Merge pull request #791 from lsst/tickets/DM-39183

Update qserv-ingest documentation

doc/admin/index.rst

@@ -3,17 +3,13 @@
    **Information in this guide is known to be outdated.** A documentation sprint is underway which will
    include updates and revisions to this guide.
 
-.. highlight:: sql
-
 ####################
 Administration Guide
 ####################
 
 .. toctree::
    :maxdepth: 4
 
-   docker
    k8s
    qserv-ingest/index
-   configuration
    test-set

doc/admin/k8s.rst

@@ -1,5 +1,26 @@
+##############################
 Run Qserv on top of Kubernetes
 ##############################
 
+Deployment
+==========
+
 The offical way to install Qserv is to run it on top of Kubernetes using
 the `qserv-operator <https://qserv-operator.lsst.io>`_.
+
+Administration
+==============
+
+Access to Qserv dashboard
+-------------------------
+
+This access is only available for users having access to the cluster with ``kubectl``.
+
+.. code:: bash
+
+    # Open a tunnel between local machine and Qserv dashboard
+    # # Listen on first port number locally, forwarding to second port number in the pod
+    kubectl port-forward qserv-repl-ctl-0 8080:8080 &
+
+    # Access to the dashboard
+    firefox http://localhost:8080

doc/admin/qserv-ingest/ingest.yaml

@@ -0,0 +1,69 @@
+version: 15
+ingest:
+    http:
+        # Optional, default to no time-out
+        # Timeout for POST and PUT queries in seconds
+        # POST and PUT queries which might be very long for the following operations:
+        # - posting contribution
+        # - closing transaction
+        # - publishing database
+        # - building the "director" index
+        # - building table indexes at workers
+        write_timeout: 1800
+        # Optional, default to no time-out
+        # Timeout for GET queries in seconds
+        read_timeout: 10
+    metadata:
+      # Optional, default to "ingest.input.servers[0]/ingest.input.path"
+      # Allow to customize metadata URL
+      url: http://dataserver/datasets/DC2/
+
+    input:
+        # List of http servers providing input dataset
+        # The ingest process will load-balance the download of input files accross
+        # these servers.
+        # Use file:// as first element in list when using local data
+        # TODO Add support for webdav protocol
+        servers:
+            - http://dataserver
+            - http://dataserver
+        # Path to input data on the http servers
+        path: datasets/DC2/
+
+    ## URLs of Qserv services
+    ## ----------------------
+    qserv:
+        # URL which serves Qserv SQL queries
+        query_url: "mysql://qsmaster:@qserv-czar:4040"
+        # URL which serves input chunk contributions queue
+        queue_url: "mysql://qsingest:@qserv-ingest-db-0.qserv-ingest-db/qservIngest"
+        # Replication controller service URL
+        replication_url: http://qserv-repl-ctl-0.qserv-repl-ctl:8080
+
+    ## Configure replication service
+    ## Documented at https://confluence.lsstcorp.org/display/DM/1.+Setting+configuration+parameters
+    ## --------------------------------------------------------------------------------------------
+    ingestservice:
+        # Optional, default to None
+        # Override the default value stored in input metadata (in database.json file)
+        # 1: build secondary index when closing a transaction
+        # 0: build secondary index after ingest
+        auto_build_secondary_index: 1
+
+        # Optional, default to "/etc/pki/tls/certs/ca-bundle.crt"
+        # cainfo: "/etc/pki/tls/certs/ca-bundle.crt"
+
+        # Optional, default to 1
+        # ssl_verifypeer: 1
+
+        # Optional, default to None
+        # Override the default Qserv value
+        async_proc_limit: 4
+
+        # Optional, default to None
+        # Override the default Qserv value
+        low_speed_limit: 10
+
+        # Optional, default to None
+        # Override the default Qserv value
+        low_speed_time: 3600

doc/admin/qserv-ingest/run.rst

@@ -56,8 +56,8 @@ For additional informations, check official documentation for `setting the names
 
 - `git-lfs <https://git-lfs.com>`__ is required in order to retrieve integration test datasets.
 
-Prepare Qserv ingest
-====================
+Prepare and configure Qserv ingest
+==================================
 
 Get the project
 ---------------
@@ -81,13 +81,18 @@ A directory with ingest workflow configuration and an ``env.sh`` file have to be
     cp env.example.sh env.sh
 
 #. In ``env.sh``, set ``OVERLAY`` to ``<CUSTOM_INGEST>``, and eventually ``INSTANCE`` to the name of current Qserv instance.
-#. In ingest workflow configuration file (named ``manifests/<CUSTOM_INGEST>/configmap/ingest.yaml``), set:
+#. Configure ingest workflow configuration file (named ``manifests/<CUSTOM_INGEST>/configmap/ingest.yaml``):
+
+..
+    Copy example configuration file in current directory in order to update documentation
+    cp ~/src/qserv-ingest/manifests/base/configmap/ingest.yaml ~/src/qserv/doc/admin/qserv-ingest/
+
+Inline documentation for this configuration file is avalaible at ``manifests/base/configmap/ingest.yaml``:
+
+.. literalinclude:: ingest.yaml
+   :language: yaml
+   :linenos:
 
-   #. ``ingest/input/servers`` to the list of http servers providing input dataset,  The ingest process will load-balance the download of input files accross these servers.
-   #. ``ingest/input/path`` to the **path to input data on the http server**
-   #. ``ingest/qserv/query_url`` to the URL which serve Qserv SQL queries
-   #. ``ingest/qserv/queue_url`` to the URL which serve input chunk contributions queue
-   #. ``ingest/qserv/replication_url`` to the URL which serve replication controller service
 
 Launch Qserv ingest
 ===================

doc/admin/test-set.rst

@@ -2,6 +2,9 @@
 Test sets
 *********
 
+.. warning::
+    DEPRECATED
+
 Integration tests
 =================
 
@@ -28,7 +31,6 @@ Directory structure
 ^^^^^^^^^^^^^^^^^^^
 
 ::
-
     case<number>/
         README.txt - contains info about data
         queries/

doc/dev/doc.rst

@@ -2,7 +2,7 @@
 Generate documentation
 ######################
 
-Documentation is automatically built and generated on each Github Action build. This can also be performed manually by launching script below:
+Documentation is automatically built and generated on each Github Action build. This may also be done locally by manually running the following script:
 
 .. code:: sh
 
@@ -11,4 +11,12 @@ Documentation is automatically built and generated on each Github Action build.
     qserv build-images --pull-image
     # Require '--user=qserv' if uid=1000 on host machine
     rm -rf $QSERV_SRC_DIR/build/doc
-    qserv build-docs --cmake --linkcheck --user=qserv
+    # Run this only command to regenerate documentation
+    qserv build-docs --cmake --linkcheck --user=qserv
+
+Documentation can then be accessed locally with a web browser:
+
+.. code:: sh
+
+    # Access documentation
+    firefox $QSERV_SRC_DIR/build/doc/html/index.html