From 902c373f533f506c9f9a64222182d0b012d96325 Mon Sep 17 00:00:00 2001 From: Fabrice Jammes Date: Wed, 31 May 2023 11:20:38 +0200 Subject: [PATCH 1/2] Update qserv-ingest documentation --- doc/admin/configuration.rst | 51 ------------- doc/admin/docker.rst | 112 ----------------------------- doc/admin/index.rst | 4 -- doc/admin/k8s.rst | 20 ++++++ doc/admin/qserv-ingest/ingest.yaml | 69 ++++++++++++++++++ doc/admin/qserv-ingest/run.rst | 21 +++--- doc/admin/test-set.rst | 4 +- doc/dev/doc.rst | 12 +++- 8 files changed, 115 insertions(+), 178 deletions(-) delete mode 100644 doc/admin/configuration.rst delete mode 100644 doc/admin/docker.rst create mode 100644 doc/admin/qserv-ingest/ingest.yaml diff --git a/doc/admin/configuration.rst b/doc/admin/configuration.rst deleted file mode 100644 index 5e7ea7b4d6..0000000000 --- a/doc/admin/configuration.rst +++ /dev/null @@ -1,51 +0,0 @@ -*********************** -Configuration procedure -*********************** - -This documentation presents Qserv configuration tool, designed to simplify and automate Qserv installation process. - -Goals -===== - -Qserv installation procedure consists of next steps : - -.. _eups: https://github.com/RobertLuptonTheGood/eups - -- download, build and install Qserv using eups_, -- configure Qserv. - -The goal is to have a modular procedure in order to ease future evolutions and maintenance. -That's why Qserv configuration procedure must be as independant as possible of eups and other steps eups. - -Approach -======== - -eups install directory must only contains immutable data like, for example, binaries. -That's why Qserv configuration tool creates two directories: - -- QSERV_RUN_DIR which contains configuration (file and data) and execution informations (pid files, log files) -- QSERV_DATA_DIR which contains scientific data (i.e. sky data), - -The configuration tool is written in pure-python and offers several options (see --help). - -.. code-block:: bash - - qserv-configure.py --help - -Features -======== - -- Use meta-configuration file QSERV_RUN_DIR/qserv-meta.conf, and templates in QSERV_DIR/cfg/templates to generate all configuration -- Run configuration scripts for Qserv services when required (i.e. for mysql, scisql and xrootd) -- Protect existing data, if located outside of configuration directory -- Create client configuration in ~/.lsst - -Tickets completed -================= - -- https://jira.lsstcorp.org/browse/DM-622 -- https://jira.lsstcorp.org/browse/DM-930 -- https://jira.lsstcorp.org/browse/DM-595 -- https://jira.lsstcorp.org/browse/DM-895 -- https://jira.lsstcorp.org/browse/DM-2595 - diff --git a/doc/admin/docker.rst b/doc/admin/docker.rst deleted file mode 100644 index 9fa743a1c8..0000000000 --- a/doc/admin/docker.rst +++ /dev/null @@ -1,112 +0,0 @@ -################# -Docker containers -################# - -Qserv is also available as a Docker image (see https://www.docker.com/). - -.. note:: - - Our continuous integration service https://travis-ci.org/lsst/qserv produces automatically - docker containers for each commit pushed to Github. Containers names are displayed on top of travis console. - -.. note:: - - This procedure was tested with Docker 1.13. - -.. _docker-github: - -************************************** -Build Qserv image from a Github branch -************************************** - -Create Qserv latest release image, with 3 names: - -- qserv:latest -- qserv:dev -- qserv:YYY_MM - -.. code-block:: bash - - . /path/to/lsst/stack/loadLSST.bash - cd ${SRC_DIR}/qserv/admin/tools/docker - 1_build-image.sh -C - -Create Qserv cutting-edge dependencies image, named qserv:dev: - -.. code-block:: bash - - # cutting-edge dependencies needs to be tagged eups-dev - # on distribution server. - cd ${SRC_DIR}/qserv/admin/tools/docker - 1_build-image.sh -CD - -Create Qserv image for a given git tag/branch: - -In order to push produced Docker images to Docker Hub, prefix -```` with ``qserv/qserv:`` in instructions below. - -.. code-block:: bash - - # Code need to be pushed on github - cd ${SRC_DIR}/qserv/admin/tools/docker - 2_build-git-image.sh -R -T - # Current Qserv version will have eups tag named qserv-dev - -Create Qserv master and worker images from a given Qserv version: - -.. code-block:: bash - - # Code need to be pushed on github - # eups tag named qserv-dev will be used to setup Qserv version - cd ${SRC_DIR}/qserv/admin/tools/docker - 4_build-configured-image.sh -i - -********* -Use cases -********* - -Test latest release -=================== - -Run mono-node integration test against latest Qserv release: - -.. code-block:: bash - - docker run -it --rm \ - -h $(hostname)-docker -u qserv \ - fjammes/qserv:latest \ - /qserv/scripts/mono-node-test.sh - -Develop and test -================ - -Create Qserv image for a developer workstation: - -.. code-block:: bash - - cd ${SRC_DIR}/qserv/admin/tools/docker - build-work-image.sh - # change uid in image so that it match host user id - # in order to mount rw user source code in container - change-uid.sh - - -Set uid in qserv:dev w.r.t your host machine user account. This creates a qserv:dev-uid image which can mount host source code in a container with correct permissions. -Host might be a development machine or a continuous integration server. - -.. code-block:: bash - - ./change-uid.sh - -Build, configure and run Qserv from source in qserv:work-uid container using -source and run directory located on a development machine. - -.. code-block:: bash - - docker run -it --rm \ - -h $(hostname)-docker \ - -v /home/qserv/src/qserv:/home/dev/src/qserv \ - -v /home/qserv/qserv-run/:/home/dev/qserv-run \ - -u dev \ - fjammes/qserv:work-uid \ - /bin/sh -c "/home/dev/scripts/build.sh && /qserv/scripts/mono-node-test.sh" diff --git a/doc/admin/index.rst b/doc/admin/index.rst index b62dc30015..d9f6e3f44b 100644 --- a/doc/admin/index.rst +++ b/doc/admin/index.rst @@ -3,8 +3,6 @@ **Information in this guide is known to be outdated.** A documentation sprint is underway which will include updates and revisions to this guide. -.. highlight:: sql - #################### Administration Guide #################### @@ -12,8 +10,6 @@ Administration Guide .. toctree:: :maxdepth: 4 - docker k8s qserv-ingest/index - configuration test-set diff --git a/doc/admin/k8s.rst b/doc/admin/k8s.rst index b6d46a58e0..7deddf8ff4 100644 --- a/doc/admin/k8s.rst +++ b/doc/admin/k8s.rst @@ -1,5 +1,25 @@ +############################## Run Qserv on top of Kubernetes ############################## +Deployment +========== + The offical way to install Qserv is to run it on top of Kubernetes using the `qserv-operator `_. + +Administration +============== + +Access to Qserv dashboard +------------------------- + +This access is only available for users having access to the cluster with ``kubectl``. + +.. code:: bash + + # Open a tunnel between local machine and Qserv dashboard + kubectl port-forward qserv-repl-ctl-0 8080 & + + # Access to the dashboard + firefox http://localhost:8080 \ No newline at end of file diff --git a/doc/admin/qserv-ingest/ingest.yaml b/doc/admin/qserv-ingest/ingest.yaml new file mode 100644 index 0000000000..04048a22b3 --- /dev/null +++ b/doc/admin/qserv-ingest/ingest.yaml @@ -0,0 +1,69 @@ +version: 15 +ingest: + http: + # Optional + # Timeout for POST and PUT queries in seconds + # POST and PUT queries which might be very long like: + # - postint contribution + # - closing transaction + # - publishing database + # have no timeout at all + write_timeout: 1800 + # Optional + # Timeout for GET queries in seconds + read_timeout: 10 + metadata: + # Optional + # Allow to customize metadata URL + # Default to "{servers[0]}{path}" + url: http://dataserver/datasets/DC2/ + + input: + # List of http servers providing input dataset + # The ingest process will load-balance the download of input files accross + # these servers. + # Use file:// as first element in list when using local data + # TODO Add support for webdav protocol + servers: + - http://dataserver + - http://dataserver + # Path to input data on the http servers + path: datasets/DC2/ + + ## URLs of Qserv services + ## ---------------------- + qserv: + # URL which serves Qserv SQL queries + query_url: "mysql://qsmaster:@qserv-czar:4040" + # URL which serves input chunk contributions queue + queue_url: "mysql://qsingest:@qserv-ingest-db-0.qserv-ingest-db/qservIngest" + # Replication controller service URL + replication_url: http://qserv-repl-ctl-0.qserv-repl-ctl:8080 + + ## Configure replication service + ## Documented at https://confluence.lsstcorp.org/display/DM/1.+Setting+configuration+parameters + ## -------------------------------------------------------------------------------------------- + ingestservice: + # Optional, default to None + # Override the default value stored in input metadata (in database.json file) + # 1: build secondary index when closing a transaction + # 0: build secondary index after ingest + auto_build_secondary_index: 1 + + # Optional, default to "/etc/pki/tls/certs/ca-bundle.crt" + # cainfo: "/etc/pki/tls/certs/ca-bundle.crt" + + # Optional, default to 1 + # ssl_verifypeer: 1 + + # Optional, default to None + # Override the default Qserv value + async_proc_limit: 4 + + # Optional, default to None + # Override the default Qserv value + low_speed_limit: 10 + + # Optional, default to None + # Override the default Qserv value + low_speed_time: 3600 diff --git a/doc/admin/qserv-ingest/run.rst b/doc/admin/qserv-ingest/run.rst index 1e7e64934e..d9b1fe7bf4 100644 --- a/doc/admin/qserv-ingest/run.rst +++ b/doc/admin/qserv-ingest/run.rst @@ -56,8 +56,8 @@ For additional informations, check official documentation for `setting the names - `git-lfs `__ is required in order to retrieve integration test datasets. -Prepare Qserv ingest -==================== +Prepare and configure Qserv ingest +================================== Get the project --------------- @@ -81,13 +81,18 @@ A directory with ingest workflow configuration and an ``env.sh`` file have to be cp env.example.sh env.sh #. In ``env.sh``, set ``OVERLAY`` to ````, and eventually ``INSTANCE`` to the name of current Qserv instance. -#. In ingest workflow configuration file (named ``manifests//configmap/ingest.yaml``), set: +#. Configure ingest workflow configuration file (named ``manifests//configmap/ingest.yaml``): + +.. + Copy example configuration file in current directory in order to update documentation + cp ~/src/qserv-ingest/manifests/base/configmap/ingest.yaml ~/src/qserv/doc/admin/qserv-ingest/ + +Inline documentation for this configuration file is avalaible at ``manifests/base/configmap/ingest.yaml``: + +.. literalinclude:: ingest.yaml + :language: yaml + :linenos: - #. ``ingest/input/servers`` to the list of http servers providing input dataset, The ingest process will load-balance the download of input files accross these servers. - #. ``ingest/input/path`` to the **path to input data on the http server** - #. ``ingest/qserv/query_url`` to the URL which serve Qserv SQL queries - #. ``ingest/qserv/queue_url`` to the URL which serve input chunk contributions queue - #. ``ingest/qserv/replication_url`` to the URL which serve replication controller service Launch Qserv ingest =================== diff --git a/doc/admin/test-set.rst b/doc/admin/test-set.rst index 78b28dd21f..8d8a31136a 100644 --- a/doc/admin/test-set.rst +++ b/doc/admin/test-set.rst @@ -2,6 +2,9 @@ Test sets ********* +.. warning:: + DEPRECATED + Integration tests ================= @@ -28,7 +31,6 @@ Directory structure ^^^^^^^^^^^^^^^^^^^ :: - case/ README.txt - contains info about data queries/ diff --git a/doc/dev/doc.rst b/doc/dev/doc.rst index 17f04609b8..79ea9b6a20 100644 --- a/doc/dev/doc.rst +++ b/doc/dev/doc.rst @@ -2,7 +2,7 @@ Generate documentation ###################### -Documentation is automatically built and generated on each Github Action build. This can also be performed manually by launching script below: +Documentation is automatically built and generated on each Github Action build. This may also be done locally by manually running the following script: .. code:: sh @@ -11,4 +11,12 @@ Documentation is automatically built and generated on each Github Action build. qserv build-images --pull-image # Require '--user=qserv' if uid=1000 on host machine rm -rf $QSERV_SRC_DIR/build/doc - qserv build-docs --cmake --linkcheck --user=qserv \ No newline at end of file + # Run this only command to regenerate documentation + qserv build-docs --cmake --linkcheck --user=qserv + +Documentation can then be accessed locally with a web browser: + +.. code:: sh + + # Access documentation + firefox $QSERV_SRC_DIR/build/doc/html/index.html \ No newline at end of file From b8def32f8c39f731419eb66b1f88aa9d0c95266f Mon Sep 17 00:00:00 2001 From: Fabrice Jammes Date: Wed, 21 Jun 2023 22:48:14 +0200 Subject: [PATCH 2/2] Add minor fixes in documentation --- doc/admin/k8s.rst | 3 ++- doc/admin/qserv-ingest/ingest.yaml | 14 +++++++------- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/doc/admin/k8s.rst b/doc/admin/k8s.rst index 7deddf8ff4..989773640c 100644 --- a/doc/admin/k8s.rst +++ b/doc/admin/k8s.rst @@ -19,7 +19,8 @@ This access is only available for users having access to the cluster with ``kube .. code:: bash # Open a tunnel between local machine and Qserv dashboard - kubectl port-forward qserv-repl-ctl-0 8080 & + # # Listen on first port number locally, forwarding to second port number in the pod + kubectl port-forward qserv-repl-ctl-0 8080:8080 & # Access to the dashboard firefox http://localhost:8080 \ No newline at end of file diff --git a/doc/admin/qserv-ingest/ingest.yaml b/doc/admin/qserv-ingest/ingest.yaml index 04048a22b3..0e675aa097 100644 --- a/doc/admin/qserv-ingest/ingest.yaml +++ b/doc/admin/qserv-ingest/ingest.yaml @@ -1,21 +1,21 @@ version: 15 ingest: http: - # Optional + # Optional, default to no time-out # Timeout for POST and PUT queries in seconds - # POST and PUT queries which might be very long like: - # - postint contribution + # POST and PUT queries which might be very long for the following operations: + # - posting contribution # - closing transaction # - publishing database - # have no timeout at all + # - building the "director" index + # - building table indexes at workers write_timeout: 1800 - # Optional + # Optional, default to no time-out # Timeout for GET queries in seconds read_timeout: 10 metadata: - # Optional + # Optional, default to "ingest.input.servers[0]/ingest.input.path" # Allow to customize metadata URL - # Default to "{servers[0]}{path}" url: http://dataserver/datasets/DC2/ input: