Merge pull request #6 from simonsobs/docs

Documentation and several small fixes

.github/workflows/docs.yml

@@ -0,0 +1,56 @@
+name:  Deploy Documentation
+
+# on:
+#   push:
+#     tags:
+#       - '*'
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - main
+
+jobs:
+  update_docs:
+    name: Build and Deploy Documentation
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        shell: bash -l {0}
+    steps:
+      - name: Cancel Previous Runs
+        uses: styfle/[email protected]
+        with:
+          access_token: ${{ github.token }}
+
+      - name: Checkout
+        uses: actions/checkout@v3
+
+      - name: Setup Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: 3.x
+
+      - name: Create Cache ID
+        run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV
+
+      - name: Setup Cache
+        uses: actions/cache@v3
+        with:
+          key: mkdocs-material-${{ env.cache_id }}
+          path: .cache
+          restore-keys: |
+            mkdocs-material-
+
+      - name: Install Dependencies
+        run: |
+          pip install mkdocs mkdocstrings mkdocstrings-python mkdocs-material
+          git config user.name 'github-actions[bot]'
+          git config user.email 'github-actions[bot]@users.noreply.github.com'
+
+      - name: Deploy Docs
+        run: |
+          cd docs
+          mkdocs gh-deploy --force

.github/workflows/test.yml

@@ -62,7 +62,7 @@ jobs:
           conda activate "soconda_CI"
           export OMP_NUM_THREADS=2
           export MPI_DISABLE=1
-          #python3 -c 'import toast.tests; toast.tests.run()'
+          python3 -c 'import toast.tests; toast.tests.run()'
           git clone --depth=1 --single-branch --branch=master https://github.com/simonsobs/sotodlib.git
           pushd sotodlib
           python3 setup.py test

README.md

@@ -1,3 +1,5 @@
+![Documentation building status](https://github.com/simonsobs/soconda/actions/workflows/docs.yml/badge.svg) ![Latest test build](https://github.com/simonsobs/soconda/actions/workflows/test.yml/badge.svg)
+
 # Simons Observatory Conda Tools
 
 This repository contains scripts to help with:
@@ -10,204 +12,5 @@ This repository contains scripts to help with:
 
 - Creating a versioned modulefile for loading the environment
 
-## Using an Existing Environment
-
-If you are just using an already-created environment, you can follow
-instructions on project resources (Confluence) for how to load a particular
-version on a system. After loading an environment, there are a several ways to
-customize things.
-
-### Overriding a Few Packages
-
-Each `soconda` modulefile sets the user directory (for `pip install --user`) to
-be in a versioned location in your home directory. If you want to use a
-different / newer version of `sotodlib` (for example), then you can just do:
-
-    $> pip install --user https://github.com/simonsobs/sotodlib/archive/master.tar.gz
-
-If you swap to a different `soconda` environment, your user package install
-directory will also switch.
-
-### More Extensive Customization
-
-One benefit of an `soconda` environment is that it also contains conda packages
-built for some legacy compiled tools. If you want to keep those, but
-dramatically change which other conda packages are installed, you can first
-create a new personal environment while cloning an existing one:
-
-    $> conda create --clone soconda_20230809_1.0.0 /path/to/my/env
-
-Then activate your new environment and conda install whatever you like:
-
-    $> conda activate /path/to/my/env
-    $> conda install foo bar blat
-
-Note that your pip user directory will still be set to the location created by
-the original upstream module, and you can also "`pip install --user`"
-additional packages to go with your custom conda environment.
-
-### Rolling Your Own
-
-You can also just load a conda base environment on a particular system and then
-read the rest of this document to install your own environment. You can edit
-the files:
-
-    packages_conda.txt
-    packages_local.txt
-    packages_pip.txt
-
-to control which packages will be installed with conda, pip, or built locally.
-
-## Creating a Simons Observatory Environment
-
-The install script takes options to specify the location of the base environment
-(if it is not already activated) and the name (or full path) of the environment
-to create. It also allows specifying a central location to install the
-modulefile.
-
----
-**NOTE**
-
-Running the `soconda.sh` script will create log files and downloaded sources in
-the current directory. Consider running the command from within a temporary
-build directory to avoid clutter.
-
----
-
-### Base Conda Environment
-
-If you already have a conda-forge base environment, then you can skip this
-step. Otherwise, decide on the install prefix for your overall conda
-environment. For this example, we will use `/opt/conda` as the path to the
-conda base installation. Now run the bootstrap script:
-
-    $> ./tools/bootstrap_base "/opt/conda"
-
-This bootstrap will install a base system with the conda-forge channel set to
-the default. You can now source the conda initialization file and activate the
-base environment:
-
-    $> source /opt/conda/etc/profile.d/conda.sh
-    $> conda activate base
-
-After installing an `soconda` environment below, the resulting module file will
-do this on load.
-
-### Special Note on mpi4py
-
-By default, the conda package for mpi4py will be installed using the mpich
-flavor. This should work well for stand-alone workstations or single nodes. If
-you have a cluster with a customized MPI compiler, then set the `MPICC`
-environment variable to the MPI C compiler before creating an environment. That
-will cause the mpi4py package to be built using your compiler.
-
-### Example:  Local System
-
-Starting from scratch, bootstrap a small conda-forge base environment in `~/conda`:
-
-    $> ./tools/bootstrap_base.sh ~/conda
-    $> source ~/conda/etc/profile.d/conda.sh
-    $> conda activate base
-
-Create an `soconda` environment with default name and version. However, we
-decide to put all the modulefiles into a central location in the root of the
-base conda install:
-
-    $> ./soconda.sh -b ~/conda -m ~/conda/modulefiles
-
-Now we can load the module:
-
-    $> module use ~/conda/modulefiles
-    $> module avail
-    $> module load soconda/XXXXXX
-
-### Example:  NERSC
-
-At NERSC, the default provided python is from Anaconda, and cannot be easily
-customized for our needs. Instead, we have a conda-forge base system installed
-in our project software directory:
-
-    $> source /global/common/software/sobs/perlmutter/conda/etc/profile.d/conda.sh
-    $> conda activate base
-
-Now we can either install a shared software environment or use this base environment to build a conda environment in a personal directory.
-
-## Loading an Environment
-
-The created conda environment includes the root "name" (or path) specified at
-creation time and has the version appended to this. For example, if you did not
-specify the name or the version, then it would have created an environment under
-`envs/soconda_<git version>`.
-
-### Loading an Environment with a Modulefile
-
-There is a module file installed with each soconda environment. If you
-specified the module directory during install, then a module named after the
-version was created in that directory. The module file will initialize the
-conda base environment and then activate the environment you created.
-
-Ensure that the location of the modulefile is in your search path:
-
-    # (If you specified a custom module file directory)
-    $> module use /path/to/custom/modulefiles
-
-    # OR (you are using the defaults, and your base install is in /opt/conda)
-    $> module use /opt/conda/envs/soconda_1.0.0/modulefiles
-
-And then load the module:
-
-    $> module load soconda/1.0.0
-
-Doing a `module unload soconda` will deactivate the conda environment and
-remove any conda initialization from your shell environment.
-
-### Running Tests
-
-After loading an `soconda` environment, you can run some tests with:
-
-    $> ./run_tests.sh
-
-## Customizing an Environment
-
-If you want to dramatically change the package versions / content of an
-`soconda` stack, just load the existing base conda environment and edit the
-three lists of packages (`packages_[conda|pip|local].txt`) to exclude certain
-packages or add extras. Then install it to some personal location outside the
-base install (i.e. pass the full path to `soconda -e <path>`).
-
-## Deleting an Environment
-
-The `soconda` environments are self contained and you can delete them by
-removing the path or (if using a name), removing the
-`<base dir>/envs/<name of env>` directory. You can optionally delete the
-modulefile and the pip local directory in your home directory.
-
-## Advanced Details
-
-The compiled packages assume the use of the conda compilers for consistency with
-the libraries installed through conda. If you want to change those compilers you
-can remove the `compilers` conda package and manually set the `CC`, `CXX`, and `FC`
-environment variables. Full warning that this may cause problems with threading
-interfaces, thread pinning, etc, when building packages that use OpenMP.
-
-### Pixell
-
-We currently build pixell from source with the conda compilers for consistency,
-rather than installing the wheel.
-
-### So3g
-
-This package is currently built from source by default, but the pre-built wheel
-(which comes bundled with OpenMP-enabled libopenblas) should also work. To use
-that, comment out the so3g line in `packages_local.txt` and uncomment the
-line in `packages_pip.txt`.
-
-### TOAST
-
-This package is currently built from source by default, with dependencies
-installed through conda. When toast-3.0 arrives in the conda-forge toast
-feedstock, it should be added back to `packages_conda.txt`. It should also work
-to install the python wheel package by commenting out the toast entry in
-`packages_local.txt` and adding it to `packages_pip.txt`.
-
+## [Full Documentation](https://simonsobs.github.io/soconda/)
 

docs/docs/automate.md

@@ -0,0 +1,22 @@
+# Automating the Install
+
+The general process of automating the install proceeds like this:
+
+1. A cron job (or similar) runs periodically and launches a script which
+examines current tags of soconda and finds the latest one. It looks at the
+installation directory and determines whether this tag has been installed. If
+not, it runs an installation wrapper script for the current system.
+
+2. The wrapper script has hard-coded paths to the various locations and also
+knows how to build the version string to use. This script calls the
+`soconda.sh` script to install the latest tag.
+
+3. If a new tag was installed, a message is posted via slack hook to one of the
+Simons Observatory slack channels.
+
+## Example:  Perlmutter at NERSC
+
+In the `deploy` subdirectory are the relevant files. The perlmutter install
+wrapper script is in `install_perlmutter.sh`. The tag checking script run by
+the cron job is in `install_nersc_tag.sh`, and the example lines in the scron
+tab are in `scron_check_tag.slurm`.

docs/docs/dev.md

@@ -0,0 +1,57 @@
+# Developer Notes
+
+This section covers steps needed when maintaining the `soconda` tools.
+
+## Releases
+
+After updating the versions of included packages, it is a good idea to make a
+new release so that a new environment is built and deployed. After all
+outstanding changes are merged to `main`, go to the github page for this
+repository and click on the `Actions` tab. Trigger the `Test Build` workflow.
+This will take an hour or so to run. Assuming it all works, create a new
+release with some notes about what was updated. Cron jobs running on our
+computing centers will check for new tags and build them if found.
+
+## Changing Installed Packages
+
+The three top-level files: `packages_conda.txt`, `packages_local.txt`, and
+`packages_pip.txt` contain the packages that will be installed. The conda
+packages are installed first, then local conda packages are built and
+installed, and finally the pip packages are installed. When installing pip
+packages, the `soconda.sh` script first extracts the dependencies of each pip
+package and installs it instead with conda. This helps minimize the number of
+pip packages installed in the environment and that makes it easier to avoid
+dependency problems later.
+
+## Updating Bundled Recipes
+
+The conda recipes for bundled packages should be updated whenever upstream
+packages have new releases. This involves the following steps:
+
+1.  Update the version in the `meta.yaml` file of the package recipe.
+
+2.  Update the download URL if needed.
+
+3.  Manually download the new tarball of the package.  For example:
+
+        curl -SL -o pixel-0.19.0.tar.gz \
+        https://github.com/simonsobs/pixell/archive/v0.19.0.tar.gz
+
+4. Get the sha256 checksum of the tarball, and copy that into the `meta.yaml`
+entry:
+
+        openssl sha256 pixel-0.19.0.tar.gz
+        SHA256(pixell-0.19.0.tar.gz)= 8142a2a368175de845166afffe3e4efd0ac0bdc109a96eb8f4cc0360e6191fd1
+
+5. Ensure that the new version of the package does not have any updated
+dependencies or other constraints.
+
+## Adding New Package Recipes
+
+This should not be needed very often, and will require some familiarity with
+conda recipes. Create a new directory for the recipe in the `pkgs` directory.
+Add a `meta.yaml` file, a `build.sh` file, and a copy of the package license.
+See the conda documentation and existing conda-forge feedstocks for extensive
+examples. You can load an existing soconda environment as a testbed and then
+test your new recipe with `conda build`. After you can build it independently,
+add it to the `packages_local.txt` file.

docs/docs/index.md

@@ -0,0 +1,15 @@
+# Simons Observatory Conda Tools
+
+[This git repository](https://github.com/simonsobs/soconda) contains scripts to
+help with:
+
+- Installing a conda base system (optional)
+
+- Creating a conda environment with a well-defined set of conda and pip packages
+
+- Building a few legacy compiled packages into this environment
+
+- Creating a versioned modulefile for loading the environment
+
+The documentation also covers different user scenarios for loading and
+customizing a pre-installed environment.