Deploying to master from @ ZwickyTransientFacility/scope-ml@e6903ab 🚀

ZwickyTransientFacility · Sep 8, 2023 · d59557d · d59557d
commit d59557d
Show file tree

Hide file tree

Showing 152 changed files with 10,175 additions and 0 deletions.
diff --git a/.buildinfo b/.buildinfo
@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: 7e76fb2d63321ec3c347e52311e09cea
+tags: 645f666f9bcd5a90fca523b33c5a78b7
diff --git a/.nojekyll b/.nojekyll
diff --git a/CNAME b/CNAME
@@ -0,0 +1 @@
+scope.ztf.dev
diff --git a/_images/beta_lyr.png b/_images/beta_lyr.png
diff --git a/_images/bogus_1.png b/_images/bogus_1.png
diff --git a/_images/bogus_2.png b/_images/bogus_2.png
diff --git a/_images/cepheid_1O.png b/_images/cepheid_1O.png
diff --git a/_images/cepheid_F.png b/_images/cepheid_F.png
diff --git a/_images/cepheid_F1O.png b/_images/cepheid_F1O.png
diff --git a/_images/cepheid_F_1.png b/_images/cepheid_F_1.png
diff --git a/_images/cv_Novalike.png b/_images/cv_Novalike.png
diff --git a/_images/cv_SU_UMa.png b/_images/cv_SU_UMa.png
diff --git a/_images/cv_U_Gem.png b/_images/cv_U_Gem.png
diff --git a/_images/cv_Z_Cam.png b/_images/cv_Z_Cam.png
diff --git a/_images/delta_scuti.png b/_images/delta_scuti.png
diff --git a/_images/flaring_dwarfnova.png b/_images/flaring_dwarfnova.png
diff --git a/_images/flaring_mdwarf.png b/_images/flaring_mdwarf.png
diff --git a/_images/hr__beta_lyr.png b/_images/hr__beta_lyr.png
diff --git a/_images/hr__cepheid.png b/_images/hr__cepheid.png
diff --git a/_images/hr__cv.png b/_images/hr__cv.png
diff --git a/_images/hr__delta_scuti.png b/_images/hr__delta_scuti.png
diff --git a/_images/hr__flaring.png b/_images/hr__flaring.png
diff --git a/_images/hr__lpv.png b/_images/hr__lpv.png
diff --git a/_images/hr__periodic.png b/_images/hr__periodic.png
diff --git a/_images/hr__rr_lyr.png b/_images/hr__rr_lyr.png
diff --git a/_images/hr__variable.png b/_images/hr__variable.png
diff --git a/_images/hr__w_uma.png b/_images/hr__w_uma.png
diff --git a/_images/mira.png b/_images/mira.png
diff --git a/_images/period__beta_lyr.png b/_images/period__beta_lyr.png
diff --git a/_images/period__cepheid.png b/_images/period__cepheid.png
diff --git a/_images/periodic.png b/_images/periodic.png
diff --git a/_images/radec__cepheid.png b/_images/radec__cepheid.png
diff --git a/_images/radec__delta_scuti.png b/_images/radec__delta_scuti.png
diff --git a/_images/radec__flaring.png b/_images/radec__flaring.png
diff --git a/_images/radec__lpv.png b/_images/radec__lpv.png
diff --git a/_images/radec__periodic.png b/_images/radec__periodic.png
diff --git a/_images/radec__rr_lyr.png b/_images/radec__rr_lyr.png
diff --git a/_images/radec__variable.png b/_images/radec__variable.png
diff --git a/_images/radec__w_uma.png b/_images/radec__w_uma.png
diff --git a/_images/rr_lyr_ab.png b/_images/rr_lyr_ab.png
diff --git a/_images/rr_lyr_c.png b/_images/rr_lyr_c.png
diff --git a/_images/srv.png b/_images/srv.png
diff --git a/_images/variable.png b/_images/variable.png
diff --git a/_images/w_uma.png b/_images/w_uma.png
diff --git a/_sources/developer.md.txt b/_sources/developer.md.txt
@@ -0,0 +1,236 @@
+# Installation/Developer Guidelines
+
+## Initial steps
+
+- Create your own fork the [scope repository](https://github.com/ZwickyTransientFacility/scope) by clicking the "fork" button. Then, decide whether you would like to use HTTPS (easier for beginners) or SSH.
+- Following one set of instructions below, clone (download) your copy of the repository, and set up a remote called `upstream` that points to the main `scope` repository.
+
+### HTTPS:
+
+```shell script
+git clone https://github.com/<yourname>/scope.git && cd scope
+git remote add upstream https://github.com/ZwickyTransientFacility/scope.git
+```
+
+### SSH:
+
+- [Set up SSH authentication with GitHub](https://help.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh).
+
+```shell script
+git clone [email protected]:<yourname>/scope.git && cd scope
+git remote add upstream [email protected]:ZwickyTransientFacility/scope.git
+```
+
+## Setting up your environment (Windows/Linux/macOS)
+
+#### Use a package manager for installation
+
+We currently recommend running `scope` with Python 3.10. You may want to begin your installation by creating/activating a virtual environment, for example using conda. We specifically recommend installing miniforge3 (https://github.com/conda-forge/miniforge).
+
+Once you have a package manager installed, run:
+
+```bash
+conda create -n scope-env -c conda-forge python=3.10 healpy
+conda activate scope-env
+```
+
+#### Update your `PYTHONPATH`
+
+Ensure that Python can import from `scope` by modifying the `PYTHONPATH` environment variable. Use a simple text editor like `nano` to modify the appropriate file (depending on which shell you are using). For example, if using bash, run `nano ~/.bash_profile` and add the following line:
+
+```bash
+export PYTHONPATH="$PYTHONPATH:$HOME/scope"
+```
+
+Save the updated file (`Ctrl+O` in `nano`) and close/reopen your terminal for this change to be recognized. Then `cd` back into scope and activate your `scope-env` again.
+
+#### Install pre-commit
+
+We use `black` to format the code and `flake8` to verify that code complies with [PEP8](https://www.python.org/dev/peps/pep-0008/).
+Please install our pre-commit hook as follows:
+
+```shell script
+pip install pre-commit
+pre-commit install
+```
+
+This will check your changes before each commit to ensure that they
+conform with our code style standards. We use black to reformat Python
+code.
+
+The pre-commit hook will lint *changes* made to the source.
+
+---
+
+### Instructions for macOS (ARM64/Apple Silicon only)
+
+Skip to the next section if using Windows/Linux or macOS (AMD64/Intel).
+
+#### Install tensorflow for macOS/Apple Silicon
+
+You will need to install the correct version of tensorflow for the ARM64 architecture used by Macs with Apple Silicon. Apple provides an effective version here: `https://developer.apple.com/metal/tensorflow-plugin/`.
+
+To install tensorflow for macOS (including a version that runs on GPUs), run:
+```bash
+pip install tensorflow-macos
+pip install tensorflow-metal
+```
+
+#### Modify remaining requirements
+
+After successfully installing tensorflow-deps, tensorflow-macos, tensorflow-metal, modify the remaining software requirements before installing anything else. Overwrite the `.requirements/dev.txt` file with the updated requirements from `.requirements/dev-M1.txt` by running the following from the scope directory:
+
+```bash
+cp -f .requirements/dev-M1.txt .requirements/dev.txt
+```
+
+This removes `tensorflow` and `tensorflow-addons` from `.requirements/dev.txt` . The file should now list the following requirements:
+
+```txt
+deepdiff>=5.0
+gsutil>=4.60
+keras-tuner>=1.0.2
+matplotlib>=3.3
+pytest>=6.1.2
+questionary>=1.8.1
+scikit-learn>=0.24.1
+wandb>=0.12.1
+```
+
+Continue the installation by following the below instructions for all operating systems.
+
+---
+### All Windows/Linux/macOS
+
+#### Install required packages
+
+Install the required python packages by running:
+```bash
+pip install -r requirements.txt
+```
+
+#### Create and modify config.yaml
+
+From the included config.defaults.yaml, make a copy called config.yaml:
+
+```bash
+cp config.defaults.yaml config.yaml
+```
+
+Edit config.yaml to include Kowalski instance and Fritz tokens in the associated empty `token:` fields.
+
+#### Testing
+Run `./scope.py test` to test your installation. Note that for the test to pass, you will need access to the Kowalski database. If you do not have Kowalski access, you can run `./scope.py test_limited` to run a more limited (but still useful) set of tests.
+
+#### Troubleshooting
+Upon encountering installation/testing errors, manually install the package in question using  `conda install xxx` , and remove it from `.requirements/dev.txt`. After that, re-run `pip install -r requirements.txt` to continue.
+
+#### Known issues
+- Across all platforms, we are currently aware of `scope` dependency issues with Python 3.11.
+- Anaconda continues to cause problems with environment setup.
+- Using `pip` to install `healpy` on an arm64 Mac can raise an error upon import. We recommend including `healpy` as a requirement during the creation of your `conda` environment.
+- On Windows machines, `healpy` and `cesium` raise errors upon installation.
+   - For `healpy`, see [this](https://healpy.readthedocs.io/en/latest/install.html#installation-on-windows-through-the-windows-subsystem-for-linux) guide for a potential workaround.
+   - For `cesium`, try to install from the source (https://cesium-ml.org/docs/install.html#from-source) within `scope`. If you will not be running feature generation, this is not a critical error, but there will be points in the code that fail (e.g. `scope.py test`, `tools/generate_features.py`)
+
+If the installation continues to raise errors, update the conda environment and try again.
+
+### How to contribute
+
+Contributions to `scope` are made through [GitHub Pull Requests](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests), a set of proposed commits (or patches):
+
+1. Download the latest version of `scope`, and create a new branch for your work.
+
+   Here, let's say we want to contribute some documentation fixes; we'll call our branch `rewrite-contributor-guide`.
+
+   ```shell script
+   git checkout main
+   git pull upstream main
+   git checkout -b rewrite-contributor-guide
+   ```
+
+2. Make modifications to `scope` and commit your changes using `git add` and `git commit`.
+Each commit message should consist of a summary line and a longer description, e.g.:
+
+   ```text
+   Rewrite the contributor guide
+   While reading through the contributor guide, I noticed several places
+   in which instructions were out of order. I therefore reorganized all
+   sections to follow logically, and fixed several grammar mistakes along
+   the way.
+   ```
+
+1. When ready, push your branch to GitHub:
+
+   ```shell script
+   git push origin rewrite-contributor-guide
+   ```
+
+   Once the branch is uploaded, GitHub should print a URL for turning your branch into a pull request.
+   Open that URL in your browser, write an informative title and description for your pull request, and submit it.
+
+2. The team will now review your contribution, and suggest changes.
+*To simplify review, please limit pull requests to one logical set of changes.*
+To incorporate changes recommended by the reviewers, commit edits to your branch, and push to the branch again
+(there is no need to re-create the pull request, it will automatically track modifications to your branch).
+
+1. Sometimes, while you were working on your feature, the `main` branch is updated with new commits, potentially
+resulting in conflicts with your feature branch. The are two ways to resolve this situation - merging and rebasing,
+please look [here](https://www.atlassian.com/git/tutorials/merging-vs-rebasing) for a detailed discussion.
+While both ways are acceptable, since we are squashing commits from a PR before merging, we prefer the first option:
+
+    ```shell script
+    git merge rewrite-contributor-guide upstream/main
+    ```
+Developers may merge `main` into their branch as many times as they want to.
+
+1. Once the pull request has been reviewed and approved by at least one team member, it will be merged into `scope`.
+
+## Contributing Field Guide sections
+
+If you would like to contribute a Field Guide section, please follow the steps below.
+
+- Make sure to follow the steps described above in the "How to contribute" section!
+
+- Add sections to `config.defaults.yaml` under `docs.field_guide.<object_class_type>`.
+  - Use `docs.field_guide.rr_lyr_ab` as an example. You need to specify the object's
+    coordinates and a title for the generated light curve plot. Optionally,
+    you may specify a period [days] - then a phase-folded light curve will also be rendered.
+
+- Make sure your `config.yaml` file contains a valid Kowalski token.
+  - See [here](https://github.com/dmitryduev/penquins) on how to generate one
+  (Kowalski account required).
+  - You can use `config.defaults.yaml` as a template.
+
+- Make sure the structure of your config file is the same as the default,
+  i.e. you propagated the changes in `config.defaults.yaml`.
+  (otherwise the `scope.py` utility will later complain and ask you to fix that).
+
+- Add a Markdown file to `doc/` and call it `field_guide__<object_class>.md`.
+  - Use [`doc/field_guide__rr_lyrae.md`](field_guide__rr_lyrae.md) as a template.
+  - Light curve examples will be generated as `data/<object_class_type>.png` files using the info
+    you provided in the config.
+  - Add the following include statements to [`doc/field_guide.md`](field_guide.md):
+
+```
+{include} ./field_guide__<object_class>.md
+```
+
+- If you wish to render a sample Gaia-based HR diagram, you need to create a "Golden" data set
+  for that class of objects and put it under `data/golden` as `<object_class>.csv`
+  - The `csv` file must follow the same structure as [`data/golden/rr_lyr.csv`].
+   Please keep the `csv` header ("ra,dec") and provide object coordinates in degrees.
+  - The HR diagram will be generated as `data/hr__<object_class>.png`
+
+- Run the `./scope.py doc` command to generate the imagery and build the documentation.
+  - If the build is successful, you can check the results in
+    [`doc/_build/html/index.html`](_build/html/index.html)
+
+- Once you're happy with the result, commit the changes to a branch on your fork
+  and open a pull request on GitHub (see the "How to contribute" section above).
+  - The GitHub Actions CI will run a subset of the testing/deployment pipeline
+    for each commit you make to your branch -
+    make sure you get a green checkmark next to the commit hash.
+  - Once the PR is reviewed, approved, and merged,
+    the CI will automatically build and deploy the docs to
+    [`https://scope.ztf.dev`](https://scope.ztf.dev)
diff --git a/_sources/field_guide.md.txt b/_sources/field_guide.md.txt
@@ -0,0 +1,40 @@
+# Field guide
+
+This guide provides detailed information about the different types of variable sources along with examples of bogus variability.
+
+Proceed <a href="_static/taxonomy.html">here</a> to interactively inspect the taxonomy
+tree we are employing in SCoPe. Please refer to [arXiv:2102.11304](https://arxiv.org/pdf/2102.11304.pdf)
+for more details on the taxonomy.
+
+```{include} ./field_guide__variable.md
+```
+
+```{include} ./field_guide__periodic.md
+```
+
+```{include} ./field_guide__rr_lyrae.md
+```
+
+```{include} ./field_guide__w_uma.md
+```
+
+```{include} ./field_guide__delta_scuti.md
+```
+
+```{include} ./field_guide__cepheid.md
+```
+
+```{include} ./field_guide__CVs.md
+```
+
+```{include} ./field_guide__flaring.md
+```
+
+```{include} ./field_guide__beta_lyr.md
+```
+
+```{include} ./field_guide__lpv.md
+```
+
+```{include} ./field_guide__bogus.md
+```
diff --git a/_sources/field_guide__CVs.md.txt b/_sources/field_guide__CVs.md.txt
@@ -0,0 +1,55 @@
+## Cataclysmic Variables (cv)
+Cataclysmic variables are close binaries with active mass transfer from a late
+type main sequence star or brown dwarf to a white dwarf. There are several different
+types of CVs, depending on their causes of variability. These include novae (which
+have 9-15 mag outbursts from thermonuclear events on the white dwarf surface), dwarf novae which have 2-9 mag outbursts on weeks to decades timescales due to disk instabilities) and novalikes which do not have outbursts but have high and low states of accretion which cause several magnitudes of brightness change.
+
+### Classification and numbers
+- Supertypes
+  - variable
+  - aperiodic outbursts
+  - aperiodic high and low states
+  - periodic orbital variability
+- Subtypes
+  - Nova
+  - U Gem dwarf nova
+  - Z Cam dwarf nova
+  - SU UMa/WZ Sge dwarf nova
+  - Novalike
+
+- Occurrence rate: common, several thousand expected in ZTF data
+
+
+### ZTF light curves
+![ZTF CV U Gem](data/cv_U_Gem.png)
+![ZTF CV Z Cam](data/cv_Z_Cam.png)
+![ZTF CV SU UMa](data/cv_SU_UMa.png)
+![ZTF CV Novalike](data/cv_Novalike.png)
+
+#### Description
+CVs are easy to recognize by their distinctive light curve shape, colors and
+high amplitude variability. The outbursts are non-periodic.
+They can sometimes be confused with supernovae, flare stars or long period variables.
+
+#### Light curve characteristics
+- non-periodic variable but recur on some timescale
+- outburst timescale range: days to decades
+- amplitude: 9-15 mag (nova), 2-9 mag (dwarf nova), 1-4 mag (novalike)
+- light curve shape:
+    - sawtooth; steep rise and slow decay (subtype nova)
+    - symmetrical to slightly sawtooth (subtype U Gem)
+    - standstills about 1 mag below outburst level for weeks (subtype Z Cam)
+    - fast rise and extended plateau for 1-2 weeks followed by steeper decline     (subtype SU UMa/WZ Sge)
+    - extended (days-weeks) states at either high or low brightness
+- can show periodic modulation of the light curve on orbital timescales of hours
+
+
+#### Other characteristics and selection methods
+- intrinsic CV colors: blue, (g-r < 0.6).
+  Reddening is usually not important except for places in the galactic plane.
+- absolute magnitude at quiescence: 8<G<14
+
+![HR diagram of CVs](data/hr__cv.png)
+
+### References and further reading:
+- Warner 1995: Cataclysmic Variable Stars
diff --git a/_sources/field_guide__beta_lyr.md.txt b/_sources/field_guide__beta_lyr.md.txt
@@ -0,0 +1,30 @@
+## Beta Lyr (blyr)
+
+Semi-detached eclipsing binaries (also called [beta Lyrae variables](https://en.wikipedia.org/wiki/Beta_Lyrae_variable)) are binary star systems (i.e. two stars that are in orbit around each other) that are aligned such that they eclipse each other relative to our line of sight from the Earth. Unlike detached EBs, semi-detatched systems are so close to each other that the shape of (at least one of) the stars in the system are distorted, giving rise to the smooth periodic variations even when the system is not eclipsing.
+
+### Classification and numbers
+- Supertypes
+  - variable
+  - periodic
+  - binary
+  - eclipsing
+- Occurrence rate: very common, about 10<sup>5</sup> expected in ZTF data
+
+### ZTF light curves
+![ZTF betalyr](data/beta_lyr.png)
+
+#### Description
+Any type of star can be in an eclipsing binary (so they may be small, large, hot, cool, etc.), which in turn can lead to a large range in eclipse amplitudes (though typically these are between 0.1 and ~1 mag, with most in the ~0.3 to 0.75 mag range) or eclipse periods (though our observations are typically only sensitive to periods between 0.1 to ~20 days).
+
+#### Light curve characteristics
+- periodic variable
+- Range of amplitudes (~0.2 to >1 mag)
+- Intermediate periods (log Period between -0.5 and 1.4)
+- light curve shape: EB, round or sinusoidal light curves, with imposed "V-shape" dips in the phase folded light curve
+
+![HR diagram of Beta Lyrae](data/hr__beta_lyr.png)
+
+![Period histogram of Beta Lyrae](data/period__beta_lyr.png)
+
+### References and further reading:
+- Sterken & Jasschek: Light curves of variable stars
diff --git a/_sources/field_guide__bogus.md.txt b/_sources/field_guide__bogus.md.txt
@@ -0,0 +1,16 @@
+## Bogus variability (bogus)
+
+Not all light curve variability pertains to a source's intrinsic astrophysical nature. Some is caused by nearby extended objects, bright stars, blends and image artifacts, and being aware of how such bogus light curves appear can help avoid confusion.
+
+### ZTF light curves
+![ZTF bogus](data/bogus_1.png)
+![ZTF bogus](data/bogus_2.png)
+
+#### Description
+The first light curve above demonstrates a saturation ghost artifact, and the second light curve suffers from another kind of artifact. These artifacts were identified and affected data masked after ZTF began, but data from earlier releases were not retroactively masked. This produces the apparent cutoff in variability after a certain point in time.
+
+#### Light curve characteristics
+The light curves appear to be "flaring" with departures from the median by multiple magnitudes which suddenly stop in later data. This cutoff corresponds to the time when a new method of processing the data was used to mask affected points.
+
+### References and further reading:
+- [ZTF Explanatory Supplement (esp. Appendix B)](https://irsa.ipac.caltech.edu/data/ZTF/docs/ztf_explanatory_supplement.pdf)