diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 46c5d967..ef94c44e 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -26,7 +26,7 @@ jobs: - uses: actions/setup-python@v4 with: - python-version: '3.9' + python-version: '3.11' - name: Install dependencies and use default config run: | diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index ec2b95e8..d4ad64eb 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -14,7 +14,7 @@ jobs: - uses: actions/checkout@v3 - uses: actions/setup-python@v4 with: - python-version: '3.9' + python-version: '3.11' - name: Install dependencies run: | python -m pip install --upgrade pip diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 1db740e6..59524132 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -26,7 +26,7 @@ jobs: - uses: actions/setup-python@v4 with: - python-version: '3.9' + python-version: '3.11' - name: Install dependencies and use default config run: | diff --git a/.github/workflows/test_no_kowalski.yml b/.github/workflows/test_no_kowalski.yml index 771303bc..c7a8ebfd 100644 --- a/.github/workflows/test_no_kowalski.yml +++ b/.github/workflows/test_no_kowalski.yml @@ -14,7 +14,7 @@ jobs: - uses: actions/checkout@v3 - uses: actions/setup-python@v4 with: - python-version: '3.9' + python-version: '3.11' - name: Install dependencies run: | python -m pip install --upgrade pip diff --git a/.requirements/dev-M1.txt b/.requirements/dev-M1.txt deleted file mode 100644 index fe07f0ae..00000000 --- a/.requirements/dev-M1.txt +++ /dev/null @@ -1,8 +0,0 @@ -deepdiff>=5.0 -gsutil>=4.60 -keras-tuner>=1.0.2 -matplotlib>=3.3 -pytest>=6.1.2 -questionary>=1.8.1 -scikit-learn>=0.24.1 -wandb>=0.12.1 diff --git a/.requirements/dev.txt b/.requirements/dev.txt index f6e031d1..b663b35f 100644 --- a/.requirements/dev.txt +++ b/.requirements/dev.txt @@ -5,6 +5,5 @@ matplotlib>=3.3 pytest>=6.1.2 questionary>=1.8.1 scikit-learn>=0.24.1 -tensorflow>=2.12.0 -tensorflow-addons>=0.20 +tensorflow>=2.14.0 wandb>=0.12.1 diff --git a/.requirements/doc.txt b/.requirements/doc.txt index 8f431846..0d8f5f78 100644 --- a/.requirements/doc.txt +++ b/.requirements/doc.txt @@ -19,3 +19,4 @@ cesium>=0.11.1 xgboost>=1.7.5 seaborn>=0.12.2 pydot>=1.4.2 +jupyter>=1.0.0 diff --git a/doc/developer.md b/doc/developer.md index 3af8e72d..e9a2d261 100644 --- a/doc/developer.md +++ b/doc/developer.md @@ -23,18 +23,18 @@ git remote add upstream git@github.com:ZwickyTransientFacility/scope.git ## Setting up your environment (Windows/Linux/macOS) -#### Use a package manager for installation +### Use a package manager for installation -We currently recommend running `scope` with Python 3.10. You may want to begin your installation by creating/activating a virtual environment, for example using conda. We specifically recommend installing miniforge3 (https://github.com/conda-forge/miniforge). +We currently recommend running `scope` with Python 3.11. You may want to begin your installation by creating/activating a virtual environment, for example using conda. We specifically recommend installing miniforge3 (https://github.com/conda-forge/miniforge). Once you have a package manager installed, run: ```bash -conda create -n scope-env -c conda-forge python=3.10 healpy +conda create -n scope-env -c conda-forge python=3.11 h5py conda activate scope-env ``` -#### Update your `PYTHONPATH` +### Update your `PYTHONPATH` Ensure that Python can import from `scope` by modifying the `PYTHONPATH` environment variable. Use a simple text editor like `nano` to modify the appropriate file (depending on which shell you are using). For example, if using bash, run `nano ~/.bash_profile` and add the following line: @@ -44,7 +44,7 @@ export PYTHONPATH="$PYTHONPATH:$HOME/scope" Save the updated file (`Ctrl+O` in `nano`) and close/reopen your terminal for this change to be recognized. Then `cd` back into scope and activate your `scope-env` again. -#### Install pre-commit +### Install pre-commit We use `black` to format the code and `flake8` to verify that code complies with [PEP8](https://www.python.org/dev/peps/pep-0008/). Please install our pre-commit hook as follows: @@ -60,56 +60,14 @@ code. The pre-commit hook will lint *changes* made to the source. ---- - -### Instructions for macOS (ARM64/Apple Silicon only) - -Skip to the next section if using Windows/Linux or macOS (AMD64/Intel). - -#### Install tensorflow for macOS/Apple Silicon - -You will need to install the correct version of tensorflow for the ARM64 architecture used by Macs with Apple Silicon. Apple provides an effective version here: `https://developer.apple.com/metal/tensorflow-plugin/`. - -To install tensorflow for macOS (including a version that runs on GPUs), run: -```bash -pip install tensorflow-macos -pip install tensorflow-metal -``` - -#### Modify remaining requirements - -After successfully installing tensorflow-deps, tensorflow-macos, tensorflow-metal, modify the remaining software requirements before installing anything else. Overwrite the `.requirements/dev.txt` file with the updated requirements from `.requirements/dev-M1.txt` by running the following from the scope directory: - -```bash -cp -f .requirements/dev-M1.txt .requirements/dev.txt -``` - -This removes `tensorflow` and `tensorflow-addons` from `.requirements/dev.txt` . The file should now list the following requirements: - -```txt -deepdiff>=5.0 -gsutil>=4.60 -keras-tuner>=1.0.2 -matplotlib>=3.3 -pytest>=6.1.2 -questionary>=1.8.1 -scikit-learn>=0.24.1 -wandb>=0.12.1 -``` - -Continue the installation by following the below instructions for all operating systems. - ---- -### All Windows/Linux/macOS - -#### Install required packages +## Install required packages Install the required python packages by running: ```bash pip install -r requirements.txt ``` -#### Create and modify config.yaml +### Create and modify config.yaml From the included config.defaults.yaml, make a copy called config.yaml: @@ -119,23 +77,23 @@ cp config.defaults.yaml config.yaml Edit config.yaml to include Kowalski instance and Fritz tokens in the associated empty `token:` fields. -#### Testing +### Testing Run `./scope.py test` to test your installation. Note that for the test to pass, you will need access to the Kowalski database. If you do not have Kowalski access, you can run `./scope.py test_limited` to run a more limited (but still useful) set of tests. -#### Troubleshooting +### Troubleshooting Upon encountering installation/testing errors, manually install the package in question using `conda install xxx` , and remove it from `.requirements/dev.txt`. After that, re-run `pip install -r requirements.txt` to continue. -#### Known issues +### Known issues - Across all platforms, we are currently aware of `scope` dependency issues with Python 3.11. - Anaconda continues to cause problems with environment setup. -- Using `pip` to install `healpy` on an arm64 Mac can raise an error upon import. We recommend including `healpy` as a requirement during the creation of your `conda` environment. +- Using `pip` to install `healpy` on an arm64 Mac can raise an error upon import. We recommend including `h5py` as a requirement during the creation of your `conda` environment. - On Windows machines, `healpy` and `cesium` raise errors upon installation. - For `healpy`, see [this](https://healpy.readthedocs.io/en/latest/install.html#installation-on-windows-through-the-windows-subsystem-for-linux) guide for a potential workaround. - For `cesium`, try to install from the source (https://cesium-ml.org/docs/install.html#from-source) within `scope`. If you will not be running feature generation, this is not a critical error, but there will be points in the code that fail (e.g. `scope.py test`, `tools/generate_features.py`) If the installation continues to raise errors, update the conda environment and try again. -### How to contribute +## How to contribute Contributions to `scope` are made through [GitHub Pull Requests](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests), a set of proposed commits (or patches):