README.md

Satip

Satip is a library for satellite image processing, and provides all of the functionality necessary for retrieving, and storing EUMETSAT data

Installation

To install the satip library please run:

pip install satip

Or if you're working in the development environment you can run the following from the directory root:

pip install -e .

Conda

Or, if you want to use conda from a cloned Satip repository:

conda env create -f environment.yml
conda activate satip
pip install -e .

If you plan to work on the development of Satip then also consider installing these development tools:

conda install pytest flake8 jedi mypy black pre-commit
pre-commit install

Development Environment

In order to contribute:

• it's recommended that you use a Linux-based OS. This is currently used for all CI/CD testing, production, and development.
• At the time of writing (21-Dec-23), the Python version used is 3.11 with work being done to update to Python 3.12. This is subject to updates over time.

Operation

Getting your own API key

In order to contribute to development or just test-run some scripts, you will need your own Eumetsat-API-key. Please follow these steps:

1. Go to https://eoportal.eumetsat.int and register an account.
2. You can log in and go to https://data.eumetsat.int/ to check available data services. From there go to your profile and choose the option "API key" or go to https://api.eumetsat.int/api-key/ directly.
3. Please make sure that you added the key and secret to your user's environment variables.

Downloading EUMETSAT Data

The following command will download the last 2 hours of RSS imagery into NetCDF files at the specified location

python satip/app.py --api-key=<EUMETSAT API Key> --api-secret=<EUMETSAT API Secret> --save-dir="/path/to/saving/files/" --history="2 hours"

To download more historical data, the command below will download the native files, compress with bz2, and save into a subdirectory.

python satip/get_raw_eumetsat_data.py --user-key=<EUMETSAT API Key> --user-secret=<EUMETSAT API Secret>

Converting Native files to Zarr

scripts/convert_native_to_zarr.py converts EUMETSAT .nat files to Zarr datasets, using very mild lossy JPEG-XL compression. (JPEG-XL is the "new kid on the block" of image compression algorithms). JPEG-XL makes the files about a quarter the size of the equivalent bz2 compressed files, whilst the images are visually indistinguishable. JPEG-XL cannot represent NaNs so NaNs. JPEG-XL understands float32 values in the range [0, 1]. NaNs are encoded as the value 0.025. All "real" values are in the range [0.075, 1]. We leave a gap between "NaNs" and "real values" because there is very slight "ringing" around areas of constant value (see this comment for more details). Use satip.jpeg_xl_float_with_nans.JpegXlFloatWithNaNs to decode the satellite data. This class will reconstruct the NaNs and rescale the data to the range [0, 1].

Running in Production

The live service uses app.py as the entrypoint for running the live data download for OCF's forecasting service, and has a few configuration options, configurable by command line argument or environment variable.

--api-key or API_KEY is the EUMETSAT API key

--api-secret or API_SECRET is the EUMETSAT API secret

--save-dir or SAVE_DIR is the top level directory to save the output files, a latest subfolder will be added to that directory to contain the latest data

--history or HISTORY is the amount of history timesteps to use in the latest.zarr files

--db-url or DB_URL is the URL to the database to save to when a run has finished

--use-rescaler or USE_RESCALER tells whether to rescale the satellite data to between 0 and 1 or not when saving to disk. Primarily used as backwards compatibility for the current production models, all new training and production Zarrs should use the rescaled data.

Testing

To run tests, simply run pytest . from the root of the repository. To generate the test plots, run python scripts/generate_test_plots.py.

Environmental Variables

Some tests require environmental variables to be set that would be passed in by command line argument when running the code in production. These are as follows:

• EUMETSAT_USER_KEY: the EUMETSAT API key
• EUMETSAT_USER_SECRET: the EUMETSAT API secret

These can be added using the export command in your shell environment. To add these permanently, the export statements can be added to the configuration file for the shell environment (e.g. "~/.bashrc" if using bash).

Contributors ✨

Thanks goes to these wonderful people (emoji key):

Jacob Bieker 💻	Jack Kelly 💻	Ayrton Bourn 💻	Laurence Watson 💻	Notger Heinz 📖	Peter Dudfield 📖	Azah Norbline 💻
Tom Pughe 💻	Zhenbang Feng 💻

This project follows the all-contributors specification. Contributions of any kind welcome!