Download and convert weather data for use in ML pipelines
Some renewables, such as solar and wind, generate power according to the weather conditions. Any forecasting therefore requires predictions of how these conditions will change. Many meteorological organisations provide Numerical Weather Prediction (NWP) data, which can then used for model training and inference.
This data is often very large and can come in various formats. Furthermore, these formats are not necessarily suitable for training, so may require preprocessing and conversion.
This package aims to streamline the collection and processing of this NWP data.
Note
This is not built to replace tools such as Herbie. It is built to produce data specific to the needs of Open Climate Fix's models, so things like the output format and the variable selection are hard-coded. If you need a more configurable cli-driven tool, consider using herbie instead.
Install from PyPi using pip:
$ pip install nwp-consumerOr use the container image:
$ docker pull ghcr.io/openclimatefix/nwp-consumerTo download the latest available day of GFS data:*
$ nwp-consumer consumeTo create an archive of a month of GFS data:
Note
This will download several gigabytes of data to your home partition. Make sure you have plenty of free space (and time!)
$ nwp-consumer archive --year 2024 --month 1Documentation is generated via pdoc. To build the documentation, run the following command in the repository root:
$ PDOC_ALLOW_EXEC=1 python -m pdoc -o docs --docformat=google src/nwp_consumerNote
The PDOC_ALLOW_EXEC=1 environment variable is required due to a facet
of the ocf_blosc2 library, which imports itself automatically and hence
necessitates execution to be enabled.
Authentication, and model repository selection, is handled via environment variables.
Choose a repository via the MODEL_REPOSITORY environment variable. Required environment
variables can be found in the repository's metadata function. Missing variables will be
warned about at runtime.
The ZARRDIR environment variable can be set to an S3 url
(ex: s3://some-bucket-name/some-prefix). Valid credentials for accessing the bucket
must be discoverable in the environment as per
Botocore's documentation
With difficulty! This package pulls data specifically tailored to Open Climate Fix's needs, and as such, the data it pulls (and the schema that data is surfaced with) is a fixed part of the package. A large part of the value proposition of this consumer is that the data it produces is consistent and comparable between different sources, so pull requests to the effect of adding or changing this for a specific model are unlikely to be approved.
However, desired changes can be made via cloning the repo and making the relevant parameter modifications to the model's expected coordinates in it's metadata for the desired model repository.
This project uses MyPy for static type checking and Ruff for linting. Installing the development dependencies makes them available in your virtual environment.
Use them via:
$ python -m mypy .
$ python -m ruff check .Be sure to do this periodically while developing to catch any errors early and prevent headaches with the CI pipeline. It may seem like a hassle at first, but it prevents accidental creation of a whole suite of bugs.
Run the unittests with:
$ python -m unittest discover -s src/nwp_consumer -p "test_*.py"On packaging a python project using setuptools and pyproject.toml:
- The official PyPA packaging guide.
- A step-by-step practical guide on the godatadriven blog.
- The pyproject.toml metadata specification.
On hexagonal architecture:
- A concrete example using Python.
- An overview of the fundamentals incorporating Typescript
- Another example using Go.
On the directory structure:
- The official PyPA discussion on src and flat layouts.
- PR's are welcome! See the Organisation Profile for details on contributing
- Find out about our other projects in the OCF Meta Repo
- Check out the OCF blog for updates
- Follow OCF on LinkedIn
Part of the Open Climate Fix community.
