Latest release 0.3.0 (#47)

* Fix (global version) (#12) * Global Encoder-Processor-Decoder graph (#9) * feat: Initial implementation of global graphs + fixes Co-authored by: Mario Santa Cruz <[email protected]> Co-authored-by: Helen Theissen <[email protected]> Co-authored-by: Sara Hahner <[email protected]> Co-authored-by: Jesper Dramsch <[email protected]> * fix: attributes as torch.float32 * new test: attributes must be float32 * fix typo * Homogeneize base builders * improve test docstrings * homogeneize (name as class attribute) * new input config * new default * remove dataclass from attribute classes * fix: config nodes name * 6 generate graphs from icosahedral meshes (#11) * Global Encoder-Processor-Decoder graph (#9) * feat: Initial implementation of global graphs Co-authored by: Mario Santa Cruz <[email protected]> Co-authored-by: Helen Theissen <[email protected]> Co-authored-by: Jesper Dramsch <[email protected]> * fix: attributes as torch.float32 * new test: attributes must be float32 * fix typo * Homogeneize base builders * improve test docstrings * homogeneize (name as class attribute) * new input config * new default * feat: Initial implementation of global graphs Co-authored by: Mario Santa Cruz <[email protected]> * add cli command * Ignore .pt files * run pre-commit * docstring + log erros * initial tests * feat: initial version of AttributeBuilder * refactor: separate into node edge attribute builders * feat: edge_length moved to edges/attributes.py * remove __init__ * bugfix (encoder edge lengths) + refector * feat: support path and dict for `config` argument * fix: error * refactor: naming * fix: pre-commit * feat: builders icosahedral * feat: Add icosahedral graph generation Co-authored-by: Mario Santa Cruz <[email protected]> * refactor: remove create_shere * feat: Icosahedral edge builder * feat: hexagonal graph generation Co-authored-by: Mario Santa Cruz <[email protected]> * feat: hexagonal builders * fix: AOI not implemented yet * fix: abstractmethod and renaming * chore: add dependencies * test: add tests for trimesh * test: add tests for hex (h3) * fix: imports * fix: output type * refactor: delete unused file * refactor: renaming and positioning * feat: ensure src and dst always the same * fix: imports * fix: edge_name not supported * test: add tests for TriIcosahedralEdges * fix: assert missing for Hexagonal edges * test: hexagonal edges * fix: avoid same name * fix: imports * fix: conflicts * update tests * Include xhops to hexagonal edges * docs: update docstrings * fix: update attribute name * refactor: rename multiscale nodes * refactor: rename icosahedral nodes * improve: clarity of function * improve: function syntax * refactor: simplify resolution assignment * refactor: improve variable naming in icosahedral graph generation * more comments * refactor: naming * refactor: separate into functions * refactor: remove unused code * refactor: remove unused options and rename * doc: clarify cells and nodes * fix: add return statements * homogeneize: tri & hex edges * naming: xhops to x_hops * naming: x_hops in docstring * docstring * blank lines * fix: add return statement * simplify icoshaedral edges * feat: select edge builder method based on ico node type * test: adjust tests to new MultiScaleEdges * docs: improve docstrings * fix: remove LAM filtering for icosahedral, leave for next PR * h3 (v4) not supported --------- * [feature] add changelog (#21) * ci: changelog check * feat: add changelog * fix: on develop * Skip no-commit-to-branch in code QA (#17) * Clean nodes after building the graph (#23) * feat: clean graph of unneeded attributes after creation Co-authored-by: Mario Santa Cruz <[email protected]> Co-authored-by: Helen Theissen <[email protected]> Co-authored-by: Jesper Dramsch <[email protected]> * Make graph filename argument optional (#24) * hotfix: make graph saving optional Co-authored-by: Jesper Dramsch <[email protected]> Co-authored-by: Mario Santa Cruz <[email protected]> Co-authored-by: Helen Theissen <[email protected]> * doc: update CHANGELOG Co-authored-by: Jesper Dramsch <[email protected]> * chore: add rtd pr preview * HEALPix nodes (#16) [feature] HEALPix node builder Co-authored-by: theissenhelen <[email protected]> * 25 integrate reusable workflows (#26) * ci: add public pr label * ci: add downstream-ci workflow * ci: add ci-config * fix: pre-commit * docs: update changelog * ci: switch main on downstream-ci * ci: run ci on push on main * Hotfix/clean graph (#28) * Add binary dependencies to ci-config.yml (#29) * Add binary dependencies to ci-config.yml * docs: update changelog --------- Co-authored-by: theissenhelen <[email protected]> * ci: inherit pypi publish flow (#27) * ci: inherit pypi publish flow Co-authored-by: Jesper Dramsch <[email protected]> * docs: update changelog * ci: only downstream-ci for changes in src and tests --------- Co-authored-by: Jesper Dramsch <[email protected]> * Bugfix/31 ensure latlon input numpy array (#32) * #31 Ensure lat lon values passed as numpy arrays * fix formatting * Updated CHANGELOG.md * Remove bug in the graph cleaning function (#33) * fix: remove the bug in the graph cleaning function Co-authored-by: Jesper Dramsch <[email protected]> * docs: update changelog --------- Co-authored-by: Jesper Dramsch <[email protected]> Co-authored-by: Gert Mertes <[email protected]> * Move to reusable workflows for QA and Testing (#34) * ci: move to reusable workflows for QA and Testing * docs: changelog add ci reusable workflow * ci: downstream ci ignore docs * ci: remove anchor, unsupported by github * ci: add changelog release updater (#35) * ci: changelog updater PR (#36) * ci: correct inputs typo * ci: fix changelog updater * docs: ci fixes changelog * ci: premissions * ci: typo * Ci/fix pr permission (#37) * ci: add pull-requests permissions * ci: add on workflow dispatch * ci: remove 3.9 tests (#39) * ci: remove 3.9 tests * ci: remove testing of python 3.9 * dics: update changelog * Fix support for Python 3.9 (#38) * fix: support py3.9 Co-authored-by: Jesper Dramsch <[email protected]> * fix: update & homogeneize changelog * fix: update tests to support py39 * fix: style * ci: rollback 3.9 tests * refactor: annotations not used in tests --------- Co-authored-by: Jesper Dramsch <[email protected]> * Ci/fix_double_notes (#40) * ci: avoid duplication of release notes * ci: try write-all * ci: fix permissions * docs: update changelog * ci: fix indent * Inspection tools (#22) Co-authored-by: Ana Prieto Nemesio <[email protected]> Co-authored-by: Helen Theissen <[email protected]> * [pre-commit.ci] pre-commit autoupdate (#41) updates: - [github.com/psf/black-pre-commit-mirror: 24.4.2 → 24.8.0](psf/black-pre-commit-mirror@24.4.2...24.8.0) - [github.com/astral-sh/ruff-pre-commit: v0.4.6 → v0.6.2](astral-sh/ruff-pre-commit@v0.4.6...v0.6.2) - [github.com/tox-dev/pyproject-fmt: 2.1.3 → 2.2.1](tox-dev/pyproject-fmt@2.1.3...2.2.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#43) updates: - [github.com/astral-sh/ruff-pre-commit: v0.6.2 → v0.6.3](astral-sh/ruff-pre-commit@v0.6.2...v0.6.3) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * First version of documentation (#18) Documentation web page --------- Co-authored-by: Helen Theissen <[email protected]> Co-authored-by: Jesper Dramsch <[email protected]> * [Changelog] Update to 0.3.0 (#44) * [create-pull-request] automated change * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: theissenhelen <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --------- Co-authored-by: Mario Santa Cruz <[email protected]> Co-authored-by: Mario Santa Cruz <[email protected]> Co-authored-by: Jesper Dramsch <[email protected]> Co-authored-by: Gert Mertes <[email protected]> Co-authored-by: Mario Santa Cruz <[email protected]> Co-authored-by: Iain Russell <[email protected]> Co-authored-by: Rilwan (Akanni) Adewoyin <[email protected]> Co-authored-by: Ana Prieto Nemesio <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: theissenhelen <[email protected]>
ecmwf · Sep 11, 2024 · 17c2847 · 17c2847
1 parent 7bf0d23
commit 17c2847
Show file tree

Hide file tree

Showing 90 changed files with 4,703 additions and 252 deletions.
diff --git a/.github/ci-config.yml b/.github/ci-config.yml
@@ -0,0 +1,9 @@
+dependencies: |
+  ecmwf/ecbuild
+  MathisRosenhauer/libaec@master
+  ecmwf/eccodes
+  ecmwf/eckit
+  ecmwf/odc
+dependency_branch: develop
+parallelism_factor: 8
+self_build: false # Only for python packages
diff --git a/.github/workflows/changelog-pr-update.yml b/.github/workflows/changelog-pr-update.yml
@@ -0,0 +1,15 @@
+name: Check Changelog Update on PR
+on:
+  pull_request:
+    types: [assigned, opened, synchronize, reopened, labeled, unlabeled]
+    branches:
+      - main
+      - develop
+jobs:
+  Check-Changelog:
+    name: Check Changelog Action
+    runs-on: ubuntu-20.04
+    steps:
+      - uses: tarides/changelog-check-action@v2
+        with:
+          changelog: CHANGELOG.md
diff --git a/.github/workflows/changelog-release-update.yml b/.github/workflows/changelog-release-update.yml
@@ -0,0 +1,35 @@
+# .github/workflows/update-changelog.yaml
+name: "Update Changelog"
+
+on:
+  release:
+    types: [released]
+  workflow_dispatch: ~
+
+permissions:
+  pull-requests: write
+  contents: write
+
+jobs:
+  update:
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v4
+      with:
+        ref: ${{ github.event.release.target_commitish }}
+
+    - name: Update Changelog
+      uses: stefanzweifel/changelog-updater-action@v1
+      with:
+        latest-version: ${{ github.event.release.tag_name }}
+        heading-text: ${{ github.event.release.name }}
+
+    - name: Create Pull Request
+      uses: peter-evans/create-pull-request@v6
+      with:
+        branch: docs/changelog-update-${{ github.event.release.tag_name }}
+        title: '[Changelog] Update to ${{ github.event.release.tag_name }}'
+        add-paths: |
+          CHANGELOG.md
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -0,0 +1,52 @@
+name: ci
+
+on:
+  # Trigger the workflow on push to master or develop, except tag creation
+  push:
+    branches:
+      - 'main'
+      - 'develop'
+    tags-ignore:
+      - '**'
+    paths-ignore:
+      - "docs/**"
+      - "CHANGELOG.md"
+      - "README.md"
+
+  # Trigger the workflow on pull request
+  pull_request:
+    paths-ignore:
+      - "docs/**"
+      - "CHANGELOG.md"
+      - "README.md"
+
+  # Trigger the workflow manually
+  workflow_dispatch: ~
+
+  # Trigger after public PR approved for CI
+  pull_request_target:
+    types: [labeled]
+    paths-ignore:
+      - "docs/**"
+      - "CHANGELOG.md"
+      - "README.md"
+
+jobs:
+  # Run CI including downstream packages on self-hosted runners
+  downstream-ci:
+    name: downstream-ci
+    if: ${{ !github.event.pull_request.head.repo.fork && github.event.action != 'labeled' || github.event.label.name == 'approved-for-ci' }}
+    uses: ecmwf-actions/downstream-ci/.github/workflows/downstream-ci.yml@main
+    with:
+      anemoi-graphs: ecmwf/anemoi-graphs@${{ github.event.pull_request.head.sha || github.sha }}
+      codecov_upload: true
+    secrets: inherit
+
+   # Build downstream packages on HPC
+  downstream-ci-hpc:
+    name: downstream-ci-hpc
+    if: ${{ !github.event.pull_request.head.repo.fork && github.event.action != 'labeled' || github.event.label.name == 'approved-for-ci' }}
+    uses: ecmwf-actions/downstream-ci/.github/workflows/downstream-ci.yml@main
+    with:
+      anemoi-graphs: ecmwf/anemoi-graphs@${{ github.event.pull_request.head.sha || github.sha }}
+    secrets: inherit
diff --git a/.github/workflows/label-public-pr.yml b/.github/workflows/label-public-pr.yml
@@ -0,0 +1,10 @@
+# Manage labels of pull requests that originate from forks
+name: label-public-pr
+
+on:
+  pull_request_target:
+    types: [opened, synchronize]
+
+jobs:
+  label:
+    uses: ecmwf-actions/reusable-workflows/.github/workflows/label-pr.yml@v2
diff --git a/.github/workflows/python-publish.yml b/.github/workflows/python-publish.yml
@@ -4,71 +4,24 @@
 name: Upload Python Package
 
 on:
-
-  push: {}
-
   release:
     types: [created]
 
 jobs:
   quality:
-    name: Code QA
-    runs-on: ubuntu-latest
-    steps:
-    - run: sudo apt-get install -y pandoc # Needed by sphinx for notebooks
-    - uses: actions/checkout@v4
-    - uses: actions/setup-python@v5
-      with:
-        python-version: 3.x
-    - uses: pre-commit/[email protected]
+    uses: ecmwf-actions/reusable-workflows/.github/workflows/qa-precommit-run.yml@v2
+    with:
+      skip-hooks: "no-commit-to-branch"
 
   checks:
     strategy:
-      fail-fast: false
       matrix:
-        platform: ["ubuntu-latest", "macos-latest"]
-        python-version: ["3.10"]
-
-    name: Python  ${{ matrix.python-version }} on ${{ matrix.platform }}
-    runs-on: ${{ matrix.platform }}
-
-    steps:
-    - uses: actions/checkout@v4
-
-    - uses: actions/setup-python@v2
-      with:
-        python-version: ${{ matrix.python-version }}
-
-    - name: Install
-      run: |
-        pip install -e .[all,tests]
-        pip freeze
-
-    - name: Tests
-      run: pytest
+        python-version: ["3.9", "3.10"]
+    uses: ecmwf-actions/reusable-workflows/.github/workflows/qa-pytest-pyproject.yml@v2
+    with:
+      python-version: ${{ matrix.python-version }}
 
   deploy:
-
-    if: ${{ github.event_name == 'release' }}
-    runs-on: ubuntu-latest
     needs: [checks, quality]
-
-    steps:
-    - uses: actions/checkout@v4
-
-    - name: Set up Python
-      uses: actions/setup-python@v2
-      with:
-        python-version: 3.x
-
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip
-        pip install build wheel twine
-    - name: Build and publish
-      env:
-        TWINE_USERNAME: __token__
-        TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
-      run: |
-        python -m build
-        twine upload dist/*
+    uses: ecmwf-actions/reusable-workflows/.github/workflows/cd-pypi.yml@v2
+    secrets: inherit
diff --git a/.github/workflows/python-pull-request.yml b/.github/workflows/python-pull-request.yml
@@ -0,0 +1,23 @@
+# This workflow will upload a Python Package using Twine when a release is created
+# For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries
+
+name: Code Quality checks for PRs
+
+on:
+  push:
+  pull_request_target:
+    types: [opened, synchronize, reopened]
+
+jobs:
+  quality:
+    uses: ecmwf-actions/reusable-workflows/.github/workflows/qa-precommit-run.yml@v2
+    with:
+      skip-hooks: "no-commit-to-branch"
+
+  checks:
+    strategy:
+      matrix:
+        python-version: ["3.9", "3.10"]
+    uses: ecmwf-actions/reusable-workflows/.github/workflows/qa-pytest-pyproject.yml@v2
+    with:
+      python-version: ${{ matrix.python-version }}
diff --git a/.github/workflows/readthedocs-pr-update.yml b/.github/workflows/readthedocs-pr-update.yml
@@ -0,0 +1,22 @@
+name: Read the Docs PR Preview
+on:
+  pull_request_target:
+    types:
+      - opened
+      - synchronize
+      - reopened
+    # Execute this action only on PRs that touch
+    # documentation files.
+    paths:
+      - "docs/**"
+
+permissions:
+  pull-requests: write
+
+jobs:
+  documentation-links:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: readthedocs/actions/preview@v1
+        with:
+          project-slug: "anemoi-graphs"
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -21,7 +21,7 @@ repos:
   - id: check-added-large-files # Check for large files added to git
   - id: check-merge-conflict # Check for files that contain merge conflict
 - repo: https://github.com/psf/black-pre-commit-mirror
-  rev: 24.4.2
+  rev: 24.8.0
   hooks:
   - id: black
     args: [--line-length=120]
@@ -34,7 +34,7 @@ repos:
     - --force-single-line-imports
     - --profile black
 - repo: https://github.com/astral-sh/ruff-pre-commit
-  rev: v0.4.6
+  rev: v0.6.3
   hooks:
   - id: ruff
     # Next line if for documenation cod snippets
@@ -65,6 +65,6 @@ repos:
   - id: optional-dependencies-all
     args: ["--inplace", "--exclude-keys=dev,docs,tests", "--group=dev=all,docs,tests"]
 - repo: https://github.com/tox-dev/pyproject-fmt
-  rev: "2.1.3"
+  rev: "2.2.1"
   hooks:
   - id: pyproject-fmt
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -0,0 +1,79 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+Please add your functional changes to the appropriate section in the PR.
+Keep it human-readable, your future self will thank you!
+
+## [Unreleased](https://github.com/ecmwf/anemoi-graphs/compare/0.3.0...HEAD)
+
+## [0.3.0 Anemoi-graphs, minor release](https://github.com/ecmwf/anemoi-graphs/compare/0.2.1...0.3.0) - 2024-09-03
+
+### Added
+
+- HEALPixNodes - nodebuilder based on Hierarchical Equal Area isoLatitude Pixelation of a sphere.
+
+- Inspection tools: interactive plots, and distribution plots of edge & node attributes.
+
+- Graph description print in the console.
+
+- CLI entry point: 'anemoi-graphs inspect ...'.
+
+- added downstream-ci pipeline and cd-pypi reusable workflow
+
+- Changelog release updater
+
+- Create package documentation.
+
+
+### Changed
+
+- fix: added support for Python3.9.
+- fix: bug in graph cleaning method
+- fix: `anemoi-graphs create` CLI argument is casted to a Path.
+- ci: fix missing binary dependency in ci-config.yaml
+- fix: Updated `get_raw_values` method in `AreaWeights` to ensure compatibility with `scipy.spatial.SphericalVoronoi` by converting `latitudes` and `longitudes` to NumPy arrays before passing them to the `latlon_rad_to_cartesian` function. This resolves an issue where the function would fail if passed Torch Tensors directly.
+- ci: Reusable workflows for push, PR, and releases
+- ci: ignore docs for downstream ci
+- ci: changed Changelog action to create PR
+- ci: fixes and permissions on changelog updater
+
+### Removed
+
+## [0.2.1](https://github.com/ecmwf/anemoi-graphs/compare/0.2.0...0.2.1) - Anemoi-graph Release, bug fix release
+
+### Added
+
+### Changed
+
+- Fix The 'save_path' argument of the GraphCreator class is optional, allowing users to create graphs without saving them.
+
+### Removed
+
+## [0.2.0](https://github.com/ecmwf/anemoi-graphs/compare/0.1.0...0.2.0) - Anemoi-graph Release, Icosahedral graph building
+
+### Added
+
+- New node builders by iteratively refining an icosahedron: TriNodes, HexNodes.
+- New edge builders for building multi-scale connections.
+- Added Changelog
+
+### Changed
+
+### Removed
+
+## [0.1.0](https://github.com/ecmwf/anemoi-graphs/releases/tag/0.1.0) - Initial Release, Global graph building
+
+### Added
+
+- Documentation
+- Initial implementation for global graph building on the fly from Zarr and NPZ datasets
+
+### Changed
+
+### Removed
+
+<!-- Add Git Diffs for Links above -->
diff --git a/docs/_static/cutoff.jpg b/docs/_static/cutoff.jpg
diff --git a/docs/_static/hetero_data_graph.txt b/docs/_static/hetero_data_graph.txt
@@ -0,0 +1,30 @@
+HeteroData(
+  data={
+    x=[40320, 2],  # coordinates in radians (lat in [-pi/2, pi/2], lon in [0, 2pi])
+    node_type='ZarrDatasetNodes',
+    area_weight=[40320, 1],
+  },
+  hidden={
+    x=[10242, 2],  # coordinates in radians (lat in [-pi/2, pi/2], lon in [0, 2pi])
+    node_type='TriNodes',
+    area_weight=[10242, 1],
+  },
+  (data, to, hidden)={
+    edge_index=[2, 62980],
+    edge_type='CutOffEdges',
+    edge_length=[62980, 1],
+    edge_dirs=[62980, 2],
+  },
+  (hidden, to, hidden)={
+    edge_index=[2, 81900],
+    edge_type='MultiScaleEdges',
+    edge_length=[81900, 1],
+    edge_dirs=[81900, 2],
+  },
+  (hidden, to, data)={
+    edge_index=[2, 120960],
+    edge_type='KNNEdges',
+    edge_length=[120960, 1],
+    edge_dirs=[120960, 2],
+  }
+)