Skip to content

Commit

Permalink
Add async (#25)
Browse files Browse the repository at this point in the history
* Add async

* To thread

* No threadpool

* fix

* Worker

* Naming things

* pre-commit

* Await in the right place

* Use semaphore

* Docs

* Docs

* style: pre-commit fixes

* Pre-commit fix

* Docs

* Drop 3.8 tests

* Drop 3.13

* Don't car

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
  • Loading branch information
AlecThomson and pre-commit-ci[bot] authored Dec 4, 2024
1 parent 95430f1 commit 1bdd70a
Show file tree
Hide file tree
Showing 11 changed files with 413 additions and 163 deletions.
47 changes: 23 additions & 24 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,39 +5,38 @@ name: Tests

on:
push:
branches: [ "main" ]
branches: ["main"]
pull_request:
branches: [ "main" ]
branches: ["main"]

permissions:
contents: write

jobs:
build:

runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11"]
python-version: ["3.9", "3.10", "3.11", "3.12"]

steps:
- uses: actions/checkout@v3
with:
lfs: true
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v3
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install pytest black pytest-cov
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
pip install .
- name: Test with pytest
run: |
pytest --cov fixms
- name: Upload coverage reports to Codecov
uses: codecov/[email protected]
with:
token: ${{ secrets.CODECOV_TOKEN }}
- uses: actions/checkout@v3
with:
lfs: true
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v3
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install pytest black pytest-cov
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
pip install .
- name: Test with pytest
run: |
pytest --cov fixms
- name: Upload coverage reports to Codecov
uses: codecov/[email protected]
with:
token: ${{ secrets.CODECOV_TOKEN }}
105 changes: 75 additions & 30 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,32 +1,77 @@
ci:
autoupdate_commit_msg: "chore: update pre-commit hooks"
autofix_commit_msg: "style: pre-commit fixes"

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
hooks:
- id: check-yaml
- id: end-of-file-fixer
exclude: |
^tests/scienceData.RACS_0012+00.SB45305.RACS_0012+00.beam00_averaged_cal.leakage.ms/*$
- id: trailing-whitespace
exclude: |
^tests/scienceData.RACS_0012+00.SB45305.RACS_0012+00.beam00_averaged_cal.leakage.ms/*$
- repo: https://github.com/psf/black
rev: 24.10.0
hooks:
- id: black
- repo: https://github.com/PyCQA/isort
rev: 5.13.2
hooks:
- id: isort
args: ["--profile=black"]
- repo: https://github.com/adamchainz/blacken-docs
rev: "1.18.0"
hooks:
- id: blacken-docs
additional_dependencies: [black==24.*]

ci:
autofix_commit_msg: |
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
autofix_prs: true
autoupdate_branch: ''
autoupdate_commit_msg: '[pre-commit.ci] pre-commit autoupdate'
autoupdate_schedule: weekly
skip: []
submodules: false
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: "v4.6.0"
hooks:
- id: check-added-large-files
- id: check-case-conflict
- id: check-merge-conflict
- id: check-symlinks
- id: check-yaml
- id: debug-statements
- id: end-of-file-fixer
- id: mixed-line-ending
- id: name-tests-test
args: ["--pytest-test-first"]
- id: requirements-txt-fixer
- id: trailing-whitespace

- repo: https://github.com/pre-commit/pygrep-hooks
rev: "v1.10.0"
hooks:
- id: rst-directive-colons
- id: rst-inline-touching-normal

- repo: https://github.com/pre-commit/mirrors-prettier
rev: "v3.1.0"
hooks:
- id: prettier
types_or: [yaml, markdown, html, css, scss, javascript, json]
args: [--prose-wrap=always]

- repo: https://github.com/astral-sh/ruff-pre-commit
rev: "v0.5.2"
hooks:
- id: ruff
args: ["--fix", "--show-fixes"]
- id: ruff-format

- repo: https://github.com/codespell-project/codespell
rev: "v2.3.0"
hooks:
- id: codespell

- repo: https://github.com/shellcheck-py/shellcheck-py
rev: "v0.10.0.1"
hooks:
- id: shellcheck

- repo: local
hooks:
- id: disallow-caps
name: Disallow improper capitalization
language: pygrep
entry: PyBind|Numpy|Cmake|CCache|Github|PyTest
exclude: .pre-commit-config.yaml

- repo: https://github.com/abravalheri/validate-pyproject
rev: "v0.18"
hooks:
- id: validate-pyproject
additional_dependencies: ["validate-pyproject-schema-store[all]"]

- repo: https://github.com/python-jsonschema/check-jsonschema
rev: "0.29.0"
hooks:
- id: check-dependabot
- id: check-github-workflows
- id: check-readthedocs
6 changes: 3 additions & 3 deletions .readthedocs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ build:

# Build documentation in the "docs/" directory with Sphinx
sphinx:
configuration: docs/conf.py
configuration: docs/conf.py

# Optionally build your docs in additional formats such as PDF and ePub
# formats:
Expand All @@ -33,5 +33,5 @@ sphinx:

# Optionally declare the Python requirements required to build your docs
python:
install:
- requirements: docs/requirements.txt
install:
- requirements: docs/requirements.txt
75 changes: 58 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,77 +1,117 @@
[![rtd](https://readthedocs.org/projects/fixms/badge/?version=latest)](https://fixms.readthedocs.io/) [![Tests](https://github.com/AlecThomson/FixMS/actions/workflows/test.yml/badge.svg)](https://github.com/AlecThomson/FixMS/actions/workflows/test.yml) [![PyPi](https://github.com/AlecThomson/FixMS/actions/workflows/publish.yml/badge.svg)](https://pypi.org/project/fixms/) [![pre-commit.ci status](https://results.pre-commit.ci/badge/github/AlecThomson/FixMS/main.svg)](https://results.pre-commit.ci/latest/github/AlecThomson/FixMS/main) [![codecov](https://codecov.io/gh/AlecThomson/FixMS/graph/badge.svg?token=08GDOB2DM0)](https://codecov.io/gh/AlecThomson/FixMS)

[![rtd](https://readthedocs.org/projects/fixms/badge/?version=latest)](https://fixms.readthedocs.io/)
[![Tests](https://github.com/AlecThomson/FixMS/actions/workflows/test.yml/badge.svg)](https://github.com/AlecThomson/FixMS/actions/workflows/test.yml)
[![PyPi](https://github.com/AlecThomson/FixMS/actions/workflows/publish.yml/badge.svg)](https://pypi.org/project/fixms/)
[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/AlecThomson/FixMS/main.svg)](https://results.pre-commit.ci/latest/github/AlecThomson/FixMS/main)
[![codecov](https://codecov.io/gh/AlecThomson/FixMS/graph/badge.svg?token=08GDOB2DM0)](https://codecov.io/gh/AlecThomson/FixMS)

# FixMS

WARNING FOR USERS:
These scripts are still being verified for complete correctness. Please use at your own risk.
WARNING FOR USERS: These scripts are still being verified for complete
correctness. Please use at your own risk.

ASKAP utilities for updating MeasurementSets for external imagers.

ASKAP MSs are produced in a way that breaks compatibility with most other imagers (e.g. CASA, WSclean). Here we provide two modules (with CLI hooks) that perform the fixes that need to be applied in order to produce astronomically correct imagers with non-YandaSoft imagers:

1. `fix_ms_dir` : ASKAP MeasurementSets are phased towards the centre of field, but not the centre of its given beam. This utility reads the appropriate offsets to the beam centre from the `BEAM_OFFSET` and updates the `FIELD` table, as well as the phase and delay reference columns. An option is also available to restore the `FIELD` table directions to the original directions if `fix_ms_dir` has already been run. `BEAM_OFFSET` is also restored with this option.

2. `fix_ms_corrs` : ASKAP MeasurementSets, as calibrated by the obervatory, provide correlations in the instrument frame. ASKAP has a unique 'roll' axis which means, in principle, the instrument frame can be at any arbitrary rotation on the sky. This utility applies the appropriate rotation matrix to the visibilities such the 'X' is aligned North-South and 'Y' is aligned East-West (IAU convention). Further, ASKAPsoft defines Stokes I as $I=XX+YY$, whereas most other telescopes use $I=\frac{1}{2}(XX+YY)$ (note this also applies to all other Stokes paramters). This factor is also corrected for here at the same time as the rotation. If you have calibrated with non-ASKAPsoft tools, you may need to use the `--no-fix-stokes-factor` option, which will disable the factor of two correction and just do a rotation.
ASKAP MSs are produced in a way that breaks compatibility with most other
imagers (e.g. CASA, WSclean). Here we provide two modules (with CLI hooks) that
perform the fixes that need to be applied in order to produce astronomically
correct imagers with non-YandaSoft imagers:

1. `fix_ms_dir` : ASKAP MeasurementSets are phased towards the centre of field,
but not the centre of its given beam. This utility reads the appropriate
offsets to the beam centre from the `BEAM_OFFSET` and updates the `FIELD`
table, as well as the phase and delay reference columns. An option is also
available to restore the `FIELD` table directions to the original directions
if `fix_ms_dir` has already been run. `BEAM_OFFSET` is also restored with
this option.

2. `fix_ms_corrs` : ASKAP MeasurementSets, as calibrated by the obervatory,
provide correlations in the instrument frame. ASKAP has a unique 'roll' axis
which means, in principle, the instrument frame can be at any arbitrary
rotation on the sky. This utility applies the appropriate rotation matrix to
the visibilities such the 'X' is aligned North-South and 'Y' is aligned
East-West (IAU convention). Further, ASKAPsoft defines Stokes I as $I=XX+YY$,
whereas most other telescopes use $I=\frac{1}{2}(XX+YY)$ (note this also
applies to all other Stokes parameters). This factor is also corrected for
here at the same time as the rotation. If you have calibrated with
non-ASKAPsoft tools, you may need to use the `--no-fix-stokes-factor` option,
which will disable the factor of two correction and just do a rotation.

For convenience, we also provide `fix_ms` which does both of the above!

Full documentation on [Read The Docs](https://fixms.readthedocs.io/en/latest/).

## Installation

Obtain and install Python 3 (I recommend [Miniforge](https://github.com/conda-forge/miniforge) or [Miniconda](https://docs.conda.io/en/latest/miniconda.html)).
Obtain and install Python 3 (I recommend
[Miniforge](https://github.com/conda-forge/miniforge) or
[Miniconda](https://docs.conda.io/en/latest/miniconda.html)).

Install the Python scripts:

Latest:

```
pip install git+https://github.com/AlecThomson/FixMS.git
```

Stable:

```
pip install FixMS
```

## Usage

> **_NOTE:_** If running on large MeasurementSets pay attention to the
> 'chunksize' and 'max-chunks' arguments. These are used to balance memory usage
> and the disk I/O. You may need to adjust these parameters to achieve maximum
> speed on your hardware.
```
❯ fix_ms -h
usage: fix_ms [-h] [--chunksize CHUNKSIZE] [--data-column DATA_COLUMN] [--corrected-data-column CORRECTED_DATA_COLUMN] [ms]
usage: fix_ms [-h] [--chunksize CHUNKSIZE] [--max-chunks MAX_CHUNKS] [--data-column DATA_COLUMN] [--corrected-data-column CORRECTED_DATA_COLUMN] [--no-fix-stokes-factor] [ms]
Utility to correct the ASKAP beam positions and apply a rotation to apply a change of the reference frame of the visibilities
positional arguments:
ms Measurement set to update (default: None)
options:
optional arguments:
-h, --help show this help message and exit
--chunksize CHUNKSIZE
The chunksize to use when reading the MS (default: 1000)
--max-chunks MAX_CHUNKS
The maximum number of chunks to process at once (default: 1000)
--data-column DATA_COLUMN
The column to fix (default: DATA)
--corrected-data-column CORRECTED_DATA_COLUMN
The column to write the corrected data to (default: CORRECTED_DATA)
--no-fix-stokes-factor
Don't fix the Stokes factor. Use this if you have *not* used ASKAPsoft. If you have used ASKAPsoft, you should leave this option alone. (default: False)
```

```
❯ fix_ms_corrs -h
usage: fix_ms_corrs [-h] [--chunksize CHUNKSIZE] [--data-column DATA_COLUMN] [--corrected-data-column CORRECTED_DATA_COLUMN] ms
usage: fix_ms_corrs [-h] [--chunksize CHUNKSIZE] [--max-chunks MAX_CHUNKS] [--data-column DATA_COLUMN] [--corrected-data-column CORRECTED_DATA_COLUMN] [--no-fix-stokes-factor] ms
Fix the correlation rotation of ASKAP MSs. Converts the ASKAP standard correlations to the 'standard' correlations This will make them compatible with most imagers (e.g. wsclean, CASA) The new correlations are placed in a new column called 'CORRECTED_DATA'
Fix the correlation rotation of ASKAP MSs. Converts the ASKAP standard correlations to the 'standard' correlations This will make them compatible with most imagers (e.g. wsclean, CASA) The
new correlations are placed in a new column called 'CORRECTED_DATA'
positional arguments:
ms The MS to fix
options:
optional arguments:
-h, --help show this help message and exit
--chunksize CHUNKSIZE
The chunksize to use when reading the MS (default: 1000)
--max-chunks MAX_CHUNKS
The maximum number of chunks to process at once (default: 1000)
--data-column DATA_COLUMN
The column to fix (default: DATA)
--corrected-data-column CORRECTED_DATA_COLUMN
The column to write the corrected data to (default: CORRECTED_DATA)
--no-fix-stokes-factor
Don't fix the Stokes factor. Use this if you have *not* used ASKAPsoft. If you have used ASKAPsoft, you should leave this option alone. (default: False)
```

```
Expand All @@ -86,8 +126,9 @@ positional arguments:
options:
-h, --help show this help message and exit
-r, --restore Switch to restore direction to the original ASKAPsoft pipeline direction. (default: False)
```
```

## Contribution

Contributions are very welcome! Please open an issue first to discuss any bugs or updates you might have.
Contributions are very welcome! Please open an issue first to discuss any bugs
or updates you might have.
2 changes: 0 additions & 2 deletions docs/conf.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
import os
import sys
from importlib.metadata import distribution

# Configuration file for the Sphinx documentation builder.
Expand Down
12 changes: 6 additions & 6 deletions docs/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
isort
-e .
autoapi
black
sphinx
numpydoc
isort
myst_parser
autoapi
sphinx-autoapi
numpydoc
pydata-sphinx-theme
-e .
sphinx
sphinx-autoapi
8 changes: 7 additions & 1 deletion fixms/fix_ms.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
Utility to correct the ASKAP beam positions and apply a rotation
to apply a change of the reference frame of the visibilities
"""

import logging
from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser
from pathlib import Path
Expand All @@ -19,7 +20,6 @@ def get_parser() -> ArgumentParser:
parser = ArgumentParser(
description=__doc__, formatter_class=ArgumentDefaultsHelpFormatter
)

parser.add_argument(
"ms", help="Measurement set to update", type=str, default=None, nargs="?"
)
Expand All @@ -30,6 +30,12 @@ def get_parser() -> ArgumentParser:
default=1000,
help="The chunksize to use when reading the MS",
)
parser.add_argument(
"--max-chunks",
type=int,
default=1000,
help="The maximum number of chunks to process at once",
)
parser.add_argument(
"--data-column", type=str, default="DATA", help="The column to fix"
)
Expand Down
Loading

0 comments on commit 1bdd70a

Please sign in to comment.