AMPEL is a modular and scalable platform with explicit provenance tracking, suited for systematically processing large - possibly complex and heterogeneous - datasets in real-time or not. This includes selecting, analyzing, updating, combining, enriching and reacting to data.
The framework requires analysis and reaction logic to be broken down in adequate indepentent units. AMPEL is general enough to be usable in various fields, it was originaly developped to solve challenges in the context of observational astrophysics.
AMPEL requires Python 3.10+ and its codebase is fully typed.
AMPEL is made of four execution layers (tiers) that replace a traditional pipeline architecture.
The tiers are independently scheduled and the information exchange between tiers occurs via a dedicated database. The execution layer architecture along with the database structure allows for simple parallelization.
Each tier is modular and executes so-called "units".
Ampel logical units have standardized inputs and ouputs, enforced through abstract classes which units inherit.
Every change in AMPEL is triggered by a process. A process executes, at a given date and time, a processor unit that itself runs one or multiple logical units with specific configurations. Information about process executions are registred into the database. The majority of processes are associated with a specific tier but general processes are possible.
A working AMPEL system will spawn multiple processes, posssibly concurently, accross the four AMPEL tiers. This will result in the ingestion and analysis of data and the triggering of automated reactions when given data states are detected.
Channels are convenient for multi-user or multi-prupose AMPEL systems. They allow to define and enforce access rights and to privatize processes, meaning that the output generated by the processes will be only accessible to processes belonging to the same channel.
Internally, channels are just tags in database documents and ampel configuration files.
From a user perspective, a channel can be seen as a collection of private processes.
Processes associated with a given channel
Note that within AMPEL, different channels requiring the same computation will not result in the required computation being performed twice.
The AMPEL code is partitioned in different repositories.
The only mandatory repository in this list is ampel-interface
Public abstract class definitions:
https://github.com/AmpelProject/Ampel-interface
Specialized classes for Tier 0, capable of handling alerts:
https://github.com/AmpelProject/Ampel-alerts
An add-on that introduces two generic classes of datapoints:
https://github.com/AmpelProject/Ampel-photometry
Example of an instrument specific implementation:
https://github.com/AmpelAstro/Ampel-ztf
Numerous ampel units, the majority being specific to astronomy:
https://github.com/AmpelAstro/Ampel-HU-astro/
MongoDB is used to store data. The collections have been designed and indexed for fast insertion and query. Users do not interact with the database directly. Information exchange is instead regulated through (python) abstract base classes from which units are constructed. A specific set of internal classes handle database input and output.
All AMPEL software, can be combined into one container that defines an instance. These containers can be used both to process real-time data as well as to reprocess archived data. The containers themselves should be archived as well.
The latest release of ampel-core can be installed from PyPI with pip, e.g.:
pip install ampel-core
Other projects like ampel-alerts, ampel-ztf, etc. can also be installed with pip.
ampel-core uses poetry for dependency management and packaging. To work with an editable install it's recommended that you set up poetry and install ampel-core in a virtual environment by doing
poetry installAlternatively, may also use a setuptools-style editable install from setup.py:
pip install -e .Note that setup.py and requirements.txt are auto-generated; any changes you commit will be overwritten the next time pyproject.toml is updated.
- One class per file (PEP 20: Explicit is better than implicit)
- Please try to type hint code and fix potential issues using mypy
- Keep indentation preferences of existing files
- Files should contain a header. Example:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# File : ampel/pipeline/t2/T2Controller.py
# License : BSD-3-Clause
# Author : Your Name <Optional email>
# Date : 25.01.2018
# Last Modified Date: 22.08.2018
# Last Modified By : Your Name <Optional email>
- Classes and methods should include docstrings, preferably using reST style:
"""
:param param1: this is a first param
:param param2: this is a second param
:returns: description
:raises keyError: raises an exception
"""
- Classes: CamelCase
- Methods: snake_case
- Instance variables: snake_case
- Dict keys: lowercase
- Commit often
- Please write good commit messages (see jvs commits for examples of high quality messages)
J. Nordin, V. Brinnel, J. van Santen, M. Bulla, U. Feindt et al. Transient processing and analysis using AMPEL. Astronomy and Astrophysics 631 (2019) A147. doi:10.1051/0004-6361/201935634
