Pipeline

About the pipeline

Pipeline topology (simplified)

Steps explanation

• Sorting Hat: Assigns an ALeRCE ID (aid) to the alerts received by it. This step performs a crossmatch to check if the alerted object must be assigned to a previously created aid or need to create a new aid that will contain this alert.
• Previous Candidates: Decodes the extra_fields (only in ZTF alerts for now) and obtains the previous candidates in the alert. Returns an AVRO package with 3 fields: the aid, a list of detections and a list of non_detections.
• Stamp Classifier: Uses the stamp_classifier algorithm to classify images into different astronomical objects.
• Lightcurve: Retrieves the full lightcurve of a given object.
• Correction: Applies correction to pass from difference to apparent magnitude.
• Magstats: Calculates some important statistics of the object.
• Xmatch: Performs a crossmatch using the ALLWISE catalog.
• Features: Obtain the features of a given lightcurve, using FeatureExtractors.
• LC Classifier: Perform a inference given a set of features. This might not get a classification if the object lacks of certain features.

Other steps that aren't part of the alert processing pipeline

• S3: Upload the AVRO files in a AWS S3 Bucket.
• Watchlist: Receives alerts and perform crossmatch to check if there are objects on a watchlist created by an user. If there is, then the user is notified.
• Scribe: CQRS Step which allows other steps to perform asyncronous write database operations.

Glossary

• Alert: Incoming detection from a given survey
• Survey: Specific observational project that generates alerts based on changes in the sky. Note that they can use one or more telescopes
• Object: A collection of spatially close detections (as defined by ALeRCE), assumed to be from the same source
• Detection: Alert stored in the database for which a significant flux change with respect to its template image can be detected
• Non Detection: Observations included in the stream in which no significant flux change was detected. These never come as alerts and are only present in ZTF as part of an alert previous candidates

More details in the PIPELINE.md file