Proposed flagging system for non-stellar sources

[robertdstein]

@jvansanten and I discussed a proposed addition to the ingestion process. Currently, the Archive database is being queried directly for ToO follow up, and this is slow/time-consuming. Estimates of the rates suggest that, by using the Decent Filter, the alert rate can be cut to ~0.5% of the base rate by only searching real+non-stellar objects. The idea is that a channel could be defined that would use the Decent filter as a T0, and then setting up a T3 to output the candidateid of all sources (presumably non-stellar) that pass this filter. These alerts could be tagged in the ArchiveDB, and then ToO queries could search only those alerts with the non-stellar flag. A possible additional use case could be to speed up re-runs of ZTF data. In both cases, a naive estimate would be a factor ~200 speed-up. There would be an unavoidable lag, based on the frequency of T3 scheduling, in propagating the tagging through to the database following ingestion. This lag would be a caveat to using this method. Synchronisation issues could also occur, if for example the base filter was changed at some point. At present, the ToO requirement would be only from present alerts going forward, so would not need to be run backwards over the past 2 years of ZTF data.

[simeonreusch]

I just stumbled upon this, and it still sounds like a good idea.

[robertdstein]

I had literally completely forgotten that we had this idea, but I agree it sounds good.

[jvansanten]

The underpinnings of this exist in the topics feature of Ampel-ZTF-archive. At the moment topics are create-only. What is still needed is a method to add chunks of alerts to an existing topic, ideally handling permissions (ugh) and skipping alerts that are already in a previous chunk, as well as a T3 unit add said chunks.

[simeonreusch]

uh right! Maybe we can have a chat about this in the near future. Also with LSST in mind, this could relieve a lot of stress from the archive.