DM-45750: Store new-style ephemerides and APDB catalogs in ap_verify data sets

README.md

@@ -19,7 +19,7 @@ path                  | description
 `raw`                 | Raw ImSim exposures.
 `config`              | Dataset-specific configs to help the Science Pipelines work with this dataset, including the butler `export.yaml` file used by `ap_verify.py`
 `pipelines`           | To be populated with dataset-specific pipelines. Currently contains three example files specialized for ImSim data.
-`preloaded`           | Starter Butler repo containing a skymap, calibs, coadds to use as difference imaging templates, ImSim refcats covering the relevant sky region, and a pretrained machine learning model for real/bogus classification.
+`preloaded`           | Starter Butler repo containing a skymap, calibs, coadds to use as difference imaging templates, ImSim refcats covering the relevant sky region, mock APDB outputs based on the raw images, and a pretrained machine learning model for real/bogus classification.
 `scripts`             | Scripts for regenerating this dataset.
 
 Butler Collections
@@ -33,6 +33,7 @@ collection              | description
 `refcats`               | Level 7 HTM shards from relevant reference catalogs.
 `skymaps`               | Skymaps for the template coadds.
 `templates/<type>`      | Coadd images produced by a compatible version of the LSST pipelines. For example, `deepCoadd` images go in a `templates/deep` collection.
+`dia_catalogs`          | Catalogs representing the contents of the APDB at the start of each visit.
 `models`                | Pretrained machine learning models.
 `<instrument>/defaults` | A chained collection linking all of the above.
 

config/export.yaml

@@ -1158,6 +1158,13 @@ data:
     name_in_raft: S22
     raft: R43
     purpose: SCIENCE
+- type: dimension
+  element: group
+  records:
+  - instrument: LSSTCam-imSim
+    name: '943296'
+  - instrument: LSSTCam-imSim
+    name: '982985'
 - type: dimension
   element: physical_filter
   records:
@@ -1232,6 +1239,12 @@ data:
   host: null
   timespan_begin: null
   timespan_end: null
+- type: collection
+  collection_type: RUN
+  name: dia_catalogs/apdb
+  host: null
+  timespan_begin: null
+  timespan_end: null
 - type: collection
   collection_type: RUN
   name: fake-injection-catalog
@@ -1268,6 +1281,11 @@ data:
   children:
   - 2.2i/calib/gen2
   - LSSTCam-imSim/calib/curated
+- type: collection
+  collection_type: CHAINED
+  name: dia_catalogs
+  children:
+  - dia_catalogs/apdb
 - type: collection
   collection_type: CHAINED
   name: models
@@ -1291,6 +1309,7 @@ data:
   - skymaps
   - LSSTCam-imSim/calib
   - refcats
+  - dia_catalogs
   - models
   - fake-injection-catalog
 - type: dataset_type
@@ -2818,6 +2837,90 @@ data:
       htm7: 147160
     path: fake-injection-catalog/injection_catalog/r/injection_catalog_r_147160_fake-injection-catalog.parq
     formatter: lsst.daf.butler.formatters.parquet.ParquetFormatter
+- type: dataset_type
+  name: preloaded_diaForcedSources
+  dimensions:
+  - instrument
+  - detector
+  - group
+  storage_class: DataFrame
+  is_calibration: false
+- type: dataset
+  dataset_type: preloaded_diaForcedSources
+  run: dia_catalogs/apdb
+  records:
+  - dataset_id:
+    - !uuid 'a6b02083-0049-4e7b-9fb3-5d582088dd1b'
+    data_id:
+    - instrument: LSSTCam-imSim
+      detector: 168
+      group: '943296'
+    path: dia_catalogs/apdb/preloaded_diaForcedSources/943296/preloaded_diaForcedSources_LSSTCam-imSim_943296_R41_S20_dia_catalogs_apdb.parq
+    formatter: lsst.daf.butler.formatters.parquet.ParquetFormatter
+  - dataset_id:
+    - !uuid '8bb1f479-50e1-403f-bfcd-8e7e2e4524df'
+    data_id:
+    - instrument: LSSTCam-imSim
+      detector: 164
+      group: '982985'
+    path: dia_catalogs/apdb/preloaded_diaForcedSources/982985/preloaded_diaForcedSources_LSSTCam-imSim_982985_R41_S02_dia_catalogs_apdb.parq
+    formatter: lsst.daf.butler.formatters.parquet.ParquetFormatter
+- type: dataset_type
+  name: preloaded_diaObjects
+  dimensions:
+  - instrument
+  - detector
+  - group
+  storage_class: DataFrame
+  is_calibration: false
+- type: dataset
+  dataset_type: preloaded_diaObjects
+  run: dia_catalogs/apdb
+  records:
+  - dataset_id:
+    - !uuid '723aa650-698b-40d6-b01a-4002b9921ebd'
+    data_id:
+    - instrument: LSSTCam-imSim
+      detector: 168
+      group: '943296'
+    path: dia_catalogs/apdb/preloaded_diaObjects/943296/preloaded_diaObjects_LSSTCam-imSim_943296_R41_S20_dia_catalogs_apdb.parq
+    formatter: lsst.daf.butler.formatters.parquet.ParquetFormatter
+  - dataset_id:
+    - !uuid 'be1c1cad-b7e1-4e2f-b0f2-0ec5c66456a5'
+    data_id:
+    - instrument: LSSTCam-imSim
+      detector: 164
+      group: '982985'
+    path: dia_catalogs/apdb/preloaded_diaObjects/982985/preloaded_diaObjects_LSSTCam-imSim_982985_R41_S02_dia_catalogs_apdb.parq
+    formatter: lsst.daf.butler.formatters.parquet.ParquetFormatter
+- type: dataset_type
+  name: preloaded_diaSources
+  dimensions:
+  - instrument
+  - detector
+  - group
+  storage_class: DataFrame
+  is_calibration: false
+- type: dataset
+  dataset_type: preloaded_diaSources
+  run: dia_catalogs/apdb
+  records:
+  - dataset_id:
+    - !uuid '0b0b85ec-7509-4192-aeb4-3b569aae490f'
+    data_id:
+    - instrument: LSSTCam-imSim
+      detector: 168
+      group: '943296'
+    path: dia_catalogs/apdb/preloaded_diaSources/943296/preloaded_diaSources_LSSTCam-imSim_943296_R41_S20_dia_catalogs_apdb.parq
+    formatter: lsst.daf.butler.formatters.parquet.ParquetFormatter
+  - dataset_id:
+    - !uuid '0bbf567e-70a0-43bd-be4a-acf0775e09a7'
+    data_id:
+    - instrument: LSSTCam-imSim
+      detector: 164
+      group: '982985'
+    path: dia_catalogs/apdb/preloaded_diaSources/982985/preloaded_diaSources_LSSTCam-imSim_982985_R41_S02_dia_catalogs_apdb.parq
+    formatter: lsst.daf.butler.formatters.parquet.ParquetFormatter
 - type: dataset_type
   name: pretrainedModelPackage
   dimensions: []

doc/ap_verify_ci_dc2/index.rst

@@ -20,7 +20,7 @@ The input data were chosen arbitrarily, so they are not even suitable for testin
 Dataset contents
 ================
 
-The ``preloaded/`` butler repo contains two overlapping exposures of ``LSSTCam-imSim`` from DC2 ``tract=4431``, with ``patches=(9,10,16,17)``, ``goodSeeingCoadd`` datasets for those patches to make templates from, DC2 refcats covering that region, and skymaps, calibrations, and pretrained ML models necessary to process the data.
+The ``preloaded/`` butler repo contains two overlapping exposures of ``LSSTCam-imSim`` from DC2 ``tract=4431``, with ``patches=(9,10,16,17)``, ``goodSeeingCoadd`` datasets for those patches to make templates from, DC2 refcats covering that region, and skymaps, calibrations, mock APDB outputs based on the raw images, and pretrained ML models necessary to process the data.
 
 * visit=982985, detector=164
 * visit=943296, detector=168