Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-40853: Implement new source injection for AP #225

Merged
merged 3 commits into from
Jul 24, 2024
Merged

DM-40853: Implement new source injection for AP #225

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5f8f858
Fixing ApVerifyWithFakes to use new source_injection package
BrunoSanchez May 13, 2024
6a298b4
Fix E721 lint warning (type comparison)
timj Jul 11, 2024
cc4e261
Removed pipelines associated with deprecated Calibrate yamls
BrunoSanchez Jul 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 0 additions & 8 deletions pipelines/DECam/ApVerifyWithFakes.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,3 @@ instrument: lsst.obs.decam.DarkEnergyCamera
description: Fully instrumented AP pipeline with fakes, specialized for DECam
imports:
- location: $AP_VERIFY_DIR/pipelines/_ingredients/ApVerifyWithFakes.yaml
exclude:
- processCcd
# The existing Fakes insertion system is not currently compatible with the
# new CalibrateImageTask, so the fakes pipeline still is on the old tasks.
- location: $AP_PIPE_DIR/pipelines/DECam/ProcessCcdCalibrate.yaml
# Can't use $AP_PIPE_DIR/pipelines/DECam/ApPipeWithFakes.yaml here
# because the changes made by that file and _ingredients/ApVerifyWithFakes.yaml
# are hard to separate.
8 changes: 0 additions & 8 deletions pipelines/HSC/ApVerifyWithFakes.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,3 @@ instrument: lsst.obs.subaru.HyperSuprimeCam
description: Fully instrumented AP pipeline with fakes, specialized for HSC
imports:
- location: $AP_VERIFY_DIR/pipelines/_ingredients/ApVerifyWithFakes.yaml
exclude:
- processCcd
# The existing Fakes insertion system is not currently compatible with the
# new CalibrateImageTask, so the fakes pipeline still is on the old tasks.
- location: $AP_PIPE_DIR/pipelines/HSC/ProcessCcdCalibrate.yaml
# Can't use $AP_PIPE_DIR/pipelines/HSC/ApPipeWithFakes.yaml here
# because the changes made by that file and _ingredients/ApVerifyWithFakes.yaml
# are hard to separate.
8 changes: 0 additions & 8 deletions pipelines/LSSTCam-imSim/ApVerifyWithFakes.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,3 @@ instrument: lsst.obs.lsst.LsstCamImSim
description: Fully instrumented AP pipeline with fakes, specialized for LSSTCam-imSim
imports:
- location: $AP_VERIFY_DIR/pipelines/_ingredients/ApVerifyWithFakes.yaml
exclude:
- processCcd
# The existing Fakes insertion system is not currently compatible with the
# new CalibrateImageTask, so the fakes pipeline still is on the old tasks.
- location: $AP_PIPE_DIR/pipelines/LSSTCam-imSim/ProcessCcdCalibrate.yaml
# Can't use $AP_PIPE_DIR/pipelines/LSSTCam-imSim/ApPipeWithFakes.yaml here
# because the changes made by that file and _ingredients/ApVerifyWithFakes.yaml
# are hard to separate.
14 changes: 11 additions & 3 deletions pipelines/_ingredients/ApVerifyWithFakes.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,17 @@ tasks:
contracts:
# Metric inputs must match pipeline outputs
# Use of ConnectionsClass for templated fields is a workaround for DM-30210
- fakesMatch.connections.ConnectionsClass(config=fakesMatch).matchedDiaSources.name ==
- injectedMatch.connections.ConnectionsClass(config=injectedMatch).matchedDiaSources.name ==
apFakesCompletenessMag20t22.connections.ConnectionsClass(config=apFakesCompletenessMag20t22).matchedFakes.name
- fakesMatch.connections.ConnectionsClass(config=fakesMatch).matchedDiaSources.name ==
- injectedMatch.connections.ConnectionsClass(config=injectedMatch).matchedDiaSources.name ==
apFakesCompletenessMag22t24.connections.ConnectionsClass(config=apFakesCompletenessMag22t24).matchedFakes.name
- fakesMatch.connections.ConnectionsClass(config=fakesMatch).matchedDiaSources.name ==
- injectedMatch.connections.ConnectionsClass(config=injectedMatch).matchedDiaSources.name ==
apFakesCompletenessMag24t26.connections.ConnectionsClass(config=apFakesCompletenessMag24t26).matchedFakes.name
- injectedMatch.connections.ConnectionsClass(config=injectedMatch).matchedDiaSources.name ==
apFakesCountMag20t22.connections.ConnectionsClass(config=apFakesCountMag20t22).matchedFakes.name
- injectedMatch.connections.ConnectionsClass(config=injectedMatch).matchedDiaSources.name ==
apFakesCountMag22t24.connections.ConnectionsClass(config=apFakesCountMag22t24).matchedFakes.name
- injectedMatch.connections.ConnectionsClass(config=injectedMatch).matchedDiaSources.name ==
apFakesCountMag24t26.connections.ConnectionsClass(config=apFakesCountMag24t26).matchedFakes.name
- injectedMatch.connections.ConnectionsClass(config=injectedMatch).matchedDiaSources.name ==
apFakesCount.connections.ConnectionsClass(config=apFakesCount).matchedFakes.name
18 changes: 7 additions & 11 deletions pipelines/_ingredients/ConversionsForFakes.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,21 +11,15 @@ imports:
# Fakes pipeline doesn't produce non-fakes DiaSourceTable.
- consolidateDiaSourceTable
tasks:
# Conversion of fakes_src [afw.table] to fakes_source [Parquet]
writeSourceTable:
class: lsst.pipe.tasks.postprocess.WriteSourceTableTask
config:
connections.catalogType: parameters.fakesType
# TODO: TransformSourceTableTask can't be run until we create a functor
# config that doesn't depend on shapeHSM.
# Merging of fakes_source [detector-level] to fakes_sourceTable_visit [visit-level]
consolidateSourceTable:
class: lsst.pipe.tasks.postprocess.ConsolidateSourceTableTask
config:
connections.catalogType: parameters.fakesType
# Skip DPDD-ified sourceTable and just use source directly
connections.inputCatalogs: "{catalogType}source"

# This is output directly by CalibrateImageTask.
connections.inputCatalogs: initial_stars_detector
connections.outputCatalog: initial_stars
# Merging of fakes_*Diff_diaSrcTable [detector-level Parquet] to fakes_diaSourceTable [visit-level]
consolidateDiaSourceTable:
class: lsst.pipe.tasks.postprocess.ConsolidateSourceTableTask
Expand All @@ -36,22 +30,24 @@ tasks:
# and fakesType is more stable than coaddName.
connections.inputCatalogs: "fakes_{catalogType}Diff_diaSrcTable"
connections.outputCatalog: fakes_diaSourceTable

# Creation of fakes_visitSummary
consolidateVisitSummary:
class: lsst.pipe.tasks.postprocess.ConsolidateVisitSummaryTask
config:
connections.calexpType: parameters.fakesType
connections.calexp: "{calexpType}calexp"
connections.calexp: "{calexpType}initial_pvi"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here and below, should this be {fakesType} instead of {calexpType}?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so. As the parameter template inside the task is named calexpType and we point to the fakesType value already.
Bear in mind that this consolidation task is a very general task and not-ap specific, nor fakes-specific.

connections.visitSummary: "{calexpType}visitSummary"
connections.visitSummarySchema: "{calexpType}visitSummary_schema"
# Conversion of fakes_visitSummary [visit-level afw.table] to fakes_visitTable [instrument-level Parquet]
makeVisitTable:
class: lsst.pipe.tasks.postprocess.MakeVisitTableTask
config:
connections.calexpType: parameters.fakesType
connections.visitSummaries: "{calexpType}visitSummary"
# Conversion of fakes_visitSummary [visit-level afw.table] to fakes_ccdVisitTable [instrument-level Parquet]
makeCcdVisitTable:
class: lsst.pipe.tasks.postprocess.MakeCcdVisitTableTask
config:
connections.calexpType: parameters.fakesType
connections.visitSummaryRefs: "{calexpType}visitSummary"

Loading