Bug in saving vectors to configuration_metadata ?

[giovannimarchiori]

I am running k4run in the nightly and then inspecting the configuration_metadata of the produced output file. Some of the properties - typically vectors - seem not to be filled.
Example: for this algorithm (https://github.com/HEP-FCC/k4RecCalorimeter/blob/main/RecFCCeeCalorimeter/src/components/AugmentClustersFCCee.h) that I configure with

    augmentClusterAlg = AugmentClustersFCCee("Augment" + outputClusters,
                                                 inClusters=clusterAlg.clusters.Path,
                                                 outClusters="Augmented" + clusterAlg.clusters.Path,
                                                 systemIDs=[4],
                                                 systemNames=["EMB"],
                                                 numLayers=[ecalBarrelLayers],
                                                 readoutNames=[inputReadouts["ecalBarrel"]],
                                                 layerFieldNames=["layer"],
                                                 thetaRecalcWeights=[ecalBarrelThetaWeights],
                                                 # do_photon_shapeVar=runPhotonIDTool,
                                                 do_photon_shapeVar=True,  # we want these variables to train the photon ID BDT
                                                 do_widthTheta_logE_weights=logEWeightInPhotonID,
                                                 OutputLevel=INFO
                                                 )
        TopAlg += [augmentClusterAlg]

I see in the configuration_metadata:

, AugmentEMBCaloClusters.AuditAlgorithms = "";
, AugmentEMBCaloClusters.AuditExecute = "True";
, AugmentEMBCaloClusters.AuditFinalize = "";
, AugmentEMBCaloClusters.AuditInitialize = "";
, AugmentEMBCaloClusters.AuditReinitialize = "";
, AugmentEMBCaloClusters.AuditRestart = "";
, AugmentEMBCaloClusters.AuditStart = "";
, AugmentEMBCaloClusters.AuditStop = "";
, AugmentEMBCaloClusters.Blocking = "";
, AugmentEMBCaloClusters.Cardinality = "";
, AugmentEMBCaloClusters.Enable = "";
, AugmentEMBCaloClusters.ErrorMax = "";
, AugmentEMBCaloClusters.ExtraInputs = "";
, AugmentEMBCaloClusters.ExtraOutputs = "";
, AugmentEMBCaloClusters.FilterCircularDependencies = "";
, AugmentEMBCaloClusters.MonitorService = "";
, AugmentEMBCaloClusters.NeededResources = "";
, AugmentEMBCaloClusters.OutputLevel = "3";
, AugmentEMBCaloClusters.RegisterForContextService = "";
, AugmentEMBCaloClusters.Timeline = "";
, AugmentEMBCaloClusters.do_photon_shapeVar = "True";
, AugmentEMBCaloClusters.do_widthTheta_logE_weights = "False";
, AugmentEMBCaloClusters.inClusters = "'EMBCaloClusters'";
, AugmentEMBCaloClusters.layerFieldNames = "";
, AugmentEMBCaloClusters.maxDebugPrint = "10";
, AugmentEMBCaloClusters.moduleFieldNames = "";
, AugmentEMBCaloClusters.numLayers = "";
, AugmentEMBCaloClusters.outClusters = "'AugmentedEMBCaloClusters'";
, AugmentEMBCaloClusters.readoutNames = "";
, AugmentEMBCaloClusters.systemIDs = "";
, AugmentEMBCaloClusters.systemNames = "";
, AugmentEMBCaloClusters.thetaFieldNames = "";
, AugmentEMBCaloTopoClusters.thetaRecalcWeights = "";

so fields like systemIDs, systemNames, thetaFieldNames, ... are all empty. Is this normal?

On the other hand, if I look at the log file produced in output, those properties are properly printed, but others like inClusters and outClusters (I think everything that is not a GaudiProperty but rather the name of a DataHandle) are not printed

[k4run] Option name: AugmentEMBCaloTopoClusters.OutputLevel 3
[k4run] Option name: AugmentEMBCaloTopoClusters.systemIDs [4]
[k4run] Option name: AugmentEMBCaloTopoClusters.systemNames ['EMB']
[k4run] Option name: AugmentEMBCaloTopoClusters.numLayers [11]
[k4run] Option name: AugmentEMBCaloTopoClusters.thetaRecalcWeights [[-1, 3.0, 3.0, 3.0, 4.25, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0]]
[k4run] Option name: AugmentEMBCaloTopoClusters.readoutNames ['ECalBarrelModuleThetaMerged']
[k4run] Option name: AugmentEMBCaloTopoClusters.layerFieldNames ['layer']
[k4run] Option name: AugmentEMBCaloTopoClusters.thetaFieldNames ['theta']
[k4run] Option name: AugmentEMBCaloTopoClusters.moduleFieldNames ['module']
[k4run] Option name: AugmentEMBCaloTopoClusters.do_photon_shapeVar True
[k4run] Option name: AugmentEMBCaloTopoClusters.do_widthTheta_logE_weights False
[k4run] Option name: AugmentEMBCaloTopoClusters.maxDebugPrint 10

that makes it difficult to retrieve offline the settings used to produce a given sample.
Would it be possible to store in the configuration_metadata all properties including lists?
I see that for some cases this is already possible e.g. in configuration_metadata there is:
, CreatePositionedECalEndcapCells.CalibrateECalEndcap.samplingFraction = "[ 0.16419000 , 0.19289800 , 0.18783000 , 0.19320300 , 0.19392800 , 0.19228600 , 0.19995900 , 0.20015300 , 0.21263500 , 0.18034500 , 0.18488000 , 0.19476200 , 0.19777500 , 0.20050400 , 0.20555500 , 0.20360100 , 0.21087700 , 0.20837600 , 0.21634500 , 0.20145200 , 0.20213400 , 0.20756600 , 0.20815200 , 0.20988900 , 0.21174300 , 0.21318800 , 0.21586400 , 0.22972000 , 0.19251500 , 0.010323300 ]";
so I don't know exactly where the problem is and have no idea how to fix it myself...

[jmcarcell]

Let's go first with the main issue here: the empty parameters that are saved. I went into an Alma 9 node, sourced the nightlies, and ran that algorithm (AugmentClustersFCCee.h) by removing a big part of the initialize and execute since I don't have a complete steering file and I find that all the parameters are correctly saved. I find the same locally running non-functional algorithms in general, but I can see this issue happening for functional algorithms. So let's go first to the non-functional ones, can you try to give me a setup in which the parameters are not saved (meaning I need the steering file and input data if any)?.

[giovannimarchiori]

Hi @jmcarcell
thanks for your reply!
I am running with today's nightly (source /cvmfs/sw-nightlies.hsf.org/key4hep/setup.sh)
and
k4run run_digi_reco.py -n 1 --IOSvc.Input ALLEGRO_sim_nodchlayers.root
where you can grab run_digi_reco.py here: https://cernbox.cern.ch/s/pwZFH81FbOLNTw9
and ALLEGRO_sim_nodchlayers.root here: https://cernbox.cern.ch/s/MN1KlJ2AWRtJvwF

Then check the output with
podio-dump -c configuration_metadata -d ALLEGRO_sim_digi_reco.root
and see
, AugmentEMBCaloClusters.layerFieldNames = "";
and so on

[giovannimarchiori]

Hi @jmcarcell , with your fix in #262 the metadata is properly filled so I am going to close the issue

Thanks!