add documentation for IOSvc

doc/PodioInputOutput.md

@@ -0,0 +1,235 @@
+<!--
+Copyright (c) 2014-2024 Key4hep-Project.
+
+This file is part of Key4hep.
+See https://key4hep.github.io/key4hep-doc/ for further info.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Reading and writing EDM4hep files in Gaudi
+
+The facilities to read and write EDM4hep (or in general event data models based on podio) are provided by [k4FWCore](https://github.com/key4hep/k4FWCore). This page will describe their usage, but not go into too much details of their internals. This page also assumes a certain familiarity with Gaudi, i.e. most of the snippets just show a minimal configuration part, and not a complete runnable example.
+
+## Accessing event data
+
+`IOSvc` is an external Gaudi service for reading and writing EDM4hep and other PODIO-based data models. The service should be imported from `k4FWCore` and named "IOSvc" as other components may look for it under this name.
+
+```python
+from k4FWCore import IOSvc
+
+io_svc = IOSvc("IOSvc")
+```
+
+After instantiation the service should be register as an external service in the `ApplicationMgr`. Similarly, it's important to import the `ApplicationMgr` from `k4FWCore`:
+
+```python
+from k4FWCore import ApplicationMgr
+
+ApplicationMgr(
+    # other args
+    ExtSvc=[
+        io_svc,
+        # other services
+    ]
+)
+```
+
+### Reading events
+
+The `IOSvc` supports reading ROOT files containing PODIO-based data-models such as EDM4hep. Both files written with ROOT TTree or RNTuple backend are supported with the backend inferred automatically from the files themselves.
+
+The `Input` property can be used to specify the input. The `IOSvc` will not read any files unless the `Input` property is specified.
+
+::::{tab-set}
+:::{tab-item} Python
+```python
+io_svc.Input = "input.root"
+```
+:::
+:::{tab-item} CLI
+```sh
+k4run <steering-file> --IOSvc.Input input.root
+```
+:::
+::::
+
+:::{note}
+The value assigned to the `Input` will be processed as is, in particular without regular expression or glob expansion.
+:::
+
+A list of filenames can be given in order to specify multiple input files:
+
+::::{tab-set}
+:::{tab-item} Python
+```python
+io_svc.Input = ["input.root", "another_input.root", ]
+```
+:::
+:::{tab-item} CLI
+```sh
+k4run <steering-file> --IOSvc.Input input.root another_input.root
+```
+:::
+::::
+
+
+During processing, for each event in the Gaudi event loop the `IOSvc` will read a frame from the input and populate the Gaudi Transient Event Store (TES) with the collections stored in that frame.
+
+The `FirstEventEntry` property of `IOSvc` can be used to start processing from a given frame instead of from the first frame in the input:
+
+::::{tab-set}
+:::{tab-item} Python
+```python
+io_svc.FirstEventEntry = 7 # default 0
+```
+:::
+:::{tab-item} CLI
+```sh
+k4run <steering-file> --IOSvc.FirstEventEntry 7
+```
+:::
+::::
+
+A list of collection names can be assigned to the `CollectionNames` property of `IOSvc` to limit the number of collections that will be populated. Without specifying the `CollectionNames` all present collections will be read and put into TES.
+
+::::{tab-set}
+:::{tab-item} Python
+```python
+io_svc.CollectionNames = ["MCParticles", "SimTrackerHits"]
+```
+:::
+:::{tab-item} CLI
+```sh
+k4run <steering-file> --IOSvc.CollectionNames "MCParticles" "SimTrackerHits"
+```
+:::
+::::
+
+### Writing events
+
+The `IOSvc` supports writing PODIO-based data-models such as EDM4hep to the ROOT output. The `Output` property can be used to specify the output. The `IOSvc` will not write any files unless the `Output` property is specified.
+
+::::{tab-set}
+:::{tab-item} Python
+```python
+io_svc.Output = "output.root"
+```
+:::
+:::{tab-item} CLI
+```sh
+k4run <steering-file> --IOSvc.Output output.root
+```
+:::
+::::
+
+:::{note}
+Unlike the `Input`, the `Output` property should be a single string even when writing multiple files is expected. When the size limit for an output file is reached, the system will automatically open a new file and start writing to it.
+:::
+
+The writing backend can be specified with the `OutputType` property of `IOSvc`. The allowed values are `"ROOT"` for TTree-based output or `"RNTuple"` for RNTuple-based output. By default the `"ROOT"` backend is used.
+
+::::{tab-set}
+:::{tab-item} Python
+```python
+io_svc.OutputType = "RNTuple"
+```
+:::
+:::{tab-item} CLI
+```sh
+k4run <steering-file> --IOSvc.OutputType "RNTuple"
+```
+:::
+::::
+
+During processing, at the end of each event from the Gaudi event loop the `IOSvc` will write a frame with the collection present in TES. By default all the collections will be written. The `outputCommands` property of `IOSvc` can be used to specify commands to select which collections should be written. For example, the following commands will skip writing all the collections except for the collections named `MCParticles1`, `MCParticles2` and `SimTrackerHits`:
+
+::::{tab-set}
+:::{tab-item} Python
+```python
+io_svc.outputCommands = [
+    "drop *",
+    "keep MCParticles1",
+    "keep MCParticles2",
+    "keep SimTrackerHits",
+]
+```
+:::
+:::{tab-item} CLI
+```sh
+k4run <steering-file> --IOSvc.outputCommands \
+    "drop *" \
+    "keep MCParticles1" \
+    "keep MCParticles2" \
+    "keep SimTrackerHits"
+```
+:::
+::::
+
+## Accessing metadata
+
+The k4FWCore provides the `MetadataSvc` that allows accessing user metadata in PODIO-based data-models. There is no need to instantiate the `MetadataSvc` explicitly when using `IOSvc` as `IOSvc` can instantiate it on its own if needed.
+
+When both `Input` and `Output` properties of `IOSvc` are defined, all the metadata originally present in the input will be propagated to the output, possibly adding also any user metadata created during processing.
+
+Unlike event data, metadata is not exposed to users through the Gaudi TES and cannot be accessed directly by algorithms in the same way. Instead, handling metadata is encapsulated within the algorithm implementation itself. For more details on how this is managed, refer to the developer documentation.
+
+
+## Migrating from the legacy `k4DataSvc`
+
+Migrating from the legacy `k4DataSvc` or `PodioDataSvc` is rather straightforward. On a steering file level the `PodioDataSvc` should be replaced with the `IOSvc`, while the `PodioInput` and `PodioOutput` algorithms should be removed. For example:
+
+```diff
+-from Configurables import k4DataSvc
+-from Configurables import PodioInput
+-from Configurables import PodioOutput
++from k4FWCore import IOSvc
+from k4FWCore import ApplicationMgr
+from Configurables import SelectorAlg
+
+-podioevent = k4DataSvc("EventDataSvc")
+-podioevent.input = "example_input.root"
++io_svc = IOSvc("IOSvc")
++io_svc.Input= "example_output.root"
+
+-inp = PodioInput()
+-inp.collections = ["MCParticles", "SimTrackerHits", "TrackerHits", "Tracks"]
++io_svc.CollectionNames = ["MCParticles", "SimTrackerHits"]
+
+alg = SelectorAlg(
+    "Selector",
+    InputParticles="MCParticles",
+    InputHits="SimTrackerHits",
+    Output="SelectedParticles",
+)
+
+-oup = PodioOutput()
+-oup.filename = "example_output.root"
+-oup.outputCommands = ["drop MCParticles"]
++io_svc.Output = "example_output.root"
++oup.outputCommands = ["drop MCParticles"]
+
+
+ApplicationMgr(
+-    TopAlg=[inp, alg,oup],
++    TopAlg=[alg],
+    EvtSel="NONE",
+-    ExtSvc=[podioevent],
++    ExtSvc=[io_svc],
+)
+```
+
+Both functional algorithms and classic algorithms are compatible with either `IOSvc` or `PodioDataSvc`.
+
+The biggest challenge for the transition are the algorithms and services that explicitly request and operate on `PodioDataSvc`. These components are not compatible with `IOSvc` and their internals have to be adapted for the usage with `IOSvc` case by case.
+
+If you encounter an algorithm or service that seems to be incompatible `IOSvc`, please open an issue in the bugtracker to report it for further investigation.