2023R1: IOManager and Connectivity Service consolidation

[bieryAtFnal]

Introduction

The goals of this work are the following:

• discuss any remaining design or implementation issues with the IOManager changes and/or the current use of the Connectivity Service, decide on plans of action, and implement the desired changes
• modify the code that makes use of the IOManager and modify the associated configuration to take advantage of the IOManager changes and the addition of the Connectivity Service

Basically, we want to finish up the connection-related changes that part of the previous release and make full use of them.

Planned Deliverables

1. Update “old-style” configuration and code to take advantage of IOM and ConnSvc changes

   Narrative

   This deliverable has several elements:

   1. Convert from using a translation from module-local connection names to connection IDs to using agreements between modules about the contents of the connection ID string. For example, switching from

      m_trigger_decision_input = get_iom_receiver<dfmessages::TriggerDecision>(ci["trigger_decision_input"]);
      to
      m_trigger_decision_input = get_iom_receiver<dfmessages::TriggerDecision>("trigger_decisions_for_" + m_this_trb_source_id.to_string());
      

      This allows us to do the next item.
   2. Remove the "conn_ref" blocks from the generated configuration. The goal here is to reduce the complexity of our configurations.
   3. Replace statically-defined connections in the configuration. I'm not sure that we have an example of this yet.
   Observables

   (to-do: list the observables here. For reference, Eric created a page with examples [here](https://hackmd.io/@eflumer/SySCzI4so).
   Suggested testing

   1. confirm that all of the observables are present
   2. (add more validation/acceptance tests here)

2. Update the readout code (readoutmodules::CreateReadout) to use the data type from the connection ID to determine the ReadoutModel template parameters when creating the DataLinkHandler modules, instead of searching for particular strings in the connection ID.

   Narrative

   will be filled in soon...
   Observables

   will be filled in soon...
   Suggested testing

   1. confirm that all of the observables are present
   2. (add more validation/acceptance tests here)

3. Add support for looking up connections across different DAQ sessions

   Narrative

   will be filled in soon...
   Observables

   will be filled in soon...
   Suggested testing

   1. confirm that all of the observables are present
   2. (add more validation/acceptance tests here)

4. Test the performance of the Connectivity Service, and improve it if needed

   Narrative

   will be filled in soon... A specific task: Add configuration parameters to control the rate of keepalive publish calls and subscriber updates (currently 1/s hard-coded), since this can cause excess traffic on large systems.
   Observables

   1. The primary deliverables here are the performance test results. This could either be a table of results for different test conditions (e.g. number of connecting daq_application), or it could be a plot that shows the results graphically.
   2. A test program that allows us to re-run the performance tests whenever we want, especially when changes are made in the Connectivity Service.
   3. This is not a required Observable, but it is worth mentioning that there are debug messages that are printed out by the Connectivity Service, and their level of verbosity can be controlled with the CONNECTION_FLASK_DEBUG environmental variable (details at the bottom of this page).
   Suggested testing

   1. confirm that all of the observables are present
   2. (add more validation/acceptance tests here)

5. Extend the test suite with dynamic reconnection tests

   Narrative

   will be filled in soon...
   Observables

   will be filled in soon...
   Suggested testing

   1. confirm that all of the observables are present
   2. (add more validation/acceptance tests here)

6. Automatically start and stop the Connectivity Service in small-scale tests systems, when desired.

   Narrative

   In smaller-scale testing, the Connectivity Service will be started automatically by nanorc, based on a daqconf-level configuration parameter. This config param will allow us to disable such automatic starting of the service in cases in which we want to start it by hand, for example, when running a Timing DAQ session and a Data Taking DAQ session.
   Observables

   will be filled in soon...
   Suggested testing

   1. confirm that all of the observables are present
   2. (add more validation/acceptance tests here)

7. Documentation

   The original intention of this deliverable was to confirm that the documentation for describing and using connectivity service based connections has been provided. There may be other documentation that would be useful, and this item may be related to the stretch goal, but we're including it here, for reference.

   Narrative

   will be filled in soon...
   Observables

   will be filled in soon...
   Suggested testing

   1. confirm that all of the observables are present
   2. (add more validation/acceptance tests here)

Stretch Goals

1. Create a document that describes the requirements for connections between DAQ modules/processes and describes the patterns that are present and recommended in our code.

   Narrative

   In smaller-scale testing, the Connectivity Service will be started automatically by nanorc, based on a daqconf-level configuration parameter. This config param will allow us to disable such automatic starting of the service in cases in which we want to start it by hand, for example, when running a Timing DAQ session and a Data Taking DAQ session.
   Observables

   will be filled in soon...
   Suggested testing

   This would be icing on the cake, but we could verify that the code adheres to the patterns described in the document.

Deferred/Excluded Deliverables

None so far.

Reference Material

• Introductory slides from Eric
• Introductory slides from Gordon