document how wis2box-user can ensure originating centre and subcentre is encoded into produced BUFR

[maaikelimper]

During the recent WIS2 Training in Brazil, the question of how to correctly encode the BUFR_ORIGINATING_CENTRE and BUFR_ORIGINATING_SUBCENTRE into the produced BUFR was again brought up.

We had some discussion about this previously in this issue:
#536

I think the issue was closed after deciding to set these to MISSING by default but there is no documentation on this in wis2box

@david-i-berry can you comment on this ?

[david-i-berry]

This was fixed and documented here: https://csv2bufr.readthedocs.io/en/latest/installation.html#environment-variables. We need to add more info and also bring it over to the WIS2box docs as well.

[maaikelimper]

OK thanks, so this is a documentation issue, I will update the title and label accordingly

[maaikelimper]

Setting environment variables in the wis2box.env will apply to all datasets shared by the wis2box-instance. Is it also possible to define these per dataset ? (this will be required by Caribbean wis2box but possibly also for others)

[maaikelimper]

need to understand if data standard will still require this in the future under WIS2 ?

[tomkralidis]

2024-11-21

• part of the wis2box workflow includes data transformation and encoding
• we should make efforts to provide centre/subcentre properly as part of being an RI with additional capabilities
• consider adding wis2box specific metadata for centre/subcentre (per CCT c11/c12) that can be used by pipelines producing BUFR
• csv2bufr
  • allow for specifying centre/subcentre in mapping template
  • apply accordingly (either via environment variables before invoking ecCodes, or specifying the BUFR sequence directly)
• wis2box
  • csv2bufr template can be updated in memory based on discovery metadata wis2box definitions of centre/subcentre

[david-i-berry]

• csv2bufr

  • allow for specifying centre/subcentre in mapping template
  • apply accordingly (either via environment variables before invoking ecCodes, or specifying the BUFR sequence directly)

• wis2box

  • csv2bufr template can be updated in memory based on discovery metadata wis2box definitions of centre/subcentre

• csv2bufr

  • setting the centre/subcentre fields has been supported since near the start but has become inconsistent between specifying template by id and specifying by filename, with different behaviours between the two. The current main branch of csv2bufr addresses this
  • the fields are only updated based on the ENVs when missing in the specified template (both by id and filename).
    • previously the centre/subcentre were updated regardless of whether the fields were missing in the mapping file when the template was specified by id.
    • when the template was specified by filename the centre/subcentre were not inserted/updated.
  • multiple centres can be supported in one instance, with a different mapping file for each centre or dynamically as per below.

• wis2box

  • The mapping file can be updated dynamically here: https://github.com/wmo-im/wis2box-api/blob/67e423a3fef3ebc089d10203f81573a099918da1/wis2box_api/plugins/process/csv2bufr.py#L175