HTAN2 Data Model & BigQuery Architecture Planning

[aditigopalan]

After discussion with Darya on 11/20, creating this ticket to track our BigQuery architecture and data flow planning for Phase 2.

Including the following action items

• Create UML diagram for BigQuery tables. (Dar'ya made this diagram)
• Define which tables are mutable or automated.
• Determine if new tables are needed for Phase 2.
• Review if cloud workflows need updates. Discuss roles in relation to the Synapse uploads and schematic releases.
• Draft a design document for Phase 2 architecture.

[aclayton555]

24-11/12 Close-out: Dar'ya has been working on this with Aditi and Adam. Link to Figma: https://www.figma.com/board/7MaRE8wDQABuNg5GqJXN9x/HTAN2-BQ-Medallion-Architecture?node-id=0-1&p=f&t=hGRTfR7YVkW5jbqq-0

Notes from preliminary discussion on medallion tiers:

• Any time we use a SynapseID, we should add a version number
• Data model version will soon be added as an annotation to manifests
• Also chat with Ino about how (gold?) tier will be used with portal

Also discussed if/how Bigquery tables can be used as a monitoring tool to monitor sparseness. (@PozhidayevaDarya not sure if this this within scope for the designs in progress here, or if these should be developed longer term in a separate ticket?)