Port Scotland

[sgreenbury]

Contributes towards #36 and supersedes #26 since porting to dagster.

Updated: this version now implements the country base class approach #98. Further revisions to be done with #138.

---

Remaining tasks

• Revise the catalog with metadata class fields
• Add descriptions for tables from table index
• Subset some assets for specific partition key (OA11/LC1117SC)
• Check geometries available
• Add source data release
• Update pipeline to output as required following update output types of publishing assets, add cloud sensors #92 and potentially Port Northern Ireland #98
• Reshape data across tables for derived metrics to not lose any data include generating column names and hxltags programmatically conditional on the table variables
• Merge main and add required changes
• Confirm pipeline runs

[andrewphilipsmith]

@sgreenbury Please could you add detail about how this PR relates to #26? Does this supersede #26, or will both PRs need to be merged in due course? Many thanks

[sgreenbury]

@sgreenbury Please could you add detail about how this PR relates to #26? Does this supersede #26, or will both PRs need to be merged in due course? Many thanks

Yes, this supersedes #26 since porting to dagster. I'll update the first comment to clarify and close #26. Thanks!

[sgreenbury]

Following the discussion just now, one missing detail that should be implemented is capturing the SourceDataRelease for each table. This seems to be provided by the first column of the metadata catalog (e.g. 3C is release 3C for the corresponding census). From this a set of SourceDataRelease metadatas should be included (possibly in a separate module) and populated and then looked up as the source_tables are constructed.

[andrewphilipsmith]

This all looks good to me.

The various comments are all minor things (and one's I'm no doubt guilty of myself!). I think the best option would be to merge this and open issues, or new PRs to deal with most of the comments.

[sgreenbury]

Thanks @andrewphilipsmith! I'll go ahead and merge this now once the checks pass and then open new issues/PRs for the areas you mention + any fixes for the currently missing derived metrics.