1085 Ecocounter Open Data Schema+DAG+QC plots #1096
Conversation
…in both site and flow ids)
| @@ -106,46 +96,93 @@ def reminder_message(ds = None, **context): | |||
| wait_till_10th.doc_md = """ | |||
| Wait until the 10th day of the month to export data. Alternatively mark task as success to proceed immediately. | |||
| """ | |||
|
|
|||
| @task() | |||
| def get_years(ds=None): | |||
There was a problem hiding this comment.
did we add this for mapped task naming? It was using ds before right?
There was a problem hiding this comment.
Yes, the insert_and_download_data task_group is mapped over the output of get_years. The purpose is to run the exports for last two months (which may be two separate years), since new data frequently arrives within ~30 days.
There was a problem hiding this comment.
OK to remove sensitivity_changes since it's superseded by sensitivity_history.
Not sure why flow_id = 101042942 has a stray date_range = (,) .
The readme should mention:
sensitivity_historyis manually updated based on emails with with Eco-counter Technical Support Specialist (Derek Yates as of 2024).- what the numbered settings mean (least to most selective, or not standardized)
- Eco-counter must confirm what the current and new settings are whenever a sensitivity is adjusted, along with the date the change is applied.
There was a problem hiding this comment.
OK to remove sensitivity_changes since it's superseded by sensitivity_history.
✅
Not sure why flow_id = 101042942 has a stray date_range = (,) .
That flow_id never had any data. Deleted the entry.
The readme should mention...
Adding!
There was a problem hiding this comment.
How to separate scooter flows calibration factors from bike flows? Currently we apply the same "total volume" factor to both, but we will have separate validation factors for these veh types soon. (Not a blocker for this release)
Readme should state:
ecocounter.calibration_factorsexcludes factors inecocounter.manual_counts_infowith flagged asdo_not_use=true.do_not_useis entered manually on a case-by-case basis depending on which calibration study is the 'best'.- Caution is required when assigning
do_not_useto calibration studies. Avoid as much as possible applying a new factor to data that has already been calibrated and published to Open Data.
There was a problem hiding this comment.
The way to separate scooter calibration factors would be to edit ecocounter.validation_results, particularly the flow_ids CTE to link different flow_ids to those studies.
|
@gabrielwol, I'm not finding this file that my review was requested for: volumes/ecocounter/views/create-view-counts_calibrated.sql |
Renamed to "counts" from counts_calibrated |
In bigdata, I see both VIEWs |
@A-DUYVESTYN yes, we would keep only
Will work on readmes today. |
|
OK. I'm not clear on process for assigning |
… airflow imports (#1115)
What this pull request accomplishes:
New Monthly DAG to insert/download Ecocounter data. Includes prompts to review data and check if data for all days are present. @chmnata
Open Data tables + functions @A-DUYVESTYN
open_data_daily_counts+ fnopen_data_daily_counts_insert,open_data_raw_counts+ fnopen_data_raw_counts_insert,open_data_sitesCalibration factor supporting tables @A-DUYVESTYN
sensitivity_changeswhen we are satisfied with changes.Does not need significant review:
R plots
qc_graph_volumespostgres functionMisc.
counterto sites table (commonly used ID field in Ecocounter communications/dashboard)ecocounter_pull.py,pull_data_from_api.py.direction_mainfield to flows: easily group dominant/contraflow togethercounts,counts_calibratedIssue(s) this solves:
What, in particular, needs to reviewed:
What needs to be done by a sysadmin after this PR is merged
counts_old(saved the old version of counts for monitoring project dependency)discontinuitiestable