-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Miovision: discrepencies between volumes_15min_mvt and unacceptable_gaps #868
Comments
All these discrepencies were caused by 5 unacceptable_gap entries:
SELECT DISTINCT un.dt, un.intersection_uid, un.gap_start, un.gap_end
FROM miovision_api.volumes_15min_mvt AS v
JOIN miovision_api.unacceptable_gaps AS un USING (intersection_uid, datetime_bin)
WHERE v.volume IS NOT NULL
AND v.datetime_bin >= '2023-01-02' AND v.datetime_bin < '2024-02-06' The first/fourth/fifth record above are because the find_gaps script identifies gaps that overlap midnight, but started the day before. I didn't anticipate that this would make the pipeline non-idempotent. I will remove this behaviour now at the minor expense of missing small gaps that start over midnight. The other two issues I can't think of a cause for! I can re-aggregate these 2 ranges and we can keep our eyes out for similar issues in the future. |
remove 15 minute buffer + change to adding artificial point at each day
Originally posted by @gabrielwol in #863 (comment)
The text was updated successfully, but these errors were encountered: