Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cloud sensor assets' partitions get wiped every 60 seconds #124

Open
penelopeysm opened this issue Jun 26, 2024 · 0 comments
Open

Cloud sensor assets' partitions get wiped every 60 seconds #124

penelopeysm opened this issue Jun 26, 2024 · 0 comments
Labels
bug Something isn't working low-priority

Comments

@penelopeysm
Copy link
Member

penelopeysm commented Jun 26, 2024

I noticed that this is a weird side effect of this commit (Urban-Analytics-Technology-Platform/popgetter@01f84e7) which in turn was part of #108 (just merged).

The intent of that commit is that when Dagster is started up, the list of partitions is regenerated, in order to avoid inconsistencies with previous Dagster runs (e.g. if asset prefixes are changed / country assets are removed, the old partition names would persist).

However, it seems that the Dagster daemon also performs a complete code reload every 60 seconds. (You can observe this by adding a print statement at the top level of python/popgetter/__init__.py.)

What this means is that the list of partitions gets wiped and regenerated every 60 seconds. This is completely harmless in that it doesn't affect the publishing process itself, so this is almost a non-issue, but it can be a little bit disconcerting when viewing in the web UI.

@penelopeysm penelopeysm added bug Something isn't working low-priority labels Jun 26, 2024
@andrewphilipsmith andrewphilipsmith moved this to Inport in Popgetter Aug 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working low-priority
Projects
Status: Backlog:
Development

No branches or pull requests

1 participant