Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problems with data import #354

Open
odscjames opened this issue Apr 27, 2022 · 2 comments
Open

Problems with data import #354

odscjames opened this issue Apr 27, 2022 · 2 comments

Comments

@odscjames
Copy link
Collaborator

Problems were seen with this. The worker had the following Python error repeated:


   File "/home/iatidatastoreclassic/iatidatastoreclassic/iati_datastore/iatilib/crawler.py", line 303, in update_dataset
     resource = fetch_resource(dataset, ignore_hashes)
   File "/home/iatidatastoreclassic/iatidatastoreclassic/iati_datastore/iatilib/crawler.py", line 117, in fetch_resource
     content = d.raw_xml
   File "/home/iatidatastoreclassic/iatidatastoreclassic/.ve/lib/python3.8/site-packages/iatikit/data/dataset.py", line 68, in raw_xml
     with open(self.data_path, 'rb') as handler:
   TypeError: expected str, bytes or os.PathLike object, not NoneType

#353 should provide an immediate fix, but making this issue to make work visible and get to the bottom of this.

@odscjames
Copy link
Collaborator Author

One issue that came up is that the above error was not seen in Sentry. I'm not sure why; sentry is set up for the CLI (just verified). At first I thought it was RQ catching the error and putting it in it's own logs, so sentry doesn't get a chance to see it. Sentry does have RQ integrations that we may have needed - https://docs.sentry.io/platforms/python/guides/rq/ But I tried locally and an error in a task does get reported to sentry without that integration, so I'm not sure what is happening.

@odscjames
Copy link
Collaborator Author

Fixed an issue in server monitoring so we get alerts on unhealthy status in future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant