Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add retries to download functions #58

Closed
tillywoodfield opened this issue Mar 5, 2024 · 1 comment
Closed

Add retries to download functions #58

tillywoodfield opened this issue Mar 5, 2024 · 1 comment

Comments

@tillywoodfield
Copy link
Contributor

Is your feature request related to a problem? Please describe.

When calling functions from utils.download, occasionally we see a connection error which resolves itself on a manual rerun.

Describe the solution you’d like

Add a retry around all of the top-level functions from utils.download.

Describe alternatives you’ve considered

We can add a retry in the application which is using iatikit, but think it would be more useful to add it to iatikit directly so that others can benefit from it.

Additional context

Example of the connection error:

Traceback (most recent call last):
  File "/home/iatitables/iatitables/runner.py", line 6, in <module>
    iatidata.run_all(processes=1)
  File "/home/iatitables/iatitables/iatidata/__init__.py", line 1368, in run_all
    process_registry(refresh=refresh, sample=sample, processes=processes)
  File "/home/iatitables/iatitables/iatidata/__init__.py", line 335, in process_registry
    save_all(sample=sample, parts=processes, refresh=refresh)
  File "/home/iatitables/iatitables/iatidata/__init__.py", line 308, in save_all
    get_standard(refresh)
  File "/home/iatitables/iatitables/iatidata/__init__.py", line 101, in get_standard
    iatikit.download.standard()
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/iatikit/utils/download.py", line 250, in standard
    codelists()
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/iatikit/utils/download.py", line 187, in codelists
    version_codelist = get_codelist(codelist_name, version)
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/iatikit/utils/download.py", line 165, in get_codelist
    version_codelist = requests.get(codelist_url).json()
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/sessions.py", line 725, in send
    history = [resp for resp in gen]
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/sessions.py", line 725, in <listcomp>
    history = [resp for resp in gen]
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/sessions.py", line 266, in resolve_redirects
    resp = self.send(
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/iatitables/iatitables/.ve/lib/python3.10/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
@andylolz
Copy link
Member

andylolz commented Mar 6, 2024

Fixed in #59.

@andylolz andylolz closed this as completed Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants