When one composite loading results in error (for example, due to not reaching zenodo.org), sift fails silently to load any datasets

[gerritholl]

I'm trying to load a number of datasets including true_color. When loaded the first time, producing the true_color composite tries to reach zenodo.org to download (I think) spectral response functions. On my Windows notebook, this fails with a TimeoutError (employer security settings do not allow "unsupported" software to connect to the web). Sift does not handle this gracefully. A user not looking at the console will observe nothing happening at all. Rather than failing silently, Sift should continue loading those datasets it can load, then show an error message explaining why this one composite could not be loaded.

A user who does look at the console will see:

2023-11-10 15:36:28 DEBUG connectionpool:_new_conn:L1053 Starting new HTTPS connection (1): zenodo.org:443
2023-11-10 15:37:22 WARNING _qt:message_handler:L235 Retrying to obtain clipboard.
2023-11-10 15:37:31 ERROR queue:run:L82 Background task exception:
Traceback (most recent call last):
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\urllib3\connection.py", line 203, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\urllib3\util\connection.py", line 85, in create_connection
    raise err
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\urllib3\util\connection.py", line 73, in create_connection
    sock.connect(sa)
TimeoutError: [WinError 10060] Ein Verbindungsversuch ist fehlgeschlagen, da die Gegenstelle nach einer bestimmten Zeitspanne nicht richtig reagiert hat, oder die hergestellte Verbindung war fehlerhaft, da der verbundene Host nicht reagiert hat

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\urllib3\connectionpool.py", line 791, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\urllib3\connectionpool.py", line 492, in _make_request
    raise new_e
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\urllib3\connectionpool.py", line 468, in _make_request
    self._validate_conn(conn)
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\urllib3\connectionpool.py", line 1097, in _validate_conn
    conn.connect()
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\urllib3\connection.py", line 611, in connect
    self.sock = sock = self._new_conn()
                       ^^^^^^^^^^^^^^^^
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\urllib3\connection.py", line 212, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x0000018ED4A7BA50>, 'Connection to zenodo.org timed out. (connect timeout=None)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\requests\adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\urllib3\connectionpool.py", line 845, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\urllib3\util\retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='zenodo.org', port=443): Max retries exceeded with url: /record/1288441/files/pyspectral_atm_correction_luts_no_aerosol.tgz (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x0000018ED4A7BA50>, 'Connection to zenodo.org timed out. (connect timeout=None)'))

Traceback (most recent call last):
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\uwsift\queue.py", line 78, in run
    for status in task:
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\uwsift\__main__.py", line 335, in _bgnd_open_paths
    for progress in self.document.import_files(paths, **importer_kwargs):
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\uwsift\model\document.py", line 268, in import_files
    for dex, (num_prods, info) in enumerate(infos):
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\uwsift\workspace\simple_workspace.py", line 223, in collect_product_metadata_for_paths
    for prod in hauler.merge_products():
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\uwsift\workspace\importer.py", line 553, in merge_products
    self._load_all_datasets()
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\uwsift\workspace\importer.py", line 666, in _load_all_datasets
    self.scn.load(self.dataset_ids, pad_data=False, upper_right_corner="NE", **self.product_filters)
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\satpy\scene.py", line 1445, in load
    self.generate_possible_composites(unload)
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\satpy\scene.py", line 1508, in generate_possible_composites
    keepables = self._generate_composites_from_loaded_datasets()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\satpy\scene.py", line 1527, in _generate_composites_from_loaded_datasets
    return self._generate_composites_nodes_from_loaded_datasets(needed_comp_nodes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\satpy\scene.py", line 1533, in _generate_composites_nodes_from_loaded_datasets
    self._generate_composite(node, keepables)
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\satpy\scene.py", line 1557, in _generate_composite
    prereq_datasets = self._get_prereq_datasets(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\satpy\scene.py", line 1639, in _get_prereq_datasets
    self._generate_composite(prereq_node, keepables)
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\satpy\scene.py", line 1591, in _generate_composite
    composite = compositor(prereq_datasets,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\satpy\modifiers\atmosphere.py", line 105, in __call__
    corrector = Rayleigh(vis.attrs['platform_name'], vis.attrs['sensor'],
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\pyspectral\rayleigh.py", line 173, in __init__
    download_luts(aerosol_type=aerosol_type)
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\pyspectral\utils.py", line 381, in download_luts
    _download_tarball_and_extract(lut_tarball_url, local_tarball_pathname, subdir_path)
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\pyspectral\utils.py", line 399, in _download_tarball_and_extract
    response = requests.get(tarball_url)
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\requests\api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\requests\api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\requests\sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\requests\sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\requests\adapters.py", line 507, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='zenodo.org', port=443): Max retries exceeded with url: /record/1288441/files/pyspectral_atm_correction_luts_no_aerosol.tgz (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x0000018ED4A7BA50>, 'Connection to zenodo.org timed out. (connect timeout=None)'))
2023-11-10 15:37:31 DEBUG queue:_did_complete_task:L190 completed task False, and_then we do this...
Traceback (most recent call last):
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\uwsift\queue.py", line 191, in _did_complete_task
    todo(succeeded)
  File "D:\SIFT_2.0.0b0dev_windows_20231026_150745\SIFT_2.0.0b0\Lib\site-packages\uwsift\__main__.py", line 350, in _bgnd_open_paths_finish
    raise ValueError("no UUIDs provided by background open" " in _bgnd_open_paths_finish")
ValueError: no UUIDs provided by background open in _bgnd_open_paths_finish
2023-11-10 15:39:18 WARNING _qt:message_handler:L235 Retrying to obtain clipboard.

This is using sift 2.0.0b0 on a laptop running Windows 10 Enterprise LTSC. The environment apparently prohibits sift from contacting the web; I've previously tried to reach the web from a Python script, and the (not very helpful) reply from our IT support was "Python is not supported on Windows".

[djhoese]

Error handling is one of SIFT's biggest faults/limitations. It doesn't surprise me that it doesn't handle this well as it doesn't handle most low-level (or high-level?) errors well beyond putting them in the terminal. I think getting this error all the way to the user would be difficult as it is happening inside the Scene.load call in Satpy and even Satpy doesn't present that error to the user, right? I mean, does this fall under the Satpy case of not erroring when a product fails to load, but instead providing the user with whatever it could? Perhaps this is a use case for a feature you (and I?) talked about in the past of letting the satpy.config have an option to control how failed product loading is handled (exception raised or silent).

[gerritholl]

True, it'd be hard for sift to salvage anything unless it does the .load(...) calls one-by-one. I haven't tried this from Satpy directly; IT doesn't support us using our Windows notebooks for anything programming-related, saying that's what the Linux workstations/servers are for.