parsl vs typeguard 4 doesn't work

[parantapa]

Describe the bug
multiprocessing.Queue is used as a generic type in "monitoring/monitoring.py"; example case; see [1]
multiprocessing.Queue is not a "type" (at-least in Python 3.11.7)
Quoting pydoc:
"multiprocessing.Queue = Queue(maxsize=0) method of multiprocessing.context.DefaultContext instance
Returns a queue object"
This is causing code that uses MonitoringHub to fail.

[1]: See:

parsl/parsl/monitoring/monitoring.py

Line 171 in 479cf9d

comm_q: Queue[Union[Tuple[int, int], str]]

To Reproduce

1. Setup Parsl 2024.1.8 with Python 3.11.7
2. Run the following test script:

> cat test.py                                                                                                                                                                                                                                      (episim37)
import parsl
from parsl.monitoring.monitoring import MonitoringHub
from parsl.config import Config
from parsl.executors import HighThroughputExecutor
from parsl.addresses import address_by_hostname

config = Config(
   executors=[
       HighThroughputExecutor(
           label="local_htex",
           cores_per_worker=1,
           max_workers=4,
           address=address_by_hostname(),
       )
   ],
   monitoring=MonitoringHub(
       hub_address=address_by_hostname(),
       hub_port=55055,
       monitoring_debug=False,
       resource_monitoring_interval=10,
   ),
   strategy='none'
)

parsl.load(config)

3. Run script

python test.py

4. See error

Traceback (most recent call last):
  File "/home/parantapa/test.py", line 25, in <module>
    parsl.load(config)
  File "/home/parantapa/miniconda3/envs/episim37/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 1428, in load
    cls._dfk = DataFlowKernel(config)
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/parantapa/miniconda3/envs/episim37/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 116, in __init__
    self.hub_interchange_port = self.monitoring.start(self.run_id, self.run_dir)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/parantapa/miniconda3/envs/episim37/lib/python3.11/site-packages/parsl/monitoring/monitoring.py", line 171, in start
    comm_q: Queue[Union[Tuple[int, int], str]]
            ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'method' object is not subscriptable

Expected behavior
Code to not crash

Environment

• OS: ArchLinux
• Python version: 3.11.7
• Parsl version: 2024.1.8

Distributed Environment

• Where are you running the Parsl script from ? [e.g. Laptop/Workstation, Login node, Compute node]
  Laptop

• Where do you need the workers to run ? [e.g. Same as Parsl script, Compute nodes, Cloud nodes]
  Laptop

[benclifford]

can you look which version of typeguard you have installed?

[parantapa]

pip freeze | grep typeguard
typeguard==4.1.5

[benclifford]

ok, that shouldn't be happening - because parsl install requirements should be constrained by pip to this (in requirements.txt): typeguard>=2.10,<3

[parantapa]

I see. I have another package installed that needs a more recent version of typeguard. I guess that upgraded the typeguard requirement.

However, is there a chance that this can be fixed (instead of constraining typeguard?)

[benclifford]

it's been in the works for a while - there are some API changes that are awkward if maintaining support for typeguard 2, which I'd like to also keep supporting for a while

[benclifford]

crossref #2812 which fixes v4 but breaks v2

[benclifford]

@parantapa you might try Parsl now that #3044 has been merged

[parantapa]

@benclifford thanks for the heads up! this is really helpful.