Idea: Async context manager

[scasagrande]

So one idea that I've had is to implement some sort of async communication support. The motivation for this is when you have multiple commands that you would like to execute, but you don't care for the order in which things are performed. While far from an exhaustive list, here are two situations off the top of my head I can think of to illustrate my point:

• During initial experiment setup, multiple instruments might require some setup parameters (eg, output settings, limiters, initial positions, trigger priming, etc). These can all be done async because their settings don't rely on the other instruments having completed their setup.
• At some point during an experiment, you want to take a reading from multiple instruments. Typically after asking an instrument for a reading, there is a delay while that acquisition takes place. Some instruments this is quick (perhaps a measurement is already a memory buffer, and you are simply retrieving it), but some instruments this can have a noticeable delay. Either way, you'd like to start asking multiple instruments at the same time to start doing whatever it is they need to do in order to respond.

My initial thoughts for this is to have some sort of context manager which can toggle this async logic on. This way its clear exactly which calls will be attempted to be performed in an async manner, while then reverting to a default synchronous behaviour after exiting the CM.

Using pyserial serial ports as an example, we can hook serial port monitoring to know when we can read by using serial.Serial.fileno() and asyncio.get_event_loop().add_reader() (see http://stackoverflow.com/questions/31793812/using-asyncio-to-read-the-output-of-a-serial-port for this idea).

Then, to the end user, the code might look something like this:

import instruments as ik
inst1 = ik.generic_scpi.SCPIMultimeter.open_serial('/dev/ttyUSB0', baud=9800)
inst2 = ik.generic_scpi.SCPIMultimeter.open_serial('/dev/ttyUSB1', baud=9800)

with ik.nonblocking:
    value1 = inst1.measure()
    value2 = inst2.measure()

print(value1)
print(value2)

So I would imagine it blocking leaving the context manager until all async tasks have resolved.

[cgranade]

I like this idea. A lot. That said, it may be very challenging to make the generalization to asyncio in a way that preserves support for Python < 3.5, since the most natural way to do so would probably be with the new async/await keywords in 3.5. At this point, as radical as it is, going Py3 only may even be a good direction, since 3 compatibility has improved so very much over the past few years.

[scasagrande]

My plan regarding <py3.5 is just not to support this feature. I'd just revert to synchronous behavour and raise a warning.

[bilderbuchi]

I have to say I like how Lantz is doing asynchronous/concurrent access, using the concurrent futures mechanism, and also context managers (although the latter to ensure that initialisation and shutdown are handled cleanly):
https://lantz.readthedocs.io/en/0.3/overview.html?highlight=concurrent#effortless-asynchronous-get-and-set
https://lantz.readthedocs.io/en/0.3/overview.html?highlight=concurrent#context-manager

>>> result1 = fungen.update_async({'ac_mode': True, 'amplitude': Q(42, 'V')})
>>> result2 = another_fungen.update_async({'ac_mode': True, 'amplitude': Q(42, 'V')})
>>> while not result1.done() and not result2.done()
...     DoSomething()

This enables nice things like initializing multiple instruments concurrently (https://lantz.readthedocs.io/en/0.3/guides/initializing-setup.html), even including dependencies between instruments and GUI updates.

[scasagrande]

The context manager for startup/shutdown is straight forward, that should be relatively easy.

Regarding concurrency, I'm glad there is another example I can lean on. I will admit, my asyncio knowledge is on the weaker side, but this could prove to be a good way for me to build on that.

Lucky for us though, I'm going to be dropping Py34 support shortly in time with official EOL, so all supported Py3 versions can make use of asyncio!

[bilderbuchi]

From my small amount of knowledge about async stuff my gut feeling was to use concurrent futures instead of asyncio/async await/etc to do multi instrument access. I was positively surprised when I found out that Lantz used the same approach :-)

[scasagrande]

You could very well be correct! I haven't thought much about it yet.

[scasagrande]

🤔 looks like asyncio.Future is very similar to concurrent.future.Future (noting this for future me to remember)

https://docs.python.org/3/library/asyncio-future.html
https://docs.python.org/3.6/library/asyncio-task.html#future

[scasagrande]

More future (hehe) reminders for me:

https://stackoverflow.com/questions/43882301/why-is-asyncio-future-incompatible-with-concurrent-futures-future

[bilderbuchi]

Huh, interesting. In my admittedly limited experience, nearly every time someone discussed asyncio/awaiting online, the outcome seems to be that the asynchronous event loop "infects" your code, in that it's hard to cross the boundary between asynchronous and non-async code, so you end up having to use the asynchronous event loop nearly everywhere. If having mainly some i/o blocking parts in your code, concurrent futures seemed to be preferable. I'll have to read up in my copy of Fluent Python again to offer more substantial arguments.

@hgrecco can you share some light on why Lantz uses concurrent futures?

[RossSmyth]

Hi, I am interested in async support. Any update on this development? It seems that pyvisa has a pull request open which would make this easier. pyvisa/pyvisa#403

[scasagrande]

Hi @RossSmyth , nothing yet here on this. Our priority over the last few months has been a bunch of other large efforts (such as moving to pint, and removing numpy as a dep).

We're happy to start hearing any requests or suggestions for this feature as that will help us plan.