Fix delay/hang when resolving deep dictionary dependencies

[benclifford]

Description

Prior to this PR, the dict part of DEEP_DEPENDENCY_RESOLVER did not extract all futures that it should in the gather stage, meaning that sometimes the unwrap stage would happen too early. Prior to this PR, only keys and values of a dict that were directly Future objects would be waited for by the DFK, rather than deep-recursing into those key/value objects. This would result in Future.result() happening too early (and blocking, potentially forever).

This PR switches out that behaviour to call deep_traverse_to_gather recursively, instead of using an if.

This PR adds a test case which demonstrates this change. That test case will hang without the above change, but pass with this change.

This PR adds asserts to the code that unwraps Future objects, to assert that the future object is in a suitable state for immediate unwrapping (i.e. that it is done()). This changes the above bug from a hang into a dependency failure that is reported to the user, and because it happens in any unwrapping of Future objects, is probably useful in the future to anyone modifying on the deep dependency resolver.

Type of change

• Bug fix

[khk-globus]

Subtle.