Add streamline resource framework

[DavidLegg]

• Tickets addressed: N/A
• Review: By file
• Merge strategy: Merge (no squash)

Description

Implements a new resource framework, based on the one currently being used by Clipper and lessons learned from an older framework developed for Clipper.

The most important changes, compared to previous resource frameworks, are:

• Pull-based derived resources
  • PublishingResource and its related resources use a push-based model for derived resources. This required resolving derived resource values eagerly with an expensive looping task.
  • Clipper's current framework replaced that with a lazy pull-based model, which proved much more efficient. This framework adopts the same pull-based approach.
• Abstraction strategy
  • A resource's sampling strategy is "upstream" of the resource itself, and shouldn't affect how that resource can be used - hence it should not be part of the resource type. Instead of "DerivedResource" and "SamplingResource" and "SemiLazilyDerivedState", just have "DiscreteResource" and three ways of constructing one on Clipper's current model. This idea is carried forward, but taken further:
  • Instead of one resource type per dynamics type, e.g. DiscreteResource, NumericResource, etc., we have just one resource interface parameterized by dynamics type. This cleans up derived resources, since we only need Resource -> Resource derivation, not Discrete -> Discrete and Discrete -> Numeric and Numeric -> Discrete and ...
    Avoiding that combinatorial explosion makes adding new dynamics types easier. This in turn gives us better modeling fidelity, since we can define and use an unusual dynamics type if needed. For example, we define and regularly use in derivations an exact clock based on Duration.
• Improved expiry and dynamics comparison semantics
  • Dynamics are compared by equals, not DynamicsIdentity (as in Clipper model presently), which simplifies dynamics types.
  • Expiry info is carried in a wrapper around dynamics, rather than in dynamics themselves. This again simplifies dynamics types.
• Built-in error catching for effects and derived resources
  • Dynamics are wrapped in ErrorCatching, a type that can replace the normal value with an exception.
  • When applying effects or performing a derivation (through monad machinery, see below), exceptions are caught and given to the ErrorCatching layer. Errors cascade through derivations, so we don't propagate "fake" data when errors occur.
  • Allows simulation to continue when some resources "fail" - Those resources have no value (so don't have a wrong value), while unaffected resources continue to work normally.
  • When registering a resource, errors are shunted to a separate errors resource with additional debugging information attached. Hence errors and partial sim results are visible in the UI.
• Use of monads to lift operations
  • Operations can be written against the simplest type they logically work on, then lifted through monadic operators to act on resources. This keeps the handling for expiry, errors, and resources consistent and centralized. This is discussed in more detail here:
    streamline-framework.pdf
• Streamlined cell and effect handling
  • On Clipper, even with the option to define custom cells or effect types, modelers preferred to just use SettableState/Register and "set" effects. This leads to unnecessary conflicts for concurrent effects. Although effects "should" commute, because (for example) they represent increment and decrement operations, they don't actually commute because they're implemented through set effects.
  • In this framework, there's one kind of resource that permits effects, CellResource. Additionally, effects are functions on the dynamics; we don't define different effect types for different cells. Finally, we define an "automatic" effect trait, which combines concurrent effects by trying both orderings to see if the effects commute in practice.
  • In practice, we observe very few concurrent effects, so the penalty to performance from using the automatic effect trait should be negligible. The benefit is that modelers can write arbitrary effects as easily as they could write the read-compute-set style:

  var v = resource.getDynamics();
  var u = someFunction(v);
  resource.set(u);
  

  becomes

  resource.emit(v -> someFunction(v));
  

• "Reactions" implemented by efficiently trampolining with replaying tasks
  • On Clipper, we define a number of daemon tasks that "react" to some condition repeatedly throughout the simulation, performing a very small amount of work each time. For example, computing an integral every time an integrand changes, or processing a command when one is added to a queue.
  • Using a loop and threaded task is inefficient; time spent thread switching dwarfs time spent doing useful work. Clipper solved this using replaying tasks to avoid thread switching, and spawning a new task each iteration instead of looping to avoid long replays.
  • This technique is carried forward in the Reactions class, which defines several useful reaction patterns.
• Equivalence of Conditions and discrete boolean resources
  • Since resources carry expiry information, a Condition is equivalent to a profile segment on a discrete boolean resource. To realize this equivalence and halve the amount of work for methods like lessThan, we have a function when to turn a boolean resource into the condition "that resource is true". This is especially useful in conjunction with reactions.
• Model-level debugging utilities
  • Since stack traces and procedural debugger stepping are not very useful for debugging things like derived resources, the "Tracing" class offers a way to trace when a resource is invoked, and what it returns. Tracing stacks, so a chain of traced derived resources appear as such in the output, e.g. A -> B -> C -> result.
  • Additional context is attached to each effect, meant to label it with the activity that emitted it and the resource it was emitted on. Effects usually fail when applied, removed in time from the blameworthy activity. Carrying this info with the effect lets us determine its cause more quickly.
    • Note that this feature is mostly a stop-gap until topic metadata is well-supported. At that point, most of the labelling technology in this framework will likely be replaced by topic metadata.
• Flexible unit awareness framework
  • Clipper used a specially-designed QuantityState to efficiently handle units. This framework generalizes that to a UnitAware wrapper interface, which can be wrapped around any type, provided an appropriate scaling function to do the unit conversions with. This unifies unit handling, since as a fallback, we can always take unit-naive operations, and apply them to unit-aware objects by requesting the unit-naive values in the unit implicitly expected by the unit-naive operation. This way, units are controlled at the point of use, instead of the point of definition, and different uses can transparently ask for different units.
  • Some operations, like basic arithmetic, are generalized across types to be done in a truly unit-aware way. See the unit-aware overloads of methods in PolynomialResources for examples.

Verification

Some core classes were unit-tested as needed for debugging. Additionally, Pranav on SRL has been using a preview version of this framework for a few weeks to help flush out bugs and missing features.

Documentation

streamline-framework.pdf
The intended audience for this document is a modeler looking to use the framework. I'd like to convert this document into a set of wiki pages so they're more easily accessible, and expand it with examples and lessons learned by modelers using this or other resource frameworks.

Future work

Fleshing out standard dynamics types, effects, and resource derivation functions as more models use them and find use cases that are missing or poorly supported.

There are known efficiency improvements around task handling and providing expiry information directly to Aerie when sampling resources. These may have small knock-on effects to this framework.

Also, as mentioned briefly above, Aerie's topic metadata feature will likely supplant the labelling system in this framework in the future.

(PR copied from #1198, to get CI tests to work.)

[bradNASA]

If we don't resolve the infinite loop problem, we should create an issue for it, #1305 . I'm approving because I don't think this is a big enough reason to hold it back.

[mattdailis]

Thanks for all your hard work, @DavidLegg. This PR will empower mission modelers to more easily harness the power of Aerie. The terse expressiveness of your libraries, and the fact that I've had a number of people over the past few weeks ask me repeatedly, "when are you going to merge this PR?" are a testament to how much care you've put into the design and implementation of this framework.

This is the beginning, not the end; I expect us to continue to iterate on this framework. Some specific areas I'd like to see us explore are:

• Can we avoid the singleton pattern in Resources.CLOCK?
• Can Resource.signalling leverage expiry, rather than emitting no-op events?
• Can we serialize errors using events instead of resources, for better visualization?

With that, I say