Add Layer trait and Layer factory methods on Envs, Aborts, Tries, and Options

[johnhungerford]

Introduces new type Layer[Sin, Sout], representing a handler of Sout that requires further handling of Sin and that cannot transform the output of the handled effect. The latter restriction allows for commutative composition, so that layers can be combined in any order without affecting (as far as I can tell) the resulting effect.

Layer provides a mechanism for dependency injection, by allowing you to compose Envs handlers. It should be possible to add a function similar to ZIO's .provide, which automatically wires layers together to satisfy all dependencies.

However, Layer can be used more broadly as a means to simplify certain kinds of effect handling. For instance, it can be used to between different kinds of failures.

[johnhungerford]

Addresses issue #153

[fwbrasil]

It was a bit counterintuitive to me that the effects that the layer can handle is "out" and effects it can produce is "in". I image that's thinking of the operations in the pending effect set. What do you think about flipping the meaning of the type parameters? I think it's more intuitive to think of a Layer like an effect-level arrow that takes effects to handle and produces new effects if needed.

[johnhungerford]

Yeah I agree. How about Layer[Handled, S] or something along those lines, where Handled are the effects that are handled and S are the new unhandled effects?

[fwbrasil]

Maybe Input/Output to make it more explicit?

[johnhungerford]

It's now Layer[In, Out], where In is the intersection of all effects the layer will handle, and Out is the intersection of the effects that will remain to be handled.

[fwbrasil]

I'd like to avoid symbolic operators in Kyo. Maybe we could use add for ++ and andThen for >>>?

[johnhungerford]

Changed ++ to add and >>> to chain. I'm avoiding andThen for now, since that could easily mean what is currently add (since add just handles applies the first layer and then the second).

Whether to keep add as add and not change it to andThen will depend on whether it is effectively commutative so long as it does not change the type of T when it handles an effect.

[fwbrasil]

Would it be possible to express the type-level transformations in the method signature itself instead of relying on the implicit mechanism? I imagine it's not the case, just to double check.

[johnhungerford]

I'm not sure. Basically some mechanism is needed to extract the intersection between the effects that one layer handles and the effects that another layer needs to be handled, and keep track of the remaining effects of each. I'm pretty sure I can least simplify and clarify ApplyLayer, but I can't think of a way to produce the same functionality with just type bounds. I'll give it some more thought though.

[fwbrasil]

sounds good! the current version is ok with me btw

[johnhungerford]

I wasn't able to figure out an alternative, but I did improve the type class used to chain layers together. Now it's called ChainLayer, and it's easier to follow because it reverses the ZIO approach. We can now think of it in this way: the output effects of the first layer are the input effects of the second layer. The main function of ChainLayer, then, is to derive which outputs effects from the first layer remain unhandled.

[fwbrasil]

I had forgotten how joyful OSS can be! It's great how well the layers compose 🤯

Regarding effect handling ordering, the result of the computations can change depending on the order but it seems reasonable to assume that ordering is a decision users can make when building layers. We might need better names for the operators, though. ++ can give an impression that ordering isn't important.

I've been working on a similar mechanism but for another purpose. Users currently need to handle other pending effects before forking a fiber. Since Kyo supports arbitrary effects, it's not easy to "propagate" the effect to the forked computation. For this use case, I need more features than only handling the effect so I created a new Joins mechanism that provides three operations:

1. save: Saves any state necessary to handle the effect in the forked computation. For example, Envs would save the environment value at caller to provide it while executing the fiber.
2. handle: Handles the effect given the state.
3. resume: After the fiber returns, this method can suspend the effect again.

The approach works well for methods that don't return a Fiber instance like Fibers.parallel:

    def parallel[T, S](j: Joins[S])(l: Seq[T > (S with Fibers)])(
        implicit f: Flat[T > Fibers]
    ): Seq[T] > (S with Fibers) =
      j.save.map { st =>
        j.handle(st, l).map(parallel(_)).map(j.resume)
      }

But it doesn't work well for Fibers.init. It's not possible to use resume in this case because the M is inside a Fiber value. For now, I'm just returning the Fiber[M[T]] but I don't think that provides good usability.

    def init[T, S](j: Joins[S])( /*inline*/ v: => T > (S with Fibers))(implicit
        f: Flat[T > (S with Fibers)]
    ): Fiber[j.M[T]] > (S with IOs) =
      j.save.map { st =>
        init(j.handle(st, v))
      }

I wonder if we could try to unify the joins and layers. The joins impl is in this branch: https://github.com/getkyo/kyo/blob/joins

[fwbrasil]

Regarding the build failure, the issue is compilation in Scala 2. The macro for Flat is in the same module as the core code so it can't be invoked given that Scala 2 can't use macros from the same compilation unit. I've been working this around by importing Flat.unsafe. I've created #155 to move the macros to a new module, it's not trivial but seems possible.

[johnhungerford]

Re: the build failure, when I run sbt +kyo-core/test or sbt +kyo-core/compile locally it works. Is there another command I should be running to reproduce the failure?

[johnhungerford]

I suspect that Layer could be a special case of Join, but we would have to add another type parameter (or inner type) Unhandled, so that

  def handle[T, S](
        s: State,
        v: T > (E with S)
    )(implicit
        f: Flat[T > (E with S)]
    ): M[T] > (S with IOs)

would become

  def handle[T, S](
        s: State,
        v: T > (E with S)
    )(implicit
        f: Flat[T > (E with S)]
    ): M[T] > (S with Unhandled)

In the case of using it for joins, Unhandled would be IOs, but it could be other effects for other use cases.

If we went this route, what you are calling Join and what I'm calling Layer would be special cases of some more general type, let's say HandleArrow:

  trait HandleArrow[E, Unhandled] { self =>

    type M[_]
    type State

    def save: State > E

    def handle[T, S](
        s: State,
        v: T > (E with S)
    )(implicit
        f: Flat[T > (E with S)]
    ): M[T] > (S with Unhandled)

    // Not sure about this one...
    def resume[T, S](v: M[T] > S): T > (E with S)

In this case I think Layer and Join would be:

  type Layer[E, Unhandled] = HandleArrow[E, Unhandled] { type M[X] = Id[X]; type State = Unit }
  type Join[E] = HandleArrow[E, IOs]

For the use cases I have in mind, it is important that Layer have a commutative add method. If there is going to be a mechanism to automatically compose layers, it cannot matter what order they are in (this is not going to be true of andThen, of course). This is one of the reasons layer's run method cannot change T (and why I have type M[X] = Id[X] above). At least in the case of Aborts and Envs, I think that if M = Id, then layer addition would be commutative. Are there edge cases I'm not thinking of though?

[fwbrasil]

Re: the build failure, when I run sbt +kyo-core/test or sbt +kyo-core/compile locally it works. Is there another command I should be running to reproduce the failure?

I get the same error as the CI when I run sbt +test. I think it could be the state of your local compilation, maybe the macro class files are already generated so the compiler can expand it. Try sbt +clean first.

[johnhungerford]

I think the build error is resolved: I had not defined the implicit Flat parameter for Layer.run correctly.

[fwbrasil]

Could you rename to andThen? I think it'd be better to communicate that order is important.

[johnhungerford]

Done. Agreed that this is clearer.

[fwbrasil]

In Scala 2, methods that remove items from the pending effects need to have the effect to be removed as the first in the type intersection otherwise the compiler isn't able to infer types correctly. You might be able to avoid type annotations if you use effect: T > (In with S).

[johnhungerford]

I pushed this change, although it looks like I still need the type annotations in those places where I've called Layer#run.

[fwbrasil]

I see. It could be that the compiler isn't able to infer the removal based on the generic type parameter.

[fwbrasil]

Other than the two last minor comments, the change LGTM! Reasoning about Kyo's effect handling and how the layers are composed, I can't see potential issues. I normally try to design effects in Kyo by breaking up problems into smaller reusable pieces that handle specific behaviors. The use case for Layers seems different enough from Joins for them to be separate implementations. We can revisit this later as well.

Regarding ordering, I think if we have layers only for Envs, Aborts, and Tries it isn't relevant because the short-circuiting uses pre-defined fallbacks in the layers instead of M[_] values but this could not be true for other effects. Maybe we could restrict the API by providing methods like Layers.envs[T](...) instead of Envs[T].layer and then making Layer sealed? With this change, we could keep add as well.

Thanks for the contribution! This will be the first meta-effect in Kyo, it's nice to see how it generalized from ZIO's version :)

[johnhungerford]

I have renamed add to andThen so that layers are not too constrained. If/when functionality is added to automatically compose layers, we can add restrictions there to accept only layers that can be combined in any order without changing behavior.

[johnhungerford]

Ok I think it's good to go. This was fun! (This is my first significant OSS contribution, by the way. Prior to this, I've only made one trivial change to zio-http).

[fwbrasil]

Nice!! You are definitely off to a good start 🙏