res is used by other packages, yet there is no clear public/private API separation

[eivindjahren]

res/ does not mark any sub-modules as private, only some class members uses the underscore prefix to indicate it being private. Still, some other packages (subscript to name one) do use res and we would like that usage to remain. However, there is a need to consider parts of res as private in order to be able to make necessary changes.

I think there are several features on the horizon, such as the optimization "mode", which would limit the need to use res as a library. However, for the time being, there should be some line delimiting what we consider public and private.

Proposed solution

move what should be considered private (likely most of res/) into a package clearly marked as internal, e.g. _ert_internal to discurage its usage. Likely, what should remain in res/ will be only whatever is used in the wild and we condone the usage of.

[pinkwah]

Is this stuff that will be used by any of the ert packages? I think it might be difficult to delineate between usage within res, ert* and the outside world.

[eivindjahren]

By private, I mean private to the ert package so ert* would freely use anything in _ert_internal. So if we make changes to _ert_internal, we only have to move ert with it.

[eivindjahren]

Another option is to move some things from res/ to ert/. That is unfortunately not entirely trivial and somewhat messy when it comes to modules shared by ert/ and ert3/.

[eivindjahren]

There is some overlap in concepts here with ert_shared/libres_fascade.py.

[markusdregi]

I think this discussion is somewhat optimistic in how complex separating the public and private functions of res actually is.

First, one have the obvious consumption from ERT. But, at this point the fact that res and the remaining ERT implementation are in separate packages is more of a technical detail than anything else. From an ERT perspective we should feel free to move around functionality between the packages as we see fit with respect to responsibilities and structure.

The second software consuming res is Everest and that consumption is very contained; but in a fashion that is not transparent by only inspecting the ERT repo. This is somewhat cumbersome, but perhaps we could start by making all functions consumed by Everest explicitly public in some fashion. Perhaps writing documentation for them?

Then we have all the centrally (for various notions of centrally) managed workflows; that is Python workflows in semeio and subscript. The interesting part here is that they all get EnKFMain passed as an argument; which implies that everything that can be reached via the objects reachable from the Python object EnKFMain is public API. But, the good news is that we can can inspect the workflows and verify which parts of the API is being used and by this information be able to make parts of the API private.

However, say hello to my little can of worms 🪱 There is nothing special about the workflows in subscript and semeio besides that they are inspectable, changeable and under control. There are also workflows managed by the various assets that utilises EnKFMain and deducing up-front which parts they are using is very difficult. Furthermore, we don't really have a good alternative for the problems they are solving and discouraging a technical solution to a valid problem without having an alternative technical solution to provide does not make much sense to me...

I would suggest two epic tasks to get started on this:

• start logging user defined workflows and see how much interesting information with respect to usage we can deduce and
• implement a first version of a data centric workflow concept that can be introduced in both ERT2 and ERT3 where the implementation in ERT2 builds upon the current workflow concept, but abstracts heavily on top of it.

The first is more of a fold up your sleeves and get started kind of epic. While the later requires some serious thinking, experimentation, user feedback, implementation and follow up. And we will probably not be able to act upon it immediately as we should most likely align the run model and the engine of ERT2 and ERT3 before this is done to not increase implementation complexity even further. The good news is that that consolidation work is already on the milestone agenda :)

[eivindjahren]

Then we have all the centrally (for various notions of centrally) managed workflows; that is Python workflows in semeio and subscript. The interesting part here is that they all get EnKFMain passed as an argument; which implies that everything that can be reached via the objects reachable from the Python object EnKFMain is public API. But, the good news is that we can can inspect the workflows and verify which parts of the API is being used and by this information be able to make parts of the API private.

However, say hello to my little can of worms worm There is nothing special about the workflows in subscript and semeio besides that they are inspectable, changeable and under control. There are also workflows managed by the various assets that utilises EnKFMain and deducing up-front which parts they are using is very difficult. Furthermore, we don't really have a good alternative for the problems they are solving and discouraging a technical solution to a valid problem without having an alternative technical solution to provide does not make much sense to me...

I think because we are currently changing the behavior of from res.enkf import * these workflows can break with every ert version, so I don't see how the status quo benefits workflows not "centrally" managed (or semeio for that manner). What we could do, is expose a facade for EnkFMain which we strive to not change the behavior of, or at least give reasonble notice of changes. That object should probably replace/reuse ert_shared/libres_fascade.py, and could be contained in such a manner that workflows do not need to import anything from res (ie. instead of needing to run from res.enkf import ESUpdate, ErtRunContext; ESUpdate(ert).smootherUpdate(ErtRunContext(...)) you can run ert.es_smoother_update(run_context, ...).
Doing so will cause breakage, but perhaps in the long term, less breakage than the current scheme. Also the breakage will be more predictable an thought through, we could even create a migration guide.

The second software consuming res is Everest and that consumption is very contained; but in a fashion that is not transparent by only inspecting the ERT repo. This is somewhat cumbersome, but perhaps we could start by making all functions consumed by Everest explicitly public in some fashion. Perhaps writing documentation for them?

For Everest, we could make functions consumed by Everest explicitly public by the above proposed solution: What everest consume is re-exported in res/ from _ert_internal (what before was in res/ is moved to _ert_internal).

[markusdregi]

I agree to all of the things you are saying :) But for the "external" workflows the first step would be to first create a facade version of workflows...

This sounds like a good workflow milestone though :)