From 8800e06cab7dab3f77499b1b961011638cd3e69b Mon Sep 17 00:00:00 2001
From: Arnaud Bergeron <abergeron@gmail.com>
Date: Fri, 8 Jul 2022 14:47:00 -0400
Subject: [PATCH 1/5] Add documentation for client/runner.py

---
 docs/src/code/client.rst        | 1 +
 docs/src/code/client/runner.rst | 5 +++++
 2 files changed, 6 insertions(+)
 create mode 100644 docs/src/code/client/runner.rst

diff --git a/docs/src/code/client.rst b/docs/src/code/client.rst
index 82dc287ab..c77b2a20b 100644
--- a/docs/src/code/client.rst
+++ b/docs/src/code/client.rst
@@ -9,6 +9,7 @@ Client helper functions
    client/cli
    client/experiment
    client/manual
+   client/runner
 
 .. automodule:: orion.client
    :members:
diff --git a/docs/src/code/client/runner.rst b/docs/src/code/client/runner.rst
new file mode 100644
index 000000000..0de0555a8
--- /dev/null
+++ b/docs/src/code/client/runner.rst
@@ -0,0 +1,5 @@
+Runner client
+=============
+
+.. automodule:: orion.client.runner
+   :members:

From 598ab32dd0b7483b91498de0d395c40ebd92d170 Mon Sep 17 00:00:00 2001
From: Arnaud Bergeron <abergeron@gmail.com>
Date: Fri, 8 Jul 2022 14:53:34 -0400
Subject: [PATCH 2/5] Add a plan section to the developer documentation

---
 docs/src/developer/overview.rst |  1 +
 docs/src/developer/plan.rst     | 97 +++++++++++++++++++++++++++++++++
 docs/src/index.rst              |  1 +
 3 files changed, 99 insertions(+)
 create mode 100644 docs/src/developer/plan.rst

diff --git a/docs/src/developer/overview.rst b/docs/src/developer/overview.rst
index 26110cd28..4d4b683a4 100644
--- a/docs/src/developer/overview.rst
+++ b/docs/src/developer/overview.rst
@@ -9,6 +9,7 @@ The documentation for developers is organized in the following, easy to read, se
 
 * :doc:`Getting Started <installing>`. Installing the development environment
 * :doc:`Conventions <standards>`. Get familiar with the project's standards and guidelines
+* :doc:`Source code map <plan>`. Get familiar with the interactions in the code.
 * :doc:`Testing <testing>`. Implementing your changes and how to test your code
 * :doc:`Documenting <documenting>`. Documenting your changes and updating the documentation
 * :doc:`Continuous Integration <ci>`. Get familiar with our continuous integration setup
diff --git a/docs/src/developer/plan.rst b/docs/src/developer/plan.rst
new file mode 100644
index 000000000..191700a82
--- /dev/null
+++ b/docs/src/developer/plan.rst
@@ -0,0 +1,97 @@
+***************
+Source code map
+***************
+
+This document will walk the path of an orion experiment through the
+code. Not every detail is explained, but there are ample links to the
+classes and methods involved if you want to dig further in a certain
+section.
+
+Departure
+---------
+
+You start an experience by running ``orion hunt <script> <params>``.
+
+The code in :py:func:`orion.core.cli.main` will parse the command line
+arguments and route to :py:func:`orion.core.cli.hunt.main`.
+
+The command line arguments are passed to
+:py:func:`orion.core.io.experiment_builder.build_from_args`. This will
+massage the parsed command line arguments and merge that configuration
+with the config file and the defaults with various helpers from
+:py:mod:`orion.core.io.resolve_config` to build the final
+configuration. The result is eventually handled off to
+:py:func:`orion.core.io.experiment_builder.create_experiment` to
+create an :py:class:`orion.core.worker.experiment.Experiment` and set
+its properties.
+
+The created experiments finds its way back to
+:py:func:`orion.core.cli.hunt.main` and is handed off to
+:py:func:`orion.core.cli.hunt.workon` along with some more
+configuration for the workers.
+
+This method will setup a few more objects to manage the optimization
+process: a :py:class:`orion.core.worker.consumer.Consumer` to act as
+the bridge to the user script and an
+:py:class:`orion.client.experiment.ExperimentClient` to coordinate
+everything and calls
+:py:meth:`orion.client.experiment.ExperimentClient.workon` which
+mostly creates a :py:class:`orion.client.runner.Runner` and calls its
+:py:meth:`orion.client.runner.Runner.run` method.
+
+
+The Run Loop
+------------
+
+We are finally in the main run loop. It is composed of three main
+phases that repeat.
+
+
+First phase
+~~~~~~~~~~~
+
+In the first phase we call
+:py:meth:`orion.client.runner.Runner.sample`. This will check if new
+trials are required using
+:py:meth:`orion.client.runner.Runner.should_sample` and request those
+trials using :py:meth:`orion.client.experiment.ExperimentClient.suggest`.
+
+This will first check if any trials are available in the storage using
+:py:meth:`orion.core.worker.experiment.Experiment.reserve_trial`.
+
+If none are available, it will produce new trials using
+:py:meth:`orion.core.worker.producer.Producer.produce()` which loads
+the state of the algorithm from the storage, runs it to suggest new
+:py:class:`orion.core.worker.trial.Trial` and saves both the new
+trials and the new algorithm state to the storage. This is protected
+from concurrent access by other instances of ``orion hunt`` by locking
+the storage for the duration of that operation.
+
+
+The second phase
+~~~~~~~~~~~~~~~~
+
+In the second phase we call
+:py:meth:`orion.client.runner.Runner.scatter` with the trials
+generated in the first phase, if any.
+
+This schedules each trial to be executed using the configured executor
+and registers the futures that the executor returns. Execution is
+handled asynchronously and the futures enable us to keep track of the
+state of the trials.
+
+
+The third phase
+~~~~~~~~~~~~~~~
+
+In the third phase we call
+:py:meth:`orion.client.runner.Runner.gather` which will wait on all
+currently registered futures with a timeout to get some results.
+
+Once we get those results we de-register the futures and record the
+results with
+:py:meth:`orion.client.experiment.ExperimentClient.observe` or update
+the count of broken trials if they did not finish successfully.
+
+Finally we monitor the total amount of time spent waiting for trials
+to finish.
diff --git a/docs/src/index.rst b/docs/src/index.rst
index f126046e8..46bf9b2ec 100644
--- a/docs/src/index.rst
+++ b/docs/src/index.rst
@@ -67,6 +67,7 @@
    developer/overview
    developer/installing
    developer/standards
+   developer/plan
    developer/testing
    developer/documenting
    developer/ci

From 6ede51862063dff6461c316e16cec610ddfdfd8d Mon Sep 17 00:00:00 2001
From: Arnaud Bergeron <abergeron@gmail.com>
Date: Fri, 22 Jul 2022 14:38:08 -0400
Subject: [PATCH 3/5] Add more explanation to some section as per comments

---
 docs/src/developer/plan.rst | 53 ++++++++++++++++++++++++++++++++-----
 1 file changed, 46 insertions(+), 7 deletions(-)

diff --git a/docs/src/developer/plan.rst b/docs/src/developer/plan.rst
index 191700a82..0e3c9998f 100644
--- a/docs/src/developer/plan.rst
+++ b/docs/src/developer/plan.rst
@@ -16,16 +16,27 @@ The code in :py:func:`orion.core.cli.main` will parse the command line
 arguments and route to :py:func:`orion.core.cli.hunt.main`.
 
 The command line arguments are passed to
-:py:func:`orion.core.io.experiment_builder.build_from_args`. This will
-massage the parsed command line arguments and merge that configuration
-with the config file and the defaults with various helpers from
-:py:mod:`orion.core.io.resolve_config` to build the final
-configuration. The result is eventually handled off to
+:py:func:`orion.core.io.experiment_builder.build_from_args`, which
+does some setup and hands over the arguments to
+:py:func:`orion.core.io.experiment_builder.build`. This will hand over
+the configuration to
+:py:func:`orion.core.io.experiment_builder.consolidate_config` which
+will look up the experiment in the configured storage to see if it's
+already there and merge the loaded configuration with the provided one
+with various helpers from :py:mod:`orion.core.io.resolve_config` to
+build the final configuration. The result is eventually handled off to
 :py:func:`orion.core.io.experiment_builder.create_experiment` to
 create an :py:class:`orion.core.worker.experiment.Experiment` and set
 its properties.
 
-The created experiments finds its way back to
+If the experiment is new, meaning it has no storage id, then it will
+attempt to save it to storage, which may conflict in case another
+instance of ``orion hunt`` is doing the same thing. The storage is
+responsible for repoting conflicts and
+:py:func:`orion.core.io.experiment_builder.build` is called again
+recursively in that case to retry the whole operation.
+
+The created experiment finds its way back to
 :py:func:`orion.core.cli.hunt.main` and is handed off to
 :py:func:`orion.core.cli.hunt.workon` along with some more
 configuration for the workers.
@@ -60,7 +71,7 @@ This will first check if any trials are available in the storage using
 :py:meth:`orion.core.worker.experiment.Experiment.reserve_trial`.
 
 If none are available, it will produce new trials using
-:py:meth:`orion.core.worker.producer.Producer.produce()` which loads
+:py:meth:`orion.core.worker.producer.Producer.produce` which loads
 the state of the algorithm from the storage, runs it to suggest new
 :py:class:`orion.core.worker.trial.Trial` and saves both the new
 trials and the new algorithm state to the storage. This is protected
@@ -95,3 +106,31 @@ the count of broken trials if they did not finish successfully.
 
 Finally we monitor the total amount of time spent waiting for trials
 to finish.
+
+
+Stopping criteria
+~~~~~~~~~~~~~~~~~
+
+There are multiple criteria that are monitored to stop the
+experiment.
+
+The first obvious one is the configured maximum number of trials to
+run. If this is reached, then we stop running more. This is checked at
+the beginning of the loop with
+:py:attr:`orion.client.runner.Runner.is_running`.
+
+The experiment can also stop if too many trials fail, either because
+they fail to start, they crashed, were killed (like by an external job
+scheduler) or the take too much time to complete. This is checked in
+:py:meth:`orion.client.runner.Runner.gather` with
+:py:attr:`orion.client.runner.Runner.is_broken`.
+
+If one of the workers returns an unexpected result the experiment is
+also stop immediately because it is assume that something is wrong
+with either the code or the configuration and spending more time
+computing stuff will not fix it. This is also checked for in
+:py:meth:`orion.client.runner.Runner.gather`.
+
+Finally if the loop spends too much time waiting and nothing happens
+the experiment is considered stalled and will also stop. This is
+checked at the end of :py:meth:`orion.client.runner.Runner.run`.

From 934fcb3cacf8a4a141fed7ded598f051e3f7770b Mon Sep 17 00:00:00 2001
From: abergeron <abergeron@gmail.com>
Date: Tue, 2 Aug 2022 14:07:41 -0400
Subject: [PATCH 4/5] Update docs/src/developer/plan.rst

Co-authored-by: Xavier Bouthillier <xavier.bouthillier@gmail.com>
---
 docs/src/developer/plan.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/src/developer/plan.rst b/docs/src/developer/plan.rst
index 0e3c9998f..2928c30a6 100644
--- a/docs/src/developer/plan.rst
+++ b/docs/src/developer/plan.rst
@@ -32,7 +32,7 @@ its properties.
 If the experiment is new, meaning it has no storage id, then it will
 attempt to save it to storage, which may conflict in case another
 instance of ``orion hunt`` is doing the same thing. The storage is
-responsible for repoting conflicts and
+responsible for reporting conflicts and
 :py:func:`orion.core.io.experiment_builder.build` is called again
 recursively in that case to retry the whole operation.
 

From 84b323710fa13300703dd1bc17e7eda3cd6fa287 Mon Sep 17 00:00:00 2001
From: abergeron <abergeron@gmail.com>
Date: Tue, 2 Aug 2022 14:08:08 -0400
Subject: [PATCH 5/5] Update docs/src/developer/plan.rst

Co-authored-by: Xavier Bouthillier <xavier.bouthillier@gmail.com>
---
 docs/src/developer/plan.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/src/developer/plan.rst b/docs/src/developer/plan.rst
index 2928c30a6..f37c1aa13 100644
--- a/docs/src/developer/plan.rst
+++ b/docs/src/developer/plan.rst
@@ -126,7 +126,7 @@ scheduler) or the take too much time to complete. This is checked in
 :py:attr:`orion.client.runner.Runner.is_broken`.
 
 If one of the workers returns an unexpected result the experiment is
-also stop immediately because it is assume that something is wrong
+also stopped immediately because it assumed that something is wrong
 with either the code or the configuration and spending more time
 computing stuff will not fix it. This is also checked for in
 :py:meth:`orion.client.runner.Runner.gather`.