Merge branch 'development'

README.rst

@@ -231,11 +231,17 @@ scripts *(shell)*, and sequence tasks *(sequence)*.
     pfwd = { "shell" = "ssh -N -L 0.0.0.0:8080:$STAGING:8080 $STAGING & ssh -N -L 0.0.0.0:5432:$STAGINGDB:5432 $STAGINGDB &" }
     pfwdstop = { "shell" = "kill $(pgrep -f "ssh -N -L .*:(8080|5432)")" }
 
-  By default poe attempts to find a posix shell (sh, bash, or zsh in that order) on the system and uses that. When running on windows, this might not always be possible. If bash is not found on the path on windows then poe will explicitly look for `Git bash <https://gitforwindows.org>`_ at the usual location.
+  By default poe attempts to find a posix shell (sh, bash, or zsh in that order) on the
+  system and uses that. When running on windows, this might not always be possible. If
+  bash is not found on the path on windows then poe will explicitly look for
+  `Git bash <https://gitforwindows.org>`_ at the usual location.
 
   **Using different types of shell/interpreter**
 
-  It is also possible to specify an alternative interpreter (or list of compatible interpreters ordered by preference) to be invoked to execute shell task content. For example if you only expect the task to be executed on windows or other environments with powershell installed then you can specify a powershell based task like so:
+  It is also possible to specify an alternative interpreter (or list of compatible
+  interpreters ordered by preference) to be invoked to execute shell task content. For
+  example if you only expect the task to be executed on windows or other environments
+  with powershell installed then you can specify a powershell based task like so:
 
   .. code-block:: toml
 
@@ -245,13 +251,17 @@ scripts *(shell)*, and sequence tasks *(sequence)*.
     """
     interpreter = "pwsh"
 
-  If your task content is restricted to syntax that is valid for both posix shells and powershell then you can maximise increase the likelihood of it working on any system by specifying the interpreter as:
+  If your task content is restricted to syntax that is valid for both posix shells and
+  powershell then you can maximise increase the likelihood of it working on any system
+  by specifying the interpreter as:
 
   .. code-block:: toml
 
     interpreter = ["posix", "pwsh"]
 
-  It is also possible to specify python code as the shell task code as in the following example. However it is recommended to use a *script* task rather than writing complex code inline within your pyproject.toml.
+  It is also possible to specify python code as the shell task code as in the following
+  example. However it is recommended to use a *script* task rather than writing complex
+  code inline within your pyproject.toml.
 
   .. code-block:: toml
 
@@ -266,9 +276,11 @@ scripts *(shell)*, and sequence tasks *(sequence)*.
   The following interpreter values may be used:
 
   posix
-      This is the default behavoir, equivalent to ["sh", "bash", "zsh"], meaning that poe will try to find sh, and fallback to bash, then zsh.
+      This is the default behavoir, equivalent to ["sh", "bash", "zsh"], meaning that
+      poe will try to find sh, and fallback to bash, then zsh.
   sh
-      Use the basic posix shell. This is often an alias for bash or dash depending on the operating system.
+      Use the basic posix shell. This is often an alias for bash or dash depending on
+      the operating system.
   bash
       Uses whatever version of bash can be found. This is usually the most portable option.
   zsh
@@ -280,7 +292,8 @@ scripts *(shell)*, and sequence tasks *(sequence)*.
   powershell
       Uses the newest version of powershell that can be found.
 
-  The default value can be changed with the global *shell_interpreter* option as described below.
+  The default value can be changed with the global *shell_interpreter* option as
+  described below.
 
 - **Composite tasks** are defined as a sequence of other tasks as an array.
 
@@ -387,7 +400,7 @@ env key like so:
     serve.env = { PORT = "9001" }
 
 Notice this example uses deep keys which can be more convenient but aren't as well
-supported by some toml implementations.
+supported by some older toml implementations.
 
 The above example can be modified to only set the `PORT` variable if it is not already
 set by replacing the last line with the following:
@@ -411,6 +424,19 @@ You can also specify an env file (with bash-like syntax) to load per task like s
     serve.script  = "myapp:run"
     serve.envfile = ".env"
 
+It it also possible to reference existing env vars when defining a new env var for a
+task. This may be useful for aliasing or extending a variable already defined in the
+host environment, globally in the config, or in a referencd envfile. In the following
+example the value from $TF_VAR_service_port on the host environment is also made
+available as $FLASK_RUN_PORT within the task.
+
+.. code-block:: toml
+
+    [tool.poe.tasks.serve]
+    serve.cmd = "flask run"
+    serve.env   = { FLASK_RUN_PORT = "${TF_VAR_service_port}" }
+
+
 Declaring CLI arguments
 -----------------------
 
@@ -633,9 +659,13 @@ pyproject.toml file by specifying :toml:`tool.poe.env` like so
 
   [tool.poe.env]
   VAR1 = "FOO"
-  VAR2 = "BAR"
+  VAR2 = "BAR BAR BLACK ${FARM_ANIMAL}"
+
+The example above also demonstrates how – as with env vars defined at the task level –
+posix variable interpolation syntax may be used to define global env vars with reference
+to variables already defined in the host environment or in a referenced env file.
 
-As for the task level option, you can indicated that a variable should only be set if
+As with the task level option, you can indicated that a variable should only be set if
 not already set like so:
 
 .. code-block:: toml
@@ -799,7 +829,8 @@ so:
   [tool.poe]
   include = ["modules/acme_common/shared_tasks.toml", "generated_tasks.json"]
 
-Files are loaded in the order specified. If an item already exists then the included value it ignored.
+Files are loaded in the order specified. If an item already exists then the included
+value it ignored.
 
 If a referenced file is missing then poe ignores it without error, though
 failure to read the contents will result in failure.

poethepoet/context.py

@@ -3,34 +3,28 @@
     Any,
     Dict,
     Mapping,
-    MutableMapping,
     Optional,
     Tuple,
-    Union,
     TYPE_CHECKING,
 )
-from .exceptions import ExecutionError
 from .executor import PoeExecutor
-from .envfile import load_env_file
+from .env.manager import EnvVarsManager
 
 if TYPE_CHECKING:
     from .config import PoeConfig
     from .ui import PoeUi
 
-# TODO: think about factoring env var concerns out to a dedicated class
-
 
 class RunContext:
     config: "PoeConfig"
     ui: "PoeUi"
-    env: Dict[str, str]
+    env: EnvVarsManager
     dry: bool
     poe_active: Optional[str]
     project_dir: Path
     multistage: bool = False
     exec_cache: Dict[str, Any]
     captured_stdout: Dict[Tuple[str, ...], str]
-    _envfile_cache: Dict[str, Dict[str, str]]
 
     def __init__(
         self,
@@ -46,61 +40,26 @@ def __init__(
         self.project_dir = Path(config.project_dir)
         self.dry = dry
         self.poe_active = poe_active
+        self.multistage = multistage
         self.exec_cache = {}
         self.captured_stdout = {}
-        self._envfile_cache = {}
-        self.base_env = self.__build_base_env(env)
-
-    def __build_base_env(self, env: Mapping[str, str]):
-        # Get env vars from envfile referenced in global options
-        result = dict(env)
-
-        # Get env vars from envfile referenced in global options
-        if self.config.global_envfile is not None:
-            result.update(self.get_env_file(self.config.global_envfile))
-
-        # Get env vars from global options
-        self._update_env(result, self.config.global_env)
-
-        result["POE_ROOT"] = str(self.config.project_dir)
-        return result
-
-    @staticmethod
-    def _update_env(
-        env: MutableMapping[str, str],
-        extra_vars: Mapping[str, Union[str, Mapping[str, str]]],
-    ):
-        """
-        Update the given env with the given extra_vars. If a value in extra_vars is
-        indicated as `default` then only copy it over if that key is not already set on
-        env.
-        """
-        for key, value in extra_vars.items():
-            if isinstance(value, str):
-                env[key] = value
-            elif key not in env:
-                env[key] = value["default"]
+        self.env = EnvVarsManager(self.config, self.ui, base_env=env)
 
     @property
     def executor_type(self) -> Optional[str]:
         return self.config.executor["type"]
 
-    def get_env(
+    def get_task_env(
         self,
-        parent_env: Optional[Mapping[str, str]],
+        parent_env: Optional[EnvVarsManager],
         task_envfile: Optional[str],
         task_env: Optional[Mapping[str, str]],
         task_uses: Optional[Mapping[str, Tuple[str, ...]]] = None,
-    ) -> Dict[str, str]:
-        result = dict(self.base_env, **(parent_env or {}))
-
-        # Include env vars from envfile referenced in task options
-        if task_envfile is not None:
-            result.update(self.get_env_file(task_envfile))
+    ) -> EnvVarsManager:
+        if parent_env is None:
+            parent_env = self.env
 
-        # Include env vars from task options
-        if task_env is not None:
-            self._update_env(result, task_env)
+        result = parent_env.for_task(task_envfile, task_env)
 
         # Include env vars from dependencies
         if task_uses is not None:
@@ -122,7 +81,7 @@ def get_dep_values(
     def get_executor(
         self,
         invocation: Tuple[str, ...],
-        env: Mapping[str, str],
+        env: EnvVarsManager,
         task_options: Dict[str, Any],
     ) -> PoeExecutor:
         return PoeExecutor.get(
@@ -134,29 +93,3 @@ def get_executor(
             executor_config=task_options.get("executor"),
             capture_stdout=task_options.get("capture_stdout", False),
         )
-
-    def get_env_file(self, envfile_path_str: str) -> Dict[str, str]:
-        if envfile_path_str in self._envfile_cache:
-            return self._envfile_cache[envfile_path_str]
-
-        result = {}
-
-        envfile_path = self.project_dir.joinpath(envfile_path_str)
-        if envfile_path.is_file():
-            try:
-                with envfile_path.open() as envfile:
-                    result = load_env_file(envfile)
-            except ValueError as error:
-                message = error.args[0]
-                raise ExecutionError(
-                    f"Syntax error in referenced envfile: {envfile_path_str!r}; {message}"
-                ) from error
-
-        else:
-            self.ui.print_msg(
-                f"Warning: Poe failed to locate envfile at {envfile_path_str!r}",
-                verbosity=1,
-            )
-
-        self._envfile_cache[envfile_path_str] = result
-        return result

poethepoet/env/cache.py

@@ -0,0 +1,47 @@
+from pathlib import Path
+from typing import (
+    Dict,
+    Optional,
+    TYPE_CHECKING,
+)
+from ..exceptions import ExecutionError
+from .parse import parse_env_file
+
+if TYPE_CHECKING:
+    from .ui import PoeUi
+
+
+class EnvFileCache:
+    _cache: Dict[str, Dict[str, str]] = {}
+    _ui: Optional["PoeUi"]
+    _project_dir: Path
+
+    def __init__(self, project_dir: Path, ui: Optional["PoeUi"]):
+        self._project_dir = project_dir
+        self._ui = ui
+
+    def get(self, envfile_path_str: str) -> Dict[str, str]:
+        if envfile_path_str in self._cache:
+            return self._cache[envfile_path_str]
+
+        result = {}
+
+        envfile_path = self._project_dir.joinpath(envfile_path_str)
+        if envfile_path.is_file():
+            try:
+                with envfile_path.open() as envfile:
+                    result = parse_env_file(envfile.readlines())
+            except ValueError as error:
+                message = error.args[0]
+                raise ExecutionError(
+                    f"Syntax error in referenced envfile: {envfile_path_str!r}; {message}"
+                ) from error
+
+        elif self._ui is not None:
+            self._ui.print_msg(
+                f"Warning: Poe failed to locate envfile at {envfile_path_str!r}",
+                verbosity=1,
+            )
+
+        self._cache[envfile_path_str] = result
+        return result