diff --git a/README.rst b/README.rst index b4b86960a..f1cab8c8e 100644 --- a/README.rst +++ b/README.rst @@ -231,11 +231,17 @@ scripts *(shell)*, and sequence tasks *(sequence)*. pfwd = { "shell" = "ssh -N -L 0.0.0.0:8080:$STAGING:8080 $STAGING & ssh -N -L 0.0.0.0:5432:$STAGINGDB:5432 $STAGINGDB &" } pfwdstop = { "shell" = "kill $(pgrep -f "ssh -N -L .*:(8080|5432)")" } - By default poe attempts to find a posix shell (sh, bash, or zsh in that order) on the system and uses that. When running on windows, this might not always be possible. If bash is not found on the path on windows then poe will explicitly look for `Git bash `_ at the usual location. + By default poe attempts to find a posix shell (sh, bash, or zsh in that order) on the + system and uses that. When running on windows, this might not always be possible. If + bash is not found on the path on windows then poe will explicitly look for + `Git bash `_ at the usual location. **Using different types of shell/interpreter** - It is also possible to specify an alternative interpreter (or list of compatible interpreters ordered by preference) to be invoked to execute shell task content. For example if you only expect the task to be executed on windows or other environments with powershell installed then you can specify a powershell based task like so: + It is also possible to specify an alternative interpreter (or list of compatible + interpreters ordered by preference) to be invoked to execute shell task content. For + example if you only expect the task to be executed on windows or other environments + with powershell installed then you can specify a powershell based task like so: .. code-block:: toml @@ -245,13 +251,17 @@ scripts *(shell)*, and sequence tasks *(sequence)*. """ interpreter = "pwsh" - If your task content is restricted to syntax that is valid for both posix shells and powershell then you can maximise increase the likelihood of it working on any system by specifying the interpreter as: + If your task content is restricted to syntax that is valid for both posix shells and + powershell then you can maximise increase the likelihood of it working on any system + by specifying the interpreter as: .. code-block:: toml interpreter = ["posix", "pwsh"] - It is also possible to specify python code as the shell task code as in the following example. However it is recommended to use a *script* task rather than writing complex code inline within your pyproject.toml. + It is also possible to specify python code as the shell task code as in the following + example. However it is recommended to use a *script* task rather than writing complex + code inline within your pyproject.toml. .. code-block:: toml @@ -266,9 +276,11 @@ scripts *(shell)*, and sequence tasks *(sequence)*. The following interpreter values may be used: posix - This is the default behavoir, equivalent to ["sh", "bash", "zsh"], meaning that poe will try to find sh, and fallback to bash, then zsh. + This is the default behavoir, equivalent to ["sh", "bash", "zsh"], meaning that + poe will try to find sh, and fallback to bash, then zsh. sh - Use the basic posix shell. This is often an alias for bash or dash depending on the operating system. + Use the basic posix shell. This is often an alias for bash or dash depending on + the operating system. bash Uses whatever version of bash can be found. This is usually the most portable option. zsh @@ -280,7 +292,8 @@ scripts *(shell)*, and sequence tasks *(sequence)*. powershell Uses the newest version of powershell that can be found. - The default value can be changed with the global *shell_interpreter* option as described below. + The default value can be changed with the global *shell_interpreter* option as + described below. - **Composite tasks** are defined as a sequence of other tasks as an array. @@ -387,7 +400,7 @@ env key like so: serve.env = { PORT = "9001" } Notice this example uses deep keys which can be more convenient but aren't as well -supported by some toml implementations. +supported by some older toml implementations. The above example can be modified to only set the `PORT` variable if it is not already set by replacing the last line with the following: @@ -411,6 +424,19 @@ You can also specify an env file (with bash-like syntax) to load per task like s serve.script = "myapp:run" serve.envfile = ".env" +It it also possible to reference existing env vars when defining a new env var for a +task. This may be useful for aliasing or extending a variable already defined in the +host environment, globally in the config, or in a referencd envfile. In the following +example the value from $TF_VAR_service_port on the host environment is also made +available as $FLASK_RUN_PORT within the task. + +.. code-block:: toml + + [tool.poe.tasks.serve] + serve.cmd = "flask run" + serve.env = { FLASK_RUN_PORT = "${TF_VAR_service_port}" } + + Declaring CLI arguments ----------------------- @@ -633,9 +659,13 @@ pyproject.toml file by specifying :toml:`tool.poe.env` like so [tool.poe.env] VAR1 = "FOO" - VAR2 = "BAR" + VAR2 = "BAR BAR BLACK ${FARM_ANIMAL}" + +The example above also demonstrates how – as with env vars defined at the task level – +posix variable interpolation syntax may be used to define global env vars with reference +to variables already defined in the host environment or in a referenced env file. -As for the task level option, you can indicated that a variable should only be set if +As with the task level option, you can indicated that a variable should only be set if not already set like so: .. code-block:: toml @@ -799,7 +829,8 @@ so: [tool.poe] include = ["modules/acme_common/shared_tasks.toml", "generated_tasks.json"] -Files are loaded in the order specified. If an item already exists then the included value it ignored. +Files are loaded in the order specified. If an item already exists then the included +value it ignored. If a referenced file is missing then poe ignores it without error, though failure to read the contents will result in failure. diff --git a/poethepoet/context.py b/poethepoet/context.py index 65ade0c10..b2ed40fc1 100644 --- a/poethepoet/context.py +++ b/poethepoet/context.py @@ -3,34 +3,28 @@ Any, Dict, Mapping, - MutableMapping, Optional, Tuple, - Union, TYPE_CHECKING, ) -from .exceptions import ExecutionError from .executor import PoeExecutor -from .envfile import load_env_file +from .env.manager import EnvVarsManager if TYPE_CHECKING: from .config import PoeConfig from .ui import PoeUi -# TODO: think about factoring env var concerns out to a dedicated class - class RunContext: config: "PoeConfig" ui: "PoeUi" - env: Dict[str, str] + env: EnvVarsManager dry: bool poe_active: Optional[str] project_dir: Path multistage: bool = False exec_cache: Dict[str, Any] captured_stdout: Dict[Tuple[str, ...], str] - _envfile_cache: Dict[str, Dict[str, str]] def __init__( self, @@ -46,61 +40,26 @@ def __init__( self.project_dir = Path(config.project_dir) self.dry = dry self.poe_active = poe_active + self.multistage = multistage self.exec_cache = {} self.captured_stdout = {} - self._envfile_cache = {} - self.base_env = self.__build_base_env(env) - - def __build_base_env(self, env: Mapping[str, str]): - # Get env vars from envfile referenced in global options - result = dict(env) - - # Get env vars from envfile referenced in global options - if self.config.global_envfile is not None: - result.update(self.get_env_file(self.config.global_envfile)) - - # Get env vars from global options - self._update_env(result, self.config.global_env) - - result["POE_ROOT"] = str(self.config.project_dir) - return result - - @staticmethod - def _update_env( - env: MutableMapping[str, str], - extra_vars: Mapping[str, Union[str, Mapping[str, str]]], - ): - """ - Update the given env with the given extra_vars. If a value in extra_vars is - indicated as `default` then only copy it over if that key is not already set on - env. - """ - for key, value in extra_vars.items(): - if isinstance(value, str): - env[key] = value - elif key not in env: - env[key] = value["default"] + self.env = EnvVarsManager(self.config, self.ui, base_env=env) @property def executor_type(self) -> Optional[str]: return self.config.executor["type"] - def get_env( + def get_task_env( self, - parent_env: Optional[Mapping[str, str]], + parent_env: Optional[EnvVarsManager], task_envfile: Optional[str], task_env: Optional[Mapping[str, str]], task_uses: Optional[Mapping[str, Tuple[str, ...]]] = None, - ) -> Dict[str, str]: - result = dict(self.base_env, **(parent_env or {})) - - # Include env vars from envfile referenced in task options - if task_envfile is not None: - result.update(self.get_env_file(task_envfile)) + ) -> EnvVarsManager: + if parent_env is None: + parent_env = self.env - # Include env vars from task options - if task_env is not None: - self._update_env(result, task_env) + result = parent_env.for_task(task_envfile, task_env) # Include env vars from dependencies if task_uses is not None: @@ -122,7 +81,7 @@ def get_dep_values( def get_executor( self, invocation: Tuple[str, ...], - env: Mapping[str, str], + env: EnvVarsManager, task_options: Dict[str, Any], ) -> PoeExecutor: return PoeExecutor.get( @@ -134,29 +93,3 @@ def get_executor( executor_config=task_options.get("executor"), capture_stdout=task_options.get("capture_stdout", False), ) - - def get_env_file(self, envfile_path_str: str) -> Dict[str, str]: - if envfile_path_str in self._envfile_cache: - return self._envfile_cache[envfile_path_str] - - result = {} - - envfile_path = self.project_dir.joinpath(envfile_path_str) - if envfile_path.is_file(): - try: - with envfile_path.open() as envfile: - result = load_env_file(envfile) - except ValueError as error: - message = error.args[0] - raise ExecutionError( - f"Syntax error in referenced envfile: {envfile_path_str!r}; {message}" - ) from error - - else: - self.ui.print_msg( - f"Warning: Poe failed to locate envfile at {envfile_path_str!r}", - verbosity=1, - ) - - self._envfile_cache[envfile_path_str] = result - return result diff --git a/poethepoet/env/__init__.py b/poethepoet/env/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/poethepoet/env/cache.py b/poethepoet/env/cache.py new file mode 100644 index 000000000..01a1607cb --- /dev/null +++ b/poethepoet/env/cache.py @@ -0,0 +1,47 @@ +from pathlib import Path +from typing import ( + Dict, + Optional, + TYPE_CHECKING, +) +from ..exceptions import ExecutionError +from .parse import parse_env_file + +if TYPE_CHECKING: + from .ui import PoeUi + + +class EnvFileCache: + _cache: Dict[str, Dict[str, str]] = {} + _ui: Optional["PoeUi"] + _project_dir: Path + + def __init__(self, project_dir: Path, ui: Optional["PoeUi"]): + self._project_dir = project_dir + self._ui = ui + + def get(self, envfile_path_str: str) -> Dict[str, str]: + if envfile_path_str in self._cache: + return self._cache[envfile_path_str] + + result = {} + + envfile_path = self._project_dir.joinpath(envfile_path_str) + if envfile_path.is_file(): + try: + with envfile_path.open() as envfile: + result = parse_env_file(envfile.readlines()) + except ValueError as error: + message = error.args[0] + raise ExecutionError( + f"Syntax error in referenced envfile: {envfile_path_str!r}; {message}" + ) from error + + elif self._ui is not None: + self._ui.print_msg( + f"Warning: Poe failed to locate envfile at {envfile_path_str!r}", + verbosity=1, + ) + + self._cache[envfile_path_str] = result + return result diff --git a/poethepoet/env/manager.py b/poethepoet/env/manager.py new file mode 100644 index 000000000..374f3e353 --- /dev/null +++ b/poethepoet/env/manager.py @@ -0,0 +1,100 @@ +from pathlib import Path +from typing import ( + Dict, + Mapping, + Optional, + Union, + TYPE_CHECKING, +) +from .cache import EnvFileCache +from .template import apply_envvars_to_template + +if TYPE_CHECKING: + from .config import PoeConfig + from .ui import PoeUi + + +class EnvVarsManager: + _config: "PoeConfig" + _ui: Optional["PoeUi"] + _vars: Dict[str, str] + envfiles: EnvFileCache + + def __init__( + self, + config: "PoeConfig", + ui: Optional["PoeUi"], + parent_env: Optional["EnvVarsManager"] = None, + base_env: Optional[Mapping[str, str]] = None, + ): + self._config = config + self._ui = ui + self.envfiles = ( + # Reuse EnvFileCache from parent_env when possible + EnvFileCache(Path(config.project_dir), self._ui) + if parent_env is None + else parent_env.envfiles + ) + self._vars = { + **(parent_env.to_dict() if parent_env is not None else {}), + **(base_env or {}), + } + + if parent_env is None: + # Get env vars from envfile referenced in global options + if self._config.global_envfile is not None: + self._vars.update(self.envfiles.get(self._config.global_envfile)) + + # Get env vars from global options + self._apply_env_config(self._config.global_env) + + self._vars["POE_ROOT"] = str(self._config.project_dir) + + def _apply_env_config( + self, + config_env: Mapping[str, Union[str, Mapping[str, str]]], + ): + """ + Used for including env vars from global or task config. + If a value is provided as a mapping from `"default"` to `str` then it is only + used if the associated key doesn't already have a value. + """ + for key, value in config_env.items(): + if isinstance(value, str): + value_str = value + elif key not in self._vars: + value_str = value["default"] + else: + continue + + self._vars[key] = apply_envvars_to_template( + value_str, self._vars, require_braces=True + ) + + def for_task( + self, task_envfile: Optional[str], task_env: Optional[Mapping[str, str]] + ) -> "EnvVarsManager": + """ + Create a copy of self and extend it to include vars for the task. + """ + result = EnvVarsManager(self._config, self._ui, parent_env=self) + + # Include env vars from envfile referenced in task options + if task_envfile is not None: + result.update(self.envfiles.get(task_envfile)) + + # Include env vars from task options + if task_env is not None: + result._apply_env_config(task_env) + + return result + + def update(self, env_vars: Mapping[str, str]): + self._vars.update(env_vars) + return self + + def to_dict(self): + return dict(self._vars) + + def fill_template(self, template: str): + return apply_envvars_to_template(template, self._vars) diff --git a/poethepoet/envfile.py b/poethepoet/env/parse.py similarity index 79% rename from poethepoet/envfile.py rename to poethepoet/env/parse.py index 7bf96458e..c813daa62 100644 --- a/poethepoet/envfile.py +++ b/poethepoet/env/parse.py @@ -1,13 +1,21 @@ from enum import Enum import re -from typing import Dict, List, Optional, TextIO +from typing import Iterable, Optional, Sequence class ParserException(ValueError): - def __init__(self, message: str, position: int): - super().__init__(message) - self.message = message - self.position = position + def __init__(self, issue: str, offset: int, lines: Iterable[str]): + self.line_num, self.position = self._get_line_number(offset, lines) + super().__init__(f"{issue} at line {self.line_num} position {self.position}.") + + def _get_line_number(self, position: int, lines: Iterable[str]): + line_num = 1 + for line in lines: + if len(line) > position: + break + line_num += 1 + position -= len(line) + return line_num, position class ParserState(Enum): @@ -21,19 +29,6 @@ class ParserState(Enum): IN_DOUBLE_QUOTE = 3 -def load_env_file(envfile: TextIO) -> Dict[str, str]: - """ - Parses variable assignments from the given string. Expects a subset of bash syntax. - """ - - content_lines = envfile.readlines() - try: - return parse_env_file("".join(content_lines)) - except ParserException as error: - line_num, position = _get_line_number(content_lines, error.position) - raise ValueError(f"{error.message} at line {line_num} position {position}.") - - VARNAME_PATTERN = r"^[\s\t;]*(?:export[\s\t]+)?([a-zA-Z_][a-zA-Z_0-9]*)" ASSIGNMENT_PATTERN = f"{VARNAME_PATTERN}=" COMMENT_SUFFIX_PATTERN = r"^[\s\t;]*\#.*?\n" @@ -43,8 +38,8 @@ def load_env_file(envfile: TextIO) -> Dict[str, str]: DOUBLE_QUOTE_VALUE_PATTERN = r"^((?:.|\n)*?)(\"|\\+)" -def parse_env_file(content: str): - content = content + "\n" +def parse_env_file(content_lines: Sequence[str]): + content = "".join(content_lines) + "\n" result = {} cursor = 0 state = ParserState.SCAN_VAR_NAME @@ -78,11 +73,12 @@ def parse_env_file(content: str): if var_name_match: cursor += var_name_match.span()[1] raise ParserException( - f"Expected assignment operator", - cursor, + f"Expected assignment operator", cursor, content_lines ) - raise ParserException(f"Expected variable assignment", cursor) + raise ParserException( + f"Expected variable assignment", cursor, content_lines + ) var_name = match.group(1) cursor += match.end() @@ -133,7 +129,9 @@ def parse_env_file(content: str): SINGLE_QUOTE_VALUE_PATTERN, content[cursor:], re.MULTILINE ) if match is None: - raise ParserException(f"Unmatched single quote", cursor - 1) + raise ParserException( + f"Unmatched single quote", cursor - 1, content_lines + ) var_content.append(match.group(1)) cursor += match.end() state = ParserState.SCAN_VALUE @@ -145,7 +143,9 @@ def parse_env_file(content: str): DOUBLE_QUOTE_VALUE_PATTERN, content[cursor:], re.MULTILINE ) if match is None: - raise ParserException(f"Unmatched double quote", cursor - 1) + raise ParserException( + f"Unmatched double quote", cursor - 1, content_lines + ) new_var_content, backslashes_or_dquote = match.groups() var_content.append(new_var_content) cursor += match.end() @@ -164,13 +164,3 @@ def parse_env_file(content: str): cursor += 1 return result - - -def _get_line_number(lines: List[str], position: int): - line_num = 1 - for line in lines: - if len(line) > position: - break - line_num += 1 - position -= len(line) - return line_num, position diff --git a/poethepoet/helpers/env.py b/poethepoet/env/template.py similarity index 57% rename from poethepoet/helpers/env.py rename to poethepoet/env/template.py index cfcf7a253..e16b09f53 100644 --- a/poethepoet/helpers/env.py +++ b/poethepoet/env/template.py @@ -5,15 +5,25 @@ # Matches shell variable patterns, distinguishing escaped examples (to be ignored) # There may be a more direct way to doing this r"(?:" - r"(?:[^\\]|^)(?:\\(?:\\{2})*)\$([\w\d_]+)|" # $VAR preceded by an odd num of \ - r"(?:[^\\]|^)(?:\\(?:\\{2})*)\$\{([\w\d_]+)\}|" # ${VAR} preceded by an odd num of \ - r"\$([\w\d_]+)|" # $VAR - r"\${([\w\d_]+)}" # ${VAR} - r")" + r"(?:[^\\]|^)(?:\\(?:\\{2})*)\$(?P[\w\d_]+)|" # $VAR preceded by an odd num of \ + r"(?:[^\\]|^)(?:\\(?:\\{2})*)\$\{(?P[\w\d_]+)\}|" # ${VAR} preceded by an odd num of \ + r"\$(?P[\w\d_]+)|" # $VAR + r"\${(?P[\w\d_]+)}" # ${VAR} + ")" +) + +_SHELL_VAR_PATTERN_BRACES = re.compile( + # Matches shell variable patterns, distinguishing escaped examples (to be ignored) + r"(?:" + r"(?:[^\\]|^)(?:\\(?:\\{2})*)\$\{(?P[\w\d_]+)\}|" # ${VAR} preceded by an odd num of \ + r"\${(?P[\w\d_]+)}" # ${VAR} + ")" ) -def resolve_envvars(content: str, env: Mapping[str, str]) -> str: +def apply_envvars_to_template( + content: str, env: Mapping[str, str], require_braces=False +) -> str: """ Template in ${environmental} $variables from env as if we were in a shell @@ -22,19 +32,22 @@ def resolve_envvars(content: str, env: Mapping[str, str]) -> str: intentionally very limited implementation of escaping semantics for the sake of usability. """ + pattern = _SHELL_VAR_PATTERN_BRACES if require_braces else _SHELL_VAR_PATTERN + cursor = 0 resolved_parts = [] - for match in _SHELL_VAR_PATTERN.finditer(content): - groups = match.groups() - # the first two groups match escaped varnames so should be ignored - var_name = groups[2] or groups[3] - escaped_var_name = groups[0] or groups[1] + for match in pattern.finditer(content): + groups = match.groupdict() + var_name = groups.get("paren") or groups.get("naked") + escaped_var_name = groups.get("esc_paren") or groups.get("esc_naked") + if var_name: var_value = env.get(var_name) resolved_parts.append(content[cursor : match.start()]) cursor = match.end() if var_value is not None: resolved_parts.append(var_value) + elif escaped_var_name: # Remove the effective escape char resolved_parts.append(content[cursor : match.start()]) @@ -44,5 +57,6 @@ def resolve_envvars(content: str, env: Mapping[str, str]) -> str: resolved_parts.append(matched[1:]) else: resolved_parts.append(matched[0:1] + matched[2:]) + resolved_parts.append(content[cursor:]) return "".join(resolved_parts) diff --git a/poethepoet/executor/base.py b/poethepoet/executor/base.py index 7aa725398..ceb3b16aa 100644 --- a/poethepoet/executor/base.py +++ b/poethepoet/executor/base.py @@ -13,6 +13,7 @@ TYPE_CHECKING, Union, ) +from ..env.manager import EnvVarsManager from ..exceptions import PoeException from ..virtualenv import Virtualenv @@ -46,13 +47,14 @@ class PoeExecutor(metaclass=MetaPoeExecutor): working_dir: Optional["Path"] __executor_types: Dict[str, Type["PoeExecutor"]] = {} + __key__: Optional[str] = None def __init__( self, invocation: Tuple[str, ...], context: "RunContext", options: Mapping[str, str], - env: Mapping[str, str], + env: EnvVarsManager, working_dir: Optional["Path"] = None, dry: bool = False, capture_stdout: Union[str, bool] = False, @@ -70,7 +72,7 @@ def get( cls, invocation: Tuple[str, ...], context: "RunContext", - env: Mapping[str, str], + env: EnvVarsManager, working_dir: Optional["Path"] = None, dry: bool = False, executor_config: Optional[Mapping[str, str]] = None, @@ -117,7 +119,10 @@ def _resolve_implementation( return cls.__executor_types[config_executor_type] def execute(self, cmd: Sequence[str], input: Optional[bytes] = None) -> int: - raise NotImplementedError + """ + Execute the given cmd as a subprocess inside the poetry managed dev environment + """ + return self._exec_via_subproc(cmd, input=input) def _exec_via_subproc( self, @@ -130,7 +135,9 @@ def _exec_via_subproc( if self.dry: return 0 popen_kwargs: MutableMapping[str, Any] = {"shell": shell} - popen_kwargs["env"] = self.env if env is None else env + popen_kwargs["env"] = dict( + (self.env.to_dict() if env is None else env), POE_ACTIVE=self.__key__ + ) if input is not None: popen_kwargs["stdin"] = PIPE if self.capture_stdout: diff --git a/poethepoet/executor/poetry.py b/poethepoet/executor/poetry.py index ea6a26915..a1891153a 100644 --- a/poethepoet/executor/poetry.py +++ b/poethepoet/executor/poetry.py @@ -53,17 +53,11 @@ def _execute_cmd( return self._exec_via_subproc( (venv.resolve_executable(cmd[0]), *cmd[1:]), input=input, - env=dict( - venv.get_env_vars(self.env), POE_ACTIVE=PoetryExecutor.__key__ - ), + env=venv.get_env_vars(self.env.to_dict()), + shell=shell, ) - return self._exec_via_subproc( - cmd, - input=input, - env=dict(self.env, POE_ACTIVE=PoetryExecutor.__key__), - shell=shell, - ) + return self._exec_via_subproc(cmd, input=input, shell=shell) def _get_poetry_virtualenv(self, force: bool = True): """ diff --git a/poethepoet/executor/simple.py b/poethepoet/executor/simple.py index ba6c998e5..cb92085d9 100644 --- a/poethepoet/executor/simple.py +++ b/poethepoet/executor/simple.py @@ -1,4 +1,4 @@ -from typing import Dict, Optional, Sequence, Type +from typing import Dict, Type from .base import PoeExecutor @@ -9,11 +9,3 @@ class SimpleExecutor(PoeExecutor): __key__ = "simple" __options__: Dict[str, Type] = {} - - def execute(self, cmd: Sequence[str], input: Optional[bytes] = None) -> int: - """ - Execute the given cmd as a subprocess inside the poetry managed dev environment - """ - return self._exec_via_subproc( - cmd, input=input, env=dict(self.env, POE_ACTIVE=SimpleExecutor.__key__) - ) diff --git a/poethepoet/executor/virtualenv.py b/poethepoet/executor/virtualenv.py index 4cb2d310f..5ebeacb5a 100644 --- a/poethepoet/executor/virtualenv.py +++ b/poethepoet/executor/virtualenv.py @@ -20,9 +20,7 @@ def execute(self, cmd: Sequence[str], input: Optional[bytes] = None) -> int: return self._exec_via_subproc( (venv.resolve_executable(cmd[0]), *cmd[1:]), input=input, - env=dict( - venv.get_env_vars(self.env), POE_ACTIVE=VirtualenvExecutor.__key__ - ), + env=venv.get_env_vars(self.env.to_dict()), ) def _resolve_virtualenv(self) -> Virtualenv: diff --git a/poethepoet/task/base.py b/poethepoet/task/base.py index de21c74e3..322db5d95 100644 --- a/poethepoet/task/base.py +++ b/poethepoet/task/base.py @@ -17,7 +17,7 @@ from .args import PoeTaskArgs from ..exceptions import PoeException from ..helpers import is_valid_env_var -from ..helpers.env import resolve_envvars +from ..env.manager import EnvVarsManager if TYPE_CHECKING: from ..context import RunContext @@ -191,30 +191,24 @@ def _parse_named_args(self, extra_args: Sequence[str]) -> Optional[Dict[str, str return PoeTaskArgs(args_def, self.name).parse(extra_args) return None - def add_named_args_to_env( - self, env: Mapping[str, str] - ) -> Tuple[Mapping[str, str], bool]: - if self.named_args is None: - return env, False - return ( - dict( - env, - **( - { - key: str(value) - for key, value in self.named_args.items() - if value is not None - } - ), - ), - bool(self.named_args), - ) + @property + def has_named_args(self): + return bool(self.named_args) + + def get_named_arg_values(self) -> Mapping[str, str]: + if not self.named_args: + return {} + return { + key: str(value) + for key, value in self.named_args.items() + if value is not None + } def run( self, context: "RunContext", extra_args: Sequence[str] = tuple(), - env: Optional[Mapping[str, str]] = None, + parent_env: Optional[EnvVarsManager] = None, ) -> int: """ Run this task @@ -223,8 +217,8 @@ def run( return self._handle_run( context, extra_args, - context.get_env( - env, + context.get_task_env( + parent_env, self.options.get("envfile"), self.options.get("env"), upstream_invocations["uses"], @@ -235,10 +229,10 @@ def _handle_run( self, context: "RunContext", extra_args: Sequence[str], - env: Mapping[str, str], + env: EnvVarsManager, ) -> int: """ - _handle_run must be implemented by a subclass and return a single executor + This method must be implemented by a subclass and return a single executor result. """ raise NotImplementedError @@ -256,23 +250,22 @@ def _get_upstream_invocations(self, context: "RunContext"): """ NB. this memoization assumes the context (and contained env vars) will be the same in all instances for the lifetime of this object. Whilst this should be OK - for all corrent usecases is it strictly speaking something that this object + for all current usecases is it strictly speaking something that this object should not know enough to safely assume. So we probably want to revisit this. """ if self.__upstream_invocations is None: - env: Mapping - env = context.get_env( - {}, self.options.get("envfile"), self.options.get("env") + env = context.get_task_env( + None, self.options.get("envfile"), self.options.get("env") ) - env, _ = self.add_named_args_to_env(env) + env.update(self.get_named_arg_values()) self.__upstream_invocations = { "deps": [ - tuple(shlex.split(resolve_envvars(task_ref, env))) + tuple(shlex.split(env.fill_template(task_ref))) for task_ref in self.options.get("deps", tuple()) ], "uses": { - key: tuple(shlex.split(resolve_envvars(task_ref, env))) + key: tuple(shlex.split(env.fill_template(task_ref))) for key, task_ref in self.options.get("uses", {}).items() }, } diff --git a/poethepoet/task/cmd.py b/poethepoet/task/cmd.py index 8a9fb6a63..08defd813 100644 --- a/poethepoet/task/cmd.py +++ b/poethepoet/task/cmd.py @@ -3,7 +3,6 @@ import shlex from typing import ( Dict, - Mapping, Sequence, Type, Tuple, @@ -11,7 +10,7 @@ Union, ) from .base import PoeTask -from ..helpers.env import resolve_envvars +from ..env.manager import EnvVarsManager if TYPE_CHECKING: from ..config import PoeConfig @@ -35,10 +34,11 @@ def _handle_run( self, context: "RunContext", extra_args: Sequence[str], - env: Mapping[str, str], + env: EnvVarsManager, ) -> int: - env, has_named_args = self.add_named_args_to_env(env) - if has_named_args: + env.update(self.get_named_arg_values()) + + if self.has_named_args: # If named arguments are defined then it doesn't make sense to pass extra # args to the command, because they've already been parsed cmd = self._resolve_args(context, env) @@ -47,13 +47,14 @@ def _handle_run( self._print_action(" ".join(cmd), context.dry) return context.get_executor(self.invocation, env, self.options).execute(cmd) - def _resolve_args(self, context: "RunContext", env: Mapping[str, str]): + def _resolve_args(self, context: "RunContext", env: EnvVarsManager): + updated_content = env.fill_template(self.content) # Parse shell command tokens and check if they're quoted if self._is_windows: cmd_tokens = ( (compat_token, bool(_QUOTED_TOKEN_PATTERN.match(compat_token))) for compat_token in shlex.split( - resolve_envvars(self.content, env), + updated_content, posix=False, comments=True, ) @@ -63,12 +64,12 @@ def _resolve_args(self, context: "RunContext", env: Mapping[str, str]): (posix_token, bool(_QUOTED_TOKEN_PATTERN.match(compat_token))) for (posix_token, compat_token) in zip( shlex.split( - resolve_envvars(self.content, env), + updated_content, posix=True, comments=True, ), shlex.split( - resolve_envvars(self.content, env), + updated_content, posix=False, comments=True, ), diff --git a/poethepoet/task/ref.py b/poethepoet/task/ref.py index cdb06a1c6..4b14f21cf 100644 --- a/poethepoet/task/ref.py +++ b/poethepoet/task/ref.py @@ -2,7 +2,6 @@ from typing import ( Any, Dict, - Mapping, Optional, Sequence, Type, @@ -11,7 +10,7 @@ Union, ) from .base import PoeTask -from ..helpers.env import resolve_envvars +from ..env.manager import EnvVarsManager if TYPE_CHECKING: from ..config import PoeConfig @@ -32,14 +31,14 @@ def _handle_run( self, context: "RunContext", extra_args: Sequence[str], - env: Mapping[str, str], + env: EnvVarsManager, ) -> int: """ Lookup and delegate to the referenced task """ - invocation = tuple(shlex.split(resolve_envvars(self.content, env))) + invocation = tuple(shlex.split(env.fill_template(self.content))) task = self.from_config(invocation[0], self._config, self._ui, invocation) - return task.run(context=context, extra_args=extra_args, env=env) + return task.run(context=context, extra_args=extra_args, parent_env=env) @classmethod def _validate_task_def( diff --git a/poethepoet/task/script.py b/poethepoet/task/script.py index 29d14120e..3cb14a9fa 100644 --- a/poethepoet/task/script.py +++ b/poethepoet/task/script.py @@ -2,7 +2,6 @@ from typing import ( Any, Dict, - Mapping, Optional, Sequence, Tuple, @@ -12,7 +11,7 @@ ) from .base import PoeTask from ..exceptions import ScriptParseError -from ..helpers.env import resolve_envvars +from ..env.manager import EnvVarsManager from ..helpers.python import ( resolve_function_call, parse_and_validate, @@ -38,14 +37,14 @@ def _handle_run( self, context: "RunContext", extra_args: Sequence[str], - env: Mapping[str, str], + env: EnvVarsManager, ) -> int: # TODO: check whether the project really does use src layout, and don't do # sys.path.append('src') if it doesn't target_module, function_call = self.parse_script_content(self.named_args) argv = [ self.name, - *(resolve_envvars(token, env) for token in extra_args), + *(env.fill_template(token) for token in extra_args), ] cmd = ( "python", diff --git a/poethepoet/task/sequence.py b/poethepoet/task/sequence.py index 16b679847..e2f41f78a 100644 --- a/poethepoet/task/sequence.py +++ b/poethepoet/task/sequence.py @@ -2,7 +2,6 @@ Any, Dict, List, - Mapping, Optional, Sequence, Tuple, @@ -11,6 +10,7 @@ Union, ) from .base import PoeTask, TaskContent +from ..env.manager import EnvVarsManager from ..exceptions import ExecutionError, PoeException if TYPE_CHECKING: @@ -64,11 +64,11 @@ def _handle_run( self, context: "RunContext", extra_args: Sequence[str], - env: Mapping[str, str], + env: EnvVarsManager, ) -> int: - env, has_named_args = self.add_named_args_to_env(env) + env.update(self.get_named_arg_values()) - if not has_named_args and any(arg.strip() for arg in extra_args): + if not self.has_named_args and any(arg.strip() for arg in extra_args): raise PoeException(f"Sequence task {self.name!r} does not accept arguments") if len(self.subtasks) > 1: @@ -78,7 +78,9 @@ def _handle_run( ignore_fail = self.options.get("ignore_fail") non_zero_subtasks: List[str] = list() for subtask in self.subtasks: - task_result = subtask.run(context=context, extra_args=tuple(), env=env) + task_result = subtask.run( + context=context, extra_args=tuple(), parent_env=env + ) if task_result and not ignore_fail: raise ExecutionError( f"Sequence aborted after failed subtask {subtask.name!r}" diff --git a/poethepoet/task/shell.py b/poethepoet/task/shell.py index de8f5aa40..bfe534a74 100644 --- a/poethepoet/task/shell.py +++ b/poethepoet/task/shell.py @@ -5,7 +5,6 @@ Any, Dict, List, - Mapping, Optional, Sequence, Tuple, @@ -13,6 +12,7 @@ TYPE_CHECKING, Union, ) +from ..env.manager import EnvVarsManager from ..exceptions import PoeException from .base import PoeTask @@ -35,11 +35,11 @@ def _handle_run( self, context: "RunContext", extra_args: Sequence[str], - env: Mapping[str, str], + env: EnvVarsManager, ) -> int: - env, has_named_args = self.add_named_args_to_env(env) + env.update(self.get_named_arg_values()) - if not has_named_args and any(arg.strip() for arg in extra_args): + if not self.has_named_args and any(arg.strip() for arg in extra_args): raise PoeException(f"Shell task {self.name!r} does not accept arguments") interpreter_cmd = self.resolve_interpreter_cmd() diff --git a/pyproject.toml b/pyproject.toml index 74592cb04..2d9365334 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -101,7 +101,7 @@ commands = [tool.coverage.report] -omit = ["**/site-packages/**"] +omit = ["**/site-packages/**", "poethepoet/completion/*", "poethepoet/plugin.py"] [tool.pytest.ini_options] diff --git a/tests/fixtures/cmds_project/pyproject.toml b/tests/fixtures/cmds_project/pyproject.toml index c4e09117a..7913a1517 100644 --- a/tests/fixtures/cmds_project/pyproject.toml +++ b/tests/fixtures/cmds_project/pyproject.toml @@ -8,3 +8,7 @@ env = { BEST_PASSWORD = "Password1" } [tool.poe.tasks.greet] shell = "echo $formal_greeting $subject" args = ["formal-greeting", "subject"] + +[tool.poe.tasks.surfin-bird] +cmd = "echo $WORD is the word" +env = { WORD = "${SOME_INPUT_VAR}"} diff --git a/tests/fixtures/envfile_project/pyproject.toml b/tests/fixtures/envfile_project/pyproject.toml index b56abae3d..e77d2c213 100644 --- a/tests/fixtures/envfile_project/pyproject.toml +++ b/tests/fixtures/envfile_project/pyproject.toml @@ -1,11 +1,12 @@ [tool.poe] envfile = "credentials.env" +env = { HOST = "${HOST}:80" } [tool.poe.tasks.deploy-dev] cmd = """ echo "deploying to ${USER}:${PASSWORD}@${HOST}${PATH_SUFFIX}" """ - +env = { HOST = "${HOST}80" } # reference and override value from envfile [tool.poe.tasks.deploy-prod] cmd = """ diff --git a/tests/test_cmd_tasks.py b/tests/test_cmd_tasks.py index 658148b5e..4044c3329 100644 --- a/tests/test_cmd_tasks.py +++ b/tests/test_cmd_tasks.py @@ -22,3 +22,12 @@ def test_cmd_task_with_dash_case_arg(run_poe_subproc): assert result.capture == f"Poe => echo $formal_greeting $subject\n" assert result.stdout == "hey you\n" assert result.stderr == "" + + +def test_cmd_alias_env_var(run_poe_subproc): + result = run_poe_subproc( + "surfin-bird", project="cmds", env={"SOME_INPUT_VAR": "BIRD"} + ) + assert result.capture == f"Poe => echo BIRD is the word\n" + assert result.stdout == "BIRD is the word\n" + assert result.stderr == "" diff --git a/tests/test_envfile.py b/tests/test_envfile.py index 9e5246f36..445ca406a 100644 --- a/tests/test_envfile.py +++ b/tests/test_envfile.py @@ -8,15 +8,17 @@ def test_global_envfile_and_default(run_poe_subproc, is_windows): if is_windows: # On windows shlex works in non-POSIX mode which results in quotes assert ( - 'Poe => echo "deploying to admin:12345@dev.example.com"\n' in result.capture + 'Poe => echo "deploying to admin:12345@dev.example.com:8080"\n' + in result.capture ) - assert result.stdout == '"deploying to admin:12345@dev.example.com"\n' + assert result.stdout == '"deploying to admin:12345@dev.example.com:8080"\n' assert result.stderr == "" else: assert ( - "Poe => echo deploying to admin:12345@dev.example.com\n" in result.capture + "Poe => echo deploying to admin:12345@dev.example.com:8080\n" + in result.capture ) - assert result.stdout == "deploying to admin:12345@dev.example.com\n" + assert result.stdout == "deploying to admin:12345@dev.example.com:8080\n" assert result.stderr == "" diff --git a/tests/test_executors.py b/tests/test_executors.py index a6bf51000..732b446a7 100644 --- a/tests/test_executors.py +++ b/tests/test_executors.py @@ -38,7 +38,9 @@ def test_virtualenv_executor_provides_access_to_venv_content( ): # version 1.0.0 of flask isn't around much venv_path = projects["venv"].joinpath("myvenv") - for _ in with_virtualenv_and_venv(venv_path, ["flask==1.0.0"]): + for _ in with_virtualenv_and_venv( + venv_path, ["itsdangerous==2.0.1", "flask==1.0.0"] + ): # binaries from the venv are directly callable result = run_poe_subproc("server-version", project="venv") assert result.capture == "Poe => flask --version\n" @@ -76,7 +78,7 @@ def test_detect_venv( assert result.stderr == "" # if we install flask into this virtualenv then we should get a different result - install_into_virtualenv(venv_path, ["flask==1.0.0"]) + install_into_virtualenv(venv_path, ["itsdangerous==2.0.1", "flask==1.0.0"]) result = run_poe_subproc("detect_flask", project="simple") assert result.capture == "Poe => detect_flask\n" assert result.stdout.startswith("Flask found at ") diff --git a/tests/unit/test_parse_env_file.py b/tests/unit/test_parse_env_file.py index 65bea8e87..479959f1f 100644 --- a/tests/unit/test_parse_env_file.py +++ b/tests/unit/test_parse_env_file.py @@ -1,4 +1,4 @@ -from poethepoet.envfile import parse_env_file +from poethepoet.env.parse import parse_env_file import pytest valid_examples = [