Skip to content

Commit

Permalink
Merge pull request #177 from shrutimantri/add-include-exclude-examples
Browse files Browse the repository at this point in the history
feat(docs): add examples for include/exclude in namespace files
  • Loading branch information
anna-geller authored Sep 23, 2024
2 parents b6bda90 + a18a218 commit a9376bc
Show file tree
Hide file tree
Showing 3 changed files with 55 additions and 7 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
@Example(
full = true,
title = """
Execute a Python script in a Conda virtual environment. First, add the following script in the embedded VS Code editor and name it `etl_script.py`:
Execute a Python script in a Conda virtual environment. First, add the following script in the embedded Code Editor and name it `etl_script.py`:
```python
import argparse
Expand All @@ -42,7 +42,7 @@
print(result)
```
Then, make sure to set the `enabled` flag of the `namespaceFiles` property to `true` to enable [namespace files](https://kestra.io/docs/developer-guide/namespace-files).
Then, make sure to set the `enabled` flag of the `namespaceFiles` property to `true` to enable [namespace files](https://kestra.io/docs/developer-guide/namespace-files). We `include` only the `etl_script.py` file as that is the only file we require from namespace files.
This flow uses a `io.kestra.plugin.core.runner.Process` Task Runner and Conda virtual environment for process isolation and dependency management. However, note that, by default, Kestra runs tasks in a Docker container (i.e. a Docker task runner), and you can use the `taskRunner` property to customize many options, as well as `containerImage` to choose the Docker image to use.
""",
Expand All @@ -55,6 +55,8 @@
type: io.kestra.plugin.scripts.python.Commands
namespaceFiles:
enabled: true
include:
- etl_script.py
taskRunner:
type: io.kestra.plugin.core.runner.Process
beforeCommands:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@
end
```
In order to read that script from the [Namespace File](https://kestra.io/docs/developer-guide/namespace-files) called `main.rb`, you need to enable the `namespaceFiles` property.
In order to read that script from the [Namespace File](https://kestra.io/docs/developer-guide/namespace-files) called `main.rb`, you need to enable the `namespaceFiles` property. We include only `main.rb` as that is the only file we want from the `namespaceFiles`.
Also, note how we use the `inputFiles` option to read additional files into the script's working directory. In this case, we read the `data.json` file, which contains the data that we want to convert to CSV.
Expand All @@ -66,6 +66,8 @@
type: io.kestra.plugin.scripts.ruby.Commands
namespaceFiles:
enabled: true
include:
- main.rb
inputFiles:
data.json: |
[
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,46 @@
- 'echo "The current execution is: {{ execution.id }}"'
"""
),
@Example(
title = "Include only specific namespace files.",
full = true,
code = """
id: include_files
namespace: company.team
tasks:
- id: command
type: io.kestra.plugin.scripts.shell.Commands
description: "Only the included `namespaceFiles` get listed"
namespaceFiles:
enabled: true
include:
- test1.txt
- test2.yaml
commands:
- ls
"""
),
@Example(
title = "Exclude specific namespace files.",
full = true,
code = """
id: exclude_files
namespace: company.team
tasks:
- id: command
type: io.kestra.plugin.scripts.shell.Commands
description: "All `namespaceFiles` except those that are excluded will be injected into the task's working directory"
namespaceFiles:
enabled: true
exclude:
- test1.txt
- test2.yaml
commands:
- ls
"""
),
@Example(
title = "Execute Shell commands that generate files accessible by other tasks and available for download in the UI's Output tab.",
full = true,
Expand All @@ -82,10 +122,14 @@
namespace: company.team
tasks:
- id: commands
type: io.kestra.plugin.scripts.shell.Commands
commands:
- cat {{ outputs.previousTaskId.uri }}
- id: http_download
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/products.csv
- id: commands
type: io.kestra.plugin.scripts.shell.Commands
commands:
- cat {{ outputs.http_download.uri }}
"""
),
@Example(
Expand Down

0 comments on commit a9376bc

Please sign in to comment.