diff --git a/docs/README.md b/docs/README.md index dc5cb7046c..576948df3a 100644 --- a/docs/README.md +++ b/docs/README.md @@ -24,6 +24,11 @@ This creates a new environment called `monodocs-env` with all the dependencies n In the `flyteorg/flyte` root directory make sure you have activated the `monodocs-env` (or whatever you called it) environment and do: +```bash +# need to set this to a fake value to build the docs locally +$ export DOCSEARCH_API_KEY=fake-api-key +``` + ```bash $ make docs ``` diff --git a/docs/community/contribute/contribute_docs.md b/docs/community/contribute/contribute_docs.md index f97152032b..7c1d47ffc8 100644 --- a/docs/community/contribute/contribute_docs.md +++ b/docs/community/contribute/contribute_docs.md @@ -82,14 +82,14 @@ The following are some tips to include various content: * **Source code references (Embedded format)**
`.rst` example: ```{code-block} - .. rli:: https://raw.githubusercontent.com/flyteorg/// + .. literalinclude:: /examples/ :lines: - ``` `.md` example: ````{code-block} - ```{rli} https://raw.githubusercontent.com/flyteorg/// - lines: - + ```{literalinclude} /examples/ + :lines: - ``` ```` diff --git a/docs/conf.py b/docs/conf.py index 316acc60be..83fca407b7 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -343,6 +343,22 @@ "flytesnacks/README.md", "flytekit/**/README.md", "flytekit/_templates/**", + "examples/advanced_composition/**", + "examples/basics/**", + "examples/customizing_dependencies/**", + "examples/data_types_and_io/**", + "examples/development_lifecycle/**", + "examples/extending/**", + "examples/productionizing/**", + "examples/testing/**", + "flytesnacks/examples/advanced_composition/*.md", + "flytesnacks/examples/basics/*.md", + "flytesnacks/examples/customizing_dependencies/*.md", + "flytesnacks/examples/data_types_and_io/*.md", + "flytesnacks/examples/development_lifecycle/*.md", + "flytesnacks/examples/extending/*.md", + "flytesnacks/examples/productionizing/*.md", + "flytesnacks/examples/testing/*.md", "api/flytectl/index.rst", "protos/boilerplate/**", "protos/tmp/**", @@ -622,14 +638,6 @@ "flytesnacks/_build", "flytesnacks/_tags", "flytesnacks/index.md", - "examples/advanced_composition", - "examples/basics", - "examples/customizing_dependencies", - "examples/data_types_and_io", - "examples/development_lifecycle", - "examples/extending", - "examples/productionizing", - "examples/testing" ] ], "local": flytesnacks_local_path is not None, @@ -690,6 +698,15 @@ # Disable warnings from tensorflow os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3" +# Define the canonical URL if you are using a custom domain on Read the Docs +html_baseurl = os.environ.get("READTHEDOCS_CANONICAL_URL", "") + +# Tell Jinja2 templates the build is running on Read the Docs +if os.environ.get("READTHEDOCS", "") == "True": + if "html_context" not in globals(): + html_context = {} + html_context["READTHEDOCS"] = True + class CustomWarningSuppressor(logging.Filter): """Filter logs by `suppress_warnings`.""" diff --git a/docs/user_guide/advanced_composition/decorating_workflows.md b/docs/user_guide/advanced_composition/decorating_workflows.md index 751cd6a95c..ee5f02c8a3 100644 --- a/docs/user_guide/advanced_composition/decorating_workflows.md +++ b/docs/user_guide/advanced_composition/decorating_workflows.md @@ -23,7 +23,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the necessary libraries. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/decorating_workflows.py +```{literalinclude} /examples/advanced_composition/advanced_composition/decorating_workflows.py :caption: advanced_composition/decorating_workflows.py :lines: 1-6 ``` @@ -32,7 +32,7 @@ Let's define the tasks we need for setup and teardown. In this example, we use t {py:class}`unittest.mock.MagicMock` class to create a fake external service that we want to initialize at the beginning of our workflow and finish at the end. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/decorating_workflows.py +```{literalinclude} /examples/advanced_composition/advanced_composition/decorating_workflows.py :caption: advanced_composition/decorating_workflows.py :lines: 9-21 ``` @@ -45,7 +45,7 @@ external service and Flyte. We create a decorator that we want to use to wrap our workflow function. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/decorating_workflows.py +```{literalinclude} /examples/advanced_composition/advanced_composition/decorating_workflows.py :caption: advanced_composition/decorating_workflows.py :pyobject: setup_teardown ``` @@ -66,14 +66,14 @@ There are a few key pieces to note in the `setup_teardown` decorator above: We define two tasks that will constitute the workflow. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/decorating_workflows.py +```{literalinclude} /examples/advanced_composition/advanced_composition/decorating_workflows.py :caption: advanced_composition/decorating_workflows.py :lines: 63-70 ``` And then create our decorated workflow: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/decorating_workflows.py +```{literalinclude} /examples/advanced_composition/advanced_composition/decorating_workflows.py :caption: advanced_composition/decorating_workflows.py :lines: 74-82 ``` diff --git a/docs/user_guide/advanced_composition/eager_workflows.md b/docs/user_guide/advanced_composition/eager_workflows.md index 9bf3e019c8..4b83679473 100644 --- a/docs/user_guide/advanced_composition/eager_workflows.md +++ b/docs/user_guide/advanced_composition/eager_workflows.md @@ -45,7 +45,7 @@ using the `@eager` decorator. To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/eager_workflows.py +```{literalinclude} /examples/advanced_composition/advanced_composition/eager_workflows.py :caption: advanced_composition/eager_workflows.py :lines: 1-21 ``` @@ -116,7 +116,7 @@ One of the biggest benefits of eager workflows is that you can now materialize task and subworkflow outputs as Python values and do operations on them just like you would in any other Python function. Let's look at another example: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/eager_workflows.py +```{literalinclude} /examples/advanced_composition/advanced_composition/eager_workflows.py :caption: advanced_composition/eager_workflows.py :pyobject: another_eager_workflow ``` @@ -131,7 +131,7 @@ As you saw in the `simple_eager_workflow` workflow above, you can use regular Python conditionals in your eager workflows. Let's look at a more complicated example: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/eager_workflows.py +```{literalinclude} /examples/advanced_composition/advanced_composition/eager_workflows.py :caption: advanced_composition/eager_workflows.py :lines: 36-53 ``` @@ -144,7 +144,7 @@ to check if `out` is negative, but we're also using the `gt_100` task in the You can also gather the outputs of multiple tasks or subworkflows into a list: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/eager_workflows.py +```{literalinclude} /examples/advanced_composition/advanced_composition/eager_workflows.py :caption: advanced_composition/eager_workflows.py :lines: 58-69 ``` @@ -153,7 +153,7 @@ You can also gather the outputs of multiple tasks or subworkflows into a list: You can also invoke static workflows from within an eager workflow: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/eager_workflows.py +```{literalinclude} /examples/advanced_composition/advanced_composition/eager_workflows.py :caption: advanced_composition/eager_workflows.py :lines: 74-84 ``` @@ -162,7 +162,7 @@ You can also invoke static workflows from within an eager workflow: You can have nest eager subworkflows inside a parent eager workflow: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/eager_workflows.py +```{literalinclude} /examples/advanced_composition/advanced_composition/eager_workflows.py :caption: advanced_composition/eager_workflows.py :lines: 89-97 ``` @@ -171,7 +171,7 @@ You can have nest eager subworkflows inside a parent eager workflow: You can also catch exceptions in eager workflows through `EagerException`: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/eager_workflows.py +```{literalinclude} /examples/advanced_composition/advanced_composition/eager_workflows.py :caption: advanced_composition/eager_workflows.py :lines: 102-117 ``` @@ -195,7 +195,7 @@ and remotely. You can execute eager workflows locally by simply calling them like a regular `async` function: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/eager_workflows.py +```{literalinclude} /examples/advanced_composition/advanced_composition/eager_workflows.py :caption: advanced_composition/eager_workflows.py :lines: 123-125 ``` @@ -244,7 +244,7 @@ When using a sandbox cluster started with `flytectl demo start`, however, the `client_secret_group` and `client_secret_key` are not required, since the default sandbox configuration does not require key-based authentication. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/eager_workflows.py +```{literalinclude} /examples/advanced_composition/advanced_composition/eager_workflows.py :caption: advanced_composition/eager_workflows.py :lines: 130-145 ``` diff --git a/docs/user_guide/advanced_composition/intratask_checkpoints.md b/docs/user_guide/advanced_composition/intratask_checkpoints.md index d856a45714..81631e40e7 100644 --- a/docs/user_guide/advanced_composition/intratask_checkpoints.md +++ b/docs/user_guide/advanced_composition/intratask_checkpoints.md @@ -51,14 +51,14 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the necessary libraries and set the number of task retries to `3`: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/checkpoint.py +```{literalinclude} /examples/advanced_composition/advanced_composition/checkpoint.py :caption: advanced_composition/checkpoint.py :lines: 1-4 ``` We define a task to iterate precisely `n_iterations`, checkpoint its state, and recover from simulated failures: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/checkpoint.py +```{literalinclude} /examples/advanced_composition/advanced_composition/checkpoint.py :caption: advanced_composition/checkpoint.py :pyobject: use_checkpoint ``` @@ -69,14 +69,14 @@ The checkpoint system offers additional APIs, documented in the code accessible Create a workflow that invokes the task: The task will automatically undergo retries in the event of a {ref}`FlyteRecoverableException `. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/checkpoint.py +```{literalinclude} /examples/advanced_composition/advanced_composition/checkpoint.py :caption: advanced_composition/checkpoint.py :pyobject: checkpointing_example ``` The local checkpoint is not utilized here because retries are not supported: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/checkpoint.py +```{literalinclude} /examples/advanced_composition/advanced_composition/checkpoint.py :caption: advanced_composition/checkpoint.py :lines: 37-42 ``` diff --git a/docs/user_guide/advanced_composition/map_tasks.md b/docs/user_guide/advanced_composition/map_tasks.md index 26330a8cd5..f73946cd89 100644 --- a/docs/user_guide/advanced_composition/map_tasks.md +++ b/docs/user_guide/advanced_composition/map_tasks.md @@ -23,14 +23,14 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the required libraries: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/map_task.py +```{literalinclude} /examples/advanced_composition/advanced_composition/map_task.py :caption: advanced_composition/map_task.py :lines: 1 ``` Here's a simple workflow that uses {py:func}`map_task `: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/map_task.py +```{literalinclude} /examples/advanced_composition/advanced_composition/map_task.py :caption: advanced_composition/map_task.py :lines: 4-19 ``` @@ -82,7 +82,7 @@ When defining a map task, avoid calling other tasks in it. Flyte can't accuratel In this example, the map task `suboptimal_mappable_task` would not give you the best performance: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/map_task.py +```{literalinclude} /examples/advanced_composition/advanced_composition/map_task.py :caption: advanced_composition/map_task.py :lines: 31-40 ``` @@ -98,7 +98,7 @@ You might need to map a task with multiple inputs. For instance, consider a task that requires three inputs: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/map_task.py +```{literalinclude} /examples/advanced_composition/advanced_composition/map_task.py :caption: advanced_composition/map_task.py :pyobject: multi_input_task ``` @@ -107,21 +107,21 @@ You may want to map this task with only the ``quantity`` input, while keeping th Since a map task accepts only one input, you can achieve this by partially binding values to the map task. This can be done using the {py:func}`functools.partial` function: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/map_task.py +```{literalinclude} /examples/advanced_composition/advanced_composition/map_task.py :caption: advanced_composition/map_task.py :lines: 52-58 ``` Another possibility is to bind the outputs of a task to partials: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/map_task.py +```{literalinclude} /examples/advanced_composition/advanced_composition/map_task.py :caption: advanced_composition/map_task.py :lines: 63-72 ``` You can also provide multiple lists as input to a `map_task`: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/map_task.py +```{literalinclude} /examples/advanced_composition/advanced_composition/map_task.py :caption: advanced_composition/map_task.py :pyobject: map_workflow_with_lists ``` diff --git a/docs/user_guide/advanced_composition/subworkflows.md b/docs/user_guide/advanced_composition/subworkflows.md index 08a4bbb8d4..14d3cc1006 100644 --- a/docs/user_guide/advanced_composition/subworkflows.md +++ b/docs/user_guide/advanced_composition/subworkflows.md @@ -24,7 +24,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte Here's an example illustrating the calculation of slope, intercept and the corresponding y-value: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/subworkflow.py +```{literalinclude} /examples/advanced_composition/advanced_composition/subworkflow.py :caption: advanced_composition/subworkflow.py :lines: 1-35 ``` @@ -34,7 +34,7 @@ Subsequently, the `regression_line_wf` triggers `slope_intercept_wf` and then co To execute the workflow locally, use the following: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/subworkflow.py +```{literalinclude} /examples/advanced_composition/advanced_composition/subworkflow.py :caption: advanced_composition/subworkflow.py :lines: 39-40 ``` @@ -43,14 +43,14 @@ It's possible to nest a workflow that contains a subworkflow within another work Workflows can be easily constructed from other workflows, even if they function as standalone entities. Each workflow in this module has the capability to exist and run independently: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/subworkflow.py +```{literalinclude} /examples/advanced_composition/advanced_composition/subworkflow.py :caption: advanced_composition/subworkflow.py :pyobject: nested_regression_line_wf ``` You can run the nested workflow locally as well: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/subworkflow.py +```{literalinclude} /examples/advanced_composition/advanced_composition/subworkflow.py :caption: advanced_composition/subworkflow.py :lines: 52-53 ``` @@ -71,7 +71,7 @@ external workflows may offer a way to distribute the workload of a workflow acro Here's an example that illustrates the concept of external workflows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/subworkflow.py +```{literalinclude} /examples/advanced_composition/advanced_composition/subworkflow.py :caption: advanced_composition/subworkflow.py :lines: 61-71 ``` @@ -85,7 +85,7 @@ In the console screenshot above, note that the launch plan execution ID differs You can run a workflow containing an external workflow locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/subworkflow.py +```{literalinclude} /examples/advanced_composition/advanced_composition/subworkflow.py :caption: advanced_composition/subworkflow.py :lines: 75-76 ``` diff --git a/docs/user_guide/advanced_composition/waiting_for_external_inputs.md b/docs/user_guide/advanced_composition/waiting_for_external_inputs.md index 0d3a2aae28..edeb6e2b95 100644 --- a/docs/user_guide/advanced_composition/waiting_for_external_inputs.md +++ b/docs/user_guide/advanced_composition/waiting_for_external_inputs.md @@ -43,7 +43,7 @@ your workflow to mock out the behavior of some long-running computation. To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/waiting_for_external_inputs.py +```{literalinclude} /examples/advanced_composition/advanced_composition/waiting_for_external_inputs.py :caption: advanced_composition/waiting_for_external_inputs.py :lines: 1-20 ``` @@ -75,7 +75,7 @@ but before publishing it you want to give it a custom title. You can achieve this by defining a `wait_for_input` node that takes a `str` input and finalizes the report: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/waiting_for_external_inputs.py +```{literalinclude} /examples/advanced_composition/advanced_composition/waiting_for_external_inputs.py :caption: advanced_composition/waiting_for_external_inputs.py :lines: 24-49 ``` @@ -107,7 +107,7 @@ an explicit approval signal before continuing execution. Going back to our report-publishing use case, suppose that we want to block the publishing of a report for some reason (e.g. if they don't appear to be valid): -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/waiting_for_external_inputs.py +```{literalinclude} /examples/advanced_composition/advanced_composition/waiting_for_external_inputs.py :caption: advanced_composition/waiting_for_external_inputs.py :lines: 53-64 ``` @@ -120,7 +120,7 @@ You can also use the output of the `approve` function as a promise, feeding it to a subsequent task. Let's create a version of our report-publishing workflow where the approval happens after `create_report`: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/waiting_for_external_inputs.py +```{literalinclude} /examples/advanced_composition/advanced_composition/waiting_for_external_inputs.py :caption: advanced_composition/waiting_for_external_inputs.py :pyobject: approval_as_promise_wf ``` @@ -133,7 +133,7 @@ useful when we combine them with other Flyte constructs, like {ref}`conditionals To illustrate this, let's extend the report-publishing use case so that we produce an "invalid report" output in case we don't approve the final report: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/advanced_composition/advanced_composition/waiting_for_external_inputs.py +```{literalinclude} /examples/advanced_composition/advanced_composition/waiting_for_external_inputs.py :caption: advanced_composition/waiting_for_external_inputs.py :lines: 88-114 ``` diff --git a/docs/user_guide/basics/documenting_workflows.md b/docs/user_guide/basics/documenting_workflows.md index 9f5e20d5fb..954bd35302 100644 --- a/docs/user_guide/basics/documenting_workflows.md +++ b/docs/user_guide/basics/documenting_workflows.md @@ -15,14 +15,14 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the relevant libraries: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/documenting_workflows.py +```{literalinclude} /examples/basics/basics/documenting_workflows.py :caption: basics/documenting_workflows.py :lines: 1-3 ``` We import the `slope` and `intercept` tasks from the `workflow.py` file. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/documenting_workflows.py +```{literalinclude} /examples/basics/basics/documenting_workflows.py :caption: basics/documenting_workflows.py :lines: 6 ``` @@ -35,7 +35,7 @@ The initial section of the docstring provides a concise overview of the workflow The subsequent section provides a comprehensive explanation. The last part of the docstring outlines the parameters and return type. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/documenting_workflows.py +```{literalinclude} /examples/basics/basics/documenting_workflows.py :caption: basics/documenting_workflows.py :pyobject: sphinx_docstring_wf ``` @@ -49,7 +49,7 @@ The next section offers a comprehensive description. The third section of the docstring details all parameters along with their respective data types. The final section of the docstring explains the return type and its associated data type. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/documenting_workflows.py +```{literalinclude} /examples/basics/basics/documenting_workflows.py :caption: basics/documenting_workflows.py :pyobject: numpy_docstring_wf ``` @@ -63,7 +63,7 @@ The subsequent section of the docstring provides an extensive explanation. The third segment of the docstring outlines the parameters and return type, including their respective data types. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/documenting_workflows.py +```{literalinclude} /examples/basics/basics/documenting_workflows.py :caption: basics/documenting_workflows.py :pyobject: google_docstring_wf ``` diff --git a/docs/user_guide/basics/hello_world.md b/docs/user_guide/basics/hello_world.md index a63b175fbe..d310a80273 100644 --- a/docs/user_guide/basics/hello_world.md +++ b/docs/user_guide/basics/hello_world.md @@ -17,7 +17,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import `task` and `workflow` from the `flytekit` library: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/hello_world.py +```{literalinclude} /examples/basics/basics/hello_world.py :caption: basics/hello_world.py :lines: 1 ``` @@ -25,7 +25,7 @@ To begin, import `task` and `workflow` from the `flytekit` library: Define a task that produces the string "Hello, World!". Simply using the `@task` decorator to annotate the Python function: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/hello_world.py +```{literalinclude} /examples/basics/basics/hello_world.py :caption: basics/hello_world.py :pyobject: say_hello ``` @@ -33,14 +33,14 @@ Simply using the `@task` decorator to annotate the Python function: You can handle the output of a task in the same way you would with a regular Python function. Store the output in a variable and use it as a return value for a Flyte workflow: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/hello_world.py +```{literalinclude} /examples/basics/basics/hello_world.py :caption: basics/hello_world.py :pyobject: hello_world_wf ``` Run the workflow by simply calling it like a Python function: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/hello_world.py +```{literalinclude} /examples/basics/basics/hello_world.py :caption: basics/hello_world.py :lines: 19-20 ``` diff --git a/docs/user_guide/basics/imperative_workflows.md b/docs/user_guide/basics/imperative_workflows.md index db0e3a5ee0..e6d189c1ba 100644 --- a/docs/user_guide/basics/imperative_workflows.md +++ b/docs/user_guide/basics/imperative_workflows.md @@ -23,28 +23,28 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the necessary dependencies: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/imperative_workflow.py +```{literalinclude} /examples/basics/basics/imperative_workflow.py :caption: basics/imperative_workflow.py :lines: 1 ``` We import the `slope` and `intercept` tasks from the `workflow.py` file: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/imperative_workflow.py +```{literalinclude} /examples/basics/basics/imperative_workflow.py :caption: basics/imperative_workflow.py :lines: 4 ``` Create an imperative workflow: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/imperative_workflow.py +```{literalinclude} /examples/basics/basics/imperative_workflow.py :caption: basics/imperative_workflow.py :lines: 7 ``` Add the workflow inputs to the imperative workflow: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/imperative_workflow.py +```{literalinclude} /examples/basics/basics/imperative_workflow.py :caption: basics/imperative_workflow.py :lines: 11-12 ``` @@ -56,21 +56,21 @@ you can create a {ref}`launch plan `. Add the tasks that need to be triggered from within the workflow: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/imperative_workflow.py +```{literalinclude} /examples/basics/basics/imperative_workflow.py :caption: basics/imperative_workflow.py :lines: 16-19 ``` Lastly, add the workflow output: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/imperative_workflow.py +```{literalinclude} /examples/basics/basics/imperative_workflow.py :caption: basics/imperative_workflow.py :lines: 23 ``` You can execute the workflow locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/imperative_workflow.py +```{literalinclude} /examples/basics/basics/imperative_workflow.py :caption: basics/imperative_workflow.py :lines: 27-28 ``` diff --git a/docs/user_guide/basics/launch_plans.md b/docs/user_guide/basics/launch_plans.md index 63ace47a70..6f1f983ba6 100644 --- a/docs/user_guide/basics/launch_plans.md +++ b/docs/user_guide/basics/launch_plans.md @@ -27,56 +27,56 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the necessary libraries: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/launch_plan.py +```{literalinclude} /examples/basics/basics/launch_plan.py :caption: basics/launch_plan.py :lines: 1 ``` We import the workflow from the `workflow.py` file for which we're going to create a launch plan: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/launch_plan.py +```{literalinclude} /examples/basics/basics/launch_plan.py :caption: basics/launch_plan.py :lines: 5 ``` Create a default launch plan with no inputs during serialization: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/launch_plan.py +```{literalinclude} /examples/basics/basics/launch_plan.py :caption: basics/launch_plan.py :lines: 8 ``` You can run the launch plan locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/launch_plan.py +```{literalinclude} /examples/basics/basics/launch_plan.py :caption: basics/launch_plan.py :lines: 11 ``` Create a launch plan and specify the default inputs: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/launch_plan.py +```{literalinclude} /examples/basics/basics/launch_plan.py :caption: basics/launch_plan.py :lines: 14-16 ``` You can trigger the launch plan locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/launch_plan.py +```{literalinclude} /examples/basics/basics/launch_plan.py :caption: basics/launch_plan.py :lines: 19 ``` You can override the defaults as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/launch_plan.py +```{literalinclude} /examples/basics/basics/launch_plan.py :caption: basics/launch_plan.py :lines: 22 ``` It's possible to lock launch plan inputs, preventing them from being overridden during execution: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/launch_plan.py +```{literalinclude} /examples/basics/basics/launch_plan.py :caption: basics/launch_plan.py :lines: 25-27 ``` diff --git a/docs/user_guide/basics/named_outputs.md b/docs/user_guide/basics/named_outputs.md index 00b9160997..1d274c88b6 100644 --- a/docs/user_guide/basics/named_outputs.md +++ b/docs/user_guide/basics/named_outputs.md @@ -22,21 +22,21 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the required dependencies: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/named_outputs.py +```{literalinclude} /examples/basics/basics/named_outputs.py :caption: basics/named_outputs.py :lines: 1-3 ``` Define a `NamedTuple` and assign it as an output to a task: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/named_outputs.py +```{literalinclude} /examples/basics/basics/named_outputs.py :caption: basics/named_outputs.py :lines: 6-14 ``` Likewise, assign a `NamedTuple` to the output of `intercept` task: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/named_outputs.py +```{literalinclude} /examples/basics/basics/named_outputs.py :caption: basics/named_outputs.py :lines: 18-26 ``` @@ -59,14 +59,14 @@ Remember that we are extracting individual task execution outputs by dereferenci This is necessary because `NamedTuple`s function as tuples and require this dereferencing: ::: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/named_outputs.py +```{literalinclude} /examples/basics/basics/named_outputs.py :caption: basics/named_outputs.py :lines: 32-39 ``` You can run the workflow locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/named_outputs.py +```{literalinclude} /examples/basics/basics/named_outputs.py :caption: basics/named_outputs.py :lines: 43-44 ``` diff --git a/docs/user_guide/basics/shell_tasks.md b/docs/user_guide/basics/shell_tasks.md index 8680b87f5d..e1f8b23bef 100644 --- a/docs/user_guide/basics/shell_tasks.md +++ b/docs/user_guide/basics/shell_tasks.md @@ -15,7 +15,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte First, import the necessary libraries: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/shell_task.py +```{literalinclude} /examples/basics/basics/shell_task.py :caption: basics/shell_task.py :lines: 1-8 ``` @@ -24,7 +24,7 @@ With the required imports in place, you can proceed to define a shell task. To create a shell task, provide a name for it, specify the bash script to be executed, and define inputs and outputs if needed: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/shell_task.py +```{literalinclude} /examples/basics/basics/shell_task.py :caption: basics/shell_task.py :lines: 13-55 ``` @@ -40,21 +40,21 @@ Here's a breakdown of the parameters of the `ShellTask`: We define a task to instantiate `FlyteFile` and `FlyteDirectory`. A `.gitkeep` file is created in the FlyteDirectory as a placeholder to ensure the directory exists: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/shell_task.py +```{literalinclude} /examples/basics/basics/shell_task.py :caption: basics/shell_task.py :pyobject: create_entities ``` We create a workflow to define the dependencies between the tasks: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/shell_task.py +```{literalinclude} /examples/basics/basics/shell_task.py :caption: basics/shell_task.py :pyobject: shell_task_wf ``` You can run the workflow locally: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/shell_task.py +```{literalinclude} /examples/basics/basics/shell_task.py :caption: basics/shell_task.py :lines: 85-86 ``` diff --git a/docs/user_guide/basics/tasks.md b/docs/user_guide/basics/tasks.md index b76e61f5dc..22d078f9f1 100644 --- a/docs/user_guide/basics/tasks.md +++ b/docs/user_guide/basics/tasks.md @@ -34,7 +34,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import `task` from the `flytekit` library: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/basics/basics/task.py +```{literalinclude} /examples/basics/basics/task.py :caption: basics/task.py :lines: 1 ``` @@ -45,7 +45,7 @@ Learn more about the supported types in the {ref}`type-system section `, {std:ref}`FlyteFile ` and {std:ref}`FlyteDirectory `. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/cfb5ea3b0d0502ef7df1f2e14f4a0d9b78250b6a/examples/data_types_and_io/data_types_and_io/dataclass.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/dataclass.py :caption: data_types_and_io/dataclass.py :lines: 51-88 ``` @@ -67,14 +67,14 @@ flyte file, flyte directory and structured dataset. We define a workflow that calls the tasks created above. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/cfb5ea3b0d0502ef7df1f2e14f4a0d9b78250b6a/examples/data_types_and_io/data_types_and_io/dataclass.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/dataclass.py :caption: data_types_and_io/dataclass.py :pyobject: dataclass_wf ``` You can run the workflow locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/cfb5ea3b0d0502ef7df1f2e14f4a0d9b78250b6a/examples/data_types_and_io/data_types_and_io/dataclass.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/dataclass.py :caption: data_types_and_io/dataclass.py :lines: 101-102 ``` diff --git a/docs/user_guide/data_types_and_io/enum_type.md b/docs/user_guide/data_types_and_io/enum_type.md index b8e9011921..f5b1873d98 100644 --- a/docs/user_guide/data_types_and_io/enum_type.md +++ b/docs/user_guide/data_types_and_io/enum_type.md @@ -22,7 +22,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the dependencies: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/enum_type.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/enum_type.py :caption: data_types_and_io/enum_type.py :lines: 1-3 ``` @@ -30,14 +30,14 @@ To begin, import the dependencies: We define an enum and a simple coffee maker workflow that accepts an order and brews coffee ☕️ accordingly. The assumption is that the coffee maker only understands enum inputs: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/enum_type.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/enum_type.py :caption: data_types_and_io/enum_type.py :lines: 9-35 ``` The workflow can also accept an enum value: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/enum_type.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/enum_type.py :caption: data_types_and_io/enum_type.py :pyobject: coffee_maker_enum ``` @@ -51,7 +51,7 @@ pyflyte run \ You can run the workflows locally: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/enum_type.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/enum_type.py :caption: data_types_and_io/enum_type.py :lines: 44-46 ``` diff --git a/docs/user_guide/data_types_and_io/flytedirectory.md b/docs/user_guide/data_types_and_io/flytedirectory.md index 4ad2316ded..82cc5ab2a0 100644 --- a/docs/user_guide/data_types_and_io/flytedirectory.md +++ b/docs/user_guide/data_types_and_io/flytedirectory.md @@ -16,7 +16,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the libraries: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/folder.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/folder.py :caption: data_types_and_io/folder.py :lines: 1-10 ``` @@ -27,7 +27,7 @@ let's continue by considering the normalization of columns in a CSV file. The following task downloads a list of URLs pointing to CSV files and returns the folder path in a `FlyteDirectory` object. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/folder.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/folder.py :caption: data_types_and_io/folder.py :pyobject: download_files ``` @@ -57,7 +57,7 @@ demonstrates how Flyte tasks are simply entrypoints of execution, which can them other functions and routines that are written in pure Python. ::: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/folder.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/folder.py :caption: data_types_and_io/folder.py :pyobject: normalize_columns ``` @@ -65,7 +65,7 @@ other functions and routines that are written in pure Python. We then define a task that accepts the previously downloaded folder, along with some metadata about the column names of each file in the directory and the column names that we want to normalize. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/folder.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/folder.py :caption: data_types_and_io/folder.py :pyobject: normalize_all_files ``` @@ -74,14 +74,14 @@ Compose all of the above tasks into a workflow. This workflow accepts a list of URL strings pointing to a remote location containing a CSV file, a list of column names associated with each CSV file, and a list of columns that we want to normalize. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/folder.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/folder.py :caption: data_types_and_io/folder.py :pyobject: download_and_normalize_csv_files ``` You can run the workflow locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/folder.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/folder.py :caption: data_types_and_io/folder.py :lines: 94-114 ``` @@ -98,7 +98,7 @@ This feature is marked as experimental. We'd love feedback on the API! Here is a simple example, you can accept a `FlyteDirectory` as an input, walk through it and copy the files to another `FlyteDirectory` one by one. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/ddce0448141ea6d2cb148df52bf408874adb15ad/examples/data_types_and_io/data_types_and_io/file_streaming.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/file_streaming.py :caption: data_types_and_io/file_streaming.py :lines: 23-33 ``` diff --git a/docs/user_guide/data_types_and_io/flytefile.md b/docs/user_guide/data_types_and_io/flytefile.md index 76dc0f6be8..997dbe031a 100644 --- a/docs/user_guide/data_types_and_io/flytefile.md +++ b/docs/user_guide/data_types_and_io/flytefile.md @@ -23,7 +23,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte First, import the libraries: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/file.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/file.py :caption: data_types_and_io/file.py :lines: 1-8 ``` @@ -43,7 +43,7 @@ Predefined aliases for commonly used flyte file formats are also available. You can find them [here](https://github.com/flyteorg/flytekit/blob/master/flytekit/types/file/__init__.py). ::: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/file.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/file.py :caption: data_types_and_io/file.py :pyobject: normalize_columns ``` @@ -56,16 +56,16 @@ When this task finishes, the Flytekit engine returns the `FlyteFile` instance, u Lastly, define a workflow. The `normalize_csv_files` workflow has an `output_location` argument which is passed to the `location` input of the task. If it's not an empty string, the task attempts to upload its file to that location. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/0ec8388759d34566a0ffc0c3c2d7443fd4a3a46f/examples/data_types_and_io/data_types_and_io/file.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/file.py :caption: data_types_and_io/file.py :pyobject: normalize_csv_file ``` You can run the workflow locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/0ec8388759d34566a0ffc0c3c2d7443fd4a3a46f/examples/data_types_and_io/data_types_and_io/file.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/file.py :caption: data_types_and_io/file.py -:lines: 75-95 +:lines: 72-92 ``` You can enable type validation if you have the [python-magic](https://pypi.org/project/python-magic/) package installed. @@ -101,7 +101,7 @@ This feature is marked as experimental. We'd love feedback on the API! Here is a simple example of removing some columns from a CSV file and writing the result to a new file: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/ddce0448141ea6d2cb148df52bf408874adb15ad/examples/data_types_and_io/data_types_and_io/file_streaming.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/file_streaming.py :caption: data_types_and_io/file_streaming.py :lines: 8-20 ``` diff --git a/docs/user_guide/data_types_and_io/pickle_type.md b/docs/user_guide/data_types_and_io/pickle_type.md index 301ff95f9f..6a1d84bd37 100644 --- a/docs/user_guide/data_types_and_io/pickle_type.md +++ b/docs/user_guide/data_types_and_io/pickle_type.md @@ -27,7 +27,7 @@ This example demonstrates how you can utilize custom objects without registering To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/pickle_type.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/pickle_type.py :caption: data_types_and_io/pickle_type.py :lines: 1 ``` @@ -40,7 +40,7 @@ Alternatively, you can {ref}`turn this object into a dataclass ` for We have used a simple object here for demonstration purposes. ::: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/pickle_type.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/pickle_type.py :caption: data_types_and_io/pickle_type.py :lines: 7-26 ``` @@ -53,7 +53,7 @@ or significant list elements, you can specify a batch size. This feature allows for the processing of each batch as a separate pickle file. The following example demonstrates how to set the batch size. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/pickle_type.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/pickle_type.py :caption: data_types_and_io/pickle_type.py :lines: 35-58 ``` @@ -64,7 +64,7 @@ The `welcome_superheroes` task will generate two pickle files: one containing tw You can run the workflows locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/pickle_type.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/pickle_type.py :caption: data_types_and_io/pickle_type.py :lines: 62-64 ``` diff --git a/docs/user_guide/data_types_and_io/pytorch_type.md b/docs/user_guide/data_types_and_io/pytorch_type.md index 24696f6a75..685c7a38b2 100644 --- a/docs/user_guide/data_types_and_io/pytorch_type.md +++ b/docs/user_guide/data_types_and_io/pytorch_type.md @@ -18,7 +18,7 @@ At times, you may find the need to pass tensors and modules (models) within your To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/pytorch_type.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/pytorch_type.py :caption: data_types_and_io/pytorch_type.py :lines: 5-50 ``` @@ -36,7 +36,7 @@ According to the PyTorch [docs](https://pytorch.org/tutorials/beginner/saving_lo it's recommended to store the module's `state_dict` rather than the module itself, although the serialization should work in either case. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/pytorch_type.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/pytorch_type.py :caption: data_types_and_io/pytorch_type.py :lines: 63-117 ``` diff --git a/docs/user_guide/data_types_and_io/structureddataset.md b/docs/user_guide/data_types_and_io/structureddataset.md index 9a82610590..f22aaac077 100644 --- a/docs/user_guide/data_types_and_io/structureddataset.md +++ b/docs/user_guide/data_types_and_io/structureddataset.md @@ -37,14 +37,14 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the dependencies for the example: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :lines: 1-19 ``` Define a task that returns a Pandas DataFrame. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :pyobject: generate_pandas_df ``` @@ -66,7 +66,7 @@ you can just specify the column names and their types in the structured dataset First, initialize column types you want to extract from the `StructuredDataset`. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :lines: 31-32 ``` @@ -76,7 +76,7 @@ When you invoke `all()` with ``pandas.DataFrame``, the Flyte engine downloads th Keep in mind that you can invoke ``open()`` with any dataframe type that's supported or added to structured dataset. For instance, you can use ``pa.Table`` to convert the Pandas DataFrame to a PyArrow table. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :lines: 42-52 ``` @@ -89,7 +89,7 @@ You can use a custom serialization format to serialize your dataframes. Here's how you can register the Pandas to CSV handler, which is already available, and enable the CSV serialization by annotating the structured dataset with the CSV format: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :lines: 58-72 ``` @@ -198,7 +198,7 @@ enabling the use of a 2D NumPy array as a valid type within structured datasets. Extend `StructuredDatasetEncoder` and implement the `encode` function. The `encode` function converts NumPy array to an intermediate format (parquet file format in this case). -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :pyobject: NumpyEncodingHandler ``` @@ -208,7 +208,7 @@ The `encode` function converts NumPy array to an intermediate format (parquet fi Extend {py:class}`StructuredDatasetDecoder` and implement the {py:meth}`~StructuredDatasetDecoder.decode` function. The {py:meth}`~StructuredDatasetDecoder.decode` function converts the parquet file to a `numpy.ndarray`. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :pyobject: NumpyDecodingHandler ``` @@ -218,7 +218,7 @@ The {py:meth}`~StructuredDatasetDecoder.decode` function converts the parquet fi Create a default renderer for numpy array, then Flytekit will use this renderer to display schema of NumPy array on the Flyte deck. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :pyobject: NumpyRenderer ``` @@ -228,14 +228,14 @@ Specify the Python type you want to register this encoder with (`np.ndarray`), the storage engine to register this against (if not specified, it is assumed to work for all the storage backends), and the byte format, which in this case is `PARQUET`. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :lines: 128-130 ``` You can now use `numpy.ndarray` to deserialize the parquet file to NumPy and serialize a task's output (NumPy array) to a parquet file. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :lines: 135-148 ``` @@ -246,7 +246,7 @@ You can now use `numpy.ndarray` to deserialize the parquet file to NumPy and ser You can run the code locally as follows: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :lines: 152-156 ``` @@ -259,7 +259,7 @@ Like most storage formats (e.g. Avro, Parquet, and BigQuery), StructuredDataset Nested field StructuredDataset should be run when flytekit version > 1.11.0. ::: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/structured_dataset.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/structured_dataset.py :caption: data_types_and_io/structured_dataset.py :lines: 158-285 ``` diff --git a/docs/user_guide/data_types_and_io/tensorflow_type.md b/docs/user_guide/data_types_and_io/tensorflow_type.md index 43f620ce01..9035c71db5 100644 --- a/docs/user_guide/data_types_and_io/tensorflow_type.md +++ b/docs/user_guide/data_types_and_io/tensorflow_type.md @@ -9,7 +9,7 @@ This document outlines the TensorFlow types available in Flyte, which facilitate the integration of TensorFlow models and datasets in Flyte workflows. ### Import necessary libraries and modules -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/9aadec205a6e208c62e29f52873fb3d675965a51/examples/data_types_and_io/data_types_and_io/tensorflow_type.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/tensorflow_type.py :caption: data_types_and_io/tensorflow_type.py :lines: 3-12 ``` @@ -30,7 +30,7 @@ The `TensorFlowModelTransformer` allows you to save a TensorFlow model to a remo ```{note} To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/9aadec205a6e208c62e29f52873fb3d675965a51/examples/data_types_and_io/data_types_and_io/tensorflow_type.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/tensorflow_type.py :caption: data_types_and_io/tensorflow_type.py :lines: 16-34 ``` @@ -47,7 +47,7 @@ Flyte supports TFRecord files through the TFRecordFile type, which can handle se ### Usage The `TensorFlowRecordFileTransformer` enables you to work with single TFRecord files, making it easy to read and write data in TensorFlow's TFRecord format. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/9aadec205a6e208c62e29f52873fb3d675965a51/examples/data_types_and_io/data_types_and_io/tensorflow_type.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/tensorflow_type.py :caption: data_types_and_io/tensorflow_type.py :lines: 38-48 ``` @@ -66,7 +66,7 @@ Flyte supports directories containing multiple TFRecord files through the `TFRec The `TensorFlowRecordsDirTransformer` allows you to work with directories of TFRecord files, which is useful for handling large datasets that are split across multiple files. #### Example -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/9aadec205a6e208c62e29f52873fb3d675965a51/examples/data_types_and_io/data_types_and_io/tensorflow_type.py +```{literalinclude} /examples/data_types_and_io/data_types_and_io/tensorflow_type.py :caption: data_types_and_io/tensorflow_type.py :lines: 52-62 ``` diff --git a/docs/user_guide/development_lifecycle/cache_serializing.md b/docs/user_guide/development_lifecycle/cache_serializing.md index 1445de13cf..4ea14fb89e 100644 --- a/docs/user_guide/development_lifecycle/cache_serializing.md +++ b/docs/user_guide/development_lifecycle/cache_serializing.md @@ -17,7 +17,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte For any {py:func}`flytekit.task` in Flyte, there is always one required import, which is: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/task_cache_serialize.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/task_cache_serialize.py :caption: development_lifecycle/task_cache_serialize.py :lines: 1 ``` @@ -27,7 +27,7 @@ Task cache serializing is disabled by default to avoid unexpected behavior for t This operation is only useful for cacheable tasks, where one may reuse output from a previous execution. Flyte requires implicitly enabling the `cache` parameter on all cache serializable tasks. Cache key definitions follow the same rules as non-serialized cache tasks. It is important to understand the implications of the task signature and `cache_version` parameter in defining cached results. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/task_cache_serialize.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/task_cache_serialize.py :caption: development_lifecycle/task_cache_serialize.py :pyobject: square ``` diff --git a/docs/user_guide/development_lifecycle/caching.md b/docs/user_guide/development_lifecycle/caching.md index ea6a5af574..70b2fcaa64 100644 --- a/docs/user_guide/development_lifecycle/caching.md +++ b/docs/user_guide/development_lifecycle/caching.md @@ -77,19 +77,19 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte Import the necessary libraries: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/task_cache.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/task_cache.py :caption: development_lifecycle/task_cache.py :lines: 1-3 ``` For any {py:func}`flytekit.task` in Flyte, there is always one required import, which is: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/task_cache.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/task_cache.py :caption: development_lifecycle/task_cache.py :lines: 8-10 ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/task_cache.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/task_cache.py :caption: development_lifecycle/task_cache.py :pyobject: square ``` @@ -150,7 +150,7 @@ The format used by the store is opaque and not meant to be inspectable. The default behavior displayed by Flyte's memoization feature might not match the user intuition. For example, this code makes use of pandas dataframes: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/task_cache.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/task_cache.py :caption: development_lifecycle/task_cache.py :lines: 39-54 ``` @@ -159,7 +159,7 @@ If run twice with the same inputs, one would expect that `bar` would trigger a c However, with release 1.2.0, Flyte provides a new way to control memoization behavior of literals. This is done via a `typing.Annotated` call on the task signature. For example, in order to cache the result of calls to `bar`, you can rewrite the code above like this: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/task_cache.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/task_cache.py :caption: development_lifecycle/task_cache.py :lines: 64-85 ``` @@ -175,7 +175,7 @@ This feature also works in local execution. Here's a complete example of the feature: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/task_cache.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/task_cache.py :caption: development_lifecycle/task_cache.py :lines: 97-134 ``` diff --git a/docs/user_guide/development_lifecycle/decks.md b/docs/user_guide/development_lifecycle/decks.md index 366302d49e..68887615b3 100644 --- a/docs/user_guide/development_lifecycle/decks.md +++ b/docs/user_guide/development_lifecycle/decks.md @@ -28,7 +28,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte To begin, import the dependencies: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/decks.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py :lines: 1-4 ``` @@ -39,7 +39,7 @@ We create a new deck named `pca` and render Markdown content along with a You can begin by initializing an {ref}`ImageSpec ` object to encompass all the necessary dependencies. This approach automatically triggers a Docker build, alleviating the need for you to manually create a Docker image. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/b431ae399def3a749833fe81c2c291b016cf3213/examples/development_lifecycle/development_lifecycle/decks.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py :lines: 15-27 ``` @@ -51,7 +51,7 @@ To upload the image to the local registry in the demo cluster, indicate the regi Note the usage of `append` to append the Plotly deck to the Markdown deck. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/decks.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py :pyobject: pca_plot ``` @@ -96,7 +96,7 @@ When the task connected with a deck object is executed, these objects employ ren Creates a profile report from a Pandas DataFrame. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/b431ae399def3a749833fe81c2c291b016cf3213/examples/development_lifecycle/development_lifecycle/decks.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py :lines: 56-63 ``` @@ -113,7 +113,7 @@ Creates a profile report from a Pandas DataFrame. Renders DataFrame as an HTML table. This renderer doesn't necessitate plugin installation since it's accessible within the flytekit library. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/b431ae399def3a749833fe81c2c291b016cf3213/examples/development_lifecycle/development_lifecycle/decks.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py :lines: 69-76 ``` @@ -127,7 +127,7 @@ This renderer doesn't necessitate plugin installation since it's accessible with Converts a Markdown string into HTML, producing HTML as a Unicode string. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/decks.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py :pyobject: markdown_renderer ``` @@ -147,7 +147,7 @@ The median (Q2) is indicated by a line within the box. Typically, the whiskers extend to the edges of the box, plus or minus 1.5 times the interquartile range (IQR: Q3-Q1). -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/b431ae399def3a749833fe81c2c291b016cf3213/examples/development_lifecycle/development_lifecycle/decks.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py :lines: 97-103 ``` @@ -162,7 +162,7 @@ plus or minus 1.5 times the interquartile range (IQR: Q3-Q1). Converts a {ref}`FlyteFile ` or `PIL.Image.Image` object into an HTML string, where the image data is encoded as a base64 string. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/b431ae399def3a749833fe81c2c291b016cf3213/examples/development_lifecycle/development_lifecycle/decks.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py :lines: 109-123 ``` @@ -176,7 +176,7 @@ where the image data is encoded as a base64 string. Converts a Pandas dataframe into an HTML table. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/b431ae399def3a749833fe81c2c291b016cf3213/examples/development_lifecycle/development_lifecycle/decks.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/decks.py :caption: development_lifecycle/decks.py :lines: 127-135 ``` diff --git a/docs/user_guide/development_lifecycle/failure_node.md b/docs/user_guide/development_lifecycle/failure_node.md index 9bf7e2dd52..3e3cab7149 100644 --- a/docs/user_guide/development_lifecycle/failure_node.md +++ b/docs/user_guide/development_lifecycle/failure_node.md @@ -20,21 +20,21 @@ To address this issue, you can add a failure node into your workflow. This ensur To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/failure_node.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/failure_node.py :caption: development_lifecycle/failure_node.py :lines: 1-6 ``` Create a task that will fail during execution: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/failure_node.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/failure_node.py :caption: development_lifecycle/failure_node.py :lines: 10-18 ``` Create a task that will be executed if any of the tasks in the workflow fail: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/failure_node.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/failure_node.py :caption: development_lifecycle/failure_node.py :pyobject: clean_up ``` @@ -45,21 +45,21 @@ Specify the `on_failure` to a cleanup task. This task will be executed if any of The input of `clean_up` should be the exact same as the input of the workflow. ::: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/failure_node.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/failure_node.py :caption: development_lifecycle/failure_node.py :pyobject: subwf ``` By setting the failure policy to `FAIL_AFTER_EXECUTABLE_NODES_COMPLETE` to ensure that the `wf1` is executed even if the subworkflow fails. In this case, both parent and child workflows will fail, resulting in the `clean_up` task being executed twice: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/failure_node.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/failure_node.py :caption: development_lifecycle/failure_node.py :lines: 42-53 ``` You can also set the `on_failure` to a workflow. This workflow will be executed if any of the tasks in the workflow fail: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/development_lifecycle/development_lifecycle/failure_node.py +```{literalinclude} /examples/development_lifecycle/development_lifecycle/failure_node.py :caption: development_lifecycle/failure_node.py :pyobject: wf2 ``` diff --git a/docs/user_guide/extending/custom_types.md b/docs/user_guide/extending/custom_types.md index a9670e9e8f..92a2ab5a19 100644 --- a/docs/user_guide/extending/custom_types.md +++ b/docs/user_guide/extending/custom_types.md @@ -27,7 +27,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte First, we import the dependencies: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/0ec8388759d34566a0ffc0c3c2d7443fd4a3a46f/examples/extending/extending/custom_types.py +```{literalinclude} /examples/extending/extending/custom_types.py :caption: extending/custom_types.py :lines: 1-7 ``` @@ -38,7 +38,7 @@ First, we import the dependencies: Defined type here represents a list of files on the disk. We will refer to it as `MyDataset`. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/0ec8388759d34566a0ffc0c3c2d7443fd4a3a46f/examples/extending/extending/custom_types.py +```{literalinclude} /examples/extending/extending/custom_types.py :caption: extending/custom_types.py :pyobject: MyDataset ``` @@ -53,7 +53,7 @@ The `TypeTransformer` is a Generic abstract base class. The `Generic` type argum that we want to work with. In this case, it is the `MyDataset` object. ::: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/0ec8388759d34566a0ffc0c3c2d7443fd4a3a46f/examples/extending/extending/custom_types.py +```{literalinclude} /examples/extending/extending/custom_types.py :caption: extending/custom_types.py :pyobject: MyDatasetTransformer ``` @@ -61,23 +61,23 @@ that we want to work with. In this case, it is the `MyDataset` object. Before we can use MyDataset in our tasks, we need to let Flytekit know that `MyDataset` should be considered as a valid type. This is done using {py:class}`~flytekit:flytekit.extend.TypeEngine`'s `register` method. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/0ec8388759d34566a0ffc0c3c2d7443fd4a3a46f/examples/extending/extending/custom_types.py +```{literalinclude} /examples/extending/extending/custom_types.py :caption: extending/custom_types.py -:lines: 87 +:lines: 86 ``` The new type should be ready to use! Let us write an example generator and consumer for this new datatype. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/0ec8388759d34566a0ffc0c3c2d7443fd4a3a46f/examples/extending/extending/custom_types.py +```{literalinclude} /examples/extending/extending/custom_types.py :caption: extending/custom_types.py -:lines: 91-114 +:lines: 90-114 ``` This workflow can be executed and tested locally. Flytekit will exercise the entire path even if you run it locally. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/0ec8388759d34566a0ffc0c3c2d7443fd4a3a46f/examples/extending/extending/custom_types.py +```{literalinclude} /examples/extending/extending/custom_types.py :caption: extending/custom_types.py -:lines: 119-120 +:lines: 118-119 ``` [flytesnacks]: https://github.com/flyteorg/flytesnacks/tree/0ec8388759d34566a0ffc0c3c2d7443fd4a3a46f/examples/extending/ diff --git a/docs/user_guide/extending/user_container_task_plugins.md b/docs/user_guide/extending/user_container_task_plugins.md index 99a3adf155..444ad9d646 100644 --- a/docs/user_guide/extending/user_container_task_plugins.md +++ b/docs/user_guide/extending/user_container_task_plugins.md @@ -32,7 +32,7 @@ def wait_and_run(path: str) -> int: return do_next(path=path) ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/extending/extending/user_container.py +```{literalinclude} /examples/extending/extending/user_container.py :caption: extending/user_container.py :lines: 1-6 ``` @@ -42,7 +42,7 @@ def wait_and_run(path: str) -> int: As illustrated above, to achieve this structure we need to create a class named `WaitForObjectStoreFile`, which derives from {py:class}`flytekit.PythonFunctionTask` as follows. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/extending/extending/user_container.py +```{literalinclude} /examples/extending/extending/user_container.py :caption: extending/user_container.py :pyobject: WaitForObjectStoreFile ``` @@ -68,14 +68,14 @@ Refer to the [spark plugin](https://github.com/flyteorg/flytekit/tree/master/plu ### Actual usage -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/extending/extending/user_container.py +```{literalinclude} /examples/extending/extending/user_container.py :caption: extending/user_container.py :lines: 54-69 ``` And of course, you can run the workflow locally using your own new shiny plugin! -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/extending/extending/user_container.py +```{literalinclude} /examples/extending/extending/user_container.py :caption: extending/user_container.py :lines: 73-78 ``` diff --git a/docs/user_guide/productionizing/customizing_task_resources.md b/docs/user_guide/productionizing/customizing_task_resources.md index 6ba07a604b..da885ec583 100644 --- a/docs/user_guide/productionizing/customizing_task_resources.md +++ b/docs/user_guide/productionizing/customizing_task_resources.md @@ -36,35 +36,35 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte Import the dependencies: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/customizing_resources.py +```{literalinclude} /examples/productionizing/productionizing/customizing_resources.py :caption: productionizing/customizing_resources.py :lines: 1-3 ``` Define a task and configure the resources to be allocated to it: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/customizing_resources.py +```{literalinclude} /examples/productionizing/productionizing/customizing_resources.py :caption: productionizing/customizing_resources.py :pyobject: count_unique_numbers ``` Define a task that computes the square of a number: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/customizing_resources.py +```{literalinclude} /examples/productionizing/productionizing/customizing_resources.py :caption: productionizing/customizing_resources.py :pyobject: square ``` You can use the tasks decorated with memory and storage hints like regular tasks in a workflow. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/customizing_resources.py +```{literalinclude} /examples/productionizing/productionizing/customizing_resources.py :caption: productionizing/customizing_resources.py :pyobject: my_workflow ``` You can execute the workflow locally. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/customizing_resources.py +```{literalinclude} /examples/productionizing/productionizing/customizing_resources.py :caption: productionizing/customizing_resources.py :lines: 32-34 ``` @@ -82,7 +82,7 @@ Let's understand how the resources can be initialized with an example. Import the dependencies. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/customizing_resources.py +```{literalinclude} /examples/productionizing/productionizing/customizing_resources.py :caption: productionizing/customizing_resources.py :lines: 38-40 ``` @@ -90,28 +90,28 @@ Import the dependencies. Define a task and configure the resources to be allocated to it. You can use tasks decorated with memory and storage hints like regular tasks in a workflow. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/customizing_resources.py +```{literalinclude} /examples/productionizing/productionizing/customizing_resources.py :caption: productionizing/customizing_resources.py :pyobject: count_unique_numbers ``` Define a task that computes the square of a number: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/customizing_resources.py +```{literalinclude} /examples/productionizing/productionizing/customizing_resources.py :caption: productionizing/customizing_resources.py :pyobject: square_1 ``` The `with_overrides` method overrides the old resource allocations: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/customizing_resources.py +```{literalinclude} /examples/productionizing/productionizing/customizing_resources.py :caption: productionizing/customizing_resources.py :pyobject: my_pipeline ``` You can execute the workflow locally: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/customizing_resources.py +```{literalinclude} /examples/productionizing/productionizing/customizing_resources.py :caption: productionizing/customizing_resources.py :lines: 65-67 ``` diff --git a/docs/user_guide/productionizing/reference_launch_plans.md b/docs/user_guide/productionizing/reference_launch_plans.md index bce75e4681..79b4142baf 100644 --- a/docs/user_guide/productionizing/reference_launch_plans.md +++ b/docs/user_guide/productionizing/reference_launch_plans.md @@ -16,7 +16,7 @@ Reference launch plans cannot be run locally. You must mock them out. To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/reference_launch_plan.py +```{literalinclude} /examples/productionizing/productionizing/reference_launch_plan.py :caption: productionizing/reference_launch_plan.py :lines: 1-36 ``` diff --git a/docs/user_guide/productionizing/reference_tasks.md b/docs/user_guide/productionizing/reference_tasks.md index d91ecd4bbc..0d611608a6 100644 --- a/docs/user_guide/productionizing/reference_tasks.md +++ b/docs/user_guide/productionizing/reference_tasks.md @@ -16,7 +16,7 @@ Reference tasks cannot be run locally. You must mock them out. To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/reference_task.py +```{literalinclude} /examples/productionizing/productionizing/reference_task.py :caption: productionizing/reference_task.py :lines: 1-36 ``` diff --git a/docs/user_guide/productionizing/schedules.md b/docs/user_guide/productionizing/schedules.md index fa0a6eedb0..6d28906073 100644 --- a/docs/user_guide/productionizing/schedules.md +++ b/docs/user_guide/productionizing/schedules.md @@ -25,7 +25,7 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte Consider the following example workflow: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/lp_schedules.py +```{literalinclude} /examples/productionizing/productionizing/lp_schedules.py :caption: productionizing/lp_schedules.py :lines: 1-14 ``` @@ -39,7 +39,7 @@ The `date_formatter_wf` workflow can be scheduled using either the `CronSchedule [Cron](https://en.wikipedia.org/wiki/Cron) expression strings use this {ref}`syntax `. An incorrect cron schedule expression would lead to failure in triggering the schedule. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/lp_schedules.py +```{literalinclude} /examples/productionizing/productionizing/lp_schedules.py :caption: productionizing/lp_schedules.py :lines: 17-29 ``` @@ -54,7 +54,7 @@ If you prefer to use an interval rather than a cron scheduler to schedule your w Here's an example: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/lp_schedules.py +```{literalinclude} /examples/productionizing/productionizing/lp_schedules.py :caption: productionizing/lp_schedules.py :lines: 34-57 ``` diff --git a/docs/user_guide/productionizing/secrets.md b/docs/user_guide/productionizing/secrets.md index 7eba15e653..538dc5d5ad 100644 --- a/docs/user_guide/productionizing/secrets.md +++ b/docs/user_guide/productionizing/secrets.md @@ -62,7 +62,7 @@ Once you've defined a secret on the Flyte backend, `flytekit` exposes a class called {py:class}`~flytekit.Secret`s, which allows you to request a secret from the configured secret manager: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/use_secrets.py +```{literalinclude} /examples/productionizing/productionizing/use_secrets.py :caption: productionizing/use_secrets.py :lines: 1-6, 49-53 ``` @@ -76,7 +76,7 @@ In the code below we specify two variables, `SECRET_GROUP` and `SECRET_NAME`, which maps onto the `user-info` secret that we created with `kubectl` above, with a key called `user_secret`. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/use_secrets.py +```{literalinclude} /examples/productionizing/productionizing/use_secrets.py :caption: productionizing/use_secrets.py :lines: 66-67 ``` @@ -92,7 +92,7 @@ invoking the {py:func}`flytekit.current_context` function, as shown below. At runtime, flytekit looks inside the task pod for an environment variable or a mounted file with a predefined name/path and loads the value. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/use_secrets.py +```{literalinclude} /examples/productionizing/productionizing/use_secrets.py :caption: productionizing/use_secrets.py :pyobject: secret_task ``` @@ -127,14 +127,14 @@ the same secret: In this case, the secret group will be `user-info`, with three available secret keys: `user_secret`, `username`, and `password`: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/use_secrets.py +```{literalinclude} /examples/productionizing/productionizing/use_secrets.py :caption: productionizing/use_secrets.py :lines: 107-108 ``` The Secret structure allows passing two fields, matching the key and the group, as previously described: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/use_secrets.py +```{literalinclude} /examples/productionizing/productionizing/use_secrets.py :caption: productionizing/use_secrets.py :lines: 113-124 ``` @@ -155,14 +155,14 @@ In these scenarios you can specify the `mount_requirement=Secret.MountType.FILE` In the following example we force the mounting to be an environment variable: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/use_secrets.py +```{literalinclude} /examples/productionizing/productionizing/use_secrets.py :caption: productionizing/use_secrets.py :lines: 139-158 ``` These tasks can be used in your workflow as usual -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/use_secrets.py +```{literalinclude} /examples/productionizing/productionizing/use_secrets.py :caption: productionizing/use_secrets.py :pyobject: my_secret_workflow ``` @@ -172,7 +172,7 @@ These tasks can be used in your workflow as usual The simplest way to test secret accessibility is to export the secret as an environment variable. There are some helper methods available to do so: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/productionizing/productionizing/use_secrets.py +```{literalinclude} /examples/productionizing/productionizing/use_secrets.py :caption: productionizing/use_secrets.py :lines: 172-182 ``` diff --git a/docs/user_guide/testing/mocking_tasks.md b/docs/user_guide/testing/mocking_tasks.md index b95af69b14..eb1c396075 100644 --- a/docs/user_guide/testing/mocking_tasks.md +++ b/docs/user_guide/testing/mocking_tasks.md @@ -6,42 +6,42 @@ A lot of the tasks that you write you can run locally, but some of them you will To clone and run the example code on this page, see the [Flytesnacks repo][flytesnacks]. ``` -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/testing/testing/mocking.py +```{literalinclude} /examples/testing/testing/mocking.py :caption: testing/mocking.py :lines: 1-6 ``` This is a generic SQL task (and is by default not hooked up to any datastore nor handled by any plugin), and must be mocked: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/testing/testing/mocking.py +```{literalinclude} /examples/testing/testing/mocking.py :caption: testing/mocking.py :lines: 10-16 ``` This is a task that can run locally: -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/testing/testing/mocking.py +```{literalinclude} /examples/testing/testing/mocking.py :caption: testing/mocking.py :pyobject: t1 ``` Declare a workflow that chains these two tasks together. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/testing/testing/mocking.py +```{literalinclude} /examples/testing/testing/mocking.py :caption: testing/mocking.py :pyobject: my_wf ``` Without a mock, calling the workflow would typically raise an exception, but with the `task_mock` construct, which returns a `MagicMock` object, we can override the return value. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/testing/testing/mocking.py +```{literalinclude} /examples/testing/testing/mocking.py :caption: testing/mocking.py :pyobject: main_1 ``` There is another utility as well called `patch` which offers the same functionality, but in the traditional Python patching style, where the first argument is the `MagicMock` object. -```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/testing/testing/mocking.py +```{literalinclude} /examples/testing/testing/mocking.py :caption: testing/mocking.py :lines: 45-56 ```