Skip to content

Commit

Permalink
Doc updates (#336)
Browse files Browse the repository at this point in the history
  • Loading branch information
yanmastin-astro authored Jan 3, 2025
1 parent 823ca8d commit 5588811
Show file tree
Hide file tree
Showing 4 changed files with 15 additions and 17 deletions.
18 changes: 8 additions & 10 deletions docs/comparison/taskflow_api.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,17 @@
For users that employ lots of Python functions in their DAGs, [TaskFlow API](https://www.astronomer.io/docs/learn/airflow-decorators/) represent a simpler way to transform functions into tasks, with a more intuitive way of passing data between them.
They were introduced in Airflow 2 as an alternative to Airflow [traditional operators](traditional_operators.md).

Below, we illustrate how to represent an Airflow DAG using TaskFlow API and how to define the same DAG using
The following section shows how to represent an Airflow DAG using TaskFlow API and how to define the same DAG using
DAG Factory. Ultimately, both implementations use the same Airflow operators. The main difference is the language used
to declare the workflow: while one uses Python, the other uses YAML.
to declare the workflow: one uses Python and the other uses YAML.


## Goal

Let's say we'd like to create a workflow that performs the following:

1. Create a list of [PyPI](https://pypi.org/) projects to be analysed
2. Fetch the [statistics](https://pypistats.org/) for each of these projects
1. Create a list of [PyPI](https://pypi.org/) projects to be analysed.
2. Fetch the [statistics](https://pypistats.org/) for each of these projects.
3. Summarize the selected statistics as Markdown, using Python.

We will implement all these steps using the Airflow `task` decorator, and the last task will generate a Markdown table similar to:
Expand Down Expand Up @@ -58,9 +58,8 @@ Both implementations accomplish the same goal and result in the expected Markdow

### Airflow Graph view

As it can be seen in the screenshots below, both the DAG created using Python with standard Airflow and the
DAG created using YAML and DAG Factory look identical, from a graph topology perspective, and also from the underlining
operators being used.
As shown in the screenshots below, both the DAG created using Python with standard Airflow and the
DAG created using YAML and DAG Factory look identical, from a graph topology perspective, and also from the underlining operators being used.

#### Graph view: Plain Airflow Python DAG

Expand All @@ -72,7 +71,7 @@ operators being used.

### Airflow Dynamic Task Mapping

In both workflows, we are generating dynamically a task for each PyPI repo.
In both workflows, we are dynamically generating a task for each PyPI repo.

#### Mapped Tasks: Plain Airflow Python DAG

Expand All @@ -99,7 +98,6 @@ From an Airflow UI perspective, the content displayed in the "Code" view is the

![alt text](../static/example_pypi_stats_dagfactory_code.png "YAML DAG code visualization")

To overcome this limitation, DAG Factory appends the YAML content to the DAG Documentation so users can better troubleshoot
the DAG:
To overcome this limitation, DAG Factory appends the YAML content to the DAG Documentation so users can better troubleshoot the DAG:

![alt text](../static/example_pypi_stats_dagfactory_docs.png "YAML DAG docs visualization")
4 changes: 2 additions & 2 deletions docs/comparison/traditional_operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Most of the Airflow providers come with built-in traditional operators. Some exa

Below, we illustrate how to represent an Airflow DAG using traditional operators and how to define the same DAG using
DAG Factory. Ultimately, both implementations use the same Airflow operators. The main difference is the language used
to declare the workflow: while one uses Python, the other uses YAML.
to declare the workflow: one uses Python and the other uses YAML.


## Goal
Expand Down Expand Up @@ -61,7 +61,7 @@ Both implementations accomplish the same goal and result in the expected Markdow

### Airflow Graph view

As it can be seen in the screenshots below, both the DAG created using Python with standard Airflow and the
As shown in the screenshots below, both the DAG created using Python with standard Airflow and the
DAG created using YAML and DAG Factory look identical, from a graph topology perspective, and also from the underlining
operators being used.

Expand Down
6 changes: 3 additions & 3 deletions docs/getting-started/quick-start-airflow-standalone.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ The minimum requirements for **dag-factory** are:

## Step 1: Create a Python Virtual Environment

First, create and activate a virtual environment:
Create and activate a virtual environment:

```commandline
python3 -m venv dagfactory_env
Expand All @@ -20,7 +20,7 @@ source dagfactory_env/bin/activate

## Step 2: Install Apache Airflow

Next, you'll need to install [Apache Airflow®](https://airflow.apache.org). Follow these steps:
Install [Apache Airflow®](https://airflow.apache.org):

1. Create a directory for your project and navigate to it:

Expand All @@ -43,7 +43,7 @@ Next, you'll need to install [Apache Airflow®](https://airflow.apache.org). Fol
## Step 3: Install DAG Factory
Now, install the DAG Factory library in your virtual environment:
Install the DAG Factory library in your virtual environment:
```commandline
pip install dag-factory
Expand Down
4 changes: 2 additions & 2 deletions docs/getting-started/quick-start-astro-cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ The minimum requirements for **dag-factory** are:

## Step 1: Initialize Airflow Project

Start by creating a new directory and initializing your Astro CLI project:
Create a new directory and initialize your Astro CLI project:

```commandline
mkdir dag-factory-quick-start && cd dag-factory-quick-start
Expand All @@ -23,7 +23,7 @@ This will set up the necessary Airflow files and directories.

## Step 2: Install DAG Factory

To install DAG Factory in your Airflow environment:
Install DAG Factory in your Airflow environment:

1. Add dag-factory as a dependency to the `requirements.txt` file created during the project initialization.

Expand Down

0 comments on commit 5588811

Please sign in to comment.