Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix example DAG in the README and docs/index #705

Merged
merged 7 commits into from
Nov 23, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 9 additions & 36 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -31,64 +31,37 @@ Run your dbt Core projects as `Apache Airflow <https://airflow.apache.org/>`_ DA
Quickstart
__________

Check out the Quickstart guide on our `docs <https://astronomer.github.io/astronomer-cosmos/#quickstart>`_. See more examples at `/dev/dags <https://github.com/astronomer/astronomer-cosmos/tree/main/dev/dags>`_.
Check out the Quickstart guide on our `docs <https://astronomer.github.io/astronomer-cosmos/#quickstart>`_. See more examples at `/dev/dags <https://github.com/astronomer/astronomer-cosmos/tree/main/dev/dags>`_ and at the `cosmos-demo repo <https://github.com/astronomer/cosmos-demo>`_.


Example Usage
___________________

You can render an Airflow Task Group using the ``DbtTaskGroup`` class. Here's an example with the `jaffle_shop project <https://github.com/dbt-labs/jaffle_shop>`_:
You can render a Cosmos Airflow DAG using the ``DbtDag`` class. Here's an example with the `jaffle_shop project <https://github.com/dbt-labs/jaffle_shop>`_:

..
This renders on Github but not Sphinx:

.. code-block:: python
https://github.com/astronomer/astronomer-cosmos/blob/24aa38e528e299ef51ca6baf32f5a6185887d432/dev/dags/basic_cosmos_dag.py#L1-L42

from pendulum import datetime
This will generate an Airflow DAG that looks like this:

from airflow import DAG
from airflow.operators.empty import EmptyOperator
from cosmos import DbtTaskGroup, ProfileConfig, ProjectConfig
from cosmos.profiles import PostgresUserPasswordProfileMapping
.. figure:: /docs/_static/jaffle_shop_dag.png

profile_config = ProfileConfig(
profile_name="default",
target_name="dev",
profile_mapping=PostgresUserPasswordProfileMapping(
conn_id="airflow_db",
profile_args={"schema": "public"},
),
)

with DAG(
dag_id="extract_dag",
start_date=datetime(2022, 11, 27),
schedule_interval="@daily",
):
e1 = EmptyOperator(task_id="pre_dbt")

dbt_tg = DbtTaskGroup(
project_config=ProjectConfig("jaffle_shop"),
profile_config=profile_config,
)

e2 = EmptyOperator(task_id="post_dbt")

e1 >> dbt_tg >> e2

This will generate an Airflow Task Group that looks like this:

.. figure:: /docs/_static/jaffle_shop_task_group.png

Community
_________
- Join us on the Airflow `Slack <https://join.slack.com/t/apache-airflow/shared_invite/zt-1zy8e8h85-es~fn19iMzUmkhPwnyRT6Q>`_ at #airflow-dbt


Changelog
_________

We follow `Semantic Versioning <https://semver.org/>`_ for releases.
Check `CHANGELOG.rst <https://github.com/astronomer/astronomer-cosmos/blob/main/CHANGELOG.rst>`_
for the latest changes.


Contributing Guide
__________________

Expand Down
Binary file added docs/_static/jaffle_shop_dag.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
48 changes: 10 additions & 38 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -40,53 +40,25 @@ Run your dbt Core projects as `Apache Airflow <https://airflow.apache.org/>`_ DA
Example Usage
___________________

You can render an Airflow Task Group using the ``DbtTaskGroup`` class. Here's an example with the jaffle_shop project:
You can render a Cosmos Airflow DAG using the ``DbtDag`` class. Here's an example with the `jaffle_shop project <https://github.com/dbt-labs/jaffle_shop>`_:

.. code-block:: python
..
The following renders in Sphinx but not Github:

from pendulum import datetime
.. literalinclude:: ./dev/dags/basic_cosmos_dag.py
:language: python
:start-after: [START local_example]
:end-before: [END local_example]

from airflow import DAG
from airflow.operators.empty import EmptyOperator
from cosmos import DbtTaskGroup

This will generate an Airflow DAG that looks like this:

profile_config = ProfileConfig(
profile_name="default",
target_name="dev",
profile_mapping=PostgresUserPasswordProfileMapping(
conn_id="airflow_db",
profile_args={"schema": "public"},
),
)

with DAG(
dag_id="extract_dag",
start_date=datetime(2022, 11, 27),
schedule_interval="@daily",
):
e1 = EmptyOperator(task_id="pre_dbt")

dbt_tg = DbtTaskGroup(
project_config=ProjectConfig("jaffle_shop"),
profile_config=profile_config,
default_args={"retries": 2},
)

e2 = EmptyOperator(task_id="post_dbt")

e1 >> dbt_tg >> e2


This will generate an Airflow Task Group that looks like this:

.. image:: https://raw.githubusercontent.com/astronomer/astronomer-cosmos/main/docs/_static/jaffle_shop_task_group.png

.. figure:: /docs/_static/jaffle_shop_dag.png

Getting Started
_______________

To get started now, check out the `Getting Started Guide <getting_started/index.html>`_.
Check out the Quickstart guide on our `docs <https://astronomer.github.io/astronomer-cosmos/#quickstart>`_. See more examples at `/dev/dags <https://github.com/astronomer/astronomer-cosmos/tree/main/dev/dags>`_ and at the `cosmos-demo repo <https://github.com/astronomer/cosmos-demo>`_.


Changelog
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,5 @@ pydata-sphinx-theme
sphinx-autobuild
sphinx-autoapi
apache-airflow
apache-airflow-providers-cncf-kubernetes>=5.1.1
openlineage-airflow
4 changes: 3 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,8 @@ docs =[
"sphinx",
"pydata-sphinx-theme",
"sphinx-autobuild",
"sphinx-autoapi"
"sphinx-autoapi",
"apache-airflow-providers-cncf-kubernetes>=5.1.1"
]
tests = [
"packaging",
Expand Down Expand Up @@ -227,6 +228,7 @@ dependencies = [
"sphinx-autobuild",
"sphinx-autoapi",
"openlineage-airflow",
"apache-airflow-providers-cncf-kubernetes>=5.1.1"
]

[tool.hatch.envs.docs.scripts]
Expand Down
Loading