Skip to content

Commit

Permalink
Update docs regarding docs plugin in Astronomer cloud (#874)
Browse files Browse the repository at this point in the history
Resolves #871

Turns out, there is a lot of nuance in regards to how Astronomer
interacts with stuff under the hood.
  • Loading branch information
dwreeves authored Mar 6, 2024
1 parent 9e32ef7 commit 82c8db9
Showing 1 changed file with 19 additions and 1 deletion.
20 changes: 19 additions & 1 deletion docs/configuration/hosting-docs.rst
Original file line number Diff line number Diff line change
Expand Up @@ -105,11 +105,29 @@ For example, if your dbt project directory is ``/usr/local/airflow/dags/my_dbt_p
AIRFLOW__COSMOS__DBT_DOCS_DIR="/usr/local/airflow/dags/my_dbt_project/target"
Using docs out of local storage has the downside that some values in the dbt docs can become stale unless the docs are periodically refreshed and redeployed:
Using docs out of local storage has a couple downsides. First, some values in the dbt docs can become stale, unless the docs are periodically refreshed and redeployed:

- Counts of the numbers of rows.
- The compiled SQL for incremental models before and after the first run.

Second, deployment from local storage may only be partially compatible with some managed Airflow systems.
Compatibility will depend on the managed Airflow system, as each one works differently.

For example, Astronomer does not update the resources available to the webserver instance when ``--dags`` is specified during deployment, meaning that the dbt dcs will not be updated when this flag is used.

.. note::
Managed Airflow on Astronomer Cloud does not provide the webserver access to the DAGs folder.
If you want to host your docs in local storage with Astro, you should host them in a directory other than ``dags/``.
For example, you can set your ``AIRFLOW__COSMOS__DBT_DOCS_DIR`` to ``/usr/local/airflow/dbt_docs_dir`` with the following pre-deployment script:

.. code-block:: bash
dbt docs generate
mkdir dbt_docs_dir
cp dags/dbt/target/manifest.json dbt_docs_dir/manifest.json
cp dags/dbt/target/catalog.json dbt_docs_dir/catalog.json
cp dags/dbt/target/index.html dbt_docs_dir/index.html
Host from HTTP/HTTPS
~~~~~~~~~~~~~~~~~~~~

Expand Down

0 comments on commit 82c8db9

Please sign in to comment.