Fix dbt project parsing/Airflow DAG rendering for load_method=LoadMode.CUSTOM

[tatiana]

As part of testing for the 1.1.2 release (#560), we realised that if the DAG uses load_method=LoadMode.CUSTOM, all the test nodes are removed from the DAG.

Example DAG, run from the Cosmos repo (dev/dags):

import os
from datetime import datetime
from pathlib import Path

from cosmos.constants import LoadMode
from cosmos import DbtDag, ProjectConfig, ProfileConfig, RenderConfig
from cosmos.profiles import PostgresUserPasswordProfileMapping

DEFAULT_DBT_ROOT_PATH = Path(__file__).parent / "dbt"
DBT_ROOT_PATH = Path(os.getenv("DBT_ROOT_PATH", DEFAULT_DBT_ROOT_PATH))

profile_config = ProfileConfig(
    profile_name="default",
    target_name="dev",
    profile_mapping=PostgresUserPasswordProfileMapping(
        conn_id="airflow_db",
        profile_args={"schema": "public"},
    ),
)

basic_cosmos_dag = DbtDag(
    # dbt/cosmos-specific parameters
    project_config=ProjectConfig(
        DBT_ROOT_PATH / "jaffle_shop",
    ),
    render_config=RenderConfig(
        load_method=LoadMode.CUSTOM,
    ),
    profile_config=profile_config,
    operator_args={"install_deps": True},
    # normal dag parameters
    schedule_interval="@daily",
    start_date=datetime(2023, 1, 1),
    catchup=False,
    dag_id="basic_cosmos_dag",
)

Rendered DAG with load_method=LoadMode.DBT_LS:

Same DAG rendered with load_method=LoadMode.CUSTOM:

This issue is a bug in the current implementation of Cosmos custom parsing of dbt projects, highlighted since we introduced #543.

Acceptance criteria

• The aforementioned DAG should render test nodes when using load_method=LoadMode.CUSTOM

[raffifu]

This issue happen because the implementaion of DbtProject not properly parsing the test, as far as i know the only way for custom parser to get the test is from reading *.yml file and scan if there's a tests key on the file. As you can see on the snippet below, the function is only implemented to scan yaml and grab the config. I would like to contribute on this issue

astronomer-cosmos/cosmos/dbt/parser/project.py

Lines 344 to 347 in 4c8d6b0

	def _handle_config_file(self, path: Path) -> None:
	"""
	Handles a single config file.
	"""

[tatiana]

Thank you very much, @raffifu ! I appreciate your contributions.

This issue has been there for a while, and the fact the coverage in this part of the code is limited certainly didn't help us surface it before.

Please, reach out if you're not able to look into this, and I'll check it out - otherwise, the ticket is yours 🎁

[tatiana]

This was released as part of 1.2:

• https://pypi.org/project/astronomer-cosmos/1.2.0/
• https://github.com/astronomer/astronomer-cosmos/releases/tag/astronomer-cosmos-v1.2.0