Add pre-commit hook for McCabe max complexity check and fix errors

[jbandoro]

Description

Opening this up as a draft to work through the code that have been flagged as too complex given the default threshold of 10 below:

❯ pre-commit run flake8 --all-files
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

cosmos/dbt/graph.py:134:5: C901 'DbtGraph.load_via_dbt_ls' is too complex (16)
cosmos/dbt/parser/project.py:136:5: C901 'DbtModel.__post_init__' is too complex (18)
cosmos/dbt/parser/project.py:346:5: C901 'LegacyDbtProject._handle_config_file' is too complex (15)
cosmos/dbt/selector.py:87:1: C901 'select_nodes_ids_by_intersection' is too complex (16)

I'm happy to do this since it will help be get more familiar with some inner workings of cosmos, but if someone from the team wants to take over they're welcome.

If its lowered to 6 as mentioned in the issue, there are more errors that could be fixed:

❯ pre-commit run flake8 --all-files
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

cosmos/airflow/graph.py:196:1: C901 'build_airflow_graph' is too complex (9)
cosmos/config.py:66:5: C901 'ProjectConfig.__init__' is too complex (7)
cosmos/dbt/graph.py:134:5: C901 'DbtGraph.load_via_dbt_ls' is too complex (16)
cosmos/dbt/parser/project.py:63:5: C901 'DbtModelConfig._config_selector_ooo' is too complex (7)
cosmos/dbt/parser/project.py:98:1: C901 'extract_python_file_upstream_requirements' is too complex (7)
cosmos/dbt/parser/project.py:136:5: C901 'DbtModel.__post_init__' is too complex (18)
cosmos/dbt/parser/project.py:262:5: C901 'LegacyDbtProject.__post_init__' is too complex (10)
cosmos/dbt/parser/project.py:346:5: C901 'LegacyDbtProject._handle_config_file' is too complex (15)
cosmos/dbt/selector.py:52:5: C901 'SelectorConfig.load_from_statement' is too complex (7)
cosmos/dbt/selector.py:87:1: C901 'select_nodes_ids_by_intersection' is too complex (16)
cosmos/dbt/selector.py:170:1: C901 'select_nodes' is too complex (9)
cosmos/hooks/subprocess.py:34:5: C901 'FullOutputSubprocessHook.run_command' is too complex (8)
cosmos/operators/base.py:137:5: C901 'DbtBaseOperator.get_env' is too complex (7)
cosmos/operators/base.py:186:5: C901 'DbtBaseOperator.add_global_flags' is too complex (7)
cosmos/operators/local.py:136:5: C901 'DbtLocalBaseOperator.store_compiled_sql' is too complex (7)
cosmos/profiles/base.py:160:5: C901 'BaseProfileMapping.get_dbt_value' is too complex (8)
dev/dags/dbt_docs.py:45:1: C901 'which_upload' is too complex (7)

Related Issue(s)

Closes: #525

Breaking Change?

Checklist

• I have made corresponding changes to the documentation (if required)
• I have added tests that prove my fix is effective or that my feature works

[netlify]

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Name	Link
🔨 Latest commit	0295fce
🔍 Latest deploy log	https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/654180f917e778000864af24

[codecov]

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (58de67e) 93.38% compared to head (0295fce) 93.41%.
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #629      +/-   ##
==========================================
+ Coverage   93.38%   93.41%   +0.03%     
==========================================
  Files          53       53              
  Lines        2115     2158      +43     
==========================================
+ Hits         1975     2016      +41     
- Misses        140      142       +2     

Files	Coverage Δ
cosmos/dbt/graph.py	99.40% <98.41%> (+0.63%)	⬆️
cosmos/dbt/selector.py	96.96% <97.87%> (+0.38%)	⬆️
cosmos/dbt/parser/project.py	89.73% <85.13%> (-0.32%)	⬇️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tatiana]

@jbandoro Thank you for taking the time to work on the code complexity!
This type of task sometimes stays hanging in the backlog - but it's critical for the maintainability of the code. Keep up the fantastic work!

[jbandoro]

Apologies for the follow-up commits, I finally was able to get the integration tests to run completely in my local dev environment with hatch run tests.py3.10-2.6:test-integration and fixed some errors.

[tatiana]

This is outstanding work, @jbandoro. We're very grateful for this! I changed the scope of the original ticket, that stated the goal of accomplishing code complexity 6 to 10, and created a new ticket for us to reduce to 6: #641