Airflow: Remove top level connection retrievals in 4 DAGs

[gabrielwol]

You should avoid writing the top level code which is not necessary to create Operators and build DAG relations between them. This is because of the design decision for the scheduler of Airflow and the impact the top-level code parsing speed on both performance and scalability of Airflow.
...
Specifically you should not run any database access, heavy computations and networking operations.

[2024-12-16T16:31:30.370+0000] {base.py:84} INFO - Retrieving connection 'google_sheets_api'
[2024-12-16T16:31:30.590+0000] {base.py:84} INFO - Retrieving connection 'vz_api_bot'
[2024-12-16T16:31:31.087+0000] {base.py:84} INFO - Retrieving connection 'here_bot'
[2024-12-16T16:31:31.092+0000] {base.py:84} INFO - Retrieving connection 'here_bot'
[2024-12-16T16:31:31.097+0000] {base.py:84} INFO - Retrieving connection 'here_bot'
[2024-12-16T16:31:31.148+0000] {base.py:84} INFO - Retrieving connection 'here_bot'
[2024-12-16T16:31:31.152+0000] {base.py:84} INFO - Retrieving connection 'here_bot'
[2024-12-16T16:31:31.156+0000] {base.py:84} INFO - Retrieving connection 'here_bot'

[gabrielwol]

Successfully tested this:

    @task.bash()
    def task_conn()->str:
        conn = BaseHook.get_connection("here_bot")
        os.environ['HOST'] = conn.host
        os.environ['USER'] = conn.login
        os.environ['PGPASSWORD'] = conn.password
        return '''psql -h $HOST -U $USER -d bigdata -c "SELECT * FROM here.ta LIMIT 1;" '''

[2024-12-16T16:31:31.413+0000] {base.py:84} INFO - Retrieving connection 'here_bot'
[2024-12-16T16:31:31.450+0000] {subprocess.py:63} INFO - Tmp dir root location: /tmp
[2024-12-16T16:31:31.451+0000] {subprocess.py:75} INFO - Running command: ['/usr/bin/bash', '-c', 'psql -h $HOST -U $USER -d bigdata -c "SELECT * FROM here.ta LIMIT 1;" ']
[2024-12-16T16:31:31.470+0000] {subprocess.py:86} INFO - Output:
[2024-12-16T16:31:31.552+0000] {subprocess.py:93} INFO -   link_dir   |     dt     |   tod    |         tx          | length | mean | stddev | min_spd | max_spd | pct_50 | pct_85 | confidence | sample_size
...