[ADAP-958] [Bug] dbt operations failing with ImportError: cannot import name 'POLLING_PREDICATE' from 'google.api_core.future.polling

[gmyrianthous]

Is this a new bug in dbt-bigquery?

• I believe this is a new bug in dbt-bigquery
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When trying to run dbt operations with dbt-core/bigquery 1.6.0+, I get the following error:

ImportError: cannot import name 'POLLING_PREDICATE' from 'google.api_core.future.polling' 

Expected Behavior

I would expect dbt operations to run without raising any exceptions.

Steps To Reproduce

1. Create a pyproject.toml file as follows:

[tool.poetry]
name = "dbt"

[tool.poetry.dependencies]
python = "~3.8.12"
dbt-core = "1.6.6"
dbt-bigquery = "1.6.6"

[tool.poetry.group.dev.dependencies]
sqlfluff = "1.4.5"
sqlfluff-templater-dbt = "1.4.5"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

2. Generate the lock file and install dependencies

poetry lock --no-update 
poetry install

3. Run dbt compile within the poetry environment

poetry run dbt compile --profiles-dir /path/to/profiles --target mytarget

Relevant log output

10:56:40  Running with dbt=1.6.6
10:56:40  Encountered an error:
cannot import name 'POLLING_PREDICATE' from 'google.api_core.future.polling' (/Users/gmyrianthous/virtualenvs/analytics-venv/lib/python3.8/site-packages/google/api_core/future/polling.py)
10:56:40  Traceback (most recent call last):
  File "/Users/gmyrianthous/virtualenvs/analytics-venv/lib/python3.8/site-packages/dbt/cli/requires.py", line 87, in wrapper
    result, success = func(*args, **kwargs)
  File "/Users/gmyrianthous/virtualenvs/analytics-venv/lib/python3.8/site-packages/dbt/cli/requires.py", line 72, in wrapper
    return func(*args, **kwargs)
  File "/Users/gmyrianthous/virtualenvs/analytics-venv/lib/python3.8/site-packages/dbt/cli/requires.py", line 140, in wrapper
    profile = load_profile(flags.PROJECT_DIR, flags.VARS, flags.PROFILE, flags.TARGET, threads)
  File "/Users/gmyrianthous/virtualenvs/analytics-venv/lib/python3.8/site-packages/dbt/config/runtime.py", line 70, in load_profile
    profile = Profile.render(
  File "/Users/gmyrianthous/virtualenvs/analytics-venv/lib/python3.8/site-packages/dbt/config/profile.py", line 436, in render
    return cls.from_raw_profiles(
  File "/Users/gmyrianthous/virtualenvs/analytics-venv/lib/python3.8/site-packages/dbt/config/profile.py", line 401, in from_raw_profiles
    return cls.from_raw_profile_info(
  File "/Users/gmyrianthous/virtualenvs/analytics-venv/lib/python3.8/site-packages/dbt/config/profile.py", line 355, in from_raw_profile_info
    credentials: Credentials = cls._credentials_from_profile(
  File "/Users/gmyrianthous/virtualenvs/analytics-venv/lib/python3.8/site-packages/dbt/config/profile.py", line 165, in _credentials_from_profile
    cls = load_plugin(typename)
  File "/Users/gmyrianthous/virtualenvs/analytics-venv/lib/python3.8/site-packages/dbt/adapters/factory.py", line 212, in load_plugin
    return FACTORY.load_plugin(name)
  File "/Users/gmyrianthous/virtualenvs/analytics-venv/lib/python3.8/site-packages/dbt/adapters/factory.py", line 58, in load_plugin
    mod: Any = import_module("." + name, "dbt.adapters")
  File "/usr/local/Cellar/[email protected]/3.8.18/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/gmyrianthous/virtualenvs/analytics-venv/lib/python3.8/site-packages/dbt/adapters/bigquery/__init__.py", line 5, in <module>
    from dbt.adapters.bigquery.impl import BigQueryAdapter, GrantTarget, PartitionConfig  # noqa
  File "/Users/gmyrianthous/virtualenvs/analytics-venv/lib/python3.8/site-packages/dbt/adapters/bigquery/impl.py", line 32, in <module>
    from dbt.adapters.bigquery.python_submissions import (
  File "/Users/gmyrianthous/virtualenvs/analytics-venv/lib/python3.8/site-packages/dbt/adapters/bigquery/python_submissions.py", line 4, in <module>
    from google.api_core.future.polling import POLLING_PREDICATE
ImportError: cannot import name 'POLLING_PREDICATE' from 'google.api_core.future.polling' (/Users/gmyrianthous/virtualenvs/analytics-venv/lib/python3.8/site-packages/google/api_core/future/polling.py)

Environment

- OS: OSX
- Python: 3.8.12
- dbt-core: 1.6.0, 1.6.6
- dbt-bigquery: 1.6.0, 1.6.6

Additional Context

No response

[gmyrianthous]

I believe this is related to the fact that we import POLLING_PREDICATE from google.api_core.future.polling but google.api_core.future.polling.POLLING_PREDICATE is available on google-api-core version 2.11.0+.

In my current setting, I've had google-api-core v2.8.2.

Therefore, we should ensure that this version is enforced and create a pin. In fact, the following set of dependencies work just fine:

[tool.poetry]
name = "dbt"

[tool.poetry.dependencies]
python = "~3.8.12"
dbt-core = "1.6.6"
dbt-bigquery = "1.6.6"
google-api-core = ">=2.11.0"

[tool.poetry.group.dev.dependencies]
sqlfluff = "1.4.5"
sqlfluff-templater-dbt = "1.4.5"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Let me know if you also agree and I can create a PR quickly to add google-api-core>=2.11.0 as a requirement.

[dbeatty10]

Let me know if you also agree and I can create a PR quickly to add google-api-core>=2.11.0 as a requirement.

Agreed @gmyrianthous!

Just raised a PR for this and added you as a co-author: #980