Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix code to work with mashumaro 3.15 #11051

Open
wants to merge 45 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 37 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
261e124
Fix NodeVersion definition, make_semantic_model utility
gshank Nov 25, 2024
37be156
Bump up mashumaro version again
gshank Nov 25, 2024
52fa7c0
Try pinning to >=3.15,<4.0
gshank Nov 26, 2024
354af6e
specify dbt-adapters branch
gshank Nov 26, 2024
7876edb
Update v12.json schema
gshank Nov 26, 2024
c0ea26e
validate in "update_from"
gshank Nov 26, 2024
dfca4e5
error classes
gshank Nov 26, 2024
b81632e
Comments
gshank Nov 27, 2024
e47bed3
Put back test_disabled_model.py
gshank Nov 27, 2024
9d6daa1
Use older mashumaro
gshank Nov 27, 2024
baf42c4
Again
gshank Nov 27, 2024
9df05a8
Put back default for validate in _update_from_config
gshank Nov 27, 2024
524ce54
Fix NodeVersion in artifacts test
gshank Nov 27, 2024
df23c7d
Skip a couple of tests until mash 3.15
gshank Nov 30, 2024
42a1afb
Make specialized 'calculate_node_config' methods
gshank Nov 30, 2024
9c6883b
Change order of Union in source definitions partitions
gshank Dec 2, 2024
25899ca
Change order of external.partitions schema elements
gshank Dec 2, 2024
5b4af17
Changie
gshank Dec 2, 2024
86791e0
Remove unnecessary "base" parameter
gshank Dec 2, 2024
d5616de
passes unit tests
gshank Dec 2, 2024
4554eb3
comment
gshank Dec 2, 2024
b4f72e4
Use dbt-common branch
gshank Dec 3, 2024
54adabe
Fix a few tests, remove unnecessary finalize_and_validate calls
gshank Dec 3, 2024
bbfc6e6
rename config to context_config
gshank Dec 3, 2024
1afb2bb
fix hooks before validation
gshank Dec 3, 2024
7d78b47
Bump mashumaro again
gshank Dec 3, 2024
1c709d2
Remove skips from tests
gshank Dec 3, 2024
13105cd
fix expected_manifest version: "2"
gshank Dec 3, 2024
3f7ee0e
Remove unnecessary ConfigSource
gshank Dec 3, 2024
c16cbe0
Rename ContextConfig ConfigBuilder
gshank Dec 3, 2024
c0fd389
rename model_configs resource_configs
gshank Dec 3, 2024
049418e
More cleanup
gshank Dec 3, 2024
e6fc9b0
Remove unnecessary mangle_hooks, other comments and cleanup
gshank Dec 3, 2024
31ce086
More comments
gshank Dec 4, 2024
923b8b6
add test for serialization of ExternalPartitions in ExternalTable
gshank Dec 9, 2024
1a30f64
Update test_serialization to simplify for mashumaro, add test_graph_s…
gshank Dec 10, 2024
c137041
Merge branch 'main' into mashumaro_fixes
gshank Dec 10, 2024
f011789
Remove base parameter from get_config_for
gshank Dec 11, 2024
0fb57ee
Update return type of generate_node_config
gshank Dec 11, 2024
e919144
Merge branch 'main' into mashumaro_fixes
gshank Dec 12, 2024
ebc2359
Remove requirements for 3.15
gshank Dec 12, 2024
4585bd0
formatting
gshank Dec 16, 2024
715ad2f
dev-requirements
gshank Dec 19, 2024
d1274b1
Merge branch 'main' into mashumaro_fixes
gshank Dec 19, 2024
5f94de5
Put back Union order changes, use dbtClassMixin in test_serialization.py
gshank Dec 19, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20241202-115445.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: Support upgrading mashumaro to 3.15
time: 2024-12-02T11:54:45.103325-05:00
custom:
Author: gshank
Issue: "11044"
2 changes: 1 addition & 1 deletion core/dbt/artifacts/resources/v1/components.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
from dbt_common.dataclass_schema import ExtensibleDbtClassMixin, dbtClassMixin
from dbt_semantic_interfaces.type_enums import TimeGranularity

NodeVersion = Union[str, float]
NodeVersion = Union[int, float, str]


@dataclass
Expand Down
6 changes: 0 additions & 6 deletions core/dbt/artifacts/resources/v1/snapshot.py
Original file line number Diff line number Diff line change
Expand Up @@ -71,12 +71,6 @@ def final_validate(self):
if self.materialized and self.materialized != "snapshot":
raise ValidationError("A snapshot must have a materialized value of 'snapshot'")

# Called by "calculate_node_config_dict" in ContextConfigGenerator
def finalize_and_validate(self):
data = self.to_dict(omit_none=True)
self.validate(data)
return self.from_dict(data)


@dataclass
class Snapshot(CompiledResource):
Expand Down
2 changes: 1 addition & 1 deletion core/dbt/artifacts/resources/v1/source_definition.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ class ExternalTable(AdditionalPropertiesAllowed, Mergeable):
file_format: Optional[str] = None
row_format: Optional[str] = None
tbl_properties: Optional[str] = None
partitions: Optional[Union[List[str], List[ExternalPartition]]] = None
partitions: Optional[Union[List[ExternalPartition], List[str]]] = None

def __bool__(self):
return self.location is not None
Expand Down
296 changes: 150 additions & 146 deletions core/dbt/context/context_config.py

Large diffs are not rendered by default.

40 changes: 20 additions & 20 deletions core/dbt/context/providers.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
from dbt.constants import DEFAULT_ENV_PLACEHOLDER
from dbt.context.base import Var, contextmember, contextproperty
from dbt.context.configured import FQNLookup
from dbt.context.context_config import ContextConfig
from dbt.context.context_config import ConfigBuilder
from dbt.context.exceptions_jinja import wrapped_exports
from dbt.context.macro_resolver import MacroResolver, TestMacroNamespace
from dbt.context.macros import MacroNamespace, MacroNamespaceBuilder
Expand Down Expand Up @@ -367,14 +367,14 @@


class Config(Protocol):
def __init__(self, model, context_config: Optional[ContextConfig]): ...
def __init__(self, model, config_builder: Optional[ConfigBuilder]): ...


# Implementation of "config(..)" calls in models
class ParseConfigObject(Config):
def __init__(self, model, context_config: Optional[ContextConfig]):
def __init__(self, model, config_builder: Optional[ConfigBuilder]):
self.model = model
self.context_config = context_config
self.config_builder = config_builder

def _transform_config(self, config):
for oldkey in ("pre_hook", "post_hook"):
Expand All @@ -395,19 +395,19 @@

opts = self._transform_config(opts)

# it's ok to have a parse context with no context config, but you must
# it's ok to have a parse context with no config builder, but you must
# not call it!
if self.context_config is None:
raise DbtRuntimeError("At parse time, did not receive a context config")
if self.config_builder is None:
raise DbtRuntimeError("At parse time, did not receive a config builder")

Check warning on line 401 in core/dbt/context/providers.py

View check run for this annotation

Codecov / codecov/patch

core/dbt/context/providers.py#L401

Added line #L401 was not covered by tests

# Track unrendered opts to build parsed node unrendered_config later on
if get_flags().state_modified_compare_more_unrendered_values:
unrendered_config = statically_parse_unrendered_config(self.model.raw_code)
if unrendered_config:
self.context_config.add_unrendered_config_call(unrendered_config)
self.config_builder.add_unrendered_config_call(unrendered_config)

# Use rendered opts to populate context_config
self.context_config.add_config_call(opts)
# Use rendered opts to populate config builder
self.config_builder.add_config_call(opts)
return ""

def set(self, name, value):
Expand All @@ -427,7 +427,7 @@


class RuntimeConfigObject(Config):
def __init__(self, model, context_config: Optional[ContextConfig] = None):
def __init__(self, model, config_builder: Optional[ConfigBuilder] = None):
self.model = model
# we never use or get a config, only the parser cares

Expand Down Expand Up @@ -887,7 +887,7 @@
config: RuntimeConfig,
manifest: Manifest,
provider: Provider,
context_config: Optional[ContextConfig],
config_builder: Optional[ConfigBuilder],
) -> None:
if provider is None:
raise DbtInternalError(f"Invalid provider given to context: {provider}")
Expand All @@ -896,7 +896,7 @@
self.model: Union[Macro, ManifestNode] = model
super().__init__(config, manifest, model.package_name)
self.sql_results: Dict[str, Optional[AttrDict]] = {}
self.context_config: Optional[ContextConfig] = context_config
self.config_builder: Optional[ConfigBuilder] = config_builder
self.provider: Provider = provider
self.adapter = get_adapter(self.config)
# The macro namespace is used in creating the DatabaseWrapper
Expand Down Expand Up @@ -1165,7 +1165,7 @@
{%- set unique_key = config.require('unique_key') -%}
...
""" # noqa
return self.provider.Config(self.model, self.context_config)
return self.provider.Config(self.model, self.config_builder)

@contextproperty()
def execute(self) -> bool:
Expand Down Expand Up @@ -1689,12 +1689,12 @@
model: ManifestNode,
config: RuntimeConfig,
manifest: Manifest,
context_config: ContextConfig,
config_builder: ConfigBuilder,
) -> Dict[str, Any]:
# The __init__ method of ModelContext also initializes
# a ManifestContext object which creates a MacroNamespaceBuilder
# which adds every macro in the Manifest.
ctx = ModelContext(model, config, manifest, ParseProvider(), context_config)
ctx = ModelContext(model, config, manifest, ParseProvider(), config_builder)
# The 'to_dict' method in ManifestContext moves all of the macro names
# in the macro 'namespace' up to top level keys
return ctx.to_dict()
Expand Down Expand Up @@ -1896,14 +1896,14 @@
config: RuntimeConfig,
manifest: Manifest,
provider: Provider,
context_config: Optional[ContextConfig],
config_builder: Optional[ConfigBuilder],
macro_resolver: MacroResolver,
) -> None:
# this must be before super init so that macro_resolver exists for
# build_namespace
self.macro_resolver = macro_resolver
self.thread_ctx = MacroStack()
super().__init__(model, config, manifest, provider, context_config)
super().__init__(model, config, manifest, provider, config_builder)
self._build_test_namespace()
# We need to rebuild this because it's already been built by
# the ProviderContext with the wrong namespace.
Expand Down Expand Up @@ -1975,10 +1975,10 @@
model: ManifestNode,
config: RuntimeConfig,
manifest: Manifest,
context_config: ContextConfig,
config_builder: ConfigBuilder,
macro_resolver: MacroResolver,
) -> Dict[str, Any]:
ctx = TestContext(model, config, manifest, ParseProvider(), context_config, macro_resolver)
ctx = TestContext(model, config, manifest, ParseProvider(), config_builder, macro_resolver)
# The 'to_dict' method in ManifestContext moves all of the macro names
# in the macro 'namespace' up to top level keys
return ctx.to_dict()
11 changes: 1 addition & 10 deletions core/dbt/contracts/graph/model_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,14 +53,5 @@ class UnitTestNodeConfig(NodeConfig):
}


# base resource types are like resource types, except nothing has mandatory
# configs.
BASE_RESOURCE_TYPES: Dict[NodeType, Type[BaseConfig]] = RESOURCE_TYPES.copy()


def get_config_for(resource_type: NodeType, base=False) -> Type[BaseConfig]:
MichelleArk marked this conversation as resolved.
Show resolved Hide resolved
if base:
lookup = BASE_RESOURCE_TYPES
else:
lookup = RESOURCE_TYPES
return lookup.get(resource_type, NodeConfig)
return RESOURCE_TYPES.get(resource_type, NodeConfig)
Loading
Loading