Ideas for how we might structure classes for composable, effective layers of abstraction

[nicholasyager]

We're quickly reaching the point of our work on dbt-meshify where it will be important to consider how we as engineers want to interact with our abstractions. Generally speaking, we want to design classes that are composable, and that provide a clear interface to the operations that we want to perform.

With this in mind, I propose that we consider DbtProjects to be the core abstraction that we then manipulate and perform logical operations on. For example:

project = LocalDbtProject('path/to/project')

# Split a new project off of the original project
new_project: LocalDbtProject = project.split(
    project_name='new_project',
    select='+marts.finance',
    exclude='reporting.*+
)

# Add a group to the project
new_project.add_group(
    name='finance',
    access='restricted',
    select='staging.*'
)

# Save the project to the file system.
new_project.save('path/to/new_project')

`

In order to accomplish this, we can compose our classes with a dbt client and other services depending on our needs. A hypothetical example of this could be:

class Dbt:

   def __init__(self):
      self.dbt_runner = dbtRunner()
   
   def invoke(self, directory: Optional[os.Pathlike] = None, runner_args)
      starting_directory = os.getcwd()
      if directory:
          os.chdir(directory)
      result = self.dbt_runner.invoke(['--log-level','none', *runner_args])
      os.chdir(starting_directory)

      if not result.success:
          raise result.exception

      return result.result

    def parse(self, directory: os.Pathlike):
        return self.invoke(directory, ['parse'])

    def list_models(self, directory: os.Pathlike):
        return self.invoke(directory, ['ls'])


class DbtProject:

    def __init__(self, path: os.Pathlike):
        self.dbt = DbtRunner()
        self.path = path
        self.manifest = self._load_manifest()
       
  
    def _load_project():
        """Load a dbt Project configuration"""
        return yaml.load(open(os.path.join(self.path, 'dbt_project.yml')))

    def _load_manifest():
        """Load a manifest for a project."""
        return self.dbt.parse(self.path)

Using this as a baseline, we can abstract away the work that the dbtRunner is doing (perhaps to support calls to the dbt-server instead of a local dbt runner), as well as extend the functionality of a DbtProject.

Considerations

This proposal describes how we want to wrap and interact with dbt projects without actually solving the problems of splitting and merging projects. While this is a bit meta, how we describe our abstractions for dbt projects will define how difficult it will be to operate on dbt projects in the core functionality of dbt-meshify.

Open Questions

• Is there a world where we want to pull specific data from dbt Cloud?
• Do we want to split out the storage aspects of this code (copying files, initializing a dbt project, etc)?
• Naming is going to get awkward between a dbt project vs a dbt Cloud Project. Thoughts?
• What would be cool functionality!?!

[dave-connors-3]

@nicholasyager this totally makes sense to me -- I am a big fan of splitting off the project as an extension of the actual dbt functionality itself -- right now those are definitely too jumbled together! i am more than willing to refactor what we have to look more like this!

1. there will likely be a time when we may want to read data from dbt cloud -- when we get to a point of creating new resources, we may want to share connection details from an existing cloud project to new ones.
2. I think it may make sense to split out the file manipulation logic into it's own class too. Not sure exactly what that could look like, but maybe not dissimilar from the self.dbt = Dbt() line you have up above?
3. I think i'm ok will some confusing class names so long as we have them split into files that are relatively easy to reason about!
4. I think there's a lot of opportunity for us to expose some of the guts of what we're doing as actual commands at much smaller than split or connect at the project level -- I think there's utility in many of the actual model operations to be their own services too:

# add a group to your project
$ dbt-meshify group -s tag:finance

# add a contract to your model (data type and constraint yml)
dbt-meshify contract -s dim_customers

# add a model version to a model
dbt-meshify version -s fct_orders

Most of this will happen as a part of the larger split and connect commands, but I think there's definitely appetite for this tool to help with all the mesh concepts at any time, not just when you're setting up the concept from scratch!

[nicholasyager]

I really like the idea of exposing these operations via the CLI in addition to the API. I believe this approach would then make it much easier for power users to script custom splitting and merging operations.