Standard Data Connection Interface & Databricks Catalog Scanning

[natb1]

This PR implements concepts described in #11 as well as this draft design document. In short:

• It introduces a standard interface for adding new data sources/destinations to Perseus called a "DataConnection".
• It demonstrates an implementation of that interface by adding a new Databricks data source. New Databricks datasource is implemented up to loading a scan/profile report for any available Databricks data catalogs. The implementation demonstrates concepts that can be extended to deploying ETL pipelines and the final CDM.

1. Out of scope for this PR

• This solution is specifically designed to be backwards compatible with all existing database sources/destinations. It makes it easier to add new connections, easier to maintain and document those connections, but it does not refactor any code that is already working.
• This PR only enables scanning of Databricks data sources. Implementation of ETL pipeline deployment is pending additional design - in particular, design of ETL pipeline serialization. The concepts in this PR should be extensible to that use case which will be implemented in a future PR.

2. Limitations Addressed by this Design

1. Today, adding or modifying how Perseus interacts with a database involves touching many parts of the code. For example, there are many parts of the code that are switch statements that check the "type" of the database and modify the logic or "strategy" depending on that.

• Switch statements in the templates to determine which templates and which parts of the templates to render.
• Switch statements in the controller code to determine what strategy to use for testing or scanning a connection.
• Lookup tables for default ports for each database.
• etc.

As a result, adding or modifying database integrations involves deep knowledge of the full code base, and parsing/modifying many parts of the code. This is time consuming and error prone, and could result in workarounds such and relying on documentation for dealing with edge cases vs. improving UX.

2. Existing infrastructure (esp. Perseus backend, and WhiteRabbit) assumes RDBMS backends with constraints that won't be relevant for other DBMS'.

• Profiling is parameterized to address scalability limitations in RDBMS' that are not relevant for distributed DBMS' and would actually be detrimental to cost/performance.
• There are no standardized interfaces for DDL. SQL Render is primarily focused on analytic query parsing and it would likely not be feasible to extend to a generalized DBMS interface. (After more research it appears that SQL Render does, in fact, aspire to generalize DDL. The complexity of extending that to a generalized DBMS interface - that includes, for example, Delta optimization - seems extreme)
• In general, there is little support for non-relational and/or non-sql (ex. REST) integrations with DBMS'

3. New Design Elements

1. A new DataConnection typescript interface.

This interface encapsulates all of the logic necessary to implement a new data source. Implementations of this interface are registered with a DataConnection Angular service in order to be "injected" into the Perseus app.

I.e. any developer adding a new data source can do so by 1) implementing this interface and 2) registering the implementation with the service WITHOUT the need to learn or modify any other parts of the code.

2. An OPTIONAL data-connection Node (typescript backend) service to support integration with non-relational DBMS'. DataConnection implementations can integrate via ANY backend including WhiteRabbit/CDM Builder etc. The data-connection backend service is an option that provides additional flexibility for DBMS' not traditionally handled by the existing services. The service is built using an open source Node framework for data access services called loopback. There is native support for OpenAPI - the Angular frontend integration is generated programmatically from the OpenAPI spec. It also has a built in ORM. In short, it is a lightweight option for integrating with arbitrary data backends in a SOA.

For Databricks this means:

• Flexibility to integrate with EITHER the SQL Warehouse, or Jobs API depending on the configuration and use case.
• Ability to write queries optimized for Spark.
• Support for full Databricks DDL.

4. Summary of Improvements

• Data sources can be added to Perseus with less effort, easier maintainability and less risk of regressions in existing code.
• Backend data integrations can be designed with greater flexibility using open standards.
• Databricks data sources can be scanned to use as a source for OMOP ETL.

[natb1]

Updated section 2.2 with additional context on SQLRender

[ssamus]

Hi @natb1 @leeevans we have completed the review of this pull request. The feature is not fully implemented and cannot merge into the master branch. The feature must be implemented and tested before it merges with Master. Please re-create the pull request in the Dev branch.

[natb1]

@ssamus I will resubmit to develop.

Can you please provide more context? Can we have a conversation with your developer? For example:

• The PR DOES include automated testing. It also includes a framework for QA of the new UI elements.
• There is apparently no pre-existing framework for testing. Is there any additional context you could provide on the standard I could work towards for the code to be considered "tested".
• The code DOES implement the functionality described. It does include fully functioning interface that when implemented does extend the UI with additional data sources. It does enable the scanning of a Databricks workspace in production. Is there more information you can provide on the level of functionality you would like to see implemented?

Mostly, is there a way for us to have a technical dialog about contributing to the codebase?