Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a metastore method that returns all available tables #133

Open
2 tasks
yruslan opened this issue Jan 17, 2023 · 1 comment
Open
2 tasks

Add a metastore method that returns all available tables #133

yruslan opened this issue Jan 17, 2023 · 1 comment
Labels

Comments

@yruslan
Copy link
Collaborator

yruslan commented Jan 17, 2023

Background

Since the metastore can be used as a library, it could be useful to enumerate which tables exist.

Feature

Add a method to the metastore reader that lists all tables that are available.

  • Scala
  • Python

Proposed Solution

val tables: Seq[String] = metastore.getTables()

An idea to discuss. Usually, metastores have the concept of databases. We can simulate it by a dot prefix (db.table1). We can use this to segregate uat and prod tables, for instance.

// Fetch all tables starting with 'uat.'
val tablesUat: Seq[String] = metastore.getTables("uat")

Idea. What if instead of Seq[String] the method would return Seq[Table], which would contain table name and description?

@jirifilip
Copy link
Collaborator

The segregation by prefixes is a very nice idea. Users could even organize the tables into "hierarchies", like uat.bronze.customers and prod.gold.purchases (if using medallion architecture) etc. And then query all production gold tables for example.

Just adding this potential idea here so we don't forget.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants