Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ibis support for datasets / destinations #2003

Closed
sh-rp opened this issue Oct 30, 2024 · 0 comments · Fixed by #2004 · May be fixed by #2046
Closed

ibis support for datasets / destinations #2003

sh-rp opened this issue Oct 30, 2024 · 0 comments · Fixed by #2004 · May be fixed by #2046

Comments

@sh-rp
Copy link
Collaborator

sh-rp commented Oct 30, 2024

We want to explore ibis support. There are to things we can probably do:

  1. allow the user to get ibis in memory tables (or iterate over them) the same way arrow tables and dataframes work in the current implementation. this is very easy because we just convert arrow tables into ibis in memory tables and provide two new methods on a readablerelation. this will work for all destinations supported by our dataset

  2. the "real" integration would be to hand over credentials from a destination to the ibis connection. The user will the get an ibis object which is a full replacement of our dataset where they can join tables and even write back to the database or create new tables and stuff like that, all the regular ibis operations. This will work for a subset of our destinations, you can see the list of ibis backends in the backends dropdown here: https://ibis-project.org/. This should also works with our duckdb filesystem thing, excluding the write stuff of course which the user would have to do with the pond or a dlt pipeline to commit the computed data somewhere.

@sh-rp sh-rp moved this to In Progress in dlt core library Oct 30, 2024
@sh-rp sh-rp linked a pull request Nov 10, 2024 that will close this issue
@github-project-automation github-project-automation bot moved this from In Progress to Done in dlt core library Nov 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
1 participant