support API for Arrow Flight SQL

[erikerlandson]

The desired use case is data scientists or other users working with arrow-based data, such as parquet, pandas, polars, etc, and who would like to get that data into or out of Trino, without having to hack through intermediate steps or intermediate ascii sql row representations.

Flight SQL means database servers can implement a standard interface that is designed around Apache Arrow and columnar data from the start. Just like how Arrow provides a standard in-memory format, Flight SQL saves developers from having to design and implement an entirely new wire protocol. As mentioned, Flight already implements features like encryption on the wire and authentication of requests, which databases do not need to re-implement.

https://arrow.apache.org/blog/2022/02/16/introducing-arrow-flight-sql/

[erikerlandson]

cc @caldeirav @MichaelTiemannOSC

[jwilliams-ocient]

Heavy ➕.

The team at Ocient is also looking to allow Trino to connect to a FlightSQL data source. The connector implementation seems like it would be pretty straightforward if I'm following https://trino.io/docs/current/develop/example-jdbc.html (source) correctly.

[OmarSultan85]

Heavy ➕.

The team at Ocient is also looking to allow Trino to connect to a FlightSQL data source. The connector implementation seems like it would be pretty straightforward if I'm following https://trino.io/docs/current/develop/example-jdbc.html (source) correctly.

Hey Jwilliams,

Just wondering how far long are you in this track? We are looking to implement a custom connector to allow for querying data from a FlightSQL Data Source. But the documentation in the example jdbc driver doesn't seem to provide enough information on how to actually query data and run the queries.

Were you able to reach anything?

[YuriyGavrilov]

+1

[YuriyGavrilov]

just to leave this here: https://www.vldb.org/pvldb/vol10/p1022-muehleisen.pdf

[YuriyGavrilov]

some good example of Flight SQL server:

• https://github.com/voltrondata/flight-sql-server-example
• https://www.kamu.dev/blog/2023-09-datafusion-flightsql/ https://github.com/kamu-data/kamu-cli
• https://www.postgresql.org/about/news/apache-arrow-flight-sql-adapter-for-postgresql-010-2716/

[georgewfisher]

+1

[Firstero]

+1

[Jmoore1127]

+1

[alekhed]

+1

[shohamyamin]

+1

[alexmarco]

+1

[findepi]

I am not aware of anyone working on Arrow Flight SQL APIs for Trino, but Arrow-based data transport should be possible thanks to project Swift (#22271)