Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Arrow streaming without schema #60

Closed
domoritz opened this issue Aug 14, 2024 · 5 comments · May be fixed by #61
Closed

Arrow streaming without schema #60

domoritz opened this issue Aug 14, 2024 · 5 comments · May be fixed by #61
Labels
enhancement New feature or request

Comments

@domoritz
Copy link

I have a stream of record batches that I want to send but I don't have access to the schema easily. Would it make sense to have an API that just takes a stream of batches and gets the schema from the first batch?
@abdolence
Copy link
Owner

Hey. Correct me if I'm wrong but not sure if it is possible since IPC stream requires it.

Probably as a workaround you can just send some kind of dummy schema?

@abdolence
Copy link
Owner

Reading here though:
https://arrow.apache.org/docs/format/Columnar.html#serialization-and-interprocess-communication-ipc

Seems it is not defined as required. Then probably it is a good idea to have it optionally.

@abdolence abdolence added the enhancement New feature or request label Aug 14, 2024
@abdolence abdolence linked a pull request Aug 14, 2024 that will close this issue
Arrow IPC stream format doesn't require schema anymore #61
Draft
@abdolence
Copy link
Owner

If you want to give a try, you can check this PR:
#61

refering to GitHub branch in your Cargo.toml

@abdolence
Copy link
Owner

abdolence commented Aug 14, 2024 •
edited
Loading

Schema-less is not possible though to use using the official Apache Arrow implementation API for Rust. So, they treat it as required.

It is not blocking me in this crate since the streaming mostly implemented in the crate itself, but just as FYI.

image

@domoritz
Copy link
Author

Doesn't make sense as explained in #61 (comment)

@domoritz domoritz closed this as not planned Won't fix, can't repro, duplicate, stale Aug 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Arrow IPC stream format doesn't require schema anymore abdolence/axum-streams-rs
2 participants
@domoritz @abdolence