Consider supporting polars

[gerrymanoim]

Just a dumping ground for ideas.

I think we can extract most of the core logic to rust and then offer functions/packages that produce both polars data and pandas data (via arrow -> pandas conversions in python?).

[maread99]

👍 That definitely seems like the way to go for performance and flexibility. Rust is still stuck on my todo list ☹️

I'm wondering to what degree it would complicate the build and installation?

[gerrymanoim]

I'm wondering to what degree it would complicate the build and installation?

pyo3/maturin (https://www.maturin.rs/) are pretty well supported these days. We'd basically build wheels for everything and most users would just install the wheel (https://github.com/pydantic/pydantic-core/blob/main/.github/workflows/ci.yml#L446). So the build side for us would be more involved but mostly the same for the vast majority of users. Some users will build from source, but I think that's relatively uncommon/plenty of other libraries are bundling rust already.

I think figuring out the right parts of the api to support rust + pandas + polars would be a bit annoying but doable.

[maread99]

Sounds good. I've certainly noticed Rust pushing it's way into python projects.

Learning Rust is on my 2024 todo list. Unfortunately I can't see that I'll be in a position to offer any help until, at best, towards the end of the year...

[gerrymanoim]

No problem! I was going to see if I could carve out a weekend or some time on a flight to try to make a branch.

[maread99]

One thing that doesn't seem to sit right with this library is that it's on a fork. I was wondering whether any move to Rust and support for polars would justify starting over with a fresh library and just crediting the quantopian roots?

Just a thought.

[gerrymanoim]

I've definitely been annoyed by that.

I agree that maybe it would be good to develop in a new repo and then archive this one when we're ready (and do a major version update).