Add a C-compatible API

[asinghvi17]

It would be great to have a C-accessible API for Icechunk, as a "lowest common denominator" for access from Julia, C, GDAL, etc.

I am personally a contributor to Julia's Zarr implementation in https://github.com/JuliaIO/Zarr.jl, and would be happy to write a Julia integration for Icechunk given a C API from this end. Interop from Julia to Rust directly is tricky (it's a similar problem as C++ has) so a C-API is the easiest way to go here.

I imagine it could be structured pretty simply, the store would likely have to be represented as a pointer. C doesn't look to have many async capabilities but that can be handled on the calling language's end (Julia has async calls, for example, that live in a specific pool of threads).

Would be great to hear what might be needed to support this!

cc @alex-s-gardner @felixcremer @meggart from the Julia end

[rabernat]

Yes, we think this would be a great addition and support it 💯 percent!

It would be great to find a contributor who has experience creating C bindings from Rust libraries to work on this. @asinghvi17 is that something you have any experience with?

[meggart]

I have some limited experience in creating C bindings for a Rust library I have written 5 years ago for an internal project and for calling the lib from R, python, Julia and Matlab. However as this is quite some time ago I would need to check if what I have done back then is still idiomatic.

I think a good step to get us started would be if you could share a small hello-world-icechunk example as a standalone Rust application, maybe one that just opens an existing public dataset and requests a chunk of data. I or someone else if there are volunteers could then try to create an MWE that starts wrapping the subset of functions called in this example, creates c headers automatically and calls the functions from Julia. Then we at least get a first idea of the problems we might face and the amount of work it would be to get to complete C bindings.

[paraseba]

@meggart we have a couple of example applications in the examples directory. smoke_test.py is a pretty trivial one in there. That would give you an understanding of the basic python bindings.

If you want something Rust only, we also have an examples dir under icechunk. In there I'd recommend this one.

[mpiannucci]

I am happy to provide some guidance here as well. The simplest implementation of this could start with porting the zarr store interface to a C style API. The python implementation does this and should provide a pretty comprehensive guide on what pieces are required.

[meggart]

Thanks a lot for pointing to the examples. In oder to start a discussion I shared a few code snippets in #399 @asinghvi17 and others maybe you can have a look?