Golang bindings for recrypt-rs

[bugeats]

Description

recrypt-rs is a pure-Rust library that implements a set of cryptographic primitives for building a multi-hop Proxy Re-encryption scheme, known as Transform Encryption. We need to use a subset of this tooling in a golang environment. The task is to create a binding between the two platforms.

Only a subset of the recrypt features are needed at this time. The goal is to support the basic example, reproduced here:

use recrypt::prelude::*;
use recrypt::Revealed;

// create a new recrypt
let mut recrypt = Recrypt::new();

// generate a plaintext to encrypt
let pt = recrypt.gen_plaintext();

// generate a public/private keypair and some signing keys
let (priv_key, pub_key) = recrypt.generate_key_pair().unwrap();
let signing_keypair = recrypt.generate_ed25519_key_pair();

// encrypt!
let encrypted_val = recrypt.encrypt(&pt, &pub_key, &signing_keypair).unwrap();

// decrypt!
let decrypted_val = recrypt.decrypt(encrypted_val, &priv_key).unwrap();

// plaintext recovered.
assert_eq!(Revealed(pt), Revealed(decrypted_val))

Requirements

• An equivalent golang flow should be supported for the above Rust flow.
• The golang API and types should be modeled after the recrypt TypeScript definitions
  • The OOP/instance pattern from recrypt-rs should be mimicked
  • The golang API must also manage memory by clearing when an instance is destroyed
• It should be possible to maintain updates from the upstream Rust library
  • A simple update to a Cargo file will suffice
• Build scripts will produce a module that matches the current golang architecture
  • Required architectures: linux-386, linux-amd64, linux-arm, linux-arm64

FFI vs WASM

It is possible that the binding could be produced by either FFI via cgo or WASM via Wasmer. It is not important which method is used as long as the above requirements are satisfied.

There's lots of Rust auto-bindings type projects, but unfortunately there is no equivalent for golang. To use FFI is going to require manual work.

[bugeats]

Calling @coltfred and @clintfred. Your project is beautiful, but we don't know much Rust.

[bugeats]

💵 $200 bounty posted here 💵

[clintfred]

@bugeats Is https://github.com/IronCoreLabs/recrypt-wasm-binding close to what you'd need for the Wasmer binding?

[bugeats]

@clintfred yes the WASM binding is just fine, and we explored that idea in this branch. However, even after fighting with Wasmer for a while, we couldn't make it work. My naive understanding is that even though Wasmer can read the WASM artifact, it still needs to be built in such a way to support golang instead of JS.

[ernieturner]

@bugeats I took a look at this for a few hours yesterday and made a little bit of progress. I agree with your assessment that the existing recrypt-wasm-binding isn't likely to work since it includes a bunch of shim types from the wasm-bindgen crate that are unlikely to work everywhere.

Here's a summary of what I did and where I ended up.

I created a new Rust project that included recrypt as a dependency and enables the wasm feature set

recrypt = { version = "0.8.2", features = ["wasm"], default-features = false }

Then I wrote a method to just expose a single function from recrypt like so

extern crate recrypt;
use recrypt::prelude::*;

#[no_mangle]
pub extern "C" fn pt() -> *const u8 {
    let recrypt = Recrypt::new();
    let pt = recrypt.gen_plaintext();
    pt.bytes().clone().as_ptr()
}

Obviously this isn't great because I'm creating a new instance of Recrypt every time, but I wanted to start somewhere.

I was then able to run cargo build --release --target wasm32-unknown-unknown which generated me a .wasm file. I copied this file into a quick Go project and imported it like you have on your branch. Reading and instantiating the WASM binary worked, but when I tried to call the pt function, it gave me an error that I believe is because of the random number generation that is happening. I don't think the WASM binary has a way to generate random numbers as there is no external source of entropy for it to pull from. To try and prove that I changed the Rust code to use a RNG with a fixed seed.

NOTE: Don't do this! It isn't secure at all!

pub extern "C" fn pt() -> *const u8 {
    let recrypt = Recrypt::new_with_rand(ChaChaRng::from_seed([0u8; 32]));
    let pt = recrypt.gen_plaintext();

    pt.bytes().clone().as_ptr()
}

When I recompiled this and ran it from Go, it worked and gave me an index into the WASM memory for where the plaintext bytes are and I was able to see that with this

...
result, _ := pt()
pointer := result.ToI32()
memory := instance.Memory.Data()
fmt.Println(memory[pointer : pointer+384])

That printed out the not-at-all-random bytes generated from above.

I haven't dug into what work would be required to get the WASM binary to have a source of entropy. One option would be to compile it against wasm32-wasi which pulls entropy from the host environment. Support for that was added to Rand 0.7 but unfortunately we haven't upgraded to that version of Rand yet because of other dependency problems. I also don't know if compiling for wasm32-wasi would work within Go or not.

I hope this helps and isn't a bunch of information you already knew. Still a ton of unknowns and work to try, but I figured I'd post what I found out.

[moskusfe]

I'll implement the FFI/CGO binding.

[bugeats]

Hey @ernieturner thanks for taking such a deep look. Are you using the OOP pattern to encapsulate the state of the random seed generation? Is that the only thing it's used for? I wonder if these bindings would be easier to implement if all the functions were pure, and just took a seed value as an argument.

[bugeats]

@moskusfe I look forward to seeing what you come up with. We were hoping to find someone like you who might have deeper experience with FFI.

I forgot to mention in the requirements that we're going to need some basic unit tests. Hopefully we can then follow your lead in binding the entire recrypt API at a later time.

[ernieturner]

@bugeats What I did was certainly a hack and PoC, but I was just trying to get around the fact that, as far as I can tell, there isn't a way to get a source of entropy from within the WASM module. Doing what you describe, sending in a random seed, would work partially, but the main problem you'd have is that you should be using a ReseedingRng. So your initial seed could work (as long as it was truly random) but you'd then run into problems when the RNG needed to be reseeded.

At this point I don't really know enough to understand how you'd expose the Recrypt class across the boundary. Going down the purely functional route, the only way you'd be able to make these bindings work that way would be to make every method take a random seed as input and then create a new instance of Recrypt on every function call (similar to what I did above but with an actual random value). That's unlikely to be terribly performant, but I don't actually know what the cost would end up being. Probably depends on the use case and how frequently you were calling into Recrypt methods. Doing this would also solve the reseeding issue from above because you wouldn't generate enough randomness on a single function call for the RNG to require reseeding.

However, this path would definitely create a pretty big source of insecurity. Trusting that the data passed in as a seed is truly random and not a value used previously could be a pretty big weak link.

[bugeats]

@moskusfe are you still working on this? How's it going?

We were able to get some basic FFI stuff going, but the tricky part was understanding the C types to return, as well as how to handle the instantiation of the recrypt object.

[UnicodingUnicorn]

I'm curious, what happens if one just lazy_statics the recrypt object?

[coltfred]

@UnicodingUnicorn The recrypt api is thread safe, it locks on the Rng object as needed. I haven't tried it, but I would expect that if you put it in a lazy static it should work (as long as the CryptoRng you're making can also be made in the lazy static), it should work. Note that I don't think this helps with any of the problems that have been discussed so far.

[UnicodingUnicorn]

Doesn't being able to lazy_static save us having to reinitialise it in every function call? Which in turn means it isn't being reseeded everything the FFI is called.

[UnicodingUnicorn]

Is this the right way to continue?

[coltfred]

@UnicodingUnicorn Sorry for the delayed response. The problem with using a single version as lazy_static is that you still need to reseed your RNG, which isn't currently possible (as far as I'm aware) with rand 0.6 and WASM that's not running in the browser.

If you go down the path of using C bindings, then you can probably use lazy_static and the mutex internal to the Recrypt object should allow you to use it safely from multiple threads. The reason there was so much talk around reseeding and passing recrypt around was because of the assumption that WASM was being used.

Also note that Recrypt does use calls to the underlying OS to seed the default RNG we use. I'm not sure what the ramifications are of using that in lazy_static.

[UnicodingUnicorn]

Oh, I see. Sorry for missing that. By reseeding, do you mean that Recrypt will invariably use the same random seed every time it is instantiated, requiring the user to submit a seed on initialisation, or is it that it needs to be reseeded on every call to it?

[coltfred]

@UnicodingUnicorn Recrypt by default uses a RNG that's seeded based on the preferred entropy source for the architecture it's being built for. It then uses that entropy source again to reseed the RNG after some amount of bytes are consumed from that RNG. It's not reseeded each call, but based on the amount of bytes needed.