eth/abi: decode/encode array of tuples

[tbazadaykin]

I have encoded data like this:

0x0000000000000000000000000000000000000000000000000000000000000042000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000c0000000000000000000000000000000000000000000000000000000000000000100000000000000000000000018a475d6741215709ed6cc5f4d064732379b5a580000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000002e516d57426953453942795236767278346876726a71533353473572367745345352713743503252567061665a5756000000000000000000000000000000000000

and i know that it's was encoded like ['uint256', '(address,uint256)[]', 'string'] with values [66, [(0x18a475d6741215709ed6cc5f4d064732379b5a58,1)], QmWBiSE9ByR6vrx4hvrjqS3SG5r6wE4SRq7CP2RVpafZWV]
and i know that everything is ok, because it's successfully decoded with ethers.js

is it possible to decode such data with this gem?

[tomiwaAdey]

Yes, have a look at Abi decode.

[kurotaky]

Hi, Thanks for the issue. I think you mean the following.

[19] pry(main)> types
=> ["uint256", "(address,uint256)[]", "string"]

[20] pry(main)> args
=> [66, ["(\"0x18a475d6741215709ed6cc5f4d064732379b5a58\",1)"], "QmWBiSE9ByR6vrx4hvrjqS3SG5r6wE4SRq7CP2RVpafZWV"]

# or 
#[24] pry(main)> args
#=> [66, [["0x18a475d6741215709ed6cc5f4d064732379b5a58", 1]], #"QmWBiSE9ByR6vrx4hvrjqS3SG5r6wE4SRq7CP2RVpafZWV"]

[25] pry(main)> Abi.encode types, args
Eth::Abi::Type::ParseError: Unknown base type
from /Users/kurotaki/src/github.com/q9f/eth.rb/lib/eth/abi/type.rb:173:in `validate_base_type'

Perhaps handling of tuple types should be implemented. 😄

Related resources

• https://docs.soliditylang.org/en/develop/abi-spec.html?highlight=tuple#handling-tuple-types
• handling tuples (structs) web3/web3.js#1241

[q9f]

Ref EthWorks/ethereum.rb#141

[geraldb]

Hello, about help wanted for tuple encoding / decoding. FYI: I started out with your excellent eth gem - since there is no support for tuple encoding I started out with a quick hack and it all has now lead into a new serries of eth gems - see rlp-lite, ethlite, abiparser, etc. @ https://github.com/rubycocos/blockchain - for now the new abiparser gem is separate from the abi encoding/decoding in ethlite - but if time permits I try to make abiparser a better replacement.

If you are intersted in working together or have questions or comments, please feel free to ping me. All the best. Cheers. Prost. Happy New Year 2023!

[q9f]

I would love to redo the RLP and ABI parts of this gem (time permitting). I'm interested in collaborating. I'll take a look at your gem collection after the holidays. Thanks for sharing! 🖤

[geraldb]

Thanks for the response - you can find a first version of a new RLP gem (called rlp-lite) at
https://github.com/rubycocos/blockchain/tree/master/rlp-lite - this is actually a fully functional and quick & dirty merge & fast clean-up of eth.rb and web-eth.rb - as I am understanding more and more of RLP and ABI encoding and decoding I may (re)start from scratch to further simplify. I have started from scratch on a new abiparser gem (that is for now only for documentation and interface ids or such) but the mid-term / long-term plan is to replace the "old" ABI encoding & decoding inside the the rlp-lite gem (also used in ethlite, etc.) - see https://github.com/rubycocos/blockchain/tree/master/abiparser for the work-in-progress.

Note: I am not an employee and I am not getting paid for anything, thus, for now progress is slow and mostly a function of what I need for my own little (etherum pixel art collections / token) scripts (as the only user of the gems so far ;-). All the best. Cheers. Prosit 2023! Happy New Year.

[q9f]

Thanks for the response - you can find a first version of a new RLP gem (called rlp-lite) at https://github.com/rubycocos/blockchain/tree/master/rlp-lite - this is actually a fully functional and quick & dirty merge & fast clean-up of eth.rb and web-eth.rb - as I am understanding more and more of RLP and ABI encoding and decoding I may (re)start from scratch to further simplify. I have started from scratch on a new abiparser gem (that is for now only for documentation and interface ids or such) but the mid-term / long-term plan is to replace the "old" ABI encoding & decoding inside the the rlp-lite gem (also used in ethlite, etc.) - see https://github.com/rubycocos/blockchain/tree/master/abiparser for the work-in-progress.

This is great. RLP was actually my first Ethereum library (Crystal). I found it very straightforward.

However, with ABI I had issues finding a good specification/tests and some cases are still not covered in this repository, unfortunately, Ref ethereum/tests#691 (comment)

Note: I am not an employee and I am not getting paid for anything, thus, for now progress is slow and mostly a function of what I need for my own little (etherum pixel art collections / token) scripts (as the only user of the gems so far ;-). All the best. Cheers. Prosit 2023! Happy New Year.

I'm in the same boat, but happy to keep this channel open 🙌🏼

[geraldb]

Thanks again for the quick update and yes, I am too happy to keep this channel open for updates and/or figuring out how to work together.

Great to see Crystal in use - thanks for sharing (as open source). I have started adding some first ABI tests thanks for your references - will try to add the little test fixtures that are in the official repo. On the abiparser gem side I have now (that is, today) incorporated the full type parser (incl. nested tuples) Again all the best. Cheers. Prosit 2023! See you next year ;-).

Update I put together a new "stand-alone" abicoder gem that "extracts" a fully functional ABI.encode & decode with all types and with zero-dependencies (on any 3rd party gems incl. any of my own) for easy (re)use. that should make working together way easier if there's any interest in the new year. See https://github.com/rubycocos/blockchain/tree/master/abicoder. And for some first tests see https://github.com/rubycocos/blockchain/blob/master/abicoder/test/test_types.rb and https://github.com/rubycocos/blockchain/blob/master/abicoder/test/test_coder.rb.

[q9f]

reference for tuple encoding: https://github.com/ethers-io/ethers.js/blob/c80fcddf50a9023486e9f9acb1848aba4c19f7b6/packages/abi/src.ts/coders/tuple.ts

ruby: https://github.com/rubycocos/blockchain/blob/53edc7ba0f776bcd2f1531ce1445be1adac5dabc/abicoder/lib/abicoder/encoder.rb#L118 (I might look into porting this over)

I'll probably get the tests from ethers.js or web3.js

[geraldb]

Happy new year. Prosit 2023! I have started to add tests from various sources (so far I did't find anything definite) - see https://github.com/rubycocos/blockchain/tree/master/abicoder/test. Everything should work now (according to the new & latest ABI Encoding V2 (solidity) standard), that is, including nested tuple & arrays (dynamic & static / fixed). The best source for tests I found is the ethabi in rust - unfortunately the tests are "inline" in the code and encoded in rust - see https://github.com/rust-ethereum/ethabi/blob/master/ethabi/src/encoder.rs I try to add a couple more and I am kind of feeling confident that the encoding / decoding is now "complete" and I will start to move on "up the stack" to work on abigen / abiparser / etc. Anyways, the encoder and decoder are now coded on-purpose as "single-file" scripts with zero-dependencies (expect for the Type.parse) for easy (re)use. You are welcome to (re)use the gem or copy-n-paste code as you please. All the best. Cheers. PS: As a minimum common shared base maybe we can start with the tests or such (e.g. I am happy to (re)use and copy all your ABI tests, for example - also it looks like a conflict lies ahead ;-)
I am using "to-the-metal" mintest and I see you are more a fan of rspecs ;-) - so maybe json datasets like in ethereum can bridge the gap or such). Anyways, as written I am kind of confident that the encoder/decoder is now complete and I will move on but sure I am happy and try to help / assist in upgrading in eth.rb if you have any questions, comments, etc.

PS: Also to state the obvious from my point of view the recommendation and best practice is to adopt & (re)use the abicoder gem itself. I went to great-length to keep it zero-dependcy. Anyways, it's all up-to-you. Do what you think is best. Nothing wrong with copy-n-pasting from each other ;-).

PPS: I started adding some abi coder test fixtures (in .yaml), see https://github.com/rubycocos/blockchain/blob/master/test/abicoder/basic.yml, https://github.com/rubycocos/blockchain/blob/master/test/abicoder/more.yml and the test runner https://github.com/rubycocos/blockchain/blob/master/abicoder/test/test_fixtures.rb.

[carlosbalsas]

Any solution to this?

[q9f]

Solution would be to find a Ruby engineer who has time 😆

[geraldb]

Any solution to this?

Do not ask what other rubyists can do for you (for free) - Why not ask what you can do for others ;-) ?

Anyways, you can use the abicoder gem that has full support for abi encoding/decoding array of tuples, etc. See https://github.com/rubycocos/blockchain/tree/master/abicoder I am sure you can use it with eth.rb (by "monkey patching") - I started out that way too by trying to "patch" the eth json rpc call in eth.rb (and than ended-up with the ethlite gem alternative).

But I agree - short or long term the dream would be to find some funding for some paid ruby programming hours. (if you want progress faster)

PS: By the way, my design philosophy is (if you browse https://github.com/rubycocos/blockchain ) to break-up the eth "monolith" into many (re)usable modules /gems - so the idea is that in the end we end-up like eth 2.0 with a big merge :-) - but for sure for now it looks like I am entertaining myself with no (re)use in practice ;-).

[q9f]

@geraldb I ported your collection of spec tests to this gem and identified two different issues

• we cannot encode tuples (obviously, this issue)
• we cannot encode multi-dimensional arrays eth/abi: decode/encode multi-dimensional arrays #217

I looked into importing abicoder as a dependency and using your gem to resolve the entire encoding and decoding issues, however, it conflicts with the Eth::Abi namespace and I don't want to untangle this. Also, I would lose some functionality.

Instead, I am looking into adapting your types and parser into my encoder. But this will take some time on my end.

[geraldb]

@q9f Please give me a week or (reallistically) two to respond - happy to help out. Unfortunately I am in an emergency right now I got hit by 10+ take downs on github (punksnotdead) e.g. https://github.com/cryptopunksnotdead/awesome-cryptopunks-bubble, https://github.com/cryptopunksnotdead/awesome-24px, https://github.com/cryptopunksnotdead/lets-go-programming-cryptopunks, and many more. - I am new to this absurdity and trying to figure out / work out how to continue / defend myself. Thank you for your understanding.

[geraldb]

@q9f A little quick update (coding will take another week) - here are some suggestions what I can do (if that makes sense or helps out):

• break-up abicoder into abicoder and new abitypes (or such) - so you can reuse the type parser
• I can add a new "umbrella" / top-level namespace so you can continue to use your own Eth::Abi (but of the top of my head - the abicoder is using Abi only not Eth::Abi - no conflict? - anyways happy to resolve this with require "abicoder/base" or such
• about the missing abi encoding (or decoding) functionality in abicoder - please, tell if possible - and I add or move over the missing machinery
• also you are more than welcome to join the rubycocos org (&blockchain) as a committer
• and open for your suggestion / proposals, / ideas of course.

Anyways, to sum-up as a quick first item (early next week) I can break-up abicoder and package-up the new abitypes gem (for easy integration / (re)use).

PS: You might be missing the real / major tests - sorry for any confusion - because of the "monorepo" -style - the abicoder tests are in https://github.com/rubycocos/blockchain/tree/master/test/abicoder outside of the abicoder gem / dir.

[wuminzhe]

When learning ABI through @geraldb's code, I made some changes. here is my repo: https://github.com/wuminzhe/abi_coder_rb

[geraldb]

to all: sorry not trying to shill myself. i am no longer a no-coiner - still a no-bitcoiner ;-) - and i am happy to accept ether if anyone (or any org) wants to sponsor work on the abicoder (for open source work i always offer a 1+1 deal - you get two hours or more for one). please, ping me if interested. i am sorry that i am broke (out-of-money) and thus i cannot work / help for free in the next months for sure.

ps: and yes, happy to see the version by @wuminzhe to run on wasm. keep up the great work and thanks for sharing.