Listing S3 objects with space or unicode characters in the key fails

[kerryjj]

Environment

• Elixir & Erlang versions (elixir --version):
  Erlang/OTP 21 [erts-10.3.5.11] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace]
  Elixir 1.8.2 (compiled with Erlang/OTP 21)

• ExAws version mix deps |grep ex_aws
  2.1.3

• HTTP client version. IE for hackney do mix deps | grep hackney
  1.15.2

Current behavior

When using ExAws.S3.list_objects_v2 or ExAws.S3.list_objects and a key that has a space or unicode chars in it, the query params passed to S3 are double encoded.

The following command fails because the URL signatures don't match. This is because the query prefix is encoded to replace " " with +. Then the sanitize method replacces + with %2B.

ExAws.S3.list_objects_v2("my-bucket", prefix: "Copy ") |> ExAws.request()
ExAws: Request URL: "https://s3.eu-west-1.amazonaws.com/my-bucket/?list-type=2&prefix=Copy%2B" 

Expected behavior

ExAws.S3.list_objects_v2("my-bucket", prefix: "Copy ") |> ExAws.request()
ExAws: Request URL: "https://s3.eu-west-1.amazonaws.com/my-bucket/?list-type=2&prefix=Copy%20" 

I have a PR that shows a potential fix for this, so will include that here as well.

[kerryjj]

Here's the PR #704

I think this fixes the issue. Let me know what you think and if there's something I've missed here in terms of how these methods are used.

[rmayhue]

Just wondering if anyone is still working on this since the PR above was submitted a while ago and is still open.

I'm running into this exact issue where spaces in the key are generating "The request signature we calculated does not match the signature you provided."

@kerryjj - I'm guessing that you're still using PR #704 ?

[kerryjj]

@rmayhue Yes, I'm using PR #704 in production, because this is a real bug in the main repo and I can't work with out.

[rmayhue]

@kerryjj Thanks for the reply. Yesterday I forked the main repo and generated a patch from PR #704 and it applied cleanly to master so I'm now using my forked repo. Like you, I can't work without it. Thanks for your work on PR #704, you saved me a bunch of time 👍

[kerryjj]

Good to hear. I'm going to aim to get back to the PR this weekend and see if I can find some info that would help with adding some new tests. Haven't had a chance to yet and it's been quite hard to find concise info on how the encoding works in all of the AWS services as it does seem to work differently sometimes depending on which of the AWS APIs are being used.

…

On Mon, 29 Mar 2021 at 14:55, Rob Mayhue ***@***.***> wrote: @kerryjj <https://github.com/kerryjj> Thanks for the reply. Yesterday I forked the main repo and generated a patch from PR #704 <#704> and it applied cleanly to master so I'm now using my forked repo. Like you, I can't work without it. Thanks for your work on PR #704 <#704>, you saved me a bunch of time 👍 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#703 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAMPFJIXNVOKZ33DOPKVM3TGCBFXANCNFSM4PEBUO5A> .