Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cat with a timeout can truncate responses #127

Open
mrd0ll4r opened this issue Feb 7, 2023 · 0 comments
Open

cat with a timeout can truncate responses #127

mrd0ll4r opened this issue Feb 7, 2023 · 0 comments

Comments

@mrd0ll4r
Copy link
Contributor

mrd0ll4r commented Feb 7, 2023

Hello,

the IPFS API for cat returns a chunked, streaming response, but still accepts the timeout parameter. If that is specified, responses may be truncated if the timeout is reached. The API then appends an HTTP trailer named X-Stream-Error to the response. Currently, this crate swallows that error, which truncates file responses (and potentially appends the trailer as garbage at the end).

Example API call:

$ curl --raw -v -X POST "http://127.0.0.1:5001/api/v0/cat?arg=/ipfs/f01701220d45cdd10be2505683ab3d9dbcb0ea6429b61debfede8d23f10a8531a27383bd5&timeout=30s" --output tmp
*   Trying 127.0.0.1:5001...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 127.0.0.1 (127.0.0.1) port 5001 (#0)
> POST /api/v0/cat?arg=/ipfs/f01701220d45cdd10be2505683ab3d9dbcb0ea6429b61debfede8d23f10a8531a27383bd5&timeout=30s HTTP/1.1
> Host: 127.0.0.1:5001
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Access-Control-Allow-Headers: X-Stream-Output, X-Chunked-Output, X-Content-Length
< Access-Control-Expose-Headers: X-Stream-Output, X-Chunked-Output, X-Content-Length
< Content-Type: text/plain
< Server: kubo/0.18.1
< Trailer: X-Stream-Error
< Vary: Origin
< X-Content-Length: 1388014004
< X-Stream-Output: 1
< Date: Tue, 07 Feb 2023 17:30:35 GMT
< Transfer-Encoding: chunked
<
{ [4104 bytes data]
100  250M    0  250M    0     0  8533k      0 --:--:--  0:00:30 --:--:-- 9370k
* Connection #0 to host 127.0.0.1 left intact
$ tail tmp
<binary garbage>
0
X-Stream-Error: context deadline exceeded

I'm using the hyper client with a global timeout configured. I then call it like so:

client
        .cat(c)
        .map_ok(|chunk| chunk.to_vec())
        .try_concat()
        .await
        .map_err(|err| anyhow!("{}", err))

If the timeout occurs before streaming starts, the server responds with 500, which is correctly picked up.
However, if the timeout occurs after streaming has started, no error is returned and you end up with a partial file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant