Added error handling when a chuck is not downloaded from an Azure blob #19
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- To download a file from an azure blob we use the libraries "github.com/Azure/azure-pipeline-go/pipeline" and "github.com/Azure/azure-storage-blob-go/azblob". We divide the file into chunks and download it chunk by chunk until we download the whole file. For each chunk, we can retry up to 20 times. However, if the download of a chunk fails in all of 20 times (for example due to connectivity issues), we do not handle and report the error. This patch stores all the errors when downloading a chunk and in case all 20 retries have failed it stops the download and reports the error. Signed-off-by: Ioannis Sfakianakis <[email protected]>
jsfakian
force-pushed
the
CI-349-BUG-checksum-mismatch
branch
from
7c95a36 to
d84ba7d
Compare
milan-zededa
reviewed
Jan 8, 2024
| } | ||
| }, | ||
| }) | ||
| if err != nil { |
There was a problem hiding this comment.
I think this if err != nil { is not needed. As I understand it, dr.Body is just constructor and download starts below with io.CopyBuffer.
There was a problem hiding this comment.
Also, is this async in some way? I haven't looked at the azure API in quite some time.
There was a problem hiding this comment.
I do not know about the whole API, but the way we use it is as follows: We send 16 concurrent requests to download 1MB chunks. When we download all of them, we move to the next 16 chunks until we download the whole file.
|
I think you need to sign the DCO |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
"github.com/Azure/azure-storage-blob-go/azblob". We divide the file into chunks and download it chunk by chunk until we download the whole file. For each chunk, we can retry up to 20 times. However, if the download of a chunk fails in all of 20 times (for example due to connectivity issues), we do not handle
and report the error. This patch stores all the errors when downloading a chunk and in case all 20 retries have failed it stops the download and reports the error.