add DOWNLOAD staging action

[andre-merzky]

This fixes #3266

[codecov]

Codecov Report

Attention: Patch coverage is 28.57143% with 20 lines in your changes missing coverage. Please review.

Project coverage is 42.91%. Comparing base (2618c57) to head (6942125).
Report is 9 commits behind head on devel.

Files with missing lines	Patch %	Lines
src/radical/pilot/utils/staging_helper.py	33.33%	12 Missing ⚠️
src/radical/pilot/agent/staging_input/default.py	14.28%	6 Missing ⚠️
src/radical/pilot/agent/staging_output/default.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##            devel    #3282      +/-   ##
==========================================
+ Coverage   42.78%   42.91%   +0.12%     
==========================================
  Files          97       97              
  Lines       11310    11286      -24     
==========================================
+ Hits         4839     4843       +4     
+ Misses       6471     6443      -28     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[AymenFJA]

I left 2 comments @andre-merzky. Thanks.

[AymenFJA]

Correct me if I am mistaken, but some Linux flavors might not have the wget command. So to avoid this, I think Python approach is the most guaranteed to work in most cases as Python interpreter must exist. So I would suggest to do the following:

import requests 

def download(self, src: str, tgt: str, flags:str) -> None:
    """Download the remote file to the current tgt directory."""
    response = requests.get(src, stream=True)

    tgt = ru.Url(tgt).path

    try:
       response.raise_for_status()  # Check if the download was successful
    except Exception as e:
       print(f'ERROR: skipping staging, download failed due to: {e}')
       return
   
    self.mkdir(os.path.dirname(tgt), flags)

    # Save the file content
    with open(tgt, 'wb') as f:
        for chunk in response.iter_content(chunk_size=8192):
            f.write(chunk)

[andre-merzky]

Ack, we can use python for that, will change!

[AymenFJA]

LGTM, thanks @andre-merzky