Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hotfix/mkdir #3165

Merged
merged 2 commits into from
Apr 10, 2024
Merged

Hotfix/mkdir #3165

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b74bed2
implicit mkdir on local agent staging
andre-merzky Apr 10, 2024
285b223
fix the fix
andre-merzky Apr 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions src/radical/pilot/agent/staging_input/default.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,8 +181,7 @@ def _handle_task(self, task, actionables):
if action in [rpc.COPY, rpc.LINK, rpc.MOVE]:
assert src.schema == 'file', 'staging src expected as file://'

# backend will take care of dir creation - but we do it manually
# for local ops (copy, link, move)
# implicitly create target dir if needed - but only for local ops
if action != rpc.TRANSFER:
tgtdir = os.path.dirname(tgt.path)
if tgtdir != task_sandbox.path:
Expand Down
7 changes: 7 additions & 0 deletions src/radical/pilot/agent/staging_output/default.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -317,6 +317,13 @@ def _handle_task_staging(self, task, actionables):
if action in [rpc.COPY, rpc.LINK, rpc.MOVE]:
assert tgt.schema == 'file', 'staging tgt expected as file://'

# implicitly create target dir if needed - but only for local ops
if action != rpc.TRANSFER:
tgtdir = os.path.dirname(tgt.path)
if tgtdir != task_sandbox.path:
self._log.debug("mkdir %s", tgtdir)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we elevate this to a warning with something like: "WARNING: implicit directory creation! Mkdir %s"?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this would help: unless the end user is actively looking for warnings in the agent logs, it would not really show up. Also, if we provide some semantics then users begin to rely on it (as we have just seen in this case), and then a warning is not intuitive either.

Having said that, I will open a ticket to clearly document the implemented behavior - sorry for not doing that anyway...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#3166 is now open for this.

ru.rec_makedir(tgtdir)

if action == rpc.COPY:
try:
shutil.copytree(src.path, tgt.path)
Expand Down
Loading