Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TaskVineExecutor: write function data to temp directory #3592

Merged
merged 7 commits into from
Sep 6, 2024
Merged

TaskVineExecutor: write function data to temp directory #3592

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
b75a3ae
function data in temp
colinthomas-z80 Aug 19, 2024
2c18d6c
use getpass for username
colinthomas-z80 Aug 19, 2024
c201ec1
use tempfile module
colinthomas-z80 Aug 20, 2024
9f6b037
flake etc
colinthomas-z80 Aug 20, 2024
5ec7cdb
Merge branch 'master' into tmp_function_data
benclifford Aug 21, 2024
cd7229f
Merge branch 'master' into tmp_function_data
benclifford Sep 4, 2024
bfa9bd9
Merge branch 'master' into tmp_function_data
benclifford Sep 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion parsl/executors/taskvine/executor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
import threading
import uuid
from concurrent.futures import Future
from datetime import datetime
from typing import List, Literal, Optional, Union

# Import other libraries
Expand Down Expand Up @@ -215,7 +216,8 @@ def __create_data_and_logging_dirs(self):

# Create directories for data and results
log_dir = os.path.join(run_dir, self.label)
self._function_data_dir = os.path.join(run_dir, self.label, "function_data")
tmp_dir = os.path.join('/tmp/', f'{self.label}-{os.getlogin()}')
self._function_data_dir = os.path.join(tmp_dir, datetime.now().strftime('%Y%m%d%H%M%S%f'), "function_data")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is more a reference for readers of #1122 that review comment for @colinthomas-z80 - here's another instance of using a timestamp as a unique ID that #1122 might want to address

os.makedirs(log_dir)
os.makedirs(self._function_data_dir)

Expand Down
Loading