#DRAFT/WIP 20 allow custom submissions to slurm #21
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Running into this error on slurm due to https://github.com/eic/simulation_campaign_single/blob/main/scripts/run.sh#L87 |
|
Is this merging https://github.com/eic/job_submission_slurm/ into this? |
|
It'd be great if that means we can give up on having two repos. |
wdconinc
reviewed
Nov 18, 2023
scripts/submit_csv.sh
Outdated
|
|
||
|
|
||
|
|
||
|
|
||
| condor_submit -verbose -file ${SUBMIT_FILE} |
There was a problem hiding this comment.
Not needed anymore when SYSTEM==slurm?
scripts/submit_csv.sh
Outdated
| @@ -65,9 +89,44 @@ sed " | |||
| s|%INPUT_FILES%|${INPUT_FILES}|g; | |||
| s|%REQUIREMENTS%|${REQUIREMENTS}|g; | |||
| s|%CSV_FILE%|${CSV_FILE}|g; | |||
| s|%ACCOUNT%|${ACCOUNT:-rrg-wdconinc}|g; | |||
There was a problem hiding this comment.
Prefer to crash this if not specified, rather than have everyone submit under my allocation ;-)
There was a problem hiding this comment.
Example: ${ACCOUNT:?Define ACCOUNT with the slurm account}, ref: https://www.gnu.org/software/bash/manual/bash.html#Shell-Parameter-Expansion
Yup. That's the idea. |
Also give a familiar interface to submitters as the campaign coordinators. So, users can for example can run custom workflows like this on narval or ifarm without having write privileges to xrootd. CAMPAIGN_OUTPUT=$SHARED/ePIC/ePIC-Campaign-Organizer/Campaign_Output SYSTEM=slurm EBEAM=5 PBEAM=41 DETECTOR_VERSION=main DETECTOR_CONFIG=epic_craterlake JUG_XL_TAG=nightly CSV_FILE=gamma_1GeV_part2.csv ./scripts/submit_csv.sh narval_csv single SINGLE/etaScan/gamma.csv 1 |
wdconinc
reviewed
Nov 19, 2023
| @@ -28,12 +28,32 @@ shift | |||
| TARGET=${1:-2} | |||
| shift | |||
|
|
|||
| # environment variable to indicate whether the job is running on condor or slurm | |||
| SYSTEM=${SYSTEM:-condor} | |||
There was a problem hiding this comment.
SYSTEM may be a bit generic. How about SCHEDULER or something like that?
|
|
||
| # create command line | ||
| EXECUTABLE="./scripts/run.sh" | ||
| ARGUMENTS="${TYPE} EVGEN/\$(file).\$(ext) \$(nevents) \$(ichunk)" | ||
| EXECUTABLE="$PWD/scripts/run.sh" |
There was a problem hiding this comment.
Probably should use something that determines the directory of the current script, e.g. SCRIPTDIR=$(dirname $0) or so. Then this script can be run from outside the directory, and the files it generates can be better organized.
Comment on lines
+44
to
+46
| if [ ${SYSTEM} = "condor" ]; then | ||
| ARGUMENTS="${TYPE} EVGEN/\$(file).\$(ext) \$(nevents) \$(ichunk)" | ||
| elif [ ${SYSTEM} = "slurm" ]; then |
There was a problem hiding this comment.
if elif else fi can be turned into case statement
| elif [ ${SYSTEM} = "slurm" ]; then | ||
| ARGUMENTS="${TYPE}" | ||
| # FIXME: This is not ideal. It prevents from submitting multiple jobs with different JUG_XL_TAG simultaneously. | ||
| cd scripts |
There was a problem hiding this comment.
See above for SCRIPTDIR which would be useful here.
Also pushd-popd for directory stacks then.
| done | ||
| echo "Submitting ${NJOBS} to a ${SYSTEM} system" | ||
|
|
||
| if [ ${SYSTEM} = "condor" ]; then |
@wdconinc Here is a minimal script that tries to just run the same command on the remote node and surprisingly it doesn't face permissions issues when writing that file. Granted i downloaded eic-shell prior to submitting the job. Really confusing why the main job submission script is failing. Here is the slurm log file and the job finishes without issues |
where gamma_1GeV_small.csv contains And that leads to this log |
|
This issue can be resolved here with but that doesn't actually get us any further on the osg condor side. |
|
(with that diff it just fails a few lines later, but ok) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Briefly, what does this PR introduce?
Adapt for custom submissions to narval
What kind of change does this PR introduce?
Please check if this PR fulfills the following:
Does this PR introduce breaking changes? What changes might users need to make to their code?
Does this PR change default behavior?