#Draft/WIP: Simplify lookup of scratch directory

[rahmans1]

Briefly, what does this PR introduce?

Different systems use different scratch spaces. We can pass this through the template file specific to the site the jobs are being submitted from. For example, on OSG https://github.com/eic/job_submission_condor/blob/main/templates/osg_csv.sh.in.

export TMPDIR=${_CONDOR_SCRATCH_DIR}

On JLAB slurm,
export TMPDIR=/scratch/slurm

On CEDAR/narval slurm,

export TMPDIR=$SLURM_TMPDIR

What kind of change does this PR introduce?

• Bug fix (issue #__)
• New feature (issue #__)
• Documentation update
• Other: __

Please check if this PR fulfills the following:

• Tests for the changes have been added
• Documentation has been added / updated
• Changes have been communicated to collaborators

Does this PR introduce breaking changes? What changes might users need to make to their code?

Does this PR change default behavior?

[wdconinc]

How does this deal with TMPDIRs that are only set on the node itself, like the /localscratch/epicprod.$JOBID?

[rahmans1]

How does this deal with TMPDIRs that are only set on the node itself, like the /localscratch/epicprod.$JOBID?

$SLURM_TMPDIR variable usually points to the /localscratch location but jlab for example recommends using the /scratch/slurm location for working directory. https://scicomp2015.jlab.org/docs/slurm_file

Similarly $_CONDOR_SCRATCH_DIR points to /srv location when submitting through OSG jobs but at BNL condor that variable is set to /tmp and that location seems to not be writable.

It would be an easy fix if all of the servers had the same variable pointing to the same location but that seems not to be the case. This may require a bit of trial and error and may be it just doesn't work

But I was thinking something like export TMPDIR=/localscratch/epicprod$SLURM_ARRAY_JOB_ID.$SLURM_ARRAY_TASK_ID in the template file if we know what pattern the automatedly created folders by slurm will be. The environment script doesn't get executed till the job is running. So, the variables should be populated when it does.

Basically, _CONDOR_SCRATCH_DIR works for all OSG submissions. But it's not going to work for dedicated running on BNL condor or the slurm servers.