Exception while using SlurmWorker: ImportError: No module named pydra.engine.helpers

[adi611]

(Based on the discussion on 08-06-23 with @djarecka and @ghisvail)

I created a workflow based on the example on the PSI/J website. The workflow works fine with cf as the plugin and produces the expected outputs: Output.txt file and the Image.png files. More about this can be accessed at: #663. Now when I try to run the same workflow (with some relevant changes) on slurm it gives me an exception: ImportError: No module named pydra.engine.helpers. My code:

cmd1 = 'pdftotext'
args = ['/code/Week_1_2/content/Lorem_ipsum.pdf', '/code/Week_1_2/content/Output.txt']
shelly1 = pydra.ShellCommandTask(name='shelly1', executable=cmd1, args=args)
print('cmndline = ', shelly1.cmdline)

cmd2 = 'wordcloud_cli'
args2 = ['--text', '/code/Week_1_2/content/Output.txt', '--imagefile', '/code/Week_1_2/content/Image.png']
shelly2 = pydra.ShellCommandTask(name='shelly2', executable=cmd2, args=args2)
print('cmndline = ', shelly2.cmdline)

wf1 = pydra.Workflow(name = 'wf1', input_spec=['x'], x = '/content/Lorem_ipsum.pdf')
wf1.add(shelly1)
wf1.add(shelly2)

wf1.set_output([('out_s1', wf1.shelly1.lzout.stdout), ('out_s2', wf1.shelly2.lzout.stdout)])
with pydra.Submitter(plugin='slurm') as sub:
    sub(wf1)
wf1.result()

My folder structure is as follows:

GSoC/
├── pydra/
└── pydra_codes/
    └── Week_1_2/
        ├── content/
        │   └── Lorem_ipsum.pdf
        └── pydra_gsoc_week1.py

The actions I performed for running the slurm container (which are based on the GitHub Action workflow):

docker image pull mgxd/slurm:19.05.1

docker container run -itd -h ernie --name slurm \
  -v /mnt/c/Users/Aditya/OneDrive/Desktop/GSoC/pydra_codes:/code \
  -v /mnt/c/Users/Aditya/OneDrive/Desktop/GSoC/pydra:/pydra \
  -e NO_ET=TRUE mgxd/slurm:19.05.1

docker container exec slurm bash -c "sacctmgr -i add cluster name=linux \
   && supervisorctl restart slurmdbd \
   && supervisorctl restart slurmctld \
   && sacctmgr -i add account none,test Cluster=linux Description='none' Organization='none' \
   && pip install wordcloud nest_asyncio"

docker container exec -it slurm bash
yum install -y poppler-utils
exit

docker container exec slurm bash -c "python /code/Week_1_2/pydra_gsoc_week1.py"

The pdf used is at the following link.

Please let me know if there are any additional details I can provide.

[satra]

that slurm container and version is quite old at this point. i would suggest recreating/finding a container with a more recent version of slurm.

[adi611]

Thanks for the suggestion, I will look for containers with newer versions of slurm.

[djarecka]

Hi @adi611 - I think the main error you were showing us yesterday is due to some conflicts in the python paths in the container. If you add you examples to the tests which use relative imports you won't get this (although you will get another issues since the command you try to run is not installed etc.).

Since you will be looking for another newer containers, there is probably no need to spend time fixing it. And if you want to test something in the meantime, just create some tests and run with pytests and relative paths.

[adi611]

Thank you for the suggestion, I'll create tests using pytest and relative paths for interim testing purposes.