Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix the pickling error for IO objects #1929

Merged
merged 1 commit into from
Oct 20, 2023

Conversation

ndonyapour
Copy link
Contributor

Hello, I'm trying to run a CWL workflow using Toil which involves the serialization/pickling of jobs. However the recent updates to CWL throws this stack trace below

 File "/home/donyapourn2/mambaforge-pypy3/envs/toil_test/bin/toil-cwl-runner", line 8, in <module>
    sys.exit(main())
  File "/home/donyapourn2/mambaforge-pypy3/envs/toil_test/lib/python3.10/site-packages/toil/cwl/cwltoil.py", line 3903, in main
    outobj = toil.start(wf1)
  File "/home/donyapourn2/mambaforge-pypy3/envs/toil_test/lib/python3.10/site-packages/toil/common.py", line 1406, in start
    rootJobDescription = rootJob.saveAsRootJob(self._jobStore)
  File "/home/donyapourn2/mambaforge-pypy3/envs/toil_test/lib/python3.10/site-packages/toil/job.py", line 2670, in saveAsRootJob
    self._saveJobGraph(jobStore, saveSelf=True)
  File "/home/donyapourn2/mambaforge-pypy3/envs/toil_test/lib/python3.10/site-packages/toil/job.py", line 2643, in _saveJobGraph
    job.saveBody(jobStore)
  File "/home/donyapourn2/mambaforge-pypy3/envs/toil_test/lib/python3.10/site-packages/toil/job.py", line 2532, in saveBody
    pickle.dump(self, fileHandle, pickle.HIGHEST_PROTOCOL)
TypeError: cannot pickle '_io.TextIOWrapper' object
Failure! Please scroll up and find the FIRST error message.

I figured out this line breaks the serialization/pickling and this PR fixes this error.

@codecov
Copy link

codecov bot commented Oct 18, 2023

Codecov Report

Merging #1929 (f12335b) into main (56913e7) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1929      +/-   ##
==========================================
- Coverage   83.95%   83.95%   -0.01%     
==========================================
  Files          46       46              
  Lines        8190     8189       -1     
  Branches     2174     2174              
==========================================
- Hits         6876     6875       -1     
  Misses        843      843              
  Partials      471      471              
Files Coverage Δ
cwltool/context.py 100.00% <100.00%> (ø)

@tetron
Copy link
Member

tetron commented Oct 19, 2023

This is probably fine, but I don't know offhand what validate_stdout is actually used for and what the implications are of changing the default value.

@tetron
Copy link
Member

tetron commented Oct 19, 2023

In other words, was anything relying on the old behavior of that field being sys.stdout by default instead of None?

@ndonyapour
Copy link
Contributor Author

I don't think so. The validate_stdout variable is being set for runtimeContext here before the job executor is called.

@mr-c
Copy link
Member

mr-c commented Oct 19, 2023

file=runtime_context.validate_stdout,

@ndonyapour
Copy link
Contributor Author

Is the executor always called by main.py or are there situations where it can be called directly?

@mr-c
Copy link
Member

mr-c commented Oct 19, 2023

Is the executor always called by main.py or are there situations where it can be called directly?

External users of cwltool (example: toil-cwl-runner) also call the executor without using main.

Copy link
Member

@mr-c mr-c left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Copy link
Member

@mr-c mr-c left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add a pickling test , to prevent regression

@mr-c
Copy link
Member

mr-c commented Oct 19, 2023

@ndonyapour
Copy link
Contributor Author

Thank you for your response! I've checked the file variable in the line you mentioned above, and I couldn't find any other instances where it's used. Can that line be deleted? I'm currently working on a pickling test for context.py.

@ndonyapour
Copy link
Contributor Author

Added a pickling test for RuntimeContext to this PR.

@mr-c
Copy link
Member

mr-c commented Oct 19, 2023

I've checked the file variable in the line you mentioned above, and I couldn't find any other instances where it's used. Can that line be deleted?

If you check the git blame for that line, you will find it was added to enable testing of the enhanced --validate in https://github.com/common-workflow-language/cwltool/pull/1915/files#diff-b77d003dbd8ec8f0b12435dfb6950b341c9dc78cc083cf851c5259142286a210R32

So no, it is quite needed :-)

@mr-c
Copy link
Member

mr-c commented Oct 19, 2023

Can you run make cleanup to fix the formatting?

@mr-c mr-c enabled auto-merge (rebase) October 20, 2023 12:42
@mr-c mr-c merged commit fd75a25 into common-workflow-language:main Oct 20, 2023
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants