-
Notifications
You must be signed in to change notification settings - Fork 120
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Vine: add library context implementation #3900
Vine: add library context implementation #3900
Conversation
Thanh, what's the status on this one? |
This is on hold at the moment. I'm working on whether to add the abstraction of context for task groups. |
# in case of FutureFunctionCall tasks | ||
new_args = [] | ||
for arg in args: | ||
if isinstance(arg, dict) and "VineFutureFile" in arg: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This smells like trouble to me. Why does the worker need to know if this is a future file as opposed to the creation of a regular file?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this belongs to the future function calls and I don't modify that code except left-indent that code block in. @JinZhou5042 I see that you added this line in #3670. Can you shed some insights please?
@dthain the new change send a string as |
RTM? |
Tests for special functions (lambda, dynamically executed code) are added with some fix to the PR. This PR is RTM @dthain. |
Proposed Changes
This PR adds the mechanisms needed so that FunctionCalls can pull data hosted in a library on a remote node.
Merge Checklist
The following items must be completed before PRs can be merge.
Check these off to verify you have completed all steps.
make test
Run local tests prior to pushing.make format
Format source code to comply with lint policies. Note that some lint errors can only be resolved manually (e.g., Python)make lint
Run lint on source code prior to pushing.