Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Future: Deal with prefect top level map function #32

Open
evamaxfield opened this issue Jan 31, 2020 · 1 comment
Open

Future: Deal with prefect top level map function #32

evamaxfield opened this issue Jan 31, 2020 · 1 comment

Comments

@evamaxfield
Copy link
Contributor

Use Case

Please provide a use case to help us understand your request in context
In our current implementation we have decided we somewhat want the following:

raw = Raw(check_exists=True)
qc = QC(check_exists=False)
norm = Norm(check_exists=False)

with Flow() as flow:
	raw_data = raw.run()
	qc_data = qc.run(raw_data)
	normed_data = norm.run(qc_data)

but inside norm.run there may be an entire prefect workflow defined. Specifically to allow use of mapped operations. As an example here is what the pseudo-code of the norm.run may look like:

def run(qc_data, **kwargs):
    with Flow() as flow:
        processed = qc_data.map(self._single_process_function)

    return processed

This is useful for defining that certain tasks require certain resources but in the case that all tasks require the same resources it would be great to allow .run to be used in a normal top level prefect flow.

@evamaxfield
Copy link
Contributor Author

Update: I think this will likely be related to the new Prefect Result Class and related Handlers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants