Transient Dataset

When a condition plugin is specified within a pipeline, the data pipeline planner, breaks down the data flow graph into phases. The phase before the conditional plugin could be executed as one or more jobs, either in parallel or serial depending on the plugins defined. Before, the conditional plugin is executed, the output from the previous phase is written to a transient dataset. Using of conditional plugin will force the output of previous stage being committed to disk.

Upon completion (success or failure) of job, the transient dataset is garbage collected the CDAP system.