Skip to content

Latest commit

 

History

History
12 lines (8 loc) · 978 Bytes

README.md

File metadata and controls

12 lines (8 loc) · 978 Bytes
Raw

Cloud Composer Examples:

This repo constains the following examples of using Cloud Composer, Google Cloud Platform's managed Apache Airflow service:

  1. Composer Dataflow Examples

    a. Simple Load DAG: provides a common pattern to automatically trigger, via Google Cloud Function, a Dataflow job when a file arrives in Google Cloud Storage, process the data and load it into BigQuery. Alt text

  2. Composer HTTP Example

    a. Ephemeral Dataproc Spark DAG: provides an example of triggering a DAG via HTTP POST to the Airflow API to create a Dataproc cluster, submit a Spark job, and import the newly enhanced GCS files into BigQuery. Alt text