week_2_data_ingestion

Week 2: Data Ingestion

Data Lake (GCS)

• What is a Data Lake
• ELT vs. ETL
• Alternatives to components (S3/HDFS, Redshift, Snowflake etc.)
• Video
• Slides

Introduction to Workflow orchestration

• What is an Orchestration Pipeline?
• What is a DAG?
• Video

Setting up Airflow locally

• Setting up Airflow with Docker-Compose
• Video
• More information in the airflow folder

If you want to run a lighter version of Airflow with fewer services, check this video. It's optional.

Ingesting data to GCP with Airflow

• Extraction: Download and unpack the data
• Pre-processing: Convert this raw data to parquet
• Upload the parquet files to GCS
• Create an external table in BigQuery
• Video

Ingesting data to Local Postgres with Airflow

• Converting the ingestion script for loading data to Postgres to Airflow DAG
• Video

Transfer service (AWS -> GCP)

Moving files from AWS to GCP.

You will need an AWS account for this. This section is optional

• Video 1
• Video 2

Homework

In the homework, you'll create a few DAGs for processing the NY Taxi data for 2019-2021

More information here

Community notes

Did you take notes? You can share them here.

• Notes from Alvaro Navas
• Notes from Aaron Wright
• Notes from Abd
• Blog post by Isaac Kargar
• Blog, notes, walkthroughs by Sandy Behrens
• Add your notes here (above this line)