mrobee / airflow-spark Public

Notifications You must be signed in to change notification settings
Fork 0
Star 3

Apache Airflow with Apache Spark cluster support

3 stars 0 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
dags		dags
data		data
docker		docker
pipeline_scripts		pipeline_scripts
.env.sample		.env.sample
.gitignore		.gitignore
Makefile		Makefile
README.md		README.md
docker-compose.yml		docker-compose.yml

Repository files navigation

airflow-spark

Apache Airflow with Apache Spark cluster support

Usage

Build the docker image using make build-images
Put your data in data directory
Put your dag files in dags directory
Put your additional pipeline scripts in pipeline_scripts directory
Edit .env file to meet your needs
Run docker-compose up
Open http://localhost:8089 in your browser

About

Apache Airflow with Apache Spark cluster support

Report repository

Releases

No releases published

Packages

No packages published

Languages