Building an End-to-End Open-Source Modern Data Platform for Biomedical Data

This workshop at PyBCN 2022 is a detailed guide to help you navigate the modern data stack and build your own platform using open-source technologies. Data engineering has experiences enormous growth in the last years, allowing for rapid progress and innovation as more people than ever are thinking about data resources and how to better leverage them. In this talk we will explore the related technologies and build from scratch an end-to-end modern data platform for the analysis of medical data.

We will be using open-source tools and libraries, including python-based DBT, Apache Airflow and Querybook.

The platform will consist of the following components:

• Data warehouse
• Data integration
• Data transformation
• Data orchestration
• Data visualization

INSTALL REQUIREMENTS

• Install Python
• Install Java
• Install docker
  • in Linux edit your /etc/hosts and add 172.17.0.1 docker.host.internal

INSTALL COMPONENTS

• Download synthea patient data generator: synthea-with-dependencies.jar
• Install Dremio: docker pull dremio/dremio-oss
• Install PostgreSQL: docker pull postgres
  • Install psql
  • Install a SQL client such as PgAdmin or DBeaver
• Install DBT: pip install dbt-postgres dbt-dremio
• Install Airflow
• Install Querybook
• Install Datahub

Have a drink, and relax ...

• https://www.youtube.com/watch?v=y3TxcejHw-4

FIRST STEPS

• Clone this repo

git clone https://github.com/alabarga/pybcn22-modern-data-stack.git

• Generate or download synthetic data