Skip to content

Latest commit

 

History

History
62 lines (51 loc) · 4.5 KB

README.md

File metadata and controls

62 lines (51 loc) · 4.5 KB
Raw

Setting up Airflow on Windows (local)

Hi! 95% of the blog posts and aides online say "Set up Airflow quickly using this docker image!", "Use managed Airflow on Azure!", or "Set up Airflow on AWS/GCS!" And those solutions are great. And sometimes you have to do a proof of concept without those tools or the tool-set you have available is limited. All of the instructions here I have done on my own personal computer.

Scenario: you have a Windows machine/server and you want an open-source, code-based solution to scheduling your code, but a GUI interface would be appropriate for the larger team.

The 1,000 feet view of the solution:

  • Windows Server (or Windows 10 machine)
  • WSL 1 (This was the limiting factor)
  • Ubuntu 18.04.6
  • Apache Airflow 2.3

Now I get it, the number of steps below might look a little suspect, even more steps than needed. I think it's worth noting that some of these steps are essential, some are because I want to have a reproducible environment ( same Python specs inside and outside of WSL), and others cause I want to set up some foundational pieces of good coding practice.

Step Essential Reproducable Good coding Notes
Powershell X The version installed already on Windows should be fine. I personally like version 7.
Visual Studio Code X Any coding IDE will do, but I really like vscode for a lot of reasons these days
Ubuntu X
pyenv-win X I use this to not accidentally upgrade Python and to keep versions constraints the same across environments
Python X if you want to code in your Windows environment and not just in WSL, that is why this is installed
pip install X X everything listed there is for reproducibility and good coding.
Activate WSL X
VScode setup X X not essential if you have a different prefered IDE

All before opening WSL:

  • Microsoft Store Downloads: (you can access the store from Windows programs/windows search!)
    • PowerShell 7: I like the newer version provided in the store because it comes with git installed. I also encounter less bugs with setting up PnPOnline modules(a different project!)
    • Visual Studio Code: You will see a feature preview version and the regular version. Get the regular.
    • Ubuntu 18.04.6: I list the version so you can know what I used in case there are any differences in later versions. This was not an intentional choice, but listing it for the sake of reproducibility
  • PowerShell 7 set up: (open PowerShell "as administrator") and do the following
    • pyenv-win: commands are here PowerShell instructions
    • Python 3.8.10:
      • pyenv install 3.8.10
      • pyenv version and copy output for later.Mine was 3.8.10 (set by C:\Users\aapee\.pyenv\pyenv-win\version)
    • pip install: I want these available no matter what the virtual environment. (I use pip in this example because collaborators are not as familiar with anaconda/conda/miniconda. Also, miniconda felt like one more thing to install. I know conda-forge handles packages dependencies better, please don't come at me :D )
      • pyvenv
      • pipreqs
      • black
      • pylint
    • activating WSL:
      • Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
      • great instructions
  • Set up Visual Studio Code
    • Choose a Python interpreter
      • open vscode
      • ctrl+shift+p to open the settings search bar
      • type "Python: select" and click on the Python: select interpreter option
      • look for the pyenv interpreter choice (it will say 3.8.10 on the left and pyenv in blue on the right)
    • Import profile of choice, or select extensions that speak to you. I can do a separate post of my fav vscode stuff.

Upcoming Steps:

  • WSL, Python & PostgreSQL
  • Apache Airflow install & initial config
  • Apache Airflow customizations
  • REST API & vscode
  • MinIO xcom backend