Airflow on local machine

Prelude

You want to run Airflow locally. Perhaps you want to test a new DAG or some other changes before deploying.

Warning!

Airflow can only run on Windows via containers or the Linux Subsystem (WSL). Don’t bother trying otherwise. Mac & Linux are fine.

Prerequisites

• a machine
• a terminal
• Python 3

Terminal

  conda create -n airflow python
  conda activate airflow
  pip install apache-airflow
  airflow db migrate
  airflow users create -u admin -p admin -f Ad -l Min -r Admin -e admin@gmail.com

Then to run, in separate terminals:

  airflow scheduler
  airflow webserver

And the UI will be available at localhost:8080 with the example dags included.

Addendum

Wait, where do I put my custom dags?

Locally, Airflow will create the airflow.cfg file in $HOME/airflow aka $AIRFLOW_HOME. There you can customize everything you might need, including the dags_folder. If sharing an airflow.cfg with your teammates, could also simply set $AIRFLOW_HOME to a dir from your chosen Git repository.

Why the db migrate and user creation?

Well for some reason, after all these years, these manual steps are still required. The db init seems to be ran by the webserver, yet not the migrate nor creating a default admin user. Only the other deployment options seem to do this by default.