Introduction to Apache Airflow — Simple Guide

Apache Airflow is a tool to help you run and manage repeating tasks (called workflows). Think of it like a smart scheduler that knows the order of steps, when to run them, and what to do if something fails.

This guide explains Airflow in simple words, shows an easy example (a DAG), and gives step-by-step setup instructions for Windows and Linux (including Docker-based setup).

Quick overview

• A workflow in Airflow is called a DAG (Directed Acyclic Graph). A DAG is just a list of steps with rules about which step runs first.
• Each step in a DAG is a Task. Tasks can run commands, Python functions, SQL, or anything you script.
• Airflow schedules and runs Tasks according to the DAG and the schedule you give it (for example, every day at 2am).

Why use Airflow?

• It makes workflows visible: you can see the steps and status in a web UI.
• It retries failed steps and can alert you.
• It lets you reuse building blocks (operators) and share connections to databases and cloud services.

Main components

• DAG (workflow): The overall plan of steps and their order.
• Task: A single job inside the DAG (e.g., run this SQL, call this API).
• Operator: A pre-made type of task (e.g., BashOperator runs shell commands, PythonOperator runs Python code, EmailOperator sends email).
• Scheduler: The part of Airflow that decides when to start tasks based on DAG schedules.
• Webserver (UI): The browser interface where you view DAGs, runs, logs, and trigger manual runs.
• Metadata Database: A small database (usually SQLite for tests, Postgres/MySQL in production) where Airflow stores DAG runs, task status, and configuration.
• Executor/Workers: The mechanism that runs tasks. Simple installs use SequentialExecutor or LocalExecutor. For scale, use CeleryExecutor or Kubernetes.
• Connections: Saved credentials (like AWS keys or database login) that tasks can use.
• XCom: Small messages passed between tasks (e.g., one task produces a filename, another uses it).

Simple example — a hello DAG

Create a file dags/hello_world.py inside your Airflow dags folder. This DAG runs once and prints a message.

from datetime import datetime
from airflow import DAG
from airflow.operators.bash import BashOperator

with DAG(
    dag_id='hello_world',
    start_date=datetime(2026, 1, 1),
    schedule_interval=None,  # run only when triggered
    catchup=False,
) as dag:

    say_hello = BashOperator(
        task_id='say_hello',
        bash_command='echo "Hello from Airflow!"'
    )

    say_hello

• Save this into dags/ and then trigger the DAG from the Airflow UI to see the task run.
• The UI shows logs; you can open logs for say_hello to see the printed message.

Another simple example — Python task and dependency

from datetime import datetime
from airflow import DAG
from airflow.operators.python import PythonOperator

def greet():
    print("Good morning — this is a Python task in Airflow!")

with DAG('python_example', start_date=datetime(2026,1,1), schedule_interval=None, catchup=False) as dag:
    task1 = PythonOperator(task_id='greet', python_callable=greet)
    task2 = BashOperator(task_id='list_files', bash_command='ls -la')

    task1 >> task2  # run task2 after task1

Where to put your code

• dags/ — put DAG Python files here.
• plugins/ — custom operators/hooks if you write them.
• logs/ — Airflow writes task logs here (location configurable).

Setup options (recommended) — Docker Compose (works on Windows and Linux)

This is the easiest and most consistent way to run Airflow locally.

1. Install Docker (Docker Desktop on Windows or Docker Engine + Compose on Linux).
2. Create a folder and a docker-compose.yml (you can use the official Airflow example). A minimal workflow:

mkdir airflow-docker
cd airflow-docker
# Create a simple docker-compose.yml or use the official one from Apache Airflow docs.
# Then initialize and start Airflow:
export AIRFLOW_UID=$(id -u)
# On Windows (PowerShell) set AIRFLOW_UID manually to your user id if needed.

docker compose up airflow-init
docker compose up -d

3. Open the Airflow web UI at: http://localhost:8080
4. Place DAG files into the dags/ folder used by the docker-compose setup (usually a volume in the compose file). The web UI will detect them automatically.

Notes:

• Docker Compose creates a Postgres metadata DB and a scheduler and webserver for you.
• To stop: docker compose down (add --volumes if you want to remove data).

Setup option B — pip install (for Linux/macOS or WSL on Windows)

On Windows native Python, Airflow is not fully supported; use WSL2 or Docker. The steps below work well inside a Linux environment.

1. Create a virtual environment and install constraints (Airflow requires a constraints file matching your Python version).

python3 -m venv .venv
source .venv/bin/activate
PYTHON_VERSION=3.12
AIRFLOW_VERSION=2.9.0
pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"

2. Initialize the metadata database and create a user:

airflow db init
airflow users create \
  --username admin \
  --firstname Admin \
  --lastname User \
  --role Admin \
  --email admin@example.com

# Start scheduler and webserver in separate terminals
airflow scheduler &
airflow webserver --port 8080

3. Open the UI at http://localhost:8080 and log in with the user you created.

Common notes and tips (simple)

• If a task fails, Airflow shows the error and logs in the UI. You can retry the task or fix the code and re-run.
• Use schedule_interval like @daily or cron strings to run regularly.
• Use Connections to securely store credentials instead of hardcoding keys in DAGs.
• For production, use Postgres/MySQL as the metadata DB and a proper executor (Celery or Kubernetes).

Example: Passing data between tasks (XCom)

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def push(ti):
    ti.xcom_push(key='message', value='hello from push')

def pull(ti):
    val = ti.xcom_pull(key='message', task_ids='push_task')
    print('Got:', val)

with DAG('xcom_example', start_date=datetime(2026,1,1), schedule_interval=None, catchup=False) as dag:
    push_task = PythonOperator(task_id='push_task', python_callable=push)
    pull_task = PythonOperator(task_id='pull_task', python_callable=pull)
    push_task >> pull_task

XCom is small and good for passing tiny values like filenames or IDs.

Troubleshooting quick guide

• UI blank or not reachable: check that the webserver is running and port 8080 is open.
• DAG not showing: make sure the file is in the correct dags/ directory and there's no syntax error in the file.
• Tasks failing due to missing credentials: add a Connection in the UI or use environment variables.

Final words

Airflow is powerful but can feel complex at first. Start small — a few DAGs that run daily — and grow from there. Use Docker Compose locally for easiest setup. If you'd like, I can:

• Add a ready-to-run docker-compose.yml and .env for local testing.
• Create a sample repository layout with dags/, plugins/, and a docker-compose.yml so you can docker compose up right away.