Airflow DAG Not Running: A Systematic Debugging Path

Your DAG isn’t firing. Skip the guesswork. Work this path in order—each step tells you exactly what to run and where to look. In most cases it’s one of: the DAG never parsed, the DAG is paused, date math/catchup is off, pools or concurrency are saturated, the scheduler is unhealthy, workers are full, or a dependency deadlock. Run the commands below, then apply only the fix that matches what you found.

1) First: did the DAG parse successfully?

If a DAG doesn’t parse, the scheduler never knows it exists.

• List import errors:

airflow dags list-import-errors

• Confirm the DAG is visible:

airflow dags list | grep -E "(^| )my_dag( |$)" || echo "not found"

• Check parsing logs:

# Dag processor / parsing logs
ls -1 $AIRFLOW_HOME/logs/dag_processing/
# Tail the specific dag file's parse log
tail -n 200 $AIRFLOW_HOME/logs/dag_processing/<dag_file>.log
# Scheduler overview
tail -n 200 $AIRFLOW_HOME/logs/scheduler/latest/scheduler.log

Common failures: missing Python packages, circular imports, heavy module imports that exceed the dagbag timeout, invalid DAG IDs (only letters, numbers, dashes, dots, underscores). Quick checks:

# Where is Airflow loading DAGs from?
airflow config get-value core dags_folder

# Syntax and import sanity
python -m py_compile <dag_file>
python -c "import importlib; importlib.import_module('dags.my_dag')"

Control heavy dag parsing and timeouts. Prefer lazy imports inside tasks (operators) instead of at module import time. Global timeout:

# airflow configuration (airflow.cfg)
[core]
dagbag_import_timeout = 90

[scheduler]
parsing_processes = 4
min_file_process_interval = 30

There’s no per-file timeout; refactor slow dag files and avoid expensive network calls at import. If you landed here via an “airflow 3.3.0 documentation” search result: Airflow’s authoritative docs are versioned; check the matching version for your deployment.

2) Paused DAGs, schedule, start_date, and catchup

Many “won’t trigger on schedule” issues are simple.

• Is it paused?

airflow dags list | awk 'NR==1 || /my_dag/'
airflow dags unpause my_dag

• Does your schedule and start_date create a past run?

# Good pattern: explicit timezone-aware start_date
aimport pendulum
from airflow import DAG

with DAG(
    dag_id="my_dag",
    schedule="@daily",      # or cron like "0 3 * * *"
    start_date=pendulum.datetime(2024, 8, 1, tz="UTC"),
    catchup=False,           # only the latest interval
) as dag:
    ...

Rules that bite teams in week two:

• start_date must be before the first schedule boundary. If it’s in the future, nothing runs.
• catchup=False runs only the latest interval, not the backlog.
• schedule=None means manual only.

Inspect existing and pending dag runs:

airflow dags list-runs -d my_dag --no-backfill --state queued,scheduled,failed,success

Backfill specific dates if that’s what you meant:

airflow dags backfill my_dag -s 2024-07-01 -e 2024-07-03

See it quickly in the airflow ui (Graph view) and confirm no future-only schedule. If you need battle-tested patterns for schedules and SLAs, see our production Airflow DAGs service and our write-up on Airflow best practices from getting paged at 3am.

3) Pools and concurrency limits blocking runs

Even if scheduled, tasks won’t start if capacity is exhausted.

Limit	Scope	Where to check/set
Pool slots	Shared resource bucket	airflow pools list, airflow pools set <name> 8
DAG max_active_tasks	Per DAG	DAG args: max_active_tasks=8
DAG max_active_runs	Per DAG	DAG args: max_active_runs=1
Parallelism	Global scheduler	airflow config get-value core parallelism

Check if tasks are queued but not executing:

airflow dags list-runs -d my_dag --state queued,scheduled
# Show task tree to spot blocked branches
airflow tasks list my_dag --tree

Assign tasks to pools intentionally:

PythonOperator(
    task_id="heavy_etl",
    python_callable=run_etl,
    pool="warehouse_pool",  # control Snowflake/BigQuery slots
)

Right-size concurrency in code:

with DAG(
    dag_id="my_dag",
    max_active_tasks=8,
    max_active_runs=1,
    ...
):
    ...

When you see starvation across many DAGs, increase global capacity thoughtfully and monitor run latency, not just theoretical limits. If you use a managed service, our comparison of MWAA vs Astronomer vs Composer covers where each exposes pool and concurrency controls.

4) Scheduler health and worker capacity

If capacity looks fine, verify the scheduler and executors are alive.

• Is the scheduler job healthy?

# Airflow 2.x
airflow jobs check --job-type SchedulerJob || echo "scheduler unhealthy"

# System service (example)
systemctl status airflow-scheduler
journalctl -u airflow-scheduler -n 200 -e

# Logs
tail -n 200 $AIRFLOW_HOME/logs/scheduler/latest/scheduler.log

• Executors: Celery

airflow celery status
airflow celery inspect active
# Config: worker concurrency
airflow config get-value celery worker_concurrency

• Executors: Kubernetes

kubectl get pods -l component=worker -o wide
kubectl logs deploy/airflow-worker --tail=200

Symptoms to spot: heartbeats missing, rapidly growing queued tasks, repeated “No slots” messages. After fixing configuration or infra, gracefully restart the scheduler service and verify new heartbeats. For reference, the airflow scheduler process coordinates creation of runs while the dag processor parses files; both must be healthy. For BI teams sharing infra, measure “time from scheduled to first running task” on a busy day to prove improvements on your own setup.

5) Dependency deadlocks, fast checks, and prevention

Last mile: all green, but still no movement? Look for a dependency deadlock (e.g., ExternalTaskSensor waiting on a run that will never exist).

# Show upstream/downstream relationships
airflow tasks list my_dag --tree
# Inspect specific run state
airflow dags list-runs -d my_dag | head -n 10

# Metadata DB (read-only) to find stuck task instance(s)
psql $AIRFLOW_CONN_AIRFLOW_DB -c "
SELECT dag_id, task_id, run_id, state, start_date, end_date
FROM task_instance
WHERE dag_id='my_dag' AND (state='queued' OR state='upstream_failed')
ORDER BY start_date NULLS LAST
LIMIT 50;"

Fix patterns: set sensors to mode="reschedule", double-check external DAG IDs and schedules, and ensure upstream DAGs actually create the corresponding dag runs. Keep DAG and task IDs ASCII-only; avoid unicode in IDs to prevent subtle mismatches.

Prevention that actually works

• DAG parse tests in CI: run an Airflow container and import every DAG.

# GitHub Actions job excerpt
- run: pip install apache-airflow==2.*
- run: airflow db init
- run: airflow dags list  # fails CI on import errors

• Operational monitors: alert on “queued > X minutes” and scheduler heartbeat age; record run latency per DAG.
• Lightweight canary: one DAG that runs every 5 minutes and touches each critical connection.

from airflow.decorators import dag, task
import pendulum

@dag(schedule="*/5 * * * *", start_date=pendulum.now("UTC").subtract(minutes=10), catchup=False)
def canary():
    @task()
    def ping_db():
        # minimal query against your warehouse
        return 1
    ping_db()
canary()

Quick answers

• Airflow DAG won’t trigger on schedule? Check paused, then start_date vs schedule boundary, then catchup.
• Are dependencies met? Use airflow tasks list <dag> --tree and resolve blocked sensors.
• Extended ASCII/unicode? Not for DAG or task IDs; stick to [A-Za-z0-9_-.]. Python files can contain unicode with a proper encoding header.
• Control parsing timeout per file? Not supported; optimize imports and raise dagbag_import_timeout globally.
• Create DAGs dynamically? Yes, but ensure unique, stable dag_ids and keep generation fast.
• Reduce scheduling latency/task delay? Increase parallelism/workers, lower sensor load with mode="reschedule", and monitor queue age.
• airflow tasks stuck in no_status? Check worker connectivity and executor logs.

If you want deeper orchestration guidance, browse our Airflow & orchestration articles. For hands-on help, we do repo surgery and platform work daily at Vertex—start a project.