Airflow Scheduler Performance: Find the Bottleneck

Your tasks aren’t starting and SLAs are slipping. Stop tweaking knobs at random. Measure time-to-start, confirm whether the scheduler loop is healthy, and isolate whether the slowdown is parsing, database, executor path, limits, or workers. Start by timing from queued to start for recent task instances, then walk the steps below—each gives the exact command, query, or configuration to baseline and verify. If you need a DAG-centric debug path, we covered that separately here: Airflow DAG Not Running: A Systematic Debugging Path. This guide focuses on scheduler performance: a practical path to the actual bottleneck and only changes you can prove improved Airflow performance.

1) DAG parsing: when imports stall scheduling

Slow dag parsing blocks new runs from materializing. Baseline how recently each dag file is parsed, how long imports take, and how many parsing processes you run.

# Last parsed times (PostgreSQL)
psql "$AIRFLOW_DB" -c "
SELECT dag_id, now() - last_parsed_time AS since_last_parsed
FROM dag
WHERE is_active
ORDER BY since_last_parsed DESC
LIMIT 50;"

# Total active DAGs (large number of dags can raise parsing cost)
psql "$AIRFLOW_DB" -c "SELECT count(*) AS number_of_dags FROM dag WHERE is_active;"

# Parse durations from scheduler logs (adjust path)
grep -E "Processing .*\\.py|DagFileProcessor" $AIRFLOW_HOME/logs/scheduler/*/*.log | tail -n 200

# Airflow configuration: parsing knobs
airflow config get-value scheduler parsing_processes
airflow config get-value scheduler min_file_process_interval

It’s the culprit if since_last_parsed grows while few new dag runs appear and logs show the scheduler taking long “Processing … took N seconds” cycles. Offenders: network I/O performed at import time, heavyweight top-level imports, or filesystem latency. Best practices:

• Move I/O into operators; never at module import.
• Lazy-load heavy libraries inside callables.
• Consolidate utilities so each DAG imports fewer modules.

# Bad: network call during DAG import
data = load_catalog_from_s3()  # executed when dag file is parsed
with DAG("etl", start_date=...):
    ...

# Good: defer to runtime
from airflow.operators.python import PythonOperator

def load_catalog_rt(**_):
    return load_catalog_from_s3()

with DAG("etl", start_date=...):
    PythonOperator(task_id="fetch_catalog", python_callable=load_catalog_rt)

Scale only after you measure. If DagFileProcessor workers run back-to-back with little idle and the DB has headroom, raise parsing_processes (the number of processes) gradually and re-check parse lag and time-to-start. Watch storage latency; adding many copies of the scheduler’s parser can overload network filesystems. If you’re planning an upgrade, note that newer releases improved parsing; see our production plan: Airflow 3 Migration: A Production Cutover Plan.

2) Scheduler heartbeat and loop throughput

A healthy loop is predictable: a steady heartbeat, consistent batch size, and no surprise restarts. Prove that before tuning.

# Version and executor context
airflow info | egrep "executor|version"

# Heartbeat health (PostgreSQL)
psql "$AIRFLOW_DB" -c "
SELECT job_id, state, now() - latest_heartbeat AS since_hb
FROM job
WHERE job_type = 'SchedulerJob'
ORDER BY latest_heartbeat DESC
LIMIT 5;"

# Count schedulers (one scheduler or multiple schedulers?)
psql "$AIRFLOW_DB" -c "
SELECT count(*) AS running_schedulers
FROM job
WHERE job_type='SchedulerJob' AND state='running';"

# Loop throughput knobs
airflow config get-value scheduler scheduler_heartbeat_sec
airflow config get-value scheduler max_tis_per_query

Signals of trouble: large spikes in since_hb, very small “Processing X task instances” batches, or unintended multiple schedulers contending. If you meant one scheduler, stop extras; if you intentionally run multiple for performance reasons and for resiliency, ensure unique IDs and proper DB pooling. Check load on the scheduler host to rule out throttling:

# Linux host utilization
ps -C airflow -o pid,%mem,%cpu,cmd | grep SchedulerJob

# Kubernetes
kubectl top pod -n <ns> | grep scheduler

If the loop is constrained (e.g., logs show the scheduler taking many seconds while touching few tasks), raise max_tis_per_query stepwise and confirm bigger batches without increasing database waits. Newer schedulers in Airflow 2.x and Airflow 3 improved the core loop and priorities; upgrade only after you’ve baselined so you can verify improvement. For ongoing alerting, wire the above into your monitors; we show concrete alerts here: Airflow Monitoring That Warns You Before Users Do. This is the heart of scheduler performance; optimize only with proof.

3) Metadata database load: confirm or clear it

When Postgres is saturated, everything looks slow. Confirm the constraint with latency, waits, and plans—then change one factor at a time.

# Time-to-start across recent running tasks
psql "$AIRFLOW_DB" -c "
SELECT ti.dag_id, ti.task_id,
       EXTRACT(EPOCH FROM (ti.start_date - ti.queued_dttm)) AS start_delay_s
FROM task_instance ti
WHERE ti.state = 'running' AND ti.start_date IS NOT NULL
ORDER BY start_delay_s DESC
LIMIT 20;"

# Backlog snapshot
psql "$AIRFLOW_DB" -c "SELECT state, count(*) FROM task_instance GROUP BY state;"

# PostgreSQL waits
psql "$AIRFLOW_DB" -c "
SELECT wait_event_type, wait_event, count(*)
FROM pg_stat_activity
WHERE state <> 'idle'
GROUP BY 1,2
ORDER BY 3 DESC;"

# Lock contention in scheduler critical sections
psql "$AIRFLOW_DB" -c "
SELECT locktype, mode, granted, count(*)
FROM pg_locks
GROUP BY 1,2,3
ORDER BY 4 DESC;"

It’s the bottleneck if start_delay is high while workers aren’t busy and waits or locks pile up. The scheduler evaluates each dependency and writes state transitions; slow plans against task_instance or dag_run multiply delay. Actions to consider:

• Collect EXPLAIN ANALYZE for slow queries; confirm btree indexes on common filters such as (dag_id, run_id, state), (dag_id, state), (state, queued_dttm).
• Right-size vCPU and storage IOPS; then re-run the latency SQL to verify improvement.
• Enable pg_stat_statements and rank by total_time for hotspots.
• Set a sane statement_timeout to fail fast on pathological queries during backfills.
• After upgrades, run airflow db check to ensure all migrations applied.

During backfills, stagger runs to cap the number of tasks created per second. If metadata writes spike, even fast hardware suffers. The airflow documentation describes known tradeoffs; we focus on the measurements that show which knob to touch first.

4) Executor path: when submission lags behind scheduling

The scheduler and worker can run at different speeds. If the scheduler marks tasks runnable faster than the executor path can submit them, starts are delayed. Identify your executor and inspect its back end.

# What executor is configured?
airflow config get-value core executor

# Celery: worker liveness and backlog depth
airflow celery status
redis-cli llen celery           # Redis depth (if used)
rabbitmqctl list_queues name messages  # RabbitMQ depth

# Kubernetes: pod scheduling pressure
kubectl get pods -n <ns> | egrep "Running|Pending|Error"
kubectl describe pod <pod> -n <ns> | egrep -i "event|limit|throttle"

Clues: large broker backlogs, many Pending pods due to resource quotas, or API errors in logs. For Celery, match worker_concurrency to your workload type. For KubernetesExecutor, set realistic CPU/memory requests so pods place quickly and ensure namespace quotas permit bursts.

# Celery knob (measure before/after)
airflow config get-value celery worker_concurrency

# K8s worker pod template (avoid throttling)
resources:
  requests:
    cpu: "1"
    memory: "1Gi"
  limits:
    cpu: "2"
    memory: "2Gi"

Verify by watching backlog depth shrink and start_delay fall. If submission is healthy but starts still lag, return to database or limits. If you’re evaluating hosting, we summarized sane defaults and tradeoffs in Managed Airflow Compared: MWAA vs Astronomer vs Composer. Note: Some teams try running multiple schedulers to push more submissions; don’t mask an overloaded broker—fix the executor path first.

5) Pools and priority: prevent starvation

Pools are safety rails, but tiny defaults can block unrelated pipelines. Inventory pools, find where tasks are waiting, and resize intentionally.

# Inventory pools and slots
airflow pools list

# Waiting tasks by pool (PostgreSQL)
psql "$AIRFLOW_DB" -c "
SELECT COALESCE(pool, 'default_pool') AS pool, count(*) AS waiting
FROM task_instance
WHERE state = 'queued'
GROUP BY 1
ORDER BY waiting DESC;"

# Adjust slots (example)
airflow pools set analytics 20  # increase this value after verifying headroom

Review DAG code for explicit pools and priority_weight. By default, tasks land in default_pool, which may be too small for your daily workload. Separate backfills and heavy fact builds into their own pools so fresh SLAs aren’t blocked. Example:

from airflow import DAG
from airflow.operators.bash import BashOperator

with DAG(
    dag_id="warehouse_build",
    start_date=..., schedule_interval="0 * * * *",
    max_active_runs=1,
) as dag:
    build = BashOperator(
        task_id="rebuild_facts",
        bash_command="python run_fact_build.py",
        pool="analytics",
        priority_weight=5,
    )
    notify = BashOperator(
        task_id="notify",
        bash_command="python notify.py",
        pool="default_pool",
        priority_weight=1,
    )

Re-run the pool backlog SQL and confirm slots are consumed where pressure exists. If wait times persist despite free slots, return to per-DAG concurrency or the executor path. For production patterns beyond pools—retries, SLAs, and on-call routing—see Airflow DAGs — Production DAGs with retries, SLAs, on-call routing, and observability and our field notes in Airflow Best Practices From Someone Who's Been Paged at 3am.

6) Parallelism and per-DAG limits: hidden caps

Global and per-DAG caps can hold the system back even when workers are idle. Audit configuration options and tie each change to a measurable goal.

# Global caps (airflow configuration)
airflow config get-value core parallelism
airflow config get-value scheduler max_active_runs_per_dag
airflow config get-value scheduler max_active_tasks_per_dag

# Per-DAG runtime view (PostgreSQL)
psql "$AIRFLOW_DB" -c "
SELECT dag_id, count(*) AS running_runs
FROM dag_run
WHERE state='running'
GROUP BY dag_id
ORDER BY running_runs DESC;"

# System snapshot of running vs waiting
airflow jobs check  # ensure scheduler job is healthy
psql "$AIRFLOW_DB" -c "SELECT state, count(*) FROM task_instance WHERE state in ('running','queued') GROUP BY 1;"

Symptoms: exactly N active runs per DAG, or a small, flat number of tasks system-wide regardless of capacity. These settings bound the number of tasks that can be running or queued. Prefer DAG-level overrides when one Directed acyclic graph needs more headroom without flooding others.

# DAG-level overrides
from airflow import DAG
from datetime import datetime

dag = DAG(
    dag_id="orders_etl",
    start_date=datetime(2024,1,1),
    schedule_interval="*/15 * * * *",
    max_active_runs=2,     # per-DAG dag run cap
    concurrency=16,        # task instances allowed to run concurrently in this DAG
)

Change one knob at a time and re-check running_runs and waiting counts. Raise parallelism only after you confirm the database and executor can absorb the extra load. Document the before/after so you can reproduce the improvement in another airflow environment. This is classic performance tuning—tune the performance by measurement, not hunch.

7) Dynamic task mapping: control fan‑out and metadata churn

Dynamic mapping is powerful, but thousands of short Airflow tasks per run can drown the control plane. The scheduler parses mapping metadata each loop; oversized fan‑out increases per‑cycle work and database writes.

# Mapped instances per run (PostgreSQL)
psql "$AIRFLOW_DB" -c "
SELECT dag_id, run_id, task_id, count(*) AS mapped_count
FROM task_instance
WHERE map_index IS NOT NULL
  AND state IN ('queued','running')
GROUP BY 1,2,3
ORDER BY mapped_count DESC
LIMIT 20;"

It’s the issue if a few runs have extreme mapped_count, start delays are high, and workers have capacity. Strategies that hold up under load:

• Chunk upstream inputs and map over batches instead of single items.
• Use pools to bound mapped concurrency per DAG.
• Prefer fewer, slightly longer tasks to reduce per–task instance overhead.

# Example: batch mapping input
from airflow.decorators import task

def chunk(seq, size):
    for i in range(0, len(seq), size):
        yield seq[i:i+size]

@task
def expand_inputs():
    items = list(range(0, 10000))
    return list(chunk(items, 100))  # 100 batches vs 10k items

@task(pool="analytics")
def process_batch(batch):
    ...

process_batch.expand(batch=expand_inputs())

Re-run the mapped_count SQL and profile Postgres again; you should see fewer writes and shorter loops. If you truly need extreme fan‑out, profile the metadata database and executor path together and consider whether a streaming system fits that portion of the workflow better.

8) Worker capacity and a focused FAQ

Sometimes the constraint is simple: workers are full. Confirm usage and contention on compute, memory, storage, and network throughput.

# Kubernetes: actual usage
kubectl top pods -n <ns> | grep worker
kubectl top nodes

# Celery workers: liveness and processes
airflow celery status
ps aux | egrep "celery|worker" | grep -v grep

# Linux host basics
uptime
vmstat 1

Clues: high CPU usage with flat throughput, storage waits in vmstat, or network saturation during large transfers. For Python-bound compute, scale out more small airflow workers to limit GIL contention. For I/O-bound work, increase worker_concurrency modestly and verify downstream systems keep up. Always re-check start_delay and completed tasks per minute after each change.

Quick answers to common questions

• Apache Airflow scheduler — what’s it for? It evaluates each dependency, enforces concurrency, and submits runnable work; if prerequisites aren’t met, the scheduler won’t submit the task instance.
• New vs old scheduler? Airflow 2.x introduced a more efficient core loop and the mini-scheduler inside workers; Airflow 3 continues this. Curious what changed? Check release notes for your airflow version and talks from Airflow Summit.
• Are there architectural limits to thousands of short-living tasks? Yes—metadata writes and per-cycle evaluation cost. Batch, chunk, or offload micro-tasks.
• Why aren’t task‑instance dependencies fully materialized in the database? To avoid multiplying writes; the scheduler computes minimal state each cycle.
• How much RAM does Airflow need? Measure your own airflow environment at peak (kubectl top/ps) and size with headroom.
• Is Airflow a good ETL tool? It’s an orchestrator; push heavy compute to your warehouse or Spark and keep Airflow lightweight.
• Does Airbnb still use Airflow? The project began there and is now a top‑level Apache project used widely in production.
• Should I run multiple schedulers? For HA, yes—but document it and size the database. If you intended only one scheduler, shut extras down.

---

Need help to optimize and stabilize your scheduler quickly? Start with battle-tested alerts: Airflow Monitoring That Warns You Before Users Do. For a cutover plan that won’t break mornings, read Airflow 3 Migration: A Production Cutover Plan. If you want us to map your bottleneck and ship fixes this week, talk to Vertex.