Airflow Monitoring That Warns You Before Users Do

Your users don’t care that a task turned green. They care that their dashboard is current at 8am. Effective airflow monitoring focuses on what breaks in production: the control plane’s heartbeat, parse time drift, queue depth and pool saturation, realistic SLAs, and end-to-end data freshness. Use one metrics path, build dashboards that answer pager questions, and route alerts with clear ownership. Most teams stop at “task succeeded.” The bar is “the data is right.” Below is a field-tested checklist, queries, and code to monitor workflows end-to-end and get a notification before a stakeholder pings you. We’ll also show how to configure metrics and wire alerting to Slack, with severity tiers and owners who can fix it now.

Scheduler health and parse times: the pulse of the system

The scheduling service is where subtle degradation starts. Watch one key metric first: heartbeat age. Then track parse duration, import errors, and the backlog of files under dags/. Real symptoms: runs stop being created, new code from GitHub never appears, or the Airflow UI feels stale despite a recent deployment.

Airflow components to watch as a set: the controller (scheduler), executor, webserver, and metadata database. If the web interface exposes health, a quick probe catches stalls behind a reverse proxy like NGINX or Apache HTTP Server:

curl -s http://<webserver>/health | jq .
# Expect scheduler: healthy, metadatabase: healthy

Resource pinning matters. Starving CPU steals time from parsing, which delays new DAG runs and hides failures. After each deploy, compare P95 parse time to last week’s baseline; spikes often come from slow imports or heavy top-level Python. If you use a managed service (see Managed Airflow compared), still treat the controller as production infrastructure. For runbook patterns we use in practice, see Airflow best practices from someone who’s been paged at 3am.

Task duration drift and meaningful SLAs

Production breakage often starts as slow, not failed. A query that used to take 3 minutes now takes 14, making the whole pipeline late. Track duration versus historical medians per task and alert on deltas, not just failures. Pair this with SLAs so the system tells you the moment a critical run is late for its contract.

Set an SLA on critical work and use the operator’s callback for routing:

# flow_sla.py
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator

def compute():
    return "ok"

def on_sla_miss(dag, task_list, blocking_task_list, slas, *args, **kwargs):
    # send a high-severity notification with owners/runbook
    print(f"SLA miss on {task_list}")

with DAG(
    dag_id="daily_revenue",
    start_date=datetime(2024, 1, 1),
    schedule_interval="0 6 * * *",
    default_args={"owner": "analytics"},
    sla_miss_callback=on_sla_miss,
    catchup=False,
) as flow:
    t = PythonOperator(
        task_id="compute_revenue",
        python_callable=compute,
        sla=timedelta(minutes=30),
        retries=2,
        retry_delay=timedelta(minutes=5),
    )

Alert if the current duration exceeds, say, 2x the 30‑day median for that task. Don’t guess—compute your own baseline from historical task instances:

-- task_duration_baseline.sql
SELECT task_id,
       percentile_disc(0.5) WITHIN GROUP (ORDER BY duration) AS p50,
       percentile_disc(0.95) WITHIN GROUP (ORDER BY duration) AS p95
FROM (
  SELECT task_id,
         EXTRACT(EPOCH FROM (end_date - start_date)) AS duration
  FROM task_instance
  WHERE start_date > now() - interval '30 days'
    AND state = 'success'
) d
GROUP BY task_id;

Queues, pools, and executor backlog

Healthy runs get created, but nothing starts? That’s queue depth and pool saturation. Monitor queued vs running counts per queue, free slots per pool, and the time a task spends in the queued state. When concurrency settings mismatch worker capacity, you’ll see growing delays with no obvious errors—and then task failures as timeouts kick in.

Build a backlog panel straight from the metadata database and page on drift:

-- queued_vs_running.sql (Postgres-like; adjust date funcs)
WITH recent AS (
  SELECT state, queue, pool, now() - start_date AS age
  FROM task_instance
  WHERE start_date > now() - interval '2 hours'
)
SELECT 
  queue,
  pool,
  SUM(CASE WHEN state='queued' THEN 1 ELSE 0 END) AS queued,
  SUM(CASE WHEN state='running' THEN 1 ELSE 0 END) AS running,
  percentile_disc(0.95) WITHIN GROUP (ORDER BY age) FILTER (WHERE state='queued') AS p95_queued_age
FROM recent
GROUP BY 1,2
ORDER BY queued DESC;

Also watch executor errors (e.g., lost Kubernetes pods) and whether pools reflect reality. Over-commit and you thrash; under-commit and CPUs idle while SLAs slip. If “runs created, nothing starts” repeats after a deployment, compare worker capacity to the number of concurrent DAGs and tasks you allow.

End-to-end data freshness: the data is the product

Dag monitoring isn’t enough. “The task succeeded” is different from “the data is right.” Validate freshness and shape where the tables land. Track max(updated_at), row deltas, and key null rates. If you use dbt, wire tests and exposures into the orchestrator so late or wrong tables fail fast, and make freshness part of your data pipelines—not an afterthought.

Warehouse-side checks that run after the last transformation:

-- freshness_check.sql (Snowflake/BigQuery/Redshift: adapt syntax)
SELECT 
  DATE_DIFF('minute', MAX(updated_at), CURRENT_TIMESTAMP) AS minutes_stale
FROM analytics.daily_orders;

# models/daily_orders.yml (dbt)
version: 2
models:
  - name: daily_orders
    tests:
      - not_null:
          column_name: order_id
    freshness:
      warn_after: {count: 60, period: minute}
      error_after: {count: 120, period: minute}

Trigger alerts on freshness misses and suspicious row-count swings versus trailing medians. Details on running dbt from Airflow are in Airflow + dbt: run dbt with model-level visibility, and a full testing plan in a data quality testing strategy that catches real problems. If you want us to orchestrate this and own on-call, see our production Airflow DAGs services.

Metrics plumbing: from Airflow to Prometheus and Grafana

Pick one path for telemetry and keep it consistent across environments. A simple, proven path is Airflow → StatsD exporter → Prometheus → Grafana. Keep logs in their own system; metrics should be cheap to emit and fast to scrape. Use the same dashboard JSON in dev/stage/prod to make drift obvious.

Enable metrics in your Airflow configuration and configure collection once per environment:

# airflow.cfg (excerpt)
[metrics]
statsd_on = True
# point at your metrics gateway; scrape from Prometheus

Scrape your exporter with Prometheus and build panels in Grafana (Grafana Labs has good examples). Use tags for team/owner to support alert routing. The interface between metrics and alerts should be boring: a panel per failure mode, an alert per panel, and documented thresholds based on your own baselines—not guesses from a blog.

Dashboards that earn on-call trust

Every panel should answer a pager question in one glance. Build a top-level board ordered by “how production fails,” with drill-downs for flows, queues, and pools. Add annotations for deploys and controller restarts; they help correlate spikes with change velocity. When digging into a single run, cross-check the Graph view in the Airflow UI to confirm dependency order aligns with what your time-series panels show.

Panel	What it shows	Investigate when
Heartbeat age	Seconds since last pulse	Climbs past baseline; runs stop being created
Parse P95	95th percentile parse time	Spikes after a code change; new files missing
Task duration ratio	Current / 30‑day median	> 2.0 for critical tasks; SLAs at risk
Queued vs running by queue	Backlog and throughput	Queued grows, running flat; adjust capacity
Pool slots free	Free vs total slots	Near zero; long waits despite idle cluster
SLA misses (count)	Misses by flow/task	Trend rises; revisit schedules/dependencies
Data freshness (minutes)	Key tables vs targets	Stale beyond contract; escalate before users ping

Establish your own thresholds: compute 30‑day medians and p95s in the metadata store and alert on deviations. For a systematic triage path when a run won’t start, see Airflow not running: a debugging path.

Alert routing, severity, and ownership

Severity tiers prevent paging on noise. A solid split: P1 for controller down or freshness breach on tier‑1 tables; P2 for sustained queue saturation or repeated SLA misses; P3 for a single flow failure without user impact. Route P1 to pager and Slack, P2 to Slack with working-hours pager, P3 to Slack only. Every alert must have an owner and a runbook link, and your notification text should include the run URL and last few log lines.

Put ownership in code so it survives rotation and integrate alerting at the operator level:

# defaults in each flow file
default_args = {
  "owner": "data-platform",
  "email_on_failure": False,
}

def on_fail(context):
    ti = context["ti"]
    url = ti.log_url  # task log link
    msg = f"<{url}|{ti.dag_id}.{ti.task_id}> failed on {ti.execution_date}"
    # send to #data-oncall via your chat API

# task example
from airflow.operators.python import PythonOperator
failing = PythonOperator(
    task_id="maybe_slow",
    python_callable=lambda: 1/0,
    on_failure_callback=on_fail,
)

Keep an owner map for flows and for shared resources like pools and queues. When multiple teams deploy to the same platform, on-call shouldn’t guess who fixes what.

FAQ: short, honest answers

What is Apache Airflow?

Open-source software to orchestrate Directed Acyclic Graphs of tasks for analytics and machine learning.

How does Airflow work?

A controller creates runs from schedules, an executor starts tasks on workers, and a metadata database tracks state; the webserver surfaces it.

What are DAGs in Airflow?

Python-defined graphs that declare order and dependencies. You version them, test them, and they appear as DAG runs in the UI.

Why monitor Airflow?

Because “green tasks” can still deliver stale or wrong data. Monitor workflows for control-plane health, queues, SLAs, and freshness.

How can the StatsD metrics be sent to Prometheus?

Run a StatsD exporter, point Airflow’s client at it, and have Prometheus scrape the exporter; build panels in Grafana.

How can New Relic help you better monitor your stack?

Use it for host/container telemetry and log aggregation; keep core Airflow metrics in your primary time‑series system to avoid split‑brain.

Considering Wireless Airflow Monitoring?

That’s HVAC—wireless air flow monitors. Different domain; this page is about Apache Airflow.

---

If you want this implemented—metrics wired, dashboards built, alerts routed, and freshness checks proven—Vertex can help. Start with a quick assessment or have us own the build. Talk to us.