Airflow Best Practices From Someone Who's Been Paged at 3am

You see it in the on-call channel: a DAG stuck on a sensor, a backfill that double-loaded, a task that retries itself into a rate limit. If you want to sleep, design for reliability first. The short list that actually prevents 3am pages: make every task idempotent; split work into small, dependency-clear tasks; use retries with exponential backoff and timeouts; pick deferrable sensors over busy-waiting; route SLAs and alerts to a human who can act; keep secrets out of code; and parse and test DAGs in CI before they ever reach a scheduler. This piece goes deeper than a feature tour. It covers what breaks in production, why it breaks, and the patterns that avoid it. It’s based on platforms I’ve run across Disney/Hulu/Nike/Peloton/Gopuff/Kaplan and the Airflow estates we build at Vertex. If you need help turning this into production Airflow DAGs with real on-call, we do that.

Idempotency is the root requirement

If a task can run twice without changing the final state, the rest of your reliability work gets easier: aggressive retry policies become safe, backfills are boring, and you can restart at any step. Idempotency is not a definition exercise; it’s design. A few concrete patterns:

• Write once per partition. Load into a staging table, then MERGE into the target by partition key. Re-runs update the same rows.
• Derive outputs from inputs plus a stable run key (e.g., data interval start) and avoid wall clock. No ad-hoc NOW() without a parameter.
• For files in S3, write content-addressed keys or overwrite the same key per partition; never append blindly.
• For an external api, store a high-water mark and re-request with overlap, then de-duplicate on insert.

Example Snowflake pattern (40M-row orders for an X-Small is fine if you batch by day). The dag run’s logical date drives partitioning, not the current time:

-- jinja template drives partition boundaries
{% set ds = ds_nodash %}
MERGE INTO analytics.orders AS t
USING staging.orders_{{ ds }} AS s
ON t.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET
  amount = s.amount,
  updated_at = s.updated_at
WHEN NOT MATCHED THEN INSERT (order_id, amount, updated_at)
VALUES (s.order_id, s.amount, s.updated_at);

Expose the same key to downstream tasks through metadata, not ad-hoc strings. In TaskFlow, return the partition date, not a filename baked into python code. For dataset-based triggers, publish a stable Dataset URI per partition. Idempotency also enables strict timeouts: if a task exceeds its runtime budget, kill it and safely retry. That’s the core best practice that lets everything else work.

Task granularity and dependency design (with a before/after DAG)

Overly chunky tasks hide failures, make retries expensive, and stall the scheduler. Thin, composable steps with clear dependency edges are resilient. Keep transforms and side effects separate: validate, stage, merge, then publish. Keep top-level code minimal so imports are cheap when Airflow parses files.

Before: one giant Bash that does everything, no retry safety, hard-coded dates, hidden side effects:

# before.py - monolith, fragile
default_args = {'owner': 'data', 'retries': 0}
with DAG('orders_daily', start_date=datetime(2024,1,1), schedule_interval='@daily', catchup=True, default_args=default_args) as dag:
    run_all = BashOperator(
        task_id='run_all',
        bash_command='dbt run --full-refresh && python enrich.py && python publish.py',
    )

After: explicit steps, idempotent boundaries, deferrable wait, safe task execution, and clear dag code. Note the top level code is nearly empty:

# after.py - smaller, safer
from airflow.decorators import task, dag
from airflow.sensors.filesystem import FileSensor
from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor

default_args = dict(retries=3, retry_delay=timedelta(minutes=5), 
                    depends_on_past=False)

@dag(schedule='@daily', start_date=datetime(2024,1,1), catchup=True, default_args=default_args)
def orders_daily():
    wait_raw = S3KeySensor(task_id='wait_raw', bucket_key='raw/orders/{{ ds }}.json',
                           bucket_name='my-bucket', deferrable=True, timeout=3600)

    @task()
    def validate(ds: str) -> str:
        # fast checks, emit partition key only
        return ds

    @task()
    def stage(partition: str) -> str:
        # load to staging.orders_{{ ds }}
        return partition

    @task()
    def merge(partition: str):
        # MERGE into analytics.orders
        pass

    wait_raw >> validate() >> stage(validate()) >> merge(stage(validate()))

orders_daily()

Two more specifics that pay off:

• Cross-dags dependencies: prefer Datasets over ExternalTaskSensor unless you need precise alignment. Datasets reduce tight coupling between airflow dags.
• Avoid heavy logic at the top level of your dag file. Put work inside callables so imports are cheap during parsing.

If you’re designing your first dag in airflow, resist the urge to collapse steps “for speed.” You’ll debug faster and backfill safer with smaller tasks.

Retries, exponential backoff, timeouts, and SLAs that mean something

Failures in production are often transient: flaky networks, brief warehouse blips, a throttled api. Default to retries with exponential backoff and sane timeouts. Use SLAs to notify humans when data is late, not to auto-terminate tasks.

# default reliability posture
common = dict(
  retries=5,
  retry_delay=timedelta(minutes=3),
  exponential_backoff=True,
  max_retry_delay=timedelta(minutes=30),
  execution_timeout=timedelta(minutes=90),  # kill stuck work
)

with DAG('reliable_example', default_args=common, schedule='0 * * * *', start_date=datetime(2024,1,1), catchup=True) as dag:
    api_pull = SimpleHttpOperator(
       task_id='api_pull', http_conn_id='orders_api', endpoint='/orders', method='GET',
       headers={'X-Request-Id': '{{ ti.dag_id }}-{{ ts_nodash }}'},
    )

Backoff lets capacity recover. But it only works if tasks are idempotent and bounded. Set execution_timeout on every long-running task. If your warehouse job can take an hour, give it 75 minutes and fail hard—don’t let zombie queries run forever. Then your scheduler can move on.

Distinguish permanent from transient errors. Map HTTP 400/422 as fail-fast (no retry), 429/5xx as retryable. Wrap client libraries to surface this consistently. For example:

@task(retries=6, retry_delay=timedelta(minutes=2), exponential_backoff=True)
def pull_page(offset: int):
    r = client.get('/orders', params={'offset': offset})
    if r.status_code in (400, 422):
        raise AirflowFailException('Bad request, fix inputs')
    if r.status_code in (429, 500, 503):
        raise AirflowException('Transient, let Airflow retry')

SLAs belong on the final publish step or the business-facing deliverable, not on every internal hop. Use one SLA to page the human who can fix it. If you need help designing SLAs and routing, we cover alerting patterns in our Airflow articles.

Sensors vs deferrable operators: the worker-slot tax of waiting

The quickest way to starve your executor is to block workers on long-polling sensors. Old-style sensors awake every poke interval, burning a slot. Deferrable sensors hand control to the Triggerer process and free the worker until the condition is met. On Celery or kubernetes executors, that’s the difference between scaling sanity and a queue pileup.

Get this one right:

# bad: busy wait
S3KeySensor(
  task_id='wait_raw', bucket_name='my-bucket', bucket_key='raw/{{ ds }}.json',
  poke_interval=60, timeout=3600, mode='poke'  # occupies a worker
)

# better: reschedule (older pattern)
S3KeySensor(
  task_id='wait_raw', bucket_name='my-bucket', bucket_key='raw/{{ ds }}.json',
  mode='reschedule', timeout=3600
)

# best: deferrable (provider support)
S3KeySensor(
  task_id='wait_raw', bucket_name='my-bucket', bucket_key='raw/{{ ds }}.json',
  deferrable=True, timeout=3600
)

How to measure the cost in your airflow environment:

• In logs, count sensor task start/end times and compute occupied minutes. Multiply by number of concurrent sensors. That’s your slot-minutes burned.
• Watch the Triggerer; make sure it’s sized so deferrable tasks resume promptly.

Also prefer event-driven checks over tight loops: warehouse query complete sensors that poll every 5s look nice, but 60s is fine unless a user is staring at the airflow ui. For cross-pipeline hand-offs, Datasets beat ExternalTaskSensor: producers publish, consumers subscribe, and you avoid DAG import-time coupling.

Bottom line: use deferrable where available, reschedule otherwise, and never block a worker just to wait for a file in s3 or a query to finish.

Alerts that reach a human: owners, SLAs, and routing in practice

Alerts without a reachable owner are noise. Put a person or rotation in owner. Route failures to the right channel from the start, and make the alert include the run context and the next step to try. Keep the alert volume low by alerting on the last mile, not every internal task.

# minimal, useful alerts with Slack & PagerDuty
def slack_alert(context):
    ti = context['ti']
    msg = f"<{ti.log_url}|{ti.dag_id}.{ti.task_id}> failed on {ti.execution_date}. Try rerun if idempotent."
    requests.post(SLACK_WEBHOOK, json={'text': msg})

def pagerduty_alert(context):
    # send only on SLA miss for deliverables
    pass

with DAG('publish_reports', default_args={'owner': 'oncall-analytics'}, schedule='0 7 * * *', start_date=datetime(2024,1,1)) as dag:
    publish = PythonOperator(task_id='publish', python_callable=do_publish, 
                             on_failure_callback=slack_alert, 
                             sla=timedelta(minutes=45))

Use a single SLA on the publish step to trigger escalation when business data is late. Everything upstream can fail fast and retry without waking anyone. If you want richer context, include the upstream health summary in the message (counts, window). We often automate a short log summary using lightweight Artificial intelligence in Slack so the on-call sees the last errors immediately.

Keep alert routes declarative (per DAG) and test them. It’s easy to send a message to nowhere during a rename. We offer incident-ready DAG templates with retries, SLAs, and routing baked in; see production Airflow DAG patterns for how we package this.

Secrets, connections, and configuration you won’t regret

Secrets in code are forever. Use Connections for credentials and a secrets backend for rotation. On MWAA, prefer AWS Secrets Manager; on Composer, Secret Manager; on Astronomer, your cloud’s native store or Vault. This is secrets management, not convenience.

• Never put passwords in a dag file. Reference conn_ids or Variable.get for non-sensitive flags.
• Use airflow variables for knobs, not secrets. Document defaults and audit access.
• Prefer Connection extra JSON for API scopes/headers; don’t hardcode that in python code.

# using a connection instead of inline creds
SimpleHttpOperator(
  task_id='orders_api', http_conn_id='orders_api', endpoint='/orders',
  headers={ 'X-Request-Id': '{{ ts_nodash }}' }
)

For batch jobs on kubernetes, mount credentials via K8s Secrets or Workload Identity; don’t bake them into images. If you need to pass a token, pass a short-lived one. Rotate keys by updating the secret, not the DAG.

Use a jinja template to build paths without leaking values:

output_path = 's3://data-lake/curated/orders/{{ ds }}/part-{{ ti.try_number }}.parquet'

Finally, stable config belongs outside code. If toggling a feature requires a PR, your operators will invent side channels. Put flags into Variables, or better, a small config table in your warehouse keyed by metadata like environment and DAG name. That keeps deploys separate from day-to-day operations.

Testing and CI: make the parser happy before prod

Most production “outages” I’ve seen were boring: import errors, accidental heavy top-level queries, or a bad refactor that broke context. Catch this in CI by letting Airflow parse your DAGs and by running minimal unit tests. If Airflow can’t import your package fast, your web server and scheduler suffer.

• Static parse step: airflow dags list should succeed. That proves the airflow operator imports and your directory structure is sane.
• Enforce “no heavy work at import”: guard everything behind if __name__ == '__main__' in utilities. Never query a database at import.
• Unit test TaskFlow callables and Jinja rendering with a minimal context.

# GitHub Actions snippet to parse on every PR
name: dag-ci
on: [pull_request]
jobs:
  parse:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }
      - run: pip install 'apache-airflow==2.9.*' 'apache-airflow-providers-amazon'
      - run: airflow db init
      - run: airflow dags list  # proves Airflow parses your DAGs

Maintain a near-prod environment (same image, same provider packages, same IAM). Benefits: you catch network and permission failures, mismatched airflow versions, and provider diffs before users do. When you upgrade to airflow 3.0 later, you’ll be glad this exists.

Version and deploy code like any service. On MWAA, sync to the bucket that backs the dag file folder; on Astronomer, push via the CLI; on Composer, commit to the managed repo. Pin provider versions and test upgrades behind a feature branch. We cover repo layout and dbt orchestration patterns in our dbt performance guide.

MWAA vs Astronomer vs Composer: platform differences that change advice

Most Apache Airflow advice travels across platforms, but some choices do matter. Here’s what I look at when picking a managed service:

Area	MWAA	Astronomer	Composer
Upgrades	Slower; Amazon-curated images	Fast; fine-grained control	Moderate; tied to GKE image
Deferrable support	Good with recent providers	Excellent; Triggerer tuning	Good; check image channel
Secrets	AWS Secrets Manager native	Vault or cloud-native	Secret Manager native
Logging	CloudWatch	Cloud logs of your choice	GCS/Cloud Logging
Networking	VPC/IAM easy	Flexible; your VPC	VPC/SAs easy
Executors	Celery/K8s options	Celery/K8s strong	Celery/K8s strong

What changes:

• Secrets: use the platform’s native backend. Don’t roll your own.
• Deferrable: verify your image supports the provider version that has deferrable=True. If not, use mode='reschedule'.
• Deploys: MWAA syncs to an S3 path; Composer watches a GCS folder; Astronomer builds an image. Your CI must match.

Quick answers to recurring Airflow questions

When not to use airflow? If all you need is a single service with its own scheduler (e.g., dbt Cloud jobs), or you lack ops capacity to run a stateful system with a meta DB and workers, pick simpler tooling.

Is Airflow an ETL tool? It’s orchestration for a Directed acyclic graph (workflow), not a transform engine. Use warehouses, Spark, or dbt to process data; Airflow coordinates them.

What is better than Airflow? For some use cases: Dagster for type-checked graphs, Prefect for lightweight flows, serverless schedulers for few jobs. If you need broad provider coverage and scale, Airflow still wins.

Given what you’ve learned, would you still choose Airflow? Yes, when I need many integrations, mature ops, and team familiarity. I’d still start small and grow with Datasets, deferrable sensors, and clear patterns.

How many days to learn Airflow? Don’t count days. Ship one thin DAG, wire alerts, read logs. Then add retries and a sensor. You’ll learn by running it.

How much do you automate DAG task generation? Templating is fine (a simple jinja template), but avoid meta-frameworks that hide edges. Make dag code obvious in reviews.

Sharing data between tasks a bad idea? Small metadata via XCom is fine; large payloads belong in files, tables, or s3, with pointers passed via XCom.

Starting from scratch with 10–20 dags? From day one: idempotency, retries, deferrable sensors, CI parse, owners, and a staging-to-merge pattern. Add observability and SLA routing before users depend on it.

Version and deploy? Git, PRs, CI parse, then deploy to the platform’s DAGs directory or image. Pin providers; track airflow versions.

Or would you pick another orchestrator? If you want fully managed, low infra overhead, and strong typed graphs, evaluate alternatives. If you already run Airflow, lean into the patterns here. Should we use airflow? If you need many integrations, strong scheduling, and team familiarity—yes.

---

If you want a reliability review or help implementing these patterns, start with a 60‑minute working session. We’ll look at your DAGs, imports, sensors, and alerting. See our data quality notes, then book a slot.