MWAA Migration: Move Apache Airflow Without Relearning Lessons

Your team has Apache Airflow running and a clock on the move. This MWAA migration guide is the production plan: what moves as‑is, what to redesign, and the exact checks that prevent week‑two outages. We cover DAG and plugin inventory, provider and Python compatibility, networking and IAM, secrets, dependencies, logging, metadata history, schedules, worker sizing, cost, and deployment workflow. You’ll get a parallel‑run cutover and rollback plan you can execute with your own engineers. If you want a second set of hands that’s shipped platforms at Disney, Hulu, Nike, Peloton, Gopuff, and Kaplan, we do that too.

What can move unchanged vs what needs redesign

Most Apache Airflow code will run on Amazon MWAA if it’s pure Python, uses supported providers, and doesn’t assume local disk or shell tools missing on the managed images. The inverse is also true when you migrate off MWAA: anything tied to MWAA’s environment settings or CloudWatch needs a replacement. Use the table below to scope work before touching infra.

Component	Moves as‑is to MWAA	Moves as‑is off MWAA	Notes
DAG Python using core/operators	Usually	Yes	Pin provider versions for compatibility.
Custom plugins	Maybe	Maybe	Package into /plugins; avoid OS‑level deps.
Connections/Variables	Yes	Yes	Prefer Secrets Manager/Env backends.
airflow.cfg settings	Partial	Yes	Only a subset configurable on an MWAA environment.
Local file I/O	No	Maybe	Refactor to S3; MWAA web/worker filesystems are ephemeral.
Shell tools (e.g., gsutil, psql client)	No	Maybe	Package inside tasks or switch to provider hooks.
Webserver custom UI tweaks	Limited	Yes	MWAA restricts webserver customization.
Logs on local disk	No	Maybe	MWAA writes to CloudWatch and S3.
KubernetesExecutor	No	Yes	MWAA does not run your own images; consider Celery/deferrables.

If you’re weighing managed airflow options, we summarized trade‑offs in Managed Airflow Compared: MWAA vs Astronomer vs Composer. Whatever you pick, keep interfaces portable: S3 for artifacts, Secrets backend for auth, and provider hooks over ad‑hoc CLI calls.

DAG and plugin inventory you need before any button clicks

Start with a precise inventory. Count DAGs, tasks, providers used, external systems touched, and any non‑Python dependencies. This is the diff you’ll validate in parallel run.

• List DAGs, schedules, and owners from your repo and metadata database.
• Map imports to providers to pin versions early.
• Catalog secrets used by each DAG and where they come from.
• Note any local file reads/writes to replace with S3.

# Quickly see provider imports used in DAGs
$ grep -R "airflow.providers" -n dags/ | cut -d: -f3 | sort -u

# List DAGs and owners from the metadata DB (Postgres example)
SELECT dag_id, owner, schedule_interval
FROM dag
ORDER BY dag_id;

# Find shell dependencies inside tasks
$ grep -R "BashOperator\|SubProcess" dags/

Snapshot historical performance to set baselines (e.g., P95 task duration) so you can compare before/after in the cutover:

-- P95 runtime by DAG over last 30 days
SELECT ti.dag_id,
       percentile_cont(0.95) WITHIN GROUP (ORDER BY ti.duration) AS p95_s
FROM task_instance ti
JOIN dag_run dr ON dr.dag_id = ti.dag_id AND dr.run_id = ti.run_id
WHERE dr.execution_date > now() - interval '30 days'
  AND ti.state = 'success'
GROUP BY 1
ORDER BY 2 DESC;

Inventory outputs feed your plan: version pins, missing packages, IAM needs, S3 paths, and data reconciliation scope. If you don’t have DAG tests yet, add them now—our checklist in Airflow DAG Testing: Catch Breaks Before Deploy is built for this phase.

Python, providers, and version compatibility on MWAA

Compatibility drives 80% of MWAA surprises. Align your apache airflow version, Python runtime, and provider packages with what Amazon MWAA supports. Use the official constraints file for pins, and upgrade providers that pull in breaking changes.

# Example requirements.txt pins (providers match constraints)
apache-airflow-providers-amazon==<pin>
apache-airflow-providers-snowflake==<pin>
apache-airflow-providers-slack==<pin>
# Your libraries, pinned
pandas==<pin>
requests==<pin>

Then test import and DAG parse locally against the matching constraints:

pip install "apache-airflow==<apache_airflow_version>" \
  -c https://raw.githubusercontent.com/apache/airflow/<tag>/constraints-<python>.txt
pip install -r requirements.txt -c <same-constraints-url>
airflow db init && airflow dags list

Q: How do I migrate from Airflow 2 to Airflow 3? Treat it as a separate upgrade before or after the MWAA move, not during. We published a step‑by‑step production plan in Airflow 3 Migration: A Production Cutover Plan. For apache airflow 2.x to 3.x, pin providers that adopted breaking API changes, and validate deferrable operators. If you must combine both, gate releases with parse tests, unit tests, and a canary environment.

Note: If you’re migrating from Apache Airflow that’s self‑managed, match the apache airflow version and providers first, then re‑point to MWAA. This reduces unknowns in the workflow.

Networking, IAM, and secrets that bite in week two

Most “it worked in dev” failures are networking or IAM. Place the MWAA environment in private subnets with NAT for egress, and set Security Groups to reach data stores (Redshift, Snowflake, Databricks, REST endpoints). Scope the execution role to the S3 bucket for DAGs/plugins, logs, and any data buckets your tasks touch. Cross‑account? Explicitly allow the role’s ARN on the target account resources.

{
  "Version": "2012-10-17",
  "Statement": [
    {"Effect": "Allow", "Action": ["s3:GetObject"], "Resource": [
      "arn:aws:s3:::<dags-bucket>/dags/*",
      "arn:aws:s3:::<dags-bucket>/plugins/*"
    ]},
    {"Effect": "Allow", "Action": ["logs:CreateLogStream","logs:PutLogEvents"],
     "Resource": "arn:aws:logs:<region>:<account>:log-group:/aws/airflow/*:*"},
    {"Effect": "Allow", "Action": ["secretsmanager:GetSecretValue"],
     "Resource": "arn:aws:secretsmanager:<region>:<account>:secret:airflow/*"}
  ]
}

Secrets: Prefer AWS Secrets Manager or Parameter Store via Airflow’s secrets backend. This avoids exporting connections from the metadata database and makes moving off MWAA simpler later. Ensure your VPC endpoints (Interface/ Gateway) cover Secrets Manager, S3, and other services to avoid NAT surprises.

Validation tip: From a debug task, open the same TCP targets your real operators will use. If it can’t connect without a public IP, add the right VPC endpoints or a route via NAT. Treat IAM and egress like code—review and diff them in PRs.

Config, secrets, dependencies, logging, and deployment workflow

Configuration: On MWAA you configure a subset of airflow.cfg via the console/API. For secrets, enable the backend with environment variables and use namespaced keys per team.

# MWAA environment variables
AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"}

# Example Secret JSON for a connection in AWS Secrets Manager
{
  "conn_type": "snowflake",
  "host": "<account>.snowflakecomputing.com",
  "login": "<user>",
  "password": "<pwd>",
  "extra": {"warehouse": "ANALYTICS_WH"}
}

Dependencies: Build a locked requirements file and install via the environment’s requirements path. Logs: Enable CloudWatch for task, scheduler, and webserver logs; optionally send copies to S3 for longer retention. Deployment: store DAGs and plugins in an S3 bucket; changes sync automatically.

# GitHub Actions: validate, then deploy DAGs/plugins and update deps
name: deploy-airflow
on: [push]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements-dev.txt && airflow dags list
      - run: aws s3 sync dags/ s3://<bucket>/dags/ --delete
      - run: aws s3 sync plugins/ s3://<bucket>/plugins/ --delete
      - run: aws mwaa update-environment --name <env> \
               --requirements-s3-path s3://<bucket>/requirements.txt

Observability: Wire failures to on‑call. We outline production alerting patterns in Airflow Monitoring That Warns You Before Users Do. For resilient DAG code and SLAs, see Airflow DAGs — Production DAGs with retries, SLAs, on‑call routing, and observability.

Metadata history, schedules, and backfills: keeping context

You generally don’t copy MWAA’s internal metadata database, so decide what history you need. Best practice: export Connections/Variables (or switch to Secrets), and treat DAG run history as optional. For audit, copy core tables into your warehouse as read‑only lineage.

-- Minimal lineage snapshot into your warehouse
SELECT dag_id, run_id, state, execution_date, start_date, end_date
FROM dag_run
WHERE execution_date > now() - interval '180 days';

Schedules: Ensure every DAG defines schedule, catchup, and default args explicitly. Validate in the Airflow UI after deploy and confirm no DAG is paused unexpectedly. For backfills, run targeted windows with idempotent tasks writing to separate schemas or S3 prefixes.

# Targeted backfill for a monthly DAG
airflow dags backfill --start-date 2025-01-01 --end-date 2025-03-01 my_monthly_dag

# Example idempotent target in SQL (warehouse)
CREATE TABLE IF NOT EXISTS analytics.orders_monthly LIKE analytics.orders_monthly_template;
DELETE FROM analytics.orders_monthly WHERE month = '2025-01-01';
INSERT INTO analytics.orders_monthly
SELECT * FROM staging.orders WHERE order_date >= '2025-01-01' AND order_date < '2025-02-01';

Reality check: if a dbt model processing a 40M‑row orders table on an X‑Small warehouse takes 27 minutes today, your backfill window must reflect that—or you’ll build a days‑long backlog. Measure with your current runs, not guesses.

Worker sizing, performance, and cost modeling on AWS

Right‑size first, then tune. Use queued tasks, scheduler lag, and P95 task durations as signals. If the scheduler is the bottleneck, fix DAG parsing, reduce excessive dynamic task creation, and consider deferrable operators. We detail scheduler triage in Airflow Scheduler Performance: Find the Bottleneck.

How much does MWAA cost?

Model it from your own metrics:

• Environment hours x the class rate (check the official pricing page).
• Worker vCPU/memory hours from scale‑out events.
• CloudWatch logs/metrics storage and ingestion.
• S3 storage for DAGs, plugins, and logs; Amazon S3 request counts.
• NAT/data egress if private subnets pull external resources.

Tag your environment and buckets, then use Cost Explorer by tag to validate. For end‑to‑end pipeline latency, compare old vs new on P95 task and DAG runtimes.

-- Identify slowest tasks (last 7 days)
SELECT dag_id, task_id,
       percentile_cont(0.95) WITHIN GROUP (ORDER BY duration) AS p95_s,
       COUNT(*) AS runs
FROM task_instance
WHERE start_date > now() - interval '7 days' AND state = 'success'
GROUP BY 1,2
ORDER BY p95_s DESC
LIMIT 20;

If your workload is bursty, prefer fewer always‑on workers and a faster scale‑out. If steady, keep workers warm to avoid cold‑start penalties.

Cutover plan: parallel run, validation, and rollback

Run old and new side‑by‑side until the numbers match.

1. Provision a new Amazon MWAA environment in the same AWS account and VPC as prod, but write outputs to a shadow schema/S3 prefix.
2. Deploy with catchup disabled; enable only for controlled backfills.
3. Mirror secrets and IAM. Smoke‑test each external system.
4. Enable a canary set of DAGs. Compare task counts, success rates, and P95s. Reconcile row counts and checksums for critical tables (see Data Migration Reconciliation: Prove the New Platform Matches).
5. Expand to all DAGs. Keep both schedulers running for at least one full business cycle.
6. Cut traffic by toggling schedules on old, enabling on new. Keep old paused for a week.
7. Rollback: re‑enable old schedules and disable new. Keep artifacts versioned so rollback restores code and dependencies together.

Common stakeholder questions

• What is MWAA used for? Orchestrating any workflow that benefits from retries, dependencies, and observability—not just ETL.
• Has AWS abandoned this service? Check release notes and supported versions on the official docs. Validate against your airflow version roadmap.
• What is the Amazon MWAA migration guide? See Amazon’s “Migrate to a new Amazon MWAA environment” doc; use it for service specifics and this guide for production guardrails.
• 7 AWS migration strategies? Map the 7Rs: rehost (lift to MWAA), replatform (standardize on providers), refactor (modularize DAGs), retain/retire (kill unused), repurchase (move a subset to Step Functions), relocate (VPC/account move).

Into or out of MWAA: specifics for both directions

Into Amazon MWAA: focus on provider pins, S3 paths, CloudWatch logging, and Secrets Manager. Treat this as rehosting plus some replatforming. Out of MWAA to self‑managed Apache Airflow (ECS, EKS, or VMs): recreate webserver/scheduler images, port environment variables and secrets backend, and move logs off CloudWatch. Either way, keep artifacts portable: DAGs in an S3 bucket (or equivalent), providers pinned, and minimal reliance on vendor‑specific toggles.

Example DAG portability:

from datetime import datetime
from airflow import DAG
from airflow.providers.amazon.aws.operators.s3 import S3CopyObjectOperator

with DAG(
    dag_id="orders_daily_export",
    start_date=datetime(2025, 1, 1),
    schedule="@daily",
    catchup=False,
    default_args={"retries": 2},
) as dag:
    copy = S3CopyObjectOperator(
        task_id="copy_to_archive",
        source_bucket_name="raw-data",
        source_bucket_key="orders/{{ ds }}/part-0000.snappy",
        dest_bucket_name="archive-data",
        dest_bucket_key="orders/{{ ds }}/part-0000.snappy",
    )

Run the same DAG on managed or self‑managed by swapping environment config only. Validate visually in the Airflow UI and via tests. If you’re moving Airflow to Amazon from on‑prem, treat egress and DNS early—don’t wait for the first failing operator to discover it.

Where the official docs help—and where they don’t

Use official references for service behavior, quotas, supported apache airflow version, and API syntax: the MWAA environment configuration, constraints, and “What’s new” pages. They’ll tell you what Amazon Managed Workflows for Apache Airflow supports today and how to configure it. What the docs won’t do is validate your team’s assumptions, pin your exact providers, or prove idempotency on your data. That’s the job of your inventory, tests, and parallel run.

If you’re evaluating alternatives, scan Managed Airflow Compared and our operator write‑ups on Airflow best practices. For dbt‑heavy DAGs, we show model‑level visibility patterns in Airflow + dbt. For broader warehouse and BI setup, see Business Health Reporting.

---

Want a sanity check on your plan? Send the inventory and target constraints; we’ll point out the risky bits and the fastest parallel‑run path. Or start a scoped engagement: Start a project with Vertex.