Apache Airflow Best Practices (3.2.0)

DAG Authoring

Imports (Airflow 3.x)

• Use from airflow.sdk import DAG, task, Asset — the stable public API
• Legacy from airflow.models import DAG and from airflow.decorators import task are deprecated

Top-Level Code (CRITICAL)

• Avoid heavy computation at module level (executed on every DAG parse)
• Minimize imports at module level — lazy-load inside @task functions
• Never call APIs, query databases, or access Variables at top level
• If Variables needed at top level, enable experimental cache with TTL

TaskFlow API (Default Pattern)

• Use @task decorator for all Python tasks (preferred over classic operators)
• XCom serialization is automatic — return values become XCom
• Use @task.branch for branching logic
• Use @task.sensor for sensor tasks

Dynamic Task Mapping

• Use task.expand() for runtime-determined task instances
• Combine with .partial() for fixed kwargs
• Map over lists, dicts, or XCom outputs from upstream tasks

Scheduling

• Use cron expressions or timetables for schedule parameter
• Set catchup=False for most DAGs
• Use data-aware scheduling with Asset (replaces Dataset) for dependencies
• Configure SLA monitoring

Task Dependencies

• Use >> / << operators for clarity
• Group related tasks with TaskGroup
• Avoid deep nesting (max 3 levels)

Testing

Local Testing

• Use dag.test() in if __name__ == "__main__": block for IDE debugging
• Runs all tasks in single serialized process without executor

Unit Tests

• Test DAG import without errors
• Detect cycles in dependencies
• Mock external connections
• Test task logic independently

Integration Tests

• Use Airflow test mode
• Validate end-to-end workflows
• Test with sample data

Production Deployment

Performance

• Lazy-load heavy libraries inside @task functions
• Use connection pooling
• Minimize DAG parse time (target < 30s for all DAGs)
• Enable parallelism appropriately

Reliability

• Set appropriate retries and retry_delay
• Use SLA callbacks for monitoring
• Implement proper error handling with on_failure_callback
• Log important events

Migration: 2.x → 3.x

Deprecated (Remove or Replace)

Deprecated	Replacement
from airflow.models import DAG	from airflow.sdk import DAG
from airflow.decorators import task	from airflow.sdk import task
Dataset	Asset
execution_date in context	dag_run.logical_date
conf in task context	Removed — use Variables or params

Architecture Changes

• AIP-72: Task Execution Interface — tasks run in isolated subprocesses via Execution API Server
• AIP-44: Internal API — components communicate via API, not direct DB access
• New UI: React-based web interface (replaces Flask-based UI)

References

• Airflow 3.2.0 Best Practices
• Airflow SDK (Task SDK)
• Migration Guide 2.x → 3.x