unlock unlimited code runs for $79, limited time →
NOSSLETTER.techLIVE
back to archive

Introduce the (internal/db level) concept of "dynamic regions" for TIs

Language: Python+1060 -81feature
concepts demonstrated
Database schema designImmutable data structuresTask instance identificationHierarchical execution tracking
The Issue & Context

Before this change, Airflow could not distinguish between multiple executions of the same task_id within loops or mapped tasks, causing collisions and inability to track distinct iterations after clears or retries.

Root Cause Analysis

Task instances were identified solely by dag_id, task_id, run_id, and map_index, which is insufficient to differentiate multiple passes of loops or dynamic expansions; attempts to rewrite existing rows to add this info were prohibitively slow at production scale.

Solution Strategy & Architectural Pattern

Introduced a new immutable 'dynamic region' table to represent each execution of dynamic constructs like loops or mapped tasks. Task instances now reference a region_id to disambiguate multiple executions of the same task_id. The migration adds storage for regions without rewriting existing rows, using a sentinel zero value for legacy data to avoid long migration times.

Key Takeaway for Contributors

When evolving a database schema to support new execution semantics at scale, adding a dedicated immutable entity to represent dynamic execution contexts can avoid costly data rewrites and enable precise tracking of logically distinct runs.