Introduce the (internal/db level) concept of "dynamic regions" for TIs
The Issue & Context
Before this change, Airflow could not distinguish between multiple executions of the same task_id within loops or mapped tasks, causing collisions and inability to track distinct iterations after clears or retries.
Root Cause Analysis
Task instances were identified solely by dag_id, task_id, run_id, and map_index, which is insufficient to differentiate multiple passes of loops or dynamic expansions; attempts to rewrite existing rows to add this info were prohibitively slow at production scale.
Solution Strategy & Architectural Pattern
Introduced a new immutable 'dynamic region' table to represent each execution of dynamic constructs like loops or mapped tasks. Task instances now reference a region_id to disambiguate multiple executions of the same task_id. The migration adds storage for regions without rewriting existing rows, using a sentinel zero value for legacy data to avoid long migration times.
Key Takeaway for Contributors
When evolving a database schema to support new execution semantics at scale, adding a dedicated immutable entity to represent dynamic execution contexts can avoid costly data rewrites and enable precise tracking of logically distinct runs.