Migration 0094 upgrade is slow at scale due to row-by-row Python deserialization of deadline callbacks #63532
Description
Apache Airflow version
main (development)
If "Other Airflow 3 version" selected, which one?
No response
What happened?
Migration 0094_3_2_0_replace_deadline_inline_callback_with_fkey processes every row in the deadline table through Python's serde.deserialize() during upgrade. For each row it deserializes the JSON callback, instantiates the Python object, re-serializes it into the new format, then writes to the callback table.
With a large number of deadline rows this becomes very slow because the bottleneck is Python-side object instantiation — not the database I/O. The migration processes rows in batches of 1000, but within each batch every row goes through:
callback_data = deserialize(deadline.callback).serialize() | {
"prefix": CALLBACK_METRICS_PREFIX,
"dag_id": deadline.dag_id,
}
This round-trip through Python deserialization/serialization is unnecessary given that the callback JSON format is predictable. Prior to 3.2.0 the only supported callback type was AsyncCallback, which has a fixed structure
{
"__data__": {"path": "...", "kwargs": {...}},
"__classname__": "airflow.sdk.definitions.deadline.AsyncCallback",
"__version__": 0
}
Observed impact:
10M deadline rows: ~33 minutes
The migration holds a transaction open for the entire duration
Proportional to deadline row count — 1M rows would still take ~3 minutes
What you think should happen instead?
The transformation could be done entirely in SQL by extracting path and kwargs from the existing JSON and constructing the new callback row format without ever touching Python.
I think we can do like the way its been done in https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/migrations/versions/0085_3_1_0_downgrade_serialized_dag_version_to_v2.py
How to reproduce
Steps to reproduce:
- Start Airflow 3.1.8 with PostgreSQL backend
- Create a significant number of DAG runs with deadlines (or insert synthetic data into the deadline table with valid serde-formatted callback JSON)
- Upgrade to 3.2.0 (airflow db migrate)
- Observe that migration 0094 takes several minutes per million deadline rows
Operating System
Mac
Versions of Apache Airflow Providers
No response
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct