fix: nested non-DAG with-statement prematurely exits DAG context in version inflation checker#68795
Open
kalluripradeep wants to merge 2 commits into
Conversation
…ersion inflation checker AirflowRuntimeVaryingValueChecker.visit_With() called exit_dag_context() unconditionally at the end of every with-statement. When a non-DAG with-statement (e.g. `with open(...) as f`) was nested inside a DAG with-block, it would set is_in_dag_context=False, causing any tasks constructed AFTER the nested with — but still inside the DAG block — to be invisible to the checker. Those tasks were never flagged for runtime-varying values even when they used datetime.now(), random(), etc. Fix: guard the exit_dag_context() call with `if is_with_dag_context` so only the with-statement that actually entered the DAG context exits it. Regression tests added for: - nested non-DAG with followed by a task (primary case) - multiple successive nested non-DAG withs - task constructed inside a nested non-DAG with - task outside the DAG block not over-flagged after the fix
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #68796
What's the problem?
AirflowRuntimeVaryingValueChecker.visit_With()calledexit_dag_context()unconditionally at the end of everywith-statement. When a non-DAGwith-statement (e.g.with open(...) as f) was nested inside a DAGwith-block, it would resetis_in_dag_context=False. Any tasks constructed after the nestedwith— but still inside the DAG block — were invisible to the checker and never flagged for runtime-varying values even when they useddatetime.now(),random(), etc.Root cause
Fix
Guard the call with
if is_with_dag_contextso only thewith-statement that actually entered the DAG context exits it.Changes
airflow-core/src/airflow/utils/dag_version_inflation_checker.py— one-line guardairflow-core/tests/unit/utils/test_dag_version_inflation_checker.py— 4 regression tests