Auto-set ZARR3 and OCDBT to False for Pathways in config#3358
Draft
igorts-git wants to merge 1 commit intomainfrom
Draft
Auto-set ZARR3 and OCDBT to False for Pathways in config#3358igorts-git wants to merge 1 commit intomainfrom
igorts-git wants to merge 1 commit intomainfrom
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
e0d77a0 to
36ad42d
Compare
36ad42d to
fc5c0a9
Compare
|
🤖 Hi @igorts-git, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
There was a problem hiding this comment.
This pull request correctly centralizes the logic for disabling ZARR3 and OCDBT when using Pathways (enable_single_controller=True) into the configuration system. This simplifies command-line invocations by removing the need to manually set these flags and ensures consistent default behavior across the codebase.
🔍 General Feedback
- The architectural choice to move the configuration overrides to a single source of truth in
types.pyis sound and enhances maintainability. - The unit tests clearly and effectively verify that the overridden defaults behave as expected, covering both the default scenario and the edge case where
colocated_python_checkpointingis set toTrue. - Good job updating the documentation and standalone scripts to reflect the new unified approach.
| # To create a scanned checkpoint, you can use `src/maxtext/checkpoint_conversion/to_maxtext.py` and if | ||
| # using Pathways, please set `enable_single_controller=True` | ||
| # python src/maxtext/checkpoint_conversion/to_maxtext.py \ | ||
| # maxtext/configs/base.yml \ |
There was a problem hiding this comment.
🟢 The path to the config file might need to be prefixed with `src/` to match the Python script execution path, assuming the user is running this from the repository root.
Suggested change
| # maxtext/configs/base.yml \ | |
| # src/maxtext/configs/base.yml \ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Address b/490520104: ZARR3 and OCDBT flags must be manually set to False for Pathways (should be default).
When
enable_single_controller==Truewe automatically set ZARR3 and OCDBT flags to False.One potentially controversial change as part of this PR is that the checkpoint conversion script
to_maxtextnow needs to be explicitly toldenable_single_controller=True. I did not see a way around it.FIXES: b/490520104
Pair-programmed with Gemini-CLI.
Tests
Added a unit test. Manually ran a tiny Pathways workload.
Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.