Conversation
* Testing R1 Distills to confirm functional in TransformerLens * Updating order to be alphabetical * Setup StableLM architecture adapter * Resolved weight and qk issues with stablelm. Added more models * Added more models * reformatted * Created a ArchitectureAdapter for OpenElm, handled trusting remote code * Fix formatting * Removed test file, update benchmark * Add mock model test * More benchmark adjustments * removed improperly listed supported models * Updating to resolve existing weight diff issues * began working through issues with exsting architecture benchmarks * Resolve any existing weight folding issues we can possibly resolve * Fixing test failures * Clean up format and other changes * Added text quality benchmark, updated to pass CI * Cleaned up comment, tightened tolerances further for bfloat16 models * Removed unnecessary testing file * Cleanup of redundant code * Resolve type issues and format issues
* created initial model registry tool * fixed some formatting issues * ran format * Updated to resolve merge issues, added verification system with model compatibility reporting - Added batch verification system (verify_models.py) with status codes (0=unverified, 1=verified, 2=skipped, 3=failed) - HF scraper with min_downloads=500 threshold (4,846 models across 20 architectures) - 58 models verified, HF token sanitization in verification notes - Registry data stored in single supported_models.json and verification_history.json files - API, validation, and benchmark tooling updated for new registry format * Model registry updates, including new report generation and alias drift detection features. * Initial pass test of all architecture adapters * Updating the last adapters to ensure successful runs * Type and format fixes before PR --------- Co-authored-by: jlarson4 <jonahalarson@comcast.net>
* created initial model registry tool * fixed some formatting issues * ran format * Updated to resolve merge issues, added verification system with model compatibility reporting - Added batch verification system (verify_models.py) with status codes (0=unverified, 1=verified, 2=skipped, 3=failed) - HF scraper with min_downloads=500 threshold (4,846 models across 20 architectures) - 58 models verified, HF token sanitization in verification notes - Registry data stored in single supported_models.json and verification_history.json files - API, validation, and benchmark tooling updated for new registry format * Model registry updates, including new report generation and alias drift detection features. * Initial pass test of all architecture adapters * Updating the last adapters to ensure successful runs * Type and format fixes before PR * First verification test of the top 10 models for each language. Documenting verification changes via a new page in the docs site. * Fixed initial issues with Gemma models, OLMo2, OpenELM, and Llama. * Additional smaller fixes for other popular models * Added single model run for verify models * Resolving format issues * Updated to include verification of the R1 distills * fixed bug where cfg.d_mlp was sometimes dropped * Additional batch of model verification and verification system improvements * Gemma3 issue resolutions * Updating docs to add a prefix filter * Updating supported models to properly track text quality * Additional Partial verification, added text quality to supported models, improve documentation * Docstring test fix --------- Co-authored-by: Bryce Meyer <bryce13950@gmail.com> Co-authored-by: Jonah Larson <jlarson@Jonahs-MacBook-Pro.local>
* created initial model registry tool * fixed some formatting issues * ran format * Updated to resolve merge issues, added verification system with model compatibility reporting - Added batch verification system (verify_models.py) with status codes (0=unverified, 1=verified, 2=skipped, 3=failed) - HF scraper with min_downloads=500 threshold (4,846 models across 20 architectures) - 58 models verified, HF token sanitization in verification notes - Registry data stored in single supported_models.json and verification_history.json files - API, validation, and benchmark tooling updated for new registry format * Model registry updates, including new report generation and alias drift detection features. * Initial pass test of all architecture adapters * Updating the last adapters to ensure successful runs * Type and format fixes before PR * First verification test of the top 10 models for each language. Documenting verification changes via a new page in the docs site. * Fixed initial issues with Gemma models, OLMo2, OpenELM, and Llama. * Additional smaller fixes for other popular models * Added single model run for verify models * Resolving format issues * Updated to include verification of the R1 distills * fixed bug where cfg.d_mlp was sometimes dropped * Additional batch of model verification and verification system improvements * Gemma3 issue resolutions * Updating docs to add a prefix filter * Updating supported models to properly track text quality * Additional Partial verification, added text quality to supported models, improve documentation * Docstring test fix * Fixed a small bug to ensure MoE weights are properly formatted. Improve memory estimate for verify_models. Add verification for 36 more high value models * Updating docs copy --------- Co-authored-by: Bryce Meyer <bryce13950@gmail.com> Co-authored-by: Jonah Larson <jlarson@Jonahs-MacBook-Pro.local>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Prepping the model registry, adding new architectures
verify_modelsfor running supported models through the benchmark systemDetails
Screenshots
Checklist: