fix: always apply evaluator identity keywords and allow explicit domain_id#5989
Open
jam-jee wants to merge 1 commit into
Open
fix: always apply evaluator identity keywords and allow explicit domain_id#5989jam-jee wants to merge 1 commit into
jam-jee wants to merge 1 commit into
Conversation
…in_id Evaluator.create() gated the identity SearchKeywords (@subtype/@evaluatortype/@contenttype) behind `method is not None`. Since `method` is only resolved for RewardFunction, RewardPrompt evaluators were registered with no identity keywords and were therefore invisible in SageMaker Studio (which filters hub content by these keywords). The asset still existed in the hub, so the failure was silent. Always attach the identity keywords regardless of evaluator type or whether a method was resolved; only `method` remains conditional. Also add an optional `domain_id` parameter to Evaluator.create() and DataSet.create(). The @Domain keyword is already auto-injected inside Studio, but the domain cannot be inferred outside Studio (laptop/CI), leaving assets un-scoped and hidden. Callers can now pass domain_id explicitly; auto-detection is skipped when it is provided. Adds regression tests covering RewardFunction and RewardPrompt identity keywords and explicit domain_id for both Evaluator and DataSet. Ref: P467494019
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
Evaluator.create()inconsistently appliesHubContentSearchKeywords, so some evaluators are silently invisible in SageMaker Studio despite registering successfully in the hub (confirmable viadescribe-hub-content). Studio filters hub content by keywords such as@subtype:aws/evaluatorand@evaluatortype:<type>; when they are missing, the asset exists but never renders in the UI, and no error is raised.Internal ref: P467494019.
Root cause
In
Evaluator.create()the identity keywords were gated behindmethod is not None:methodis only resolved forRewardFunction. ForRewardPrompt,methodstaysNone, sotagswas empty and the only keyword ever attached was@domain(and only inside Studio). The identity keywords describe what the asset is and should never depend on whether a method was resolved.Changes
@subtype/@evaluatortype/@contenttype) inEvaluator.create(), regardless of evaluator type. Onlymethodremains conditional (reward prompts have no method).domain_idparameter toEvaluator.create()andDataSet.create().@domainis already auto-injected inside Studio, but the domain cannot be inferred outside Studio (laptop / CI), leaving assets un-scoped and hidden. Callers can now passdomain_idexplicitly; auto-detection is skipped when it is provided. Fully backward compatible (defaults toNone).RewardFunctionandRewardPrompt; explicitdomain_idis tagged and auto-detection is not invoked, for bothEvaluatorandDataSet.Testing
tests/unit/ai_registrypasses (90 tests), including the 3 new regression tests.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
General
Tests