Skip to content

fix: always apply evaluator identity keywords and allow explicit domain_id#5989

Open
jam-jee wants to merge 1 commit into
aws:masterfrom
jam-jee:fix/evaluator-hubcontent-search-keywords
Open

fix: always apply evaluator identity keywords and allow explicit domain_id#5989
jam-jee wants to merge 1 commit into
aws:masterfrom
jam-jee:fix/evaluator-hubcontent-search-keywords

Conversation

@jam-jee

@jam-jee jam-jee commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Issue

Evaluator.create() inconsistently applies HubContentSearchKeywords, so some evaluators are silently invisible in SageMaker Studio despite registering successfully in the hub (confirmable via describe-hub-content). Studio filters hub content by keywords such as @subtype:aws/evaluator and @evaluatortype:<type>; when they are missing, the asset exists but never renders in the UI, and no error is raised.

Internal ref: P467494019.

Root cause

In Evaluator.create() the identity keywords were gated behind method is not None:

if method is not None:
    tags = [
        (TAG_KEY_METHOD, method.value),
        ("@subtype", ...), ("@evaluatortype", ...), ("@contenttype", ...),
    ]
else:
    tags = []

method is only resolved for RewardFunction. For RewardPrompt, method stays None, so tags was empty and the only keyword ever attached was @domain (and only inside Studio). The identity keywords describe what the asset is and should never depend on whether a method was resolved.

Changes

  • Always attach the identity keywords (@subtype / @evaluatortype / @contenttype) in Evaluator.create(), regardless of evaluator type. Only method remains conditional (reward prompts have no method).
  • Add an optional domain_id parameter to Evaluator.create() and DataSet.create(). @domain is already auto-injected inside Studio, but the domain cannot be inferred outside Studio (laptop / CI), leaving assets un-scoped and hidden. Callers can now pass domain_id explicitly; auto-detection is skipped when it is provided. Fully backward compatible (defaults to None).
  • Regression tests: identity keywords always present for both RewardFunction and RewardPrompt; explicit domain_id is tagged and auto-detection is not invoked, for both Evaluator and DataSet.

Testing

tests/unit/ai_registry passes (90 tests), including the 3 new regression tests.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

General

  • I have read the CONTRIBUTING doc
  • I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
  • I used the commit message format described in CONTRIBUTING
  • I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
  • I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes

…in_id

Evaluator.create() gated the identity SearchKeywords
(@subtype/@evaluatortype/@contenttype) behind `method is not None`. Since
`method` is only resolved for RewardFunction, RewardPrompt evaluators were
registered with no identity keywords and were therefore invisible in SageMaker
Studio (which filters hub content by these keywords). The asset still existed in
the hub, so the failure was silent.

Always attach the identity keywords regardless of evaluator type or whether a
method was resolved; only `method` remains conditional.

Also add an optional `domain_id` parameter to Evaluator.create() and
DataSet.create(). The @Domain keyword is already auto-injected inside Studio, but
the domain cannot be inferred outside Studio (laptop/CI), leaving assets
un-scoped and hidden. Callers can now pass domain_id explicitly; auto-detection
is skipped when it is provided.

Adds regression tests covering RewardFunction and RewardPrompt identity keywords
and explicit domain_id for both Evaluator and DataSet.

Ref: P467494019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant