Skip to content

fix(rubrics): read JudgeRubric questions from Pydantic messages#2005

Open
crazywriter1 wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
crazywriter1:fix/judge-rubric-pydantic-question
Open

fix(rubrics): read JudgeRubric questions from Pydantic messages#2005
crazywriter1 wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
crazywriter1:fix/judge-rubric-pydantic-question

Conversation

@crazywriter1

@crazywriter1 crazywriter1 commented Jul 14, 2026

Copy link
Copy Markdown

Summary

  • JudgeRubric.judge() only extracted question from dict prompts (isinstance(..., dict))
  • Rollouts normalize prompts to Pydantic Message objects, so the Question block was silently empty
  • Read content via parser helpers (_message_field / _content_to_text) for dict and Message prompts

Test plan

  • Dict prompt still yields the last user content as {question}
  • Pydantic-like message with .content no longer produces an empty Question
  • Multimodal text parts are joined into the question string (via _content_to_text)

Note

Low Risk
Small, localized change to prompt parsing in judge rubric scoring with no auth or data-path changes.

Overview
Fixes judge scoring when rollouts pass Pydantic Message prompts instead of raw dicts.

JudgeRubric.judge() previously built the {question} field only when the last list prompt entry was a dict with "content", so normalized rollout prompts left the Question block empty and weakened judge decisions.

The last message’s content is now read through Parser._message_field and Parser._content_to_text, so dict and object messages (including multimodal text parts) populate {question} consistently.

Reviewed by Cursor Bugbot for commit 3cd51ce. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Fix JudgeRubric.judge to extract question text from Pydantic messages

Previously, when prompt was a list, the method assumed raw dicts and fell back to empty strings for Pydantic Message objects. Now uses self.parser._message_field and self.parser._content_to_text in judge_rubric.py to correctly extract the question string from both dict and Pydantic message formats.

Macroscope summarized 3cd51ce.

Rollouts normalize prompts to Message objects, but judge() only read
dict content and silently sent an empty Question to the judge model.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant