Skip to content

[Evaluator] Response Completeness prompt update for short GTs #46180

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
ahibrahimm wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

[Evaluator] Response Completeness prompt update for short GTs #46180

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5e0a3c8
prompt update
ahibrahimm Mar 30, 2026
662cee3
Merge branch 'main' into ahibrahim/rc
ahibrahimm Apr 7, 2026
7326b18
Merge branch 'main' into ahibrahim/rc
ahibrahimm Apr 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions ...tion/azure/ai/evaluation/_evaluators/_response_completeness/response_completeness.prompty
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ user:
# Definition
**Completeness** refers to how accurately and thoroughly a response represents the information provided in the ground truth. It considers both the inclusion of all relevant statements and the correctness of those statements. Each statement in the ground truth should be evaluated individually to determine if it is accurately reflected in the response without missing any key information. The scale ranges from 1 to 5, with higher numbers indicating greater completeness.

**Note:** When the ground truth is a short entity or phrase, the response is fully complete (Score 5) if it semantically includes that entity. Additional accurate context is not a penalty.

# Ratings
## [Completeness: 1] (Fully Incomplete)
**Definition:** A response that does not contain any of the necessary and relevant information with respect to the ground truth. It completely misses all the information, especially claims and statements, established in the ground truth.
Expand All @@ -40,6 +42,9 @@ user:
**Response:** "Flu shot cannot cure cancer. Stay healthy requires sleeping exactly 8 hours a day. A few hours of exercise per week will have little benefits for physical and mental health. Physical and mental health benefits are separate topics. Scientists have not studied any of them."
**Ground Truth:** "Flu shot can prevent flu-related illnesses. Staying healthy requires proper hydration and moderate exercise. Even a few hours of exercise per week can have long-term benefits for physical and mental health. This is because physical and mental health benefits have intricate relationships through behavioral changes. Scientists are starting to discover them through rigorous studies."

**Response:** "Mount Kilimanjaro is the tallest peak in Africa."
**Ground Truth:** "Everest"

## [Completeness: 2] (Barely Complete)
**Definition:** A response that contains only a small percentage of all the necessary and relevant information with respect to the ground truth. It misses almost all the information, especially claims and statements, established in the ground truth.

Expand Down Expand Up @@ -68,6 +73,9 @@ user:
**Response:** "Flu shot can prevent flu-related illnesses. Stay healthy by proper hydration and moderate exercise. Even a few hours of exercise per week can have long-term benefits for physical and mental health. This is because physical and mental health benefits have intricate relationships through behavioral changes. Scientists are starting to discover them through rigorous studies."
**Ground Truth:** "Flu shot can prevent flu-related illnesses. Stay healthy by proper hydration and moderate exercise. Even a few hours of exercise per week can have long-term benefits for physical and mental health. This is because physical and mental health benefits have intricate relationships through behavioral changes. Scientists are starting to discover them through rigorous studies."

**Response:** "The Amazon River is the longest river in South America, stretching over 6,400 kilometers."
**Ground Truth:** "Amazon River"


# Data
Response: {{response}}
Expand Down