-
Notifications
You must be signed in to change notification settings - Fork 0
docs: add AND/OR composition examples for composite evaluator and SDK #950
Copy link
Copy link
Open
Description
Problem
Users who need AND (all must pass) or OR (any must pass) assertion logic have no examples showing how to achieve this with existing primitives. The composite evaluator and SDK inline assertions both support these patterns, but they are undocumented.
Issue #949 was closed because AND/OR logic is already expressible — but only if users know how. We need examples.
Examples to Add
AND logic (all must pass) — YAML
assertions:
- name: all_must_pass
type: composite
aggregator:
type: threshold
threshold: 1.0 # 100% of children must pass
assertions:
- type: contains
value: "capital"
- type: contains
value: "France"OR logic (any must pass) — YAML
assertions:
- name: any_match
type: composite
aggregator:
type: weighted_average # any non-zero child raises the average above 0
assertions:
- type: contains
value: "Paris"
- type: contains
value: "paris"
- type: icontains
value: "the capital of france is paris"Note: weighted_average is not a true OR — if only 1 of 3 passes, score = 0.33. For true OR, use a code-grader aggregator or an inline assertion.
OR logic (any must pass) — SDK inline assertion
await evaluate({
tests: [{
id: "capital",
input: "What is the capital of France?",
assert: [
({ output }) => ({
name: "mentions-paris",
score: /paris/i.test(output) || output.includes("the capital of France") ? 1 : 0,
}),
],
}],
task: myAgent,
});Safety gate (AND with early fail) — YAML
assertions:
- name: safety_then_quality
type: composite
aggregator:
type: threshold
threshold: 1.0
assertions:
- type: llm-grader
prompt: ./safety.md
required: true # gates aggregate if safety fails
- type: llm-grader
prompt: ./quality.mdWhere to add
examples/features/composite/README.md— add AND/OR patterns sectionexamples/features/composite/evals/— addand-or-patterns.eval.yamlexample file- Main
README.mdor docs — mention AND/OR is achievable via composite evaluator
Acceptance Signals
- AND pattern example exists in
examples/features/composite/with working YAML - OR pattern example exists (both YAML composite and SDK inline)
- Safety gate pattern documented
- README updated to reference these patterns
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels