Skip to content

docs: add AND/OR composition examples for composite evaluator and SDK #950

@christso

Description

@christso

Problem

Users who need AND (all must pass) or OR (any must pass) assertion logic have no examples showing how to achieve this with existing primitives. The composite evaluator and SDK inline assertions both support these patterns, but they are undocumented.

Issue #949 was closed because AND/OR logic is already expressible — but only if users know how. We need examples.

Examples to Add

AND logic (all must pass) — YAML

assertions:
  - name: all_must_pass
    type: composite
    aggregator:
      type: threshold
      threshold: 1.0       # 100% of children must pass
    assertions:
      - type: contains
        value: "capital"
      - type: contains
        value: "France"

OR logic (any must pass) — YAML

assertions:
  - name: any_match
    type: composite
    aggregator:
      type: weighted_average   # any non-zero child raises the average above 0
    assertions:
      - type: contains
        value: "Paris"
      - type: contains
        value: "paris"
      - type: icontains
        value: "the capital of france is paris"

Note: weighted_average is not a true OR — if only 1 of 3 passes, score = 0.33. For true OR, use a code-grader aggregator or an inline assertion.

OR logic (any must pass) — SDK inline assertion

await evaluate({
  tests: [{
    id: "capital",
    input: "What is the capital of France?",
    assert: [
      ({ output }) => ({
        name: "mentions-paris",
        score: /paris/i.test(output) || output.includes("the capital of France") ? 1 : 0,
      }),
    ],
  }],
  task: myAgent,
});

Safety gate (AND with early fail) — YAML

assertions:
  - name: safety_then_quality
    type: composite
    aggregator:
      type: threshold
      threshold: 1.0
    assertions:
      - type: llm-grader
        prompt: ./safety.md
        required: true       # gates aggregate if safety fails
      - type: llm-grader
        prompt: ./quality.md

Where to add

  1. examples/features/composite/README.md — add AND/OR patterns section
  2. examples/features/composite/evals/ — add and-or-patterns.eval.yaml example file
  3. Main README.md or docs — mention AND/OR is achievable via composite evaluator

Acceptance Signals

  1. AND pattern example exists in examples/features/composite/ with working YAML
  2. OR pattern example exists (both YAML composite and SDK inline)
  3. Safety gate pattern documented
  4. README updated to reference these patterns

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions