Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions examples/features/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ Focused examples for specific AgentV capabilities. Find your use case below, the
| [suite-level-input](suite-level-input/) | Prepend a shared system prompt to every test in the suite |
| [suite-level-input-files](suite-level-input-files/) | Share file attachments across every test in the suite |
| [env-interpolation](env-interpolation/) | Inject environment variables into eval config with `${{ VAR }}` |
| [test-vars-templating](test-vars-templating/) | Inject per-test `vars` into `{{name}}` templates in eval fields |

---

Expand Down Expand Up @@ -169,6 +170,7 @@ Focused examples for specific AgentV capabilities. Find your use case below, the
| [sdk-programmatic-api](sdk-programmatic-api/) | TypeScript SDK |
| [suite-level-input](suite-level-input/) | Dataset & input |
| [suite-level-input-files](suite-level-input-files/) | Dataset & input |
| [test-vars-templating](test-vars-templating/) | Dataset & input |
| [threshold-grader](threshold-grader/) | LLM grading |
| [tool-evaluation-plugins](tool-evaluation-plugins/) | Tool & agent evaluation |
| [tool-trajectory-advanced](tool-trajectory-advanced/) | Tool & agent evaluation |
Expand Down
16 changes: 16 additions & 0 deletions examples/features/test-vars-templating/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Per-Test Vars Templating

Demonstrates `tests[].vars` with `{{name}}` placeholders in eval files.

## Usage

```bash
agentv eval examples/features/test-vars-templating/evals/dataset.eval.yaml
```

## Features

- **Per-test data**: each test defines its own `vars` object
- **Template substitution**: `{{question}}` and dotted paths like `{{expected.answer}}`
- **Suite-level templates**: shared `input` can reference per-test vars too
- **Separate from env interpolation**: `{{question}}` uses test data, `${{ VAR }}` uses environment variables
40 changes: 40 additions & 0 deletions examples/features/test-vars-templating/evals/dataset.eval.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Per-test vars templating example
#
# tests[].vars provides per-test data for {{name}} placeholders in eval fields.
# Placeholders support dotted paths like {{expected.answer}}.
#
# Usage:
# agentv eval examples/features/test-vars-templating/evals/dataset.eval.yaml

description: Demonstrates tests[].vars templating in eval fields

execution:
target: llm

input:
- role: system
content: "You are a concise assistant answering {{category}} questions."

tests:
- id: capital-france
vars:
category: geography
question: What is the capital of France?
expected:
answer: Paris
criteria: "Answers {{question}} correctly"
input: "Question: {{question}}"
expected_output: "{{expected.answer}}"

- id: greet-ada
vars:
category: etiquette
person:
name: Ada
expected:
answer: Hello, Ada!
criteria: "Greets {{person.name}} warmly"
input:
- role: user
content: "Say hello to {{person.name}}."
expected_output: "{{expected.answer}}"
1 change: 1 addition & 0 deletions packages/core/src/evaluation/validation/eval-validator.ts
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ const DEPRECATED_TOP_LEVEL_FIELDS = new Map<string, string>([
/** Known fields at the test level. */
const KNOWN_TEST_FIELDS = new Set([
'id',
'vars',
'criteria',
'input',
'input_files',
Expand Down
23 changes: 23 additions & 0 deletions packages/core/test/evaluation/validation/eval-validator.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,29 @@ describe('validateEvalFile', () => {
expect(result.errors).toHaveLength(0);
});

it('accepts vars without unknown-field warnings', async () => {
const filePath = path.join(tempDir, 'test-vars.yaml');
await writeFile(
filePath,
`tests:
- id: test-1
vars:
question: "What is 2+2?"
expected:
answer: "4"
criteria: "Answers {{question}} correctly"
input: "Question: {{question}}"
expected_output: "{{expected.answer}}"
`,
);

const result = await validateEvalFile(filePath);

expect(result.valid).toBe(true);
const warnings = result.errors.filter((e) => e.severity === 'warning');
expect(warnings).toHaveLength(0);
});

describe('assert field validation', () => {
it('validates assert array items have type field', async () => {
const filePath = path.join(tempDir, 'assert-missing-type.yaml');
Expand Down
Loading