Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
277 changes: 0 additions & 277 deletions apps/web/src/content/docs/docs/next/evaluation/batch-cli.mdx

This file was deleted.

86 changes: 0 additions & 86 deletions apps/web/src/content/docs/docs/next/evaluation/examples.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -302,92 +302,6 @@ tests:
value: "Matches the reference answer: {{ expected_output }}"
````

## Batch CLI

Evaluate external batch runners that process all tests in one invocation:

```yaml
description: Batch CLI demo (AML screening)
providers:
- batch_cli

prompts:
- "{{ input }}"

tests:
- id: aml-001
criteria: |-
Batch runner returns JSON with decision=CLEAR.
vars:
input:
- role: system
content: You are a deterministic AML screening batch checker.
- role: user
content:
request:
type: aml_screening_check
jurisdiction: AU
effective_date: 2025-01-01
row:
id: aml-001
customer_name: Example Customer A
origin_country: NZ
destination_country: AU
transaction_type: INTERNATIONAL_TRANSFER
amount: 5000
currency: USD
expected_output:
- role: assistant
content:
decision: CLEAR

assert:
- name: decision-check
type: script
command: [bun, run, ./scripts/check-batch-cli-output.ts]
cwd: .

- id: aml-002
criteria: |-
Batch runner returns JSON with decision=REVIEW.
vars:
input:
- role: system
content: You are a deterministic AML screening batch checker.
- role: user
content:
request:
type: aml_screening_check
jurisdiction: AU
effective_date: 2025-01-01
row:
id: aml-002
customer_name: Example Customer B
origin_country: IR
destination_country: AU
transaction_type: INTERNATIONAL_TRANSFER
amount: 2000
currency: USD
expected_output:
- role: assistant
content:
decision: REVIEW

assert:
- name: decision-check
type: script
command: [bun, run, ./scripts/check-batch-cli-output.ts]
cwd: .
```

### Batch CLI Pattern Notes

- `target: batch_cli` -- configure the CLI provider with `batch_requests: true`
- The batch runner reads the eval YAML via `--eval` flag and outputs JSONL keyed by `id`
- Put structured data in `user.content` as objects for the runner to extract
- Use `vars.expected_output` with object fields for structured expected output
- Each test has its own grader to validate its portion of the output

## Shared Prompt Context

Share a common prompt or system instruction across all tests with top-level
Expand Down
Loading
Loading