Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 14 additions & 14 deletions apps/web/src/content/docs/docs/next/graders/script-graders.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -288,41 +288,41 @@ Prefer Vitest verifiers when the checks naturally fit `expect(...)`. Use `define

**SDK exports:** `defineScriptGrader`, `defineVitestWorkspaceGrader`, `defineWorkspaceGrader`, `Message`, `ToolCall`, `Trace`, `TraceSummary`, `ScriptGraderInput`, `ScriptGraderResult`, `ScriptGraderCheck`, `Workspace`, `WorkspaceCheck`

## Target Access
## Provider Access

Script graders can call an LLM through a target proxy for metrics that require multiple LLM calls (contextual precision, semantic similarity, etc.).
Script graders can call an LLM through a provider proxy for metrics that require multiple LLM calls (contextual precision, semantic similarity, etc.).

### Configuration

Add a `target` block to the grader config:
Add a `provider` block to the grader config:

```yaml
assert:
- name: contextual-precision
type: script
command: [bun, scripts/contextual-precision.ts]
target:
provider:
max_calls: 10 # Default: 50
```

### Usage

Use `createTargetClient` from the SDK:
Use `createProviderClient` from the SDK:

```typescript
#!/usr/bin/env bun
import { createTargetClient, defineScriptGrader } from 'agentv';
import { createProviderClient, defineScriptGrader } from 'agentv';

export default defineScriptGrader(async ({ input, output }) => {
const inputText = input
.filter((message) => message.role === 'user')
.map((message) => typeof message.content === 'string' ? message.content : '')
.join('\n');
const outputText = output ?? '';
const target = createTargetClient();
if (!target) return { pass: false, score: 0, reason: 'Target not configured' };
const provider = createProviderClient();
if (!provider) return { pass: false, score: 0, reason: 'Provider proxy not configured' };

const response = await target.invoke({
const response = await provider.invoke({
question: `Is this relevant to: ${inputText}? Response: ${outputText}`,
systemPrompt: 'Respond with JSON: { "relevant": true/false }'
});
Expand All @@ -336,14 +336,14 @@ export default defineScriptGrader(async ({ input, output }) => {
});
```

Use `target.invokeBatch(requests)` for multiple calls in parallel.
Use `provider.invokeBatch(requests)` for multiple calls in parallel.

**Environment variables** (set automatically when `target` is configured):
**Environment variables** (set automatically when `provider` is configured):

| Variable | Description |
|----------|-------------|
| `AGENTV_TARGET_PROXY_URL` | Local proxy URL |
| `AGENTV_TARGET_PROXY_TOKEN` | Bearer token for authentication |
| `AGENTV_PROVIDER_PROXY_URL` | Local proxy URL |
| `AGENTV_PROVIDER_PROXY_TOKEN` | Bearer token for authentication |

## Advanced Input Fields

Expand All @@ -353,7 +353,7 @@ Beyond the basic fields (`input`, `output`, `expected_output`), script graders r
|-------|------|-------------|
| `input` | `Message[]` | Full resolved input message array |
| `output` | `string \| null` | Final answer / scored result only |
| `messages` | `Message[]` | Transcript messages from the target execution |
| `messages` | `Message[]` | Transcript messages from the provider execution |
| `expected_output` | `Message[]` | Expected/reference output messages |
| `output_path` | `string` | Temp file containing large final answer JSON, when `output` is omitted |
| `input_files` | `string[]` | Paths to input files referenced in the eval |
Expand Down
6 changes: 3 additions & 3 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,10 +47,10 @@ Focused demonstrations of specific AgentV capabilities. Each example includes it
- [assert-set](features/assert-set/) - Assertion grouping patterns
- [weighted-graders](features/weighted-graders/) - Weighted graders
- [execution-metrics](features/execution-metrics/) - Metrics tracking (tokens, cost, latency)
- [script-grader-with-llm-calls](features/script-grader-with-llm-calls/) - script graders with target proxy for LLM calls
- [script-grader-with-llm-calls](features/script-grader-with-llm-calls/) - script graders with provider proxy for LLM calls
- [batch-cli](features/batch-cli/) - Batch CLI evaluation
- [document-extraction](features/document-extraction/) - Document data extraction
- [local-cli](features/local-cli/) - Local CLI targets
- [local-cli](features/local-cli/) - Local CLI providers
- [compare](features/compare/) - Baseline comparison
- [deterministic-graders](features/deterministic-graders/) - Deterministic assertions (contains, regex, JSON validation)
- [vitest-workspace-grader](features/vitest-workspace-grader/) - Vitest-style deterministic workspace verifiers
Expand Down Expand Up @@ -92,7 +92,7 @@ example-name/
│ └── *.md # LLM grader prompts (optional)
├── scripts/ # Helper scripts (optional)
├── .agentv/
│ └── providers.yaml # Target configuration (optional)
│ └── providers.yaml # Provider configuration (optional)
├── package.json # Dependencies (if using agentv)
└── README.md # Example documentation
```
Expand Down
24 changes: 12 additions & 12 deletions examples/features/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Focused examples for specific AgentV capabilities. Find your use case below, the
| Example | Description |
|---------|-------------|
| [script-grader-sdk](script-grader-sdk/) | TypeScript script graders using `defineScriptGrader()` from `agentv` |
| [script-grader-with-llm-calls](script-grader-with-llm-calls/) | script graders that make LLM calls via a target proxy |
| [script-grader-with-llm-calls](script-grader-with-llm-calls/) | script graders that make LLM calls via a provider proxy |
| [eval-assert-demo](eval-assert-demo/) | script graders runnable both in a suite and individually via `agentv eval assert` |
| [functional-grading](functional-grading/) | Install dependencies, compile, and run tests against agent-generated code |

Expand All @@ -51,7 +51,7 @@ Focused examples for specific AgentV capabilities. Find your use case below, the
|---------|-------------|
| [trajectory-assertions-simple](trajectory-assertions-simple/) | Validate expected tool calls with Promptfoo trajectory assertions |
| [trajectory-assertions-advanced](trajectory-assertions-advanced/) | Promptfoo trajectory assertions with `expected_output` and per-call checks |
| [latency-assertions](latency-assertions/) | Tool sequence and argument checks for a latency-flavored mock target |
| [latency-assertions](latency-assertions/) | Tool sequence and argument checks for a latency-flavored mock provider |
| [tool-evaluation-plugins](tool-evaluation-plugins/) | F1 precision/recall scoring for tool-call accuracy |
| [trace-evaluation](trace-evaluation/) | Inspect agent internals: LLM call counts, tool executions, step durations |

Expand Down Expand Up @@ -103,10 +103,10 @@ Focused examples for specific AgentV capabilities. Find your use case below, the
| [workspace-setup-script](workspace-setup-script/) | Multi-step setup with a `beforeAll` lifecycle extension |
| [workspace-multi-repo](workspace-multi-repo/) | Multi-repo environment using a VS Code `.code-workspace` file |
| [workspace-shared-config](workspace-shared-config/) | Define an environment recipe once and reference it across eval files |
| [repo-lifecycle](repo-lifecycle/) | Clone a git repo into the workspace and target the agent at it |
| [repo-lifecycle](repo-lifecycle/) | Clone a git repo into the workspace and point the agent at it |
| [file-changes](file-changes/) | Capture workspace file changes made by the agent across test runs |
| [file-changes-graders](file-changes-graders/) | Grade file diffs with rubrics and LLM graders |
| [local-cli](local-cli/) | Define and invoke local CLI targets |
| [local-cli](local-cli/) | Define and invoke local CLI providers |
| [batch-cli](batch-cli/) | Run bulk evaluations from the CLI |

---
Expand Down Expand Up @@ -138,7 +138,7 @@ Focused examples for specific AgentV capabilities. Find your use case below, the
| [assert-extended](assert-extended/) | Deterministic assertions |
| [basic](basic/) | Getting started |
| [basic-jsonl](basic-jsonl/) | Getting started |
| [batch-cli](batch-cli/) | Workspace & targets |
| [batch-cli](batch-cli/) | Workspace & providers |
| [benchmark-tooling](benchmark-tooling/) | Benchmarking |
| [script-grader-sdk](script-grader-sdk/) | Custom graders |
| [script-grader-with-llm-calls](script-grader-with-llm-calls/) | Custom graders |
Expand All @@ -152,17 +152,17 @@ Focused examples for specific AgentV capabilities. Find your use case below, the
| [eval-assert-demo](eval-assert-demo/) | Custom graders |
| [execution-metrics](execution-metrics/) | Cost, latency & tokens |
| [external-datasets](external-datasets/) | Dataset & input |
| [file-changes](file-changes/) | Workspace & targets |
| [file-changes-graders](file-changes-graders/) | Workspace & targets |
| [file-changes](file-changes/) | Workspace & providers |
| [file-changes-graders](file-changes-graders/) | Workspace & providers |
| [functional-grading](functional-grading/) | Custom graders |
| [input-files-shorthand](input-files-shorthand/) | Dataset & input |
| [latency-assertions](latency-assertions/) | Tool & agent evaluation |
| [local-cli](local-cli/) | Workspace & targets |
| [local-cli](local-cli/) | Workspace & providers |
| [multi-turn-conversation](multi-turn-conversation/) | LLM grading |
| [nlp-metrics](nlp-metrics/) | Deterministic assertions |
| [file-transforms](file-transforms/) | LLM grading |
| [prompt-template-sdk](prompt-template-sdk/) | TypeScript SDK |
| [repo-lifecycle](repo-lifecycle/) | Workspace & targets |
| [repo-lifecycle](repo-lifecycle/) | Workspace & providers |
| [rubric](rubric/) | LLM grading |
| [scenarios](scenarios/) | Dataset & prompt templates |
| [sdk-config-file](sdk-config-file/) | TypeScript SDK |
Expand All @@ -181,6 +181,6 @@ Focused examples for specific AgentV capabilities. Find your use case below, the
| [trial-output-consistency](trial-output-consistency/) | Benchmarking |
| [trials](trials/) | Benchmarking |
| [weighted-graders](weighted-graders/) | LLM grading |
| [workspace-multi-repo](workspace-multi-repo/) | Workspace & targets |
| [workspace-setup-script](workspace-setup-script/) | Workspace & targets |
| [workspace-shared-config](workspace-shared-config/) | Workspace & targets |
| [workspace-multi-repo](workspace-multi-repo/) | Workspace & providers |
| [workspace-setup-script](workspace-setup-script/) | Workspace & providers |
| [workspace-shared-config](workspace-shared-config/) | Workspace & providers |
42 changes: 21 additions & 21 deletions examples/features/script-grader-with-llm-calls/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,37 +141,37 @@ The current `extractRetrievalContext()` in `utils.ts` flattens for simplicity. F

## Security

The target proxy is designed with security in mind:
The provider proxy is designed with security in mind:
- Binds to **loopback only** (127.0.0.1) - not accessible from network
- Uses **bearer token authentication** - unique per execution
- Enforces **max_calls limit** - prevents runaway costs
- **Auto-shutdown** - proxy terminates when grader completes

## Configuration

Enable target access by adding a `target` block to your `script_grader` grader:
Enable provider access by adding a `provider` block to your `script_grader` grader:

```yaml
graders:
- name: contextual_precision
type: script
command: [bun, run, scripts/contextual-precision.ts]
target:
provider:
max_calls: 10 # At least N nodes to evaluate
- name: contextual_recall
type: script
command: [bun, run, scripts/contextual-recall.ts]
target:
provider:
max_calls: 15 # 1 for extraction + N statements for attribution
```

## Usage in Code

```typescript
import { createTargetClient, defineScriptGrader } from 'agentv';
import { createProviderClient, defineScriptGrader } from 'agentv';

export default defineScriptGrader(async ({ question, config }) => {
const target = createTargetClient();
const provider = createProviderClient();
const retrievalContext = config?.retrieval_context ?? [];

// Batch evaluation of all nodes
Expand All @@ -180,50 +180,50 @@ export default defineScriptGrader(async ({ question, config }) => {
systemPrompt: 'Respond with JSON: { "relevant": true/false }'
}));

const responses = await target.invokeBatch(requests);
const responses = await provider.invokeBatch(requests);

// Calculate weighted precision score...
});
```

## Querying Proxy Info

You can query information about the target proxy:
You can query information about the provider proxy:

```typescript
const info = await target.getInfo();
console.log(`Target: ${info.targetName}`);
const info = await provider.getInfo();
console.log(`Provider: ${info.providerLabel}`);
console.log(`Calls: ${info.callCount}/${info.maxCalls}`);
console.log(`Available targets: ${info.availableTargets.join(', ')}`);
console.log(`Available providers: ${info.availableProviderLabels.join(', ')}`);
```

## Target Override
## Provider Override

Use different targets for different purposes within the same grader:
Use different providers for different purposes within the same grader:

```typescript
// Use a coding agent for complex tasks
const agentResponses = await target.invokeBatch(
const agentResponses = await provider.invokeBatch(
nodes.map(node => ({
question: `Is this relevant? ${node}`,
target: 'pi' // Override default target
provider: 'pi' // Override default provider
}))
);

// Use a base LLM for simple evaluation
const response = await target.invoke({
const response = await provider.invoke({
question: complexAnalysisPrompt,
target: 'gemini-llm' // Use different target
provider: 'gemini-llm' // Use different provider
});
```

## Environment Variables

When `target` is configured, these environment variables are automatically set:
- `AGENTV_TARGET_PROXY_URL` - Local proxy URL (e.g., `http://127.0.0.1:45123`)
- `AGENTV_TARGET_PROXY_TOKEN` - Bearer token for authentication
When `provider` is configured, these environment variables are automatically set:
- `AGENTV_PROVIDER_PROXY_URL` - Local proxy URL (e.g., `http://127.0.0.1:45123`)
- `AGENTV_PROVIDER_PROXY_TOKEN` - Bearer token for authentication

The `createTargetClient()` function reads these automatically.
The `createProviderClient()` function reads these automatically.

## Running

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ assert:
- bun
- run
- ../scripts/contextual-precision.ts
target:
provider:
max_calls: 10
providers:
- llm
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ assert:
- bun
- run
- ../scripts/contextual-recall.ts
target:
provider:
max_calls: 15
providers:
- llm
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@
* Retrieval context is extracted from expected_output.tool_calls output,
* which represents the expected agent behavior (calling a retrieval tool).
*
* Requires `target: { max_calls: N }` in the grader YAML config,
* Requires `provider: { max_calls: N }` in the grader YAML config,
* where N >= number of retrieval context nodes to evaluate.
*/
import { createTargetClient, defineScriptGrader } from 'agentv';
import { createProviderClient, defineScriptGrader } from 'agentv';
import { extractRetrievalContext } from './utils.js';

interface RelevanceResult {
Expand Down Expand Up @@ -63,22 +63,22 @@ export default defineScriptGrader(async (input) => {
};
}

const target = createTargetClient();
const provider = createProviderClient();

if (!target) {
if (!provider) {
return {
score: 0,
assertions: [
{
text: 'Target not available - ensure `target` block is configured in grader YAML',
text: 'Provider proxy not available - ensure `provider` block is configured in grader YAML',
passed: false,
},
],
};
}

// Step 1: Use batch invocation to determine relevance of each node
// Demonstrates target override - uses gemini-llm regardless of default target
// Demonstrates provider override - uses gemini-llm regardless of default provider
const requests = retrievalContext.map((node, index) => ({
question: `Determine if this retrieved context node is relevant to answering the question.

Expand All @@ -95,10 +95,10 @@ Is this node relevant to answering the question? Respond with JSON only:
}`,
systemPrompt:
'You are a precise relevance grader for RAG systems. Determine if a retrieved node contains information useful for answering the given question. Output valid JSON only.',
target: 'gemini-llm', // Override: use gemini-llm for relevance checks
provider: 'gemini-llm', // Override: use gemini-llm for relevance checks
}));

const responses = await target.invokeBatch(requests);
const responses = await provider.invokeBatch(requests);

// Step 2: Parse relevance scores for each node
const relevanceScores: boolean[] = [];
Expand Down
Loading
Loading