diff --git a/.agents/product-boundary.md b/.agents/product-boundary.md index 002000609..f1efe6161 100644 --- a/.agents/product-boundary.md +++ b/.agents/product-boundary.md @@ -44,6 +44,7 @@ AgentV should adopt Promptfoo-compatible eval matrix authoring where it strength Keep these AgentV-native boundaries explicit: - `providers` are systems under test. AgentV accepts Promptfoo-shaped provider declarations: strings like `openai:gpt-4.1-mini`, complete package provider strings like `package:@agentv/promptfoo-providers:CodexCliProvider` or `package:@agentv/promptfoo-providers/codex-cli:Provider`, provider option objects with `id`, `label`, `config`, `env`, `prompts`, `transform`, `delay`, and `inputs`, and provider maps like `{ "openai:gpt-4": { label, config } }`. Package provider strings must include the exported class/function segment after the final colon. In AgentV, `id` names the backend/spec and `label` is the stable AgentV identity used for selection and result grouping. +- Provider invocation stays per authored provider-case execution: the runner dispatches one `ProviderRequest` to `Provider.invoke()` for each resolved prompt/test/provider execution, then applies repeat samples and retries as separate executions. Provider adapters may batch, queue, pool, rate-limit, fan out, or share sessions internally, but they must preserve one response or error, trace identity, metrics, artifacts, and cancellation behavior per original `ProviderRequest`. Do not add runner-selected batch grouping as public eval YAML, project config, or CLI flags. Any drain, cleanup, or lifecycle hook for queued provider work is provider-owned adapter plumbing, not an eval-runner batch selection surface. This follows the Promptfoo provider-boundary evidence captured in av-kfik.55 while keeping AgentV's repo-native artifacts and cancellation contract explicit. - AgentV-only fields such as top-level `environment` and provider-local runtime/testbed overlays are AgentV semantics. Promptfoo may ignore or strip unknown keys and will not execute AgentV environment setup without a transpiler or wrapper. - Internal runtime, artifacts, and Dashboard grouping may keep target-named fields until a separate artifact migration. - Coding-agent testbeds use `environment` recipes for host/Docker substrate, setup, fixtures, services, and cwd. Do not make Promptfoo lifecycle `extensions` or public `workspace` authoring the canonical testbed contract. diff --git a/CONCEPTS.md b/CONCEPTS.md index 5c5a8f838..162f31d5b 100644 --- a/CONCEPTS.md +++ b/CONCEPTS.md @@ -6,7 +6,9 @@ Shared domain vocabulary for this project — entities, named processes, and sta **Provider** — The eval YAML or config declaration for a system under test. AgentV is Promptfoo-compatible at this provider declaration layer: entries may be strings such as `openai:gpt-4.1-mini`, complete package provider strings such as `package:@agentv/promptfoo-providers:CodexCliProvider` or `package:@agentv/promptfoo-providers/codex-cli:Provider`, provider option objects with `id`, `label`, `config`, `env`, `prompts`, `transform`, `delay`, and `inputs`, or provider maps such as `{ "openai:gpt-4": { label, config } }`. Package provider strings must include the exported class/function segment after the final colon. In AgentV, `id` names the backend/spec and `label` is the stable AgentV selection identity. AgentV-only provider fields such as `runtime` or provider-local testbed/runtime overlays are AgentV semantics, not full Promptfoo execution compatibility. A single eval file can declare multiple providers to compare AI systems side by side. Providers select agents/adapters; they do not own the authored host/Docker testbed recipe. -**Provider adapter** — The plugin/runtime implementation that connects AgentV's evaluation engine to a specific AI system, such as Copilot CLI, Copilot SDK, Claude API, or Pi. Provider adapters implement the request/response contract: given a test case, invoke the AI system and return its output. Public `providers[]` entries select and configure these adapters by `id`. +**Provider adapter** — The plugin/runtime implementation that connects AgentV's evaluation engine to a specific AI system, such as Copilot CLI, Copilot SDK, Claude API, or Pi. Provider adapters implement the request/response contract: given a `ProviderRequest`, invoke the AI system and return one `ProviderResponse` or one error for that same request. Public `providers[]` entries select and configure these adapters by `id`. + +**Provider invocation boundary** — The eval runner dispatches one `ProviderRequest` per authored provider-case execution, including each repeat sample or retry attempt. Batching, queueing, pooling, rate limiting, fanout, and shared-session management are provider-owned internals behind `Provider.invoke()`, not public eval config. Provider-owned batching must preserve one response or error, trace and transcript identity, metrics, artifact paths, and cancellation behavior for each original `ProviderRequest`. A drain, cleanup, or session lifecycle hook can exist only as provider-owned adapter plumbing; it must not become runner-selected batch grouping in YAML or CLI configuration. **Target** — Internal/runtime and artifact vocabulary for the stable comparison dimension produced by a selected provider. Result rows and run-bundle indexes keep the `target` field as the stable source identity until a separate artifact migration changes that contract. diff --git a/apps/web/src/content/docs/docs/next/targets/custom-providers.mdx b/apps/web/src/content/docs/docs/next/targets/custom-providers.mdx index 6fba3cc06..bb16a65c5 100644 --- a/apps/web/src/content/docs/docs/next/targets/custom-providers.mdx +++ b/apps/web/src/content/docs/docs/next/targets/custom-providers.mdx @@ -21,6 +21,24 @@ interface Provider { } ``` +## Invocation Boundary + +AgentV resolves an eval into authored provider-case executions. For each +execution, including repeats and retries, the eval runner sends one +`ProviderRequest` to `Provider.invoke()`. + +Provider adapters can batch, queue, pool sessions, rate-limit, or fan out work +internally when that is the right way to talk to the backend. Keep that behavior +behind the adapter boundary. Each original `ProviderRequest` must settle to +exactly one `ProviderResponse` or one error, and the adapter must preserve that +request's response, trace and transcript identity, metrics, artifact paths, and +cancellation through `request.signal`. + +Do not expose runner-selected batch grouping as eval YAML or project config. If +an adapter needs a drain, cleanup, or session lifecycle step to flush queued +provider work, treat it as provider-owned plumbing rather than a public eval +batching mode. + ### ProviderRequest The request object passed to `invoke()`: diff --git a/apps/web/src/content/docs/docs/v4.42.4/targets/custom-providers.mdx b/apps/web/src/content/docs/docs/v4.42.4/targets/custom-providers.mdx index 4455a220f..69939d8b6 100644 --- a/apps/web/src/content/docs/docs/v4.42.4/targets/custom-providers.mdx +++ b/apps/web/src/content/docs/docs/v4.42.4/targets/custom-providers.mdx @@ -23,6 +23,24 @@ interface Provider { } ``` +## Invocation Boundary + +AgentV resolves an eval into authored provider-case executions. For each +execution, including repeats and retries, the eval runner sends one +`ProviderRequest` to `Provider.invoke()`. + +Provider adapters can batch, queue, pool sessions, rate-limit, or fan out work +internally when that is the right way to talk to the backend. Keep that behavior +behind the adapter boundary. Each original `ProviderRequest` must settle to +exactly one `ProviderResponse` or one error, and the adapter must preserve that +request's response, trace and transcript identity, metrics, artifact paths, and +cancellation through `request.signal`. + +Do not expose runner-selected batch grouping as eval YAML or project config. If +an adapter needs a drain, cleanup, or session lifecycle step to flush queued +provider work, treat it as provider-owned plumbing rather than a public eval +batching mode. + ### ProviderRequest The request object passed to `invoke()`: