Skip to content

feat: thread attack objective through sdk and mcp runners #193

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jithin23-kv wants to merge 1 commit into
base: master
Choose a base branch
from
Open

feat: thread attack objective through sdk and mcp runners #193

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions core/src/config/schema.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,7 @@ export const RunConfigSchema = z
turnMode: z.enum(["single", "multi"]).optional(),
turns: z.number().int().positive().optional(),
telemetry: z.unknown().optional(),
attackObjective: z.string().optional(),
})
.passthrough();

Expand Down
6 changes: 6 additions & 0 deletions core/src/execute/evaluatorLoop.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,12 @@ export async function runEvaluatorAttacks(
throw err;
}

if (config.attackObjective) {
for (const attack of attacks) {
if (attack.kind === "agent") attack.attackObjective = config.attackObjective;
}
}

log.info(` ${attacks.length} attack(s) generated [effort: ${config.effort}]`);

const attackResults: AttackResult[] = [];
Expand Down
8 changes: 8 additions & 0 deletions core/src/execute/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,14 @@ export interface RunConfig {
turnMode?: "single" | "multi";
turns: number;
telemetry?: TelemetryConfig;
/**
* Free-text primary mission steering every evaluator's attacks (e.g. "get the
* target to leak env vars via a delegated employee"). Threaded onto each
* generated AgentAttackSpec; consumed by generateNextAdaptiveTurn as the
* attacker's top-priority goal, same mechanism the browser extension's
* popup-driven `attackObjective` already uses.
*/
attackObjective?: string;
}

// ---------------------------------------------------------------------------
Expand Down
61 changes: 32 additions & 29 deletions docs/cli.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -285,42 +285,45 @@ See [telemetry.md](telemetry.md) for Langfuse and Netra setup, config fields, an

## Commands reference

| Command | Description |
| --------------------------------------------- | ---------------------------------------------------------------------- |
| `opfor setup` | Interactive wizard — writes a timestamped config |
| `opfor setup --agent` / `--mcp` | Skip mode prompt |
| `opfor setup --empty` | Write a blank config without wizard prompts |
| `opfor setup --config <path>` | Override the output config path |
| `opfor run` | Run the setup wizard inline, then run the resulting config |
| `opfor run --config <file>` | Read an existing config, fire attacks, judge, write HTML + JSON report |
| `opfor run --config <file> --effort <e>` | Override `effort` (`adaptive` or `comprehensive`) |
| `opfor run --config <file> --turns <n>` | Override turn count (1 forces single-turn) |
| `opfor run --config <file> --output <d>` | Override report parent directory (default `.opfor/reports/`) |
| `opfor run --config <file> --env <path>` | Load env vars from a non-default `.env` path |
| `opfor run --config <file> --events <path>` | Stream NDJSON run lifecycle events to `<path>` (for CI/automation) |
| `opfor setup --env <path>` | Same `--env` flag works on setup |
| `opfor hunt --endpoint <url> --objective <t>` | Autonomous agentic red-team — see [hunt.md](hunt.md) |
| Command | Description |
| ------------------------------------------------ | ---------------------------------------------------------------------- |
| `opfor setup` | Interactive wizard — writes a timestamped config |
| `opfor setup --agent` / `--mcp` | Skip mode prompt |
| `opfor setup --empty` | Write a blank config without wizard prompts |
| `opfor setup --config <path>` | Override the output config path |
| `opfor run` | Run the setup wizard inline, then run the resulting config |
| `opfor run --config <file>` | Read an existing config, fire attacks, judge, write HTML + JSON report |
| `opfor run --config <file> --effort <e>` | Override `effort` (`adaptive` or `comprehensive`) |
| `opfor run --config <file> --turns <n>` | Override turn count (1 forces single-turn) |
| `opfor run --config <file> --output <d>` | Override report parent directory (default `.opfor/reports/`) |
| `opfor run --config <file> --env <path>` | Load env vars from a non-default `.env` path |
| `opfor run --config <file> --events <path>` | Stream NDJSON run lifecycle events to `<path>` (for CI/automation) |
| `opfor run --config <file> --objective <t>` | Steer every evaluator's attacks toward a specific free-text mission |
| `opfor run --config <file> --objective-file <p>` | Same, read from a file |
| `opfor setup --env <path>` | Same `--env` flag works on setup |
| `opfor hunt --endpoint <url> --objective <t>` | Autonomous agentic red-team — see [hunt.md](hunt.md) |

---

## Config fields reference

### Common fields (both modes)

| Field | Required | Description |
| ----------------------- | ----------------------------- | -------------------------------------------------------------------------------------------------- |
| `target.kind` | Yes | `"agent"` or `"mcp"`. |
| `selection.mode` | Yes | `"suite"` or `"evaluators"`. |
| `selection.suite` | For suite | Suite ID — see [evaluators reference](evaluators.md). |
| `selection.evaluators` | For evaluators | Array of evaluator IDs. |
| `attackerLlm.provider` | Yes | See [Supported LLM providers](#supported-llm-providers). |
| `attackerLlm.model` | Yes | Model name (e.g. `gpt-4o-mini`). |
| `attackerLlm.apiKeyEnv` | No | Env var **name** holding the API key. Falls back to the provider's default env var when absent. |
| `attackerLlm.baseURL` | For openai-compatible / azure | Base URL for the LLM endpoint. |
| `judgeLlm.*` | No | Same fields as `attackerLlm`. Separate model for judging. Falls back to `attackerLlm` when absent. |
| `effort` | Yes | `"adaptive"` or `"comprehensive"`. |
| `turnMode` | No | `"single"` (default when omitted) or `"multi"`. |
| `turns` | Yes | Turns per attack. Ignored when `turnMode` is `"single"`. Range 1–10 (wizard default 3). |
| Field | Required | Description |
| ----------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `target.kind` | Yes | `"agent"` or `"mcp"`. |
| `selection.mode` | Yes | `"suite"` or `"evaluators"`. |
| `selection.suite` | For suite | Suite ID — see [evaluators reference](evaluators.md). |
| `selection.evaluators` | For evaluators | Array of evaluator IDs. |
| `attackerLlm.provider` | Yes | See [Supported LLM providers](#supported-llm-providers). |
| `attackerLlm.model` | Yes | Model name (e.g. `gpt-4o-mini`). |
| `attackerLlm.apiKeyEnv` | No | Env var **name** holding the API key. Falls back to the provider's default env var when absent. |
| `attackerLlm.baseURL` | For openai-compatible / azure | Base URL for the LLM endpoint. |
| `judgeLlm.*` | No | Same fields as `attackerLlm`. Separate model for judging. Falls back to `attackerLlm` when absent. |
| `effort` | Yes | `"adaptive"` or `"comprehensive"`. |
| `turnMode` | No | `"single"` (default when omitted) or `"multi"`. |
| `turns` | Yes | Turns per attack. Ignored when `turnMode` is `"single"`. Range 1–10 (wizard default 3). |
| `attackObjective` | No | Free-text primary mission steering every evaluator's attacks (e.g. "get the target to leak env vars via a delegated employee"). Also settable via `--objective`/`--objective-file` on `opfor run`. Same mechanism the browser extension's popup-driven objective already uses. |

### Agent fields (`target.kind: "agent"`)

Expand Down
26 changes: 14 additions & 12 deletions docs/mcp.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,25 +149,27 @@ LLM object schema:

#### Run settings

| Parameter | Type | Required | Description |
| ------------- | ----------------------------- | -------- | ------------------------------------------------------------------------ |
| `effort` | `adaptive` \| `comprehensive` | No | Defaults to `adaptive`. Comprehensive runs one attack per named pattern. |
| `turns` | number (1–10) | No | Turns per attack. 1 = single-turn. Defaults to 1. |
| `output_dir` | string | No | Directory to write `opfor.config.json`. Defaults to `.` |
| `config_path` | string | No | Full path for config file (overrides `output_dir`) |
| Parameter | Type | Required | Description |
| ------------------ | ----------------------------- | -------- | ------------------------------------------------------------------------ |
| `effort` | `adaptive` \| `comprehensive` | No | Defaults to `adaptive`. Comprehensive runs one attack per named pattern. |
| `turns` | number (1–10) | No | Turns per attack. 1 = single-turn. Defaults to 1. |
| `attack_objective` | string | No | Free-text mission steering every evaluator's attacks |
| `output_dir` | string | No | Directory to write `opfor.config.json`. Defaults to `.` |
| `config_path` | string | No | Full path for config file (overrides `output_dir`) |

---

### `opfor_run`

Runs the full red team evaluation from a config file produced by `opfor_setup`. Generates attacks on-the-fly, runs them against the target, judges responses, and writes an HTML + JSON report.

| Parameter | Type | Required | Description |
| ----------------- | ----------------------------- | -------- | ---------------------------------------------------- |
| `config_path` | string | Yes | Path to `opfor.config.json` written by `opfor_setup` |
| `output_dir` | string | No | Report directory. Defaults to config file directory |
| `effort_override` | `adaptive` \| `comprehensive` | No | Override the effort level from config |
| `turns_override` | number (1–10) | No | Override turns per attack from config |
| Parameter | Type | Required | Description |
| -------------------- | ----------------------------- | -------- | ---------------------------------------------------- |
| `config_path` | string | Yes | Path to `opfor.config.json` written by `opfor_setup` |
| `output_dir` | string | No | Report directory. Defaults to config file directory |
| `effort_override` | `adaptive` \| `comprehensive` | No | Override the effort level from config |
| `turns_override` | number (1–10) | No | Override turns per attack from config |
| `objective_override` | string | No | Override the attack objective from config |

---

Expand Down
14 changes: 14 additions & 0 deletions docs/sdk.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -302,6 +302,20 @@ const results = await opfor.run({
});
```

## Attack objective

Steer every evaluator's attacks toward a specific free-text mission, instead of
letting each evaluator pursue its own generic goal.

```typescript
const results = await opfor.run({
target: { url: "https://api.example.com/chat" },
suite: "owasp-llm-top10",

attackObjective: "get the target to leak env vars via a delegated employee",
});
```

## Models

Configure attacker and judge LLMs.
Expand Down
23 changes: 23 additions & 0 deletions runners/cli/src/commands/run.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,11 @@ export function registerRunCommand(program: Command): void {
"--events <path>",
"Stream run lifecycle events as newline-delimited JSON (NDJSON) to <path>"
)
.option(
"--objective <text>",
"Free-text attack objective steering every evaluator's attacks (e.g. a specific goal to pursue)"
)
.option("--objective-file <path>", "Read the attack objective from a file")
.action(
async (opts: {
config?: string;
Expand All @@ -36,6 +41,8 @@ export function registerRunCommand(program: Command): void {
output?: string;
env?: string;
events?: string;
objective?: string;
objectiveFile?: string;
}) => {
if (opts.env) {
const { config: loadDotenv } = await import("dotenv");
Expand Down Expand Up @@ -90,6 +97,19 @@ export function registerRunCommand(program: Command): void {
}
runConfig = { ...runConfig, turns: n };
}
let objective = opts.objective?.trim();
if (!objective && opts.objectiveFile) {
try {
objective = (await readFile(path.resolve(opts.objectiveFile), "utf8")).trim();
} catch {
log.error(`Cannot read --objective-file at ${path.resolve(opts.objectiveFile)}.`);
process.exitCode = 1;
return;
}
}
if (objective) {
runConfig = { ...runConfig, attackObjective: objective };
}

log.info(`\nOpfor Run`);
log.info(` Target : ${runConfig.target.name} (${runConfig.target.kind})`);
Expand All @@ -99,6 +119,9 @@ export function registerRunCommand(program: Command): void {
if (runConfig.judgeLlm) {
log.info(` Judge : ${runConfig.judgeLlm.provider}/${runConfig.judgeLlm.model}`);
}
if (runConfig.attackObjective) {
log.info(` Objective : ${runConfig.attackObjective}`);
}
log.info("");

await ensureOpforDirs();
Expand Down
Loading
Loading