Skip to content

feat: chaos for proxy mode (drop, disconnect, malformed)#122

Merged
jpr5 merged 5 commits intomainfrom
feat/chaos-proxy-phase2-v2
Apr 23, 2026
Merged

feat: chaos for proxy mode (drop, disconnect, malformed)#122
jpr5 merged 5 commits intomainfrom
feat/chaos-proxy-phase2-v2

Conversation

@marthakelly
Copy link
Copy Markdown
Contributor

@marthakelly marthakelly commented Apr 22, 2026 •
edited by jpr5
Loading

Summary

Chaos (drop, disconnect, malformed) now applies in proxy mode. Previously chaos was skipped for proxied requests, making them unrealistically reliable.

What it does

  • Pre-flight drop/disconnect pre-empt upstream (nothing is contacted).
  • Post-response malformed proxies to upstream, captures the response, replaces the body with invalid JSON before relay.
  • Chaos is rolled once per request, after fixture match, with resolved config (headers > fixture.chaos > server defaults).
  • SSE bypass: when upstream responds with streaming SSE, malformed chaos is skipped (bytes are already on the wire) and aimock_chaos_bypassed_total metric fires.
  • Source labeling: aimock_chaos_triggered_total and journal entries carry source: "fixture" | "proxy" to distinguish where chaos fired.
  • CORS: Access-Control-Allow-Headers includes chaos control headers and X-Test-Id for browser-based test harnesses.
  • Video status: handleVideoStatus (GET /v1/videos/:id) now evaluates chaos, consistent with all other handlers.

Design highlights

  • applyChaos split into roll+dispatch (applyChaosAction) so callers that pre-roll can't re-roll by accident.
  • proxyAndRecord returns "not_configured" | "relayed" | "handled_by_hook".
  • JournalEntry.response.source disambiguates null-fixture entries.
  • Chaos source is conditional (fixture ? "fixture" : "proxy") across all 15+ call sites.

Scope

In: malformed / drop / disconnect for proxy mode, non-streaming.
Deferred: status_override, truncate, streaming (SSE/NDJSON) mutation.

Test plan

  • Pre-flight drop/disconnect: upstream counter stays at 0, client gets 500/disconnect
  • Post-response malformed: upstream IS called, body is invalid JSON, journal records chaosAction
  • Single-roll semantics: drop journals the matched fixture + increments match count
  • SSE bypass: malformed chaos skipped, bypass metric emitted, triggered metric NOT emitted
  • Proxy failure interaction: 502 reaches client, journal correct, no chaosAction
  • Prometheus source labels: source="fixture" vs source="proxy" on chaos counter
  • Video status chaos: drop fires before video-not-found 404
  • CORS preflight: OPTIONS response includes all chaos + test-id headers
  • Binary-safe hook: beforeWriteResponse receives raw upstream bytes
  • Content-type preserved on no-chaos replay

@marthakelly marthakelly changed the base branch from feat/chaos-proxy-phase1 to main April 22, 2026 20:51
@marthakelly marthakelly changed the title feat: post-response chaos for proxy mode (phase 2) feat: chaos for proxy mode (drop, disconnect, malformed) Apr 22, 2026
@marthakelly marthakelly force-pushed the feat/chaos-proxy-phase2-v2 branch from 12abf48 to 696522a Compare April 23, 2026 01:21
@pkg-pr-new
Copy link
Copy Markdown

pkg-pr-new Bot commented Apr 23, 2026 •
edited
Loading

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@122

commit: a0be96a

@marthakelly marthakelly marked this pull request as ready for review April 23, 2026 01:26
@marthakelly marthakelly force-pushed the feat/chaos-proxy-phase2-v2 branch from 0082dc3 to 65ffff8 Compare April 23, 2026 15:58
@jpr5 jpr5 force-pushed the feat/chaos-proxy-phase2-v2 branch 2 times, most recently from f4b4ba1 to 1c67ed0 Compare April 23, 2026 17:16
@jpr5 jpr5 mentioned this pull request Apr 23, 2026
Merged
5 tasks
@jpr5 jpr5 force-pushed the feat/chaos-proxy-phase2-v2 branch from 54a524d to d66e3bf Compare April 23, 2026 18:47
@marthakelly @jpr5
feat: chaos injection for proxy mode (drop, disconnect, malformed)
8a8f745 
Chaos (drop, disconnect, malformed) now applies in proxy mode.
Previously chaos was skipped for proxied requests.

- Pre-flight drop/disconnect pre-empt upstream (nothing is contacted)
- Post-response malformed proxies to upstream, captures the response,
  replaces the body with invalid JSON before relay
- Chaos is rolled once per request, after fixture match, with resolved
  config (headers > fixture.chaos > server-level)
- SSE bypass: when the hook can't mutate a streamed response, chaos is
  skipped and an onHookBypassed callback fires
- CORS allow list includes chaos and test-id custom headers
- Chaos source label is now explicit (fixture vs proxy) across all
  applyChaos call sites
@marthakelly @jpr5
feat: wire chaos across all provider handlers
cae5cf3 
Thread applyChaos through all 12 provider handlers with conditional
source label (fixture ? "fixture" : "proxy"). Add chaos to
handleVideoStatus for completeness.
@marthakelly @jpr5
test: chaos proxy mode coverage
31735b8 
- proxy-only: pre-flight drop/disconnect, post-response malformed, SSE
  bypass, proxy failure interaction
- chaos-fixture-mode: fixture-mode pre-flight chaos, single roll,
  journal recording, video status chaos drop
- metrics: chaos counter carries source=proxy label
- recorder: pin binary-safe hook contract, proxy failure path
- server: CORS preflight includes chaos control headers
@marthakelly @jpr5
docs: chaos testing for proxy mode
4535798 
Document proxy mode behavior (drop/disconnect skip upstream, malformed
proxies then corrupts), SSE bypass, sequential rate semantics, journal
source field, Prometheus metrics, and per-request header API. Add
CHANGELOG entry for chaos-in-proxy-mode feature.
@jpr5 jpr5 force-pushed the feat/chaos-proxy-phase2-v2 branch from d66e3bf to 4535798 Compare April 23, 2026 18:49
@marthakelly marthakelly mentioned this pull request Apr 23, 2026
Closed
@jpr5 jpr5 merged commit b7b305e into main Apr 23, 2026
22 checks passed
@jpr5 jpr5 deleted the feat/chaos-proxy-phase2-v2 branch April 23, 2026 18:57
jpr5 added a commit that referenced this pull request Apr 23, 2026
@jpr5
fix: 50+ fixes across recorder, providers, CI, competitive matrix (#133)
0683040 
## Summary

Addresses all bucket (c) and (d) items from PR #122's 7-agent CR, plus
correctness issues discovered during this PR's own iterative CR process
(6 rounds, 4 fix cycles).

### Server core
- `readBody` 10MB size limit (DoS surface closed)
- Control API: error detail in responses, one-shot race fix (synchronous
splice), fixtures_loaded gauge updates on mutations
- `normalizeCompatPath` magic number replaced with readable expression
- `source` field ("fixture" | "proxy") added to JournalEntry type

### Recorder correctness
- Crash hardening: `headersSent` guards, `clientDisconnected` tracking,
encoding_format logging
- Content+toolCalls: preserve both (previously dropped content)
- Cohere v2 native detection branch in `buildFixtureResponse`
- Tool-call IDs preserved from 5 provider formats
- Reasoning/thinking extracted from 4 provider formats (including
tool-call-only)
- Multi-block text: `filter+join` instead of `find`
(Anthropic/Gemini/Bedrock)
- Thinking-only Anthropic responses handled (not error fixture)
- Empty-content responses recorded correctly (not error fixture)
- Ollama `/api/generate` format detection
- Streaming collapse reasoning propagated

### Provider handlers
- **Bedrock/Converse**: ContentWithToolCallsResponse guards,
ResponseOverrides wired into all builders + streaming, Converse-wrapped
stream format (matches collapse parser), `text_delta` type field,
Converse error envelope, webSearches warnings
- **Cohere v2**: reasoning in all builders + streaming, webSearches
warnings, response_format forwarding, assistant tool_calls preservation,
full ResponseOverrides (finish_reason, usage, id)
- **Ollama**: ContentWithToolCallsResponse, default stream:true, field
validation
- **Gemini**: tool_call_id collision fix (shared callCounter),
thought-part filtering
- **All**: journal source field on proxy paths, Azure embedding routing

### Competitive matrix pipeline
- `computeChanges` matches actual HTML structure (span class="no")
- `applyChanges` regex matches actual td>span structure
- `updateProviderCounts` scoped to competitor column (not global)
- `extractFeatures` tightened keyword patterns
- Added mokksy/ai-mocks competitor

### CI + framework
- `--auto` instead of `--admin` on fix-drift merge
- Slack secrets via env vars across 5 workflows
- Script injection prevention in notify-pr.yml (env vars + jq)
- Portable `grep -oE`, SLACK_WEBHOOK guards
- Router: RegExp `g`-flag `lastIndex` reset
- Jest/Vitest: save/restore pre-existing env vars, loadFixtures warnings

## Test plan
- [ ] 2573/2573 tests passing (69 test files)
- [ ] TSC clean
- [ ] Prettier + ESLint clean
- [ ] 0 merge conflicts with main
- [ ] 6 CR rounds converged (R1-R4 on original, R1-R6 on additions)

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@marthakelly @jpr5