From c42378c5a7078d5388b45a08ffbdd3a996cfda2e Mon Sep 17 00:00:00 2001 From: Alan Jowett Date: Thu, 19 Mar 2026 18:43:07 -0700 Subject: [PATCH 1/2] Add specification integrity roadmap to docs/roadmap.md Document the phased evolution from cross-document spec audits (shipped) through bidirectional code/spec audits, invariant extraction, and standards/protocol analysis. Includes vision for continuous semantic integration as a CI consumer of PromptKit prompts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/roadmap.md | 85 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) diff --git a/docs/roadmap.md b/docs/roadmap.md index 4a90083..d7e2241 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -78,6 +78,91 @@ A lighter-weight option: a VS Code extension that provides: - One-click assembly with output to a new editor tab - Integration with VS Code's Copilot Chat via chat participants +## Specification Integrity & Drift Detection + +> **Status: Phase 1 shipped.** The traceability audit template and +> specification-drift taxonomy landed in PR #35. Phases 2–4 are planned. + +PromptKit is evolving toward a **specification integrity engine** — a +set of composable templates that detect gaps, contradictions, and drift +across the artifacts that define a software system: requirements, design, +validation plans, source code, and tests. + +### Phase 1: Cross-Document Specification Audits ✅ + +Shipped: `audit-traceability` template, `specification-analyst` persona, +`traceability-audit` protocol, and `specification-drift` taxonomy (D1–D7). + +Audits requirements, design, and validation documents for: +- Untraced requirements (D1) and untested requirements (D2) +- Orphaned design decisions (D3) and orphaned test cases (D4) +- Assumption drift (D5) and constraint violations (D6) +- Acceptance criteria mismatch (D7) — illusory test coverage + +The design document is optional, enabling two-document (requirements ↔ +validation) or three-document audits. Extends the `document-lifecycle` +pipeline as stage 4. + +### Phase 2: Bidirectional Code ↔ Spec Audits + +The specification-drift taxonomy reserves D8–D13 for these templates: + +- **`audit-code-compliance`** — Given requirements + design, audit source + code for unimplemented requirements, violated constraints, and + undocumented behavior. Answers: "Does the code implement what was + specified?" +- **`audit-test-compliance`** — Given requirements + validation plan, + audit test code for unimplemented test cases, missing assertions, and + coverage gaps. Answers: "Do the tests verify what the plan says they + should?" +- **Drift detection** — Surface spec-only behavior (specified but not + implemented), code-only behavior (implemented but not specified), and + mismatched assumptions between documents and code. + +These templates reuse the `specification-analyst` persona and extend the +`specification-drift` taxonomy. New protocols will handle the distinct +challenge of mapping document-level claims to code-level behavior. + +### Phase 3: Invariant Extraction + +- **Invariant extraction template** — Extract MUST/SHOULD/MAY constraints, + state transitions, timing assumptions, and error conditions from + existing specifications or code. Produces structured, machine-readable + invariant sets that can feed into audit templates. +- **Spec evolution diffing** — Compare two versions of the same + specification to detect breaking changes, relaxed constraints, and + shifted assumptions. Same pattern as traceability audit but applied + across time rather than across document types. + +### Phase 4: Standards & Protocol Analysis + +Applying the spec integrity methodology to formal standards (RFCs, IETF +protocols, API specifications): + +- **`standards-analyst` persona** — Domain expertise in normative language + (RFC 2119), protocol state machines, negotiation semantics, and + interoperability requirements. +- **RFC structural audits** — Internal consistency, normative language + correctness, state machine extraction, contradiction detection. +- **RFC → validation spec generation** — Convert MUST/SHOULD/MAY + requirements from RFCs into structured test conditions covering + negotiation, error handling, timing, and interoperability. +- **RFC ↔ implementation audits** — Audit reference implementations + against their governing RFCs for drift, undocumented capabilities, + and security mismatches. + +### Vision: Continuous Semantic Integration + +The long-term direction is enabling specification integrity checks as +part of continuous integration — every PR triggers doc ↔ code ↔ validation +audits, drift is caught at commit time, and specifications stay aligned +with implementation. + +This is an **integration concern** rather than a PromptKit template — the +component that runs audits in CI would be a separate tool that *uses* +PromptKit prompts. PromptKit's role is providing the composable audit +methodology; the CI integration consumes it. + ## New Templates Planned templates based on common engineering workflows: From adb21738987308e2b7681f6f5127f2ae97df8ae8 Mon Sep 17 00:00:00 2001 From: Alan Jowett Date: Thu, 19 Mar 2026 18:53:57 -0700 Subject: [PATCH 2/2] Refine RFC roadmap: input normalization + xml2rfc output format Replace the original Phase 4 (which proposed a new persona and multiple standalone capabilities) with a leaner approach: RFCs are requirements documents with a specific format, so RFC support is primarily an input normalization template (RFC -> requirements-document) and an output format (xml2rfc v3). All downstream audit machinery reuses existing components. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/roadmap.md | 50 +++++++++++++++++++++++++++++++++---------------- 1 file changed, 34 insertions(+), 16 deletions(-) diff --git a/docs/roadmap.md b/docs/roadmap.md index d7e2241..ebcbd36 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -134,22 +134,40 @@ challenge of mapping document-level claims to code-level behavior. shifted assumptions. Same pattern as traceability audit but applied across time rather than across document types. -### Phase 4: Standards & Protocol Analysis - -Applying the spec integrity methodology to formal standards (RFCs, IETF -protocols, API specifications): - -- **`standards-analyst` persona** — Domain expertise in normative language - (RFC 2119), protocol state machines, negotiation semantics, and - interoperability requirements. -- **RFC structural audits** — Internal consistency, normative language - correctness, state machine extraction, contradiction detection. -- **RFC → validation spec generation** — Convert MUST/SHOULD/MAY - requirements from RFCs into structured test conditions covering - negotiation, error handling, timing, and interoperability. -- **RFC ↔ implementation audits** — Audit reference implementations - against their governing RFCs for drift, undocumented capabilities, - and security mismatches. +### Phase 4: RFC & Standards Support + +An RFC is fundamentally a requirements document with a specific format +and RFC 2119 normative language — the same MUST/SHOULD/MAY keywords +that PromptKit's `requirements-elicitation` protocol already produces. +This means RFC support is primarily an **input normalization** and +**output format** problem, not a new capability stack. + +**RFC in** — `extract-rfc-requirements` template: +- Takes an RFC (or internet-draft) as input, produces a standard + requirements-document as output. +- Reuses the `specification-analyst` persona (no new persona needed — + RFCs are specs). +- Needs a thin `rfc-extraction` protocol covering: section + classification, normative statement extraction, state machine + identification, cross-RFC dependency tracking, and IANA/security + considerations parsing. +- Once normalized to a requirements-document, all existing audit + machinery applies — `audit-traceability` for RFC ↔ design ↔ validation, + and future `audit-code-compliance` for RFC ↔ implementation. + +**Spec out** — `rfc-document` format: +- Produces xml2rfc v3 (RFC 7991) output: ``, `
`, + `MUST`, ``, proper `` blocks. +- Output is structurally valid xml2rfc that feeds directly into the + `xml2rfc` toolchain for rendering. +- Pairs with `author-requirements-doc` or a new `author-rfc` template + for writing internet-drafts from scratch. + +**Downstream** — everything else reuses existing components: +- RFC ↔ implementation audits = `audit-code-compliance` with + RFC-derived requirements as input. +- RFC ↔ validation = `audit-traceability` as-is. +- RFC version diffing = spec evolution diffing from Phase 3. ### Vision: Continuous Semantic Integration