Feature Request: Prompt Execution Profiler (Token & Structure Analysis)

[Alan-Jowett]

Feature Request: Prompt Execution Profiler (Token & Structure Analysis)

Summary
As PromptKit generates increasingly structured, multi‑layered prompts, it becomes difficult to understand where tokens are being consumed and why. This makes it hard to optimize personas, protocols, templates, and context blocks. A Prompt Execution Profiler would analyze LLM session logs and surface actionable insights about inefficiencies in prompt structure and execution.

Problem
Right now, prompt optimization is largely guesswork. We lack visibility into:

• which parts of a prompt are actually used
• where the model re‑derives information unnecessarily
• which personas or protocol steps cause rework
• where drift or self‑correction occurs
• which context blocks are dead weight
• token hotspots across the prompt lifecycle

Without observability, we can’t systematically improve prompt efficiency or reduce token burn.

Proposal
Introduce a PromptKit Profiler that ingests session logs and produces a structured analysis of prompt execution. The profiler would:

1. Parse logs into prompts, responses, retries, deltas, and token counts
2. Map segments to PromptKit components (persona, protocol step, template, context block)
3. Detect inefficiencies, such as:
   • redundant reasoning
   • unnecessary re‑derivations
   • unused context
   • persona verbosity
   • protocol loops
   • drift or contradictions
4. Attribute token burn to specific components
5. Recommend optimizations, such as:
   • compressing personas
   • pruning context
   • tightening protocol steps
   • restructuring templates
   • caching invariants

Why an LLM Can Do This
The inefficiencies are semantic, not computational. LLMs are well‑suited to detect:

• repeated reasoning patterns
• irrelevant context
• structural redundancy
• over‑specification
• drift between steps

Session logs already contain the raw signals; the profiler would turn them into actionable insight.

Outcome
A Prompt Execution Profiler would provide:

• observability into prompt performance
• measurable optimization opportunities
• reduced token burn
• more efficient personas, protocols, and templates
• a foundation for future PromptKit tooling (linter, optimizer, debugger)

This would make PromptKit more scalable, predictable, and cost‑efficient as prompt complexity grows.

[Alan-Jowett]

Thank you for the PR @themechbro! The React dashboard looks polished. I realized that the issue wasn't super crisp on the requirements and I apologize.

Scoping the v1 so contributors have a concrete target.

Key insight: The profiler is a PromptKit template, not a script

This follows PromptKit's architecture: the profiler is a prompt that an LLM executes against a session log. The LLM does the reasoning about token usage, redundancy, and inefficiency — we don't write a parser, we write a template that instructs the LLM how to analyze a session.

What we're measuring

The profiler analyzes a full LLM session log (all turns — prompt, responses, follow-ups) to answer: "Where did this session spend tokens inefficiently, and which PromptKit components contributed?"

The key inefficiencies to detect:

• Redundant reasoning: The LLM re-derives something it already concluded earlier
• False starts / backtracking: The LLM starts down a path, abandons it, restarts
• Unnecessary re-derivation: Information in the prompt is re-derived from scratch instead of referenced
• Protocol loops: A protocol phase is executed multiple times when once should suffice
• Unused context: Context blocks in the prompt that the LLM never references

v1 scope: A new PromptKit template

Template: profile-session (or similar)
Persona: Could reuse specification-analyst or introduce a prompt-engineer persona
Format: User-selectable at prompt assembly time:

• Markdown report — analysis and optimization recommendations for human/LLM review
• JSON — structured metrics for consumption by a dashboard or other tooling

What the template instructs the LLM to do

Given a session log + the assembled prompt that was used:

1. Segment the log into turns (prompt → response → follow-up → ...)
2. Estimate token usage per turn and per section
3. Map turns to PromptKit components: Which protocol phase is each response executing? Which persona behaviors are being invoked?
4. Detect inefficiencies: Redundant reasoning, backtracking, re-derivation, unused context — attributed to specific PromptKit components (e.g., "Protocol Phase 3 was executed twice because the self-verification guardrail triggered a redo")
5. Produce optimization recommendations: Tied back to PromptKit components (e.g., "Persona section is 3K tokens — consider compressing behavioral constraints" or "Protocol Phase 2 and Phase 4 have overlapping checks — consider merging")

Output (user-selectable)

• Markdown: Executive summary, per-component breakdown, top optimization opportunities, recommended component changes
• JSON: Token counts per turn, per component, inefficiency annotations with type/severity/location

What v1 is NOT

• Not a standalone script, dashboard, or web application
• Not a real-time tool — batch analysis of completed sessions
• Not an optimizer — it identifies inefficiencies and recommends changes, the maintainer decides

Why this matters

As PromptKit prompts get more complex (the audit-traceability prompt is 27K chars with 6 phases), understanding where the LLM spends effort during execution lets us write tighter protocols and leaner personas. The profiler closes the feedback loop: assemble → execute → profile → improve → reassemble.

Future: Visualization Dashboard

Once the template produces structured JSON output, a visualization dashboard would be a natural next step — rendering token breakdowns, component attribution, and inefficiency hotspots as interactive charts. A React/Recharts approach (like the one explored in PR #63) could be a great starting point for that UI layer. The template comes first (it produces the data), the dashboard comes second (it visualizes it).