Add packrat-style memoization to grammar matcher

[curtisman]

Summary

Add packrat-style memoization to the grammar matcher's sub-rule entry path. When the matcher enters a RulesPart alternation, it now records whether the entry succeeded or failed and caches the result keyed by (rules, index, leadingSpacingMode, pendingWildcard, requireValue, carrier). Subsequent entries with the same context short-circuit: failures return false immediately, and successes replay a captured delta onto the live state without re-executing the sub-rule body.

Design

Failure caching: A MemoMarkerBacktrack sentinel frame is pushed onto the backtrack chain before each alternation cursor. When the marker surfaces in tryNextBacktrack (all alternatives exhausted) with anyMemberFinalized === false, the entry context is recorded as "failed" in the per-call MemoCache. Future entries with the same key skip the entire sub-rule.

Success caching: When a sub-rule member's body completes, captureSuccessDelta saves chain-head pointers into the live state's immutable cons-lists, plus a captureBase for deferred rebasing. On cache hit, applyDelta walks the chain segments at replay time, creating new nodes with rebased IDs (replayBase - captureBase). Additional deltas are pushed as MemoReplayBacktrack frames in reverse order so LIFO pop yields source order.

Suffix-failure pruning: After a memo-replay success, if the remainder of the grammar fails for a given delta, the suffixStateKey (packed endOffset + spacing modes + pending wildcard) is recorded on the replay frame. Subsequent deltas with the same suffix-state key are skipped without executing the suffix, eliminating redundant suffix failures. This turns the pathological case from ~830ms to ~140ms (6x on top of the initial memoization speedup).

Chain-pointer delta capture: captureSuccessDelta stores valuesHead/valuesStop and valueIdsHead/valueIdsStop pointers directly into the immutable cons-lists. No chain walking, no flat-array allocation, no deep-copy at capture time (O(1) pointer saves). Rebasing via rebaseMatchedValue is deferred entirely to applyDelta at replay time. For the pathological grammar with ~49K deltas (most never replayed), this eliminates ~192K intermediate allocations at capture time.

Soundness boundaries: Success caching is suppressed (noSuccessCache) for repeat entries (whose finalize pushes a CONTINUE backtrack frame not capturable in a single delta), entries with an active pending wildcard (the captured substring depends on the wildcard's start position, which is not in the cache key), and entries where the outer is not tracking values (the cache key does not distinguish null vs non-null tracking). Failure caching is always sound.

Cache key: Encoded as a packed number (index << 5) | flagBits to avoid per-lookup string allocation. The 5 low bits encode leadingSpacingMode (2 bits), pendingWildcard, requireValue, and carrier.

Opt-in/out: Controlled by GrammarMatchOptions.memoization (defaults to true). When false, no marker is pushed and the matcher behaves identically to the pre-memoization code path.

Helpers

• memoLookup: consolidated cache-probe + marker-push logic extracted from enterRulesAlternation, handling failure short-circuit, success replay, and fresh-marker push in one place.
• requiresValue: predicate extracted from inline checks at multiple rule-entry sites.
• copyValueIdChainWithDelta: single-pass iterative deep-copy used by applyDelta to rebase ValueIdNode chains at replay time and for carrier chain reconstruction.
• suffixStateKey: packs delta fields influencing suffix matching into a numeric key for fast equality checks.

Performance

Stage	Pathological grammar time
No memoization	~13.5s
+ Memoization	~830ms (16x)
+ Suffix-failure pruning	~140ms (6x)
+ Chain-pointer capture	~140ms (eliminates capture allocations; same wall time, less GC pressure)

Testing

• memoizationCoverage.spec.ts (762 lines): 13 describe blocks covering flag parity, failure cache, success delta fidelity, valueId rebasing, cache key discrimination, carrier mode, noSuccessCache boundaries, replay LIFO ordering, wildcard deltas, lastMatchedPartInfo preservation, policy cross-product parity, and suppression with active memo frames.
• suffixFailurePruning.spec.ts (new): Tests lossless parity (memo on vs off produce identical results), activation (verifies pruning fires and reduces work), and key discrimination (different endOffsets/spacing modes are not conflated).
• Fuzz harness: "memo-parity" validation in fuzzHarness.ts / fuzzRunner.ts runs every generated grammar with memo on vs off and compares results. 5 fuzz suites in grammarFuzz.spec.ts crossing memo parity with policy overrides.
• Pathology test: re-enabled from xit to it with a 2000ms timeout.
• All 70 test suites (16,103 tests) pass.