feat: Add AIConfigTracker with at-most-once tracking and resumption tokens

[mattrmc1]

Summary

Implements the full LDAIConfigTracker interface — previously a stub. Callers can now record AI operation metrics (duration, tokens, success/error, feedback, tool calls, judge results) with at-most-once enforcement, extract metrics from runner operations via trackMetricsOf, and reconstruct trackers across processes via resumption tokens.

Tracking methods

void trackDuration(Duration duration);
<T> T trackDurationOf(Callable<T> operation) throws Exception;

Records wall-clock duration. Null silently dropped (debug log); negatives clamped to zero. trackDurationOf wraps a Callable, measures via System.nanoTime(), records duration in finally even on exception.

<T> T trackMetricsOf(Function<? super T, AIMetrics> metricsExtractor, Callable<T> operation) throws Exception;

All-in-one wrapper: starts timer, invokes operation, stops clock before calling the extractor (slow extractors don't inflate duration). On success: prefers runner-reported durationMs over wall-clock, then delegates to trackSuccess/trackError, trackTokens, trackToolCalls. On exception: records wall-clock duration, calls trackError, rethrows. If the extractor itself throws, operation duration is still recorded before propagating — trackError is NOT called since the AI operation succeeded.

void trackSuccess();
void trackError();

Share a single AtomicReference<Boolean> guard — only the first to fire wins.

void trackFeedback(FeedbackKind kind);

Validates and resolves the event name before claiming the at-most-once guard, so null/invalid input doesn't burn the slot.

void trackTokens(TokenUsage tokens);

Emits events for each positive count (total, input, output). All-zero usage does not consume the at-most-once slot.

void trackToolCall(String toolKey);
void trackToolCalls(List<String> toolKeys);

Multi-fire (not at-most-once). Each call emits a separate $ld:ai:tool_call event.

void trackJudgeResult(JudgeResult result);

Silently dropped when not sampled, not successful, or when metricKey is blank/null or score is null/non-finite. Multi-fire.

void trackTimeToFirstToken(Duration duration);

Records time-to-first-token duration. At-most-once.

Resumption tokens

String getResumptionToken();                                            // on LDAIConfigTracker
LDAIConfigTracker createTracker(String resumptionToken, LDContext context); // on LDAIClient

getResumptionToken() returns URL-safe Base64 (no padding) JSON containing { runId, configKey, variationKey, version, graphKey }. variationKey and graphKey omitted when null. No length cap — large config keys are supported. Empty runId / configKey are rejected on decode.

Tracker factory wiring

LDAIClientImpl now creates real LDAIConfigTrackerImpl instances. A private trackerFactory method captures config identity and returns a Supplier<LDAIConfigTracker> producing a fresh tracker with a new runId on each call. Default configs also get real trackers. Default version is 1.

NoOpAIConfigTracker deleted — no longer needed.

New types

FeedbackKind — enum: POSITIVE, NEGATIVE.

TokenUsage — immutable record: total, input, output.

AIMetrics — immutable builder: success, optional tokens, durationMs, toolCalls.

JudgeResult — immutable builder: metricKey, score, sampled, success, optional judgeConfigKey, reasoning, errorMessage.

MetricSummary — snapshot of all tracked metrics plus resumption token.

TrackData — run identity fields with toLDValue().

Thread safety

All at-most-once slots use AtomicReference<T>.compareAndSet(null, value) — single atomic guard+value, no race window. Tool calls use CopyOnWriteArrayList.

Test plan

• ./gradlew :lib:sdk:server-ai:test passes
• LDAIConfigTrackerImplTest — duration (emit, clamp, at-most-once, null), durationOf (success + exception), success/error (emit, shared guard both directions), feedback (emit, at-most-once, null slot preservation), tokens (positive counts, zero skip, slot preservation), tool calls (multi-fire, null), judge result (sampled/success/metricKey/score guards, multi-fire), trackMetricsOf (success path, error path, extractor failure duration tracking, null AIMetrics guard), variationKey/graphKey in payload, concurrency (20-thread contention), constructor null rejection
• ResumptionTokensTest — encode/decode round-trips, large keys, special character escaping, null/malformed rejection, empty runId/configKey rejection

---

Note

Medium Risk
New public tracking API and telemetry emission change observability behavior; resumption tokens embed flag-targeting metadata if exposed to clients.

Overview
Replaces the no-op LDAIConfigTracker stub with a full implementation that emits LaunchDarkly custom metrics for AI runs (duration, time-to-first-token, success/error, feedback, tokens, tool calls, and judge scores).

LDAIClientImpl now supplies a per-config Supplier that creates LDAIConfigTrackerImpl instances (new UUID runId per createTracker()), including when falling back to caller defaults. NoOpAIConfigTracker is removed. LDAIClient#createTracker(String, LDContext) decodes a resumption token to continue the same run across requests.

The expanded LDAIConfigTracker API adds trackMetricsOf, getSummary, getTrackData, and getResumptionToken, with at-most-once semantics on most metrics (tool calls and judge results are multi-fire). LDAITrackingTypes holds the new immutable value types; ResumptionTokens encodes/decodes URL-safe Base64 JSON for run identity (docs warn tokens can expose variation key / version and should stay server-side).

^{Reviewed by Cursor Bugbot for commit 121b140. Bugbot is set up for automated code reviews on this repo. Configure here.}

[jsonbailey]

A few low-severity / cosmetic notes on the resumption token handling.

[jsonbailey]

Consider documenting here (the producer side, where a caller decides what to do with the value) that the resumption token embeds the flag's variationKey and version — so it should be kept server-side and not exposed to untrusted clients (e.g. round-tripped through a browser), where it could leak flag targeting details.

[jsonbailey]

Thanks for adding the security note — it reads well. One follow-up: it currently lives on createTracker (the consumer side). It should be on getResumptionToken() at a minimum, since that's where a caller is holding the token and deciding where to send it — that's the point where the warning actually changes behavior. Keeping it on createTracker as well is fine (a copy on both ends doesn't hurt), but the producer side is the important one.

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.