-
Notifications
You must be signed in to change notification settings - Fork 10
feat: Add AIConfigTracker with at-most-once tracking and resumption tokens #179
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
mattrmc1
wants to merge
15
commits into
main
Choose a base branch
from
mmccarthy/AIC-2664/ai-config-tracker-overhaul
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
7c4dbde
[AIC-2664] Impl trackers (first pass)
mattrmc1 a0c8784
fix: default tracker version to 1 and remove version clamp from token…
mattrmc1 bed4ca2
guard against null AIMetrics
mattrmc1 2b47c86
fix: guard against blank metricKey and infinite/invalid score
mattrmc1 4ef3de2
fix: MAX_TOKEN_BYTES -> MAX_TOKEN_LENGTH
mattrmc1 1be0a1e
fix: guard against empty runId and configKey
mattrmc1 8e81ea0
fix: Add warning comment to createTracker public call
mattrmc1 e81e2f5
fix: use trim + isEmpty to support java 8
mattrmc1 c21fdd7
fix: stop trackMetricsOf clock before running metrics extractor
mattrmc1 4c96dca
fix: record operation duration when trackMetricsOf extractor throws
mattrmc1 4da5478
fix: downgrade null-arg track logs from warn to debug per spec
mattrmc1 394a044
fix: remove unnecessary NoOpAIConfigTracker
mattrmc1 5381bf4
fix: remove resumption-token length cap
mattrmc1 3aa5d08
fix: Add security note to LDAIConfigTracker.getResumptionToken()
mattrmc1 121b140
fix: Add security note to MetricSummary.getResumptionToken()
mattrmc1 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
167 changes: 160 additions & 7 deletions
167
lib/sdk/server-ai/src/main/java/com/launchdarkly/sdk/server/ai/LDAIConfigTracker.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,16 +1,169 @@ | ||
| package com.launchdarkly.sdk.server.ai; | ||
|
|
||
| import com.launchdarkly.sdk.server.ai.datamodel.LDAITrackingTypes.AIMetrics; | ||
| import com.launchdarkly.sdk.server.ai.datamodel.LDAITrackingTypes.FeedbackKind; | ||
| import com.launchdarkly.sdk.server.ai.datamodel.LDAITrackingTypes.JudgeResult; | ||
| import com.launchdarkly.sdk.server.ai.datamodel.LDAITrackingTypes.MetricSummary; | ||
| import com.launchdarkly.sdk.server.ai.datamodel.LDAITrackingTypes.TokenUsage; | ||
| import com.launchdarkly.sdk.server.ai.datamodel.LDAITrackingTypes.TrackData; | ||
|
|
||
| import java.time.Duration; | ||
| import java.util.List; | ||
| import java.util.concurrent.Callable; | ||
| import java.util.function.Function; | ||
|
|
||
| /** | ||
| * Reports events related to a single AI run of an {@link AIConfig}. | ||
| * <p> | ||
| * A tracker is obtained from a retrieved config via {@link AIConfig#createTracker()}. Each tracker | ||
| * corresponds to one AI run and is used to record metrics such as model usage, duration, and | ||
| * feedback against the AI Config it was created from. | ||
| * A tracker is obtained from a retrieved config via {@link AIConfig#createTracker()}, or | ||
| * reconstructed from a resumption token via {@link LDAIClient#createTracker(String, com.launchdarkly.sdk.LDContext)}. | ||
| * Each tracker corresponds to one AI run and is used to record metrics such as model usage, | ||
| * duration, and feedback against the AI Config it was created from. | ||
| * <p> | ||
| * Most tracking methods are at-most-once: a second call to the same method on the same tracker | ||
| * is silently dropped. {@link #trackToolCall(String)} and {@link #trackJudgeResult(JudgeResult)} | ||
| * are multi-fire — each call records a distinct event. | ||
| * <p> | ||
| * <strong>This interface is an intentional placeholder.</strong> The metric- and feedback-reporting | ||
| * methods (and resumption-token support) are introduced in a later step of the AI SDK build-out; it | ||
| * is defined here so that the public config types expose a stable {@code createTracker()} surface. | ||
| * The only implementation in this release is an internal no-op. | ||
| * Implementations are thread-safe. | ||
| */ | ||
| public interface LDAIConfigTracker { | ||
|
|
||
| /** | ||
| * Returns the correlation metadata for this tracker's run. | ||
| * | ||
| * @return the track data, never {@code null} | ||
| */ | ||
| TrackData getTrackData(); | ||
|
|
||
| /** | ||
| * Returns the resumption token for this run. | ||
| * <p> | ||
| * The resumption token encodes the run's identity and can be passed to | ||
| * {@link LDAIClient#createTracker(String, com.launchdarkly.sdk.LDContext)} to reconstruct a | ||
| * tracker on a subsequent request (for example, in a streaming scenario). | ||
| * <p> | ||
| * <strong>Security note:</strong> resumption tokens embed flag-evaluation details such as the | ||
| * variation key and config version. Keep tokens server-side and do not round-trip them through | ||
| * untrusted clients where they could leak flag-targeting information. | ||
| * | ||
| * @return the resumption token, or {@code null} if not available | ||
| */ | ||
| String getResumptionToken(); | ||
|
|
||
| /** | ||
| * Records the duration of the AI generation. | ||
| * <p> | ||
| * At-most-once: subsequent calls on the same tracker are silently dropped. | ||
| * | ||
| * @param duration the duration; ignored if {@code null} | ||
| */ | ||
| void trackDuration(Duration duration); | ||
|
|
||
| /** | ||
| * Executes the given operation and records its wall-clock duration. | ||
| * <p> | ||
| * The duration is recorded even if the operation throws. Equivalent to wrapping the operation | ||
| * in a try/finally that calls {@link #trackDuration(Duration)}. | ||
| * | ||
| * @param <T> the return type of the operation | ||
| * @param operation the operation to execute and time; must not be {@code null} | ||
| * @return the result of the operation | ||
| * @throws Exception if the operation throws | ||
| */ | ||
| <T> T trackDurationOf(Callable<T> operation) throws Exception; | ||
|
|
||
| /** | ||
| * Records the time from request start to receipt of the first token. | ||
| * <p> | ||
| * At-most-once: subsequent calls on the same tracker are silently dropped. | ||
| * | ||
| * @param duration the time to first token; ignored if {@code null} | ||
| */ | ||
| void trackTimeToFirstToken(Duration duration); | ||
|
|
||
| /** | ||
| * Records that the AI generation succeeded. | ||
| * <p> | ||
| * At-most-once and mutually exclusive with {@link #trackError()}: whichever is called first wins. | ||
| */ | ||
| void trackSuccess(); | ||
|
|
||
| /** | ||
| * Records that the AI generation failed. | ||
| * <p> | ||
| * At-most-once and mutually exclusive with {@link #trackSuccess()}: whichever is called first wins. | ||
| */ | ||
| void trackError(); | ||
|
|
||
| /** | ||
| * Records user feedback for this AI generation. | ||
| * <p> | ||
| * At-most-once: subsequent calls on the same tracker are silently dropped. | ||
| * | ||
| * @param kind the feedback kind; ignored if {@code null} | ||
| */ | ||
| void trackFeedback(FeedbackKind kind); | ||
|
|
||
| /** | ||
| * Records token usage for this AI generation. | ||
| * <p> | ||
| * At-most-once: subsequent calls on the same tracker are silently dropped. Calls where all | ||
| * counts are zero do not consume the at-most-once slot. | ||
| * | ||
| * @param tokens the token usage; ignored if {@code null} | ||
| */ | ||
| void trackTokens(TokenUsage tokens); | ||
|
|
||
| /** | ||
| * Records a single tool call made during this AI generation. | ||
| * <p> | ||
| * Multi-fire: every call emits an event. | ||
| * | ||
| * @param toolKey the tool key; ignored if {@code null} | ||
| */ | ||
| void trackToolCall(String toolKey); | ||
|
|
||
| /** | ||
| * Records multiple tool calls made during this AI generation. | ||
| * <p> | ||
| * Equivalent to calling {@link #trackToolCall(String)} for each key. | ||
| * | ||
| * @param toolKeys the tool keys; ignored if {@code null} | ||
| */ | ||
| void trackToolCalls(List<String> toolKeys); | ||
|
|
||
| /** | ||
| * Records the result of a judge evaluation. | ||
| * <p> | ||
| * Multi-fire per judge metric key. The result is silently skipped if it was not sampled, if | ||
| * the evaluation did not succeed, or if the metric key or score is absent. | ||
| * | ||
| * @param result the judge result; ignored if {@code null} | ||
| */ | ||
| void trackJudgeResult(JudgeResult result); | ||
|
|
||
| /** | ||
| * Executes the given operation and tracks its metrics using the extracted {@link AIMetrics}. | ||
| * <p> | ||
| * Tracks duration (preferring runner-reported duration when present), success or error, tokens, | ||
| * and tool calls. If the operation throws, {@link #trackError()} is called and the exception | ||
| * is re-thrown. | ||
| * | ||
| * @param <T> the return type of the operation | ||
| * @param metricsExtractor a function that extracts {@link AIMetrics} from the operation result; | ||
| * exceptions from the extractor propagate to the caller | ||
| * @param operation the AI operation to execute; must not be {@code null} | ||
| * @return the result of the operation | ||
| * @throws Exception if the operation or the metrics extractor throws | ||
| */ | ||
| <T> T trackMetricsOf( | ||
| Function<? super T, AIMetrics> metricsExtractor, | ||
| Callable<T> operation) throws Exception; | ||
|
|
||
| /** | ||
| * Returns a snapshot of all metrics tracked so far on this tracker. | ||
| * | ||
| * @return the metric summary, never {@code null} | ||
| */ | ||
| MetricSummary getSummary(); | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.