feat: add Runner, RunnerResult, Judge, and Evaluator

[mattrmc1]

Summary

Adds the AIEVALS types — Runner, RunnerResult, Judge, and Evaluator — and wires Evaluator.noop() into all config types. Callers can now implement a Runner to wrap any model provider, construct a Judge to evaluate AI outputs against a judge prompt with structured {score, reasoning} output, and coordinate multiple judges through an Evaluator.

New types

public interface Runner {
  RunnerResult run(String input, Map<String, Object> outputType) throws Exception;
  default RunnerResult run(String input) throws Exception;
}

Wraps a model provider SDK. outputType carries a JSON-Schema-like map when structured output is needed. Single-arg overload delegates with outputType = null.

RunnerResult.builder(String content, AIMetrics metrics)
    .raw(Object raw)
    .parsed(Map<String, Object> parsed)
    .build();

Immutable result of a Runner invocation. parsed is defensively copied and returned as unmodifiable.

public Judge(AIJudgeConfig config, Runner runner, LDLogger logger);

JudgeResult evaluate(String input, String output);
JudgeResult evaluate(String input, String output, double samplingRate);
JudgeResult evaluateMessages(List<Message> messages, RunnerResult response);
JudgeResult evaluateMessages(List<Message> messages, RunnerResult response, double samplingRate);

Evaluates AI output by invoking a runner with a formatted evaluation prompt and parsing the structured response. Sampling gate runs first — below the rate, returns sampled=false immediately. Creates a fresh tracker per evaluation via config.createTracker(). Parses score (Number, [0.0, 1.0]) and reasoning (String, optional). Runner exceptions are caught and returned as JudgeResult(success=false) — judge failures are results, not exceptions. Does not call trackJudgeResult.

public static Evaluator noop();
public Evaluator(Map<String, Judge> judges, JudgeConfiguration judgeConfiguration, LDLogger logger);

CompletableFuture<List<JudgeResult>> evaluate(String input, String output);

Coordinates sequential execution of judges. Missing judges skipped with a warning. Evaluator.noop() returns a singleton whose evaluate immediately returns an empty list. For v1.0, all configs receive Evaluator.noop().

Config type changes

AIConfig base class gains an Evaluator field and getEvaluator() accessor. AICompletionConfig and AIAgentConfig constructors accept an Evaluator. AIJudgeConfig always wires Evaluator.noop() internally — judges do not evaluate themselves.

Test plan

• ./gradlew :lib:sdk:server-ai:test passes
• JudgeTest — successful evaluation, score boundary validation (0 and 1), reasoning optional, runner exception handling (caught not rethrown), null/missing parsed output, score out of range, sampling rates (0 always skips, 1 always runs), message formatting, getter accessors
• EvaluatorTest — noop returns empty list, noop singleton identity, single/multiple judge execution, missing judge skipped, evaluator does not call trackJudgeResult, returned future is already complete
• RunnerResultTest — builder field assignment, immutability, defensive copy of parsed map

---

Note

Medium Risk
New public API surface and constructor changes on internal config types; runtime behavior stays noop for config retrieval, but Judge/Evaluator will invoke Runners and affect metrics when wired in.

Overview
Introduces the AI evaluation stack for the server AI SDK: a Runner abstraction (plus immutable RunnerResult) for provider calls with optional structured output, a Judge that runs judge prompts via a runner and returns JudgeResult (sampling, score validation, metrics via tracker), and an Evaluator that runs configured judges sequentially and exposes Evaluator.noop() for v1.

AIConfig now carries an Evaluator with getEvaluator(); completion and agent configs accept it in their constructors, while judge configs always use noop. LDAIClientImpl passes Evaluator.noop() for all retrieved completion/agent configs until managed wiring lands later.

Unit tests cover judge evaluation paths, evaluator coordination, and RunnerResult immutability.

^{Reviewed by Cursor Bugbot for commit 59835e3. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit faa4981. Configure here.}