Skip to content

fix(llmobs): openai-java payload mapping for responses, tool metadata, and prompt tracking #10644

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ygree wants to merge 41 commits into
base: master
Choose a base branch
from
Open

fix(llmobs): openai-java payload mapping for responses, tool metadata, and prompt tracking #10644

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
cbd6226
llmobs: set model tag even when llmobs disabled
ygree Feb 24, 2026
4f27673
Set metadata.stream tag no matter it's true or false
ygree Mar 2, 2026
d128d6b
Set chat/completion CACHE_READ_INPUT_TOKENS tag
ygree Mar 2, 2026
3fc5ceb
Set error nad error_type tags
ygree Mar 2, 2026
021a9d1
Use "" instead of null for the role in CompletionDecorator to comply …
ygree Mar 2, 2026
0637931
Use "" instead of null for the content to comply with TestOpenAiLlmIn…
ygree Mar 2, 2026
0cb41e1
Add missing metatadata.tool_choice
ygree Mar 3, 2026
a42f8aa
Add missing tool_definitions
ygree Mar 3, 2026
6e10255
Add source:integration tag
ygree Mar 3, 2026
34f3a07
Add missing _dd attribute to the llmobs span event
ygree Mar 4, 2026
a0c1139
Add missing error tags
ygree Mar 4, 2026
effc343
Remove error from the llmobs span event. It must be part of meta block
ygree Mar 4, 2026
c0e3876
Add missing meta.text.verbosity
ygree Mar 4, 2026
b000770
Add summaryText and encrypted_content
ygree Mar 4, 2026
53471a2
Add missing tool_calls and tool_results for responses
ygree Mar 4, 2026
2207c46
Always set stream param to produce the same request body to be aligne…
ygree Mar 4, 2026
ca6e2d1
Add OpenAI prompt-tracking reconstruction for responses (input.prompt…
ygree Mar 5, 2026
7d683b6
Fix OpenAI Responses prompt tracking to use response instructions fir…
ygree Mar 5, 2026
2c17ddc
Set LLMObs error-path defaults in Java to always emit model_name and …
ygree Mar 5, 2026
ad3b782
Add OpenAI Responses tool definition extraction to populate LLMObs to…
ygree Mar 5, 2026
1810327
Fix ChatCompletionServiceTest
ygree Mar 5, 2026
46221e4
Extract JsonValueUtils
ygree Mar 5, 2026
61ad667
Refactor OpenAI responses instrumentation to reuse ToolCallExtractor …
ygree Mar 5, 2026
f0957b7
Fix test assertions
ygree Mar 5, 2026
f3f1f75
Add integration tag
ygree Mar 6, 2026
668e955
Add ddtrace.verion
ygree Mar 6, 2026
d57402e
Improve test assertions
ygree Mar 6, 2026
a3051e3
Merge branch 'master' into ygree/llmobs-systest-fixes
ygree Mar 6, 2026
0c879ba
Fix format
ygree Mar 6, 2026
f4e3a8b
Include input messages when instructions are present in prompt tracking
ygree Mar 18, 2026
028d64f
Fix instructions role to system in prompt tracking
ygree Mar 18, 2026
82f4303
Merge branch 'master' into ygree/llmobs-systest-fixes
ygree Mar 19, 2026
717a8f0
fix LLMObsSpanMapperTest
ygree Mar 20, 2026
8420f0a
Catch exception not throwable
ygree Mar 24, 2026
91707fa
Add JsonValueUtilsTest
ygree Mar 24, 2026
3d12515
Test that on HTTP error, the OpenAI response span retains model_name …
ygree Mar 25, 2026
576cec7
Add "create response with prompt tracking" test to improve coverage o…
ygree Mar 25, 2026
ba0cb27
Add "create response with custom tool call" test to improve coverage …
ygree Mar 25, 2026
8be92d7
Prevent NPE when tag value is null
ygree Mar 26, 2026
1036ed4
Replace catch Throwable with catch Exception
ygree Mar 26, 2026
9911c51
responseCreateParamsWithPromptTracking support both known and unknown…
ygree Mar 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions dd-java-agent/agent-llmobs/src/main/java/datadog/trace/llmobs/domain/DDLLMObsSpan.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

import datadog.context.ContextScope;
import datadog.trace.api.DDSpanTypes;
import datadog.trace.api.DDTraceApiInfo;
import datadog.trace.api.DDTraceId;
import datadog.trace.api.WellKnownTags;
import datadog.trace.api.llmobs.LLMObs;
Expand Down Expand Up @@ -39,6 +40,7 @@ public class DDLLMObsSpan implements LLMObsSpan {

private static final String SERVICE = LLMOBS_TAG_PREFIX + "service";
private static final String VERSION = LLMOBS_TAG_PREFIX + "version";
private static final String DDTRACE_VERSION = LLMOBS_TAG_PREFIX + "ddtrace.version";
private static final String ENV = LLMOBS_TAG_PREFIX + "env";

private static final String LLM_OBS_INSTRUMENTATION_NAME = "llmobs";
Expand Down Expand Up @@ -76,6 +78,7 @@ public DDLLMObsSpan(
span.setTag(ENV, wellKnownTags.getEnv());
span.setTag(SERVICE, wellKnownTags.getService());
span.setTag(VERSION, wellKnownTags.getVersion());
span.setTag(DDTRACE_VERSION, DDTraceApiInfo.VERSION);

span.setTag(SPAN_KIND, kind);
spanKind = kind;
Expand Down
9 changes: 9 additions & 0 deletions ...va-agent/agent-llmobs/src/test/groovy/datadog/trace/llmobs/domain/DDLLMObsSpanTest.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import static datadog.trace.agent.test.utils.TraceUtils.runUnderTrace

import datadog.trace.agent.tooling.TracerInstaller
import datadog.trace.api.DDTags
import datadog.trace.api.DDTraceApiInfo
import datadog.trace.api.IdGenerationStrategy
import datadog.trace.api.WellKnownTags
import datadog.trace.api.telemetry.LLMObsMetricCollector
Expand Down Expand Up @@ -134,6 +135,8 @@ class DDLLMObsSpanTest extends DDSpecification{
def tagVersion = innerSpan.getTag(LLMOBS_TAG_PREFIX + "version")
tagVersion instanceof UTF8BytesString
"v1" == tagVersion.toString()

DDTraceApiInfo.VERSION == innerSpan.getTag(LLMOBS_TAG_PREFIX + "ddtrace.version")
}

def "test span with overwrites"() {
Expand Down Expand Up @@ -219,6 +222,8 @@ class DDLLMObsSpanTest extends DDSpecification{
def tagVersion = innerSpan.getTag(LLMOBS_TAG_PREFIX + "version")
tagVersion instanceof UTF8BytesString
"v1" == tagVersion.toString()

DDTraceApiInfo.VERSION == innerSpan.getTag(LLMOBS_TAG_PREFIX + "ddtrace.version")
}

def "test llm span string input formatted to messages"() {
Expand Down Expand Up @@ -270,6 +275,8 @@ class DDLLMObsSpanTest extends DDSpecification{
def tagVersion = innerSpan.getTag(LLMOBS_TAG_PREFIX + "version")
tagVersion instanceof UTF8BytesString
"v1" == tagVersion.toString()

DDTraceApiInfo.VERSION == innerSpan.getTag(LLMOBS_TAG_PREFIX + "ddtrace.version")
}

def "test llm span with messages"() {
Expand Down Expand Up @@ -326,6 +333,8 @@ class DDLLMObsSpanTest extends DDSpecification{
def tagVersion = innerSpan.getTag(LLMOBS_TAG_PREFIX + "version")
tagVersion instanceof UTF8BytesString
"v1" == tagVersion.toString()

DDTraceApiInfo.VERSION == innerSpan.getTag(LLMOBS_TAG_PREFIX + "ddtrace.version")
}

def "finish records span.finished telemetry when LLMObs enabled"() {
Expand Down
2 changes: 1 addition & 1 deletion dd-java-agent/instrumentation/openai-java/openai-java-3.0/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
apply from: "$rootDir/gradle/java.gradle"

def minVer = '3.0.0'
def minVer = '3.0.1'
Copy link
Contributor Author

@ygree ygree Mar 24, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ResponseTextConfig fun verbosity(): Optional<Verbosity> was added in 3.0.1 openai/openai-java@c1de354#diff-6b385fb153d457757ba112e6117593cb59da6af308cce0f9b6f26e3885befc6cR73


muzzle {
pass {
Expand Down
131 changes: 116 additions & 15 deletions ...-3.0/src/main/java/datadog/trace/instrumentation/openai_java/ChatCompletionDecorator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,19 @@
package datadog.trace.instrumentation.openai_java;

import static datadog.trace.instrumentation.openai_java.JsonValueUtils.jsonValueMapToObject;
import static datadog.trace.instrumentation.openai_java.JsonValueUtils.jsonValueToObject;

import com.openai.core.JsonValue;
import com.openai.helpers.ChatCompletionAccumulator;
import com.openai.models.FunctionDefinition;
import com.openai.models.chat.completions.ChatCompletion;
import com.openai.models.chat.completions.ChatCompletionChunk;
import com.openai.models.chat.completions.ChatCompletionCreateParams;
import com.openai.models.chat.completions.ChatCompletionFunctionTool;
import com.openai.models.chat.completions.ChatCompletionMessage;
import com.openai.models.chat.completions.ChatCompletionMessageParam;
import com.openai.models.chat.completions.ChatCompletionMessageToolCall;
import com.openai.models.chat.completions.ChatCompletionTool;
import datadog.trace.api.Config;
import datadog.trace.api.llmobs.LLMObs;
import datadog.trace.bootstrap.instrumentation.api.AgentSpan;
Expand All @@ -31,19 +38,24 @@ public void withChatCompletionCreateParams(
AgentSpan span, ChatCompletionCreateParams params, boolean stream) {
span.setResourceName(CHAT_COMPLETIONS_CREATE);
span.setTag(CommonTags.OPENAI_REQUEST_ENDPOINT, "/v1/chat/completions");
if (!llmObsEnabled) {
if (params == null) {
return;
}
Optional<String> modelName = params.model()._value().asString();
modelName.ifPresent(str -> span.setTag(CommonTags.OPENAI_REQUEST_MODEL, str));

span.setTag(CommonTags.SPAN_KIND, Tags.LLMOBS_LLM_SPAN_KIND);
if (params == null) {
if (!llmObsEnabled) {
return;
}
params
.model()
._value()
.asString()
.ifPresent(str -> span.setTag(CommonTags.OPENAI_REQUEST_MODEL, str));

// Keep model_name and output shape stable on error paths where no response is available.
modelName.ifPresent(
str -> {
span.setTag(CommonTags.MODEL_NAME, str);
span.setTag(CommonTags.OUTPUT, Collections.singletonList(LLMObs.LLMMessage.from("", "")));
});

span.setTag(CommonTags.SPAN_KIND, Tags.LLMOBS_LLM_SPAN_KIND);

span.setTag(
CommonTags.INPUT,
Expand All @@ -55,9 +67,7 @@ public void withChatCompletionCreateParams(
// maxTokens is deprecated but integration tests missing to provide maxCompletionTokens
params.maxTokens().ifPresent(v -> metadata.put("max_tokens", v));
params.temperature().ifPresent(v -> metadata.put("temperature", v));
if (stream) {
metadata.put("stream", true);
}
metadata.put("stream", stream);
params
.streamOptions()
.ifPresent(
Expand All @@ -72,6 +82,92 @@ public void withChatCompletionCreateParams(
params.n().ifPresent(v -> metadata.put("n", v));
params.seed().ifPresent(v -> metadata.put("seed", v));
span.setTag(CommonTags.METADATA, metadata);
params
.toolChoice()
.ifPresent(
toolChoice -> {
String choice = null;
if (toolChoice.isAuto()) {
choice = "auto";
} else if (toolChoice.isAllowedToolChoice()) {
choice = "allowed_tools";
} else if (toolChoice.isNamedToolChoice()) {
choice = "function";
} else if (toolChoice.isNamedToolChoiceCustom()) {
choice = "custom";
}
if (choice != null) {
metadata.put("tool_choice", choice);
}
});

List<ChatCompletionTool> tools = params.tools().orElse(Collections.emptyList());
if (!tools.isEmpty()) {
span.setTag(CommonTags.TOOL_DEFINITIONS, extractToolDefinitions(tools));
}
}

private List<Map<String, Object>> extractToolDefinitions(List<ChatCompletionTool> tools) {
List<Map<String, Object>> toolDefinitions = new ArrayList<>();
for (ChatCompletionTool tool : tools) {
if (tool.isFunction()) {
Map<String, Object> toolDef = extractFunctionToolDef(tool.asFunction());
if (toolDef != null) {
toolDefinitions.add(toolDef);
}
}
}
return toolDefinitions;
}

private static Map<String, Object> extractFunctionToolDef(ChatCompletionFunctionTool funcTool) {
// Try typed access first (works when built programmatically)
Optional<FunctionDefinition> funcDefOpt = funcTool._function().asKnown();
if (funcDefOpt.isPresent()) {
FunctionDefinition funcDef = funcDefOpt.get();
Map<String, Object> toolDef = new HashMap<>();
toolDef.put("name", funcDef.name());
funcDef.description().ifPresent(desc -> toolDef.put("description", desc));
funcDef
.parameters()
.ifPresent(
params ->
toolDef.put("schema", jsonValueMapToObject(params._additionalProperties())));
return toolDef;
}

// Fall back to raw JSON extraction (when deserialized from HTTP request)
Optional<JsonValue> rawOpt = funcTool._function().asUnknown();
if (!rawOpt.isPresent()) {
return null;
}
Optional<Map<String, JsonValue>> objOpt = rawOpt.get().asObject();
if (!objOpt.isPresent()) {
return null;
}
Map<String, JsonValue> obj = objOpt.get();
JsonValue nameValue = obj.get("name");
if (nameValue == null) {
return null;
}
Optional<String> nameOpt = nameValue.asString();
if (!nameOpt.isPresent()) {
return null;
}
Map<String, Object> toolDef = new HashMap<>();
toolDef.put("name", nameOpt.get());
JsonValue descValue = obj.get("description");
if (descValue != null) {
descValue.asString().ifPresent(desc -> toolDef.put("description", desc));
}
JsonValue paramsValue = obj.get("parameters");
if (paramsValue != null) {
Object schema = jsonValueToObject(paramsValue);
if (schema != null) {
toolDef.put("schema", schema);
}
}
return toolDef;
}

private static LLMObs.LLMMessage llmMessage(ChatCompletionMessageParam m) {
Expand All @@ -97,13 +193,14 @@ private static LLMObs.LLMMessage llmMessage(ChatCompletionMessageParam m) {
}

public void withChatCompletion(AgentSpan span, ChatCompletion completion) {
if (!llmObsEnabled) {
return;
}
String modelName = completion.model();
span.setTag(CommonTags.OPENAI_RESPONSE_MODEL, modelName);
span.setTag(CommonTags.MODEL_NAME, modelName);

if (!llmObsEnabled) {
return;
}

List<LLMObs.LLMMessage> output =
completion.choices().stream()
.map(ChatCompletionDecorator::llmMessage)
Expand All @@ -117,6 +214,10 @@ public void withChatCompletion(AgentSpan span, ChatCompletion completion) {
span.setTag(CommonTags.INPUT_TOKENS, usage.promptTokens());
span.setTag(CommonTags.OUTPUT_TOKENS, usage.completionTokens());
span.setTag(CommonTags.TOTAL_TOKENS, usage.totalTokens());
usage
.promptTokensDetails()
.flatMap(details -> details.cachedTokens())
.ifPresent(v -> span.setTag(CommonTags.CACHE_READ_INPUT_TOKENS, v));
});
}

Expand All @@ -127,7 +228,7 @@ private static LLMObs.LLMMessage llmMessage(ChatCompletion.Choice choice) {
if (roleOpt.isPresent()) {
role = String.valueOf(roleOpt.get());
}
String content = msg.content().orElse(null);
String content = msg.content().orElse("");

Optional<List<ChatCompletionMessageToolCall>> toolCallsOpt = msg.toolCalls();
if (toolCallsOpt.isPresent() && !toolCallsOpt.get().isEmpty()) {
Expand Down
1 change: 1 addition & 0 deletions ...ava-3.0/src/main/java/datadog/trace/instrumentation/openai_java/ChatCompletionModule.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ public String[] helperClassNames() {
packageName + ".CommonTags",
packageName + ".ChatCompletionDecorator",
packageName + ".OpenAiDecorator",
packageName + ".JsonValueUtils",
packageName + ".HttpResponseWrapper",
packageName + ".HttpStreamResponseWrapper",
packageName + ".HttpStreamResponseStreamWrapper",
Expand Down
9 changes: 9 additions & 0 deletions ...a/openai-java-3.0/src/main/java/datadog/trace/instrumentation/openai_java/CommonTags.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,20 @@ interface CommonTags {
String MODEL_PROVIDER = TAG_PREFIX + LLMObsTags.MODEL_PROVIDER;

String ML_APP = TAG_PREFIX + LLMObsTags.ML_APP;
String INTEGRATION = TAG_PREFIX + "integration";
String VERSION = TAG_PREFIX + "version";
String DDTRACE_VERSION = TAG_PREFIX + "ddtrace.version";
String SOURCE = TAG_PREFIX + "source";

String ERROR = TAG_PREFIX + "error";
String ERROR_TYPE = TAG_PREFIX + "error_type";

String ENV = TAG_PREFIX + "env";
String SERVICE = TAG_PREFIX + "service";
String PARENT_ID = TAG_PREFIX + "parent_id";

String TOOL_DEFINITIONS = TAG_PREFIX + "tool_definitions";

String METRIC_PREFIX = "_ml_obs_metric.";
String INPUT_TOKENS = METRIC_PREFIX + "input_tokens";
String OUTPUT_TOKENS = METRIC_PREFIX + "output_tokens";
Expand All @@ -33,4 +41,5 @@ interface CommonTags {
String CACHE_READ_INPUT_TOKENS = METRIC_PREFIX + "cache_read_input_tokens";

String REQUEST_REASONING = "_ml_obs_request.reasoning";
String REQUEST_PROMPT = "_ml_obs_request.prompt";
}
Loading
Loading