Add trace.builder.tags.precedence.enabled to let explicit builder tags win

[dougqh]

Floating an idea — feedback welcome before this is hardened

This is a draft to float a precedence fix behind an off-by-default flag. Looking for reactions to the concept (and the rollout plan) before investing in telemetry + a forked on-path test.

The wart

At span build, tag contributors are applied last-wins in this order:

context.setAllTags(mergedTracerTags, mergedTracerTagsNeedsIntercept);
context.setAllTags(tagLedger);        // explicit builder tags, applied 2nd...
context.setAllTags(coreTags, coreTagsNeedsIntercept);   // ...then OVERWRITTEN by these
context.setAllTags(rootSpanTags, rootSpanTagsNeedsIntercept);
context.setAllTags(contextualTags);

So inbound-header / root-span / contextual tags silently override explicit per-span tags set via the builder. This is illogical — the explicit, more-specific tag should win — and it's been flagged in-code since 2020 (git blame: Björn): "maybe the tags set in the builder should come last, so that they override other tags."

Provenance audit (is it a wart or load-bearing?)

contributor	source	present when
coreTags	inbound header-derived tags (TagContext.getTags())	extraction/root only; null for local children
rootSpanTags	process/identity tags (runtime-id, language, _dd.*)	local root only; null for local children
contextualTags	classloader-config overrides	local root w/o service name only

• Blast radius is structurally bounded: for local children (parent is a DDSpanContext — the common non-root case), coreTags and rootSpanTags are both null, so the hazard can only occur on root / extraction-path spans.
• Not load-bearing for authoritative tags: the only plausibly-authoritative set is rootSpanTags (process identity), but those are _dd./runtime-namespaced tags users don't set on a builder → real collision ≈ 0. coreTags (header tags) colliding with explicit instrumentation tags is exactly the case where the builder tag should win.

Conclusion: documented long-standing wart, safe to correct, low blast radius.

The change

Off-by-default flag trace.builder.tags.precedence.enabled. When enabled, the ledger is applied last so explicit builder tags win:

context.setAllTags(mergedTracerTags, mergedTracerTagsNeedsIntercept); // floor, unchanged
if (BUILDER_TAGS_PRECEDENCE) {
  context.setAllTags(coreTags, coreTagsNeedsIntercept);
  context.setAllTags(rootSpanTags, rootSpanTagsNeedsIntercept);
  context.setAllTags(contextualTags);
  context.setAllTags(tagLedger);      // ← builder tags win
} else { /* historical order */ }

BUILDER_TAGS_PRECEDENCE is a constant-folded static final (mirroring SPAN_BUILDER_REUSE_ENABLED right above it), so the JIT dead-code-eliminates the unused ordering branch on the hot span-build path — the flag is process-constant, so there's no per-span branch cost.

"Does any existing test assert the old order?"

Ran the span-builder + interceptor suites with the logical order forced on:

• CoreSpanBuilderTest — passes
• all taginterceptor.* — pass
• the new BuilderTagsPrecedenceTest (pins the default order) — flips, confirming the flag works + propagates

→ nothing in those suites asserts the historical order. (The full dd-trace-core suite isn't a clean instrument locally — writer/intake/serialization/testcontainer tests fail for env reasons regardless of this flag — so the full-suite check is deferred to CI.)

Proposed rollout (the point of the flag)

flag default-off → opt-in canary → telemetry on the real collision rate (root spans only) → flip default → remove the historical path + flag. The collision-rate counter is not yet built — intentionally deferred until the concept lands.

Testing

• New BuilderTagsPrecedenceTest characterizes the default (off) order (green).
• Forked on-path test (property set at JVM launch, since the flag is a class-load static final) is a TODO once the concept is agreed.

---

tag: ai generated · default-off, no behavior change unless explicitly enabled.

🤖 Generated with Claude Code

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.87 s	14.72 s	[-0.0%; +2.0%] (no difference)
startup:insecure-bank:tracing:Agent	13.68 s	13.63 s	[-0.4%; +1.2%] (no difference)
startup:petclinic:appsec:Agent	16.87 s	16.48 s	[+1.5%; +3.2%] (significantly worse)
startup:petclinic:iast:Agent	16.73 s	16.43 s	[-2.6%; +6.3%] (no difference)
startup:petclinic:profiling:Agent	16.86 s	16.82 s	[-0.8%; +1.2%] (no difference)
startup:petclinic:sca:Agent	16.82 s	16.30 s	[-1.2%; +7.6%] (no difference)
startup:petclinic:tracing:Agent	16.18 s	16.16 s	[-1.1%; +1.4%] (no difference)

Commit: 61bfa55f · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.