Skip to content

Python: fix Anthropic streaming double-counting token usage#7162

Open
he-yufeng wants to merge 1 commit into
microsoft:mainfrom
he-yufeng:fix/anthropic-streaming-usage-double-count
Open

Python: fix Anthropic streaming double-counting token usage#7162
he-yufeng wants to merge 1 commit into
microsoft:mainfrom
he-yufeng:fix/anthropic-streaming-usage-double-count

Conversation

@he-yufeng

Copy link
Copy Markdown
Contributor

Motivation & Context

On the Anthropic streaming path, usage tokens are double-counted. _process_stream_event emits a usage Content on both message_start and message_delta, and ChatResponse.from_updates sums every usage Content it sees. But Anthropic's message_delta usage is a cumulative total for the message, not a per-delta increment (from their streaming docs: "The token counts shown in the usage field of the message_delta event are cumulative"). Summing the message_start seed onto the cumulative message_delta total inflates usage_details: output_token_count lands at 26 when the API reported 25 (the seed is output_tokens: 1), and for server-tool turns that also report cumulative input on message_delta, the prompt tokens get double-counted too.

The streaming total should match what the non-streaming path produces.

Description & Review Guide

  • What are the major changes? Anthropic streams cumulative usage snapshots, but from_updates expects each update's usage to be an increment it can sum. _process_stream_event now takes a per-stream emitted_usage accumulator and emits the increment of each snapshot over what has already been emitted, so the summation reconstructs the final cumulative usage instead of inflating it. The accumulator lives in the per-request _stream() generator rather than on the client, so concurrent streams don't interfere, and when no accumulator is passed (a direct single-event call) the snapshot is returned unchanged, so existing callers and tests are untouched.
  • What is the impact of these changes? response.usage_details after a streaming Anthropic call now equals the cumulative totals the API reported, matching the non-streaming path. This fixes both the unconditional output-seed inflation and the server-tool input double-count.
  • What do you want reviewers to focus on? The reconciliation in _incremental_usage, and that message_start's input-side counts are preserved when message_delta carries only output_tokens (the basic case).

Two regression tests cover the basic case (output 25 / input 10, not 26) and the server-tool case where message_delta also reports cumulative input (output 25 / input 12, not doubled).

Related Issue

Fixes #7143

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible

Copilot AI review requested due to automatic review settings July 17, 2026 03:54
@giles17 giles17 added the python Usage: [Issues, PRs], Target: Python label Jul 17, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes incorrect token accounting for Anthropic streaming responses in the Python Anthropic package by converting Anthropic’s cumulative usage snapshots into incremental usage updates that can be safely summed by ChatResponse.from_updates, aligning streaming totals with the non-streaming path.

Changes:

  • Thread a per-stream UsageDetails accumulator through the streaming generator to emit usage increments instead of cumulative snapshots.
  • Update _process_stream_event to optionally accept the per-stream accumulator and apply _incremental_usage to both message_start and message_delta usage.
  • Add regression tests covering both the basic cumulative-output case and the server-tool cumulative-input case.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
python/packages/anthropic/agent_framework_anthropic/_chat_client.py Adds per-stream usage reconciliation (_incremental_usage) and threads an accumulator through streaming event processing to prevent double-counting.
python/packages/anthropic/tests/test_anthropic_client.py Adds regression tests ensuring streaming usage totals match Anthropic-reported cumulative totals for both output-only and input+output delta usage.

Comment on lines +1188 to +1195
emitted_counts = cast("dict[str, int]", emitted)
delta: dict[str, int] = {}
for key, value in cast("dict[str, int | None]", cumulative).items():
if value is None:
continue
delta[key] = value - emitted_counts.get(key, 0)
emitted_counts[key] = value
return cast("UsageDetails", delta)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python: [Bug]: Anthropic streaming double-counts token usage (message_start usage is summed with the cumulative message_delta usage)

3 participants