Skip to content

git: clarify when NOT to call status/diff/log — cuts agent extra-call rate (measured +21.7 pts)#4520

Open
sujugithub wants to merge 1 commit into
modelcontextprotocol:mainfrom
sujugithub:hitrate/git-descriptions
Open

git: clarify when NOT to call status/diff/log — cuts agent extra-call rate (measured +21.7 pts)#4520
sujugithub wants to merge 1 commit into
modelcontextprotocol:mainfrom
sujugithub:hitrate/git-descriptions

Conversation

@sujugithub

Copy link
Copy Markdown

What

Rewrites 7 tool description strings in src/git/src/mcp_server_git/server.py. No code or behavior changes — only the text agents read when choosing tools.

Why

I measured how a mid-tier agent actually uses this server (12 realistic scenarios × 5 runs each, live server, every tool call recorded). Tool selection is already excellent — 100% hit rate, the diff-trio naming works. The failure mode is ritual verification calls: after git_add, agents call git_status or git_diff_staged to check it worked; before git_show HEAD, they call git_log to find a hash they don't need. A quarter of all runs were padded this way.

Measured effect of this diff (nothing else changed)

metric before after
tool hit rate 100% 100%
argument correctness 100% 100%
strict success (right tool + args + no extra calls) 75.0% 96.7%

Method: agent = gpt-oss-120b (deliberately mid-tier — weaker agents are hurt most by underspecified descriptions), N=5 runs per scenario, descriptions applied via in-memory override so before/after used the identical server build. Happy to share the full per-run data and scenario suite, or re-run against any revision of the wording.

Notes

  • The pattern in each rewrite: state that the operation is complete on success ("does not require calling git_status to verify"), and say which sibling tool covers the adjacent intent.
  • One deviation from the measured text: the tested git_show wording implied revision was optional; since GitShow.revision is required, this PR says Pass revision=\"HEAD\" instead — same intent, schema-accurate.
  • The measurement harness (scenarios + runner) is being open-sourced; I'll link it here once public.

🤖 Generated with Claude Code

Measured with an agent harness (12 scenarios x 5 runs each, live server):
tool selection was already perfect (100% hit rate) but agents padded runs
with ritual git_status/git_diff_staged/git_log verification calls. With only
these description changes: strict success (right tool + right args + no
extra calls) 75.0% -> 96.7%. No behavior changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants