Skip to content

memory: disambiguate tool descriptions to reduce agent tool-misuse (measured +34.5 pts)#4519

Open
sujugithub wants to merge 1 commit into
modelcontextprotocol:mainfrom
sujugithub:hitrate/memory-descriptions
Open

memory: disambiguate tool descriptions to reduce agent tool-misuse (measured +34.5 pts)#4519
sujugithub wants to merge 1 commit into
modelcontextprotocol:mainfrom
sujugithub:hitrate/memory-descriptions

Conversation

@sujugithub

Copy link
Copy Markdown

What

Rewrites 6 tool description strings in src/memory/index.ts. No code or behavior changes — only the text agents read when choosing tools.

Why

Agents pick tools from names + descriptions + schemas alone. I measured how a mid-tier agent actually uses this server (11 realistic scenarios × 5 runs each, live server, every tool call recorded) and found two systematic failures:

  1. Wrong-tool confusion: create_entities vs add_observations, and search_nodes vs open_nodes — the current descriptions don't say when to prefer the sibling.
  2. Ritual extra calls: agents call open_nodes/search_nodes before writes and deletes that don't need a lookup first.

Measured effect of this diff (nothing else changed)

metric before after
tool hit rate (right tool chosen) 80.0% 100.0%
argument correctness 100% 100%
strict success (right tool + args + no extra calls) 61.8% 96.4%

Method: agent = gpt-oss-120b (a deliberately mid-tier model — weaker agents are hurt most by ambiguous descriptions), N=5 runs per scenario, descriptions applied via in-memory override so before/after used the identical server build. Happy to share the full per-run data and the scenario suite, or re-run against any revision of this text — treat the exact wording as a starting point.

Notes

  • Each rewritten description leads with when to use the tool and names the sibling to prefer otherwise ("Use add_observations instead to add facts to existing entities").
  • The measurement harness (scenarios + runner) is being open-sourced; I'll link it here once public.

🤖 Generated with Claude Code

Measured with an agent harness (11 scenarios x 5 runs each, live server):
agents confused create_entities/add_observations and search_nodes/open_nodes,
and padded tasks with unnecessary lookups. With only these description
changes: tool hit rate 80.0% -> 100.0%, strict success (right tool + right
args + no extra calls) 61.8% -> 96.4%. No behavior changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants