Tool Selection Quality: Is Your Agent Calling the Right Tool?

What is tool selection quality in an AI agent?

Tool selection quality is whether an agent picked the right tool for a step and called it with the right arguments, judged from the transcript of what it did. It sits next to outcome grading, which only checks the final state the agent left behind. You need both, but not everywhere: grade the tool call where the choice itself carries risk, and grade the outcome everywhere else.

That last sentence is the whole post. The rest is how we decide which is which when we build the eval harness around an agent we are about to ship.

Why is this suddenly a metric people sell?

Because agents stopped being chatbots. Once a model can call tools, it can do real things: look up an order, send an email, write a row, issue a refund. The answer it gives you can be right while the path it took was wrong, and the path is where the damage happens.

Eval platforms noticed early. Galileo's agentic evaluations launch in January 2025 listed tool selection quality as a metric alongside errors in individual tool calls and overall session success, and the release said the feature was available to all Galileo users (Galileo press release, PR Newswire, January 23, 2025, read October 23, 2026). That is a reasonable way to package it if you are buying a platform.

The more useful source is Anthropic's engineering guide, "Demystifying evals for AI agents," published January 9, 2026 and read on October 23, 2026. It lists "tool calls verification (tools used, parameters)" as one kind of code-based grader. It also says something that sounds like it contradicts that: it is "often better to grade what the agent produced, not the path it took," and checking that an agent followed a specific sequence of tool calls proved "too rigid and results in overly brittle tests."

Both are true. The trick is knowing when each one applies.

When should you grade the tool call instead of the outcome?

Start from the default: grade the outcome. If the job is "answer the customer's question about their delivery window," you care that the answer is correct and grounded. Whether the agent looked up the order first and the shipment second, or the other way round, does not matter. Pinning the order of calls in a test just means the test breaks the next time a model update finds a shorter path. That is the brittleness Anthropic warns about: a suite can go red after a model upgrade even when every answer got better.

Then carve out the exceptions. We grade the call itself when the path is where the risk lives:

• Side effects. Any tool that writes, sends, deletes, books or pays. If the agent reached the right answer through a tool that also updated a record it should not have touched, the run fails, no matter how good the reply reads.
• Limits on arguments. A refund, a discount, a credit. Anthropic's own example for a support agent checks for a refund call with an amount constraint, written as process_refund with an amount of at most 100. The outcome ("customer is happy") can look perfect while the argument blew through policy.
• Data access. A tool that can read another customer's data, or a broader slice of data than the task needs. Reaching for it at all is the bug.
• Cost. A tool that calls an expensive downstream API or kicks off a long job. An agent that calls it three times to answer one question is a bill problem before it is a quality problem.

Everywhere else, let the agent find its own way and judge what it produced.

What are the ways an agent picks the wrong tool?

When we read transcripts from a failing run, the tool problems fall into four buckets. Naming them makes the eval easier to write, because each one needs a different check.

Wrong tool. The agent had a search_orders tool and a search_customers tool and used the second to answer a question about the first. Sometimes it even gets lucky and the answer comes out right. The check is a required tool for that kind of task, or a forbidden one.

Right tool, wrong arguments. It called the correct lookup with a date range that was off by a month, or passed the customer's name where the tool wanted an ID. These are the quiet ones, because the call succeeds and returns something plausible. The check is an argument constraint: this field must match, that value must stay inside a range.

A needless call. The agent already had the answer in context and called a tool anyway, or called the same tool twice with the same arguments. Harmless once, expensive at volume, and a sign the tool descriptions are confusing it. The check is a ceiling on calls for that tool in that task.

A missing call. The opposite failure, and the most dangerous for a business. The agent answered from what it thinks it knows instead of looking it up. The reply reads confidently and is out of date. The check is a required tool call before any answer about live data.

None of these needs a fancy platform to detect. They all come straight out of the transcript.

Why do more tools make selection worse?

Every tool you hand an agent goes into its context as a name, a description and a schema. Add enough of them and two things happen: the descriptions start to overlap, and the model has more to read before it decides. Overlap is the bigger problem in practice. If get_order, lookup_order and order_status all exist because three teams each wrapped the same API, the model has to guess, and so would a new hire.

We wrote about what too many tools do to an agent's context in more depth. The short version for tool selection: a selection failure is often a tool design failure. Before you write an eval for it, check whether a person reading only the tool names and descriptions would pick correctly. If they would not, fix the tools first. The eval will pass more often for the right reason.

How do you build a tool selection eval without making it brittle?

The shape we use is simple. We teach it here because the method is the useful part; the eval sets themselves stay with the client.

• Assert what must and must not happen, not the order. Required tools, forbidden tools and argument constraints. Leave the sequence free unless the order is itself the rule (for example, verify identity before changing an address).
• Allow equivalent paths. If two tools can legitimately answer the question, the eval accepts either.
• Grade the outcome too. A tool check that passes on a wrong final answer is not a pass. We run a code-based tool check and an outcome grader on the same trial.
• Run more than one trial. Model outputs vary between runs, and Anthropic's guide says as much and runs multiple trials for that reason. A tool choice that is right four times out of five is a real finding, not noise to average away.
• Keep the tool checks narrow. Put them on the tasks where the path carries risk, as above. Ten precise checks on the refund flow beat a hundred sequence checks across every task.

Where this runs matters as much as what it checks. The side-effect checks are the ones we let block a release; the rest warn. We covered which evals should block a merge separately, and it applies directly here.

This is one piece of our agent orchestration methodology: figure out where the agent can do damage, build the guard there, and ship with the evals running from the first week, not added after the first incident.

FAQ: tool selection quality

Is tool selection quality the same as task success? No. Task success asks whether the final state is right. Tool selection asks whether the agent got there through the right calls. An agent can succeed with a bad path and fail with a good one.

Do I need an eval platform to measure it? Not to start. Tool calls are in the transcript, and a code-based check against required tools, forbidden tools and argument limits is a few lines of test code. Platforms help with dashboards and volume.

Should an LLM judge grade tool choice? Usually not for the risky calls. Whether a refund stayed under a limit is a deterministic check, so check it in code. A model judge is better saved for questions with no exact answer, like whether the reply was helpful.

Does this change which model or API we should use? It can. Some teams find they need tighter control over tools than a hosted product allows, which is part of the decision between the Claude API or a hosted tool.

Where should we start? List every tool your agent can call that has a side effect. Write one required-or-forbidden check for each. That alone catches the failures that cost money.

Putting an agent in front of customers?

If you are about to give an agent tools that touch real customers or real money, the evals around those tools are the part worth getting right first. We design and build that as custom software for AI workflows. Tell us what your agent needs to do and we will tell you where we would put the guards.