Test MCP tool selection with a multi-model Promptfoo eval suite

Decision

Test how models use our MCP tools with Promptfoo, configured in YAML and kept in the repository beside the server. A custom provider runs the real loop: it connects to a running MCP server, lists its tools, hands them to the model and executes every tool call the model makes. A simulated user carries each scenario over several turns. Deterministic assertions check the tool-call trace, and an LLM judge scores the outcome against a rubric. The matrix covers models from Anthropic and OpenAI, including small ones.

Keep the eval's system prompt short and generic. It tells the model to act through tools and names no tool. When a scenario fails, fix the tool's name, description or parameters, because that is what the model reads when it chooses.

Run the suite before changing a tool description or the server's instructions. It runs by hand against a local, preview or production server, and CI does not run it.

Why

An MCP client picks a tool from its name, its description and its parameter schema. Those are the user interface for a language model: an ambiguous description or a required parameter the user never mentioned sends the model to the wrong tool. Our server is fixed and the clients vary, much like browser compatibility, so we need a matrix of which models handle our tools.

Early on, low pass rates prompted a rewrite of the eval's system prompt into a table that mapped trigger phrases to tool names. Pass rates jumped, and the suite stopped telling us anything about real input, because real clients never see that prompt. Once we fixed the tool descriptions instead, small models reached comparable pass rates with no table. That is why failures get fixed in the descriptions.

Rejected alternatives

• Langfuse. We already trace production with OpenTelemetry and Tempo, so it would duplicate that. Promptfoo runs locally, compares runs and lives in git.
• A system prompt tuned to the scenarios. It raises pass rates by overfitting and teaches nothing that holds for clients we don't control.

Consequences

The evals shape the API: parameters become optional and descriptions change wording because a model misread them. Each full run costs money in model calls, including the judge's. The suite needs a live server and a bearer token for an eval user. A regression shows up only when someone runs it.