Neura Market
Tool Definition Quality Score (TDQS) logo

Tool Definition Quality Score (TDQS)

Free

Open framework for scoring MCP tool definitions for AI agents

FreeFree tier
Type
Open Source
Visit Website GitHub (40 stars)

About Tool Definition Quality Score (TDQS)

TDQS (Tool Definition Quality Score) is an open framework for scoring how well an MCP tool definition communicates to an AI agent. Developed by Glama, it is used to score every tool of every MCP server and hosted connector in its registry. The framework is a complete specification including a rubric, exact prompts, aggregation formulas, and operational practices for running at registry scale. It scores tool definitions based on a four-stage pipeline: context signals, hard gates, LLM rubric evaluation, and deterministic post-processing. The goal is to provide an explainable, reproducible, and cheap quality signal for ranking and recommending MCP servers. TDQS is motivated by empirical studies showing that 97% of tool descriptions contain defects and that well-written descriptions significantly improve tool selection and task success.

Key Features

Four-stage scoring pipeline: context signals, hard gates, LLM rubric evaluation, deterministic post-processing
Produces per-dimension justifications for every score
Based on empirical studies of MCP tool description quality
Includes a complete rubric and prompts for LLM evaluation
Cheap enough to run on every schema change
Improvement guidelines provided as a checklist
Supports caching, diffing, and auditing of scores

Pros & Cons

Pros
  • Explainable scores with per-dimension breakdowns that maintainers can act on
  • Reproducible evaluation process ensures consistent results
  • Cheap to run on every schema change, enabling frequent updates
  • Based on empirical research on tool description defects
  • Includes clear rubric and improvement checklist for developers
Cons
  • Only evaluates tool definitions, not actual tool behavior or functionality
  • Relies on LLM evaluation which may introduce variability

Best For

Evaluating MCP server tool definitions for qualityImproving tool description quality to increase AI agent selection ratesRanking and recommending MCP servers in a registryAuditing tool definitions for defects and smells

FAQ

What is TDQS?
TDQS (Tool Definition Quality Score) is an open framework developed by Glama for scoring how well an MCP tool definition communicates to an AI agent.
How is the TDQS score computed?
The score is computed through a four-stage pipeline: context signals, hard gates, LLM rubric evaluation, and deterministic post-processing. The overall score combines tool definition quality (70%) and server coherence (30%).
What inputs does TDQS use?
TDQS scores exactly what an MCP client sees from tools/list: name, title, description, inputSchema, outputSchema, annotations, and sibling tool names.
Is TDQS free and open source?
Yes, TDQS is an open-source framework hosted on GitHub under the glama-ai organization, freely available for use and contribution.