toolbloat 0.2.0__tar.gz

toolbloat-0.2.0/.gitignore

@@ -0,0 +1,15 @@
+__pycache__/
+*.py[cod]
+.venv/
+dist/
+build/
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+.env
+.env.*
+!.env.example
+.toolbloat/
+toolbloat-report/
+.DS_Store
+toolbloat-demo/

toolbloat-0.2.0/CHANGELOG.md

@@ -0,0 +1,48 @@
+# Changelog
+
+All notable changes to this project are documented here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses
+[Semantic Versioning](https://semver.org/).
+
+## [0.2.0] - 2026-11-04
+
+### Added
+- GitHub Action (`uses: diegoperezg7/toolbloat@v0.2.0`): counts the tokens of the MCP tool
+  definitions in a config, writes the table to the job summary and fails over `max-tokens`.
+  No model calls, no keys.
+- `toolbloat tokens --max-tokens N`: exit code 2 when the total is over the budget, 1 when a server
+  could not be measured.
+- `toolbloat tokens --markdown FILE`: append a Markdown summary (for `$GITHUB_STEP_SUMMARY`).
+
+## [0.1.1] - 2026-11-03
+
+### Changed
+- The privacy prompt before sending anything to a cloud provider now lists every server that will be
+  sent, the config file it came from (or `--server`), its command and the provider, and refuses
+  without `--yes` when there is no interactive terminal. `toolbloat intents` asks too.
+- The HTML report formats numbers, percentages and amounts for the active language
+  (English `6,064` / `$2.73`, Spanish `6.064` / `2,73 US$`).
+
+## [0.1.0] - 2026-11-03
+
+### Added
+- `toolbloat tokens`: start MCP servers over stdio (official MCP Python SDK), list their tools and
+  count definition tokens per tool, per server and in total (tiktoken o200k, approximate).
+- `toolbloat intents`: write one or more realistic requests per tool with a model, cached in an
+  editable YAML file.
+- `toolbloat run`: ask each model which tool it would call, with only the tool's own server and with
+  the whole stack. Reports selection accuracy, argument validity against the JSON Schema, a
+  cross-server confusion matrix, tools no model chose, tokens saved by removing them and a monthly
+  token and cost projection. Tools are never executed.
+- `toolbloat report`: rebuild the HTML report from `results.json`.
+- `toolbloat demo`: a recorded real run on six MCP reference servers, no keys needed.
+- Before sending tool definitions to a cloud provider, lists the servers and the provider and asks
+  for confirmation (`--yes` to skip, required without a terminal).
+- Reads `.mcp.json`, Cursor and Claude Desktop configs (command and args only, `env` is never shown).
+- Providers: Ollama, Mistral, OpenRouter, OpenAI through one OpenAI-compatible client, with
+  `--max-cost`, backoff on rate limits and a resumable call cache.
+- Static HTML report in English and Spanish, light and dark, mobile friendly.
+
+[0.2.0]: https://github.com/diegoperezg7/toolbloat/releases/tag/v0.2.0
+[0.1.1]: https://github.com/diegoperezg7/toolbloat/releases/tag/v0.1.1
+[0.1.0]: https://github.com/diegoperezg7/toolbloat/releases/tag/v0.1.0

toolbloat-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Diego Pérez
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

toolbloat-0.2.0/NOTICE

@@ -0,0 +1,232 @@
+toolbloat
+Copyright (c) 2026 Diego Pérez. Licensed under the MIT License (see LICENSE).
+
+This repository and the toolbloat wheel include third-party material:
+
+  - src/toolbloat/data/demo/results.json, docs/index.html and the screenshots/GIF in docs/assets
+    contain tool names, descriptions and JSON Schemas returned by the MCP reference servers
+    (@modelcontextprotocol/server-filesystem 2.1.0, server-memory 2.0.1, server-everything 2.1.0,
+    server-sequential-thinking 2.0.0, mcp-server-time 2026.10.28, mcp-server-fetch 2026.10.28),
+    from https://github.com/modelcontextprotocol/servers
+  - examples/reference-stack.mcp.json names those packages.
+
+No code from those servers is included. Their license, reproduced verbatim from
+https://github.com/modelcontextprotocol/servers/blob/main/LICENSE (retrieved 2026-11-03):
+
+--------------------------------------------------------------------------------
+The MCP project is undergoing a licensing transition from the MIT License to the Apache License, Version 2.0 ("Apache-2.0"). All new code and specification contributions to the project are licensed under Apache-2.0. Documentation contributions (excluding specifications) are licensed under CC-BY-4.0.
+
+Contributions for which relicensing consent has been obtained are licensed under Apache-2.0. Contributions made by authors who originally licensed their work under the MIT License and who have not yet granted explicit permission to relicense remain licensed under the MIT License.
+
+No rights beyond those granted by the applicable original license are conveyed for such contributions.
+
+---
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to the Licensor for inclusion in the Work by the copyright
+      owner or by an individual or Legal Entity authorized to submit on behalf
+      of the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+---
+
+MIT License
+
+Copyright (c) 2024-2025 Model Context Protocol a Series of LF Projects, LLC.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+---
+
+Creative Commons Attribution 4.0 International (CC-BY-4.0)
+
+Documentation in this project (excluding specifications) is licensed under
+CC-BY-4.0. See https://creativecommons.org/licenses/by/4.0/legalcode for
+the full license text.

toolbloat-0.2.0/PKG-INFO

@@ -0,0 +1,296 @@
+Metadata-Version: 2.5
+Name: toolbloat
+Version: 0.2.0
+Summary: See what your MCP servers cost in tokens on every turn, and whether stacking them makes your model pick the wrong tool.
+Project-URL: Homepage, https://github.com/diegoperezg7/toolbloat
+Project-URL: Report, https://diegoperezg7.github.io/toolbloat/
+Project-URL: Issues, https://github.com/diegoperezg7/toolbloat/issues
+Author: Diego Pérez
+License-Expression: MIT
+License-File: LICENSE
+License-File: NOTICE
+Keywords: agents,evaluation,llm,mcp,model-context-protocol,tokens,tool-calling
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Requires-Dist: jinja2==3.1.6
+Requires-Dist: jsonschema==4.26.0
+Requires-Dist: mcp==2.3.0
+Requires-Dist: openai==3.42.1
+Requires-Dist: pyyaml==6.0.3
+Requires-Dist: rich==15.0.0
+Requires-Dist: tiktoken==0.14.0
+Requires-Dist: typer==0.27.3
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="docs/assets/banner-dark.svg">
+    <img alt="toolbloat" src="docs/assets/banner-light.svg" width="640">
+  </picture>
+</p>
+
+<p align="center">
+  <a href="https://github.com/diegoperezg7/toolbloat/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/diegoperezg7/toolbloat/actions/workflows/ci.yml/badge.svg"></a>
+  <a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-8f3bb3.svg"></a>
+  <a href="https://github.com/diegoperezg7/toolbloat/releases"><img alt="Release" src="https://img.shields.io/github/v/release/diegoperezg7/toolbloat?color=8f3bb3"></a>
+  <a href="https://github.com/diegoperezg7/toolbloat/stargazers"><img alt="Stars" src="https://img.shields.io/github/stars/diegoperezg7/toolbloat?style=flat&color=8f3bb3"></a>
+</p>
+
+<p align="center"><b>See what your MCP servers cost in tokens on every turn, and whether stacking them makes your model pick the wrong tool.</b></p>
+
+<p align="center"><a href="README.es.md">Leer en español</a> · <a href="https://diegoperezg7.github.io/toolbloat/">Example report</a></p>
+
+<p align="center"><img alt="toolbloat demo in a terminal" src="docs/assets/demo.gif" width="760"></p>
+
+## Why
+
+Every MCP server you connect sends its tool definitions to the model on every turn. Tools exist to
+see that cost: Claude Code shows MCP tool tokens in [`/context`](https://code.claude.com/docs/en/slash-commands),
+[mcp-bloat](https://github.com/pranayasharma/mcp-bloat) audits them per server, and
+[mcp-compressor](https://github.com/atlassian-labs/mcp-compressor) or
+[squeez](https://github.com/claudioemmanuel/squeez) shrink them. Tools also exist to check one
+server's tool descriptions: [toolfit](https://github.com/sreshtalluri/toolfit) and
+[toolscore](https://github.com/Aryan-MP/toolscore) find which tools of *a* server confuse a model,
+and [LiveMCPBench](https://github.com/icip-cas/LiveMCPBench) is a fixed benchmark over a large toolset.
+
+None of them answers the question you have once you run five servers at once: **does stacking
+them make my model pick the wrong tool, how much does that depend on the model, and what am I
+paying per turn for it?** toolbloat measures exactly that, on your own stack:
+
+- tokens per tool, per server and in total, plus the prompt tokens each provider actually bills;
+- selection accuracy per model with **only the tool's own server** loaded vs. **the whole stack**;
+- which requests stacking breaks, and a **cross-server confusion matrix**;
+- whether the arguments the model proposes validate against the tool's JSON Schema;
+- tools no model chose, the tokens you would save by removing them, and a monthly projection.
+
+Tools are **never executed**: toolbloat only reads the tool call the model proposes.
+
+## Quickstart
+
+See a recorded real run, no keys and no setup (needs [uv](https://docs.astral.sh/uv/)):
+
+```bash
+uvx --from git+https://github.com/diegoperezg7/toolbloat@v0.2.0 toolbloat demo --open
+```
+
+or from a clone:
+
+```bash
+git clone https://github.com/diegoperezg7/toolbloat && cd toolbloat && uv run toolbloat demo --open
+```
+
+`uvx toolbloat` is coming to PyPI.
+
+Then measure your own stack:
+
+```bash
+# tokens only, no model calls: reads ./.mcp.json, Cursor and Claude Desktop configs
+toolbloat tokens
+
+# full run with a local model (nothing leaves your machine)
+toolbloat run -m ollama:qwen3:4b-16k
+
+# or compare several models (keys from env vars), capped at 1 USD
+toolbloat run -m ollama:qwen3:4b-16k -m mistral:ministral-8b-latest -m mistral:mistral-small-latest --max-cost 1
+```
+
+## Example output
+
+<!-- demo:start -->
+```text
+$ toolbloat demo
+Recorded run from 2026-11-03 · toolbloat 0.1.0 · Apple M4, 16 GB RAM, macOS 27.2
+This replays a real `toolbloat run`; nothing is executed now. Measure your own stack with `toolbloat
+run`.
+Tokens per server (o200k, approx.)     
+┏━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━┓
+┃ server     ┃ tools ┃ tokens ┃ share ┃
+┡━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━┩
+│ everything │    16 │  1,825 │   30% │
+│ filesystem │    14 │  1,806 │   30% │
+│ memory     │     9 │  1,045 │   17% │
+│ thinking   │     1 │    873 │   14% │
+│ time       │     2 │    281 │    5% │
+│ fetch      │     1 │    234 │    4% │
+├────────────┼───────┼────────┼───────┤
+│ total      │    43 │  6,064 │  100% │
+└────────────┴───────┴────────┴───────┘
+Every turn re-sends ~6,064 tokens of tool definitions: ~18.2M tokens/month at 100 turns/day (before 
+prompt caching).
+Tool selection accuracy (alone vs. whole stack)                                          
+┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
+┃ model                        ┃ alone ┃ stacked ┃  change ┃ broken ┃ args ok ┃ $/month ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
+│ mistral:ministral-8b-latest  │   86% │     72% │ -14 pts │      6 │    100% │   $2.73 │
+│ mistral:ministral-14b-latest │   91% │     86% │  -5 pts │      2 │    100% │   $3.64 │
+└──────────────────────────────┴───────┴─────────┴─────────┴────────┴─────────┴─────────┘
+broken = right alone, wrong with the whole stack · args ok = correct calls whose arguments match the
+JSON Schema
+Most common wrong picks with the whole stack                                          
+┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
+┃ expected                             ┃ model chose                         ┃ times ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
+│ memory/create_relations              │ memory/create_entities              │     2 │
+│ filesystem/write_file                │ filesystem/create_directory         │     2 │
+│ memory/delete_observations           │ memory/search_nodes                 │     2 │
+│ filesystem/edit_file                 │ filesystem/read_text_file           │     2 │
+│ filesystem/list_directory_with_sizes │ filesystem/list_allowed_directories │     2 │
+│ filesystem/read_file                 │ filesystem/list_allowed_directories │     1 │
+│ filesystem/read_text_file            │ filesystem/list_allowed_directories │     1 │
+│ filesystem/read_media_file           │ filesystem/search_files             │     1 │
+└──────────────────────────────────────┴─────────────────────────────────────┴───────┘
+5 tool(s) were never chosen with the whole stack: filesystem/edit_file, memory/create_relations, 
+filesystem/list_directory_with_sizes, memory/delete_observations, filesystem/write_file. Removing 
+them saves ~690 tokens per turn (~2.1M/month).
+API cost of the calls in this report: $0.1447
+```
+<!-- demo:end -->
+
+The full report for this run is at **<https://diegoperezg7.github.io/toolbloat/>**.
+
+<p align="center">
+  <img alt="toolbloat HTML report, light" src="docs/assets/report-light.png" width="49%">
+  <img alt="toolbloat HTML report, dark" src="docs/assets/report-dark.png" width="49%">
+</p>
+
+## Use it in CI
+
+A GitHub Action keeps the token cost of your MCP stack, or of the MCP server you maintain, from
+growing unnoticed. It starts the stdio servers in your config, counts their tool definitions, writes
+the table to the job summary and fails the job when the total goes over `max-tokens`. It never calls
+a model and needs no keys.
+
+```yaml
+# .github/workflows/mcp-tokens.yml
+on: [pull_request]
+jobs:
+  tokens:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: diegoperezg7/toolbloat@v0.2.0
+        with:
+          config: .mcp.json        # space-separated list of config files
+          max-tokens: "8000"       # leave empty to only report
+```
+
+On the six reference servers of the demo the summary reads 6,064 tokens, so a budget of 5,000
+fails by 1,064. The same check runs locally:
+
+```bash
+toolbloat tokens --config .mcp.json --max-tokens 8000   # exit 2 if over, 1 if a server cannot be measured
+```
+
+The action runs the server commands from the config in the repository it checks out. On pull
+requests from forks GitHub gives the job no secrets, but those commands still run: review config
+changes as you would review code.
+
+## How it works
+
+```mermaid
+flowchart LR
+    A[".mcp.json / Cursor /<br>Claude Desktop / --server"] --> B["start each server<br>over stdio"]
+    B --> C["tools/list<br>(never tools/call)"]
+    C --> D["count tokens<br>per tool and server"]
+    C --> E["intents.yaml<br>1+ request per tool<br>(LLM-written, editable)"]
+    E --> F{"each model, each request"}
+    F -->|"only the tool's server"| G["alone"]
+    F -->|"every server"| H["stacked"]
+    G --> I["accuracy, broken by stacking,<br>confusion matrix, valid args,<br>never-chosen tools, $/month"]
+    H --> I
+    D --> I
+    I --> J["terminal summary +<br>static HTML report + results.json"]
+```
+
+1. **Discover.** Each server is started with the official MCP Python SDK (`mcp==2.3.0`) and asked
+   for `tools/list`. Only `command` and `args` are read from your configs for display; `env` is
+   passed to the server but never printed or saved.
+2. **Count.** Each tool is serialised as an OpenAI-style function (name, description, JSON Schema)
+   and counted with tiktoken `o200k_base`. Providers wrap tools in their own templates and
+   tokenizers, so this is **approximate**; the report also shows the prompt tokens each provider billed.
+3. **Write intents.** A model writes realistic requests for each tool (`.toolbloat/intents.yaml`).
+   They are cached; edit them, add `also_ok` for tools that are equally right, or `skip: true`.
+4. **Ask.** For every request, each model gets the same system prompt ("call exactly one tool") at
+   temperature 0, once with only the right server's tools and once with every tool. Answers are
+   cached in `toolbloat-report/calls.jsonl`, so interrupted runs resume for free.
+5. **Report.** `results.json`, a Rich summary in the terminal and a single-file HTML report.
+
+## Configuration
+
+| Option | Default | What it does |
+| --- | --- | --- |
+| `-m, --model provider:model` | `ollama:qwen3:4b` | Repeatable. Providers: `ollama`, `mistral`, `openrouter`, `openai`. |
+| `-c, --config PATH` | auto | `.mcp.json`, `.cursor/mcp.json`, `~/.cursor/mcp.json`, Claude Desktop config. Repeatable. |
+| `-s, --server "name=cmd args"` | – | Add a server by hand. Repeatable. |
+| `--intents PATH` | `.toolbloat/intents.yaml` | Cached, editable requests. |
+| `--intent-model` | first `--model` | Model that writes missing intents. |
+| `--intents-per-tool N` | `1` | More requests per tool = steadier numbers, more calls. |
+| `--max-cost USD` | `1.0` | Checked before every call; the run stops with partial results. Local models cost 0. |
+| `--price provider:model=IN,OUT` | – | USD per million tokens for a model without a built-in price. |
+| `--turns-per-day N` | `100` | For the monthly projection. |
+| `--out DIR` | `toolbloat-report` | `results.json`, `report.html`, `calls.jsonl`. |
+
+Keys come only from environment variables: `MISTRAL_API_KEY`, `OPENROUTER_API_KEY`,
+`OPENAI_API_KEY`. `OLLAMA_HOST` points to another Ollama. See [.env.example](.env.example).
+
+**Ollama context window.** Ollama loads models with a small context window by default (4,096 tokens
+for `qwen3:4b` here). A stack of a few servers is larger than that and gets cut silently;
+toolbloat warns when a model bills fewer prompt tokens than the definitions. Create a variant with a
+bigger window once (`num_ctx` 16384) and use it:
+
+```bash
+printf 'FROM qwen3:4b\nPARAMETER num_ctx 16384\n' > Modelfile && ollama create qwen3:4b-16k -f Modelfile
+```
+
+Tested with a partial run of `ollama:qwen3:4b-16k` (32 real calls on the reference stack; the largest prompt was 5,193
+tokens, well under the 16,384 window). It is not in the recorded demo: a local 4B model with thinking took several
+minutes per call on a shared 16 GB laptop.
+
+**Privacy.** With a cloud provider, your tool definitions and the test requests are sent to it.
+Before anything is sent, toolbloat lists the servers whose definitions will go out and the provider
+they go to, and asks for confirmation; without a terminal (CI) it refuses unless you pass `--yes`.
+With Ollama nothing leaves your machine. There is no telemetry. Running toolbloat starts the MCP
+servers in your config, which is third-party code you already run with your agent: see
+[SECURITY.md](SECURITY.md).
+
+## The demo data
+
+`toolbloat demo` replays a real `toolbloat run` from 2026-11-03 on six servers from the
+[MCP reference servers](https://github.com/modelcontextprotocol/servers), which start without
+credentials: `server-filesystem`, `server-memory`, `server-everything`, `server-sequential-thinking`
+(npm) and `mcp-server-time`, `mcp-server-fetch` (PyPI). They are published under MIT, with the
+project moving new contributions to Apache-2.0 ([LICENSE](https://github.com/modelcontextprotocol/servers/blob/main/LICENSE)).
+toolbloat does not bundle their code, only the tool definitions they returned.
+
+Models in the recording: `mistral:ministral-8b-latest` and `mistral:ministral-14b-latest`, each on all 43 requests in both conditions (172 calls, 0.14 USD in total at list price). Requests were written by `ministral-8b-latest` and reviewed by hand. Third-party notices for the bundled tool definitions are in [NOTICE](NOTICE). The config is in
+[`examples/reference-stack.mcp.json`](examples/reference-stack.mcp.json) and the reviewed intents in
+[`examples/reference-stack.intents.yaml`](examples/reference-stack.intents.yaml).
+
+## Roadmap
+
+- Execute tools in a sandbox to check results, not just the choice.
+- Suggest description rewrites for the tools that get confused.
+- HTTP and SSE transports.
+- Compare a stack with and without a compressor such as mcp-compressor.
+
+## Part of a series
+
+Open source tools that evaluate agents and LLMs on your own data:
+
+- [memarena](https://github.com/diegoperezg7/memarena): Benchmark memory layers on your own conversations.
+- [parsearena](https://github.com/diegoperezg7/parsearena): Compare PDF parsers side by side and their effect on retrieval.
+- [commitarena](https://github.com/diegoperezg7/commitarena): Replay real bug fixes from your git history against any model.
+- [rulelab](https://github.com/diegoperezg7/rulelab): Find which AGENTS.md rules earn their tokens.
+- [doesitfire](https://github.com/diegoperezg7/doesitfire): Test whether your SKILL.md activates in Claude Code, Codex and OpenCode.
+
+## License
+
+[MIT](LICENSE) © 2026 Diego Pérez