@tokcalc/mcp-server 0.1.1 → 0.1.3

package/README.md

@@ -1,86 +1,406 @@
-# @tokcalc/mcp-server
-
-**LLM serving capacity planner for AI agents.**
-
-Open-source MCP (Model Context Protocol) server that lets AI agents (Cursor, Claude Desktop, Cline) estimate LLM serving capacity — model fit, KV cache, throughput, latency, multi-GPU topology, and cost.
-
-## Tools
-
-| Tool | What it does |
-|---|---|
-| `estimate_capacity` | VRAM/KV/throughput/latency/cost for one config |
-| `compare_gpus` | Ranked GPU comparison for one workload |
-| `recommend_topology` | TP/CP topology recommendation |
-| `estimate_api_vs_self_host` | Break-even analysis |
-| `list_models` | Discover supported model IDs (35 models) |
-| `list_gpus` | Discover supported GPU IDs (30 GPUs) |
-
-All tools are **read-only** — no side effects, no cloud credentials, no deployments.
-
-## Install
-
-### Claude Desktop
-
-Add to `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) or `%APPDATA%\Claude\claude_desktop_config.json` (Windows):
-
-```json
-{
-  "mcpServers": {
-    "tokcalc": {
-      "command": "npx",
-      "args": ["-y", "@tokcalc/mcp-server"]
-    }
-  }
-}
-```
-
-Restart Claude Desktop. The `tokcalc` server will be available as an MCP tool source.
-
-### Cursor
-
-Add to `.cursor/mcp.json` in your project:
-
-```json
-{
-  "mcpServers": {
-    "tokcalc": {
-      "command": "npx",
-      "args": ["-y", "@tokcalc/mcp-server"]
-    }
-  }
-}
-```
-
-### Cline (VS Code)
-
-Add the same config to Cline's MCP settings.
-
-## Example prompts
-
-Ask your AI agent:
-
-> "I need to serve Llama 3.3 70B at 32K context for 50 concurrent users. What GPU topology do you recommend, and how much will it cost per month?"
-
-> "Compare H100 vs H200 for serving Qwen 2.5 72B in FP8 with continuous batching."
-
-> "At what daily request volume does self-hosting Llama 70B on H200 beat the GPT-4o API?"
-
-The agent calls `list_models` → `list_gpus` → `recommend_topology` → `estimate_capacity` and returns a structured plan with throughput ranges, latency, VRAM, cost, and confidence levels.
-
-## Supported models (35)
-
-Llama 3/3.1/3.3, Llama 4 Scout/Maverick, Mistral 7B, Mixtral 8x7B/8x22B, Mistral Large 3, Pixtral 12B, Codestral, Qwen 2/2.5/3 (incl. MoE + VL), DeepSeek V3/R1/Coder V2, Gemma 2, Phi-3/4, SmolLM2, Falcon 3, OLMo 2, BGE-M3, E5, GTE.
-
-## Supported GPUs (30)
-
-NVIDIA H100/H200/B200/B300, A100, L40S, L4, T4, V100, RTX 4090/3090/5090, RTX PRO 6000 Blackwell, AMD MI300X/MI325X, Intel Gaudi 3, Google TPU v5p/Trillium, Groq LPU, Cerebras CS-3, Apple M2/M3/M4 Ultra/Max.
-
-## License
-
-Apache 2.0 — same as the main tokcalc project.
-
-## Links
-
-- [Live calculator](https://tokcalc.vercel.app)
-- [GitHub](https://github.com/stevecrates489-commits/tokcalc)
-- [CONTRIBUTING](https://github.com/stevecrates489-commits/tokcalc/blob/main/CONTRIBUTING.md)
+<div align="center">
+
+# tokcalc
+
+### The open-source LLM serving capacity planner
+
+**Plan your LLM deployment before you rent the GPUs.**
+
+[![Live demo](https://img.shields.io/badge/live-demo-10b981?style=flat-square)](https://tokcalc.dev)
+[![License: Apache 2.0](https://img.shields.io/badge/license-Apache_2.0-blue?style=flat-square)](LICENSE)
+[![PRs welcome](https://img.shields.io/badge/PRs-welcome-10b981?style=flat-square)](CONTRIBUTING.md)
+[![Made with Next.js](https://img.shields.io/badge/made_with-Next.js_16-black?style=flat-square)](https://nextjs.org)
+
+</div>
+
+---
+
+<div align="center">
+
+![tokcalc demo](/public/tokcalc-demo.gif)
+
+*Switch model → multi-GPU → long-context capacity planner → Build vs Buy → Reference catalog*
+
+</div>
+
+---
+
+tokcalc turns your LLM traffic, context length, latency SLOs, cache behavior,
+and model choice into a **defensible serving topology and cost plan** —
+with transparent formulas and cited benchmarks.
+
+It's the engineering-grade pre-deployment decision layer for LLM inference.
+Not another static "tokens per second" calculator.
+
+## Try it now
+
+**[tokcalc.dev](https://tokcalc.dev)** — no signup, no tracking, no paywall.
+
+Pick a model, a GPU, and a workload. Get an instant capacity plan:
+
+- Generation speed (tok/s)
+- Time-to-first-token (TTFT) + inter-token latency (ITL)
+- VRAM budget with KV-cache sizing
+- Multi-GPU topology recommendation (Single GPU → TP×2/4/8 → Context Parallel)
+- Monthly cost + break-even vs API pricing
+- **Shareable URL** — your config encoded in the URL hash, send to colleagues
+
+## Why tokcalc?
+
+The market has dozens of "tokens per second" calculators and self-host-vs-API
+break-even tools (induwara.lk, gigagpu, kickllm, cloudparity, curlscape,
+profitable.ai). None of them are unified capacity planners.
+
+### What tokcalc answers that competitors can't
+
+> *"Can I serve Qwen 2.5 72B at 128K context on 2× H100 with 20 concurrent users?"*
+
+> *"How many H200s do I need for 1,000 req/min with P95 TTFT < 2s?"*
+
+> *"Does FP8 or AWQ save more money once quality, KV cache, and engine support are included?"*
+
+> *"At what daily volume does an H100 beat GPT-4o pricing?"*
+
+> *"What happens to cost and latency if an agent makes 8 model calls, has 3 tool calls, and its context grows by 5K tokens each turn?"*
+
+> *"Would prefix caching, continuous batching, or PD disaggregation save more for this workload?"*
+
+## Features
+
+### Calculator tab
+
+| Feature | What it computes |
+|---|---|
+| **Model fit / VRAM** | Will the model + KV cache fit in the GPU's memory? |
+| **Throughput** | Decode tok/s (per-stream) + aggregate (batched) + prefill tok/s |
+| **Latency split** | Time-to-first-token (= prefill) + inter-token latency (= decode) |
+| **Continuous batching** | User-tunable 1.0–4× multiplier (cited 1.5–4× SOSP range) |
+| **Reasoning tokens** | Hidden reasoning budget added to billed output (o1 / R1 / Claude thinking) |
+| **Prompt caching** | Self-hosted vLLM APC + Anthropic 5m/1h TTL + OpenAI 50%-off cached tokens |
+| **Speculative decoding** | User-tunable 1.2–4× boost factor |
+| **Multi-GPU TP** | 1× → 8× tensor parallel with NVLink efficiency factor |
+| **Long-context capacity** | KV memory + max concurrency + prefill time at 4K → 1M context |
+| **Topology recommendation** | Single GPU → TP×2 → TP×4 → TP×8 → TP×8 + Context Parallel (RingAttention) |
+| **Cost economics** | GPU $/hr → $/M output tokens → $/request → monthly cost |
+
+### Build vs Buy tab
+
+Independent calculator (separate state) that compares:
+- **Self-host**: model + GPU + quant + utilization + batch → $/M tokens + monthly cost
+- **API**: 13 providers (OpenAI / Anthropic / Gemini / Groq / DeepSeek / Mistral / Together)
+- **Verdict**: Self-host cheaper / API cheaper / Not enough volume — with break-even reqs/day
+
+### Reference tab
+
+5 sub-tables — fully transparent, every record source-linked where available:
+- **Models** (35 entries: Llama 4 Scout/Maverick, Qwen 3 family, DeepSeek V3/R1, Pixtral, BGE-M3, ...)
+- **GPUs** (30 entries: H100/H200/B200/B300, AMD MI300X/MI325X, Intel Gaudi 3, TPU v5p/Trillium, Groq LPU, Cerebras WSE-3, Apple M2/M3/M4 Ultra, ...)
+- **Quantization** (16 formats: FP16/BF16, GGUF Q2_K→Q8_0, GPTQ, AWQ, EXL2, FP8, NVFP4)
+- **API pricing** (13 models with input/cached/output + retired/current status)
+- **Cloud GPU pricing** (all GPUs with $/hr > 0 + typical providers)
+
+### The long-context capacity planner (the differentiator)
+
+This is the formula the Perplexity research brief called "the most important
+tokcalc should visibly expose":
+
+$$
+\text{KV bytes/request} = 2 \cdot L \cdot T \cdot H_{\text{kv}} \cdot D_h \cdot B
+$$
+
+For dense attention, prefill cost grows **superlinearly** with context length:
+
+$$
+\text{prefill FLOPs} = \underbrace{2 \cdot N \cdot T}_{\text{linear}} + \underbrace{T^2 \cdot H_{\text{kv}} \cdot D_h \cdot L}_{\text{attention}}
+$$
+
+tokcalc shows you:
+- Max concurrent users at 4K / 8K / 16K / 32K / 64K / 128K / 256K / 512K / 1M context
+- KV memory per request at each context length
+- Prefill time (with superlinear attention correction beyond 32K)
+- Required topology (Single GPU → TP×2/4/8 → TP×8 + Context Parallel)
+- RingAttention citation when CP is needed
+
+## The math, transparently
+
+Every number above comes from a formula you can inspect. No black boxes.
+
+<details>
+<summary><b>Decode tokens/sec (memory-bandwidth bound)</b></summary>
+
+$$
+\text{decode tok/sec} \approx \frac{\text{HBM BW} \cdot \eta_{\text{mem}} \cdot \text{quant\_eff}}{\text{model size}}
+$$
+
+Where:
+- `HBM BW` = GPU memory bandwidth (e.g., 3350 GB/s for H100 SXM)
+- `η_mem = 0.65` = typical real-world memory utilization (35% overhead)
+- `quant_eff` = dequantization efficiency multiplier (1.0 for FP16, 1.5 for FP8 on H100, 0.85 for INT4)
+- `model size = active_params × bytes_per_param` (uses ACTIVE params for MoE, not total)
+
+Refs: PagedAttention paper ([arxiv.org/abs/2309.06180](https://arxiv.org/abs/2309.06180))
+</details>
+
+<details>
+<summary><b>Prefill tokens/sec (compute bound)</b></summary>
+
+$$
+\text{prefill tok/sec} \approx \frac{\text{GPU FLOPS} \cdot \eta_{\text{compute}}}{2 \cdot \text{active params}}
+$$
+
+Where:
+- `GPU FLOPS` = dense FP16/BF16 TFLOPS (sparse values not used)
+- `η_compute = 0.50` = typical compute utilization
+- Factor of 2 = one multiply + one add per parameter per token
+
+For long context (>32K), the superlinear attention correction above applies.
+</details>
+
+<details>
+<summary><b>Continuous batching multiplier (workload-specific)</b></summary>
+
+$$
+\text{aggregate tok/sec} = \text{decode tok/sec} \cdot \text{batch size} \cdot \text{continuous batching multiplier}
+$$
+
+**Critical caveat**: There is no universal continuous batching multiplier. vLLM
+reported 14–24× vs HF Transformers (extreme), 2.2–2.5× vs TGI. SOSP paper
+finds 2–4× typical vs FasterTransformer/Orca. tokcalc defaults to a
+conservative **1.5×** and lets you tune.
+
+Refs:
+- [vLLM blog (2023-06-20)](https://vllm.ai/blog/2023-06-20-vllm)
+- [PagedAttention paper](https://arxiv.org/abs/2309.06180)
+- [Anyscale continuous batching study](https://www.anyscale.com/blog/continuous-batching-llm-inference)
+</details>
+
+<details>
+<summary><b>Prompt caching economics (Anthropic / OpenAI)</b></summary>
+
+For shared prefix of length $T_p$, suffix of length $T_u$, output $O$,
+cache hit rate $h$:
+
+$$
+\text{API input cost} = N \cdot \left[ (1-h) \cdot T_p \cdot P_{\text{write}} + h \cdot T_p \cdot P_{\text{read}} + T_u \cdot P_{\text{input}} \right]
+$$
+
+Anthropic multipliers (verified 2025-2026):
+- 5-minute cache write: 1.25× base input
+- 1-hour cache write: 2.0× base input
+- Cache read: 0.1× base input (90% savings)
+
+OpenAI: cached input discounted 50%, no separate write fee.
+
+Refs:
+- [Anthropic prompt caching docs](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching)
+- [OpenAI prompt caching guide](https://platform.openai.com/docs/guides/prompt-caching)
+</details>
+
+<details>
+<summary><b>Multi-GPU topology recommendation</b></summary>
+
+$$
+\text{total needed} = \text{model weights} + (\text{KV per request} \cdot \text{batch size})
+$$
+
+Walk the smallest topology that fits:
+- **Single GPU**: total ≤ VRAM × 1
+- **Tensor Parallel ×2/4/8**: total ≤ VRAM × N (weights + KV split evenly)
+- **TP×8 + Context Parallel**: total > VRAM × 8 — use RingAttention to shard KV across nodes
+
+Refs: [RingAttention paper](https://arxiv.org/abs/2310.01889)
+</details>
+
+<details>
+<summary><b>Self-host vs API break-even</b></summary>
+
+$$
+\text{self-host $/M tokens} = \frac{\text{GPU $/hr}}{3600 \cdot \text{effective tok/s} \cdot \text{utilization}} \cdot 10^6
+$$
+
+$$
+\text{break-even req/day} = \frac{\text{monthly self-host cost}}{30 \cdot \text{API cost per request}}
+$$
+
+The decisive term is **effective utilization** — not peak throughput. A GPU
+running at 10% utilization pays 10× more per token than the theoretical minimum.
+</details>
+
+## Comparison with adjacent tools
+
+| Capability | induwara / techfuelhq / pcmasterstudio | gigagpu / kickllm / cloudparity | HF Open LLM Leaderboard / MLPerf | **tokcalc** |
+|---|:---:|:---:|:---:|:---:|
+| Model × GPU × quant tok/s | ✓ | some | some | **✓** |
+| Model-fit / VRAM | basic | rare | rare | **✓** |
+| KV-cache by context + concurrency | — | — | implicit | **✓** |
+| Prefill vs decode split (TTFT/ITL) | — | — | engine-specific | **✓** |
+| Continuous batching / paged attention | — | — | docs only | **✓** |
+| Long-context (128K–1M) planning | — | — | — | **✓** |
+| Topology recommendation (TP/CP) | — | — | partial | **✓** |
+| Prompt-cache economics | — | partial API only | — | **✓** |
+| Reasoning tokens (o1/R1/Claude thinking) | — | — | — | **✓** |
+| API vs self-host break-even | some | ✓ | — | **✓** |
+| Transparent formulas / open source | mixed | usually no | mixed | **✓** |
+| Cited benchmark evidence per config | rare | rare | ✓ (not planning) | **✓ (in progress)** |
+| Shareable URL per config | — | — | — | **✓** |
+
+## Roadmap
+
+### Shipped
+- ✅ 35 models, 30 GPUs, 16 quantization formats
+- ✅ Continuous batching, reasoning tokens, prompt caching
+- ✅ TTFT/ITL split, long-context superlinear attention
+- ✅ Long-context capacity planner + topology recommendation
+- ✅ Build-vs-Buy calculator (13 API providers with retired/current status)
+- ✅ Reference catalog (5 sub-tables)
+- ✅ Share URL + localStorage persistence
+- ✅ Dark mode toggle
+- ✅ Plain-English glossary (28 terms with hover tooltips)
+- ✅ OG image + Twitter card + social metadata
+
+### Next 30 days
+- ⏳ GitHub Action (`tokcalc/plan` PR comment)
+- ⏳ MCP server (read-only capacity-planning tools for AI agents)
+- ⏳ 3 SEO landing pages (`/compare/h100-vs-h200`, `/gguf-q4-k-m-vs-q5-k-m`, `/vllm-vs-sglang`)
+- ⏳ i18n: Chinese, Japanese, Korean
+
+### Next 90 days
+- ⏳ Workload-trace / SLO capacity planner (prompt/output/arrival distributions, p50/p95 TTFT/ITL)
+- ⏳ P/D disaggregation planner (separate prefill + decode pools)
+- ⏳ Cache-aware economics (prefix-sharing distribution, multi-turn/agent traces)
+- ⏳ Engine-aware presets (vLLM / SGLang / TensorRT-LLM / llama.cpp)
+- ⏳ Versioned price + benchmark provenance system
+
+### Long-term
+- 🔮 Agentic workflow calculator (multi-turn + tool calls + growing context)
+- 🔮 Multi-LoRA capacity planner (Punica / S-LoRA economics)
+- 🔮 VLM image-token accounting (per-model patch/tile tokenization)
+- 🔮 Embedding model mode (vectors/sec, separate workload)
+- 🔮 Training/fine-tuning estimator (LoRA / QLoRA / full-SFT FLOPs)
+- 🔮 Energy / carbon per million tokens (region-specific grid intensity)
+
+## Open core model
+
+tokcalc is **open core** — the calculator and catalog are open source; the
+cloud/data/team features are paid.
+
+| Asset | License | Notes |
+|---|---|---|
+| Source code | Apache 2.0 | This repo. Free to use, modify, distribute |
+| Model/GPU/quant catalog | CC0 1.0 | Public domain data. Anyone can use, no attribution required |
+| Benchmark provenance data | CC-BY-SA 4.0 | Anyone can use, but must attribute + share-alike |
+| Documentation | CC-BY 4.0 | Attribution required if copied |
+| "tokcalc" name + logo | Trademark | Even without formal registration, common-law rights apply |
+| Cloud SaaS layer | Proprietary | Real-time pricing API, benchmark DB, team workspaces (coming soon) |
+
+### Why this structure
+- **Trust**: Open-source formulas build credibility vs opaque competitors
+- **Community**: Contributors can submit models, GPUs, quants, benchmarks
+- **Defensibility**: Trademark + cloud features + URL-share viral loop protect against forks
+- **Revenue**: Cloud tier funds ongoing development + pricing/benchmark data maintenance
+
+## Contributing
+
+We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for:
+
+- How to add a model (with HuggingFace config.json as source)
+- How to add a GPU (with critical guardrails for B200/B300 null FP16 fields)
+- How to add a quantization format (with measured file sizes as source)
+- How to submit a benchmark (3-tier confidence model)
+- How to improve a formula (cite the source, no magic numbers)
+
+### Most-needed contributions
+- 🟢 New models (Qwen 3, Mistral Large 4, Llama 4 variants as they release)
+- 🟢 New GPUs (B300, AMD MI400, Apple M5 Ultra when shipping)
+- 🟢 New quantization formats (mxFP8, MXFP4, BitNet 2)
+- 🟢 Real benchmark data (run vLLM benchmarks and submit with provenance)
+- 🟢 Translations (especially Chinese, Japanese, Korean)
+
+## MCP server — use tokcalc from AI agents
+
+tokcalc ships an [MCP (Model Context Protocol)](https://modelcontextprotocol.io) server that lets AI agents (Cursor, Claude Desktop, Cline) call tokcalc during design reviews.
+
+### 6 read-only tools
+
+| Tool | What it does |
+|---|---|
+| `estimate_capacity` | VRAM/KV/throughput/latency/cost for one config |
+| `compare_gpus` | Ranked GPU comparison for one workload |
+| `recommend_topology` | TP/CP topology recommendation |
+| `estimate_api_vs_self_host` | Break-even analysis |
+| `list_models` | Discover supported model IDs |
+| `list_gpus` | Discover supported GPU IDs |
+
+All tools are **read-only** — no side effects, no cloud credentials, no deployments.
+
+### Install
+
+Add to your Claude Desktop config (`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS):
+
+```json
+{
+  "mcpServers": {
+    "tokcalc": {
+      "command": "npx",
+      "args": ["-y", "@tokcalc/mcp-server"]
+    }
+  }
+}
+```
+
+Or run locally:
+
+```bash
+git clone https://github.com/stevecrates489-commits/tokcalc.git
+cd tokcalc
+bun install
+bun mini-services/mcp-server/index.ts
+```
+
+### Example agent prompt
+
+> "I need to serve Llama 3.3 70B at 32K context for 50 concurrent users. What GPU topology do you recommend, and how much will it cost per month?"
+
+The agent calls `list_models` → `list_gpus` → `recommend_topology` → `estimate_capacity` → returns a structured plan with VRAM, throughput, latency, cost, and confidence.
+
+## Tech stack
+
+- **Framework**: Next.js 16 with App Router
+- **Language**: TypeScript 5
+- **Styling**: Tailwind CSS 4 + shadcn/ui (New York)
+- **Charts**: Recharts
+- **State**: React hooks (useState + useEffect + useMemo)
+- **Theme**: next-themes (dark mode default)
+- **Database**: None (pure client-side, no backend required for core features)
+
+## Acknowledgments
+
+tokcalc builds on the work of:
+
+- **vLLM team** — PagedAttention, continuous batching ([arxiv.org/abs/2309.06180](https://arxiv.org/abs/2309.06180))
+- **llama.cpp / ggml-org** — GGUF format and quantization variants
+- **MLPerf / MLCommons** — standardized inference benchmark methodology
+- **Stanford HELM** — efficiency-aware model evaluation framework
+- **Anthropic / OpenAI / Google** — published prompt-caching pricing rules
+- **NVIDIA / AMD / Intel / Google / Groq / Cerebras** — published hardware specs
+
+Every formula has a citation. Every model/GPU/quant entry has a source URL
+where available. If you spot an unsourced claim, please open an issue.
+
+## Star history
+
+[![Star History Chart](https://api.star-history.com/svg?repos=tokcalc/tokcalc&type=Date)](https://star-history.com/#tokcalc/tokcalc&Date)
+
+---
+
+<div align="center">
+
+**[Live demo](https://tokcalc.dev)** ·
+**[Documentation](https://tokcalc.dev/docs)** ·
+**[Contributing](CONTRIBUTING.md)** ·
+**[License](LICENSE)** ·
+**[Code of Conduct](CODE_OF_CONDUCT.md)**
+
+Made with care by the tokcalc community. Apache 2.0 licensed.
+
+</div>