@tokcalc/mcp-server 0.1.3 → 0.2.0-alpha.1

package/tool-results/read_1789888151021_69f58eec6a5b.txt

@@ -1,653 +0,0 @@
-     1→# tokcalc expansion research brief
-     2→
-     3→**Research cutoff:** 20 September 2026 (IST)  
-     4→**Scope:** Sections A–H requested by the maintainer.  
-     5→**Evidence policy:** Numeric claims are linked to primary/official pages or papers wherever available. Dynamic prices, retired models, and provider availability are explicitly statused; n/a means not verified/publicly disclosed, not zero. A benchmark multiplier is never treated as a universal constant.
-     6→
-     7→## Executive recommendation
-     8→
-     9→Prioritize a **scenario-based inference calculator** rather than adding dozens of static model/GPU rows first. The highest-value first release is: (1) continuous-batching/KV-cache capacity, (2) prefix-cache economics and TTFT, (3) reasoning/agent turn modeling, (4) build-vs-buy cost curves, and (5) benchmark calibration. Add models and accelerators through versioned, provenance-bearing data records—not hard-coded constants.
-    10→
-    11→## Section G — TOP 10 PRIORITIZED FEATURE EXPANSIONS
-    12→
-    13→| Rank | Feature | Description | Why users want it / evidence | Complexity / rough effort | Required data | Dependencies |
-    14→|---:|---|---|---|---|---|---|
-    15→| 1 | **Continuous batching + KV-cache simulator** | Estimate dynamic batch capacity, TTFT, ITL/TPOT, throughput and queueing under prompt/output/QPS distributions. | vLLM reported 14–24× vs HF and 2.2–2.5× vs TGI in one-completion tests, while other tests report up to 23× over naive batching; the spread proves users need condition-aware estimates, not one multiplier. [vLLM](https://vllm.ai/blog/2023-06-20-vllm) [Anyscale](https://www.anyscale.com/blog/continuous-batching-llm-inference) | **High, 32–56h** | Model architecture, KV precision, GPU memory, engine presets, prompt/output distributions, concurrency/QPS | Existing model/GPU tables; becomes the base for caching and long-context modes |
-    16→| 2 | **Prefix/prompt caching economics** | Model prefix hit rate, TTL/write/read rates, reduced prefill tokens and API cost. | Anthropic cache reads are 0.1× base input; 5-minute writes are 1.25× and 1-hour writes 2×. OpenAI documents cached-input discounts, while vLLM APC reuses KV state but does not reduce decode. [Anthropic](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) [OpenAI](https://developers.openai.com/api/docs/guides/prompt-caching.md) [vLLM](https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/) | **Medium, 16–28h** | Provider/model price records, cache TTL, hit rate, prefix tokens, suffix tokens, KV cache | Pricing table; prefill model from #1 |
-    17→| 3 | **Reasoning + agent workflow mode** | Model hidden-reasoning budget, visible output, tool rounds, tool-result growth, and cumulative latency/cost. | OpenAI bills hidden reasoning tokens as output; Anthropic thinking has a minimum 1,024-token budget; tool schemas/results are part of token accounting. [OpenAI](https://help.openai.com/en/articles/4936856-understanding-and-counting-tokens) [Anthropic thinking](https://platform.claude.com/docs/en/build-with-claude/extended-thinking) [Anthropic tools](https://platform.claude.com/docs/en/build-with-claude/tool-use) | **Medium, 18–32h** | Reasoning-token distribution, turn count, tool schema/result tokens, model/API rates | #1 for latency; #2 for stable prefixes |
-    18→| 4 | **Build-vs-buy break-even calculator** | Compare self-hosted GPU cost per request/token against API cost, utilization, replicas, cold starts and SLO. | It directly extends tokcalc’s existing cost calculator; self-host cost is hourly cost divided by effective tokens/hour, while API cost is input/cache/output plus tool/media charges. Rates are volatile, so users need timestamped scenarios. [Modal](https://modal.com/pricing) [OpenAI](https://developers.openai.com/api/docs/models/gpt-4o) | **Medium, 16–24h** | GPU/provider price snapshots, API price records, throughput benchmarks, utilization, HA/egress/cold-start assumptions | #1–#3; Section E/F data |
-    19→| 5 | **Benchmark import + calibration layer** | Import AIPerf/vLLM/MLPerf measurements and fit model/GPU/engine-specific correction factors. | The same model/GPU can vary materially by batch, context, quantization and engine; AIPerf exposes TTFT/ITL/p50/p99 and vLLM exposes serving benchmarks. [AIPerf](https://github.com/ai-dynamo/aiperf) [vLLM](https://github.com/vllm-project/vllm) | **Medium, 20–36h** | Benchmark schema, CSV/JSON importer, source/license metadata | #1 and all hardware/model records |
-    20→| 6 | **Engine-aware quantization presets** | Distinguish GGUF/GPTQ/AWQ/EXL2/FP8/NVFP4 by effective bpw, KV precision, kernel and engine support. | GGUF Llama 3.1 8B measured 2.95 GiB Q2_K to 7.95 GiB Q8_0 vs 14.96 GiB FP16; FP8 H100 results range from ~1.4–1.5× to 2.3× in cited setups. [llama.cpp](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/quantize/README.md) [TensorRT-LLM](https://nvidia.github.io/TensorRT-LLM/1.2.0/blogs/quantization-in-TRT-LLM.html) | **Medium, 20–32h** | Quantization records, engine/version compatibility, measured size/PPL/speed | #1; model and GPU schema |
-    21→| 7 | **Long-context capacity mode** | Show KV growth, prefill attention cost, multi-GPU/interconnect need, and quality caveats at 128K–1M. | RingAttention served 256K in a 32-TPU-v5e setup; KV memory is linear in context while prefill attention work is superlinear for dense attention. [RingAttention](https://arxiv.org/html/2310.01889v4) | **High, 28–48h** | Context limits, layers/KV heads/head dim, KV precision, interconnect, distributed-attention presets | #1, #6; model architecture |
-    22→| 8 | **Embedding calculator** | Add encoder throughput, vectors/sec, dimensions/storage, chunking and API input cost as a separate workload type. | BGE-M3 supports 1,024-dimensional vectors and 8,192-token sequences; E5-large-v2 examples use 512 tokens; autoregressive output tok/s is the wrong metric. [BGE-M3](https://huggingface.co/BAAI/bge-m3) [E5](https://huggingface.co/intfloat/e5-large-v2) | **Medium, 12–20h** | Embedding model metadata, batch throughput, dimensions, vector dtype, API prices | Separate workload abstraction; can reuse provider pricing |
-    23→| 9 | **VLM image/audio token accounting** | Convert image tiles/patches/video frames and audio duration into prefill/cost/latency inputs. | Gemini documents 258 tokens for small images and 258-token 768×768 tiles; OpenAI vision uses 32×32 patch accounting for applicable models. [Gemini](https://ai.google.dev/gemini-api/docs/image-understanding) [OpenAI vision](https://platform.openai.com/docs/guides/images-vision.md) | **High, 24–40h** | Processor/token rules by model, resolution/detail, frame rate, audio RTF/quality | #1, #2; VLM/audio model records |
-    24→| 10 | **Training/fine-tuning estimator** | Estimate LoRA/QLoRA/full-SFT memory, FLOPs, wall time and cost. | LoRA reports up to 10,000× fewer trainable parameters and 3× lower GPU memory in its evaluated setup; QLoRA demonstrated 65B fine-tuning on one 48GB GPU. [LoRA](https://arxiv.org/abs/2106.09685) [QLoRA](https://arxiv.org/html/2305.14314v1) | **High, 32–56h** | Training optimizer/activation/checkpointing assumptions, adapter rank, dataset tokens, GPU prices | GPU table; separate training FLOPs model |
-    25→
-    26→**Defer:** PD disaggregation, multi-LoRA, energy/carbon, cold-start distributions and research quantizers until the benchmark/provenance layer exists. PD disaggregation is experimental in current vLLM docs and does not improve throughput by itself. [vLLM PD](https://docs.vllm.ai/en/stable/features/disagg_prefill)
-    27→
-    28→## Maintainer data-model rules
-    29→
-    30→- Version every model, GPU, quantization and price record with source URL, capture time, status, scope, engine version and confidence.
-    31→- Store dense FP16/BF16, sparse/FP8/FP4 values separately; never substitute AI TOPS or system PFLOPS for dense per-chip FLOPs.
-    32→- Store metric scope (chip, board, node, system), memory scope (VRAM, HBM, unified), and benchmark conditions.
-    33→- Keep advertised context separate from verified context; DeepSeek V3 is an example where the card says 128K while config exposes 163,840. [DeepSeek V3](https://huggingface.co/deepseek-ai/DeepSeek-V3)
-    34→- For quantization, store nominal bits, effective bits/weight, activation/KV bits, group size, scale bits, actual checkpoint size, calibration requirement, engine support, and raw quality metric—not a universal accuracy-loss percentage.
-    35→- Keep current, retired, enterprise/contact-sales, historical, announced and unverified as explicit statuses.
-    36→
-    37→---
-    38→# tokcalc SECTION A — implementation-ready research (as of 2026-09-20)
-    39→
-    40→Status: best effort with primary docs/papers and vendor benchmark pages. Numeric claims are cited inline. No universal speedup exists; benchmark conditions matter.
-    41→
-    42→## 1. Continuous batching / PagedAttention
-    43→- **Definition:** Continuous (iteration-level) batching admits/removes sequences every decode iteration; PagedAttention stores KV cache in fixed-size non-contiguous blocks, avoiding contiguous preallocation/fragmentation.
-    44→- **User value:** Higher utilization/throughput, better tail behavior, and more concurrent contexts than static batching; a calculator should expose TTFT, TPOT/ITL, throughput, queueing, and KV-cache capacity rather than one speedup.
-    45→- **Complexity / priority:** High / Must-have.
-    46→- **Quantitative evidence:** vLLM's original A10G/A100 ShareGPT benchmark reports 14–24x throughput vs HF and 2.2–2.5x vs TGI for one completion/request; for three parallel completions it reports 8.5–15x vs HF and 3.3–3.5x vs TGI, with under 4% KV memory waste; its PagedAttention sharing reduced parallel-sampling memory up to 55% and translated to up to 2.2x throughput. [vLLM benchmark](https://vllm.ai/blog/2023-06-20-vllm)
-    47→- A separate A100/OPT-13B study reports up to 23x throughput with vLLM continuous-batching + memory optimizations, 8x over naive batching, and 4x for an optimized FasterTransformer; it used 1,000 requests with 512-token prompts and variable decode lengths, so do not present 23x as a universal multiplier. [Anyscale study](https://www.anyscale.com/blog/continuous-batching-llm-inference)
-    48→- The SOSP paper summarizes 2–4x throughput at the same latency vs FasterTransformer/Orca; this is a distinct workload/implementation comparison. [PagedAttention paper](https://arxiv.org/abs/2309.06180)
-    49→- **Formula implications:** KV bytes/request ≈ 2 × layers × KV heads × head_dim × sequence_tokens × bytes/element; paged capacity is floor(free_KV_bytes/block_bytes), and dynamic batch size is constrained by total KV blocks plus scheduler token budget. Compute throughput separately for prefill and decode; report measured ranges, not a fixed multiplier.
-    50→- **Caveats:** Results are old/engine- and workload-specific; vLLM's 2024 update reports 1.8–2.7x throughput gains vs its prior v0.5.3 and workload-dependent results vs TensorRT-LLM/SGLang, with higher TTFT at low QPS when multi-step scheduling is used. [vLLM 0.6 benchmark](https://vllm.ai/blog/2024-09-05-perf-update)
-    51→
-    52→## 2. Prefix / prompt caching
-    53→- **Definition:** Reuse KV states for an identical prompt prefix so only the uncached suffix is prefilled.
-    54→- **User value:** Lower TTFT and input-token cost for repeated system prompts, RAG documents, tools, and multi-turn histories.
-    55→- **Complexity / priority:** Medium (local engines), Medium (API cost model) / Must-have.
-    56→- **Anthropic:** Prompt caching supports automatic or explicit breakpoints (up to 4), default ephemeral TTL 5 minutes, optional 1-hour TTL; 5-minute writes are 1.25x base input price, 1-hour writes 2x, cache reads 0.1x (model pricing varies). Anthropic's token formula is `total_input = cache_read + cache_creation + uncached_input`; cached prompt work does not remove decode time. [Anthropic prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)
-    57→- **OpenAI:** Prompt caching is enabled by default for supported models; docs say cached input is discounted up to 90%. Current docs describe 1.25x cache-write and 0.1x cache-read multipliers for GPT-5.6+; 1,024 visible tokens is the stated minimum for GPT-5.6+, while earlier models are model/settings-dependent. [OpenAI prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching.md)
-    58→- **Formula implications:** For prefix length P, suffix U, output O, base input price c_i, output c_o, cache write multiplier w and read multiplier r: first-hit cost = P·w·c_i + U·c_i + O·c_o; later hit = P·r·c_i + U·c_i + O·c_o. Add hit-rate h: expected prefix cost per request ≈ [(1−h)w + h r]P c_i. Include TTL/eviction and cache-warming behavior.
-    59→- **Caveats:** Exact match is required; any changed tools, images, model/settings, or prefix token invalidates suffix reuse. APC only speeds prefill, not decode. Cache locality/routing, model-specific minimums, retention policy, and cache-write charges must be configurable. [vLLM APC](https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/) [OpenAI cache details](https://developers.openai.com/api/docs/guides/prompt-caching.md)
-    60→
-    61→## 3. Chunked prefill
-    62→- **Definition:** Split large prompt prefills into chunks and schedule them with decode work under a `max_num_batched_tokens` budget.
-    63→- **User value:** Prevents a long prefill from monopolizing the GPU, improving inter-token latency (ITL), decode fairness, and often utilization/throughput.
-    64→- **Complexity / priority:** Medium–High / Must-have.
-    65→- **Quantitative evidence:** vLLM docs recommend smaller token budgets (example 2,048) for better ITL and higher budgets (>8,192) for optimal throughput on smaller models/large GPUs; these are tuning guidance, not guaranteed gains. [vLLM optimization docs](https://docs.vllm.ai/en/stable/configuration/optimization)
-    66→- **Formula implications:** At each scheduler step allocate `B = max_num_batched_tokens − decode_tokens` to prefill; estimate TTFT ≈ queue + ceil(prompt_tokens/chunk_tokens)·prefill_step and ITL as decode step plus prefill interference. Add workload sweep over chunk size and prompt/decode mix.
-    67→- **Caveats:** Smaller chunks lower ITL but can hurt TTFT/throughput; optimal value depends on GPU, model, sequence mix, and QPS. I found no authoritative universal percentage speedup; flag this as unverified rather than inventing one.
-    68→
-    69→## 4. Prefill/decode disaggregation (PD)
-    70→- **Definition:** Run prefill and autoregressive decode in separate workers/instances and transfer KV cache/results through a connector.
-    71→- **User value:** Tune TTFT and tail ITL independently, use different TP/PP layouts, and isolate compute-heavy prefill from memory-bound decode.
-    72→- **Complexity / priority:** High / Nice (Future for small self-hosted deployments).
-    73→- **Quantitative evidence:** Current vLLM docs explicitly say PD disaggregation is experimental, gives independent TTFT/ITL tuning and tail-ITL control, and **does not improve throughput** by itself; correct chunked prefill can target similar behavior but is harder to tune. [vLLM disaggregated prefilling](https://docs.vllm.ai/en/stable/features/disagg_prefill)
-    74→- **Formula implications:** Model end-to-end latency as prefill service + KV transfer (`KV_bytes / interconnect_bandwidth`) + decode queue/service; capacity is separate prefill and decode worker pools. Add transfer overhead, connector, routing, and queueing fields. 
-    75→- **Caveats:** Do not market a generic throughput multiplier. Gains depend on SLO and traffic mix; transfer bandwidth, topology, KV format, and connector maturity dominate. A 2026 vLLM blog discusses single-node MI300X PD but was not used for a numeric claim. [vLLM PD blog](https://vllm.ai/blog/2026-04-07-moriio-kv-connector)
-    76→
-    77→## 5. Multi-LoRA (Punica/S-LoRA)
-    78→- **Definition:** Serve many tenant/task-specific low-rank adapters over one shared base model using heterogeneous batching and fused adapter kernels.
-    79→- **User value:** Personalization and multi-tenant model variants without loading a full base model per tenant; expose adapter count, rank, routing mix, and shared-base memory.
-    80→- **Complexity / priority:** High / Nice.
-    81→- **Quantitative evidence:** Punica reports 12x throughput vs state-of-the-art systems on fixed GPU clusters while adding 2 ms/token, evaluated on Llama 2 7B/13B/70B on A100 clusters. [Punica paper](https://arxiv.org/abs/2310.18547) [MLSys paper](https://proceedings.mlsys.org/paper_files/paper/2024/file/054de805fcceb78a201f5e9d53c85908-Paper-Conference.pdf)
-    82→- S-LoRA reports up to 4x throughput vs vLLM-packed and up to 30x vs PEFT in its evaluated settings, while serving orders-of-magnitude more adapters. [S-LoRA paper](https://arxiv.org/html/2311.03285) [MLSys paper](https://proceedings.mlsys.org/paper_files/paper/2024/file/906419cd502575b617cc489a1a696a67-Paper-Conference.pdf)
-    83→- **Formula implications:** Base weight memory paid once; adapter memory ≈ number_adapters × 2 × rank × (sum adapted input/output dimensions) × bytes/element. Throughput is a function of adapter-mix diversity, rank, batching, and adapter transfer/cache misses—not adapter count alone.
-    84→- **Caveats:** Paper multipliers are not universal and compare particular baselines/hardware; use benchmark-backed scenario presets.
-    85→
-    86→## 6. Reasoning-model workloads
-    87→- **Definition:** Models generate hidden/internal reasoning tokens in addition to visible answer tokens and may reason between tool calls.
-    88→- **User value:** Accurate cost, latency, context, and GPU-capacity estimates for o1/o3/R1/Claude-thinking-like workloads.
-    89→- **Complexity / priority:** Medium / Must-have (as a workload mode).
-    90→- **Quantitative evidence:** OpenAI states reasoning tokens are not visible but count toward output usage and are billed as output tokens; `max_output_tokens` includes reasoning, visible output, and formatting tokens. [OpenAI token guide](https://help.openai.com/en/articles/4936856-understanding-and-counting-tokens) [OpenAI reasoning guide](https://developers.openai.com/api/docs/guides/reasoning)
-    91→- Anthropic manual extended thinking requires a minimum 1,024-token `budget_tokens`, less than `max_tokens`; actual usage can stop before the budget and thinking tokens count toward the turn's output ceiling. [Anthropic extended thinking](https://platform.claude.com/docs/en/build-with-claude/extended-thinking)
-    92→- **Formula implications:** `billed_output = reasoning_tokens + visible_output_tokens`; request latency ≈ prefill + decode steps for both classes. Allow a distribution/range for hidden reasoning tokens, not a fixed ratio. Include tool-interleaving turns.
-    93→- **Caveats:** Provider/model semantics differ; no defensible universal “reasoning is X times slower/costlier” multiplier was verified.
-    94→
-    95→## 7. Agentic workloads
-    96→- **Definition:** A workflow of repeated model turns interleaved with tool calls/results, often carrying forward growing conversation state.
-    97→- **User value:** Estimate cumulative token cost, latency, context overflow, and throughput for agents rather than one-shot completions.
-    98→- **Complexity / priority:** Medium / Must-have (scenario mode).
-    99→- **Quantitative evidence:** Anthropic's tool-use docs state client tools require a model response (`tool_use`), external execution, then a new request carrying `tool_result`; tool schemas, tool-use blocks, tool-result blocks, and a special tool-use system prompt contribute billed input/output tokens. [Anthropic tool use](https://platform.claude.com/docs/en/build-with-claude/tool-use)
-   100→- **Formula implications:** For T turns, total input tokens = Σ(stable prefix + prior history + tool schemas + tool results + new user input); total latency = Σ(model latency + tool execution + network/queue). Add number of sequential rounds, parallel tool fan-out, average result tokens, schema tokens, and cache-hit rate. Parallel tool calls reduce sequential model round trips but exact latency savings are workload-dependent.
-   101→- **Caveats:** I found no authoritative general benchmark that isolates a universal tool-calling latency overhead; report user-supplied or measured distributions, not a guessed fixed penalty.
-   102→
-   103→## 8. Long-context serving (128K–1M)
-   104→- **Definition:** Serve contexts far beyond 32K using RoPE/context extension and distributed/blockwise attention while managing KV-cache growth.
-   105→- **User value:** Model long documents, codebases, and agent histories; quantify memory, TTFT, throughput collapse, and multi-GPU requirements.
-   106→- **Complexity / priority:** High / Nice.
-   107→- **Quantitative evidence:** RingAttention describes blockwise KV communication and reports that Llama 7B on 32 TPUv5e can serve up to 256K context at batch size 1 in its setup; its analysis says scaling to 1M context can cost roughly 20–40x more training FLOPs for small models. [RingAttention paper](https://arxiv.org/abs/2310.01889) [HTML](https://arxiv.org/html/2310.01889v4)
-   108→- YaRN reports 10x fewer extension-training tokens and 2.5x fewer training steps than prior context-extension methods, with 32K–128K Llama extensions in its experiments. [YaRN paper](https://arxiv.org/abs/2309.00071)
-   109→- **Formula implications:** KV memory remains linear in sequence length: `KV ≈ 2·L·H_kv·D·S·B`; decoder attention is memory-bandwidth/KV-bound, while prefill attention work grows superlinearly with S for dense attention. Ring/blockwise methods shard memory and communication, so add number of devices, interconnect bandwidth, and block size.
-   110→- **Caveats:** Max context ≠ effective retrieval quality; RoPE extension may need fine-tuning and quality validation. Long-context throughput numbers are setup-specific; no universal 1M-token multiplier verified.
-   111→
-   112→## 9. Embedding throughput
-   113→- **Definition:** Encoder-style models map input text to fixed-size vectors in one forward pass; throughput is input sequences/tokens per second, not autoregressive output tokens/s.
-   114→- **User value:** RAG/search users need indexing throughput, vector dimensions/storage, batch sizing, and API cost separate from generation.
-   115→- **Complexity / priority:** Medium / Nice.
-   116→- **Quantitative evidence:** BGE-M3 model card reports 1,024-dimensional vectors and up to 8,192-token sequences. [BGE-M3 card](https://huggingface.co/BAAI/bge-m3)
-   117→- E5-large-v2's model card example truncates/tokenizes at 512 tokens. [E5-large-v2 card](https://huggingface.co/intfloat/e5-large-v2)
-   118→- MTEB spans 8 embedding tasks, 58 datasets, and 112 languages, evidence that embeddings are a distinct workload/evaluation ecosystem (not direct product-demand measurement). [MTEB paper](https://arxiv.org/abs/2210.07316)
-   119→- **Formula implications:** `vectors/s = batch_size × effective_tokens / encoder_latency`; storage/request = dimensions × bytes/element; API cost = input_tokens × price/M tokens. Add pooling/normalization, truncation/chunking, batch size, and concurrency.
-   120→- **Caveats:** Do not infer demand or compare embedding quality from throughput alone; MTEB is quality-focused and vendor throughput depends on batch/model/GPU. I found no single authoritative market-demand number.
-   121→
-   122→## 10. Vision-language/image tokens
-   123→- **Definition:** Image preprocessing converts pixels/patches/tiles into visual tokens or embeddings that consume prefill compute, KV memory, context, and often billable input tokens.
-   124→- **User value:** Avoid severe underestimation of multimodal TTFT, memory, and API cost; support image count, resolution, detail level, frame sampling, and token pruning.
-   125→- **Complexity / priority:** High / Must-have if VLM models are supported.
-   126→- **Quantitative evidence:** Gemini docs allocate 258 tokens for images with both dimensions ≤384 px; larger images are tiled to 768×768, each tile costing 258 tokens, with a crop-unit/tile formula. [Gemini image understanding](https://ai.google.dev/gemini-api/docs/image-understanding)
-   127→- OpenAI's current vision docs specify 32×32 patch tokenization for applicable models: `patch_count = ceil(width/32) × ceil(height/32)`, then model/detail-specific patch budgets/resizing; images count toward input token and TPM limits. [OpenAI images/vision](https://platform.openai.com/docs/guides/images-vision.md)
-   128→- vLLM supports content-hash or stable-ID multimodal caching and video token pruning, useful for local serving formulas. [vLLM multimodal inputs](https://docs.vllm.ai/en/stable/features/multimodal_inputs/)
-   129→- **Formula implications:** `total_prefill_tokens = text_tokens + Σ(image_tokens) + Σ(video_frame_tokens)`; cost = total billable input × input rate; KV memory applies after projected visual tokens enter the language backbone. Add resolution/detail/model-specific tokenizer tables.
-   130→- **Caveats:** Image-token rules are model/provider-specific and changing; don't use one universal “image = N tokens.” Llama/Qwen/Pixtral tokenization must be sourced per model card/processor.
-   131→
-   132→## 11. Training throughput & fine-tuning
-   133→- **Definition:** A training calculator estimates forward/backward FLOPs, optimizer/activation/parameter memory, wall time, and cost for full SFT, LoRA, and QLoRA.
-   134→- **User value:** Users can choose feasible GPUs and compare fine-tuning economics before provisioning.
-   135→- **Complexity / priority:** High / Nice (Future full trainer; adapter estimator sooner).
-   136→- **Quantitative evidence:** Chinchilla's paper trained >400 models spanning 70M–16B parameters and 5B–500B tokens; its compute-optimal finding is to scale model size and training-token count approximately equally under a fixed compute budget. [Chinchilla paper](https://arxiv.org/abs/2203.15556)
-   137→- LoRA reports up to 10,000× fewer trainable parameters and 3× lower GPU memory vs GPT-3 full fine-tuning with Adam in its evaluated setup. [LoRA paper](https://arxiv.org/abs/2106.09685)
-   138→- QLoRA reports fine-tuning a 65B model on one 48GB GPU; its paper describes reducing >780GB average memory for 16-bit fine-tuning to <48GB in the evaluated setup. [QLoRA paper](https://arxiv.org/html/2305.14314v1)
-   139→- **Formula implications:** First approximation for dense decoder training compute is proportional to `6 × parameters × training_tokens`; show as an assumption, then calibrate with measured FLOPs/utilization. LoRA trainable params = sum over adapted matrices `r·(d_in+d_out)`; QLoRA adds quantized frozen weights plus adapter/optimizer/activation memory.
-   140→- **Caveats:** 6ND is an approximation; attention, sequence length, activation checkpointing, optimizer, packing, MoE routing, and communication alter real cost. LoRA ratios are not universal.
-   141→
-   142→## 12. Build-vs-buy calculator
-   143→- **Definition:** Compare self-hosted GPU cost/latency/capacity against API pricing for the same input/output/workload distribution.
-   144→- **User value:** Directly answers whether utilization, latency/SLO, privacy, or operational complexity justifies owning/renting GPUs.
-   145→- **Complexity / priority:** Medium / Must-have.
-   146→- **Formula implications:** Self-host cost/request = `(GPU_hourly_cost × wall_time + CPU/RAM/egress/ops amortization)/requests`; per-million-token cost = hourly_cost / effective_tokens_per_hour × 1e6. API cost = uncached input×input_rate + cached input×cached_rate + output×output_rate + tool/media charges. Break-even volume solves self-host fixed+variable = API variable.
-   147→- **Required knobs:** utilization, concurrency, TTFT/TPOT SLO, prompt/output distributions, cache hit rate, quantization, replicas/HA, region/egress, idle/cold-start time, and API batch discounts.
-   148→- **Caveats:** Prices change quickly and throughput benchmarks are not apples-to-apples; link every provider/model price and timestamp it. No universal demand statistic was verified, but this feature follows directly from the cost equations and existing tokcalc scope.
-   149→
-   150→## 13. Energy / carbon per million tokens
-   151→- **Definition:** Attribute measured/estimated electricity and regional carbon intensity to prefill/decode work, then normalize by tokens.
-   152→- **User value:** Sustainability reporting and energy-aware model/GPU selection; helps expose long-context/reasoning cost not visible in dollars.
-   153→- **Complexity / priority:** Medium–High / Nice.
-   154→- **Quantitative evidence:** A 2025 infrastructure-aware benchmark gives `Carbon (kgCO2e) = E_query × CIF` and separates prompt-level operational footprint; it reports ~1.14 gCO2e per 400-token query for Mistral Large 2 as aligned with Mistral's LCA, but this is workload/region-specific. [Hungry AI paper](https://arxiv.org/html/2505.09598v6) [Mistral LCA reference is linked in paper]
-   155→- CodeCarbon measures/estimates CPU, GPU, and RAM electricity and applies regional carbon intensity. [CodeCarbon](https://github.com/mlco2/codecarbon)
-   156→- **Formula implications:** `E_Wh = ∫ power_W dt / 3600`; `gCO2e = E_Wh × carbon_intensity_g/Wh × PUE`; report `gCO2e / 1e6 output_tokens` and optionally input/output separately. Include facility overhead and region as explicit fields.
-   157→- **Caveats:** API providers rarely publish per-request power; GPU TDP is not measured draw; utilization, batching, PUE, cooling, and grid mix dominate. I found no stable universal gCO2e/M-token constant.
-   158→
-   159→## 14. Serverless GPU cold starts
-   160→- **Definition:** Time from scale-to-zero request to a ready model worker/first token, including container/image pull, library init, weight transfer, CUDA graph capture, and model warmup.
-   161→- **User value:** Quantify tail latency and economics for bursty workloads; compare warm pool, snapshot, quantization, and scale-to-zero strategies.
-   162→- **Complexity / priority:** Medium / Nice.
-   163→- **Quantitative evidence:** Modal says model weights can load from a Volume at 1–2 GB/s, roughly nearly 1 second of cold-start latency per GB of weights; its high-performance guide reports GPU memory snapshots can yield 10x cold-start reductions. [Modal high-performance LLM inference](https://modal.com/docs/guide/high-performance-llm-inference)
-   164→- Modal's Ministral 3 test reports median cold start falling from ~118s to ~12s with GPU snapshotting. [Modal blog](https://modal.com/blog/mistral-3)
-   165→- RunPod's vLLM cold-start guide says uncached image pulls can take 30–90s; official RunPod materials advertise sub-200ms FlashBoot for active endpoints, which should be treated as a warm/snapshot path, not a generic model boot. [RunPod guide](https://www.runpod.io/blog/cut-vllm-cold-starts-runpod-serverless) [RunPod product](https://www.runpod.io/product/serverless)
-   166→- **Formula implications:** `cold_start ≈ image_pull + container_init + weight_load(model_GB / effective_GBps) + GPU_load + compile/capture + warmup`; amortize over requests per warm interval. Add p50/p95/p99, warm-worker fraction, snapshot support, model size, and provider.
-   167→- **Caveats:** Marketing numbers are platform/configuration-specific; exact cold-start distributions for Replicate/RunPod/Modal vary by region, cached image/model, GPU availability, and model. Treat every provider value as timestamped benchmark input, not a constant.
-   168→
-   169→## Cross-cutting implementation schema
-   170→Store per feature/model/provider: source URL, benchmark date, hardware, engine/version, model/quantization, prompt/output distributions, batch/concurrency, QPS, metric (TTFT/TPOT/ITL/throughput/cost/energy), and confidence. Keep measured benchmark multipliers separate from analytical estimates.
-   171→
-   172→## Gaps / failed attempts
-   173→- vLLM docs and vendor pages were readable, but several arXiv abstract pages and older dynamic docs timed out in extraction; equivalent official pages or arXiv results were used where content was available.
-   174→- No authoritative universal chunked-prefill speedup, agent tool-call latency multiplier, long-context 1M throughput multiplier, embedding market-demand figure, or universal energy/token constant was verified.
-   175→- OpenAI's public marketing pricing page returned 403 in direct fetch; the official developer pricing/prompt-caching docs were available and should be used for current model rates.
-   176→
-   177→
-   178→---
-   179→
-   180→# Tokcalc Section B — implementation-ready model catalog
-   181→
-   182→**Cutoff:** 2026-09-20. **Status:** best effort within the declared HF/API universe; not exhaustive of every community checkpoint. Architecture fields are taken from direct config JSON when readable. HF `createdAt` is a repository-date proxy, not necessarily announcement date. `n/a` means not disclosed or inaccessible; it must remain null in the calculator.
-   183→
-   184→## Open LLM, coding, and reasoning rows
-   185→
-   186→|Official HF model|Family|Total / active params|Layers|Hidden|Q / KV heads|Head dim|Vocab|Context|Release/date proxy|Sources|
-   187→|---|---|---:|---:|---:|---:|---:|---:|---:|---|---|
-   188→|[meta-llama/Llama-4-Scout-17B-16E-Instruct](https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct)|Llama 4|[~109B / 17B](https://huggingface.co/blog/llama4-release)|n/a|n/a|n/a / n/a|n/a|n/a|[10M advertised](https://developer.meta.com/ai/models/llama-4/)|[2025-04-05](https://huggingface.co/api/models/meta-llama/Llama-4-Scout-17B-16E-Instruct)|[card](https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct) · [config](https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct/raw/main/config.json)|
-   189→|[meta-llama/Llama-4-Maverick-17B-128E-Instruct](https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct)|Llama 4|[~400B / 17B](https://huggingface.co/blog/llama4-release)|n/a|n/a|n/a / n/a|n/a|n/a|[1M advertised](https://developer.meta.com/ai/models/llama-4/)|[2025-04-05](https://huggingface.co/api/models/meta-llama/Llama-4-Maverick-17B-128E-Instruct)|[card](https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct) · [config](https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct/raw/main/config.json)|
-   190→|[meta-llama/Llama-4-Behemoth-17B-16E-Instruct](https://huggingface.co/meta-llama/Llama-4-Behemoth-17B-16E-Instruct)|Llama 4|[~2T / 288B](https://ai.meta.com/blog/llama-4-multimodal-intelligence)|n/a|n/a|n/a / n/a|n/a|n/a|n/a|[2025-04-05 announced](https://huggingface.co/api/models/meta-llama/Llama-4-Behemoth-17B-16E-Instruct)|[card](https://huggingface.co/meta-llama/Llama-4-Behemoth-17B-16E-Instruct) · [config](https://huggingface.co/meta-llama/Llama-4-Behemoth-17B-16E-Instruct/raw/main/config.json)|
-   191→|[Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B)|Qwen 3|[0.6B](https://huggingface.co/api/models/Qwen/Qwen3-0.6B)|[28](https://huggingface.co/Qwen/Qwen3-0.6B/raw/main/config.json)|[1024](https://huggingface.co/Qwen/Qwen3-0.6B/raw/main/config.json)|[16](https://huggingface.co/Qwen/Qwen3-0.6B/raw/main/config.json) / [8](https://huggingface.co/Qwen/Qwen3-0.6B/raw/main/config.json)|[128](https://huggingface.co/Qwen/Qwen3-0.6B/raw/main/config.json)|[151936](https://huggingface.co/Qwen/Qwen3-0.6B/raw/main/config.json)|[40960](https://huggingface.co/Qwen/Qwen3-0.6B/raw/main/config.json)|[2025-04-27](https://huggingface.co/api/models/Qwen/Qwen3-0.6B)|[card](https://huggingface.co/Qwen/Qwen3-0.6B) · [config](https://huggingface.co/Qwen/Qwen3-0.6B/raw/main/config.json)|
-   192→|[Qwen/Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B)|Qwen 3|[4B](https://huggingface.co/api/models/Qwen/Qwen3-4B)|[36](https://huggingface.co/Qwen/Qwen3-4B/raw/main/config.json)|[2560](https://huggingface.co/Qwen/Qwen3-4B/raw/main/config.json)|[32](https://huggingface.co/Qwen/Qwen3-4B/raw/main/config.json) / [8](https://huggingface.co/Qwen/Qwen3-4B/raw/main/config.json)|[128](https://huggingface.co/Qwen/Qwen3-4B/raw/main/config.json)|[151936](https://huggingface.co/Qwen/Qwen3-4B/raw/main/config.json)|[40960](https://huggingface.co/Qwen/Qwen3-4B/raw/main/config.json)|[2025-04-27](https://huggingface.co/api/models/Qwen/Qwen3-4B)|[card](https://huggingface.co/Qwen/Qwen3-4B) · [config](https://huggingface.co/Qwen/Qwen3-4B/raw/main/config.json)|
-   193→|[Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B)|Qwen 3|[8B](https://huggingface.co/api/models/Qwen/Qwen3-8B)|[36](https://huggingface.co/Qwen/Qwen3-8B/raw/main/config.json)|[4096](https://huggingface.co/Qwen/Qwen3-8B/raw/main/config.json)|[32](https://huggingface.co/Qwen/Qwen3-8B/raw/main/config.json) / [8](https://huggingface.co/Qwen/Qwen3-8B/raw/main/config.json)|[128](https://huggingface.co/Qwen/Qwen3-8B/raw/main/config.json)|[151936](https://huggingface.co/Qwen/Qwen3-8B/raw/main/config.json)|[40960](https://huggingface.co/Qwen/Qwen3-8B/raw/main/config.json)|[2025-04-27](https://huggingface.co/api/models/Qwen/Qwen3-8B)|[card](https://huggingface.co/Qwen/Qwen3-8B) · [config](https://huggingface.co/Qwen/Qwen3-8B/raw/main/config.json)|
-   194→|[Qwen/Qwen3-14B](https://huggingface.co/Qwen/Qwen3-14B)|Qwen 3|[14B](https://huggingface.co/api/models/Qwen/Qwen3-14B)|[40](https://huggingface.co/Qwen/Qwen3-14B/raw/main/config.json)|[5120](https://huggingface.co/Qwen/Qwen3-14B/raw/main/config.json)|[40](https://huggingface.co/Qwen/Qwen3-14B/raw/main/config.json) / [8](https://huggingface.co/Qwen/Qwen3-14B/raw/main/config.json)|[128](https://huggingface.co/Qwen/Qwen3-14B/raw/main/config.json)|[151936](https://huggingface.co/Qwen/Qwen3-14B/raw/main/config.json)|[40960](https://huggingface.co/Qwen/Qwen3-14B/raw/main/config.json)|[2025-04-27](https://huggingface.co/api/models/Qwen/Qwen3-14B)|[card](https://huggingface.co/Qwen/Qwen3-14B) · [config](https://huggingface.co/Qwen/Qwen3-14B/raw/main/config.json)|
-   195→|[Qwen/Qwen3-32B](https://huggingface.co/Qwen/Qwen3-32B)|Qwen 3|[32B](https://huggingface.co/api/models/Qwen/Qwen3-32B)|[64](https://huggingface.co/Qwen/Qwen3-32B/raw/main/config.json)|[5120](https://huggingface.co/Qwen/Qwen3-32B/raw/main/config.json)|[64](https://huggingface.co/Qwen/Qwen3-32B/raw/main/config.json) / [8](https://huggingface.co/Qwen/Qwen3-32B/raw/main/config.json)|[128](https://huggingface.co/Qwen/Qwen3-32B/raw/main/config.json)|[151936](https://huggingface.co/Qwen/Qwen3-32B/raw/main/config.json)|[40960](https://huggingface.co/Qwen/Qwen3-32B/raw/main/config.json)|[2025-04-27](https://huggingface.co/api/models/Qwen/Qwen3-32B)|[card](https://huggingface.co/Qwen/Qwen3-32B) · [config](https://huggingface.co/Qwen/Qwen3-32B/raw/main/config.json)|
-   196→|[Qwen/Qwen3-30B-A3B](https://huggingface.co/Qwen/Qwen3-30B-A3B)|Qwen 3|[30B](https://huggingface.co/api/models/Qwen/Qwen3-30B-A3B)|[48](https://huggingface.co/Qwen/Qwen3-30B-A3B/raw/main/config.json)|[2048](https://huggingface.co/Qwen/Qwen3-30B-A3B/raw/main/config.json)|[32](https://huggingface.co/Qwen/Qwen3-30B-A3B/raw/main/config.json) / [4](https://huggingface.co/Qwen/Qwen3-30B-A3B/raw/main/config.json)|[128](https://huggingface.co/Qwen/Qwen3-30B-A3B/raw/main/config.json)|[151936](https://huggingface.co/Qwen/Qwen3-30B-A3B/raw/main/config.json)|[40960](https://huggingface.co/Qwen/Qwen3-30B-A3B/raw/main/config.json)|[2025-04-27](https://huggingface.co/api/models/Qwen/Qwen3-30B-A3B)|[card](https://huggingface.co/Qwen/Qwen3-30B-A3B) · [config](https://huggingface.co/Qwen/Qwen3-30B-A3B/raw/main/config.json)|
-   197→|[Qwen/Qwen3-235B-A22B](https://huggingface.co/Qwen/Qwen3-235B-A22B)|Qwen 3|[235B](https://huggingface.co/api/models/Qwen/Qwen3-235B-A22B)|[94](https://huggingface.co/Qwen/Qwen3-235B-A22B/raw/main/config.json)|[4096](https://huggingface.co/Qwen/Qwen3-235B-A22B/raw/main/config.json)|[64](https://huggingface.co/Qwen/Qwen3-235B-A22B/raw/main/config.json) / [4](https://huggingface.co/Qwen/Qwen3-235B-A22B/raw/main/config.json)|[128](https://huggingface.co/Qwen/Qwen3-235B-A22B/raw/main/config.json)|[151936](https://huggingface.co/Qwen/Qwen3-235B-A22B/raw/main/config.json)|[40960](https://huggingface.co/Qwen/Qwen3-235B-A22B/raw/main/config.json)|[2025-04-27](https://huggingface.co/api/models/Qwen/Qwen3-235B-A22B)|[card](https://huggingface.co/Qwen/Qwen3-235B-A22B) · [config](https://huggingface.co/Qwen/Qwen3-235B-A22B/raw/main/config.json)|
-   198→|[Qwen/Qwen2.5-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct)|Qwen 2.5-VL|[7B](https://huggingface.co/api/models/Qwen/Qwen2.5-VL-7B-Instruct)|[28](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct/raw/main/config.json)|[3584](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct/raw/main/config.json)|[28](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct/raw/main/config.json) / [4](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct/raw/main/config.json)|n/a|[152064](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct/raw/main/config.json)|[128000](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct/raw/main/config.json)|[2025-01-26](https://huggingface.co/api/models/Qwen/Qwen2.5-VL-7B-Instruct)|[card](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) · [config](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct/raw/main/config.json)|
-   199→|[Qwen/Qwen2.5-VL-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct)|Qwen 2.5-VL|[32B](https://huggingface.co/api/models/Qwen/Qwen2.5-VL-32B-Instruct)|[64](https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct/raw/main/config.json)|[5120](https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct/raw/main/config.json)|[40](https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct/raw/main/config.json) / [8](https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct/raw/main/config.json)|n/a|[152064](https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct/raw/main/config.json)|[128000](https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct/raw/main/config.json)|[2025-03-21](https://huggingface.co/api/models/Qwen/Qwen2.5-VL-32B-Instruct)|[card](https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct) · [config](https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct/raw/main/config.json)|
-   200→|[Qwen/Qwen2.5-VL-72B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct)|Qwen 2.5-VL|[72B](https://huggingface.co/api/models/Qwen/Qwen2.5-VL-72B-Instruct)|[80](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct/raw/main/config.json)|[8192](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct/raw/main/config.json)|[64](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct/raw/main/config.json) / [8](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct/raw/main/config.json)|n/a|[152064](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct/raw/main/config.json)|[128000](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct/raw/main/config.json)|[2025-01-27](https://huggingface.co/api/models/Qwen/Qwen2.5-VL-72B-Instruct)|[card](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct) · [config](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct/raw/main/config.json)|
-   201→|[Qwen/Qwen2.5-MoE-A2.7B](https://huggingface.co/Qwen/Qwen2.5-MoE-A2.7B)|Other|[n/a](https://huggingface.co/api/models/Qwen/Qwen2.5-MoE-A2.7B)|n/a|n/a|n/a / n/a|n/a|n/a|n/a|n/a|[card](https://huggingface.co/Qwen/Qwen2.5-MoE-A2.7B) · [config](https://huggingface.co/Qwen/Qwen2.5-MoE-A2.7B/raw/main/config.json)|
-   202→|[Qwen/Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct)|Qwen 2.5 Coder|[7B](https://huggingface.co/api/models/Qwen/Qwen2.5-Coder-7B-Instruct)|[28](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct/raw/main/config.json)|[3584](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct/raw/main/config.json)|[28](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct/raw/main/config.json) / [4](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct/raw/main/config.json)|n/a|[152064](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct/raw/main/config.json)|[32768](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct/raw/main/config.json)|[2024-09-17](https://huggingface.co/api/models/Qwen/Qwen2.5-Coder-7B-Instruct)|[card](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct) · [config](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct/raw/main/config.json)|
-   203→|[Qwen/Qwen2.5-Coder-14B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-14B-Instruct)|Qwen 2.5 Coder|[14B](https://huggingface.co/api/models/Qwen/Qwen2.5-Coder-14B-Instruct)|[48](https://huggingface.co/Qwen/Qwen2.5-Coder-14B-Instruct/raw/main/config.json)|[5120](https://huggingface.co/Qwen/Qwen2.5-Coder-14B-Instruct/raw/main/config.json)|[40](https://huggingface.co/Qwen/Qwen2.5-Coder-14B-Instruct/raw/main/config.json) / [8](https://huggingface.co/Qwen/Qwen2.5-Coder-14B-Instruct/raw/main/config.json)|n/a|[152064](https://huggingface.co/Qwen/Qwen2.5-Coder-14B-Instruct/raw/main/config.json)|[32768](https://huggingface.co/Qwen/Qwen2.5-Coder-14B-Instruct/raw/main/config.json)|[2024-11-06](https://huggingface.co/api/models/Qwen/Qwen2.5-Coder-14B-Instruct)|[card](https://huggingface.co/Qwen/Qwen2.5-Coder-14B-Instruct) · [config](https://huggingface.co/Qwen/Qwen2.5-Coder-14B-Instruct/raw/main/config.json)|
-   204→|[Qwen/Qwen2.5-Coder-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct)|Qwen 2.5 Coder|[32B](https://huggingface.co/api/models/Qwen/Qwen2.5-Coder-32B-Instruct)|[64](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct/raw/main/config.json)|[5120](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct/raw/main/config.json)|[40](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct/raw/main/config.json) / [8](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct/raw/main/config.json)|n/a|[152064](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct/raw/main/config.json)|[32768](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct/raw/main/config.json)|[2024-11-06](https://huggingface.co/api/models/Qwen/Qwen2.5-Coder-32B-Instruct)|[card](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct) · [config](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct/raw/main/config.json)|
-   205→|[deepseek-ai/DeepSeek-V3](https://huggingface.co/deepseek-ai/DeepSeek-V3)|DeepSeek|[671B](https://huggingface.co/api/models/deepseek-ai/DeepSeek-V3)|[61](https://huggingface.co/deepseek-ai/DeepSeek-V3/raw/main/config.json)|[7168](https://huggingface.co/deepseek-ai/DeepSeek-V3/raw/main/config.json)|[128](https://huggingface.co/deepseek-ai/DeepSeek-V3/raw/main/config.json) / [128](https://huggingface.co/deepseek-ai/DeepSeek-V3/raw/main/config.json)|n/a|[129280](https://huggingface.co/deepseek-ai/DeepSeek-V3/raw/main/config.json)|[163840](https://huggingface.co/deepseek-ai/DeepSeek-V3/raw/main/config.json)|[2024-12-25](https://huggingface.co/api/models/deepseek-ai/DeepSeek-V3)|[card](https://huggingface.co/deepseek-ai/DeepSeek-V3) · [config](https://huggingface.co/deepseek-ai/DeepSeek-V3/raw/main/config.json)|
-   206→|[deepseek-ai/DeepSeek-V3-0324](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324)|DeepSeek|[671B](https://huggingface.co/api/models/deepseek-ai/DeepSeek-V3-0324)|[61](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/raw/main/config.json)|[7168](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/raw/main/config.json)|[128](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/raw/main/config.json) / [128](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/raw/main/config.json)|n/a|[129280](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/raw/main/config.json)|[163840](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/raw/main/config.json)|[2025-03-24](https://huggingface.co/api/models/deepseek-ai/DeepSeek-V3-0324)|[card](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324) · [config](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/raw/main/config.json)|
-   207→|[deepseek-ai/DeepSeek-R1](https://huggingface.co/deepseek-ai/DeepSeek-R1)|DeepSeek|[671B](https://huggingface.co/api/models/deepseek-ai/DeepSeek-R1)|[61](https://huggingface.co/deepseek-ai/DeepSeek-R1/raw/main/config.json)|[7168](https://huggingface.co/deepseek-ai/DeepSeek-R1/raw/main/config.json)|[128](https://huggingface.co/deepseek-ai/DeepSeek-R1/raw/main/config.json) / [128](https://huggingface.co/deepseek-ai/DeepSeek-R1/raw/main/config.json)|n/a|[129280](https://huggingface.co/deepseek-ai/DeepSeek-R1/raw/main/config.json)|[163840](https://huggingface.co/deepseek-ai/DeepSeek-R1/raw/main/config.json)|[2025-01-20](https://huggingface.co/api/models/deepseek-ai/DeepSeek-R1)|[card](https://huggingface.co/deepseek-ai/DeepSeek-R1) · [config](https://huggingface.co/deepseek-ai/DeepSeek-R1/raw/main/config.json)|
-   208→|[deepseek-ai/DeepSeek-R1-0528](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528)|DeepSeek|[671B](https://huggingface.co/api/models/deepseek-ai/DeepSeek-R1-0528)|[61](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528/raw/main/config.json)|[7168](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528/raw/main/config.json)|[128](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528/raw/main/config.json) / [128](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528/raw/main/config.json)|n/a|[129280](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528/raw/main/config.json)|[163840](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528/raw/main/config.json)|[2025-05-28](https://huggingface.co/api/models/deepseek-ai/DeepSeek-R1-0528)|[card](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528) · [config](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528/raw/main/config.json)|
-   209→|[deepseek-ai/DeepSeek-Coder-V2-Instruct](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct)|DeepSeek Coder V2|[236B](https://huggingface.co/api/models/deepseek-ai/DeepSeek-Coder-V2-Instruct)|[60](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct/raw/main/config.json)|[5120](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct/raw/main/config.json)|[128](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct/raw/main/config.json) / [128](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct/raw/main/config.json)|n/a|[102400](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct/raw/main/config.json)|[163840](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct/raw/main/config.json)|[2024-06-14](https://huggingface.co/api/models/deepseek-ai/DeepSeek-Coder-V2-Instruct)|[card](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct) · [config](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct/raw/main/config.json)|
-   210→|[mistralai/Mistral-Large-Instruct-2411](https://huggingface.co/mistralai/Mistral-Large-Instruct-2411)|Mistral|[122.6B](https://huggingface.co/api/models/mistralai/Mistral-Large-Instruct-2411)|[88](https://huggingface.co/mistralai/Mistral-Large-Instruct-2411/raw/main/config.json)|[12288](https://huggingface.co/mistralai/Mistral-Large-Instruct-2411/raw/main/config.json)|[96](https://huggingface.co/mistralai/Mistral-Large-Instruct-2411/raw/main/config.json) / [8](https://huggingface.co/mistralai/Mistral-Large-Instruct-2411/raw/main/config.json)|[128](https://huggingface.co/mistralai/Mistral-Large-Instruct-2411/raw/main/config.json)|[32768](https://huggingface.co/mistralai/Mistral-Large-Instruct-2411/raw/main/config.json)|[131072](https://huggingface.co/mistralai/Mistral-Large-Instruct-2411/raw/main/config.json)|[2024-11-14](https://huggingface.co/api/models/mistralai/Mistral-Large-Instruct-2411)|[card](https://huggingface.co/mistralai/Mistral-Large-Instruct-2411) · [config](https://huggingface.co/mistralai/Mistral-Large-Instruct-2411/raw/main/config.json)|
-   211→|[mistralai/Codestral-22B-v0.1](https://huggingface.co/mistralai/Codestral-22B-v0.1)|Mistral|[22.2B](https://huggingface.co/api/models/mistralai/Codestral-22B-v0.1)|[56](https://huggingface.co/mistralai/Codestral-22B-v0.1/raw/main/config.json)|[6144](https://huggingface.co/mistralai/Codestral-22B-v0.1/raw/main/config.json)|[48](https://huggingface.co/mistralai/Codestral-22B-v0.1/raw/main/config.json) / [8](https://huggingface.co/mistralai/Codestral-22B-v0.1/raw/main/config.json)|n/a|[32768](https://huggingface.co/mistralai/Codestral-22B-v0.1/raw/main/config.json)|[32768](https://huggingface.co/mistralai/Codestral-22B-v0.1/raw/main/config.json)|[2024-05-29](https://huggingface.co/api/models/mistralai/Codestral-22B-v0.1)|[card](https://huggingface.co/mistralai/Codestral-22B-v0.1) · [config](https://huggingface.co/mistralai/Codestral-22B-v0.1/raw/main/config.json)|
-   212→|[mistralai/Ministral-3B-Instruct-2410](https://huggingface.co/mistralai/Ministral-3B-Instruct-2410)|Mistral|[3B](https://huggingface.co/api/models/mistralai/Ministral-3B-Instruct-2410)|n/a|n/a|n/a / n/a|n/a|n/a|n/a|[2024-10; repo 401](https://huggingface.co/api/models/mistralai/Ministral-3B-Instruct-2410)|[card](https://huggingface.co/mistralai/Ministral-3B-Instruct-2410) · [config](https://huggingface.co/mistralai/Ministral-3B-Instruct-2410/raw/main/config.json)|
-   213→|[mistralai/Ministral-8B-Instruct-2410](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410)|Mistral|[8.0B](https://huggingface.co/api/models/mistralai/Ministral-8B-Instruct-2410)|[36](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410/raw/main/config.json)|[4096](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410/raw/main/config.json)|[32](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410/raw/main/config.json) / [8](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410/raw/main/config.json)|[128](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410/raw/main/config.json)|[131072](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410/raw/main/config.json)|[32768](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410/raw/main/config.json)|[2024-10-15](https://huggingface.co/api/models/mistralai/Ministral-8B-Instruct-2410)|[card](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410) · [config](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410/raw/main/config.json)|
-   214→|[mistralai/Pixtral-12B-2409](https://huggingface.co/mistralai/Pixtral-12B-2409)|Pixtral|[12B](https://huggingface.co/api/models/mistralai/Pixtral-12B-2409)|n/a|n/a|n/a / n/a|n/a|n/a|[128K advertised](https://huggingface.co/mistralai/Pixtral-12B-2409/raw/main/config.json)|[2024-09](https://huggingface.co/api/models/mistralai/Pixtral-12B-2409)|[card](https://huggingface.co/mistralai/Pixtral-12B-2409) · [config](https://huggingface.co/mistralai/Pixtral-12B-2409/raw/main/config.json)|
-   215→|[google/gemma-3-1b-it](https://huggingface.co/google/gemma-3-1b-it)|Gemma 3|[1B](https://huggingface.co/api/models/google/gemma-3-1b-it)|n/a|n/a|n/a / n/a|n/a|n/a|[32K](https://huggingface.co/google/gemma-3-1b-it/raw/main/config.json)|[2025-03](https://huggingface.co/api/models/google/gemma-3-1b-it)|[card](https://huggingface.co/google/gemma-3-1b-it) · [config](https://huggingface.co/google/gemma-3-1b-it/raw/main/config.json)|
-   216→|[google/gemma-3-4b-it](https://huggingface.co/google/gemma-3-4b-it)|Gemma 3|[4B](https://huggingface.co/api/models/google/gemma-3-4b-it)|n/a|n/a|n/a / n/a|n/a|n/a|[128K](https://huggingface.co/google/gemma-3-4b-it/raw/main/config.json)|[2025-03](https://huggingface.co/api/models/google/gemma-3-4b-it)|[card](https://huggingface.co/google/gemma-3-4b-it) · [config](https://huggingface.co/google/gemma-3-4b-it/raw/main/config.json)|
-   217→|[google/gemma-3-12b-it](https://huggingface.co/google/gemma-3-12b-it)|Gemma 3|[12B](https://huggingface.co/api/models/google/gemma-3-12b-it)|n/a|n/a|n/a / n/a|n/a|n/a|[128K](https://huggingface.co/google/gemma-3-12b-it/raw/main/config.json)|[2025-03](https://huggingface.co/api/models/google/gemma-3-12b-it)|[card](https://huggingface.co/google/gemma-3-12b-it) · [config](https://huggingface.co/google/gemma-3-12b-it/raw/main/config.json)|
-   218→|[google/gemma-3-27b-it](https://huggingface.co/google/gemma-3-27b-it)|Gemma 3|[27B](https://huggingface.co/api/models/google/gemma-3-27b-it)|n/a|n/a|n/a / n/a|n/a|n/a|[128K](https://huggingface.co/google/gemma-3-27b-it/raw/main/config.json)|[2025-03](https://huggingface.co/api/models/google/gemma-3-27b-it)|[card](https://huggingface.co/google/gemma-3-27b-it) · [config](https://huggingface.co/google/gemma-3-27b-it/raw/main/config.json)|
-   219→|[google/gemma-2-2b-it](https://huggingface.co/google/gemma-2-2b-it)|Gemma 2|[2B](https://huggingface.co/api/models/google/gemma-2-2b-it)|n/a|n/a|n/a / n/a|n/a|n/a|[8K](https://huggingface.co/google/gemma-2-2b-it/raw/main/config.json)|[2024-07](https://huggingface.co/api/models/google/gemma-2-2b-it)|[card](https://huggingface.co/google/gemma-2-2b-it) · [config](https://huggingface.co/google/gemma-2-2b-it/raw/main/config.json)|
-   220→|[google/gemma-2-9b-it](https://huggingface.co/google/gemma-2-9b-it)|Gemma 2|[9B](https://huggingface.co/api/models/google/gemma-2-9b-it)|n/a|n/a|n/a / n/a|n/a|n/a|[8K](https://huggingface.co/google/gemma-2-9b-it/raw/main/config.json)|[2024-06](https://huggingface.co/api/models/google/gemma-2-9b-it)|[card](https://huggingface.co/google/gemma-2-9b-it) · [config](https://huggingface.co/google/gemma-2-9b-it/raw/main/config.json)|
-   221→|[google/gemma-2-27b-it](https://huggingface.co/google/gemma-2-27b-it)|Gemma 2|[27B](https://huggingface.co/api/models/google/gemma-2-27b-it)|n/a|n/a|n/a / n/a|n/a|n/a|[8K](https://huggingface.co/google/gemma-2-27b-it/raw/main/config.json)|[2024-06](https://huggingface.co/api/models/google/gemma-2-27b-it)|[card](https://huggingface.co/google/gemma-2-27b-it) · [config](https://huggingface.co/google/gemma-2-27b-it/raw/main/config.json)|
-   222→|[microsoft/phi-4](https://huggingface.co/microsoft/phi-4)|Phi|[14.7B](https://huggingface.co/api/models/microsoft/phi-4)|[40](https://huggingface.co/microsoft/phi-4/raw/main/config.json)|[5120](https://huggingface.co/microsoft/phi-4/raw/main/config.json)|[40](https://huggingface.co/microsoft/phi-4/raw/main/config.json) / [10](https://huggingface.co/microsoft/phi-4/raw/main/config.json)|n/a|[100352](https://huggingface.co/microsoft/phi-4/raw/main/config.json)|[16384](https://huggingface.co/microsoft/phi-4/raw/main/config.json)|[2024-12-11](https://huggingface.co/api/models/microsoft/phi-4)|[card](https://huggingface.co/microsoft/phi-4) · [config](https://huggingface.co/microsoft/phi-4/raw/main/config.json)|
-   223→|[microsoft/Phi-4-mini-instruct](https://huggingface.co/microsoft/Phi-4-mini-instruct)|Phi|[3.8B](https://huggingface.co/api/models/microsoft/Phi-4-mini-instruct)|[32](https://huggingface.co/microsoft/Phi-4-mini-instruct/raw/main/config.json)|[3072](https://huggingface.co/microsoft/Phi-4-mini-instruct/raw/main/config.json)|[24](https://huggingface.co/microsoft/Phi-4-mini-instruct/raw/main/config.json) / [8](https://huggingface.co/microsoft/Phi-4-mini-instruct/raw/main/config.json)|n/a|[200064](https://huggingface.co/microsoft/Phi-4-mini-instruct/raw/main/config.json)|[131072](https://huggingface.co/microsoft/Phi-4-mini-instruct/raw/main/config.json)|[2025-02-19](https://huggingface.co/api/models/microsoft/Phi-4-mini-instruct)|[card](https://huggingface.co/microsoft/Phi-4-mini-instruct) · [config](https://huggingface.co/microsoft/Phi-4-mini-instruct/raw/main/config.json)|
-   224→|[microsoft/Phi-3.5-mini-instruct](https://huggingface.co/microsoft/Phi-3.5-mini-instruct)|Phi|[3.8B](https://huggingface.co/api/models/microsoft/Phi-3.5-mini-instruct)|[32](https://huggingface.co/microsoft/Phi-3.5-mini-instruct/raw/main/config.json)|[3072](https://huggingface.co/microsoft/Phi-3.5-mini-instruct/raw/main/config.json)|[32](https://huggingface.co/microsoft/Phi-3.5-mini-instruct/raw/main/config.json) / [32](https://huggingface.co/microsoft/Phi-3.5-mini-instruct/raw/main/config.json)|n/a|[32064](https://huggingface.co/microsoft/Phi-3.5-mini-instruct/raw/main/config.json)|[131072](https://huggingface.co/microsoft/Phi-3.5-mini-instruct/raw/main/config.json)|[2024-08-16](https://huggingface.co/api/models/microsoft/Phi-3.5-mini-instruct)|[card](https://huggingface.co/microsoft/Phi-3.5-mini-instruct) · [config](https://huggingface.co/microsoft/Phi-3.5-mini-instruct/raw/main/config.json)|
-   225→|[microsoft/Phi-3.5-MoE-instruct](https://huggingface.co/microsoft/Phi-3.5-MoE-instruct)|Phi|[41.9B](https://huggingface.co/api/models/microsoft/Phi-3.5-MoE-instruct)|[32](https://huggingface.co/microsoft/Phi-3.5-MoE-instruct/raw/main/config.json)|[4096](https://huggingface.co/microsoft/Phi-3.5-MoE-instruct/raw/main/config.json)|[32](https://huggingface.co/microsoft/Phi-3.5-MoE-instruct/raw/main/config.json) / [8](https://huggingface.co/microsoft/Phi-3.5-MoE-instruct/raw/main/config.json)|n/a|[32064](https://huggingface.co/microsoft/Phi-3.5-MoE-instruct/raw/main/config.json)|[131072](https://huggingface.co/microsoft/Phi-3.5-MoE-instruct/raw/main/config.json)|[2024-08-17](https://huggingface.co/api/models/microsoft/Phi-3.5-MoE-instruct)|[card](https://huggingface.co/microsoft/Phi-3.5-MoE-instruct) · [config](https://huggingface.co/microsoft/Phi-3.5-MoE-instruct/raw/main/config.json)|
-   226→|[WizardLM/WizardLM-2-8x22B](https://huggingface.co/WizardLM/WizardLM-2-8x22B)|WizardLM|[141B](https://huggingface.co/api/models/WizardLM/WizardLM-2-8x22B)|n/a|n/a|n/a / n/a|n/a|n/a|n/a|[2024-04](https://huggingface.co/api/models/WizardLM/WizardLM-2-8x22B)|[card](https://huggingface.co/WizardLM/WizardLM-2-8x22B) · [config](https://huggingface.co/WizardLM/WizardLM-2-8x22B/raw/main/config.json)|
-   227→|[microsoft/Orca-2-13b](https://huggingface.co/microsoft/Orca-2-13b)|Orca|[n/a](https://huggingface.co/api/models/microsoft/Orca-2-13b)|[40](https://huggingface.co/microsoft/Orca-2-13b/raw/main/config.json)|[5120](https://huggingface.co/microsoft/Orca-2-13b/raw/main/config.json)|[40](https://huggingface.co/microsoft/Orca-2-13b/raw/main/config.json) / [40](https://huggingface.co/microsoft/Orca-2-13b/raw/main/config.json)|n/a|[32003](https://huggingface.co/microsoft/Orca-2-13b/raw/main/config.json)|[4096](https://huggingface.co/microsoft/Orca-2-13b/raw/main/config.json)|[2023-11-14](https://huggingface.co/api/models/microsoft/Orca-2-13b)|[card](https://huggingface.co/microsoft/Orca-2-13b) · [config](https://huggingface.co/microsoft/Orca-2-13b/raw/main/config.json)|
-   228→|[microsoft/BitNet-b1.58-2B-4T](https://huggingface.co/microsoft/BitNet-b1.58-2B-4T)|BitNet|[850M](https://huggingface.co/api/models/microsoft/BitNet-b1.58-2B-4T)|[30](https://huggingface.co/microsoft/BitNet-b1.58-2B-4T/raw/main/config.json)|[2560](https://huggingface.co/microsoft/BitNet-b1.58-2B-4T/raw/main/config.json)|[20](https://huggingface.co/microsoft/BitNet-b1.58-2B-4T/raw/main/config.json) / [5](https://huggingface.co/microsoft/BitNet-b1.58-2B-4T/raw/main/config.json)|n/a|[128256](https://huggingface.co/microsoft/BitNet-b1.58-2B-4T/raw/main/config.json)|[4096](https://huggingface.co/microsoft/BitNet-b1.58-2B-4T/raw/main/config.json)|[2025-04-15](https://huggingface.co/api/models/microsoft/BitNet-b1.58-2B-4T)|[card](https://huggingface.co/microsoft/BitNet-b1.58-2B-4T) · [config](https://huggingface.co/microsoft/BitNet-b1.58-2B-4T/raw/main/config.json)|
-   229→|[HuggingFaceTB/SmolLM2-135M](https://huggingface.co/HuggingFaceTB/SmolLM2-135M)|SmolLM2|[135M](https://huggingface.co/api/models/HuggingFaceTB/SmolLM2-135M)|[30](https://huggingface.co/HuggingFaceTB/SmolLM2-135M/raw/main/config.json)|[576](https://huggingface.co/HuggingFaceTB/SmolLM2-135M/raw/main/config.json)|[9](https://huggingface.co/HuggingFaceTB/SmolLM2-135M/raw/main/config.json) / [3](https://huggingface.co/HuggingFaceTB/SmolLM2-135M/raw/main/config.json)|n/a|[49152](https://huggingface.co/HuggingFaceTB/SmolLM2-135M/raw/main/config.json)|[8192](https://huggingface.co/HuggingFaceTB/SmolLM2-135M/raw/main/config.json)|[2024-10-31](https://huggingface.co/api/models/HuggingFaceTB/SmolLM2-135M)|[card](https://huggingface.co/HuggingFaceTB/SmolLM2-135M) · [config](https://huggingface.co/HuggingFaceTB/SmolLM2-135M/raw/main/config.json)|
-   230→|[HuggingFaceTB/SmolLM2-360M](https://huggingface.co/HuggingFaceTB/SmolLM2-360M)|SmolLM2|[362M](https://huggingface.co/api/models/HuggingFaceTB/SmolLM2-360M)|[32](https://huggingface.co/HuggingFaceTB/SmolLM2-360M/raw/main/config.json)|[960](https://huggingface.co/HuggingFaceTB/SmolLM2-360M/raw/main/config.json)|[15](https://huggingface.co/HuggingFaceTB/SmolLM2-360M/raw/main/config.json) / [5](https://huggingface.co/HuggingFaceTB/SmolLM2-360M/raw/main/config.json)|n/a|[49152](https://huggingface.co/HuggingFaceTB/SmolLM2-360M/raw/main/config.json)|[8192](https://huggingface.co/HuggingFaceTB/SmolLM2-360M/raw/main/config.json)|[2024-10-31](https://huggingface.co/api/models/HuggingFaceTB/SmolLM2-360M)|[card](https://huggingface.co/HuggingFaceTB/SmolLM2-360M) · [config](https://huggingface.co/HuggingFaceTB/SmolLM2-360M/raw/main/config.json)|
-   231→|[HuggingFaceTB/SmolLM2-1.7B](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B)|SmolLM2|[1.7B](https://huggingface.co/api/models/HuggingFaceTB/SmolLM2-1.7B)|[24](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B/raw/main/config.json)|[2048](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B/raw/main/config.json)|[32](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B/raw/main/config.json) / [32](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B/raw/main/config.json)|n/a|[49152](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B/raw/main/config.json)|[8192](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B/raw/main/config.json)|[2024-10-30](https://huggingface.co/api/models/HuggingFaceTB/SmolLM2-1.7B)|[card](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B) · [config](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B/raw/main/config.json)|
-   232→|[TinyLlama/TinyLlama-1.1B-Chat-v1.0](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0)|TinyLlama|[1.1B](https://huggingface.co/api/models/TinyLlama/TinyLlama-1.1B-Chat-v1.0)|[22](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0/raw/main/config.json)|[2048](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0/raw/main/config.json)|[32](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0/raw/main/config.json) / [4](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0/raw/main/config.json)|n/a|[32000](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0/raw/main/config.json)|[2048](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0/raw/main/config.json)|[2023-12-30](https://huggingface.co/api/models/TinyLlama/TinyLlama-1.1B-Chat-v1.0)|[card](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0) · [config](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0/raw/main/config.json)|
-   233→|[allenai/OLMo-2-1124-7B-Instruct](https://huggingface.co/allenai/OLMo-2-1124-7B-Instruct)|OLMo|[7.3B](https://huggingface.co/api/models/allenai/OLMo-2-1124-7B-Instruct)|[32](https://huggingface.co/allenai/OLMo-2-1124-7B-Instruct/raw/main/config.json)|[4096](https://huggingface.co/allenai/OLMo-2-1124-7B-Instruct/raw/main/config.json)|[32](https://huggingface.co/allenai/OLMo-2-1124-7B-Instruct/raw/main/config.json) / [32](https://huggingface.co/allenai/OLMo-2-1124-7B-Instruct/raw/main/config.json)|n/a|[100352](https://huggingface.co/allenai/OLMo-2-1124-7B-Instruct/raw/main/config.json)|[4096](https://huggingface.co/allenai/OLMo-2-1124-7B-Instruct/raw/main/config.json)|[2024-12-18](https://huggingface.co/api/models/allenai/OLMo-2-1124-7B-Instruct)|[card](https://huggingface.co/allenai/OLMo-2-1124-7B-Instruct) · [config](https://huggingface.co/allenai/OLMo-2-1124-7B-Instruct/raw/main/config.json)|
-   234→|[tiiuae/Falcon3-7B-Instruct](https://huggingface.co/tiiuae/Falcon3-7B-Instruct)|Falcon 3|[7.5B](https://huggingface.co/api/models/tiiuae/Falcon3-7B-Instruct)|[28](https://huggingface.co/tiiuae/Falcon3-7B-Instruct/raw/main/config.json)|[3072](https://huggingface.co/tiiuae/Falcon3-7B-Instruct/raw/main/config.json)|[12](https://huggingface.co/tiiuae/Falcon3-7B-Instruct/raw/main/config.json) / [4](https://huggingface.co/tiiuae/Falcon3-7B-Instruct/raw/main/config.json)|[256](https://huggingface.co/tiiuae/Falcon3-7B-Instruct/raw/main/config.json)|[131072](https://huggingface.co/tiiuae/Falcon3-7B-Instruct/raw/main/config.json)|[32768](https://huggingface.co/tiiuae/Falcon3-7B-Instruct/raw/main/config.json)|[2024-11-29](https://huggingface.co/api/models/tiiuae/Falcon3-7B-Instruct)|[card](https://huggingface.co/tiiuae/Falcon3-7B-Instruct) · [config](https://huggingface.co/tiiuae/Falcon3-7B-Instruct/raw/main/config.json)|
-   235→|[01-ai/Yi-1.5-9B-Chat](https://huggingface.co/01-ai/Yi-1.5-9B-Chat)|Yi 1.5|[8.8B](https://huggingface.co/api/models/01-ai/Yi-1.5-9B-Chat)|[48](https://huggingface.co/01-ai/Yi-1.5-9B-Chat/raw/main/config.json)|[4096](https://huggingface.co/01-ai/Yi-1.5-9B-Chat/raw/main/config.json)|[32](https://huggingface.co/01-ai/Yi-1.5-9B-Chat/raw/main/config.json) / [4](https://huggingface.co/01-ai/Yi-1.5-9B-Chat/raw/main/config.json)|n/a|[64000](https://huggingface.co/01-ai/Yi-1.5-9B-Chat/raw/main/config.json)|[4096](https://huggingface.co/01-ai/Yi-1.5-9B-Chat/raw/main/config.json)|[2024-05-10](https://huggingface.co/api/models/01-ai/Yi-1.5-9B-Chat)|[card](https://huggingface.co/01-ai/Yi-1.5-9B-Chat) · [config](https://huggingface.co/01-ai/Yi-1.5-9B-Chat/raw/main/config.json)|
-   236→|[internlm/internlm2_5-7b-chat](https://huggingface.co/internlm/internlm2_5-7b-chat)|InternLM 2.5|[7.7B](https://huggingface.co/api/models/internlm/internlm2_5-7b-chat)|[32](https://huggingface.co/internlm/internlm2_5-7b-chat/raw/main/config.json)|[4096](https://huggingface.co/internlm/internlm2_5-7b-chat/raw/main/config.json)|[32](https://huggingface.co/internlm/internlm2_5-7b-chat/raw/main/config.json) / [8](https://huggingface.co/internlm/internlm2_5-7b-chat/raw/main/config.json)|n/a|[92544](https://huggingface.co/internlm/internlm2_5-7b-chat/raw/main/config.json)|[32768](https://huggingface.co/internlm/internlm2_5-7b-chat/raw/main/config.json)|[2024-06-27](https://huggingface.co/api/models/internlm/internlm2_5-7b-chat)|[card](https://huggingface.co/internlm/internlm2_5-7b-chat) · [config](https://huggingface.co/internlm/internlm2_5-7b-chat/raw/main/config.json)|
-   237→|[deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct)|DeepSeek Coder V2|[16B](https://huggingface.co/api/models/deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct)|[27](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct/raw/main/config.json)|[2048](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct/raw/main/config.json)|[16](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct/raw/main/config.json) / [16](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct/raw/main/config.json)|n/a|[102400](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct/raw/main/config.json)|[163840](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct/raw/main/config.json)|[2024-06-14](https://huggingface.co/api/models/deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct)|[card](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct) · [config](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct/raw/main/config.json)|
-   238→|[codellama/CodeLlama-7b-hf](https://huggingface.co/codellama/CodeLlama-7b-hf)|CodeLlama|[6.7B](https://huggingface.co/api/models/codellama/CodeLlama-7b-hf)|[32](https://huggingface.co/codellama/CodeLlama-7b-hf/raw/main/config.json)|[4096](https://huggingface.co/codellama/CodeLlama-7b-hf/raw/main/config.json)|[32](https://huggingface.co/codellama/CodeLlama-7b-hf/raw/main/config.json) / [32](https://huggingface.co/codellama/CodeLlama-7b-hf/raw/main/config.json)|n/a|[32016](https://huggingface.co/codellama/CodeLlama-7b-hf/raw/main/config.json)|[16384](https://huggingface.co/codellama/CodeLlama-7b-hf/raw/main/config.json)|[2023-08-24](https://huggingface.co/api/models/codellama/CodeLlama-7b-hf)|[card](https://huggingface.co/codellama/CodeLlama-7b-hf) · [config](https://huggingface.co/codellama/CodeLlama-7b-hf/raw/main/config.json)|
-   239→|[codellama/CodeLlama-34b-Instruct-hf](https://huggingface.co/codellama/CodeLlama-34b-Instruct-hf)|CodeLlama|[33.7B](https://huggingface.co/api/models/codellama/CodeLlama-34b-Instruct-hf)|[48](https://huggingface.co/codellama/CodeLlama-34b-Instruct-hf/raw/main/config.json)|[8192](https://huggingface.co/codellama/CodeLlama-34b-Instruct-hf/raw/main/config.json)|[64](https://huggingface.co/codellama/CodeLlama-34b-Instruct-hf/raw/main/config.json) / [8](https://huggingface.co/codellama/CodeLlama-34b-Instruct-hf/raw/main/config.json)|n/a|[32000](https://huggingface.co/codellama/CodeLlama-34b-Instruct-hf/raw/main/config.json)|[16384](https://huggingface.co/codellama/CodeLlama-34b-Instruct-hf/raw/main/config.json)|[2023-08-24](https://huggingface.co/api/models/codellama/CodeLlama-34b-Instruct-hf)|[card](https://huggingface.co/codellama/CodeLlama-34b-Instruct-hf) · [config](https://huggingface.co/codellama/CodeLlama-34b-Instruct-hf/raw/main/config.json)|
-   240→|[Qwen/QwQ-32B](https://huggingface.co/Qwen/QwQ-32B)|QwQ|[32.8B](https://huggingface.co/api/models/Qwen/QwQ-32B)|[64](https://huggingface.co/Qwen/QwQ-32B/raw/main/config.json)|[5120](https://huggingface.co/Qwen/QwQ-32B/raw/main/config.json)|[40](https://huggingface.co/Qwen/QwQ-32B/raw/main/config.json) / [8](https://huggingface.co/Qwen/QwQ-32B/raw/main/config.json)|n/a|[152064](https://huggingface.co/Qwen/QwQ-32B/raw/main/config.json)|[40960](https://huggingface.co/Qwen/QwQ-32B/raw/main/config.json)|[2025-03-05](https://huggingface.co/api/models/Qwen/QwQ-32B)|[card](https://huggingface.co/Qwen/QwQ-32B) · [config](https://huggingface.co/Qwen/QwQ-32B/raw/main/config.json)|
-   241→
-   242→## Embeddings
-   243→
-   244→|Model|Family|Params|Layers|Hidden|Heads|Vocab|Max sequence|Source|
-   245→|---|---|---:|---:|---:|---:|---:|---:|---|
-   246→|[BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3)|Embeddings|[n/a](https://huggingface.co/api/models/BAAI/bge-m3)|[24](https://huggingface.co/BAAI/bge-m3/raw/main/config.json)|[1024](https://huggingface.co/BAAI/bge-m3/raw/main/config.json)|[16](https://huggingface.co/BAAI/bge-m3/raw/main/config.json)|[250002](https://huggingface.co/BAAI/bge-m3/raw/main/config.json)|[8194](https://huggingface.co/BAAI/bge-m3/raw/main/config.json)|[card](https://huggingface.co/BAAI/bge-m3) · [config](https://huggingface.co/BAAI/bge-m3/raw/main/config.json)|
-   247→|[intfloat/e5-large-v2](https://huggingface.co/intfloat/e5-large-v2)|Embeddings|[335M](https://huggingface.co/api/models/intfloat/e5-large-v2)|[24](https://huggingface.co/intfloat/e5-large-v2/raw/main/config.json)|[1024](https://huggingface.co/intfloat/e5-large-v2/raw/main/config.json)|[16](https://huggingface.co/intfloat/e5-large-v2/raw/main/config.json)|[30522](https://huggingface.co/intfloat/e5-large-v2/raw/main/config.json)|[512](https://huggingface.co/intfloat/e5-large-v2/raw/main/config.json)|[card](https://huggingface.co/intfloat/e5-large-v2) · [config](https://huggingface.co/intfloat/e5-large-v2/raw/main/config.json)|
-   248→|[thenlper/gte-large](https://huggingface.co/thenlper/gte-large)|Embeddings|[335M](https://huggingface.co/api/models/thenlper/gte-large)|[24](https://huggingface.co/thenlper/gte-large/raw/main/config.json)|[1024](https://huggingface.co/thenlper/gte-large/raw/main/config.json)|[16](https://huggingface.co/thenlper/gte-large/raw/main/config.json)|[30522](https://huggingface.co/thenlper/gte-large/raw/main/config.json)|[512](https://huggingface.co/thenlper/gte-large/raw/main/config.json)|[card](https://huggingface.co/thenlper/gte-large) · [config](https://huggingface.co/thenlper/gte-large/raw/main/config.json)|
-   249→|[nomic-ai/nomic-embed-text-v1.5](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5)|Embeddings|[137M](https://huggingface.co/api/models/nomic-ai/nomic-embed-text-v1.5)|[12](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/raw/main/config.json)|[768](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/raw/main/config.json)|[12](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/raw/main/config.json)|[30528](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/raw/main/config.json)|[2048](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/raw/main/config.json)|[card](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5) · [config](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/raw/main/config.json)|
-   250→|[jinaai/jina-embeddings-v3](https://huggingface.co/jinaai/jina-embeddings-v3)|Embeddings|[572M](https://huggingface.co/api/models/jinaai/jina-embeddings-v3)|[24](https://huggingface.co/jinaai/jina-embeddings-v3/raw/main/config.json)|[1024](https://huggingface.co/jinaai/jina-embeddings-v3/raw/main/config.json)|[16](https://huggingface.co/jinaai/jina-embeddings-v3/raw/main/config.json)|[250002](https://huggingface.co/jinaai/jina-embeddings-v3/raw/main/config.json)|[8194](https://huggingface.co/jinaai/jina-embeddings-v3/raw/main/config.json)|[card](https://huggingface.co/jinaai/jina-embeddings-v3) · [config](https://huggingface.co/jinaai/jina-embeddings-v3/raw/main/config.json)|
-   251→
-   252→## Vision-language
-   253→
-   254→|Model|Family|Total params|Architecture summary|Vocab|Context|Source|
-   255→|---|---|---:|---|---:|---:|---|
-   256→|[mistralai/Pixtral-12B-2409](https://huggingface.co/mistralai/Pixtral-12B-2409)|Pixtral|[12B](https://huggingface.co/api/models/mistralai/Pixtral-12B-2409)|[n/a / n/a / n/a](https://huggingface.co/mistralai/Pixtral-12B-2409/raw/main/config.json)|n/a|n/a|[card](https://huggingface.co/mistralai/Pixtral-12B-2409) · [config](https://huggingface.co/mistralai/Pixtral-12B-2409/raw/main/config.json)|
-   257→|[Qwen/Qwen2.5-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct)|Qwen 2.5-VL|[7B](https://huggingface.co/api/models/Qwen/Qwen2.5-VL-7B-Instruct)|[28 / 3584 / 28 q](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct/raw/main/config.json)|[152064](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct/raw/main/config.json)|[128000](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct/raw/main/config.json)|[card](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) · [config](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct/raw/main/config.json)|
-   258→|[Qwen/Qwen2.5-VL-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct)|Qwen 2.5-VL|[32B](https://huggingface.co/api/models/Qwen/Qwen2.5-VL-32B-Instruct)|[64 / 5120 / 40 q](https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct/raw/main/config.json)|[152064](https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct/raw/main/config.json)|[128000](https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct/raw/main/config.json)|[card](https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct) · [config](https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct/raw/main/config.json)|
-   259→|[Qwen/Qwen2.5-VL-72B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct)|Qwen 2.5-VL|[72B](https://huggingface.co/api/models/Qwen/Qwen2.5-VL-72B-Instruct)|[80 / 8192 / 64 q](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct/raw/main/config.json)|[152064](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct/raw/main/config.json)|[128000](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct/raw/main/config.json)|[card](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct) · [config](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct/raw/main/config.json)|
-   260→|[OpenGVLab/InternVL2-8B](https://huggingface.co/OpenGVLab/InternVL2-8B)|Vision-language|[8B combined](https://huggingface.co/api/models/OpenGVLab/InternVL2-8B)|[32 LLM / 24 vision / 4096 LLM / 1024 vision / 32 LLM / 16 vision](https://huggingface.co/OpenGVLab/InternVL2-8B/raw/main/config.json)|n/a|[92553](https://huggingface.co/OpenGVLab/InternVL2-8B/raw/main/config.json)|[card](https://huggingface.co/OpenGVLab/InternVL2-8B) · [config](https://huggingface.co/OpenGVLab/InternVL2-8B/raw/main/config.json)|
-   261→|[THUDM/cogvlm2-llama3-chat-19B](https://huggingface.co/THUDM/cogvlm2-llama3-chat-19B)|Vision-language|[19B combined](https://huggingface.co/api/models/THUDM/cogvlm2-llama3-chat-19B)|[32 / 4096 / 32 / n/a](https://huggingface.co/THUDM/cogvlm2-llama3-chat-19B/raw/main/config.json)|n/a|[128256](https://huggingface.co/THUDM/cogvlm2-llama3-chat-19B/raw/main/config.json)|[card](https://huggingface.co/THUDM/cogvlm2-llama3-chat-19B) · [config](https://huggingface.co/THUDM/cogvlm2-llama3-chat-19B/raw/main/config.json)|
-   262→
-   263→## Audio ASR
-   264→
-   265→|Model|Params|Layers/hidden|Vocab|Source|
-   266→|---|---:|---|---:|---|
-   267→|[openai/whisper-large-v3-turbo](https://huggingface.co/openai/whisper-large-v3-turbo)|[809M](https://huggingface.co/api/models/openai/whisper-large-v3-turbo)|[n/a / 1280](https://huggingface.co/openai/whisper-large-v3-turbo/raw/main/config.json)|[51866](https://huggingface.co/openai/whisper-large-v3-turbo/raw/main/config.json)|[card](https://huggingface.co/openai/whisper-large-v3-turbo)|
-   268→|[nvidia/parakeet-tdt-0.6b-v2](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2)|[0.6B](https://huggingface.co/api/models/nvidia/parakeet-tdt-0.6b-v2)|[n/a / n/a](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2/raw/main/config.json)|n/a|[card](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2)|
-   269→|[distil-whisper/distil-large-v3](https://huggingface.co/distil-whisper/distil-large-v3)|[756M](https://huggingface.co/api/models/distil-whisper/distil-large-v3)|[32 / 1280](https://huggingface.co/distil-whisper/distil-large-v3/raw/main/config.json)|[51866](https://huggingface.co/distil-whisper/distil-large-v3/raw/main/config.json)|[card](https://huggingface.co/distil-whisper/distil-large-v3)|
-   270→
-   271→ASR throughput should be modeled as audio-seconds processed per wall-clock second (real-time factor), not text token/sec. Keep audio duration, sample rate, chunking, beam/decoder settings, and WER separate from decoder LLM token metrics. Parakeet’s card reports WER but no universal token/sec.
-   272→
-   273→## Release/status exceptions and verified facts
-   274→
-   275→- **Llama 4 Scout/Maverick:** Meta describes both as MoE with 17B active; HF release material reports approximately 109B total for Scout and 400B for Maverick. Behemoth is announced at nearly 2T total and 288B active, but the official HF path was inaccessible/weights unavailable in this check. [Meta announcement](https://ai.meta.com/blog/llama-4-multimodal-intelligence) · [HF release](https://huggingface.co/blog/llama4-release)
-   276→- **DeepSeek:** V3 and R1 are 671B total / 37B active per the HF card; V3 card says 128K context, while its config exposes 163840—store both fields and show a conflict warning. Coder V2 full is 236B / 21B active and Lite is 16B / 2.4B active. [V3](https://huggingface.co/deepseek-ai/DeepSeek-V3) · [R1](https://huggingface.co/deepseek-ai/DeepSeek-R1) · [Coder V2](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct)
-   277→- **DeepSeek V3.5 / R2:** No verified official HF model card or primary release page under those names was found in the checked universe as of the cutoff. Keep them as `status=pending/unverified`, not as inference rows.
-   278→- **Qwen MoE naming:** The public documented model is **Qwen1.5-MoE-A2.7B**, 14.3B total / 2.7B active; the requested “Qwen2.5-MoE-A2.7B” repo name was not found. [card](https://huggingface.co/Qwen/Qwen1.5-MoE-A2.7B)
-   279→- **WizardLM-2 8x22B:** official repo access returned 401; an HF mirror identifies it as 141B MoE based on Mixtral-8x22B. Use 141B only with `source_confidence=mirror`; keep architecture null until official config access. [mirror](https://huggingface.co/alpindale/WizardLM-2-8x22B)
-   280→- **Pixtral 12B:** Mistral marks Pixtral 12B as no longer maintained/replaced; retain as a historical compatibility row. [status](https://mistral.ai/news/pixtral-12b/)
-   281→
-   282→## Closed API models (pricing-only, detailed pricing deferred)
-   283→
-   284→Do not add local architecture/inference rows for GPT-4o/GPT-4o-mini/o1/o1-mini/o3-mini, Claude 3.5 Sonnet/Haiku/Claude 3 Opus, or Gemini 2.0 Flash/1.5 Pro/1.5 Flash. Add versioned pricing records with `provider`, `model_id`, `input_usd_per_million`, `output_usd_per_million`, cached-input price, effective date, and deprecation status. Pricing research is intentionally deferred to the parent API-pricing track.
-   285→
-   286→## Suggested schema and implementation priorities
-   287→
-   288→Store `total_params` separately from `active_params`: memory sizing uses total weights, while MoE per-token compute uses active experts plus shared/trunk compute. Store both `context_config` and `context_advertised`, plus `release_date_type`, `source_checked_at`, `status`, and `source_confidence`. Must-have first: Qwen3 dense/MoE, DeepSeek V3/R1, Llama 4 Scout/Maverick, Gemma 3, Phi-4, Qwen2.5 Coder, Codestral, BGE-M3/E5, and Qwen2.5-VL. Nice: OLMo-2, Falcon3, Yi1.5, InternLM2.5, Pixtral, InternVL2, CogVLM2, Whisper Turbo, Parakeet. Future/status-only: Behemoth, unverified V3.5/R2, requested Qwen2.5-MoE alias, and official WizardLM-2 until accessible.
-   289→
-   290→
-   291→---
-   292→
-   293→# tokcalc Section C/E evidence ledger (snapshot 2026-09-20)
-   294→
-   295→## Conventions
-   296→Official peak FLOPS values are theoretical. NVIDIA/AMD pages frequently publish 2:4 sparsity; dense = one-half of sparse only when footnote says so. Rows explicitly state per-chip vs system. Prices are dynamic, region/tier dependent; capture date 2026-09-20 IST. N/P means not publicly listed in the source, not zero.
-   297→
-   298→## Section C — accelerators
-   299→
-   300→![AMD Instinct MI355X product image](https://www.amd.com/content/dam/amd/en/images/products/data-centers/3366850-instinct-accelerator-mi350x-prod-front.jpg)
-   301→
-   302→Official visual: [NVIDIA RTX PRO 6000 Blackwell family](https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-6000-family/).
-   303→
-   304→
-   305→| Vendor/product (status; year) | Memory / bandwidth | Dense FP16/BF16 | Interconnect | Benchmark evidence | Official source |
-   306→|---|---|---|---|---|---|
-   307→| NVIDIA H200 SXM / NVL (shipping; 2024) | 141 GB HBM3e, 4.8 TB/s | SXM 1,979 TFLOPS each FP16/BF16; NVL 1,671 TFLOPS | NVLink differs by form factor; use SKU-specific datasheet | N/P on official spec page | [H200 GPU](https://www.nvidia.com/en-us/data-center/h200), [H200 datasheet](https://resources.nvidia.com/en-us-gpu-resources/hpc-datasheet-sc23) |
-   308→| NVIDIA B200 SXM (shipping in HGX/DGX systems; 2025) | 180 GB HBM3e, ~7.7–8.0 TB/s (official DGX gives 1,440 GB/64 TB/s for 8 GPUs) | Current official page exposes FP4/FP8 system metrics but not a clean FP16/BF16 dense row; store N/P until SKU datasheet is verified. Do not use AI TOPS or FP8 as FP16. | 5th-gen NVLink; DGX B200 14.4 TB/s aggregate NVLink | N/P in official page | [NVIDIA B200/HGX table](https://www.nvidia.com/en-us/data-center/b200), [DGX B200](https://www.nvidia.com/en-us/data-center/dgx-b200/) |
-   309→| NVIDIA B300 SXM (shipping/available in 2026 cloud listings; 2025 announcement) | 288 GB HBM3e, 8 TB/s | DGX B300 official page gives FP4 144 sparse/108 dense and FP8 72 sparse; it does not expose FP16/BF16 dense. Store N/P rather than infer from FP8. | 5th-gen NVLink; DGX B300 14.4 TB/s aggregate | N/P | [DGX B300](https://www.nvidia.com/en-us/data-center/dgx-b300), [B300/HGX table](https://www.nvidia.com/en-us/data-center/b200) |
-   310→| NVIDIA GB200 NVL72 system (shipping announced systems; 2025) | System: 13.4 TB HBM3e GPU memory, 576 TB/s; one GB200 superchip: 372 GB HBM3e, 16 TB/s | 360 PFLOPS FP16/BF16 shown for NVL72 in sparse format => ~180 PFLOPS dense; superchip 10 PFLOPS shown, verify footnote before storing | NVLink 130 TB/s system; 3.6 TB/s/superchip | N/P official | [GB200 NVL72](https://www.nvidia.com/en-us/data-center/gb200-nvl72/) |
-   311→| NVIDIA GB300 NVL72 (announced/availability status SKU-dependent; 2025/2026) | N/P in official source reviewed; do not borrow B300 or GB200 values | N/P | N/P | N/P | [DGX B300](https://www.nvidia.com/en-us/data-center/dgx-b300), [Together GPU pricing](https://www.together.ai/pricing) (GB300 row shown —, not a spec source) |
-   312→| NVIDIA RTX PRO 6000 Blackwell Workstation/Server/Max-Q (shipping; 2025) | 96 GB GDDR7 ECC; server product lists 1,597 GB/s | FP16/BF16 dense N/P in official page | PCIe Gen5 x16 | N/P | [RTX PRO 6000 family](https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-6000-family/), [server specs](https://www.nvidia.com/en-us/data-center/rtx-pro-6000-blackwell-server-edition/) |
-   313→| NVIDIA RTX 5000 Ada workstation (shipping; 2023) | 32 GB GDDR6 ECC; official page does not expose bandwidth in extracted text | N/P (page gives 65.3 FP32 and 1,044.4 Tensor performance, not dense FP16/BF16) | PCIe Gen4 x16 | N/P | [RTX 5000 Ada](https://www.nvidia.com/en-us/products/workstations/rtx-5000/), [datasheet](https://resources.nvidia.com/en-us-briefcase-for-datasheets/rtx-5000-datashe?ncid=no-ncid) |
-   314→| NVIDIA RTX 4500 Ada workstation (shipping; 2023) | 24 GB GDDR6 ECC; bandwidth N/P official page | N/P (634.0 Tensor listed, not FP16/BF16 dense) | PCIe Gen4 x16 | N/P | [RTX 4500 Ada](https://www.nvidia.com/en-us/products/workstations/rtx-4500/), [datasheet](https://www.nvidia.com/content/dam/en-zz/Solutions/products/workstations/nvidia-rtx-4500-datasheet.pdf) |
-   315→| NVIDIA RTX 5090 consumer (shipping; 2025) | 32 GB GDDR7, 1,792 GB/s | N/P official FP16/BF16 (AI TOPS is not interchangeable) | N/P | N/P | [RTX 5090](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/) |
-   316→| NVIDIA RTX 5080 consumer (shipping; 2025) | 16 GB GDDR7, 960 GB/s | N/P official FP16/BF16 | N/P | N/P | [RTX 5080](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5080) |
-   317→| NVIDIA RTX 4080 Super consumer (shipping; 2024) | 16 GB GDDR6X, 736 GB/s | N/P official FP16/BF16 | N/P | N/P | [NVIDIA GeForce 40 series](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/) |
-   318→| NVIDIA RTX 4070 Ti Super consumer (shipping; 2024) | 16 GB GDDR6X, 672 GB/s | N/P official FP16/BF16 | N/P | N/P | [NVIDIA GeForce 40 series](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/) |
-   319→| AMD Instinct MI300X (shipping; 2023/2024) | 192 GB HBM3, 5.325 TB/s | 1,307.4 TFLOPS FP16 and BF16 dense | 7 x 128 GB/s Infinity Fabric links; PCIe Gen5 x16 | N/P official | [MI300X product](https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html), [datasheet PDF](https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/data-sheets/amd-instinct-mi300x-data-sheet.pdf) |
-   320→| AMD Instinct MI325X (shipping; 2024) | 256 GB HBM3E, 6 TB/s | 1,307.4 TFLOPS FP16/BF16 dense (official page) | N/P in extracted product page | N/P | [MI325X](https://www.amd.com/en/products/accelerators/instinct/mi300/mi325x.html), [datasheet](https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/product-briefs/instinct-mi325x-datasheet.pdf) |
-   321→| AMD Instinct MI350X (shipping/launch 2025) | 288 GB HBM3E, 8 TB/s | Product page shows 4.6 PFLOPS FP16/BF16 with sparsity; dense ~2.3 PFLOPS (only if applying AMD sparse footnote; store both) | N/P product page; platform data required | N/P | [MI350X](https://www.amd.com/en/products/accelerators/instinct/mi350/mi350x.html), [AMD 2025 blog](https://www.amd.com/en/blogs/2025/amd-instinct-mi350-series-and-beyond-accelerating-the-future-of-ai-and-hpc.html) |
-   322→| AMD Instinct MI355X (shipping/launch 2025) | 288 GB HBM3E, 8 TB/s | Official page shows 5.0 PFLOPS FP16/BF16 with sparsity; dense ~2.5 PFLOPS (derived half) | Platform: 8-GPU 2.3 TB total; 1,194.8 GB/s aggregate bidirectional peer-to-peer | N/P | [MI355X](https://www.amd.com/en/products/accelerators/instinct/mi350/mi355x.html), [platform](https://www.amd.com/en/products/accelerators/instinct/mi350/mi355x/platform.html) |
-   323→| Intel Gaudi 2 (shipping; 2022) | 96 GB HBM2e, 2.45 TB/s; 48 MB SRAM | Official architecture page gives memory but not clean dense BF16/FP16 number in extracted text; keep N/P rather than infer | 24 x 100GbE RoCE RDMA NICs | Intel published comparative benchmarks; capture separately | [Gaudi architecture](https://docs.habana.ai/en/latest/Gaudi_Overview/Gaudi_Architecture.html), [Gaudi2 white paper](https://cdrdv2-public.intel.com/839363/Intel-Gaudi2-AI-Accelerators-whitepaper.pdf) |
-   324→| Intel Gaudi 3 (shipping/PCIe available; 2024; expanded availability 2025) | 128 GB HBM2e, 3.7 TB/s; 96 MB SRAM | 1.678 PFLOPS BF16 matrix (official Intel brief) | 12 x 200GbE; PCIe Gen5 x16 128 GB/s bidirectional; 1,200 GB/s accelerator interconnect in Intel comparison | Intel performance/positioning PDF includes Llama 3 8B/70B/405B test matrix; numbers need SKU/config extraction | [Gaudi 3 architecture](https://docs.habana.ai/en/latest/Gaudi_Overview/Gaudi_Architecture.html), [PCIe brief](https://cdrdv2-public.intel.com/817488/Gaudi%203%20PCIe%20Product%20Brief_RB_1_V6.pdf), [availability](https://newsroom.intel.com/artificial-intelligence/intel-gaudi-3-expands-availability-drive-ai-innovation-scale) |
-   325→| Google TPU v5e (shipping/GA; 2023) | 16 GB HBM, 800 GiB/s | 197 TFLOPS BF16/chip | 400 GB/s bidirectional ICI/chip; 256-chip pod; 50.63 PFLOPS/pod | N/P official LLM tok/s | [v5e docs](https://docs.cloud.google.com/tpu/docs/v5e) |
-   326→| Google TPU v5p (shipping/GA; 2023) | 95 GiB HBM, 2,765 GB/s | 459 TFLOPS BF16/chip | 1,200 GB/s bidirectional ICI/chip; 8,960-chip pod | N/P official LLM tok/s | [v5p docs](https://docs.cloud.google.com/tpu/docs/v5p), [launch](https://cloud.google.com/blog/products/ai-machine-learning/introducing-cloud-tpu-v5p-and-ai-hypercomputer) |
-   327→| Google TPU v6e Trillium (GA Dec 2024; shipping) | 32 GB HBM; ~1,638 GB/s (Google page gives doubled v5e bandwidth; v6e docs expose 1,638 GB/s) | 918 TFLOPS BF16/chip | 800 GB/s bidirectional ICI/chip; up to 256 chips/pod | Google says up to 4.7x peak compute/chip vs v5e | [v6e docs](https://docs.cloud.google.com/tpu/docs/v6e), [GA release](https://cloud.google.com/blog/products/compute/trillium-tpu-is-ga) |
-   328→| Google TPU v7x Ironwood (GA/available in Google docs by 2026; 2025/2026) | 192 GiB HBM, 7,380 GB/s | 2,307 TFLOPS BF16/chip | 1,200 GB/s bidirectional ICI/chip; 9,216 chips/pod | Google TPU page says 42.5 exaflops/pod; system-level | [TPU7x docs](https://docs.cloud.google.com/tpu/docs/tpu7x), [TPU landing page](https://cloud.google.com/tpu) |
-   329→| Groq LPU (shipping cloud; architecture generation/SKU varies) | Groq says on-chip SRAM bandwidth >80 TB/s; exact per-chip capacity/FP16 FLOPS not disclosed on reviewed page | N/P; do not substitute FP8 rack number | Platform page: 256 LPUs/rack, 40 PB/s SRAM bandwidth, 315 PFLOPS FP8/rack (system) | Artificial Analysis: Llama 3 70B 284 tok/s; Groq later docs show Llama 3.3 70B ~280 tok/s; speculative endpoint 1,665 tok/s (different model/config) | [Groq Llama 3 benchmark](https://groq.com/blog/12-hours-later-groq-is-running-llama-3-instruct-8-70b-by-meta-ai-on-its-lpu-inference-enginge), [LPU](https://groq.com/blog/the-groq-lpu-explained), [platform](https://groq.com/platform), [model docs](https://console.groq.com/docs/model/llama-3.3-70b-versatile) |
-   330→| Cerebras WSE-3 / CS-3 (shipping; 2024) | CS-3 system: 44 GB on-chip SRAM, 21 PB/s memory bandwidth | WSE-3/CS-3: 125 PFLOPS peak AI performance is system/chip marketing metric, not directly dense FP16/BF16; mark N/P for exact dense | CS-3: 214 Pb/s interconnect bandwidth | Cerebras Inference: 1,800 tok/s Llama 3.1 8B and 450 tok/s Llama 3.1 70B; benchmark is service/system-level | [CS-3 system](https://www.cerebras.ai/system), [WSE-3 release](https://www.cerebras.ai/press-release/cerebras-announces-third-generation-wafer-scale-engine), [inference benchmark](https://www.cerebras.ai/blog/introducing-cerebras-inference-ai-at-instant-speed) |
-   331→| SambaNova RDU/SN40L (shipping cloud; 2024 generation) | Exact capacity/bandwidth and clean dense FP16/BF16 not verified from an official accessible source in this run | N/P | N/P | No official Llama 3 throughput row verified; do not fabricate | [SambaNova platform](https://sambanova.ai/technology) (spec gap; revisit with browser/tab if needed) |
-   332→| Apple M2 Ultra (shipping; 2023) | Unified memory 192 GB max; >800 GB/s bandwidth (Apple Mac Studio page) | Apple does not publish FP16/BF16 dense TFLOPS in reviewed sources | Unified memory, no NVLink; UltraFusion interconnect details not required for single chip | N/P official LLM tok/s | [M2 Ultra](https://www.apple.com/newsroom/2023/06/apple-unveils-m2-ultra/), [Mac Studio](https://www.apple.com/newsroom/2023/06/apple-unveils-new-mac-studio-and-mac-pro/) |
-   333→| Apple M3 Ultra (shipping; 2025) | Unified memory 96–512 GB; >800 GB/s unified bandwidth (Mac Studio page) | Apple does not publish FP16/BF16 dense TFLOPS | Unified memory; no NVLink | N/P | [M3 Ultra](https://www.apple.com/newsroom/2025/03/apple-reveals-m3-ultra-taking-apple-silicon-to-a-new-extreme), [Mac Studio](https://www.apple.com/newsroom/2025/03/apple-unveils-new-mac-studio-the-most-powerful-mac-ever/) |
-   334→| Apple M4 Ultra | No official Apple M4 Ultra shipping/announcement evidence verified in this run; mark unavailable rather than assume | N/P | N/P | N/P | Apple Newsroom search found M4 consumer products, not M4 Ultra; keep status unresolved |
-   335→| NVIDIA Tesla V100 (still installed/cloud; launched 2017) | 16/32 GB HBM2, 900 GB/s PCIe or 1,134 GB/s V100S; NVLink 300 GB/s bidirectional | Tensor FP16 112 TFLOPS PCIe / 125 TFLOPS SXM2 (marketing tensor metric; document as tensor, not scalar dense) | PCIe 32 GB/s; NVLink 300 GB/s | NVIDIA app guide: frameworks up to 50x vs P100 (system/framework benchmark, not LLM) | [V100 datasheet](https://images.nvidia.com/content/technologies/volta/pdf/tesla-volta-v100-datasheet-letter-fnl-web.pdf), [V100 page](https://www.nvidia.com/en-gb/data-center/tesla-v100) |
-   336→| NVIDIA Tesla P100 (legacy/hobbyist; launched 2016) | 12/16 GB HBM2, 549/732 GB/s | 18.7 TFLOPS PCIe or >21 TFLOPS native half (variant-dependent) | PCIe Gen3; no NVLink on PCIe card; SXM2 NVLink context differs | NVIDIA guide: frameworks up to 3x vs K80 | [P100 PCIe datasheet](https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/tesla-p100/pdf/nvidia-tesla-p100-PCIe-datasheet.pdf), [P100 page](https://www.nvidia.com/en-au/data-center/tesla-p100/) |
-   337→| NVIDIA Tesla K80 (legacy/hobbyist; launched 2014) | 24 GB GDDR5 aggregate (12 GB/GPU), 480 GB/s aggregate | N/P official FP16; 8.73 TFLOPS FP32 aggregate/board | PCIe Gen3; dual-GPU board | NVIDIA says up to 2x application throughput with two onboard GPUs | [K80 page](https://www.nvidia.com/en-gb/data-center/tesla-k80), [K80 board spec](https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/tesla-product-literature/Tesla-K80-BoardSpec-07317-001-v05.pdf) |
-   338→
-   339→### Section C implementation notes
-   340→- Add `dense_fp16_tflops`, `dense_bf16_tflops`, `sparse_fp16_bf16_tflops`, `metric_scope` (`chip`, `board`, `system`), `memory_capacity_gb`, `memory_type`, `memory_bandwidth_gbps`, `interconnect_type`, `interconnect_bandwidth_gbps`, `release_year`, `shipping_status`, `benchmark_tokens_s`, `benchmark_model`, `benchmark_config`, `benchmark_source`, `as_of`.
-   341→- Never put AI TOPS, FP8, FP4, sparse, or system PFLOPS into the dense FP16/BF16 field. For AMD/NVIDIA sparse rows, retain sparse value and derive dense only when vendor footnote explicitly says 2x.
-   342→- Apple unified memory is not VRAM; use `memory_scope=unified` and note CPU/GPU shared pool.
-   343→
-   344→## Section E — cloud pricing snapshot (USD; capture 2026-09-20; rates volatile)
-   345→
-   346→Prices below are exact search/extraction outputs from direct provider pages; provider pages can show different tiers/regions at different times. `—` means no current public rate verified, not that capacity never exists.
-   347→
-   348→| Provider + direct page | H100 80GB | H200 | A100 80GB | B200 | L40S | RTX 4090 | MI300X | Mac M2 Ultra | Billing/caveat |
-   349→|---|---:|---:|---:|---:|---:|---:|---:|---:|---|
-   350→| RunPod [pricing](https://www.runpod.io/pricing), [GPU models](https://www.runpod.io/gpu-models), [comparison](https://www.runpod.io/articles/comparison/choosing-gpus) | $4.79/hr serverless listing (dedicated/community varies) | $3.59/hr GPU-model snapshot; $4.31/hr clusters | $1.59/hr PCIe comparison; cluster page $1.79 SXM | $5.98/hr GPU-model snapshot; comparison $6.79 | $1.09/hr comparison | $0.34/hr Community article; product page says from $0.34 (another page headline $0.74) | — | — | Per-second; Community/Secure/Serverless/Cluster tiers and region supply change rate |
-   351→| Lambda [instances](https://lambda.ai/instances) | $3.99/GPU-hr SXM | — | $2.79/GPU-hr 80GB SXM in 8-GPU plan | $6.69/GPU-hr | — | — | — | — | Pay by minute; A100 80GB price is per GPU within 8-GPU instance |
-   352→| Modal [pricing](https://modal.com/pricing) | $0.001097/s = $3.9492/hr | $0.001261/s = $4.5396/hr | $0.000694/s = $2.4984/hr | $0.001736/s = $6.2496/hr | $0.000542/s = $1.9512/hr | — | — | — | Per-second compute; exact page is dynamic |
-   353→| Vast.ai [pricing](https://vast.ai/pricing) | H100 SXM start $1.73, median $2.67/hr | H200 start $1.98, median $4.69/hr | A100 SXM start $0.20, median $0.80; PCIe start $0.33, median $0.78 | B200 start $6.25, median $8.13/hr | live guide ~$0.47 start/$0.53 median | live guide ~$0.13 start (median not captured) | — | — | Marketplace; on-demand per-second, interruptible 50%+ cheaper, reserved up to 50% off; location/host quality matters |
-   354→| Together AI [pricing](https://www.together.ai/pricing), [GPU clusters](https://www.together.ai/gpu-clusters) | $3.99/GPU-hr on-demand; $3.19 reserved | $5.99/GPU-hr | — | HGX B200 $8.19/GPU-hr | — | — | — | — | Cluster service, typically 8–256 GPU scale; reserved terms differ |
-   355→| Replicate [pricing](https://replicate.com/pricing), [L40S announcement](https://replicate.com/blog/nvidia-l40s-gpus-are-here) | $0.006100/s = $21.96/hr (single H100 listing) | pricing page lists H200 but amount not captured in extraction | — | — | $3.51/hr historical public-model hardware price | — | — | — | Per-second/request execution; multi-GPU H100 listed $0.0122/s = $43.92/hr; public/private model rates can differ |
-   356→| CoreWeave [pricing](https://www.coreweave.com/pricing) | Official current page dynamic; no exact H100 amount extracted; third-party normalization $6.16/GPU-hr from $49.24 8-GPU node | third-party normalization $6.31/GPU-hr from $50.44 8-GPU node | third-party normalized $1.19/GPU-hr | $68.80/hr on-demand / $34.87 spot (node scope must be confirmed) | — | — | — | — | 8-GPU HGX minimum is common; use provider page to verify current region/node |
-   357→| TensorDock [cloud GPU table](https://www.tensordock.com/cloud-gpus.html) | $2.25/hr H100 SXM5 | — | $1.80/hr SXM4; $1.50/hr PCIe | — | $0.95/hr L40 (not L40S; do not relabel) | $0.35/hr (site has a $0.37 product headline) | — | — | Typical hourly, varies by host; resources not included |
-   358→| Hugging Face Endpoints [pricing](https://huggingface.co/docs/inference-endpoints/en/pricing), [pricing page](https://huggingface.co/pricing) | $10/hr (GCP H100 x1 endpoint table) | — | $2.50/hr A100 large (HF pricing page; endpoint table may vary) | — | — | — | — | — | Endpoint compute; provider/region and scale-to-zero settings matter |
-   359→| AWS EC2 [P5](https://aws.amazon.com/ec2/instance-types/p5/), [on-demand](https://aws.amazon.com/ec2/pricing/on-demand), [capacity blocks](https://aws.amazon.com/ec2/capacityblocks/pricing/) | P5.48xlarge 8x H100: $55.04/hr on-demand => $6.88/GPU-hr (third-party link to official); Capacity Blocks examples $41.528 instance/$5.191 GPU Atlanta, $37.76/$4.720 Sydney/London | P5e/P5en supports 8x H200 (1128 GB total) but exact public rate not extracted | P4d page exists but exact current 80GB rate not extracted | — | G5/G6 prices not extracted | — | — | — | On-demand vs Capacity Blocks differs; region, OS, tenancy and egress excluded |
-   360→| Google Cloud [GPU pricing](https://cloud.google.com/products/compute/gpus-pricing), [TPU pricing](https://cloud.google.com/tpu/pricing) | A3/H100 exact current rate not extracted from official table | A3 Ultra/H200 exact rate not extracted | — | A4/B200 exact rate not extracted | — | — | — | — | Public calculator/table is region and VM-shape dependent; do not use nonofficial estimate |
-   361→| Azure [VM pricing](https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/) | ND H100 v5 exact current rate not extracted | ND H200 v5 exact current rate not extracted | — | — | — | — | — | — | Region, Linux/Windows, reservation/spot matter; mark unavailable until calculator query |
-   362→| Oracle Cloud [GPU compute](https://www.oracle.com/cloud/compute/gpu/) | Official exact public rate not extracted (third-party reports $10/hr; do not treat as official) | — | third-party reports $4/hr; not verified official | — | — | — | 8x MI300X node third-party $48/hr = $6/GPU-hr, not official Oracle page | — | OCI shape pricing, region and tenancy; verify in OCI cost estimator |
-   363→
-   364→### Google TPU prices (useful for calculator)
-   365→- v5e: $1.20/chip-hour on-demand, us-central1; v5p: $4.20/chip-hour on-demand, us-east1/us-east5; Trillium v6e: $2.70/chip-hour on-demand in us-east1/us-east5; Ironwood table shows $12.00/chip-hour us-central1. Prices are from [Google Cloud TPU pricing](https://cloud.google.com/tpu/pricing), with reservation/commitment columns lower than on-demand. v5e/v5p/v6e rates are chip-hour, not pod-hour.
-   366→
-   367→### Pricing interpretation
-   368→- Treat rates as timestamped observations, not stable constants. Store `provider`, `gpu_sku`, `scope` (per GPU/per node/per endpoint), `billing_unit`, `tier`, `region`, `captured_at`, `source_url`, `price_usd`, `availability`, and `notes`.
-   369→- Directly missing providers/GPUs must remain null. Do not copy a H100 rate to H200, MI300X or Mac M2 Ultra.
-   370→
-   371→## Access gaps / unsuccessful attempts
-   372→- Direct extraction timed out for several dynamic provider pages (RunPod, Lambda, Modal, Vast, Replicate, CoreWeave, TensorDock, AWS, Google, Azure, Oracle); search indexing exposed current table snippets but dynamic pages require browser/tab or provider calculators for a fully reproducible snapshot.
-   373→- NVIDIA H200 and Google TPU pages intermittently timed out; official indexed datasheet snippets supplied H200 compute and TPU specs.
-   374→- SambaNova official RDU exact numeric spec/LLM benchmark and M4 Ultra shipping status were not verified; mark unavailable.
-   375→- No official exact current rates were found for Together A100/L40S/4090/MI300X/Mac, HF H200/B200 etc.; preserve nulls.
-   376→- Apple does not publish FP16/BF16 dense TFLOPS in reviewed pages; do not invent from GPU-core counts.
-   377→
-   378→
-   379→---
-   380→
-   381→# Section D — More Quantization Formats (tokcalc)
-   382→
-   383→Research date: 2026-09-20. Status: best effort, source-backed within the primary/official sources listed below. Quantization quality and speed are highly model-, calibration-, engine-, GPU-, batch-, context-, and kernel-dependent. Do not present any single benchmark as a universal factor.
-   384→
-   385→## 1. GGUF / llama.cpp variants
-   386→
-   387→GGUF (GPT-Generated Unified Format, as named by the official specification) is the file/container format used by llama.cpp; these are block-quantized weight schemes, usually WnA16 at inference (integer/FP weights with dequantization into FP16/BF16 compute). Source: [GGUF specification](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md). The official llama.cpp quantizer reports effective bits/weight (including scale/metadata overhead), model size, and benchmark speeds. Source: [llama.cpp quantize README](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/quantize/README.md). The legacy per-variant PPL deltas below come from llama.cpp's `quantize.cpp` table for Llama-3-8B; they are not universal. Source: [quantize.cpp](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/quantize/quantize.cpp).
-   388→
-   389→### Official Llama-3.1-8B GGUF benchmark table
-   390→
-   391→The current README's FP16 reference is 14.96 GiB, 16.0005 bpw, 29.17 generated tokens/s; the quantized rows are the same benchmark run (prompt 512, generation 128). Size reduction and decode speedup are calculated from those published values: `1 - size/14.96` and `gen_tps/29.17`. They are a model-dependent llama.cpp reference, not a generic promise.
-   392→
-   393→| Variant | Effective bpw | Size GiB | Approx reduction vs FP16 | Published gen t/s | Derived decode speedup vs FP16 | Accuracy/perplexity evidence and caveat |
-   394→|---|---:|---:|---:|---:|---:|---|
-   395→| Q2_K | 3.1593 | 2.95 | 80.3% | 79.85 | 2.74x (+174%) | Legacy Llama-3-8B table: +3.5199 PPL; severe quality loss at this level. |
-   396→| Q3_K_S | 3.6429 | 3.41 | 77.2% | 69.84 | 2.39x (+139%) | Legacy Llama-3-8B: +1.6321 PPL; low-bit degradation can be task-visible. |
-   397→| Q3_K_M | 3.9960 | 3.74 | 75.0% | 71.68 | 2.46x (+146%) | Legacy Llama-3-8B: +0.6569 PPL. |
-   398→| Q4_0 | approximately 4 (legacy effective bpw varies by tensor/metadata) | 4.34 | 71.0% | not published in current README row | not quantified | Legacy Llama-3-8B: +0.4685 PPL; older/simple scheme. |
-   399→| Q4_1 | approximately 4 | 4.78 | 68.0% | not published in current README row | not quantified | Legacy Llama-3-8B: +0.4511 PPL. |
-   400→| Q4_K_S | 4.6672 | 4.36 | 70.9% | 76.71 | 2.63x (+163%) | Legacy Llama-3-8B: +0.2689 PPL. |
-   401→| Q4_K_M | 4.8944 | 4.58 | 69.4% | 71.93 | 2.47x (+147%) | Legacy Llama-3-8B: +0.1754 PPL; common quality/size compromise. |
-   402→| Q5_0 | approximately 5 | 5.21 | 65.2% | not published in current README row | not quantified | Legacy Llama-3-8B: +0.1316 PPL. |
-   403→| Q5_1 | approximately 5 | 5.65 | 62.2% | not published in current README row | not quantified | Legacy Llama-3-8B: +0.1062 PPL. |
-   404→| Q5_K_S | 5.5704 | 5.21 | 65.2% | 69.53 | 2.38x (+138%) | Legacy Llama-3-8B: +0.1049 PPL. |
-   405→| Q5_K_M | 5.7036 | 5.33 | 64.4% | 67.23 | 2.30x (+130%) | Legacy Llama-3-8B: +0.0569 PPL; high-fidelity compromise. |
-   406→| Q6_K | 6.5633 | 6.14 | 59.0% | 58.67 | 2.01x (+101%) | Legacy Llama-3-8B: +0.0217 PPL; usually near-FP16 quality, but still model/task dependent. |
-   407→| Q8_0 | 8.5008 | 7.95 | 46.9% | 50.93 | 1.75x (+75%) | Legacy Llama-3-8B: +0.0026 PPL; near-FP16 PPL in that test. |
-   408→| FP16 reference | 16.0005 | 14.96 | — | 29.17 | 1.00x | Baseline in current README. |
-   409→
-   410→A second, independent 2026 evaluation of Llama-3.1-8B-Instruct reports FP16 GGUF 15,317 MiB and PPL 7.32; Q3_K_S 3,487 MiB/PPL 8.96, Q3_K_M 3,825 MiB/7.96, Q4_0 4,438 MiB/7.74, Q4_1 4,885 MiB/7.72, Q4_K_S 4,468 MiB/7.62, Q4_K_M 4,685 MiB/7.56, Q5_0 5,332 MiB/7.43, Q5_1 5,780 MiB/7.43, Q5_K_S 5,332 MiB/7.43, Q5_K_M 5,460 MiB/7.40, Q6_K 6,283 MiB/7.35, and Q8_0 8,138 MiB/7.33. Source: [2026 GGUF evaluation](https://arxiv.org/html/2601.14277v1). This demonstrates why tokcalc should store benchmark provenance and not convert PPL deltas to a universal “accuracy loss %”.
-   411→
-   412→![GGUF compression versus quality-loss plot from the 2026 evaluation](https://arxiv.org/html/2601.14277v1/output.png)
-   413→
-   414→The plot is useful as a maintainer-facing visual, but its points are still one Llama-3.1-8B-Instruct evaluation—not a universal quality curve.
-   415→
-   416→Engine support: llama.cpp is the reference engine and supports the family of 1.5/2/3/4/5/6/8-bit integer quantization. Source: [llama.cpp README](https://github.com/ggml-org/llama.cpp). vLLM has GGUF loading, but its own documentation calls it “highly experimental and under-optimized,” with single-file limitations in current docs. Source: [vLLM GGUF docs](https://docs.vllm.ai/en/stable/features/quantization/gguf). TGI, TensorRT-LLM and ExLlamaV2 are not GGUF-native runtimes; do not mark them as direct GGUF support without conversion.
-   417→
-   418→Dequant overhead: no format-wide constant. GGUF WnA16 generally dequantizes blocks during matmul; the published llama.cpp decode ratios above are the most useful end-to-end evidence. At larger batches/compute-bound workloads, a low-bit weight-only format may not speed up because dequantization and kernel utilization dominate.
-   419→
-   420→## 2. Weight-only PTQ formats and runtimes
-   421→
-   422→| Format / full name | Effective bits and approximate size vs FP16 | Accuracy/perplexity caveat | Dequant/kernel overhead and measured speed | Engine support (verified) | Primary/official sources |
-   423→|---|---|---|---|---|---|
-   424→| GPTQ — GPT Quantization / “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers” | Common W3/W4, also W2 and W8 implementations. Ideal raw weight storage is about 25% of FP16 at 4-bit, 18.75% at 3-bit, 12.5% at 2-bit; group scales/zeros add metadata, so use actual checkpoint size. | Paper reports 3–4 bits with negligible degradation for tested GPT models, but low-bit quality is model/calibration dependent; use measured PPL/task metrics. | GPTQ paper reports end-to-end inference speedups around 3.25x on A100 and 4.5x on A6000 in its tested setup; this is not a general 2026 multiplier. W4A16 requires dequantization/packing; high batch can become compute-bound. | vLLM supports GPTQ across Volta/Turing/Ampere/Ada/Hopper in its support table; TGI supports pre-quantized GPTQ; TensorRT-LLM supports W4A16/W4A8 GPTQ; ExLlamaV2 supports GPTQ. Sources: [vLLM quantization](https://docs.vllm.ai/en/latest/features/quantization/), [TGI quantization](https://huggingface.co/docs/text-generation-inference/en/conceptual/quantization), [TRT-LLM quantization](https://nvidia.github.io/TensorRT-LLM/latest/features/quantization.html), [ExLlamaV2](https://github.com/turboderp-org/exllamav2). | [GPTQ paper](https://arxiv.org/abs/2210.17323), [GPTQ code](https://github.com/IST-DASLab/gptq) |
-   425→| AWQ — Activation-aware Weight Quantization | Usually W4A16; protected salient channels are scaled, not stored as FP16 mixed-precision. Approx raw weight storage ~25% of FP16 plus scales/metadata. | AWQ protects approximately 1% salient weights based on activation distribution; paper reports strong 4-bit quality, but quality depends on calibration/data/model. | Paper reports >3x over Hugging Face FP16 in TinyChat on tested desktop/mobile settings. AutoAWQ README reports up to 3x and fused-module 1.6–2.5x; it also warns W4A16 can lose speedup in compute-bound regimes due to dequant overhead. | vLLM supports AWQ; TGI supports pre-quantized AWQ; TensorRT-LLM supports W4A16/W4A8 AWQ; Transformers/AutoAWQ ecosystem supports it. | [AWQ paper](https://arxiv.org/abs/2306.00978), [llm-awq code](https://github.com/mit-han-lab/llm-awq), [AutoAWQ](https://github.com/casper-hansen/autoawq), [TRT-LLM matrix](https://nvidia.github.io/TensorRT-LLM/latest/features/quantization.html) |
-   426→| EXL2 — ExLlamaV2 mixed-bitrate format | Mixed 2/3/4/5/6/8-bit layers; target average bitrate can be any value roughly 2–8 bpw. 4 bpw is ~25% raw weight bytes vs FP16, 3 bpw ~18.75%; actual file includes scales/metadata. Typical Llama-3 8B practical config: 4.0 bpw is a conservative quality/VRAM point; 3.0–3.5 bpw saves more but requires validation. | Layer-wise bit allocation minimizes calibration error at target bitrate. Do not map “4.0 bpw” to one universal accuracy number. | ExLlamaV2 README gives RTX 4090 decode examples: Llama2 7B EXL2 3.0 bpw 257 t/s, 4.0 bpw 211, 5.0 bpw 179; CodeLlama 34B 4.0 bpw 50 t/s. No FP16 matched baseline in the same table, so report these as absolute measured speeds, not claimed universal speedups. Remapping is designed for little/no performance impact, but kernels and GPU dominate. | ExLlamaV2/ExLlamaV3 and TabbyAPI are primary; TGI explicitly supports pre-quantized EXL2; vLLM/TensorRT-LLM are not general EXL2-native formats in their primary support tables. | [ExLlamaV2](https://github.com/turboderp-org/exllamav2), [TGI quantization](https://huggingface.co/docs/text-generation-inference/en/conceptual/quantization) |
-   427→| SmoothQuant — SmoothQuant: Accurate and Efficient Post-Training Quantization for LLMs | W8A8 (8-bit weights + 8-bit activations), not merely a file format; ~50% raw tensor storage vs FP16 when both weights/activations are 8-bit. It migrates activation outliers into weights using per-channel scales. | Paper reports negligible loss for tested models; calibration and model sensitivity matter. W8A8 is generally safer than very-low-bit W4 but can differ by layer. | Paper reports up to 1.56x speedup and 2x memory reduction; one A100 OPT benchmark reports up to 1.51x. These are paper-specific. Dequant/requant/scaling overhead is replaced by INT8 kernels; do not model a generic factor. | Commonly implemented in TensorRT-LLM/Model Optimizer and related HF/vLLM compression stacks; not a native GGUF/ExL2 file. TensorRT-LLM documents INT8 SQ and W8A8 recipes. | [SmoothQuant paper](https://arxiv.org/abs/2211.10438), [TensorRT-LLM quantization blog](https://nvidia.github.io/TensorRT-LLM/1.2.0/blogs/quantization-in-TRT-LLM.html) |
-   428→| QuaRot — Outlier-Free 4-Bit Inference in Rotated LLMs | End-to-end W4A4KV4 in the paper, plus 6/8-bit variants. Raw 4-bit weights are ~25% of FP16; activations/KV also become 4-bit, so total runtime savings depend heavily on KV/context. | Llama-2 WikiText-2 PPL: 7B FP baseline 5.47 vs QuaRot 4-bit GPTQ 6.10 / QuaRot-128G 5.93; 13B 4.88 vs 5.40/5.26; 70B 3.32 vs 3.79/3.61. Llama-2-70B average zero-shot 75.98% vs FP16 77.07%. | Online Hadamard transforms add at most 7% forward-pass overhead; paper reports up to 2.16x prefill speedup, 3.71x decode-memory reduction, and 3.2–4.3x linear-layer speedup in tested setups. Do not treat as end-to-end universal. | Research PyTorch/HF implementation with CUTLASS INT4, FlashInfer KV quantization, FlashAttention; no broad direct vLLM/TGI/ExLlamaV2/TensorRT-LLM turnkey format documented. | [QuaRot paper](https://arxiv.org/html/2404.00456v1), [QuaRot code](https://github.com/spcl/QuaRot) |
-   429→| SpinQuant — LLM quantization with learned rotations | 4-bit weights, activations, and KV cache in the paper (W4A4KV4); raw 4-bit weight storage ~25% of FP16 before metadata. Learned rotations are optimized to make the model more quantization-friendly. | On Llama-2 7B, the paper reports a 2.9-point zero-shot gap to full precision, outperforming LLM-QAT by 19.1 points and SmoothQuant by 25.0 points; on hard-to-quantize Llama-3 8B it reduces the gap to FP by up to 45.1% relative to QuaRot. These are paper-specific task results, not a generic accuracy-loss percentage. | No matched end-to-end speedup range was verified from the primary source; rotation application and low-bit kernels are implementation-dependent. Store overhead as unknown unless measured for the chosen engine. | Official PyTorch/CUDA research implementation and model/checkpoint code; no verified turnkey support in llama.cpp, vLLM, TGI, TensorRT-LLM, or ExLlamaV2 primary matrices. | [SpinQuant paper](https://arxiv.org/abs/2405.16406), [official code](https://github.com/facebookresearch/SpinQuant) |
-   430→| QuIP# — QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks | Primarily 2-bit weights using E8 lattice codebooks; model zoo also 3/4-bit. 2-bit raw weights ~12.5% FP16 before codebook/scales. | Authors report near-FP16 performance at 2 bits for tested Llama 1/2 models; specialized calibration/rotations and model architecture matter. No universal PPL. | Custom CUDA kernels; third-party vLLM benchmark reports ~159 t/s Llama-7B batch 1 on RTX 4090, but no matched FP16 baseline in that source, so do not convert to generic speedup. | Official CUDA code/model zoo; niche integrations (third-party vLLM/QuIP-for-all). No verified mainstream vLLM/TGI/TensorRT-LLM/ExLlamaV2 direct support in primary docs. | [QuIP# paper](https://arxiv.org/abs/2402.04396), [official code](https://github.com/Cornell-RelaxML/quip-sharp), [QuIP-for-all benchmark](https://github.com/chu-tianxiang/QuIP-for-all) |
-   431→| HQQ — Half-Quadratic Quantization | Calibration-free 8/4/3/2/1-bit; actual storage includes scales/zeros. 4-bit raw ~25% FP16, 2-bit ~12.5%; no single canonical checkpoint overhead. | No calibration data required. Official README says axis=0 generally has better quality than axis=1 at low bits, while optimized inference backends often use axis=1; validate PPL/task quality. | Official repo reports ~158 t/s Llama-3-8B 4-bit on RTX 4090 with optimized backend; no FP16 matched baseline, so no universal speedup. On-the-fly quantization trades setup/dequant overhead for convenience. | HQQ/PyTorch, torch.compile, ATen/CUDA, GemLite, torchao int4, HF Transformers/PEFT, and vLLM via GemLite/on-the-fly path. | [HQQ code](https://github.com/dropbox/hqq), [HF HQQ docs](https://huggingface.co/docs/transformers/quantization/hqq), [HQQ technical blog](https://dropbox.github.io/hqq_blog/) |
-   432→| EETQ — Easy and Efficient Quantization for Transformers | INT8 weight-only per-channel (W8A16); raw weights ~50% FP16 plus per-channel scales. | Usually high fidelity relative to lower-bit W4, but accuracy varies by model; no universal PPL/accuracy percentage found in official docs. | Uses FasterTransformer/TensorRT-LLM GEMM/GEMV kernels and FlashAttention2; no stable official cross-model speedup range found. On-the-fly TGI quantization adds startup cost; runtime dequantization/kernel behavior is engine-dependent. | HF Transformers and TGI; TGI supports `--quantize eetq` on the fly. Not a direct GGUF/EXL2 format. vLLM current primary support table does not establish EETQ as a universal backend. | [EETQ repo](https://github.com/NetEase-FuXi/EETQ), [HF EETQ docs](https://github.com/huggingface/transformers/blob/main/docs/source/en/quantization/eetq.md), [TGI quantization](https://huggingface.co/docs/text-generation-inference/en/conceptual/quantization) |
-   433→| BitBLAS — mixed-precision BLAS kernel library, not a model file format | Supports many W/A combinations (e.g., W4A16 GPTQ, INT2/INT8 BitNet, FP4/FP8/INT4/INT2/NF4); effective size is inherited from chosen weight format. | Accuracy is determined by the source quantizer/checkpoint (GPTQ/AWQ/BitNet etc.), not BitBLAS itself. | BitBLAS README reports BitNet WINT2AINT8 GEMV/GEMM 8x/2x vs cuBLAS FP16 on A100; kernel microbenchmark, not a universal end-to-end LLM speedup. | PyTorch, GPTQModel/AutoGPTQ, vLLM integration, BitNet integration; hardware-specific A100/A6000/V100/RTX 4090 matrices. | [BitBLAS repo](https://github.com/microsoft/BitBLAS), [Ladder paper](https://www.usenix.org/conference/osdi24/presentation/wang-lei) |
-   434→
-   435→## 3. Native low-precision formats
-   436→
-   437→| Format | Definition / effective bits and size | Accuracy caveat | Dequant/compute overhead and measured speed | Engine support/status | Primary/official sources |
-   438→|---|---|---|---|---|---|
-   439→| BitNet b1.58 | Native ternary weights in {-1,0,+1}; paper names 1.58 bits because ternary entropy is log2(3), but practical packed formats may use ~2 bits/weight plus scales/metadata. Compared with FP16, ideal raw weight storage is ~9.9% at 1.58 bits or 12.5% at 2-bit packing. | Requires a model trained or specifically converted for ternary weights; ordinary FP16 model PTQ is not equivalent. Original paper and 2B4T report parity/competitive quality against similarly sized baselines in tested tasks, but not broad universal accuracy. | Official bitnet.cpp reports CPU speedups: roughly 1.37–5.07x on ARM and 2.37–6.17x on x86, with energy reductions in its tested setups; these are runtime/model/platform-specific. No general GPU speed claim should be inserted. | Microsoft bitnet.cpp is official CPU-focused inference (x86/ARM, I2_S/TL1/TL2 kernels; GPU/NPU support status should be checked per release). Not a drop-in GGUF/GPTQ quant of arbitrary models. | [BitNet foundational paper](https://arxiv.org/abs/2402.17764), [CPU inference report](https://arxiv.org/abs/2410.16144), [official BitNet runtime](https://github.com/microsoft/BitNet), [2B4T model](https://huggingface.co/microsoft/BitNet-b1.58-2B-4T) |
-   440→| FP8 — 8-bit floating point (E4M3/E5M2) | W8A8 or W8A16 depending recipe; raw tensor storage ~50% FP16. E4M3/E5M2 range/precision differ; scale metadata may be per-tensor, rowwise, or blockwise. | Usually near-FP16 on supported models with calibration; TensorRT-LLM’s example reports MMLU losses 0.14–0.89% for FP8 across Falcon/Llama/MPT rows, but this is not a universal accuracy percentage. | TensorRT-LLM Llama-2-7B H100 benchmark: FP8 1.40–1.51x at BS 1–8, and 2.3x at latency-constrained BS16. A tuned Llama-3.3-70B H100 case reports 6049 vs 2474 token/s = 2.44x (+144.5%) vs tuned FP16, with TTFT -40.4%; use only with conditions. | Production-grade on H100/H200/Ada/Blackwell paths in TensorRT-LLM; vLLM W8A8 officially Hopper/Ada and some AMD MI300X paths; TGI supports FP8 on the fly. | [FP8 paper](https://arxiv.org/abs/2209.05433), [TRT-LLM benchmark](https://nvidia.github.io/TensorRT-LLM/1.2.0/blogs/quantization-in-TRT-LLM.html), [TRT-LLM FP8 tuning guide](https://nvidia.github.io/TensorRT-LLM/0.19.0/performance/performance-tuning-guide/fp8-quantization.html), [vLLM FP8 docs](https://docs.vllm.ai/en/v0.11.2/features/quantization/fp8) |
-   441→| NVFP4 — NVIDIA FP4 (E2M1) with hierarchical scaling | Each value is 1 sign + 2 exponent + 1 mantissa = 4 bits; NVIDIA’s 2025 technical blog specifies an E4M3 FP8 scale per 16-value microblock plus a tensor-level FP32 scalar. Approx effective weight storage ~4.5 bits/value before tensor-scale amortization (4 + 8/16), about 3.56x smaller than FP16; do not call it “exactly 4 bits/weight.” | NVIDIA’s Jun-2025 blog reports 1% or less accuracy degradation vs FP8 on seven DeepSeek-R1-0528 evaluations (AIME 2024 was 2% better) for that PTQ recipe; general quality remains model/recipe dependent. NVFP4 cannot be assumed safe for ordinary FP16 PTQ without validation. | Blackwell tensor cores execute native FP4; NVIDIA blog reports memory/compute benefits but not a portable end-to-end tokens/s multiplier. The official TensorRT-LLM support matrix marks NVFP4 on Blackwell, not Hopper. Avoid a universal speedup unless a benchmark supplies model, GPU, batch, and baseline. | TensorRT-LLM/Model Optimizer/Transformer Engine on Blackwell; TRT-LLM docs list NVFP4, MXFP4, FP8 and KV-cache support by SM generation. | [NVIDIA NVFP4 blog (Jun 2025)](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference), [Transformer Engine NVFP4 docs](https://docs.nvidia.com/deeplearning/transformer-engine-releases/release-2.13/user-guide/features/low_precision_training/nvfp4/nvfp4.html), [TRT-LLM matrix](https://nvidia.github.io/TensorRT-LLM/latest/features/quantization.html), [Model Optimizer](https://github.com/NVIDIA/Model-Optimizer) |
-   442→| mxFP8 / OCP MX — microscaling FP8 | OCP MX groups 32 elements with a shared E8M0 scale and 8-bit FP8 elements (E4M3 or E5M2). Effective storage ≈8 + 8/32 = 8.25 bits/value, ~48.4% of FP16 raw bytes (1.94x reduction), plus tensor/layout metadata. MXFP4/MXFP6 also exist; user specifically asked mxFP8. | MX paper reports MXFP8 near-lossless in broad evaluated settings; exact LLM PTQ quality varies by format, scale granularity, and hardware. No universal PPL loss. | Scale metadata and block-scale operations add implementation complexity; speedups are hardware/kernel-specific. Do not promise FP8’s 2x without a matched benchmark. | OCP spec is open; NVIDIA Blackwell/TRT-LLM supports MXFP8-related block scaling paths, and vLLM lists MXFP8/MXFP4 in current project support. Broad cross-vendor production support is still emerging; vLLM issue/docs show ongoing work. | [OCP MX v1.0 spec](https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf), [MX paper](https://arxiv.org/abs/2310.10537), [Microsoft microxcaling](https://github.com/microsoft/microxcaling), [vLLM project](https://github.com/vllm-project/vllm) |
-   443→
-   444→## 4. What tokcalc should store
-   445→
-   446→Do not model “quantization” as one scalar percentage. Store a normalized format/checkpoint record with:
-   447→
-   448→```ts
-   449→type QuantizationRecord = {
-   450→  id: string;                    // gguf:q4_k_m, gptq:w4a16:g128, fp8:e4m3, nvfp4, ...
-   451→  family: 'gguf'|'gptq'|'awq'|'exl2'|'smoothquant'|'rotation'|'hqq'|'eetq'|'kernel'|'native_fp'|'mx';
-   452→  displayName: string;
-   453→  fullName?: string;
-   454→  weightBits: number;            // nominal element bits (4, 8, 1.58, ...)
-   455→  effectiveBitsPerWeight?: number;// includes scales/zeros/metadata if measured
-   456→  activationBits?: number;
-   457→  kvCacheBits?: number;
-   458→  granularity?: 'tensor'|'channel'|'row'|'group'|'block'|'microblock'|'mixed_layer';
-   459→  groupSize?: number;            // e.g. 128; NVFP4 microblock 16; OCP MX 32
-   460→  scaleBits?: number;
-   461→  zeroPoint?: boolean;
-   462→  accumulationType?: 'fp16'|'bf16'|'fp32'|'int32'|string;
-   463→  sizeBytesMeasured?: number;    // prefer actual checkpoint size
-   464→  sizeBytesEstimated?: number;   // estimate from params*effectiveBits/8 + nonquantized tensors
-   465→  nonQuantizedFraction?: number;// embeddings, norms, lm_head, etc.
-   466→  calibration?: 'none'|'data_free'|'calibration_required'|'qat'|'unknown';
-   467→  modelCompatibility?: string[];
-   468→  engineSupport?: Record<string, 'native'|'plugin'|'experimental'|'conversion'|'unsupported'|'unknown'>;
-   469→  benchmarkRefs?: Array<{url:string; model:string; gpu:string; batch?:number; context?:number; metric:string; value:number; baseline?:number;}>;
-   470→  qualityRefs?: Array<{url:string; model:string; dataset:string; metric:'ppl'|'accuracy'|'loss'; baseline:number; quantized:number;}>;
-   471→  statusAsOf: string;
-   472→};
-   473→```
-   474→
-   475→Recommended formulas (show assumptions):
-   476→
-   477→- Raw weight estimate: `params * effectiveBitsPerWeight / 8`.
-   478→- Better checkpoint estimate: `quantized tensors + non-quantized tensors + scales/zeros + metadata`; use an actual file size whenever possible.
-   479→- KV memory: `tokens * layers * 2 * kv_heads * head_dim * bytes_per_kv_element`, then multiply by replication/parallelism factors; weight quantization does not automatically quantize KV.
-   480→- Throughput: store separate prefill and decode values, not one “tokens/sec”. Store batch, prompt/output lengths, context, GPU, engine version, kernel, tensor parallelism, and whether CUDA graphs/FlashAttention/speculative decoding were enabled.
-   481→- Accuracy: store raw metric and baseline/quantized pair (e.g., PPL 7.32→7.56 or MMLU 69.1→68.5), not “accuracy loss = 0.87%” unless source defines that calculation. PPL is not accuracy; a PPL percentage change cannot be reported as an accuracy percentage.
-   482→- Speed: represent `speedup = quant_tps / baseline_tps` only when source gives matched baseline conditions. Keep measured range by workload category (prefill/decode, batch 1/high batch) and label it “benchmark-specific”.
-   483→
-   484→## 5. Implementation recommendation for tokcalc
-   485→
-   486→1. Must-have: GGUF variants with actual effective bpw, measured size, and a low/medium/high quality note. Default user-facing presets: Q4_K_M, Q5_K_M, Q6_K, Q8_0. Include Q2/Q3 only with warning badges.
-   487→2. Must-have: GPTQ/AWQ/EXL2 W4A16 as separate engine-aware records; do not merge them into “INT4.”
-   488→3. Must-have: FP8 W8A8 for H100/H200/Ada/Blackwell with engine-specific benchmark ranges and a warning that speed is workload dependent.
-   489→4. Nice: NVFP4 and MXFP8 as hardware-gated Blackwell/OCP records, with storage estimate including scales.
-   490→5. Future/research: QuaRot, SpinQuant, QuIP#, HQQ, BitBLAS, BitNet, SmoothQuant, EETQ as an “experimental / specialized” tab. They need checkpoint/runtime compatibility metadata, not merely an extra quant dropdown.
-   491→
-   492→## 6. Evidence gaps / blockers
-   493→
-   494→- No primary source gave a universal accuracy-loss percentage, dequantization factor, or real-world speedup for every requested format. Such values are not scientifically portable; use null/“benchmark required” rather than fabricated defaults.
-   495→- Current llama.cpp README does not publish matched decode speeds for Q4_0/Q4_1/Q5_0/Q5_1, so those rows deliberately leave speedup unquantified.
-   496→- HQQ, QuIP#, EETQ, and NVFP4 official pages expose absolute or selected benchmark numbers but not matched FP16 end-to-end baselines in all cases; do not convert them to speedup without a baseline.
-   497→- vLLM/TGI support changes quickly. TGI documentation says it is in maintenance mode and recommends vLLM/SGLang/local engines for new work; tokcalc should timestamp engine support and version-pin it.
-   498→- Search/extraction timed out on several dynamic GitHub/NVIDIA/Hugging Face pages; where possible I used raw GitHub files, official docs, papers, and search-confirmed page titles. No claim above relies solely on a search snippet when a primary source was available.
-   499→
-   500→
-   501→---
-   502→
-   503→# tokcalc Sections F/H — verified/best-effort research (2026-09-20)
-   504→
-   505→Scope: official/current pages checked 2026-09-20. Prices USD per 1M tokens unless noted. “N/A current” means the requested model is absent/retired/contact-sales on the provider’s current official catalog; historical list prices are separated and must not be treated as buyable today.
-   506→
-   507→## F — API pricing
-   508→
-   509→### OpenAI (current official aliases)
-   510→
-   511→| Model / official source | Input | Cached input | Output | Status/effective note |
-   512→|---|---:|---:|---:|---|
-   513→| GPT-4o [`gpt-4o`](https://developers.openai.com/api/docs/models/gpt-4o) | $2.50 | $1.25 | $10.00 | Current alias; `gpt-4o-2024-05-13` snapshot deprecated. |
-   514→| GPT-4o mini [`gpt-4o-mini`](https://developers.openai.com/api/docs/models/gpt-4o-mini) | $0.15 | $0.075 | $0.60 | Current alias; older snapshots may be deprecated. |
-   515→| o1 [`o1`](https://developers.openai.com/api/docs/models/o1) | $15.00 | $7.50 | $60.00 | Current alias; `o1-2024-12-17` and `o1-preview-2024-09-12` deprecated. |
-   516→| o1-mini [`o1-mini`](https://developers.openai.com/api/docs/models/o1-mini) | $1.10 | $0.55 | $4.40 | Current alias but docs recommend newer o3-mini; `o1-mini-2024-09-12` deprecated. |
-   517→| o3-mini [`o3-mini`](https://developers.openai.com/api/docs/models/o3-mini) | $1.10 | $0.55 | $4.40 | Current alias; `o3-mini-2025-01-31` snapshot deprecated. |
-   518→
-   519→OpenAI prompt caching: enabled by default for supported models; a reusable matching prefix is billed at the model’s cached-input rate, and cached tokens appear in `input_tokens_details.cached_tokens`. The current guide says savings can be up to 90% on newer pricing, but the requested models’ official model pages explicitly show 50% of standard input (the cached rates in the table). Caching is prefix-based; cache writes are not an additive fee—the token uses uncached, cached, or write rate. Minimum cacheable length varies by model (the current guide gives 1,024 for newer GPT-5.6+; do not assume that threshold for these legacy models). Source: [`Prompt caching`](https://developers.openai.com/api/docs/guides/prompt-caching).
-   520→
-   521→### Anthropic
-   522→
-   523→| Requested model | Input | Cached read (derived from official multiplier) | 5m write | 1h write | Output | Status/effective note |
-   524→|---|---:|---:|---:|---:|---:|---|
-   525→| Claude 3.5 Sonnet (`claude-3-5-sonnet-20240620`, `...20241022`) | $3.00 | $0.30 | $3.75 | $6.00 | $15.00 | Historical launch price from [Anthropic announcement](https://www.anthropic.com/news/claude-3-5-sonnet); both API models retired Oct 28 2025; replacement Claude Sonnet 4.6 per [deprecations](https://platform.claude.com/docs/en/about-claude/model-deprecations). |
-   526→| Claude 3.5 Haiku (`claude-3-5-haiku-20241022`) | $0.80 | $0.08 | $1.00 | $1.60 | $4.00 | Official current pricing table still shows these list rates but marks it retired except Bedrock/Google Cloud; Anthropic API retirement Feb 19 2026; replacement Haiku 4.5. [Pricing](https://platform.claude.com/docs/en/about-claude/pricing), [deprecations](https://platform.claude.com/docs/en/about-claude/model-deprecations). |
-   527→| Claude 3 Opus (`claude-3-opus-20240229`) | $15.00 | $1.50 | $18.75 | $30.00 | $75.00 | Historical official launch rate in [Claude 3 announcement](https://www.anthropic.com/news/claude-3-family); retired Jan 5 2026; replacement Opus 4.8 per [deprecations](https://platform.claude.com/docs/en/about-claude/model-deprecations). |
-   528→
-   529→Anthropic caching semantics: cache write = 1.25× base for 5-minute TTL or 2× for 1-hour TTL; cache hit/refresh = 0.1× base (these are the official general multipliers; computed legacy-model rates above). A cache hit pays for the cached tokens, and caching breaks even after one read for 5m or two reads for 1h. Cache writes and reads are input-token operations, not output. Source: [Anthropic pricing](https://platform.claude.com/docs/en/about-claude/pricing), [prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching). Partner platforms set their own retirement schedules, so Bedrock/Google Cloud availability may differ.
-   530→
-   531→### Google Gemini
-   532→
-   533→| Requested model | Input | Cached/context-cache input | Output | Status/effective note |
-   534→|---|---:|---:|---:|---|
-   535→| Gemini 2.0 Flash (`gemini-2.0-flash`) | $0.10 | Not safely current; historical context-cache terms varied | $0.40 | Historical launch price from [Google Developers Blog](https://developers.googleblog.com/start-building-with-the-gemini-2-0-flash-family); officially shut down June 1 2026 per [model page](https://ai.google.dev/gemini-api/docs/models/gemini-2.0-flash). |
-   536→| Gemini 1.5 Pro (`gemini-1.5-pro` / `-002`) | $1.25 ≤128K; $2.50 >128K | Historical explicit cache rate not verified in accessible official current docs | $5.00 ≤128K; $10.00 >128K | Shut down Sept 29 2025 per [Google changelog](https://ai.google.dev/gemini-api/docs/changelog). Historical rates are from archived official pricing page: [Wayback capture](https://web.archive.org/web/20250613043647/https://ai.google.dev/gemini-api/docs/pricing). |
-   537→| Gemini 1.5 Flash (`gemini-1.5-flash` / `-002`) | $0.075 ≤128K; $0.15 >128K | $0.01875 ≤128K (archived official table; do not extrapolate >128K without re-verification) | $0.30 ≤128K; $0.60 >128K | Shut down Sept 29 2025 per [Google changelog](https://ai.google.dev/gemini-api/docs/changelog). Price reduction/effective date Aug 12 is documented in [Google Developers Blog](https://developers.googleblog.com/en/gemini-15-flash-updates-google-ai-studio-gemini-api/). |
-   538→
-   539→Google context caching is not a single universal discount: explicit/implicit caching, storage duration, and token rates depend on model/version and API surface. The current caching docs no longer list 1.5/2.0 because these models are shut down; use the archived table only for historical comparison and set status=retired in tokcalc. Current docs describe cache-hit token accounting via `usage.total_cached_tokens` for newer models: [caching docs](https://ai.google.dev/gemini-api/docs/caching).
-   540→
-   541→### Together AI
-   542→
-   543→| Requested model | Input | Cached | Output | Status/deployment/effective note |
-   544→|---|---:|---:|---:|---|
-   545→| Llama 3.1 405B | N/A public | N/A | N/A | Official page says not available on Together Serverless API; no public dedicated token rate. [Model page](https://www.together.ai/models/llama-3-1-405b). |
-   546→| Llama 3.1 70B | $0.88 | Not listed | $0.88 | Current official model card, Dedicated deployment; released 2024-07-22 and last updated 2026-02-24. [Model page](https://www.together.ai/models/llama-3-1-70b). |
-   547→| Qwen 2.5 72B | $1.20 | Not listed | $1.20 | Current official model card, Dedicated deployment; last updated 2026-02-05. [Model page](https://www.together.ai/models/qwen-2-5). |
-   548→
-   549→Together caching: automatic prefix matching; only longest matching prefix is cached. Current serverless overview documents this, but requested Llama 70B/Qwen 72B cards are Dedicated and publish no cached price; do not invent a discount. [Together serverless overview](https://docs.together.ai/docs/serverless/overview). Current pricing page shows cached rates only for selected newer serverless models, not these requested cards: [Together pricing](https://www.together.ai/pricing).
-   550→
-   551→### Groq
-   552→
-   553→| Requested model | Current input | Current cached | Current output | Status/effective note |
-   554→|---|---:|---:|---:|---|
-   555→| Llama 3.1 70B | N/A | N/A | N/A | Not in current supported-model catalog; historical `llama-3.1-70b-versatile` was decommissioned. Current production catalog has Llama 3.1 8B and Llama 3.3 70B as Enterprise “Contact Sales.” [Supported models](https://console.groq.com/docs/models), [deprecations](https://console.groq.com/docs/deprecations). |
-   556→| Llama 3.1 8B | Contact Sales | 50% discount if available | Contact Sales | Current catalog Enterprise-only Contact Sales; prior public rate ($0.05 input/$0.08 output) is stale, not a current self-serve rate. [Supported models](https://console.groq.com/docs/models), [prompt caching](https://console.groq.com/docs/prompt-caching). |
-   557→| Mixtral 8x7B | N/A | N/A | N/A | No longer in current supported-model catalog; historical price should be marked retired/unavailable. |
-   558→
-   559→Groq prompt caching: no additional fee and 50% discount on cached input; Batch already discounts all tokens 50%, with no extra cache discount on batch. [Groq prompt caching](https://console.groq.com/docs/prompt-caching).
-   560→
-   561→### Fireworks AI
-   562→
-   563→| Requested model | Input | Cached | Output | Status/effective note |
-   564→|---|---:|---:|---:|---|
-   565→| Llama 3.1 70B Instruct | $0.90 generic >16B band | Default ~50% discount (exact model rate required) | $0.90 generic band | Fireworks model FAQ text says serverless $0.90/M, but the same model page’s specification currently says Serverless Not supported / Deploy on Demand—flag this contradiction. [Model page](https://fireworks.ai/models/fireworks/llama-v3p1-70b-instruct), [serverless pricing](https://docs.fireworks.ai/serverless/pricing). |
-   566→| Llama 3.1 405B Instruct | N/A serverless | N/A | N/A | Official model page says Serverless Not supported; on-demand only, no public per-token rate. [Model page](https://fireworks.ai/models/fireworks/llama-v3p1-405b-instruct). |
-   567→| Mixtral 8x7B | $0.50 generic MoE band | Model-specific cache rate required; default discount 50% only | $0.50 generic MoE band | Official serverless pricing gives MoE up to 56B (example Mixtral 8x7B) at $0.50/M. |
-   568→| Mixtral 8x22B | $1.20 generic MoE band | Model-specific cache rate required; default discount 50% only | $1.20 generic MoE band | Official serverless pricing gives MoE 56.1B–176B (example Mixtral 8x22B) at $1.20/M. |
-   569→
-   570→Fireworks caching is on by default for serverless; default cached discount 50%, exact discount/rate varies by model. Batch adds an additional 50% savings on cached tokens per docs, so calculator needs separate `standard`, `cached`, and `batch` dimensions rather than one discount. [Prompt caching](https://docs.fireworks.ai/guides/prompt-caching), [batch inference](https://docs.fireworks.ai/guides/batch-inference).
-   571→
-   572→### Cerebras
-   573→
-   574→| Requested model | Current official rate | Status/effective note |
-   575→|---|---|---|
-   576→| Llama 3.1 70B | Current N/A; historical $0.60/M (input/output split not specified) | Official 2024 launch press release says Llama 3.1 70B priced at 60 cents per million tokens; current pricing page/catalog no longer lists this requested model. [Launch release](https://www.cerebras.ai/press-release/cerebras-launches-the-worlds-fastest-ai-inference), [current docs pricing](https://inference-docs.cerebras.ai/support/pricing). |
-   577→| Llama 3.1 405B | Current N/A; historical $6 input/$12 output | Official announcement says customer trials/GA Q1 2025 at $6/M input and $12/M output; stale by 2026. [Announcement](https://www.cerebras.ai/blog/llama-405b-inference). |
-   578→
-   579→Cerebras prompt caching is automatic, prefix-based in 128-token blocks, guaranteed TTL 5 minutes (may persist up to 1 hour), and cached input is billed at the same standard input rate (no discount). Cached tokens are reported under `usage.prompt_tokens_details.cached_tokens`; cached tokens count toward total TPM but not uncached TPM. [Cerebras prompt caching](https://inference-docs.cerebras.ai/capabilities/prompt-caching).
-   580→
-   581→### DeepSeek API
-   582→
-   583→| Requested model | Current official rate | Status/effective note |
-   584→|---|---|---|
-   585→| DeepSeek V3 | N/A current | Current official pricing page has moved to `deepseek-flash` = DeepSeek-V4.1-Flash and `deepseek-v4-pro`; requested V3 is not listed. [Current pricing](https://api-docs.deepseek.com/quick_start/pricing/). |
-   586→| DeepSeek R1 | N/A current | Current official pricing page does not list R1; mark retired/unavailable unless a specific partner endpoint is selected. |
-   587→
-   588→Current successor rates (not substitutes for requested rows): DeepSeek-V4.1-Flash: cache-hit $0.003/M off-peak or $0.006 peak, cache-miss $0.15/$0.30, output $0.60/$1.20; DeepSeek-V4-Pro-0813: cache-hit $0.022/$0.044, miss $0.66/$1.32, output $1.98/$3.96 (off-peak/peak). Off-peak is all hours outside 01:00–04:00 and 06:00–10:00 UTC Mon–Fri excluding Chinese public holidays. Current page says legacy V4 Flash names are served by V4.1 Flash and retired. [DeepSeek pricing](https://api-docs.deepseek.com/quick_start/pricing/).
-   589→
-   590→### Mistral
-   591→
-   592→| Requested model | Input | Cached input | Output | Status/effective note |
-   593→|---|---:|---:|---:|---|
-   594→| Mistral Large 2.0 (`mistral-large-2407`) | N/A current | N/A | N/A | Retired 2025-03-30; replacement Mistral Large 3. [Model card](https://docs.mistral.ai/models/mistral-large-2-0-24-07). |
-   595→| Mistral Large 2.1 (`mistral-large-2411`, if intended by “Large 2”) | N/A current | N/A | N/A | Deprecated 2026-02-27; replacement mapping in docs is inconsistent across pages, but current model overview lists retirement/deprecation. [Models overview](https://docs.mistral.ai/models). |
-   596→| Codestral current (`codestral-25-08`) | $0.30 | $0.03 | $0.90 | Current code-model pricing. [Pricing](https://docs.mistral.ai/inference/pricing), [model card](https://docs.mistral.ai/models/model-cards/codestral-25-08). |
-   597→| Current replacement Mistral Large 3 (`mistral-large-2512`) | $0.50 | Up to 90% input saving / exact cache rate to verify | $1.50 | Current replacement, 256K context, open-weight MoE; [model card](https://docs.mistral.ai/models/mistral-large-3-25-12). |
-   598→
-   599→Mistral docs say cached input tokens reduce input cost by up to 90%; API endpoint docs specify cached tokens billed at 10% of standard input. Batch processing is 50% off. [Mistral pricing](https://docs.mistral.ai/inference/pricing), [chat endpoint](https://docs.mistral.ai/api/endpoint/chat), [pricing FAQ](https://mistral.ai/pricing).
-   600→
-   601→## Provider caching matrix
-   602→
-   603→| Provider | Cache semantics | Rate/discount | Calculator implementation |
-   604→|---|---|---|---|
-   605→| OpenAI | Matching reusable prefix; usage reports cached tokens; model pages give explicit cached input | Requested models 50% of input; newer model guide says up to 90% | `cache_hit_input_rate` per model; no additive write fee for these models |
-   606→| Anthropic | Explicit cache writes, 5m/1h TTL, cache reads | 5m write 1.25×, 1h write 2×, read 0.1× base | Separate write/read token counters and TTL |
-   607→| Google | Model/API-specific explicit/implicit context cache; legacy docs required archive | Gemini 1.5 Flash historical cached input $0.01875/M ≤128K; other requested rates unavailable/currently retired | Versioned historical pricing only; status retired |
-   608→| Together | Automatic longest-prefix match on selected serverless models; requested dedicated cards omit cache rate | No verified requested-model discount | Do not assume cached discount on dedicated rows |
-   609→| Groq | Automatic cache, prefix semantics | 50% cached-input discount, no additional fee; Batch does not stack | `min(cache_rate, batch_rate)` / provider rule |
-   610→| Fireworks | Serverless prompt caching on by default; model-specific cached prices | Default 50%, exact varies; batch adds 50% on cached tokens | Model-specific cached rate + batch modifier |
-   611→| Cerebras | Automatic 128-token prefix blocks, ephemeral | No discount; standard input rate | Track cached tokens for latency/TPM, not cost |
-   612→| DeepSeek | Current V4 page publishes cache-hit vs miss and peak/off-peak | Explicit rates, dynamic time window | Time-aware pricing schedule |
-   613→| Mistral | Cached input billed as 10% standard; prefix caching | Up to 90% savings; Batch 50% | Cached-input rate + batch modifier |
-   614→
-   615→## H — Public benchmark databases / suites
-   616→
-   617→These are a mixture of result databases/leaderboards and reproducible benchmark runners. Only MLPerf results, HF datasets, and Arena are pre-existing result stores; local runners produce tokcalc validation data after execution.
-   618→
-   619→| Name + URL | What it measures | Update cadence | API/download | Commercial license |
-   620→|---|---|---|---|---|
-   621→| **MLPerf Inference Results v4.1** — [repo](https://github.com/mlcommons/inference_results_v4.1), [datacenter page](https://mlcommons.org/benchmarks/inference-datacenter/) | Standardized system speed/throughput and accuracy across inference scenarios; result repo has structured open/closed submissions. | Release/submission rounds, not continuous; v4.1 repo last pushed 2025-07-24. | Git clone/download structured `open/closed` result trees; no stable API/CSV service. | Repo metadata/license is Apache-2.0; commercial reuse of code/results generally permitted subject to attribution/trademark/rules. [LICENSE](https://github.com/mlcommons/inference_results_v4.1/blob/main/LICENSE.md). |
-   622→| **MLPerf Inference benchmark suite** — [repo](https://github.com/mlcommons/inference) | Reproducible benchmark runner: throughput/speed, latency/scenario constraints, and task accuracy. | Active release/submission cadence; repo updated Sep 2026. | Clone/reference implementations and configs; no direct results API. | Apache-2.0 [LICENSE](https://github.com/mlcommons/inference/blob/master/LICENSE.md). |
-   623→| **Stanford HELM** — [repo](https://github.com/stanford-crfm/helm), [leaderboards](https://crfm.stanford.edu/helm/) | Accuracy plus efficiency, bias, toxicity, robustness and safety across LLM/VLM suites; web leaderboards and result runs. | Entered maintenance mode June 1 2026; existing leaderboards remain inspectable/reproducible. | Python CLI (`helm-run`, `helm-summarize`), local server, downloadable run artifacts; no hosted API promised. | Apache-2.0 [LICENSE](https://github.com/stanford-crfm/helm/blob/main/LICENSE); individual datasets/model outputs can have separate terms. |
-   624→| **Hugging Face Open LLM Leaderboard results** — [hub org](https://huggingface.co/open-llm-leaderboard), [results dataset](https://huggingface.co/datasets/open-llm-leaderboard/results) | Open-model academic capability scores (MMLU-style, GPQA, IFEval, etc.), not throughput/latency. | Bot-driven/irregular; results dataset metadata showed last modified 2025-03-15 in this check, while requests dataset updates more frequently. | HF datasets API/Hub download (Parquet/Arrow via `datasets`); no separate API required. | Results dataset metadata had no explicit license in API response; commercial reuse **not confirmed**. Verify each dataset/model card before redistribution. |
-   625→| **LMSYS Chatbot Arena conversations/leaderboard** — [dataset](https://huggingface.co/datasets/lmsys/chatbot_arena_conversations), [FastChat repo](https://github.com/lm-sys/FastChat) | Human pairwise preferences and Elo-style ranking from live user votes; qualitative “in the wild” quality, not hardware throughput. | Frequently refreshed leaderboard; raw conversation dataset snapshot is older/static and official Arena has no guaranteed public API. | HF dataset download (Parquet/JSON); FastChat includes evaluation/serving code. | FastChat code Apache-2.0, but dataset is mixed: prompts CC-BY-4.0, model outputs CC-BY-NC-4.0; **not safe for commercial output redistribution**. [dataset card](https://huggingface.co/datasets/lmsys/chatbot_arena_conversations). |
-   626→| **EleutherAI lm-evaluation-harness** — [repo](https://github.com/EleutherAI/lm-evaluation-harness) | 60+ academic task families, hundreds of variants; log-likelihood/generation accuracy, not serving throughput. | Active; main pushed Sep 14 2026 and release v0.4.13 Aug 31 2026 per repo metadata. | CLI, Python `simple_evaluate()` API; emits local JSON/results; no centralized hosted result DB. | MIT [LICENSE](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/LICENSE.md), but each benchmark dataset has own terms. |
-   627→| **Hugging Face Lighteval / Open Benchmark Index** — [repo](https://github.com/huggingface/lighteval), [docs](https://huggingface.co/docs/lighteval/en/index), [index Space](https://huggingface.co/spaces/OpenEvals/open_benchmark_index) | Broad evaluation task/metric runner across HF, vLLM, TGI, API endpoints; accuracy/coding/reasoning and custom metrics. | Active; repo pushed Sep 17 2026; result storage is run-dependent. | Python/CLI API; can store results on HF Hub, S3 or local; no single canonical CSV DB. | Lighteval code MIT [LICENSE](https://github.com/huggingface/lighteval/blob/main/LICENSE); datasets/tasks may have individual terms. |
-   628→| **NVIDIA AIPerf** — [repo](https://github.com/ai-dynamo/aiperf) | Real serving measurements: request latency, TTFT/ITL, p50/p90/p99, output-token throughput/user and aggregate throughput; designed for generative APIs. | Active; pushed Sep 19 2026. | CLI output and detailed reports; OpenTelemetry metric streaming (`aiperf[otel]`); no central result DB. | Apache-2.0 [LICENSE](https://github.com/ai-dynamo/aiperf/blob/main/LICENSE). |
-   629→| **Triton Perf Analyzer** — [repo](https://github.com/triton-inference-server/perf_analyzer) | Load-test throughput and latency under concurrency/request-rate/time/count windows for Triton/TorchServe; not LLM-specific unless serving an LLM endpoint. | Active repo, but README warns genai-perf is deprecated and points to AIPerf. | CLI; console/log output and measurement artifacts, no public central database/API. | BSD-3-Clause [LICENSE](https://github.com/triton-inference-server/perf_analyzer/blob/main/LICENSE). |
-   630→| **vLLM benchmark scripts** — [repo](https://github.com/vllm-project/vllm), benchmark docs under `benchmarks/` | Local LLM serving throughput, TTFT/ITL and latency under request/concurrency/prompt-output workloads; highly relevant to tokcalc validation. | Active repo; pushed Sep 20 2026. | Clone/run scripts; emits JSON/CSV-style local benchmark output depending on command; no canonical public result DB. | Apache-2.0 [LICENSE](https://github.com/vllm-project/vllm/blob/main/LICENSE). |
-   631→
-   632→## Integration recommendation
-   633→Use MLPerf/HELM/HF datasets for accuracy and standardized reference points, but use AIPerf + vLLM for direct tokcalc throughput/latency validation. Store each benchmark point with `provider`, `model_id`, `engine_version`, `hardware`, `quantization`, `batch/concurrency`, `input_tokens`, `output_tokens`, `TTFT`, `ITL`, `tokens_per_second`, `timestamp`, `source_url`, and `license_status`; never merge different batch/concurrency/context conditions into one “speed” number.
-   634→
-   635→## Gaps/attempts
-   636→- Official pricing pages for OpenAI/Anthropic/Gemini and several docs are dynamic; direct HTML parsing was used where extraction timed out.
-   637→- Current official catalogs have moved on substantially by 2026-09-20: Anthropic 3.x, Gemini 1.5/2.0, Mistral Large 2, DeepSeek V3/R1, Groq Mixtral/Llama 3.1 70B, and Cerebras Llama 3.1 rows are retired, absent, or enterprise-only. Historical prices are included solely as labeled baselines.
-   638→- Together/FIreworks exact cached rates are not published on every requested model card; generic provider defaults are not a substitute for a model-specific quote.
-   639→- vLLM/Triton/AIPerf/lm-eval/Lighteval are benchmark runners, not pre-populated databases; this distinction matters for tokcalc’s ingestion design.
-   640→
-   641→
-   642→## TL;DR for the maintainer — do these five things this week
-   643→
-   644→1. **Ship the workload schema and KV-cache calculator first.** Add prompt length, output length, concurrency/QPS, TTFT, ITL/TPOT, prefill/decode split, KV precision, and memory-fit warnings.
-   645→2. **Add prompt-cache economics.** Let users enter prefix length and hit rate, then show first-hit versus steady-state API cost and TTFT; implement provider-specific TTL/write/read rules.
-   646→3. **Add reasoning/agent scenarios.** Model hidden reasoning tokens, tool schemas/results, sequential rounds and parallel tool calls instead of multiplying output length by a guessed constant.
-   647→4. **Build timestamped provider pricing and benchmark imports.** Prefer AIPerf/vLLM for throughput validation, MLPerf for standardized system references, and keep retired/enterprise-only API rows visible but never current.
-   648→5. **Add engine-aware quantization and a small high-value catalog refresh.** Start with GGUF Q4_K_M/Q5_K_M/Q6_K/Q8_0, GPTQ/AWQ/EXL2 W4A16, FP8, plus Llama 4, Qwen 3, DeepSeek V3/R1 and current GPU rows; do not promise universal accuracy or speedup percentages.
-   649→
-   650→### Research limitations
-   651→
-   652→This is a deep, source-backed snapshot, not a guarantee that every provider price or model catalog remains current after the cutoff. Several provider pages are dynamic, some requested models are retired by the cutoff, and some vendors do not publish dense FP16/BF16 or exact cloud rates. Those fields are intentionally left unavailable in the tables rather than inferred.
-   653→