@nxuss/lemma 1.5.0 → 1.7.0

package/README.md

@@ -1,11 +1,11 @@
-# 🧠 Lemma v1.4.0
+# 🧠 Lemma v1.7.0
 
 [![npm version](https://img.shields.io/npm/v/@nxuss/lemma.svg?color=orange&style=flat-square)](https://www.npmjs.com/package/@nxuss/lemma)
 [![License](https://img.shields.io/npm/l/@nxuss/lemma.svg?color=blue&style=flat-square)](https://github.com/Nxusbets/lemma/blob/main/LICENSE)
-[![MCP Tools](https://img.shields.io/badge/MCP%20tools-92-ff69b4?style=flat-square)](https://github.com/Nxusbets/lemma)
+[![MCP Tools](https://img.shields.io/badge/MCP%20tools-94-ff69b4?style=flat-square)](https://github.com/Nxusbets/lemma)
 [![Security](https://img.shields.io/badge/privacy-firewall%20active-blueviolet?style=flat-square)](https://github.com/Nxusbets/lemma)
 [![Caching](https://img.shields.io/badge/semantic%20memory-enabled-success?style=flat-square)](https://github.com/Nxusbets/lemma)
-[![Tests](https://img.shields.io/badge/tests-199%20passing-00c853?style=flat-square)](https://github.com/Nxusbets/lemma)
+[![Tests](https://img.shields.io/badge/tests-267%20passing-00c853?style=flat-square)](https://github.com/Nxusbets/lemma)
 
 ```text
     __                                
@@ -17,13 +17,125 @@
 
 > **The Intelligent local-first AI Gateway — Privacy, Performance, Static Code Intelligence, and Infrastructure Command Center for the Agentic Era.**
 
-Lemma is a high-performance orchestration layer that sits between your development environment and LLM providers. It transforms the way you build with AI by providing **Shared Semantic Memory**, **Autonomous Cost Optimization**, **Privacy Guardrails**, **22 Infrastructure Management Tools**, and a suite of **92 MCP tools** — including zero-token static code intelligence that runs entirely on your machine with no LLM calls required.
+Lemma is a high-performance orchestration layer that sits between your development environment and LLM providers. It transforms the way you build with AI by providing **Shared Semantic Memory**, **Autonomous Cost Optimization**, **Privacy Guardrails**, **22 Infrastructure Management Tools**, and a suite of **94 MCP tools** — including zero-token static code intelligence that runs entirely on your machine with no LLM calls required.
+
+Since v1.7.0 those 94 tools are all registered, but only the ones your project can actually use
+are advertised to the model each turn. The rest stay one `lemma_toolbox` call away. See
+[Tool profiles](#-tool-profiles).
 
 ---
 
 ## 📋 Changelog
 
-### v1.4.0 — Correctness-first patching, search, and memory freshness *(NEW)*
+### v1.7.0 — The cost of being connected *(NEW)*
+
+Every tool schema Lemma advertises goes into the model's system prompt and is **re-sent on every
+request of the session** — including schemas for tools the project can never use. On this repo that
+was ~13,950 tokens per turn before Lemma saved anything.
+
+| Change | What it does | Why it matters |
+| :--- | :--- | :--- |
+| **Project-scoped tool surface** | Core tools are always advertised; situational groups (infrastructure, review) only when the repo shows evidence they apply. `mcp.toolProfile` accepts `auto`, `core`, or `full`. | ~3,420 tokens of schemas per turn instead of ~11,600. |
+| **`lemma_toolbox`** *(new)* | Lists tools whose schemas were withheld, returns any tool's schema, and invokes it. | A withheld schema is a deferred cost, never a lost capability. |
+| **Leaner TURBOMODE prompt** | ~2,340 → ~1,700 tokens. Instructions for tools the session doesn't advertise are gone; every correctness rule stays, enforced by tests. | The prompt stops paying for advice the model can't act on directly. |
+| **Overhead in `token_budget`** | Reports profile, advertised vs. hidden counts, active groups, and the real schema token cost. | The fixed cost is visible instead of assumed. |
+
+**Total fixed cost per turn on this repo: ~5,120 tokens, down from ~13,950.**
+
+**The Brain reported cache hits that never happened.** Duplicate detection runs an internal search
+before storing a memory, and that search shared the counters behind the reported hit rate — while
+searching with a minimum similarity of 0, so it matched nearly every time. Every store inflated the
+session hit rate and bumped an unrelated entry's hit counter. Internal lookups no longer count.
+**Reported hit rates will be lower after upgrading; the lower number is the correct one.**
+
+**Three features silently did nothing.** `get_telepathic_hints`, `auto_context_bundle`, and
+`test_autofix_interceptor` read prior knowledge from `<project>/.lemma/memory/` — a directory no
+code path writes. They returned empty in every project while the Brain held the answers. All three
+now query the Brain directly.
+
+**Also in this release:**
+
+- **Memories are scoped by project** via a `projectId` derived from the git remote. Entries stored
+  before this release stay visible everywhere rather than disappearing.
+- **Brain entries are ~38% smaller** — `terms` duplicated `Object.keys(termFreq)` and is now
+  rebuilt on load instead of stored. Existing files are read unchanged.
+- **`get_project_onboarding` works without the proxy**, falling back to a locally derived map
+  instead of returning an error on the first call of the session.
+- **`lemma-checkpoint`** *(new CLI)* captures a checkpoint from repository state, so it no longer
+  depends on the assistant remembering to call the tool. See [Session checkpoints](#-session-checkpoints).
+- **Removed the `lemma-autopilot` prompt** — it told the model to call tools unconditionally, the
+  opposite of the net-savings accounting added in v1.6.0. `lemma-turbomode` is the supported prompt.
+
+---
+
+### v1.6.0 — The savings ledger can now report a loss
+
+Every release up to this one measured only what Lemma **saved**. That is single-entry bookkeeping:
+a number that can only go up is not evidence of anything, because the ledger had no way to express
+the outcome *"this session cost more than it returned."* Meanwhile Lemma's own overhead — the
+TURBOMODE prompt and 93 tool schemas injected into the model's system prompt, plus every tool
+result that lands in context — was real spend that appeared nowhere.
+
+| Change | What it does | Why it matters |
+| :--- | :--- | :--- |
+| **Cost side of the ledger** | Records `mcpInstructions` (prompt + all tool schemas, charged **once at MCP connect** — ~13,700 tokens on this repo), `toolResult` (charged on **every** call, including ones that also report a saving), and `cacheMiss`. | A session now starts visibly underwater and has to earn its way out, instead of starting at zero and only ever climbing. |
+| **Net position** | `getSnapshot()` returns `net.netTokens`, `net.netCost`, and `net.ratio` (saved ÷ spent). The terminal summary prints **`NET COST`** instead of `NET SAVED` when the ratio drops below 1.0. | The report is falsifiable. If Lemma isn't paying for itself, it says so. |
+| **`POST /api/cost-record`** *(new)* | Counterpart to `/api/savings-record`, so out-of-process callers (MCP server, CLI) can charge the ledger. Unknown sources are rejected with a 400. | Costs incurred outside the proxy were previously unrecordable. |
+| **Versioned ledger file** | `_version: 2`. Savings stay top-level so pre-v2 readers keep working; costs are additive under `_costs`. A v1 ledger loads with zeroed costs rather than failing. | No migration step, no data loss on upgrade. |
+
+**Three ways the old numbers were wrong**, now fixed:
+
+- **Cache hits were credited a hardcoded 2,000 tokens** — every semantic, synthesized, and cloud
+  hit reported the same invented figure regardless of what was actually served. Savings are now
+  measured from the cached payload that replaced the LLM call. The 2,000 placeholder survives only
+  in `stats.json` for dashboard continuity and is explicitly **barred from the ledger**: a guess
+  must not become part of a net-position claim.
+- **`pruneHistoryMessages` counted chars as tokens**, inflating history-prune savings ~4x. It now
+  converts with the same `chars / 4` heuristic used everywhere else.
+- **A cache *miss* is not charged.** The lookup is local, so a miss costs latency, not tokens.
+  Charging it an invented figure would repeat the exact error this release exists to correct.
+
+**Also fixed**: `reportSessionOverhead()` ran *after* `await trySamplingContext()`, which blocks on
+a `sampling/createMessage` round-trip that clients without sampling support never answer — so the
+overhead charge was stalled until the SDK timeout, or dropped entirely. It now runs first.
+
+**TURBOMODE prompt rewritten** at the top: it no longer promises to "multiply your LLM membership
+50x" or tell the model to poll `token_budget` every 5 turns (a scheduled call that produces a
+report instead of work, and costs tokens each time). It now states the real goal — reduce what
+enters context without costing a correct answer — and warns that a tool call which saves less than
+its own round-trip is a net loss.
+
+---
+
+### v1.5.0 — Command output stops being the biggest unoptimized token sink
+
+`run_workspace_command` returned raw stdout/stderr in full — a failing `npm test` put thousands of
+lines into context to communicate three broken assertions. Long output is now distilled by
+deterministic per-format parsers (no LLM summarization: a summarizer that invents a test name is
+worse than the dump it replaced).
+
+| Format | What survives distillation |
+| :--- | :--- |
+| jest / vitest | Failing test titles, assertion reason and diff, project-local source location. Code frames and `node_modules` stack frames dropped. |
+| tsc | Grouped by file; a repeated diagnostic printed once with its positions (`TS2322 ×40 … at 1:7, 2:7, …`) instead of forty identical lines. |
+| eslint (stylish) | Grouped by file with a rule-frequency header, capped per file. |
+| anything else | Head, tail, and the error/failure lines from the middle, each with its line number, plus an explicit count of what was withheld. |
+
+Measured on this repo: a 30-failure jest run went from 32,324 → 7,183 chars (78% withheld); a
+65-error `tsc --noEmit` from 5,597 → 527 (91%).
+
+**Lossless, not lossy.** The complete output is written to disk *before* anything is trimmed and
+the reply carries a handle; the footer always states how much was withheld. `output_region` *(new
+tool)* retrieves stored output verbatim — the whole run, one named section (a failing test title or
+file path), or a line range. Small outputs pass through untouched with no handle.
+
+**Also fixed**: stderr was discarded on success. `execSync` leaves stderr attached to the parent
+unless `stdio` is overridden, so a *passing* jest or tsc run — both report on stderr — returned an
+empty "Output:". Switched to `spawnSync`, which also raises the output limit to 20MB.
+
+---
+
+### v1.4.0 — Correctness-first patching, search, and memory freshness
 
 An audit of the everyday MCP path — read a file, search the workspace, patch it, remember the
 answer — found the same failure mode repeating: a tool that looked done but silently degraded
@@ -173,6 +285,70 @@ Or use environment variables: `JENKINS_URL`, `JENKINS_USERNAME`, `JENKINS_API_TO
 
 ---
 
+## 🎚️ Tool profiles
+
+Every tool schema an MCP server advertises is injected into the model's system prompt and re-sent
+on **every request** of the session. Advertising all 94 tools costs ~11,600 tokens per turn — much
+of it schemas for tools a given project can never use.
+
+Lemma advertises a scoped surface instead. Nothing is removed: every tool stays registered and
+callable, and `lemma_toolbox` reaches the ones whose schemas were withheld.
+
+| Profile | Advertises |
+| :--- | :--- |
+| `auto` *(default)* | Core tools, plus situational groups the repo shows evidence for |
+| `core` | Core tools only |
+| `full` | Everything, as before v1.7.0 |
+
+```jsonc
+// lemma.config.json
+{ "mcp": { "toolProfile": "full" } }   // or set LEMMA_TOOL_PROFILE=full
+```
+
+**Core** covers workspace reading and navigation, the patch/validate write path, AST context,
+caching (`state_hash_cache`, `smarter_cache`, `search_memory`, `store_memory`), routing advice, and
+the ledger. **Situational groups** are advertised on evidence — the infrastructure group when a
+`Jenkinsfile`, `kustomization.yaml`, `.openshift/`, or a `.tf` file is present, or a provider is
+already connected.
+
+Reaching anything else:
+
+```jsonc
+lemma_toolbox { "action": "list",   "query": "terraform drift" }   // what exists
+lemma_toolbox { "action": "schema", "tool":  "terraform_drift" }   // its inputs
+lemma_toolbox { "action": "call",   "tool":  "terraform_drift", "args": {} }
+```
+
+`token_budget` reports the profile in use, advertised vs. hidden counts, and what the advertised
+schemas actually cost this session.
+
+---
+
+## 💾 Session checkpoints
+
+`conversation_checkpoint` saves what a session established so the next one doesn't re-derive it —
+but it only runs if the assistant remembers to call it at the end, which is not something to build
+on. `lemma-checkpoint` derives a checkpoint from repository state instead:
+
+```bash
+lemma-checkpoint [projectRoot] [--summary "text"]
+```
+
+It records the branch, changed files, and recent commits to `<project>/.lemma/session/`, exits 0
+even outside a git repository, and is meant to be wired to your host's session-end event:
+
+```jsonc
+// ~/.claude/settings.json
+{ "hooks": { "SessionEnd": [{ "hooks": [{ "type": "command",
+  "command": "lemma-checkpoint \"${CLAUDE_PROJECT_DIR:-$PWD}\"" }] }] } }
+```
+
+Decisions made and tasks left pending exist only in the conversation, so an automatic capture
+leaves those fields empty rather than inventing them — a checkpoint saved by the assistant stays
+the richer record when one exists.
+
+---
+
 ## 💎 Free vs Pro
 
 ### 🆓 Free (33 Tools)
@@ -441,7 +617,7 @@ lemma start
 | `lemma start` | Launches proxy, cache engine, and dashboard at `http://localhost:8081`. |
 | `lemma stop` | Gracefully shuts down the background proxy. |
 | `lemma status` | Health of all modules and active connections. |
-| `lemma stats` | Real-time token and cost savings metrics. |
+| `lemma stats` | Real-time token and cost metrics — savings, Lemma's own overhead, and the net position between them. Reports `NET COST` when the session is underwater. |
 | `lemma mcp` | Starts the stdin/stdout MCP server for IDE integration. |
 | `lemma heal` | Auto-diagnose and repair the latest server crash from live context logs. |
 | `lemma autopilot` | Toggle autonomous background watcher mode (auto-applies fixes). |

package/bin/checkpoint.js

@@ -0,0 +1,49 @@
+#!/usr/bin/env node
+'use strict';
+
+/**
+ * Session-end checkpoint capture.
+ *
+ * Meant to be wired to a host's session-end event (e.g. a Claude Code Stop/SessionEnd hook)
+ * so a checkpoint gets written even when the assistant never calls the MCP tool. Writes to
+ * <project>/.lemma/session/ and exits 0 no matter what — a hook that fails must never be the
+ * reason a session ends badly.
+ *
+ * Usage: lemma-checkpoint [projectRoot] [--summary "text"]
+ */
+
+const path = require('path');
+
+function main() {
+  const args = process.argv.slice(2);
+  const summaryIndex = args.indexOf('--summary');
+  const summary = summaryIndex >= 0 ? args[summaryIndex + 1] : undefined;
+  const positional = args.filter((a, i) => !a.startsWith('--') && i !== summaryIndex + 1);
+  const projectRoot = path.resolve(positional[0] || process.cwd());
+
+  const distPath = path.join(__dirname, '..', 'dist', 'cjs', 'utils', 'ConversationCheckpoint.js');
+  let captureAutoCheckpoint;
+  try {
+    ({ captureAutoCheckpoint } = require(distPath));
+  } catch {
+    console.error('[lemma-checkpoint] Build not found. Run "npm run build" first.');
+    return;
+  }
+
+  const result = captureAutoCheckpoint(projectRoot, summary);
+  if (!result) {
+    console.error('[lemma-checkpoint] Not a git repository — nothing to capture.');
+    return;
+  }
+  console.error(
+    `[lemma-checkpoint] Saved: ${result.filesModified.length} file(s) changed → ${path.join(projectRoot, '.lemma/session')}`
+  );
+}
+
+try {
+  main();
+} catch (err) {
+  // A checkpoint is best-effort context, never a reason to fail the host's shutdown.
+  console.error('[lemma-checkpoint] Skipped:', err && err.message);
+}
+process.exit(0);

package/bin/init.js

@@ -278,6 +278,25 @@ tokens sale más cara que los tokens que ahorra.
 | "¿Cuánto ahorramos de verdad?" | \`mcp__lemma__token_receipt\` | Bitácora auditable real, no una cifra inventada |
 | Guardar una solución para el futuro | \`mcp__lemma__store_memory\` | Solo después de verificarla (tests pasaron, el usuario confirmó) — una solución mala guardada se vuelve un cache hit falso más adelante |
 
+## No todas las tools de Lemma están en tu lista
+
+Cada schema que Lemma anuncia se re-envía en **cada** request de la sesión, así que solo se
+anuncian las que este proyecto puede usar de verdad. El resto sigue registrado y disponible:
+
+- \`mcp__lemma__lemma_toolbox\` con \`action="list"\` (y un \`query\`) para ver qué existe.
+- \`action="schema"\` para leer los inputs de una tool antes de llamarla.
+- \`action="call"\` para ejecutarla.
+
+Antes de decirle al usuario que Lemma no puede hacer algo, revisa el toolbox. Si el proyecto
+necesita todas las tools anunciadas de forma permanente, pon \`mcp.toolProfile: "full"\` en
+\`lemma.config.json\`.
+
+## Antes de pedirle código al usuario
+
+No le pidas que pegue archivos, logs o documentación que puedes obtener tú: agota primero
+\`search_workspace\`, \`get_ast_hologram\` (estructura: da nombres y líneas, no firmas) y
+\`read_workspace_file\`. Cada ida y vuelta con el usuario cuesta más que la lectura.
+
 ## Al iniciar en este proyecto
 
 1. Llama \`mcp__lemma__get_project_onboarding\` para orientarte con el contexto ya indexado.
@@ -286,8 +305,8 @@ tokens sale más cara que los tokens que ahorra.
 
 ## Al reportar o investigar un bug
 
-1. \`mcp__lemma__get_telepathic_hints\` con el archivo relevante para contexto de runtime.
-2. \`mcp__lemma__search_memory\` con el mensaje de error, por si ya se resolvió algo parecido.
+1. \`mcp__lemma__search_memory\` con el mensaje de error, por si ya se resolvió algo parecido.
+2. \`lemma_toolbox\` → \`get_telepathic_hints\` con el archivo relevante para contexto de runtime.
 
 ## Una sola regla dura
 

package/dist/cjs/cli/lemma-proxy.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"lemma-proxy.d.ts","sourceRoot":"","sources":["../../../src/cli/lemma-proxy.ts"],"names":[],"mappings":";AA4cA,wBAAgB,+BAA+B,CAAC,OAAO,CAAC,EAAE;IAAE,SAAS,CAAC,EAAE,OAAO,CAAA;CAAE,GAAG,IAAI,CA4DvF;AAijED,wBAAsB,eAAe,CAAC,KAAK,EAAE,OAAO,GAAG,OAAO,CAAC;IAAE,OAAO,EAAE,OAAO,CAAC;IAAC,OAAO,EAAE,MAAM,CAAC;IAAC,KAAK,CAAC,EAAE,MAAM,CAAC;IAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;IAAC,UAAU,CAAC,EAAE,MAAM,CAAC;IAAC,SAAS,CAAC,EAAE,MAAM,CAAA;CAAE,CAAC,CA6KhL"}
+{"version":3,"file":"lemma-proxy.d.ts","sourceRoot":"","sources":["../../../src/cli/lemma-proxy.ts"],"names":[],"mappings":";AA4cA,wBAAgB,+BAA+B,CAAC,OAAO,CAAC,EAAE;IAAE,SAAS,CAAC,EAAE,OAAO,CAAA;CAAE,GAAG,IAAI,CA4DvF;AA8lED,wBAAsB,eAAe,CAAC,KAAK,EAAE,OAAO,GAAG,OAAO,CAAC;IAAE,OAAO,EAAE,OAAO,CAAC;IAAC,OAAO,EAAE,MAAM,CAAC;IAAC,KAAK,CAAC,EAAE,MAAM,CAAC;IAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;IAAC,UAAU,CAAC,EAAE,MAAM,CAAC;IAAC,SAAS,CAAC,EAAE,MAAM,CAAA;CAAE,CAAC,CA6KhL"}

package/dist/cjs/cli/lemma-proxy.js

@@ -587,6 +587,31 @@ async function loadUsage() {
 async function loadStats() {
     return readJson(ProjectStore_1.STATS_FILE, {});
 }
+/**
+ * Tokens a cache hit actually avoided regenerating, measured from the cached payload
+ * that was served instead of an LLM call. Replaces the old hardcoded 2000 guess.
+ */
+function measureCachedTokens(data) {
+    try {
+        const chars = JSON.stringify(data)?.length;
+        if (!chars)
+            return undefined;
+        return Math.floor(chars / 4);
+    }
+    catch {
+        return undefined;
+    }
+}
+/**
+ * @param tokensSaved  Measured tokens saved. When omitted for a cache hit, stats.json
+ *   still falls back to a 2000-token placeholder for dashboard continuity, but that
+ *   guess is deliberately NOT written to the savings ledger — the ledger only accepts
+ *   measured values, otherwise its net position is built on invented numbers.
+ *
+ * Note: a cache *miss* is not charged to the cost ledger. The lookup is local (no LLM
+ * round-trip), so a miss costs latency, not tokens. Charging it a fabricated token
+ * figure would repeat exactly the error the cost ledger exists to correct.
+ */
 async function recordStat(stats, project, fromCache, latencyMs, provider, tokensSaved) {
     if (!stats[project]) {
         stats[project] = { total: 0, hits: 0, misses: 0, totalLatency: 0, totalTokensSaved: 0, providers: {} };
@@ -600,9 +625,11 @@ async function recordStat(stats, project, fromCache, latencyMs, provider, tokens
     if (!s.providers[provider])
         s.providers[provider] = { hits: 0, misses: 0 };
     fromCache ? s.providers[provider].hits++ : s.providers[provider].misses++;
-    if (fromCache && tkSaved > 0) {
-        // Automatically report cache hits directly to the server session ledger for absolute sync consistency
-        SavingsLedger_1.savingsLedger.recordTokens('cache', tkSaved);
+    // Only measured savings reach the ledger. `tokensSaved === undefined` means the
+    // caller never counted, so tkSaved is the 2000 placeholder — usable for the
+    // dashboard, not for a net-position claim.
+    if (fromCache && typeof tokensSaved === 'number' && tokensSaved > 0) {
+        SavingsLedger_1.savingsLedger.recordTokens('cache', tokensSaved);
     }
     logEvent({
         type: fromCache ? 'cache:hit' : 'cache:miss',
@@ -1508,6 +1535,23 @@ class LemmaServer {
                 }
                 res.json({ ok: true, snapshot: SavingsLedger_1.savingsLedger.getSnapshot().total });
             });
+            // Cost Record Endpoint — the counterpart to /api/savings-record. Without it the
+            // ledger only ever counts wins, which makes its totals unfalsifiable.
+            this.app.post('/api/cost-record', (req, res) => {
+                const { source, tokens, chars } = req.body || {};
+                const validSources = ['mcpInstructions', 'toolResult', 'cacheMiss'];
+                if (!source || !validSources.includes(source)) {
+                    return res.status(400).json({ error: 'Invalid cost source' });
+                }
+                if (typeof tokens === 'number' && tokens > 0) {
+                    SavingsLedger_1.savingsLedger.recordCost(source, tokens);
+                }
+                else if (typeof chars === 'number' && chars > 0) {
+                    SavingsLedger_1.savingsLedger.recordCostChars(source, chars);
+                }
+                const snap = SavingsLedger_1.savingsLedger.getSnapshot();
+                res.json({ ok: true, totalCost: snap.totalCost, net: snap.net });
+            });
             // MCP Tool Event Endpoint — called by MCP server when tools execute
             this.app.post('/api/mcp-event', (req, res) => {
                 try {
@@ -1733,7 +1777,7 @@ class LemmaServer {
             const semHit = await semanticGet(provider, prompt, this.projectName);
             if (semHit) {
                 if (semHit.similarity >= 0.90) {
-                    await recordStat(this.stats, this.projectName, true, Date.now() - t0, provider, 2000);
+                    await recordStat(this.stats, this.projectName, true, Date.now() - t0, provider, measureCachedTokens(semHit.data));
                     const unmaskedData = semanticScrubber.unmask(semHit.data, tokenMap);
                     const cacheHeader = semHit.hiveMind ? 'HIT-HIVE-MIND' : 'HIT';
                     if (isStream)
@@ -1784,7 +1828,7 @@ Adjusted Answer:`;
                                 else if (provider === 'anthropic') {
                                     synthesizedData.content[0].text = adjustedText;
                                 }
-                                await recordStat(this.stats, this.projectName, true, Date.now() - t0, provider, 2000);
+                                await recordStat(this.stats, this.projectName, true, Date.now() - t0, provider, measureCachedTokens(synthesizedData));
                                 const unmaskedData = semanticScrubber.unmask(synthesizedData, tokenMap);
                                 if (isStream) {
                                     return SseRelay_1.SseRelay.simulateHit(res, unmaskedData, { provider: provider, tier, similarity: semHit.similarity });
@@ -1804,7 +1848,7 @@ Adjusted Answer:`;
             const cloudHit = await cloudSync.get(prompt);
             if (cloudHit) {
                 cacheSet(provider, prompt, cloudHit.data);
-                await recordStat(this.stats, this.projectName, true, Date.now() - t0, provider, 2000);
+                await recordStat(this.stats, this.projectName, true, Date.now() - t0, provider, measureCachedTokens(cloudHit.data));
                 const unmaskedData = semanticScrubber.unmask(cloudHit.data, tokenMap);
                 if (isStream)
                     return SseRelay_1.SseRelay.simulateHit(res, unmaskedData, { provider: provider, tier, similarity: cloudHit.similarity || 0.95 });