pi-mega-compact 0.13.7 → 0.14.1

package/README.md

@@ -13,13 +13,14 @@ A local-first context compressor for the [pi coding agent](https://github.com/ea
 - **Durable memory** — on a cadence the store auto-reviews and safe-keeps decisions, facts, and preferences as first-class RAG memories, so long-running projects remember what mattered.
 - **Prompt-cache optimization** — message separation + cache striping (default OFF, opt-in). Targets 82-90% cache hit rate by structuring context around provider cache boundaries. Enables with `MEGACOMPACT_MESSAGE_SEPARATION` and `MEGACOMPACT_CACHE_STRIPING`.
 - **Context health + KV cache poison validation** (v0.12) — a real-time composite 0-1 health score per turn from five sub-scores (drift, output quality, error rate, cache health, cache poison). Catches garbled/hallucinated output and provider-side KV-cache corruption *before* they waste tokens — the failure mode where a large-context model (e.g. DeepSeek V4 Flash, 1M window) degrades at <1% usage. Tri-layer cache poison validation: prefix hash (L1 FNV-1a), output-quality-by-cache-hit (L2 semantic), error-rate correlation (L3 behavioral). The dashboard **Health tab** shows a gauge, sparkline, sub-score bars, alerts, and per-model breakdown. Auto-mitigation (force compaction on degraded context, prefix break on poisoned cache) is default OFF — toggle it in the Maintenance tab.
-- **RAG suite** — query reformulation (TF-IDF + RRF), tiered routing (L0 cache -> L1 FTS5 -> L2 PGlite), recall-quality metrics (CRAG), memory graph, and HyDE (hypothetical document embeddings). All default ON with graceful fallback — opt out via `MEGACOMPACT_<NAME>_DISABLED=true`. HyDE auto-activates only when an LLM embedder (Ollama/HTTP) is configured — it's a no-op with the default TrigramEmbedder. Toggle any flag from the dashboard Setup tab.
-- **Stacked memory graph** — the dashboard shows memory composition over time from 3 content sources (turns, durable memories, wiki) with a 9-gate validation system and a graph-health indicator. Per-model provider cache breakdown in the Cache tab.
+- **RAG suite** — query reformulation (TF-IDF + RRF), tiered routing (L0 cache -> L1 FTS5 -> L2 PGlite), recall-quality metrics (CRAG), memory graph, and HyDE (hypothetical document embeddings). All default ON with graceful fallback — opt out via `MEGACOMPACT_<NAME>_DISABLED=true`. HyDE auto-activates only when an LLM embedder (Ollama/HTTP) is configured — it's a no-op with the default TrigramEmbedder. The dashboard **Metrics tab** (v0.13) shows RAG recall-quality metrics (pass rate, avg lift, telemetry turns, HyDE runs) and the Overview tab has a RAG health card. Toggle any flag from the dashboard Setup tab.
 - **Debug bundle** — Maintenance tab in the dashboard has a **Gather Debug Logs** button that collects events, config, and store state into a shareable archive for bug reports.
-- **Local-first by default** — node:sqlite + trigram embeddings by default, zero calls off your machine. Bring your own localhost embedder (Ollama, ONNX, TEI) for better semantic matches — the embedder endpoint is loopback-only by default (`MEGACOMPACT_ALLOW_REMOTE_EMBEDDER=1` opts in to a remote/third-party endpoint). The optional dashboard is localhost-only. No cloud, no API calls, no telemetry.
+- **Local-first by default** — node:sqlite + trigram embeddings by default, zero calls off your machine. Bring your own localhost embedder (Ollama, ONNX, TEI) for better semantic matches — the embedder endpoint is loopback-only by default (`MEGACOMPACT_ALLOW_REMOTE_EMBEDDER=1` opts in to a remote/third-party endpoint). The optional dashboard is localhost-only. No cloud, no API calls, no telemetry. The optional Cost API lookup (`MEGACOMPACT_COST_API_ENABLED`, default OFF) is the one network exception — it fetches model pricing from a user-configured endpoint to show real $/token rates in the dashboard.
 - **Team-run aware** — fine-grained durable trim fires at agent settle during sub-agent runs, so long multi-agent work doesn't just collapse at the end.
-- **Multi-pi dashboard** — one dashboard tab per active pi process with the context stack, per-repo stats, and a live SSE feed across all of them. The React SPA has 15 lazy-loaded tabs — **Overview**, **Cache**, **Sessions**, **Turns** (per-turn memory + recall + rewind), and **Health** (context-health gauge + cache-poison alerts) are primary; **Repos**, **Events**, **Config**, **Setup** (embedding wizard), **Metrics** (perf latency/TPS/CPU), **Game**, **Achievements**, **Topics** (wiki), **Memory Map** (D3 graph), and **Maintenance** (debug bundle) are advanced.
-- **Auto-categorizing wiki.** Every 3 compactions (seeds from turns before that), the store clusters your real memory embeddings (k-means) and labels each cluster with its most discriminative terms (TF-IDF) — no LLM, no Ollama, fully local. The dashboard **Wiki tab** browses topics, searches by label or term, and drills down into the member memories of each cluster.
+- **Multi-pi dashboard** — one dashboard tab per active pi process with the context stack, per-repo stats, and a live SSE feed across all of them. The React SPA (Tailwind v3 + shadcn/ui, v0.13) has 12 lazy-loaded tabs — **Overview**, **Cache**, **Sessions**, and **Turns** (per-turn memory + recall + rewind) are primary; **Repos**, **Events**, **Setup** (embedding wizard + comprehensive settings panel), **Metrics** (perf latency/TPS/CPU + RAG recall quality), **Wiki** (auto-categorizing topic browser + evolution graph), **Memory Map** (D3 graph), **Maintenance** (debug bundle), and **Health** (context-health gauge + cache-poison alerts) are advanced. The Overview tab dynamically resolves the most recently active repo's snapshot so it always shows live data.
+- **Auto-categorizing wiki.** Every 3 compactions (seeds from turns before that), the store clusters your real memory embeddings (k-means) and labels each cluster with its most discriminative terms (TF-IDF) — no LLM, no Ollama, fully local. The dashboard **Wiki tab** (v0.13) browses topics, searches by label or term, drills down into the member memories of each cluster, and shows a topic evolution graph over time. Curation supports rename/merge/split operations with durable overrides that survive full rebuilds.
+- **Comprehensive settings panel** (v0.13.6) — every adjustable `MEGACOMPACT_*` env var is surfaced in the dashboard Setup tab, grouped by category (RAG Pipeline, Wiki/Turns, Dedup Tiers, Dedup Thresholds, RAPTOR Tuning). Toggle booleans, edit numerics with units, change strings — all write to `.mega-compact.env` via POST. The regression gate enforces that no new config flag ships without a dashboard entry.
+- **Stacked memory graph** — the dashboard shows memory composition over time from 3 content sources (turns, durable memories, wiki) with a 9-gate validation system and a graph-health indicator. Per-model provider cache breakdown in the Cache tab.
 
 ## Install
 
@@ -85,6 +86,12 @@ Set env vars before starting pi. Defaults are in `src/config/dedup.ts`.
 | `MEGACOMPACT_CONTEXT_HEALTH_OUTPUT_QUALITY` | `true` | Sub-score: repetition, coherence, token-salad detection |
 | `MEGACOMPACT_CONTEXT_HEALTH_CACHE_POISON` | `true` | Sub-score: tri-layer KV cache poison validation |
 | `MEGACOMPACT_CONTEXT_HEALTH_MITIGATE` | `false` | Opt-in: auto-compaction on degraded context, prefix break on poisoned cache |
+| `MEGACOMPACT_WIKI_ENHANCED` | `true` | Enhanced wiki with topic overrides + evolution tracking |
+| `MEGACOMPACT_WIKI_INCREMENTAL` | `true` | Incremental wiki updates (no full rebuild) |
+| `MEGACOMPACT_AUTO_WIKI` | `true` | Auto-generate wiki from turns |
+| `MEGACOMPACT_NEW_UI` | `true` | Use the new Tailwind/shadcn dashboard shell |
+| `MEGACOMPACT_COST_API_ENABLED` | `false` | Opt-in: fetch model pricing from an external API (PREVENT-PI-004 network exception). Enriches dashboard cost data for models not in the local pricing table |
+| `MEGACOMPACT_COST_API_URL` | _(unset)_ | OpenRouter-compatible model pricing endpoint (e.g. `https://openrouter.ai/api/v1/models`). Only contacted when `MEGACOMPACT_COST_API_ENABLED=true` |
 
 Full config reference: [`docs/CONFIGURATION.md`](docs/CONFIGURATION.md)
 
@@ -107,7 +114,7 @@ Detailed architecture: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md)
 
 ```bash
 npm run build     # TypeScript compile
-npm test          # Build + 1145 tests
+npm test          # Build + 1197 tests
 npm run lint      # Type check + guardrails scan
 ```
 

package/dist/extensions/dashboard-server/api-contracts/endpoints/registry.js

@@ -259,6 +259,12 @@ export const ENDPOINTS = {
         path: "/api/raptor-tree",
         description: "Hierarchical RAPTOR tree (summary nodes by level) for a session; defaults to the latest session with nodes.",
     },
+    /** GET /api/raptor-build-history — Build history (coherence, depth, timeout) for a session. */
+    raptorBuildHistory: {
+        method: "GET",
+        path: "/api/raptor-build-history",
+        description: "RAPTOR build history rows (node/leaf count, depth, coherence score, timed_out) for a session; defaults to the latest session with builds.",
+    },
     /** GET /api/embedder-health — Round-trip a test embed through the active embedder. */
     embedderHealth: {
         method: "GET",

package/dist/extensions/dashboard-server/index-reader.js

@@ -14,6 +14,7 @@ import { join } from "node:path";
 import { DatabaseSync } from "node:sqlite";
 import { readProviderCacheForRepo, } from "../../src/store/sqlite/perf-samples.js";
 import { lookupModelInputRate } from "../../src/pricing.js";
+import { costApiConfig, lookupCostApiPricing } from "../../src/costApi.js";
 export function getIndexDir() {
     const override = process.env.MEGACOMPACT_INDEX_DIR;
     if (override && override.trim() !== "")
@@ -38,32 +39,54 @@ export function readIndex() {
         const rows = db
             .prepare("SELECT * FROM repo_registry ORDER BY last_seen DESC")
             .all();
-        const mapped = rows.map((r) => ({
-            repoRoot: String(r.repo_root ?? ""),
-            displayName: String(r.display_name ?? ""),
-            stateDir: String(r.state_dir ?? ""),
-            checkpointCount: Number(r.checkpoint_count ?? 0),
-            tokensSaved: Number(r.tokens_saved ?? 0),
-            compressedOriginalBytes: Number(r.compressed_original_bytes ?? 0),
-            lastCompactedAt: r.last_compacted_at ?? null,
-            provider: r.provider ?? null,
-            providerName: r.provider_name ?? null,
-            modelName: r.model_name ?? null,
-            inputRate: (r.input_rate ||
-                lookupModelInputRate(String(r.model_name ?? ""))) ?? null,
-            outputRate: r.output_rate ?? null,
-            lastSeen: Number(r.last_seen ?? 0),
-            // Defaults — enriched below from each repo's own store.
-            tokensKept: 0,
-            tokensDropped: 0,
-            sessions: 0,
-            contextWindow: null,
-            maxTokens: null,
-            reasoning: null,
-            providerCachePct: null,
-            providerCacheRead: null,
-            providerCacheWrite: null,
-        }));
+        // Lazily fetch cost-API pricing once per readIndex call (cached for 1 hour
+        // inside costApi.ts). Avoids per-row network calls.
+        const costApi = costApiConfig();
+        const mapped = rows.map((r) => {
+            const modelName = String(r.model_name ?? "");
+            const localRate = r.input_rate ||
+                lookupModelInputRate(modelName) ||
+                null;
+            let inputRate = localRate;
+            let outputRate = r.output_rate ?? null;
+            // Fall back to the cost API when the local table has no data.
+            if (costApi.enabled && (inputRate == null || outputRate == null) && modelName) {
+                const apiPricing = lookupCostApiPricing(modelName);
+                if (apiPricing) {
+                    if (inputRate == null && apiPricing.inputRate != null) {
+                        inputRate = apiPricing.inputRate;
+                    }
+                    if (outputRate == null && apiPricing.outputRate != null) {
+                        outputRate = apiPricing.outputRate;
+                    }
+                }
+            }
+            return {
+                repoRoot: String(r.repo_root ?? ""),
+                displayName: String(r.display_name ?? ""),
+                stateDir: String(r.state_dir ?? ""),
+                checkpointCount: Number(r.checkpoint_count ?? 0),
+                tokensSaved: Number(r.tokens_saved ?? 0),
+                compressedOriginalBytes: Number(r.compressed_original_bytes ?? 0),
+                lastCompactedAt: r.last_compacted_at ?? null,
+                provider: r.provider ?? null,
+                providerName: r.provider_name ?? null,
+                modelName: r.model_name ?? null,
+                inputRate,
+                outputRate,
+                lastSeen: Number(r.last_seen ?? 0),
+                // Defaults — enriched below from each repo's own store.
+                tokensKept: 0,
+                tokensDropped: 0,
+                sessions: 0,
+                contextWindow: null,
+                maxTokens: null,
+                reasoning: null,
+                providerCachePct: null,
+                providerCacheRead: null,
+                providerCacheWrite: null,
+            };
+        });
         // Enrich each repo with per-store token + model detail read directly via
         // node:sqlite (same zero-dependency invariant as readIndex; no store graph
         // import). Best-effort: a missing/corrupt store degrades to the defaults

package/dist/extensions/dashboard-server/routes-cache.js

@@ -9,7 +9,8 @@
  * PREVENT-002 (parameterized SQL). Loopback-only localhost dashboard endpoint.
  */
 import { createRequire } from "node:module";
-import { computeCacheSavings } from "../../src/pricing.js";
+import { computeCacheSavings, lookupModelInputRate } from "../../src/pricing.js";
+import { costApiConfig, lookupCostApiPricing } from "../../src/costApi.js";
 // ---------------------------------------------------------------------------
 // handleProviderCache — "/api/provider-cache"
 // ---------------------------------------------------------------------------
@@ -90,14 +91,25 @@ export function handleProviderCache(req, res, ctx) {
             let savings = null;
             try {
                 const snap = latestModelSnapshot(stateDir); // guardrails-allow PREVENT-PI-004: local SQLite read (loopback dashboard)
-                if (snap && snap.inputRate > 0) {
-                    const s = computeCacheSavings(lifetime.totalCacheRead, lifetime.totalCacheWrite, snap.inputRate);
+                let rate = snap?.inputRate ?? 0;
+                let modelName = snap?.modelName ?? snap?.modelId ?? "unknown";
+                // Fallback chain: SQLite snapshot → local pricing table → cost API
+                if (rate <= 0 && modelName !== "unknown") {
+                    rate = lookupModelInputRate(modelName) ?? 0;
+                }
+                if (rate <= 0 && costApiConfig().enabled && modelName !== "unknown") {
+                    // guardrails-allow PREVENT-PI-004: cost API pricing lookup (user-opted-in)
+                    const apiPricing = lookupCostApiPricing(modelName);
+                    rate = apiPricing?.inputRate ?? 0;
+                }
+                if (rate > 0) {
+                    const s = computeCacheSavings(lifetime.totalCacheRead, lifetime.totalCacheWrite, rate);
                     savings = {
                         cacheReadSaved: s.cacheReadSaved,
                         cacheWriteCost: s.cacheWriteCost,
                         netSaved: s.netSaved,
-                        model: snap.modelName ?? snap.modelId ?? "unknown",
-                        inputRate: snap.inputRate,
+                        model: modelName,
+                        inputRate: rate,
                     };
                 }
             }

package/dist/extensions/dashboard-server/routes-rag-settings-helpers.js

@@ -129,6 +129,13 @@ export const SETTINGS = [
             num("MEGACOMPACT_EMBED_CACHE", "Embed Cache Size", "Embedding cache entries (0 = disabled)", 256, 0, 10000),
         ],
     },
+    {
+        name: "Cost API",
+        settings: [
+            boolDirect("MEGACOMPACT_COST_API_ENABLED", "Cost API Lookup", "Fetch model pricing from an external API (PREVENT-PI-004 network exception — opt-in). Enriches the dashboard Cache/Repos tabs with real $/token rates for models not in the local pricing table.", false),
+            str("MEGACOMPACT_COST_API_URL", "Cost API URL", "OpenRouter-compatible model pricing endpoint (e.g. https://openrouter.ai/api/v1/models). Only contacted when Cost API Lookup is ON.", ""),
+        ],
+    },
 ];
 /** Flat index of every setting by key for O(1) lookup + validation. */
 export const SETTING_BY_KEY = new Map(SETTINGS.flatMap((c) => c.settings).map((s) => [s.key, s]));

package/dist/extensions/dashboard-server/routes-raptor.js

@@ -86,3 +86,63 @@ export function handleRaptorTree(req, res, ctx) {
         return true;
     }
 }
+/** Find the most recent session that has build history rows (else null). */
+function latestBuildSession(stateDir) {
+    try {
+        const db = openStore(stateDir);
+        const row = db
+            .prepare("SELECT DISTINCT session_id FROM raptor_build_history ORDER BY completed_at DESC LIMIT 1")
+            .get();
+        return row?.session_id ?? null;
+    }
+    catch {
+        return null;
+    }
+}
+export function handleRaptorBuildHistory(req, res, ctx) {
+    if (req.method !== "GET")
+        return false;
+    const parsed = parseUrl(req.url ?? "", true);
+    if (parsed.pathname !== "/api/raptor-build-history")
+        return false;
+    try {
+        const { listBuildHistory } = _req("../../src/dedup/raptor/buildHistory.js");
+        let sessionId = typeof parsed.query.sessionId === "string" ? parsed.query.sessionId : "";
+        if (!sessionId) {
+            sessionId = latestBuildSession(ctx.stateDir) ?? "";
+            if (!sessionId) {
+                const body = { builds: [], empty: true };
+                res.writeHead(200, { "Content-Type": "application/json" });
+                res.end(JSON.stringify(body));
+                return true;
+            }
+        }
+        const rows = listBuildHistory(sessionId, ctx.stateDir);
+        if (rows.length === 0) {
+            const body = { builds: [], empty: true };
+            res.writeHead(200, { "Content-Type": "application/json" });
+            res.end(JSON.stringify(body));
+            return true;
+        }
+        const dtos = rows.map((r) => ({
+            buildId: r.buildId,
+            sessionId: r.sessionId,
+            startedAt: r.startedAt,
+            completedAt: r.completedAt,
+            nodeCount: r.nodeCount,
+            leafCount: r.leafCount,
+            depth: r.depth,
+            coherenceScore: r.coherenceScore,
+            timedOut: r.timedOut,
+        }));
+        const body = { builds: dtos, empty: false };
+        res.writeHead(200, { "Content-Type": "application/json" });
+        res.end(JSON.stringify(body));
+        return true;
+    }
+    catch (e) {
+        res.writeHead(500, { "Content-Type": "application/json" });
+        res.end(JSON.stringify({ error: String(e) }));
+        return true;
+    }
+}

package/dist/extensions/dashboard-server/routes.js

@@ -15,7 +15,7 @@ export { handleProviderCache } from "./routes-cache.js";
 export { handleMemoryStatus } from "./routes-memory.js";
 export { handleSetupStatus, handleSetupDetect, handleSetupConfigure } from "./routes-setup.js";
 export { handleMemoryMap } from "./routes-memory-map.js";
-export { handleRaptorTree } from "./routes-raptor.js";
+export { handleRaptorTree, handleRaptorBuildHistory } from "./routes-raptor.js";
 export { handleCacheStripes } from "./routes-cache.js";
 export { handleContextHealth, handleCachePoison, handleHealthSettings } from "./routes-health.js";
 export { handleEmbedderHealth } from "./routes-embedder-health.js";

package/dist/extensions/dashboard-server/server.js

@@ -13,7 +13,7 @@ import { fileURLToPath } from "node:url";
 import { createRequire } from "node:module";
 import { NEW_UI } from "../../src/config.js";
 import { log, setLogPath, setDashboardServerVersion } from "./state.js";
-import { buildRouteContext, handleIndex, handleRepoIndex, handleEvents, handleGameState, handleGameScores, handlePerf, handleAchievements, handleSessions, handleTopics, handleTurns, handleMaintenance, handleProviderCache, handleCacheStripes, handleMemoryStatus, handleSetupStatus, handleSetupDetect, handleSetupConfigure, handleMemoryMap, handleRaptorTree, handleContextHealth, handleCachePoison, handleHealthSettings, handleEmbedderHealth, handleRagSettings, handleRagMetrics, handleWiki, handleStatic, } from "./routes.js";
+import { buildRouteContext, handleIndex, handleRepoIndex, handleEvents, handleGameState, handleGameScores, handlePerf, handleAchievements, handleSessions, handleTopics, handleTurns, handleMaintenance, handleProviderCache, handleCacheStripes, handleMemoryStatus, handleSetupStatus, handleSetupDetect, handleSetupConfigure, handleMemoryMap, handleRaptorTree, handleRaptorBuildHistory, handleContextHealth, handleCachePoison, handleHealthSettings, handleEmbedderHealth, handleRagSettings, handleRagMetrics, handleWiki, handleStatic, } from "./routes.js";
 export async function launchDashboardServer(stateDir) {
     // Our own package version — exposed at /api/version so the launcher can
     // detect a stale server (started by an older build) and replace it on
@@ -219,6 +219,8 @@ export async function launchDashboardServer(stateDir) {
             return;
         if (handleRaptorTree(req, res, ctx))
             return;
+        if (handleRaptorBuildHistory(req, res, ctx))
+            return;
         if (handleContextHealth(req, res, ctx))
             return;
         if (handleCachePoison(req, res, ctx))

package/dist/extensions/mega-pipeline/recall.js

@@ -79,17 +79,23 @@ export function doRecall(runtime, config, ctx, query, source) {
         runtime.pushTicker(`${C.amber}↩${C.reset} recalled ${top.checkpoint.checkpointId} · ${scorePct}% · ${label}`);
         runtime.lastWhy = `why: recalled@${scorePct}% (${result.toInject.length} chkpt)`;
     }
+    let sumTokens = 0;
+    for (const h of result.toInject)
+        sumTokens += h.checkpoint.tokenEstimate;
     if (result.toInject.length > 0) {
-        let sumTokens = 0;
-        for (const h of result.toInject)
-            sumTokens += h.checkpoint.tokenEstimate;
         runtime.rt.recallInjections += result.toInject.length;
         runtime.rt.cacheHitTokens += sumTokens;
         incRecallInjected(result.toInject.length, runtime.currentStateDir);
         incCacheHitTokens(sumTokens, runtime.currentStateDir);
-        // S43: record recall provenance — which checkpoints/summaries served this
-        // turn, their score + source path. Linked to the turn row written at
-        // turn_end via the conversation+turnIndex. Best-effort + non-fatal.
+    }
+    // S43: record recall provenance — which checkpoints/summaries served this
+    // turn, their score + source path. Linked to the turn row written at
+    // turn_end via the conversation+turnIndex. Best-effort + non-fatal.
+    // Persists telemetry (HyDE + recall metrics) even when recall returned
+    // no hits, so empty-recall HyDE invocations are still visible in the
+    // dashboard Turns/Metrics tabs.
+    const hasTelemetry = result.hydeInfo != null || result.recallMetrics != null;
+    if (result.toInject.length > 0 || hasTelemetry) {
         try {
             const convId = ensureConversationIdFor(config, sid, runtime.currentStateDir);
             const turnId = recordTurnWrite(config, {
@@ -101,16 +107,18 @@ export function doRecall(runtime, config, ctx, query, source) {
                 hyde: result.hydeInfo ?? undefined,
                 recallMetrics: result.recallMetrics ?? undefined,
             }, runtime.currentStateDir);
-            recordRecallWrite(config, turnId, result.toInject.map((h) => ({
-                checkpointId: h.checkpoint.checkpointId,
-                score: h.score,
-                source: h.raptorLevel !== undefined
-                    ? "raptor"
-                    : h.repoId
-                        ? "cross-repo"
-                        : "flat",
-                raptorLevel: h.raptorLevel,
-            })), runtime.currentStateDir);
+            if (result.toInject.length > 0) {
+                recordRecallWrite(config, turnId, result.toInject.map((h) => ({
+                    checkpointId: h.checkpoint.checkpointId,
+                    score: h.score,
+                    source: h.raptorLevel !== undefined
+                        ? "raptor"
+                        : h.repoId
+                            ? "cross-repo"
+                            : "flat",
+                    raptorLevel: h.raptorLevel,
+                })), runtime.currentStateDir);
+            }
         }
         catch {
             /* non-fatal: recall provenance never breaks the recall path */

package/dist/extensions/mega-runtime/capture-model.js

@@ -8,6 +8,7 @@
 import { resolveRepoRoot } from "../mega-config.js";
 import { recordModelSnapshot, recordRepoModel } from "../../src/store/sqlite.js";
 import { lookupModelInputRate } from "../../src/pricing.js";
+import { costApiConfig, lookupCostApiPricing } from "../../src/costApi.js";
 // -------------------------------------------------------------- captureModel
 export function captureModelImpl(ctx, ectx) {
     const m = ectx.model;
@@ -29,13 +30,25 @@ export function captureModelImpl(ctx, ectx) {
     }
     const modelId = m.id;
     const fallbackInput = lookupModelInputRate(modelId);
+    let inputRate = m.cost?.input || fallbackInput || 0;
+    let outputRate = m.cost?.output ?? 0;
+    // Cost API last-resort fallback for models not in the local pricing table
+    if (inputRate <= 0 && costApiConfig().enabled) {
+        // guardrails-allow PREVENT-PI-004: cost API pricing lookup (user-opted-in)
+        const apiPricing = lookupCostApiPricing(modelId);
+        if (apiPricing) {
+            inputRate = apiPricing.inputRate ?? 0;
+            if (outputRate <= 0)
+                outputRate = apiPricing.outputRate ?? 0;
+        }
+    }
     const snap = {
         provider: m.provider,
         providerName,
         modelId,
         modelName: m.name ?? null,
-        inputRate: m.cost?.input || fallbackInput || 0,
-        outputRate: m.cost?.output ?? 0,
+        inputRate,
+        outputRate,
         contextWindow: m.contextWindow ?? 0,
         maxTokens: m.maxTokens ?? 0,
         reasoning: !!m.reasoning,

package/dist/extensions/mega-runtime/runtime-snapshot.js

@@ -45,6 +45,12 @@ export function snapshotImpl(self, ctx) {
     const sig = materialSigImpl(self);
     if (ctx && self.widgetData && self.lastSnapshotSig === sig) {
         self.renderWidget(ctx);
+        // Refresh heartbeat on a time cadence even when material state hasn't
+        // changed, so the session doesn't appear stale and get pruned by the
+        // dashboard's 30-min cutoff. The full snapshot recompute is still gated
+        // (expensive SQLite opens + writeFileSync), but a heartbeat row is
+        // cheap (single INSERT OR REPLACE on the index DB).
+        refreshHeartbeatIfStale(self, ctx, 10_000);
         return;
     }
     const perfT0 = performance.now();
@@ -216,4 +222,24 @@ export function snapshotImpl(self, ctx) {
         /* non-fatal: perf instrumentation never blocks the agent */
     }
     self.lastSnapshotSig = sig;
+    self.lastHeartbeatAt = Date.now();
+}
+// ----------------------------------------------------------- heartbeat refresh
+/** Refresh the session heartbeat if enough time has elapsed since the last
+ *  write. Called from the material-change gate fast path (where the full
+ *  snapshot recompute is skipped) so the session stays visible in the
+ *  dashboard during idle/typing periods. Non-fatal — best-effort like the
+ *  main heartbeat write. */
+function refreshHeartbeatIfStale(self, ctx, intervalMs) {
+    const now = Date.now();
+    if (now - self.lastHeartbeatAt < intervalMs)
+        return;
+    try {
+        const repo = resolveRepoRoot(ctx?.cwd ?? self.currentStateDir) ?? self.currentStateDir;
+        recordSessionHeartbeat(process.pid, self.rt.sessionId, repo, self.currentStateDir, self.lastCtxWindow || 0);
+        self.lastHeartbeatAt = now;
+    }
+    catch {
+        /* non-fatal: heartbeat refresh must never block the snapshot path */
+    }
 }

package/dist/extensions/mega-runtime/runtime.js

@@ -163,6 +163,7 @@ export class MegaRuntime {
     // (already-registered) widget factory is refreshed. Kills the per-event
     // main-thread block during typing/idle streaming with no material change.
     lastSnapshotSig = null;
+    lastHeartbeatAt = 0;
     // v0.8.5: bumped whenever the cached game-state memo is evicted (bumpGameState
     // for in-process /mega-game writes, the fs.watch callback for cross-process
     // dashboard-server writes, and bindRepo on repo switch) so the snapshot gate

package/dist/src/config/dedup.js

@@ -43,7 +43,8 @@ export function loadDedupConfig() {
         L2_ENABLED: envBool("MEGACOMPACT_L2_ENABLED", true),
         // Fix D: RAPTOR promoted to live recall. Default ON; canary.ts sequences it
         // last (L0→L1→L2→RAPTOR) and auto-disables on p95 breach, so promotion is
-        // safe. `RAPTOR_SHADOW_MODE=false` still gates serving during transition.
+        // safe. Shadow mode (RAPTOR_SHADOW_MODE=true) opts out of serving while
+        // still building + persisting the tree (transition/eval use).
         RAPTOR_ENABLED: envBool("MEGACOMPACT_RAPTOR_ENABLED", true),
         MARK_ONLY_L0: envBool("MEGACOMPACT_MARK_ONLY_L0", false),
         MARK_ONLY_L1: envBool("MEGACOMPACT_MARK_ONLY_L1", false),

package/dist/src/costApi.js

@@ -0,0 +1,113 @@
+/**
+ * costApi.ts — optional model pricing lookup from an external API.
+ *
+ * PREVENT-PI-004 EXEMPTION: This module makes network calls to fetch model
+ * pricing data. It is disabled by default and must be explicitly opted in via
+ * MEGACOMPACT_COST_API_ENABLED=true + MEGACOMPACT_COST_API_URL=<url>.
+ *
+ * The dashboard server uses this to enrich the inputRate/outputRate fields
+ * when the local pricing table (pricing.ts) has no data for a model.
+ *
+ * Results are cached for 1 hour to minimize network calls.
+ */
+import { spawnSync } from "node:child_process"; // guardrails-allow PREVENT-PI-004: user-opted-in cost API — fetches model pricing from a user-configured endpoint, never called unless MEGACOMPACT_COST_API_ENABLED=true
+let _cache = null;
+const CACHE_TTL_MS = 60 * 60 * 1000; // 1 hour
+export function costApiConfig() {
+    const enabled = process.env.MEGACOMPACT_COST_API_ENABLED === "true" ||
+        process.env.MEGACOMPACT_COST_API_ENABLED === "1";
+    const url = process.env.MEGACOMPACT_COST_API_URL ?? "";
+    return { enabled, url };
+}
+/**
+ * Fetch model pricing from the configured API endpoint.
+ *
+ * Expects a JSON response of shape:
+ *   { "data": [{ "id": "model-name", "pricing": { "prompt": "0.000003", "completion": "0.000015" } }] }
+ * (OpenRouter-compatible format). Values are USD per token as strings.
+ *
+ * Returns a map keyed by lowercase model id → { inputRate, outputRate }.
+ * Returns an empty map on any error. Never throws.
+ */
+export function fetchModelPricing() {
+    const cfg = costApiConfig();
+    if (!cfg.enabled || !cfg.url)
+        return new Map();
+    // Return cached data if fresh.
+    if (_cache && Date.now() - _cache.timestamp < CACHE_TTL_MS) {
+        return _cache.data;
+    }
+    const result = _doFetch(cfg.url);
+    _cache = { timestamp: Date.now(), data: result };
+    return result;
+}
+function _doFetch(url) {
+    // Synchronous bridge: spawnSync an inline worker so the call blocks without
+    // deadlocking fetch (mirrors HttpEmbedder / RAPTOR summarizer pattern).
+    const WORKER = String.raw `
+		try {
+			const r = await fetch(process.env.COST_API_URL, { // guardrails-allow PREVENT-PI-004: user-opted-in cost API — fetches model pricing from a user-configured endpoint
+				method: "GET",
+				headers: { "accept": "application/json" },
+			});
+			const j = await r.json();
+			process.stdout.write(JSON.stringify({ ok: r.ok, data: j }));
+		} catch (e) {
+			process.stdout.write(JSON.stringify({ ok: false, error: String(e && e.message ? e.message : e) }));
+		}
+	`;
+    const res = spawnSync(process.execPath, ["-e", WORKER], {
+        encoding: "utf8",
+        timeout: 15_000,
+        env: { ...process.env, COST_API_URL: url },
+    });
+    const out = new Map();
+    let parsed;
+    try {
+        parsed = JSON.parse(res.stdout);
+    }
+    catch {
+        return out;
+    }
+    if (!parsed.ok || !parsed.data)
+        return out;
+    // OpenRouter-compatible: { data: [{ id, pricing: { prompt, completion } }] }
+    const models = parsed.data?.data;
+    if (!Array.isArray(models))
+        return out;
+    for (const m of models) {
+        if (!m.id)
+            continue;
+        const prompt = m.pricing?.prompt;
+        const completion = m.pricing?.completion;
+        const inputRate = prompt != null ? parseFloat(prompt) : null;
+        const outputRate = completion != null ? parseFloat(completion) : null;
+        if (inputRate != null || outputRate != null) {
+            out.set(m.id.toLowerCase(), { inputRate, outputRate });
+        }
+    }
+    return out;
+}
+/**
+ * Look up pricing for a model name from the cost API cache.
+ * Tries exact match, then prefix match (same pattern as lookupModelInputRate).
+ */
+export function lookupCostApiPricing(model) {
+    const data = fetchModelPricing();
+    if (data.size === 0)
+        return null;
+    const lower = model.toLowerCase();
+    // Exact match.
+    if (data.has(lower))
+        return data.get(lower) ?? null;
+    // Prefix match.
+    for (const [key, pricing] of data) {
+        if (lower.startsWith(key))
+            return pricing;
+    }
+    return null;
+}
+/** Clear the pricing cache (useful for testing). */
+export function clearCostApiCache() {
+    _cache = null;
+}

package/dist/src/dedup/raptor/incremental.js

@@ -12,11 +12,38 @@
 import { cosineSimilarity } from "../../embedder.js";
 import { defaultEmbedder as getDefaultEmbedder } from "../../embedder.js";
 import { buildRaptorTree } from "./tree.js";
+import { applyHallucinationGuardrails, sourceTokenSet } from "./guardrails.js";
 import { meanVector } from "./kmeans.js";
-import { summarizeCluster } from "./summarizer.js";
+import { summarizeCluster, extractiveClusterSummary } from "./summarizer.js";
 import { saveRaptorTree, listRaptorNodes, } from "../../store/sqlite.js";
 /** When >FRESH_THRESHOLD of existing leaves are new, a full rebuild is cheaper. */
 const FRESH_THRESHOLD = 0.5;
+/**
+ * Summarize + apply hallucination guardrails (mirrors tree.ts:summarizeInto).
+ * If the guardrail marks the summary as extractive_fallback (drift/hallucination),
+ * re-summarize with the deterministic extractive path so we never serve a
+ * drifted LLM summary from an incremental update.
+ */
+function summarizeGuarded(messages, centroid, embedder, consistencyThreshold) {
+    let cs = summarizeCluster(messages);
+    const sources = messages.map((m) => m.text);
+    const guard = applyHallucinationGuardrails({
+        summary: cs.summary,
+        sources,
+        centroid,
+        embedder,
+        sourceTokens: sourceTokenSet(sources),
+        consistencyThreshold,
+    });
+    if (guard.marker === "extractive_fallback") {
+        cs = extractiveClusterSummary(messages);
+    }
+    return {
+        summary: cs.summary,
+        tokenEstimate: cs.tokenEstimate,
+        qualityMarker: guard.marker === "extractive_fallback" ? "low" : guard.marker,
+    };
+}
 /**
  * Incrementally update a persisted RAPTOR tree with new leaves.
  *
@@ -139,6 +166,7 @@ export function incrementRaptorTree(existingLeaves, newLeaves, opts) {
         for (const l of reallyNewLeaves)
             leafMap.set(l.id, l);
         // Recompute affected Level-0 nodes.
+        const embedder = opts.embedder ?? getDefaultEmbedder();
         for (const nid of affectedNodeIds) {
             const node = tree.nodes.get(nid);
             if (!node)
@@ -151,7 +179,7 @@ export function incrementRaptorTree(existingLeaves, newLeaves, opts) {
                 // Summarize: flatten messages from all covered leaves.
                 const messages = coveredLeaves.flatMap((l) => l.messages);
                 if (messages.length > 0) {
-                    const cs = summarizeCluster(messages);
+                    const cs = summarizeGuarded(messages, node.embedding, embedder, opts.consistencyThreshold);
                     node.summary = cs.summary;
                     node.tokenEstimate = cs.tokenEstimate;
                 }
@@ -177,7 +205,7 @@ export function incrementRaptorTree(existingLeaves, newLeaves, opts) {
                         node.embedding = meanVector(covered.map((l) => l.embedding));
                         const messages = covered.flatMap((l) => l.messages);
                         if (messages.length > 0) {
-                            const cs = summarizeCluster(messages);
+                            const cs = summarizeGuarded(messages, node.embedding, embedder, opts.consistencyThreshold);
                             node.summary = cs.summary;
                             node.tokenEstimate = cs.tokenEstimate;
                         }