pi-mega-compact 0.13.4 → 0.13.6

package/README.md

@@ -1,6 +1,6 @@
 # pi-mega-compact
 
-A local context compressor for the [pi coding agent](https://github.com/earendil-works/pi). Keeps long sessions running without overflowing the context window. All on your machine — no cloud, no API calls, no telemetry.
+A local-first context compressor for the [pi coding agent](https://github.com/earendil-works/pi). Keeps long sessions running without overflowing the context window. Local by default — no cloud, no API calls, no telemetry. Bring your own localhost embedder (Ollama, ONNX, TEI) for better semantic matches, or opt in to a remote endpoint if you need to.
 
 ## Features
 
@@ -13,10 +13,10 @@ A local context compressor for the [pi coding agent](https://github.com/earendil
 - **Durable memory** — on a cadence the store auto-reviews and safe-keeps decisions, facts, and preferences as first-class RAG memories, so long-running projects remember what mattered.
 - **Prompt-cache optimization** — message separation + cache striping (default OFF, opt-in). Targets 82-90% cache hit rate by structuring context around provider cache boundaries. Enables with `MEGACOMPACT_MESSAGE_SEPARATION` and `MEGACOMPACT_CACHE_STRIPING`.
 - **Context health + KV cache poison validation** (v0.12) — a real-time composite 0-1 health score per turn from five sub-scores (drift, output quality, error rate, cache health, cache poison). Catches garbled/hallucinated output and provider-side KV-cache corruption *before* they waste tokens — the failure mode where a large-context model (e.g. DeepSeek V4 Flash, 1M window) degrades at <1% usage. Tri-layer cache poison validation: prefix hash (L1 FNV-1a), output-quality-by-cache-hit (L2 semantic), error-rate correlation (L3 behavioral). The dashboard **Health tab** shows a gauge, sparkline, sub-score bars, alerts, and per-model breakdown. Auto-mitigation (force compaction on degraded context, prefix break on poisoned cache) is default OFF — toggle it in the Maintenance tab.
-- **RAG suite** — query reformulation (TF-IDF + RRF), tiered routing (L0 cache -> L1 FTS5 -> L2 PGlite), and recall-quality metrics (CRAG). All default OFF, opt-in via flags `MEGACOMPACT_QUERY_REFORMULATION`, `MEGACOMPACT_TIERED_ROUTER`, `MEGACOMPACT_RECALL_METRICS`.
+- **RAG suite** — query reformulation (TF-IDF + RRF), tiered routing (L0 cache -> L1 FTS5 -> L2 PGlite), recall-quality metrics (CRAG), memory graph, and HyDE (hypothetical document embeddings). All default ON with graceful fallback — opt out via `MEGACOMPACT_<NAME>_DISABLED=true`. HyDE auto-activates only when an LLM embedder (Ollama/HTTP) is configured — it's a no-op with the default TrigramEmbedder. Toggle any flag from the dashboard Setup tab.
 - **Stacked memory graph** — the dashboard shows memory composition over time from 3 content sources (turns, durable memories, wiki) with a 9-gate validation system and a graph-health indicator. Per-model provider cache breakdown in the Cache tab.
 - **Debug bundle** — Maintenance tab in the dashboard has a **Gather Debug Logs** button that collects events, config, and store state into a shareable archive for bug reports.
-- **Fully local** — node:sqlite + trigram embeddings by default. Bring your own localhost embedder (ONNX, Ollama, TEI) for better semantic matches. Zero calls off your machine except the optional, localhost-only dashboard.
+- **Local-first by default** — node:sqlite + trigram embeddings by default, zero calls off your machine. Bring your own localhost embedder (Ollama, ONNX, TEI) for better semantic matches — the embedder endpoint is loopback-only by default (`MEGACOMPACT_ALLOW_REMOTE_EMBEDDER=1` opts in to a remote/third-party endpoint). The optional dashboard is localhost-only. No cloud, no API calls, no telemetry.
 - **Team-run aware** — fine-grained durable trim fires at agent settle during sub-agent runs, so long multi-agent work doesn't just collapse at the end.
 - **Multi-pi dashboard** — one dashboard tab per active pi process with the context stack, per-repo stats, and a live SSE feed across all of them. The React SPA has 15 lazy-loaded tabs — **Overview**, **Cache**, **Sessions**, **Turns** (per-turn memory + recall + rewind), and **Health** (context-health gauge + cache-poison alerts) are primary; **Repos**, **Events**, **Config**, **Setup** (embedding wizard), **Metrics** (perf latency/TPS/CPU), **Game**, **Achievements**, **Topics** (wiki), **Memory Map** (D3 graph), and **Maintenance** (debug bundle) are advanced.
 - **Auto-categorizing wiki.** Every 3 compactions (seeds from turns before that), the store clusters your real memory embeddings (k-means) and labels each cluster with its most discriminative terms (TF-IDF) — no LLM, no Ollama, fully local. The dashboard **Wiki tab** browses topics, searches by label or term, and drills down into the member memories of each cluster.
@@ -67,13 +67,16 @@ Set env vars before starting pi. Defaults are in `src/config/dedup.ts`.
 | `MEGACOMPACT_AUTO` | `true` | Enable auto-compaction |
 | `MEGACOMPACT_DEDUP_SIM` | `0.90` | Cosine threshold for near-dup collapse |
 | `MEGACOMPACT_CROSSREPO_ENABLED` | `true` | Cross-repo recall on resume |
-| `MEGACOMPACT_EMBEDDING_URL` | _(unset)_ | BYO localhost embedder endpoint |
+| `MEGACOMPACT_EMBEDDING_URL` | _(unset)_ | BYO localhost embedder endpoint (loopback-only by default) |
+| `MEGACOMPACT_ALLOW_REMOTE_EMBEDDER` | `false` | Opt-in: allow non-loopback embedder endpoint (e.g. hosted API) |
 | `MEGACOMPACT_TUI_WIDGET` | `true` | Render the above-editor panel |
 | `MEGACOMPACT_MESSAGE_SEPARATION` | `false` | Opt-in: separate messages at provider cache boundaries |
 | `MEGACOMPACT_CACHE_STRIPING` | `false` | Opt-in: stripe cache stripes across conversation |
-| `MEGACOMPACT_QUERY_REFORMULATION` | `false` | Opt-in: TF-IDF + RRF query reformulation |
-| `MEGACOMPACT_TIERED_ROUTER` | `false` | Opt-in: L0->L1->L2 recall routing |
-| `MEGACOMPACT_RECALL_METRICS` | `false` | Opt-in: CRAG recall quality metrics |
+| `MEGACOMPACT_QUERY_REFORMULATION_DISABLED` | `false` | Default ON — set `true` to disable TF-IDF + RRF query reformulation |
+| `MEGACOMPACT_TIERED_ROUTER_DISABLED` | `false` | Default ON — set `true` to disable L0->L1->L2 recall routing |
+| `MEGACOMPACT_RECALL_METRICS_DISABLED` | `false` | Default ON — set `true` to disable CRAG recall quality metrics |
+| `MEGACOMPACT_MEMORY_GRAPH_DISABLED` | `false` | Default ON — set `true` to disable memory graph |
+| `MEGACOMPACT_HYDE_DISABLED` | `false` | Default ON — set `true` to disable HyDE (auto-activates only with LLM embedder) |
 | `MEGACOMPACT_MEMORY_GRAPH_SEED_TURNS` | `true` | Seed memory graph from turns (default ON) |
 | `MEGACOMPACT_WIKI_SEED_FROM_TURNS` | `true` | Seed wiki from turns (default ON) |
 | `MEGACOMPACT_RAPTOR_INCREMENTAL` | `true` | Incremental RAPTOR updates (default ON) |
@@ -104,7 +107,7 @@ Detailed architecture: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md)
 
 ```bash
 npm run build     # TypeScript compile
-npm test          # Build + 1088 tests
+npm test          # Build + 1145 tests
 npm run lint      # Type check + guardrails scan
 ```
 

package/dist/extensions/dashboard-server/api-contracts/endpoints/registry.js

@@ -151,6 +151,49 @@ export const ENDPOINTS = {
         path: "/api/topics/:topicId/memories",
         description: "Member memories assigned to a wiki topic (drill-down).",
     },
+    // ─── Wiki Revival (W2) ───────────────────────────────────────────
+    /** GET /api/wiki/index — wiki landing page with resolved labels. */
+    wikiIndex: {
+        method: "GET",
+        path: "/api/wiki/index",
+        description: "Wiki index: auto-categorized topics with resolved (override-aware) labels.",
+    },
+    /** GET /api/wiki/topic/:topicId — single topic page + provenance. */
+    wikiTopic: {
+        method: "GET",
+        path: "/api/wiki/topic/:topicId",
+        description: "Single wiki topic page: summary, key memories, and memory provenance.",
+    },
+    /** PUT /api/wiki/topic/:topicId/label — rename a topic (user curation). */
+    renameTopic: {
+        method: "PUT",
+        path: "/api/wiki/topic/:topicId/label",
+        description: "Rename a wiki topic (persists a label override; emits wiki_topic_renamed).",
+    },
+    /** POST /api/wiki/merge — merge one topic into another. */
+    mergeTopics: {
+        method: "POST",
+        path: "/api/wiki/merge",
+        description: "Merge source topic into target (reassigns memories; emits wiki_topics_merged).",
+    },
+    /** POST /api/wiki/topic/:topicId/split — split listed memories into a new topic. */
+    splitTopic: {
+        method: "POST",
+        path: "/api/wiki/topic/:topicId/split",
+        description: "Split listed member memories into a new topic (emits wiki_topic_split).",
+    },
+    /** GET /api/wiki/topic/:topicId/timeline — per-topic time buckets. */
+    topicTimeline: {
+        method: "GET",
+        path: "/api/wiki/topic/:topicId/timeline",
+        description: "Per-topic memory-assignment timeline buckets for charts.",
+    },
+    /** GET /api/wiki/evolution — node/edge D3 evolution graph. */
+    topicEvolution: {
+        method: "GET",
+        path: "/api/wiki/evolution",
+        description: "Topic evolution nodes + edges + time buckets from topic_evolution.",
+    },
     // ─── S49B Maintenance Tab ─────────────────────────────────────────
     /** GET /api/maintenance — DB stats (table row counts + storage). */
     maintenanceStats: {
@@ -235,4 +278,11 @@ export const ENDPOINTS = {
         path: "/api/rag-settings",
         description: "Update RAG feature flag states by writing _DISABLED env vars to .mega-compact.env.",
     },
+    // ─── RAG Metrics (Sprint H2) ──────────────────────────────────────
+    /** GET /api/rag-metrics — HyDE + recall-quality telemetry aggregates. */
+    ragMetrics: {
+        method: "GET",
+        path: "/api/rag-metrics",
+        description: "HyDE invocation + recall-quality telemetry: flags, totals, recent turns, and daily series.",
+    },
 };

package/dist/extensions/dashboard-server/api-contracts/rag-metrics.js

@@ -0,0 +1 @@
+export {};

package/dist/extensions/dashboard-server/api-contracts/rag-settings.js

@@ -1,8 +1,8 @@
 /**
- * api-contracts/rag-settings.ts — RAG feature flag API response types.
+ * api-contracts/rag-settings.ts — comprehensive settings API response types.
  *
- * Types for the /api/rag-settings endpoint used by the dashboard RAG Settings
- * panel to read and toggle the five RAG feature flags (B1–B5).
+ * Types for the /api/rag-settings endpoint used by the dashboard Setup panel to
+ * read and update every adjustable MEGACOMPACT_* setting, grouped by category.
  *
  * PREVENT-PI-004: type definitions only, no network code.
  * PREVENT-011: no `any` type — all types are explicit.

package/dist/extensions/dashboard-server/api-contracts/wiki.js

@@ -0,0 +1 @@
+export {};

package/dist/extensions/dashboard-server/routes-rag-metrics.js

@@ -0,0 +1,60 @@
+/**
+ * dashboard-server/routes-rag-metrics.ts — GET /api/rag-metrics route (H2).
+ *
+ * Aggregates the H1 HyDE + recall-quality telemetry columns from turns.db into
+ * a compact response (flags, totals, recent rows, daily series).
+ *
+ * Guardrails: PREVENT-PI-004 (loopback-only), PREVENT-002 (parameterized),
+ * PREVENT-011 (no `any`).
+ */
+import { openTurnStore } from "../../src/store/turns/connection.js";
+import { listTelemetryTurns, aggregateDailyTelemetry, } from "../../src/store/turns/hydeStore.js";
+import { RAG_HYDE_ENABLED, RAG_RECALL_METRICS } from "../../src/config.js";
+function sendJson(res, status, body) {
+    // guardrails-allow PREVENT-PI-004: loopback dashboard response (local)
+    res.writeHead(status, { "Content-Type": "application/json" });
+    res.end(JSON.stringify(body));
+}
+export function handleRagMetrics(req, res, ctx) {
+    const url = req.url ?? "";
+    if (url !== "/api/rag-metrics" || req.method !== "GET")
+        return false;
+    let db;
+    try {
+        db = openTurnStore(ctx.stateDir);
+    }
+    catch {
+        db = null;
+    }
+    const recent = db ? listTelemetryTurns(db, { limit: 200 }) : [];
+    const daily = db ? aggregateDailyTelemetry(db, 14) : [];
+    const hydeRanTurns = recent.filter((t) => t.hydeRan === 1).length;
+    const liftSum = recent.reduce((s, t) => s + t.hydeLift, 0);
+    const genSum = recent.reduce((s, t) => s + t.hydeGenerationMs, 0);
+    const scored = recent.filter((t) => t.recallScore > 0 || t.recallPass === 1);
+    const avgScore = scored.length > 0
+        ? Math.round((scored.reduce((s, t) => s + t.recallScore, 0) / scored.length) * 100) / 100
+        : null;
+    const passCount = scored.filter((t) => t.recallPass === 1).length;
+    const recentPassRate = scored.length > 0 ? Math.round((passCount / scored.length) * 100) / 100 : 0;
+    const body = {
+        flags: {
+            hydeEnabled: RAG_HYDE_ENABLED(),
+            recallMetricsEnabled: RAG_RECALL_METRICS(),
+        },
+        totals: {
+            telemetryTurns: recent.length,
+            hydeRanTurns,
+            avgLift: recent.length > 0
+                ? Math.round((liftSum / recent.length) * 100) / 100
+                : 0,
+            avgScore,
+            avgGenerationMs: recent.length > 0 ? Math.round(genSum / recent.length) : 0,
+            recentPassRate,
+        },
+        recent: recent.slice(0, 50),
+        daily,
+    };
+    sendJson(res, 200, body);
+    return true;
+}

package/dist/extensions/dashboard-server/routes-rag-settings-helpers.js

@@ -0,0 +1,167 @@
+/**
+ * dashboard-server/routes-rag-settings-helpers.ts — SETTINGS inventory.
+ *
+ * The full catalog of user-adjustable MEGACOMPACT_* settings surfaced by the
+ * dashboard Setup panel, grouped by category. Kept separate from the route
+ * handler so the inventory can grow without bloating the handler file.
+ *
+ * PREVENT-011: no `any` type.
+ */
+// Shorthand builders to keep the inventory terse and unambiguous.
+const boolFlag = (key, label, description, def, requiresLlm = false) => ({
+    key,
+    label,
+    description,
+    type: "boolean",
+    default: def,
+    disabledConvention: true,
+    requiresLlm,
+});
+const boolDirect = (key, label, description, def) => ({
+    key,
+    label,
+    description,
+    type: "boolean",
+    default: def,
+    disabledConvention: false,
+    requiresLlm: false,
+});
+const num = (key, label, description, def, min, max, unit) => ({
+    key,
+    label,
+    description,
+    type: "number",
+    default: def,
+    disabledConvention: false,
+    requiresLlm: false,
+    min,
+    max,
+    ...(unit ? { unit } : {}),
+});
+const str = (key, label, description, def) => ({
+    key,
+    label,
+    description,
+    type: "string",
+    default: def,
+    disabledConvention: false,
+    requiresLlm: false,
+});
+/** Every adjustable setting, grouped by category. Read-only after module load. */
+export const SETTINGS = [
+    {
+        name: "RAG Pipeline",
+        settings: [
+            boolFlag("MEGACOMPACT_QUERY_REFORMULATION", "Query Reformulation", "TF-IDF keyword expansion for vague recall queries", true),
+            boolFlag("MEGACOMPACT_TIERED_ROUTER", "Tiered Recall Router", "L0 cache → L1 FTS5 → L2 HNSW routing", true),
+            boolFlag("MEGACOMPACT_RECALL_METRICS", "Recall Quality Metrics", "Precision/recall scoring and logging", true),
+            boolFlag("MEGACOMPACT_MEMORY_GRAPH", "Memory Graph", "Dashboard-oriented memory graph traversal", true),
+            boolFlag("MEGACOMPACT_HYDE", "HyDE", "Generate hypothetical answer via LLM, embed it, RRF-fuse", true, true),
+            boolFlag("MEGACOMPACT_NEW_UI", "New Dashboard UI", "Tailwind + shadcn visual design (sidebar, glass panels)", true),
+        ],
+    },
+    {
+        name: "Wiki / Turns",
+        settings: [
+            boolFlag("MEGACOMPACT_WIKI_ENHANCED", "Wiki Enhanced", "User curation (rename/merge/split) + topic evolution", true),
+            boolFlag("MEGACOMPACT_WIKI_INCREMENTAL", "Wiki Incremental", "Assign new memories to nearest centroid", true),
+            boolDirect("MEGACOMPACT_AUTO_WIKI", "Auto Wiki", "Auto-categorizing wiki (k-means + TF-IDF)", true),
+            boolDirect("MEGACOMPACT_WIKI_SEED_FROM_TURNS", "Wiki Seed from Turns", "Seed topic model from raw_transcript when chunks thin", true),
+            boolDirect("MEGACOMPACT_TURNS_DB", "Turns DB", "Isolated turns.db store", true),
+            num("MEGACOMPACT_WIKI_SILHOUETTE_MIN", "Silhouette Minimum", "Full rebuild trigger threshold", 0.2, 0, 1),
+            num("MEGACOMPACT_TURNS_RETENTION_DAYS", "Turn Retention", "Per-turn provenance retention window", 30, 1, 365, "days"),
+            num("MEGACOMPACT_TURNS_KEEP_MIN", "Turns Keep Minimum", "Min turns kept per conversation during prune", 5, 1, 100),
+            num("MEGACOMPACT_WIKI_LABEL_TOP_TERMS", "Wiki Label Terms", "How many TF-IDF terms form a topic label", 5, 1, 20),
+            num("MEGACOMPACT_WIKI_REBUILD_EVERY", "Wiki Rebuild Every", "Rebuild topic model every Nth compaction", 3, 1, 50),
+        ],
+    },
+    {
+        name: "Dedup Tiers",
+        settings: [
+            boolDirect("MEGACOMPACT_L0_ENABLED", "L0 Exact Match", "Exact-hash dedup tier", true),
+            boolDirect("MEGACOMPACT_L1_ENABLED", "L1 MinHash/LSH", "Near-dup tier", true),
+            boolDirect("MEGACOMPACT_L2_ENABLED", "L2 Semantic", "Cosine similarity dedup tier", true),
+            boolDirect("MEGACOMPACT_RAPTOR_ENABLED", "RAPTOR", "Hierarchical clustering tier", true),
+            boolDirect("MEGACOMPACT_RAPTOR_MULTILEVEL", "RAPTOR Multi-level", "Multi-level retrieval (score all tree levels)", true),
+            boolDirect("MEGACOMPACT_RAPTOR_LEAF_EXPANSION", "RAPTOR Leaf Expansion", "Leaf node expansion in retrieval", true),
+            boolDirect("MEGACOMPACT_RAPTOR_INCREMENTAL", "RAPTOR Incremental", "Incremental rebuild mode", true),
+            boolDirect("MEGACOMPACT_RAPTOR_INJECT_SUMMARIES", "RAPTOR Inject Summaries", "Inject top-level summary nodes into recall", true),
+            boolDirect("MEGACOMPACT_MARK_ONLY_L0", "Mark Only L0", "L0 runs but does not collapse", false),
+            boolDirect("MEGACOMPACT_MARK_ONLY_L1", "Mark Only L1", "L1 runs but does not collapse", false),
+            boolDirect("MEGACOMPACT_MARK_ONLY_L2", "Mark Only L2", "L2 runs but does not collapse", false),
+            boolDirect("MEGACOMPACT_MINILM", "MiniLM Embedder", "Use MiniLM instead of trigram", false),
+        ],
+    },
+    {
+        name: "Dedup Thresholds",
+        settings: [
+            num("MEGACOMPACT_L2_THRESHOLD", "L2 Cosine Threshold", "L2 semantic dedup firing point", 0.85, 0, 1),
+            num("MEGACOMPACT_L1_JACCARD", "L1 Jaccard Threshold", "L1 MinHash near-dup threshold", 0.8, 0, 1),
+            num("MEGACOMPACT_DEDUP_SIM", "Dedup Similarity", "Legacy content-similarity fallback", 0.9, 0, 1),
+            num("MEGACOMPACT_MMR_LAMBDA", "MMR Lambda", "Maximal Marginal Relevance diversity", 0.5, 0, 1),
+            num("MEGACOMPACT_SEMDEDUP_COSINE", "SemDeDup Cosine", "Offline SemDeDup pair threshold", 0.95, 0, 1),
+            num("MEGACOMPACT_CONSOLIDATE_COSINE", "Consolidate Cosine", "Memory consolidation merge threshold", 0.7, 0, 1),
+            num("MEGACOMPACT_SIMILARITY_BUDGET_MS", "Similarity Budget", "Time budget for similarity computation", 50, 1, 10000, "ms"),
+            num("MEGACOMPACT_L1_VERIFY_BUDGET_MS", "L1 Verify Budget", "L1 verification time budget", 20, 1, 10000, "ms"),
+            num("MEGACOMPACT_L1_CANDIDATE_CAP", "L1 Candidate Cap", "L1 candidate cap", 100, 1, 10000),
+        ],
+    },
+    {
+        name: "RAPTOR Tuning",
+        settings: [
+            num("MEGACOMPACT_RAPTOR_BUDGET_MS", "RAPTOR Budget", "RAPTOR time budget", 5000, 100, 60000, "ms"),
+            num("MEGACOMPACT_RAPTOR_CLUSTERS", "RAPTOR Clusters", "Clusters per level", 5, 1, 50),
+            num("MEGACOMPACT_RAPTOR_CONSISTENCY", "RAPTOR Consistency", "Consistency threshold", 0.6, 0, 1),
+            num("MEGACOMPACT_RAPTOR_MAX_LEAF_EXP", "RAPTOR Max Leaf Expansion", "Max leaf expansion nodes", 10, 1, 100),
+            num("MEGACOMPACT_RAPTOR_FRESHNESS_HOURS", "RAPTOR Freshness", "Skip rebuild when tree is fresh", 4, 1, 168, "hours"),
+            num("MEGACOMPACT_FP_RATE_L0", "FP Rate L0", "FP alert threshold for exact tier", 0.01, 0, 1),
+            num("MEGACOMPACT_FP_RATE_L1L2", "FP Rate L1/L2", "FP alert threshold for fuzzy tiers", 0.05, 0, 1),
+            num("MEGACOMPACT_ALERT_WINDOW_MS", "Alert Window", "Alert evaluation window", 600000, 1000, 3600000, "ms"),
+            num("MEGACOMPACT_P95_BUDGET_MS", "P95 Budget", "Canary p95 budget per tier", 100, 1, 10000, "ms"),
+        ],
+    },
+    {
+        name: "Models",
+        settings: [
+            str("MEGACOMPACT_HYDE_MODEL", "HyDE Model", "LLM model name for HyDE generation", "llama3.2"),
+            str("MEGACOMPACT_RAPTOR_MODEL", "RAPTOR Summary Model", "Ollama model for cluster summarization (empty = extractive)", ""),
+            str("MEGACOMPACT_RAPTOR_URL", "RAPTOR Ollama URL", "Ollama endpoint for RAPTOR summarization", "http://127.0.0.1:11434"),
+            num("MEGACOMPACT_EMBED_CACHE", "Embed Cache Size", "Embedding cache entries (0 = disabled)", 256, 0, 10000),
+        ],
+    },
+];
+/** Flat index of every setting by key for O(1) lookup + validation. */
+export const SETTING_BY_KEY = new Map(SETTINGS.flatMap((c) => c.settings).map((s) => [s.key, s]));
+/**
+ * Adjustable MEGACOMPACT_* env vars intentionally NOT surfaced in the settings
+ * UI. Handled by other panels, install-time paths, dev-only, internal tuning,
+ * or special string formats the generic form cannot represent.
+ * Kept as string literals so regression_check.py's coverage scan sees them.
+ */
+export const EXCLUDED_SETTINGS = [
+    // Embedder telemetry — handled by EmbedderSetup.
+    "MEGACOMPACT_EMBEDDING_URL",
+    "MEGACOMPACT_EMBEDDING_KEY",
+    "MEGACOMPACT_EMBEDDING_HEADERS",
+    "MEGACOMPACT_EMBEDDING_DIM",
+    "MEGACOMPACT_OLLAMA_MODEL",
+    "MEGACOMPACT_ALLOW_REMOTE_EMBEDDER",
+    // Install-time paths.
+    "MEGACOMPACT_STATE_DIR",
+    "MEGACOMPACT_INDEX_DIR",
+    "MEGACOMPACT_VECTOR_INDEX_DIR",
+    "MEGACOMPACT_TURNS_DB_PATH",
+    // Dev-only.
+    "MEGACOMPACT_EXT_SCAN_DIR",
+    "MEGACOMPACT_EXT_USER_DIR",
+    // PGlite internal tuning.
+    "MEGACOMPACT_PGLITE_DISABLED",
+    "MEGACOMPACT_PGLITE_OPEN_TIMEOUT_MS",
+    "MEGACOMPACT_PGLITE_CLOSE_TIMEOUT_MS",
+    "MEGACOMPACT_PGLITE_QUERY_TIMEOUT_MS",
+    // Special string formats the generic form cannot represent.
+    "MEGACOMPACT_WIKI_K_RANGE",
+    "MEGACOMPACT_RAPTOR_LEVEL_WEIGHTS",
+    // Internal calibration, not user-facing.
+    "MEGACOMPACT_FTS5_MAX_BM25",
+];

package/dist/extensions/dashboard-server/routes-rag-settings.js

@@ -1,9 +1,13 @@
 /**
- * dashboard-server/routes-rag-settings.ts — RAG Settings route handler.
+ * dashboard-server/routes-rag-settings.ts — comprehensive settings route handler.
  *
- * GET /api/rag-settings  — Returns the state of all RAG feature flags (B1–B5).
- * POST /api/rag-settings — Toggles flags by writing MEGACOMPACT_*_DISABLED
- *                          lines to the per-repo .mega-compact.env file.
+ * GET /api/rag-settings  — Returns every adjustable MEGACOMPACT_* setting,
+ *                          grouped by category, with live values from env.
+ * POST /api/rag-settings — Updates a single setting by writing its line to the
+ *                          per-repo .mega-compact.env file.
+ *
+ * The SETTINGS inventory lives in routes-rag-settings-helpers.ts so this file
+ * stays within the line budget.
  *
  * Guardrails: PREVENT-PI-004 (loopback-only), PREVENT-001 (null-safe JSON),
  * PREVENT-011 (no `any`). Each JSON write carries a guardrails-allow annotation.
@@ -11,47 +15,83 @@
 import { readFileSync, existsSync, writeFileSync, mkdirSync } from "node:fs";
 import { join } from "node:path";
 import { detectCurrentEmbedder } from "./routes-setup.js";
+import { SETTINGS, SETTING_BY_KEY } from "./routes-rag-settings-helpers.js";
 // ---------------------------------------------------------------------------
-// RAG_FLAGS — the five feature flags surfaced by this panel.
+// Env-reading helpers — one per setting type.
 // ---------------------------------------------------------------------------
-const RAG_FLAGS = [
-    {
-        key: "MEGACOMPACT_QUERY_REFORMULATION",
-        label: "Query Reformulation",
-        description: "TF-IDF keyword expansion for vague recall queries (TrigramEmbedder path)",
-        requiresLlm: false,
-    },
-    {
-        key: "MEGACOMPACT_TIERED_ROUTER",
-        label: "Tiered Recall Router",
-        description: "L0 cache → L1 FTS5 → L2 HNSW routing for faster recall",
-        requiresLlm: false,
-    },
-    {
-        key: "MEGACOMPACT_RECALL_METRICS",
-        label: "Recall Quality Metrics",
-        description: "Precision/recall scoring and logging for recall evaluation",
-        requiresLlm: false,
-    },
-    {
-        key: "MEGACOMPACT_MEMORY_GRAPH",
-        label: "Memory Graph",
-        description: "Dashboard-oriented memory graph traversal across sessions",
-        requiresLlm: false,
-    },
-    {
-        key: "MEGACOMPACT_HYDE",
-        label: "HyDE (Hypothetical Document Embeddings)",
-        description: "Generate hypothetical answer via LLM, embed it, RRF-fuse with raw-query results",
-        requiresLlm: true,
-    },
-];
-/** Strip any MEGACOMPACT_*_DISABLED assignment lines from env file content. */
-const DISABLED_LINE = /^export\s+MEGACOMPACT_\w+_DISABLED=/;
 function isDisabled(key) {
     const v = process.env[key + "_DISABLED"];
     return v === "true" || v === "1";
 }
+function isEnabled(key, def) {
+    const v = process.env[key];
+    if (v === undefined)
+        return def;
+    return v === "true" || v === "1";
+}
+function readNumber(key, def) {
+    const v = process.env[key];
+    if (v === undefined)
+        return def;
+    const n = Number(v);
+    return Number.isNaN(n) ? def : n;
+}
+function resolveValue(spec) {
+    if (spec.type === "boolean") {
+        return spec.disabledConvention
+            ? !isDisabled(spec.key)
+            : isEnabled(spec.key, spec.default);
+    }
+    if (spec.type === "number") {
+        return readNumber(spec.key, spec.default);
+    }
+    return process.env[spec.key] ?? spec.default;
+}
+function toSettingState(spec, category) {
+    return {
+        key: spec.key,
+        label: spec.label,
+        description: spec.description,
+        category,
+        type: spec.type,
+        value: resolveValue(spec),
+        default: spec.default,
+        disabledConvention: spec.disabledConvention,
+        requiresLlm: spec.requiresLlm,
+        ...(spec.unit !== undefined ? { unit: spec.unit } : {}),
+        ...(spec.min !== undefined ? { min: spec.min } : {}),
+        ...(spec.max !== undefined ? { max: spec.max } : {}),
+    };
+}
+// ---------------------------------------------------------------------------
+// Env file I/O.
+// ---------------------------------------------------------------------------
+/** Strip lines assigning KEY or KEY_DISABLED from env file content. */
+const keyLinePattern = (key) => new RegExp(`^export\\s+${key}(_DISABLED)?=`);
+function envPathOf(ctx) {
+    return join(ctx.stateDir, ".mega-compact.env");
+}
+function readEnvLines(envPath, key) {
+    if (!existsSync(envPath))
+        return [];
+    return readFileSync(envPath, "utf-8")
+        .split("\n")
+        .filter((l) => !keyLinePattern(key).test(l));
+}
+function writeSetting(envPath, key, line) {
+    const lines = readEnvLines(envPath, key);
+    lines.push(line);
+    if (lines[lines.length - 1] !== "")
+        lines.push("");
+    writeFileSync(envPath, lines.join("\n"), "utf-8");
+}
+/** Strip KEY/KEY_DISABLED lines, leaving the file otherwise intact. */
+function removeSetting(envPath, key) {
+    writeFileSync(envPath, readEnvLines(envPath, key).join("\n"), "utf-8");
+}
+// ---------------------------------------------------------------------------
+// Request body parsing.
+// ---------------------------------------------------------------------------
 function readJsonBody(req, cb) {
     let body = "";
     let tooBig = false;
@@ -85,15 +125,12 @@ export function handleRagSettings(req, res, ctx) {
     if (req.url !== "/api/rag-settings")
         return false;
     if (req.method === "GET") {
-        const flags = RAG_FLAGS.map((f) => ({
-            key: f.key,
-            label: f.label,
-            description: f.description,
-            enabled: !isDisabled(f.key),
-            requiresLlm: f.requiresLlm,
+        const categories = SETTINGS.map((cat) => ({
+            name: cat.name,
+            settings: cat.settings.map((s) => toSettingState(s, cat.name)),
         }));
         const body = {
-            flags,
+            categories,
             llmActive: detectCurrentEmbedder() === "http",
         };
         // guardrails-allow PREVENT-PI-004: loopback dashboard response (local)
@@ -109,47 +146,63 @@ export function handleRagSettings(req, res, ctx) {
                 res.end(JSON.stringify({ error: parsed.error }));
                 return;
             }
-            const body = parsed.value;
-            const desired = body.flags;
-            if (typeof desired !== "object" || desired === null || Array.isArray(desired)) {
+            const reqBody = parsed.value;
+            const key = reqBody.key;
+            const value = reqBody.value;
+            if (typeof key !== "string" || typeof value !== "string") {
                 // guardrails-allow PREVENT-PI-004: loopback dashboard response (local)
                 res.writeHead(400, { "Content-Type": "application/json" });
-                res.end(JSON.stringify({ error: "invalid_flags" }));
+                res.end(JSON.stringify({ error: "invalid_body" }));
                 return;
             }
-            // Only accept known RAG flag keys.
-            const known = new Set(RAG_FLAGS.map((f) => f.key));
-            for (const key of Object.keys(desired)) {
-                if (!known.has(key)) {
-                    // guardrails-allow PREVENT-PI-004: loopback dashboard response (local)
-                    res.writeHead(400, { "Content-Type": "application/json" });
-                    res.end(JSON.stringify({ error: `unknown_flag: ${key}` }));
-                    return;
-                }
-            }
-            const envPath = join(ctx.stateDir, ".mega-compact.env");
-            let lines = [];
-            if (existsSync(envPath)) {
-                const content = readFileSync(envPath, "utf-8");
-                lines = content.split("\n").filter((line) => !DISABLED_LINE.test(line));
-            }
-            for (const key of Object.keys(desired)) {
-                if (desired[key] === false) {
-                    lines.push(`export ${key}_DISABLED="true"`);
-                }
+            const spec = SETTING_BY_KEY.get(key);
+            if (!spec) {
+                // guardrails-allow PREVENT-PI-004: loopback dashboard response (local)
+                res.writeHead(400, { "Content-Type": "application/json" });
+                res.end(JSON.stringify({ error: `unknown_setting: ${key}` }));
+                return;
             }
-            if (lines.length > 0 && lines[lines.length - 1] !== "")
-                lines.push("");
+            const envPath = envPathOf(ctx);
             try {
                 mkdirSync(ctx.stateDir, { recursive: true });
-                writeFileSync(envPath, lines.join("\n"), "utf-8");
+                if (spec.type === "boolean") {
+                    if (value !== "true" && value !== "false") {
+                        throw new SettingError("boolean_value_required");
+                    }
+                    if (spec.disabledConvention) {
+                        // false → opt out via KEY_DISABLED="true"; true → strip the line.
+                        if (value === "false") {
+                            writeSetting(envPath, spec.key, `export ${spec.key}_DISABLED="true"`);
+                        }
+                        else {
+                            removeSetting(envPath, spec.key);
+                        }
+                    }
+                    else {
+                        writeSetting(envPath, spec.key, `export ${spec.key}="${value}"`);
+                    }
+                }
+                else if (spec.type === "number") {
+                    const n = Number(value);
+                    if (Number.isNaN(n))
+                        throw new SettingError("invalid_number");
+                    if (spec.min !== undefined && n < spec.min)
+                        throw new SettingError("below_min");
+                    if (spec.max !== undefined && n > spec.max)
+                        throw new SettingError("above_max");
+                    writeSetting(envPath, spec.key, `export ${spec.key}="${n}"`);
+                }
+                else {
+                    writeSetting(envPath, spec.key, `export ${spec.key}="${value}"`);
+                }
             }
             catch (e) {
+                const msg = e instanceof SettingError
+                    ? e.message
+                    : `write_failed: ${e instanceof Error ? e.message : String(e)}`;
                 // guardrails-allow PREVENT-PI-004: loopback dashboard response (local)
-                res.writeHead(500, { "Content-Type": "application/json" });
-                res.end(JSON.stringify({
-                    error: `write_failed: ${e instanceof Error ? e.message : String(e)}`,
-                }));
+                res.writeHead(400, { "Content-Type": "application/json" });
+                res.end(JSON.stringify({ error: msg }));
                 return;
             }
             const resp = {
@@ -167,3 +220,9 @@ export function handleRagSettings(req, res, ctx) {
     res.end(JSON.stringify({ error: "method_not_allowed" }));
     return true;
 }
+class SettingError extends Error {
+    constructor(message) {
+        super(message);
+        this.name = "SettingError";
+    }
+}