@gamaze/hicortex 0.13.0 → 0.13.2

package/README.md

@@ -65,7 +65,7 @@ The plugin connects to `http://127.0.0.1:8787` by default. For a remote server,
 
 | When | What | How |
 |------|------|-----|
-| Agent start | Recent lessons fetched fresh and injected into context | CC SessionStart hook (calls `hicortex lessons-context`) / Hermes plugin prefetch / OC `before_agent_start` hook |
+| Agent start | Standing context (`## Context`) + recent lessons fetched fresh and injected | CC SessionStart hook (calls `hicortex lessons-context`) / Hermes plugin `system_prompt_block` (+ `prefetch` recall) / OC `before_agent_start` hook |
 | Nightly | Denoise sessions → POST /distill → server distills + embeds + stores → consolidate (score, reflect, link, decay) | Automatic pipeline — no manual steps |
 
 ## Memory Domains & Tags
@@ -126,7 +126,7 @@ Beyond auto-distilled memories and lessons, Hicortex holds a **hand-edited conte
 
 - **Storage:** plain files on the server at `~/.hicortex/context/*.md` — one file per section (recommended starter sections `user.md` + `rules.md`, which you create — nothing is pre-populated; add more by dropping in a file). It lives outside the memories table; consolidation never touches it.
 - **Edit:** the web editor at `http://localhost:8787/context/ui` (one tab per section, Save), or the CLI `hicortex context show [name]` / `hicortex context edit <name>`.
-- **Delivery:** injected into the harnesses listed in `contextClients` (default `["cc"]` — Claude Code only in v1; `"all"` or any subset of `cc`/`hermes`/`oc`).
+- **Delivery:** injected into the harnesses listed in `contextClients` (default `["cc"]`; `"all"` or any subset of `cc`/`hermes`/`oc` — all three are supported since 0.13).
 - **Deletion** is filesystem-only — remove the file on the server (as the daemon user).
 
 ### Per-agent context (0.13)
@@ -191,12 +191,9 @@ Config at `~/.hicortex/config.json`. Created by `init`. Key options:
 |-------|-------------|
 | `mode` | `"server"` (default) or `"client"` |
 | `serverUrl` | Remote server URL (client mode) |
-| `llmModel` | Model for importance scoring (server mode) |
-| `distillModel` | Model for session distillation — 9b+ recommended (server mode) |
-| `distillBaseUrl` | Separate Ollama instance for distillation (server mode) |
+| `llmModel` | The one model the whole pipeline uses (server mode). One model is the happy path; route individual stages with `models` — see [Advanced: per-stage models](#advanced-per-stage-models) |
+| `models` | Optional nested per-stage model overrides (`score`/`distill`/`reflect`/`classify`) — see [Advanced: per-stage models](#advanced-per-stage-models) |
 | `distillFallback` | `"strict"` (default) — abort on remote distill failure, retry next run; `"local"` — fall back to base model (lower quality, 0.9.0 behaviour) |
-| `reflectModel` | Model for nightly reflection — largest available (server mode) |
-| `reflectBaseUrl` | Separate Ollama instance for reflection (server mode) |
 | `authToken` | Bearer token for endpoint auth. Generated on first `init` in server mode. Find the active token with `hicortex status` or in `~/.hicortex/config.json`. |
 | `licenseKey` | Commercial license key (optional; for display in `hicortex status`) |
 | `domains` | Your memory domain list (`[{name, description}]`). Scaffolded by `init`; edit freely — see [Memory Domains & Tags](#memory-domains--tags) |
@@ -263,6 +260,30 @@ LLM selection is **user-controlled**: `npx @gamaze/hicortex init` detects candid
 
 If no LLM is configured, the server starts in **recall-only mode**: search, lessons, and context work; `/distill` and consolidation are disabled. Run `npx @gamaze/hicortex init` to configure.
 
+### Advanced: per-stage models
+
+The happy path is **one model** (`llmModel`) for the whole pipeline. If you want to route the four pipeline stages to different models or endpoints, add a nested `models` block to `~/.hicortex/config.json`:
+
+```json
+{
+  "llmBackend": "ollama",
+  "llmModel": "qwen3.5:4b",
+  "models": {
+    "score":    { "model": "qwen3.5:4b" },
+    "distill":  { "model": "qwen3.5:35b-a3b", "baseUrl": "http://gpu-box:11434" },
+    "reflect":  { "model": "qwen3.5:35b-a3b", "baseUrl": "http://gpu-box:11434" },
+    "classify": { "model": "gemma4-31b",      "baseUrl": "http://gpu-box:11434" }
+  }
+}
+```
+
+The four tiers: `score` (importance scoring — this **is** the base model), `distill` (session distillation, 9b+ recommended), `reflect` (nightly reflection, largest available), `classify` (memory-domain tagging). Each accepts `model`, `baseUrl`, `apiKey`, and `provider`. Omitting a tier inherits: `distill` and `reflect` fall back to the **base** (`score`) model; **`classify` falls back to the `reflect` tier** (not the base — `classify` delegates to the reflect path when unset).
+
+- **Flat keys still work.** `distillModel`/`distillBaseUrl`/`reflectModel`/`reflectBaseUrl`/`classifyModel`/`classifyBaseUrl` remain supported at **lower precedence** — a `models` entry wins over the flat key of the same name.
+- **A tier's `apiKey`/`provider` require the tier's own `baseUrl`.** They only take effect when the tier sets `baseUrl`; set on a tier without a `baseUrl` they are **ignored with a warning** (a bare `models.reflect: { model, apiKey }` would otherwise silently bill to the base key). Set `provider` when a tier's `baseUrl` points at a different provider type than the base (e.g. an OpenAI-compatible API while the base is Ollama); set `apiKey` for an API-provider tier over an Ollama base, whose base key is empty (`""`).
+- **`init` writes a flat `llmModel`.** The already-configured guard now recognizes a nested-only config (`models.score`), so re-running `init` on one is a no-op. But if a config has **both** a flat `llmModel` and a `models.score.model`, the nested value **shadows** the flat one (nested > flat) — keep the model in one place.
+- **`score.provider` (and `score.apiKey` on an Ollama base) are ignored** (a warning is logged): the base provider comes from `llmBackend` (or is auto-detected from the base endpoint), and the Ollama base path sends no api key.
+
 ## Database
 
 Canonical location: `~/.hicortex/hicortex.db`. The OC plugin no longer owns its own database — it is a thin client to the server. Previously, OC installations at `~/.openclaw/data/hicortex.db` were migrated automatically on upgrade; this migration path remains in the server's `resolveDbPath` for any pre-0.10.0 installations.

package/dist/capture-cursors.d.ts

@@ -0,0 +1,73 @@
+/**
+ * Per-session capture cursors (#189).
+ *
+ * The client is the source of truth for "how much of each session has already
+ * been captured". A cursor is advanced ONLY after the server confirms the
+ * corresponding segment(s) were ingested, so a crash between POST and cursor
+ * write can at worst cause a bounded, idempotent re-send — never silent loss.
+ *
+ * Storage: a SEPARATE small file `<hicortex-home>/capture-cursors.json`
+ * (NOT state.json — that file carries the large moduleIndex, and per-session
+ * whole-file rewrites there would be an avoidable corruption/IO surface). Same
+ * temp+rename atomic write discipline as state.ts.
+ *
+ * Cursor unit is reader-defined:
+ *   - JSONL readers (CC/Pi/OC): count of successfully-PARSED entries consumed.
+ *   - Hermes: max `messages.id` consumed (INTEGER PRIMARY KEY AUTOINCREMENT —
+ *     never reused, strictly increasing).
+ *
+ * `gen` (generation) is bumped by a reader's shrink-guard when a source file is
+ * truncated/rotated below the stored cursor. It is woven into the segment id
+ * (`g<gen>.<start>-<end>`) so post-reset segments can NEVER collide with a
+ * pre-reset id on the server's content-blind segment-exact dedup — a collision
+ * there would be silent LOSS, not the intended dup-over-loss (#189 review, fix 8).
+ *
+ * Keys are `<prefix>:<sessionId>`:
+ *   cc:<sid>  pi:<sid>  oc:<agentId>:<sid>  hermes:<profile>:<sid>
+ */
+export interface CursorEntry {
+    /** Reader-defined cursor value (entry count for JSONL, max id for Hermes). */
+    cursor: number;
+    /** Shrink-guard generation — woven into segment ids to avoid post-reset id collisions. */
+    gen: number;
+    /** ISO timestamp of the last advance — used for 90-day pruning. */
+    updated: string;
+}
+export type CursorFile = Record<string, CursorEntry>;
+/** Snapshot of a session's captured position, passed to the readers. */
+export interface CursorPosition {
+    cursor: number;
+    gen: number;
+}
+/** Map of cursor key → captured position. */
+export type CursorMap = Record<string, CursorPosition>;
+/** Load the cursor file. Missing/corrupt → empty (every key defaults to 0). */
+export declare function loadCursors(stateDir?: string): CursorFile;
+/**
+ * Atomically persist the cursor file (write-temp + rename). THROWS on failure —
+ * a persistent inability to record cursors must surface as a transient failure
+ * (hold the watermark), never a silent warn-and-continue that re-captures the
+ * same content every night (#189 review, fix 7).
+ */
+export declare function saveCursors(file: CursorFile, stateDir?: string): void;
+/**
+ * A read-once, write-on-advance cursor store. Single-flight (see
+ * acquireCaptureLock in capture.ts) guarantees no concurrent writer, so an
+ * in-memory copy flushed atomically on each advance is safe.
+ */
+export interface CursorStore {
+    /** Current position for `key` ({cursor:0, gen:0} when never captured). */
+    get(key: string): CursorPosition;
+    /** Advance `key` to `cursor` at `gen` and persist. Throws if the write fails. */
+    advance(key: string, cursor: number, gen: number): void;
+    /** Snapshot of key → position for passing to readers. */
+    map(): CursorMap;
+}
+/** Open a file-backed cursor store for `stateDir`. */
+export declare function openCursorStore(stateDir?: string): CursorStore;
+/**
+ * Drop cursor entries older than `days` (default 90). Best-effort: a prune-write
+ * failure is logged, not thrown (prune runs only after a clean nightly and must
+ * not fail the run). Returns the number pruned.
+ */
+export declare function pruneCursors(stateDir?: string, days?: number): number;

package/dist/capture-cursors.js

@@ -0,0 +1,133 @@
+"use strict";
+/**
+ * Per-session capture cursors (#189).
+ *
+ * The client is the source of truth for "how much of each session has already
+ * been captured". A cursor is advanced ONLY after the server confirms the
+ * corresponding segment(s) were ingested, so a crash between POST and cursor
+ * write can at worst cause a bounded, idempotent re-send — never silent loss.
+ *
+ * Storage: a SEPARATE small file `<hicortex-home>/capture-cursors.json`
+ * (NOT state.json — that file carries the large moduleIndex, and per-session
+ * whole-file rewrites there would be an avoidable corruption/IO surface). Same
+ * temp+rename atomic write discipline as state.ts.
+ *
+ * Cursor unit is reader-defined:
+ *   - JSONL readers (CC/Pi/OC): count of successfully-PARSED entries consumed.
+ *   - Hermes: max `messages.id` consumed (INTEGER PRIMARY KEY AUTOINCREMENT —
+ *     never reused, strictly increasing).
+ *
+ * `gen` (generation) is bumped by a reader's shrink-guard when a source file is
+ * truncated/rotated below the stored cursor. It is woven into the segment id
+ * (`g<gen>.<start>-<end>`) so post-reset segments can NEVER collide with a
+ * pre-reset id on the server's content-blind segment-exact dedup — a collision
+ * there would be silent LOSS, not the intended dup-over-loss (#189 review, fix 8).
+ *
+ * Keys are `<prefix>:<sessionId>`:
+ *   cc:<sid>  pi:<sid>  oc:<agentId>:<sid>  hermes:<profile>:<sid>
+ */
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.loadCursors = loadCursors;
+exports.saveCursors = saveCursors;
+exports.openCursorStore = openCursorStore;
+exports.pruneCursors = pruneCursors;
+const paths_js_1 = require("./paths.js");
+const node_fs_1 = require("node:fs");
+const node_path_1 = require("node:path");
+const HICORTEX_HOME = (0, paths_js_1.hicortexHome)();
+const CURSORS_FILE = "capture-cursors.json";
+/** Coerce a stored cursor value to a valid non-negative integer (fix 13). */
+function sanitizeCursor(raw, key) {
+    if (typeof raw === "number" && Number.isFinite(raw) && raw >= 0) {
+        return Math.floor(raw);
+    }
+    console.warn(`[hicortex] capture-cursors: ignoring invalid cursor for ${key} (${JSON.stringify(raw)}) — treating as 0`);
+    return 0;
+}
+function sanitizeGen(raw) {
+    return typeof raw === "number" && Number.isFinite(raw) && raw >= 0 ? Math.floor(raw) : 0;
+}
+/** Load the cursor file. Missing/corrupt → empty (every key defaults to 0). */
+function loadCursors(stateDir = HICORTEX_HOME) {
+    let parsed;
+    try {
+        parsed = JSON.parse((0, node_fs_1.readFileSync)((0, node_path_1.join)(stateDir, CURSORS_FILE), "utf-8"));
+    }
+    catch {
+        return {};
+    }
+    if (!parsed || typeof parsed !== "object")
+        return {};
+    const out = {};
+    for (const [key, val] of Object.entries(parsed)) {
+        if (!val || typeof val !== "object")
+            continue;
+        const v = val;
+        out[key] = {
+            cursor: sanitizeCursor(v.cursor, key),
+            gen: sanitizeGen(v.gen),
+            updated: typeof v.updated === "string" ? v.updated : new Date().toISOString(),
+        };
+    }
+    return out;
+}
+/**
+ * Atomically persist the cursor file (write-temp + rename). THROWS on failure —
+ * a persistent inability to record cursors must surface as a transient failure
+ * (hold the watermark), never a silent warn-and-continue that re-captures the
+ * same content every night (#189 review, fix 7).
+ */
+function saveCursors(file, stateDir = HICORTEX_HOME) {
+    (0, node_fs_1.mkdirSync)(stateDir, { recursive: true });
+    const path = (0, node_path_1.join)(stateDir, CURSORS_FILE);
+    const tmp = `${path}.tmp`;
+    (0, node_fs_1.writeFileSync)(tmp, JSON.stringify(file, null, 2));
+    (0, node_fs_1.renameSync)(tmp, path);
+}
+/** Open a file-backed cursor store for `stateDir`. */
+function openCursorStore(stateDir = HICORTEX_HOME) {
+    const file = loadCursors(stateDir);
+    return {
+        get(key) {
+            const e = file[key];
+            return { cursor: e?.cursor ?? 0, gen: e?.gen ?? 0 };
+        },
+        advance(key, cursor, gen) {
+            file[key] = { cursor, gen, updated: new Date().toISOString() };
+            saveCursors(file, stateDir); // throws on failure — caller holds the watermark
+        },
+        map() {
+            const out = {};
+            for (const [k, v] of Object.entries(file))
+                out[k] = { cursor: v.cursor, gen: v.gen };
+            return out;
+        },
+    };
+}
+/**
+ * Drop cursor entries older than `days` (default 90). Best-effort: a prune-write
+ * failure is logged, not thrown (prune runs only after a clean nightly and must
+ * not fail the run). Returns the number pruned.
+ */
+function pruneCursors(stateDir = HICORTEX_HOME, days = 90) {
+    const file = loadCursors(stateDir);
+    const cutoff = Date.now() - days * 24 * 60 * 60 * 1000;
+    let pruned = 0;
+    for (const [key, entry] of Object.entries(file)) {
+        const t = Date.parse(entry.updated);
+        if (!Number.isNaN(t) && t < cutoff) {
+            delete file[key];
+            pruned++;
+        }
+    }
+    if (pruned > 0) {
+        try {
+            saveCursors(file, stateDir);
+        }
+        catch (err) {
+            console.warn(`[hicortex] capture-cursors prune write failed: ${err instanceof Error ? err.message : String(err)}`);
+            return 0;
+        }
+    }
+    return pruned;
+}

package/dist/capture.d.ts

@@ -0,0 +1,124 @@
+/**
+ * Incremental, cursor-aware capture loop (#189).
+ *
+ * Extracted from the two near-identical loops that lived in nightly.ts (server
+ * and client mode). Both now share this logic: pack each session's delta into
+ * ordered segments below the server's distill cap, POST them in order with a
+ * deterministic `segment_id`, and advance the per-session cursor ONLY after
+ * server-confirmed success — so a multi-day session grows across nights with no
+ * loss and no silent truncation.
+ *
+ * The POST transport is injected (`post`) so the mode-specific bits (localhost
+ * vs remote URL, Authorization header, timeout) stay in nightly.ts and the
+ * multi-night simulation can run as a pure unit test with no HTTP listener.
+ */
+import type { TranscriptBatch } from "./transcript-reader.js";
+import type { CursorStore } from "./capture-cursors.js";
+/**
+ * Max denoised chars per segment. Kept below the server's 80K distill cap
+ * (distiller.ts MAX_TRANSCRIPT_CHARS) with ~20K headroom so NO capture path can
+ * hit the silent truncation. LOAD-BEARING for #189 recovery: a re-ingested
+ * week-long session is re-sliced into ≤60K segments here instead of being
+ * truncated at 80K server-side. (Judgment constant — tunable later.)
+ */
+export declare const SEGMENT_MAX_CHARS = 60000;
+/**
+ * Minimum denoised chars for a FRESH whole session (startCursor 0) to be worth
+ * capturing — mirrors the long-standing pre-#189 200-char degenerate-session
+ * gate. It is applied ONLY to a whole-session capture that denoises to a single
+ * sub-200 segment. A delta beyond cursor 0 is always sent, however small: a
+ * session's concluding tail must never be held back, because once the session
+ * stops growing its mtime never re-crosses the watermark and the tail would be
+ * lost forever (#189 review, fix 5).
+ */
+export declare const MIN_SEGMENT_CHARS = 200;
+/** One packed, ready-to-POST segment of a session's delta. */
+export interface Segment {
+    /** Denoised text body of the POST. */
+    text: string;
+    /** Cursor value the segment starts at. */
+    segStart: number;
+    /** Cursor value the segment ends at. */
+    segEnd: number;
+    /**
+     * Disambiguator for hard-split pieces of a single oversized entry (A2). Empty
+     * for normal segments; ".p0", ".p1", … when one entry is split mid-text.
+     * Keeps the server's `<sid>#<segment_id>#<i>` keys distinct so no piece's
+     * memories collide on the UNIQUE index.
+     */
+    idSuffix: string;
+}
+/** The wire body for POST /distill. */
+export interface DistillBody {
+    text: string;
+    source_agent: string;
+    project: string;
+    session_id: string;
+    segment_id: string;
+    session_date: string;
+    privacy: string;
+}
+/** Normalized POST result the caller's transport returns. */
+export interface PostResult {
+    status: number;
+    distilled?: number;
+    dropped?: string[];
+    skipped?: boolean;
+    error?: string;
+}
+export type PostFn = (body: DistillBody) => Promise<PostResult>;
+export interface CaptureOptions {
+    post: PostFn;
+    cursorStore: CursorStore;
+    dryRun?: boolean;
+    /** Segment size cap; defaults to SEGMENT_MAX_CHARS. Lowered in tests. */
+    segmentMaxChars?: number;
+}
+export interface CaptureResult {
+    memoriesIngested: number;
+    sessionsSent: number;
+    hadTransientFailure: boolean;
+    /**
+     * Set when the loop stopped early on a terminal server response: "limit"
+     * (429 memory cap) or "auth" (401). The caller decides watermark handling.
+     */
+    stopped?: "limit" | "auth";
+}
+/**
+ * Split an already-denoised string into ≤maxChars pieces (A2 hard-split).
+ * Prefers paragraph, then line, then hard boundaries — mirrors the distiller's
+ * own splitIntoChunks, but WITHOUT its <200-char drop (every piece must survive,
+ * dup-over-loss).
+ */
+export declare function hardSplitText(text: string, maxChars?: number): string[];
+/**
+ * Pack a session's delta entries into ordered ≤maxChars segments.
+ *
+ * Sizing uses per-entry denoise lengths plus the "\n\n" joiners (A8) so the
+ * estimate matches what the server receives; the actual body is a re-denoise of
+ * the grouped entries (extractConversationText) so cleaning/redaction stay
+ * coherent. A single entry larger than maxChars is emitted as its own run of
+ * hard-split pieces (A2).
+ */
+export declare function packSegments(entries: unknown[], startCursor: number, entryCursors: number[], maxChars?: number): Segment[];
+/**
+ * Capture a list of session delta batches: pack, POST in order, advance
+ * per-session cursors on success. Segments of one session POST in order; the
+ * first hard failure stops THAT session (cursor holds at the last confirmed
+ * boundary) while other sessions continue. A 429/401 stops the whole loop.
+ */
+export declare function captureBatches(batches: TranscriptBatch[], opts: CaptureOptions): Promise<CaptureResult>;
+/**
+ * Acquire an exclusive capture lock for `stateDir`. Returns a release function,
+ * or null if another LIVE, non-stale run holds it after waiting up to `waitMs`.
+ *
+ * Staleness = dead pid OR lockfile mtime older than LOCK_TTL_MS. A stale lock is
+ * reclaimed (with a re-verify + O_EXCL re-race to narrow the TOCTOU window,
+ * fix 12). `waitMs` lets the full nightly wait out a transient `--capture-only`
+ * overlap instead of dropping the night's capture (fix 10); pass 0 to fail fast.
+ *
+ * This stops a `nightly` and a `nightly --capture-only` (an encouraged workflow)
+ * from running the capture loop concurrently, which would race cursor writes and
+ * emit divergent segment boundaries → real duplication.
+ */
+export declare function acquireCaptureLock(stateDir: string, waitMs?: number): Promise<(() => void) | null>;