kongbrain 0.3.8 → 0.3.10

package/README.npm.md

@@ -0,0 +1,400 @@
+# KongBrain
+
+[![npm](https://img.shields.io/npm/v/kongbrain?style=for-the-badge&logo=npm&color=cb3837)](https://www.npmjs.com/package/kongbrain)
+[![ClawHub](https://img.shields.io/badge/ClawHub-kongbrain-ff6b35?style=for-the-badge)](https://clawhub.ai/packages/kongbrain)
+[![GitHub Stars](https://img.shields.io/github/stars/42U/kongbrain?style=for-the-badge&logo=github&color=gold)](https://github.com/42U/kongbrain)
+[![License: MIT](https://img.shields.io/github/license/42U/kongbrain?style=for-the-badge&logo=opensourceinitiative&color=blue)](https://opensource.org/licenses/MIT)
+[![Node.js](https://img.shields.io/badge/Node.js-20+-339933?style=for-the-badge&logo=node.js&logoColor=white)](https://nodejs.org)
+[![SurrealDB](https://img.shields.io/badge/SurrealDB-3.0-ff00a0?style=for-the-badge&logo=surrealdb&logoColor=white)](https://surrealdb.com)
+[![OpenClaw](https://img.shields.io/badge/OpenClaw-Plugin-ff6b35?style=for-the-badge)](https://github.com/openclaw/openclaw)
+[![Tests](https://img.shields.io/badge/Tests-88_passing-brightgreen?style=for-the-badge&logo=vitest&logoColor=white)](https://vitest.dev)
+
+**A graph-backed cognitive engine for [OpenClaw](https://github.com/openclaw/openclaw).**
+
+> *OpenClaw ships with a lobster brain. It works — lobsters have survived 350 million years — but they also solve problems by walking backwards and occasionally eating each other.*
+>
+> *When a conversation gets too long, the lobster brain does what lobsters do best: it panics, truncates everything before message 47, and carries on like nothing happened. Your carefully explained architecture? Gone. That bug you described in detail twenty minutes ago? Never heard of it.*
+>
+> *KongBrain is a brain transplant. You're replacing that crustacean context window with a primate cortex — backed by a graph database, vector embeddings, and the kind of persistent memory that lets your AI remember what you said last Tuesday — and judge you for it.*
+
+Persistent memory graph. Vector-embedded, self-scoring, wired to learn across sessions. It extracts skills from what worked, traces causal chains through what broke, reflects on its own failures, and earns an identity through real experience. Every session compounds on the last.
+
+Your assistant stops forgetting. Then it starts getting smarter.
+
+[Quick Start](#quick-start) | [Architecture](#architecture) | [How It Works](#how-it-works) | [Tools](#tools) | [Development](#development)
+
+---
+
+## What Changes
+
+| | Lobster Brain (default) | Ape Brain (KongBrain) |
+|---|---|---|
+| **Memory** | Sliding window. Old messages fall off a cliff. | Graph-persistent. Every turn, concept, skill, and causal chain stored with vector embeddings. |
+| **Recall** | Whatever fits in the context window right now. | Cosine similarity + graph expansion + learned attention scoring across your entire history. |
+| **Adaptation** | Same retrieval budget every turn, regardless of intent. | 10 intent categories. Simple question? Minimal retrieval. Complex debugging? Full graph search + elevated thinking. |
+| **Learning** | None. Every session starts from zero. | Skills extracted from successful workflows, causal chains graduated into reusable procedures, corrections remembered permanently. |
+| **Self-awareness** | Thermostat-level. | Periodic cognitive checks grade its own retrieval quality, detect contradictions, suppress noise, and extract your preferences. Eventually graduates a soul document. |
+| **Compaction** | LLM-summarizes your conversation mid-flow (disruptive). | Graph retrieval IS the compaction. No interruptions, no lossy summaries. |
+
+## Quick Start
+
+From zero to ape brain in under 5 minutes.
+
+### 1. Install OpenClaw (if you haven't already)
+
+```bash
+npm install -g openclaw
+```
+
+### 2. Start SurrealDB
+
+Install SurrealDB via your platform's package manager (see [surrealdb.com/install](https://surrealdb.com/docs/surrealdb/installation)):
+
+```bash
+# macOS
+brew install surrealdb/tap/surreal
+
+# Linux (Debian/Ubuntu)
+curl -sSf https://install.surrealdb.com | sh
+export PATH="$HOME/.surrealdb:$PATH"
+```
+
+Then start it locally — **change the credentials before use**:
+
+```bash
+surreal start --user youruser --pass yourpass --bind 127.0.0.1:8042 surrealkv:~/.kongbrain/surreal.db
+```
+
+Or with Docker:
+
+```bash
+docker run -d --name surrealdb -p 127.0.0.1:8042:8000 \
+  -v ~/.kongbrain/surreal-data:/data \
+  surrealdb/surrealdb:latest start \
+  --user youruser --pass yourpass surrealkv:/data/surreal.db
+```
+
+> **Security note:** Always bind to `127.0.0.1` (not `0.0.0.0`) unless you need remote access. Never use default credentials in production.
+
+### 3. Install KongBrain
+
+```bash
+# From ClawHub (recommended)
+openclaw plugins install clawhub:kongbrain
+
+# From npm (fallback)
+openclaw plugins install kongbrain
+```
+
+> **Note:** Bare `openclaw plugins install kongbrain` checks ClawHub first, then falls back to npm. Use the `clawhub:` prefix to install from ClawHub explicitly.
+
+### 4. Activate
+
+Add to your OpenClaw config (`~/.openclaw/openclaw.json`):
+
+```json
+{
+  "plugins": {
+    "allow": ["kongbrain"],
+    "slots": {
+      "contextEngine": "kongbrain"
+    }
+  }
+}
+```
+
+### 5. Talk to your ape
+
+```bash
+openclaw tui
+```
+
+That's it. KongBrain uses whatever LLM provider and model you already have configured in OpenClaw (Anthropic, OpenAI, Google, Ollama, whatever). No separate API keys needed for the brain itself.
+
+The BGE-M3 embedding model (~420MB) downloads automatically on first startup from [Hugging Face](https://huggingface.co/BAAI/bge-m3). All database tables and indexes are created automatically on first run. No manual setup required.
+
+<details>
+<summary><strong>Configuration Options</strong></summary>
+
+All options have sensible defaults. Override via plugin config or environment variables:
+
+| Option | Env Var | Default |
+|--------|---------|---------|
+| `surreal.url` | `SURREAL_URL` | `ws://127.0.0.1:8042/rpc` |
+| `surreal.user` | `SURREAL_USER` | (required) |
+| `surreal.pass` | `SURREAL_PASS` | (required) |
+| `surreal.ns` | `SURREAL_NS` | `kong` |
+| `surreal.db` | `SURREAL_DB` | `memory` |
+| `embedding.modelPath` | `KONGBRAIN_EMBEDDING_MODEL` | Auto-downloaded BGE-M3 Q4_K_M |
+| `embedding.dimensions` | - | `1024` |
+
+Full config example:
+
+```json
+{
+  "plugins": {
+    "allow": ["kongbrain"],
+    "slots": {
+      "contextEngine": "kongbrain"
+    },
+    "entries": {
+      "kongbrain": {
+        "config": {
+          "surreal": {
+            "url": "ws://127.0.0.1:8042/rpc",
+            "user": "youruser",
+            "pass": "yourpass",
+            "ns": "kong",
+            "db": "memory"
+          }
+        }
+      }
+    }
+  }
+}
+```
+
+</details>
+
+---
+
+## Architecture
+
+### The IKONG Pillars
+
+KongBrain's cognitive architecture follows five functional pillars:
+
+| Pillar | Role | What it does |
+|--------|------|-------------|
+| **I**ntelligence | Adaptive reasoning | Intent classification, complexity estimation, thinking depth, orchestrator preflight |
+| **K**nowledge | Persistent memory | Memory graph, concepts, skills, reflections, identity chunks, core memory tiers |
+| **O**peration | Execution | Tool orchestration, skill procedures, causal chain tracking, artifact management |
+| **N**etwork | Graph traversal | Cross-pillar edge following, neighbor expansion, causal path walking |
+| **G**raph | Persistence | SurrealDB storage, BGE-M3 vector search, HNSW indexes, embedding pipeline |
+
+A 6th pillar, **Persona**, is unlocked at soul graduation: *"You have a Soul, an identity grounded in real experience. Be unique, be genuine, be yourself."*
+
+### Structural Pillars
+
+The graph entity model in SurrealDB:
+
+| Pillar | Table | What it anchors |
+|--------|-------|-----------------|
+| 1. Agent | `agent` | Who is operating (name, model) |
+| 2. Project | `project` | What we're working on (status, tags) |
+| 3. Task | `task` | Individual sessions as units of work |
+| 4. Artifact | `artifact` | Files and outputs tracked across sessions |
+| 5. Concept | `concept` | Semantic knowledge nodes extracted from sessions |
+
+On startup, the agent bootstraps the full chain: `Agent → owns → Project`, `Agent → performed → Task`, `Task → task_part_of → Project`, `Session → session_task → Task`. Graph expansion traverses these edges during retrieval.
+
+### The Knowledge Graph
+
+SurrealDB with HNSW vector indexes (1024-dim cosine). Everything is embedded and queryable.
+
+| Table | What it stores |
+|-------|---------------|
+| `turn` | Every conversation message (role, text, embedding, token count, model, usage) |
+| `memory` | Compacted episodic knowledge (importance 0-10, confidence, access tracking) |
+| `skill` | Learned procedures with steps, preconditions, success/failure counts |
+| `reflection` | Metacognitive lessons (efficiency, failure patterns, approach strategy) |
+| `causal_chain` | Cause-effect patterns (trigger, outcome, chain type, success, confidence) |
+| `identity_chunk` | Agent self-knowledge fragments (source, importance, embedding) |
+| `monologue` | Thinking traces preserved across sessions |
+| `core_memory` | Tier 0 (always loaded) + Tier 1 (session-pinned) directives |
+| `soul` | Emergent identity document, earned through graduation |
+
+<details>
+<summary><strong>Adaptive Reasoning</strong>: per-turn intent classification and budget allocation</summary>
+
+Every turn gets classified by intent and assigned an adaptive config:
+
+| Intent | Thinking | Tool Limit | Token Budget | Retrieval Share |
+|--------|----------|------------|--------------|-----------------|
+| `simple-question` | low | 3 | 4K | 10% |
+| `code-read` | medium | 5 | 6K | 15% |
+| `code-write` | high | 8 | 8K | 20% |
+| `code-debug` | high | 10 | 8K | 20% |
+| `deep-explore` | medium | 15 | 6K | 15% |
+| `reference-prior` | medium | 5 | 10K | 25% |
+| `meta-session` | low | 2 | 3K | 7% (skip retrieval) |
+| `multi-step` | high | 12 | 8K | 20% |
+| `continuation` | low | 8 | 4K | skip retrieval |
+
+**Fast path:** Short inputs (<20 chars, no `?`) skip classification entirely.
+**Confidence gate:** Below 0.40 confidence, falls back to conservative config.
+
+</details>
+
+<details>
+<summary><strong>Context Injection Pipeline</strong></summary>
+
+1. **Embed** user input via BGE-M3 (or hit prefetch cache at 0.85 cosine threshold)
+2. **Vector search** across 6 tables (turn, identity_chunk, concept, memory, artifact, monologue)
+3. **Graph expand**: fetch neighbors via structural + semantic edges, compute cosine similarity
+4. **Score** all candidates with WMR (Working Memory Ranker):
+   ```
+   score = W * [similarity, recency, importance, access, neighbor_bonus, utility, reflection_boost]
+   ```
+5. **Budget trim**: inject Tier 0/1 core memory first (15% of context), then ranked results up to 21% retrieval budget
+6. **Stage** retrieval snapshot for post-hoc quality evaluation
+
+</details>
+
+<details>
+<summary><strong>ACAN</strong>: learned cross-attention scorer</summary>
+
+A ~130K-parameter cross-attention network that replaces the fixed WMR weights once enough data accumulates.
+
+- **Activation:** 5,000+ labeled retrieval outcomes
+- **Training:** Pure TypeScript SGD with manual backprop, 80 epochs
+- **Staleness:** Retrains when data grows 50%+ or weights age > 7 days
+
+</details>
+
+<details>
+<summary><strong>Soul & Graduation</strong>: earned identity, not assigned</summary>
+
+The agent earns an identity document through accumulated experience. Graduation requires **all 7 thresholds met** AND a **quality score >= 0.6**:
+
+| Signal | Threshold |
+|--------|-----------|
+| Sessions completed | 15 |
+| Reflections stored | 10 |
+| Causal chains traced | 5 |
+| Concepts extracted | 30 |
+| Memory compactions | 5 |
+| Monologue traces | 5 |
+| Time span | 3 days |
+
+**Quality scoring** from 4 real performance signals: retrieval utilization (30%), skill success rate (25%), critical reflection rate (25%), tool failure rate (20%).
+
+**Maturity stages:** nascent (0-3/7) → developing (4/7) → emerging (5/7) → maturing (6/7) → ready (7/7 + quality gate). The agent and user are notified at each stage transition.
+
+**Soul evolution:** Every 10 sessions after graduation, the soul is re-evaluated against new experience and revised if the agent has meaningfully changed.
+
+**Soul document structure:** Working style, self-observations, earned values (grounded in specific evidence), revision history. Seeded as Tier 0 core memory, loaded every single turn.
+
+</details>
+
+<details>
+<summary><strong>Reflection System</strong>: metacognitive self-correction</summary>
+
+Triggers at session end when metrics indicate problems:
+
+| Condition | Threshold |
+|-----------|-----------|
+| Retrieval utilization | < 20% average |
+| Tool failure rate | > 20% |
+| Steering candidates | any detected |
+| Context waste | > 0.5% of context window |
+
+The LLM generates a 2-4 sentence reflection: root cause, error pattern, what to do differently. Stored with importance 7.0, deduped at 0.85 cosine similarity.
+
+</details>
+
+---
+
+## How It Works
+
+### Every Turn
+
+```
+User Input
+    |
+    v
+Preflight ──────── Intent classification (25ms, zero-shot BGE-M3 cosine)
+    |                  10 categories: simple-question, code-read, code-write,
+    |                  code-debug, deep-explore, reference-prior, meta-session,
+    |                  multi-step, continuation, unknown
+    v
+Prefetch ────────── Predictive background vector searches (LRU cache, 5-min TTL)
+    |
+    v
+Context Injection ─ Vector search -> graph expand -> 6-signal scoring -> budget trim
+    |                  Searches: turns, concepts, memories, artifacts, identity, monologues
+    |                  Scores: similarity, recency, importance, access, neighbor, utility
+    |                  Budget: 21% of context window reserved for retrieval
+    v
+Agent Loop ──────── LLM + tool execution
+    |                  Planning gate: announces plan before touching tools
+    |                  Smart truncation: preserves tail of large tool outputs
+    v
+Turn Storage ────── Every message embedded + stored + linked via graph edges
+    |                  responds_to, part_of, mentions, produced
+    v
+Quality Eval ────── Measures retrieval utilization (text overlap, trigrams, unigrams)
+    |                  Tracks tool success, context waste, feeds ACAN training
+    v
+Memory Daemon ───── Worker thread extracts 9 knowledge types via LLM:
+    |                  causal chains, monologues, concepts, corrections,
+    |                  preferences, artifacts, decisions, skills, resolved memories
+    v
+Postflight ──────── Records orchestrator metrics (non-blocking)
+```
+
+### Between Sessions
+
+At session end, KongBrain runs a combined extraction pass: skill graduation, metacognitive reflection, causal chain consolidation, soul graduation check, and soul evolution. A handoff note is written so the next session wakes up knowing what happened.
+
+At session start, a wake-up briefing is synthesized from the handoff, recent monologues, soul content (if graduated), and identity state, then injected as inner speech so the agent knows who it is and what it was doing.
+
+<details>
+<summary><strong>Memory Daemon</strong>: background knowledge extraction</summary>
+
+A worker thread running throughout the session. Batches turns every ~12K tokens, calls the configured LLM to extract:
+
+- **Causal chains**: trigger/outcome sequences with success/confidence
+- **Monologue traces**: thinking blocks that reveal problem-solving approach
+- **Concepts**: semantic nodes (architecture patterns, domain terms)
+- **Corrections**: user-provided fixes (importance: 9)
+- **Preferences**: behavioral rules learned from feedback
+- **Artifacts**: file paths created or modified
+- **Decisions**: important conclusions reached
+- **Skills**: multi-step procedures (if 5+ tool calls in session)
+- **Resolved memories**: completed tasks and confirmed facts
+
+</details>
+
+---
+
+## Tools
+
+Three tools are registered for the LLM:
+
+- **`recall`**: Search graph memory by query
+- **`core_memory`**: Read/write persistent core directives (tiered: always-loaded vs session-pinned)
+- **`introspect`**: Inspect database state, verify memory counts, run diagnostics, check graduation status, migrate workspace files
+
+---
+
+## Development
+
+```bash
+git clone https://github.com/42U/kongbrain.git
+cd kongbrain
+pnpm install
+pnpm build
+pnpm test
+```
+
+Link your local build to OpenClaw:
+
+```bash
+openclaw plugins install . --link
+```
+
+Then set `plugins.slots.contextEngine` to `"kongbrain"` in `~/.openclaw/openclaw.json` and run `openclaw`.
+
+## Contributing
+
+1. Clone the repo and install dependencies (`pnpm install`)
+2. Make your changes
+3. Build (`pnpm build`) and run tests (`pnpm test`)
+4. Open a PR against `master`
+
+The lobster doesn't accept contributions. The ape does.
+
+---
+
+MIT License | Built by [42U](https://github.com/42U)

package/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: kongbrain
 description: Graph-backed persistent memory engine for OpenClaw. Replaces the default context window with SurrealDB + vector embeddings that learn across sessions.
-version: 0.3.8
+version: 0.3.10
 homepage: https://github.com/42U/kongbrain
 metadata:
   openclaw:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "kongbrain",
-  "version": "0.3.8",
+  "version": "0.3.10",
   "description": "Graph-backed persistent memory engine for OpenClaw. Replaces the default context window with SurrealDB + vector embeddings that learn across sessions.",
   "type": "module",
   "license": "MIT",
@@ -41,6 +41,8 @@
     }
   },
   "scripts": {
+    "prepack": "cp README.md README.github.md && cp README.npm.md README.md",
+    "postpack": "cp README.github.md README.md && rm README.github.md",
     "test": "vitest run",
     "test:watch": "vitest",
     "build": "tsc --noEmit",

package/src/concept-extract.ts

@@ -0,0 +1,155 @@
+/**
+ * Shared concept-extraction helpers.
+ *
+ * Regex-based extraction of concept names from text, plus helpers to
+ * upsert extracted concepts and link them via arbitrary edge types.
+ */
+
+import type { SurrealStore } from "./surreal.js";
+import type { EmbeddingService } from "./embeddings.js";
+import { swallow } from "./errors.js";
+
+// Same regexes used by the original extractAndLinkConcepts in context-engine.
+export const CONCEPT_RE = /\b(?:(?:use|using|implement|create|add|configure|setup|install|import)\s+)([A-Z][a-zA-Z0-9_-]+(?:\s+[A-Z][a-zA-Z0-9_-]+)?)/g;
+export const TECH_TERMS = /\b(api|database|schema|migration|endpoint|middleware|component|service|module|handler|controller|model|interface|type|class|function|method|hook|plugin|extension|config|cache|queue|worker|daemon)\b/gi;
+
+/** Extract concept name strings from free text using regex heuristics. */
+export function extractConceptNames(text: string): string[] {
+  const concepts = new Set<string>();
+
+  let match: RegExpExecArray | null;
+  const re1 = new RegExp(CONCEPT_RE.source, CONCEPT_RE.flags);
+  while ((match = re1.exec(text)) !== null) {
+    concepts.add(match[1].trim());
+  }
+
+  const re2 = new RegExp(TECH_TERMS.source, TECH_TERMS.flags);
+  while ((match = re2.exec(text)) !== null) {
+    concepts.add(match[1].toLowerCase());
+  }
+
+  return [...concepts].slice(0, 10);
+}
+
+/**
+ * Upsert concepts from text and link them to a source node via the given edge.
+ *
+ * Used for:
+ *  - turn  → "mentions"          → concept  (existing behaviour)
+ *  - memory → "about_concept"    → concept  (Fix 1)
+ *  - artifact → "artifact_mentions" → concept (Fix 2)
+ */
+export async function upsertAndLinkConcepts(
+  sourceId: string,
+  edgeName: string,
+  text: string,
+  store: SurrealStore,
+  embeddings: EmbeddingService,
+  logTag: string,
+  opts?: { taskId?: string; projectId?: string },
+): Promise<void> {
+  const names = extractConceptNames(text);
+  if (names.length === 0) return;
+
+  for (const name of names) {
+    try {
+      let embedding: number[] | null = null;
+      if (embeddings.isAvailable()) {
+        try { embedding = await embeddings.embed(name); } catch { /* ok */ }
+      }
+      const conceptId = await store.upsertConcept(name, embedding);
+      if (conceptId) {
+        await store.relate(sourceId, edgeName, conceptId)
+          .catch(e => swallow(`${logTag}:relate`, e));
+
+        // derived_from: concept → task
+        if (opts?.taskId) {
+          await store.relate(conceptId, "derived_from", opts.taskId)
+            .catch(e => swallow(`${logTag}:derived_from`, e));
+        }
+        // relevant_to: concept → project
+        if (opts?.projectId) {
+          await store.relate(conceptId, "relevant_to", opts.projectId)
+            .catch(e => swallow(`${logTag}:relevant_to`, e));
+        }
+      }
+    } catch (e) {
+      swallow(`${logTag}:upsert`, e);
+    }
+  }
+}
+
+/**
+ * Link a newly-upserted concept to existing concepts via narrower/broader
+ * edges when one concept's name is a substring of the other (indicating a
+ * parent-child hierarchy, e.g. "React" → "React hooks").
+ */
+export async function linkConceptHierarchy(
+  conceptId: string,
+  conceptName: string,
+  store: SurrealStore,
+  embeddings: EmbeddingService,
+  logTag: string,
+): Promise<void> {
+  try {
+    const existing = await store.queryFirst<{ id: string; content: string }>(
+      `SELECT id, content FROM concept WHERE id != $cid LIMIT 50`,
+      { cid: conceptId },
+    );
+    if (existing.length === 0) return;
+
+    const lowerName = conceptName.toLowerCase();
+    let relatedCount = 0;
+
+    for (const other of existing) {
+      const otherLower = (other.content ?? "").toLowerCase();
+      if (!otherLower || otherLower === lowerName) continue;
+
+      const otherId = String(other.id);
+
+      if (lowerName.includes(otherLower) && lowerName !== otherLower) {
+        // New concept is more specific (e.g. "React hooks" contains "React")
+        await store.relate(conceptId, "narrower", otherId)
+          .catch(e => swallow(`${logTag}:narrower`, e));
+        await store.relate(otherId, "broader", conceptId)
+          .catch(e => swallow(`${logTag}:broader`, e));
+      } else if (otherLower.includes(lowerName) && otherLower !== lowerName) {
+        // New concept is more general (e.g. "React" contained in "React hooks")
+        await store.relate(conceptId, "broader", otherId)
+          .catch(e => swallow(`${logTag}:broader`, e));
+        await store.relate(otherId, "narrower", conceptId)
+          .catch(e => swallow(`${logTag}:narrower`, e));
+      }
+    }
+
+    // related_to: peer-level semantic association via embedding similarity
+    if (embeddings.isAvailable()) {
+      try {
+        const conceptEmb = await embeddings.embed(conceptName);
+        if (conceptEmb?.length) {
+          const similar = await store.queryFirst<{ id: string; score: number }>(
+            `SELECT id, vector::similarity::cosine(embedding, $vec) AS score
+             FROM concept
+             WHERE id != $cid
+               AND embedding != NONE AND array::len(embedding) > 0
+             ORDER BY score DESC
+             LIMIT 3`,
+            { vec: conceptEmb, cid: conceptId },
+          );
+          for (const s of similar) {
+            if (s.score < 0.75) break;
+            const simId = String(s.id);
+            await store.relate(conceptId, "related_to", simId)
+              .catch(e => swallow(`${logTag}:related_to`, e));
+            await store.relate(simId, "related_to", conceptId)
+              .catch(e => swallow(`${logTag}:related_to`, e));
+          }
+        }
+      } catch (e) {
+        swallow(`${logTag}:related_to_search`, e);
+      }
+    }
+  } catch (e) {
+    swallow(`${logTag}:hierarchy`, e);
+  }
+}