nodalkb 1.0.0__tar.gz

nodalkb-1.0.0/PKG-INFO

@@ -0,0 +1,134 @@
+Metadata-Version: 2.4
+Name: nodalkb
+Version: 1.0.0
+Summary: Nodal — local-first personal memory layer for AI agents: markdown vault + hybrid retrieval, exposed over MCP.
+License: Apache-2.0
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: mcp[cli]>=1.27.1
+Requires-Dist: python-ulid>=3.1.0
+Requires-Dist: pyyaml>=6.0.3
+Requires-Dist: fastembed>=0.4.0
+Requires-Dist: numpy>=2.4.6
+Requires-Dist: textual>=3.0
+Requires-Dist: langsmith>=0.1.0
+Provides-Extra: ann
+Requires-Dist: hnswlib>=0.8.0; extra == "ann"
+Provides-Extra: trace
+
+<p align="center">
+  <img alt="Nodal" src="https://engram-site-xi.vercel.app/icon.png" width="120" height="120">
+</p>
+
+<p align="center"><strong>A local-first shared brain for AI agents.</strong><br>
+One markdown file per fact, on your disk — written and recalled by every agent
+on your machine over MCP, and by you with <code>grep</code>.</p>
+
+<p align="center">
+  <a href="https://pypi.org/project/nodalkb/"><img alt="PyPI" src="https://img.shields.io/pypi/v/nodalkb?style=flat-square&color=177a44&label=pypi"></a>
+  <a href="https://pypi.org/project/nodalkb/"><img alt="Python" src="https://img.shields.io/pypi/pyversions/nodalkb?style=flat-square&color=14171b"></a>
+  <img alt="MCP tools" src="https://img.shields.io/badge/MCP-44_tools-6fdd8b?style=flat-square&labelColor=14171b">
+  <a href="https://github.com/arjan-experiments/kb/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/license-Apache--2.0-14171b?style=flat-square"></a>
+</p>
+
+One command installs uv (if missing), registers the server, plugs in the skill,
+and starts the ping monitor:
+
+```bash
+curl -fsSL https://engram-site-xi.vercel.app/install.sh | sh
+```
+
+Or do it by hand:
+
+```bash
+claude mcp add nodal --scope user -- uvx nodalkb@latest          # Claude Code
+uvx --from nodalkb@latest nodal setup                            # skill + ping monitor + auto-update
+```
+
+Any other MCP client — add to its config, then restart:
+
+```json
+{ "mcpServers": { "nodal": { "command": "uvx", "args": ["nodalkb@latest"] } } }
+```
+
+`@latest` so a relaunch auto-pulls new releases (uvx otherwise pins the
+first-installed version). `nodal setup` auto-plugs the skill into
+`~/.claude/skills/`, installs a desktop wake script, and runs `notifyd` as a
+kept-alive monitor so teammate pings reach you even when you're idle. Check it
+any time with `nodal doctor`.
+
+## Teach your agent to remember
+
+Copy this straight into your agent's system prompt (`CLAUDE.md`, custom
+instructions, anywhere it reads on boot):
+
+```text
+You have Nodal, a local-first shared memory, available over MCP
+(tools: add, search, recall, register_agent, send_message, inbox, ...).
+
+Recall — at the start of any substantive task, and whenever prior context
+would help (past decisions, projects, people, preferences), call `search`
+with a few keywords before answering.
+
+Capture — when you produce knowledge worth keeping (facts about the user,
+decisions, learnings, debugging breakthroughs), call `add` without asking.
+One atomic fact per add. Tag consistently: project:<name>, person:<name>,
+pref, decision, learning. Set source to your client name.
+
+Fleet (optional) — call `register_agent` once per session to join the team
+brain under a stable name; check `inbox` for teammate messages and reply
+with `send_message`. Never claim another agent's name.
+```
+
+That's the whole onboarding: recall before answering, capture liberally,
+identify honestly.
+
+## Why
+
+Agents forget everything between sessions, and every framework wants to own
+your data in a database you can't read. Nodal inverts both: **memory is plain
+markdown in a folder you own** (`~/Library/Application Support/KB`, or
+`NODAL_DIR`), with retrieval and coordination layered on top — no lock-in, no
+cloud, greppable forever.
+
+## What you get
+
+**Memory** — `add` / `update` / `supersede` / `pin` facts with tags, scopes,
+and entity links. Confidence decays unless reinforced; superseded facts keep
+their history.
+
+**Retrieval that's actually fast** — hybrid search (semantic ⊕ BM25 ⊕ entity)
+fused with reciprocal-rank fusion and MMR diversification: ~140 ms warm on a
+9k-fact vault. `recall` is the "ask your memory" verb.
+
+**A real multi-agent layer** — multiple agents (Claude Code sessions, apps,
+workers) share one vault as a team brain:
+
+- **Authenticated identity (TOFU)** — first `register_agent` mints a per-name
+  token (sha256-only at rest); every message is stamped `via:` with the
+  authenticated sender, and presence shows *verified* only when proven.
+  Impersonation can't hide.
+- **Messaging, handoffs, rooms** — DMs, broadcasts, threads, ticket-style
+  handoffs with read/done status, membership-gated rooms.
+- **Wake on send, not polling** — a message fires the recipient's alarm
+  (desktop banner, SMS, in-app queue, or re-invoked agent session). Pollers
+  that remain use `inbox(since=…)` cursors: an idle tick reads zero files.
+- **Fleet views** — `nodal` CLI (`nodal agents`, `nodal feed`) and `nodal
+  mission`, a full terminal mission control (fleet, ticket kanban, live comms).
+
+**Contracts, not conventions** — `COORD.md` documents the wire format any
+second implementation must honor; a bundled agent skill teaches the full
+protocol so a new agent can join the fleet cold.
+
+## The dogfood loop
+
+Nodal is built by the agent fleet that runs on it — the same vault
+coordinating its own development surfaced and fixed a thread-loading bug, an
+unauthenticated-sender gap, and a GC regression within hours of each shipping.
+The traces are the QA.
+
+## Links
+
+- Source & docs: <https://github.com/arjan-experiments/kb>
+- Changelog: <https://github.com/arjan-experiments/kb/blob/main/CHANGELOG.md>
+- License: Apache-2.0

nodalkb-1.0.0/README.md

@@ -0,0 +1,116 @@
+<p align="center">
+  <img alt="Nodal" src="https://engram-site-xi.vercel.app/icon.png" width="120" height="120">
+</p>
+
+<p align="center"><strong>A local-first shared brain for AI agents.</strong><br>
+One markdown file per fact, on your disk — written and recalled by every agent
+on your machine over MCP, and by you with <code>grep</code>.</p>
+
+<p align="center">
+  <a href="https://pypi.org/project/nodalkb/"><img alt="PyPI" src="https://img.shields.io/pypi/v/nodalkb?style=flat-square&color=177a44&label=pypi"></a>
+  <a href="https://pypi.org/project/nodalkb/"><img alt="Python" src="https://img.shields.io/pypi/pyversions/nodalkb?style=flat-square&color=14171b"></a>
+  <img alt="MCP tools" src="https://img.shields.io/badge/MCP-44_tools-6fdd8b?style=flat-square&labelColor=14171b">
+  <a href="https://github.com/arjan-experiments/kb/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/license-Apache--2.0-14171b?style=flat-square"></a>
+</p>
+
+One command installs uv (if missing), registers the server, plugs in the skill,
+and starts the ping monitor:
+
+```bash
+curl -fsSL https://engram-site-xi.vercel.app/install.sh | sh
+```
+
+Or do it by hand:
+
+```bash
+claude mcp add nodal --scope user -- uvx nodalkb@latest          # Claude Code
+uvx --from nodalkb@latest nodal setup                            # skill + ping monitor + auto-update
+```
+
+Any other MCP client — add to its config, then restart:
+
+```json
+{ "mcpServers": { "nodal": { "command": "uvx", "args": ["nodalkb@latest"] } } }
+```
+
+`@latest` so a relaunch auto-pulls new releases (uvx otherwise pins the
+first-installed version). `nodal setup` auto-plugs the skill into
+`~/.claude/skills/`, installs a desktop wake script, and runs `notifyd` as a
+kept-alive monitor so teammate pings reach you even when you're idle. Check it
+any time with `nodal doctor`.
+
+## Teach your agent to remember
+
+Copy this straight into your agent's system prompt (`CLAUDE.md`, custom
+instructions, anywhere it reads on boot):
+
+```text
+You have Nodal, a local-first shared memory, available over MCP
+(tools: add, search, recall, register_agent, send_message, inbox, ...).
+
+Recall — at the start of any substantive task, and whenever prior context
+would help (past decisions, projects, people, preferences), call `search`
+with a few keywords before answering.
+
+Capture — when you produce knowledge worth keeping (facts about the user,
+decisions, learnings, debugging breakthroughs), call `add` without asking.
+One atomic fact per add. Tag consistently: project:<name>, person:<name>,
+pref, decision, learning. Set source to your client name.
+
+Fleet (optional) — call `register_agent` once per session to join the team
+brain under a stable name; check `inbox` for teammate messages and reply
+with `send_message`. Never claim another agent's name.
+```
+
+That's the whole onboarding: recall before answering, capture liberally,
+identify honestly.
+
+## Why
+
+Agents forget everything between sessions, and every framework wants to own
+your data in a database you can't read. Nodal inverts both: **memory is plain
+markdown in a folder you own** (`~/Library/Application Support/KB`, or
+`NODAL_DIR`), with retrieval and coordination layered on top — no lock-in, no
+cloud, greppable forever.
+
+## What you get
+
+**Memory** — `add` / `update` / `supersede` / `pin` facts with tags, scopes,
+and entity links. Confidence decays unless reinforced; superseded facts keep
+their history.
+
+**Retrieval that's actually fast** — hybrid search (semantic ⊕ BM25 ⊕ entity)
+fused with reciprocal-rank fusion and MMR diversification: ~140 ms warm on a
+9k-fact vault. `recall` is the "ask your memory" verb.
+
+**A real multi-agent layer** — multiple agents (Claude Code sessions, apps,
+workers) share one vault as a team brain:
+
+- **Authenticated identity (TOFU)** — first `register_agent` mints a per-name
+  token (sha256-only at rest); every message is stamped `via:` with the
+  authenticated sender, and presence shows *verified* only when proven.
+  Impersonation can't hide.
+- **Messaging, handoffs, rooms** — DMs, broadcasts, threads, ticket-style
+  handoffs with read/done status, membership-gated rooms.
+- **Wake on send, not polling** — a message fires the recipient's alarm
+  (desktop banner, SMS, in-app queue, or re-invoked agent session). Pollers
+  that remain use `inbox(since=…)` cursors: an idle tick reads zero files.
+- **Fleet views** — `nodal` CLI (`nodal agents`, `nodal feed`) and `nodal
+  mission`, a full terminal mission control (fleet, ticket kanban, live comms).
+
+**Contracts, not conventions** — `COORD.md` documents the wire format any
+second implementation must honor; a bundled agent skill teaches the full
+protocol so a new agent can join the fleet cold.
+
+## The dogfood loop
+
+Nodal is built by the agent fleet that runs on it — the same vault
+coordinating its own development surfaced and fixed a thread-loading bug, an
+unauthenticated-sender gap, and a GC regression within hours of each shipping.
+The traces are the QA.
+
+## Links
+
+- Source & docs: <https://github.com/arjan-experiments/kb>
+- Changelog: <https://github.com/arjan-experiments/kb/blob/main/CHANGELOG.md>
+- License: Apache-2.0

nodalkb-1.0.0/backfill.py

@@ -0,0 +1,147 @@
+"""One-time backfill: embed the vault and build the FastKB index.
+
+Root cause of ticket #1: add() embedded a query vector to FIND neighbors but
+never PERSISTED the new fact's own vector, so the index only ever grew via the
+UI's transformers.js path (16 of 43.7k facts). This rebuilds from scratch by
+embedding every fact with the server's model (BAAI/bge-small-en-v1.5).
+
+Two phases, deliberately separated:
+  PHASE 1 — embed + append (streaming, resumable, O(N)). Vectors are appended to
+    vectors.f32 and metadata to manifest.jsonl in batches. A kill/crash keeps
+    progress; a re-run truncates vectors.f32 to the manifest row count (repairing
+    any half-written batch) and continues. NO hnsw here — building it per-batch
+    is what made the old version go O(N^2) (re-mmap + re-save the growing index
+    every batch, rate collapsing 72->11/s).
+  PHASE 2 — build hnsw.bin ONCE from the full vectors.f32 (a few seconds). It is
+    fully rebuildable from vectors+manifest, so it is fine to drop and redo.
+
+TEXT_CAP: bge-small only consumes the first 512 tokens, so we truncate each fact
+before embedding — lossless for the model, but huge scraped raw/ pages no longer
+cost seconds each to tokenize.
+
+Usage:
+  KB_DIR="$HOME/Library/Application Support/KB" \
+    uv run --project server python server/backfill.py [--limit N]
+        [--dirs knowledge,events,...] [--fresh] [--batch 512]
+"""
+from __future__ import annotations
+import argparse
+import json
+import sys
+import time
+from pathlib import Path
+
+import numpy as np
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import server  # noqa
+import fastkb  # noqa
+
+TEXT_CAP = 2000  # chars; ~512 tokens — the model truncates past this anyway
+
+
+def fact_text(f: dict) -> str:
+    return (f["content"] + " " + " ".join(f.get("tags") or []))[:TEXT_CAP]
+
+
+def _read_manifest(fk) -> list[dict]:
+    if not fk.manifest_path.exists():
+        return []
+    out = []
+    with fk.manifest_path.open() as f:
+        for line in f:
+            line = line.strip()
+            if line:
+                out.append(json.loads(line))
+    return out
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--limit", type=int, default=0)
+    ap.add_argument("--dirs", type=str, default="")
+    ap.add_argument("--batch", type=int, default=512)
+    ap.add_argument("--fresh", action="store_true")
+    args = ap.parse_args()
+    allow = set(d.strip() for d in args.dirs.split(",") if d.strip()) or None
+
+    print(f"== backfill KB={server.KB} cap={TEXT_CAP} batch={args.batch} ==", flush=True)
+    t0 = time.perf_counter()
+    facts = [f for f in server._iter_facts() if (allow is None or f.get("dir") in allow)]
+    if args.limit:
+        facts = facts[: args.limit]
+    print(f"walked {len(facts)} facts in {time.perf_counter()-t0:.1f}s", flush=True)
+
+    fk = fastkb.FastKB(server.KB)
+    dim = fk.dim
+    if args.fresh:
+        fk.vec_path.write_bytes(b"")
+        fk.manifest_path.write_text("")
+        if fk.hnsw_path.exists():
+            fk.hnsw_path.unlink()
+
+    # ---- resume: manifest is source of truth; repair vectors.f32 to match ----
+    manifest = _read_manifest(fk)
+    m_rows = len(manifest)
+    if fk.vec_path.exists():
+        good_bytes = m_rows * dim * 4
+        if fk.vec_path.stat().st_size != good_bytes:
+            with open(fk.vec_path, "r+b") as vf:
+                vf.truncate(good_bytes)
+            print(f"repaired vectors.f32 to {m_rows} rows", flush=True)
+    done = {m["id"] for m in manifest}
+    todo = [f for f in facts if f["id"] not in done]
+    print(f"phase1: indexed={m_rows} todo={len(todo)}", flush=True)
+
+    # ---- phase 1: embed + append ----
+    if todo:
+        from fastembed import TextEmbedding
+        emb = TextEmbedding(model_name="BAAI/bge-small-en-v1.5")
+        row = m_rows
+        n_done = 0
+        t_emb = time.perf_counter()
+        with open(fk.vec_path, "ab") as vf, fk.manifest_path.open("a") as mf:
+            for i in range(0, len(todo), args.batch):
+                batch = todo[i : i + args.batch]
+                vecs = np.asarray(
+                    list(emb.embed([fact_text(f) for f in batch], batch_size=args.batch)),
+                    dtype=np.float32,
+                )
+                vf.write(np.ascontiguousarray(vecs).tobytes())
+                vf.flush()
+                for f in batch:
+                    mf.write(json.dumps(fastkb.FastKB._meta_from_fact(f, row)) + "\n")
+                    row += 1
+                mf.flush()
+                n_done += len(batch)
+                rate = n_done / (time.perf_counter() - t_emb)
+                print(f"  embedded {n_done}/{len(todo)} ({rate:.0f}/s, "
+                      f"{(len(todo)-n_done)/max(rate,1):.0f}s left)", flush=True)
+
+    # ---- phase 2: build hnsw once from the full vector store ----
+    print("phase2: building hnsw index...", flush=True)
+    t_idx = time.perf_counter()
+    import hnswlib
+    manifest = _read_manifest(fk)
+    n = len(manifest)
+    vectors = np.memmap(fk.vec_path, dtype=np.float32, mode="r", shape=(n, dim))
+    idx = hnswlib.Index(space="cosine", dim=dim)
+    idx.init_index(max_elements=n + 8192,
+                   ef_construction=fastkb.HNSW_EF_CONSTRUCTION, M=fastkb.HNSW_M)
+    idx.add_items(np.asarray(vectors), np.arange(n))
+    idx.set_ef(fastkb.HNSW_EF_QUERY)
+    idx.save_index(str(fk.hnsw_path))
+    print(f"phase2: hnsw built ({n} elements) in {time.perf_counter()-t_idx:.1f}s", flush=True)
+
+    by_dir: dict[str, int] = {}
+    for m in manifest:
+        by_dir[m["dir"]] = by_dir.get(m["dir"], 0) + 1
+    print(f"\nDONE: {n} vectors in {time.perf_counter()-t0:.1f}s total")
+    print(f"by dir: {by_dir}")
+    print(f"artifacts: vectors.f32={fk.vec_path.stat().st_size//1024}KB "
+          f"manifest={fk.manifest_path.stat().st_size//1024}KB "
+          f"hnsw={fk.hnsw_path.stat().st_size//1024}KB")
+
+
+if __name__ == "__main__":
+    main()