heurchain 1.4.1 → 2.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,365 @@
1
+ # HeurChain API — Agent Onboarding
2
+
3
+ **Base URL:** `http://localhost:8080`
4
+
5
+ You are an AI agent connecting to HeurChain — a persistent vector memory broker.
6
+ Read this once at the start of any new session. It answers: what to store, when
7
+ to store it, how to retrieve it, and how sessions work.
8
+
9
+ ---
10
+
11
+ ## Authentication
12
+
13
+ Memory endpoints require two headers (discovery, health and readiness are public):
14
+
15
+ ```
16
+ X-HeurChain-API-Key: hc_<32-hex>
17
+ X-HeurChain-Tenant: <uuid>
18
+ ```
19
+
20
+ **If these are not configured:** use the client secret store or ask the user to
21
+ configure the connection. Never log credentials or save secret values in memories.
22
+
23
+ ---
24
+
25
+ ## First-run checklist
26
+
27
+ Run these steps in order at the start of every new session:
28
+
29
+ 1. **`GET /memory/manifest`** — see what's already stored (tag counts, agent
30
+ counts, 10 most-recent items). Do this BEFORE asking the user for anything
31
+ that might already be in memory (credentials, project context, decisions).
32
+ 2. **Use the configured connection.** Recall credential locations and purposes,
33
+ never copy secret values into agent memory.
34
+ 3. **`POST /session/start`** — open a session. Save the returned `session_id`
35
+ and pass it in `metadata.session_id` on every `/store` call this conversation.
36
+ 4. **`GET /agent/{your_agent_id}`** — recall your agent's own recent memories.
37
+ Review before starting work.
38
+ 5. **At session end:** call `POST /session/end`, then write a session summary
39
+ (see _What to store and when_ below).
40
+
41
+ ---
42
+
43
+ ## Inspect, correct, forget and checkpoint
44
+
45
+ POST /memory/get and /memory/list inspect exact records, including archives.
46
+ POST /memory/update requires expected_revision and a correction reason. Retained
47
+ superseded versions are explicit history, not ordinary recall.
48
+ POST /memory/forget requires expected_revision and confirm=true. Verify the
49
+ cleanup_complete receipt. It covers that ID and its retained versions, not
50
+ backups, independent copies, encrypted silos or previously returned client context.
51
+ POST /memory/checkpoint saves concise working context; /memory/resume reads it by
52
+ ID without dense indexing. Retain checkpoint_id in the harness handoff and verify
53
+ stored=true. A host hook or proactive agent call must invoke it; MCP does not
54
+ intercept automatic compaction. New snapshots need new IDs.
55
+ List/search/query accept conjunctive structured filters for author, namespace,
56
+ tags, lifecycle dates and scalar metadata. Search/query return active records.
57
+ Keep secrets/private reasoning out of all memories and checkpoints.
58
+
59
+ ## Core write/read endpoints
60
+
61
+ ### `POST /store` — write a memory item
62
+
63
+ ```json
64
+ {
65
+ "text": "the content to remember",
66
+ "agent_id": "kimi",
67
+ "tags": ["fact", "project-context"],
68
+ "metadata": {
69
+ "session_id": "ses_abc123",
70
+ "source": "user-statement",
71
+ "project": "openclaw"
72
+ }
73
+ }
74
+ ```
75
+
76
+ Optional top-level fields:
77
+ - `id` — stable key for this item. Auto-UUID if omitted. Reposting with the same `id` overwrites (safe to retry on transient errors).
78
+ - `embedding` (float[]) — bring your own 768-dim vector. Omit to let the server embed via BAAI/bge-base-en-v1.5.
79
+ - `namespace` — decay tier: `ops` | `notes` | `self`. Default `notes`.
80
+
81
+ #### Choosing a namespace
82
+
83
+ Memories lose activation as they age and gain it each time they are retrieved.
84
+ Below a threshold an item is moved to the cold vault, where it is preserved but
85
+ **stops appearing in `/search` and `/query` results**. The namespace sets how
86
+ fast that clock runs:
87
+
88
+ | namespace | decays | use for |
89
+ |-----------|--------|---------|
90
+ | `ops` | fastest | transient operational detail — job output, retry chatter, one-off status |
91
+ | `notes` (default) | medium | ordinary working memory |
92
+ | `self` | slowest | durable facts about the user, the agent, or the project you want kept |
93
+
94
+ Pick `self` for anything you would be annoyed to lose in a month. An
95
+ unrecognised value returns `400` rather than silently falling back, so a typo
96
+ cannot quietly retier your memories.
97
+
98
+ Response `200`:
99
+ ```json
100
+ { "id": "uuid-or-your-id", "stored": true }
101
+ ```
102
+
103
+ ---
104
+
105
+ ### `POST /query` — semantic similarity search
106
+
107
+ ```json
108
+ {
109
+ "text": "what should I retrieve",
110
+ "agent_id": "kimi",
111
+ "tags": ["credential"],
112
+ "top_k": 5,
113
+ "threshold": 0.6
114
+ }
115
+ ```
116
+
117
+ - `agent_id` — optional; filters to that agent's writes only
118
+ - `tags` — optional; AND semantics (all tags must match)
119
+ - `top_k` — default 10, max 100
120
+ - `threshold` — cosine similarity floor, default 0.0
121
+
122
+ Response `200`:
123
+ ```json
124
+ {
125
+ "count": 2,
126
+ "results": [
127
+ {
128
+ "id": "uuid",
129
+ "score": 0.87,
130
+ "text": "the stored content",
131
+ "agent_id": "kimi",
132
+ "tags": ["credential"],
133
+ "metadata": { "session_id": "ses_abc123" },
134
+ "created_at": "2026-05-13T18:00:00Z"
135
+ }
136
+ ]
137
+ }
138
+ ```
139
+
140
+ ---
141
+
142
+ ### `POST /search` — hybrid BM25 + dense search (RRF fusion)
143
+
144
+ Send text, limit (default 10, maximum 50), qdrant_candidates (default 50,
145
+ maximum 200), and optional agent_id/tags. The broker embeds the text and fuses
146
+ positive BM25 matches with dense results. Agent matching is exact; tags match
147
+ any requested tag. These filters apply before the final result limit.
148
+
149
+ Results contain source text, metadata, canonical document IDs, and available
150
+ BM25/dense ranks and scores. The retrieval object reports backend status,
151
+ candidate limits, unresolved IDs and missing text. A failed leg may return a
152
+ degraded result, including an empty result; inspect bm25_status/dense_status.
153
+
154
+ Non-silo /store now projects into both indexes. HTTP 200 with indexed=true
155
+ confirms completed indexing. HTTP 202 with stored=true and indexed=false means
156
+ the memory is durable but indexing is incomplete: retry the same document ID.
157
+ There is no background retry queue. Silo storage retains its encrypted path.
158
+
159
+ ---
160
+
161
+ ### `POST /embed` — embed without storing
162
+
163
+ ```json
164
+ { "text": "..." }
165
+ ```
166
+
167
+ Response: `{ "embedding": [...768 floats...] }`. Use when you control storage yourself.
168
+
169
+ ---
170
+
171
+ ## Session lifecycle
172
+
173
+ Sessions correlate writes across a conversation so you can replay the full
174
+ context of any prior session later.
175
+
176
+ ### `POST /session/start`
177
+
178
+ ```json
179
+ { "agent_id": "kimi", "ttl_seconds": 86400 }
180
+ ```
181
+
182
+ Response: `{ "session_id": "ses_..." }`
183
+
184
+ Save `session_id` and pass it in `metadata.session_id` on every `/store` call
185
+ this conversation. `ttl_seconds` defaults to 86400 (24 h) if omitted.
186
+
187
+ ### `POST /session/end`
188
+
189
+ ```json
190
+ { "session_id": "ses_..." }
191
+ ```
192
+
193
+ Response: `{ "ok": true }`
194
+
195
+ Call at conversation end, then immediately store a session summary (see below).
196
+
197
+ ### `GET /session/{session_id}/context?n=20`
198
+
199
+ Returns up to 50 items written under that session, ordered by recency:
200
+
201
+ ```json
202
+ {
203
+ "items": [
204
+ { "id": "...", "text": "...", "tags": [...], "created_at": "..." }
205
+ ]
206
+ }
207
+ ```
208
+
209
+ Use this to resume a prior conversation: pass the previous session ID to
210
+ re-hydrate context without re-querying the user.
211
+
212
+ ---
213
+
214
+ ## Agent recall
215
+
216
+ ### `GET /agent/{agent_id}` — your agent's recent memories
217
+
218
+ Returns recent memories written by this `agent_id`. Call at session start to
219
+ re-orient before starting any task.
220
+
221
+ ### `GET /agents` — list active agents
222
+
223
+ Returns distinct `agent_id` values from the last 7 days. Use to discover what
224
+ other agents have contributed to this tenant's memory.
225
+
226
+ ---
227
+
228
+ ## Memory manifest
229
+
230
+ ### `GET /memory/manifest`
231
+
232
+ ```json
233
+ {
234
+ "total": 42,
235
+ "tags": [{ "tag": "credential", "count": 3 }],
236
+ "agents": [{ "agent_id": "kimi", "count": 18 }],
237
+ "recent": [{ "id": "...", "agent_id": "...", "tags": [...], "created_at": "..." }],
238
+ "tag_taxonomy": ["credential","decision","fact","project-context","system-config","conversation","code"],
239
+ "generated_at": "2026-05-13T18:00:00Z"
240
+ }
241
+ ```
242
+
243
+ **Always call this at the start of any non-trivial task.**
244
+
245
+ ---
246
+
247
+ ## Vault
248
+
249
+ ### `GET /vault/stats`
250
+
251
+ Vector count, byte estimate, oldest/newest item timestamps for this tenant.
252
+
253
+ ### `POST /vault/flush-working-memory`
254
+
255
+ Deletes items tagged `conversation` (or a custom tag set you specify). Use at
256
+ session end to prune conversational chaff before it pollutes future retrieval.
257
+
258
+ ### `GET /vault/export`
259
+
260
+ A zip archive containing every vector item for this tenant, one Markdown file
261
+ per item (YAML frontmatter: id, tenant, agent_id, created_at, tags; then the
262
+ body text; then the item's metadata as a fenced JSON block). Embeddings are not
263
+ included, so a restore re-embeds each body server-side. Use for backup or
264
+ migration.
265
+
266
+ ### `POST /admin/vault/rehydrate?tenant_id=...`
267
+
268
+ Admin-only recovery of one vaulted document. Send a JSON body with a `doc_id` field; the
269
+ broker restores Redis, BM25, and Qdrant from the preserved embedding and
270
+ returns `duration_ms`. A newer active record returns `409 Conflict`.
271
+
272
+ ---
273
+
274
+ ## What to store and when
275
+
276
+ **What goes in `text`:** anything that should survive past this conversation.
277
+
278
+ | Content | Tag | Example |
279
+ |---|---|---|
280
+ | Credential location | system-config | Which secret store holds the configured connection, without its value |
281
+ | Architecture decision | `decision` | `"Chose PostgreSQL over MySQL — reason: JSON operators"` |
282
+ | User-stated fact | `fact` | `"The user's email is alex@example.com"` |
283
+ | Project layout | `project-context` | `"OpenClaw: VM 242, user rm, pnpm monorepo at ~/openclaw"` |
284
+ | Server / env config | `system-config` | `"HeurChain tenant abc-123, key rotated 2026-05-13"` |
285
+ | Session summary | `conversation` | One-paragraph summary of what was done and what's pending |
286
+ | Code snippet | `code` | Small, self-contained snippet useful across sessions |
287
+
288
+ **What NOT to store:** raw conversation transcripts, large file contents, data
289
+ that changes every run (dynamic UUIDs, ephemeral timestamps), anything that
290
+ belongs in a repo or config file.
291
+
292
+ **When to call `/store`:**
293
+
294
+ - **Connection configured**: record its purpose and secret-store location only;
295
+ never the API key, token or password itself.
296
+ - **Decision made** — when you and the user agree on an approach, architecture
297
+ choice, or policy. Tag `decision`.
298
+ - **User states a fact** — name, email, preference, account info. Tag `fact`.
299
+ - **Context eviction** — context window filling up; store a summary before
300
+ truncating. Tag `conversation`.
301
+ - **Session end** — one-paragraph summary of accomplishments and open items.
302
+ Tag `conversation`.
303
+ - **NOT after every message** — that produces retrieval noise. Store selectively.
304
+
305
+ **Chunking:** max 512 tokens (~2000 chars) per item. Split larger content into
306
+ segments; link them via `metadata.session_id` or a shared `id` prefix
307
+ (e.g. `doc-plan-1`, `doc-plan-2`).
308
+
309
+ ---
310
+
311
+ ## Tags
312
+
313
+ Tag rules: lowercase alphanumeric, dash or underscore, max 32 chars, max 16 per item.
314
+
315
+ **Taxonomy** — use these names for cross-agent consistency:
316
+
317
+ | Tag | Use for |
318
+ |---|---|
319
+ | `credential` | Credential locations and purposes; never secret values |
320
+ | `decision` | Architecture, tooling, or policy choices |
321
+ | `fact` | User-stated facts, preferences, account info |
322
+ | `project-context` | Project structure, stack, file locations |
323
+ | `system-config` | Server config, env vars, host addresses |
324
+ | `conversation` | Session summaries, ephemeral notes |
325
+ | `code` | Code snippets, scripts, config files |
326
+
327
+ ---
328
+
329
+ ## Error handling
330
+
331
+ | Status | Meaning | Action |
332
+ |---|---|---|
333
+ | `200` | Success | — |
334
+ | `400` | Bad request (missing field, malformed JSON) | Fix payload, do not retry |
335
+ | `401` | Auth failed (bad/missing key or tenant) | Ask user for correct credentials |
336
+ | `404` | Item or session not found | Do not retry |
337
+ | `429` | Rate limit exceeded | Body: `{"limit":60,"current":63}`. Back off 60 s then retry |
338
+ | `5xx` | Server error | Exponential backoff, max 3 attempts. Report if persistent |
339
+
340
+ All error responses are JSON: `{ "error": "description" }`.
341
+
342
+ ---
343
+
344
+ ## OpenClaw / Hermes integration
345
+
346
+ - Set `agent_id` to `"openclaw"` or `"hermes"` consistently across all sessions for this tenant.
347
+ - At first boot: store your MCP server list, model routing config, and workspace
348
+ path under tag `system-config`.
349
+ - Each session start: call `GET /agent/openclaw` (or `/agent/hermes`) before any task.
350
+ - If the HeurChain MCP server is configured, prefer MCP tools
351
+ (`mcp__heurchain__heurchain_search`, `mcp__heurchain__heurchain_export`).
352
+ Fall back to direct HTTP if MCP is unavailable.
353
+ - `POST /promote` — increments a memory item's retrieval weight. Call after a
354
+ human confirms a stored fact is correct.
355
+
356
+ ---
357
+
358
+ ## Discovery
359
+
360
+ - `GET /` — this document (`text/markdown` default; `Accept: application/json` for structured spec). Send tenant headers to get a manifest summary appended.
361
+ - `GET /health` — service liveness (no auth)
362
+ - `GET /ready` — Redis/SQLite and authenticated Qdrant read connectivity (no auth); 503 if a dependency fails. Does not attest index coverage.
363
+ - `GET /memory/manifest` — tenant memory summary (auth required)
364
+ - `GET /openapi.json` — OpenAPI 3 spec (no auth)
365
+
package/CHANGELOG.md CHANGED
@@ -1,103 +1,18 @@
1
- # HeurChain Changelog
1
+ # Changelog (npm package)
2
2
 
3
- ## v1.4.1 — 2026-05-07
3
+ ## [2.0.0] - 2026-11-10
4
4
 
5
- ### Architecture
5
+ ### Changed (breaking)
6
+ - HeurChain is now a self-hosted Go memory broker, version 0.6.1, run with Docker Compose
7
+ or systemd from https://github.com/peterjohannmedina/heurchain. This package no longer
8
+ ships a server: `server.js` and its dependencies are gone, and `main` is removed.
9
+ - `AGENT_CONFIG.json` is rewritten for the broker 0.6.1 API (schema
10
+ `heurchain-agent-config/3.0`): two-header API-key auth, MCP over Streamable HTTP, the
11
+ session protocol, every HTTP operation and every MCP tool.
6
12
 
7
- - **Briefing worker is pure memory delivery — no inference.** HeurChain is a memory substrate. The briefing worker assembles raw proc entries and session turns; it never calls an LLM. The receiving agent decides whether to surface items, query deeper via `heurchain_search`, or synthesize with its own model.
8
- - `runCompressor` remains for consolidation cue compression only (background worker, separate concern).
13
+ ### Added
14
+ - `AGENT_ONBOARDING.md`: the broker's own agent guide, as served at `GET /`.
9
15
 
10
- ### Changes
16
+ ## [1.4.2] and earlier
11
17
 
12
- - Removed LLM synthesis path from `generateBriefing()`. Raw proc + session items IS the correct output, not a fallback.
13
- - Default `BRIEFING_INTERVAL_MS` changed from 6 hours → 24 hours (daily cadence).
14
- - Added optional webhook push: set proc key `briefing_webhook_url` for an agent and the worker will POST `{ agent, timestamp, items }` to that URL after each briefing generation (fire-and-forget, 10s timeout). Agent's orchestration layer handles the payload independently.
15
-
16
- ---
17
-
18
- ## v1.4.0 — 2026-05-07
19
-
20
- ### New: Ollama-first compressor provider chain
21
-
22
- - `COMPRESSOR_PROVIDER`: `ollama` (default when `OLLAMA_BASE_URL` is set) | `anthropic` | `none`
23
- - Ollama default model: `qwen2.5:1.5b` — runs on CPU, ~1 GB RAM, no GPU or external API required
24
- - `anthropic` provider optional — requires `ANTHROPIC_API_KEY`
25
- - `none` provider: consolidation still fires, marks keys consolidated, skips cue generation
26
- - Removes hard dependency on external API for self-hosted deployments
27
-
28
- ### New: Proactive memory briefing worker
29
-
30
- - `runProactiveBriefings()` runs on `BRIEFING_INTERVAL_MS` schedule (default 24h) plus once 30s after startup
31
- - Discovers active agents by scanning `proc:agent:*` keys
32
- - Assembles `"HeurChain: {item}"` reminders from proc memory and recent session buffer turns
33
- - Caches result at `briefing:{agent}:latest` (Redis, 24h TTL)
34
- - `/api/session-context` now returns a `briefing` field — pre-generated reminders ready at session start with zero added latency (cache hit path)
35
- - Per-agent controls via proc keys: `briefing_disabled`, `briefing_interval_ms`, `briefing_instructions`, `briefing_webhook_url`
36
-
37
- ---
38
-
39
- ## v1.3.0 — 2026-05-07
40
-
41
- ### New: Frictionless REST layer
42
-
43
- - `GET /api/session-context?agent=X&hint=Y` — token-budgeted payload (proc knobs + last session + top BM25 hits) designed for system-prompt injection via hook at session start. Zero MCP tool calls required.
44
- - `GET /api/search?q=...` — BM25 search over the full index via plain HTTP. No SSE session, no tool definition overhead.
45
- - `POST /api/buffer` — fire-and-forget turn capture. Flushes to Redis on 5-min heartbeat or session end. No agent action required.
46
- - `POST/GET/DELETE /api/proc` — procedural memory tier. Stable agent preferences (coding style, language, patterns) stored with no TTL or decay.
47
-
48
- ### New: Slim MCP profile
49
-
50
- - Connect with `?profile=slim` on `/sse` to load 14 core tools instead of 29 (~750 token savings on tool definitions per context window).
51
- - Infrastructure tools (Prometheus, Grafana, Proxmox, Ceph) only load on `?profile=full`.
52
-
53
- ### New: MCP tools
54
-
55
- - `proc_set` / `proc_get` — read and write procedural memory from within an MCP session.
56
-
57
- ### Fix: BM25 index was Redis-blind
58
-
59
- - Silent bug in node-redis `scanIterator` caused it to yield key pages (arrays) instead of individual keys. Every `redis.get(array)` call failed silently, so BM25 was built from Obsidian vault files only — Redis keys were never indexed. Fixed via `scanKeys()` normalizer. Index now correctly covers both tiers (465 docs vs 447 before).
60
-
61
- ---
62
-
63
- ## v1.2.1 — 2026-05-06
64
-
65
- ### Security
66
-
67
- - Path traversal fix: `validateKey()` now rejects `..` segments, null bytes, and slash characters before resolving vault paths.
68
- - Bearer token auth (`MCP_AUTH_TOKEN`) enforced on all endpoints. `/health` remains open for healthcheck probes.
69
- - Grafana tool fails closed when credentials are not configured rather than exposing unauthenticated access.
70
-
71
- ### Fixes
72
-
73
- - SSE session cap (50) with cleanup on both `req.on("close")` and `res.on("close")` to prevent transport leaks.
74
- - BM25 rebuild is atomic: `isBuilding` flag + `pendingOps` queue replays writes that arrived during a rebuild onto the fresh index after swap. No silent memory loss during hourly rebuild.
75
- - `bulkAdd()` for O(N log N) index builds instead of O(N²) per-insert `_recalcStats()` calls.
76
- - Per-key try/catch in `buildIndex` Redis scan — a single bad key type no longer aborts the entire scan.
77
- - Tier validation: unknown tier strings fall back to `"all"` instead of returning empty results.
78
-
79
- ### Features
80
-
81
- - `heurchain_export` — glacier sync: exports Redis longterm-tier entries to Obsidian vault as markdown. Idempotent.
82
- - `minScore` parameter on `heurchain_search` to filter low-signal results.
83
- - Namespace isolation: `namespace` parameter scopes search and write operations to an agent-specific key prefix.
84
- - Session noise filter: suppresses low-signal session metadata keys from BM25 results.
85
-
86
- ---
87
-
88
- ## v1.2.0 — 2026-05-05
89
-
90
- ### Features
91
-
92
- - Native BM25 search (Okapi BM25, k1=1.5, b=0.75) implemented in-process. Replaced dead broker call to port 3012.
93
- - Redis integration: `redis-cache` service added to Docker stack; `REDIS_URL` fixed to use Docker service name.
94
- - `heurchain_search` now indexes all Redis string keys + Obsidian vault `.md` files on startup with hourly rebuild.
95
-
96
- ---
97
-
98
- ## v1.1.0 — initial release
99
-
100
- - MCP SSE server with 27 tools across cache, vault, search, monitoring, infrastructure, and user context categories.
101
- - Tiered storage: Redis (longterm) + Obsidian vault (working).
102
- - nginx gateway for SSE proxying.
103
- - Ansible role for managed host deployment.
18
+ The Node.js MCP server. Deprecated; still installable as `heurchain@1`.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 HeurChain contributors
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.