@psiclawops/hypermem 0.9.3 → 0.9.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (69) hide show
  1. package/CHANGELOG.md +27 -2
  2. package/INSTALL.md +75 -68
  3. package/README.md +18 -36
  4. package/assets/default-config.json +41 -0
  5. package/bench/data-access-bench.mjs +1 -1
  6. package/bin/hypermem-doctor.mjs +76 -2
  7. package/bin/hypermem-status.mjs +255 -7
  8. package/dist/adaptive-lifecycle.d.ts +39 -0
  9. package/dist/adaptive-lifecycle.d.ts.map +1 -1
  10. package/dist/adaptive-lifecycle.js +87 -9
  11. package/dist/background-indexer.d.ts.map +1 -1
  12. package/dist/background-indexer.js +16 -14
  13. package/dist/compositor.d.ts.map +1 -1
  14. package/dist/compositor.js +239 -20
  15. package/dist/cross-agent.d.ts +1 -1
  16. package/dist/cross-agent.js +17 -17
  17. package/dist/hybrid-retrieval.d.ts +8 -0
  18. package/dist/hybrid-retrieval.d.ts.map +1 -1
  19. package/dist/hybrid-retrieval.js +112 -10
  20. package/dist/index.d.ts +16 -5
  21. package/dist/index.d.ts.map +1 -1
  22. package/dist/index.js +19 -4
  23. package/dist/message-store.d.ts +62 -1
  24. package/dist/message-store.d.ts.map +1 -1
  25. package/dist/message-store.js +355 -2
  26. package/dist/open-domain.d.ts.map +1 -1
  27. package/dist/open-domain.js +3 -2
  28. package/dist/proactive-pass.d.ts +42 -2
  29. package/dist/proactive-pass.d.ts.map +1 -1
  30. package/dist/proactive-pass.js +294 -39
  31. package/dist/seed.d.ts +1 -1
  32. package/dist/seed.js +1 -1
  33. package/dist/session-flusher.d.ts +2 -2
  34. package/dist/session-flusher.js +2 -2
  35. package/dist/spawn-context.d.ts +1 -1
  36. package/dist/spawn-context.js +1 -1
  37. package/dist/topic-store.js +5 -5
  38. package/dist/topic-synthesizer.d.ts.map +1 -1
  39. package/dist/topic-synthesizer.js +10 -4
  40. package/dist/trigger-registry.d.ts +1 -1
  41. package/dist/trigger-registry.js +4 -4
  42. package/dist/types.d.ts +101 -2
  43. package/dist/types.d.ts.map +1 -1
  44. package/dist/vector-store.d.ts +10 -1
  45. package/dist/vector-store.d.ts.map +1 -1
  46. package/dist/vector-store.js +45 -9
  47. package/docs/DIAGNOSTICS.md +88 -1
  48. package/docs/INTEGRATION_VALIDATION.md +40 -1
  49. package/docs/MIGRATION.md +1 -1
  50. package/docs/TUNING.md +47 -6
  51. package/install.sh +5 -60
  52. package/memory-plugin/dist/index.js +192 -0
  53. package/memory-plugin/openclaw.plugin.json +199 -2
  54. package/memory-plugin/package.json +2 -2
  55. package/package.json +29 -10
  56. package/plugin/dist/index.d.ts +2 -0
  57. package/plugin/dist/index.d.ts.map +1 -1
  58. package/plugin/dist/index.js +178 -11
  59. package/plugin/dist/index.js.map +1 -1
  60. package/plugin/openclaw.plugin.json +199 -2
  61. package/plugin/package.json +2 -2
  62. package/scripts/install-packed-runtime.mjs +99 -0
  63. package/scripts/install-runtime.mjs +164 -4
  64. package/ARCHITECTURE.md +0 -298
  65. package/docs/KNOWN_LIMITATIONS.md +0 -35
  66. package/docs/PHASE1-VALIDATION.md +0 -132
  67. package/docs/RELEASE_0.8.0_VALIDATION.md +0 -70
  68. package/docs/RELEASE_PROCESS.md +0 -10
  69. package/docs/ROADMAP.md +0 -266
package/ARCHITECTURE.md DELETED
@@ -1,298 +0,0 @@
1
- # hypermem Architecture
2
-
3
- _Agent-centric memory that outlives sessions, backed by SQLite memory databases._
4
-
5
- ---
6
-
7
- ## Memory Layers
8
-
9
- ```
10
- L1 SQLite Cache (Hot) Active session working memory
11
- │ Slots: system, identity, messages, facts, context
12
- │ Sub-millisecond reads, evicts on session end
13
- │ Fleet cache: agent profiles, fleet summary
14
-
15
- L2 Agent Messages DB Raw conversation history (per agent)
16
- │ messages.db Write-heavy, rotatable (100MB / 90 days)
17
- │ messages_YYYYQN.db Rotated archives, read-only
18
-
19
- L3 Agent Vectors DB Semantic search index (per agent)
20
- │ vectors.db Index rebuilt from L2+L4 (reconstructable)
21
- │ KNN search via sqlite-vec (nomic-embed-text, 768d)
22
- │ Rate-limited embedding via token bucket
23
-
24
- L4 Library DB Fleet-wide structured knowledge
25
- library.db Facts, knowledge, preferences, fleet registry
26
- Episodes, topics, work items, system state
27
- Desired state, agent capabilities
28
- Knowledge graph (DAG links between entities)
29
- ```
30
-
31
- > Note: some internal method names and telemetry reasons still contain `redis`
32
- > for backward compatibility. The runtime hot layer is SQLite `:memory:` cache,
33
- > not an external Redis service.
34
-
35
- ## Database Schema
36
-
37
- ### messages.db (per agent, schema v10)
38
- - `agent_meta` — agent metadata
39
- - `conversations` — session tracking
40
- - `messages` — raw message log (text, tool calls, tool results)
41
- - `schema_version` — migration tracking
42
- - **Rotation:** When size > 100MB or age > 90 days, renamed to `messages_YYYYQN.db`
43
- - **WAL mode** with checkpoint on rotation
44
-
45
- ### vectors.db (per agent)
46
- - `vec_facts` — 768-dimensional vectors for facts (sqlite-vec virtual table)
47
- - `vec_knowledge` — vectors for knowledge entries
48
- - `vec_episodes` — vectors for episode descriptions
49
- - `vec_sessions` — vectors for session summaries
50
- - `vec_index_map` — tracks what's been indexed (source_table, source_id, source_db)
51
- - `embedding_cache` — avoids redundant Ollama API calls
52
-
53
- ### library.db (shared, schema v19)
54
- - `facts` — verifiable claims with confidence, domain, expiry, supersedes chains
55
- - `knowledge` — domain/key/value structured data
56
- - `knowledge_links` — DAG edges between entities (fact↔fact, fact↔knowledge, etc.)
57
- - `episodes` — significant events with impact and participants
58
- - `topics` — cross-session thread tracking
59
- - `preferences` — operator/user behavioral patterns
60
- - `fleet_agents` — agent registry with tier, org, capabilities (JSON)
61
- - `fleet_orgs` — organizational structure
62
- - `agent_capabilities` — queryable skills, tools, MCP servers per agent
63
- - `agent_desired_state` — intended configuration vs. actual (drift detection)
64
- - `agent_config_events` — change audit trail
65
- - `system_registry` — service state tracking
66
- - `system_events` — service lifecycle events
67
- - `work_items` — work queue entries with FTS5
68
- - `session_registry` — session lifecycle tracking
69
-
70
- ## Compositor
71
-
72
- Assembles LLM prompts from all four layers with token budgeting:
73
-
74
- ```
75
- User message arrives
76
-
77
- ├── L1 Hot cache: system prompt, identity, cached slots
78
- ├── L2 Messages: recent conversation history (budget-truncated)
79
- ├── L3 Vectors: KNN semantic recall on user's latest message
80
- │ └── Related facts/knowledge/episodes with relevance scores
81
- ├── L4 Library: structured knowledge injection
82
- │ ├── Facts (up to 30% of remaining budget): active, non-expired, sorted by confidence
83
- │ ├── Knowledge (up to 20% of remaining): grouped by domain, top 15 entries
84
- │ └── Preferences (up to 10% of remaining): behavioral patterns, grouped by subject
85
- └── Cross-session (up to 20% of remaining): context from related sessions
86
- ```
87
-
88
- Each slot gets a proportional budget cap. Smart truncation at line boundaries.
89
- Multi-provider output: Anthropic and OpenAI message formats.
90
-
91
- ### Tuning Parameters (TUNE-001–007)
92
-
93
- Compositor behavior is tuned via parameters tracked in `tune/TUNING_REGISTRY.md`:
94
-
95
- | ID | Parameter | Value | Effect |
96
- |---|---|---|---|
97
- | TUNE-001 | Semantic recall min RRF score | 0.008 | Drops noise results from hybrid search |
98
- | TUNE-002 | Facts confidence floor | 0.5 | Excludes low-confidence facts from injection |
99
- | TUNE-003 | Differentiated fact confidence | 0.60–0.75 by type | Decisions/incidents score higher than config/prefs |
100
- | TUNE-004 | config_change episode significance | 0.5 (was 0.4) | Config changes no longer silently dropped |
101
- | TUNE-005 | Extraction slot guard | suppress on default | No-op when strategy is lightweight/default |
102
- | TUNE-006 | Advisor slot guard | suppress bare seat list | No-op when no domain routes matched |
103
- | TUNE-007 | Identity anchor guard | suppress on default identity | No-op when identity resolves to 'default' |
104
-
105
- ### Safety Mechanisms
106
-
107
- - **Budget safety valve:** Post-assembly check — if estimated tokens exceed budget × 1.05, trims oldest history messages until under budget. System/identity/current prompt are never touched.
108
- - **Compaction fence:** Per-conversation boundary protecting the LLM's recent tail from compaction. Only moves forward (monotone progress). No fence = no compaction (explicit opt-in).
109
- - **Preservation gate:** Nomic-space geometric verification that summaries stay faithful to source content. Centroid alignment + source coverage → combined score (threshold: 0.65).
110
-
111
- ## Fleet Cache (Hot Cache Layer)
112
-
113
- ```
114
- fleet:agent:{id} — Composite profile: registry + capabilities + desired state
115
- fleet:summary — Fleet-wide stats: agent count, drift count, tier breakdown
116
- ```
117
-
118
- - **Cache-aside** on reads: hot cache first, SQLite fallback, warm on miss
119
- - **Write-through invalidation** on fleet mutations
120
- - **Hydration** on gateway startup: bulk-populate from library.db
121
- - TTL: agent profiles 10min, summary 2min
122
-
123
- ## Knowledge Graph
124
-
125
- DAG traversal over `knowledge_links` for relationship discovery:
126
-
127
- - **Entity types:** fact, knowledge, topic, episode, agent, preference
128
- - **Link types:** supports, contradicts, supersedes, references, derived_from, depends_on, extends, covers, related, authored_by
129
- - **BFS traversal:** configurable depth, result limit, direction filter, type filter
130
- - **Shortest path:** between any two entities
131
- - **Analytics:** most-connected entities, link count by type
132
-
133
- ## Rate Limiter
134
-
135
- Token-bucket rate limiter for embedding API calls:
136
-
137
- - Burst capacity with steady refill (default: 5/s, burst 10)
138
- - Priority queue: high (user-facing recall) > normal > low (batch indexing)
139
- - Reserved tokens for high-priority requests
140
- - `createRateLimitedEmbedder()` wraps any embedding function
141
-
142
- ## Cross-Agent Access Control
143
-
144
- Visibility-tiered access model for cross-agent knowledge queries:
145
-
146
- - **`fleet`** — visible to all agents (default tier)
147
- - **`org`** — visible to agents in the same organizational unit
148
- - **`council`** — visible to council-tier agents and the org lead
149
- - **`private`** — visible only to the owning agent
150
-
151
- ### Org Registry
152
-
153
- `visibilityFilter()` resolves access levels using an `OrgRegistry` — a mapping of agents to tiers, orgs, and capabilities. Currently loaded from a hardcoded `defaultOrgRegistry()` in `cross-agent.ts`.
154
-
155
- **Known limitation:** `defaultOrgRegistry()` duplicates fleet structure that lives authoritatively in `fleet_agents` + `fleet_orgs` in library.db. The planned migration is to live-load the registry from library.db on startup, with the hardcoded registry retained only as a cold-start fallback.
156
-
157
- ### Unknown Agent Fallback (Restrictive Default)
158
-
159
- When a target agent is not found in the registry, `visibilityFilter()` applies a **restrictive default**: fleet-only visibility with a logged warning. This is a deliberate safety-side behavior — unknown agents see only fleet-visible data rather than failing the query entirely. The warning surfaces registry gaps for operators to fix.
160
-
161
- This means:
162
- - Queries succeed but return a narrowed result set
163
- - New agents provisioned in the DB but not yet in the registry will have reduced cross-agent visibility until registered
164
- - The warning message names the missing agent and suggests adding it to the registry
165
-
166
- ## Context Engine Plugin
167
-
168
- `plugin/src/index.ts` — OpenClaw context engine plugin (`hypercompositor`, fills `contextEngine` slot):
169
-
170
- ```
171
- gateway:startup → Init hypermem, auto-rotate DBs, seed fleet registry from workspace identities, hydrate fleet cache
172
- agent:bootstrap → Warm session (history, facts, profile → hot cache)
173
- context:assemble → Full four-layer prompt assembly within token budget
174
- agent:afterTurn → Ingest new messages to SQLite + hot cache, trigger background indexer
175
- ```
176
-
177
- Registers with `ownsCompaction: true` — runtime skips legacy compaction entirely.
178
-
179
- ## Memory Plugin
180
-
181
- `memory-plugin/src/index.ts` — Lightweight memory provider (`hypermem`, fills `memory` slot):
182
-
183
- - Registers `MemoryPluginCapability` with a `MemorySearchManager` backed by HyperMem's hybrid FTS5 + KNN retrieval
184
- - Provides the `memory_search` tool through the official memory slot interface
185
- - Public artifacts provider lists `MEMORY.md` and `memory/*.md` for all configured agents
186
- - Stateless wrapper: lifecycle is owned by the context engine plugin
187
-
188
- ### Plugin Data Flow
189
-
190
- ```
191
- ┌──────────────────────────────────────────────────┐
192
- │ HOT CACHE (SQLite :memory: CacheLayer) │
193
- │ │
194
- │ hm:{a}:{s}:history ── Session archive (250 cap │
195
- │ (append-only) at bootstrap, 1000 soft │
196
- │ cap ongoing) │
197
- │ │
198
- │ hm:{a}:{s}:window ── Submission buffer │
199
- │ (compositor output) (120s TTL) │
200
- │ │
201
- │ hm:{a}:{s}:cursor ── Last-sent pointer │
202
- │ (compositor metadata) (24h TTL) │
203
- │ │
204
- │ hm:{a}:{s}:system ── System prompt slot │
205
- │ hm:{a}:{s}:identity ── Identity slot │
206
- │ hm:{a}:{s}:facts ── Cached facts slot │
207
- │ hm:{a}:{s}:context ── Cross-session slot │
208
- └──────────────────────────────────────────────────┘
209
-
210
- Data Flow (current — P0 stabilized, window/cursor active):
211
-
212
- bootstrap() assemble() afterTurn()
213
- ─────────── ────────── ───────────
214
- ▸ sessionExists() → skip if hot compose() slice(prePromptCount)
215
- ▸ SQLite ─→ warmSession() ─→ getHistory(limit) ✅ ─→ record*Message()
216
- ─→ pushHistory(250) ─→ dedup by id ─→ pushHistory(1, dedup)
217
- ─→ cache history ─→ budget assembly ─→ cache history
218
- ─→ write window bundle ─→ invalidateWindow()
219
- ─→ write cursor metadata ─→ background indexer
220
- ─→ → runtime → provider
221
-
222
- ### Key Invariants
223
-
224
- 1. Hot-cache `history` is the warm archive. Append-only. Nothing reads it for direct submission.
225
- 2. Hot-cache `window` is the compositor's output cache. Written ONLY by `compose()`. Read ONLY by `assemble()`. Invalidated by `afterTurn`.
226
- 3. Hot-cache `cursor` tracks the newest message in the last window. Used by background indexer for high-signal mining.
227
- 4. `warmSession()` seeds `history` only (capped at 250). Never writes `window`.
228
- 5. `pushHistory()` tail-checks before append (no duplicate IDs in the hot-cache history list).
229
- 6. `compose()` deduplicates history by `id` before budget assembly.
230
- 7. `getHistory()` honors its `limit` parameter on BOTH hot-cache and SQLite paths.
231
-
232
- Open and deferred work is tracked outside this public architecture reference.
233
-
234
- ### Runtime Contract
235
-
236
- **Exclusive dispatch:** The OpenClaw runtime calls either `afterTurn()` OR `ingest()`/`ingestBatch()`, never both. Since hypermem implements `afterTurn`, it must handle message ingestion there. `ingest()` exists for API compatibility but is never called by the runtime in practice.
237
-
238
- **Provider translation:** The plugin sets `skipProviderTranslation: true` on compose requests. The compositor returns NeutralMessages; the plugin converts to AgentMessages. The runtime handles provider-specific translation. Two-stage translation (compositor → provider format → plugin → agent format) was the root cause of Incident 1 (silent tool call drops).
239
-
240
- ## Module Map (29 files, ~12,300 lines)
241
-
242
- | Module | Lines | Layer | Purpose |
243
- |---|---|---|---|
244
- | `index.ts` | ~1,340 | All | Facade — all public API |
245
- | `compositor.ts` | ~1,140 | L1-L4 | Prompt assembly + token budgeting + safety valve + window/cursor write |
246
- | `library-schema.ts` | ~780 | L4 | Library schema v19 + migrations |
247
- | `background-indexer.ts` | ~680 | L2-L4 | LLM-powered extraction framework |
248
- | `vector-store.ts` | ~600 | L3 | Semantic search + embedding |
249
- | `hybrid-retrieval.ts` | ~450 | L3-L4 | FTS5 + KNN with Reciprocal Rank Fusion |
250
- | `fleet-store.ts` | ~440 | L4 | Fleet registry + capabilities |
251
- | `db.ts` | ~440 | - | Database manager + rotation |
252
- | `knowledge-graph.ts` | ~420 | L4 | DAG traversal + shortest path |
253
- | `cache.ts` | ~700 | L1 | SQLite `:memory:` hot-cache operations, window cache, cursor, fleet cache |
254
- | `doc-chunker.ts` | ~400 | - | Section-aware markdown/file parser |
255
- | `work-store.ts` | ~400 | L4 | Work queue + FTS5 |
256
- | `provider-translator.ts` | ~390 | - | Neutral ↔ provider format conversion |
257
- | `doc-chunk-store.ts` | ~375 | L4 | Chunk storage + deduplication |
258
- | `message-store.ts` | ~370 | L2 | Conversation recording + querying |
259
- | `types.ts` | ~370 | - | Shared type definitions + SessionCursor |
260
- | `cross-agent.ts` | ~330 | L2-L4 | Cross-agent knowledge queries + visibility |
261
- | `desired-state-store.ts` | ~310 | L4 | Config drift detection |
262
- | `knowledge-store.ts` | ~300 | L4 | Domain/key/value structured data |
263
- | `secret-scanner.ts` | ~285 | - | Credential/secret detection |
264
- | `system-store.ts` | ~250 | L4 | Service state tracking |
265
- | `seed.ts` | ~250 | L4 | Workspace seeder + collection inference |
266
- | `fact-store.ts` | ~230 | L4 | Facts with confidence + expiry |
267
- | `rate-limiter.ts` | ~230 | L3 | Token-bucket for embedding API |
268
- | `schema.ts` | ~200 | L2 | Messages schema + migrations |
269
- | `episode-store.ts` | ~180 | L4 | Significant event tracking |
270
- | `preference-store.ts` | ~170 | L4 | Operator behavioral patterns |
271
- | `topic-store.ts` | ~160 | L4 | Cross-session thread tracking |
272
- | `plugin/src/index.ts` | ~590 | - | `hypercompositor` context engine plugin + window invalidation |
273
- | `memory-plugin/src/index.ts` | ~290 | - | `hypermem` memory slot plugin (memory_search via hybrid retrieval) |
274
-
275
- ## Test Coverage (105 assertions, 11 suites)
276
-
277
- _Test count reflects assertions, not individual test blocks. Suites contain inline assertions._
278
-
279
- | Suite | Key coverage |
280
- |---|---|
281
- | smoke | End-to-end create/write/read/close, provider translation |
282
- | redis-integration | Legacy suite name, covers hot-cache ops, slots, history limits, window cache, cursor, warming, dedup |
283
- | cross-agent | Cross-agent queries, fleet search, visibility tiers |
284
- | vector-search | Embedding, KNN, batch indexing |
285
- | library | All L4 collections (facts → desired state) |
286
- | compositor | Four-layer composition, budgets, providers, safety valve, Gate 1 |
287
- | fleet-cache | Fleet hot-cache hydration and cache-aside behavior |
288
- | rotation | DB rotation, auto-rotate, collision handling |
289
- | knowledge-graph | DAG traversal, shortest path, analytics |
290
- | rate-limiter | Token bucket, priority, timeout, embedder |
291
- | doc-chunker | Markdown/file chunking, section-aware parsing, seeder |
292
-
293
- ## Dependencies
294
-
295
- - `node:sqlite` (Node 22+ built-in) — zero-dependency SQLite
296
- - No external cache service dependency — hot cache is SQLite `:memory:`
297
- - `sqlite-vec` — optional, vector search extension
298
- - Ollama (localhost:11434) — optional, embedding generation
@@ -1,35 +0,0 @@
1
- # Known Limitations — HyperMem 0.8.0
2
-
3
- ## Global-scope fact write authorization
4
-
5
- Facts written with `scope='global'` are readable fleet-wide at L4 priority. The write path has no authorization gate — any agent with access to the HyperMem API can write a global-scope fact.
6
-
7
- **Impact:** In a trusted single-operator deployment (the intended target), this is acceptable. All agents are operator-controlled and share the same trust boundary.
8
-
9
- **Planned fix:** Introduce a write-authority model that gates global-scope writes to designated agents (council seats or explicitly allowlisted agent IDs). See [docs/ROADMAP.md](ROADMAP.md).
10
-
11
- **Workaround:** Restrict HyperMem API access to trusted agents only. Do not expose the API to untrusted external agents.
12
-
13
- ## Cross-agent org registry is hardcoded
14
-
15
- `visibilityFilter()` in `cross-agent.ts` resolves agent tiers and visibility from a hardcoded `defaultOrgRegistry()`. This duplicates fleet topology that lives authoritatively in `fleet_agents` + `fleet_orgs` in library.db.
16
-
17
- **Impact:** New agents added to library.db but not the hardcoded registry get fleet-only visibility until the registry is updated in code.
18
-
19
- **Planned fix:** Live-load registry from library.db on startup, hardcoded as cold-start fallback only. See [docs/ROADMAP.md](ROADMAP.md).
20
-
21
- ## Cursor durability across restarts
22
-
23
- The compositor writes a session cursor to hot cache (SQLite `:memory:`) with a 24h TTL. On gateway restart, the cursor is lost until the next `compose()` call repopulates it.
24
-
25
- **Impact:** Background indexer may miss the cursor on the first turn after a restart, falling back to full-history scan.
26
-
27
- **Planned fix:** Dual-write cursor to messages.db so it survives restarts. See [docs/ROADMAP.md](ROADMAP.md).
28
-
29
- ## Cross-session context has no boundary markers
30
-
31
- `buildCrossSessionContext()` renders flat previews with no per-message boundaries or sender identity labels.
32
-
33
- **Impact:** Context from different sessions blends together, making attribution ambiguous in multi-agent scenarios.
34
-
35
- **Planned fix:** WQ-20260402-001. See [docs/ROADMAP.md](ROADMAP.md).
@@ -1,132 +0,0 @@
1
- # HyperMem Phase 1 Validation Runbook
2
-
3
- Operator-facing guide for running and interpreting the Phase 1 validation suite.
4
-
5
- ---
6
-
7
- ## Quick Start
8
-
9
- Run the full Phase 1 validation flow:
10
-
11
- ```bash
12
- npm run build && node scripts/validate-compose.mjs && node scripts/validate-config-surface.mjs
13
- ```
14
-
15
- Or run individual validations:
16
-
17
- ```bash
18
- # Compose validation (facts, library, budget pressure)
19
- node scripts/validate-compose.mjs
20
-
21
- # Config surface parity (install.sh, docs, README)
22
- node scripts/validate-config-surface.mjs
23
-
24
- # Config key resolution tests
25
- node test/config-validation.mjs
26
-
27
- # Retrieval regression harness
28
- node test/retrieval-regression.mjs
29
-
30
- # Compositor integration (includes budget-pressure fixture)
31
- node test/compositor.mjs
32
-
33
- # Plugin pipeline (requires plugin build)
34
- npm run validate:plugin-pipeline
35
-
36
- # Startup fleet seeding on cold boot
37
- node test/fleet-startup-seeding.mjs
38
-
39
- # Release path hardening harness (builds core + plugin)
40
- npm run validate:release-path
41
-
42
- # Compose report (operator-readable diagnostics)
43
- node scripts/compose-report.mjs
44
- ```
45
-
46
- ---
47
-
48
- ## What Each Validation Covers
49
-
50
- | Validation | What it proves |
51
- |---|---|
52
- | `validate-compose.mjs` | End-to-end compose with seeded facts, knowledge retrieval, and budget pressure |
53
- | `validate-config-surface.mjs` | Config keys present in install.sh, INSTALL.md, TUNING.md, README |
54
- | `config-validation.mjs` | contextWindowOverrides sanitization, budget resolution, maintenance defaults |
55
- | `retrieval-regression.mjs` | Scope isolation, superseded-fact filtering, budget pressure, knowledge retrieval |
56
- | `compositor.mjs` | Four-layer compose, trigger routing, keystone injection, budget-pressure filtering |
57
- | `plugin-pipeline.mjs` | Real plugin assemble() path with seeded L4 memory, tight-budget proof |
58
- | `fleet-startup-seeding.mjs` | Cold-start fleet population from workspace identity files plus idempotent repeat-boot behavior |
59
- | `release-gateway-path.mjs` | Real plugin release-path proof: tool-chain ejection counters, ArtifactRef, replay marker, and degradation telemetry |
60
- | `compose-report.mjs` | Operator-readable diagnostics showing layer counts and budget decisions |
61
-
62
- ---
63
-
64
- ## Interpreting Healthy Output
65
-
66
- A passing run shows all checks green:
67
-
68
- ```
69
- ALL 12 CHECKS PASSED ✅
70
- ```
71
-
72
- Key diagnostics in the compose report:
73
-
74
- - **factsIncluded > 0**: facts were retrieved for the prompt
75
- - **tokenCount <= budget**: compositor respected the token ceiling
76
- - **retrievalMode**: `trigger`, `fallback_knn`, or `fts_only` — shows which retrieval path fired
77
- - **scopeFiltered >= 0**: cross-session facts correctly filtered
78
-
79
- Maintenance diagnostics (when `verboseLogging` is enabled):
80
-
81
- ```
82
- [indexer] Maintenance: considered=5 skipped=2 scanned=3 mutated=0 duration=12ms exit=complete
83
- ```
84
-
85
- - **considered**: conversations examined
86
- - **skipped**: conversations within cooldown window
87
- - **scanned**: conversations where sweeps ran
88
- - **mutated**: total messages deleted or truncated
89
- - **exit**: `complete`, `cap-reached`, `cooldown`, or `no-conversations`
90
-
91
- ---
92
-
93
- ## Common Cases
94
-
95
- ### Empty context (no facts/knowledge seeded)
96
-
97
- Expected: `factsIncluded=0`, `contextBlock` may be empty or contain only history. This is normal for fresh installs or agents with no indexed conversations.
98
-
99
- ### Missing fact in context
100
-
101
- Check: is the fact's `superseded_by` column NULL? Superseded facts are filtered. Is the fact's `agent_id` correct for the composing agent? Cross-agent facts are scope-filtered.
102
-
103
- ### Over-budget compose
104
-
105
- The compositor respects token budgets with a small tolerance (5-15%). If `tokenCount` significantly exceeds `tokenBudget`, check `compositor.budgetFraction` and `contextWindowReserve` in config.
106
-
107
- ### Maintenance not running
108
-
109
- Check that `maintenance.periodicInterval` is set in config.json. Default is 300000ms (5 min). If `verboseLogging` is enabled, you should see maintenance diagnostics every tick.
110
-
111
- ---
112
-
113
- ## What Remains Unverified After Phase 1
114
-
115
- - **Vector/semantic retrieval**: all Phase 1 tests run FTS-only (no Ollama dependency)
116
- - **Full hot-cache plugin path inside Phase 1 itself**: these checks use the direct compositor, not the full cache-layer assemble lifecycle. Use `npm run validate:release-path` for that proof.
117
- - **Multi-agent fleet interactions**: scope isolation is tested, but fleet-wide maintenance behavior is not
118
- - **Provider-specific formatting**: tests use `provider: 'anthropic'` only
119
- - **Real model token counting**: tests use the char/4 heuristic estimator, not tiktoken
120
-
121
- ---
122
-
123
- ## Maintenance Tuning Reference
124
-
125
- See [TUNING.md](./TUNING.md#background-maintenance) for the full knob reference. Key defaults:
126
-
127
- | Setting | Default | Effect |
128
- |---|---|---|
129
- | `maintenance.periodicInterval` | 300000ms | Background tick cadence |
130
- | `maintenance.maxActiveConversations` | 5 | Conversations processed per agent per tick |
131
- | `maintenance.recentConversationCooldownMs` | 30000ms | Skip recently processed conversations |
132
- | `maintenance.maxCandidatesPerPass` | 200 | Cap on mutations per tick |
@@ -1,70 +0,0 @@
1
- # HyperMem 0.8.0 Release Path Validation
2
-
3
- This is the operator runbook for the release hardening harness.
4
-
5
- ## Command
6
-
7
- ```bash
8
- npm run validate:release-path
9
- ```
10
-
11
- That command builds core + plugin, then runs `test/release-gateway-path.mjs` against the built plugin dist in an isolated temporary HOME.
12
-
13
- ## What it proves
14
-
15
- The harness exercises the real context-engine plugin path, not just direct compositor helpers.
16
-
17
- It verifies four release-path behaviors in one run:
18
-
19
- 1. **normal compose path** returns assembled context through `engine.assemble()`
20
- 2. **artifact degradation** emits a canonical `[artifact:...]` reference in system context
21
- 3. **tool-chain ejection** is recorded through real compose-path co-ejection counters in telemetry
22
- 4. **tool-loop replay recovery** emits the canonical `[replay state=entering ...]` marker when runtime history is hot and the hot cache is cold
23
-
24
- ## Telemetry contract
25
-
26
- When `HYPERMEM_TELEMETRY=1`, the plugin now emits a `degradation` JSONL event alongside the existing `assemble`, `trim`, and `trim-guard` events.
27
-
28
- Per event fields:
29
-
30
- - `agentId`
31
- - `sessionKey`
32
- - `turnId`
33
- - `path` (`compose` or `toolLoop`)
34
- - `toolChainCoEjections`
35
- - `toolChainStubReplacements`
36
- - `artifactDegradations`
37
- - `artifactOversizeThresholdTokens`
38
- - `replayState`
39
- - `replayReason` (legacy machine reason strings may still contain `redis` for compatibility)
40
-
41
- The release harness asserts those counters against prompt-visible behavior, so the telemetry is not just emitted, it is verified.
42
-
43
- ## Inspecting artifacts manually
44
-
45
- By default the harness deletes its temp HOME on success.
46
-
47
- To keep the temp workspace and telemetry file:
48
-
49
- ```bash
50
- HYPERMEM_KEEP_RELEASE_TMP=1 npm run validate:release-path
51
- ```
52
-
53
- The script will print the preserved temp path. The telemetry file lives at:
54
-
55
- ```text
56
- <tmp>/release-telemetry.jsonl
57
- ```
58
-
59
- ## Healthy result
60
-
61
- ```text
62
- ALL 12 CHECKS PASSED ✅
63
- ```
64
-
65
- A healthy run means the built plugin can prove the Phase C prompt-path contracts that matter for `0.8.0`:
66
-
67
- - degraded content uses the canonical visible shapes where it is prompt-visible
68
- - degradation counters line up with what entered the model path
69
- - replay recovery is visible at the plugin boundary
70
- - the proof runs against the real assemble lifecycle, not a mocked helper only
@@ -1,10 +0,0 @@
1
- # HyperMem Release Process
2
-
3
- **Canonical source:** [PsiClawOps/publication_process_internal](https://github.com/PsiClawOps/publication_process_internal)
4
-
5
- - **Universal process:** `PROCESS.md` (5-stage publication pipeline)
6
- - **HyperMem-specific:** `repos/hypermem.md` (scrub lists, stubs, build checks)
7
- - **Install verification:** `INSTALL_VERIFICATION.md` (independent vetting gate)
8
- - **Style:** `STYLE_GUIDE.md` + `FLEET_OUTPUT.md`
9
-
10
- Do not duplicate process content here. If something is missing from the canonical repo, add it there.