@psiclawops/hypermem 0.9.3 → 0.9.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +27 -2
- package/INSTALL.md +75 -68
- package/README.md +18 -36
- package/assets/default-config.json +41 -0
- package/bench/data-access-bench.mjs +1 -1
- package/bin/hypermem-doctor.mjs +76 -2
- package/bin/hypermem-status.mjs +255 -7
- package/dist/adaptive-lifecycle.d.ts +39 -0
- package/dist/adaptive-lifecycle.d.ts.map +1 -1
- package/dist/adaptive-lifecycle.js +87 -9
- package/dist/background-indexer.d.ts.map +1 -1
- package/dist/background-indexer.js +16 -14
- package/dist/compositor.d.ts.map +1 -1
- package/dist/compositor.js +239 -20
- package/dist/cross-agent.d.ts +1 -1
- package/dist/cross-agent.js +17 -17
- package/dist/hybrid-retrieval.d.ts +8 -0
- package/dist/hybrid-retrieval.d.ts.map +1 -1
- package/dist/hybrid-retrieval.js +112 -10
- package/dist/index.d.ts +16 -5
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +19 -4
- package/dist/message-store.d.ts +62 -1
- package/dist/message-store.d.ts.map +1 -1
- package/dist/message-store.js +355 -2
- package/dist/open-domain.d.ts.map +1 -1
- package/dist/open-domain.js +3 -2
- package/dist/proactive-pass.d.ts +42 -2
- package/dist/proactive-pass.d.ts.map +1 -1
- package/dist/proactive-pass.js +294 -39
- package/dist/seed.d.ts +1 -1
- package/dist/seed.js +1 -1
- package/dist/session-flusher.d.ts +2 -2
- package/dist/session-flusher.js +2 -2
- package/dist/spawn-context.d.ts +1 -1
- package/dist/spawn-context.js +1 -1
- package/dist/topic-store.js +5 -5
- package/dist/topic-synthesizer.d.ts.map +1 -1
- package/dist/topic-synthesizer.js +10 -4
- package/dist/trigger-registry.d.ts +1 -1
- package/dist/trigger-registry.js +4 -4
- package/dist/types.d.ts +101 -2
- package/dist/types.d.ts.map +1 -1
- package/dist/vector-store.d.ts +10 -1
- package/dist/vector-store.d.ts.map +1 -1
- package/dist/vector-store.js +45 -9
- package/docs/DIAGNOSTICS.md +88 -1
- package/docs/INTEGRATION_VALIDATION.md +40 -1
- package/docs/MIGRATION.md +1 -1
- package/docs/TUNING.md +47 -6
- package/install.sh +5 -60
- package/memory-plugin/dist/index.js +192 -0
- package/memory-plugin/openclaw.plugin.json +199 -2
- package/memory-plugin/package.json +2 -2
- package/package.json +29 -10
- package/plugin/dist/index.d.ts +2 -0
- package/plugin/dist/index.d.ts.map +1 -1
- package/plugin/dist/index.js +178 -11
- package/plugin/dist/index.js.map +1 -1
- package/plugin/openclaw.plugin.json +199 -2
- package/plugin/package.json +2 -2
- package/scripts/install-packed-runtime.mjs +99 -0
- package/scripts/install-runtime.mjs +164 -4
- package/ARCHITECTURE.md +0 -298
- package/docs/KNOWN_LIMITATIONS.md +0 -35
- package/docs/PHASE1-VALIDATION.md +0 -132
- package/docs/RELEASE_0.8.0_VALIDATION.md +0 -70
- package/docs/RELEASE_PROCESS.md +0 -10
- package/docs/ROADMAP.md +0 -266
package/ARCHITECTURE.md
DELETED
|
@@ -1,298 +0,0 @@
|
|
|
1
|
-
# hypermem Architecture
|
|
2
|
-
|
|
3
|
-
_Agent-centric memory that outlives sessions, backed by SQLite memory databases._
|
|
4
|
-
|
|
5
|
-
---
|
|
6
|
-
|
|
7
|
-
## Memory Layers
|
|
8
|
-
|
|
9
|
-
```
|
|
10
|
-
L1 SQLite Cache (Hot) Active session working memory
|
|
11
|
-
│ Slots: system, identity, messages, facts, context
|
|
12
|
-
│ Sub-millisecond reads, evicts on session end
|
|
13
|
-
│ Fleet cache: agent profiles, fleet summary
|
|
14
|
-
│
|
|
15
|
-
L2 Agent Messages DB Raw conversation history (per agent)
|
|
16
|
-
│ messages.db Write-heavy, rotatable (100MB / 90 days)
|
|
17
|
-
│ messages_YYYYQN.db Rotated archives, read-only
|
|
18
|
-
│
|
|
19
|
-
L3 Agent Vectors DB Semantic search index (per agent)
|
|
20
|
-
│ vectors.db Index rebuilt from L2+L4 (reconstructable)
|
|
21
|
-
│ KNN search via sqlite-vec (nomic-embed-text, 768d)
|
|
22
|
-
│ Rate-limited embedding via token bucket
|
|
23
|
-
│
|
|
24
|
-
L4 Library DB Fleet-wide structured knowledge
|
|
25
|
-
library.db Facts, knowledge, preferences, fleet registry
|
|
26
|
-
Episodes, topics, work items, system state
|
|
27
|
-
Desired state, agent capabilities
|
|
28
|
-
Knowledge graph (DAG links between entities)
|
|
29
|
-
```
|
|
30
|
-
|
|
31
|
-
> Note: some internal method names and telemetry reasons still contain `redis`
|
|
32
|
-
> for backward compatibility. The runtime hot layer is SQLite `:memory:` cache,
|
|
33
|
-
> not an external Redis service.
|
|
34
|
-
|
|
35
|
-
## Database Schema
|
|
36
|
-
|
|
37
|
-
### messages.db (per agent, schema v10)
|
|
38
|
-
- `agent_meta` — agent metadata
|
|
39
|
-
- `conversations` — session tracking
|
|
40
|
-
- `messages` — raw message log (text, tool calls, tool results)
|
|
41
|
-
- `schema_version` — migration tracking
|
|
42
|
-
- **Rotation:** When size > 100MB or age > 90 days, renamed to `messages_YYYYQN.db`
|
|
43
|
-
- **WAL mode** with checkpoint on rotation
|
|
44
|
-
|
|
45
|
-
### vectors.db (per agent)
|
|
46
|
-
- `vec_facts` — 768-dimensional vectors for facts (sqlite-vec virtual table)
|
|
47
|
-
- `vec_knowledge` — vectors for knowledge entries
|
|
48
|
-
- `vec_episodes` — vectors for episode descriptions
|
|
49
|
-
- `vec_sessions` — vectors for session summaries
|
|
50
|
-
- `vec_index_map` — tracks what's been indexed (source_table, source_id, source_db)
|
|
51
|
-
- `embedding_cache` — avoids redundant Ollama API calls
|
|
52
|
-
|
|
53
|
-
### library.db (shared, schema v19)
|
|
54
|
-
- `facts` — verifiable claims with confidence, domain, expiry, supersedes chains
|
|
55
|
-
- `knowledge` — domain/key/value structured data
|
|
56
|
-
- `knowledge_links` — DAG edges between entities (fact↔fact, fact↔knowledge, etc.)
|
|
57
|
-
- `episodes` — significant events with impact and participants
|
|
58
|
-
- `topics` — cross-session thread tracking
|
|
59
|
-
- `preferences` — operator/user behavioral patterns
|
|
60
|
-
- `fleet_agents` — agent registry with tier, org, capabilities (JSON)
|
|
61
|
-
- `fleet_orgs` — organizational structure
|
|
62
|
-
- `agent_capabilities` — queryable skills, tools, MCP servers per agent
|
|
63
|
-
- `agent_desired_state` — intended configuration vs. actual (drift detection)
|
|
64
|
-
- `agent_config_events` — change audit trail
|
|
65
|
-
- `system_registry` — service state tracking
|
|
66
|
-
- `system_events` — service lifecycle events
|
|
67
|
-
- `work_items` — work queue entries with FTS5
|
|
68
|
-
- `session_registry` — session lifecycle tracking
|
|
69
|
-
|
|
70
|
-
## Compositor
|
|
71
|
-
|
|
72
|
-
Assembles LLM prompts from all four layers with token budgeting:
|
|
73
|
-
|
|
74
|
-
```
|
|
75
|
-
User message arrives
|
|
76
|
-
│
|
|
77
|
-
├── L1 Hot cache: system prompt, identity, cached slots
|
|
78
|
-
├── L2 Messages: recent conversation history (budget-truncated)
|
|
79
|
-
├── L3 Vectors: KNN semantic recall on user's latest message
|
|
80
|
-
│ └── Related facts/knowledge/episodes with relevance scores
|
|
81
|
-
├── L4 Library: structured knowledge injection
|
|
82
|
-
│ ├── Facts (up to 30% of remaining budget): active, non-expired, sorted by confidence
|
|
83
|
-
│ ├── Knowledge (up to 20% of remaining): grouped by domain, top 15 entries
|
|
84
|
-
│ └── Preferences (up to 10% of remaining): behavioral patterns, grouped by subject
|
|
85
|
-
└── Cross-session (up to 20% of remaining): context from related sessions
|
|
86
|
-
```
|
|
87
|
-
|
|
88
|
-
Each slot gets a proportional budget cap. Smart truncation at line boundaries.
|
|
89
|
-
Multi-provider output: Anthropic and OpenAI message formats.
|
|
90
|
-
|
|
91
|
-
### Tuning Parameters (TUNE-001–007)
|
|
92
|
-
|
|
93
|
-
Compositor behavior is tuned via parameters tracked in `tune/TUNING_REGISTRY.md`:
|
|
94
|
-
|
|
95
|
-
| ID | Parameter | Value | Effect |
|
|
96
|
-
|---|---|---|---|
|
|
97
|
-
| TUNE-001 | Semantic recall min RRF score | 0.008 | Drops noise results from hybrid search |
|
|
98
|
-
| TUNE-002 | Facts confidence floor | 0.5 | Excludes low-confidence facts from injection |
|
|
99
|
-
| TUNE-003 | Differentiated fact confidence | 0.60–0.75 by type | Decisions/incidents score higher than config/prefs |
|
|
100
|
-
| TUNE-004 | config_change episode significance | 0.5 (was 0.4) | Config changes no longer silently dropped |
|
|
101
|
-
| TUNE-005 | Extraction slot guard | suppress on default | No-op when strategy is lightweight/default |
|
|
102
|
-
| TUNE-006 | Advisor slot guard | suppress bare seat list | No-op when no domain routes matched |
|
|
103
|
-
| TUNE-007 | Identity anchor guard | suppress on default identity | No-op when identity resolves to 'default' |
|
|
104
|
-
|
|
105
|
-
### Safety Mechanisms
|
|
106
|
-
|
|
107
|
-
- **Budget safety valve:** Post-assembly check — if estimated tokens exceed budget × 1.05, trims oldest history messages until under budget. System/identity/current prompt are never touched.
|
|
108
|
-
- **Compaction fence:** Per-conversation boundary protecting the LLM's recent tail from compaction. Only moves forward (monotone progress). No fence = no compaction (explicit opt-in).
|
|
109
|
-
- **Preservation gate:** Nomic-space geometric verification that summaries stay faithful to source content. Centroid alignment + source coverage → combined score (threshold: 0.65).
|
|
110
|
-
|
|
111
|
-
## Fleet Cache (Hot Cache Layer)
|
|
112
|
-
|
|
113
|
-
```
|
|
114
|
-
fleet:agent:{id} — Composite profile: registry + capabilities + desired state
|
|
115
|
-
fleet:summary — Fleet-wide stats: agent count, drift count, tier breakdown
|
|
116
|
-
```
|
|
117
|
-
|
|
118
|
-
- **Cache-aside** on reads: hot cache first, SQLite fallback, warm on miss
|
|
119
|
-
- **Write-through invalidation** on fleet mutations
|
|
120
|
-
- **Hydration** on gateway startup: bulk-populate from library.db
|
|
121
|
-
- TTL: agent profiles 10min, summary 2min
|
|
122
|
-
|
|
123
|
-
## Knowledge Graph
|
|
124
|
-
|
|
125
|
-
DAG traversal over `knowledge_links` for relationship discovery:
|
|
126
|
-
|
|
127
|
-
- **Entity types:** fact, knowledge, topic, episode, agent, preference
|
|
128
|
-
- **Link types:** supports, contradicts, supersedes, references, derived_from, depends_on, extends, covers, related, authored_by
|
|
129
|
-
- **BFS traversal:** configurable depth, result limit, direction filter, type filter
|
|
130
|
-
- **Shortest path:** between any two entities
|
|
131
|
-
- **Analytics:** most-connected entities, link count by type
|
|
132
|
-
|
|
133
|
-
## Rate Limiter
|
|
134
|
-
|
|
135
|
-
Token-bucket rate limiter for embedding API calls:
|
|
136
|
-
|
|
137
|
-
- Burst capacity with steady refill (default: 5/s, burst 10)
|
|
138
|
-
- Priority queue: high (user-facing recall) > normal > low (batch indexing)
|
|
139
|
-
- Reserved tokens for high-priority requests
|
|
140
|
-
- `createRateLimitedEmbedder()` wraps any embedding function
|
|
141
|
-
|
|
142
|
-
## Cross-Agent Access Control
|
|
143
|
-
|
|
144
|
-
Visibility-tiered access model for cross-agent knowledge queries:
|
|
145
|
-
|
|
146
|
-
- **`fleet`** — visible to all agents (default tier)
|
|
147
|
-
- **`org`** — visible to agents in the same organizational unit
|
|
148
|
-
- **`council`** — visible to council-tier agents and the org lead
|
|
149
|
-
- **`private`** — visible only to the owning agent
|
|
150
|
-
|
|
151
|
-
### Org Registry
|
|
152
|
-
|
|
153
|
-
`visibilityFilter()` resolves access levels using an `OrgRegistry` — a mapping of agents to tiers, orgs, and capabilities. Currently loaded from a hardcoded `defaultOrgRegistry()` in `cross-agent.ts`.
|
|
154
|
-
|
|
155
|
-
**Known limitation:** `defaultOrgRegistry()` duplicates fleet structure that lives authoritatively in `fleet_agents` + `fleet_orgs` in library.db. The planned migration is to live-load the registry from library.db on startup, with the hardcoded registry retained only as a cold-start fallback.
|
|
156
|
-
|
|
157
|
-
### Unknown Agent Fallback (Restrictive Default)
|
|
158
|
-
|
|
159
|
-
When a target agent is not found in the registry, `visibilityFilter()` applies a **restrictive default**: fleet-only visibility with a logged warning. This is a deliberate safety-side behavior — unknown agents see only fleet-visible data rather than failing the query entirely. The warning surfaces registry gaps for operators to fix.
|
|
160
|
-
|
|
161
|
-
This means:
|
|
162
|
-
- Queries succeed but return a narrowed result set
|
|
163
|
-
- New agents provisioned in the DB but not yet in the registry will have reduced cross-agent visibility until registered
|
|
164
|
-
- The warning message names the missing agent and suggests adding it to the registry
|
|
165
|
-
|
|
166
|
-
## Context Engine Plugin
|
|
167
|
-
|
|
168
|
-
`plugin/src/index.ts` — OpenClaw context engine plugin (`hypercompositor`, fills `contextEngine` slot):
|
|
169
|
-
|
|
170
|
-
```
|
|
171
|
-
gateway:startup → Init hypermem, auto-rotate DBs, seed fleet registry from workspace identities, hydrate fleet cache
|
|
172
|
-
agent:bootstrap → Warm session (history, facts, profile → hot cache)
|
|
173
|
-
context:assemble → Full four-layer prompt assembly within token budget
|
|
174
|
-
agent:afterTurn → Ingest new messages to SQLite + hot cache, trigger background indexer
|
|
175
|
-
```
|
|
176
|
-
|
|
177
|
-
Registers with `ownsCompaction: true` — runtime skips legacy compaction entirely.
|
|
178
|
-
|
|
179
|
-
## Memory Plugin
|
|
180
|
-
|
|
181
|
-
`memory-plugin/src/index.ts` — Lightweight memory provider (`hypermem`, fills `memory` slot):
|
|
182
|
-
|
|
183
|
-
- Registers `MemoryPluginCapability` with a `MemorySearchManager` backed by HyperMem's hybrid FTS5 + KNN retrieval
|
|
184
|
-
- Provides the `memory_search` tool through the official memory slot interface
|
|
185
|
-
- Public artifacts provider lists `MEMORY.md` and `memory/*.md` for all configured agents
|
|
186
|
-
- Stateless wrapper: lifecycle is owned by the context engine plugin
|
|
187
|
-
|
|
188
|
-
### Plugin Data Flow
|
|
189
|
-
|
|
190
|
-
```
|
|
191
|
-
┌──────────────────────────────────────────────────┐
|
|
192
|
-
│ HOT CACHE (SQLite :memory: CacheLayer) │
|
|
193
|
-
│ │
|
|
194
|
-
│ hm:{a}:{s}:history ── Session archive (250 cap │
|
|
195
|
-
│ (append-only) at bootstrap, 1000 soft │
|
|
196
|
-
│ cap ongoing) │
|
|
197
|
-
│ │
|
|
198
|
-
│ hm:{a}:{s}:window ── Submission buffer │
|
|
199
|
-
│ (compositor output) (120s TTL) │
|
|
200
|
-
│ │
|
|
201
|
-
│ hm:{a}:{s}:cursor ── Last-sent pointer │
|
|
202
|
-
│ (compositor metadata) (24h TTL) │
|
|
203
|
-
│ │
|
|
204
|
-
│ hm:{a}:{s}:system ── System prompt slot │
|
|
205
|
-
│ hm:{a}:{s}:identity ── Identity slot │
|
|
206
|
-
│ hm:{a}:{s}:facts ── Cached facts slot │
|
|
207
|
-
│ hm:{a}:{s}:context ── Cross-session slot │
|
|
208
|
-
└──────────────────────────────────────────────────┘
|
|
209
|
-
|
|
210
|
-
Data Flow (current — P0 stabilized, window/cursor active):
|
|
211
|
-
|
|
212
|
-
bootstrap() assemble() afterTurn()
|
|
213
|
-
─────────── ────────── ───────────
|
|
214
|
-
▸ sessionExists() → skip if hot compose() slice(prePromptCount)
|
|
215
|
-
▸ SQLite ─→ warmSession() ─→ getHistory(limit) ✅ ─→ record*Message()
|
|
216
|
-
─→ pushHistory(250) ─→ dedup by id ─→ pushHistory(1, dedup)
|
|
217
|
-
─→ cache history ─→ budget assembly ─→ cache history
|
|
218
|
-
─→ write window bundle ─→ invalidateWindow()
|
|
219
|
-
─→ write cursor metadata ─→ background indexer
|
|
220
|
-
─→ → runtime → provider
|
|
221
|
-
|
|
222
|
-
### Key Invariants
|
|
223
|
-
|
|
224
|
-
1. Hot-cache `history` is the warm archive. Append-only. Nothing reads it for direct submission.
|
|
225
|
-
2. Hot-cache `window` is the compositor's output cache. Written ONLY by `compose()`. Read ONLY by `assemble()`. Invalidated by `afterTurn`.
|
|
226
|
-
3. Hot-cache `cursor` tracks the newest message in the last window. Used by background indexer for high-signal mining.
|
|
227
|
-
4. `warmSession()` seeds `history` only (capped at 250). Never writes `window`.
|
|
228
|
-
5. `pushHistory()` tail-checks before append (no duplicate IDs in the hot-cache history list).
|
|
229
|
-
6. `compose()` deduplicates history by `id` before budget assembly.
|
|
230
|
-
7. `getHistory()` honors its `limit` parameter on BOTH hot-cache and SQLite paths.
|
|
231
|
-
|
|
232
|
-
Open and deferred work is tracked outside this public architecture reference.
|
|
233
|
-
|
|
234
|
-
### Runtime Contract
|
|
235
|
-
|
|
236
|
-
**Exclusive dispatch:** The OpenClaw runtime calls either `afterTurn()` OR `ingest()`/`ingestBatch()`, never both. Since hypermem implements `afterTurn`, it must handle message ingestion there. `ingest()` exists for API compatibility but is never called by the runtime in practice.
|
|
237
|
-
|
|
238
|
-
**Provider translation:** The plugin sets `skipProviderTranslation: true` on compose requests. The compositor returns NeutralMessages; the plugin converts to AgentMessages. The runtime handles provider-specific translation. Two-stage translation (compositor → provider format → plugin → agent format) was the root cause of Incident 1 (silent tool call drops).
|
|
239
|
-
|
|
240
|
-
## Module Map (29 files, ~12,300 lines)
|
|
241
|
-
|
|
242
|
-
| Module | Lines | Layer | Purpose |
|
|
243
|
-
|---|---|---|---|
|
|
244
|
-
| `index.ts` | ~1,340 | All | Facade — all public API |
|
|
245
|
-
| `compositor.ts` | ~1,140 | L1-L4 | Prompt assembly + token budgeting + safety valve + window/cursor write |
|
|
246
|
-
| `library-schema.ts` | ~780 | L4 | Library schema v19 + migrations |
|
|
247
|
-
| `background-indexer.ts` | ~680 | L2-L4 | LLM-powered extraction framework |
|
|
248
|
-
| `vector-store.ts` | ~600 | L3 | Semantic search + embedding |
|
|
249
|
-
| `hybrid-retrieval.ts` | ~450 | L3-L4 | FTS5 + KNN with Reciprocal Rank Fusion |
|
|
250
|
-
| `fleet-store.ts` | ~440 | L4 | Fleet registry + capabilities |
|
|
251
|
-
| `db.ts` | ~440 | - | Database manager + rotation |
|
|
252
|
-
| `knowledge-graph.ts` | ~420 | L4 | DAG traversal + shortest path |
|
|
253
|
-
| `cache.ts` | ~700 | L1 | SQLite `:memory:` hot-cache operations, window cache, cursor, fleet cache |
|
|
254
|
-
| `doc-chunker.ts` | ~400 | - | Section-aware markdown/file parser |
|
|
255
|
-
| `work-store.ts` | ~400 | L4 | Work queue + FTS5 |
|
|
256
|
-
| `provider-translator.ts` | ~390 | - | Neutral ↔ provider format conversion |
|
|
257
|
-
| `doc-chunk-store.ts` | ~375 | L4 | Chunk storage + deduplication |
|
|
258
|
-
| `message-store.ts` | ~370 | L2 | Conversation recording + querying |
|
|
259
|
-
| `types.ts` | ~370 | - | Shared type definitions + SessionCursor |
|
|
260
|
-
| `cross-agent.ts` | ~330 | L2-L4 | Cross-agent knowledge queries + visibility |
|
|
261
|
-
| `desired-state-store.ts` | ~310 | L4 | Config drift detection |
|
|
262
|
-
| `knowledge-store.ts` | ~300 | L4 | Domain/key/value structured data |
|
|
263
|
-
| `secret-scanner.ts` | ~285 | - | Credential/secret detection |
|
|
264
|
-
| `system-store.ts` | ~250 | L4 | Service state tracking |
|
|
265
|
-
| `seed.ts` | ~250 | L4 | Workspace seeder + collection inference |
|
|
266
|
-
| `fact-store.ts` | ~230 | L4 | Facts with confidence + expiry |
|
|
267
|
-
| `rate-limiter.ts` | ~230 | L3 | Token-bucket for embedding API |
|
|
268
|
-
| `schema.ts` | ~200 | L2 | Messages schema + migrations |
|
|
269
|
-
| `episode-store.ts` | ~180 | L4 | Significant event tracking |
|
|
270
|
-
| `preference-store.ts` | ~170 | L4 | Operator behavioral patterns |
|
|
271
|
-
| `topic-store.ts` | ~160 | L4 | Cross-session thread tracking |
|
|
272
|
-
| `plugin/src/index.ts` | ~590 | - | `hypercompositor` context engine plugin + window invalidation |
|
|
273
|
-
| `memory-plugin/src/index.ts` | ~290 | - | `hypermem` memory slot plugin (memory_search via hybrid retrieval) |
|
|
274
|
-
|
|
275
|
-
## Test Coverage (105 assertions, 11 suites)
|
|
276
|
-
|
|
277
|
-
_Test count reflects assertions, not individual test blocks. Suites contain inline assertions._
|
|
278
|
-
|
|
279
|
-
| Suite | Key coverage |
|
|
280
|
-
|---|---|
|
|
281
|
-
| smoke | End-to-end create/write/read/close, provider translation |
|
|
282
|
-
| redis-integration | Legacy suite name, covers hot-cache ops, slots, history limits, window cache, cursor, warming, dedup |
|
|
283
|
-
| cross-agent | Cross-agent queries, fleet search, visibility tiers |
|
|
284
|
-
| vector-search | Embedding, KNN, batch indexing |
|
|
285
|
-
| library | All L4 collections (facts → desired state) |
|
|
286
|
-
| compositor | Four-layer composition, budgets, providers, safety valve, Gate 1 |
|
|
287
|
-
| fleet-cache | Fleet hot-cache hydration and cache-aside behavior |
|
|
288
|
-
| rotation | DB rotation, auto-rotate, collision handling |
|
|
289
|
-
| knowledge-graph | DAG traversal, shortest path, analytics |
|
|
290
|
-
| rate-limiter | Token bucket, priority, timeout, embedder |
|
|
291
|
-
| doc-chunker | Markdown/file chunking, section-aware parsing, seeder |
|
|
292
|
-
|
|
293
|
-
## Dependencies
|
|
294
|
-
|
|
295
|
-
- `node:sqlite` (Node 22+ built-in) — zero-dependency SQLite
|
|
296
|
-
- No external cache service dependency — hot cache is SQLite `:memory:`
|
|
297
|
-
- `sqlite-vec` — optional, vector search extension
|
|
298
|
-
- Ollama (localhost:11434) — optional, embedding generation
|
|
@@ -1,35 +0,0 @@
|
|
|
1
|
-
# Known Limitations — HyperMem 0.8.0
|
|
2
|
-
|
|
3
|
-
## Global-scope fact write authorization
|
|
4
|
-
|
|
5
|
-
Facts written with `scope='global'` are readable fleet-wide at L4 priority. The write path has no authorization gate — any agent with access to the HyperMem API can write a global-scope fact.
|
|
6
|
-
|
|
7
|
-
**Impact:** In a trusted single-operator deployment (the intended target), this is acceptable. All agents are operator-controlled and share the same trust boundary.
|
|
8
|
-
|
|
9
|
-
**Planned fix:** Introduce a write-authority model that gates global-scope writes to designated agents (council seats or explicitly allowlisted agent IDs). See [docs/ROADMAP.md](ROADMAP.md).
|
|
10
|
-
|
|
11
|
-
**Workaround:** Restrict HyperMem API access to trusted agents only. Do not expose the API to untrusted external agents.
|
|
12
|
-
|
|
13
|
-
## Cross-agent org registry is hardcoded
|
|
14
|
-
|
|
15
|
-
`visibilityFilter()` in `cross-agent.ts` resolves agent tiers and visibility from a hardcoded `defaultOrgRegistry()`. This duplicates fleet topology that lives authoritatively in `fleet_agents` + `fleet_orgs` in library.db.
|
|
16
|
-
|
|
17
|
-
**Impact:** New agents added to library.db but not the hardcoded registry get fleet-only visibility until the registry is updated in code.
|
|
18
|
-
|
|
19
|
-
**Planned fix:** Live-load registry from library.db on startup, hardcoded as cold-start fallback only. See [docs/ROADMAP.md](ROADMAP.md).
|
|
20
|
-
|
|
21
|
-
## Cursor durability across restarts
|
|
22
|
-
|
|
23
|
-
The compositor writes a session cursor to hot cache (SQLite `:memory:`) with a 24h TTL. On gateway restart, the cursor is lost until the next `compose()` call repopulates it.
|
|
24
|
-
|
|
25
|
-
**Impact:** Background indexer may miss the cursor on the first turn after a restart, falling back to full-history scan.
|
|
26
|
-
|
|
27
|
-
**Planned fix:** Dual-write cursor to messages.db so it survives restarts. See [docs/ROADMAP.md](ROADMAP.md).
|
|
28
|
-
|
|
29
|
-
## Cross-session context has no boundary markers
|
|
30
|
-
|
|
31
|
-
`buildCrossSessionContext()` renders flat previews with no per-message boundaries or sender identity labels.
|
|
32
|
-
|
|
33
|
-
**Impact:** Context from different sessions blends together, making attribution ambiguous in multi-agent scenarios.
|
|
34
|
-
|
|
35
|
-
**Planned fix:** WQ-20260402-001. See [docs/ROADMAP.md](ROADMAP.md).
|
|
@@ -1,132 +0,0 @@
|
|
|
1
|
-
# HyperMem Phase 1 Validation Runbook
|
|
2
|
-
|
|
3
|
-
Operator-facing guide for running and interpreting the Phase 1 validation suite.
|
|
4
|
-
|
|
5
|
-
---
|
|
6
|
-
|
|
7
|
-
## Quick Start
|
|
8
|
-
|
|
9
|
-
Run the full Phase 1 validation flow:
|
|
10
|
-
|
|
11
|
-
```bash
|
|
12
|
-
npm run build && node scripts/validate-compose.mjs && node scripts/validate-config-surface.mjs
|
|
13
|
-
```
|
|
14
|
-
|
|
15
|
-
Or run individual validations:
|
|
16
|
-
|
|
17
|
-
```bash
|
|
18
|
-
# Compose validation (facts, library, budget pressure)
|
|
19
|
-
node scripts/validate-compose.mjs
|
|
20
|
-
|
|
21
|
-
# Config surface parity (install.sh, docs, README)
|
|
22
|
-
node scripts/validate-config-surface.mjs
|
|
23
|
-
|
|
24
|
-
# Config key resolution tests
|
|
25
|
-
node test/config-validation.mjs
|
|
26
|
-
|
|
27
|
-
# Retrieval regression harness
|
|
28
|
-
node test/retrieval-regression.mjs
|
|
29
|
-
|
|
30
|
-
# Compositor integration (includes budget-pressure fixture)
|
|
31
|
-
node test/compositor.mjs
|
|
32
|
-
|
|
33
|
-
# Plugin pipeline (requires plugin build)
|
|
34
|
-
npm run validate:plugin-pipeline
|
|
35
|
-
|
|
36
|
-
# Startup fleet seeding on cold boot
|
|
37
|
-
node test/fleet-startup-seeding.mjs
|
|
38
|
-
|
|
39
|
-
# Release path hardening harness (builds core + plugin)
|
|
40
|
-
npm run validate:release-path
|
|
41
|
-
|
|
42
|
-
# Compose report (operator-readable diagnostics)
|
|
43
|
-
node scripts/compose-report.mjs
|
|
44
|
-
```
|
|
45
|
-
|
|
46
|
-
---
|
|
47
|
-
|
|
48
|
-
## What Each Validation Covers
|
|
49
|
-
|
|
50
|
-
| Validation | What it proves |
|
|
51
|
-
|---|---|
|
|
52
|
-
| `validate-compose.mjs` | End-to-end compose with seeded facts, knowledge retrieval, and budget pressure |
|
|
53
|
-
| `validate-config-surface.mjs` | Config keys present in install.sh, INSTALL.md, TUNING.md, README |
|
|
54
|
-
| `config-validation.mjs` | contextWindowOverrides sanitization, budget resolution, maintenance defaults |
|
|
55
|
-
| `retrieval-regression.mjs` | Scope isolation, superseded-fact filtering, budget pressure, knowledge retrieval |
|
|
56
|
-
| `compositor.mjs` | Four-layer compose, trigger routing, keystone injection, budget-pressure filtering |
|
|
57
|
-
| `plugin-pipeline.mjs` | Real plugin assemble() path with seeded L4 memory, tight-budget proof |
|
|
58
|
-
| `fleet-startup-seeding.mjs` | Cold-start fleet population from workspace identity files plus idempotent repeat-boot behavior |
|
|
59
|
-
| `release-gateway-path.mjs` | Real plugin release-path proof: tool-chain ejection counters, ArtifactRef, replay marker, and degradation telemetry |
|
|
60
|
-
| `compose-report.mjs` | Operator-readable diagnostics showing layer counts and budget decisions |
|
|
61
|
-
|
|
62
|
-
---
|
|
63
|
-
|
|
64
|
-
## Interpreting Healthy Output
|
|
65
|
-
|
|
66
|
-
A passing run shows all checks green:
|
|
67
|
-
|
|
68
|
-
```
|
|
69
|
-
ALL 12 CHECKS PASSED ✅
|
|
70
|
-
```
|
|
71
|
-
|
|
72
|
-
Key diagnostics in the compose report:
|
|
73
|
-
|
|
74
|
-
- **factsIncluded > 0**: facts were retrieved for the prompt
|
|
75
|
-
- **tokenCount <= budget**: compositor respected the token ceiling
|
|
76
|
-
- **retrievalMode**: `trigger`, `fallback_knn`, or `fts_only` — shows which retrieval path fired
|
|
77
|
-
- **scopeFiltered >= 0**: cross-session facts correctly filtered
|
|
78
|
-
|
|
79
|
-
Maintenance diagnostics (when `verboseLogging` is enabled):
|
|
80
|
-
|
|
81
|
-
```
|
|
82
|
-
[indexer] Maintenance: considered=5 skipped=2 scanned=3 mutated=0 duration=12ms exit=complete
|
|
83
|
-
```
|
|
84
|
-
|
|
85
|
-
- **considered**: conversations examined
|
|
86
|
-
- **skipped**: conversations within cooldown window
|
|
87
|
-
- **scanned**: conversations where sweeps ran
|
|
88
|
-
- **mutated**: total messages deleted or truncated
|
|
89
|
-
- **exit**: `complete`, `cap-reached`, `cooldown`, or `no-conversations`
|
|
90
|
-
|
|
91
|
-
---
|
|
92
|
-
|
|
93
|
-
## Common Cases
|
|
94
|
-
|
|
95
|
-
### Empty context (no facts/knowledge seeded)
|
|
96
|
-
|
|
97
|
-
Expected: `factsIncluded=0`, `contextBlock` may be empty or contain only history. This is normal for fresh installs or agents with no indexed conversations.
|
|
98
|
-
|
|
99
|
-
### Missing fact in context
|
|
100
|
-
|
|
101
|
-
Check: is the fact's `superseded_by` column NULL? Superseded facts are filtered. Is the fact's `agent_id` correct for the composing agent? Cross-agent facts are scope-filtered.
|
|
102
|
-
|
|
103
|
-
### Over-budget compose
|
|
104
|
-
|
|
105
|
-
The compositor respects token budgets with a small tolerance (5-15%). If `tokenCount` significantly exceeds `tokenBudget`, check `compositor.budgetFraction` and `contextWindowReserve` in config.
|
|
106
|
-
|
|
107
|
-
### Maintenance not running
|
|
108
|
-
|
|
109
|
-
Check that `maintenance.periodicInterval` is set in config.json. Default is 300000ms (5 min). If `verboseLogging` is enabled, you should see maintenance diagnostics every tick.
|
|
110
|
-
|
|
111
|
-
---
|
|
112
|
-
|
|
113
|
-
## What Remains Unverified After Phase 1
|
|
114
|
-
|
|
115
|
-
- **Vector/semantic retrieval**: all Phase 1 tests run FTS-only (no Ollama dependency)
|
|
116
|
-
- **Full hot-cache plugin path inside Phase 1 itself**: these checks use the direct compositor, not the full cache-layer assemble lifecycle. Use `npm run validate:release-path` for that proof.
|
|
117
|
-
- **Multi-agent fleet interactions**: scope isolation is tested, but fleet-wide maintenance behavior is not
|
|
118
|
-
- **Provider-specific formatting**: tests use `provider: 'anthropic'` only
|
|
119
|
-
- **Real model token counting**: tests use the char/4 heuristic estimator, not tiktoken
|
|
120
|
-
|
|
121
|
-
---
|
|
122
|
-
|
|
123
|
-
## Maintenance Tuning Reference
|
|
124
|
-
|
|
125
|
-
See [TUNING.md](./TUNING.md#background-maintenance) for the full knob reference. Key defaults:
|
|
126
|
-
|
|
127
|
-
| Setting | Default | Effect |
|
|
128
|
-
|---|---|---|
|
|
129
|
-
| `maintenance.periodicInterval` | 300000ms | Background tick cadence |
|
|
130
|
-
| `maintenance.maxActiveConversations` | 5 | Conversations processed per agent per tick |
|
|
131
|
-
| `maintenance.recentConversationCooldownMs` | 30000ms | Skip recently processed conversations |
|
|
132
|
-
| `maintenance.maxCandidatesPerPass` | 200 | Cap on mutations per tick |
|
|
@@ -1,70 +0,0 @@
|
|
|
1
|
-
# HyperMem 0.8.0 Release Path Validation
|
|
2
|
-
|
|
3
|
-
This is the operator runbook for the release hardening harness.
|
|
4
|
-
|
|
5
|
-
## Command
|
|
6
|
-
|
|
7
|
-
```bash
|
|
8
|
-
npm run validate:release-path
|
|
9
|
-
```
|
|
10
|
-
|
|
11
|
-
That command builds core + plugin, then runs `test/release-gateway-path.mjs` against the built plugin dist in an isolated temporary HOME.
|
|
12
|
-
|
|
13
|
-
## What it proves
|
|
14
|
-
|
|
15
|
-
The harness exercises the real context-engine plugin path, not just direct compositor helpers.
|
|
16
|
-
|
|
17
|
-
It verifies four release-path behaviors in one run:
|
|
18
|
-
|
|
19
|
-
1. **normal compose path** returns assembled context through `engine.assemble()`
|
|
20
|
-
2. **artifact degradation** emits a canonical `[artifact:...]` reference in system context
|
|
21
|
-
3. **tool-chain ejection** is recorded through real compose-path co-ejection counters in telemetry
|
|
22
|
-
4. **tool-loop replay recovery** emits the canonical `[replay state=entering ...]` marker when runtime history is hot and the hot cache is cold
|
|
23
|
-
|
|
24
|
-
## Telemetry contract
|
|
25
|
-
|
|
26
|
-
When `HYPERMEM_TELEMETRY=1`, the plugin now emits a `degradation` JSONL event alongside the existing `assemble`, `trim`, and `trim-guard` events.
|
|
27
|
-
|
|
28
|
-
Per event fields:
|
|
29
|
-
|
|
30
|
-
- `agentId`
|
|
31
|
-
- `sessionKey`
|
|
32
|
-
- `turnId`
|
|
33
|
-
- `path` (`compose` or `toolLoop`)
|
|
34
|
-
- `toolChainCoEjections`
|
|
35
|
-
- `toolChainStubReplacements`
|
|
36
|
-
- `artifactDegradations`
|
|
37
|
-
- `artifactOversizeThresholdTokens`
|
|
38
|
-
- `replayState`
|
|
39
|
-
- `replayReason` (legacy machine reason strings may still contain `redis` for compatibility)
|
|
40
|
-
|
|
41
|
-
The release harness asserts those counters against prompt-visible behavior, so the telemetry is not just emitted, it is verified.
|
|
42
|
-
|
|
43
|
-
## Inspecting artifacts manually
|
|
44
|
-
|
|
45
|
-
By default the harness deletes its temp HOME on success.
|
|
46
|
-
|
|
47
|
-
To keep the temp workspace and telemetry file:
|
|
48
|
-
|
|
49
|
-
```bash
|
|
50
|
-
HYPERMEM_KEEP_RELEASE_TMP=1 npm run validate:release-path
|
|
51
|
-
```
|
|
52
|
-
|
|
53
|
-
The script will print the preserved temp path. The telemetry file lives at:
|
|
54
|
-
|
|
55
|
-
```text
|
|
56
|
-
<tmp>/release-telemetry.jsonl
|
|
57
|
-
```
|
|
58
|
-
|
|
59
|
-
## Healthy result
|
|
60
|
-
|
|
61
|
-
```text
|
|
62
|
-
ALL 12 CHECKS PASSED ✅
|
|
63
|
-
```
|
|
64
|
-
|
|
65
|
-
A healthy run means the built plugin can prove the Phase C prompt-path contracts that matter for `0.8.0`:
|
|
66
|
-
|
|
67
|
-
- degraded content uses the canonical visible shapes where it is prompt-visible
|
|
68
|
-
- degradation counters line up with what entered the model path
|
|
69
|
-
- replay recovery is visible at the plugin boundary
|
|
70
|
-
- the proof runs against the real assemble lifecycle, not a mocked helper only
|
package/docs/RELEASE_PROCESS.md
DELETED
|
@@ -1,10 +0,0 @@
|
|
|
1
|
-
# HyperMem Release Process
|
|
2
|
-
|
|
3
|
-
**Canonical source:** [PsiClawOps/publication_process_internal](https://github.com/PsiClawOps/publication_process_internal)
|
|
4
|
-
|
|
5
|
-
- **Universal process:** `PROCESS.md` (5-stage publication pipeline)
|
|
6
|
-
- **HyperMem-specific:** `repos/hypermem.md` (scrub lists, stubs, build checks)
|
|
7
|
-
- **Install verification:** `INSTALL_VERIFICATION.md` (independent vetting gate)
|
|
8
|
-
- **Style:** `STYLE_GUIDE.md` + `FLEET_OUTPUT.md`
|
|
9
|
-
|
|
10
|
-
Do not duplicate process content here. If something is missing from the canonical repo, add it there.
|