@unblocklabs/unblock-memory 0.2.5 → 0.2.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +68 -42
- package/dist/src/analysis.d.ts +17 -1
- package/dist/src/analysis.js +123 -19
- package/dist/src/curation.d.ts +70 -0
- package/dist/src/curation.js +191 -0
- package/dist/src/manager.d.ts +20 -1
- package/dist/src/manager.js +126 -1
- package/dist/src/plugin.js +88 -3
- package/dist/src/runtime.js +1 -0
- package/dist/src/session-projector.js +1 -18
- package/dist/src/session-sync.js +3 -2
- package/openclaw.plugin.json +5 -3
- package/package.json +2 -2
- package/skills/memory-curator/SKILL.md +72 -66
package/README.md
CHANGED
|
@@ -53,14 +53,9 @@ directories, or globs into named corpora:
|
|
|
53
53
|
chatTypes: ["channel", "group"],
|
|
54
54
|
},
|
|
55
55
|
{
|
|
56
|
-
name: "
|
|
56
|
+
name: "knowledge",
|
|
57
57
|
kind: "files",
|
|
58
|
-
paths: ["knowledge
|
|
59
|
-
},
|
|
60
|
-
{
|
|
61
|
-
name: "reflections",
|
|
62
|
-
kind: "files",
|
|
63
|
-
paths: ["knowledge/reflections/**/*.md"],
|
|
58
|
+
paths: ["knowledge/**/*.md"],
|
|
64
59
|
},
|
|
65
60
|
],
|
|
66
61
|
// Optional: omit unless the local analysis worker is installed.
|
|
@@ -85,7 +80,7 @@ context resident after first use. Set it to `false` to restore QMD's five-minute
|
|
|
85
80
|
idle unload behavior.
|
|
86
81
|
|
|
87
82
|
`memory_search` searches every configured corpus by default. Pass
|
|
88
|
-
`corpora: ["
|
|
83
|
+
`corpora: ["knowledge"]` to search selected corpora or `corpora: ["all"]` to
|
|
89
84
|
request all of them explicitly. Search results include their corpus name and
|
|
90
85
|
remain readable by passing the returned `qmd://` path to `memory_get`.
|
|
91
86
|
|
|
@@ -115,18 +110,22 @@ The optional `sessions` corpus reads the current agent's normal OpenClaw SQLite
|
|
|
115
110
|
store and indexes its active user/assistant transcript branch. It defaults to
|
|
116
111
|
channel and group conversations; add `direct` explicitly to include DMs. Run
|
|
117
112
|
`memory_sync_sessions` to start a refresh, then use `memory_sync_status` to
|
|
118
|
-
check its progress or result. Projections are private derived Markdown
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
113
|
+
check its progress or result. Projections are private derived Markdown under the
|
|
114
|
+
agent's `unblock-memory/sessions` state directory and can be rebuilt from
|
|
115
|
+
OpenClaw at any time. Their embedded text contains only `# Transcript` and
|
|
116
|
+
timestamped speaker messages; filtering metadata remains in the session
|
|
117
|
+
manifest. The projected file modification time matches the session start time
|
|
118
|
+
for meaningful chronological cluster reads. Session results include provider,
|
|
119
|
+
chat type, conversation identity, and start time as an ISO 8601 timestamp. They
|
|
120
|
+
participate in the same search and clustering index as file memory. This phase
|
|
121
|
+
does not sync sessions at startup or on a schedule; refreshes are manual through
|
|
124
122
|
`memory_sync_sessions`.
|
|
125
123
|
|
|
126
124
|
Indexes live at `~/.openclaw/agents/<agentId>/unblock-memory/index.sqlite` (or the
|
|
127
|
-
equivalent configured OpenClaw state directory).
|
|
128
|
-
|
|
129
|
-
|
|
125
|
+
equivalent configured OpenClaw state directory). Durable agent-supplied event
|
|
126
|
+
dates and maintenance proposals live separately in `curation.sqlite`, so a QMD
|
|
127
|
+
index rebuild does not discard them. The first lookup builds the index;
|
|
128
|
+
Markdown filesystem changes queue a debounced, serialized background refresh.
|
|
130
129
|
|
|
131
130
|
## Memory analysis
|
|
132
131
|
|
|
@@ -166,17 +165,38 @@ Unblock Memory exposes:
|
|
|
166
165
|
- `memory_list_clusters` to cheaply list current clusters and report whether the
|
|
167
166
|
retained analysis is stale
|
|
168
167
|
- `memory_recluster` to explicitly rebuild clusters when the list is missing or stale
|
|
169
|
-
- `memory_fetch_cluster` to return
|
|
170
|
-
short `clusterId` returned by `memory_list_clusters`
|
|
168
|
+
- `memory_fetch_cluster` to return a sorted, paginated selection of QMD chunks
|
|
169
|
+
for a short `clusterId` returned by `memory_list_clusters`
|
|
171
170
|
|
|
172
171
|
`memory_recluster` optionally accepts UMAP controls (`method`, components,
|
|
173
172
|
neighbors, and minimum distance), HDBSCAN controls (minimum cluster size,
|
|
174
173
|
minimum samples, selection method and epsilon, and single-cluster behavior),
|
|
175
174
|
and a deterministic seed. Omitting them uses the worker's defaults.
|
|
176
175
|
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
176
|
+
`memory_fetch_cluster` accepts `topK` (1–50), a zero-based `offset`, and
|
|
177
|
+
`sort`: `representative` (the default), `score_desc`, `score_asc`, `date_desc`,
|
|
178
|
+
or `date_asc`. Score is cluster membership probability for normal clusters and
|
|
179
|
+
outlier score for noise. Each member reports raw `sourceModifiedAt` separately
|
|
180
|
+
from `eventTime` and `eventTimeBasis`. Session start times and dated memory paths
|
|
181
|
+
resolve programmatically; reviewed annotations resolve otherwise ambiguous
|
|
182
|
+
chunks or whole documents. Date sorting uses resolved event time when available
|
|
183
|
+
and the clearly labeled source modification time only as a fallback. Responses
|
|
184
|
+
include page totals and the next offset when more members remain.
|
|
185
|
+
|
|
186
|
+
A chronological cluster read creates a coalesced maintenance proposal only for
|
|
187
|
+
returned documents whose event time remains ambiguous; it does not scan the
|
|
188
|
+
whole corpus for chores. Persisted exact-duplicate analysis can likewise create
|
|
189
|
+
review proposals for non-session Markdown. `memory_list_maintenance_tasks`
|
|
190
|
+
returns at most ten tasks, while `memory_update_maintenance_task` can resolve,
|
|
191
|
+
defer, or mark one irrelevant and optionally attach a supported event date.
|
|
192
|
+
These tools never edit or delete source Markdown. Duplicate cleanup remains a
|
|
193
|
+
reviewed source change outside the maintenance tool, and generated session
|
|
194
|
+
projections must never be edited directly.
|
|
195
|
+
|
|
196
|
+
Member excerpts are capped at 2 KB each and 12 KB across a response; source
|
|
197
|
+
aliases are capped at five per member and 50 across a response. These budgets
|
|
198
|
+
are shared across the page so every returned member receives a useful excerpt
|
|
199
|
+
and at least one source path, including a full 50-member page.
|
|
180
200
|
|
|
181
201
|
If indexing changes content or vectors, the previous derived analysis is kept
|
|
182
202
|
and marked stale. Cluster reads include the analysis timestamp, stale timestamp,
|
|
@@ -186,33 +206,37 @@ A failed rebuild leaves the stale result intact, while a successful rebuild
|
|
|
186
206
|
atomically replaces it. Analysis is never scheduled automatically. If the worker
|
|
187
207
|
is absent or fails, `memory_search` and `memory_get` continue to work.
|
|
188
208
|
|
|
189
|
-
## Curating
|
|
209
|
+
## Curating knowledge
|
|
190
210
|
|
|
191
211
|
The plugin bundles the `memory-curator` skill for turning useful clusters into
|
|
192
212
|
durable knowledge. It becomes available when the plugin is enabled. If the
|
|
193
213
|
agent has an explicit skill allowlist, include `memory-curator`.
|
|
194
214
|
|
|
195
|
-
Keep
|
|
215
|
+
Keep maintained knowledge outside `memory/**` so each file belongs to only one
|
|
216
|
+
corpus. Use stable topic files updated in place:
|
|
196
217
|
|
|
197
218
|
```text
|
|
198
219
|
knowledge/
|
|
199
|
-
├──
|
|
200
|
-
|
|
201
|
-
└──
|
|
202
|
-
|
|
220
|
+
├── fleet.md
|
|
221
|
+
├── people/
|
|
222
|
+
│ └── rico.md
|
|
223
|
+
└── projects/
|
|
224
|
+
└── unblock-memory.md
|
|
203
225
|
```
|
|
204
226
|
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
and
|
|
208
|
-
|
|
209
|
-
|
|
227
|
+
Knowledge is the agent's maintained, current understanding of its unique world:
|
|
228
|
+
facts such as fleet membership, local decisions and preferences, assessments,
|
|
229
|
+
and explicit uncertainty that would be expensive to reconstruct from scattered
|
|
230
|
+
history. Each claim should carry its own epistemic qualification so it remains
|
|
231
|
+
honest when semantic chunking retrieves it alone. Remove stale conclusions
|
|
232
|
+
instead of preserving history, changelogs, or `Supersedes` passages in the same
|
|
233
|
+
file; raw memory and sessions retain the evidence history.
|
|
210
234
|
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
authoritative.
|
|
214
|
-
|
|
215
|
-
|
|
235
|
+
Public or vendor-owned facts, generic command syntax, and behavior likely to
|
|
236
|
+
change with third-party releases should normally be looked up from the current
|
|
237
|
+
authoritative source. A local policy or deliberate divergence may belong in
|
|
238
|
+
knowledge, but the local decision—not copied generic documentation—is the
|
|
239
|
+
durable content.
|
|
216
240
|
|
|
217
241
|
For a manual run, ask the agent:
|
|
218
242
|
|
|
@@ -226,10 +250,12 @@ For recurring curation, use an OpenClaw automation with the same thin message:
|
|
|
226
250
|
Use $memory-curator to run the scheduled memory curation cycle.
|
|
227
251
|
```
|
|
228
252
|
|
|
229
|
-
The skill
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
253
|
+
The skill treats a cluster as an incomplete attention signal. It frames the
|
|
254
|
+
question raised, uses representative, score, and chronological views as useful,
|
|
255
|
+
searches existing knowledge and adjacent corpora, and investigates live systems,
|
|
256
|
+
files, documentation, or the web when those are better evidence. It then updates
|
|
257
|
+
a stable knowledge topic or correctly writes nothing. Its own writes are indexed
|
|
258
|
+
for the next cycle; it does not recluster recursively in the same run.
|
|
233
259
|
|
|
234
260
|
Existing `unblock-qmd` indexes are derived caches and may be left in place;
|
|
235
261
|
Unblock Memory rebuilds its own index from configured corpora.
|
package/dist/src/analysis.d.ts
CHANGED
|
@@ -1,5 +1,8 @@
|
|
|
1
1
|
import type { QMDStore } from "@unblocklabs/qmd";
|
|
2
2
|
type AnalysisDatabase = QMDStore["internal"]["db"];
|
|
3
|
+
export type TemporalReadOptions = {
|
|
4
|
+
sessionCollection?: string;
|
|
5
|
+
};
|
|
3
6
|
export type MemoryReclusterOptions = {
|
|
4
7
|
space?: {
|
|
5
8
|
method?: "umap" | "none";
|
|
@@ -30,9 +33,15 @@ type MemoryAnalysisMember = {
|
|
|
30
33
|
x: number;
|
|
31
34
|
y: number;
|
|
32
35
|
representativeRank: number | null;
|
|
36
|
+
sourceModifiedAt: string;
|
|
37
|
+
eventTime: string | null;
|
|
38
|
+
eventTimeBasis: "path" | "frontmatter" | "session" | "agent_verified" | null;
|
|
39
|
+
eventTimeSource: string;
|
|
40
|
+
contentFingerprint: string;
|
|
33
41
|
text: string;
|
|
34
42
|
sourcePaths: string[];
|
|
35
43
|
};
|
|
44
|
+
export type MemoryClusterSort = "representative" | "score_desc" | "score_asc" | "date_desc" | "date_asc";
|
|
36
45
|
export type MemoryAnalysisSummary = {
|
|
37
46
|
status: "ok";
|
|
38
47
|
runId: string;
|
|
@@ -72,6 +81,13 @@ export type MemoryClusterDetail = AnalysisReadMetadata & {
|
|
|
72
81
|
runId?: string;
|
|
73
82
|
cluster?: Omit<MemoryClusterSummary, "preview">;
|
|
74
83
|
members?: MemoryAnalysisMember[];
|
|
84
|
+
page?: {
|
|
85
|
+
offset: number;
|
|
86
|
+
returned: number;
|
|
87
|
+
total: number;
|
|
88
|
+
hasMore: boolean;
|
|
89
|
+
nextOffset?: number;
|
|
90
|
+
};
|
|
75
91
|
};
|
|
76
92
|
export declare function ensureMemoryAnalysisSchema(db: AnalysisDatabase): void;
|
|
77
93
|
export declare function markMemoryAnalysisStale(db: AnalysisDatabase): void;
|
|
@@ -85,5 +101,5 @@ export declare function runAnalysisWorker(params: {
|
|
|
85
101
|
export declare function latestAnalysisRunId(db: AnalysisDatabase): string | undefined;
|
|
86
102
|
export declare function readAnalysisSummary(db: AnalysisDatabase): MemoryAnalysisSummary | undefined;
|
|
87
103
|
export declare function readClusters(db: AnalysisDatabase, requestedLimit?: number): MemoryClusterList;
|
|
88
|
-
export declare function readCluster(db: AnalysisDatabase, clusterReferenceId: string, requestedLimit?: number): MemoryClusterDetail;
|
|
104
|
+
export declare function readCluster(db: AnalysisDatabase, clusterReferenceId: string, requestedLimit?: number, requestedOffset?: number, sort?: MemoryClusterSort, temporal?: TemporalReadOptions): MemoryClusterDetail;
|
|
89
105
|
export {};
|
package/dist/src/analysis.js
CHANGED
|
@@ -48,6 +48,17 @@ export function ensureMemoryAnalysisSchema(db) {
|
|
|
48
48
|
CREATE INDEX IF NOT EXISTS idx_memory_analysis_memberships_cluster
|
|
49
49
|
ON memory_analysis_memberships(run_id, cluster_id, representative_rank);
|
|
50
50
|
|
|
51
|
+
CREATE TABLE IF NOT EXISTS memory_analysis_duplicate_occurrences (
|
|
52
|
+
run_id TEXT NOT NULL,
|
|
53
|
+
content_fingerprint TEXT NOT NULL,
|
|
54
|
+
canonical_hash TEXT NOT NULL,
|
|
55
|
+
canonical_seq INTEGER NOT NULL,
|
|
56
|
+
duplicate_hash TEXT NOT NULL,
|
|
57
|
+
duplicate_seq INTEGER NOT NULL,
|
|
58
|
+
PRIMARY KEY (run_id, duplicate_hash, duplicate_seq),
|
|
59
|
+
FOREIGN KEY (run_id) REFERENCES memory_analysis_runs(id) ON DELETE CASCADE
|
|
60
|
+
);
|
|
61
|
+
|
|
51
62
|
CREATE VIEW IF NOT EXISTS memory_analysis_available_memberships AS
|
|
52
63
|
SELECT
|
|
53
64
|
m.run_id, m.hash, m.seq, m.cluster_id, m.probability, m.outlier_score,
|
|
@@ -61,6 +72,18 @@ export function ensureMemoryAnalysisSchema(db) {
|
|
|
61
72
|
WHERE d.hash = m.hash AND d.active = 1
|
|
62
73
|
);
|
|
63
74
|
`);
|
|
75
|
+
db.exec(`
|
|
76
|
+
CREATE TEMP TABLE IF NOT EXISTS memory_temporal_annotations (
|
|
77
|
+
collection TEXT NOT NULL,
|
|
78
|
+
path TEXT NOT NULL,
|
|
79
|
+
qmd_hash TEXT,
|
|
80
|
+
qmd_seq INTEGER,
|
|
81
|
+
event_time TEXT NOT NULL,
|
|
82
|
+
basis TEXT NOT NULL,
|
|
83
|
+
document_wide INTEGER NOT NULL,
|
|
84
|
+
PRIMARY KEY (collection, path, qmd_hash, qmd_seq, document_wide)
|
|
85
|
+
);
|
|
86
|
+
`);
|
|
64
87
|
}
|
|
65
88
|
export function markMemoryAnalysisStale(db) {
|
|
66
89
|
db.prepare(`
|
|
@@ -208,8 +231,8 @@ function byteSlice(text, maxBytes) {
|
|
|
208
231
|
return "";
|
|
209
232
|
return bytes.subarray(0, maxBytes - 3).toString("utf8").replace(/\uFFFD$/u, "") + "…";
|
|
210
233
|
}
|
|
211
|
-
function members(db, runId, clusterId, limit, maxExcerptBytes = MAX_EXCERPT_BYTES, maxTotalBytes = MAX_TOTAL_EXCERPT_BYTES, maxTotalAliases = MAX_TOTAL_ALIASES) {
|
|
212
|
-
const
|
|
234
|
+
function members(db, runId, clusterId, limit, offset = 0, sort = "representative", maxExcerptBytes = MAX_EXCERPT_BYTES, maxTotalBytes = MAX_TOTAL_EXCERPT_BYTES, maxTotalAliases = MAX_TOTAL_ALIASES, temporal = {}) {
|
|
235
|
+
const representativeOrder = clusterId === -1
|
|
213
236
|
? "m.outlier_score DESC, m.hash, m.seq"
|
|
214
237
|
: `CASE WHEN m.representative_rank IS NULL THEN 1 ELSE 0 END,
|
|
215
238
|
m.representative_rank,
|
|
@@ -217,23 +240,87 @@ function members(db, runId, clusterId, limit, maxExcerptBytes = MAX_EXCERPT_BYTE
|
|
|
217
240
|
m.outlier_score,
|
|
218
241
|
m.hash,
|
|
219
242
|
m.seq`;
|
|
243
|
+
const score = clusterId === -1 ? "m.outlier_score" : "m.probability";
|
|
244
|
+
const order = {
|
|
245
|
+
representative: representativeOrder,
|
|
246
|
+
score_desc: `${score} DESC, m.hash, m.seq`,
|
|
247
|
+
score_asc: `${score} ASC, m.hash, m.seq`,
|
|
248
|
+
date_desc: "julianday(COALESCE(m.event_time, m.source_modified_at)) DESC, m.hash, m.seq",
|
|
249
|
+
date_asc: "julianday(COALESCE(m.event_time, m.source_modified_at)) ASC, m.hash, m.seq",
|
|
250
|
+
}[sort];
|
|
220
251
|
const rows = db.prepare(`
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
252
|
+
WITH candidate_times AS (
|
|
253
|
+
SELECT
|
|
254
|
+
m.hash,
|
|
255
|
+
m.seq,
|
|
256
|
+
d.collection,
|
|
257
|
+
d.path,
|
|
258
|
+
d.modified_at AS source_modified_at,
|
|
259
|
+
CASE
|
|
260
|
+
WHEN d.collection = ? THEN d.modified_at
|
|
261
|
+
WHEN d.path GLOB '*[12][0-9][0-9][0-9]-[01][0-9]-[0-3][0-9].md'
|
|
262
|
+
THEN substr(d.path, length(d.path) - 12, 10) || 'T00:00:00.000Z'
|
|
263
|
+
ELSE annotation.event_time
|
|
264
|
+
END AS event_time,
|
|
265
|
+
CASE
|
|
266
|
+
WHEN d.collection = ? THEN 'session'
|
|
267
|
+
WHEN d.path GLOB '*[12][0-9][0-9][0-9]-[01][0-9]-[0-3][0-9].md' THEN 'path'
|
|
268
|
+
ELSE annotation.basis
|
|
269
|
+
END AS event_time_basis,
|
|
270
|
+
CASE
|
|
271
|
+
WHEN d.collection = ? THEN 1
|
|
272
|
+
WHEN d.path GLOB '*[12][0-9][0-9][0-9]-[01][0-9]-[0-3][0-9].md' THEN 2
|
|
273
|
+
WHEN annotation.event_time IS NOT NULL THEN 3
|
|
274
|
+
ELSE 4
|
|
275
|
+
END AS priority
|
|
276
|
+
FROM memory_analysis_available_memberships m
|
|
277
|
+
JOIN documents d ON d.hash = m.hash AND d.active = 1
|
|
278
|
+
LEFT JOIN memory_temporal_annotations annotation
|
|
279
|
+
ON annotation.collection = d.collection
|
|
280
|
+
AND annotation.path = d.path
|
|
281
|
+
AND (annotation.document_wide = 1 OR
|
|
282
|
+
(annotation.qmd_hash = m.hash AND annotation.qmd_seq = m.seq))
|
|
283
|
+
WHERE m.run_id = ? AND m.cluster_id = ?
|
|
284
|
+
), ranked_times AS (
|
|
285
|
+
SELECT *, ROW_NUMBER() OVER (
|
|
286
|
+
PARTITION BY hash, seq
|
|
287
|
+
ORDER BY priority, julianday(COALESCE(event_time, source_modified_at)) DESC, collection, path
|
|
288
|
+
) AS rank
|
|
289
|
+
FROM candidate_times
|
|
290
|
+
), member_rows AS (
|
|
291
|
+
SELECT
|
|
292
|
+
m.hash, m.seq, m.probability, m.outlier_score, m.x, m.y,
|
|
293
|
+
m.representative_rank, m.pos, m.chunk_len, m.doc,
|
|
294
|
+
(
|
|
295
|
+
SELECT d.modified_at
|
|
296
|
+
FROM documents d
|
|
297
|
+
WHERE d.hash = m.hash AND d.active = 1
|
|
298
|
+
ORDER BY julianday(d.modified_at) DESC, d.collection, d.path
|
|
299
|
+
LIMIT 1
|
|
300
|
+
) AS source_modified_at,
|
|
301
|
+
temporal.event_time,
|
|
302
|
+
temporal.event_time_basis,
|
|
303
|
+
temporal.collection AS event_collection,
|
|
304
|
+
temporal.path AS event_path
|
|
305
|
+
FROM memory_analysis_available_memberships m
|
|
306
|
+
JOIN ranked_times temporal
|
|
307
|
+
ON temporal.hash = m.hash AND temporal.seq = m.seq AND temporal.rank = 1
|
|
308
|
+
WHERE m.run_id = ? AND m.cluster_id = ?
|
|
309
|
+
)
|
|
310
|
+
SELECT * FROM member_rows m
|
|
311
|
+
ORDER BY ${order}
|
|
312
|
+
LIMIT ? OFFSET ?
|
|
313
|
+
`).all(temporal.sessionCollection ?? "", temporal.sessionCollection ?? "", temporal.sessionCollection ?? "", runId, clusterId, runId, clusterId, limit, offset);
|
|
229
314
|
let remaining = maxTotalBytes;
|
|
230
315
|
let remainingAliases = maxTotalAliases;
|
|
231
|
-
return rows.map((row) => {
|
|
232
|
-
const
|
|
233
|
-
|
|
234
|
-
|
|
316
|
+
return rows.map((row, index) => {
|
|
317
|
+
const remainingRows = rows.length - index;
|
|
318
|
+
const excerptBudget = Math.min(maxExcerptBytes, Math.floor(remaining / remainingRows));
|
|
319
|
+
const fullText = row.doc.slice(row.pos, row.pos + row.chunk_len);
|
|
320
|
+
const text = byteSlice(fullText, excerptBudget);
|
|
235
321
|
remaining -= Buffer.byteLength(text);
|
|
236
|
-
const
|
|
322
|
+
const aliasBudget = Math.min(MAX_ALIASES_PER_MEMBER, Math.floor(remainingAliases / remainingRows));
|
|
323
|
+
const aliases = sourcePaths(db, row.hash, aliasBudget);
|
|
237
324
|
remainingAliases -= aliases.length;
|
|
238
325
|
return {
|
|
239
326
|
hash: row.hash,
|
|
@@ -243,6 +330,11 @@ function members(db, runId, clusterId, limit, maxExcerptBytes = MAX_EXCERPT_BYTE
|
|
|
243
330
|
x: row.x,
|
|
244
331
|
y: row.y,
|
|
245
332
|
representativeRank: row.representative_rank,
|
|
333
|
+
sourceModifiedAt: row.source_modified_at,
|
|
334
|
+
eventTime: row.event_time,
|
|
335
|
+
eventTimeBasis: row.event_time_basis,
|
|
336
|
+
eventTimeSource: `qmd://${row.event_collection}/${row.event_path}`,
|
|
337
|
+
contentFingerprint: createHash("sha256").update(fullText).digest("hex"),
|
|
246
338
|
text,
|
|
247
339
|
sourcePaths: aliases,
|
|
248
340
|
};
|
|
@@ -276,7 +368,7 @@ function readMetadata(run) {
|
|
|
276
368
|
}
|
|
277
369
|
function toSummary(db, run, row, includePreview, previewBytes = 600, aliasLimit = MAX_TOTAL_ALIASES) {
|
|
278
370
|
const preview = includePreview
|
|
279
|
-
? members(db, run.id, row.cluster_id, 1, previewBytes, previewBytes, aliasLimit)[0]
|
|
371
|
+
? members(db, run.id, row.cluster_id, 1, 0, "representative", previewBytes, previewBytes, aliasLimit)[0]
|
|
280
372
|
: undefined;
|
|
281
373
|
return {
|
|
282
374
|
clusterId: clusterReference(run.id, row.cluster_id),
|
|
@@ -362,7 +454,7 @@ function resolveClusterId(db, runId, reference) {
|
|
|
362
454
|
`).all(runId, runId);
|
|
363
455
|
return clusterIds.find((row) => clusterReference(runId, row.cluster_id) === reference)?.cluster_id;
|
|
364
456
|
}
|
|
365
|
-
export function readCluster(db, clusterReferenceId, requestedLimit = DEFAULT_MEMBER_LIMIT) {
|
|
457
|
+
export function readCluster(db, clusterReferenceId, requestedLimit = DEFAULT_MEMBER_LIMIT, requestedOffset = 0, sort = "representative", temporal = {}) {
|
|
366
458
|
const run = latestValidRun(db);
|
|
367
459
|
if (!run) {
|
|
368
460
|
return { status: "not_analyzed", ...readMetadata() };
|
|
@@ -388,6 +480,11 @@ export function readCluster(db, clusterReferenceId, requestedLimit = DEFAULT_MEM
|
|
|
388
480
|
return { status: "not_found", runId: run.id, ...metadata };
|
|
389
481
|
}
|
|
390
482
|
const limit = Math.max(1, Math.min(MAX_MEMBER_LIMIT, Math.floor(requestedLimit)));
|
|
483
|
+
const offset = Math.max(0, Math.floor(requestedOffset));
|
|
484
|
+
const total = availableSize(db, run.id, clusterId);
|
|
485
|
+
const pageMembers = members(db, run.id, clusterId, limit, offset, sort, MAX_EXCERPT_BYTES, MAX_TOTAL_EXCERPT_BYTES, MAX_TOTAL_ALIASES, temporal);
|
|
486
|
+
const nextOffset = offset + pageMembers.length;
|
|
487
|
+
const hasMore = nextOffset < total;
|
|
391
488
|
return {
|
|
392
489
|
status: "ok",
|
|
393
490
|
runId: run.id,
|
|
@@ -395,9 +492,16 @@ export function readCluster(db, clusterReferenceId, requestedLimit = DEFAULT_MEM
|
|
|
395
492
|
cluster: {
|
|
396
493
|
clusterId: clusterReferenceId,
|
|
397
494
|
size: row.size,
|
|
398
|
-
availableSize:
|
|
495
|
+
availableSize: total,
|
|
399
496
|
meanProbability: row.mean_probability,
|
|
400
497
|
},
|
|
401
|
-
members:
|
|
498
|
+
members: pageMembers,
|
|
499
|
+
page: {
|
|
500
|
+
offset,
|
|
501
|
+
returned: pageMembers.length,
|
|
502
|
+
total,
|
|
503
|
+
hasMore,
|
|
504
|
+
...(hasMore ? { nextOffset } : {}),
|
|
505
|
+
},
|
|
402
506
|
};
|
|
403
507
|
}
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
declare const TEMPORAL_BASES: readonly ["path", "frontmatter", "session", "agent_verified"];
|
|
2
|
+
export type TemporalBasis = typeof TEMPORAL_BASES[number];
|
|
3
|
+
declare const MAINTENANCE_TASK_TYPES: readonly ["ambiguous_event_time", "exact_duplicate"];
|
|
4
|
+
export type MaintenanceTaskType = typeof MAINTENANCE_TASK_TYPES[number];
|
|
5
|
+
declare const MAINTENANCE_STATUSES: readonly ["pending", "resolved", "deferred", "irrelevant"];
|
|
6
|
+
export type MaintenanceStatus = typeof MAINTENANCE_STATUSES[number];
|
|
7
|
+
export type TemporalAnnotation = {
|
|
8
|
+
corpus: string;
|
|
9
|
+
collection: string;
|
|
10
|
+
path: string;
|
|
11
|
+
contentFingerprint: string;
|
|
12
|
+
eventTime: string;
|
|
13
|
+
basis: TemporalBasis;
|
|
14
|
+
evidence: string;
|
|
15
|
+
qmdHash: string | null;
|
|
16
|
+
qmdSeq: number | null;
|
|
17
|
+
createdAt: string;
|
|
18
|
+
updatedAt: string;
|
|
19
|
+
};
|
|
20
|
+
export type MaintenanceTask = {
|
|
21
|
+
id: string;
|
|
22
|
+
type: MaintenanceTaskType;
|
|
23
|
+
corpus: string;
|
|
24
|
+
collection: string;
|
|
25
|
+
path: string;
|
|
26
|
+
reason: string;
|
|
27
|
+
contentFingerprint: string;
|
|
28
|
+
detail: string | null;
|
|
29
|
+
resolutionNote: string | null;
|
|
30
|
+
status: MaintenanceStatus;
|
|
31
|
+
createdAt: string;
|
|
32
|
+
updatedAt: string;
|
|
33
|
+
};
|
|
34
|
+
export declare function chunkFingerprint(text: string): string;
|
|
35
|
+
export declare class CurationStore {
|
|
36
|
+
#private;
|
|
37
|
+
constructor(path: string);
|
|
38
|
+
close(): void;
|
|
39
|
+
annotations(): TemporalAnnotation[];
|
|
40
|
+
addTask(candidate: {
|
|
41
|
+
type: MaintenanceTaskType;
|
|
42
|
+
corpus: string;
|
|
43
|
+
collection: string;
|
|
44
|
+
path: string;
|
|
45
|
+
reason: string;
|
|
46
|
+
contentFingerprint?: string;
|
|
47
|
+
detail?: string;
|
|
48
|
+
}): void;
|
|
49
|
+
listTasks(params?: {
|
|
50
|
+
status?: MaintenanceStatus;
|
|
51
|
+
limit?: number;
|
|
52
|
+
}): MaintenanceTask[];
|
|
53
|
+
updateTask(params: {
|
|
54
|
+
id: string;
|
|
55
|
+
status: Exclude<MaintenanceStatus, "pending">;
|
|
56
|
+
note?: string;
|
|
57
|
+
annotation?: {
|
|
58
|
+
scope: "chunk" | "document";
|
|
59
|
+
eventTime: string;
|
|
60
|
+
basis: TemporalBasis;
|
|
61
|
+
evidence: string;
|
|
62
|
+
};
|
|
63
|
+
}): MaintenanceTask | undefined;
|
|
64
|
+
updateAnnotationLocation(params: {
|
|
65
|
+
annotation: TemporalAnnotation;
|
|
66
|
+
qmdHash: string | null;
|
|
67
|
+
qmdSeq: number | null;
|
|
68
|
+
}): void;
|
|
69
|
+
}
|
|
70
|
+
export {};
|
|
@@ -0,0 +1,191 @@
|
|
|
1
|
+
import { createHash, randomUUID } from "node:crypto";
|
|
2
|
+
import { chmodSync, mkdirSync } from "node:fs";
|
|
3
|
+
import { dirname } from "node:path";
|
|
4
|
+
import { DatabaseSync } from "node:sqlite";
|
|
5
|
+
const TEMPORAL_BASES = ["path", "frontmatter", "session", "agent_verified"];
|
|
6
|
+
const MAINTENANCE_TASK_TYPES = ["ambiguous_event_time", "exact_duplicate"];
|
|
7
|
+
const MAINTENANCE_STATUSES = ["pending", "resolved", "deferred", "irrelevant"];
|
|
8
|
+
function annotation(row) {
|
|
9
|
+
return {
|
|
10
|
+
corpus: row.corpus,
|
|
11
|
+
collection: row.collection,
|
|
12
|
+
path: row.path,
|
|
13
|
+
contentFingerprint: row.content_fingerprint,
|
|
14
|
+
eventTime: row.event_time,
|
|
15
|
+
basis: row.basis,
|
|
16
|
+
evidence: row.evidence,
|
|
17
|
+
qmdHash: row.qmd_hash,
|
|
18
|
+
qmdSeq: row.qmd_seq,
|
|
19
|
+
createdAt: row.created_at,
|
|
20
|
+
updatedAt: row.updated_at,
|
|
21
|
+
};
|
|
22
|
+
}
|
|
23
|
+
function task(row) {
|
|
24
|
+
return {
|
|
25
|
+
id: row.id,
|
|
26
|
+
type: row.type,
|
|
27
|
+
corpus: row.corpus,
|
|
28
|
+
collection: row.collection,
|
|
29
|
+
path: row.path,
|
|
30
|
+
reason: row.reason,
|
|
31
|
+
contentFingerprint: row.content_fingerprint,
|
|
32
|
+
detail: row.detail,
|
|
33
|
+
resolutionNote: row.resolution_note,
|
|
34
|
+
status: row.status,
|
|
35
|
+
createdAt: row.created_at,
|
|
36
|
+
updatedAt: row.updated_at,
|
|
37
|
+
};
|
|
38
|
+
}
|
|
39
|
+
export function chunkFingerprint(text) {
|
|
40
|
+
return createHash("sha256").update(text).digest("hex");
|
|
41
|
+
}
|
|
42
|
+
export class CurationStore {
|
|
43
|
+
#db;
|
|
44
|
+
constructor(path) {
|
|
45
|
+
mkdirSync(dirname(path), { recursive: true, mode: 0o700 });
|
|
46
|
+
this.#db = new DatabaseSync(path);
|
|
47
|
+
chmodSync(path, 0o600);
|
|
48
|
+
this.#db.exec(`
|
|
49
|
+
PRAGMA journal_mode = WAL;
|
|
50
|
+
PRAGMA busy_timeout = 5000;
|
|
51
|
+
|
|
52
|
+
CREATE TABLE IF NOT EXISTS temporal_annotations (
|
|
53
|
+
corpus TEXT NOT NULL,
|
|
54
|
+
collection TEXT NOT NULL,
|
|
55
|
+
path TEXT NOT NULL,
|
|
56
|
+
content_fingerprint TEXT NOT NULL DEFAULT '',
|
|
57
|
+
event_time TEXT NOT NULL,
|
|
58
|
+
basis TEXT NOT NULL CHECK (basis IN ('path', 'frontmatter', 'session', 'agent_verified')),
|
|
59
|
+
evidence TEXT NOT NULL,
|
|
60
|
+
qmd_hash TEXT,
|
|
61
|
+
qmd_seq INTEGER,
|
|
62
|
+
created_at TEXT NOT NULL,
|
|
63
|
+
updated_at TEXT NOT NULL,
|
|
64
|
+
PRIMARY KEY (corpus, collection, path, content_fingerprint)
|
|
65
|
+
);
|
|
66
|
+
|
|
67
|
+
`);
|
|
68
|
+
this.#ensureMaintenanceSchema();
|
|
69
|
+
}
|
|
70
|
+
#ensureMaintenanceSchema() {
|
|
71
|
+
this.#db.exec(`
|
|
72
|
+
CREATE TABLE IF NOT EXISTS maintenance_tasks (
|
|
73
|
+
id TEXT PRIMARY KEY,
|
|
74
|
+
type TEXT NOT NULL CHECK (type IN ('ambiguous_event_time', 'exact_duplicate')),
|
|
75
|
+
corpus TEXT NOT NULL,
|
|
76
|
+
collection TEXT NOT NULL,
|
|
77
|
+
path TEXT NOT NULL,
|
|
78
|
+
reason TEXT NOT NULL,
|
|
79
|
+
content_fingerprint TEXT NOT NULL,
|
|
80
|
+
detail TEXT,
|
|
81
|
+
resolution_note TEXT,
|
|
82
|
+
status TEXT NOT NULL CHECK (status IN ('pending', 'resolved', 'deferred', 'irrelevant')),
|
|
83
|
+
created_at TEXT NOT NULL,
|
|
84
|
+
updated_at TEXT NOT NULL,
|
|
85
|
+
UNIQUE (type, corpus, collection, path, reason, content_fingerprint)
|
|
86
|
+
);
|
|
87
|
+
`);
|
|
88
|
+
this.#db.exec(`
|
|
89
|
+
CREATE INDEX IF NOT EXISTS maintenance_tasks_status_created
|
|
90
|
+
ON maintenance_tasks(status, created_at);
|
|
91
|
+
`);
|
|
92
|
+
}
|
|
93
|
+
close() {
|
|
94
|
+
this.#db.close();
|
|
95
|
+
}
|
|
96
|
+
annotations() {
|
|
97
|
+
return this.#db.prepare(`
|
|
98
|
+
SELECT * FROM temporal_annotations
|
|
99
|
+
ORDER BY collection, path, content_fingerprint
|
|
100
|
+
`).all().map((row) => annotation(row));
|
|
101
|
+
}
|
|
102
|
+
addTask(candidate) {
|
|
103
|
+
const now = new Date().toISOString();
|
|
104
|
+
this.#db.prepare(`
|
|
105
|
+
INSERT INTO maintenance_tasks
|
|
106
|
+
(id, type, corpus, collection, path, reason, content_fingerprint, detail, status, created_at, updated_at)
|
|
107
|
+
VALUES (?, ?, ?, ?, ?, ?, ?, ?, 'pending', ?, ?)
|
|
108
|
+
ON CONFLICT(type, corpus, collection, path, reason, content_fingerprint) DO UPDATE SET
|
|
109
|
+
detail = CASE
|
|
110
|
+
WHEN maintenance_tasks.status = 'pending' THEN excluded.detail
|
|
111
|
+
ELSE maintenance_tasks.detail
|
|
112
|
+
END,
|
|
113
|
+
updated_at = CASE
|
|
114
|
+
WHEN maintenance_tasks.status = 'pending' THEN excluded.updated_at
|
|
115
|
+
ELSE maintenance_tasks.updated_at
|
|
116
|
+
END
|
|
117
|
+
`).run(randomUUID(), candidate.type, candidate.corpus, candidate.collection, candidate.path, candidate.reason, candidate.contentFingerprint ?? "", candidate.detail ?? null, now, now);
|
|
118
|
+
}
|
|
119
|
+
listTasks(params = {}) {
|
|
120
|
+
const status = params.status ?? "pending";
|
|
121
|
+
const limit = Math.max(1, Math.min(10, Math.floor(params.limit ?? 5)));
|
|
122
|
+
return this.#db.prepare(`
|
|
123
|
+
SELECT * FROM maintenance_tasks
|
|
124
|
+
WHERE status = ?
|
|
125
|
+
ORDER BY created_at, id
|
|
126
|
+
LIMIT ?
|
|
127
|
+
`).all(status, limit).map((row) => task(row));
|
|
128
|
+
}
|
|
129
|
+
updateTask(params) {
|
|
130
|
+
this.#db.exec("BEGIN IMMEDIATE");
|
|
131
|
+
try {
|
|
132
|
+
const row = this.#db.prepare("SELECT * FROM maintenance_tasks WHERE id = ?")
|
|
133
|
+
.get(params.id);
|
|
134
|
+
if (!row) {
|
|
135
|
+
this.#db.exec("COMMIT");
|
|
136
|
+
return undefined;
|
|
137
|
+
}
|
|
138
|
+
const now = new Date().toISOString();
|
|
139
|
+
if (row.type === "ambiguous_event_time" && params.status === "resolved" && !params.annotation) {
|
|
140
|
+
throw new Error("resolving an ambiguous event-time task requires a date annotation");
|
|
141
|
+
}
|
|
142
|
+
if (params.annotation) {
|
|
143
|
+
if (row.type !== "ambiguous_event_time") {
|
|
144
|
+
throw new Error("date annotations can only resolve ambiguous event-time tasks");
|
|
145
|
+
}
|
|
146
|
+
if (params.status !== "resolved") {
|
|
147
|
+
throw new Error("date annotations require resolved status");
|
|
148
|
+
}
|
|
149
|
+
if (!Number.isFinite(Date.parse(params.annotation.eventTime))) {
|
|
150
|
+
throw new Error("date annotation eventTime must be an ISO 8601 timestamp");
|
|
151
|
+
}
|
|
152
|
+
const fingerprint = params.annotation.scope === "document" ? "" : row.content_fingerprint;
|
|
153
|
+
if (params.annotation.scope === "chunk" && !fingerprint) {
|
|
154
|
+
throw new Error("chunk annotation requires a content fingerprint");
|
|
155
|
+
}
|
|
156
|
+
this.#db.prepare(`
|
|
157
|
+
INSERT INTO temporal_annotations
|
|
158
|
+
(corpus, collection, path, content_fingerprint, event_time, basis, evidence,
|
|
159
|
+
qmd_hash, qmd_seq, created_at, updated_at)
|
|
160
|
+
VALUES (?, ?, ?, ?, ?, ?, ?, NULL, NULL, ?, ?)
|
|
161
|
+
ON CONFLICT(corpus, collection, path, content_fingerprint) DO UPDATE SET
|
|
162
|
+
event_time = excluded.event_time,
|
|
163
|
+
basis = excluded.basis,
|
|
164
|
+
evidence = excluded.evidence,
|
|
165
|
+
qmd_hash = NULL,
|
|
166
|
+
qmd_seq = NULL,
|
|
167
|
+
updated_at = excluded.updated_at
|
|
168
|
+
`).run(row.corpus, row.collection, row.path, fingerprint, params.annotation.eventTime, params.annotation.basis, params.annotation.evidence, now, now);
|
|
169
|
+
}
|
|
170
|
+
this.#db.prepare(`
|
|
171
|
+
UPDATE maintenance_tasks
|
|
172
|
+
SET status = ?, resolution_note = ?, updated_at = ?
|
|
173
|
+
WHERE id = ?
|
|
174
|
+
`).run(params.status, params.note ?? null, now, params.id);
|
|
175
|
+
const updated = task(this.#db.prepare("SELECT * FROM maintenance_tasks WHERE id = ?").get(params.id));
|
|
176
|
+
this.#db.exec("COMMIT");
|
|
177
|
+
return updated;
|
|
178
|
+
}
|
|
179
|
+
catch (error) {
|
|
180
|
+
this.#db.exec("ROLLBACK");
|
|
181
|
+
throw error;
|
|
182
|
+
}
|
|
183
|
+
}
|
|
184
|
+
updateAnnotationLocation(params) {
|
|
185
|
+
this.#db.prepare(`
|
|
186
|
+
UPDATE temporal_annotations
|
|
187
|
+
SET qmd_hash = ?, qmd_seq = ?
|
|
188
|
+
WHERE corpus = ? AND collection = ? AND path = ? AND content_fingerprint = ?
|
|
189
|
+
`).run(params.qmdHash, params.qmdSeq, params.annotation.corpus, params.annotation.collection, params.annotation.path, params.annotation.contentFingerprint);
|
|
190
|
+
}
|
|
191
|
+
}
|
package/dist/src/manager.d.ts
CHANGED
|
@@ -1,7 +1,8 @@
|
|
|
1
1
|
import type { QMDStore } from "@unblocklabs/qmd";
|
|
2
|
-
import { type AnalysisRunner, type MemoryAnalysisSummary, type MemoryClusterDetail, type MemoryClusterList, type MemoryReclusterOptions } from "./analysis.js";
|
|
2
|
+
import { type AnalysisRunner, type MemoryAnalysisSummary, type MemoryClusterDetail, type MemoryClusterList, type MemoryClusterSort, type MemoryReclusterOptions } from "./analysis.js";
|
|
3
3
|
import type { CorpusMemorySearchResult, CorpusSearchOptions, MemoryEmbeddingProbeResult, MemoryProviderStatus, MemoryReadResult, MemorySearchManagerContract, MemorySyncParams } from "./contracts.js";
|
|
4
4
|
import type { ChatType } from "./config.js";
|
|
5
|
+
import { type MaintenanceStatus, type TemporalBasis } from "./curation.js";
|
|
5
6
|
import { type SessionSyncResult } from "./session-sync.js";
|
|
6
7
|
import { type ResolvedSource } from "./sources.js";
|
|
7
8
|
export type ManagerStore = Pick<QMDStore, "update" | "embed" | "getStatus" | "listCollections" | "searchLex" | "vsearch" | "get" | "getDocumentBody" | "close">;
|
|
@@ -28,6 +29,7 @@ export declare class QmdMemoryManager implements MemorySearchManagerContract {
|
|
|
28
29
|
#private;
|
|
29
30
|
constructor(params: {
|
|
30
31
|
dbPath: string;
|
|
32
|
+
curationPath?: string;
|
|
31
33
|
workspaceDir: string;
|
|
32
34
|
sources: readonly ResolvedSource[];
|
|
33
35
|
storeFactory?: () => Promise<ManagerStore>;
|
|
@@ -44,7 +46,24 @@ export declare class QmdMemoryManager implements MemorySearchManagerContract {
|
|
|
44
46
|
fetchCluster(params: {
|
|
45
47
|
clusterId: string;
|
|
46
48
|
topK?: number;
|
|
49
|
+
offset?: number;
|
|
50
|
+
sort?: MemoryClusterSort;
|
|
47
51
|
}): Promise<MemoryClusterDetail>;
|
|
52
|
+
listMaintenanceTasks(params?: {
|
|
53
|
+
status?: MaintenanceStatus;
|
|
54
|
+
limit?: number;
|
|
55
|
+
}): import("./curation.js").MaintenanceTask[];
|
|
56
|
+
updateMaintenanceTask(params: {
|
|
57
|
+
id: string;
|
|
58
|
+
status: Exclude<MaintenanceStatus, "pending">;
|
|
59
|
+
note?: string;
|
|
60
|
+
annotation?: {
|
|
61
|
+
scope: "chunk" | "document";
|
|
62
|
+
eventTime: string;
|
|
63
|
+
basis: TemporalBasis;
|
|
64
|
+
evidence: string;
|
|
65
|
+
};
|
|
66
|
+
}): import("./curation.js").MaintenanceTask | undefined;
|
|
48
67
|
search(query: string, opts?: CorpusSearchOptions): Promise<CorpusMemorySearchResult[]>;
|
|
49
68
|
readFile(params: {
|
|
50
69
|
relPath: string;
|
package/dist/src/manager.js
CHANGED
|
@@ -2,6 +2,7 @@ import { mkdir, stat } from "node:fs/promises";
|
|
|
2
2
|
import { dirname } from "node:path";
|
|
3
3
|
import chokidar from "chokidar";
|
|
4
4
|
import { ensureMemoryAnalysisSchema, latestAnalysisRunId, markMemoryAnalysisStale, readAnalysisSummary, readCluster, readClusters, runAnalysisWorker, } from "./analysis.js";
|
|
5
|
+
import { CurationStore, chunkFingerprint, } from "./curation.js";
|
|
5
6
|
import { readSessionManifest, sessionMetadataByPath, syncSessionProjections, } from "./session-sync.js";
|
|
6
7
|
import { parseSafeVirtualPath } from "./sources.js";
|
|
7
8
|
const DEFAULT_READ_LINES = 120;
|
|
@@ -135,6 +136,7 @@ function sessionAllowedPaths(metadataByPath, collection, filter) {
|
|
|
135
136
|
export class QmdMemoryManager {
|
|
136
137
|
#dbPath;
|
|
137
138
|
#workspaceDir;
|
|
139
|
+
#curationPath;
|
|
138
140
|
#sources;
|
|
139
141
|
#storeFactory;
|
|
140
142
|
#keepModelsWarm;
|
|
@@ -142,6 +144,7 @@ export class QmdMemoryManager {
|
|
|
142
144
|
#analysisRunner;
|
|
143
145
|
#sessions;
|
|
144
146
|
#store;
|
|
147
|
+
#curation;
|
|
145
148
|
#cleanupRemovedDocuments;
|
|
146
149
|
#operationChain;
|
|
147
150
|
#watcher;
|
|
@@ -155,6 +158,7 @@ export class QmdMemoryManager {
|
|
|
155
158
|
#sessionManifestMtimeNs;
|
|
156
159
|
constructor(params) {
|
|
157
160
|
this.#dbPath = params.dbPath;
|
|
161
|
+
this.#curationPath = params.curationPath ?? `${params.dbPath}.curation.sqlite`;
|
|
158
162
|
this.#workspaceDir = params.workspaceDir;
|
|
159
163
|
this.#sources = new Map(params.sources.map((source) => [source.collection, source]));
|
|
160
164
|
this.#storeFactory = params.storeFactory;
|
|
@@ -391,7 +395,126 @@ export class QmdMemoryManager {
|
|
|
391
395
|
return this.#enqueue(async () => readClusters((await this.#getAnalysisStore()).internal.db, limit));
|
|
392
396
|
}
|
|
393
397
|
fetchCluster(params) {
|
|
394
|
-
return this.#enqueue(async () =>
|
|
398
|
+
return this.#enqueue(async () => {
|
|
399
|
+
const db = (await this.#getAnalysisStore()).internal.db;
|
|
400
|
+
this.#loadTemporalAnnotations(db);
|
|
401
|
+
const detail = readCluster(db, params.clusterId, params.topK, params.offset, params.sort, { sessionCollection: this.#sessions?.collection });
|
|
402
|
+
if (params.sort === "date_asc" || params.sort === "date_desc") {
|
|
403
|
+
for (const member of detail.members ?? []) {
|
|
404
|
+
if (member.eventTime !== null)
|
|
405
|
+
continue;
|
|
406
|
+
const safe = parseSafeVirtualPath(member.eventTimeSource, this.#sources);
|
|
407
|
+
if (!safe)
|
|
408
|
+
continue;
|
|
409
|
+
this.#getCuration().addTask({
|
|
410
|
+
type: "ambiguous_event_time",
|
|
411
|
+
corpus: safe.source.corpus,
|
|
412
|
+
collection: safe.source.collection,
|
|
413
|
+
path: safe.relativePath,
|
|
414
|
+
reason: "cluster chronology has no reliable event time",
|
|
415
|
+
contentFingerprint: member.contentFingerprint,
|
|
416
|
+
detail: "Inspect the document and relevant evidence; annotate a date only when one can be supported.",
|
|
417
|
+
});
|
|
418
|
+
}
|
|
419
|
+
}
|
|
420
|
+
if (detail.runId && detail.members) {
|
|
421
|
+
this.#addDuplicateTasks(db, detail.runId, detail.members);
|
|
422
|
+
}
|
|
423
|
+
return detail;
|
|
424
|
+
});
|
|
425
|
+
}
|
|
426
|
+
listMaintenanceTasks(params = {}) {
|
|
427
|
+
return this.#getCuration().listTasks(params);
|
|
428
|
+
}
|
|
429
|
+
updateMaintenanceTask(params) {
|
|
430
|
+
return this.#getCuration().updateTask(params);
|
|
431
|
+
}
|
|
432
|
+
#getCuration() {
|
|
433
|
+
this.#curation ??= new CurationStore(this.#curationPath);
|
|
434
|
+
return this.#curation;
|
|
435
|
+
}
|
|
436
|
+
#loadTemporalAnnotations(db) {
|
|
437
|
+
db.exec("DELETE FROM memory_temporal_annotations");
|
|
438
|
+
const findChunks = db.prepare(`
|
|
439
|
+
SELECT d.hash, vectors.seq, vectors.pos, vectors.chunk_len, content.doc
|
|
440
|
+
FROM documents d
|
|
441
|
+
JOIN content ON content.hash = d.hash
|
|
442
|
+
JOIN content_vectors vectors ON vectors.hash = d.hash
|
|
443
|
+
WHERE d.collection = ? AND d.path = ? AND d.active = 1
|
|
444
|
+
ORDER BY vectors.seq
|
|
445
|
+
`);
|
|
446
|
+
const insert = db.prepare(`
|
|
447
|
+
INSERT OR REPLACE INTO memory_temporal_annotations
|
|
448
|
+
(collection, path, qmd_hash, qmd_seq, event_time, basis, document_wide)
|
|
449
|
+
VALUES (?, ?, ?, ?, ?, ?, ?)
|
|
450
|
+
`);
|
|
451
|
+
const curation = this.#getCuration();
|
|
452
|
+
for (const annotation of curation.annotations()) {
|
|
453
|
+
if (!annotation.contentFingerprint) {
|
|
454
|
+
insert.run(annotation.collection, annotation.path, null, null, annotation.eventTime, annotation.basis, 1);
|
|
455
|
+
continue;
|
|
456
|
+
}
|
|
457
|
+
const rows = findChunks.all(annotation.collection, annotation.path);
|
|
458
|
+
const matched = rows.find((row) => chunkFingerprint(row.doc.slice(row.pos, row.pos + row.chunk_len)) === annotation.contentFingerprint);
|
|
459
|
+
curation.updateAnnotationLocation({
|
|
460
|
+
annotation,
|
|
461
|
+
qmdHash: matched?.hash ?? null,
|
|
462
|
+
qmdSeq: matched?.seq ?? null,
|
|
463
|
+
});
|
|
464
|
+
if (matched) {
|
|
465
|
+
insert.run(annotation.collection, annotation.path, matched.hash, matched.seq, annotation.eventTime, annotation.basis, 0);
|
|
466
|
+
}
|
|
467
|
+
}
|
|
468
|
+
}
|
|
469
|
+
#addDuplicateTasks(db, runId, members) {
|
|
470
|
+
if (members.length === 0)
|
|
471
|
+
return;
|
|
472
|
+
const pageMatch = members.map(() => "(duplicates.canonical_hash = ? AND duplicates.canonical_seq = ?) OR " +
|
|
473
|
+
"(duplicates.duplicate_hash = ? AND duplicates.duplicate_seq = ?)").join(" OR ");
|
|
474
|
+
const pageParams = members.flatMap((member) => [member.hash, member.seq, member.hash, member.seq]);
|
|
475
|
+
const sessionCollections = [...this.#sources.values()]
|
|
476
|
+
.filter((source) => source.kind === "sessions")
|
|
477
|
+
.map((source) => source.collection);
|
|
478
|
+
const excludeSessions = sessionCollections.length > 0
|
|
479
|
+
? `duplicate_document.collection NOT IN (${sessionCollections.map(() => "?").join(", ")})`
|
|
480
|
+
: "1 = 1";
|
|
481
|
+
const rows = db.prepare(`
|
|
482
|
+
SELECT
|
|
483
|
+
duplicate_document.collection,
|
|
484
|
+
duplicate_document.path,
|
|
485
|
+
duplicates.content_fingerprint,
|
|
486
|
+
COUNT(*) AS occurrence_count
|
|
487
|
+
FROM memory_analysis_duplicate_occurrences duplicates
|
|
488
|
+
JOIN (SELECT DISTINCT hash FROM documents WHERE active = 1) canonical_document
|
|
489
|
+
ON canonical_document.hash = duplicates.canonical_hash
|
|
490
|
+
JOIN documents duplicate_document
|
|
491
|
+
ON duplicate_document.hash = duplicates.duplicate_hash
|
|
492
|
+
AND duplicate_document.active = 1
|
|
493
|
+
WHERE duplicates.run_id = ?
|
|
494
|
+
AND (${pageMatch})
|
|
495
|
+
AND ${excludeSessions}
|
|
496
|
+
GROUP BY duplicate_document.collection, duplicate_document.path,
|
|
497
|
+
duplicates.content_fingerprint
|
|
498
|
+
ORDER BY duplicate_document.collection, duplicate_document.path,
|
|
499
|
+
duplicates.content_fingerprint
|
|
500
|
+
LIMIT 10
|
|
501
|
+
`).all(runId, ...pageParams, ...sessionCollections);
|
|
502
|
+
const curation = this.#getCuration();
|
|
503
|
+
for (const row of rows) {
|
|
504
|
+
const source = this.#sources.get(row.collection);
|
|
505
|
+
if (!source || source.kind === "sessions")
|
|
506
|
+
continue;
|
|
507
|
+
curation.addTask({
|
|
508
|
+
type: "exact_duplicate",
|
|
509
|
+
corpus: source.corpus,
|
|
510
|
+
collection: row.collection,
|
|
511
|
+
path: row.path,
|
|
512
|
+
reason: "exact chunk content repeats in this source document",
|
|
513
|
+
contentFingerprint: row.content_fingerprint,
|
|
514
|
+
detail: `${row.occurrence_count} exact duplicate occurrence${row.occurrence_count === 1 ? "" : "s"}. ` +
|
|
515
|
+
"Review the source and propose cleanup only if repetition is accidental.",
|
|
516
|
+
});
|
|
517
|
+
}
|
|
395
518
|
}
|
|
396
519
|
async #getAnalysisStore() {
|
|
397
520
|
const store = await this.#getStore();
|
|
@@ -529,5 +652,7 @@ export class QmdMemoryManager {
|
|
|
529
652
|
await this.#operationChain?.catch(() => undefined);
|
|
530
653
|
await this.#store?.close();
|
|
531
654
|
this.#store = undefined;
|
|
655
|
+
this.#curation?.close();
|
|
656
|
+
this.#curation = undefined;
|
|
532
657
|
}
|
|
533
658
|
}
|
package/dist/src/plugin.js
CHANGED
|
@@ -192,6 +192,14 @@ function createListClustersTool(runtime, ctx) {
|
|
|
192
192
|
const fetchClusterParameters = Type.Object({
|
|
193
193
|
clusterId: Type.String({ pattern: "^[0-9a-f]{10}$" }),
|
|
194
194
|
topK: Type.Optional(Type.Integer({ minimum: 1, maximum: 50 })),
|
|
195
|
+
offset: Type.Optional(Type.Integer({ minimum: 0 })),
|
|
196
|
+
sort: Type.Optional(Type.Union([
|
|
197
|
+
Type.Literal("representative"),
|
|
198
|
+
Type.Literal("score_desc"),
|
|
199
|
+
Type.Literal("score_asc"),
|
|
200
|
+
Type.Literal("date_desc"),
|
|
201
|
+
Type.Literal("date_asc"),
|
|
202
|
+
])),
|
|
195
203
|
}, { additionalProperties: false });
|
|
196
204
|
function createFetchClusterTool(runtime, ctx) {
|
|
197
205
|
const active = getContext(ctx);
|
|
@@ -200,14 +208,89 @@ function createFetchClusterTool(runtime, ctx) {
|
|
|
200
208
|
return {
|
|
201
209
|
name: "memory_fetch_cluster",
|
|
202
210
|
label: "Fetch Memory Cluster",
|
|
203
|
-
description: "Fetch
|
|
211
|
+
description: "Fetch a sorted page of QMD chunks for a clusterId returned by memory_list_clusters.",
|
|
204
212
|
parameters: fetchClusterParameters,
|
|
205
213
|
async execute(_toolCallId, params) {
|
|
206
|
-
const { clusterId, topK } = Value.Parse(fetchClusterParameters, params);
|
|
214
|
+
const { clusterId, topK, offset, sort } = Value.Parse(fetchClusterParameters, params);
|
|
207
215
|
const { manager, error } = await runtime.getMemorySearchManager(active);
|
|
208
216
|
if (!manager)
|
|
209
217
|
return jsonResult({ status: "unavailable", error: error ?? "memory unavailable" });
|
|
210
|
-
return jsonResult(await manager.fetchCluster({ clusterId, topK }));
|
|
218
|
+
return jsonResult(await manager.fetchCluster({ clusterId, topK, offset, sort }));
|
|
219
|
+
},
|
|
220
|
+
};
|
|
221
|
+
}
|
|
222
|
+
const maintenanceStatus = Type.Union([
|
|
223
|
+
Type.Literal("pending"),
|
|
224
|
+
Type.Literal("resolved"),
|
|
225
|
+
Type.Literal("deferred"),
|
|
226
|
+
Type.Literal("irrelevant"),
|
|
227
|
+
]);
|
|
228
|
+
const listMaintenanceParameters = Type.Object({
|
|
229
|
+
status: Type.Optional(maintenanceStatus),
|
|
230
|
+
limit: Type.Optional(Type.Integer({ minimum: 1, maximum: 10 })),
|
|
231
|
+
}, { additionalProperties: false });
|
|
232
|
+
function createListMaintenanceTool(runtime, ctx) {
|
|
233
|
+
const active = getContext(ctx);
|
|
234
|
+
if (!active)
|
|
235
|
+
return null;
|
|
236
|
+
return {
|
|
237
|
+
name: "memory_list_maintenance_tasks",
|
|
238
|
+
label: "List Memory Maintenance Tasks",
|
|
239
|
+
description: "List a bounded curation inbox of memory chronology and duplicate-review proposals.",
|
|
240
|
+
parameters: listMaintenanceParameters,
|
|
241
|
+
async execute(_toolCallId, params) {
|
|
242
|
+
const options = Value.Parse(listMaintenanceParameters, params);
|
|
243
|
+
const { manager, error } = await runtime.getMemorySearchManager(active);
|
|
244
|
+
if (!manager)
|
|
245
|
+
return jsonResult({ status: "unavailable", error: error ?? "memory unavailable" });
|
|
246
|
+
return jsonResult({ status: "ok", tasks: manager.listMaintenanceTasks(options) });
|
|
247
|
+
},
|
|
248
|
+
};
|
|
249
|
+
}
|
|
250
|
+
const isoTimestamp = Type.String({
|
|
251
|
+
pattern: "^\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}(?:\\.\\d{1,9})?(?:Z|[+-]\\d{2}:\\d{2})$",
|
|
252
|
+
});
|
|
253
|
+
const updateMaintenanceParameters = Type.Object({
|
|
254
|
+
taskId: Type.String({ pattern: "\\S" }),
|
|
255
|
+
action: Type.Union([
|
|
256
|
+
Type.Literal("resolve"),
|
|
257
|
+
Type.Literal("defer"),
|
|
258
|
+
Type.Literal("irrelevant"),
|
|
259
|
+
]),
|
|
260
|
+
note: Type.Optional(Type.String({ minLength: 1, maxLength: 500 })),
|
|
261
|
+
annotation: Type.Optional(Type.Object({
|
|
262
|
+
scope: Type.Optional(Type.Union([Type.Literal("chunk"), Type.Literal("document")])),
|
|
263
|
+
eventTime: isoTimestamp,
|
|
264
|
+
basis: Type.Union([
|
|
265
|
+
Type.Literal("path"),
|
|
266
|
+
Type.Literal("frontmatter"),
|
|
267
|
+
Type.Literal("session"),
|
|
268
|
+
Type.Literal("agent_verified"),
|
|
269
|
+
]),
|
|
270
|
+
evidence: Type.String({ minLength: 1, maxLength: 500 }),
|
|
271
|
+
}, { additionalProperties: false })),
|
|
272
|
+
}, { additionalProperties: false });
|
|
273
|
+
function createUpdateMaintenanceTool(runtime, ctx) {
|
|
274
|
+
const active = getContext(ctx);
|
|
275
|
+
if (!active)
|
|
276
|
+
return null;
|
|
277
|
+
return {
|
|
278
|
+
name: "memory_update_maintenance_task",
|
|
279
|
+
label: "Update Memory Maintenance Task",
|
|
280
|
+
description: "Resolve, defer, or dismiss a memory-maintenance proposal. This tool never edits source Markdown.",
|
|
281
|
+
parameters: updateMaintenanceParameters,
|
|
282
|
+
async execute(_toolCallId, params) {
|
|
283
|
+
const { taskId, action, note, annotation } = Value.Parse(updateMaintenanceParameters, params);
|
|
284
|
+
const { manager, error } = await runtime.getMemorySearchManager(active);
|
|
285
|
+
if (!manager)
|
|
286
|
+
return jsonResult({ status: "unavailable", error: error ?? "memory unavailable" });
|
|
287
|
+
const updated = manager.updateMaintenanceTask({
|
|
288
|
+
id: taskId,
|
|
289
|
+
status: action === "resolve" ? "resolved" : action === "defer" ? "deferred" : "irrelevant",
|
|
290
|
+
note,
|
|
291
|
+
...(annotation ? { annotation: { ...annotation, scope: annotation.scope ?? "chunk" } } : {}),
|
|
292
|
+
});
|
|
293
|
+
return jsonResult(updated ? { status: "ok", task: updated } : { status: "not_found" });
|
|
211
294
|
},
|
|
212
295
|
};
|
|
213
296
|
}
|
|
@@ -304,4 +387,6 @@ export function registerUnblockMemory(api) {
|
|
|
304
387
|
api.registerTool((ctx) => createReclusterTool(runtime, ctx), { names: ["memory_recluster"] });
|
|
305
388
|
api.registerTool((ctx) => createListClustersTool(runtime, ctx), { names: ["memory_list_clusters"] });
|
|
306
389
|
api.registerTool((ctx) => createFetchClusterTool(runtime, ctx), { names: ["memory_fetch_cluster"] });
|
|
390
|
+
api.registerTool((ctx) => createListMaintenanceTool(runtime, ctx), { names: ["memory_list_maintenance_tasks"] });
|
|
391
|
+
api.registerTool((ctx) => createUpdateMaintenanceTool(runtime, ctx), { names: ["memory_update_maintenance_task"] });
|
|
307
392
|
}
|
package/dist/src/runtime.js
CHANGED
|
@@ -204,6 +204,7 @@ export class QmdMemoryRuntime {
|
|
|
204
204
|
const manager = new QmdMemoryManager({
|
|
205
205
|
workspaceDir,
|
|
206
206
|
dbPath: join(stateDir, "index.sqlite"),
|
|
207
|
+
curationPath: join(stateDir, "curation.sqlite"),
|
|
207
208
|
sources,
|
|
208
209
|
keepModelsWarm: this.#keepEmbeddingModelWarm,
|
|
209
210
|
analysisExecutable: this.#analysisExecutable,
|
|
@@ -100,9 +100,6 @@ function formatTimestamp(value, timezone) {
|
|
|
100
100
|
return `${part("year")}-${part("month")}-${part("day")} ` +
|
|
101
101
|
`${part("hour")}:${part("minute")}:${part("second")} ${part("timeZoneName")}`.trim();
|
|
102
102
|
}
|
|
103
|
-
function inline(value) {
|
|
104
|
-
return value.replace(/[\r\n]+/gu, " ").replaceAll("`", "\\`");
|
|
105
|
-
}
|
|
106
103
|
export function projectSession(input) {
|
|
107
104
|
const messages = input.events.flatMap((event) => {
|
|
108
105
|
const projected = projectMessage(event, input);
|
|
@@ -110,22 +107,8 @@ export function projectSession(input) {
|
|
|
110
107
|
});
|
|
111
108
|
if (messages.length === 0)
|
|
112
109
|
return undefined;
|
|
113
|
-
const provider = input.provider ?? "unknown";
|
|
114
|
-
const header = [
|
|
115
|
-
"# Session",
|
|
116
|
-
"",
|
|
117
|
-
`- Session ID: \`${inline(input.sessionId)}\``,
|
|
118
|
-
`- Provider: ${inline(provider)}`,
|
|
119
|
-
`- Chat type: ${input.chatType}`,
|
|
120
|
-
...(input.label ? [`- Conversation: ${inline(input.label)}`] : []),
|
|
121
|
-
...(input.conversationId ? [`- Conversation ID: \`${inline(input.conversationId)}\``] : []),
|
|
122
|
-
`- Started: ${new Date(input.startedAt).toISOString()}`,
|
|
123
|
-
"",
|
|
124
|
-
"## Transcript",
|
|
125
|
-
"",
|
|
126
|
-
];
|
|
127
110
|
const transcript = messages.map((message) => `${formatTimestamp(message.timestamp, input.timezone)} — ${message.speaker}: ${message.text}`);
|
|
128
|
-
return
|
|
111
|
+
return `# Transcript\n\n${transcript.join("\n\n")}\n`;
|
|
129
112
|
}
|
|
130
113
|
function hash(value) {
|
|
131
114
|
return createHash("sha256").update(value).digest("hex").slice(0, 16);
|
package/dist/src/session-sync.js
CHANGED
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
import { createHash, randomUUID } from "node:crypto";
|
|
2
2
|
import { existsSync, lstatSync } from "node:fs";
|
|
3
|
-
import { chmod, mkdir, readFile, rename, unlink, writeFile } from "node:fs/promises";
|
|
3
|
+
import { chmod, mkdir, readFile, rename, unlink, utimes, writeFile } from "node:fs/promises";
|
|
4
4
|
import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path";
|
|
5
5
|
import { DatabaseSync } from "node:sqlite";
|
|
6
6
|
import { projectSession, sessionDocumentPath, } from "./session-projector.js";
|
|
7
7
|
const MANIFEST_VERSION = 1;
|
|
8
|
-
const PROJECTOR_VERSION =
|
|
8
|
+
const PROJECTOR_VERSION = 2;
|
|
9
9
|
const SUPPORTED_SCHEMA_VERSION = 17;
|
|
10
10
|
const REQUIRED_COLUMNS = {
|
|
11
11
|
schema_meta: ["meta_key", "role", "schema_version", "agent_id", "app_version"],
|
|
@@ -261,6 +261,7 @@ export async function syncSessionProjections(params) {
|
|
|
261
261
|
}
|
|
262
262
|
const target = projectionPath(params.outputDir, documentPath);
|
|
263
263
|
await atomicWrite(target, content, 0o600);
|
|
264
|
+
await utimes(target, new Date(), new Date(metadata.startedAt));
|
|
264
265
|
if (previous?.documentPath && previous.documentPath !== documentPath) {
|
|
265
266
|
await remove(projectionPath(params.outputDir, previous.documentPath));
|
|
266
267
|
}
|
package/openclaw.plugin.json
CHANGED
|
@@ -1,18 +1,20 @@
|
|
|
1
1
|
{
|
|
2
2
|
"id": "unblock-memory",
|
|
3
3
|
"name": "Unblock Memory",
|
|
4
|
-
"version": "0.2.
|
|
4
|
+
"version": "0.2.7",
|
|
5
5
|
"description": "Indexes, retrieves, and analyzes configured workspace memory with existing QMD vectors.",
|
|
6
6
|
"kind": "memory",
|
|
7
7
|
"activation": { "onStartup": false },
|
|
8
8
|
"skills": ["./skills"],
|
|
9
|
-
"contracts": { "tools": ["memory_search", "memory_get", "memory_sync_sessions", "memory_sync_status", "memory_recluster", "memory_list_clusters", "memory_fetch_cluster"] },
|
|
9
|
+
"contracts": { "tools": ["memory_search", "memory_get", "memory_sync_sessions", "memory_sync_status", "memory_recluster", "memory_list_clusters", "memory_fetch_cluster", "memory_list_maintenance_tasks", "memory_update_maintenance_task"] },
|
|
10
10
|
"toolMetadata": {
|
|
11
11
|
"memory_sync_sessions": { "sideEffecting": true },
|
|
12
12
|
"memory_sync_status": { "replaySafe": true },
|
|
13
13
|
"memory_recluster": { "sideEffecting": true },
|
|
14
14
|
"memory_list_clusters": { "replaySafe": true },
|
|
15
|
-
"memory_fetch_cluster": { "replaySafe": true }
|
|
15
|
+
"memory_fetch_cluster": { "replaySafe": true },
|
|
16
|
+
"memory_list_maintenance_tasks": { "replaySafe": true },
|
|
17
|
+
"memory_update_maintenance_task": { "sideEffecting": true }
|
|
16
18
|
},
|
|
17
19
|
"uiHints": {
|
|
18
20
|
"keepEmbeddingModelWarm": {
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@unblocklabs/unblock-memory",
|
|
3
|
-
"version": "0.2.
|
|
3
|
+
"version": "0.2.7",
|
|
4
4
|
"description": "Workspace-native memory for OpenClaw, powered by QMD",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
|
@@ -30,7 +30,7 @@
|
|
|
30
30
|
"preflight": "npm run knip && npm run build && npm run typecheck && npm test && npm run plugin:inspect && npm run plugin:inspect:runtime && npm pack --dry-run"
|
|
31
31
|
},
|
|
32
32
|
"dependencies": {
|
|
33
|
-
"@unblocklabs/qmd": "https://github.com/unblocklabs-ai/qmd/releases/download/v2.9.
|
|
33
|
+
"@unblocklabs/qmd": "https://github.com/unblocklabs-ai/qmd/releases/download/v2.9.3/unblocklabs-qmd-2.9.3.tgz",
|
|
34
34
|
"chokidar": "5.0.0",
|
|
35
35
|
"picomatch": "^4.0.5",
|
|
36
36
|
"typebox": "1.3.6"
|
|
@@ -1,78 +1,84 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: memory-curator
|
|
3
|
-
description:
|
|
3
|
+
description: Investigate Unblock Memory clusters and maintain supported, agent-specific knowledge that would otherwise be difficult to reconstruct.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Memory Curator
|
|
7
7
|
|
|
8
|
-
Use
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
8
|
+
Use semantic clusters as attention signals for maintaining the agent's current
|
|
9
|
+
understanding of its world. A cluster shows similarity, not a complete timeline,
|
|
10
|
+
truth, or consensus. Do not write from a cluster alone, and prefer no write over
|
|
11
|
+
weak, duplicative, or easily looked-up knowledge.
|
|
12
|
+
|
|
13
|
+
## Investigate
|
|
14
|
+
|
|
15
|
+
1. Call `memory_list_clusters`. If analysis is missing or stale, call
|
|
16
|
+
`memory_recluster`, then list again.
|
|
17
|
+
2. Fetch a useful cluster with `memory_fetch_cluster`. Start with
|
|
18
|
+
`sort: "representative"`; use `score_desc`, `date_asc`, or `date_desc` and
|
|
19
|
+
pagination when relevance, evolution, or recent state matters. Treat
|
|
20
|
+
`eventTime` as event chronology; `sourceModifiedAt` is only a labeled
|
|
21
|
+
fallback when `eventTime` is unresolved.
|
|
22
|
+
3. State the question the cluster raises: what may be repeated, contradictory,
|
|
23
|
+
changing, or worth understanding?
|
|
24
|
+
4. Search existing knowledge with `memory_search`, using
|
|
25
|
+
`corpora: ["knowledge"]`. If that corpus is not configured, report that and
|
|
26
|
+
do not create an unindexed file.
|
|
27
|
+
5. Investigate the evidence needed to answer the question. Follow important
|
|
28
|
+
`qmd://` sources with `memory_get`, search adjacent memory or sessions, inspect
|
|
29
|
+
other clusters, and check live systems, local files, authoritative docs, or
|
|
30
|
+
the web when they are the right source. The selected cluster is not assumed
|
|
31
|
+
to contain the whole timeline or the current truth.
|
|
32
|
+
6. Distinguish underlying evidence from earlier agent-created knowledge.
|
|
33
|
+
Derived repetition can locate or challenge a conclusion, but is not
|
|
34
|
+
independent corroboration.
|
|
35
|
+
|
|
36
|
+
## Decide what belongs in knowledge
|
|
37
|
+
|
|
38
|
+
Write knowledge when the result is agent-specific, evolving, and expensive to
|
|
39
|
+
reconstruct, such as current fleet composition, relationships, preferences,
|
|
40
|
+
project decisions, local conventions, or an assessment synthesized across
|
|
41
|
+
time.
|
|
42
|
+
|
|
43
|
+
Prefer a lookup for public or vendor-owned facts, generic command syntax, and
|
|
44
|
+
behavior likely to change with a third-party release. Those sources may verify
|
|
45
|
+
a local conclusion, but do not copy ordinary documentation into knowledge. A
|
|
46
|
+
local policy or deliberate divergence can belong when its local meaning and
|
|
47
|
+
rationale are the durable part.
|
|
48
|
+
|
|
49
|
+
Choose one outcome: update knowledge or no write. No write is successful when
|
|
50
|
+
the evidence is insufficient, the information is already accurate, or lookup
|
|
51
|
+
is better than memory.
|
|
52
|
+
|
|
53
|
+
## Maintain knowledge
|
|
54
|
+
|
|
55
|
+
Use a stable `knowledge/<topic>.md` file and update it in place. Facts, agent
|
|
56
|
+
assessments, and uncertainty may coexist. Qualify each claim where it appears so
|
|
57
|
+
it remains honest if semantic chunking retrieves it alone:
|
|
58
|
+
|
|
59
|
+
- identify what is verified and when or against which current source;
|
|
60
|
+
- label an interpretation as the agent's assessment rather than a settled fact;
|
|
61
|
+
- state uncertainty or an open question directly with the affected claim.
|
|
62
|
+
|
|
63
|
+
Keep the file focused on current understanding. Remove stale conclusions rather
|
|
64
|
+
than retaining `Supersedes`, history, changelog, migration, or old-process
|
|
65
|
+
sections. Raw memory and sessions preserve the evidence history. Include a
|
|
66
|
+
human-readable update time and useful evidence citations, but do not force a
|
|
67
|
+
rigid document template.
|
|
68
68
|
|
|
69
69
|
## Finish the cycle
|
|
70
70
|
|
|
71
71
|
- Do not rewrite raw memory or session projections.
|
|
72
|
-
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
72
|
+
- Review a small page from `memory_list_maintenance_tasks`. For ambiguous dates,
|
|
73
|
+
investigate supporting evidence and use `memory_update_maintenance_task` to
|
|
74
|
+
attach a chunk or document date only when supported; otherwise defer or mark
|
|
75
|
+
it irrelevant. For exact-duplicate proposals, decide whether cleanup should
|
|
76
|
+
be proposed, but do not treat repetition across historical files as an error.
|
|
77
|
+
The maintenance tools never change source Markdown, and generated session
|
|
78
|
+
projections must never be manually cleaned.
|
|
79
|
+
- Verify an updated file with `memory_search`, using
|
|
80
|
+
`corpora: ["knowledge"]`, and check all-corpora ranking when useful.
|
|
81
|
+
- Report the questions investigated, evidence consulted beyond each cluster,
|
|
82
|
+
files changed, current uncertainties, and intentional no-write decisions.
|
|
77
83
|
- Do not recluster again after this cycle's writes. Let them enter the next
|
|
78
84
|
scheduled cycle so the run cannot recursively react to its own output.
|