agent-rag-mcp 0.2.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +19 -2
- package/dist/src/adapters/structured-text.mjs +111 -45
- package/dist/src/cli/format.mjs +112 -0
- package/dist/src/cli/index.mjs +63 -6
- package/dist/src/ingest/pipeline.mjs +10 -9
- package/dist/src/store/canonical.mjs +65 -0
- package/dist/src/store/identity.mjs +1 -1
- package/dist/src/store/lifecycle.mjs +2 -0
- package/dist/src/store/manifest.mjs +13 -4
- package/dist/src/store/rehash.mjs +162 -0
- package/dist/src/store/schema.mjs +11 -1
- package/package.json +6 -1
- package/skills/agent-rag/SKILL.md +35 -0
package/README.md
CHANGED
|
@@ -32,8 +32,25 @@ The `rag` commands available today are:
|
|
|
32
32
|
- `rag backup --to <dir>` — Backs up the manifest and object store.
|
|
33
33
|
- `rag restore --from <dir>` — Restores a backup under this root.
|
|
34
34
|
- `rag reindex` — Rebuilds the database from the manifest and objects.
|
|
35
|
+
- `rag rehash` — Previews recomputed hashes for stored versions; `rag rehash --apply` writes the changed hash facts in one transaction. Preview first, and back up with `rag backup --to <backup-dir>` before applying.
|
|
36
|
+
- `rag rehash --apply` writes `manifest.json` before committing the database transaction. An interruption in that window can leave the manifest ahead of the database; rerunning `rag rehash --apply` or running `rag reindex` converges to the requested facts.
|
|
35
37
|
|
|
36
|
-
|
|
38
|
+
### CLI output
|
|
39
|
+
|
|
40
|
+
Interactive terminals render document lists as aligned columns and other supported results as concise summaries. When output is piped, commands keep the single-line JSON envelope. Set `RAG_FORMAT=json` or `RAG_FORMAT=pretty` to override terminal detection; an invalid value falls back to automatic detection. `rag --help` defaults to raw usage text on a terminal, while `RAG_FORMAT=json` restores its JSON envelope for scripts that parse the legacy help output. Set `NO_COLOR` (any value, including empty) to disable ANSI styling without changing the output layout.
|
|
41
|
+
|
|
42
|
+
`rag ls` includes each document's `documentId`. Copy that value to relabel the same document:
|
|
43
|
+
|
|
44
|
+
```sh
|
|
45
|
+
rag ls
|
|
46
|
+
rag relabel <document-id> --label "replacement label"
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Structured files can use `.yaml`, `.yml`, `.json`, or `.jsonl`; each structured file yields sections from its parsed data tree. YAML uses the YAML 1.2 core schema, so words such as `yes` and `on` remain strings and unquoted timestamps remain strings.
|
|
50
|
+
|
|
51
|
+
### Content identity
|
|
52
|
+
|
|
53
|
+
Structured JSON, JSONL, and YAML are hashed from compact canonical JSON for their parsed data tree, without a format prefix. Their `doc_version` rows use `content_kind='structured'` and `hash_version=2`; equivalent trees therefore share a content hash across formats and key order. Other formats retain the existing normalized extracted-text hash and use `content_kind=NULL` with `hash_version=1`. Existing version rows stay at version 1 with a null content kind until repaired with `rag rehash`; the repair never rewrites stored object bytes and skips structured documents that cannot be canonically hashed.
|
|
37
54
|
|
|
38
55
|
The default knowledge-base root is `%USERPROFILE%\.rag\`. Each client has its own `rag.db` under `.rag\clients\<client>\`; projects are logical within that client and are identified by `project_id`. The namespace is the pair `(client, project)`. By default, `failClosed` is `true`; without a namespace, reads are allowed, and when `failClosed` is set to `false`, writes go to the reserved `default/default` namespace. The Tier 3 human review queue is available through `rag review list`.
|
|
39
56
|
|
|
@@ -78,7 +95,7 @@ The namespace travels in every `rag_find` or `rag_read` call; it does not come f
|
|
|
78
95
|
- `npm test` runs the tests with Node's `node:test`.
|
|
79
96
|
- `npm run typecheck` runs the TypeScript typecheck without emitting files.
|
|
80
97
|
|
|
81
|
-
## Known limits
|
|
98
|
+
## Known limits
|
|
82
99
|
|
|
83
100
|
Version 0.1.0 has no embeddings or Tier 2; scanned PDFs go through OCR. Document parsing runs in a child process with a 512 MB V8 heap cap.
|
|
84
101
|
|
|
@@ -1,3 +1,5 @@
|
|
|
1
|
+
import { parseDocument } from "yaml";
|
|
2
|
+
import { canonicalizeStructuredContent, NonFiniteNumberCanonicalizationError } from "../store/canonical.mjs";
|
|
1
3
|
class StructuredTextExtractionError extends Error {
|
|
2
4
|
code;
|
|
3
5
|
constructor(code) {
|
|
@@ -29,70 +31,127 @@ function supportsReference(ref, options) {
|
|
|
29
31
|
}
|
|
30
32
|
function renderJsonValue(value) {
|
|
31
33
|
if (typeof value === "string") return value;
|
|
32
|
-
return JSON.stringify(value,
|
|
34
|
+
return JSON.stringify(value, (_key, child) => typeof child === "bigint" ? child.toString() : child, 2) ?? "null";
|
|
35
|
+
}
|
|
36
|
+
function parseJsonContent(content) {
|
|
37
|
+
return JSON.parse(content, (_key, value, context) => {
|
|
38
|
+
const source = context?.source;
|
|
39
|
+
if (typeof value === "number" && !Number.isSafeInteger(value) && source !== void 0 && /^-?(?:0|[1-9][0-9]*)$/.test(source)) {
|
|
40
|
+
return BigInt(source);
|
|
41
|
+
}
|
|
42
|
+
return value;
|
|
43
|
+
});
|
|
44
|
+
}
|
|
45
|
+
function canonicalFields(value) {
|
|
46
|
+
try {
|
|
47
|
+
return { canonicalContent: canonicalizeStructuredContent(value) };
|
|
48
|
+
} catch (error) {
|
|
49
|
+
if (error instanceof NonFiniteNumberCanonicalizationError) return { canonicalizationError: error.code };
|
|
50
|
+
throw error;
|
|
51
|
+
}
|
|
33
52
|
}
|
|
34
53
|
function extractJson(content) {
|
|
35
54
|
let parsed;
|
|
36
55
|
try {
|
|
37
|
-
parsed =
|
|
56
|
+
parsed = parseJsonContent(content);
|
|
38
57
|
} catch {
|
|
39
58
|
throw new StructuredTextExtractionError("extraction_invalid_json");
|
|
40
59
|
}
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
}
|
|
49
|
-
return [{ ordinal: 0, heading: null, level: null, text: renderJsonValue(parsed) }];
|
|
60
|
+
const sections = typeof parsed === "object" && parsed !== null && !Array.isArray(parsed) ? Object.entries(parsed).map(([heading, value], ordinal) => ({
|
|
61
|
+
ordinal,
|
|
62
|
+
heading,
|
|
63
|
+
level: 1,
|
|
64
|
+
text: renderJsonValue(value)
|
|
65
|
+
})) : [{ ordinal: 0, heading: null, level: null, text: renderJsonValue(parsed) }];
|
|
66
|
+
return { sections, ...canonicalFields(parsed) };
|
|
50
67
|
}
|
|
51
68
|
function extractJsonl(content) {
|
|
52
69
|
const sections = [];
|
|
70
|
+
const records = [];
|
|
53
71
|
for (const line of content.split(/\r?\n/)) {
|
|
54
72
|
const text = line.trim();
|
|
55
73
|
if (text.length === 0) continue;
|
|
56
74
|
try {
|
|
57
|
-
|
|
75
|
+
records.push(parseJsonContent(text));
|
|
58
76
|
} catch {
|
|
59
77
|
throw new StructuredTextExtractionError("extraction_invalid_json");
|
|
60
78
|
}
|
|
61
79
|
sections.push({ ordinal: sections.length, heading: null, level: null, text });
|
|
62
80
|
}
|
|
63
|
-
return sections;
|
|
81
|
+
return { sections, ...canonicalFields(records) };
|
|
64
82
|
}
|
|
65
|
-
function
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
83
|
+
function normalizeYamlValue(value, ancestors = /* @__PURE__ */ new Set()) {
|
|
84
|
+
if (typeof value !== "object" || value === null) return value;
|
|
85
|
+
if (ancestors.has(value)) throw new StructuredTextExtractionError("extraction_invalid_yaml");
|
|
86
|
+
ancestors.add(value);
|
|
87
|
+
try {
|
|
88
|
+
if (value instanceof Map) {
|
|
89
|
+
const result2 = /* @__PURE__ */ Object.create(null);
|
|
90
|
+
for (const [key, child] of value) {
|
|
91
|
+
const stringKey = typeof key === "string" ? key : canonicalizeStructuredContent(normalizeYamlValue(key, ancestors));
|
|
92
|
+
result2[stringKey] = normalizeYamlValue(child, ancestors);
|
|
93
|
+
}
|
|
94
|
+
return result2;
|
|
95
|
+
}
|
|
96
|
+
if (Array.isArray(value)) return value.map((child) => normalizeYamlValue(child, ancestors));
|
|
97
|
+
const result = /* @__PURE__ */ Object.create(null);
|
|
98
|
+
for (const [key, child] of Object.entries(value)) result[key] = normalizeYamlValue(child, ancestors);
|
|
99
|
+
return result;
|
|
100
|
+
} finally {
|
|
101
|
+
ancestors.delete(value);
|
|
75
102
|
}
|
|
76
|
-
|
|
77
|
-
|
|
103
|
+
}
|
|
104
|
+
function renderYamlValue(value) {
|
|
105
|
+
if (typeof value === "string") return value;
|
|
106
|
+
return JSON.stringify(value, (_key, child) => typeof child === "bigint" ? child.toString() : child, 2) ?? "null";
|
|
107
|
+
}
|
|
108
|
+
function yamlTopLevelStarts(document) {
|
|
109
|
+
const root = document.contents;
|
|
110
|
+
const starts = /* @__PURE__ */ new Map();
|
|
111
|
+
for (const pair of root?.items ?? []) {
|
|
112
|
+
const position = pair.key?.range?.[0];
|
|
113
|
+
if (position === void 0) continue;
|
|
114
|
+
const rawKey = pair.key && Object.hasOwn(pair.key, "value") ? pair.key.value : pair.key?.toJSON?.();
|
|
115
|
+
const key = normalizeYamlValue(rawKey);
|
|
116
|
+
const stringKey = typeof key === "string" ? key : canonicalizeStructuredContent(key);
|
|
117
|
+
starts.set(stringKey, position);
|
|
78
118
|
}
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
119
|
+
return starts;
|
|
120
|
+
}
|
|
121
|
+
function extractYaml(content) {
|
|
122
|
+
let parsed;
|
|
123
|
+
let topLevelStarts;
|
|
124
|
+
let canonicalizationError;
|
|
125
|
+
try {
|
|
126
|
+
const document = parseDocument(content, { schema: "core", uniqueKeys: false, intAsBigInt: true });
|
|
127
|
+
if (document.errors.length > 0) throw new Error("invalid YAML document");
|
|
128
|
+
parsed = normalizeYamlValue(document.toJS({ mapAsMap: true }));
|
|
129
|
+
topLevelStarts = yamlTopLevelStarts(document);
|
|
130
|
+
} catch (error) {
|
|
131
|
+
if (!(error instanceof NonFiniteNumberCanonicalizationError)) {
|
|
132
|
+
throw new StructuredTextExtractionError("extraction_invalid_yaml");
|
|
87
133
|
}
|
|
88
|
-
|
|
89
|
-
|
|
134
|
+
canonicalizationError = error.code;
|
|
135
|
+
parsed = null;
|
|
136
|
+
topLevelStarts = /* @__PURE__ */ new Map();
|
|
90
137
|
}
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
138
|
+
if (canonicalizationError !== void 0) {
|
|
139
|
+
return { sections: [{ ordinal: 0, heading: null, level: null, text: content.trim() }], canonicalizationError };
|
|
140
|
+
}
|
|
141
|
+
let sections;
|
|
142
|
+
if (typeof parsed === "object" && parsed !== null && !Array.isArray(parsed)) {
|
|
143
|
+
const entries = Object.entries(parsed).map(([heading, value]) => ({ heading, value, start: topLevelStarts.get(heading) })).filter((entry) => entry.start !== void 0).sort((left, right) => left.start - right.start);
|
|
144
|
+
sections = entries.map(({ heading, start }, ordinal) => ({
|
|
145
|
+
ordinal,
|
|
146
|
+
heading,
|
|
147
|
+
level: 1,
|
|
148
|
+
text: content.slice(start, entries[ordinal + 1]?.start ?? content.length).trim()
|
|
149
|
+
}));
|
|
150
|
+
if (sections.length === 0) sections = [{ ordinal: 0, heading: null, level: null, text: renderYamlValue(parsed) }];
|
|
151
|
+
} else {
|
|
152
|
+
sections = [{ ordinal: 0, heading: null, level: null, text: renderYamlValue(parsed) }];
|
|
153
|
+
}
|
|
154
|
+
return { sections, ...canonicalFields(parsed) };
|
|
96
155
|
}
|
|
97
156
|
function createStructuredTextExtractor(options) {
|
|
98
157
|
return {
|
|
@@ -108,19 +167,26 @@ function createStructuredTextExtractor(options) {
|
|
|
108
167
|
if (content.length === 0) {
|
|
109
168
|
return { sections: [], versionLabel: null, mediaType: null };
|
|
110
169
|
}
|
|
111
|
-
let
|
|
170
|
+
let extracted;
|
|
112
171
|
switch (options.kind) {
|
|
113
172
|
case "json":
|
|
114
|
-
|
|
173
|
+
extracted = extractJson(content);
|
|
115
174
|
break;
|
|
116
175
|
case "jsonl":
|
|
117
|
-
|
|
176
|
+
extracted = extractJsonl(content);
|
|
118
177
|
break;
|
|
119
178
|
case "yaml":
|
|
120
|
-
|
|
179
|
+
extracted = extractYaml(content);
|
|
121
180
|
break;
|
|
122
181
|
}
|
|
123
|
-
|
|
182
|
+
const result = {
|
|
183
|
+
sections: extracted.sections,
|
|
184
|
+
versionLabel: null,
|
|
185
|
+
mediaType: options.mediaTypes[0] ?? null,
|
|
186
|
+
...extracted.canonicalContent === void 0 ? {} : { canonicalContent: extracted.canonicalContent },
|
|
187
|
+
...extracted.canonicalizationError === void 0 ? {} : { canonicalizationError: extracted.canonicalizationError }
|
|
188
|
+
};
|
|
189
|
+
return result;
|
|
124
190
|
}
|
|
125
191
|
};
|
|
126
192
|
}
|
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
const ANSI_ESCAPE = /\u001B(?:\[[0-?]*[ -/]*[@-~]|\][^\u0007]*(?:\u0007|\u001B\\))/g;
|
|
2
|
+
function record(value) {
|
|
3
|
+
return value !== null && typeof value === "object" && !Array.isArray(value) ? value : null;
|
|
4
|
+
}
|
|
5
|
+
function oneLine(value) {
|
|
6
|
+
return String(value ?? "").replace(/[\r\n]+/g, " ");
|
|
7
|
+
}
|
|
8
|
+
function styled(value, code, enabled) {
|
|
9
|
+
return enabled ? `\x1B[${code}m${value}\x1B[0m` : value;
|
|
10
|
+
}
|
|
11
|
+
function resolveOutputFormat(env, isTTY) {
|
|
12
|
+
const requested = env.RAG_FORMAT?.trim().toLowerCase();
|
|
13
|
+
if (requested === "json" || requested === "pretty") return requested;
|
|
14
|
+
return isTTY ? "pretty" : "json";
|
|
15
|
+
}
|
|
16
|
+
function useColor(env) {
|
|
17
|
+
return !Object.hasOwn(env, "NO_COLOR");
|
|
18
|
+
}
|
|
19
|
+
function stripAnsi(value) {
|
|
20
|
+
return value.replace(ANSI_ESCAPE, "");
|
|
21
|
+
}
|
|
22
|
+
function padVisible(value, width) {
|
|
23
|
+
return value.padEnd(width + value.length - stripAnsi(value).length);
|
|
24
|
+
}
|
|
25
|
+
function renderDocuments(documents, color) {
|
|
26
|
+
if (documents.length === 0) return "No documents.";
|
|
27
|
+
const columns = [
|
|
28
|
+
{ header: "DOCUMENT ID", value: (row) => row.documentId },
|
|
29
|
+
{ header: "ENTITY", value: (row) => row.entityKey },
|
|
30
|
+
{ header: "LABEL", value: (row) => row.label },
|
|
31
|
+
{ header: "VERSION", value: (row) => row.versionLabel }
|
|
32
|
+
];
|
|
33
|
+
const rows = documents.map((item) => record(item) ?? {});
|
|
34
|
+
const widths = columns.map((column) => Math.max(
|
|
35
|
+
column.header.length,
|
|
36
|
+
...rows.map((row) => stripAnsi(oneLine(column.value(row))).length)
|
|
37
|
+
));
|
|
38
|
+
const header = columns.map((column, index) => padVisible(
|
|
39
|
+
styled(column.header, "1", color),
|
|
40
|
+
widths[index]
|
|
41
|
+
)).join(" ").trimEnd();
|
|
42
|
+
const body = rows.map((row) => columns.map((column, index) => padVisible(
|
|
43
|
+
oneLine(column.value(row)),
|
|
44
|
+
widths[index]
|
|
45
|
+
)).join(" ").trimEnd());
|
|
46
|
+
return [header, ...body].join("\n");
|
|
47
|
+
}
|
|
48
|
+
function renderDoctor(result) {
|
|
49
|
+
const checked = Number(result.checked ?? 0);
|
|
50
|
+
if (result.ok === true) return `Doctor: healthy (${checked} objects checked)`;
|
|
51
|
+
const missing = Array.isArray(result.missing) ? result.missing.length : 0;
|
|
52
|
+
const corrupt = Array.isArray(result.corrupt) ? result.corrupt.length : 0;
|
|
53
|
+
const details = [`${missing} missing`, `${corrupt} corrupt`];
|
|
54
|
+
if (typeof result.chunks === "number" && typeof result.indexedChunks === "number" && result.chunks !== result.indexedChunks) {
|
|
55
|
+
details.push(`index ${result.indexedChunks}/${result.chunks}`);
|
|
56
|
+
}
|
|
57
|
+
if (typeof result.unreferencedObjects === "number" && result.unreferencedObjects > 0) {
|
|
58
|
+
details.push(`${result.unreferencedObjects} orphan objects`);
|
|
59
|
+
}
|
|
60
|
+
if (typeof result.leftoverTemporaries === "number" && result.leftoverTemporaries > 0) {
|
|
61
|
+
details.push(`${result.leftoverTemporaries} leftover temporaries`);
|
|
62
|
+
}
|
|
63
|
+
return `Doctor: issues found (${checked} checked; ${details.join(", ")})`;
|
|
64
|
+
}
|
|
65
|
+
function renderImport(result) {
|
|
66
|
+
const summary = record(result.summary) ?? {};
|
|
67
|
+
const count = (key) => typeof summary[key] === "number" ? summary[key] : 0;
|
|
68
|
+
return `Import: ${count("inserted")} inserted, ${count("unchanged")} unchanged, ${count("skipped")} skipped, ${count("failed")} failed (${count("total")} total)`;
|
|
69
|
+
}
|
|
70
|
+
function renderDocumentAction(result) {
|
|
71
|
+
const status = oneLine(result.status);
|
|
72
|
+
const documentId = oneLine(result.documentId);
|
|
73
|
+
if (status === "relabeled") return `Relabeled document ${documentId}: ${oneLine(result.label)}`;
|
|
74
|
+
if (status === "retired") return `Retired document ${documentId}`;
|
|
75
|
+
const action = status === "inserted" ? "Added" : status === "unchanged" ? "Unchanged" : "Updated";
|
|
76
|
+
const label = typeof result.label === "string" ? ` ${oneLine(result.label)}` : " document";
|
|
77
|
+
return `${action}${label} (${status}; documentId: ${documentId})`;
|
|
78
|
+
}
|
|
79
|
+
function renderError(error) {
|
|
80
|
+
const code = oneLine(error.code || "error");
|
|
81
|
+
return `Error [${code}]: ${oneLine(error.message || "Unknown error")}`;
|
|
82
|
+
}
|
|
83
|
+
function renderOutput(output, env) {
|
|
84
|
+
const envelope = record(output);
|
|
85
|
+
if (!envelope) return JSON.stringify(output) ?? "null";
|
|
86
|
+
const error = record(envelope.error);
|
|
87
|
+
if (error) return renderError(error);
|
|
88
|
+
const result = record(envelope.result);
|
|
89
|
+
if (!result) return JSON.stringify(output) ?? "null";
|
|
90
|
+
const color = useColor(env);
|
|
91
|
+
if (Array.isArray(result.documents)) return renderDocuments(result.documents, color);
|
|
92
|
+
if (typeof result.documents === "number" && typeof result.versions === "number" && typeof result.objects === "number") {
|
|
93
|
+
return `Documents: ${result.documents} | Versions: ${result.versions} | Objects: ${result.objects}`;
|
|
94
|
+
}
|
|
95
|
+
if (typeof result.ok === "boolean" && typeof result.checked === "number") return renderDoctor(result);
|
|
96
|
+
if (typeof result.updated === "number" && typeof result.unchanged === "number" && typeof result.skipped === "number") {
|
|
97
|
+
const label = result.applied === true ? "Rehash applied" : "Rehash preview";
|
|
98
|
+
const summary = `${label}: ${result.updated} updated, ${result.unchanged} unchanged, ${result.skipped} skipped`;
|
|
99
|
+
return typeof result.backupReminder === "string" ? `${summary}
|
|
100
|
+
${oneLine(result.backupReminder)}` : summary;
|
|
101
|
+
}
|
|
102
|
+
if (Array.isArray(result.files) && record(result.summary)) return renderImport(result);
|
|
103
|
+
if (typeof result.status === "string" && typeof result.documentId === "string") return renderDocumentAction(result);
|
|
104
|
+
return JSON.stringify(output) ?? "null";
|
|
105
|
+
}
|
|
106
|
+
export {
|
|
107
|
+
padVisible,
|
|
108
|
+
renderOutput,
|
|
109
|
+
resolveOutputFormat,
|
|
110
|
+
stripAnsi,
|
|
111
|
+
useColor
|
|
112
|
+
};
|
package/dist/src/cli/index.mjs
CHANGED
|
@@ -10,7 +10,9 @@ import { importDirectory } from "../ingest/import.mjs";
|
|
|
10
10
|
import { clientDbPath, openDatabase, validateClientName } from "../store/db.mjs";
|
|
11
11
|
import { backupRoot, reindexRoot, restoreRoot, writeRootManifest } from "../store/backup.mjs";
|
|
12
12
|
import { readObject } from "../store/objects.mjs";
|
|
13
|
+
import { rehashNamespace } from "../store/rehash.mjs";
|
|
13
14
|
import { getActiveLimits } from "../limits.mjs";
|
|
15
|
+
import { renderOutput, resolveOutputFormat } from "./format.mjs";
|
|
14
16
|
import {
|
|
15
17
|
linkReviewVersions,
|
|
16
18
|
listReviewCandidates,
|
|
@@ -36,6 +38,7 @@ Commands:
|
|
|
36
38
|
backup --to <dir> Back up the manifest and object store
|
|
37
39
|
restore --from <dir> Restore a backup into this root
|
|
38
40
|
reindex Rebuild the database from manifest and objects
|
|
41
|
+
rehash [--apply] Recompute stored-version content hashes (preview by default)
|
|
39
42
|
|
|
40
43
|
Options:
|
|
41
44
|
--root <path> Knowledge-base root
|
|
@@ -45,7 +48,20 @@ Options:
|
|
|
45
48
|
--from <dir> Backup source
|
|
46
49
|
--type <type> Human relation type for rag review link
|
|
47
50
|
--label <text> Replacement label for rag relabel
|
|
48
|
-
--
|
|
51
|
+
--apply Persist the rag rehash results
|
|
52
|
+
--help Show this help
|
|
53
|
+
|
|
54
|
+
Content identity:
|
|
55
|
+
JSON, JSONL, and YAML use canonical tree hashes (content_kind=structured, hash_version=2).
|
|
56
|
+
Other formats keep normalized-text hashes (content_kind=NULL, hash_version=1).`;
|
|
57
|
+
const HELP = `${USAGE}
|
|
58
|
+
|
|
59
|
+
Output format:
|
|
60
|
+
Interactive terminals render human-readable output; pipes use JSON.
|
|
61
|
+
RAG_FORMAT=json|pretty overrides auto-detection; NO_COLOR disables ANSI styling.
|
|
62
|
+
--help defaults to raw usage on a terminal; RAG_FORMAT=json restores its JSON envelope.
|
|
63
|
+
|
|
64
|
+
Use the documentId shown by rag ls with rag relabel <document-id> --label <text>.`;
|
|
49
65
|
const NAMED_ERROR_CODES = /* @__PURE__ */ new Set([
|
|
50
66
|
"usage",
|
|
51
67
|
"ingest_failed",
|
|
@@ -80,6 +96,7 @@ const NAMED_ERROR_CODES = /* @__PURE__ */ new Set([
|
|
|
80
96
|
"backup_failed",
|
|
81
97
|
"destination_exists",
|
|
82
98
|
"namespace_mismatch",
|
|
99
|
+
"rehash_failed",
|
|
83
100
|
"restore_destination_not_empty",
|
|
84
101
|
"restore_failed",
|
|
85
102
|
"foreign_client_in_manifest",
|
|
@@ -150,12 +167,17 @@ function parseArgs(argv) {
|
|
|
150
167
|
const positional = [];
|
|
151
168
|
const values = {};
|
|
152
169
|
let help = false;
|
|
170
|
+
let apply = false;
|
|
153
171
|
for (let index = 0; index < argv.length; index += 1) {
|
|
154
172
|
const argument = argv[index];
|
|
155
173
|
if (argument === "--help") {
|
|
156
174
|
help = true;
|
|
157
175
|
continue;
|
|
158
176
|
}
|
|
177
|
+
if (argument === "--apply") {
|
|
178
|
+
apply = true;
|
|
179
|
+
continue;
|
|
180
|
+
}
|
|
159
181
|
const assignment = /^--(to|from|type|label)=(.*)$/.exec(argument);
|
|
160
182
|
if (assignment) {
|
|
161
183
|
values[assignment[1]] = assignment[2];
|
|
@@ -182,7 +204,7 @@ function parseArgs(argv) {
|
|
|
182
204
|
if (argument.startsWith("--")) throw usageError(`Unknown option: ${argument}`);
|
|
183
205
|
positional.push(argument);
|
|
184
206
|
}
|
|
185
|
-
return { command: positional[0] ?? null, positional: positional.slice(1), help, values };
|
|
207
|
+
return { command: positional[0] ?? null, positional: positional.slice(1), help, values, apply };
|
|
186
208
|
}
|
|
187
209
|
function requireNamespace(config, mode) {
|
|
188
210
|
const { namespace } = resolveCommandNamespace(config, mode);
|
|
@@ -223,11 +245,16 @@ function queryDocuments(root, namespace) {
|
|
|
223
245
|
try {
|
|
224
246
|
const project = db.prepare(`SELECT p.project_id FROM project AS p
|
|
225
247
|
JOIN client AS c ON c.client_id = p.client_id WHERE c.name = ? AND p.slug = ?`).get(namespace.client, namespace.project);
|
|
226
|
-
const rows = project ? db.prepare(`SELECT e.entity_key AS entityKey, d.label AS label, dv.version_label AS versionLabel
|
|
248
|
+
const rows = project ? db.prepare(`SELECT d.document_id AS documentId, e.entity_key AS entityKey, d.label AS label, dv.version_label AS versionLabel
|
|
227
249
|
FROM document AS d JOIN entity AS e ON e.entity_id = d.entity_id
|
|
228
250
|
JOIN doc_version AS dv ON dv.document_id = d.document_id AND dv.is_current = 1
|
|
229
251
|
WHERE d.project_id = ? AND d.deleted_at IS NULL ORDER BY e.entity_key, d.label`).all(project.project_id) : [];
|
|
230
|
-
const documents = rows.map((row) => ({
|
|
252
|
+
const documents = rows.map((row) => ({
|
|
253
|
+
documentId: row.documentId,
|
|
254
|
+
entityKey: row.entityKey,
|
|
255
|
+
label: row.label,
|
|
256
|
+
versionLabel: row.versionLabel
|
|
257
|
+
}));
|
|
231
258
|
return { namespace, result: { documents } };
|
|
232
259
|
} finally {
|
|
233
260
|
db.close();
|
|
@@ -385,6 +412,9 @@ async function runCli(options) {
|
|
|
385
412
|
if (parsed.values.label !== void 0 && parsed.command !== "relabel") {
|
|
386
413
|
return errorOutput(config.namespace, usageError('Option --label is only accepted by "rag relabel".'));
|
|
387
414
|
}
|
|
415
|
+
if (parsed.apply && parsed.command !== "rehash") {
|
|
416
|
+
return errorOutput(config.namespace, usageError('Option --apply is only accepted by "rag rehash".'));
|
|
417
|
+
}
|
|
388
418
|
if (parsed.command === "relabel" && parsed.values.label === void 0) {
|
|
389
419
|
return errorOutput(config.namespace, usageError("Option --label requires a non-blank value.", "invalid_label"));
|
|
390
420
|
}
|
|
@@ -643,13 +673,26 @@ async function runCli(options) {
|
|
|
643
673
|
return commandErrorOutput(config.namespace, error);
|
|
644
674
|
}
|
|
645
675
|
}
|
|
676
|
+
if (parsed.command === "rehash") {
|
|
677
|
+
if (parsed.positional.length !== 0) return commandErrorOutput(config.namespace, usageError("rehash does not accept positional arguments."));
|
|
678
|
+
try {
|
|
679
|
+
const namespace = requireNamespace(config, parsed.apply ? "write" : "read");
|
|
680
|
+
if (!namespace) throw new NamespaceRequiredError();
|
|
681
|
+
const result = await rehashNamespace(path.resolve(options.cwd, config.root), namespace, parsed.apply);
|
|
682
|
+
return { exitCode: 0, output: { namespace, result } };
|
|
683
|
+
} catch (error) {
|
|
684
|
+
return commandErrorOutput(config.namespace, error);
|
|
685
|
+
}
|
|
686
|
+
}
|
|
646
687
|
return errorOutput(config.namespace, usageError(parsed.command === null ? "Command is required." : `Unknown command: ${parsed.command}`));
|
|
647
688
|
}
|
|
648
689
|
async function main(dependencies = {}) {
|
|
649
690
|
const writeStdout = dependencies.writeStdout ?? ((line) => process.stdout.write(line));
|
|
691
|
+
const writeStderr = dependencies.writeStderr ?? ((line) => process.stderr.write(line));
|
|
650
692
|
const setExitCode = dependencies.setExitCode ?? ((exitCode) => {
|
|
651
693
|
process.exitCode = exitCode;
|
|
652
694
|
});
|
|
695
|
+
const format = resolveOutputFormat(process.env, process.stdout.isTTY === true);
|
|
653
696
|
try {
|
|
654
697
|
const result = await (dependencies.runCli ?? runCli)({
|
|
655
698
|
argv: process.argv.slice(2),
|
|
@@ -657,12 +700,25 @@ async function main(dependencies = {}) {
|
|
|
657
700
|
cwd: process.cwd(),
|
|
658
701
|
homeDir: process.env.RAG_HOME ?? homedir()
|
|
659
702
|
});
|
|
660
|
-
|
|
703
|
+
const output = result.output;
|
|
704
|
+
if (format === "pretty" && result.exitCode === 0 && typeof output === "object" && output !== null && "result" in output && output.result === USAGE) {
|
|
705
|
+
writeStdout(`${HELP}
|
|
706
|
+
`);
|
|
707
|
+
} else if (format === "pretty") {
|
|
708
|
+
const rendered = `${renderOutput(output, process.env)}
|
|
709
|
+
`;
|
|
710
|
+
if (typeof output === "object" && output !== null && "error" in output) writeStderr(rendered);
|
|
711
|
+
else writeStdout(rendered);
|
|
712
|
+
} else {
|
|
713
|
+
writeStdout(`${JSON.stringify(output)}
|
|
661
714
|
`);
|
|
715
|
+
}
|
|
662
716
|
setExitCode(result.exitCode);
|
|
663
717
|
} catch (error) {
|
|
664
718
|
const message = publicErrorMessage(error);
|
|
665
|
-
|
|
719
|
+
if (format === "pretty") writeStderr(`${renderOutput({ error: { code: "internal_error", message } }, process.env)}
|
|
720
|
+
`);
|
|
721
|
+
else writeStdout(`${JSON.stringify({ namespace: null, error: { code: "internal_error", message } })}
|
|
666
722
|
`);
|
|
667
723
|
setExitCode(1);
|
|
668
724
|
}
|
|
@@ -671,6 +727,7 @@ if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.me
|
|
|
671
727
|
await main();
|
|
672
728
|
}
|
|
673
729
|
export {
|
|
730
|
+
HELP,
|
|
674
731
|
USAGE,
|
|
675
732
|
main,
|
|
676
733
|
runCli
|
|
@@ -6,7 +6,8 @@ import { ResourceLimitError } from "../ports/extractor.mjs";
|
|
|
6
6
|
import { clientDbPath, openDatabase } from "../store/db.mjs";
|
|
7
7
|
import { writeRootManifest } from "../store/backup.mjs";
|
|
8
8
|
import { deriveTier0Relations, normalizeEntityKey } from "../store/identity.mjs";
|
|
9
|
-
import {
|
|
9
|
+
import { computeContentHashFacts } from "../store/rehash.mjs";
|
|
10
|
+
import { fingerprintDistance, NEAR_DUPLICATE_MAX_DISTANCE } from "../store/similarity.mjs";
|
|
10
11
|
import { putObjectFromPath, readObject } from "../store/objects.mjs";
|
|
11
12
|
import { MAX_SOURCE_FILE_BYTES as SHARED_MAX_SOURCE_FILE_BYTES } from "../limits.mjs";
|
|
12
13
|
import { parseDocument, ParseHostError } from "./parse-host.mjs";
|
|
@@ -86,9 +87,8 @@ async function ingestFile(input) {
|
|
|
86
87
|
recordFailedRun(db, projectId, input.sourcePath, sha256, runStartedAt, error.message, error.code);
|
|
87
88
|
throw error;
|
|
88
89
|
}
|
|
89
|
-
const
|
|
90
|
-
const contentHash =
|
|
91
|
-
const normalizedHash = fingerprintNormalizedText(normalizedText);
|
|
90
|
+
const hashFacts = computeContentHashFacts(extracted);
|
|
91
|
+
const { contentHash, normalizedHash, contentKind, hashVersion } = hashFacts;
|
|
92
92
|
const sourceLabel = path.basename(input.sourcePath);
|
|
93
93
|
const entityKey = normalizeEntityKey(input.entityKey ?? path.parse(sourceLabel).name);
|
|
94
94
|
const label = input.label ?? sourceLabel;
|
|
@@ -125,8 +125,8 @@ async function ingestFile(input) {
|
|
|
125
125
|
db.prepare(`INSERT INTO ingest_run (run_id, project_id, started_at, status, source_uri, sha256)
|
|
126
126
|
VALUES (?, ?, ?, 'running', ?, ?)`).run(runId, projectId, now, input.sourcePath, stored.sha256);
|
|
127
127
|
db.prepare("UPDATE doc_version SET is_current = 0, superseded_at = ? WHERE document_id = ? AND is_current = 1").run(now, documentId);
|
|
128
|
-
db.prepare(`INSERT INTO doc_version (doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, extractor, extractor_version,
|
|
129
|
-
ingest_run_id, is_current, ingested_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 1, ?)`).run(docVersionId, documentId, stored.sha256, extracted.versionLabel, contentHash, normalizedHash, extractorName, extractorVersion, runId, now);
|
|
128
|
+
db.prepare(`INSERT INTO doc_version (doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, content_kind, hash_version, extractor, extractor_version,
|
|
129
|
+
ingest_run_id, is_current, ingested_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1, ?)`).run(docVersionId, documentId, stored.sha256, extracted.versionLabel, contentHash, normalizedHash, contentKind, hashVersion, extractorName, extractorVersion, runId, now);
|
|
130
130
|
let offset = 0;
|
|
131
131
|
extracted.sections.forEach((section, ordinal) => {
|
|
132
132
|
const sectionId = randomUUID();
|
|
@@ -146,7 +146,8 @@ async function ingestFile(input) {
|
|
|
146
146
|
contentHash,
|
|
147
147
|
entityKey,
|
|
148
148
|
projectId,
|
|
149
|
-
versionLabel: extracted.versionLabel
|
|
149
|
+
versionLabel: extracted.versionLabel,
|
|
150
|
+
hashVersion
|
|
150
151
|
};
|
|
151
152
|
const insertRelation = db.prepare(`INSERT INTO doc_relation
|
|
152
153
|
(relation_id, from_doc_version_id, to_doc_version_id, relation_type, score, decided_by, decided_at, evidence)
|
|
@@ -154,7 +155,7 @@ async function ingestFile(input) {
|
|
|
154
155
|
const currentDocument = db.prepare("SELECT deleted_at FROM document WHERE document_id = ?").get(documentId);
|
|
155
156
|
if (currentDocument?.deleted_at === null) {
|
|
156
157
|
const priorVersions = db.prepare(`SELECT dv.doc_version_id AS docVersionId, dv.sha256,
|
|
157
|
-
dv.content_hash AS contentHash, dv.normalized_hash AS normalizedHash,
|
|
158
|
+
dv.content_hash AS contentHash, dv.normalized_hash AS normalizedHash, dv.hash_version AS hashVersion,
|
|
158
159
|
e.entity_key AS entityKey, d.project_id AS projectId, dv.version_label AS versionLabel
|
|
159
160
|
FROM doc_version AS dv
|
|
160
161
|
JOIN document AS d ON d.document_id = dv.document_id
|
|
@@ -173,7 +174,7 @@ async function ingestFile(input) {
|
|
|
173
174
|
JSON.stringify(relation.evidence)
|
|
174
175
|
);
|
|
175
176
|
}
|
|
176
|
-
if (prior.normalizedHash !== null) {
|
|
177
|
+
if (prior.hashVersion === hashVersion && prior.normalizedHash !== null) {
|
|
177
178
|
const distance = fingerprintDistance(normalizedHash, prior.normalizedHash);
|
|
178
179
|
if (distance <= NEAR_DUPLICATE_MAX_DISTANCE) {
|
|
179
180
|
const relationId = `rel_${createHash("sha256").update(JSON.stringify([docVersionId, prior.docVersionId, "near_duplicate"])).digest("hex")}`;
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
const TEXT_HASH_VERSION = 1;
|
|
2
|
+
const STRUCTURED_HASH_VERSION = 2;
|
|
3
|
+
class NonFiniteNumberCanonicalizationError extends Error {
|
|
4
|
+
code = "non_finite_number";
|
|
5
|
+
constructor() {
|
|
6
|
+
super("Canonical structured content contains a non-finite number.");
|
|
7
|
+
this.name = "NonFiniteNumberCanonicalizationError";
|
|
8
|
+
}
|
|
9
|
+
}
|
|
10
|
+
function compareCodePoints(left, right) {
|
|
11
|
+
const leftPoints = Array.from(left, (character) => character.codePointAt(0));
|
|
12
|
+
const rightPoints = Array.from(right, (character) => character.codePointAt(0));
|
|
13
|
+
const sharedLength = Math.min(leftPoints.length, rightPoints.length);
|
|
14
|
+
for (let index = 0; index < sharedLength; index += 1) {
|
|
15
|
+
const leftPoint = leftPoints[index];
|
|
16
|
+
const rightPoint = rightPoints[index];
|
|
17
|
+
if (leftPoint !== rightPoint) return leftPoint - rightPoint;
|
|
18
|
+
}
|
|
19
|
+
return leftPoints.length - rightPoints.length;
|
|
20
|
+
}
|
|
21
|
+
function encode(value, ancestors) {
|
|
22
|
+
if (value === null) return "null";
|
|
23
|
+
if (typeof value === "string") return JSON.stringify(value);
|
|
24
|
+
if (typeof value === "boolean") return value ? "true" : "false";
|
|
25
|
+
if (typeof value === "number") {
|
|
26
|
+
if (!Number.isFinite(value)) throw new NonFiniteNumberCanonicalizationError();
|
|
27
|
+
return JSON.stringify(value);
|
|
28
|
+
}
|
|
29
|
+
if (typeof value === "bigint") {
|
|
30
|
+
if (value <= BigInt(Number.MAX_SAFE_INTEGER) && value >= BigInt(Number.MIN_SAFE_INTEGER)) {
|
|
31
|
+
return JSON.stringify(Number(value));
|
|
32
|
+
}
|
|
33
|
+
return JSON.stringify(value.toString());
|
|
34
|
+
}
|
|
35
|
+
if (Array.isArray(value)) {
|
|
36
|
+
if (ancestors.has(value)) throw new TypeError("Structured content must be acyclic.");
|
|
37
|
+
ancestors.add(value);
|
|
38
|
+
try {
|
|
39
|
+
return `[${value.map((item) => item === void 0 ? "null" : encode(item, ancestors)).join(",")}]`;
|
|
40
|
+
} finally {
|
|
41
|
+
ancestors.delete(value);
|
|
42
|
+
}
|
|
43
|
+
}
|
|
44
|
+
if (typeof value === "object") {
|
|
45
|
+
const object = value;
|
|
46
|
+
if (ancestors.has(object)) throw new TypeError("Structured content must be acyclic.");
|
|
47
|
+
ancestors.add(object);
|
|
48
|
+
try {
|
|
49
|
+
const members = Object.keys(object).filter((key) => object[key] !== void 0).sort(compareCodePoints).map((key) => `${JSON.stringify(key)}:${encode(object[key], ancestors)}`);
|
|
50
|
+
return `{${members.join(",")}}`;
|
|
51
|
+
} finally {
|
|
52
|
+
ancestors.delete(object);
|
|
53
|
+
}
|
|
54
|
+
}
|
|
55
|
+
return "null";
|
|
56
|
+
}
|
|
57
|
+
function canonicalizeStructuredContent(value) {
|
|
58
|
+
return encode(value, /* @__PURE__ */ new Set());
|
|
59
|
+
}
|
|
60
|
+
export {
|
|
61
|
+
NonFiniteNumberCanonicalizationError,
|
|
62
|
+
STRUCTURED_HASH_VERSION,
|
|
63
|
+
TEXT_HASH_VERSION,
|
|
64
|
+
canonicalizeStructuredContent
|
|
65
|
+
};
|
|
@@ -57,7 +57,7 @@ function deriveTier0Relations(a, b) {
|
|
|
57
57
|
const relations = [];
|
|
58
58
|
if (a.sha256 === b.sha256) {
|
|
59
59
|
relations.push(makeRelation(a.docVersionId, b.docVersionId, "same_bytes", { sha256: a.sha256 }));
|
|
60
|
-
} else if (a.contentHash !== null && b.contentHash !== null && a.contentHash === b.contentHash) {
|
|
60
|
+
} else if (a.contentHash !== null && b.contentHash !== null && a.contentHash === b.contentHash && (a.hashVersion ?? 1) === (b.hashVersion ?? 1)) {
|
|
61
61
|
relations.push(makeRelation(a.docVersionId, b.docVersionId, "same_content", { contentHash: a.contentHash }));
|
|
62
62
|
}
|
|
63
63
|
const sameEntity = normalizeEntityKey(a.entityKey) === normalizeEntityKey(b.entityKey);
|
|
@@ -64,6 +64,7 @@ function listReviewCandidates(db, projectId) {
|
|
|
64
64
|
JOIN entity AS right_entity ON right_entity.entity_id = right_document.entity_id
|
|
65
65
|
WHERE left_document.project_id = ? AND right_document.project_id = left_document.project_id
|
|
66
66
|
AND left_version.content_hash IS NOT NULL AND left_version.content_hash = right_version.content_hash
|
|
67
|
+
AND left_version.hash_version = right_version.hash_version
|
|
67
68
|
AND left_document.deleted_at IS NULL AND right_document.deleted_at IS NULL
|
|
68
69
|
AND left_entity.deleted_at IS NULL AND right_entity.deleted_at IS NULL
|
|
69
70
|
AND NOT EXISTS (
|
|
@@ -86,6 +87,7 @@ function listReviewCandidates(db, projectId) {
|
|
|
86
87
|
JOIN entity AS to_entity ON to_entity.entity_id = to_document.entity_id
|
|
87
88
|
WHERE near_relation.relation_type = 'near_duplicate' AND near_relation.decided_by <> 'human'
|
|
88
89
|
AND from_document.project_id = ? AND to_document.project_id = from_document.project_id
|
|
90
|
+
AND from_version.hash_version = to_version.hash_version
|
|
89
91
|
AND from_document.deleted_at IS NULL AND to_document.deleted_at IS NULL
|
|
90
92
|
AND from_entity.deleted_at IS NULL AND to_entity.deleted_at IS NULL
|
|
91
93
|
AND NOT EXISTS (
|
|
@@ -21,7 +21,7 @@ function buildManifest(db) {
|
|
|
21
21
|
ingestRuns: orderedRows(db, "ingest_run"),
|
|
22
22
|
entities: db.prepare("SELECT entity_id, project_id, entity_key, kind, title, created_at, updated_at, deleted_at FROM entity ORDER BY entity_id").all(),
|
|
23
23
|
documents: db.prepare("SELECT document_id, entity_id, project_id, label, created_at, updated_at, deleted_at FROM document ORDER BY document_id").all(),
|
|
24
|
-
versions: db.prepare("SELECT doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, extractor, extractor_version, ingest_run_id, is_current, ingested_at, superseded_at FROM doc_version ORDER BY doc_version_id").all(),
|
|
24
|
+
versions: db.prepare("SELECT doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, content_kind, hash_version, extractor, extractor_version, ingest_run_id, is_current, ingested_at, superseded_at FROM doc_version ORDER BY doc_version_id").all(),
|
|
25
25
|
sections: db.prepare("SELECT section_id, doc_version_id, ordinal, heading, level, char_start, char_end FROM section ORDER BY doc_version_id, ordinal, section_id").all(),
|
|
26
26
|
chunks: db.prepare("SELECT chunk_id, doc_version_id, section_id, ordinal, text, char_start, char_end, token_count FROM chunk ORDER BY doc_version_id, ordinal, chunk_id").all(),
|
|
27
27
|
relations: db.prepare("SELECT relation_id, from_doc_version_id, to_doc_version_id, relation_type, score, decided_by, decided_at, evidence FROM doc_relation ORDER BY relation_id").all(),
|
|
@@ -38,7 +38,7 @@ const ROW_COLUMNS = {
|
|
|
38
38
|
ingestRuns: ["run_id", "project_id", "started_at", "finished_at", "status", "source_uri", "sha256", "result_code", "detail"],
|
|
39
39
|
entities: ["entity_id", "project_id", "entity_key", "kind", "title", "created_at", "updated_at", "deleted_at"],
|
|
40
40
|
documents: ["document_id", "entity_id", "project_id", "label", "created_at", "updated_at", "deleted_at"],
|
|
41
|
-
versions: ["doc_version_id", "document_id", "sha256", "version_label", "content_hash", "normalized_hash", "extractor", "extractor_version", "ingest_run_id", "is_current", "ingested_at", "superseded_at"],
|
|
41
|
+
versions: ["doc_version_id", "document_id", "sha256", "version_label", "content_hash", "normalized_hash", "content_kind", "hash_version", "extractor", "extractor_version", "ingest_run_id", "is_current", "ingested_at", "superseded_at"],
|
|
42
42
|
sections: ["section_id", "doc_version_id", "ordinal", "heading", "level", "char_start", "char_end"],
|
|
43
43
|
chunks: ["chunk_id", "doc_version_id", "section_id", "ordinal", "text", "char_start", "char_end", "token_count"],
|
|
44
44
|
relations: ["relation_id", "from_doc_version_id", "to_doc_version_id", "relation_type", "score", "decided_by", "decided_at", "evidence"],
|
|
@@ -47,8 +47,12 @@ const ROW_COLUMNS = {
|
|
|
47
47
|
};
|
|
48
48
|
function validateRowKeys(value, field) {
|
|
49
49
|
const expected = ROW_COLUMNS[field];
|
|
50
|
+
const legacyVersionColumns = field === "versions" ? expected.filter((column) => column !== "content_kind" && column !== "hash_version") : null;
|
|
50
51
|
for (const row of value) {
|
|
51
|
-
if (!isRecord(row)
|
|
52
|
+
if (!isRecord(row)) throw new ManifestError("manifest_invalid", `Manifest ${field} row keys must match expected columns: ${expected.join(", ")}.`);
|
|
53
|
+
const keys = Object.keys(row);
|
|
54
|
+
const matches = (columns) => keys.length === columns.length && columns.every((column) => Object.hasOwn(row, column));
|
|
55
|
+
if (!matches(expected) && !(legacyVersionColumns !== null && matches(legacyVersionColumns))) {
|
|
52
56
|
throw new ManifestError("manifest_invalid", `Manifest ${field} row keys must match expected columns: ${expected.join(", ")}.`);
|
|
53
57
|
}
|
|
54
58
|
}
|
|
@@ -99,7 +103,12 @@ function importManifest(db, value) {
|
|
|
99
103
|
insertRows(db, "INSERT INTO entity (entity_id, project_id, entity_key, kind, title, created_at, updated_at, deleted_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", value.entities, (r) => [r.entity_id, r.project_id, r.entity_key, r.kind, r.title, r.created_at, r.updated_at, r.deleted_at]);
|
|
100
104
|
insertRows(db, "INSERT INTO document (document_id, entity_id, project_id, label, created_at, updated_at, deleted_at) VALUES (?, ?, ?, ?, ?, ?, ?)", value.documents, (r) => [r.document_id, r.entity_id, r.project_id, r.label, r.created_at, r.updated_at, r.deleted_at]);
|
|
101
105
|
insertRows(db, "INSERT INTO object (sha256, byte_size, rel_path, media_type, created_at, last_verified_at) VALUES (?, ?, ?, ?, ?, ?)", value.objects, (r) => [r.sha256, r.byte_size, r.rel_path, r.media_type, r.created_at, r.last_verified_at]);
|
|
102
|
-
|
|
106
|
+
const versions = value.versions.map((version) => ({
|
|
107
|
+
...version,
|
|
108
|
+
content_kind: version.content_kind ?? null,
|
|
109
|
+
hash_version: version.hash_version ?? 1
|
|
110
|
+
}));
|
|
111
|
+
insertRows(db, "INSERT INTO doc_version (doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, content_kind, hash_version, extractor, extractor_version, ingest_run_id, is_current, ingested_at, superseded_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", versions, (r) => [r.doc_version_id, r.document_id, r.sha256, r.version_label, r.content_hash, r.normalized_hash, r.content_kind, r.hash_version, r.extractor, r.extractor_version, r.ingest_run_id, r.is_current, r.ingested_at, r.superseded_at]);
|
|
103
112
|
insertRows(db, "INSERT INTO section (section_id, doc_version_id, ordinal, heading, level, char_start, char_end) VALUES (?, ?, ?, ?, ?, ?, ?)", value.sections, (r) => [r.section_id, r.doc_version_id, r.ordinal, r.heading, r.level, r.char_start, r.char_end]);
|
|
104
113
|
insertRows(db, "INSERT INTO chunk (chunk_id, doc_version_id, section_id, ordinal, text, char_start, char_end, token_count) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", value.chunks, (r) => [r.chunk_id, r.doc_version_id, r.section_id, r.ordinal, r.text, r.char_start, r.char_end, r.token_count]);
|
|
105
114
|
db.exec("INSERT INTO chunk_fts(chunk_fts) VALUES ('rebuild');");
|
|
@@ -0,0 +1,162 @@
|
|
|
1
|
+
import { createHash } from "node:crypto";
|
|
2
|
+
import * as fs from "node:fs/promises";
|
|
3
|
+
import * as path from "node:path";
|
|
4
|
+
import { NonFiniteNumberCanonicalizationError, STRUCTURED_HASH_VERSION, TEXT_HASH_VERSION } from "./canonical.mjs";
|
|
5
|
+
import { clientDbPath, openDatabase } from "./db.mjs";
|
|
6
|
+
import { writeRootManifest } from "./backup.mjs";
|
|
7
|
+
import { readObject } from "./objects.mjs";
|
|
8
|
+
import { fingerprintNormalizedText } from "./similarity.mjs";
|
|
9
|
+
import { parseDocument } from "../ingest/parse-host.mjs";
|
|
10
|
+
function computeContentHashFacts(extracted) {
|
|
11
|
+
const normalizedText = extracted.sections.map((section) => section.text).join("\n").replace(/\s+/g, " ").trim().toLowerCase();
|
|
12
|
+
let contentKind;
|
|
13
|
+
let hashVersion;
|
|
14
|
+
let hashInput;
|
|
15
|
+
let skipped = false;
|
|
16
|
+
try {
|
|
17
|
+
if (extracted.canonicalizationError === "non_finite_number") throw new NonFiniteNumberCanonicalizationError();
|
|
18
|
+
contentKind = extracted.canonicalContent === void 0 ? null : "structured";
|
|
19
|
+
hashVersion = extracted.canonicalContent === void 0 ? TEXT_HASH_VERSION : STRUCTURED_HASH_VERSION;
|
|
20
|
+
hashInput = extracted.canonicalContent ?? normalizedText;
|
|
21
|
+
} catch (error) {
|
|
22
|
+
if (!(error instanceof NonFiniteNumberCanonicalizationError)) throw error;
|
|
23
|
+
contentKind = null;
|
|
24
|
+
hashVersion = TEXT_HASH_VERSION;
|
|
25
|
+
hashInput = normalizedText;
|
|
26
|
+
skipped = true;
|
|
27
|
+
}
|
|
28
|
+
return {
|
|
29
|
+
contentHash: createHash("sha256").update(hashInput).digest("hex"),
|
|
30
|
+
normalizedHash: fingerprintNormalizedText(hashInput),
|
|
31
|
+
contentKind,
|
|
32
|
+
hashVersion,
|
|
33
|
+
skipped
|
|
34
|
+
};
|
|
35
|
+
}
|
|
36
|
+
function errorWithCode(code, message) {
|
|
37
|
+
return Object.assign(new Error(message), { code });
|
|
38
|
+
}
|
|
39
|
+
function mediaHint(extractor, label) {
|
|
40
|
+
const labelExtension = path.extname(label).toLowerCase();
|
|
41
|
+
switch (extractor) {
|
|
42
|
+
case "json":
|
|
43
|
+
return ".json";
|
|
44
|
+
case "jsonl":
|
|
45
|
+
return ".jsonl";
|
|
46
|
+
case "yaml":
|
|
47
|
+
return labelExtension === ".yml" ? ".yml" : ".yaml";
|
|
48
|
+
case "plain-text":
|
|
49
|
+
return labelExtension === ".md" ? ".md" : ".txt";
|
|
50
|
+
case "pdf":
|
|
51
|
+
return ".pdf";
|
|
52
|
+
case "docx":
|
|
53
|
+
return ".docx";
|
|
54
|
+
case "xlsx":
|
|
55
|
+
return ".xlsx";
|
|
56
|
+
case "pptx":
|
|
57
|
+
return ".pptx";
|
|
58
|
+
case "image-ocr":
|
|
59
|
+
return [".png", ".jpg", ".jpeg"].includes(labelExtension) ? labelExtension : ".png";
|
|
60
|
+
default:
|
|
61
|
+
throw errorWithCode("rehash_failed", `Cannot re-extract version with unknown extractor "${extractor}".`);
|
|
62
|
+
}
|
|
63
|
+
}
|
|
64
|
+
function sameFacts(row, facts) {
|
|
65
|
+
return row.content_hash === facts.contentHash && row.normalized_hash === facts.normalizedHash && row.content_kind === facts.contentKind && row.hash_version === facts.hashVersion;
|
|
66
|
+
}
|
|
67
|
+
async function rehashNamespace(root, namespace, apply) {
|
|
68
|
+
const dbPath = clientDbPath(root, namespace.client);
|
|
69
|
+
try {
|
|
70
|
+
await fs.access(dbPath);
|
|
71
|
+
} catch (error) {
|
|
72
|
+
if (error.code === "ENOENT") {
|
|
73
|
+
throw errorWithCode("database_not_found", `Client database not found: ${dbPath}`);
|
|
74
|
+
}
|
|
75
|
+
throw errorWithCode("io_error", `Unable to inspect client database: ${dbPath}`);
|
|
76
|
+
}
|
|
77
|
+
const db = openDatabase(dbPath);
|
|
78
|
+
try {
|
|
79
|
+
const project = db.prepare(`SELECT project.project_id FROM project
|
|
80
|
+
JOIN client ON client.client_id = project.client_id WHERE client.name = ? AND project.slug = ?`).get(namespace.client, namespace.project);
|
|
81
|
+
if (!project) return { applied: apply, updated: 0, unchanged: 0, skipped: 0, ...!apply ? { backupReminder: reminder() } : {} };
|
|
82
|
+
const rows = db.prepare(`SELECT version.doc_version_id, version.sha256, version.content_hash, version.normalized_hash,
|
|
83
|
+
version.content_kind, version.hash_version, version.extractor, document.label
|
|
84
|
+
FROM doc_version AS version
|
|
85
|
+
JOIN document ON document.document_id = version.document_id
|
|
86
|
+
WHERE document.project_id = ? ORDER BY version.doc_version_id`).all(project.project_id);
|
|
87
|
+
const updates = [];
|
|
88
|
+
let unchanged = 0;
|
|
89
|
+
let skipped = 0;
|
|
90
|
+
for (const row of rows) {
|
|
91
|
+
let bytes;
|
|
92
|
+
try {
|
|
93
|
+
bytes = await readObject(root, row.sha256);
|
|
94
|
+
} catch (error) {
|
|
95
|
+
const code = error.code === "ENOENT" ? "object_missing" : "object_read_failed";
|
|
96
|
+
throw errorWithCode(code, `Cannot read object ${row.sha256} for version ${row.doc_version_id}.`);
|
|
97
|
+
}
|
|
98
|
+
const objectHash = createHash("sha256").update(bytes).digest("hex");
|
|
99
|
+
if (objectHash !== row.sha256) {
|
|
100
|
+
throw errorWithCode("object_corrupt", `Object ${row.sha256} failed SHA-256 verification for version ${row.doc_version_id}.`);
|
|
101
|
+
}
|
|
102
|
+
let extracted;
|
|
103
|
+
try {
|
|
104
|
+
extracted = await parseDocument(
|
|
105
|
+
path.join(root, "objects", row.sha256.slice(0, 2), row.sha256.slice(2, 4), row.sha256),
|
|
106
|
+
namespace,
|
|
107
|
+
void 0,
|
|
108
|
+
{ mediaHint: mediaHint(row.extractor, row.label) }
|
|
109
|
+
);
|
|
110
|
+
} catch (error) {
|
|
111
|
+
if (error.code === "rehash_failed") throw error;
|
|
112
|
+
throw errorWithCode("rehash_failed", `Cannot re-extract version ${row.doc_version_id}: ${error instanceof Error ? error.message : String(error)}`);
|
|
113
|
+
}
|
|
114
|
+
const facts = computeContentHashFacts(extracted);
|
|
115
|
+
if (facts.skipped) {
|
|
116
|
+
skipped += 1;
|
|
117
|
+
} else if (sameFacts(row, facts)) {
|
|
118
|
+
unchanged += 1;
|
|
119
|
+
} else {
|
|
120
|
+
updates.push({
|
|
121
|
+
docVersionId: row.doc_version_id,
|
|
122
|
+
contentHash: facts.contentHash,
|
|
123
|
+
normalizedHash: facts.normalizedHash,
|
|
124
|
+
contentKind: facts.contentKind,
|
|
125
|
+
hashVersion: facts.hashVersion
|
|
126
|
+
});
|
|
127
|
+
}
|
|
128
|
+
}
|
|
129
|
+
if (apply && updates.length > 0) {
|
|
130
|
+
db.exec("BEGIN IMMEDIATE;");
|
|
131
|
+
try {
|
|
132
|
+
const update = db.prepare(`UPDATE doc_version SET content_hash = ?, normalized_hash = ?, content_kind = ?, hash_version = ?
|
|
133
|
+
WHERE doc_version_id = ?`);
|
|
134
|
+
for (const row of updates) update.run(row.contentHash, row.normalizedHash, row.contentKind, row.hashVersion, row.docVersionId);
|
|
135
|
+
await writeRootManifest(root, db);
|
|
136
|
+
db.exec("COMMIT;");
|
|
137
|
+
} catch (error) {
|
|
138
|
+
try {
|
|
139
|
+
db.exec("ROLLBACK;");
|
|
140
|
+
} catch {
|
|
141
|
+
}
|
|
142
|
+
throw errorWithCode("rehash_failed", `Backfill transaction failed: ${error instanceof Error ? error.message : String(error)}`);
|
|
143
|
+
}
|
|
144
|
+
}
|
|
145
|
+
return {
|
|
146
|
+
applied: apply,
|
|
147
|
+
updated: updates.length,
|
|
148
|
+
unchanged,
|
|
149
|
+
skipped,
|
|
150
|
+
...!apply ? { backupReminder: reminder() } : {}
|
|
151
|
+
};
|
|
152
|
+
} finally {
|
|
153
|
+
db.close();
|
|
154
|
+
}
|
|
155
|
+
}
|
|
156
|
+
function reminder() {
|
|
157
|
+
return "Before applying, back up this namespace with: rag backup --to <backup-dir>.";
|
|
158
|
+
}
|
|
159
|
+
export {
|
|
160
|
+
computeContentHashFacts,
|
|
161
|
+
rehashNamespace
|
|
162
|
+
};
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
const SCHEMA_VERSION =
|
|
1
|
+
const SCHEMA_VERSION = 2;
|
|
2
2
|
const MIGRATIONS = [
|
|
3
3
|
{
|
|
4
4
|
version: 1,
|
|
@@ -170,6 +170,16 @@ BEFORE DELETE ON doc_version
|
|
|
170
170
|
BEGIN
|
|
171
171
|
SELECT RAISE(ABORT, 'doc_version history is immutable');
|
|
172
172
|
END;`
|
|
173
|
+
},
|
|
174
|
+
{
|
|
175
|
+
version: 2,
|
|
176
|
+
sql: `CREATE TABLE schema_migration (
|
|
177
|
+
version INTEGER PRIMARY KEY,
|
|
178
|
+
applied_at TEXT NOT NULL
|
|
179
|
+
);
|
|
180
|
+
|
|
181
|
+
ALTER TABLE doc_version ADD COLUMN content_kind TEXT;
|
|
182
|
+
ALTER TABLE doc_version ADD COLUMN hash_version INTEGER NOT NULL DEFAULT 1;`
|
|
173
183
|
}
|
|
174
184
|
];
|
|
175
185
|
export {
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "agent-rag-mcp",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.4.0",
|
|
4
4
|
"description": "Deterministic, namespace-isolated document retrieval with a CLI and MCP server.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"repository": {
|
|
@@ -28,12 +28,16 @@
|
|
|
28
28
|
"bin",
|
|
29
29
|
"dist",
|
|
30
30
|
"extensions",
|
|
31
|
+
"skills",
|
|
31
32
|
"README.md",
|
|
32
33
|
"LICENSE"
|
|
33
34
|
],
|
|
34
35
|
"pi": {
|
|
35
36
|
"extensions": [
|
|
36
37
|
"./extensions/agent-rag.ts"
|
|
38
|
+
],
|
|
39
|
+
"skills": [
|
|
40
|
+
"./skills"
|
|
37
41
|
]
|
|
38
42
|
},
|
|
39
43
|
"engines": {
|
|
@@ -58,6 +62,7 @@
|
|
|
58
62
|
"mammoth": "1.13.0",
|
|
59
63
|
"pdfjs-dist": "6.4.299",
|
|
60
64
|
"tesseract.js": "7.0.0",
|
|
65
|
+
"yaml": "^2.9.1",
|
|
61
66
|
"zod": "4.6.7"
|
|
62
67
|
}
|
|
63
68
|
}
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: agent-rag
|
|
3
|
+
description: "Trigger: search or read the indexed knowledge base with explicit client/project scope and verifiable citations."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
## Choose a tool
|
|
7
|
+
|
|
8
|
+
- Use `rag_find` to search a client/project namespace and get a bounded catalog of matching citations.
|
|
9
|
+
- Use `rag_read` only when a citation needs more surrounding text; pass its `docVersionId` and `chunkOrdinal` from `rag_find`.
|
|
10
|
+
- The orchestrator only receives `rag_find`; delegate exploration when a citation needs reading.
|
|
11
|
+
|
|
12
|
+
## Evidence and scope
|
|
13
|
+
|
|
14
|
+
- The `namespace` is always explicit: pass an object with non-empty `client` and `project`; never infer it from cwd or discover another namespace.
|
|
15
|
+
- Cite source wording with a verbatim quote. Citations must be quoted, never paraphrased as source text.
|
|
16
|
+
- Keep the citation's document/version and section context with the quote; say when a search returns no matches.
|
|
17
|
+
- Treat `rag_find` excerpts as bounded evidence, not the complete source. Use `rag_read` for a bounded passage only when needed.
|
|
18
|
+
|
|
19
|
+
## Config and CLI
|
|
20
|
+
|
|
21
|
+
- CLI namespace/root options are `--client`, `--project`, `--root`; environment alternatives are `RAG_CLIENT`, `RAG_PROJECT`, `RAG_ROOT`.
|
|
22
|
+
- The CLI home is `RAG_HOME ?? os.homedir()`. The MCP server and Pi extension pass cwd as `homeDir`.
|
|
23
|
+
- The only config keys read in `src/config/discovery.mts` are `client`, `project`, `root`, and `failClosed`; unknown keys are silently ignored.
|
|
24
|
+
- Config candidates, first existing only: `<cwd>/rag.config.json`, `<dirname(cwd)>/rag.config.json` (one level up only), then `<homeDir>/.rag/config.json`.
|
|
25
|
+
- Only JSON syntax errors (`config_invalid`) or config-file I/O errors (`config_unreadable`) raise those config errors.
|
|
26
|
+
- Each field resolves independently to its first available value: client `--client` > `RAG_CLIENT` > file > null; project follows the same chain.
|
|
27
|
+
- Root precedence is `--root` > `RAG_ROOT` > file > `<homeDir>/.rag`; fields can resolve from different sources.
|
|
28
|
+
- Discovery reports its source as `args`, `env`, `cwd`, `parent`, `home`, or `unscoped`.
|
|
29
|
+
- `client` must match `/^[A-Za-z0-9][A-Za-z0-9._-]*$/`; `.` and `..` are rejected with `invalid_client_name`. Its DB is `<root>/clients/<client>/rag.db`.
|
|
30
|
+
- `project` is logical only: its slug is stored in `project.slug` rows in that client DB; it creates no directory and has no name validation.
|
|
31
|
+
- `root` defaults to `<homeDir>/.rag`; the CLI resolves a relative root against cwd for each command.
|
|
32
|
+
- Root layout: `clients/<client>/rag.db`, `objects/<aa>/<bb>/<sha256>`, and `manifest.json`.
|
|
33
|
+
- `failClosed` defaults to true and is file-only (no flag or environment variable). For writes without a namespace it throws `namespace_required`; false uses reserved `default/default`. Reads are unaffected.
|
|
34
|
+
- `rag init` does not write `rag.config.json`; it creates only the client DB, object store, and root manifest.
|
|
35
|
+
- MCP tools require `namespace: {client, project}` on every call. The server resolves only `root` from discovery; config, `RAG_CLIENT`, and `RAG_PROJECT` never provide tool identity.
|