agent-rag-mcp 0.3.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +7 -1
- package/dist/src/adapters/structured-text.mjs +111 -45
- package/dist/src/cli/format.mjs +6 -0
- package/dist/src/cli/index.mjs +29 -2
- package/dist/src/ingest/pipeline.mjs +10 -9
- package/dist/src/store/canonical.mjs +65 -0
- package/dist/src/store/identity.mjs +1 -1
- package/dist/src/store/lifecycle.mjs +2 -0
- package/dist/src/store/manifest.mjs +13 -4
- package/dist/src/store/rehash.mjs +162 -0
- package/dist/src/store/schema.mjs +11 -1
- package/package.json +6 -1
- package/skills/agent-rag/SKILL.md +35 -0
package/README.md
CHANGED
|
@@ -32,6 +32,8 @@ The `rag` commands available today are:
|
|
|
32
32
|
- `rag backup --to <dir>` — Backs up the manifest and object store.
|
|
33
33
|
- `rag restore --from <dir>` — Restores a backup under this root.
|
|
34
34
|
- `rag reindex` — Rebuilds the database from the manifest and objects.
|
|
35
|
+
- `rag rehash` — Previews recomputed hashes for stored versions; `rag rehash --apply` writes the changed hash facts in one transaction. Preview first, and back up with `rag backup --to <backup-dir>` before applying.
|
|
36
|
+
- `rag rehash --apply` writes `manifest.json` before committing the database transaction. An interruption in that window can leave the manifest ahead of the database; rerunning `rag rehash --apply` or running `rag reindex` converges to the requested facts.
|
|
35
37
|
|
|
36
38
|
### CLI output
|
|
37
39
|
|
|
@@ -44,7 +46,11 @@ rag ls
|
|
|
44
46
|
rag relabel <document-id> --label "replacement label"
|
|
45
47
|
```
|
|
46
48
|
|
|
47
|
-
Structured files can use `.yaml`, `.yml`, `.json`, or `.jsonl`; each structured file yields
|
|
49
|
+
Structured files can use `.yaml`, `.yml`, `.json`, or `.jsonl`; each structured file yields sections from its parsed data tree. YAML uses the YAML 1.2 core schema, so words such as `yes` and `on` remain strings and unquoted timestamps remain strings.
|
|
50
|
+
|
|
51
|
+
### Content identity
|
|
52
|
+
|
|
53
|
+
Structured JSON, JSONL, and YAML are hashed from compact canonical JSON for their parsed data tree, without a format prefix. Their `doc_version` rows use `content_kind='structured'` and `hash_version=2`; equivalent trees therefore share a content hash across formats and key order. Other formats retain the existing normalized extracted-text hash and use `content_kind=NULL` with `hash_version=1`. Existing version rows stay at version 1 with a null content kind until repaired with `rag rehash`; the repair never rewrites stored object bytes and skips structured documents that cannot be canonically hashed.
|
|
48
54
|
|
|
49
55
|
The default knowledge-base root is `%USERPROFILE%\.rag\`. Each client has its own `rag.db` under `.rag\clients\<client>\`; projects are logical within that client and are identified by `project_id`. The namespace is the pair `(client, project)`. By default, `failClosed` is `true`; without a namespace, reads are allowed, and when `failClosed` is set to `false`, writes go to the reserved `default/default` namespace. The Tier 3 human review queue is available through `rag review list`.
|
|
50
56
|
|
|
@@ -1,3 +1,5 @@
|
|
|
1
|
+
import { parseDocument } from "yaml";
|
|
2
|
+
import { canonicalizeStructuredContent, NonFiniteNumberCanonicalizationError } from "../store/canonical.mjs";
|
|
1
3
|
class StructuredTextExtractionError extends Error {
|
|
2
4
|
code;
|
|
3
5
|
constructor(code) {
|
|
@@ -29,70 +31,127 @@ function supportsReference(ref, options) {
|
|
|
29
31
|
}
|
|
30
32
|
function renderJsonValue(value) {
|
|
31
33
|
if (typeof value === "string") return value;
|
|
32
|
-
return JSON.stringify(value,
|
|
34
|
+
return JSON.stringify(value, (_key, child) => typeof child === "bigint" ? child.toString() : child, 2) ?? "null";
|
|
35
|
+
}
|
|
36
|
+
function parseJsonContent(content) {
|
|
37
|
+
return JSON.parse(content, (_key, value, context) => {
|
|
38
|
+
const source = context?.source;
|
|
39
|
+
if (typeof value === "number" && !Number.isSafeInteger(value) && source !== void 0 && /^-?(?:0|[1-9][0-9]*)$/.test(source)) {
|
|
40
|
+
return BigInt(source);
|
|
41
|
+
}
|
|
42
|
+
return value;
|
|
43
|
+
});
|
|
44
|
+
}
|
|
45
|
+
function canonicalFields(value) {
|
|
46
|
+
try {
|
|
47
|
+
return { canonicalContent: canonicalizeStructuredContent(value) };
|
|
48
|
+
} catch (error) {
|
|
49
|
+
if (error instanceof NonFiniteNumberCanonicalizationError) return { canonicalizationError: error.code };
|
|
50
|
+
throw error;
|
|
51
|
+
}
|
|
33
52
|
}
|
|
34
53
|
function extractJson(content) {
|
|
35
54
|
let parsed;
|
|
36
55
|
try {
|
|
37
|
-
parsed =
|
|
56
|
+
parsed = parseJsonContent(content);
|
|
38
57
|
} catch {
|
|
39
58
|
throw new StructuredTextExtractionError("extraction_invalid_json");
|
|
40
59
|
}
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
}
|
|
49
|
-
return [{ ordinal: 0, heading: null, level: null, text: renderJsonValue(parsed) }];
|
|
60
|
+
const sections = typeof parsed === "object" && parsed !== null && !Array.isArray(parsed) ? Object.entries(parsed).map(([heading, value], ordinal) => ({
|
|
61
|
+
ordinal,
|
|
62
|
+
heading,
|
|
63
|
+
level: 1,
|
|
64
|
+
text: renderJsonValue(value)
|
|
65
|
+
})) : [{ ordinal: 0, heading: null, level: null, text: renderJsonValue(parsed) }];
|
|
66
|
+
return { sections, ...canonicalFields(parsed) };
|
|
50
67
|
}
|
|
51
68
|
function extractJsonl(content) {
|
|
52
69
|
const sections = [];
|
|
70
|
+
const records = [];
|
|
53
71
|
for (const line of content.split(/\r?\n/)) {
|
|
54
72
|
const text = line.trim();
|
|
55
73
|
if (text.length === 0) continue;
|
|
56
74
|
try {
|
|
57
|
-
|
|
75
|
+
records.push(parseJsonContent(text));
|
|
58
76
|
} catch {
|
|
59
77
|
throw new StructuredTextExtractionError("extraction_invalid_json");
|
|
60
78
|
}
|
|
61
79
|
sections.push({ ordinal: sections.length, heading: null, level: null, text });
|
|
62
80
|
}
|
|
63
|
-
return sections;
|
|
81
|
+
return { sections, ...canonicalFields(records) };
|
|
64
82
|
}
|
|
65
|
-
function
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
83
|
+
function normalizeYamlValue(value, ancestors = /* @__PURE__ */ new Set()) {
|
|
84
|
+
if (typeof value !== "object" || value === null) return value;
|
|
85
|
+
if (ancestors.has(value)) throw new StructuredTextExtractionError("extraction_invalid_yaml");
|
|
86
|
+
ancestors.add(value);
|
|
87
|
+
try {
|
|
88
|
+
if (value instanceof Map) {
|
|
89
|
+
const result2 = /* @__PURE__ */ Object.create(null);
|
|
90
|
+
for (const [key, child] of value) {
|
|
91
|
+
const stringKey = typeof key === "string" ? key : canonicalizeStructuredContent(normalizeYamlValue(key, ancestors));
|
|
92
|
+
result2[stringKey] = normalizeYamlValue(child, ancestors);
|
|
93
|
+
}
|
|
94
|
+
return result2;
|
|
95
|
+
}
|
|
96
|
+
if (Array.isArray(value)) return value.map((child) => normalizeYamlValue(child, ancestors));
|
|
97
|
+
const result = /* @__PURE__ */ Object.create(null);
|
|
98
|
+
for (const [key, child] of Object.entries(value)) result[key] = normalizeYamlValue(child, ancestors);
|
|
99
|
+
return result;
|
|
100
|
+
} finally {
|
|
101
|
+
ancestors.delete(value);
|
|
75
102
|
}
|
|
76
|
-
|
|
77
|
-
|
|
103
|
+
}
|
|
104
|
+
function renderYamlValue(value) {
|
|
105
|
+
if (typeof value === "string") return value;
|
|
106
|
+
return JSON.stringify(value, (_key, child) => typeof child === "bigint" ? child.toString() : child, 2) ?? "null";
|
|
107
|
+
}
|
|
108
|
+
function yamlTopLevelStarts(document) {
|
|
109
|
+
const root = document.contents;
|
|
110
|
+
const starts = /* @__PURE__ */ new Map();
|
|
111
|
+
for (const pair of root?.items ?? []) {
|
|
112
|
+
const position = pair.key?.range?.[0];
|
|
113
|
+
if (position === void 0) continue;
|
|
114
|
+
const rawKey = pair.key && Object.hasOwn(pair.key, "value") ? pair.key.value : pair.key?.toJSON?.();
|
|
115
|
+
const key = normalizeYamlValue(rawKey);
|
|
116
|
+
const stringKey = typeof key === "string" ? key : canonicalizeStructuredContent(key);
|
|
117
|
+
starts.set(stringKey, position);
|
|
78
118
|
}
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
119
|
+
return starts;
|
|
120
|
+
}
|
|
121
|
+
function extractYaml(content) {
|
|
122
|
+
let parsed;
|
|
123
|
+
let topLevelStarts;
|
|
124
|
+
let canonicalizationError;
|
|
125
|
+
try {
|
|
126
|
+
const document = parseDocument(content, { schema: "core", uniqueKeys: false, intAsBigInt: true });
|
|
127
|
+
if (document.errors.length > 0) throw new Error("invalid YAML document");
|
|
128
|
+
parsed = normalizeYamlValue(document.toJS({ mapAsMap: true }));
|
|
129
|
+
topLevelStarts = yamlTopLevelStarts(document);
|
|
130
|
+
} catch (error) {
|
|
131
|
+
if (!(error instanceof NonFiniteNumberCanonicalizationError)) {
|
|
132
|
+
throw new StructuredTextExtractionError("extraction_invalid_yaml");
|
|
87
133
|
}
|
|
88
|
-
|
|
89
|
-
|
|
134
|
+
canonicalizationError = error.code;
|
|
135
|
+
parsed = null;
|
|
136
|
+
topLevelStarts = /* @__PURE__ */ new Map();
|
|
90
137
|
}
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
138
|
+
if (canonicalizationError !== void 0) {
|
|
139
|
+
return { sections: [{ ordinal: 0, heading: null, level: null, text: content.trim() }], canonicalizationError };
|
|
140
|
+
}
|
|
141
|
+
let sections;
|
|
142
|
+
if (typeof parsed === "object" && parsed !== null && !Array.isArray(parsed)) {
|
|
143
|
+
const entries = Object.entries(parsed).map(([heading, value]) => ({ heading, value, start: topLevelStarts.get(heading) })).filter((entry) => entry.start !== void 0).sort((left, right) => left.start - right.start);
|
|
144
|
+
sections = entries.map(({ heading, start }, ordinal) => ({
|
|
145
|
+
ordinal,
|
|
146
|
+
heading,
|
|
147
|
+
level: 1,
|
|
148
|
+
text: content.slice(start, entries[ordinal + 1]?.start ?? content.length).trim()
|
|
149
|
+
}));
|
|
150
|
+
if (sections.length === 0) sections = [{ ordinal: 0, heading: null, level: null, text: renderYamlValue(parsed) }];
|
|
151
|
+
} else {
|
|
152
|
+
sections = [{ ordinal: 0, heading: null, level: null, text: renderYamlValue(parsed) }];
|
|
153
|
+
}
|
|
154
|
+
return { sections, ...canonicalFields(parsed) };
|
|
96
155
|
}
|
|
97
156
|
function createStructuredTextExtractor(options) {
|
|
98
157
|
return {
|
|
@@ -108,19 +167,26 @@ function createStructuredTextExtractor(options) {
|
|
|
108
167
|
if (content.length === 0) {
|
|
109
168
|
return { sections: [], versionLabel: null, mediaType: null };
|
|
110
169
|
}
|
|
111
|
-
let
|
|
170
|
+
let extracted;
|
|
112
171
|
switch (options.kind) {
|
|
113
172
|
case "json":
|
|
114
|
-
|
|
173
|
+
extracted = extractJson(content);
|
|
115
174
|
break;
|
|
116
175
|
case "jsonl":
|
|
117
|
-
|
|
176
|
+
extracted = extractJsonl(content);
|
|
118
177
|
break;
|
|
119
178
|
case "yaml":
|
|
120
|
-
|
|
179
|
+
extracted = extractYaml(content);
|
|
121
180
|
break;
|
|
122
181
|
}
|
|
123
|
-
|
|
182
|
+
const result = {
|
|
183
|
+
sections: extracted.sections,
|
|
184
|
+
versionLabel: null,
|
|
185
|
+
mediaType: options.mediaTypes[0] ?? null,
|
|
186
|
+
...extracted.canonicalContent === void 0 ? {} : { canonicalContent: extracted.canonicalContent },
|
|
187
|
+
...extracted.canonicalizationError === void 0 ? {} : { canonicalizationError: extracted.canonicalizationError }
|
|
188
|
+
};
|
|
189
|
+
return result;
|
|
124
190
|
}
|
|
125
191
|
};
|
|
126
192
|
}
|
package/dist/src/cli/format.mjs
CHANGED
|
@@ -93,6 +93,12 @@ function renderOutput(output, env) {
|
|
|
93
93
|
return `Documents: ${result.documents} | Versions: ${result.versions} | Objects: ${result.objects}`;
|
|
94
94
|
}
|
|
95
95
|
if (typeof result.ok === "boolean" && typeof result.checked === "number") return renderDoctor(result);
|
|
96
|
+
if (typeof result.updated === "number" && typeof result.unchanged === "number" && typeof result.skipped === "number") {
|
|
97
|
+
const label = result.applied === true ? "Rehash applied" : "Rehash preview";
|
|
98
|
+
const summary = `${label}: ${result.updated} updated, ${result.unchanged} unchanged, ${result.skipped} skipped`;
|
|
99
|
+
return typeof result.backupReminder === "string" ? `${summary}
|
|
100
|
+
${oneLine(result.backupReminder)}` : summary;
|
|
101
|
+
}
|
|
96
102
|
if (Array.isArray(result.files) && record(result.summary)) return renderImport(result);
|
|
97
103
|
if (typeof result.status === "string" && typeof result.documentId === "string") return renderDocumentAction(result);
|
|
98
104
|
return JSON.stringify(output) ?? "null";
|
package/dist/src/cli/index.mjs
CHANGED
|
@@ -10,6 +10,7 @@ import { importDirectory } from "../ingest/import.mjs";
|
|
|
10
10
|
import { clientDbPath, openDatabase, validateClientName } from "../store/db.mjs";
|
|
11
11
|
import { backupRoot, reindexRoot, restoreRoot, writeRootManifest } from "../store/backup.mjs";
|
|
12
12
|
import { readObject } from "../store/objects.mjs";
|
|
13
|
+
import { rehashNamespace } from "../store/rehash.mjs";
|
|
13
14
|
import { getActiveLimits } from "../limits.mjs";
|
|
14
15
|
import { renderOutput, resolveOutputFormat } from "./format.mjs";
|
|
15
16
|
import {
|
|
@@ -37,6 +38,7 @@ Commands:
|
|
|
37
38
|
backup --to <dir> Back up the manifest and object store
|
|
38
39
|
restore --from <dir> Restore a backup into this root
|
|
39
40
|
reindex Rebuild the database from manifest and objects
|
|
41
|
+
rehash [--apply] Recompute stored-version content hashes (preview by default)
|
|
40
42
|
|
|
41
43
|
Options:
|
|
42
44
|
--root <path> Knowledge-base root
|
|
@@ -46,7 +48,12 @@ Options:
|
|
|
46
48
|
--from <dir> Backup source
|
|
47
49
|
--type <type> Human relation type for rag review link
|
|
48
50
|
--label <text> Replacement label for rag relabel
|
|
49
|
-
--
|
|
51
|
+
--apply Persist the rag rehash results
|
|
52
|
+
--help Show this help
|
|
53
|
+
|
|
54
|
+
Content identity:
|
|
55
|
+
JSON, JSONL, and YAML use canonical tree hashes (content_kind=structured, hash_version=2).
|
|
56
|
+
Other formats keep normalized-text hashes (content_kind=NULL, hash_version=1).`;
|
|
50
57
|
const HELP = `${USAGE}
|
|
51
58
|
|
|
52
59
|
Output format:
|
|
@@ -89,6 +96,7 @@ const NAMED_ERROR_CODES = /* @__PURE__ */ new Set([
|
|
|
89
96
|
"backup_failed",
|
|
90
97
|
"destination_exists",
|
|
91
98
|
"namespace_mismatch",
|
|
99
|
+
"rehash_failed",
|
|
92
100
|
"restore_destination_not_empty",
|
|
93
101
|
"restore_failed",
|
|
94
102
|
"foreign_client_in_manifest",
|
|
@@ -159,12 +167,17 @@ function parseArgs(argv) {
|
|
|
159
167
|
const positional = [];
|
|
160
168
|
const values = {};
|
|
161
169
|
let help = false;
|
|
170
|
+
let apply = false;
|
|
162
171
|
for (let index = 0; index < argv.length; index += 1) {
|
|
163
172
|
const argument = argv[index];
|
|
164
173
|
if (argument === "--help") {
|
|
165
174
|
help = true;
|
|
166
175
|
continue;
|
|
167
176
|
}
|
|
177
|
+
if (argument === "--apply") {
|
|
178
|
+
apply = true;
|
|
179
|
+
continue;
|
|
180
|
+
}
|
|
168
181
|
const assignment = /^--(to|from|type|label)=(.*)$/.exec(argument);
|
|
169
182
|
if (assignment) {
|
|
170
183
|
values[assignment[1]] = assignment[2];
|
|
@@ -191,7 +204,7 @@ function parseArgs(argv) {
|
|
|
191
204
|
if (argument.startsWith("--")) throw usageError(`Unknown option: ${argument}`);
|
|
192
205
|
positional.push(argument);
|
|
193
206
|
}
|
|
194
|
-
return { command: positional[0] ?? null, positional: positional.slice(1), help, values };
|
|
207
|
+
return { command: positional[0] ?? null, positional: positional.slice(1), help, values, apply };
|
|
195
208
|
}
|
|
196
209
|
function requireNamespace(config, mode) {
|
|
197
210
|
const { namespace } = resolveCommandNamespace(config, mode);
|
|
@@ -399,6 +412,9 @@ async function runCli(options) {
|
|
|
399
412
|
if (parsed.values.label !== void 0 && parsed.command !== "relabel") {
|
|
400
413
|
return errorOutput(config.namespace, usageError('Option --label is only accepted by "rag relabel".'));
|
|
401
414
|
}
|
|
415
|
+
if (parsed.apply && parsed.command !== "rehash") {
|
|
416
|
+
return errorOutput(config.namespace, usageError('Option --apply is only accepted by "rag rehash".'));
|
|
417
|
+
}
|
|
402
418
|
if (parsed.command === "relabel" && parsed.values.label === void 0) {
|
|
403
419
|
return errorOutput(config.namespace, usageError("Option --label requires a non-blank value.", "invalid_label"));
|
|
404
420
|
}
|
|
@@ -657,6 +673,17 @@ async function runCli(options) {
|
|
|
657
673
|
return commandErrorOutput(config.namespace, error);
|
|
658
674
|
}
|
|
659
675
|
}
|
|
676
|
+
if (parsed.command === "rehash") {
|
|
677
|
+
if (parsed.positional.length !== 0) return commandErrorOutput(config.namespace, usageError("rehash does not accept positional arguments."));
|
|
678
|
+
try {
|
|
679
|
+
const namespace = requireNamespace(config, parsed.apply ? "write" : "read");
|
|
680
|
+
if (!namespace) throw new NamespaceRequiredError();
|
|
681
|
+
const result = await rehashNamespace(path.resolve(options.cwd, config.root), namespace, parsed.apply);
|
|
682
|
+
return { exitCode: 0, output: { namespace, result } };
|
|
683
|
+
} catch (error) {
|
|
684
|
+
return commandErrorOutput(config.namespace, error);
|
|
685
|
+
}
|
|
686
|
+
}
|
|
660
687
|
return errorOutput(config.namespace, usageError(parsed.command === null ? "Command is required." : `Unknown command: ${parsed.command}`));
|
|
661
688
|
}
|
|
662
689
|
async function main(dependencies = {}) {
|
|
@@ -6,7 +6,8 @@ import { ResourceLimitError } from "../ports/extractor.mjs";
|
|
|
6
6
|
import { clientDbPath, openDatabase } from "../store/db.mjs";
|
|
7
7
|
import { writeRootManifest } from "../store/backup.mjs";
|
|
8
8
|
import { deriveTier0Relations, normalizeEntityKey } from "../store/identity.mjs";
|
|
9
|
-
import {
|
|
9
|
+
import { computeContentHashFacts } from "../store/rehash.mjs";
|
|
10
|
+
import { fingerprintDistance, NEAR_DUPLICATE_MAX_DISTANCE } from "../store/similarity.mjs";
|
|
10
11
|
import { putObjectFromPath, readObject } from "../store/objects.mjs";
|
|
11
12
|
import { MAX_SOURCE_FILE_BYTES as SHARED_MAX_SOURCE_FILE_BYTES } from "../limits.mjs";
|
|
12
13
|
import { parseDocument, ParseHostError } from "./parse-host.mjs";
|
|
@@ -86,9 +87,8 @@ async function ingestFile(input) {
|
|
|
86
87
|
recordFailedRun(db, projectId, input.sourcePath, sha256, runStartedAt, error.message, error.code);
|
|
87
88
|
throw error;
|
|
88
89
|
}
|
|
89
|
-
const
|
|
90
|
-
const contentHash =
|
|
91
|
-
const normalizedHash = fingerprintNormalizedText(normalizedText);
|
|
90
|
+
const hashFacts = computeContentHashFacts(extracted);
|
|
91
|
+
const { contentHash, normalizedHash, contentKind, hashVersion } = hashFacts;
|
|
92
92
|
const sourceLabel = path.basename(input.sourcePath);
|
|
93
93
|
const entityKey = normalizeEntityKey(input.entityKey ?? path.parse(sourceLabel).name);
|
|
94
94
|
const label = input.label ?? sourceLabel;
|
|
@@ -125,8 +125,8 @@ async function ingestFile(input) {
|
|
|
125
125
|
db.prepare(`INSERT INTO ingest_run (run_id, project_id, started_at, status, source_uri, sha256)
|
|
126
126
|
VALUES (?, ?, ?, 'running', ?, ?)`).run(runId, projectId, now, input.sourcePath, stored.sha256);
|
|
127
127
|
db.prepare("UPDATE doc_version SET is_current = 0, superseded_at = ? WHERE document_id = ? AND is_current = 1").run(now, documentId);
|
|
128
|
-
db.prepare(`INSERT INTO doc_version (doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, extractor, extractor_version,
|
|
129
|
-
ingest_run_id, is_current, ingested_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 1, ?)`).run(docVersionId, documentId, stored.sha256, extracted.versionLabel, contentHash, normalizedHash, extractorName, extractorVersion, runId, now);
|
|
128
|
+
db.prepare(`INSERT INTO doc_version (doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, content_kind, hash_version, extractor, extractor_version,
|
|
129
|
+
ingest_run_id, is_current, ingested_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1, ?)`).run(docVersionId, documentId, stored.sha256, extracted.versionLabel, contentHash, normalizedHash, contentKind, hashVersion, extractorName, extractorVersion, runId, now);
|
|
130
130
|
let offset = 0;
|
|
131
131
|
extracted.sections.forEach((section, ordinal) => {
|
|
132
132
|
const sectionId = randomUUID();
|
|
@@ -146,7 +146,8 @@ async function ingestFile(input) {
|
|
|
146
146
|
contentHash,
|
|
147
147
|
entityKey,
|
|
148
148
|
projectId,
|
|
149
|
-
versionLabel: extracted.versionLabel
|
|
149
|
+
versionLabel: extracted.versionLabel,
|
|
150
|
+
hashVersion
|
|
150
151
|
};
|
|
151
152
|
const insertRelation = db.prepare(`INSERT INTO doc_relation
|
|
152
153
|
(relation_id, from_doc_version_id, to_doc_version_id, relation_type, score, decided_by, decided_at, evidence)
|
|
@@ -154,7 +155,7 @@ async function ingestFile(input) {
|
|
|
154
155
|
const currentDocument = db.prepare("SELECT deleted_at FROM document WHERE document_id = ?").get(documentId);
|
|
155
156
|
if (currentDocument?.deleted_at === null) {
|
|
156
157
|
const priorVersions = db.prepare(`SELECT dv.doc_version_id AS docVersionId, dv.sha256,
|
|
157
|
-
dv.content_hash AS contentHash, dv.normalized_hash AS normalizedHash,
|
|
158
|
+
dv.content_hash AS contentHash, dv.normalized_hash AS normalizedHash, dv.hash_version AS hashVersion,
|
|
158
159
|
e.entity_key AS entityKey, d.project_id AS projectId, dv.version_label AS versionLabel
|
|
159
160
|
FROM doc_version AS dv
|
|
160
161
|
JOIN document AS d ON d.document_id = dv.document_id
|
|
@@ -173,7 +174,7 @@ async function ingestFile(input) {
|
|
|
173
174
|
JSON.stringify(relation.evidence)
|
|
174
175
|
);
|
|
175
176
|
}
|
|
176
|
-
if (prior.normalizedHash !== null) {
|
|
177
|
+
if (prior.hashVersion === hashVersion && prior.normalizedHash !== null) {
|
|
177
178
|
const distance = fingerprintDistance(normalizedHash, prior.normalizedHash);
|
|
178
179
|
if (distance <= NEAR_DUPLICATE_MAX_DISTANCE) {
|
|
179
180
|
const relationId = `rel_${createHash("sha256").update(JSON.stringify([docVersionId, prior.docVersionId, "near_duplicate"])).digest("hex")}`;
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
const TEXT_HASH_VERSION = 1;
|
|
2
|
+
const STRUCTURED_HASH_VERSION = 2;
|
|
3
|
+
class NonFiniteNumberCanonicalizationError extends Error {
|
|
4
|
+
code = "non_finite_number";
|
|
5
|
+
constructor() {
|
|
6
|
+
super("Canonical structured content contains a non-finite number.");
|
|
7
|
+
this.name = "NonFiniteNumberCanonicalizationError";
|
|
8
|
+
}
|
|
9
|
+
}
|
|
10
|
+
function compareCodePoints(left, right) {
|
|
11
|
+
const leftPoints = Array.from(left, (character) => character.codePointAt(0));
|
|
12
|
+
const rightPoints = Array.from(right, (character) => character.codePointAt(0));
|
|
13
|
+
const sharedLength = Math.min(leftPoints.length, rightPoints.length);
|
|
14
|
+
for (let index = 0; index < sharedLength; index += 1) {
|
|
15
|
+
const leftPoint = leftPoints[index];
|
|
16
|
+
const rightPoint = rightPoints[index];
|
|
17
|
+
if (leftPoint !== rightPoint) return leftPoint - rightPoint;
|
|
18
|
+
}
|
|
19
|
+
return leftPoints.length - rightPoints.length;
|
|
20
|
+
}
|
|
21
|
+
function encode(value, ancestors) {
|
|
22
|
+
if (value === null) return "null";
|
|
23
|
+
if (typeof value === "string") return JSON.stringify(value);
|
|
24
|
+
if (typeof value === "boolean") return value ? "true" : "false";
|
|
25
|
+
if (typeof value === "number") {
|
|
26
|
+
if (!Number.isFinite(value)) throw new NonFiniteNumberCanonicalizationError();
|
|
27
|
+
return JSON.stringify(value);
|
|
28
|
+
}
|
|
29
|
+
if (typeof value === "bigint") {
|
|
30
|
+
if (value <= BigInt(Number.MAX_SAFE_INTEGER) && value >= BigInt(Number.MIN_SAFE_INTEGER)) {
|
|
31
|
+
return JSON.stringify(Number(value));
|
|
32
|
+
}
|
|
33
|
+
return JSON.stringify(value.toString());
|
|
34
|
+
}
|
|
35
|
+
if (Array.isArray(value)) {
|
|
36
|
+
if (ancestors.has(value)) throw new TypeError("Structured content must be acyclic.");
|
|
37
|
+
ancestors.add(value);
|
|
38
|
+
try {
|
|
39
|
+
return `[${value.map((item) => item === void 0 ? "null" : encode(item, ancestors)).join(",")}]`;
|
|
40
|
+
} finally {
|
|
41
|
+
ancestors.delete(value);
|
|
42
|
+
}
|
|
43
|
+
}
|
|
44
|
+
if (typeof value === "object") {
|
|
45
|
+
const object = value;
|
|
46
|
+
if (ancestors.has(object)) throw new TypeError("Structured content must be acyclic.");
|
|
47
|
+
ancestors.add(object);
|
|
48
|
+
try {
|
|
49
|
+
const members = Object.keys(object).filter((key) => object[key] !== void 0).sort(compareCodePoints).map((key) => `${JSON.stringify(key)}:${encode(object[key], ancestors)}`);
|
|
50
|
+
return `{${members.join(",")}}`;
|
|
51
|
+
} finally {
|
|
52
|
+
ancestors.delete(object);
|
|
53
|
+
}
|
|
54
|
+
}
|
|
55
|
+
return "null";
|
|
56
|
+
}
|
|
57
|
+
function canonicalizeStructuredContent(value) {
|
|
58
|
+
return encode(value, /* @__PURE__ */ new Set());
|
|
59
|
+
}
|
|
60
|
+
export {
|
|
61
|
+
NonFiniteNumberCanonicalizationError,
|
|
62
|
+
STRUCTURED_HASH_VERSION,
|
|
63
|
+
TEXT_HASH_VERSION,
|
|
64
|
+
canonicalizeStructuredContent
|
|
65
|
+
};
|
|
@@ -57,7 +57,7 @@ function deriveTier0Relations(a, b) {
|
|
|
57
57
|
const relations = [];
|
|
58
58
|
if (a.sha256 === b.sha256) {
|
|
59
59
|
relations.push(makeRelation(a.docVersionId, b.docVersionId, "same_bytes", { sha256: a.sha256 }));
|
|
60
|
-
} else if (a.contentHash !== null && b.contentHash !== null && a.contentHash === b.contentHash) {
|
|
60
|
+
} else if (a.contentHash !== null && b.contentHash !== null && a.contentHash === b.contentHash && (a.hashVersion ?? 1) === (b.hashVersion ?? 1)) {
|
|
61
61
|
relations.push(makeRelation(a.docVersionId, b.docVersionId, "same_content", { contentHash: a.contentHash }));
|
|
62
62
|
}
|
|
63
63
|
const sameEntity = normalizeEntityKey(a.entityKey) === normalizeEntityKey(b.entityKey);
|
|
@@ -64,6 +64,7 @@ function listReviewCandidates(db, projectId) {
|
|
|
64
64
|
JOIN entity AS right_entity ON right_entity.entity_id = right_document.entity_id
|
|
65
65
|
WHERE left_document.project_id = ? AND right_document.project_id = left_document.project_id
|
|
66
66
|
AND left_version.content_hash IS NOT NULL AND left_version.content_hash = right_version.content_hash
|
|
67
|
+
AND left_version.hash_version = right_version.hash_version
|
|
67
68
|
AND left_document.deleted_at IS NULL AND right_document.deleted_at IS NULL
|
|
68
69
|
AND left_entity.deleted_at IS NULL AND right_entity.deleted_at IS NULL
|
|
69
70
|
AND NOT EXISTS (
|
|
@@ -86,6 +87,7 @@ function listReviewCandidates(db, projectId) {
|
|
|
86
87
|
JOIN entity AS to_entity ON to_entity.entity_id = to_document.entity_id
|
|
87
88
|
WHERE near_relation.relation_type = 'near_duplicate' AND near_relation.decided_by <> 'human'
|
|
88
89
|
AND from_document.project_id = ? AND to_document.project_id = from_document.project_id
|
|
90
|
+
AND from_version.hash_version = to_version.hash_version
|
|
89
91
|
AND from_document.deleted_at IS NULL AND to_document.deleted_at IS NULL
|
|
90
92
|
AND from_entity.deleted_at IS NULL AND to_entity.deleted_at IS NULL
|
|
91
93
|
AND NOT EXISTS (
|
|
@@ -21,7 +21,7 @@ function buildManifest(db) {
|
|
|
21
21
|
ingestRuns: orderedRows(db, "ingest_run"),
|
|
22
22
|
entities: db.prepare("SELECT entity_id, project_id, entity_key, kind, title, created_at, updated_at, deleted_at FROM entity ORDER BY entity_id").all(),
|
|
23
23
|
documents: db.prepare("SELECT document_id, entity_id, project_id, label, created_at, updated_at, deleted_at FROM document ORDER BY document_id").all(),
|
|
24
|
-
versions: db.prepare("SELECT doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, extractor, extractor_version, ingest_run_id, is_current, ingested_at, superseded_at FROM doc_version ORDER BY doc_version_id").all(),
|
|
24
|
+
versions: db.prepare("SELECT doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, content_kind, hash_version, extractor, extractor_version, ingest_run_id, is_current, ingested_at, superseded_at FROM doc_version ORDER BY doc_version_id").all(),
|
|
25
25
|
sections: db.prepare("SELECT section_id, doc_version_id, ordinal, heading, level, char_start, char_end FROM section ORDER BY doc_version_id, ordinal, section_id").all(),
|
|
26
26
|
chunks: db.prepare("SELECT chunk_id, doc_version_id, section_id, ordinal, text, char_start, char_end, token_count FROM chunk ORDER BY doc_version_id, ordinal, chunk_id").all(),
|
|
27
27
|
relations: db.prepare("SELECT relation_id, from_doc_version_id, to_doc_version_id, relation_type, score, decided_by, decided_at, evidence FROM doc_relation ORDER BY relation_id").all(),
|
|
@@ -38,7 +38,7 @@ const ROW_COLUMNS = {
|
|
|
38
38
|
ingestRuns: ["run_id", "project_id", "started_at", "finished_at", "status", "source_uri", "sha256", "result_code", "detail"],
|
|
39
39
|
entities: ["entity_id", "project_id", "entity_key", "kind", "title", "created_at", "updated_at", "deleted_at"],
|
|
40
40
|
documents: ["document_id", "entity_id", "project_id", "label", "created_at", "updated_at", "deleted_at"],
|
|
41
|
-
versions: ["doc_version_id", "document_id", "sha256", "version_label", "content_hash", "normalized_hash", "extractor", "extractor_version", "ingest_run_id", "is_current", "ingested_at", "superseded_at"],
|
|
41
|
+
versions: ["doc_version_id", "document_id", "sha256", "version_label", "content_hash", "normalized_hash", "content_kind", "hash_version", "extractor", "extractor_version", "ingest_run_id", "is_current", "ingested_at", "superseded_at"],
|
|
42
42
|
sections: ["section_id", "doc_version_id", "ordinal", "heading", "level", "char_start", "char_end"],
|
|
43
43
|
chunks: ["chunk_id", "doc_version_id", "section_id", "ordinal", "text", "char_start", "char_end", "token_count"],
|
|
44
44
|
relations: ["relation_id", "from_doc_version_id", "to_doc_version_id", "relation_type", "score", "decided_by", "decided_at", "evidence"],
|
|
@@ -47,8 +47,12 @@ const ROW_COLUMNS = {
|
|
|
47
47
|
};
|
|
48
48
|
function validateRowKeys(value, field) {
|
|
49
49
|
const expected = ROW_COLUMNS[field];
|
|
50
|
+
const legacyVersionColumns = field === "versions" ? expected.filter((column) => column !== "content_kind" && column !== "hash_version") : null;
|
|
50
51
|
for (const row of value) {
|
|
51
|
-
if (!isRecord(row)
|
|
52
|
+
if (!isRecord(row)) throw new ManifestError("manifest_invalid", `Manifest ${field} row keys must match expected columns: ${expected.join(", ")}.`);
|
|
53
|
+
const keys = Object.keys(row);
|
|
54
|
+
const matches = (columns) => keys.length === columns.length && columns.every((column) => Object.hasOwn(row, column));
|
|
55
|
+
if (!matches(expected) && !(legacyVersionColumns !== null && matches(legacyVersionColumns))) {
|
|
52
56
|
throw new ManifestError("manifest_invalid", `Manifest ${field} row keys must match expected columns: ${expected.join(", ")}.`);
|
|
53
57
|
}
|
|
54
58
|
}
|
|
@@ -99,7 +103,12 @@ function importManifest(db, value) {
|
|
|
99
103
|
insertRows(db, "INSERT INTO entity (entity_id, project_id, entity_key, kind, title, created_at, updated_at, deleted_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", value.entities, (r) => [r.entity_id, r.project_id, r.entity_key, r.kind, r.title, r.created_at, r.updated_at, r.deleted_at]);
|
|
100
104
|
insertRows(db, "INSERT INTO document (document_id, entity_id, project_id, label, created_at, updated_at, deleted_at) VALUES (?, ?, ?, ?, ?, ?, ?)", value.documents, (r) => [r.document_id, r.entity_id, r.project_id, r.label, r.created_at, r.updated_at, r.deleted_at]);
|
|
101
105
|
insertRows(db, "INSERT INTO object (sha256, byte_size, rel_path, media_type, created_at, last_verified_at) VALUES (?, ?, ?, ?, ?, ?)", value.objects, (r) => [r.sha256, r.byte_size, r.rel_path, r.media_type, r.created_at, r.last_verified_at]);
|
|
102
|
-
|
|
106
|
+
const versions = value.versions.map((version) => ({
|
|
107
|
+
...version,
|
|
108
|
+
content_kind: version.content_kind ?? null,
|
|
109
|
+
hash_version: version.hash_version ?? 1
|
|
110
|
+
}));
|
|
111
|
+
insertRows(db, "INSERT INTO doc_version (doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, content_kind, hash_version, extractor, extractor_version, ingest_run_id, is_current, ingested_at, superseded_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", versions, (r) => [r.doc_version_id, r.document_id, r.sha256, r.version_label, r.content_hash, r.normalized_hash, r.content_kind, r.hash_version, r.extractor, r.extractor_version, r.ingest_run_id, r.is_current, r.ingested_at, r.superseded_at]);
|
|
103
112
|
insertRows(db, "INSERT INTO section (section_id, doc_version_id, ordinal, heading, level, char_start, char_end) VALUES (?, ?, ?, ?, ?, ?, ?)", value.sections, (r) => [r.section_id, r.doc_version_id, r.ordinal, r.heading, r.level, r.char_start, r.char_end]);
|
|
104
113
|
insertRows(db, "INSERT INTO chunk (chunk_id, doc_version_id, section_id, ordinal, text, char_start, char_end, token_count) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", value.chunks, (r) => [r.chunk_id, r.doc_version_id, r.section_id, r.ordinal, r.text, r.char_start, r.char_end, r.token_count]);
|
|
105
114
|
db.exec("INSERT INTO chunk_fts(chunk_fts) VALUES ('rebuild');");
|
|
@@ -0,0 +1,162 @@
|
|
|
1
|
+
import { createHash } from "node:crypto";
|
|
2
|
+
import * as fs from "node:fs/promises";
|
|
3
|
+
import * as path from "node:path";
|
|
4
|
+
import { NonFiniteNumberCanonicalizationError, STRUCTURED_HASH_VERSION, TEXT_HASH_VERSION } from "./canonical.mjs";
|
|
5
|
+
import { clientDbPath, openDatabase } from "./db.mjs";
|
|
6
|
+
import { writeRootManifest } from "./backup.mjs";
|
|
7
|
+
import { readObject } from "./objects.mjs";
|
|
8
|
+
import { fingerprintNormalizedText } from "./similarity.mjs";
|
|
9
|
+
import { parseDocument } from "../ingest/parse-host.mjs";
|
|
10
|
+
function computeContentHashFacts(extracted) {
|
|
11
|
+
const normalizedText = extracted.sections.map((section) => section.text).join("\n").replace(/\s+/g, " ").trim().toLowerCase();
|
|
12
|
+
let contentKind;
|
|
13
|
+
let hashVersion;
|
|
14
|
+
let hashInput;
|
|
15
|
+
let skipped = false;
|
|
16
|
+
try {
|
|
17
|
+
if (extracted.canonicalizationError === "non_finite_number") throw new NonFiniteNumberCanonicalizationError();
|
|
18
|
+
contentKind = extracted.canonicalContent === void 0 ? null : "structured";
|
|
19
|
+
hashVersion = extracted.canonicalContent === void 0 ? TEXT_HASH_VERSION : STRUCTURED_HASH_VERSION;
|
|
20
|
+
hashInput = extracted.canonicalContent ?? normalizedText;
|
|
21
|
+
} catch (error) {
|
|
22
|
+
if (!(error instanceof NonFiniteNumberCanonicalizationError)) throw error;
|
|
23
|
+
contentKind = null;
|
|
24
|
+
hashVersion = TEXT_HASH_VERSION;
|
|
25
|
+
hashInput = normalizedText;
|
|
26
|
+
skipped = true;
|
|
27
|
+
}
|
|
28
|
+
return {
|
|
29
|
+
contentHash: createHash("sha256").update(hashInput).digest("hex"),
|
|
30
|
+
normalizedHash: fingerprintNormalizedText(hashInput),
|
|
31
|
+
contentKind,
|
|
32
|
+
hashVersion,
|
|
33
|
+
skipped
|
|
34
|
+
};
|
|
35
|
+
}
|
|
36
|
+
function errorWithCode(code, message) {
|
|
37
|
+
return Object.assign(new Error(message), { code });
|
|
38
|
+
}
|
|
39
|
+
function mediaHint(extractor, label) {
|
|
40
|
+
const labelExtension = path.extname(label).toLowerCase();
|
|
41
|
+
switch (extractor) {
|
|
42
|
+
case "json":
|
|
43
|
+
return ".json";
|
|
44
|
+
case "jsonl":
|
|
45
|
+
return ".jsonl";
|
|
46
|
+
case "yaml":
|
|
47
|
+
return labelExtension === ".yml" ? ".yml" : ".yaml";
|
|
48
|
+
case "plain-text":
|
|
49
|
+
return labelExtension === ".md" ? ".md" : ".txt";
|
|
50
|
+
case "pdf":
|
|
51
|
+
return ".pdf";
|
|
52
|
+
case "docx":
|
|
53
|
+
return ".docx";
|
|
54
|
+
case "xlsx":
|
|
55
|
+
return ".xlsx";
|
|
56
|
+
case "pptx":
|
|
57
|
+
return ".pptx";
|
|
58
|
+
case "image-ocr":
|
|
59
|
+
return [".png", ".jpg", ".jpeg"].includes(labelExtension) ? labelExtension : ".png";
|
|
60
|
+
default:
|
|
61
|
+
throw errorWithCode("rehash_failed", `Cannot re-extract version with unknown extractor "${extractor}".`);
|
|
62
|
+
}
|
|
63
|
+
}
|
|
64
|
+
function sameFacts(row, facts) {
|
|
65
|
+
return row.content_hash === facts.contentHash && row.normalized_hash === facts.normalizedHash && row.content_kind === facts.contentKind && row.hash_version === facts.hashVersion;
|
|
66
|
+
}
|
|
67
|
+
async function rehashNamespace(root, namespace, apply) {
|
|
68
|
+
const dbPath = clientDbPath(root, namespace.client);
|
|
69
|
+
try {
|
|
70
|
+
await fs.access(dbPath);
|
|
71
|
+
} catch (error) {
|
|
72
|
+
if (error.code === "ENOENT") {
|
|
73
|
+
throw errorWithCode("database_not_found", `Client database not found: ${dbPath}`);
|
|
74
|
+
}
|
|
75
|
+
throw errorWithCode("io_error", `Unable to inspect client database: ${dbPath}`);
|
|
76
|
+
}
|
|
77
|
+
const db = openDatabase(dbPath);
|
|
78
|
+
try {
|
|
79
|
+
const project = db.prepare(`SELECT project.project_id FROM project
|
|
80
|
+
JOIN client ON client.client_id = project.client_id WHERE client.name = ? AND project.slug = ?`).get(namespace.client, namespace.project);
|
|
81
|
+
if (!project) return { applied: apply, updated: 0, unchanged: 0, skipped: 0, ...!apply ? { backupReminder: reminder() } : {} };
|
|
82
|
+
const rows = db.prepare(`SELECT version.doc_version_id, version.sha256, version.content_hash, version.normalized_hash,
|
|
83
|
+
version.content_kind, version.hash_version, version.extractor, document.label
|
|
84
|
+
FROM doc_version AS version
|
|
85
|
+
JOIN document ON document.document_id = version.document_id
|
|
86
|
+
WHERE document.project_id = ? ORDER BY version.doc_version_id`).all(project.project_id);
|
|
87
|
+
const updates = [];
|
|
88
|
+
let unchanged = 0;
|
|
89
|
+
let skipped = 0;
|
|
90
|
+
for (const row of rows) {
|
|
91
|
+
let bytes;
|
|
92
|
+
try {
|
|
93
|
+
bytes = await readObject(root, row.sha256);
|
|
94
|
+
} catch (error) {
|
|
95
|
+
const code = error.code === "ENOENT" ? "object_missing" : "object_read_failed";
|
|
96
|
+
throw errorWithCode(code, `Cannot read object ${row.sha256} for version ${row.doc_version_id}.`);
|
|
97
|
+
}
|
|
98
|
+
const objectHash = createHash("sha256").update(bytes).digest("hex");
|
|
99
|
+
if (objectHash !== row.sha256) {
|
|
100
|
+
throw errorWithCode("object_corrupt", `Object ${row.sha256} failed SHA-256 verification for version ${row.doc_version_id}.`);
|
|
101
|
+
}
|
|
102
|
+
let extracted;
|
|
103
|
+
try {
|
|
104
|
+
extracted = await parseDocument(
|
|
105
|
+
path.join(root, "objects", row.sha256.slice(0, 2), row.sha256.slice(2, 4), row.sha256),
|
|
106
|
+
namespace,
|
|
107
|
+
void 0,
|
|
108
|
+
{ mediaHint: mediaHint(row.extractor, row.label) }
|
|
109
|
+
);
|
|
110
|
+
} catch (error) {
|
|
111
|
+
if (error.code === "rehash_failed") throw error;
|
|
112
|
+
throw errorWithCode("rehash_failed", `Cannot re-extract version ${row.doc_version_id}: ${error instanceof Error ? error.message : String(error)}`);
|
|
113
|
+
}
|
|
114
|
+
const facts = computeContentHashFacts(extracted);
|
|
115
|
+
if (facts.skipped) {
|
|
116
|
+
skipped += 1;
|
|
117
|
+
} else if (sameFacts(row, facts)) {
|
|
118
|
+
unchanged += 1;
|
|
119
|
+
} else {
|
|
120
|
+
updates.push({
|
|
121
|
+
docVersionId: row.doc_version_id,
|
|
122
|
+
contentHash: facts.contentHash,
|
|
123
|
+
normalizedHash: facts.normalizedHash,
|
|
124
|
+
contentKind: facts.contentKind,
|
|
125
|
+
hashVersion: facts.hashVersion
|
|
126
|
+
});
|
|
127
|
+
}
|
|
128
|
+
}
|
|
129
|
+
if (apply && updates.length > 0) {
|
|
130
|
+
db.exec("BEGIN IMMEDIATE;");
|
|
131
|
+
try {
|
|
132
|
+
const update = db.prepare(`UPDATE doc_version SET content_hash = ?, normalized_hash = ?, content_kind = ?, hash_version = ?
|
|
133
|
+
WHERE doc_version_id = ?`);
|
|
134
|
+
for (const row of updates) update.run(row.contentHash, row.normalizedHash, row.contentKind, row.hashVersion, row.docVersionId);
|
|
135
|
+
await writeRootManifest(root, db);
|
|
136
|
+
db.exec("COMMIT;");
|
|
137
|
+
} catch (error) {
|
|
138
|
+
try {
|
|
139
|
+
db.exec("ROLLBACK;");
|
|
140
|
+
} catch {
|
|
141
|
+
}
|
|
142
|
+
throw errorWithCode("rehash_failed", `Backfill transaction failed: ${error instanceof Error ? error.message : String(error)}`);
|
|
143
|
+
}
|
|
144
|
+
}
|
|
145
|
+
return {
|
|
146
|
+
applied: apply,
|
|
147
|
+
updated: updates.length,
|
|
148
|
+
unchanged,
|
|
149
|
+
skipped,
|
|
150
|
+
...!apply ? { backupReminder: reminder() } : {}
|
|
151
|
+
};
|
|
152
|
+
} finally {
|
|
153
|
+
db.close();
|
|
154
|
+
}
|
|
155
|
+
}
|
|
156
|
+
function reminder() {
|
|
157
|
+
return "Before applying, back up this namespace with: rag backup --to <backup-dir>.";
|
|
158
|
+
}
|
|
159
|
+
export {
|
|
160
|
+
computeContentHashFacts,
|
|
161
|
+
rehashNamespace
|
|
162
|
+
};
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
const SCHEMA_VERSION =
|
|
1
|
+
const SCHEMA_VERSION = 2;
|
|
2
2
|
const MIGRATIONS = [
|
|
3
3
|
{
|
|
4
4
|
version: 1,
|
|
@@ -170,6 +170,16 @@ BEFORE DELETE ON doc_version
|
|
|
170
170
|
BEGIN
|
|
171
171
|
SELECT RAISE(ABORT, 'doc_version history is immutable');
|
|
172
172
|
END;`
|
|
173
|
+
},
|
|
174
|
+
{
|
|
175
|
+
version: 2,
|
|
176
|
+
sql: `CREATE TABLE schema_migration (
|
|
177
|
+
version INTEGER PRIMARY KEY,
|
|
178
|
+
applied_at TEXT NOT NULL
|
|
179
|
+
);
|
|
180
|
+
|
|
181
|
+
ALTER TABLE doc_version ADD COLUMN content_kind TEXT;
|
|
182
|
+
ALTER TABLE doc_version ADD COLUMN hash_version INTEGER NOT NULL DEFAULT 1;`
|
|
173
183
|
}
|
|
174
184
|
];
|
|
175
185
|
export {
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "agent-rag-mcp",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.4.0",
|
|
4
4
|
"description": "Deterministic, namespace-isolated document retrieval with a CLI and MCP server.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"repository": {
|
|
@@ -28,12 +28,16 @@
|
|
|
28
28
|
"bin",
|
|
29
29
|
"dist",
|
|
30
30
|
"extensions",
|
|
31
|
+
"skills",
|
|
31
32
|
"README.md",
|
|
32
33
|
"LICENSE"
|
|
33
34
|
],
|
|
34
35
|
"pi": {
|
|
35
36
|
"extensions": [
|
|
36
37
|
"./extensions/agent-rag.ts"
|
|
38
|
+
],
|
|
39
|
+
"skills": [
|
|
40
|
+
"./skills"
|
|
37
41
|
]
|
|
38
42
|
},
|
|
39
43
|
"engines": {
|
|
@@ -58,6 +62,7 @@
|
|
|
58
62
|
"mammoth": "1.13.0",
|
|
59
63
|
"pdfjs-dist": "6.4.299",
|
|
60
64
|
"tesseract.js": "7.0.0",
|
|
65
|
+
"yaml": "^2.9.1",
|
|
61
66
|
"zod": "4.6.7"
|
|
62
67
|
}
|
|
63
68
|
}
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: agent-rag
|
|
3
|
+
description: "Trigger: search or read the indexed knowledge base with explicit client/project scope and verifiable citations."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
## Choose a tool
|
|
7
|
+
|
|
8
|
+
- Use `rag_find` to search a client/project namespace and get a bounded catalog of matching citations.
|
|
9
|
+
- Use `rag_read` only when a citation needs more surrounding text; pass its `docVersionId` and `chunkOrdinal` from `rag_find`.
|
|
10
|
+
- The orchestrator only receives `rag_find`; delegate exploration when a citation needs reading.
|
|
11
|
+
|
|
12
|
+
## Evidence and scope
|
|
13
|
+
|
|
14
|
+
- The `namespace` is always explicit: pass an object with non-empty `client` and `project`; never infer it from cwd or discover another namespace.
|
|
15
|
+
- Cite source wording with a verbatim quote. Citations must be quoted, never paraphrased as source text.
|
|
16
|
+
- Keep the citation's document/version and section context with the quote; say when a search returns no matches.
|
|
17
|
+
- Treat `rag_find` excerpts as bounded evidence, not the complete source. Use `rag_read` for a bounded passage only when needed.
|
|
18
|
+
|
|
19
|
+
## Config and CLI
|
|
20
|
+
|
|
21
|
+
- CLI namespace/root options are `--client`, `--project`, `--root`; environment alternatives are `RAG_CLIENT`, `RAG_PROJECT`, `RAG_ROOT`.
|
|
22
|
+
- The CLI home is `RAG_HOME ?? os.homedir()`. The MCP server and Pi extension pass cwd as `homeDir`.
|
|
23
|
+
- The only config keys read in `src/config/discovery.mts` are `client`, `project`, `root`, and `failClosed`; unknown keys are silently ignored.
|
|
24
|
+
- Config candidates, first existing only: `<cwd>/rag.config.json`, `<dirname(cwd)>/rag.config.json` (one level up only), then `<homeDir>/.rag/config.json`.
|
|
25
|
+
- Only JSON syntax errors (`config_invalid`) or config-file I/O errors (`config_unreadable`) raise those config errors.
|
|
26
|
+
- Each field resolves independently to its first available value: client `--client` > `RAG_CLIENT` > file > null; project follows the same chain.
|
|
27
|
+
- Root precedence is `--root` > `RAG_ROOT` > file > `<homeDir>/.rag`; fields can resolve from different sources.
|
|
28
|
+
- Discovery reports its source as `args`, `env`, `cwd`, `parent`, `home`, or `unscoped`.
|
|
29
|
+
- `client` must match `/^[A-Za-z0-9][A-Za-z0-9._-]*$/`; `.` and `..` are rejected with `invalid_client_name`. Its DB is `<root>/clients/<client>/rag.db`.
|
|
30
|
+
- `project` is logical only: its slug is stored in `project.slug` rows in that client DB; it creates no directory and has no name validation.
|
|
31
|
+
- `root` defaults to `<homeDir>/.rag`; the CLI resolves a relative root against cwd for each command.
|
|
32
|
+
- Root layout: `clients/<client>/rag.db`, `objects/<aa>/<bb>/<sha256>`, and `manifest.json`.
|
|
33
|
+
- `failClosed` defaults to true and is file-only (no flag or environment variable). For writes without a namespace it throws `namespace_required`; false uses reserved `default/default`. Reads are unaffected.
|
|
34
|
+
- `rag init` does not write `rag.config.json`; it creates only the client DB, object store, and root manifest.
|
|
35
|
+
- MCP tools require `namespace: {client, project}` on every call. The server resolves only `root` from discovery; config, `RAG_CLIENT`, and `RAG_PROJECT` never provide tool identity.
|