agent-rag-mcp 0.3.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -32,6 +32,8 @@ The `rag` commands available today are:
32
32
  - `rag backup --to <dir>` — Backs up the manifest and object store.
33
33
  - `rag restore --from <dir>` — Restores a backup under this root.
34
34
  - `rag reindex` — Rebuilds the database from the manifest and objects.
35
+ - `rag rehash` — Previews recomputed hashes for stored versions; `rag rehash --apply` writes the changed hash facts in one transaction. Preview first, and back up with `rag backup --to <backup-dir>` before applying.
36
+ - `rag rehash --apply` writes `manifest.json` before committing the database transaction. An interruption in that window can leave the manifest ahead of the database; rerunning `rag rehash --apply` or running `rag reindex` converges to the requested facts.
35
37
 
36
38
  ### CLI output
37
39
 
@@ -44,7 +46,11 @@ rag ls
44
46
  rag relabel <document-id> --label "replacement label"
45
47
  ```
46
48
 
47
- Structured files can use `.yaml`, `.yml`, `.json`, or `.jsonl`; each structured file yields one section per top-level key.
49
+ Structured files can use `.yaml`, `.yml`, `.json`, or `.jsonl`; each structured file yields sections from its parsed data tree. YAML uses the YAML 1.2 core schema, so words such as `yes` and `on` remain strings and unquoted timestamps remain strings.
50
+
51
+ ### Content identity
52
+
53
+ Structured JSON, JSONL, and YAML are hashed from compact canonical JSON for their parsed data tree, without a format prefix. Their `doc_version` rows use `content_kind='structured'` and `hash_version=2`; equivalent trees therefore share a content hash across formats and key order. Other formats retain the existing normalized extracted-text hash and use `content_kind=NULL` with `hash_version=1`. Existing version rows stay at version 1 with a null content kind until repaired with `rag rehash`; the repair never rewrites stored object bytes and skips structured documents that cannot be canonically hashed.
48
54
 
49
55
  The default knowledge-base root is `%USERPROFILE%\.rag\`. Each client has its own `rag.db` under `.rag\clients\<client>\`; projects are logical within that client and are identified by `project_id`. The namespace is the pair `(client, project)`. By default, `failClosed` is `true`; without a namespace, reads are allowed, and when `failClosed` is set to `false`, writes go to the reserved `default/default` namespace. The Tier 3 human review queue is available through `rag review list`.
50
56
 
@@ -1,3 +1,5 @@
1
+ import { parseDocument } from "yaml";
2
+ import { canonicalizeStructuredContent, NonFiniteNumberCanonicalizationError } from "../store/canonical.mjs";
1
3
  class StructuredTextExtractionError extends Error {
2
4
  code;
3
5
  constructor(code) {
@@ -29,70 +31,127 @@ function supportsReference(ref, options) {
29
31
  }
30
32
  function renderJsonValue(value) {
31
33
  if (typeof value === "string") return value;
32
- return JSON.stringify(value, null, 2) ?? "null";
34
+ return JSON.stringify(value, (_key, child) => typeof child === "bigint" ? child.toString() : child, 2) ?? "null";
35
+ }
36
+ function parseJsonContent(content) {
37
+ return JSON.parse(content, (_key, value, context) => {
38
+ const source = context?.source;
39
+ if (typeof value === "number" && !Number.isSafeInteger(value) && source !== void 0 && /^-?(?:0|[1-9][0-9]*)$/.test(source)) {
40
+ return BigInt(source);
41
+ }
42
+ return value;
43
+ });
44
+ }
45
+ function canonicalFields(value) {
46
+ try {
47
+ return { canonicalContent: canonicalizeStructuredContent(value) };
48
+ } catch (error) {
49
+ if (error instanceof NonFiniteNumberCanonicalizationError) return { canonicalizationError: error.code };
50
+ throw error;
51
+ }
33
52
  }
34
53
  function extractJson(content) {
35
54
  let parsed;
36
55
  try {
37
- parsed = JSON.parse(content);
56
+ parsed = parseJsonContent(content);
38
57
  } catch {
39
58
  throw new StructuredTextExtractionError("extraction_invalid_json");
40
59
  }
41
- if (typeof parsed === "object" && parsed !== null && !Array.isArray(parsed)) {
42
- return Object.entries(parsed).map(([heading, value], ordinal) => ({
43
- ordinal,
44
- heading,
45
- level: 1,
46
- text: renderJsonValue(value)
47
- }));
48
- }
49
- return [{ ordinal: 0, heading: null, level: null, text: renderJsonValue(parsed) }];
60
+ const sections = typeof parsed === "object" && parsed !== null && !Array.isArray(parsed) ? Object.entries(parsed).map(([heading, value], ordinal) => ({
61
+ ordinal,
62
+ heading,
63
+ level: 1,
64
+ text: renderJsonValue(value)
65
+ })) : [{ ordinal: 0, heading: null, level: null, text: renderJsonValue(parsed) }];
66
+ return { sections, ...canonicalFields(parsed) };
50
67
  }
51
68
  function extractJsonl(content) {
52
69
  const sections = [];
70
+ const records = [];
53
71
  for (const line of content.split(/\r?\n/)) {
54
72
  const text = line.trim();
55
73
  if (text.length === 0) continue;
56
74
  try {
57
- JSON.parse(text);
75
+ records.push(parseJsonContent(text));
58
76
  } catch {
59
77
  throw new StructuredTextExtractionError("extraction_invalid_json");
60
78
  }
61
79
  sections.push({ ordinal: sections.length, heading: null, level: null, text });
62
80
  }
63
- return sections;
81
+ return { sections, ...canonicalFields(records) };
64
82
  }
65
- function extractYaml(content) {
66
- const lines = content.split("\n").map((line) => line.replace(/\r$/, ""));
67
- const topLevelKey = /^([A-Za-z0-9_.-]+):/;
68
- const starts = [];
69
- for (let index = 0; index < lines.length; index += 1) {
70
- const line = lines[index];
71
- if (line === void 0) continue;
72
- if (/^[ \t]*\t/.test(line)) throw new StructuredTextExtractionError("extraction_invalid_yaml");
73
- const match = topLevelKey.exec(line);
74
- if (match?.[1] !== void 0) starts.push({ index, heading: match[1] });
83
+ function normalizeYamlValue(value, ancestors = /* @__PURE__ */ new Set()) {
84
+ if (typeof value !== "object" || value === null) return value;
85
+ if (ancestors.has(value)) throw new StructuredTextExtractionError("extraction_invalid_yaml");
86
+ ancestors.add(value);
87
+ try {
88
+ if (value instanceof Map) {
89
+ const result2 = /* @__PURE__ */ Object.create(null);
90
+ for (const [key, child] of value) {
91
+ const stringKey = typeof key === "string" ? key : canonicalizeStructuredContent(normalizeYamlValue(key, ancestors));
92
+ result2[stringKey] = normalizeYamlValue(child, ancestors);
93
+ }
94
+ return result2;
95
+ }
96
+ if (Array.isArray(value)) return value.map((child) => normalizeYamlValue(child, ancestors));
97
+ const result = /* @__PURE__ */ Object.create(null);
98
+ for (const [key, child] of Object.entries(value)) result[key] = normalizeYamlValue(child, ancestors);
99
+ return result;
100
+ } finally {
101
+ ancestors.delete(value);
75
102
  }
76
- if (starts.length === 0) {
77
- return [{ ordinal: 0, heading: null, level: null, text: content }];
103
+ }
104
+ function renderYamlValue(value) {
105
+ if (typeof value === "string") return value;
106
+ return JSON.stringify(value, (_key, child) => typeof child === "bigint" ? child.toString() : child, 2) ?? "null";
107
+ }
108
+ function yamlTopLevelStarts(document) {
109
+ const root = document.contents;
110
+ const starts = /* @__PURE__ */ new Map();
111
+ for (const pair of root?.items ?? []) {
112
+ const position = pair.key?.range?.[0];
113
+ if (position === void 0) continue;
114
+ const rawKey = pair.key && Object.hasOwn(pair.key, "value") ? pair.key.value : pair.key?.toJSON?.();
115
+ const key = normalizeYamlValue(rawKey);
116
+ const stringKey = typeof key === "string" ? key : canonicalizeStructuredContent(key);
117
+ starts.set(stringKey, position);
78
118
  }
79
- let hasCurrentBlock = false;
80
- for (const line of lines) {
81
- const trimmed = line.trim();
82
- if (trimmed.length === 0 || trimmed.startsWith("#") || trimmed === "---" || trimmed === "...") continue;
83
- const match = topLevelKey.exec(line);
84
- if (match?.[1] !== void 0) {
85
- hasCurrentBlock = true;
86
- continue;
119
+ return starts;
120
+ }
121
+ function extractYaml(content) {
122
+ let parsed;
123
+ let topLevelStarts;
124
+ let canonicalizationError;
125
+ try {
126
+ const document = parseDocument(content, { schema: "core", uniqueKeys: false, intAsBigInt: true });
127
+ if (document.errors.length > 0) throw new Error("invalid YAML document");
128
+ parsed = normalizeYamlValue(document.toJS({ mapAsMap: true }));
129
+ topLevelStarts = yamlTopLevelStarts(document);
130
+ } catch (error) {
131
+ if (!(error instanceof NonFiniteNumberCanonicalizationError)) {
132
+ throw new StructuredTextExtractionError("extraction_invalid_yaml");
87
133
  }
88
- const indentation = line.match(/^ */)?.[0].length ?? 0;
89
- if (indentation === 0 || !hasCurrentBlock) throw new StructuredTextExtractionError("extraction_invalid_yaml");
134
+ canonicalizationError = error.code;
135
+ parsed = null;
136
+ topLevelStarts = /* @__PURE__ */ new Map();
90
137
  }
91
- return starts.map((start, ordinal) => {
92
- const nextStart = starts[ordinal + 1];
93
- const block = lines.slice(start.index, nextStart?.index ?? lines.length).join("\n").trim();
94
- return { ordinal, heading: start.heading, level: 1, text: block };
95
- });
138
+ if (canonicalizationError !== void 0) {
139
+ return { sections: [{ ordinal: 0, heading: null, level: null, text: content.trim() }], canonicalizationError };
140
+ }
141
+ let sections;
142
+ if (typeof parsed === "object" && parsed !== null && !Array.isArray(parsed)) {
143
+ const entries = Object.entries(parsed).map(([heading, value]) => ({ heading, value, start: topLevelStarts.get(heading) })).filter((entry) => entry.start !== void 0).sort((left, right) => left.start - right.start);
144
+ sections = entries.map(({ heading, start }, ordinal) => ({
145
+ ordinal,
146
+ heading,
147
+ level: 1,
148
+ text: content.slice(start, entries[ordinal + 1]?.start ?? content.length).trim()
149
+ }));
150
+ if (sections.length === 0) sections = [{ ordinal: 0, heading: null, level: null, text: renderYamlValue(parsed) }];
151
+ } else {
152
+ sections = [{ ordinal: 0, heading: null, level: null, text: renderYamlValue(parsed) }];
153
+ }
154
+ return { sections, ...canonicalFields(parsed) };
96
155
  }
97
156
  function createStructuredTextExtractor(options) {
98
157
  return {
@@ -108,19 +167,26 @@ function createStructuredTextExtractor(options) {
108
167
  if (content.length === 0) {
109
168
  return { sections: [], versionLabel: null, mediaType: null };
110
169
  }
111
- let sections;
170
+ let extracted;
112
171
  switch (options.kind) {
113
172
  case "json":
114
- sections = extractJson(content);
173
+ extracted = extractJson(content);
115
174
  break;
116
175
  case "jsonl":
117
- sections = extractJsonl(content);
176
+ extracted = extractJsonl(content);
118
177
  break;
119
178
  case "yaml":
120
- sections = extractYaml(content);
179
+ extracted = extractYaml(content);
121
180
  break;
122
181
  }
123
- return { sections, versionLabel: null, mediaType: options.mediaTypes[0] ?? null };
182
+ const result = {
183
+ sections: extracted.sections,
184
+ versionLabel: null,
185
+ mediaType: options.mediaTypes[0] ?? null,
186
+ ...extracted.canonicalContent === void 0 ? {} : { canonicalContent: extracted.canonicalContent },
187
+ ...extracted.canonicalizationError === void 0 ? {} : { canonicalizationError: extracted.canonicalizationError }
188
+ };
189
+ return result;
124
190
  }
125
191
  };
126
192
  }
@@ -93,6 +93,12 @@ function renderOutput(output, env) {
93
93
  return `Documents: ${result.documents} | Versions: ${result.versions} | Objects: ${result.objects}`;
94
94
  }
95
95
  if (typeof result.ok === "boolean" && typeof result.checked === "number") return renderDoctor(result);
96
+ if (typeof result.updated === "number" && typeof result.unchanged === "number" && typeof result.skipped === "number") {
97
+ const label = result.applied === true ? "Rehash applied" : "Rehash preview";
98
+ const summary = `${label}: ${result.updated} updated, ${result.unchanged} unchanged, ${result.skipped} skipped`;
99
+ return typeof result.backupReminder === "string" ? `${summary}
100
+ ${oneLine(result.backupReminder)}` : summary;
101
+ }
96
102
  if (Array.isArray(result.files) && record(result.summary)) return renderImport(result);
97
103
  if (typeof result.status === "string" && typeof result.documentId === "string") return renderDocumentAction(result);
98
104
  return JSON.stringify(output) ?? "null";
@@ -10,6 +10,7 @@ import { importDirectory } from "../ingest/import.mjs";
10
10
  import { clientDbPath, openDatabase, validateClientName } from "../store/db.mjs";
11
11
  import { backupRoot, reindexRoot, restoreRoot, writeRootManifest } from "../store/backup.mjs";
12
12
  import { readObject } from "../store/objects.mjs";
13
+ import { rehashNamespace } from "../store/rehash.mjs";
13
14
  import { getActiveLimits } from "../limits.mjs";
14
15
  import { renderOutput, resolveOutputFormat } from "./format.mjs";
15
16
  import {
@@ -37,6 +38,7 @@ Commands:
37
38
  backup --to <dir> Back up the manifest and object store
38
39
  restore --from <dir> Restore a backup into this root
39
40
  reindex Rebuild the database from manifest and objects
41
+ rehash [--apply] Recompute stored-version content hashes (preview by default)
40
42
 
41
43
  Options:
42
44
  --root <path> Knowledge-base root
@@ -46,7 +48,12 @@ Options:
46
48
  --from <dir> Backup source
47
49
  --type <type> Human relation type for rag review link
48
50
  --label <text> Replacement label for rag relabel
49
- --help Show this help`;
51
+ --apply Persist the rag rehash results
52
+ --help Show this help
53
+
54
+ Content identity:
55
+ JSON, JSONL, and YAML use canonical tree hashes (content_kind=structured, hash_version=2).
56
+ Other formats keep normalized-text hashes (content_kind=NULL, hash_version=1).`;
50
57
  const HELP = `${USAGE}
51
58
 
52
59
  Output format:
@@ -89,6 +96,7 @@ const NAMED_ERROR_CODES = /* @__PURE__ */ new Set([
89
96
  "backup_failed",
90
97
  "destination_exists",
91
98
  "namespace_mismatch",
99
+ "rehash_failed",
92
100
  "restore_destination_not_empty",
93
101
  "restore_failed",
94
102
  "foreign_client_in_manifest",
@@ -159,12 +167,17 @@ function parseArgs(argv) {
159
167
  const positional = [];
160
168
  const values = {};
161
169
  let help = false;
170
+ let apply = false;
162
171
  for (let index = 0; index < argv.length; index += 1) {
163
172
  const argument = argv[index];
164
173
  if (argument === "--help") {
165
174
  help = true;
166
175
  continue;
167
176
  }
177
+ if (argument === "--apply") {
178
+ apply = true;
179
+ continue;
180
+ }
168
181
  const assignment = /^--(to|from|type|label)=(.*)$/.exec(argument);
169
182
  if (assignment) {
170
183
  values[assignment[1]] = assignment[2];
@@ -191,7 +204,7 @@ function parseArgs(argv) {
191
204
  if (argument.startsWith("--")) throw usageError(`Unknown option: ${argument}`);
192
205
  positional.push(argument);
193
206
  }
194
- return { command: positional[0] ?? null, positional: positional.slice(1), help, values };
207
+ return { command: positional[0] ?? null, positional: positional.slice(1), help, values, apply };
195
208
  }
196
209
  function requireNamespace(config, mode) {
197
210
  const { namespace } = resolveCommandNamespace(config, mode);
@@ -399,6 +412,9 @@ async function runCli(options) {
399
412
  if (parsed.values.label !== void 0 && parsed.command !== "relabel") {
400
413
  return errorOutput(config.namespace, usageError('Option --label is only accepted by "rag relabel".'));
401
414
  }
415
+ if (parsed.apply && parsed.command !== "rehash") {
416
+ return errorOutput(config.namespace, usageError('Option --apply is only accepted by "rag rehash".'));
417
+ }
402
418
  if (parsed.command === "relabel" && parsed.values.label === void 0) {
403
419
  return errorOutput(config.namespace, usageError("Option --label requires a non-blank value.", "invalid_label"));
404
420
  }
@@ -657,6 +673,17 @@ async function runCli(options) {
657
673
  return commandErrorOutput(config.namespace, error);
658
674
  }
659
675
  }
676
+ if (parsed.command === "rehash") {
677
+ if (parsed.positional.length !== 0) return commandErrorOutput(config.namespace, usageError("rehash does not accept positional arguments."));
678
+ try {
679
+ const namespace = requireNamespace(config, parsed.apply ? "write" : "read");
680
+ if (!namespace) throw new NamespaceRequiredError();
681
+ const result = await rehashNamespace(path.resolve(options.cwd, config.root), namespace, parsed.apply);
682
+ return { exitCode: 0, output: { namespace, result } };
683
+ } catch (error) {
684
+ return commandErrorOutput(config.namespace, error);
685
+ }
686
+ }
660
687
  return errorOutput(config.namespace, usageError(parsed.command === null ? "Command is required." : `Unknown command: ${parsed.command}`));
661
688
  }
662
689
  async function main(dependencies = {}) {
@@ -6,7 +6,8 @@ import { ResourceLimitError } from "../ports/extractor.mjs";
6
6
  import { clientDbPath, openDatabase } from "../store/db.mjs";
7
7
  import { writeRootManifest } from "../store/backup.mjs";
8
8
  import { deriveTier0Relations, normalizeEntityKey } from "../store/identity.mjs";
9
- import { fingerprintDistance, fingerprintNormalizedText, NEAR_DUPLICATE_MAX_DISTANCE } from "../store/similarity.mjs";
9
+ import { computeContentHashFacts } from "../store/rehash.mjs";
10
+ import { fingerprintDistance, NEAR_DUPLICATE_MAX_DISTANCE } from "../store/similarity.mjs";
10
11
  import { putObjectFromPath, readObject } from "../store/objects.mjs";
11
12
  import { MAX_SOURCE_FILE_BYTES as SHARED_MAX_SOURCE_FILE_BYTES } from "../limits.mjs";
12
13
  import { parseDocument, ParseHostError } from "./parse-host.mjs";
@@ -86,9 +87,8 @@ async function ingestFile(input) {
86
87
  recordFailedRun(db, projectId, input.sourcePath, sha256, runStartedAt, error.message, error.code);
87
88
  throw error;
88
89
  }
89
- const normalizedText = extracted.sections.map((section) => section.text).join("\n").replace(/\s+/g, " ").trim().toLowerCase();
90
- const contentHash = createHash("sha256").update(normalizedText).digest("hex");
91
- const normalizedHash = fingerprintNormalizedText(normalizedText);
90
+ const hashFacts = computeContentHashFacts(extracted);
91
+ const { contentHash, normalizedHash, contentKind, hashVersion } = hashFacts;
92
92
  const sourceLabel = path.basename(input.sourcePath);
93
93
  const entityKey = normalizeEntityKey(input.entityKey ?? path.parse(sourceLabel).name);
94
94
  const label = input.label ?? sourceLabel;
@@ -125,8 +125,8 @@ async function ingestFile(input) {
125
125
  db.prepare(`INSERT INTO ingest_run (run_id, project_id, started_at, status, source_uri, sha256)
126
126
  VALUES (?, ?, ?, 'running', ?, ?)`).run(runId, projectId, now, input.sourcePath, stored.sha256);
127
127
  db.prepare("UPDATE doc_version SET is_current = 0, superseded_at = ? WHERE document_id = ? AND is_current = 1").run(now, documentId);
128
- db.prepare(`INSERT INTO doc_version (doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, extractor, extractor_version,
129
- ingest_run_id, is_current, ingested_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 1, ?)`).run(docVersionId, documentId, stored.sha256, extracted.versionLabel, contentHash, normalizedHash, extractorName, extractorVersion, runId, now);
128
+ db.prepare(`INSERT INTO doc_version (doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, content_kind, hash_version, extractor, extractor_version,
129
+ ingest_run_id, is_current, ingested_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1, ?)`).run(docVersionId, documentId, stored.sha256, extracted.versionLabel, contentHash, normalizedHash, contentKind, hashVersion, extractorName, extractorVersion, runId, now);
130
130
  let offset = 0;
131
131
  extracted.sections.forEach((section, ordinal) => {
132
132
  const sectionId = randomUUID();
@@ -146,7 +146,8 @@ async function ingestFile(input) {
146
146
  contentHash,
147
147
  entityKey,
148
148
  projectId,
149
- versionLabel: extracted.versionLabel
149
+ versionLabel: extracted.versionLabel,
150
+ hashVersion
150
151
  };
151
152
  const insertRelation = db.prepare(`INSERT INTO doc_relation
152
153
  (relation_id, from_doc_version_id, to_doc_version_id, relation_type, score, decided_by, decided_at, evidence)
@@ -154,7 +155,7 @@ async function ingestFile(input) {
154
155
  const currentDocument = db.prepare("SELECT deleted_at FROM document WHERE document_id = ?").get(documentId);
155
156
  if (currentDocument?.deleted_at === null) {
156
157
  const priorVersions = db.prepare(`SELECT dv.doc_version_id AS docVersionId, dv.sha256,
157
- dv.content_hash AS contentHash, dv.normalized_hash AS normalizedHash,
158
+ dv.content_hash AS contentHash, dv.normalized_hash AS normalizedHash, dv.hash_version AS hashVersion,
158
159
  e.entity_key AS entityKey, d.project_id AS projectId, dv.version_label AS versionLabel
159
160
  FROM doc_version AS dv
160
161
  JOIN document AS d ON d.document_id = dv.document_id
@@ -173,7 +174,7 @@ async function ingestFile(input) {
173
174
  JSON.stringify(relation.evidence)
174
175
  );
175
176
  }
176
- if (prior.normalizedHash !== null) {
177
+ if (prior.hashVersion === hashVersion && prior.normalizedHash !== null) {
177
178
  const distance = fingerprintDistance(normalizedHash, prior.normalizedHash);
178
179
  if (distance <= NEAR_DUPLICATE_MAX_DISTANCE) {
179
180
  const relationId = `rel_${createHash("sha256").update(JSON.stringify([docVersionId, prior.docVersionId, "near_duplicate"])).digest("hex")}`;
@@ -0,0 +1,65 @@
1
+ const TEXT_HASH_VERSION = 1;
2
+ const STRUCTURED_HASH_VERSION = 2;
3
+ class NonFiniteNumberCanonicalizationError extends Error {
4
+ code = "non_finite_number";
5
+ constructor() {
6
+ super("Canonical structured content contains a non-finite number.");
7
+ this.name = "NonFiniteNumberCanonicalizationError";
8
+ }
9
+ }
10
+ function compareCodePoints(left, right) {
11
+ const leftPoints = Array.from(left, (character) => character.codePointAt(0));
12
+ const rightPoints = Array.from(right, (character) => character.codePointAt(0));
13
+ const sharedLength = Math.min(leftPoints.length, rightPoints.length);
14
+ for (let index = 0; index < sharedLength; index += 1) {
15
+ const leftPoint = leftPoints[index];
16
+ const rightPoint = rightPoints[index];
17
+ if (leftPoint !== rightPoint) return leftPoint - rightPoint;
18
+ }
19
+ return leftPoints.length - rightPoints.length;
20
+ }
21
+ function encode(value, ancestors) {
22
+ if (value === null) return "null";
23
+ if (typeof value === "string") return JSON.stringify(value);
24
+ if (typeof value === "boolean") return value ? "true" : "false";
25
+ if (typeof value === "number") {
26
+ if (!Number.isFinite(value)) throw new NonFiniteNumberCanonicalizationError();
27
+ return JSON.stringify(value);
28
+ }
29
+ if (typeof value === "bigint") {
30
+ if (value <= BigInt(Number.MAX_SAFE_INTEGER) && value >= BigInt(Number.MIN_SAFE_INTEGER)) {
31
+ return JSON.stringify(Number(value));
32
+ }
33
+ return JSON.stringify(value.toString());
34
+ }
35
+ if (Array.isArray(value)) {
36
+ if (ancestors.has(value)) throw new TypeError("Structured content must be acyclic.");
37
+ ancestors.add(value);
38
+ try {
39
+ return `[${value.map((item) => item === void 0 ? "null" : encode(item, ancestors)).join(",")}]`;
40
+ } finally {
41
+ ancestors.delete(value);
42
+ }
43
+ }
44
+ if (typeof value === "object") {
45
+ const object = value;
46
+ if (ancestors.has(object)) throw new TypeError("Structured content must be acyclic.");
47
+ ancestors.add(object);
48
+ try {
49
+ const members = Object.keys(object).filter((key) => object[key] !== void 0).sort(compareCodePoints).map((key) => `${JSON.stringify(key)}:${encode(object[key], ancestors)}`);
50
+ return `{${members.join(",")}}`;
51
+ } finally {
52
+ ancestors.delete(object);
53
+ }
54
+ }
55
+ return "null";
56
+ }
57
+ function canonicalizeStructuredContent(value) {
58
+ return encode(value, /* @__PURE__ */ new Set());
59
+ }
60
+ export {
61
+ NonFiniteNumberCanonicalizationError,
62
+ STRUCTURED_HASH_VERSION,
63
+ TEXT_HASH_VERSION,
64
+ canonicalizeStructuredContent
65
+ };
@@ -57,7 +57,7 @@ function deriveTier0Relations(a, b) {
57
57
  const relations = [];
58
58
  if (a.sha256 === b.sha256) {
59
59
  relations.push(makeRelation(a.docVersionId, b.docVersionId, "same_bytes", { sha256: a.sha256 }));
60
- } else if (a.contentHash !== null && b.contentHash !== null && a.contentHash === b.contentHash) {
60
+ } else if (a.contentHash !== null && b.contentHash !== null && a.contentHash === b.contentHash && (a.hashVersion ?? 1) === (b.hashVersion ?? 1)) {
61
61
  relations.push(makeRelation(a.docVersionId, b.docVersionId, "same_content", { contentHash: a.contentHash }));
62
62
  }
63
63
  const sameEntity = normalizeEntityKey(a.entityKey) === normalizeEntityKey(b.entityKey);
@@ -64,6 +64,7 @@ function listReviewCandidates(db, projectId) {
64
64
  JOIN entity AS right_entity ON right_entity.entity_id = right_document.entity_id
65
65
  WHERE left_document.project_id = ? AND right_document.project_id = left_document.project_id
66
66
  AND left_version.content_hash IS NOT NULL AND left_version.content_hash = right_version.content_hash
67
+ AND left_version.hash_version = right_version.hash_version
67
68
  AND left_document.deleted_at IS NULL AND right_document.deleted_at IS NULL
68
69
  AND left_entity.deleted_at IS NULL AND right_entity.deleted_at IS NULL
69
70
  AND NOT EXISTS (
@@ -86,6 +87,7 @@ function listReviewCandidates(db, projectId) {
86
87
  JOIN entity AS to_entity ON to_entity.entity_id = to_document.entity_id
87
88
  WHERE near_relation.relation_type = 'near_duplicate' AND near_relation.decided_by <> 'human'
88
89
  AND from_document.project_id = ? AND to_document.project_id = from_document.project_id
90
+ AND from_version.hash_version = to_version.hash_version
89
91
  AND from_document.deleted_at IS NULL AND to_document.deleted_at IS NULL
90
92
  AND from_entity.deleted_at IS NULL AND to_entity.deleted_at IS NULL
91
93
  AND NOT EXISTS (
@@ -21,7 +21,7 @@ function buildManifest(db) {
21
21
  ingestRuns: orderedRows(db, "ingest_run"),
22
22
  entities: db.prepare("SELECT entity_id, project_id, entity_key, kind, title, created_at, updated_at, deleted_at FROM entity ORDER BY entity_id").all(),
23
23
  documents: db.prepare("SELECT document_id, entity_id, project_id, label, created_at, updated_at, deleted_at FROM document ORDER BY document_id").all(),
24
- versions: db.prepare("SELECT doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, extractor, extractor_version, ingest_run_id, is_current, ingested_at, superseded_at FROM doc_version ORDER BY doc_version_id").all(),
24
+ versions: db.prepare("SELECT doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, content_kind, hash_version, extractor, extractor_version, ingest_run_id, is_current, ingested_at, superseded_at FROM doc_version ORDER BY doc_version_id").all(),
25
25
  sections: db.prepare("SELECT section_id, doc_version_id, ordinal, heading, level, char_start, char_end FROM section ORDER BY doc_version_id, ordinal, section_id").all(),
26
26
  chunks: db.prepare("SELECT chunk_id, doc_version_id, section_id, ordinal, text, char_start, char_end, token_count FROM chunk ORDER BY doc_version_id, ordinal, chunk_id").all(),
27
27
  relations: db.prepare("SELECT relation_id, from_doc_version_id, to_doc_version_id, relation_type, score, decided_by, decided_at, evidence FROM doc_relation ORDER BY relation_id").all(),
@@ -38,7 +38,7 @@ const ROW_COLUMNS = {
38
38
  ingestRuns: ["run_id", "project_id", "started_at", "finished_at", "status", "source_uri", "sha256", "result_code", "detail"],
39
39
  entities: ["entity_id", "project_id", "entity_key", "kind", "title", "created_at", "updated_at", "deleted_at"],
40
40
  documents: ["document_id", "entity_id", "project_id", "label", "created_at", "updated_at", "deleted_at"],
41
- versions: ["doc_version_id", "document_id", "sha256", "version_label", "content_hash", "normalized_hash", "extractor", "extractor_version", "ingest_run_id", "is_current", "ingested_at", "superseded_at"],
41
+ versions: ["doc_version_id", "document_id", "sha256", "version_label", "content_hash", "normalized_hash", "content_kind", "hash_version", "extractor", "extractor_version", "ingest_run_id", "is_current", "ingested_at", "superseded_at"],
42
42
  sections: ["section_id", "doc_version_id", "ordinal", "heading", "level", "char_start", "char_end"],
43
43
  chunks: ["chunk_id", "doc_version_id", "section_id", "ordinal", "text", "char_start", "char_end", "token_count"],
44
44
  relations: ["relation_id", "from_doc_version_id", "to_doc_version_id", "relation_type", "score", "decided_by", "decided_at", "evidence"],
@@ -47,8 +47,12 @@ const ROW_COLUMNS = {
47
47
  };
48
48
  function validateRowKeys(value, field) {
49
49
  const expected = ROW_COLUMNS[field];
50
+ const legacyVersionColumns = field === "versions" ? expected.filter((column) => column !== "content_kind" && column !== "hash_version") : null;
50
51
  for (const row of value) {
51
- if (!isRecord(row) || Object.keys(row).length !== expected.length || expected.some((column) => !Object.hasOwn(row, column))) {
52
+ if (!isRecord(row)) throw new ManifestError("manifest_invalid", `Manifest ${field} row keys must match expected columns: ${expected.join(", ")}.`);
53
+ const keys = Object.keys(row);
54
+ const matches = (columns) => keys.length === columns.length && columns.every((column) => Object.hasOwn(row, column));
55
+ if (!matches(expected) && !(legacyVersionColumns !== null && matches(legacyVersionColumns))) {
52
56
  throw new ManifestError("manifest_invalid", `Manifest ${field} row keys must match expected columns: ${expected.join(", ")}.`);
53
57
  }
54
58
  }
@@ -99,7 +103,12 @@ function importManifest(db, value) {
99
103
  insertRows(db, "INSERT INTO entity (entity_id, project_id, entity_key, kind, title, created_at, updated_at, deleted_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", value.entities, (r) => [r.entity_id, r.project_id, r.entity_key, r.kind, r.title, r.created_at, r.updated_at, r.deleted_at]);
100
104
  insertRows(db, "INSERT INTO document (document_id, entity_id, project_id, label, created_at, updated_at, deleted_at) VALUES (?, ?, ?, ?, ?, ?, ?)", value.documents, (r) => [r.document_id, r.entity_id, r.project_id, r.label, r.created_at, r.updated_at, r.deleted_at]);
101
105
  insertRows(db, "INSERT INTO object (sha256, byte_size, rel_path, media_type, created_at, last_verified_at) VALUES (?, ?, ?, ?, ?, ?)", value.objects, (r) => [r.sha256, r.byte_size, r.rel_path, r.media_type, r.created_at, r.last_verified_at]);
102
- insertRows(db, "INSERT INTO doc_version (doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, extractor, extractor_version, ingest_run_id, is_current, ingested_at, superseded_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", value.versions, (r) => [r.doc_version_id, r.document_id, r.sha256, r.version_label, r.content_hash, r.normalized_hash, r.extractor, r.extractor_version, r.ingest_run_id, r.is_current, r.ingested_at, r.superseded_at]);
106
+ const versions = value.versions.map((version) => ({
107
+ ...version,
108
+ content_kind: version.content_kind ?? null,
109
+ hash_version: version.hash_version ?? 1
110
+ }));
111
+ insertRows(db, "INSERT INTO doc_version (doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, content_kind, hash_version, extractor, extractor_version, ingest_run_id, is_current, ingested_at, superseded_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", versions, (r) => [r.doc_version_id, r.document_id, r.sha256, r.version_label, r.content_hash, r.normalized_hash, r.content_kind, r.hash_version, r.extractor, r.extractor_version, r.ingest_run_id, r.is_current, r.ingested_at, r.superseded_at]);
103
112
  insertRows(db, "INSERT INTO section (section_id, doc_version_id, ordinal, heading, level, char_start, char_end) VALUES (?, ?, ?, ?, ?, ?, ?)", value.sections, (r) => [r.section_id, r.doc_version_id, r.ordinal, r.heading, r.level, r.char_start, r.char_end]);
104
113
  insertRows(db, "INSERT INTO chunk (chunk_id, doc_version_id, section_id, ordinal, text, char_start, char_end, token_count) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", value.chunks, (r) => [r.chunk_id, r.doc_version_id, r.section_id, r.ordinal, r.text, r.char_start, r.char_end, r.token_count]);
105
114
  db.exec("INSERT INTO chunk_fts(chunk_fts) VALUES ('rebuild');");
@@ -0,0 +1,162 @@
1
+ import { createHash } from "node:crypto";
2
+ import * as fs from "node:fs/promises";
3
+ import * as path from "node:path";
4
+ import { NonFiniteNumberCanonicalizationError, STRUCTURED_HASH_VERSION, TEXT_HASH_VERSION } from "./canonical.mjs";
5
+ import { clientDbPath, openDatabase } from "./db.mjs";
6
+ import { writeRootManifest } from "./backup.mjs";
7
+ import { readObject } from "./objects.mjs";
8
+ import { fingerprintNormalizedText } from "./similarity.mjs";
9
+ import { parseDocument } from "../ingest/parse-host.mjs";
10
+ function computeContentHashFacts(extracted) {
11
+ const normalizedText = extracted.sections.map((section) => section.text).join("\n").replace(/\s+/g, " ").trim().toLowerCase();
12
+ let contentKind;
13
+ let hashVersion;
14
+ let hashInput;
15
+ let skipped = false;
16
+ try {
17
+ if (extracted.canonicalizationError === "non_finite_number") throw new NonFiniteNumberCanonicalizationError();
18
+ contentKind = extracted.canonicalContent === void 0 ? null : "structured";
19
+ hashVersion = extracted.canonicalContent === void 0 ? TEXT_HASH_VERSION : STRUCTURED_HASH_VERSION;
20
+ hashInput = extracted.canonicalContent ?? normalizedText;
21
+ } catch (error) {
22
+ if (!(error instanceof NonFiniteNumberCanonicalizationError)) throw error;
23
+ contentKind = null;
24
+ hashVersion = TEXT_HASH_VERSION;
25
+ hashInput = normalizedText;
26
+ skipped = true;
27
+ }
28
+ return {
29
+ contentHash: createHash("sha256").update(hashInput).digest("hex"),
30
+ normalizedHash: fingerprintNormalizedText(hashInput),
31
+ contentKind,
32
+ hashVersion,
33
+ skipped
34
+ };
35
+ }
36
+ function errorWithCode(code, message) {
37
+ return Object.assign(new Error(message), { code });
38
+ }
39
+ function mediaHint(extractor, label) {
40
+ const labelExtension = path.extname(label).toLowerCase();
41
+ switch (extractor) {
42
+ case "json":
43
+ return ".json";
44
+ case "jsonl":
45
+ return ".jsonl";
46
+ case "yaml":
47
+ return labelExtension === ".yml" ? ".yml" : ".yaml";
48
+ case "plain-text":
49
+ return labelExtension === ".md" ? ".md" : ".txt";
50
+ case "pdf":
51
+ return ".pdf";
52
+ case "docx":
53
+ return ".docx";
54
+ case "xlsx":
55
+ return ".xlsx";
56
+ case "pptx":
57
+ return ".pptx";
58
+ case "image-ocr":
59
+ return [".png", ".jpg", ".jpeg"].includes(labelExtension) ? labelExtension : ".png";
60
+ default:
61
+ throw errorWithCode("rehash_failed", `Cannot re-extract version with unknown extractor "${extractor}".`);
62
+ }
63
+ }
64
+ function sameFacts(row, facts) {
65
+ return row.content_hash === facts.contentHash && row.normalized_hash === facts.normalizedHash && row.content_kind === facts.contentKind && row.hash_version === facts.hashVersion;
66
+ }
67
+ async function rehashNamespace(root, namespace, apply) {
68
+ const dbPath = clientDbPath(root, namespace.client);
69
+ try {
70
+ await fs.access(dbPath);
71
+ } catch (error) {
72
+ if (error.code === "ENOENT") {
73
+ throw errorWithCode("database_not_found", `Client database not found: ${dbPath}`);
74
+ }
75
+ throw errorWithCode("io_error", `Unable to inspect client database: ${dbPath}`);
76
+ }
77
+ const db = openDatabase(dbPath);
78
+ try {
79
+ const project = db.prepare(`SELECT project.project_id FROM project
80
+ JOIN client ON client.client_id = project.client_id WHERE client.name = ? AND project.slug = ?`).get(namespace.client, namespace.project);
81
+ if (!project) return { applied: apply, updated: 0, unchanged: 0, skipped: 0, ...!apply ? { backupReminder: reminder() } : {} };
82
+ const rows = db.prepare(`SELECT version.doc_version_id, version.sha256, version.content_hash, version.normalized_hash,
83
+ version.content_kind, version.hash_version, version.extractor, document.label
84
+ FROM doc_version AS version
85
+ JOIN document ON document.document_id = version.document_id
86
+ WHERE document.project_id = ? ORDER BY version.doc_version_id`).all(project.project_id);
87
+ const updates = [];
88
+ let unchanged = 0;
89
+ let skipped = 0;
90
+ for (const row of rows) {
91
+ let bytes;
92
+ try {
93
+ bytes = await readObject(root, row.sha256);
94
+ } catch (error) {
95
+ const code = error.code === "ENOENT" ? "object_missing" : "object_read_failed";
96
+ throw errorWithCode(code, `Cannot read object ${row.sha256} for version ${row.doc_version_id}.`);
97
+ }
98
+ const objectHash = createHash("sha256").update(bytes).digest("hex");
99
+ if (objectHash !== row.sha256) {
100
+ throw errorWithCode("object_corrupt", `Object ${row.sha256} failed SHA-256 verification for version ${row.doc_version_id}.`);
101
+ }
102
+ let extracted;
103
+ try {
104
+ extracted = await parseDocument(
105
+ path.join(root, "objects", row.sha256.slice(0, 2), row.sha256.slice(2, 4), row.sha256),
106
+ namespace,
107
+ void 0,
108
+ { mediaHint: mediaHint(row.extractor, row.label) }
109
+ );
110
+ } catch (error) {
111
+ if (error.code === "rehash_failed") throw error;
112
+ throw errorWithCode("rehash_failed", `Cannot re-extract version ${row.doc_version_id}: ${error instanceof Error ? error.message : String(error)}`);
113
+ }
114
+ const facts = computeContentHashFacts(extracted);
115
+ if (facts.skipped) {
116
+ skipped += 1;
117
+ } else if (sameFacts(row, facts)) {
118
+ unchanged += 1;
119
+ } else {
120
+ updates.push({
121
+ docVersionId: row.doc_version_id,
122
+ contentHash: facts.contentHash,
123
+ normalizedHash: facts.normalizedHash,
124
+ contentKind: facts.contentKind,
125
+ hashVersion: facts.hashVersion
126
+ });
127
+ }
128
+ }
129
+ if (apply && updates.length > 0) {
130
+ db.exec("BEGIN IMMEDIATE;");
131
+ try {
132
+ const update = db.prepare(`UPDATE doc_version SET content_hash = ?, normalized_hash = ?, content_kind = ?, hash_version = ?
133
+ WHERE doc_version_id = ?`);
134
+ for (const row of updates) update.run(row.contentHash, row.normalizedHash, row.contentKind, row.hashVersion, row.docVersionId);
135
+ await writeRootManifest(root, db);
136
+ db.exec("COMMIT;");
137
+ } catch (error) {
138
+ try {
139
+ db.exec("ROLLBACK;");
140
+ } catch {
141
+ }
142
+ throw errorWithCode("rehash_failed", `Backfill transaction failed: ${error instanceof Error ? error.message : String(error)}`);
143
+ }
144
+ }
145
+ return {
146
+ applied: apply,
147
+ updated: updates.length,
148
+ unchanged,
149
+ skipped,
150
+ ...!apply ? { backupReminder: reminder() } : {}
151
+ };
152
+ } finally {
153
+ db.close();
154
+ }
155
+ }
156
+ function reminder() {
157
+ return "Before applying, back up this namespace with: rag backup --to <backup-dir>.";
158
+ }
159
+ export {
160
+ computeContentHashFacts,
161
+ rehashNamespace
162
+ };
@@ -1,4 +1,4 @@
1
- const SCHEMA_VERSION = 1;
1
+ const SCHEMA_VERSION = 2;
2
2
  const MIGRATIONS = [
3
3
  {
4
4
  version: 1,
@@ -170,6 +170,16 @@ BEFORE DELETE ON doc_version
170
170
  BEGIN
171
171
  SELECT RAISE(ABORT, 'doc_version history is immutable');
172
172
  END;`
173
+ },
174
+ {
175
+ version: 2,
176
+ sql: `CREATE TABLE schema_migration (
177
+ version INTEGER PRIMARY KEY,
178
+ applied_at TEXT NOT NULL
179
+ );
180
+
181
+ ALTER TABLE doc_version ADD COLUMN content_kind TEXT;
182
+ ALTER TABLE doc_version ADD COLUMN hash_version INTEGER NOT NULL DEFAULT 1;`
173
183
  }
174
184
  ];
175
185
  export {
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "agent-rag-mcp",
3
- "version": "0.3.0",
3
+ "version": "0.4.0",
4
4
  "description": "Deterministic, namespace-isolated document retrieval with a CLI and MCP server.",
5
5
  "license": "MIT",
6
6
  "repository": {
@@ -28,12 +28,16 @@
28
28
  "bin",
29
29
  "dist",
30
30
  "extensions",
31
+ "skills",
31
32
  "README.md",
32
33
  "LICENSE"
33
34
  ],
34
35
  "pi": {
35
36
  "extensions": [
36
37
  "./extensions/agent-rag.ts"
38
+ ],
39
+ "skills": [
40
+ "./skills"
37
41
  ]
38
42
  },
39
43
  "engines": {
@@ -58,6 +62,7 @@
58
62
  "mammoth": "1.13.0",
59
63
  "pdfjs-dist": "6.4.299",
60
64
  "tesseract.js": "7.0.0",
65
+ "yaml": "^2.9.1",
61
66
  "zod": "4.6.7"
62
67
  }
63
68
  }
@@ -0,0 +1,35 @@
1
+ ---
2
+ name: agent-rag
3
+ description: "Trigger: search or read the indexed knowledge base with explicit client/project scope and verifiable citations."
4
+ ---
5
+
6
+ ## Choose a tool
7
+
8
+ - Use `rag_find` to search a client/project namespace and get a bounded catalog of matching citations.
9
+ - Use `rag_read` only when a citation needs more surrounding text; pass its `docVersionId` and `chunkOrdinal` from `rag_find`.
10
+ - The orchestrator only receives `rag_find`; delegate exploration when a citation needs reading.
11
+
12
+ ## Evidence and scope
13
+
14
+ - The `namespace` is always explicit: pass an object with non-empty `client` and `project`; never infer it from cwd or discover another namespace.
15
+ - Cite source wording with a verbatim quote. Citations must be quoted, never paraphrased as source text.
16
+ - Keep the citation's document/version and section context with the quote; say when a search returns no matches.
17
+ - Treat `rag_find` excerpts as bounded evidence, not the complete source. Use `rag_read` for a bounded passage only when needed.
18
+
19
+ ## Config and CLI
20
+
21
+ - CLI namespace/root options are `--client`, `--project`, `--root`; environment alternatives are `RAG_CLIENT`, `RAG_PROJECT`, `RAG_ROOT`.
22
+ - The CLI home is `RAG_HOME ?? os.homedir()`. The MCP server and Pi extension pass cwd as `homeDir`.
23
+ - The only config keys read in `src/config/discovery.mts` are `client`, `project`, `root`, and `failClosed`; unknown keys are silently ignored.
24
+ - Config candidates, first existing only: `<cwd>/rag.config.json`, `<dirname(cwd)>/rag.config.json` (one level up only), then `<homeDir>/.rag/config.json`.
25
+ - Only JSON syntax errors (`config_invalid`) or config-file I/O errors (`config_unreadable`) raise those config errors.
26
+ - Each field resolves independently to its first available value: client `--client` > `RAG_CLIENT` > file > null; project follows the same chain.
27
+ - Root precedence is `--root` > `RAG_ROOT` > file > `<homeDir>/.rag`; fields can resolve from different sources.
28
+ - Discovery reports its source as `args`, `env`, `cwd`, `parent`, `home`, or `unscoped`.
29
+ - `client` must match `/^[A-Za-z0-9][A-Za-z0-9._-]*$/`; `.` and `..` are rejected with `invalid_client_name`. Its DB is `<root>/clients/<client>/rag.db`.
30
+ - `project` is logical only: its slug is stored in `project.slug` rows in that client DB; it creates no directory and has no name validation.
31
+ - `root` defaults to `<homeDir>/.rag`; the CLI resolves a relative root against cwd for each command.
32
+ - Root layout: `clients/<client>/rag.db`, `objects/<aa>/<bb>/<sha256>`, and `manifest.json`.
33
+ - `failClosed` defaults to true and is file-only (no flag or environment variable). For writes without a namespace it throws `namespace_required`; false uses reserved `default/default`. Reads are unaffected.
34
+ - `rag init` does not write `rag.config.json`; it creates only the client DB, object store, and root manifest.
35
+ - MCP tools require `namespace: {client, project}` on every call. The server resolves only `root` from discovery; config, `RAG_CLIENT`, and `RAG_PROJECT` never provide tool identity.