agent-rag-mcp 0.2.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -32,8 +32,25 @@ The `rag` commands available today are:
32
32
  - `rag backup --to <dir>` — Backs up the manifest and object store.
33
33
  - `rag restore --from <dir>` — Restores a backup under this root.
34
34
  - `rag reindex` — Rebuilds the database from the manifest and objects.
35
+ - `rag rehash` — Previews recomputed hashes for stored versions; `rag rehash --apply` writes the changed hash facts in one transaction. Preview first, and back up with `rag backup --to <backup-dir>` before applying.
36
+ - `rag rehash --apply` writes `manifest.json` before committing the database transaction. An interruption in that window can leave the manifest ahead of the database; rerunning `rag rehash --apply` or running `rag reindex` converges to the requested facts.
35
37
 
36
- Structured files can use `.yaml`, `.yml`, `.json`, or `.jsonl`; each structured file yields one section per top-level key.
38
+ ### CLI output
39
+
40
+ Interactive terminals render document lists as aligned columns and other supported results as concise summaries. When output is piped, commands keep the single-line JSON envelope. Set `RAG_FORMAT=json` or `RAG_FORMAT=pretty` to override terminal detection; an invalid value falls back to automatic detection. `rag --help` defaults to raw usage text on a terminal, while `RAG_FORMAT=json` restores its JSON envelope for scripts that parse the legacy help output. Set `NO_COLOR` (any value, including empty) to disable ANSI styling without changing the output layout.
41
+
42
+ `rag ls` includes each document's `documentId`. Copy that value to relabel the same document:
43
+
44
+ ```sh
45
+ rag ls
46
+ rag relabel <document-id> --label "replacement label"
47
+ ```
48
+
49
+ Structured files can use `.yaml`, `.yml`, `.json`, or `.jsonl`; each structured file yields sections from its parsed data tree. YAML uses the YAML 1.2 core schema, so words such as `yes` and `on` remain strings and unquoted timestamps remain strings.
50
+
51
+ ### Content identity
52
+
53
+ Structured JSON, JSONL, and YAML are hashed from compact canonical JSON for their parsed data tree, without a format prefix. Their `doc_version` rows use `content_kind='structured'` and `hash_version=2`; equivalent trees therefore share a content hash across formats and key order. Other formats retain the existing normalized extracted-text hash and use `content_kind=NULL` with `hash_version=1`. Existing version rows stay at version 1 with a null content kind until repaired with `rag rehash`; the repair never rewrites stored object bytes and skips structured documents that cannot be canonically hashed.
37
54
 
38
55
  The default knowledge-base root is `%USERPROFILE%\.rag\`. Each client has its own `rag.db` under `.rag\clients\<client>\`; projects are logical within that client and are identified by `project_id`. The namespace is the pair `(client, project)`. By default, `failClosed` is `true`; without a namespace, reads are allowed, and when `failClosed` is set to `false`, writes go to the reserved `default/default` namespace. The Tier 3 human review queue is available through `rag review list`.
39
56
 
@@ -78,7 +95,7 @@ The namespace travels in every `rag_find` or `rag_read` call; it does not come f
78
95
  - `npm test` runs the tests with Node's `node:test`.
79
96
  - `npm run typecheck` runs the TypeScript typecheck without emitting files.
80
97
 
81
- ## Known limits in 0.1.0
98
+ ## Known limits
82
99
 
83
100
  Version 0.1.0 has no embeddings or Tier 2; scanned PDFs go through OCR. Document parsing runs in a child process with a 512 MB V8 heap cap.
84
101
 
@@ -1,3 +1,5 @@
1
+ import { parseDocument } from "yaml";
2
+ import { canonicalizeStructuredContent, NonFiniteNumberCanonicalizationError } from "../store/canonical.mjs";
1
3
  class StructuredTextExtractionError extends Error {
2
4
  code;
3
5
  constructor(code) {
@@ -29,70 +31,127 @@ function supportsReference(ref, options) {
29
31
  }
30
32
  function renderJsonValue(value) {
31
33
  if (typeof value === "string") return value;
32
- return JSON.stringify(value, null, 2) ?? "null";
34
+ return JSON.stringify(value, (_key, child) => typeof child === "bigint" ? child.toString() : child, 2) ?? "null";
35
+ }
36
+ function parseJsonContent(content) {
37
+ return JSON.parse(content, (_key, value, context) => {
38
+ const source = context?.source;
39
+ if (typeof value === "number" && !Number.isSafeInteger(value) && source !== void 0 && /^-?(?:0|[1-9][0-9]*)$/.test(source)) {
40
+ return BigInt(source);
41
+ }
42
+ return value;
43
+ });
44
+ }
45
+ function canonicalFields(value) {
46
+ try {
47
+ return { canonicalContent: canonicalizeStructuredContent(value) };
48
+ } catch (error) {
49
+ if (error instanceof NonFiniteNumberCanonicalizationError) return { canonicalizationError: error.code };
50
+ throw error;
51
+ }
33
52
  }
34
53
  function extractJson(content) {
35
54
  let parsed;
36
55
  try {
37
- parsed = JSON.parse(content);
56
+ parsed = parseJsonContent(content);
38
57
  } catch {
39
58
  throw new StructuredTextExtractionError("extraction_invalid_json");
40
59
  }
41
- if (typeof parsed === "object" && parsed !== null && !Array.isArray(parsed)) {
42
- return Object.entries(parsed).map(([heading, value], ordinal) => ({
43
- ordinal,
44
- heading,
45
- level: 1,
46
- text: renderJsonValue(value)
47
- }));
48
- }
49
- return [{ ordinal: 0, heading: null, level: null, text: renderJsonValue(parsed) }];
60
+ const sections = typeof parsed === "object" && parsed !== null && !Array.isArray(parsed) ? Object.entries(parsed).map(([heading, value], ordinal) => ({
61
+ ordinal,
62
+ heading,
63
+ level: 1,
64
+ text: renderJsonValue(value)
65
+ })) : [{ ordinal: 0, heading: null, level: null, text: renderJsonValue(parsed) }];
66
+ return { sections, ...canonicalFields(parsed) };
50
67
  }
51
68
  function extractJsonl(content) {
52
69
  const sections = [];
70
+ const records = [];
53
71
  for (const line of content.split(/\r?\n/)) {
54
72
  const text = line.trim();
55
73
  if (text.length === 0) continue;
56
74
  try {
57
- JSON.parse(text);
75
+ records.push(parseJsonContent(text));
58
76
  } catch {
59
77
  throw new StructuredTextExtractionError("extraction_invalid_json");
60
78
  }
61
79
  sections.push({ ordinal: sections.length, heading: null, level: null, text });
62
80
  }
63
- return sections;
81
+ return { sections, ...canonicalFields(records) };
64
82
  }
65
- function extractYaml(content) {
66
- const lines = content.split("\n").map((line) => line.replace(/\r$/, ""));
67
- const topLevelKey = /^([A-Za-z0-9_.-]+):/;
68
- const starts = [];
69
- for (let index = 0; index < lines.length; index += 1) {
70
- const line = lines[index];
71
- if (line === void 0) continue;
72
- if (/^[ \t]*\t/.test(line)) throw new StructuredTextExtractionError("extraction_invalid_yaml");
73
- const match = topLevelKey.exec(line);
74
- if (match?.[1] !== void 0) starts.push({ index, heading: match[1] });
83
+ function normalizeYamlValue(value, ancestors = /* @__PURE__ */ new Set()) {
84
+ if (typeof value !== "object" || value === null) return value;
85
+ if (ancestors.has(value)) throw new StructuredTextExtractionError("extraction_invalid_yaml");
86
+ ancestors.add(value);
87
+ try {
88
+ if (value instanceof Map) {
89
+ const result2 = /* @__PURE__ */ Object.create(null);
90
+ for (const [key, child] of value) {
91
+ const stringKey = typeof key === "string" ? key : canonicalizeStructuredContent(normalizeYamlValue(key, ancestors));
92
+ result2[stringKey] = normalizeYamlValue(child, ancestors);
93
+ }
94
+ return result2;
95
+ }
96
+ if (Array.isArray(value)) return value.map((child) => normalizeYamlValue(child, ancestors));
97
+ const result = /* @__PURE__ */ Object.create(null);
98
+ for (const [key, child] of Object.entries(value)) result[key] = normalizeYamlValue(child, ancestors);
99
+ return result;
100
+ } finally {
101
+ ancestors.delete(value);
75
102
  }
76
- if (starts.length === 0) {
77
- return [{ ordinal: 0, heading: null, level: null, text: content }];
103
+ }
104
+ function renderYamlValue(value) {
105
+ if (typeof value === "string") return value;
106
+ return JSON.stringify(value, (_key, child) => typeof child === "bigint" ? child.toString() : child, 2) ?? "null";
107
+ }
108
+ function yamlTopLevelStarts(document) {
109
+ const root = document.contents;
110
+ const starts = /* @__PURE__ */ new Map();
111
+ for (const pair of root?.items ?? []) {
112
+ const position = pair.key?.range?.[0];
113
+ if (position === void 0) continue;
114
+ const rawKey = pair.key && Object.hasOwn(pair.key, "value") ? pair.key.value : pair.key?.toJSON?.();
115
+ const key = normalizeYamlValue(rawKey);
116
+ const stringKey = typeof key === "string" ? key : canonicalizeStructuredContent(key);
117
+ starts.set(stringKey, position);
78
118
  }
79
- let hasCurrentBlock = false;
80
- for (const line of lines) {
81
- const trimmed = line.trim();
82
- if (trimmed.length === 0 || trimmed.startsWith("#") || trimmed === "---" || trimmed === "...") continue;
83
- const match = topLevelKey.exec(line);
84
- if (match?.[1] !== void 0) {
85
- hasCurrentBlock = true;
86
- continue;
119
+ return starts;
120
+ }
121
+ function extractYaml(content) {
122
+ let parsed;
123
+ let topLevelStarts;
124
+ let canonicalizationError;
125
+ try {
126
+ const document = parseDocument(content, { schema: "core", uniqueKeys: false, intAsBigInt: true });
127
+ if (document.errors.length > 0) throw new Error("invalid YAML document");
128
+ parsed = normalizeYamlValue(document.toJS({ mapAsMap: true }));
129
+ topLevelStarts = yamlTopLevelStarts(document);
130
+ } catch (error) {
131
+ if (!(error instanceof NonFiniteNumberCanonicalizationError)) {
132
+ throw new StructuredTextExtractionError("extraction_invalid_yaml");
87
133
  }
88
- const indentation = line.match(/^ */)?.[0].length ?? 0;
89
- if (indentation === 0 || !hasCurrentBlock) throw new StructuredTextExtractionError("extraction_invalid_yaml");
134
+ canonicalizationError = error.code;
135
+ parsed = null;
136
+ topLevelStarts = /* @__PURE__ */ new Map();
90
137
  }
91
- return starts.map((start, ordinal) => {
92
- const nextStart = starts[ordinal + 1];
93
- const block = lines.slice(start.index, nextStart?.index ?? lines.length).join("\n").trim();
94
- return { ordinal, heading: start.heading, level: 1, text: block };
95
- });
138
+ if (canonicalizationError !== void 0) {
139
+ return { sections: [{ ordinal: 0, heading: null, level: null, text: content.trim() }], canonicalizationError };
140
+ }
141
+ let sections;
142
+ if (typeof parsed === "object" && parsed !== null && !Array.isArray(parsed)) {
143
+ const entries = Object.entries(parsed).map(([heading, value]) => ({ heading, value, start: topLevelStarts.get(heading) })).filter((entry) => entry.start !== void 0).sort((left, right) => left.start - right.start);
144
+ sections = entries.map(({ heading, start }, ordinal) => ({
145
+ ordinal,
146
+ heading,
147
+ level: 1,
148
+ text: content.slice(start, entries[ordinal + 1]?.start ?? content.length).trim()
149
+ }));
150
+ if (sections.length === 0) sections = [{ ordinal: 0, heading: null, level: null, text: renderYamlValue(parsed) }];
151
+ } else {
152
+ sections = [{ ordinal: 0, heading: null, level: null, text: renderYamlValue(parsed) }];
153
+ }
154
+ return { sections, ...canonicalFields(parsed) };
96
155
  }
97
156
  function createStructuredTextExtractor(options) {
98
157
  return {
@@ -108,19 +167,26 @@ function createStructuredTextExtractor(options) {
108
167
  if (content.length === 0) {
109
168
  return { sections: [], versionLabel: null, mediaType: null };
110
169
  }
111
- let sections;
170
+ let extracted;
112
171
  switch (options.kind) {
113
172
  case "json":
114
- sections = extractJson(content);
173
+ extracted = extractJson(content);
115
174
  break;
116
175
  case "jsonl":
117
- sections = extractJsonl(content);
176
+ extracted = extractJsonl(content);
118
177
  break;
119
178
  case "yaml":
120
- sections = extractYaml(content);
179
+ extracted = extractYaml(content);
121
180
  break;
122
181
  }
123
- return { sections, versionLabel: null, mediaType: options.mediaTypes[0] ?? null };
182
+ const result = {
183
+ sections: extracted.sections,
184
+ versionLabel: null,
185
+ mediaType: options.mediaTypes[0] ?? null,
186
+ ...extracted.canonicalContent === void 0 ? {} : { canonicalContent: extracted.canonicalContent },
187
+ ...extracted.canonicalizationError === void 0 ? {} : { canonicalizationError: extracted.canonicalizationError }
188
+ };
189
+ return result;
124
190
  }
125
191
  };
126
192
  }
@@ -0,0 +1,112 @@
1
+ const ANSI_ESCAPE = /\u001B(?:\[[0-?]*[ -/]*[@-~]|\][^\u0007]*(?:\u0007|\u001B\\))/g;
2
+ function record(value) {
3
+ return value !== null && typeof value === "object" && !Array.isArray(value) ? value : null;
4
+ }
5
+ function oneLine(value) {
6
+ return String(value ?? "").replace(/[\r\n]+/g, " ");
7
+ }
8
+ function styled(value, code, enabled) {
9
+ return enabled ? `\x1B[${code}m${value}\x1B[0m` : value;
10
+ }
11
+ function resolveOutputFormat(env, isTTY) {
12
+ const requested = env.RAG_FORMAT?.trim().toLowerCase();
13
+ if (requested === "json" || requested === "pretty") return requested;
14
+ return isTTY ? "pretty" : "json";
15
+ }
16
+ function useColor(env) {
17
+ return !Object.hasOwn(env, "NO_COLOR");
18
+ }
19
+ function stripAnsi(value) {
20
+ return value.replace(ANSI_ESCAPE, "");
21
+ }
22
+ function padVisible(value, width) {
23
+ return value.padEnd(width + value.length - stripAnsi(value).length);
24
+ }
25
+ function renderDocuments(documents, color) {
26
+ if (documents.length === 0) return "No documents.";
27
+ const columns = [
28
+ { header: "DOCUMENT ID", value: (row) => row.documentId },
29
+ { header: "ENTITY", value: (row) => row.entityKey },
30
+ { header: "LABEL", value: (row) => row.label },
31
+ { header: "VERSION", value: (row) => row.versionLabel }
32
+ ];
33
+ const rows = documents.map((item) => record(item) ?? {});
34
+ const widths = columns.map((column) => Math.max(
35
+ column.header.length,
36
+ ...rows.map((row) => stripAnsi(oneLine(column.value(row))).length)
37
+ ));
38
+ const header = columns.map((column, index) => padVisible(
39
+ styled(column.header, "1", color),
40
+ widths[index]
41
+ )).join(" ").trimEnd();
42
+ const body = rows.map((row) => columns.map((column, index) => padVisible(
43
+ oneLine(column.value(row)),
44
+ widths[index]
45
+ )).join(" ").trimEnd());
46
+ return [header, ...body].join("\n");
47
+ }
48
+ function renderDoctor(result) {
49
+ const checked = Number(result.checked ?? 0);
50
+ if (result.ok === true) return `Doctor: healthy (${checked} objects checked)`;
51
+ const missing = Array.isArray(result.missing) ? result.missing.length : 0;
52
+ const corrupt = Array.isArray(result.corrupt) ? result.corrupt.length : 0;
53
+ const details = [`${missing} missing`, `${corrupt} corrupt`];
54
+ if (typeof result.chunks === "number" && typeof result.indexedChunks === "number" && result.chunks !== result.indexedChunks) {
55
+ details.push(`index ${result.indexedChunks}/${result.chunks}`);
56
+ }
57
+ if (typeof result.unreferencedObjects === "number" && result.unreferencedObjects > 0) {
58
+ details.push(`${result.unreferencedObjects} orphan objects`);
59
+ }
60
+ if (typeof result.leftoverTemporaries === "number" && result.leftoverTemporaries > 0) {
61
+ details.push(`${result.leftoverTemporaries} leftover temporaries`);
62
+ }
63
+ return `Doctor: issues found (${checked} checked; ${details.join(", ")})`;
64
+ }
65
+ function renderImport(result) {
66
+ const summary = record(result.summary) ?? {};
67
+ const count = (key) => typeof summary[key] === "number" ? summary[key] : 0;
68
+ return `Import: ${count("inserted")} inserted, ${count("unchanged")} unchanged, ${count("skipped")} skipped, ${count("failed")} failed (${count("total")} total)`;
69
+ }
70
+ function renderDocumentAction(result) {
71
+ const status = oneLine(result.status);
72
+ const documentId = oneLine(result.documentId);
73
+ if (status === "relabeled") return `Relabeled document ${documentId}: ${oneLine(result.label)}`;
74
+ if (status === "retired") return `Retired document ${documentId}`;
75
+ const action = status === "inserted" ? "Added" : status === "unchanged" ? "Unchanged" : "Updated";
76
+ const label = typeof result.label === "string" ? ` ${oneLine(result.label)}` : " document";
77
+ return `${action}${label} (${status}; documentId: ${documentId})`;
78
+ }
79
+ function renderError(error) {
80
+ const code = oneLine(error.code || "error");
81
+ return `Error [${code}]: ${oneLine(error.message || "Unknown error")}`;
82
+ }
83
+ function renderOutput(output, env) {
84
+ const envelope = record(output);
85
+ if (!envelope) return JSON.stringify(output) ?? "null";
86
+ const error = record(envelope.error);
87
+ if (error) return renderError(error);
88
+ const result = record(envelope.result);
89
+ if (!result) return JSON.stringify(output) ?? "null";
90
+ const color = useColor(env);
91
+ if (Array.isArray(result.documents)) return renderDocuments(result.documents, color);
92
+ if (typeof result.documents === "number" && typeof result.versions === "number" && typeof result.objects === "number") {
93
+ return `Documents: ${result.documents} | Versions: ${result.versions} | Objects: ${result.objects}`;
94
+ }
95
+ if (typeof result.ok === "boolean" && typeof result.checked === "number") return renderDoctor(result);
96
+ if (typeof result.updated === "number" && typeof result.unchanged === "number" && typeof result.skipped === "number") {
97
+ const label = result.applied === true ? "Rehash applied" : "Rehash preview";
98
+ const summary = `${label}: ${result.updated} updated, ${result.unchanged} unchanged, ${result.skipped} skipped`;
99
+ return typeof result.backupReminder === "string" ? `${summary}
100
+ ${oneLine(result.backupReminder)}` : summary;
101
+ }
102
+ if (Array.isArray(result.files) && record(result.summary)) return renderImport(result);
103
+ if (typeof result.status === "string" && typeof result.documentId === "string") return renderDocumentAction(result);
104
+ return JSON.stringify(output) ?? "null";
105
+ }
106
+ export {
107
+ padVisible,
108
+ renderOutput,
109
+ resolveOutputFormat,
110
+ stripAnsi,
111
+ useColor
112
+ };
@@ -10,7 +10,9 @@ import { importDirectory } from "../ingest/import.mjs";
10
10
  import { clientDbPath, openDatabase, validateClientName } from "../store/db.mjs";
11
11
  import { backupRoot, reindexRoot, restoreRoot, writeRootManifest } from "../store/backup.mjs";
12
12
  import { readObject } from "../store/objects.mjs";
13
+ import { rehashNamespace } from "../store/rehash.mjs";
13
14
  import { getActiveLimits } from "../limits.mjs";
15
+ import { renderOutput, resolveOutputFormat } from "./format.mjs";
14
16
  import {
15
17
  linkReviewVersions,
16
18
  listReviewCandidates,
@@ -36,6 +38,7 @@ Commands:
36
38
  backup --to <dir> Back up the manifest and object store
37
39
  restore --from <dir> Restore a backup into this root
38
40
  reindex Rebuild the database from manifest and objects
41
+ rehash [--apply] Recompute stored-version content hashes (preview by default)
39
42
 
40
43
  Options:
41
44
  --root <path> Knowledge-base root
@@ -45,7 +48,20 @@ Options:
45
48
  --from <dir> Backup source
46
49
  --type <type> Human relation type for rag review link
47
50
  --label <text> Replacement label for rag relabel
48
- --help Show this help`;
51
+ --apply Persist the rag rehash results
52
+ --help Show this help
53
+
54
+ Content identity:
55
+ JSON, JSONL, and YAML use canonical tree hashes (content_kind=structured, hash_version=2).
56
+ Other formats keep normalized-text hashes (content_kind=NULL, hash_version=1).`;
57
+ const HELP = `${USAGE}
58
+
59
+ Output format:
60
+ Interactive terminals render human-readable output; pipes use JSON.
61
+ RAG_FORMAT=json|pretty overrides auto-detection; NO_COLOR disables ANSI styling.
62
+ --help defaults to raw usage on a terminal; RAG_FORMAT=json restores its JSON envelope.
63
+
64
+ Use the documentId shown by rag ls with rag relabel <document-id> --label <text>.`;
49
65
  const NAMED_ERROR_CODES = /* @__PURE__ */ new Set([
50
66
  "usage",
51
67
  "ingest_failed",
@@ -80,6 +96,7 @@ const NAMED_ERROR_CODES = /* @__PURE__ */ new Set([
80
96
  "backup_failed",
81
97
  "destination_exists",
82
98
  "namespace_mismatch",
99
+ "rehash_failed",
83
100
  "restore_destination_not_empty",
84
101
  "restore_failed",
85
102
  "foreign_client_in_manifest",
@@ -150,12 +167,17 @@ function parseArgs(argv) {
150
167
  const positional = [];
151
168
  const values = {};
152
169
  let help = false;
170
+ let apply = false;
153
171
  for (let index = 0; index < argv.length; index += 1) {
154
172
  const argument = argv[index];
155
173
  if (argument === "--help") {
156
174
  help = true;
157
175
  continue;
158
176
  }
177
+ if (argument === "--apply") {
178
+ apply = true;
179
+ continue;
180
+ }
159
181
  const assignment = /^--(to|from|type|label)=(.*)$/.exec(argument);
160
182
  if (assignment) {
161
183
  values[assignment[1]] = assignment[2];
@@ -182,7 +204,7 @@ function parseArgs(argv) {
182
204
  if (argument.startsWith("--")) throw usageError(`Unknown option: ${argument}`);
183
205
  positional.push(argument);
184
206
  }
185
- return { command: positional[0] ?? null, positional: positional.slice(1), help, values };
207
+ return { command: positional[0] ?? null, positional: positional.slice(1), help, values, apply };
186
208
  }
187
209
  function requireNamespace(config, mode) {
188
210
  const { namespace } = resolveCommandNamespace(config, mode);
@@ -223,11 +245,16 @@ function queryDocuments(root, namespace) {
223
245
  try {
224
246
  const project = db.prepare(`SELECT p.project_id FROM project AS p
225
247
  JOIN client AS c ON c.client_id = p.client_id WHERE c.name = ? AND p.slug = ?`).get(namespace.client, namespace.project);
226
- const rows = project ? db.prepare(`SELECT e.entity_key AS entityKey, d.label AS label, dv.version_label AS versionLabel
248
+ const rows = project ? db.prepare(`SELECT d.document_id AS documentId, e.entity_key AS entityKey, d.label AS label, dv.version_label AS versionLabel
227
249
  FROM document AS d JOIN entity AS e ON e.entity_id = d.entity_id
228
250
  JOIN doc_version AS dv ON dv.document_id = d.document_id AND dv.is_current = 1
229
251
  WHERE d.project_id = ? AND d.deleted_at IS NULL ORDER BY e.entity_key, d.label`).all(project.project_id) : [];
230
- const documents = rows.map((row) => ({ entityKey: row.entityKey, label: row.label, versionLabel: row.versionLabel }));
252
+ const documents = rows.map((row) => ({
253
+ documentId: row.documentId,
254
+ entityKey: row.entityKey,
255
+ label: row.label,
256
+ versionLabel: row.versionLabel
257
+ }));
231
258
  return { namespace, result: { documents } };
232
259
  } finally {
233
260
  db.close();
@@ -385,6 +412,9 @@ async function runCli(options) {
385
412
  if (parsed.values.label !== void 0 && parsed.command !== "relabel") {
386
413
  return errorOutput(config.namespace, usageError('Option --label is only accepted by "rag relabel".'));
387
414
  }
415
+ if (parsed.apply && parsed.command !== "rehash") {
416
+ return errorOutput(config.namespace, usageError('Option --apply is only accepted by "rag rehash".'));
417
+ }
388
418
  if (parsed.command === "relabel" && parsed.values.label === void 0) {
389
419
  return errorOutput(config.namespace, usageError("Option --label requires a non-blank value.", "invalid_label"));
390
420
  }
@@ -643,13 +673,26 @@ async function runCli(options) {
643
673
  return commandErrorOutput(config.namespace, error);
644
674
  }
645
675
  }
676
+ if (parsed.command === "rehash") {
677
+ if (parsed.positional.length !== 0) return commandErrorOutput(config.namespace, usageError("rehash does not accept positional arguments."));
678
+ try {
679
+ const namespace = requireNamespace(config, parsed.apply ? "write" : "read");
680
+ if (!namespace) throw new NamespaceRequiredError();
681
+ const result = await rehashNamespace(path.resolve(options.cwd, config.root), namespace, parsed.apply);
682
+ return { exitCode: 0, output: { namespace, result } };
683
+ } catch (error) {
684
+ return commandErrorOutput(config.namespace, error);
685
+ }
686
+ }
646
687
  return errorOutput(config.namespace, usageError(parsed.command === null ? "Command is required." : `Unknown command: ${parsed.command}`));
647
688
  }
648
689
  async function main(dependencies = {}) {
649
690
  const writeStdout = dependencies.writeStdout ?? ((line) => process.stdout.write(line));
691
+ const writeStderr = dependencies.writeStderr ?? ((line) => process.stderr.write(line));
650
692
  const setExitCode = dependencies.setExitCode ?? ((exitCode) => {
651
693
  process.exitCode = exitCode;
652
694
  });
695
+ const format = resolveOutputFormat(process.env, process.stdout.isTTY === true);
653
696
  try {
654
697
  const result = await (dependencies.runCli ?? runCli)({
655
698
  argv: process.argv.slice(2),
@@ -657,12 +700,25 @@ async function main(dependencies = {}) {
657
700
  cwd: process.cwd(),
658
701
  homeDir: process.env.RAG_HOME ?? homedir()
659
702
  });
660
- writeStdout(`${JSON.stringify(result.output)}
703
+ const output = result.output;
704
+ if (format === "pretty" && result.exitCode === 0 && typeof output === "object" && output !== null && "result" in output && output.result === USAGE) {
705
+ writeStdout(`${HELP}
706
+ `);
707
+ } else if (format === "pretty") {
708
+ const rendered = `${renderOutput(output, process.env)}
709
+ `;
710
+ if (typeof output === "object" && output !== null && "error" in output) writeStderr(rendered);
711
+ else writeStdout(rendered);
712
+ } else {
713
+ writeStdout(`${JSON.stringify(output)}
661
714
  `);
715
+ }
662
716
  setExitCode(result.exitCode);
663
717
  } catch (error) {
664
718
  const message = publicErrorMessage(error);
665
- writeStdout(`${JSON.stringify({ namespace: null, error: { code: "internal_error", message } })}
719
+ if (format === "pretty") writeStderr(`${renderOutput({ error: { code: "internal_error", message } }, process.env)}
720
+ `);
721
+ else writeStdout(`${JSON.stringify({ namespace: null, error: { code: "internal_error", message } })}
666
722
  `);
667
723
  setExitCode(1);
668
724
  }
@@ -671,6 +727,7 @@ if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.me
671
727
  await main();
672
728
  }
673
729
  export {
730
+ HELP,
674
731
  USAGE,
675
732
  main,
676
733
  runCli
@@ -6,7 +6,8 @@ import { ResourceLimitError } from "../ports/extractor.mjs";
6
6
  import { clientDbPath, openDatabase } from "../store/db.mjs";
7
7
  import { writeRootManifest } from "../store/backup.mjs";
8
8
  import { deriveTier0Relations, normalizeEntityKey } from "../store/identity.mjs";
9
- import { fingerprintDistance, fingerprintNormalizedText, NEAR_DUPLICATE_MAX_DISTANCE } from "../store/similarity.mjs";
9
+ import { computeContentHashFacts } from "../store/rehash.mjs";
10
+ import { fingerprintDistance, NEAR_DUPLICATE_MAX_DISTANCE } from "../store/similarity.mjs";
10
11
  import { putObjectFromPath, readObject } from "../store/objects.mjs";
11
12
  import { MAX_SOURCE_FILE_BYTES as SHARED_MAX_SOURCE_FILE_BYTES } from "../limits.mjs";
12
13
  import { parseDocument, ParseHostError } from "./parse-host.mjs";
@@ -86,9 +87,8 @@ async function ingestFile(input) {
86
87
  recordFailedRun(db, projectId, input.sourcePath, sha256, runStartedAt, error.message, error.code);
87
88
  throw error;
88
89
  }
89
- const normalizedText = extracted.sections.map((section) => section.text).join("\n").replace(/\s+/g, " ").trim().toLowerCase();
90
- const contentHash = createHash("sha256").update(normalizedText).digest("hex");
91
- const normalizedHash = fingerprintNormalizedText(normalizedText);
90
+ const hashFacts = computeContentHashFacts(extracted);
91
+ const { contentHash, normalizedHash, contentKind, hashVersion } = hashFacts;
92
92
  const sourceLabel = path.basename(input.sourcePath);
93
93
  const entityKey = normalizeEntityKey(input.entityKey ?? path.parse(sourceLabel).name);
94
94
  const label = input.label ?? sourceLabel;
@@ -125,8 +125,8 @@ async function ingestFile(input) {
125
125
  db.prepare(`INSERT INTO ingest_run (run_id, project_id, started_at, status, source_uri, sha256)
126
126
  VALUES (?, ?, ?, 'running', ?, ?)`).run(runId, projectId, now, input.sourcePath, stored.sha256);
127
127
  db.prepare("UPDATE doc_version SET is_current = 0, superseded_at = ? WHERE document_id = ? AND is_current = 1").run(now, documentId);
128
- db.prepare(`INSERT INTO doc_version (doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, extractor, extractor_version,
129
- ingest_run_id, is_current, ingested_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 1, ?)`).run(docVersionId, documentId, stored.sha256, extracted.versionLabel, contentHash, normalizedHash, extractorName, extractorVersion, runId, now);
128
+ db.prepare(`INSERT INTO doc_version (doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, content_kind, hash_version, extractor, extractor_version,
129
+ ingest_run_id, is_current, ingested_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1, ?)`).run(docVersionId, documentId, stored.sha256, extracted.versionLabel, contentHash, normalizedHash, contentKind, hashVersion, extractorName, extractorVersion, runId, now);
130
130
  let offset = 0;
131
131
  extracted.sections.forEach((section, ordinal) => {
132
132
  const sectionId = randomUUID();
@@ -146,7 +146,8 @@ async function ingestFile(input) {
146
146
  contentHash,
147
147
  entityKey,
148
148
  projectId,
149
- versionLabel: extracted.versionLabel
149
+ versionLabel: extracted.versionLabel,
150
+ hashVersion
150
151
  };
151
152
  const insertRelation = db.prepare(`INSERT INTO doc_relation
152
153
  (relation_id, from_doc_version_id, to_doc_version_id, relation_type, score, decided_by, decided_at, evidence)
@@ -154,7 +155,7 @@ async function ingestFile(input) {
154
155
  const currentDocument = db.prepare("SELECT deleted_at FROM document WHERE document_id = ?").get(documentId);
155
156
  if (currentDocument?.deleted_at === null) {
156
157
  const priorVersions = db.prepare(`SELECT dv.doc_version_id AS docVersionId, dv.sha256,
157
- dv.content_hash AS contentHash, dv.normalized_hash AS normalizedHash,
158
+ dv.content_hash AS contentHash, dv.normalized_hash AS normalizedHash, dv.hash_version AS hashVersion,
158
159
  e.entity_key AS entityKey, d.project_id AS projectId, dv.version_label AS versionLabel
159
160
  FROM doc_version AS dv
160
161
  JOIN document AS d ON d.document_id = dv.document_id
@@ -173,7 +174,7 @@ async function ingestFile(input) {
173
174
  JSON.stringify(relation.evidence)
174
175
  );
175
176
  }
176
- if (prior.normalizedHash !== null) {
177
+ if (prior.hashVersion === hashVersion && prior.normalizedHash !== null) {
177
178
  const distance = fingerprintDistance(normalizedHash, prior.normalizedHash);
178
179
  if (distance <= NEAR_DUPLICATE_MAX_DISTANCE) {
179
180
  const relationId = `rel_${createHash("sha256").update(JSON.stringify([docVersionId, prior.docVersionId, "near_duplicate"])).digest("hex")}`;
@@ -0,0 +1,65 @@
1
+ const TEXT_HASH_VERSION = 1;
2
+ const STRUCTURED_HASH_VERSION = 2;
3
+ class NonFiniteNumberCanonicalizationError extends Error {
4
+ code = "non_finite_number";
5
+ constructor() {
6
+ super("Canonical structured content contains a non-finite number.");
7
+ this.name = "NonFiniteNumberCanonicalizationError";
8
+ }
9
+ }
10
+ function compareCodePoints(left, right) {
11
+ const leftPoints = Array.from(left, (character) => character.codePointAt(0));
12
+ const rightPoints = Array.from(right, (character) => character.codePointAt(0));
13
+ const sharedLength = Math.min(leftPoints.length, rightPoints.length);
14
+ for (let index = 0; index < sharedLength; index += 1) {
15
+ const leftPoint = leftPoints[index];
16
+ const rightPoint = rightPoints[index];
17
+ if (leftPoint !== rightPoint) return leftPoint - rightPoint;
18
+ }
19
+ return leftPoints.length - rightPoints.length;
20
+ }
21
+ function encode(value, ancestors) {
22
+ if (value === null) return "null";
23
+ if (typeof value === "string") return JSON.stringify(value);
24
+ if (typeof value === "boolean") return value ? "true" : "false";
25
+ if (typeof value === "number") {
26
+ if (!Number.isFinite(value)) throw new NonFiniteNumberCanonicalizationError();
27
+ return JSON.stringify(value);
28
+ }
29
+ if (typeof value === "bigint") {
30
+ if (value <= BigInt(Number.MAX_SAFE_INTEGER) && value >= BigInt(Number.MIN_SAFE_INTEGER)) {
31
+ return JSON.stringify(Number(value));
32
+ }
33
+ return JSON.stringify(value.toString());
34
+ }
35
+ if (Array.isArray(value)) {
36
+ if (ancestors.has(value)) throw new TypeError("Structured content must be acyclic.");
37
+ ancestors.add(value);
38
+ try {
39
+ return `[${value.map((item) => item === void 0 ? "null" : encode(item, ancestors)).join(",")}]`;
40
+ } finally {
41
+ ancestors.delete(value);
42
+ }
43
+ }
44
+ if (typeof value === "object") {
45
+ const object = value;
46
+ if (ancestors.has(object)) throw new TypeError("Structured content must be acyclic.");
47
+ ancestors.add(object);
48
+ try {
49
+ const members = Object.keys(object).filter((key) => object[key] !== void 0).sort(compareCodePoints).map((key) => `${JSON.stringify(key)}:${encode(object[key], ancestors)}`);
50
+ return `{${members.join(",")}}`;
51
+ } finally {
52
+ ancestors.delete(object);
53
+ }
54
+ }
55
+ return "null";
56
+ }
57
+ function canonicalizeStructuredContent(value) {
58
+ return encode(value, /* @__PURE__ */ new Set());
59
+ }
60
+ export {
61
+ NonFiniteNumberCanonicalizationError,
62
+ STRUCTURED_HASH_VERSION,
63
+ TEXT_HASH_VERSION,
64
+ canonicalizeStructuredContent
65
+ };
@@ -57,7 +57,7 @@ function deriveTier0Relations(a, b) {
57
57
  const relations = [];
58
58
  if (a.sha256 === b.sha256) {
59
59
  relations.push(makeRelation(a.docVersionId, b.docVersionId, "same_bytes", { sha256: a.sha256 }));
60
- } else if (a.contentHash !== null && b.contentHash !== null && a.contentHash === b.contentHash) {
60
+ } else if (a.contentHash !== null && b.contentHash !== null && a.contentHash === b.contentHash && (a.hashVersion ?? 1) === (b.hashVersion ?? 1)) {
61
61
  relations.push(makeRelation(a.docVersionId, b.docVersionId, "same_content", { contentHash: a.contentHash }));
62
62
  }
63
63
  const sameEntity = normalizeEntityKey(a.entityKey) === normalizeEntityKey(b.entityKey);
@@ -64,6 +64,7 @@ function listReviewCandidates(db, projectId) {
64
64
  JOIN entity AS right_entity ON right_entity.entity_id = right_document.entity_id
65
65
  WHERE left_document.project_id = ? AND right_document.project_id = left_document.project_id
66
66
  AND left_version.content_hash IS NOT NULL AND left_version.content_hash = right_version.content_hash
67
+ AND left_version.hash_version = right_version.hash_version
67
68
  AND left_document.deleted_at IS NULL AND right_document.deleted_at IS NULL
68
69
  AND left_entity.deleted_at IS NULL AND right_entity.deleted_at IS NULL
69
70
  AND NOT EXISTS (
@@ -86,6 +87,7 @@ function listReviewCandidates(db, projectId) {
86
87
  JOIN entity AS to_entity ON to_entity.entity_id = to_document.entity_id
87
88
  WHERE near_relation.relation_type = 'near_duplicate' AND near_relation.decided_by <> 'human'
88
89
  AND from_document.project_id = ? AND to_document.project_id = from_document.project_id
90
+ AND from_version.hash_version = to_version.hash_version
89
91
  AND from_document.deleted_at IS NULL AND to_document.deleted_at IS NULL
90
92
  AND from_entity.deleted_at IS NULL AND to_entity.deleted_at IS NULL
91
93
  AND NOT EXISTS (
@@ -21,7 +21,7 @@ function buildManifest(db) {
21
21
  ingestRuns: orderedRows(db, "ingest_run"),
22
22
  entities: db.prepare("SELECT entity_id, project_id, entity_key, kind, title, created_at, updated_at, deleted_at FROM entity ORDER BY entity_id").all(),
23
23
  documents: db.prepare("SELECT document_id, entity_id, project_id, label, created_at, updated_at, deleted_at FROM document ORDER BY document_id").all(),
24
- versions: db.prepare("SELECT doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, extractor, extractor_version, ingest_run_id, is_current, ingested_at, superseded_at FROM doc_version ORDER BY doc_version_id").all(),
24
+ versions: db.prepare("SELECT doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, content_kind, hash_version, extractor, extractor_version, ingest_run_id, is_current, ingested_at, superseded_at FROM doc_version ORDER BY doc_version_id").all(),
25
25
  sections: db.prepare("SELECT section_id, doc_version_id, ordinal, heading, level, char_start, char_end FROM section ORDER BY doc_version_id, ordinal, section_id").all(),
26
26
  chunks: db.prepare("SELECT chunk_id, doc_version_id, section_id, ordinal, text, char_start, char_end, token_count FROM chunk ORDER BY doc_version_id, ordinal, chunk_id").all(),
27
27
  relations: db.prepare("SELECT relation_id, from_doc_version_id, to_doc_version_id, relation_type, score, decided_by, decided_at, evidence FROM doc_relation ORDER BY relation_id").all(),
@@ -38,7 +38,7 @@ const ROW_COLUMNS = {
38
38
  ingestRuns: ["run_id", "project_id", "started_at", "finished_at", "status", "source_uri", "sha256", "result_code", "detail"],
39
39
  entities: ["entity_id", "project_id", "entity_key", "kind", "title", "created_at", "updated_at", "deleted_at"],
40
40
  documents: ["document_id", "entity_id", "project_id", "label", "created_at", "updated_at", "deleted_at"],
41
- versions: ["doc_version_id", "document_id", "sha256", "version_label", "content_hash", "normalized_hash", "extractor", "extractor_version", "ingest_run_id", "is_current", "ingested_at", "superseded_at"],
41
+ versions: ["doc_version_id", "document_id", "sha256", "version_label", "content_hash", "normalized_hash", "content_kind", "hash_version", "extractor", "extractor_version", "ingest_run_id", "is_current", "ingested_at", "superseded_at"],
42
42
  sections: ["section_id", "doc_version_id", "ordinal", "heading", "level", "char_start", "char_end"],
43
43
  chunks: ["chunk_id", "doc_version_id", "section_id", "ordinal", "text", "char_start", "char_end", "token_count"],
44
44
  relations: ["relation_id", "from_doc_version_id", "to_doc_version_id", "relation_type", "score", "decided_by", "decided_at", "evidence"],
@@ -47,8 +47,12 @@ const ROW_COLUMNS = {
47
47
  };
48
48
  function validateRowKeys(value, field) {
49
49
  const expected = ROW_COLUMNS[field];
50
+ const legacyVersionColumns = field === "versions" ? expected.filter((column) => column !== "content_kind" && column !== "hash_version") : null;
50
51
  for (const row of value) {
51
- if (!isRecord(row) || Object.keys(row).length !== expected.length || expected.some((column) => !Object.hasOwn(row, column))) {
52
+ if (!isRecord(row)) throw new ManifestError("manifest_invalid", `Manifest ${field} row keys must match expected columns: ${expected.join(", ")}.`);
53
+ const keys = Object.keys(row);
54
+ const matches = (columns) => keys.length === columns.length && columns.every((column) => Object.hasOwn(row, column));
55
+ if (!matches(expected) && !(legacyVersionColumns !== null && matches(legacyVersionColumns))) {
52
56
  throw new ManifestError("manifest_invalid", `Manifest ${field} row keys must match expected columns: ${expected.join(", ")}.`);
53
57
  }
54
58
  }
@@ -99,7 +103,12 @@ function importManifest(db, value) {
99
103
  insertRows(db, "INSERT INTO entity (entity_id, project_id, entity_key, kind, title, created_at, updated_at, deleted_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", value.entities, (r) => [r.entity_id, r.project_id, r.entity_key, r.kind, r.title, r.created_at, r.updated_at, r.deleted_at]);
100
104
  insertRows(db, "INSERT INTO document (document_id, entity_id, project_id, label, created_at, updated_at, deleted_at) VALUES (?, ?, ?, ?, ?, ?, ?)", value.documents, (r) => [r.document_id, r.entity_id, r.project_id, r.label, r.created_at, r.updated_at, r.deleted_at]);
101
105
  insertRows(db, "INSERT INTO object (sha256, byte_size, rel_path, media_type, created_at, last_verified_at) VALUES (?, ?, ?, ?, ?, ?)", value.objects, (r) => [r.sha256, r.byte_size, r.rel_path, r.media_type, r.created_at, r.last_verified_at]);
102
- insertRows(db, "INSERT INTO doc_version (doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, extractor, extractor_version, ingest_run_id, is_current, ingested_at, superseded_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", value.versions, (r) => [r.doc_version_id, r.document_id, r.sha256, r.version_label, r.content_hash, r.normalized_hash, r.extractor, r.extractor_version, r.ingest_run_id, r.is_current, r.ingested_at, r.superseded_at]);
106
+ const versions = value.versions.map((version) => ({
107
+ ...version,
108
+ content_kind: version.content_kind ?? null,
109
+ hash_version: version.hash_version ?? 1
110
+ }));
111
+ insertRows(db, "INSERT INTO doc_version (doc_version_id, document_id, sha256, version_label, content_hash, normalized_hash, content_kind, hash_version, extractor, extractor_version, ingest_run_id, is_current, ingested_at, superseded_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", versions, (r) => [r.doc_version_id, r.document_id, r.sha256, r.version_label, r.content_hash, r.normalized_hash, r.content_kind, r.hash_version, r.extractor, r.extractor_version, r.ingest_run_id, r.is_current, r.ingested_at, r.superseded_at]);
103
112
  insertRows(db, "INSERT INTO section (section_id, doc_version_id, ordinal, heading, level, char_start, char_end) VALUES (?, ?, ?, ?, ?, ?, ?)", value.sections, (r) => [r.section_id, r.doc_version_id, r.ordinal, r.heading, r.level, r.char_start, r.char_end]);
104
113
  insertRows(db, "INSERT INTO chunk (chunk_id, doc_version_id, section_id, ordinal, text, char_start, char_end, token_count) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", value.chunks, (r) => [r.chunk_id, r.doc_version_id, r.section_id, r.ordinal, r.text, r.char_start, r.char_end, r.token_count]);
105
114
  db.exec("INSERT INTO chunk_fts(chunk_fts) VALUES ('rebuild');");
@@ -0,0 +1,162 @@
1
+ import { createHash } from "node:crypto";
2
+ import * as fs from "node:fs/promises";
3
+ import * as path from "node:path";
4
+ import { NonFiniteNumberCanonicalizationError, STRUCTURED_HASH_VERSION, TEXT_HASH_VERSION } from "./canonical.mjs";
5
+ import { clientDbPath, openDatabase } from "./db.mjs";
6
+ import { writeRootManifest } from "./backup.mjs";
7
+ import { readObject } from "./objects.mjs";
8
+ import { fingerprintNormalizedText } from "./similarity.mjs";
9
+ import { parseDocument } from "../ingest/parse-host.mjs";
10
+ function computeContentHashFacts(extracted) {
11
+ const normalizedText = extracted.sections.map((section) => section.text).join("\n").replace(/\s+/g, " ").trim().toLowerCase();
12
+ let contentKind;
13
+ let hashVersion;
14
+ let hashInput;
15
+ let skipped = false;
16
+ try {
17
+ if (extracted.canonicalizationError === "non_finite_number") throw new NonFiniteNumberCanonicalizationError();
18
+ contentKind = extracted.canonicalContent === void 0 ? null : "structured";
19
+ hashVersion = extracted.canonicalContent === void 0 ? TEXT_HASH_VERSION : STRUCTURED_HASH_VERSION;
20
+ hashInput = extracted.canonicalContent ?? normalizedText;
21
+ } catch (error) {
22
+ if (!(error instanceof NonFiniteNumberCanonicalizationError)) throw error;
23
+ contentKind = null;
24
+ hashVersion = TEXT_HASH_VERSION;
25
+ hashInput = normalizedText;
26
+ skipped = true;
27
+ }
28
+ return {
29
+ contentHash: createHash("sha256").update(hashInput).digest("hex"),
30
+ normalizedHash: fingerprintNormalizedText(hashInput),
31
+ contentKind,
32
+ hashVersion,
33
+ skipped
34
+ };
35
+ }
36
+ function errorWithCode(code, message) {
37
+ return Object.assign(new Error(message), { code });
38
+ }
39
+ function mediaHint(extractor, label) {
40
+ const labelExtension = path.extname(label).toLowerCase();
41
+ switch (extractor) {
42
+ case "json":
43
+ return ".json";
44
+ case "jsonl":
45
+ return ".jsonl";
46
+ case "yaml":
47
+ return labelExtension === ".yml" ? ".yml" : ".yaml";
48
+ case "plain-text":
49
+ return labelExtension === ".md" ? ".md" : ".txt";
50
+ case "pdf":
51
+ return ".pdf";
52
+ case "docx":
53
+ return ".docx";
54
+ case "xlsx":
55
+ return ".xlsx";
56
+ case "pptx":
57
+ return ".pptx";
58
+ case "image-ocr":
59
+ return [".png", ".jpg", ".jpeg"].includes(labelExtension) ? labelExtension : ".png";
60
+ default:
61
+ throw errorWithCode("rehash_failed", `Cannot re-extract version with unknown extractor "${extractor}".`);
62
+ }
63
+ }
64
+ function sameFacts(row, facts) {
65
+ return row.content_hash === facts.contentHash && row.normalized_hash === facts.normalizedHash && row.content_kind === facts.contentKind && row.hash_version === facts.hashVersion;
66
+ }
67
+ async function rehashNamespace(root, namespace, apply) {
68
+ const dbPath = clientDbPath(root, namespace.client);
69
+ try {
70
+ await fs.access(dbPath);
71
+ } catch (error) {
72
+ if (error.code === "ENOENT") {
73
+ throw errorWithCode("database_not_found", `Client database not found: ${dbPath}`);
74
+ }
75
+ throw errorWithCode("io_error", `Unable to inspect client database: ${dbPath}`);
76
+ }
77
+ const db = openDatabase(dbPath);
78
+ try {
79
+ const project = db.prepare(`SELECT project.project_id FROM project
80
+ JOIN client ON client.client_id = project.client_id WHERE client.name = ? AND project.slug = ?`).get(namespace.client, namespace.project);
81
+ if (!project) return { applied: apply, updated: 0, unchanged: 0, skipped: 0, ...!apply ? { backupReminder: reminder() } : {} };
82
+ const rows = db.prepare(`SELECT version.doc_version_id, version.sha256, version.content_hash, version.normalized_hash,
83
+ version.content_kind, version.hash_version, version.extractor, document.label
84
+ FROM doc_version AS version
85
+ JOIN document ON document.document_id = version.document_id
86
+ WHERE document.project_id = ? ORDER BY version.doc_version_id`).all(project.project_id);
87
+ const updates = [];
88
+ let unchanged = 0;
89
+ let skipped = 0;
90
+ for (const row of rows) {
91
+ let bytes;
92
+ try {
93
+ bytes = await readObject(root, row.sha256);
94
+ } catch (error) {
95
+ const code = error.code === "ENOENT" ? "object_missing" : "object_read_failed";
96
+ throw errorWithCode(code, `Cannot read object ${row.sha256} for version ${row.doc_version_id}.`);
97
+ }
98
+ const objectHash = createHash("sha256").update(bytes).digest("hex");
99
+ if (objectHash !== row.sha256) {
100
+ throw errorWithCode("object_corrupt", `Object ${row.sha256} failed SHA-256 verification for version ${row.doc_version_id}.`);
101
+ }
102
+ let extracted;
103
+ try {
104
+ extracted = await parseDocument(
105
+ path.join(root, "objects", row.sha256.slice(0, 2), row.sha256.slice(2, 4), row.sha256),
106
+ namespace,
107
+ void 0,
108
+ { mediaHint: mediaHint(row.extractor, row.label) }
109
+ );
110
+ } catch (error) {
111
+ if (error.code === "rehash_failed") throw error;
112
+ throw errorWithCode("rehash_failed", `Cannot re-extract version ${row.doc_version_id}: ${error instanceof Error ? error.message : String(error)}`);
113
+ }
114
+ const facts = computeContentHashFacts(extracted);
115
+ if (facts.skipped) {
116
+ skipped += 1;
117
+ } else if (sameFacts(row, facts)) {
118
+ unchanged += 1;
119
+ } else {
120
+ updates.push({
121
+ docVersionId: row.doc_version_id,
122
+ contentHash: facts.contentHash,
123
+ normalizedHash: facts.normalizedHash,
124
+ contentKind: facts.contentKind,
125
+ hashVersion: facts.hashVersion
126
+ });
127
+ }
128
+ }
129
+ if (apply && updates.length > 0) {
130
+ db.exec("BEGIN IMMEDIATE;");
131
+ try {
132
+ const update = db.prepare(`UPDATE doc_version SET content_hash = ?, normalized_hash = ?, content_kind = ?, hash_version = ?
133
+ WHERE doc_version_id = ?`);
134
+ for (const row of updates) update.run(row.contentHash, row.normalizedHash, row.contentKind, row.hashVersion, row.docVersionId);
135
+ await writeRootManifest(root, db);
136
+ db.exec("COMMIT;");
137
+ } catch (error) {
138
+ try {
139
+ db.exec("ROLLBACK;");
140
+ } catch {
141
+ }
142
+ throw errorWithCode("rehash_failed", `Backfill transaction failed: ${error instanceof Error ? error.message : String(error)}`);
143
+ }
144
+ }
145
+ return {
146
+ applied: apply,
147
+ updated: updates.length,
148
+ unchanged,
149
+ skipped,
150
+ ...!apply ? { backupReminder: reminder() } : {}
151
+ };
152
+ } finally {
153
+ db.close();
154
+ }
155
+ }
156
+ function reminder() {
157
+ return "Before applying, back up this namespace with: rag backup --to <backup-dir>.";
158
+ }
159
+ export {
160
+ computeContentHashFacts,
161
+ rehashNamespace
162
+ };
@@ -1,4 +1,4 @@
1
- const SCHEMA_VERSION = 1;
1
+ const SCHEMA_VERSION = 2;
2
2
  const MIGRATIONS = [
3
3
  {
4
4
  version: 1,
@@ -170,6 +170,16 @@ BEFORE DELETE ON doc_version
170
170
  BEGIN
171
171
  SELECT RAISE(ABORT, 'doc_version history is immutable');
172
172
  END;`
173
+ },
174
+ {
175
+ version: 2,
176
+ sql: `CREATE TABLE schema_migration (
177
+ version INTEGER PRIMARY KEY,
178
+ applied_at TEXT NOT NULL
179
+ );
180
+
181
+ ALTER TABLE doc_version ADD COLUMN content_kind TEXT;
182
+ ALTER TABLE doc_version ADD COLUMN hash_version INTEGER NOT NULL DEFAULT 1;`
173
183
  }
174
184
  ];
175
185
  export {
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "agent-rag-mcp",
3
- "version": "0.2.0",
3
+ "version": "0.4.0",
4
4
  "description": "Deterministic, namespace-isolated document retrieval with a CLI and MCP server.",
5
5
  "license": "MIT",
6
6
  "repository": {
@@ -28,12 +28,16 @@
28
28
  "bin",
29
29
  "dist",
30
30
  "extensions",
31
+ "skills",
31
32
  "README.md",
32
33
  "LICENSE"
33
34
  ],
34
35
  "pi": {
35
36
  "extensions": [
36
37
  "./extensions/agent-rag.ts"
38
+ ],
39
+ "skills": [
40
+ "./skills"
37
41
  ]
38
42
  },
39
43
  "engines": {
@@ -58,6 +62,7 @@
58
62
  "mammoth": "1.13.0",
59
63
  "pdfjs-dist": "6.4.299",
60
64
  "tesseract.js": "7.0.0",
65
+ "yaml": "^2.9.1",
61
66
  "zod": "4.6.7"
62
67
  }
63
68
  }
@@ -0,0 +1,35 @@
1
+ ---
2
+ name: agent-rag
3
+ description: "Trigger: search or read the indexed knowledge base with explicit client/project scope and verifiable citations."
4
+ ---
5
+
6
+ ## Choose a tool
7
+
8
+ - Use `rag_find` to search a client/project namespace and get a bounded catalog of matching citations.
9
+ - Use `rag_read` only when a citation needs more surrounding text; pass its `docVersionId` and `chunkOrdinal` from `rag_find`.
10
+ - The orchestrator only receives `rag_find`; delegate exploration when a citation needs reading.
11
+
12
+ ## Evidence and scope
13
+
14
+ - The `namespace` is always explicit: pass an object with non-empty `client` and `project`; never infer it from cwd or discover another namespace.
15
+ - Cite source wording with a verbatim quote. Citations must be quoted, never paraphrased as source text.
16
+ - Keep the citation's document/version and section context with the quote; say when a search returns no matches.
17
+ - Treat `rag_find` excerpts as bounded evidence, not the complete source. Use `rag_read` for a bounded passage only when needed.
18
+
19
+ ## Config and CLI
20
+
21
+ - CLI namespace/root options are `--client`, `--project`, `--root`; environment alternatives are `RAG_CLIENT`, `RAG_PROJECT`, `RAG_ROOT`.
22
+ - The CLI home is `RAG_HOME ?? os.homedir()`. The MCP server and Pi extension pass cwd as `homeDir`.
23
+ - The only config keys read in `src/config/discovery.mts` are `client`, `project`, `root`, and `failClosed`; unknown keys are silently ignored.
24
+ - Config candidates, first existing only: `<cwd>/rag.config.json`, `<dirname(cwd)>/rag.config.json` (one level up only), then `<homeDir>/.rag/config.json`.
25
+ - Only JSON syntax errors (`config_invalid`) or config-file I/O errors (`config_unreadable`) raise those config errors.
26
+ - Each field resolves independently to its first available value: client `--client` > `RAG_CLIENT` > file > null; project follows the same chain.
27
+ - Root precedence is `--root` > `RAG_ROOT` > file > `<homeDir>/.rag`; fields can resolve from different sources.
28
+ - Discovery reports its source as `args`, `env`, `cwd`, `parent`, `home`, or `unscoped`.
29
+ - `client` must match `/^[A-Za-z0-9][A-Za-z0-9._-]*$/`; `.` and `..` are rejected with `invalid_client_name`. Its DB is `<root>/clients/<client>/rag.db`.
30
+ - `project` is logical only: its slug is stored in `project.slug` rows in that client DB; it creates no directory and has no name validation.
31
+ - `root` defaults to `<homeDir>/.rag`; the CLI resolves a relative root against cwd for each command.
32
+ - Root layout: `clients/<client>/rag.db`, `objects/<aa>/<bb>/<sha256>`, and `manifest.json`.
33
+ - `failClosed` defaults to true and is file-only (no flag or environment variable). For writes without a namespace it throws `namespace_required`; false uses reserved `default/default`. Reads are unaffected.
34
+ - `rag init` does not write `rag.config.json`; it creates only the client DB, object store, and root manifest.
35
+ - MCP tools require `namespace: {client, project}` on every call. The server resolves only `root` from discovery; config, `RAG_CLIENT`, and `RAG_PROJECT` never provide tool identity.