agent-rag-mcp 0.7.0 → 0.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -16,6 +16,12 @@ Install globally so an MCP adapter can resolve `agent-rag-mcp` from its `PATH`:
16
16
  npm install --global agent-rag-mcp
17
17
  ```
18
18
 
19
+ ### Standalone terminal interface
20
+
21
+ Run `agent-rag` in an interactive terminal to browse and administer the knowledge base without Pi. Use `agent-rag --root <path>` to select a different storage root. Otherwise, it uses the same root discovery as the CLI, including `rag.config.json`, `RAG_ROOT`, and the CLI home default. The current root and selected client/project appear in the menu. This interface requires a terminal; use `rag` for scripts and JSON output.
22
+
23
+ Choose a client and project to browse all its documents (including retired ones), inspect current and historical versions, read extracted text in pages, search current documents, or download a selected version's original file. Downloads verify the stored SHA-256 and refuse to overwrite an existing destination. The menu also supports file/directory imports, status, doctor, review decisions, relabeling and retiring documents, entity renaming, backup, restore, reindex and rehash. Write operations ask for confirmation. Backup includes the selected client and all its projects, not other clients; restore requires an empty destination root and restores the selected client. Semantic search remains available through the Pi interface and MCP tools.
24
+
19
25
  The `rag` commands available today are:
20
26
 
21
27
  - `rag init` — Initializes the namespace database and object store.
@@ -87,9 +93,27 @@ The orchestrator receives only `rag_find`, which returns citations and bounded e
87
93
 
88
94
  The extension adds a `rag` section to the agent system prompt describing the namespace and delegation rules; it contains no client, project, or path value. For truncated excerpts, coverage questions, or longer verbatim quotes, it requires deeper text to be read by a subagent that exposes `rag_read` and forbids asserting absence from a catalog. Set `RAG_PROMPT=off` (exactly that value) to suppress it; any other value, including unset, keeps it.
89
95
 
96
+ In interactive Pi sessions, `/rag` asks for client and project, then offers search, status, document list, review queue and doctor. Status reports stored vector records and bytes for the project, including older versions and model identities; it is not an index-completeness count. Search offers lexical, semantic and hybrid modes and displays up to ten citations. Selecting a result reads a bounded passage only if the extension's `read` capability is enabled. Setting `RAG_TUI_ADMIN=1` also enables importing a file or directory, retiring a document, recording review decisions, backup and restore; each mutation requires interactive confirmation. Restore still requires an empty destination root. This is an opt-in UI gate, not authentication or an OS security boundary. The command requires an interactive UI and calls the packaged `rag` CLI for administration.
97
+
90
98
  The namespace travels in every `rag_find` or `rag_read` call; it does not come from the environment. For example, include `"namespace": {"client":"acme","project":"research"}` in each tool call's arguments. Do not configure `RAG_CLIENT` or `RAG_PROJECT` as a substitute for the per-call namespace.
91
99
 
92
- `rag_find` returns `total`, the number of matching chunks before the `limit` slice; `citations.length` is how many citations were actually returned, so comparing them shows how many matches were cut by the limit. Its `excerptChars` input defaults to 800. Every successful `rag_read` result includes `totalChunks` and `nextChunkOrdinal`; the latter is the next zero-based chunk ordinal, or `null` when there is no following chunk. `truncated` means this chunk was not delivered whole, so re-read the same ordinal with higher limits; it does not indicate whether another chunk exists. `rag_read` accepts `startLine`, the zero-based first line to return (default `0`, maximum `10,000`), and returns `returnedBytes` and `returnedLines` for the page, `chunkBytes` for the full chunk, and `truncatedBy` (`"bytes"`, `"lines"`, or `null`) alongside `truncated`. For a chunk that exceeds the line limit, ensure each page fits within `maxBytes`, then repeat calls advancing `startLine` by `returnedLines`; this covers every line without overlap or gaps, and `truncatedBy: null` marks the last page.
100
+ Namespace filtering and `RAG_CAPABILITIES` limit requests and tools, but they do not authenticate callers or enforce the `namespace_acl` table. A caller with access to a server can request any namespace accessible under its root. Use an authenticated authorization boundary and operating-system permissions when clients or users must not access each other's data.
101
+
102
+ To constrain one MCP server process or Pi extension to particular namespaces, set `RAG_ALLOWED_NAMESPACES` to a JSON array such as `[{"client":"acme","project":"research"}]`. An empty array denies all namespaces; malformed configuration fails startup. When unset, all namespaces under the root remain available. This setting does not identify callers, restrict direct filesystem access or apply to the CLI; enforce those boundaries separately.
103
+
104
+ The CLI defaults to the user's `.rag` directory, while the MCP server and Pi extension resolve their default relative to their working directory. Set the same absolute `RAG_ROOT` for all processes that must share an index. Installing the Pi package also registers native tools; if using the MCP adapter at the same time, configure and verify each route deliberately to avoid duplicate tools or inconsistent capabilities.
105
+
106
+ `rag_find` searches current document versions by default. Set `includeHistorical: true` to include superseded versions; each citation has an `isCurrent` flag so historical evidence is distinguishable. `total` is the number of matching chunks before the `limit` slice; `citations.length` is how many citations were returned. Its `excerptChars` input defaults to 800. Each `rag_read` result reports `totalChunks` and `nextChunkOrdinal` for traversing chunks; `truncated` concerns the current chunk only. It reports `chunkBytes`, `returnedBytes`, `returnedLines`, `truncatedBy` (`"bytes"`, `"lines"`, or `null`) and `nextByteOffset` for the current page.
107
+
108
+ For byte-exact pagination, start with `startByte: 0`, a bounded `maxBytes` (for example 8,192), and a positive `maxLines`. Repeat the same citation using the returned `nextByteOffset` as `startByte` until it is `null`; concatenate page texts directly, without adding separators. Offsets count UTF-8 bytes in the stored chunk and must be character boundaries. The byte-page mode includes any line separator cut by `maxLines` in the page, preserving exact text. If `nextByteOffset` does not advance, increase `maxBytes` to fit the next UTF-8 character and ensure `maxLines` is positive. Leave room for JSON escaping and transport overhead when choosing page sizes; if the transport truncates a response, retry that page with a smaller limit. Do not combine `startByte` and `startLine`. Existing line pagination remains available with `startLine` (default 0, maximum 10,000), advancing it by `returnedLines` only when each complete page fits within `maxBytes`; line pages omit the separating newline at a line-limit cut.
109
+
110
+ Search matches every query term by default (`matchMode: "all"`). Set `matchMode: "any"` to return candidates containing at least one query term. This broader mode may return unrelated documents and does not provide semantic search.
111
+
112
+ For semantic search, configure `RAG_EMBEDDING_MODEL` with an embedding-capable model already installed in a local Ollama instance at `127.0.0.1:11434`, then set `searchMode: "semantic"` or `"hybrid"` on `rag_find`. The first search embeds the query and indexes eligible chunks in small batches; subsequent searches embed the query and only new or changed chunks. Each chunk must be at most 8,192 characters. Set `RAG_EMBEDDING_BATCH_SIZE` from 1 to 32 (default 8) and `RAG_EMBEDDING_BATCH_BYTES` from 1,024 to 1,048,576 (default 65,536) to bound the number and total UTF-8 bytes of chunk texts in a provider batch. This is not a hard process-memory limit and does not include model, query, vectors, or JSON overhead. A chunk exceeding the byte budget fails explicitly. Search scans the cached vectors and keeps the top 100 semantic candidates in memory; hybrid combines the top 100 lexical and semantic ranks using reciprocal rank fusion. Semantic `score` is cosine similarity (higher is better), and `matchStart`/`matchEnd` are `null` when there is no literal match. Results are ranked candidates, not proof of relevance.
113
+
114
+ Alternatively, explicitly set `RAG_EMBEDDING_PROVIDER=azure` along with `RAG_AZURE_OPENAI_ENDPOINT`, `RAG_AZURE_OPENAI_DEPLOYMENT` and `RAG_AZURE_OPENAI_API_KEY` in the server or Pi process environment. Azure OpenAI v1 embeddings accepts the resource HTTPS endpoint and deployment name. The key should be provided through a secret manager or private environment, never committed or passed in a tool call. Semantic and hybrid searches send the query and any uncached eligible chunk text to the configured Azure endpoint, which may incur charges. They do not send documents until invoked. Confirm data-processing authorization and expected costs before enabling it on real documents. Omitting the provider keeps Ollama as the default; selecting Azure never falls back silently to another provider.
115
+
116
+ Vectors are stored as float32 in the derived `clients/<client>/semantic-cache.sqlite`, separated by project, provider, endpoint, model/deployment and `RAG_EMBEDDING_REVISION`. Semantic searches write to this cache and require a writable client directory even when the process exposes only `find`. Protect it like the indexed documents: vectors may contain sensitive information. The cache is rebuilt on demand and is not included in backup or manifest. Historical and retired vectors may remain in the cache, but search only returns eligible versions. Deleting the cache with the server stopped forces re-embedding and can incur Azure charges. If an Ollama model tag or Azure deployment is changed in place, change `RAG_EMBEDDING_REVISION` to a new value; otherwise vectors from the previous model revision can be reused incorrectly. No automatic model-version detection or cache garbage collection is implemented.
93
117
 
94
118
  ## Requirements
95
119
 
@@ -98,10 +122,12 @@ The namespace travels in every `rag_find` or `rag_read` call; it does not come f
98
122
  - `npm run build` builds each source file with esbuild (per-file).
99
123
  - `npm test` runs the tests with Node's `node:test`.
100
124
  - `npm run typecheck` runs the TypeScript typecheck without emitting files.
125
+ - `npm run evaluate:retrieval` runs a small synthetic retrieval baseline, including a semantic query that lexical search may miss. It does not evaluate answer generation or real-world quality.
126
+ - Set `RAG_EVAL_SEMANTIC=1` and `RAG_EMBEDDING_MODEL` when running the evaluation to compare local Ollama rankings on synthetic documents; this uses the local provider only and does not measure abstention or production quality.
101
127
 
102
128
  ## Known limits
103
129
 
104
- Version 0.1.0 has no embeddings or Tier 2; scanned PDFs go through OCR. Document parsing runs in a child process with a 512 MB V8 heap cap.
130
+ The current vector cache performs an exact linear scan and has no automatic garbage collection or Tier 2. PDF pages without extractable text go through OCR, including in mixed PDFs. OCR defaults to English; set `RAG_OCR_LANGUAGE=spa` for Spanish or `eng` for English before ingestion. Both language datasets ship as dependencies and OCR does not download data at runtime. OCR is limited to 100 pages per document. Document parsing runs in a child process with a 512 MB V8 heap cap. Existing indexed PDFs retain their previous extraction; ingesting identical bytes is idempotent and does not re-extract them.
105
131
 
106
132
  At ingest, DOCX files split by heading; heading groups over 8,192 characters are subdivided at whitespace, keeping the heading at the start of the first section. Files without headings use greedy paragraph chunks capped at 8,192 characters, and oversized paragraphs split at whitespace. Already-ingested DOCX documents keep their existing single chunk until re-ingested; `rag reindex` does not re-extract source documents.
107
133
 
@@ -0,0 +1,26 @@
1
+ #!/usr/bin/env node
2
+ import fs from "node:fs";
3
+ import path from "node:path";
4
+ import { spawnSync } from "node:child_process";
5
+ import { fileURLToPath } from "node:url";
6
+
7
+ const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
8
+ const built = path.join(root, "dist", "src", "tui", "index.mjs");
9
+ const source = path.join(root, "src", "tui", "index.mts");
10
+ const mode = process.env.RAG_ENTRY ?? "auto";
11
+ if (!["auto", "source", "dist"].includes(mode)) {
12
+ console.error(`agent-rag: invalid RAG_ENTRY: ${mode}`);
13
+ process.exitCode = 1;
14
+ } else {
15
+ const entry = (mode === "source" ? [source] : mode === "dist" ? [built] : [built, source]).find(fs.existsSync);
16
+ if (!entry) {
17
+ console.error("agent-rag: TUI entry not found; build the package first.");
18
+ process.exitCode = 1;
19
+ } else {
20
+ const flags = entry === source ? ["--experimental-strip-types", "--experimental-sqlite"] : ["--experimental-sqlite"];
21
+ const child = spawnSync(process.execPath, [...flags, entry, ...process.argv.slice(2)], { stdio: "inherit" });
22
+ if (child.error) { console.error(`agent-rag: ${child.error.message}`); process.exitCode = 1; }
23
+ else if (child.signal) process.kill(process.pid, child.signal);
24
+ else process.exitCode = child.status ?? 1;
25
+ }
26
+ }
@@ -1,8 +1,4 @@
1
- import*as h from"node:fs";import*as N from"node:path";var I=class extends Error{code;path;constructor(t,n,r){let o=r instanceof Error?r.message:String(r);super(n==="config_invalid"?`Invalid JSON in config file ${t}: ${o}`:`Unable to read config file ${t}: ${o}`,{cause:r}),this.name="ConfigDiscoveryError",this.code=n,this.path=t}};function ie(e){let t;try{t=h.readFileSync(e,"utf8")}catch(n){throw new I(e,"config_unreadable",n)}try{return JSON.parse(t)}catch(n){throw new I(e,"config_invalid",n)}}function D({cwd:e,homeDir:t,argv:n,env:r}){let o={};for(let f=0;f<n.length;f+=1){let F=/^--(client|project|root)=(.*)$/.exec(n[f]);F?o[F[1]]=F[2]:/^--(client|project|root)$/.test(n[f])&&n[f+1]!==void 0&&(o[n[f].slice(2)]=n[f+1],f+=1)}let c={client:r.RAG_CLIENT,project:r.RAG_PROJECT,root:r.RAG_ROOT},d=N.join(e,"rag.config.json"),a=N.join(N.dirname(e),"rag.config.json"),u=N.join(t,".rag","config.json"),l=h.existsSync(d)?d:h.existsSync(a)?a:h.existsSync(u)?u:null,i=l===null?{}:ie(l),p=typeof i.client=="string"?i.client:void 0,E=typeof i.project=="string"?i.project:void 0,s=o.client??c.client??p??null,m=o.project??c.project??E??null,T=s===null&&m===null?null:{client:s,project:m},y=o.client!==void 0||o.project!==void 0?"args":c.client!==void 0||c.project!==void 0?"env":p!==void 0||E!==void 0?l===d?"cwd":l===a?"parent":"home":"unscoped";return{namespace:T,source:y,configPath:y==="cwd"||y==="parent"||y==="home"?l:null,failClosed:typeof i.failClosed=="boolean"?i.failClosed:!0,root:o.root??c.root??(typeof i.root=="string"?i.root:N.join(t,".rag"))}}import*as O from"node:fs";import{DatabaseSync as ae}from"node:sqlite";import*as B from"node:path";import{DatabaseSync as xe}from"node:sqlite";var M=class extends Error{code="invalid_client_name";constructor(t){super(`Invalid client name "${t}". Client names must match [A-Za-z0-9][A-Za-z0-9._-]* and cannot be "." or "..".`),this.name="InvalidClientNameError"}};function oe(e){if(e==="."||e===".."||!/^[A-Za-z0-9][A-Za-z0-9._-]*$/.test(e))throw new M(e)}function v(e,t){return oe(t),B.join(e,"clients",t,"rag.db")}var se=800,P=/[\p{L}\p{M}\p{N}]+/gu;function ce(e){return e.match(P)??[]}function de(e){return e.map(t=>`"${t.replaceAll('"','""')}"`).join(" AND ")}function K(e){return e.normalize("NFD").replace(new RegExp("\\p{M}","gu"),"").toLowerCase()}function ue(e,t){let n=new Set(t.map(K));for(let r of e.matchAll(P)){let o=r[0];if(!n.has(K(o)))continue;let c=r.index;return{start:c,end:c+o.length}}return null}function le(e,t,n,r){let o=Math.min(r,e.length);if(o===0)return"";let c=Math.min(n-t,o),d=Math.max(0,Math.min(t-Math.floor((o-c)/2),e.length-o)),a=d+o,u=d,l=a,i=Array.from(e.matchAll(P),s=>({start:s.index,end:s.index+s[0].length})),p=i.find(s=>s.start<d&&d<s.end);p&&(d=i.find(s=>s.start>=p.end)?.start??e.length);let E=i.find(s=>s.start<a&&a<s.end);if(E){let s=i.filter(m=>m.end<=E.start);a=s[s.length-1]?.end??0}return a<=d?e.slice(u,l):e.slice(d,a)}function pe(e,t){return e<t?-1:e>t?1:0}function b(e,t,n){return{namespace:e,error:{code:t,message:n}}}function z(e){let{root:t,namespace:n,query:r}=e,o=e.excerptLength??se;if(!Number.isInteger(o)||o<1)return b(n,"invalid_excerpt_length","excerptLength must be a positive integer.");let c=ce(r);if(c.length===0)return{namespace:n,result:{citations:[],total:0}};let d;try{let a=v(t,n.client);if(!O.existsSync(a)||!O.statSync(a).isFile())return b(n,"database_not_found",`No database exists for namespace ${n.client}/${n.project}.`);d=new ae(a,{readOnly:!0});let u=d.prepare(`SELECT dv.doc_version_id AS docVersionId, d.document_id AS documentId,
2
- e.entity_key AS entityKey, d.label AS label, dv.version_label AS versionLabel,
3
- s.ordinal AS sectionOrdinal, c.ordinal AS chunkOrdinal, c.text AS text,
4
- c.char_start AS charStart, c.char_end AS charEnd, bm25(chunk_fts) AS score
5
- FROM chunk_fts
1
+ import*as de from"node:path";import*as X from"node:fs";import*as M from"node:path";var H=class extends Error{code;path;constructor(n,r,i){let t=i instanceof Error?i.message:String(i);super(r==="config_invalid"?`Invalid JSON in config file ${n}: ${t}`:`Unable to read config file ${n}: ${t}`,{cause:i}),this.name="ConfigDiscoveryError",this.code=r,this.path=n}};function He(e){let n;try{n=X.readFileSync(e,"utf8")}catch(r){throw new H(e,"config_unreadable",r)}try{return JSON.parse(n)}catch(r){throw new H(e,"config_invalid",r)}}function G({cwd:e,homeDir:n,argv:r,env:i}){let t={};for(let y=0;y<r.length;y+=1){let h=/^--(client|project|root)=(.*)$/.exec(r[y]);h?t[h[1]]=h[2]:/^--(client|project|root)$/.test(r[y])&&r[y+1]!==void 0&&(t[r[y].slice(2)]=r[y+1],y+=1)}let a={client:i.RAG_CLIENT,project:i.RAG_PROJECT,root:i.RAG_ROOT},d=M.join(e,"rag.config.json"),u=M.join(M.dirname(e),"rag.config.json"),l=M.join(n,".rag","config.json"),c=X.existsSync(d)?d:X.existsSync(u)?u:X.existsSync(l)?l:null,o=c===null?{}:He(c),m=typeof o.client=="string"?o.client:void 0,p=typeof o.project=="string"?o.project:void 0,s=t.client??a.client??m??null,_=t.project??a.project??p??null,T=s===null&&_===null?null:{client:s,project:_},E=t.client!==void 0||t.project!==void 0?"args":a.client!==void 0||a.project!==void 0?"env":m!==void 0||p!==void 0?c===d?"cwd":c===u?"parent":"home":"unscoped";return{namespace:T,source:E,configPath:E==="cwd"||E==="parent"||E==="home"?c:null,failClosed:typeof o.failClosed=="boolean"?o.failClosed:!0,root:t.root??a.root??(typeof o.root=="string"?o.root:M.join(n,".rag"))}}import*as Y from"node:fs";import{DatabaseSync as Ye}from"node:sqlite";import*as Ee from"node:path";import{DatabaseSync as At}from"node:sqlite";var fe=class extends Error{code="invalid_client_name";constructor(n){super(`Invalid client name "${n}". Client names must match [A-Za-z0-9][A-Za-z0-9._-]* and cannot be "." or "..".`),this.name="InvalidClientNameError"}};function ze(e){if(e==="."||e===".."||!/^[A-Za-z0-9][A-Za-z0-9._-]*$/.test(e))throw new fe(e)}function x(e,n){return ze(n),Ee.join(e,"clients",n,"rag.db")}var Je=800,_e=/[\p{L}\p{M}\p{N}]+/gu;function Ke(e){return e.match(_e)??[]}function qe(e,n){return e.map(r=>`"${r.replaceAll('"','""')}"`).join(n==="any"?" OR ":" AND ")}function Ae(e){return e.normalize("NFD").replace(new RegExp("\\p{M}","gu"),"").toLowerCase()}function We(e,n){let r=new Set(n.map(Ae));for(let i of e.matchAll(_e)){let t=i[0];if(!r.has(Ae(t)))continue;let a=i.index;return{start:a,end:a+t.length}}return null}function Ze(e,n,r,i){let t=Math.min(i,e.length);if(t===0)return"";let a=Math.min(r-n,t),d=Math.max(0,Math.min(n-Math.floor((t-a)/2),e.length-t)),u=d+t,l=d,c=u,o=Array.from(e.matchAll(_e),s=>({start:s.index,end:s.index+s[0].length})),m=o.find(s=>s.start<d&&d<s.end);m&&(d=o.find(s=>s.start>=m.end)?.start??e.length);let p=o.find(s=>s.start<u&&u<s.end);if(p){let s=o.filter(_=>_.end<=p.start);u=s[s.length-1]?.end??0}return u<=d?e.slice(l,c):e.slice(d,u)}function Qe(e,n){return e<n?-1:e>n?1:0}function z(e,n,r){return{namespace:e,error:{code:n,message:r}}}function P(e){let{root:n,namespace:r,query:i}=e,t=e.excerptLength??Je;if(!Number.isInteger(t)||t<1)return z(r,"invalid_excerpt_length","excerptLength must be a positive integer.");let a=Ke(i);if(a.length===0)return{namespace:r,result:{citations:[],total:0}};let d;try{let u=x(n,r.client);if(!Y.existsSync(u)||!Y.statSync(u).isFile())return z(r,"database_not_found",`No database exists for namespace ${r.client}/${r.project}.`);d=new Ye(u,{readOnly:!0}),d.exec("BEGIN");let l=`FROM chunk_fts
6
2
  JOIN chunk AS c ON c.rowid = chunk_fts.rowid
7
3
  JOIN doc_version AS dv ON dv.doc_version_id = c.doc_version_id
8
4
  JOIN document AS d ON d.document_id = dv.document_id
@@ -11,18 +7,47 @@ import*as h from"node:fs";import*as N from"node:path";var I=class extends Error{
11
7
  JOIN client AS cl ON cl.client_id = p.client_id
12
8
  LEFT JOIN section AS s ON s.section_id = c.section_id
13
9
  WHERE chunk_fts MATCH ? AND cl.name = ? AND p.slug = ?
14
- AND d.deleted_at IS NULL AND e.deleted_at IS NULL
15
- ORDER BY score ASC, dv.doc_version_id COLLATE BINARY ASC, c.ordinal ASC`).all(de(c),n.client,n.project),l=[];for(let i of u){let p=ue(i.text,c);if(!p)return b(n,"match_offsets_unavailable","A matched FTS token could not be located in its stored chunk text.");l.push({docVersionId:i.docVersionId,documentId:i.documentId,entityKey:i.entityKey,label:i.label,versionLabel:i.versionLabel,sectionOrdinal:i.sectionOrdinal,chunkOrdinal:i.chunkOrdinal,charStart:i.charStart,charEnd:i.charEnd,matchStart:p.start,matchEnd:p.end,score:i.score,excerpt:le(i.text,p.start,p.end,o)})}return l.sort((i,p)=>i.score-p.score||pe(i.docVersionId,p.docVersionId)||i.chunkOrdinal-p.chunkOrdinal),{namespace:n,result:{citations:l,total:l.length}}}catch(a){let u=a instanceof Error?a.message:String(a);return b(n,"database_unavailable",u)}finally{d?.close()}}import*as x from"node:fs";import{DatabaseSync as Ee}from"node:sqlite";function C(e,t,n){return{namespace:e,error:{code:t,message:n}}}function me(e,t){let n=Buffer.from(e,"utf8");if(n.byteLength<=t)return{text:e,truncated:!1};let r=t;for(;r>0;)try{return{text:new TextDecoder("utf-8",{fatal:!0}).decode(n.subarray(0,r)),truncated:!0}}catch{r-=1}return{text:"",truncated:!0}}function _e(e,t){if(t===0)return 0;let n=0;for(let r=0;r<e.length;r+=1)if(!(e[r]!=="\r"&&e[r]!==`
16
- `)&&(e[r]==="\r"&&e[r+1]===`
17
- `&&(r+=1),n+=1,n===t))return r+1;return e.length}function Te(e){if(e.length===0)return 0;let t=1;for(let n=0;n<e.length;n+=1)e[n]==="\r"&&e[n+1]===`
18
- `?(t+=1,n+=1):(e[n]==="\r"||e[n]===`
19
- `)&&(t+=1);return t}function fe(e,t){if(e.length===0)return{text:e,truncated:!1};if(t===0)return{text:"",truncated:!0};let n=1;for(let r=0;r<e.length;r+=1){if(e[r]!=="\r"&&e[r]!==`
20
- `)continue;let o=r;if(e[r]==="\r"&&e[r+1]===`
21
- `&&(r+=1),n===t)return{text:e.slice(0,o),truncated:!0};n+=1}return{text:e,truncated:!1}}async function V(e){let{root:t,namespace:n,docVersionId:r,chunkOrdinal:o,maxBytes:c,maxLines:d,startLine:a=0}=e;if(!Number.isInteger(c)||c<0||!Number.isInteger(d)||d<0||!Number.isInteger(a)||a<0)return C(n,"invalid_read_limit","maxBytes, maxLines, and startLine must be non-negative integers.");let u;try{let l=v(t,n.client);if(!x.existsSync(l)||!x.statSync(l).isFile())return C(n,"database_not_found",`No database exists for namespace ${n.client}/${n.project}.`);u=new Ee(l,{readOnly:!0});let i=u.prepare(`SELECT c.text AS text,
10
+ AND d.deleted_at IS NULL AND e.deleted_at IS NULL AND (dv.is_current = 1 OR ? = 1)`,c=[qe(a,e.matchMode??"all"),r.client,r.project,e.includeHistorical?1:0],{total:o}=d.prepare(`SELECT COUNT(*) AS total ${l}`).get(...c),m=d.prepare(`SELECT dv.doc_version_id AS docVersionId, d.document_id AS documentId,
11
+ e.entity_key AS entityKey, d.label AS label, dv.version_label AS versionLabel, dv.is_current AS isCurrent,
12
+ s.ordinal AS sectionOrdinal, c.ordinal AS chunkOrdinal, c.text AS text,
13
+ c.char_start AS charStart, c.char_end AS charEnd, bm25(chunk_fts) AS score
14
+ ${l}
15
+ ORDER BY score ASC, dv.doc_version_id COLLATE BINARY ASC, c.ordinal ASC LIMIT ?`).all(...c,e.maxResults??-1);d.exec("COMMIT");let p=[];for(let s of m){let _=We(s.text,a);if(!_)return z(r,"match_offsets_unavailable","A matched FTS token could not be located in its stored chunk text.");p.push({docVersionId:s.docVersionId,documentId:s.documentId,entityKey:s.entityKey,label:s.label,versionLabel:s.versionLabel,isCurrent:s.isCurrent===1,sectionOrdinal:s.sectionOrdinal,chunkOrdinal:s.chunkOrdinal,charStart:s.charStart,charEnd:s.charEnd,matchStart:_.start,matchEnd:_.end,score:s.score,excerpt:Ze(s.text,_.start,_.end,t)})}return p.sort((s,_)=>s.score-_.score||Qe(s.docVersionId,_.docVersionId)||s.chunkOrdinal-_.chunkOrdinal),{namespace:r,result:{citations:p,total:o}}}catch(u){let l=u instanceof Error?u.message:String(u);return z(r,"database_unavailable",l)}finally{d?.close()}}import*as J from"node:fs";import{DatabaseSync as et}from"node:sqlite";function $(e,n,r){return{namespace:e,error:{code:n,message:r}}}function tt(e,n){let r=Buffer.from(e,"utf8");if(r.byteLength<=n)return{text:e,truncated:!1};let i=n;for(;i>0;)try{return{text:new TextDecoder("utf-8",{fatal:!0}).decode(r.subarray(0,i)),truncated:!0}}catch{i-=1}return{text:"",truncated:!0}}function nt(e,n){if(n===0)return 0;let r=0;for(let i=0;i<e.length;i+=1)if(!(e[i]!=="\r"&&e[i]!==`
16
+ `)&&(e[i]==="\r"&&e[i+1]===`
17
+ `&&(i+=1),r+=1,r===n))return i+1;return e.length}function rt(e){if(e.length===0)return 0;let n=1;for(let r=0;r<e.length;r+=1)e[r]==="\r"&&e[r+1]===`
18
+ `?(n+=1,r+=1):(e[r]==="\r"||e[r]===`
19
+ `)&&(n+=1);return n}function it(e,n,r=!1){if(e.length===0)return{text:e,truncated:!1};if(n===0)return{text:"",truncated:!0};let i=1;for(let t=0;t<e.length;t+=1){if(e[t]!=="\r"&&e[t]!==`
20
+ `)continue;let a=t;if(e[t]==="\r"&&e[t+1]===`
21
+ `&&(t+=1),i===n)return{text:e.slice(0,r?t+1:a),truncated:!0};i+=1}return{text:e,truncated:!1}}async function he(e){let{root:n,namespace:r,docVersionId:i,chunkOrdinal:t,maxBytes:a,maxLines:d,startLine:u=0,startByte:l}=e;if(!Number.isInteger(a)||a<0||!Number.isInteger(d)||d<0||!Number.isInteger(u)||u<0||l!==void 0&&(!Number.isSafeInteger(l)||l<0)||l!==void 0&&e.startLine!==void 0)return $(r,"invalid_read_limit","maxBytes, maxLines, and startLine must be non-negative integers.");let c;try{let o=x(n,r.client);if(!J.existsSync(o)||!J.statSync(o).isFile())return $(r,"database_not_found",`No database exists for namespace ${r.client}/${r.project}.`);c=new et(o,{readOnly:!0});let m=c.prepare(`SELECT c.text AS text,
22
22
  (SELECT COUNT(*) FROM chunk AS total_chunks WHERE total_chunks.doc_version_id = c.doc_version_id) AS totalChunks
23
23
  FROM chunk AS c
24
24
  JOIN doc_version AS dv ON dv.doc_version_id = c.doc_version_id
25
25
  JOIN document AS d ON d.document_id = dv.document_id
26
26
  JOIN project AS p ON p.project_id = d.project_id
27
27
  JOIN client AS cl ON cl.client_id = p.client_id
28
- WHERE c.doc_version_id = ? AND c.ordinal = ? AND cl.name = ? AND p.slug = ?`).get(r,o,n.client,n.project);if(!i)return C(n,"citation_not_found","The citation does not exist in the requested document version and namespace.");let p=i.text.slice(_e(i.text,a)),E=me(p,c),s=fe(E.text,d),m=E.truncated?"bytes":s.truncated?"lines":null,T=i.totalChunks;return{namespace:n,result:{text:s.text,truncated:m!==null,totalChunks:T,nextChunkOrdinal:o+1<T?o+1:null,chunkBytes:Buffer.byteLength(i.text,"utf8"),returnedBytes:Buffer.byteLength(s.text,"utf8"),returnedLines:Te(s.text),truncatedBy:m}}}catch(l){let i=l instanceof Error?l.message:String(l);return C(n,"object_read_failed",i)}finally{u?.close()}}import{z as g}from"zod";var $=100,q=2e3,H=1048576,S=1e4,Ne=10,ge=800,he=65536,Re=200;function J(e){let t=new Set;for(let n of(e??"find,read").split(",").map(r=>r.trim())){if(n!=="find"&&n!=="read")throw new Error(`invalid_capability: Unknown RAG_CAPABILITIES value "${n}".`);t.add(n)}return t}var k=g.string().min(1).refine(e=>e.trim().length>0),G=g.object({client:k,project:k}).strict();function A(e,t,n){return g.number().int().min(e).max(t).optional().transform(r=>r??n)}var Q=g.object({namespace:G,query:k,limit:A(1,$,Ne),excerptChars:A(1,q,ge)}).strict(),W=g.object({namespace:G,docVersionId:k,chunkOrdinal:g.number().multipleOf(1).min(0).refine(Number.isSafeInteger),maxBytes:A(0,H,he),maxLines:A(0,S,Re),startLine:A(0,S,0)}).strict();function Le(e){let t=e.find(c=>c.code==="unrecognized_keys"&&c.path.length===0)??e.find(c=>c.code==="unrecognized_keys");if(!t||t.code!=="unrecognized_keys")return null;let n=t.keys[0];if(n===void 0)return null;let r=t.path.map(String).join(".");return{code:"unknown_field",message:`Unknown field "${r.length>0?`${r}.${n}`:n}".`}}function _(e,t){return e.some(n=>n.path[0]===t)}function L(e,t,n){return{code:"invalid_limit",message:`${e} must be an integer from ${t} to ${n}.`}}function Z(e,t){let n=Le(e);if(n)return n;if(_(e,"namespace"))return{code:"namespace_required",message:"namespace must contain non-empty client and project strings."};if(t==="find"){if(_(e,"query"))return{code:"invalid_query",message:"query must be a non-empty string."};if(_(e,"limit"))return L("limit",1,$);if(_(e,"excerptChars"))return L("excerptChars",1,q)}else{if(_(e,"docVersionId"))return{code:"invalid_input",message:"docVersionId must be a non-empty string."};if(_(e,"chunkOrdinal"))return{code:"invalid_input",message:"chunkOrdinal must be a non-negative integer."};if(_(e,"maxBytes"))return L("maxBytes",0,H);if(_(e,"maxLines"))return L("maxLines",0,S);if(_(e,"startLine"))return L("startLine",0,S)}return e.some(r=>r.path.length===0)?{code:"invalid_input",message:"Tool arguments must be an object."}:{code:"invalid_input",message:"Tool arguments are invalid."}}function Ae(e){let t=G.safeParse(e);return t.success?t.data:null}function ee(e){let t=Q.safeParse(e);return t.success?{ok:!0,input:t.data,namespace:t.data.namespace}:{ok:!1,namespace:U(e),error:Z(t.error.issues,"find")}}function ne(e){let t=W.safeParse(e);return t.success?{ok:!0,input:t.data,namespace:t.data.namespace}:{ok:!1,namespace:U(e),error:Z(t.error.issues,"read")}}function ye(e){return typeof e=="object"&&e!==null&&!Array.isArray(e)}function U(e){return ye(e)?Ae(e.namespace):null}function te(e){return g.toJSONSchema(e,{io:"input"})}var X={name:"rag_find",description:"Search a client/project knowledge base and return bounded citations.",inputSchema:te(Q)},j={name:"rag_read",description:"Read bounded text from a cited document chunk in the explicit namespace.",inputSchema:te(W)};function Y(e,t=!1){return{content:[{type:"text",text:JSON.stringify(e)}],details:e,...t?{isError:!0}:{}}}function R(e,t){return Y({namespace:e,error:t},!0)}function w(e){return R(e,{code:"cancelled",message:"Tool call was cancelled."})}function re(e,t){let n=t instanceof Error?t.message:String(t);return R(U(e),{code:"retrieval_failed",message:n})}var Ie="The namespace travels on every call: never infer `client` or `project` from the cwd, repo name, config, env, or a placeholder like `default`. If the user has not stated it, ask before the first call.\nUse the RAG tool the role actually exposes; when only `rag_find` is available, delegate for deeper text instead of paraphrasing a catalog entry.\nWhen an excerpt is truncated, when the user asks whether the document covers something, or when a longer verbatim quote is needed, hand off to the subagent that exposes `rag_read` with the namespace and the citation's `docVersionId`; never assert absence from a catalog.";function ze(e){e.on("before_agent_start",n=>{if(process.env.RAG_PROMPT==="off")return;let r=n.systemPromptOptions.sections??(n.systemPromptOptions.sections={});r.rag=Ie});let t=J(process.env.RAG_CAPABILITIES);t.has("find")&&e.registerTool({name:X.name,label:"RAG Find",description:X.description,parameters:X.inputSchema,async execute(n,r,o,c,d){try{let a=ee(r);if(!a.ok)return R(a.namespace,a.error);let{namespace:u,query:l,limit:i,excerptChars:p}=a.input;if(o?.aborted)return w(u);let E=D({cwd:d.cwd,homeDir:d.cwd,argv:[],env:process.env}).root,s=z({root:E,namespace:u,query:l,excerptLength:p});return o?.aborted?w(u):"error"in s?R(u,s.error):Y({namespace:u,result:{...s.result,citations:s.result.citations.slice(0,i)}})}catch(a){return re(r,a)}}}),t.has("read")&&e.registerTool({name:j.name,label:"RAG Read",description:j.description,parameters:j.inputSchema,async execute(n,r,o,c,d){try{let a=ne(r);if(!a.ok)return R(a.namespace,a.error);let{namespace:u,docVersionId:l,chunkOrdinal:i,maxBytes:p,maxLines:E,startLine:s}=a.input;if(o?.aborted)return w(u);let m=D({cwd:d.cwd,homeDir:d.cwd,argv:[],env:process.env}).root,T=await V({root:m,namespace:u,docVersionId:l,chunkOrdinal:i,maxBytes:p,maxLines:E,startLine:s});return o?.aborted?w(u):"error"in T?R(u,T.error):Y({namespace:u,result:T.result})}catch(a){return re(r,a)}}})}export{ze as default};
28
+ WHERE c.doc_version_id = ? AND c.ordinal = ? AND cl.name = ? AND p.slug = ?`).get(i,t,r.client,r.project);if(!m)return $(r,"citation_not_found","The citation does not exist in the requested document version and namespace.");let p=Buffer.from(m.text,"utf8"),s=l??Buffer.byteLength(m.text.slice(0,nt(m.text,u)),"utf8");if(s>p.length||s<p.length&&(p[s]&192)===128)return $(r,"invalid_start_byte","startByte must be a UTF-8 character boundary within the chunk.");let _=p.subarray(s).toString("utf8"),T=tt(_,a),E=it(T.text,d,l!==void 0),h=E.truncated&&(l===void 0||E.text.length<_.length)?"lines":T.truncated?"bytes":null,f=m.totalChunks;return{namespace:r,result:{text:E.text,truncated:h!==null,totalChunks:f,nextChunkOrdinal:t+1<f?t+1:null,chunkBytes:Buffer.byteLength(m.text,"utf8"),returnedBytes:Buffer.byteLength(E.text,"utf8"),returnedLines:rt(E.text)-(l!==void 0&&/[\r\n]$/.test(E.text)?1:0),truncatedBy:h,nextByteOffset:h===null?null:s+Buffer.byteLength(E.text,"utf8")}}}catch(o){let m=o instanceof Error?o.message:String(o);return $(r,"object_read_failed",m)}finally{c?.close()}}import*as q from"node:fs";import*as W from"node:path";import{createHash as Re}from"node:crypto";import{DatabaseSync as Ie}from"node:sqlite";var K=8192,ve=3e4,ot=100;async function Se(e,n,r){let i=AbortSignal.timeout(ve),t=r?AbortSignal.any([r,i]):i,a=await fetch("http://127.0.0.1:11434/api/embed",{method:"POST",headers:{"content-type":"application/json"},body:JSON.stringify({model:e,input:n,truncate:!1}),signal:t,redirect:"error"});if(!a.ok)throw new Error(`Local embedding service returned HTTP ${a.status}.`);let d=await a.json();if(!d||typeof d!="object"||!("embeddings"in d))throw new Error("Embedding response is invalid.");return d.embeddings}function Oe(e){let n;try{n=new URL(e)}catch{throw new Error("invalid_azure_endpoint")}let r=/^[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.(?:openai\.azure\.com|services\.ai\.azure\.com|cognitiveservices\.azure\.com)$/i.test(n.hostname),i=["","/","/openai/v1","/openai/v1/"].includes(n.pathname);if(n.protocol!=="https:"||!r||!i||n.username||n.password||n.port||n.search||n.hash)throw new Error("invalid_azure_endpoint");return`${n.origin}/openai/v1/embeddings`}function at(e,n){let r=Oe(e);if(!n.trim())throw new Error("missing_azure_api_key");return async(i,t,a)=>{let d=AbortSignal.timeout(ve),u=a?AbortSignal.any([a,d]):d,l=await fetch(r,{method:"POST",headers:{"content-type":"application/json","api-key":n},body:JSON.stringify({model:i,input:t,encoding_format:"float"}),signal:u,redirect:"error"});if(!l.ok)throw new Error("Azure embedding request failed.");let c=await l.json();if(!c||typeof c!="object"||!("data"in c)||!Array.isArray(c.data)||c.data.length!==t.length)throw new Error("Invalid Azure embedding response.");let o=new Array(t.length);for(let m of c.data){if(!m||typeof m!="object"||!Number.isInteger(m.index)||m.index<0||m.index>=t.length||o[m.index]!==void 0||!Array.isArray(m.embedding))throw new Error("Invalid Azure embedding response.");o[m.index]=m.embedding}return o}}async function Te(e,n=process.env,r){let i=n.RAG_EMBEDDING_PROVIDER??"ollama",t=n.RAG_EMBEDDING_BATCH_SIZE===void 0?8:Number(n.RAG_EMBEDDING_BATCH_SIZE);if(!Number.isInteger(t)||t<1||t>32)return{namespace:e.namespace,error:{code:"invalid_embedding_batch_size",message:"RAG_EMBEDDING_BATCH_SIZE must be an integer from 1 to 32."}};let a=n.RAG_EMBEDDING_BATCH_BYTES===void 0?65536:Number(n.RAG_EMBEDDING_BATCH_BYTES);if(!Number.isInteger(a)||a<1024||a>1048576)return{namespace:e.namespace,error:{code:"invalid_embedding_batch_bytes",message:"RAG_EMBEDDING_BATCH_BYTES must be an integer from 1024 to 1048576."}};let d=n.RAG_EMBEDDING_REVISION??"";if(i==="ollama")return Le({...e,model:n.RAG_EMBEDDING_MODEL??"",cacheIdentity:JSON.stringify(["ollama",n.RAG_EMBEDDING_MODEL??"",d]),batchSize:t,maxBatchBytes:a},Se,r);if(i!=="azure")return{namespace:e.namespace,error:{code:"invalid_embedding_provider",message:"RAG_EMBEDDING_PROVIDER must be ollama or azure."}};let u=n.RAG_AZURE_OPENAI_ENDPOINT,l=n.RAG_AZURE_OPENAI_API_KEY,c=n.RAG_AZURE_OPENAI_DEPLOYMENT;if(!u||!l||!c?.trim())return{namespace:e.namespace,error:{code:"semantic_not_configured",message:"Azure embeddings require endpoint, deployment and API key configuration."}};let o;try{o=at(u,l)}catch{return{namespace:e.namespace,error:{code:"invalid_azure_endpoint",message:"Azure endpoint must be a supported HTTPS resource URL."}}}return Le({...e,model:c,cacheIdentity:JSON.stringify(["azure",Oe(u),c,d]),batchSize:t,maxBatchBytes:a},o,r)}function ge(e,n){let r=0,i=0,t=0;for(let a=0;a<e.length;a+=1)r+=e[a]*n[a],i+=e[a]**2,t+=n[a]**2;if(!i||!t||!Number.isFinite(r)||!Number.isFinite(i)||!Number.isFinite(t))throw new Error("Embedding vector must be finite and nonzero.");return r/Math.sqrt(i*t)}async function Le(e,n=Se,r){let{root:i,namespace:t,query:a,includeHistorical:d,excerptLength:u,model:l}=e,c=e.batchSize??8,o=e.maxBatchBytes??65536;if(!l.trim())return{namespace:t,error:{code:"semantic_not_configured",message:"Configure an embedding model."}};if(!Number.isInteger(c)||c<1||c>32)return{namespace:t,error:{code:"invalid_embedding_batch_size",message:"Embedding batch size must be an integer from 1 to 32."}};if(!Number.isInteger(o)||o<1024||o>1048576)return{namespace:t,error:{code:"invalid_embedding_batch_bytes",message:"Embedding batch bytes must be an integer from 1024 to 1048576."}};if(a.length>K)return{namespace:t,error:{code:"semantic_query_too_large",message:`Semantic queries must not exceed ${K} characters.`}};let m,p,s="database";try{let _=x(i,t.client);if(!q.existsSync(_)||!q.statSync(_).isFile())return{namespace:t,error:{code:"database_not_found",message:"Namespace database was not found."}};m=new Ie(_,{readOnly:!0});let E=m.prepare(`SELECT dv.doc_version_id AS docVersionId, d.document_id AS documentId,
29
+ e.entity_key AS entityKey, d.label AS label, dv.version_label AS versionLabel, dv.is_current AS isCurrent,
30
+ s.ordinal AS sectionOrdinal, c.ordinal AS chunkOrdinal, c.char_start AS charStart, c.char_end AS charEnd, c.text AS text
31
+ FROM chunk AS c JOIN doc_version AS dv ON dv.doc_version_id = c.doc_version_id
32
+ JOIN document AS d ON d.document_id = dv.document_id JOIN entity AS e ON e.entity_id = d.entity_id
33
+ JOIN project AS p ON p.project_id = d.project_id JOIN client AS cl ON cl.client_id = p.client_id
34
+ LEFT JOIN section AS s ON s.section_id = c.section_id
35
+ WHERE cl.name = ? AND p.slug = ? AND d.deleted_at IS NULL AND e.deleted_at IS NULL AND (dv.is_current = 1 OR ? = 1)
36
+ ORDER BY dv.doc_version_id COLLATE BINARY, c.ordinal`).iterate(t.client,t.project,d?1:0)[Symbol.iterator](),y=E.next();if(y.done)return{namespace:t,result:{citations:[],total:0,searchMode:"semantic"}};s="cache",p=new Ie(W.join(W.dirname(_),"semantic-cache.sqlite")),p.exec("PRAGMA busy_timeout = 5000; PRAGMA journal_mode = WAL;"),p.exec(`CREATE TABLE IF NOT EXISTS vector_cache (
37
+ project TEXT NOT NULL, identity TEXT NOT NULL, doc_version_id TEXT NOT NULL, ordinal INTEGER NOT NULL,
38
+ text_hash TEXT NOT NULL, dimensions INTEGER NOT NULL, vector BLOB NOT NULL,
39
+ PRIMARY KEY(project, identity, doc_version_id, ordinal)
40
+ ) WITHOUT ROWID`),p.exec("CREATE INDEX IF NOT EXISTS idx_vector_content ON vector_cache(project, identity, text_hash)");let h=Re("sha256").update(e.cacheIdentity??JSON.stringify(["direct",l])).digest("hex"),f=p.prepare("SELECT dimensions, vector FROM vector_cache WHERE project = ? AND identity = ? AND doc_version_id = ? AND ordinal = ? AND text_hash = ?"),g=p.prepare("SELECT dimensions, vector FROM vector_cache WHERE project = ? AND identity = ? AND text_hash = ? LIMIT 1"),b=p.prepare(`INSERT INTO vector_cache(project, identity, doc_version_id, ordinal, text_hash, dimensions, vector) VALUES (?, ?, ?, ?, ?, ?, ?)
41
+ ON CONFLICT(project, identity, doc_version_id, ordinal) DO UPDATE SET text_hash=excluded.text_hash, dimensions=excluded.dimensions, vector=excluded.vector`),L=(N,v,O)=>{if(!Array.isArray(N)||N.length!==v||!Array.isArray(N[0]))throw new Error("Invalid vector count.");let S=O??N[0].length;if(S<1||S>8192||N.some(B=>!Array.isArray(B)||B.length!==S||B.some(U=>typeof U!="number"||!Number.isFinite(U))))throw new Error("Invalid vector dimensions.");return S};s="embedding";let I=await n(l,[a],r),R=L(I,1),A=I[0];ge(A,A);let C=[],ye=0,be=(N,v)=>{C.push({docVersionId:N.docVersionId,documentId:N.documentId,entityKey:N.entityKey,label:N.label,versionLabel:N.versionLabel,isCurrent:N.isCurrent===1,sectionOrdinal:N.sectionOrdinal,chunkOrdinal:N.chunkOrdinal,charStart:N.charStart,charEnd:N.charEnd,matchStart:null,matchEnd:null,score:ge(A,v),excerpt:N.text.slice(0,u)}),C.sort((O,S)=>S.score-O.score||O.docVersionId.localeCompare(S.docVersionId)||O.chunkOrdinal-S.chunkOrdinal),C.length>ot&&C.pop()},j=[],le=0,me=async()=>{if(!j.length)return;s="embedding";let N=await n(l,j.map(v=>v.row.text),r);if(L(N,j.length,R),r?.aborted)throw new Error("cancelled");for(let v of N)ge(v,v);s="cache",p.exec("BEGIN IMMEDIATE");try{for(let[v,O]of j.entries()){let S=N[v],B=Buffer.allocUnsafe(R*4);S.forEach((U,pe)=>{if(!Number.isFinite(Math.fround(U)))throw new Error("Embedding value exceeds float32.");B.writeFloatLE(U,pe*4)}),b.run(t.project,h,O.row.docVersionId,O.row.chunkOrdinal,O.hash,R,B),be(O.row,Array.from({length:R},(U,pe)=>B.readFloatLE(pe*4)))}p.exec("COMMIT")}catch(v){throw p.exec("ROLLBACK"),v}j=[],le=0};for(let N of{*[Symbol.iterator](){yield y.value,yield*{[Symbol.iterator]:()=>E}}}){if(r?.aborted)throw new Error("cancelled");if(N.text.length>K)return{namespace:t,error:{code:"semantic_chunk_too_large",message:`A chunk exceeds ${K} characters; semantic search did not finish.`}};ye+=1;let v=Re("sha256").update(N.text).digest("hex");s="cache";let O=f.get(t.project,h,N.docVersionId,N.chunkOrdinal,v)??g.get(t.project,h,v);if(O){if(O.dimensions!==R||O.vector.byteLength!==R*4)return{namespace:t,error:{code:"embedding_dimension_mismatch",message:"Cached vectors do not match this model. Set a new RAG_EMBEDDING_REVISION before rebuilding."}};let S=Buffer.from(O.vector);be(N,Array.from({length:R},(B,U)=>S.readFloatLE(U*4)))}else{let S=Buffer.byteLength(N.text,"utf8");if(S>o)return{namespace:t,error:{code:"semantic_chunk_too_large",message:"A chunk exceeds RAG_EMBEDDING_BATCH_BYTES."}};le+S>o&&await me(),j.push({row:N,hash:v}),le+=S,j.length>=c&&await me()}}return await me(),{namespace:t,result:{citations:C,total:ye,searchMode:"semantic"}}}catch{return r?.aborted?{namespace:t,error:{code:"cancelled",message:"Search was cancelled."}}:s==="embedding"?{namespace:t,error:{code:"embedding_unavailable",message:"Embedding request failed; check the configured provider, model and access."}}:s==="cache"?{namespace:t,error:{code:"semantic_cache_unavailable",message:"Semantic cache could not be read or written."}}:{namespace:t,error:{code:"database_unavailable",message:"Could not load namespace chunks."}}}finally{p?.close(),m?.close()}}function Ne(e,n){let r=new Map;for(let i of[e,n])i.forEach((t,a)=>{let d=JSON.stringify([t.docVersionId,t.chunkOrdinal]),u=r.get(d);u?u.score+=1/(60+a+1):r.set(d,{citation:t,score:1/(60+a+1)})});return[...r.values()].map(({citation:i,score:t})=>({...i,score:t})).sort((i,t)=>t.score-i.score||i.docVersionId.localeCompare(t.docVersionId)||i.chunkOrdinal-t.chunkOrdinal)}import*as V from"node:fs";import*as Z from"node:path";import{execFile as st}from"node:child_process";import{fileURLToPath as ct}from"node:url";function dt(){let e=Z.dirname(ct(import.meta.url));for(let n of["../bin/rag.mjs","../../bin/rag.mjs","../../../bin/rag.mjs"]){let r=Z.resolve(e,n);if(V.existsSync(r)&&V.statSync(r).isFile())return r}throw new Error("The packaged rag CLI could not be found.")}async function we(e,n,r,i){if(["status","ls","review","doctor","rm","backup"].includes(i[0]??"")&&!V.existsSync(x(n,r.client)))return{namespace:r,error:{code:"database_not_found",message:"Client database was not found."}};let t={...process.env,RAG_FORMAT:"json",RAG_ROOT:n};delete t.RAG_ENTRY;let a=dt();return new Promise((d,u)=>{st(process.execPath,[a,...i,"--root",n,"--client",r.client,"--project",r.project],{cwd:e,env:t,timeout:12e4,maxBuffer:1048576,windowsHide:!0},(l,c)=>{try{let o=JSON.parse(c.trim());if(!o||typeof o!="object"||Array.isArray(o))throw new Error("invalid result");d(o)}catch{u(new Error(l?.killed?"RAG command timed out.":"RAG command failed without a valid result."))}})})}import*as Ce from"node:fs";import*as Q from"node:path";import{DatabaseSync as ut}from"node:sqlite";function xe(e,n){let r=Q.join(Q.dirname(x(e,n.client)),"semantic-cache.sqlite");if(!Ce.existsSync(r))return{storedVectorRecords:0,vectorBytes:0};let i=new ut(r,{readOnly:!0});try{let t=i.prepare("SELECT COUNT(*) AS count, COALESCE(SUM(length(vector)), 0) AS bytes FROM vector_cache WHERE project = ?").get(n.project);return{storedVectorRecords:t.count,vectorBytes:t.bytes}}finally{i.close()}}import{z as w}from"zod";var ke=100,De=2e3,Be=1048576,ee=1e4,lt=10,mt=800,pt=65536,ft=200;function Ue(e){let n=new Set;for(let r of(e??"find,read").split(",").map(i=>i.trim())){if(r!=="find"&&r!=="read")throw new Error(`invalid_capability: Unknown RAG_CAPABILITIES value "${r}".`);n.add(r)}return n}var te=w.string().min(1).refine(e=>e.trim().length>0),re=w.object({client:te,project:te}).strict();function je(e){if(e===void 0)return null;let n;try{n=JSON.parse(e)}catch{throw new Error("invalid_namespace_allowlist: RAG_ALLOWED_NAMESPACES must be a JSON array.")}let r=w.array(re).safeParse(n);if(!r.success)throw new Error("invalid_namespace_allowlist: RAG_ALLOWED_NAMESPACES must be an array of client/project objects.");return r.data}function ie(e,n){return e===null||e.some(r=>r.client===n.client&&r.project===n.project)}function ne(e,n,r){return w.number().int().min(e).max(n).optional().transform(i=>i??r)}var Me=w.object({namespace:re,query:te,includeHistorical:w.boolean().optional().default(!1),matchMode:w.enum(["all","any"]).optional().default("all"),searchMode:w.enum(["lexical","semantic","hybrid"]).optional().default("lexical"),limit:ne(1,ke,lt),excerptChars:ne(1,De,mt)}).strict(),Xe=w.object({namespace:re,docVersionId:te,chunkOrdinal:w.number().multipleOf(1).min(0).refine(Number.isSafeInteger),maxBytes:ne(0,Be,pt),maxLines:ne(0,ee,ft),startLine:w.number().int().min(0).max(ee).optional(),startByte:w.number().int().min(0).max(Number.MAX_SAFE_INTEGER).optional()}).strict().refine(e=>e.startLine===void 0||e.startByte===void 0,{path:["startByte"],message:"startByte and startLine cannot be combined."});function Et(e){let n=e.find(a=>a.code==="unrecognized_keys"&&a.path.length===0)??e.find(a=>a.code==="unrecognized_keys");if(!n||n.code!=="unrecognized_keys")return null;let r=n.keys[0];if(r===void 0)return null;let i=n.path.map(String).join(".");return{code:"unknown_field",message:`Unknown field "${i.length>0?`${i}.${r}`:r}".`}}function k(e,n){return e.some(r=>r.path[0]===n)}function F(e,n,r){return{code:"invalid_limit",message:`${e} must be an integer from ${n} to ${r}.`}}function Fe(e,n){let r=Et(e);if(r)return r;if(k(e,"namespace"))return{code:"namespace_required",message:"namespace must contain non-empty client and project strings."};if(n==="find"){if(k(e,"query"))return{code:"invalid_query",message:"query must be a non-empty string."};if(k(e,"limit"))return F("limit",1,ke);if(k(e,"excerptChars"))return F("excerptChars",1,De)}else{if(e.some(i=>i.path[0]==="startByte"&&i.code==="custom"))return{code:"invalid_input",message:"startByte and startLine cannot be combined."};if(k(e,"docVersionId"))return{code:"invalid_input",message:"docVersionId must be a non-empty string."};if(k(e,"chunkOrdinal"))return{code:"invalid_input",message:"chunkOrdinal must be a non-negative integer."};if(k(e,"maxBytes"))return F("maxBytes",0,Be);if(k(e,"maxLines"))return F("maxLines",0,ee);if(k(e,"startLine"))return F("startLine",0,ee);if(k(e,"startByte"))return F("startByte",0,Number.MAX_SAFE_INTEGER)}return e.some(i=>i.path.length===0)?{code:"invalid_input",message:"Tool arguments must be an object."}:{code:"invalid_input",message:"Tool arguments are invalid."}}function _t(e){let n=re.safeParse(e);return n.success?n.data:null}function Ge(e){let n=Me.safeParse(e);return n.success?{ok:!0,input:n.data,namespace:n.data.namespace}:{ok:!1,namespace:oe(e),error:Fe(n.error.issues,"find")}}function Pe(e){let n=Xe.safeParse(e);return n.success?{ok:!0,input:n.data,namespace:n.data.namespace}:{ok:!1,namespace:oe(e),error:Fe(n.error.issues,"read")}}function ht(e){return typeof e=="object"&&e!==null&&!Array.isArray(e)}function oe(e){return ht(e)?_t(e.namespace):null}function $e(e){return w.toJSONSchema(e,{io:"input"})}var ae={name:"rag_find",description:"Search current document versions. Semantic or hybrid search sends indexed text to the explicitly configured provider; use matchMode any for broader lexical matching or includeHistorical for older versions.",inputSchema:$e(Me)},se={name:"rag_read",description:"Read bounded text from a cited document chunk in the explicit namespace. For exact pagination use startByte: 0, modest maxBytes and positive maxLines; repeat with nextByteOffset until null, concatenating text directly. Do not combine startByte and startLine.",inputSchema:$e(Xe)};function ue(e,n=!1){return{content:[{type:"text",text:JSON.stringify(e)}],details:e,...n?{isError:!0}:{}}}function D(e,n){return ue({namespace:e,error:n},!0)}function ce(e){return D(e,{code:"cancelled",message:"Tool call was cancelled."})}function Ve(e,n){let r=n instanceof Error?n.message:String(n);return D(oe(e),{code:"retrieval_failed",message:r})}var gt="The namespace travels on every call: never infer `client` or `project` from the cwd, repo name, config, env, or a placeholder like `default`. If the user has not stated it, ask before the first call.\nUse the RAG tool the role actually exposes; when only `rag_find` is available, delegate for deeper text instead of paraphrasing a catalog entry.\nWhen an excerpt is truncated, when the user asks whether the document covers something, or when a longer verbatim quote is needed, hand off to the subagent that exposes `rag_read` with the namespace and the citation's `docVersionId`; never assert absence from a catalog.";function Zt(e){e.on("before_agent_start",i=>{if(process.env.RAG_PROMPT==="off")return;let t=i.systemPromptOptions.sections??(i.systemPromptOptions.sections={});t.rag=gt});let n=Ue(process.env.RAG_CAPABILITIES),r=je(process.env.RAG_ALLOWED_NAMESPACES);e.registerCommand("rag",{description:"Search the document knowledge base",async handler(i,t){if(!t.hasUI)return;if(!n.has("find")){t.ui.notify("RAG search is not enabled for this session.","warning");return}let a=(await t.ui.input("RAG client"))?.trim();if(!a)return;let d=(await t.ui.input("RAG project"))?.trim();if(!d)return;let u={client:a,project:d};if(!ie(r,u)){t.ui.notify("Namespace is not enabled for this process.","error");return}let l=process.env.RAG_TUI_ADMIN==="1",c=["Search","Status","Documents","Review queue","Doctor",...l?["Import file","Import directory","Retire document","Backup","Restore"]:[]],o=await t.ui.select(`RAG \xB7 ${a} / ${d}`,c);if(!o||!c.includes(o))return;if(o!=="Search"){try{let T=de.resolve(t.cwd,G({cwd:t.cwd,homeDir:t.cwd,argv:[],env:process.env}).root),E=async f=>{let g=await we(t.cwd,T,u,f);if(g.error&&typeof g.error=="object"){let b=g.error;return t.ui.notify(`${b.code??"error"}: ${b.message??"Command failed."}`,"error"),null}return g.result&&typeof g.result=="object"?g.result:{}};if(o==="Backup"||o==="Restore"){let f=(await t.ui.input(o==="Backup"?"New backup destination":"Existing backup directory"))?.trim();if(!f)return;let g=de.resolve(t.cwd,f),b=o==="Backup"?`Source: ${T}
42
+ New destination: ${g}`:`Source: ${g}
43
+ Restore to empty root: ${T}`;if(!await t.ui.confirm(o,`${b}
44
+ Namespace: ${a} / ${d}`))return;let L=await E(o==="Backup"?["backup","--to",g]:["restore","--from",g]);L&&t.ui.notify(JSON.stringify(L),"info");return}if(o==="Import file"||o==="Import directory"){let f=(await t.ui.input(o==="Import file"?"File path":"Directory path"))?.trim();if(!f)return;let g=de.resolve(t.cwd,f);if(!await t.ui.confirm(o,`${g}
45
+ Destination: ${a} / ${d}
46
+ Root: ${T}`))return;let b=await E([o==="Import file"?"add":"import",g]);b&&t.ui.notify(JSON.stringify(b),"info");return}if(o==="Retire document"){let f=await E(["ls"]);if(!f||!Array.isArray(f.documents)||!f.documents.length){f&&t.ui.notify("No active documents.","info");return}let g=f.documents,b=g.map((A,C)=>`${C+1}. ${A.label} \xB7 ${A.documentId}`),L=await t.ui.select("Retire a document",b);if(!L)return;let I=g[b.indexOf(L)];if(!I||!await t.ui.confirm("Retire document",`${I.label}
47
+ ${I.documentId}
48
+ ${a} / ${d}`))return;let R=await E(["rm",I.documentId]);R&&t.ui.notify(JSON.stringify(R),"info");return}let h=await E(o==="Status"?["status"]:o==="Documents"?["ls"]:o==="Review queue"?["review","list"]:["doctor"]);if(!h)return;if(o==="Status"){t.ui.notify(JSON.stringify({...h,semanticCache:xe(T,u)}),"info");return}if(o==="Documents"&&Array.isArray(h.documents)){let f=h.documents;if(!f.length){t.ui.notify("No active documents.","info");return}await t.ui.select(`Documents (${f.length})`,f.map((g,b)=>`${b+1}. ${g.label} \xB7 ${g.versionLabel??"current"} \xB7 ${g.documentId}`));return}if(o==="Review queue"&&Array.isArray(h.candidates)){let f=h.candidates;if(!f.length){t.ui.notify("No review candidates.","info");return}let g=f.map((A,C)=>`${C+1}. ${A.fromDocVersionId} \u2194 ${A.toDocVersionId}`),b=await t.ui.select(`Review candidates (${f.length})`,g);if(!b||!l)return;let L=f[g.indexOf(b)];if(!L)return;let I=await t.ui.select("Human review decision",["same_entity","conflicts_with"]);if(!I||!["same_entity","conflicts_with"].includes(I)||!await t.ui.confirm("Record review decision",`${L.fromDocVersionId}
49
+ ${L.toDocVersionId}
50
+ ${I}
51
+ ${a} / ${d}`))return;let R=await E(["review","link",L.fromDocVersionId,L.toDocVersionId,"--type",I]);R&&t.ui.notify(JSON.stringify(R),"info");return}t.ui.notify(JSON.stringify(h),"info")}catch(T){t.ui.notify(T instanceof Error?T.message:String(T),"error")}return}let m=(await t.ui.input("Search documents"))?.trim();if(!m)return;let p=await t.ui.select("Retrieval method",["Lexical","Semantic","Hybrid"]);if(!p||!["Lexical","Semantic","Hybrid"].includes(p))return;let s=await t.ui.select("Match query terms",["All terms","Any term"]);if(!s||!["All terms","Any term"].includes(s))return;let _=await t.ui.select("Document versions",["Current versions","Include historical versions"]);if(!(!_||!["Current versions","Include historical versions"].includes(_)))try{let T=G({cwd:t.cwd,homeDir:t.cwd,argv:[],env:process.env}).root,E=_==="Include historical versions",y=s==="Any term"?"any":"all",h=p==="Lexical"?P({root:T,namespace:u,query:m,excerptLength:240,matchMode:y,includeHistorical:E,maxResults:10}):await Te({root:T,namespace:u,query:m,excerptLength:240,includeHistorical:E});if("error"in h&&h.error){t.ui.notify(`${h.error.code}: ${h.error.message}`,"error");return}let f=h.result.citations;if(p==="Hybrid"){let A=P({root:T,namespace:u,query:m,excerptLength:240,matchMode:y,includeHistorical:E,maxResults:100});if("error"in A){t.ui.notify(`${A.error.code}: ${A.error.message}`,"error");return}f=Ne(A.result.citations,f)}let g=f.slice(0,10);if(g.length===0){t.ui.notify("No matches. This does not establish that the information is absent.","info");return}let b=g.map((A,C)=>`${C+1}. ${A.label} [${A.versionLabel??A.docVersionId}; ${A.isCurrent?"current":"historical"}] \u2014 ${A.excerpt.replace(/\s+/g," ")}`),L=await t.ui.select(`RAG results (${h.result.total} matches)`,b);if(!L)return;let I=g[b.indexOf(L)];if(!I)return;if(!n.has("read")){t.ui.notify("Full reading is not enabled for this session.","info");return}let R=await he({root:T,namespace:u,docVersionId:I.docVersionId,chunkOrdinal:I.chunkOrdinal,maxBytes:4096,maxLines:40});"error"in R?t.ui.notify(`${R.error.code}: ${R.error.message}`,"error"):t.ui.notify(`${I.label} \xB7 ${I.docVersionId}
52
+ ${R.result.text}${R.result.truncated?`
53
+ [truncated]`:""}`,"info")}catch(T){t.ui.notify(T instanceof Error?T.message:String(T),"error")}}}),n.has("find")&&e.registerTool({name:ae.name,label:"RAG Find",description:ae.description,parameters:ae.inputSchema,async execute(i,t,a,d,u){try{let l=Ge(t);if(!l.ok)return D(l.namespace,l.error);let{namespace:c,query:o,limit:m,excerptChars:p,includeHistorical:s,matchMode:_,searchMode:T}=l.input;if(!ie(r,c))return D(c,{code:"namespace_denied",message:"Namespace is not enabled for this process."});if(a?.aborted)return ce(c);let E=G({cwd:u.cwd,homeDir:u.cwd,argv:[],env:process.env}).root,y=T!=="lexical"?await Te({root:E,namespace:c,query:o,excerptLength:p,includeHistorical:s},process.env,a):P({root:E,namespace:c,query:o,excerptLength:p,includeHistorical:s,matchMode:_,maxResults:m});if(a?.aborted)return ce(c);if("error"in y&&y.error)return D(c,y.error);if(T==="hybrid"){let h=P({root:E,namespace:c,query:o,excerptLength:p,includeHistorical:s,matchMode:_,maxResults:100});if("error"in h)return D(c,h.error);let f=Ne(h.result.citations,y.result.citations);return ue({namespace:c,result:{citations:f.slice(0,m),total:y.result.total,searchMode:"hybrid"}})}return ue({namespace:c,result:{...y.result,citations:y.result.citations.slice(0,m)}})}catch(l){return Ve(t,l)}}}),n.has("read")&&e.registerTool({name:se.name,label:"RAG Read",description:se.description,parameters:se.inputSchema,async execute(i,t,a,d,u){try{let l=Pe(t);if(!l.ok)return D(l.namespace,l.error);let{namespace:c,docVersionId:o,chunkOrdinal:m,maxBytes:p,maxLines:s,startLine:_,startByte:T}=l.input;if(!ie(r,c))return D(c,{code:"namespace_denied",message:"Namespace is not enabled for this process."});if(a?.aborted)return ce(c);let E=G({cwd:u.cwd,homeDir:u.cwd,argv:[],env:process.env}).root,y=await he({root:E,namespace:c,docVersionId:o,chunkOrdinal:m,maxBytes:p,maxLines:s,startLine:_,startByte:T});return a?.aborted?ce(c):"error"in y?D(c,y.error):ue({namespace:c,result:y.result})}catch(l){return Ve(t,l)}}})}export{Zt as default};
@@ -6,13 +6,21 @@ const JPEG_MEDIA_TYPE = "image/jpeg";
6
6
  const IMAGE_MEDIA_TYPES = /* @__PURE__ */ new Set([PNG_MEDIA_TYPE, JPEG_MEDIA_TYPE]);
7
7
  const IMAGE_EXTENSION = /\.(?:png|jpe?g)$/i;
8
8
  const MAX_IMAGE_PIXELS = 5e7;
9
- const englishLanguageData = createRequire(import.meta.url)("@tesseract.js-data/eng");
9
+ const require2 = createRequire(import.meta.url);
10
+ const englishLanguageData = require2("@tesseract.js-data/eng");
11
+ const spanishLanguageData = require2("@tesseract.js-data/spa");
10
12
  const OCR_LANGUAGE_DATA_PATH = englishLanguageData.langPath;
11
13
  const OCR_WORKER_OPTIONS = {
12
14
  langPath: OCR_LANGUAGE_DATA_PATH,
13
15
  cacheMethod: "none",
14
16
  gzip: englishLanguageData.gzip
15
17
  };
18
+ function resolveOcrConfig(value = process.env.RAG_OCR_LANGUAGE) {
19
+ const language = value ?? "eng";
20
+ if (language !== "eng" && language !== "spa") throw new Error(`Unsupported RAG_OCR_LANGUAGE: ${language}; expected eng or spa.`);
21
+ const data = language === "spa" ? spanishLanguageData : englishLanguageData;
22
+ return { language, options: { langPath: data.langPath, cacheMethod: "none", gzip: data.gzip } };
23
+ }
16
24
  function supports(ref) {
17
25
  if (typeof ref === "string") {
18
26
  const mediaType2 = ref.split(";")[0]?.trim().toLowerCase();
@@ -73,7 +81,8 @@ async function extract(bytes) {
73
81
  if (width === 0 || height === 0 || width > MAX_IMAGE_PIXELS / height) {
74
82
  throw new ResourceLimitError("image_too_large", `Image exceeds MAX_IMAGE_PIXELS (${MAX_IMAGE_PIXELS} pixels).`);
75
83
  }
76
- const worker = await Tesseract.createWorker("eng", Tesseract.OEM.LSTM_ONLY, OCR_WORKER_OPTIONS);
84
+ const { language, options } = resolveOcrConfig();
85
+ const worker = await Tesseract.createWorker(language, Tesseract.OEM.LSTM_ONLY, options);
77
86
  try {
78
87
  await worker.setParameters({ tessedit_pageseg_mode: Tesseract.PSM.SINGLE_BLOCK });
79
88
  const { data } = await worker.recognize(Buffer.from(bytes));
@@ -89,7 +98,9 @@ async function extract(bytes) {
89
98
  }
90
99
  const imageOcrExtractor = {
91
100
  name: "image-ocr",
92
- version: "7.0.0",
101
+ get version() {
102
+ return `7.0.0-${resolveOcrConfig().language}`;
103
+ },
93
104
  mediaTypes: [PNG_MEDIA_TYPE, JPEG_MEDIA_TYPE],
94
105
  supports,
95
106
  extract
@@ -98,5 +109,6 @@ export {
98
109
  MAX_IMAGE_PIXELS,
99
110
  OCR_LANGUAGE_DATA_PATH,
100
111
  OCR_WORKER_OPTIONS,
101
- imageOcrExtractor
112
+ imageOcrExtractor,
113
+ resolveOcrConfig
102
114
  };
@@ -1,7 +1,7 @@
1
1
  import { createCanvas } from "@napi-rs/canvas";
2
2
  import Tesseract from "tesseract.js";
3
3
  import { getDocument } from "pdfjs-dist/legacy/build/pdf.mjs";
4
- import { OCR_WORKER_OPTIONS } from "./image-ocr.mjs";
4
+ import { resolveOcrConfig } from "./image-ocr.mjs";
5
5
  import { MAX_OCR_PDF_PAGES, MAX_PDF_OCR_RENDER_PIXELS, MAX_PDF_SOURCE_IMAGE_PIXELS } from "../limits.mjs";
6
6
  import { ResourceLimitError } from "../ports/extractor.mjs";
7
7
  const PDF_MEDIA_TYPE = "application/pdf";
@@ -29,51 +29,55 @@ async function extract(bytes) {
29
29
  if (document.numPages > MAX_PDF_PAGES) {
30
30
  throw new ResourceLimitError("too_many_pages", `PDF exceeds MAX_PDF_PAGES (${MAX_PDF_PAGES} pages).`);
31
31
  }
32
- const sections = [];
32
+ const pages = [];
33
33
  for (let pageNumber = 1; pageNumber <= document.numPages; pageNumber += 1) {
34
34
  const page = await document.getPage(pageNumber);
35
35
  const textContent = await page.getTextContent();
36
36
  const text = textContent.items.filter((item) => "str" in item).map((item) => "str" in item ? item.str : "").join(" ").replace(/\s+/g, " ").trim();
37
- if (text.length > 0) {
38
- sections.push({ ordinal: sections.length, heading: null, level: null, text });
39
- }
37
+ pages.push({ text, ocr: false });
40
38
  }
41
- if (sections.length > 0) return { sections, versionLabel: null, mediaType: PDF_MEDIA_TYPE };
42
- if (document.numPages > MAX_OCR_PDF_PAGES) {
39
+ const pagesNeedingOcr = pages.flatMap((page, index) => page.text.length === 0 ? [index] : []);
40
+ if (pagesNeedingOcr.length > MAX_OCR_PDF_PAGES) {
43
41
  throw new ResourceLimitError(
44
42
  "too_many_pages",
45
43
  `PDF exceeds MAX_OCR_PDF_PAGES (${MAX_OCR_PDF_PAGES} pages) for OCR.`
46
44
  );
47
45
  }
48
- const worker = await Tesseract.createWorker("eng", Tesseract.OEM.LSTM_ONLY, OCR_WORKER_OPTIONS);
49
- try {
50
- await worker.setParameters({ tessedit_pageseg_mode: Tesseract.PSM.SINGLE_BLOCK });
51
- for (let pageNumber = 1; pageNumber <= document.numPages; pageNumber += 1) {
52
- const page = await document.getPage(pageNumber);
53
- const viewport = page.getViewport({ scale: PDF_OCR_RENDER_SCALE });
54
- const width = Math.ceil(viewport.width);
55
- const height = Math.ceil(viewport.height);
56
- if (width === 0 || height === 0 || width > MAX_PDF_OCR_RENDER_PIXELS / height) {
57
- throw new ResourceLimitError(
58
- "image_too_large",
59
- `PDF page exceeds MAX_PDF_OCR_RENDER_PIXELS (${MAX_PDF_OCR_RENDER_PIXELS} pixels).`
60
- );
61
- }
62
- const canvas = createCanvas(width, height);
63
- const canvasContext = canvas.getContext("2d");
64
- await page.render({
65
- canvas,
66
- canvasContext,
67
- viewport
68
- }).promise;
69
- const { data } = await worker.recognize(canvas.toBuffer("image/png"));
70
- const text = data.text.replace(/\s+/g, " ").trim();
71
- if (text.length > 0) {
72
- sections.push({ ordinal: pageNumber - 1, heading: `Page ${pageNumber}`, level: null, text });
46
+ if (pagesNeedingOcr.length > 0) {
47
+ const { language, options } = resolveOcrConfig();
48
+ const worker = await Tesseract.createWorker(language, Tesseract.OEM.LSTM_ONLY, options);
49
+ try {
50
+ await worker.setParameters({ tessedit_pageseg_mode: Tesseract.PSM.SINGLE_BLOCK });
51
+ for (const pageIndex of pagesNeedingOcr) {
52
+ const pageNumber = pageIndex + 1;
53
+ const page = await document.getPage(pageNumber);
54
+ const viewport = page.getViewport({ scale: PDF_OCR_RENDER_SCALE });
55
+ const width = Math.ceil(viewport.width);
56
+ const height = Math.ceil(viewport.height);
57
+ if (width === 0 || height === 0 || width > MAX_PDF_OCR_RENDER_PIXELS / height) {
58
+ throw new ResourceLimitError(
59
+ "image_too_large",
60
+ `PDF page exceeds MAX_PDF_OCR_RENDER_PIXELS (${MAX_PDF_OCR_RENDER_PIXELS} pixels).`
61
+ );
62
+ }
63
+ const canvas = createCanvas(width, height);
64
+ const canvasContext = canvas.getContext("2d");
65
+ await page.render({
66
+ canvas,
67
+ canvasContext,
68
+ viewport
69
+ }).promise;
70
+ const { data } = await worker.recognize(canvas.toBuffer("image/png"));
71
+ const text = data.text.replace(/\s+/g, " ").trim();
72
+ pages[pageIndex] = { text, ocr: true };
73
73
  }
74
+ } finally {
75
+ await worker.terminate();
74
76
  }
75
- } finally {
76
- await worker.terminate();
77
+ }
78
+ const sections = [];
79
+ for (const [index, page] of pages.entries()) {
80
+ if (page.text.length > 0) sections.push({ ordinal: sections.length, heading: page.ocr ? `Page ${index + 1}` : null, level: null, text: page.text });
77
81
  }
78
82
  return { sections, versionLabel: null, mediaType: PDF_MEDIA_TYPE };
79
83
  } finally {
@@ -82,7 +86,9 @@ async function extract(bytes) {
82
86
  }
83
87
  const pdfExtractor = {
84
88
  name: "pdf",
85
- version: "6.4.299",
89
+ get version() {
90
+ return `6.4.299-mixed-pages-1-${resolveOcrConfig().language}`;
91
+ },
86
92
  mediaTypes: [PDF_MEDIA_TYPE],
87
93
  supports,
88
94
  extract
@@ -0,0 +1,18 @@
1
+ import * as fs from "node:fs";
2
+ import * as path from "node:path";
3
+ import { DatabaseSync } from "node:sqlite";
4
+ import { clientDbPath } from "../store/db.mjs";
5
+ function cacheStatus(root, namespace) {
6
+ const file = path.join(path.dirname(clientDbPath(root, namespace.client)), "semantic-cache.sqlite");
7
+ if (!fs.existsSync(file)) return { storedVectorRecords: 0, vectorBytes: 0 };
8
+ const db = new DatabaseSync(file, { readOnly: true });
9
+ try {
10
+ const row = db.prepare("SELECT COUNT(*) AS count, COALESCE(SUM(length(vector)), 0) AS bytes FROM vector_cache WHERE project = ?").get(namespace.project);
11
+ return { storedVectorRecords: row.count, vectorBytes: row.bytes };
12
+ } finally {
13
+ db.close();
14
+ }
15
+ }
16
+ export {
17
+ cacheStatus
18
+ };
@@ -0,0 +1,40 @@
1
+ import * as fs from "node:fs";
2
+ import * as path from "node:path";
3
+ import { execFile } from "node:child_process";
4
+ import { fileURLToPath } from "node:url";
5
+ import { clientDbPath } from "../store/db.mjs";
6
+ function findCli() {
7
+ const here = path.dirname(fileURLToPath(import.meta.url));
8
+ for (const relative of ["../bin/rag.mjs", "../../bin/rag.mjs", "../../../bin/rag.mjs"]) {
9
+ const candidate = path.resolve(here, relative);
10
+ if (fs.existsSync(candidate) && fs.statSync(candidate).isFile()) return candidate;
11
+ }
12
+ throw new Error("The packaged rag CLI could not be found.");
13
+ }
14
+ async function runPiCli(cwd, root, namespace, args) {
15
+ if (["status", "ls", "review", "doctor", "rm", "backup"].includes(args[0] ?? "") && !fs.existsSync(clientDbPath(root, namespace.client))) {
16
+ return { namespace, error: { code: "database_not_found", message: "Client database was not found." } };
17
+ }
18
+ const env = { ...process.env, RAG_FORMAT: "json", RAG_ROOT: root };
19
+ delete env.RAG_ENTRY;
20
+ const cli = findCli();
21
+ return new Promise((resolve, reject) => {
22
+ execFile(
23
+ process.execPath,
24
+ [cli, ...args, "--root", root, "--client", namespace.client, "--project", namespace.project],
25
+ { cwd, env, timeout: 12e4, maxBuffer: 1048576, windowsHide: true },
26
+ (error, stdout) => {
27
+ try {
28
+ const value = JSON.parse(stdout.trim());
29
+ if (!value || typeof value !== "object" || Array.isArray(value)) throw new Error("invalid result");
30
+ resolve(value);
31
+ } catch {
32
+ reject(new Error(error?.killed ? "RAG command timed out." : "RAG command failed without a valid result."));
33
+ }
34
+ }
35
+ );
36
+ });
37
+ }
38
+ export {
39
+ runPiCli
40
+ };
@@ -6,8 +6,8 @@ const WORDS = /[\p{L}\p{M}\p{N}]+/gu;
6
6
  function tokenizeQuery(query) {
7
7
  return query.match(WORDS) ?? [];
8
8
  }
9
- function toFtsExpression(tokens) {
10
- return tokens.map((token) => `"${token.replaceAll('"', '""')}"`).join(" AND ");
9
+ function toFtsExpression(tokens, matchMode) {
10
+ return tokens.map((token) => `"${token.replaceAll('"', '""')}"`).join(matchMode === "any" ? " OR " : " AND ");
11
11
  }
12
12
  function normalizeToken(token) {
13
13
  return token.normalize("NFD").replace(new RegExp("\\p{M}", "gu"), "").toLowerCase();
@@ -62,11 +62,8 @@ function findInNamespace(input) {
62
62
  return failure(namespace, "database_not_found", `No database exists for namespace ${namespace.client}/${namespace.project}.`);
63
63
  }
64
64
  db = new DatabaseSync(dbPath, { readOnly: true });
65
- const rows = db.prepare(`SELECT dv.doc_version_id AS docVersionId, d.document_id AS documentId,
66
- e.entity_key AS entityKey, d.label AS label, dv.version_label AS versionLabel,
67
- s.ordinal AS sectionOrdinal, c.ordinal AS chunkOrdinal, c.text AS text,
68
- c.char_start AS charStart, c.char_end AS charEnd, bm25(chunk_fts) AS score
69
- FROM chunk_fts
65
+ db.exec("BEGIN");
66
+ const fromClause = `FROM chunk_fts
70
67
  JOIN chunk AS c ON c.rowid = chunk_fts.rowid
71
68
  JOIN doc_version AS dv ON dv.doc_version_id = c.doc_version_id
72
69
  JOIN document AS d ON d.document_id = dv.document_id
@@ -75,8 +72,16 @@ function findInNamespace(input) {
75
72
  JOIN client AS cl ON cl.client_id = p.client_id
76
73
  LEFT JOIN section AS s ON s.section_id = c.section_id
77
74
  WHERE chunk_fts MATCH ? AND cl.name = ? AND p.slug = ?
78
- AND d.deleted_at IS NULL AND e.deleted_at IS NULL
79
- ORDER BY score ASC, dv.doc_version_id COLLATE BINARY ASC, c.ordinal ASC`).all(toFtsExpression(tokens), namespace.client, namespace.project);
75
+ AND d.deleted_at IS NULL AND e.deleted_at IS NULL AND (dv.is_current = 1 OR ? = 1)`;
76
+ const parameters = [toFtsExpression(tokens, input.matchMode ?? "all"), namespace.client, namespace.project, input.includeHistorical ? 1 : 0];
77
+ const { total } = db.prepare(`SELECT COUNT(*) AS total ${fromClause}`).get(...parameters);
78
+ const rows = db.prepare(`SELECT dv.doc_version_id AS docVersionId, d.document_id AS documentId,
79
+ e.entity_key AS entityKey, d.label AS label, dv.version_label AS versionLabel, dv.is_current AS isCurrent,
80
+ s.ordinal AS sectionOrdinal, c.ordinal AS chunkOrdinal, c.text AS text,
81
+ c.char_start AS charStart, c.char_end AS charEnd, bm25(chunk_fts) AS score
82
+ ${fromClause}
83
+ ORDER BY score ASC, dv.doc_version_id COLLATE BINARY ASC, c.ordinal ASC LIMIT ?`).all(...parameters, input.maxResults ?? -1);
84
+ db.exec("COMMIT");
80
85
  const citations = [];
81
86
  for (const row of rows) {
82
87
  const range = matchRange(row.text, tokens);
@@ -89,6 +94,7 @@ function findInNamespace(input) {
89
94
  entityKey: row.entityKey,
90
95
  label: row.label,
91
96
  versionLabel: row.versionLabel,
97
+ isCurrent: row.isCurrent === 1,
92
98
  sectionOrdinal: row.sectionOrdinal,
93
99
  chunkOrdinal: row.chunkOrdinal,
94
100
  charStart: row.charStart,
@@ -100,7 +106,7 @@ function findInNamespace(input) {
100
106
  });
101
107
  }
102
108
  citations.sort((left, right) => left.score - right.score || compareText(left.docVersionId, right.docVersionId) || left.chunkOrdinal - right.chunkOrdinal);
103
- return { namespace, result: { citations, total: citations.length } };
109
+ return { namespace, result: { citations, total } };
104
110
  } catch (error) {
105
111
  const message = error instanceof Error ? error.message : String(error);
106
112
  return failure(namespace, "database_unavailable", message);