agent-rag-mcp 0.8.0 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +9 -1
- package/bin/agent-rag.mjs +26 -0
- package/dist/extensions/agent-rag.mjs +18 -18
- package/dist/src/retrieve/read.mjs +16 -9
- package/dist/src/server.mjs +2 -2
- package/dist/src/tools/schema.mjs +11 -3
- package/dist/src/tui/index.mjs +296 -0
- package/package.json +2 -1
- package/skills/agent-rag/SKILL.md +1 -1
package/README.md
CHANGED
|
@@ -16,6 +16,12 @@ Install globally so an MCP adapter can resolve `agent-rag-mcp` from its `PATH`:
|
|
|
16
16
|
npm install --global agent-rag-mcp
|
|
17
17
|
```
|
|
18
18
|
|
|
19
|
+
### Standalone terminal interface
|
|
20
|
+
|
|
21
|
+
Run `agent-rag` in an interactive terminal to browse and administer the knowledge base without Pi. Use `agent-rag --root <path>` to select a different storage root. Otherwise, it uses the same root discovery as the CLI, including `rag.config.json`, `RAG_ROOT`, and the CLI home default. The current root and selected client/project appear in the menu. This interface requires a terminal; use `rag` for scripts and JSON output.
|
|
22
|
+
|
|
23
|
+
Choose a client and project to browse all its documents (including retired ones), inspect current and historical versions, read extracted text in pages, search current documents, or download a selected version's original file. Downloads verify the stored SHA-256 and refuse to overwrite an existing destination. The menu also supports file/directory imports, status, doctor, review decisions, relabeling and retiring documents, entity renaming, backup, restore, reindex and rehash. Write operations ask for confirmation. Backup includes the selected client and all its projects, not other clients; restore requires an empty destination root and restores the selected client. Semantic search remains available through the Pi interface and MCP tools.
|
|
24
|
+
|
|
19
25
|
The `rag` commands available today are:
|
|
20
26
|
|
|
21
27
|
- `rag init` — Initializes the namespace database and object store.
|
|
@@ -97,7 +103,9 @@ To constrain one MCP server process or Pi extension to particular namespaces, se
|
|
|
97
103
|
|
|
98
104
|
The CLI defaults to the user's `.rag` directory, while the MCP server and Pi extension resolve their default relative to their working directory. Set the same absolute `RAG_ROOT` for all processes that must share an index. Installing the Pi package also registers native tools; if using the MCP adapter at the same time, configure and verify each route deliberately to avoid duplicate tools or inconsistent capabilities.
|
|
99
105
|
|
|
100
|
-
`rag_find` searches current document versions by default. Set `includeHistorical: true` to include superseded versions; each citation has an `isCurrent` flag so historical evidence is distinguishable.
|
|
106
|
+
`rag_find` searches current document versions by default. Set `includeHistorical: true` to include superseded versions; each citation has an `isCurrent` flag so historical evidence is distinguishable. `total` is the number of matching chunks before the `limit` slice; `citations.length` is how many citations were returned. Its `excerptChars` input defaults to 800. Each `rag_read` result reports `totalChunks` and `nextChunkOrdinal` for traversing chunks; `truncated` concerns the current chunk only. It reports `chunkBytes`, `returnedBytes`, `returnedLines`, `truncatedBy` (`"bytes"`, `"lines"`, or `null`) and `nextByteOffset` for the current page.
|
|
107
|
+
|
|
108
|
+
For byte-exact pagination, start with `startByte: 0`, a bounded `maxBytes` (for example 8,192), and a positive `maxLines`. Repeat the same citation using the returned `nextByteOffset` as `startByte` until it is `null`; concatenate page texts directly, without adding separators. Offsets count UTF-8 bytes in the stored chunk and must be character boundaries. The byte-page mode includes any line separator cut by `maxLines` in the page, preserving exact text. If `nextByteOffset` does not advance, increase `maxBytes` to fit the next UTF-8 character and ensure `maxLines` is positive. Leave room for JSON escaping and transport overhead when choosing page sizes; if the transport truncates a response, retry that page with a smaller limit. Do not combine `startByte` and `startLine`. Existing line pagination remains available with `startLine` (default 0, maximum 10,000), advancing it by `returnedLines` only when each complete page fits within `maxBytes`; line pages omit the separating newline at a line-limit cut.
|
|
101
109
|
|
|
102
110
|
Search matches every query term by default (`matchMode: "all"`). Set `matchMode: "any"` to return candidates containing at least one query term. This broader mode may return unrelated documents and does not provide semantic search.
|
|
103
111
|
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
import fs from "node:fs";
|
|
3
|
+
import path from "node:path";
|
|
4
|
+
import { spawnSync } from "node:child_process";
|
|
5
|
+
import { fileURLToPath } from "node:url";
|
|
6
|
+
|
|
7
|
+
const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
|
8
|
+
const built = path.join(root, "dist", "src", "tui", "index.mjs");
|
|
9
|
+
const source = path.join(root, "src", "tui", "index.mts");
|
|
10
|
+
const mode = process.env.RAG_ENTRY ?? "auto";
|
|
11
|
+
if (!["auto", "source", "dist"].includes(mode)) {
|
|
12
|
+
console.error(`agent-rag: invalid RAG_ENTRY: ${mode}`);
|
|
13
|
+
process.exitCode = 1;
|
|
14
|
+
} else {
|
|
15
|
+
const entry = (mode === "source" ? [source] : mode === "dist" ? [built] : [built, source]).find(fs.existsSync);
|
|
16
|
+
if (!entry) {
|
|
17
|
+
console.error("agent-rag: TUI entry not found; build the package first.");
|
|
18
|
+
process.exitCode = 1;
|
|
19
|
+
} else {
|
|
20
|
+
const flags = entry === source ? ["--experimental-strip-types", "--experimental-sqlite"] : ["--experimental-sqlite"];
|
|
21
|
+
const child = spawnSync(process.execPath, [...flags, entry, ...process.argv.slice(2)], { stdio: "inherit" });
|
|
22
|
+
if (child.error) { console.error(`agent-rag: ${child.error.message}`); process.exitCode = 1; }
|
|
23
|
+
else if (child.signal) process.kill(process.pid, child.signal);
|
|
24
|
+
else process.exitCode = child.status ?? 1;
|
|
25
|
+
}
|
|
26
|
+
}
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import*as de from"node:path";import*as X from"node:fs";import*as
|
|
1
|
+
import*as de from"node:path";import*as X from"node:fs";import*as M from"node:path";var H=class extends Error{code;path;constructor(n,r,i){let t=i instanceof Error?i.message:String(i);super(r==="config_invalid"?`Invalid JSON in config file ${n}: ${t}`:`Unable to read config file ${n}: ${t}`,{cause:i}),this.name="ConfigDiscoveryError",this.code=r,this.path=n}};function He(e){let n;try{n=X.readFileSync(e,"utf8")}catch(r){throw new H(e,"config_unreadable",r)}try{return JSON.parse(n)}catch(r){throw new H(e,"config_invalid",r)}}function G({cwd:e,homeDir:n,argv:r,env:i}){let t={};for(let y=0;y<r.length;y+=1){let h=/^--(client|project|root)=(.*)$/.exec(r[y]);h?t[h[1]]=h[2]:/^--(client|project|root)$/.test(r[y])&&r[y+1]!==void 0&&(t[r[y].slice(2)]=r[y+1],y+=1)}let a={client:i.RAG_CLIENT,project:i.RAG_PROJECT,root:i.RAG_ROOT},d=M.join(e,"rag.config.json"),u=M.join(M.dirname(e),"rag.config.json"),l=M.join(n,".rag","config.json"),c=X.existsSync(d)?d:X.existsSync(u)?u:X.existsSync(l)?l:null,o=c===null?{}:He(c),m=typeof o.client=="string"?o.client:void 0,p=typeof o.project=="string"?o.project:void 0,s=t.client??a.client??m??null,_=t.project??a.project??p??null,T=s===null&&_===null?null:{client:s,project:_},E=t.client!==void 0||t.project!==void 0?"args":a.client!==void 0||a.project!==void 0?"env":m!==void 0||p!==void 0?c===d?"cwd":c===u?"parent":"home":"unscoped";return{namespace:T,source:E,configPath:E==="cwd"||E==="parent"||E==="home"?c:null,failClosed:typeof o.failClosed=="boolean"?o.failClosed:!0,root:t.root??a.root??(typeof o.root=="string"?o.root:M.join(n,".rag"))}}import*as Y from"node:fs";import{DatabaseSync as Ye}from"node:sqlite";import*as Ee from"node:path";import{DatabaseSync as At}from"node:sqlite";var fe=class extends Error{code="invalid_client_name";constructor(n){super(`Invalid client name "${n}". Client names must match [A-Za-z0-9][A-Za-z0-9._-]* and cannot be "." or "..".`),this.name="InvalidClientNameError"}};function ze(e){if(e==="."||e===".."||!/^[A-Za-z0-9][A-Za-z0-9._-]*$/.test(e))throw new fe(e)}function x(e,n){return ze(n),Ee.join(e,"clients",n,"rag.db")}var Je=800,_e=/[\p{L}\p{M}\p{N}]+/gu;function Ke(e){return e.match(_e)??[]}function qe(e,n){return e.map(r=>`"${r.replaceAll('"','""')}"`).join(n==="any"?" OR ":" AND ")}function Ae(e){return e.normalize("NFD").replace(new RegExp("\\p{M}","gu"),"").toLowerCase()}function We(e,n){let r=new Set(n.map(Ae));for(let i of e.matchAll(_e)){let t=i[0];if(!r.has(Ae(t)))continue;let a=i.index;return{start:a,end:a+t.length}}return null}function Ze(e,n,r,i){let t=Math.min(i,e.length);if(t===0)return"";let a=Math.min(r-n,t),d=Math.max(0,Math.min(n-Math.floor((t-a)/2),e.length-t)),u=d+t,l=d,c=u,o=Array.from(e.matchAll(_e),s=>({start:s.index,end:s.index+s[0].length})),m=o.find(s=>s.start<d&&d<s.end);m&&(d=o.find(s=>s.start>=m.end)?.start??e.length);let p=o.find(s=>s.start<u&&u<s.end);if(p){let s=o.filter(_=>_.end<=p.start);u=s[s.length-1]?.end??0}return u<=d?e.slice(l,c):e.slice(d,u)}function Qe(e,n){return e<n?-1:e>n?1:0}function z(e,n,r){return{namespace:e,error:{code:n,message:r}}}function P(e){let{root:n,namespace:r,query:i}=e,t=e.excerptLength??Je;if(!Number.isInteger(t)||t<1)return z(r,"invalid_excerpt_length","excerptLength must be a positive integer.");let a=Ke(i);if(a.length===0)return{namespace:r,result:{citations:[],total:0}};let d;try{let u=x(n,r.client);if(!Y.existsSync(u)||!Y.statSync(u).isFile())return z(r,"database_not_found",`No database exists for namespace ${r.client}/${r.project}.`);d=new Ye(u,{readOnly:!0}),d.exec("BEGIN");let l=`FROM chunk_fts
|
|
2
2
|
JOIN chunk AS c ON c.rowid = chunk_fts.rowid
|
|
3
3
|
JOIN doc_version AS dv ON dv.doc_version_id = c.doc_version_id
|
|
4
4
|
JOIN document AS d ON d.document_id = dv.document_id
|
|
@@ -7,25 +7,25 @@ import*as de from"node:path";import*as X from"node:fs";import*as B from"node:pat
|
|
|
7
7
|
JOIN client AS cl ON cl.client_id = p.client_id
|
|
8
8
|
LEFT JOIN section AS s ON s.section_id = c.section_id
|
|
9
9
|
WHERE chunk_fts MATCH ? AND cl.name = ? AND p.slug = ?
|
|
10
|
-
AND d.deleted_at IS NULL AND e.deleted_at IS NULL AND (dv.is_current = 1 OR ? = 1)`,
|
|
10
|
+
AND d.deleted_at IS NULL AND e.deleted_at IS NULL AND (dv.is_current = 1 OR ? = 1)`,c=[qe(a,e.matchMode??"all"),r.client,r.project,e.includeHistorical?1:0],{total:o}=d.prepare(`SELECT COUNT(*) AS total ${l}`).get(...c),m=d.prepare(`SELECT dv.doc_version_id AS docVersionId, d.document_id AS documentId,
|
|
11
11
|
e.entity_key AS entityKey, d.label AS label, dv.version_label AS versionLabel, dv.is_current AS isCurrent,
|
|
12
12
|
s.ordinal AS sectionOrdinal, c.ordinal AS chunkOrdinal, c.text AS text,
|
|
13
13
|
c.char_start AS charStart, c.char_end AS charEnd, bm25(chunk_fts) AS score
|
|
14
14
|
${l}
|
|
15
|
-
ORDER BY score ASC, dv.doc_version_id COLLATE BINARY ASC, c.ordinal ASC LIMIT ?`).all(...
|
|
15
|
+
ORDER BY score ASC, dv.doc_version_id COLLATE BINARY ASC, c.ordinal ASC LIMIT ?`).all(...c,e.maxResults??-1);d.exec("COMMIT");let p=[];for(let s of m){let _=We(s.text,a);if(!_)return z(r,"match_offsets_unavailable","A matched FTS token could not be located in its stored chunk text.");p.push({docVersionId:s.docVersionId,documentId:s.documentId,entityKey:s.entityKey,label:s.label,versionLabel:s.versionLabel,isCurrent:s.isCurrent===1,sectionOrdinal:s.sectionOrdinal,chunkOrdinal:s.chunkOrdinal,charStart:s.charStart,charEnd:s.charEnd,matchStart:_.start,matchEnd:_.end,score:s.score,excerpt:Ze(s.text,_.start,_.end,t)})}return p.sort((s,_)=>s.score-_.score||Qe(s.docVersionId,_.docVersionId)||s.chunkOrdinal-_.chunkOrdinal),{namespace:r,result:{citations:p,total:o}}}catch(u){let l=u instanceof Error?u.message:String(u);return z(r,"database_unavailable",l)}finally{d?.close()}}import*as J from"node:fs";import{DatabaseSync as et}from"node:sqlite";function $(e,n,r){return{namespace:e,error:{code:n,message:r}}}function tt(e,n){let r=Buffer.from(e,"utf8");if(r.byteLength<=n)return{text:e,truncated:!1};let i=n;for(;i>0;)try{return{text:new TextDecoder("utf-8",{fatal:!0}).decode(r.subarray(0,i)),truncated:!0}}catch{i-=1}return{text:"",truncated:!0}}function nt(e,n){if(n===0)return 0;let r=0;for(let i=0;i<e.length;i+=1)if(!(e[i]!=="\r"&&e[i]!==`
|
|
16
16
|
`)&&(e[i]==="\r"&&e[i+1]===`
|
|
17
17
|
`&&(i+=1),r+=1,r===n))return i+1;return e.length}function rt(e){if(e.length===0)return 0;let n=1;for(let r=0;r<e.length;r+=1)e[r]==="\r"&&e[r+1]===`
|
|
18
18
|
`?(n+=1,r+=1):(e[r]==="\r"||e[r]===`
|
|
19
|
-
`)&&(n+=1);return n}function it(e,n){if(e.length===0)return{text:e,truncated:!1};if(n===0)return{text:"",truncated:!0};let
|
|
20
|
-
`)continue;let t
|
|
21
|
-
`&&(
|
|
19
|
+
`)&&(n+=1);return n}function it(e,n,r=!1){if(e.length===0)return{text:e,truncated:!1};if(n===0)return{text:"",truncated:!0};let i=1;for(let t=0;t<e.length;t+=1){if(e[t]!=="\r"&&e[t]!==`
|
|
20
|
+
`)continue;let a=t;if(e[t]==="\r"&&e[t+1]===`
|
|
21
|
+
`&&(t+=1),i===n)return{text:e.slice(0,r?t+1:a),truncated:!0};i+=1}return{text:e,truncated:!1}}async function he(e){let{root:n,namespace:r,docVersionId:i,chunkOrdinal:t,maxBytes:a,maxLines:d,startLine:u=0,startByte:l}=e;if(!Number.isInteger(a)||a<0||!Number.isInteger(d)||d<0||!Number.isInteger(u)||u<0||l!==void 0&&(!Number.isSafeInteger(l)||l<0)||l!==void 0&&e.startLine!==void 0)return $(r,"invalid_read_limit","maxBytes, maxLines, and startLine must be non-negative integers.");let c;try{let o=x(n,r.client);if(!J.existsSync(o)||!J.statSync(o).isFile())return $(r,"database_not_found",`No database exists for namespace ${r.client}/${r.project}.`);c=new et(o,{readOnly:!0});let m=c.prepare(`SELECT c.text AS text,
|
|
22
22
|
(SELECT COUNT(*) FROM chunk AS total_chunks WHERE total_chunks.doc_version_id = c.doc_version_id) AS totalChunks
|
|
23
23
|
FROM chunk AS c
|
|
24
24
|
JOIN doc_version AS dv ON dv.doc_version_id = c.doc_version_id
|
|
25
25
|
JOIN document AS d ON d.document_id = dv.document_id
|
|
26
26
|
JOIN project AS p ON p.project_id = d.project_id
|
|
27
27
|
JOIN client AS cl ON cl.client_id = p.client_id
|
|
28
|
-
WHERE c.doc_version_id = ? AND c.ordinal = ? AND cl.name = ? AND p.slug = ?`).get(i,t,r.client,r.project);if(!
|
|
28
|
+
WHERE c.doc_version_id = ? AND c.ordinal = ? AND cl.name = ? AND p.slug = ?`).get(i,t,r.client,r.project);if(!m)return $(r,"citation_not_found","The citation does not exist in the requested document version and namespace.");let p=Buffer.from(m.text,"utf8"),s=l??Buffer.byteLength(m.text.slice(0,nt(m.text,u)),"utf8");if(s>p.length||s<p.length&&(p[s]&192)===128)return $(r,"invalid_start_byte","startByte must be a UTF-8 character boundary within the chunk.");let _=p.subarray(s).toString("utf8"),T=tt(_,a),E=it(T.text,d,l!==void 0),h=E.truncated&&(l===void 0||E.text.length<_.length)?"lines":T.truncated?"bytes":null,f=m.totalChunks;return{namespace:r,result:{text:E.text,truncated:h!==null,totalChunks:f,nextChunkOrdinal:t+1<f?t+1:null,chunkBytes:Buffer.byteLength(m.text,"utf8"),returnedBytes:Buffer.byteLength(E.text,"utf8"),returnedLines:rt(E.text)-(l!==void 0&&/[\r\n]$/.test(E.text)?1:0),truncatedBy:h,nextByteOffset:h===null?null:s+Buffer.byteLength(E.text,"utf8")}}}catch(o){let m=o instanceof Error?o.message:String(o);return $(r,"object_read_failed",m)}finally{c?.close()}}import*as q from"node:fs";import*as W from"node:path";import{createHash as Re}from"node:crypto";import{DatabaseSync as Ie}from"node:sqlite";var K=8192,ve=3e4,ot=100;async function Se(e,n,r){let i=AbortSignal.timeout(ve),t=r?AbortSignal.any([r,i]):i,a=await fetch("http://127.0.0.1:11434/api/embed",{method:"POST",headers:{"content-type":"application/json"},body:JSON.stringify({model:e,input:n,truncate:!1}),signal:t,redirect:"error"});if(!a.ok)throw new Error(`Local embedding service returned HTTP ${a.status}.`);let d=await a.json();if(!d||typeof d!="object"||!("embeddings"in d))throw new Error("Embedding response is invalid.");return d.embeddings}function Oe(e){let n;try{n=new URL(e)}catch{throw new Error("invalid_azure_endpoint")}let r=/^[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.(?:openai\.azure\.com|services\.ai\.azure\.com|cognitiveservices\.azure\.com)$/i.test(n.hostname),i=["","/","/openai/v1","/openai/v1/"].includes(n.pathname);if(n.protocol!=="https:"||!r||!i||n.username||n.password||n.port||n.search||n.hash)throw new Error("invalid_azure_endpoint");return`${n.origin}/openai/v1/embeddings`}function at(e,n){let r=Oe(e);if(!n.trim())throw new Error("missing_azure_api_key");return async(i,t,a)=>{let d=AbortSignal.timeout(ve),u=a?AbortSignal.any([a,d]):d,l=await fetch(r,{method:"POST",headers:{"content-type":"application/json","api-key":n},body:JSON.stringify({model:i,input:t,encoding_format:"float"}),signal:u,redirect:"error"});if(!l.ok)throw new Error("Azure embedding request failed.");let c=await l.json();if(!c||typeof c!="object"||!("data"in c)||!Array.isArray(c.data)||c.data.length!==t.length)throw new Error("Invalid Azure embedding response.");let o=new Array(t.length);for(let m of c.data){if(!m||typeof m!="object"||!Number.isInteger(m.index)||m.index<0||m.index>=t.length||o[m.index]!==void 0||!Array.isArray(m.embedding))throw new Error("Invalid Azure embedding response.");o[m.index]=m.embedding}return o}}async function Te(e,n=process.env,r){let i=n.RAG_EMBEDDING_PROVIDER??"ollama",t=n.RAG_EMBEDDING_BATCH_SIZE===void 0?8:Number(n.RAG_EMBEDDING_BATCH_SIZE);if(!Number.isInteger(t)||t<1||t>32)return{namespace:e.namespace,error:{code:"invalid_embedding_batch_size",message:"RAG_EMBEDDING_BATCH_SIZE must be an integer from 1 to 32."}};let a=n.RAG_EMBEDDING_BATCH_BYTES===void 0?65536:Number(n.RAG_EMBEDDING_BATCH_BYTES);if(!Number.isInteger(a)||a<1024||a>1048576)return{namespace:e.namespace,error:{code:"invalid_embedding_batch_bytes",message:"RAG_EMBEDDING_BATCH_BYTES must be an integer from 1024 to 1048576."}};let d=n.RAG_EMBEDDING_REVISION??"";if(i==="ollama")return Le({...e,model:n.RAG_EMBEDDING_MODEL??"",cacheIdentity:JSON.stringify(["ollama",n.RAG_EMBEDDING_MODEL??"",d]),batchSize:t,maxBatchBytes:a},Se,r);if(i!=="azure")return{namespace:e.namespace,error:{code:"invalid_embedding_provider",message:"RAG_EMBEDDING_PROVIDER must be ollama or azure."}};let u=n.RAG_AZURE_OPENAI_ENDPOINT,l=n.RAG_AZURE_OPENAI_API_KEY,c=n.RAG_AZURE_OPENAI_DEPLOYMENT;if(!u||!l||!c?.trim())return{namespace:e.namespace,error:{code:"semantic_not_configured",message:"Azure embeddings require endpoint, deployment and API key configuration."}};let o;try{o=at(u,l)}catch{return{namespace:e.namespace,error:{code:"invalid_azure_endpoint",message:"Azure endpoint must be a supported HTTPS resource URL."}}}return Le({...e,model:c,cacheIdentity:JSON.stringify(["azure",Oe(u),c,d]),batchSize:t,maxBatchBytes:a},o,r)}function ge(e,n){let r=0,i=0,t=0;for(let a=0;a<e.length;a+=1)r+=e[a]*n[a],i+=e[a]**2,t+=n[a]**2;if(!i||!t||!Number.isFinite(r)||!Number.isFinite(i)||!Number.isFinite(t))throw new Error("Embedding vector must be finite and nonzero.");return r/Math.sqrt(i*t)}async function Le(e,n=Se,r){let{root:i,namespace:t,query:a,includeHistorical:d,excerptLength:u,model:l}=e,c=e.batchSize??8,o=e.maxBatchBytes??65536;if(!l.trim())return{namespace:t,error:{code:"semantic_not_configured",message:"Configure an embedding model."}};if(!Number.isInteger(c)||c<1||c>32)return{namespace:t,error:{code:"invalid_embedding_batch_size",message:"Embedding batch size must be an integer from 1 to 32."}};if(!Number.isInteger(o)||o<1024||o>1048576)return{namespace:t,error:{code:"invalid_embedding_batch_bytes",message:"Embedding batch bytes must be an integer from 1024 to 1048576."}};if(a.length>K)return{namespace:t,error:{code:"semantic_query_too_large",message:`Semantic queries must not exceed ${K} characters.`}};let m,p,s="database";try{let _=x(i,t.client);if(!q.existsSync(_)||!q.statSync(_).isFile())return{namespace:t,error:{code:"database_not_found",message:"Namespace database was not found."}};m=new Ie(_,{readOnly:!0});let E=m.prepare(`SELECT dv.doc_version_id AS docVersionId, d.document_id AS documentId,
|
|
29
29
|
e.entity_key AS entityKey, d.label AS label, dv.version_label AS versionLabel, dv.is_current AS isCurrent,
|
|
30
30
|
s.ordinal AS sectionOrdinal, c.ordinal AS chunkOrdinal, c.char_start AS charStart, c.char_end AS charEnd, c.text AS text
|
|
31
31
|
FROM chunk AS c JOIN doc_version AS dv ON dv.doc_version_id = c.doc_version_id
|
|
@@ -33,21 +33,21 @@ import*as de from"node:path";import*as X from"node:fs";import*as B from"node:pat
|
|
|
33
33
|
JOIN project AS p ON p.project_id = d.project_id JOIN client AS cl ON cl.client_id = p.client_id
|
|
34
34
|
LEFT JOIN section AS s ON s.section_id = c.section_id
|
|
35
35
|
WHERE cl.name = ? AND p.slug = ? AND d.deleted_at IS NULL AND e.deleted_at IS NULL AND (dv.is_current = 1 OR ? = 1)
|
|
36
|
-
ORDER BY dv.doc_version_id COLLATE BINARY, c.ordinal`).iterate(t.client,t.project,d?1:0)[Symbol.iterator](),y=
|
|
36
|
+
ORDER BY dv.doc_version_id COLLATE BINARY, c.ordinal`).iterate(t.client,t.project,d?1:0)[Symbol.iterator](),y=E.next();if(y.done)return{namespace:t,result:{citations:[],total:0,searchMode:"semantic"}};s="cache",p=new Ie(W.join(W.dirname(_),"semantic-cache.sqlite")),p.exec("PRAGMA busy_timeout = 5000; PRAGMA journal_mode = WAL;"),p.exec(`CREATE TABLE IF NOT EXISTS vector_cache (
|
|
37
37
|
project TEXT NOT NULL, identity TEXT NOT NULL, doc_version_id TEXT NOT NULL, ordinal INTEGER NOT NULL,
|
|
38
38
|
text_hash TEXT NOT NULL, dimensions INTEGER NOT NULL, vector BLOB NOT NULL,
|
|
39
39
|
PRIMARY KEY(project, identity, doc_version_id, ordinal)
|
|
40
|
-
) WITHOUT ROWID`),
|
|
41
|
-
ON CONFLICT(project, identity, doc_version_id, ordinal) DO UPDATE SET text_hash=excluded.text_hash, dimensions=excluded.dimensions, vector=excluded.vector`),v=(g,L,O)=>{if(!Array.isArray(g)||g.length!==L||!Array.isArray(g[0]))throw new Error("Invalid vector count.");let S=O??g[0].length;if(S<1||S>8192||g.some(U=>!Array.isArray(U)||U.length!==S||U.some(j=>typeof j!="number"||!Number.isFinite(j))))throw new Error("Invalid vector dimensions.");return S};c="embedding";let I=await n(l,[a],r),R=v(I,1),A=I[0];ge(A,A);let w=[],ye=0,be=(g,L)=>{w.push({docVersionId:g.docVersionId,documentId:g.documentId,entityKey:g.entityKey,label:g.label,versionLabel:g.versionLabel,isCurrent:g.isCurrent===1,sectionOrdinal:g.sectionOrdinal,chunkOrdinal:g.chunkOrdinal,charStart:g.charStart,charEnd:g.charEnd,matchStart:null,matchEnd:null,score:ge(A,L),excerpt:g.text.slice(0,u)}),w.sort((O,S)=>S.score-O.score||O.docVersionId.localeCompare(S.docVersionId)||O.chunkOrdinal-S.chunkOrdinal),w.length>ot&&w.pop()},M=[],le=0,me=async()=>{if(!M.length)return;c="embedding";let g=await n(l,M.map(L=>L.row.text),r);if(v(g,M.length,R),r?.aborted)throw new Error("cancelled");for(let L of g)ge(L,L);c="cache",m.exec("BEGIN IMMEDIATE");try{for(let[L,O]of M.entries()){let S=g[L],U=Buffer.allocUnsafe(R*4);S.forEach((j,pe)=>{if(!Number.isFinite(Math.fround(j)))throw new Error("Embedding value exceeds float32.");U.writeFloatLE(j,pe*4)}),b.run(t.project,N,O.row.docVersionId,O.row.chunkOrdinal,O.hash,R,U),be(O.row,Array.from({length:R},(j,pe)=>U.readFloatLE(pe*4)))}m.exec("COMMIT")}catch(L){throw m.exec("ROLLBACK"),L}M=[],le=0};for(let g of{*[Symbol.iterator](){yield y.value,yield*{[Symbol.iterator]:()=>T}}}){if(r?.aborted)throw new Error("cancelled");if(g.text.length>q)return{namespace:t,error:{code:"semantic_chunk_too_large",message:`A chunk exceeds ${q} characters; semantic search did not finish.`}};ye+=1;let L=Re("sha256").update(g.text).digest("hex");c="cache";let O=E.get(t.project,N,g.docVersionId,g.chunkOrdinal,L)??_.get(t.project,N,L);if(O){if(O.dimensions!==R||O.vector.byteLength!==R*4)return{namespace:t,error:{code:"embedding_dimension_mismatch",message:"Cached vectors do not match this model. Set a new RAG_EMBEDDING_REVISION before rebuilding."}};let S=Buffer.from(O.vector);be(g,Array.from({length:R},(U,j)=>S.readFloatLE(j*4)))}else{let S=Buffer.byteLength(g.text,"utf8");if(S>o)return{namespace:t,error:{code:"semantic_chunk_too_large",message:"A chunk exceeds RAG_EMBEDDING_BATCH_BYTES."}};le+S>o&&await me(),M.push({row:g,hash:L}),le+=S,M.length>=s&&await me()}}return await me(),{namespace:t,result:{citations:w,total:ye,searchMode:"semantic"}}}catch{return r?.aborted?{namespace:t,error:{code:"cancelled",message:"Search was cancelled."}}:c==="embedding"?{namespace:t,error:{code:"embedding_unavailable",message:"Embedding request failed; check the configured provider, model and access."}}:c==="cache"?{namespace:t,error:{code:"semantic_cache_unavailable",message:"Semantic cache could not be read or written."}}:{namespace:t,error:{code:"database_unavailable",message:"Could not load namespace chunks."}}}finally{m?.close(),p?.close()}}function Ne(e,n){let r=new Map;for(let i of[e,n])i.forEach((t,a)=>{let d=JSON.stringify([t.docVersionId,t.chunkOrdinal]),u=r.get(d);u?u.score+=1/(60+a+1):r.set(d,{citation:t,score:1/(60+a+1)})});return[...r.values()].map(({citation:i,score:t})=>({...i,score:t})).sort((i,t)=>t.score-i.score||i.docVersionId.localeCompare(t.docVersionId)||i.chunkOrdinal-t.chunkOrdinal)}import*as P from"node:fs";import*as Q from"node:path";import{execFile as st}from"node:child_process";import{fileURLToPath as ct}from"node:url";function dt(){let e=Q.dirname(ct(import.meta.url));for(let n of["../bin/rag.mjs","../../bin/rag.mjs","../../../bin/rag.mjs"]){let r=Q.resolve(e,n);if(P.existsSync(r)&&P.statSync(r).isFile())return r}throw new Error("The packaged rag CLI could not be found.")}async function we(e,n,r,i){if(["status","ls","review","doctor","rm","backup"].includes(i[0]??"")&&!P.existsSync(x(n,r.client)))return{namespace:r,error:{code:"database_not_found",message:"Client database was not found."}};let t={...process.env,RAG_FORMAT:"json",RAG_ROOT:n};delete t.RAG_ENTRY;let a=dt();return new Promise((d,u)=>{st(process.execPath,[a,...i,"--root",n,"--client",r.client,"--project",r.project],{cwd:e,env:t,timeout:12e4,maxBuffer:1048576,windowsHide:!0},(l,s)=>{try{let o=JSON.parse(s.trim());if(!o||typeof o!="object"||Array.isArray(o))throw new Error("invalid result");d(o)}catch{u(new Error(l?.killed?"RAG command timed out.":"RAG command failed without a valid result."))}})})}import*as Ce from"node:fs";import*as ee from"node:path";import{DatabaseSync as ut}from"node:sqlite";function xe(e,n){let r=ee.join(ee.dirname(x(e,n.client)),"semantic-cache.sqlite");if(!Ce.existsSync(r))return{storedVectorRecords:0,vectorBytes:0};let i=new ut(r,{readOnly:!0});try{let t=i.prepare("SELECT COUNT(*) AS count, COALESCE(SUM(length(vector)), 0) AS bytes FROM vector_cache WHERE project = ?").get(n.project);return{storedVectorRecords:t.count,vectorBytes:t.bytes}}finally{i.close()}}import{z as C}from"zod";var ke=100,De=2e3,Ue=1048576,te=1e4,lt=10,mt=800,pt=65536,ft=200;function je(e){let n=new Set;for(let r of(e??"find,read").split(",").map(i=>i.trim())){if(r!=="find"&&r!=="read")throw new Error(`invalid_capability: Unknown RAG_CAPABILITIES value "${r}".`);n.add(r)}return n}var ne=C.string().min(1).refine(e=>e.trim().length>0),re=C.object({client:ne,project:ne}).strict();function Me(e){if(e===void 0)return null;let n;try{n=JSON.parse(e)}catch{throw new Error("invalid_namespace_allowlist: RAG_ALLOWED_NAMESPACES must be a JSON array.")}let r=C.array(re).safeParse(n);if(!r.success)throw new Error("invalid_namespace_allowlist: RAG_ALLOWED_NAMESPACES must be an array of client/project objects.");return r.data}function ie(e,n){return e===null||e.some(r=>r.client===n.client&&r.project===n.project)}function V(e,n,r){return C.number().int().min(e).max(n).optional().transform(i=>i??r)}var Be=C.object({namespace:re,query:ne,includeHistorical:C.boolean().optional().default(!1),matchMode:C.enum(["all","any"]).optional().default("all"),searchMode:C.enum(["lexical","semantic","hybrid"]).optional().default("lexical"),limit:V(1,ke,lt),excerptChars:V(1,De,mt)}).strict(),Xe=C.object({namespace:re,docVersionId:ne,chunkOrdinal:C.number().multipleOf(1).min(0).refine(Number.isSafeInteger),maxBytes:V(0,Ue,pt),maxLines:V(0,te,ft),startLine:V(0,te,0)}).strict();function Et(e){let n=e.find(a=>a.code==="unrecognized_keys"&&a.path.length===0)??e.find(a=>a.code==="unrecognized_keys");if(!n||n.code!=="unrecognized_keys")return null;let r=n.keys[0];if(r===void 0)return null;let i=n.path.map(String).join(".");return{code:"unknown_field",message:`Unknown field "${i.length>0?`${i}.${r}`:r}".`}}function k(e,n){return e.some(r=>r.path[0]===n)}function $(e,n,r){return{code:"invalid_limit",message:`${e} must be an integer from ${n} to ${r}.`}}function Fe(e,n){let r=Et(e);if(r)return r;if(k(e,"namespace"))return{code:"namespace_required",message:"namespace must contain non-empty client and project strings."};if(n==="find"){if(k(e,"query"))return{code:"invalid_query",message:"query must be a non-empty string."};if(k(e,"limit"))return $("limit",1,ke);if(k(e,"excerptChars"))return $("excerptChars",1,De)}else{if(k(e,"docVersionId"))return{code:"invalid_input",message:"docVersionId must be a non-empty string."};if(k(e,"chunkOrdinal"))return{code:"invalid_input",message:"chunkOrdinal must be a non-negative integer."};if(k(e,"maxBytes"))return $("maxBytes",0,Ue);if(k(e,"maxLines"))return $("maxLines",0,te);if(k(e,"startLine"))return $("startLine",0,te)}return e.some(i=>i.path.length===0)?{code:"invalid_input",message:"Tool arguments must be an object."}:{code:"invalid_input",message:"Tool arguments are invalid."}}function _t(e){let n=re.safeParse(e);return n.success?n.data:null}function Ge(e){let n=Be.safeParse(e);return n.success?{ok:!0,input:n.data,namespace:n.data.namespace}:{ok:!1,namespace:oe(e),error:Fe(n.error.issues,"find")}}function Pe(e){let n=Xe.safeParse(e);return n.success?{ok:!0,input:n.data,namespace:n.data.namespace}:{ok:!1,namespace:oe(e),error:Fe(n.error.issues,"read")}}function ht(e){return typeof e=="object"&&e!==null&&!Array.isArray(e)}function oe(e){return ht(e)?_t(e.namespace):null}function $e(e){return C.toJSONSchema(e,{io:"input"})}var ae={name:"rag_find",description:"Search current document versions. Semantic or hybrid search sends indexed text to the explicitly configured provider; use matchMode any for broader lexical matching or includeHistorical for older versions.",inputSchema:$e(Be)},se={name:"rag_read",description:"Read bounded text from a cited document chunk in the explicit namespace.",inputSchema:$e(Xe)};function ue(e,n=!1){return{content:[{type:"text",text:JSON.stringify(e)}],details:e,...n?{isError:!0}:{}}}function D(e,n){return ue({namespace:e,error:n},!0)}function ce(e){return D(e,{code:"cancelled",message:"Tool call was cancelled."})}function Ve(e,n){let r=n instanceof Error?n.message:String(n);return D(oe(e),{code:"retrieval_failed",message:r})}var gt="The namespace travels on every call: never infer `client` or `project` from the cwd, repo name, config, env, or a placeholder like `default`. If the user has not stated it, ask before the first call.\nUse the RAG tool the role actually exposes; when only `rag_find` is available, delegate for deeper text instead of paraphrasing a catalog entry.\nWhen an excerpt is truncated, when the user asks whether the document covers something, or when a longer verbatim quote is needed, hand off to the subagent that exposes `rag_read` with the namespace and the citation's `docVersionId`; never assert absence from a catalog.";function Zt(e){e.on("before_agent_start",i=>{if(process.env.RAG_PROMPT==="off")return;let t=i.systemPromptOptions.sections??(i.systemPromptOptions.sections={});t.rag=gt});let n=je(process.env.RAG_CAPABILITIES),r=Me(process.env.RAG_ALLOWED_NAMESPACES);e.registerCommand("rag",{description:"Search the document knowledge base",async handler(i,t){if(!t.hasUI)return;if(!n.has("find")){t.ui.notify("RAG search is not enabled for this session.","warning");return}let a=(await t.ui.input("RAG client"))?.trim();if(!a)return;let d=(await t.ui.input("RAG project"))?.trim();if(!d)return;let u={client:a,project:d};if(!ie(r,u)){t.ui.notify("Namespace is not enabled for this process.","error");return}let l=process.env.RAG_TUI_ADMIN==="1",s=["Search","Status","Documents","Review queue","Doctor",...l?["Import file","Import directory","Retire document","Backup","Restore"]:[]],o=await t.ui.select(`RAG \xB7 ${a} / ${d}`,s);if(!o||!s.includes(o))return;if(o!=="Search"){try{let h=de.resolve(t.cwd,F({cwd:t.cwd,homeDir:t.cwd,argv:[],env:process.env}).root),T=async E=>{let _=await we(t.cwd,h,u,E);if(_.error&&typeof _.error=="object"){let b=_.error;return t.ui.notify(`${b.code??"error"}: ${b.message??"Command failed."}`,"error"),null}return _.result&&typeof _.result=="object"?_.result:{}};if(o==="Backup"||o==="Restore"){let E=(await t.ui.input(o==="Backup"?"New backup destination":"Existing backup directory"))?.trim();if(!E)return;let _=de.resolve(t.cwd,E),b=o==="Backup"?`Source: ${h}
|
|
42
|
-
New destination: ${
|
|
43
|
-
Restore to empty root: ${
|
|
44
|
-
Namespace: ${a} / ${d}`))return;let
|
|
40
|
+
) WITHOUT ROWID`),p.exec("CREATE INDEX IF NOT EXISTS idx_vector_content ON vector_cache(project, identity, text_hash)");let h=Re("sha256").update(e.cacheIdentity??JSON.stringify(["direct",l])).digest("hex"),f=p.prepare("SELECT dimensions, vector FROM vector_cache WHERE project = ? AND identity = ? AND doc_version_id = ? AND ordinal = ? AND text_hash = ?"),g=p.prepare("SELECT dimensions, vector FROM vector_cache WHERE project = ? AND identity = ? AND text_hash = ? LIMIT 1"),b=p.prepare(`INSERT INTO vector_cache(project, identity, doc_version_id, ordinal, text_hash, dimensions, vector) VALUES (?, ?, ?, ?, ?, ?, ?)
|
|
41
|
+
ON CONFLICT(project, identity, doc_version_id, ordinal) DO UPDATE SET text_hash=excluded.text_hash, dimensions=excluded.dimensions, vector=excluded.vector`),L=(N,v,O)=>{if(!Array.isArray(N)||N.length!==v||!Array.isArray(N[0]))throw new Error("Invalid vector count.");let S=O??N[0].length;if(S<1||S>8192||N.some(B=>!Array.isArray(B)||B.length!==S||B.some(U=>typeof U!="number"||!Number.isFinite(U))))throw new Error("Invalid vector dimensions.");return S};s="embedding";let I=await n(l,[a],r),R=L(I,1),A=I[0];ge(A,A);let C=[],ye=0,be=(N,v)=>{C.push({docVersionId:N.docVersionId,documentId:N.documentId,entityKey:N.entityKey,label:N.label,versionLabel:N.versionLabel,isCurrent:N.isCurrent===1,sectionOrdinal:N.sectionOrdinal,chunkOrdinal:N.chunkOrdinal,charStart:N.charStart,charEnd:N.charEnd,matchStart:null,matchEnd:null,score:ge(A,v),excerpt:N.text.slice(0,u)}),C.sort((O,S)=>S.score-O.score||O.docVersionId.localeCompare(S.docVersionId)||O.chunkOrdinal-S.chunkOrdinal),C.length>ot&&C.pop()},j=[],le=0,me=async()=>{if(!j.length)return;s="embedding";let N=await n(l,j.map(v=>v.row.text),r);if(L(N,j.length,R),r?.aborted)throw new Error("cancelled");for(let v of N)ge(v,v);s="cache",p.exec("BEGIN IMMEDIATE");try{for(let[v,O]of j.entries()){let S=N[v],B=Buffer.allocUnsafe(R*4);S.forEach((U,pe)=>{if(!Number.isFinite(Math.fround(U)))throw new Error("Embedding value exceeds float32.");B.writeFloatLE(U,pe*4)}),b.run(t.project,h,O.row.docVersionId,O.row.chunkOrdinal,O.hash,R,B),be(O.row,Array.from({length:R},(U,pe)=>B.readFloatLE(pe*4)))}p.exec("COMMIT")}catch(v){throw p.exec("ROLLBACK"),v}j=[],le=0};for(let N of{*[Symbol.iterator](){yield y.value,yield*{[Symbol.iterator]:()=>E}}}){if(r?.aborted)throw new Error("cancelled");if(N.text.length>K)return{namespace:t,error:{code:"semantic_chunk_too_large",message:`A chunk exceeds ${K} characters; semantic search did not finish.`}};ye+=1;let v=Re("sha256").update(N.text).digest("hex");s="cache";let O=f.get(t.project,h,N.docVersionId,N.chunkOrdinal,v)??g.get(t.project,h,v);if(O){if(O.dimensions!==R||O.vector.byteLength!==R*4)return{namespace:t,error:{code:"embedding_dimension_mismatch",message:"Cached vectors do not match this model. Set a new RAG_EMBEDDING_REVISION before rebuilding."}};let S=Buffer.from(O.vector);be(N,Array.from({length:R},(B,U)=>S.readFloatLE(U*4)))}else{let S=Buffer.byteLength(N.text,"utf8");if(S>o)return{namespace:t,error:{code:"semantic_chunk_too_large",message:"A chunk exceeds RAG_EMBEDDING_BATCH_BYTES."}};le+S>o&&await me(),j.push({row:N,hash:v}),le+=S,j.length>=c&&await me()}}return await me(),{namespace:t,result:{citations:C,total:ye,searchMode:"semantic"}}}catch{return r?.aborted?{namespace:t,error:{code:"cancelled",message:"Search was cancelled."}}:s==="embedding"?{namespace:t,error:{code:"embedding_unavailable",message:"Embedding request failed; check the configured provider, model and access."}}:s==="cache"?{namespace:t,error:{code:"semantic_cache_unavailable",message:"Semantic cache could not be read or written."}}:{namespace:t,error:{code:"database_unavailable",message:"Could not load namespace chunks."}}}finally{p?.close(),m?.close()}}function Ne(e,n){let r=new Map;for(let i of[e,n])i.forEach((t,a)=>{let d=JSON.stringify([t.docVersionId,t.chunkOrdinal]),u=r.get(d);u?u.score+=1/(60+a+1):r.set(d,{citation:t,score:1/(60+a+1)})});return[...r.values()].map(({citation:i,score:t})=>({...i,score:t})).sort((i,t)=>t.score-i.score||i.docVersionId.localeCompare(t.docVersionId)||i.chunkOrdinal-t.chunkOrdinal)}import*as V from"node:fs";import*as Z from"node:path";import{execFile as st}from"node:child_process";import{fileURLToPath as ct}from"node:url";function dt(){let e=Z.dirname(ct(import.meta.url));for(let n of["../bin/rag.mjs","../../bin/rag.mjs","../../../bin/rag.mjs"]){let r=Z.resolve(e,n);if(V.existsSync(r)&&V.statSync(r).isFile())return r}throw new Error("The packaged rag CLI could not be found.")}async function we(e,n,r,i){if(["status","ls","review","doctor","rm","backup"].includes(i[0]??"")&&!V.existsSync(x(n,r.client)))return{namespace:r,error:{code:"database_not_found",message:"Client database was not found."}};let t={...process.env,RAG_FORMAT:"json",RAG_ROOT:n};delete t.RAG_ENTRY;let a=dt();return new Promise((d,u)=>{st(process.execPath,[a,...i,"--root",n,"--client",r.client,"--project",r.project],{cwd:e,env:t,timeout:12e4,maxBuffer:1048576,windowsHide:!0},(l,c)=>{try{let o=JSON.parse(c.trim());if(!o||typeof o!="object"||Array.isArray(o))throw new Error("invalid result");d(o)}catch{u(new Error(l?.killed?"RAG command timed out.":"RAG command failed without a valid result."))}})})}import*as Ce from"node:fs";import*as Q from"node:path";import{DatabaseSync as ut}from"node:sqlite";function xe(e,n){let r=Q.join(Q.dirname(x(e,n.client)),"semantic-cache.sqlite");if(!Ce.existsSync(r))return{storedVectorRecords:0,vectorBytes:0};let i=new ut(r,{readOnly:!0});try{let t=i.prepare("SELECT COUNT(*) AS count, COALESCE(SUM(length(vector)), 0) AS bytes FROM vector_cache WHERE project = ?").get(n.project);return{storedVectorRecords:t.count,vectorBytes:t.bytes}}finally{i.close()}}import{z as w}from"zod";var ke=100,De=2e3,Be=1048576,ee=1e4,lt=10,mt=800,pt=65536,ft=200;function Ue(e){let n=new Set;for(let r of(e??"find,read").split(",").map(i=>i.trim())){if(r!=="find"&&r!=="read")throw new Error(`invalid_capability: Unknown RAG_CAPABILITIES value "${r}".`);n.add(r)}return n}var te=w.string().min(1).refine(e=>e.trim().length>0),re=w.object({client:te,project:te}).strict();function je(e){if(e===void 0)return null;let n;try{n=JSON.parse(e)}catch{throw new Error("invalid_namespace_allowlist: RAG_ALLOWED_NAMESPACES must be a JSON array.")}let r=w.array(re).safeParse(n);if(!r.success)throw new Error("invalid_namespace_allowlist: RAG_ALLOWED_NAMESPACES must be an array of client/project objects.");return r.data}function ie(e,n){return e===null||e.some(r=>r.client===n.client&&r.project===n.project)}function ne(e,n,r){return w.number().int().min(e).max(n).optional().transform(i=>i??r)}var Me=w.object({namespace:re,query:te,includeHistorical:w.boolean().optional().default(!1),matchMode:w.enum(["all","any"]).optional().default("all"),searchMode:w.enum(["lexical","semantic","hybrid"]).optional().default("lexical"),limit:ne(1,ke,lt),excerptChars:ne(1,De,mt)}).strict(),Xe=w.object({namespace:re,docVersionId:te,chunkOrdinal:w.number().multipleOf(1).min(0).refine(Number.isSafeInteger),maxBytes:ne(0,Be,pt),maxLines:ne(0,ee,ft),startLine:w.number().int().min(0).max(ee).optional(),startByte:w.number().int().min(0).max(Number.MAX_SAFE_INTEGER).optional()}).strict().refine(e=>e.startLine===void 0||e.startByte===void 0,{path:["startByte"],message:"startByte and startLine cannot be combined."});function Et(e){let n=e.find(a=>a.code==="unrecognized_keys"&&a.path.length===0)??e.find(a=>a.code==="unrecognized_keys");if(!n||n.code!=="unrecognized_keys")return null;let r=n.keys[0];if(r===void 0)return null;let i=n.path.map(String).join(".");return{code:"unknown_field",message:`Unknown field "${i.length>0?`${i}.${r}`:r}".`}}function k(e,n){return e.some(r=>r.path[0]===n)}function F(e,n,r){return{code:"invalid_limit",message:`${e} must be an integer from ${n} to ${r}.`}}function Fe(e,n){let r=Et(e);if(r)return r;if(k(e,"namespace"))return{code:"namespace_required",message:"namespace must contain non-empty client and project strings."};if(n==="find"){if(k(e,"query"))return{code:"invalid_query",message:"query must be a non-empty string."};if(k(e,"limit"))return F("limit",1,ke);if(k(e,"excerptChars"))return F("excerptChars",1,De)}else{if(e.some(i=>i.path[0]==="startByte"&&i.code==="custom"))return{code:"invalid_input",message:"startByte and startLine cannot be combined."};if(k(e,"docVersionId"))return{code:"invalid_input",message:"docVersionId must be a non-empty string."};if(k(e,"chunkOrdinal"))return{code:"invalid_input",message:"chunkOrdinal must be a non-negative integer."};if(k(e,"maxBytes"))return F("maxBytes",0,Be);if(k(e,"maxLines"))return F("maxLines",0,ee);if(k(e,"startLine"))return F("startLine",0,ee);if(k(e,"startByte"))return F("startByte",0,Number.MAX_SAFE_INTEGER)}return e.some(i=>i.path.length===0)?{code:"invalid_input",message:"Tool arguments must be an object."}:{code:"invalid_input",message:"Tool arguments are invalid."}}function _t(e){let n=re.safeParse(e);return n.success?n.data:null}function Ge(e){let n=Me.safeParse(e);return n.success?{ok:!0,input:n.data,namespace:n.data.namespace}:{ok:!1,namespace:oe(e),error:Fe(n.error.issues,"find")}}function Pe(e){let n=Xe.safeParse(e);return n.success?{ok:!0,input:n.data,namespace:n.data.namespace}:{ok:!1,namespace:oe(e),error:Fe(n.error.issues,"read")}}function ht(e){return typeof e=="object"&&e!==null&&!Array.isArray(e)}function oe(e){return ht(e)?_t(e.namespace):null}function $e(e){return w.toJSONSchema(e,{io:"input"})}var ae={name:"rag_find",description:"Search current document versions. Semantic or hybrid search sends indexed text to the explicitly configured provider; use matchMode any for broader lexical matching or includeHistorical for older versions.",inputSchema:$e(Me)},se={name:"rag_read",description:"Read bounded text from a cited document chunk in the explicit namespace. For exact pagination use startByte: 0, modest maxBytes and positive maxLines; repeat with nextByteOffset until null, concatenating text directly. Do not combine startByte and startLine.",inputSchema:$e(Xe)};function ue(e,n=!1){return{content:[{type:"text",text:JSON.stringify(e)}],details:e,...n?{isError:!0}:{}}}function D(e,n){return ue({namespace:e,error:n},!0)}function ce(e){return D(e,{code:"cancelled",message:"Tool call was cancelled."})}function Ve(e,n){let r=n instanceof Error?n.message:String(n);return D(oe(e),{code:"retrieval_failed",message:r})}var gt="The namespace travels on every call: never infer `client` or `project` from the cwd, repo name, config, env, or a placeholder like `default`. If the user has not stated it, ask before the first call.\nUse the RAG tool the role actually exposes; when only `rag_find` is available, delegate for deeper text instead of paraphrasing a catalog entry.\nWhen an excerpt is truncated, when the user asks whether the document covers something, or when a longer verbatim quote is needed, hand off to the subagent that exposes `rag_read` with the namespace and the citation's `docVersionId`; never assert absence from a catalog.";function Zt(e){e.on("before_agent_start",i=>{if(process.env.RAG_PROMPT==="off")return;let t=i.systemPromptOptions.sections??(i.systemPromptOptions.sections={});t.rag=gt});let n=Ue(process.env.RAG_CAPABILITIES),r=je(process.env.RAG_ALLOWED_NAMESPACES);e.registerCommand("rag",{description:"Search the document knowledge base",async handler(i,t){if(!t.hasUI)return;if(!n.has("find")){t.ui.notify("RAG search is not enabled for this session.","warning");return}let a=(await t.ui.input("RAG client"))?.trim();if(!a)return;let d=(await t.ui.input("RAG project"))?.trim();if(!d)return;let u={client:a,project:d};if(!ie(r,u)){t.ui.notify("Namespace is not enabled for this process.","error");return}let l=process.env.RAG_TUI_ADMIN==="1",c=["Search","Status","Documents","Review queue","Doctor",...l?["Import file","Import directory","Retire document","Backup","Restore"]:[]],o=await t.ui.select(`RAG \xB7 ${a} / ${d}`,c);if(!o||!c.includes(o))return;if(o!=="Search"){try{let T=de.resolve(t.cwd,G({cwd:t.cwd,homeDir:t.cwd,argv:[],env:process.env}).root),E=async f=>{let g=await we(t.cwd,T,u,f);if(g.error&&typeof g.error=="object"){let b=g.error;return t.ui.notify(`${b.code??"error"}: ${b.message??"Command failed."}`,"error"),null}return g.result&&typeof g.result=="object"?g.result:{}};if(o==="Backup"||o==="Restore"){let f=(await t.ui.input(o==="Backup"?"New backup destination":"Existing backup directory"))?.trim();if(!f)return;let g=de.resolve(t.cwd,f),b=o==="Backup"?`Source: ${T}
|
|
42
|
+
New destination: ${g}`:`Source: ${g}
|
|
43
|
+
Restore to empty root: ${T}`;if(!await t.ui.confirm(o,`${b}
|
|
44
|
+
Namespace: ${a} / ${d}`))return;let L=await E(o==="Backup"?["backup","--to",g]:["restore","--from",g]);L&&t.ui.notify(JSON.stringify(L),"info");return}if(o==="Import file"||o==="Import directory"){let f=(await t.ui.input(o==="Import file"?"File path":"Directory path"))?.trim();if(!f)return;let g=de.resolve(t.cwd,f);if(!await t.ui.confirm(o,`${g}
|
|
45
45
|
Destination: ${a} / ${d}
|
|
46
|
-
Root: ${
|
|
46
|
+
Root: ${T}`))return;let b=await E([o==="Import file"?"add":"import",g]);b&&t.ui.notify(JSON.stringify(b),"info");return}if(o==="Retire document"){let f=await E(["ls"]);if(!f||!Array.isArray(f.documents)||!f.documents.length){f&&t.ui.notify("No active documents.","info");return}let g=f.documents,b=g.map((A,C)=>`${C+1}. ${A.label} \xB7 ${A.documentId}`),L=await t.ui.select("Retire a document",b);if(!L)return;let I=g[b.indexOf(L)];if(!I||!await t.ui.confirm("Retire document",`${I.label}
|
|
47
47
|
${I.documentId}
|
|
48
|
-
${a} / ${d}`))return;let R=await
|
|
49
|
-
${
|
|
48
|
+
${a} / ${d}`))return;let R=await E(["rm",I.documentId]);R&&t.ui.notify(JSON.stringify(R),"info");return}let h=await E(o==="Status"?["status"]:o==="Documents"?["ls"]:o==="Review queue"?["review","list"]:["doctor"]);if(!h)return;if(o==="Status"){t.ui.notify(JSON.stringify({...h,semanticCache:xe(T,u)}),"info");return}if(o==="Documents"&&Array.isArray(h.documents)){let f=h.documents;if(!f.length){t.ui.notify("No active documents.","info");return}await t.ui.select(`Documents (${f.length})`,f.map((g,b)=>`${b+1}. ${g.label} \xB7 ${g.versionLabel??"current"} \xB7 ${g.documentId}`));return}if(o==="Review queue"&&Array.isArray(h.candidates)){let f=h.candidates;if(!f.length){t.ui.notify("No review candidates.","info");return}let g=f.map((A,C)=>`${C+1}. ${A.fromDocVersionId} \u2194 ${A.toDocVersionId}`),b=await t.ui.select(`Review candidates (${f.length})`,g);if(!b||!l)return;let L=f[g.indexOf(b)];if(!L)return;let I=await t.ui.select("Human review decision",["same_entity","conflicts_with"]);if(!I||!["same_entity","conflicts_with"].includes(I)||!await t.ui.confirm("Record review decision",`${L.fromDocVersionId}
|
|
49
|
+
${L.toDocVersionId}
|
|
50
50
|
${I}
|
|
51
|
-
${a} / ${d}`))return;let R=await
|
|
51
|
+
${a} / ${d}`))return;let R=await E(["review","link",L.fromDocVersionId,L.toDocVersionId,"--type",I]);R&&t.ui.notify(JSON.stringify(R),"info");return}t.ui.notify(JSON.stringify(h),"info")}catch(T){t.ui.notify(T instanceof Error?T.message:String(T),"error")}return}let m=(await t.ui.input("Search documents"))?.trim();if(!m)return;let p=await t.ui.select("Retrieval method",["Lexical","Semantic","Hybrid"]);if(!p||!["Lexical","Semantic","Hybrid"].includes(p))return;let s=await t.ui.select("Match query terms",["All terms","Any term"]);if(!s||!["All terms","Any term"].includes(s))return;let _=await t.ui.select("Document versions",["Current versions","Include historical versions"]);if(!(!_||!["Current versions","Include historical versions"].includes(_)))try{let T=G({cwd:t.cwd,homeDir:t.cwd,argv:[],env:process.env}).root,E=_==="Include historical versions",y=s==="Any term"?"any":"all",h=p==="Lexical"?P({root:T,namespace:u,query:m,excerptLength:240,matchMode:y,includeHistorical:E,maxResults:10}):await Te({root:T,namespace:u,query:m,excerptLength:240,includeHistorical:E});if("error"in h&&h.error){t.ui.notify(`${h.error.code}: ${h.error.message}`,"error");return}let f=h.result.citations;if(p==="Hybrid"){let A=P({root:T,namespace:u,query:m,excerptLength:240,matchMode:y,includeHistorical:E,maxResults:100});if("error"in A){t.ui.notify(`${A.error.code}: ${A.error.message}`,"error");return}f=Ne(A.result.citations,f)}let g=f.slice(0,10);if(g.length===0){t.ui.notify("No matches. This does not establish that the information is absent.","info");return}let b=g.map((A,C)=>`${C+1}. ${A.label} [${A.versionLabel??A.docVersionId}; ${A.isCurrent?"current":"historical"}] \u2014 ${A.excerpt.replace(/\s+/g," ")}`),L=await t.ui.select(`RAG results (${h.result.total} matches)`,b);if(!L)return;let I=g[b.indexOf(L)];if(!I)return;if(!n.has("read")){t.ui.notify("Full reading is not enabled for this session.","info");return}let R=await he({root:T,namespace:u,docVersionId:I.docVersionId,chunkOrdinal:I.chunkOrdinal,maxBytes:4096,maxLines:40});"error"in R?t.ui.notify(`${R.error.code}: ${R.error.message}`,"error"):t.ui.notify(`${I.label} \xB7 ${I.docVersionId}
|
|
52
52
|
${R.result.text}${R.result.truncated?`
|
|
53
|
-
[truncated]`:""}`,"info")}catch(
|
|
53
|
+
[truncated]`:""}`,"info")}catch(T){t.ui.notify(T instanceof Error?T.message:String(T),"error")}}}),n.has("find")&&e.registerTool({name:ae.name,label:"RAG Find",description:ae.description,parameters:ae.inputSchema,async execute(i,t,a,d,u){try{let l=Ge(t);if(!l.ok)return D(l.namespace,l.error);let{namespace:c,query:o,limit:m,excerptChars:p,includeHistorical:s,matchMode:_,searchMode:T}=l.input;if(!ie(r,c))return D(c,{code:"namespace_denied",message:"Namespace is not enabled for this process."});if(a?.aborted)return ce(c);let E=G({cwd:u.cwd,homeDir:u.cwd,argv:[],env:process.env}).root,y=T!=="lexical"?await Te({root:E,namespace:c,query:o,excerptLength:p,includeHistorical:s},process.env,a):P({root:E,namespace:c,query:o,excerptLength:p,includeHistorical:s,matchMode:_,maxResults:m});if(a?.aborted)return ce(c);if("error"in y&&y.error)return D(c,y.error);if(T==="hybrid"){let h=P({root:E,namespace:c,query:o,excerptLength:p,includeHistorical:s,matchMode:_,maxResults:100});if("error"in h)return D(c,h.error);let f=Ne(h.result.citations,y.result.citations);return ue({namespace:c,result:{citations:f.slice(0,m),total:y.result.total,searchMode:"hybrid"}})}return ue({namespace:c,result:{...y.result,citations:y.result.citations.slice(0,m)}})}catch(l){return Ve(t,l)}}}),n.has("read")&&e.registerTool({name:se.name,label:"RAG Read",description:se.description,parameters:se.inputSchema,async execute(i,t,a,d,u){try{let l=Pe(t);if(!l.ok)return D(l.namespace,l.error);let{namespace:c,docVersionId:o,chunkOrdinal:m,maxBytes:p,maxLines:s,startLine:_,startByte:T}=l.input;if(!ie(r,c))return D(c,{code:"namespace_denied",message:"Namespace is not enabled for this process."});if(a?.aborted)return ce(c);let E=G({cwd:u.cwd,homeDir:u.cwd,argv:[],env:process.env}).root,y=await he({root:E,namespace:c,docVersionId:o,chunkOrdinal:m,maxBytes:p,maxLines:s,startLine:_,startByte:T});return a?.aborted?ce(c):"error"in y?D(c,y.error):ue({namespace:c,result:y.result})}catch(l){return Ve(t,l)}}})}export{Zt as default};
|
|
@@ -41,7 +41,7 @@ function countLines(text) {
|
|
|
41
41
|
}
|
|
42
42
|
return lines;
|
|
43
43
|
}
|
|
44
|
-
function limitLines(text, maxLines) {
|
|
44
|
+
function limitLines(text, maxLines, includeSeparator = false) {
|
|
45
45
|
if (text.length === 0) return { text, truncated: false };
|
|
46
46
|
if (maxLines === 0) return { text: "", truncated: true };
|
|
47
47
|
let lines = 1;
|
|
@@ -49,14 +49,14 @@ function limitLines(text, maxLines) {
|
|
|
49
49
|
if (text[index] !== "\r" && text[index] !== "\n") continue;
|
|
50
50
|
const separatorStart = index;
|
|
51
51
|
if (text[index] === "\r" && text[index + 1] === "\n") index += 1;
|
|
52
|
-
if (lines === maxLines) return { text: text.slice(0, separatorStart), truncated: true };
|
|
52
|
+
if (lines === maxLines) return { text: text.slice(0, includeSeparator ? index + 1 : separatorStart), truncated: true };
|
|
53
53
|
lines += 1;
|
|
54
54
|
}
|
|
55
55
|
return { text, truncated: false };
|
|
56
56
|
}
|
|
57
57
|
async function readFromCitation(input) {
|
|
58
|
-
const { root, namespace, docVersionId, chunkOrdinal, maxBytes, maxLines, startLine = 0 } = input;
|
|
59
|
-
if (!Number.isInteger(maxBytes) || maxBytes < 0 || !Number.isInteger(maxLines) || maxLines < 0 || !Number.isInteger(startLine) || startLine < 0) {
|
|
58
|
+
const { root, namespace, docVersionId, chunkOrdinal, maxBytes, maxLines, startLine = 0, startByte } = input;
|
|
59
|
+
if (!Number.isInteger(maxBytes) || maxBytes < 0 || !Number.isInteger(maxLines) || maxLines < 0 || !Number.isInteger(startLine) || startLine < 0 || startByte !== void 0 && (!Number.isSafeInteger(startByte) || startByte < 0) || startByte !== void 0 && input.startLine !== void 0) {
|
|
60
60
|
return failure(namespace, "invalid_read_limit", "maxBytes, maxLines, and startLine must be non-negative integers.");
|
|
61
61
|
}
|
|
62
62
|
let db;
|
|
@@ -75,10 +75,16 @@ async function readFromCitation(input) {
|
|
|
75
75
|
JOIN client AS cl ON cl.client_id = p.client_id
|
|
76
76
|
WHERE c.doc_version_id = ? AND c.ordinal = ? AND cl.name = ? AND p.slug = ?`).get(docVersionId, chunkOrdinal, namespace.client, namespace.project);
|
|
77
77
|
if (!citation) return failure(namespace, "citation_not_found", "The citation does not exist in the requested document version and namespace.");
|
|
78
|
-
const
|
|
78
|
+
const chunkBuffer = Buffer.from(citation.text, "utf8");
|
|
79
|
+
const offset = startByte ?? Buffer.byteLength(citation.text.slice(0, startLineOffset(citation.text, startLine)), "utf8");
|
|
80
|
+
if (offset > chunkBuffer.length || offset < chunkBuffer.length && (chunkBuffer[offset] & 192) === 128) {
|
|
81
|
+
return failure(namespace, "invalid_start_byte", "startByte must be a UTF-8 character boundary within the chunk.");
|
|
82
|
+
}
|
|
83
|
+
const chunkText = chunkBuffer.subarray(offset).toString("utf8");
|
|
79
84
|
const byteLimited = limitUtf8(chunkText, maxBytes);
|
|
80
|
-
const lineLimited = limitLines(byteLimited.text, maxLines);
|
|
81
|
-
const
|
|
85
|
+
const lineLimited = limitLines(byteLimited.text, maxLines, startByte !== void 0);
|
|
86
|
+
const lineCut = lineLimited.truncated && (startByte === void 0 || lineLimited.text.length < chunkText.length);
|
|
87
|
+
const truncatedBy = lineCut ? "lines" : byteLimited.truncated ? "bytes" : null;
|
|
82
88
|
const totalChunks = citation.totalChunks;
|
|
83
89
|
return {
|
|
84
90
|
namespace,
|
|
@@ -89,8 +95,9 @@ async function readFromCitation(input) {
|
|
|
89
95
|
nextChunkOrdinal: chunkOrdinal + 1 < totalChunks ? chunkOrdinal + 1 : null,
|
|
90
96
|
chunkBytes: Buffer.byteLength(citation.text, "utf8"),
|
|
91
97
|
returnedBytes: Buffer.byteLength(lineLimited.text, "utf8"),
|
|
92
|
-
returnedLines: countLines(lineLimited.text),
|
|
93
|
-
truncatedBy
|
|
98
|
+
returnedLines: countLines(lineLimited.text) - (startByte !== void 0 && /[\r\n]$/.test(lineLimited.text) ? 1 : 0),
|
|
99
|
+
truncatedBy,
|
|
100
|
+
nextByteOffset: truncatedBy === null ? null : offset + Buffer.byteLength(lineLimited.text, "utf8")
|
|
94
101
|
}
|
|
95
102
|
};
|
|
96
103
|
} catch (error) {
|
package/dist/src/server.mjs
CHANGED
|
@@ -60,9 +60,9 @@ async function main() {
|
|
|
60
60
|
}
|
|
61
61
|
const check = validateRagReadInput(args);
|
|
62
62
|
if (!check.ok) return toolError(check.namespace, check.error);
|
|
63
|
-
const { namespace, docVersionId, chunkOrdinal, maxBytes, maxLines, startLine } = check.input;
|
|
63
|
+
const { namespace, docVersionId, chunkOrdinal, maxBytes, maxLines, startLine, startByte } = check.input;
|
|
64
64
|
if (!isNamespaceAllowed(allowlist, namespace)) return toolError(namespace, { code: "namespace_denied", message: "Namespace is not enabled for this process." });
|
|
65
|
-
const read = await readFromCitation({ root, namespace, docVersionId, chunkOrdinal, maxBytes, maxLines, startLine });
|
|
65
|
+
const read = await readFromCitation({ root, namespace, docVersionId, chunkOrdinal, maxBytes, maxLines, startLine, startByte });
|
|
66
66
|
if ("error" in read) return toolError(namespace, read.error);
|
|
67
67
|
return toolResult({ namespace, result: read.result });
|
|
68
68
|
} catch (error) {
|
|
@@ -55,8 +55,12 @@ const ragReadInputSchema = z.object({
|
|
|
55
55
|
chunkOrdinal: z.number().multipleOf(1).min(0).refine(Number.isSafeInteger),
|
|
56
56
|
maxBytes: defaultedInteger(0, MAX_BYTES, DEFAULT_MAX_BYTES),
|
|
57
57
|
maxLines: defaultedInteger(0, MAX_LINES, DEFAULT_MAX_LINES),
|
|
58
|
-
startLine:
|
|
59
|
-
|
|
58
|
+
startLine: z.number().int().min(0).max(MAX_LINES).optional(),
|
|
59
|
+
startByte: z.number().int().min(0).max(Number.MAX_SAFE_INTEGER).optional()
|
|
60
|
+
}).strict().refine((value) => value.startLine === void 0 || value.startByte === void 0, {
|
|
61
|
+
path: ["startByte"],
|
|
62
|
+
message: "startByte and startLine cannot be combined."
|
|
63
|
+
});
|
|
60
64
|
function unknownField(issues) {
|
|
61
65
|
const issue = issues.find((item) => item.code === "unrecognized_keys" && item.path.length === 0) ?? issues.find((item) => item.code === "unrecognized_keys");
|
|
62
66
|
if (!issue || issue.code !== "unrecognized_keys") return null;
|
|
@@ -88,6 +92,9 @@ function inputFailure(issues, kind) {
|
|
|
88
92
|
if (hasIssue(issues, "limit")) return limitFailure("limit", 1, MAX_LIMIT);
|
|
89
93
|
if (hasIssue(issues, "excerptChars")) return limitFailure("excerptChars", 1, MAX_EXCERPT_CHARS);
|
|
90
94
|
} else {
|
|
95
|
+
if (issues.some((issue) => issue.path[0] === "startByte" && issue.code === "custom")) {
|
|
96
|
+
return { code: "invalid_input", message: "startByte and startLine cannot be combined." };
|
|
97
|
+
}
|
|
91
98
|
if (hasIssue(issues, "docVersionId")) {
|
|
92
99
|
return { code: "invalid_input", message: "docVersionId must be a non-empty string." };
|
|
93
100
|
}
|
|
@@ -97,6 +104,7 @@ function inputFailure(issues, kind) {
|
|
|
97
104
|
if (hasIssue(issues, "maxBytes")) return limitFailure("maxBytes", 0, MAX_BYTES);
|
|
98
105
|
if (hasIssue(issues, "maxLines")) return limitFailure("maxLines", 0, MAX_LINES);
|
|
99
106
|
if (hasIssue(issues, "startLine")) return limitFailure("startLine", 0, MAX_LINES);
|
|
107
|
+
if (hasIssue(issues, "startByte")) return limitFailure("startByte", 0, Number.MAX_SAFE_INTEGER);
|
|
100
108
|
}
|
|
101
109
|
if (issues.some((issue) => issue.path.length === 0)) {
|
|
102
110
|
return { code: "invalid_input", message: "Tool arguments must be an object." };
|
|
@@ -145,7 +153,7 @@ const RAG_FIND_TOOL = {
|
|
|
145
153
|
};
|
|
146
154
|
const RAG_READ_TOOL = {
|
|
147
155
|
name: "rag_read",
|
|
148
|
-
description: "Read bounded text from a cited document chunk in the explicit namespace.",
|
|
156
|
+
description: "Read bounded text from a cited document chunk in the explicit namespace. For exact pagination use startByte: 0, modest maxBytes and positive maxLines; repeat with nextByteOffset until null, concatenating text directly. Do not combine startByte and startLine.",
|
|
149
157
|
inputSchema: advertisedInputSchema(ragReadInputSchema)
|
|
150
158
|
};
|
|
151
159
|
const RAG_TOOLS = [RAG_FIND_TOOL, RAG_READ_TOOL];
|
|
@@ -0,0 +1,296 @@
|
|
|
1
|
+
import * as fs from "node:fs";
|
|
2
|
+
import * as fsp from "node:fs/promises";
|
|
3
|
+
import * as path from "node:path";
|
|
4
|
+
import { createHash } from "node:crypto";
|
|
5
|
+
import { homedir } from "node:os";
|
|
6
|
+
import { createInterface } from "node:readline/promises";
|
|
7
|
+
import { DatabaseSync } from "node:sqlite";
|
|
8
|
+
import { fileURLToPath } from "node:url";
|
|
9
|
+
import { discoverConfig } from "../config/discovery.mjs";
|
|
10
|
+
import { runCli } from "../cli/index.mjs";
|
|
11
|
+
import { findInNamespace } from "../retrieve/find.mjs";
|
|
12
|
+
import { readFromCitation } from "../retrieve/read.mjs";
|
|
13
|
+
import { clientDbPath } from "../store/db.mjs";
|
|
14
|
+
import { objectRelPath } from "../store/objects.mjs";
|
|
15
|
+
function safe(value) {
|
|
16
|
+
return String(value ?? "").replace(/[\x00-\x1f\x7f-\x9f]/g, " ");
|
|
17
|
+
}
|
|
18
|
+
function displayText(value) {
|
|
19
|
+
return value.replace(/[\x00-\x08\x0b-\x1f\x7f-\x9f]/g, " ");
|
|
20
|
+
}
|
|
21
|
+
function database(root, client) {
|
|
22
|
+
const file = clientDbPath(root, client);
|
|
23
|
+
if (!fs.existsSync(file)) throw new Error(`Database not found: ${file}`);
|
|
24
|
+
return new DatabaseSync(file, { readOnly: true });
|
|
25
|
+
}
|
|
26
|
+
function listNamespaces(root) {
|
|
27
|
+
const clients = path.join(root, "clients");
|
|
28
|
+
if (!fs.existsSync(clients)) return [];
|
|
29
|
+
const output = [];
|
|
30
|
+
for (const item of fs.readdirSync(clients, { withFileTypes: true })) {
|
|
31
|
+
if (!item.isDirectory() || !fs.existsSync(path.join(clients, item.name, "rag.db"))) continue;
|
|
32
|
+
const db = database(root, item.name);
|
|
33
|
+
try {
|
|
34
|
+
const rows = db.prepare(`SELECT c.name AS client, p.slug AS project,
|
|
35
|
+
(SELECT count(*) FROM document d WHERE d.project_id = p.project_id AND d.deleted_at IS NULL) AS documents
|
|
36
|
+
FROM project p JOIN client c ON c.client_id = p.client_id WHERE c.name = ? ORDER BY p.slug`).all(item.name);
|
|
37
|
+
output.push(...rows.map((row) => ({ ...row })));
|
|
38
|
+
} finally {
|
|
39
|
+
db.close();
|
|
40
|
+
}
|
|
41
|
+
}
|
|
42
|
+
return output.sort((a, b) => a.client.localeCompare(b.client) || a.project.localeCompare(b.project));
|
|
43
|
+
}
|
|
44
|
+
function listDocuments(root, ns) {
|
|
45
|
+
const db = database(root, ns.client);
|
|
46
|
+
try {
|
|
47
|
+
return db.prepare(`SELECT d.document_id AS documentId, d.label, e.entity_key AS entity, d.deleted_at AS deleted
|
|
48
|
+
FROM document d JOIN entity e ON e.entity_id=d.entity_id JOIN project p ON p.project_id=d.project_id
|
|
49
|
+
JOIN client c ON c.client_id=p.client_id WHERE c.name=? AND p.slug=? ORDER BY d.label, d.document_id`).all(ns.client, ns.project);
|
|
50
|
+
} finally {
|
|
51
|
+
db.close();
|
|
52
|
+
}
|
|
53
|
+
}
|
|
54
|
+
function listVersions(root, ns, documentId) {
|
|
55
|
+
const db = database(root, ns.client);
|
|
56
|
+
try {
|
|
57
|
+
return db.prepare(`SELECT v.doc_version_id AS id, v.sha256, v.version_label AS label, v.is_current AS current, v.ingested_at AS date
|
|
58
|
+
FROM doc_version v JOIN document d ON d.document_id=v.document_id JOIN project p ON p.project_id=d.project_id
|
|
59
|
+
JOIN client c ON c.client_id=p.client_id WHERE c.name=? AND p.slug=? AND d.document_id=? ORDER BY v.is_current DESC, v.ingested_at DESC, v.doc_version_id`).all(ns.client, ns.project, documentId);
|
|
60
|
+
} finally {
|
|
61
|
+
db.close();
|
|
62
|
+
}
|
|
63
|
+
}
|
|
64
|
+
async function sha256(file) {
|
|
65
|
+
const hash = createHash("sha256");
|
|
66
|
+
for await (const data of fs.createReadStream(file)) hash.update(data);
|
|
67
|
+
return hash.digest("hex");
|
|
68
|
+
}
|
|
69
|
+
async function exportVersion(root, ns, documentId, versionId, destination) {
|
|
70
|
+
const version = listVersions(root, ns, documentId).find((v) => v.id === versionId);
|
|
71
|
+
if (!version) throw new Error("Document version not found in the selected project.");
|
|
72
|
+
const source = path.join(root, objectRelPath(version.sha256));
|
|
73
|
+
await fsp.copyFile(source, destination, fs.constants.COPYFILE_EXCL);
|
|
74
|
+
try {
|
|
75
|
+
if (await sha256(destination) !== version.sha256) throw new Error("Export failed: stored object checksum does not match.");
|
|
76
|
+
} catch (error) {
|
|
77
|
+
await fsp.unlink(destination).catch(() => {
|
|
78
|
+
});
|
|
79
|
+
throw error;
|
|
80
|
+
}
|
|
81
|
+
}
|
|
82
|
+
async function choose(io, title, entries) {
|
|
83
|
+
io.print(`
|
|
84
|
+
${title}`);
|
|
85
|
+
entries.forEach((entry, index) => io.print(` ${index + 1}. ${safe(entry)}`));
|
|
86
|
+
io.print(" 0. Back");
|
|
87
|
+
for (; ; ) {
|
|
88
|
+
const value = (await io.ask("> ")).trim();
|
|
89
|
+
if (value === "0" || value === "") return null;
|
|
90
|
+
if (/^[1-9]\d*$/.test(value) && Number(value) <= entries.length) return Number(value) - 1;
|
|
91
|
+
io.print("Choose a listed number or 0.");
|
|
92
|
+
}
|
|
93
|
+
}
|
|
94
|
+
async function confirm(io, text) {
|
|
95
|
+
return (await io.ask(`${text} [y/N] `)).trim().toLowerCase() === "y";
|
|
96
|
+
}
|
|
97
|
+
async function askPath(io, message, cwd) {
|
|
98
|
+
const input = (await io.ask(`${message} (blank to cancel): `)).trim().replace(/^"(.*)"$/, "$1");
|
|
99
|
+
return input ? path.resolve(cwd, input) : null;
|
|
100
|
+
}
|
|
101
|
+
async function readPages(io, root, ns, version, initialChunk = 0) {
|
|
102
|
+
let chunk = initialChunk, offset = 0;
|
|
103
|
+
const history = [];
|
|
104
|
+
for (; ; ) {
|
|
105
|
+
const result = await readFromCitation({ root, namespace: ns, docVersionId: version, chunkOrdinal: chunk, startByte: offset, maxBytes: 4096, maxLines: 25 });
|
|
106
|
+
if ("error" in result) {
|
|
107
|
+
io.print(`Error: ${safe(result.error.message)}`);
|
|
108
|
+
return;
|
|
109
|
+
}
|
|
110
|
+
io.print(`
|
|
111
|
+
--- Chunk ${chunk + 1}/${result.result.totalChunks} \xB7 byte ${offset}/${result.result.chunkBytes} ---`);
|
|
112
|
+
io.print(displayText(result.result.text));
|
|
113
|
+
const action = (await io.ask("[n]ext, [p]revious, [q] back: ")).trim().toLowerCase();
|
|
114
|
+
if (action === "q" || action === "") return;
|
|
115
|
+
if (action === "p") {
|
|
116
|
+
const previous = history.pop();
|
|
117
|
+
if (previous) ({ chunk, offset } = previous);
|
|
118
|
+
continue;
|
|
119
|
+
}
|
|
120
|
+
if (action !== "n") continue;
|
|
121
|
+
if (result.result.nextByteOffset !== null) {
|
|
122
|
+
if (result.result.nextByteOffset <= offset) {
|
|
123
|
+
io.print("Cannot advance the page.");
|
|
124
|
+
return;
|
|
125
|
+
}
|
|
126
|
+
history.push({ chunk, offset });
|
|
127
|
+
offset = result.result.nextByteOffset;
|
|
128
|
+
} else if (result.result.nextChunkOrdinal !== null) {
|
|
129
|
+
history.push({ chunk, offset });
|
|
130
|
+
chunk = result.result.nextChunkOrdinal;
|
|
131
|
+
offset = 0;
|
|
132
|
+
} else io.print("End of document.");
|
|
133
|
+
}
|
|
134
|
+
}
|
|
135
|
+
async function browse(io, root, ns, cwd) {
|
|
136
|
+
for (; ; ) {
|
|
137
|
+
const docs = listDocuments(root, ns);
|
|
138
|
+
const choice = await choose(io, `Documents (${docs.length}; includes retired)`, docs.map((d) => `${d.label} \xB7 ${d.entity}${d.deleted ? " \xB7 retired" : ""}`));
|
|
139
|
+
if (choice === null) return;
|
|
140
|
+
const doc = docs[choice];
|
|
141
|
+
for (; ; ) {
|
|
142
|
+
const versions = listVersions(root, ns, doc.documentId);
|
|
143
|
+
const selected = await choose(io, `${safe(doc.label)} \xB7 versions`, versions.map((v) => `${v.current ? "current" : "historical"} \xB7 ${v.label ?? "unlabeled"} \xB7 ${v.date} \xB7 ${v.id}`));
|
|
144
|
+
if (selected === null) break;
|
|
145
|
+
const version = versions[selected];
|
|
146
|
+
const action = await choose(io, `Version ${safe(version.id)}`, ["Read extracted text", "Download original", "Relabel document", "Retire document"]);
|
|
147
|
+
if (action === 0) await readPages(io, root, ns, version.id);
|
|
148
|
+
if (action === 1) {
|
|
149
|
+
const destination = await askPath(io, "Destination file (must not exist)", cwd);
|
|
150
|
+
if (destination) {
|
|
151
|
+
await exportVersion(root, ns, doc.documentId, version.id, destination);
|
|
152
|
+
io.print(`Saved: ${safe(destination)}`);
|
|
153
|
+
}
|
|
154
|
+
}
|
|
155
|
+
if (action === 2) {
|
|
156
|
+
const label = (await io.ask("New label (blank to cancel): ")).trim();
|
|
157
|
+
if (label && await confirm(io, `Relabel ${safe(doc.label)} in ${safe(ns.client)}/${safe(ns.project)}?`)) await command(io, root, ns, cwd, ["relabel", doc.documentId, "--label", label]);
|
|
158
|
+
}
|
|
159
|
+
if (action === 3 && !doc.deleted && await confirm(io, `Retire ${safe(doc.label)} in ${safe(ns.client)}/${safe(ns.project)}?`)) {
|
|
160
|
+
await command(io, root, ns, cwd, ["rm", doc.documentId]);
|
|
161
|
+
break;
|
|
162
|
+
}
|
|
163
|
+
}
|
|
164
|
+
}
|
|
165
|
+
}
|
|
166
|
+
async function command(io, root, ns, cwd, args) {
|
|
167
|
+
const result = await runCli({ argv: [...args, "--root", root, "--client", ns.client, "--project", ns.project], env: process.env, cwd });
|
|
168
|
+
io.print(displayText(JSON.stringify(result.output, null, 2)));
|
|
169
|
+
}
|
|
170
|
+
async function search(io, root, ns) {
|
|
171
|
+
const query = (await io.ask("Search terms (blank to cancel): ")).trim();
|
|
172
|
+
if (!query) return;
|
|
173
|
+
const found = findInNamespace({ root, namespace: ns, query, maxResults: 100 });
|
|
174
|
+
if ("error" in found) {
|
|
175
|
+
io.print(`Error: ${safe(found.error.message)}`);
|
|
176
|
+
return;
|
|
177
|
+
}
|
|
178
|
+
io.print(`${found.result.total} matching chunks; showing up to 100. No match does not prove absence.`);
|
|
179
|
+
const selected = await choose(io, "Search results", found.result.citations.map((c) => `${c.label} \xB7 ${c.isCurrent ? "current" : "historical"} \xB7 ${c.excerpt}`));
|
|
180
|
+
if (selected !== null) {
|
|
181
|
+
const citation = found.result.citations[selected];
|
|
182
|
+
await readPages(io, root, ns, citation.docVersionId, citation.chunkOrdinal);
|
|
183
|
+
}
|
|
184
|
+
}
|
|
185
|
+
async function review(io, root, ns, cwd) {
|
|
186
|
+
const response = await runCli({ argv: ["review", "list", "--root", root, "--client", ns.client, "--project", ns.project], env: process.env, cwd });
|
|
187
|
+
const envelope = response.output;
|
|
188
|
+
if (response.exitCode !== 0) {
|
|
189
|
+
io.print(displayText(JSON.stringify(response.output, null, 2)));
|
|
190
|
+
return;
|
|
191
|
+
}
|
|
192
|
+
const candidates = envelope.result?.candidates ?? [];
|
|
193
|
+
if (!candidates.length) {
|
|
194
|
+
io.print("No pending review candidates.");
|
|
195
|
+
return;
|
|
196
|
+
}
|
|
197
|
+
const selected = await choose(io, "Review candidates", candidates.map((c) => `${c.fromDocVersionId} / ${c.toDocVersionId}`));
|
|
198
|
+
if (selected === null) return;
|
|
199
|
+
const relation = await choose(io, "Decision", ["same_entity", "conflicts_with"]);
|
|
200
|
+
if (relation === null) return;
|
|
201
|
+
const candidate = candidates[selected], type = ["same_entity", "conflicts_with"][relation];
|
|
202
|
+
if (await confirm(io, `Record ${type} for ${candidate.fromDocVersionId} / ${candidate.toDocVersionId}?`)) {
|
|
203
|
+
await command(io, root, ns, cwd, ["review", "link", candidate.fromDocVersionId, candidate.toDocVersionId, "--type", type]);
|
|
204
|
+
}
|
|
205
|
+
}
|
|
206
|
+
async function runTui(io, root, cwd) {
|
|
207
|
+
let ns = null;
|
|
208
|
+
for (; ; ) {
|
|
209
|
+
io.print(`
|
|
210
|
+
AGENT RAG \xB7 root: ${safe(root)} \xB7 project: ${ns ? `${safe(ns.client)} / ${safe(ns.project)}` : "none"}`);
|
|
211
|
+
const action = await choose(io, "Main menu", ["Select client/project", "Create/select a new project", "Browse documents and versions", "Search", "Import file", "Import directory", "Status", "Doctor", "Review queue", "Backup database", "Restore backup", "Reindex", "Rehash preview", "Rename entity", "Apply rehash", "Exit"]);
|
|
212
|
+
if (action === null || action === 15) return;
|
|
213
|
+
try {
|
|
214
|
+
if (action === 0) {
|
|
215
|
+
const namespaces = listNamespaces(root);
|
|
216
|
+
const selected = await choose(io, "Clients and projects", namespaces.map((x) => `${x.client} / ${x.project} \xB7 ${x.documents} documents`));
|
|
217
|
+
if (selected !== null) ns = { client: namespaces[selected].client, project: namespaces[selected].project };
|
|
218
|
+
continue;
|
|
219
|
+
}
|
|
220
|
+
if (action === 1) {
|
|
221
|
+
const client = (await io.ask("Client (blank to cancel): ")).trim();
|
|
222
|
+
if (!client) continue;
|
|
223
|
+
const project = (await io.ask("Project (blank to cancel): ")).trim();
|
|
224
|
+
if (!project) continue;
|
|
225
|
+
clientDbPath(root, client);
|
|
226
|
+
ns = { client, project };
|
|
227
|
+
io.print("Project selected. Importing a document will create it if needed.");
|
|
228
|
+
continue;
|
|
229
|
+
}
|
|
230
|
+
if (!ns) {
|
|
231
|
+
io.print("Select a client and project first.");
|
|
232
|
+
continue;
|
|
233
|
+
}
|
|
234
|
+
if (action === 2) await browse(io, root, ns, cwd);
|
|
235
|
+
if (action === 3) await search(io, root, ns);
|
|
236
|
+
if (action === 4 || action === 5) {
|
|
237
|
+
const input = await askPath(io, action === 4 ? "File to import" : "Directory to import", cwd);
|
|
238
|
+
if (input && await confirm(io, `Import ${safe(input)} into ${safe(ns.client)}/${safe(ns.project)}?`)) await command(io, root, ns, cwd, [action === 4 ? "add" : "import", input]);
|
|
239
|
+
}
|
|
240
|
+
if (action === 6) await command(io, root, ns, cwd, ["status"]);
|
|
241
|
+
if (action === 7) await command(io, root, ns, cwd, ["doctor"]);
|
|
242
|
+
if (action === 8) await review(io, root, ns, cwd);
|
|
243
|
+
if (action === 9 || action === 10) {
|
|
244
|
+
const input = await askPath(io, action === 9 ? "New backup directory" : "Existing backup directory", cwd);
|
|
245
|
+
const description = action === 9 ? `Back up client ${safe(ns.client)} and all its projects from ${safe(root)} to ${safe(input)}?` : `Restore client ${safe(ns.client)} from ${safe(input)} into empty root ${safe(root)}?`;
|
|
246
|
+
if (input && await confirm(io, description)) await command(io, root, ns, cwd, [action === 9 ? "backup" : "restore", action === 9 ? "--to" : "--from", input]);
|
|
247
|
+
}
|
|
248
|
+
if (action === 11 && await confirm(io, `Rebuild the database for ${safe(ns.client)} in ${safe(root)}?`)) await command(io, root, ns, cwd, ["reindex"]);
|
|
249
|
+
if (action === 12) await command(io, root, ns, cwd, ["rehash"]);
|
|
250
|
+
if (action === 13) {
|
|
251
|
+
const from = (await io.ask("Current entity key (blank to cancel): ")).trim();
|
|
252
|
+
if (!from) continue;
|
|
253
|
+
const to = (await io.ask("New entity key (blank to cancel): ")).trim();
|
|
254
|
+
if (!to) continue;
|
|
255
|
+
if (await confirm(io, `Rename ${safe(from)} to ${safe(to)} in ${safe(ns.client)}/${safe(ns.project)}?`)) await command(io, root, ns, cwd, ["entity", "rename", from, to]);
|
|
256
|
+
}
|
|
257
|
+
if (action === 14 && await confirm(io, `Apply rehash to ${safe(ns.client)}/${safe(ns.project)}? Preview and back up first.`)) await command(io, root, ns, cwd, ["rehash", "--apply"]);
|
|
258
|
+
} catch (error) {
|
|
259
|
+
io.print(`Error: ${safe(error instanceof Error ? error.message : error)}`);
|
|
260
|
+
}
|
|
261
|
+
}
|
|
262
|
+
}
|
|
263
|
+
async function main() {
|
|
264
|
+
const args = process.argv.slice(2);
|
|
265
|
+
if (args.includes("--help")) {
|
|
266
|
+
console.log("Usage: agent-rag [--root <path>]\nInteractive knowledge-base browser and administrator.");
|
|
267
|
+
return;
|
|
268
|
+
}
|
|
269
|
+
if (!(args.length === 0 || args.length === 2 && args[0] === "--root" && args[1])) throw new Error("Usage: agent-rag [--root <path>]");
|
|
270
|
+
if (!process.stdin.isTTY || !process.stdout.isTTY) throw new Error("agent-rag requires an interactive terminal. Use rag for scripts.");
|
|
271
|
+
const cwd = process.cwd();
|
|
272
|
+
const config = discoverConfig({ cwd, homeDir: process.env.RAG_HOME ?? homedir(), argv: args, env: process.env });
|
|
273
|
+
const root = path.resolve(cwd, config.root);
|
|
274
|
+
const rl = createInterface({ input: process.stdin, output: process.stdout });
|
|
275
|
+
try {
|
|
276
|
+
await runTui({ ask: (prompt) => rl.question(prompt), print: (text) => console.log(text) }, root, cwd);
|
|
277
|
+
} finally {
|
|
278
|
+
rl.close();
|
|
279
|
+
}
|
|
280
|
+
}
|
|
281
|
+
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
|
282
|
+
try {
|
|
283
|
+
await main();
|
|
284
|
+
} catch (error) {
|
|
285
|
+
console.error(`agent-rag: ${safe(error instanceof Error ? error.message : error)}`);
|
|
286
|
+
process.exitCode = 1;
|
|
287
|
+
}
|
|
288
|
+
}
|
|
289
|
+
export {
|
|
290
|
+
exportVersion,
|
|
291
|
+
listDocuments,
|
|
292
|
+
listNamespaces,
|
|
293
|
+
listVersions,
|
|
294
|
+
main,
|
|
295
|
+
runTui
|
|
296
|
+
};
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "agent-rag-mcp",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.9.0",
|
|
4
4
|
"description": "Deterministic, namespace-isolated document retrieval with a CLI and MCP server.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"repository": {
|
|
@@ -21,6 +21,7 @@
|
|
|
21
21
|
],
|
|
22
22
|
"type": "module",
|
|
23
23
|
"bin": {
|
|
24
|
+
"agent-rag": "bin/agent-rag.mjs",
|
|
24
25
|
"rag": "bin/rag.mjs",
|
|
25
26
|
"agent-rag-mcp": "bin/agent-rag-mcp.mjs"
|
|
26
27
|
},
|
|
@@ -10,7 +10,7 @@ description: "Trigger: search or read the indexed knowledge base with explicit c
|
|
|
10
10
|
- Searches require all query terms by default. Use `matchMode: "any"` to broaden lexical matching when appropriate; partial matches are candidates, not proof of relevance or semantic matches.
|
|
11
11
|
- `searchMode: "semantic"` or `"hybrid"` requires a configured embedding provider. These modes send the query and indexed chunk text to the configured provider (local Ollama or explicitly selected Azure), are bounded, and may fail for large corpora. Scores are ranking signals, not evidence; read citations before relying on them.
|
|
12
12
|
- Use `rag_read` only when a citation needs more surrounding text; pass its `docVersionId` and `chunkOrdinal` from `rag_find`.
|
|
13
|
-
-
|
|
13
|
+
- For complete chunk coverage, call `rag_read` with `startByte: 0`, a modest `maxBytes` such as 8192 and positive `maxLines`; repeat with the returned `nextByteOffset` until null, concatenating texts directly. If the offset does not advance, increase `maxBytes` to fit the next UTF-8 character. If a transport truncates a response, retry the same page with a smaller limit. Do not combine `startByte` with `startLine`.
|
|
14
14
|
- The orchestrator only receives `rag_find`; delegate exploration when a citation needs reading.
|
|
15
15
|
|
|
16
16
|
## Evidence and scope
|