ownsearch 0.1.2 → 0.1.4

package/README.md

@@ -1,36 +1,116 @@
 # ownsearch
 
-**ownsearch** is a local search layer for agents.
+**ownsearch** is a local retrieval layer for agents.
 
-It indexes approved folders into a local Qdrant store, exposes retrieval through an MCP server, and lets your agents search private knowledge without a hosted search service.
+It indexes approved folders into a local Qdrant vector store, embeds content with Gemini, and exposes grounded retrieval through an MCP server so agents can search private documents without shipping those documents to a hosted RAG backend.
 
-V1 is intentionally text-first: simple, reliable local retrieval for docs, code, and PDFs. Over time, **ownsearch** will expand to support multimodal files and data, including images, audio, video, and richer cross-modal search workflows.
+This package is designed for **text-first, local, agentic RAG**:
+
+- local folders instead of SaaS document ingestion
+- MCP-native access for agents
+- grounded chunk retrieval instead of opaque long-context guessing
+- predictable local storage with Docker-backed Qdrant
+
+## Why it exists
+
+Most agents waste time and tokens when they do one of two things:
+
+- search too broadly with weak semantic queries
+- skip retrieval and guess from partial context
+
+`ownsearch` is meant to reduce both failure modes by:
+
+- indexing local knowledge once
+- making retrieval cheap and reusable
+- giving agents a structured way to fetch only the chunks they need
+- improving answer quality with reranking, deduplication, and grounded chunk access
+
+## Core use cases
+
+`ownsearch` is a good fit when an agent needs to work over:
+
+- product documentation
+- technical design docs
+- code-adjacent text files
+- contracts and policy documents
+- research notes
+- knowledge bases stored in folders
+- PDF, DOCX, RTF, markdown, and plain-text heavy repositories
+
+Typical agent workflows:
+
+- answer questions over local docs
+- locate the exact source file or section for a fact
+- summarize a set of related files
+- compare policy, spec, or contract language across documents
+- support coding agents with repo-local documentation search
+- reduce token cost by retrieving only relevant chunks instead of loading entire files
 
 ## What it does
 
-- Indexes local folders into a persistent vector store
-- Chunks and embeds supported files with Gemini
-- Supports incremental reindexing for changed and deleted files
-- Exposes search and context retrieval through MCP
-- Lets agents retrieve ranked hits, exact chunks, or grounded context bundles
+- indexes local folders into a persistent vector store
+- chunks and embeds supported files with Gemini
+- supports incremental reindexing for changed and deleted files
+- exposes search and context retrieval through MCP
+- reranks and deduplicates result sets before returning them
+- lets agents retrieve ranked hits, exact chunks, or bundled grounded context
 
-## V1 scope
+## Current power
 
-- text and code files
-- extracted text from PDFs
-- Gemini `gemini-embedding-001`
-- Docker-backed Qdrant
-- stdio MCP server for local agent attachment
+What is already strong in the current package:
 
-## Quickstart
+- local-first setup with Docker-backed Qdrant
+- deterministic readiness checks through `ownsearch doctor`
+- multi-platform MCP config generation
+- bundled retrieval skill for better query planning
+- support for common text document formats
+- large plain text and code files are no longer blocked by the extracted-document size cap
+- repeatable smoke validation for mixed text corpora
+
+## V1 supported document types
+
+The current package is intended for text-first corpora, including:
+
+- plain text and code files
+- markdown and MDX
+- JSON, YAML, TOML, CSV, XML, HTML
+- PDF via text extraction
+- DOCX via text extraction
+- RTF via text extraction
+
+## Deployment readiness
+
+This package is ready to deploy for **text-first local document folders** when:
 
-Install `ownsearch` globally:
+- Node.js `20+` is available
+- Docker is available and Qdrant can run locally
+- `GEMINI_API_KEY` is configured
+- the document corpus is primarily text-based
+
+Installation:
 
 ```bash
 npm install -g ownsearch
+```
 
-Set it up, index a folder, and start searching:
+Deployment checklist:
 
+```bash
+npm install -g ownsearch
+ownsearch setup
+ownsearch doctor
+ownsearch index C:\path\to\folder --name my-folder
+ownsearch serve-mcp
+```
+
+If `ownsearch doctor` returns:
+
+- `verdict.status: "ready"` then the package is operational
+- `verdict.status: "action_required"` then follow the listed `nextSteps`
+
+## Quickstart
+
+```bash
 ownsearch setup
 ownsearch doctor
 ownsearch index ./docs --name docs
@@ -38,48 +118,183 @@ ownsearch list-roots
 ownsearch search "what is this repo about?" --limit 5
 ownsearch search-context "what is this repo about?" --limit 8 --max-chars 12000
 ownsearch serve-mcp
+```
+
+On first run, `ownsearch setup` can:
+
+- prompt for `GEMINI_API_KEY`
+- link users to Google AI Studio
+- save the key to `~/.ownsearch/.env`
+- print exact next commands for CLI and MCP usage
+- optionally print an MCP config snippet for a selected agent
+
+## Real-world fit
+
+`ownsearch` is a strong fit for:
+
+- engineering teams with private docs that should stay local
+- coding agents that need repo-adjacent design docs and runbooks
+- consultants or operators working across contract, policy, or knowledge folders
+- researchers who want grounded retrieval over local notes and exported reports
+- teams trying to reduce agent token burn by retrieving small grounded context bundles instead of pasting entire files
 
-To connect ownsearch to a supported agent, print a config snippet for your client:
+It is less suitable when:
 
+- the corpus is mostly scanned documents that need OCR
+- the workflow depends on spreadsheets, slides, or legacy Office formats
+- the main requirement is hosted multi-user search rather than local agent retrieval
+
+## Agent integration
+
+To print MCP config snippets:
+
+```bash
 ownsearch print-agent-config codex
-ownsearch print-agent-config claude-desktop
 ownsearch print-agent-config cursor
-Local development
+ownsearch print-agent-config vscode
+ownsearch print-agent-config github-copilot
+ownsearch print-agent-config copilot-cli
+ownsearch print-agent-config windsurf
+ownsearch print-agent-config continue
+ownsearch print-agent-config claude-desktop
+```
+
+Supported config targets currently include:
+
+- `codex`
+- `cursor`
+- `vscode`
+- `github-copilot`
+- `copilot-cli`
+- `windsurf`
+- `continue`
+- `claude-desktop`
+
+Notes:
 
-If you want to run ownsearch from source while developing locally:
+- `claude-desktop` currently returns guidance rather than a raw JSON snippet because current Claude Desktop docs prefer desktop extensions (`.mcpb`) over manual JSON server configs
+- all other supported targets return concrete MCP config payloads
 
-npm install
-npm run build
-node dist/cli.js setup
-node dist/cli.js index ./docs --name docs
-node dist/cli.js search "what is this repo about?" --limit 5
-node dist/cli.js serve-mcp
+## Bundled skill
+
+The package ships with a bundled retrieval skill:
+
+```bash
+ownsearch print-skill ownsearch-rag-search
+```
+
+The skill is intended to help an agent:
+
+- rewrite weak user requests into stronger retrieval queries
+- decide when to use `search_context` vs `search` vs `get_chunks`
+- recover from poor first-pass retrieval
+- avoid duplicate-heavy answer synthesis
+- stay grounded when retrieval is probabilistic
+
+## CLI commands
+
+- `ownsearch setup`
+  Starts or reconnects to the local Qdrant Docker container, creates local config, persists `GEMINI_API_KEY`, and prints next-step commands.
+- `ownsearch doctor`
+  Checks config, Gemini key presence, Qdrant connectivity, collection settings, and emits a deterministic readiness verdict.
+- `ownsearch index <folder> --name <name>`
+  Indexes a folder incrementally into the local vector collection.
+- `ownsearch list-roots`
+  Lists approved indexed roots.
+- `ownsearch search "<query>"`
+  Returns reranked search hits from the vector store.
+- `ownsearch search-context "<query>"`
+  Returns a compact grounded context bundle for agents.
+- `ownsearch delete-root <rootId>`
+  Removes a root from config and deletes its vectors from Qdrant.
+- `ownsearch store-status`
+  Shows collection status and vector configuration.
+- `ownsearch serve-mcp`
+  Starts the stdio MCP server.
+- `ownsearch print-agent-config <agent>`
+  Prints MCP config snippets or platform guidance.
+- `ownsearch print-skill [skill]`
+  Prints a bundled OwnSearch skill.
 
 ## MCP tools
 
-* `index_path`
-* `search`
-* `search_context`
-* `get_chunks`
-* `list_roots`
-* `delete_root`
-* `store_status`
+The MCP server currently exposes:
 
-## Notes
+- `index_path`
+- `search`
+- `search_context`
+- `get_chunks`
+- `list_roots`
+- `delete_root`
+- `store_status`
 
-* Config is stored in `~/.ownsearch/config.json`
-* Qdrant runs locally in Docker as `ownsearch-qdrant`
-* `GEMINI_API_KEY` must be available in the environment or `.env`
+Recommended retrieval flow:
 
-## Roadmap
+1. Use `search_context` for fast grounded retrieval.
+2. Use `search` when ranking and source inspection matter.
+3. Use `get_chunks` when exact wording or detailed comparison matters.
 
-Planned after the text-first v1:
+## Validation
+
+The package includes a repeatable smoke suite:
+
+```bash
+npm run smoke:text-docs
+```
+
+That smoke run currently validates:
+
+- `.txt` retrieval
+- `.rtf` retrieval
+- `.docx` retrieval
+- `.pdf` retrieval
+- large plain text file bypass of the extracted-document byte cap
+
+## Limitations
+
+This package is deploy-ready for text-first corpora, but it is not universal document intelligence.
+
+Current hard limitations:
+
+- no OCR for image-only PDFs
+- no `.doc` support
+- no spreadsheet or presentation extraction such as `.xlsx` or `.pptx`
+- no multimodal embeddings yet
+- reranking is heuristic and local, not yet model-based
+- very large corpora can still become expensive because embedding cost scales with chunk count
+
+Operational limitations:
+
+- retrieval quality still depends on query quality
+- extracted document quality depends on source document quality
+- duplicate-heavy corpora are improved by current reranking, but not fully solved for all edge cases
+- scanned or low-quality PDFs may require OCR before indexing
+
+## Future scope
+
+Planned next-stage improvements:
+
+- pluggable learned rerankers
+- stronger deduplication across overlapping corpora
+- richer document extraction
+- watch mode for automatic local reindexing
+- HTTP MCP transport
+- optional hosted deployment mode
+- multimodal indexing and retrieval for:
+  - images
+  - audio
+  - video
+  - richer document formats
+
+The multimodal phase will require careful collection migration because Gemini text and multimodal embedding spaces are not interchangeable across model families.
+
+## Notes
 
-* richer document extraction
-* better reranking and deduplication
-* watch mode for automatic reindexing
-* HTTP MCP transport
-* multimodal indexing for images, audio, video, and richer document formats
+- config is stored in `~/.ownsearch/config.json`
+- shared CLI and MCP secrets can be stored in `~/.ownsearch/.env`
+- Qdrant runs locally in Docker as `ownsearch-qdrant`
+- `GEMINI_API_KEY` may come from the shell environment, the current working directory `.env`, or `~/.ownsearch/.env`
+- `maxFileBytes` primarily applies to extracted document formats such as PDF, DOCX, and RTF, not to large plain text and code files
 
 ## License
 

package/dist/{chunk-NLETDGQ5.js → chunk-ZQAY3FE3.js}

@@ -3,6 +3,18 @@ function buildContextBundle(query, hits, maxChars = 12e3) {
   const results = [];
   let totalChars = 0;
   for (const hit of hits) {
+    const last = results.at(-1);
+    if (last && last.rootId === hit.rootId && last.relativePath === hit.relativePath && hit.chunkIndex === last.chunkIndex + 1) {
+      const mergedContent = `${last.content}
+${hit.content}`.trim();
+      const mergedDelta = mergedContent.length - last.content.length;
+      if (totalChars + mergedDelta <= maxChars) {
+        last.content = mergedContent;
+        last.chunkIndex = hit.chunkIndex;
+        totalChars += mergedDelta;
+        continue;
+      }
+    }
     if (results.length > 0 && totalChars + hit.content.length > maxChars) {
       break;
     }
@@ -25,9 +37,11 @@ function buildContextBundle(query, hits, maxChars = 12e3) {
 }
 
 // src/config.ts
+import fsSync from "fs";
 import fs from "fs/promises";
 import os from "os";
 import path2 from "path";
+import dotenv from "dotenv";
 
 // src/constants.ts
 var CONFIG_DIR_NAME = ".ownsearch";
@@ -43,26 +57,32 @@ var DEFAULT_CHUNK_OVERLAP = 200;
 var DEFAULT_MAX_FILE_BYTES = 50 * 1024 * 1024;
 var SUPPORTED_TEXT_EXTENSIONS = /* @__PURE__ */ new Set([
   ".c",
+  ".conf",
   ".cpp",
   ".cs",
   ".css",
   ".csv",
+  ".docx",
   ".env",
   ".go",
   ".h",
   ".hpp",
   ".html",
+  ".ini",
   ".java",
   ".js",
   ".json",
   ".jsx",
+  ".log",
   ".md",
   ".mdx",
   ".mjs",
   ".pdf",
   ".ps1",
+  ".properties",
   ".py",
   ".rb",
+  ".rtf",
   ".rs",
   ".sh",
   ".sql",
@@ -74,6 +94,11 @@ var SUPPORTED_TEXT_EXTENSIONS = /* @__PURE__ */ new Set([
   ".yaml",
   ".yml"
 ]);
+var EXTRACTED_DOCUMENT_EXTENSIONS = /* @__PURE__ */ new Set([
+  ".pdf",
+  ".docx",
+  ".rtf"
+]);
 var IGNORED_DIRECTORIES = /* @__PURE__ */ new Set([
   ".git",
   ".hg",
@@ -141,9 +166,34 @@ function getConfigDir() {
 function getConfigPath() {
   return path2.join(getConfigDir(), CONFIG_FILE_NAME);
 }
+function getEnvPath() {
+  return path2.join(getConfigDir(), ".env");
+}
+function getCwdEnvPath() {
+  return path2.resolve(process.cwd(), ".env");
+}
 async function ensureConfigDir() {
   await fs.mkdir(getConfigDir(), { recursive: true });
 }
+function loadOwnSearchEnv() {
+  for (const envPath of [getCwdEnvPath(), getEnvPath()]) {
+    if (!fsSync.existsSync(envPath)) {
+      continue;
+    }
+    const parsed = dotenv.parse(fsSync.readFileSync(envPath, "utf8"));
+    for (const [key, value] of Object.entries(parsed)) {
+      if (process.env[key] === void 0) {
+        process.env[key] = value;
+      }
+    }
+  }
+}
+function readEnvFile(envPath) {
+  if (!fsSync.existsSync(envPath)) {
+    return {};
+  }
+  return dotenv.parse(fsSync.readFileSync(envPath, "utf8"));
+}
 async function loadConfig() {
   await ensureConfigDir();
   const configPath = getConfigPath();
@@ -171,6 +221,11 @@ async function saveConfig(config) {
   await fs.writeFile(getConfigPath(), `${JSON.stringify(config, null, 2)}
 `, "utf8");
 }
+async function saveGeminiApiKey(apiKey) {
+  await ensureConfigDir();
+  await fs.writeFile(getEnvPath(), `GEMINI_API_KEY=${apiKey.trim()}
+`, "utf8");
+}
 function createRootDefinition(rootPath, name) {
   const now = (/* @__PURE__ */ new Date()).toISOString();
   const rootName = name?.trim() || path2.basename(rootPath);
@@ -270,6 +325,112 @@ async function embedQuery(query) {
 
 // src/qdrant.ts
 import { QdrantClient } from "@qdrant/js-client-rest";
+
+// src/rerank.ts
+function normalize(input) {
+  return input.toLowerCase().replace(/[^a-z0-9\s]/g, " ").replace(/\s+/g, " ").trim();
+}
+function tokenize(input) {
+  return normalize(input).split(" ").filter((token) => token.length > 1);
+}
+function unique(items) {
+  return Array.from(new Set(items));
+}
+function lexicalOverlap(queryTokens, haystack) {
+  if (queryTokens.length === 0) {
+    return 0;
+  }
+  const haystackTokens = new Set(tokenize(haystack));
+  let matches = 0;
+  for (const token of queryTokens) {
+    if (haystackTokens.has(token)) {
+      matches += 1;
+    }
+  }
+  return matches / queryTokens.length;
+}
+function nearDuplicate(a, b) {
+  const aTokens = unique(tokenize(a.content)).slice(0, 48);
+  const bTokens = unique(tokenize(b.content)).slice(0, 48);
+  if (aTokens.length === 0 || bTokens.length === 0) {
+    return false;
+  }
+  const bSet = new Set(bTokens);
+  let intersection = 0;
+  for (const token of aTokens) {
+    if (bSet.has(token)) {
+      intersection += 1;
+    }
+  }
+  const union = (/* @__PURE__ */ new Set([...aTokens, ...bTokens])).size;
+  return union > 0 && intersection / union >= 0.8;
+}
+function contentSignature(content) {
+  return tokenize(content).slice(0, 24).join(" ");
+}
+function rerankAndDeduplicate(query, hits, limit) {
+  const normalizedQuery = normalize(query);
+  const queryTokens = unique(tokenize(query));
+  const ranked = hits.map((hit) => {
+    const overlap = lexicalOverlap(queryTokens, hit.content);
+    const pathOverlap = lexicalOverlap(queryTokens, `${hit.relativePath} ${hit.rootName}`);
+    const exactPhrase = normalizedQuery.length > 0 && normalize(hit.content).includes(normalizedQuery) ? 0.2 : 0;
+    const score = hit.score + overlap * 0.22 + pathOverlap * 0.08 + exactPhrase;
+    return { ...hit, rerankScore: score };
+  }).sort((left, right) => right.rerankScore - left.rerankScore);
+  const selected = [];
+  const signatureSet = /* @__PURE__ */ new Set();
+  const perFileCounts = /* @__PURE__ */ new Map();
+  const preferredPerFileLimit = 2;
+  function canTake(hit, enforcePerFileLimit) {
+    const signature = contentSignature(hit.content);
+    if (signature && signatureSet.has(signature)) {
+      return false;
+    }
+    if (selected.some((existing) => nearDuplicate(existing, hit))) {
+      return false;
+    }
+    if (enforcePerFileLimit) {
+      const current = perFileCounts.get(hit.relativePath) ?? 0;
+      if (current >= preferredPerFileLimit) {
+        return false;
+      }
+    }
+    return true;
+  }
+  function add(hit) {
+    selected.push(hit);
+    const signature = contentSignature(hit.content);
+    if (signature) {
+      signatureSet.add(signature);
+    }
+    perFileCounts.set(hit.relativePath, (perFileCounts.get(hit.relativePath) ?? 0) + 1);
+  }
+  for (const hit of ranked) {
+    if (selected.length >= limit) {
+      break;
+    }
+    if (canTake(hit, true)) {
+      add(hit);
+    }
+  }
+  if (selected.length < limit) {
+    for (const hit of ranked) {
+      if (selected.length >= limit) {
+        break;
+      }
+      if (selected.some((existing) => existing.id === hit.id)) {
+        continue;
+      }
+      if (canTake(hit, false)) {
+        add(hit);
+      }
+    }
+  }
+  return selected.map(({ rerankScore: _rerankScore, ...hit }) => hit);
+}
+
+// src/qdrant.ts
 var OwnSearchStore = class {
   constructor(client2, collectionName, vectorSize) {
     this.client = client2;
@@ -427,7 +588,7 @@ var OwnSearchStore = class {
     }
     const results = await this.client.search(this.collectionName, {
       vector,
-      limit: filters.pathSubstring ? Math.max(limit * 3, limit) : limit,
+      limit: Math.max(filters.pathSubstring ? limit * 8 : limit * 6, 24),
       with_payload: true,
       filter: must.length ? { must } : void 0
     });
@@ -441,11 +602,8 @@ var OwnSearchStore = class {
       chunkIndex: Number(result.payload?.chunk_index ?? 0),
       content: String(result.payload?.content ?? "")
     }));
-    if (!filters.pathSubstring) {
-      return hits.slice(0, limit);
-    }
-    const needle = filters.pathSubstring.toLowerCase();
-    return hits.filter((hit) => hit.relativePath.toLowerCase().includes(needle)).slice(0, limit);
+    const filtered = !filters.pathSubstring ? hits : hits.filter((hit) => hit.relativePath.toLowerCase().includes(filters.pathSubstring.toLowerCase()));
+    return rerankAndDeduplicate(filters.queryText ?? "", filtered, limit);
   }
   async getChunks(ids) {
     if (ids.length === 0) {
@@ -500,9 +658,17 @@ function chunkText(content, chunkSize, chunkOverlap) {
   while (start < normalized.length) {
     let end = Math.min(start + chunkSize, normalized.length);
     if (end < normalized.length) {
-      const lastBoundary = normalized.lastIndexOf("\n", end);
-      if (lastBoundary > start + Math.floor(chunkSize * 0.5)) {
-        end = lastBoundary;
+      const minimumBoundary = start + Math.floor(chunkSize * 0.5);
+      const newlineBoundary = normalized.lastIndexOf("\n", end);
+      const whitespaceBoundary = normalized.lastIndexOf(" ", end);
+      const punctuationBoundary = Math.max(
+        normalized.lastIndexOf(". ", end),
+        normalized.lastIndexOf("? ", end),
+        normalized.lastIndexOf("! ", end)
+      );
+      const boundary = Math.max(newlineBoundary, whitespaceBoundary, punctuationBoundary);
+      if (boundary > minimumBoundary) {
+        end = boundary;
       }
     }
     const chunk = normalized.slice(start, end).trim();
@@ -520,10 +686,14 @@ function chunkText(content, chunkSize, chunkOverlap) {
 // src/files.ts
 import fs2 from "fs/promises";
 import path3 from "path";
+import mammoth from "mammoth";
 import { PDFParse } from "pdf-parse";
 function sanitizeExtractedText(input) {
   return input.replace(/\u0000/g, "").replace(/[\u0001-\u0008\u000B\u000C\u000E-\u001F\u007F]/g, " ").replace(/\r\n/g, "\n");
 }
+function extractRtfText(input) {
+  return input.replace(/\\par[d]?/g, "\n").replace(/\\tab/g, "	").replace(/\\'[0-9a-fA-F]{2}/g, " ").replace(/\\[a-zA-Z]+-?\d* ?/g, "").replace(/[{}]/g, " ");
+}
 async function collectTextFiles(rootPath, maxFileBytes) {
   const files = [];
   const absoluteRoot = path3.resolve(rootPath);
@@ -543,6 +713,11 @@ async function collectTextFiles(rootPath, maxFileBytes) {
       await parser.destroy();
     }
   }
+  async function parseDocx(filePath) {
+    const buffer = await fs2.readFile(filePath);
+    const result = await mammoth.extractRawText({ buffer });
+    return result.value ?? "";
+  }
   async function walk(currentPath) {
     const entries = await fs2.readdir(currentPath, { withFileTypes: true });
     for (const entry of entries) {
@@ -565,7 +740,7 @@ async function collectTextFiles(rootPath, maxFileBytes) {
         continue;
       }
       const stats = await fs2.stat(nextPath);
-      if (stats.size > maxFileBytes) {
+      if (EXTRACTED_DOCUMENT_EXTENSIONS.has(extension) && stats.size > maxFileBytes) {
         debugLog("skip-size", nextPath, stats.size);
         continue;
       }
@@ -573,6 +748,10 @@ async function collectTextFiles(rootPath, maxFileBytes) {
       try {
         if (extension === ".pdf") {
           content = await parsePdf(nextPath);
+        } else if (extension === ".docx") {
+          content = await parseDocx(nextPath);
+        } else if (extension === ".rtf") {
+          content = extractRtfText(await fs2.readFile(nextPath, "utf8"));
         } else {
           content = await fs2.readFile(nextPath, "utf8");
         }
@@ -715,7 +894,12 @@ async function indexPath(rootPath, options = {}) {
 export {
   buildContextBundle,
   getConfigPath,
+  getEnvPath,
+  getCwdEnvPath,
+  loadOwnSearchEnv,
+  readEnvFile,
   loadConfig,
+  saveGeminiApiKey,
   deleteRootDefinition,
   findRoot,
   listRoots,

package/dist/cli.js

@@ -7,15 +7,21 @@ import {
   embedQuery,
   findRoot,
   getConfigPath,
+  getCwdEnvPath,
+  getEnvPath,
   indexPath,
   listRoots,
-  loadConfig
-} from "./chunk-NLETDGQ5.js";
+  loadConfig,
+  loadOwnSearchEnv,
+  readEnvFile,
+  saveGeminiApiKey
+} from "./chunk-ZQAY3FE3.js";
 
 // src/cli.ts
-import "dotenv/config";
+import fs from "fs/promises";
 import path from "path";
 import { spawn } from "child_process";
+import readline from "readline/promises";
 import { fileURLToPath } from "url";
 import { Command } from "commander";
 
@@ -61,23 +67,294 @@ async function ensureQdrantDocker() {
 }
 
 // src/cli.ts
+loadOwnSearchEnv();
 var program = new Command();
+var PACKAGE_NAME = "ownsearch";
+var GEMINI_API_KEY_URL = "https://aistudio.google.com/apikey";
+var BUNDLED_SKILL_NAME = "ownsearch-rag-search";
+var SUPPORTED_AGENTS = [
+  "codex",
+  "claude-desktop",
+  "continue",
+  "copilot-cli",
+  "cursor",
+  "github-copilot",
+  "vscode",
+  "windsurf"
+];
 function requireGeminiKey() {
   if (!process.env.GEMINI_API_KEY) {
     throw new OwnSearchError("Set GEMINI_API_KEY before running OwnSearch.");
   }
 }
+function buildAgentConfig(agent) {
+  const stdioConfig = {
+    command: "npx",
+    args: ["-y", PACKAGE_NAME, "serve-mcp"],
+    env: {
+      GEMINI_API_KEY: "${GEMINI_API_KEY}"
+    }
+  };
+  switch (agent) {
+    case "codex":
+      return {
+        platform: "codex",
+        configScope: "Add this server entry to your Codex MCP configuration.",
+        config: { ownsearch: stdioConfig }
+      };
+    case "claude-desktop":
+      return {
+        platform: "claude-desktop",
+        installMethod: "Desktop Extension (.mcpb)",
+        note: "Current Claude Desktop documentation recommends local MCP installation through Desktop Extensions instead of manual JSON config files.",
+        nextStep: "OwnSearch does not yet ship an .mcpb bundle. Use Cursor, VS Code, Windsurf, Continue, or GitHub Copilot with the snippets below for now."
+      };
+    case "continue":
+      return {
+        platform: "continue",
+        configPath: ".continue/mcpServers/ownsearch.json",
+        note: "Continue can ingest JSON MCP configs directly.",
+        config: { ownsearch: stdioConfig }
+      };
+    case "copilot-cli":
+      return {
+        platform: "copilot-cli",
+        configPath: "~/.copilot/mcp-config.json",
+        config: {
+          mcpServers: {
+            ownsearch: {
+              type: "local",
+              command: stdioConfig.command,
+              args: stdioConfig.args,
+              env: stdioConfig.env,
+              tools: ["*"]
+            }
+          }
+        }
+      };
+    case "cursor":
+      return {
+        platform: "cursor",
+        configPath: "~/.cursor/mcp.json or .cursor/mcp.json",
+        config: { ownsearch: stdioConfig }
+      };
+    case "github-copilot":
+    case "vscode":
+      return {
+        platform: agent,
+        configPath: ".vscode/mcp.json or VS Code user profile mcp.json",
+        config: {
+          servers: {
+            ownsearch: stdioConfig
+          }
+        }
+      };
+    case "windsurf":
+      return {
+        platform: "windsurf",
+        configPath: "~/.codeium/mcp_config.json",
+        config: {
+          mcpServers: {
+            ownsearch: stdioConfig
+          }
+        }
+      };
+    default:
+      throw new OwnSearchError(`Unsupported agent: ${agent}`);
+  }
+}
+async function readBundledSkill(skillName) {
+  const currentFilePath = fileURLToPath(import.meta.url);
+  const packageRoot = path.resolve(path.dirname(currentFilePath), "..");
+  const skillPath = path.join(packageRoot, "skills", skillName, "SKILL.md");
+  return fs.readFile(skillPath, "utf8");
+}
+function getDoctorVerdict(input) {
+  const nextSteps = [];
+  if (!input.geminiApiKeyPresent) {
+    nextSteps.push("Run `ownsearch setup` and save a Gemini API key.");
+  }
+  if (!input.qdrantReachable) {
+    nextSteps.push("Run `ownsearch setup` to start or reconnect to the local Qdrant container.");
+  }
+  if (input.geminiApiKeyPresent && input.qdrantReachable && input.rootCount === 0) {
+    nextSteps.push("Run `ownsearch index C:\\path\\to\\folder --name my-folder` to add your first indexed root.");
+  }
+  if (nextSteps.length === 0) {
+    nextSteps.push("Run `ownsearch index C:\\path\\to\\folder --name my-folder` to add more content, or `ownsearch serve-mcp` to connect an agent.");
+    return {
+      status: "ready",
+      summary: input.rootCount > 0 ? "OwnSearch is ready for indexing, search, and MCP agent use." : "OwnSearch is ready. Qdrant and Gemini are configured.",
+      nextSteps
+    };
+  }
+  return {
+    status: "action_required",
+    summary: "OwnSearch is not fully ready yet.",
+    nextSteps
+  };
+}
+async function promptForGeminiKey() {
+  if (!process.stdin.isTTY || !process.stdout.isTTY) {
+    return false;
+  }
+  const rl = readline.createInterface({
+    input: process.stdin,
+    output: process.stdout
+  });
+  try {
+    console.log(`Generate a Gemini API key here: ${GEMINI_API_KEY_URL}`);
+    console.log(`OwnSearch will save it to ${getEnvPath()}`);
+    for (; ; ) {
+      const apiKey = (await rl.question("Paste GEMINI_API_KEY and press Enter (Ctrl+C to cancel): ")).trim();
+      if (!apiKey) {
+        console.log("GEMINI_API_KEY is required for indexing and search.");
+        continue;
+      }
+      await saveGeminiApiKey(apiKey);
+      process.env.GEMINI_API_KEY = apiKey;
+      return true;
+    }
+  } finally {
+    rl.close();
+  }
+}
+function getGeminiApiKeySource() {
+  if (readEnvFile(getEnvPath()).GEMINI_API_KEY) {
+    return "ownsearch-env";
+  }
+  if (readEnvFile(getCwdEnvPath()).GEMINI_API_KEY) {
+    return "cwd-env";
+  }
+  if (process.env.GEMINI_API_KEY) {
+    return "process-env";
+  }
+  return "missing";
+}
+async function ensureManagedGeminiKey() {
+  const source = getGeminiApiKeySource();
+  if (source === "ownsearch-env") {
+    return { present: true, source, savedToManagedEnv: false };
+  }
+  if (process.env.GEMINI_API_KEY) {
+    await saveGeminiApiKey(process.env.GEMINI_API_KEY);
+    return { present: true, source, savedToManagedEnv: true };
+  }
+  const prompted = await promptForGeminiKey();
+  return {
+    present: prompted,
+    source: prompted ? "prompt" : "missing",
+    savedToManagedEnv: prompted
+  };
+}
+function printSetupNextSteps() {
+  console.log("");
+  console.log("Next commands:");
+  console.log("  CLI indexing:");
+  console.log("    ownsearch index C:\\path\\to\\folder --name my-folder");
+  console.log("  CLI search:");
+  console.log('    ownsearch search "your question here" --limit 5');
+  console.log("  CLI grounded context:");
+  console.log('    ownsearch search-context "your question here" --limit 8 --max-chars 12000');
+  console.log("  MCP server for agents:");
+  console.log("    ownsearch serve-mcp");
+  console.log("  Agent config snippets:");
+  console.log("    ownsearch print-agent-config codex");
+  console.log("    ownsearch print-agent-config claude-desktop");
+  console.log("    ownsearch print-agent-config cursor");
+  console.log("    ownsearch print-agent-config vscode");
+  console.log("    ownsearch print-agent-config github-copilot");
+  console.log("    ownsearch print-agent-config copilot-cli");
+  console.log("    ownsearch print-agent-config windsurf");
+  console.log("    ownsearch print-agent-config continue");
+  console.log("  Bundled retrieval skill:");
+  console.log(`    ownsearch print-skill ${BUNDLED_SKILL_NAME}`);
+}
+async function promptForAgentChoice() {
+  if (!process.stdin.isTTY || !process.stdout.isTTY) {
+    return void 0;
+  }
+  const rl = readline.createInterface({
+    input: process.stdin,
+    output: process.stdout
+  });
+  try {
+    console.log("");
+    console.log("Connect to an agent now?");
+    console.log("  1. codex");
+    console.log("  2. claude-desktop");
+    console.log("  3. cursor");
+    console.log("  4. vscode");
+    console.log("  5. windsurf");
+    console.log("  6. copilot-cli");
+    console.log("  7. continue");
+    console.log("  8. skip");
+    for (; ; ) {
+      const answer = (await rl.question("Select 1-8: ")).trim().toLowerCase();
+      switch (answer) {
+        case "1":
+        case "codex":
+          return "codex";
+        case "2":
+        case "claude-desktop":
+        case "claude":
+          return "claude-desktop";
+        case "3":
+        case "cursor":
+          return "cursor";
+        case "4":
+        case "vscode":
+        case "github-copilot":
+          return "vscode";
+        case "5":
+        case "windsurf":
+          return "windsurf";
+        case "6":
+        case "copilot-cli":
+        case "copilot":
+          return "copilot-cli";
+        case "7":
+        case "continue":
+          return "continue";
+        case "8":
+        case "skip":
+        case "":
+          return void 0;
+        default:
+          console.log("Enter 1, 2, 3, 4, 5, 6, 7, or 8.");
+      }
+    }
+  } finally {
+    rl.close();
+  }
+}
+function printAgentConfigSnippet(agent) {
+  console.log("");
+  console.log(`MCP config for ${agent}:`);
+  console.log(JSON.stringify(buildAgentConfig(agent), null, 2));
+}
 program.name("ownsearch").description("Gemini-powered local search MCP server backed by Qdrant.").version("0.1.0");
 program.command("setup").description("Create config and start a local Qdrant Docker container.").action(async () => {
   const config = await loadConfig();
   const result = await ensureQdrantDocker();
+  const gemini = await ensureManagedGeminiKey();
   console.log(JSON.stringify({
     configPath: getConfigPath(),
+    envPath: getEnvPath(),
     qdrantUrl: config.qdrantUrl,
-    qdrantStarted: result.started
+    qdrantStarted: result.started,
+    geminiApiKeyPresent: gemini.present,
+    geminiApiKeySource: gemini.source,
+    geminiApiKeySavedToManagedEnv: gemini.savedToManagedEnv
   }, null, 2));
-  if (!process.env.GEMINI_API_KEY) {
-    console.log("GEMINI_API_KEY is not set. Indexing and search will require it later.");
+  if (!gemini.present) {
+    console.log(`GEMINI_API_KEY is not set. Re-run setup or add it to ${getEnvPath()} before indexing or search.`);
+    return;
+  }
+  printSetupNextSteps();
+  const agent = await promptForAgentChoice();
+  if (agent) {
+    printAgentConfigSnippet(agent);
   }
 });
 program.command("index").argument("<folder>", "Folder path to index").option("-n, --name <name>", "Display name for the indexed root").option("--max-file-bytes <n>", "Override the file size limit for this run", (value) => Number(value)).description("Index a local folder into Qdrant using Gemini embeddings.").action(async (folder, options) => {
@@ -96,6 +373,7 @@ program.command("search").argument("<query>", "Natural language query").option("
     const hits = await store.search(
       vector,
       {
+        queryText: query,
         rootIds: options.rootId,
         pathSubstring: options.path
       },
@@ -112,6 +390,7 @@ program.command("search-context").argument("<query>", "Natural language query").
     const hits = await store.search(
       vector,
       {
+        queryText: query,
         rootIds: options.rootId,
         pathSubstring: options.path
       },
@@ -148,9 +427,17 @@ program.command("doctor").description("Check local prerequisites and package con
   } catch (error) {
     qdrantReachable = false;
   }
+  const verdict = getDoctorVerdict({
+    geminiApiKeyPresent: Boolean(process.env.GEMINI_API_KEY),
+    qdrantReachable,
+    rootCount: roots.length
+  });
   console.log(JSON.stringify({
+    verdict,
     configPath: getConfigPath(),
+    envPath: getEnvPath(),
     geminiApiKeyPresent: Boolean(process.env.GEMINI_API_KEY),
+    geminiApiKeySource: getGeminiApiKeySource(),
     qdrantUrl: config.qdrantUrl,
     qdrantReachable,
     collection: config.qdrantCollection,
@@ -158,6 +445,7 @@ program.command("doctor").description("Check local prerequisites and package con
     vectorSize: config.vectorSize,
     chunkSize: config.chunkSize,
     chunkOverlap: config.chunkOverlap,
+    maxExtractedDocumentBytes: config.maxFileBytes,
     maxFileBytes: config.maxFileBytes,
     rootCount: roots.length
   }, null, 2));
@@ -173,23 +461,16 @@ program.command("serve-mcp").description("Start the stdio MCP server.").action(a
     process.exitCode = code ?? 0;
   });
 });
-program.command("print-agent-config").argument("<agent>", "codex | claude-desktop | cursor").description("Print an MCP config snippet for a supported agent.").action(async (agent) => {
-  const config = {
-    command: "npx",
-    args: ["-y", "ownsearch", "serve-mcp"],
-    env: {
-      GEMINI_API_KEY: "${GEMINI_API_KEY}"
-    }
-  };
-  switch (agent) {
-    case "codex":
-    case "claude-desktop":
-    case "cursor":
-      console.log(JSON.stringify({ ownsearch: config }, null, 2));
-      return;
-    default:
-      throw new OwnSearchError(`Unsupported agent: ${agent}`);
+program.command("print-agent-config").argument("<agent>", SUPPORTED_AGENTS.join(" | ")).description("Print an MCP config snippet for a supported agent.").action(async (agent) => {
+  if (SUPPORTED_AGENTS.includes(agent)) {
+    console.log(JSON.stringify(buildAgentConfig(agent), null, 2));
+    return;
   }
+  throw new OwnSearchError(`Unsupported agent: ${agent}`);
+});
+program.command("print-skill").argument("[skill]", `Bundled skill name (default ${BUNDLED_SKILL_NAME})`).description("Print a bundled OwnSearch skill that helps agents query retrieval tools more effectively.").action(async (skill) => {
+  const skillName = skill?.trim() || BUNDLED_SKILL_NAME;
+  console.log(await readBundledSkill(skillName));
 });
 program.parseAsync(process.argv).catch((error) => {
   const message = error instanceof Error ? error.message : String(error);

package/dist/mcp/server.js

@@ -7,14 +7,15 @@ import {
   embedQuery,
   findRoot,
   indexPath,
-  loadConfig
-} from "../chunk-NLETDGQ5.js";
+  loadConfig,
+  loadOwnSearchEnv
+} from "../chunk-ZQAY3FE3.js";
 
 // src/mcp/server.ts
-import "dotenv/config";
 import { Server } from "@modelcontextprotocol/sdk/server/index.js";
 import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
 import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";
+loadOwnSearchEnv();
 function asText(result) {
   return {
     content: [
@@ -164,6 +165,7 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
         const hits = await store.search(
           vector,
           {
+            queryText: args.query,
             rootIds: args.rootIds,
             pathSubstring: args.pathSubstring
           },
@@ -184,6 +186,7 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
         const hits = await store.search(
           vector,
           {
+            queryText: args.query,
             rootIds: args.rootIds,
             pathSubstring: args.pathSubstring
           },

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ownsearch",
-  "version": "0.1.2",
+  "version": "0.1.4",
   "description": "Text-first local document search MCP server backed by Gemini embeddings and Qdrant.",
   "type": "module",
   "bin": {
@@ -8,13 +8,15 @@
   },
   "files": [
     "dist",
-    "README.md"
+    "README.md",
+    "skills"
   ],
   "scripts": {
     "build": "tsup src/cli.ts src/mcp/server.ts --format esm --dts --clean --external pdf-parse",
     "dev": "tsx src/cli.ts",
     "prepare": "npm run build",
     "prepublishOnly": "npm run typecheck && npm run build",
+    "smoke:text-docs": "tsx scripts/smoke-text-docs.mts",
     "serve-mcp": "tsx src/mcp/server.ts",
     "typecheck": "tsc --noEmit"
   },
@@ -48,12 +50,14 @@
     "@qdrant/js-client-rest": "^1.17.0",
     "commander": "^14.0.1",
     "dotenv": "^17.3.1",
+    "mammoth": "^1.12.0",
     "pdf-parse": "^2.4.5",
     "zod": "^3.25.76"
   },
   "devDependencies": {
     "@types/node": "^24.6.0",
     "@types/pdf-parse": "^1.1.5",
+    "docx": "^9.6.1",
     "tsup": "^8.5.0",
     "tsx": "^4.20.6",
     "typescript": "^5.9.3"

package/skills/ownsearch-rag-search/SKILL.md

@@ -0,0 +1,139 @@
+---
+name: ownsearch-rag-search
+description: Improve retrieval quality when an agent uses OwnSearch MCP tools to search local documents. Use for semantic search, grounded answering, query rewriting, multi-query retrieval, exact chunk fetches, duplicate-heavy result sets, or whenever a user request must be translated into stronger OwnSearch search_context/search/get_chunks calls.
+---
+
+# OwnSearch RAG Search
+
+## Overview
+
+Use this skill to bridge the gap between what a user asks and what OwnSearch should retrieve. Treat retrieval as probabilistic: rewrite weak queries, run multiple targeted searches when needed, prefer grounded context over guesswork, and fetch exact chunks before making precise claims.
+
+## Retrieval Workflow
+
+1. Classify the user request.
+2. Generate one to four retrieval queries.
+3. Start with `search_context` for the strongest query.
+4. Expand to additional searches only if evidence is weak, duplicate-heavy, or incomplete.
+5. Use `get_chunks` after `search` when the answer needs exact wording, detailed comparison, or citation-grade grounding.
+6. Answer only from retrieved evidence. Say when the retrieved context is insufficient.
+
+## Query Planning
+
+Generate retrieval queries with these patterns:
+
+- Literal query: preserve the exact noun phrase, error string, rule name, or title the user used.
+- Canonical query: replace vague wording with domain terms likely to appear in documents.
+- Paraphrase query: restate the intent in simpler or more explicit language.
+- Source-biased query: add likely file names, section names, or path hints when the user names a source.
+
+Good examples:
+
+- User ask: "How do concentration checks work?"
+  Queries:
+  - `concentration checks`
+  - `maintain concentration after taking damage`
+  - `constitution saving throw concentration spell`
+
+- User ask: "Where does the repo explain local MCP setup?"
+  Queries:
+  - `local MCP setup`
+  - `Model Context Protocol setup`
+  - `serve-mcp agent config`
+
+- User ask: "What did the contract say about payment timing?"
+  Queries:
+  - `payment timing`
+  - `payment due within`
+  - `invoice due date net terms`
+
+## Tool Use Rules
+
+Use `search_context` when:
+
+- the user wants an answer, summary, explanation, or quick grounding
+- the answer can be supported by a few chunks
+- low latency matters more than exhaustive recall
+
+Use `search` when:
+
+- you want to inspect ranking and source distribution
+- you need to compare multiple candidates
+- you suspect duplicates or poor recall
+
+Use `get_chunks` when:
+
+- exact wording matters
+- the answer depends on adjacent details
+- you need to quote or carefully verify a claim
+- you need to compare similar hits before answering
+
+## Duplicate Handling
+
+Assume top results can still contain semantic duplicates.
+
+When results are duplicate-heavy:
+
+- keep only the strongest chunk per repeated claim unless neighboring chunks add new facts
+- prefer source diversity when multiple files say the same thing
+- if one document clearly appears authoritative, prefer that source but mention corroboration when useful
+- if the top results are all from one file and the answer still seems incomplete, issue a second query with a different phrasing
+
+## Failure Recovery
+
+If the first search is weak:
+
+- shorten the query
+- remove conversational filler
+- swap vague words for canonical terms
+- split compound questions into separate searches
+- add likely section names or file hints
+- search once for the concept and once for the expected answer shape
+
+Examples:
+
+- "Can you tell me what they said about when we can terminate this thing?"
+  Retry with:
+  - `termination`
+  - `termination notice`
+  - `right to terminate`
+  - `termination for cause`
+
+- "Why is my build exploding around env handling?"
+  Retry with:
+  - `environment variables`
+  - `dotenv`
+  - `GEMINI_API_KEY`
+  - `setup envPath`
+
+## Answering Rules
+
+- Do not invent facts that were not retrieved.
+- Prefer citing file paths or chunk provenance when the client supports it.
+- If retrieval is partial, say which part is grounded and which part is uncertain.
+- If evidence conflicts, surface the conflict instead of averaging it away.
+- If nothing relevant is retrieved after a few query variants, say so explicitly.
+
+## Minimal Playbook
+
+For a normal grounded answer:
+
+1. Derive two or three strong retrieval queries.
+2. Call `search_context` with the best query.
+3. If results look sufficient, answer from them.
+4. If results look weak or ambiguous, call `search` with another variant.
+5. Fetch exact chunks for the best IDs before making precise claims.
+
+For a locate-the-source task:
+
+1. Use `search` first.
+2. Inspect which files dominate.
+3. Use `get_chunks` on top hits.
+4. Return the most relevant files and sections, not just a prose answer.
+
+For a compare-or-summarize task:
+
+1. Run one query per subtopic.
+2. Collect grounded chunks from each.
+3. Merge only non-duplicate evidence.
+4. Summarize with explicit source-backed differences.

package/skills/ownsearch-rag-search/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "OwnSearch RAG Search"
+  short_description: "Sharpen OwnSearch retrieval"
+  default_prompt: "Use $ownsearch-rag-search to turn my request into stronger OwnSearch retrieval and grounded answering."