docshark 0.1.17 → 0.1.21

package/README.md

@@ -0,0 +1,176 @@
+# 🦈 DocShark
+
+[![Built with Bun](https://img.shields.io/badge/Bun-%23000000.svg?style=flat&logo=bun&logoColor=white)](https://bun.sh/)
+[![NPM Version](https://img.shields.io/npm/v/docshark.svg?style=flat&color=blue)](https://www.npmjs.com/package/docshark)
+[![MCP Compatible](https://img.shields.io/badge/MCP-Ready-0D1117.svg?style=flat&logo=github&logoColor=white)](https://modelcontextprotocol.io/)
+[![GitHub Release](https://img.shields.io/github/v/release/Michael-Obele/docshark?color=success)](https://github.com/Michael-Obele/docshark/releases)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+**DocShark** is a powerful MCP (Model Context Protocol) server designed to scrape, index, and search any documentation website. It creates a local, highly-searchable knowledge base from public documentation pages using FTS5 (Full-Text Search) and BM25 ranking, allowing AI assistants to query the latest docs effortlessly.
+
+---
+
+## 🚀 Features
+
+- **Automated Crawling**: Discovers pages via `sitemap.xml` with fallback to BFS link crawling.
+- **Smart Extraction**: Uses Readability and Turndown to extract main content and convert it to clean Markdown, filtering out navbars and sidebars.
+- **Semantic Chunking**: Splits content based on headings, preserving contextual headers for better AI understanding.
+- **High-Performance Search**: Built-in SQLite + FTS5 indexing with BM25 ranking for accurate and lightning-fast search results.
+- **JS-Rendered Site Support**: Tiered fetching strategy automatically detects React/Vue SPAs (empty shells) and upgrades to `puppeteer-core` if you have it installed (zero-config, auto-fallback).
+- **Polite Crawling**: Respects `robots.txt` and implements rate limiting to prevent overloading documentation servers.
+- **Standard MCP Tooling**: Connect perfectly with Desktop Claude, VS Code, Cursor, and any other MCP-compatible clients via standard `stdio` or `http`/`sse` transports.
+
+## 📦 What We Have Done (Phase 1)
+
+**Phase 1: Core Engine** is fully implemented and tested.
+
+- ✅ Custom SQLite Database with FTS5 virtual tables and auto-sync triggers.
+- ✅ Web scraping engine supporting standard `fetch()` and `puppeteer-core`.
+- ✅ Markdown processor utilizing Readability + Turndown.
+- ✅ Heading-based semantic chunker (500-1200 tokens per chunk).
+- ✅ Asynchronous job manager and queue system.
+- ✅ Complete HTTP API (REST endpoints + SSE event streams).
+- ✅ Seamless integration of 4 MCP tools: `manage_library`, `search_docs`, `list_libraries`, and `get_doc_page`.
+- ✅ Robust CLI interface (`start`, `add`, `rename`, `search`, `list`).
+
+## 🏗️ What We Are Doing
+
+We are actively polishing the integration between the core engine and external MCP clients (like VS Code Agents and Claude Desktop).
+
+## 🔮 What We Plan To Do (Phase 2 & Beyond)
+
+- **Web Dashboard**: An intuitive SvelteKit dashboard to manage your synced libraries, view crawl progress in real-time (via SSE), and test searches manually.
+- **Incremental Crawling**: Smarter `refresh` jobs that compare `ETag` and `Last-Modified` headers to only re-scrape updated pages.
+- **Vector Search (RAG)**: Integration of lightweight vector embeddings for semantic similarity search alongside the existing FTS5 keyword search.
+- **Advanced Scraping Setup**: Support for custom CSS selectors to define exactly where content lives in non-standard documentation websites.
+
+---
+
+## 🛠️ Usage
+
+### Quick Start (from npm)
+
+You can run DocShark directly without installing it globally using `bunx`:
+
+```bash
+# Add a documentation library to the index
+bunx docshark add https://valibot.dev/guides/ --depth 2
+
+# Search your indexed docs
+bunx docshark search "schema validation"
+```
+
+### Installation
+
+To install DocShark globally as a CLI tool:
+
+DocShark is intended to be installed and run with Bun.
+
+```bash
+# Global Bun installation
+bun add -g docshark
+```
+
+After installation, you can use the `docshark` command:
+
+```bash
+docshark list
+
+# Update the global Bun installation when a new release is published
+docshark update
+
+# Script-friendly update check
+docshark update --check --quiet
+```
+
+Interactive CLI runs will also let you know when a newer version is available. Update notices are intentionally skipped for MCP `stdio` mode so they never interfere with protocol output.
+
+For scripts, `docshark update --check` exits `0` when current, `10` when a newer version is available, and `1` when the version check could not be completed.
+
+## 🔌 MCP Integration
+
+### VS Code (GitHub Copilot / MCP Extension)
+
+Add DocShark to your `.vscode/settings.json` or global MCP configuration:
+
+```json
+{
+  "mcpServers": {
+    "docshark": {
+      "command": "bunx",
+      "args": ["-y", "docshark", "start", "--stdio"]
+    }
+  }
+}
+```
+
+### Cursor
+
+1. Open **Cursor Settings** > **Models** > **MCP**.
+2. Click **+ Add New MCP Server**.
+3. Name: `docshark`
+4. Type: `command`
+5. Command: `bunx -y docshark start --stdio`
+
+### Claude Desktop
+
+Edit your Claude Desktop configuration file:
+
+- **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`
+- **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`
+
+```json
+{
+  "mcpServers": {
+    "docshark": {
+      "command": "bunx",
+      "args": ["-y", "docshark", "start", "--stdio"]
+    }
+  }
+}
+```
+
+---
+
+## 🛠️ Development
+
+### Local Setup
+
+Ensure you have [Bun](https://bun.sh/) installed.
+
+```bash
+# Clone the repository
+git clone https://github.com/Michael-Obele/docshark.git
+cd docshark
+
+# Install dependencies
+bun install
+
+# (Optional) Enable auto-detection & scraping of Javascript React/Vue single-page apps
+bun add puppeteer-core
+
+# Start the DocShark MCP server in HTTP mode for local testing
+bun run src/cli.ts start --port 6380
+```
+
+### Local CLI Debugging
+
+```bash
+# Run CLI directly while developing
+bun run src/cli.ts list
+```
+
+## 🔄 Versioning & Changelog
+
+This project uses [Google's Release Please](https://github.com/googleapis/release-please) to automate versioning and changelog generation.
+
+- **Semantic Versioning**: Our versions automatically bump (e.g. `0.0.1` -> `0.0.2` or `0.1.0`) based on standard Conventional Commits (`feat:`, `fix:`, `chore:`, etc.).
+- **Automated**: A PR is automatically created on `master` when standard commits are merged, generating a standard `CHANGELOG.md`.
+
+## 📜 License
+
+This project is open-source and available under the [MIT License](LICENSE).
+
+---
+
+_Built to empower AI agents with the latest knowledge._

package/dist/cli.js

@@ -5,6 +5,7 @@ import { startHttpServer } from "./http.js";
 import { StdioTransport } from "@tmcp/transport-stdio";
 import { server, db, searchEngine, libraryService } from "./server.js";
 import { maybeNotifyAboutUpdate, runUpdateCommand } from "./cli-update.js";
+import { formatSearchResults } from "./search/format-results.js";
 import { VERSION } from "./version.js";
 const useColor = process.stdout.isTTY;
 const color = {
@@ -160,12 +161,7 @@ cli
         console.log(`\nNo results found for "${query}".\n`);
         return;
     }
-    for (const r of results) {
-        console.log(`\n--- ${r.page_title} (${r.library_display_name}) ---`);
-        console.log(`Section: ${r.heading_context}`);
-        console.log(r.content.slice(0, 300));
-        console.log(`Source: ${r.page_url}\n`);
-    }
+    console.log(`\n${formatSearchResults(query, results)}\n`);
 });
 cli
     .command("list", "List indexed libraries")

package/dist/jobs/manager.d.ts

@@ -9,6 +9,7 @@ export declare class JobManager {
     /** Start a crawl job for a library */
     startCrawl(libraryId: string, opts?: {
         incremental?: boolean;
+        sessionId?: string;
     }): CrawlJob;
     /** Get status of a specific job */
     getJob(jobId: string): CrawlJob | undefined;

package/dist/jobs/manager.js

@@ -12,7 +12,7 @@ export class JobManager {
     /** Start a crawl job for a library */
     startCrawl(libraryId, opts) {
         const jobId = nanoid();
-        const job = this.db.createJob({ id: jobId, libraryId });
+        const job = this.db.createJob({ id: jobId, libraryId, sessionId: opts?.sessionId });
         // Run crawl async (non-blocking)
         const worker = new CrawlWorker(this.db, this.eventBus);
         this.activeJobs.set(jobId, worker);

package/dist/scraper/rate-limiter.js

@@ -2,7 +2,7 @@
 export class RateLimiter {
     delayMs;
     lastRequest = 0;
-    constructor(delayMs = 500) {
+    constructor(delayMs = 200) {
         this.delayMs = delayMs;
     }
     async wait() {

package/dist/scripts/sync-version.js

@@ -4,20 +4,32 @@ import { fileURLToPath } from "node:url";
 const scriptDir = dirname(fileURLToPath(import.meta.url));
 const packageJsonPath = resolve(scriptDir, "../../package.json");
 const versionFilePath = resolve(scriptDir, "../version.ts");
+const releaseVersion = process.env.DOCSHARK_VERSION?.trim();
 const packageJson = JSON.parse(readFileSync(packageJsonPath, "utf8"));
-if (typeof packageJson.version !== "string") {
+const targetVersion = releaseVersion && releaseVersion.length > 0
+    ? releaseVersion
+    : packageJson.version;
+if (typeof targetVersion !== "string") {
     throw new Error("packages/core/package.json is missing a valid version.");
 }
 const nextVersionFile = [
     "// This file is automatically updated by the version sync script.",
-    `export const VERSION = '${packageJson.version}';`,
+    `export const VERSION = '${targetVersion}';`,
     "",
 ].join("\n");
 const currentVersionFile = readFileSync(versionFilePath, "utf8");
 if (currentVersionFile !== nextVersionFile) {
     writeFileSync(versionFilePath, nextVersionFile, "utf8");
-    console.log(`Updated src/version.ts to ${packageJson.version}`);
+    console.log(`Updated src/version.ts to ${targetVersion}`);
 }
 else {
-    console.log(`src/version.ts already matches ${packageJson.version}`);
+    console.log(`src/version.ts already matches ${targetVersion}`);
+}
+if (releaseVersion && packageJson.version !== targetVersion) {
+    const nextPackageJson = {
+        ...packageJson,
+        version: targetVersion,
+    };
+    writeFileSync(packageJsonPath, `${JSON.stringify(nextPackageJson, null, 2)}\n`, "utf8");
+    console.log(`Updated package.json to ${targetVersion}`);
 }

package/dist/search/format-results.d.ts

@@ -0,0 +1,2 @@
+import type { SearchResult } from './types.js';
+export declare function formatSearchResults(query: string, results: SearchResult[]): string;

package/dist/search/format-results.js

@@ -0,0 +1,23 @@
+import { sanitizeDocContent } from './sanitize.js';
+function formatReasons(reasons) {
+    if (reasons.length === 0) {
+        return '';
+    }
+    return `**Why this ranked highly:** ${reasons.join(', ')}\n\n`;
+}
+export function formatSearchResults(query, results) {
+    const formatted = results
+        .map((result, index) => {
+        let block = `### ${index + 1}. ${result.page_title} — ${result.library_display_name}\n`;
+        block += `**Source:** ${result.page_url}\n`;
+        if (result.heading_context.trim().length > 0) {
+            block += `**Section:** ${result.heading_context}\n`;
+        }
+        // Sanitize content to prevent prompt injection
+        const sanitizedContent = sanitizeDocContent(result.content);
+        block += `${formatReasons(result.reasons)}${sanitizedContent}`;
+        return block;
+    })
+        .join('\n\n---\n\n');
+    return `## Results for "${query}"\n\n${formatted}`;
+}

package/dist/search/query-planner.d.ts

@@ -0,0 +1,7 @@
+import type { SearchPlan } from "./types.js";
+export declare function normalizeSearchText(value: string): string;
+export declare class QueryPlanner {
+    build(query: string, library?: string): SearchPlan;
+    private detectIntent;
+    private extractVersion;
+}

package/dist/search/query-planner.js

@@ -0,0 +1,88 @@
+const STOP_WORDS = new Set([
+    "a",
+    "an",
+    "and",
+    "are",
+    "at",
+    "do",
+    "for",
+    "how",
+    "i",
+    "in",
+    "is",
+    "of",
+    "on",
+    "or",
+    "the",
+    "to",
+    "what",
+    "with",
+]);
+const PHRASE_HINTS = [
+    "getting started",
+    "quickstart",
+    "overview",
+    "reference",
+    "api",
+    "troubleshooting",
+];
+export function normalizeSearchText(value) {
+    return value
+        .toLowerCase()
+        .replace(/[^a-z0-9@/._\-\s]+/g, " ")
+        .replace(/\s+/g, " ")
+        .trim();
+}
+function sanitizeToken(value) {
+    return value.replace(/^[^a-z0-9@/._-]+|[^a-z0-9@/._-]+$/gi, "").toLowerCase();
+}
+export class QueryPlanner {
+    build(query, library) {
+        const normalizedQuery = normalizeSearchText(query);
+        const rawTokens = normalizedQuery
+            .split(/\s+/)
+            .map((token) => sanitizeToken(token))
+            .filter(Boolean);
+        const filteredKeywords = Array.from(new Set(rawTokens.filter((token) => token.length > 1 && !STOP_WORDS.has(token))));
+        return {
+            original_query: query,
+            normalized_query: normalizedQuery,
+            intent: this.detectIntent(normalizedQuery),
+            keywords: filteredKeywords.length > 0
+                ? filteredKeywords
+                : Array.from(new Set(rawTokens)),
+            phrases: PHRASE_HINTS.filter((phrase) => normalizedQuery.includes(phrase)),
+            requested_library: library,
+            requested_version: this.extractVersion(normalizedQuery),
+        };
+    }
+    detectIntent(query) {
+        if (query.includes("getting started") ||
+            query.includes("quickstart") ||
+            query.startsWith("install ") ||
+            query.startsWith("setup ")) {
+            return "getting_started";
+        }
+        if (query.includes("overview") ||
+            query.startsWith("what is ") ||
+            query.startsWith("about ")) {
+            return "overview";
+        }
+        if (/[a-z]+\.[a-z]+/.test(query) ||
+            /[A-Z][a-zA-Z]+\(/.test(query) ||
+            query.includes(" api") ||
+            query.endsWith(" api") ||
+            query.includes("reference") ||
+            query.includes("@")) {
+            return "api_lookup";
+        }
+        if (/error|fail|issue|problem|broken|debug|fix|troubleshoot/.test(query)) {
+            return "troubleshooting";
+        }
+        return "general";
+    }
+    extractVersion(query) {
+        const match = query.match(/\bv(?:ersion)?\s*(\d+)\b/);
+        return match?.[1];
+    }
+}

package/dist/search/sanitize.d.ts

@@ -0,0 +1,10 @@
+/**
+ * Output sanitization to prevent prompt injection attacks
+ * Removes suspicious patterns that could escape agent context
+ */
+export declare function sanitizeOutput(text: string): string;
+/**
+ * Sanitize a single chunk of documentation content
+ * Removes malicious content while preserving code blocks and formatting
+ */
+export declare function sanitizeDocContent(content: string): string;

package/dist/search/sanitize.js

@@ -0,0 +1,40 @@
+/**
+ * Output sanitization to prevent prompt injection attacks
+ * Removes suspicious patterns that could escape agent context
+ */
+export function sanitizeOutput(text) {
+    return text
+        // Remove template directives
+        .replace(/\{[#%].*?[#%]\}/g, '')
+        // Remove system prompt markers
+        .replace(/\[SYSTEM[\]:].*?\[\/SYSTEM\]/gi, '')
+        .replace(/\[ADMIN[\]:].*?\[\/ADMIN\]/gi, '')
+        // Remove potential prompt injection patterns
+        .replace(/ignore\s+above.*?instructions/gi, '')
+        .replace(/forget\s+previous.*?context/gi, '')
+        .trim();
+}
+/**
+ * Sanitize a single chunk of documentation content
+ * Removes malicious content while preserving code blocks and formatting
+ */
+export function sanitizeDocContent(content) {
+    // First sanitize for injection patterns
+    let sanitized = sanitizeOutput(content);
+    // Escape potential dangerous markdown constructs
+    // but preserve code blocks (between triple backticks)
+    const codeBlockPattern = /```[\s\S]*?```/g;
+    const codeBlocks = sanitized.match(codeBlockPattern) || [];
+    // Temporarily replace code blocks
+    let temp = sanitized;
+    codeBlocks.forEach((block, i) => {
+        temp = temp.replace(block, `__CODE_BLOCK_${i}__`);
+    });
+    // Sanitize outside code blocks
+    temp = sanitizeOutput(temp);
+    // Restore code blocks
+    codeBlocks.forEach((block, i) => {
+        temp = temp.replace(`__CODE_BLOCK_${i}__`, block);
+    });
+    return temp;
+}

package/dist/search/types.d.ts

@@ -0,0 +1,33 @@
+export type SearchIntent = "general" | "overview" | "getting_started" | "api_lookup" | "troubleshooting";
+export interface SearchOptions {
+    library?: string;
+    limit?: number;
+}
+export interface SearchPlan {
+    original_query: string;
+    normalized_query: string;
+    intent: SearchIntent;
+    keywords: string[];
+    phrases: string[];
+    requested_version?: string;
+    requested_library?: string;
+}
+export interface SearchCandidate {
+    content: string;
+    heading_context: string;
+    page_url: string;
+    page_path: string;
+    page_title: string;
+    library_name: string;
+    library_display_name: string;
+    lexical_score: number;
+    has_code_block: boolean;
+    token_count: number;
+    chunk_index: number;
+}
+export interface SearchResult extends SearchCandidate {
+    rerank_score: number;
+    reasons: string[];
+    path_type: string;
+    version_tag: string | null;
+}

package/dist/search/types.js

@@ -0,0 +1 @@
+export {};

package/dist/server.js

@@ -5,6 +5,7 @@ import * as v from "valibot";
 import { tool } from "tmcp/utils";
 import { Database } from "./storage/db.js";
 import { SearchEngine } from "./storage/search.js";
+import { formatSearchResults } from "./search/format-results.js";
 import { LibraryService } from "./services/library.js";
 import { JobManager } from "./jobs/manager.js";
 import { VERSION } from "./version.js";
@@ -32,29 +33,27 @@ export const server = new McpServer({
 // ──────────────────────────────────────
 server.tool({
     name: "search_docs",
-    description: "Search through indexed documentation libraries for relevant information. " +
-        "Returns ranked documentation sections with code examples and source URLs. " +
-        "Use this when you need to find information about a library, framework, API, " +
-        "or any technical concept.",
+    description: "Search indexed docs by keyword or library. Returns ranked sections with URLs.",
+    annotations: {
+        readOnlyHint: true,
+        idempotentHint: true,
+    },
     schema: v.object({
         query: v.pipe(v.string(), v.description("Search query. Use natural language.")),
         library: v.optional(v.pipe(v.string(), v.description("Filter to a specific library."))),
         limit: v.optional(v.pipe(v.number(), v.integer(), v.minValue(1), v.maxValue(20)), 5),
     }),
 }, async ({ query, library, limit }) => {
-    const results = searchEngine.search(query, { library, limit });
-    if (results.length === 0)
-        return tool.text(`No results found for "${query}".`);
-    const formatted = results
-        .map((r, i) => {
-        let block = `### ${i + 1}. ${r.page_title} — ${r.library_display_name}\n`;
-        block += `**Source:** ${r.page_url}\n`;
-        block += `**Section:** ${r.heading_context}\n\n`;
-        block += r.content;
-        return block;
-    })
-        .join("\n\n---\n\n");
-    return tool.text(`## Results for "${query}"\n\n${formatted}`);
+    try {
+        const results = searchEngine.search(query, { library, limit });
+        if (results.length === 0)
+            return tool.text(`No results found for "${query}".`);
+        return tool.text(formatSearchResults(query, results));
+    }
+    catch (err) {
+        const message = err instanceof Error ? err.message : "Search failed";
+        return tool.text(`❌ Error: ${message}`);
+    }
 });
 function requireValue(value, message) {
     if (value === undefined || value === null || value === "") {
@@ -89,52 +88,84 @@ function formatLibraryInfo(libraryId) {
     return output;
 }
 // ──────────────────────────────────────
-// Tool 2: list_libraries — Discovery tool
+// Tool 2: list_libraries — Discovery tool with pagination
 // ──────────────────────────────────────
 server.tool({
     name: "list_libraries",
-    description: "List all documentation libraries currently indexed and available for searching. " +
-        "Use this to discover what docs are available before running search_docs.",
+    description: "List indexed documentation libraries. Paginated results.",
+    annotations: {
+        readOnlyHint: true,
+        idempotentHint: true,
+    },
     schema: v.object({
         status: v.optional(v.pipe(v.picklist(["indexed", "crawling", "error", "all"]), v.description('Filter by status. Default: "all".')), "all"),
+        page: v.optional(v.pipe(v.number(), v.integer(), v.minValue(1)), 1),
+        limit: v.optional(v.pipe(v.number(), v.integer(), v.minValue(1), v.maxValue(50)), 20),
     }),
-}, async ({ status }) => {
-    const libraries = db.listLibraries(status);
-    if (libraries.length === 0) {
-        return tool.text("No libraries indexed yet. Use manage_library with action=add to add a documentation website.");
+}, async ({ status, page = 1, limit = 20 }) => {
+    try {
+        const libraries = db.listLibraries(status);
+        if (libraries.length === 0) {
+            return tool.text("No libraries indexed yet. Use manage_library with action=add to add a documentation website.");
+        }
+        // Paginate results
+        const start = (page - 1) * limit;
+        const end = start + limit;
+        const paginated = libraries.slice(start, end);
+        const hasMore = end < libraries.length;
+        // Minified response (no pretty-printing)
+        let output = `## Libraries (${start + 1}-${Math.min(end, libraries.length)} of ${libraries.length})\n\n`;
+        output += "| Library | URL | Pages | Chunks | Status |\n";
+        output += "| ------- | --- | ----- | ------ | ------ |\n";
+        for (const lib of paginated) {
+            output += `|${lib.name}|${lib.url}|${lib.page_count}|${lib.chunk_count}|${lib.status}|\n`;
+        }
+        if (hasMore) {
+            output += `\n**More available.** Use page=${page + 1} to fetch next page.`;
+        }
+        return tool.text(output);
     }
-    let output = `## Indexed Libraries (${libraries.length} total)\n\n`;
-    output += "| Library | URL | Pages | Chunks | Status |\n";
-    output += "| ------- | --- | ----- | ------ | ------ |\n";
-    for (const lib of libraries) {
-        output += `| ${lib.name} | ${lib.url} | ${lib.page_count} | ${lib.chunk_count} | ${lib.status} |\n`;
+    catch (err) {
+        const message = err instanceof Error ? err.message : "Failed to list libraries";
+        return tool.text(`❌ Error: ${message}`);
     }
-    return tool.text(output);
 });
 // ──────────────────────────────────────
 // Tool 3: get_doc_page — Full page read
 // ──────────────────────────────────────
 server.tool({
     name: "get_doc_page",
-    description: "Retrieve the complete content of a specific documentation page as markdown. " +
-        "Use when search results reference a page and you need full context.",
+    description: "Retrieve complete documentation page as markdown.",
+    annotations: {
+        readOnlyHint: true,
+        idempotentHint: true,
+    },
     schema: v.object({
         url: v.optional(v.pipe(v.string(), v.description("The full URL of the documentation page."))),
         library: v.optional(v.pipe(v.string(), v.description("Library name to search within."))),
         path: v.optional(v.pipe(v.string(), v.description("Relative path within the library."))),
     }),
 }, async ({ url, library, path }) => {
-    const page = db.getPage({ url, library, path });
-    if (!page)
-        return tool.text("Page not found. Use search_docs to find the correct page.");
-    return tool.text(`# ${page.title}\n**Source:** ${page.url}\n\n${page.content_markdown}`);
+    try {
+        const page = db.getPage({ url, library, path });
+        if (!page)
+            return tool.text("Page not found. Use search_docs to find the correct page.");
+        return tool.text(`# ${page.title}\n**Source:** ${page.url}\n\n${page.content_markdown}`);
+    }
+    catch (err) {
+        const message = err instanceof Error ? err.message : "Failed to fetch page";
+        return tool.text(`❌ Error: ${message}`);
+    }
 });
 // ──────────────────────────────────────
 // Tool 4: manage_library — Create, rename, refresh, remove, inspect
 // ──────────────────────────────────────
 server.tool({
     name: "manage_library",
-    description: "Manage a documentation library lifecycle. Use action=add to crawl a new source, action=rename to change the library name, action=refresh to re-crawl, action=remove to delete it, or action=info to inspect its pages and stats.",
+    description: "Manage library lifecycle: add/rename/refresh/remove/info. Destructive actions require confirmation.",
+    annotations: {
+        destructiveHint: true,
+    },
     schema: v.object({
         action: v.pipe(v.picklist(["add", "rename", "refresh", "remove", "info"]), v.description("The management action to perform.")),
         url: v.optional(v.pipe(v.string(), v.url(), v.description("Base URL of the documentation website."))),
@@ -170,7 +201,7 @@ server.tool({
                 const libraryName = requireValue(input.library, "library is required for action=refresh.");
                 const lib = db.getLibraryByName(libraryName);
                 if (!lib)
-                    return tool.text(`Library "${libraryName}" not found. Use list_libraries to see available.`);
+                    return tool.text(`❌ Library "${libraryName}" not found. Use list_libraries to see available.`);
                 const job = jobManager.startCrawl(lib.id, { incremental: true });
                 return tool.text(`🔄 Refresh started for "${lib.display_name}".\nJob ${job.id}: checking for updated pages...`);
             }
@@ -178,7 +209,7 @@ server.tool({
                 const libraryName = requireValue(input.library, "library is required for action=remove.");
                 const lib = db.getLibraryByName(libraryName);
                 if (!lib)
-                    return tool.text(`Library "${libraryName}" not found.`);
+                    return tool.text(`❌ Library "${libraryName}" not found.`);
                 db.removeLibrary(lib.id);
                 return tool.text(`🗑️ Library "${lib.display_name}" removed.\nDeleted ${lib.page_count} pages and ${lib.chunk_count} chunks.`);
             }
@@ -186,14 +217,14 @@ server.tool({
                 const libraryName = requireValue(input.library, "library is required for action=info.");
                 const lib = db.getLibraryByName(libraryName);
                 if (!lib)
-                    return tool.text(`Library "${libraryName}" not found. Use list_libraries to see available libraries.`);
+                    return tool.text(`❌ Library "${libraryName}" not found. Use list_libraries to see available libraries.`);
                 return tool.text(formatLibraryInfo(lib.id));
             }
         }
     }
     catch (err) {
         const message = err instanceof Error ? err.message : "Unknown error";
-        return tool.text(`❌ Failed: ${message}`);
+        return tool.text(`❌ Error: ${message}`);
     }
-    return tool.text(`❌ Failed: Unsupported action.`);
+    return tool.text(`❌ Error: Unsupported action.`);
 });

package/dist/storage/db.d.ts

@@ -51,6 +51,7 @@ export declare class Database {
     createJob(job: {
         id: string;
         libraryId: string;
+        sessionId?: string;
     }): CrawlJob;
     getJob(id: string): CrawlJob | undefined;
     updateJob(id: string, updates: Partial<Pick<CrawlJob, "status" | "pages_discovered" | "pages_crawled" | "pages_failed" | "chunks_created" | "error_message" | "started_at" | "completed_at">>): void;

package/dist/storage/db.js

@@ -92,6 +92,7 @@ export class Database {
       CREATE TABLE IF NOT EXISTS crawl_jobs (
         id               TEXT PRIMARY KEY,
         library_id       TEXT NOT NULL REFERENCES libraries(id) ON DELETE CASCADE,
+        session_id       TEXT,
         status           TEXT NOT NULL DEFAULT 'queued',
         pages_discovered INTEGER NOT NULL DEFAULT 0,
         pages_crawled    INTEGER NOT NULL DEFAULT 0,
@@ -213,8 +214,8 @@ export class Database {
     // ──────────────────────────────────────
     createJob(job) {
         this.db
-            .prepare("INSERT INTO crawl_jobs (id, library_id) VALUES (?, ?)")
-            .run(job.id, job.libraryId);
+            .prepare("INSERT INTO crawl_jobs (id, library_id, session_id) VALUES (?, ?, ?)")
+            .run(job.id, job.libraryId, job.sessionId ?? null);
         return this.db
             .prepare("SELECT * FROM crawl_jobs WHERE id = ?")
             .get(job.id);

package/dist/storage/search.d.ts

@@ -1,21 +1,23 @@
-import type { Database } from './db.js';
-export interface SearchResult {
-    content: string;
-    heading_context: string;
-    page_url: string;
-    page_title: string;
-    library_name: string;
-    library_display_name: string;
-    relevance_score: number;
-    has_code_block: boolean;
-    token_count: number;
-}
+import type { Database } from "./db.js";
+import type { SearchOptions, SearchResult } from "../search/types.js";
+export type { SearchOptions, SearchResult } from "../search/types.js";
 export declare class SearchEngine {
     private db;
+    private planner;
     constructor(db: Database);
-    search(query: string, opts?: {
-        library?: string;
-        limit?: number;
-    }): SearchResult[];
-    private sanitizeQuery;
+    search(query: string, opts?: SearchOptions): SearchResult[];
+    private fetchCandidates;
+    private buildFtsQuery;
+    private quoteTerm;
+    private rerank;
+    private scoreCandidate;
+    private collapseDuplicates;
+    private preferenceScore;
+    private canonicalPageKey;
+    private inferPathType;
+    private pathTypeScore;
+    private keywordOverlap;
+    private hasPhraseMatch;
+    private primaryTitle;
+    private extractVersionTag;
 }

package/dist/storage/search.js

@@ -1,49 +1,323 @@
+import { QueryPlanner, normalizeSearchText } from "../search/query-planner.js";
 export class SearchEngine {
     db;
+    planner = new QueryPlanner();
     constructor(db) {
         this.db = db;
     }
     search(query, opts = {}) {
         const limit = opts.limit ?? 5;
-        const ftsQuery = this.sanitizeQuery(query);
+        const plan = this.planner.build(query, opts.library);
+        const ftsQuery = this.buildFtsQuery(plan);
         if (!ftsQuery)
             return [];
         try {
-            const stmt = this.db.raw().prepare(`
-        SELECT
-          c.content,
-          c.heading_context,
-          c.has_code_block,
-          c.token_count,
-          p.url   AS page_url,
-          p.title AS page_title,
-          l.name  AS library_name,
-          l.display_name AS library_display_name,
-          bm25(chunks_fts, 1.0, 0.5) AS relevance_score
-        FROM chunks_fts
-        JOIN chunks c ON chunks_fts.rowid = c.rowid
-        JOIN pages p  ON c.page_id = p.id
-        JOIN libraries l ON c.library_id = l.id
-        WHERE chunks_fts MATCH ?
-          AND (? IS NULL OR l.name = ?)
-        ORDER BY relevance_score
-        LIMIT ?
-      `);
-            return stmt.all(ftsQuery, opts.library ?? null, opts.library ?? null, limit);
+            const candidates = this.fetchCandidates(ftsQuery, opts.library, limit);
+            if (candidates.length === 0) {
+                return [];
+            }
+            const reranked = this.rerank(plan, candidates);
+            return this.collapseDuplicates(plan, reranked).slice(0, limit);
         }
         catch (err) {
-            // FTS5 query might fail with bad syntax — return empty
             console.warn(`[DocShark] Search failed:`, err.message);
             return [];
         }
     }
-    sanitizeQuery(query) {
-        // Remove FTS5 special operators for safety, wrap terms in quotes
-        return query
-            .replace(/['"]/g, '')
-            .split(/\s+/)
-            .filter(Boolean)
-            .map((term) => `"${term}"`)
-            .join(' OR ');
+    fetchCandidates(ftsQuery, library, limit) {
+        const candidateLimit = Math.min(Math.max(limit * 12, 25), 80);
+        const stmt = this.db.raw().prepare(`
+      SELECT
+        c.content,
+        COALESCE(c.heading_context, '') AS heading_context,
+        c.has_code_block,
+        COALESCE(c.token_count, 0) AS token_count,
+        c.chunk_index,
+        p.url AS page_url,
+        p.path AS page_path,
+        COALESCE(p.title, 'Untitled') AS page_title,
+        l.name AS library_name,
+        l.display_name AS library_display_name,
+        bm25(chunks_fts, 1.0, 0.7) AS lexical_score
+      FROM chunks_fts
+      JOIN chunks c ON chunks_fts.rowid = c.rowid
+      JOIN pages p ON c.page_id = p.id
+      JOIN libraries l ON c.library_id = l.id
+      WHERE chunks_fts MATCH ?
+        AND (? IS NULL OR l.name = ?)
+      ORDER BY lexical_score
+      LIMIT ?
+    `);
+        const rows = stmt.all(ftsQuery, library ?? null, library ?? null, candidateLimit);
+        return rows.map((row) => ({
+            ...row,
+            has_code_block: row.has_code_block === 1,
+        }));
+    }
+    buildFtsQuery(plan) {
+        const clauses = new Set();
+        const exactQuery = this.quoteTerm(plan.normalized_query);
+        if (plan.normalized_query.length > 0) {
+            clauses.add(exactQuery);
+        }
+        for (const phrase of plan.phrases) {
+            clauses.add(this.quoteTerm(phrase));
+        }
+        for (const keyword of plan.keywords) {
+            clauses.add(this.quoteTerm(keyword));
+        }
+        if (plan.keywords.length > 1 && plan.keywords.length <= 6) {
+            clauses.add(`(${plan.keywords.map((keyword) => this.quoteTerm(keyword)).join(" AND ")})`);
+        }
+        return Array.from(clauses).join(" OR ");
+    }
+    quoteTerm(value) {
+        return `"${value.replace(/["']/g, "").trim()}"`;
+    }
+    rerank(plan, candidates) {
+        const total = Math.max(candidates.length, 1);
+        return candidates
+            .map((candidate, index) => this.scoreCandidate(plan, candidate, index, total))
+            .sort((left, right) => {
+            if (right.rerank_score !== left.rerank_score) {
+                return right.rerank_score - left.rerank_score;
+            }
+            return left.lexical_score - right.lexical_score;
+        });
+    }
+    scoreCandidate(plan, candidate, index, total) {
+        const title = normalizeSearchText(candidate.page_title);
+        const primaryTitle = normalizeSearchText(this.primaryTitle(candidate.page_title));
+        const heading = normalizeSearchText(candidate.heading_context);
+        const path = normalizeSearchText(candidate.page_path);
+        const libraryText = normalizeSearchText(`${candidate.library_name} ${candidate.library_display_name}`);
+        const contentPreview = normalizeSearchText(candidate.content.slice(0, 800));
+        const pathType = this.inferPathType(candidate.page_path, candidate.page_title);
+        const versionTag = this.extractVersionTag(candidate.page_path);
+        const reasons = [];
+        let score = 0;
+        const lexicalRankScore = 0.35 * (1 - index / total);
+        score += lexicalRankScore;
+        const titleExact = primaryTitle.includes(plan.normalized_query) &&
+            plan.normalized_query.length > 0;
+        if (titleExact) {
+            score += 0.22;
+            reasons.push("exact title match");
+        }
+        const titleOverlap = this.keywordOverlap(plan.keywords, primaryTitle || title);
+        if (titleOverlap > 0) {
+            score += 0.14 * titleOverlap;
+            if (titleOverlap === 1 && !titleExact) {
+                reasons.push("all query keywords appear in title");
+            }
+        }
+        const headingOverlap = this.keywordOverlap(plan.keywords, heading);
+        if (headingOverlap > 0) {
+            score += 0.1 * headingOverlap;
+            if (headingOverlap >= 0.6) {
+                reasons.push("heading context aligns with the query");
+            }
+        }
+        const pathOverlap = this.keywordOverlap(plan.keywords, path);
+        if (pathOverlap > 0) {
+            score += 0.08 * pathOverlap;
+        }
+        const libraryOverlap = this.keywordOverlap(plan.keywords, libraryText);
+        if (libraryOverlap > 0) {
+            score += 0.08 * libraryOverlap;
+            if (libraryOverlap === 1) {
+                reasons.push("library name aligns with the query");
+            }
+        }
+        else if (plan.keywords.length === 1) {
+            score -= 0.03;
+        }
+        const phraseMatch = this.hasPhraseMatch(plan, title, heading, contentPreview);
+        if (phraseMatch) {
+            score += 0.08;
+            reasons.push("exact phrase match");
+        }
+        const pathPrior = this.pathTypeScore(plan.intent, pathType);
+        if (pathPrior > 0) {
+            score += pathPrior;
+            if (pathPrior >= 0.1) {
+                reasons.push(`matched ${pathType.replace(/_/g, "-")} page type`);
+            }
+        }
+        if (candidate.has_code_block) {
+            const codeSignal = plan.intent === "api_lookup" || plan.intent === "troubleshooting"
+                ? 0.07
+                : plan.intent === "getting_started"
+                    ? 0.03
+                    : 0.015;
+            score += codeSignal;
+            if (codeSignal >= 0.03) {
+                reasons.push("includes code sample");
+            }
+        }
+        if (candidate.token_count >= 60 && candidate.token_count <= 260) {
+            score += 0.03;
+        }
+        if (plan.requested_version) {
+            if (versionTag === plan.requested_version) {
+                score += 0.12;
+                reasons.push(`matches requested version v${versionTag}`);
+            }
+        }
+        else if (!versionTag) {
+            score += 0.08;
+            reasons.push("canonical unversioned page");
+        }
+        else {
+            score += Math.min(parseInt(versionTag, 10), 20) / 300;
+        }
+        const uniqueReasons = Array.from(new Set(reasons)).slice(0, 4);
+        return {
+            ...candidate,
+            path_type: pathType,
+            version_tag: versionTag,
+            rerank_score: Number(score.toFixed(6)),
+            reasons: uniqueReasons,
+        };
+    }
+    collapseDuplicates(plan, candidates) {
+        const bestByPage = new Map();
+        for (const candidate of candidates) {
+            const existing = bestByPage.get(candidate.page_url);
+            if (!existing || candidate.rerank_score > existing.rerank_score) {
+                bestByPage.set(candidate.page_url, candidate);
+            }
+        }
+        const bestByCanonicalPage = new Map();
+        const pageResults = Array.from(bestByPage.values()).sort((left, right) => right.rerank_score - left.rerank_score);
+        for (const candidate of pageResults) {
+            const canonicalKey = this.canonicalPageKey(candidate);
+            const existing = bestByCanonicalPage.get(canonicalKey);
+            if (!existing ||
+                this.preferenceScore(plan, candidate) >
+                    this.preferenceScore(plan, existing)) {
+                bestByCanonicalPage.set(canonicalKey, candidate);
+            }
+        }
+        return Array.from(bestByCanonicalPage.values()).sort((left, right) => {
+            if (right.rerank_score !== left.rerank_score) {
+                return right.rerank_score - left.rerank_score;
+            }
+            return left.lexical_score - right.lexical_score;
+        });
+    }
+    preferenceScore(plan, result) {
+        let score = result.rerank_score;
+        if (plan.requested_version) {
+            if (result.version_tag === plan.requested_version) {
+                score += 0.2;
+            }
+        }
+        else if (!result.version_tag) {
+            score += 0.12;
+        }
+        else {
+            score += parseInt(result.version_tag, 10) / 500;
+        }
+        if (plan.intent === "overview" || plan.intent === "getting_started") {
+            if (result.path_type === "getting_started" ||
+                result.path_type === "overview") {
+                score += 0.02;
+            }
+        }
+        return score;
+    }
+    canonicalPageKey(result) {
+        const canonicalPath = result.page_path
+            .toLowerCase()
+            .replace(/\/v\d+(?=\/|$)/g, "")
+            .replace(/\/+/g, "/")
+            .replace(/\/$/, "") || "/";
+        const canonicalTitle = normalizeSearchText(result.page_title)
+            .replace(/\bv\d+\b/g, "")
+            .trim();
+        return `${result.library_name}:${canonicalPath}:${canonicalTitle}`;
+    }
+    inferPathType(pagePath, pageTitle) {
+        const value = `${pagePath} ${pageTitle}`.toLowerCase();
+        if (/getting-started|quickstart|install|installation|setup/.test(value)) {
+            return "getting_started";
+        }
+        if (/overview|introduction|what is|basics/.test(value)) {
+            return "overview";
+        }
+        if (/\/api|\bapi\b|reference|\/apis\//.test(value)) {
+            return "api";
+        }
+        if (/troubleshoot|troubleshooting|errors?|faq|debug/.test(value)) {
+            return "troubleshooting";
+        }
+        return "guide";
+    }
+    pathTypeScore(intent, pathType) {
+        switch (intent) {
+            case "overview":
+                if (pathType === "overview")
+                    return 0.16;
+                if (pathType === "getting_started")
+                    return 0.1;
+                if (pathType === "guide")
+                    return 0.06;
+                return 0;
+            case "getting_started":
+                if (pathType === "getting_started")
+                    return 0.18;
+                if (pathType === "guide")
+                    return 0.08;
+                if (pathType === "overview")
+                    return 0.06;
+                return 0;
+            case "api_lookup":
+                if (pathType === "api")
+                    return 0.18;
+                if (pathType === "guide")
+                    return 0.04;
+                return 0;
+            case "troubleshooting":
+                if (pathType === "troubleshooting")
+                    return 0.16;
+                if (pathType === "guide")
+                    return 0.07;
+                return 0;
+            default:
+                if (pathType === "getting_started")
+                    return 0.08;
+                if (pathType === "overview")
+                    return 0.07;
+                if (pathType === "api")
+                    return 0.05;
+                if (pathType === "guide")
+                    return 0.03;
+                return 0;
+        }
+    }
+    keywordOverlap(keywords, haystack) {
+        if (keywords.length === 0 || haystack.length === 0) {
+            return 0;
+        }
+        const matches = keywords.filter((keyword) => haystack.includes(keyword)).length;
+        return matches / keywords.length;
+    }
+    hasPhraseMatch(plan, title, heading, content) {
+        if (plan.normalized_query.includes(" ") &&
+            (title.includes(plan.normalized_query) ||
+                heading.includes(plan.normalized_query))) {
+            return true;
+        }
+        return plan.phrases.some((phrase) => title.includes(phrase) ||
+            heading.includes(phrase) ||
+            content.includes(phrase));
+    }
+    primaryTitle(title) {
+        return title.split(/\||—|-/)[0]?.trim() ?? title;
+    }
+    extractVersionTag(path) {
+        const match = path.toLowerCase().match(/\/v(\d+)(?=\/|$)/);
+        return match?.[1] ?? null;
     }
 }

package/dist/tools/search-docs.js

@@ -1,6 +1,7 @@
 // src/tools/search-docs.ts — Primary search tool (80% of usage)
 import * as v from 'valibot';
 import { tool } from 'tmcp/utils';
+import { formatSearchResults } from '../search/format-results.js';
 export function createSearchDocsTool(searchEngine) {
     return {
         definition: {
@@ -20,16 +21,7 @@ export function createSearchDocsTool(searchEngine) {
             if (results.length === 0) {
                 return tool.text(`No results found for "${query}".`);
             }
-            const formatted = results
-                .map((r, i) => {
-                let block = `### ${i + 1}. ${r.page_title} — ${r.library_display_name}\n`;
-                block += `**Source:** ${r.page_url}\n`;
-                block += `**Section:** ${r.heading_context}\n\n`;
-                block += r.content;
-                return block;
-            })
-                .join('\n\n---\n\n');
-            return tool.text(`## Results for "${query}"\n\n${formatted}`);
+            return tool.text(formatSearchResults(query, results));
         },
     };
 }

package/dist/types.d.ts

@@ -42,6 +42,7 @@ export interface ChunkRecord {
 export interface CrawlJob {
     id: string;
     library_id: string;
+    session_id?: string;
     status: 'queued' | 'running' | 'completed' | 'failed' | 'cancelled';
     pages_discovered: number;
     pages_crawled: number;

package/dist/version.d.ts

@@ -1 +1 @@
-export declare const VERSION = "0.1.17";
+export declare const VERSION = "0.1.21";

package/dist/version.js

@@ -1,2 +1,2 @@
 // This file is automatically updated by the version sync script.
-export const VERSION = '0.1.17';
+export const VERSION = '0.1.21';

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "docshark",
-  "version": "0.1.17",
+  "version": "0.1.21",
   "description": "🦈 Documentation MCP Server — scrape, index, and search any doc website",
   "type": "module",
   "main": "./dist/index.js",
@@ -56,7 +56,7 @@
   "dependencies": {
     "@mozilla/readability": "^0.6.0",
     "@tmcp/adapter-valibot": "^0.1.5",
-    "@tmcp/transport-http": "^0.8.4",
+    "@tmcp/transport-http": "^0.8.5",
     "@tmcp/transport-sse": "^0.5.3",
     "@tmcp/transport-stdio": "^0.4.1",
     "cac": "^7.0.0",
@@ -66,7 +66,7 @@
     "puppeteer-core": "^24.37.5",
     "robots-parser": "^3.0.1",
     "srvx": "^0.11.8",
-    "tmcp": "^1.19.2",
+    "tmcp": "^1.19.3",
     "turndown": "^7.2.2",
     "turndown-plugin-gfm": "^1.0.2",
     "valibot": "^1.2.0"