docshark 0.1.17 → 0.1.20

package/README.md

@@ -0,0 +1,176 @@
+# 🦈 DocShark
+
+[![Built with Bun](https://img.shields.io/badge/Bun-%23000000.svg?style=flat&logo=bun&logoColor=white)](https://bun.sh/)
+[![NPM Version](https://img.shields.io/npm/v/docshark.svg?style=flat&color=blue)](https://www.npmjs.com/package/docshark)
+[![MCP Compatible](https://img.shields.io/badge/MCP-Ready-0D1117.svg?style=flat&logo=github&logoColor=white)](https://modelcontextprotocol.io/)
+[![GitHub Release](https://img.shields.io/github/v/release/Michael-Obele/docshark?color=success)](https://github.com/Michael-Obele/docshark/releases)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+**DocShark** is a powerful MCP (Model Context Protocol) server designed to scrape, index, and search any documentation website. It creates a local, highly-searchable knowledge base from public documentation pages using FTS5 (Full-Text Search) and BM25 ranking, allowing AI assistants to query the latest docs effortlessly.
+
+---
+
+## 🚀 Features
+
+- **Automated Crawling**: Discovers pages via `sitemap.xml` with fallback to BFS link crawling.
+- **Smart Extraction**: Uses Readability and Turndown to extract main content and convert it to clean Markdown, filtering out navbars and sidebars.
+- **Semantic Chunking**: Splits content based on headings, preserving contextual headers for better AI understanding.
+- **High-Performance Search**: Built-in SQLite + FTS5 indexing with BM25 ranking for accurate and lightning-fast search results.
+- **JS-Rendered Site Support**: Tiered fetching strategy automatically detects React/Vue SPAs (empty shells) and upgrades to `puppeteer-core` if you have it installed (zero-config, auto-fallback).
+- **Polite Crawling**: Respects `robots.txt` and implements rate limiting to prevent overloading documentation servers.
+- **Standard MCP Tooling**: Connect perfectly with Desktop Claude, VS Code, Cursor, and any other MCP-compatible clients via standard `stdio` or `http`/`sse` transports.
+
+## 📦 What We Have Done (Phase 1)
+
+**Phase 1: Core Engine** is fully implemented and tested.
+
+- ✅ Custom SQLite Database with FTS5 virtual tables and auto-sync triggers.
+- ✅ Web scraping engine supporting standard `fetch()` and `puppeteer-core`.
+- ✅ Markdown processor utilizing Readability + Turndown.
+- ✅ Heading-based semantic chunker (500-1200 tokens per chunk).
+- ✅ Asynchronous job manager and queue system.
+- ✅ Complete HTTP API (REST endpoints + SSE event streams).
+- ✅ Seamless integration of 4 MCP tools: `manage_library`, `search_docs`, `list_libraries`, and `get_doc_page`.
+- ✅ Robust CLI interface (`start`, `add`, `rename`, `search`, `list`).
+
+## 🏗️ What We Are Doing
+
+We are actively polishing the integration between the core engine and external MCP clients (like VS Code Agents and Claude Desktop).
+
+## 🔮 What We Plan To Do (Phase 2 & Beyond)
+
+- **Web Dashboard**: An intuitive SvelteKit dashboard to manage your synced libraries, view crawl progress in real-time (via SSE), and test searches manually.
+- **Incremental Crawling**: Smarter `refresh` jobs that compare `ETag` and `Last-Modified` headers to only re-scrape updated pages.
+- **Vector Search (RAG)**: Integration of lightweight vector embeddings for semantic similarity search alongside the existing FTS5 keyword search.
+- **Advanced Scraping Setup**: Support for custom CSS selectors to define exactly where content lives in non-standard documentation websites.
+
+---
+
+## 🛠️ Usage
+
+### Quick Start (from npm)
+
+You can run DocShark directly without installing it globally using `bunx`:
+
+```bash
+# Add a documentation library to the index
+bunx docshark add https://valibot.dev/guides/ --depth 2
+
+# Search your indexed docs
+bunx docshark search "schema validation"
+```
+
+### Installation
+
+To install DocShark globally as a CLI tool:
+
+DocShark is intended to be installed and run with Bun.
+
+```bash
+# Global Bun installation
+bun add -g docshark
+```
+
+After installation, you can use the `docshark` command:
+
+```bash
+docshark list
+
+# Update the global Bun installation when a new release is published
+docshark update
+
+# Script-friendly update check
+docshark update --check --quiet
+```
+
+Interactive CLI runs will also let you know when a newer version is available. Update notices are intentionally skipped for MCP `stdio` mode so they never interfere with protocol output.
+
+For scripts, `docshark update --check` exits `0` when current, `10` when a newer version is available, and `1` when the version check could not be completed.
+
+## 🔌 MCP Integration
+
+### VS Code (GitHub Copilot / MCP Extension)
+
+Add DocShark to your `.vscode/settings.json` or global MCP configuration:
+
+```json
+{
+  "mcpServers": {
+    "docshark": {
+      "command": "bunx",
+      "args": ["-y", "docshark", "start", "--stdio"]
+    }
+  }
+}
+```
+
+### Cursor
+
+1. Open **Cursor Settings** > **Models** > **MCP**.
+2. Click **+ Add New MCP Server**.
+3. Name: `docshark`
+4. Type: `command`
+5. Command: `bunx -y docshark start --stdio`
+
+### Claude Desktop
+
+Edit your Claude Desktop configuration file:
+
+- **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`
+- **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`
+
+```json
+{
+  "mcpServers": {
+    "docshark": {
+      "command": "bunx",
+      "args": ["-y", "docshark", "start", "--stdio"]
+    }
+  }
+}
+```
+
+---
+
+## 🛠️ Development
+
+### Local Setup
+
+Ensure you have [Bun](https://bun.sh/) installed.
+
+```bash
+# Clone the repository
+git clone https://github.com/Michael-Obele/docshark.git
+cd docshark
+
+# Install dependencies
+bun install
+
+# (Optional) Enable auto-detection & scraping of Javascript React/Vue single-page apps
+bun add puppeteer-core
+
+# Start the DocShark MCP server in HTTP mode for local testing
+bun run src/cli.ts start --port 6380
+```
+
+### Local CLI Debugging
+
+```bash
+# Run CLI directly while developing
+bun run src/cli.ts list
+```
+
+## 🔄 Versioning & Changelog
+
+This project uses [Google's Release Please](https://github.com/googleapis/release-please) to automate versioning and changelog generation.
+
+- **Semantic Versioning**: Our versions automatically bump (e.g. `0.0.1` -> `0.0.2` or `0.1.0`) based on standard Conventional Commits (`feat:`, `fix:`, `chore:`, etc.).
+- **Automated**: A PR is automatically created on `master` when standard commits are merged, generating a standard `CHANGELOG.md`.
+
+## 📜 License
+
+This project is open-source and available under the [MIT License](LICENSE).
+
+---
+
+_Built to empower AI agents with the latest knowledge._

package/dist/scripts/sync-version.js

@@ -4,20 +4,32 @@ import { fileURLToPath } from "node:url";
 const scriptDir = dirname(fileURLToPath(import.meta.url));
 const packageJsonPath = resolve(scriptDir, "../../package.json");
 const versionFilePath = resolve(scriptDir, "../version.ts");
+const releaseVersion = process.env.DOCSHARK_VERSION?.trim();
 const packageJson = JSON.parse(readFileSync(packageJsonPath, "utf8"));
-if (typeof packageJson.version !== "string") {
+const targetVersion = releaseVersion && releaseVersion.length > 0
+    ? releaseVersion
+    : packageJson.version;
+if (typeof targetVersion !== "string") {
     throw new Error("packages/core/package.json is missing a valid version.");
 }
 const nextVersionFile = [
     "// This file is automatically updated by the version sync script.",
-    `export const VERSION = '${packageJson.version}';`,
+    `export const VERSION = '${targetVersion}';`,
     "",
 ].join("\n");
 const currentVersionFile = readFileSync(versionFilePath, "utf8");
 if (currentVersionFile !== nextVersionFile) {
     writeFileSync(versionFilePath, nextVersionFile, "utf8");
-    console.log(`Updated src/version.ts to ${packageJson.version}`);
+    console.log(`Updated src/version.ts to ${targetVersion}`);
 }
 else {
-    console.log(`src/version.ts already matches ${packageJson.version}`);
+    console.log(`src/version.ts already matches ${targetVersion}`);
+}
+if (releaseVersion && packageJson.version !== targetVersion) {
+    const nextPackageJson = {
+        ...packageJson,
+        version: targetVersion,
+    };
+    writeFileSync(packageJsonPath, `${JSON.stringify(nextPackageJson, null, 2)}\n`, "utf8");
+    console.log(`Updated package.json to ${targetVersion}`);
 }

package/dist/version.d.ts

@@ -1 +1 @@
-export declare const VERSION = "0.1.17";
+export declare const VERSION = "0.1.20";

package/dist/version.js

@@ -1,2 +1,2 @@
 // This file is automatically updated by the version sync script.
-export const VERSION = '0.1.17';
+export const VERSION = '0.1.20';

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "docshark",
-  "version": "0.1.17",
+  "version": "0.1.20",
   "description": "🦈 Documentation MCP Server — scrape, index, and search any doc website",
   "type": "module",
   "main": "./dist/index.js",