zigsm 1.4.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Andrey Ryapov, "Zig and WebAssembly" contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,180 @@
+# ZSM
+
+ZSM - fork of **Zig Docs MCP**
+
+> [!NOTE]
+> This fork keeps the original behavior and documentation flow nearly unchanged. The main addition vs upstream is addition of embeddings for better semantic retrieval in search tools.
+
+Model Context Protocol (MCP) server that provides up-to-date documentation for the Zig programming language standard library and builtin functions.
+
+It uses the same approach as Zig's official autodoc (ziglang.org) by reading STD lib source files directly through a WASM module. However instead of returning HTML, it outputs Markdown which significantly reduces token usage.
+
+By default, the server uses your locally installed Zig compiler to serve documentation, ensuring you always get docs that match your actual Zig version. It can also fetch documentation from ziglang.org if needed.
+
+> [!TIP]
+> Add `use zigdocs` to your prompt if you want to explicitly instruct the LLM to use Zig docs tools. Otherwise, LLM will automatically decide when to utilize MCP tools based on the context of your questions.
+
+<p align="center" width="100%">
+  <img src="https://raw.githubusercontent.com/zig-wasm/.github/refs/heads/main/static/readme_mcp_1.gif" width="49%" />
+  <img src="https://raw.githubusercontent.com/zig-wasm/.github/refs/heads/main/static/readme_mcp_2.gif" width="49%" />
+</p>
+
+## Tools
+
+- **`list_builtin_functions`** - Lists all available Zig builtin functions. Builtin functions are provided by the compiler and are prefixed with '@'. The comptime keyword on a parameter means that the parameter must be known at compile time. Use this to discover what functions are available, then use 'get_builtin_function' to get detailed documentation.
+- **`get_builtin_function`** - Search for Zig builtin functions by name and get their documentation, signatures, and usage information. Returns all matching functions ranked by relevance.
+- **`search_std_lib`** - Search the Zig standard library for declarations by name. Returns a list of matching items with their fully qualified names. Use this to discover available types, functions, and constants in the standard library.
+- **`get_std_lib_item`** - Get detailed documentation for a specific standard library item by its fully qualified name (e.g., "std.ArrayList.init"). Returns comprehensive documentation including function signatures, parameters, errors, examples, and source code. Set `get_source_file: true` to retrieve the entire source file where the item is implemented.
+
+When `VOYAGE_API_KEY` is set, search tools automatically use hybrid ranking (lexical + embeddings) and fall back to lexical-only results if embeddings are unavailable.
+
+## Commands
+
+The CLI provides flexible options for version control and update management:
+
+```bash
+# Start MCP server
+zsm --doc-source local
+
+# Use specific Zig version from ziglang.org instead of local Zig
+zsm --doc-source remote --version 0.14.1
+
+# Enable automatic daily updates
+zsm --doc-source remote --update-policy daily
+
+# Update documentation without starting MCP server (only for remote)
+zsm update --version 0.15.1
+
+# Start local web server to view documentation
+zsm view --version 0.15.1
+```
+
+**Version options `--version`**:
+- `master` (default) - Latest development version from Zig's master branch
+- `0.14.1`, `0.14.0`, etc. - Specific Zig release versions
+
+**Update policies `--update-policy`**:
+- `manual` (default) - No automatic updates, manual control only
+- `daily` - Check for documentation updates once per day
+- `startup` - Update documentation every time the server starts
+
+**Documentation sources `--doc-source`**:
+- `local` (default) - Use your locally installed Zig compiler's documentation server (`zig std`)
+- `remote` - Download documentation from ziglang.org
+
+## Documentation Sources
+
+### Local Mode (Default)
+
+The server automatically uses your local Zig installation to serve documentation via `zig std`. This ensures:
+- Documentation always matches your installed Zig version
+- No network requests needed for standard library docs
+- Faster response times
+
+### Remote Mode
+
+When using `--doc-source remote`, documentation is fetched from ziglang.org and cached in platform-specific directories:
+- Linux: `~/.cache/zigsm/`
+- macOS: `~/Library/Caches/zigsm/`
+- Windows: `%LOCALAPPDATA%\zigsm\`
+
+## Embeddings (Voyage)
+
+Optional environment variables:
+- `VOYAGE_API_KEY` - Enables embeddings for doc search when set.
+- `VOYAGE_MODEL` - Embedding model override (default: `voyage-3-lite`).
+
+Embeddings are cached per Zig version, doc source, and model in the same cache directory used for docs.
+
+## Installation
+
+The installation examples below use the local documentation source by default. In local mode, docs are served by your installed Zig via `zig std`, requiring no network and matching your actual Zig version. This is the recommended setup for most users. For downloading docs from ziglang.org instead, see Remote Documentation (Optional) below.
+
+### Claude Code
+Using npx (Node.js)
+```bash
+claude mcp add zsm -- npx -y zigsm@latest
+```
+
+Using bunx (Bun)
+```bash
+claude mcp add zsm -- bunx zigsm@latest
+```
+
+### Roo Code
+
+1. Click the **MCP** button in Roo Code
+2. Select **"Edit Global MCP"** or **"Edit Project MCP"**
+3. Add the configuration from the JSON template below
+
+### Augment Code
+
+Navigate to **Settings → MCP Servers → Add Server** and use the JSON template below.
+
+### Claude Desktop
+
+Add the JSON configuration below to your MCP settings file.
+
+### JSON Configuration Template
+
+**Node.js:**
+```json
+{
+  "mcpServers": {
+    "zsm": {
+      "command": "npx",
+      "args": ["-y", "zigsm@latest"]
+    }
+  }
+}
+```
+
+**Bun:**
+```json
+{
+  "mcpServers": {
+    "zsm": {
+      "command": "bunx",
+      "args": ["zigsm@latest"]
+    }
+  }
+}
+```
+
+### Remote Documentation (Optional)
+
+If you prefer downloading documentation from ziglang.org instead of using your local Zig, enable remote mode explicitly and choose a version:
+
+Using npx (Node.js)
+```bash
+claude mcp add zsm -- npx -y zigsm@latest --doc-source remote --version master
+```
+
+Using bunx (Bun)
+```bash
+claude mcp add zsm -- bunx zigsm@latest --doc-source remote --version 0.14.1
+```
+
+**Node.js (remote):**
+```json
+{
+  "mcpServers": {
+    "zsm": {
+      "command": "npx",
+      "args": ["-y", "zigsm@latest", "--doc-source", "remote", "--version", "master"]
+    }
+  }
+}
+```
+
+**Bun (remote):**
+```json
+{
+  "mcpServers": {
+    "zsm": {
+      "command": "bunx",
+      "args": ["zigsm@latest", "--doc-source", "remote", "--version", "0.14.1"]
+    }
+  }
+}
+```

package/dist/docs.js

@@ -0,0 +1,176 @@
+import * as fs from "node:fs";
+import * as http from "node:http";
+import * as path from "node:path";
+import { fileURLToPath } from "node:url";
+import envPaths from "env-paths";
+import extractBuiltinFunctions from "./extract-builtin-functions.js";
+import { getLocalStdSources, getZigVersion } from "./local-std-server.js";
+export async function ensureDocs(zigVersion, updatePolicy = "manual", isMcpMode = true, docSource = "local") {
+    const paths = envPaths("zigsm", { suffix: "" });
+    const metadataPath = path.join(paths.cache, zigVersion, "metadata.json");
+    let shouldUpdate = false;
+    if (updatePolicy === "startup") {
+        shouldUpdate = true;
+    }
+    else if (updatePolicy === "daily") {
+        if (!fs.existsSync(metadataPath)) {
+            shouldUpdate = true;
+        }
+        else {
+            try {
+                const content = fs.readFileSync(metadataPath, "utf8");
+                const metadata = JSON.parse(content);
+                const dayInMs = 24 * 60 * 60 * 1000;
+                shouldUpdate = Date.now() - metadata.lastUpdate >= dayInMs;
+            }
+            catch {
+                shouldUpdate = true;
+            }
+        }
+    }
+    if (shouldUpdate) {
+        try {
+            if (!isMcpMode)
+                console.log(`Updating documentation for Zig version: ${zigVersion}`);
+            const builtinFunctions = await extractBuiltinFunctions(zigVersion, isMcpMode, true);
+            await downloadSourcesTar(zigVersion, isMcpMode, true, docSource);
+            const dir = path.dirname(metadataPath);
+            if (!fs.existsSync(dir)) {
+                fs.mkdirSync(dir, { recursive: true });
+            }
+            fs.writeFileSync(metadataPath, JSON.stringify({
+                lastUpdate: Date.now(),
+                version: zigVersion,
+            }, null, 2));
+            if (!isMcpMode)
+                console.log(`Successfully updated documentation for Zig version: ${zigVersion}`);
+            return builtinFunctions;
+        }
+        catch (error) {
+            if (error instanceof Error && error.message.includes("404")) {
+                console.error(`Error: Zig version '${zigVersion}' not found on ziglang.org. Please check the version number.`);
+            }
+            else {
+                console.error(`Error updating documentation for version ${zigVersion}:`, error);
+            }
+            throw error;
+        }
+    }
+    return await extractBuiltinFunctions(zigVersion, isMcpMode);
+}
+export async function downloadSourcesTar(zigVersion, isMcpMode = false, forceUpdate = false, docSource = "local") {
+    if (docSource === "local") {
+        try {
+            if (!isMcpMode)
+                console.log("Using local Zig std server for documentation");
+            const localVersion = getZigVersion();
+            if (!isMcpMode)
+                console.log(`Local Zig version: ${localVersion}`);
+            return await getLocalStdSources();
+        }
+        catch (error) {
+            if (!isMcpMode) {
+                console.log(`Failed to use local Zig std server: ${error}`);
+                console.log("Falling back to remote documentation");
+            }
+        }
+    }
+    const paths = envPaths("zigsm", { suffix: "" });
+    const versionCacheDir = path.join(paths.cache, zigVersion);
+    const sourcesPath = path.join(versionCacheDir, "sources.tar");
+    if (fs.existsSync(sourcesPath) && !forceUpdate) {
+        if (!isMcpMode)
+            console.log(`Using cached sources.tar from ${sourcesPath}`);
+        return new Uint8Array(fs.readFileSync(sourcesPath));
+    }
+    const url = `https://ziglang.org/documentation/${zigVersion}/std/sources.tar`;
+    if (!isMcpMode)
+        console.log(`Downloading sources.tar from: ${url}`);
+    const response = await fetch(url);
+    if (!response.ok) {
+        throw new Error(`Failed to download sources.tar from ${url}: ${response.status} ${response.statusText}`);
+    }
+    const buffer = await response.arrayBuffer();
+    const uint8Array = new Uint8Array(buffer);
+    if (!fs.existsSync(versionCacheDir)) {
+        fs.mkdirSync(versionCacheDir, { recursive: true });
+    }
+    fs.writeFileSync(sourcesPath, uint8Array);
+    if (!isMcpMode)
+        console.log(`Downloaded sources.tar to ${sourcesPath}`);
+    return uint8Array;
+}
+async function downloadSourcesTarPath(zigVersion) {
+    const paths = envPaths("zigsm", { suffix: "" });
+    const versionCacheDir = path.join(paths.cache, zigVersion);
+    const sourcesPath = path.join(versionCacheDir, "sources.tar");
+    if (fs.existsSync(sourcesPath)) {
+        console.log(`Using cached sources.tar from ${sourcesPath}`);
+        return sourcesPath;
+    }
+    await downloadSourcesTar(zigVersion, false);
+    return sourcesPath;
+}
+export async function startViewServer(zigVersion) {
+    try {
+        const sourcesPath = await downloadSourcesTarPath(zigVersion);
+        const currentDir = path.dirname(fileURLToPath(import.meta.url));
+        const wasmPath = path.join(currentDir, "main.wasm");
+        const indexPath = path.join(currentDir, "index.html");
+        const stdJsPath = path.join(currentDir, "std.js");
+        const port = 8080;
+        const server = http.createServer((req, res) => {
+            let filePath;
+            const url = req.url || "/";
+            if (url === "/" || url === "/index.html") {
+                filePath = indexPath;
+            }
+            else if (url === "/std.js") {
+                filePath = stdJsPath;
+            }
+            else if (url === "/main.wasm") {
+                filePath = wasmPath;
+            }
+            else if (url === "/sources.tar") {
+                filePath = sourcesPath;
+            }
+            else {
+                res.writeHead(404);
+                res.end("File not found");
+                return;
+            }
+            if (!fs.existsSync(filePath)) {
+                res.writeHead(404);
+                res.end("File not found");
+                return;
+            }
+            const ext = path.extname(filePath).toLowerCase();
+            const contentTypes = {
+                ".html": "text/html",
+                ".js": "text/javascript",
+                ".css": "text/css",
+                ".wasm": "application/wasm",
+                ".tar": "application/x-tar",
+            };
+            const contentType = contentTypes[ext] || "application/octet-stream";
+            res.writeHead(200, { "Content-Type": contentType });
+            fs.createReadStream(filePath).pipe(res);
+        });
+        server.listen(port, () => {
+            const url = `http://localhost:${port}`;
+            console.log(`Server started at ${url}`);
+            console.log(`Serving Zig ${zigVersion} documentation`);
+            console.log("Press Ctrl+C to stop the server");
+        });
+        process.on("SIGINT", () => {
+            console.log("\nShutting down server...");
+            server.close(() => {
+                process.exit(0);
+            });
+        });
+    }
+    catch (error) {
+        console.error("Error starting view server:", error);
+        throw error;
+    }
+}

package/dist/extract-builtin-functions.js

@@ -0,0 +1,155 @@
+import * as fs from "node:fs";
+import * as path from "node:path";
+import * as cheerio from "cheerio";
+import envPaths from "env-paths";
+async function extractBuiltinFunctions(zigVersion, isMcpMode = true, forceUpdate = false) {
+    const paths = envPaths("zigsm", { suffix: "" });
+    const versionCacheDir = path.join(paths.cache, zigVersion);
+    const outputPath = path.join(versionCacheDir, "builtin-functions.json");
+    if (fs.existsSync(outputPath) && !forceUpdate) {
+        if (!isMcpMode)
+            console.log(`Using cached builtin functions from ${outputPath}`);
+        try {
+            const content = fs.readFileSync(outputPath, "utf8");
+            return JSON.parse(content);
+        }
+        catch (error) {
+            console.error(`Error reading cached file, re-extracting:`, error);
+        }
+    }
+    const url = `https://ziglang.org/documentation/${zigVersion}/`;
+    if (!isMcpMode)
+        console.log(`Downloading HTML from: ${url}`);
+    const response = await fetch(url);
+    if (!response.ok) {
+        throw new Error(`Failed to download HTML from ${url}: ${response.status} ${response.statusText}`);
+    }
+    const html = await response.text();
+    const $ = cheerio.load(html);
+    const builtins = [];
+    // Find "Builtin Functions" section
+    const builtinFunctionsSection = $('h2[id="Builtin-Functions"]');
+    if (builtinFunctionsSection.length === 0) {
+        throw new Error("Could not find Builtin Functions section in HTML");
+    }
+    let current = builtinFunctionsSection.next();
+    while (current.length && current.prop("tagName")?.toLowerCase() !== "h2") {
+        if (current.is("h3[id]")) {
+            const h3 = current;
+            const func = h3.find("a").first().text();
+            if (func.startsWith("@")) {
+                const signature = h3.next("pre").text().trim();
+                const descriptionParts = [];
+                const seeAlsoLinks = [];
+                let descCurrent = h3.next("pre").next();
+                while (descCurrent.length &&
+                    !["h2", "h3"].includes(descCurrent.prop("tagName")?.toLowerCase() || "")) {
+                    if (descCurrent.is("p")) {
+                        const pHtml = descCurrent.html() || "";
+                        const $p = cheerio.load(pHtml, {
+                            xml: { xmlMode: false, decodeEntities: false },
+                        });
+                        $p("a").each((_, a) => {
+                            const link = $p(a);
+                            const href = link.attr("href") || "";
+                            const text = link.text();
+                            let markdownLink;
+                            if (href.startsWith("#")) {
+                                markdownLink = `[${text}](https://ziglang.org/documentation/${zigVersion}/${href})`;
+                            }
+                            else {
+                                markdownLink = `[${text}](${href})`;
+                            }
+                            link.replaceWith(markdownLink);
+                        });
+                        $p("code").each((_, code) => {
+                            const el = $p(code);
+                            el.replaceWith(`\`${el.text()}\``);
+                        });
+                        const pText = $p.root().text();
+                        descriptionParts.push(pText.replace(/\s+/g, " ").trim());
+                    }
+                    else if (descCurrent.is("ul")) {
+                        // Convert each <li> to Markdown, handling <a> and <code> tags
+                        descCurrent.children("li").each((_, li) => {
+                            const liHtml = $(li).html() || "";
+                            const $li = cheerio.load(liHtml, {
+                                xml: { xmlMode: false, decodeEntities: false },
+                            });
+                            $li("a").each((_, a) => {
+                                const link = $li(a);
+                                const href = link.attr("href") || "";
+                                const text = link.text();
+                                let markdownLink;
+                                if (href.startsWith("#")) {
+                                    markdownLink = `[${text}](https://ziglang.org/documentation/${zigVersion}/${href})`;
+                                }
+                                else {
+                                    markdownLink = `[${text}](${href})`;
+                                }
+                                link.replaceWith(markdownLink);
+                            });
+                            $li("code").each((_, code) => {
+                                const el = $li(code);
+                                el.replaceWith(`\`${el.text()}\``);
+                            });
+                            const liText = $li.root().text().replace(/\s+/g, " ").trim();
+                            if (liText.length > 0) {
+                                descriptionParts.push(`* ${liText}`);
+                            }
+                        });
+                    }
+                    else if (descCurrent.is("figure")) {
+                        // Extract <figcaption> and <pre> content
+                        const figcaption = descCurrent.find("figcaption").first().text().trim();
+                        const pre = descCurrent.find("pre").first();
+                        const code = pre.text();
+                        let lang = "";
+                        let label = "";
+                        if (figcaption) {
+                            label = `**${figcaption}**\n`;
+                            if (figcaption.endsWith(".zig")) {
+                                lang = "zig";
+                            }
+                            else if (figcaption.toLowerCase().includes("shell")) {
+                                lang = "sh";
+                            }
+                        }
+                        if (code) {
+                            // Format as Markdown code block
+                            const codeBlock = `${label}\n\`\`\`${lang}\n${code.trim()}\n\`\`\``;
+                            descriptionParts.push(codeBlock.trim());
+                        }
+                    }
+                    descCurrent = descCurrent.next();
+                }
+                // Join doc blocks with a single newline and collapse multiple newlines
+                let docs = descriptionParts.join("\n");
+                docs = docs.replace(/\n{2,}/g, "\n").replace(/\n+$/g, "");
+                if (docs.toLowerCase().endsWith("see also:")) {
+                    docs = docs.slice(0, -"see also:".length).trim();
+                }
+                if (seeAlsoLinks.length > 0) {
+                    if (docs.length > 0) {
+                        docs += "\n";
+                    }
+                    docs += `See also:\n* ${seeAlsoLinks.join("\n* ")}`;
+                }
+                builtins.push({
+                    func,
+                    signature,
+                    docs,
+                });
+            }
+        }
+        current = current.next();
+    }
+    if (!fs.existsSync(versionCacheDir)) {
+        fs.mkdirSync(versionCacheDir, { recursive: true });
+    }
+    fs.writeFileSync(outputPath, JSON.stringify(builtins, null, 2));
+    if (!isMcpMode)
+        console.log(`Extracted ${builtins.length} builtin functions to ${outputPath}`);
+    return builtins;
+}
+export default extractBuiltinFunctions;

package/dist/index.html

@@ -0,0 +1,47 @@
+<!doctype html>
+<html>
+  <head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Zig Documentation</title>
+    <style>
+      body {
+        font-family: monospace;
+        font-size: 14px;
+        margin: 20px;
+      }
+      #search {
+        width: 100%;
+        padding: 10px;
+        margin-bottom: 20px;
+        border: 1px solid #555;
+        font-family: monospace;
+      }
+      #content {
+        white-space: pre-wrap;
+        line-height: 1.4;
+      }
+      .hidden {
+        display: none;
+      }
+      #errors {
+        background: #400;
+        padding: 10px;
+        margin: 10px 0;
+        border: 1px solid #600;
+      }
+    </style>
+  </head>
+  <body>
+    <input type="search" id="search" autocomplete="off" spellcheck="false" placeholder="Search documentation...">
+    <div id="content"></div>
+    <div id="errors" class="hidden">
+      <h1>Errors</h1>
+      <pre id="errorsText"></pre>
+    </div>
+    <script type="module">
+      import { startDocsViewer } from './std.js';
+      startDocsViewer();
+    </script>
+  </body>
+</html>