mcp-rustdoc 2.0.0 → 3.1.0

package/README.md

@@ -1,6 +1,6 @@
 # mcp-rustdoc
 
-An MCP server that gives AI assistants deep access to the Rust ecosystem. It scrapes docs.rs with surgical DOM extraction (cheerio) and queries the crates.io API, exposing six tools that cover everything from high-level crate overviews to individual method signatures, feature gates, trait impls, and code examples.
+An MCP server that gives AI assistants deep access to the Rust ecosystem. It scrapes docs.rs (and `doc.rust-lang.org` for `std`/`core`/`alloc`) with surgical DOM extraction (cheerio) and queries the crates.io API, exposing nine tools that cover everything from high-level crate overviews to individual method signatures, feature gates, trait impls, and code examples. Responses are cached in memory (5-minute TTL) to avoid redundant fetches.
 
 ## Tools
 
@@ -12,9 +12,24 @@ An MCP server that gives AI assistants deep access to the Rust ecosystem. It scr
 | `get_crate_items` | Items in a module with types, feature gates, and descriptions |
 | `lookup_crate_item` | Item detail: signature, docs, methods, variants, optionally trait impls + examples |
 | `search_crate` | Ranked symbol search (exact > prefix > substring) with canonical paths |
+| `search_crates` | Search crates.io by keyword — returns name, description, downloads, version |
+| `get_crate_versions` | All published versions with dates and yanked status (crates.io API) |
+| `get_source_code` | Raw source code of a file from docs.rs or doc.rust-lang.org |
 
 Every tool accepts an optional `version` parameter to pin a specific crate version instead of `latest`.
 
+### Standard library support
+
+All documentation tools work with `std`, `core`, and `alloc` — the Rust standard library crates hosted at `doc.rust-lang.org`. Use them exactly like any other crate:
+
+```
+> lookup_crate_docs({ crateName: "std" })
+> get_crate_items({ crateName: "std", modulePath: "collections" })
+> search_crate({ crateName: "core", query: "Option" })
+```
+
+`get_crate_metadata` returns a helpful message for std crates since they aren't published on crates.io.
+
 ## Install
 
 No clone needed. Just configure your AI coding assistant with `npx`:
@@ -335,6 +350,75 @@ Searches all items in a crate by name. Results are ranked: exact match on the ba
 
 ---
 
+### `search_crates`
+
+Search for Rust crates on crates.io by keyword.
+
+| Parameter | Type | Required | Description |
+|---|---|---|---|
+| `query` | string | yes | Search keywords |
+| `page` | number | no | Page number (default 1) |
+| `perPage` | number | no | Results per page (default 10, max 50) |
+
+```
+> search_crates({ query: "http" })
+
+# Crate search: "http" — 1234 results (page 1)
+
+  http v1.2.0 (50,000,000 downloads) — A set of types for representing HTTP requests and responses.
+  hyper v1.5.2 (120,000,000 downloads) — A fast and correct HTTP library.
+  reqwest v0.12.12 (100,000,000 downloads) — higher level HTTP client library
+  ...
+```
+
+---
+
+### `get_crate_versions`
+
+List all published versions of a crate from crates.io.
+
+| Parameter | Type | Required | Description |
+|---|---|---|---|
+| `crateName` | string | yes | Crate name |
+
+```
+> get_crate_versions({ crateName: "serde" })
+
+# serde — 312 versions
+
+  1.0.219  2025-02-01
+  1.0.218  2025-01-12
+  1.0.217  2024-12-23
+  ...
+  0.1.0  2014-12-09
+```
+
+---
+
+### `get_source_code`
+
+Fetch the raw source code of a file from docs.rs (or doc.rust-lang.org for std crates).
+
+| Parameter | Type | Required | Description |
+|---|---|---|---|
+| `crateName` | string | yes | Crate name |
+| `path` | string | yes | Source path relative to crate root (e.g. `"src/lib.rs"`, `"src/sync/mutex.rs"`) |
+| `version` | string | no | Pinned version |
+
+```
+> get_source_code({ crateName: "tokio", path: "src/sync/mutex.rs" })
+
+# Source: tokio/src/sync/mutex.rs
+https://docs.rs/tokio/latest/src/tokio/sync/mutex.rs
+
+\`\`\`rust
+use crate::sync::batch_semaphore as semaphore;
+...
+\`\`\`
+```
+
+---
+
 ## Recommended workflows
 
 ### Exploring a new crate
@@ -362,6 +446,7 @@ Searches all items in a crate by name. Results are ranked: exact match on the ba
 src/
   index.ts              Entry point — registers tools + prompt, starts stdio server
   lib.ts                Shared: URL builders, HTTP/DOM helpers, crates.io API, extractors
+  cache.ts              In-memory TTL cache (5-minute default)
   types/
     html-to-text.d.ts   Type declarations for html-to-text
   tools/
@@ -370,6 +455,9 @@ src/
     get-items.ts        get_crate_items
     lookup-item.ts      lookup_crate_item
     search.ts           search_crate
+    search-crates.ts    search_crates
+    crate-versions.ts   get_crate_versions
+    source-code.ts      get_source_code
     crate-metadata.ts   get_crate_metadata
     crate-brief.ts      get_crate_brief
 ```
@@ -377,7 +465,8 @@ src/
 ### Data sources
 
 - **docs.rs** — HTML pages parsed with cheerio for surgical DOM extraction (only the elements needed, not full-page conversion)
-- **crates.io API** — JSON endpoints for metadata, features, and dependencies
+- **doc.rust-lang.org** — Same rustdoc HTML format, used for `std`, `core`, and `alloc`
+- **crates.io API** — JSON endpoints for metadata, features, dependencies, and search
 
 ### Design decisions
 
@@ -385,6 +474,7 @@ src/
 - **Ranked search** — `all.html` contains every public item; scoring by exact/prefix/substring gives better results than flat substring matching
 - **Version parameter everywhere** — Agents working on projects with pinned dependencies need to read docs for specific versions
 - **Optional sections** — `includeImpls` and `includeExamples` default to off so the base response stays compact; agents opt in when they need more detail
+- **In-memory cache** — All HTTP responses are cached for 5 minutes, avoiding redundant fetches when agents issue multiple related tool calls
 
 ## License
 

package/dist/index.js

@@ -5,9 +5,23 @@ import { z } from "zod";
 import axios from "axios";
 import { load } from "cheerio";
 import { convert } from "html-to-text";
+const DEFAULT_TTL = 5 * 60 * 1e3;
+const store = /* @__PURE__ */ new Map();
+function cacheGet(key) {
+  const entry = store.get(key);
+  if (!entry) return null;
+  if (Date.now() > entry.expiry) {
+    store.delete(key);
+    return null;
+  }
+  return entry.data;
+}
+function cacheSet(key, data, ttl = DEFAULT_TTL) {
+  store.set(key, { data, expiry: Date.now() + ttl });
+}
 const DOCS_BASE = "https://docs.rs";
 const CRATES_IO = "https://crates.io/api/v1";
-const USER_AGENT = "mcp-rust-docs/2.0.0";
+const USER_AGENT = "mcp-rust-docs/3.1.0";
 const MAX_DOC_LENGTH = 6e3;
 const MAX_SEARCH_RESULTS = 100;
 const SECTION_TO_TYPE = {
@@ -38,10 +52,18 @@ const TYPE_FILE_PREFIX = {
   attr: "attr.",
   derive: "derive."
 };
+const STD_CRATES = /* @__PURE__ */ new Set(["std", "core", "alloc"]);
+function isStdCrate(name) {
+  return STD_CRATES.has(name);
+}
+function stdDocsUrl(crate, path) {
+  return `https://doc.rust-lang.org/stable/${crate}/${path}`;
+}
 function crateSlug(name) {
   return name.replace(/-/g, "_");
 }
 function docsUrl(crate, path = "index.html", version = "latest") {
+  if (isStdCrate(crate)) return stdDocsUrl(crate, path);
   return `${DOCS_BASE}/${crate}/${version}/${crateSlug(crate)}/${path}`;
 }
 function modToUrlPrefix(modulePath) {
@@ -51,7 +73,13 @@ function modToRustPrefix(modulePath) {
   return modulePath ? modulePath.replace(/\./g, "::") + "::" : "";
 }
 async function fetchDom(url) {
+  const cached = cacheGet(`dom:${url}`);
+  if (cached) {
+    console.log(`[cache hit] ${url}`);
+    return load(cached);
+  }
   const { data } = await axios.get(url, { timeout: 15e3 });
+  cacheSet(`dom:${url}`, data);
   return load(data);
 }
 function cleanHtml(html) {
@@ -74,12 +102,18 @@ function errorResult(msg) {
 }
 const cratesIoHeaders = { "User-Agent": USER_AGENT };
 async function fetchCrateInfo(name) {
+  const cacheKey = `crate-info:${name}`;
+  const cached = cacheGet(cacheKey);
+  if (cached) {
+    console.log(`[cache hit] crate-info ${name}`);
+    return cached;
+  }
   const { data } = await axios.get(`${CRATES_IO}/crates/${name}`, {
     headers: cratesIoHeaders,
     timeout: 1e4
   });
   const c = data.crate;
-  return {
+  const info = {
     name: c.name,
     version: c.max_stable_version || c.max_version,
     description: c.description,
@@ -87,34 +121,52 @@ async function fetchCrateInfo(name) {
     repository: c.repository,
     downloads: c.downloads
   };
+  cacheSet(cacheKey, info);
+  return info;
 }
 async function fetchCrateVersionInfo(name, version) {
+  const cacheKey = `crate-version:${name}@${version}`;
+  const cached = cacheGet(cacheKey);
+  if (cached) {
+    console.log(`[cache hit] crate-version ${name}@${version}`);
+    return cached;
+  }
   const { data } = await axios.get(`${CRATES_IO}/crates/${name}/${version}`, {
     headers: cratesIoHeaders,
     timeout: 1e4
   });
   const v = data.version;
   const features = v.features ?? {};
-  return {
+  const info = {
     num: v.num,
     features,
     defaultFeatures: features["default"] ?? [],
     yanked: v.yanked,
     license: v.license
   };
+  cacheSet(cacheKey, info);
+  return info;
 }
 async function fetchCrateDeps(name, version) {
+  const cacheKey = `crate-deps:${name}@${version}`;
+  const cached = cacheGet(cacheKey);
+  if (cached) {
+    console.log(`[cache hit] crate-deps ${name}@${version}`);
+    return cached;
+  }
   const { data } = await axios.get(`${CRATES_IO}/crates/${name}/${version}/dependencies`, {
     headers: cratesIoHeaders,
     timeout: 1e4
   });
-  return (data.dependencies ?? []).map((d) => ({
+  const deps = (data.dependencies ?? []).map((d) => ({
     name: d.crate_id,
     req: d.req,
     optional: d.optional,
     kind: d.kind,
     features: d.features ?? []
   }));
+  cacheSet(cacheKey, deps);
+  return deps;
 }
 function extractItemFeatureGate($) {
   const gate = $(".item-info .stab.portability").first().text().trim();
@@ -157,7 +209,7 @@ const itemTypeEnum = z.enum([
   "derive"
 ]);
 const versionParam = z.string().optional().describe('Crate version (e.g. "1.49.0"). Defaults to latest.');
-function register$5(server2) {
+function register$8(server2) {
   server2.tool(
     "lookup_crate_docs",
     "Fetch the main documentation for a Rust crate. Returns overview, version, sections, and re-exports.",
@@ -165,7 +217,6 @@ function register$5(server2) {
       crateName: z.string().describe('Crate name (e.g. "tokio", "serde-json")'),
       version: versionParam
     },
-    // @ts-expect-error — MCP SDK deep type instantiation with Zod schemas
     async ({ crateName, version }) => {
       try {
         const ver = version ?? "latest";
@@ -195,7 +246,7 @@ function register$5(server2) {
     }
   );
 }
-function register$4(server2) {
+function register$7(server2) {
   server2.tool(
     "get_crate_items",
     "List public items in a crate root or module. Returns names, types, feature gates, and short descriptions.",
@@ -240,7 +291,7 @@ function register$4(server2) {
     }
   );
 }
-function register$3(server2) {
+function register$6(server2) {
   server2.tool(
     "lookup_crate_item",
     "Get detailed documentation for a specific item. Returns signature, docs, feature gate, methods, trait impls, and optionally examples.",
@@ -253,7 +304,6 @@ function register$3(server2) {
       includeExamples: z.boolean().optional().describe("Include code examples from the docs. Default: false."),
       includeImpls: z.boolean().optional().describe("Include trait implementation list. Default: false.")
     },
-    // @ts-expect-error — MCP SDK deep type instantiation with Zod schemas
     async ({ crateName, itemType, itemName, modulePath, version, includeExamples, includeImpls }) => {
       try {
         const ver = version ?? "latest";
@@ -339,7 +389,7 @@ function scoreMatch(name, query) {
   if (lower.includes(q)) return 20;
   return 0;
 }
-function register$2(server2) {
+function register$5(server2) {
   server2.tool(
     "search_crate",
     "Search for items by name within a Rust crate. Returns ranked results with canonical paths and item types.",
@@ -383,7 +433,7 @@ function register$2(server2) {
     }
   );
 }
-function register$1(server2) {
+function register$4(server2) {
   server2.tool(
     "get_crate_metadata",
     "Get crate metadata from crates.io: version, features, default features, optional dependencies, and links.",
@@ -393,6 +443,12 @@ function register$1(server2) {
     },
     async ({ crateName, version }) => {
       try {
+        if (isStdCrate(crateName)) {
+          return textResult(
+            `"${crateName}" is part of the Rust standard library and is not published on crates.io.
+Use lookup_crate_docs, get_crate_items, lookup_crate_item, or search_crate to browse its documentation.`
+          );
+        }
         const info = await fetchCrateInfo(crateName);
         const ver = version ?? info.version;
         const [versionInfo, deps] = await Promise.all([
@@ -444,7 +500,7 @@ function register$1(server2) {
     }
   );
 }
-function register(server2) {
+function register$3(server2) {
   server2.tool(
     "get_crate_brief",
     "Bundle call: fetches crate metadata, overview docs, module list, re-exports, and optionally items from focused modules — all in one shot.",
@@ -455,9 +511,16 @@ function register(server2) {
     },
     async ({ crateName, version, focusModules }) => {
       try {
-        const info = await fetchCrateInfo(crateName);
-        const ver = version ?? info.version;
-        const versionInfo = await fetchCrateVersionInfo(crateName, ver);
+        const isStd = isStdCrate(crateName);
+        let info = null;
+        let versionInfo = null;
+        if (!isStd) {
+          const crateInfo = await fetchCrateInfo(crateName);
+          const ver2 = version ?? crateInfo.version;
+          versionInfo = await fetchCrateVersionInfo(crateName, ver2);
+          info = crateInfo;
+        }
+        const ver = isStd ? "latest" : version ?? info.version;
         const rootUrl = docsUrl(crateName, "index.html", ver);
         const $ = await fetchDom(rootUrl);
         const doc = truncate(cleanHtml($("details.toggle.top-doc").html() ?? ""), 3e3);
@@ -477,22 +540,33 @@ function register(server2) {
           });
           if (items.length) itemsBySection[type] = items;
         });
-        const parts = [
-          `# ${info.name} v${versionInfo.num}`,
-          rootUrl,
-          "",
-          info.description,
-          "",
-          `license: ${versionInfo.license} | downloads: ${info.downloads.toLocaleString()}`
-        ];
-        if (info.repository) parts.push(`repo: ${info.repository}`);
-        const { defaultFeatures, features } = versionInfo;
-        parts.push(
-          "",
-          "## Features",
-          `  default = [${defaultFeatures.join(", ")}]`,
-          `  all: ${Object.keys(features).filter((f) => f !== "default").sort().join(", ")}`
-        );
+        const parts = [];
+        if (isStd) {
+          const pageVersion = $(".sidebar-crate .version").text().trim() || "stable";
+          parts.push(
+            `# ${crateName} (Rust standard library) v${pageVersion}`,
+            rootUrl,
+            "",
+            `The "${crateName}" crate is part of the Rust standard library.`
+          );
+        } else {
+          parts.push(
+            `# ${info.name} v${versionInfo.num}`,
+            rootUrl,
+            "",
+            info.description,
+            "",
+            `license: ${versionInfo.license} | downloads: ${info.downloads.toLocaleString()}`
+          );
+          if (info.repository) parts.push(`repo: ${info.repository}`);
+          const { defaultFeatures, features } = versionInfo;
+          parts.push(
+            "",
+            "## Features",
+            `  default = [${defaultFeatures.join(", ")}]`,
+            `  all: ${Object.keys(features).filter((f) => f !== "default").sort().join(", ")}`
+          );
+        }
         if (doc) parts.push("", "## Overview", doc);
         if (reexports.length) {
           parts.push("", "## Re-exports", ...reexports.map((r) => `  ${r}`));
@@ -536,8 +610,157 @@ function register(server2) {
     }
   );
 }
+function register$2(server2) {
+  server2.tool(
+    "search_crates",
+    "Search for Rust crates on crates.io by keyword. Returns name, description, downloads, and version.",
+    {
+      query: z.string().describe("Search keywords"),
+      page: z.number().min(1).optional().describe("Page number (default 1)"),
+      perPage: z.number().min(1).max(50).optional().describe("Results per page (default 10, max 50)")
+    },
+    async ({ query, page: rawPage, perPage: rawPerPage }) => {
+      const page = rawPage ?? 1;
+      const perPage = rawPerPage ?? 10;
+      try {
+        const cacheKey = `search-crates:${query}:${page}:${perPage}`;
+        const cached = cacheGet(cacheKey);
+        if (cached) {
+          console.log(`[cache hit] search-crates "${query}" page=${page}`);
+          return textResult(cached);
+        }
+        const { data } = await axios.get(`${CRATES_IO}/crates`, {
+          params: { q: query, per_page: perPage, page },
+          headers: { "User-Agent": USER_AGENT },
+          timeout: 1e4
+        });
+        const crates = data.crates ?? [];
+        if (!crates.length) {
+          return textResult(`No crates found for "${query}".`);
+        }
+        const total = data.meta?.total ?? crates.length;
+        const lines = crates.map((c) => {
+          const ver = c.max_stable_version || c.max_version;
+          const dl = c.downloads.toLocaleString();
+          const desc = c.description ? ` — ${c.description.trim()}` : "";
+          return `  ${c.name} v${ver} (${dl} downloads)${desc}`;
+        });
+        const result = [
+          `# Crate search: "${query}" — ${total} results (page ${page})`,
+          "",
+          ...lines
+        ].join("\n");
+        cacheSet(cacheKey, result);
+        return textResult(result);
+      } catch (e) {
+        return errorResult(`Could not search crates.io for "${query}". ${e.message}`);
+      }
+    }
+  );
+}
+function register$1(server2) {
+  server2.tool(
+    "get_crate_versions",
+    "List all published versions of a crate from crates.io, with yanked status and release dates.",
+    {
+      crateName: z.string().describe("Crate name")
+    },
+    async ({ crateName }) => {
+      try {
+        if (isStdCrate(crateName)) {
+          return textResult(
+            `"${crateName}" is part of the Rust standard library and is not published on crates.io.
+Its version matches the Rust toolchain version.`
+          );
+        }
+        const cacheKey = `crate-versions:${crateName}`;
+        const cached = cacheGet(cacheKey);
+        if (cached) {
+          console.log(`[cache hit] crate-versions ${crateName}`);
+          return textResult(cached);
+        }
+        const { data } = await axios.get(`${CRATES_IO}/crates/${crateName}/versions`, {
+          headers: { "User-Agent": USER_AGENT },
+          timeout: 1e4
+        });
+        const versions = (data.versions ?? []).map((v) => ({
+          num: v.num,
+          yanked: v.yanked,
+          created_at: v.created_at,
+          license: v.license ?? ""
+        }));
+        if (!versions.length) {
+          return textResult(`No versions found for "${crateName}".`);
+        }
+        const lines = versions.map((v) => {
+          const date = v.created_at.slice(0, 10);
+          const yanked = v.yanked ? " [YANKED]" : "";
+          return `  ${v.num}  ${date}${yanked}`;
+        });
+        const result = [
+          `# ${crateName} — ${versions.length} versions`,
+          "",
+          ...lines
+        ].join("\n");
+        cacheSet(cacheKey, result);
+        return textResult(result);
+      } catch (e) {
+        return errorResult(`Could not fetch versions for "${crateName}". ${e.message}`);
+      }
+    }
+  );
+}
+function register(server2) {
+  server2.tool(
+    "get_source_code",
+    "Fetch the source code of a Rust item from docs.rs. Returns the raw source implementation.",
+    {
+      crateName: z.string().describe("Crate name"),
+      path: z.string().describe('Source path relative to crate root (e.g. "src/lib.rs", "src/sync/mutex.rs")'),
+      version: versionParam
+    },
+    async ({ crateName, path, version }) => {
+      try {
+        if (isStdCrate(crateName)) {
+          const url2 = `https://doc.rust-lang.org/stable/src/${crateName}/${path}`;
+          const $2 = await fetchDom(url2);
+          const code2 = $2("#source-code").text().trim() || $2("pre.rust").text().trim();
+          if (!code2) return errorResult(`No source code found at ${url2}`);
+          return textResult([
+            `# Source: ${crateName}/${path}`,
+            url2,
+            "",
+            "```rust",
+            truncate(code2, 12e3),
+            "```"
+          ].join("\n"));
+        }
+        const ver = version ?? "latest";
+        const url = `${DOCS_BASE}/${crateName}/${ver}/src/${crateSlug(crateName)}/${path}`;
+        const $ = await fetchDom(url);
+        const code = $("#source-code").text().trim() || $("pre.rust").text().trim() || $(".src-line-numbers + code").text().trim();
+        if (!code) {
+          return errorResult(`No source code found at ${url}. Check that the path is correct.`);
+        }
+        return textResult([
+          `# Source: ${crateName}/${path}`,
+          url,
+          "",
+          "```rust",
+          truncate(code, 12e3),
+          "```"
+        ].join("\n"));
+      } catch (e) {
+        return errorResult(`Could not fetch source for "${crateName}/${path}". ${e.message}`);
+      }
+    }
+  );
+}
 console.log = (...args) => console.error(...args);
-const server = new McpServer({ name: "rust-docs", version: "2.0.0" });
+const server = new McpServer({ name: "rust-docs", version: "3.1.0" });
+register$8(server);
+register$7(server);
+register$6(server);
 register$5(server);
 register$4(server);
 register$3(server);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mcp-rustdoc",
-  "version": "2.0.0",
+  "version": "3.1.0",
   "type": "module",
   "description": "MCP server for fetching and browsing Rust crate documentation from docs.rs and crates.io",
   "main": "dist/index.js",
@@ -39,18 +39,18 @@
   },
   "homepage": "https://github.com/kieled/mcp-rustdoc#readme",
   "engines": {
-    "node": ">=18"
+    "node": ">=20"
   },
   "dependencies": {
-    "@modelcontextprotocol/sdk": "^1.6.1",
-    "axios": "^1.6.0",
-    "cheerio": "^1.0.0",
-    "html-to-text": "^9.0.5",
-    "zod": "^3.23.0"
+    "@modelcontextprotocol/sdk": "1.26.0",
+    "axios": "1.13.4",
+    "cheerio": "1.2.0",
+    "html-to-text": "9.0.5",
+    "zod": "4.3.6"
   },
   "devDependencies": {
-    "@types/node": "^22.0.0",
-    "typescript": "^5.7.0",
-    "vite": "^6.0.0"
+    "@types/node": "25.2.1",
+    "typescript": "5.9.3",
+    "vite": "7.3.1"
   }
 }