swagger-digest 0.2.0 → 0.3.0

package/README.md

@@ -91,6 +91,37 @@ By default the digest is returned directly in the tool result (so it lands in th
 - **No headless-browser fallback:** the optional Playwright fallback is disabled in the MCP server.
 - Diagnostics go to stderr only; stdout carries protocol frames exclusively.
 
+## Benchmarks
+
+Ten public APIs, digest vs. raw spec JSON in an LLM context, ground truth from the parsed raw specs. Run in November 2026 against swagger-digest 0.2.0; token counts are heuristic estimates, not tokenizer-exact.
+
+**7/10 wins · 100% operation coverage on every digest produced · 1.6–3.7× lighter · 1 call vs 1–5**
+
+| Spec                | Ops   | Raw tokens             | Digest tokens           | Compression | Op coverage | Winner  |
+| ------------------- | ----- | ---------------------- | ----------------------- | ----------- | ----------- | ------- |
+| Petstore v2         | 20    | 3,461                  | 2,202                   | 1.6×        | 20/20       | 🟢 MCP  |
+| Petstore v3         | 19    | 4,277                  | 2,624                   | 1.6×        | 19/19       | 🟢 MCP  |
+| Petstore v3 UI page | 19    | 4,277 (+4 URL guesses) | ❌ failed*              | —           | —           | 🟡 Raw* |
+| Azure Compute       | 157   | 157,568                | 72,549                  | 2.2×        | 157/157     | 🟢 MCP  |
+| X/Twitter v2        | 259   | 333,985                | 105,719                 | 3.2×        | 259/259     | 🟢 MCP  |
+| Box                 | 297   | 453,200                | 162,829                 | 2.8×        | 297/297     | 🟢 MCP  |
+| Twilio              | 197   | 469,913                | 155,709                 | 3.0×        | 197/197     | 🟢 MCP  |
+| Stripe              | 612   | 2,079,211              | 564,308                 | 3.7×        | 612/612     | 🟢 MCP  |
+| Kubernetes          | 1,172 | 1,085,939              | 1,341,885 (expanded!)   | ❌ 0.8×*    | 1172/1172   | 🔴 Raw  |
+| GitHub (full)       | 1,272 | 3,551,639              | ❌ failed (14 MB fetch) | —           | —           | 🟡 Raw* |
+
+> For its target use case — understanding an API in an LLM context — the digest wins on the spec class that actually matters (up to ~600 operations): 100% endpoint coverage, never truncated, one call, and 1.6–3.7× lighter. The bigger the spec, the bigger the win.
+
+\* Raw's three "wins" carry heavy caveats: the Petstore UI-page case needed brute-forced candidate URLs (the digest's discovery has since been fixed for that class of pages); GitHub's 3.55M-token raw spec would truncate in any model's context anyway; and the Kubernetes expansion is now prevented by the default token cap described below.
+
+### Known failure modes (and their mitigations)
+
+| Case                                                         | What happens                                                           | Mitigation                                                                                                                             |
+| ------------------------------------------------------------ | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
+| JS-only Swagger UI pages with dynamically computed spec URLs | Literal extraction finds nothing after `url:`                          | **Fixed in 0.3.0** (embedded spec-URL scan). Otherwise pass `spec_url` / `--spec` explicitly                                           |
+| Mega-specs (Kubernetes-class, ≥1M tokens)                    | Digest can _expand_ past raw size — descriptions inlined per operation | **Default in 0.3.0**: automatic 250k-token cap (opt out with `--no-auto-trim` / `auto_trim: false`). Explicit `max_tokens` always wins |
+| Huge fetches (14 MB GitHub spec)                             | Fetch fails at the 10 MB response cap                                  | Use a filtered direct spec URL                                                                                                         |
+
 ### Filtering and token budget
 
 Large specs produce large digests. The CLI can remove whole operations before rendering and/or trim the remaining digest to an approximate token budget:
@@ -116,6 +147,8 @@ How `--max-tokens` trims (in this fixed order, re-checked against the estimate a
 
 A `## Trimmed` note at the end of the digest records exactly which steps were applied. After every step, schemas that became unreachable (referenced only by collapsed/removed operations) are dropped from the Data types section. The token estimator is a heuristic, **not** a real tokenizer — treat `--max-tokens` as a guide, not a guarantee. Unknown tag names passed to `--include-tag`/`--exclude-tag` produce a warning on stderr listing the tags that exist in the spec.
 
+**Default token cap:** without an explicit `--max-tokens`, digests are automatically trimmed to an estimated 250,000 tokens (a warning names the untrimmed estimate). Mega-specs can otherwise expand _past_ their raw size once descriptions are inlined per operation. Opt out with `--no-auto-trim`; an explicit `--max-tokens` always takes precedence. The MCP server behaves the same via its `auto_trim` argument (default true).
+
 The same options are available programmatically via the third argument of `transformOpenApiToMarkdown(spec, sourceUrl, { includeTags, excludeTags, pathPrefix, maxTokens, onWarning })`.
 
 ## What it extracts
@@ -123,7 +156,7 @@ The same options are available programmatically via the third argument of `trans
 The retrieval strategy deliberately avoids scraping rendered operation accordions or simulating UI clicks:
 
 1. Fetch and parse a direct `.json`, `.yaml`, or `.yml` URL.
-2. Fetch the Swagger UI page and inspect script configuration for `url: "..."` or an inline `spec: { ... }` object.
+2. Fetch the Swagger UI page and inspect script configuration for `url: "..."` or an inline `spec: { ... }` object, plus any spec-like URLs embedded in same-origin initializer scripts (covers initializers that compute the spec URL dynamically).
 3. Try common spec endpoints on the same origin: `/v3/api-docs`, `/swagger/v1/swagger.json`, `/swagger.json`, `/openapi.json`, and `/api-docs`.
 4. Optionally launch a headless Chromium browser and read the parsed specification from Swagger UI's in-memory state.
 

package/dist/{chunk-2NORV6KM.js → chunk-DJES77HX.js}

@@ -6,7 +6,7 @@ import {
   parseHeaderPairs,
   transformOpenApiToMarkdown,
   writeDigest
-} from "./chunk-U7OLLL6J.js";
+} from "./chunk-MXTHI25H.js";
 
 // src/mcp/digest.ts
 var TOKEN_HINT_THRESHOLD = 8e4;
@@ -69,6 +69,7 @@ async function runSwaggerDigest(args, options = {}) {
     excludeTags: args.exclude_tags,
     pathPrefix: args.path_prefix,
     maxTokens: args.max_tokens,
+    autoTrim: args.auto_trim !== false,
     onWarning: warn
   });
   const operationCount = countDigestOperations(markdown);
@@ -128,6 +129,9 @@ function createMcpServer(options = {}) {
         max_tokens: z.number().int().positive().optional().describe(
           "Trim the digest until an approximate token estimate fits this budget (heuristic, not tokenizer-accurate)"
         ),
+        auto_trim: z.boolean().optional().describe(
+          "Auto-trim digests exceeding the 250,000-token default budget when max_tokens is not given. Defaults to true."
+        ),
         write_to_file: z.object({ directory: z.string(), filename: z.string() }).optional().describe("Save the digest to a file instead of returning it inline")
       }
     },

package/dist/{chunk-U7OLLL6J.js → chunk-MXTHI25H.js}

@@ -586,6 +586,35 @@ function extractSpecUrlsFromScripts(scripts, pageUrl) {
   }
   return found;
 }
+function extractEmbeddedSpecUrlsFromScripts(scripts, pageUrl) {
+  const found = [];
+  const push = (raw) => {
+    try {
+      const url = new URL(raw);
+      if (url.protocol !== "http:" && url.protocol !== "https:") return;
+      if (!/\.(?:json|ya?ml)$/i.test(url.pathname)) return;
+      if (!/(?:openapi|swagger|api[-._]?docs)/i.test(url.pathname)) return;
+      const resolved = url.toString();
+      if (!found.includes(resolved)) found.push(resolved);
+    } catch {
+    }
+  };
+  for (const script of scripts) {
+    for (const match of script.matchAll(/https?:\/\/[^\s"'`<>(){}[\]\\,=]+/gi)) {
+      push(match[0]);
+    }
+  }
+  const sameOrigin = found.filter((url) => {
+    try {
+      return new URL(url).origin === pageUrl.origin;
+    } catch {
+      return false;
+    }
+  });
+  const crossOrigin = found.filter((url) => !sameOrigin.includes(url));
+  return [...sameOrigin, ...crossOrigin].slice(0, MAX_EMBEDDED_SPEC_URLS);
+}
+var MAX_EMBEDDED_SPEC_URLS = 8;
 function findObjectLiteralArrayAfterKey(source, key) {
   const keyPattern = new RegExp(`\\b${key}\\s*:`);
   const match = keyPattern.exec(source);
@@ -703,7 +732,7 @@ async function fetchExternalConfigScripts(srcs, pageUrl, ctx, errors) {
     try {
       const response = await fetchText(candidate.toString(), ctx);
       scripts.push(response.text);
-      if (extractSpecUrlFromScripts(scripts, pageUrl) || scripts.some((script) => findObjectLiteralAfterKey(script, "spec"))) {
+      if (extractSpecUrlFromScripts(scripts, pageUrl) || extractEmbeddedSpecUrlsFromScripts(scripts, pageUrl).length > 0 || scripts.some((script) => findObjectLiteralAfterKey(script, "spec"))) {
         break;
       }
     } catch (error) {
@@ -983,6 +1012,13 @@ async function fetchOpenApiSpec(input, options = {}) {
     const externalUrls = extractSpecUrls(externalScripts, pageUrl).filter((url) => !inlineUrls.includes(url));
     const fromExternal = await probeSpecUrls([...inlineUrls, ...externalUrls], ctx, recordFailure, noteMultiple);
     if (fromExternal) return fromExternal;
+    const triedUrls = /* @__PURE__ */ new Set([...inlineUrls, ...externalUrls]);
+    const embeddedUrls = [
+      ...extractEmbeddedSpecUrlsFromScripts(scripts, pageUrl),
+      ...extractEmbeddedSpecUrlsFromScripts(externalScripts, pageUrl)
+    ].filter((url) => !triedUrls.has(url));
+    const fromEmbedded = await probeSpecUrls(embeddedUrls, ctx, recordFailure);
+    if (fromEmbedded) return fromEmbedded;
     for (const script of scripts) {
       const literal = findObjectLiteralAfterKey(script, "spec");
       if (!literal) continue;
@@ -1904,6 +1940,7 @@ function normalizeInlinePayloadSchemas(spec) {
 }
 
 // src/transformer/index.ts
+var DEFAULT_MAX_TOKENS = 25e4;
 var HTTP_METHODS2 = ["get", "put", "post", "delete", "options", "head", "patch", "trace"];
 function tagsOfEndpoint(endpoint) {
   const tags = endpoint.operation.tags;
@@ -2076,16 +2113,22 @@ function transformOpenApiToMarkdown(spec, sourceUrl, options = {}) {
     collapsedTags: /* @__PURE__ */ new Set(),
     indexOnly: false
   };
-  if (options.maxTokens === void 0) {
-    return renderDigest(normalizedSpec, sourceUrl, endpoints, tags, fullStage);
-  }
   const worst = (markdown2) => {
     const report = estimateTokens(markdown2);
     return Math.max(...Object.values(report.models).map((model) => model.tokens));
   };
-  const fits = (markdown2) => worst(markdown2) <= options.maxTokens;
+  const effectiveMaxTokens = options.maxTokens ?? (options.autoTrim === false ? void 0 : DEFAULT_MAX_TOKENS);
+  if (effectiveMaxTokens === void 0) {
+    return renderDigest(normalizedSpec, sourceUrl, endpoints, tags, fullStage);
+  }
+  const fits = (markdown2) => worst(markdown2) <= effectiveMaxTokens;
   const full = renderDigest(normalizedSpec, sourceUrl, endpoints, tags, fullStage);
   if (fits(full)) return full;
+  if (options.maxTokens === void 0) {
+    options.onWarning?.(
+      `Untrimmed digest \u2248 ${worst(full).toLocaleString()} tokens (estimated), over the ${DEFAULT_MAX_TOKENS.toLocaleString()} default budget; auto-trimming. Pass maxTokens to control the budget or autoTrim: false to disable.`
+    );
+  }
   const stages = buildTrimStages(endpoints, tags);
   for (let i = 1; i < stages.length; i += 1) {
     const stage = stages[i];
@@ -2094,14 +2137,14 @@ function transformOpenApiToMarkdown(spec, sourceUrl, options = {}) {
       return appendTrimNote(
         markdown2,
         stages.slice(1, i + 1).map((entry) => entry.action).filter((action) => Boolean(action)),
-        options.maxTokens
+        effectiveMaxTokens
       );
     }
   }
   const last = stages[stages.length - 1];
   const markdown = renderDigest(normalizedSpec, sourceUrl, endpoints, tags, last);
   const allActions = stages.slice(1).map((stage) => stage.action).filter((action) => Boolean(action));
-  return appendTrimNote(markdown, allActions, options.maxTokens);
+  return appendTrimNote(markdown, allActions, effectiveMaxTokens);
 }
 
 // src/headers.ts
@@ -2196,6 +2239,7 @@ export {
   parseSpecText,
   fetchOpenApiSpec,
   estimateTokens,
+  DEFAULT_MAX_TOKENS,
   transformOpenApiToMarkdown,
   parseHeaderArgument,
   parseHeaderPairs,

package/dist/cli.js

@@ -10,7 +10,7 @@ import {
   parseHeaderPairs,
   transformOpenApiToMarkdown,
   writeDigest
-} from "./chunk-U7OLLL6J.js";
+} from "./chunk-MXTHI25H.js";
 
 // src/cli.ts
 import { input } from "@inquirer/prompts";
@@ -61,7 +61,7 @@ async function main() {
     "Custom request header, repeatable. Sent only to the URL you supplied.",
     collectOption,
     []
-  ).option("--cookie <string>", "Convenience for a Cookie request header (same-origin scoping applies)").option("--include-tag <tag>", "Keep only operations with this tag, repeatable", collectOption, []).option("--exclude-tag <tag>", "Drop operations with this tag, repeatable", collectOption, []).option("--path-prefix <prefix>", "Keep only operations whose path starts with this prefix").option("--max-tokens <n>", "Trim the digest until the approximate token estimate fits this budget").option("--spec <url>", "Explicit OpenAPI spec URL; overrides discovery from the Swagger UI page").option("--stdout", "Print the digest to standard output instead of writing a file").addHelpText(
+  ).option("--cookie <string>", "Convenience for a Cookie request header (same-origin scoping applies)").option("--include-tag <tag>", "Keep only operations with this tag, repeatable", collectOption, []).option("--exclude-tag <tag>", "Drop operations with this tag, repeatable", collectOption, []).option("--path-prefix <prefix>", "Keep only operations whose path starts with this prefix").option("--max-tokens <n>", "Trim the digest until the approximate token estimate fits this budget").option("--no-auto-trim", "Disable the automatic 250,000-token default cap for digests without --max-tokens").option("--spec <url>", "Explicit OpenAPI spec URL; overrides discovery from the Swagger UI page").option("--stdout", "Print the digest to standard output instead of writing a file").addHelpText(
     "after",
     [
       "",
@@ -132,6 +132,7 @@ async function main() {
     excludeTags: options.excludeTag,
     pathPrefix: options.pathPrefix,
     maxTokens: parseMaxTokens(options.maxTokens),
+    autoTrim: options.autoTrim !== false,
     onWarning: (message) => console.error(chalk.yellow(`Warning: ${message}`))
   });
   spinner.succeed(`Parsed ${title} (${result.retrievalMethod})`);

package/dist/index.d.ts

@@ -57,6 +57,11 @@ interface McpDigestArgs {
     path_prefix?: string;
     /** Trim the digest until the approximate token estimate fits this budget. */
     max_tokens?: number;
+    /**
+     * Auto-trim digests exceeding the 250,000-token default budget when
+     * max_tokens is not given. Defaults to true.
+     */
+    auto_trim?: boolean;
     /** Save the digest to a file instead of returning it inline. */
     write_to_file?: {
         directory: string;
@@ -114,9 +119,24 @@ interface TransformOptions {
      * budget. The estimator is not tokenizer-accurate; treat the budget as a guide.
      */
     maxTokens?: number;
+    /**
+     * When no explicit `maxTokens` is given, cap the digest at
+     * {@link DEFAULT_MAX_TOKENS} estimated tokens by running the same trim
+     * pipeline, warning when it engages. Defaults to true. Mega-specs
+     * (Kubernetes-class) otherwise produce digests *larger* than the raw spec.
+     */
+    autoTrim?: boolean;
     /** Receives non-fatal notices, e.g. unknown tag names. */
     onWarning?: (message: string) => void;
 }
+/**
+ * Default token budget applied when `maxTokens` is undefined and `autoTrim`
+ * is not disabled. Chosen from benchmark data: large real-world specs
+ * (Stripe-class, ~600 operations) digest to ~560k tokens untrimmed, while
+ * kitchen-sink specs expand past their raw size — 250k keeps the largest
+ * useful full-detail digests intact and trims only true outliers.
+ */
+declare const DEFAULT_MAX_TOKENS = 250000;
 /** Convert an OpenAPI 2.0, 3.0, or 3.1 document to a compact Markdown digest. */
 declare function transformOpenApiToMarkdown(spec: JsonObject, sourceUrl?: string, options?: TransformOptions): string;
 
@@ -153,5 +173,5 @@ declare function writeDigest(options: {
     filename: string;
 }): Promise<string>;
 
-export { buildRequestHeaders, createMcpServer, defaultFilename, defaultOutputDirectory, estimateTokens, fetchOpenApiSpec, isOpenApiSpec, normalizeFilename, parseSpecText, resolveOutputDirectory, runSwaggerDigest, toTypeScriptIdentifier, transformOpenApiToMarkdown, writeDigest };
+export { DEFAULT_MAX_TOKENS, buildRequestHeaders, createMcpServer, defaultFilename, defaultOutputDirectory, estimateTokens, fetchOpenApiSpec, isOpenApiSpec, normalizeFilename, parseSpecText, resolveOutputDirectory, runSwaggerDigest, toTypeScriptIdentifier, transformOpenApiToMarkdown, writeDigest };
 export type { DigestServerOptions, DigestTokenReport, FetchOptions, FetchResult, JsonObject, McpDigestArgs, ModelTokenEstimate, TransformOptions };

package/dist/index.js

@@ -2,8 +2,9 @@ import {
   buildRequestHeaders,
   createMcpServer,
   runSwaggerDigest
-} from "./chunk-2NORV6KM.js";
+} from "./chunk-DJES77HX.js";
 import {
+  DEFAULT_MAX_TOKENS,
   defaultFilename,
   defaultOutputDirectory,
   estimateTokens,
@@ -15,8 +16,9 @@ import {
   toTypeScriptIdentifier,
   transformOpenApiToMarkdown,
   writeDigest
-} from "./chunk-U7OLLL6J.js";
+} from "./chunk-MXTHI25H.js";
 export {
+  DEFAULT_MAX_TOKENS,
   buildRequestHeaders,
   createMcpServer,
   defaultFilename,

package/dist/mcp/index.js

@@ -1,8 +1,8 @@
 #!/usr/bin/env node
 import {
   createMcpServer
-} from "../chunk-2NORV6KM.js";
-import "../chunk-U7OLLL6J.js";
+} from "../chunk-DJES77HX.js";
+import "../chunk-MXTHI25H.js";
 
 // src/mcp/index.ts
 import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "swagger-digest",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "description": "Turn Swagger UI pages and OpenAPI specs into compact, AI-friendly Markdown API digests.",
   "type": "module",
   "main": "dist/index.js",