website-to-markdown 1.1.1 → 2.0.0

package/README.md

@@ -25,11 +25,75 @@ if (page.outcome === 'ok') console.log(page.title, page.markdown, page.links)
 ```
 
 `fetchPageAsMarkdown` never throws: a page that cannot be read comes back as
-an `outcome` (`unchanged`, `gone`, `blocked`, `transient`, `unsupported`).
-Pass the last read's `etag`, `lastModified` or `htmlHash` to get `unchanged`
-without converting again. `convertDocumentBytes` converts an uploaded file;
+an `outcome`, and anything unexpected is logged and comes back `transient`.
+`convertDocumentBytes` converts an uploaded file (PDF, office files, images);
 `urlToMarkdown` is the one-shot form that throws.
 
+| `outcome` | Meaning | `markdown` |
+|---|---|---|
+| `ok` | Read and converted. | the page |
+| `unchanged` | A 304, or the same HTML as the `htmlHash` given. Keep the stored copy. | empty |
+| `gone` | 404 or 410. | empty |
+| `blocked` | A 4xx other than 429, a bot wall Browser Rendering could not pass, or the SSRF guard refused the URL or a redirect. | empty |
+| `transient` | 5xx, a 429 after three paused retries, a network failure, or every conversion tier missed. Try again next run. | empty |
+| `unsupported` | Not a URL we read (bad URL, asset, oversized HTML or document). It will not get better on its own. | empty |
+
+`source` says which tier produced the markdown: `content-negotiation` (the
+origin served markdown), `workers-ai`, `browser-rendering`, `not-modified`
+(for `unchanged`) or `firecrawl-parse` (the `parsePdf` tier, whichever parser
+it is). `captureIssue` says why a page is incomplete or missing, when known:
+
+| `captureIssue` | |
+|---|---|
+| `markdown_limit` | Markdown cut at `MAX_MARKDOWN_CHARS` (300,000). |
+| `html_byte_limit` | HTML over 2 MB is not converted; over 4 MB it is not downloaded. |
+| `document_byte_limit` | A linked document over 2 MB (PDF: 12 MB). |
+| `workers_ai_rate_limited` | Workers AI kept answering 429. |
+| `browser_rendering_rate_limited` | Browser Rendering kept answering 429. |
+| `render_incomplete` | The page is an app shell and rendering did not fill it. |
+| `pdf_page_limit_unverified` | The PDF parser read 60 pages or could not say how many; the rest may be missing. |
+| `pdf_extraction_failed` | Workers AI and `parsePdf` both returned nothing. |
+
+### Reading a page again
+
+Pass the last read's `etag` and `lastModified` (sent as conditional headers)
+and `htmlHash` (the hash of the HTML served; most origins send no
+validators). Identical HTML comes back `unchanged` without a Workers AI call.
+The hash includes a conversion version, so after an upgrade that changes the
+conversion every page is converted once more.
+
+```ts
+const next = await converter.fetchPageAsMarkdown({
+	url: stored.url,
+	etag: stored.etag,
+	lastModified: stored.lastModified,
+	htmlHash: stored.htmlHash
+})
+```
+
+`render: false` skips the Browser Rendering tier. `country: 'DK'` adds
+`?country=DK` to the request (not to the page's `url`), which pins a Shopify
+store's market when it would otherwise pick one from the server's IP.
+
+Every page's markdown goes through `cleanConvertedMarkdown`: Cloudflare's
+obfuscated e-mail links are decoded, JSON-LD code fences and `<?xml`/`<!DOCTYPE`
+text removed, and a label glued to its own `mailto:`/`tel:` value split. It is
+exported, and idempotent, for markdown stored by an older version.
+
+### Limits
+
+| | |
+|---|---|
+| HTML converted | 2 MB (read up to 4 MB for links and metadata) |
+| Documents | 2 MB, PDFs 12 MB, PDF pages read by `parsePdf` 60 |
+| Markdown kept | 300,000 characters |
+| Origin request timeout | 30 s, redirects 10 |
+| Pages in flight | `URL_TO_MARKDOWN_PAGE_CONCURRENCY` (80), for the caller's own pool |
+| Per origin host | 16 in flight; 2 (and 2 starts/s) for 15 minutes after a 429 |
+| Workers AI | 24 in flight, 12 starts/s |
+| Browser Rendering | 80 in flight, 28 starts/s |
+| Sitemaps | 20 files, 20,000 URLs, 10 MB per file; robots.txt 500 KiB |
+
 Create one converter per process: it owns the Cloudflare pools and the
 per-host limits.
 
@@ -43,16 +107,36 @@ per-host limits.
 | `reportTimeouts` | `true` | `false` keeps Workers AI timeouts from `onApiError`; the page falls through to the next tier anyway. |
 | `logger` | silent | `{ info, warn }` |
 | `parsePdf` | none | OCR tier for a PDF Workers AI returned nothing for. |
-| `isHostAllowed` | public addresses only | SSRF guard run before every origin request. |
+| `isHostAllowed` | public addresses only | SSRF guard run before every origin request and every redirect. |
+| `allowLocalhost` | `false` | Let the default guard and `http://` through to `localhost`, `127.0.0.1` and `::1`. Set it outside production for a dev server. |
+| `fetch` | global `fetch` | What origin requests go through, behind the guard. Not used for Cloudflare's API. |
 | `crawlSnapshot` | none | `{ root, logger? }` for the returned `withCrawlSnapshot`. |
 
 Besides the converters, the factory returns:
 
+- `guardedFetch(url, init?, { maxRedirects? })`: `fetch` behind the
+  converter's guard, which follows redirects itself and checks every hop.
+  Throws `HostNotAllowedError` when one is refused. Use it for any other
+  request to a customer-supplied URL (images, logos, an MCP server).
 - `isHostAllowed(hostname)`: the SSRF guard the converter runs, bound to its
-  `logger`. Use it for any other fetch of a customer-supplied URL.
+  `logger`.
 - `withCrawlSnapshot(input, run)`: [crawl snapshots](#crawl-snapshots) under
   `crawlSnapshot.root`. A `null` input just runs `run`.
 
+### SSRF guard
+
+`isHostAllowed(hostname, { logger?, allowLocalhost? })` resolves the name and
+refuses it when any address is private, loopback, link-local, CGNAT,
+multicast or reserved. An IP literal (`10.0.0.1`, `[fd00::1]`,
+`[::ffff:169.254.169.254]`) is judged as written. A failed DNS lookup is let
+through so the request reports the real error. Results are cached for 60 s;
+DNS rebinding between the check and the connection is not covered.
+
+`guardedFetch(url, init?, { isHostAllowed?, fetch?, maxRedirects? })` is the
+standalone form. On a redirect it does what `fetch` does: `Authorization` and
+`Cookie` are dropped when the origin changes, and a 303 (or a POST answered
+with 301/302) becomes a GET.
+
 A limiter is a function `run => Promise` with `tripCooldown(ms)` and
 `dispose()`:
 
@@ -103,6 +187,10 @@ const CrawlSnapshotInput = z.object({
 })
 ```
 
+`website-to-markdown/schema` also has `FETCH_OUTCOMES`, `FETCH_SOURCES` and
+`CAPTURE_ISSUES`, the runtime lists behind the types, for narrowing a value
+read back from storage: `z.enum(FETCH_OUTCOMES)`.
+
 ## Sitemaps
 
 ```ts
@@ -113,6 +201,29 @@ const pages = await discoverSitemapUrls('https://example.com', robots.sitemaps)
 const readable = pages.filter((page) => robotsAllows(robots.rules, page.url))
 ```
 
+Both read through `guardedFetch` with the default guard and send the browser
+user agent. Pass `{ fetch: converter.guardedFetch }` as the last argument to
+use the converter's guard, or any `fetch` in tests. Neither throws: a site
+without a robots.txt or a sitemap gives `[]`.
+
+## Testing
+
+```ts
+import { fakePage, fakeSite } from 'website-to-markdown/testing'
+
+const page = fakePage({ url: 'https://a.dk', markdown: '# A' }) // a complete ok page
+
+const site = fakeSite({
+	'https://a.dk': { markdown: '# A', links: ['https://a.dk/kontakt'], etag: 'v1' }
+})
+await site.fetchPageAsMarkdown({ url: 'https://a.dk', etag: 'v1' }) // unchanged, 304
+await site.fetchPageAsMarkdown({ url: 'https://a.dk/x' }) // gone, 404
+site.calls // every input, in order
+```
+
+`createMemoryConcurrencyLimit` as `createLimit` keeps a test off a shared
+limiter, and the `fetch` option keeps it off the network.
+
 ## CLI
 
 ```sh
@@ -129,3 +240,22 @@ bunx website-to-markdown https://example.com --env-file .env.cloudflare
 
 infisical run -- bunx website-to-markdown https://example.com
 ```
+
+## Upgrading from 1.x
+
+- Localhost is refused unless you pass `allowLocalhost: true` (1.x allowed it
+  whenever `NODE_ENV` was not `production`). For a dev server:
+  `createWebsiteToMarkdown({ …, allowLocalhost: process.env.NODE_ENV !== 'production' })`.
+  The standalone `isHostAllowed(hostname, logger)` is now
+  `isHostAllowed(hostname, { logger, allowLocalhost })`.
+- `FetchedPage` has no optional fields: `alternates`, `alternateLanguages`,
+  `captureIssue` and `htmlHash` are always set. Build test pages with
+  `fakePage` from `website-to-markdown/testing`.
+- `captureIssue` is the `CaptureIssue` union instead of `string`. A Browser
+  Rendering 429 is `browser_rendering_rate_limited`, no longer
+  `workers_ai_rate_limited`.
+- Redirects are followed by the library, so every hop passes the guard; a
+  redirect to a refused host is `blocked`. Sitemaps and robots.txt go through
+  the guard too.
+- The conversion version is bumped, so the first read of each page after the
+  upgrade converts it again even when its `htmlHash` matches.

package/dist/cli.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"cli.d.ts","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AAOA,OAAO,EACN,KAAK,KAAK,EAKV,MAAM,YAAY,CAAA;AAInB,eAAO,MAAM,GAAG,0BAqFd,CAAA;AAQF,0DAA0D;AAC1D,wBAAsB,IAAI,CAAC,IAAI,EAAE,SAAS,MAAM,EAAE,EAAE,EAAE,CAAC,EAAE,KAAK,mBAG7D"}
+{"version":3,"file":"cli.d.ts","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AAOA,OAAO,EACN,KAAK,KAAK,EAKV,MAAM,YAAY,CAAA;AAInB,eAAO,MAAM,GAAG,0BAuFd,CAAA;AAQF,0DAA0D;AAC1D,wBAAsB,IAAI,CAAC,IAAI,EAAE,SAAS,MAAM,EAAE,EAAE,EAAE,CAAC,EAAE,KAAK,mBAG7D"}

package/dist/cli.js

@@ -46,7 +46,9 @@ export const cli = defineCli({
                 cloudflare: {
                     apiToken: CLOUDFLARE_API_TOKEN,
                     accountId: CLOUDFLARE_ACCOUNT_ID
-                }
+                },
+                // Run by a person on their own machine, so a dev server is fair game.
+                allowLocalhost: true
             });
             ctx.stderr.write(`Fetching ${url}…\n`);
             const page = await fetchPageAsMarkdown({ url, render });

package/dist/cli.js.map

@@ -1 +1 @@
-{"version":3,"file":"cli.js","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AACA;;;GAGG;AACH,OAAO,EAAE,QAAQ,EAAE,MAAM,kBAAkB,CAAA;AAC3C,OAAO,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAA;AACpC,OAAO,EAEN,QAAQ,EACR,aAAa,EACb,SAAS,EACT,aAAa,EACb,MAAM,YAAY,CAAA;AACnB,OAAO,EAAE,CAAC,EAAE,MAAM,KAAK,CAAA;AACvB,OAAO,EAAE,uBAAuB,EAAE,MAAM,iBAAiB,CAAA;AAEzD,MAAM,CAAC,MAAM,GAAG,GAAG,SAAS,CAAC;IAC5B,IAAI,EAAE,qBAAqB;IAC3B,kDAAkD;IAClD,OAAO,EAAE,CAAC,KAAK,EAAE,EAAE,EAAE,EAAE;QACtB,IACC,KAAK,YAAY,aAAa;YAC9B,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC,EACnE,CAAC;YACF,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,KAAK,CAAC,IAAI,CAAC,CAAA;YAC3B,OAAO,CAAC,CAAA;QACT,CAAC;IACF,CAAC;IACD,QAAQ,EAAE,aAAa,CAAC;QACvB,8DAA8D;QAC9D,MAAM,EAAE,KAAK;QACb,WAAW,EACV,uFAAuF;QACxF,KAAK,EAAE,CAAC,CAAC,MAAM,CAAC;YACf,GAAG,EAAE,CAAC;iBACJ,MAAM,EAAE;iBACR,QAAQ,CAAC,gCAAgC,CAAC;iBAC1C,IAAI,CAAC,EAAE,UAAU,EAAE,IAAI,EAAE,CAAC;YAC5B,MAAM,EAAE,CAAC;iBACP,OAAO,EAAE;iBACT,OAAO,CAAC,IAAI,CAAC;iBACb,QAAQ,CACR,kEAAkE,CAClE;YACF,OAAO,EAAE,CAAC;iBACR,MAAM,EAAE;iBACR,QAAQ,EAAE;iBACV,QAAQ,CAAC,6DAA6D,CAAC;SACzE,CAAC;QACF,GAAG,EAAE,KAAK,EAAE,EAAE,GAAG,EAAE,MAAM,EAAE,OAAO,EAAE,EAAE,GAAG,EAAE,EAAE;YAC5C,MAAM,IAAI,GAAG,OAAO,CAAC,CAAC,CAAC,QAAQ,CAAC,MAAM,WAAW,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,CAAA;YAChE,MAAM,oBAAoB,GACzB,GAAG,CAAC,GAAG,CAAC,oBAAoB,IAAI,IAAI,CAAC,oBAAoB,CAAA;YAC1D,MAAM,qBAAqB,GAC1B,GAAG,CAAC,GAAG,CAAC,qBAAqB,IAAI,IAAI,CAAC,qBAAqB,CAAA;YAC5D,IAAI,CAAC,oBAAoB,IAAI,CAAC,qBAAqB;gBAClD,MAAM,IAAI,QAAQ,CACjB,qDAAqD,CACrD,CAAA;YAEF,MAAM,EAAE,mBAAmB,EAAE,GAAG,uBAAuB,CAAC;gBACvD,UAAU,EAAE;oBACX,QAAQ,EAAE,oBAAoB;oBAC9B,SAAS,EAAE,qBAAqB;iBAChC;aACD,CAAC,CAAA;YAEF,GAAG,CAAC,MAAM,CAAC,KAAK,CAAC,YAAY,GAAG,KAAK,CAAC,CAAA;YAEtC,MAAM,IAAI,GAAG,MAAM,mBAAmB,CAAC,EAAE,GAAG,EAAE,MAAM,EAAE,CAAC,CAAA;YAEvD,GAAG,CAAC,MAAM,CAAC,KAAK,CACf,GAAG,IAAI,CAAC,SAAS,CAChB;gBACC,OAAO,EAAE,IAAI,CAAC,OAAO;gBACrB,MAAM,EAAE,IAAI,CAAC,MAAM;gBACnB,MAAM,EAAE,IAAI,CAAC,MAAM;gBACnB,GAAG,EAAE,IAAI,CAAC,GAAG;gBACb,QAAQ,EAAE,IAAI,CAAC,QAAQ;gBACvB,WAAW,EAAE,IAAI,CAAC,WAAW;gBAC7B,IAAI,EAAE,IAAI,CAAC,IAAI;gBACf,YAAY,EAAE,IAAI,CAAC,YAAY;gBAC/B,KAAK,EAAE,IAAI,CAAC,KAAK;gBACjB,IAAI,EAAE,IAAI,CAAC,IAAI;gBACf,SAAS,EAAE,IAAI,CAAC,SAAS;gBACzB,eAAe,EAAE,IAAI,CAAC,eAAe;gBACrC,YAAY,EAAE,IAAI,CAAC,MAAM,CAAC,MAAM;gBAChC,KAAK,EAAE,IAAI,CAAC,KAAK,CAAC,MAAM;gBACxB,MAAM,EAAE,IAAI,CAAC,MAAM,CAAC,MAAM;gBAC1B,aAAa,EAAE,IAAI,CAAC,QAAQ,CAAC,MAAM;gBACnC,UAAU,EAAE,IAAI,CAAC,UAAU;aAC3B,EACD,IAAI,EACJ,CAAC,CACD,IAAI,CACL,CAAA;YAED,IAAI,IAAI,CAAC,OAAO,KAAK,IAAI;gBAAE,GAAG,CAAC,QAAQ,GAAG,CAAC,CAAA;YAC3C,OAAO,IAAI,CAAC,QAAQ,CAAA;QACrB,CAAC;KACD,CAAC;CACF,CAAC,CAAA;AAEF,SAAS,WAAW,CAAC,IAAY;IAChC,OAAO,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC,KAAK,CAAC,GAAG,EAAE;QACxC,MAAM,IAAI,QAAQ,CAAC,0BAA0B,IAAI,GAAG,CAAC,CAAA;IACtD,CAAC,CAAC,CAAA;AACH,CAAC;AAED,0DAA0D;AAC1D,MAAM,CAAC,KAAK,UAAU,IAAI,CAAC,IAAuB,EAAE,EAAU;IAC7D,MAAM,IAAI,GAAG,MAAM,GAAG,CAAC,GAAG,CAAC,IAAI,EAAE,EAAE,CAAC,CAAA;IACpC,OAAO,IAAI,KAAK,CAAC,IAAI,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,GAAG,CAAC,UAAU,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAA;AACzE,CAAC;AAED,IAAI,MAAM,CAAC,IAAI,CAAC,IAAI;IAAE,OAAO,CAAC,QAAQ,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAA","sourcesContent":["#!/usr/bin/env bun\n/**\n * Fetch a public URL as markdown and print the result. Page metadata goes to\n * stderr; markdown goes to stdout.\n */\nimport { readFile } from 'node:fs/promises'\nimport { parseEnv } from 'node:util'\nimport {\n\ttype CliIO,\n\tCliError,\n\tCliUsageError,\n\tdefineCli,\n\tdefineCommand\n} from 'hanzio/cli'\nimport { z } from 'zod'\nimport { createWebsiteToMarkdown } from './fetch-page.js'\n\nexport const cli = defineCli({\n\tname: 'website-to-markdown',\n\t// No URL prints the help and exits 1, as 1.0 did.\n\tonError: (error, io) => {\n\t\tif (\n\t\t\terror instanceof CliUsageError &&\n\t\t\terror.message.split('\\n').some((line) => line.startsWith('<url>: '))\n\t\t) {\n\t\t\tio.stdout.write(error.help)\n\t\t\treturn 1\n\t\t}\n\t},\n\tcommands: defineCommand({\n\t\t// Unknown flags and extra positionals are ignored, as in 1.0.\n\t\tstrict: false,\n\t\tdescription:\n\t\t\t'Fetch a public URL as markdown. Needs CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID.',\n\t\tinput: z.object({\n\t\t\turl: z\n\t\t\t\t.string()\n\t\t\t\t.describe('e.g. https://example.com/about')\n\t\t\t\t.meta({ positional: true }),\n\t\t\trender: z\n\t\t\t\t.boolean()\n\t\t\t\t.default(true)\n\t\t\t\t.describe(\n\t\t\t\t\t'Use the Browser Rendering tier; --no-render keeps to HTML tiers.'\n\t\t\t\t),\n\t\t\tenvFile: z\n\t\t\t\t.string()\n\t\t\t\t.optional()\n\t\t\t\t.describe('Read CLOUDFLARE_* from a dotenv file; the environment wins.')\n\t\t}),\n\t\trun: async ({ url, render, envFile }, ctx) => {\n\t\t\tconst file = envFile ? parseEnv(await readEnvFile(envFile)) : {}\n\t\t\tconst CLOUDFLARE_API_TOKEN =\n\t\t\t\tctx.env.CLOUDFLARE_API_TOKEN || file.CLOUDFLARE_API_TOKEN\n\t\t\tconst CLOUDFLARE_ACCOUNT_ID =\n\t\t\t\tctx.env.CLOUDFLARE_ACCOUNT_ID || file.CLOUDFLARE_ACCOUNT_ID\n\t\t\tif (!CLOUDFLARE_API_TOKEN || !CLOUDFLARE_ACCOUNT_ID)\n\t\t\t\tthrow new CliError(\n\t\t\t\t\t'Set CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID.'\n\t\t\t\t)\n\n\t\t\tconst { fetchPageAsMarkdown } = createWebsiteToMarkdown({\n\t\t\t\tcloudflare: {\n\t\t\t\t\tapiToken: CLOUDFLARE_API_TOKEN,\n\t\t\t\t\taccountId: CLOUDFLARE_ACCOUNT_ID\n\t\t\t\t}\n\t\t\t})\n\n\t\t\tctx.stderr.write(`Fetching ${url}…\\n`)\n\n\t\t\tconst page = await fetchPageAsMarkdown({ url, render })\n\n\t\t\tctx.stderr.write(\n\t\t\t\t`${JSON.stringify(\n\t\t\t\t\t{\n\t\t\t\t\t\toutcome: page.outcome,\n\t\t\t\t\t\tstatus: page.status,\n\t\t\t\t\t\tsource: page.source,\n\t\t\t\t\t\turl: page.url,\n\t\t\t\t\t\tfinalUrl: page.finalUrl,\n\t\t\t\t\t\tcontentType: page.contentType,\n\t\t\t\t\t\tetag: page.etag,\n\t\t\t\t\t\tlastModified: page.lastModified,\n\t\t\t\t\t\ttitle: page.title,\n\t\t\t\t\t\tlang: page.lang,\n\t\t\t\t\t\tcanonical: page.canonical,\n\t\t\t\t\t\tmetaDescription: page.metaDescription,\n\t\t\t\t\t\tjsonLdBlocks: page.jsonLd.length,\n\t\t\t\t\t\tlinks: page.links.length,\n\t\t\t\t\t\timages: page.images.length,\n\t\t\t\t\t\tmarkdownChars: page.markdown.length,\n\t\t\t\t\t\tdurationMs: page.durationMs\n\t\t\t\t\t},\n\t\t\t\t\tnull,\n\t\t\t\t\t2\n\t\t\t\t)}\\n`\n\t\t\t)\n\n\t\t\tif (page.outcome !== 'ok') ctx.exitCode = 1\n\t\t\treturn page.markdown\n\t\t}\n\t})\n})\n\nfunction readEnvFile(path: string) {\n\treturn readFile(path, 'utf8').catch(() => {\n\t\tthrow new CliError(`Cannot read --env-file ${path}.`)\n\t})\n}\n\n/** The bin. `--help` without a URL exits 1, as in 1.0. */\nexport async function main(argv: readonly string[], io?: CliIO) {\n\tconst code = await cli.run(argv, io)\n\treturn code === 0 && argv.every((arg) => arg.startsWith('-')) ? 1 : code\n}\n\nif (import.meta.main) process.exitCode = await main(process.argv.slice(2))\n"]}
+{"version":3,"file":"cli.js","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AACA;;;GAGG;AACH,OAAO,EAAE,QAAQ,EAAE,MAAM,kBAAkB,CAAA;AAC3C,OAAO,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAA;AACpC,OAAO,EAEN,QAAQ,EACR,aAAa,EACb,SAAS,EACT,aAAa,EACb,MAAM,YAAY,CAAA;AACnB,OAAO,EAAE,CAAC,EAAE,MAAM,KAAK,CAAA;AACvB,OAAO,EAAE,uBAAuB,EAAE,MAAM,iBAAiB,CAAA;AAEzD,MAAM,CAAC,MAAM,GAAG,GAAG,SAAS,CAAC;IAC5B,IAAI,EAAE,qBAAqB;IAC3B,kDAAkD;IAClD,OAAO,EAAE,CAAC,KAAK,EAAE,EAAE,EAAE,EAAE;QACtB,IACC,KAAK,YAAY,aAAa;YAC9B,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC,EACnE,CAAC;YACF,EAAE,CAAC,MAAM,CAAC,KAAK,CAAC,KAAK,CAAC,IAAI,CAAC,CAAA;YAC3B,OAAO,CAAC,CAAA;QACT,CAAC;IACF,CAAC;IACD,QAAQ,EAAE,aAAa,CAAC;QACvB,8DAA8D;QAC9D,MAAM,EAAE,KAAK;QACb,WAAW,EACV,uFAAuF;QACxF,KAAK,EAAE,CAAC,CAAC,MAAM,CAAC;YACf,GAAG,EAAE,CAAC;iBACJ,MAAM,EAAE;iBACR,QAAQ,CAAC,gCAAgC,CAAC;iBAC1C,IAAI,CAAC,EAAE,UAAU,EAAE,IAAI,EAAE,CAAC;YAC5B,MAAM,EAAE,CAAC;iBACP,OAAO,EAAE;iBACT,OAAO,CAAC,IAAI,CAAC;iBACb,QAAQ,CACR,kEAAkE,CAClE;YACF,OAAO,EAAE,CAAC;iBACR,MAAM,EAAE;iBACR,QAAQ,EAAE;iBACV,QAAQ,CAAC,6DAA6D,CAAC;SACzE,CAAC;QACF,GAAG,EAAE,KAAK,EAAE,EAAE,GAAG,EAAE,MAAM,EAAE,OAAO,EAAE,EAAE,GAAG,EAAE,EAAE;YAC5C,MAAM,IAAI,GAAG,OAAO,CAAC,CAAC,CAAC,QAAQ,CAAC,MAAM,WAAW,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,CAAA;YAChE,MAAM,oBAAoB,GACzB,GAAG,CAAC,GAAG,CAAC,oBAAoB,IAAI,IAAI,CAAC,oBAAoB,CAAA;YAC1D,MAAM,qBAAqB,GAC1B,GAAG,CAAC,GAAG,CAAC,qBAAqB,IAAI,IAAI,CAAC,qBAAqB,CAAA;YAC5D,IAAI,CAAC,oBAAoB,IAAI,CAAC,qBAAqB;gBAClD,MAAM,IAAI,QAAQ,CACjB,qDAAqD,CACrD,CAAA;YAEF,MAAM,EAAE,mBAAmB,EAAE,GAAG,uBAAuB,CAAC;gBACvD,UAAU,EAAE;oBACX,QAAQ,EAAE,oBAAoB;oBAC9B,SAAS,EAAE,qBAAqB;iBAChC;gBACD,sEAAsE;gBACtE,cAAc,EAAE,IAAI;aACpB,CAAC,CAAA;YAEF,GAAG,CAAC,MAAM,CAAC,KAAK,CAAC,YAAY,GAAG,KAAK,CAAC,CAAA;YAEtC,MAAM,IAAI,GAAG,MAAM,mBAAmB,CAAC,EAAE,GAAG,EAAE,MAAM,EAAE,CAAC,CAAA;YAEvD,GAAG,CAAC,MAAM,CAAC,KAAK,CACf,GAAG,IAAI,CAAC,SAAS,CAChB;gBACC,OAAO,EAAE,IAAI,CAAC,OAAO;gBACrB,MAAM,EAAE,IAAI,CAAC,MAAM;gBACnB,MAAM,EAAE,IAAI,CAAC,MAAM;gBACnB,GAAG,EAAE,IAAI,CAAC,GAAG;gBACb,QAAQ,EAAE,IAAI,CAAC,QAAQ;gBACvB,WAAW,EAAE,IAAI,CAAC,WAAW;gBAC7B,IAAI,EAAE,IAAI,CAAC,IAAI;gBACf,YAAY,EAAE,IAAI,CAAC,YAAY;gBAC/B,KAAK,EAAE,IAAI,CAAC,KAAK;gBACjB,IAAI,EAAE,IAAI,CAAC,IAAI;gBACf,SAAS,EAAE,IAAI,CAAC,SAAS;gBACzB,eAAe,EAAE,IAAI,CAAC,eAAe;gBACrC,YAAY,EAAE,IAAI,CAAC,MAAM,CAAC,MAAM;gBAChC,KAAK,EAAE,IAAI,CAAC,KAAK,CAAC,MAAM;gBACxB,MAAM,EAAE,IAAI,CAAC,MAAM,CAAC,MAAM;gBAC1B,aAAa,EAAE,IAAI,CAAC,QAAQ,CAAC,MAAM;gBACnC,UAAU,EAAE,IAAI,CAAC,UAAU;aAC3B,EACD,IAAI,EACJ,CAAC,CACD,IAAI,CACL,CAAA;YAED,IAAI,IAAI,CAAC,OAAO,KAAK,IAAI;gBAAE,GAAG,CAAC,QAAQ,GAAG,CAAC,CAAA;YAC3C,OAAO,IAAI,CAAC,QAAQ,CAAA;QACrB,CAAC;KACD,CAAC;CACF,CAAC,CAAA;AAEF,SAAS,WAAW,CAAC,IAAY;IAChC,OAAO,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC,KAAK,CAAC,GAAG,EAAE;QACxC,MAAM,IAAI,QAAQ,CAAC,0BAA0B,IAAI,GAAG,CAAC,CAAA;IACtD,CAAC,CAAC,CAAA;AACH,CAAC;AAED,0DAA0D;AAC1D,MAAM,CAAC,KAAK,UAAU,IAAI,CAAC,IAAuB,EAAE,EAAU;IAC7D,MAAM,IAAI,GAAG,MAAM,GAAG,CAAC,GAAG,CAAC,IAAI,EAAE,EAAE,CAAC,CAAA;IACpC,OAAO,IAAI,KAAK,CAAC,IAAI,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,GAAG,CAAC,UAAU,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAA;AACzE,CAAC;AAED,IAAI,MAAM,CAAC,IAAI,CAAC,IAAI;IAAE,OAAO,CAAC,QAAQ,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAA","sourcesContent":["#!/usr/bin/env bun\n/**\n * Fetch a public URL as markdown and print the result. Page metadata goes to\n * stderr; markdown goes to stdout.\n */\nimport { readFile } from 'node:fs/promises'\nimport { parseEnv } from 'node:util'\nimport {\n\ttype CliIO,\n\tCliError,\n\tCliUsageError,\n\tdefineCli,\n\tdefineCommand\n} from 'hanzio/cli'\nimport { z } from 'zod'\nimport { createWebsiteToMarkdown } from './fetch-page.js'\n\nexport const cli = defineCli({\n\tname: 'website-to-markdown',\n\t// No URL prints the help and exits 1, as 1.0 did.\n\tonError: (error, io) => {\n\t\tif (\n\t\t\terror instanceof CliUsageError &&\n\t\t\terror.message.split('\\n').some((line) => line.startsWith('<url>: '))\n\t\t) {\n\t\t\tio.stdout.write(error.help)\n\t\t\treturn 1\n\t\t}\n\t},\n\tcommands: defineCommand({\n\t\t// Unknown flags and extra positionals are ignored, as in 1.0.\n\t\tstrict: false,\n\t\tdescription:\n\t\t\t'Fetch a public URL as markdown. Needs CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID.',\n\t\tinput: z.object({\n\t\t\turl: z\n\t\t\t\t.string()\n\t\t\t\t.describe('e.g. https://example.com/about')\n\t\t\t\t.meta({ positional: true }),\n\t\t\trender: z\n\t\t\t\t.boolean()\n\t\t\t\t.default(true)\n\t\t\t\t.describe(\n\t\t\t\t\t'Use the Browser Rendering tier; --no-render keeps to HTML tiers.'\n\t\t\t\t),\n\t\t\tenvFile: z\n\t\t\t\t.string()\n\t\t\t\t.optional()\n\t\t\t\t.describe('Read CLOUDFLARE_* from a dotenv file; the environment wins.')\n\t\t}),\n\t\trun: async ({ url, render, envFile }, ctx) => {\n\t\t\tconst file = envFile ? parseEnv(await readEnvFile(envFile)) : {}\n\t\t\tconst CLOUDFLARE_API_TOKEN =\n\t\t\t\tctx.env.CLOUDFLARE_API_TOKEN || file.CLOUDFLARE_API_TOKEN\n\t\t\tconst CLOUDFLARE_ACCOUNT_ID =\n\t\t\t\tctx.env.CLOUDFLARE_ACCOUNT_ID || file.CLOUDFLARE_ACCOUNT_ID\n\t\t\tif (!CLOUDFLARE_API_TOKEN || !CLOUDFLARE_ACCOUNT_ID)\n\t\t\t\tthrow new CliError(\n\t\t\t\t\t'Set CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID.'\n\t\t\t\t)\n\n\t\t\tconst { fetchPageAsMarkdown } = createWebsiteToMarkdown({\n\t\t\t\tcloudflare: {\n\t\t\t\t\tapiToken: CLOUDFLARE_API_TOKEN,\n\t\t\t\t\taccountId: CLOUDFLARE_ACCOUNT_ID\n\t\t\t\t},\n\t\t\t\t// Run by a person on their own machine, so a dev server is fair game.\n\t\t\t\tallowLocalhost: true\n\t\t\t})\n\n\t\t\tctx.stderr.write(`Fetching ${url}…\\n`)\n\n\t\t\tconst page = await fetchPageAsMarkdown({ url, render })\n\n\t\t\tctx.stderr.write(\n\t\t\t\t`${JSON.stringify(\n\t\t\t\t\t{\n\t\t\t\t\t\toutcome: page.outcome,\n\t\t\t\t\t\tstatus: page.status,\n\t\t\t\t\t\tsource: page.source,\n\t\t\t\t\t\turl: page.url,\n\t\t\t\t\t\tfinalUrl: page.finalUrl,\n\t\t\t\t\t\tcontentType: page.contentType,\n\t\t\t\t\t\tetag: page.etag,\n\t\t\t\t\t\tlastModified: page.lastModified,\n\t\t\t\t\t\ttitle: page.title,\n\t\t\t\t\t\tlang: page.lang,\n\t\t\t\t\t\tcanonical: page.canonical,\n\t\t\t\t\t\tmetaDescription: page.metaDescription,\n\t\t\t\t\t\tjsonLdBlocks: page.jsonLd.length,\n\t\t\t\t\t\tlinks: page.links.length,\n\t\t\t\t\t\timages: page.images.length,\n\t\t\t\t\t\tmarkdownChars: page.markdown.length,\n\t\t\t\t\t\tdurationMs: page.durationMs\n\t\t\t\t\t},\n\t\t\t\t\tnull,\n\t\t\t\t\t2\n\t\t\t\t)}\\n`\n\t\t\t)\n\n\t\t\tif (page.outcome !== 'ok') ctx.exitCode = 1\n\t\t\treturn page.markdown\n\t\t}\n\t})\n})\n\nfunction readEnvFile(path: string) {\n\treturn readFile(path, 'utf8').catch(() => {\n\t\tthrow new CliError(`Cannot read --env-file ${path}.`)\n\t})\n}\n\n/** The bin. `--help` without a URL exits 1, as in 1.0. */\nexport async function main(argv: readonly string[], io?: CliIO) {\n\tconst code = await cli.run(argv, io)\n\treturn code === 0 && argv.every((arg) => arg.startsWith('-')) ? 1 : code\n}\n\nif (import.meta.main) process.exitCode = await main(process.argv.slice(2))\n"]}

package/dist/content-type.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"content-type.d.ts","sourceRoot":"","sources":["../src/content-type.ts"],"names":[],"mappings":"AA2DA,kFAAkF;AAClF,wBAAgB,uBAAuB,CAAC,WAAW,EAAE,MAAM,GAAG,IAAI;;;SAIjE;AAED,MAAM,MAAM,WAAW,GAAG,UAAU,GAAG,MAAM,GAAG,MAAM,GAAG,UAAU,GAAG,OAAO,CAAA;AAE7E,iFAAiF;AACjF,wBAAgB,mBAAmB,CAAC,WAAW,EAAE,MAAM,GAAG,IAAI,GAAG,WAAW,CAO3E"}
+{"version":3,"file":"content-type.d.ts","sourceRoot":"","sources":["../src/content-type.ts"],"names":[],"mappings":"AA6DA,kFAAkF;AAClF,wBAAgB,uBAAuB,CAAC,WAAW,EAAE,MAAM,GAAG,IAAI;;;SAIjE;AAED,MAAM,MAAM,WAAW,GAAG,UAAU,GAAG,MAAM,GAAG,MAAM,GAAG,UAAU,GAAG,OAAO,CAAA;AAE7E,iFAAiF;AACjF,wBAAgB,mBAAmB,CAAC,WAAW,EAAE,MAAM,GAAG,IAAI,GAAG,WAAW,CAO3E"}

package/dist/content-type.js

@@ -1,15 +1,16 @@
-const MARKDOWN_CONTENT_TYPES = new Set([
+import { isOneOf } from 'hanzio';
+const MARKDOWN_CONTENT_TYPES = [
     'text/markdown',
     'text/x-markdown',
     'application/markdown'
-]);
-const HTML_CONTENT_TYPES = new Set(['text/html', 'application/xhtml+xml']);
-const TEXTUAL_CONTENT_TYPES = new Set([
+];
+const HTML_CONTENT_TYPES = ['text/html', 'application/xhtml+xml'];
+const TEXTUAL_CONTENT_TYPES = [
     'application/xml',
     'application/json',
     'application/rss+xml',
     'application/atom+xml'
-]);
+];
 /**
  * The formats Workers AI `toMarkdown` accepts as documents (its `/supported`
  * list, read 2026-09-08). Images are accepted there too but are deliberately
@@ -44,7 +45,7 @@ function normalizeContentType(contentType) {
 function isTextualContentType(type) {
     return (type === '' ||
         type.startsWith('text/') ||
-        TEXTUAL_CONTENT_TYPES.has(type) ||
+        isOneOf(TEXTUAL_CONTENT_TYPES, type) ||
         // `application/ld+json`, `application/rdf+xml`, … but not `image/svg+xml`.
         (type.startsWith('application/') &&
             (type.endsWith('+json') || type.endsWith('+xml'))));
@@ -58,9 +59,9 @@ export function convertibleDocumentType(contentType) {
 /** What the fetch layer should do with a body, decided from the header alone. */
 export function classifyContentType(contentType) {
     const type = normalizeContentType(contentType);
-    if (MARKDOWN_CONTENT_TYPES.has(type))
+    if (isOneOf(MARKDOWN_CONTENT_TYPES, type))
         return 'markdown';
-    if (HTML_CONTENT_TYPES.has(type))
+    if (isOneOf(HTML_CONTENT_TYPES, type))
         return 'html';
     if (CONVERTIBLE_DOCUMENT_EXTENSIONS.has(type))
         return 'document';

package/dist/content-type.js.map

@@ -1 +1 @@
-{"version":3,"file":"content-type.js","sourceRoot":"","sources":["../src/content-type.ts"],"names":[],"mappings":"AAAA,MAAM,sBAAsB,GAAG,IAAI,GAAG,CAAC;IACtC,eAAe;IACf,iBAAiB;IACjB,sBAAsB;CACtB,CAAC,CAAA;AAEF,MAAM,kBAAkB,GAAG,IAAI,GAAG,CAAC,CAAC,WAAW,EAAE,uBAAuB,CAAC,CAAC,CAAA;AAE1E,MAAM,qBAAqB,GAAG,IAAI,GAAG,CAAC;IACrC,iBAAiB;IACjB,kBAAkB;IAClB,qBAAqB;IACrB,sBAAsB;CACtB,CAAC,CAAA;AAEF;;;;;GAKG;AACH,MAAM,+BAA+B,GAAG,IAAI,GAAG,CAAC;IAC/C,CAAC,iBAAiB,EAAE,KAAK,CAAC;IAC1B;QACC,yEAAyE;QACzE,MAAM;KACN;IACD,CAAC,yCAAyC,EAAE,KAAK,CAAC;IAClD,CAAC,mEAAmE,EAAE,MAAM,CAAC;IAC7E,CAAC,gDAAgD,EAAE,MAAM,CAAC;IAC1D,CAAC,uDAAuD,EAAE,MAAM,CAAC;IACjE,CAAC,0BAA0B,EAAE,KAAK,CAAC;IACnC,CAAC,gDAAgD,EAAE,KAAK,CAAC;IACzD;QACC,2EAA2E;QAC3E,MAAM;KACN;IACD;QACC,wEAAwE;QACxE,MAAM;KACN;IACD,CAAC,iDAAiD,EAAE,KAAK,CAAC;CAC1D,CAAC,CAAA;AAEF,SAAS,oBAAoB,CAAC,WAAsC;IACnE,OAAO,WAAW,EAAE,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,CAAC,WAAW,EAAE,IAAI,EAAE,CAAA;AAC9D,CAAC;AAED,SAAS,oBAAoB,CAAC,IAAY;IACzC,OAAO,CACN,IAAI,KAAK,EAAE;QACX,IAAI,CAAC,UAAU,CAAC,OAAO,CAAC;QACxB,qBAAqB,CAAC,GAAG,CAAC,IAAI,CAAC;QAC/B,2EAA2E;QAC3E,CAAC,IAAI,CAAC,UAAU,CAAC,cAAc,CAAC;YAC/B,CAAC,IAAI,CAAC,QAAQ,CAAC,OAAO,CAAC,IAAI,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC,CACnD,CAAA;AACF,CAAC;AAED,kFAAkF;AAClF,MAAM,UAAU,uBAAuB,CAAC,WAA0B;IACjE,MAAM,QAAQ,GAAG,oBAAoB,CAAC,WAAW,CAAC,CAAA;IAClD,MAAM,SAAS,GAAG,+BAA+B,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAA;IAC/D,OAAO,SAAS,CAAC,CAAC,CAAC,EAAE,SAAS,EAAE,QAAQ,EAAE,CAAC,CAAC,CAAC,IAAI,CAAA;AAClD,CAAC;AAID,iFAAiF;AACjF,MAAM,UAAU,mBAAmB,CAAC,WAA0B;IAC7D,MAAM,IAAI,GAAG,oBAAoB,CAAC,WAAW,CAAC,CAAA;IAC9C,IAAI,sBAAsB,CAAC,GAAG,CAAC,IAAI,CAAC;QAAE,OAAO,UAAU,CAAA;IACvD,IAAI,kBAAkB,CAAC,GAAG,CAAC,IAAI,CAAC;QAAE,OAAO,MAAM,CAAA;IAC/C,IAAI,+BAA+B,CAAC,GAAG,CAAC,IAAI,CAAC;QAAE,OAAO,UAAU,CAAA;IAChE,IAAI,oBAAoB,CAAC,IAAI,CAAC;QAAE,OAAO,MAAM,CAAA;IAC7C,OAAO,OAAO,CAAA;AACf,CAAC","sourcesContent":["const MARKDOWN_CONTENT_TYPES = new Set([\n\t'text/markdown',\n\t'text/x-markdown',\n\t'application/markdown'\n])\n\nconst HTML_CONTENT_TYPES = new Set(['text/html', 'application/xhtml+xml'])\n\nconst TEXTUAL_CONTENT_TYPES = new Set([\n\t'application/xml',\n\t'application/json',\n\t'application/rss+xml',\n\t'application/atom+xml'\n])\n\n/**\n * The formats Workers AI `toMarkdown` accepts as documents (its `/supported`\n * list, read 2026-09-08). Images are accepted there too but are deliberately\n * absent: this list decides what the CRAWLER downloads and converts, and a\n * site's gallery must stay an asset. Uploads accept images on their own list.\n */\nconst CONVERTIBLE_DOCUMENT_EXTENSIONS = new Map([\n\t['application/pdf', 'pdf'],\n\t[\n\t\t'application/vnd.openxmlformats-officedocument.wordprocessingml.document',\n\t\t'docx'\n\t],\n\t['application/vnd.oasis.opendocument.text', 'odt'],\n\t['application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', 'xlsx'],\n\t['application/vnd.ms-excel.sheet.macroenabled.12', 'xlsm'],\n\t['application/vnd.ms-excel.sheet.binary.macroenabled.12', 'xlsb'],\n\t['application/vnd.ms-excel', 'xls'],\n\t['application/vnd.oasis.opendocument.spreadsheet', 'ods'],\n\t[\n\t\t'application/vnd.openxmlformats-officedocument.presentationml.presentation',\n\t\t'pptx'\n\t],\n\t[\n\t\t'application/vnd.openxmlformats-officedocument.presentationml.slideshow',\n\t\t'ppsx'\n\t],\n\t['application/vnd.oasis.opendocument.presentation', 'odp']\n])\n\nfunction normalizeContentType(contentType: string | null | undefined) {\n\treturn contentType?.split(';')[0]?.trim().toLowerCase() ?? ''\n}\n\nfunction isTextualContentType(type: string) {\n\treturn (\n\t\ttype === '' ||\n\t\ttype.startsWith('text/') ||\n\t\tTEXTUAL_CONTENT_TYPES.has(type) ||\n\t\t// `application/ld+json`, `application/rdf+xml`, … but not `image/svg+xml`.\n\t\t(type.startsWith('application/') &&\n\t\t\t(type.endsWith('+json') || type.endsWith('+xml')))\n\t)\n}\n\n/** The upload descriptor for a binary we can convert, or null for text/assets. */\nexport function convertibleDocumentType(contentType: string | null) {\n\tconst mimeType = normalizeContentType(contentType)\n\tconst extension = CONVERTIBLE_DOCUMENT_EXTENSIONS.get(mimeType)\n\treturn extension ? { extension, mimeType } : null\n}\n\nexport type ContentKind = 'markdown' | 'html' | 'text' | 'document' | 'asset'\n\n/** What the fetch layer should do with a body, decided from the header alone. */\nexport function classifyContentType(contentType: string | null): ContentKind {\n\tconst type = normalizeContentType(contentType)\n\tif (MARKDOWN_CONTENT_TYPES.has(type)) return 'markdown'\n\tif (HTML_CONTENT_TYPES.has(type)) return 'html'\n\tif (CONVERTIBLE_DOCUMENT_EXTENSIONS.has(type)) return 'document'\n\tif (isTextualContentType(type)) return 'text'\n\treturn 'asset'\n}\n"]}
+{"version":3,"file":"content-type.js","sourceRoot":"","sources":["../src/content-type.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,OAAO,EAAiB,MAAM,QAAQ,CAAA;AAE/C,MAAM,sBAAsB,GAAG;IAC9B,eAAe;IACf,iBAAiB;IACjB,sBAAsB;CACb,CAAA;AAEV,MAAM,kBAAkB,GAAG,CAAC,WAAW,EAAE,uBAAuB,CAAU,CAAA;AAE1E,MAAM,qBAAqB,GAAG;IAC7B,iBAAiB;IACjB,kBAAkB;IAClB,qBAAqB;IACrB,sBAAsB;CACb,CAAA;AAEV;;;;;GAKG;AACH,MAAM,+BAA+B,GAAG,IAAI,GAAG,CAAC;IAC/C,CAAC,iBAAiB,EAAE,KAAK,CAAC;IAC1B;QACC,yEAAyE;QACzE,MAAM;KACN;IACD,CAAC,yCAAyC,EAAE,KAAK,CAAC;IAClD,CAAC,mEAAmE,EAAE,MAAM,CAAC;IAC7E,CAAC,gDAAgD,EAAE,MAAM,CAAC;IAC1D,CAAC,uDAAuD,EAAE,MAAM,CAAC;IACjE,CAAC,0BAA0B,EAAE,KAAK,CAAC;IACnC,CAAC,gDAAgD,EAAE,KAAK,CAAC;IACzD;QACC,2EAA2E;QAC3E,MAAM;KACN;IACD;QACC,wEAAwE;QACxE,MAAM;KACN;IACD,CAAC,iDAAiD,EAAE,KAAK,CAAC;CAC1D,CAAC,CAAA;AAEF,SAAS,oBAAoB,CAAC,WAA6B;IAC1D,OAAO,WAAW,EAAE,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,CAAC,WAAW,EAAE,IAAI,EAAE,CAAA;AAC9D,CAAC;AAED,SAAS,oBAAoB,CAAC,IAAY;IACzC,OAAO,CACN,IAAI,KAAK,EAAE;QACX,IAAI,CAAC,UAAU,CAAC,OAAO,CAAC;QACxB,OAAO,CAAC,qBAAqB,EAAE,IAAI,CAAC;QACpC,2EAA2E;QAC3E,CAAC,IAAI,CAAC,UAAU,CAAC,cAAc,CAAC;YAC/B,CAAC,IAAI,CAAC,QAAQ,CAAC,OAAO,CAAC,IAAI,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC,CACnD,CAAA;AACF,CAAC;AAED,kFAAkF;AAClF,MAAM,UAAU,uBAAuB,CAAC,WAA0B;IACjE,MAAM,QAAQ,GAAG,oBAAoB,CAAC,WAAW,CAAC,CAAA;IAClD,MAAM,SAAS,GAAG,+BAA+B,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAA;IAC/D,OAAO,SAAS,CAAC,CAAC,CAAC,EAAE,SAAS,EAAE,QAAQ,EAAE,CAAC,CAAC,CAAC,IAAI,CAAA;AAClD,CAAC;AAID,iFAAiF;AACjF,MAAM,UAAU,mBAAmB,CAAC,WAA0B;IAC7D,MAAM,IAAI,GAAG,oBAAoB,CAAC,WAAW,CAAC,CAAA;IAC9C,IAAI,OAAO,CAAC,sBAAsB,EAAE,IAAI,CAAC;QAAE,OAAO,UAAU,CAAA;IAC5D,IAAI,OAAO,CAAC,kBAAkB,EAAE,IAAI,CAAC;QAAE,OAAO,MAAM,CAAA;IACpD,IAAI,+BAA+B,CAAC,GAAG,CAAC,IAAI,CAAC;QAAE,OAAO,UAAU,CAAA;IAChE,IAAI,oBAAoB,CAAC,IAAI,CAAC;QAAE,OAAO,MAAM,CAAA;IAC7C,OAAO,OAAO,CAAA;AACf,CAAC","sourcesContent":["import { isOneOf, type Nullable } from 'hanzio'\n\nconst MARKDOWN_CONTENT_TYPES = [\n\t'text/markdown',\n\t'text/x-markdown',\n\t'application/markdown'\n] as const\n\nconst HTML_CONTENT_TYPES = ['text/html', 'application/xhtml+xml'] as const\n\nconst TEXTUAL_CONTENT_TYPES = [\n\t'application/xml',\n\t'application/json',\n\t'application/rss+xml',\n\t'application/atom+xml'\n] as const\n\n/**\n * The formats Workers AI `toMarkdown` accepts as documents (its `/supported`\n * list, read 2026-09-08). Images are accepted there too but are deliberately\n * absent: this list decides what the CRAWLER downloads and converts, and a\n * site's gallery must stay an asset. Uploads accept images on their own list.\n */\nconst CONVERTIBLE_DOCUMENT_EXTENSIONS = new Map([\n\t['application/pdf', 'pdf'],\n\t[\n\t\t'application/vnd.openxmlformats-officedocument.wordprocessingml.document',\n\t\t'docx'\n\t],\n\t['application/vnd.oasis.opendocument.text', 'odt'],\n\t['application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', 'xlsx'],\n\t['application/vnd.ms-excel.sheet.macroenabled.12', 'xlsm'],\n\t['application/vnd.ms-excel.sheet.binary.macroenabled.12', 'xlsb'],\n\t['application/vnd.ms-excel', 'xls'],\n\t['application/vnd.oasis.opendocument.spreadsheet', 'ods'],\n\t[\n\t\t'application/vnd.openxmlformats-officedocument.presentationml.presentation',\n\t\t'pptx'\n\t],\n\t[\n\t\t'application/vnd.openxmlformats-officedocument.presentationml.slideshow',\n\t\t'ppsx'\n\t],\n\t['application/vnd.oasis.opendocument.presentation', 'odp']\n])\n\nfunction normalizeContentType(contentType: Nullable<string>) {\n\treturn contentType?.split(';')[0]?.trim().toLowerCase() ?? ''\n}\n\nfunction isTextualContentType(type: string) {\n\treturn (\n\t\ttype === '' ||\n\t\ttype.startsWith('text/') ||\n\t\tisOneOf(TEXTUAL_CONTENT_TYPES, type) ||\n\t\t// `application/ld+json`, `application/rdf+xml`, … but not `image/svg+xml`.\n\t\t(type.startsWith('application/') &&\n\t\t\t(type.endsWith('+json') || type.endsWith('+xml')))\n\t)\n}\n\n/** The upload descriptor for a binary we can convert, or null for text/assets. */\nexport function convertibleDocumentType(contentType: string | null) {\n\tconst mimeType = normalizeContentType(contentType)\n\tconst extension = CONVERTIBLE_DOCUMENT_EXTENSIONS.get(mimeType)\n\treturn extension ? { extension, mimeType } : null\n}\n\nexport type ContentKind = 'markdown' | 'html' | 'text' | 'document' | 'asset'\n\n/** What the fetch layer should do with a body, decided from the header alone. */\nexport function classifyContentType(contentType: string | null): ContentKind {\n\tconst type = normalizeContentType(contentType)\n\tif (isOneOf(MARKDOWN_CONTENT_TYPES, type)) return 'markdown'\n\tif (isOneOf(HTML_CONTENT_TYPES, type)) return 'html'\n\tif (CONVERTIBLE_DOCUMENT_EXTENSIONS.has(type)) return 'document'\n\tif (isTextualContentType(type)) return 'text'\n\treturn 'asset'\n}\n"]}

package/dist/crawl-snapshot.d.ts

@@ -1,3 +1,4 @@
+import { type Nullable } from 'hanzio';
 import type { Logger } from './logger.js';
 import { CRAWL_SNAPSHOT_NAME_PATTERN, type CrawlSnapshotInput } from './schema/index.js';
 export { CRAWL_SNAPSHOT_NAME_PATTERN, type CrawlSnapshotInput };
@@ -7,7 +8,7 @@ export type CrawlSnapshotOptions = {
     /** Told about replay misses. */
     logger?: Logger | undefined;
 };
-export declare function withCrawlSnapshot<T>(input: CrawlSnapshotInput | null | undefined, run: () => Promise<T>, options: CrawlSnapshotOptions): Promise<T>;
+export declare function withCrawlSnapshot<T>(input: Nullable<CrawlSnapshotInput>, run: () => Promise<T>, options: CrawlSnapshotOptions): Promise<T>;
 /**
  * `read` through the active snapshot. Recording stores what the network
  * returned; replaying returns what was stored. A read the recording never made

package/dist/crawl-snapshot.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"crawl-snapshot.d.ts","sourceRoot":"","sources":["../src/crawl-snapshot.ts"],"names":[],"mappings":"AAIA,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,aAAa,CAAA;AACzC,OAAO,EACN,2BAA2B,EAC3B,KAAK,kBAAkB,EACvB,MAAM,mBAAmB,CAAA;AAE1B,OAAO,EAAE,2BAA2B,EAAE,KAAK,kBAAkB,EAAE,CAAA;AAsB/D,MAAM,MAAM,oBAAoB,GAAG;IAClC,0CAA0C;IAC1C,IAAI,EAAE,MAAM,CAAA;IACZ,gCAAgC;IAChC,MAAM,CAAC,EAAE,MAAM,GAAG,SAAS,CAAA;CAC3B,CAAA;AAID,wBAAgB,iBAAiB,CAAC,CAAC,EAClC,KAAK,EAAE,kBAAkB,GAAG,IAAI,GAAG,SAAS,EAC5C,GAAG,EAAE,MAAM,OAAO,CAAC,CAAC,CAAC,EACrB,OAAO,EAAE,oBAAoB,cAmB7B;AAED;;;;;GAKG;AACH,wBAAsB,oBAAoB,CAAC,CAAC,EAC3C,GAAG,EAAE,MAAM,EACX,IAAI,EAAE,MAAM,OAAO,CAAC,CAAC,CAAC,GACpB,OAAO,CAAC,CAAC,CAAC,CAeZ;AAED,8EAA8E;AAC9E,wBAAgB,oBAAoB,IAAI,kBAAkB,GAAG,IAAI,CAGhE"}
+{"version":3,"file":"crawl-snapshot.d.ts","sourceRoot":"","sources":["../src/crawl-snapshot.ts"],"names":[],"mappings":"AAGA,OAAO,EAAa,KAAK,QAAQ,EAAE,MAAM,QAAQ,CAAA;AAEjD,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,aAAa,CAAA;AACzC,OAAO,EACN,2BAA2B,EAC3B,KAAK,kBAAkB,EACvB,MAAM,mBAAmB,CAAA;AAE1B,OAAO,EAAE,2BAA2B,EAAE,KAAK,kBAAkB,EAAE,CAAA;AAsB/D,MAAM,MAAM,oBAAoB,GAAG;IAClC,0CAA0C;IAC1C,IAAI,EAAE,MAAM,CAAA;IACZ,gCAAgC;IAChC,MAAM,CAAC,EAAE,MAAM,GAAG,SAAS,CAAA;CAC3B,CAAA;AAID,wBAAgB,iBAAiB,CAAC,CAAC,EAClC,KAAK,EAAE,QAAQ,CAAC,kBAAkB,CAAC,EACnC,GAAG,EAAE,MAAM,OAAO,CAAC,CAAC,CAAC,EACrB,OAAO,EAAE,oBAAoB,cAqB7B;AAED;;;;;GAKG;AACH,wBAAsB,oBAAoB,CAAC,CAAC,EAC3C,GAAG,EAAE,MAAM,EACX,IAAI,EAAE,MAAM,OAAO,CAAC,CAAC,CAAC,GACpB,OAAO,CAAC,CAAC,CAAC,CAiBZ;AAUD,8EAA8E;AAC9E,wBAAgB,oBAAoB,IAAI,kBAAkB,GAAG,IAAI,CAGhE"}

package/dist/crawl-snapshot.js

@@ -1,6 +1,7 @@
 import { AsyncLocalStorage } from 'node:async_hooks';
 import { mkdir, readFile, writeFile } from 'node:fs/promises';
 import { resolve } from 'node:path';
+import { invariant } from 'hanzio';
 import { sha256Hex } from './hash.js';
 import { CRAWL_SNAPSHOT_NAME_PATTERN } from './schema/index.js';
 export { CRAWL_SNAPSHOT_NAME_PATTERN };
@@ -8,12 +9,8 @@ const snapshot = new AsyncLocalStorage();
 export function withCrawlSnapshot(input, run, options) {
     if (!input)
         return run();
-    if (process.env.NODE_ENV === 'production') {
-        throw new Error('A crawl snapshot is a local bench tool');
-    }
-    if (!CRAWL_SNAPSHOT_NAME_PATTERN.test(input.name)) {
-        throw new Error(`Invalid crawl snapshot name: ${input.name}`);
-    }
+    invariant(process.env.NODE_ENV !== 'production', 'A crawl snapshot is a local bench tool');
+    invariant(CRAWL_SNAPSHOT_NAME_PATTERN.test(input.name), `Invalid crawl snapshot name: ${input.name}`);
     return snapshot.run({
         mode: input.mode,
         name: input.name,
@@ -34,8 +31,10 @@ export async function throughCrawlSnapshot(key, read) {
     const file = resolve(active.directory, `${sha256Hex(key).slice(0, 32)}.json`);
     if (active.mode === 'replay') {
         const stored = await readFile(file, 'utf8').catch(() => null);
-        if (stored !== null)
-            return JSON.parse(stored).value;
+        // A file cut short by a killed recording is a miss, not a crash.
+        const parsed = stored === null ? null : parseStored(stored);
+        if (parsed)
+            return parsed.value;
         active.logger?.warn('crawl-snapshot.miss', { key });
     }
     const value = await read();
@@ -43,6 +42,14 @@ export async function throughCrawlSnapshot(key, read) {
     await writeFile(file, JSON.stringify({ key, value }));
     return value;
 }
+function parseStored(stored) {
+    try {
+        return JSON.parse(stored);
+    }
+    catch {
+        return null;
+    }
+}
 /** The snapshot this job reads or writes, to hand on to the job it queues. */
 export function currentCrawlSnapshot() {
     const active = snapshot.getStore();

package/dist/crawl-snapshot.js.map

@@ -1 +1 @@
-{"version":3,"file":"crawl-snapshot.js","sourceRoot":"","sources":["../src/crawl-snapshot.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,iBAAiB,EAAE,MAAM,kBAAkB,CAAA;AACpD,OAAO,EAAE,KAAK,EAAE,QAAQ,EAAE,SAAS,EAAE,MAAM,kBAAkB,CAAA;AAC7D,OAAO,EAAE,OAAO,EAAE,MAAM,WAAW,CAAA;AACnC,OAAO,EAAE,SAAS,EAAE,MAAM,WAAW,CAAA;AAErC,OAAO,EACN,2BAA2B,EAE3B,MAAM,mBAAmB,CAAA;AAE1B,OAAO,EAAE,2BAA2B,EAA2B,CAAA;AA6B/D,MAAM,QAAQ,GAAG,IAAI,iBAAiB,EAAiB,CAAA;AAEvD,MAAM,UAAU,iBAAiB,CAChC,KAA4C,EAC5C,GAAqB,EACrB,OAA6B;IAE7B,IAAI,CAAC,KAAK;QAAE,OAAO,GAAG,EAAE,CAAA;IACxB,IAAI,OAAO,CAAC,GAAG,CAAC,QAAQ,KAAK,YAAY,EAAE,CAAC;QAC3C,MAAM,IAAI,KAAK,CAAC,wCAAwC,CAAC,CAAA;IAC1D,CAAC;IACD,IAAI,CAAC,2BAA2B,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC;QACnD,MAAM,IAAI,KAAK,CAAC,gCAAgC,KAAK,CAAC,IAAI,EAAE,CAAC,CAAA;IAC9D,CAAC;IAED,OAAO,QAAQ,CAAC,GAAG,CAClB;QACC,IAAI,EAAE,KAAK,CAAC,IAAI;QAChB,IAAI,EAAE,KAAK,CAAC,IAAI;QAChB,SAAS,EAAE,OAAO,CAAC,OAAO,CAAC,IAAI,EAAE,KAAK,CAAC,IAAI,CAAC;QAC5C,MAAM,EAAE,OAAO,CAAC,MAAM;KACtB,EACD,GAAG,CACH,CAAA;AACF,CAAC;AAED;;;;;GAKG;AACH,MAAM,CAAC,KAAK,UAAU,oBAAoB,CACzC,GAAW,EACX,IAAsB;IAEtB,MAAM,MAAM,GAAG,QAAQ,CAAC,QAAQ,EAAE,CAAA;IAClC,IAAI,CAAC,MAAM;QAAE,OAAO,IAAI,EAAE,CAAA;IAE1B,MAAM,IAAI,GAAG,OAAO,CAAC,MAAM,CAAC,SAAS,EAAE,GAAG,SAAS,CAAC,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,OAAO,CAAC,CAAA;IAC7E,IAAI,MAAM,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;QAC9B,MAAM,MAAM,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,CAAA;QAC7D,IAAI,MAAM,KAAK,IAAI;YAAE,OAAQ,IAAI,CAAC,KAAK,CAAC,MAAM,CAAkB,CAAC,KAAK,CAAA;QACtE,MAAM,CAAC,MAAM,EAAE,IAAI,CAAC,qBAAqB,EAAE,EAAE,GAAG,EAAE,CAAC,CAAA;IACpD,CAAC;IAED,MAAM,KAAK,GAAG,MAAM,IAAI,EAAE,CAAA;IAC1B,MAAM,KAAK,CAAC,MAAM,CAAC,SAAS,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAA;IAClD,MAAM,SAAS,CAAC,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,EAAE,GAAG,EAAE,KAAK,EAAE,CAAC,CAAC,CAAA;IACrD,OAAO,KAAK,CAAA;AACb,CAAC;AAED,8EAA8E;AAC9E,MAAM,UAAU,oBAAoB;IACnC,MAAM,MAAM,GAAG,QAAQ,CAAC,QAAQ,EAAE,CAAA;IAClC,OAAO,MAAM,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,MAAM,CAAC,IAAI,EAAE,IAAI,EAAE,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,IAAI,CAAA;AAChE,CAAC","sourcesContent":["import { AsyncLocalStorage } from 'node:async_hooks'\nimport { mkdir, readFile, writeFile } from 'node:fs/promises'\nimport { resolve } from 'node:path'\nimport { sha256Hex } from './hash.js'\nimport type { Logger } from './logger.js'\nimport {\n\tCRAWL_SNAPSHOT_NAME_PATTERN,\n\ttype CrawlSnapshotInput\n} from './schema/index.js'\n\nexport { CRAWL_SNAPSHOT_NAME_PATTERN, type CrawlSnapshotInput }\n\n/**\n * A crawl recorded once and replayed, for benchmarks. Two runs of a pipeline\n * then read the same pages, so a difference between them is the code: not a\n * site that published a page in between, a 429, or a host that timed out on\n * one run and not the other.\n *\n * Recorded: the network reads - pages as `fetchPageAsMarkdown` returns them,\n * robots.txt and sitemaps - plus anything the caller routes through\n * `throughCrawlSnapshot`. Everything else runs live on every replay, which is\n * the point.\n *\n * Set per job through `withCrawlSnapshot`, never globally: a worker drains\n * other tenants' jobs on the same lane. Never in production.\n */\n\ntype CrawlSnapshot = CrawlSnapshotInput & {\n\tdirectory: string\n\tlogger: Logger | undefined\n}\n\nexport type CrawlSnapshotOptions = {\n\t/** Directory the snapshots live under. */\n\troot: string\n\t/** Told about replay misses. */\n\tlogger?: Logger | undefined\n}\n\nconst snapshot = new AsyncLocalStorage<CrawlSnapshot>()\n\nexport function withCrawlSnapshot<T>(\n\tinput: CrawlSnapshotInput | null | undefined,\n\trun: () => Promise<T>,\n\toptions: CrawlSnapshotOptions\n) {\n\tif (!input) return run()\n\tif (process.env.NODE_ENV === 'production') {\n\t\tthrow new Error('A crawl snapshot is a local bench tool')\n\t}\n\tif (!CRAWL_SNAPSHOT_NAME_PATTERN.test(input.name)) {\n\t\tthrow new Error(`Invalid crawl snapshot name: ${input.name}`)\n\t}\n\n\treturn snapshot.run(\n\t\t{\n\t\t\tmode: input.mode,\n\t\t\tname: input.name,\n\t\t\tdirectory: resolve(options.root, input.name),\n\t\t\tlogger: options.logger\n\t\t},\n\t\trun\n\t)\n}\n\n/**\n * `read` through the active snapshot. Recording stores what the network\n * returned; replaying returns what was stored. A read the recording never made\n * (a code change that discovers a new URL) goes to the network and is added,\n * with a warning, so the snapshot stays complete without hiding the drift.\n */\nexport async function throughCrawlSnapshot<T>(\n\tkey: string,\n\tread: () => Promise<T>\n): Promise<T> {\n\tconst active = snapshot.getStore()\n\tif (!active) return read()\n\n\tconst file = resolve(active.directory, `${sha256Hex(key).slice(0, 32)}.json`)\n\tif (active.mode === 'replay') {\n\t\tconst stored = await readFile(file, 'utf8').catch(() => null)\n\t\tif (stored !== null) return (JSON.parse(stored) as { value: T }).value\n\t\tactive.logger?.warn('crawl-snapshot.miss', { key })\n\t}\n\n\tconst value = await read()\n\tawait mkdir(active.directory, { recursive: true })\n\tawait writeFile(file, JSON.stringify({ key, value }))\n\treturn value\n}\n\n/** The snapshot this job reads or writes, to hand on to the job it queues. */\nexport function currentCrawlSnapshot(): CrawlSnapshotInput | null {\n\tconst active = snapshot.getStore()\n\treturn active ? { mode: active.mode, name: active.name } : null\n}\n"]}
+{"version":3,"file":"crawl-snapshot.js","sourceRoot":"","sources":["../src/crawl-snapshot.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,iBAAiB,EAAE,MAAM,kBAAkB,CAAA;AACpD,OAAO,EAAE,KAAK,EAAE,QAAQ,EAAE,SAAS,EAAE,MAAM,kBAAkB,CAAA;AAC7D,OAAO,EAAE,OAAO,EAAE,MAAM,WAAW,CAAA;AACnC,OAAO,EAAE,SAAS,EAAiB,MAAM,QAAQ,CAAA;AACjD,OAAO,EAAE,SAAS,EAAE,MAAM,WAAW,CAAA;AAErC,OAAO,EACN,2BAA2B,EAE3B,MAAM,mBAAmB,CAAA;AAE1B,OAAO,EAAE,2BAA2B,EAA2B,CAAA;AA6B/D,MAAM,QAAQ,GAAG,IAAI,iBAAiB,EAAiB,CAAA;AAEvD,MAAM,UAAU,iBAAiB,CAChC,KAAmC,EACnC,GAAqB,EACrB,OAA6B;IAE7B,IAAI,CAAC,KAAK;QAAE,OAAO,GAAG,EAAE,CAAA;IACxB,SAAS,CACR,OAAO,CAAC,GAAG,CAAC,QAAQ,KAAK,YAAY,EACrC,wCAAwC,CACxC,CAAA;IACD,SAAS,CACR,2BAA2B,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,EAC5C,gCAAgC,KAAK,CAAC,IAAI,EAAE,CAC5C,CAAA;IAED,OAAO,QAAQ,CAAC,GAAG,CAClB;QACC,IAAI,EAAE,KAAK,CAAC,IAAI;QAChB,IAAI,EAAE,KAAK,CAAC,IAAI;QAChB,SAAS,EAAE,OAAO,CAAC,OAAO,CAAC,IAAI,EAAE,KAAK,CAAC,IAAI,CAAC;QAC5C,MAAM,EAAE,OAAO,CAAC,MAAM;KACtB,EACD,GAAG,CACH,CAAA;AACF,CAAC;AAED;;;;;GAKG;AACH,MAAM,CAAC,KAAK,UAAU,oBAAoB,CACzC,GAAW,EACX,IAAsB;IAEtB,MAAM,MAAM,GAAG,QAAQ,CAAC,QAAQ,EAAE,CAAA;IAClC,IAAI,CAAC,MAAM;QAAE,OAAO,IAAI,EAAE,CAAA;IAE1B,MAAM,IAAI,GAAG,OAAO,CAAC,MAAM,CAAC,SAAS,EAAE,GAAG,SAAS,CAAC,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,OAAO,CAAC,CAAA;IAC7E,IAAI,MAAM,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;QAC9B,MAAM,MAAM,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,CAAA;QAC7D,iEAAiE;QACjE,MAAM,MAAM,GAAG,MAAM,KAAK,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,WAAW,CAAI,MAAM,CAAC,CAAA;QAC9D,IAAI,MAAM;YAAE,OAAO,MAAM,CAAC,KAAK,CAAA;QAC/B,MAAM,CAAC,MAAM,EAAE,IAAI,CAAC,qBAAqB,EAAE,EAAE,GAAG,EAAE,CAAC,CAAA;IACpD,CAAC;IAED,MAAM,KAAK,GAAG,MAAM,IAAI,EAAE,CAAA;IAC1B,MAAM,KAAK,CAAC,MAAM,CAAC,SAAS,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAA;IAClD,MAAM,SAAS,CAAC,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,EAAE,GAAG,EAAE,KAAK,EAAE,CAAC,CAAC,CAAA;IACrD,OAAO,KAAK,CAAA;AACb,CAAC;AAED,SAAS,WAAW,CAAI,MAAc;IACrC,IAAI,CAAC;QACJ,OAAO,IAAI,CAAC,KAAK,CAAC,MAAM,CAAiB,CAAA;IAC1C,CAAC;IAAC,MAAM,CAAC;QACR,OAAO,IAAI,CAAA;IACZ,CAAC;AACF,CAAC;AAED,8EAA8E;AAC9E,MAAM,UAAU,oBAAoB;IACnC,MAAM,MAAM,GAAG,QAAQ,CAAC,QAAQ,EAAE,CAAA;IAClC,OAAO,MAAM,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,MAAM,CAAC,IAAI,EAAE,IAAI,EAAE,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,IAAI,CAAA;AAChE,CAAC","sourcesContent":["import { AsyncLocalStorage } from 'node:async_hooks'\nimport { mkdir, readFile, writeFile } from 'node:fs/promises'\nimport { resolve } from 'node:path'\nimport { invariant, type Nullable } from 'hanzio'\nimport { sha256Hex } from './hash.js'\nimport type { Logger } from './logger.js'\nimport {\n\tCRAWL_SNAPSHOT_NAME_PATTERN,\n\ttype CrawlSnapshotInput\n} from './schema/index.js'\n\nexport { CRAWL_SNAPSHOT_NAME_PATTERN, type CrawlSnapshotInput }\n\n/**\n * A crawl recorded once and replayed, for benchmarks. Two runs of a pipeline\n * then read the same pages, so a difference between them is the code: not a\n * site that published a page in between, a 429, or a host that timed out on\n * one run and not the other.\n *\n * Recorded: the network reads - pages as `fetchPageAsMarkdown` returns them,\n * robots.txt and sitemaps - plus anything the caller routes through\n * `throughCrawlSnapshot`. Everything else runs live on every replay, which is\n * the point.\n *\n * Set per job through `withCrawlSnapshot`, never globally: a worker drains\n * other tenants' jobs on the same lane. Never in production.\n */\n\ntype CrawlSnapshot = CrawlSnapshotInput & {\n\tdirectory: string\n\tlogger: Logger | undefined\n}\n\nexport type CrawlSnapshotOptions = {\n\t/** Directory the snapshots live under. */\n\troot: string\n\t/** Told about replay misses. */\n\tlogger?: Logger | undefined\n}\n\nconst snapshot = new AsyncLocalStorage<CrawlSnapshot>()\n\nexport function withCrawlSnapshot<T>(\n\tinput: Nullable<CrawlSnapshotInput>,\n\trun: () => Promise<T>,\n\toptions: CrawlSnapshotOptions\n) {\n\tif (!input) return run()\n\tinvariant(\n\t\tprocess.env.NODE_ENV !== 'production',\n\t\t'A crawl snapshot is a local bench tool'\n\t)\n\tinvariant(\n\t\tCRAWL_SNAPSHOT_NAME_PATTERN.test(input.name),\n\t\t`Invalid crawl snapshot name: ${input.name}`\n\t)\n\n\treturn snapshot.run(\n\t\t{\n\t\t\tmode: input.mode,\n\t\t\tname: input.name,\n\t\t\tdirectory: resolve(options.root, input.name),\n\t\t\tlogger: options.logger\n\t\t},\n\t\trun\n\t)\n}\n\n/**\n * `read` through the active snapshot. Recording stores what the network\n * returned; replaying returns what was stored. A read the recording never made\n * (a code change that discovers a new URL) goes to the network and is added,\n * with a warning, so the snapshot stays complete without hiding the drift.\n */\nexport async function throughCrawlSnapshot<T>(\n\tkey: string,\n\tread: () => Promise<T>\n): Promise<T> {\n\tconst active = snapshot.getStore()\n\tif (!active) return read()\n\n\tconst file = resolve(active.directory, `${sha256Hex(key).slice(0, 32)}.json`)\n\tif (active.mode === 'replay') {\n\t\tconst stored = await readFile(file, 'utf8').catch(() => null)\n\t\t// A file cut short by a killed recording is a miss, not a crash.\n\t\tconst parsed = stored === null ? null : parseStored<T>(stored)\n\t\tif (parsed) return parsed.value\n\t\tactive.logger?.warn('crawl-snapshot.miss', { key })\n\t}\n\n\tconst value = await read()\n\tawait mkdir(active.directory, { recursive: true })\n\tawait writeFile(file, JSON.stringify({ key, value }))\n\treturn value\n}\n\nfunction parseStored<T>(stored: string) {\n\ttry {\n\t\treturn JSON.parse(stored) as { value: T }\n\t} catch {\n\t\treturn null\n\t}\n}\n\n/** The snapshot this job reads or writes, to hand on to the job it queues. */\nexport function currentCrawlSnapshot(): CrawlSnapshotInput | null {\n\tconst active = snapshot.getStore()\n\treturn active ? { mode: active.mode, name: active.name } : null\n}\n"]}

package/dist/fetch-page.d.ts

@@ -1,12 +1,14 @@
+import { type Nullable } from 'hanzio';
 import type { ApiError } from 'hanzio/api-wrapper';
 import { type CloudflareCredentials } from './client.js';
 import { type CrawlSnapshotInput, type CrawlSnapshotOptions } from './crawl-snapshot.js';
 import { type CreateConcurrencyLimit } from './limit.js';
 import { type Logger } from './logger.js';
+import { type Fetch, type GuardedFetchOptions } from './net-guard.js';
+import type { CaptureIssue, FetchOutcome, FetchSource } from './schema/index.js';
+export type { CaptureIssue, FetchOutcome, FetchSource };
 /** Stored markdown ceiling; a page over this is a listing or an upload, not knowledge. */
 export declare const MAX_MARKDOWN_CHARS = 300000;
-export type FetchOutcome = 'ok' | 'unchanged' | 'gone' | 'blocked' | 'transient' | 'unsupported';
-export type FetchSource = 'content-negotiation' | 'workers-ai' | 'browser-rendering' | 'not-modified' | 'firecrawl-parse';
 export type FetchedPage = {
     /** The URL we requested, after http->https upgrade and www fallback. */
     url: string;
@@ -27,13 +29,13 @@ export type FetchedPage = {
     lang: string | null;
     /** <link rel="canonical">, absolute. */
     canonical: string | null;
-    /** `<link rel="alternate" hreflang>` targets, absolute, `x-default` excluded. Optional: many callers build a page from parts. */
-    alternates?: string[];
+    /** `<link rel="alternate" hreflang>` targets, absolute, `x-default` excluded. */
+    alternates: string[];
     /** `alternates` by language: URL -> ISO 639-1 (`da` of `da-DK`). */
-    alternateLanguages?: Record<string, string>;
+    alternateLanguages: Record<string, string>;
     /** <meta name="description"> or the og: equivalent. */
     metaDescription: string | null;
-    /** Parsed application/ld+json blocks (any shape; the profile layer reads LocalBusiness/OpeningHoursSpecification). */
+    /** Parsed application/ld+json blocks, any shape. */
     jsonLd: unknown[];
     /**
      * Absolute, deduped link targets: the markdown's, unioned with every
@@ -44,38 +46,39 @@ export type FetchedPage = {
     /** Absolute, deduped image URLs found in the markdown. */
     images: string[];
     durationMs: number;
-    captureIssue?: string | null | undefined;
+    /** Why the page is incomplete or missing, when the library knows. */
+    captureIssue: CaptureIssue | null;
     /**
      * Hash of the HTML the origin served, when an origin tier read HTML. Given
      * back on the next read (`fetchPageAsMarkdown({ htmlHash })`), identical HTML
      * is answered as `unchanged` without converting it again.
      */
-    htmlHash?: string | null | undefined;
+    htmlHash: string | null;
 };
 export type PageBody = {
-    captureIssue?: string | null | undefined;
+    captureIssue?: Nullable<CaptureIssue>;
     outcome: FetchOutcome;
     source: FetchSource | null;
-    markdown?: string | null | undefined;
+    markdown?: Nullable<string>;
     /** The HTML an origin tier read, when one did - the only source of metadata. */
-    html?: string | null | undefined;
+    html?: Nullable<string>;
 };
 export type FetchPageInput = {
     url: string;
     /** Stored validators from the last run; a 304 returns outcome 'unchanged'. */
-    etag?: string | null | undefined;
-    lastModified?: string | null | undefined;
+    etag?: Nullable<string>;
+    lastModified?: Nullable<string>;
     /** Allow the Browser Rendering tier. Default true. */
     render?: boolean | undefined;
     /**
      * ISO country of the market to read the page in. A Shopify store redirects
-     * by the visitor's IP - our worker reads a Danish shop as its EU market in
-     * euros and English (Good Habit audit §3.2) - and `?country=DK` pins it.
-     * Applied to the request only: the page keeps the URL it was asked for.
+     * by the visitor's IP - a worker abroad reads a Danish shop as its EU market
+     * in euros and English - and `?country=DK` pins it. Applied to the request
+     * only: the page keeps the URL it was asked for.
      */
-    country?: string | null | undefined;
+    country?: Nullable<string>;
     /** The last read's `htmlHash`; identical HTML comes back `unchanged`. */
-    htmlHash?: string | null | undefined;
+    htmlHash?: Nullable<string>;
     signal?: AbortSignal | undefined;
 };
 export type PdfParser = (input: {
@@ -115,10 +118,17 @@ export type WebsiteToMarkdownOptions = {
     /** The OCR tier for a PDF Workers AI returned nothing for. */
     parsePdf?: PdfParser;
     /**
-     * SSRF guard run before every origin request. Defaults to
-     * {@link isHostAllowed}: public addresses only, localhost outside production.
+     * SSRF guard run before every origin request and every redirect it
+     * follows. Defaults to {@link isHostAllowed}: public addresses only.
      */
     isHostAllowed?: (hostname: string) => Promise<boolean>;
+    /**
+     * Let the default guard and `http://` through to `localhost`, `127.0.0.1`
+     * and `::1`, e.g. outside production for a dev server. Default false.
+     */
+    allowLocalhost?: boolean;
+    /** The `fetch` origin requests go through, behind the guard. Default the global. */
+    fetch?: Fetch;
 };
 /**
  * One converter: its Cloudflare client, pools and per-host politeness limits.
@@ -136,14 +146,20 @@ export declare function createWebsiteToMarkdown(options: WebsiteToMarkdownOption
     }) => Promise<PageBody>;
     urlToMarkdown: (urlString: string) => Promise<{
         markdown: string;
-        source: FetchSource;
+        source: "browser-rendering" | "content-negotiation" | "workers-ai" | "not-modified" | "firecrawl-parse";
     }>;
     convertFileWithWorkersAi: (input: import("./client.js").ConvertFileInput) => Promise<string | null>;
     renderUrlToMarkdown: (url: string, signal?: AbortSignal) => Promise<string | null>;
     /** The SSRF guard the converter runs, bound to its `logger`. */
     isHostAllowed: (hostname: string) => Promise<boolean>;
+    /**
+     * `fetch` behind the converter's guard, run on the URL and on every
+     * redirect. Throws `HostNotAllowedError` when one is refused. Use it for
+     * any other request to a customer-supplied URL.
+     */
+    guardedFetch(input: string | URL, init?: RequestInit, fetchOptions?: Pick<GuardedFetchOptions, "maxRedirects">): Promise<Response>;
     /** `withCrawlSnapshot` bound to the `crawlSnapshot` option. */
-    withCrawlSnapshot<T>(input: CrawlSnapshotInput | null | undefined, run: () => Promise<T>): Promise<T>;
+    withCrawlSnapshot<T>(input: Nullable<CrawlSnapshotInput>, run: () => Promise<T>): Promise<T>;
 };
 export type WebsiteToMarkdown = ReturnType<typeof createWebsiteToMarkdown>;
 //# sourceMappingURL=fetch-page.d.ts.map

package/dist/fetch-page.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"fetch-page.d.ts","sourceRoot":"","sources":["../src/fetch-page.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,oBAAoB,CAAA;AAElD,OAAO,EACN,KAAK,qBAAqB,EAG1B,MAAM,aAAa,CAAA;AAMpB,OAAO,EACN,KAAK,kBAAkB,EACvB,KAAK,oBAAoB,EAIzB,MAAM,qBAAqB,CAAA;AAQ5B,OAAO,EAAE,KAAK,sBAAsB,EAAoB,MAAM,YAAY,CAAA;AAC1E,OAAO,EAAsB,KAAK,MAAM,EAAgB,MAAM,aAAa,CAAA;AAsB3E,0FAA0F;AAC1F,eAAO,MAAM,kBAAkB,SAAU,CAAA;AAEzC,MAAM,MAAM,YAAY,GACrB,IAAI,GACJ,WAAW,GACX,MAAM,GACN,SAAS,GACT,WAAW,GACX,aAAa,CAAA;AAChB,MAAM,MAAM,WAAW,GACpB,qBAAqB,GACrB,YAAY,GACZ,mBAAmB,GACnB,cAAc,GACd,iBAAiB,CAAA;AAEpB,MAAM,MAAM,WAAW,GAAG;IACzB,wEAAwE;IACxE,GAAG,EAAE,MAAM,CAAA;IACX,mDAAmD;IACnD,QAAQ,EAAE,MAAM,CAAA;IAChB,iFAAiF;IACjF,MAAM,EAAE,MAAM,CAAA;IACd,OAAO,EAAE,YAAY,CAAA;IACrB,MAAM,EAAE,WAAW,GAAG,IAAI,CAAA;IAC1B,WAAW,EAAE,MAAM,GAAG,IAAI,CAAA;IAC1B,IAAI,EAAE,MAAM,GAAG,IAAI,CAAA;IACnB,YAAY,EAAE,MAAM,GAAG,IAAI,CAAA;IAC3B,2EAA2E;IAC3E,QAAQ,EAAE,MAAM,CAAA;IAChB,2EAA2E;IAC3E,KAAK,EAAE,MAAM,GAAG,IAAI,CAAA;IACpB,yDAAyD;IACzD,IAAI,EAAE,MAAM,GAAG,IAAI,CAAA;IACnB,wCAAwC;IACxC,SAAS,EAAE,MAAM,GAAG,IAAI,CAAA;IACxB,iIAAiI;IACjI,UAAU,CAAC,EAAE,MAAM,EAAE,CAAA;IACrB,oEAAoE;IACpE,kBAAkB,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAA;IAC3C,uDAAuD;IACvD,eAAe,EAAE,MAAM,GAAG,IAAI,CAAA;IAC9B,sHAAsH;IACtH,MAAM,EAAE,OAAO,EAAE,CAAA;IACjB;;;;OAIG;IACH,KAAK,EAAE,MAAM,EAAE,CAAA;IACf,0DAA0D;IAC1D,MAAM,EAAE,MAAM,EAAE,CAAA;IAChB,UAAU,EAAE,MAAM,CAAA;IAClB,YAAY,CAAC,EAAE,MAAM,GAAG,IAAI,GAAG,SAAS,CAAA;IACxC;;;;OAIG;IACH,QAAQ,CAAC,EAAE,MAAM,GAAG,IAAI,GAAG,SAAS,CAAA;CACpC,CAAA;AAYD,MAAM,MAAM,QAAQ,GAAG;IACtB,YAAY,CAAC,EAAE,MAAM,GAAG,IAAI,GAAG,SAAS,CAAA;IACxC,OAAO,EAAE,YAAY,CAAA;IACrB,MAAM,EAAE,WAAW,GAAG,IAAI,CAAA;IAC1B,QAAQ,CAAC,EAAE,MAAM,GAAG,IAAI,GAAG,SAAS,CAAA;IACpC,gFAAgF;IAChF,IAAI,CAAC,EAAE,MAAM,GAAG,IAAI,GAAG,SAAS,CAAA;CAChC,CAAA;AAyTD,MAAM,MAAM,cAAc,GAAG;IAC5B,GAAG,EAAE,MAAM,CAAA;IACX,8EAA8E;IAC9E,IAAI,CAAC,EAAE,MAAM,GAAG,IAAI,GAAG,SAAS,CAAA;IAChC,YAAY,CAAC,EAAE,MAAM,GAAG,IAAI,GAAG,SAAS,CAAA;IACxC,sDAAsD;IACtD,MAAM,CAAC,EAAE,OAAO,GAAG,SAAS,CAAA;IAC5B;;;;;OAKG;IACH,OAAO,CAAC,EAAE,MAAM,GAAG,IAAI,GAAG,SAAS,CAAA;IACnC,yEAAyE;IACzE,QAAQ,CAAC,EAAE,MAAM,GAAG,IAAI,GAAG,SAAS,CAAA;IACpC,MAAM,CAAC,EAAE,WAAW,GAAG,SAAS,CAAA;CAChC,CAAA;AAED,MAAM,MAAM,SAAS,GAAG,CAAC,KAAK,EAAE;IAC/B,KAAK,EAAE,WAAW,CAAA;IAClB,QAAQ,EAAE,MAAM,CAAA;IAChB,QAAQ,EAAE,MAAM,CAAA;IAChB,2EAA2E;IAC3E,QAAQ,EAAE,MAAM,CAAA;IAChB,MAAM,CAAC,EAAE,WAAW,GAAG,SAAS,CAAA;CAChC,KAAK,OAAO,CAAC;IAAE,QAAQ,EAAE,MAAM,CAAC;IAAC,QAAQ,EAAE,MAAM,GAAG,IAAI,CAAA;CAAE,CAAC,CAAA;AAE5D,MAAM,MAAM,wBAAwB,GAAG;IACtC,4EAA4E;IAC5E,UAAU,EAAE,qBAAqB,CAAA;IACjC;;;;OAIG;IACH,WAAW,CAAC,EAAE,sBAAsB,CAAA;IACpC,sEAAsE;IACtE,UAAU,CAAC,EAAE,CAAC,KAAK,EAAE,QAAQ,KAAK,IAAI,CAAA;IACtC;;;;OAIG;IACH,cAAc,CAAC,EAAE,OAAO,CAAA;IACxB,MAAM,CAAC,EAAE,MAAM,CAAA;IACf;;;OAGG;IACH,aAAa,CAAC,EAAE,oBAAoB,CAAA;IACpC,8DAA8D;IAC9D,QAAQ,CAAC,EAAE,SAAS,CAAA;IACpB;;;OAGG;IACH,aAAa,CAAC,EAAE,CAAC,QAAQ,EAAE,MAAM,KAAK,OAAO,CAAC,OAAO,CAAC,CAAA;CACtD,CAAA;AAED;;;GAGG;AACH,wBAAgB,uBAAuB,CAAC,OAAO,EAAE,wBAAwB;iCA8UhE,cAAc,KACnB,OAAO,CAAC,WAAW,CAAC;kCA7QoB;QAC1C,KAAK,EAAE,WAAW,CAAA;QAClB,QAAQ,EAAE,MAAM,CAAA;QAChB,QAAQ,EAAE,MAAM,CAAA;QAChB,SAAS,EAAE,MAAM,CAAA;QACjB,iBAAiB,CAAC,EAAE,MAAM,GAAG,SAAS,CAAA;QACtC,MAAM,CAAC,EAAE,WAAW,GAAG,SAAS,CAAA;KAChC,KAAG,OAAO,CAAC,QAAQ,CAAC;+BA4amB,MAAM;;;;;;IAiB7C,gEAAgE;8BA7gBtC,MAAM,KAAK,OAAO,CAAC,OAAO,CAAC;IA+gBrD,+DAA+D;sBAC7C,CAAC,SACX,kBAAkB,GAAG,IAAI,GAAG,SAAS,OACvC,MAAM,OAAO,CAAC,CAAC,CAAC;EAYvB;AAED,MAAM,MAAM,iBAAiB,GAAG,UAAU,CAAC,OAAO,uBAAuB,CAAC,CAAA"}
+{"version":3,"file":"fetch-page.d.ts","sourceRoot":"","sources":["../src/fetch-page.ts"],"names":[],"mappings":"AAAA,OAAO,EASN,KAAK,QAAQ,EAMb,MAAM,QAAQ,CAAA;AACf,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,oBAAoB,CAAA;AAClD,OAAO,EACN,KAAK,qBAAqB,EAG1B,MAAM,aAAa,CAAA;AAEpB,OAAO,EACN,KAAK,kBAAkB,EACvB,KAAK,oBAAoB,EAIzB,MAAM,qBAAqB,CAAA;AAQ5B,OAAO,EAAE,KAAK,sBAAsB,EAAoB,MAAM,YAAY,CAAA;AAC1E,OAAO,EAAsB,KAAK,MAAM,EAAgB,MAAM,aAAa,CAAA;AAW3E,OAAO,EACN,KAAK,KAAK,EAEV,KAAK,mBAAmB,EAKxB,MAAM,gBAAgB,CAAA;AAGvB,OAAO,KAAK,EAAE,YAAY,EAAE,YAAY,EAAE,WAAW,EAAE,MAAM,mBAAmB,CAAA;AAKhF,YAAY,EAAE,YAAY,EAAE,YAAY,EAAE,WAAW,EAAE,CAAA;AAcvD,0FAA0F;AAC1F,eAAO,MAAM,kBAAkB,SAAU,CAAA;AAEzC,MAAM,MAAM,WAAW,GAAG;IACzB,wEAAwE;IACxE,GAAG,EAAE,MAAM,CAAA;IACX,mDAAmD;IACnD,QAAQ,EAAE,MAAM,CAAA;IAChB,iFAAiF;IACjF,MAAM,EAAE,MAAM,CAAA;IACd,OAAO,EAAE,YAAY,CAAA;IACrB,MAAM,EAAE,WAAW,GAAG,IAAI,CAAA;IAC1B,WAAW,EAAE,MAAM,GAAG,IAAI,CAAA;IAC1B,IAAI,EAAE,MAAM,GAAG,IAAI,CAAA;IACnB,YAAY,EAAE,MAAM,GAAG,IAAI,CAAA;IAC3B,2EAA2E;IAC3E,QAAQ,EAAE,MAAM,CAAA;IAChB,2EAA2E;IAC3E,KAAK,EAAE,MAAM,GAAG,IAAI,CAAA;IACpB,yDAAyD;IACzD,IAAI,EAAE,MAAM,GAAG,IAAI,CAAA;IACnB,wCAAwC;IACxC,SAAS,EAAE,MAAM,GAAG,IAAI,CAAA;IACxB,iFAAiF;IACjF,UAAU,EAAE,MAAM,EAAE,CAAA;IACpB,oEAAoE;IACpE,kBAAkB,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAA;IAC1C,uDAAuD;IACvD,eAAe,EAAE,MAAM,GAAG,IAAI,CAAA;IAC9B,oDAAoD;IACpD,MAAM,EAAE,OAAO,EAAE,CAAA;IACjB;;;;OAIG;IACH,KAAK,EAAE,MAAM,EAAE,CAAA;IACf,0DAA0D;IAC1D,MAAM,EAAE,MAAM,EAAE,CAAA;IAChB,UAAU,EAAE,MAAM,CAAA;IAClB,qEAAqE;IACrE,YAAY,EAAE,YAAY,GAAG,IAAI,CAAA;IACjC;;;;OAIG;IACH,QAAQ,EAAE,MAAM,GAAG,IAAI,CAAA;CACvB,CAAA;AAYD,MAAM,MAAM,QAAQ,GAAG;IACtB,YAAY,CAAC,EAAE,QAAQ,CAAC,YAAY,CAAC,CAAA;IACrC,OAAO,EAAE,YAAY,CAAA;IACrB,MAAM,EAAE,WAAW,GAAG,IAAI,CAAA;IAC1B,QAAQ,CAAC,EAAE,QAAQ,CAAC,MAAM,CAAC,CAAA;IAC3B,gFAAgF;IAChF,IAAI,CAAC,EAAE,QAAQ,CAAC,MAAM,CAAC,CAAA;CACvB,CAAA;AAsTD,MAAM,MAAM,cAAc,GAAG;IAC5B,GAAG,EAAE,MAAM,CAAA;IACX,8EAA8E;IAC9E,IAAI,CAAC,EAAE,QAAQ,CAAC,MAAM,CAAC,CAAA;IACvB,YAAY,CAAC,EAAE,QAAQ,CAAC,MAAM,CAAC,CAAA;IAC/B,sDAAsD;IACtD,MAAM,CAAC,EAAE,OAAO,GAAG,SAAS,CAAA;IAC5B;;;;;OAKG;IACH,OAAO,CAAC,EAAE,QAAQ,CAAC,MAAM,CAAC,CAAA;IAC1B,yEAAyE;IACzE,QAAQ,CAAC,EAAE,QAAQ,CAAC,MAAM,CAAC,CAAA;IAC3B,MAAM,CAAC,EAAE,WAAW,GAAG,SAAS,CAAA;CAChC,CAAA;AAED,MAAM,MAAM,SAAS,GAAG,CAAC,KAAK,EAAE;IAC/B,KAAK,EAAE,WAAW,CAAA;IAClB,QAAQ,EAAE,MAAM,CAAA;IAChB,QAAQ,EAAE,MAAM,CAAA;IAChB,2EAA2E;IAC3E,QAAQ,EAAE,MAAM,CAAA;IAChB,MAAM,CAAC,EAAE,WAAW,GAAG,SAAS,CAAA;CAChC,KAAK,OAAO,CAAC;IAAE,QAAQ,EAAE,MAAM,CAAC;IAAC,QAAQ,EAAE,MAAM,GAAG,IAAI,CAAA;CAAE,CAAC,CAAA;AAE5D,MAAM,MAAM,wBAAwB,GAAG;IACtC,4EAA4E;IAC5E,UAAU,EAAE,qBAAqB,CAAA;IACjC;;;;OAIG;IACH,WAAW,CAAC,EAAE,sBAAsB,CAAA;IACpC,sEAAsE;IACtE,UAAU,CAAC,EAAE,CAAC,KAAK,EAAE,QAAQ,KAAK,IAAI,CAAA;IACtC;;;;OAIG;IACH,cAAc,CAAC,EAAE,OAAO,CAAA;IACxB,MAAM,CAAC,EAAE,MAAM,CAAA;IACf;;;OAGG;IACH,aAAa,CAAC,EAAE,oBAAoB,CAAA;IACpC,8DAA8D;IAC9D,QAAQ,CAAC,EAAE,SAAS,CAAA;IACpB;;;OAGG;IACH,aAAa,CAAC,EAAE,CAAC,QAAQ,EAAE,MAAM,KAAK,OAAO,CAAC,OAAO,CAAC,CAAA;IACtD;;;OAGG;IACH,cAAc,CAAC,EAAE,OAAO,CAAA;IACxB,oFAAoF;IACpF,KAAK,CAAC,EAAE,KAAK,CAAA;CACb,CAAA;AAED;;;GAGG;AACH,wBAAgB,uBAAuB,CAAC,OAAO,EAAE,wBAAwB;iCAyXhE,cAAc,KACnB,OAAO,CAAC,WAAW,CAAC;kCAtToB;QAC1C,KAAK,EAAE,WAAW,CAAA;QAClB,QAAQ,EAAE,MAAM,CAAA;QAChB,QAAQ,EAAE,MAAM,CAAA;QAChB,SAAS,EAAE,MAAM,CAAA;QACjB,iBAAiB,CAAC,EAAE,MAAM,GAAG,SAAS,CAAA;QACtC,MAAM,CAAC,EAAE,WAAW,GAAG,SAAS,CAAA;KAChC,KAAG,OAAO,CAAC,QAAQ,CAAC;+BA2dmB,MAAM;;;;;;IAiB7C,gEAAgE;8BArkBtC,MAAM,KAAK,OAAO,CAAC,OAAO,CAAC;IAukBrD;;;;OAIG;wBAEK,MAAM,GAAG,GAAG,SACZ,WAAW,iBACH,IAAI,CAAC,mBAAmB,EAAE,cAAc,CAAC;IAIzD,+DAA+D;sBAC7C,CAAC,SACX,QAAQ,CAAC,kBAAkB,CAAC,OAC9B,MAAM,OAAO,CAAC,CAAC,CAAC;EAavB;AAED,MAAM,MAAM,iBAAiB,GAAG,UAAU,CAAC,OAAO,uBAAuB,CAAC,CAAA"}