llms-ink 0.3.0 → 0.4.0

package/README.md

@@ -74,6 +74,16 @@ const { parsed, issues } = await ink.checkText(await readFile('public/llms.txt',
 Without `base`, `site` and `url` are `null`; without `base` and `links`, `llmsFull` is `null` and
 `links.checked` is 0. Up to 2 MB.
 
+### `ink.markdown(html, { url?, signal? })` → `Promise<string>`
+
+A page's HTML as Markdown, the way llms.ink reads pages: hidden-instruction text (comments, hidden elements,
+off-screen text addressed to AI), scripts, navigation, headers, footers and forms are dropped first. `url` (never
+fetched) resolves relative links. Up to 5 MB; nothing is stored.
+
+```ts
+const md = await ink.markdown(await readFile('dist/pricing/index.html', 'utf8'), { url: 'https://example.com/pricing/' });
+```
+
 ### `ink.stream(url, { mode?, fresh?, signal? })` → `AsyncGenerator<StreamEvent>`
 
 The same build as `llms()`, as it happens:
@@ -104,7 +114,15 @@ After `vite build` (not the dev server or SSR builds), writes `llms.txt` (and `l
 `site` into `build.outDir`. llms.ink reads the live site, so the files describe the previous deploy. If llms.ink
 fails, the plugin warns and keeps any existing file (e.g. one from `public/`); `strict: true` fails the build.
 Options: `site`, `full`, `key` (default `LLMS_INK_KEY`), `native` (default `true`; a file llms.ink generated before
-is regenerated, not echoed), `strict`, `timeout` (ms, default 300000), `baseUrl`.
+is regenerated, not echoed), `strict`, `timeout` (ms, default 300000), `baseUrl`, `markdownPages`.
+
+`markdownPages: true` also writes a Markdown version next to every HTML page of this build (`about.html` →
+`about.md`, `docs/index.html` → `docs/index.md`), converted with `ink.markdown()` from the local output, so the
+`.md` pages match what you're deploying. Existing `.md` files are kept; `/404`, `/500` and pages over 5 MB are
+skipped; a rate limit is waited out (`Retry-After`). Failures warn and the build goes on, unless `strict`. Pass
+`{ concurrency, exclude }` (default 4 at a time; `exclude` is a RegExp on the page path) to tune it. Large sites:
+use an API key (60 conversions a minute anonymous per IP, the key's own limit otherwise). They're static files:
+`/about.md` works as is; for `Accept: text/markdown` on the page URL, add a rewrite to `<page>.md` on your host.
 
 ### Astro: `llms-ink/astro`
 
@@ -137,6 +155,29 @@ Everything else, and llms.ink's own fetcher, gets `undefined` (Express: `next()`
 `LLMS_INK_KEY`; recommended on shared hosting IPs), `markdown`, `baseUrl`. Don't also ship a static `llms.txt`:
 most hosts serve it before the middleware runs.
 
+#### AI traffic
+
+```ts
+const llms = llmsInkHandler({ site: 'https://example.com', traffic: { key: env.LLMS_INK_KEY } });
+const response = (await llms(request, ctx)) ?? (await app(request)); // Workers: ctx; Next.js proxy: the event
+
+expressApp.use(llmsInkExpress({ site: 'https://example.com', traffic: { key: process.env.LLMS_INK_KEY } }));
+```
+
+With `traffic: { key, site?, flushEvery? }` the middleware also counts requests from AI agents (`AI_AGENTS`):
+user fetches (`ChatGPT-User`, `Claude-User`, `Perplexity-User`…, someone asked an assistant) and crawlers (`GPTBot`,
+`OAI-SearchBot`, `ClaudeBot`, `PerplexityBot`, `Googlebot`, `bingbot`…). It counts every request it sees, served or
+not, per path and agent in memory, and every `flushEvery` seconds (default 60, at least 10) sends the day's totals to
+`POST /api/v0/traffic`, at most 1000 rows per request. Only `{ site, period, rows: [{ path, agent, kind, count }] }`
+leaves your server: no IPs, user agents, query strings or cookies; browsers aren't counted. Sends never block a
+response: with `ctx` they go to `ctx.waitUntil`, without one (Node, Bun, Deno) an unref'd timer sends. Failed sends
+keep their counts and retry with backoff (honoring `Retry-After`); a `401` or `403 site_claimed` stops counting with
+one warning. `site` defaults to the middleware's hostname; the first key to send for a site owns it. Counts not yet
+sent are lost when the process stops: call `await llms.flush()` on shutdown. Read them at
+[llms.ink/traffic](https://llms.ink/traffic/) or `GET /api/v0/traffic?site=&days=` with the same key.
+`classifyAgent(userAgent)` and `trafficCounter({ key, site })` (`record(request, ctx?)`, `flush()`) are exported for
+other servers.
+
 ## Errors
 
 Every failure is an `LlmsInkError` with `status`, `code`, `title`, `detail`, `errors` (per-parameter issues of

package/dist/astro.d.ts

@@ -1,5 +1,6 @@
 /**
- * Astro integration: writes llms.txt (and llms-full.txt) from llms.ink into the build output.
+ * Astro integration: writes llms.txt (and llms-full.txt) from llms.ink into the build output, and with
+ * `markdownPages: true` a Markdown version next to every built HTML page.
  *
  * @example
  * ```ts
@@ -13,7 +14,8 @@
  * @module
  */
 import type { AstroIntegration } from 'astro';
-import { type LlmsFilesOptions } from './files.ts';
+import { type LlmsFilesOptions, type MarkdownPagesOptions } from './files.ts';
+export type { MarkdownPagesOptions };
 /** {@link LlmsFilesOptions} with `site` optional: it defaults to Astro's `site` (plus `base`). */
 export type LlmsInkAstroOptions = Omit<LlmsFilesOptions, 'site'> & {
     site?: string;
@@ -22,7 +24,9 @@ export type LlmsInkAstroOptions = Omit<LlmsFilesOptions, 'site'> & {
  * After `astro build`, fetches llms.txt — and llms-full.txt with `full: true` — for `site` (default: the
  * `site` and `base` of the Astro config) and writes them into the static output directory. llms.ink reads the
  * live site, so the files describe the previous deploy. When llms.ink can't be reached it warns and keeps any
- * existing file (e.g. one copied from `public/`); `strict: true` fails the build instead.
+ * existing file (e.g. one copied from `public/`); `strict: true` fails the build instead. `markdownPages: true`
+ * converts each prerendered `.html` page of this build through `POST /api/v0/markdown` and writes `about.md` /
+ * `docs/index.md` next to it; page URLs come from `site`.
  */
 export declare function llmsInk(options?: LlmsInkAstroOptions): AstroIntegration;
 export default llmsInk;

package/dist/astro.js

@@ -1,5 +1,6 @@
 /**
- * Astro integration: writes llms.txt (and llms-full.txt) from llms.ink into the build output.
+ * Astro integration: writes llms.txt (and llms-full.txt) from llms.ink into the build output, and with
+ * `markdownPages: true` a Markdown version next to every built HTML page.
  *
  * @example
  * ```ts
@@ -18,7 +19,9 @@ import { llmsFilesWriter } from "./files.js";
  * After `astro build`, fetches llms.txt — and llms-full.txt with `full: true` — for `site` (default: the
  * `site` and `base` of the Astro config) and writes them into the static output directory. llms.ink reads the
  * live site, so the files describe the previous deploy. When llms.ink can't be reached it warns and keeps any
- * existing file (e.g. one copied from `public/`); `strict: true` fails the build instead.
+ * existing file (e.g. one copied from `public/`); `strict: true` fails the build instead. `markdownPages: true`
+ * converts each prerendered `.html` page of this build through `POST /api/v0/markdown` and writes `about.md` /
+ * `docs/index.md` next to it; page URLs come from `site`.
  */
 export function llmsInk(options = {}) {
     const settings = { ...options, site: options.site ?? '' };

package/dist/files.d.ts

@@ -1,9 +1,16 @@
 /**
  * Shared by the Vite plugin and the Astro integration: fetch llms.txt (and llms-full.txt) for a site and write
- * them into a build's output directory.
+ * them into a build's output directory; with `markdownPages`, write a `.md` version next to every built HTML page.
  *
  * @module
  */
+/** `markdownPages` settings beyond `true`. */
+export interface MarkdownPagesOptions {
+    /** Pages converted at once. Default 4. */
+    concurrency?: number;
+    /** Pages to skip, matched against the page path (`/docs/`, `/about`). Default: `/404` and `/500` pages. */
+    exclude?: RegExp;
+}
 /** Options for the Vite plugin and the Astro integration. */
 export interface LlmsFilesOptions {
     /** The deployed site to describe, e.g. `https://example.com`. llms.ink reads the live site. */
@@ -18,7 +25,14 @@ export interface LlmsFilesOptions {
      * generates (`native=0`).
      */
     native?: boolean;
-    /** Fail the build when a file can't be fetched. Default `false`: warn and keep any existing file. */
+    /**
+     * Write a Markdown version of every HTML page of this build next to it (`about.html` → `about.md`,
+     * `docs/index.html` → `docs/index.md`), converted by llms.ink from the local output (`POST /api/v0/markdown`):
+     * chrome and hidden-instruction text dropped, relative links resolved against `site`. Existing `.md` files are
+     * never overwritten; pages over 5 MB are skipped. Default `false`.
+     */
+    markdownPages?: boolean | MarkdownPagesOptions;
+    /** Fail the build when a file can't be fetched or a page can't be converted. Default `false`: warn and go on. */
     strict?: boolean;
     /** Milliseconds to wait for llms.ink per file. Default 300000 (building a big site can take minutes). */
     timeout?: number;
@@ -30,8 +44,12 @@ export interface Log {
     info(message: string): void;
     warn(message: string): void;
 }
+/** The endpoint's body limit. */
+export declare const MAX_PAGE_BYTES = 5000000;
 /**
  * A writer that fetches the files once and writes them into `dir` on every call (watch builds call it again;
- * a failed fetch is retried on the next call).
+ * a failed fetch is retried on the next call). Markdown pages are converted on every call: they describe this build.
  */
 export declare function llmsFilesWriter(options: LlmsFilesOptions): (dir: string, log: Log) => Promise<void>;
+/** `docs/index.html` → `/docs/`, `about.html` → `/about`, `index.html` → `/`. */
+export declare function pagePath(file: string): string;

package/dist/files.js

@@ -1,44 +1,50 @@
 /**
  * Shared by the Vite plugin and the Astro integration: fetch llms.txt (and llms-full.txt) for a site and write
- * them into a build's output directory.
+ * them into a build's output directory; with `markdownPages`, write a `.md` version next to every built HTML page.
  *
  * @module
  */
-import { access, mkdir, writeFile } from 'node:fs/promises';
+import { access, mkdir, readdir, readFile, stat, writeFile } from 'node:fs/promises';
 import { join } from 'node:path';
-import { LlmsInk } from "./index.js";
+import { setTimeout as wait } from 'node:timers/promises';
+import { LlmsInk, LlmsInkError } from "./index.js";
 const FILES = [['llms.txt', 'llms'], ['llms-full.txt', 'full']];
+/** The endpoint's body limit. */
+export const MAX_PAGE_BYTES = 5_000_000;
+const PAGE_TIMEOUT = 60_000;
+/** Tries per page when llms.ink says to slow down (429 with Retry-After). */
+const RATE_TRIES = 4;
+const DEFAULT_EXCLUDE = /^\/(404|500)(\.html|\/)?$/;
+const reason = (error) => (error instanceof Error ? error.message : String(error));
 /**
  * A writer that fetches the files once and writes them into `dir` on every call (watch builds call it again;
- * a failed fetch is retried on the next call).
+ * a failed fetch is retried on the next call). Markdown pages are converted on every call: they describe this build.
  */
 export function llmsFilesWriter(options) {
     let fetched;
     return async (dir, log) => {
         const { site, strict = false } = options;
-        if (!site) {
-            const message = 'No `site` set: llms.txt not written.';
-            if (strict)
-                throw new Error(message);
-            return log.warn(message);
-        }
-        fetched ??= fetchFiles(options);
-        const results = await fetched;
         const failures = [];
-        for (const result of results) {
-            const path = join(dir, result.name);
-            if ('text' in result) {
-                await mkdir(dir, { recursive: true });
-                await writeFile(path, result.text);
-                log.info(`Wrote ${result.name} for ${site} (${result.source}, ~${result.tokens} tokens).`);
-                continue;
+        if (!site)
+            failures.push('No `site` set: llms.txt not written.');
+        else {
+            fetched ??= fetchFiles(options);
+            for (const result of await fetched) {
+                const path = join(dir, result.name);
+                if ('text' in result) {
+                    await mkdir(dir, { recursive: true });
+                    await writeFile(path, result.text);
+                    log.info(`Wrote ${result.name} for ${site} (${result.source}, ~${result.tokens} tokens).`);
+                    continue;
+                }
+                const kept = await access(path).then(() => 'kept the existing file', () => 'none written');
+                failures.push(`Couldn't get ${result.name} for ${site}: ${reason(result.error)} (${kept}).`);
             }
-            const reason = result.error instanceof Error ? result.error.message : String(result.error);
-            const kept = await access(path).then(() => 'kept the existing file', () => 'none written');
-            failures.push(`Couldn't get ${result.name} for ${site}: ${reason} (${kept}).`);
+            if (failures.length)
+                fetched = undefined;
         }
-        if (failures.length)
-            fetched = undefined;
+        if (options.markdownPages)
+            failures.push(...(await writeMarkdownPages(dir, options, log)));
         if (strict && failures.length)
             throw new Error(failures.join(' '));
         for (const message of failures)
@@ -50,3 +56,74 @@ async function fetchFiles({ site, full, key, native = true, timeout = 300_000, b
     const files = full ? FILES : FILES.slice(0, 1);
     return Promise.all(files.map(([name, mode]) => ink.llms(site, { mode, native, signal: AbortSignal.timeout(timeout) }).then(({ text, tokens, source }) => ({ name, text, tokens, source }), (error) => ({ name, error }))));
 }
+/** Every `.html` file under `dir`, as paths relative to it with `/` separators, sorted. Symlinks aren't followed. */
+async function htmlFiles(dir, prefix = '') {
+    const entries = await readdir(join(dir, prefix), { withFileTypes: true }).catch(() => []);
+    const nested = await Promise.all(entries.map((e) => {
+        const path = prefix ? `${prefix}/${e.name}` : e.name;
+        if (e.isDirectory())
+            return htmlFiles(dir, path);
+        return e.isFile() && e.name.endsWith('.html') ? [path] : [];
+    }));
+    return nested.flat().sort();
+}
+/** `docs/index.html` → `/docs/`, `about.html` → `/about`, `index.html` → `/`. */
+export function pagePath(file) {
+    const path = `/${file.replace(/\.html$/, '')}`;
+    return path.endsWith('/index') ? path.slice(0, -'index'.length) : path;
+}
+/**
+ * Converts each HTML page of `dir` through llms.ink and writes `<page>.md` next to it. Returns the failure messages
+ * (the caller warns or, with `strict`, fails the build).
+ */
+async function writeMarkdownPages(dir, options, log) {
+    const settings = typeof options.markdownPages === 'object' ? options.markdownPages : {};
+    const exclude = settings.exclude ?? DEFAULT_EXCLUDE;
+    const concurrency = Math.max(1, Math.floor(settings.concurrency ?? 4));
+    const ink = new LlmsInk({ key: options.key, baseUrl: options.baseUrl });
+    const site = options.site ? new URL(options.site.endsWith('/') ? options.site : `${options.site}/`) : null;
+    const pages = (await htmlFiles(dir)).filter((file) => !exclude.test(pagePath(file)));
+    const failures = [];
+    let written = 0;
+    let skipped = 0;
+    const convert = async (file) => {
+        const target = join(dir, file.replace(/\.html$/, '.md'));
+        const md = file.replace(/\.html$/, '.md');
+        if (await access(target).then(() => true, () => false)) {
+            skipped++;
+            return log.info(`Kept ${md}: it already exists.`);
+        }
+        const source = join(dir, file);
+        if ((await stat(source)).size > MAX_PAGE_BYTES) {
+            skipped++;
+            return log.warn(`Skipped ${file}: over ${MAX_PAGE_BYTES.toLocaleString('en-US')} bytes.`);
+        }
+        const html = await readFile(source, 'utf8');
+        // Page URL under `site` (its path included: Astro's `base`).
+        const url = site ? new URL(pagePath(file).slice(1), site).href : undefined;
+        for (let attempt = 1;; attempt++) {
+            try {
+                const text = await ink.markdown(html, { url, signal: AbortSignal.timeout(PAGE_TIMEOUT) });
+                await writeFile(target, text);
+                written++;
+                return;
+            }
+            catch (error) {
+                const limited = error instanceof LlmsInkError && error.status === 429 && error.code === 'rate_limited';
+                if (limited && attempt < RATE_TRIES) {
+                    await wait(Math.min(error.retryAfter ?? 10, 60) * 1000);
+                    continue;
+                }
+                failures.push(`Couldn't convert ${file} to Markdown: ${reason(error).replace(/\.$/, '')}.`);
+                return;
+            }
+        }
+    };
+    let next = 0;
+    await Promise.all(Array.from({ length: Math.min(concurrency, pages.length) }, async () => {
+        while (next < pages.length)
+            await convert(pages[next++]);
+    }));
+    log.info(`Wrote ${written} Markdown page${written === 1 ? '' : 's'}${skipped ? ` (${skipped} skipped)` : ''}${failures.length ? `; ${failures.length} failed` : ''}.`);
+    return failures;
+}

package/dist/index.d.ts

@@ -13,12 +13,12 @@
  *
  * @module
  */
-export declare const VERSION = "0.3.0";
+export declare const VERSION = "0.4.0";
 export declare const DEFAULT_BASE_URL = "https://llms.ink";
 /** `llms` for the index (llms.txt), `full` for llms-full.txt. */
 export type Mode = 'llms' | 'full';
 /** What the link is. */
-export type Kind = 'site' | 'section' | 'page' | 'repo' | 'document' | 'video';
+export type Kind = 'site' | 'section' | 'page' | 'repo' | 'document' | 'video' | 'channel';
 /** `native` = published by the site; `generated` = made by llms.ink. */
 export type Source = 'native' | 'generated';
 /** A transcript line; `start` and `end` are seconds into the source video. */
@@ -137,7 +137,7 @@ export interface CheckResult {
     checkedAt: string;
 }
 /** Every error code the API answers with (https://llms.ink/docs/#errors). */
-export type ErrorCode = 'invalid_request' | 'invalid_url' | 'invalid_key' | 'not_found' | 'unknown_version' | 'method_not_allowed' | 'payload_too_large' | 'unprocessable' | 'rate_limited' | 'quota_exceeded' | 'internal' | 'upstream_error' | 'unavailable' | 'timeout' | 'loop_detected';
+export type ErrorCode = 'invalid_request' | 'invalid_url' | 'invalid_key' | 'key_required' | 'site_claimed' | 'not_found' | 'unknown_version' | 'method_not_allowed' | 'payload_too_large' | 'too_large' | 'unsupported_media_type' | 'unprocessable' | 'rate_limited' | 'quota_exceeded' | 'internal' | 'upstream_error' | 'unavailable' | 'timeout' | 'loop_detected';
 /** `invalid_request` only: one invalid parameter. */
 export interface Issue {
     path: string;
@@ -244,6 +244,12 @@ export interface CheckTextOptions {
     links?: boolean;
     signal?: AbortSignal;
 }
+/** Options for {@link LlmsInk.markdown}. */
+export interface MarkdownOptions {
+    /** Where the page lives (never fetched): relative links in the Markdown resolve against it. */
+    url?: string;
+    signal?: AbortSignal;
+}
 /**
  * A client for https://llms.ink.
  *
@@ -266,6 +272,11 @@ export declare class LlmsInk {
     check(url: string, options?: CheckOptions): Promise<CheckResult>;
     /** Parses and checks llms.txt text you already have (`POST /api/v0/check`); nothing is published. */
     checkText(text: string, options?: CheckTextOptions): Promise<CheckResult>;
+    /**
+     * A page's HTML as Markdown (`POST /api/v0/markdown`, up to 5 MB): hidden-instruction text, scripts, navigation,
+     * headers and footers dropped, the way llms.ink reads pages. Nothing is fetched or stored.
+     */
+    markdown(html: string, options?: MarkdownOptions): Promise<string>;
     /**
      * Like {@link LlmsInk.llms}, as the build happens: yields `progress` and `partial` events, then one
      * `result`. Errors — including a terminal `error` event — are thrown as {@link LlmsInkError}s.

package/dist/index.js

@@ -14,7 +14,7 @@
  * @module
  */
 import { env } from "./env.js";
-export const VERSION = '0.3.0';
+export const VERSION = '0.4.0';
 export const DEFAULT_BASE_URL = 'https://llms.ink';
 /**
  * An API error, or llms.ink couldn't be reached (then `status` and `code` are undefined).
@@ -117,6 +117,15 @@ export class LlmsInk {
         const res = await this.#request('/api/v0/check', params, 'application/json', options.signal, text);
         return (await this.#json(res));
     }
+    /**
+     * A page's HTML as Markdown (`POST /api/v0/markdown`, up to 5 MB): hidden-instruction text, scripts, navigation,
+     * headers and footers dropped, the way llms.ink reads pages. Nothing is fetched or stored.
+     */
+    async markdown(html, options = {}) {
+        const params = options.url ? { url: options.url } : {};
+        const res = await this.#request('/api/v0/markdown', params, 'text/markdown', options.signal, html, 'text/html; charset=utf-8');
+        return res.text();
+    }
     /**
      * Like {@link LlmsInk.llms}, as the build happens: yields `progress` and `partial` events, then one
      * `result`. Errors — including a terminal `error` event — are thrown as {@link LlmsInkError}s.
@@ -183,10 +192,10 @@ export class LlmsInk {
         const known = event.type === 'progress' || event.type === 'partial' || event.type === 'result';
         return known ? event : undefined;
     }
-    async #request(path, params, accept, signal, body) {
+    async #request(path, params, accept, signal, body, type = 'text/markdown; charset=utf-8') {
         const headers = { accept };
         if (body !== undefined)
-            headers['content-type'] = 'text/markdown; charset=utf-8';
+            headers['content-type'] = type;
         if (!IS_BROWSER)
             headers['user-agent'] = `llms-ink-js/${VERSION}`;
         if (this.key)

package/dist/middleware.d.ts

@@ -1,7 +1,7 @@
 /**
- * Serve llms.txt, llms-full.txt and (optionally) Markdown versions of pages from llms.ink, inside your own app.
- * A fetch-API handler for Cloudflare Workers, Next.js middleware, Deno, Bun and Hono, plus an Express/Connect
- * adapter.
+ * Serve llms.txt, llms-full.txt and (optionally) Markdown versions of pages from llms.ink, inside your own app, and
+ * (optionally) count AI-agent traffic. A fetch-API handler for Cloudflare Workers, Next.js middleware, Deno, Bun and
+ * Hono, plus an Express/Connect adapter.
  *
  * @example
  * ```ts
@@ -9,14 +9,16 @@
  *
  * const llms = llmsInkHandler({ site: 'https://example.com', markdown: true });
  * export default {
- *   async fetch(request: Request): Promise<Response> {
- *     return (await llms(request)) ?? fetch(request); // undefined: not ours, the app answers
+ *   async fetch(request: Request, env: unknown, ctx: ExecutionContext): Promise<Response> {
+ *     return (await llms(request, ctx)) ?? fetch(request); // undefined: not ours, the app answers
  *   },
  * };
  * ```
  *
  * @module
  */
+import { type TrafficOptions, type WaitUntil } from './traffic.ts';
+export { AI_AGENTS, classifyAgent, trafficCounter, type AgentKind, type AiAgent, type TrafficBatch, type TrafficCounter, type TrafficOptions, type TrafficRow, type WaitUntil } from './traffic.ts';
 /** Options for {@link llmsInkHandler} and {@link llmsInkExpress}. */
 export interface LlmsInkHandlerOptions {
     /** The deployed site to describe, e.g. `https://example.com`. */
@@ -33,9 +35,22 @@ export interface LlmsInkHandlerOptions {
     markdown?: boolean;
     /** Defaults to the `LLMS_INK_API` environment variable, else https://llms.ink. */
     baseUrl?: string;
+    /**
+     * Count requests from AI agents (ChatGPT-User, Claude-User, GPTBot, ClaudeBot… see `AI_AGENTS`) per path and send
+     * daily aggregates to llms.ink every `flushEvery` seconds; read them at https://llms.ink/traffic/. Sends only
+     * `{ path, agent, kind, count }`: no IPs, user agents, query strings or cookies. Off unless set.
+     */
+    traffic?: TrafficOptions;
+}
+/**
+ * A fetch-API handler: a `Response` for the requests it serves, `undefined` for everything else. Pass the
+ * platform's `ctx` (Workers' `ExecutionContext`, Next.js's `NextFetchEvent`) so traffic sends outlive the response.
+ */
+export interface LlmsInkHandler {
+    (request: Request, ctx?: WaitUntil): Promise<Response | undefined>;
+    /** Sends the traffic counted so far (with `traffic`); resolves at once without it. */
+    flush(): Promise<void>;
 }
-/** A fetch-API handler: a `Response` for the requests it serves, `undefined` for everything else. */
-export type LlmsInkHandler = (request: Request) => Promise<Response | undefined>;
 /**
  * Answers `GET`/`HEAD` `/llms.txt` and `/llms-full.txt` with llms.ink's generated files for `site`
  * (`native=0`, so a site whose /llms.txt is this handler can't loop), and, with `markdown: true`, Markdown
@@ -59,6 +74,10 @@ export interface NodeResponse {
     end(chunk: Uint8Array): unknown;
 }
 /** Express/Connect middleware: `next()` when the request isn't one {@link llmsInkHandler} serves. */
-export type NodeMiddleware = (req: NodeRequest, res: NodeResponse, next: (err?: unknown) => void) => void;
+export interface NodeMiddleware {
+    (req: NodeRequest, res: NodeResponse, next: (err?: unknown) => void): void;
+    /** Sends the traffic counted so far (with `traffic`). */
+    flush(): Promise<void>;
+}
 /** {@link llmsInkHandler} as Express/Connect middleware: `app.use(llmsInkExpress({ site }))`. */
 export declare function llmsInkExpress(options: LlmsInkHandlerOptions): NodeMiddleware;

package/dist/middleware.js

@@ -1,7 +1,7 @@
 /**
- * Serve llms.txt, llms-full.txt and (optionally) Markdown versions of pages from llms.ink, inside your own app.
- * A fetch-API handler for Cloudflare Workers, Next.js middleware, Deno, Bun and Hono, plus an Express/Connect
- * adapter.
+ * Serve llms.txt, llms-full.txt and (optionally) Markdown versions of pages from llms.ink, inside your own app, and
+ * (optionally) count AI-agent traffic. A fetch-API handler for Cloudflare Workers, Next.js middleware, Deno, Bun and
+ * Hono, plus an Express/Connect adapter.
  *
  * @example
  * ```ts
@@ -9,8 +9,8 @@
  *
  * const llms = llmsInkHandler({ site: 'https://example.com', markdown: true });
  * export default {
- *   async fetch(request: Request): Promise<Response> {
- *     return (await llms(request)) ?? fetch(request); // undefined: not ours, the app answers
+ *   async fetch(request: Request, env: unknown, ctx: ExecutionContext): Promise<Response> {
+ *     return (await llms(request, ctx)) ?? fetch(request); // undefined: not ours, the app answers
  *   },
  * };
  * ```
@@ -19,6 +19,8 @@
  */
 import { env } from "./env.js";
 import { DEFAULT_BASE_URL, VERSION } from "./index.js";
+import { trafficCounter } from "./traffic.js";
+export { AI_AGENTS, classifyAgent, trafficCounter } from "./traffic.js";
 const FILES = { '/llms.txt': 'llms', '/llms-full.txt': 'full' };
 /** llms.ink's own fetcher (`compatible; llms.ink/0.1`): answering it from llms.ink would loop. */
 const LLMS_INK_FETCHER = /\bllms\.ink\/\d/;
@@ -44,6 +46,7 @@ export function llmsInkHandler(options) {
         headers.authorization = `Bearer ${key}`;
     const cache = new Map();
     const pending = new Map();
+    const traffic = options.traffic ? trafficCounter({ ...options.traffic, site: options.traffic.site ?? site.hostname, baseUrl: base }) : null;
     /** Fetches `upstream`; on failure the last good copy, else llms.ink's error. */
     async function refresh(upstream) {
         const last = cache.get(upstream);
@@ -85,7 +88,8 @@ export function llmsInkHandler(options) {
         // Concurrent requests share one fetch; each gets its own copy of the body.
         return (await loading).clone();
     }
-    return async (request) => {
+    const handler = async (request, ctx) => {
+        traffic?.record(request, ctx);
         if (request.method !== 'GET' && request.method !== 'HEAD')
             return undefined;
         if (LLMS_INK_FETCHER.test(request.headers.get('user-agent') ?? ''))
@@ -110,11 +114,12 @@ export function llmsInkHandler(options) {
         }
         return request.method === 'HEAD' ? new Response(null, response) : response;
     };
+    return Object.assign(handler, { flush: () => traffic?.flush() ?? Promise.resolve() });
 }
 /** {@link llmsInkHandler} as Express/Connect middleware: `app.use(llmsInkExpress({ site }))`. */
 export function llmsInkExpress(options) {
     const handle = llmsInkHandler(options);
-    return (req, res, next) => {
+    const middleware = (req, res, next) => {
         const headers = new Headers();
         for (const name of ['accept', 'user-agent']) {
             const value = req.headers[name];
@@ -132,6 +137,7 @@ export function llmsInkExpress(options) {
         })
             .catch(next);
     };
+    return Object.assign(middleware, { flush: handle.flush });
 }
 function pick(from, names) {
     const to = new Headers();

package/dist/traffic.d.ts

@@ -0,0 +1,68 @@
+/**
+ * AI traffic: which AI agents fetch which pages of your site, counted in memory and sent to llms.ink as daily
+ * aggregates (`POST /api/v0/traffic`). Only requests from the agents in {@link AI_AGENTS} are counted; what leaves
+ * your server is `{ site, period, rows: [{ path, agent, kind, count }] }` — no IPs, no user agents, no query
+ * strings, no cookies, nothing about people who visit.
+ *
+ * @module
+ */
+/** `user`: fetched because someone asked an assistant; `crawler`: indexing for search or collecting for training. */
+export type AgentKind = 'user' | 'crawler';
+/** One AI agent: the name it puts in its User-Agent (matched case-insensitively), what kind it is, who runs it. */
+export interface AiAgent {
+    agent: string;
+    kind: AgentKind;
+    operator: string;
+}
+/**
+ * The agents counted, from the operators' published bot lists. Robots-only tokens (`Google-Extended`,
+ * `Applebot-Extended`) never appear in a User-Agent, so they aren't here: Googlebot and Applebot do that crawling.
+ */
+export declare const AI_AGENTS: readonly AiAgent[];
+/** The AI agent behind a User-Agent header, or null for browsers, other bots and anything unknown. */
+export declare function classifyAgent(userAgent: string | null | undefined): AiAgent | null;
+/** Options for {@link trafficCounter}, and the middleware's `traffic` option. */
+export interface TrafficOptions {
+    /** API key (`lk_…`). The first key to send counts for a site owns it; read them back with the same key. */
+    key: string;
+    /** Hostname the counts are filed under, e.g. `example.com`. The middleware defaults it to its `site`. */
+    site?: string;
+    /** Seconds between sends. Default 60, at least 10. */
+    flushEvery?: number;
+}
+/** One aggregate: requests from one agent to one path in one UTC day. */
+export interface TrafficRow {
+    path: string;
+    agent: string;
+    kind: AgentKind;
+    count: number;
+}
+/** The body of `POST /api/v0/traffic`. */
+export interface TrafficBatch {
+    site: string;
+    /** UTC day, `YYYY-MM-DD`. */
+    period: string;
+    rows: TrafficRow[];
+}
+/** Workers' `ExecutionContext`, Next.js's `NextFetchEvent`: keeps a send alive after the response. */
+export interface WaitUntil {
+    waitUntil(promise: Promise<unknown>): void;
+}
+/** Counts AI-agent requests and sends them in the background. */
+export interface TrafficCounter {
+    /**
+     * Counts the request when an AI agent sent it (`GET`/`HEAD` only). When a send is due it starts one and hands it
+     * to `ctx.waitUntil`; without `ctx`, a background timer sends. Never throws and never waits.
+     */
+    record(request: Request, ctx?: WaitUntil): void;
+    /** Sends everything counted so far. Never rejects; failed sends are kept and retried with backoff. */
+    flush(): Promise<void>;
+}
+/**
+ * A counter for one site. Use it directly in servers the middleware doesn't cover:
+ * `counter.record(request, ctx)` for each request, `await counter.flush()` before shutting down.
+ */
+export declare function trafficCounter(options: TrafficOptions & {
+    site: string;
+    baseUrl?: string;
+}): TrafficCounter;

package/dist/traffic.js

@@ -0,0 +1,221 @@
+/**
+ * AI traffic: which AI agents fetch which pages of your site, counted in memory and sent to llms.ink as daily
+ * aggregates (`POST /api/v0/traffic`). Only requests from the agents in {@link AI_AGENTS} are counted; what leaves
+ * your server is `{ site, period, rows: [{ path, agent, kind, count }] }` — no IPs, no user agents, no query
+ * strings, no cookies, nothing about people who visit.
+ *
+ * @module
+ */
+import { DEFAULT_BASE_URL, VERSION } from "./index.js";
+/**
+ * The agents counted, from the operators' published bot lists. Robots-only tokens (`Google-Extended`,
+ * `Applebot-Extended`) never appear in a User-Agent, so they aren't here: Googlebot and Applebot do that crawling.
+ */
+export const AI_AGENTS = [
+    // Fetches on behalf of someone asking an assistant right now.
+    { agent: 'ChatGPT-User', kind: 'user', operator: 'OpenAI' },
+    { agent: 'Claude-User', kind: 'user', operator: 'Anthropic' },
+    { agent: 'Perplexity-User', kind: 'user', operator: 'Perplexity' },
+    { agent: 'MistralAI-User', kind: 'user', operator: 'Mistral' },
+    { agent: 'DuckAssistBot', kind: 'user', operator: 'DuckDuckGo' },
+    { agent: 'meta-externalfetcher', kind: 'user', operator: 'Meta' },
+    { agent: 'Google-NotebookLM', kind: 'user', operator: 'Google' },
+    { agent: 'Amzn-User', kind: 'user', operator: 'Amazon' },
+    // Crawlers: AI search indexes, training collection, and the search engines AI answers draw on.
+    { agent: 'OAI-SearchBot', kind: 'crawler', operator: 'OpenAI' },
+    { agent: 'GPTBot', kind: 'crawler', operator: 'OpenAI' },
+    { agent: 'Claude-SearchBot', kind: 'crawler', operator: 'Anthropic' },
+    { agent: 'ClaudeBot', kind: 'crawler', operator: 'Anthropic' },
+    { agent: 'anthropic-ai', kind: 'crawler', operator: 'Anthropic' },
+    { agent: 'PerplexityBot', kind: 'crawler', operator: 'Perplexity' },
+    { agent: 'Google-CloudVertexBot', kind: 'crawler', operator: 'Google' },
+    { agent: 'Googlebot', kind: 'crawler', operator: 'Google' },
+    { agent: 'bingbot', kind: 'crawler', operator: 'Microsoft' },
+    { agent: 'Applebot', kind: 'crawler', operator: 'Apple' },
+    { agent: 'Amzn-SearchBot', kind: 'crawler', operator: 'Amazon' },
+    { agent: 'Amazonbot', kind: 'crawler', operator: 'Amazon' },
+    { agent: 'meta-externalagent', kind: 'crawler', operator: 'Meta' },
+    { agent: 'Bytespider', kind: 'crawler', operator: 'ByteDance' },
+    { agent: 'CCBot', kind: 'crawler', operator: 'Common Crawl' },
+    { agent: 'cohere-ai', kind: 'crawler', operator: 'Cohere' },
+    { agent: 'YouBot', kind: 'crawler', operator: 'You.com' },
+    { agent: 'DuckDuckBot', kind: 'crawler', operator: 'DuckDuckGo' },
+];
+const BY_TOKEN = new Map(AI_AGENTS.map((a) => [a.agent.toLowerCase(), a]));
+/** One pass over the User-Agent: a known name not glued to a preceding letter, digit or hyphen. */
+const MATCH = new RegExp(`(?:^|[^a-z0-9-])(${AI_AGENTS.map((a) => a.agent.replace(/[.*+?^${}()|[\]\\-]/g, '\\$&')).join('|')})`, 'i');
+/** The AI agent behind a User-Agent header, or null for browsers, other bots and anything unknown. */
+export function classifyAgent(userAgent) {
+    if (!userAgent)
+        return null;
+    const match = MATCH.exec(userAgent);
+    return match ? BY_TOKEN.get(match[1].toLowerCase()) ?? null : null;
+}
+/** Rows per POST (the API's limit). */
+const ROWS_PER_POST = 1000;
+/** Distinct path/agent/day rows kept in memory; past it, new paths count as `(other)`. */
+const MAX_PENDING = 5000;
+/** The path new rows are folded into once {@link MAX_PENDING} is reached. */
+const OTHER_PATH = '/(other)';
+const MAX_PATH = 512;
+const MAX_BACKOFF_MS = 3_600_000;
+const SEND_TIMEOUT_MS = 10_000;
+/** The path only: no query string or fragment (URL.pathname has neither), capped in length. */
+function cleanPath(url) {
+    const path = new URL(url).pathname || '/';
+    return path.length > MAX_PATH ? path.slice(0, MAX_PATH) : path;
+}
+/**
+ * A counter for one site. Use it directly in servers the middleware doesn't cover:
+ * `counter.record(request, ctx)` for each request, `await counter.flush()` before shutting down.
+ */
+export function trafficCounter(options) {
+    const key = options.key?.trim();
+    if (!key)
+        throw new TypeError('traffic needs an llms.ink API key (traffic: { key: "lk_…" }).');
+    // `https://Example.com/x` or `example.com` → `example.com`.
+    const site = new URL(/^https?:\/\//i.test(options.site) ? options.site : `https://${options.site}`).hostname.toLowerCase();
+    const every = Math.max(10, options.flushEvery ?? 60) * 1000;
+    const endpoint = `${(options.baseUrl ?? DEFAULT_BASE_URL).replace(/\/+$/, '')}/api/v0/traffic`;
+    const headers = { authorization: `Bearer ${key}`, 'content-type': 'application/json', 'user-agent': `llms-ink-js/${VERSION}` };
+    /** `period\tpath\tagent` → row. */
+    const pending = new Map();
+    let lastSend = Date.now();
+    let retryAt = 0;
+    let backoff = 0;
+    let inflight = null;
+    let timer;
+    /** Set when llms.ink refuses the key or the site: counting stops, once, with a warning. */
+    let stopped = false;
+    function add(period, path, agent, kind, count) {
+        let id = `${period}\t${path}\t${agent}`;
+        let row = pending.get(id);
+        if (!row && pending.size >= MAX_PENDING) {
+            path = OTHER_PATH;
+            id = `${period}\t${path}\t${agent}`;
+            row = pending.get(id);
+        }
+        if (row)
+            row.count += count;
+        else
+            pending.set(id, { period, path, agent, kind, count });
+    }
+    function stop(message) {
+        stopped = true;
+        pending.clear();
+        clearTimeout(timer);
+        console.warn(`llms-ink traffic: ${message} Counting stopped.`);
+    }
+    function retryLater(rows, retryAfter) {
+        for (const r of rows)
+            add(r.period, r.path, r.agent, r.kind, r.count);
+        backoff = Math.min(MAX_BACKOFF_MS, Math.max(every, backoff * 2));
+        retryAt = Date.now() + Math.max(backoff, (retryAfter ?? 0) * 1000);
+    }
+    /** One POST; resolves to what to do with its rows. */
+    async function send(batch) {
+        let res;
+        try {
+            res = await fetch(endpoint, { method: 'POST', headers, body: JSON.stringify(batch), signal: AbortSignal.timeout(SEND_TIMEOUT_MS) });
+        }
+        catch {
+            return { retryAfter: null };
+        }
+        if (res.ok) {
+            await res.body?.cancel();
+            return 'sent';
+        }
+        // llms.ink's problem details (`code`, `detail`) say why a batch was refused.
+        const problem = await res.json().catch(() => null);
+        const why = problem ? `${problem.code ?? ''}: ${problem.detail ?? ''}` : '';
+        if (res.status === 429 || res.status >= 500)
+            return { retryAfter: Number(res.headers.get('retry-after')) || null };
+        if (res.status === 401 || res.status === 403) {
+            stop(`llms.ink refused ${site} (${res.status} ${why || 'check the key'}).`);
+            return 'stop';
+        }
+        console.warn(`llms-ink traffic: llms.ink rejected a batch for ${site} (${res.status} ${why}).`);
+        return 'drop';
+    }
+    async function sendAll(rows) {
+        const batches = [];
+        const byPeriod = new Map();
+        for (const r of rows)
+            byPeriod.set(r.period, [...(byPeriod.get(r.period) ?? []), r]);
+        for (const [period, list] of byPeriod) {
+            for (let i = 0; i < list.length; i += ROWS_PER_POST) {
+                const chunk = list.slice(i, i + ROWS_PER_POST);
+                batches.push({ batch: { site, period, rows: chunk.map(({ path, agent, kind, count }) => ({ path, agent, kind, count })) }, rows: chunk });
+            }
+        }
+        for (let i = 0; i < batches.length; i++) {
+            const outcome = await send(batches[i].batch);
+            if (outcome === 'stop')
+                return;
+            if (typeof outcome === 'object') {
+                retryLater(batches.slice(i).flatMap((b) => b.rows), outcome.retryAfter);
+                return;
+            }
+        }
+        backoff = 0;
+        retryAt = 0;
+    }
+    /** Waits for a send in flight, then sends what was counted meanwhile. */
+    async function flush() {
+        while (inflight)
+            await inflight;
+        if (!pending.size || stopped)
+            return;
+        clearTimeout(timer);
+        timer = undefined;
+        const rows = [...pending.values()];
+        pending.clear();
+        lastSend = Date.now();
+        inflight = sendAll(rows)
+            .catch(() => { })
+            .finally(() => {
+            inflight = null;
+            if (pending.size && !stopped)
+                arm();
+        });
+        await inflight;
+    }
+    /** A background send where nothing keeps one alive (Node, Bun, Deno); unref'd so it never holds the process open. */
+    function arm() {
+        if (timer)
+            return;
+        const delay = Math.max(1000, Math.max(lastSend + every, retryAt) - Date.now());
+        timer = setTimeout(() => {
+            timer = undefined;
+            void flush();
+        }, delay);
+        // Node's and Bun's timers have unref(); elsewhere setTimeout returns a number.
+        if (typeof timer === 'object' && 'unref' in timer)
+            timer.unref();
+    }
+    return {
+        record(request, ctx) {
+            try {
+                if (stopped || (request.method !== 'GET' && request.method !== 'HEAD'))
+                    return;
+                const agent = classifyAgent(request.headers.get('user-agent'));
+                if (!agent)
+                    return;
+                const now = Date.now();
+                add(new Date(now).toISOString().slice(0, 10), cleanPath(request.url), agent.agent, agent.kind, 1);
+                const due = now - lastSend >= every || pending.size >= ROWS_PER_POST;
+                if (!inflight && now >= retryAt && due) {
+                    const sending = flush();
+                    ctx?.waitUntil(sending);
+                }
+                else if (!ctx) {
+                    arm();
+                }
+            }
+            catch {
+                // Counting must never break the site.
+            }
+        },
+        flush,
+    };
+}

package/dist/vite.d.ts

@@ -1,5 +1,6 @@
 /**
- * Vite plugin: writes llms.txt (and llms-full.txt) from llms.ink into the build output.
+ * Vite plugin: writes llms.txt (and llms-full.txt) from llms.ink into the build output, and with
+ * `markdownPages: true` a Markdown version next to every built HTML page.
  *
  * @example
  * ```ts
@@ -13,13 +14,14 @@
  * @module
  */
 import type { Plugin } from 'vite';
-import { type LlmsFilesOptions } from './files.ts';
-export type { LlmsFilesOptions };
+import { type LlmsFilesOptions, type MarkdownPagesOptions } from './files.ts';
+export type { LlmsFilesOptions, MarkdownPagesOptions };
 /**
  * After `vite build` (not the dev server, not SSR builds), fetches llms.txt — and llms-full.txt with
  * `full: true` — for `site` and writes them into `build.outDir`. llms.ink reads the live site, so the files
  * describe the previous deploy. When llms.ink can't be reached it warns and keeps any existing file (e.g. one
- * copied from `public/`); `strict: true` fails the build instead.
+ * copied from `public/`); `strict: true` fails the build instead. `markdownPages: true` converts each `.html` file
+ * of this build (not the deploy) through `POST /api/v0/markdown` and writes `about.md` / `docs/index.md` next to it.
  */
 export declare function llmsInk(options: LlmsFilesOptions): Plugin;
 export default llmsInk;

package/dist/vite.js

@@ -1,5 +1,6 @@
 /**
- * Vite plugin: writes llms.txt (and llms-full.txt) from llms.ink into the build output.
+ * Vite plugin: writes llms.txt (and llms-full.txt) from llms.ink into the build output, and with
+ * `markdownPages: true` a Markdown version next to every built HTML page.
  *
  * @example
  * ```ts
@@ -18,7 +19,8 @@ import { llmsFilesWriter } from "./files.js";
  * After `vite build` (not the dev server, not SSR builds), fetches llms.txt — and llms-full.txt with
  * `full: true` — for `site` and writes them into `build.outDir`. llms.ink reads the live site, so the files
  * describe the previous deploy. When llms.ink can't be reached it warns and keeps any existing file (e.g. one
- * copied from `public/`); `strict: true` fails the build instead.
+ * copied from `public/`); `strict: true` fails the build instead. `markdownPages: true` converts each `.html` file
+ * of this build (not the deploy) through `POST /api/v0/markdown` and writes `about.md` / `docs/index.md` next to it.
  */
 export function llmsInk(options) {
     const write = llmsFilesWriter(options);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "llms-ink",
-  "version": "0.3.0",
+  "version": "0.4.0",
   "description": "Turn any link — site, docs section, page, GitHub repo, PDF or video — into llms.txt and llms-full.txt. Typed client for llms.ink, plus a Vite plugin, an Astro integration and middleware that serve the files from your site. No dependencies.",
   "type": "module",
   "exports": {
@@ -31,9 +31,9 @@
   "scripts": {
     "build": "rm -rf dist && tsc -p tsconfig.json",
     "prepublishOnly": "npm run build && npm test",
-    "test": "npm run build && node --test test/client.test.mjs test/plugins.test.mjs test/middleware.test.mjs",
+    "test": "npm run build && node --test test/client.test.mjs test/plugins.test.mjs test/middleware.test.mjs test/traffic.test.mjs",
     "test:live": "npm run build && node --test test/live.test.mjs",
-    "test:bun": "npm run build && bun test ./test/client.test.mjs ./test/plugins.test.mjs ./test/middleware.test.mjs",
+    "test:bun": "npm run build && bun test ./test/client.test.mjs ./test/plugins.test.mjs ./test/middleware.test.mjs ./test/traffic.test.mjs",
     "test:bun:live": "npm run build && bun test --timeout 300000 ./test/live.test.mjs"
   },
   "keywords": ["llms.txt", "llms-full.txt", "llm", "ai", "markdown", "docs", "context", "rag", "sdk", "llms-ink", "vite-plugin", "astro-integration", "middleware"],