llms-ink 0.1.0 → 0.3.0

package/README.md

@@ -2,7 +2,8 @@
 
 Turn any link — site, docs section, page, GitHub repo, PDF or video — into llms.txt and llms-full.txt. A typed
 client for the [llms.ink](https://llms.ink) API with no dependencies. Runs on Node 18+, Bun, Deno, Cloudflare
-Workers and browsers.
+Workers and browsers. Also a [Vite plugin, an Astro integration and middleware](#serve-llmstxt-from-your-site) that
+serve llms.txt from your own site.
 
 ```sh
 npm install llms-ink        # or: bun add llms-ink
@@ -32,10 +33,11 @@ console.log(report.found, report.summary, report.issues);
 - `baseUrl`: defaults to `LLMS_INK_API`, else `https://llms.ink`.
 - `fetch`: a `fetch` to use instead of the global one.
 
-### `ink.llms(url, { mode?, fresh?, signal? })` → `Promise<Result>`
+### `ink.llms(url, { mode?, fresh?, native?, signal? })` → `Promise<Result>`
 
 `mode` is `'llms'` (default) or `'full'` (up to 20 pages inlined). `fresh: true` skips the cache and rebuilds
-(counts against the daily quota). The result:
+(counts against the daily quota). `native: false` always generates the file and never returns the site's own
+llms.txt (by default a site's own file is returned as is, unless llms.ink generated it). The result:
 
 ```ts
 { url, kind, source, mode, title, text, tokens, pages, model, cached, generatedAt, media? }
@@ -50,7 +52,27 @@ Building a big site or a video can take minutes. Bound a call with a signal:
 ### `ink.check(url, { signal? })` → `Promise<CheckResult>`
 
 Checks the site's (or docs section's) published llms.txt against the llmstxt.org format and fetches its first
-25 links: `{ site, url, found, status, title, tokens, sections, links, llmsFull, issues, summary, checkedAt }`.
+25 links: `{ site, url, found, status, title, tokens, sections, links, llmsFull, parsed, issues, summary, checkedAt }`.
+
+`parsed` is the file's structure (`null` when nothing was found):
+
+```ts
+{ title, summary, details, sections: [{ name, optional, text, links: [{ title, url, notes, line }] }] }
+```
+
+### `ink.checkText(text, { base?, links?, signal? })` → `Promise<CheckResult>`
+
+Parses and checks llms.txt text you already have, e.g. before publishing it:
+
+```ts
+const { parsed, issues } = await ink.checkText(await readFile('public/llms.txt', 'utf8'), {
+  base: 'https://example.com/llms.txt', // resolves relative links; reported as url/site
+  links: true,                          // also fetch up to 25 links and look for llms-full.txt
+});
+```
+
+Without `base`, `site` and `url` are `null`; without `base` and `links`, `llmsFull` is `null` and
+`links.checked` is 0. Up to 2 MB.
 
 ### `ink.stream(url, { mode?, fresh?, signal? })` → `AsyncGenerator<StreamEvent>`
 
@@ -66,6 +88,55 @@ for await (const event of ink.stream('docs.astro.build', { mode: 'full' })) {
 
 Breaking out of the loop cancels the request.
 
+## Serve llms.txt from your site
+
+### Vite: `llms-ink/vite`
+
+```ts
+// vite.config.ts
+import { defineConfig } from 'vite';
+import { llmsInk } from 'llms-ink/vite';
+
+export default defineConfig({ plugins: [llmsInk({ site: 'https://example.com', full: true })] });
+```
+
+After `vite build` (not the dev server or SSR builds), writes `llms.txt` (and `llms-full.txt` with `full`) for
+`site` into `build.outDir`. llms.ink reads the live site, so the files describe the previous deploy. If llms.ink
+fails, the plugin warns and keeps any existing file (e.g. one from `public/`); `strict: true` fails the build.
+Options: `site`, `full`, `key` (default `LLMS_INK_KEY`), `native` (default `true`; a file llms.ink generated before
+is regenerated, not echoed), `strict`, `timeout` (ms, default 300000), `baseUrl`.
+
+### Astro: `llms-ink/astro`
+
+```ts
+// astro.config.mjs
+import { defineConfig } from 'astro/config';
+import llmsInk from 'llms-ink/astro';
+
+export default defineConfig({ site: 'https://example.com', integrations: [llmsInk({ full: true })] });
+```
+
+The same, on `astro:build:done`, into the static output. `site` defaults to Astro's `site` plus `base`.
+
+### Middleware: `llms-ink/middleware`
+
+```ts
+import { llmsInkHandler, llmsInkExpress } from 'llms-ink/middleware';
+
+const llms = llmsInkHandler({ site: 'https://example.com', markdown: true });
+const response = (await llms(request)) ?? (await app(request)); // Workers, Next.js proxy, Deno, Bun, Hono
+
+expressApp.use(llmsInkExpress({ site: 'https://example.com' })); // Express, Connect
+```
+
+Answers `GET`/`HEAD` `/llms.txt` and `/llms-full.txt` from llms.ink, always generated (`native=0`, so it can't
+loop through itself), with llms.ink's `Cache-Control`. Each answer is kept in memory for its `max-age` and served as
+a last good copy (`Warning: 110`, `Cache-Control: max-age=60`) when llms.ink fails. With `markdown: true` it also
+answers `/<page>.md` and requests whose `Accept` prefers `text/markdown` over `text/html` with the page as Markdown.
+Everything else, and llms.ink's own fetcher, gets `undefined` (Express: `next()`). Options: `site`, `key` (default
+`LLMS_INK_KEY`; recommended on shared hosting IPs), `markdown`, `baseUrl`. Don't also ship a static `llms.txt`:
+most hosts serve it before the middleware runs.
+
 ## Errors
 
 Every failure is an `LlmsInkError` with `status`, `code`, `title`, `detail`, `errors` (per-parameter issues of

package/dist/astro.d.ts

@@ -0,0 +1,28 @@
+/**
+ * Astro integration: writes llms.txt (and llms-full.txt) from llms.ink into the build output.
+ *
+ * @example
+ * ```ts
+ * // astro.config.mjs
+ * import { defineConfig } from 'astro/config';
+ * import llmsInk from 'llms-ink/astro';
+ *
+ * export default defineConfig({ site: 'https://example.com', integrations: [llmsInk({ full: true })] });
+ * ```
+ *
+ * @module
+ */
+import type { AstroIntegration } from 'astro';
+import { type LlmsFilesOptions } from './files.ts';
+/** {@link LlmsFilesOptions} with `site` optional: it defaults to Astro's `site` (plus `base`). */
+export type LlmsInkAstroOptions = Omit<LlmsFilesOptions, 'site'> & {
+    site?: string;
+};
+/**
+ * After `astro build`, fetches llms.txt — and llms-full.txt with `full: true` — for `site` (default: the
+ * `site` and `base` of the Astro config) and writes them into the static output directory. llms.ink reads the
+ * live site, so the files describe the previous deploy. When llms.ink can't be reached it warns and keeps any
+ * existing file (e.g. one copied from `public/`); `strict: true` fails the build instead.
+ */
+export declare function llmsInk(options?: LlmsInkAstroOptions): AstroIntegration;
+export default llmsInk;

package/dist/astro.js

@@ -0,0 +1,39 @@
+/**
+ * Astro integration: writes llms.txt (and llms-full.txt) from llms.ink into the build output.
+ *
+ * @example
+ * ```ts
+ * // astro.config.mjs
+ * import { defineConfig } from 'astro/config';
+ * import llmsInk from 'llms-ink/astro';
+ *
+ * export default defineConfig({ site: 'https://example.com', integrations: [llmsInk({ full: true })] });
+ * ```
+ *
+ * @module
+ */
+import { fileURLToPath } from 'node:url';
+import { llmsFilesWriter } from "./files.js";
+/**
+ * After `astro build`, fetches llms.txt — and llms-full.txt with `full: true` — for `site` (default: the
+ * `site` and `base` of the Astro config) and writes them into the static output directory. llms.ink reads the
+ * live site, so the files describe the previous deploy. When llms.ink can't be reached it warns and keeps any
+ * existing file (e.g. one copied from `public/`); `strict: true` fails the build instead.
+ */
+export function llmsInk(options = {}) {
+    const settings = { ...options, site: options.site ?? '' };
+    const write = llmsFilesWriter(settings);
+    return {
+        name: 'llms-ink',
+        hooks: {
+            'astro:config:done': ({ config }) => {
+                if (!settings.site && config.site)
+                    settings.site = new URL(config.base, config.site).href;
+            },
+            'astro:build:done': async ({ dir, logger }) => {
+                await write(fileURLToPath(dir), logger);
+            },
+        },
+    };
+}
+export default llmsInk;

package/dist/env.d.ts

@@ -0,0 +1,2 @@
+/** `process.env[name]` where there is a `process` (Node, Bun, Deno with env access); undefined elsewhere. */
+export declare function env(name: string): string | undefined;

package/dist/env.js

@@ -0,0 +1,11 @@
+/** `process.env[name]` where there is a `process` (Node, Bun, Deno with env access); undefined elsewhere. */
+export function env(name) {
+    // Read off globalThis, not a declared global, so it neither needs nor clashes with @types/node.
+    const scope = globalThis;
+    try {
+        return scope.process?.env?.[name]?.trim() || undefined;
+    }
+    catch {
+        return undefined; // Deno without --allow-env throws
+    }
+}

package/dist/files.d.ts

@@ -0,0 +1,37 @@
+/**
+ * Shared by the Vite plugin and the Astro integration: fetch llms.txt (and llms-full.txt) for a site and write
+ * them into a build's output directory.
+ *
+ * @module
+ */
+/** Options for the Vite plugin and the Astro integration. */
+export interface LlmsFilesOptions {
+    /** The deployed site to describe, e.g. `https://example.com`. llms.ink reads the live site. */
+    site: string;
+    /** Also write llms-full.txt (the content of up to 20 pages). Default `false`. */
+    full?: boolean;
+    /** API key (`lk_…`). Defaults to the `LLMS_INK_KEY` environment variable; anonymous without one. */
+    key?: string;
+    /**
+     * Default `true`: when the live site publishes its own llms.txt, that file is used as is, unless it's one
+     * llms.ink generated before (it carries the llms.ink credit line), which is regenerated. `false` always
+     * generates (`native=0`).
+     */
+    native?: boolean;
+    /** Fail the build when a file can't be fetched. Default `false`: warn and keep any existing file. */
+    strict?: boolean;
+    /** Milliseconds to wait for llms.ink per file. Default 300000 (building a big site can take minutes). */
+    timeout?: number;
+    /** Defaults to the `LLMS_INK_API` environment variable, else https://llms.ink. */
+    baseUrl?: string;
+}
+/** Where messages go: Vite's or Astro's logger. */
+export interface Log {
+    info(message: string): void;
+    warn(message: string): void;
+}
+/**
+ * A writer that fetches the files once and writes them into `dir` on every call (watch builds call it again;
+ * a failed fetch is retried on the next call).
+ */
+export declare function llmsFilesWriter(options: LlmsFilesOptions): (dir: string, log: Log) => Promise<void>;

package/dist/files.js

@@ -0,0 +1,52 @@
+/**
+ * Shared by the Vite plugin and the Astro integration: fetch llms.txt (and llms-full.txt) for a site and write
+ * them into a build's output directory.
+ *
+ * @module
+ */
+import { access, mkdir, writeFile } from 'node:fs/promises';
+import { join } from 'node:path';
+import { LlmsInk } from "./index.js";
+const FILES = [['llms.txt', 'llms'], ['llms-full.txt', 'full']];
+/**
+ * A writer that fetches the files once and writes them into `dir` on every call (watch builds call it again;
+ * a failed fetch is retried on the next call).
+ */
+export function llmsFilesWriter(options) {
+    let fetched;
+    return async (dir, log) => {
+        const { site, strict = false } = options;
+        if (!site) {
+            const message = 'No `site` set: llms.txt not written.';
+            if (strict)
+                throw new Error(message);
+            return log.warn(message);
+        }
+        fetched ??= fetchFiles(options);
+        const results = await fetched;
+        const failures = [];
+        for (const result of results) {
+            const path = join(dir, result.name);
+            if ('text' in result) {
+                await mkdir(dir, { recursive: true });
+                await writeFile(path, result.text);
+                log.info(`Wrote ${result.name} for ${site} (${result.source}, ~${result.tokens} tokens).`);
+                continue;
+            }
+            const reason = result.error instanceof Error ? result.error.message : String(result.error);
+            const kept = await access(path).then(() => 'kept the existing file', () => 'none written');
+            failures.push(`Couldn't get ${result.name} for ${site}: ${reason} (${kept}).`);
+        }
+        if (failures.length)
+            fetched = undefined;
+        if (strict && failures.length)
+            throw new Error(failures.join(' '));
+        for (const message of failures)
+            log.warn(message);
+    };
+}
+async function fetchFiles({ site, full, key, native = true, timeout = 300_000, baseUrl }) {
+    const ink = new LlmsInk({ key, baseUrl });
+    const files = full ? FILES : FILES.slice(0, 1);
+    return Promise.all(files.map(([name, mode]) => ink.llms(site, { mode, native, signal: AbortSignal.timeout(timeout) }).then(({ text, tokens, source }) => ({ name, text, tokens, source }), (error) => ({ name, error }))));
+}

package/dist/index.d.ts

@@ -13,7 +13,7 @@
  *
  * @module
  */
-export declare const VERSION = "0.1.0";
+export declare const VERSION = "0.3.0";
 export declare const DEFAULT_BASE_URL = "https://llms.ink";
 /** `llms` for the index (llms.txt), `full` for llms-full.txt. */
 export type Mode = 'llms' | 'full';
@@ -72,12 +72,40 @@ export interface CheckIssue {
     line: number | null;
     message: string;
 }
-/** `GET /api/v0/check`: a published llms.txt checked against the format and its links. */
-export interface CheckResult {
-    /** The site's origin. */
-    site: string;
-    /** The llms.txt that was checked (or the last place looked when none was found). */
+/** One link of a section: `- [title](url): notes`. */
+export interface ParsedLink {
+    title: string;
+    /** Resolved against `base` / the fetched URL when possible, else as written. */
     url: string;
+    /** Text after `:`, trimmed. */
+    notes: string | null;
+    line: number;
+}
+/** An H2 section of llms.txt. */
+export interface ParsedSection {
+    name: string;
+    /** The section is named "Optional" (case-insensitive). */
+    optional: boolean;
+    /** Non-link content of the section. */
+    text: string | null;
+    links: ParsedLink[];
+}
+/** The structure of an llms.txt file. */
+export interface ParsedLlms {
+    /** The H1. */
+    title: string | null;
+    /** The blockquote, without `> `. */
+    summary: string | null;
+    /** Markdown between the summary and the first H2. */
+    details: string | null;
+    sections: ParsedSection[];
+}
+/** `GET` / `POST /api/v0/check`: an llms.txt checked against the format and its links. */
+export interface CheckResult {
+    /** The site's origin; null for {@link LlmsInk.checkText} without `base`. */
+    site: string | null;
+    /** The llms.txt that was checked (or the last place looked when none was found); null for text without `base`. */
+    url: string | null;
     found: boolean;
     /** HTTP status of the llms.txt request. */
     status: number;
@@ -87,7 +115,7 @@ export interface CheckResult {
     sections: number;
     links: {
         total: number;
-        /** Links fetched this run (the first 25). */
+        /** Links fetched this run (the first 25); 0 when links weren't checked. */
         checked: number;
         /** `status` is 0 when the link didn't answer in time. */
         broken: Array<{
@@ -95,8 +123,10 @@ export interface CheckResult {
             status: number;
         }>;
     };
-    /** Whether llms-full.txt is published next to it. */
-    llmsFull: boolean;
+    /** Whether llms-full.txt is published next to it; null when not checked. */
+    llmsFull: boolean | null;
+    /** The parsed file; null when nothing was found. */
+    parsed: ParsedLlms | null;
     /** Errors first, then warnings and notes, in line order. */
     issues: CheckIssue[];
     summary: {
@@ -107,7 +137,7 @@ export interface CheckResult {
     checkedAt: string;
 }
 /** Every error code the API answers with (https://llms.ink/docs/#errors). */
-export type ErrorCode = 'invalid_request' | 'invalid_url' | 'invalid_key' | 'not_found' | 'unknown_version' | 'method_not_allowed' | 'payload_too_large' | 'unprocessable' | 'rate_limited' | 'quota_exceeded' | 'internal' | 'upstream_error' | 'unavailable' | 'timeout';
+export type ErrorCode = 'invalid_request' | 'invalid_url' | 'invalid_key' | 'not_found' | 'unknown_version' | 'method_not_allowed' | 'payload_too_large' | 'unprocessable' | 'rate_limited' | 'quota_exceeded' | 'internal' | 'upstream_error' | 'unavailable' | 'timeout' | 'loop_detected';
 /** `invalid_request` only: one invalid parameter. */
 export interface Issue {
     path: string;
@@ -193,6 +223,12 @@ export interface LlmsOptions {
     mode?: Mode;
     /** Skip the cache and rebuild (counts against the daily quota). */
     fresh?: boolean;
+    /**
+     * `false` always generates the file from the site's pages and never returns the site's own llms.txt or
+     * llms-full.txt (`native=0`). Defaults to `true`: a site's own file is returned as is, unless it carries the
+     * llms.ink credit line (then it's regenerated).
+     */
+    native?: boolean;
     /** Aborts the request; the abort error is rethrown as is. */
     signal?: AbortSignal;
 }
@@ -200,6 +236,14 @@ export interface LlmsOptions {
 export interface CheckOptions {
     signal?: AbortSignal;
 }
+/** Options for {@link LlmsInk.checkText}. */
+export interface CheckTextOptions {
+    /** Public URL the text lives at: resolves relative links and is reported as `url` / `site`. */
+    base?: string;
+    /** Also check up to 25 linked URLs (and llms-full.txt next to `base`). Defaults to `false`. */
+    links?: boolean;
+    signal?: AbortSignal;
+}
 /**
  * A client for https://llms.ink.
  *
@@ -220,6 +264,8 @@ export declare class LlmsInk {
     llms(url: string, options?: LlmsOptions): Promise<Result>;
     /** Checks a site's published llms.txt against the llmstxt.org format and that its links answer. */
     check(url: string, options?: CheckOptions): Promise<CheckResult>;
+    /** Parses and checks llms.txt text you already have (`POST /api/v0/check`); nothing is published. */
+    checkText(text: string, options?: CheckTextOptions): Promise<CheckResult>;
     /**
      * Like {@link LlmsInk.llms}, as the build happens: yields `progress` and `partial` events, then one
      * `result`. Errors — including a terminal `error` event — are thrown as {@link LlmsInkError}s.

package/dist/index.js

@@ -13,7 +13,8 @@
  *
  * @module
  */
-export const VERSION = '0.1.0';
+import { env } from "./env.js";
+export const VERSION = '0.3.0';
 export const DEFAULT_BASE_URL = 'https://llms.ink';
 /**
  * An API error, or llms.ink couldn't be reached (then `status` and `code` are undefined).
@@ -57,15 +58,6 @@ const BY_CODE = {
     quota_exceeded: QuotaExceededError,
 };
 const errorFor = (detail, init) => new (BY_CODE[init.code ?? ''] ?? LlmsInkError)(detail, init);
-/** `process.env[name]` where there is a `process` (Node, Bun, Deno with env access); undefined elsewhere. */
-function env(name) {
-    try {
-        return (typeof process === 'undefined' ? undefined : process.env?.[name]?.trim()) || undefined;
-    }
-    catch {
-        return undefined; // Deno without --allow-env throws
-    }
-}
 /** Browsers refuse or preflight a custom User-Agent, so it's only sent from servers. */
 const IS_BROWSER = 'document' in globalThis || 'WorkerGlobalScope' in globalThis;
 /** `Retry-After` as seconds: delta-seconds, or an HTTP date turned into seconds from now. */
@@ -107,12 +99,22 @@ export class LlmsInk {
      * any public link.
      */
     async llms(url, options = {}) {
-        const res = await this.#get('/api/v0/llms', llmsParams(url, options), 'application/json', options.signal);
+        const res = await this.#request('/api/v0/llms', llmsParams(url, options), 'application/json', options.signal);
         return (await this.#json(res));
     }
     /** Checks a site's published llms.txt against the llmstxt.org format and that its links answer. */
     async check(url, options = {}) {
-        const res = await this.#get('/api/v0/check', { url }, 'application/json', options.signal);
+        const res = await this.#request('/api/v0/check', { url }, 'application/json', options.signal);
+        return (await this.#json(res));
+    }
+    /** Parses and checks llms.txt text you already have (`POST /api/v0/check`); nothing is published. */
+    async checkText(text, options = {}) {
+        const params = {};
+        if (options.base)
+            params.base = options.base;
+        if (options.links)
+            params.links = '1';
+        const res = await this.#request('/api/v0/check', params, 'application/json', options.signal, text);
         return (await this.#json(res));
     }
     /**
@@ -121,7 +123,7 @@ export class LlmsInk {
      * Breaking out of the loop cancels the request.
      */
     async *stream(url, options = {}) {
-        const res = await this.#get('/api/v0/llms', llmsParams(url, options), 'application/x-ndjson', options.signal);
+        const res = await this.#request('/api/v0/llms', llmsParams(url, options), 'application/x-ndjson', options.signal);
         if (!res.headers.get('content-type')?.includes('ndjson')) {
             yield { type: 'result', result: (await this.#json(res)) };
             return;
@@ -181,16 +183,19 @@ export class LlmsInk {
         const known = event.type === 'progress' || event.type === 'partial' || event.type === 'result';
         return known ? event : undefined;
     }
-    async #get(path, params, accept, signal) {
+    async #request(path, params, accept, signal, body) {
         const headers = { accept };
+        if (body !== undefined)
+            headers['content-type'] = 'text/markdown; charset=utf-8';
         if (!IS_BROWSER)
             headers['user-agent'] = `llms-ink-js/${VERSION}`;
         if (this.key)
             headers.authorization = `Bearer ${this.key}`;
-        const target = `${this.baseUrl}${path}?${new URLSearchParams(params)}`;
+        const query = new URLSearchParams(params).toString();
+        const target = `${this.baseUrl}${path}${query ? `?${query}` : ''}`;
         let res;
         try {
-            res = await this.#fetch(target, { headers, signal });
+            res = await this.#fetch(target, body === undefined ? { headers, signal } : { method: 'POST', headers, body, signal });
         }
         catch (err) {
             if (isAbort(err))
@@ -211,11 +216,16 @@ export class LlmsInk {
         }
     }
 }
-function llmsParams(url, { mode = 'llms', fresh }) {
+function llmsParams(url, { mode = 'llms', fresh, native = true }) {
     if (mode !== 'llms' && mode !== 'full') {
         throw new TypeError(`mode must be 'llms' (the index) or 'full' (llms-full.txt), not ${JSON.stringify(mode)}`);
     }
-    return fresh ? { url, mode, fresh: '1' } : { url, mode };
+    const params = { url, mode };
+    if (fresh)
+        params.fresh = '1';
+    if (!native)
+        params.native = '0';
+    return params;
 }
 /** An HTTP error response → the typed error for its problem `code`. */
 async function problem(res) {

package/dist/middleware.d.ts

@@ -0,0 +1,64 @@
+/**
+ * Serve llms.txt, llms-full.txt and (optionally) Markdown versions of pages from llms.ink, inside your own app.
+ * A fetch-API handler for Cloudflare Workers, Next.js middleware, Deno, Bun and Hono, plus an Express/Connect
+ * adapter.
+ *
+ * @example
+ * ```ts
+ * import { llmsInkHandler } from 'llms-ink/middleware';
+ *
+ * const llms = llmsInkHandler({ site: 'https://example.com', markdown: true });
+ * export default {
+ *   async fetch(request: Request): Promise<Response> {
+ *     return (await llms(request)) ?? fetch(request); // undefined: not ours, the app answers
+ *   },
+ * };
+ * ```
+ *
+ * @module
+ */
+/** Options for {@link llmsInkHandler} and {@link llmsInkExpress}. */
+export interface LlmsInkHandlerOptions {
+    /** The deployed site to describe, e.g. `https://example.com`. */
+    site: string;
+    /**
+     * API key (`lk_…`), sent as `Authorization: Bearer`. Defaults to the `LLMS_INK_KEY` environment variable
+     * where there is one. Recommended: proxies on shared hosting IPs otherwise share the anonymous limits.
+     */
+    key?: string;
+    /**
+     * Also answer `<path>.md`, and page requests whose `Accept` prefers `text/markdown` over `text/html`, with the
+     * page as Markdown (`https://llms.ink/full/<page>`). Default `false`.
+     */
+    markdown?: boolean;
+    /** Defaults to the `LLMS_INK_API` environment variable, else https://llms.ink. */
+    baseUrl?: string;
+}
+/** A fetch-API handler: a `Response` for the requests it serves, `undefined` for everything else. */
+export type LlmsInkHandler = (request: Request) => Promise<Response | undefined>;
+/**
+ * Answers `GET`/`HEAD` `/llms.txt` and `/llms-full.txt` with llms.ink's generated files for `site`
+ * (`native=0`, so a site whose /llms.txt is this handler can't loop), and, with `markdown: true`, Markdown
+ * versions of pages. Responses keep llms.ink's `Cache-Control`; each one is kept in memory for its `max-age`
+ * and served as a last good copy (with `Warning: 110`) when llms.ink fails. Returns `undefined` for every
+ * other request, and for llms.ink's own fetcher.
+ */
+export declare function llmsInkHandler(options: LlmsInkHandlerOptions): LlmsInkHandler;
+/** Node's `IncomingMessage` (Express, Connect, plain `node:http`), as far as the adapter reads it. */
+export interface NodeRequest {
+    method?: string;
+    url?: string;
+    /** Express: the URL before mounting stripped a prefix. */
+    originalUrl?: string;
+    headers: Record<string, string | string[] | undefined>;
+}
+/** Node's `ServerResponse`, as far as the adapter writes it. */
+export interface NodeResponse {
+    statusCode: number;
+    setHeader(name: string, value: string): unknown;
+    end(chunk: Uint8Array): unknown;
+}
+/** Express/Connect middleware: `next()` when the request isn't one {@link llmsInkHandler} serves. */
+export type NodeMiddleware = (req: NodeRequest, res: NodeResponse, next: (err?: unknown) => void) => void;
+/** {@link llmsInkHandler} as Express/Connect middleware: `app.use(llmsInkExpress({ site }))`. */
+export declare function llmsInkExpress(options: LlmsInkHandlerOptions): NodeMiddleware;

package/dist/middleware.js

@@ -0,0 +1,183 @@
+/**
+ * Serve llms.txt, llms-full.txt and (optionally) Markdown versions of pages from llms.ink, inside your own app.
+ * A fetch-API handler for Cloudflare Workers, Next.js middleware, Deno, Bun and Hono, plus an Express/Connect
+ * adapter.
+ *
+ * @example
+ * ```ts
+ * import { llmsInkHandler } from 'llms-ink/middleware';
+ *
+ * const llms = llmsInkHandler({ site: 'https://example.com', markdown: true });
+ * export default {
+ *   async fetch(request: Request): Promise<Response> {
+ *     return (await llms(request)) ?? fetch(request); // undefined: not ours, the app answers
+ *   },
+ * };
+ * ```
+ *
+ * @module
+ */
+import { env } from "./env.js";
+import { DEFAULT_BASE_URL, VERSION } from "./index.js";
+const FILES = { '/llms.txt': 'llms', '/llms-full.txt': 'full' };
+/** llms.ink's own fetcher (`compatible; llms.ink/0.1`): answering it from llms.ink would loop. */
+const LLMS_INK_FETCHER = /\bllms\.ink\/\d/;
+/** Response headers passed through from llms.ink. */
+const PASS = ['content-type', 'cache-control', 'x-llms-kind', 'x-llms-source', 'x-llms-tokens'];
+/** Pages kept in memory as last good copies; the oldest is dropped first. */
+const MAX_ENTRIES = 200;
+/** How long downstream caches may keep a stale copy served while llms.ink fails. */
+const STALE_CACHE_CONTROL = 'public, max-age=60';
+/**
+ * Answers `GET`/`HEAD` `/llms.txt` and `/llms-full.txt` with llms.ink's generated files for `site`
+ * (`native=0`, so a site whose /llms.txt is this handler can't loop), and, with `markdown: true`, Markdown
+ * versions of pages. Responses keep llms.ink's `Cache-Control`; each one is kept in memory for its `max-age`
+ * and served as a last good copy (with `Warning: 110`) when llms.ink fails. Returns `undefined` for every
+ * other request, and for llms.ink's own fetcher.
+ */
+export function llmsInkHandler(options) {
+    const base = (options.baseUrl ?? env('LLMS_INK_API') ?? DEFAULT_BASE_URL).replace(/\/+$/, '');
+    const key = (options.key ?? env('LLMS_INK_KEY'))?.trim();
+    const site = new URL(/^https?:\/\//i.test(options.site) ? options.site : `https://${options.site}`);
+    const headers = { accept: 'text/plain', 'user-agent': `llms-ink-js/${VERSION}` };
+    if (key)
+        headers.authorization = `Bearer ${key}`;
+    const cache = new Map();
+    const pending = new Map();
+    /** Fetches `upstream`; on failure the last good copy, else llms.ink's error. */
+    async function refresh(upstream) {
+        const last = cache.get(upstream);
+        let res;
+        try {
+            res = await fetch(upstream, { headers });
+        }
+        catch (err) {
+            return last ? stale(last) : failure(502, 'upstream_error', `Couldn't reach llms.ink: ${err instanceof Error ? err.message : err}`);
+        }
+        // An older llms.ink without `format=text` answers JSON; never serve that as the file.
+        const isText = res.headers.get('content-type')?.startsWith('text/');
+        if (!res.ok || !isText) {
+            if (last) {
+                await res.body?.cancel();
+                return stale(last);
+            }
+            if (res.ok)
+                return failure(502, 'upstream_error', `llms.ink answered ${res.headers.get('content-type')}, not text.`);
+            return new Response(res.body, { status: res.status, headers: pick(res.headers, [...PASS, 'x-error-code', 'retry-after']) });
+        }
+        const maxAge = Number(/(?:^|,)\s*max-age=(\d+)/i.exec(res.headers.get('cache-control') ?? '')?.[1] ?? 0);
+        const entry = { body: await res.text(), headers: pick(res.headers, PASS), at: Date.now(), maxAge };
+        cache.delete(upstream);
+        cache.set(upstream, entry);
+        if (cache.size > MAX_ENTRIES)
+            cache.delete(cache.keys().next().value);
+        return fresh(entry);
+    }
+    async function serve(upstream) {
+        const hit = cache.get(upstream);
+        if (hit && Date.now() - hit.at < hit.maxAge * 1000)
+            return fresh(hit);
+        let loading = pending.get(upstream);
+        if (!loading) {
+            loading = refresh(upstream).finally(() => pending.delete(upstream));
+            pending.set(upstream, loading);
+        }
+        // Concurrent requests share one fetch; each gets its own copy of the body.
+        return (await loading).clone();
+    }
+    return async (request) => {
+        if (request.method !== 'GET' && request.method !== 'HEAD')
+            return undefined;
+        if (LLMS_INK_FETCHER.test(request.headers.get('user-agent') ?? ''))
+            return undefined;
+        const { pathname } = new URL(request.url);
+        const mode = FILES[pathname];
+        let response;
+        if (mode) {
+            const query = new URLSearchParams({ url: site.href, mode, native: '0', format: 'text' });
+            response = await serve(`${base}/api/v0/llms?${query}`);
+        }
+        else if (options.markdown && pathname.endsWith('.md')) {
+            const page = pathname.slice(0, -3).replace(/\/index$/, '/');
+            response = await serve(`${base}/full/${site.host}${page}`);
+        }
+        else if (options.markdown && isPage(pathname) && prefersMarkdown(request.headers.get('accept') ?? '')) {
+            response = await serve(`${base}/full/${site.host}${pathname}`);
+            response.headers.append('vary', 'Accept');
+        }
+        else {
+            return undefined;
+        }
+        return request.method === 'HEAD' ? new Response(null, response) : response;
+    };
+}
+/** {@link llmsInkHandler} as Express/Connect middleware: `app.use(llmsInkExpress({ site }))`. */
+export function llmsInkExpress(options) {
+    const handle = llmsInkHandler(options);
+    return (req, res, next) => {
+        const headers = new Headers();
+        for (const name of ['accept', 'user-agent']) {
+            const value = req.headers[name];
+            if (typeof value === 'string')
+                headers.set(name, value);
+        }
+        const request = new Request(new URL(req.originalUrl ?? req.url ?? '/', 'http://localhost'), { method: req.method, headers });
+        handle(request)
+            .then(async (response) => {
+            if (!response)
+                return next();
+            res.statusCode = response.status;
+            response.headers.forEach((value, name) => res.setHeader(name, value));
+            res.end(new Uint8Array(await response.arrayBuffer()));
+        })
+            .catch(next);
+    };
+}
+function pick(from, names) {
+    const to = new Headers();
+    for (const name of names) {
+        const value = from.get(name);
+        if (value !== null)
+            to.set(name, value);
+    }
+    return to;
+}
+function fresh(entry) {
+    const headers = new Headers(entry.headers);
+    headers.set('age', String(Math.floor((Date.now() - entry.at) / 1000)));
+    return new Response(entry.body, { headers });
+}
+function stale(entry) {
+    const response = fresh(entry);
+    response.headers.set('cache-control', STALE_CACHE_CONTROL);
+    response.headers.set('warning', '110 - "Response is Stale"');
+    return response;
+}
+function failure(status, code, detail) {
+    return new Response(`${detail}\n`, { status, headers: { 'content-type': 'text/plain; charset=utf-8', 'x-error-code': code } });
+}
+/** A page path: no file extension, or `.html`/`.htm`. */
+function isPage(pathname) {
+    const last = pathname.slice(pathname.lastIndexOf('/') + 1);
+    return !last.includes('.') || /\.html?$/i.test(last);
+}
+/**
+ * Whether `Accept` names `text/markdown` with a quality at least that of `text/html` (or, when `text/html`
+ * isn't listed, `text/*`, then `*\/*`). Browsers never list `text/markdown`, so they get the app.
+ */
+function prefersMarkdown(accept) {
+    let markdown = 0;
+    const html = {};
+    for (const part of accept.toLowerCase().split(',')) {
+        const [type = '', ...params] = part.split(';').map((s) => s.trim());
+        const q = params.find((p) => p.startsWith('q='));
+        const quality = q === undefined ? 1 : Number(q.slice(2));
+        if (!Number.isFinite(quality))
+            continue;
+        if (type === 'text/markdown' || type === 'text/x-markdown')
+            markdown = Math.max(markdown, quality);
+        else if (type === 'text/html' || type === 'text/*' || type === '*/*')
+            html[type] = Math.max(html[type] ?? 0, quality);
+    }
+    return markdown > 0 && markdown >= (html['text/html'] ?? html['text/*'] ?? html['*/*'] ?? 0);
+}

package/dist/vite.d.ts

@@ -0,0 +1,25 @@
+/**
+ * Vite plugin: writes llms.txt (and llms-full.txt) from llms.ink into the build output.
+ *
+ * @example
+ * ```ts
+ * // vite.config.ts
+ * import { defineConfig } from 'vite';
+ * import { llmsInk } from 'llms-ink/vite';
+ *
+ * export default defineConfig({ plugins: [llmsInk({ site: 'https://example.com', full: true })] });
+ * ```
+ *
+ * @module
+ */
+import type { Plugin } from 'vite';
+import { type LlmsFilesOptions } from './files.ts';
+export type { LlmsFilesOptions };
+/**
+ * After `vite build` (not the dev server, not SSR builds), fetches llms.txt — and llms-full.txt with
+ * `full: true` — for `site` and writes them into `build.outDir`. llms.ink reads the live site, so the files
+ * describe the previous deploy. When llms.ink can't be reached it warns and keeps any existing file (e.g. one
+ * copied from `public/`); `strict: true` fails the build instead.
+ */
+export declare function llmsInk(options: LlmsFilesOptions): Plugin;
+export default llmsInk;

package/dist/vite.js

@@ -0,0 +1,47 @@
+/**
+ * Vite plugin: writes llms.txt (and llms-full.txt) from llms.ink into the build output.
+ *
+ * @example
+ * ```ts
+ * // vite.config.ts
+ * import { defineConfig } from 'vite';
+ * import { llmsInk } from 'llms-ink/vite';
+ *
+ * export default defineConfig({ plugins: [llmsInk({ site: 'https://example.com', full: true })] });
+ * ```
+ *
+ * @module
+ */
+import { resolve } from 'node:path';
+import { llmsFilesWriter } from "./files.js";
+/**
+ * After `vite build` (not the dev server, not SSR builds), fetches llms.txt — and llms-full.txt with
+ * `full: true` — for `site` and writes them into `build.outDir`. llms.ink reads the live site, so the files
+ * describe the previous deploy. When llms.ink can't be reached it warns and keeps any existing file (e.g. one
+ * copied from `public/`); `strict: true` fails the build instead.
+ */
+export function llmsInk(options) {
+    const write = llmsFilesWriter(options);
+    let config;
+    return {
+        name: 'llms-ink',
+        apply: 'build',
+        configResolved(resolved) {
+            config = resolved;
+        },
+        async closeBundle(error) {
+            if (error)
+                return;
+            // Vite 6+ runs the hook once per environment; only the browser build is served as static files.
+            const environment = this.environment;
+            if (environment ? environment.name !== 'client' : config.build.ssr)
+                return;
+            const { outDir } = environment ? environment.config.build : config.build;
+            await write(resolve(config.root, outDir), {
+                info: (message) => config.logger.info(`[llms-ink] ${message}`),
+                warn: (message) => config.logger.warn(`[llms-ink] ${message}`),
+            });
+        },
+    };
+}
+export default llmsInk;

package/package.json

@@ -1,32 +1,57 @@
 {
   "name": "llms-ink",
-  "version": "0.1.0",
-  "description": "Turn any link — site, docs section, page, GitHub repo, PDF or video — into llms.txt and llms-full.txt. Typed client for llms.ink, no dependencies.",
+  "version": "0.3.0",
+  "description": "Turn any link — site, docs section, page, GitHub repo, PDF or video — into llms.txt and llms-full.txt. Typed client for llms.ink, plus a Vite plugin, an Astro integration and middleware that serve the files from your site. No dependencies.",
   "type": "module",
   "exports": {
     ".": {
       "types": "./dist/index.d.ts",
       "default": "./dist/index.js"
+    },
+    "./vite": {
+      "types": "./dist/vite.d.ts",
+      "default": "./dist/vite.js"
+    },
+    "./astro": {
+      "types": "./dist/astro.d.ts",
+      "default": "./dist/astro.js"
+    },
+    "./middleware": {
+      "types": "./dist/middleware.d.ts",
+      "default": "./dist/middleware.js"
     }
   },
   "main": "./dist/index.js",
   "types": "./dist/index.d.ts",
   "files": ["dist"],
   "sideEffects": false,
+  "repository": { "type": "git", "url": "git+https://github.com/lukax/llms-ink-js.git" },
+  "bugs": "https://github.com/lukax/llms-ink-js/issues",
   "engines": { "node": ">=18" },
   "scripts": {
-    "build": "tsc -p tsconfig.json",
+    "build": "rm -rf dist && tsc -p tsconfig.json",
     "prepublishOnly": "npm run build && npm test",
-    "test": "npm run build && node --test test/client.test.mjs",
+    "test": "npm run build && node --test test/client.test.mjs test/plugins.test.mjs test/middleware.test.mjs",
     "test:live": "npm run build && node --test test/live.test.mjs",
-    "test:bun": "npm run build && bun test ./test/client.test.mjs",
+    "test:bun": "npm run build && bun test ./test/client.test.mjs ./test/plugins.test.mjs ./test/middleware.test.mjs",
     "test:bun:live": "npm run build && bun test --timeout 300000 ./test/live.test.mjs"
   },
-  "keywords": ["llms.txt", "llms-full.txt", "llm", "ai", "markdown", "docs", "context", "rag", "sdk", "llms-ink"],
+  "keywords": ["llms.txt", "llms-full.txt", "llm", "ai", "markdown", "docs", "context", "rag", "sdk", "llms-ink", "vite-plugin", "astro-integration", "middleware"],
   "homepage": "https://llms.ink/docs/",
   "author": "Lucas Espindola (https://lucasespindola.com)",
   "license": "MIT",
+  "peerDependencies": {
+    "astro": ">=4",
+    "vite": ">=5"
+  },
+  "peerDependenciesMeta": {
+    "astro": { "optional": true },
+    "vite": { "optional": true }
+  },
   "devDependencies": {
-    "typescript": "^5.9.3"
+    "@types/node": "^22.20.5",
+    "astro": "^7.4.1",
+    "typescript": "^5.9.3",
+    "vite": "^8.3.8"
   }
 }