llms-ink 0.0.0-stage → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Lucas Espindola
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -1,3 +1,152 @@
1
- # Temporary Holding Version
1
+ # llms-ink
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ Turn any link — site, docs section, page, GitHub repo, PDF or video — into llms.txt and llms-full.txt. A typed
4
+ client for the [llms.ink](https://llms.ink) API with no dependencies. Runs on Node 18+, Bun, Deno, Cloudflare
5
+ Workers and browsers. Also a [Vite plugin, an Astro integration and middleware](#serve-llmstxt-from-your-site) that
6
+ serve llms.txt from your own site.
7
+
8
+ ```sh
9
+ npm install llms-ink # or: bun add llms-ink
10
+ deno add jsr:@lukax/llms-ink # or: npx jsr add @lukax/llms-ink
11
+ ```
12
+
13
+ ```ts
14
+ import { LlmsInk } from 'llms-ink'; // Deno/JSR: from 'jsr:@lukax/llms-ink'
15
+
16
+ const ink = new LlmsInk();
17
+
18
+ const index = await ink.llms('developers.cloudflare.com/kv/'); // llms.txt: an index of links
19
+ const full = await ink.llms('github.com/honojs/hono', { mode: 'full' }); // llms-full.txt: the content
20
+ console.log(full.title, full.tokens, full.text);
21
+
22
+ const report = await ink.check('llmstxt.org'); // is a site's own llms.txt valid, do its links answer?
23
+ console.log(report.found, report.summary, report.issues);
24
+ ```
25
+
26
+ ## API
27
+
28
+ ### `new LlmsInk({ key?, baseUrl?, fetch? })`
29
+
30
+ - `key`: an API key (`lk_…`), sent as `Authorization: Bearer`. Defaults to the `LLMS_INK_KEY` environment
31
+ variable where there is one (Node, Bun, Deno with `--allow-env`). Without a key, requests are anonymous and
32
+ share the free limits. An invalid or revoked key throws `InvalidKeyError`; it never falls back to anonymous.
33
+ - `baseUrl`: defaults to `LLMS_INK_API`, else `https://llms.ink`.
34
+ - `fetch`: a `fetch` to use instead of the global one.
35
+
36
+ ### `ink.llms(url, { mode?, fresh?, native?, signal? })` → `Promise<Result>`
37
+
38
+ `mode` is `'llms'` (default) or `'full'` (up to 20 pages inlined). `fresh: true` skips the cache and rebuilds
39
+ (counts against the daily quota). `native: false` always generates the file and never returns the site's own
40
+ llms.txt (by default a site's own file is returned as is, unless llms.ink generated it). The result:
41
+
42
+ ```ts
43
+ { url, kind, source, mode, title, text, tokens, pages, model, cached, generatedAt, media? }
44
+ ```
45
+
46
+ `kind` is `site | section | page | repo | document | video`; `source` is `native` (the site publishes it) or
47
+ `generated`. `media` is set for videos (duration, thumbnail, downloads, transcript segments).
48
+
49
+ Building a big site or a video can take minutes. Bound a call with a signal:
50
+ `ink.llms(url, { signal: AbortSignal.timeout(120_000) })`. Aborts are rethrown as is.
51
+
52
+ ### `ink.check(url, { signal? })` → `Promise<CheckResult>`
53
+
54
+ Checks the site's (or docs section's) published llms.txt against the llmstxt.org format and fetches its first
55
+ 25 links: `{ site, url, found, status, title, tokens, sections, links, llmsFull, issues, summary, checkedAt }`.
56
+
57
+ ### `ink.stream(url, { mode?, fresh?, signal? })` → `AsyncGenerator<StreamEvent>`
58
+
59
+ The same build as `llms()`, as it happens:
60
+
61
+ ```ts
62
+ for await (const event of ink.stream('docs.astro.build', { mode: 'full' })) {
63
+ if (event.type === 'progress') console.log(event.stage, event.detail);
64
+ if (event.type === 'partial') preview(event.text); // incomplete so far
65
+ if (event.type === 'result') save(event.result); // always last
66
+ }
67
+ ```
68
+
69
+ Breaking out of the loop cancels the request.
70
+
71
+ ## Serve llms.txt from your site
72
+
73
+ ### Vite: `llms-ink/vite`
74
+
75
+ ```ts
76
+ // vite.config.ts
77
+ import { defineConfig } from 'vite';
78
+ import { llmsInk } from 'llms-ink/vite';
79
+
80
+ export default defineConfig({ plugins: [llmsInk({ site: 'https://example.com', full: true })] });
81
+ ```
82
+
83
+ After `vite build` (not the dev server or SSR builds), writes `llms.txt` (and `llms-full.txt` with `full`) for
84
+ `site` into `build.outDir`. llms.ink reads the live site, so the files describe the previous deploy. If llms.ink
85
+ fails, the plugin warns and keeps any existing file (e.g. one from `public/`); `strict: true` fails the build.
86
+ Options: `site`, `full`, `key` (default `LLMS_INK_KEY`), `native` (default `true`; a file llms.ink generated before
87
+ is regenerated, not echoed), `strict`, `timeout` (ms, default 300000), `baseUrl`.
88
+
89
+ ### Astro: `llms-ink/astro`
90
+
91
+ ```ts
92
+ // astro.config.mjs
93
+ import { defineConfig } from 'astro/config';
94
+ import llmsInk from 'llms-ink/astro';
95
+
96
+ export default defineConfig({ site: 'https://example.com', integrations: [llmsInk({ full: true })] });
97
+ ```
98
+
99
+ The same, on `astro:build:done`, into the static output. `site` defaults to Astro's `site` plus `base`.
100
+
101
+ ### Middleware: `llms-ink/middleware`
102
+
103
+ ```ts
104
+ import { llmsInkHandler, llmsInkExpress } from 'llms-ink/middleware';
105
+
106
+ const llms = llmsInkHandler({ site: 'https://example.com', markdown: true });
107
+ const response = (await llms(request)) ?? (await app(request)); // Workers, Next.js proxy, Deno, Bun, Hono
108
+
109
+ expressApp.use(llmsInkExpress({ site: 'https://example.com' })); // Express, Connect
110
+ ```
111
+
112
+ Answers `GET`/`HEAD` `/llms.txt` and `/llms-full.txt` from llms.ink, always generated (`native=0`, so it can't
113
+ loop through itself), with llms.ink's `Cache-Control`. Each answer is kept in memory for its `max-age` and served as
114
+ a last good copy (`Warning: 110`, `Cache-Control: max-age=60`) when llms.ink fails. With `markdown: true` it also
115
+ answers `/<page>.md` and requests whose `Accept` prefers `text/markdown` over `text/html` with the page as Markdown.
116
+ Everything else, and llms.ink's own fetcher, gets `undefined` (Express: `next()`). Options: `site`, `key` (default
117
+ `LLMS_INK_KEY`; recommended on shared hosting IPs), `markdown`, `baseUrl`. Don't also ship a static `llms.txt`:
118
+ most hosts serve it before the middleware runs.
119
+
120
+ ## Errors
121
+
122
+ Every failure is an `LlmsInkError` with `status`, `code`, `title`, `detail`, `errors` (per-parameter issues of
123
+ `invalid_request`) and `retryAfter` (seconds, from `Retry-After`). When llms.ink can't be reached, `status` and
124
+ `code` are undefined and `cause` holds the network error. Codes are listed at
125
+ [llms.ink/docs](https://llms.ink/docs/#errors).
126
+
127
+ | Class | Status | Code | When |
128
+ |---|---|---|---|
129
+ | `InvalidKeyError` | 401 | `invalid_key` | The key is malformed, unknown or revoked. |
130
+ | `RateLimitedError` | 429 | `rate_limited` | Too many requests this minute; wait `retryAfter` seconds. |
131
+ | `QuotaExceededError` | 429 | `quota_exceeded` | Daily quota of new links used up; `retryAfter` is the seconds until midnight UTC. Cached links still work. |
132
+
133
+ ```ts
134
+ import { LlmsInk, RateLimitedError } from 'llms-ink';
135
+
136
+ try {
137
+ await ink.llms('example.com');
138
+ } catch (err) {
139
+ if (err instanceof RateLimitedError) await new Promise((r) => setTimeout(r, err.retryAfter! * 1000));
140
+ else throw err;
141
+ }
142
+ ```
143
+
144
+ ## Limits
145
+
146
+ Anonymous: 10 requests a minute per IP, 60 new (uncached) links a day per IP. Cached results are free. A key
147
+ gets its own daily quota (1000 new links by default) and 60 requests a minute;
148
+ [ask for one](https://lucasespindola.com/message?text=Hi+Lucas%2C+I%E2%80%99d+like+an+llms.ink+API+key+for+...).
149
+
150
+ ## License
151
+
152
+ MIT
@@ -0,0 +1,28 @@
1
+ /**
2
+ * Astro integration: writes llms.txt (and llms-full.txt) from llms.ink into the build output.
3
+ *
4
+ * @example
5
+ * ```ts
6
+ * // astro.config.mjs
7
+ * import { defineConfig } from 'astro/config';
8
+ * import llmsInk from 'llms-ink/astro';
9
+ *
10
+ * export default defineConfig({ site: 'https://example.com', integrations: [llmsInk({ full: true })] });
11
+ * ```
12
+ *
13
+ * @module
14
+ */
15
+ import type { AstroIntegration } from 'astro';
16
+ import { type LlmsFilesOptions } from './files.ts';
17
+ /** {@link LlmsFilesOptions} with `site` optional: it defaults to Astro's `site` (plus `base`). */
18
+ export type LlmsInkAstroOptions = Omit<LlmsFilesOptions, 'site'> & {
19
+ site?: string;
20
+ };
21
+ /**
22
+ * After `astro build`, fetches llms.txt — and llms-full.txt with `full: true` — for `site` (default: the
23
+ * `site` and `base` of the Astro config) and writes them into the static output directory. llms.ink reads the
24
+ * live site, so the files describe the previous deploy. When llms.ink can't be reached it warns and keeps any
25
+ * existing file (e.g. one copied from `public/`); `strict: true` fails the build instead.
26
+ */
27
+ export declare function llmsInk(options?: LlmsInkAstroOptions): AstroIntegration;
28
+ export default llmsInk;
package/dist/astro.js ADDED
@@ -0,0 +1,39 @@
1
+ /**
2
+ * Astro integration: writes llms.txt (and llms-full.txt) from llms.ink into the build output.
3
+ *
4
+ * @example
5
+ * ```ts
6
+ * // astro.config.mjs
7
+ * import { defineConfig } from 'astro/config';
8
+ * import llmsInk from 'llms-ink/astro';
9
+ *
10
+ * export default defineConfig({ site: 'https://example.com', integrations: [llmsInk({ full: true })] });
11
+ * ```
12
+ *
13
+ * @module
14
+ */
15
+ import { fileURLToPath } from 'node:url';
16
+ import { llmsFilesWriter } from "./files.js";
17
+ /**
18
+ * After `astro build`, fetches llms.txt — and llms-full.txt with `full: true` — for `site` (default: the
19
+ * `site` and `base` of the Astro config) and writes them into the static output directory. llms.ink reads the
20
+ * live site, so the files describe the previous deploy. When llms.ink can't be reached it warns and keeps any
21
+ * existing file (e.g. one copied from `public/`); `strict: true` fails the build instead.
22
+ */
23
+ export function llmsInk(options = {}) {
24
+ const settings = { ...options, site: options.site ?? '' };
25
+ const write = llmsFilesWriter(settings);
26
+ return {
27
+ name: 'llms-ink',
28
+ hooks: {
29
+ 'astro:config:done': ({ config }) => {
30
+ if (!settings.site && config.site)
31
+ settings.site = new URL(config.base, config.site).href;
32
+ },
33
+ 'astro:build:done': async ({ dir, logger }) => {
34
+ await write(fileURLToPath(dir), logger);
35
+ },
36
+ },
37
+ };
38
+ }
39
+ export default llmsInk;
package/dist/env.d.ts ADDED
@@ -0,0 +1,2 @@
1
+ /** `process.env[name]` where there is a `process` (Node, Bun, Deno with env access); undefined elsewhere. */
2
+ export declare function env(name: string): string | undefined;
package/dist/env.js ADDED
@@ -0,0 +1,11 @@
1
+ /** `process.env[name]` where there is a `process` (Node, Bun, Deno with env access); undefined elsewhere. */
2
+ export function env(name) {
3
+ // Read off globalThis, not a declared global, so it neither needs nor clashes with @types/node.
4
+ const scope = globalThis;
5
+ try {
6
+ return scope.process?.env?.[name]?.trim() || undefined;
7
+ }
8
+ catch {
9
+ return undefined; // Deno without --allow-env throws
10
+ }
11
+ }
@@ -0,0 +1,37 @@
1
+ /**
2
+ * Shared by the Vite plugin and the Astro integration: fetch llms.txt (and llms-full.txt) for a site and write
3
+ * them into a build's output directory.
4
+ *
5
+ * @module
6
+ */
7
+ /** Options for the Vite plugin and the Astro integration. */
8
+ export interface LlmsFilesOptions {
9
+ /** The deployed site to describe, e.g. `https://example.com`. llms.ink reads the live site. */
10
+ site: string;
11
+ /** Also write llms-full.txt (the content of up to 20 pages). Default `false`. */
12
+ full?: boolean;
13
+ /** API key (`lk_…`). Defaults to the `LLMS_INK_KEY` environment variable; anonymous without one. */
14
+ key?: string;
15
+ /**
16
+ * Default `true`: when the live site publishes its own llms.txt, that file is used as is, unless it's one
17
+ * llms.ink generated before (it carries the llms.ink credit line), which is regenerated. `false` always
18
+ * generates (`native=0`).
19
+ */
20
+ native?: boolean;
21
+ /** Fail the build when a file can't be fetched. Default `false`: warn and keep any existing file. */
22
+ strict?: boolean;
23
+ /** Milliseconds to wait for llms.ink per file. Default 300000 (building a big site can take minutes). */
24
+ timeout?: number;
25
+ /** Defaults to the `LLMS_INK_API` environment variable, else https://llms.ink. */
26
+ baseUrl?: string;
27
+ }
28
+ /** Where messages go: Vite's or Astro's logger. */
29
+ export interface Log {
30
+ info(message: string): void;
31
+ warn(message: string): void;
32
+ }
33
+ /**
34
+ * A writer that fetches the files once and writes them into `dir` on every call (watch builds call it again;
35
+ * a failed fetch is retried on the next call).
36
+ */
37
+ export declare function llmsFilesWriter(options: LlmsFilesOptions): (dir: string, log: Log) => Promise<void>;
package/dist/files.js ADDED
@@ -0,0 +1,52 @@
1
+ /**
2
+ * Shared by the Vite plugin and the Astro integration: fetch llms.txt (and llms-full.txt) for a site and write
3
+ * them into a build's output directory.
4
+ *
5
+ * @module
6
+ */
7
+ import { access, mkdir, writeFile } from 'node:fs/promises';
8
+ import { join } from 'node:path';
9
+ import { LlmsInk } from "./index.js";
10
+ const FILES = [['llms.txt', 'llms'], ['llms-full.txt', 'full']];
11
+ /**
12
+ * A writer that fetches the files once and writes them into `dir` on every call (watch builds call it again;
13
+ * a failed fetch is retried on the next call).
14
+ */
15
+ export function llmsFilesWriter(options) {
16
+ let fetched;
17
+ return async (dir, log) => {
18
+ const { site, strict = false } = options;
19
+ if (!site) {
20
+ const message = 'No `site` set: llms.txt not written.';
21
+ if (strict)
22
+ throw new Error(message);
23
+ return log.warn(message);
24
+ }
25
+ fetched ??= fetchFiles(options);
26
+ const results = await fetched;
27
+ const failures = [];
28
+ for (const result of results) {
29
+ const path = join(dir, result.name);
30
+ if ('text' in result) {
31
+ await mkdir(dir, { recursive: true });
32
+ await writeFile(path, result.text);
33
+ log.info(`Wrote ${result.name} for ${site} (${result.source}, ~${result.tokens} tokens).`);
34
+ continue;
35
+ }
36
+ const reason = result.error instanceof Error ? result.error.message : String(result.error);
37
+ const kept = await access(path).then(() => 'kept the existing file', () => 'none written');
38
+ failures.push(`Couldn't get ${result.name} for ${site}: ${reason} (${kept}).`);
39
+ }
40
+ if (failures.length)
41
+ fetched = undefined;
42
+ if (strict && failures.length)
43
+ throw new Error(failures.join(' '));
44
+ for (const message of failures)
45
+ log.warn(message);
46
+ };
47
+ }
48
+ async function fetchFiles({ site, full, key, native = true, timeout = 300_000, baseUrl }) {
49
+ const ink = new LlmsInk({ key, baseUrl });
50
+ const files = full ? FILES : FILES.slice(0, 1);
51
+ return Promise.all(files.map(([name, mode]) => ink.llms(site, { mode, native, signal: AbortSignal.timeout(timeout) }).then(({ text, tokens, source }) => ({ name, text, tokens, source }), (error) => ({ name, error }))));
52
+ }
@@ -0,0 +1,235 @@
1
+ /**
2
+ * Client for the llms.ink JSON API (https://llms.ink/docs/): any public link — site, docs section, page,
3
+ * GitHub repo, PDF or video — as llms.txt or llms-full.txt. No dependencies; uses the global `fetch`, so it
4
+ * runs on Node ≥ 18, Bun, Deno, Cloudflare Workers and browsers.
5
+ *
6
+ * @example
7
+ * ```ts
8
+ * import { LlmsInk } from "llms-ink";
9
+ *
10
+ * const ink = new LlmsInk();
11
+ * const { title, text } = await ink.llms("developers.cloudflare.com/kv/", { mode: "full" });
12
+ * ```
13
+ *
14
+ * @module
15
+ */
16
+ export declare const VERSION = "0.2.0";
17
+ export declare const DEFAULT_BASE_URL = "https://llms.ink";
18
+ /** `llms` for the index (llms.txt), `full` for llms-full.txt. */
19
+ export type Mode = 'llms' | 'full';
20
+ /** What the link is. */
21
+ export type Kind = 'site' | 'section' | 'page' | 'repo' | 'document' | 'video';
22
+ /** `native` = published by the site; `generated` = made by llms.ink. */
23
+ export type Source = 'native' | 'generated';
24
+ /** A transcript line; `start` and `end` are seconds into the source video. */
25
+ export interface Segment {
26
+ start: number;
27
+ end: number;
28
+ text: string;
29
+ }
30
+ /** Video results only: what `/api/v0/frame` and `/api/v0/media` can serve. */
31
+ export interface Media {
32
+ /** Seconds. */
33
+ duration: number | null;
34
+ hasVideo: boolean;
35
+ hasAudio: boolean;
36
+ thumbnail: string | null;
37
+ downloads: Array<'mp4' | 'm4a'>;
38
+ isLive?: boolean;
39
+ transcript?: {
40
+ source: string;
41
+ segments: Segment[];
42
+ };
43
+ }
44
+ /** `GET /api/v0/llms`: the llms.txt or llms-full.txt of a link and what it was built from. */
45
+ export interface Result {
46
+ /** The link after normalization. */
47
+ url: string;
48
+ kind: Kind;
49
+ source: Source;
50
+ mode: Mode;
51
+ title: string;
52
+ /** The llms.txt or llms-full.txt content (Markdown). */
53
+ text: string;
54
+ /** Approximate token count (characters ÷ 4). */
55
+ tokens: number;
56
+ /** Links in llms.txt (mode `llms`) or pages inlined (mode `full`). */
57
+ pages: number;
58
+ /** Model that wrote the descriptions; null for native files or path-based grouping. */
59
+ model: string | null;
60
+ cached: boolean;
61
+ /** ISO 8601 date-time. */
62
+ generatedAt: string;
63
+ media?: Media;
64
+ }
65
+ /** One finding of the checker. */
66
+ export interface CheckIssue {
67
+ /** `error` breaks the format or a link; `warning` is worth fixing; `info` is a note. */
68
+ level: 'error' | 'warning' | 'info';
69
+ /** Stable name of the check, e.g. `missing_title`, `broken_link`. */
70
+ code: string;
71
+ /** Line in the file, when the issue has one. */
72
+ line: number | null;
73
+ message: string;
74
+ }
75
+ /** `GET /api/v0/check`: a published llms.txt checked against the format and its links. */
76
+ export interface CheckResult {
77
+ /** The site's origin. */
78
+ site: string;
79
+ /** The llms.txt that was checked (or the last place looked when none was found). */
80
+ url: string;
81
+ found: boolean;
82
+ /** HTTP status of the llms.txt request. */
83
+ status: number;
84
+ title: string | null;
85
+ tokens: number;
86
+ /** H2 sections. */
87
+ sections: number;
88
+ links: {
89
+ total: number;
90
+ /** Links fetched this run (the first 25). */
91
+ checked: number;
92
+ /** `status` is 0 when the link didn't answer in time. */
93
+ broken: Array<{
94
+ url: string;
95
+ status: number;
96
+ }>;
97
+ };
98
+ /** Whether llms-full.txt is published next to it. */
99
+ llmsFull: boolean;
100
+ /** Errors first, then warnings and notes, in line order. */
101
+ issues: CheckIssue[];
102
+ summary: {
103
+ errors: number;
104
+ warnings: number;
105
+ };
106
+ /** ISO 8601 date-time. */
107
+ checkedAt: string;
108
+ }
109
+ /** Every error code the API answers with (https://llms.ink/docs/#errors). */
110
+ export type ErrorCode = 'invalid_request' | 'invalid_url' | 'invalid_key' | 'not_found' | 'unknown_version' | 'method_not_allowed' | 'payload_too_large' | 'unprocessable' | 'rate_limited' | 'quota_exceeded' | 'internal' | 'upstream_error' | 'unavailable' | 'timeout' | 'loop_detected';
111
+ /** `invalid_request` only: one invalid parameter. */
112
+ export interface Issue {
113
+ path: string;
114
+ message: string;
115
+ }
116
+ /** An error body (RFC 9457 problem details), sent as `application/problem+json`. */
117
+ export interface Problem {
118
+ type: string;
119
+ title: string;
120
+ status: number;
121
+ code: ErrorCode;
122
+ detail: string;
123
+ errors?: Issue[];
124
+ }
125
+ /** Build stages reported while streaming. */
126
+ export type Stage = 'cache' | 'index' | 'extract' | 'ready';
127
+ /**
128
+ * One line of the `application/x-ndjson` stream of `GET /api/v0/llms`. `partial` text is explicitly
129
+ * incomplete; `result` is the last event. A terminal `error` event is thrown as an {@link LlmsInkError}.
130
+ */
131
+ export type StreamEvent = {
132
+ type: 'progress';
133
+ stage: Stage;
134
+ detail: string;
135
+ } | {
136
+ type: 'partial';
137
+ text: string;
138
+ } | {
139
+ type: 'result';
140
+ result: Result;
141
+ };
142
+ /** Fields of an {@link LlmsInkError} besides the message. */
143
+ export interface LlmsInkErrorInit {
144
+ status?: number;
145
+ code?: string;
146
+ title?: string;
147
+ errors?: Issue[];
148
+ retryAfter?: number;
149
+ cause?: unknown;
150
+ }
151
+ /**
152
+ * An API error, or llms.ink couldn't be reached (then `status` and `code` are undefined).
153
+ * `code` is the problem's machine-readable code (`invalid_url`, `unprocessable`, `upstream_error`…),
154
+ * `detail` the human-readable message, `errors` the per-parameter issues of `invalid_request`.
155
+ * `retryAfter` is the `Retry-After` header in seconds, when the response had one.
156
+ */
157
+ export declare class LlmsInkError extends Error {
158
+ readonly detail: string;
159
+ readonly status: number | undefined;
160
+ readonly code: string | undefined;
161
+ readonly title: string | undefined;
162
+ readonly errors: Issue[];
163
+ readonly retryAfter: number | undefined;
164
+ constructor(detail: string, init?: LlmsInkErrorInit);
165
+ }
166
+ /** 401 `invalid_key`: the API key is malformed, unknown or revoked. Requests never fall back to anonymous. */
167
+ export declare class InvalidKeyError extends LlmsInkError {
168
+ }
169
+ /** 429 `rate_limited`: too many requests this minute; wait `retryAfter` seconds. */
170
+ export declare class RateLimitedError extends LlmsInkError {
171
+ }
172
+ /**
173
+ * 429 `quota_exceeded`: the daily quota of new links is used up; `retryAfter` is the seconds until it resets
174
+ * (midnight UTC). Cached links still work.
175
+ */
176
+ export declare class QuotaExceededError extends LlmsInkError {
177
+ }
178
+ /** Options for {@link LlmsInk}. */
179
+ export interface LlmsInkOptions {
180
+ /**
181
+ * API key (`lk_…`), sent as `Authorization: Bearer`. Defaults to the `LLMS_INK_KEY` environment variable
182
+ * where there is one; without a key, requests are anonymous and share the free limits.
183
+ */
184
+ key?: string;
185
+ /** Defaults to the `LLMS_INK_API` environment variable, else https://llms.ink. */
186
+ baseUrl?: string;
187
+ /** A `fetch` to use instead of the global one. */
188
+ fetch?: typeof fetch;
189
+ }
190
+ /** Options for {@link LlmsInk.llms} and {@link LlmsInk.stream}. */
191
+ export interface LlmsOptions {
192
+ /** `llms` (default) for the index, `full` for llms-full.txt (the content inlined, up to 20 pages). */
193
+ mode?: Mode;
194
+ /** Skip the cache and rebuild (counts against the daily quota). */
195
+ fresh?: boolean;
196
+ /**
197
+ * `false` always generates the file from the site's pages and never returns the site's own llms.txt or
198
+ * llms-full.txt (`native=0`). Defaults to `true`: a site's own file is returned as is, unless it carries the
199
+ * llms.ink credit line (then it's regenerated).
200
+ */
201
+ native?: boolean;
202
+ /** Aborts the request; the abort error is rethrown as is. */
203
+ signal?: AbortSignal;
204
+ }
205
+ /** Options for {@link LlmsInk.check}. */
206
+ export interface CheckOptions {
207
+ signal?: AbortSignal;
208
+ }
209
+ /**
210
+ * A client for https://llms.ink.
211
+ *
212
+ * Building a big site or a video can take minutes; pass a `signal` (e.g. `AbortSignal.timeout(300_000)`) to
213
+ * bound a call, or use {@link LlmsInk.stream} to watch progress.
214
+ */
215
+ export declare class LlmsInk {
216
+ #private;
217
+ /** The API key in use, if any. */
218
+ readonly key: string | undefined;
219
+ /** Origin of the API, without a trailing slash. */
220
+ readonly baseUrl: string;
221
+ constructor(options?: LlmsInkOptions);
222
+ /**
223
+ * llms.txt (`mode: 'llms'`, an index of links) or llms-full.txt (`mode: 'full'`, the content inlined) for
224
+ * any public link.
225
+ */
226
+ llms(url: string, options?: LlmsOptions): Promise<Result>;
227
+ /** Checks a site's published llms.txt against the llmstxt.org format and that its links answer. */
228
+ check(url: string, options?: CheckOptions): Promise<CheckResult>;
229
+ /**
230
+ * Like {@link LlmsInk.llms}, as the build happens: yields `progress` and `partial` events, then one
231
+ * `result`. Errors — including a terminal `error` event — are thrown as {@link LlmsInkError}s.
232
+ * Breaking out of the loop cancels the request.
233
+ */
234
+ stream(url: string, options?: LlmsOptions): AsyncGenerator<StreamEvent, void, undefined>;
235
+ }
package/dist/index.js ADDED
@@ -0,0 +1,237 @@
1
+ /**
2
+ * Client for the llms.ink JSON API (https://llms.ink/docs/): any public link — site, docs section, page,
3
+ * GitHub repo, PDF or video — as llms.txt or llms-full.txt. No dependencies; uses the global `fetch`, so it
4
+ * runs on Node ≥ 18, Bun, Deno, Cloudflare Workers and browsers.
5
+ *
6
+ * @example
7
+ * ```ts
8
+ * import { LlmsInk } from "llms-ink";
9
+ *
10
+ * const ink = new LlmsInk();
11
+ * const { title, text } = await ink.llms("developers.cloudflare.com/kv/", { mode: "full" });
12
+ * ```
13
+ *
14
+ * @module
15
+ */
16
+ import { env } from "./env.js";
17
+ export const VERSION = '0.2.0';
18
+ export const DEFAULT_BASE_URL = 'https://llms.ink';
19
+ /**
20
+ * An API error, or llms.ink couldn't be reached (then `status` and `code` are undefined).
21
+ * `code` is the problem's machine-readable code (`invalid_url`, `unprocessable`, `upstream_error`…),
22
+ * `detail` the human-readable message, `errors` the per-parameter issues of `invalid_request`.
23
+ * `retryAfter` is the `Retry-After` header in seconds, when the response had one.
24
+ */
25
+ export class LlmsInkError extends Error {
26
+ detail;
27
+ status;
28
+ code;
29
+ title;
30
+ errors;
31
+ retryAfter;
32
+ constructor(detail, init = {}) {
33
+ super(detail, init.cause === undefined ? undefined : { cause: init.cause });
34
+ this.name = new.target.name;
35
+ this.detail = detail;
36
+ this.status = init.status;
37
+ this.code = init.code;
38
+ this.title = init.title;
39
+ this.errors = init.errors ?? [];
40
+ this.retryAfter = init.retryAfter;
41
+ }
42
+ }
43
+ /** 401 `invalid_key`: the API key is malformed, unknown or revoked. Requests never fall back to anonymous. */
44
+ export class InvalidKeyError extends LlmsInkError {
45
+ }
46
+ /** 429 `rate_limited`: too many requests this minute; wait `retryAfter` seconds. */
47
+ export class RateLimitedError extends LlmsInkError {
48
+ }
49
+ /**
50
+ * 429 `quota_exceeded`: the daily quota of new links is used up; `retryAfter` is the seconds until it resets
51
+ * (midnight UTC). Cached links still work.
52
+ */
53
+ export class QuotaExceededError extends LlmsInkError {
54
+ }
55
+ const BY_CODE = {
56
+ invalid_key: InvalidKeyError,
57
+ rate_limited: RateLimitedError,
58
+ quota_exceeded: QuotaExceededError,
59
+ };
60
+ const errorFor = (detail, init) => new (BY_CODE[init.code ?? ''] ?? LlmsInkError)(detail, init);
61
+ /** Browsers refuse or preflight a custom User-Agent, so it's only sent from servers. */
62
+ const IS_BROWSER = 'document' in globalThis || 'WorkerGlobalScope' in globalThis;
63
+ /** `Retry-After` as seconds: delta-seconds, or an HTTP date turned into seconds from now. */
64
+ function retryAfter(value) {
65
+ if (!value)
66
+ return undefined;
67
+ const v = value.trim();
68
+ if (/^\d+$/.test(v))
69
+ return Number(v);
70
+ const when = Date.parse(v);
71
+ return Number.isNaN(when) ? undefined : Math.max(0, Math.ceil((when - Date.now()) / 1000));
72
+ }
73
+ /** Aborts (`signal`, `AbortSignal.timeout`) are the caller's doing and pass through unwrapped. */
74
+ const isAbort = (err) => err instanceof Error && (err.name === 'AbortError' || err.name === 'TimeoutError');
75
+ /**
76
+ * A client for https://llms.ink.
77
+ *
78
+ * Building a big site or a video can take minutes; pass a `signal` (e.g. `AbortSignal.timeout(300_000)`) to
79
+ * bound a call, or use {@link LlmsInk.stream} to watch progress.
80
+ */
81
+ export class LlmsInk {
82
+ /** The API key in use, if any. */
83
+ key;
84
+ /** Origin of the API, without a trailing slash. */
85
+ baseUrl;
86
+ #fetch;
87
+ constructor(options = {}) {
88
+ const key = (options.key ?? env('LLMS_INK_KEY'))?.trim();
89
+ this.key = key || undefined;
90
+ this.baseUrl = (options.baseUrl ?? env('LLMS_INK_API') ?? DEFAULT_BASE_URL).replace(/\/+$/, '');
91
+ if (!options.fetch && typeof globalThis.fetch !== 'function') {
92
+ throw new TypeError('No global fetch: use Node 18+ or pass options.fetch.');
93
+ }
94
+ // Wrapped so the global is called unbound-safe (browsers throw "Illegal invocation" otherwise).
95
+ this.#fetch = options.fetch ?? ((input, init) => globalThis.fetch(input, init));
96
+ }
97
+ /**
98
+ * llms.txt (`mode: 'llms'`, an index of links) or llms-full.txt (`mode: 'full'`, the content inlined) for
99
+ * any public link.
100
+ */
101
+ async llms(url, options = {}) {
102
+ const res = await this.#get('/api/v0/llms', llmsParams(url, options), 'application/json', options.signal);
103
+ return (await this.#json(res));
104
+ }
105
+ /** Checks a site's published llms.txt against the llmstxt.org format and that its links answer. */
106
+ async check(url, options = {}) {
107
+ const res = await this.#get('/api/v0/check', { url }, 'application/json', options.signal);
108
+ return (await this.#json(res));
109
+ }
110
+ /**
111
+ * Like {@link LlmsInk.llms}, as the build happens: yields `progress` and `partial` events, then one
112
+ * `result`. Errors — including a terminal `error` event — are thrown as {@link LlmsInkError}s.
113
+ * Breaking out of the loop cancels the request.
114
+ */
115
+ async *stream(url, options = {}) {
116
+ const res = await this.#get('/api/v0/llms', llmsParams(url, options), 'application/x-ndjson', options.signal);
117
+ if (!res.headers.get('content-type')?.includes('ndjson')) {
118
+ yield { type: 'result', result: (await this.#json(res)) };
119
+ return;
120
+ }
121
+ if (!res.body)
122
+ throw new LlmsInkError(`${this.baseUrl} sent an empty stream.`, { status: res.status });
123
+ const reader = res.body.getReader();
124
+ const decoder = new TextDecoder();
125
+ let buffer = '';
126
+ let done = false;
127
+ try {
128
+ while (!done) {
129
+ let chunk;
130
+ try {
131
+ chunk = await reader.read();
132
+ }
133
+ catch (err) {
134
+ if (isAbort(err))
135
+ throw err;
136
+ throw new LlmsInkError(`The stream from ${this.baseUrl} broke off: ${err instanceof Error ? err.message : err}`, { cause: err });
137
+ }
138
+ done = chunk.done;
139
+ buffer += done ? decoder.decode() : decoder.decode(chunk.value, { stream: true });
140
+ const lines = buffer.split('\n');
141
+ buffer = done ? '' : lines.pop();
142
+ for (const line of lines) {
143
+ const event = line.trim() ? this.#event(line) : undefined;
144
+ if (!event)
145
+ continue;
146
+ yield event;
147
+ if (event.type === 'result')
148
+ return;
149
+ }
150
+ }
151
+ }
152
+ finally {
153
+ reader.cancel().catch(() => { });
154
+ }
155
+ throw new LlmsInkError(`The stream from ${this.baseUrl} ended without a result.`);
156
+ }
157
+ /** A stream line → its event; unknown event types (added later) are skipped, `error` is thrown. */
158
+ #event(line) {
159
+ let event;
160
+ try {
161
+ event = JSON.parse(line);
162
+ }
163
+ catch (err) {
164
+ throw new LlmsInkError(`${this.baseUrl} sent a stream line that isn't JSON.`, { cause: err });
165
+ }
166
+ if (!event || typeof event !== 'object' || !('type' in event))
167
+ return undefined;
168
+ if (event.type === 'error') {
169
+ const { detail, code, status } = event;
170
+ throw errorFor(detail || 'The build failed.', { status, code });
171
+ }
172
+ // The server's own event shapes, by type.
173
+ const known = event.type === 'progress' || event.type === 'partial' || event.type === 'result';
174
+ return known ? event : undefined;
175
+ }
176
+ async #get(path, params, accept, signal) {
177
+ const headers = { accept };
178
+ if (!IS_BROWSER)
179
+ headers['user-agent'] = `llms-ink-js/${VERSION}`;
180
+ if (this.key)
181
+ headers.authorization = `Bearer ${this.key}`;
182
+ const target = `${this.baseUrl}${path}?${new URLSearchParams(params)}`;
183
+ let res;
184
+ try {
185
+ res = await this.#fetch(target, { headers, signal });
186
+ }
187
+ catch (err) {
188
+ if (isAbort(err))
189
+ throw err;
190
+ throw new LlmsInkError(`Couldn't reach ${this.baseUrl}: ${err instanceof Error ? err.message : err}`, { cause: err });
191
+ }
192
+ if (!res.ok)
193
+ throw await problem(res);
194
+ return res;
195
+ }
196
+ async #json(res) {
197
+ const body = await res.text();
198
+ try {
199
+ return JSON.parse(body);
200
+ }
201
+ catch (err) {
202
+ throw new LlmsInkError(`${this.baseUrl} answered with something other than JSON.`, { status: res.status, cause: err });
203
+ }
204
+ }
205
+ }
206
+ function llmsParams(url, { mode = 'llms', fresh, native = true }) {
207
+ if (mode !== 'llms' && mode !== 'full') {
208
+ throw new TypeError(`mode must be 'llms' (the index) or 'full' (llms-full.txt), not ${JSON.stringify(mode)}`);
209
+ }
210
+ const params = { url, mode };
211
+ if (fresh)
212
+ params.fresh = '1';
213
+ if (!native)
214
+ params.native = '0';
215
+ return params;
216
+ }
217
+ /** An HTTP error response → the typed error for its problem `code`. */
218
+ async function problem(res) {
219
+ let data = {};
220
+ try {
221
+ const parsed = JSON.parse(await res.text());
222
+ if (parsed && typeof parsed === 'object')
223
+ data = parsed;
224
+ }
225
+ catch {
226
+ // not problem+json: fall back to the status line
227
+ }
228
+ const status = typeof data.status === 'number' ? data.status : res.status;
229
+ const detail = data.detail || data.title || `HTTP ${res.status} ${res.statusText || 'error'}`.trim();
230
+ return errorFor(detail, {
231
+ status,
232
+ code: data.code,
233
+ title: data.title,
234
+ errors: data.errors,
235
+ retryAfter: retryAfter(res.headers.get('retry-after')),
236
+ });
237
+ }
@@ -0,0 +1,64 @@
1
+ /**
2
+ * Serve llms.txt, llms-full.txt and (optionally) Markdown versions of pages from llms.ink, inside your own app.
3
+ * A fetch-API handler for Cloudflare Workers, Next.js middleware, Deno, Bun and Hono, plus an Express/Connect
4
+ * adapter.
5
+ *
6
+ * @example
7
+ * ```ts
8
+ * import { llmsInkHandler } from 'llms-ink/middleware';
9
+ *
10
+ * const llms = llmsInkHandler({ site: 'https://example.com', markdown: true });
11
+ * export default {
12
+ * async fetch(request: Request): Promise<Response> {
13
+ * return (await llms(request)) ?? fetch(request); // undefined: not ours, the app answers
14
+ * },
15
+ * };
16
+ * ```
17
+ *
18
+ * @module
19
+ */
20
+ /** Options for {@link llmsInkHandler} and {@link llmsInkExpress}. */
21
+ export interface LlmsInkHandlerOptions {
22
+ /** The deployed site to describe, e.g. `https://example.com`. */
23
+ site: string;
24
+ /**
25
+ * API key (`lk_…`), sent as `Authorization: Bearer`. Defaults to the `LLMS_INK_KEY` environment variable
26
+ * where there is one. Recommended: proxies on shared hosting IPs otherwise share the anonymous limits.
27
+ */
28
+ key?: string;
29
+ /**
30
+ * Also answer `<path>.md`, and page requests whose `Accept` prefers `text/markdown` over `text/html`, with the
31
+ * page as Markdown (`https://llms.ink/full/<page>`). Default `false`.
32
+ */
33
+ markdown?: boolean;
34
+ /** Defaults to the `LLMS_INK_API` environment variable, else https://llms.ink. */
35
+ baseUrl?: string;
36
+ }
37
+ /** A fetch-API handler: a `Response` for the requests it serves, `undefined` for everything else. */
38
+ export type LlmsInkHandler = (request: Request) => Promise<Response | undefined>;
39
+ /**
40
+ * Answers `GET`/`HEAD` `/llms.txt` and `/llms-full.txt` with llms.ink's generated files for `site`
41
+ * (`native=0`, so a site whose /llms.txt is this handler can't loop), and, with `markdown: true`, Markdown
42
+ * versions of pages. Responses keep llms.ink's `Cache-Control`; each one is kept in memory for its `max-age`
43
+ * and served as a last good copy (with `Warning: 110`) when llms.ink fails. Returns `undefined` for every
44
+ * other request, and for llms.ink's own fetcher.
45
+ */
46
+ export declare function llmsInkHandler(options: LlmsInkHandlerOptions): LlmsInkHandler;
47
+ /** Node's `IncomingMessage` (Express, Connect, plain `node:http`), as far as the adapter reads it. */
48
+ export interface NodeRequest {
49
+ method?: string;
50
+ url?: string;
51
+ /** Express: the URL before mounting stripped a prefix. */
52
+ originalUrl?: string;
53
+ headers: Record<string, string | string[] | undefined>;
54
+ }
55
+ /** Node's `ServerResponse`, as far as the adapter writes it. */
56
+ export interface NodeResponse {
57
+ statusCode: number;
58
+ setHeader(name: string, value: string): unknown;
59
+ end(chunk: Uint8Array): unknown;
60
+ }
61
+ /** Express/Connect middleware: `next()` when the request isn't one {@link llmsInkHandler} serves. */
62
+ export type NodeMiddleware = (req: NodeRequest, res: NodeResponse, next: (err?: unknown) => void) => void;
63
+ /** {@link llmsInkHandler} as Express/Connect middleware: `app.use(llmsInkExpress({ site }))`. */
64
+ export declare function llmsInkExpress(options: LlmsInkHandlerOptions): NodeMiddleware;
@@ -0,0 +1,183 @@
1
+ /**
2
+ * Serve llms.txt, llms-full.txt and (optionally) Markdown versions of pages from llms.ink, inside your own app.
3
+ * A fetch-API handler for Cloudflare Workers, Next.js middleware, Deno, Bun and Hono, plus an Express/Connect
4
+ * adapter.
5
+ *
6
+ * @example
7
+ * ```ts
8
+ * import { llmsInkHandler } from 'llms-ink/middleware';
9
+ *
10
+ * const llms = llmsInkHandler({ site: 'https://example.com', markdown: true });
11
+ * export default {
12
+ * async fetch(request: Request): Promise<Response> {
13
+ * return (await llms(request)) ?? fetch(request); // undefined: not ours, the app answers
14
+ * },
15
+ * };
16
+ * ```
17
+ *
18
+ * @module
19
+ */
20
+ import { env } from "./env.js";
21
+ import { DEFAULT_BASE_URL, VERSION } from "./index.js";
22
+ const FILES = { '/llms.txt': 'llms', '/llms-full.txt': 'full' };
23
+ /** llms.ink's own fetcher (`compatible; llms.ink/0.1`): answering it from llms.ink would loop. */
24
+ const LLMS_INK_FETCHER = /\bllms\.ink\/\d/;
25
+ /** Response headers passed through from llms.ink. */
26
+ const PASS = ['content-type', 'cache-control', 'x-llms-kind', 'x-llms-source', 'x-llms-tokens'];
27
+ /** Pages kept in memory as last good copies; the oldest is dropped first. */
28
+ const MAX_ENTRIES = 200;
29
+ /** How long downstream caches may keep a stale copy served while llms.ink fails. */
30
+ const STALE_CACHE_CONTROL = 'public, max-age=60';
31
+ /**
32
+ * Answers `GET`/`HEAD` `/llms.txt` and `/llms-full.txt` with llms.ink's generated files for `site`
33
+ * (`native=0`, so a site whose /llms.txt is this handler can't loop), and, with `markdown: true`, Markdown
34
+ * versions of pages. Responses keep llms.ink's `Cache-Control`; each one is kept in memory for its `max-age`
35
+ * and served as a last good copy (with `Warning: 110`) when llms.ink fails. Returns `undefined` for every
36
+ * other request, and for llms.ink's own fetcher.
37
+ */
38
+ export function llmsInkHandler(options) {
39
+ const base = (options.baseUrl ?? env('LLMS_INK_API') ?? DEFAULT_BASE_URL).replace(/\/+$/, '');
40
+ const key = (options.key ?? env('LLMS_INK_KEY'))?.trim();
41
+ const site = new URL(/^https?:\/\//i.test(options.site) ? options.site : `https://${options.site}`);
42
+ const headers = { accept: 'text/plain', 'user-agent': `llms-ink-js/${VERSION}` };
43
+ if (key)
44
+ headers.authorization = `Bearer ${key}`;
45
+ const cache = new Map();
46
+ const pending = new Map();
47
+ /** Fetches `upstream`; on failure the last good copy, else llms.ink's error. */
48
+ async function refresh(upstream) {
49
+ const last = cache.get(upstream);
50
+ let res;
51
+ try {
52
+ res = await fetch(upstream, { headers });
53
+ }
54
+ catch (err) {
55
+ return last ? stale(last) : failure(502, 'upstream_error', `Couldn't reach llms.ink: ${err instanceof Error ? err.message : err}`);
56
+ }
57
+ // An older llms.ink without `format=text` answers JSON; never serve that as the file.
58
+ const isText = res.headers.get('content-type')?.startsWith('text/');
59
+ if (!res.ok || !isText) {
60
+ if (last) {
61
+ await res.body?.cancel();
62
+ return stale(last);
63
+ }
64
+ if (res.ok)
65
+ return failure(502, 'upstream_error', `llms.ink answered ${res.headers.get('content-type')}, not text.`);
66
+ return new Response(res.body, { status: res.status, headers: pick(res.headers, [...PASS, 'x-error-code', 'retry-after']) });
67
+ }
68
+ const maxAge = Number(/(?:^|,)\s*max-age=(\d+)/i.exec(res.headers.get('cache-control') ?? '')?.[1] ?? 0);
69
+ const entry = { body: await res.text(), headers: pick(res.headers, PASS), at: Date.now(), maxAge };
70
+ cache.delete(upstream);
71
+ cache.set(upstream, entry);
72
+ if (cache.size > MAX_ENTRIES)
73
+ cache.delete(cache.keys().next().value);
74
+ return fresh(entry);
75
+ }
76
+ async function serve(upstream) {
77
+ const hit = cache.get(upstream);
78
+ if (hit && Date.now() - hit.at < hit.maxAge * 1000)
79
+ return fresh(hit);
80
+ let loading = pending.get(upstream);
81
+ if (!loading) {
82
+ loading = refresh(upstream).finally(() => pending.delete(upstream));
83
+ pending.set(upstream, loading);
84
+ }
85
+ // Concurrent requests share one fetch; each gets its own copy of the body.
86
+ return (await loading).clone();
87
+ }
88
+ return async (request) => {
89
+ if (request.method !== 'GET' && request.method !== 'HEAD')
90
+ return undefined;
91
+ if (LLMS_INK_FETCHER.test(request.headers.get('user-agent') ?? ''))
92
+ return undefined;
93
+ const { pathname } = new URL(request.url);
94
+ const mode = FILES[pathname];
95
+ let response;
96
+ if (mode) {
97
+ const query = new URLSearchParams({ url: site.href, mode, native: '0', format: 'text' });
98
+ response = await serve(`${base}/api/v0/llms?${query}`);
99
+ }
100
+ else if (options.markdown && pathname.endsWith('.md')) {
101
+ const page = pathname.slice(0, -3).replace(/\/index$/, '/');
102
+ response = await serve(`${base}/full/${site.host}${page}`);
103
+ }
104
+ else if (options.markdown && isPage(pathname) && prefersMarkdown(request.headers.get('accept') ?? '')) {
105
+ response = await serve(`${base}/full/${site.host}${pathname}`);
106
+ response.headers.append('vary', 'Accept');
107
+ }
108
+ else {
109
+ return undefined;
110
+ }
111
+ return request.method === 'HEAD' ? new Response(null, response) : response;
112
+ };
113
+ }
114
+ /** {@link llmsInkHandler} as Express/Connect middleware: `app.use(llmsInkExpress({ site }))`. */
115
+ export function llmsInkExpress(options) {
116
+ const handle = llmsInkHandler(options);
117
+ return (req, res, next) => {
118
+ const headers = new Headers();
119
+ for (const name of ['accept', 'user-agent']) {
120
+ const value = req.headers[name];
121
+ if (typeof value === 'string')
122
+ headers.set(name, value);
123
+ }
124
+ const request = new Request(new URL(req.originalUrl ?? req.url ?? '/', 'http://localhost'), { method: req.method, headers });
125
+ handle(request)
126
+ .then(async (response) => {
127
+ if (!response)
128
+ return next();
129
+ res.statusCode = response.status;
130
+ response.headers.forEach((value, name) => res.setHeader(name, value));
131
+ res.end(new Uint8Array(await response.arrayBuffer()));
132
+ })
133
+ .catch(next);
134
+ };
135
+ }
136
+ function pick(from, names) {
137
+ const to = new Headers();
138
+ for (const name of names) {
139
+ const value = from.get(name);
140
+ if (value !== null)
141
+ to.set(name, value);
142
+ }
143
+ return to;
144
+ }
145
+ function fresh(entry) {
146
+ const headers = new Headers(entry.headers);
147
+ headers.set('age', String(Math.floor((Date.now() - entry.at) / 1000)));
148
+ return new Response(entry.body, { headers });
149
+ }
150
+ function stale(entry) {
151
+ const response = fresh(entry);
152
+ response.headers.set('cache-control', STALE_CACHE_CONTROL);
153
+ response.headers.set('warning', '110 - "Response is Stale"');
154
+ return response;
155
+ }
156
+ function failure(status, code, detail) {
157
+ return new Response(`${detail}\n`, { status, headers: { 'content-type': 'text/plain; charset=utf-8', 'x-error-code': code } });
158
+ }
159
+ /** A page path: no file extension, or `.html`/`.htm`. */
160
+ function isPage(pathname) {
161
+ const last = pathname.slice(pathname.lastIndexOf('/') + 1);
162
+ return !last.includes('.') || /\.html?$/i.test(last);
163
+ }
164
+ /**
165
+ * Whether `Accept` names `text/markdown` with a quality at least that of `text/html` (or, when `text/html`
166
+ * isn't listed, `text/*`, then `*\/*`). Browsers never list `text/markdown`, so they get the app.
167
+ */
168
+ function prefersMarkdown(accept) {
169
+ let markdown = 0;
170
+ const html = {};
171
+ for (const part of accept.toLowerCase().split(',')) {
172
+ const [type = '', ...params] = part.split(';').map((s) => s.trim());
173
+ const q = params.find((p) => p.startsWith('q='));
174
+ const quality = q === undefined ? 1 : Number(q.slice(2));
175
+ if (!Number.isFinite(quality))
176
+ continue;
177
+ if (type === 'text/markdown' || type === 'text/x-markdown')
178
+ markdown = Math.max(markdown, quality);
179
+ else if (type === 'text/html' || type === 'text/*' || type === '*/*')
180
+ html[type] = Math.max(html[type] ?? 0, quality);
181
+ }
182
+ return markdown > 0 && markdown >= (html['text/html'] ?? html['text/*'] ?? html['*/*'] ?? 0);
183
+ }
package/dist/vite.d.ts ADDED
@@ -0,0 +1,25 @@
1
+ /**
2
+ * Vite plugin: writes llms.txt (and llms-full.txt) from llms.ink into the build output.
3
+ *
4
+ * @example
5
+ * ```ts
6
+ * // vite.config.ts
7
+ * import { defineConfig } from 'vite';
8
+ * import { llmsInk } from 'llms-ink/vite';
9
+ *
10
+ * export default defineConfig({ plugins: [llmsInk({ site: 'https://example.com', full: true })] });
11
+ * ```
12
+ *
13
+ * @module
14
+ */
15
+ import type { Plugin } from 'vite';
16
+ import { type LlmsFilesOptions } from './files.ts';
17
+ export type { LlmsFilesOptions };
18
+ /**
19
+ * After `vite build` (not the dev server, not SSR builds), fetches llms.txt — and llms-full.txt with
20
+ * `full: true` — for `site` and writes them into `build.outDir`. llms.ink reads the live site, so the files
21
+ * describe the previous deploy. When llms.ink can't be reached it warns and keeps any existing file (e.g. one
22
+ * copied from `public/`); `strict: true` fails the build instead.
23
+ */
24
+ export declare function llmsInk(options: LlmsFilesOptions): Plugin;
25
+ export default llmsInk;
package/dist/vite.js ADDED
@@ -0,0 +1,47 @@
1
+ /**
2
+ * Vite plugin: writes llms.txt (and llms-full.txt) from llms.ink into the build output.
3
+ *
4
+ * @example
5
+ * ```ts
6
+ * // vite.config.ts
7
+ * import { defineConfig } from 'vite';
8
+ * import { llmsInk } from 'llms-ink/vite';
9
+ *
10
+ * export default defineConfig({ plugins: [llmsInk({ site: 'https://example.com', full: true })] });
11
+ * ```
12
+ *
13
+ * @module
14
+ */
15
+ import { resolve } from 'node:path';
16
+ import { llmsFilesWriter } from "./files.js";
17
+ /**
18
+ * After `vite build` (not the dev server, not SSR builds), fetches llms.txt — and llms-full.txt with
19
+ * `full: true` — for `site` and writes them into `build.outDir`. llms.ink reads the live site, so the files
20
+ * describe the previous deploy. When llms.ink can't be reached it warns and keeps any existing file (e.g. one
21
+ * copied from `public/`); `strict: true` fails the build instead.
22
+ */
23
+ export function llmsInk(options) {
24
+ const write = llmsFilesWriter(options);
25
+ let config;
26
+ return {
27
+ name: 'llms-ink',
28
+ apply: 'build',
29
+ configResolved(resolved) {
30
+ config = resolved;
31
+ },
32
+ async closeBundle(error) {
33
+ if (error)
34
+ return;
35
+ // Vite 6+ runs the hook once per environment; only the browser build is served as static files.
36
+ const environment = this.environment;
37
+ if (environment ? environment.name !== 'client' : config.build.ssr)
38
+ return;
39
+ const { outDir } = environment ? environment.config.build : config.build;
40
+ await write(resolve(config.root, outDir), {
41
+ info: (message) => config.logger.info(`[llms-ink] ${message}`),
42
+ warn: (message) => config.logger.warn(`[llms-ink] ${message}`),
43
+ });
44
+ },
45
+ };
46
+ }
47
+ export default llmsInk;
package/package.json CHANGED
@@ -1,6 +1,55 @@
1
1
  {
2
2
  "name": "llms-ink",
3
- "version": "0.0.0-stage",
4
- "stub": true,
5
- "description": "Temporary package placeholder for staged publishing"
6
- }
3
+ "version": "0.2.0",
4
+ "description": "Turn any link — site, docs section, page, GitHub repo, PDF or video — into llms.txt and llms-full.txt. Typed client for llms.ink, plus a Vite plugin, an Astro integration and middleware that serve the files from your site. No dependencies.",
5
+ "type": "module",
6
+ "exports": {
7
+ ".": {
8
+ "types": "./dist/index.d.ts",
9
+ "default": "./dist/index.js"
10
+ },
11
+ "./vite": {
12
+ "types": "./dist/vite.d.ts",
13
+ "default": "./dist/vite.js"
14
+ },
15
+ "./astro": {
16
+ "types": "./dist/astro.d.ts",
17
+ "default": "./dist/astro.js"
18
+ },
19
+ "./middleware": {
20
+ "types": "./dist/middleware.d.ts",
21
+ "default": "./dist/middleware.js"
22
+ }
23
+ },
24
+ "main": "./dist/index.js",
25
+ "types": "./dist/index.d.ts",
26
+ "files": ["dist"],
27
+ "sideEffects": false,
28
+ "engines": { "node": ">=18" },
29
+ "scripts": {
30
+ "build": "rm -rf dist && tsc -p tsconfig.json",
31
+ "prepublishOnly": "npm run build && npm test",
32
+ "test": "npm run build && node --test test/client.test.mjs test/plugins.test.mjs test/middleware.test.mjs",
33
+ "test:live": "npm run build && node --test test/live.test.mjs",
34
+ "test:bun": "npm run build && bun test ./test/client.test.mjs ./test/plugins.test.mjs ./test/middleware.test.mjs",
35
+ "test:bun:live": "npm run build && bun test --timeout 300000 ./test/live.test.mjs"
36
+ },
37
+ "keywords": ["llms.txt", "llms-full.txt", "llm", "ai", "markdown", "docs", "context", "rag", "sdk", "llms-ink", "vite-plugin", "astro-integration", "middleware"],
38
+ "homepage": "https://llms.ink/docs/",
39
+ "author": "Lucas Espindola (https://lucasespindola.com)",
40
+ "license": "MIT",
41
+ "peerDependencies": {
42
+ "astro": ">=4",
43
+ "vite": ">=5"
44
+ },
45
+ "peerDependenciesMeta": {
46
+ "astro": { "optional": true },
47
+ "vite": { "optional": true }
48
+ },
49
+ "devDependencies": {
50
+ "@types/node": "^22.20.5",
51
+ "astro": "^7.4.1",
52
+ "typescript": "^5.9.3",
53
+ "vite": "^8.3.8"
54
+ }
55
+ }