static-llms-txt 0.0.0-stage → 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 OdysseyX LLC
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,130 @@
-# Temporary Holding Version
+# static-llms-txt
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Write `llms.txt` and `llms-full.txt` for any static site, from the HTML it already built.
+
+Most llms.txt generators are tied to one framework and read its Markdown sources. This one reads the output folder, so it works after Eleventy, Hugo, Jekyll, Astro, Gatsby, Docusaurus, MkDocs, a hand-written site, or anything else that ends in `.html` files.
+
+- **`llms.txt`**: an index in the [llmstxt.org](https://llmstxt.org) shape. Title, summary, one link per page with its description.
+- **`llms-full.txt`**: the main content of every page as Markdown in one file, each page under its own heading with a `Source:` link.
+- **No network, no API key, no config file.** One dependency (`node-html-parser`).
+- Skips what should not be there: navigation, footers, scripts, `noindex` pages, redirects, 404 pages.
+- Leaves an `llms.txt` you wrote by hand alone.
+
+## Use it from the command line
+
+```sh
+npx static-llms-txt ./dist --site https://example.com
+```
+
+Put it after your build:
+
+```json
+{
+  "scripts": {
+    "build": "eleventy && static-llms-txt _site --site https://example.com"
+  }
+}
+```
+
+| Option | What it does |
+| --- | --- |
+| `--site <url>` | Public address of the site. Makes every link absolute. Without it, the page's canonical link is used. |
+| `--title <text>` | Site title. Default: taken from the home page. |
+| `--description <text>` | One-line summary. Default: the home page meta description. |
+| `--content <selector>` | CSS selector of the main content. Default: `main`, then `article`, then `body`. |
+| `--exclude <glob>` | Skip matching files, for example `"tags/**"`. Repeatable. |
+| `--clean-urls` | Link `/about` instead of `/about.html`. |
+| `--no-full` | Write `llms.txt` only. |
+| `--force` | Overwrite an `llms.txt` this tool did not write. |
+| `--ask-ai` | Also add an Ask AI box to every page. See below. |
+
+## Use it from code
+
+```js
+import { generate } from "static-llms-txt";
+
+const result = generate("./dist", {
+  site: "https://example.com",
+  exclude: ["tags/**", "drafts/**"],
+});
+console.log(result.pages.length, "pages", result.written);
+```
+
+CommonJS works too: `const { generate } = require("static-llms-txt")`.
+
+| Function | What it does |
+| --- | --- |
+| `generate(dir, options)` | Scan a folder and write both files into it. Returns `{ pages, llms, llmsFull, written, skipped }`. |
+| `scan(dir, options)` | Read every page. Returns `[{ url, path, title, description, sections, text }]`. |
+| `build(pages, options)` | Turn a list of pages (from `scan`, or your own `{ url, title, html }` objects) into the text of both files, without touching the disk. |
+| `readHtml(html, options)` | One HTML document to title, description and Markdown content. |
+| `inject(dir, options)` / `scriptTag(options)` | The optional Ask AI part. |
+
+Options for `generate` and `scan`: `site`, `title`, `description`, `content`, `remove` (extra selectors to drop), `exclude` (globs), `cleanUrls`, `minLength` (default 40 characters), `full`, `force`, `out`, `write`. Types are included.
+
+## What the output looks like
+
+`llms.txt`
+
+```md
+# Acme Docs
+
+> Acme makes anvils.
+
+## Pages
+
+- [Acme](https://acme.dev/): Acme makes anvils.
+- [Guide](https://acme.dev/guide/): How to start.
+
+## Optional
+
+- [Full text of every page](https://acme.dev/llms-full.txt)
+```
+
+`llms-full.txt`
+
+```md
+# Guide
+
+Source: https://acme.dev/guide/
+
+Start here with the guide.
+
+## Step one
+
+Source: https://acme.dev/guide/#step-one
+
+Unpack the anvil carefully.
+```
+
+## Optional: an Ask AI box that answers from llms-full.txt
+
+Once the site has an `llms-full.txt`, a single script tag can answer visitor questions from it. `--ask-ai` adds that tag to every page:
+
+```sh
+npx static-llms-txt ./dist --site https://example.com --ask-ai --key YOUR_ONETAG_KEY
+```
+
+The box is [OneTag AI](https://onetagai.com/?ref=static-llms-txt)'s open-source `ask-docs.js`. It finds the pages that match the question in `llms-full.txt`, sends them with the question to the OneTag AI API, and shows the answer with links to the pages it used. There is no backend to run.
+
+The key is a site key. It is public by design, like an analytics ID, and it ends up in the built HTML. The free plan is 1,000,000 tokens a month with a hard cap: when it runs out the box says so, and no bill can appear. Get one without opening a browser:
+
+```sh
+curl -X POST https://onetagai.com/api/signup -H "Content-Type: application/json" -d '{"email":"you@example.com"}'
+```
+
+Without `--key` the tool uses a demo key that answers on localhost only, so you can try it before signing up. Everything above this section works without OneTag AI.
+
+## Framework plugins built on this
+
+- `eleventy-plugin-onetag` for Eleventy
+- `docusaurus-plugin-onetag` for Docusaurus
+- `gatsby-plugin-onetag` for Gatsby
+- `hexo-onetag` for Hexo
+- `vitepress-plugin-onetag` for VitePress
+- `nuxt-onetag` for Nuxt
+- [`astro-onetag`](https://www.npmjs.com/package/astro-onetag) and [`starlight-onetag`](https://www.npmjs.com/package/starlight-onetag) for Astro
+
+## License
+
+MIT

package/cli.cjs

@@ -0,0 +1,67 @@
+#!/usr/bin/env node
+"use strict";
+const path = require("node:path");
+const { generate, inject } = require("./index.cjs");
+
+const HELP = `static-llms-txt: write llms.txt and llms-full.txt from a built static site.
+
+Usage
+  npx static-llms-txt <folder> [options]
+
+Options
+  --site <url>          Public address of the site, e.g. https://example.com. Used for absolute links.
+  --title <text>        Site title. Default: taken from the home page.
+  --description <text>  One-line summary. Default: the home page meta description.
+  --content <selector>  CSS selector of the main content. Default: main, article, then body.
+  --exclude <glob>      Skip matching files, e.g. "tags/**". Repeatable.
+  --clean-urls          Link /about instead of /about.html.
+  --no-full             Write llms.txt only.
+  --force               Overwrite an llms.txt this tool did not write.
+  --ask-ai              Also add an "Ask AI" box to every page (one script tag, answers from llms-full.txt).
+  --key <key>           OneTag AI site key for --ask-ai. Free: https://onetagai.com/?ref=static-llms-txt
+  -h, --help            Show this help.
+`;
+
+function parseArgs(argv) {
+  const o = { exclude: [] };
+  const rest = [];
+  for (let i = 0; i < argv.length; i++) {
+    const a = argv[i];
+    const next = () => { if (i + 1 >= argv.length) throw new Error("Missing value after " + a); return argv[++i]; };
+    if (a === "-h" || a === "--help") o.help = true;
+    else if (a === "--site") o.site = next();
+    else if (a === "--title") o.title = next();
+    else if (a === "--description") o.description = next();
+    else if (a === "--content") o.content = next();
+    else if (a === "--exclude") o.exclude.push(next());
+    else if (a === "--clean-urls") o.cleanUrls = true;
+    else if (a === "--no-full") o.full = false;
+    else if (a === "--force") o.force = true;
+    else if (a === "--ask-ai") o.askAi = true;
+    else if (a === "--key") o.key = next();
+    else if (a.startsWith("-")) throw new Error("Unknown option " + a + ". Run with --help to see the options.");
+    else rest.push(a);
+  }
+  o.dir = rest[0];
+  return o;
+}
+
+function main() {
+  let o;
+  try { o = parseArgs(process.argv.slice(2)); } catch (e) { console.error(e.message); process.exit(2); }
+  if (o.help || !o.dir) { console.log(HELP); process.exit(o.help ? 0 : 2); }
+  let result;
+  try { result = generate(o.dir, o); } catch (e) { console.error(e.message); process.exit(1); }
+  const dir = path.resolve(o.dir);
+  for (const f of result.written) console.log("wrote " + path.join(dir, f));
+  for (const f of result.skipped) console.log("kept  " + path.join(dir, f) + " (not written by this tool; pass --force to replace it)");
+  console.log(result.pages.length + " pages");
+  if (!o.site && result.pages.some((p) => !/^https?:/.test(p.url))) console.log("Tip: pass --site https://your-site.com so the links are absolute.");
+  if (o.askAi) {
+    const changed = inject(dir, { key: o.key || "ot_demo000000000000000000000", ref: "static-llms-txt", exclude: o.exclude });
+    console.log("added the Ask AI script to " + changed.length + " pages");
+    if (!o.key) console.log("No --key given: the demo key answers on localhost only. Get a free key:\n  curl -X POST https://onetagai.com/api/signup -H \"Content-Type: application/json\" -d '{\"email\":\"you@example.com\"}'");
+  }
+}
+
+main();

package/index.cjs

@@ -0,0 +1,356 @@
+// static-llms-txt: write llms.txt and llms-full.txt for a static site from the HTML it already built.
+// It reads a folder of .html files, keeps the main content of every page as Markdown, and writes
+//   llms.txt       an index: one link per page (https://llmstxt.org)
+//   llms-full.txt  every page in one file, one "# Title" + "Source: URL" block per page
+// No network, no API key. The optional Ask AI part (askAi / inject) only adds a script tag.
+"use strict";
+const fs = require("node:fs");
+const path = require("node:path");
+const { parse } = require("node-html-parser");
+
+const MARK = "<!-- generated by static-llms-txt -->";
+const SCRIPT = "https://onetagai.com/ask-docs.js";
+const DROP = "script,style,noscript,template,svg,canvas,iframe,object,embed,nav,footer,aside,form,button,select,dialog,[hidden],[aria-hidden=true],.sr-only,.visually-hidden";
+const ROOTS = ["main", "[role=main]", "article", "#content", ".content", "body"];
+const BLOCK = new Set(["p", "div", "section", "article", "main", "header", "figure", "figcaption", "details", "summary", "address", "dd", "dt", "dl", "tr", "table", "thead", "tbody", "tfoot", "ul", "ol", "li", "blockquote", "pre", "h1", "h2", "h3", "h4", "h5", "h6", "hr", "br"]);
+
+function decode(text) {
+  return String(text)
+    .replace(/&nbsp;/g, " ").replace(/&amp;/g, "&").replace(/&lt;/g, "<").replace(/&gt;/g, ">")
+    .replace(/&quot;/g, '"').replace(/&#0?39;|&apos;/g, "'")
+    .replace(/&#(\d+);/g, (m, n) => safeChar(Number(n), m)).replace(/&#x([0-9a-f]+);/gi, (m, n) => safeChar(parseInt(n, 16), m));
+}
+function safeChar(code, fallback) {
+  try { return String.fromCodePoint(code); } catch (e) { return fallback; }
+}
+const squash = (s) => decode(s).replace(/\s+/g, " ");
+const tagOf = (node) => (node.rawTagName || "").toLowerCase();
+
+function touchingLinks(node) {
+  const prev = node.previousSibling;
+  return !!prev && prev.nodeType === 1 && tagOf(node) === "a" && tagOf(prev) === "a";
+}
+
+// Inline text of a node: links and emphasis become their text, <code> keeps backticks.
+function inline(node) {
+  if (node.nodeType === 3) return squash(node.rawText);
+  if (node.nodeType !== 1) return "";
+  const tag = tagOf(node);
+  if (tag === "br") return "\n";
+  if (tag === "img") return "";
+  // Block children (a heading inside a list item) and back-to-back links get a space, so words do not run together.
+  let inner = "";
+  for (const c of node.childNodes) {
+    const t = c.nodeType === 1 ? tagOf(c) : "";
+    const spaced = t && (BLOCK.has(t) || touchingLinks(c));
+    inner += (spaced ? " " : "") + inline(c) + (t && BLOCK.has(t) && t !== "br" ? " " : "");
+  }
+  if (tag === "code" && inner.trim()) return "`" + inner.trim() + "`";
+  return inner;
+}
+
+function codeBlock(node) {
+  const code = node.querySelector("code") || node;
+  const cls = (code.getAttribute("class") || "") + " " + (node.getAttribute("class") || "") + " " + (node.getAttribute("data-language") || node.getAttribute("data-lang") || "");
+  const lang = (/(?:language|lang|highlight-source)-([\w+#.-]+)/.exec(cls) || [])[1] || (node.getAttribute("data-language") || node.getAttribute("data-lang") || "");
+  // Line-per-element highlighters (Shiki, Expressive Code) put each line in its own block element.
+  const lines = code.querySelectorAll(".line, .ec-line");
+  const text = lines.length > 1 ? lines.map((l) => decode(l.text)).join("\n") : decode(code.text);
+  const body = text.replace(/\s+$/, "").replace(/^\n+/, "");
+  if (!body.trim()) return "";
+  const fence = body.includes("```") ? "~~~~" : "```";
+  return fence + lang + "\n" + body + "\n" + fence;
+}
+
+function table(node) {
+  const rows = node.querySelectorAll("tr").map((tr) => tr.querySelectorAll("th,td").map((c) => inline(c).replace(/\s+/g, " ").replace(/\|/g, "\\|").trim()));
+  const filled = rows.filter((r) => r.length);
+  if (!filled.length) return "";
+  const out = ["| " + filled[0].join(" | ") + " |", "| " + filled[0].map(() => "---").join(" | ") + " |"];
+  for (const r of filled.slice(1)) out.push("| " + r.join(" | ") + " |");
+  return out.join("\n");
+}
+
+function list(node, depth) {
+  const ordered = tagOf(node) === "ol";
+  const out = [];
+  let n = 0;
+  for (const li of node.childNodes) {
+    if (li.nodeType !== 1 || tagOf(li) !== "li") continue;
+    n++;
+    const own = [], nested = [];
+    for (const c of li.childNodes) {
+      const t = c.nodeType === 1 ? tagOf(c) : "";
+      if (t === "ul" || t === "ol") nested.push(list(c, depth + 1));
+      else if (t === "pre") nested.push(codeBlock(c).replace(/^/gm, "  ".repeat(depth + 1)));
+      else own.push(t && BLOCK.has(t) ? " " + inline(c) + " " : (t && touchingLinks(c) ? " " : "") + inline(c));
+    }
+    const text = own.join("").replace(/\s+/g, " ").trim();
+    if (text) out.push("  ".repeat(depth) + (ordered ? n + ". " : "- ") + text);
+    for (const x of nested) if (x) out.push(x);
+  }
+  return out.join("\n");
+}
+
+// Markdown blocks of a content root. `shift` moves page headings down so a page's own
+// sections sit under its "# Title" in llms-full.txt (h2 → ##, h3 → ###, …).
+function blocks(root, ctx) {
+  const out = [];
+  let run = [];
+  const flush = () => {
+    const text = run.join("").replace(/[ \t]*\n[ \t]*/g, "\n").replace(/[ \t]+/g, " ").trim();
+    if (text) out.push(text);
+    run = [];
+  };
+  const walk = (node) => {
+    if (node.nodeType === 3) { run.push(squash(node.rawText)); return; }
+    if (node.nodeType !== 1) return;
+    const tag = tagOf(node);
+    if (!BLOCK.has(tag)) {
+      if (node.childNodes.some((c) => c.nodeType === 1 && BLOCK.has(tagOf(c)))) { node.childNodes.forEach(walk); return; }
+      run.push((touchingLinks(node) ? " " : "") + inline(node));
+      return;
+    }
+    if (tag === "br") { run.push("\n"); return; }
+    flush();
+    if (/^h[1-6]$/.test(tag)) {
+      const text = inline(node).replace(/\s+/g, " ").replace(/^#+\s*|\s*[#¶🔗]+$/g, "").trim();
+      if (!text) return;
+      const level = Number(tag[1]);
+      if (level === 1 && !ctx.h1) { ctx.h1 = text; return; }   // the page title, written once by the caller
+      const id = node.getAttribute("id") || (node.querySelector("[id]") || { getAttribute: () => "" }).getAttribute("id") || "";
+      out.push({ heading: Math.max(2, level), text, id });
+      return;
+    }
+    if (tag === "pre") { const b = codeBlock(node); if (b) out.push(b); return; }
+    if (tag === "ul" || tag === "ol") { const b = list(node, 0); if (b) out.push(b); return; }
+    if (tag === "table") { const b = table(node); if (b) out.push(b); return; }
+    if (tag === "hr") return;
+    if (tag === "blockquote") {
+      const inner = blocks(node, ctx).filter((b) => typeof b === "string").join("\n\n");
+      if (inner) out.push(inner.replace(/^/gm, "> "));
+      return;
+    }
+    node.childNodes.forEach(walk);
+    flush();
+  };
+  root.childNodes.forEach(walk);
+  flush();
+  return out;
+}
+
+function meta(doc, selector) {
+  const el = doc.querySelector(selector);
+  return el ? decode(el.getAttribute("content") || "").replace(/\s+/g, " ").trim() : "";
+}
+
+// One HTML document → { title, description, canonical, noindex, redirect, sections, text }.
+// sections: [{ heading, id, text }] where the first one (heading "") is the text before any h2.
+function readHtml(html, options) {
+  const o = options || {};
+  const doc = parse(String(html), { comment: false, blockTextElements: { script: false, style: false, noscript: false } });
+  const robots = meta(doc, 'meta[name="robots"]') + " " + meta(doc, 'meta[name="googlebot"]');
+  const refresh = doc.querySelector('meta[http-equiv="refresh"], meta[http-equiv="Refresh"]');
+  const canonicalEl = doc.querySelector('link[rel="canonical"]');
+  const titleEl = doc.querySelector("title");
+  const docTitle = titleEl ? decode(titleEl.text).replace(/\s+/g, " ").trim() : "";
+  const info = {
+    title: "", docTitle,
+    siteName: meta(doc, 'meta[property="og:site_name"]'),
+    description: meta(doc, 'meta[name="description"]') || meta(doc, 'meta[property="og:description"]'),
+    canonical: (canonicalEl && canonicalEl.getAttribute("href")) || meta(doc, 'meta[property="og:url"]') || "",
+    noindex: /noindex/i.test(robots),
+    redirect: !!refresh,
+    lang: (doc.querySelector("html") && doc.querySelector("html").getAttribute("lang")) || "",
+    sections: [], text: "",
+  };
+  const selectors = [].concat(o.content || []).filter(Boolean).concat(ROOTS);
+  let root = null;
+  for (const s of selectors) { try { root = doc.querySelector(s); } catch (e) { root = null; } if (root) break; }
+  if (!root) return info;
+  for (const s of [DROP].concat(o.exclude || [])) {
+    let found = [];
+    try { found = root.querySelectorAll(s); } catch (e) { /* a selector the parser cannot read is skipped */ }
+    for (const el of found) el.remove();
+  }
+  const ctx = { h1: "" };
+  const parts = blocks(root, ctx);
+  let cur = { heading: "", level: 0, id: "", lines: [] };
+  const sections = [cur];
+  for (const b of parts) {
+    if (typeof b === "string") { cur.lines.push(b); continue; }
+    if (b.heading === 2) { cur = { heading: b.text, level: 2, id: b.id, lines: [] }; sections.push(cur); }
+    else cur.lines.push("#".repeat(b.heading) + " " + b.text);
+  }
+  info.sections = sections.map((s) => ({ heading: s.heading, id: s.id, text: s.lines.join("\n\n").trim() })).filter((s) => s.heading || s.text);
+  info.text = info.sections.map((s) => (s.heading ? "## " + s.heading + "\n\n" : "") + s.text).join("\n\n").trim();
+  info.title = ctx.h1 || meta(doc, 'meta[property="og:title"]') || docTitle;
+  return info;
+}
+
+// "a/**/b*.html" → RegExp, matched against paths relative to the site folder with "/" separators.
+function globToRegExp(glob) {
+  const g = String(glob).replace(/^\.?\//, "");
+  let re = "";
+  for (let i = 0; i < g.length; i++) {
+    const c = g[i];
+    if (c === "*") {
+      if (g[i + 1] === "*") { re += ".*"; i++; if (g[i + 1] === "/") i++; }
+      else re += "[^/]*";
+    } else if (c === "?") re += "[^/]";
+    else re += c.replace(/[.+^${}()|[\]\\]/g, "\\$&");
+  }
+  return new RegExp("^" + re + "$");
+}
+
+function walkHtml(dir) {
+  const out = [];
+  const visit = (abs, rel) => {
+    for (const entry of fs.readdirSync(abs, { withFileTypes: true })) {
+      if (entry.name.startsWith(".") || entry.name === "node_modules") continue;
+      const childAbs = path.join(abs, entry.name), childRel = rel ? rel + "/" + entry.name : entry.name;
+      if (entry.isDirectory()) visit(childAbs, childRel);
+      else if (/\.html?$/i.test(entry.name)) out.push(childRel);
+    }
+  };
+  visit(dir, "");
+  return out;
+}
+
+function urlPath(rel, cleanUrls) {
+  let p = "/" + rel;
+  if (/(^|\/)index\.html?$/i.test(rel)) p = p.replace(/index\.html?$/i, "");
+  else if (cleanUrls) p = p.replace(/\.html?$/i, "");
+  return p.split("/").map((seg) => encodeURIComponent(seg).replace(/%2B/g, "+")).join("/");
+}
+
+function joinUrl(site, p) {
+  if (!site) return "";
+  return String(site).replace(/\/+$/, "") + (p.startsWith("/") ? p : "/" + p);
+}
+
+// Home page first, then by depth, then by path: the order pages appear in both files.
+function byPath(a, b) {
+  const da = a.path.split("/").filter(Boolean).length, db = b.path.split("/").filter(Boolean).length;
+  return da - db || (a.path < b.path ? -1 : a.path > b.path ? 1 : 0);
+}
+
+// Read every page of a built site. Returns [{ file, path, url, title, description, sections, text }].
+function scan(dir, options) {
+  const o = options || {};
+  const root = path.resolve(dir);
+  if (!fs.existsSync(root) || !fs.statSync(root).isDirectory()) throw new Error("static-llms-txt: " + root + " is not a folder. Pass the folder your site was built into.");
+  const excludes = [].concat(o.exclude || []).map(globToRegExp);
+  const pages = [];
+  for (const rel of walkHtml(root)) {
+    if (/(^|\/)(404|500)(\.html?|\/index\.html?)$/i.test(rel)) continue;
+    if (excludes.some((re) => re.test(rel))) continue;
+    const info = readHtml(fs.readFileSync(path.join(root, rel), "utf8"), { content: o.content, exclude: o.remove });
+    if (info.noindex || info.redirect) continue;
+    if (info.text.length < (o.minLength === undefined ? 40 : o.minLength)) continue;
+    const p = urlPath(rel, o.cleanUrls);
+    const url = o.site ? joinUrl(o.site, p) : /^https?:\/\//i.test(info.canonical) ? info.canonical : "";
+    pages.push({ file: rel, path: p, url, title: info.title || p, docTitle: info.docTitle, siteName: info.siteName, description: info.description, sections: info.sections, text: info.text });
+  }
+  return pages.sort(byPath);
+}
+
+const oneLine = (s) => String(s || "").replace(/\s+/g, " ").trim();
+
+// pages: [{ url, path?, title, description?, text? | sections? | html? }] → { llms, llmsFull }.
+function build(pages, options) {
+  const o = options || {};
+  const list = pages.map((p) => {
+    if (p.html && !p.text && !p.sections) {
+      const info = readHtml(p.html, { content: o.content, exclude: o.remove });
+      return Object.assign({}, p, { title: p.title || info.title, description: p.description || info.description, sections: info.sections, text: info.text });
+    }
+    return p;
+  }).filter((p) => p && (p.text || (p.sections && p.sections.length)));
+  const home = list.find((p) => (p.path || "") === "/") || list[0] || {};
+  const title = oneLine(o.title || home.siteName || (home.docTitle || home.title || "").split(/\s+[|·–—-]\s+/).pop() || "Site");
+  const description = oneLine(o.description || home.description || "");
+  const link = (p) => p.url || (o.site ? joinUrl(o.site, p.path || "/") : p.path || "");
+
+  const index = ["# " + title, ""];
+  if (description) index.push("> " + description, "");
+  index.push("## " + (o.section || "Pages"), "");
+  for (const p of list) {
+    const d = oneLine(p.description);
+    index.push("- [" + oneLine(p.title).replace(/[[\]]/g, "") + "](" + link(p) + ")" + (d ? ": " + d : ""));
+  }
+  if (o.full !== false) index.push("", "## Optional", "", "- [Full text of every page](" + (o.site ? joinUrl(o.site, "/llms-full.txt") : "/llms-full.txt") + ")");
+  index.push("", MARK, "");
+
+  const full = [];
+  for (const p of list) {
+    const url = link(p);
+    const abs = /^https?:\/\//i.test(url);
+    full.push("# " + oneLine(p.title), "");
+    if (abs) full.push("Source: " + url, "");
+    const sections = p.sections || [{ heading: "", id: "", text: p.text }];
+    for (const s of sections) {
+      if (s.heading) {
+        full.push("## " + oneLine(s.heading), "");
+        if (abs && s.id) full.push("Source: " + url.replace(/#.*$/, "") + "#" + encodeURIComponent(s.id), "");
+      }
+      if (s.text) full.push(s.text, "");
+    }
+  }
+  full.push(MARK, "");
+  return { title, description, pages: list, llms: index.join("\n"), llmsFull: full.join("\n").replace(/\n{3,}/g, "\n\n") };
+}
+
+// A file we did not write (another plugin's llms.txt, or one written by hand) is left alone.
+function writeOwn(file, text, force) {
+  if (!force && fs.existsSync(file) && !fs.readFileSync(file, "utf8").includes(MARK)) return false;
+  fs.mkdirSync(path.dirname(file), { recursive: true });
+  fs.writeFileSync(file, text);
+  return true;
+}
+
+// Scan a built site and write llms.txt and llms-full.txt into it.
+function generate(dir, options) {
+  const o = options || {};
+  const pages = o.pages || scan(dir, o);
+  const result = build(pages, o);
+  const out = path.resolve(o.out || dir);
+  const written = [], skipped = [];
+  const put = (name, text) => (o.write === false ? null : (writeOwn(path.join(out, name), text, o.force) ? written : skipped).push(name));
+  put("llms.txt", result.llms);
+  if (o.full !== false) put("llms-full.txt", result.llmsFull);
+  return Object.assign(result, { written, skipped });
+}
+
+const attr = (v) => String(v).replace(/&/g, "&amp;").replace(/"/g, "&quot;").replace(/</g, "&lt;");
+
+// The Ask AI script tag. Options become data-* attributes; `ref` names the plugin that added it.
+function scriptTag(options) {
+  const o = options || {};
+  const names = ["key", "ref", "label", "placeholder", "position", "button", "color", "theme", "lang", "rules", "index"];
+  let tag = '<script async src="' + attr(o.src || SCRIPT) + '"';
+  for (const n of names) if (o[n] !== undefined && o[n] !== null && o[n] !== "") tag += " data-" + n + '="' + attr(o[n]) + '"';
+  return tag + "></script>";
+}
+
+// Add the Ask AI script tag before </body> of every page in a built site. Running it twice changes nothing.
+function inject(dir, options) {
+  const root = path.resolve(dir);
+  const o = options || {};
+  const excludes = [].concat(o.exclude || []).map(globToRegExp);
+  const tag = scriptTag(o);
+  const changed = [];
+  for (const rel of walkHtml(root)) {
+    if (excludes.some((re) => re.test(rel))) continue;
+    const file = path.join(root, rel);
+    const html = fs.readFileSync(file, "utf8");
+    if (/<script[^>]+\/ask-docs\.js/i.test(html)) continue;
+    const at = html.search(/<\/body\s*>(?![\s\S]*<\/body\s*>)/i);
+    if (at === -1) continue;
+    fs.writeFileSync(file, html.slice(0, at) + tag + "\n" + html.slice(at));
+    changed.push(rel);
+  }
+  return changed;
+}
+
+module.exports = { generate, scan, build, readHtml, inject, scriptTag, globToRegExp, MARK, SCRIPT };

package/index.d.ts

@@ -0,0 +1,129 @@
+export interface Section {
+  /** Text of the h2 that opens the section. Empty for the text before the first h2. */
+  heading: string;
+  /** The heading's id attribute, used for #anchor links. */
+  id: string;
+  /** Markdown text of the section. */
+  text: string;
+}
+
+export interface PageInfo {
+  title: string;
+  docTitle: string;
+  siteName: string;
+  description: string;
+  canonical: string;
+  noindex: boolean;
+  redirect: boolean;
+  lang: string;
+  sections: Section[];
+  text: string;
+}
+
+export interface Page {
+  /** Absolute address of the page. */
+  url?: string;
+  /** Path from the site root, e.g. "/guide/". */
+  path?: string;
+  /** File path relative to the scanned folder (set by scan). */
+  file?: string;
+  title: string;
+  description?: string;
+  /** Markdown text. Give text, sections or html. */
+  text?: string;
+  sections?: Section[];
+  /** Raw HTML of the page; build() reads it with readHtml(). */
+  html?: string;
+}
+
+export interface ReadOptions {
+  /** CSS selector (or list) of the main content. Default: main, [role=main], article, #content, .content, body. */
+  content?: string | string[];
+  /** Extra CSS selectors to drop from the content. */
+  exclude?: string | string[];
+}
+
+export interface Options {
+  /** Public address of the site, e.g. "https://example.com". Makes every link absolute. */
+  site?: string;
+  /** Site title. Default: taken from the home page. */
+  title?: string;
+  /** One-line summary under the title. Default: the home page meta description. */
+  description?: string;
+  /** Heading of the page list in llms.txt. Default "Pages". */
+  section?: string;
+  /** CSS selector (or list) of the main content. */
+  content?: string | string[];
+  /** Extra CSS selectors to drop from every page. */
+  remove?: string | string[];
+  /** Globs of files to skip, relative to the folder, e.g. "tags/**". */
+  exclude?: string | string[];
+  /** Link /about instead of /about.html. */
+  cleanUrls?: boolean;
+  /** Pages with less text than this are skipped. Default 40 characters. */
+  minLength?: number;
+  /** false writes llms.txt only. */
+  full?: boolean;
+  /** Overwrite an llms.txt that this tool did not write. */
+  force?: boolean;
+  /** Write to this folder instead of the scanned one. */
+  out?: string;
+  /** false returns the text without writing files. */
+  write?: boolean;
+  /** Use these pages instead of scanning the folder. */
+  pages?: Page[];
+}
+
+export interface Result {
+  title: string;
+  description: string;
+  pages: Page[];
+  /** Text of llms.txt. */
+  llms: string;
+  /** Text of llms-full.txt. */
+  llmsFull: string;
+}
+
+export interface GenerateResult extends Result {
+  written: string[];
+  skipped: string[];
+}
+
+export interface ScriptOptions {
+  /** OneTag AI site key. Public by design. */
+  key?: string;
+  /** Short name of the tool that added the tag. */
+  ref?: string;
+  label?: string;
+  placeholder?: string;
+  position?: "auto" | "float" | "left" | "none";
+  /** CSS selector of your own button that opens the box. */
+  button?: string;
+  /** Accent color as hex without "#". */
+  color?: string;
+  theme?: "light" | "dark";
+  lang?: string;
+  rules?: string;
+  index?: string;
+  src?: string;
+  /** Globs of files to leave alone (inject only). */
+  exclude?: string | string[];
+}
+
+/** Scan a built site and write llms.txt and llms-full.txt into it. */
+export function generate(dir: string, options?: Options): GenerateResult;
+/** Read every page of a built site. */
+export function scan(dir: string, options?: Options): Page[];
+/** Turn a list of pages into the text of both files. */
+export function build(pages: Page[], options?: Options): Result;
+/** Read one HTML document: title, description and main content as Markdown. */
+export function readHtml(html: string, options?: ReadOptions): PageInfo;
+/** Add the Ask AI script tag before </body> of every page. Returns the files changed. */
+export function inject(dir: string, options?: ScriptOptions): string[];
+/** The Ask AI script tag as a string. */
+export function scriptTag(options?: ScriptOptions): string;
+export function globToRegExp(glob: string): RegExp;
+export const MARK: string;
+export const SCRIPT: string;
+declare const core: { generate: typeof generate; scan: typeof scan; build: typeof build; readHtml: typeof readHtml; inject: typeof inject; scriptTag: typeof scriptTag; globToRegExp: typeof globToRegExp; MARK: string; SCRIPT: string };
+export default core;

package/index.mjs

@@ -0,0 +1,3 @@
+import core from "./index.cjs";
+export const { generate, scan, build, readHtml, inject, scriptTag, globToRegExp, MARK, SCRIPT } = core;
+export default core;

package/package.json

@@ -1,6 +1,39 @@
 {
   "name": "static-llms-txt",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "0.1.0",
+  "description": "Write llms.txt and llms-full.txt for any static site from the HTML it already built. Works after Eleventy, Hugo, Jekyll, Astro, Gatsby, Docusaurus or plain HTML. No API key.",
+  "main": "./index.cjs",
+  "module": "./index.mjs",
+  "types": "./index.d.ts",
+  "exports": {
+    ".": {
+      "types": "./index.d.ts",
+      "import": "./index.mjs",
+      "require": "./index.cjs"
+    }
+  },
+  "bin": {
+    "static-llms-txt": "./cli.cjs"
+  },
+  "files": ["index.cjs", "index.mjs", "index.d.ts", "cli.cjs", "README.md", "LICENSE"],
+  "scripts": {
+    "test": "node --test test/*.test.mjs",
+    "prepublishOnly": "npm test"
+  },
+  "keywords": ["llms.txt", "llms-txt", "llms-full.txt", "llmstxt", "static-site", "html-to-markdown", "ssg", "eleventy", "hugo", "jekyll", "ai", "seo", "geo", "cli"],
+  "license": "MIT",
+  "author": "OneTag AI (https://onetagai.com)",
+  "homepage": "https://github.com/fengyiqicoder/onetagai/tree/main/packages/static-llms-txt#readme",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/fengyiqicoder/onetagai.git",
+    "directory": "packages/static-llms-txt"
+  },
+  "bugs": "https://github.com/fengyiqicoder/onetagai/issues",
+  "engines": {
+    "node": ">=18"
+  },
+  "dependencies": {
+    "node-html-parser": "^7.0.1"
+  }
+}