@zvndev/darkmown 2.3.1-next.4.g92bbead → 2.3.1-next.5.g6f7d03c

package/README.md

@@ -162,11 +162,78 @@ site_url: https://example.com
 ---
 ```
 
-- **`site_url`** (absolute origin, **no trailing slash**) turns on `sitemap.xml` and `rss.xml` and is the absolute prefix for every URL in them. The home `title`/`description` become the RSS channel title/description.
-- **`robots.txt`** is *always* emitted (`User-agent: * / Allow: /`); the `Sitemap:` line is added only when `site_url` is set.
+- **`site_url`** (absolute origin, **no trailing slash**) turns on `sitemap.xml`, `rss.xml`, and every page's **canonical URL**, and is the absolute prefix for every URL in them. The home `title`/`description` become the RSS channel title/description.
+- **`robots.txt`** is *always* emitted; the `Sitemap:` line is added only when `site_url` is set. See [AI crawlers](#ai-crawlers) below for the explicit per-crawler groups.
 - **`sitemap.xml`** lists every built page (reactive pages included — they're indexable HTML). Each `<lastmod>` is the page's frontmatter `date:` if set, else its git last-commit date, else the file's mtime. No `<priority>`/`<changefreq>`.
 - **`rss.xml`** syndicates your **posts** — any page with a `date:` in its frontmatter. Newest first, capped at the 20 most recent. Each item's `<description>` is the page's `excerpt:`, else its `description:`, else (for a plain `.md` post) its first paragraph. Every page links the feed with `<link rel="alternate" type="application/rss+xml">` so readers can autodiscover it.
 
+### Canonical URLs
+
+With `site_url` set, every page's `<head>` states its own absolute URL twice, as `<link rel="canonical">` and `og:url`:
+
+```html
+<link rel="canonical" href="https://example.com/docs/">
+<meta property="og:url" content="https://example.com/docs/">
+```
+
+Both are built from the **same route string** the sitemap and your internal links use, so the three can never disagree about which URL form is canonical. Darkmown routes are **trailing-slashed** (`/docs/`, matching the `dist/docs/index.html` the build writes), and `darkmown deploy vercel` writes `trailingSlash: true` so the host serves that same form directly. If you host it yourself, configure the host to match: a host that redirects `/docs/` to `/docs` puts a redirect hop on every internal navigation and makes your own sitemap advertise URLs that redirect.
+
+A paginated listing canonicalises **each page to itself** (`/blog/page/2/` points at `/blog/page/2/`), never back to page one, because pages 2..N hold different content and pointing them at page one asks a crawler to drop them.
+
+`og:type` is `article` for a page with a `date:` (or an article `schema:`), and `website` otherwise. There is no `twitter:title`/`twitter:description`: X's card parser falls back to the Open Graph tags, so they would be duplicate bytes on every page. `twitter:card` has no Open Graph equivalent, so it *is* stated.
+
+### Structured data (`schema:`)
+
+Opt a page into JSON-LD with one frontmatter key. This is ordinary indexing hygiene for Google's conventional **rich results**. Google's own generative-search guidance is explicit that "there's no special schema.org markup you need to add" and that structured data "isn't required for generative AI search", so this is not an AI-citation lever and is not sold as one.
+
+```md
+---
+title: Zero JavaScript, by default
+description: A Darkmown page ships no framework JavaScript unless it declares reactive behavior.
+date: 2026-06-25
+author: Ada Lovelace
+schema: BlogPosting
+---
+```
+
+```html
+<script type="application/ld+json">{"@context":"https://schema.org","@type":"BlogPosting",
+"headline":"Zero JavaScript, by default","datePublished":"2026-06-25",
+"author":{"@type":"Person","name":"Ada Lovelace"},"url":"https://example.com/blog/zero-js/"}</script>
+```
+
+- **Types** (a compile-time whitelist, like every other Darkmown vocabulary): `Article`, `BlogPosting`, `TechArticle`, `WebSite`, `Organization`. Anything else is a compile error naming every valid type. Pass a list for a page that is honestly both: `schema: [WebSite, Organization]`.
+- **Supporting keys:** `author:` (a name or a list) and `updated:` for articles; `organization:` and `logo:` for `Organization`. Everything else is reused from `title`, `description`, `image`, `date`, `lang`, and the canonical URL.
+- **Only what the page has.** A key you did not write produces no property at all: never a blank one and never a guess. There is deliberately **no** way to emit `aggregateRating`, `review`, or `offers`: fabricated ratings are a manual-action risk, and Darkmown cannot know them. `FAQPage` is likewise not supported, because it would mean inferring a Q&A structure out of prose.
+- **`BreadcrumbList` is automatic** for nested routes (two or more path segments) once `site_url` is set. The trail is built from routes that **actually exist**, with their real titles, so a crumb never links a page you never wrote: `/vs/markdoc/` on a site with no `/vs/` page yields `Home > vs Markdoc`.
+- Zero runtime cost. JSON-LD is an inert data block, so a static page stays static (`runtime: false`) and ships no JavaScript. It is verified in a real browser to survive Darkmown's strict, `unsafe-inline`-free `script-src`.
+
+### AI crawlers
+
+`robots.txt` names every major AI crawler and answer engine explicitly, grouped by operator, with a link to the documentation each token was verified against:
+
+```
+# OpenAI: https://developers.openai.com/api/docs/bots
+# OAI-SearchBot = ChatGPT Search crawling and citations. GPTBot = potential model training. …
+User-agent: OAI-SearchBot
+User-agent: GPTBot
+User-agent: ChatGPT-User
+Allow: /
+```
+
+Covered: **OpenAI** (`OAI-SearchBot`, `GPTBot`, `ChatGPT-User`), **Anthropic** (`Claude-SearchBot`, `ClaudeBot`, `Claude-User`), **Google** (`Google-Extended`), **Apple** (`Applebot-Extended`), **Perplexity** (`PerplexityBot`, `Perplexity-User`), **Meta** (`meta-externalagent`), **Mistral** (`MistralAI-Index`, `MistralAI-Training`, `MistralAI-User`), **Amazon** (`Amazonbot`), and **Common Crawl** (`CCBot`). Search crawling, model training, and live user-triggered fetches are *different* permissions from the same operator, so each group carries a comment saying which is which.
+
+Flip the whole set to `Disallow: /` from the home page:
+
+```md
+---
+site_url: https://example.com
+ai_crawlers: deny
+---
+```
+
+`allow` is the default (and matches what `User-agent: *` already grants). A value that is neither `allow` nor `deny` **fails the build** rather than defaulting: the intent behind a typo like `ai_crawlers: block` is almost always to opt *out*, and silently allowing would be the one unrecoverable outcome.
+
 ```md
 ---
 title: Hello, Darkmown
@@ -1039,12 +1106,14 @@ Genuine *internal* invariants — the expression-AST parser reading the compiler
 Darkmown ships its own description of the `.wd` language — the same tables the compiler validates against — so a tool can teach a model to write `.wd` and constrain its output to what compiles.
 
 - **Directive catalog** — `darkmown catalog` prints structured JSON: every directive, `@loop` clause, loop variable, button action, format pipe, and predicate operator, each with a syntax template, a one-line description, one concrete example, and whether it needs the reactive runtime. Importable too: `import { directiveCatalog } from "@zvndev/darkmown/catalog"`.
-- **Cheatsheet (`llms.txt`)** — `darkmown catalog --llms` prints a compact (~90-line) markdown cheatsheet generated from the same data — the artifact you stuff into a model's system prompt. Every build also writes it to `dist/llms.txt`.
+- **Cheatsheet (`llms.txt`)**: `darkmown catalog --llms` prints a compact markdown cheatsheet generated from the same data. It is the artifact you stuff into a model's system prompt. Every build also writes it to `dist/llms.txt`, followed by a one-line **index of every page on your site** (title, absolute URL, description) and a pointer to `llms-full.txt`.
+- **Corpus (`llms-full.txt`)**: `darkmown catalog --llms-full` prints the complete reference the index points at: every directive with its full syntax template and example, every clause, action, pipe and operator, every frontmatter key, and **every compile-error code with its cause and fix**. An AI edit loop that hits `[WD201]` can resolve it from the same fetch it learned the syntax from. Every build writes it to `dist/llms-full.txt`, with the **full source text of every page** appended.
 - **GBNF grammar** — [`grammar/wd-directives.gbnf`](grammar/wd-directives.gbnf) is a generated [GBNF](https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md) grammar for `.wd` directive lines. Point llama.cpp (or any grammar-constrained decoder) at it to make invalid directive lines — JS idioms like `cart.filter(...)`, HTML muscle-memory like `::: card class="…"` — literally unrepresentable during generation. Regenerate with `node scripts/gen-grammar.mjs`.
 
 ```sh
 darkmown catalog --llms > system-prompt.md   # paste into a model's context
-darkmown build                               # also emits dist/llms.txt
+darkmown catalog --llms-full                 # the complete reference + error codes
+darkmown build                               # also emits dist/llms.txt + dist/llms-full.txt
 ```
 
 Because the catalog, cheatsheet, grammar, and error `example`s are all generated from the compiler's own tables, they cannot drift from what actually compiles (enforced by tests).

package/docs/spec-alignment.md

@@ -2,6 +2,19 @@
 
 This audit compares the current Darkmown implementation with the original MARROW/WhateverDown vision and the follow-up decisions made on 2026-06-11.
 
+## Technical SEO: canonical URLs, structured data, AI crawlers, llms-full.txt (2026-08-09)
+
+The 1.3.0 cycle shipped the crawler *files* (sitemap, rss, robots). This closes the gaps a real deployment exposed: the files existed but nothing on the page said which URL was canonical, and the host disagreed with the framework about the URL form.
+
+- **Canonical URLs (`<link rel="canonical">` + `og:url`).** With `site_url` set, every page states its own absolute URL. The builder resolves it (it is the only layer that knows both the origin and the concrete route a page is written to, including a paginated `/page/2/`) and threads it into the shell as `options.site`; `src/compiler/page.js` derives both tags from that one value, using the **same route string** the sitemap builds from. Canonical, `og:url`, sitemap `<loc>`, and the internal links therefore cannot drift apart: regression-tested by walking every built route and asserting all three agree. A **paginated page canonicalises to itself**, never to page one (pointing pages 2..N at page one asks a crawler to drop them). With no `site_url` there is no canonical at all: the framework never invents an origin.
+- **BUGFIX: the deploy config disagreed with the framework's own URL form.** Darkmown routes are trailing-slashed everywhere (`routeFromFile` → `/docs/`, `routes.json`, `_headers`, the sitemap, the dev server, and the `dist/docs/index.html` the build writes), and every generated link uses that form. `vercel.json` and the `darkmown deploy vercel` default set `trailingSlash: false`, so the host 308-redirected *every* internal navigation and the sitemap advertised URLs that redirect. Both now set `trailingSlash: true`, which is the one-line fix: the alternative (dropping the slash) would have meant changing the route model that `routes.json`, the headers files, and the output layout are all built on. Verified in a browser: every internal link on the home page resolves 200 with `maxRedirects: 0`.
+- **BUGFIX: `www` and the apex both served 200.** `https://www.darkmown.com/` returned content rather than redirecting, so the site was reachable under two hostnames with no canonicalisation signal. `vercel.json` now carries a host-conditioned 308 to the apex, which is also what the new canonical tags name.
+- **Structured data (`schema:`), deliberately narrow.** An opt-in frontmatter key emits a JSON-LD block: `Article`/`BlogPosting`/`TechArticle`, `WebSite`, `Organization`, plus an **automatic `BreadcrumbList`** for nested routes. Frontmatter, not a directive, because JSON-LD describes the *document* (as `title`/`description`/`image`/`lang` already do), which also keeps it available to plain `.md` without breaking the "`.md` never gets directive behavior" invariant, and lets the `html: false` home page describe itself with zero raw HTML. The type list is a compile-time whitelist like every other Darkmown vocabulary (`WD016` names every valid type). **Scope is a correctness decision, not an omission:** Google's generative-search guidance states there is no special AI schema and that structured data is not required for generative results, so this is positioned as ordinary rich-result hygiene rather than an AI-citation lever. There is deliberately **no path to `aggregateRating`/`review`/`offers`** (fabricated social proof is a manual-action risk) and **no `FAQPage`** (it would mean inferring a Q&A structure out of prose). A key the author did not write produces no property, never a blank or a guess. Breadcrumb crumbs are built only from routes that **actually exist**, with their real titles, so a crumb can never link a page that was never written. Zero runtime cost, and verified in Chromium that the inert data block survives the strict, `unsafe-inline`-free `script-src` (worth checking, not assuming: inline `speculationrules` needed an explicit CSP token).
+- **Explicit AI-crawler groups in `robots.txt`.** Every token is copied from its own operator's published documentation, with the docs URL written into the generated file next to the group; anything unverifiable was left out (`cohere-ai` among them). Search crawling, model training, and live user-triggered fetches are *different* permissions from the same operator, so each group carries a comment saying which is which (`OAI-SearchBot` = ChatGPT Search citations vs `GPTBot` = training is the case most often collapsed by hand-editing). `ai_crawlers: deny` on the home page flips the whole set to `Disallow: /`; an unrecognised value **fails the build** (`WD907`) rather than defaulting, because the intent behind a typo is almost always to opt out and failing open would be the one unrecoverable outcome.
+- **`llms-full.txt`, and `llms.txt` becomes a real index.** Following the convention: `llms.txt` keeps the cheatsheet and gains a one-line-per-page index of the site (title, absolute URL, description) plus a pointer to the corpus; `llms-full.txt` carries the complete authoring reference (full syntax template, description and example per directive, every clause/action/pipe/operator, every frontmatter key, and **every compile-error code with its cause and fix**) followed by the full source text of every page. Both are rendered by `src/catalog.js` from the same `directiveCatalog()` and the same route list, so the index cannot disagree with the corpus. `darkmown catalog --llms-full` prints the framework half.
+- **Frontmatter is now catalogued.** `directiveCatalog().frontmatterKeys` documents every key the framework actually reads, drift-guarded against the source that consumes it, so an AI author cannot be taught frontmatter that does nothing.
+- **Zero runtime impact.** All build-time: `src/runtime.js` is untouched and the 8 KB budget is unchanged. Static pages stay `runtime: false` (asserted per route in `tests/seo-meta.test.js`), and a page carrying JSON-LD ships no JavaScript at all.
+
 ## 2.3.0 additions — error codes, supply chain, platform coverage (2026-08-09)
 
 - **Every author-facing compile error carries a stable `WDxxx` code.** The original vision treated compile errors as a first-class DX surface ("errors include file path + corrective suggestion"); this closes the last gap by making them *identifiable*. 138 codes, grouped by subsystem, mirrored on `err.wd.code` beside the existing `file`/`line`/`hint`/`example`. `src/errors.js` is the single source of truth, `docs/errors.md` is generated from it, and `directiveCatalog()` exposes it as data for tooling. Drift guards run both directions: an author-facing throw without a code, a duplicate code, or an undocumented code all fail the build. Internal invariants (the AST parser reading the compiler's own validated output, the reader contract) stay plain `Error`s by design — they signal a framework bug, not an authoring mistake — and that split is itself enforced by test.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@zvndev/darkmown",
-  "version": "2.3.1-next.4.g92bbead",
+  "version": "2.3.1-next.5.g6f7d03c",
   "private": false,
   "publishConfig": {
     "access": "public"

package/site/pages/app.wd

@@ -1,7 +1,7 @@
 ---
 html: true
 title: A live app in one Markdown file
-description: One .wd file — static prose, fetched data, looped results with nested conditionals, a persistent counter, and a form that round-trips to a real server. No JavaScript written, no components, no build config.
+description: One .wd file: static prose, fetched data, looped results with nested conditionals, a persistent counter, and a form that round-trips to a real server. No JavaScript written, no components, no build config.
 ---
 
 @include /nav.wd
@@ -10,21 +10,21 @@ description: One .wd file — static prose, fetched data, looped results with ne
 
 # A live app, in one Markdown file
 
-Everything on this page — the prose, the fetched list, the badges, the form, the counter — is a single `.wd` file. No components. No JSX. No build config. Watch where JavaScript actually enters.
+Everything on this page (the prose, the fetched list, the badges, the form, the counter) is a single `.wd` file. No components. No JSX. No build config. Watch where JavaScript actually enters.
 
 ## Static until proven reactive
 
-This whole section is plain Markdown. The list below is unrolled **at build time** from a JSON file — it renders into HTML with no runtime at all:
+This whole section is plain Markdown. The list below is unrolled **at build time** from a JSON file, and it renders into HTML with no runtime at all:
 
 @loop /features.json into feature
-- **{ feature.title }** — { feature.body }
+- **{ feature.title }**: { feature.body }
 @endloop
 
-<aside class="boundary">If you deleted everything below this line, this page would ship <strong>zero</strong> bytes of JavaScript. Everything above is static HTML. Below is where the one ~7.7&nbsp;KB gzipped runtime loads (CI-enforced under 8 KB) — and only because the document asks for it.</aside>
+<aside class="boundary">If you deleted everything below this line, this page would ship <strong>zero</strong> bytes of JavaScript. Everything above is static HTML. Below is where the one ~7.7&nbsp;KB gzipped runtime loads (CI-enforced under 8 KB), and only because the document asks for it.</aside>
 
 ## Fetched data, looped, with nested conditions
 
-`:fetch` pulls JSON over the network into reactive state — from a static file here, or any API. Until it lands, the page shows a fallback. Each row then decides its own badge from **nested** conditionals, in plain Markdown:
+`:fetch` pulls JSON over the network into reactive state, from a static file here or any API. Until it lands, the page shows a fallback. Each row then decides its own badge from **nested** conditionals, in plain Markdown:
 
 :fetch roadmap from "/__wd/data/roadmap.json"
 
@@ -47,7 +47,7 @@ This whole section is plain Markdown. The list below is unrolled **at build time
 Loading the roadmap…
 :endif
 
-No ternaries, no `.map()`, no framework — the outer `:if` checks `blocked`, the inner one checks `shipped`, per row, and both stay reactive.
+No ternaries, no `.map()`, no framework. The outer `:if` checks `blocked`, the inner one checks `shipped`, per row, and both stay reactive.
 
 ## A counter that survives reload
 
@@ -63,15 +63,15 @@ You are watching { stars } item(s).
 :button "Reset" -> stars = 0
 
 :if milestone
-Nice — you are tracking the whole near-term roadmap.
+Nice, you are tracking the whole near-term roadmap.
 :endif
 :::
 
 ## A form that talks to a real server
 
-This form posts to the demo echo endpoint and shows the reply. With JavaScript it is a `fetch`; without it, the exact same markup is a native POST. On Vercel/dev the sample endpoint replies; on static-only hosts, point `action=` at your own backend — Darkmown adapts instead of owning one:
+This form posts to the demo echo endpoint and shows the reply. With JavaScript it is a `fetch`; without it, the exact same markup is a native POST. On Vercel/dev the sample endpoint replies; on static-only hosts, point `action=` at your own backend. Darkmown adapts instead of owning one:
 
-:form action="/api/echo" into reply
+:form action="/api/echo/" into reply
 :input note placeholder="Tell us what you'd build with this"
 :submit "Send to the server"
 :endform
@@ -88,6 +88,6 @@ Something went wrong: { reply_error }
 
 ---
 
-That was one file. Static prose, build-time loops, fetched data, nested conditionals, persistent state, computed values, and a server round-trip — and the only JavaScript on the page is the shared runtime Darkmown loads for you. [Read the docs](/docs/) or [deploy your own](/#deploy).
+That was one file. Static prose, build-time loops, fetched data, nested conditionals, persistent state, computed values, and a server round-trip, and the only JavaScript on the page is the shared runtime Darkmown loads for you. [Read the docs](/docs/) or [deploy your own](/#deploy).
 
 </main>

package/site/pages/blog/_schema.wd

@@ -9,6 +9,7 @@ date: date
 description: string
 excerpt: string?
 tags: string[]?
+schema: string?
 ---
 
 The blog collection schema. Posts are plain `.md` files next to this file;

package/site/pages/blog/collections-and-pagination.md

@@ -3,6 +3,7 @@ title: Collections and pagination, with zero ceremony
 description: Any folder under site/pages is a queryable collection. Loop it by name, sort it, paginate it — all at build time, all zero-JS.
 date: 2026-06-28
 excerpt: This very page is an entry in a collection. The list you came from is one @loop over a folder — no content root, no marker file, no config.
+schema: BlogPosting
 tags: [collections, content]
 ---
 

package/site/pages/blog/drafts-that-never-leak.md

@@ -3,6 +3,7 @@ title: Drafts that never leak
 description: draft true excludes a page everywhere in a production build — dist, routes.json, sitemap, and rss.
 date: 2026-06-27
 excerpt: Set draft true on a post and it vanishes from a production build — dist, routes.json, the sitemap, and the feed — even with a date.
+schema: BlogPosting
 tags: [drafts, workflow]
 ---
 

package/site/pages/blog/shipping-feeds.md

@@ -3,6 +3,7 @@ title: Shipping feeds without a config file
 description: Site identity lives in the home page frontmatter — one site_url field turns on sitemap.xml and rss.xml.
 date: 2026-06-26
 excerpt: One reserved frontmatter field on the home page — site_url — turns on the whole feeds story. No config loader, no plugin.
+schema: BlogPosting
 tags: [feeds, seo]
 ---
 

package/site/pages/blog/typed-frontmatter.md

@@ -3,6 +3,7 @@ title: Typed frontmatter with a four-line schema
 description: Drop a _schema.wd in a collection and every entry is validated at build time — missing fields, wrong types, and typos all fail the build.
 date: 2026-06-29
 excerpt: A typo'd frontmatter key used to ship silently. With a collection schema, titel instead of title fails the build with a file and line.
+schema: BlogPosting
 tags: [collections, schema, dx]
 ---
 

package/site/pages/blog/zero-js-by-default.md

@@ -3,6 +3,7 @@ title: Zero JavaScript, by default
 description: A Darkmown page ships no framework JavaScript unless it declares reactive behavior. The blog you are reading proves it.
 date: 2026-06-25
 excerpt: View source on this page. There is no runtime script tag, because nothing on it is reactive — and the framework knows the difference.
+schema: BlogPosting
 tags: [performance, philosophy]
 ---
 

package/site/pages/data.wd

@@ -21,7 +21,7 @@ This page demos the Tier 1.5 surface: declarative fetch, forms that write state,
 :if team
 @loop team into member
 ::: card .member-card
-**{ member.name }** — { member.role }
+**{ member.name }**: { member.role }
 :if member.lead
 Team lead ★
 :else
@@ -37,14 +37,14 @@ That loop also proves `:if` over loop items: the lead badge is decided per row.
 
 ## The four states of a fetch
 
-Every `:fetch` tracks the whole lifecycle — `loading`, `error`, `empty`, and data — so a page can answer each one. Here the same primitive shows an **error** (a missing URL 404s) and an **empty** result (a file that is just `[]`):
+Every `:fetch` tracks the whole lifecycle (`loading`, `error`, `empty`, and data) so a page can answer each one. Here the same primitive shows an **error** (a missing URL 404s) and an **empty** result (a file that is just `[]`):
 
-*(Heads up: the browser console will log a `404` for `intentionally-missing.json` on this page — that failure is deliberate, so the error branch below has something real to show.)*
+*(Heads up: the browser console will log a `404` for `intentionally-missing.json` on this page. That failure is deliberate, so the error branch below has something real to show.)*
 
 :fetch broken from "/__wd/data/intentionally-missing.json"
 
 :if broken_error
-The request failed on purpose — `broken_error` holds: { broken_error }
+The request failed on purpose. `broken_error` holds: { broken_error }
 :else if broken_loading
 Trying the missing endpoint…
 :else
@@ -56,7 +56,7 @@ Waiting…
 @loop roster into person
 - { person.name }
 @empty
-The roster came back empty — `@empty` renders instead of the rows.
+The roster came back empty, so `@empty` renders instead of the rows.
 @endloop
 
 The error branch uses an `:else if` chain; the empty branch uses the loop's `@empty`. No state machine to wire by hand.
@@ -71,12 +71,12 @@ A real API usually wants a token. `:fetch … headers=<key>` spreads a state obj
 :if feed_error
 The API rejected the token: { feed_error }
 :else if feed
-Authenticated fetch returned { feed.length } row(s). Open the network tab — the `Authorization` header rides along on the request.
+Authenticated fetch returned { feed.length } row(s). Open the network tab: the `Authorization` header rides along on the request.
 :else
 Loading the authenticated feed…
 :endif
 
-This demo points `from=` at a static file (which ignores the header), so it stays self-contained — point it at your real API and the bearer token travels with every request. When that token expires, add `refresh="/auth/refresh"` and Darkmown renews it on a `401` and retries once, automatically.
+This demo points `from=` at a static file (which ignores the header), so it stays self-contained. Point it at your real API and the bearer token travels with every request. When that token expires, add `refresh="/auth/refresh"` and Darkmown renews it on a `401` and retries once, automatically.
 
 ## Forms write state, no backend required
 
@@ -89,16 +89,16 @@ This demo points `from=` at a static file (which ignores the header), so it stay
 :endform
 
 :if profile
-Hello **{ profile.name }** — good luck with { profile.quest }.
+Hello **{ profile.name }**, good luck with { profile.quest }.
 :else
 Submit the form and this sentence reacts. Nothing leaves the page.
 :endif
 
 ## Server round-trips, adapter style
 
-Darkmown does not own your backend — point a form at any endpoint. With JS, the submit becomes a fetch and the JSON reply lands in state; without JS, the same markup is a plain native POST:
+Darkmown does not own your backend: point a form at any endpoint. With JS, the submit becomes a fetch and the JSON reply lands in state; without JS, the same markup is a plain native POST:
 
-:form action="/api/echo" into reply
+:form action="/api/echo/" into reply
 :input message placeholder="Say something to the server"
 :submit "Send to server"
 :endform
@@ -113,7 +113,7 @@ The reply will appear here.
 The request failed: { reply_error }
 :endif
 
-In `darkmown dev` the echo endpoint is built into the dev server; on darkmown.com it is a real serverless function behind the same URL. On static-only hosts such as a plain Cloudflare Pages deploy, replace `action="/api/echo"` with your own API endpoint. Either way, Darkmown itself stays static — sessions ride on `:fetch` plus ordinary cookies against your real API.
+In `darkmown dev` the echo endpoint is built into the dev server; on darkmown.com it is a real serverless function behind the same URL. On static-only hosts such as a plain Cloudflare Pages deploy, replace `action="/api/echo/"` with your own API endpoint. Either way, Darkmown itself stays static, and sessions ride on `:fetch` plus ordinary cookies against your real API.
 
 ## Persistent state survives reload
 
@@ -130,7 +130,7 @@ The cart holds { items.length } item(s) worth ${ total }.
 :button "Empty cart" -> items = []
 :::
 
-`:computed total = items.length * 4` derives state from state — the price updates with every cart change, persistence included.
+`:computed total = items.length * 4` derives state from state, so the price updates with every cart change, persistence included.
 
 ## The escape hatch
 
@@ -140,7 +140,7 @@ Colocated `data.js` uses the `window.wd` API (`wd.get`, `wd.set`) for anything d
 
 ## Lazy loading
 
-The list below uses `when=visible` — the request only fires once this section scrolls into view. Watch the network tab:
+The list below uses `when=visible`, so the request only fires once this section scrolls into view. Watch the network tab:
 
 :fetch quotes from "/__wd/data/quotes.json" when=visible
 
@@ -173,20 +173,20 @@ Scroll me into view and I will load.
 :endform
 
 :if inquiry
-Thanks **{ inquiry.fullname }** — we logged your "{ inquiry.topic }" note. Services: { inquiry.services }. Urgency: { inquiry.urgency }.
+Thanks **{ inquiry.fullname }**, we logged your "{ inquiry.topic }" note. Services: { inquiry.services }. Urgency: { inquiry.urgency }.
 :else
-Pick a topic and send — it is captured client-side, no backend.
+Pick a topic and send. It is captured client-side, no backend.
 :endif
 
 ## Dynamic links from data, and link buttons
 
-A build-time `@loop` can drive the `href` itself — `[{ link.label }]({ link.url })` resolves the destination at compile time, so these stay plain static links:
+A build-time `@loop` can drive the `href` itself: `[{ link.label }]({ link.url })` resolves the destination at compile time, so these stay plain static links:
 
 @loop /links.json into link
 - [{ link.label }]({ link.url })
 @endloop
 
-A trailing `{.class}` styles a single link as a button — no wrapper container:
+A trailing `{.class}` styles a single link as a button, with no wrapper container:
 
 [Read the docs](/docs/){.pulse-button}
 

package/site/pages/docs/index.wd

@@ -3,6 +3,7 @@ html: true
 title: Darkmown docs
 description: Learn Darkmown routing, directives, state, loops, forms, and the static-to-reactive model.
 transitions: true
+schema: TechArticle
 ---
 
 @include /nav.wd
@@ -31,6 +32,9 @@ Darkmown is a Markdown-native framework for mostly-static sites with tiny reacti
 - [Interpolation](#interpolation)
 - [Frontmatter](#frontmatter)
 - [SEO & feeds](#seo--feeds)
+- [Canonical URLs](#canonical-urls)
+- [Structured data](#structured-data)
+- [AI crawlers](#ai-crawlers)
 - [Drafts](#drafts)
 - [Instant navigation](#instant-navigation)
 - [Images](#images)
@@ -296,6 +300,57 @@ site_url: https://example.com
 
 A page becomes an **RSS post** by carrying a `date:`; `excerpt:` is its feed summary (else `description:`, else — for a plain `.md` — its first paragraph). The sitemap's `<lastmod>` is the `date:` if set, else the file's git/last-modified date. `robots.txt` is always written; the `Sitemap:` line only appears with a `site_url`. Without `site_url`, robots still emits and the build prints a hint — it never breaks.
 
+## Canonical URLs
+
+With `site_url` set, every page states its own absolute URL twice, as `<link rel="canonical">` and as `og:url`, both built from the **same route string** the sitemap and your internal links use, so nothing can drift out of agreement:
+
+```html
+<link rel="canonical" href="https://example.com/docs/">
+<meta property="og:url" content="https://example.com/docs/">
+```
+
+Darkmown routes are **trailing-slashed** (`/docs/`), matching the `dist/docs/index.html` the build writes. `darkmown deploy vercel` writes `trailingSlash: true` so the host serves exactly that form. Host it elsewhere and configure the same: a host that redirects `/docs/` to `/docs` puts a redirect hop on every internal navigation and makes your own sitemap advertise URLs that redirect.
+
+A paginated listing canonicalises **each page to itself** (`/blog/page/2/` points at `/blog/page/2/`), never back to page one. Pages 2+ hold different content, and pointing them at page one asks a crawler to drop them.
+
+`og:type` is `article` for a page with a `date:` (or an article `schema:`), `website` otherwise. There is no `twitter:title`/`twitter:description`: X's card parser falls back to the Open Graph tags, so they would be duplicate bytes on every page.
+
+## Structured data
+
+One frontmatter key emits a JSON-LD block into the head. This is ordinary indexing hygiene for Google's conventional **rich results**; Google's own generative-search guidance says there is no special AI schema and that structured data is not required for generative results, so Darkmown does not sell it as an AI-citation lever.
+
+```md
+---
+title: Zero JavaScript, by default
+date: 2026-06-25
+author: Ada Lovelace
+schema: BlogPosting
+---
+```
+
+The types are a compile-time whitelist, like every other Darkmown vocabulary: `Article`, `BlogPosting`, `TechArticle`, `WebSite`, `Organization`. Anything else is a compile error that names every valid type. Pass a list for a page that is honestly both: `schema: [WebSite, Organization]`. Supporting keys are `author:` (a name or a list) and `updated:` for articles, `organization:` and `logo:` for `Organization`; everything else is reused from `title`, `description`, `image`, `date`, `lang`, and the canonical URL.
+
+**Only what the page has.** A key you did not write produces no property at all: never a blank one, never a guess. There is deliberately no way to emit `aggregateRating`, `review`, or `offers`: fabricated ratings are a manual-action risk and Darkmown cannot know them. `FAQPage` is not supported either, because it would mean inferring a Q&A structure out of prose.
+
+**`BreadcrumbList` is automatic** for nested routes (two or more path segments) once `site_url` is set, built from routes that actually exist with their real titles, so a crumb never links a page you never wrote. JSON-LD is an inert data block, so a static page stays static and ships no JavaScript.
+
+## AI crawlers
+
+`robots.txt` names every major AI crawler and answer engine explicitly, grouped by operator, each group linking the documentation its tokens were verified against and noting what that operator's crawlers actually do:
+
+```
+# OpenAI: https://developers.openai.com/api/docs/bots
+# OAI-SearchBot = ChatGPT Search crawling and citations. GPTBot = potential model training. …
+User-agent: OAI-SearchBot
+User-agent: GPTBot
+User-agent: ChatGPT-User
+Allow: /
+```
+
+Covered: OpenAI, Anthropic, Google (`Google-Extended`), Apple (`Applebot-Extended`), Perplexity, Meta, Mistral, Amazon, and Common Crawl. Search crawling, model training, and live user-triggered fetches are **different** permissions from the same operator, which is why each group says which is which.
+
+Flip the whole set to `Disallow: /` from the home page with `ai_crawlers: deny`. `allow` is the default. A value that is neither fails the build rather than defaulting. The intent behind a typo like `ai_crawlers: block` is almost always to opt *out*, and silently allowing would be the one unrecoverable outcome.
+
 ## Drafts
 
 Mark a page `draft: true` and it stays out of production: `darkmown build` leaves it out of `dist`, `routes.json`, the sitemap, and the feed — even if it has a `date:`. `darkmown dev` still builds and serves it (with a "DRAFT" banner you only see in dev), and `darkmown build --drafts` includes drafts for a staging deploy. This is separate from the permanent `.`/`-`/`_` filename hiding — a hidden name is private forever; `draft:` is a switch you flip when the page is ready.
@@ -496,6 +551,7 @@ Darkmown builds 100% static HTML — never per-request rendering. When you need
 
 - A backend endpoint is a Web-standard handler in a top-level `api/` directory: `export default (request, context) => Response`. `api/echo.js` → `/api/echo`; `api/users/[id].js` → `/api/users/:id`.
 - One shape runs everywhere: `darkmown dev` (local runner), Vercel (native Edge Functions), and Cloudflare Pages (the build emits a `dist/_worker.js`). `:fetch`/`:form` point at `/api/*` with no extra config.
+- If your host rewrites trailing slashes, point at the form it will settle on. Vercel's `trailingSlash: true` redirects `/api/echo` to `/api/echo/`, and the local runner answers both, so a missing slash costs a redirect that only appears once deployed. The redirect is a `308`, so a form POST still arrives intact; you are paying a round trip, not losing data.
 - `darkmown deploy vercel` or `darkmown deploy cloudflare` builds and ships in one command, printing your URL.
 - For a remote/custom backend, point `:fetch`/`:form` at an absolute URL and widen the CSP `connect-src` / `form-action`. Darkmown owns no server — it adapts to yours.
 
@@ -610,11 +666,13 @@ See [SECURITY.md](https://github.com/ZVN-DEV/darkmown/blob/master/SECURITY.md) i
 Darkmown ships a machine-readable description of its own language, so an AI tool can learn `.wd` and be constrained to output that compiles. It is all generated from the compiler's own tables, so it never drifts from the real grammar.
 
 - **`darkmown catalog`** prints structured JSON: every directive, `@loop` clause, loop variable, button action, format pipe, and predicate operator — each with a syntax template, a one-line description, one concrete example, and whether it needs the reactive runtime. Import it too: `import { directiveCatalog } from "@zvndev/darkmown/catalog"`.
-- **`darkmown catalog --llms`** prints a compact (~90-line) cheatsheet — the artifact you paste into a model's system prompt. Every build also writes it to `dist/llms.txt`.
+- **`darkmown catalog --llms`** prints a compact cheatsheet, the artifact you paste into a model's system prompt. Every build writes it to `dist/llms.txt` too, followed by a one-line index of every page on your site and a pointer to the full corpus.
+- **`darkmown catalog --llms-full`** prints the complete reference that index points at: every directive with its full syntax and example, every clause, action, pipe, operator, and frontmatter key, and every compile-error code with its cause and fix. Every build writes it to `dist/llms-full.txt`, with the full source text of every page appended.
 - **`grammar/wd-directives.gbnf`** is a generated GBNF grammar for `.wd` directive lines. Point a grammar-constrained decoder (llama.cpp and friends) at it to make invalid directive lines — JS idioms, HTML muscle-memory — impossible to generate.
 
 ```sh
 darkmown catalog --llms > system-prompt.md
+darkmown catalog --llms-full            # the complete reference + every error code
 ```
 
 Compile errors help here too: every corrective `Use:` hint ends with a concrete `— e.g. <valid line>`, and each thrown error carries a structured `err.wd = { file, line, hint, example }` mirror alongside its message — so an edit-and-retry loop gets the fix without parsing prose.
@@ -640,7 +698,7 @@ The `darkmown` command (installed by `npx @zvndev/darkmown`) is the whole tool
 - `darkmown dev` — the live compiler: browser reload, an in-browser error overlay, incremental rebuilds, and a local runner for `api/*` functions.
 - `darkmown build [--target cloudflare] [--drafts]` — write 100% static output to `dist/` (plus `sitemap.xml` / `rss.xml` / `robots.txt`); `--drafts` includes `draft: true` pages for a staging deploy.
 - `darkmown serve` — preview the built `dist/` locally (static only; use `dev` to exercise `api/*`).
-- `darkmown catalog [--llms]` — print the `.wd` directive catalog as JSON, or with `--llms` a compact cheatsheet for priming an AI model (see [AI authoring](#ai-authoring)).
+- `darkmown catalog [--llms|--llms-full]`: print the `.wd` directive catalog as JSON, a compact cheatsheet for priming an AI model, or the complete reference including every error code (see [AI authoring](#ai-authoring)).
 - `darkmown deploy <vercel|cloudflare> [--prod]` — build (target-aware) and deploy through the platform CLI, printing your URL; an unauthenticated CLI surfaces the login to run first.
 - `darkmown version` / `darkmown help` — print the installed version, or the full command summary and directive syntax.
 

package/site/pages/folio/cart.wd

@@ -107,7 +107,7 @@ Tax (8%)
 
 This is a real form. With the runtime it posts over `fetch` and shows the reply inline; with JavaScript off, the identical markup degrades to a native POST. The demo endpoint simply echoes your order back.
 
-:form action="/api/echo" into receipt
+:form action="/api/echo/" into receipt
 :input name placeholder="Full name" required
 :input email type=email placeholder="you@studio.com" required
 :select shipping
@@ -173,7 +173,7 @@ Total with tax — **{ total | money }**
 
 :button "Clear basket" -> cart clear
 
-:form action="/api/echo" into receipt
+:form action="/api/echo/" into receipt
 :input name placeholder="Full name" required
 :input email type=email placeholder="you@studio.com" required
 :select shipping

package/site/pages/forge/index.wd

@@ -248,7 +248,7 @@ A month-to-month list price would run **${ annual }/yr**. Choose yearly at check
 Drop in your details and we'll spin up your **${ total }/mo** workspace — no charge for 14 days.
 
 ::: div .checkout-card
-:form action="/api/echo" into receipt
+:form action="/api/echo/" into receipt
 :input name placeholder="Your name"
 :input email placeholder="Work email"
 :submit "Start 14-day trial"

package/site/pages/index.wd

@@ -4,6 +4,7 @@ description: A .md file is a page. Rename it to .wd and the same file gets loops
 image: https://darkmown.com/__wd/media/og.png
 site_url: https://darkmown.com
 transitions: true
+schema: WebSite
 ---
 
 @include /nav.wd
@@ -95,7 +96,7 @@ Cart: **{ count }** item(s)
 
 Not a screenshot, and not a React site describing a Markdown tool. `site/pages/index.wd` is a `.wd` file with no raw HTML in it. The first two numbers below are live: a `:fetch` reading `api/stats.js`, a serverless function in this same repo.
 
-:fetch stats from "/api/stats" timeout=6000
+:fetch stats from "/api/stats/" timeout=6000
 
 ::: livestrip
 
@@ -147,7 +148,7 @@ JS on a static page
 
 Say hello and the form round-trips to a real serverless function, then renders the reply:
 
-:form action="/api/echo" into reply
+:form action="/api/echo/" into reply
 :input note placeholder="What would you build with this?"
 :submit "Send it"
 :endform

package/site/pages/vs/markdoc.wd

@@ -4,6 +4,7 @@ description: Markdoc gives Markdown conditionals, partials and variables, but no
 image: https://darkmown.com/__wd/media/og.png
 site_url: https://darkmown.com
 transitions: true
+schema: TechArticle
 ---
 
 @include /nav.wd