@jimhoyd/urlcode 0.4.2 → 0.4.6

package/docs/archive/2026-09-19/USABILITY-REVIEW.md

@@ -1,139 +0,0 @@
-# Historical record
-
-Archived 2026-09-19. This records an earlier implementation or proposal, not
-current instructions. See the [current roadmap](../../../ROADMAP.md),
-[current contract](../../SPECIFICATION.md) and [open decisions](../../OPEN-DECISIONS.md).
-Remaining acceptance work is not declared complete by archiving this record.
-
-<!-- trust-model-prose: historical-file -->
-<!-- guidance-claims: ignore-file -->
-
-# Usability review: URLCode against the tools people reach for instead
-
-Status: an honest assessment written 2026-09-18 against the source in the four
-repositories at that date. It is opinion grounded in the code and docs, not a
-benchmark. Its purpose is to name the friction a first-time user or an AI
-agent meets, compare it with what competing tools do, and rank the changes
-that would matter most. It does not change any contract by itself.
-
-> **Update:** written the day before `0.4.0-alpha.2` reversed the default
-> execution mode. Where the table below credits functions and middleware with
-> isolation, that now describes a route declaring `sandbox: true`; the default
-> is trusted, in-process execution with full Node, npm and filesystem access.
-> See [SPIKE-DEFAULT-TRUST-MODEL.md](../../SPIKE-DEFAULT-TRUST-MODEL.md).
-
-## 1. Who the competitors are, by rung
-
-URLCode is not one product; it competes with a different tool on each rung
-of [the ladder](../../FRAMEWORK.md#the-ladder).
-
-| Rung | What people use today | Where URLCode is better | Where it is worse today |
-|---|---|---|---|
-| Redirects, responses | Netlify `_redirects`, Cloudflare `_redirects`/Rules, Vercel `vercel.json`, nginx | One portable file that validates before deploy, tests with fixtures, counts routes, converts from those formats and refuses lossy conversions | Those files are two columns; `urlcode.yaml` needs `version`, a route key and a handler object. Nobody needs a validator for ten redirects |
-| Pages, files | Any static host, Astro, Hugo | Same file, no build step, native ranges/ETags, `site` conventions | No content pipeline, no templating for pages outside the extension kit |
-| Functions, middleware | Cloudflare Workers, Vercel Functions, Hono, Express | Real isolation with a fresh heap per call; typed args from YAML; secrets only by revision-pinned grant; runs the same on a laptop | No `fetch`, no timers, no streaming, no npm packages in guests. Every framework above lets you call an API from a function; here you declare a `proxy` or stop |
-| Live links | Bitly, Dub, Short.io, a Postgres table | Records without reloads, versioned writes, private management API, no account system to run | Single host SQLite that needs a patched Node build; no dashboard until you install admin |
-| Accounts | Clerk, Auth0, Auth.js, Better Auth, Lucia, Supabase Auth, Devise | Everything is operator-owned and reviewable: no vendor, no callback URLs on someone else's dashboard, no per-MAU bill; passkeys, OIDC, TOTP, recovery and an account page in one package | Install is a reviewed-tarball ritual, a host file, a SHA-256 pin and a JSON-on-stdin bootstrap. Clerk is `npm i` and one env var. Better Auth is a config object and a CLI migration |
-| Administration | Django admin, Rails Administrate, Retool, Forest Admin, Supabase Studio | Built on the same service with two-person cases and audited reveal, which none of the generic admins give you | Only manages auth's entities; there is no way to expose the project's own data. Django admin is one line per model |
-| Full app in one place | Rails, Django, Laravel, Next.js + Clerk + Prisma, Supabase | The whole thing is declarative and portable; an agent can generate and validate it without a build; the trust boundary between generated code and the operator is explicit | Everything that is not a route, a function or an account is out of scope: no data model, no ORM, no forms for your own tables, no client bundle, no email templates for your app |
-
-The honest summary: URLCode wins on portability, validation, isolation and
-operator ownership, and loses on the first fifteen minutes and on anything
-that needs application data.
-
-## 2. The first fifteen minutes
-
-What a new person does today, compared with the fastest competitor at each
-step. Times are what the docs imply, not measurements.
-
-| Step | URLCode today | Fastest competitor | Gap |
-|---|---|---|---|
-| Install | `npm i -g @jimhoyd/urlcode` or brew or a checksum script; three channels documented at equal weight | `npx create-next-app` | Too many equal choices on the first screen. Pick one, put the rest in the install guide |
-| First project | `urlcode init` writes a function route, a redirect and fixtures | `wrangler init` | Equal |
-| First redirect | 5 lines of YAML for one redirect | 1 line in `_redirects` | Acceptable once, heavy for 200. `bulk-import` exists but a person starting small never learns it |
-| First function | Must know: `parameters` with `schema`, `args` with `from: path`, the Request/Response subset, that `fetch` is absent | Hono: `app.get('/x', c => c.json(...))` | The typed-args design is right but needs a two-line minimal form. Today the smallest function route is 10 lines |
-| Add accounts | Clone two private repos, build tarballs with `pack-sources.mjs --core --auth --ui --admin --core-revision SHA`, install four tarballs, `urlcode-auth init`, compute `inspectExtensionRevision`, paste the SHA into an env var, `bootstrap` with JSON on stdin, `serve --host-file --origin` | Clerk: `npm i @clerk/nextjs`, one env var, wrap the app | This is the largest gap in the framework. Most of it is a consequence of the packages being unpublished and of the revision pin, and it is the step the README leads with |
-| Add admin | `urlcode-admin init` writes everything | Django `admin.site.register(Model)` | Comparable once installed |
-| Deploy | Container, Node process, or an adapter that refuses functions, links and extensions | `vercel deploy` | The refusals are correct but the person learns them at deploy time. `urlcode capabilities` exists and is not in the quick start |
-
-## 3. What an AI agent meets
-
-The framework's claim is that an agent should build from a handful of
-redirects to a full application without rebuilding the core. Measured
-against that claim:
-
-- **Strong:** one JSON Schema, a generated field reference, fixtures the agent
-  must write, `validate`, `test` and `audit` that give exact failures with the
-  route named, an MCP server for read-only inspection, and a capability
-  matrix that lists what does not exist. Few frameworks tell an agent what it
-  cannot do. This is the right foundation.
-- **Weak:** the documentation was organized by feature history rather than by
-  task. Before this review, `llms.txt` listed 40 documents at equal weight,
-  three of them status logs, and the README opened with release history and
-  "unreleased source" caveats. An agent reading it spent its context on
-  provenance rather than on the shape of a project. The framework page and the
-  reorganized index in this change address that; the remaining cost is the
-  size of the reference documents themselves (the YAML guide, policies and
-  dynamic links are each over 400 lines).
-- **Missing:** an agent cannot yet discover the extension packages' YAML from
-  the core schema. `extensions.auth.config` is validated by auth's schema at
-  activation, but there is no way to ask the installed runtime "what config
-  does `auth` accept" without the host file. A `urlcode extensions --schema`
-  command that reads the host file and prints each extension's configuration
-  and policy schemas would close this, and would let `urlcode mcp` serve them.
-- **Missing:** no single command creates the whole layered project. Today it
-  is `urlcode init`, then `urlcode-auth init`, then `urlcode-admin init`, each
-  with its own directory conventions. One `urlcode init --with auth,admin`
-  that delegates to the installed extension packages' scaffolds would make
-  the ladder real for an agent.
-
-## 4. Ranked recommendations
-
-Ordered by how much each would change the experience per unit of work, and
-whether it touches a contract.
-
-1. **Publish the three extension packages** (even as `0.1.0-alpha` with the
-   caveats their status files carry). Every install step in section 2's
-   "add accounts" row except the revision pin existed because they were
-   unpublished. This was a decision, not code; the repositories already had
-   the release checks. No contract change. Done 2026-09-18: all three are on
-   npm as `0.1.0-alpha.x` (review still pending, issue 58).
-2. **`urlcode init --with auth,admin,ui`.** Delegate to each installed
-   package's existing scaffold; write one host file and one README. No contract
-   change; a CLI addition in core that calls into optional peers.
-3. **Print extension schemas.** `urlcode extensions --host-file … --json`
-   listing each registered extension's name, version, configuration schema and
-   policy schema, and expose it through `urlcode mcp`. No contract change.
-4. **A short form for the common function route.** Allow `function:
-   functions/hello.mjs` as a string with path parameters inferred as required
-   strings of bounded length, expanding to today's long form. This is a schema
-   addition (`version: "1"` stays valid) and the single largest cut in YAML
-   for first-time users and agents. Needs the usual generated-reference and
-   cookbook updates.
-5. **Lead the README with the ladder, not the release history.** Done in this
-   change; keep it that way. Move status caveats to the readiness register.
-6. **One install channel on the first screen.** npm first; brew, script and
-   container in the install guide.
-7. **Fold the `presentation`/`ui` split.** Auth and admin still render through
-   the primitives while the kit is the documented way to restyle. Finishing
-   kit adoption (already listed in the ui status file) removes the one place
-   where the framework's story and its code differ.
-8. **Later, and a real contract question:** application data. The thing every
-   competitor on the last rung has and URLCode does not is a place for the
-   project's own records with an admin view. The runtime already has one
-   bounded store (links) and one admin surface (auth's entities). Whether a
-   declared `collection` handler with operator-owned SQLite and an admin
-   registration belongs in the framework is the next spike worth writing.
-   Without it, "full-fledged application" means "site with accounts".
-
-## 5. What is fine and should stay
-
-- The operator host file. It is the reason the project can be untrusted and
-  portable at the same time; Clerk's convenience is bought with a vendor in
-  the loop. Keep the boundary, make the file generated.
-- The revision pin. It is unusual and it is what makes "an agent changed the
-  YAML" a reviewable event rather than a silent grant. Keep it, print it
-  loudly, and make `init` write it.
-- Refusing instead of degrading on every target. The failure names the route.
-- The capability matrix in the AI guide. Extend it to the extension packages
-  rather than softening it.

package/docs/archive/2026-09-20/OPEN-DECISIONS-COMPLETED.md

@@ -1,116 +0,0 @@
-# Historical record
-
-Archived 2026-09-20 from [open decisions](../../OPEN-DECISIONS.md), which was
-reconciled against its issues that day (#242). These sections record completed
-work and dated observations exactly as they stood on 2026-09-19 (core `db375bf`
-and the package sources of that date); they are not current instructions and
-their version numbers, dist-tags, commits and pins are not current. Versions
-and channels: [version alignment](../../VERSION-ALIGNMENT.md) and
-`npm run release:status`. Archiving does not declare any security, deployment,
-recovery or provider check performed. The live middleware decision remains in
-open decisions.
-
-<!-- trust-model-prose: historical-file -->
-<!-- guidance-claims: ignore-file -->
-<!-- local-links: historical-file -->
-
-## Closed questions removed from the active list
-
-- Publishing convention is recorded in [version alignment](../../VERSION-ALIGNMENT.md):
-  publishable manifests on main, releases through reviewed tags/workflows.
-- Core `0.4.0-alpha.2` and current extension releases exist; publishing that
-  already-shipped version is not a next step.
-- Auth/admin kit adoption and shared form helpers are implemented in their code.
-- The UI primitive fallback is retired, which settles the question this table
-  carried. Auth and admin now render every screen through the kit and refuse
-  activation without it (`packages/auth/src/auth.ts`: "there is no
-  shared-primitive fallback"), so "keep both" no longer describes the code.
-- The template pins `0.4.0-alpha.2`. Its skill differences were read against that
-  pin: omitted handlers and advice about the removed management API are stale,
-  not intentional older-version behavior.
-- The guidance checks run through `npm run check` inside `verify`; a regex check
-  is not a schema validator for every example. Extending its coverage is tracked
-  separately, not a reason to weaken review or bypass required checks.
-
-## Done: the monorepo migration is complete
-
-**Closed 2026-09-19.** `urlcode-ui`, `urlcode-auth` and `urlcode-admin` are
-workspace packages under `packages/`, and all three have been released from
-this repository — `@jimhoyd/urlcode-ui@0.1.0-alpha.6`,
-`@jimhoyd/urlcode-auth@0.1.0-alpha.6`, `@jimhoyd/urlcode-admin@0.1.0-alpha.4`,
-each on `alpha` with `latest` deliberately held behind. Core's dist-tags are
-unchanged. The three source repositories are gone; their history survives only
-as verified `git bundle`s, because the repository allows squash merges only and
-the imported commits did not survive onto `main`.
-
-The operational runbook is [DEVELOPMENT-PIPELINE.md](../../DEVELOPMENT-PIPELINE.md)
-and [RELEASE-SECURITY.md](../../RELEASE-SECURITY.md). The plan itself is archived at
-[archive/2026-09-19/SPIKE-MONOREPO.md](../2026-09-19/SPIKE-MONOREPO.md),
-whose closing note records what the plan got wrong — chiefly that its
-strongest argument, the reach of the enforcing checks, only became true after
-both checkers were changed to discover workspace packages.
-
-The section below is kept for the middleware decision it records, which is
-still the reason there is no `packages/middleware`.
-
-## Accepted: monorepo first — middleware withdrawn rather than consolidated
-
-The maintainer confirmed that monorepo work is starting now. The earlier
-recommendation to keep repositories separate for now is superseded.
-
-**Reversed 2026-09-19: the middleware half of this decision no longer applies.**
-This section used to say "move middleware into the monorepo as its own package
-first, preserving its existing API and behavior," and explicitly: "do not
-unpublish or retire the middleware package as part of the initial move."
-That instruction was overtaken. `@jimhoyd/urlcode-middleware` has been
-**unpublished** from npm at `0.1.0-alpha.2` and `jimhoyd-com/urlcode-middleware`
-**deleted** — the package was withdrawn outright instead of migrated, so there
-is no `packages/middleware` to create and no subsequent fold-into-core step.
-The reversal is recorded here rather than deleted because the instruction it
-replaces was explicit, and a reader who remembers it should be able to see that
-it was changed deliberately and not simply forgotten.
-
-Nothing was lost in capability terms: per-route middleware is **native to
-core** via the `middleware:` array ([MIDDLEWARE.md](../../MIDDLEWARE.md)), and the
-deleted package only ever offered the same behavior through the extension
-seam. Trust remains the default and explicit sandbox choices keep their
-meaning. The generic extension wrapping hook (`ExtensionInstance.middleware`,
-`RuntimeExtension.cacheSensitive`) stays in core's contract for other
-extensions — it is no longer exercised by any shipped package, which is worth
-knowing before it is assumed to be covered. Static targets continue rejecting
-request-time middleware because there is no server to run it.
-
-Migration scope is therefore **core, auth, admin and UI**. Template and the
-distribution tap stay outside that package move. The observed stale peer pins
-and checkout-limited guidance checks strengthen the case for shared
-verification: carry those checks across the new package paths, since merely
-moving files does not prove every generated skill is covered. The cleanup PRs
-that superseded the earlier zero-open-PR survey have since merged, and a fresh
-survey again reports zero open pull requests across all four in-scope
-repositories — re-run it per repository immediately before that repository
-moves rather than trusting this line.
-
-The [archived monorepo plan](../2026-09-19/SPIKE-MONOREPO.md) records
-migration context.
-[Issue 172](https://github.com/jimhoyd-com/urlcode/issues/172), which tracked
-"consolidate middleware into core after moving it into the monorepo," was
-**closed on 2026-09-19** as moot — there was nothing left to consolidate.
-Migration starting is not a claim that it has landed.
-
-## Source review baseline
-
-| Repository | Reviewed commit | Code checked |
-|---|---|---|
-| core | `db375bf` | Runtime dispatch, schema normalization, capabilities, static compiler, prerender, MCP and resource generators |
-| auth | `71957dd` | Lifecycle hooks, UI rendering and shared helper imports |
-| admin | `f3b4882` | UI rendering, auth-service integration and shared helper imports |
-| UI | `0e96f7f` | Shared forms, kit/host exports and copied core contract |
-| ~~middleware~~ | `f201f4b` | Extension wrapping, per-entry sandbox dispatch and scaffolding — **repository deleted 2026-09-19; this baseline is unreachable except through the local `urlcode-middleware.bundle`** |
-| template | `4e09e50` | Exact core pin, generated guide and both vendored skills |
-
-The Homebrew tap (`73eaaef`) still selects stable core `0.3.0`; its old trust
-behavior belongs to that pin and must not be rewritten as alpha.2 behavior.
-The other organization tap and Scoop bucket contain Gitroll, not URLCode.
-The deleted documentation repository is historical context, not a second source
-of current contracts; its former GitHub links no longer resolve. This review is targeted source inspection, not an audit of
-every execution path or an independent security assessment.

package/docs/archive/README.md

@@ -1,29 +0,0 @@
-# Historical records
-
-These documents retain earlier reasoning and evidence. They are not authoring
-instructions or an active backlog. Read [the current roadmap](../../ROADMAP.md),
-[open decisions](../OPEN-DECISIONS.md) and [implemented contract](../SPECIFICATION.md)
-first. An old example can describe a removed feature or an earlier trust model.
-
-Archived on 2026-09-19 after checking the current code:
-
-| Record | Why archived | Current owner of remaining work |
-|---|---|---|
-| [Monorepo spike](2026-09-19/SPIKE-MONOREPO.md) | The migration it proposed is complete: ui, auth and admin are workspace packages and all three have been released from this repository | [Development pipeline](../DEVELOPMENT-PIPELINE.md) and [release security](../RELEASE-SECURITY.md) |
-| [Previous roadmap](2026-09-19/ROADMAP.md) | Release chronology mixed with current planning | Current roadmap and release readiness |
-| [Detailed phases](2026-09-19/NEXT-STEPS.md) | Most authoring/tooling/UI work delivered; repeated stale backlog | Current roadmap, issues and open decisions |
-| [Previous decisions](2026-09-19/OPEN-DECISIONS.md) | Mixed settled releases, obsolete repository counts and live questions | Current open decisions |
-| [Next-phase implementation](2026-09-19/NEXT-PHASE-PLAN.md) | Delivered source snapshot for alpha.1 | Specification and provider evidence |
-| [Extension delivery sequence](2026-09-19/EXTENSION-IMPLEMENTATION.md) | Registration, auth/admin and UI now implemented | Package status files and core issue 58 |
-| [Policy/extension exploration](2026-09-19/SPIKE-EXTENSIONS.md) | Main design delivered; obsolete link APIs | Policies, plugins and extensions guides |
-| [Extension model review](2026-09-19/SPIKE-EXTENSION-MODEL.md) | Historical design rationale | Current extension contract |
-| [Usability review](2026-09-19/USABILITY-REVIEW.md) | Predates trusted execution and retired packages | New benchmark observations |
-| [Management security](2026-09-19/MANAGEMENT-SECURITY.md) | Describes the removed core link-management API | No current core replacement |
-| [Lambda compile spike](2026-09-19/SPIKE-LAMBDA-COMPILE.md) | Proposal declined 2026-09-19 in favour of one Node deployment per project | Open decisions; AWS and Vercel guides |
-| [Completed decisions (2026-09-20)](2026-09-20/OPEN-DECISIONS-COMPLETED.md) | Closed questions, the completed monorepo migration and the 2026-09-19 source baseline, moved out of open decisions (#242) | Current open decisions and release readiness |
-
-Open proposals stay outside the archive with their status stated at the top.
-The accepted trust-by-default decision remains directly linked from the current
-security contract; its `SPIKE-` filename is retained for existing references.
-Security audits and operational evidence remain available: “implemented” never
-means their unperformed review, deployment or recovery work is complete.

package/docs/policies/agents.md

@@ -1,182 +0,0 @@
-# The `agents` policy
-
-Denies or reports requests by their `User-Agent`, before anything else runs.
-It is the cheapest refusal in the request chain (`agents`, then `throttle`,
-then the cache lookup), so a denied crawler never counts against a quota,
-never reaches the sandbox and never fills the origin cache.
-
-```yaml
-version: "1"
-policies:                        # project defaults, or per route under routes.<pattern>.policies
-  agents:
-    deny: [ai-crawlers]          # bundled list names, or project-relative .json files
-    allow: [monitoring]          # allow always wins over deny
-    denyPatterns: ["^curl/"]     # linear-time regex subset, matched case-insensitively
-    allowPatterns: ["^Mozilla/5\\.0 \\(compatible; Googlebot"]
-    denyEmpty: false             # deny a missing or blank User-Agent
-    status: 403                  # 400-599
-    mode: enforce                # enforce | report
-```
-
-The `hardened` profile sets `deny: [ai-crawlers]` with status 403. A route can
-override any key, or set `agents: false` to switch the policy off for itself.
-
-## Semantics
-
-- Matching is against the `User-Agent` request header only and is
-  case-insensitive. Nothing else about the request (address, path, other
-  headers) takes part.
-- Evaluation order: if any `allow` list or `allowPatterns` entry matches, the
-  request passes and nothing is logged. Otherwise the first `deny` list (in
-  the order written) or `denyPatterns` entry that matches denies it.
-- `urlcode audit`, `test` and `benchmark` send generated probes as
-  `Mozilla/5.0 (compatible; RouteProbe/0.1)`;
-  a fixture may set its own `user-agent` header. Upstream lists include
-  short unanchored names (the `ai-crawlers` list carries `Code`, which
-  matches any agent containing that word), so a custom allow pattern for
-  your own tooling is worth declaring before denying a whole list.
-- `denyEmpty: true` denies a request with no `User-Agent`, or one that is only
-  whitespace. Allow rules cannot match an empty header, so this always wins
-  for empty headers.
-- A denial answers `status` (default 403) with `content-type:
-  text/plain; charset=utf-8`, `cache-control: no-store` and the body
-  `Forbidden\n`. Response policies that also run at request time (cache,
-  compression) are skipped on the early response; `security` headers are
-  still applied.
-- Every denial is logged as `{ event: 'agents', route, list, outcome }`, where
-  `list` is the list name, the file path as written in YAML, `pattern` for
-  `denyPatterns`, or `empty` for `denyEmpty`. The raw header value is never
-  logged: a `User-Agent` is attacker-controlled text.
-- `mode: report` logs the same event with `outcome: 'reported'` and never
-  denies. Run a new deny list in report mode for a release, read the log, then
-  switch to `enforce`.
-
-## Allow before deny
-
-Broad deny lists without an explicit allow are the most common self-inflicted
-outage in this space. `crawlers` contains every search engine; `ai-crawlers`
-contains agents some operators want (for example `ChatGPT-User` or
-`Applebot-Extended` when a site chooses to appear in AI search). Keep an
-explicit `allow` or `allowPatterns` entry for the agents you depend on, and
-anchor it: `^Mozilla/5\.0 \(compatible; Googlebot` cannot be satisfied by a
-`Googlebot` token dropped in the middle of an unrelated string.
-
-Matching is string matching. A client can claim any `User-Agent`, and the
-genuine search crawlers publish the way to check a claim (reverse DNS for
-Googlebot, bingbot and Applebot; the IETF `web-bot-auth` HTTP Message
-Signature drafts for newer agents). That verification needs network calls and
-vendor data, so it belongs in a [host plugin](../archive/2026-09-19/SPIKE-EXTENSIONS.md) that
-runs after this policy, not in the runtime.
-
-## Bundled lists
-
-| Name | Contents | Upstream | Licence | Refresh |
-| --- | --- | --- | --- | --- |
-| `ai-crawlers` | every agent in `robots.json` (AI training, AI search and assistant crawlers) | [ai-robots-txt/ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) | MIT | pinned tag in `scripts/sync-agent-lists.ts` |
-| `crawlers` | every crawler, bot and automated client in `crawler-user-agents.json` | [monperrus/crawler-user-agents](https://github.com/monperrus/crawler-user-agents) | MIT (revisions after 2016-11-07 only) | same |
-| `seo` | `crawlers` entries tagged `seo` (backlink and rank trackers such as AhrefsBot, SemrushBot, MJ12bot) | derived from crawler-user-agents | MIT | same |
-| `monitoring` | `crawlers` entries tagged `monitoring` (UptimeRobot, Pingdom, StatusCake, ...) | derived from crawler-user-agents | MIT | same |
-
-Each list lives in `data/agents/<name>.json` as
-`{ name, description, source: { repository, url, license, tag, commit, file, fetchedAt }, entries }`
-with one entry per pattern: `{ name, pattern, source, sourceRevision, addedAt }`.
-The generated `data/agents/index.js` mirrors the `[name, pattern]` pairs so the
-policy module (which also runs inside the Cloudflare Worker) needs no
-filesystem. The upstream licences are reproduced verbatim under
-`data/agents/LICENSES/` and named in `NOTICE`, as Apache-2.0 section 4(d)
-requires. `urlcode audit` and the runtime's policy inventory report the list
-names, pattern counts and the pinned revision each list was built from, so a
-deploy carries a known list version and a rollback rolls the list back too.
-
-### Refreshing
-
-```sh
-node scripts/sync-agent-lists.ts            # fetch pinned upstreams, validate, write data/agents/ (also: npm run sync:agents)
-node scripts/sync-agent-lists.ts --check    # exit 1 when the committed files are stale
-```
-
-The script fetches each upstream at the tag and commit pinned in its `sources`
-table, normalises entries to the schema above (an ai.robots.txt agent name
-becomes an escaped literal pattern; a crawler-user-agents pattern is kept and
-named by its literal prefix), validates every pattern against the subset below
-(rewriting `{n,}` to `{n,64}`, and dropping and printing anything else),
-preserves `addedAt` from the previous file or the upstream `addition_date`, and
-writes the JSON lists, the index and the licence copies. To move to a newer
-upstream release, change the `tag` and `commit` pins and rerun; the revision
-appears in every entry and in the audit output. Refresh through an ordinary
-pull request so the list diff is reviewed like code. Behind an HTTPS proxy,
-set `NODE_USE_ENV_PROXY=1` so `fetch` honours `HTTPS_PROXY` and
-`NODE_EXTRA_CA_CERTS`.
-
-## Project lists
-
-`deny` and `allow` also accept a project-relative path ending in `.json`:
-
-```yaml
-policies:
-  agents:
-    deny: [agents/deny.json]
-```
-
-The file is either an array of entries or `{ "entries": [...] }`, each entry
-`{ "name": "curl", "pattern": "^curl/" }` (`name` optional; the other fields
-of the bundled schema are ignored). It must stay inside the project, hold at
-most 4096 entries and pass the same pattern validation as YAML patterns. The
-log names the path as written. The file is read once at activation; the
-Cloudflare build embeds its entries in the artifact so the Worker never reads a
-file.
-
-## The pattern subset
-
-Patterns in `denyPatterns`, `allowPatterns` and every list file are validated
-at activation. A pattern outside the subset fails activation with a
-`ConfigError` naming the route, the key and the reason, so a project cannot
-turn the matcher into a denial-of-service vector by editing YAML. Allowed:
-
-- anchors `^` and `$`; literals; `.`
-- escapes: `\d \w \s \D \W \S \b \B \t \n \r \f \v \0`, `\xHH`, `\uHHHH`, and
-  a backslash before any punctuation (`\.`, `\/`, `\(`, `\-`, ...)
-- character classes `[...]` and `[^...]` with ranges and the escapes above
-- groups `(...)` and `(?:...)`, and alternation `|`
-- quantifiers `*`, `+`, `?`, `{n}` and `{n,m}` with `m <= 64`, on a single
-  atom (a literal, escape, class or `.`); `?` may also follow a group that
-  contains no quantifier, for optional words such as `(?:bot)?`
-- at most 256 bytes
-
-Rejected: backreferences (`\1`, `\k<name>`), lookahead and lookbehind, named
-groups, unicode property escapes, `\c` control escapes, nested character
-classes, `{n,}` and bounds above 64, lazy or stacked quantifiers (`+?`, `**`),
-quantifiers on anchors, and `*`, `+` or `{n,m}` on a group (so `(a+)+` and
-`(ab)*` fail). Each list is compiled into one alternated `RegExp` with the `i`
-flag, so a request costs one pass per list rather than one per pattern.
-
-## robots.txt
-
-Denying an agent is not the same as asking it to stay away. Well-behaved
-crawlers read `/robots.txt` (RFC 9309) before fetching anything, and the AI
-crawlers in `ai-crawlers` are the ones that upstream tracks as respecting or
-ignoring it. Generate one from the same bundled list with
-[`site.robots`](../SITE.md) so the file and the policy cannot drift apart:
-
-```yaml
-site:
-  robots:
-    disallow: [ai-crawlers]
-policies:
-  agents:
-    deny: [ai-crawlers]
-```
-
-This serves `/robots.txt` with one `User-agent:` line per list entry followed
-by `Disallow: /`, then `User-agent: *` / `Allow: /`. A hand-written
-`/robots.txt` `respond` route still works and takes precedence over the
-generated one. The `agents` policy then enforces the same decision for clients
-that ignore the file.
-
-## Targets
-
-| Target | Support |
-| --- | --- |
-| self-hosted (`urlcode serve`) | native |
-| Vercel, AWS | native |
-| Cloudflare | compiled: the build validates the policy, embeds the effective configuration (and any project list entries) in the artifact, and the Worker compiles it at startup without filesystem access |

package/docs/policies/cache.md

@@ -1,152 +0,0 @@
-# `policies.cache`
-
-Named HTTP caching strategies, enforced on the host. A route picks one
-strategy from a fixed catalogue; every row is a pattern with a name outside
-this project and a defined header output, so a browser, a CDN and the
-runtime's own origin memory cache all read the same thing. Explicit fields
-(`maxAge`, `staleWhileRevalidate`, `staleIfError`, `cdnMaxAge`, `originTtl`)
-override what a strategy implies.
-
-```yaml
-policies:
-  cache: { strategy: revalidate }      # project default
-routes:
-  /feed:
-    function: { source: feed.mjs }
-    policies:
-      cache: { strategy: swr, maxAge: 30, staleWhileRevalidate: 300, vary: [Accept-Language] }
-```
-
-A route may also write `cache` directly, next to `function` or `middleware`,
-instead of nesting it under `policies`. The compiler expands it to
-`policies.cache` before anything else reads the project ([policies](../POLICIES.md)),
-so it is pure sugar over the form above and is refused alongside a direct
-`policies.cache` on the same route:
-
-```yaml
-routes:
-  /feed:
-    function: { source: feed.mjs }
-    cache: { strategy: swr, maxAge: 30 }   # same as policies: { cache: {...} }
-```
-
-## Strategies
-
-| `strategy` | Emitted headers | Origin memory cache | Typical use |
-|---|---|---|---|
-| `no-store` | `Cache-Control: no-store` | off | personalized, secret-bearing |
-| `revalidate` | `Cache-Control: no-cache`, `ETag` (a strong SHA-256 tag is computed for results without one); answers `304` to `If-None-Match`, and to `If-Modified-Since` when the result carries `Last-Modified` | off | HTML, anything that must be fresh but is cheap to validate |
-| `public` | `Cache-Control: public, max-age=N` (`maxAge` required) | on only when `originTtl` > 0 | stable API answers, feeds |
-| `immutable` | `Cache-Control: public, max-age=31536000, immutable` (RFC 8246; `maxAge` overrides the year) | off | content-hashed URLs only; refused elsewhere unless `force: true` |
-| `swr` | `Cache-Control: public, max-age=N, stale-while-revalidate=M` (RFC 5861; both required) | on: fresh for `originTtl` (default `maxAge`), then stale served once | hot functions, link previews |
-| `sie` | as `swr` plus `stale-if-error=K` (`staleIfError` required; `staleWhileRevalidate` optional) | on, as `swr` | keep answering during an upstream failure (headers only at the origin; see below) |
-| `micro` | `Cache-Control: no-store` to clients | on: `originTtl` default 1 s, at most 5 s unless `force: true` | the NGINX micro-cache: absorb a thundering herd without changing what a browser sees |
-| `cdn-only` | `Cache-Control: no-store` plus `CDN-Cache-Control: max-age=N` (RFC 9213; `cdnMaxAge` required) | off | let the CDN cache while browsers do not |
-| `private` | `Cache-Control: private, max-age=N` (`maxAge` required) | off | per-user data a browser may keep |
-
-`immutable` accepts a route whose pattern has a segment with eight or more
-hex characters (`/app.3f2a9c1d.js`) or a parameter named like a digest
-(`{hash}`, `{digest}`, `{sha}`, `{version}`, `{build}`, `{rev}`,
-`{fingerprint}`). Anything else fails configuration with the route named
-unless `force: true`.
-
-## Who owns `Cache-Control`
-
-Explicit beats strategy, in this order:
-
-1. A `response.headers.cache-control` declared in YAML on the route is kept
-   as written; the strategy does not touch it.
-2. An asset handler's `cacheControl` (`page`, `download`, `static`) is kept
-   when the cache policy is only inherited from the project or a profile.
-   When the route itself declares `policies.cache`, the strategy overrides it.
-3. A handler result that already says `no-store` or `private` is kept and is
-   never stored: a personalized answer under a `public` route stays private.
-4. Otherwise the strategy's header replaces whatever the handler emitted.
-
-`Vary` is merged with the declared `vary` names (no duplicates, `*` left
-alone), for every strategy, so the origin key and the wire header agree.
-Compression adds `Accept-Encoding` to `Vary` after this policy.
-
-## Origin memory cache
-
-On for `swr`, `sie`, `micro`, and `public` with `originTtl` set. It is an
-origin-side micro-cache, so a client's own `Cache-Control: no-cache` or
-`max-age=0` request directive does not bypass it (RFC 9111 §5.2.1 lets an
-origin decide that); conditional and range requests do, as below. Rules:
-
-- Only `GET` and `HEAD` are looked up; only `GET` results are stored, and a
-  `HEAD` hit serves the `GET` entry's headers with an empty body and the
-  entry's `Content-Length`. A route whose `methods` exclude `GET` has the
-  origin cache off.
-- Only statuses in `statuses` (default `200, 301, 302, 404, 410`) are stored.
-- A request carrying `If-None-Match`, `If-Modified-Since`, `If-Match`,
-  `If-Unmodified-Since` or `Range` is never answered from the origin cache:
-  a stored entry is a full `200` representation and the handler owns
-  validators and ranges, so such requests always reach it (an asset route
-  keeps its own `304`, `412` and `206` answers).
-- Never stored: results carrying `Set-Cookie`; routes declaring `secrets`;
-  results whose handler `Cache-Control` says `private` or `no-store`; bodies
-  larger than `maxBytes` (default 1 MiB).
-- Key: route pattern, request path, query string and the values of the
-  declared `vary` request headers. The method is not part of the key so
-  `HEAD` shares the `GET` entry.
-- Hits carry `Age` (RFC 9111). No non-standard headers are added.
-- Concurrent misses for one key coalesce: the first request reaches the
-  handler, up to 64 others wait for its result, and any beyond that proceed
-  to the handler themselves. If the fill fails, waiters fall through to the
-  handler rather than receiving the error.
-- Bounds: `maxEntries` per route configuration (default 10000) and 64 MiB of
-  bodies across the whole runtime; the least recently used entry is evicted
-  first. The store belongs to one runtime instance and is dropped on close and
-  reload, so a deploy never serves the previous code's output.
-- Stored entries hold the handler's headers as they were after YAML
-  `response.headers` and this policy ran, before security headers and
-  compression. Those run again on every hit, so a hit is compressed and
-  hardened the same way as a miss. Bodies are stored uncompressed.
-
-### `swr` at the origin: stale served once, next request refreshes
-
-A policy has no handle to the route's handler, so it cannot revalidate in the
-background. The origin-side approximation is: a request that finds an entry
-past `originTtl` but within `staleWhileRevalidate` is answered from the stale
-entry immediately, and the entry is flagged so the next request for that key
-goes to the handler and replaces it. If that refresh fails the flag is
-cleared and the stale entry may be served once more, as long as it is within
-the window. Beyond `max-age + stale-while-revalidate` nothing is served from
-memory. Clients and CDNs that honour RFC 5861 do their own background
-revalidation from the emitted header.
-
-### `sie` limitation
-
-`stale-if-error` is header-only at the origin. Policies observe errors but
-cannot replace a thrown error with a result (the runtime rethrows), so a
-handler failure is not answered from a stale entry by this runtime. Downstream
-caches honouring RFC 5861 still serve stale on a 5xx. Origin-side
-stale-if-error needs the plugin API or a runtime change that lets a policy's
-error hook return a result.
-
-## Targets
-
-| Target | Support | Note |
-|---|---|---|
-| node | native | headers and origin cache per runtime instance |
-| vercel | native | headers; the origin cache is per function instance, so hit rates depend on instance reuse |
-| aws | native | headers; the origin cache is per Lambda instance |
-| cloudflare | refused | the Worker build has no policy runtime; activation fails with the route named |
-
-## Diagnostics
-
-`describe()` (in `urlcode test`, the readiness plan and plugin activation)
-reports `strategy`, the emitted `cacheControl` string (or `explicit response
-header` / `asset handler` when something else owns it), `cdnCacheControl`,
-`originTtl`, `staleWhileRevalidate`, `staleIfError`, `vary` and whether the
-origin cache is `on`.
-
-Log events, through the runtime's `log`:
-
-```json
-{ "event": "cache", "route": "/feed", "outcome": "hit" }
-```
-
-`outcome` is `hit`, `stale`, `miss` or `store`. The route is the configured
-pattern, never the request path or key.