@jimhoyd/urlcode 0.3.0

package/docs/SITE.md

@@ -0,0 +1,150 @@
+# Site conventions
+
+The optional top-level `site` block generates the small well-known files a
+site is expected to serve. Every key is off unless declared, and each declared
+key becomes one ordinary native route (`respond` or `page`) that is merged into
+the route table before compilation. Everything downstream is unchanged: the
+route appears in `urlcode routes` with `generated: "site.<key>"`, the audit
+generates fixtures for it, host policies apply to it and every deployment
+target compiles it like a route the project wrote by hand. `site` is accepted
+only in the entry `urlcode.yaml`, not in included files.
+
+```yaml
+version: "1"
+site:
+  robots:
+    disallow: [ai-crawlers, /admin]
+    allow: [/admin/public]
+    sitemap: true
+  sitemap:
+    exclude: [/drafts/*]
+    changefreq: weekly
+  favicon: public/favicon.svg
+  securityTxt:
+    contact: [mailto:security@example.com]
+    expires: "2027-01-01T00:00:00Z"
+    policy: [https://example.com/security-policy]
+    preferredLanguages: [en, fr]
+  llms: llms.txt
+routes:
+  /: {page: {file: public/index.html}}
+```
+
+The [cookbook](../examples/cookbook/urlcode.yaml) declares `robots`, `favicon`,
+`securityTxt` and `llms` with request fixtures; `sitemap` is exercised by the
+unit tests because it needs a public origin (below).
+
+## Precedence with declared routes
+
+A route the project declares at a generated path always wins. The generated
+route is dropped and the activation log records
+`{"event":"site","key":"robots","path":"/robots.txt","status":"shadowed"}`.
+This is how a project keeps a hand-written `robots.txt` while still using
+`site.favicon`, or migrates one file at a time.
+
+## The public origin
+
+`robots.txt`'s `Sitemap:` line and every `<loc>` in `sitemap.xml` are absolute
+URLs, so the runtime needs to know the origin the site is served from. It
+never guesses from a request: a generated file is a fixed response compiled at
+activation, and a `Host` header is client-controlled. The origin is the
+operator's `--origin https://links.example` flag on `serve`, `dev`,
+`validate`, `test`, `routes`, `audit`, `benchmark` and `build`, or the
+`origin` option of `startServer`/`createRuntime`/`buildCloudflare`.
+
+- `site.robots.sitemap: true` without an origin omits the `Sitemap:` line and
+  logs `{"event":"site","key":"robots","severity":"info",...}` at activation.
+- `site.sitemap` without an origin **refuses activation**: a sitemap of relative
+  URLs is invalid under the protocol, so there is no useful degraded form.
+
+## Keys
+
+### `robots` → `/robots.txt` (RFC 9309)
+
+| Field | Meaning |
+| --- | --- |
+| `disallow` | bundled agent list names (`ai-crawlers`, `crawlers`, `seo`, `monitoring`, the same lists as the [`agents` policy](policies/agents.md)) and/or paths starting with `/` |
+| `allow` | the same shapes, emitted as `Allow:` |
+| `sitemap` | `true` appends `Sitemap: <origin>/sitemap.xml` when the origin is known |
+| `extra` | literal lines appended verbatim (comments, `Crawl-delay`, ...) |
+
+List names become one group of `User-agent:` lines followed by `Disallow: /`
+(or `Allow: /`). Paths go under `User-agent: *`; with no paths that group is
+`Allow: /`. A list entry whose name is not usable as a product token (it has
+spaces or pattern metacharacters) is skipped and counted in an info log line.
+The result is served as `text/plain; charset=utf-8`.
+
+```
+User-agent: GPTBot
+User-agent: ClaudeBot
+Disallow: /
+
+User-agent: *
+Disallow: /admin
+Allow: /admin/public
+
+Sitemap: https://links.example/sitemap.xml
+```
+
+### `sitemap` → `/sitemap.xml` (sitemaps.org protocol 0.9)
+
+`true`, or `{ exclude, changefreq, priority }`. The sitemap lists every active
+literal GET route that serves HTML: a `page` whose file is `.html`/`.htm` or
+whose `contentType` is `text/html`; a `respond` route whose `Content-Type`
+response header is `text/html`; and the `.html` files under a `static` mount
+(the mount's `index` file is listed as its directory URL). Excluded:
+parameterized routes, redirects, functions, downloads and stored links,
+disabled or expired routes, `/robots.txt` and `/sitemap.xml`, any route whose
+`response.headers` set `X-Robots-Tag` containing `noindex`, and anything an
+`exclude` pattern matches (an exact path, or a prefix ending in `/*`).
+`lastmod` is the file's modification date for asset routes and omitted
+otherwise. URLs are sorted and XML-escaped; the response is
+`application/xml; charset=utf-8`.
+
+More than 50,000 URLs, or a document over the 1 MiB declared-response limit,
+refuses activation with a message saying to generate the file at build time
+and serve it as a static asset instead; the runtime does not split or index
+sitemaps.
+
+### `favicon` → `/favicon.ico`
+
+A project-relative `.ico`, `.svg` or `.png` file, served as a `page` route
+with `image/x-icon`, `image/svg+xml` or `image/png` and
+`cacheControl: public, max-age=3600` (the closest value in the
+[allowed vocabulary](ASSETS.md)). The route has the usual asset semantics:
+ETag, Last-Modified, conditional requests and ranges.
+
+### `securityTxt` → `/.well-known/security.txt` (RFC 9116)
+
+| Field | Rule |
+| --- | --- |
+| `contact` (required) | one or more `mailto:`, `tel:` or `https:` URIs |
+| `expires` (required) | UTC ISO timestamp; must be in the future at activation; more than a year away logs a warning (the RFC recommends less than a year) |
+| `policy`, `acknowledgments`, `canonical` | `https:` URIs |
+| `encryption` | `https:`, `dns:` or `openpgp4fpr:` URIs |
+| `preferredLanguages` | language tags, emitted as one comma-separated line |
+
+Fields are emitted in the RFC's order (Acknowledgments, Canonical, Contact,
+Encryption, Expires, Policy, Preferred-Languages), one value per line. The
+file is unsigned; add a signed copy as a declared route if you need one. The
+`.well-known` segment is an ordinary route segment (only `.` and `..` are
+refused).
+
+### `llms` → `/llms.txt`
+
+A project-relative text file served as a `page` route with
+`text/plain; charset=utf-8` and the default `no-cache`.
+
+## Per-target support
+
+| Target | `robots`, `sitemap`, `securityTxt` (`respond`) | `favicon`, `llms` (`page`) |
+| --- | --- | --- |
+| self-hosted, Vercel, AWS | served | served |
+| Cloudflare | compiled into the artifact (`build --origin` for absolute URLs) | refused at build time like any `page` route: the target has no asset binding; serve them from the platform's static assets |
+
+## Not in this release
+
+No per-route `noindex` field (use `response.headers: {X-Robots-Tag: noindex}`,
+which the sitemap honours), no sitemap index or split files, no `humans.txt`,
+no signed `security.txt`, and no project `.json` agent lists in `robots`
+(bundled names only).

package/docs/SPECIFICATION.md

@@ -0,0 +1,232 @@
+# Implemented 0.1 contract
+
+This document and [JSON Schema](../schemas/urlcode.schema.json) describe
+0.2.0. `version: "1"` is the stable project-format contract for the 0.1 release
+line. Later planned features are rejected until implemented.
+
+## Files and validation
+
+`urlcode.yaml` contains `version`, `routes`, and optional `includes` (an explicit
+list of project-relative YAML files). Included files have the same version/routes
+shape and cannot recursively include files. Duplicate paths across files fail.
+See [organization examples](ORGANIZATION.md) for one-file, multi-file and mixed
+layouts. All references resolve from the project root, not the including file.
+File references resolve inside the project; escaping symlinks fail. No glob,
+remote config or arbitrary infrastructure configuration.
+
+YAML 1.2 JSON-compatible values only: string mapping keys, finite numbers,
+booleans and null. No duplicate keys, aliases, anchors, tags, merge keys,
+multiple documents, reserved prototype keys or nesting of 40+ levels. Unknown
+schema fields fail. Files are limited to 32 MiB each, 256 includes and 100,000
+routes total, with a 64 MiB aggregate YAML source cap. Loading runs in a worker
+with a 256 MiB old-generation heap, a 10-second wall deadline and at most two
+concurrent loads per Node isolate. These are not a total process RSS bound.
+Route compilation yields every 64 routes and checks a 10-second cooperative
+deadline; individual synchronous operations are not preempted. At most 1,000 parameterized routes and 1,024 distinct input schemas.
+
+## Routes
+
+Keys are absolute case-sensitive paths. Trailing slashes are significant.
+Parameters occupy whole segments, e.g. `/p/{id}`, with distinct identifier names.
+Each parameter matches exactly one nonempty segment, never across `/`; it is not
+greedy. No regex paths, host matching or dot segments. Only static directory mounts
+support a terminal `/*` wildcard with an otherwise literal path. Route keys cannot contain
+percent encoding, spaces, backslashes or query strings. Path length is limited
+to 2,048 characters and 32 segments. `/_urlcode` is reserved.
+
+One handler per route: `function`, `redirect`, `page`, `static`, `download`, `respond` or `link`.
+See [asset configuration](ASSETS.md) for file handlers. Optional properties:
+
+- `methods`: unique HTTP methods; default GET and HEAD. Explicit lists are exact;
+  adding GET does not implicitly add HEAD. Wrong method returns 405 plus Allow.
+- `enabled`: false returns 404, the same as unknown paths.
+- `expires`: UTC ISO timestamp (`...ssZ` or `...ss.sssZ`); expired routes return 410.
+- `description`: optional authoring metadata.
+- `middleware`: ordered list of up to 16 `{source, export?}` modules wrapping any
+  handler. See [middleware](MIDDLEWARE.md) for the portable contract.
+- `parameters`, `env`, `secrets`: inputs and explicit binding references.
+
+Literal paths win; parameter routes with more literal segments win next;
+static mounts follow, longest prefix first. A missing file in the selected mount
+returns 404 without falling back to a shorter mount.
+Equally specific overlapping patterns fail even if methods differ. Match a route
+before checking its methods; do not fall back to a less specific route for 405.
+Requests decode the path once; invalid UTF-8/percent encoding, encoded slashes or
+backslashes, control characters and dot segments return 400. Query values decode
+once. Incoming query data is not automatically forwarded.
+
+See [route matching and new links](ROUTING.md) for examples, precedence, wildcard
+limits, reload behavior and the distinction between YAML routes and live link data.
+
+The optional top-level `site` block (entry file only) generates native routes
+for site conventions: `robots` → `/robots.txt`, `sitemap` → `/sitemap.xml`,
+`favicon` → `/favicon.ico`, `securityTxt` → `/.well-known/security.txt` and
+`llms` → `/llms.txt`. Each is an ordinary `respond` or `page` route merged in
+before compilation and counted by `routes`/`audit`; a declared route at the same
+path wins and the generated one is logged as shadowed. Absolute URLs come from
+the operator's `--origin`; `sitemap` refuses activation without one. See
+[site conventions](SITE.md).
+
+## HTTP request/response configuration
+
+Routes accept `request.body` validation and `response.headers` overrides. The
+`respond` handler serves declared text/JSON with a status without running code.
+See [HTTP configuration](HTTP.md) for the exact supported fields, precedence,
+security restrictions and examples.
+
+## Policies
+
+Optional top-level `policies` and `profiles` keys, and `routes.<path>.policies`,
+declare host-enforced behavior around a route: `throttle`, `agents`, `security`,
+`compression` and `cache`. All are off unless declared; a route's keys merge
+over the project's, `false` disables one, and a target that cannot enforce a
+policy refuses activation naming the route. The five policies are implemented
+for the self-hosted server; Vercel and AWS accept `agents`, `security`, `cache`
+and route-partitioned `throttle`; the Cloudflare build compiles `agents` and
+`security` only. See [policies](POLICIES.md) for the pipeline position, merge
+rules and the per-target table, and [plugins](PLUGINS.md) for the host hook API
+operators pass in code.
+
+## Inputs
+
+Declare each path placeholder as a required string. Query/header inputs may be
+string, integer, number or boolean. Query arrays declare scalar `items`; repeated
+keys retain order (maximum 100 values). Header names are case-insensitive.
+Duplicate scalar query/header inputs return 400. Required missing inputs return
+400; defaults apply only to absent inputs. Empty strings are present values.
+
+Supported validation: `type`, scalar `enum`, `default`, string `minLength`/
+`maxLength`, numeric `minimum`/`maximum`, query array `items` and `maxItems`.
+Regular-expression patterns, structured bodies, cookies, nested inputs and
+OpenAPI `style`/`explode` fields are not implemented. This uses a documented
+OpenAPI-like input subset; it is not an OpenAPI document or full JSON Schema
+input vocabulary. String limits are at most 8,192 characters.
+
+Integers use `-?(0|[1-9][0-9]*)` and must be safe JavaScript integers. Numbers
+allow the same grammar plus a fractional suffix; no exponent, plus sign, leading
+zeros, whitespace, NaN or infinity. Booleans are exactly `true` or `false`.
+Unknown query keys are ignored unless explicitly passed by a redirect.
+
+## Redirects
+
+`redirect.url` is an absolute HTTP(S) URL with literal scheme/host and no embedded
+credentials or whitespace/control characters. `{pathInput}` placeholders are
+allowed only in the destination pathname and encoded as single components.
+No environment/secret interpolation. Status defaults to 302; allowed values are
+301, 302, 303, 307 and 308.
+
+`redirect.query.map` maps output keys to `{from: path|query|header, name: input}`
+references to declared inputs. Typed/defaulted values are used. Arrays produce
+repeated output keys. Absent optional inputs are omitted.
+`redirect.query.pass` is false or an explicit allowlist; unrestricted `true` is
+not supported. Conflicts between destination keys, maps and passthrough fail.
+Declared passthrough inputs use validated/defaulted values; undeclared allowlisted
+keys preserve repeated values. Headers are forwarded only through explicit maps.
+
+## Stored links
+
+The entry `urlcode.yaml` must opt in with `dynamicLinks: true` (default false).
+Included files cannot set this project-level flag. Parameterized redirects and
+functions do not require it.
+
+`link: {collection: links, code: {from: path, name: code}}` resolves a declared
+path input against an operator-bound store. GET/HEAD only. The logical collection
+is portable; file paths and store credentials are external deployment bindings.
+No general storage capability is exposed to guest code. See [dynamic links](DYNAMIC-LINKS.md)
+for validation, persistence, mutation, expiry and read-after-write behavior.
+
+## Functions
+
+```yaml
+version: "1"
+routes:
+  /hello/{name}:
+    parameters:
+      - name: name
+        in: path
+        required: true
+        schema: {type: string, minLength: 1}
+    function:
+      source: functions/hello.mjs
+      export: default
+      args:
+        name: {from: path, name: name}
+    env:
+      GREETING: {value: Hello}
+```
+
+```js
+export default function hello(request, { args, env }) {
+  return Response.json({ message: `${env.GREETING}, ${args.name}!` });
+}
+```
+
+ES modules only (`.mjs` or `.js`, independent of Node package settings).
+TypeScript is not included. `export` defaults to `default`. Functions execute
+inside QuickJS/WASM, never through Node imports. Only relative `.js`/`.mjs`
+project imports are supported, with a snapshotted dependency graph. No bare/npm,
+Node built-in, remote, dynamic source imports or `import.meta`. Runtime-created
+imports remain restricted to the route's middleware and handler dependency graphs; there is no fallback.
+Source limits: 128 modules, 1 MiB per module, 4 MiB total.
+
+The current guest API is a **text/JSON subset**, not the complete native Fetch
+API: Request `url`, `method`, `headers`, `text()`, `json()`; Headers append/set/
+delete/get/has/entries/getSetCookie; Response constructor with string/null body,
+`status`, `headers`, `ok`, `text()`, `json()`, static `json()` and `redirect()`.
+Requests decode body bytes as UTF-8. Binary/streaming bodies, URL helpers,
+fetch/WebSocket, crypto and filesystem are not exposed. Promise/async and
+bounded timers (128 pending per invocation) work inside the guest. Unsupported
+APIs fail; they never execute on the host. Do not claim full browser/Node API parity.
+
+Context contains `inputs.path/query/header`, `args`, `env`, `secrets`. Arguments
+may be scalar literals, input references, `{env: alias}` or `{secret: alias}`.
+Bindings use `{value: "literal"}`, `{env: EXTERNAL_NAME}` or `{secret: logical_name}`.
+Literal non-secret values need no grant. Every external environment or secret
+binding is denied unless an operator policy grants that exact name to the route
+and matches the SHA-256 of the current configuration/source snapshot.
+A project cannot grant itself capabilities. See [policy setup](FUNCTION-SECURITY.md).
+Missing bindings also reject activation. Inspection parses source without running it.
+
+Development may read `.env.local`; process values win. Serving never reads it.
+Dotenv supports single-line NAME=value, paired single/double quotes, blank lines
+and full-line comments, without expansion/escapes/shell execution. Loading a
+value does not authorize exposing it to a function; the policy still applies.
+
+Every invocation has a fresh guest heap and module state. No cross-request
+counters, cached secrets or prototype mutation. QuickJS heap limit is 32 MiB,
+stack limit 512 KiB; outer worker and deployment limits are additional defenses,
+not a claim that total process RSS is capped at 32 MiB. Two workers, no queue;
+saturation returns 503. The independent 5-second deadline terminates a worker
+and returns 504. Generic failures return 502; worker replacement is bounded.
+
+HEAD invokes the handler as HEAD and suppresses body output. Code must guard
+its own application side effects when future brokered integrations are enabled.
+Guest console output is discarded. Functions see the configured public origin,
+not arbitrary Host/forwarded headers. Request/response bodies default to 1 MiB;
+response headers 16 KiB, maximum 256 pairs. Hop-by-hop headers are stripped;
+cookies are preserved individually. Default response cache policy is `no-store`.
+
+No unrestricted host execution option exists. Network integrations and persistent
+state need future explicit, tested capability brokers. Approved secrets can be
+returned by code that receives them; isolation does not automatically enforce
+information-flow rules on authorized inputs. Keep grants narrow and review the
+exact pinned revision. The sandbox still needs independent security review before hostile multi-tenant use.
+
+## Reload and status
+
+`dev` polls project YAML/JSON/JS and `.env.local` every 500 ms, plus declared
+asset files/directories (including binary assets and explicit build directories).
+Asset polling uses file metadata; production assets stay fixed until restart.
+The general source scan excludes common
+build/dependency directories and hidden files. Includes and source dependencies
+must be normal watched files; changes in symlink targets or `node_modules`
+require restart. A candidate fully validates and initializes its functions
+and snapshots its assets before activation. Invalid candidates leave the old snapshot serving. In-flight
+function calls finish on their original snapshot; new requests use the new one.
+Production `serve` is a fixed snapshot; restart/redeploy for code, secret or
+operator-policy changes. Config/code edits invalidate old binding grants.
+
+The health `version` combines route-definition and asset-representation digests,
+not a full artifact digest
+or secret fingerprint. Production release identity should be the Git commit and
+container image digest. See [operations](OPERATIONS.md).