@jimhoyd/urlcode 0.4.2 → 0.4.6

package/docs/SECURITY-AUDIT.md

@@ -1,164 +0,0 @@
-# Security review — 2026-09-18
-
-Scope: trust-model change (docs/SPIKE-DEFAULT-TRUST-MODEL.md). `function`/
-`middleware` routes now run trusted and unsandboxed by default, with
-`sandbox: true` as an explicit per-route opt-in to the isolation this document's
-earlier entries describe. This is a maintainer-decided policy reversal, not a
-finding; it is recorded here because it changes what every earlier entry's
-"guest"/"sandbox" language means going forward.
-
-**What did not change:** the `sandbox: true` execution path — QuickJS/WASM
-isolation, module-graph restriction, fresh heap per call, worker deadline —
-is byte-for-byte the same as every earlier entry describes; nothing in this
-change touched `src/functions.ts`, `src/function-worker.ts` or `src/guest-api.ts`.
-Binding grants are unaffected either way: `env`/`secrets` still reach only a
-route that explicitly declared them and an operator policy pinned to the
-project revision explicitly granted, whether that route is trusted or
-sandboxed.
-
-**What did change, and the resulting residual risk:** a `function`/
-`middleware` route with no `sandbox` field (the common case going forward, and
-every existing project's routes after an upgrade with no YAML change) now runs
-in the host process with full Node access — the isolation earlier entries'
-"remaining gaps" language assumed for *all* guest code no longer applies to
-it. The residual-risk framing in earlier entries ("before exposing hostile
-multi-tenant workloads, obtain independent review of WASM/host boundaries")
-is scoped to `sandbox: true` routes specifically; it was never a claim that
-covered a route that opts out of the sandbox, and after this change most
-routes do exactly that by default. Whether a given project's own function/
-middleware code is safe to trust with full host access is now the project's
-judgment call, not something this runtime's isolation reviews (past or
-future) can speak to. New source review of the dispatch decision itself
-(`src/runtime.ts`, `src/trusted-functions.ts`, `src/policy.ts`'s split
-grant-hashing) accompanies the change; see the pull request that introduced
-`sandbox` for its own description of what was and was not verified.
-
-# Security review — 2026-09-17
-
-Scope: follow-up source review of worker/connection replacement, probe admission,
-request correlation, operational logging, dependency/release supply chain and the
-live-link Node requirement. Regression tests accompany the fixes. Internal review,
-not an independent penetration test.
-
-This is a dated record, not current guidance. The `link` handler, its SQLite
-store and the link-store/management findings below were removed from core in
-a later change; they describe the runtime as it existed on this date, not the
-current one.
-
-## Findings fixed in this revision
-
-| Finding | Impact and evidence | Fix / regression |
-|---|---|---|
-| Function workers latched off permanently after bounded churn | Availability: replacement stopped after three exits in a minute and was never retried, so eight deadline-exceeded requests — reachable from ordinary request input to any function whose runtime depends on its input — disabled every function route for the life of the process. Reproduced against `serve`, which never reloads: `/fast` and readiness stayed 503 indefinitely | Replacement now backs off (250 ms doubling to a 30-second ceiling) and keeps retrying; a completed invocation clears the backoff. Load is shed while a slot is down, and `function_worker` events record each attempt. Regression drives eight deadlines, then asserts the pool serves and readiness returns to 200 |
-| Link-store connections were never replaced | Availability: one operation reaching the five-second deadline, or any abrupt worker exit, terminated the connection with no replacement path, permanently failing that pool while the records themselves were intact on disk | Connections share the same backed-off replacement and emit `link_store_worker` events. Activation still fails closed and is never retried behind the caller. Covered by the existing 22-test live-link suite; the post-activation failure branch has no automated test because the worker could not be crashed deterministically from a test |
-| Health probes bypassed admission control | Resource exposure: probes were answered outside the in-flight budget, an unmetered path on a public listener, and report the configuration digest and route count without authentication | Probes keep a separate bounded budget (16 by default) so they stay available under application saturation without being unmetered. Operations documents keeping them internal |
-| Request correlation broke at the ingress, and logs could not attribute anything | Operability: `x-request-id` was always regenerated, so traces did not survive a proxy hop, and request records carried no method or route, leaving per-route error rates and abuse (including the exhaustion above) undetectable | Opt-in `--trust-request-id` accepts a single safe upstream value; `--request-log detailed` adds the method and the matched route pattern. Pattern and method come from reviewed configuration, never from request text. Regression asserts a spoofed ID is ignored by default and that path, parameter and query text never reach the log |
-| Deployment capacity controls were unreachable from the supported deployment | Availability: workers, deadlines and byte limits existed only in the embedding JS API while the container entrypoint is the CLI, so the supported target was fixed at two function workers with no way to tune | `serve`/`dev` accept `--workers`, `--function-timeout-ms`, `--max-response-bytes`, `--max-body-bytes`, `--max-in-flight` and `--max-in-flight-health`, validated before the listener starts |
-| Live-link tests failed rather than skipped on an unpatched Node build | Signal loss: a current Node 22 release bundling SQLite 3.51.2 turned 21 tests red for an environmental reason, hiding real regressions behind expected noise | The suite skips with the detected version named, one CI job asserts the suite actually runs somewhere, `doctor` reports `liveLinks`, and activation names the detected version |
-| Release artifacts and dependencies were not gated | Supply chain: advisories were checked by hand on a date, and Actions and the container base image floated on mutable tags | `npm audit --omit=dev` fails CI on runtime advisories, Actions and the base image are pinned by SHA/digest, and Dependabot proposes npm, Actions and image updates weekly |
-
-Replacement backs off but does not stop. A cause that keeps recurring keeps the
-instance shedding load with readiness at 503 rather than recovering silently;
-that is an operator signal, not self-healing. Alert on sustained
-`function_worker` and `link_store_worker` restart events.
-
-# Security review — 2026-09-16
-
-Scope: source review of HTTP serving/management, log handling, filesystem and
-configuration loading, sandbox/module/binding boundaries, stored-link pooling,
-scaffolding, package/container configuration and current dependency advisories.
-Local adversarial regression tests accompany fixes. This is an internal review,
-not an independent penetration test or certification. No production systems,
-third-party targets or external accounts were attacked.
-
-## Findings fixed in this revision
-
-| Finding | Impact and evidence | Fix / regression |
-|---|---|---|
-| Asynchronous log sink errors escaped request error handling | Availability: a local Writable reporting an async error terminated the process with an unhandled error event | Shared sink error handling suppresses further writes to a failed sink; synchronous failures also contained. Regression covers both failure modes and listener reuse |
-| Management lacked bounded in-flight HTTP admission and socket inactivity enforcement | Resource exposure: connection limits alone did not bound active pipelined work or stalled output; public listener protections were absent here | Default 32 admitted requests through finish/disconnect, 10-second socket inactivity timeout, overload 503 and idempotent shutdown. Regression stalls an authenticated upload, checks overload, waits for disconnect, then verifies recovery |
-| Development watcher read entire unrelated JSON/JS/YAML files | Local availability: a repository file outside the validated dependency graph could cause excessive allocations or fail watching solely because its contents were unreadable | Watch fingerprints use file metadata; actual configuration/source loading retains validation. Regression starts dev with an unreadable unrelated JSON file |
-| Management had no request-level audit events | Operational visibility: mutations, authentication failures and aborted requests lacked structured records | Redacted events include timestamp, request ID, collection, action, authentication result, status and finish/abort outcome. Tests prove tokens, short codes and destinations are absent |
-| Management accepted normalized dot-segment path aliases | Defense-in-depth: URL normalization admitted alternative endpoint spellings that could disagree with upstream path policies; no authorization bypass was demonstrated | Reject raw path normalization before endpoint dispatch; authentication still precedes path handling |
-
-Management event status 0 means no response headers were sent before disconnect.
-An aborted request may have committed a mutation: reconcile record/version before
-retrying. These logs are best effort, may be dropped under pressure, identify a configured credential ID (or a legacy shared token), and are not a
-tamper-evident journal. Successful store mutations now have separate transactional
-audit records; see [management security](archive/2026-09-19/MANAGEMENT-SECURITY.md). A failed sink needs collector/operator recovery;
-URLCode cannot report failures reliably through the same broken output stream.
-
-## Boundaries checked
-
-- Guest code stays inside QuickJS/WASM, with fresh invocation state, denied host
-  APIs, bounded modules/memory/deadlines and no unrestricted fallback. Import
-  allowlists and external revision-pinned binding grants remain in force.
-- Public serving cannot mutate the native link store. Management is a separate
-  token-protected listener with conditional writes, scoped collection and bounded
-  bodies. YAML requires explicit live-link opt-in; it cannot grant guest storage.
-- SQL uses bound parameters; read/write pools have independent bounded admission,
-  patched-SQLite checks, read-only serving and commit/version tests.
-- Asset/scaffold paths reject traversal and unsafe filesystem references. Static
-  publication is explicit; sensitive-name filtering is not a secret detector.
-- Log output excludes request URLs, bodies, headers, tokens and user exceptions.
-  Management adds safe operational context without logging stored destinations.
-- `npm audit` reported zero known advisories on this date. This does not cover
-  every Node, SQLite, WASM engine, operating-system or container vulnerability.
-
-Existing and new automated tests cover these contracts; they do not constitute a
-proof that the sandbox engine or complete application is vulnerability-free.
-
-## Remaining gaps, prioritized
-
-**Before exposing hostile multi-tenant workloads:** obtain independent review of
-WASM/host boundaries, module loading and resource exhaustion. The
-[independent-review package](SANDBOX-REVIEW.md) defines scope and closure evidence.
-YAML parsing now has an aggregate source cap and a bounded worker with a hard
-wall deadline. Route compilation has a cooperative deadline; process-wide RSS,
-individual host operations and overlapping snapshots still require deployment
-limits. No anonymous code/config upload service is approved by these changes.
-
-**Management remains private:** literal loopback binding, external per-credential
-collection/action scopes, expiry and hot revocation, plus atomic SQLite mutation
-audits are implemented. Legacy shared tokens remain for compatibility. There is
-no public user-account system, MFA/SSO, built-in rate limiter, credential issuance
-service or external tamper-evident archive. See [management security](archive/2026-09-19/MANAGEMENT-SECURITY.md).
-
-**Before claiming operational readiness:** execute sustained mixed-workload soak,
-backup restoration, disk-full, process-kill, proxy timeout and rollback drills on
-the real deployment. Monitor disk/WAL growth, pool failures/rejections, RSS,
-readiness, missing logs and restarts. SQLite is single-host; there is no automatic
-multi-host failover or server-database adapter. Logging retention/rotation is owned
-by the external collector and still needs deployment recipes and verification.
-
-**Post-0.1 release hardening:** establish a private vulnerability reporting/support
-policy and patch response ownership before managed or hostile multi-tenant use.
-Release artifacts need immutable image and
-dependency identities, upstream vulnerability monitoring and a reviewed update
-process. At the audit cutoff, CI actions/base images used mutable version tags. The
-repository-governance follow-up pins them and enables dependency maintenance,
-secret protection, CodeQL and private reporting; see [governance](../GOVERNANCE.md).
-CI now also gates runtime advisories with `npm audit --omit=dev`. A manual main-only
-signed candidate/SBOM workflow is now defined; see
-[release security](RELEASE-SECURITY.md). A tagged release publishes to npm with provenance. The 0.1.0 release is licensed under
-Apache-2.0; the license does not close the independent-review gates.
-
-**Application responsibility:** HTML/JS assets are active browser content; choose
-appropriate CSP, cookie flags, authorization and cache policy. Granted secrets
-can intentionally be returned by code receiving them. Filename filters cannot
-identify secrets stored under innocent names. Operator directories/volumes must
-remain protected from other host processes; filesystem checks do not protect
-against a privileged host attacker racing mutations.
-
-See [release gates](RELEASE-READINESS.md), [function security](FUNCTION-SECURITY.md),
-[operations](OPERATIONS.md) and [resilience](RESILIENCE.md). Free-product and
-portability boundaries remain unchanged; this audit is not deployment readiness proof.
-
-Repeatable local/CI drills now cover mixed HTTP load, quiesced backup restoration,
-configuration rollback and disposable volume exhaustion/recovery. Real deployment
-acceptance remains open; see [operational proof](OPERATIONAL-PROOF.md).
-
-The hardening CI pass also exposed a failed-store initialization cleanup race on
-Windows: rejection could precede worker termination and leave the DB file briefly
-locked. Initialization now closes the DB and awaits worker termination before
-returning failure. The missing-metadata regression exercises this cleanup path.

package/docs/SITE.md

@@ -1,150 +0,0 @@
-# Site conventions
-
-The optional top-level `site` block generates the small well-known files a
-site is expected to serve. Every key is off unless declared, and each declared
-key becomes one ordinary native route (`respond` or `page`) that is merged into
-the route table before compilation. Everything downstream is unchanged: the
-route appears in `urlcode routes` with `generated: "site.<key>"`, the audit
-generates fixtures for it, host policies apply to it and every deployment
-target compiles it like a route the project wrote by hand. `site` is accepted
-only in the entry `urlcode.yaml`, not in included files.
-
-```yaml
-version: "1"
-site:
-  robots:
-    disallow: [ai-crawlers, /admin]
-    allow: [/admin/public]
-    sitemap: true
-  sitemap:
-    exclude: [/drafts/*]
-    changefreq: weekly
-  favicon: public/favicon.svg
-  securityTxt:
-    contact: [mailto:security@example.com]
-    expires: "2027-01-01T00:00:00Z"
-    policy: [https://example.com/security-policy]
-    preferredLanguages: [en, fr]
-  llms: llms.txt
-routes:
-  /: {page: {file: public/index.html}}
-```
-
-The [cookbook](../examples/cookbook/urlcode.yaml) declares `robots`, `favicon`,
-`securityTxt` and `llms` with request fixtures; `sitemap` is exercised by the
-unit tests because it needs a public origin (below).
-
-## Precedence with declared routes
-
-A route the project declares at a generated path always wins. The generated
-route is dropped and the activation log records
-`{"event":"site","key":"robots","path":"/robots.txt","status":"shadowed"}`.
-This is how a project keeps a hand-written `robots.txt` while still using
-`site.favicon`, or migrates one file at a time.
-
-## The public origin
-
-`robots.txt`'s `Sitemap:` line and every `<loc>` in `sitemap.xml` are absolute
-URLs, so the runtime needs to know the origin the site is served from. It
-never guesses from a request: a generated file is a fixed response compiled at
-activation, and a `Host` header is client-controlled. The origin is the
-operator's `--origin https://links.example` flag on `serve`, `dev`,
-`validate`, `test`, `routes`, `audit`, `benchmark` and `build`, or the
-`origin` option of `startServer`/`createRuntime`/`buildCloudflare`.
-
-- `site.robots.sitemap: true` without an origin omits the `Sitemap:` line and
-  logs `{"event":"site","key":"robots","severity":"info",...}` at activation.
-- `site.sitemap` without an origin **refuses activation**: a sitemap of relative
-  URLs is invalid under the protocol, so there is no useful degraded form.
-
-## Keys
-
-### `robots` → `/robots.txt` (RFC 9309)
-
-| Field | Meaning |
-| --- | --- |
-| `disallow` | bundled agent list names (`ai-crawlers`, `crawlers`, `seo`, `monitoring`, the same lists as the [`agents` policy](policies/agents.md)) and/or paths starting with `/` |
-| `allow` | the same shapes, emitted as `Allow:` |
-| `sitemap` | `true` appends `Sitemap: <origin>/sitemap.xml` when the origin is known |
-| `extra` | literal lines appended verbatim (comments, `Crawl-delay`, ...) |
-
-List names become one group of `User-agent:` lines followed by `Disallow: /`
-(or `Allow: /`). Paths go under `User-agent: *`; with no paths that group is
-`Allow: /`. A list entry whose name is not usable as a product token (it has
-spaces or pattern metacharacters) is skipped and counted in an info log line.
-The result is served as `text/plain; charset=utf-8`.
-
-```
-User-agent: GPTBot
-User-agent: ClaudeBot
-Disallow: /
-
-User-agent: *
-Disallow: /admin
-Allow: /admin/public
-
-Sitemap: https://links.example/sitemap.xml
-```
-
-### `sitemap` → `/sitemap.xml` (sitemaps.org protocol 0.9)
-
-`true`, or `{ exclude, changefreq, priority }`. The sitemap lists every active
-literal GET route that serves HTML: a `page` whose file is `.html`/`.htm` or
-whose `contentType` is `text/html`; a `respond` route whose `Content-Type`
-response header is `text/html`; and the `.html` files under a `static` mount
-(the mount's `index` file is listed as its directory URL). Excluded:
-parameterized routes, redirects, functions, downloads and stored links,
-disabled or expired routes, `/robots.txt` and `/sitemap.xml`, any route whose
-`response.headers` set `X-Robots-Tag` containing `noindex`, and anything an
-`exclude` pattern matches (an exact path, or a prefix ending in `/*`).
-`lastmod` is the file's modification date for asset routes and omitted
-otherwise. URLs are sorted and XML-escaped; the response is
-`application/xml; charset=utf-8`.
-
-More than 50,000 URLs, or a document over the 1 MiB declared-response limit,
-refuses activation with a message saying to generate the file at build time
-and serve it as a static asset instead; the runtime does not split or index
-sitemaps.
-
-### `favicon` → `/favicon.ico`
-
-A project-relative `.ico`, `.svg` or `.png` file, served as a `page` route
-with `image/x-icon`, `image/svg+xml` or `image/png` and
-`cacheControl: public, max-age=3600` (the closest value in the
-[allowed vocabulary](ASSETS.md)). The route has the usual asset semantics:
-ETag, Last-Modified, conditional requests and ranges.
-
-### `securityTxt` → `/.well-known/security.txt` (RFC 9116)
-
-| Field | Rule |
-| --- | --- |
-| `contact` (required) | one or more `mailto:`, `tel:` or `https:` URIs |
-| `expires` (required) | UTC ISO timestamp; must be in the future at activation; more than a year away logs a warning (the RFC recommends less than a year) |
-| `policy`, `acknowledgments`, `canonical` | `https:` URIs |
-| `encryption` | `https:`, `dns:` or `openpgp4fpr:` URIs |
-| `preferredLanguages` | language tags, emitted as one comma-separated line |
-
-Fields are emitted in the RFC's order (Acknowledgments, Canonical, Contact,
-Encryption, Expires, Policy, Preferred-Languages), one value per line. The
-file is unsigned; add a signed copy as a declared route if you need one. The
-`.well-known` segment is an ordinary route segment (only `.` and `..` are
-refused).
-
-### `llms` → `/llms.txt`
-
-A project-relative text file served as a `page` route with
-`text/plain; charset=utf-8` and the default `no-cache`.
-
-## Per-target support
-
-| Target | `robots`, `sitemap`, `securityTxt` (`respond`) | `favicon`, `llms` (`page`) |
-| --- | --- | --- |
-| self-hosted, Vercel, AWS | served | served |
-| Cloudflare | compiled into the artifact (`build --origin` for absolute URLs) | refused at build time like any `page` route: the target has no asset binding; serve them from the platform's static assets |
-
-## Not in this release
-
-No per-route `noindex` field (use `response.headers: {X-Robots-Tag: noindex}`,
-which the sitemap honours), no sitemap index or split files, no `humans.txt`,
-no signed `security.txt`, and no project `.json` agent lists in `robots`
-(bundled names only).