@jimhoyd/urlcode 0.3.0

package/docs/MIDDLEWARE.md

@@ -0,0 +1,83 @@
+# Middleware
+
+Middleware is reusable JavaScript around any route handler. It is optional and
+route-local; plain redirects and assets retain their native fast path when no
+middleware is attached. Adding middleware requires sandbox execution.
+
+```yaml
+version: "1"
+routes:
+  /go:
+    middleware:
+      - source: middleware/headers.mjs
+        export: default
+    redirect:
+      url: https://example.com
+```
+
+```js
+export default async function headers(request, context, next) {
+  context.state.startedAt = Date.now();
+  const response = await next();
+  response.headers.set('x-example-middleware', 'active');
+  return response;
+}
+```
+
+Paths resolve from the project root. Up to 16 entries are allowed; each accepts
+`source` and optional `export` (default `default`). Reuse a module across routes.
+There are no global middleware settings or middleware-specific argument fields.
+This is URLCode's small portable API, not Express/Node middleware compatibility.
+
+## Order and responses
+
+For `[first, second]`, execution is first-before, second-before, handler,
+second-after, first-after. Call `await next()` to obtain the downstream response.
+Call it at most once, during the middleware invocation, with no arguments.
+Always return a `Response`. Return early to skip downstream middleware and the
+handler, for example `return new Response('Denied', {status: 403})`.
+Middleware may catch downstream JavaScript exceptions and return a fallback.
+
+Every middleware and the function share the same request and context. Validated
+`inputs`, function `args`, and route-scoped `env`/`secrets` are available along
+with a fresh `context.state` object for this chain. State never survives the
+request. Header edits are visible downstream; editing the request does not
+reroute it or change already validated inputs. Request bodies are single-use:
+reading one consumes it for downstream code. There is no `clone()` or streaming
+API; pass parsed data through `context.state` when needed.
+
+Function responses support the existing text/JSON guest API. To transform their
+body, read it and return a new `Response`. Native redirect/respond/link/page/static/
+download bodies are opaque and cannot be read through `text()` or `json()`.
+Returning the same native response preserves original bytes, including binary
+files, ranges and HEAD lengths. You may add headers, but cannot change its
+original status or existing native headers while preserving that body. To replace
+status, destination or content, return a new `Response` instead. Replacement
+responses follow the normal sandbox text/JSON and size limits. To wrap a shared
+template around file content, render it through a function at build time and
+publish the result: see [prerendering](PRERENDER.md).
+
+Route selection, enabled/expiry checks, methods and input/body validation run
+before middleware. Their errors do not pass through the chain. A missing file
+inside a selected static mount is a downstream 404 response. Stored-link lookup
+errors (missing, disabled, expired or unavailable store) precede middleware. YAML
+`response.headers` apply last and override matching middleware headers. Runtime
+framing and asset metadata protections still apply.
+
+## Isolation and testing
+
+The whole chain and handler run in one fresh QuickJS/WASM guest with one memory
+budget and one deadline. No Node, filesystem, shell, fetch or ambient environment
+is exposed. Modules can only access this route's declared dependency graphs.
+All middleware receive that route's approved bindings, so review the whole chain;
+source changes invalidate grants. See [security](FUNCTION-SECURITY.md).
+
+Invalid responses and repeated `next()` calls fail with 502, exhausted capacity
+returns 503, and the shared deadline returns 504. Middleware cannot extend the
+deadline or catch the outer worker termination. Forgotten downstream work is
+still drained within that deadline.
+
+Include explicit request fixtures for middleware-wrapped routes: test success,
+early responses, validation failures and every configured method. Audit will
+report missing coverage instead of assuming native handler behavior. Benchmark
+with middleware enabled to measure its actual sandbox overhead.

package/docs/MONITORING.md

@@ -0,0 +1,132 @@
+# Monitoring a URLCode deployment
+
+URLCode emits one JSON object per line on stdout, answers two unauthenticated
+probes and, when an operator opts in, serves in-process counters in Prometheus
+format. Everything below is built from those sources, and the example
+configuration in [`examples/monitoring/`](../examples/monitoring/) is the
+runnable form of it. The contract behind all of it (the event catalogue, the
+observer interface for in-process sinks, the metrics snapshot and the
+exposition endpoint) is in [observability](OBSERVABILITY.md).
+
+What you can measure is shaped by a deliberate logging decision: records carry
+**no URL, query string, header, body, binding or user exception text**. You can
+alert on error rates and latency per configured route; you cannot get per-URL
+analytics from these logs, and adding it would mean logging user data. See
+[operations](OPERATIONS.md).
+
+## Probes
+
+| Endpoint | Meaning | Alert when |
+|---|---|---|
+| `GET /_urlcode/health` | The process is alive and not shutting down. | It fails at all (`UrlcodeDown`). |
+| `GET /_urlcode/ready` | The active snapshot, every function worker and every configured link-store reader are available. | It fails for longer than replacement takes (`UrlcodeNotReady`). |
+
+Both return `{status, version, routes}`. A third endpoint,
+`GET /_urlcode/metrics`, exists only with `startServer({ metrics: true })` and
+answers 404 otherwise; see [metrics](#metrics) below. Keep both: alerting only on health hides
+a process that is up and serving nothing, while alerting only on readiness pages
+for a brief, self-healing worker replacement. Probes have their own bounded
+admission budget (`--max-in-flight-health`), so they keep answering while the
+application sheds load — but they are unauthenticated and disclose the
+configuration digest and route count, so keep them internal.
+
+## Log records
+
+| Event | Fields | Why it matters |
+|---|---|---|
+| `request` | `requestId`, `status`, `durationMs`; plus `method` and `route` with `--request-log detailed` | Error rate and latency per route. `route` is the configured pattern such as `/u/{id}`, never the requested path. |
+| `reload` | `status` (`ok`/`rejected`); `version` and `routes` on `ok` | A `rejected` reload means the last-good snapshot is still serving and a deploy did not take effect. |
+| `watch` | `status` | Development watcher failure; not used by `serve`. |
+| `function_worker` | `status` (`started`/`restarting`), `slot`; `attempt` and `delayMs` on `restarting` | Sustained `restarting` means a function is failing on real traffic. |
+| `link_store_worker` | `status`, `readOnly`, `attempt`, `delayMs` | The same signal for link-store connections. `status: "restarting"` reports an automatic replacement with its backoff; sustained restarts mean the underlying fault is not recoverable. |
+| `link_observer` | `status` (`failed`/`dropped`/`closed`); `reason` on `failed`; `dropped` on `dropped`; the delivery totals on `closed` | Only when an operator enables `linkEvents`. The link event channel below could not keep up or its collector failed. `dropped` means click records were discarded; like `logs_dropped`, anything built on that channel is incomplete while it fires. |
+| `logs_dropped` | `count` | The logger shed records because the collector fell behind. Every other signal is unreliable while this fires. |
+| `observer` | `status` (`failed`), `name` | An in-process observer threw; the request was unaffected. Written to the log only, never to observers. Sustained failures mean the observer's own sink is broken. |
+| `throttle`, `agents`, `cache` | `route`, `outcome`; `remaining` or `list` | Policy decisions; see [policies](POLICIES.md). `throttle` logs `allowed` only in report mode. |
+| `site` | `key`, `path`, `status` (`generated`/`shadowed`); or `severity` (`info`/`warning`) and `message` | Activation records for [site conventions](SITE.md). `shadowed` means a declared route took the path; an `info`/`warning` line reports an omitted `Sitemap:` line (no `--origin`), skipped list names or a far-future `security.txt` expiry. |
+| `management_request` | `timestamp`, `requestId`, `collection`, `action`, `authenticated`, `principal`, `status`, `outcome`, `durationMs` | Operator activity on the link-management API. `status` 0 means no response headers were sent before the peer disconnected; such a request may still have committed a mutation. |
+
+### The link event channel
+
+`link_request` is **not** a stdout record. It is delivered to an `observe()`
+function the embedding operator process supplies, after the response is over, so
+it can never change, delay or fail a redirect. It carries `requestId`,
+`collection`, `route`, `method`, `status`, `outcome`
+(`completed`/`aborted`/`missing`/`disabled`/`expired`/`invalid_code`/`invalid_record`/`unavailable`)
+and `durationMs`. The short code is redacted unless `includeCode` is set, because
+a code identifies the link somebody followed.
+
+The queue is bounded: under overload it drops events and reports the count
+through `link_observer` rather than growing memory. Alert on those drops if you
+count clicks — a quiet channel and a dropping channel look identical downstream.
+
+Startup prints `listening` with the effective `origin`, which is what functions
+and absolute URLs see. Behind a proxy or tunnel this must be your public origin;
+forwarded headers are deliberately not trusted. See [tunnels](TUNNELS.md).
+
+## Metrics
+
+`startServer({ metrics: true })` serves `GET /_urlcode/metrics` in Prometheus
+text format: requests by status class and by configured route, in-flight
+gauges, shed 503s, reloads, worker restarts and healthy slots, policy
+outcomes, link outcomes, dropped logs and observer errors, all prefixed
+`urlcode_`. The same numbers are available in process as `app.metrics()`. The
+endpoint shares the probes' admission budget and bind host and is off by
+default; it discloses route patterns and traffic shape, so keep it internal
+like the probes. [`examples/monitoring/prometheus-scrape.yaml`](../examples/monitoring/prometheus-scrape.yaml)
+scrapes it directly, without a log pipeline. Field names and label sets are
+fixed in [observability](OBSERVABILITY.md).
+
+If you would rather keep everything in one process, an observer passed as
+`startServer({ observers })` receives every log record and a periodic metrics
+snapshot; the same page shows an OpenTelemetry sketch.
+
+## What to alert on
+
+The example rules in
+[`examples/monitoring/prometheus-rules.yaml`](../examples/monitoring/prometheus-rules.yaml)
+cover:
+
+- **`UrlcodeDown`** — liveness probe failing. Process-level; check the supervisor.
+- **`UrlcodeNotReady`** — readiness failing for more than a few minutes.
+- **`UrlcodeServerErrors`** — over 5% 5xx on a route. 502 is a function error,
+  503 is capacity, 504 is a deadline.
+- **`UrlcodeCapacityShedding`** — sustained 503. Raise `--max-in-flight` or
+  `--workers`, or find what is occupying the pool.
+- **`UrlcodeWorkerRestartLoop`** — replacement backs off but never stops, so a
+  persistent cause appears as a steady restart rate rather than a stopped pool.
+  This is the signal that a function is exceeding its deadline on real traffic.
+- **`UrlcodeLogsDropped`** — the collector is behind, so the other rules are
+  blind until it recovers.
+
+Pick service objectives for your own application; these thresholds are a
+starting point, not a recommendation for your workload.
+
+## Wiring it up
+
+1. Send the process's stdout to a collector. The runtime never writes log files
+   and owns no rotation or retention; that belongs to the collector.
+2. Get counters either by scraping `/_urlcode/metrics` (enable `metrics`
+   and load [`examples/monitoring/prometheus-scrape.yaml`](../examples/monitoring/prometheus-scrape.yaml))
+   or by deriving them from the JSON records — see
+   [`examples/monitoring/vector.toml`](../examples/monitoring/vector.toml), which
+   produces `urlcode_requests_total`, `urlcode_worker_restarts_total` and
+   `urlcode_logs_dropped_total`. Fluent Bit, Promtail and Alloy work equally
+   well; the field names are what matter. The endpoint labels requests by
+   `status_class` where the log pipeline keeps the exact `status`; the example
+   rules carry both forms.
+3. Probe both endpoints with blackbox_exporter — see
+   [`examples/monitoring/blackbox-jobs.yaml`](../examples/monitoring/blackbox-jobs.yaml).
+4. Load the alert rules and set the thresholds to your objectives.
+
+Latency percentiles need a histogram; `durationMs` is per record, so have the
+log pipeline bucket it rather than averaging in the alert.
+
+## What this does not give you
+
+Dashboards here describe one process. There is no built-in tracing, no
+per-URL analytics, no distributed aggregation, no metrics persistence across
+restarts and no automatic capacity management. The example configuration is a starting point that has not been
+run against a production workload; validate it in your own environment before
+relying on it, and run the drills in
+[release readiness](RELEASE-READINESS.md) before treating any of it as proof.

package/docs/OBSERVABILITY.md

@@ -0,0 +1,229 @@
+# Observability
+
+URLCode reports what it does through three interfaces built on one stream of
+events: the JSON log on stdout, operator **observers** that receive the same
+events in process, and a **metrics snapshot** of counters derived from them,
+optionally served in Prometheus format. Observers mirror [host plugins](PLUGINS.md):
+they are JavaScript an operator passes to `startServer` or `createRuntime`,
+never something a project's YAML can name, and they run with the host's
+privileges. [Monitoring](MONITORING.md) is the operator's guide to probes,
+recipes and alerts; this page is the contract.
+
+```js
+import { startServer } from '@jimhoyd/urlcode';
+
+await startServer({
+  project: './site',
+  observers: [myObserver],   // in-process event and metrics sinks
+  metrics: true,             // GET /_urlcode/metrics, Prometheus text format
+});
+```
+
+## Event catalogue
+
+`events` in `@jimhoyd/urlcode/observability` is a frozen object mapping each event name
+to the complete list of fields it may carry. A test runs a real server and
+holds every record to it, so a field or event that is not in the table below
+does not ship. Fields marked *optional* are present only in the situations
+named.
+
+| Event | Fields | Emitted when |
+|---|---|---|
+| `request` | `requestId` string, `status` integer, `durationMs` number; `method` string and `route` string or `null` with `--request-log detailed` | Every response the server wrote, including probes and shed 503s. `route` is the configured pattern (`/u/{id}`) or the probe path, never the requested path. |
+| `reload` | `status` `ok`/`rejected`; `version` string and `routes` integer on `ok` | `app.reload()` or the development watcher swapped, or refused to swap, the snapshot. |
+| `watch` | `status` `failed` | The development watcher could not fingerprint the project. |
+| `function_worker` | `status` `started`/`restarting`, `slot` integer; `attempt` and `delayMs` integers on `restarting` | A function worker became ready or is scheduled for replacement. |
+| `link_store_worker` | `status` `started`/`restarting`, `readOnly` boolean; `attempt` and `delayMs` on `restarting` | The same for a link-store connection. |
+| `link_observer` | `status` `failed`/`dropped`/`closed`; `reason` `timeout`/`error` on `failed`; `dropped` integer on `dropped`; `queued`, `delivered`, `dropped`, `failed`, `timedOut`, `closed` on `closed` | Only with `linkEvents`: the link collector failed, the queue shed events, or the channel drained at shutdown. |
+| `link_request` | `requestId`, `collection`, `route`, `code` (only with `includeCode`), `method`, `status`, `outcome`, `durationMs` | Delivered to the `linkEvents.observe` function only, after the response is over. It never reaches the log or observers; its `outcome` is counted in the metrics. |
+| `logs_dropped` | `count` integer | The JSON logger shed records because stdout was not writable. Written by the logger itself, so observers do not see it. |
+| `observer` | `status` `failed`, `name` string | An observer hook threw or rejected. Written to the default log only, never to observers. |
+| `throttle` | `route`, `outcome` `allowed`/`exceeded`, `remaining` integer | A throttle decision. `allowed` is logged only in `mode: report`; enforce mode logs refusals. |
+| `agents` | `route`, `list` string, `outcome` `denied`/`reported` | A User-Agent matched a list. The list name is logged, never the header. |
+| `cache` | `route`, `outcome` `hit`/`stale`/`miss`/`store` | A cache lookup or store. |
+| `management_request` | `timestamp`, `requestId`, `collection`, `action`, `authenticated`, `principal`, `status`, `outcome`, `durationMs` | The link-management API (`startLinkApi`), which has its own logger; `principal` is the operator-configured principal id. |
+| `listening` | `address`, `port`, `mode`, `origin` | Printed once by the CLI at startup, not emitted by the server. |
+
+Every event carries `event` (its name). Numbers are JSON numbers, never
+strings.
+
+### Privacy guarantees
+
+No event, snapshot or exposition carries a request URL, path, query string,
+header, body, client address, User-Agent string, binding, secret, stored link
+destination or user exception text. `route` is always a configured pattern
+from reviewed YAML. `requestId` is server-generated unless
+`--trust-request-id` accepts one from a trusted proxy. `code` in
+`link_request` is redacted unless the operator sets `includeCode`. An observer
+that logs should keep the same rule; nothing in an event lets it break it.
+
+## Observers
+
+```js
+const myObserver = {
+  name: 'forwarder',            // ^[a-z][a-z0-9-]{0,63}$, unique per server
+  version: '1.0.0',             // any string up to 64 characters
+  onEvent(event) {},            // every record the JSON logger writes, in order
+  onMetrics(snapshot) {},       // the metrics snapshot, on the interval and at close
+  async onClose() {},           // release resources; reverse order
+};
+```
+
+In TypeScript the contract is `Observer` from `@jimhoyd/urlcode/observability` (also
+exported from `urlcode`), with `ObserverEvent` for a record and
+`MetricsSnapshot` for what `onMetrics` receives; the declarations ship with the
+package:
+
+```ts
+import type { Observer, ObserverEvent, MetricsSnapshot } from '@jimhoyd/urlcode/observability';
+
+const myObserver: Observer = {
+  name: 'forwarder',
+  version: '1.0.0',
+  onEvent(event: ObserverEvent) { queue.push(event); },
+  onMetrics(snapshot: MetricsSnapshot) { gauge.set(snapshot.requests.inFlight); },
+};
+```
+
+Validation (`validateObservers`) matches plugins: at most 32 observers, each
+an object with a kebab-case `name` no other observer uses, a `version` string,
+every declared hook a function and at least one present. It runs before the
+listener starts, so a bad observer fails startup rather than a request.
+
+`onEvent` receives the same object the logger serialised, after the logger,
+observers in array order. Do not mutate it. It runs on the request path, so
+keep it cheap: buffer and flush on a timer rather than awaiting a network
+call. A hook that throws or returns a rejecting promise is isolated: the
+request is unaffected, the next observer still runs, `observers.errors` in
+the snapshot increments and one `observer` record goes to the default log.
+Nothing is retried; an observer that needs delivery guarantees owns its own
+queue, like the [link event channel](MONITORING.md#the-link-event-channel).
+
+`onMetrics` receives a fresh snapshot every `metricsIntervalMs`
+(`startServer` option, `0` off by default, 1 s to 1 h) and once at `close()`.
+`onClose` runs in reverse order after the runtime and the link channel have
+drained. `app.observers` lists the `{ name, version }` pairs.
+
+`createRuntime(project, { observers })` takes the same array for embedding
+without the server: the runtime's own sink and counters are then yours, and
+`runtime.metrics()` returns its snapshot. `startServer` never passes its
+observers down to the runtimes it creates, so a reload does not re-register
+them and counters survive reloads.
+
+`createObserverSink(observers, fallbackLog)` is the fan-out itself, exported
+for tests and custom hosts: it returns a `log(event)` function with `.metrics`,
+`.publish(snapshot)` and `.close()`.
+
+## Metrics snapshot
+
+`app.metrics()` and `runtime.metrics()` return a plain object, safe to
+`JSON.stringify`, of counters since the process started serving. Numbers
+only; the one keyed table is `requests.byRoute`, keyed by configured pattern
+and capped at 10 000 keys.
+
+| Field | Type | Meaning |
+|---|---|---|
+| `version` | gauge | Snapshot shape version, currently `1`. |
+| `uptimeSeconds`, `rssBytes` | gauge | Process facts. |
+| `requests.total`, `requests.byStatusClass.{2xx,3xx,4xx,5xx}` | counter | Application responses, including shed 503s. |
+| `requests.inFlight` | gauge | Requests holding application admission now. |
+| `requests.byRoute[pattern]` | counter | Responses per matched route. A shed or unmatched request has no route. |
+| `health.total`, `health.byStatusClass`, `health.inFlight` | counter, gauge | The probe budget: `/_urlcode/health`, `/_urlcode/ready` and `/_urlcode/metrics`. |
+| `shed.requests`, `shed.health` | counter | 503s answered because an admission budget was full. |
+| `reloads.ok`, `reloads.rejected` | counter | Snapshot swaps. |
+| `watch.failed` | counter | Development watcher failures. |
+| `functionWorkers.started`, `functionWorkers.restarts` | counter | Worker starts and scheduled replacements. |
+| `functionWorkers.healthySlots`, `functionWorkers.slots` | gauge | Ready slots and configured slots of the serving runtime. |
+| `linkStoreWorkers.started`, `linkStoreWorkers.restarts` | counter | The same for link-store connections. |
+| `policies.throttle.{allowed,exceeded}` | counter | Throttle decisions (see the catalogue for what enforce mode logs). |
+| `policies.agents.{denied,reported}` | counter | Agents decisions. |
+| `policies.cache.{hit,stale,miss,store}` | counter | Cache outcomes. |
+| `linkRequests.{completed,aborted,missing,disabled,expired,invalid_code,invalid_record,unavailable}` | counter | Dynamic link outcomes, counted whether or not `linkEvents` is set. |
+| `linkObserver.failed`, `linkObserver.dropped` | counter | Link collector failures and dropped events. |
+| `logsDropped` | counter | Records the JSON logger shed. |
+| `observers.errors` | counter | Observer hooks that threw or rejected. |
+
+Policy counters are derived from the `throttle`, `agents` and `cache` events
+as they pass through the sink, so the policies themselves have no metrics
+code. Runtime facts that never become events (admission, shedding, slot
+health) are recorded by the server directly. Counters are per process;
+aggregation across replicas is the scraper's job.
+
+## Prometheus exposition
+
+`startServer({ metrics: true })` serves `GET /_urlcode/metrics` as
+`text/plain; version=0.0.4`, rendered from the same snapshot by
+`renderPrometheus(snapshot)`, a pure function you can also call yourself.
+Every metric is prefixed `urlcode_`; counters end in `_total`; the only labels
+are `status_class`, `route` and `outcome`.
+
+```
+# HELP urlcode_requests_total Application requests answered since start, by status class.
+# TYPE urlcode_requests_total counter
+urlcode_requests_total{status_class="2xx"} 1042
+urlcode_route_requests_total{route="/u/{id}"} 977
+urlcode_requests_in_flight 3
+urlcode_shed_total{outcome="requests"} 0
+urlcode_reloads_total{outcome="ok"} 2
+urlcode_function_worker_restarts_total 0
+urlcode_function_worker_healthy_slots 2
+urlcode_throttle_total{outcome="exceeded"} 14
+urlcode_cache_total{outcome="hit"} 511
+urlcode_link_requests_total{outcome="completed"} 0
+urlcode_logs_dropped_total 0
+urlcode_observer_errors_total 0
+urlcode_uptime_seconds 86400
+urlcode_process_rss_bytes 71303168
+```
+
+The endpoint is **off by default**. It shares the probes' admission budget
+(`--max-in-flight-health`) and the same bind host, which is `127.0.0.1`
+unless `--host` says otherwise. Like the probes it is unauthenticated and
+discloses route patterns and traffic shape, so **do not expose it publicly**:
+keep it on an internal interface or restrict it at the ingress. A scrape
+counts under `health`, not under application requests. See
+[`examples/monitoring/prometheus-scrape.yaml`](../examples/monitoring/prometheus-scrape.yaml).
+
+## OpenTelemetry sketch
+
+The runtime has no OpenTelemetry dependency. An observer can forward events
+to an OTLP exporter and map the snapshot onto instruments; this is a sketch,
+not shipped code, and omits batching, resource attributes and error handling.
+
+```js
+// Sketch. `logs` and `meter` come from the OpenTelemetry SDK the operator
+// configures; the runtime knows nothing about them.
+export function otelObserver({ logger, meter }) {
+  const requests = meter.createCounter('urlcode.requests', { unit: '{request}' });
+  const inFlight = meter.createObservableGauge('urlcode.requests.in_flight');
+  let last;
+  inFlight.addCallback(result => { if (last) result.observe(last.requests.inFlight); });
+  return {
+    name: 'otel', version: '0.1.0',
+    onEvent(event) {
+      // Every field is already safe to attach as an attribute.
+      logger.emit({ body: event.event, attributes: event });
+      if (event.event === 'request') requests.add(1, { status_class: `${Math.floor(event.status / 100)}xx`, route: event.route ?? '' });
+    },
+    onMetrics(snapshot) { last = snapshot; },   // the gauge reads the latest snapshot
+  };
+}
+```
+
+Counters in the snapshot are cumulative, so they map to OpenTelemetry
+`Counter` instruments read through an observable callback, or to a
+Prometheus receiver scraping `/_urlcode/metrics` directly. `durationMs` on
+`request` is the input for a `Histogram`; the runtime does not bucket it.
+
+## What is not provided
+
+- **Tracing.** There are no spans and no context propagation; `requestId` is
+  the only correlation key, and a trusted proxy can supply it.
+- **Sampling.** Every event is delivered to every observer, or shed by the
+  logger under back-pressure and reported as `logs_dropped`.
+- **Persistence and aggregation.** Counters live in process memory, reset on
+  restart, and describe one process. Retention and cross-replica sums belong
+  to the collector.
+- **Per-URL analytics.** By design; see the privacy guarantees.
+- **Adapters.** Vercel, Lambda and Cloudflare handlers emit through the
+  platform's own logging and do not take observers.

package/docs/OPERATIONAL-PROOF.md

@@ -0,0 +1,42 @@
+# Operational evidence and deployment acceptance
+
+`node scripts/operational-drills.ts` creates and deletes its own temporary project
+and store. It runs mixed real HTTP requests against native redirects, isolated
+functions and live links while updating records; asserts responses; rejects a bad
+reload; activates and rolls back a configuration; then closes all users of the
+store, explicitly verifies a successful WAL checkpoint, and restores a copied database. It checks integrity, record/version and
+latest durable audit revision. `URLCODE_SOAK_SECONDS=60` selects a longer run
+(1–3600 seconds, default 5). Output is JSON with request count, batch p99 duration,
+RSS and restore time. Batch latency is not per-request p99 or a capacity promise.
+CI runs the short drill on all six supported Node/OS combinations.
+
+The container job additionally uses `--disk-full-dir /state` on a disposable
+16 MiB tmpfs. It reserves recovery space, writes until a real volume exhaustion
+failure, checks available disk space, releases the reserve, proves a subsequent
+mutation succeeds, and checks integrity and equal committed link/audit counts.
+Never point this option at production storage: it intentionally consumes up to
+84 MiB of writes in a newly created temporary child directory. The CI mount is
+nonpersistent. An I/O device failure or power cut is a different failure mode.
+
+## Required proof on the intended deployment
+
+Local/CI passes do not close these gates. The deployment owner must record:
+
+| Exercise | Acceptance evidence |
+|---|---|
+| Soak | At least 24 hours at expected peak and burst load through actual TLS/ingress; native/function/live-link mix, slow clients, response correctness, p50/p95/p99, throughput, error/rejection rate, RSS plateau, CPU, FD and disk/WAL growth. Define numerical SLOs before starting |
+| Restore | Restore a consistent backup onto a separate host/volume. Reconcile collection counts, latest committed revisions and audit journal; measure RPO/RTO against agreed targets. Test encrypted backup access and credential recovery |
+| Logical export/restore | Run `links export` against a store under write load, restore it with `links import` onto a separate store, and reconcile record counts and every field. Prove the export's digest verifies, that a truncated stream is rejected, and that stale management ETags are discarded because the restore reassigns versions. An export carries no audit journal, so pair it with a file backup |
+| Kill/restart | Kill server and writer with outstanding reads/writes. Reconcile uncertain mutations by version/request ID, run integrity checks, restore readiness, and prove no duplicate successful conditional writes |
+| Disk/log exhaustion | Fill the actual disposable staging storage type and log destination. Verify bounded 503s, audit/mutation atomicity, alert delivery through an independent sink, reserve-space recovery and readiness after recovery |
+| Rollback | Deploy candidate by exact digest beside last-good, run route assertions, switch ingress, drain, then switch back. Verify configuration/policy compatibility and audit continuity; never downgrade to an unaudited writer |
+| Monitoring | Deliver test alerts for missing logs, sustained errors, pool rejection/failure, low disk, restart storms, backup age and failed readiness to a named on-call owner |
+
+Record date, operator, source/app/policy/image digests, topology, hardware/limits,
+commands, duration, synthetic dataset size, raw metrics/log locations, result,
+RPO/RTO and unresolved findings. A reviewer signs the acceptance record; a blank
+record is not a pass. Do not copy only the main file of a live WAL database.
+The executable local drill uses a fully quiesced, explicitly checkpointed database intentionally. A last
+read-only connection can leave WAL frames even after every connection closes;
+closing alone is insufficient proof that the main database file is a full backup. For a
+live-backup system, prove its SQLite-consistent snapshot method separately.