@jimhoyd/urlcode 0.3.0 → 0.4.0-alpha.1

package/docs/INTERCHANGE.md

@@ -0,0 +1,134 @@
+# Redirect interchange and bulk authoring
+
+`importRoutes` and `exportRoutes` return a conversion report without writing files,
+activating a runtime, reading bindings, loading function sources or running code.
+The CLI can preview the report before creating a new output file. Failed reports
+never contain a partial document or output; duplicate paths never overwrite a row.
+
+```js
+import { importRoutes, exportRoutes } from '@jimhoyd/urlcode';
+const report = await importRoutes({
+  format: 'csv', source: 'migration.csv',
+  text: 'path,url,status\n/old,https://example.test/new,301\n'
+});
+if (!report.ok) throw new Error(JSON.stringify(report.diagnostics));
+// report.output is URLCode project YAML; report.document is the validated project.
+const csv = await exportRoutes({ format: 'csv', document: report.document });
+```
+
+Each report has `ok`, `lossless`, `routeCount` and `diagnostics`. Diagnostics name
+an input source, physical CSV/text row or JSON/YAML array index where available,
+a path where appropriate, a code, severity and explanation. Destinations and
+credentials are not echoed in validation errors. Output is sorted by literal
+path, independent of input order. Import does not merge an existing project;
+normal project loading rejects collisions across includes when output is added.
+
+## CLI usage
+
+```sh
+# Bulk row formats can be inferred from the input extension or named explicitly.
+urlcode import csv migration.csv --out routes.yaml --dry-run
+urlcode import csv migration.csv --out routes.yaml
+urlcode import migration.json --report json --dry-run
+
+# Provider migrations require explicit acknowledgment of the reported differences.
+urlcode import netlify _redirects --out imported.yaml --dry-run
+urlcode import netlify _redirects --out imported.yaml --accept-provider-differences
+urlcode export --target netlify --project ./project --out _redirects --accept-provider-differences
+urlcode export --target cloudflare --project ./project --dry-run
+urlcode export --target vercel --project ./project --dry-run
+urlcode export --target netlify-toml --project ./project --dry-run
+```
+
+General import syntax is `urlcode import [format] FILE`; `--format` explicitly
+overrides the positional format or extension. Exports use `--target` for a
+provider format, or `--format csv|json|yaml` for row data. `--out FILE` creates a
+new file with mode 0600 and never overwrites one. `--dry-run` validates and
+returns a report without creating the requested output. `--report json` always
+emits the report. Failures exit nonzero and leave no converted output.
+
+Reports include source/row diagnostics and the successful candidate output.
+Acknowledged provider conversions always emit a report with their warnings,
+even when `--out` is provided; raw provider output is never silently printed as
+if it were lossless. Project export flattens validated includes, but refuses
+other unsupported project behavior instead of dropping it. For a large input
+that needs a complete sharded project, use [bulk import](BULK.md).
+
+## Supported forms
+
+| Format | Input/output subset |
+| --- | --- |
+| `csv` | Header exactly `path,url,status`; optional empty status defaults to 302; quoted fields and escaped quotes |
+| `json` | Array of `{path,url,status?}`; status must be a number |
+| `yaml` | The same row array in strict URLCode YAML syntax; not a full project file |
+| `netlify` | `_redirects`: literal source, absolute HTTP(S) destination, optional status (default 301) |
+| `cloudflare` | Pages `_redirects`: same columns, default 302; 2,000 static rules, 1,000 characters per rule |
+| `vercel` | `vercel.json` containing only `redirects`; literal `source`, absolute `destination`, and either boolean `permanent` (308/307) or `statusCode` |
+| `netlify-toml` | Only `[[redirects]]` blocks with `from`, `to`, optional numeric `status` (default 301); unescaped double-quoted strings |
+
+All statuses are restricted to 301, 302, 303, 307 and 308. Literal ASCII paths
+and absolute HTTP(S) destinations are required. The existing schema and semantic
+compiler validate resulting routes. No provider pattern compiler is introduced.
+
+The TOML subset deliberately rejects general TOML constructs, build settings,
+escapes, inline comments, nested conditions, force flags and other tables. A full
+`netlify.toml` must be separated into a redirects-only input by the operator. This
+avoids silently discarding build settings or reinterpreting unsupported syntax.
+
+Functions, assets, middleware, conditions, parameters, query maps/allowlists,
+headers, policies, includes and any other extra route/project fields cannot be
+exported by this subset. Relative destinations, wildcards, provider placeholder
+syntax, rewrites and duplicate paths fail. Use a runtime adapter for richer
+behavior. CSV/JSON/YAML round-trips are lossless within the declared simple subset.
+
+## Provider semantics require explicit acknowledgment
+
+Provider conversions fail by default. `acceptProviderDifferences: true` (CLI
+`--accept-provider-differences`) permits a **non-lossless migration candidate**;
+it does not suppress invalid rules, discard unsupported fields or claim exact
+behavior. Every such successful report retains a warning and `lossless: false`.
+
+URLCode drops incoming query parameters and defaults to GET/HEAD. Native
+redirect systems can apply to other methods and have different normalization
+and request-query behavior. Netlify automatically forwards queries for common
+redirect statuses and can give existing files precedence. Cloudflare Pages
+redirects override assets, but Pages Functions can bypass `_redirects`.
+Review behavior for requests with queries, non-GET methods, slash/URL normalization
+and conflicting assets/functions before deploying. The acknowledged subset is
+literal GET/HEAD requests without these conflicts. No provider deployment was
+performed as part of interchange unit tests.
+
+Provider references checked 2026-09-17:
+
+- [Netlify redirect options](https://docs.netlify.com/manage/routing/redirects/redirect-options/)
+- [Netlify TOML configuration](https://docs.netlify.com/build/configure-builds/file-based-configuration/)
+- [Cloudflare Pages redirects](https://developers.cloudflare.com/pages/configuration/redirects/)
+- [Vercel redirect configuration](https://vercel.com/docs/project-configuration/vercel-json)
+
+## Resource limits
+
+Imports reject more than 32 MiB of input or 100,000 rows. Diagnostic collection
+stops after 100 invalid rows; normal route compilation retains its deadline and
+validation rules. Large output can still exceed the runtime YAML worker's heap
+or deadline: conversion success does not promise activation of a single 100k
+route YAML file. Split large migration inputs into independently reviewed files
+and use explicit `includes`; aggregate project limits continue to apply.
+
+Reports also expose bounded classification counts:
+
+- `convertedRoutes`: routes actually returned in successful output; zero when
+  any error prevents output.
+- `nativeEquivalentRoutes`: returned routes with no conversion warnings; always
+  zero for acknowledged provider conversions.
+- `runtimeRequiredRoutes`: route diagnostics identifying behavior outside the
+  simple export subset.
+- `unsupportedRows`: known invalid/duplicate input-row diagnostics.
+- `providerDifferenceRoutes`: candidate routes subject to the provider semantic
+  warning; these overlap converted routes after acknowledgment.
+- `fullyScanned`: false on parser/global failures or truncated diagnostics. Counts
+  then describe only examined input, never an inferred total.
+
+These counts are not a partition of arbitrary malformed input. `routeCount`
+retains its original meaning: routes in the candidate table, or rows
+parsed before an early global failure. A failed report contains no candidate
+output even if some rows were convertible.

package/docs/MIDDLEWARE-EXAMPLES.md

@@ -0,0 +1,75 @@
+# Middleware examples
+
+Fourteen reusable middleware modules covering fifteen patterns (`auth.mjs`
+exports both `bearer` and `basic`), each in
+[`examples/cookbook/middleware`](../examples/cookbook/middleware) with a route in
+[`routes/middleware.yaml`](../examples/cookbook/routes/middleware.yaml) and request
+fixtures in the cookbook tests. The same modules ship as the `middleware`
+[local recipe](RECIPES.md):
+
+```sh
+urlcode recipes add middleware --out ./my-middleware
+urlcode test --project ./my-middleware
+```
+
+They cover what Express, Koa, Fastify, Hono, Next.js and edge runtimes usually
+ship as middleware, rewritten for URLCode's [guest API](MIDDLEWARE.md): text/JSON
+`Request`/`Response`, `Headers`, `context.inputs/args/env/secrets/state`, timers
+and nothing else. There is no `crypto`, `URL`, `fetch`, storage or console, and
+`context.state` dies with the request. Patterns that need any of those are listed
+at the end so nobody wastes time porting them.
+
+| Pattern | Module | Framework equivalent | Demonstrates |
+| --- | --- | --- | --- |
+| Bearer token gate | `auth.mjs` `bearer` | Hono `bearerAuth`, `express-bearer-token` | Early 401, `www-authenticate`, constant-time compare against an `env` binding |
+| Basic authentication | `auth.mjs` `basic` | Hono `basicAuth`, `express-basic-auth` | Hand-written base64 (no `atob`), both checks always evaluated, `state.user` |
+| CORS | `cors.mjs` | Express `cors`, Hono `cors` | `OPTIONS` answered before the handler, origin allowlist, `vary: origin` |
+| Correlation id and timing | `request-id.mjs` | Hono `requestId`/`timing`, Express `response-time` | Validating a caller header, `server-timing`, coexisting with the runtime's own `x-request-id` |
+| Maintenance switch | `maintenance.mjs` | Next.js and Netlify Edge maintenance examples | 503 with `retry-after`, bypass header, flipping behavior from a binding |
+| Error boundary | `errors.mjs` | Koa `onerror`, Express error handlers | Catching a downstream throw, JSON 500 instead of a bare 502 |
+| JSON envelope | `envelope.mjs` | Response transformers | Reading a function body once, passing native bodies through untouched |
+| Content negotiation | `negotiate.mjs` | Express `res.format` | Parsing `accept` with q-values, 406, `vary: accept` |
+| Method override | `methods.mjs` `override` | Express `method-override` | Bounded tunneling through POST, 405 with `allow` |
+| ETag and 304 | `etag.mjs` | Express `etag`, Fastify `@fastify/etag` | FNV-1a weak tag, `if-none-match`, null-body 304 |
+| A/B bucket | `bucket.mjs` | Vercel and Cloudflare A/B examples | Cookie parsing, `set-cookie`, replacing a native redirect |
+| Locale redirect | `locale.mjs` | Next.js i18n middleware | `accept-language` ranking, allowlisted languages, `vary` |
+| Referer allowlist | `referer.mjs` | Hotlink protection rules | Gating a native download without reading it |
+| Body validation | `body.mjs` | `express-validator`, Fastify schemas | Single-use body, 422 error list, handoff through `state` |
+| Debug echo | `debug.mjs` | Request loggers | Inspecting inputs, args and redacted headers when the console is silent |
+
+## Reading the modules
+
+Every module follows the shape in [middleware](MIDDLEWARE.md): read the request,
+optionally return early, otherwise `await next()` once and return a `Response`.
+Three habits recur and are worth copying:
+
+- **Configuration lives in bindings.** Tokens, allowlists and switches are read
+  from `context.env`. The cookbook binds literal values so it runs without grants;
+  a deployed project uses `{secret: name}` for credentials and an operator grant.
+- **Native bodies stay opaque.** `envelope`, `negotiate` and `etag` only rewrite a
+  body when the downstream response is a function response with a readable
+  content type. `bucket`, `locale` and `referer` wrap native redirects and
+  downloads without touching their bytes; to change the destination they return
+  a new `Response` instead.
+- **Chains compose through `state`.** `/fragile` runs `request-id` before
+  `errors`, so the fallback JSON carries the correlation id. `/profile` parses
+  the body once in middleware and the function reads `context.state.body`.
+
+## Limits these examples respect
+
+- **No `crypto`.** The auth modules compare a shared token; they cannot verify
+  HMAC signatures, JWTs or password hashes. Signed URLs would be a runtime
+  feature, not a middleware example.
+- **No cross-request state.** Rate limiting, caching, sessions and CSRF tokens
+  need storage the guest does not have. Throttling and cache headers exist as
+  native [policies](POLICIES.md) instead.
+- **No logging target.** The guest console is a no-op; `debug.mjs` returns the
+  information to the caller instead, and only when both a binding and a header
+  ask for it. Remove that route before publishing a project.
+- **Runtime headers win.** The runtime stamps `x-request-id` on every response,
+  which is why the tracing example uses `x-correlation-id`. YAML
+  `response.headers` also override middleware headers.
+
+Each cookbook route has fixtures for its success path, its early responses, its
+validation failures and every declared method, which is what `urlcode audit`
+expects before it reports a middleware-wrapped route as covered.

package/docs/MIDDLEWARE.md

@@ -28,6 +28,8 @@ Paths resolve from the project root. Up to 16 entries are allowed; each accepts
 `source` and optional `export` (default `default`). Reuse a module across routes.
 There are no global middleware settings or middleware-specific argument fields.
 This is URLCode's small portable API, not Express/Node middleware compatibility.
+Fourteen ready-to-copy patterns (auth, CORS, tracing, error boundary, ETag and
+more) are in [middleware examples](MIDDLEWARE-EXAMPLES.md).
 
 ## Order and responses
 

package/docs/NEXT-PHASE-PLAN.md

@@ -0,0 +1,90 @@
+# Next-phase implementation status
+
+Repository review and source implementation, 2026-09-17. Phase A established
+the capability catalog; the subsequent source work implements the bounded
+Phase B–D features below. This describes the source additions after `0.3.0` that ship in
+`0.4.0-alpha.1`, not a claim that provider deployments have been verified.
+
+## Implementation and evidence
+
+| Phase | Implemented source | Evidence and remaining limits |
+| --- | --- | --- |
+| A: capabilities and normalized representation | Shared catalog, route/project analysis, CLI/SDK and adapter preflight reuse the existing compiled IR | Capability tests and target refusals; compiled routes with secrets/closures are never a portable public artifact |
+| B1: provider conformance | Synthetic 12-case common-subset fixture; local self-hosted/AWS/Vercel/Cloudflare replay; bounded HTTPS deployment runner and versioned reports | Local adapter evidence exists; actual AWS/Vercel/Cloudflare deployments and provider-specific transport/policy guarantees remain unverified |
+| B2: Netlify/Cloudflare conversion | Strict literal redirect import/export, source diagnostics, dry-run, no-clobber output and explicit provider-difference acknowledgment | Provider normalization, query forwarding, method coverage and asset precedence differ; acknowledged migrations are explicitly non-lossless |
+| B3: Vercel/TOML conversion | Conservative Vercel redirect subset and redirects-only Netlify TOML grammar | Unsupported fields, patterns, conditions, forced rules and general TOML syntax are rejected rather than discarded |
+| C1: bounded proxy | Self-hosted native proxy and external revision-pinned HTTPS-origin grants, connection-pinned public DNS, body/time/concurrency limits and header filtering | No guest fetch, host execution fallback, automatic redirects or retries; providers refuse proxy; independent security review remains open |
+| C2: conditions | Exact bounded query/header/cookie/host/method predicates; explicit disjoint redirect/respond cases and fallback; no-store | Duplicate YAML keys remain errors; ambiguous cases fail; self-hosted/AWS/Vercel share logic, Cloudflare refuses pending artifact support |
+| C3: best-effort signals | Self-hosted bounded webhook broker, external pinned grants, fixed redacted event shape, accepted/delivered/failed/dropped counters and shutdown handling | No queue, retry, ordering or durability guarantee; saturation drops; providers refuse signals |
+| D: recipes | Three ordinary local Git-owned recipe projects with list/show/add and new-directory dry-run publication | Runtime integration tests exercise redirect, JSON API and built TypeScript recipe; no remote registry or implicit project merge |
+| D: bulk | Strict CSV/JSON/YAML conversion, input fingerprint/source provenance, sorted 1,000-route include shards | 1k/10k/100k local measurements pass without relaxing loader limits; no implicit merge or arbitrary bulk mutation |
+| D: TypeScript guests | Fixed trusted build-time compiler, bounded relative graph, rewritten JavaScript imports, referenced-asset snapshot and safe new output | Transpilation is not type checking; no tsconfig/plugins/package execution or dotenv copying; runtime remains QuickJS JavaScript only |
+| D: consolidated SDK/MCP | Inspection, semantic validation, path explanation, compatibility, conversion previews and recipe discovery; operator-rooted stdio MCP | Read-only tooling; no arbitrary path, credential, guest execution or write authority; not a remote authenticated service |
+
+See [interchange](INTERCHANGE.md), [provider evidence](PROVIDER-VERIFICATION.md),
+[egress](EGRESS.md), [conditions](CONDITIONS.md), [recipes](RECIPES.md),
+[bulk measurements](BULK.md), [TypeScript authoring](TYPESCRIPT-AUTHORING.md), and
+[tooling/MCP](TOOLING.md) for the executable interfaces and exact restrictions.
+
+## Preserved architecture
+
+The schema and semantic compiler remain the behavior contract. `CompiledRoute`
+and `CompiledRouteTable` remain the runtime IR; `MatchableRoute` remains the
+shared matching representation. New handlers and conditions extend those paths
+rather than introducing a second route compiler. Capability analysis precedes
+binding resolution and activation and distinguishes implementation support from
+actual deployment evidence. Unknown or unsupported targets fail closed.
+
+Project YAML describes route behavior. Provider infrastructure and outbound
+origin grants stay in operator configuration. Functions remain untrusted
+QuickJS/WASM guests with no host-code fallback. Existing explicit external
+bindings and new egress grants remain pinned to the exact configuration/source
+revision. Compilation and conversion do not resolve credentials or manufacture
+grants. File authors publish new projects without overwriting unrelated work.
+
+The five policy modules remain the authority for their target-sensitive
+compatibility. Delegated compression does not imply identical edge behavior;
+coalesced headers and normalized URLs remain transport limitations. Conditional
+routes are no-store to prevent cross-branch shared-cache leakage. Conditions are
+selection rules, not authentication or authority.
+
+## Validation and outstanding external work
+
+New regression suites cover conservative conversion refusals, source diagnostics,
+local provider replay, conditions/ambiguity, proxy and signal security boundaries,
+recipe execution, TypeScript graph limits, bulk sharding and MCP authority limits.
+Package smoke exercises actual archive installation with production dependencies,
+CLI authoring/conversion/MCP and the public declaration surface. Schema changes
+require regenerated reference documentation and executable examples. Local integration passed `npm run verify` (384 tests: 383 passed, one existing
+TLS-fixture skip) and `npm run test:package`, including a production-only install
+with TypeScript 6.0.3. Required CI checks and normal pull-request review still
+apply to each exact proposed revision.
+
+The bulk benchmark uses fresh sequential processes and records conversion,
+normal runtime activation, memory samples and checked runtime lookups for
+1,000, 10,000 and 100,000 synthetic redirects. Splitting the last dataset into
+100 includes avoids the earlier single-document worker memory failure without
+increasing the 256 MiB worker heap or ten-second loader deadline. This is local
+capacity evidence, not a cross-platform SLO or peak-memory bound.
+
+### Deferred follow-up: live provider testing (non-blocking)
+
+Decision, 2026-09-17: defer live Cloudflare, AWS and Vercel testing and return to
+it later. This does not block the current implementation work or pull-request
+review and merge, subject to the normal required checks and authorization.
+Provider deployment status remains **unverified** until real tests are recorded;
+deferral does not change capability claims or remove other release/security gates.
+
+When resumed, choose a provider and supply an operator-owned test account/project
+with normal local login access and authorization for a temporary deployment
+(including any hosting charges), or supply an already deployed conformance
+fixture URL. Run the existing provider verification tool and record the results.
+No credentials or provider setup are needed from the user for the current work.
+
+Actual provider provisioning/deployment observations require operator-owned
+accounts and explicit fixture URLs. Real ingress normalization, repeated header
+and cookie behavior, distributed policy guarantees, soak/recovery tests and
+independent assessment of the new network bridge remain separate release and
+operational gates. No implementation test, capability report, benchmark, or CI
+pass substitutes for that evidence. The self-hosted release remains useful and
+portable without requiring provider accounts or a paid control plane.