hatch3r 1.7.0 → 1.7.5

package/rules/hatch3r-operability.mdc

@@ -0,0 +1,145 @@
+---
+description: Operability patterns in user code — liveness / readiness / startup probes, graceful shutdown, feature flags, runbook URL annotations, health endpoints
+globs: ["**/services/**", "**/handlers/**", "**/health*", "**/probes/**", "**/k8s/**", "**/manifests/**", "**/charts/**", "**/feature*", "**/flags/**"]
+alwaysApply: false
+---
+# Operability
+
+## Liveness, Readiness, and Startup Probes (Kubernetes)
+
+Three probe kinds; each answers a different question and triggers a different action. Conflating them is the most common operability bug — a liveness probe that reaches downstream causes pod-restart cascades on dependency outages.
+
+- **Liveness** — shallow self-check: process alive, event loop responsive, no deadlock on the request handler. NEVER checks external dependencies. Failure → kubelet restarts the pod. Default: `httpGet /health/live`, `periodSeconds: 10`, `failureThreshold: 3`, `timeoutSeconds: 1`.
+- **Readiness** — deep dependency check: DB reachable, cache reachable, required downstream healthy, migrations applied. Failure → endpoint controller removes the pod from Service load-balancer rotation; pod stays running and recovers when dependencies return. Default: `httpGet /health/ready`, `periodSeconds: 5`, `failureThreshold: 2`, `timeoutSeconds: 2`.
+- **Startup** — same checks as readiness but with longer initial allowance for slow-starting apps (large model load, warm cache build, JIT warmup, schema migration). Once succeeds once, liveness and readiness take over. Default: `httpGet /health/startup`, `periodSeconds: 5`, `failureThreshold: 60` (allowing 5 minutes), `timeoutSeconds: 2`.
+
+Concrete examples per ecosystem:
+
+- **Node Express** — `/health/live`, `/health/ready`, `/health/startup` on the same router with different handler chains. The live handler returns 200 unconditionally if the event loop is responsive; the ready handler awaits `Promise.allSettled` over DB ping, cache ping, downstream ping with per-check 1s timeout.
+- **Spring Boot Actuator** — `/actuator/health/liveness` and `/actuator/health/readiness` exposed out of the box; add startup via a custom `HealthIndicator`. Configure `management.endpoint.health.probes.enabled=true`.
+- **Go (net/http)** — three handlers on `http.ServeMux`; ready handler aggregates checks via `errgroup.Group.Wait()` with per-check `context.WithTimeout`.
+
+Anti-pattern: a single `/health` endpoint that both Kubernetes probes hit. The pod is killed during DB outage because the liveness probe failed on the same deep check the readiness probe is supposed to own.
+
+## Graceful Shutdown
+
+Handle `SIGTERM` per the sequence below. Skipping the preStop delay causes the well-known endpoint-propagation race that drops up to several seconds of in-flight traffic.
+
+- Step 1: Stop accepting new connections (close the HTTP listener; stop pulling from the queue).
+- Step 2: Mark `/health/ready` to return 503 (the endpoint controller removes the pod from the Service).
+- Step 3: Wait `preStop` window of 1–3s for endpoint-removal propagation. Kubernetes does NOT propagate endpoint removal before delivering SIGTERM — without an explicit `preStop` `sleep`, traffic continues to arrive for the first 1–3s of shutdown.
+- Step 4: Drain in-flight requests (configurable deadline 30–45s; cap by `terminationGracePeriodSeconds`).
+- Step 5: Close DB connections, drain queue consumers, flush log buffers, close OpenTelemetry exporters.
+- Step 6: Exit 0.
+
+Pod manifest baseline:
+
+```
+terminationGracePeriodSeconds: 45
+lifecycle:
+  preStop:
+    exec:
+      command: ["sh", "-c", "sleep 3"]
+```
+
+Force kill after grace period defeats the drain. If 45s is insufficient, raise `terminationGracePeriodSeconds` rather than skip the drain — but investigate why the service holds long-running requests at shutdown.
+
+For queue consumers (Kafka, NATS, SQS): on SIGTERM stop pulling new messages first, finish in-flight processing, then commit offsets and disconnect. Skipping the commit step on shutdown produces duplicate processing on the next pod's first poll.
+
+## Feature Flags — OpenFeature
+
+OpenFeature is a CNCF Incubating SDK (pre-1.0 spec) that wraps any provider — LaunchDarkly, Unleash, Flagsmith, GrowthBook, Statsig — behind a consistent API. New code targets OpenFeature; provider-specific SDK imports become a finding.
+
+Flag types by lifecycle:
+
+- **Release flag** — gate new code path during rollout. Remove within 1 sprint of full enablement; otherwise it becomes flag debt.
+- **Experiment flag** — A/B test traffic split for measurement. Retire after the experiment concludes and the decision is shipped.
+- **Ops flag** — kill switch for risky features and feature-level circuit breakers. Permanent by design. Cross-reference the kill-switch pattern below.
+- **Permission flag** — entitlement gating (per-tenant feature availability). Prefer reusing the authorization layer where possible; flag only the surface that the auth system does not naturally cover.
+
+Flag-debt budget: 50–100 active flags per service. Run a monthly cleanup cadence — list flags with `enabled = true for 100% of traffic for >30 days` and either remove them or convert to permanent config.
+
+Default-off for new release flags; default-on only for kill switches (so the absence of provider connectivity does not silently disable the feature the on-call needs to disable). Evaluate flags at request entry once and pass the resolved value down — re-evaluating mid-request risks split-brain behavior when the flag flips during the call.
+
+## Kill-Switch Pattern
+
+Every risky feature ships with a kill switch (Ops flag) and a documented procedure for flipping it. On-call must be able to disable the feature without redeploying. Document the flag name in `docs/runbooks/<service>.md` alongside the alert.
+
+Test the kill switch on every release — a kill switch that nobody has flipped in production is unverified. Quarterly drill: flip the flag, observe the metric drop, restore.
+
+Cross-reference `rules/hatch3r-progressive-delivery.md` — kill switch is the rollback mechanism when the staged rollout has already finished and the regression is discovered after 100%.
+
+## Runbook URL on Every Alert
+
+Alert without a runbook is a finding under this rule. Every alert carries a runbook URL annotation:
+
+- **Prometheus:** `annotations.runbook_url: "https://internal.example.com/runbooks/<alert-name>.md"`.
+- **Datadog / Grafana:** equivalent `runbook_url` or `notification_message` template field.
+
+Runbook format:
+
+- **Symptoms** — what the on-call sees on the dashboard or in the alert payload.
+- **Triage** — first three commands or queries to narrow the cause.
+- **Mitigation** — kill switch, rollback command, scale action.
+- **Root cause** — links to past postmortems for similar symptoms.
+- **Follow-ups** — open issues, related dashboards, owner team.
+
+Empirical observation from 2024–2026 incidents: LLM-driven auto-diagnosis quality is dominated by runbook quality rather than by model choice. A high-fidelity runbook with concrete commands beats a generic prompt against a frontier model on the same dataset.
+
+Runbooks live in the service repository under `docs/runbooks/<alert-name>.md` so they ship with the code that emits the alert. Renaming an alert without updating the runbook URL produces a 404 link from the alert payload — a CI check on the alert manifest catches this on every PR that touches alert names.
+
+## Health Endpoint Conventions
+
+- `/health/live` — 200 + `{ "status": "ok", "version": "<semver>", "build_sha": "<short-sha>" }` on success; 503 + `{ "status": "down", "reason": "<diagnostic>" }` on failure.
+- `/health/ready` — same shape; 503 includes the failing downstream name (e.g. `reason: "postgres-primary unreachable"`).
+- `/health/startup` — same as ready but with allowance for warmup.
+
+Never expose secrets, connection strings, or per-request internals from health endpoints — they are unauthenticated by default. Detailed diagnostics live behind authentication on a separate admin endpoint.
+
+Probe response time budget: under 100ms for `/health/live`, under 500ms for `/health/ready` and `/health/startup`. A probe that times out the kubelet causes pod restart cascades unrelated to the underlying issue.
+
+Cache the ready-check result for 1–2s — Kubernetes polls every 5s by default; checking each downstream on every poll multiplies dependency load by the replica count.
+
+## Capacity Planning
+
+- Nightly load test in CI against a staging environment representative of production. Compare baseline-vs-current p50, p95, p99 latency and error rate; fail the run on a 20% regression vs the previous green baseline.
+- Saturation tracking — alert when CPU sustained above 70% for 15 minutes, memory above 80% for 15 minutes, connection pool above 80% utilization for 5 minutes.
+- Rightsizing — target 20–30% headroom on CPU and memory at peak. Below 10% headroom is undersized; above 50% sustained is oversized and a cost finding.
+- HPA (Horizontal Pod Autoscaler) target CPU at 60–70% — leaves room for the scale-up lag to bring new replicas online before the existing pods saturate.
+- Cold-start budget for serverless and JVM services: measure p95 cold-start latency; provision the warm-pool to keep cold-start traffic below 1% of total. KEDA or platform autoscaler keeps the warm-pool at the right size.
+
+## Multi-Region and Disaster Recovery
+
+- RTO (Recovery Time Objective) and RPO (Recovery Point Objective) documented per service tier in the service catalog.
+- Active-active for tier-1 services targeting RTO under 5 minutes and RPO under 1 minute.
+- Active-passive acceptable for tier-2 services targeting RTO under 1 hour.
+- Failover drill quarterly; the drill is the test of the runbook. A runbook that has not been executed in a drill is a draft, not a runbook.
+- Data residency — honor regional data boundaries (EU-US DPF, regional cloud regions). Cross-region replication respects the residency contract.
+- DNS-based failover (Route 53, Cloud DNS) has a propagation tail measured in minutes; for sub-minute RTO use load-balancer-level failover (cross-region target groups) or anycast.
+
+## 2024–2026 Outage Lessons
+
+- **CrowdStrike, July 2024** — global config push without canary. Mitigation: staged rollout (cross-reference `rules/hatch3r-progressive-delivery.md`).
+- **AWS us-east-1, October 2025** — cascading failure across services with hidden us-east-1 control-plane dependency. Mitigation: dependency mapping, multi-region for control plane.
+- **Azure East-US2, September 2025** — single-region outage on customer-perceived multi-region service. Mitigation: active-active across regions.
+- **Cloudflare, November 2025** — config-change incident. Mitigation: treat config as code, canary config changes.
+
+Most outages have an organizational root cause (change management, capacity planning, ownership). The patterns above defend against the technical leg of the failure; organizational defenses are out of scope for this rule.
+
+Postmortem cadence: within 5 business days of incident close, blameless, owned by the on-call rotation, action items tracked in the service backlog with target due dates. Postmortem reviewers cross-check that the runbook was updated and that an automated detection (alert, dashboard, or unit test) was added for the failure mode that escaped detection.
+
+## Cross-References
+
+- `rules/hatch3r-resilience-patterns.md` — circuit breakers, retries, timeouts.
+- `rules/hatch3r-progressive-delivery.md` — canary, blue-green, kill-switch usage during rollout.
+- `rules/hatch3r-observability-metrics.md` — SLOs, RED metrics, burn-rate alerts feed the runbook.
+
+## References
+
+- Kubernetes probes — `kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes`
+- Kubernetes pod termination — `kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination`
+- Google SRE workbook, graceful-shutdown chapter — `sre.google/workbook`
+- OpenFeature specification — `openfeature.dev`
+- LaunchDarkly docs — `docs.launchdarkly.com`
+- Unleash docs — `docs.getunleash.io`
+- 2024–2026 outage postmortems: CrowdStrike Jul 2024, AWS us-east-1 Oct 2025, Azure East-US2 Sep 2025, Cloudflare Nov 2025.

package/rules/hatch3r-passkey-server.md

@@ -0,0 +1,181 @@
+---
+id: hatch3r-passkey-server
+type: rule
+description: Server-side WebAuthn / passkey ceremony — registration, authentication, attestation, counter, RP-ID, recovery, FIDO CXP/CXF awareness
+scope: "**/auth/**,**/passkey*,**/webauthn*,**/fido*,**/credentials/**,**/api/**,**/handlers/**"
+tags: [security, implementation]
+quality_charter: agents/shared/quality-charter.md
+cache_friendly: true
+---
+# Passkey Server — WebAuthn Ceremony
+
+## Scope
+
+Server-side WebAuthn / passkey implementation. Companion to `rules/hatch3r-ux-states-and-flows.md` and `rules/hatch3r-ai-ux-patterns.md` (frontend Conditional UI), and to `rules/hatch3r-auth-patterns.md` (auth model, AAL mapping, step-up). WebAuthn Level 3 (W3C Candidate Recommendation, Jan 2026) is the spec baseline.
+
+## Server Libraries — Use Vetted, Never Hand-Roll
+
+| Runtime | Library |
+|---------|---------|
+| Node.js / Deno / Bun | `@simplewebauthn/server` |
+| JVM (Java / Kotlin) | `webauthn4j` |
+| Go | `go-webauthn/webauthn` |
+| Python | `py_webauthn` (Duo) |
+| Rust | `webauthn-rs` |
+| .NET | `Fido2NetLib` |
+
+Hand-rolling the ceremony (signature parsing, CBOR decoding, attestation chain validation) is a Critical anti-pattern.
+
+## Registration Ceremony
+
+Two HTTP endpoints. Always.
+
+### `POST /webauthn/register/options`
+
+Server generates and returns `PublicKeyCredentialCreationOptions`:
+
+- `challenge` — 32 random bytes, cached server-side with a 5-minute TTL keyed to the user session.
+- `rp.id` — registrable domain (e.g., `example.com`). Never include scheme or port.
+- `rp.name` — human-readable RP name shown in the OS passkey UI.
+- `user.id` — server-side opaque ID (16-64 random bytes), NOT email and NOT username. Stable per account; never reused after deletion.
+- `user.name`, `user.displayName` — visible identifiers (e.g., email + display name).
+- `pubKeyCredParams` — `[{type: "public-key", alg: -7 /* ES256 */}, {type: "public-key", alg: -257 /* RS256 */}]`.
+- `authenticatorSelection.userVerification: "required"` for new accounts.
+- `authenticatorSelection.residentKey: "required"` to enable discoverable / passkey-first UX.
+- `attestation: "none"` for consumer apps. `"direct"` only when high-assurance attestation is required (enterprise / regulated).
+- `excludeCredentials` — array of the user's already-registered credential IDs to prevent duplicate registration on the same authenticator.
+
+### `POST /webauthn/register/verify`
+
+Client returns `AuthenticatorAttestationResponse`. Server verifies:
+
+- Challenge matches the cached challenge for this session; consume after verification.
+- `origin` is in the allowlist (typically `https://{rp.id}` + known origins).
+- RP-ID hash in `authData` matches SHA-256 of `rp.id`.
+- Signature validates against the attestation public key (when attestation requested).
+- Attestation chain validates against FIDO Metadata Service when `attestation: "direct"`.
+
+Persist on success: `credential_id` (base64url), `public_key` (COSE-encoded), `counter`, `aaguid`, `transports[]`, `backup_eligible`, `backup_state`, `user_id` (FK), `created_at`.
+
+## Authentication Ceremony
+
+### `POST /webauthn/login/options`
+
+Server returns `PublicKeyCredentialRequestOptions`:
+
+- `challenge` — fresh 32 random bytes, cached with 5-minute TTL keyed to the login attempt ID.
+- `rp.id` — same registrable domain as registration.
+- `allowCredentials` — array of the user's credential IDs when identifier is known. Empty array (or omit) for discoverable login (passkey-first UX); the OS picker handles credential selection.
+- `userVerification: "required"` for the AAL2-required path.
+
+### `POST /webauthn/login/verify`
+
+Client returns `AuthenticatorAssertionResponse`. Server verifies:
+
+- Challenge matches; consume after verification.
+- `origin` is in the allowlist.
+- RP-ID hash matches SHA-256 of `rp.id`.
+- Signature validates against the stored public key for the asserted credential ID.
+- Counter check (see below).
+
+On success, update the stored counter and `backup_state`; issue the session.
+
+## RP-ID Rules
+
+- `rp.id` is the registrable domain. For `app.example.com`, `rp.id` may be `example.com` (broader) or `app.example.com` (narrower). Broader RP-ID allows credentials to work across subdomains.
+- Credentials are bound to the RP-ID used at registration. Changing the RP-ID later requires re-registration.
+- Multi-region domains: plan the RP-ID before launch. Use Related Origin Requests (WebAuthn L3) for legitimate multi-origin scenarios.
+
+## Counter Checking — Clone Detection
+
+Every authenticator increments its signature counter on each assertion (some authenticators report 0 always — document expected behavior per AAGUID).
+
+- On registration, persist the initial counter.
+- On authentication, the asserted counter MUST be greater than the stored counter.
+- If asserted counter <= stored counter (and not both 0): treat as possible clone. Reject the assertion, notify the user via email, and require recovery.
+- Always update the stored counter after successful authentication.
+
+## AAGUID Handling
+
+- `aaguid` (Authenticator Attestation GUID) identifies the authenticator model family.
+- When `attestation: "none"`, do NOT use AAGUID for security decisions — it is unattested and can be spoofed.
+- Use AAGUID for analytics (which device families users prefer) and for cross-referencing the FIDO Metadata Service when `attestation: "direct"`.
+
+## Backup State Flags
+
+WebAuthn L3 introduces two single-bit flags in `authData`:
+
+- `backup_eligible` (BE) — credential CAN be backed up / synced (passkey).
+- `backup_state` (BS) — credential IS currently backed up.
+
+Persist both. Display "passkey synced across your devices" UI hint when `BS=1` (cross-reference frontend `rules/hatch3r-ux-states-and-flows.md`). Flag `BE=0` credentials as device-bound (security key) in recovery flows — losing the device means losing the credential.
+
+## Recovery Patterns
+
+- **Multiple passkeys per user** — REQUIRED. Encourage registration of >=2 authenticators (laptop sync + phone, or laptop + hardware key) during onboarding. UI surfaces missing-second-passkey state as a recoverable warning.
+- **Account recovery flow** — verified email + identity-proofing question + step-up auth via remaining passkey. Never plain SMS. Document the recovery SLA.
+- **Admin recovery** — enterprise tenants get an admin-initiated recovery path with audit trail; consumer accounts do not.
+- **FIDO CXP/CXF awareness** — Cross-Platform Export Format (Feb 2026 draft) enables passkey migration across managers (1Password, Apple Keychain, Google Password Manager, Bitwarden). Plan the migration UX so users discover that exporting from manager A to manager B is supported.
+
+## User Verification (UV)
+
+- `userVerification: "required"` — authenticator must verify the user via biometric or PIN. Replaces the password factor. Use for AAL2 and all sensitive operations.
+- `userVerification: "preferred"` — authenticator verifies if able, otherwise presence-only. Acceptable for AAL1.
+- `userVerification: "discouraged"` — presence only. Rare; only for tap-to-confirm flows.
+- Map per AAL level — cross-reference `rules/hatch3r-auth-patterns.md` MFA section.
+
+## Step-Up Authentication
+
+High-risk operations require a fresh WebAuthn assertion even when the session is valid:
+
+- Delete account, change email, change password.
+- Rotate API key, add OAuth client, change billing.
+- Transfer funds above the project's risk threshold.
+
+Issue a short-lived (5-15 min) step-up token scoped to the operation. Re-assertion uses `userVerification: "required"` regardless of session AAL.
+
+## Discoverable Credentials (Passkey-First UX)
+
+- Register with `residentKey: "required"` so the credential is stored on the authenticator with a username hint.
+- Authenticate with empty `allowCredentials` and frontend `mediation: "conditional"` (Conditional UI) so the username field offers passkeys directly.
+- Cross-reference `rules/hatch3r-ux-states-and-flows.md` for the frontend autocomplete attribute pattern.
+
+## Telemetry
+
+Track per-feature:
+
+- Registration success rate, registration failure rate by reason.
+- Authentication success rate, authentication failure rate by reason.
+- Time-to-completion for registration and assertion (P50/P95).
+- AAGUID distribution.
+- `backup_state` rate (sync vs device-bound).
+- Counter-mismatch incidents (potential clone).
+
+Cross-reference `skills/hatch3r-observability-verify` for instrumentation patterns.
+
+## Migration From Passwords
+
+Opt-in flow after a successful password+TOTP login:
+
+1. Prompt "Add a passkey for faster sign-in?" with one-tap registration.
+2. On success, mark the account `passkey_enabled: true`.
+3. After >=1 passkey is registered AND the user has confirmed passkey works on >=2 sessions, prompt "Switch to passkey-only sign-in?"
+4. On opt-in confirmation, disable password sign-in for the account. Keep a recovery path (email + secondary passkey).
+
+## Anti-Patterns
+
+- Never store the private key server-side. Passkeys are public-key only — the private key never leaves the authenticator.
+- Never use email as `user.id`. Use a server-side opaque ID; email changes break credential binding otherwise.
+- Never accept attestation as user identity. Attestation identifies the authenticator model, not the human.
+- Never skip the origin check. RP-ID alone does not prevent phishing — origin allowlist does.
+- Never log credential public keys or signatures in plaintext audit logs.
+- Never accept a counter <= stored counter as success.
+
+## References
+
+- W3C Web Authentication Level 3 (Candidate Recommendation, Jan 2026).
+- FIDO Alliance — Client to Authenticator Protocol (CTAP) 2.2.
+- FIDO Alliance — Credential Exchange Protocol / Credential Exchange Format (CXP / CXF, Feb 2026 draft).
+- `@simplewebauthn/server` documentation.
+- MDN — Web Authentication API.
+- FIDO Metadata Service (MDS3) for attestation verification.

package/rules/hatch3r-passkey-server.mdc

@@ -0,0 +1,177 @@
+---
+description: Server-side WebAuthn / passkey ceremony — registration, authentication, attestation, counter, RP-ID, recovery, FIDO CXP/CXF awareness
+globs: ["**/auth/**", "**/passkey*", "**/webauthn*", "**/fido*", "**/credentials/**", "**/api/**", "**/handlers/**"]
+alwaysApply: false
+---
+# Passkey Server — WebAuthn Ceremony
+
+## Scope
+
+Server-side WebAuthn / passkey implementation. Companion to `rules/hatch3r-ux-states-and-flows.md` and `rules/hatch3r-ai-ux-patterns.md` (frontend Conditional UI), and to `rules/hatch3r-auth-patterns.md` (auth model, AAL mapping, step-up). WebAuthn Level 3 (W3C Candidate Recommendation, Jan 2026) is the spec baseline.
+
+## Server Libraries — Use Vetted, Never Hand-Roll
+
+| Runtime | Library |
+|---------|---------|
+| Node.js / Deno / Bun | `@simplewebauthn/server` |
+| JVM (Java / Kotlin) | `webauthn4j` |
+| Go | `go-webauthn/webauthn` |
+| Python | `py_webauthn` (Duo) |
+| Rust | `webauthn-rs` |
+| .NET | `Fido2NetLib` |
+
+Hand-rolling the ceremony (signature parsing, CBOR decoding, attestation chain validation) is a Critical anti-pattern.
+
+## Registration Ceremony
+
+Two HTTP endpoints. Always.
+
+### `POST /webauthn/register/options`
+
+Server generates and returns `PublicKeyCredentialCreationOptions`:
+
+- `challenge` — 32 random bytes, cached server-side with a 5-minute TTL keyed to the user session.
+- `rp.id` — registrable domain (e.g., `example.com`). Never include scheme or port.
+- `rp.name` — human-readable RP name shown in the OS passkey UI.
+- `user.id` — server-side opaque ID (16-64 random bytes), NOT email and NOT username. Stable per account; never reused after deletion.
+- `user.name`, `user.displayName` — visible identifiers (e.g., email + display name).
+- `pubKeyCredParams` — `[{type: "public-key", alg: -7 /* ES256 */}, {type: "public-key", alg: -257 /* RS256 */}]`.
+- `authenticatorSelection.userVerification: "required"` for new accounts.
+- `authenticatorSelection.residentKey: "required"` to enable discoverable / passkey-first UX.
+- `attestation: "none"` for consumer apps. `"direct"` only when high-assurance attestation is required (enterprise / regulated).
+- `excludeCredentials` — array of the user's already-registered credential IDs to prevent duplicate registration on the same authenticator.
+
+### `POST /webauthn/register/verify`
+
+Client returns `AuthenticatorAttestationResponse`. Server verifies:
+
+- Challenge matches the cached challenge for this session; consume after verification.
+- `origin` is in the allowlist (typically `https://{rp.id}` + known origins).
+- RP-ID hash in `authData` matches SHA-256 of `rp.id`.
+- Signature validates against the attestation public key (when attestation requested).
+- Attestation chain validates against FIDO Metadata Service when `attestation: "direct"`.
+
+Persist on success: `credential_id` (base64url), `public_key` (COSE-encoded), `counter`, `aaguid`, `transports[]`, `backup_eligible`, `backup_state`, `user_id` (FK), `created_at`.
+
+## Authentication Ceremony
+
+### `POST /webauthn/login/options`
+
+Server returns `PublicKeyCredentialRequestOptions`:
+
+- `challenge` — fresh 32 random bytes, cached with 5-minute TTL keyed to the login attempt ID.
+- `rp.id` — same registrable domain as registration.
+- `allowCredentials` — array of the user's credential IDs when identifier is known. Empty array (or omit) for discoverable login (passkey-first UX); the OS picker handles credential selection.
+- `userVerification: "required"` for the AAL2-required path.
+
+### `POST /webauthn/login/verify`
+
+Client returns `AuthenticatorAssertionResponse`. Server verifies:
+
+- Challenge matches; consume after verification.
+- `origin` is in the allowlist.
+- RP-ID hash matches SHA-256 of `rp.id`.
+- Signature validates against the stored public key for the asserted credential ID.
+- Counter check (see below).
+
+On success, update the stored counter and `backup_state`; issue the session.
+
+## RP-ID Rules
+
+- `rp.id` is the registrable domain. For `app.example.com`, `rp.id` may be `example.com` (broader) or `app.example.com` (narrower). Broader RP-ID allows credentials to work across subdomains.
+- Credentials are bound to the RP-ID used at registration. Changing the RP-ID later requires re-registration.
+- Multi-region domains: plan the RP-ID before launch. Use Related Origin Requests (WebAuthn L3) for legitimate multi-origin scenarios.
+
+## Counter Checking — Clone Detection
+
+Every authenticator increments its signature counter on each assertion (some authenticators report 0 always — document expected behavior per AAGUID).
+
+- On registration, persist the initial counter.
+- On authentication, the asserted counter MUST be greater than the stored counter.
+- If asserted counter <= stored counter (and not both 0): treat as possible clone. Reject the assertion, notify the user via email, and require recovery.
+- Always update the stored counter after successful authentication.
+
+## AAGUID Handling
+
+- `aaguid` (Authenticator Attestation GUID) identifies the authenticator model family.
+- When `attestation: "none"`, do NOT use AAGUID for security decisions — it is unattested and can be spoofed.
+- Use AAGUID for analytics (which device families users prefer) and for cross-referencing the FIDO Metadata Service when `attestation: "direct"`.
+
+## Backup State Flags
+
+WebAuthn L3 introduces two single-bit flags in `authData`:
+
+- `backup_eligible` (BE) — credential CAN be backed up / synced (passkey).
+- `backup_state` (BS) — credential IS currently backed up.
+
+Persist both. Display "passkey synced across your devices" UI hint when `BS=1` (cross-reference frontend `rules/hatch3r-ux-states-and-flows.md`). Flag `BE=0` credentials as device-bound (security key) in recovery flows — losing the device means losing the credential.
+
+## Recovery Patterns
+
+- **Multiple passkeys per user** — REQUIRED. Encourage registration of >=2 authenticators (laptop sync + phone, or laptop + hardware key) during onboarding. UI surfaces missing-second-passkey state as a recoverable warning.
+- **Account recovery flow** — verified email + identity-proofing question + step-up auth via remaining passkey. Never plain SMS. Document the recovery SLA.
+- **Admin recovery** — enterprise tenants get an admin-initiated recovery path with audit trail; consumer accounts do not.
+- **FIDO CXP/CXF awareness** — Cross-Platform Export Format (Feb 2026 draft) enables passkey migration across managers (1Password, Apple Keychain, Google Password Manager, Bitwarden). Plan the migration UX so users discover that exporting from manager A to manager B is supported.
+
+## User Verification (UV)
+
+- `userVerification: "required"` — authenticator must verify the user via biometric or PIN. Replaces the password factor. Use for AAL2 and all sensitive operations.
+- `userVerification: "preferred"` — authenticator verifies if able, otherwise presence-only. Acceptable for AAL1.
+- `userVerification: "discouraged"` — presence only. Rare; only for tap-to-confirm flows.
+- Map per AAL level — cross-reference `rules/hatch3r-auth-patterns.md` MFA section.
+
+## Step-Up Authentication
+
+High-risk operations require a fresh WebAuthn assertion even when the session is valid:
+
+- Delete account, change email, change password.
+- Rotate API key, add OAuth client, change billing.
+- Transfer funds above the project's risk threshold.
+
+Issue a short-lived (5-15 min) step-up token scoped to the operation. Re-assertion uses `userVerification: "required"` regardless of session AAL.
+
+## Discoverable Credentials (Passkey-First UX)
+
+- Register with `residentKey: "required"` so the credential is stored on the authenticator with a username hint.
+- Authenticate with empty `allowCredentials` and frontend `mediation: "conditional"` (Conditional UI) so the username field offers passkeys directly.
+- Cross-reference `rules/hatch3r-ux-states-and-flows.md` for the frontend autocomplete attribute pattern.
+
+## Telemetry
+
+Track per-feature:
+
+- Registration success rate, registration failure rate by reason.
+- Authentication success rate, authentication failure rate by reason.
+- Time-to-completion for registration and assertion (P50/P95).
+- AAGUID distribution.
+- `backup_state` rate (sync vs device-bound).
+- Counter-mismatch incidents (potential clone).
+
+Cross-reference `skills/hatch3r-observability-verify` for instrumentation patterns.
+
+## Migration From Passwords
+
+Opt-in flow after a successful password+TOTP login:
+
+1. Prompt "Add a passkey for faster sign-in?" with one-tap registration.
+2. On success, mark the account `passkey_enabled: true`.
+3. After >=1 passkey is registered AND the user has confirmed passkey works on >=2 sessions, prompt "Switch to passkey-only sign-in?"
+4. On opt-in confirmation, disable password sign-in for the account. Keep a recovery path (email + secondary passkey).
+
+## Anti-Patterns
+
+- Never store the private key server-side. Passkeys are public-key only — the private key never leaves the authenticator.
+- Never use email as `user.id`. Use a server-side opaque ID; email changes break credential binding otherwise.
+- Never accept attestation as user identity. Attestation identifies the authenticator model, not the human.
+- Never skip the origin check. RP-ID alone does not prevent phishing — origin allowlist does.
+- Never log credential public keys or signatures in plaintext audit logs.
+- Never accept a counter <= stored counter as success.
+
+## References
+
+- W3C Web Authentication Level 3 (Candidate Recommendation, Jan 2026).
+- FIDO Alliance — Client to Authenticator Protocol (CTAP) 2.2.
+- FIDO Alliance — Credential Exchange Protocol / Credential Exchange Format (CXP / CXF, Feb 2026 draft).
+- `@simplewebauthn/server` documentation.
+- MDN — Web Authentication API.
+- FIDO Metadata Service (MDS3) for attestation verification.

package/rules/hatch3r-progressive-delivery.md

@@ -0,0 +1,120 @@
+---
+id: hatch3r-progressive-delivery
+type: rule
+description: Progressive delivery — canary, blue-green, feature-flag rollout with auto-rollback on SLO burn; staged rollout to prevent CrowdStrike-class incidents
+scope: "**/.github/workflows/**,**/deploy/**,**/k8s/**,**/manifests/**,**/argo/**,**/flagger/**,**/spinnaker/**,**/rollout*"
+tags: [devops]
+quality_charter: agents/shared/quality-charter.md
+cache_friendly: true
+---
+# Progressive Delivery
+
+## Three Rollout Strategies
+
+Choose one per service based on risk profile and resource budget:
+
+- **Canary** — shift traffic in increments (1% → 10% → 50% → 100%) over hours; analyze metrics at each stage; auto-rollback on SLO burn. Argo Rollouts is the most-deployed Kubernetes-native option; Flagger is the Flux-aligned alternative. Lowest resource overhead; gradual exposure.
+- **Blue-green** — bring the new version fully online alongside the old version; switch traffic atomically; keep old version warm for fast rollback. Higher resource cost (2x at switchover); near-instant rollback. Suits stateful upgrades and small fleets.
+- **Feature-flag rollout** — deploy the new code dark (path exists but is gated); flip an OpenFeature flag to release. Decouples deploy from release; pairs with the kill-switch pattern in `rules/hatch3r-operability.md`.
+
+A service may combine strategies — deploy via blue-green for infrastructure changes, then gate user-visible behavior behind a feature flag for incremental release.
+
+## Canary Analysis
+
+Automate the success / fail decision at each stage; never rely on a human eyeballing the dashboard.
+
+- **Argo Rollouts AnalysisTemplate** — declarative Prometheus / Datadog / New Relic / Wavefront queries with success conditions.
+- **Flagger metric checks** — Prometheus / Datadog / CloudWatch / Stackdriver via `MetricTemplate`.
+- **Datadog deployment tracking** — auto-rollback triggered by deployment-tagged anomaly detection.
+- **Spinnaker / Kayenta** — Mann-Whitney U test on baseline-vs-canary metrics; statistical rigor for low-traffic services where raw thresholds give noisy signals.
+
+Metrics gated at every stage:
+
+- Error-rate ratio (canary vs control) — fail on >1.2x baseline for 2 consecutive intervals.
+- p95 / p99 latency (canary vs control) — fail on >1.2x baseline for 2 consecutive intervals.
+- Business KPIs (checkout success rate, signup completion) — fail on >5% drop from baseline.
+- Saturation metrics (CPU, memory) — fail on canary above 90% when control is below 70%.
+
+Two consecutive intervals (typically 1-minute scrape windows) avoids single-scrape noise tripping a false rollback.
+
+Baseline definition: a stable subset of production pods running the prior version, receiving the same traffic mix as the canary. Comparing canary against absolute thresholds rather than against a live baseline produces false positives during traffic spikes and false negatives during quiet windows.
+
+## SLO-Burn Auto-Rollback
+
+Wire canary analysis to the service's SLO (cross-reference `rules/hatch3r-observability-metrics.md`). If a multi-window multi-burn-rate alert fires during a canary stage, rollback automatically — do not wait for the next stage gate.
+
+Two-window burn-rate alert pattern: trigger rollback when error budget burns at 14.4x over a 5-minute window AND 6x over a 1-hour window (the Google SRE "fast burn" pattern). Slow-burn alerts (3x over 6h AND 1x over 3d) do not auto-rollback — they page the on-call.
+
+## Staged Rollout Cadence
+
+Default cadence for non-trivial production deploys, enforced by the rollout controller:
+
+- Stage 1: 1% for 30 minutes minimum.
+- Stage 2: 10% for 1 hour minimum.
+- Stage 3: 50% for 2 hours minimum.
+- Stage 4: 100%.
+
+Override only with explicit on-call approval, documented in the deploy log. CrowdStrike July 2024 root cause: a global config push without staged exposure; a 1% canary would have detected the kernel panic before reaching any production tenant.
+
+Trivial deploys (docs-only, config-only with prior canary, dependency patch with no behavior change) may skip stages with a documented justification in the PR.
+
+Stage holding times scale with peak-hour traffic — a 30-minute stage at 03:00 may see less traffic than a 5-minute stage at 13:00. Tune cadence per service so each stage observes at least 1000 canary requests; below that the metrics-gate is statistically underpowered.
+
+## Blast-Radius Reasoning
+
+Every PR description includes the blast-radius block:
+
+- **Services affected** — list of deployed services.
+- **Regions** — list of target regions and rollout order.
+- **Traffic %** — peak traffic share over the deploy window.
+- **Rollback time** — measured rollback duration (under 5 minutes is the target).
+- **Rollback command** — exact CLI invocation, copy-pasteable.
+
+A reviewer-enforced checklist item rejects PRs missing the block. Blast-radius is the artifact the incident commander reads at 03:00.
+
+## Deploy Windows and Change Calendar
+
+- No production deploys: Friday 12:00 PM through Monday 09:00 AM in the timezone of the on-call region.
+- No production deploys during named freezes (Black Friday window, end-of-quarter close, major event launch).
+- Hotfixes require explicit incident-commander sign-off, captured in the deploy log.
+
+Deploy-window violations are a finding even when the deploy succeeded — the on-call coverage assumption is what is being tested, not the deploy.
+
+## Rollback Playbook
+
+Every deploy ships a rollback command in the service runbook. Examples:
+
+- Argo Rollouts: `kubectl argo rollouts abort <rollout> -n <ns>` then `kubectl argo rollouts undo <rollout> -n <ns>`.
+- Flagger canary: `kubectl patch canary <name> -n <ns> --type merge -p '{"spec":{"skipAnalysis":false}}'` then trigger a revert deployment.
+- Kubernetes deployment (no rollout controller): `kubectl rollout undo deployment/<name> -n <ns>`.
+
+Time-to-rollback target: under 5 minutes from rollback decision to traffic on the previous version. If the rollback path is untested in the last quarterly drill, the deploy is blocked until the path is verified.
+
+Database migrations follow expand-contract: the new code reads both old and new schema; the migration runs as a separate deploy; the old code path is removed in a subsequent release. Rollback of a deploy that owns its own destructive migration is not possible — separate the migration from the deploy.
+
+## Config-Change Canary
+
+Treat configuration as code — same staged rollout cadence, same auto-rollback wiring. CrowdStrike July 2024 and Cloudflare November 2025 both originated in config changes pushed globally without canary.
+
+- Feature flags rolled out via OpenFeature targeting rules (1% → 10% → 50% → 100% by user / tenant / region) cover application-level config.
+- Infrastructure config (kernel modules, network policies, service-mesh routing) needs a canary cluster or canary node pool, not a global push.
+
+A "config tweak" is a deploy by another name; the change-management envelope is identical.
+
+GitOps-managed clusters (Flux, Argo CD) gain canary semantics for free via Flagger or Argo Rollouts. Direct `kubectl apply` to the cluster bypasses the controller and produces audit-trail gaps — block direct cluster writes outside of break-glass procedures.
+
+## Cross-References
+
+- `rules/hatch3r-operability.md` — kill switches and runbooks for post-rollout recovery.
+- `rules/hatch3r-observability-metrics.md` — SLO definitions and burn-rate alerts that gate the canary.
+- `rules/hatch3r-resilience-patterns.md` — circuit breakers shield the user from a failing canary while the rollback completes.
+
+## References
+
+- Argo Rollouts — `argoproj.github.io/argo-rollouts`
+- Flagger — `flagger.app`
+- Spinnaker — `spinnaker.io`
+- Kayenta — `github.com/spinnaker/kayenta`
+- Datadog deployment tracking — `docs.datadoghq.com/tracing/services/deployment_tracking`
+- Google SRE workbook, alerting chapter (multi-window multi-burn-rate) — `sre.google/workbook/alerting-on-slos`
+- 2024–2026 outage postmortems: CrowdStrike Jul 2024, AWS us-east-1 Oct 2025, Azure East-US2 Sep 2025, Cloudflare Nov 2025.