create-freeholder 0.0.0-bootstrap.0 → 0.2.0-beta.2

package/dist/template/deploy/app-store-privacy.md

@@ -0,0 +1,82 @@
+<!--
+Copyright (C) 2026 Tony Aly
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Apple App Store privacy checklist
+
+*MASTER.md §35, C10.16. Companion: [Play data-safety](play-data-safety.md).*
+
+This is the list an owner walks when submitting the customer app. It is not
+Apple's review, and a green CI run is not a review outcome. Apple accepts or
+rejects the binary; the checklist only makes the answers we give them match
+the permissions the binary actually requests.
+
+CI fails the customer-app job when the demo discovery contract cannot be
+parsed, when an asset `freeholder-app init` writes is missing, or when
+`apps/mobile/app.json` `ios.privacyManifests` does not match those
+permissions. That is the tested half of "submission-ready". The other half is
+this form.
+
+## What the app is
+
+A white-label client of a Freeholder instance. It asks for the business's web
+address, signs the customer in, and calls the instance's public API. There is
+no in-app purchase of anything the platform sells — digital sales stay on the
+website, which is what the store rules permit for goods consumed outside the
+app. There is no third-party analytics SDK and no background location.
+
+## Privacy Nutrition Label
+
+Answer from the privacy manifest in `apps/mobile/app.json`. If you add a
+permission, change the manifest in the same PR; CI will fail a mismatch.
+
+| Data type | Collected? | Linked to identity? | Used for tracking? | Purpose |
+|---|---|---|---|---|
+| Email address | Yes — sign-in | Yes | No | App functionality |
+| Name | Yes — customer profile | Yes | No | App functionality |
+| User ID | Yes — the instance's contact id | Yes | No | App functionality |
+| Phone number | No — the profile may *display* a number the instance already holds; the app does not collect one | — | — | — |
+| Location | No. §35.1 forbids background location. | — | — | — |
+| Photos or videos | No. Galleries are the business's files, not the customer's camera roll. Camera-roll ingest is C10.18 and will have to update this table. | — | — | — |
+| Contacts | No | — | — | — |
+| Product interaction | Optional to declare. Bookings and invoices are app functionality, not tracking. | Yes, if declared | No | App functionality |
+| Advertising data | No | — | — | — |
+| Diagnostics | No third-party crash SDK | — | — | — |
+
+Tracking is **false**. There are no tracking domains.
+
+## PrivacyInfo / required-reason APIs
+
+Declared because Expo dependencies call them, not because we read them
+ourselves:
+
+| API | Reason | Why |
+|---|---|---|
+| UserDefaults (`CA92.1`) | `expo-secure-store` — the session lives in the platform keychain |
+| File timestamp (`C617.1`) | `expo-file-system` — the private cache's bounded files |
+| Disk space (`E174.1`) | `expo-file-system` — refuse a write that would blow the 20 MiB vault |
+
+Do not add Face ID (`NSFaceIDUsageDescription`) unless the binary starts
+requesting biometric unlock at the OS prompt. A fingerprint is a convenience
+over a held session, not a new data type.
+
+## App Review notes
+
+- **Login:** a demo instance (Aurora Coast Photography) or the owner's own
+  site. The app asks for the web address on first launch; it is not compiled
+  for a single tenant.
+- **Screenshots:** `npx freeholder-app init` writes branded placeholders when
+  a simulator capture is not available. Real screenshots from seeded content
+  are better, not required for the command or for CI.
+- **Payments:** invoice payment opens in the system browser. Do not claim
+  in-app purchase.
+- **Push:** device registration exists on the instance; no production push
+  carrier is configured yet, and the binary does not request
+  `POST_NOTIFICATIONS`. Adding `expo-notifications` must update the manifest
+  in the same change.
+
+## Honest limit
+
+Apple's review is Apple's. This document and the CI gate keep our answers
+honest; they do not predict a review outcome.

package/dist/template/deploy/background-jobs.md

@@ -0,0 +1,229 @@
+# Background-job operations
+
+Copyright (C) 2026 Tony Aly
+SPDX-License-Identifier: Apache-2.0
+
+Freeholder runs background work through pg-boss in the same Postgres database
+as the application. It does not require Redis or a hosted queue. pg-boss owns
+its execution tables in the `pgboss` schema. Freeholder owns the bounded
+`job_idempotency_keys` and payload-free `job_runtime_heartbeats` tables in
+`public`.
+
+## The transaction boundary
+
+A service enqueues caused work with `ctx.queueJob(...)`. Infrastructure code
+that already owns a Drizzle transaction calls `enqueueJob(tx, ...)`. There is
+no exported best-effort `send()` function.
+
+The pg-boss row is inserted through its Drizzle transaction adapter. The
+business mutation, audit row, idempotency claim and queue row therefore share
+one commit. If any insert fails or the service throws later, all four roll
+back. Code must not enqueue after a service returns or from an uncommitted
+event listener.
+
+Outbound webhook fan-out is the first production caller of this contract. Its
+delivery rows and immediate `core.deliverWebhooks` nudge are atomic. If queue
+storage is unavailable, the listener fails and the transactional outbox keeps
+the source event for retry.
+
+Any workflow that can wait on DNS, TLS, an object store, or a third-party API
+uses the same boundary. The request service validates the actor and inserts a
+job in its transaction; the worker reads a short service-layer snapshot,
+performs provider I/O with no database transaction open, and calls a narrow
+system mutation to apply the result. Catalogue refresh and social ingest,
+health, GBP and publication delivery are executable examples. The source gate
+in `tests/core/long-running-service-boundary.test.ts` prevents those provider
+calls from drifting back into their service handlers.
+
+## Queue policy
+
+Every `defineJob()` declaration owns these limits beside its handler:
+
+| Policy | Default | Meaning |
+|---|---:|---|
+| retry limit | 5 | Attempts after the first failure |
+| retry delay | 30 seconds | Initial delay |
+| exponential backoff | on | Delay grows with pg-boss jitter |
+| retry ceiling | 1 hour | Maximum delay |
+| concurrency | 1 | Global active jobs for that queue across all replicas |
+| lease | 15 minutes | Lost-worker timeout |
+| heartbeat | 60 seconds | Automatically refreshed by the active worker |
+| terminal history | 30 days | Time pg-boss keeps completed/failed/cancelled rows |
+
+Definitions may tighten or expand these within validated bounds. Freeholder
+passes both local and database-backed group concurrency to pg-boss. Merely
+setting per-process concurrency is insufficient when two app replicas share a
+database.
+
+Queue definitions are reconciled at boot, so retry/lease changes take effect
+on an existing instance without deleting its queued work. Queue names and
+policies are stable contracts; renaming one is a data migration, not a code
+cleanup.
+
+## Idempotency
+
+`idempotencyKey` identifies one logical operation within a job name. Use an
+opaque business identifier such as `invoice-send:<invoice-id>:<revision>`;
+never put a password, API token, message body or other secret in the key.
+
+- Matching name, key and canonical payload return the original job ID with
+  `deduplicated: true`.
+- Reusing the same name/key with different data fails the caller transaction.
+  It is a programming error, not permission to replace already queued work.
+- Concurrent callers serialize on the database unique index and converge on
+  one job.
+- Claims live for 30 days by default. Callers may set a bounded TTL from one
+  minute to one year when the business operation has a different replay
+  horizon.
+- `core.pruneJobKeys` deletes expired claims daily. Once a claim expires, the
+  key may intentionally represent a new operation and receives a new job ID.
+
+Handlers still must tolerate retries. An idempotency key prevents duplicate
+enqueue requests; it does not make an external provider atomic with Postgres.
+A mail, payment or webhook handler uses the provider's own idempotency key or
+records its provider reference before declaring success.
+
+## Leases and cancellation
+
+pg-boss marks a fetched job active and Freeholder configures a heartbeat for
+every queue. The worker refreshes it automatically. If a process disappears,
+the supervisor returns the expired job to retry until its retry limit is
+exhausted.
+
+Handlers receive a `JobExecutionContext` with:
+
+- `signal`, aborted during worker shutdown;
+- `attempt`, starting at one;
+- `heartbeat()`, for an explicit safe-point refresh;
+- `isCancelled()` and `throwIfCancelled()`, for durable cooperative
+  cancellation.
+
+Queued and retrying work stops before execution when cancelled. Active code
+cannot be safely interrupted in the middle of an arbitrary external side
+effect, so long handlers call `throwIfCancelled()` between atomic units. A
+handler must not swallow `JobCancelledError` and continue.
+
+`cancelJob(tx, name, id)` and `retryJob(tx, name, id)` accept the caller's
+transaction. The permissioned services and owner console use these same
+transactional primitives; the UI does not edit pg-boss tables directly.
+
+## Owner history and recovery
+
+Staff with `platform` view access can inspect retained work at `/admin/jobs`.
+The summary and filters read pg-boss itself, so the screen cannot drift from a
+shadow history table. Run detail shows queue policy, retry count, lease times,
+source identity and retained payload/output. Secret-shaped fields are redacted
+recursively before they reach the UI. Agents and API keys cannot inspect this
+surface even when they hold a broad platform scope because job payloads may
+contain cross-module customer data.
+
+Every ordinary queue sends work that exhausts its retry policy to the retained
+`core.deadLetter` queue. Dead letters keep their original queue, run ID, age
+and retry count for 90 days. No worker consumes that queue automatically: a
+person must inspect the failure, correct its cause, and deliberately redrive
+it. The overview briefing calls out failed, dead-lettered and lease-overdue
+work without exposing payload data.
+
+Cancel, retry and redrive require `platform` manage access, a fresh step-up
+proof and exact typed confirmation (`CANCEL`, `RETRY`, or `REDRIVE`). Each
+action runs through a service transaction, writes the audit trail and emits a
+committed event. Redrive is intentionally rate-shaped one retained failure at
+a time in the owner UI; automation can use the same service with a bounded
+batch after a human has diagnosed the source queue.
+
+An active run is shown as lease-overdue when its latest heartbeat (or start,
+before the first heartbeat) is older than the configured heartbeat/lease
+window. That is an operational signal, not permission to enqueue a duplicate:
+the pg-boss supervisor remains responsible for lease expiry and retry.
+
+## Process layouts
+
+The default is one process: instrumentation registers definitions, reconciles
+queues and schedules, then mounts workers. This is the correct Tier-1 shape
+for a single Replit or Droplet deployment.
+
+`FREEHOLDER_JOBS=off` prevents that process from executing handlers. It still
+starts the producer half, because web requests must enqueue transactionally
+for a dedicated worker. A separate process must run the same image with
+`FREEHOLDER_JOBS=on`. Readiness and Doctor inspect durable heartbeats across
+all processes, so a web process with workers disabled is healthy only while a
+current-version worker is actually alive.
+
+During `next build`, neither producer nor worker starts. In tests, workers are
+off unless explicitly forced on, preventing maintenance schedules from racing
+database fixtures.
+
+## Graceful shutdown
+
+The production process owns `SIGINT` and `SIGTERM` through Next's documented
+`NEXT_MANUAL_SIG_HANDLE=true` contract. `pnpm start` sets the variable before
+loading the Next CLI, and the standalone container sets it before starting
+`server.js`. Do not remove either setting or add a second signal handler: two
+independent shutdown paths can race and return an active job to the queue while
+its original handler is still running.
+
+On either signal Freeholder stops startup retries, marks the queue runtime as
+stopping, asks pg-boss to drain active leases for up to 30 seconds, records the
+stopped heartbeat, and then exits with the signal's conventional status. The
+process-wide deadline is 35 seconds so the final heartbeat has time to commit.
+If draining fails or exceeds that deadline, Freeholder logs only the stable
+phase message and exits unsuccessfully; the expired lease is then recovered by
+pg-boss rather than being silently abandoned.
+
+Give the process at least 40 seconds of termination grace in Docker, a platform
+service definition, or a systemd unit. A shorter supervisor timeout can send
+`SIGKILL` before the queue lease is released. Development uses Next's normal
+signal handling; the manual contract is installed only by the production
+entrypoints.
+
+## Runtime health and alerts
+
+Every queue runtime writes one complete heartbeat every 15 seconds. A row
+identifies only the runtime role and platform version plus aggregate queue
+counts; it never stores a queue name, job ID, payload or error message. Rows
+older than seven days are removed during heartbeat maintenance.
+
+`GET /api/health/live` proves only that the web process can answer. Use it for
+liveness, where a dependency outage must not cause a restart loop.
+`GET /api/health` is readiness: it boots the request graph and then requires a
+healthy heartbeat from a worker running the exact platform version. A
+heartbeat is stale after 45 seconds. During a rolling deploy, the new web
+version therefore stays out of service until its matching worker is live.
+
+Startup migration, module synchronization, demo seed and worker mounting are
+retried in-process with bounded exponential backoff. A database outage keeps
+liveness available and readiness unavailable; recovery does not require a
+manual process restart. Startup logs name only the failed phase and never the
+caught database/provider message.
+
+The readiness body exposes only aggregate evidence: producer and worker
+counts, heartbeat age, queued/ready/active/failed counts, dead-letter count and
+the oldest runnable queue lag. Deferred work is not lag. A runnable job older
+than five minutes marks the worker degraded and makes readiness fail. Dead
+letters mark the report degraded but do not make a capable worker unavailable;
+they require human recovery rather than a restart.
+
+Runtime errors are logged as a bounded phase and stable code. Raw errors are
+not logged because database and provider messages can contain customer data.
+Queue-lag and dead-letter alerts likewise contain counts only and log on state
+transitions rather than every heartbeat.
+
+## Failure and recovery checklist
+
+1. Run `pnpm doctor` and inspect the live worker heartbeat, queue lag and
+   dead-letter count.
+2. Confirm at least one current-version worker has database connectivity and
+   is not configured with `FREEHOLDER_JOBS=off`.
+3. Preserve both the `public` and `pgboss` schemas in database backups. A
+   backup that excludes pg-boss can lose committed pending work.
+4. Restart the worker. Active jobs whose leases lapse retry automatically;
+   do not insert replacement rows by hand.
+5. Open `/admin/jobs`, inspect the sanitized failure and fix the underlying
+   code or configuration. Retry a retained failed/cancelled run, or redrive a
+   dead letter, only after the cause is understood.
+6. A repeated deterministic failure is code or configuration trouble. Raising
+   its retry limit only delays the diagnosis.
+
+Never log full job payloads merely to diagnose queue state. Payloads may carry
+customer data; IDs, names, attempts, states and sanitized error messages are
+the operational surface.

package/dist/template/deploy/browser-journeys.md

@@ -0,0 +1,78 @@
+# Real-browser product journeys
+
+Copyright (C) 2026 Tony Aly
+SPDX-License-Identifier: Apache-2.0
+
+Freeholder runs its primary owner journey against the production build in
+Chromium. The service tests remain the detailed source of truth for validation
+and permission matrices; this gate proves that the browser, Server Actions,
+redirects, cookies, rendered pages and external protocols still join up.
+
+## What the journey proves
+
+`tests/browser/journeys.spec.ts` starts with an empty migrated database and
+walks one continuous instance through:
+
+- first-boot owner, business and location setup, including the permanent
+  wizard lock and the narrow pre-2FA setup authorization boundary;
+- authenticator enrollment, one-time recovery-code display, wrong-password
+  feedback, password/TOTP login and session establishment;
+- form creation, page creation, schema-derived block editing, autosave and
+  deliberate publication;
+- the public form's signed timestamp, submission confirmation, submission
+  inbox and canonical Contact resolution;
+- direct Contact creation and its shared detail record;
+- human-reviewed `fr-CA` translation and localized public rendering;
+- least-privilege API-key creation, one-time token reveal, MCP tool discovery
+  and a scoped `contacts.list` JSON-RPC call; and
+- forgotten-password request, one-time reset, all-session revocation, refusal
+  of the old password and restoration with the new password plus a recovery
+  code.
+
+Production never reveals a reset token. The test follows the public forgot
+form, then replaces only the resulting disposable database row's token hash
+with the SHA-256 hash of a known test token before opening `/reset`. This is a
+fixture seam, not an application endpoint or authentication bypass. Every
+other credential is created and consumed through the rendered product.
+
+## C11 cross-module journeys
+
+`pnpm test:journeys` also runs `tests/browser/c11-*.spec.ts`. Those proofs
+start from a seeded owner (first-boot remains C1.22) and walk the visitor-
+facing seams C11.01 and C11.04 name: a cookieless localized enquiry/chat,
+HTTP `quotes.setConversion`, quote accept, contract sign, C5.25 customer
+invoice pay with `paidMinor`, contact timeline copy, `/admin/reports` after
+pay, and the app-free `/capture/[token]` ingest page.
+
+The rest of C11.01–C11.08 is mixed vitest composition in
+`tests/core/c11-0*.test.ts`: catalog tax/stock/restock checkout, availability
+query plus deposit/signed waiver/due reminder, interrupted capture then
+gallery/social, subscription change/cancel with `hasAccess` after expiry,
+agent approval plus builder prompt/apply/rollback and code gates, mocked
+mail/calendar playbooks, and demo/import-ledger/update rollback. Hosted
+Stripe/PayPal charges, live OAuth, live multi-network publish, and
+WordPress-to-CMS page materialization are not claimed — adapter doubles and
+the import ledger are named in MASTER.md §43.16. C11.08 stays open.
+
+## Run it locally
+
+Set `TEST_DATABASE_URL` or `BROWSER_DATABASE_URL` to a throwaway PostgreSQL
+database whose name contains `test` or `a11y`. The configuration refuses any
+other database name before connecting or truncating.
+
+```bash
+pnpm exec playwright install chromium
+pnpm build
+pnpm test:journeys
+```
+
+Run the complete serial browser gate with `pnpm test:browser`. The two suites
+share `tests/browser/database.ts`; each resets the database on entry and again
+on exit, then leaves only the default roles. This matters in CI because image,
+SEO and upgrade checks use the same disposable PostgreSQL service afterward.
+
+Screenshots, videos and traces from failures are under `test-results/browser`.
+CI's HTML report is under `playwright-report/browser`. Neither is committed.
+
+`BROWSER_BASE_URL` overrides the default `http://localhost:3100` only when the
+replacement URL reaches the same local test process and disposable database.

package/dist/template/deploy/c11-08-restore-drill-2026-09-28.log

@@ -0,0 +1,41 @@
+# C11.08 live Tier-1 restore drill — 2026-09-28
+# operator: Tony Aly (executed by Kimi Code agent on the owner's machine)
+# source: droplet freeholder-ref id=604232042 ip=104.131.118.196 image=ghcr.io/campdenman/freeholder:edge@sha256:0827de964a434adf8b065c4f518d4bbadc73f659108f58ab3a3e052af0a0a46a
+# target: droplet freeholder-restore (provisioned in this drill)
+# repo: worktree /home/tony/code/freeholder-c1108 branch docs/c11-08-restore-drill @ f9f3a05 (origin/main)
+
+[2026-09-29T00:14:50Z] DRILL START
+[2026-09-29T00:14:50Z] source pre-check: app+db containers up; db 361 tables / 79430 rows; assets=10000 ready; media_objects=10000 attached; users=1 (owner); pages=1 (perf-home published); connected_accounts=0
+[2026-09-29T00:14:50Z] source media byte check: /opt/freeholder/media/.data holds 0 files; app 404s its own media objects -> dataset is DB-seeded (tests/helpers/performance.ts writes asset rows, no object bytes). Recorded as a source-data finding for the media leg.
+[2026-09-29T00:19:38Z] STEP 1 EXPORT complete
+  pg_dump archive: /root/c11-08-freeholder.dump on source, copied to /home/tony/c11-08-drill/freeholder.dump
+  archive sha256: e78527321bb2fb93675a57d83f7e7782499c4936e5a8dc19b8eb119a46279123
+  archive size: 6918459 bytes
+  ownership export: /home/tony/c11-08-drill/ownership-export (374 tables, 150812 rows, 10000 media assets, 377 checksumed files)
+  media integrity (DB level): missingInventoryKeys=0 unreferencedInventoryKeys=0
+  recovery.json: secretValuesIncluded=false; CREDENTIAL_KEY fingerprint 5138ef637205fa71e479874daa992bbfc6eb1f2e8893a9a9c0404ed3574074e1 (independently recomputed on source, key never displayed)
+[2026-09-29T00:19:38Z] STEP 1b media byte finding: source media_objects=10000 attached rows but 0 object bytes in the source store; the app 404s its own objects. Byte-level transfer has nothing to copy; recorded for the media leg.
+[2026-09-29T01:13:24Z] STEP 2 PROVISION complete
+  droplet freeholder-restore id=604504844 ip=209.97.155.248 region=nyc3 size=s-1vcpu-1gb image=ubuntu-24-04-x64 ssh-key=59638965 cloud-init=deploy/digitalocean-droplet/infra/cloud-init.yml
+  recipe files scp'd to /opt/freeholder (compose.yml, Caddyfile, backup.sh from deploy/digitalocean-droplet/infra @ f9f3a05)
+  .env written on-box: fresh POSTGRES_PASSWORD/SESSION_SECRET/BOOTSTRAP_SECRET/CREDENTIAL_KEY (openssl rand -hex 32), APP_URL + FREEHOLDER_DOMAIN=http://209.97.155.248 (http-only site label for the drill box), FREEHOLDER_UNSAFE_LOCAL_STORAGE=1
+  deviation (recorded): target mirrors the source's local-disk storage mode because the source dataset is local-disk and no freeholder Spaces bucket exists on the account (freeholder-demo-media 404s in all regions); creating new paid resources was not part of the sanctioned drill footprint. §18 Spaces remains the production mandate; Caddyfile unchanged.
+  compose patched on-box: app service gained ./media/.data:/app/.data volume (mirrors source box; base recipe compose assumes S3)
+  NOTE: 'docker compose pull' fetched current :edge @sha256:34854f00... which is NEWER than the source's running image @sha256:0827de96...; target pinned via FREEHOLDER_IMAGE to the source digest 0827de964a434adf8b065c4f518d4bbadc73f659108f58ab3a3e052af0a0a46a so the restore is same-schema and bootable (recipe rollback mechanism)
+[2026-09-29T01:30:27Z] STEP 3 RESTORE complete (recipe.yaml restore/migrate-in command shape)
+  empty target db (fresh volume) -> docker compose -f compose.yml exec -T db pg_restore --no-owner --no-privileges -U freeholder -d freeholder < freeholder.dump
+  archive re-verified on target: sha256 e78527321bb2fb93675a57d83f7e7782499c4936e5a8dc19b8eb119a46279123, 6918459 bytes; pg_restore exit 0, zero errors
+  target app booted against restored db: 'schema is up to date' (same image digest as source), jobs worker resumed 86 restored jobs
+  verification export on target: 374 tables, 150905 rows, 10000 media assets
+  per-table diff source-vs-target: 366/374 identical (rows + canonical sha256). 8 differing tables are all runtime-operational: pgboss.job(+26), pgboss.job_common(+52), pgboss.queue, pgboss.schedule, pgboss.version, public.analytics_events(+4), public.audit_log(+10), public.job_runtime_heartbeats(+1) -> target app's own live heartbeat/job activity between restore and re-export, NOT business data.
+  owned data exact match: contacts 5001, messages 20000, orders 2000, products 500, assets 10000, pages 1, media_objects 10000, users 1, invoices 2000
+  media manifests identical: assets, objects, integrity (missingInventoryKeys=0, unreferencedInventoryKeys=0) round-tripped the restore byte-for-byte at DB level
+  MEDIA LEG (byte level): source store holds 0 of 10000 object bytes (pre-existing seed condition: tests/helpers/performance.ts writes asset/media_objects rows only; the source app 404s its own media). 'pnpm media:transfer' refuses local-disk stores by design ('local storage is not a Tier-1 migration target') - guardrail captured. Nothing exists to byte-copy; the restore reproduced the source media state exactly on the target.
+[2026-09-29T01:30:27Z] STEP 4 VERIFY complete
+  doctor (recipe verify op, from worktree vs target app): verdict=warn, 16 ok / 14 warn / 0 fail of 30 checks, exit 1. All warns expected for a scratch drill box (http APP_URL, no mail/SMS/push/AI adapters, media-on-disk per documented deviation, update policy undeclared). File storage check: ok (wrote, read, deleted a test file). TOTP enrolled via --enroll-totp (designed path for a fresh disposable instance); owner password set via the recipe's owner-password.mjs --disable-2fa (verify.md 'get back in'), value never leaves the box.
+  seeded public page: GET http://209.97.155.248/perf-home -> 200 via Caddy direct AND via ssh port-forward; body 26351 bytes, 'Performance home' rendered; only diff vs source-rendered page is the origin in canonical/og URLs (localhost:3000 vs 209.97.155.248). Proof the imported site is live on the second instance.
+[2026-09-29T01:37:30Z] STEP 5 DRILL RECORD written: deploy/c11-08-restore-summary.md (restore-drill date, operator, source archive checksum, restored table/row totals, media discrepancy counts, key fingerprint match, retention/erasure checks, verification results, deviations)
+[2026-09-29T01:37:30Z] MASTER.md C11.08 checked with dated F04/F05/F07/F09/F12 evidence; plan-gate exit 0 (281 checked, +1); plan-gate tests 24/24
+[2026-09-29T01:37:30Z] evidence PR opened: https://github.com/CampDenman/freeholder/pull/443 (branch docs/c11-08-restore-drill off origin/main f9f3a05), queued for merge
+[2026-09-29T01:37:30Z] FINAL STATE: source freeholder-ref id=604232042 104.131.118.196 healthy (app+db up, /perf-home 200, read-only throughout; transient dump removed from its disk). target freeholder-restore id=604504844 209.97.155.248 running restored stack (app+db+caddy, /perf-home 200). BOTH DROPLETS LEAVE-RUNNING per drill instructions - owner decides teardown.
+[2026-09-29T01:37:30Z] DRILL COMPLETE - full journey verified; media byte absence recorded as a source-data finding (no bytes existed to restore; manifests identical)

package/dist/template/deploy/c11-08-restore-summary.md

@@ -0,0 +1,113 @@
+# C11.08 live Tier-1 restore drill — record (2026-09-28)
+
+The clause C11.08 left open was *"restore on another Tier-1 target remains the
+ownership-drill pair matrix rather than a second live instance."* This drill
+ran the real thing end to end on live DigitalOcean infrastructure, per §23's
+mandated sequence (export archive → provision target from recipe → import →
+storage sync → verify before cutover) and the drill-record fields in
+[`ownership-recovery.md`](ownership-recovery.md). Raw transcript:
+[`c11-08-restore-drill-2026-09-28.log`](c11-08-restore-drill-2026-09-28.log).
+
+## Drill record (required fields)
+
+- **Restore-drill date:** 2026-09-28 (UTC 2026-09-29T00:14Z – 01:30Z).
+- **Operator:** Tony Aly (executed by the Kimi Code agent on the owner's
+  machine; every step timestamped in the log).
+- **Source archive checksum:** SHA-256
+  `e78527321bb2fb93675a57d83f7e7782499c4936e5a8dc19b8eb119a46279123`,
+  6,918,459 bytes, custom-format `pg_dump --no-owner --no-privileges`. Verified
+  identical on the target box before `pg_restore`.
+- **Source:** droplet `freeholder-ref` (id 604232042, 104.131.118.196),
+  app+db in compose at /opt/freeholder, running
+  `ghcr.io/campdenman/freeholder:edge@sha256:0827de96…0a46a`. Export was
+  read-only; the source stayed healthy throughout (its `/perf-home` still
+  serves 200 after the drill) and the transient dump file was removed from its
+  disk.
+- **Target:** droplet `freeholder-restore` (id 604504844, 209.97.155.248,
+  nyc3, s-1vcpu-1gb) provisioned from
+  [`deploy/digitalocean-droplet`](../deploy/digitalocean-droplet/README.md):
+  recipe `cloud-init.yml`, then recipe `compose.yml`/`Caddyfile`/`backup.sh`
+  scp'd to /opt/freeholder, `.env` written on-box with freshly generated
+  secrets, image pinned to the source digest via the recipe's `FREEHOLDER_IMAGE`
+  mechanism (`:edge` had moved to a newer build between the source's last pull
+  and this drill; the pin keeps the restore same-schema — both digests are in
+  the log).
+- **Restore command (the recipe's own shape):** into an empty database —
+  `docker compose -f compose.yml exec -T db pg_restore --no-owner
+  --no-privileges -U freeholder -d freeholder < freeholder.dump` (the
+  `recipe.yaml` `restore`/`migrate-in` operation), exit 0, zero errors.
+- **Restored table/row totals:** source logical export 374 tables / 150,812
+  rows / 10,000 media assets. Target re-export after restore: 374 tables /
+  150,905 rows / 10,000 media assets. Per-table comparison: **366 of 374
+  tables identical** in row count *and* canonical SHA-256. The 8 differing
+  tables are all runtime-operational — `pgboss.job` (+26), `pgboss.job_common`
+  (+52), `pgboss.queue`, `pgboss.schedule`, `pgboss.version`,
+  `public.analytics_events` (+4), `public.audit_log` (+10),
+  `public.job_runtime_heartbeats` (+1) — written by the *target app's own* live
+  job worker/heartbeat between restore and re-export, not by the restore. Every
+  owned table matches exactly: contacts 5,001; messages 20,000; orders 2,000;
+  invoices 2,000; products 500; assets 10,000; media_objects 10,000; pages 1;
+  users 1.
+- **Media discrepancy counts:** the media manifest (assets, objects, integrity)
+  is **byte-identical** across the restore — `missingInventoryKeys` 0,
+  `unreferencedInventoryKeys` 0 on both sides. The byte-level finding:
+  **10,000 of 10,000 media objects have no source bytes.** The source dataset
+  is DB-seeded (`tests/helpers/performance.ts` inserts `assets`/
+  `media_objects` rows; it never writes object bytes), so the *source instance
+  itself* 404s its own media. `pnpm media:transfer` refuses local-disk stores
+  by design ("local storage is not a Tier-1 migration target") — the guardrail
+  fired as written and is captured in the log. Nothing existed to byte-copy, so
+  the restore reproduced the source's media state exactly. This is a
+  source-data finding, not a restore failure; it is worth a seed-harness
+  follow-up so future ref datasets carry real object bytes.
+- **Key fingerprint match:** source `CREDENTIAL_KEY` fingerprint
+  `5138ef637205fa71e479874daa992bbfc6eb1f2e8893a9a9c0404ed3574074e1`,
+  recomputed on the source box (key never displayed) and matching the export's
+  `recovery.json` (`secretValuesIncluded: false`, `validFormat: true`). The
+  target was provisioned with fresh secrets per the drill plan; because the
+  source holds **zero** `connected_accounts` rows, the fresh key strands no
+  ciphertext. The runbook's "keep the same CREDENTIAL_KEY" rule matters only
+  when encrypted credentials exist.
+- **Retention/erasure checks:** the dataset is synthetic test data (no personal
+  rows); the restored copy lives only on the disposable drill target. Per
+  `ownership-recovery.md`'s post-restore review, `connections.grants` and
+  `connected_accounts` were re-checked on the target: 0 rows, so no agent
+  grants point at production mailboxes and nothing needed revoking.
+- **Verification before cutover (§23 step 4):** the recipe `verify` operation
+  (`pnpm doctor`) ran against the target app from the worktree:
+  **verdict `warn` — 16 ok / 14 warn / 0 fail of 30 checks** (exit 1). Every
+  warning is expected for a scratch drill box: http `APP_URL`, no mail/SMS/
+  push/AI adapters, media-on-disk per the documented deviation below, update
+  policy undeclared. The File-storage check passed (wrote, read and deleted a
+  test file). Owner access used the recipe's own recovery path
+  (`scripts/owner-password.mjs --disable-2fa`, then doctor `--enroll-totp` —
+  the designed enrollment for a fresh disposable instance); no secret left the
+  box.
+- **Served-page proof:** `GET http://209.97.155.248/perf-home` → 200 through
+  Caddy on the target (and via ssh port-forward), 26,351 bytes, the seeded
+  "Performance home" page rendered. Diff against the source-rendered page
+  shows only the origin in canonical/OG URLs (`localhost:3000` vs the target
+  IP) — the imported site is live on the second instance.
+
+## Recorded deviations (all in the log)
+
+1. **Storage mode:** the droplet recipe mandates Spaces (§18); the target
+   mirrors the source's actual local-disk storage (`FREEHOLDER_UNSAFE_LOCAL_
+   STORAGE=1` + the `./media/.data` volume, as the source box runs) because the
+   source dataset is local-disk and the one historic freeholder Spaces bucket
+   no longer exists (404 in all regions). Creating new paid resources was not
+   part of the sanctioned drill footprint (two $6 droplets). Doctor reports
+   media-on-disk as a warn, as `verify.md` anticipates.
+2. **Bind-mount ownership:** the app container runs as uid 1001; docker
+   auto-created the host `./media/.data` as root, so the first doctor run
+   failed File-storage with EACCES until the directory was `chown`ed to
+   1001:1001 (the source box already carries that ownership). Recorded as a
+   real-world sharp edge for local-storage overrides.
+3. **Image pin:** `FREEHOLDER_IMAGE` set to the source digest; current `:edge`
+   had moved on. The recipe's own rollback variable carries the pin.
+
+## Infrastructure left running
+
+Both droplets are **leave-running** for the owner to decide (per drill
+instructions): `freeholder-ref` (604232042, 104.131.118.196) and
+`freeholder-restore` (604504844, 209.97.155.248). Two $6/mo droplets.