@rehearsal-db/core 0.1.0-beta.1 → 0.1.0-beta.10

package/docs/getting-started.md

@@ -1,247 +1,171 @@
 # Getting started
 
-This guide creates a safe local Rehearsal project from synthetic data. It does not
-connect to production, request hosted credentials, or require an existing baseline.
+This guide sets up Rehearsal in an existing project. It creates a disposable database on
+your computer. Normal setup and rehearsal commands never connect to a hosted database.
 
-## Prerequisites
+Want to experiment first? Use the [safe hands-on tutorial](tutorial.md).
 
-- Node.js 24
-- npm
-- Supabase CLI 2.117.0 (the version proved by package CI)
-- Docker Desktop, Colima, or another Docker-compatible engine
-- a Supabase project with timestamped SQL migrations
+## 1. Check the required tools
 
-Confirm the tools first:
+You need Node.js 24, npm, and a running Docker-compatible engine:
 
 ```bash
 node --version
 npm --version
-supabase --version
 docker info
 ```
 
-If you want to see the complete lifecycle before touching your own project, clone the
-Rehearsal repository and run:
+`node --version` must begin with `v24`. If `docker info` fails, start Docker Desktop,
+Colima, or your usual Docker engine.
 
-```bash
-npm ci --ignore-scripts
-npm run test:fixture
-```
+Choose the extra requirement for your project:
 
-That proof installs the packed package into a disposable fictional project, restores a
-row and Storage object, applies a valid migration, rejects invalid SQL, and removes only
-its labeled local runtime. It does not need a hosted Supabase project or credentials.
+- **Supabase:** install Supabase CLI 2.117.0 and run `supabase --version`.
+- **PostgreSQL:** download the local database image once:
 
-## 1. Install and initialize
+  ```bash
+  docker pull postgres:17-alpine
+  ```
 
-```bash
-npm install --save-dev @rehearsal-db/core
-npx rehearsal init
-```
+Rehearsal will use that image with downloads disabled.
+
+## 2. Install and open the guide
 
-`init` is a preview. Read the generated configuration, then explicitly write it:
+Run these commands from the directory containing your project's `package.json`:
 
 ```bash
-npx rehearsal init --write
+npm install --save-dev @rehearsal-db/core@beta
+npx rehearsal
 ```
 
-Rehearsal never overwrites an existing configuration.
+Choose **Set the stage**, then choose Supabase or PostgreSQL. Rehearsal previews every file
+before asking to create it. It will not replace an existing file.
 
-The generated configuration is intentionally incomplete until you review its ports,
-project ID, application commands, and project-owned input paths. Do not run `doctor`
-until the next section's files exist.
+Review the generated `rehearsal.config.mjs`, especially:
 
-## 2. Add the project-owned inputs
+- the migration directory;
+- the application start and proof commands;
+- the local ports.
 
-Create the paths named by `rehearsal.config.ts`:
+The proof command should check behavior changed by the migration. A test that only checks
+whether the home page loads is usually too weak.
 
-- a dedicated local Supabase `config.toml`;
-- a sanitization policy describing every exported field;
-- an active baseline below `.rehearsal/`;
-- an application proof command that exits nonzero when the restored app is wrong.
-
-For the generated default paths:
-
-```bash
-mkdir -p infrastructure/rehearsal/supabase infrastructure/rehearsal rehearsal
-cp supabase/config.toml infrastructure/rehearsal/supabase/config.toml
-```
+## 3. Prepare safe baseline inputs
 
-Edit the copied Supabase config. Give it the same unique `project_id`, API port, database
-port, and Studio port used by `rehearsal.config.ts`. Disable services your proof does
-not need. This must remain an unlinked local config; never run `supabase link` from it.
+A **baseline** is the locked starting point restored before each rehearsal. Start with a
+small synthetic dataset. Do not use raw production data.
 
-Also replace the generated `application.startCommand`, `application.proofCommand`, and
-`verification.commands` with real commands from your project. The proof should check a
-restored relationship and the candidate schema—not merely that `/` returns 200.
+Rehearsal needs:
 
-Start with synthetic rows shaped like your schema. Do not start onboarding with
-production data. The following SQL, policy, row, and ledger form one matched example;
-do not mix them with differently shaped snippets.
+| Input               | What it contains                                     |
+| ------------------- | ---------------------------------------------------- |
+| Records             | Safe rows in NDJSON format: one JSON object per line |
+| Migration ledger    | The historical migrations represented by those rows  |
+| Sanitization policy | A decision for every included table and column       |
+| Storage manifest    | Optional Supabase Storage files only                 |
 
-Historical migration `supabase/migrations/20260101000000_create_widgets.sql`:
+A record looks like this:
 
-```sql
-create table public.widgets (
-  id bigint generated by default as identity primary key,
-  name text not null,
-  created_at timestamptz not null default now()
-);
-
-insert into storage.buckets (id, name, public)
-values ('fixture-assets', 'fixture-assets', false);
+```json
+{ "table": "widgets", "row": { "id": 1, "name": "Synthetic Widget" } }
 ```
 
-Sanitization policy `infrastructure/rehearsal/sanitization-policy.json`:
+Put your safe records and ledger inside the project. Conventional names such as
+`rehearsal/sanitized-data.ndjson` and `rehearsal/migration-ledger.json` are discovered
+automatically. The guide inspects their structure but does not print row values.
 
-```json
-{
-  "policyVersion": 1,
-  "migrationCutoff": "20260101000000",
-  "tables": [
-    {
-      "name": "widgets",
-      "group": "synthetic",
-      "sourceRows": "STREAM AND SANITIZE",
-      "columns": [
-        {
-          "name": "id",
-          "action": "KEEP EXACTLY",
-          "generated": "NEVER",
-          "identity": "YES",
-          "foreignKey": null
-        },
-        {
-          "name": "name",
-          "action": "REPLACE WITH SYNTHETIC",
-          "generated": "NEVER",
-          "identity": "NO",
-          "foreignKey": null
-        },
-        {
-          "name": "created_at",
-          "action": "DERIVE",
-          "generated": "NEVER",
-          "identity": "NO",
-          "foreignKey": null
-        }
-      ]
-    }
-  ]
-}
-```
+Follow the guide's recommended actions:
 
-Safe row `rehearsal/synthetic-data.ndjson` (one JSON object per physical line):
+1. **Prepare the script** creates a draft sanitization policy.
+2. **Review the script** asks you to classify every table and column.
+3. **Create the baseline** validates and activates the reviewed inputs.
 
-```json
-{
-  "table": "widgets",
-  "row": {
-    "id": 1,
-    "name": "Synthetic Widget",
-    "created_at": "2026-01-01T00:00:00.000Z"
-  }
-}
-```
+Rehearsal will not activate a policy containing undecided fields.
 
-Migration ledger `rehearsal/migration-ledger.json`:
+The ledger must describe the exact historical SQL included in the baseline. Do not type
+years of migration history by hand for a real project. Generate it through reviewed
+project tooling. The [tutorial](tutorial.md) contains a complete synthetic example, and
+[Baselines](baselines.md) explains the file formats.
 
-```json
-[
-  {
-    "version": "20260101000000",
-    "name": "create_widgets",
-    "statements": [
-      "create table public.widgets (\n\tid bigint generated by default as identity primary key,\n\tname text not null,\n\tcreated_at timestamptz not null default now()\n)",
-      "insert into storage.buckets (id, name, public)\nvalues ('fixture-assets', 'fixture-assets', false)"
-    ]
-  }
-]
-```
+## 4. Reach READY
 
-The ledger is evidence, not a second migration language. Its version, name, order, and
-statements must exactly represent the historical migration bytes in the baseline. SQL
-that is merely equivalent is rejected. For a production-shaped baseline, generate this
-ledger through the project's reviewed source/export tooling; do not reconstruct years of
-history by hand.
+The guide displays four setup stages. Continue with its recommended action until all
+checks are complete and the project reports `READY`.
 
-Then activate the safe input:
+You can run the same check directly:
 
 ```bash
-npx rehearsal baseline create \
-  --records=rehearsal/synthetic-data.ndjson \
-  --ledger=rehearsal/migration-ledger.json
+npx rehearsal doctor
 ```
 
-The independent fixture in this repository is the executable reference example.
+If it reports `NOT READY`, fix the listed item and run it again. Do not bypass a check or
+add a hosted connection string.
 
-Expected result:
+## 5. Run the rehearsal
 
-```text
-Activated synthetic baseline ...: 1 rows across 1 tables; 1 migrations through 20260101000000.
-```
+Choose **Run a rehearsal**. Rehearsal shows the exact candidate migrations and asks for
+confirmation before it changes the disposable runtime.
 
-## 3. Check readiness
+A successful run:
 
-```bash
-npx rehearsal doctor
-```
+1. restores the baseline;
+2. applies only the migrations you approved;
+3. verifies the local database;
+4. launches the ordinary application when readiness is configured;
+5. runs the ordinary test command and declared positive/negative HTTP proofs;
+6. stops only the application process it launched.
+
+Your original baseline remains unchanged.
 
-Do not continue until it ends with `READY`. Doctor checks the local-only boundary,
-dependencies, artifact integrity, migration lineage, ports, and project commands.
+If a source owner later approves a production-shaped copy, use the separate workflow in
+[Optional production-shaped preparation](production-source.md). Do not add source
+credentials to the normal config or runtime environment.
 
-If it reports `NOT READY`, fix each named prerequisite and rerun it. Do not bypass a
-check or copy a hosted connection string into the generated runtime environment.
+Once temporary source access is ready, `npx rehearsal refresh` previews the safe
+replacement workflow. It means “get a newly verified copy.” By contrast, `reset` means
+“discard local edits and restore the copy I already have.”
 
-## 4. Review pending work
+## 6. Use the sandbox and clean up
+
+Choose **Open the sandbox app**, or run:
 
 ```bash
-npx rehearsal explain
-npx rehearsal candidates
-npx rehearsal inspect baseline
-npx rehearsal inspect migrations
+npx rehearsal open
 ```
 
-The plan prints an exact candidate digest. It does not start services or change data.
+Open the local URL printed by Rehearsal. The app stays available for hands-on testing
+until you press `Ctrl+C`. Only the app stops; your local database and Storage changes are
+kept. Run `npx rehearsal open` again to continue where you left off.
 
-## 5. Run the rehearsal
+Use the guide for normal runtime tasks:
 
-```bash
-npx rehearsal run --confirm-candidates=<sha256>
-```
+- **Verify** checks the current runtime.
+- **Reset** discards runtime edits and restores the baseline.
+- **Stop** stops the runtime but keeps its local state.
+- **Discard** removes this project's disposable runtime and volume.
+
+Press `Ctrl+Z` at any prompt to exit the entire guide. It will not leave a suspended
+process behind.
 
-Rehearsal restores the immutable baseline, applies only the confirmed migration suffix,
-verifies the runtime, and runs the project-owned application proof. The resulting local
-database is writable, so you can test real application changes without mutating the
-baseline.
+## Command-line setup
 
-## 6. Work, verify, and reset
+The guide is recommended for people. Scripts and CI can use explicit commands:
 
 ```bash
-npx rehearsal status
-npx rehearsal start
-npx rehearsal verify
-npx rehearsal reset
-npx rehearsal stop
+npx rehearsal setup --target=postgresql --write
+npx rehearsal doctor
+npx rehearsal candidates
+npx rehearsal run --confirm-candidates=PASTE_DIGEST_HERE
 ```
 
-Changes persist in the disposable runtime until `reset` or runtime removal. `start`
-resumes that runtime without resetting it. `reset` restores the exact baseline. `stop`
-stops only this project's runtime.
-
-## Next steps
-
-- Follow [the full tutorial](tutorial.md).
-- Read [the security model](security-model.md) before designing a production export.
-- Define an exhaustive [sanitization policy](sanitization.md).
-- Learn the [baseline lifecycle](baselines.md).
+Replace `postgresql` with `supabase` when needed. See [CLI commands](commands.md) for the
+complete reference and [Troubleshooting](troubleshooting.md) when a check fails.
 
-## Generated files
+## Files Rehearsal creates
 
-- `rehearsal.config.ts` is written only by `init --write`.
-- `.rehearsal/generations/<id>/` contains one immutable baseline generation.
-- `.rehearsal/current` selects the active generation atomically.
-- `.rehearsal/runtime/` contains the disposable local Supabase project and receipts.
-- `.rehearsal/runtime.env` contains generated loopback-only application credentials.
+- `rehearsal.config.mjs` — reviewed project configuration
+- `.rehearsal/` — ignored baselines, runtime files, and receipts
+- `.rehearsal/runtime.env` — generated local-only application variables
+- `infrastructure/rehearsal/supabase/config.toml` — Supabase-only local configuration
 
-Ignore all of `.rehearsal/`. Do not commit it even when its inputs were synthetic.
+Keep `.rehearsal/` out of source control, even when its data is synthetic.

package/docs/glossary.md

@@ -1,4 +1,4 @@
-# Glossary and architecture
+# Glossary
 
 **Baseline** — immutable sanitized starting artifact.
 
@@ -6,6 +6,10 @@
 
 **Candidate digest** — checksum binding the exact candidate filenames and bytes.
 
+**Dependent target** — another isolated local database managed with the primary runtime.
+
+**Combined digest** — one checksum binding candidate or cleanup sets across every target.
+
 **Project-owned** — application-specific policy or code that remains outside the package.
 
 **Represented migration** — historical migration whose exact bytes are bound into the
@@ -19,13 +23,12 @@ baseline.
 
 ```mermaid
 flowchart LR
-  A[Project-owned approved source] --> B[Project-owned sanitization]
-  B --> C[Immutable baseline]
-  D[Project migrations] --> E[Digest planner]
-  C --> F[Disposable local Supabase]
-  E --> F
-  F --> G[Project application proof]
-  G --> H[Verified local receipt]
+  A[Safe rows] --> B[Locked baseline]
+  C[Migration files] --> D[Reviewed plan]
+  B --> E[Disposable local database]
+  D --> E
+  E --> F[Application proof]
+  F --> G[Verified result]
 ```
 
 The reusable package owns the path from a completed baseline plus migration directory to

package/docs/production-source.md

@@ -1,33 +1,176 @@
-# Designing a production source boundary
-
-The npm package does not connect to production and does not ship an extraction command.
-That separation is a security boundary: each project must authorize, sanitize, and
-audit its own source.
-
-## Recommended flow
-
-1. Define versioned read-only export views containing only approved columns.
-2. Grant a temporary role `SELECT` on those views and nothing else.
-3. Use a short-lived credential outside shell history and source control.
-4. Stream records through the project-owned sanitization policy.
-5. Write into a private building generation below `.rehearsal`.
-6. Verify counts, checksums, policy coverage, migration history, and secret canaries.
-7. Atomically activate the completed generation.
-8. Revoke and verify removal of the temporary credential.
-9. Delete raw intermediate files and retain only the sanitized artifact.
-
-Use server-side cursors or another bounded streaming mechanism. A full in-memory dump
-is not acceptable for large sources. Extraction logs should contain counts and hashes,
-never row bodies.
-
-## What not to do
-
-- Do not point `rehearsal.config.ts` at a hosted URL.
-- Do not put a production connection string in `.rehearsal/runtime.env`.
-- Do not grant table-wide access merely because a view is inconvenient.
-- Do not commit sanitized baselines; sanitized data is still data.
-- Do not let the reusable package infer which fields are safe.
-- Do not leave the export role provisioned after refresh.
-
-The local execution engine remains useful with synthetic or manually approved baselines.
-Production-shaped onboarding is a separate project security exercise.
+# Optional production-shaped preparation
+
+Start with synthetic data. Use this workflow only when the source owner has approved the
+exact tables, columns, owner scope, assets, and privacy policy.
+
+Ordinary Rehearsal commands remain local-only. Source preparation is separate and cannot
+be triggered by `run`, `reset`, `migrate`, `verify`, or application launch.
+
+## 1. Add the config paths
+
+```js
+preparation: {
+  sourcePolicy: "infrastructure/rehearsal/source-access-policy.json",
+  privacyKey: ".rehearsal/secrets/privacy.key",
+  batchRows: 500,
+  maximumRows: 1000000,
+  maximumBytes: 2147483648,
+  diskHeadroomBytes: 67108864,
+},
+```
+
+The privacy key stays ignored under `.rehearsal`. The policy is safe to review in source
+control because it contains hashes and environment-variable names, not credentials,
+URLs, or raw owner identifiers.
+
+## 2. Declare the source boundary
+
+`source-access-policy.json` uses this shape:
+
+```json
+{
+  "accessVersion": 1,
+  "targetFingerprint": "<sha256 of the approved PostgreSQL host, port, and database>",
+  "administratorEnvironmentVariable": "REHEARSAL_SOURCE_ADMIN_URL",
+  "reader": {
+    "role": "rehearsal_example_reader",
+    "ownerRole": "rehearsal_example_owner",
+    "credentialFile": ".rehearsal/secrets/source-reader.env",
+    "validForMinutes": 30
+  },
+  "exportSchema": "rehearsal_export_example",
+  "migrationLedger": {
+    "schema": "supabase_migrations",
+    "table": "schema_migrations",
+    "versionColumn": "version",
+    "nameColumn": "name",
+    "statementsColumn": "statements"
+  },
+  "relations": [
+    {
+      "source": { "schema": "public", "table": "widgets" },
+      "view": "widgets",
+      "targetSchema": "public",
+      "targetTable": "widgets",
+      "columns": ["id", "owner_id", "name"],
+      "orderBy": ["id"],
+      "rowScope": { "kind": "approved-public" }
+    },
+    {
+      "source": { "schema": "public", "table": "profiles" },
+      "view": "approved_owner_profile",
+      "targetSchema": "app_api",
+      "targetTable": "profiles",
+      "columns": ["id", "email"],
+      "orderBy": ["id"],
+      "rowScope": {
+        "kind": "approved-owner",
+        "column": "id",
+        "valueEnvironmentVariable": "REHEARSAL_APPROVED_OWNER_ID"
+      }
+    }
+  ],
+  "assets": []
+}
+```
+
+Calculate fingerprints with the exported helpers from `@rehearsal-db/core/source-access`
+or a reviewed local script. A fingerprint binds the destination without recording its
+address. Changing the host, port, or database invalidates the plan.
+
+`approved-public` means every selected row is licensed or otherwise approved for this
+use. `approved-owner` requires one explicit owner value from the named environment
+variable. Rehearsal never chooses an owner by finding an administrator.
+
+`targetSchema` is optional and defaults to `public`. Set it when the restored relation
+belongs elsewhere, and use the same schema in privacy policy version 2. This is relation
+identity, not a request to search every schema.
+
+For Storage, add an `assetReader` block and approved `assets` entries. The reader URL and
+token come from named environment variables. Objects outside the exact bucket/prefix are
+not listed or downloaded. Size, object-count, path, ETag, and exact-byte checks apply.
+
+## 3. Use executable privacy policy version 2
+
+Every selected column has one action and, where required, a bounded recipe:
+
+```json
+{
+  "policyVersion": 2,
+  "migrationCutoff": "20260101000000",
+  "tables": [
+    {
+      "schema": "app_api",
+      "name": "profiles",
+      "sourceRows": "STREAM AND SANITIZE",
+      "columns": [
+        {
+          "name": "id",
+          "action": "PSEUDONYMIZE",
+          "recipe": { "format": "uuid", "namespace": "account-id" },
+          "generated": "NEVER",
+          "identity": "NO",
+          "foreignKey": null
+        },
+        {
+          "name": "email",
+          "action": "PSEUDONYMIZE",
+          "recipe": { "format": "email", "namespace": "account-email" },
+          "generated": "NEVER",
+          "identity": "NO",
+          "foreignKey": null
+        }
+      ]
+    }
+  ]
+}
+```
+
+Supported recipes are intentionally small: keyed UUID/email/text/integer pseudonyms,
+explicit constants, bounded date shifts, and fully classified JSON objects. Unknown
+tables, columns, JSON keys, formats, and recipes stop the refresh. There is no JavaScript
+or SQL escape hatch.
+
+## 4. Preview, apply, refresh, and retire
+
+Keep credentials out of shell history by loading them from an owner-only environment
+manager or file, then run:
+
+```bash
+npx rehearsal privacy key --write
+npx rehearsal source plan
+npx rehearsal source apply --confirm-source-access=<full-sha256>
+npx rehearsal refresh
+npx rehearsal refresh --confirm-refresh=<full-sha256>
+npx rehearsal source retire
+npx rehearsal source retire --confirm-source-retirement=<full-sha256>
+```
+
+`source apply` creates new time-limited roles and export views only when every exact name
+is unused. It verifies the reader cannot read raw tables, write, escalate roles, or call
+an exposed security-definer network function. Externally provisioned readers are not yet
+supported; use the managed reader so Rehearsal can prove its scope and retirement.
+
+`refresh` previews the complete replacement first. After exact confirmation, its baseline
+step uses one read-only repeatable-read transaction for the schema shape,
+migration ledger, and rows. It streams in bounded batches, records only counts and
+hashes, stages privately, rechecks schema/ledger drift, and activates atomically. It then
+resets the local runtime stack and applies configured baseline retention. A failed new
+runtime is rolled back to the previous baseline and runtime. The lower-level `baseline
+refresh` command remains available when automation should activate a baseline without
+resetting the runtime or pruning old generations.
+
+Storage cannot share PostgreSQL's transaction snapshot. Rehearsal instead verifies each
+object's inventory version and byte count during transfer, then records its SHA-256 in
+the immutable baseline.
+
+`source retire` inventories the exact recorded views before removal, then removes only
+those views, the two recorded roles, and local credential/receipt files. Unrelated roles,
+grants, objects, baselines, runtimes, and Docker resources are preserved.
+
+## Limits
+
+Rehearsal does not decide consent, licensing, retention rights, privileged-role meaning,
+or which owner is yours. It does not provide an OS firewall, backup, disaster recovery,
+or universal database support. Real Google account selection and provider callbacks
+still require direct human observation; a synthetic redirect cannot prove them.