@everystack/mcp 0.3.2 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (104) hide show
  1. package/dist/adding-database.md +36 -23
  2. package/dist/cli.md +44 -3
  3. package/dist/core.md +19 -1
  4. package/dist/database-operations.md +236 -0
  5. package/dist/derived-objects.md +225 -0
  6. package/dist/gates/detectors/embedded-data-bundle.d.ts +13 -0
  7. package/dist/gates/detectors/embedded-data-bundle.d.ts.map +1 -0
  8. package/dist/gates/detectors/embedded-data-bundle.js +56 -0
  9. package/dist/gates/detectors/hand-written-migration.d.ts +24 -0
  10. package/dist/gates/detectors/hand-written-migration.d.ts.map +1 -0
  11. package/dist/gates/detectors/hand-written-migration.js +54 -0
  12. package/dist/gates/detectors/secret-in-public-env.d.ts +12 -0
  13. package/dist/gates/detectors/secret-in-public-env.d.ts.map +1 -0
  14. package/dist/gates/detectors/secret-in-public-env.js +38 -0
  15. package/dist/gates/engine.d.ts +28 -0
  16. package/dist/gates/engine.d.ts.map +1 -0
  17. package/dist/gates/engine.js +74 -0
  18. package/dist/gates/registry.d.ts +14 -0
  19. package/dist/gates/registry.d.ts.map +1 -0
  20. package/dist/gates/registry.js +21 -0
  21. package/dist/gates/telemetry.d.ts +47 -0
  22. package/dist/gates/telemetry.d.ts.map +1 -0
  23. package/dist/gates/telemetry.js +121 -0
  24. package/dist/gates/types.d.ts +67 -0
  25. package/dist/gates/types.d.ts.map +1 -0
  26. package/dist/gates/types.js +14 -0
  27. package/dist/governance/cli.d.ts +57 -0
  28. package/dist/governance/cli.d.ts.map +1 -0
  29. package/dist/governance/cli.js +169 -0
  30. package/dist/governance/grounding.d.ts +78 -0
  31. package/dist/governance/grounding.d.ts.map +1 -0
  32. package/dist/governance/grounding.js +299 -0
  33. package/dist/index.cjs +119 -107
  34. package/dist/index.d.ts +2 -0
  35. package/dist/index.d.ts.map +1 -0
  36. package/dist/index.js +94 -0
  37. package/dist/project-claude-md.md +35 -18
  38. package/dist/prompts/add-feature.d.ts +3 -0
  39. package/dist/prompts/add-feature.d.ts.map +1 -0
  40. package/dist/prompts/add-feature.js +154 -0
  41. package/dist/prompts/claude-md.d.ts +12 -0
  42. package/dist/prompts/claude-md.d.ts.map +1 -0
  43. package/dist/prompts/claude-md.js +87 -0
  44. package/dist/prompts/debug.d.ts +3 -0
  45. package/dist/prompts/debug.d.ts.map +1 -0
  46. package/dist/prompts/debug.js +129 -0
  47. package/dist/prompts/deploy.d.ts +3 -0
  48. package/dist/prompts/deploy.d.ts.map +1 -0
  49. package/dist/prompts/deploy.js +118 -0
  50. package/dist/prompts/design-schema.d.ts +3 -0
  51. package/dist/prompts/design-schema.d.ts.map +1 -0
  52. package/dist/prompts/design-schema.js +97 -0
  53. package/dist/prompts/governance-setup.d.ts +9 -0
  54. package/dist/prompts/governance-setup.d.ts.map +1 -0
  55. package/dist/prompts/governance-setup.js +76 -0
  56. package/dist/prompts/index.d.ts +3 -0
  57. package/dist/prompts/index.d.ts.map +1 -0
  58. package/dist/prompts/index.js +20 -0
  59. package/dist/prompts/new-app.d.ts +3 -0
  60. package/dist/prompts/new-app.d.ts.map +1 -0
  61. package/dist/prompts/new-app.js +203 -0
  62. package/dist/prompts/runbook.d.ts +12 -0
  63. package/dist/prompts/runbook.d.ts.map +1 -0
  64. package/dist/prompts/runbook.js +70 -0
  65. package/dist/prompts/secure.d.ts +3 -0
  66. package/dist/prompts/secure.d.ts.map +1 -0
  67. package/dist/prompts/secure.js +219 -0
  68. package/dist/resources/index.d.ts +12 -0
  69. package/dist/resources/index.d.ts.map +1 -0
  70. package/dist/resources/index.js +182 -0
  71. package/dist/schema-patterns.md +92 -106
  72. package/dist/tools/check-environment.d.ts +25 -0
  73. package/dist/tools/check-environment.d.ts.map +1 -0
  74. package/dist/tools/check-environment.js +281 -0
  75. package/dist/tools/index.d.ts +3 -0
  76. package/dist/tools/index.d.ts.map +1 -0
  77. package/dist/tools/index.js +18 -0
  78. package/dist/tools/project-status.d.ts +28 -0
  79. package/dist/tools/project-status.d.ts.map +1 -0
  80. package/dist/tools/project-status.js +138 -0
  81. package/dist/tools/project-validate.d.ts +19 -0
  82. package/dist/tools/project-validate.d.ts.map +1 -0
  83. package/dist/tools/project-validate.js +323 -0
  84. package/dist/tools/schema-analyze.d.ts +46 -0
  85. package/dist/tools/schema-analyze.d.ts.map +1 -0
  86. package/dist/tools/schema-analyze.js +336 -0
  87. package/package.json +3 -3
  88. package/src/gates/detectors/hand-written-migration.ts +29 -13
  89. package/src/index.ts +1 -1
  90. package/src/prompts/add-feature.ts +4 -4
  91. package/src/prompts/claude-md.ts +4 -2
  92. package/src/prompts/debug.ts +3 -4
  93. package/src/prompts/deploy.ts +17 -8
  94. package/src/prompts/design-schema.ts +45 -59
  95. package/src/prompts/new-app.ts +21 -19
  96. package/src/prompts/secure.ts +3 -3
  97. package/src/resources/adding-database.md +36 -23
  98. package/src/resources/cli.md +44 -3
  99. package/src/resources/core.md +19 -1
  100. package/src/resources/database-operations.md +236 -0
  101. package/src/resources/derived-objects.md +225 -0
  102. package/src/resources/index.ts +15 -1
  103. package/src/resources/project-claude-md.md +35 -18
  104. package/src/resources/schema-patterns.md +92 -106
@@ -13,7 +13,7 @@ You already have a running Expo app. Now you're adding:
13
13
  - **@everystack/logging** — structured logging so you can see what your app is doing.
14
14
  - **@everystack/security** — device attestation and biometric auth.
15
15
  - **@everystack/query** — React hooks that make it easy to fetch and display data.
16
- - **Drizzle ORM** defines your database tables in TypeScript. You describe what data you want to store, and Drizzle creates the tables for you.
16
+ - **@everystack/model** — declares your database tables in TypeScript with `defineModel`. You describe what data you want to store and who can read/write it, and everystack generates the migration and the typed schema for you.
17
17
 
18
18
  ## Step 1: Install PostgreSQL
19
19
 
@@ -64,24 +64,31 @@ You should see a table with the value `1`. If so, your database is ready.
64
64
  From your project directory:
65
65
 
66
66
  ```bash
67
- pnpm add @everystack/api @everystack/auth @everystack/admin @everystack/logging @everystack/security @everystack/query
67
+ pnpm add @everystack/api @everystack/auth @everystack/admin @everystack/logging @everystack/security @everystack/query @everystack/model
68
68
  pnpm add drizzle-orm
69
- pnpm add -D drizzle-kit
70
69
  ```
71
70
 
72
- ## Step 4: Create your database schema
71
+ ## Step 4: Declare your database schema
73
72
 
74
- The schema defines what data your app stores. Create a file at `db/schema.ts`.
73
+ The schema defines what data your app stores. Create your tables in `db/models/` — one
74
+ `defineModel` per table (for example `db/models/post.ts`), composed onto a `defineModule`
75
+ in `db/models/index.ts`.
75
76
 
76
- A schema is like designing a spreadsheet before you start filling in data. You decide what columns each table has, what type of data goes in each column, and how tables relate to each other.
77
+ A model is like designing a spreadsheet before you start filling in data. You decide what columns each table has, what type of data goes in each column, and how tables relate to each other — plus who can read and write each table.
77
78
 
78
- Read everystack://schema-patterns for design patterns and examples. Key conventions:
79
+ You do **not** hand-write `db/schema.ts` and you do **not** run `drizzle-kit generate` —
80
+ everystack generates the migration and the typed artifact from your Models.
79
81
 
80
- - UUID primary keys with `defaultRandom()`
81
- - `created_at` and `updated_at` timestamps on all tables
82
- - Foreign key references with proper cascading
83
- - Relations defined for both SSR and API query embedding
84
- - Snake_case column names
82
+ Read everystack://schema-patterns for the `defineModel` patterns and examples. Key conventions:
83
+
84
+ - UUID primary keys with `field.uuid().primaryKey().defaultRandom()`
85
+ - `createdAt` and `updatedAt` timestamps on all tables
86
+ - Foreign key references with `.references('users', 'id')`
87
+ - Relations declared on the Model (used for both SSR and API query embedding)
88
+ - Authorization declared with `can()` abilities (compiles to grants + RLS)
89
+
90
+ For views, materialized views, functions, and triggers, read everystack://derived-objects
91
+ — they are declared as descriptors and deploy via `db:reconcile`, not migrations.
85
92
 
86
93
  ## Step 5: Configure the handler
87
94
 
@@ -121,29 +128,35 @@ export async function POST(request: Request) {
121
128
  // ... PATCH, DELETE
122
129
  ```
123
130
 
124
- ## Step 8: Run migrations
131
+ ## Step 8: Move your database to match your Models
125
132
 
126
- Migrations create the actual tables in your database based on your schema.
133
+ `db:sync` makes your local dev database match your checkout tables, authz, and the
134
+ derived layer — in one command. You never run `drizzle-kit` or write a SQL migration.
127
135
 
128
136
  ```bash
129
- npx drizzle-kit generate
130
- npx drizzle-kit migrate
137
+ everystack db:sync --database-url "$DATABASE_URL"
131
138
  ```
132
139
 
133
- The first command generates SQL files from your schema. The second runs them against your database.
140
+ This is the dev edit loop: change `db/models/`, run `db:sync`, test, repeat. When you're
141
+ ready to record the change as a migration for protected stages, run `everystack db:generate`
142
+ (and later reach a deployed stage with `db:plan` → `db:apply`).
143
+
144
+ Read everystack://database-operations for the full workflow — the dev loop, importing data,
145
+ and safely migrating a deployed stage.
134
146
 
135
147
  ## Step 9: Security
136
148
 
137
149
  **This is not optional.** Read everystack://security for the three-layer security model.
138
150
 
139
- Your database needs Row Level Security (RLS) policies. These are rules that control who can see and change what data. Even if your app code has a bug, the database enforces access control.
151
+ Your database needs Row Level Security (RLS) rules that control who can see and change what data. Even if your app code has a bug, the database enforces access control.
140
152
 
141
- At minimum:
142
- - Create database roles (`anon`, `authenticated`, `admin`)
143
- - Add RLS policies to every table
144
- - Set up GRANTs so each role can only access what it should
153
+ In everystack you **declare** this on your Models rather than hand-writing SQL: `can()`
154
+ abilities on each `defineModel` compile to database roles (`anon`, `authenticated`,
155
+ `admin`), GRANTs, and RLS policies, and `db:sync` deploys them. `db:check` (the CI gate)
156
+ refuses a table without RLS. Never hand-write `CREATE ROLE`, `GRANT`, or `CREATE POLICY`.
145
157
 
146
- The security resource has copy-paste SQL templates for common patterns.
158
+ The security resource explains the policies your abilities compile to and the patterns
159
+ behind them.
147
160
 
148
161
  ## Step 10: Verify
149
162
 
package/dist/cli.md CHANGED
@@ -28,11 +28,52 @@ everystack update --channel staging --platform web
28
28
  Flags: `--channel` (default: production), `--message`, `--platform` (ios/android/web/all), `--skip-export`.
29
29
 
30
30
  ### Database
31
+
32
+ You declare tables in `db/models/` (`defineModel`) and derived objects as descriptors;
33
+ these verbs move a database. Full operational guidance: everystack://database-operations.
34
+
35
+ **Schema (declare → deploy)**
36
+ ```bash
37
+ everystack db:sync # Make a dev DB match your checkout — state + authz + compute, one verb
38
+ everystack db:generate # STATE layer (tables) → next migration file (--dry-run prints the SQL, writes nothing)
39
+ everystack db:reconcile # Deploy the compute layer only (views/matviews/functions/triggers)
40
+ everystack db:check # CI gate: declared state composes + generated artifacts match regeneration
41
+ everystack db:fingerprint # Content-address the live base schema vs the Models — MATCH/MISMATCH
42
+ everystack db:diff # The state edge between two declared states — no DB, CI-pure
43
+ ```
44
+
45
+ **Safe remote migration (protected stages)**
46
+ ```bash
47
+ everystack db:plan --stage X --out X.plan.json # Mint a reviewable edge — read-only, ephemeral
48
+ everystack db:apply --plan X.plan.json --stage X # Verify → apply → verify, credential-free via the ops Lambda
49
+ everystack db:approvers --stage X --set "…" # Declare who may run destructive applies (STS-verified)
50
+ ```
51
+
52
+ **Backups & safety**
53
+ ```bash
54
+ everystack db:snapshot --stage X # Physical RDS snapshot (instant DR)
55
+ everystack db:snapshots --stage X # List manual RDS snapshots
56
+ everystack db:backup --stage X # Logical pg_dump → private S3 backups bucket
57
+ everystack db:backups --stage X # List logical backups
58
+ everystack db:backup:download <id> --stage X # Presigned download URL (1h)
59
+ everystack db:restore --from <id> --stage X --confirm # Restore a backup INTO a stage (destructive)
60
+ ```
61
+
62
+ **Local dev databases**
63
+ ```bash
64
+ everystack db:template:refresh # Build the dev template from declared state + seed
65
+ everystack db:branch # Per-git-branch dev DB from the template (--list / --prune)
66
+ everystack db:fork --from-stage src --stage tgt --confirm # Fork a deployed stage's DB into a feature stage
67
+ ```
68
+
69
+ **Data**
31
70
  ```bash
32
- everystack db:migrate # Run migrations via Lambda
33
- everystack db:seed # Seed database (dev only)
71
+ everystack db:seed --stage dev # Declarative seed script (dev only; refuses on production)
72
+ everystack db:backfill --apply # One-shot data moves from db/backfills/*.sql (own record)
73
+ everystack pipeline:run --stage X # Ingest pipeline (credential-free in the ops Lambda)
74
+ everystack db:migrate # Run the generated migration on a deployed stage via Lambda invoke
34
75
  everystack db:psql --stage dev -c "SELECT * FROM posts" # Read-only SQL via Lambda
35
- everystack console --stage dev # Interactive REPL with db + schema
76
+ everystack console --stage dev # Interactive REPL with db + schema
36
77
  ```
37
78
 
38
79
  The REPL has `db`, `schema`, `eq`, `and`, `or`, `gt`, `lt`, `count`, `sum`, `avg`, `sql`, `desc`, `asc` in scope.
package/dist/core.md CHANGED
@@ -158,11 +158,24 @@ const { data } = await api.rpc('timeline', { limit: 20 });
158
158
 
159
159
  All infrastructure commands use **AWS IAM credentials** (not shared secrets).
160
160
 
161
+ You declare tables in `db/models/` with `defineModel` and the derived layer as descriptors
162
+ (see everystack://derived-objects); the CLI moves the database — you never run
163
+ `drizzle-kit` or hand-write a migration. Full operational guidance:
164
+ everystack://database-operations.
165
+
161
166
  | Command | What it does |
162
167
  |---------|-------------|
163
168
  | `everystack update --channel production` | OTA deploy (no infrastructure changes) |
164
- | `everystack db:migrate` | Run Drizzle migrations via Lambda invoke |
169
+ | `everystack db:sync` | Dev DB your checkout (state + authz + compute), one verb |
170
+ | `everystack db:generate` | STATE layer (tables) → next migration file |
171
+ | `everystack db:reconcile` | Deploy the compute layer (views/matviews/functions/triggers) |
172
+ | `everystack db:check` | CI gate: declared state composes + artifacts match regeneration |
173
+ | `everystack db:plan --stage X` | Mint a reviewable, fingerprint-pinned edge (read-only) |
174
+ | `everystack db:apply --plan … --stage X` | Verify → apply → verify, credential-free via the ops Lambda |
175
+ | `everystack db:snapshot / db:backup --stage X` | Physical RDS snapshot / logical pg_dump before a risky migration |
165
176
  | `everystack db:seed` | Seed database via Lambda invoke (dev only) |
177
+ | `everystack db:backfill --apply` | One-shot data moves from `db/backfills/*.sql` |
178
+ | `everystack db:migrate` | Run the generated migration on a deployed stage via Lambda invoke |
166
179
  | `everystack db:psql --stage dev -c "SQL"` | Execute read-only SQL via Lambda |
167
180
  | `everystack console --stage dev` | Interactive REPL with db + schema in scope |
168
181
  | `everystack logs:errors --stage dev` | Query recent error logs |
@@ -173,6 +186,11 @@ All infrastructure commands use **AWS IAM credentials** (not shared secrets).
173
186
  | `everystack diag URL` | Diagnose deployed page freshness |
174
187
  | `everystack analyze:ssr` | Static analysis for SSR anti-patterns |
175
188
 
189
+ On a dev database you `db:sync` freely; a protected stage takes `db:plan` → `db:apply`
190
+ (fingerprint-verified, snapshot + approver gates on destructive plans); `db:check` is the
191
+ per-PR CI gate; `db:reconcile` deploys compute; `db:seed`/`db:backfill` move data;
192
+ `db:snapshot`/`db:backup` are the safety net.
193
+
176
194
  ## Key Patterns
177
195
 
178
196
  **Schema-agnostic.** The library knows nothing about your tables. You pass your Drizzle schema to `createHandler()`. Your schema, your migrations, your database.
@@ -0,0 +1,236 @@
1
+ # Database Operations: local dev, data import, and SAFE remote migration
2
+
3
+ > The one thing to internalize: **on a dev database you `db:sync` freely; on any protected
4
+ > stage you `db:plan`, snapshot, `db:apply`, and a human approves destruction.** The
5
+ > database's own declared state is the authority — every change is verified against a
6
+ > content-addressed fingerprint, and the operator never holds a production database URL.
7
+
8
+ Read this to change a deployed schema, import data, or set up a local dev database. For
9
+ declaring the objects themselves, see `everystack://schema-patterns` (tables) and
10
+ `everystack://derived-objects` (views/matviews/functions/triggers).
11
+
12
+ ## The model: two layers, three verbs by target
13
+
14
+ The schema is **state** (tables, constraints, authz, sequences — migrated) and **compute**
15
+ (functions, views, matviews, triggers — reconciled). You almost never run the layer verbs
16
+ by hand; you run one of three verbs, chosen by WHERE you are applying:
17
+
18
+ | Target | Verb | What it does |
19
+ |--------|------|--------------|
20
+ | **Dev database** (yours, direct URL) | `db:sync` | Make the database match your checkout — state + compute + authz, one verb. Fast, no ceremony. |
21
+ | **Protected stage** (dev/staging/prod, deployed) | `db:plan` → `db:apply` | Mint a reviewable edge, then apply it credential-free with before/after verification. |
22
+ | **CI / every PR** | `db:check` | Gate: the declared state composes and generated artifacts match regeneration. Exit 1 on drift. |
23
+
24
+ `db:sync` is for databases you own directly. `db:plan`/`db:apply` is for databases you
25
+ reach through the deployed ops Lambda — you never hold their URL.
26
+
27
+ ## Local development
28
+
29
+ ### A per-branch dev database
30
+
31
+ ```bash
32
+ everystack db:template:refresh # build the dev template <base>_tpl from the DECLARED
33
+ # state + the app's db:seed script (never a data copy)
34
+ everystack db:branch # mint a dev DB for the current git branch from the template
35
+ # (seed rows inherited); db:sync evolves it as you edit
36
+ everystack db:branch --list # what branch DBs exist
37
+ everystack db:branch --prune --confirm # drop DBs whose branches are gone (mapped via COMMENT)
38
+ ```
39
+
40
+ The template is built from your **declared** models + seed, never copied from a real
41
+ database — so a fresh branch DB is reproducible and PII-free.
42
+
43
+ ### The edit loop
44
+
45
+ ```bash
46
+ # edit db/models/ — a table field, or a descriptor body in derived.ts
47
+ everystack db:sync --database-url "$DATABASE_URL" # the DB now matches your checkout
48
+ pnpm test
49
+ # iterate; commit when happy — the model/descriptor diff IS the change
50
+ ```
51
+
52
+ `db:sync` flags: `--allow-drops` (include destructive changes; held back by default),
53
+ `--overwrite-drift` (rebuild a hand-edited derived object from source).
54
+
55
+ ### Seeing the change as SQL first
56
+
57
+ ```bash
58
+ everystack db:generate --dry-run --database-url "$DATABASE_URL" # the edge as SQL, writes NOTHING
59
+ everystack db:diff --from-models db/models/index.ts --check # models-vs-models edge, no DB at all (CI-pure)
60
+ ```
61
+
62
+ `db:generate` (without `--dry-run`) writes the next migration file for the STATE layer;
63
+ `db:sync` and `db:apply` are how it reaches a database.
64
+
65
+ ## Importing data properly
66
+
67
+ Data has three legitimate homes. **Never hand-write ad-hoc `INSERT`s in a migration or run
68
+ them through `psql` for anything repeatable** — that is unversioned, unrepeatable, and a
69
+ cheat gate flags bundled data.
70
+
71
+ 1. **`db:seed`** — the app's declarative seed script, for dev/demo rows. Runs on dev
72
+ databases (and dev-tier stages); refuses on production.
73
+ ```bash
74
+ everystack db:seed --stage dev
75
+ ```
76
+ 2. **`db:backfill`** — one-shot DATA moves authored as SQL in `db/backfills/*.sql`. Identity
77
+ is the file's **content hash** (renaming or reformatting is a no-op), each database keeps
78
+ its own record (`everystack.backfill_log`), an edited-after-run file is **BLOCKED**, and
79
+ it **never runs as a schema side effect** — you run it deliberately.
80
+ ```bash
81
+ everystack db:backfill # list pending (dry)
82
+ everystack db:backfill --apply # run the pending backfills
83
+ ```
84
+ Use it for expand → backfill → contract: add the new column (schema), move the data
85
+ (backfill), drop the old column later (a separate destructive plan).
86
+ 3. **The ingest pipeline** — for real, ongoing data ingestion (`defineStage`/`definePipeline`,
87
+ run via `pipeline:run`). This is the framework path for loading a dataset, not a script.
88
+ ```bash
89
+ everystack pipeline:run --stage dev # credential-free, runs in the ops Lambda
90
+ ```
91
+
92
+ ## Migrating a REMOTE stage SAFELY
93
+
94
+ This is the ceremony. Every step exists because a shortcut here loses data or races another
95
+ operator. Do them in order.
96
+
97
+ ### 1. Gate it in CI
98
+
99
+ ```bash
100
+ everystack db:check # the declared state composes; generated artifacts match regeneration
101
+ ```
102
+
103
+ `db:check` is the per-PR gate. With a scratch PostgreSQL it builds the whole declared
104
+ database from scratch and requires a fingerprint MATCH. A PR that fails `db:check` is not
105
+ mergeable — the schema is broken before it ever reaches a stage.
106
+
107
+ ### 2. Mint a reviewable plan
108
+
109
+ ```bash
110
+ everystack db:plan --stage production --out prod.plan.json
111
+ ```
112
+
113
+ `db:plan` is **read-only**. It asks the target its live fingerprint, diffs your checkout
114
+ against it, and writes one plan file: the edge plus both endpoint fingerprints. It prints
115
+ the **shape first** — a histogram (`shape: 402 policies, 398 grants, 201 rls, …`) before any
116
+ SQL wall — and itemizes every DESTRUCTIVE statement in red with the rebuild cost. A plan
117
+ with held drops refuses to mint unless you pass `--allow-drops` (destruction is carried
118
+ explicitly). Plans are **ephemeral** — attach them to the PR/release, never commit them.
119
+
120
+ ### 3. Read the plan
121
+
122
+ Look at the shape, the destructive itemization, and the rebuild estimate. This is the
123
+ review surface — the point where a human (or you) decides the edge is what was intended.
124
+
125
+ ### 4. Apply it credential-free
126
+
127
+ ```bash
128
+ everystack db:apply --plan prod.plan.json --stage production
129
+ ```
130
+
131
+ With `--stage`, the WRITE runs **inside the ops Lambda over IAM — the operator never holds
132
+ the database URL**. `db:apply` verifies → applies → verifies, and **refuses** unless:
133
+
134
+ - **the live fingerprint still equals `plan.from`** (the concurrency lock — nobody changed
135
+ the target since you minted the plan; if they did, re-mint), AND
136
+ - **your checkout descends from the commit that declares the target's state** (the
137
+ fast-forward rule — "rebase first"; a stale checkout can't silently revert merged work).
138
+
139
+ Every refusal is recorded in `everystack.schema_log`. `--database-url` is local-dev-only;
140
+ protected stages always go through `--stage`.
141
+
142
+ ### 5. Destructive plans carry more gates
143
+
144
+ A plan is DESTRUCTIVE if it drops (table/column/type) or narrows a type (a lossy
145
+ `SET DATA TYPE`). Those additionally require, all of them:
146
+
147
+ - **`--confirm`** — always, explicitly.
148
+ - **A snapshot** — taken automatically via `db:backup` when you pass `--stage` (or supply
149
+ one with `--snapshot-ref`). No snapshot, no destructive apply.
150
+ - **The stage's approver set** — if declared, `db:apply` verifies the caller's AWS identity
151
+ (STS) against it. Set it with `db:approvers`:
152
+ ```bash
153
+ everystack db:approvers --stage production --set "cto,arn:aws:iam::…:user/cto"
154
+ everystack db:approvers --stage production --set '' # disable destructive applies entirely
155
+ ```
156
+
157
+ Prefer never to destroy in place. The safe path for removing a column is the **contract
158
+ phase**: `field.deprecated()` first (the column stays readable, new writes are rejected,
159
+ generated types strike it through), then the physical drop later as its own confirmed,
160
+ snapshotted, approver-gated plan.
161
+
162
+ ## Snapshot-first: the safety net
163
+
164
+ Two backup mechanisms, different shapes — know when to reach for each:
165
+
166
+ | | `db:snapshot` | `db:backup` |
167
+ |---|---|---|
168
+ | Kind | **Physical** RDS snapshot (whole instance) | **Logical** `pg_dump` of one database → private S3 |
169
+ | Speed to safety | Instant (control-plane, no VPC) | Minutes (dumps the data) |
170
+ | Restore | RDS restore (new instance) — instant DR | `db:restore` INTO a stage; downloadable, portable |
171
+ | Requires | RDS (not Aurora Serverless v2 the same way) | any Postgres reachable by the ops Lambda |
172
+
173
+ ```bash
174
+ everystack db:snapshot --stage production # physical, instant — the "oh no" button
175
+ everystack db:snapshots --stage production # list manual snapshots
176
+ everystack db:backup --stage production # logical pg_dump → S3 backups bucket
177
+ everystack db:backups --stage production # list logical backups
178
+ everystack db:backup:download <id> --stage production # presigned download URL (1h)
179
+ ```
180
+
181
+ **RDS snapshot first?** For a destructive apply, `db:apply` already forces a logical backup —
182
+ you cannot destroy without one. But before any large or risky migration on a stage that
183
+ holds real data, take a **physical `db:snapshot` first** as well: it is instant, it captures
184
+ the whole instance, and restoring it is the fastest path back if something the plan didn't
185
+ foresee goes wrong. Cheap insurance; take it.
186
+
187
+ ## Importing data into a REMOTE stage
188
+
189
+ - **`db:seed --stage dev`** — the declarative seed, dev-tier only (refuses on production).
190
+ - **`pipeline:run --stage <name>`** — the ingest pipeline, credential-free in the ops
191
+ Lambda. The way to load a real dataset into a deployed stage.
192
+ - **`db:fork --from-stage <src> --stage <tgt> --confirm`** — copy a DEPLOYED stage's
193
+ database (schema + data) into a feature stage: backup → presigned URL (the cross-stage
194
+ hand-off) → restore via the target's ops Lambda. **Production is never a target**, and
195
+ forking FROM a prod-tier stage warns about PII. Use it to get realistic data into a
196
+ throwaway stage.
197
+ - **`db:restore --from <id> --stage <name> --confirm`** — restore a logical backup INTO a
198
+ stage. Destructive (it replaces the target), so `--confirm` is required.
199
+
200
+ ## Why `--stage` is the safe boundary
201
+
202
+ Operator verbs that touch a protected database run their WRITE inside that stage's **ops
203
+ Lambda**, invoked over IAM. The operator's own credentials never resolve the database URL —
204
+ your AWS identity IS the authorization, CloudTrail records the access, and a leaked laptop
205
+ can't dump production. `--database-url` exists only for local dev databases you own outright.
206
+
207
+ ## Command index
208
+
209
+ ```bash
210
+ # Declare (edit files), then:
211
+ everystack db:sync # dev DB ← your checkout (state + compute), one verb
212
+ everystack db:reconcile # compute layer only (views/matviews/functions/triggers)
213
+ everystack db:generate # STATE layer → migration file / --apply / --dry-run
214
+ everystack db:check # CI gate: declared state composes; artifacts match
215
+ everystack db:fingerprint # content-address live schema vs models — MATCH/MISMATCH
216
+
217
+ # Reach a protected stage safely:
218
+ everystack db:plan --stage X --out X.plan.json # mint a reviewable edge (read-only)
219
+ everystack db:apply --plan X.plan.json --stage X # verify → apply → verify (credential-free)
220
+ everystack db:approvers --stage X --set "…" # who may run destructive applies
221
+
222
+ # Back up / restore:
223
+ everystack db:snapshot --stage X # physical RDS snapshot (instant DR)
224
+ everystack db:backup --stage X # logical pg_dump → S3
225
+ everystack db:restore --from <id> --stage X --confirm
226
+
227
+ # Local dev databases:
228
+ everystack db:template:refresh # build the dev template from declared state + seed
229
+ everystack db:branch # per-branch dev DB from the template
230
+
231
+ # Data:
232
+ everystack db:seed --stage dev # declarative seed (dev only)
233
+ everystack db:backfill --apply # one-shot data moves (db/backfills/*.sql)
234
+ everystack pipeline:run --stage X # ingest pipeline (credential-free)
235
+ everystack db:fork --from-stage prod --stage feature --confirm # copy a stage's data
236
+ ```
@@ -0,0 +1,225 @@
1
+ # Derived Objects: functions, views, materialized views, triggers, sequences
2
+
3
+ > Derived objects don't migrate — they **deploy**. Base tables hold irreplaceable data
4
+ > and change through generated migrations. Functions, views, and materialized views are
5
+ > **compute**: declared source whose results are rebuildable from the tables at any time.
6
+ > You edit their descriptors, and `db:reconcile` makes the database match — no migration
7
+ > file, no journal entry, no numbered anything. Git is the history.
8
+
9
+ Read this when you need to add or change a view, materialized view, function, trigger, or
10
+ standalone sequence. For tables (`defineModel`) and the operational verbs
11
+ (`db:sync`/`db:plan`/`db:apply`), see `everystack://schema-patterns` and
12
+ `everystack://database-operations`.
13
+
14
+ ## Two layers, two lifecycles
15
+
16
+ A schema is two different things, and conflating them is the mistake:
17
+
18
+ | Layer | Objects | Changes via | History |
19
+ |-------|---------|-------------|---------|
20
+ | **State** | tables, constraints, authz, sequences | Models → `db:generate` → migration | migration + `db:pull` |
21
+ | **Compute** | functions, views, matviews, triggers | descriptors → `db:reconcile` | git |
22
+
23
+ Versioning compute through migrations is a category error — the same as writing a
24
+ migration to edit a Lambda handler. A one-line fix to a materialized view should be a
25
+ one-line git diff, not a restatement of the whole view inside a numbered file. (In one
26
+ real analytics app, 34 of 43 migrations were pure derived-layer churn, including a 701 KB
27
+ migration that fixed a one-line regex.) Descriptors end that.
28
+
29
+ ## The declared home
30
+
31
+ The whole database — both layers — lives in `db/models/`, in one format:
32
+
33
+ ```
34
+ db/
35
+ ├── models/ # the declared database: defineModel tables + derived descriptors
36
+ │ ├── post.ts
37
+ │ ├── derived.ts # defineView / defineMaterializedView / defineFunction / defineSql
38
+ │ └── index.ts # defineModule({ models, sequences, derived })
39
+ └── schema.generated.ts # generated artifact (never edited; db:check refuses drift)
40
+ ```
41
+
42
+ **`db/sql/` is retired.** Raw-SQL derived objects moved into descriptors, and every CLI
43
+ verb hard-fails on a `db/sql/` directory that still carries `.sql` files. If you are
44
+ adopting an app that still has one, see "Adopting an existing database" below.
45
+
46
+ The boundary rule, uniform with models: **structure is declared** (identity,
47
+ dependencies, authz, security posture, indexes, signatures, trigger events);
48
+ **expressions stay `sql``** (the SELECT, the function body, a WHEN predicate). No query
49
+ DSL, no string mini-grammars in declared properties.
50
+
51
+ ```ts
52
+ import { defineView, defineMaterializedView, defineFunction, defineSql, defineModule,
53
+ trigger, arg, setof, field, can, index, sql } from '@everystack/model';
54
+
55
+ // A VIEW — securityInvoker is the one forced decision.
56
+ export const activeUsers = defineView('active_users', {
57
+ securityInvoker: true, // REQUIRED, no default:
58
+ // true = readers' own RLS/grants apply through the view
59
+ // false = definer semantics (the sealed-slice pattern —
60
+ // a view grant WITHOUT a table grant; the body must seal)
61
+ abilities: [can('read')], // grants are AUTHORED, never inherited
62
+ dependsOn: [users], // declared, then VERIFIED against the live edges (drift-checked)
63
+ fields: { id: field.uuid().notNull(), name: field.text().notNull() }, // OPTIONAL: typed SSR
64
+ // reads — emits a pgView().existing() into schema.generated.ts;
65
+ // never touches the DDL or the content hash
66
+ as: sql`SELECT id, name FROM users WHERE deleted_at IS NULL`,
67
+ });
68
+
69
+ // A MATERIALIZED VIEW — grants-only authz; refresh/populate declare the HOW.
70
+ export const leaderboard = defineMaterializedView('leaderboard', {
71
+ abilities: [can('read')],
72
+ dependsOn: [scores, activeUsers],
73
+ indexes: [index(['user_id']).unique(), index('tags').using('gin')],
74
+ refresh: 'concurrently', // compile-gated: REQUIRES a unique, non-partial index
75
+ populate: 'deferred', // create WITH NO DATA, refresh outside the creation lock
76
+ as: sql`SELECT user_id, sum(points) AS points FROM scores GROUP BY user_id`,
77
+ });
78
+
79
+ // A FUNCTION — signature structured, body raw.
80
+ export const searchPosts = defineFunction('search_posts', {
81
+ args: [arg('query', 'text')],
82
+ returns: setof(posts), // a descriptor ref — a typo is a TS error, the dep comes free
83
+ language: 'sql',
84
+ volatility: 'stable', // default 'volatile'; 'immutable' enables expression indexes
85
+ abilities: [can('execute', { role: 'authenticated' })], // role REQUIRED; PUBLIC is always revoked
86
+ body: sql`SELECT * FROM posts WHERE ts @@ websearch_to_tsquery(query)`,
87
+ });
88
+
89
+ // A SECURITY DEFINER function MUST pin its search_path (the classic landmine):
90
+ export const promote = defineFunction('promote_user', {
91
+ args: [arg('uid', 'uuid')], returns: 'void', language: 'plpgsql',
92
+ security: 'definer', searchPath: ['pg_catalog', 'public'], // required with definer
93
+ abilities: [can('execute', { role: 'admin' })],
94
+ body: sql`BEGIN UPDATE users SET role = 'admin' WHERE id = uid; END`,
95
+ });
96
+
97
+ // A TRIGGER FUNCTION is a standalone function returning 'trigger'…
98
+ export const touchUpdatedAt = defineFunction('touch_updated_at', {
99
+ returns: 'trigger', language: 'plpgsql',
100
+ body: sql`BEGIN NEW.updated_at = now(); RETURN NEW; END`,
101
+ });
102
+
103
+ // …and the TRIGGER is declared ON THE MODEL it fires on (owner consent, visible where the
104
+ // table lives) — see defineModel below.
105
+
106
+ export const appModule = defineModule({
107
+ models: [posts, users],
108
+ derived: [activeUsers, leaderboard, searchPosts, promote, touchUpdatedAt],
109
+ // sequences: [invoiceNo], // standalone sequences are STATE — see below
110
+ });
111
+ ```
112
+
113
+ Triggers ride the model:
114
+
115
+ ```ts
116
+ export const Post = defineModel('posts', {
117
+ fields: { id: field.uuid().primaryKey(), updatedAt: field.timestamptz() },
118
+ abilities: [can('read')],
119
+ triggers: [trigger('touch', {
120
+ timing: 'before', events: ['update'], forEach: 'row',
121
+ execute: touchUpdatedAt, // the trigger function MUST be in the module's derived set
122
+ })],
123
+ });
124
+ ```
125
+
126
+ ## The rules that make it safe (all enforced)
127
+
128
+ - **`securityInvoker` is a required decision** on every view — no default. Pick `true`
129
+ (readers' own RLS applies) or `false` (definer semantics, the sealed slice).
130
+ - **Grants are authored, never inherited.** `can('read')` on a relation → `anon` +
131
+ `authenticated`; `can('read', { role })` narrows. A granted view whose readers can't
132
+ reach its underlying tables is a **compile error** — the reachability gate.
133
+ - **SECURITY DEFINER without a pinned `searchPath` is rejected** at define time.
134
+ - **`can('execute')` needs a role** — bare execute has no safe audience; PUBLIC is always
135
+ revoked by the compiler.
136
+ - **A missing import is caught loudly.** An `undefined` in `dependsOn`/`derived`/a
137
+ trigger's `execute` (a renamed or missing named import) is rejected at define time,
138
+ naming the descriptor and the index — never a silent drop.
139
+ - **`dependsOn` is declared, then verified.** The reachability gate walks it at compile
140
+ time; the live catalog's actual edges check it at plan time. A view that reads a table
141
+ it never declared (or declares one it never reads) is a `db:check` failure.
142
+ - **REFRESH CONCURRENTLY requires a unique, non-partial index** — compile-gated.
143
+ - **Content hashing is comment- and whitespace-insensitive** — reformatting a body is a
144
+ no-op; only a real change reconciles.
145
+
146
+ The escape hatch `defineSql` handles what the structured vocabulary can't say yet
147
+ (aggregates, operators, event triggers) — still declared, hashed, dependency-ordered. It
148
+ refuses a `CREATE [MATERIALIZED] VIEW` so accidental relations land in the read model
149
+ where the gates can see them; its SQL is otherwise trusted as authored.
150
+
151
+ ## Sequences are STATE, not compute
152
+
153
+ A standalone sequence (one not owned by a serial column) holds a counter value, so it
154
+ migrates with the base schema — declare it with `defineSequence` and pass it on
155
+ `defineModule({ sequences })`. It is fingerprinted with the tables and created before them.
156
+ This is the one derived-looking object that is state, not compute.
157
+
158
+ ```ts
159
+ export const invoiceNo = defineSequence('invoice_no', { start: 1000 });
160
+ ```
161
+
162
+ ## Deploying: `db:reconcile`
163
+
164
+ ```bash
165
+ everystack db:reconcile # plan (read-only; works via --stage or --database-url)
166
+ everystack db:reconcile --check # CI gate: exit 1 when anything differs
167
+ everystack db:reconcile --apply --database-url … # execute (direct connection required)
168
+ ```
169
+
170
+ Planning works over the read-only ops Lambda (`--stage`); **`--apply` needs a direct
171
+ connection** — the deployed query path is read-only by design. On a dev database, prefer
172
+ `db:sync` (it does state + compute in one verb); reach for `db:reconcile --apply` when you
173
+ want compute only.
174
+
175
+ A plan shows the rebuild cost before you say yes — dependent cascades, matview sizes, row
176
+ counts:
177
+
178
+ ```
179
+ = 51 up to date
180
+ ~ replace public.score (source changed)
181
+ ↻ refresh public.season_scores (depends on replaced function public.score; 1.2 GB, ~4M rows)
182
+ + create public.weekly_top (new in source)
183
+ ≠ regrant public.leaderboard — live grants differ (REVOKE SELECT ON leaderboard FROM intern)
184
+ estimated rebuild: 1.2 GB across 1 matview(s)
185
+ ```
186
+
187
+ `--apply` sends all DDL as **one transaction** (a failed create rolls back the drops
188
+ before it), records provenance and a `schema_log` memoir (the full DDL, when, by whom,
189
+ from which git commit), and rebuilds dependents automatically in dependency order.
190
+
191
+ ## Semantics worth knowing
192
+
193
+ - **Drift is never overwritten silently.** A hand-edited managed object (someone ran
194
+ `CREATE OR REPLACE` against the DB) makes the plan report drift and `--apply` refuse.
195
+ Fix the source to match, or rebuild from source with `--overwrite-drift`.
196
+ - **Grant drift converges without ceremony** — a hand-run GRANT/REVOKE surfaces as an
197
+ idempotent `regrant` back to the declared abilities. Declared authz is authoritative.
198
+ - **Changing a function REFRESHes the matviews that call it** — their rows are stale the
199
+ moment the function changes, and the reconciler knows it.
200
+ - **Unmanaged objects are never touched** — live derived objects with no source and no
201
+ provenance are listed and left alone.
202
+
203
+ ## Adopting an existing database (baseline / rebaseline)
204
+
205
+ - **First contact wants `--baseline`.** Against a database that already has your derived
206
+ layer (built by old migrations), the reconciler refuses to guess whether live matches
207
+ source. `everystack db:pull` renders the live layer as descriptors (bodies from the
208
+ catalog's canonical deparse, grants as abilities, `dependsOn` from live edges). The path:
209
+ **pull → review → commit → `db:reconcile --apply --baseline` → a silent check.**
210
+ Use `db:pull --derived-out db/models/derived.ts` to write the derived layer straight
211
+ into its own file when you already have a hand-maintained models barrel.
212
+ - **A `db/sql`-era migration wants `--rebaseline`.** When provenance exists but the source
213
+ was re-rendered (descriptors render canonical SQL, so the source hash differs even for
214
+ identical SQL), `--rebaseline` re-records the new source hash for every object whose live
215
+ definition still matches what the reconciler last applied — no drop, no create, no
216
+ matview repopulation. The live side is VERIFIED; the source side is TRUSTED. Migrate
217
+ first (verbatim), edit after. `--baseline` and `--rebaseline` compose in one run.
218
+
219
+ ## Current limitations
220
+
221
+ - **Function identity is by name** — overloads are rejected at define time. Rename one.
222
+ - **Function signature changes** (return/arg types) can't `CREATE OR REPLACE` — drop the
223
+ old function in a reviewed step, or rename.
224
+ - Postgres can't see function→function dependencies inside bodies — `dependsOn` is your
225
+ declaration there.