@everystack/mcp 0.3.3 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/adding-database.md +36 -23
- package/dist/cli.md +44 -3
- package/dist/core.md +19 -1
- package/dist/database-operations.md +236 -0
- package/dist/derived-objects.md +225 -0
- package/dist/index.cjs +112 -102
- package/dist/project-claude-md.md +15 -12
- package/dist/schema-patterns.md +92 -106
- package/package.json +3 -3
- package/src/gates/detectors/hand-written-migration.ts +12 -5
- package/src/index.ts +1 -1
- package/src/prompts/add-feature.ts +4 -4
- package/src/prompts/debug.ts +3 -4
- package/src/prompts/deploy.ts +17 -8
- package/src/prompts/design-schema.ts +45 -59
- package/src/prompts/new-app.ts +21 -19
- package/src/prompts/secure.ts +3 -3
- package/src/resources/adding-database.md +36 -23
- package/src/resources/cli.md +44 -3
- package/src/resources/core.md +19 -1
- package/src/resources/database-operations.md +236 -0
- package/src/resources/derived-objects.md +225 -0
- package/src/resources/index.ts +15 -1
- package/src/resources/project-claude-md.md +15 -12
- package/src/resources/schema-patterns.md +92 -106
package/dist/adding-database.md
CHANGED
|
@@ -13,7 +13,7 @@ You already have a running Expo app. Now you're adding:
|
|
|
13
13
|
- **@everystack/logging** — structured logging so you can see what your app is doing.
|
|
14
14
|
- **@everystack/security** — device attestation and biometric auth.
|
|
15
15
|
- **@everystack/query** — React hooks that make it easy to fetch and display data.
|
|
16
|
-
- **
|
|
16
|
+
- **@everystack/model** — declares your database tables in TypeScript with `defineModel`. You describe what data you want to store and who can read/write it, and everystack generates the migration and the typed schema for you.
|
|
17
17
|
|
|
18
18
|
## Step 1: Install PostgreSQL
|
|
19
19
|
|
|
@@ -64,24 +64,31 @@ You should see a table with the value `1`. If so, your database is ready.
|
|
|
64
64
|
From your project directory:
|
|
65
65
|
|
|
66
66
|
```bash
|
|
67
|
-
pnpm add @everystack/api @everystack/auth @everystack/admin @everystack/logging @everystack/security @everystack/query
|
|
67
|
+
pnpm add @everystack/api @everystack/auth @everystack/admin @everystack/logging @everystack/security @everystack/query @everystack/model
|
|
68
68
|
pnpm add drizzle-orm
|
|
69
|
-
pnpm add -D drizzle-kit
|
|
70
69
|
```
|
|
71
70
|
|
|
72
|
-
## Step 4:
|
|
71
|
+
## Step 4: Declare your database schema
|
|
73
72
|
|
|
74
|
-
The schema defines what data your app stores. Create
|
|
73
|
+
The schema defines what data your app stores. Create your tables in `db/models/` — one
|
|
74
|
+
`defineModel` per table (for example `db/models/post.ts`), composed onto a `defineModule`
|
|
75
|
+
in `db/models/index.ts`.
|
|
75
76
|
|
|
76
|
-
A
|
|
77
|
+
A model is like designing a spreadsheet before you start filling in data. You decide what columns each table has, what type of data goes in each column, and how tables relate to each other — plus who can read and write each table.
|
|
77
78
|
|
|
78
|
-
|
|
79
|
+
You do **not** hand-write `db/schema.ts` and you do **not** run `drizzle-kit generate` —
|
|
80
|
+
everystack generates the migration and the typed artifact from your Models.
|
|
79
81
|
|
|
80
|
-
-
|
|
81
|
-
|
|
82
|
-
-
|
|
83
|
-
-
|
|
84
|
-
-
|
|
82
|
+
Read everystack://schema-patterns for the `defineModel` patterns and examples. Key conventions:
|
|
83
|
+
|
|
84
|
+
- UUID primary keys with `field.uuid().primaryKey().defaultRandom()`
|
|
85
|
+
- `createdAt` and `updatedAt` timestamps on all tables
|
|
86
|
+
- Foreign key references with `.references('users', 'id')`
|
|
87
|
+
- Relations declared on the Model (used for both SSR and API query embedding)
|
|
88
|
+
- Authorization declared with `can()` abilities (compiles to grants + RLS)
|
|
89
|
+
|
|
90
|
+
For views, materialized views, functions, and triggers, read everystack://derived-objects
|
|
91
|
+
— they are declared as descriptors and deploy via `db:reconcile`, not migrations.
|
|
85
92
|
|
|
86
93
|
## Step 5: Configure the handler
|
|
87
94
|
|
|
@@ -121,29 +128,35 @@ export async function POST(request: Request) {
|
|
|
121
128
|
// ... PATCH, DELETE
|
|
122
129
|
```
|
|
123
130
|
|
|
124
|
-
## Step 8:
|
|
131
|
+
## Step 8: Move your database to match your Models
|
|
125
132
|
|
|
126
|
-
|
|
133
|
+
`db:sync` makes your local dev database match your checkout — tables, authz, and the
|
|
134
|
+
derived layer — in one command. You never run `drizzle-kit` or write a SQL migration.
|
|
127
135
|
|
|
128
136
|
```bash
|
|
129
|
-
|
|
130
|
-
npx drizzle-kit migrate
|
|
137
|
+
everystack db:sync --database-url "$DATABASE_URL"
|
|
131
138
|
```
|
|
132
139
|
|
|
133
|
-
|
|
140
|
+
This is the dev edit loop: change `db/models/`, run `db:sync`, test, repeat. When you're
|
|
141
|
+
ready to record the change as a migration for protected stages, run `everystack db:generate`
|
|
142
|
+
(and later reach a deployed stage with `db:plan` → `db:apply`).
|
|
143
|
+
|
|
144
|
+
Read everystack://database-operations for the full workflow — the dev loop, importing data,
|
|
145
|
+
and safely migrating a deployed stage.
|
|
134
146
|
|
|
135
147
|
## Step 9: Security
|
|
136
148
|
|
|
137
149
|
**This is not optional.** Read everystack://security for the three-layer security model.
|
|
138
150
|
|
|
139
|
-
Your database needs Row Level Security (RLS)
|
|
151
|
+
Your database needs Row Level Security (RLS) — rules that control who can see and change what data. Even if your app code has a bug, the database enforces access control.
|
|
140
152
|
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
153
|
+
In everystack you **declare** this on your Models rather than hand-writing SQL: `can()`
|
|
154
|
+
abilities on each `defineModel` compile to database roles (`anon`, `authenticated`,
|
|
155
|
+
`admin`), GRANTs, and RLS policies, and `db:sync` deploys them. `db:check` (the CI gate)
|
|
156
|
+
refuses a table without RLS. Never hand-write `CREATE ROLE`, `GRANT`, or `CREATE POLICY`.
|
|
145
157
|
|
|
146
|
-
The security resource
|
|
158
|
+
The security resource explains the policies your abilities compile to and the patterns
|
|
159
|
+
behind them.
|
|
147
160
|
|
|
148
161
|
## Step 10: Verify
|
|
149
162
|
|
package/dist/cli.md
CHANGED
|
@@ -28,11 +28,52 @@ everystack update --channel staging --platform web
|
|
|
28
28
|
Flags: `--channel` (default: production), `--message`, `--platform` (ios/android/web/all), `--skip-export`.
|
|
29
29
|
|
|
30
30
|
### Database
|
|
31
|
+
|
|
32
|
+
You declare tables in `db/models/` (`defineModel`) and derived objects as descriptors;
|
|
33
|
+
these verbs move a database. Full operational guidance: everystack://database-operations.
|
|
34
|
+
|
|
35
|
+
**Schema (declare → deploy)**
|
|
36
|
+
```bash
|
|
37
|
+
everystack db:sync # Make a dev DB match your checkout — state + authz + compute, one verb
|
|
38
|
+
everystack db:generate # STATE layer (tables) → next migration file (--dry-run prints the SQL, writes nothing)
|
|
39
|
+
everystack db:reconcile # Deploy the compute layer only (views/matviews/functions/triggers)
|
|
40
|
+
everystack db:check # CI gate: declared state composes + generated artifacts match regeneration
|
|
41
|
+
everystack db:fingerprint # Content-address the live base schema vs the Models — MATCH/MISMATCH
|
|
42
|
+
everystack db:diff # The state edge between two declared states — no DB, CI-pure
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
**Safe remote migration (protected stages)**
|
|
46
|
+
```bash
|
|
47
|
+
everystack db:plan --stage X --out X.plan.json # Mint a reviewable edge — read-only, ephemeral
|
|
48
|
+
everystack db:apply --plan X.plan.json --stage X # Verify → apply → verify, credential-free via the ops Lambda
|
|
49
|
+
everystack db:approvers --stage X --set "…" # Declare who may run destructive applies (STS-verified)
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
**Backups & safety**
|
|
53
|
+
```bash
|
|
54
|
+
everystack db:snapshot --stage X # Physical RDS snapshot (instant DR)
|
|
55
|
+
everystack db:snapshots --stage X # List manual RDS snapshots
|
|
56
|
+
everystack db:backup --stage X # Logical pg_dump → private S3 backups bucket
|
|
57
|
+
everystack db:backups --stage X # List logical backups
|
|
58
|
+
everystack db:backup:download <id> --stage X # Presigned download URL (1h)
|
|
59
|
+
everystack db:restore --from <id> --stage X --confirm # Restore a backup INTO a stage (destructive)
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
**Local dev databases**
|
|
63
|
+
```bash
|
|
64
|
+
everystack db:template:refresh # Build the dev template from declared state + seed
|
|
65
|
+
everystack db:branch # Per-git-branch dev DB from the template (--list / --prune)
|
|
66
|
+
everystack db:fork --from-stage src --stage tgt --confirm # Fork a deployed stage's DB into a feature stage
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
**Data**
|
|
31
70
|
```bash
|
|
32
|
-
everystack db:
|
|
33
|
-
everystack db:
|
|
71
|
+
everystack db:seed --stage dev # Declarative seed script (dev only; refuses on production)
|
|
72
|
+
everystack db:backfill --apply # One-shot data moves from db/backfills/*.sql (own record)
|
|
73
|
+
everystack pipeline:run --stage X # Ingest pipeline (credential-free in the ops Lambda)
|
|
74
|
+
everystack db:migrate # Run the generated migration on a deployed stage via Lambda invoke
|
|
34
75
|
everystack db:psql --stage dev -c "SELECT * FROM posts" # Read-only SQL via Lambda
|
|
35
|
-
everystack console --stage dev
|
|
76
|
+
everystack console --stage dev # Interactive REPL with db + schema
|
|
36
77
|
```
|
|
37
78
|
|
|
38
79
|
The REPL has `db`, `schema`, `eq`, `and`, `or`, `gt`, `lt`, `count`, `sum`, `avg`, `sql`, `desc`, `asc` in scope.
|
package/dist/core.md
CHANGED
|
@@ -158,11 +158,24 @@ const { data } = await api.rpc('timeline', { limit: 20 });
|
|
|
158
158
|
|
|
159
159
|
All infrastructure commands use **AWS IAM credentials** (not shared secrets).
|
|
160
160
|
|
|
161
|
+
You declare tables in `db/models/` with `defineModel` and the derived layer as descriptors
|
|
162
|
+
(see everystack://derived-objects); the CLI moves the database — you never run
|
|
163
|
+
`drizzle-kit` or hand-write a migration. Full operational guidance:
|
|
164
|
+
everystack://database-operations.
|
|
165
|
+
|
|
161
166
|
| Command | What it does |
|
|
162
167
|
|---------|-------------|
|
|
163
168
|
| `everystack update --channel production` | OTA deploy (no infrastructure changes) |
|
|
164
|
-
| `everystack db:
|
|
169
|
+
| `everystack db:sync` | Dev DB ← your checkout (state + authz + compute), one verb |
|
|
170
|
+
| `everystack db:generate` | STATE layer (tables) → next migration file |
|
|
171
|
+
| `everystack db:reconcile` | Deploy the compute layer (views/matviews/functions/triggers) |
|
|
172
|
+
| `everystack db:check` | CI gate: declared state composes + artifacts match regeneration |
|
|
173
|
+
| `everystack db:plan --stage X` | Mint a reviewable, fingerprint-pinned edge (read-only) |
|
|
174
|
+
| `everystack db:apply --plan … --stage X` | Verify → apply → verify, credential-free via the ops Lambda |
|
|
175
|
+
| `everystack db:snapshot / db:backup --stage X` | Physical RDS snapshot / logical pg_dump before a risky migration |
|
|
165
176
|
| `everystack db:seed` | Seed database via Lambda invoke (dev only) |
|
|
177
|
+
| `everystack db:backfill --apply` | One-shot data moves from `db/backfills/*.sql` |
|
|
178
|
+
| `everystack db:migrate` | Run the generated migration on a deployed stage via Lambda invoke |
|
|
166
179
|
| `everystack db:psql --stage dev -c "SQL"` | Execute read-only SQL via Lambda |
|
|
167
180
|
| `everystack console --stage dev` | Interactive REPL with db + schema in scope |
|
|
168
181
|
| `everystack logs:errors --stage dev` | Query recent error logs |
|
|
@@ -173,6 +186,11 @@ All infrastructure commands use **AWS IAM credentials** (not shared secrets).
|
|
|
173
186
|
| `everystack diag URL` | Diagnose deployed page freshness |
|
|
174
187
|
| `everystack analyze:ssr` | Static analysis for SSR anti-patterns |
|
|
175
188
|
|
|
189
|
+
On a dev database you `db:sync` freely; a protected stage takes `db:plan` → `db:apply`
|
|
190
|
+
(fingerprint-verified, snapshot + approver gates on destructive plans); `db:check` is the
|
|
191
|
+
per-PR CI gate; `db:reconcile` deploys compute; `db:seed`/`db:backfill` move data;
|
|
192
|
+
`db:snapshot`/`db:backup` are the safety net.
|
|
193
|
+
|
|
176
194
|
## Key Patterns
|
|
177
195
|
|
|
178
196
|
**Schema-agnostic.** The library knows nothing about your tables. You pass your Drizzle schema to `createHandler()`. Your schema, your migrations, your database.
|
|
@@ -0,0 +1,236 @@
|
|
|
1
|
+
# Database Operations: local dev, data import, and SAFE remote migration
|
|
2
|
+
|
|
3
|
+
> The one thing to internalize: **on a dev database you `db:sync` freely; on any protected
|
|
4
|
+
> stage you `db:plan`, snapshot, `db:apply`, and a human approves destruction.** The
|
|
5
|
+
> database's own declared state is the authority — every change is verified against a
|
|
6
|
+
> content-addressed fingerprint, and the operator never holds a production database URL.
|
|
7
|
+
|
|
8
|
+
Read this to change a deployed schema, import data, or set up a local dev database. For
|
|
9
|
+
declaring the objects themselves, see `everystack://schema-patterns` (tables) and
|
|
10
|
+
`everystack://derived-objects` (views/matviews/functions/triggers).
|
|
11
|
+
|
|
12
|
+
## The model: two layers, three verbs by target
|
|
13
|
+
|
|
14
|
+
The schema is **state** (tables, constraints, authz, sequences — migrated) and **compute**
|
|
15
|
+
(functions, views, matviews, triggers — reconciled). You almost never run the layer verbs
|
|
16
|
+
by hand; you run one of three verbs, chosen by WHERE you are applying:
|
|
17
|
+
|
|
18
|
+
| Target | Verb | What it does |
|
|
19
|
+
|--------|------|--------------|
|
|
20
|
+
| **Dev database** (yours, direct URL) | `db:sync` | Make the database match your checkout — state + compute + authz, one verb. Fast, no ceremony. |
|
|
21
|
+
| **Protected stage** (dev/staging/prod, deployed) | `db:plan` → `db:apply` | Mint a reviewable edge, then apply it credential-free with before/after verification. |
|
|
22
|
+
| **CI / every PR** | `db:check` | Gate: the declared state composes and generated artifacts match regeneration. Exit 1 on drift. |
|
|
23
|
+
|
|
24
|
+
`db:sync` is for databases you own directly. `db:plan`/`db:apply` is for databases you
|
|
25
|
+
reach through the deployed ops Lambda — you never hold their URL.
|
|
26
|
+
|
|
27
|
+
## Local development
|
|
28
|
+
|
|
29
|
+
### A per-branch dev database
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
everystack db:template:refresh # build the dev template <base>_tpl from the DECLARED
|
|
33
|
+
# state + the app's db:seed script (never a data copy)
|
|
34
|
+
everystack db:branch # mint a dev DB for the current git branch from the template
|
|
35
|
+
# (seed rows inherited); db:sync evolves it as you edit
|
|
36
|
+
everystack db:branch --list # what branch DBs exist
|
|
37
|
+
everystack db:branch --prune --confirm # drop DBs whose branches are gone (mapped via COMMENT)
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
The template is built from your **declared** models + seed, never copied from a real
|
|
41
|
+
database — so a fresh branch DB is reproducible and PII-free.
|
|
42
|
+
|
|
43
|
+
### The edit loop
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
# edit db/models/ — a table field, or a descriptor body in derived.ts
|
|
47
|
+
everystack db:sync --database-url "$DATABASE_URL" # the DB now matches your checkout
|
|
48
|
+
pnpm test
|
|
49
|
+
# iterate; commit when happy — the model/descriptor diff IS the change
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
`db:sync` flags: `--allow-drops` (include destructive changes; held back by default),
|
|
53
|
+
`--overwrite-drift` (rebuild a hand-edited derived object from source).
|
|
54
|
+
|
|
55
|
+
### Seeing the change as SQL first
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
everystack db:generate --dry-run --database-url "$DATABASE_URL" # the edge as SQL, writes NOTHING
|
|
59
|
+
everystack db:diff --from-models db/models/index.ts --check # models-vs-models edge, no DB at all (CI-pure)
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
`db:generate` (without `--dry-run`) writes the next migration file for the STATE layer;
|
|
63
|
+
`db:sync` and `db:apply` are how it reaches a database.
|
|
64
|
+
|
|
65
|
+
## Importing data properly
|
|
66
|
+
|
|
67
|
+
Data has three legitimate homes. **Never hand-write ad-hoc `INSERT`s in a migration or run
|
|
68
|
+
them through `psql` for anything repeatable** — that is unversioned, unrepeatable, and a
|
|
69
|
+
cheat gate flags bundled data.
|
|
70
|
+
|
|
71
|
+
1. **`db:seed`** — the app's declarative seed script, for dev/demo rows. Runs on dev
|
|
72
|
+
databases (and dev-tier stages); refuses on production.
|
|
73
|
+
```bash
|
|
74
|
+
everystack db:seed --stage dev
|
|
75
|
+
```
|
|
76
|
+
2. **`db:backfill`** — one-shot DATA moves authored as SQL in `db/backfills/*.sql`. Identity
|
|
77
|
+
is the file's **content hash** (renaming or reformatting is a no-op), each database keeps
|
|
78
|
+
its own record (`everystack.backfill_log`), an edited-after-run file is **BLOCKED**, and
|
|
79
|
+
it **never runs as a schema side effect** — you run it deliberately.
|
|
80
|
+
```bash
|
|
81
|
+
everystack db:backfill # list pending (dry)
|
|
82
|
+
everystack db:backfill --apply # run the pending backfills
|
|
83
|
+
```
|
|
84
|
+
Use it for expand → backfill → contract: add the new column (schema), move the data
|
|
85
|
+
(backfill), drop the old column later (a separate destructive plan).
|
|
86
|
+
3. **The ingest pipeline** — for real, ongoing data ingestion (`defineStage`/`definePipeline`,
|
|
87
|
+
run via `pipeline:run`). This is the framework path for loading a dataset, not a script.
|
|
88
|
+
```bash
|
|
89
|
+
everystack pipeline:run --stage dev # credential-free, runs in the ops Lambda
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
## Migrating a REMOTE stage SAFELY
|
|
93
|
+
|
|
94
|
+
This is the ceremony. Every step exists because a shortcut here loses data or races another
|
|
95
|
+
operator. Do them in order.
|
|
96
|
+
|
|
97
|
+
### 1. Gate it in CI
|
|
98
|
+
|
|
99
|
+
```bash
|
|
100
|
+
everystack db:check # the declared state composes; generated artifacts match regeneration
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
`db:check` is the per-PR gate. With a scratch PostgreSQL it builds the whole declared
|
|
104
|
+
database from scratch and requires a fingerprint MATCH. A PR that fails `db:check` is not
|
|
105
|
+
mergeable — the schema is broken before it ever reaches a stage.
|
|
106
|
+
|
|
107
|
+
### 2. Mint a reviewable plan
|
|
108
|
+
|
|
109
|
+
```bash
|
|
110
|
+
everystack db:plan --stage production --out prod.plan.json
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
`db:plan` is **read-only**. It asks the target its live fingerprint, diffs your checkout
|
|
114
|
+
against it, and writes one plan file: the edge plus both endpoint fingerprints. It prints
|
|
115
|
+
the **shape first** — a histogram (`shape: 402 policies, 398 grants, 201 rls, …`) before any
|
|
116
|
+
SQL wall — and itemizes every DESTRUCTIVE statement in red with the rebuild cost. A plan
|
|
117
|
+
with held drops refuses to mint unless you pass `--allow-drops` (destruction is carried
|
|
118
|
+
explicitly). Plans are **ephemeral** — attach them to the PR/release, never commit them.
|
|
119
|
+
|
|
120
|
+
### 3. Read the plan
|
|
121
|
+
|
|
122
|
+
Look at the shape, the destructive itemization, and the rebuild estimate. This is the
|
|
123
|
+
review surface — the point where a human (or you) decides the edge is what was intended.
|
|
124
|
+
|
|
125
|
+
### 4. Apply it credential-free
|
|
126
|
+
|
|
127
|
+
```bash
|
|
128
|
+
everystack db:apply --plan prod.plan.json --stage production
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
With `--stage`, the WRITE runs **inside the ops Lambda over IAM — the operator never holds
|
|
132
|
+
the database URL**. `db:apply` verifies → applies → verifies, and **refuses** unless:
|
|
133
|
+
|
|
134
|
+
- **the live fingerprint still equals `plan.from`** (the concurrency lock — nobody changed
|
|
135
|
+
the target since you minted the plan; if they did, re-mint), AND
|
|
136
|
+
- **your checkout descends from the commit that declares the target's state** (the
|
|
137
|
+
fast-forward rule — "rebase first"; a stale checkout can't silently revert merged work).
|
|
138
|
+
|
|
139
|
+
Every refusal is recorded in `everystack.schema_log`. `--database-url` is local-dev-only;
|
|
140
|
+
protected stages always go through `--stage`.
|
|
141
|
+
|
|
142
|
+
### 5. Destructive plans carry more gates
|
|
143
|
+
|
|
144
|
+
A plan is DESTRUCTIVE if it drops (table/column/type) or narrows a type (a lossy
|
|
145
|
+
`SET DATA TYPE`). Those additionally require, all of them:
|
|
146
|
+
|
|
147
|
+
- **`--confirm`** — always, explicitly.
|
|
148
|
+
- **A snapshot** — taken automatically via `db:backup` when you pass `--stage` (or supply
|
|
149
|
+
one with `--snapshot-ref`). No snapshot, no destructive apply.
|
|
150
|
+
- **The stage's approver set** — if declared, `db:apply` verifies the caller's AWS identity
|
|
151
|
+
(STS) against it. Set it with `db:approvers`:
|
|
152
|
+
```bash
|
|
153
|
+
everystack db:approvers --stage production --set "cto,arn:aws:iam::…:user/cto"
|
|
154
|
+
everystack db:approvers --stage production --set '' # disable destructive applies entirely
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
Prefer never to destroy in place. The safe path for removing a column is the **contract
|
|
158
|
+
phase**: `field.deprecated()` first (the column stays readable, new writes are rejected,
|
|
159
|
+
generated types strike it through), then the physical drop later as its own confirmed,
|
|
160
|
+
snapshotted, approver-gated plan.
|
|
161
|
+
|
|
162
|
+
## Snapshot-first: the safety net
|
|
163
|
+
|
|
164
|
+
Two backup mechanisms, different shapes — know when to reach for each:
|
|
165
|
+
|
|
166
|
+
| | `db:snapshot` | `db:backup` |
|
|
167
|
+
|---|---|---|
|
|
168
|
+
| Kind | **Physical** RDS snapshot (whole instance) | **Logical** `pg_dump` of one database → private S3 |
|
|
169
|
+
| Speed to safety | Instant (control-plane, no VPC) | Minutes (dumps the data) |
|
|
170
|
+
| Restore | RDS restore (new instance) — instant DR | `db:restore` INTO a stage; downloadable, portable |
|
|
171
|
+
| Requires | RDS (not Aurora Serverless v2 the same way) | any Postgres reachable by the ops Lambda |
|
|
172
|
+
|
|
173
|
+
```bash
|
|
174
|
+
everystack db:snapshot --stage production # physical, instant — the "oh no" button
|
|
175
|
+
everystack db:snapshots --stage production # list manual snapshots
|
|
176
|
+
everystack db:backup --stage production # logical pg_dump → S3 backups bucket
|
|
177
|
+
everystack db:backups --stage production # list logical backups
|
|
178
|
+
everystack db:backup:download <id> --stage production # presigned download URL (1h)
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
**RDS snapshot first?** For a destructive apply, `db:apply` already forces a logical backup —
|
|
182
|
+
you cannot destroy without one. But before any large or risky migration on a stage that
|
|
183
|
+
holds real data, take a **physical `db:snapshot` first** as well: it is instant, it captures
|
|
184
|
+
the whole instance, and restoring it is the fastest path back if something the plan didn't
|
|
185
|
+
foresee goes wrong. Cheap insurance; take it.
|
|
186
|
+
|
|
187
|
+
## Importing data into a REMOTE stage
|
|
188
|
+
|
|
189
|
+
- **`db:seed --stage dev`** — the declarative seed, dev-tier only (refuses on production).
|
|
190
|
+
- **`pipeline:run --stage <name>`** — the ingest pipeline, credential-free in the ops
|
|
191
|
+
Lambda. The way to load a real dataset into a deployed stage.
|
|
192
|
+
- **`db:fork --from-stage <src> --stage <tgt> --confirm`** — copy a DEPLOYED stage's
|
|
193
|
+
database (schema + data) into a feature stage: backup → presigned URL (the cross-stage
|
|
194
|
+
hand-off) → restore via the target's ops Lambda. **Production is never a target**, and
|
|
195
|
+
forking FROM a prod-tier stage warns about PII. Use it to get realistic data into a
|
|
196
|
+
throwaway stage.
|
|
197
|
+
- **`db:restore --from <id> --stage <name> --confirm`** — restore a logical backup INTO a
|
|
198
|
+
stage. Destructive (it replaces the target), so `--confirm` is required.
|
|
199
|
+
|
|
200
|
+
## Why `--stage` is the safe boundary
|
|
201
|
+
|
|
202
|
+
Operator verbs that touch a protected database run their WRITE inside that stage's **ops
|
|
203
|
+
Lambda**, invoked over IAM. The operator's own credentials never resolve the database URL —
|
|
204
|
+
your AWS identity IS the authorization, CloudTrail records the access, and a leaked laptop
|
|
205
|
+
can't dump production. `--database-url` exists only for local dev databases you own outright.
|
|
206
|
+
|
|
207
|
+
## Command index
|
|
208
|
+
|
|
209
|
+
```bash
|
|
210
|
+
# Declare (edit files), then:
|
|
211
|
+
everystack db:sync # dev DB ← your checkout (state + compute), one verb
|
|
212
|
+
everystack db:reconcile # compute layer only (views/matviews/functions/triggers)
|
|
213
|
+
everystack db:generate # STATE layer → migration file / --apply / --dry-run
|
|
214
|
+
everystack db:check # CI gate: declared state composes; artifacts match
|
|
215
|
+
everystack db:fingerprint # content-address live schema vs models — MATCH/MISMATCH
|
|
216
|
+
|
|
217
|
+
# Reach a protected stage safely:
|
|
218
|
+
everystack db:plan --stage X --out X.plan.json # mint a reviewable edge (read-only)
|
|
219
|
+
everystack db:apply --plan X.plan.json --stage X # verify → apply → verify (credential-free)
|
|
220
|
+
everystack db:approvers --stage X --set "…" # who may run destructive applies
|
|
221
|
+
|
|
222
|
+
# Back up / restore:
|
|
223
|
+
everystack db:snapshot --stage X # physical RDS snapshot (instant DR)
|
|
224
|
+
everystack db:backup --stage X # logical pg_dump → S3
|
|
225
|
+
everystack db:restore --from <id> --stage X --confirm
|
|
226
|
+
|
|
227
|
+
# Local dev databases:
|
|
228
|
+
everystack db:template:refresh # build the dev template from declared state + seed
|
|
229
|
+
everystack db:branch # per-branch dev DB from the template
|
|
230
|
+
|
|
231
|
+
# Data:
|
|
232
|
+
everystack db:seed --stage dev # declarative seed (dev only)
|
|
233
|
+
everystack db:backfill --apply # one-shot data moves (db/backfills/*.sql)
|
|
234
|
+
everystack pipeline:run --stage X # ingest pipeline (credential-free)
|
|
235
|
+
everystack db:fork --from-stage prod --stage feature --confirm # copy a stage's data
|
|
236
|
+
```
|
|
@@ -0,0 +1,225 @@
|
|
|
1
|
+
# Derived Objects: functions, views, materialized views, triggers, sequences
|
|
2
|
+
|
|
3
|
+
> Derived objects don't migrate — they **deploy**. Base tables hold irreplaceable data
|
|
4
|
+
> and change through generated migrations. Functions, views, and materialized views are
|
|
5
|
+
> **compute**: declared source whose results are rebuildable from the tables at any time.
|
|
6
|
+
> You edit their descriptors, and `db:reconcile` makes the database match — no migration
|
|
7
|
+
> file, no journal entry, no numbered anything. Git is the history.
|
|
8
|
+
|
|
9
|
+
Read this when you need to add or change a view, materialized view, function, trigger, or
|
|
10
|
+
standalone sequence. For tables (`defineModel`) and the operational verbs
|
|
11
|
+
(`db:sync`/`db:plan`/`db:apply`), see `everystack://schema-patterns` and
|
|
12
|
+
`everystack://database-operations`.
|
|
13
|
+
|
|
14
|
+
## Two layers, two lifecycles
|
|
15
|
+
|
|
16
|
+
A schema is two different things, and conflating them is the mistake:
|
|
17
|
+
|
|
18
|
+
| Layer | Objects | Changes via | History |
|
|
19
|
+
|-------|---------|-------------|---------|
|
|
20
|
+
| **State** | tables, constraints, authz, sequences | Models → `db:generate` → migration | migration + `db:pull` |
|
|
21
|
+
| **Compute** | functions, views, matviews, triggers | descriptors → `db:reconcile` | git |
|
|
22
|
+
|
|
23
|
+
Versioning compute through migrations is a category error — the same as writing a
|
|
24
|
+
migration to edit a Lambda handler. A one-line fix to a materialized view should be a
|
|
25
|
+
one-line git diff, not a restatement of the whole view inside a numbered file. (In one
|
|
26
|
+
real analytics app, 34 of 43 migrations were pure derived-layer churn, including a 701 KB
|
|
27
|
+
migration that fixed a one-line regex.) Descriptors end that.
|
|
28
|
+
|
|
29
|
+
## The declared home
|
|
30
|
+
|
|
31
|
+
The whole database — both layers — lives in `db/models/`, in one format:
|
|
32
|
+
|
|
33
|
+
```
|
|
34
|
+
db/
|
|
35
|
+
├── models/ # the declared database: defineModel tables + derived descriptors
|
|
36
|
+
│ ├── post.ts
|
|
37
|
+
│ ├── derived.ts # defineView / defineMaterializedView / defineFunction / defineSql
|
|
38
|
+
│ └── index.ts # defineModule({ models, sequences, derived })
|
|
39
|
+
└── schema.generated.ts # generated artifact (never edited; db:check refuses drift)
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
**`db/sql/` is retired.** Raw-SQL derived objects moved into descriptors, and every CLI
|
|
43
|
+
verb hard-fails on a `db/sql/` directory that still carries `.sql` files. If you are
|
|
44
|
+
adopting an app that still has one, see "Adopting an existing database" below.
|
|
45
|
+
|
|
46
|
+
The boundary rule, uniform with models: **structure is declared** (identity,
|
|
47
|
+
dependencies, authz, security posture, indexes, signatures, trigger events);
|
|
48
|
+
**expressions stay `sql``** (the SELECT, the function body, a WHEN predicate). No query
|
|
49
|
+
DSL, no string mini-grammars in declared properties.
|
|
50
|
+
|
|
51
|
+
```ts
|
|
52
|
+
import { defineView, defineMaterializedView, defineFunction, defineSql, defineModule,
|
|
53
|
+
trigger, arg, setof, field, can, index, sql } from '@everystack/model';
|
|
54
|
+
|
|
55
|
+
// A VIEW — securityInvoker is the one forced decision.
|
|
56
|
+
export const activeUsers = defineView('active_users', {
|
|
57
|
+
securityInvoker: true, // REQUIRED, no default:
|
|
58
|
+
// true = readers' own RLS/grants apply through the view
|
|
59
|
+
// false = definer semantics (the sealed-slice pattern —
|
|
60
|
+
// a view grant WITHOUT a table grant; the body must seal)
|
|
61
|
+
abilities: [can('read')], // grants are AUTHORED, never inherited
|
|
62
|
+
dependsOn: [users], // declared, then VERIFIED against the live edges (drift-checked)
|
|
63
|
+
fields: { id: field.uuid().notNull(), name: field.text().notNull() }, // OPTIONAL: typed SSR
|
|
64
|
+
// reads — emits a pgView().existing() into schema.generated.ts;
|
|
65
|
+
// never touches the DDL or the content hash
|
|
66
|
+
as: sql`SELECT id, name FROM users WHERE deleted_at IS NULL`,
|
|
67
|
+
});
|
|
68
|
+
|
|
69
|
+
// A MATERIALIZED VIEW — grants-only authz; refresh/populate declare the HOW.
|
|
70
|
+
export const leaderboard = defineMaterializedView('leaderboard', {
|
|
71
|
+
abilities: [can('read')],
|
|
72
|
+
dependsOn: [scores, activeUsers],
|
|
73
|
+
indexes: [index(['user_id']).unique(), index('tags').using('gin')],
|
|
74
|
+
refresh: 'concurrently', // compile-gated: REQUIRES a unique, non-partial index
|
|
75
|
+
populate: 'deferred', // create WITH NO DATA, refresh outside the creation lock
|
|
76
|
+
as: sql`SELECT user_id, sum(points) AS points FROM scores GROUP BY user_id`,
|
|
77
|
+
});
|
|
78
|
+
|
|
79
|
+
// A FUNCTION — signature structured, body raw.
|
|
80
|
+
export const searchPosts = defineFunction('search_posts', {
|
|
81
|
+
args: [arg('query', 'text')],
|
|
82
|
+
returns: setof(posts), // a descriptor ref — a typo is a TS error, the dep comes free
|
|
83
|
+
language: 'sql',
|
|
84
|
+
volatility: 'stable', // default 'volatile'; 'immutable' enables expression indexes
|
|
85
|
+
abilities: [can('execute', { role: 'authenticated' })], // role REQUIRED; PUBLIC is always revoked
|
|
86
|
+
body: sql`SELECT * FROM posts WHERE ts @@ websearch_to_tsquery(query)`,
|
|
87
|
+
});
|
|
88
|
+
|
|
89
|
+
// A SECURITY DEFINER function MUST pin its search_path (the classic landmine):
|
|
90
|
+
export const promote = defineFunction('promote_user', {
|
|
91
|
+
args: [arg('uid', 'uuid')], returns: 'void', language: 'plpgsql',
|
|
92
|
+
security: 'definer', searchPath: ['pg_catalog', 'public'], // required with definer
|
|
93
|
+
abilities: [can('execute', { role: 'admin' })],
|
|
94
|
+
body: sql`BEGIN UPDATE users SET role = 'admin' WHERE id = uid; END`,
|
|
95
|
+
});
|
|
96
|
+
|
|
97
|
+
// A TRIGGER FUNCTION is a standalone function returning 'trigger'…
|
|
98
|
+
export const touchUpdatedAt = defineFunction('touch_updated_at', {
|
|
99
|
+
returns: 'trigger', language: 'plpgsql',
|
|
100
|
+
body: sql`BEGIN NEW.updated_at = now(); RETURN NEW; END`,
|
|
101
|
+
});
|
|
102
|
+
|
|
103
|
+
// …and the TRIGGER is declared ON THE MODEL it fires on (owner consent, visible where the
|
|
104
|
+
// table lives) — see defineModel below.
|
|
105
|
+
|
|
106
|
+
export const appModule = defineModule({
|
|
107
|
+
models: [posts, users],
|
|
108
|
+
derived: [activeUsers, leaderboard, searchPosts, promote, touchUpdatedAt],
|
|
109
|
+
// sequences: [invoiceNo], // standalone sequences are STATE — see below
|
|
110
|
+
});
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
Triggers ride the model:
|
|
114
|
+
|
|
115
|
+
```ts
|
|
116
|
+
export const Post = defineModel('posts', {
|
|
117
|
+
fields: { id: field.uuid().primaryKey(), updatedAt: field.timestamptz() },
|
|
118
|
+
abilities: [can('read')],
|
|
119
|
+
triggers: [trigger('touch', {
|
|
120
|
+
timing: 'before', events: ['update'], forEach: 'row',
|
|
121
|
+
execute: touchUpdatedAt, // the trigger function MUST be in the module's derived set
|
|
122
|
+
})],
|
|
123
|
+
});
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
## The rules that make it safe (all enforced)
|
|
127
|
+
|
|
128
|
+
- **`securityInvoker` is a required decision** on every view — no default. Pick `true`
|
|
129
|
+
(readers' own RLS applies) or `false` (definer semantics, the sealed slice).
|
|
130
|
+
- **Grants are authored, never inherited.** `can('read')` on a relation → `anon` +
|
|
131
|
+
`authenticated`; `can('read', { role })` narrows. A granted view whose readers can't
|
|
132
|
+
reach its underlying tables is a **compile error** — the reachability gate.
|
|
133
|
+
- **SECURITY DEFINER without a pinned `searchPath` is rejected** at define time.
|
|
134
|
+
- **`can('execute')` needs a role** — bare execute has no safe audience; PUBLIC is always
|
|
135
|
+
revoked by the compiler.
|
|
136
|
+
- **A missing import is caught loudly.** An `undefined` in `dependsOn`/`derived`/a
|
|
137
|
+
trigger's `execute` (a renamed or missing named import) is rejected at define time,
|
|
138
|
+
naming the descriptor and the index — never a silent drop.
|
|
139
|
+
- **`dependsOn` is declared, then verified.** The reachability gate walks it at compile
|
|
140
|
+
time; the live catalog's actual edges check it at plan time. A view that reads a table
|
|
141
|
+
it never declared (or declares one it never reads) is a `db:check` failure.
|
|
142
|
+
- **REFRESH CONCURRENTLY requires a unique, non-partial index** — compile-gated.
|
|
143
|
+
- **Content hashing is comment- and whitespace-insensitive** — reformatting a body is a
|
|
144
|
+
no-op; only a real change reconciles.
|
|
145
|
+
|
|
146
|
+
The escape hatch `defineSql` handles what the structured vocabulary can't say yet
|
|
147
|
+
(aggregates, operators, event triggers) — still declared, hashed, dependency-ordered. It
|
|
148
|
+
refuses a `CREATE [MATERIALIZED] VIEW` so accidental relations land in the read model
|
|
149
|
+
where the gates can see them; its SQL is otherwise trusted as authored.
|
|
150
|
+
|
|
151
|
+
## Sequences are STATE, not compute
|
|
152
|
+
|
|
153
|
+
A standalone sequence (one not owned by a serial column) holds a counter value, so it
|
|
154
|
+
migrates with the base schema — declare it with `defineSequence` and pass it on
|
|
155
|
+
`defineModule({ sequences })`. It is fingerprinted with the tables and created before them.
|
|
156
|
+
This is the one derived-looking object that is state, not compute.
|
|
157
|
+
|
|
158
|
+
```ts
|
|
159
|
+
export const invoiceNo = defineSequence('invoice_no', { start: 1000 });
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
## Deploying: `db:reconcile`
|
|
163
|
+
|
|
164
|
+
```bash
|
|
165
|
+
everystack db:reconcile # plan (read-only; works via --stage or --database-url)
|
|
166
|
+
everystack db:reconcile --check # CI gate: exit 1 when anything differs
|
|
167
|
+
everystack db:reconcile --apply --database-url … # execute (direct connection required)
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
Planning works over the read-only ops Lambda (`--stage`); **`--apply` needs a direct
|
|
171
|
+
connection** — the deployed query path is read-only by design. On a dev database, prefer
|
|
172
|
+
`db:sync` (it does state + compute in one verb); reach for `db:reconcile --apply` when you
|
|
173
|
+
want compute only.
|
|
174
|
+
|
|
175
|
+
A plan shows the rebuild cost before you say yes — dependent cascades, matview sizes, row
|
|
176
|
+
counts:
|
|
177
|
+
|
|
178
|
+
```
|
|
179
|
+
= 51 up to date
|
|
180
|
+
~ replace public.score (source changed)
|
|
181
|
+
↻ refresh public.season_scores (depends on replaced function public.score; 1.2 GB, ~4M rows)
|
|
182
|
+
+ create public.weekly_top (new in source)
|
|
183
|
+
≠ regrant public.leaderboard — live grants differ (REVOKE SELECT ON leaderboard FROM intern)
|
|
184
|
+
estimated rebuild: 1.2 GB across 1 matview(s)
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
`--apply` sends all DDL as **one transaction** (a failed create rolls back the drops
|
|
188
|
+
before it), records provenance and a `schema_log` memoir (the full DDL, when, by whom,
|
|
189
|
+
from which git commit), and rebuilds dependents automatically in dependency order.
|
|
190
|
+
|
|
191
|
+
## Semantics worth knowing
|
|
192
|
+
|
|
193
|
+
- **Drift is never overwritten silently.** A hand-edited managed object (someone ran
|
|
194
|
+
`CREATE OR REPLACE` against the DB) makes the plan report drift and `--apply` refuse.
|
|
195
|
+
Fix the source to match, or rebuild from source with `--overwrite-drift`.
|
|
196
|
+
- **Grant drift converges without ceremony** — a hand-run GRANT/REVOKE surfaces as an
|
|
197
|
+
idempotent `regrant` back to the declared abilities. Declared authz is authoritative.
|
|
198
|
+
- **Changing a function REFRESHes the matviews that call it** — their rows are stale the
|
|
199
|
+
moment the function changes, and the reconciler knows it.
|
|
200
|
+
- **Unmanaged objects are never touched** — live derived objects with no source and no
|
|
201
|
+
provenance are listed and left alone.
|
|
202
|
+
|
|
203
|
+
## Adopting an existing database (baseline / rebaseline)
|
|
204
|
+
|
|
205
|
+
- **First contact wants `--baseline`.** Against a database that already has your derived
|
|
206
|
+
layer (built by old migrations), the reconciler refuses to guess whether live matches
|
|
207
|
+
source. `everystack db:pull` renders the live layer as descriptors (bodies from the
|
|
208
|
+
catalog's canonical deparse, grants as abilities, `dependsOn` from live edges). The path:
|
|
209
|
+
**pull → review → commit → `db:reconcile --apply --baseline` → a silent check.**
|
|
210
|
+
Use `db:pull --derived-out db/models/derived.ts` to write the derived layer straight
|
|
211
|
+
into its own file when you already have a hand-maintained models barrel.
|
|
212
|
+
- **A `db/sql`-era migration wants `--rebaseline`.** When provenance exists but the source
|
|
213
|
+
was re-rendered (descriptors render canonical SQL, so the source hash differs even for
|
|
214
|
+
identical SQL), `--rebaseline` re-records the new source hash for every object whose live
|
|
215
|
+
definition still matches what the reconciler last applied — no drop, no create, no
|
|
216
|
+
matview repopulation. The live side is VERIFIED; the source side is TRUSTED. Migrate
|
|
217
|
+
first (verbatim), edit after. `--baseline` and `--rebaseline` compose in one run.
|
|
218
|
+
|
|
219
|
+
## Current limitations
|
|
220
|
+
|
|
221
|
+
- **Function identity is by name** — overloads are rejected at define time. Rename one.
|
|
222
|
+
- **Function signature changes** (return/arg types) can't `CREATE OR REPLACE` — drop the
|
|
223
|
+
old function in a reviewed step, or rename.
|
|
224
|
+
- Postgres can't see function→function dependencies inside bodies — `dependsOn` is your
|
|
225
|
+
declaration there.
|