@rivi-software/d1-migrations 0.0.0-stage → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +228 -2
- package/dist/bundle.d.ts +36 -0
- package/dist/bundle.js +125 -0
- package/dist/cli.d.ts +2 -0
- package/dist/cli.js +4 -0
- package/dist/files.d.ts +44 -0
- package/dist/files.js +79 -0
- package/dist/index.d.ts +60 -0
- package/dist/index.js +77 -0
- package/dist/journal.d.ts +54 -0
- package/dist/journal.js +95 -0
- package/dist/run-cli.d.ts +10 -0
- package/dist/run-cli.js +101 -0
- package/package.json +68 -3
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 RIVI Software
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,229 @@
|
|
|
1
|
-
#
|
|
1
|
+
# @rivi-software/d1-migrations
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
[](https://www.npmjs.com/package/@rivi-software/d1-migrations) [](https://github.com/rivi-software/tools/actions/workflows/ci.yml) [](LICENSE)
|
|
4
|
+
|
|
5
|
+
D1 migrations from inside a Worker, for the databases an app creates at runtime, such as one database per customer.
|
|
6
|
+
|
|
7
|
+
`wrangler d1 migrations apply` runs from the command line, and only on the databases the Wrangler config names. This package applies the same migrations from a Worker, to any database the Worker holds:
|
|
8
|
+
- the same files, in the same order
|
|
9
|
+
- the same statements
|
|
10
|
+
- the same migrations table
|
|
11
|
+
|
|
12
|
+
Wrangler and the Worker can then each apply the migrations the other has not.
|
|
13
|
+
|
|
14
|
+
It has three parts:
|
|
15
|
+
- `bundleD1Migrations()` (Node, at build time) reads the migration files and writes a module the Worker imports.
|
|
16
|
+
- `migrateD1(db, migrations)` (in a Worker) applies the migrations a database has not run yet.
|
|
17
|
+
- `d1-migrations journal` (command line, in CI) keeps a SHA-256 of each migration, and fails when a migration that was recorded is edited or deleted. Wrangler records a migration by its name only, so an edit after a migration ran goes unnoticed: databases migrated before and after the edit differ.
|
|
18
|
+
|
|
19
|
+
## Install
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
npm install @rivi-software/d1-migrations
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
The bundle step splits each migration with your project's own Wrangler, so it needs `wrangler` 4 installed. Wrangler is an optional peer dependency: the Worker never imports it. The tests run against Wrangler 4.156.2.
|
|
26
|
+
|
|
27
|
+
The bundle step and the `d1-migrations` command need Node 22 or later. `migrateD1` uses only the D1 binding, so it runs in any Worker. The package is ESM only.
|
|
28
|
+
|
|
29
|
+
## Usage
|
|
30
|
+
|
|
31
|
+
### At build time
|
|
32
|
+
|
|
33
|
+
Bundle the migrations before the Worker builds. Either run the command from a `package.json` script:
|
|
34
|
+
|
|
35
|
+
```json
|
|
36
|
+
{
|
|
37
|
+
"scripts": {
|
|
38
|
+
"prebuild": "d1-migrations bundle --out src/generated/migrations.ts"
|
|
39
|
+
}
|
|
40
|
+
}
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
or call the function from a build script:
|
|
44
|
+
|
|
45
|
+
```ts
|
|
46
|
+
import { bundleD1Migrations } from "@rivi-software/d1-migrations/bundle";
|
|
47
|
+
|
|
48
|
+
await bundleD1Migrations({ dir: "migrations", out: "src/generated/migrations.ts" });
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
Pass the same settings the binding has in the Wrangler config:
|
|
52
|
+
|
|
53
|
+
| Option | Command line | Wrangler config field | Default |
|
|
54
|
+
| --- | --- | --- | --- |
|
|
55
|
+
| `dir` | `--dir` | `migrations_dir` | `migrations` |
|
|
56
|
+
| `pattern` | `--pattern` | `migrations_pattern` | `<dir>/*.sql` |
|
|
57
|
+
| `table` | `--table` | `migrations_table` | `d1_migrations` |
|
|
58
|
+
| `cwd` | (the current directory) | | `process.cwd()` |
|
|
59
|
+
|
|
60
|
+
Paths are relative to `cwd`, which should be the directory that holds the Wrangler config. For Drizzle's layout, pass `--dir drizzle --pattern "drizzle/*/migration.sql"`.
|
|
61
|
+
|
|
62
|
+
The module holds each migration's statements and the table's name. It only changes when a migration does, and the bundle step leaves an unchanged module unwritten, so no watcher wakes up. Commit it, or have Git ignore it and bundle before every build and test run.
|
|
63
|
+
|
|
64
|
+
### In the Worker
|
|
65
|
+
|
|
66
|
+
Pass `migrateD1` the binding of a database the Worker holds. For example, the binding of a Workers for Platforms script that was uploaded with its customer's database:
|
|
67
|
+
|
|
68
|
+
```ts
|
|
69
|
+
import { migrateD1 } from "@rivi-software/d1-migrations";
|
|
70
|
+
import migrations from "./generated/migrations";
|
|
71
|
+
|
|
72
|
+
let migrated: Promise<unknown> | undefined;
|
|
73
|
+
|
|
74
|
+
export default {
|
|
75
|
+
async fetch(request, env) {
|
|
76
|
+
// Once per isolate. A failed migration is tried again on the next request.
|
|
77
|
+
migrated ??= migrateD1(env.DB, migrations).catch((error) => {
|
|
78
|
+
migrated = undefined;
|
|
79
|
+
throw error;
|
|
80
|
+
});
|
|
81
|
+
await migrated;
|
|
82
|
+
// …
|
|
83
|
+
},
|
|
84
|
+
} satisfies ExportedHandler<Env>;
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
`migrateD1` resolves to `{ applied }`, the names of the migrations it applied. On a database that is up to date, it costs two queries and applies nothing. A migration that fails throws a `D1MigrationError` with these fields:
|
|
88
|
+
- `migration`: the migration that failed
|
|
89
|
+
- `applied`: the migrations this call applied before it
|
|
90
|
+
- `cause`: D1's error
|
|
91
|
+
|
|
92
|
+
The database keeps the migrations before it, and nothing of the one that failed.
|
|
93
|
+
|
|
94
|
+
### In CI
|
|
95
|
+
|
|
96
|
+
Record each new migration in the journal, and commit the journal with it:
|
|
97
|
+
|
|
98
|
+
```bash
|
|
99
|
+
npx d1-migrations journal
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
In CI, check the migrations against it:
|
|
103
|
+
|
|
104
|
+
```bash
|
|
105
|
+
npx d1-migrations journal --check
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
`--check` writes nothing, and fails in three cases:
|
|
109
|
+
- **Edited.** A recorded migration's text changed.
|
|
110
|
+
- **Deleted.** A recorded migration's file is gone.
|
|
111
|
+
- **Unrecorded.** A migration is missing from the journal.
|
|
112
|
+
|
|
113
|
+
`journal` without `--check` adds new migrations, and never changes or drops an entry. A migration that may have run on any database must stay as it is, so a change belongs in a new migration. To change a migration that has run on no database yet, delete its entry from the journal, then run `journal` again.
|
|
114
|
+
|
|
115
|
+
The journal is `JOURNAL.json` in the migrations directory. `--journal <file>` picks another file, and `--dir` and `--pattern` work as they do for `bundle`.
|
|
116
|
+
|
|
117
|
+
### In tests with `@cloudflare/vitest-pool-workers`
|
|
118
|
+
|
|
119
|
+
The module also exports `migrations` and `table` by name, which is the shape `applyD1Migrations()` from `cloudflare:test` takes:
|
|
120
|
+
|
|
121
|
+
```ts
|
|
122
|
+
import { applyD1Migrations, env } from "cloudflare:test";
|
|
123
|
+
import { migrations, table } from "../src/generated/migrations";
|
|
124
|
+
|
|
125
|
+
await applyD1Migrations(env.DB, migrations, table);
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
## How it works
|
|
129
|
+
|
|
130
|
+
- **Which files, in which order.** It finds the files the way Wrangler does: the files under `dir` that `pattern` matches, without dotfiles. It sorts them the way Wrangler does, one path segment at a time:
|
|
131
|
+
- a segment's leading number (`0010` in `0010_users.sql`) comes first
|
|
132
|
+
- a tie, or two segments without a number, sorts by text
|
|
133
|
+
- a numbered segment sorts before one without a number
|
|
134
|
+
|
|
135
|
+
A migration's name is its path under `dir`, such as `0001_init.sql` or `0000_init/migration.sql`. That is the name Wrangler records.
|
|
136
|
+
- **Splitting.** `wrangler d1 migrations apply --local` adds the row that records the migration to the end of the migration's text, as `INSERT INTO <table> (name) values ('<name>');` on a new line. It then splits the whole text with its splitter, and runs the statements in one batch.
|
|
137
|
+
|
|
138
|
+
The bundle step splits that same text with that same function (`unstable_splitSqlQuery`), then takes the INSERT off again. Comments, strings, triggers with `BEGIN … END` and `CASE … END`, and a leading `BEGIN TRANSACTION`/`COMMIT` pair therefore come out exactly as Wrangler splits them.
|
|
139
|
+
|
|
140
|
+
When a migration's last statement is not complete, the bundle step fails instead. That is a missing semicolon, or a comment, string or trigger body left open. Wrangler would run that statement together with its INSERT.
|
|
141
|
+
- **Applying.** `migrateD1` creates the migrations table with Wrangler's statement, byte for byte. SQLite keeps that text in its schema, so the two schemas match exactly. It then reads which migrations the table records.
|
|
142
|
+
|
|
143
|
+
Each migration the table lacks goes in one `batch()` with the INSERT that records it. D1 runs a batch as a transaction, so a failed migration leaves nothing behind.
|
|
144
|
+
|
|
145
|
+
Calls can overlap, such as the first requests to a new database. The table's `name` is unique, so when another call applies a migration first, this call's batch rolls back. The call then finds the migration recorded, and goes on to the next one. A migration the table records but the bundle lacks, such as one a newer deploy applied, is left alone.
|
|
146
|
+
- **The journal.** It hashes each migration's text with CRLF line endings read as LF, so a Windows checkout with Git's `autocrlf` matches. The hashes stay in the repository. The migrations table stays exactly as Wrangler makes it, with no hash column.
|
|
147
|
+
|
|
148
|
+
The tests prove the two paths agree:
|
|
149
|
+
- They apply the same migrations to fresh local databases, once with `wrangler d1 migrations apply --local` and once with `bundleD1Migrations` and `migrateD1`.
|
|
150
|
+
- They compare the schemas (each object's `CREATE` text), every table's rows, and the migrations table's names in the order they were applied.
|
|
151
|
+
- They do that for three sets of migrations:
|
|
152
|
+
- a set with comments, strings, a trigger, a view and seed rows
|
|
153
|
+
- Drizzle's layout with a custom table
|
|
154
|
+
- a set of file names that tests the order
|
|
155
|
+
- Wrangler and `migrateD1` then each apply what the other has not.
|
|
156
|
+
|
|
157
|
+
## Limits
|
|
158
|
+
|
|
159
|
+
- **Wrangler's splitter is an unstable export.** The bundle step needs a Wrangler 4 that still exports `unstable_splitSqlQuery`. Cloudflare's own `@cloudflare/vitest-pool-workers` imports it too. A Wrangler without it makes the bundle step fail with an error that says so. Workers already built are unaffected. When a Wrangler upgrade changes how it splits the cases in this package's tests, those tests fail.
|
|
160
|
+
- **Compared with the local path only.** `wrangler d1 migrations apply --remote` sends the migration's text to D1's API, which splits it on Cloudflare's side. The tests run locally, so they compare `migrateD1` with `--local`.
|
|
161
|
+
- **A database the Worker holds.** `migrateD1` takes a D1 binding, or anything with its `prepare` and `batch`, such as a session. A database the Worker has no binding for needs the REST API, which this version does not cover.
|
|
162
|
+
- **One batch per migration.** D1's limits apply to each migration as they do under Wrangler. A statement can be at most 100 KB, and a statement that rewrites very many rows can exceed D1's execution limits. Split a large data change over several migrations.
|
|
163
|
+
- **Forward only.** No down migrations, as with Wrangler.
|
|
164
|
+
- **The journal guards the repository.** It catches an edited migration before it merges, not on a database where it already ran.
|
|
165
|
+
- **The Worker carries the migrations.** Every migration's SQL ends up in the Worker's bundle, and counts toward its size.
|
|
166
|
+
|
|
167
|
+
## Exports
|
|
168
|
+
|
|
169
|
+
From `@rivi-software/d1-migrations` (Workers):
|
|
170
|
+
|
|
171
|
+
| Export | What |
|
|
172
|
+
| --- | --- |
|
|
173
|
+
| `migrateD1(db, migrations)` | Applies the migrations `db` lacks: `{ applied }` |
|
|
174
|
+
| `D1MigrationError` | A failed migration: `migration`, `applied`, `cause` |
|
|
175
|
+
| `DEFAULT_MIGRATIONS_TABLE` | `d1_migrations` |
|
|
176
|
+
| Types | `D1Migration` (`{ name, queries }`), `D1Migrations` (`{ table?, migrations }`), `MigrateD1Result`, `D1DatabaseLike`, `D1StatementLike` |
|
|
177
|
+
|
|
178
|
+
From `@rivi-software/d1-migrations/bundle` (Node):
|
|
179
|
+
|
|
180
|
+
| Export | What |
|
|
181
|
+
| --- | --- |
|
|
182
|
+
| `bundleD1Migrations(options)` | Writes the module: `{ out, table, migrations, changed }`. Options `out`, `dir`, `pattern`, `table`, `cwd` |
|
|
183
|
+
| `readD1Migrations(options?)` | The migrations, split, without writing anything |
|
|
184
|
+
| `listMigrationFiles(source?)` | The migration files in Wrangler's order: `{ dir, files: [{ name, path }] }` |
|
|
185
|
+
| `compareMigrationPaths(a, b)` | Wrangler's order for two migration names |
|
|
186
|
+
| Types | `BundleD1MigrationsOptions`, `BundleD1MigrationsResult`, `ReadD1MigrationsOptions`, `MigrationsSource`, `MigrationFile`, `MigrationFiles` |
|
|
187
|
+
|
|
188
|
+
From `@rivi-software/d1-migrations/journal` (Node):
|
|
189
|
+
|
|
190
|
+
| Export | What |
|
|
191
|
+
| --- | --- |
|
|
192
|
+
| `updateJournal(options?)` | Records new migrations: `{ journal, added, problems }`. It writes nothing when `problems` holds an edited or deleted migration |
|
|
193
|
+
| `checkJournal(options?)` | `{ journal, problems }`, each problem `{ name, kind }`, where `kind` is `edited`, `deleted` or `unrecorded` |
|
|
194
|
+
| `hashMigration(sql)` | The hash the journal records |
|
|
195
|
+
| `JOURNAL_FILE` | `JOURNAL.json` |
|
|
196
|
+
| Types | `JournalOptions` (`dir`, `pattern`, `cwd`, `journal`), `Journal`, `JournalEntry`, `JournalProblem`, `JournalCheck`, `JournalUpdate`, `MigrationsSource` |
|
|
197
|
+
|
|
198
|
+
The `d1-migrations` command runs `bundle` and `journal [--check]`. `d1-migrations` with no command prints its options. It exits with `1` when the bundle step fails or the journal disagrees, and with `2` on a usage error.
|
|
199
|
+
|
|
200
|
+
## Development
|
|
201
|
+
|
|
202
|
+
The tests in `tests/` cover:
|
|
203
|
+
- the order and the file matching
|
|
204
|
+
- the splitting, pinned against the installed Wrangler
|
|
205
|
+
- the generated module
|
|
206
|
+
- `migrateD1` on Miniflare's D1: a failed migration, overlapping calls, a custom table
|
|
207
|
+
- the journal and the command
|
|
208
|
+
- the agreement with `wrangler d1 migrations apply --local`
|
|
209
|
+
|
|
210
|
+
They run Wrangler and Miniflare locally, and call no Cloudflare API. In this directory, run `pnpm run test`, `pnpm run check` and `pnpm run build`.
|
|
211
|
+
|
|
212
|
+
## License
|
|
213
|
+
|
|
214
|
+
Licensed under the [MIT License](LICENSE). Copyright © 2026 RIVI Software.
|
|
215
|
+
|
|
216
|
+
<br>
|
|
217
|
+
|
|
218
|
+
<p align="center">
|
|
219
|
+
<a href="https://rivi.software">
|
|
220
|
+
<picture>
|
|
221
|
+
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rivi-software/tools/main/assets/rivi-mark-on-dark.svg">
|
|
222
|
+
<img src="https://raw.githubusercontent.com/rivi-software/tools/main/assets/rivi-mark.svg" alt="RIVI Software" width="36" height="36">
|
|
223
|
+
</picture>
|
|
224
|
+
</a>
|
|
225
|
+
</p>
|
|
226
|
+
|
|
227
|
+
<p align="center">
|
|
228
|
+
<sub>Part of <a href="https://github.com/rivi-software/tools#readme">RIVI tools</a>, the packages <a href="https://rivi.software">RIVI Software</a> builds its own apps with.</sub>
|
|
229
|
+
</p>
|
package/dist/bundle.d.ts
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
import { type MigrationsSource } from "./files.js";
|
|
2
|
+
import { type D1Migration } from "./index.js";
|
|
3
|
+
export { compareMigrationPaths, listMigrationFiles } from "./files.js";
|
|
4
|
+
export type { MigrationFile, MigrationFiles, MigrationsSource } from "./files.js";
|
|
5
|
+
export type ReadD1MigrationsOptions = MigrationsSource & {
|
|
6
|
+
/** The binding's `migrations_table`. Default `d1_migrations`. */
|
|
7
|
+
table?: string;
|
|
8
|
+
};
|
|
9
|
+
export type BundleD1MigrationsOptions = ReadD1MigrationsOptions & {
|
|
10
|
+
/**
|
|
11
|
+
* The module to write, relative to `cwd`. A `.ts` file gets type annotations; any other gets
|
|
12
|
+
* plain JavaScript with a JSDoc type.
|
|
13
|
+
*/
|
|
14
|
+
out: string;
|
|
15
|
+
};
|
|
16
|
+
export type BundleD1MigrationsResult = {
|
|
17
|
+
/** The module's absolute path. */
|
|
18
|
+
out: string;
|
|
19
|
+
table: string;
|
|
20
|
+
migrations: D1Migration[];
|
|
21
|
+
/** Whether the module changed. An unchanged module is not written, so no watcher wakes. */
|
|
22
|
+
changed: boolean;
|
|
23
|
+
};
|
|
24
|
+
/**
|
|
25
|
+
* The migrations of `options`, in the order `wrangler d1 migrations apply` applies them, each split
|
|
26
|
+
* into the statements that command runs. Throws when a migration's last statement is not complete
|
|
27
|
+
* (a missing semicolon, or an unclosed comment, string or trigger body): Wrangler would run it
|
|
28
|
+
* together with the statement that records the migration.
|
|
29
|
+
*/
|
|
30
|
+
export declare function readD1Migrations(options?: ReadD1MigrationsOptions): Promise<D1Migration[]>;
|
|
31
|
+
/**
|
|
32
|
+
* Reads the migrations ({@link readD1Migrations}) and writes them to `out` as a module whose default
|
|
33
|
+
* export goes straight to `migrateD1()`: `{ table, migrations }`. It also exports `table` and
|
|
34
|
+
* `migrations` by name, the shape `applyD1Migrations()` from `cloudflare:test` takes.
|
|
35
|
+
*/
|
|
36
|
+
export declare function bundleD1Migrations(options: BundleD1MigrationsOptions): Promise<BundleD1MigrationsResult>;
|
package/dist/bundle.js
ADDED
|
@@ -0,0 +1,125 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* The build-time half: reads the migration files in the order Wrangler applies them, splits each
|
|
3
|
+
* into statements with Wrangler's own splitter, and writes a module the Worker imports and passes
|
|
4
|
+
* to `migrateD1()`. Runs in Node, with `wrangler` installed in the project.
|
|
5
|
+
*/
|
|
6
|
+
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
|
7
|
+
import path from "node:path";
|
|
8
|
+
import { listMigrationFiles } from "./files.js";
|
|
9
|
+
import { DEFAULT_MIGRATIONS_TABLE } from "./index.js";
|
|
10
|
+
export { compareMigrationPaths, listMigrationFiles } from "./files.js";
|
|
11
|
+
/**
|
|
12
|
+
* The migrations of `options`, in the order `wrangler d1 migrations apply` applies them, each split
|
|
13
|
+
* into the statements that command runs. Throws when a migration's last statement is not complete
|
|
14
|
+
* (a missing semicolon, or an unclosed comment, string or trigger body): Wrangler would run it
|
|
15
|
+
* together with the statement that records the migration.
|
|
16
|
+
*/
|
|
17
|
+
export async function readD1Migrations(options = {}) {
|
|
18
|
+
return readFiles(listMigrationFiles(options).files, options.table ?? DEFAULT_MIGRATIONS_TABLE);
|
|
19
|
+
}
|
|
20
|
+
/**
|
|
21
|
+
* Reads the migrations ({@link readD1Migrations}) and writes them to `out` as a module whose default
|
|
22
|
+
* export goes straight to `migrateD1()`: `{ table, migrations }`. It also exports `table` and
|
|
23
|
+
* `migrations` by name, the shape `applyD1Migrations()` from `cloudflare:test` takes.
|
|
24
|
+
*/
|
|
25
|
+
export async function bundleD1Migrations(options) {
|
|
26
|
+
const cwd = path.resolve(options.cwd ?? ".");
|
|
27
|
+
const out = path.resolve(cwd, options.out);
|
|
28
|
+
const table = options.table ?? DEFAULT_MIGRATIONS_TABLE;
|
|
29
|
+
const { dir, files } = listMigrationFiles({ ...options, cwd });
|
|
30
|
+
const migrations = await readFiles(files, table);
|
|
31
|
+
const source = renderModule({
|
|
32
|
+
table,
|
|
33
|
+
migrations,
|
|
34
|
+
from: `${(path.relative(cwd, dir) || ".").replace(/\\/g, "/")}/`,
|
|
35
|
+
typescript: /\.[cm]?tsx?$/.test(out),
|
|
36
|
+
});
|
|
37
|
+
const previous = await readFile(out, "utf8").catch(() => undefined);
|
|
38
|
+
if (previous !== source) {
|
|
39
|
+
await mkdir(path.dirname(out), { recursive: true });
|
|
40
|
+
await writeFile(out, source);
|
|
41
|
+
}
|
|
42
|
+
return { out, table, migrations, changed: previous !== source };
|
|
43
|
+
}
|
|
44
|
+
async function readFiles(files, table) {
|
|
45
|
+
const split = await wranglerSplitter();
|
|
46
|
+
const migrations = [];
|
|
47
|
+
for (const file of files)
|
|
48
|
+
migrations.push({
|
|
49
|
+
name: file.name,
|
|
50
|
+
queries: splitMigration(split, file.name, await readFile(file.path, "utf8"), table),
|
|
51
|
+
});
|
|
52
|
+
return migrations;
|
|
53
|
+
}
|
|
54
|
+
async function wranglerSplitter() {
|
|
55
|
+
let wrangler;
|
|
56
|
+
try {
|
|
57
|
+
wrangler = await import("wrangler");
|
|
58
|
+
}
|
|
59
|
+
catch (error) {
|
|
60
|
+
throw new Error("@rivi-software/d1-migrations splits migrations with Wrangler's splitter: install wrangler 4 in this project.", { cause: error });
|
|
61
|
+
}
|
|
62
|
+
if (typeof wrangler.unstable_splitSqlQuery !== "function")
|
|
63
|
+
throw new Error("This version of Wrangler does not export unstable_splitSqlQuery, which @rivi-software/d1-migrations splits migrations with. Install the Wrangler version the package's README names, and report the issue at https://github.com/rivi-software/tools/issues.");
|
|
64
|
+
return wrangler.unstable_splitSqlQuery;
|
|
65
|
+
}
|
|
66
|
+
/**
|
|
67
|
+
* The statements `wrangler d1 migrations apply --local` runs for a migration, without the one that
|
|
68
|
+
* records it. Wrangler adds `INSERT INTO <table> (name) values ('<name>');` on a new line after the
|
|
69
|
+
* migration's text and splits the two together, so this does too, then takes the INSERT off again.
|
|
70
|
+
*/
|
|
71
|
+
function splitMigration(split, name, sql, table) {
|
|
72
|
+
const record = `INSERT INTO "${table.replace(/"/g, '""')}" (name)\nvalues ('${name.replace(/'/g, "''")}')`;
|
|
73
|
+
const statements = split(`${sql}\n${record};`);
|
|
74
|
+
// A text with one statement comes back whole, its semicolon included; several come back trimmed,
|
|
75
|
+
// without theirs. A comment after the migration's last statement stays with the INSERT.
|
|
76
|
+
const last = (statements.pop() ?? "").trim().replace(/;$/, "");
|
|
77
|
+
if (!last.endsWith(record) || !isBlank(last.slice(0, -record.length)))
|
|
78
|
+
throw new Error(`Migration ${name} does not end with a complete statement: its last statement lacks a semicolon, or a comment, string or trigger body is left open. wrangler d1 migrations apply would run it together with the INSERT that records the migration.`);
|
|
79
|
+
return statements;
|
|
80
|
+
}
|
|
81
|
+
/** Whether `sql` holds nothing but whitespace and comments. */
|
|
82
|
+
function isBlank(sql) {
|
|
83
|
+
let index = 0;
|
|
84
|
+
while (index < sql.length) {
|
|
85
|
+
if (/\s/.test(sql[index]))
|
|
86
|
+
index++;
|
|
87
|
+
else if (sql.startsWith("--", index)) {
|
|
88
|
+
const end = sql.indexOf("\n", index);
|
|
89
|
+
index = end === -1 ? sql.length : end + 1;
|
|
90
|
+
}
|
|
91
|
+
else if (sql.startsWith("/*", index)) {
|
|
92
|
+
const end = sql.indexOf("*/", index + 2);
|
|
93
|
+
if (end === -1)
|
|
94
|
+
return false;
|
|
95
|
+
index = end + 2;
|
|
96
|
+
}
|
|
97
|
+
else
|
|
98
|
+
return false;
|
|
99
|
+
}
|
|
100
|
+
return true;
|
|
101
|
+
}
|
|
102
|
+
function renderModule(input) {
|
|
103
|
+
const string = (value) => JSON.stringify(value);
|
|
104
|
+
const items = input.migrations.map((migration) => {
|
|
105
|
+
const queries = migration.queries.map((query) => ` ${string(query)},\n`).join("");
|
|
106
|
+
return ` {\n name: ${string(migration.name)},\n queries: [${queries ? `\n${queries} ` : ""}],\n },\n`;
|
|
107
|
+
});
|
|
108
|
+
const list = `[${items.length ? `\n${items.join("")}` : ""}]`;
|
|
109
|
+
return [
|
|
110
|
+
`// Generated by @rivi-software/d1-migrations from ${input.from}. Do not edit: change the`,
|
|
111
|
+
`// migrations and bundle them again.`,
|
|
112
|
+
``,
|
|
113
|
+
...(input.typescript
|
|
114
|
+
? [`import type { D1Migration } from "@rivi-software/d1-migrations";`, ``]
|
|
115
|
+
: []),
|
|
116
|
+
`export const table = ${string(input.table)};`,
|
|
117
|
+
``,
|
|
118
|
+
input.typescript
|
|
119
|
+
? `export const migrations: D1Migration[] = ${list};`
|
|
120
|
+
: `/** @type {import("@rivi-software/d1-migrations").D1Migration[]} */\nexport const migrations = ${list};`,
|
|
121
|
+
``,
|
|
122
|
+
`export default { table, migrations };`,
|
|
123
|
+
``,
|
|
124
|
+
].join("\n");
|
|
125
|
+
}
|
package/dist/cli.d.ts
ADDED
package/dist/cli.js
ADDED
package/dist/files.d.ts
ADDED
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
/** Where the migration files are, as a D1 binding in the Wrangler config says it. */
|
|
2
|
+
export type MigrationsSource = {
|
|
3
|
+
/**
|
|
4
|
+
* The directory that holds the migrations, relative to `cwd`: the binding's `migrations_dir`.
|
|
5
|
+
* Default `migrations`.
|
|
6
|
+
*/
|
|
7
|
+
dir?: string;
|
|
8
|
+
/**
|
|
9
|
+
* A glob for the migration files, relative to `cwd` and under `dir`: the binding's
|
|
10
|
+
* `migrations_pattern`, such as `migrations/*\/migration.sql` for Drizzle's layout. Default
|
|
11
|
+
* `<dir>/*.sql`.
|
|
12
|
+
*/
|
|
13
|
+
pattern?: string;
|
|
14
|
+
/**
|
|
15
|
+
* The directory `dir` and `pattern` are relative to: the one that holds the Wrangler config.
|
|
16
|
+
* Default `process.cwd()`.
|
|
17
|
+
*/
|
|
18
|
+
cwd?: string;
|
|
19
|
+
};
|
|
20
|
+
/** A migration file: its name in the migrations table, and where it is on disk. */
|
|
21
|
+
export type MigrationFile = {
|
|
22
|
+
/** Its path under the migrations directory, with `/` between segments: `0001_init.sql`. */
|
|
23
|
+
name: string;
|
|
24
|
+
/** Its absolute path. */
|
|
25
|
+
path: string;
|
|
26
|
+
};
|
|
27
|
+
/** The resolved directory of a {@link MigrationsSource}, with its migration files in order. */
|
|
28
|
+
export type MigrationFiles = {
|
|
29
|
+
/** The migrations directory, absolute. */
|
|
30
|
+
dir: string;
|
|
31
|
+
files: MigrationFile[];
|
|
32
|
+
};
|
|
33
|
+
/**
|
|
34
|
+
* The migration files of `source`, in the order `wrangler d1 migrations apply` applies them. The
|
|
35
|
+
* names are the ones Wrangler records in its migrations table. Throws when the directory is
|
|
36
|
+
* missing, or when `pattern` is not under `dir`, as Wrangler does.
|
|
37
|
+
*/
|
|
38
|
+
export declare function listMigrationFiles(source?: MigrationsSource): MigrationFiles;
|
|
39
|
+
/**
|
|
40
|
+
* Wrangler's order for migration names. Segment by segment, a segment's leading number (the digits
|
|
41
|
+
* before its first `_`) sorts first, then the segment's text. A segment with a number sorts before
|
|
42
|
+
* one without. A shorter path sorts before a longer one that starts with it.
|
|
43
|
+
*/
|
|
44
|
+
export declare function compareMigrationPaths(a: string, b: string): number;
|
package/dist/files.js
ADDED
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
import { readdirSync, statSync } from "node:fs";
|
|
2
|
+
import path from "node:path";
|
|
3
|
+
import { Minimatch } from "minimatch";
|
|
4
|
+
/**
|
|
5
|
+
* The migration files of `source`, in the order `wrangler d1 migrations apply` applies them. The
|
|
6
|
+
* names are the ones Wrangler records in its migrations table. Throws when the directory is
|
|
7
|
+
* missing, or when `pattern` is not under `dir`, as Wrangler does.
|
|
8
|
+
*/
|
|
9
|
+
export function listMigrationFiles(source = {}) {
|
|
10
|
+
const cwd = path.resolve(source.cwd ?? ".");
|
|
11
|
+
const dir = normalizeRelativePath(source.dir ?? "migrations");
|
|
12
|
+
const pattern = normalizeRelativePath(source.pattern ?? `${dir}/*.sql`);
|
|
13
|
+
if (dir !== "." && !pattern.startsWith(`${dir}/`))
|
|
14
|
+
throw new Error(`The migrations pattern "${pattern}" must start with "${dir}/", the migrations directory.`);
|
|
15
|
+
const root = path.resolve(cwd, dir);
|
|
16
|
+
if (!isDirectory(root))
|
|
17
|
+
throw new Error(`No migrations directory at ${root}.`);
|
|
18
|
+
// Wrangler matches each path relative to the migrations directory, skipping dotfiles.
|
|
19
|
+
const matcher = new Minimatch(dir === "." ? pattern : pattern.slice(dir.length + 1), {
|
|
20
|
+
dot: false,
|
|
21
|
+
});
|
|
22
|
+
const names = [];
|
|
23
|
+
const walk = (absolute, relative) => {
|
|
24
|
+
for (const entry of readdirSync(absolute, { withFileTypes: true })) {
|
|
25
|
+
const name = relative === "" ? entry.name : `${relative}/${entry.name}`;
|
|
26
|
+
if (entry.isDirectory()) {
|
|
27
|
+
if (matcher.match(name, true))
|
|
28
|
+
walk(path.join(absolute, entry.name), name);
|
|
29
|
+
}
|
|
30
|
+
else if (entry.isFile() && matcher.match(name))
|
|
31
|
+
names.push(name);
|
|
32
|
+
}
|
|
33
|
+
};
|
|
34
|
+
walk(root, "");
|
|
35
|
+
names.sort(compareMigrationPaths);
|
|
36
|
+
return {
|
|
37
|
+
dir: root,
|
|
38
|
+
files: names.map((name) => ({ name, path: path.join(root, ...name.split("/")) })),
|
|
39
|
+
};
|
|
40
|
+
}
|
|
41
|
+
/**
|
|
42
|
+
* Wrangler's order for migration names. Segment by segment, a segment's leading number (the digits
|
|
43
|
+
* before its first `_`) sorts first, then the segment's text. A segment with a number sorts before
|
|
44
|
+
* one without. A shorter path sorts before a longer one that starts with it.
|
|
45
|
+
*/
|
|
46
|
+
export function compareMigrationPaths(a, b) {
|
|
47
|
+
const aSegments = a.split("/");
|
|
48
|
+
const bSegments = b.split("/");
|
|
49
|
+
for (let i = 0; i < Math.min(aSegments.length, bSegments.length); i++) {
|
|
50
|
+
const order = compareSegments(aSegments[i], bSegments[i]);
|
|
51
|
+
if (order !== 0)
|
|
52
|
+
return order;
|
|
53
|
+
}
|
|
54
|
+
return aSegments.length - bSegments.length;
|
|
55
|
+
}
|
|
56
|
+
function compareSegments(a, b) {
|
|
57
|
+
const aNumber = Number.parseInt(a.split("_")[0], 10);
|
|
58
|
+
const bNumber = Number.parseInt(b.split("_")[0], 10);
|
|
59
|
+
const aNumbered = Number.isFinite(aNumber);
|
|
60
|
+
const bNumbered = Number.isFinite(bNumber);
|
|
61
|
+
if (aNumbered && bNumbered && aNumber !== bNumber)
|
|
62
|
+
return aNumber - bNumber;
|
|
63
|
+
if (aNumbered !== bNumbered)
|
|
64
|
+
return aNumbered ? -1 : 1;
|
|
65
|
+
return a < b ? -1 : a > b ? 1 : 0;
|
|
66
|
+
}
|
|
67
|
+
/** Forward slashes, no `./` and no trailing `/`, as Wrangler normalizes the two config fields. */
|
|
68
|
+
function normalizeRelativePath(value) {
|
|
69
|
+
const normalized = path.posix.normalize(value.replace(/\\/g, "/"));
|
|
70
|
+
return normalized.length > 1 && normalized.endsWith("/") ? normalized.slice(0, -1) : normalized;
|
|
71
|
+
}
|
|
72
|
+
function isDirectory(file) {
|
|
73
|
+
try {
|
|
74
|
+
return statSync(file).isDirectory();
|
|
75
|
+
}
|
|
76
|
+
catch {
|
|
77
|
+
return false;
|
|
78
|
+
}
|
|
79
|
+
}
|
package/dist/index.d.ts
ADDED
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Applies D1 migrations from inside a Worker, to a database the Worker holds: one created at
|
|
3
|
+
* runtime, such as one database per customer, as well as one in the Wrangler config. It records them
|
|
4
|
+
* in Wrangler's own migrations table, so `wrangler d1 migrations apply` and the Worker agree on what
|
|
5
|
+
* has run. The migrations come from `bundleD1Migrations()` (`@rivi-software/d1-migrations/bundle`),
|
|
6
|
+
* which reads and splits the files at build time.
|
|
7
|
+
*/
|
|
8
|
+
/** The table Wrangler records applied migrations in when the binding sets no `migrations_table`. */
|
|
9
|
+
export declare const DEFAULT_MIGRATIONS_TABLE = "d1_migrations";
|
|
10
|
+
/** One migration: its name in the migrations table, and its SQL statements in order. */
|
|
11
|
+
export interface D1Migration {
|
|
12
|
+
readonly name: string;
|
|
13
|
+
readonly queries: readonly string[];
|
|
14
|
+
}
|
|
15
|
+
/** The migrations to apply, in order, and the table that records them. */
|
|
16
|
+
export interface D1Migrations {
|
|
17
|
+
/** The migrations table. Default {@link DEFAULT_MIGRATIONS_TABLE}. */
|
|
18
|
+
readonly table?: string;
|
|
19
|
+
readonly migrations: readonly D1Migration[];
|
|
20
|
+
}
|
|
21
|
+
/** The part of a prepared statement `migrateD1` uses. A `D1PreparedStatement` has it. */
|
|
22
|
+
export interface D1StatementLike {
|
|
23
|
+
bind(...values: unknown[]): D1StatementLike;
|
|
24
|
+
run(): Promise<unknown>;
|
|
25
|
+
all(): Promise<{
|
|
26
|
+
results: unknown[];
|
|
27
|
+
}>;
|
|
28
|
+
}
|
|
29
|
+
/** The part of a database `migrateD1` uses. A `D1Database` binding has it, and so does a session. */
|
|
30
|
+
export interface D1DatabaseLike {
|
|
31
|
+
prepare(query: string): D1StatementLike;
|
|
32
|
+
batch(statements: D1StatementLike[]): Promise<unknown>;
|
|
33
|
+
}
|
|
34
|
+
export type MigrateD1Result = {
|
|
35
|
+
/** The migrations this call applied, in order. Empty when the database was up to date. */
|
|
36
|
+
applied: string[];
|
|
37
|
+
};
|
|
38
|
+
/** A migration failed. The database keeps the migrations before it, and nothing of this one. */
|
|
39
|
+
export declare class D1MigrationError extends Error {
|
|
40
|
+
readonly name = "D1MigrationError";
|
|
41
|
+
/** The migration that failed. */
|
|
42
|
+
readonly migration: string;
|
|
43
|
+
/** The migrations this call applied before it. */
|
|
44
|
+
readonly applied: string[];
|
|
45
|
+
constructor(migration: string, applied: string[], cause: unknown);
|
|
46
|
+
}
|
|
47
|
+
/**
|
|
48
|
+
* Applies the migrations `db` has not run yet, in order, and returns their names.
|
|
49
|
+
*
|
|
50
|
+
* - It creates the migrations table as Wrangler does, and records each migration in it as Wrangler
|
|
51
|
+
* does, so either can apply the next one.
|
|
52
|
+
* - Each migration and the row that records it go in one `batch()`, which D1 runs as a transaction:
|
|
53
|
+
* a migration that fails leaves neither behind, and throws a {@link D1MigrationError}.
|
|
54
|
+
* - Calls that overlap on one database are safe. The table's `name` is unique, so when another call
|
|
55
|
+
* applies a migration first, this call's batch rolls back; it then finds the migration recorded
|
|
56
|
+
* and goes on to the next one.
|
|
57
|
+
* - A migration the table records but `migrations` lacks, such as one a newer deploy applied, is
|
|
58
|
+
* left alone.
|
|
59
|
+
*/
|
|
60
|
+
export declare function migrateD1(db: D1DatabaseLike, migrations: D1Migrations): Promise<MigrateD1Result>;
|
package/dist/index.js
ADDED
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Applies D1 migrations from inside a Worker, to a database the Worker holds: one created at
|
|
3
|
+
* runtime, such as one database per customer, as well as one in the Wrangler config. It records them
|
|
4
|
+
* in Wrangler's own migrations table, so `wrangler d1 migrations apply` and the Worker agree on what
|
|
5
|
+
* has run. The migrations come from `bundleD1Migrations()` (`@rivi-software/d1-migrations/bundle`),
|
|
6
|
+
* which reads and splits the files at build time.
|
|
7
|
+
*/
|
|
8
|
+
/** The table Wrangler records applied migrations in when the binding sets no `migrations_table`. */
|
|
9
|
+
export const DEFAULT_MIGRATIONS_TABLE = "d1_migrations";
|
|
10
|
+
/** A migration failed. The database keeps the migrations before it, and nothing of this one. */
|
|
11
|
+
export class D1MigrationError extends Error {
|
|
12
|
+
name = "D1MigrationError";
|
|
13
|
+
/** The migration that failed. */
|
|
14
|
+
migration;
|
|
15
|
+
/** The migrations this call applied before it. */
|
|
16
|
+
applied;
|
|
17
|
+
constructor(migration, applied, cause) {
|
|
18
|
+
super(`D1 migration ${migration} failed: ${cause instanceof Error ? cause.message : String(cause)}`, { cause });
|
|
19
|
+
this.migration = migration;
|
|
20
|
+
this.applied = applied;
|
|
21
|
+
}
|
|
22
|
+
}
|
|
23
|
+
/**
|
|
24
|
+
* Applies the migrations `db` has not run yet, in order, and returns their names.
|
|
25
|
+
*
|
|
26
|
+
* - It creates the migrations table as Wrangler does, and records each migration in it as Wrangler
|
|
27
|
+
* does, so either can apply the next one.
|
|
28
|
+
* - Each migration and the row that records it go in one `batch()`, which D1 runs as a transaction:
|
|
29
|
+
* a migration that fails leaves neither behind, and throws a {@link D1MigrationError}.
|
|
30
|
+
* - Calls that overlap on one database are safe. The table's `name` is unique, so when another call
|
|
31
|
+
* applies a migration first, this call's batch rolls back; it then finds the migration recorded
|
|
32
|
+
* and goes on to the next one.
|
|
33
|
+
* - A migration the table records but `migrations` lacks, such as one a newer deploy applied, is
|
|
34
|
+
* left alone.
|
|
35
|
+
*/
|
|
36
|
+
export async function migrateD1(db, migrations) {
|
|
37
|
+
const table = quoteIdentifier(migrations.table ?? DEFAULT_MIGRATIONS_TABLE);
|
|
38
|
+
// Byte for byte Wrangler's statement: SQLite keeps its text in sqlite_schema, so a database
|
|
39
|
+
// migrated here has the same schema as one `wrangler d1 migrations apply` migrated.
|
|
40
|
+
await db
|
|
41
|
+
.prepare(`CREATE TABLE IF NOT EXISTS ${table}(
|
|
42
|
+
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
43
|
+
name TEXT UNIQUE,
|
|
44
|
+
applied_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL
|
|
45
|
+
);`)
|
|
46
|
+
.run();
|
|
47
|
+
const recorded = async () => {
|
|
48
|
+
const { results } = await db.prepare(`SELECT name FROM ${table}`).all();
|
|
49
|
+
return new Set(results.map((row) => row.name));
|
|
50
|
+
};
|
|
51
|
+
const done = await recorded();
|
|
52
|
+
const record = db.prepare(`INSERT INTO ${table} (name) values (?)`);
|
|
53
|
+
const applied = [];
|
|
54
|
+
for (const migration of migrations.migrations) {
|
|
55
|
+
if (done.has(migration.name))
|
|
56
|
+
continue;
|
|
57
|
+
try {
|
|
58
|
+
await db.batch([
|
|
59
|
+
...migration.queries.map((query) => db.prepare(query)),
|
|
60
|
+
record.bind(migration.name),
|
|
61
|
+
]);
|
|
62
|
+
applied.push(migration.name);
|
|
63
|
+
}
|
|
64
|
+
catch (error) {
|
|
65
|
+
// Another call applied it first, and this batch rolled back.
|
|
66
|
+
const recordedNow = await recorded().catch(() => undefined);
|
|
67
|
+
if (recordedNow?.has(migration.name))
|
|
68
|
+
continue;
|
|
69
|
+
throw new D1MigrationError(migration.name, applied, error);
|
|
70
|
+
}
|
|
71
|
+
}
|
|
72
|
+
return { applied };
|
|
73
|
+
}
|
|
74
|
+
/** An SQL identifier in double quotes, as Wrangler quotes the migrations table's name. */
|
|
75
|
+
function quoteIdentifier(name) {
|
|
76
|
+
return `"${name.replace(/"/g, '""')}"`;
|
|
77
|
+
}
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
import { type MigrationsSource } from "./files.js";
|
|
2
|
+
export type { MigrationsSource } from "./files.js";
|
|
3
|
+
/** The journal's file name in the migrations directory, unless `journal` names another file. */
|
|
4
|
+
export declare const JOURNAL_FILE = "JOURNAL.json";
|
|
5
|
+
export type JournalOptions = MigrationsSource & {
|
|
6
|
+
/** The journal, relative to `cwd`. Default `JOURNAL.json` in the migrations directory. */
|
|
7
|
+
journal?: string;
|
|
8
|
+
};
|
|
9
|
+
export type JournalEntry = {
|
|
10
|
+
/** The migration's name, as its migrations table records it. */
|
|
11
|
+
name: string;
|
|
12
|
+
/** The SHA-256 of its text with CRLF line endings read as LF, in hexadecimal. */
|
|
13
|
+
sha256: string;
|
|
14
|
+
};
|
|
15
|
+
/** What the journal's file holds. */
|
|
16
|
+
export type Journal = {
|
|
17
|
+
version: 1;
|
|
18
|
+
migrations: JournalEntry[];
|
|
19
|
+
};
|
|
20
|
+
/**
|
|
21
|
+
* A migration that disagrees with the journal: `edited` (its text changed after it was recorded),
|
|
22
|
+
* `deleted` (recorded, but its file is gone) or `unrecorded` (a file the journal lacks).
|
|
23
|
+
*/
|
|
24
|
+
export type JournalProblem = {
|
|
25
|
+
name: string;
|
|
26
|
+
kind: "edited" | "deleted" | "unrecorded";
|
|
27
|
+
};
|
|
28
|
+
export type JournalCheck = {
|
|
29
|
+
/** The journal's absolute path. */
|
|
30
|
+
journal: string;
|
|
31
|
+
/** Empty when every migration matches the journal. */
|
|
32
|
+
problems: JournalProblem[];
|
|
33
|
+
};
|
|
34
|
+
export type JournalUpdate = {
|
|
35
|
+
/** The journal's absolute path. */
|
|
36
|
+
journal: string;
|
|
37
|
+
/** The migrations this call recorded, in order. */
|
|
38
|
+
added: string[];
|
|
39
|
+
/**
|
|
40
|
+
* Recorded migrations that were edited or deleted. When there are any, the journal is left as it
|
|
41
|
+
* was: a migration that may have run never changes, so a change belongs in a new migration.
|
|
42
|
+
*/
|
|
43
|
+
problems: JournalProblem[];
|
|
44
|
+
};
|
|
45
|
+
/** The SHA-256 the journal records for a migration's text. */
|
|
46
|
+
export declare function hashMigration(sql: string): string;
|
|
47
|
+
/** Compares the migrations with the journal. A missing journal records nothing. */
|
|
48
|
+
export declare function checkJournal(options?: JournalOptions): Promise<JournalCheck>;
|
|
49
|
+
/**
|
|
50
|
+
* Records the migrations the journal lacks, and writes it. It never changes or drops an entry: when
|
|
51
|
+
* a recorded migration was edited or deleted, it returns those problems and writes nothing. To
|
|
52
|
+
* accept an edit to a migration that has run nowhere yet, delete its entry from the journal first.
|
|
53
|
+
*/
|
|
54
|
+
export declare function updateJournal(options?: JournalOptions): Promise<JournalUpdate>;
|
package/dist/journal.js
ADDED
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* The journal: a JSON file beside the migrations with a SHA-256 of each one. Wrangler records an
|
|
3
|
+
* applied migration by its name only, so an edit to a migration after it ran goes unnoticed, and
|
|
4
|
+
* databases migrated before and after the edit differ. The journal records each migration once;
|
|
5
|
+
* `checkJournal()`, in CI, fails when a recorded migration is edited or deleted, or when a new one
|
|
6
|
+
* was never recorded.
|
|
7
|
+
*/
|
|
8
|
+
import { createHash } from "node:crypto";
|
|
9
|
+
import { readFile, writeFile } from "node:fs/promises";
|
|
10
|
+
import path from "node:path";
|
|
11
|
+
import { compareMigrationPaths, listMigrationFiles } from "./files.js";
|
|
12
|
+
/** The journal's file name in the migrations directory, unless `journal` names another file. */
|
|
13
|
+
export const JOURNAL_FILE = "JOURNAL.json";
|
|
14
|
+
/** The SHA-256 the journal records for a migration's text. */
|
|
15
|
+
export function hashMigration(sql) {
|
|
16
|
+
// A checkout with Git's autocrlf has CRLF line endings: still the same migration.
|
|
17
|
+
return createHash("sha256").update(sql.replace(/\r\n/g, "\n")).digest("hex");
|
|
18
|
+
}
|
|
19
|
+
/** Compares the migrations with the journal. A missing journal records nothing. */
|
|
20
|
+
export async function checkJournal(options = {}) {
|
|
21
|
+
const { journal, recorded, current } = await load(options);
|
|
22
|
+
return { journal, problems: compare(recorded, current) };
|
|
23
|
+
}
|
|
24
|
+
function compare(recorded, current) {
|
|
25
|
+
const problems = [];
|
|
26
|
+
for (const [name, sha256] of recorded) {
|
|
27
|
+
const actual = current.get(name);
|
|
28
|
+
if (actual === undefined)
|
|
29
|
+
problems.push({ name, kind: "deleted" });
|
|
30
|
+
else if (actual !== sha256)
|
|
31
|
+
problems.push({ name, kind: "edited" });
|
|
32
|
+
}
|
|
33
|
+
for (const name of current.keys())
|
|
34
|
+
if (!recorded.has(name))
|
|
35
|
+
problems.push({ name, kind: "unrecorded" });
|
|
36
|
+
return problems.sort((a, b) => compareMigrationPaths(a.name, b.name));
|
|
37
|
+
}
|
|
38
|
+
/**
|
|
39
|
+
* Records the migrations the journal lacks, and writes it. It never changes or drops an entry: when
|
|
40
|
+
* a recorded migration was edited or deleted, it returns those problems and writes nothing. To
|
|
41
|
+
* accept an edit to a migration that has run nowhere yet, delete its entry from the journal first.
|
|
42
|
+
*/
|
|
43
|
+
export async function updateJournal(options = {}) {
|
|
44
|
+
const { journal, exists, recorded, current } = await load(options);
|
|
45
|
+
const problems = compare(recorded, current);
|
|
46
|
+
const blocking = problems.filter((problem) => problem.kind !== "unrecorded");
|
|
47
|
+
if (blocking.length)
|
|
48
|
+
return { journal, added: [], problems: blocking };
|
|
49
|
+
const added = problems.map((problem) => problem.name);
|
|
50
|
+
for (const name of added)
|
|
51
|
+
recorded.set(name, current.get(name));
|
|
52
|
+
if (added.length || !exists) {
|
|
53
|
+
const content = {
|
|
54
|
+
version: 1,
|
|
55
|
+
migrations: [...recorded]
|
|
56
|
+
.sort(([a], [b]) => compareMigrationPaths(a, b))
|
|
57
|
+
.map(([name, sha256]) => ({ name, sha256 })),
|
|
58
|
+
};
|
|
59
|
+
await writeFile(journal, `${JSON.stringify(content, null, 2)}\n`);
|
|
60
|
+
}
|
|
61
|
+
return { journal, added, problems: [] };
|
|
62
|
+
}
|
|
63
|
+
async function load(options) {
|
|
64
|
+
const cwd = path.resolve(options.cwd ?? ".");
|
|
65
|
+
const { dir, files } = listMigrationFiles({ ...options, cwd });
|
|
66
|
+
const journal = options.journal
|
|
67
|
+
? path.resolve(cwd, options.journal)
|
|
68
|
+
: path.join(dir, JOURNAL_FILE);
|
|
69
|
+
const text = await readFile(journal, "utf8").catch((error) => {
|
|
70
|
+
if (error.code === "ENOENT")
|
|
71
|
+
return undefined;
|
|
72
|
+
throw error;
|
|
73
|
+
});
|
|
74
|
+
const recorded = new Map((text === undefined ? [] : parseJournal(text, journal).migrations).map((entry) => [entry.name, entry.sha256]));
|
|
75
|
+
const current = new Map();
|
|
76
|
+
for (const file of files)
|
|
77
|
+
current.set(file.name, hashMigration(await readFile(file.path, "utf8")));
|
|
78
|
+
return { journal, exists: text !== undefined, recorded, current };
|
|
79
|
+
}
|
|
80
|
+
function parseJournal(text, file) {
|
|
81
|
+
let value;
|
|
82
|
+
try {
|
|
83
|
+
value = JSON.parse(text);
|
|
84
|
+
}
|
|
85
|
+
catch (error) {
|
|
86
|
+
throw new Error(`${file} is not valid JSON.`, { cause: error });
|
|
87
|
+
}
|
|
88
|
+
const migrations = value?.migrations;
|
|
89
|
+
if (!Array.isArray(migrations) ||
|
|
90
|
+
!migrations.every((entry) => typeof entry?.name === "string" &&
|
|
91
|
+
typeof entry?.sha256 === "string" &&
|
|
92
|
+
/^[0-9a-f]{64}$/.test(entry.sha256)))
|
|
93
|
+
throw new Error(`${file} is not a migrations journal: it needs a "migrations" list of { "name", "sha256" } entries.`);
|
|
94
|
+
return { version: 1, migrations };
|
|
95
|
+
}
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
export declare const USAGE = "Usage:\n d1-migrations bundle --out <file> [--dir <dir>] [--pattern <glob>] [--table <name>]\n d1-migrations journal [--check] [--dir <dir>] [--pattern <glob>] [--journal <file>]\n\n bundle Writes the migrations to a module for migrateD1(), split into statements.\n journal Records new migrations in the journal. With --check, writes nothing and\n fails when a migration was edited, deleted or never recorded.\n\n --dir The binding's migrations_dir (default: migrations)\n --pattern The binding's migrations_pattern (default: <dir>/*.sql)\n --table The binding's migrations_table (default: d1_migrations)\n --journal The journal (default: <dir>/JOURNAL.json)\n\nPaths are relative to the current directory, the one with the Wrangler config.";
|
|
2
|
+
type Context = {
|
|
3
|
+
/** The directory paths are relative to. */
|
|
4
|
+
cwd: string;
|
|
5
|
+
log: (line: string) => void;
|
|
6
|
+
error: (line: string) => void;
|
|
7
|
+
};
|
|
8
|
+
/** Runs `d1-migrations` with `args` and returns its exit code. */
|
|
9
|
+
export declare function runCli(args: string[], context?: Context): Promise<number>;
|
|
10
|
+
export {};
|
package/dist/run-cli.js
ADDED
|
@@ -0,0 +1,101 @@
|
|
|
1
|
+
import path from "node:path";
|
|
2
|
+
import { parseArgs } from "node:util";
|
|
3
|
+
import { bundleD1Migrations } from "./bundle.js";
|
|
4
|
+
import { checkJournal, updateJournal } from "./journal.js";
|
|
5
|
+
export const USAGE = `Usage:
|
|
6
|
+
d1-migrations bundle --out <file> [--dir <dir>] [--pattern <glob>] [--table <name>]
|
|
7
|
+
d1-migrations journal [--check] [--dir <dir>] [--pattern <glob>] [--journal <file>]
|
|
8
|
+
|
|
9
|
+
bundle Writes the migrations to a module for migrateD1(), split into statements.
|
|
10
|
+
journal Records new migrations in the journal. With --check, writes nothing and
|
|
11
|
+
fails when a migration was edited, deleted or never recorded.
|
|
12
|
+
|
|
13
|
+
--dir The binding's migrations_dir (default: migrations)
|
|
14
|
+
--pattern The binding's migrations_pattern (default: <dir>/*.sql)
|
|
15
|
+
--table The binding's migrations_table (default: d1_migrations)
|
|
16
|
+
--journal The journal (default: <dir>/JOURNAL.json)
|
|
17
|
+
|
|
18
|
+
Paths are relative to the current directory, the one with the Wrangler config.`;
|
|
19
|
+
/** Runs `d1-migrations` with `args` and returns its exit code. */
|
|
20
|
+
export async function runCli(args, context = { cwd: process.cwd(), log: console.log, error: console.error }) {
|
|
21
|
+
const [command, ...rest] = args;
|
|
22
|
+
let values;
|
|
23
|
+
try {
|
|
24
|
+
({ values } = parseArgs({
|
|
25
|
+
args: rest,
|
|
26
|
+
strict: true,
|
|
27
|
+
options: {
|
|
28
|
+
out: { type: "string" },
|
|
29
|
+
dir: { type: "string" },
|
|
30
|
+
pattern: { type: "string" },
|
|
31
|
+
table: { type: "string" },
|
|
32
|
+
journal: { type: "string" },
|
|
33
|
+
check: { type: "boolean" },
|
|
34
|
+
},
|
|
35
|
+
}));
|
|
36
|
+
}
|
|
37
|
+
catch (error) {
|
|
38
|
+
context.error(`${error instanceof Error ? error.message : error}\n\n${USAGE}`);
|
|
39
|
+
return 2;
|
|
40
|
+
}
|
|
41
|
+
const source = { cwd: context.cwd, dir: values.dir, pattern: values.pattern };
|
|
42
|
+
const shown = (file) => path.relative(context.cwd, file) || file;
|
|
43
|
+
try {
|
|
44
|
+
if (command === "bundle" && values.out && !values.journal && !values.check) {
|
|
45
|
+
const result = await bundleD1Migrations({ ...source, table: values.table, out: values.out });
|
|
46
|
+
const count = `${result.migrations.length} migration${result.migrations.length === 1 ? "" : "s"}`;
|
|
47
|
+
context.log(result.changed
|
|
48
|
+
? `Bundled ${count} into ${shown(result.out)}.`
|
|
49
|
+
: `${shown(result.out)} already holds the ${count}.`);
|
|
50
|
+
return 0;
|
|
51
|
+
}
|
|
52
|
+
if (command === "journal" && !values.out && !values.table) {
|
|
53
|
+
const options = { ...source, journal: values.journal };
|
|
54
|
+
if (values.check) {
|
|
55
|
+
const { journal, problems } = await checkJournal(options);
|
|
56
|
+
if (!problems.length) {
|
|
57
|
+
context.log(`Every migration matches ${shown(journal)}.`);
|
|
58
|
+
return 0;
|
|
59
|
+
}
|
|
60
|
+
context.error(`The migrations disagree with ${shown(journal)}:`);
|
|
61
|
+
for (const problem of problems)
|
|
62
|
+
context.error(` ${describe(problem)}`);
|
|
63
|
+
context.error(advice(problems));
|
|
64
|
+
return 1;
|
|
65
|
+
}
|
|
66
|
+
const { journal, added, problems } = await updateJournal(options);
|
|
67
|
+
if (problems.length) {
|
|
68
|
+
context.error(`${shown(journal)} is unchanged. These recorded migrations changed:`);
|
|
69
|
+
for (const problem of problems)
|
|
70
|
+
context.error(` ${describe(problem)}`);
|
|
71
|
+
context.error(advice(problems));
|
|
72
|
+
return 1;
|
|
73
|
+
}
|
|
74
|
+
context.log(added.length
|
|
75
|
+
? `Recorded in ${shown(journal)}: ${added.join(", ")}.`
|
|
76
|
+
: `${shown(journal)} already records every migration.`);
|
|
77
|
+
return 0;
|
|
78
|
+
}
|
|
79
|
+
}
|
|
80
|
+
catch (error) {
|
|
81
|
+
context.error(error instanceof Error ? error.message : String(error));
|
|
82
|
+
return 1;
|
|
83
|
+
}
|
|
84
|
+
context.error(USAGE);
|
|
85
|
+
return 2;
|
|
86
|
+
}
|
|
87
|
+
function describe(problem) {
|
|
88
|
+
switch (problem.kind) {
|
|
89
|
+
case "edited":
|
|
90
|
+
return `${problem.name} was edited after it was recorded.`;
|
|
91
|
+
case "deleted":
|
|
92
|
+
return `${problem.name} was recorded, and its file is gone.`;
|
|
93
|
+
case "unrecorded":
|
|
94
|
+
return `${problem.name} is not recorded: run d1-migrations journal.`;
|
|
95
|
+
}
|
|
96
|
+
}
|
|
97
|
+
function advice(problems) {
|
|
98
|
+
return problems.some((problem) => problem.kind !== "unrecorded")
|
|
99
|
+
? "A migration that may have run must stay as it is: put the change in a new migration. If it has run on no database yet, delete its entry from the journal and run d1-migrations journal."
|
|
100
|
+
: "Run d1-migrations journal, and commit the journal with the migrations.";
|
|
101
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,71 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@rivi-software/d1-migrations",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"
|
|
5
|
-
"
|
|
3
|
+
"version": "0.1.0",
|
|
4
|
+
"description": "D1 migrations from inside a Worker, for databases an app creates at runtime: the same files, order and d1_migrations table as wrangler d1 migrations apply, and a journal of hashes that fails CI when an applied migration is edited",
|
|
5
|
+
"keywords": [
|
|
6
|
+
"cloudflare",
|
|
7
|
+
"cloudflare-workers",
|
|
8
|
+
"d1",
|
|
9
|
+
"migrations",
|
|
10
|
+
"sqlite",
|
|
11
|
+
"wrangler",
|
|
12
|
+
"multi-tenant"
|
|
13
|
+
],
|
|
14
|
+
"license": "MIT",
|
|
15
|
+
"type": "module",
|
|
16
|
+
"exports": {
|
|
17
|
+
".": {
|
|
18
|
+
"types": "./dist/index.d.ts",
|
|
19
|
+
"default": "./dist/index.js"
|
|
20
|
+
},
|
|
21
|
+
"./bundle": {
|
|
22
|
+
"types": "./dist/bundle.d.ts",
|
|
23
|
+
"default": "./dist/bundle.js"
|
|
24
|
+
},
|
|
25
|
+
"./journal": {
|
|
26
|
+
"types": "./dist/journal.d.ts",
|
|
27
|
+
"default": "./dist/journal.js"
|
|
28
|
+
}
|
|
29
|
+
},
|
|
30
|
+
"bin": {
|
|
31
|
+
"d1-migrations": "./dist/cli.js"
|
|
32
|
+
},
|
|
33
|
+
"files": [
|
|
34
|
+
"dist"
|
|
35
|
+
],
|
|
36
|
+
"sideEffects": false,
|
|
37
|
+
"dependencies": {
|
|
38
|
+
"minimatch": "^10.2.5"
|
|
39
|
+
},
|
|
40
|
+
"peerDependencies": {
|
|
41
|
+
"wrangler": "^4.0.0"
|
|
42
|
+
},
|
|
43
|
+
"peerDependenciesMeta": {
|
|
44
|
+
"wrangler": {
|
|
45
|
+
"optional": true
|
|
46
|
+
}
|
|
47
|
+
},
|
|
48
|
+
"devDependencies": {
|
|
49
|
+
"@cloudflare/workers-types": "5.20261102.1",
|
|
50
|
+
"miniflare": "5.20261102.0-beta",
|
|
51
|
+
"wrangler": "4.156.2"
|
|
52
|
+
},
|
|
53
|
+
"engines": {
|
|
54
|
+
"node": ">=22"
|
|
55
|
+
},
|
|
56
|
+
"repository": {
|
|
57
|
+
"type": "git",
|
|
58
|
+
"url": "git+https://github.com/rivi-software/tools.git",
|
|
59
|
+
"directory": "packages/d1-migrations"
|
|
60
|
+
},
|
|
61
|
+
"homepage": "https://github.com/rivi-software/tools/tree/main/packages/d1-migrations#readme",
|
|
62
|
+
"bugs": "https://github.com/rivi-software/tools/issues",
|
|
63
|
+
"publishConfig": {
|
|
64
|
+
"access": "public"
|
|
65
|
+
},
|
|
66
|
+
"scripts": {
|
|
67
|
+
"build": "rm -rf dist && tsc -p tsconfig.build.json",
|
|
68
|
+
"check": "tsc --noEmit",
|
|
69
|
+
"test": "vitest run"
|
|
70
|
+
}
|
|
6
71
|
}
|