@cipherstash/migrate 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +19 -0
- package/LICENSE.md +21 -0
- package/README.md +89 -0
- package/dist/index.cjs +524 -0
- package/dist/index.cjs.map +1 -0
- package/dist/index.d.cts +740 -0
- package/dist/index.d.ts +740 -0
- package/dist/index.js +465 -0
- package/dist/index.js.map +1 -0
- package/package.json +66 -0
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
# @cipherstash/migrate
|
|
2
|
+
|
|
3
|
+
## 0.2.0
|
|
4
|
+
|
|
5
|
+
### Minor Changes
|
|
6
|
+
|
|
7
|
+
- add4357: Add `stash encrypt` command group and `@cipherstash/migrate` library for plaintext → encrypted column migrations.
|
|
8
|
+
|
|
9
|
+
New CLI commands:
|
|
10
|
+
|
|
11
|
+
- `stash encrypt status` — per-column migration status (phase, backfill progress, drift between intent and state, EQL registration).
|
|
12
|
+
- `stash encrypt plan` — diff `.cipherstash/migrations.json` (intent) vs observed state.
|
|
13
|
+
- `stash encrypt backfill --table <t> --column <c>` — resumable, idempotent, chunked encryption of plaintext into `<col>_encrypted`. Uses the user's encryption client (Protect/Stack). SIGINT-safe; re-run to resume. The first run on a column prompts to confirm dual-writes are deployed (or accept `--confirm-dual-writes-deployed` for non-interactive contexts), records the `dual_writing` transition in `cs_migrations`, then runs the chunked encryption loop. `--force` re-encrypts every plaintext row regardless of current state — recovery path for drift caused by an earlier backfill running before dual-writes were actually live.
|
|
14
|
+
- `stash encrypt cutover --table <t> --column <c>` — runs `eql_v2.rename_encrypted_columns()` inside a transaction; optionally forces Proxy config refresh via `CIPHERSTASH_PROXY_URL`. After cutover, apps reading `<col>` transparently receive the encrypted column.
|
|
15
|
+
- `stash encrypt drop --table <t> --column <c>` — generates a migration file that drops the old plaintext column.
|
|
16
|
+
|
|
17
|
+
`stash db install` now also installs a `cipherstash.cs_migrations` table used to track per-column migration runtime state (current phase, backfill cursor, rows processed). The table is append-only (event-log shape) and kept separate from `eql_v2_configuration` which remains the authoritative EQL intent store used by Proxy.
|
|
18
|
+
|
|
19
|
+
The new `@cipherstash/migrate` package exposes the same primitives as a library for users who want to embed backfill in their own workers or cron jobs — all commands are thin wrappers around its exports (`runBackfill`, `appendEvent`, `latestByColumn`, `progress`, `renameEncryptedColumns`, `reloadConfig`, `readManifest`, `writeManifest`).
|
package/LICENSE.md
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 CipherStash
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,89 @@
|
|
|
1
|
+
# @cipherstash/migrate
|
|
2
|
+
|
|
3
|
+
Primitives for migrating existing plaintext columns to CipherStash's `eql_v2_encrypted` in production Postgres databases, safely and resumably.
|
|
4
|
+
|
|
5
|
+
Backs the `stash encrypt` CLI command group, but also exported for direct use — embed `runBackfill()` in your own worker or cron job when you'd rather not pipe gigabytes through a CLI process.
|
|
6
|
+
|
|
7
|
+
## Lifecycle
|
|
8
|
+
|
|
9
|
+
Each column walks through these phases:
|
|
10
|
+
|
|
11
|
+
```
|
|
12
|
+
schema-added → dual-writing → backfilling → backfilled → cut-over → dropped
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
State is tracked in an append-only `cipherstash.cs_migrations` table installed by `stash db install`. The EQL intent (which indexes, which cast_as) continues to live in `eql_v2_configuration` so Proxy continues to work against the same database.
|
|
16
|
+
|
|
17
|
+
## API
|
|
18
|
+
|
|
19
|
+
```ts
|
|
20
|
+
import {
|
|
21
|
+
installMigrationsSchema,
|
|
22
|
+
appendEvent,
|
|
23
|
+
latestByColumn,
|
|
24
|
+
progress,
|
|
25
|
+
runBackfill,
|
|
26
|
+
renameEncryptedColumns,
|
|
27
|
+
reloadConfig,
|
|
28
|
+
readManifest,
|
|
29
|
+
writeManifest,
|
|
30
|
+
} from '@cipherstash/migrate'
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
### `installMigrationsSchema(client)`
|
|
34
|
+
|
|
35
|
+
Creates `cipherstash.cs_migrations` idempotently. Normally called by `stash db install`.
|
|
36
|
+
|
|
37
|
+
### `runBackfill({ db, encryptionClient, tableSchema, tableName, plaintextColumn, encryptedColumn, pkColumn, schemaColumnKey, chunkSize?, signal?, onProgress? })`
|
|
38
|
+
|
|
39
|
+
Chunked, resumable, idempotent backfill of plaintext → encrypted. Per chunk, in a single transaction: select next page → encrypt via `client.bulkEncryptModels` → `UPDATE … FROM (VALUES …)` → `INSERT` a `backfill_checkpoint` event. Guards with `encrypted IS NULL` so re-runs never double-write.
|
|
40
|
+
|
|
41
|
+
- `db`: a `pg.PoolClient` (the runner drives transactions on it).
|
|
42
|
+
- `encryptionClient`: your initialised `@cipherstash/stack` `EncryptionClient` (or anything that exposes `bulkEncryptModels(models, table)` returning `{ data } | { failure }`).
|
|
43
|
+
- `tableSchema`: the `EncryptedTable` for the target table from your encryption client file.
|
|
44
|
+
- `signal`: optional `AbortSignal`. If aborted between chunks, the backfill exits cleanly and leaves a resumable checkpoint.
|
|
45
|
+
|
|
46
|
+
Returns `{ resumed, rowsProcessed, rowsTotal, completed }`.
|
|
47
|
+
|
|
48
|
+
### `appendEvent(client, { tableName, columnName, event, phase, … })` / `progress(client, table, column)` / `latestByColumn(client)`
|
|
49
|
+
|
|
50
|
+
Direct access to the `cs_migrations` event log. Use these if you're building your own migration UI or orchestration on top.
|
|
51
|
+
|
|
52
|
+
### `renameEncryptedColumns(client)` / `reloadConfig(client)`
|
|
53
|
+
|
|
54
|
+
Thin wrappers around `eql_v2.rename_encrypted_columns()` (the cut-over primitive) and `eql_v2.reload_config()` (Proxy refresh hint — no-op when connected directly to Postgres).
|
|
55
|
+
|
|
56
|
+
### `readManifest(cwd)` / `writeManifest(manifest, cwd)`
|
|
57
|
+
|
|
58
|
+
Read/write `.cipherstash/migrations.json` — the repo-side intent declaration. Zod-validated. The manifest is optional; commands work without it but you lose the `plan` diff.
|
|
59
|
+
|
|
60
|
+
## Drop-in usage in a BullMQ/Inngest worker
|
|
61
|
+
|
|
62
|
+
```ts
|
|
63
|
+
import pg from 'pg'
|
|
64
|
+
import { runBackfill } from '@cipherstash/migrate'
|
|
65
|
+
import { encryptionClient, usersTable } from './src/encryption/index.js'
|
|
66
|
+
|
|
67
|
+
const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL })
|
|
68
|
+
|
|
69
|
+
export async function handler({ signal }: { signal: AbortSignal }) {
|
|
70
|
+
const db = await pool.connect()
|
|
71
|
+
try {
|
|
72
|
+
return await runBackfill({
|
|
73
|
+
db,
|
|
74
|
+
encryptionClient,
|
|
75
|
+
tableSchema: usersTable,
|
|
76
|
+
tableName: 'users',
|
|
77
|
+
schemaColumnKey: 'email',
|
|
78
|
+
plaintextColumn: 'email',
|
|
79
|
+
encryptedColumn: 'email_encrypted',
|
|
80
|
+
pkColumn: 'id',
|
|
81
|
+
chunkSize: 2000,
|
|
82
|
+
signal,
|
|
83
|
+
onProgress: (p) => console.log(`${p.rowsProcessed}/${p.rowsTotal}`),
|
|
84
|
+
})
|
|
85
|
+
} finally {
|
|
86
|
+
db.release()
|
|
87
|
+
}
|
|
88
|
+
}
|
|
89
|
+
```
|
package/dist/index.cjs
ADDED
|
@@ -0,0 +1,524 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
var __create = Object.create;
|
|
3
|
+
var __defProp = Object.defineProperty;
|
|
4
|
+
var __getOwnPropDesc = Object.getOwnPropertyDescriptor;
|
|
5
|
+
var __getOwnPropNames = Object.getOwnPropertyNames;
|
|
6
|
+
var __getProtoOf = Object.getPrototypeOf;
|
|
7
|
+
var __hasOwnProp = Object.prototype.hasOwnProperty;
|
|
8
|
+
var __export = (target, all) => {
|
|
9
|
+
for (var name in all)
|
|
10
|
+
__defProp(target, name, { get: all[name], enumerable: true });
|
|
11
|
+
};
|
|
12
|
+
var __copyProps = (to, from, except, desc) => {
|
|
13
|
+
if (from && typeof from === "object" || typeof from === "function") {
|
|
14
|
+
for (let key of __getOwnPropNames(from))
|
|
15
|
+
if (!__hasOwnProp.call(to, key) && key !== except)
|
|
16
|
+
__defProp(to, key, { get: () => from[key], enumerable: !(desc = __getOwnPropDesc(from, key)) || desc.enumerable });
|
|
17
|
+
}
|
|
18
|
+
return to;
|
|
19
|
+
};
|
|
20
|
+
var __toESM = (mod, isNodeMode, target) => (target = mod != null ? __create(__getProtoOf(mod)) : {}, __copyProps(
|
|
21
|
+
// If the importer is in node compatibility mode or this is not an ESM
|
|
22
|
+
// file that has been converted to a CommonJS file using a Babel-
|
|
23
|
+
// compatible transform (i.e. "__esModule" has not been set), then set
|
|
24
|
+
// "default" to the CommonJS "module.exports" for node compatibility.
|
|
25
|
+
isNodeMode || !mod || !mod.__esModule ? __defProp(target, "default", { value: mod, enumerable: true }) : target,
|
|
26
|
+
mod
|
|
27
|
+
));
|
|
28
|
+
var __toCommonJS = (mod) => __copyProps(__defProp({}, "__esModule", { value: true }), mod);
|
|
29
|
+
|
|
30
|
+
// src/index.ts
|
|
31
|
+
var index_exports = {};
|
|
32
|
+
__export(index_exports, {
|
|
33
|
+
MIGRATIONS_SCHEMA_SQL: () => MIGRATIONS_SCHEMA_SQL,
|
|
34
|
+
activateConfig: () => activateConfig,
|
|
35
|
+
appendEvent: () => appendEvent,
|
|
36
|
+
countEncryptedWithActiveConfig: () => countEncryptedWithActiveConfig,
|
|
37
|
+
countUnencrypted: () => countUnencrypted,
|
|
38
|
+
discardPendingConfig: () => discardPendingConfig,
|
|
39
|
+
fetchUnencryptedPage: () => fetchUnencryptedPage,
|
|
40
|
+
installMigrationsSchema: () => installMigrationsSchema,
|
|
41
|
+
latestByColumn: () => latestByColumn,
|
|
42
|
+
manifestPath: () => manifestPath,
|
|
43
|
+
migrateConfig: () => migrateConfig,
|
|
44
|
+
progress: () => progress,
|
|
45
|
+
qualifyTable: () => qualifyTable,
|
|
46
|
+
quoteIdent: () => quoteIdent,
|
|
47
|
+
readManifest: () => readManifest,
|
|
48
|
+
readyForEncryption: () => readyForEncryption,
|
|
49
|
+
reloadConfig: () => reloadConfig,
|
|
50
|
+
renameEncryptedColumns: () => renameEncryptedColumns,
|
|
51
|
+
runBackfill: () => runBackfill,
|
|
52
|
+
selectPendingColumns: () => selectPendingColumns,
|
|
53
|
+
setManifestTargetPhase: () => setManifestTargetPhase,
|
|
54
|
+
upsertManifestColumn: () => upsertManifestColumn,
|
|
55
|
+
writeManifest: () => writeManifest
|
|
56
|
+
});
|
|
57
|
+
module.exports = __toCommonJS(index_exports);
|
|
58
|
+
|
|
59
|
+
// src/install.ts
|
|
60
|
+
var MIGRATIONS_SCHEMA_SQL = `
|
|
61
|
+
CREATE SCHEMA IF NOT EXISTS cipherstash;
|
|
62
|
+
|
|
63
|
+
CREATE TABLE IF NOT EXISTS cipherstash.cs_migrations (
|
|
64
|
+
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
|
65
|
+
table_name text NOT NULL,
|
|
66
|
+
column_name text NOT NULL,
|
|
67
|
+
event text NOT NULL,
|
|
68
|
+
phase text NOT NULL,
|
|
69
|
+
cursor_value text,
|
|
70
|
+
rows_processed bigint,
|
|
71
|
+
rows_total bigint,
|
|
72
|
+
details jsonb,
|
|
73
|
+
created_at timestamptz NOT NULL DEFAULT now()
|
|
74
|
+
);
|
|
75
|
+
|
|
76
|
+
CREATE INDEX IF NOT EXISTS cs_migrations_column_id_desc
|
|
77
|
+
ON cipherstash.cs_migrations (table_name, column_name, id DESC);
|
|
78
|
+
`;
|
|
79
|
+
async function installMigrationsSchema(client) {
|
|
80
|
+
await client.query(MIGRATIONS_SCHEMA_SQL);
|
|
81
|
+
}
|
|
82
|
+
|
|
83
|
+
// src/state.ts
|
|
84
|
+
async function appendEvent(client, input) {
|
|
85
|
+
const result = await client.query(
|
|
86
|
+
`INSERT INTO cipherstash.cs_migrations
|
|
87
|
+
(table_name, column_name, event, phase, cursor_value, rows_processed, rows_total, details)
|
|
88
|
+
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
|
|
89
|
+
RETURNING id, table_name, column_name, event, phase, cursor_value, rows_processed, rows_total, details, created_at`,
|
|
90
|
+
[
|
|
91
|
+
input.tableName,
|
|
92
|
+
input.columnName,
|
|
93
|
+
input.event,
|
|
94
|
+
input.phase,
|
|
95
|
+
input.cursorValue ?? null,
|
|
96
|
+
input.rowsProcessed ?? null,
|
|
97
|
+
input.rowsTotal ?? null,
|
|
98
|
+
input.details ?? null
|
|
99
|
+
]
|
|
100
|
+
);
|
|
101
|
+
return rowToState(result.rows[0]);
|
|
102
|
+
}
|
|
103
|
+
async function latestByColumn(client) {
|
|
104
|
+
const result = await client.query(
|
|
105
|
+
`SELECT DISTINCT ON (table_name, column_name)
|
|
106
|
+
id, table_name, column_name, event, phase, cursor_value, rows_processed, rows_total, details, created_at
|
|
107
|
+
FROM cipherstash.cs_migrations
|
|
108
|
+
ORDER BY table_name, column_name, id DESC`
|
|
109
|
+
);
|
|
110
|
+
const map = /* @__PURE__ */ new Map();
|
|
111
|
+
for (const row of result.rows) {
|
|
112
|
+
const state = rowToState(row);
|
|
113
|
+
map.set(`${state.tableName}.${state.columnName}`, state);
|
|
114
|
+
}
|
|
115
|
+
return map;
|
|
116
|
+
}
|
|
117
|
+
async function progress(client, tableName, columnName) {
|
|
118
|
+
const result = await client.query(
|
|
119
|
+
`SELECT id, table_name, column_name, event, phase, cursor_value, rows_processed, rows_total, details, created_at
|
|
120
|
+
FROM cipherstash.cs_migrations
|
|
121
|
+
WHERE table_name = $1 AND column_name = $2
|
|
122
|
+
ORDER BY id DESC
|
|
123
|
+
LIMIT 1`,
|
|
124
|
+
[tableName, columnName]
|
|
125
|
+
);
|
|
126
|
+
if (result.rows.length === 0) return null;
|
|
127
|
+
return rowToState(result.rows[0]);
|
|
128
|
+
}
|
|
129
|
+
function rowToState(row) {
|
|
130
|
+
return {
|
|
131
|
+
id: String(row.id),
|
|
132
|
+
tableName: row.table_name,
|
|
133
|
+
columnName: row.column_name,
|
|
134
|
+
event: row.event,
|
|
135
|
+
phase: row.phase,
|
|
136
|
+
cursorValue: row.cursor_value,
|
|
137
|
+
rowsProcessed: row.rows_processed === null ? null : Number(row.rows_processed),
|
|
138
|
+
rowsTotal: row.rows_total === null ? null : Number(row.rows_total),
|
|
139
|
+
details: row.details,
|
|
140
|
+
createdAt: row.created_at
|
|
141
|
+
};
|
|
142
|
+
}
|
|
143
|
+
|
|
144
|
+
// src/eql.ts
|
|
145
|
+
async function selectPendingColumns(client) {
|
|
146
|
+
const result = await client.query("SELECT table_name, column_name FROM eql_v2.select_pending_columns()");
|
|
147
|
+
return result.rows.map((row) => ({
|
|
148
|
+
tableName: row.table_name,
|
|
149
|
+
columnName: row.column_name
|
|
150
|
+
}));
|
|
151
|
+
}
|
|
152
|
+
async function readyForEncryption(client) {
|
|
153
|
+
const result = await client.query(
|
|
154
|
+
"SELECT eql_v2.ready_for_encryption() AS ready"
|
|
155
|
+
);
|
|
156
|
+
return result.rows[0]?.ready === true;
|
|
157
|
+
}
|
|
158
|
+
async function renameEncryptedColumns(client) {
|
|
159
|
+
await client.query("SELECT eql_v2.rename_encrypted_columns()");
|
|
160
|
+
}
|
|
161
|
+
async function migrateConfig(client) {
|
|
162
|
+
await client.query("SELECT eql_v2.migrate_config()");
|
|
163
|
+
}
|
|
164
|
+
async function activateConfig(client) {
|
|
165
|
+
await client.query("SELECT eql_v2.activate_config()");
|
|
166
|
+
}
|
|
167
|
+
async function discardPendingConfig(client) {
|
|
168
|
+
await client.query(
|
|
169
|
+
"DELETE FROM public.eql_v2_configuration WHERE state = 'pending'"
|
|
170
|
+
);
|
|
171
|
+
}
|
|
172
|
+
async function reloadConfig(client) {
|
|
173
|
+
await client.query("SELECT eql_v2.reload_config()");
|
|
174
|
+
}
|
|
175
|
+
async function countEncryptedWithActiveConfig(client, tableName, columnName) {
|
|
176
|
+
const result = await client.query(
|
|
177
|
+
"SELECT eql_v2.count_encrypted_with_active_config($1, $2) AS count",
|
|
178
|
+
[tableName, columnName]
|
|
179
|
+
);
|
|
180
|
+
return BigInt(result.rows[0]?.count ?? "0");
|
|
181
|
+
}
|
|
182
|
+
|
|
183
|
+
// src/sql.ts
|
|
184
|
+
function quoteIdent(identifier) {
|
|
185
|
+
return `"${identifier.replace(/"/g, '""')}"`;
|
|
186
|
+
}
|
|
187
|
+
|
|
188
|
+
// src/cursor.ts
|
|
189
|
+
async function fetchUnencryptedPage(client, opts) {
|
|
190
|
+
const pk = quoteIdent(opts.pkColumn);
|
|
191
|
+
const plain = quoteIdent(opts.plaintextColumn);
|
|
192
|
+
const enc = quoteIdent(opts.encryptedColumn);
|
|
193
|
+
const table = qualifyTable(opts.tableName);
|
|
194
|
+
const params = [];
|
|
195
|
+
let where = opts.force ? `${plain} IS NOT NULL` : `${plain} IS NOT NULL AND ${enc} IS NULL`;
|
|
196
|
+
if (opts.after !== null) {
|
|
197
|
+
params.push(opts.after);
|
|
198
|
+
where += ` AND ${pk} > $${params.length}`;
|
|
199
|
+
}
|
|
200
|
+
params.push(opts.limit);
|
|
201
|
+
const limitParam = `$${params.length}`;
|
|
202
|
+
const sql = `
|
|
203
|
+
SELECT ${pk}::text AS pk, ${plain} AS plaintext
|
|
204
|
+
FROM ${table}
|
|
205
|
+
WHERE ${where}
|
|
206
|
+
ORDER BY ${pk} ASC
|
|
207
|
+
LIMIT ${limitParam}
|
|
208
|
+
`;
|
|
209
|
+
const result = await client.query(
|
|
210
|
+
sql,
|
|
211
|
+
params
|
|
212
|
+
);
|
|
213
|
+
const rows = result.rows;
|
|
214
|
+
const lastPk = rows.length > 0 ? rows[rows.length - 1]?.pk : null;
|
|
215
|
+
return { rows, lastPk };
|
|
216
|
+
}
|
|
217
|
+
async function countUnencrypted(client, tableName, plaintextColumn, encryptedColumn, force = false) {
|
|
218
|
+
const plain = quoteIdent(plaintextColumn);
|
|
219
|
+
const enc = quoteIdent(encryptedColumn);
|
|
220
|
+
const table = qualifyTable(tableName);
|
|
221
|
+
const where = force ? `${plain} IS NOT NULL` : `${plain} IS NOT NULL AND ${enc} IS NULL`;
|
|
222
|
+
const result = await client.query(
|
|
223
|
+
`SELECT count(*)::text AS count FROM ${table} WHERE ${where}`
|
|
224
|
+
);
|
|
225
|
+
return Number(result.rows[0]?.count ?? 0);
|
|
226
|
+
}
|
|
227
|
+
function qualifyTable(tableName) {
|
|
228
|
+
if (tableName.includes(".")) {
|
|
229
|
+
const parts = tableName.split(".");
|
|
230
|
+
return parts.map(quoteIdent).join(".");
|
|
231
|
+
}
|
|
232
|
+
return quoteIdent(tableName);
|
|
233
|
+
}
|
|
234
|
+
|
|
235
|
+
// src/backfill.ts
|
|
236
|
+
var import_stack = require("@cipherstash/stack");
|
|
237
|
+
async function runBackfill(options) {
|
|
238
|
+
const chunkSize = options.chunkSize ?? 1e3;
|
|
239
|
+
const { db, tableName, pkColumn, plaintextColumn, encryptedColumn } = options;
|
|
240
|
+
const force = options.force ?? false;
|
|
241
|
+
const rowsTotal = await countUnencrypted(
|
|
242
|
+
db,
|
|
243
|
+
tableName,
|
|
244
|
+
plaintextColumn,
|
|
245
|
+
encryptedColumn,
|
|
246
|
+
force
|
|
247
|
+
);
|
|
248
|
+
const last = await progress(db, tableName, plaintextColumn);
|
|
249
|
+
const resumeCursor = last?.event === "backfill_checkpoint" ? last.cursorValue : null;
|
|
250
|
+
const resumed = resumeCursor !== null;
|
|
251
|
+
const priorProcessed = last?.event === "backfill_checkpoint" ? last.rowsProcessed ?? 0 : 0;
|
|
252
|
+
await appendEvent(db, {
|
|
253
|
+
tableName,
|
|
254
|
+
columnName: plaintextColumn,
|
|
255
|
+
event: "backfill_started",
|
|
256
|
+
phase: "backfilling",
|
|
257
|
+
cursorValue: resumeCursor,
|
|
258
|
+
rowsProcessed: priorProcessed,
|
|
259
|
+
rowsTotal: priorProcessed + rowsTotal,
|
|
260
|
+
details: { chunkSize, resumed, ...force ? { force: true } : {} }
|
|
261
|
+
});
|
|
262
|
+
let cursor = resumeCursor;
|
|
263
|
+
let rowsProcessed = priorProcessed;
|
|
264
|
+
const rowsTotalWithResumed = priorProcessed + rowsTotal;
|
|
265
|
+
let chunkIndex = 0;
|
|
266
|
+
let completed = false;
|
|
267
|
+
try {
|
|
268
|
+
while (true) {
|
|
269
|
+
if (options.signal?.aborted) break;
|
|
270
|
+
const page = await fetchUnencryptedPage(db, {
|
|
271
|
+
tableName,
|
|
272
|
+
pkColumn,
|
|
273
|
+
plaintextColumn,
|
|
274
|
+
encryptedColumn,
|
|
275
|
+
after: cursor,
|
|
276
|
+
limit: chunkSize,
|
|
277
|
+
force
|
|
278
|
+
});
|
|
279
|
+
if (page.rows.length === 0) {
|
|
280
|
+
completed = true;
|
|
281
|
+
break;
|
|
282
|
+
}
|
|
283
|
+
const coerce = options.transformPlaintext ?? ((v) => v);
|
|
284
|
+
const models = page.rows.map((row) => ({
|
|
285
|
+
__pk: row.pk,
|
|
286
|
+
[options.schemaColumnKey]: coerce(row.plaintext)
|
|
287
|
+
}));
|
|
288
|
+
const encryptResult = await options.encryptionClient.bulkEncryptModels(
|
|
289
|
+
models,
|
|
290
|
+
options.tableSchema
|
|
291
|
+
);
|
|
292
|
+
if (encryptResult.failure) {
|
|
293
|
+
throw new Error(
|
|
294
|
+
`bulkEncryptModels failed: ${encryptResult.failure.message}`
|
|
295
|
+
);
|
|
296
|
+
}
|
|
297
|
+
for (const [i, row] of encryptResult.data.entries()) {
|
|
298
|
+
const value = row[options.schemaColumnKey];
|
|
299
|
+
if (!(0, import_stack.isEncryptedPayload)(value)) {
|
|
300
|
+
const pk = row.__pk ?? page.rows[i]?.pk;
|
|
301
|
+
const valueType = value === null ? "null" : typeof value;
|
|
302
|
+
throw new Error(
|
|
303
|
+
`Encryption client returned a non-ciphertext value (type: ${valueType}) at model key "${options.schemaColumnKey}" for pk=${pk}. This usually means the schema column key does not match your EncryptedTable. Verify that your schema declares a column keyed "${options.schemaColumnKey}", or pass --schema-column-key <name> to override.`
|
|
304
|
+
);
|
|
305
|
+
}
|
|
306
|
+
}
|
|
307
|
+
await db.query("BEGIN");
|
|
308
|
+
try {
|
|
309
|
+
await writeEncryptedChunk(db, {
|
|
310
|
+
tableName,
|
|
311
|
+
pkColumn,
|
|
312
|
+
encryptedColumn,
|
|
313
|
+
schemaColumnKey: options.schemaColumnKey,
|
|
314
|
+
encryptedRows: encryptResult.data,
|
|
315
|
+
force
|
|
316
|
+
});
|
|
317
|
+
rowsProcessed += page.rows.length;
|
|
318
|
+
cursor = page.lastPk;
|
|
319
|
+
await appendEvent(db, {
|
|
320
|
+
tableName,
|
|
321
|
+
columnName: plaintextColumn,
|
|
322
|
+
event: "backfill_checkpoint",
|
|
323
|
+
phase: "backfilling",
|
|
324
|
+
cursorValue: cursor,
|
|
325
|
+
rowsProcessed,
|
|
326
|
+
rowsTotal: rowsTotalWithResumed,
|
|
327
|
+
details: { chunkIndex, chunkRows: page.rows.length }
|
|
328
|
+
});
|
|
329
|
+
await db.query("COMMIT");
|
|
330
|
+
} catch (err) {
|
|
331
|
+
await db.query("ROLLBACK").catch(() => {
|
|
332
|
+
});
|
|
333
|
+
throw err;
|
|
334
|
+
}
|
|
335
|
+
options.onProgress?.({
|
|
336
|
+
rowsProcessed,
|
|
337
|
+
rowsTotal: rowsTotalWithResumed,
|
|
338
|
+
lastPk: cursor,
|
|
339
|
+
chunkSize,
|
|
340
|
+
chunkIndex
|
|
341
|
+
});
|
|
342
|
+
chunkIndex += 1;
|
|
343
|
+
}
|
|
344
|
+
if (completed) {
|
|
345
|
+
await appendEvent(db, {
|
|
346
|
+
tableName,
|
|
347
|
+
columnName: plaintextColumn,
|
|
348
|
+
event: "backfilled",
|
|
349
|
+
phase: "backfilled",
|
|
350
|
+
cursorValue: cursor,
|
|
351
|
+
rowsProcessed,
|
|
352
|
+
rowsTotal: rowsTotalWithResumed,
|
|
353
|
+
details: { chunkCount: chunkIndex }
|
|
354
|
+
});
|
|
355
|
+
}
|
|
356
|
+
} catch (err) {
|
|
357
|
+
await appendEvent(db, {
|
|
358
|
+
tableName,
|
|
359
|
+
columnName: plaintextColumn,
|
|
360
|
+
event: "error",
|
|
361
|
+
phase: "backfilling",
|
|
362
|
+
cursorValue: cursor,
|
|
363
|
+
rowsProcessed,
|
|
364
|
+
rowsTotal: rowsTotalWithResumed,
|
|
365
|
+
details: {
|
|
366
|
+
message: err instanceof Error ? err.message : String(err),
|
|
367
|
+
chunkIndex
|
|
368
|
+
}
|
|
369
|
+
});
|
|
370
|
+
throw err;
|
|
371
|
+
}
|
|
372
|
+
return {
|
|
373
|
+
resumed,
|
|
374
|
+
rowsProcessed,
|
|
375
|
+
rowsTotal: rowsTotalWithResumed,
|
|
376
|
+
completed
|
|
377
|
+
};
|
|
378
|
+
}
|
|
379
|
+
async function writeEncryptedChunk(db, opts) {
|
|
380
|
+
if (opts.encryptedRows.length === 0) return;
|
|
381
|
+
const table = qualifyTable(opts.tableName);
|
|
382
|
+
const pk = quoteIdent(opts.pkColumn);
|
|
383
|
+
const enc = quoteIdent(opts.encryptedColumn);
|
|
384
|
+
const params = [];
|
|
385
|
+
const valuesSql = opts.encryptedRows.map((row) => {
|
|
386
|
+
const pkValue = row.__pk;
|
|
387
|
+
const encryptedValue = row[opts.schemaColumnKey];
|
|
388
|
+
params.push(pkValue);
|
|
389
|
+
const pkParam = `$${params.length}`;
|
|
390
|
+
params.push(encryptedValue);
|
|
391
|
+
const encParam = `$${params.length}::jsonb`;
|
|
392
|
+
return `(${pkParam}, ${encParam})`;
|
|
393
|
+
}).join(", ");
|
|
394
|
+
const where = opts.force ? `t.${pk}::text = v.pk::text` : `t.${pk}::text = v.pk::text AND t.${enc} IS NULL`;
|
|
395
|
+
const sql = `
|
|
396
|
+
UPDATE ${table} AS t
|
|
397
|
+
SET ${enc} = v.enc
|
|
398
|
+
FROM (VALUES ${valuesSql}) AS v(pk, enc)
|
|
399
|
+
WHERE ${where}
|
|
400
|
+
`;
|
|
401
|
+
await db.query(sql, params);
|
|
402
|
+
}
|
|
403
|
+
|
|
404
|
+
// src/manifest.ts
|
|
405
|
+
var fs = __toESM(require("fs/promises"), 1);
|
|
406
|
+
var path = __toESM(require("path"), 1);
|
|
407
|
+
var import_zod = require("zod");
|
|
408
|
+
var IndexKind = import_zod.z.enum(["unique", "match", "ore", "ste_vec"]);
|
|
409
|
+
var CastAs = import_zod.z.enum([
|
|
410
|
+
"text",
|
|
411
|
+
"int",
|
|
412
|
+
"small_int",
|
|
413
|
+
"big_int",
|
|
414
|
+
"real",
|
|
415
|
+
"double",
|
|
416
|
+
"boolean",
|
|
417
|
+
"date",
|
|
418
|
+
"jsonb",
|
|
419
|
+
"json",
|
|
420
|
+
"float",
|
|
421
|
+
"decimal",
|
|
422
|
+
"timestamp"
|
|
423
|
+
]);
|
|
424
|
+
var ManifestColumnSchema = import_zod.z.object({
|
|
425
|
+
/** Physical column name, e.g. `email`. */
|
|
426
|
+
column: import_zod.z.string(),
|
|
427
|
+
/**
|
|
428
|
+
* EQL cast type. Text by default. See the EQL docs for the full list
|
|
429
|
+
* (`text | int | small_int | big_int | real | double | boolean | date |
|
|
430
|
+
* jsonb | json | float | decimal | timestamp`).
|
|
431
|
+
*/
|
|
432
|
+
castAs: CastAs.default("text"),
|
|
433
|
+
/** Desired EQL index set. Driver of the `indexes: {…}` block in EQL config. */
|
|
434
|
+
indexes: import_zod.z.array(IndexKind).default([]),
|
|
435
|
+
/**
|
|
436
|
+
* The phase the user wants this column to reach. `cut-over` is the
|
|
437
|
+
* typical end state (reads transparently decrypted); advance to
|
|
438
|
+
* `dropped` only once you're confident the plaintext column is no
|
|
439
|
+
* longer needed.
|
|
440
|
+
*/
|
|
441
|
+
targetPhase: import_zod.z.enum(["schema-added", "dual-writing", "backfilled", "cut-over", "dropped"]).default("cut-over"),
|
|
442
|
+
/**
|
|
443
|
+
* Override for primary-key detection during backfill. Omit to let the
|
|
444
|
+
* CLI auto-detect via `information_schema`.
|
|
445
|
+
*/
|
|
446
|
+
pkColumn: import_zod.z.string().optional()
|
|
447
|
+
});
|
|
448
|
+
var ManifestSchema = import_zod.z.object({
|
|
449
|
+
version: import_zod.z.literal(1).default(1),
|
|
450
|
+
/** Map of table name → array of column intents for that table. */
|
|
451
|
+
tables: import_zod.z.record(import_zod.z.array(ManifestColumnSchema))
|
|
452
|
+
});
|
|
453
|
+
function manifestPath(cwd = process.cwd()) {
|
|
454
|
+
return path.join(cwd, ".cipherstash", "migrations.json");
|
|
455
|
+
}
|
|
456
|
+
async function readManifest(cwd = process.cwd()) {
|
|
457
|
+
const filePath = manifestPath(cwd);
|
|
458
|
+
let raw;
|
|
459
|
+
try {
|
|
460
|
+
raw = await fs.readFile(filePath, "utf-8");
|
|
461
|
+
} catch (err) {
|
|
462
|
+
if (err.code === "ENOENT") return null;
|
|
463
|
+
throw err;
|
|
464
|
+
}
|
|
465
|
+
const parsed = ManifestSchema.parse(JSON.parse(raw));
|
|
466
|
+
return parsed;
|
|
467
|
+
}
|
|
468
|
+
async function writeManifest(manifest, cwd = process.cwd()) {
|
|
469
|
+
const filePath = manifestPath(cwd);
|
|
470
|
+
await fs.mkdir(path.dirname(filePath), { recursive: true });
|
|
471
|
+
const validated = ManifestSchema.parse(manifest);
|
|
472
|
+
await fs.writeFile(
|
|
473
|
+
filePath,
|
|
474
|
+
`${JSON.stringify(validated, null, 2)}
|
|
475
|
+
`,
|
|
476
|
+
"utf-8"
|
|
477
|
+
);
|
|
478
|
+
}
|
|
479
|
+
async function upsertManifestColumn(table, column, cwd = process.cwd()) {
|
|
480
|
+
const existing = await readManifest(cwd) ?? { version: 1, tables: {} };
|
|
481
|
+
const tableColumns = existing.tables[table] ?? [];
|
|
482
|
+
const remaining = tableColumns.filter((c) => c.column !== column.column);
|
|
483
|
+
existing.tables[table] = [...remaining, column];
|
|
484
|
+
await writeManifest(existing, cwd);
|
|
485
|
+
}
|
|
486
|
+
async function setManifestTargetPhase(table, columnName, targetPhase, cwd = process.cwd()) {
|
|
487
|
+
const existing = await readManifest(cwd);
|
|
488
|
+
if (!existing) return;
|
|
489
|
+
const tableColumns = existing.tables[table];
|
|
490
|
+
if (!tableColumns) return;
|
|
491
|
+
const current = tableColumns.find((c) => c.column === columnName);
|
|
492
|
+
if (!current) return;
|
|
493
|
+
existing.tables[table] = tableColumns.map(
|
|
494
|
+
(c) => c.column === columnName ? { ...c, targetPhase } : c
|
|
495
|
+
);
|
|
496
|
+
await writeManifest(existing, cwd);
|
|
497
|
+
}
|
|
498
|
+
// Annotate the CommonJS export names for ESM import in node:
|
|
499
|
+
0 && (module.exports = {
|
|
500
|
+
MIGRATIONS_SCHEMA_SQL,
|
|
501
|
+
activateConfig,
|
|
502
|
+
appendEvent,
|
|
503
|
+
countEncryptedWithActiveConfig,
|
|
504
|
+
countUnencrypted,
|
|
505
|
+
discardPendingConfig,
|
|
506
|
+
fetchUnencryptedPage,
|
|
507
|
+
installMigrationsSchema,
|
|
508
|
+
latestByColumn,
|
|
509
|
+
manifestPath,
|
|
510
|
+
migrateConfig,
|
|
511
|
+
progress,
|
|
512
|
+
qualifyTable,
|
|
513
|
+
quoteIdent,
|
|
514
|
+
readManifest,
|
|
515
|
+
readyForEncryption,
|
|
516
|
+
reloadConfig,
|
|
517
|
+
renameEncryptedColumns,
|
|
518
|
+
runBackfill,
|
|
519
|
+
selectPendingColumns,
|
|
520
|
+
setManifestTargetPhase,
|
|
521
|
+
upsertManifestColumn,
|
|
522
|
+
writeManifest
|
|
523
|
+
});
|
|
524
|
+
//# sourceMappingURL=index.cjs.map
|