@voltro/sql-postgres 0.43.2 → 0.44.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +34 -0
- package/dist/index.d.ts +13 -0
- package/dist/index.js +109 -100
- package/package.json +3 -3
package/CHANGELOG.md
CHANGED
|
@@ -39,6 +39,40 @@ _Changes staged for the next release accumulate here (rolled up from
|
|
|
39
39
|
|
|
40
40
|
---
|
|
41
41
|
|
|
42
|
+
## [0.44.0] — 2026-08-19
|
|
43
|
+
|
|
44
|
+
### ⚠ BREAKING
|
|
45
|
+
|
|
46
|
+
- **@voltro/data-transfer, @voltro/cli** — `runImport` returns an `ImportOutcome` instead of the bundle's `Manifest`, and `--mode replace` runs as ONE transaction by default.
|
|
47
|
+
|
|
48
|
+
**Why the return type changed.** The summary line counted the rows the BUNDLE carries, not the rows the run wrote. Those differ most exactly where it matters: a bundle directory carries its own resume ledger, so a copied directory imports nothing — correctly, with a warning naming the file to delete — and the run then printed `import complete … rows: 242950` over a target it had not touched. The warning was one line above, which is one line too far for anyone piping the output through `tail -1`. The outcome carries `rowsWritten`, `rowsSkipped` and `fullyResumed`, the CLI reports written-vs-carried, and a fully-skipped run says so on its LAST line.
|
|
49
|
+
|
|
50
|
+
Migration: `runImport(...)` now resolves to `{ manifest, tablesWritten, rowsWritten, tablesSkipped, rowsSkipped, fullyResumed }`. Read `.manifest` where you read the manifest before.
|
|
51
|
+
|
|
52
|
+
**Why replace is atomic now.** The all-or-nothing guarantee was written for the emptying step, and read — reasonably — as covering the run. A replace that died partway through the LOAD left the target emptied of its old rows and holding part of the new ones, measured on a live instance over sixteen minutes. There is no useful state for a replace to stop in, so it is a default rather than a flag you have to know about. It also closes the window that produced that failure: with the tables emptied and the load uncommitted, a concurrent writer in the application waits instead of inserting a row the bundle is about to insert too.
|
|
53
|
+
|
|
54
|
+
`--no-atomic` (CLI) / `atomic: false` (API) opts out. The trade is stated where it bites: every write to those tables waits for the load, and on postgres the bulk `COPY` loader cannot join a transaction it does not own — an atomic run now says that once rather than being quietly slower.
|
|
55
|
+
|
|
56
|
+
Two smaller things from the same report: `--mode replace` against a running instance warns about the empty-target window when it is NOT atomic, and the `--target api` upload reports progress per chunk plus a line explaining that the final request stays open for the whole import — sixteen minutes of silence is indistinguishable from a hang, and one operator killed a run that had finished.
|
|
57
|
+
|
|
58
|
+
**`voltro update` carries you across this** — codemod `0.44.0/01_import-outcome`.
|
|
59
|
+
|
|
60
|
+
### Fixed
|
|
61
|
+
|
|
62
|
+
- **@voltro/sql-mysql, @voltro/sql-postgres, @voltro/sql-sqlite, @voltro/sql-mssql, @voltro/data-transfer** — A write and its write-recorders now succeed or fail together on every SQL dialect, and the data importer's retry is idempotent.
|
|
63
|
+
|
|
64
|
+
A recorder (a versioning trail, an audit log) runs on the caller's connection and is ALLOWED to fail — a recorder that throws must take the write down with it, that is its contract. Outside a caller transaction the two were not one unit: the row's statement committed on its own, and the recorder's INSERT ran afterwards as a second autocommit statement. So a recorder that threw left a COMMITTED row behind a write that reported failure. Measured directly: `insert` throws, the row is in the table, and a second attempt at the same row is `ER_DUP_ENTRY` on PRIMARY.
|
|
65
|
+
|
|
66
|
+
Anything that retries a failed write then meets its own row. The data importer retries by design — it holds a row whose write failed and tries again once the remaining tables have streamed — so a `--mode replace` that had just emptied a table failed on a duplicate key IN that table. From outside, that is the impossible-looking thing: an import that emptied a table and then failed because something was already in it. Reproduced verbatim, including the table name and `[ER_DUP_ENTRY/1062]`, by putting the old retry back.
|
|
67
|
+
|
|
68
|
+
Both ends are closed, and they are independent on purpose:
|
|
69
|
+
|
|
70
|
+
- **The cause.** A table that HAS recorders writes inside a transaction now, so the row and the trail commit together or not at all — on mysql, mariadb, postgres, sqlite and mssql, verified by one suite that asks all five the same question. A table with no recorders — the default — takes the direct path unchanged; `recordsTable` is a Map-size check first, so it costs one comparison. - **The defence.** The importer's retry upserts instead of inserting in `replace` mode. That covers every OTHER way a write can land while reporting failure: a driver timeout on a write the server applied, a connection lost after the commit, a concurrent writer inserting the same key. It is sound precisely because the table was emptied by this same run — there is nothing in it that is not ours.
|
|
71
|
+
|
|
72
|
+
The test that pinned the divergence went red when the fix landed, exactly as its own note said it would, and is inverted with that note kept.
|
|
73
|
+
|
|
74
|
+
---
|
|
75
|
+
|
|
42
76
|
## [0.43.2] — 2026-08-18
|
|
43
77
|
|
|
44
78
|
### Fixed
|
package/dist/index.d.ts
CHANGED
|
@@ -379,6 +379,19 @@ export declare class PostgresDataStore implements DataStore {
|
|
|
379
379
|
* The gap being closed is specific: a plain write's statement commits ITSELF,
|
|
380
380
|
* so the transport fires while this is still awaiting the driver.
|
|
381
381
|
*/
|
|
382
|
+
/**
|
|
383
|
+
* A write and its write-recorders, as ONE unit — see `MysqlStore.
|
|
384
|
+
* writeWithRecorders` for the failure this closes.
|
|
385
|
+
*
|
|
386
|
+
* Same divergence on this dialect: the row's statement commits on its own and
|
|
387
|
+
* the recorder's INSERT is a second autocommit statement, so a recorder that
|
|
388
|
+
* throws leaves a committed row behind a write that reported failure. A caller
|
|
389
|
+
* that retries then meets its own row.
|
|
390
|
+
*
|
|
391
|
+
* Only tables that HAVE a recorder pay for the transaction; `recordsTable` is
|
|
392
|
+
* a Map-size check first, so the default costs one comparison.
|
|
393
|
+
*/
|
|
394
|
+
private writeWithRecorders;
|
|
382
395
|
private localWrite;
|
|
383
396
|
insert(table: string, row: Row): Promise<Row>;
|
|
384
397
|
insertMany(table: string, rows: ReadonlyArray<Row>): Promise<ReadonlyArray<Row>>;
|
package/dist/index.js
CHANGED
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
import { PgClient as e, PgClient as t } from "@effect/sql-pg";
|
|
2
|
-
import { Config as n, Duration as r, Effect as i, Fiber as a, Layer as o, ManagedRuntime as s, Metric as c,
|
|
3
|
-
import
|
|
4
|
-
import { DEFAULT_ACQUIRE_TIMEOUT_MS as
|
|
5
|
-
import { EventEmitter as
|
|
6
|
-
import { createLogger as
|
|
2
|
+
import { Config as n, Duration as r, Effect as i, Fiber as a, Layer as o, ManagedRuntime as s, Metric as c, Option as l, Redacted as u, Schedule as d, Stream as f } from "effect";
|
|
3
|
+
import p from "pg";
|
|
4
|
+
import { DEFAULT_ACQUIRE_TIMEOUT_MS as m, EagerCardinalityError as h, attachEagerLoads as g, attributionFields as ee, attributionKey as te, beginLocalWrite as ne, bulkInsertLimitsFor as re, chunkRowsForInsert as ie, compileEagerJson as ae, compilePredicate as _, compileRawFragment as oe, compileSelect as se, encodeRowForSchema as v, endLocalWrite as ce, externalChangeEvent as le, hasEagerLoads as ue, isTableReactive as y, makeEagerFallbackReporter as de, observeDbOp as b, raiseChangeListenerCeiling as fe, recordsTable as x, registerPendingAttribution as S, requireTable as C, resolveEchoAttribution as w, runRetryingTransaction as pe, runStoreTransaction as T, runWriteRecorders as me, stampGeneratedId as E, stampGeneratedIds as he, withCapturedAttribution as D } from "@voltro/database";
|
|
5
|
+
import { EventEmitter as ge } from "node:events";
|
|
6
|
+
import { createLogger as _e } from "@voltro/logger";
|
|
7
7
|
import { SqlClient as O, TransactionConnection as k } from "@effect/sql/SqlClient";
|
|
8
8
|
//#region src/sqlLayer.ts
|
|
9
9
|
var A = (e) => {
|
|
10
|
-
let t = e.acquireTimeoutMs ??
|
|
10
|
+
let t = e.acquireTimeoutMs ?? m;
|
|
11
11
|
return t > 0 ? t : void 0;
|
|
12
12
|
}, j = (e) => e ? { rejectUnauthorized: !1 } : !1, M = /^[A-Za-z_][A-Za-z0-9_]*$/, N = (e) => {
|
|
13
13
|
let t = A(e);
|
|
@@ -15,7 +15,7 @@ var A = (e) => {
|
|
|
15
15
|
host: n.succeed(e.host),
|
|
16
16
|
port: n.succeed(e.port),
|
|
17
17
|
username: n.succeed(e.username),
|
|
18
|
-
password: n.succeed(
|
|
18
|
+
password: n.succeed(u.make(e.password)),
|
|
19
19
|
database: n.succeed(e.database),
|
|
20
20
|
...e.maxConnections === void 0 ? {} : { maxConnections: n.succeed(e.maxConnections) },
|
|
21
21
|
...e.ssl === void 0 ? {} : { ssl: n.succeed(j(e.ssl)) },
|
|
@@ -28,7 +28,7 @@ var A = (e) => {
|
|
|
28
28
|
if (!M.test(t.schema)) throw Error(`DB_SCHEMA '${t.schema}' is not a valid postgres identifier (expected ${M}).`);
|
|
29
29
|
n = { options: `-c search_path="${t.schema}"` };
|
|
30
30
|
}
|
|
31
|
-
let r = A(t), a = i.acquireRelease(i.sync(() => new
|
|
31
|
+
let r = A(t), a = i.acquireRelease(i.sync(() => new p.Pool({
|
|
32
32
|
host: t.host,
|
|
33
33
|
port: t.port,
|
|
34
34
|
user: t.username,
|
|
@@ -83,37 +83,37 @@ var A = (e) => {
|
|
|
83
83
|
...n,
|
|
84
84
|
...t
|
|
85
85
|
};
|
|
86
|
-
}, L = (e) => P(I(e)),
|
|
86
|
+
}, L = (e) => P(I(e)), ve = /* @__PURE__ */ new Set(["40001", "40P01"]), ye = (e) => {
|
|
87
87
|
let t = e;
|
|
88
88
|
for (let e = 0; e < 5 && typeof t == "object" && t; e++) {
|
|
89
89
|
let e = t.code;
|
|
90
90
|
if (typeof e == "string") return e;
|
|
91
91
|
t = t.cause;
|
|
92
92
|
}
|
|
93
|
-
},
|
|
94
|
-
let t =
|
|
95
|
-
return t !== void 0 &&
|
|
96
|
-
},
|
|
97
|
-
rehydrated: c.tagged(
|
|
98
|
-
tombstone: c.tagged(
|
|
99
|
-
unrecovered: c.tagged(
|
|
100
|
-
},
|
|
101
|
-
i.runSync(c.increment(
|
|
102
|
-
},
|
|
93
|
+
}, R = (e) => {
|
|
94
|
+
let t = ye(e);
|
|
95
|
+
return t !== void 0 && ve.has(t);
|
|
96
|
+
}, z = (e) => R(e) ? "retry" : "noRetry", B = c.counter("voltro_cdc_oversized_total", { description: "Change events whose row images exceeded the postgres NOTIFY payload cap, by what re-hydration recovered (rehydrated = row re-read · tombstone = delete, primary key only · unrecovered = content lost to taps)." }), be = {
|
|
97
|
+
rehydrated: c.tagged(B, "outcome", "rehydrated"),
|
|
98
|
+
tombstone: c.tagged(B, "outcome", "tombstone"),
|
|
99
|
+
unrecovered: c.tagged(B, "outcome", "unrecovered")
|
|
100
|
+
}, V = (e) => {
|
|
101
|
+
i.runSync(c.increment(be[e]));
|
|
102
|
+
}, xe = {
|
|
103
103
|
rehydrateTimeoutMs: 5e3,
|
|
104
104
|
rehydrateRetries: 2
|
|
105
|
-
},
|
|
105
|
+
}, H = (e) => {
|
|
106
106
|
let t = process.env[e];
|
|
107
107
|
if (t === void 0 || t.trim() === "") return;
|
|
108
108
|
let n = Number(t);
|
|
109
109
|
return Number.isFinite(n) && n >= 0 ? n : void 0;
|
|
110
|
-
},
|
|
111
|
-
let t =
|
|
110
|
+
}, U = (e) => {
|
|
111
|
+
let t = xe, n = e ?? {};
|
|
112
112
|
return {
|
|
113
|
-
rehydrateTimeoutMs: Math.max(1, n.rehydrateTimeoutMs ??
|
|
114
|
-
rehydrateRetries: Math.max(0, n.rehydrateRetries ??
|
|
113
|
+
rehydrateTimeoutMs: Math.max(1, n.rehydrateTimeoutMs ?? H("VOLTRO_CDC_REHYDRATE_TIMEOUT_MS") ?? t.rehydrateTimeoutMs),
|
|
114
|
+
rehydrateRetries: Math.max(0, n.rehydrateRetries ?? H("VOLTRO_CDC_REHYDRATE_RETRIES") ?? t.rehydrateRetries)
|
|
115
115
|
};
|
|
116
|
-
},
|
|
116
|
+
}, Se = (e) => i.suspend(() => {
|
|
117
117
|
let t = e.notification, n = t.id, a = {
|
|
118
118
|
table: t.table,
|
|
119
119
|
op: t.op
|
|
@@ -127,9 +127,9 @@ var A = (e) => {
|
|
|
127
127
|
outcome: "unrecovered",
|
|
128
128
|
reason: e
|
|
129
129
|
});
|
|
130
|
-
if (typeof n != "string" && typeof n != "number") return
|
|
130
|
+
if (typeof n != "string" && typeof n != "number") return V("unrecovered"), i.succeed(o("no-key"));
|
|
131
131
|
if (t.op === "delete") {
|
|
132
|
-
|
|
132
|
+
V("tombstone");
|
|
133
133
|
let e = {
|
|
134
134
|
event: {
|
|
135
135
|
...a,
|
|
@@ -141,12 +141,12 @@ var A = (e) => {
|
|
|
141
141
|
};
|
|
142
142
|
return i.succeed(e);
|
|
143
143
|
}
|
|
144
|
-
let s =
|
|
144
|
+
let s = d.exponential(r.millis(50), 2).pipe(d.intersect(d.recurs(e.tunables.rehydrateRetries)));
|
|
145
145
|
return e.fetchRow({
|
|
146
146
|
schema: t.schema,
|
|
147
147
|
table: t.table,
|
|
148
148
|
key: n
|
|
149
|
-
}).pipe(i.retry(s), i.timeout(r.millis(e.tunables.rehydrateTimeoutMs)), i.map((e) => e === null ? (
|
|
149
|
+
}).pipe(i.retry(s), i.timeout(r.millis(e.tunables.rehydrateTimeoutMs)), i.map((e) => e === null ? (V("unrecovered"), o("row-gone")) : (V("rehydrated"), {
|
|
150
150
|
event: {
|
|
151
151
|
...a,
|
|
152
152
|
old: null,
|
|
@@ -154,8 +154,8 @@ var A = (e) => {
|
|
|
154
154
|
oversized: "rehydrated"
|
|
155
155
|
},
|
|
156
156
|
outcome: "rehydrated"
|
|
157
|
-
})), i.catchAllCause(() => (
|
|
158
|
-
}),
|
|
157
|
+
})), i.catchAllCause(() => (V("unrecovered"), i.succeed(o("read-failed")))));
|
|
158
|
+
}), Ce = (e) => {
|
|
159
159
|
let t = /* @__PURE__ */ new Set();
|
|
160
160
|
return (n, r) => {
|
|
161
161
|
if (r.outcome === "unrecovered") {
|
|
@@ -173,25 +173,25 @@ var A = (e) => {
|
|
|
173
173
|
note: r.outcome === "tombstone" ? "a delete of an oversized row delivers the primary key only — the pre-image is unrecoverable" : "the re-read returns the row as it is NOW, not the image at commit"
|
|
174
174
|
}));
|
|
175
175
|
};
|
|
176
|
-
},
|
|
177
|
-
let t = e.tracerLayer ? o.mergeAll(e.sqlLayer, e.tracerLayer) : e.sqlLayer, n = s.make(t), r = await n.runPromise(O), i = e.changeStrategy ?? "inline", a = new
|
|
176
|
+
}, W = ["json"], G = _e({ scope: "voltro:postgres" }), K = async (e) => {
|
|
177
|
+
let t = e.tracerLayer ? o.mergeAll(e.sqlLayer, e.tracerLayer) : e.sqlLayer, n = s.make(t), r = await n.runPromise(O), i = e.changeStrategy ?? "inline", a = new we(r, n, i, e.cdcChannel ?? "framework_changes", U({
|
|
178
178
|
...e.cdcRehydrateTimeoutMs === void 0 ? {} : { rehydrateTimeoutMs: e.cdcRehydrateTimeoutMs },
|
|
179
179
|
...e.cdcRehydrateRetries === void 0 ? {} : { rehydrateRetries: e.cdcRehydrateRetries }
|
|
180
180
|
}));
|
|
181
181
|
return i === "cdc" && await a.startCdcConsumer(), a;
|
|
182
|
-
},
|
|
182
|
+
}, we = class {
|
|
183
183
|
sql;
|
|
184
184
|
runtime;
|
|
185
185
|
changeStrategy;
|
|
186
186
|
cdcChannel;
|
|
187
187
|
cdcRehydrate;
|
|
188
|
-
emitter = new
|
|
188
|
+
emitter = new ge();
|
|
189
189
|
cdcFiber = null;
|
|
190
190
|
inflightTxns = 0;
|
|
191
|
-
reportOversized =
|
|
192
|
-
reportEagerFallback =
|
|
193
|
-
constructor(e, t, n, r, i =
|
|
194
|
-
this.sql = e, this.runtime = t, this.changeStrategy = n, this.cdcChannel = r, this.cdcRehydrate = i,
|
|
191
|
+
reportOversized = Ce(G);
|
|
192
|
+
reportEagerFallback = de(G);
|
|
193
|
+
constructor(e, t, n, r, i = U()) {
|
|
194
|
+
this.sql = e, this.runtime = t, this.changeStrategy = n, this.cdcChannel = r, this.cdcRehydrate = i, fe(this.emitter);
|
|
195
195
|
}
|
|
196
196
|
txnSpec(e, t, n) {
|
|
197
197
|
return {
|
|
@@ -199,7 +199,7 @@ var A = (e) => {
|
|
|
199
199
|
dialect: "postgres",
|
|
200
200
|
withTransaction: (e) => this.sql.withTransaction(e),
|
|
201
201
|
runPromiseExit: (e) => this.runtime.runPromiseExit(e),
|
|
202
|
-
isRetryable:
|
|
202
|
+
isRetryable: R,
|
|
203
203
|
span: {
|
|
204
204
|
name: t,
|
|
205
205
|
attributes: {
|
|
@@ -210,7 +210,7 @@ var A = (e) => {
|
|
|
210
210
|
};
|
|
211
211
|
}
|
|
212
212
|
withNamespace(e) {
|
|
213
|
-
return e === null ? this : new
|
|
213
|
+
return e === null ? this : new Te(this, e);
|
|
214
214
|
}
|
|
215
215
|
async runInNamespace(e, t) {
|
|
216
216
|
let n = this.sql;
|
|
@@ -220,7 +220,7 @@ var A = (e) => {
|
|
|
220
220
|
...this.txnSpec("PostgresDataStore.runInNamespace", "store.namespace", "namespace.transaction"),
|
|
221
221
|
prepare: (t) => i.provideService(n`SET LOCAL search_path TO ${n(e)}`, k, t),
|
|
222
222
|
work: t,
|
|
223
|
-
makeView: (e, t) => new
|
|
223
|
+
makeView: (e, t) => new q(this, e, t)
|
|
224
224
|
});
|
|
225
225
|
} finally {
|
|
226
226
|
this.inflightTxns--;
|
|
@@ -228,12 +228,12 @@ var A = (e) => {
|
|
|
228
228
|
}
|
|
229
229
|
__postgresReplicationFriend = { runEffect: (e) => this.runtime.runPromise(e) };
|
|
230
230
|
async executeQuery(e, t, n = null) {
|
|
231
|
-
let r =
|
|
231
|
+
let r = se(e, this.sql, n), a = t ? i.provideService(r, k, t) : r;
|
|
232
232
|
return b("postgres", "select", () => this.runtime.runPromise(a));
|
|
233
233
|
}
|
|
234
234
|
async executeInsert(e, t, n, r, a) {
|
|
235
235
|
t = E(e, t);
|
|
236
|
-
let o = this.sql, s = o`INSERT INTO ${o(e)} ${o.insert(v(t, e,
|
|
236
|
+
let o = this.sql, s = o`INSERT INTO ${o(e)} ${o.insert(v(t, e, W))} RETURNING *`, c = n ? i.provideService(s, k, n) : s, l = (await this.runtime.runPromise(c))[0];
|
|
237
237
|
if (!l) throw Error(`PostgresDataStore.insert: no row returned for table '${e}'`);
|
|
238
238
|
return await this.routeEvent({
|
|
239
239
|
table: e,
|
|
@@ -243,7 +243,7 @@ var A = (e) => {
|
|
|
243
243
|
}, r, n, a), l;
|
|
244
244
|
}
|
|
245
245
|
async executeUpdate(e, t, n, r, a, o) {
|
|
246
|
-
let s = this.sql, c = s`UPDATE ${s(e)} SET ${s.update(v(n, e,
|
|
246
|
+
let s = this.sql, c = s`UPDATE ${s(e)} SET ${s.update(v(n, e, W))} WHERE ${s("id")} = ${t} RETURNING *`, l = r ? i.provideService(c, k, r) : c, u = (await this.runtime.runPromise(l))[0];
|
|
247
247
|
return u ? (await this.routeEvent({
|
|
248
248
|
table: e,
|
|
249
249
|
op: "update",
|
|
@@ -261,7 +261,7 @@ var A = (e) => {
|
|
|
261
261
|
if (r) return l(r);
|
|
262
262
|
this.inflightTxns++;
|
|
263
263
|
try {
|
|
264
|
-
return await
|
|
264
|
+
return await pe({
|
|
265
265
|
...this.txnSpec("PostgresDataStore.upsert", "store.upsert", "upsert.transaction"),
|
|
266
266
|
body: (e) => l(e)
|
|
267
267
|
});
|
|
@@ -269,7 +269,7 @@ var A = (e) => {
|
|
|
269
269
|
this.inflightTxns--;
|
|
270
270
|
}
|
|
271
271
|
}
|
|
272
|
-
let c = n.conflictColumns.map((e) => s`${s(e)}`), l = Object.keys(t).filter((e) => t[e] !== void 0), u = n.update === void 0 ? l.filter((e) => e !== "id" && !n.conflictColumns.includes(e)) : n.update, d = u.length > 0 ? s.csv(u.map((e) => s`${s(e)} = EXCLUDED.${s(e)}`)) : s`${s(n.conflictColumns[0])} = EXCLUDED.${s(n.conflictColumns[0])}`, f = s`INSERT INTO ${s(e)} ${s.insert(v(t, e,
|
|
272
|
+
let c = n.conflictColumns.map((e) => s`${s(e)}`), l = Object.keys(t).filter((e) => t[e] !== void 0), u = n.update === void 0 ? l.filter((e) => e !== "id" && !n.conflictColumns.includes(e)) : n.update, d = u.length > 0 ? s.csv(u.map((e) => s`${s(e)} = EXCLUDED.${s(e)}`)) : s`${s(n.conflictColumns[0])} = EXCLUDED.${s(n.conflictColumns[0])}`, f = s`INSERT INTO ${s(e)} ${s.insert(v(t, e, W))} ON CONFLICT (${s.csv(c)}) DO UPDATE SET ${d} RETURNING *`, p = r ? i.provideService(f, k, r) : f, m = (await this.runtime.runPromise(p))[0];
|
|
273
273
|
if (!m) throw Error(`PostgresDataStore.upsert: no row returned for table '${e}'`);
|
|
274
274
|
{
|
|
275
275
|
let n = t.id !== void 0 && t.id === m.id ? "insert" : "update";
|
|
@@ -284,7 +284,7 @@ var A = (e) => {
|
|
|
284
284
|
}
|
|
285
285
|
async executeInsertIgnore(e, t, n, r, a, o) {
|
|
286
286
|
t = E(e, t);
|
|
287
|
-
let s = this.sql, c = n.conflictColumns.map((e) => s`${s(e)}`), l = s`INSERT INTO ${s(e)} ${s.insert(v(t, e,
|
|
287
|
+
let s = this.sql, c = n.conflictColumns.map((e) => s`${s(e)}`), l = s`INSERT INTO ${s(e)} ${s.insert(v(t, e, W))} ON CONFLICT (${s.csv(c)}) DO NOTHING RETURNING *`, u = r ? i.provideService(l, k, r) : l, d = (await this.runtime.runPromise(u))[0];
|
|
288
288
|
if (d) return await this.routeEvent({
|
|
289
289
|
table: e,
|
|
290
290
|
op: "insert",
|
|
@@ -296,8 +296,8 @@ var A = (e) => {
|
|
|
296
296
|
return h[0];
|
|
297
297
|
}
|
|
298
298
|
async executeInsertMany(e, t, n, r, a) {
|
|
299
|
-
if (t =
|
|
300
|
-
let o = this.sql, s = t.map((t) => v(t, e,
|
|
299
|
+
if (t = he(e, t), t.length === 0) return [];
|
|
300
|
+
let o = this.sql, s = t.map((t) => v(t, e, W)), c = ie(s, re("postgres")), l = (t) => o`INSERT INTO ${o(e)} ${o.insert(t)} RETURNING *`, u;
|
|
301
301
|
if (c.length === 1) {
|
|
302
302
|
let e = l(c[0]), t = n ? i.provideService(e, k, n) : e;
|
|
303
303
|
u = await this.runtime.runPromise(t);
|
|
@@ -333,7 +333,7 @@ var A = (e) => {
|
|
|
333
333
|
}, r, n, a), !0) : !1;
|
|
334
334
|
}
|
|
335
335
|
async appendInTxn(e, t, n) {
|
|
336
|
-
let r = this.sql, a = r`INSERT INTO ${r(e)} ${r.insert(v(t, e,
|
|
336
|
+
let r = this.sql, a = r`INSERT INTO ${r(e)} ${r.insert(v(t, e, W))}`;
|
|
337
337
|
await this.runtime.runPromise(n ? i.provideService(a, k, n) : a);
|
|
338
338
|
}
|
|
339
339
|
async maxInTxn(e, t, n, r) {
|
|
@@ -342,9 +342,9 @@ var A = (e) => {
|
|
|
342
342
|
}
|
|
343
343
|
async routeEvent(e, t, n = null, r) {
|
|
344
344
|
if (e = {
|
|
345
|
-
...
|
|
345
|
+
...ee(r),
|
|
346
346
|
...e
|
|
347
|
-
},
|
|
347
|
+
}, x(e.table) && await me({
|
|
348
348
|
append: (e, t) => this.appendInTxn(e, t, n),
|
|
349
349
|
maxOf: (e, t, r) => this.maxInTxn(e, t, r, n)
|
|
350
350
|
}, {
|
|
@@ -357,7 +357,7 @@ var A = (e) => {
|
|
|
357
357
|
}), y(e.table)) {
|
|
358
358
|
if (this.changeStrategy === "cdc") {
|
|
359
359
|
let t = (e.op === "delete" ? e.old : e.new)?.id;
|
|
360
|
-
t != null &&
|
|
360
|
+
t != null && S(te(e.table, e.op, t), {
|
|
361
361
|
...e.traceId === void 0 ? {} : { traceId: e.traceId },
|
|
362
362
|
...e.subjectId === void 0 ? {} : { subjectId: e.subjectId }
|
|
363
363
|
});
|
|
@@ -370,17 +370,17 @@ var A = (e) => {
|
|
|
370
370
|
return this.runWithEager(e, null);
|
|
371
371
|
}
|
|
372
372
|
raw(e, t) {
|
|
373
|
-
let n =
|
|
373
|
+
let n = oe(e, this.sql);
|
|
374
374
|
return b("postgres", "raw", () => this.runtime.runPromise(n));
|
|
375
375
|
}
|
|
376
376
|
async runWithEager(e, t, n = null) {
|
|
377
|
-
if (!
|
|
378
|
-
let r = n === null ?
|
|
377
|
+
if (!ue(e)) return this.executeQuery(e, t, n);
|
|
378
|
+
let r = n === null ? ae(e, this.sql, "postgres") : null;
|
|
379
379
|
if (r !== null) try {
|
|
380
380
|
let e = t ? i.provideService(r.fragment, k, t) : r.fragment, n = await b("postgres", "select", () => this.runtime.runPromise(e));
|
|
381
381
|
return r.decode(n);
|
|
382
382
|
} catch (t) {
|
|
383
|
-
if (t instanceof
|
|
383
|
+
if (t instanceof h) throw t;
|
|
384
384
|
this.reportEagerFallback({
|
|
385
385
|
dialect: "postgres",
|
|
386
386
|
table: e.table,
|
|
@@ -394,7 +394,7 @@ var A = (e) => {
|
|
|
394
394
|
reason: "not-compilable"
|
|
395
395
|
});
|
|
396
396
|
let a = await this.executeQuery(e, t, n);
|
|
397
|
-
return
|
|
397
|
+
return g(a, e.eager, e.sourceTable ?? C(e.table), (e) => this.executeQuery(e, t, n));
|
|
398
398
|
}
|
|
399
399
|
getInternalRunWithEager() {
|
|
400
400
|
return (e, t) => this.runWithEager(e, t);
|
|
@@ -402,40 +402,49 @@ var A = (e) => {
|
|
|
402
402
|
async queryInNamespace(e, t) {
|
|
403
403
|
return this.runWithEager(t, null, e);
|
|
404
404
|
}
|
|
405
|
+
async writeWithRecorders(e, t) {
|
|
406
|
+
let n = [], r = this.sql, a = await this.runtime.runPromise(r.withTransaction(i.flatMap(i.serviceOption(k), (r) => l.isNone(r) ? i.fail(/* @__PURE__ */ Error("PostgresDataStore.write: TransactionConnection missing.")) : i.tryPromise({
|
|
407
|
+
try: () => e(t, r.value, n),
|
|
408
|
+
catch: (e) => e
|
|
409
|
+
}))));
|
|
410
|
+
for (let e of n) this.emitChange(e);
|
|
411
|
+
return a;
|
|
412
|
+
}
|
|
405
413
|
async localWrite(e, t, n) {
|
|
414
|
+
let r = (e) => x(t) ? this.writeWithRecorders(n, e) : n(e, null, null);
|
|
406
415
|
return b("postgres", e, async () => {
|
|
407
|
-
if (this.changeStrategy !== "cdc") return D(
|
|
408
|
-
|
|
416
|
+
if (this.changeStrategy !== "cdc") return D(r);
|
|
417
|
+
ne(t);
|
|
409
418
|
try {
|
|
410
|
-
return await D(
|
|
419
|
+
return await D(r);
|
|
411
420
|
} finally {
|
|
412
|
-
|
|
421
|
+
ce(t);
|
|
413
422
|
}
|
|
414
423
|
});
|
|
415
424
|
}
|
|
416
425
|
insert(e, t) {
|
|
417
|
-
return this.localWrite("insert", e, (n) => this.executeInsert(e, t,
|
|
426
|
+
return this.localWrite("insert", e, (n, r, i) => this.executeInsert(e, t, r, i, n));
|
|
418
427
|
}
|
|
419
428
|
insertMany(e, t) {
|
|
420
|
-
return this.localWrite("insert", e, (n) => this.executeInsertMany(e, t,
|
|
429
|
+
return this.localWrite("insert", e, (n, r, i) => this.executeInsertMany(e, t, r, i, n));
|
|
421
430
|
}
|
|
422
431
|
patchJson(e, t, n, r) {
|
|
423
|
-
return this.localWrite("update", e, (i) => this.executePatchJson(e, t, n, r,
|
|
432
|
+
return this.localWrite("update", e, (i, a, o) => this.executePatchJson(e, t, n, r, a, o, i));
|
|
424
433
|
}
|
|
425
434
|
update(e, t, n) {
|
|
426
|
-
return this.localWrite("update", e, (r) => this.executeUpdate(e, t, n,
|
|
435
|
+
return this.localWrite("update", e, (r, i, a) => this.executeUpdate(e, t, n, i, a, r));
|
|
427
436
|
}
|
|
428
437
|
delete(e, t) {
|
|
429
|
-
return this.localWrite("delete", e, (n) => this.executeDelete(e, t,
|
|
438
|
+
return this.localWrite("delete", e, (n, r, i) => this.executeDelete(e, t, r, i, n));
|
|
430
439
|
}
|
|
431
440
|
async updateMany(e, t, n) {
|
|
432
|
-
return this.localWrite("update", e, (r) => this.executeUpdateMany(e, t, n,
|
|
441
|
+
return this.localWrite("update", e, (r, i, a) => this.executeUpdateMany(e, t, n, i, a, r));
|
|
433
442
|
}
|
|
434
443
|
async deleteMany(e, t) {
|
|
435
|
-
return this.localWrite("delete", e, (n) => this.executeDeleteMany(e, t,
|
|
444
|
+
return this.localWrite("delete", e, (n, r, i) => this.executeDeleteMany(e, t, r, i, n));
|
|
436
445
|
}
|
|
437
446
|
async executeUpdateMany(e, t, n, r, a, o) {
|
|
438
|
-
let s = this.sql, c = _(n.where, s), l = s`UPDATE ${s(e)} SET ${s.update(v(t, e,
|
|
447
|
+
let s = this.sql, c = _(n.where, s), l = s`UPDATE ${s(e)} SET ${s.update(v(t, e, W))} WHERE ${c} RETURNING *`, u = r ? i.provideService(l, k, r) : l, d = await this.runtime.runPromise(u);
|
|
439
448
|
for (let t of d) await this.routeEvent({
|
|
440
449
|
table: e,
|
|
441
450
|
op: "update",
|
|
@@ -455,10 +464,10 @@ var A = (e) => {
|
|
|
455
464
|
return u.length;
|
|
456
465
|
}
|
|
457
466
|
upsert(e, t, n) {
|
|
458
|
-
return this.localWrite("upsert", e, (r) => this.executeUpsert(e, t, n,
|
|
467
|
+
return this.localWrite("upsert", e, (r, i, a) => this.executeUpsert(e, t, n, i, a, r));
|
|
459
468
|
}
|
|
460
469
|
insertIgnore(e, t, n) {
|
|
461
|
-
return this.localWrite("upsert", e, (r) => this.executeInsertIgnore(e, t, n,
|
|
470
|
+
return this.localWrite("upsert", e, (r, i, a) => this.executeInsertIgnore(e, t, n, i, a, r));
|
|
462
471
|
}
|
|
463
472
|
emitChange(e) {
|
|
464
473
|
this.emitter.emit("change", e);
|
|
@@ -528,7 +537,7 @@ var A = (e) => {
|
|
|
528
537
|
return await T({
|
|
529
538
|
...this.txnSpec("PostgresDataStore.transactional", "store.transactional", "transaction"),
|
|
530
539
|
work: e,
|
|
531
|
-
makeView: (e, t) => new
|
|
540
|
+
makeView: (e, t) => new q(this, e, t)
|
|
532
541
|
});
|
|
533
542
|
} finally {
|
|
534
543
|
this.inflightTxns--;
|
|
@@ -545,8 +554,8 @@ var A = (e) => {
|
|
|
545
554
|
injectExternalChange(e) {
|
|
546
555
|
if (!y(e.table)) return;
|
|
547
556
|
let t = (e.op === "delete" ? e.old : e.new)?.id;
|
|
548
|
-
|
|
549
|
-
this.emitter.emit("change",
|
|
557
|
+
w(e.table, e.op, t, (t) => {
|
|
558
|
+
this.emitter.emit("change", le(e, t));
|
|
550
559
|
});
|
|
551
560
|
}
|
|
552
561
|
run(e) {
|
|
@@ -556,7 +565,7 @@ var A = (e) => {
|
|
|
556
565
|
if (this.inflightTxns > 0) {
|
|
557
566
|
let t = Date.now() + e;
|
|
558
567
|
for (; this.inflightTxns > 0 && Date.now() < t;) await new Promise((e) => setTimeout(e, 25));
|
|
559
|
-
this.inflightTxns > 0 &&
|
|
568
|
+
this.inflightTxns > 0 && G.warn("close: grace period expired with in-flight transactions — forcing dispose", {
|
|
560
569
|
gracePeriodMs: e,
|
|
561
570
|
inflight: this.inflightTxns
|
|
562
571
|
});
|
|
@@ -581,7 +590,7 @@ var A = (e) => {
|
|
|
581
590
|
try {
|
|
582
591
|
t = JSON.parse(e);
|
|
583
592
|
} catch (e) {
|
|
584
|
-
return
|
|
593
|
+
return G.warn("cdc: bad payload", { channel: this.cdcChannel }, e), i.void;
|
|
585
594
|
}
|
|
586
595
|
if (t.oversized !== !0) {
|
|
587
596
|
let e = {
|
|
@@ -594,7 +603,7 @@ var A = (e) => {
|
|
|
594
603
|
this.injectExternalChange(e);
|
|
595
604
|
});
|
|
596
605
|
}
|
|
597
|
-
return y(t.table) ?
|
|
606
|
+
return y(t.table) ? Se({
|
|
598
607
|
notification: t,
|
|
599
608
|
fetchRow: ({ schema: e, table: t, key: n }) => this.readRowAsCdcJson(e, t, n),
|
|
600
609
|
tunables: this.cdcRehydrate
|
|
@@ -602,9 +611,9 @@ var A = (e) => {
|
|
|
602
611
|
this.reportOversized(t, e), this.injectExternalChange(e.event);
|
|
603
612
|
})) : i.void;
|
|
604
613
|
});
|
|
605
|
-
this.cdcFiber = this.runtime.runFork(e.pipe(
|
|
614
|
+
this.cdcFiber = this.runtime.runFork(e.pipe(f.runForEach(n)));
|
|
606
615
|
}
|
|
607
|
-
},
|
|
616
|
+
}, q = class {
|
|
608
617
|
parent;
|
|
609
618
|
txn;
|
|
610
619
|
attr;
|
|
@@ -673,7 +682,7 @@ var A = (e) => {
|
|
|
673
682
|
this.events.length = 0;
|
|
674
683
|
}
|
|
675
684
|
}
|
|
676
|
-
},
|
|
685
|
+
}, Te = class {
|
|
677
686
|
parent;
|
|
678
687
|
namespace;
|
|
679
688
|
constructor(e, t) {
|
|
@@ -727,14 +736,14 @@ var A = (e) => {
|
|
|
727
736
|
return r === void 0 ? Promise.reject(/* @__PURE__ */ Error("PostgresNamespaceView.raw: underlying store has no raw()")) : r(e, t);
|
|
728
737
|
});
|
|
729
738
|
}
|
|
730
|
-
},
|
|
739
|
+
}, J = (e) => e.__postgresReplicationFriend ?? null, Ee = (e, t) => {
|
|
731
740
|
let [n, r] = e.split("/"), [i, a] = t.split("/");
|
|
732
741
|
if (!n || !r || !i || !a) throw Error(`postgres LSN compare: invalid format (${e} vs ${t})`);
|
|
733
742
|
let o = parseInt(n, 16), s = parseInt(r, 16), c = parseInt(i, 16);
|
|
734
743
|
return o === c ? s - parseInt(a, 16) : o - c;
|
|
735
|
-
},
|
|
744
|
+
}, Y = () => ({
|
|
736
745
|
async capturePrimaryPosition(e) {
|
|
737
|
-
let t =
|
|
746
|
+
let t = J(e);
|
|
738
747
|
if (t === null) throw Error("postgresReplicationAdapter: primary is not a PostgresDataStore (missing __postgresReplicationFriend). Pass the postgres store directly.");
|
|
739
748
|
return t.runEffect(i.gen(function* () {
|
|
740
749
|
let e = (yield* (yield* O)`SELECT pg_current_wal_lsn()::text AS lsn`)[0]?.lsn;
|
|
@@ -742,7 +751,7 @@ var A = (e) => {
|
|
|
742
751
|
}));
|
|
743
752
|
},
|
|
744
753
|
async probeReplicaPosition(e) {
|
|
745
|
-
let t =
|
|
754
|
+
let t = J(e);
|
|
746
755
|
if (t === null) throw Error("postgresReplicationAdapter: replica is not a PostgresDataStore.");
|
|
747
756
|
return t.runEffect(i.gen(function* () {
|
|
748
757
|
let e = (yield* (yield* O)`
|
|
@@ -752,16 +761,16 @@ var A = (e) => {
|
|
|
752
761
|
}));
|
|
753
762
|
},
|
|
754
763
|
compare(e, t) {
|
|
755
|
-
return
|
|
764
|
+
return Ee(t, e) >= 0 ? "caught-up" : "behind";
|
|
756
765
|
}
|
|
757
|
-
}), X = (e) => `"${e.replace(/"/g, "\"\"")}"`,
|
|
766
|
+
}), X = (e) => `"${e.replace(/"/g, "\"\"")}"`, De = (e) => e.replace(/\\/g, "\\\\").replace(/\t/g, "\\t").replace(/\n/g, "\\n").replace(/\r/g, "\\r").replace(/\v/g, "\\v").replace(/\f/g, "\\f").replace(/[\b]/g, "\\b"), Oe = (e) => {
|
|
758
767
|
let t = "";
|
|
759
768
|
for (let n of e) t += n.toString(16).padStart(2, "0");
|
|
760
769
|
return t;
|
|
761
|
-
}, Z = (e) => `{${e.map((e) => e == null ? "NULL" : Array.isArray(e) ? Z(e) : typeof e == "number" || typeof e == "bigint" ? String(e) : typeof e == "boolean" ? e ? "t" : "f" : `"${(e instanceof Date ? e.toISOString() : String(e)).replace(/\\/g, "\\\\").replace(/"/g, "\\\"")}"`).join(",")}}`, Q = (e, t) => e == null ? null : typeof e == "string" ? e : typeof e == "number" || typeof e == "bigint" ? String(e) : typeof e == "boolean" ? e ? "t" : "f" : e instanceof Date ? e.toISOString() : e instanceof Uint8Array ? `\\x${
|
|
770
|
+
}, Z = (e) => `{${e.map((e) => e == null ? "NULL" : Array.isArray(e) ? Z(e) : typeof e == "number" || typeof e == "bigint" ? String(e) : typeof e == "boolean" ? e ? "t" : "f" : `"${(e instanceof Date ? e.toISOString() : String(e)).replace(/\\/g, "\\\\").replace(/"/g, "\\\"")}"`).join(",")}}`, Q = (e, t) => e == null ? null : typeof e == "string" ? e : typeof e == "number" || typeof e == "bigint" ? String(e) : typeof e == "boolean" ? e ? "t" : "f" : e instanceof Date ? e.toISOString() : e instanceof Uint8Array ? `\\x${Oe(e)}` : Array.isArray(e) && t === "array" ? Z(e) : JSON.stringify(e), $ = (e, t, n = {}) => t.map((t) => {
|
|
762
771
|
let r = Q(e[t], n[t]);
|
|
763
|
-
return r === null ? "\\N" :
|
|
764
|
-
}).join(" "),
|
|
772
|
+
return r === null ? "\\N" : De(r);
|
|
773
|
+
}).join(" "), ke = class {
|
|
765
774
|
text;
|
|
766
775
|
chunks;
|
|
767
776
|
done;
|
|
@@ -792,7 +801,7 @@ var A = (e) => {
|
|
|
792
801
|
handleReadyForQuery() {
|
|
793
802
|
this.settle(void 0);
|
|
794
803
|
}
|
|
795
|
-
},
|
|
804
|
+
}, Ae = (e) => ({
|
|
796
805
|
...e.url ? { connectionString: e.url } : {
|
|
797
806
|
host: e.host ?? "localhost",
|
|
798
807
|
port: e.port ?? 5432,
|
|
@@ -801,9 +810,9 @@ var A = (e) => {
|
|
|
801
810
|
database: e.database ?? "postgres"
|
|
802
811
|
},
|
|
803
812
|
...e.ssl === !0 ? { ssl: { rejectUnauthorized: !1 } } : {}
|
|
804
|
-
}),
|
|
813
|
+
}), je = 65536, Me = (e) => {
|
|
805
814
|
let t, n, r = () => t ? Promise.resolve(t) : (n ??= (async () => {
|
|
806
|
-
let n = new
|
|
815
|
+
let n = new p.Client(Ae(e));
|
|
807
816
|
return await n.connect(), t = n, n;
|
|
808
817
|
})(), n);
|
|
809
818
|
return {
|
|
@@ -811,9 +820,9 @@ var A = (e) => {
|
|
|
811
820
|
copyInto: async ({ table: e, columns: t, columnTypes: n, rows: i }) => {
|
|
812
821
|
if (i.length === 0) return 0;
|
|
813
822
|
let a = await r(), o = `COPY ${X(e)} (${t.map(X).join(", ")}) FROM STDIN`, s = [], c = "";
|
|
814
|
-
for (let e of i) c += $(e, t, n ?? {}), c += "\n", c.length >=
|
|
823
|
+
for (let e of i) c += $(e, t, n ?? {}), c += "\n", c.length >= je && (s.push(Buffer.from(c, "utf8")), c = "");
|
|
815
824
|
return c.length > 0 && s.push(Buffer.from(c, "utf8")), await new Promise((e, t) => {
|
|
816
|
-
let n = new
|
|
825
|
+
let n = new ke(o, s, (n, r) => {
|
|
817
826
|
n ? t(n) : e(r);
|
|
818
827
|
});
|
|
819
828
|
a.query(n);
|
|
@@ -824,12 +833,12 @@ var A = (e) => {
|
|
|
824
833
|
t = void 0, n = void 0, e && await e.end().catch(() => void 0);
|
|
825
834
|
}
|
|
826
835
|
};
|
|
827
|
-
},
|
|
836
|
+
}, Ne = {
|
|
828
837
|
id: "postgres",
|
|
829
838
|
makeSqlLayer: (e) => L(e),
|
|
830
|
-
makeStore: (e) =>
|
|
839
|
+
makeStore: (e) => K(e),
|
|
831
840
|
compileContains: (e, t, n) => n`${e} ILIKE ${`%${t.replace(/[\\%_]/g, (e) => `\\${e}`)}%`} ESCAPE '\\'`,
|
|
832
|
-
retryFilter:
|
|
841
|
+
retryFilter: z
|
|
833
842
|
};
|
|
834
843
|
//#endregion
|
|
835
|
-
export { e as PgClient, I as connectionFromConfig, $ as encodeCopyRow, Q as encodeCopyValue,
|
|
844
|
+
export { e as PgClient, I as connectionFromConfig, $ as encodeCopyRow, Q as encodeCopyValue, Me as makePgCopySession, K as makePostgresDataStore, P as makePostgresSqlLayer, L as makePostgresSqlLayerFromConfig, Ne as postgresDialect, Y as postgresReplicationAdapter, z as postgresRetryFilter };
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@voltro/sql-postgres",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.44.0",
|
|
4
4
|
"description": "PostgreSQL dialect adapter for Voltro's cross-dialect DataStore (LISTEN/NOTIFY reactivity, logical-replication CDC).",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"voltro",
|
|
@@ -35,8 +35,8 @@
|
|
|
35
35
|
"dependencies": {
|
|
36
36
|
"@effect/sql": "^0.52.0",
|
|
37
37
|
"@effect/sql-pg": "^0.53.0",
|
|
38
|
-
"@voltro/database": "0.
|
|
39
|
-
"@voltro/logger": "0.
|
|
38
|
+
"@voltro/database": "0.44.0",
|
|
39
|
+
"@voltro/logger": "0.44.0",
|
|
40
40
|
"pg": "^8.23.0"
|
|
41
41
|
},
|
|
42
42
|
"peerDependencies": {
|