postgrejs 3.3.0 → 3.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +38 -1
- package/README.md +56 -6
- package/connection/connection.d.ts +97 -0
- package/connection/connection.js +141 -2
- package/connection/intl-connection.d.ts +113 -3
- package/connection/intl-connection.js +520 -31
- package/connection/prepared-statement.d.ts +64 -1
- package/connection/prepared-statement.js +95 -5
- package/data-types/float4-type.js +2 -1
- package/data-types/float8-type.js +2 -1
- package/data-types/int2-type.js +2 -1
- package/data-types/int4-type.js +2 -1
- package/data-types/int8-type.js +2 -1
- package/data-types/numeric-type.js +7 -0
- package/data-types/oid-type.js +2 -1
- package/index.d.ts +3 -0
- package/index.js +3 -0
- package/interfaces/batch-result.d.ts +42 -0
- package/interfaces/batch-result.js +1 -0
- package/interfaces/copy-from-rows-options.d.ts +58 -0
- package/interfaces/copy-from-rows-options.js +1 -0
- package/interfaces/database-connection-params.d.ts +25 -0
- package/interfaces/pipeline-request.d.ts +16 -0
- package/interfaces/pipeline-request.js +1 -0
- package/interfaces/query-options.d.ts +6 -0
- package/package.json +5 -2
- package/protocol/database-error.d.ts +16 -0
- package/protocol/database-error.js +15 -0
- package/protocol/pg-socket.d.ts +76 -0
- package/protocol/pg-socket.js +124 -12
- package/util/assert-integer.d.ts +43 -0
- package/util/assert-integer.js +58 -0
- package/util/copy-from-rows.d.ts +56 -0
- package/util/copy-from-rows.js +95 -0
- package/util/encode-copy-binary.d.ts +48 -0
- package/util/encode-copy-binary.js +112 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,6 +1,43 @@
|
|
|
1
1
|
## Changelog
|
|
2
2
|
|
|
3
|
-
### [v3.
|
|
3
|
+
### [v3.4.0](https://github.com/panates/postgrejs/compare/v3.3.0...v3.4.0) -
|
|
4
|
+
|
|
5
|
+
- feat: add PreparedStatement.executeBatch() for multi-set execution
|
|
6
|
+
|
|
7
|
+
#### 🚀 New Features
|
|
8
|
+
|
|
9
|
+
- feat: add copyFromRows() for binary COPY bulk loading @Eray Hanoğlu
|
|
10
|
+
- feat: add Connection.pipeline() for many statements in one round trip @Eray Hanoğlu
|
|
11
|
+
- feat: reuse prepared statements for repeated queries @Eray Hanoğlu
|
|
12
|
+
|
|
13
|
+
#### 🪲 Fixes
|
|
14
|
+
|
|
15
|
+
- fix: refuse values that have no numeric reading instead of storing zero @Eray Hanoğlu
|
|
16
|
+
|
|
17
|
+
#### 📖 Documentation Changes
|
|
18
|
+
|
|
19
|
+
- docs: regenerate both benchmark reports from an interleaved run @Eray Hanoğlu
|
|
20
|
+
- docs: refresh the Node report and correct a stale asymmetry note @Eray Hanoğlu
|
|
21
|
+
- docs: record statement-level rollback in the README @Eray Hanoğlu
|
|
22
|
+
- docs: record the prepared statement cache in the README @Eray Hanoğlu
|
|
23
|
+
- docs: list pipeline() in the feature list and comparison table @Eray Hanoğlu
|
|
24
|
+
- docs: list binary COPY in the feature list and comparison table @Eray Hanoğlu
|
|
25
|
+
- docs: list batch execution in the feature list and comparison table @Eray Hanoğlu
|
|
26
|
+
|
|
27
|
+
#### ⚡️ Performance Improvments
|
|
28
|
+
|
|
29
|
+
- perf: carry the rollbackOnError savepoint in the statement's own round trip @Eray Hanoğlu
|
|
30
|
+
- perf: let the prepared statement cache serve queries inside a transaction @Eray Hanoğlu
|
|
31
|
+
|
|
32
|
+
#### 💬 General Changes
|
|
33
|
+
|
|
34
|
+
- bench: add Bulk Load scenarios for text and binary COPY @Eray Hanoğlu
|
|
35
|
+
- bench: add Unit of Work scenario for pipeline() @Eray Hanoğlu
|
|
36
|
+
- bench: stop measuring postgres.js on an unprepared path @Eray Hanoğlu
|
|
37
|
+
- bench: interleave repeats instead of running them back to back @Eray Hanoğlu
|
|
38
|
+
- ci: pin GITHUB_TOKEN to read-only in the test and qc workflows @Eray Hanoğlu
|
|
39
|
+
|
|
40
|
+
### [v3.3.0](https://github.com/panates/postgrejs/compare/v3.2.0...v3.3.0) - 14 September 2026
|
|
4
41
|
|
|
5
42
|
#### 🚀 New Features
|
|
6
43
|
|
package/README.md
CHANGED
|
@@ -23,8 +23,8 @@ onto a client meant for text.
|
|
|
23
23
|
The numbers back it up. In PostgreJS's own benchmark suite - run head-to-head against
|
|
24
24
|
[`pg`](https://github.com/brianc/node-postgres) (node-postgres) and [`postgres`](https://github.com/porsager/postgres)
|
|
25
25
|
(postgres.js) on identical workloads - PostgreJS opens a connection up to **3x faster** than postgres.js, pushes
|
|
26
|
-
pooled queries through up to **6.
|
|
27
|
-
|
|
26
|
+
pooled queries through up to **6.5x faster** than pg, and fetches large result sets nearly **5x faster** than pg
|
|
27
|
+
too. It's also the only one of the three Node drivers with a complete binary wire protocol across every data
|
|
28
28
|
type, rather than falling back to text for most of them. See [`doc/BENCHMARKS.md`](doc/BENCHMARKS.md) for the full
|
|
29
29
|
methodology and every scenario.
|
|
30
30
|
|
|
@@ -74,8 +74,17 @@ usage.
|
|
|
74
74
|
resource management.
|
|
75
75
|
- **Binary Wire Protocol:** Implements the full binary wire protocol for all PostgreSQL data types, ensuring robust and
|
|
76
76
|
efficient data handling.
|
|
77
|
-
- **Prepared Statements:** Named prepared statements for optimized query execution
|
|
77
|
+
- **Prepared Statements:** Named prepared statements for optimized query execution, and a per-connection cache that
|
|
78
|
+
reuses one automatically for SQL the connection has run before - a repeated query costs `Bind`/`Execute` instead of
|
|
79
|
+
parsing again, worth 3.2x on fifty concurrent calls and 2.9x again on a query inside an open transaction. On by
|
|
80
|
+
default, `prepare: false` to opt out.
|
|
81
|
+
- **Statement-Level Rollback:** Inside a transaction each statement runs under a savepoint of its own, so a failed
|
|
82
|
+
statement leaves the transaction usable instead of aborting the whole block. The `SAVEPOINT`/`RELEASE` pair travels
|
|
83
|
+
in the statement's own round trip rather than costing one each - 36µs for the pair, against 507µs sent separately.
|
|
84
|
+
`rollbackOnError: false` to opt out.
|
|
78
85
|
- **Cursors:** Features fast double-link cache cursors for efficient data retrieval.
|
|
86
|
+
- **Batch Execution:** `executeBatch()` runs one prepared statement over many parameter sets under a single `Sync`,
|
|
87
|
+
reporting each set's row count - 1000 updates in 20ms where the same calls pipelined individually take 188ms.
|
|
79
88
|
- **Notifications:** High-level implementation for PostgreSQL notifications (LISTEN/NOTIFY), enabling real-time data
|
|
80
89
|
updates.
|
|
81
90
|
- **Extensibility:** Extensible data-types and type mapping to accommodate custom requirements.
|
|
@@ -84,8 +93,13 @@ usage.
|
|
|
84
93
|
- **Performance Optimization:** Low memory utilization and boosted performance through the use of shared buffers.
|
|
85
94
|
- **Authorization:** Supports various password algorithms including Clear text, MD5, and SASL, ensuring secure
|
|
86
95
|
authentication.
|
|
87
|
-
- **Bulk Import/Export:** `COPY TO STDOUT` and `COPY FROM STDIN` as Node streams, with backpressure in both directions
|
|
96
|
+
- **Bulk Import/Export:** `COPY TO STDOUT` and `COPY FROM STDIN` as Node streams, with backpressure in both directions,
|
|
97
|
+
plus `copyFromRows()`, which encodes rows straight into binary `COPY` - around 4x faster than the CSV equivalent and
|
|
98
|
+
no text escaping to get wrong. Takes arrays or objects from anything iterable, so a file larger than memory streams in.
|
|
88
99
|
- **Query Pipelining:** Pooled queries can share connections so a burst is not capped by pool size - opt-in per call.
|
|
100
|
+
`pipeline()` goes further for a known set of statements: several different ones travel under a single `Sync`, so they
|
|
101
|
+
cost one round trip instead of one each and commit or roll back together - around 2x faster than the same calls
|
|
102
|
+
through `Promise.all()`, which is already pipelined.
|
|
89
103
|
- **Dynamic SQL:** A `sql` tag builds statements from composable fragments - values become parameters, names are quoted,
|
|
90
104
|
and `sql.values()`/`sql.set()` write INSERT and UPDATE clauses from objects.
|
|
91
105
|
- **Multiple Hosts:** A connection can list several servers and pick one by role
|
|
@@ -114,7 +128,7 @@ usage.
|
|
|
114
128
|
|
|
115
129
|
How PostgreJS compares to [`pg`](https://github.com/brianc/node-postgres) (node-postgres) and
|
|
116
130
|
[`postgres`](https://github.com/porsager/postgres) (postgres.js). Every row was checked against the libraries' own
|
|
117
|
-
source rather than their documentation — versions compared: **PostgreJS 3.
|
|
131
|
+
source rather than their documentation — versions compared: **PostgreJS 3.4.0, pg 8.23.0, postgres.js 3.4.9**. ✅ built
|
|
118
132
|
in · 🟡 partial or needs a separate package · ❌ not supported.
|
|
119
133
|
|
|
120
134
|
| Feature | PostgreJS | pg | postgres.js |
|
|
@@ -146,10 +160,14 @@ in · 🟡 partial or needs a separate package · ❌ not supported.
|
|
|
146
160
|
| ***Querying*** | | | |
|
|
147
161
|
| Query parameters | ✅ | ✅ | ✅ |
|
|
148
162
|
| Parameter type casting | ✅ | 🟡 <sup>10</sup> | ✅ |
|
|
149
|
-
| Prepared statements
|
|
163
|
+
| Prepared statements <sup>24</sup> | ✅ automatic | ✅ manual | ✅ automatic |
|
|
164
|
+
| Statement-level rollback <sup>25</sup> | ✅ automatic | ❌ | 🟡 manual |
|
|
165
|
+
| Batch execution <sup>21</sup> | ✅ | ❌ | ❌ |
|
|
166
|
+
| Multi-statement round trip <sup>23</sup> | ✅ | ❌ | ❌ |
|
|
150
167
|
| Multi-statement scripts | ✅ | ✅ | ✅ |
|
|
151
168
|
| Server-side cursors | ✅ | 🟡 <sup>11</sup> | ✅ |
|
|
152
169
|
| `COPY TO` / `COPY FROM` | ✅ | 🟡 <sup>12</sup> | ✅ |
|
|
170
|
+
| Binary COPY encoding <sup>22</sup> | ✅ | ❌ | ❌ |
|
|
153
171
|
| Row count after a COPY | ✅ | ✅ | ❌ |
|
|
154
172
|
| ***Transaction management*** | | | |
|
|
155
173
|
| Transaction API | ✅ | ❌ | ✅ |
|
|
@@ -215,6 +233,38 @@ in · 🟡 partial or needs a separate package · ❌ not supported.
|
|
|
215
233
|
documentation lists what stops working with it: `pg-cursor`, `pg-query-stream` and
|
|
216
234
|
`pg-copy-streams` all "operate directly on the binary stream and therefore are
|
|
217
235
|
incompatible" - so server-side cursors, row streaming and COPY are what it costs.
|
|
236
|
+
- <sup>21</sup> One statement executed over many parameter sets behind a single `Sync`, each set's row count reported
|
|
237
|
+
separately. pg and postgres.js both emit a `Sync` per execution (`syncBuffer` in pg's `connection.js`, the
|
|
238
|
+
concatenated `ExecuteUnnamed` in postgres.js's), so a burst of executions costs a server round of implicit-transaction
|
|
239
|
+
bookkeeping each, pipelined or not. Writing one multi-row statement by hand is a separate approach that all three
|
|
240
|
+
support, and both PostgreJS and postgres.js ship value builders for it - it is faster still where it applies, but it
|
|
241
|
+
is one statement rather than many, and PostgreSQL's 65535-parameter ceiling bounds it.
|
|
242
|
+
- <sup>22</sup> Turning JavaScript rows into the binary `COPY` format, rather than carrying a payload the caller
|
|
243
|
+
formatted first. All three can carry a `COPY` stream, but only as bytes: postgres.js hands back a `Writable` that
|
|
244
|
+
wraps raw chunks in `CopyData`, and pg reaches the same point through `pg-copy-streams`. Producing the format needs a
|
|
245
|
+
binary encoder per type, which neither driver has - see Binary encoders above, where the same gap shows up for query
|
|
246
|
+
parameters. The capability is still reachable with pg through a third package, `pg-copy-streams-binary`, which brings
|
|
247
|
+
its own encoders; postgres.js has no equivalent.
|
|
248
|
+
- <sup>23</sup> Several different statements sent under one `Sync`, so they cost a single round of the server's
|
|
249
|
+
transaction bookkeeping rather than one each, arrive as one implicit transaction, and come back as per-statement
|
|
250
|
+
results. Distinct from the Pipelining row above, which is about not waiting between queries: all three do that, and
|
|
251
|
+
all three still close every statement with its own `Sync` - pg sends one immediately after each Execute
|
|
252
|
+
(`query.js`), and postgres.js concatenates Execute and Sync into a single constant (`ExecuteUnnamed`). Atomicity on
|
|
253
|
+
its own is reachable anywhere with an explicit `BEGIN`/`COMMIT`; what that cannot recover is the round trip, since
|
|
254
|
+
the per-statement Syncs and two extra statements remain.
|
|
255
|
+
- <sup>24</sup> Whether a repeated query is parsed again every time. PostgreJS and postgres.js both keep a per-connection
|
|
256
|
+
cache keyed on the SQL and reuse a server-side statement, so a repeat costs `Bind`/`Execute` rather than
|
|
257
|
+
`Parse`/`Bind`/`Describe`/`Execute`; pg prepares only the statements you name yourself (`query.js`: "named queries
|
|
258
|
+
must always be prepared"), with nothing caching by SQL text. The two automatic ones differ in when they start:
|
|
259
|
+
postgres.js on first sight, PostgreJS on the second use, which leaves a genuinely one-shot query at its unprepared
|
|
260
|
+
cost. Both default to on and both take `prepare: false`, which matters for PgBouncer in transaction pooling mode
|
|
261
|
+
before 1.21, where a named statement does not survive to the next call.
|
|
262
|
+
- <sup>25</sup> Whether one failed statement inside a transaction leaves the rest of the block runnable. PostgreJS puts
|
|
263
|
+
every statement under a savepoint of its own by default (`rollbackOnError`), rolling back to it when the statement
|
|
264
|
+
fails, and carries the `SAVEPOINT`/`RELEASE` pair inside the statement's own round trip rather than paying a round
|
|
265
|
+
trip for each. postgres.js has savepoints, but as a scope the caller opens around a callback (`savepoint(name, fn)`
|
|
266
|
+
in `src/index.js`), so a statement is protected only where someone wrapped it; pg has no savepoint handling in
|
|
267
|
+
`lib/` at all, so a failed statement leaves the block aborted until the caller rolls back themselves.
|
|
218
268
|
|
|
219
269
|
|
|
220
270
|
## Benchmarks
|
|
@@ -1,7 +1,9 @@
|
|
|
1
1
|
import { ConnectionState } from '../constants.js';
|
|
2
|
+
import type { CopyFromRowsOptions, CopyFromRowsResult } from '../interfaces/copy-from-rows-options.js';
|
|
2
3
|
import type { ConnectionConfiguration } from '../interfaces/database-connection-params.js';
|
|
3
4
|
import type { FunctionCallOptions } from '../interfaces/function-call-options.js';
|
|
4
5
|
import type { FunctionCallResult } from '../interfaces/function-call-result.js';
|
|
6
|
+
import type { PipelineRequest } from '../interfaces/pipeline-request.js';
|
|
5
7
|
import type { QueryOptions } from '../interfaces/query-options.js';
|
|
6
8
|
import type { QueryResult } from '../interfaces/query-result.js';
|
|
7
9
|
import type { ScriptExecuteOptions } from '../interfaces/script-execute-options.js';
|
|
@@ -11,6 +13,7 @@ import type { DatabaseError } from '../protocol/database-error.js';
|
|
|
11
13
|
import type { Protocol } from '../protocol/protocol.js';
|
|
12
14
|
import { SafeEventEmitter } from '../safe-event-emitter.js';
|
|
13
15
|
import type { Maybe, OID } from '../types.js';
|
|
16
|
+
import type { CopyRowSource } from '../util/copy-from-rows.js';
|
|
14
17
|
import { QueryRequest } from '../util/sql-tag.js';
|
|
15
18
|
import type { CopyFromStream, CopyToStream } from './copy-stream.js';
|
|
16
19
|
import { IntlConnection } from './intl-connection.js';
|
|
@@ -136,6 +139,100 @@ export declare class Connection extends SafeEventEmitter implements AsyncDisposa
|
|
|
136
139
|
*
|
|
137
140
|
* @param sql {string} - A COPY ... FROM STDIN statement
|
|
138
141
|
*/
|
|
142
|
+
/**
|
|
143
|
+
* Bulk-loads rows into a table with `COPY ... FROM STDIN (FORMAT binary)`,
|
|
144
|
+
* encoding every value with its own type's binary encoder instead of
|
|
145
|
+
* making the caller format a text payload.
|
|
146
|
+
*
|
|
147
|
+
* ```ts
|
|
148
|
+
* const { rowCount } = await connection.copyFromRows('users', [
|
|
149
|
+
* [1, 'John', 10.5],
|
|
150
|
+
* [2, 'Jane', 20.0],
|
|
151
|
+
* ], { columns: ['id', 'name', 'amount'] });
|
|
152
|
+
* ```
|
|
153
|
+
*
|
|
154
|
+
* Rows may be positional arrays or objects read by column name, and the
|
|
155
|
+
* source may be anything iterable - including an async iterable or a
|
|
156
|
+
* Readable, so a file far larger than memory streams straight in:
|
|
157
|
+
*
|
|
158
|
+
* ```ts
|
|
159
|
+
* await connection.copyFromRows('users', (async function* () {
|
|
160
|
+
* for await (const { value } of jsonRows) yield [value.id, value.name];
|
|
161
|
+
* })(), { columns: ['id', 'name'] });
|
|
162
|
+
* ```
|
|
163
|
+
*
|
|
164
|
+
* Rows are pulled rather than pushed, so the source only produces the
|
|
165
|
+
* next row once the previous chunk has reached the socket - backpressure
|
|
166
|
+
* is the loop pausing, not a queue growing.
|
|
167
|
+
*
|
|
168
|
+
* Binary is both faster and cheaper to produce than text: 200,000 rows of
|
|
169
|
+
* int4/text/float8/timestamptz measured 232ms against 483ms for the
|
|
170
|
+
* equivalent CSV `copyFrom()`, client-side encoding included, because
|
|
171
|
+
* writing an int32 costs less than formatting a decimal string.
|
|
172
|
+
*
|
|
173
|
+
* Column types are read from the server (one Describe round trip) unless
|
|
174
|
+
* `columnTypes` supplies them. They have to be exact - binary COPY does
|
|
175
|
+
* no conversion - which is also why a column whose type has no binary
|
|
176
|
+
* encoder is rejected before any row is sent rather than mid-stream.
|
|
177
|
+
*
|
|
178
|
+
* `copyFrom()` remains the way to send an already-formatted text or CSV
|
|
179
|
+
* payload.
|
|
180
|
+
*
|
|
181
|
+
* A value the column's type cannot encode - `'abc'` for an integer
|
|
182
|
+
* column - aborts the copy by default, naming the row and column;
|
|
183
|
+
* `onInvalidValue` can instead null the value or drop the row, and the
|
|
184
|
+
* result says how many that happened to. NaN and Infinity in a float or
|
|
185
|
+
* numeric column are not invalid: PostgreSQL stores them as values
|
|
186
|
+
* distinct from NULL, so they go through untouched.
|
|
187
|
+
*
|
|
188
|
+
* @returns Rows sent, plus what a tolerant `onInvalidValue` swallowed.
|
|
189
|
+
*/
|
|
190
|
+
/**
|
|
191
|
+
* Runs several different statements in one round trip: every
|
|
192
|
+
* Parse/Bind/Describe/Execute goes out before any response is waited
|
|
193
|
+
* for, and a single Sync closes the lot.
|
|
194
|
+
*
|
|
195
|
+
* ```ts
|
|
196
|
+
* const [renamed, , total] = await connection.pipeline([
|
|
197
|
+
* sql`update users set name = ${name} where id = ${id}`,
|
|
198
|
+
* sql`insert into audit(msg) values (${msg})`,
|
|
199
|
+
* sql`select count(*)::int as n from users`,
|
|
200
|
+
* ]);
|
|
201
|
+
* renamed.rowsAffected; // 1
|
|
202
|
+
* total.rows?.[0]; // [42]
|
|
203
|
+
* ```
|
|
204
|
+
*
|
|
205
|
+
* The counterpart to `PreparedStatement.executeBatch()`, which runs one
|
|
206
|
+
* statement over many parameter sets; this runs many statements once
|
|
207
|
+
* each. Twenty statements measured 2.3ms against 5.0ms for the same
|
|
208
|
+
* calls through `Promise.all()`, which PostgreJS already pipelines -
|
|
209
|
+
* what the single Sync removes is the server finishing an implicit
|
|
210
|
+
* transaction and answering ReadyForQuery twenty times over, which is
|
|
211
|
+
* why socket reads drop from twenty to one.
|
|
212
|
+
*
|
|
213
|
+
* Three things follow from that framing rather than from choices made
|
|
214
|
+
* above it:
|
|
215
|
+
*
|
|
216
|
+
* - **One transaction.** Unless an explicit transaction is already open,
|
|
217
|
+
* the statements commit or roll back together.
|
|
218
|
+
* - **A failing statement stops the rest.** PostgreSQL discards
|
|
219
|
+
* everything between an error and the Sync, so statements after a
|
|
220
|
+
* rejected one never run. The error carries `failedIndex`.
|
|
221
|
+
* - **No statement can see another's results.** They are all sent before
|
|
222
|
+
* any reply arrives, so anything conditional on an earlier result
|
|
223
|
+
* belongs in a separate call rather than here.
|
|
224
|
+
*
|
|
225
|
+
* Results map to statements by position. Statements that return rows get
|
|
226
|
+
* them decoded as `query()` would, `rowDecoder` included; `fetchCount`
|
|
227
|
+
* does not apply, since a portal suspended mid-pipeline would break that
|
|
228
|
+
* mapping.
|
|
229
|
+
*
|
|
230
|
+
* @param requests Statements to run, as `sql` tag output, plain SQL
|
|
231
|
+
* strings, or `{ sql, params }` objects.
|
|
232
|
+
* @param options Applied to every statement.
|
|
233
|
+
*/
|
|
234
|
+
pipeline(requests: (string | QueryRequest | PipelineRequest)[], options?: QueryOptions): Promise<QueryResult[]>;
|
|
235
|
+
copyFromRows(table: string, source: CopyRowSource, options?: CopyFromRowsOptions): Promise<CopyFromRowsResult>;
|
|
139
236
|
copyFrom(sql: string): Promise<CopyFromStream>;
|
|
140
237
|
/**
|
|
141
238
|
* Creates a PreparedStatement instance
|
package/connection/connection.js
CHANGED
|
@@ -251,6 +251,139 @@ export class Connection extends SafeEventEmitter {
|
|
|
251
251
|
*
|
|
252
252
|
* @param sql {string} - A COPY ... FROM STDIN statement
|
|
253
253
|
*/
|
|
254
|
+
/**
|
|
255
|
+
* Bulk-loads rows into a table with `COPY ... FROM STDIN (FORMAT binary)`,
|
|
256
|
+
* encoding every value with its own type's binary encoder instead of
|
|
257
|
+
* making the caller format a text payload.
|
|
258
|
+
*
|
|
259
|
+
* ```ts
|
|
260
|
+
* const { rowCount } = await connection.copyFromRows('users', [
|
|
261
|
+
* [1, 'John', 10.5],
|
|
262
|
+
* [2, 'Jane', 20.0],
|
|
263
|
+
* ], { columns: ['id', 'name', 'amount'] });
|
|
264
|
+
* ```
|
|
265
|
+
*
|
|
266
|
+
* Rows may be positional arrays or objects read by column name, and the
|
|
267
|
+
* source may be anything iterable - including an async iterable or a
|
|
268
|
+
* Readable, so a file far larger than memory streams straight in:
|
|
269
|
+
*
|
|
270
|
+
* ```ts
|
|
271
|
+
* await connection.copyFromRows('users', (async function* () {
|
|
272
|
+
* for await (const { value } of jsonRows) yield [value.id, value.name];
|
|
273
|
+
* })(), { columns: ['id', 'name'] });
|
|
274
|
+
* ```
|
|
275
|
+
*
|
|
276
|
+
* Rows are pulled rather than pushed, so the source only produces the
|
|
277
|
+
* next row once the previous chunk has reached the socket - backpressure
|
|
278
|
+
* is the loop pausing, not a queue growing.
|
|
279
|
+
*
|
|
280
|
+
* Binary is both faster and cheaper to produce than text: 200,000 rows of
|
|
281
|
+
* int4/text/float8/timestamptz measured 232ms against 483ms for the
|
|
282
|
+
* equivalent CSV `copyFrom()`, client-side encoding included, because
|
|
283
|
+
* writing an int32 costs less than formatting a decimal string.
|
|
284
|
+
*
|
|
285
|
+
* Column types are read from the server (one Describe round trip) unless
|
|
286
|
+
* `columnTypes` supplies them. They have to be exact - binary COPY does
|
|
287
|
+
* no conversion - which is also why a column whose type has no binary
|
|
288
|
+
* encoder is rejected before any row is sent rather than mid-stream.
|
|
289
|
+
*
|
|
290
|
+
* `copyFrom()` remains the way to send an already-formatted text or CSV
|
|
291
|
+
* payload.
|
|
292
|
+
*
|
|
293
|
+
* A value the column's type cannot encode - `'abc'` for an integer
|
|
294
|
+
* column - aborts the copy by default, naming the row and column;
|
|
295
|
+
* `onInvalidValue` can instead null the value or drop the row, and the
|
|
296
|
+
* result says how many that happened to. NaN and Infinity in a float or
|
|
297
|
+
* numeric column are not invalid: PostgreSQL stores them as values
|
|
298
|
+
* distinct from NULL, so they go through untouched.
|
|
299
|
+
*
|
|
300
|
+
* @returns Rows sent, plus what a tolerant `onInvalidValue` swallowed.
|
|
301
|
+
*/
|
|
302
|
+
/**
|
|
303
|
+
* Runs several different statements in one round trip: every
|
|
304
|
+
* Parse/Bind/Describe/Execute goes out before any response is waited
|
|
305
|
+
* for, and a single Sync closes the lot.
|
|
306
|
+
*
|
|
307
|
+
* ```ts
|
|
308
|
+
* const [renamed, , total] = await connection.pipeline([
|
|
309
|
+
* sql`update users set name = ${name} where id = ${id}`,
|
|
310
|
+
* sql`insert into audit(msg) values (${msg})`,
|
|
311
|
+
* sql`select count(*)::int as n from users`,
|
|
312
|
+
* ]);
|
|
313
|
+
* renamed.rowsAffected; // 1
|
|
314
|
+
* total.rows?.[0]; // [42]
|
|
315
|
+
* ```
|
|
316
|
+
*
|
|
317
|
+
* The counterpart to `PreparedStatement.executeBatch()`, which runs one
|
|
318
|
+
* statement over many parameter sets; this runs many statements once
|
|
319
|
+
* each. Twenty statements measured 2.3ms against 5.0ms for the same
|
|
320
|
+
* calls through `Promise.all()`, which PostgreJS already pipelines -
|
|
321
|
+
* what the single Sync removes is the server finishing an implicit
|
|
322
|
+
* transaction and answering ReadyForQuery twenty times over, which is
|
|
323
|
+
* why socket reads drop from twenty to one.
|
|
324
|
+
*
|
|
325
|
+
* Three things follow from that framing rather than from choices made
|
|
326
|
+
* above it:
|
|
327
|
+
*
|
|
328
|
+
* - **One transaction.** Unless an explicit transaction is already open,
|
|
329
|
+
* the statements commit or roll back together.
|
|
330
|
+
* - **A failing statement stops the rest.** PostgreSQL discards
|
|
331
|
+
* everything between an error and the Sync, so statements after a
|
|
332
|
+
* rejected one never run. The error carries `failedIndex`.
|
|
333
|
+
* - **No statement can see another's results.** They are all sent before
|
|
334
|
+
* any reply arrives, so anything conditional on an earlier result
|
|
335
|
+
* belongs in a separate call rather than here.
|
|
336
|
+
*
|
|
337
|
+
* Results map to statements by position. Statements that return rows get
|
|
338
|
+
* them decoded as `query()` would, `rowDecoder` included; `fetchCount`
|
|
339
|
+
* does not apply, since a portal suspended mid-pipeline would break that
|
|
340
|
+
* mapping.
|
|
341
|
+
*
|
|
342
|
+
* @param requests Statements to run, as `sql` tag output, plain SQL
|
|
343
|
+
* strings, or `{ sql, params }` objects.
|
|
344
|
+
* @param options Applied to every statement.
|
|
345
|
+
*/
|
|
346
|
+
async pipeline(requests, options = {}) {
|
|
347
|
+
if (!Array.isArray(requests))
|
|
348
|
+
throw new TypeError('pipeline() requires an array of statements');
|
|
349
|
+
if (options.cursor)
|
|
350
|
+
throw new Error('pipeline() cannot return a cursor - every statement runs to ' +
|
|
351
|
+
'completion under one Sync, so there is no portal left to fetch from');
|
|
352
|
+
if (!requests.length)
|
|
353
|
+
return [];
|
|
354
|
+
const normalized = requests.map(r => typeof r === 'string' ? { sql: r } : { sql: r.sql, params: r.params });
|
|
355
|
+
/* c8 ignore start */
|
|
356
|
+
if (this.listenerCount('debug')) {
|
|
357
|
+
this.emit('debug', {
|
|
358
|
+
location: 'Connection.pipeline',
|
|
359
|
+
connection: this,
|
|
360
|
+
message: `[${this.processID}] pipeline | ${normalized.length} statements`,
|
|
361
|
+
});
|
|
362
|
+
}
|
|
363
|
+
/* c8 ignore stop */
|
|
364
|
+
return await this._captureErrorStack(this._intlCon.executePipeline(normalized, options)).catch((e) => {
|
|
365
|
+
// Name the statement the server actually rejected, not the first
|
|
366
|
+
// one - the error is otherwise attributed to whichever SQL happens
|
|
367
|
+
// to be handy, which in a pipeline is the wrong statement.
|
|
368
|
+
const i = e.failedIndex;
|
|
369
|
+
throw this._handleError(e, i != null && normalized[i] ? normalized[i].sql : normalized[0].sql);
|
|
370
|
+
});
|
|
371
|
+
}
|
|
372
|
+
async copyFromRows(table, source, options) {
|
|
373
|
+
/* c8 ignore start */
|
|
374
|
+
if (this.listenerCount('debug')) {
|
|
375
|
+
this.emit('debug', {
|
|
376
|
+
location: 'Connection.copyFromRows',
|
|
377
|
+
connection: this,
|
|
378
|
+
message: `[${this.processID}] copyFromRows | ${table}`,
|
|
379
|
+
table,
|
|
380
|
+
});
|
|
381
|
+
}
|
|
382
|
+
/* c8 ignore stop */
|
|
383
|
+
return await this._captureErrorStack(this._intlCon.copyFromRows(table, source, options)).catch((e) => {
|
|
384
|
+
throw this._handleError(e, `COPY ${table} FROM STDIN (FORMAT binary)`);
|
|
385
|
+
});
|
|
386
|
+
}
|
|
254
387
|
async copyFrom(sql) {
|
|
255
388
|
/* c8 ignore start */
|
|
256
389
|
if (this.listenerCount('debug')) {
|
|
@@ -456,11 +589,17 @@ export class Connection extends SafeEventEmitter {
|
|
|
456
589
|
const effectiveAutoCommit = options?.autoCommit != null
|
|
457
590
|
? options.autoCommit
|
|
458
591
|
: this._intlCon.config.autoCommit;
|
|
592
|
+
// An open transaction takes this path too: queryCached() puts the
|
|
593
|
+
// rollbackOnError savepoint in the statement's own round trip rather
|
|
594
|
+
// than leaving it to the prepare()/execute()/close() route below. The
|
|
595
|
+
// one thing it cannot express is an explicit autoCommit:true while a
|
|
596
|
+
// transaction is open - "commit once this statement is done" - which
|
|
597
|
+
// stays with PreparedStatement's own wrapper.
|
|
459
598
|
if (!options?.cursor &&
|
|
460
599
|
effectiveAutoCommit !== false &&
|
|
461
|
-
!this._intlCon.inTransaction) {
|
|
600
|
+
!(options?.autoCommit === true && this._intlCon.inTransaction)) {
|
|
462
601
|
const params = options?.params?.map(prm => prm instanceof BindParam ? prm.value : prm);
|
|
463
|
-
return await this._captureErrorStack(this._intlCon.
|
|
602
|
+
return await this._captureErrorStack(this._intlCon.queryCached(sql, paramTypes, params, options || {}), this.query, options?.asyncErrorHandling).catch((e) => {
|
|
464
603
|
throw this._handleError(e, sql);
|
|
465
604
|
});
|
|
466
605
|
}
|
|
@@ -1,5 +1,9 @@
|
|
|
1
1
|
import { ConnectionState } from '../constants.js';
|
|
2
|
+
import type { DataTypeMap } from '../data-type-map.js';
|
|
3
|
+
import type { BatchResult } from '../interfaces/batch-result.js';
|
|
4
|
+
import type { CopyFromRowsOptions, CopyFromRowsResult } from '../interfaces/copy-from-rows-options.js';
|
|
2
5
|
import type { ConnectionConfiguration } from '../interfaces/database-connection-params.js';
|
|
6
|
+
import type { FieldInfo } from '../interfaces/field-info.js';
|
|
3
7
|
import type { FunctionCallOptions } from '../interfaces/function-call-options.js';
|
|
4
8
|
import type { FunctionCallResult } from '../interfaces/function-call-result.js';
|
|
5
9
|
import type { QueryOptions } from '../interfaces/query-options.js';
|
|
@@ -9,10 +13,25 @@ import type { ScriptResult } from '../interfaces/script-result.js';
|
|
|
9
13
|
import { PgSocket } from '../protocol/pg-socket.js';
|
|
10
14
|
import { Protocol } from '../protocol/protocol.js';
|
|
11
15
|
import { SafeEventEmitter } from '../safe-event-emitter.js';
|
|
12
|
-
import type { Maybe, OID } from '../types.js';
|
|
16
|
+
import type { AnyParseFunction, Maybe, OID } from '../types.js';
|
|
17
|
+
import { type CopyRowSource } from '../util/copy-from-rows.js';
|
|
13
18
|
import type { Connection } from './connection.js';
|
|
14
19
|
import { CopyFromStream, CopyToStream } from './copy-stream.js';
|
|
15
20
|
export declare class IntlConnection extends SafeEventEmitter {
|
|
21
|
+
/**
|
|
22
|
+
* Server-side prepared statements this connection has built, keyed by
|
|
23
|
+
* the SQL text and its parameter OIDs - both, because the OIDs are part
|
|
24
|
+
* of what was parsed, not just how it is bound.
|
|
25
|
+
*
|
|
26
|
+
* A Map, relied on for its insertion order: re-inserting on every hit
|
|
27
|
+
* makes the first entry the least recently used, which is what gets
|
|
28
|
+
* closed when the cache is full. Statements belong to the session, so
|
|
29
|
+
* this is per connection and dies with it.
|
|
30
|
+
*/
|
|
31
|
+
private _preparedCache;
|
|
32
|
+
/** SQL seen once but not yet prepared - see PREPARE_AFTER_USES. */
|
|
33
|
+
private _preparedCandidates;
|
|
34
|
+
private _preparedCounter;
|
|
16
35
|
private _executeReusedParserCache;
|
|
17
36
|
protected _refCount: number;
|
|
18
37
|
protected _transactionDepth: number;
|
|
@@ -136,6 +155,21 @@ export declare class IntlConnection extends SafeEventEmitter {
|
|
|
136
155
|
* Runs a COPY ... FROM STDIN and hands back a stream to feed it.
|
|
137
156
|
* Resolves once the server is ready for data.
|
|
138
157
|
*/
|
|
158
|
+
/**
|
|
159
|
+
* Streams rows into a table with `COPY ... FROM STDIN (FORMAT binary)`,
|
|
160
|
+
* encoding each value with its own type's binary encoder.
|
|
161
|
+
*
|
|
162
|
+
* Binary COPY performs no conversion server-side, so the destination
|
|
163
|
+
* column types have to be known exactly. They are read from the server
|
|
164
|
+
* unless the caller supplies them: a Describe of `select <columns> from
|
|
165
|
+
* <table> where false` against the unnamed statement returns a
|
|
166
|
+
* RowDescription carrying each column's OID, which costs one round trip
|
|
167
|
+
* and leaves nothing to close (the next Parse replaces the unnamed
|
|
168
|
+
* statement). Letting PostgreSQL resolve the name is the reason to do it
|
|
169
|
+
* this way rather than reading the catalog here - schemas, quoting,
|
|
170
|
+
* search_path and views then behave exactly as they will for the COPY.
|
|
171
|
+
*/
|
|
172
|
+
copyFromRows(table: string, source: CopyRowSource, options?: CopyFromRowsOptions): Promise<CopyFromRowsResult>;
|
|
139
173
|
copyFrom(sql: string): Promise<CopyFromStream>;
|
|
140
174
|
protected _execute(sql: string, options?: ScriptExecuteOptions, cb?: (event: string, ...args: any[]) => void): Promise<ScriptResult>;
|
|
141
175
|
/**
|
|
@@ -145,7 +179,43 @@ export declare class IntlConnection extends SafeEventEmitter {
|
|
|
145
179
|
* +close()'s 7 round trips whenever there's no cursor and no explicit/
|
|
146
180
|
* active transaction to wrap (see Connection.query() for that gating).
|
|
147
181
|
*/
|
|
148
|
-
|
|
182
|
+
/**
|
|
183
|
+
* Runs a one-shot extended query, reusing a server-side prepared
|
|
184
|
+
* statement when this connection has run the same SQL before.
|
|
185
|
+
*
|
|
186
|
+
* A repeated query then costs Bind/Execute instead of
|
|
187
|
+
* Parse/Bind/Describe/Execute, which the server can answer without
|
|
188
|
+
* planning again - fifty concurrent calls of one statement measured
|
|
189
|
+
* 0.97ms against 3.13ms, and socket reads dropped from 38 to 8.
|
|
190
|
+
*
|
|
191
|
+
* Nothing is prepared on first sight: SQL has to be seen
|
|
192
|
+
* PREPARE_AFTER_USES times before it earns a name, so a query that runs
|
|
193
|
+
* once costs exactly what it costs today.
|
|
194
|
+
*
|
|
195
|
+
* A cached statement can be invalidated under us - `alter table` between
|
|
196
|
+
* two calls makes PostgreSQL answer 0A000, "cached plan must not change
|
|
197
|
+
* result type". That is caught here: the entry is dropped and the query
|
|
198
|
+
* runs again unprepared, so a migration against a live connection
|
|
199
|
+
* recovers instead of failing every call from then on.
|
|
200
|
+
*/
|
|
201
|
+
queryCached(sql: string, paramTypes: Maybe<Maybe<OID>[]>, params: Maybe<Maybe<any>[]>, options: QueryOptions): Promise<QueryResult>;
|
|
202
|
+
/**
|
|
203
|
+
* The savepoint rollbackOnError calls for, or undefined when this call
|
|
204
|
+
* needs none: outside a transaction there is nothing to roll back to,
|
|
205
|
+
* and a statement that is itself a transaction command manages its own
|
|
206
|
+
* boundaries (wrapping COMMIT in a savepoint would be nonsense).
|
|
207
|
+
*/
|
|
208
|
+
protected _inlineSavepointFor(sql: string, options: QueryOptions): Maybe<string>;
|
|
209
|
+
protected _queryCached(sql: string, paramTypes: Maybe<Maybe<OID>[]>, params: Maybe<Maybe<any>[]>, options: QueryOptions, savepoint: Maybe<string>): Promise<QueryResult>;
|
|
210
|
+
/** Closes the least recently used statement once the cache is full. */
|
|
211
|
+
protected _evictPreparedIfFull(): Promise<void>;
|
|
212
|
+
/**
|
|
213
|
+
* Closes an evicted statement server-side. Close answers CloseComplete
|
|
214
|
+
* but no ReadyForQuery, so the Sync goes separately and both are awaited
|
|
215
|
+
* together - the same shape PreparedStatement._close() uses.
|
|
216
|
+
*/
|
|
217
|
+
protected _closePreparedStatement(name: string): Promise<void>;
|
|
218
|
+
queryOnce(sql: string, paramTypes: Maybe<Maybe<OID>[]>, params: Maybe<Maybe<any>[]>, options: QueryOptions, savepoint?: string): Promise<QueryResult>;
|
|
149
219
|
/**
|
|
150
220
|
* Parse+Describe(statement)+Sync as a single round trip - used by
|
|
151
221
|
* PreparedStatement.prepare() instead of a separately-awaited Parse then
|
|
@@ -164,7 +234,47 @@ export declare class IntlConnection extends SafeEventEmitter {
|
|
|
164
234
|
* the RowDescription fields prepareOnce() already fetched instead of
|
|
165
235
|
* re-Describing a fresh portal on every call.
|
|
166
236
|
*/
|
|
167
|
-
|
|
237
|
+
/**
|
|
238
|
+
* Parsers and wrapped field descriptions for a statement whose
|
|
239
|
+
* RowDescription prepare() already fetched, memoised per RowDescription
|
|
240
|
+
* so repeated executes of the same statement don't rebuild them.
|
|
241
|
+
* Shared by executeReused() and executeBatchReused().
|
|
242
|
+
*/
|
|
243
|
+
protected _resolveReusedParsers(cachedFields: Protocol.RowDescription[], typeMap: DataTypeMap, options: QueryOptions): {
|
|
244
|
+
parsers: AnyParseFunction[];
|
|
245
|
+
resultFields: FieldInfo[];
|
|
246
|
+
};
|
|
247
|
+
/**
|
|
248
|
+
* Executes one prepared statement once per parameter set, as a single
|
|
249
|
+
* Bind/Execute stream closed by one Sync (see
|
|
250
|
+
* PgSocket.sendBatchBindExecuteMessages() for the wire shape and what
|
|
251
|
+
* the single Sync implies).
|
|
252
|
+
*
|
|
253
|
+
* Sets map to results positionally: the server answers each Execute with
|
|
254
|
+
* its own CommandComplete, and rows that arrive before one belong to the
|
|
255
|
+
* set it closes. When the server rejects a set, everything behind it is
|
|
256
|
+
* discarded unexecuted - the completed prefix is attached to the thrown
|
|
257
|
+
* error as `batchResults`, and the rejected set's index as `failedIndex`,
|
|
258
|
+
* so a caller can tell how far the batch actually got.
|
|
259
|
+
*/
|
|
260
|
+
/**
|
|
261
|
+
* Runs several different statements as one Parse/Bind/Describe/Execute
|
|
262
|
+
* stream closed by a single Sync (see PgSocket.sendPipelineMessages() for
|
|
263
|
+
* the wire shape and what that framing implies).
|
|
264
|
+
*
|
|
265
|
+
* The counterpart to executeBatchReused(): that one runs a single
|
|
266
|
+
* prepared statement over many parameter sets, this one runs many
|
|
267
|
+
* statements once each. Results map to statements positionally - the
|
|
268
|
+
* server answers each Execute with its own CommandComplete, and any rows
|
|
269
|
+
* that arrive before one belong to the statement it closes.
|
|
270
|
+
*/
|
|
271
|
+
executePipeline(requests: {
|
|
272
|
+
sql: string;
|
|
273
|
+
params?: any[];
|
|
274
|
+
paramTypes?: Maybe<OID>[];
|
|
275
|
+
}[], options: QueryOptions): Promise<QueryResult[]>;
|
|
276
|
+
executeBatchReused(statementName: string, cachedFields: Protocol.RowDescription[] | undefined, paramTypes: Maybe<Maybe<OID>[]>, paramSets: Maybe<any>[][], options: QueryOptions): Promise<BatchResult>;
|
|
277
|
+
executeReused(statementName: string, cachedFields: Protocol.RowDescription[] | undefined, paramTypes: Maybe<Maybe<OID>[]>, params: Maybe<Maybe<any>[]>, options: QueryOptions, savepoint?: string): Promise<QueryResult>;
|
|
168
278
|
emit(event: string | symbol, ...args: any[]): boolean;
|
|
169
279
|
protected _onError(err: Error): void;
|
|
170
280
|
}
|