@steadybrew/postgresjs 4.3.0 → 4.4.1

package/README.md

@@ -592,6 +592,8 @@ for await (const chunk of readableStream) {
 
 > **NOTE** This is a low-level API which does not provide any type safety. To make this work, you must match your [`copy query` parameters](https://www.postgresql.org/docs/14/sql-copy.html) correctly to your [Node.js stream read or write](https://nodejs.org/api/stream.html) code. Ensure [Node.js stream backpressure](https://nodejs.org/en/learn/modules/backpressuring-in-streams) is handled correctly to avoid memory exhaustion.
 
+A `COPY` holds its connection until it finishes. Postgres.js cannot tell that a simple-protocol query is a `COPY` until the server answers, so it sends nothing behind a simple query whose text contains the word `copy` (`sql.unsafe()` or `sql.file()` without parameters, `.simple()`, `.readable()`, `.writable()`). Queries sent through `sql` meanwhile run on another connection or wait for one. When every connection is running a `COPY` that has not been ended, for example with `max: 1`, queries, `sql.reserve()` and `sql.begin()` through `sql` reject with `COPY_IN_PROGRESS` instead, because only that `COPY` finishing could free a connection. Statements on a `sql.reserve()` or `sql.begin()` handle reject with `COPY_IN_PROGRESS` while its `COPY` runs.
+
 ### Canceling Queries in Progress
 
 Postgres.js supports, [canceling queries in progress](https://www.postgresql.org/docs/7.1/protocol-protocol.html#AEN39000). It works by opening a new connection with a protocol level startup message to cancel the current query running on a specific connection. That means there is no guarantee that the query will be canceled, and due to the possible race conditions it might even result in canceling another query. This is fine for long running queries, but in the case of high load and fast queries it might be better to simply ignore results instead of canceling.
@@ -869,7 +871,7 @@ const sql = postgres({
 When you call `.listen`, a dedicated connection will be created to ensure that you receive notifications instantly. This connection will be used for any further calls to `.listen`. The connection will automatically reconnect according to a backoff reconnection pattern to not overload the database server.
 
 ### Listen `await sql.listen(channel, onnotify, [onlisten]) -> { state }`
-`.listen` takes the channel name, a function to handle each notify, and an optional function to run every time listen is registered and ready (happens on initial connect and reconnects). It returns a promise which resolves once the `LISTEN` query to Postgres completes, or if there is already a listener active.
+`.listen` takes the channel name, a function to handle each notify, and an optional function to run every time listen is registered and ready (happens on initial connect and reconnects). It returns a promise which resolves once the `LISTEN` query to Postgres completes, or if there is already a listener active. If either function throws, the error is rethrown as an uncaught exception, and the other listeners and the returned promise are not affected.
 
 ```js
 await sql.listen('news', payload => {
@@ -922,10 +924,17 @@ const { unsubscribe } = await sql.subscribe(
   },
   () => {
     // Callback on initial connect and potential reconnects
+  },
+  (error) => {
+    // Callback on connection errors and changes that could not be decoded
   }
 )
 ```
 
+If any of these callbacks throws, the error is rethrown as an uncaught exception, and other subscribers and the replication stream are not affected.
+
+`onerror` is called with errors of the replication connection, after which the subscription reconnects on a new replication slot and calls `onsubscribe` again (these are also logged with `console.error`, and changes committed while it was disconnected are not delivered; if the reconnect itself fails, it is retried with backoff, its error is logged with `console.error` and passed to `onerror`, and it never ends the process), and with errors thrown while a change is decoded by a [transform](#data-transformation) or a custom type parser. Such a change is skipped, and the replication stream and later changes carry on. A decode error goes only to the subscriptions that would have received the change, matched by operation and table; a subscription filtered by key gets every skipped change of its table, as the key is read from the decoded row, even if the table has no primary key. If a column name transform throws, each change of that table is skipped and reported until the server describes the table again, for example after `ALTER TABLE`. If one of the matching subscriptions has no `onerror`, the error is rethrown once as an uncaught exception, which ends the process under Node's default handling. `onerror` stays registered across reconnects until `unsubscribe()`.
+
 ### Subscribe pattern
 
 You can subscribe to specific operations, tables, or even rows with primary keys.
@@ -1051,6 +1060,8 @@ const sql = postgres('postgres://username:password@host:port/database', {
 })
 ```
 
+The `onnotice`, `onparameter`, `onnotify`, `onclose`, `onidle` and `debug` callbacks passed in the options are called synchronously from inside the driver. If one throws, the error is rethrown as an uncaught exception and the connection and its queries are not affected; under Node's default handling that ends the process.
+
 Note that `max_lifetime = 60 * (30 + Math.random() * 30)` by default. This resolves to an interval between 30 and 60 minutes to optimize for the benefits of prepared statements **and** working nicely with Linux's OOM killer.
 
 ### Dynamic passwords
@@ -1457,7 +1468,12 @@ If the only attempt times out, the error is `CONNECT_TIMEOUT`. Once earlier atte
 ##### COPY_IN_PROGRESS
 > You cannot execute queries during copy
 
-This error is thrown if trying to run a query during a copy operation (writable / readable).
+This error is thrown for a statement on a `sql.reserve()` or `sql.begin()` handle while its copy operation (writable / readable) runs, including statements sent before the server started it. A query, `sql.reserve()` or `sql.begin()` through `sql` gets it when every connection is running a copy operation whose writable has not been ended; otherwise it waits for a connection.
+
+##### TRANSACTION_ROLLED_BACK
+> The server rolled the transaction back instead of committing it
+
+This error is thrown by `sql.begin()` when the server answers the final `COMMIT` (or `PREPARE TRANSACTION`) with `ROLLBACK`, because the transaction was aborted by a statement whose error the callback did not surface, such as a failing query that was not awaited. Nothing was committed. When such a query's error was already recorded, it is the error's `cause`.
 
 ## TypeScript support
 

package/cf/src/connection.js

@@ -7,7 +7,7 @@ import Stream from 'node:stream'
 import { performance } from '../polyfills.js'
 
 import { stringify, handleValue, addArrayType } from './types.js'
-import { Errors } from './errors.js'
+import { Errors, callback } from './errors.js'
 import Result from './result.js'
 import Queue from './queue.js'
 import clampedTimeout from './timeout.js'
@@ -15,7 +15,7 @@ import { SSLRequest, tlsConfig, cancelRequest } from './transport.js'
 import { Query, CLOSE } from './query.js'
 import b from './bytes.js'
 
-export default Connection
+export { Connection }
 
 let uid = 1
 
@@ -28,6 +28,8 @@ const Sync = b().S().end()
 const Phase = { Closed: 0, Backoff: 1, Connecting: 2, Initializing: 3, Ready: 4, Draining: 5, Closing: 6 }
 const phaseNames = Object.keys(Phase)
 
+const endsAbortedState = /^\s*(rollback|abort)\b/i
+
 const retryRoutines = new Set([
   'FetchPreparedStatement',
   'RevalidateCachedQuery',
@@ -112,6 +114,7 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
     , chunk = null
     , nonce = null
     , query = null
+    , barrier = null
     , final = null
 
   const check = globalThis[Symbol.for('postgres.js:check')]
@@ -226,6 +229,9 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
     owner: null,
     get active() { return query ? 1 : 0 },
     get pipelined() { return sent.length },
+    get accepting() { return accepting() },
+    get copying() { return stream !== null },
+    get copyHeld() { return stream !== null && !stream.writableEnded },
     get acquiring() { return acquisition && !acquisition.owner.cancelled ? 1 : 0 },
     count: 0,
     id
@@ -270,8 +276,12 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
     return !query && sent.length === 0
   }
 
+  function accepting() {
+    return !query || (!barrier && sent.length < max_pipeline)
+  }
+
   function backoffMs() {
-    return (typeof backoff === 'function' ? backoff(options.shared.retries) : backoff) * 1000
+    return backoffSeconds(backoff, options.shared.retries) * 1000
   }
 
   function standbyPass() {
@@ -448,7 +458,7 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
       queryError(pending, err)
       pending.cancelled && pending.cancelled.resolve()
     }
-    query = results = errorResponse = chunk = nextWriteTimer = null
+    query = results = errorResponse = chunk = nextWriteTimer = barrier = null
     result = new Result()
     rows = 0
     nonce = serverSignature = null
@@ -512,8 +522,10 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
   }
 
   function submit(q) {
-    if (stream)
-      return queryError(q, Errors.generic('COPY_IN_PROGRESS', 'You cannot execute queries during copy'))
+    if (stream) {
+      queryError(q, Errors.generic('COPY_IN_PROGRESS', 'You cannot execute queries during copy'))
+      return true
+    }
 
     if (q.cancelled)
       return true
@@ -527,17 +539,21 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
         : (query = q, query.active = true)
 
       build(q)
-      return write(toBuffer(q))
-        && !q.describeFirst
-        && !q.cursorFn
-        && sent.length < max_pipeline
+      const written = write(toBuffer(q))
+      if (q.describeFirst || q.cursorFn || mayCopy(q))
+        barrier = q
+      return written && accepting()
     } catch (error) {
       sent.remove(q) || write(Sync)
       queryError(q, error)
-      return sent.length < max_pipeline
+      return accepting()
     }
   }
 
+  function mayCopy(q) {
+    return q.options.simple && /\bcopy\b/i.test(q.string)
+  }
+
   function toBuffer(q) {
     if (q.parameters.length >= 65534)
       throw Errors.generic('MAX_PARAMETERS_EXCEEDED', 'Max number of parameters (65534) exceeded')
@@ -596,7 +612,7 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
       ? statements[q.signature]
       : { string, types, name: q.prepare ? statementId + statementCount++ : '' }
 
-    typeof options.debug === 'function' && options.debug(id, string, parameters, types)
+    typeof options.debug === 'function' && callback(options.debug, id, string, parameters, types)
   }
 
   function write(x, fn) {
@@ -767,18 +783,21 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
     backendParameters[k] = v
     if (options.parameters[k] !== v) {
       options.parameters[k] = v
-      onparameter && onparameter(k, v)
+      callback(onparameter, k, v)
     }
   }
 
-  function ReadyForQuery() {
+  function ReadyForQuery(x) {
+    query === barrier && (barrier = null)
     if (query) {
       if (errorResponse) {
         if (query.initialization)
           return dispatch('initFailed', errorResponse)
+        query.prepared && retryRoutines.has(errorResponse.routine) && delete statements[query.signature]
         query.retried
           ? errored(query.retried)
-          : query.prepared && retryRoutines.has(errorResponse.routine) && query.owner === connection.owner
+          : query.prepared && retryRoutines.has(errorResponse.routine) && query.owner === connection.owner && !barrier
+            && (x[5] !== 69 || endsAbortedState.test(query.string))
             ? retry(query, errorResponse)
             : errored(errorResponse)
       } else {
@@ -1061,7 +1080,6 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
   }
 
   function retry(q, error) {
-    delete statements[q.signature]
     q.retried = error
     execute(q)
   }
@@ -1072,7 +1090,8 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
 
     let index = 9
     while (x[index++] !== 0);
-    onnotify(
+    callback(
+      onnotify,
       x.toString('utf8', 9, index - 1),
       x.toString('utf8', index, x.length - 1)
     )
@@ -1118,6 +1137,7 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
       }
     })
     query.resolve(stream)
+    connection.owner && connection.owner.next()
   }
 
   function CopyOutResponse() {
@@ -1125,6 +1145,7 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
       read() { socket.resume() }
     })
     query.resolve(stream)
+    connection.owner && connection.owner.next()
   }
 
   /* c8 ignore next 3 */
@@ -1147,6 +1168,7 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
       }
     })
     query.resolve(stream)
+    connection.owner && connection.owner.next()
   }
 
   function CopyData(x) {
@@ -1159,10 +1181,12 @@ function Connection(options, queues = {}, { onopen = noop, onend = noop, onclose
   }
 
   function NoticeResponse(x) {
+    if (onnotice === false)
+      return
+
     onnotice
-      ? onnotice(parseError(x))
+      ? callback(onnotice, parseError(x))
       : console.log(parseError(x)) // eslint-disable-line
-
   }
 
   /* c8 ignore next 3 */
@@ -1313,3 +1337,19 @@ function timer(fn, seconds) {
     fn.apply(null, args)
   }
 }
+
+export function defaultBackoff(retries) {
+  return (0.5 + Math.random() / 2) * Math.min(3 ** retries / 100, 20)
+}
+
+export function backoffSeconds(backoff, retries) {
+  if (typeof backoff !== 'function')
+    return backoff
+
+  try {
+    return backoff(retries)
+  } catch (e) {
+    callback(() => { throw e })
+    return defaultBackoff(retries)
+  }
+}

package/cf/src/errors.js

@@ -6,6 +6,14 @@ export class PostgresError extends Error {
   }
 }
 
+export function callback(fn, ...args) {
+  try {
+    fn && fn(...args)
+  } catch (e) {
+    queueMicrotask(() => { throw e })
+  }
+}
+
 export const Errors = {
   connection,
   postgres,

package/cf/src/index.js

@@ -9,6 +9,7 @@ import {
   Parameter,
   Identifier,
   Builder,
+  escapeIdentifier,
   toPascal,
   pascal,
   toCamel,
@@ -20,12 +21,12 @@ import {
   fromKebab
 } from './types.js'
 
-import Connection from './connection.js'
+import { Connection, defaultBackoff } from './connection.js'
 import { cancelRequest } from './transport.js'
 import clampedTimeout from './timeout.js'
 import { Query, CLOSE } from './query.js'
 import Queue from './queue.js'
-import { Errors, PostgresError } from './errors.js'
+import { Errors, PostgresError, callback } from './errors.js'
 import Subscribe from './subscribe.js'
 import largeObject from './large.js'
 
@@ -219,7 +220,7 @@ function Postgres(a, b) {
         })
       },
       onnotify(c, x) {
-        c in listen.channels && listen.channels[c].listeners.forEach(l => l.fn(x))
+        c in listen.channels && listen.channels[c].listeners.forEach(l => callback(l.fn, x))
       }
     }))
 
@@ -229,7 +230,7 @@ function Postgres(a, b) {
     if (exists) {
       channels[name].listeners.push(listener)
       const result = await channels[name].result
-      listener.onlisten && listener.onlisten()
+      callback(listener.onlisten)
       return { state: result.state, unlisten }
     }
 
@@ -237,7 +238,7 @@ function Postgres(a, b) {
       sql.unsafe('"' + name.replace(/"/g, '""') + '"')
     }`, listeners: [listener] }
     const result = await channels[name].result
-    listener.onlisten && listener.onlisten()
+    callback(listener.onlisten)
     return { state: result.state, unlisten }
 
     async function unlisten() {
@@ -273,7 +274,9 @@ function Postgres(a, b) {
           const query = { reserve: c => resolve(own(c, l)), reject }
           closed.length
             ? connect(closed.shift(), query)
-            : queries.push(query)
+            : allHeldByCopy()
+              ? reject(copyInProgress())
+              : queries.push(query)
         })
     return l
   }
@@ -295,9 +298,9 @@ function Postgres(a, b) {
   function send(l, q, expected = Lease.Active) {
     const c = l.connection
     l.state === expected
-      ? c.queue === full
-        ? (q.queue = l.queue, l.queue.push(q))
-        : c.execute(q) || move(c, full)
+      ? c.copying || (c.queue !== full && c.accepting)
+        ? c.execute(q) || move(c, full)
+        : (move(c, full), q.queue = l.queue, l.queue.push(q))
       : q.reject(leaseError(l.state))
   }
 
@@ -353,14 +356,16 @@ function Postgres(a, b) {
     }
 
     async function finish(statement) {
+      let result
       l.state = Lease.Settling
       try {
-        await statement(Sql(q => send(l, q, Lease.Settling)))
+        result = await statement(Sql(q => send(l, q, Lease.Settling)))
       } catch (e) {
         settle(l, Lease.Discarded)
         throw e
       }
       settle(l, Lease.Released)
+      return result
     }
 
     async function scope(fn, name) {
@@ -375,7 +380,7 @@ function Postgres(a, b) {
       let uncaughtError
         , result
 
-      name && await sql`savepoint ${ sql(name) }`
+      name && await sql.unsafe('savepoint ' + escapeIdentifier(name))
       try {
         result = await new Promise((resolve, reject) => {
           const x = fn(sql)
@@ -386,8 +391,8 @@ function Postgres(a, b) {
           throw uncaughtError
       } catch (e) {
         l.state === Lease.Active && await (name
-          ? sql`rollback to ${ sql(name) }`
-          : finish(final => final`rollback`)
+          ? sql.unsafe('rollback to ' + escapeIdentifier(name))
+          : finish(final => final.unsafe('rollback'))
         )
         throw e instanceof PostgresError && e.code === '25P02' && uncaughtError || e
       }
@@ -395,10 +400,18 @@ function Postgres(a, b) {
       if (l.state !== Lease.Active)
         throw l.error || leaseError(l.state)
 
-      name || await finish(final => prepare
-        ? final`prepare transaction ${ final.unsafe(prepare) }`
-        : final`commit`
-      )
+      if (!name) {
+        const done = await finish(final => prepare
+          ? final.unsafe('prepare transaction ' + prepare)
+          : final.unsafe('commit')
+        )
+        if (done.command === 'ROLLBACK') {
+          throw Object.assign(
+            Errors.generic('TRANSACTION_ROLLED_BACK', 'The server rolled the transaction back instead of committing it'),
+            uncaughtError && { cause: uncaughtError }
+          )
+        }
+      }
 
       return result
 
@@ -427,14 +440,6 @@ function Postgres(a, b) {
     return c
   }
 
-  function callback(fn, ...args) {
-    try {
-      fn && fn(...args)
-    } catch (e) {
-      queueMicrotask(() => { throw e })
-    }
-  }
-
   function json(x) {
     return new Parameter(x, 3802)
   }
@@ -457,9 +462,32 @@ function Postgres(a, b) {
     if (closed.length)
       return connect(closed.shift(), query)
 
-    busy.length
-      ? go(busy.shift(), query)
-      : queries.push(query)
+    const c = takeBusy()
+    c
+      ? go(c, query)
+      : allHeldByCopy()
+        ? query.reject(copyInProgress())
+        : queries.push(query)
+  }
+
+  function allHeldByCopy() {
+    return connections.every(c => c.copyHeld)
+  }
+
+  function copyInProgress() {
+    return Errors.generic('COPY_IN_PROGRESS', 'You cannot execute queries during copy')
+  }
+
+  function takeBusy() {
+    while (busy.length) {
+      const c = busy.shift()
+      if (c.accepting)
+        return c
+
+      move(c, full)
+    }
+
+    return null
   }
 
   function takeOpen() {
@@ -546,6 +574,9 @@ function Postgres(a, b) {
     if (initial && initial.reserve)
       return initial.reserve(c)
 
+    if (!initial && !c.accepting)
+      return move(c, full)
+
     if (!initial && !queries.length)
       return move(c, open)
 
@@ -584,7 +615,7 @@ function Postgres(a, b) {
   function onclose(c, e) {
     move(c, closed)
     c.owner && settle(c.owner, Lease.Closed, e)
-    options.onclose && options.onclose(c.id)
+    callback(options.onclose, c.id)
     if (ending)
       return closedWhileEnding(c, e)
 
@@ -639,7 +670,7 @@ function parseOptions(a, b) {
     connect_timeout : 30,
     max_lifetime    : max_lifetime,
     max_pipeline    : 100,
-    backoff         : backoff,
+    backoff         : defaultBackoff,
     keep_alive      : 60,
     prepare         : true,
     debug           : false,
@@ -698,10 +729,6 @@ function tsa(o, url, env) {
   throw new Error('target_session_attrs ' + x + ' is not supported')
 }
 
-function backoff(retries) {
-  return (0.5 + Math.random() / 2) * Math.min(3 ** retries / 100, 20)
-}
-
 function max_lifetime() {
   return 60 * (30 + Math.random() * 30)
 }