velocious 1.0.626 → 1.0.627

package/README.md

@@ -1827,7 +1827,7 @@ database: {
 }
 ```
 
-`pool.max` caps live async-tracked connections for that pool and defaults to `10` when omitted. When the cap is reached, new checkouts wait until a matching checked-in connection can be handed over or capacity is freed. Set `pool.max` to `null` only when a process is deliberately allowed to open an unbounded number of database connections. The built-in debug endpoint reports each in-use connection's `checkedOutForMs`, each idle connection's `idleForMs`, and queued `pendingCheckouts[].waitingForMs` so production diagnostics can distinguish long-held checkouts from pool-capacity waits.
+`pool.max` caps live async-tracked connections for that pool and defaults to `10` when omitted. When the cap is reached, new checkouts wait until a matching checked-in connection can be handed over or capacity is freed. Set `pool.max` to `null` only when a process is deliberately allowed to open an unbounded number of database connections. The built-in debug endpoint reports each in-use connection's `checkedOutForMs`, each idle connection's `idleForMs`, queued `pendingCheckouts[].waitingForMs`, and matching idle capacity plus checkout-drain state so production diagnostics can distinguish long-held checkouts, pool-capacity waits, and an invariant violation where compatible capacity is unexpectedly idle.
 
 Debug snapshots also expose cumulative connection-creation, checkout-wait and
 timeout, idle-reap, and peak-live-connection telemetry. Opt-in test profiles can
@@ -2028,7 +2028,7 @@ await client.close()
 
 For long-lived Node clients, the constructor also accepts opt-in liveness options (all default off, so browser/Expo usage is unchanged): `webSocketImplementation` (inject Node's `ws`, since the global/undici WebSocket exposes neither protocol ping nor an unref-able socket), `heartbeatIntervalMs` (a ping heartbeat that drops a socket whose peer stops ponging, so a client notices a vanished server), and `unref` (unref the underlying socket so an idle connection can't keep the process alive on its own). See [docs/websocket-channels.md](docs/websocket-channels.md).
 
-`await client.close()` is a final graceful shutdown that releases resumable server-session state; unexpected transport drops first attempt to resume that state. A successful resume retains the existing server-side connection and channel instances. If the server instead rejects the old session with `session-gone`, SnapReq promotes the already-established fresh session, reopens still-live one-to-one connection handles, and re-subscribes still-live channel handles. Those public handles remain usable and channel readiness resolves on the fresh session; explicitly closed handles stay closed. See [the WebSocket channel lifecycle guarantees](docs/websocket-channels.md#lifecycle-guarantees-phase-1b).
+`await client.close()` is a final graceful shutdown that releases resumable server-session state; unexpected transport drops first attempt to resume that state. On a multi-worker server, the client automatically puts the prior session identity in the reconnect upgrade URL so the host can route it to its owning worker; routing is session-based and never source-IP-based. A successful resume retains the existing server-side connection and channel instances. If the server instead rejects the old session with `session-gone`, SnapReq promotes the already-established fresh session, reopens still-live one-to-one connection handles, and re-subscribes still-live channel handles. Those public handles remain usable and channel readiness resolves on the fresh session; explicitly closed handles stay closed. See [the WebSocket channel lifecycle guarantees](docs/websocket-channels.md#lifecycle-guarantees-phase-1b).
 
 ## Subscribe to events
 
@@ -2701,6 +2701,18 @@ It exposes `GET /api/stats`, `/api/jobs`, `/api/jobs/:id`, `/api/schedule` and `
 npx velocious server --host 0.0.0.0 --port 8082
 ```
 
+Threaded servers default to `os.availableParallelism()` HTTP workers. Pass
+`--workers` or configure `httpServer.workers` to override that count. Ordinary
+connections are distributed round-robin even behind a loopback reverse proxy,
+while resumable WebSockets return to their session's worker. Each worker owns a
+separate configuration and database pools, so per-worker limits multiply across
+the effective worker count (for example, four workers with `pool.max: 10` can
+open 40 connections for that pool). The debug snapshot exposes the configured
+and effective counts; default in-process mode uses one effective handler.
+Only requests carrying a resumable-session query wait for upgrade headers before
+worker assignment; ordinary and malformed requests continue directly to the
+request parser.
+
 When the server runs in the `development` environment, Velocious watches application `src/` trees and hot-reloads by recycling HTTP workers after `.js`/`.mjs`/`.cjs`/`.json`/`.ejs` changes. That picks up edited controllers, models, resources, routes, and views without a manual server restart while keeping production/test behavior unchanged.
 
 Starting the HTTP server creates `tmp/server.lock` under the configured application directory before Beacon, workers, or the TCP listener start. A second server for the same app fails fast with the lock owner details instead of partially starting. Normal shutdown removes the lock; stale locks with a dead local PID are reclaimed automatically, while locks from another host or unreadable metadata should be removed manually only after confirming no server is running. See [docs/http-server.md](docs/http-server.md#server-lock).

package/build/database/pool/async-tracked-multi-connection.js

@@ -2,6 +2,7 @@
 
 import { AsyncLocalStorage } from "async_hooks"
 import BasePool, { POOL_CONFIGURATION_KEY } from "./base.js"
+import DatabasePoolCheckoutTimeoutError from "./checkout-timeout-error.js"
 import { currentTestProfileContext } from "../../testing/test-profile-context.js"
 
 /**
@@ -101,6 +102,9 @@ export default class VelociousDatabasePoolAsyncTrackedMultiConnection extends Ba
    * @type {Promise<void> | undefined} */
   pendingCheckoutDrainPromise = undefined
 
+  /** Whether a caller requested another pass through the pending checkout queue. */
+  pendingCheckoutDrainRequested = false
+
   /**
    * Idle connection reaper timer.
    * @type {ReturnType<typeof setTimeout> | undefined} */
@@ -602,18 +606,45 @@ export default class VelociousDatabasePoolAsyncTrackedMultiConnection extends Ba
    * @returns {Promise<void>} - Resolves when pending checkouts have been drained as far as possible.
    */
   async drainPendingCheckouts() {
-    if (this.pendingCheckoutDrainPromise) {
-      await this.pendingCheckoutDrainPromise
-      return
-    }
+    this.pendingCheckoutDrainRequested = true
 
-    this.pendingCheckoutDrainPromise = this.drainPendingCheckoutsActual()
+    if (!this.pendingCheckoutDrainPromise) this.startPendingCheckoutDrain()
+    await this.pendingCheckoutDrainPromise
+  }
 
+  /**
+   * Starts the single checkout-drain owner. The shared promise is cleared before
+   * it settles, closing the resolved-promise/stale-field interval in which a new
+   * request could otherwise be lost.
+   * @returns {void}
+   */
+  startPendingCheckoutDrain() {
+    const {promise, reject, resolve} = Promise.withResolvers()
+
+    this.pendingCheckoutDrainPromise = promise
+    void this.runRequestedPendingCheckoutDrains({reject, resolve})
+  }
+
+  /**
+   * Runs drain passes until every request observed during the active pass has
+   * received a later pass.
+   * @param {{reject: (reason?: ReturnType<typeof JSON.parse>) => void, resolve: (value?: void) => void}} deferred - Shared drain settlement.
+   * @returns {Promise<void>}
+   */
+  async runRequestedPendingCheckoutDrains({reject, resolve}) {
     try {
-      await this.pendingCheckoutDrainPromise
-    } finally {
+      while (this.pendingCheckoutDrainRequested) {
+        this.pendingCheckoutDrainRequested = false
+        await this.drainPendingCheckoutsActual()
+      }
+    } catch (error) {
       this.pendingCheckoutDrainPromise = undefined
+      reject(error)
+      return
     }
+
+    this.pendingCheckoutDrainPromise = undefined
+    resolve()
   }
 
   /**
@@ -726,13 +757,13 @@ export default class VelociousDatabasePoolAsyncTrackedMultiConnection extends Ba
   /**
    * Runs pending checkout timeout error.
    * @param {PendingCheckout} checkout - Timed-out checkout.
-   * @returns {Error} - Timeout error.
+   * @returns {DatabasePoolCheckoutTimeoutError} - Timeout error.
    */
   pendingCheckoutTimeoutError(checkout) {
     const checkoutName = checkout.options.name ? ` Checkout name: ${JSON.stringify(checkout.options.name)}.` : ""
     const diagnostics = this.pendingCheckoutTimeoutDiagnostics(checkout)
 
-    return new Error(`Timed out after ${checkout.timeoutMillis}ms waiting for database connection checkout from pool "${this.identifier}".${checkoutName} ${diagnostics}`)
+    return new DatabasePoolCheckoutTimeoutError(`Timed out after ${checkout.timeoutMillis}ms waiting for database connection checkout from pool "${this.identifier}".${checkoutName} ${diagnostics}`)
   }
 
   /**
@@ -1255,7 +1286,13 @@ export default class VelociousDatabasePoolAsyncTrackedMultiConnection extends Ba
       connections,
       connectionsBeingSpawned: this.connectionsBeingSpawned,
       idleCount: this.connections.length + [...this.lifecycleRetainedConnections.values()].filter((connection) => connection.getIdSeq() === undefined).length,
+      idleMatchingPendingCheckoutCount: this.connections.filter((connection) => {
+        return !this.connectionHasOpenTransaction(connection)
+          && this.pendingCheckouts.some((checkout) => this.connectionMatchesReuseKey(connection, checkout.reuseKey))
+      }).length,
       inUseCount: Object.keys(this.connectionsInUse).length,
+      pendingCheckoutDrainActive: Boolean(this.pendingCheckoutDrainPromise),
+      pendingCheckoutDrainRequested: this.pendingCheckoutDrainRequested,
       pendingCheckouts: this.pendingCheckoutDebugSnapshots(now),
       pendingCheckoutCount: this.pendingCheckouts.length,
       telemetry: {...this.telemetry}

package/build/database/pool/base.js

@@ -40,10 +40,13 @@ import sha256Hex from "../../utils/sha256-hex.js"
  * @property {Array<Record<string, ReturnType<typeof JSON.parse>>>} connections - Live connection snapshots.
  * @property {number} connectionsBeingSpawned - Number of in-progress connection spawns.
  * @property {number} idleCount - Number of idle connections.
+ * @property {number} [idleMatchingPendingCheckoutCount] - Idle connections that can satisfy at least one pending checkout.
  * @property {string} identifier - Database identifier.
  * @property {number} inUseCount - Number of checked-out connections.
  * @property {Array<DatabasePoolPendingCheckoutDebugSnapshot>} [pendingCheckouts] - Waiting checkout snapshots.
  * @property {number} pendingCheckoutCount - Number of queued checkout requests.
+ * @property {boolean} [pendingCheckoutDrainActive] - Whether a checkout drain pass is active.
+ * @property {boolean} [pendingCheckoutDrainRequested] - Whether another checkout drain pass was requested.
  * @property {string} poolClass - Pool class name.
  * @property {{connectionCreationCount: number, connectionCreationFailureCount: number, connectionCreationMaxMs: number, connectionCreationTotalMs: number, checkoutTimeoutCount: number, checkoutWaitCount: number, checkoutWaitMaxMs: number, checkoutWaitTotalMs: number, idleReapCount: number, idleReapDisposalCount: number, idleReapFailureCount: number, idleReapMaxMs: number, idleReapTotalMs: number, peakLiveConnections: number}} [telemetry] - Cumulative pool lifecycle telemetry.
  */

package/build/database/pool/checkout-timeout-error.js

@@ -0,0 +1,16 @@
+// @ts-check
+
+/** Stable framework error for a database pool checkout that exceeded its wait limit. */
+export default class DatabasePoolCheckoutTimeoutError extends Error {
+  /**
+   * Builds a database pool checkout timeout error.
+   * @param {string} message - Detailed sanitized pool timeout message.
+   * @param {{cause?: unknown}} [args] - Error options.
+   */
+  constructor(message, {cause} = {}) {
+    super(message, cause === undefined ? undefined : {cause})
+
+    this.name = "DatabasePoolCheckoutTimeoutError"
+    this.code = "VELOCIOUS_DATABASE_POOL_CHECKOUT_TIMEOUT"
+  }
+}

package/build/http-client/websocket-client.js

@@ -4,6 +4,7 @@ import SnapReqWebSocketClient from "snapreq/websocket"
 import {deserializeFrontendModelTransportValue} from "../frontend-models/transport-serialization.js"
 
 const DEFAULT_URL = "ws://127.0.0.1:3006/websocket"
+const SESSION_ROUTING_PARAMETER = "velociousSessionId"
 
 /**
  * Velocious's WebSocket client. The cross-platform connection/session/channel
@@ -28,6 +29,57 @@ export default class VelociousWebsocketClient extends SnapReqWebSocketClient {
     this.runningReconnectTasks = new Set()
     /** @type {Promise<void> | null} */
     this.gracefulClosePromise = null
+    this.routingBaseUrl = this.url
+  }
+
+  /**
+   * Restores a persisted session before opening the socket so the host can route
+   * the HTTP upgrade to the worker that owns its paused state.
+   * @returns {Promise<void>}
+   */
+  async _restoreSessionIdForRouting() {
+    // SnapReq initializes these internal session fields in its constructor, but
+    // its declaration does not expose that definite-assignment lifecycle here.
+    const routingState = /** @type {{_sessionId: string | null, _sessionStore: {get: () => string | null | undefined | Promise<string | null | undefined>} | undefined, _sessionStoreRestored: boolean}} */ (/** @type {unknown} */ (this))
+
+    if (routingState._sessionId || routingState._sessionStoreRestored || !routingState._sessionStore) return
+
+    routingState._sessionStoreRestored = true
+
+    try {
+      const storedId = await routingState._sessionStore.get()
+
+      if (typeof storedId === "string" && storedId.length > 0) routingState._sessionId = storedId
+    } catch (error) {
+      this._debug("sessionStore.get failed", error)
+    }
+  }
+
+  /**
+   * Builds the WebSocket URL carrying only the current resumable session routing hint.
+   * @returns {string} - WebSocket URL.
+   */
+  _sessionRoutingUrl() {
+    const url = new URL(this.routingBaseUrl)
+
+    if (this._sessionId) {
+      url.searchParams.set(SESSION_ROUTING_PARAMETER, this._sessionId)
+    } else {
+      url.searchParams.delete(SESSION_ROUTING_PARAMETER)
+    }
+
+    return url.toString()
+  }
+
+  /**
+   * Restores routing state before delegating socket creation to SnapReq.
+   * @param {Parameters<SnapReqWebSocketClient["_connect"]>[0]} [options] - Connect options.
+   * @returns {Promise<void>} - Resolves when the session is ready.
+   */
+  async _connect(options) {
+    await this._restoreSessionIdForRouting()
+    this.url = this._sessionRoutingUrl()
+    await super._connect(options)
   }
 
   /**

package/build/http-server/client/index.js

@@ -314,6 +314,12 @@ export default class VeoliciousHttpServerClient {
       this.websocketSession = undefined
       this.events.emit("close")
     })
+    this.websocketSession.events.on("ownershipClaimed", ({sessionId}) => {
+      this.events.emit("websocketSessionOwned", {sessionId})
+    })
+    this.websocketSession.events.on("ownershipReleased", ({sessionId}) => {
+      this.events.emit("websocketSessionReleased", {sessionId})
+    })
     this.state = "websocket"
     this.events.emit("output", response)
     void this.websocketSession.initializeChannel()

package/build/http-server/client/websocket-session.js

@@ -215,6 +215,9 @@ export default class VelociousHttpServerClientWebsocketSession {
      */
     this._resumeIdentityPromise = undefined
 
+    /** @type {string | null} */
+    this._claimedSessionId = null
+
     /**
      * Accumulates payloads for a fragmented websocket message per
      * RFC 6455. Non-null while mid-fragment; cleared when the frame
@@ -266,6 +269,7 @@ export default class VelociousHttpServerClientWebsocketSession {
    * @returns {void}
    */
   sendSessionEstablished() {
+    this._claimOwnership()
     this.sendJson({
       type: "session-established",
       sessionId: this.sessionId,
@@ -313,6 +317,7 @@ export default class VelociousHttpServerClientWebsocketSession {
   }
 
   destroy() {
+    this._releaseOwnership()
     this._stopHeartbeat()
     this._resetFragmentBuffer()
     this._clearBufferedFrameChunks()
@@ -325,6 +330,25 @@ export default class VelociousHttpServerClientWebsocketSession {
     this.events.removeAllListeners()
   }
 
+  /** Claims this session id for host-side reconnect routing. */
+  _claimOwnership() {
+    if (this._claimedSessionId === this.sessionId) return
+    if (this._claimedSessionId) this._releaseOwnership()
+
+    this._claimedSessionId = this.sessionId
+    this.events.emit("ownershipClaimed", {sessionId: this.sessionId})
+  }
+
+  /** Releases the currently claimed session id exactly once. */
+  _releaseOwnership() {
+    const sessionId = this._claimedSessionId
+
+    if (!sessionId) return
+
+    this._claimedSessionId = null
+    this.events.emit("ownershipReleased", {sessionId})
+  }
+
   /**
    * Runs has subscription.
    * @param {string} channel - Channel name.
@@ -1339,6 +1363,7 @@ export default class VelociousHttpServerClientWebsocketSession {
     }
 
     this._stopHeartbeat()
+    this._releaseOwnership()
     this.configuration._websocketSessions.delete(this)
     void this._runMessageHandlerClose()
     void this._teardownChannel()
@@ -1355,6 +1380,7 @@ export default class VelociousHttpServerClientWebsocketSession {
    */
   _finalizeGraceExpiry() {
     this._stopHeartbeat()
+    this._releaseOwnership()
     this._resetFragmentBuffer()
     this._clearBufferedFrameChunks()
     this._abandonInboundMessages()
@@ -1499,6 +1525,9 @@ export default class VelociousHttpServerClientWebsocketSession {
 
     this.configuration._clearPausedWebsocketSession(resumeSessionId)
 
+    this._releaseOwnership()
+    paused._releaseOwnership()
+
     // Transfer resumable state onto this (live) session. The paused
     // session shell is discarded after the transfer.
     for (const [connectionId, connection] of paused._connections) {
@@ -1526,6 +1555,7 @@ export default class VelociousHttpServerClientWebsocketSession {
     paused._paused = false
     paused.destroy()
 
+    this._claimOwnership()
     this.sendJson({type: "session-resumed", sessionId: resumeSessionId})
     for (const body of queued) this.sendJson(body)
     await this._fireOnResume()