velocious 1.0.530 → 1.0.532

package/README.md

@@ -2013,7 +2013,7 @@ New jobs default to `executionMode: "pooled"`: a worker runs them in warm, reusa
 
 `maxConcurrentForkedJobs` (default: `4`) caps how many out-of-process `executionMode: "forked"` or `executionMode: "spawned"` jobs one worker may keep in flight. Forked jobs use `child_process.fork()` with an attached IPC channel. After the main process acknowledges their durable status report, forked and spawned one-shot runners exit without waiting for graceful Beacon/database teardown; the OS closes their process-owned resources. A missing or rejected status acknowledgement makes the runner exit as failed instead of reporting clean success. Spawned jobs use the legacy `background-jobs-runner` CLI process via `child_process.spawn()` and are only for callers that intentionally want that spawned behavior.
 
-`jobTimeoutMs` (or `VELOCIOUS_BACKGROUND_JOBS_JOB_TIMEOUT_MS`, milliseconds; default: disabled) is a wall-clock backstop for a single `"forked"` runner. A forked job still running after the timeout is terminated (`SIGTERM`, then `SIGKILL` after the reaping grace) and reported `failed`, so a genuinely-hung runner can't pin a worker — and its whole-app boot and DB connections — indefinitely (notably a retired-release worker draining after a deploy). It's a coarse safety net, not per-job tuning: it applies to every forked runner, so set it well above the longest legitimate forked job. Omit it, or set `null`/`<= 0`, to disable. `"inline"` jobs are not covered — they share the worker's process and can't be killed without killing the worker. See [docs/background-jobs.md](docs/background-jobs.md#forked-job-timeout-hung-runner-backstop).
+`jobTimeoutMs` (or `VELOCIOUS_BACKGROUND_JOBS_JOB_TIMEOUT_MS`, milliseconds; default: disabled) is a wall-clock backstop for `"forked"` and `"pooled"` jobs. A job still running after the timeout is terminated (`SIGTERM`, then `SIGKILL` after the reaping grace) and reported `failed`, so a genuinely-hung job can't pin a worker's capacity — and its whole-app boot and DB connections — indefinitely (notably a retired-release worker draining after a deploy). For a **pooled** job the whole child running it is killed, so its concurrent in-flight siblings on that child are also reported `failed` and requeued — a hung JS job can't be cancelled any other way — before a replacement child is spawned. It's a coarse safety net, not per-job tuning: it applies to every forked and pooled job, so set it well above the longest legitimate job. Omit it, or set `null`/`<= 0`, to disable. `"inline"` jobs are not covered — they share the worker's process and can't be killed without killing the worker. See [docs/background-jobs.md](docs/background-jobs.md#job-timeout-hung-runner-backstop).
 
 ### Dispatch strategy
 
@@ -2327,4 +2327,6 @@ npx velocious db:tenants:migrate projectTenant --parallel 20
 
 At runtime, the apartment-style `Tenant` façade (`velocious/build/src/tenants/tenant.js`) is the single entry point: `Tenant.with(tenant, callback)` / `Tenant.current()` to switch into and read a tenant context, `Tenant.each({identifier, callback, parallel?, filter?})` to run a callback within every provider-listed tenant, and `Tenant.drop({identifier, tenant})` (plus the `db:tenants:drop` CLI command) to drop a tenant's database through the provider's `dropDatabase` hook. `Tenant.with` and `Tenant.each` run their callbacks inside `ensureConnections`, so switching into a tenant establishes its database connections (global + tenant) and passes them to the callback — the tenant is immediately queryable without the caller wiring up connections (already-open connections are reused, so nesting does not double-connect). `Tenant.with` is generic over its callback's return type, so a value returned from inside the tenant context comes back to the caller with its type preserved (no cast needed). `Tenant.aggregateAcross({identifier, aggregates, keyColumns, subquery, tenants?, filter?})` runs one aggregate over the same table across many tenant databases and returns the merged result — grouping tenants by server and using a single cross-database `UNION ALL` where the driver supports two-part `` `database`.`table` `` references (MySQL/MariaDB) or one query per tenant otherwise (PostgreSQL/SQLite/MSSQL).
 
+`SchemaCloner` adds a missing auto-increment column and its separate source unique index in one schema alteration, including on MySQL/MariaDB where an auto-increment column must be keyed when it is created.
+
 See [docs/tenant-databases.md](docs/tenant-databases.md) for the full configuration and migration pattern.

package/build/background-jobs/worker.js

@@ -72,7 +72,7 @@ export default class BackgroundJobsWorker {
    * @param {number} [args.pooledRunnerMaxLifetimeMs] - Override the per-runner recycle lifetime.
    * @param {number} [args.forkedChildSigkillGraceMs] - Override the grace period between SIGTERM and SIGKILL when reaping lingering process runners on stop.
    * @param {number} [args.heartbeatIntervalMs] - Override the liveness heartbeat interval (default 15000ms).
-   * @param {number} [args.jobTimeoutMs] - Override the forked-job wall-clock timeout from `configuration.getBackgroundJobsConfig()`. `0` disables it.
+   * @param {number} [args.jobTimeoutMs] - Override the wall-clock timeout for forked and pooled jobs from `configuration.getBackgroundJobsConfig()`. `0` disables it.
    */
   constructor({configuration, host, port, maxConcurrentForkedJobs, maxConcurrentInlineJobs, pooledRunnerCount, pooledRunnerConcurrency, pooledRunnerMaxJobs, pooledRunnerMaxRssBytes, pooledRunnerMaxLifetimeMs, forkedChildSigkillGraceMs, heartbeatIntervalMs, jobTimeoutMs} = {}) {
     /**
@@ -129,7 +129,7 @@ export default class BackgroundJobsWorker {
       ? forkedChildSigkillGraceMs
       : FORKED_CHILD_SIGKILL_GRACE_MS
     /**
-     * Constructor override for the forked-job wall-clock timeout. When unset the
+     * Constructor override for the forked and pooled wall-clock job timeout. When unset the
      * timeout is read from `configuration.getBackgroundJobsConfig().jobTimeoutMs`
      * at fork time (default: disabled).
      * @type {number | undefined}
@@ -186,7 +186,7 @@ export default class BackgroundJobsWorker {
     this.inflightPooledJobs = new Set()
     /** @type {Set<import("node:child_process").ChildProcess>} */
     this.pooledChildren = new Set()
-    /** @type {Map<import("node:child_process").ChildProcess, {createdAtMs: number, jobsRun: number, inflight: Map<string, {payload: import("./types.js").BackgroundJobPayload & {id: string}, resolve?: (value: void) => void}>, retiring: boolean, settling?: boolean}>} */
+    /** @type {Map<import("node:child_process").ChildProcess, {createdAtMs: number, jobsRun: number, inflight: Map<string, {payload: import("./types.js").BackgroundJobPayload & {id: string}, resolve?: (value: void) => void, timeoutTimer?: ReturnType<typeof setTimeout> | null}>, retiring: boolean, settling?: boolean, timeoutSigkillTimer?: ReturnType<typeof setTimeout> | null}>} */
     this.pooledChildStates = new Map()
   }
 
@@ -659,7 +659,9 @@ export default class BackgroundJobsWorker {
     if (!state) throw new Error("Pooled runner state missing")
 
     return new Promise((resolve) => {
-      state.inflight.set(payload.id, {payload, resolve})
+      const timeoutTimer = this._armPooledJobTimeout({child, jobId: payload.id})
+
+      state.inflight.set(payload.id, {payload, resolve, timeoutTimer})
       try {
         child.send({type: "job", payload})
       } catch (error) {
@@ -668,6 +670,58 @@ export default class BackgroundJobsWorker {
     })
   }
 
+  /**
+   * Arms a per-job wall-clock backstop for a pooled job. A pooled child hosts many
+   * concurrent jobs, so a single genuinely-hung job would otherwise pin its
+   * runner's concurrency slot forever — the lifetime recycle only retires a child
+   * once its in-flight set drains, which a hung job never does. On overrun the
+   * whole child is terminated so the hung job (and its siblings) requeue. Returns
+   * the timer, or null when no timeout is configured.
+   * @param {object} args - Options.
+   * @param {import("node:child_process").ChildProcess} args.child - Pooled child.
+   * @param {string} args.jobId - Job id whose overrun is guarded.
+   * @returns {ReturnType<typeof setTimeout> | null} - The armed timer, or null.
+   */
+  _armPooledJobTimeout({child, jobId}) {
+    const timeoutMs = this._resolveJobTimeoutMs()
+
+    if (!(typeof timeoutMs === "number" && timeoutMs > 0)) return null
+
+    return setTimeout(() => this._onPooledJobTimeout({child, jobId}), timeoutMs)
+  }
+
+  /**
+   * Fired when a pooled job overruns its timeout. Terminates the child running it
+   * (SIGTERM, then SIGKILL after the grace) — a hung JS job cannot be cancelled
+   * any other way. The non-clean exit flows through `_handlePooledChildFailure`,
+   * which reports every in-flight job on the child failed (so they requeue) and
+   * drops it from tracking; capacity is refilled on the next dispatch.
+   * @param {object} args - Options.
+   * @param {import("node:child_process").ChildProcess} args.child - Pooled child.
+   * @param {string} args.jobId - Job id that overran.
+   * @returns {void}
+   */
+  _onPooledJobTimeout({child, jobId}) {
+    const state = this.pooledChildStates.get(child)
+
+    // Already settling/gone, or the job finished in the race with this timer.
+    if (!state || state.settling || !state.inflight.has(jobId)) return
+
+    try {
+      child.kill("SIGTERM")
+    } catch {
+      // Child already exited; nothing to do.
+    }
+
+    state.timeoutSigkillTimer = setTimeout(() => {
+      try {
+        child.kill("SIGKILL")
+      } catch {
+        // Child already exited; nothing to do.
+      }
+    }, this.forkedChildSigkillGraceMs)
+  }
+
   /**
    * Creates a reusable pooled child.
    * @returns {import("node:child_process").ChildProcess} - New pooled child.
@@ -705,6 +759,7 @@ export default class BackgroundJobsWorker {
     const entry = state.inflight.get(record.jobId)
     if (!entry) return
 
+    if (entry.timeoutTimer) clearTimeout(entry.timeoutTimer)
     state.inflight.delete(record.jobId)
     state.jobsRun += 1
     const resolve = entry.resolve
@@ -789,7 +844,15 @@ export default class BackgroundJobsWorker {
   async _handlePooledChildFailure({child, error}) {
     const state = this.pooledChildStates.get(child)
     if (state?.settling) return
-    if (state) state.settling = true
+    if (state) {
+      state.settling = true
+      // Cancel this child's pending timers before its in-flight set is reported —
+      // the SIGKILL grace from a timeout kill, and every armed per-job backstop.
+      if (state.timeoutSigkillTimer) clearTimeout(state.timeoutSigkillTimer)
+      for (const inflightEntry of state.inflight.values()) {
+        if (inflightEntry.timeoutTimer) clearTimeout(inflightEntry.timeoutTimer)
+      }
+    }
     this.pooledChildren.delete(child)
     this.inflightProcessChildren.delete(child)
 
@@ -908,7 +971,7 @@ export default class BackgroundJobsWorker {
    * @returns {ForkedJobTimeoutState} - Timeout state.
    */
   _armForkedJobTimeout({child}) {
-    const timeoutMs = this._resolveForkedJobTimeoutMs()
+    const timeoutMs = this._resolveJobTimeoutMs()
     /** @type {ForkedJobTimeoutState} */
     const state = {timedOut: false, timeoutMs, timer: null, sigkillTimer: null}
 
@@ -920,12 +983,12 @@ export default class BackgroundJobsWorker {
   }
 
   /**
-   * Resolves the effective forked-job timeout in ms, or null when disabled. The
+   * Resolves the effective wall-clock job timeout in ms (shared by forked and pooled jobs), or null when disabled. The
    * constructor override wins; otherwise the value comes from the background-jobs
    * configuration. A non-positive value disables the backstop.
    * @returns {number | null} - Timeout in ms, or null when disabled.
    */
-  _resolveForkedJobTimeoutMs() {
+  _resolveJobTimeoutMs() {
     const raw = typeof this.jobTimeoutMsOverride === "number"
       ? this.jobTimeoutMsOverride
       : (this.configuration ? this.configuration.getBackgroundJobsConfig().jobTimeoutMs : null)

package/build/database/drivers/mysql/sql/alter-table.js

@@ -1,6 +1,48 @@
 // @ts-check
 
 import AlterTableBase from "../../../query/alter-table-base.js"
+import TableColumn from "../../../table-data/table-column.js"
 
 export default class VelociousDatabaseConnectionDriversMysqlSqlAlterTable extends AlterTableBase {
+  /**
+   * Builds MySQL ALTER TABLE statements, adding indexes atomically with columns.
+   * @returns {Promise<string[]>} - Resolves with SQL statements.
+   */
+  async toSQLs() {
+    const sqls = await super.toSQLs()
+    const indexes = this.tableData.getIndexes()
+
+    if (indexes.length === 0) return sqls
+    if (sqls.length !== 1) throw new Error("Expected one MySQL ALTER TABLE statement when adding indexes")
+
+    const options = this.getOptions()
+    let sql = sqls[0]
+
+    for (const index of indexes) {
+      sql += ", ADD"
+
+      if (index.getUnique()) sql += " UNIQUE"
+
+      sql += " INDEX"
+
+      const indexName = index.getName()
+
+      if (typeof indexName === "string") {
+        sql += ` ${options.quoteIndexName(indexName)}`
+      }
+
+      sql += " ("
+      sql += index
+        .getColumns()
+        .map((column) => {
+          const columnName = column instanceof TableColumn ? column.getName() : column
+
+          return options.quoteColumnName(columnName)
+        })
+        .join(", ")
+      sql += ")"
+    }
+
+    return [sql]
+  }
 }

package/build/database/query/preloader/belongs-to.js

@@ -58,8 +58,8 @@ export default class VelociousDatabaseQueryPreloaderBelongsTo {
 
     /**
      * Foreign key values.
-     * @type {Array<number | string>} */
-    const foreignKeyValues = []
+     * @type {Set<number | string>} */
+    const foreignKeyValues = new Set()
 
     for (const model of modelsToLoad) {
       const foreignKeyValue = /** @type {string | number | null | undefined} */ (model.readColumn(foreignKey))
@@ -69,7 +69,7 @@ export default class VelociousDatabaseQueryPreloaderBelongsTo {
       // throws on non-string primary-key columns.
       if (foreignKeyValue === null || foreignKeyValue === undefined) continue
 
-      if (!foreignKeyValues.includes(foreignKeyValue)) foreignKeyValues.push(foreignKeyValue)
+      foreignKeyValues.add(foreignKeyValue)
     }
 
     /**
@@ -83,7 +83,7 @@ export default class VelociousDatabaseQueryPreloaderBelongsTo {
     let targetModels = []
 
     // Only query when at least one model has a non-null foreign key.
-    if (foreignKeyValues.length > 0) {
+    if (foreignKeyValues.size > 0) {
       await ensureModelClassInitialized(targetModelClass, this.relationship.getConfiguration())
 
       /**
@@ -91,7 +91,7 @@ export default class VelociousDatabaseQueryPreloaderBelongsTo {
        * @type {Record<string, string | number | Array<string | number>>} */
       const whereArgs = {}
 
-      whereArgs[primaryKey] = foreignKeyValues
+      whereArgs[primaryKey] = [...foreignKeyValues]
 
       // Load target models to be preloaded on the given models
       let query = targetModelClass.where(whereArgs)
@@ -178,15 +178,15 @@ export default class VelociousDatabaseQueryPreloaderBelongsTo {
 
     /**
      * Foreign key values by type.
-     * @type {Record<string, Array<number | string>>} */
+     * @type {Record<string, Set<number | string>>} */
     const foreignKeyValuesByType = {}
 
     for (const meta of modelMeta) {
       if (meta.targetType === undefined || meta.targetType === null) continue
       if (meta.foreignKeyValue === undefined || meta.foreignKeyValue === null) continue
 
-      if (!foreignKeyValuesByType[meta.targetType]) foreignKeyValuesByType[meta.targetType] = []
-      if (!foreignKeyValuesByType[meta.targetType].includes(meta.foreignKeyValue)) foreignKeyValuesByType[meta.targetType].push(meta.foreignKeyValue)
+      if (!foreignKeyValuesByType[meta.targetType]) foreignKeyValuesByType[meta.targetType] = new Set()
+      foreignKeyValuesByType[meta.targetType].add(meta.foreignKeyValue)
     }
 
     /**
@@ -209,7 +209,7 @@ export default class VelociousDatabaseQueryPreloaderBelongsTo {
        * @type {Record<string, string | number | Array<string | number>>} */
       const whereArgs = {}
 
-      whereArgs[primaryKey] = foreignKeyValuesByType[targetType]
+      whereArgs[primaryKey] = [...foreignKeyValuesByType[targetType]]
 
       let query = targetModelClass.where(whereArgs)
 

package/build/database/query/preloader/has-many.js

@@ -130,8 +130,8 @@ export default class VelociousDatabaseQueryPreloaderHasMany {
 
     /**
      * Models primary key values.
-     * @type {Array<number | string>} */
-    const modelsPrimaryKeyValues = []
+     * @type {Set<number | string>} */
+    const modelsPrimaryKeyValues = new Set()
 
     /**
      * Models by primary key value.
@@ -148,7 +148,7 @@ export default class VelociousDatabaseQueryPreloaderHasMany {
 
       preloadCollections[primaryKeyValue] = []
 
-      if (!modelsPrimaryKeyValues.includes(primaryKeyValue)) modelsPrimaryKeyValues.push(primaryKeyValue)
+      modelsPrimaryKeyValues.add(primaryKeyValue)
       if (!(primaryKeyValue in modelsByPrimaryKeyValue)) modelsByPrimaryKeyValue[primaryKeyValue] = []
 
       modelsByPrimaryKeyValue[primaryKeyValue].push(model)
@@ -156,7 +156,7 @@ export default class VelociousDatabaseQueryPreloaderHasMany {
 
     // Step 1: Query the through table to build parent→target ID mapping
     const throughModels = await throughModelClass
-      .where({[throughForeignKey]: modelsPrimaryKeyValues})
+      .where({[throughForeignKey]: [...modelsPrimaryKeyValues]})
       .toArray()
 
     /**
@@ -260,8 +260,8 @@ export default class VelociousDatabaseQueryPreloaderHasMany {
 
     /**
      * Models primary key values.
-     * @type {Array<number | string>} */
-    const modelsPrimaryKeyValues = []
+     * @type {Set<number | string>} */
+    const modelsPrimaryKeyValues = new Set()
 
     /**
      * Models by primary key value.
@@ -278,7 +278,7 @@ export default class VelociousDatabaseQueryPreloaderHasMany {
 
       preloadCollections[primaryKeyValue] = []
 
-      if (!modelsPrimaryKeyValues.includes(primaryKeyValue)) modelsPrimaryKeyValues.push(primaryKeyValue)
+      modelsPrimaryKeyValues.add(primaryKeyValue)
       if (!(primaryKeyValue in modelsByPrimaryKeyValue)) modelsByPrimaryKeyValue[primaryKeyValue] = []
 
       modelsByPrimaryKeyValue[primaryKeyValue].push(model)
@@ -289,7 +289,7 @@ export default class VelociousDatabaseQueryPreloaderHasMany {
      * @type {Record<string, string | number | Array<string | number>>} */
     const whereArgs = {}
 
-    whereArgs[foreignKey] = modelsPrimaryKeyValues
+    whereArgs[foreignKey] = [...modelsPrimaryKeyValues]
 
     if (this.relationship.getPolymorphic()) {
       const typeColumn = this.relationship.getPolymorphicTypeColumn()

package/build/database/query/preloader/has-one.js

@@ -23,8 +23,8 @@ export default class VelociousDatabaseQueryPreloaderHasOne {
   async run() {
     /**
      * Models primary key values.
-     * @type {Array<number | string>} */
-    const modelsPrimaryKeyValues = []
+     * @type {Set<number | string>} */
+    const modelsPrimaryKeyValues = new Set()
 
     /**
      * Models by primary key value.
@@ -63,20 +63,20 @@ export default class VelociousDatabaseQueryPreloaderHasOne {
 
       preloadCollections[primaryKeyValue] = undefined
 
-      if (!modelsPrimaryKeyValues.includes(primaryKeyValue)) modelsPrimaryKeyValues.push(primaryKeyValue)
+      modelsPrimaryKeyValues.add(primaryKeyValue)
       if (!(primaryKeyValue in modelsByPrimaryKeyValue)) modelsByPrimaryKeyValue[primaryKeyValue] = []
 
       modelsByPrimaryKeyValue[primaryKeyValue].push(model)
     }
 
-    if (modelsPrimaryKeyValues.length == 0) return satisfiedTargets
+    if (modelsPrimaryKeyValues.size == 0) return satisfiedTargets
 
     /**
      * Where args.
      * @type {Record<string, string | number | Array<string | number>>} */
     const whereArgs = {}
 
-    whereArgs[foreignKey] = modelsPrimaryKeyValues
+    whereArgs[foreignKey] = [...modelsPrimaryKeyValues]
 
     if (this.relationship.getPolymorphic()) {
       const typeColumn = this.relationship.getPolymorphicTypeColumn()

package/build/database/tenants/schema-cloner.js

@@ -143,6 +143,37 @@ export default class SchemaCloner {
       tableData.addColumn(sourceColumn.getName(), this.columnArgsFromSourceColumn(sourceColumn, {isNewColumn: false}))
     }
 
+    if (this.targetDb.getType() === "mysql") {
+      const missingAutoIncrementColumnNames = new Set(
+        missingColumns
+          .filter((sourceColumn) => sourceColumn.getAutoIncrement())
+          .map((sourceColumn) => sourceColumn.getName())
+      )
+      const targetIndexesByName = new Map(
+        (await targetTable.getIndexes()).map((targetIndex) => [targetIndex.getName(), targetIndex])
+      )
+
+      for (const sourceIndex of await sourceTable.getIndexes()) {
+        if (
+          !sourceIndex.isPrimaryKey() &&
+          sourceIndex.isUnique() &&
+          missingAutoIncrementColumnNames.has(sourceIndex.getColumnNames()[0])
+        ) {
+          const targetIndex = targetIndexesByName.get(sourceIndex.getName())
+
+          if (targetIndex) {
+            if (this.indexesMatch(sourceIndex, targetIndex)) continue
+
+            this.assertSafeIndexReplacement({sourceIndex, tableName, targetIndex})
+            await this.dropTargetIndex({tableName, targetIndex})
+            targetIndexesByName.delete(targetIndex.getName())
+          }
+
+          tableData.addIndex(this.tableDataIndexFromSourceIndex(sourceIndex))
+        }
+      }
+    }
+
     for (const alterSql of await this.targetDb.alterTableSQLs(tableData)) {
       await this.targetDb.query(alterSql)
     }
@@ -186,15 +217,7 @@ export default class SchemaCloner {
 
       if (targetIndex) {
         if (!this.indexesMatch(sourceIndex, targetIndex)) {
-          // Replacing a non-unique index with a unique one is unsafe because
-          // the target may have duplicate values that will reject the new
-          // unique constraint, and the old index was already dropped by this
-          // point. The opposite direction (unique → non-unique) is always
-          // safe — non-unique indexes never fail on duplicate values.
-          if (sourceIndex.isUnique() && !targetIndex.isUnique()) {
-            throw new Error(`Schema clone index drift for ${tableName}.${sourceIndex.getName()}: cannot safely replace a non-unique index with a unique one.`)
-          }
-
+          this.assertSafeIndexReplacement({sourceIndex, tableName, targetIndex})
           await this.dropTargetIndex({tableName, targetIndex})
           targetIndexesByName.delete(targetIndex.getName())
           targetIndexSignatures.delete(this.indexSignature(targetIndex))
@@ -241,6 +264,20 @@ export default class SchemaCloner {
     }
   }
 
+  /**
+   * Refuses index replacements that could fail after the existing index is dropped.
+   * @param {{sourceIndex: import("../drivers/base-columns-index.js").default, tableName: string, targetIndex: import("../drivers/base-columns-index.js").default}} args - Source and target index definitions.
+   * @returns {void}
+   */
+  assertSafeIndexReplacement({sourceIndex, tableName, targetIndex}) {
+    // Replacing a non-unique index with a unique one is unsafe because the
+    // target may have duplicate values that will reject the new constraint.
+    // The opposite direction is safe because non-unique indexes accept them.
+    if (sourceIndex.isUnique() && !targetIndex.isUnique()) {
+      throw new Error(`Schema clone index drift for ${tableName}.${sourceIndex.getName()}: cannot safely replace a non-unique index with a unique one.`)
+    }
+  }
+
   /**
    * Baselines the target ledger so the cloned schema is recorded as already-applied.
    * @returns {Promise<string[]>} The versions newly recorded on the target.

package/build/src/background-jobs/worker.d.ts

@@ -22,7 +22,7 @@ export default class BackgroundJobsWorker {
      * @param {number} [args.pooledRunnerMaxLifetimeMs] - Override the per-runner recycle lifetime.
      * @param {number} [args.forkedChildSigkillGraceMs] - Override the grace period between SIGTERM and SIGKILL when reaping lingering process runners on stop.
      * @param {number} [args.heartbeatIntervalMs] - Override the liveness heartbeat interval (default 15000ms).
-     * @param {number} [args.jobTimeoutMs] - Override the forked-job wall-clock timeout from `configuration.getBackgroundJobsConfig()`. `0` disables it.
+     * @param {number} [args.jobTimeoutMs] - Override the wall-clock timeout for forked and pooled jobs from `configuration.getBackgroundJobsConfig()`. `0` disables it.
      */
     constructor({ configuration, host, port, maxConcurrentForkedJobs, maxConcurrentInlineJobs, pooledRunnerCount, pooledRunnerConcurrency, pooledRunnerMaxJobs, pooledRunnerMaxRssBytes, pooledRunnerMaxLifetimeMs, forkedChildSigkillGraceMs, heartbeatIntervalMs, jobTimeoutMs }?: {
         configuration?: import("../configuration.js").default | undefined;
@@ -87,7 +87,7 @@ export default class BackgroundJobsWorker {
      */
     forkedChildSigkillGraceMs: number;
     /**
-     * Constructor override for the forked-job wall-clock timeout. When unset the
+     * Constructor override for the forked and pooled wall-clock job timeout. When unset the
      * timeout is read from `configuration.getBackgroundJobsConfig().jobTimeoutMs`
      * at fork time (default: disabled).
      * @type {number | undefined}
@@ -142,7 +142,7 @@ export default class BackgroundJobsWorker {
     inflightPooledJobs: Set<Promise<void>>;
     /** @type {Set<import("node:child_process").ChildProcess>} */
     pooledChildren: Set<import("node:child_process").ChildProcess>;
-    /** @type {Map<import("node:child_process").ChildProcess, {createdAtMs: number, jobsRun: number, inflight: Map<string, {payload: import("./types.js").BackgroundJobPayload & {id: string}, resolve?: (value: void) => void}>, retiring: boolean, settling?: boolean}>} */
+    /** @type {Map<import("node:child_process").ChildProcess, {createdAtMs: number, jobsRun: number, inflight: Map<string, {payload: import("./types.js").BackgroundJobPayload & {id: string}, resolve?: (value: void) => void, timeoutTimer?: ReturnType<typeof setTimeout> | null}>, retiring: boolean, settling?: boolean, timeoutSigkillTimer?: ReturnType<typeof setTimeout> | null}>} */
     pooledChildStates: Map<import("node:child_process").ChildProcess, {
         createdAtMs: number;
         jobsRun: number;
@@ -151,9 +151,11 @@ export default class BackgroundJobsWorker {
                 id: string;
             };
             resolve?: (value: void) => void;
+            timeoutTimer?: ReturnType<typeof setTimeout> | null;
         }>;
         retiring: boolean;
         settling?: boolean;
+        timeoutSigkillTimer?: ReturnType<typeof setTimeout> | null;
     }>;
     /**
      * Runs start.
@@ -295,6 +297,37 @@ export default class BackgroundJobsWorker {
     _runPooledJob(payload: import("./types.js").BackgroundJobPayload & {
         id: string;
     }): Promise<void>;
+    /**
+     * Arms a per-job wall-clock backstop for a pooled job. A pooled child hosts many
+     * concurrent jobs, so a single genuinely-hung job would otherwise pin its
+     * runner's concurrency slot forever — the lifetime recycle only retires a child
+     * once its in-flight set drains, which a hung job never does. On overrun the
+     * whole child is terminated so the hung job (and its siblings) requeue. Returns
+     * the timer, or null when no timeout is configured.
+     * @param {object} args - Options.
+     * @param {import("node:child_process").ChildProcess} args.child - Pooled child.
+     * @param {string} args.jobId - Job id whose overrun is guarded.
+     * @returns {ReturnType<typeof setTimeout> | null} - The armed timer, or null.
+     */
+    _armPooledJobTimeout({ child, jobId }: {
+        child: import("node:child_process").ChildProcess;
+        jobId: string;
+    }): ReturnType<typeof setTimeout> | null;
+    /**
+     * Fired when a pooled job overruns its timeout. Terminates the child running it
+     * (SIGTERM, then SIGKILL after the grace) — a hung JS job cannot be cancelled
+     * any other way. The non-clean exit flows through `_handlePooledChildFailure`,
+     * which reports every in-flight job on the child failed (so they requeue) and
+     * drops it from tracking; capacity is refilled on the next dispatch.
+     * @param {object} args - Options.
+     * @param {import("node:child_process").ChildProcess} args.child - Pooled child.
+     * @param {string} args.jobId - Job id that overran.
+     * @returns {void}
+     */
+    _onPooledJobTimeout({ child, jobId }: {
+        child: import("node:child_process").ChildProcess;
+        jobId: string;
+    }): void;
     /**
      * Creates a reusable pooled child.
      * @returns {import("node:child_process").ChildProcess} - New pooled child.
@@ -397,12 +430,12 @@ export default class BackgroundJobsWorker {
         child: import("node:child_process").ChildProcess;
     }): ForkedJobTimeoutState;
     /**
-     * Resolves the effective forked-job timeout in ms, or null when disabled. The
+     * Resolves the effective wall-clock job timeout in ms (shared by forked and pooled jobs), or null when disabled. The
      * constructor override wins; otherwise the value comes from the background-jobs
      * configuration. A non-positive value disables the backstop.
      * @returns {number | null} - Timeout in ms, or null when disabled.
      */
-    _resolveForkedJobTimeoutMs(): number | null;
+    _resolveJobTimeoutMs(): number | null;
     /**
      * Fired when a forked runner overruns its timeout. Sends SIGTERM for a clean
      * shutdown, then SIGKILL after the grace for a runner that ignores it. The

package/build/src/background-jobs/worker.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"worker.d.ts","sourceRoot":"","sources":["../../../src/background-jobs/worker.js"],"names":[],"mappings":"AAiDA;;;;;;;GAOG;AAEH;IACE;;;;;;;;;;;;;;;;OAgBG;IACH,iRAdG;QAAqD,aAAa;QAC5C,IAAI;QACJ,IAAI;QACJ,uBAAuB;QACvB,uBAAuB;QACvB,iBAAiB;QACjB,uBAAuB;QACvB,mBAAmB;QACnB,uBAAuB;QACvB,yBAAyB;QACzB,yBAAyB;QACzB,mBAAmB;QACnB,YAAY;KACpC,EAmHA;IAjHC;;gEAE4D;IAC5D,sBADU,OAAO,CAAC,OAAO,qBAAqB,EAAE,OAAO,CAAC,CAC4C;IACpG;;mEAE+D;IAC/D,eADU,OAAO,qBAAqB,EAAE,OAAO,GAAG,SAAS,CAC7B;IAC9B,yBAAgB;IAChB,yBAAgB;IAChB;;;;;OAKG;IACH,iCAFU,MAAM,GAAG,SAAS,CAIf;IACb;;oCAEgC;IAChC,iCADU,MAAM,GAAG,SAAS,CAGf;IACb;;;;OAIG;IACH,yBAFU,MAAM,CAEwD;IACxE;;wBAEoB;IACpB,yBADU,MAAM,CACwD;IACxE,8CAAmE;IACnE,oDAA+E;IAC/E,gDAAuE;IACvE,oDAA8E;IAC9E,sDAAkF;IAClF,0BAA4D;IAC5D,gCAAwE;IACxE,4BAAkE;IAClE,gCAAwF;IACxF,kCAAyF;IACzF;;;;OAIG;IACH,2BAFU,MAAM,CAIiB;IACjC;;;;;OAKG;IACH,sBAFU,MAAM,GAAG,SAAS,CAE2D;IACvF,oBAAuB;IACvB,8DAA4B;IAC5B,4BAEyB;IACzB;;4DAEwD;IACxD,iBADU,UAAU,CAAC,OAAO,WAAW,CAAC,GAAG,SAAS,CACpB;IAChC;;;;;;OAMG;IACH,iBAFU,GAAG,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAEI;IAChC;;wCAEoC;IACpC,YADU,UAAU,GAAG,SAAS,CACL;IAC3B;;0DAEsD;IACtD,gBADU,4BAA4B,GAAG,SAAS,CACnB;IAC/B;;;;;;OAMG;IACH,oBAFU,GAAG,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAEO;IACnC;;;;OAIG;IACH,qBAFU,GAAG,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAEQ;IACpC;;;;;;OAMG;IACH,yBAFU,GAAG,CAAC,OAAO,oBAAoB,EAAE,YAAY,CAAC,CAEhB;IACxC,iCAAiC;IACjC,oBADW,GAAG,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CACM;IACnC,6DAA6D;IAC7D,gBADW,GAAG,CAAC,OAAO,oBAAoB,EAAE,YAAY,CAAC,CAC1B;IAC/B,yQAAyQ;IACzQ,mBADW,GAAG,CAAC,OAAO,oBAAoB,EAAE,YAAY,EAAE;QAAC,WAAW,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,QAAQ,EAAE,GAAG,CAAC,MAAM,EAAE;YAAC,OAAO,EAAE,OAAO,YAAY,EAAE,oBAAoB,GAAG;gBAAC,EAAE,EAAE,MAAM,CAAA;aAAC,CAAC;YAAC,OAAO,CAAC,EAAE,CAAC,KAAK,EAAE,IAAI,KAAK,IAAI,CAAA;SAAC,CAAC,CAAC;QAAC,QAAQ,EAAE,OAAO,CAAC;QAAC,QAAQ,CAAC,EAAE,OAAO,CAAA;KAAC,CAAC,CACnO;IAGpC;;;OAGG;IACH,SAFa,OAAO,CAAC,IAAI,CAAC,CAgCzB;IAED;;;;;;;;;;;;;;OAcG;IACH,qBAHG;QAAsB,SAAS;KAC/B,GAAU,OAAO,CAAC,IAAI,CAAC,CAiCzB;IAED;;;;;;OAMG;IACH,yBAJW,GAAG,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,cAClB,MAAM,GACJ,OAAO,CAAC,IAAI,CAAC,CAgBzB;IAED;;;;;OAKG;IACH,6BAFa,OAAO,CAAC,IAAI,CAAC,CAsBzB;IAED,0BAqCC;IAED;;;;;OAKG;IACH,mBAFa,IAAI,CAgBhB;IAED;;;OAGG;IACH,kBAFa,IAAI,CAOhB;IAED;;;;OAIG;IACH,oBAHW,OAAO,YAAY,EAAE,oBAAoB,GACvC,OAAO,CAAC,IAAI,CAAC,CAsBzB;IAED;;;;;;OAMG;IACH,6CAJG;QAA8D,aAAa,EAAnE,OAAO,YAAY,EAAE,0BAA0B;QACgB,OAAO,EAAtE,OAAO,YAAY,EAAE,oBAAoB,GAAG;YAAC,EAAE,EAAE,MAAM,CAAA;SAAC;KAChE,GAAU,OAAO,CAAC,IAAI,CAAC,CAMzB;IAED;;;;OAIG;IACH,0BAHW,OAAO,YAAY,EAAE,oBAAoB,GAAG;QAAC,EAAE,EAAE,MAAM,CAAA;KAAC,GACtD,IAAI,CAgChB;IAED;;;;OAIG;IACH,kCAHW,OAAO,YAAY,EAAE,oBAAoB,GACvC,OAAO,YAAY,EAAE,0BAA0B,CAW3D;IAED;;;;OAIG;IACH,uCAHW,MAAM,GACJ,OAAO,YAAY,EAAE,0BAA0B,CAQ3D;IAED;;;;OAIG;IACH,6BAHW,OAAO,CAAC,IAAI,CAAC,GACX,IAAI,CAyBhB;IAED;;;;OAIG;IACH,gCAHW,OAAO,YAAY,EAAE,oBAAoB,GAAG;QAAC,EAAE,EAAE,MAAM,CAAA;KAAC,GACtD,OAAO,CAAC,IAAI,CAAC,CAyBzB;IAED;;;;;OAKG;IACH,uBAFa,IAAI,CAUhB;IAED;;;OAGG;IACH,iBAFa,OAAO,YAAY,EAAE,0BAA0B,GAAG,IAAI,CAgBlE;IAED;;;;OAIG;IACH,2BAHW,OAAO,CAAC,IAAI,CAAC,GACX,IAAI,CAWhB;IAED;;;;;OAKG;IACH,yBAFa,MAAM,CAgBlB;IAED;;;;;;;OAOG;IACH,uBAHW,OAAO,YAAY,EAAE,oBAAoB,GAAG;QAAC,EAAE,EAAE,MAAM,CAAA;KAAC,GACtD,OAAO,CAAC,IAAI,CAAC,CAuBzB;IAED;;;OAGG;IACH,sBAFa,OAAO,oBAAoB,EAAE,YAAY,CAiBrD;IAED;;;;;;;OAOG;IACH,8CAJG;QAAwD,KAAK,EAArD,OAAO,oBAAoB,EAAE,YAAY;QACjC,OAAO,EAAf,OAAC;KACT,GAAU,IAAI,CAmChB;IAED;;;;;;;;OAQG;IACH,+BAHW,OAAO,oBAAoB,EAAE,YAAY,GACvC,IAAI,CAUhB;IAED;;;;OAIG;IACH,2BAHW,OAAO,oBAAoB,EAAE,YAAY,GACvC,IAAI,CAOhB;IAED;;;;OAIG;IACH,0BAHW,OAAO,oBAAoB,EAAE,YAAY,GACvC,IAAI,CAOhB;IAED;;;;;;;;;;OAUG;IACH,4CAJG;QAAwD,KAAK,EAArD,OAAO,oBAAoB,EAAE,YAAY;QACjC,KAAK,EAAb,OAAC;KACT,GAAU,OAAO,CAAC,IAAI,CAAC,CAwBzB;IAED;;;;OAIG;IACH,uBAHW,OAAO,YAAY,EAAE,oBAAoB,GACvC,OAAO,CAAC,IAAI,CAAC,CAkBzB;IAED;;;;OAIG;IACH,kBAHW,OAAO,YAAY,EAAE,oBAAoB,GAAG;QAAC,EAAE,EAAE,MAAM,CAAA;KAAC,GACtD,OAAO,CAAC,IAAI,CAAC,CAYzB;IAED;;;OAGG;IACH,sBAFa,OAAO,oBAAoB,EAAE,YAAY,CAmBrD;IAED;;;;;;OAMG;IACH,wCAJG;QAAwD,KAAK,EAArD,OAAO,oBAAoB,EAAE,YAAY;QACsB,OAAO,EAAtE,OAAO,YAAY,EAAE,oBAAoB,GAAG;YAAC,EAAE,EAAE,MAAM,CAAA;SAAC;KAChE,GAAU,OAAO,CAAC,IAAI,CAAC,CAezB;IAED;;;;;;;;;;;OAWG;IACH,gCAHG;QAAwD,KAAK,EAArD,OAAO,oBAAoB,EAAE,YAAY;KACjD,GAAU,qBAAqB,CAYjC;IAED;;;;;OAKG;IACH,8BAFa,MAAM,GAAG,IAAI,CAazB;IAED;;;;;;;;;OASG;IACH,sCAJG;QAAwD,KAAK,EAArD,OAAO,oBAAoB,EAAE,YAAY;QACb,KAAK,EAAjC,qBAAqB;KAC7B,GAAU,IAAI,CAkBhB;IAED;;;;;OAKG;IACH,8BAHW,qBAAqB,GACnB,IAAI,CAYhB;IAED;;;;;;;;;;OAUG;IACH,gFARG;QAAwD,KAAK,EAArD,OAAO,oBAAoB,EAAE,YAAY;QACrB,IAAI,EAAxB,MAAM,GAAG,IAAI;QACiD,MAAM,EAApE,MAAM,cAAc,SAAS,EAAE,SAAS,CAAC,OAAO,GAAG,IAAI;QACQ,OAAO,EAAtE,OAAO,YAAY,EAAE,oBAAoB,GAAG;YAAC,EAAE,EAAE,MAAM,CAAA;SAAC;QAC5B,OAAO,EAAnC,CAAC,KAAK,EAAE,IAAI,KAAK,IAAI;QACQ,YAAY;KACjD,GAAU,IAAI,CAiBhB;IAED;;;;;;OAMG;IACH,4CAJG;QAA4B,IAAI,EAAxB,MAAM,GAAG,IAAI;QACiD,MAAM,EAApE,MAAM,cAAc,SAAS,EAAE,SAAS,CAAC,OAAO,GAAG,IAAI;KAC/D,GAAU,OAAO,CAInB;IAED;;;;;;;;OAQG;IACH,4DANG;QAAwD,KAAK,EAArD,OAAO,oBAAoB,EAAE,YAAY;QAC7B,KAAK,EAAjB,KAAK;QAC0D,OAAO,EAAtE,OAAO,YAAY,EAAE,oBAAoB,GAAG;YAAC,EAAE,EAAE,MAAM,CAAA;SAAC;QAC5B,OAAO,EAAnC,CAAC,KAAK,EAAE,IAAI,KAAK,IAAI;KAC7B,GAAU,IAAI,CAQhB;IAED;;;;;;OAMG;IACH,uCAJG;QAAwD,KAAK,EAArD,OAAO,oBAAoB,EAAE,YAAY;QACsB,OAAO,EAAtE,OAAO,YAAY,EAAE,oBAAoB,GAAG;YAAC,EAAE,EAAE,MAAM,CAAA;SAAC;KAChE,GAAU,IAAI,CAShB;IAED;;;;;;OAMG;IACH,8CAJG;QAAuE,OAAO,EAAtE,OAAO,YAAY,EAAE,oBAAoB,GAAG;YAAC,EAAE,EAAE,MAAM,CAAA;SAAC;QAChD,KAAK,EAAb,OAAC;KACT,GAAU,IAAI,CAWhB;IAED;;;;OAIG;IACH,mBAHW,OAAO,YAAY,EAAE,oBAAoB,GACvC,OAAO,CAAC,IAAI,CAAC,CAwCzB;IAED;;;;;;;;;;OAUG;IACH,+EARG;QAAqB,KAAK,EAAlB,MAAM;QACuB,MAAM,EAAnC,WAAW,GAAG,QAAQ;QACb,KAAK,GAAd,OAAC;QACa,SAAS;QACT,aAAa;QACb,QAAQ;KAC9B,GAAU,OAAO,CAAC,IAAI,CAAC,CAUzB;IAED;;;;;;;;;;;;OAYG;IACH,2FARG;QAAqB,KAAK,EAAlB,MAAM;QACuB,MAAM,EAAnC,WAAW,GAAG,QAAQ;QACb,KAAK,GAAd,OAAC;QACa,SAAS;QACT,aAAa;QACb,QAAQ;KAC9B,GAAU,IAAI,CAahB;CACF;;;;;;;;cA3lCa,OAAO;;;;eACP,MAAM,GAAG,IAAI;;;;WACb,UAAU,CAAC,OAAO,UAAU,CAAC,GAAG,IAAI;;;;kBACpC,UAAU,CAAC,OAAO,UAAU,CAAC,GAAG,IAAI;;uBAnD3B,kBAAkB;yCAGA,sBAAsB"}
+{"version":3,"file":"worker.d.ts","sourceRoot":"","sources":["../../../src/background-jobs/worker.js"],"names":[],"mappings":"AAiDA;;;;;;;GAOG;AAEH;IACE;;;;;;;;;;;;;;;;OAgBG;IACH,iRAdG;QAAqD,aAAa;QAC5C,IAAI;QACJ,IAAI;QACJ,uBAAuB;QACvB,uBAAuB;QACvB,iBAAiB;QACjB,uBAAuB;QACvB,mBAAmB;QACnB,uBAAuB;QACvB,yBAAyB;QACzB,yBAAyB;QACzB,mBAAmB;QACnB,YAAY;KACpC,EAmHA;IAjHC;;gEAE4D;IAC5D,sBADU,OAAO,CAAC,OAAO,qBAAqB,EAAE,OAAO,CAAC,CAC4C;IACpG;;mEAE+D;IAC/D,eADU,OAAO,qBAAqB,EAAE,OAAO,GAAG,SAAS,CAC7B;IAC9B,yBAAgB;IAChB,yBAAgB;IAChB;;;;;OAKG;IACH,iCAFU,MAAM,GAAG,SAAS,CAIf;IACb;;oCAEgC;IAChC,iCADU,MAAM,GAAG,SAAS,CAGf;IACb;;;;OAIG;IACH,yBAFU,MAAM,CAEwD;IACxE;;wBAEoB;IACpB,yBADU,MAAM,CACwD;IACxE,8CAAmE;IACnE,oDAA+E;IAC/E,gDAAuE;IACvE,oDAA8E;IAC9E,sDAAkF;IAClF,0BAA4D;IAC5D,gCAAwE;IACxE,4BAAkE;IAClE,gCAAwF;IACxF,kCAAyF;IACzF;;;;OAIG;IACH,2BAFU,MAAM,CAIiB;IACjC;;;;;OAKG;IACH,sBAFU,MAAM,GAAG,SAAS,CAE2D;IACvF,oBAAuB;IACvB,8DAA4B;IAC5B,4BAEyB;IACzB;;4DAEwD;IACxD,iBADU,UAAU,CAAC,OAAO,WAAW,CAAC,GAAG,SAAS,CACpB;IAChC;;;;;;OAMG;IACH,iBAFU,GAAG,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAEI;IAChC;;wCAEoC;IACpC,YADU,UAAU,GAAG,SAAS,CACL;IAC3B;;0DAEsD;IACtD,gBADU,4BAA4B,GAAG,SAAS,CACnB;IAC/B;;;;;;OAMG;IACH,oBAFU,GAAG,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAEO;IACnC;;;;OAIG;IACH,qBAFU,GAAG,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAEQ;IACpC;;;;;;OAMG;IACH,yBAFU,GAAG,CAAC,OAAO,oBAAoB,EAAE,YAAY,CAAC,CAEhB;IACxC,iCAAiC;IACjC,oBADW,GAAG,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CACM;IACnC,6DAA6D;IAC7D,gBADW,GAAG,CAAC,OAAO,oBAAoB,EAAE,YAAY,CAAC,CAC1B;IAC/B,0XAA0X;IAC1X,mBADW,GAAG,CAAC,OAAO,oBAAoB,EAAE,YAAY,EAAE;QAAC,WAAW,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,QAAQ,EAAE,GAAG,CAAC,MAAM,EAAE;YAAC,OAAO,EAAE,OAAO,YAAY,EAAE,oBAAoB,GAAG;gBAAC,EAAE,EAAE,MAAM,CAAA;aAAC,CAAC;YAAC,OAAO,CAAC,EAAE,CAAC,KAAK,EAAE,IAAI,KAAK,IAAI,CAAC;YAAC,YAAY,CAAC,EAAE,UAAU,CAAC,OAAO,UAAU,CAAC,GAAG,IAAI,CAAA;SAAC,CAAC,CAAC;QAAC,QAAQ,EAAE,OAAO,CAAC;QAAC,QAAQ,CAAC,EAAE,OAAO,CAAC;QAAC,mBAAmB,CAAC,EAAE,UAAU,CAAC,OAAO,UAAU,CAAC,GAAG,IAAI,CAAA;KAAC,CAAC,CACpV;IAGpC;;;OAGG;IACH,SAFa,OAAO,CAAC,IAAI,CAAC,CAgCzB;IAED;;;;;;;;;;;;;;OAcG;IACH,qBAHG;QAAsB,SAAS;KAC/B,GAAU,OAAO,CAAC,IAAI,CAAC,CAiCzB;IAED;;;;;;OAMG;IACH,yBAJW,GAAG,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,cAClB,MAAM,GACJ,OAAO,CAAC,IAAI,CAAC,CAgBzB;IAED;;;;;OAKG;IACH,6BAFa,OAAO,CAAC,IAAI,CAAC,CAsBzB;IAED,0BAqCC;IAED;;;;;OAKG;IACH,mBAFa,IAAI,CAgBhB;IAED;;;OAGG;IACH,kBAFa,IAAI,CAOhB;IAED;;;;OAIG;IACH,oBAHW,OAAO,YAAY,EAAE,oBAAoB,GACvC,OAAO,CAAC,IAAI,CAAC,CAsBzB;IAED;;;;;;OAMG;IACH,6CAJG;QAA8D,aAAa,EAAnE,OAAO,YAAY,EAAE,0BAA0B;QACgB,OAAO,EAAtE,OAAO,YAAY,EAAE,oBAAoB,GAAG;YAAC,EAAE,EAAE,MAAM,CAAA;SAAC;KAChE,GAAU,OAAO,CAAC,IAAI,CAAC,CAMzB;IAED;;;;OAIG;IACH,0BAHW,OAAO,YAAY,EAAE,oBAAoB,GAAG;QAAC,EAAE,EAAE,MAAM,CAAA;KAAC,GACtD,IAAI,CAgChB;IAED;;;;OAIG;IACH,kCAHW,OAAO,YAAY,EAAE,oBAAoB,GACvC,OAAO,YAAY,EAAE,0BAA0B,CAW3D;IAED;;;;OAIG;IACH,uCAHW,MAAM,GACJ,OAAO,YAAY,EAAE,0BAA0B,CAQ3D;IAED;;;;OAIG;IACH,6BAHW,OAAO,CAAC,IAAI,CAAC,GACX,IAAI,CAyBhB;IAED;;;;OAIG;IACH,gCAHW,OAAO,YAAY,EAAE,oBAAoB,GAAG;QAAC,EAAE,EAAE,MAAM,CAAA;KAAC,GACtD,OAAO,CAAC,IAAI,CAAC,CAyBzB;IAED;;;;;OAKG;IACH,uBAFa,IAAI,CAUhB;IAED;;;OAGG;IACH,iBAFa,OAAO,YAAY,EAAE,0BAA0B,GAAG,IAAI,CAgBlE;IAED;;;;OAIG;IACH,2BAHW,OAAO,CAAC,IAAI,CAAC,GACX,IAAI,CAWhB;IAED;;;;;OAKG;IACH,yBAFa,MAAM,CAgBlB;IAED;;;;;;;OAOG;IACH,uBAHW,OAAO,YAAY,EAAE,oBAAoB,GAAG;QAAC,EAAE,EAAE,MAAM,CAAA;KAAC,GACtD,OAAO,CAAC,IAAI,CAAC,CAyBzB;IAED;;;;;;;;;;;OAWG;IACH,uCAJG;QAAwD,KAAK,EAArD,OAAO,oBAAoB,EAAE,YAAY;QAC5B,KAAK,EAAlB,MAAM;KACd,GAAU,UAAU,CAAC,OAAO,UAAU,CAAC,GAAG,IAAI,CAQhD;IAED;;;;;;;;;;OAUG;IACH,sCAJG;QAAwD,KAAK,EAArD,OAAO,oBAAoB,EAAE,YAAY;QAC5B,KAAK,EAAlB,MAAM;KACd,GAAU,IAAI,CAqBhB;IAED;;;OAGG;IACH,sBAFa,OAAO,oBAAoB,EAAE,YAAY,CAiBrD;IAED;;;;;;;OAOG;IACH,8CAJG;QAAwD,KAAK,EAArD,OAAO,oBAAoB,EAAE,YAAY;QACjC,OAAO,EAAf,OAAC;KACT,GAAU,IAAI,CAoChB;IAED;;;;;;;;OAQG;IACH,+BAHW,OAAO,oBAAoB,EAAE,YAAY,GACvC,IAAI,CAUhB;IAED;;;;OAIG;IACH,2BAHW,OAAO,oBAAoB,EAAE,YAAY,GACvC,IAAI,CAOhB;IAED;;;;OAIG;IACH,0BAHW,OAAO,oBAAoB,EAAE,YAAY,GACvC,IAAI,CAOhB;IAED;;;;;;;;;;OAUG;IACH,4CAJG;QAAwD,KAAK,EAArD,OAAO,oBAAoB,EAAE,YAAY;QACjC,KAAK,EAAb,OAAC;KACT,GAAU,OAAO,CAAC,IAAI,CAAC,CAgCzB;IAED;;;;OAIG;IACH,uBAHW,OAAO,YAAY,EAAE,oBAAoB,GACvC,OAAO,CAAC,IAAI,CAAC,CAkBzB;IAED;;;;OAIG;IACH,kBAHW,OAAO,YAAY,EAAE,oBAAoB,GAAG;QAAC,EAAE,EAAE,MAAM,CAAA;KAAC,GACtD,OAAO,CAAC,IAAI,CAAC,CAYzB;IAED;;;OAGG;IACH,sBAFa,OAAO,oBAAoB,EAAE,YAAY,CAmBrD;IAED;;;;;;OAMG;IACH,wCAJG;QAAwD,KAAK,EAArD,OAAO,oBAAoB,EAAE,YAAY;QACsB,OAAO,EAAtE,OAAO,YAAY,EAAE,oBAAoB,GAAG;YAAC,EAAE,EAAE,MAAM,CAAA;SAAC;KAChE,GAAU,OAAO,CAAC,IAAI,CAAC,CAezB;IAED;;;;;;;;;;;OAWG;IACH,gCAHG;QAAwD,KAAK,EAArD,OAAO,oBAAoB,EAAE,YAAY;KACjD,GAAU,qBAAqB,CAYjC;IAED;;;;;OAKG;IACH,wBAFa,MAAM,GAAG,IAAI,CAazB;IAED;;;;;;;;;OASG;IACH,sCAJG;QAAwD,KAAK,EAArD,OAAO,oBAAoB,EAAE,YAAY;QACb,KAAK,EAAjC,qBAAqB;KAC7B,GAAU,IAAI,CAkBhB;IAED;;;;;OAKG;IACH,8BAHW,qBAAqB,GACnB,IAAI,CAYhB;IAED;;;;;;;;;;OAUG;IACH,gFARG;QAAwD,KAAK,EAArD,OAAO,oBAAoB,EAAE,YAAY;QACrB,IAAI,EAAxB,MAAM,GAAG,IAAI;QACiD,MAAM,EAApE,MAAM,cAAc,SAAS,EAAE,SAAS,CAAC,OAAO,GAAG,IAAI;QACQ,OAAO,EAAtE,OAAO,YAAY,EAAE,oBAAoB,GAAG;YAAC,EAAE,EAAE,MAAM,CAAA;SAAC;QAC5B,OAAO,EAAnC,CAAC,KAAK,EAAE,IAAI,KAAK,IAAI;QACQ,YAAY;KACjD,GAAU,IAAI,CAiBhB;IAED;;;;;;OAMG;IACH,4CAJG;QAA4B,IAAI,EAAxB,MAAM,GAAG,IAAI;QACiD,MAAM,EAApE,MAAM,cAAc,SAAS,EAAE,SAAS,CAAC,OAAO,GAAG,IAAI;KAC/D,GAAU,OAAO,CAInB;IAED;;;;;;;;OAQG;IACH,4DANG;QAAwD,KAAK,EAArD,OAAO,oBAAoB,EAAE,YAAY;QAC7B,KAAK,EAAjB,KAAK;QAC0D,OAAO,EAAtE,OAAO,YAAY,EAAE,oBAAoB,GAAG;YAAC,EAAE,EAAE,MAAM,CAAA;SAAC;QAC5B,OAAO,EAAnC,CAAC,KAAK,EAAE,IAAI,KAAK,IAAI;KAC7B,GAAU,IAAI,CAQhB;IAED;;;;;;OAMG;IACH,uCAJG;QAAwD,KAAK,EAArD,OAAO,oBAAoB,EAAE,YAAY;QACsB,OAAO,EAAtE,OAAO,YAAY,EAAE,oBAAoB,GAAG;YAAC,EAAE,EAAE,MAAM,CAAA;SAAC;KAChE,GAAU,IAAI,CAShB;IAED;;;;;;OAMG;IACH,8CAJG;QAAuE,OAAO,EAAtE,OAAO,YAAY,EAAE,oBAAoB,GAAG;YAAC,EAAE,EAAE,MAAM,CAAA;SAAC;QAChD,KAAK,EAAb,OAAC;KACT,GAAU,IAAI,CAWhB;IAED;;;;OAIG;IACH,mBAHW,OAAO,YAAY,EAAE,oBAAoB,GACvC,OAAO,CAAC,IAAI,CAAC,CAwCzB;IAED;;;;;;;;;;OAUG;IACH,+EARG;QAAqB,KAAK,EAAlB,MAAM;QACuB,MAAM,EAAnC,WAAW,GAAG,QAAQ;QACb,KAAK,GAAd,OAAC;QACa,SAAS;QACT,aAAa;QACb,QAAQ;KAC9B,GAAU,OAAO,CAAC,IAAI,CAAC,CAUzB;IAED;;;;;;;;;;;;OAYG;IACH,2FARG;QAAqB,KAAK,EAAlB,MAAM;QACuB,MAAM,EAAnC,WAAW,GAAG,QAAQ;QACb,KAAK,GAAd,OAAC;QACa,SAAS;QACT,aAAa;QACb,QAAQ;KAC9B,GAAU,IAAI,CAahB;CACF;;;;;;;;cA1pCa,OAAO;;;;eACP,MAAM,GAAG,IAAI;;;;WACb,UAAU,CAAC,OAAO,UAAU,CAAC,GAAG,IAAI;;;;kBACpC,UAAU,CAAC,OAAO,UAAU,CAAC,GAAG,IAAI;;uBAnD3B,kBAAkB;yCAGA,sBAAsB"}