@12-apps/jobs 4.5.0 → 4.6.0

package/ADOPTING.md

@@ -157,6 +157,7 @@ default from the environment, which is what makes the mount one line:
 | `events` | — | `JobEvents` — dead-letters, completions and removed schedules, for the host to wire to its own notifier / audit / realtime. See below. |
 | `retention` | package default | How long finished jobs are kept: a day of successes, a week of failures. Override for a longer support window. All four numbers must be positive and finite — see the warning below. |
 | `defaultConcurrency` | `5` | Per-queue concurrency when no job on the queue states one. A stated `concurrency: 1` still wins. |
+| `stall` | BullMQ's numbers | `JobStallConfig` — `lockDurationMs` (30 s), `stalledIntervalMs` (30 s) and `maxStalledCount` (1), for every queue and per queue under `queues`. See "When a worker stalls" below. |
 | `db` | — | `() => SweepLeaseDb` — enables `withSweepLease`. Without it the lease throws on first use (loud, never a silent skip). |
 | `installShutdownHooks` | `true` | `SIGTERM`/`SIGINT` drain in-flight jobs (workers only). Off in tests. |
 
@@ -208,6 +209,9 @@ An observer that throws or rejects is logged and swallowed — somebody else's
 code must never be able to fail the job it is watching, nor the reconcile that
 was cleaning up a stale schedule.
 
+`onJobStalled` fires when a running job lost its lock and was put back to run
+again; see the next section.
+
 `onScheduleRemoved` fires only for REMOVAL, not installation. Installation is
 an idempotent upsert that runs on every boot of every worker, so auditing it
 would write one row per schedule per deploy; removal is a deploy permanently
@@ -278,6 +282,56 @@ owns the schema folder:
 published as generic, and it closes three fail-open paths that were invisible
 by construction.
 
+### When a worker stalls
+
+A worker holds a lock on every job it runs and renews it every half
+`lockDurationMs`. If its event loop is blocked, the process is paused or Redis
+stops answering for longer than the lock, the lock expires. The worker then
+logs `could not renew lock for job …` and, when the handler returns,
+`Missing lock for job … moveToFinished`, because it can no longer record the
+result. The stalled checker, every `stalledIntervalMs`, finds the job without a
+lock and moves it back to `wait`, and it runs again. **A stall is a re-run,
+never a lost run.** That is one more reason handlers must be idempotent.
+
+What the driver does about it:
+
+- **Every stall is an ERROR line** through the host's logger, naming the job,
+  its run id, how many times it has been started and how many times it has stalled, and it fires
+  `events.onJobStalled`. The two lock errors above stay errors too. The job
+  recovers on its own, but the stall that caused it hits every job on that
+  worker, and it needs looking at.
+- **Every failed attempt is an ERROR line** naming the job, its run id, the
+  attempt out of the budget, and whether it will be retried.
+- **A job that stalls more than `maxStalledCount` times is failed**, as
+  unrecoverable. It lands in the failed set (kept for `retention.failed`) and
+  reaches `onJobFailed` with `terminal: true`, the dead-letter.
+- **Scheduled jobs are bounded too.** BullMQ never applies `maxStalledCount` to
+  a job-scheduler job, so a tick whose handler takes the worker down every time
+  would be restarted forever. The driver sets BullMQ's `maxStartedAttempts` to
+  the queue's largest `attempts` plus `maxStalledCount`. The start that goes
+  over it is failed as a terminal dead-letter, and the next tick of the
+  schedule is a fresh job. One consequence for anyone retrying a failed job by
+  hand: its earlier starts still count toward that cap, so a job retried by
+  hand can be refused as it starts. Retry with
+  `job.retry("failed", { resetAttemptsMade: true, resetAttemptsStarted: true })`
+  to give it a fresh budget.
+
+A queue whose handlers can legitimately hold the event loop for a while (a
+single-flight sweep queue doing batch work, say) can take a longer lock without
+changing the others:
+
+```ts
+createApiJobs({
+  jobs: () => import("./lib/jobs"),
+  stall: { queues: { [SWEEP_QUEUE]: { lockDurationMs: 60_000 } } },
+});
+```
+
+A longer lock is also a slower recovery when a worker really dies, since the
+job is only put back once its lock has expired. Every number is validated:
+`createApiJobs` refuses a bad one at assembly with `InvalidJobStallError`, and
+`createBullMqJobDriver` refuses it again.
+
 ## What changed in behaviour
 
 Nothing was **removed** from the API — every 2.0.0 export still exists with the
@@ -304,6 +358,7 @@ from success.
 | `events?: JobEvents` on `createApiJobs`, and on both driver factories | Dead-letters (`onJobFailed` with `terminal`), completions (`onJobCompleted`) and removed schedules (`onScheduleRemoved`). The package still notifies/audits/publishes nothing itself. |
 | `retention?: JobRetention`, `defaultConcurrency?: number` | The two operational numbers that were hardcoded. The defaults are unchanged (a day / a week; concurrency 5), so omitting them is a no-op. `retention` is validated at assembly and again in the driver — a non-positive window stops bounding the backend rather than shrinking it. |
 | `assertValidRetention`, `InvalidJobRetentionError` | The retention check, exported so a host that assembles its own window can run it. Lives in `core`, so importing it never pulls `bullmq` into a bundle that only enqueues. |
+| `stall?: JobStallConfig`, `events.onJobStalled`, `DEFAULT_STALL_POLICY`, `assertValidStall`, `InvalidJobStallError` | The worker's lock and stall settings, which used to be BullMQ's implicit defaults, now spelled out and configurable per queue. Each stall is reported as an error and to the host. A job, scheduled or not, that keeps stalling is failed as a dead-letter instead of being restarted forever. The defaults are BullMQ's own numbers, so omitting `stall` changes nothing but the new bound on restarts. |
 | `DEFAULT_QUEUE` | The queue name a definition falls back to, exported instead of duplicated as a literal in every host. |
 | `resolveRegisteredJob`, `InvalidJobDefinitionError`, `NoJobsRegisteredError`, `JobsConfigError` | The gate and the three refusals, so a host can catch them by type. |
 

package/README.md

@@ -103,6 +103,13 @@ simply stops happening:
   scheduler removed from Redis, instead of firing forever at a handler that no
   longer exists. That removal is reported to `events.onScheduleRemoved`,
   because it is destructive and a deploy did it.
+- **A stalled job is re-run, and reported.** When a worker's lock on a job
+  expires (a blocked event loop, a paused process, a silent Redis link), BullMQ
+  puts the job back and runs it again. Each stall is an error line and
+  `events.onJobStalled`. A job that keeps stalling (more than `maxStalledCount`
+  times, or, for a scheduled job, more starts than its attempts plus that) is
+  failed as a terminal dead-letter instead of being restarted forever. The
+  lock and stall numbers are configurable per queue through `stall`.
 - The `inline` driver honours `attempts` but not delays or schedules, and both
   omissions are logged rather than silent. It is refused in production —
   by name, by env var and by instance.
@@ -130,6 +137,8 @@ createApiJobs({
       if (terminal) void notifyOperators(name, error);
     },
     onJobCompleted: ({ name, runId }) => void publishRealtime(name, runId),
+    // A job lost its lock mid-run and was put back to run again.
+    onJobStalled: ({ name, runId, stalledCount }) => void notifyOperators(name, runId, stalledCount),
     // A deploy just cancelled a recurring job, permanently.
     onScheduleRemoved: ({ name, queue }) => void audit("schedule.removed", { name, queue }),
   },

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@12-apps/jobs",
-  "version": "4.5.0",
+  "version": "4.6.0",
   "type": "module",
   "sideEffects": false,
   "description": "Generic background-job library: a typed job registry with retries, exponential backoff and cron schedules, behind a swappable driver port (BullMQ/Redis in production, inline execution in tests and zero-config dev). The runtime half (./server) is one factory — createApiJobs: driver resolution, worker bootstrap, graceful drain, the single-writer sweep lease and a health endpoint — with a Hono adapter (./hono) and the package-owned SweepLease Prisma partial + migrations. Knows nothing about the host app's domain, ORM or transport.",

package/src/core/stall.ts

@@ -0,0 +1,91 @@
+import type { JobStallConfig, JobStallPolicy } from "./types";
+
+/**
+ * The stall settings a host may configure: how long a worker's lock
+ * on a running job lasts, how often the stalled checker looks, and how many
+ * re-runs a stalled job gets before it is failed.
+ *
+ * Lives in `core`, beside `retention`, for the same reason: the root entry
+ * point re-exports the error, and nothing here may pull `bullmq` into a bundle
+ * that only enqueues. It is arithmetic on three numbers.
+ */
+
+/**
+ * BullMQ's own defaults, spelled out. The driver passes them explicitly
+ * rather than leaving them implicit, so the numbers a stall is measured
+ * against are readable in this package instead of inside BullMQ's worker
+ * constructor, and a BullMQ upgrade that changed them could not move them
+ * silently.
+ */
+export const DEFAULT_STALL_POLICY: JobStallPolicy = {
+  lockDurationMs: 30_000,
+  stalledIntervalMs: 30_000,
+  maxStalledCount: 1,
+};
+
+/** Raised for a stall setting BullMQ would reject, or one that cannot work. */
+export class InvalidJobStallError extends Error {
+  constructor(field: string, value: unknown, rule: string) {
+    super(`stall.${field} must be ${rule}, got ${JSON.stringify(value)}.`);
+    this.name = "InvalidJobStallError";
+  }
+}
+
+function assertPolicyFields(where: string, policy: Partial<JobStallPolicy>): void {
+  for (const field of ["lockDurationMs", "stalledIntervalMs"] as const) {
+    const value = policy[field];
+    if (value === undefined) continue;
+    // A zero or NaN lock is renewed never and expires at once: every job
+    // would be reported stalled and re-run while it is still running.
+    if (typeof value !== "number" || !Number.isInteger(value) || value <= 0) {
+      throw new InvalidJobStallError(`${where}${field}`, value, "a positive integer (ms)");
+    }
+  }
+  const count = policy.maxStalledCount;
+  if (count === undefined) return;
+  if (typeof count !== "number" || !Number.isInteger(count) || count < 0) {
+    throw new InvalidJobStallError(`${where}maxStalledCount`, count, "a non-negative integer");
+  }
+}
+
+/**
+ * Refuse a stall config that cannot work. `undefined` means "the defaults";
+ * only the values a host actually spells out are checked.
+ */
+export function assertValidStall(stall: JobStallConfig | undefined): void {
+  if (stall === undefined) return;
+  if (typeof stall !== "object" || stall === null) {
+    throw new InvalidJobStallError("<root>", stall, "an object");
+  }
+  assertPolicyFields("", stall);
+  for (const [queue, policy] of Object.entries(stall.queues ?? {})) {
+    if (typeof policy !== "object" || policy === null) {
+      throw new InvalidJobStallError(`queues.${queue}`, policy, "an object");
+    }
+    assertPolicyFields(`queues.${queue}.`, policy);
+  }
+}
+
+function definedFields(policy: Partial<JobStallPolicy> | undefined): Partial<JobStallPolicy> {
+  const out: Partial<JobStallPolicy> = {};
+  if (!policy) return out;
+  if (policy.lockDurationMs !== undefined) out.lockDurationMs = policy.lockDurationMs;
+  if (policy.stalledIntervalMs !== undefined) out.stalledIntervalMs = policy.stalledIntervalMs;
+  if (policy.maxStalledCount !== undefined) out.maxStalledCount = policy.maxStalledCount;
+  return out;
+}
+
+/**
+ * The policy one queue's worker runs with: the package defaults, then the
+ * host's values for every queue, then the host's values for THIS queue.
+ */
+export function resolveStallPolicy(
+  queue: string,
+  stall: JobStallConfig | undefined,
+): JobStallPolicy {
+  return {
+    ...DEFAULT_STALL_POLICY,
+    ...definedFields(stall),
+    ...definedFields(stall?.queues?.[queue]),
+  };
+}

package/src/core/types.ts

@@ -186,6 +186,33 @@ export interface JobFailedEvent extends JobRunEvent {
   terminal: boolean;
 }
 
+/**
+ * A job whose worker lost its lock before the job finished.
+ *
+ * The worker's event loop or its Redis link stalled for longer than the lock,
+ * so the backend could no longer tell whether anyone was still running the
+ * job. It has been moved back to `wait` and WILL run again: a stall is a
+ * re-run, never a lost run, which is why handlers must be idempotent. A job
+ * that stalls past its budget is failed with `terminal: true` through
+ * {@link JobEvents.onJobFailed}, like any other dead-letter.
+ */
+export interface JobStalledEvent {
+  /** The job's name, or `undefined` when the job was already gone. */
+  name: string | undefined;
+  queue: string;
+  /** The driver's id for the run that stalled. */
+  runId: string;
+  /**
+   * How many times this run had been started when the stall was reported.
+   * Read back after BullMQ moved the job, so it may already count the re-run
+   * that the stall caused.
+   */
+  startedCount: number;
+  maxAttempts: number;
+  /** How many times this run has stalled so far, this one included. */
+  stalledCount: number;
+}
+
 /** A cron schedule that existed in the backend and no longer exists in code. */
 export interface ScheduleRemovedEvent {
   /** The scheduler key, which is the name of the job that installed it. */
@@ -214,6 +241,12 @@ export interface JobEvents {
    * only one of the two most hosts want to act on.
    */
   onJobFailed?(event: JobFailedEvent): void | Promise<void>;
+  /**
+   * A running job lost its lock and was put back to run again. Always worth a
+   * look: the job itself recovers, but the stall that caused it (a blocked
+   * event loop, a Redis link that went quiet) hits every job on that worker.
+   */
+  onJobStalled?(event: JobStalledEvent): void | Promise<void>;
   /**
    * A schedule was removed from the backend because no code declares it any
    * more. Destructive and unrecoverable from the queue's side, which is why it
@@ -241,6 +274,37 @@ export interface JobRetention {
   failed: JobRetentionWindow;
 }
 
+/**
+ * How a worker holds, and gives up, a running job.
+ *
+ * A worker holds a LOCK on every job it runs and renews it every half
+ * `lockDurationMs`. When the renewal cannot happen in time (the event loop was
+ * blocked, the process was paused, Redis did not answer), the lock expires and
+ * the stalled checker, every `stalledIntervalMs`, moves the job back to `wait`
+ * so it runs again. `maxStalledCount` is how many of those re-runs a job gets
+ * before it is failed as a dead-letter instead.
+ */
+export interface JobStallPolicy {
+  /** How long a lock lasts without a renewal. BullMQ's default is 30 s. */
+  lockDurationMs: number;
+  /** How often the stalled checker runs. BullMQ's default is 30 s. */
+  stalledIntervalMs: number;
+  /**
+   * Re-runs after a stall before the job is failed. `0` fails it on its first
+   * stall. BullMQ's default is 1.
+   */
+  maxStalledCount: number;
+}
+
+/**
+ * The host's stall settings: values for every queue, and per-queue values over
+ * them. A long-running or single-flight queue usually wants a longer lock than
+ * a queue of quick sends.
+ */
+export interface JobStallConfig extends Partial<JobStallPolicy> {
+  queues?: Record<string, Partial<JobStallPolicy>>;
+}
+
 /**
  * The driver port. `inline` and `bullmq` implement it; a third (SQS, pg-boss)
  * would need no change above this line.

package/src/drivers/bullmq-policy.ts

@@ -1,7 +1,7 @@
 import { UnrecoverableError, type JobsOptions } from "bullmq";
 
 import { assertValidRetention } from "../core/retention";
-import type { AnyJobDefinition, JobRetention } from "../core/types";
+import type { AnyJobDefinition, JobRetention, JobStallPolicy } from "../core/types";
 
 /**
  * The BullMQ driver's three policy decisions, kept together and out of the
@@ -84,3 +84,50 @@ export function isTerminalFailure(
 ): boolean {
   return attemptsMade >= maxAttempts || error instanceof UnrecoverableError;
 }
+
+/**
+ * The most times one run of a job may be STARTED on this queue before the
+ * worker fails it as a dead-letter: every attempt it is allowed,
+ * plus every stall it is allowed to recover from.
+ *
+ * This is the only bound a SCHEDULED job has. BullMQ never applies
+ * `maxStalledCount` to a job-scheduler job (`moveStalledJobsToWait`'s
+ * `isRepeatableJob` branch), so a tick whose handler takes the worker down
+ * every time it runs, a poison pill, would otherwise be put back and started
+ * again forever, taking every other job on the queue down with it. With the
+ * cap, the start that goes over it is failed as unrecoverable, reaches the
+ * failed set and `onJobFailed` with `terminal: true`, and the next tick of the
+ * schedule is a fresh job with its own budget.
+ *
+ * Worker-wide, so it takes the largest attempt budget on the queue: a cap
+ * below one job's legitimate retries would dead-letter it early.
+ */
+export function maxStartedAttemptsFor(
+  group: readonly AnyJobDefinition[],
+  maxStalledCount: number,
+): number {
+  const attempts = group.map((definition) => Math.max(1, definition.attempts ?? 1));
+  return Math.max(1, ...attempts) + maxStalledCount;
+}
+
+/**
+ * A queue's lock and stall settings, in BullMQ's option names. Spelled out
+ * rather than left to BullMQ's defaults: the numbers a stall is judged by
+ * belong where the host can read and configure them.
+ */
+export function workerStallOptions(
+  group: readonly AnyJobDefinition[],
+  stall: JobStallPolicy,
+): {
+  lockDuration: number;
+  stalledInterval: number;
+  maxStalledCount: number;
+  maxStartedAttempts: number;
+} {
+  return {
+    lockDuration: stall.lockDurationMs,
+    stalledInterval: stall.stalledIntervalMs,
+    maxStalledCount: stall.maxStalledCount,
+    maxStartedAttempts: maxStartedAttemptsFor(group, stall.maxStalledCount),
+  };
+}

package/src/drivers/bullmq-reporting.ts

@@ -0,0 +1,127 @@
+import type { Job } from "bullmq";
+
+import type { EmitJobEvent } from "../core/events";
+import type { JobLogger } from "../core/types";
+
+import { isTerminalFailure } from "./bullmq-policy";
+
+/**
+ * What the BullMQ driver says when a job goes wrong: one ERROR line per failed
+ * attempt and per stall, each naming the job, its run and its attempt, plus
+ * the matching `JobEvents` hook for the host.
+ *
+ * Every failure gets its own line on purpose. A host's logger is usually its
+ * error reporter too, and a failure that is only counted, or only reported
+ * when it is the last one, leaves nothing to investigate when the pattern is
+ * the problem: a job that fails twice a day and succeeds on retry, or a
+ * worker that stalls every night and recovers.
+ */
+interface ReportingState {
+  logger: JobLogger;
+  emit: EmitJobEvent;
+}
+
+/** The fields of a BullMQ job a failure report reads. */
+type FailedJob = Pick<Job, "name" | "id" | "attemptsMade" | "opts">;
+
+/** Report one failed attempt, and `onJobFailed` with whether it was the last. */
+export function reportFailure(
+  state: ReportingState,
+  queueName: string,
+  job: FailedJob | undefined,
+  error: unknown,
+): void {
+  if (!job) {
+    // BullMQ passes no job when the failure is not tied to one it can read.
+    state.logger.error(`a job on queue "${queueName}" failed (run ?):`, error);
+    return;
+  }
+  const maxAttempts = job.opts.attempts ?? 1;
+  const terminal = isTerminalFailure(job.attemptsMade, maxAttempts, error);
+  const runId = job.id ?? `${job.name}:unknown`;
+  // The outcome is stated rather than left to be inferred from the attempt
+  // count: a stall-limit or no-handler failure is terminal on attempt 1 of 3.
+  state.logger.error(
+    `job "${job.name}" failed (run ${runId}, attempt ${job.attemptsMade}/${maxAttempts}, ` +
+      `${terminal ? "no retry left" : "will retry"}):`,
+    error,
+  );
+  state.emit((events) =>
+    events.onJobFailed?.({
+      name: job.name,
+      queue: queueName,
+      runId,
+      attempt: job.attemptsMade,
+      maxAttempts,
+      error,
+      terminal,
+    }),
+  );
+}
+
+/** The fields of a BullMQ job a stall report reads. */
+type StalledJob = Pick<
+  Job,
+  "name" | "attemptsStarted" | "stalledCounter" | "opts" | "repeatJobKey"
+>;
+
+/** Read the stalled job back; a failed read is `undefined`, never a throw. */
+async function readStalledJob(
+  getJob: (id: string) => Promise<StalledJob | undefined>,
+  jobId: string,
+): Promise<StalledJob | undefined> {
+  try {
+    return await getJob(jobId);
+  } catch {
+    return undefined;
+  }
+}
+
+/** What the stall report says happens next. */
+function nextAfterStall(job: StalledJob | undefined, maxStalledCount: number): string {
+  const stalledCount = job?.stalledCounter ?? 0;
+  if (job && !job.repeatJobKey && stalledCount > maxStalledCount) {
+    return `it has stalled more than the ${maxStalledCount} time(s) allowed, so it is failed as a dead-letter.`;
+  }
+  return "it was moved back to wait and will run again.";
+}
+
+/**
+ * Report one stalled job, and `onJobStalled`.
+ *
+ * BullMQ's `stalled` event carries only the id, so the job is read back for
+ * its name and counters. That read is best-effort: a job already trimmed, or a
+ * Redis that did not answer, still produces the line, with the id alone.
+ *
+ * The line says what happens next. A job over its stall budget is failed as a
+ * dead-letter rather than re-run, except a scheduled one: BullMQ never applies
+ * `maxStalledCount` to those, and they are bounded by `maxStartedAttempts`
+ * when they next start instead.
+ */
+export async function reportStall(
+  state: ReportingState,
+  queueName: string,
+  jobId: string,
+  getJob: (id: string) => Promise<StalledJob | undefined>,
+  maxStalledCount: number,
+): Promise<void> {
+  const job = await readStalledJob(getJob, jobId);
+  const maxAttempts = job?.opts.attempts ?? 1;
+  const startedCount = job?.attemptsStarted ?? 0;
+  const stalledCount = job?.stalledCounter ?? 0;
+  state.logger.error(
+    `job "${job?.name ?? "?"}" on queue "${queueName}" stalled (run ${jobId}, ` +
+      `started ${startedCount} time(s) of ${maxAttempts} attempt(s), stalled ${stalledCount} ` +
+      `time(s)): its lock expired before it finished; ${nextAfterStall(job, maxStalledCount)}`,
+  );
+  state.emit((events) =>
+    events.onJobStalled?.({
+      name: job?.name,
+      queue: queueName,
+      runId: jobId,
+      startedCount,
+      maxAttempts,
+      stalledCount,
+    }),
+  );
+}

package/src/drivers/bullmq.ts

@@ -28,6 +28,7 @@ import { Queue, UnrecoverableError, Worker, type JobsOptions } from "bullmq";
 import { createEventEmitter, type EmitJobEvent } from "../core/events";
 import { DEFAULT_QUEUE } from "../core/queues";
 import { resolveRegisteredJob } from "../core/registry";
+import { assertValidStall, resolveStallPolicy } from "../core/stall";
 import type {
   AnyJobDefinition,
   EnqueueOptions,
@@ -37,15 +38,19 @@ import type {
   JobEvents,
   JobLogger,
   JobRetention,
+  JobStallConfig,
 } from "../core/types";
 
 import {
   DEFAULT_CONCURRENCY,
   DEFAULT_JOB_RETENTION,
   isTerminalFailure,
+  maxStartedAttemptsFor,
   resolveConcurrency,
   retentionOptions,
+  workerStallOptions,
 } from "./bullmq-policy";
+import { reportFailure, reportStall } from "./bullmq-reporting";
 import { parseRedisUrl, type RedisConnectionOptions } from "./redis-url";
 
 /** The one ioredis command this driver reads outside BullMQ's own surface. */
@@ -66,8 +71,13 @@ export interface BullMqJobDriverOptions {
   retention?: JobRetention;
   /** Per-queue concurrency when no definition on the queue states one. */
   defaultConcurrency?: number;
-  /** Where completions, dead-letters and removed schedules are reported. */
+  /** Where completions, dead-letters, stalls and removed schedules are reported. */
   events?: JobEvents;
+  /**
+   * Lock and stall settings, for every queue and per queue. Defaults to
+   * BullMQ's own numbers, spelled out in `DEFAULT_STALL_POLICY`.
+   */
+  stall?: JobStallConfig;
 }
 
 /** Everything the driver's helpers need, threaded instead of closed over. */
@@ -77,6 +87,7 @@ interface DriverState {
   prefix?: string;
   retention: Pick<JobsOptions, "removeOnComplete" | "removeOnFail">;
   defaultConcurrency: number;
+  stall: JobStallConfig | undefined;
   /** Reports to the host's observer; see `core/events`. Never throws. */
   emit: EmitJobEvent;
   queues: Map<string, Queue>;
@@ -203,6 +214,7 @@ function startWorker(
 ): Worker {
   const onThisQueue = new Set(group.map((definition) => definition.name));
   const concurrency = resolveConcurrency(group, state.defaultConcurrency);
+  const stall = resolveStallPolicy(queueName, state.stall);
 
   const worker = new Worker(
     queueName,
@@ -240,30 +252,19 @@ function startWorker(
       connection: state.connection,
       concurrency,
       ...(state.prefix ? { prefix: state.prefix } : {}),
+      ...workerStallOptions(group, stall),
     },
   );
 
-  worker.on("failed", (job, error) => {
-    const maxAttempts = job?.opts.attempts ?? 1;
-    const attempts = job ? `${job.attemptsMade}/${maxAttempts}` : "?";
-    state.logger.error(
-      `job "${job?.name ?? queueName}" failed (attempt ${attempts}):`,
-      error,
-    );
-    if (!job) return;
-    const terminal = isTerminalFailure(job.attemptsMade, maxAttempts, error);
-    state.emit((events) =>
-      events.onJobFailed?.({
-        name: job.name,
-        queue: queueName,
-        runId: job.id ?? `${job.name}:unknown`,
-        attempt: job.attemptsMade,
-        maxAttempts,
-        error,
-        terminal,
-      }),
-    );
+  worker.on("failed", (job, error) => reportFailure(state, queueName, job, error));
+  worker.on("stalled", (jobId) => {
+    const getJob = (id: string) => queueFor(state, queueName).getJob(id);
+    void reportStall(state, queueName, jobId, getJob, stall.maxStalledCount);
   });
+  // Lock-renewal and lost-lock errors ("could not renew lock for job …",
+  // "Missing lock for job … moveToFinished") arrive here. They stay ERRORS:
+  // each one is a stall somebody should look at, even though the job itself
+  // recovers.
   worker.on("error", (error) =>
     state.logger.error(`worker "${queueName}" error:`, error),
   );
@@ -285,6 +286,9 @@ function groupByQueue(
 }
 
 export function createBullMqJobDriver(options: BullMqJobDriverOptions): JobDriver {
+  // The same backstop as retention: a host building the driver directly off
+  // `@12-apps/jobs/bullmq` must not reach BullMQ with a lock of 0 or NaN.
+  assertValidStall(options.stall);
   const state: DriverState = {
     connection: parseRedisUrl(options.redisUrl),
     logger: options.logger,
@@ -294,6 +298,7 @@ export function createBullMqJobDriver(options: BullMqJobDriverOptions): JobDrive
       typeof options.defaultConcurrency === "number" && options.defaultConcurrency > 0
         ? options.defaultConcurrency
         : DEFAULT_CONCURRENCY,
+    stall: options.stall,
     emit: createEventEmitter(options.events, options.logger),
     queues: new Map(),
     workers: [],
@@ -353,5 +358,8 @@ export function createBullMqJobDriver(options: BullMqJobDriverOptions): JobDrive
 export const __testables = {
   resolveConcurrency,
   isTerminalFailure,
+  maxStartedAttemptsFor,
+  reportFailure,
+  reportStall,
   DEFAULT_CONCURRENCY,
 };

package/src/index.ts

@@ -98,12 +98,17 @@ export type {
   JobRetentionWindow,
   JobRunEvent,
   JobSchedule,
+  JobStallConfig,
+  JobStalledEvent,
+  JobStallPolicy,
   ScheduleRemovedEvent,
 } from "./core/types";
 
 // Retention validation lives in `core` so this barrel can carry it without
 // pulling `bullmq` (and ioredis) into a bundle that only ever enqueues.
 export { assertValidRetention, InvalidJobRetentionError } from "./core/retention";
+// The stall settings' check and defaults, for the same reason.
+export { assertValidStall, DEFAULT_STALL_POLICY, InvalidJobStallError } from "./core/stall";
 
 export { createInlineJobDriver } from "./drivers/inline";
 export type { InlineJobDriver, InlineJobRun } from "./drivers/inline";

package/src/server/config.ts

@@ -4,6 +4,7 @@ import type {
   JobEvents,
   JobLogger,
   JobRetention,
+  JobStallConfig,
 } from "../core/types";
 import type { SweepLeaseDbProvider } from "../lease/sweep-lease";
 
@@ -100,6 +101,13 @@ export interface JobsServerConfig {
    * that states `concurrency: 1` still gets 1 — a stated value always wins.
    */
   defaultConcurrency?: number;
+  /**
+   * How a worker holds a running job, and how many times a job that lost that
+   * hold (a stall) is run again before it is failed. For every queue, with
+   * per-queue values over them. Defaults to BullMQ's own numbers (a 30 s lock,
+   * a 30 s stalled check, one re-run). Validated at assembly.
+   */
+  stall?: JobStallConfig;
   /**
    * Where the `sweep_leases` table lives — enables `withSweepLease` on the
    * factory's return. Omit it and the lease helper rejects on first use,
@@ -161,6 +169,7 @@ export interface ResolvedConfig {
   events: JobEvents | undefined;
   retention: JobRetention | undefined;
   defaultConcurrency: number | undefined;
+  stall: JobStallConfig | undefined;
 }
 
 /**
@@ -179,5 +188,6 @@ export function resolveConfig(config: JobsServerConfig): ResolvedConfig {
     events: config.events,
     retention: config.retention,
     defaultConcurrency: config.defaultConcurrency,
+    stall: config.stall,
   };
 }

package/src/server/create-api-jobs.ts

@@ -1,5 +1,6 @@
 import { assertJobsRegistered, listJobs } from "../core/registry";
 import { assertValidRetention } from "../core/retention";
+import { assertValidStall } from "../core/stall";
 import {
   configureJobs,
   getJobDriver,
@@ -339,6 +340,9 @@ export function createApiJobs(config: JobsServerConfig): JobsApi {
   // all — silently, weeks before anyone notices. The driver re-checks it, for
   // a host that builds one directly off `@12-apps/jobs/bullmq`.
   assertValidRetention(config?.retention);
+  // And the stall settings: a lock of 0 or NaN expires at once, and every job
+  // would be reported stalled and re-run while it was still running.
+  assertValidStall(config?.stall);
 
   const state: RuntimeState = {
     starting: false,

package/src/server/index.ts

@@ -32,4 +32,5 @@ export type { JobsDriverChoice, JobsServerConfig, JobsSource } from "./config";
 // and `start()` throw without also reaching for the root entry point.
 export { NoJobsRegisteredError } from "../core/registry";
 export { InvalidJobRetentionError } from "../core/retention";
-export type { JobEvents, JobRetention } from "../core/types";
+export { InvalidJobStallError } from "../core/stall";
+export type { JobEvents, JobRetention, JobStallConfig } from "../core/types";

package/src/server/resolve-driver.ts

@@ -144,6 +144,7 @@ async function resolveBullMq(
         events: resolved.events,
         retention: resolved.retention,
         defaultConcurrency: resolved.defaultConcurrency,
+        stall: resolved.stall,
       }),
       deliberatelyOff: false,
     };