@nicnocquee/dataqueue 1.34.0 → 1.35.0-beta.20260224110011

package/ai/docs-content.json

@@ -15,7 +15,7 @@
     "slug": "api",
     "title": "API Reference",
     "description": "",
-    "content": "This section documents the main classes, types, and functions available for managing job queues, processing jobs, and interacting with the database.\n\n## API Surface\n\n- [JobQueue](/api/job-queue)\n- [JobOptions](/api/job-options)\n- [JobRecord](/api/job-record)\n- [JobEvent](/api/job-event)\n- [Processor](/api/processor)\n- [ProcessorOptions](/api/processor-options)\n- [JobHandlers](/api/job-handlers)\n- [Database Utility](/api/db-util)\n- [Tags](/api/tags)"
+    "content": "This section documents the main classes, types, and functions available for managing job queues, processing jobs, and interacting with the database.\n\n## API Surface\n\n- [JobQueue](/api/job-queue)\n- [JobOptions](/api/job-options)\n- [JobRecord](/api/job-record)\n- [JobEvent](/api/job-event)\n- [Processor](/api/processor)\n- [ProcessorOptions](/api/processor-options)\n- [Supervisor](/api/job-queue#background-supervisor)\n- [SupervisorOptions](/api/job-queue#supervisoroptions)\n- [SupervisorRunResult](/api/job-queue#supervisorrunresult)\n- [JobHandlers](/api/job-handlers)\n- [Database Utility](/api/db-util)\n- [Tags](/api/tags)"
   },
   {
     "slug": "api/job-event",
@@ -39,19 +39,19 @@
     "slug": "api/job-queue",
     "title": "JobQueue",
     "description": "",
-    "content": "## Initialization\n\n### initJobQueue\n\n```ts\ninitJobQueue(config: JobQueueConfig): JobQueue\n```\n\nInitializes the job queue system with the provided configuration. The `JobQueueConfig` is a discriminated union -- you provide either a PostgreSQL or Redis configuration.\n\n#### PostgresJobQueueConfig\n\n```ts\ninterface PostgresJobQueueConfig {\n  backend?: 'postgres'; // Optional, defaults to 'postgres'\n  databaseConfig: {\n    connectionString?: string;\n    host?: string;\n    port?: number;\n    database?: string;\n    user?: string;\n    password?: string;\n    ssl?: DatabaseSSLConfig;\n  };\n  verbose?: boolean;\n}\n```\n\n#### RedisJobQueueConfig\n\n```ts\ninterface RedisJobQueueConfig {\n  backend: 'redis'; // Required\n  redisConfig: {\n    url?: string;\n    host?: string;\n    port?: number;\n    password?: string;\n    db?: number;\n    tls?: RedisTLSConfig;\n    keyPrefix?: string; // Default: 'dq:'\n  };\n  verbose?: boolean;\n}\n```\n\n#### JobQueueConfig\n\n```ts\ntype JobQueueConfig = PostgresJobQueueConfig | RedisJobQueueConfig;\n```\n\n#### DatabaseSSLConfig\n\n```ts\ninterface DatabaseSSLConfig {\n  ca?: string;\n  cert?: string;\n  key?: string;\n  rejectUnauthorized?: boolean;\n}\n```\n\n- `ca` - Client certificate authority (CA) as PEM string or file path. If the value starts with 'file://', it will be loaded from file, otherwise treated as PEM string.\n- `cert` - Client certificate as PEM string or file path. If the value starts with 'file://', it will be loaded from file, otherwise treated as PEM string.\n- `key` - Client private key as PEM string or file path. If the value starts with 'file://', it will be loaded from file, otherwise treated as PEM string.\n- `rejectUnauthorized` - Whether to reject unauthorized certificates (default: true)\n\n#### RedisTLSConfig\n\n```ts\ninterface RedisTLSConfig {\n  ca?: string;\n  cert?: string;\n  key?: string;\n  rejectUnauthorized?: boolean;\n}\n```\n\n---\n\n## Adding Jobs\n\n### addJob\n\n```ts\naddJob(job: JobOptions): Promise<number>\n```\n\nAdds a job to the queue. Returns the job ID.\n\n#### JobOptions\n\n```ts\ninterface JobOptions {\n  jobType: string;\n  payload: any;\n  maxAttempts?: number;\n  priority?: number;\n  runAt?: Date | null;\n  timeoutMs?: number;\n  tags?: string[];\n}\n```\n\n---\n\n## Retrieving Jobs\n\n### getJob\n\n```ts\ngetJob(id: number): Promise<JobRecord | null>\n```\n\nRetrieves a job by its ID.\n\n### getJobs\n\n```ts\ngetJobs(\n  filters?: {\n    jobType?: string;\n    priority?: number;\n    runAt?: Date | { gt?: Date; gte?: Date; lt?: Date; lte?: Date; eq?: Date };\n    tags?: { values: string[]; mode?: 'all' | 'any' | 'none' | 'exact' };\n  },\n  limit?: number,\n  offset?: number\n): Promise<JobRecord[]>\n```\n\nRetrieves jobs matching the provided filters, with optional pagination.\n\n### getJobsByStatus\n\n```ts\ngetJobsByStatus(status: string, limit?: number, offset?: number): Promise<JobRecord[]>\n```\n\nRetrieves jobs by their status, with pagination.\n\n### getAllJobs\n\n```ts\ngetAllJobs(limit?: number, offset?: number): Promise<JobRecord[]>\n```\n\nRetrieves all jobs, with optional pagination.\n\n### getJobsByTags\n\n```ts\ngetJobsByTags(tags: string[], mode?: TagQueryMode, limit?: number, offset?: number): Promise<JobRecord[]>\n```\n\nRetrieves jobs by tag(s).\n\n---\n\n## Managing Jobs\n\n### retryJob\n\n```ts\nretryJob(jobId: number): Promise<void>\n```\n\nRetries a job given its ID.\n\n### cancelJob\n\n```ts\ncancelJob(jobId: number): Promise<void>\n```\n\nCancels a job given its ID.\n\n### editJob\n\n```ts\neditJob(jobId: number, updates: EditJobOptions): Promise<void>\n```\n\nEdits a pending job given its ID. Only works for jobs with status 'pending'. Silently fails for other statuses (processing, completed, failed, cancelled).\n\n#### EditJobOptions\n\n```ts\ninterface EditJobOptions {\n  payload?: any;\n  maxAttempts?: number;\n  priority?: number;\n  runAt?: Date | null;\n  timeoutMs?: number;\n  tags?: string[];\n}\n```\n\nAll fields are optional - only provided fields will be updated. Note that `jobType` cannot be changed.\n\n#### Example\n\n```ts\n// Edit a pending job's payload and priority\nawait jobQueue.editJob(jobId, {\n  payload: { to: 'newemail@example.com', subject: 'Updated' },\n  priority: 10,\n});\n\n// Edit only the scheduled run time\nawait jobQueue.editJob(jobId, {\n  runAt: new Date(Date.now() + 60000), // Run in 1 minute\n});\n\n// Edit multiple fields at once\nawait jobQueue.editJob(jobId, {\n  payload: { to: 'updated@example.com' },\n  priority: 5,\n  maxAttempts: 10,\n  timeoutMs: 30000,\n  tags: ['urgent', 'priority'],\n});\n```\n\n### editAllPendingJobs\n\n```ts\neditAllPendingJobs(\n  filters?: {\n    jobType?: string;\n    priority?: number;\n    runAt?: Date | { gt?: Date; gte?: Date; lt?: Date; lte?: Date; eq?: Date };\n    tags?: { values: string[]; mode?: 'all' | 'any' | 'none' | 'exact' };\n  },\n  updates: EditJobOptions\n): Promise<number>\n```\n\nEdits all pending jobs that match the filters. Only works for jobs with status 'pending'. Non-pending jobs are not affected. Returns the number of jobs that were edited.\n\n#### Parameters\n\n- `filters` (optional): Filters to select which jobs to edit. If not provided, all pending jobs are edited.\n  - `jobType`: Filter by job type\n  - `priority`: Filter by priority\n  - `runAt`: Filter by scheduled run time (supports `gt`, `gte`, `lt`, `lte`, `eq` operators or exact Date match)\n  - `tags`: Filter by tags with mode ('all', 'any', 'none', 'exact')\n- `updates`: The fields to update (same as `EditJobOptions`). All fields are optional - only provided fields will be updated.\n\n#### Returns\n\nThe number of jobs that were successfully edited.\n\n#### Examples\n\n```ts\n// Edit all pending jobs\nconst editedCount = await jobQueue.editAllPendingJobs(undefined, {\n  priority: 10,\n});\n\n// Edit all pending email jobs\nconst editedCount = await jobQueue.editAllPendingJobs(\n  { jobType: 'email' },\n  {\n    priority: 5,\n  },\n);\n\n// Edit all pending jobs with 'urgent' tag\nconst editedCount = await jobQueue.editAllPendingJobs(\n  { tags: { values: ['urgent'], mode: 'any' } },\n  {\n    priority: 10,\n    maxAttempts: 5,\n  },\n);\n\n// Edit all pending jobs scheduled in the future\nconst editedCount = await jobQueue.editAllPendingJobs(\n  { runAt: { gte: new Date() } },\n  {\n    priority: 10,\n  },\n);\n\n// Edit with combined filters\nconst editedCount = await jobQueue.editAllPendingJobs(\n  {\n    jobType: 'email',\n    tags: { values: ['urgent'], mode: 'any' },\n  },\n  {\n    priority: 10,\n    maxAttempts: 5,\n  },\n);\n```\n\n**Note:** Only pending jobs are edited. Jobs with other statuses (processing, completed, failed, cancelled) are not affected. Edit events are recorded for each affected job, just like single job edits.\n\n### cancelAllUpcomingJobs\n\n```ts\ncancelAllUpcomingJobs(filters?: {\n  jobType?: string;\n  priority?: number;\n  runAt?: Date | { gt?: Date; gte?: Date; lt?: Date; lte?: Date; eq?: Date };\n  tags?: { values: string[]; mode?: 'all' | 'any' | 'none' | 'exact' };\n}): Promise<number>\n```\n\nCancels all upcoming jobs that match the filters. Returns the number of jobs cancelled.\n\n### cleanupOldJobs\n\n```ts\ncleanupOldJobs(daysToKeep?: number): Promise<number>\n```\n\nCleans up jobs older than the specified number of days. Returns the number of jobs removed.\n\n### reclaimStuckJobs\n\n```ts\nreclaimStuckJobs(maxProcessingTimeMinutes?: number): Promise<number>\n```\n\nReclaims jobs stuck in 'processing' for too long. Returns the number of jobs reclaimed. If a job has a `timeoutMs` that is longer than the `maxProcessingTimeMinutes` threshold, the job's own timeout is used instead, preventing premature reclamation of long-running jobs.\n\n---\n\n## Job Events\n\n### getJobEvents\n\n```ts\ngetJobEvents(jobId: number): Promise<JobEvent[]>\n```\n\nRetrieves the job events for a job.\n\n#### JobEvent\n\n```ts\ninterface JobEvent {\n  id: number;\n  jobId: number;\n  eventType: JobEventType;\n  createdAt: Date;\n  metadata: any;\n}\n```\n\n#### JobEventType\n\n```ts\nenum JobEventType {\n  Added = 'added',\n  Processing = 'processing',\n  Completed = 'completed',\n  Failed = 'failed',\n  Cancelled = 'cancelled',\n  Retried = 'retried',\n  Edited = 'edited',\n}\n```\n\n---\n\n## Processing Jobs\n\n### createProcessor\n\n```ts\ncreateProcessor(\n  handlers: JobHandlers,\n  options?: ProcessorOptions\n): Processor\n```\n\nCreates a job processor with the provided handlers and options.\n\n#### ProcessorOptions\n\n```ts\ninterface ProcessorOptions {\n  workerId?: string;\n  batchSize?: number;\n  concurrency?: number;\n  pollInterval?: number;\n  onError?: (error: Error) => void;\n  verbose?: boolean;\n  jobType?: string | string[];\n}\n```\n\n---\n\n## Accessing the Underlying Client\n\n### getPool\n\n```ts\ngetPool(): Pool\n```\n\nReturns the PostgreSQL connection pool instance. Only available when using the PostgreSQL backend.\n\n> **Note:** Throws an error if called when using the Redis backend.\n\n### getRedisClient\n\n```ts\ngetRedisClient(): Redis\n```\n\nReturns the `ioredis` client instance. Only available when using the Redis backend.\n\n> **Note:** Throws an error if called when using the PostgreSQL backend."
+    "content": "## Initialization\n\n### initJobQueue\n\n```ts\ninitJobQueue(config: JobQueueConfig): JobQueue\n```\n\nInitializes the job queue system with the provided configuration. The `JobQueueConfig` is a discriminated union -- you provide either a PostgreSQL or Redis configuration.\n\n#### PostgresJobQueueConfig\n\nProvide either `databaseConfig` (the library creates a pool) or `pool` (bring your own `pg.Pool`). At least one must be set.\n\n```ts\ninterface PostgresJobQueueConfig {\n  backend?: 'postgres'; // Optional, defaults to 'postgres'\n  databaseConfig?: {\n    connectionString?: string;\n    host?: string;\n    port?: number;\n    database?: string;\n    user?: string;\n    password?: string;\n    ssl?: DatabaseSSLConfig;\n  };\n  pool?: import('pg').Pool; // Bring your own pool\n  verbose?: boolean;\n}\n```\n\n#### RedisJobQueueConfig\n\nProvide either `redisConfig` (the library creates an ioredis client) or `client` (bring your own). At least one must be set.\n\n```ts\ninterface RedisJobQueueConfig {\n  backend: 'redis'; // Required\n  redisConfig?: {\n    url?: string;\n    host?: string;\n    port?: number;\n    password?: string;\n    db?: number;\n    tls?: RedisTLSConfig;\n    keyPrefix?: string; // Default: 'dq:'\n  };\n  client?: unknown; // Bring your own ioredis client\n  keyPrefix?: string; // Key prefix when using external client (default: 'dq:')\n  verbose?: boolean;\n}\n```\n\n#### JobQueueConfig\n\n```ts\ntype JobQueueConfig = PostgresJobQueueConfig | RedisJobQueueConfig;\n```\n\n#### DatabaseSSLConfig\n\n```ts\ninterface DatabaseSSLConfig {\n  ca?: string;\n  cert?: string;\n  key?: string;\n  rejectUnauthorized?: boolean;\n}\n```\n\n- `ca` - Client certificate authority (CA) as PEM string or file path. If the value starts with 'file://', it will be loaded from file, otherwise treated as PEM string.\n- `cert` - Client certificate as PEM string or file path. If the value starts with 'file://', it will be loaded from file, otherwise treated as PEM string.\n- `key` - Client private key as PEM string or file path. If the value starts with 'file://', it will be loaded from file, otherwise treated as PEM string.\n- `rejectUnauthorized` - Whether to reject unauthorized certificates (default: true)\n\n#### RedisTLSConfig\n\n```ts\ninterface RedisTLSConfig {\n  ca?: string;\n  cert?: string;\n  key?: string;\n  rejectUnauthorized?: boolean;\n}\n```\n\n---\n\n## Adding Jobs\n\n### addJob\n\n```ts\naddJob(job: JobOptions, options?: AddJobOptions): Promise<number>\n```\n\nAdds a job to the queue. Returns the job ID.\n\n#### JobOptions\n\n```ts\ninterface JobOptions {\n  jobType: string;\n  payload: any;\n  maxAttempts?: number;\n  priority?: number;\n  runAt?: Date | null;\n  timeoutMs?: number;\n  tags?: string[];\n  idempotencyKey?: string;\n  retryDelay?: number; // Base delay between retries in seconds (default: 60)\n  retryBackoff?: boolean; // Use exponential backoff (default: true)\n  retryDelayMax?: number; // Max delay cap in seconds (default: none)\n  group?: { id: string; tier?: string }; // Optional group for global concurrency limits\n}\n```\n\n- `retryDelay` - Base delay between retries in seconds. When `retryBackoff` is true, this is the base for exponential backoff (`retryDelay * 2^attempts`). When false, retries use this fixed delay. Default: `60`.\n- `retryBackoff` - Whether to use exponential backoff. When true, delay doubles with each attempt and includes jitter. Default: `true`.\n- `retryDelayMax` - Maximum delay cap in seconds. Only meaningful when `retryBackoff` is true. No limit when omitted.\n- `group` - Optional grouping metadata. Use `group.id` to enforce global per-group limits with `ProcessorOptions.groupConcurrency`. `group.tier` is reserved for future policies.\n\n#### AddJobOptions\n\n```ts\ninterface AddJobOptions {\n  db?: DatabaseClient;\n}\n```\n\n- `db` — An external database client (e.g., a `pg.PoolClient` inside a transaction). When provided, the INSERT runs on this client instead of the internal pool. **PostgreSQL only.** Throws if used with the Redis backend.\n\n### addJobs\n\n```ts\naddJobs(jobs: JobOptions[], options?: AddJobOptions): Promise<number[]>\n```\n\nAdds multiple jobs to the queue in a single operation. More efficient than calling `addJob` in a loop because it batches the INSERT into a single database round-trip (PostgreSQL) or a single atomic Lua script (Redis).\n\nReturns an array of job IDs in the same order as the input array.\n\nEach job can independently have its own `priority`, `runAt`, `tags`, `idempotencyKey`, and other options. Idempotency keys are handled per-job — duplicates resolve to the existing job's ID without creating a new row.\n\nPassing an empty array returns `[]` immediately without touching the database.\n\n```ts\nconst jobIds = await jobQueue.addJobs([\n  {\n    jobType: 'email',\n    payload: { to: 'a@example.com', subject: 'Hi', body: '...' },\n  },\n  {\n    jobType: 'email',\n    payload: { to: 'b@example.com', subject: 'Hi', body: '...' },\n    priority: 10,\n  },\n  {\n    jobType: 'report',\n    payload: { reportId: '123', userId: '456' },\n    tags: ['monthly'],\n  },\n]);\n// jobIds = [1, 2, 3]\n```\n\nThe `{ db }` option works the same as `addJob` — pass a transactional client to batch-insert within an existing transaction (PostgreSQL only).\n\n#### DatabaseClient\n\n```ts\ninterface DatabaseClient {\n  query(\n    text: string,\n    values?: any[],\n  ): Promise<{ rows: any[]; rowCount: number | null }>;\n}\n```\n\nAny object matching this interface works — `pg.Pool`, `pg.PoolClient`, `pg.Client`, or ORM query runners that expose a raw `query()` method.\n\n---\n\n## Retrieving Jobs\n\n### getJob\n\n```ts\ngetJob(id: number): Promise<JobRecord | null>\n```\n\nRetrieves a job by its ID.\n\n### getJobs\n\n```ts\ngetJobs(\n  filters?: {\n    jobType?: string;\n    priority?: number;\n    runAt?: Date | { gt?: Date; gte?: Date; lt?: Date; lte?: Date; eq?: Date };\n    tags?: { values: string[]; mode?: 'all' | 'any' | 'none' | 'exact' };\n  },\n  limit?: number,\n  offset?: number\n): Promise<JobRecord[]>\n```\n\nRetrieves jobs matching the provided filters, with optional pagination.\n\n### getJobsByStatus\n\n```ts\ngetJobsByStatus(status: string, limit?: number, offset?: number): Promise<JobRecord[]>\n```\n\nRetrieves jobs by their status, with pagination.\n\n### getAllJobs\n\n```ts\ngetAllJobs(limit?: number, offset?: number): Promise<JobRecord[]>\n```\n\nRetrieves all jobs, with optional pagination.\n\n### getJobsByTags\n\n```ts\ngetJobsByTags(tags: string[], mode?: TagQueryMode, limit?: number, offset?: number): Promise<JobRecord[]>\n```\n\nRetrieves jobs by tag(s).\n\n---\n\n## Managing Jobs\n\n### retryJob\n\n```ts\nretryJob(jobId: number): Promise<void>\n```\n\nRetries a job given its ID.\n\n### cancelJob\n\n```ts\ncancelJob(jobId: number): Promise<void>\n```\n\nCancels a job given its ID.\n\n### editJob\n\n```ts\neditJob(jobId: number, updates: EditJobOptions): Promise<void>\n```\n\nEdits a pending job given its ID. Only works for jobs with status 'pending'. Silently fails for other statuses (processing, completed, failed, cancelled).\n\n#### EditJobOptions\n\n```ts\ninterface EditJobOptions {\n  payload?: any;\n  maxAttempts?: number;\n  priority?: number;\n  runAt?: Date | null;\n  timeoutMs?: number;\n  tags?: string[];\n  retryDelay?: number | null;\n  retryBackoff?: boolean | null;\n  retryDelayMax?: number | null;\n}\n```\n\nAll fields are optional - only provided fields will be updated. Note that `jobType` cannot be changed. Set retry fields to `null` to revert to legacy default behavior.\n\n#### Example\n\n```ts\n// Edit a pending job's payload and priority\nawait jobQueue.editJob(jobId, {\n  payload: { to: 'newemail@example.com', subject: 'Updated' },\n  priority: 10,\n});\n\n// Edit only the scheduled run time\nawait jobQueue.editJob(jobId, {\n  runAt: new Date(Date.now() + 60000), // Run in 1 minute\n});\n\n// Edit multiple fields at once\nawait jobQueue.editJob(jobId, {\n  payload: { to: 'updated@example.com' },\n  priority: 5,\n  maxAttempts: 10,\n  timeoutMs: 30000,\n  tags: ['urgent', 'priority'],\n});\n```\n\n### editAllPendingJobs\n\n```ts\neditAllPendingJobs(\n  filters?: {\n    jobType?: string;\n    priority?: number;\n    runAt?: Date | { gt?: Date; gte?: Date; lt?: Date; lte?: Date; eq?: Date };\n    tags?: { values: string[]; mode?: 'all' | 'any' | 'none' | 'exact' };\n  },\n  updates: EditJobOptions\n): Promise<number>\n```\n\nEdits all pending jobs that match the filters. Only works for jobs with status 'pending'. Non-pending jobs are not affected. Returns the number of jobs that were edited.\n\n#### Parameters\n\n- `filters` (optional): Filters to select which jobs to edit. If not provided, all pending jobs are edited.\n  - `jobType`: Filter by job type\n  - `priority`: Filter by priority\n  - `runAt`: Filter by scheduled run time (supports `gt`, `gte`, `lt`, `lte`, `eq` operators or exact Date match)\n  - `tags`: Filter by tags with mode ('all', 'any', 'none', 'exact')\n- `updates`: The fields to update (same as `EditJobOptions`). All fields are optional - only provided fields will be updated.\n\n#### Returns\n\nThe number of jobs that were successfully edited.\n\n#### Examples\n\n```ts\n// Edit all pending jobs\nconst editedCount = await jobQueue.editAllPendingJobs(undefined, {\n  priority: 10,\n});\n\n// Edit all pending email jobs\nconst editedCount = await jobQueue.editAllPendingJobs(\n  { jobType: 'email' },\n  {\n    priority: 5,\n  },\n);\n\n// Edit all pending jobs with 'urgent' tag\nconst editedCount = await jobQueue.editAllPendingJobs(\n  { tags: { values: ['urgent'], mode: 'any' } },\n  {\n    priority: 10,\n    maxAttempts: 5,\n  },\n);\n\n// Edit all pending jobs scheduled in the future\nconst editedCount = await jobQueue.editAllPendingJobs(\n  { runAt: { gte: new Date() } },\n  {\n    priority: 10,\n  },\n);\n\n// Edit with combined filters\nconst editedCount = await jobQueue.editAllPendingJobs(\n  {\n    jobType: 'email',\n    tags: { values: ['urgent'], mode: 'any' },\n  },\n  {\n    priority: 10,\n    maxAttempts: 5,\n  },\n);\n```\n\n**Note:** Only pending jobs are edited. Jobs with other statuses (processing, completed, failed, cancelled) are not affected. Edit events are recorded for each affected job, just like single job edits.\n\n### cancelAllUpcomingJobs\n\n```ts\ncancelAllUpcomingJobs(filters?: {\n  jobType?: string;\n  priority?: number;\n  runAt?: Date | { gt?: Date; gte?: Date; lt?: Date; lte?: Date; eq?: Date };\n  tags?: { values: string[]; mode?: 'all' | 'any' | 'none' | 'exact' };\n}): Promise<number>\n```\n\nCancels all upcoming jobs that match the filters. Returns the number of jobs cancelled.\n\n### cleanupOldJobs\n\n```ts\ncleanupOldJobs(daysToKeep?: number): Promise<number>\n```\n\nCleans up jobs older than the specified number of days. Returns the number of jobs removed.\n\n### reclaimStuckJobs\n\n```ts\nreclaimStuckJobs(maxProcessingTimeMinutes?: number): Promise<number>\n```\n\nReclaims jobs stuck in 'processing' for too long. Returns the number of jobs reclaimed. If a job has a `timeoutMs` that is longer than the `maxProcessingTimeMinutes` threshold, the job's own timeout is used instead, preventing premature reclamation of long-running jobs.\n\n---\n\n## Job Events\n\n### getJobEvents\n\n```ts\ngetJobEvents(jobId: number): Promise<JobEvent[]>\n```\n\nRetrieves the job events for a job.\n\n#### JobEvent\n\n```ts\ninterface JobEvent {\n  id: number;\n  jobId: number;\n  eventType: JobEventType;\n  createdAt: Date;\n  metadata: any;\n}\n```\n\n#### JobEventType\n\n```ts\nenum JobEventType {\n  Added = 'added',\n  Processing = 'processing',\n  Completed = 'completed',\n  Failed = 'failed',\n  Cancelled = 'cancelled',\n  Retried = 'retried',\n  Edited = 'edited',\n}\n```\n\n---\n\n## Event Hooks\n\nDataQueue emits real-time events for job lifecycle transitions. Register listeners using `on`, `once`, `off`, and `removeAllListeners`. Works identically with both PostgreSQL and Redis backends.\n\n### QueueEventMap\n\n```ts\ninterface QueueEventMap {\n  'job:added': { jobId: number; jobType: string };\n  'job:processing': { jobId: number; jobType: string };\n  'job:completed': { jobId: number; jobType: string };\n  'job:failed': {\n    jobId: number;\n    jobType: string;\n    error: Error;\n    willRetry: boolean;\n  };\n  'job:cancelled': { jobId: number };\n  'job:retried': { jobId: number };\n  'job:waiting': { jobId: number; jobType: string };\n  'job:progress': { jobId: number; progress: number };\n  error: Error;\n}\n```\n\n### on\n\n```ts\non(event: QueueEventName, listener: (data) => void): void\n```\n\nRegister a listener that fires every time the event is emitted.\n\n### once\n\n```ts\nonce(event: QueueEventName, listener: (data) => void): void\n```\n\nRegister a one-time listener that auto-removes after the first invocation.\n\n### off\n\n```ts\noff(event: QueueEventName, listener: (data) => void): void\n```\n\nRemove a previously registered listener. Pass the exact function reference used with `on` or `once`.\n\n### removeAllListeners\n\n```ts\nremoveAllListeners(event?: QueueEventName): void\n```\n\nRemove all listeners for a specific event, or all listeners for all events when called without arguments.\n\nSee [Event Hooks](/usage/event-hooks) for detailed usage examples.\n\n---\n\n## Processing Jobs\n\n### createProcessor\n\n```ts\ncreateProcessor(\n  handlers: JobHandlers,\n  options?: ProcessorOptions\n): Processor\n```\n\nCreates a job processor with the provided handlers and options.\n\n#### ProcessorOptions\n\n```ts\ninterface ProcessorOptions {\n  workerId?: string;\n  batchSize?: number;\n  concurrency?: number;\n  groupConcurrency?: number;\n  pollInterval?: number;\n  onError?: (error: Error) => void;\n  verbose?: boolean;\n  jobType?: string | string[];\n}\n```\n\n- `groupConcurrency` - Optional global per-group concurrency limit (positive integer). Applies only to jobs with `group.id`; ungrouped jobs are unaffected.\n\n---\n\n## Background Supervisor\n\n### createSupervisor\n\n```ts\ncreateSupervisor(options?: SupervisorOptions): Supervisor\n```\n\nCreates a background supervisor that automatically runs maintenance tasks on a configurable interval: reclaiming stuck jobs, cleaning up old completed jobs/events, and expiring timed-out waitpoint tokens.\n\n#### SupervisorOptions\n\n```ts\ninterface SupervisorOptions {\n  intervalMs?: number; // default: 60000\n  stuckJobsTimeoutMinutes?: number; // default: 10\n  cleanupJobsDaysToKeep?: number; // default: 30 (0 to disable)\n  cleanupEventsDaysToKeep?: number; // default: 30 (0 to disable)\n  cleanupBatchSize?: number; // default: 1000\n  reclaimStuckJobs?: boolean; // default: true\n  expireTimedOutTokens?: boolean; // default: true\n  onError?: (error: Error) => void; // default: console.error\n  verbose?: boolean;\n}\n```\n\n#### Supervisor\n\n```ts\ninterface Supervisor {\n  start(): Promise<SupervisorRunResult>;\n  startInBackground(): void;\n  stop(): void;\n  stopAndDrain(timeoutMs?: number): Promise<void>;\n  isRunning(): boolean;\n}\n```\n\n- `start()` runs all tasks once and returns the results (serverless-friendly).\n- `startInBackground()` starts a background loop that runs every `intervalMs`.\n- `stopAndDrain()` stops the loop and waits for the current run to finish.\n\n#### SupervisorRunResult\n\n```ts\ninterface SupervisorRunResult {\n  reclaimedJobs: number;\n  cleanedUpJobs: number;\n  cleanedUpEvents: number;\n  expiredTokens: number;\n}\n```\n\nSee [Long-Running Server](/usage/long-running-server#background-supervisor) for usage examples.\n\n---\n\n## Accessing the Underlying Client\n\n### getPool\n\n```ts\ngetPool(): Pool\n```\n\nReturns the PostgreSQL connection pool instance. Only available when using the PostgreSQL backend.\n\n> **Note:** Throws an error if called when using the Redis backend.\n\n### getRedisClient\n\n```ts\ngetRedisClient(): Redis\n```\n\nReturns the `ioredis` client instance. Only available when using the Redis backend.\n\n> **Note:** Throws an error if called when using the PostgreSQL backend."
   },
   {
     "slug": "api/job-record",
     "title": "JobRecord",
     "description": "",
-    "content": "The `JobRecord` interface represents a job stored in the queue, including its status, attempts, and metadata.\n\n## Fields\n\n- `id`: _number_ — Unique job ID.\n- `jobType`: _string_ — The type of the job.\n- `payload`: _any_ — The job payload.\n- `status`:\n  _'pending' | 'processing' | 'completed' | 'failed' | 'cancelled'_ —\n  Current job status.\n- `createdAt`: _Date_ — When the job was created.\n- `updated_at`: _Date_ — When the job was last updated.\n- `locked_at`: _Date | null_ — When the job was locked for\n  processing.\n- `locked_by`: _string | null_ — Worker that locked the job.\n- `attempts`: _number_ — Number of attempts so far.\n- `maxAttempts`: _number_ — Maximum allowed attempts.\n- `nextAttemptAt`: _Date | null_ — When the next attempt is\n  scheduled.\n- `priority`: _number_ — Job priority.\n- `runAt`: _Date_ — When the job is scheduled to run.\n- `pendingReason?`: _string | null_ — Reason for pending\n  status.\n- `errorHistory?`: _\\{ message: string; timestamp: string \\}[]_ — Error history for the job.\n- `timeoutMs?`: _number | null_ — Timeout for this job in\n  milliseconds.\n- `failureReason?`: _FailureReason | null_ — Reason for last\n  failure, if any.\n- `completedAt`: _Date | null_ — When the job was completed.\n- `startedAt`: _Date | null_ — When the job was first picked up\n  for processing.\n- `lastRetriedAt`: _Date | null_ — When the job was last\n  retried.\n- `lastFailedAt`: _Date | null_ — When the job last failed.\n- `lastCancelledAt`: _Date | null_ — When the job was last\n  cancelled.\n- `tags?`: _string[]_ — Tags for this job. Used for grouping, searching, or batch operations.\n- `idempotencyKey?`: _string | null_ — The idempotency key for this job, if one was provided when the job was created.\n- `progress?`: _number | null_ — Progress percentage (0–100) reported by the handler via `ctx.setProgress()`. `null` if no progress has been reported. See [Progress Tracking](/usage/progress-tracking).\n\n## Example\n\n```json\n{\n  \"id\": 1,\n  \"jobType\": \"email\",\n  \"payload\": { \"to\": \"user@example.com\", \"subject\": \"Hello\" },\n  \"status\": \"pending\",\n  \"createdAt\": \"2024-06-01T12:00:00Z\",\n  \"tags\": [\"welcome\", \"user\"],\n  \"idempotencyKey\": \"welcome-email-user-123\",\n  \"progress\": null\n}\n```"
+    "content": "The `JobRecord` interface represents a job stored in the queue, including its status, attempts, and metadata.\n\n## Fields\n\n- `id`: _number_ — Unique job ID.\n- `jobType`: _string_ — The type of the job.\n- `payload`: _any_ — The job payload.\n- `status`:\n  _'pending' | 'processing' | 'completed' | 'failed' | 'cancelled'_ —\n  Current job status.\n- `createdAt`: _Date_ — When the job was created.\n- `updated_at`: _Date_ — When the job was last updated.\n- `locked_at`: _Date | null_ — When the job was locked for\n  processing.\n- `locked_by`: _string | null_ — Worker that locked the job.\n- `attempts`: _number_ — Number of attempts so far.\n- `maxAttempts`: _number_ — Maximum allowed attempts.\n- `nextAttemptAt`: _Date | null_ — When the next attempt is\n  scheduled.\n- `priority`: _number_ — Job priority.\n- `runAt`: _Date_ — When the job is scheduled to run.\n- `pendingReason?`: _string | null_ — Reason for pending\n  status.\n- `errorHistory?`: _\\{ message: string; timestamp: string \\}[]_ — Error history for the job.\n- `timeoutMs?`: _number | null_ — Timeout for this job in\n  milliseconds.\n- `failureReason?`: _FailureReason | null_ — Reason for last\n  failure, if any.\n- `completedAt`: _Date | null_ — When the job was completed.\n- `startedAt`: _Date | null_ — When the job was first picked up\n  for processing.\n- `lastRetriedAt`: _Date | null_ — When the job was last\n  retried.\n- `lastFailedAt`: _Date | null_ — When the job last failed.\n- `lastCancelledAt`: _Date | null_ — When the job was last\n  cancelled.\n- `tags?`: _string[]_ — Tags for this job. Used for grouping, searching, or batch operations.\n- `idempotencyKey?`: _string | null_ — The idempotency key for this job, if one was provided when the job was created.\n- `progress?`: _number | null_ — Progress percentage (0–100) reported by the handler via `ctx.setProgress()`. `null` if no progress has been reported. See [Progress Tracking](/usage/progress-tracking).\n- `output?`: _unknown_ — Handler output stored via `ctx.setOutput(data)` or by returning a value from the handler. `null` if no output has been stored. See [Job Output](/usage/job-output).\n\n## Example\n\n```json\n{\n  \"id\": 1,\n  \"jobType\": \"email\",\n  \"payload\": { \"to\": \"user@example.com\", \"subject\": \"Hello\" },\n  \"status\": \"completed\",\n  \"createdAt\": \"2024-06-01T12:00:00Z\",\n  \"tags\": [\"welcome\", \"user\"],\n  \"idempotencyKey\": \"welcome-email-user-123\",\n  \"progress\": 100,\n  \"output\": { \"messageId\": \"abc-123\", \"sentAt\": \"2024-06-01T12:00:05Z\" }\n}\n```"
   },
   {
     "slug": "api/processor",
     "title": "Processor",
     "description": "",
-    "content": "The `Processor` interface represents a job processor that can process jobs from the queue, either in the background or synchronously.\n\n## Creating a processor\n\nCreate a processor by calling `createProcessor` on the queue.\n\n```ts\nconst jobQueue = getJobQueue();\nconst processor = queue.createProcessor(handlers, options);\n```\n\n### ProcessorOptions\n\n```ts\ninterface ProcessorOptions {\n  workerId?: string;\n  batchSize?: number;\n  concurrency?: number;\n  pollInterval?: number;\n  onError?: (error: Error) => void;\n  verbose?: boolean;\n  jobType?: string | string[];\n}\n```\n\n## Methods\n\n### startInBackground\n\n```ts\nstartInBackground(): void\n```\n\nStart the job processor in the background. This will run continuously and process jobs as they become available. It polls for new jobs every `pollInterval` milliseconds (default: 5 seconds).\n\n### stop\n\n```ts\nstop(): void\n```\n\nStop the job processor that runs in the background. Does not wait for in-flight jobs to finish.\n\n### stopAndDrain\n\n```ts\nstopAndDrain(timeoutMs?: number): Promise<void>\n```\n\nStop the processor and wait for the current in-flight batch to finish before resolving. Accepts an optional timeout in milliseconds (default: `30000`). If the batch does not complete within the timeout, the promise resolves anyway so your process is not stuck indefinitely. Useful for graceful shutdown (e.g., SIGTERM handling). See [Long-Running Server](/usage/long-running-server) for a full example.\n\n### isRunning\n\n```ts\nisRunning(): boolean\n```\n\nCheck if the job processor is running.\n\n### start\n\n```ts\nstart(): Promise<number>\n```\n\nStart the job processor synchronously. This will process jobs immediately and then stop. Returns the number of jobs processed."
+    "content": "The `Processor` interface represents a job processor that can process jobs from the queue, either in the background or synchronously.\n\n## Creating a processor\n\nCreate a processor by calling `createProcessor` on the queue.\n\n```ts\nconst jobQueue = getJobQueue();\nconst processor = queue.createProcessor(handlers, options);\n```\n\n### ProcessorOptions\n\n```ts\ninterface ProcessorOptions {\n  workerId?: string;\n  batchSize?: number;\n  concurrency?: number;\n  groupConcurrency?: number;\n  pollInterval?: number;\n  onError?: (error: Error) => void;\n  verbose?: boolean;\n  jobType?: string | string[];\n}\n```\n\n- `groupConcurrency` sets a global per-group concurrency cap across all workers/instances for jobs with `group.id`.\n- Must be a positive integer when provided.\n- Jobs without `group.id` are not affected.\n\n## Methods\n\n### startInBackground\n\n```ts\nstartInBackground(): void\n```\n\nStart the job processor in the background. This will run continuously and process jobs as they become available. It polls for new jobs every `pollInterval` milliseconds (default: 5 seconds).\n\n### stop\n\n```ts\nstop(): void\n```\n\nStop the job processor that runs in the background. Does not wait for in-flight jobs to finish.\n\n### stopAndDrain\n\n```ts\nstopAndDrain(timeoutMs?: number): Promise<void>\n```\n\nStop the processor and wait for the current in-flight batch to finish before resolving. Accepts an optional timeout in milliseconds (default: `30000`). If the batch does not complete within the timeout, the promise resolves anyway so your process is not stuck indefinitely. Useful for graceful shutdown (e.g., SIGTERM handling). See [Long-Running Server](/usage/long-running-server) for a full example.\n\n### isRunning\n\n```ts\nisRunning(): boolean\n```\n\nCheck if the job processor is running.\n\n### start\n\n```ts\nstart(): Promise<number>\n```\n\nStart the job processor synchronously. This will process jobs immediately and then stop. Returns the number of jobs processed."
   },
   {
     "slug": "api/tags",
@@ -117,7 +117,7 @@
     "slug": "intro/comparison",
     "title": "Comparison",
     "description": "How DataQueue compares to BullMQ and Trigger.dev",
-    "content": "Choosing a job queue depends on your stack, infrastructure preferences, and the features you need. Here is a side-by-side comparison of **DataQueue**, **BullMQ**, and **Trigger.dev**.\n\n| Feature                 | DataQueue                                       | BullMQ                                      | Trigger.dev                             |\n| ----------------------- | ----------------------------------------------- | ------------------------------------------- | --------------------------------------- |\n| **Backend**             | PostgreSQL or Redis                             | Redis only                                  | Cloud or self-hosted (Postgres + Redis) |\n| **Type Safety**         | Full generic `PayloadMap`                       | Basic types                                 | Full TypeScript tasks                   |\n| **Scheduling**          | `runAt`, Cron                                   | Cron, delayed, recurring                    | Cron, delayed                           |\n| **Retries**             | Exponential backoff, configurable `maxAttempts` | Exponential backoff, custom strategies, DLQ | Auto retries, bulk replay, DLQ          |\n| **Priority**            | Integer priority                                | Priority levels                             | Queue-based priority                    |\n| **Concurrency Control** | `batchSize` + `concurrency`                     | Built-in                                    | Per-task + shared limits                |\n| **Rate Limiting**       | -                                               | Yes                                         | Via concurrency limits                  |\n| **Job Flows / DAGs**    | -                                               | Parent-child flows                          | Workflows                               |\n| **Dashboard**           | Built-in Next.js package                        | Third-party (Bull Board, etc.)              | Built-in web dashboard                  |\n| **Wait / Pause Jobs**   | `waitFor`, `waitUntil`, token system            | -                                           | Durable execution                       |\n| **Human-in-the-Loop**   | Token system                                    | -                                           | Yes                                     |\n| **Progress Tracking**   | Yes (0-100%)                                    | Yes                                         | Yes (realtime)                          |\n| **Serverless-First**    | Yes                                             | No (needs long-running process)             | Yes (cloud)                             |\n| **Self-Hosted**         | Yes                                             | Yes (your Redis)                            | Yes (containers)                        |\n| **Cloud Option**        | -                                               | -                                           | Yes                                     |\n| **License**             | MIT                                             | MIT                                         | Apache-2.0                              |\n| **Pricing**             | Free (OSS)                                      | Free (OSS)                                  | Free tier + paid plans                  |\n| **Infrastructure**      | Your own Postgres or Redis                      | Your own Redis                              | Their cloud or your infra               |\n\n## Where DataQueue shines\n\n- **Serverless-first** — designed from the ground up for Vercel, AWS Lambda, and other serverless platforms. No long-running process required.\n- **Use your existing database** — back your queue with PostgreSQL or Redis. No additional infrastructure to provision or pay for.\n- **Wait and token system** — pause jobs with `waitFor`, `waitUntil`, or token-based waits for human-in-the-loop workflows, all within a single handler function.\n- **Type-safe PayloadMap** — a generic `PayloadMap` gives you compile-time validation of every job type and its payload, catching bugs before they reach production.\n- **Built-in Next.js dashboard** — add a full admin UI to your Next.js app with a single route file. No separate service to deploy."
+    "content": "Choosing a job queue depends on your stack, infrastructure preferences, and the features you need. Here is a side-by-side comparison of **DataQueue**, **BullMQ**, and **Trigger.dev**.\n\n| Feature                 | DataQueue                                               | BullMQ                                      | Trigger.dev                             |\n| ----------------------- | ------------------------------------------------------- | ------------------------------------------- | --------------------------------------- |\n| **Backend**             | PostgreSQL or Redis                                     | Redis only                                  | Cloud or self-hosted (Postgres + Redis) |\n| **Type Safety**         | Full generic `PayloadMap`                               | Basic types                                 | Full TypeScript tasks                   |\n| **Scheduling**          | `runAt`, Cron                                           | Cron, delayed, recurring                    | Cron, delayed                           |\n| **Retries**             | Exponential backoff, configurable `maxAttempts`         | Exponential backoff, custom strategies, DLQ | Auto retries, bulk replay, DLQ          |\n| **Priority**            | Integer priority                                        | Priority levels                             | Queue-based priority                    |\n| **Concurrency Control** | `batchSize` + `concurrency` + global `groupConcurrency` | Built-in                                    | Per-task + shared limits                |\n| **Rate Limiting**       | -                                                       | Yes                                         | Via concurrency limits                  |\n| **Job Flows / DAGs**    | -                                                       | Parent-child flows                          | Workflows                               |\n| **Dashboard**           | Built-in Next.js package                                | Third-party (Bull Board, etc.)              | Built-in web dashboard                  |\n| **Wait / Pause Jobs**   | `waitFor`, `waitUntil`, token system                    | -                                           | Durable execution                       |\n| **Human-in-the-Loop**   | Token system                                            | -                                           | Yes                                     |\n| **Progress Tracking**   | Yes (0-100%)                                            | Yes                                         | Yes (realtime)                          |\n| **Serverless-First**    | Yes                                                     | No (needs long-running process)             | Yes (cloud)                             |\n| **Self-Hosted**         | Yes                                                     | Yes (your Redis)                            | Yes (containers)                        |\n| **Cloud Option**        | -                                                       | -                                           | Yes                                     |\n| **License**             | MIT                                                     | MIT                                         | Apache-2.0                              |\n| **Pricing**             | Free (OSS)                                              | Free (OSS)                                  | Free tier + paid plans                  |\n| **Infrastructure**      | Your own Postgres or Redis                              | Your own Redis                              | Their cloud or your infra               |\n\n## Where DataQueue shines\n\n- **Serverless-first** — designed from the ground up for Vercel, AWS Lambda, and other serverless platforms. No long-running process required.\n- **Use your existing database** — back your queue with PostgreSQL or Redis. No additional infrastructure to provision or pay for.\n- **Wait and token system** — pause jobs with `waitFor`, `waitUntil`, or token-based waits for human-in-the-loop workflows, all within a single handler function.\n- **Type-safe PayloadMap** — a generic `PayloadMap` gives you compile-time validation of every job type and its payload, catching bugs before they reach production.\n- **Built-in Next.js dashboard** — add a full admin UI to your Next.js app with a single route file. No separate service to deploy."
   },
   {
     "slug": "intro",
@@ -141,7 +141,7 @@
     "slug": "usage/add-job",
     "title": "Add Job",
     "description": "",
-    "content": "You can add jobs to the queue from your application logic, such as in a [server function](https://react.dev/reference/rsc/server-functions):\n\n```typescript title=\"@/app/actions/send-email.ts\"\n'use server';\n\nimport { getJobQueue } from '@/lib/queue';\nimport { revalidatePath } from 'next/cache';\n\nexport const sendEmail = async ({\n  name,\n  email,\n}: {\n  name: string;\n  email: string;\n}) => {\n  // Add a welcome email job\n  const jobQueue = getJobQueue();try {\n    const runAt = new Date(Date.now() + 5 * 1000); // Run 5 seconds from nowconst job = await jobQueue.addJob({\n      jobType: 'send_email',\n      payload: {\n        to: email,\n        subject: 'Welcome to our platform!',\n        body: `Hi ${name}, welcome to our platform!`,\n      },\n      priority: 10, // Higher number = higher priority\n      runAt: runAt,\n      tags: ['welcome', 'user'], // Add tags for grouping/searching\n    });\n\n    revalidatePath('/');\n    return { job };\n  } catch (error) {\n    console.error('Error adding job:', error);\n    throw error;\n  }\n};\n```\n\nIn the example above, a job is added to the queue to send an email. The job type is `send_email`, and the payload includes the recipient's email, subject, and body.\n\nWhen adding a job, you can set its `priority`, schedule when it should run using `runAt`, and specify a timeout in milliseconds with `timeoutMs`.\n\nYou can also add `tags` (an array of strings) to group, search, or batch jobs by category. See [Tags](/api/tags) for more details.\n\n## Idempotency\n\nYou can provide an `idempotencyKey` when adding a job to prevent duplicate jobs. If a job with the same key already exists in the queue, `addJob` returns the existing job's ID instead of creating a new one.\n\nThis is useful for preventing duplicates caused by retries, double-clicks, webhook replays, or serverless function re-invocations.\n\n```typescript title=\"@/app/actions/send-welcome.ts\"\n'use server';\n\nimport { getJobQueue } from '@/lib/queue';\n\nexport const sendWelcomeEmail = async (userId: string, email: string) => {\n  const jobQueue = getJobQueue();const jobId = await jobQueue.addJob({\n    jobType: 'send_email',\n    payload: {\n      to: email,\n      subject: 'Welcome!',\n      body: `Welcome to our platform!`,\n    },\n    idempotencyKey: `welcome-email-${userId}`, // prevents duplicate welcome emails\n  });\n\n  return { jobId };\n};\n```\n\nIn the example above, calling `sendWelcomeEmail` multiple times for the same `userId` will only create one job. Subsequent calls return the existing job's ID.\n\n### Behavior\n\n- **No key provided**: Works exactly as before, no uniqueness check is performed.\n- **Key provided, no conflict**: The job is inserted and its new ID is returned.\n- **Key provided, conflict**: The existing job's ID is returned. The existing job is **not** updated.\n- **Scope**: The key is unique across the entire `job_queue` table regardless of job status. Once a key exists, it cannot be reused until the job is cleaned up via [`cleanupOldJobs`](/usage/cleanup-jobs)."
+    "content": "You can add jobs to the queue from your application logic, such as in a [server function](https://react.dev/reference/rsc/server-functions):\n\n```typescript title=\"@/app/actions/send-email.ts\"\n'use server';\n\nimport { getJobQueue } from '@/lib/queue';\nimport { revalidatePath } from 'next/cache';\n\nexport const sendEmail = async ({\n  name,\n  email,\n}: {\n  name: string;\n  email: string;\n}) => {\n  // Add a welcome email job\n  const jobQueue = getJobQueue();try {\n    const runAt = new Date(Date.now() + 5 * 1000); // Run 5 seconds from nowconst job = await jobQueue.addJob({\n      jobType: 'send_email',\n      payload: {\n        to: email,\n        subject: 'Welcome to our platform!',\n        body: `Hi ${name}, welcome to our platform!`,\n      },\n      priority: 10, // Higher number = higher priority\n      runAt: runAt,\n      tags: ['welcome', 'user'], // Add tags for grouping/searching\n    });\n\n    revalidatePath('/');\n    return { job };\n  } catch (error) {\n    console.error('Error adding job:', error);\n    throw error;\n  }\n};\n```\n\nIn the example above, a job is added to the queue to send an email. The job type is `send_email`, and the payload includes the recipient's email, subject, and body.\n\nWhen adding a job, you can set its `priority`, schedule when it should run using `runAt`, and specify a timeout in milliseconds with `timeoutMs`.\n\nYou can also add `tags` (an array of strings) to group, search, or batch jobs by category. See [Tags](/api/tags) for more details.\n\n## Batch Insert\n\nWhen you need to enqueue many jobs at once, use `addJobs` instead of calling `addJob` in a loop. It batches the inserts into a single database round-trip (PostgreSQL) or a single atomic Lua script (Redis), which is significantly faster.\n\n```typescript title=\"@/app/actions/send-bulk.ts\"\n'use server';\n\nimport { getJobQueue } from '@/lib/queue';\n\nexport const sendBulkEmails = async (\n  recipients: { email: string; name: string }[],\n) => {\n  const jobQueue = getJobQueue();const jobIds = await jobQueue.addJobs(\n    recipients.map((r) => ({\n      jobType: 'send_email' as const,\n      payload: {\n        to: r.email,\n        subject: 'Newsletter',\n        body: `Hi ${r.name}, here's your update!`,\n      },\n      tags: ['newsletter'],\n    })),\n  );\n  // jobIds[i] corresponds to recipients[i]\n  return { jobIds };\n};\n```\n\n`addJobs` returns an array of job IDs in the **same order** as the input array. Each job can independently have its own `priority`, `runAt`, `tags`, `idempotencyKey`, and other options.\n\n- **Empty array**: `addJobs([])` returns `[]` immediately without touching the database.\n- **Idempotency**: Each job's `idempotencyKey` is handled independently. Duplicate keys resolve to the existing job's ID.\n- **Transactional**: The `{ db }` option works with `addJobs` the same way as `addJob` (PostgreSQL only).\n\n## Idempotency\n\nYou can provide an `idempotencyKey` when adding a job to prevent duplicate jobs. If a job with the same key already exists in the queue, `addJob` returns the existing job's ID instead of creating a new one.\n\nThis is useful for preventing duplicates caused by retries, double-clicks, webhook replays, or serverless function re-invocations.\n\n```typescript title=\"@/app/actions/send-welcome.ts\"\n'use server';\n\nimport { getJobQueue } from '@/lib/queue';\n\nexport const sendWelcomeEmail = async (userId: string, email: string) => {\n  const jobQueue = getJobQueue();const jobId = await jobQueue.addJob({\n    jobType: 'send_email',\n    payload: {\n      to: email,\n      subject: 'Welcome!',\n      body: `Welcome to our platform!`,\n    },\n    idempotencyKey: `welcome-email-${userId}`, // prevents duplicate welcome emails\n  });\n\n  return { jobId };\n};\n```\n\nIn the example above, calling `sendWelcomeEmail` multiple times for the same `userId` will only create one job. Subsequent calls return the existing job's ID.\n\n### Behavior\n\n- **No key provided**: Works exactly as before, no uniqueness check is performed.\n- **Key provided, no conflict**: The job is inserted and its new ID is returned.\n- **Key provided, conflict**: The existing job's ID is returned. The existing job is **not** updated.\n- **Scope**: The key is unique across the entire `job_queue` table regardless of job status. Once a key exists, it cannot be reused until the job is cleaned up via [`cleanupOldJobs`](/usage/cleanup-jobs).\n\n## Transactional Job Creation\n\n> **Note:** Transactional job creation is only available with the **PostgreSQL** backend.\n\nYou can insert a job within an existing database transaction by passing an external database client via the `db` option. This guarantees that the job is enqueued **atomically** with your other database writes — if the transaction rolls back, the job is never enqueued.\n\nThis is useful when you need to ensure that a job is only created when a related database operation succeeds (e.g., creating a user and enqueuing a welcome email in the same transaction).\n\n```typescript title=\"@/app/actions/register.ts\"\n'use server';\n\nimport { Pool } from 'pg';\nimport { getJobQueue } from '@/lib/queue';\n\nconst pool = new Pool({ connectionString: process.env.DATABASE_URL });\n\nexport const registerUser = async (email: string, name: string) => {\n  const client = await pool.connect();\n  try {\n    await client.query('BEGIN');\n\n    // Insert the user\n    await client.query('INSERT INTO users (email, name) VALUES ($1, $2)', [\n      email,\n      name,\n    ]);\n\n    // Enqueue the welcome email in the same transactionconst jobQueue = getJobQueue();\n    await jobQueue.addJob(\n      {\n        jobType: 'send_email',\n        payload: { to: email, subject: 'Welcome!', body: `Hi ${name}!` },\n      },\n      { db: client }, // Use the transaction client\n    );\n\n    await client.query('COMMIT');\n  } catch (error) {\n    await client.query('ROLLBACK');\n    throw error;\n  } finally {\n    client.release();\n  }\n};\n```\n\n### How it works\n\n- When `db` is provided, the `INSERT` into the `job_queue` table and the associated job event are both executed on the supplied client.\n- The library does **not** call `client.release()` — you are responsible for managing the client lifecycle.\n- If the transaction is rolled back, both the job and its event are discarded.\n- When `db` is **not** provided, `addJob` behaves exactly as before (gets a connection from the internal pool)."
   },
   {
     "slug": "usage/building-with-ai",
@@ -159,7 +159,7 @@
     "slug": "usage/cleanup-jobs",
     "title": "Cleanup Jobs",
     "description": "",
-    "content": "If you have a lot of jobs, you may want to clean up old ones—for example, keeping only jobs from the last 30 days. You can do this by calling the `cleanupOldJobs` method. The example below shows an API route (`/api/cron/cleanup`) that can be triggered by a cron job:\n\n```typescript title=\"@/app/api/cron/cleanup.ts\"\nimport { getJobQueue } from '@/lib/queue';\nimport { NextResponse } from 'next/server';\n\nexport async function GET(request: Request) {\n  const authHeader = request.headers.get('authorization');\n  if (authHeader !== `Bearer ${process.env.CRON_SECRET}`) {\n    return NextResponse.json({ message: 'Unauthorized' }, { status: 401 });\n  }\n\n  try {const jobQueue = getJobQueue();\n\n    // Clean up old jobs (keep only the last 30 days)\n    const deleted = await jobQueue.cleanupOldJobs(30);\n    console.log(`Deleted ${deleted} old jobs`);\n\n    return NextResponse.json({\n      message: 'Old jobs cleaned up',\n      deleted,\n    });\n  } catch (error) {\n    console.error('Error cleaning up jobs:', error);\n    return NextResponse.json(\n      { message: 'Failed to clean up jobs' },\n      { status: 500 },\n    );\n  }\n}\n```\n\n#### Scheduling the Cleanup Job with Cron\n\nAdd the following to your `vercel.json` to call the cleanup route every day at midnight:\n\n```json title=\"vercel.json\"\n{\n  \"crons\": [\n    {\n      \"path\": \"/api/cron/cleanup\",\n      \"schedule\": \"0 0 * * *\"\n    }\n  ]\n}\n```"
+    "content": "> **Note:** Running a long-lived server? Use\n  [`createSupervisor()`](/usage/long-running-server#background-supervisor) to\n  automate job cleanup instead of calling `cleanupOldJobs` manually.\n\nIf you have a lot of jobs, you may want to clean up old ones—for example, keeping only jobs from the last 30 days. You can do this by calling the `cleanupOldJobs` method. The example below shows an API route (`/api/cron/cleanup`) that can be triggered by a cron job:\n\n```typescript title=\"@/app/api/cron/cleanup.ts\"\nimport { getJobQueue } from '@/lib/queue';\nimport { NextResponse } from 'next/server';\n\nexport async function GET(request: Request) {\n  const authHeader = request.headers.get('authorization');\n  if (authHeader !== `Bearer ${process.env.CRON_SECRET}`) {\n    return NextResponse.json({ message: 'Unauthorized' }, { status: 401 });\n  }\n\n  try {const jobQueue = getJobQueue();\n\n    // Clean up old jobs (keep only the last 30 days)\n    const deleted = await jobQueue.cleanupOldJobs(30);\n    console.log(`Deleted ${deleted} old jobs`);\n\n    return NextResponse.json({\n      message: 'Old jobs cleaned up',\n      deleted,\n    });\n  } catch (error) {\n    console.error('Error cleaning up jobs:', error);\n    return NextResponse.json(\n      { message: 'Failed to clean up jobs' },\n      { status: 500 },\n    );\n  }\n}\n```\n\n#### Scheduling the Cleanup Job with Cron\n\nAdd the following to your `vercel.json` to call the cleanup route every day at midnight:\n\n```json title=\"vercel.json\"\n{\n  \"crons\": [\n    {\n      \"path\": \"/api/cron/cleanup\",\n      \"schedule\": \"0 0 * * *\"\n    }\n  ]\n}\n```"
   },
   {
     "slug": "usage/cron-jobs",
@@ -185,11 +185,17 @@
     "description": "",
     "content": "You can edit a pending job by its ID to update its properties before it is processed. Only jobs with status 'pending' can be edited. Attempting to edit a job with any other status (processing, completed, failed, cancelled) will silently fail.\n\n## Basic Usage\n\n```typescript title=\"@/app/api/edit-job/route.ts\"\nimport { NextRequest, NextResponse } from 'next/server';\nimport { getJobQueue } from '@/lib/queue';\n\nexport async function POST(request: NextRequest) {\n  try {\n    const { jobId, updates } = await request.json();const jobQueue = getJobQueue();\n    await jobQueue.editJob(jobId, updates);\n    return NextResponse.json({ message: 'Job updated' });\n  } catch (error) {\n    console.error('Error editing job:', error);\n    return NextResponse.json(\n      { message: 'Failed to edit job' },\n      { status: 500 },\n    );\n  }\n}\n```\n\n## Editable Fields\n\nAll fields in `EditJobOptions` are optional - only the fields you provide will be updated. The following fields can be edited:\n\n- `payload` - The job payload data\n- `priority` - Job priority (higher runs first)\n- `maxAttempts` - Maximum number of attempts\n- `runAt` - When to run the job (Date or null)\n- `timeoutMs` - Timeout for the job in milliseconds\n- `tags` - Tags for grouping, searching, or batch operations\n\n**Note:** `jobType` cannot be changed. If you need to change the job type, you should cancel the job and create a new one.\n\n## Examples\n\n### Edit Payload\n\n```typescript\n// Update the payload of a pending job\nawait jobQueue.editJob(jobId, {\n  payload: { to: 'newemail@example.com', subject: 'Updated Subject' },\n});\n```\n\n### Edit Priority\n\n```typescript\n// Increase the priority of a job\nawait jobQueue.editJob(jobId, {\n  priority: 10,\n});\n```\n\n### Edit Scheduled Time\n\n```typescript\n// Reschedule a job to run in 1 hour\nawait jobQueue.editJob(jobId, {\n  runAt: new Date(Date.now() + 60 * 60 * 1000),\n});\n\n// Schedule a job to run immediately (or as soon as possible)\nawait jobQueue.editJob(jobId, {\n  runAt: null,\n});\n```\n\n### Edit Multiple Fields\n\n```typescript\n// Update multiple fields at once\nawait jobQueue.editJob(jobId, {\n  payload: { to: 'updated@example.com', subject: 'New Subject' },\n  priority: 5,\n  maxAttempts: 10,\n  timeoutMs: 30000,\n  tags: ['urgent', 'priority'],\n});\n```\n\n### Partial Updates\n\n```typescript\n// Only update what you need - other fields remain unchanged\nawait jobQueue.editJob(jobId, {\n  priority: 10,\n  // payload, maxAttempts, runAt, timeoutMs, and tags remain unchanged\n});\n```\n\n### Clear Tags or Timeout\n\n```typescript\n// Remove tags by setting to undefined\nawait jobQueue.editJob(jobId, {\n  tags: undefined,\n});\n\n// Remove timeout by setting to undefined\nawait jobQueue.editJob(jobId, {\n  timeoutMs: undefined,\n});\n```\n\n## Batch Editing\n\nYou can edit multiple pending jobs at once using `editAllPendingJobs`. This is useful when you need to update many jobs that match certain criteria. The function returns the number of jobs that were edited.\n\n### Basic Batch Edit\n\n```typescript\n// Edit all pending jobs\nconst editedCount = await jobQueue.editAllPendingJobs(undefined, {\n  priority: 10,\n});\nconsole.log(`Edited ${editedCount} jobs`);\n```\n\n### Filter by Job Type\n\n```typescript\n// Edit all pending email jobs\nconst editedCount = await jobQueue.editAllPendingJobs(\n  { jobType: 'email' },\n  {\n    priority: 5,\n  },\n);\n```\n\n### Filter by Priority\n\n```typescript\n// Edit all pending jobs with priority 1\nconst editedCount = await jobQueue.editAllPendingJobs(\n  { priority: 1 },\n  {\n    priority: 5,\n  },\n);\n```\n\n### Filter by Tags\n\n```typescript\n// Edit all pending jobs with 'urgent' tag\nconst editedCount = await jobQueue.editAllPendingJobs(\n  { tags: { values: ['urgent'], mode: 'any' } },\n  {\n    priority: 10,\n  },\n);\n```\n\n### Filter by Scheduled Time\n\n```typescript\n// Edit all pending jobs scheduled in the future\nconst editedCount = await jobQueue.editAllPendingJobs(\n  { runAt: { gte: new Date() } },\n  {\n    priority: 10,\n  },\n);\n\n// Edit all pending jobs scheduled before a specific date\nconst editedCount = await jobQueue.editAllPendingJobs(\n  { runAt: { lt: new Date('2024-12-31') } },\n  {\n    priority: 5,\n  },\n);\n```\n\n### Combined Filters\n\n```typescript\n// Edit all pending email jobs with 'urgent' tag\nconst editedCount = await jobQueue.editAllPendingJobs(\n  {\n    jobType: 'email',\n    tags: { values: ['urgent'], mode: 'any' },\n  },\n  {\n    priority: 10,\n    maxAttempts: 5,\n  },\n);\n```\n\n### Batch Edit Notes\n\n- Only pending jobs are edited. Jobs with other statuses (processing, completed, failed, cancelled) are not affected.\n- The function returns the number of jobs that were successfully edited.\n- Edit events are recorded for each affected job, just like single job edits.\n- If no fields are provided in the updates object, the function returns 0 and no jobs are modified.\n\n## When to Use Edit vs Cancel vs Retry\n\n- **Edit**: Use when you want to modify a pending job's properties before it runs\n- **Cancel**: Use when you want to completely remove a pending job from the queue\n- **Retry**: Use when you want to retry a failed job (sets status back to pending)\n\n## Error Handling\n\nThe `editJob` function silently fails if you try to edit a non-pending job. This means:\n\n- No error is thrown\n- The job remains unchanged\n- The operation completes successfully (but does nothing)\n\nTo check if an edit was successful, you can:\n\n```typescript\nconst job = await jobQueue.getJob(jobId);\nif (job?.status === 'pending') {\n  // Job is still pending, edit might have succeeded\n  // Check if the fields you wanted to update actually changed\n  if (job.priority === newPriority) {\n    console.log('Edit successful');\n  }\n} else {\n  console.log('Job is not pending, edit was ignored');\n}\n```\n\n## Event Tracking\n\nWhen a job is edited, an 'edited' event is recorded in the job's event history. The event metadata contains the fields that were updated:\n\n```typescript\nconst events = await jobQueue.getJobEvents(jobId);\nconst editEvent = events.find((e) => e.eventType === 'edited');\nif (editEvent) {\n  console.log('Updated fields:', editEvent.metadata);\n  // { payload: {...}, priority: 10, ... }\n}\n```\n\n## Best Practices\n\n1. **Check job status before editing**: If you're unsure whether a job is pending, check its status first:\n\n```typescript\nconst job = await jobQueue.getJob(jobId);\nif (job?.status === 'pending') {\n  await jobQueue.editJob(jobId, updates);\n} else {\n  console.log('Job is not pending, cannot edit');\n}\n```\n\n2. **Use partial updates**: Only update the fields you need to change. This is more efficient and reduces the chance of accidentally overwriting other fields.\n\n3. **Validate updates**: Ensure the updated values are valid for your job handlers. For example, if your handler expects a specific payload structure, make sure the updated payload matches.\n\n4. **Consider race conditions**: If a job might be picked up for processing while you're editing it, be aware that the edit might not take effect if the job transitions to 'processing' status between your check and the edit operation.\n\n5. **Monitor events**: Use job events to track when and what was edited for audit purposes."
   },
+  {
+    "slug": "usage/event-hooks",
+    "title": "Event Hooks",
+    "description": "",
+    "content": "DataQueue emits real-time events for job lifecycle transitions, progress updates, and internal errors. Use event hooks to integrate with logging, metrics, alerting, or any custom logic without polling.\n\nEvent hooks work identically with both the PostgreSQL and Redis backends.\n\n## Listening for Events\n\nRegister listeners with `on()`, `once()`, or remove them with `off()` and `removeAllListeners()`.\n\n```typescript\nconst queue = initJobQueue<MyPayloadMap>(config);\n\nqueue.on('job:completed', (event) => {\n  console.log(`Job ${event.jobId} (${event.jobType}) completed`);\n});\n\nqueue.on('job:failed', (event) => {\n  console.error(`Job ${event.jobId} failed: ${event.error.message}`);\n  if (!event.willRetry) {\n    alertOps(`Permanent failure for job ${event.jobId}`);\n  }\n});\n\nqueue.on('error', (error) => {\n  logger.error('Queue internal error:', error);\n});\n```\n\n## Available Events\n\n| Event            | Payload                                | When                                                                              |\n| ---------------- | -------------------------------------- | --------------------------------------------------------------------------------- |\n| `job:added`      | `{ jobId, jobType }`                   | After `addJob()` or `addJobs()`                                                   |\n| `job:processing` | `{ jobId, jobType }`                   | When a processor claims and starts a job                                          |\n| `job:completed`  | `{ jobId, jobType }`                   | When a handler completes successfully                                             |\n| `job:failed`     | `{ jobId, jobType, error, willRetry }` | When a handler throws or times out                                                |\n| `job:cancelled`  | `{ jobId }`                            | After `cancelJob()`                                                               |\n| `job:retried`    | `{ jobId }`                            | After `retryJob()`                                                                |\n| `job:waiting`    | `{ jobId, jobType }`                   | When a handler enters a wait (`ctx.waitFor`, `ctx.waitUntil`, `ctx.waitForToken`) |\n| `job:progress`   | `{ jobId, progress }`                  | When a handler calls `ctx.setProgress()`                                          |\n| `error`          | `Error`                                | Internal errors from the processor or supervisor                                  |\n\n## One-Time Listeners\n\nUse `once()` when you only need to react to the first occurrence of an event.\n\n```typescript\nqueue.once('job:added', (event) => {\n  console.log('First job added:', event.jobId);\n});\n```\n\n## Removing Listeners\n\n```typescript\nconst listener = (event) => console.log(event);\n\nqueue.on('job:completed', listener);\n\n// Remove a specific listener\nqueue.off('job:completed', listener);\n\n// Remove all listeners for one event\nqueue.removeAllListeners('job:completed');\n\n// Remove all listeners for all events\nqueue.removeAllListeners();\n```\n\n## Error Monitoring\n\nThe `error` event fires for internal errors in the processor and supervisor. It works alongside the existing `onError` callback in `ProcessorOptions` and `SupervisorOptions` -- both fire independently.\n\n```typescript\nqueue.on('error', (error) => {\n  Sentry.captureException(error);\n});\n\n// onError still works as before\nconst processor = queue.createProcessor(handlers, {\n  onError: (error) => console.error('Processor error:', error),\n});\n```\n\n## Failure Retry Detection\n\nThe `job:failed` event includes a `willRetry` boolean that tells you whether the job will be retried automatically.\n\n```typescript\nqueue.on('job:failed', (event) => {\n  if (event.willRetry) {\n    metrics.increment('job.retry', { jobType: event.jobType });\n  } else {\n    metrics.increment('job.permanent_failure', { jobType: event.jobType });\n    pagerDuty.alert(`Job ${event.jobId} permanently failed`);\n  }\n});\n```\n\n## Progress Tracking\n\nThe `job:progress` event fires whenever a handler calls `ctx.setProgress()`, giving you real-time progress updates.\n\n```typescript\nqueue.on('job:progress', (event) => {\n  websocket.broadcast(`job:${event.jobId}`, { progress: event.progress });\n});\n```\n\n> **Note:** Events are emitted synchronously after the corresponding database operation\n  completes. Slow event listeners will delay the return of methods like\n  `addJob()` or the processing of the next job. Use async patterns in listeners\n  if they perform I/O."
+  },
   {
     "slug": "usage/failed-jobs",
     "title": "Failed Jobs",
     "description": "",
-    "content": "A job handler can fail for many reasons, such as a bug in the code or running out of resources.\n\nWhen a job fails, it will be marked as `failed` and retried up to `maxAttempts` times. You can set the `maxAttempts` value when adding the job to the queue.\n\nEach retry is scheduled after `2^attempts * 1 minute` from the previous attempt. You can view the error history for a job in its `errorHistory` field."
+    "content": "A job handler can fail for many reasons, such as a bug in the code or running out of resources.\n\nWhen a job fails, it is marked as `failed` and retried up to `maxAttempts` times (default: 3). You can view the error history for a job in its `errorHistory` field.\n\n## Retry configuration\n\nYou can control the retry behavior per-job using three options:\n\n| Option          | Type      | Default | Description                                    |\n| --------------- | --------- | ------- | ---------------------------------------------- |\n| `retryDelay`    | `number`  | `60`    | Base delay between retries in **seconds**      |\n| `retryBackoff`  | `boolean` | `true`  | Use exponential backoff (doubles each attempt) |\n| `retryDelayMax` | `number`  | _none_  | Maximum cap for the delay in **seconds**       |\n\n### Fixed delay\n\nSet `retryBackoff: false` to use a constant delay between retries:\n\n```ts\nawait jobQueue.addJob({\n  jobType: 'email',\n  payload: { to: 'user@example.com' },\n  maxAttempts: 5,\n  retryDelay: 30, // 30 seconds between each retry\n  retryBackoff: false,\n});\n```\n\nEvery retry will wait exactly 30 seconds.\n\n### Exponential backoff (default)\n\nWhen `retryBackoff` is `true` (the default), the delay doubles with each attempt. A small amount of random jitter is added to prevent thundering herd problems:\n\n```ts\nawait jobQueue.addJob({\n  jobType: 'email',\n  payload: { to: 'user@example.com' },\n  maxAttempts: 5,\n  retryDelay: 10, // base: 10 seconds\n  retryBackoff: true, // enabled by default\n});\n```\n\nThis produces approximate delays of 10s, 20s, 40s, 80s, ... (with jitter).\n\n### Capping the delay\n\nUse `retryDelayMax` to prevent the delay from growing unbounded:\n\n```ts\nawait jobQueue.addJob({\n  jobType: 'email',\n  payload: { to: 'user@example.com' },\n  maxAttempts: 10,\n  retryDelay: 5,\n  retryBackoff: true,\n  retryDelayMax: 300, // never wait more than 5 minutes\n});\n```\n\nDelays: ~5s, ~10s, ~20s, ~40s, ~80s, ~160s, ~300s, ~300s, ...\n\n### Default behavior\n\nIf none of the retry options are set, the legacy formula `2^attempts * 1 minute` is used. This means the first retry is after ~2 minutes, then ~4 minutes, then ~8 minutes, and so on.\n\n## Jitter\n\nWhen exponential backoff is enabled, each computed delay is multiplied by a random factor between 0.5 and 1.0. This prevents multiple failed jobs from retrying at exactly the same time, which could overload downstream services.\n\n## Cron schedules\n\nRetry configuration can also be set on cron schedules. Every job enqueued by the schedule inherits the retry settings:\n\n```ts\nawait jobQueue.addCronJob({\n  scheduleName: 'daily-report',\n  cronExpression: '0 9 * * *',\n  jobType: 'report',\n  payload: { type: 'daily' },\n  retryDelay: 60,\n  retryBackoff: true,\n  retryDelayMax: 600,\n});\n```\n\n## Editing retry config\n\nYou can update the retry configuration of a pending job:\n\n```ts\nawait jobQueue.editJob(jobId, {\n  retryDelay: 15,\n  retryBackoff: false,\n});\n```"
   },
   {
     "slug": "usage/force-kill-timeout",
@@ -207,7 +213,7 @@
     "slug": "usage/init-queue",
     "title": "Initialize Queue",
     "description": "",
-    "content": "After defining your job types, payloads, and handlers, you need to initialize the job queue which sets up the connection to your database backend.\n\n## PostgreSQL\n\n```typescript title=\"@lib/queue.ts\"\nimport { initJobQueue } from '@nicnocquee/dataqueue';\nimport { type JobPayloadMap } from './types/job-payload-map';\n\nlet jobQueue: ReturnType<typeof initJobQueue<JobPayloadMap>> | null = null;\n\nexport const getJobQueue = () => {\n  if (!jobQueue) {jobQueue = initJobQueue<JobPayloadMap>({\n      databaseConfig: {\n        connectionString: process.env.PG_DATAQUEUE_DATABASE, // Set this in your environment\n      },\n      verbose: process.env.NODE_ENV === 'development',\n    });\n  }\n  return jobQueue;\n};\n```\n\n> **Note:** The value of `connectionString` must be a [valid Postgres connection\n  string](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING-URIS).\n  For example:\n\n```dotenv\nPG_DATAQUEUE_DATABASE=postgresql://postgres:password@localhost:5432/my_database?search_path=my_schema\n```\n\n\n## Redis\n\nTo use Redis as the backend, set `backend: 'redis'` and provide `redisConfig` instead of `databaseConfig`:\n\n```typescript title=\"@lib/queue.ts\"\nimport { initJobQueue } from '@nicnocquee/dataqueue';\nimport { type JobPayloadMap } from './types/job-payload-map';\n\nlet jobQueue: ReturnType<typeof initJobQueue<JobPayloadMap>> | null = null;\n\nexport const getJobQueue = () => {\n  if (!jobQueue) {jobQueue = initJobQueue<JobPayloadMap>({\n      backend: 'redis',\n      redisConfig: {\n        url: process.env.REDIS_URL, // e.g. redis://localhost:6379\n      },\n      verbose: process.env.NODE_ENV === 'development',\n    });\n  }\n  return jobQueue;\n};\n```\n\nYou can also connect using individual connection options instead of a URL:\n\n```typescript title=\"@lib/queue.ts\"\njobQueue = initJobQueue<JobPayloadMap>({\n  backend: 'redis',\n  redisConfig: {\n    host: 'localhost',\n    port: 6379,\n    password: process.env.REDIS_PASSWORD,\n    db: 0,\n    keyPrefix: 'myapp:', // Optional, defaults to 'dq:'\n  },\n  verbose: process.env.NODE_ENV === 'development',\n});\n```\n\n> **Note:** The `keyPrefix` option lets you namespace all Redis keys. This is useful when\n  sharing a Redis instance between multiple applications or multiple queues. The\n  default prefix is `dq:`.\n\n---\n\n## Using the Queue\n\nOnce initialized, you use the queue instance identically regardless of backend. The API is the same for both PostgreSQL and Redis.\n\n```typescript title=\"@/app/actions/send-email.ts\"\nimport { getJobQueue } from '@/lib/queue';\n\nconst sendEmail = async () => {const jobQueue = getJobQueue();\n  await jobQueue.addJob({\n    jobType: 'send_email',\n    payload: {\n      to: 'test@example.com',\n      subject: 'Hello',\n      body: 'Hello, world!',\n    },\n  });\n};\n```\n\n---\n\n## SSL Configuration (PostgreSQL)\n\nMost managed Postgres providers (like DigitalOcean, Supabase, etc.) require SSL connections and use their own CA certificate (.crt file) to sign the server's certificate. To securely verify the server's identity, you must configure your client to trust this CA certificate.\n\nYou can configure SSL for your database connection in several ways, depending on your environment and security requirements.\n\n### Using PEM Strings from Environment Variables\n\nThis is ideal for serverless environments where you cannot mount files. Store your CA certificate, and optionally client certificate and key, as environment variables then pass them to the `ssl` property of the `databaseConfig` object.\n\n```typescript title=\"@lib/queue.ts\"\nimport { initJobQueue } from '@nicnocquee/dataqueue';\nimport { type JobPayloadMap } from './types/job-payload-map';\n\nlet jobQueue: ReturnType<typeof initJobQueue<JobPayloadMap>> | null = null;\n\nexport const getJobQueue = () => {\n  if (!jobQueue) {\n    jobQueue = initJobQueue<JobPayloadMap>({\n      databaseConfig: {\n        connectionString: process.env.PG_DATAQUEUE_DATABASE, // Set this in your environment\n        ssl: {\n          ca: process.env.PGSSLROOTCERT, // PEM string: the content of your .crt file\n          cert: process.env.PGSSLCERT, // PEM string (optional, for client authentication)\n          key: process.env.PGSSLKEY, // PEM string (optional, for client authentication)\n          rejectUnauthorized: true, // Always true for CA-signed certs\n        },\n      },\n      verbose: process.env.NODE_ENV === 'development',\n    });\n  }\n  return jobQueue;\n};\n```\n\n> **Note:** When using a custom CA certificate and `connectionString`, you must remove the\n  `sslmode` parameter from the connection string. Otherwise, the connection will\n  fail.\n\n### Using File Paths\n\nIf you have the CA certificate, client certificate, or key on disk, provide their absolute paths using the `file://` prefix. Only values starting with `file://` will be loaded from the file system; all others are treated as PEM strings.\n\n```typescript title=\"@lib/queue.ts\"\nimport { initJobQueue } from '@nicnocquee/dataqueue';\nimport { type JobPayloadMap } from './types/job-payload-map';\n\nlet jobQueue: ReturnType<typeof initJobQueue<JobPayloadMap>> | null = null;\n\nexport const getJobQueue = () => {\n  if (!jobQueue) {\n    jobQueue = initJobQueue<JobPayloadMap>({\n      databaseConfig: {\n        connectionString: process.env.PG_DATAQUEUE_DATABASE,\n        ssl: {\n          ca: 'file:///absolute/path/to/ca.crt', // Path to your provider's CA cert\n          cert: 'file:///absolute/path/to/client.crt', // optional, for client authentication\n          key: 'file:///absolute/path/to/client.key', // optional, for client authentication\n          rejectUnauthorized: true,\n        },\n      },\n      verbose: process.env.NODE_ENV === 'development',\n    });\n  }\n  return jobQueue;\n};\n```\n\n> **Note:** When using a custom CA certificate and `connectionString`, you must remove the\n  `sslmode` parameter from the connection string. Otherwise, the connection will\n  fail.\n\n### Skipping Certificate Validation\n\nFor convenience, you can skip certificate validation (not recommended for production) by setting `rejectUnauthorized` to `false` and without providing a custom CA certificate.\n\n```typescript title=\"@lib/queue.ts\"\nimport { initJobQueue } from '@nicnocquee/dataqueue';\nimport { type JobPayloadMap } from './types/job-payload-map';\n\nlet jobQueue: ReturnType<typeof initJobQueue<JobPayloadMap>> | null = null;\n\nexport const getJobQueue = () => {\n  if (!jobQueue) {\n    jobQueue = initJobQueue<JobPayloadMap>({\n      databaseConfig: {\n        connectionString: process.env.PG_DATAQUEUE_DATABASE,\n        ssl: {\n          rejectUnauthorized: false,\n        },\n      },\n      verbose: process.env.NODE_ENV === 'development',\n    });\n  }\n  return jobQueue;\n};\n```\n\n> **Note:** When using `rejectUnauthorized: false` and `connectionString`, you must remove\n  the `sslmode` parameter from the connection string. Otherwise, the connection\n  will fail.\n\n---\n\n## TLS Configuration (Redis)\n\nIf your Redis server requires TLS (common with managed services like AWS ElastiCache, Redis Cloud, etc.), provide TLS options in the `redisConfig`:\n\n```typescript title=\"@lib/queue.ts\"\njobQueue = initJobQueue<JobPayloadMap>({\n  backend: 'redis',\n  redisConfig: {\n    url: process.env.REDIS_URL,\n    tls: {\n      ca: process.env.REDIS_CA_CERT, // PEM string\n      rejectUnauthorized: true,\n    },\n  },\n});\n```"
+    "content": "After defining your job types, payloads, and handlers, you need to initialize the job queue which sets up the connection to your database backend.\n\n## PostgreSQL\n\n```typescript title=\"@lib/queue.ts\"\nimport { initJobQueue } from '@nicnocquee/dataqueue';\nimport { type JobPayloadMap } from './types/job-payload-map';\n\nlet jobQueue: ReturnType<typeof initJobQueue<JobPayloadMap>> | null = null;\n\nexport const getJobQueue = () => {\n  if (!jobQueue) {jobQueue = initJobQueue<JobPayloadMap>({\n      databaseConfig: {\n        connectionString: process.env.PG_DATAQUEUE_DATABASE, // Set this in your environment\n      },\n      verbose: process.env.NODE_ENV === 'development',\n    });\n  }\n  return jobQueue;\n};\n```\n\n> **Note:** The value of `connectionString` must be a [valid Postgres connection\n  string](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING-URIS).\n  For example:\n\n```dotenv\nPG_DATAQUEUE_DATABASE=postgresql://postgres:password@localhost:5432/my_database?search_path=my_schema\n```\n\n\n## Redis\n\nTo use Redis as the backend, set `backend: 'redis'` and provide `redisConfig` instead of `databaseConfig`:\n\n```typescript title=\"@lib/queue.ts\"\nimport { initJobQueue } from '@nicnocquee/dataqueue';\nimport { type JobPayloadMap } from './types/job-payload-map';\n\nlet jobQueue: ReturnType<typeof initJobQueue<JobPayloadMap>> | null = null;\n\nexport const getJobQueue = () => {\n  if (!jobQueue) {jobQueue = initJobQueue<JobPayloadMap>({\n      backend: 'redis',\n      redisConfig: {\n        url: process.env.REDIS_URL, // e.g. redis://localhost:6379\n      },\n      verbose: process.env.NODE_ENV === 'development',\n    });\n  }\n  return jobQueue;\n};\n```\n\nYou can also connect using individual connection options instead of a URL:\n\n```typescript title=\"@lib/queue.ts\"\njobQueue = initJobQueue<JobPayloadMap>({\n  backend: 'redis',\n  redisConfig: {\n    host: 'localhost',\n    port: 6379,\n    password: process.env.REDIS_PASSWORD,\n    db: 0,\n    keyPrefix: 'myapp:', // Optional, defaults to 'dq:'\n  },\n  verbose: process.env.NODE_ENV === 'development',\n});\n```\n\n> **Note:** The `keyPrefix` option lets you namespace all Redis keys. This is useful when\n  sharing a Redis instance between multiple applications or multiple queues. The\n  default prefix is `dq:`.\n\n---\n\n## Bring Your Own Pool / Client\n\nInstead of providing connection configuration, you can pass an existing connection instance. This is useful when your application already manages its own connection pool and you want to share it with dataqueue.\n\n### PostgreSQL — External Pool\n\n```typescript title=\"@lib/queue.ts\"\nimport { Pool } from 'pg';\nimport { initJobQueue } from '@nicnocquee/dataqueue';\nimport { type JobPayloadMap } from './types/job-payload-map';\n\nconst pool = new Pool({ connectionString: process.env.DATABASE_URL });\n\nconst jobQueue = initJobQueue<JobPayloadMap>({\n  pool,verbose: process.env.NODE_ENV === 'development',\n});\n```\n\n### Redis — External Client\n\n```typescript title=\"@lib/queue.ts\"\nimport IORedis from 'ioredis';\nimport { initJobQueue } from '@nicnocquee/dataqueue';\nimport { type JobPayloadMap } from './types/job-payload-map';\n\nconst redis = new IORedis(process.env.REDIS_URL);\n\nconst jobQueue = initJobQueue<JobPayloadMap>({\n  backend: 'redis',\n  client: redis,keyPrefix: 'myapp:',\n  verbose: process.env.NODE_ENV === 'development',\n});\n```\n\n> **Note:** **Connection ownership:** When you provide your own `pool` or `client`, the\n  library will **not** close it on shutdown. You are responsible for calling\n  `pool.end()` or `client.quit()` when your application exits.\n\n---\n\n## Using the Queue\n\nOnce initialized, you use the queue instance identically regardless of backend. The API is the same for both PostgreSQL and Redis.\n\n```typescript title=\"@/app/actions/send-email.ts\"\nimport { getJobQueue } from '@/lib/queue';\n\nconst sendEmail = async () => {const jobQueue = getJobQueue();\n  await jobQueue.addJob({\n    jobType: 'send_email',\n    payload: {\n      to: 'test@example.com',\n      subject: 'Hello',\n      body: 'Hello, world!',\n    },\n  });\n};\n```\n\n---\n\n## SSL Configuration (PostgreSQL)\n\nMost managed Postgres providers (like DigitalOcean, Supabase, etc.) require SSL connections and use their own CA certificate (.crt file) to sign the server's certificate. To securely verify the server's identity, you must configure your client to trust this CA certificate.\n\nYou can configure SSL for your database connection in several ways, depending on your environment and security requirements.\n\n### Using PEM Strings from Environment Variables\n\nThis is ideal for serverless environments where you cannot mount files. Store your CA certificate, and optionally client certificate and key, as environment variables then pass them to the `ssl` property of the `databaseConfig` object.\n\n```typescript title=\"@lib/queue.ts\"\nimport { initJobQueue } from '@nicnocquee/dataqueue';\nimport { type JobPayloadMap } from './types/job-payload-map';\n\nlet jobQueue: ReturnType<typeof initJobQueue<JobPayloadMap>> | null = null;\n\nexport const getJobQueue = () => {\n  if (!jobQueue) {\n    jobQueue = initJobQueue<JobPayloadMap>({\n      databaseConfig: {\n        connectionString: process.env.PG_DATAQUEUE_DATABASE, // Set this in your environment\n        ssl: {\n          ca: process.env.PGSSLROOTCERT, // PEM string: the content of your .crt file\n          cert: process.env.PGSSLCERT, // PEM string (optional, for client authentication)\n          key: process.env.PGSSLKEY, // PEM string (optional, for client authentication)\n          rejectUnauthorized: true, // Always true for CA-signed certs\n        },\n      },\n      verbose: process.env.NODE_ENV === 'development',\n    });\n  }\n  return jobQueue;\n};\n```\n\n> **Note:** When using a custom CA certificate and `connectionString`, you must remove the\n  `sslmode` parameter from the connection string. Otherwise, the connection will\n  fail.\n\n### Using File Paths\n\nIf you have the CA certificate, client certificate, or key on disk, provide their absolute paths using the `file://` prefix. Only values starting with `file://` will be loaded from the file system; all others are treated as PEM strings.\n\n```typescript title=\"@lib/queue.ts\"\nimport { initJobQueue } from '@nicnocquee/dataqueue';\nimport { type JobPayloadMap } from './types/job-payload-map';\n\nlet jobQueue: ReturnType<typeof initJobQueue<JobPayloadMap>> | null = null;\n\nexport const getJobQueue = () => {\n  if (!jobQueue) {\n    jobQueue = initJobQueue<JobPayloadMap>({\n      databaseConfig: {\n        connectionString: process.env.PG_DATAQUEUE_DATABASE,\n        ssl: {\n          ca: 'file:///absolute/path/to/ca.crt', // Path to your provider's CA cert\n          cert: 'file:///absolute/path/to/client.crt', // optional, for client authentication\n          key: 'file:///absolute/path/to/client.key', // optional, for client authentication\n          rejectUnauthorized: true,\n        },\n      },\n      verbose: process.env.NODE_ENV === 'development',\n    });\n  }\n  return jobQueue;\n};\n```\n\n> **Note:** When using a custom CA certificate and `connectionString`, you must remove the\n  `sslmode` parameter from the connection string. Otherwise, the connection will\n  fail.\n\n### Skipping Certificate Validation\n\nFor convenience, you can skip certificate validation (not recommended for production) by setting `rejectUnauthorized` to `false` and without providing a custom CA certificate.\n\n```typescript title=\"@lib/queue.ts\"\nimport { initJobQueue } from '@nicnocquee/dataqueue';\nimport { type JobPayloadMap } from './types/job-payload-map';\n\nlet jobQueue: ReturnType<typeof initJobQueue<JobPayloadMap>> | null = null;\n\nexport const getJobQueue = () => {\n  if (!jobQueue) {\n    jobQueue = initJobQueue<JobPayloadMap>({\n      databaseConfig: {\n        connectionString: process.env.PG_DATAQUEUE_DATABASE,\n        ssl: {\n          rejectUnauthorized: false,\n        },\n      },\n      verbose: process.env.NODE_ENV === 'development',\n    });\n  }\n  return jobQueue;\n};\n```\n\n> **Note:** When using `rejectUnauthorized: false` and `connectionString`, you must remove\n  the `sslmode` parameter from the connection string. Otherwise, the connection\n  will fail.\n\n---\n\n## TLS Configuration (Redis)\n\nIf your Redis server requires TLS (common with managed services like AWS ElastiCache, Redis Cloud, etc.), provide TLS options in the `redisConfig`:\n\n```typescript title=\"@lib/queue.ts\"\njobQueue = initJobQueue<JobPayloadMap>({\n  backend: 'redis',\n  redisConfig: {\n    url: process.env.REDIS_URL,\n    tls: {\n      ca: process.env.REDIS_CA_CERT, // PEM string\n      rejectUnauthorized: true,\n    },\n  },\n});\n```"
   },
   {
     "slug": "usage/job-events",
@@ -221,6 +227,12 @@
     "description": "",
     "content": "The first thing you need to do is define your job types and their corresponding payload types. A payload is the data passed to the job handler. A job handler is a function that runs when a job is processed.\n\n### Define Job Types and Payloads\n\nJob types and their payloads are specific to your app. You can define them in any file. The important thing is that they are an object type, where the keys are the job types and the values are the payload types. In this example, `send_email`, `generate_report`, and `generate_image` are the job types, and their values are the payload types.\n\n```typescript title=\"@lib/types/job-payload-map.ts\"\n// Define the job payload map for this app.\n// This ensures that the job payload is typed correctly when adding jobs.\n// The keys are the job types, and the values are the payload types.\nexport type JobPayloadMap = {\n  send_email: {\n    to: string;\n    subject: string;\n    body: string;\n  };\n  generate_report: {\n    reportId: string;\n    userId: string;\n  };\n  generate_image: {\n    prompt: string;\n  };\n};\n```\n\n### Define Job Handlers\n\nNext, define the job handlers by exporting a `JobHandlers` object that maps job types to handler functions. If you forget to add a handler for a job type, TypeScript will show an error.\n\n```typescript title=\"@lib/job-handlers.ts\"\nimport { sendEmail } from './services/email'; // Function to send the email\nimport { generateReport } from './services/generate-report'; // Function to generate the report\nimport { JobHandlers } from '@nicnocquee/dataqueue';\n\nexport const jobHandlers: JobHandlers<JobPayloadMap> = {\n  send_email: async (payload) => {\n    const { to, subject, body } = payload;\n    await sendEmail(to, subject, body);\n  },\n  generate_report: async (payload) => {\n    const { reportId, userId } = payload;\n    await generateReport(reportId, userId);\n  },\n  generate_image: async (payload, signal) => {\n    const { prompt } = payload;\n    await generateImageAi(prompt, signal);\n  },\n};\n```\n\nIn the example above, we define three job handlers: `send_email`, `generate_report`, and `generate_image`. Each handler is a function that takes a payload, an `AbortSignal`, and a `JobContext` as arguments. The `AbortSignal` is used to abort the job if it takes too long to complete. The `JobContext` provides methods to extend the job's timeout while it's running.\n\n### Job Handler Signature\n\nA job handler receives three arguments: the job payload, an `AbortSignal`, and a `JobContext`.\n\n```typescript\n(payload: Payload, signal: AbortSignal, ctx: JobContext) => Promise<void>;\n```\n\nYou can omit arguments you don't need. For example, if you only need the payload:\n\n```typescript\nconst handler = async (payload) => {\n  // ...\n};\n```\n\n### JobContext\n\nThe third argument provides methods for timeout management and progress reporting:\n\n- `ctx.prolong(ms?)` — Proactively reset the timeout. If `ms` is provided, sets the deadline to `ms` milliseconds from now. If omitted, resets to the original `timeoutMs`.\n- `ctx.onTimeout(callback)` — Register a callback that fires when the timeout is about to hit, before the `AbortSignal` is triggered. Return a number (ms) to extend, or return nothing to let the timeout proceed.\n- `ctx.setProgress(percent)` — Report progress as a percentage (0–100). The value is persisted to the database and can be read by clients via `getJob()` or the React SDK's `useJob()` hook.\n\nSee [Job Timeout](/usage/job-timeout) for timeout examples and [Progress Tracking](/usage/progress-tracking) for progress reporting."
   },
+  {
+    "slug": "usage/job-output",
+    "title": "Job Output",
+    "description": "Store and retrieve results from job handlers",
+    "content": "Jobs can store an output value when they complete. This is useful when you need to retrieve the result of a background task — for example, a generated report URL, a processed image path, or computation results.\n\n## Storing Output\n\nThere are two ways to store output from a handler:\n\n### 1. Return a value from the handler\n\nThe simplest approach — return any JSON-serializable value from your handler function:\n\n```typescript title=\"@lib/job-handlers.ts\"\nimport { JobHandlers } from '@nicnocquee/dataqueue';\n\nexport const jobHandlers: JobHandlers<JobPayloadMap> = {\n  generate_report: async (payload, signal, ctx) => {\n    const url = await generateReport(payload.reportId);return { url, generatedAt: new Date().toISOString() };},\n};\n```\n\n### 2. Use `ctx.setOutput(data)`\n\nFor more control, call `ctx.setOutput()` explicitly. This is useful when you want to store intermediate results during execution:\n\n```typescript title=\"@lib/job-handlers.ts\"\nexport const jobHandlers: JobHandlers<JobPayloadMap> = {\n  process_images: async (payload, signal, ctx) => {\n    const results: string[] = [];\n\n    for (const image of payload.images) {\n      const url = await processImage(image);\n      results.push(url);\n\n      await ctx.setProgress(\n        Math.round((results.length / payload.images.length) * 100),\n      );\n      await ctx.setOutput({ processedUrls: results });}\n  },\n};\n```\n\n### Precedence\n\nIf both `ctx.setOutput()` is called **and** the handler returns a value, the `ctx.setOutput()` value takes precedence. The handler's return value is ignored in that case.\n\n### Rules\n\n- **JSON-serializable**: The output value must be JSON-serializable (objects, arrays, strings, numbers, booleans, null).\n- **Last write wins**: Calling `ctx.setOutput()` multiple times overwrites the previous value.\n- **Best-effort persistence**: Like `setProgress`, output writes to the database are best-effort — errors do not kill the handler.\n\n## Reading Output\n\nOutput is stored in the `output` field of the [JobRecord](/api/job-record):\n\n```typescript\nconst job = await jobQueue.getJob(jobId);\nconsole.log(job?.output); // null | any JSON value\n```\n\n- Before the handler stores output, the value is `null`.\n- After the job completes, the output is preserved and can be read at any time.\n- Handlers that return `undefined` (or `void`) do not store output — the field remains `null`.\n\n## Tracking Output in React\n\nIf you're using the [React SDK](/usage/react-sdk), the `useJob` hook exposes `output` directly:\n\n```tsx\nimport { useJob } from '@nicnocquee/dataqueue-react';\n\nfunction JobResult({ jobId }: { jobId: number }) {\n  const { status, output, progress } = useJob(jobId, {\n    fetcher: (id) =>\n      fetch(`/api/jobs/${id}`)\n        .then((r) => r.json())\n        .then((d) => d.job),\n  });\n\n  if (status === 'completed' && output) {\n    return <a href={(output as any).url}>Download Report</a>;\n  }\n\n  return (\n    <div>\n      <p>Status: {status}</p>\n      <progress value={progress ?? 0} max={100} />\n    </div>\n  );\n}\n```\n\n## Listening for Output Events\n\nYou can subscribe to the `job:output` event to be notified whenever a handler calls `ctx.setOutput()`:\n\n```typescript\njobQueue.on('job:output', ({ jobId, output }) => {\n  console.log(`Job ${jobId} stored output:`, output);\n});\n```\n\n## Database Migration\n\n> **Note:** If you're using the **PostgreSQL** backend, make sure to run the latest\n  migrations to add the `output` column. See [Database\n  Migration](/usage/database-migration).\n\nThe Redis backend requires no migration — the `output` field is stored automatically as part of the job hash."
+  },
   {
     "slug": "usage/job-timeout",
     "title": "Job Timeout",
@@ -231,25 +243,25 @@
     "slug": "usage/long-running-server",
     "title": "Long-Running Server",
     "description": "",
-    "content": "The [Process Jobs](/usage/process-jobs) page covers processing jobs in a serverless environment using cron-triggered API routes. If you're running a long-lived server (Express, Fastify, plain Node.js, etc.), you can instead run the processor continuously in the background and handle lifecycle management yourself.\n\n## Starting the Processor in the Background\n\nUse `startInBackground()` to run the processor as a continuous polling loop. It will check for new jobs every `pollInterval` milliseconds (default: 5 seconds) and process them automatically.\n\n```typescript\nimport { initJobQueue } from '@nicnocquee/dataqueue';\nimport { jobHandlers } from './job-handlers';\n\nconst jobQueue = initJobQueue({\n  databaseConfig: {\n    connectionString: process.env.PG_DATAQUEUE_DATABASE,\n  },\n});\n\nconst processor = jobQueue.createProcessor(jobHandlers, {\n  workerId: `server-${process.pid}`,\n  batchSize: 10,\n  concurrency: 3,\n  pollInterval: 5000, // check for new jobs every 5 seconds\n  onError: (error) => {\n    // Called when an unexpected error occurs during batch processing.\n    // Use this to send errors to your monitoring service.\n    console.error('Processor error:', error);\n  },\n});\n\nprocessor.startInBackground();\n```\n\nWhen a full batch is returned (i.e., the number of processed jobs equals `batchSize`), the processor immediately fetches the next batch when the current batch is finished, so it can drain a large backlog quickly. Once a batch returns fewer jobs than `batchSize`, it waits `pollInterval` before fetching the next batch.\n\n### Configuration Tips\n\n- **`pollInterval`** -- Lower values (e.g., `1000`) reduce latency for new jobs but increase database load. Higher values (e.g., `10000`) are gentler on the database but introduce more delay. 5 seconds is a good default.\n- **`concurrency`** -- Keep this proportional to your server's resources. If jobs call external APIs with rate limits, keep it low.\n- **`batchSize`** -- Larger batches reduce polling overhead but hold a database lock longer during claim. 10-20 is typical.\n- **`onError`** -- Always set this in production. Without it, errors default to `console.error` which is easy to miss.\n\n## Graceful Shutdown\n\nWhen your server receives a termination signal (e.g., `SIGTERM` from a container orchestrator), you should stop the processor and wait for in-flight jobs to finish before exiting. Use `stopAndDrain()` for this.\n\n```typescript\nasync function shutdown() {\n  console.log('Shutting down...');\n\n  // Stop polling and wait for the current batch to finish (up to 30 seconds)\n  await processor.stopAndDrain(30000);\n\n  // Close the database connection pool\n  // PostgreSQL:\n  jobQueue.getPool().end();\n  // Redis:\n  // jobQueue.getRedisClient().quit();\n\n  console.log('Shutdown complete');\n  process.exit(0);\n}\n\nprocess.on('SIGTERM', shutdown);\nprocess.on('SIGINT', shutdown);\n```\n\n`stopAndDrain()` accepts an optional timeout in milliseconds (default: 30000). If the current batch does not finish within that time, the promise resolves anyway so your process is not stuck indefinitely.\n\n> **Note:** Use `stopAndDrain()` instead of `stop()` for graceful shutdown. `stop()` halts\n  the polling loop immediately without waiting for in-flight jobs, which can\n  leave jobs stuck in the `processing` state until they are reclaimed.\n\n## Scheduling Maintenance Tasks\n\nIn a serverless setup, you use cron-triggered API routes for [cleanup](/usage/cleanup-jobs) and [reclaim](/usage/reclaim-jobs). In a long-running server, you can use `setInterval` instead.\n\n```typescript\n// Reclaim stuck jobs every 10 minutes\nconst reclaimInterval = setInterval(\n  async () => {\n    try {\n      const reclaimed = await jobQueue.reclaimStuckJobs(10);\n      if (reclaimed > 0) console.log(`Reclaimed ${reclaimed} stuck jobs`);\n    } catch (error) {\n      console.error('Reclaim error:', error);\n    }\n  },\n  10 * 60 * 1000,\n);\n\n// Clean up completed jobs older than 30 days, once per day\nconst cleanupInterval = setInterval(\n  async () => {\n    try {\n      const deleted = await jobQueue.cleanupOldJobs(30);\n      if (deleted > 0) console.log(`Cleaned up ${deleted} old jobs`);\n\n      const deletedEvents = await jobQueue.cleanupOldJobEvents(30);\n      if (deletedEvents > 0)\n        console.log(`Cleaned up ${deletedEvents} old job events`);\n    } catch (error) {\n      console.error('Cleanup error:', error);\n    }\n  },\n  24 * 60 * 60 * 1000,\n);\n```\n\nMake sure to clear these intervals during shutdown:\n\n```typescript\nasync function shutdown() {\n  clearInterval(reclaimInterval);\n  clearInterval(cleanupInterval);\n\n  await processor.stopAndDrain(30000);\n\n  jobQueue.getPool().end();\n  process.exit(0);\n}\n```\n\n> **Note:** If you use the [wait/token](/usage/wait) feature (PostgreSQL only), also call\n  `expireTimedOutTokens()` on an interval to expire tokens that have passed\n  their timeout.\n\n## Full Example\n\nHere is a complete Express server that ties everything together:\n\n```typescript title=\"server.ts\"\nimport express from 'express';\nimport { initJobQueue } from '@nicnocquee/dataqueue';\nimport { jobHandlers } from './job-handlers';\n\n// --- Initialize the queue ---\nconst jobQueue = initJobQueue({\n  databaseConfig: {\n    connectionString: process.env.PG_DATAQUEUE_DATABASE!,\n  },\n});\n\n// --- Create and start the processor ---\nconst processor = jobQueue.createProcessor(jobHandlers, {\n  workerId: `server-${process.pid}`,\n  batchSize: 10,\n  concurrency: 3,\n  pollInterval: 5000,\n  onError: (error) => {\n    console.error('Processor error:', error);\n  },\n});\n\nprocessor.startInBackground();\n\n// --- Schedule maintenance ---\nconst reclaimInterval = setInterval(\n  async () => {\n    try {\n      await jobQueue.reclaimStuckJobs(10);\n    } catch (e) {\n      console.error('Reclaim error:', e);\n    }\n  },\n  10 * 60 * 1000,\n);\n\nconst cleanupInterval = setInterval(\n  async () => {\n    try {\n      await jobQueue.cleanupOldJobs(30);\n      await jobQueue.cleanupOldJobEvents(30);\n    } catch (e) {\n      console.error('Cleanup error:', e);\n    }\n  },\n  24 * 60 * 60 * 1000,\n);\n\n// --- Express app ---\nconst app = express();\napp.use(express.json());\n\napp.post('/jobs', async (req, res) => {\n  const { jobType, payload } = req.body;\n  const jobId = await jobQueue.addJob({ jobType, payload });\n  res.json({ jobId });\n});\n\napp.get('/jobs/:id', async (req, res) => {\n  const job = await jobQueue.getJob(Number(req.params.id));\n  if (!job) return res.status(404).json({ error: 'Not found' });\n  res.json(job);\n});\n\nconst server = app.listen(3000, () => {\n  console.log('Server running on port 3000');\n});\n\n// --- Graceful shutdown ---\nasync function shutdown() {\n  console.log('Shutting down gracefully...');\n\n  // Stop accepting new HTTP connections\n  server.close();\n\n  // Clear maintenance intervals\n  clearInterval(reclaimInterval);\n  clearInterval(cleanupInterval);\n\n  // Wait for in-flight jobs to finish\n  await processor.stopAndDrain(30000);\n\n  // Close the database pool\n  jobQueue.getPool().end();\n\n  console.log('Shutdown complete');\n  process.exit(0);\n}\n\nprocess.on('SIGTERM', shutdown);\nprocess.on('SIGINT', shutdown);\n```"
+    "content": "The [Process Jobs](/usage/process-jobs) page covers processing jobs in a serverless environment using cron-triggered API routes. If you're running a long-lived server (Express, Fastify, plain Node.js, etc.), you can instead run the processor continuously in the background and handle lifecycle management yourself.\n\n## Starting the Processor in the Background\n\nUse `startInBackground()` to run the processor as a continuous polling loop. It will check for new jobs every `pollInterval` milliseconds (default: 5 seconds) and process them automatically.\n\n```typescript\nimport { initJobQueue } from '@nicnocquee/dataqueue';\nimport { jobHandlers } from './job-handlers';\n\nconst jobQueue = initJobQueue({\n  databaseConfig: {\n    connectionString: process.env.PG_DATAQUEUE_DATABASE,\n  },\n});\n\nconst processor = jobQueue.createProcessor(jobHandlers, {\n  workerId: `server-${process.pid}`,\n  batchSize: 10,\n  concurrency: 3,\n  pollInterval: 5000, // check for new jobs every 5 seconds\n  onError: (error) => {\n    // Called when an unexpected error occurs during batch processing.\n    // Use this to send errors to your monitoring service.\n    console.error('Processor error:', error);\n  },\n});\n\nprocessor.startInBackground();\n```\n\nWhen a full batch is returned (i.e., the number of processed jobs equals `batchSize`), the processor immediately fetches the next batch when the current batch is finished, so it can drain a large backlog quickly. Once a batch returns fewer jobs than `batchSize`, it waits `pollInterval` before fetching the next batch.\n\n### Configuration Tips\n\n- **`pollInterval`** -- Lower values (e.g., `1000`) reduce latency for new jobs but increase database load. Higher values (e.g., `10000`) are gentler on the database but introduce more delay. 5 seconds is a good default.\n- **`concurrency`** -- Keep this proportional to your server's resources. If jobs call external APIs with rate limits, keep it low.\n- **`batchSize`** -- Larger batches reduce polling overhead but hold a database lock longer during claim. 10-20 is typical.\n- **`onError`** -- Always set this in production. Without it, errors default to `console.error` which is easy to miss.\n\n## Graceful Shutdown\n\nWhen your server receives a termination signal (e.g., `SIGTERM` from a container orchestrator), you should stop the processor and wait for in-flight jobs to finish before exiting. Use `stopAndDrain()` for this.\n\n```typescript\nasync function shutdown() {\n  console.log('Shutting down...');\n\n  // Stop polling and wait for the current batch to finish (up to 30 seconds)\n  await processor.stopAndDrain(30000);\n\n  // Close the database connection pool\n  // PostgreSQL:\n  jobQueue.getPool().end();\n  // Redis:\n  // jobQueue.getRedisClient().quit();\n\n  console.log('Shutdown complete');\n  process.exit(0);\n}\n\nprocess.on('SIGTERM', shutdown);\nprocess.on('SIGINT', shutdown);\n```\n\n`stopAndDrain()` accepts an optional timeout in milliseconds (default: 30000). If the current batch does not finish within that time, the promise resolves anyway so your process is not stuck indefinitely.\n\n> **Note:** Use `stopAndDrain()` instead of `stop()` for graceful shutdown. `stop()` halts\n  the polling loop immediately without waiting for in-flight jobs, which can\n  leave jobs stuck in the `processing` state until they are reclaimed.\n\n## Background Supervisor\n\nIn production, you need periodic maintenance: reclaiming stuck jobs, cleaning up old completed jobs/events, and expiring timed-out waitpoint tokens. The **supervisor** automates all of this.\n\n```typescript\nconst supervisor = jobQueue.createSupervisor({\n  intervalMs: 60_000, // run maintenance every 60 seconds\n  stuckJobsTimeoutMinutes: 10, // reclaim jobs stuck > 10 minutes\n  cleanupJobsDaysToKeep: 30, // delete completed jobs older than 30 days\n  cleanupEventsDaysToKeep: 30, // delete job events older than 30 days\n  onError: (error) => {\n    console.error('Supervisor error:', error);\n  },\n});\n\nsupervisor.startInBackground();\n```\n\nThe supervisor runs independently from the processor. Each maintenance task is isolated -- if one fails, the others still run. All operations are idempotent, so it's safe to run multiple supervisor instances across a cluster.\n\n### Supervisor Options\n\n| Option                    | Default         | Description                                                  |\n| ------------------------- | --------------- | ------------------------------------------------------------ |\n| `intervalMs`              | `60000`         | How often the maintenance loop runs (ms)                     |\n| `stuckJobsTimeoutMinutes` | `10`            | Reclaim jobs stuck in `processing` longer than this          |\n| `cleanupJobsDaysToKeep`   | `30`            | Delete completed jobs older than this (days). `0` to disable |\n| `cleanupEventsDaysToKeep` | `30`            | Delete job events older than this (days). `0` to disable     |\n| `cleanupBatchSize`        | `1000`          | Batch size for cleanup deletions                             |\n| `reclaimStuckJobs`        | `true`          | Enable/disable stuck job reclaiming                          |\n| `expireTimedOutTokens`    | `true`          | Enable/disable waitpoint token expiry                        |\n| `onError`                 | `console.error` | Called when a maintenance task throws                        |\n| `verbose`                 | `false`         | Enable verbose logging                                       |\n\n### Graceful Shutdown with the Supervisor\n\nWhen shutting down, drain both the processor and the supervisor:\n\n```typescript\nasync function shutdown() {\n  console.log('Shutting down...');\n\n  await Promise.all([\n    processor.stopAndDrain(30000),\n    supervisor.stopAndDrain(30000),\n  ]);\n\n  jobQueue.getPool().end();\n  console.log('Shutdown complete');\n  process.exit(0);\n}\n\nprocess.on('SIGTERM', shutdown);\nprocess.on('SIGINT', shutdown);\n```\n\n### One-Shot Mode (Serverless)\n\nIf you prefer to run maintenance from a cron-triggered API route or a serverless function, use `start()` instead of `startInBackground()`. It runs all tasks once and returns the results:\n\n```typescript\nconst supervisor = jobQueue.createSupervisor();\nconst result = await supervisor.start();\n// { reclaimedJobs: 3, cleanedUpJobs: 120, cleanedUpEvents: 45, expiredTokens: 0 }\n```\n\n## Manual Maintenance\n\n> **Note:** The supervisor above is the recommended approach. Manual maintenance is still\n  supported if you need fine-grained control over individual tasks.\n\nIn a serverless setup, you use cron-triggered API routes for [cleanup](/usage/cleanup-jobs) and [reclaim](/usage/reclaim-jobs). In a long-running server, you can use `setInterval` instead.\n\n```typescript\n// Reclaim stuck jobs every 10 minutes\nconst reclaimInterval = setInterval(\n  async () => {\n    try {\n      const reclaimed = await jobQueue.reclaimStuckJobs(10);\n      if (reclaimed > 0) console.log(`Reclaimed ${reclaimed} stuck jobs`);\n    } catch (error) {\n      console.error('Reclaim error:', error);\n    }\n  },\n  10 * 60 * 1000,\n);\n\n// Clean up completed jobs older than 30 days, once per day\nconst cleanupInterval = setInterval(\n  async () => {\n    try {\n      const deleted = await jobQueue.cleanupOldJobs(30);\n      if (deleted > 0) console.log(`Cleaned up ${deleted} old jobs`);\n\n      const deletedEvents = await jobQueue.cleanupOldJobEvents(30);\n      if (deletedEvents > 0)\n        console.log(`Cleaned up ${deletedEvents} old job events`);\n    } catch (error) {\n      console.error('Cleanup error:', error);\n    }\n  },\n  24 * 60 * 60 * 1000,\n);\n```\n\nMake sure to clear these intervals during shutdown:\n\n```typescript\nasync function shutdown() {\n  clearInterval(reclaimInterval);\n  clearInterval(cleanupInterval);\n\n  await processor.stopAndDrain(30000);\n\n  jobQueue.getPool().end();\n  process.exit(0);\n}\n```\n\n## Full Example\n\nHere is a complete Express server that ties everything together:\n\n```typescript title=\"server.ts\"\nimport express from 'express';\nimport { initJobQueue } from '@nicnocquee/dataqueue';\nimport { jobHandlers } from './job-handlers';\n\n// --- Initialize the queue ---\nconst jobQueue = initJobQueue({\n  databaseConfig: {\n    connectionString: process.env.PG_DATAQUEUE_DATABASE!,\n  },\n});\n\n// --- Create and start the processor ---\nconst processor = jobQueue.createProcessor(jobHandlers, {\n  workerId: `server-${process.pid}`,\n  batchSize: 10,\n  concurrency: 3,\n  pollInterval: 5000,\n  onError: (error) => {\n    console.error('Processor error:', error);\n  },\n});\n\nprocessor.startInBackground();\n\n// --- Start the background supervisor ---\nconst supervisor = jobQueue.createSupervisor({\n  intervalMs: 60_000,\n  stuckJobsTimeoutMinutes: 10,\n  cleanupJobsDaysToKeep: 30,\n  cleanupEventsDaysToKeep: 30,\n  onError: (error) => {\n    console.error('Supervisor error:', error);\n  },\n});\n\nsupervisor.startInBackground();\n\n// --- Express app ---\nconst app = express();\napp.use(express.json());\n\napp.post('/jobs', async (req, res) => {\n  const { jobType, payload } = req.body;\n  const jobId = await jobQueue.addJob({ jobType, payload });\n  res.json({ jobId });\n});\n\napp.get('/jobs/:id', async (req, res) => {\n  const job = await jobQueue.getJob(Number(req.params.id));\n  if (!job) return res.status(404).json({ error: 'Not found' });\n  res.json(job);\n});\n\nconst server = app.listen(3000, () => {\n  console.log('Server running on port 3000');\n});\n\n// --- Graceful shutdown ---\nasync function shutdown() {\n  console.log('Shutting down gracefully...');\n\n  // Stop accepting new HTTP connections\n  server.close();\n\n  // Wait for in-flight work to finish\n  await Promise.all([\n    processor.stopAndDrain(30000),\n    supervisor.stopAndDrain(30000),\n  ]);\n\n  // Close the database pool\n  jobQueue.getPool().end();\n\n  console.log('Shutdown complete');\n  process.exit(0);\n}\n\nprocess.on('SIGTERM', shutdown);\nprocess.on('SIGINT', shutdown);\n```"
   },
   {
     "slug": "usage/process-jobs",
     "title": "Process Jobs",
     "description": "",
-    "content": "So far, we haven't actually performed any jobs—we've only added them to the queue. Now, let's process those jobs.\n\n> **Note:** This page covers processing jobs in a **serverless environment** using\n  cron-triggered API routes. If you're running a long-lived server (Express,\n  Fastify, etc.), see [Long-Running Server](/usage/long-running-server).\n\nIn a serverless environment, we can't have a long-running process that constantly monitors and processes the queue.\n\nInstead, we create an API endpoint that checks the queue and processes jobs in batches. This endpoint is then triggered by a cron job. For example, you can create an API endpoint at `app/api/cron/process` to process jobs in batches:\n\n```typescript title=\"@/app/api/cron/process.ts\"\nimport { jobHandlers } from '@/lib/job-handler';\nimport { getJobQueue } from '@/lib/queue';\nimport { NextResponse } from 'next/server';\n\nexport async function GET(request: Request) {\n  // Secure the cron route: https://vercel.com/docs/cron-jobs/manage-cron-jobs#securing-cron-jobs\n  const authHeader = request.headers.get('authorization');\n  if (authHeader !== `Bearer ${process.env.CRON_SECRET}`) {\n    return NextResponse.json({ message: 'Unauthorized' }, { status: 401 });\n  }\n\n  try {const jobQueue = getJobQueue();\n\n    // Control how many jobs are processed in parallel per batch using the `concurrency` option.\n    // For example, to process up to 3 jobs in parallel per batch:\n    const processor = jobQueue.createProcessor(jobHandlers, {\n      workerId: `cron-${Date.now()}`,\n      batchSize: 10, // up to 10 jobs per batch\n      concurrency: 3, // up to 3 jobs processed in parallel\n      verbose: true,\n    });\n\n    const processed = await processor.start();\n\n    return NextResponse.json({\n      message: 'Job processing completed',\n      processed,\n    });\n  } catch (error) {\n    console.error('Error processing jobs:', error);\n    return NextResponse.json(\n      { message: 'Failed to process jobs' },\n      { status: 500 },\n    );\n  }\n}\n```\n\nIn the example above, we use the `createProcessor` method to create a processor. When you call the processor's `start` function, it processes jobs in the queue up to the `batchSize` limit.\n\n### Batch Size\n\nServerless platforms like Vercel limit how long a function can run. If you set `batchSize` too high, the function might run too long and get killed. Choose a `batchSize` that fits your use case.\n\nYou can also process only certain job types by setting the `jobType` option. If a job type is more resource-intensive, use a lower `batchSize` for that type.\n\nFor example, you can define two endpoints: one for low-resource jobs and another for high-resource jobs, each with different `batchSize` and `concurrency` values.\n\n### Concurrency\n\nSome jobs are resource-intensive, like image processing, LLM calls, or calling a rate-limited external service. In these cases, set the `concurrency` option to control how many jobs run in parallel per batch.\n\nThe default is `3`. Set it to `1` to process jobs one at a time. Use a lower value to avoid exhausting resources in constrained environments.\n\n### Triggering the Processor via Cron\n\nDefining an endpoint isn't enough—you need to trigger it regularly. For example, use Vercel cron to trigger the endpoint every minute by adding this to your `vercel.json`:\n\n```json title=\"vercel.json\"\n{\n  \"$schema\": \"https://openapi.vercel.sh/vercel.json\",\n  \"crons\": [\n    {\n      \"path\": \"/api/cron/process\",\n      \"schedule\": \"* * * * *\"\n    }\n  ]\n}\n```\n\nFor Vercel cron, set the `CRON_SECRET` environment variable, as it's sent in the `authorization` header. If you use a different cron service, set the `authorization` header to the value of `CRON_SECRET`:\n\n```\nAuthorization: Bearer <VALUE_OF_CRON_SECRET>\n```\n\nDuring development, you can create a small script to run the cron job continuously in the background. For example, you can create a `cron.sh` file like [this one](https://github.com/nicnocquee/dataqueue/blob/main/apps/demo/cron.sh), then add it to your `package.json` scripts:\n\n```json title=\"package.json\"\n{\n  \"scripts\": {\n    \"cron\": \"bash cron.sh\"\n  }\n}\n```\n\nThen, you can run the cron job by running `pnpm cron` from the apps/demo directory."
+    "content": "So far, we haven't actually performed any jobs—we've only added them to the queue. Now, let's process those jobs.\n\n> **Note:** This page covers processing jobs in a **serverless environment** using\n  cron-triggered API routes. If you're running a long-lived server (Express,\n  Fastify, etc.), see [Long-Running Server](/usage/long-running-server).\n\nIn a serverless environment, we can't have a long-running process that constantly monitors and processes the queue.\n\nInstead, we create an API endpoint that checks the queue and processes jobs in batches. This endpoint is then triggered by a cron job. For example, you can create an API endpoint at `app/api/cron/process` to process jobs in batches:\n\n```typescript title=\"@/app/api/cron/process.ts\"\nimport { jobHandlers } from '@/lib/job-handler';\nimport { getJobQueue } from '@/lib/queue';\nimport { NextResponse } from 'next/server';\n\nexport async function GET(request: Request) {\n  // Secure the cron route: https://vercel.com/docs/cron-jobs/manage-cron-jobs#securing-cron-jobs\n  const authHeader = request.headers.get('authorization');\n  if (authHeader !== `Bearer ${process.env.CRON_SECRET}`) {\n    return NextResponse.json({ message: 'Unauthorized' }, { status: 401 });\n  }\n\n  try {const jobQueue = getJobQueue();\n\n    // Control how many jobs are processed in parallel per batch using the `concurrency` option.\n    // For example, to process up to 3 jobs in parallel per batch:\n    const processor = jobQueue.createProcessor(jobHandlers, {\n      workerId: `cron-${Date.now()}`,\n      batchSize: 10, // up to 10 jobs per batch\n      concurrency: 3, // up to 3 jobs processed in parallel\n      groupConcurrency: 2, // optional: max 2 concurrent jobs per group.id globally\n      verbose: true,\n    });\n\n    const processed = await processor.start();\n\n    return NextResponse.json({\n      message: 'Job processing completed',\n      processed,\n    });\n  } catch (error) {\n    console.error('Error processing jobs:', error);\n    return NextResponse.json(\n      { message: 'Failed to process jobs' },\n      { status: 500 },\n    );\n  }\n}\n```\n\nIn the example above, we use the `createProcessor` method to create a processor. When you call the processor's `start` function, it processes jobs in the queue up to the `batchSize` limit.\n\n### Batch Size\n\nServerless platforms like Vercel limit how long a function can run. If you set `batchSize` too high, the function might run too long and get killed. Choose a `batchSize` that fits your use case.\n\nYou can also process only certain job types by setting the `jobType` option. If a job type is more resource-intensive, use a lower `batchSize` for that type.\n\nFor example, you can define two endpoints: one for low-resource jobs and another for high-resource jobs, each with different `batchSize` and `concurrency` values.\n\n### Concurrency\n\nSome jobs are resource-intensive, like image processing, LLM calls, or calling a rate-limited external service. In these cases, set the `concurrency` option to control how many jobs run in parallel per batch.\n\nThe default is `3`. Set it to `1` to process jobs one at a time. Use a lower value to avoid exhausting resources in constrained environments.\n\n### Group-Based Concurrency\n\nWhen jobs should be limited per tenant/customer/account across all workers, set `groupConcurrency` on the processor and provide `group.id` when enqueuing jobs:\n\n```typescript\nawait jobQueue.addJob({\n  jobType: 'send_email',\n  payload: { to: 'user@example.com' },\n  group: { id: 'tenant_123' },\n});\n\nconst processor = jobQueue.createProcessor(jobHandlers, {\n  batchSize: 20,\n  concurrency: 10,\n  groupConcurrency: 2,\n});\n```\n\nWith this configuration, at most 2 jobs from the same `group.id` run at once globally, while ungrouped jobs continue to flow normally.\n\n### Triggering the Processor via Cron\n\nDefining an endpoint isn't enough—you need to trigger it regularly. For example, use Vercel cron to trigger the endpoint every minute by adding this to your `vercel.json`:\n\n```json title=\"vercel.json\"\n{\n  \"$schema\": \"https://openapi.vercel.sh/vercel.json\",\n  \"crons\": [\n    {\n      \"path\": \"/api/cron/process\",\n      \"schedule\": \"* * * * *\"\n    }\n  ]\n}\n```\n\nFor Vercel cron, set the `CRON_SECRET` environment variable, as it's sent in the `authorization` header. If you use a different cron service, set the `authorization` header to the value of `CRON_SECRET`:\n\n```\nAuthorization: Bearer <VALUE_OF_CRON_SECRET>\n```\n\nDuring development, you can create a small script to run the cron job continuously in the background. For example, you can create a `cron.sh` file like [this one](https://github.com/nicnocquee/dataqueue/blob/main/apps/demo/cron.sh), then add it to your `package.json` scripts:\n\n```json title=\"package.json\"\n{\n  \"scripts\": {\n    \"cron\": \"bash cron.sh\"\n  }\n}\n```\n\nThen, you can run the cron job by running `pnpm cron` from the apps/demo directory."
   },
   {
     "slug": "usage/progress-tracking",
     "title": "Progress Tracking",
     "description": "Report and track job progress from handlers",
-    "content": "Jobs can report their progress as a percentage (0–100) while they run. This is useful for long-running tasks like file processing, data imports, or image generation where you want to show a progress bar or percentage to the user.\n\n## Reporting Progress from a Handler\n\nUse `ctx.setProgress(percent)` inside your job handler to report progress:\n\n```typescript title=\"@lib/job-handlers.ts\"\nimport { JobHandlers } from '@nicnocquee/dataqueue';\n\nexport const jobHandlers: JobHandlers<JobPayloadMap> = {\n  generate_report: async (payload, signal, ctx) => {\n    const chunks = await loadData(payload.reportId);\n\n    for (let i = 0; i < chunks.length; i++) {\n      if (signal.aborted) return;\n\n      await processChunk(chunks[i]);\n\n      // Report progress (0-100)\n      await ctx.setProgress(Math.round(((i + 1) / chunks.length) * 100));\n    }\n  },\n};\n```\n\n### setProgress Rules\n\n- **Range**: The value must be between 0 and 100 (inclusive). Values outside this range throw an error.\n- **Rounding**: Fractional values are rounded to the nearest integer (`33.7` becomes `34`).\n- **Best-effort persistence**: Progress is written to the database but errors during the write do not kill the handler — processing continues.\n\n## Reading Progress\n\nProgress is stored in the `progress` field of the [JobRecord](/api/job-record):\n\n```typescript\nconst job = await jobQueue.getJob(jobId);\nconsole.log(job?.progress); // null | 0–100\n```\n\n- Before the handler calls `setProgress`, the value is `null`.\n- After the job completes, the last progress value is preserved (typically `100`).\n\n## Tracking Progress in React\n\nIf you're using the [React SDK](/usage/react-sdk), the `useJob` hook exposes `progress` directly:\n\n```tsx\nimport { useJob } from '@nicnocquee/dataqueue-react';\n\nfunction JobProgress({ jobId }: { jobId: number }) {\n  const { status, progress } = useJob(jobId, {\n    fetcher: (id) =>\n      fetch(`/api/jobs/${id}`)\n        .then((r) => r.json())\n        .then((d) => d.job),\n  });\n\n  return (\n    <div>\n      <p>Status: {status}</p>\n      <progress value={progress ?? 0} max={100} />\n      <span>{progress ?? 0}%</span>\n    </div>\n  );\n}\n```\n\n## Database Migration\n\n> **Note:** If you're using the **PostgreSQL** backend, make sure to run the latest\n  migrations to add the `progress` column. See [Database\n  Migration](/usage/database-migration).\n\nThe Redis backend requires no migration — the `progress` field is stored automatically as part of the job hash."
+    "content": "Jobs can report their progress as a percentage (0–100) while they run. This is useful for long-running tasks like file processing, data imports, or image generation where you want to show a progress bar or percentage to the user.\n\n## Reporting Progress from a Handler\n\nUse `ctx.setProgress(percent)` inside your job handler to report progress:\n\n```typescript title=\"@lib/job-handlers.ts\"\nimport { JobHandlers } from '@nicnocquee/dataqueue';\n\nexport const jobHandlers: JobHandlers<JobPayloadMap> = {\n  generate_report: async (payload, signal, ctx) => {\n    const chunks = await loadData(payload.reportId);\n\n    for (let i = 0; i < chunks.length; i++) {\n      if (signal.aborted) return;\n\n      await processChunk(chunks[i]);\n\n      // Report progress (0-100)\n      await ctx.setProgress(Math.round(((i + 1) / chunks.length) * 100));\n    }\n  },\n};\n```\n\n### setProgress Rules\n\n- **Range**: The value must be between 0 and 100 (inclusive). Values outside this range throw an error.\n- **Rounding**: Fractional values are rounded to the nearest integer (`33.7` becomes `34`).\n- **Best-effort persistence**: Progress is written to the database but errors during the write do not kill the handler — processing continues.\n\n## Reading Progress\n\nProgress is stored in the `progress` field of the [JobRecord](/api/job-record):\n\n```typescript\nconst job = await jobQueue.getJob(jobId);\nconsole.log(job?.progress); // null | 0–100\n```\n\n- Before the handler calls `setProgress`, the value is `null`.\n- After the job completes, the last progress value is preserved (typically `100`).\n\n## Tracking Progress in React\n\nIf you're using the [React SDK](/usage/react-sdk), the `useJob` hook exposes `progress` directly:\n\n```tsx\nimport { useJob } from '@nicnocquee/dataqueue-react';\n\nfunction JobProgress({ jobId }: { jobId: number }) {\n  const { status, progress } = useJob(jobId, {\n    fetcher: (id) =>\n      fetch(`/api/jobs/${id}`)\n        .then((r) => r.json())\n        .then((d) => d.job),\n  });\n\n  return (\n    <div>\n      <p>Status: {status}</p>\n      <progress value={progress ?? 0} max={100} />\n      <span>{progress ?? 0}%</span>\n    </div>\n  );\n}\n```\n\n## Database Migration\n\n> **Note:** If you're using the **PostgreSQL** backend, make sure to run the latest\n  migrations to add the `progress` column. See [Database\n  Migration](/usage/database-migration).\n\nThe Redis backend requires no migration — the `progress` field is stored automatically as part of the job hash.\n\n## Related\n\n- [Job Output](/usage/job-output) — Store and retrieve results from job handlers using `ctx.setOutput()` or handler return values."
   },
   {
     "slug": "usage/quick-start",
     "title": "Quick Start",
     "description": "Get started with DataQueue",
-    "content": "In this docs, we'll use a Next.js with App Router project which is deployed to Vercel as an example.\n\n## Next.js Shortcut\n\nIf you're using Next.js, you can scaffold everything — API routes, a job queue singleton, a cron script, and all dependencies — with a single command:\n\n```bash\nnpx dataqueue-cli init\n```\n\nThe command auto-detects your project structure (App Router vs Pages Router, `src/` directory vs root) and creates all the files you need. See the [`init` CLI reference](/cli/init) for full details.\n\nIf you prefer to set things up manually, follow the steps below.\n\n## PostgreSQL Backend\n\n1. [Run migrations before deploying your app](/usage/database-migration)\n2. [Define job handlers](/usage/job-handlers)\n3. [Initialize the job queue](/usage/init-queue)\n4. [Add a job](/usage/add-job)\n5. Create three API routes to [process jobs](/usage/process-jobs), [reclaim stuck jobs](/usage/reclaim-jobs), and [cleanup old jobs](/usage/cleanup-jobs)\n6. [Call those API routes periodically](/usage/process-jobs#triggering-the-processor-via-cron) via a cron service (like Vercel cron) or a small script like [this one](https://github.com/nicnocquee/dataqueue/blob/main/apps/demo/cron.sh) during development.\n\n## Redis Backend\n\n1. [Install `ioredis`](/intro/install)\n2. [Define job handlers](/usage/job-handlers)\n3. [Initialize the job queue with Redis config](/usage/init-queue#redis)\n4. [Add a job](/usage/add-job)\n5. Create three API routes to [process jobs](/usage/process-jobs), [reclaim stuck jobs](/usage/reclaim-jobs), and [cleanup old jobs](/usage/cleanup-jobs)\n6. [Call those API routes periodically](/usage/process-jobs#triggering-the-processor-via-cron) via a cron service (like Vercel cron) or a small script like [this one](https://github.com/nicnocquee/dataqueue/blob/main/apps/demo/cron.sh) during development.\n\n> **Note:** The Redis backend requires **no database migrations**. Just install `ioredis`,\n  configure the connection, and you're ready to go.\n\n## Long-Running Server\n\nIf you're running a persistent server (Express, Fastify, plain Node.js, etc.) instead of a serverless environment, the setup is slightly different:\n\n1. [Run migrations](/usage/database-migration) (PostgreSQL) or [install `ioredis`](/intro/install) (Redis)\n2. [Define job handlers](/usage/job-handlers)\n3. [Initialize the job queue](/usage/init-queue)\n4. Start the processor in the background with `startInBackground()`\n5. Schedule maintenance tasks (`reclaimStuckJobs`, `cleanupOldJobs`) on intervals\n6. Handle `SIGTERM`/`SIGINT` for graceful shutdown with `stopAndDrain()`\n\nSee [Long-Running Server](/usage/long-running-server) for a complete walkthrough and full example."
+    "content": "In this docs, we'll use a Next.js with App Router project which is deployed to Vercel as an example.\n\n## Next.js Shortcut\n\nIf you're using Next.js, you can scaffold everything — API routes, a job queue singleton, a cron script, and all dependencies — with a single command:\n\n```bash\nnpx dataqueue-cli init\n```\n\nThe command auto-detects your project structure (App Router vs Pages Router, `src/` directory vs root) and creates all the files you need. See the [`init` CLI reference](/cli/init) for full details.\n\nIf you prefer to set things up manually, follow the steps below.\n\n## PostgreSQL Backend\n\n1. [Run migrations before deploying your app](/usage/database-migration)\n2. [Define job handlers](/usage/job-handlers)\n3. [Initialize the job queue](/usage/init-queue)\n4. [Add a job](/usage/add-job)\n5. Create three API routes to [process jobs](/usage/process-jobs), [reclaim stuck jobs](/usage/reclaim-jobs), and [cleanup old jobs](/usage/cleanup-jobs)\n6. [Call those API routes periodically](/usage/process-jobs#triggering-the-processor-via-cron) via a cron service (like Vercel cron) or a small script like [this one](https://github.com/nicnocquee/dataqueue/blob/main/apps/demo/cron.sh) during development.\n\n## Redis Backend\n\n1. [Install `ioredis`](/intro/install)\n2. [Define job handlers](/usage/job-handlers)\n3. [Initialize the job queue with Redis config](/usage/init-queue#redis)\n4. [Add a job](/usage/add-job)\n5. Create three API routes to [process jobs](/usage/process-jobs), [reclaim stuck jobs](/usage/reclaim-jobs), and [cleanup old jobs](/usage/cleanup-jobs)\n6. [Call those API routes periodically](/usage/process-jobs#triggering-the-processor-via-cron) via a cron service (like Vercel cron) or a small script like [this one](https://github.com/nicnocquee/dataqueue/blob/main/apps/demo/cron.sh) during development.\n\n> **Note:** The Redis backend requires **no database migrations**. Just install `ioredis`,\n  configure the connection, and you're ready to go.\n\n## Long-Running Server\n\nIf you're running a persistent server (Express, Fastify, plain Node.js, etc.) instead of a serverless environment, the setup is slightly different:\n\n1. [Run migrations](/usage/database-migration) (PostgreSQL) or [install `ioredis`](/intro/install) (Redis)\n2. [Define job handlers](/usage/job-handlers)\n3. [Initialize the job queue](/usage/init-queue)\n4. Start the processor in the background with `startInBackground()`\n5. Start the [background supervisor](/usage/long-running-server#background-supervisor) with `createSupervisor()` to automate maintenance (reclaim stuck jobs, cleanup old data)\n6. Handle `SIGTERM`/`SIGINT` for graceful shutdown with `stopAndDrain()`\n\nSee [Long-Running Server](/usage/long-running-server) for a complete walkthrough and full example."
   },
   {
     "slug": "usage/react-sdk",
@@ -261,13 +273,13 @@
     "slug": "usage/reclaim-jobs",
     "title": "Reclaim Jobs",
     "description": "",
-    "content": "Sometimes, a job can get stuck in the `processing` state. This usually happens if the process is killed or an unhandled error occurs after the job status is updated, but before it is marked as `completed` or `failed`.\n\nTo recover stuck jobs, use the `reclaimStuckJobs` method. The example below shows how to create an API route (`/api/cron/reclaim`) that can be triggered by a cron job:\n\n```typescript title=\"@/app/api/cron/reclaim.ts\"\nimport { getJobQueue } from '@/lib/queue';\nimport { NextResponse } from 'next/server';\n\nexport async function GET(request: Request) {\n  // Secure the cron route: https://vercel.com/docs/cron-jobs/manage-cron-jobs#securing-cron-jobs\n  const authHeader = request.headers.get('authorization');\n  if (authHeader !== `Bearer ${process.env.CRON_SECRET}`) {\n    return NextResponse.json({ message: 'Unauthorized' }, { status: 401 });\n  }\n\n  try {const jobQueue = getJobQueue();\n\n    // Reclaim jobs stuck for more than 10 minutes\n    const reclaimed = await jobQueue.reclaimStuckJobs(10);\n    console.log(`Reclaimed ${reclaimed} stuck jobs`);\n\n    return NextResponse.json({\n      message: 'Stuck jobs reclaimed',\n      reclaimed,\n    });\n  } catch (error) {\n    console.error('Error reclaiming jobs:', error);\n    return NextResponse.json(\n      { message: 'Failed to reclaim jobs' },\n      { status: 500 },\n    );\n  }\n}\n```\n\n#### Per-Job Timeout Awareness\n\n`reclaimStuckJobs` respects each job's individual `timeoutMs`. If a job has a `timeoutMs` that is longer than `maxProcessingTimeMinutes`, it will not be reclaimed until its own timeout has elapsed. For example, if you call `reclaimStuckJobs(10)` and a job has `timeoutMs: 1800000` (30 minutes), that job will only be reclaimed after 30 minutes — not 10.\n\nJobs without a `timeoutMs` continue to use the global `maxProcessingTimeMinutes` threshold as before.\n\n#### Scheduling the Reclaim Job with Cron\n\nAdd the following to your `vercel.json` to call the cron route every 10 minutes:\n\n```json title=\"vercel.json\"\n{\n  \"crons\": [\n    {\n      \"path\": \"/api/cron/reclaim\",\n      \"schedule\": \"*/10 * * * *\"\n    }\n  ]\n}\n```"
+    "content": "> **Note:** Running a long-lived server? Use\n  [`createSupervisor()`](/usage/long-running-server#background-supervisor) to\n  automate stuck-job reclaiming instead of calling `reclaimStuckJobs` manually.\n\nSometimes, a job can get stuck in the `processing` state. This usually happens if the process is killed or an unhandled error occurs after the job status is updated, but before it is marked as `completed` or `failed`.\n\nTo recover stuck jobs, use the `reclaimStuckJobs` method. The example below shows how to create an API route (`/api/cron/reclaim`) that can be triggered by a cron job:\n\n```typescript title=\"@/app/api/cron/reclaim.ts\"\nimport { getJobQueue } from '@/lib/queue';\nimport { NextResponse } from 'next/server';\n\nexport async function GET(request: Request) {\n  // Secure the cron route: https://vercel.com/docs/cron-jobs/manage-cron-jobs#securing-cron-jobs\n  const authHeader = request.headers.get('authorization');\n  if (authHeader !== `Bearer ${process.env.CRON_SECRET}`) {\n    return NextResponse.json({ message: 'Unauthorized' }, { status: 401 });\n  }\n\n  try {const jobQueue = getJobQueue();\n\n    // Reclaim jobs stuck for more than 10 minutes\n    const reclaimed = await jobQueue.reclaimStuckJobs(10);\n    console.log(`Reclaimed ${reclaimed} stuck jobs`);\n\n    return NextResponse.json({\n      message: 'Stuck jobs reclaimed',\n      reclaimed,\n    });\n  } catch (error) {\n    console.error('Error reclaiming jobs:', error);\n    return NextResponse.json(\n      { message: 'Failed to reclaim jobs' },\n      { status: 500 },\n    );\n  }\n}\n```\n\n#### Per-Job Timeout Awareness\n\n`reclaimStuckJobs` respects each job's individual `timeoutMs`. If a job has a `timeoutMs` that is longer than `maxProcessingTimeMinutes`, it will not be reclaimed until its own timeout has elapsed. For example, if you call `reclaimStuckJobs(10)` and a job has `timeoutMs: 1800000` (30 minutes), that job will only be reclaimed after 30 minutes — not 10.\n\nJobs without a `timeoutMs` continue to use the global `maxProcessingTimeMinutes` threshold as before.\n\n#### Scheduling the Reclaim Job with Cron\n\nAdd the following to your `vercel.json` to call the cron route every 10 minutes:\n\n```json title=\"vercel.json\"\n{\n  \"crons\": [\n    {\n      \"path\": \"/api/cron/reclaim\",\n      \"schedule\": \"*/10 * * * *\"\n    }\n  ]\n}\n```"
   },
   {
     "slug": "usage/scaling",
     "title": "Scaling to Thousands of Jobs",
     "description": "",
-    "content": "DataQueue is designed to handle high-volume workloads out of the box. This page covers how to tune your setup for thousands (or more) of concurrent jobs, whether you're using PostgreSQL or Redis.\n\n## How Throughput Works\n\nWhen a processor runs, it follows this cycle:\n\n1. **Claim** a batch of ready jobs from the database (atomically, using `FOR UPDATE SKIP LOCKED` in PostgreSQL or Lua scripts in Redis).\n2. **Process** up to `concurrency` jobs in parallel from that batch.\n3. **Repeat** immediately if the batch was full, or wait `pollInterval` before checking again.\n\nThe theoretical maximum throughput of a single processor instance is:\n\n```\nthroughput = batchSize / (avgJobDuration + pollInterval)\n```\n\nFor example, with `batchSize: 20`, `concurrency: 10`, `pollInterval: 2000ms`, and jobs averaging 500ms each, a single processor can handle roughly **10 jobs/second** (600/minute).\n\n> **Note:** When a full batch is returned, the processor immediately polls again without\n  waiting. This means it can drain backlogs much faster than the formula\n  suggests during peak load.\n\n## Tuning Processor Settings\n\n### Batch Size\n\n`batchSize` controls how many jobs are claimed per polling cycle.\n\n| Environment                 | Recommended | Why                                    |\n| --------------------------- | ----------- | -------------------------------------- |\n| Serverless (Vercel, Lambda) | 1-10        | Functions have execution time limits   |\n| Long-running server         | 10-50       | Larger batches reduce polling overhead |\n| High-throughput worker      | 50-100      | Maximizes throughput per poll cycle    |\n\n### Concurrency\n\n`concurrency` controls how many jobs from the batch run in parallel.\n\n- **CPU-bound jobs** (image processing, compression): keep concurrency low (1-4) to avoid CPU saturation.\n- **IO-bound jobs** (API calls, email sending): higher concurrency (5-20) works well since jobs spend most time waiting.\n- **Rate-limited APIs**: match concurrency to the API's rate limit to avoid throttling.\n\n### Poll Interval\n\n`pollInterval` controls how often the processor checks for new jobs when idle.\n\n- **1000ms**: Low latency, higher database load. Good for real-time workloads.\n- **5000ms** (default): Balanced. Good for most use cases.\n- **10000-30000ms**: Gentle on the database. Use when latency tolerance is high.\n\n```typescript\nconst processor = jobQueue.createProcessor(jobHandlers, {\n  batchSize: 30,\n  concurrency: 10,\n  pollInterval: 2000,\n});\n```\n\n## Horizontal Scaling with Multiple Workers\n\nDataQueue supports running **multiple processor instances** simultaneously -- on the same server, across multiple servers, or in separate containers. No coordination is needed between workers.\n\n### How It Works\n\n- Each processor gets a unique `workerId` (auto-generated, or set manually).\n- PostgreSQL uses `FOR UPDATE SKIP LOCKED` to ensure no two workers claim the same job.\n- Redis uses atomic Lua scripts for the same guarantee.\n- Workers can safely run on different machines pointing at the same database.\n\n### Example: Multiple Workers\n\n```typescript\n// Worker 1 (e.g., on server A)\nconst processor1 = jobQueue.createProcessor(jobHandlers, {\n  workerId: 'worker-a',\n  batchSize: 20,\n  concurrency: 5,\n});\nprocessor1.startInBackground();\n\n// Worker 2 (e.g., on server B)\nconst processor2 = jobQueue.createProcessor(jobHandlers, {\n  workerId: 'worker-b',\n  batchSize: 20,\n  concurrency: 5,\n});\nprocessor2.startInBackground();\n```\n\n### Specialized Workers\n\nUse the `jobType` filter to create workers dedicated to specific job types. This lets you scale different workloads independently:\n\n```typescript\n// Fast worker for lightweight jobs\nconst emailWorker = jobQueue.createProcessor(jobHandlers, {\n  jobType: 'email',\n  batchSize: 50,\n  concurrency: 20,\n  pollInterval: 1000,\n});\n\n// Slow worker for heavy jobs\nconst reportWorker = jobQueue.createProcessor(jobHandlers, {\n  jobType: 'report',\n  batchSize: 5,\n  concurrency: 1,\n  pollInterval: 5000,\n});\n```\n\n## PostgreSQL Scaling\n\n### Connection Pool\n\nEach processor instance uses database connections from the pool. The default `max` is 10, which works for a single processor. If you run multiple processors in the same process, increase the pool size:\n\n```typescript\nconst jobQueue = initJobQueue({\n  databaseConfig: {\n    connectionString: process.env.DATABASE_URL,\n    max: 20, // increase for multiple processors\n  },\n});\n```\n\n> **Note:** If you run processors on separate servers, each has its own pool. The total\n  connections across all servers should stay within your database's\n  `max_connections` setting (typically 100 for managed databases like Neon or\n  Supabase).\n\n### Table Maintenance\n\nAs completed jobs accumulate, the `job_queue` table grows. This doesn't affect processing speed (claim queries use partial indexes that only cover active jobs), but it does increase storage and slow down full-table queries like `getJobs()`.\n\n**Run cleanup regularly:**\n\n```typescript\n// Delete completed jobs older than 30 days\nawait jobQueue.cleanupOldJobs(30);\n\n// Delete old job events\nawait jobQueue.cleanupOldJobEvents(30);\n```\n\nCleanup operations are batched internally (1000 rows at a time) so they won't lock the table or timeout even with hundreds of thousands of old jobs.\n\n**Run reclaim regularly:**\n\n```typescript\n// Reclaim jobs stuck in 'processing' for more than 10 minutes\nawait jobQueue.reclaimStuckJobs(10);\n```\n\nThis recovers jobs from workers that crashed or timed out.\n\n### Monitoring Table Size\n\nYou can monitor the `job_queue` table size with this query:\n\n```sql\nSELECT\n  pg_size_pretty(pg_total_relation_size('job_queue')) AS total_size,\n  (SELECT count(*) FROM job_queue WHERE status = 'pending') AS pending,\n  (SELECT count(*) FROM job_queue WHERE status = 'processing') AS processing,\n  (SELECT count(*) FROM job_queue WHERE status = 'completed') AS completed,\n  (SELECT count(*) FROM job_queue WHERE status = 'failed') AS failed;\n```\n\n### Performance Indexes\n\nDataQueue includes optimized partial indexes out of the box:\n\n- `idx_job_queue_claimable` -- speeds up job claiming (pending jobs by priority).\n- `idx_job_queue_failed_retry` -- speeds up retry scheduling.\n- `idx_job_queue_stuck` -- speeds up stuck job reclamation.\n- `idx_job_queue_cleanup` -- speeds up cleanup of old completed jobs.\n\nThese are created automatically when you run migrations.\n\n## Redis Scaling\n\n### Memory Usage\n\nRedis stores everything in memory. Estimate memory usage as:\n\n| Jobs    | Approximate Memory |\n| ------- | ------------------ |\n| 1,000   | 1-2 MB             |\n| 10,000  | 10-20 MB           |\n| 100,000 | 100-200 MB         |\n\nThis assumes payloads under 1 KB each. Larger payloads increase memory proportionally.\n\n### Key Best Practices\n\n- **Enable persistence**: Use AOF (Append Only File) for durability. Without it, a Redis restart loses all jobs.\n- **Use `keyPrefix`** to isolate multiple queues in the same Redis instance:\n\n```typescript\nconst jobQueue = initJobQueue({\n  backend: 'redis',\n  redisConfig: {\n    url: process.env.REDIS_URL,\n    keyPrefix: 'myapp:jobs:',\n  },\n});\n```\n\n- **Run cleanup regularly** to free memory from completed jobs. Like PostgreSQL, Redis cleanup is batched internally using cursor-based scanning, so it's safe to run even with a large number of completed jobs.\n\n## Payload Best Practices\n\nKeep payloads small. Store references (IDs, URLs) rather than full data:\n\n```typescript\n// Good: small payload with reference\nawait jobQueue.addJob({\n  jobType: 'processImage',\n  payload: { imageId: 'img_abc123', bucket: 's3://uploads' },\n});\n\n// Avoid: large payload with embedded data\nawait jobQueue.addJob({\n  jobType: 'processImage',\n  payload: { imageData: '<base64 string of 5MB image>' }, // too large\n});\n```\n\nA good target is **under 10 KB per payload**. This keeps database queries fast and Redis memory predictable.\n\n## Example: Processing 10,000 Jobs per Hour\n\nHere's a configuration that can comfortably process ~10,000 jobs per hour, assuming each job takes about 1 second:\n\n```typescript title=\"worker.ts\"\nimport { initJobQueue } from '@nicnocquee/dataqueue';\nimport { jobHandlers } from './job-handlers';\n\nconst jobQueue = initJobQueue({\n  databaseConfig: {\n    connectionString: process.env.DATABASE_URL,\n    max: 15, // room for processor + maintenance queries\n  },\n});\n\nconst processor = jobQueue.createProcessor(jobHandlers, {\n  batchSize: 30,\n  concurrency: 10, // 10 jobs in parallel\n  pollInterval: 2000,\n  onError: (error) => {\n    console.error('Processor error:', error);\n  },\n});\n\nprocessor.startInBackground();\n\n// Maintenance\nsetInterval(\n  async () => {\n    try {\n      await jobQueue.reclaimStuckJobs(10);\n    } catch (e) {\n      console.error(e);\n    }\n  },\n  10 * 60 * 1000,\n);\n\nsetInterval(\n  async () => {\n    try {\n      await jobQueue.cleanupOldJobs(30);\n      await jobQueue.cleanupOldJobEvents(30);\n    } catch (e) {\n      console.error(e);\n    }\n  },\n  24 * 60 * 60 * 1000,\n);\n\n// Graceful shutdown\nprocess.on('SIGTERM', async () => {\n  await processor.stopAndDrain(30000);\n  jobQueue.getPool().end();\n  process.exit(0);\n});\n```\n\nWith 10 concurrent jobs and ~1s each, a single worker handles roughly **10 jobs/second** = **36,000 jobs/hour**. For higher throughput, add more worker instances.\n\n## Quick Reference\n\n| Scale               | Workers | batchSize | concurrency | pollInterval | Notes                            |\n| ------------------- | ------- | --------- | ----------- | ------------ | -------------------------------- |\n| < 100 jobs/hour     | 1       | 10        | 3           | 5000ms       | Default settings work fine       |\n| 100-1,000/hour      | 1       | 20        | 5           | 3000ms       | Single worker is sufficient      |\n| 1,000-10,000/hour   | 1-2     | 30        | 10          | 2000ms       | Add a second worker if needed    |\n| 10,000-100,000/hour | 2-5     | 50        | 15          | 1000ms       | Multiple workers recommended     |\n| 100,000+/hour       | 5+      | 50-100    | 20          | 1000ms       | Specialized workers per job type |"
+    "content": "DataQueue is designed to handle high-volume workloads out of the box. This page covers how to tune your setup for thousands (or more) of concurrent jobs, whether you're using PostgreSQL or Redis.\n\n## How Throughput Works\n\nWhen a processor runs, it follows this cycle:\n\n1. **Claim** a batch of ready jobs from the database (atomically, using `FOR UPDATE SKIP LOCKED` in PostgreSQL or Lua scripts in Redis).\n2. **Process** up to `concurrency` jobs in parallel from that batch.\n3. **Repeat** immediately if the batch was full, or wait `pollInterval` before checking again.\n\nThe theoretical maximum throughput of a single processor instance is:\n\n```\nthroughput = batchSize / (avgJobDuration + pollInterval)\n```\n\nFor example, with `batchSize: 20`, `concurrency: 10`, `pollInterval: 2000ms`, and jobs averaging 500ms each, a single processor can handle roughly **10 jobs/second** (600/minute).\n\n> **Note:** When a full batch is returned, the processor immediately polls again without\n  waiting. This means it can drain backlogs much faster than the formula\n  suggests during peak load.\n\n## Tuning Processor Settings\n\n### Batch Size\n\n`batchSize` controls how many jobs are claimed per polling cycle.\n\n| Environment                 | Recommended | Why                                    |\n| --------------------------- | ----------- | -------------------------------------- |\n| Serverless (Vercel, Lambda) | 1-10        | Functions have execution time limits   |\n| Long-running server         | 10-50       | Larger batches reduce polling overhead |\n| High-throughput worker      | 50-100      | Maximizes throughput per poll cycle    |\n\n### Concurrency\n\n`concurrency` controls how many jobs from the batch run in parallel.\n\n- **CPU-bound jobs** (image processing, compression): keep concurrency low (1-4) to avoid CPU saturation.\n- **IO-bound jobs** (API calls, email sending): higher concurrency (5-20) works well since jobs spend most time waiting.\n- **Rate-limited APIs**: match concurrency to the API's rate limit to avoid throttling.\n\n### Group Concurrency\n\n`groupConcurrency` caps how many jobs with the same `group.id` can be in `processing` at the same time across all worker instances.\n\n- Use this for multi-tenant fairness and per-account rate protection.\n- It is global coordination (PostgreSQL + Redis), not per-process.\n- Ungrouped jobs are unaffected.\n\n```typescript\nawait jobQueue.addJob({\n  jobType: 'send_email',\n  payload: { to: 'user@example.com' },\n  group: { id: 'tenant_abc' },\n});\n\nconst processor = jobQueue.createProcessor(jobHandlers, {\n  batchSize: 30,\n  concurrency: 10,\n  groupConcurrency: 2,\n  pollInterval: 2000,\n});\n```\n\n### Poll Interval\n\n`pollInterval` controls how often the processor checks for new jobs when idle.\n\n- **1000ms**: Low latency, higher database load. Good for real-time workloads.\n- **5000ms** (default): Balanced. Good for most use cases.\n- **10000-30000ms**: Gentle on the database. Use when latency tolerance is high.\n\n```typescript\nconst processor = jobQueue.createProcessor(jobHandlers, {\n  batchSize: 30,\n  concurrency: 10,\n  pollInterval: 2000,\n});\n```\n\n## Horizontal Scaling with Multiple Workers\n\nDataQueue supports running **multiple processor instances** simultaneously -- on the same server, across multiple servers, or in separate containers. No coordination is needed between workers.\n\n### How It Works\n\n- Each processor gets a unique `workerId` (auto-generated, or set manually).\n- PostgreSQL uses `FOR UPDATE SKIP LOCKED` to ensure no two workers claim the same job.\n- Redis uses atomic Lua scripts for the same guarantee.\n- Workers can safely run on different machines pointing at the same database.\n\n### Example: Multiple Workers\n\n```typescript\n// Worker 1 (e.g., on server A)\nconst processor1 = jobQueue.createProcessor(jobHandlers, {\n  workerId: 'worker-a',\n  batchSize: 20,\n  concurrency: 5,\n});\nprocessor1.startInBackground();\n\n// Worker 2 (e.g., on server B)\nconst processor2 = jobQueue.createProcessor(jobHandlers, {\n  workerId: 'worker-b',\n  batchSize: 20,\n  concurrency: 5,\n});\nprocessor2.startInBackground();\n```\n\n### Specialized Workers\n\nUse the `jobType` filter to create workers dedicated to specific job types. This lets you scale different workloads independently:\n\n```typescript\n// Fast worker for lightweight jobs\nconst emailWorker = jobQueue.createProcessor(jobHandlers, {\n  jobType: 'email',\n  batchSize: 50,\n  concurrency: 20,\n  pollInterval: 1000,\n});\n\n// Slow worker for heavy jobs\nconst reportWorker = jobQueue.createProcessor(jobHandlers, {\n  jobType: 'report',\n  batchSize: 5,\n  concurrency: 1,\n  pollInterval: 5000,\n});\n```\n\n## PostgreSQL Scaling\n\n### Connection Pool\n\nEach processor instance uses database connections from the pool. The default `max` is 10, which works for a single processor. If you run multiple processors in the same process, increase the pool size:\n\n```typescript\nconst jobQueue = initJobQueue({\n  databaseConfig: {\n    connectionString: process.env.DATABASE_URL,\n    max: 20, // increase for multiple processors\n  },\n});\n```\n\n> **Note:** If you run processors on separate servers, each has its own pool. The total\n  connections across all servers should stay within your database's\n  `max_connections` setting (typically 100 for managed databases like Neon or\n  Supabase).\n\n### Table Maintenance\n\nAs completed jobs accumulate, the `job_queue` table grows. This doesn't affect processing speed (claim queries use partial indexes that only cover active jobs), but it does increase storage and slow down full-table queries like `getJobs()`.\n\n**Run cleanup regularly:**\n\n```typescript\n// Delete completed jobs older than 30 days\nawait jobQueue.cleanupOldJobs(30);\n\n// Delete old job events\nawait jobQueue.cleanupOldJobEvents(30);\n```\n\nCleanup operations are batched internally (1000 rows at a time) so they won't lock the table or timeout even with hundreds of thousands of old jobs.\n\n**Run reclaim regularly:**\n\n```typescript\n// Reclaim jobs stuck in 'processing' for more than 10 minutes\nawait jobQueue.reclaimStuckJobs(10);\n```\n\nThis recovers jobs from workers that crashed or timed out.\n\n### Monitoring Table Size\n\nYou can monitor the `job_queue` table size with this query:\n\n```sql\nSELECT\n  pg_size_pretty(pg_total_relation_size('job_queue')) AS total_size,\n  (SELECT count(*) FROM job_queue WHERE status = 'pending') AS pending,\n  (SELECT count(*) FROM job_queue WHERE status = 'processing') AS processing,\n  (SELECT count(*) FROM job_queue WHERE status = 'completed') AS completed,\n  (SELECT count(*) FROM job_queue WHERE status = 'failed') AS failed;\n```\n\n### Performance Indexes\n\nDataQueue includes optimized partial indexes out of the box:\n\n- `idx_job_queue_claimable` -- speeds up job claiming (pending jobs by priority).\n- `idx_job_queue_failed_retry` -- speeds up retry scheduling.\n- `idx_job_queue_stuck` -- speeds up stuck job reclamation.\n- `idx_job_queue_cleanup` -- speeds up cleanup of old completed jobs.\n\nThese are created automatically when you run migrations.\n\n## Redis Scaling\n\n### Memory Usage\n\nRedis stores everything in memory. Estimate memory usage as:\n\n| Jobs    | Approximate Memory |\n| ------- | ------------------ |\n| 1,000   | 1-2 MB             |\n| 10,000  | 10-20 MB           |\n| 100,000 | 100-200 MB         |\n\nThis assumes payloads under 1 KB each. Larger payloads increase memory proportionally.\n\n### Key Best Practices\n\n- **Enable persistence**: Use AOF (Append Only File) for durability. Without it, a Redis restart loses all jobs.\n- **Use `keyPrefix`** to isolate multiple queues in the same Redis instance:\n\n```typescript\nconst jobQueue = initJobQueue({\n  backend: 'redis',\n  redisConfig: {\n    url: process.env.REDIS_URL,\n    keyPrefix: 'myapp:jobs:',\n  },\n});\n```\n\n- **Run cleanup regularly** to free memory from completed jobs. Like PostgreSQL, Redis cleanup is batched internally using cursor-based scanning, so it's safe to run even with a large number of completed jobs.\n\n## Payload Best Practices\n\nKeep payloads small. Store references (IDs, URLs) rather than full data:\n\n```typescript\n// Good: small payload with reference\nawait jobQueue.addJob({\n  jobType: 'processImage',\n  payload: { imageId: 'img_abc123', bucket: 's3://uploads' },\n});\n\n// Avoid: large payload with embedded data\nawait jobQueue.addJob({\n  jobType: 'processImage',\n  payload: { imageData: '<base64 string of 5MB image>' }, // too large\n});\n```\n\nA good target is **under 10 KB per payload**. This keeps database queries fast and Redis memory predictable.\n\n## Example: Processing 10,000 Jobs per Hour\n\nHere's a configuration that can comfortably process ~10,000 jobs per hour, assuming each job takes about 1 second:\n\n```typescript title=\"worker.ts\"\nimport { initJobQueue } from '@nicnocquee/dataqueue';\nimport { jobHandlers } from './job-handlers';\n\nconst jobQueue = initJobQueue({\n  databaseConfig: {\n    connectionString: process.env.DATABASE_URL,\n    max: 15, // room for processor + maintenance queries\n  },\n});\n\nconst processor = jobQueue.createProcessor(jobHandlers, {\n  batchSize: 30,\n  concurrency: 10, // 10 jobs in parallel\n  pollInterval: 2000,\n  onError: (error) => {\n    console.error('Processor error:', error);\n  },\n});\n\nprocessor.startInBackground();\n\n// Maintenance\nsetInterval(\n  async () => {\n    try {\n      await jobQueue.reclaimStuckJobs(10);\n    } catch (e) {\n      console.error(e);\n    }\n  },\n  10 * 60 * 1000,\n);\n\nsetInterval(\n  async () => {\n    try {\n      await jobQueue.cleanupOldJobs(30);\n      await jobQueue.cleanupOldJobEvents(30);\n    } catch (e) {\n      console.error(e);\n    }\n  },\n  24 * 60 * 60 * 1000,\n);\n\n// Graceful shutdown\nprocess.on('SIGTERM', async () => {\n  await processor.stopAndDrain(30000);\n  jobQueue.getPool().end();\n  process.exit(0);\n});\n```\n\nWith 10 concurrent jobs and ~1s each, a single worker handles roughly **10 jobs/second** = **36,000 jobs/hour**. For higher throughput, add more worker instances.\n\n## Quick Reference\n\n| Scale               | Workers | batchSize | concurrency | pollInterval | Notes                            |\n| ------------------- | ------- | --------- | ----------- | ------------ | -------------------------------- |\n| < 100 jobs/hour     | 1       | 10        | 3           | 5000ms       | Default settings work fine       |\n| 100-1,000/hour      | 1       | 20        | 5           | 3000ms       | Single worker is sufficient      |\n| 1,000-10,000/hour   | 1-2     | 30        | 10          | 2000ms       | Add a second worker if needed    |\n| 10,000-100,000/hour | 2-5     | 50        | 15          | 1000ms       | Multiple workers recommended     |\n| 100,000+/hour       | 5+      | 50-100    | 20          | 1000ms       | Specialized workers per job type |"
   },
   {
     "slug": "usage/wait",