@nicnocquee/dataqueue 1.35.0-beta.20260224075710 → 1.35.0-beta.20260224110011

package/ai/docs-content.json

@@ -39,7 +39,7 @@
     "slug": "api/job-queue",
     "title": "JobQueue",
     "description": "",
-    "content": "## Initialization\n\n### initJobQueue\n\n```ts\ninitJobQueue(config: JobQueueConfig): JobQueue\n```\n\nInitializes the job queue system with the provided configuration. The `JobQueueConfig` is a discriminated union -- you provide either a PostgreSQL or Redis configuration.\n\n#### PostgresJobQueueConfig\n\nProvide either `databaseConfig` (the library creates a pool) or `pool` (bring your own `pg.Pool`). At least one must be set.\n\n```ts\ninterface PostgresJobQueueConfig {\n  backend?: 'postgres'; // Optional, defaults to 'postgres'\n  databaseConfig?: {\n    connectionString?: string;\n    host?: string;\n    port?: number;\n    database?: string;\n    user?: string;\n    password?: string;\n    ssl?: DatabaseSSLConfig;\n  };\n  pool?: import('pg').Pool; // Bring your own pool\n  verbose?: boolean;\n}\n```\n\n#### RedisJobQueueConfig\n\nProvide either `redisConfig` (the library creates an ioredis client) or `client` (bring your own). At least one must be set.\n\n```ts\ninterface RedisJobQueueConfig {\n  backend: 'redis'; // Required\n  redisConfig?: {\n    url?: string;\n    host?: string;\n    port?: number;\n    password?: string;\n    db?: number;\n    tls?: RedisTLSConfig;\n    keyPrefix?: string; // Default: 'dq:'\n  };\n  client?: unknown; // Bring your own ioredis client\n  keyPrefix?: string; // Key prefix when using external client (default: 'dq:')\n  verbose?: boolean;\n}\n```\n\n#### JobQueueConfig\n\n```ts\ntype JobQueueConfig = PostgresJobQueueConfig | RedisJobQueueConfig;\n```\n\n#### DatabaseSSLConfig\n\n```ts\ninterface DatabaseSSLConfig {\n  ca?: string;\n  cert?: string;\n  key?: string;\n  rejectUnauthorized?: boolean;\n}\n```\n\n- `ca` - Client certificate authority (CA) as PEM string or file path. If the value starts with 'file://', it will be loaded from file, otherwise treated as PEM string.\n- `cert` - Client certificate as PEM string or file path. If the value starts with 'file://', it will be loaded from file, otherwise treated as PEM string.\n- `key` - Client private key as PEM string or file path. If the value starts with 'file://', it will be loaded from file, otherwise treated as PEM string.\n- `rejectUnauthorized` - Whether to reject unauthorized certificates (default: true)\n\n#### RedisTLSConfig\n\n```ts\ninterface RedisTLSConfig {\n  ca?: string;\n  cert?: string;\n  key?: string;\n  rejectUnauthorized?: boolean;\n}\n```\n\n---\n\n## Adding Jobs\n\n### addJob\n\n```ts\naddJob(job: JobOptions, options?: AddJobOptions): Promise<number>\n```\n\nAdds a job to the queue. Returns the job ID.\n\n#### JobOptions\n\n```ts\ninterface JobOptions {\n  jobType: string;\n  payload: any;\n  maxAttempts?: number;\n  priority?: number;\n  runAt?: Date | null;\n  timeoutMs?: number;\n  tags?: string[];\n  idempotencyKey?: string;\n  retryDelay?: number; // Base delay between retries in seconds (default: 60)\n  retryBackoff?: boolean; // Use exponential backoff (default: true)\n  retryDelayMax?: number; // Max delay cap in seconds (default: none)\n}\n```\n\n- `retryDelay` - Base delay between retries in seconds. When `retryBackoff` is true, this is the base for exponential backoff (`retryDelay * 2^attempts`). When false, retries use this fixed delay. Default: `60`.\n- `retryBackoff` - Whether to use exponential backoff. When true, delay doubles with each attempt and includes jitter. Default: `true`.\n- `retryDelayMax` - Maximum delay cap in seconds. Only meaningful when `retryBackoff` is true. No limit when omitted.\n\n#### AddJobOptions\n\n```ts\ninterface AddJobOptions {\n  db?: DatabaseClient;\n}\n```\n\n- `db` — An external database client (e.g., a `pg.PoolClient` inside a transaction). When provided, the INSERT runs on this client instead of the internal pool. **PostgreSQL only.** Throws if used with the Redis backend.\n\n### addJobs\n\n```ts\naddJobs(jobs: JobOptions[], options?: AddJobOptions): Promise<number[]>\n```\n\nAdds multiple jobs to the queue in a single operation. More efficient than calling `addJob` in a loop because it batches the INSERT into a single database round-trip (PostgreSQL) or a single atomic Lua script (Redis).\n\nReturns an array of job IDs in the same order as the input array.\n\nEach job can independently have its own `priority`, `runAt`, `tags`, `idempotencyKey`, and other options. Idempotency keys are handled per-job — duplicates resolve to the existing job's ID without creating a new row.\n\nPassing an empty array returns `[]` immediately without touching the database.\n\n```ts\nconst jobIds = await jobQueue.addJobs([\n  {\n    jobType: 'email',\n    payload: { to: 'a@example.com', subject: 'Hi', body: '...' },\n  },\n  {\n    jobType: 'email',\n    payload: { to: 'b@example.com', subject: 'Hi', body: '...' },\n    priority: 10,\n  },\n  {\n    jobType: 'report',\n    payload: { reportId: '123', userId: '456' },\n    tags: ['monthly'],\n  },\n]);\n// jobIds = [1, 2, 3]\n```\n\nThe `{ db }` option works the same as `addJob` — pass a transactional client to batch-insert within an existing transaction (PostgreSQL only).\n\n#### DatabaseClient\n\n```ts\ninterface DatabaseClient {\n  query(\n    text: string,\n    values?: any[],\n  ): Promise<{ rows: any[]; rowCount: number | null }>;\n}\n```\n\nAny object matching this interface works — `pg.Pool`, `pg.PoolClient`, `pg.Client`, or ORM query runners that expose a raw `query()` method.\n\n---\n\n## Retrieving Jobs\n\n### getJob\n\n```ts\ngetJob(id: number): Promise<JobRecord | null>\n```\n\nRetrieves a job by its ID.\n\n### getJobs\n\n```ts\ngetJobs(\n  filters?: {\n    jobType?: string;\n    priority?: number;\n    runAt?: Date | { gt?: Date; gte?: Date; lt?: Date; lte?: Date; eq?: Date };\n    tags?: { values: string[]; mode?: 'all' | 'any' | 'none' | 'exact' };\n  },\n  limit?: number,\n  offset?: number\n): Promise<JobRecord[]>\n```\n\nRetrieves jobs matching the provided filters, with optional pagination.\n\n### getJobsByStatus\n\n```ts\ngetJobsByStatus(status: string, limit?: number, offset?: number): Promise<JobRecord[]>\n```\n\nRetrieves jobs by their status, with pagination.\n\n### getAllJobs\n\n```ts\ngetAllJobs(limit?: number, offset?: number): Promise<JobRecord[]>\n```\n\nRetrieves all jobs, with optional pagination.\n\n### getJobsByTags\n\n```ts\ngetJobsByTags(tags: string[], mode?: TagQueryMode, limit?: number, offset?: number): Promise<JobRecord[]>\n```\n\nRetrieves jobs by tag(s).\n\n---\n\n## Managing Jobs\n\n### retryJob\n\n```ts\nretryJob(jobId: number): Promise<void>\n```\n\nRetries a job given its ID.\n\n### cancelJob\n\n```ts\ncancelJob(jobId: number): Promise<void>\n```\n\nCancels a job given its ID.\n\n### editJob\n\n```ts\neditJob(jobId: number, updates: EditJobOptions): Promise<void>\n```\n\nEdits a pending job given its ID. Only works for jobs with status 'pending'. Silently fails for other statuses (processing, completed, failed, cancelled).\n\n#### EditJobOptions\n\n```ts\ninterface EditJobOptions {\n  payload?: any;\n  maxAttempts?: number;\n  priority?: number;\n  runAt?: Date | null;\n  timeoutMs?: number;\n  tags?: string[];\n  retryDelay?: number | null;\n  retryBackoff?: boolean | null;\n  retryDelayMax?: number | null;\n}\n```\n\nAll fields are optional - only provided fields will be updated. Note that `jobType` cannot be changed. Set retry fields to `null` to revert to legacy default behavior.\n\n#### Example\n\n```ts\n// Edit a pending job's payload and priority\nawait jobQueue.editJob(jobId, {\n  payload: { to: 'newemail@example.com', subject: 'Updated' },\n  priority: 10,\n});\n\n// Edit only the scheduled run time\nawait jobQueue.editJob(jobId, {\n  runAt: new Date(Date.now() + 60000), // Run in 1 minute\n});\n\n// Edit multiple fields at once\nawait jobQueue.editJob(jobId, {\n  payload: { to: 'updated@example.com' },\n  priority: 5,\n  maxAttempts: 10,\n  timeoutMs: 30000,\n  tags: ['urgent', 'priority'],\n});\n```\n\n### editAllPendingJobs\n\n```ts\neditAllPendingJobs(\n  filters?: {\n    jobType?: string;\n    priority?: number;\n    runAt?: Date | { gt?: Date; gte?: Date; lt?: Date; lte?: Date; eq?: Date };\n    tags?: { values: string[]; mode?: 'all' | 'any' | 'none' | 'exact' };\n  },\n  updates: EditJobOptions\n): Promise<number>\n```\n\nEdits all pending jobs that match the filters. Only works for jobs with status 'pending'. Non-pending jobs are not affected. Returns the number of jobs that were edited.\n\n#### Parameters\n\n- `filters` (optional): Filters to select which jobs to edit. If not provided, all pending jobs are edited.\n  - `jobType`: Filter by job type\n  - `priority`: Filter by priority\n  - `runAt`: Filter by scheduled run time (supports `gt`, `gte`, `lt`, `lte`, `eq` operators or exact Date match)\n  - `tags`: Filter by tags with mode ('all', 'any', 'none', 'exact')\n- `updates`: The fields to update (same as `EditJobOptions`). All fields are optional - only provided fields will be updated.\n\n#### Returns\n\nThe number of jobs that were successfully edited.\n\n#### Examples\n\n```ts\n// Edit all pending jobs\nconst editedCount = await jobQueue.editAllPendingJobs(undefined, {\n  priority: 10,\n});\n\n// Edit all pending email jobs\nconst editedCount = await jobQueue.editAllPendingJobs(\n  { jobType: 'email' },\n  {\n    priority: 5,\n  },\n);\n\n// Edit all pending jobs with 'urgent' tag\nconst editedCount = await jobQueue.editAllPendingJobs(\n  { tags: { values: ['urgent'], mode: 'any' } },\n  {\n    priority: 10,\n    maxAttempts: 5,\n  },\n);\n\n// Edit all pending jobs scheduled in the future\nconst editedCount = await jobQueue.editAllPendingJobs(\n  { runAt: { gte: new Date() } },\n  {\n    priority: 10,\n  },\n);\n\n// Edit with combined filters\nconst editedCount = await jobQueue.editAllPendingJobs(\n  {\n    jobType: 'email',\n    tags: { values: ['urgent'], mode: 'any' },\n  },\n  {\n    priority: 10,\n    maxAttempts: 5,\n  },\n);\n```\n\n**Note:** Only pending jobs are edited. Jobs with other statuses (processing, completed, failed, cancelled) are not affected. Edit events are recorded for each affected job, just like single job edits.\n\n### cancelAllUpcomingJobs\n\n```ts\ncancelAllUpcomingJobs(filters?: {\n  jobType?: string;\n  priority?: number;\n  runAt?: Date | { gt?: Date; gte?: Date; lt?: Date; lte?: Date; eq?: Date };\n  tags?: { values: string[]; mode?: 'all' | 'any' | 'none' | 'exact' };\n}): Promise<number>\n```\n\nCancels all upcoming jobs that match the filters. Returns the number of jobs cancelled.\n\n### cleanupOldJobs\n\n```ts\ncleanupOldJobs(daysToKeep?: number): Promise<number>\n```\n\nCleans up jobs older than the specified number of days. Returns the number of jobs removed.\n\n### reclaimStuckJobs\n\n```ts\nreclaimStuckJobs(maxProcessingTimeMinutes?: number): Promise<number>\n```\n\nReclaims jobs stuck in 'processing' for too long. Returns the number of jobs reclaimed. If a job has a `timeoutMs` that is longer than the `maxProcessingTimeMinutes` threshold, the job's own timeout is used instead, preventing premature reclamation of long-running jobs.\n\n---\n\n## Job Events\n\n### getJobEvents\n\n```ts\ngetJobEvents(jobId: number): Promise<JobEvent[]>\n```\n\nRetrieves the job events for a job.\n\n#### JobEvent\n\n```ts\ninterface JobEvent {\n  id: number;\n  jobId: number;\n  eventType: JobEventType;\n  createdAt: Date;\n  metadata: any;\n}\n```\n\n#### JobEventType\n\n```ts\nenum JobEventType {\n  Added = 'added',\n  Processing = 'processing',\n  Completed = 'completed',\n  Failed = 'failed',\n  Cancelled = 'cancelled',\n  Retried = 'retried',\n  Edited = 'edited',\n}\n```\n\n---\n\n## Event Hooks\n\nDataQueue emits real-time events for job lifecycle transitions. Register listeners using `on`, `once`, `off`, and `removeAllListeners`. Works identically with both PostgreSQL and Redis backends.\n\n### QueueEventMap\n\n```ts\ninterface QueueEventMap {\n  'job:added': { jobId: number; jobType: string };\n  'job:processing': { jobId: number; jobType: string };\n  'job:completed': { jobId: number; jobType: string };\n  'job:failed': {\n    jobId: number;\n    jobType: string;\n    error: Error;\n    willRetry: boolean;\n  };\n  'job:cancelled': { jobId: number };\n  'job:retried': { jobId: number };\n  'job:waiting': { jobId: number; jobType: string };\n  'job:progress': { jobId: number; progress: number };\n  error: Error;\n}\n```\n\n### on\n\n```ts\non(event: QueueEventName, listener: (data) => void): void\n```\n\nRegister a listener that fires every time the event is emitted.\n\n### once\n\n```ts\nonce(event: QueueEventName, listener: (data) => void): void\n```\n\nRegister a one-time listener that auto-removes after the first invocation.\n\n### off\n\n```ts\noff(event: QueueEventName, listener: (data) => void): void\n```\n\nRemove a previously registered listener. Pass the exact function reference used with `on` or `once`.\n\n### removeAllListeners\n\n```ts\nremoveAllListeners(event?: QueueEventName): void\n```\n\nRemove all listeners for a specific event, or all listeners for all events when called without arguments.\n\nSee [Event Hooks](/usage/event-hooks) for detailed usage examples.\n\n---\n\n## Processing Jobs\n\n### createProcessor\n\n```ts\ncreateProcessor(\n  handlers: JobHandlers,\n  options?: ProcessorOptions\n): Processor\n```\n\nCreates a job processor with the provided handlers and options.\n\n#### ProcessorOptions\n\n```ts\ninterface ProcessorOptions {\n  workerId?: string;\n  batchSize?: number;\n  concurrency?: number;\n  pollInterval?: number;\n  onError?: (error: Error) => void;\n  verbose?: boolean;\n  jobType?: string | string[];\n}\n```\n\n---\n\n## Background Supervisor\n\n### createSupervisor\n\n```ts\ncreateSupervisor(options?: SupervisorOptions): Supervisor\n```\n\nCreates a background supervisor that automatically runs maintenance tasks on a configurable interval: reclaiming stuck jobs, cleaning up old completed jobs/events, and expiring timed-out waitpoint tokens.\n\n#### SupervisorOptions\n\n```ts\ninterface SupervisorOptions {\n  intervalMs?: number; // default: 60000\n  stuckJobsTimeoutMinutes?: number; // default: 10\n  cleanupJobsDaysToKeep?: number; // default: 30 (0 to disable)\n  cleanupEventsDaysToKeep?: number; // default: 30 (0 to disable)\n  cleanupBatchSize?: number; // default: 1000\n  reclaimStuckJobs?: boolean; // default: true\n  expireTimedOutTokens?: boolean; // default: true\n  onError?: (error: Error) => void; // default: console.error\n  verbose?: boolean;\n}\n```\n\n#### Supervisor\n\n```ts\ninterface Supervisor {\n  start(): Promise<SupervisorRunResult>;\n  startInBackground(): void;\n  stop(): void;\n  stopAndDrain(timeoutMs?: number): Promise<void>;\n  isRunning(): boolean;\n}\n```\n\n- `start()` runs all tasks once and returns the results (serverless-friendly).\n- `startInBackground()` starts a background loop that runs every `intervalMs`.\n- `stopAndDrain()` stops the loop and waits for the current run to finish.\n\n#### SupervisorRunResult\n\n```ts\ninterface SupervisorRunResult {\n  reclaimedJobs: number;\n  cleanedUpJobs: number;\n  cleanedUpEvents: number;\n  expiredTokens: number;\n}\n```\n\nSee [Long-Running Server](/usage/long-running-server#background-supervisor) for usage examples.\n\n---\n\n## Accessing the Underlying Client\n\n### getPool\n\n```ts\ngetPool(): Pool\n```\n\nReturns the PostgreSQL connection pool instance. Only available when using the PostgreSQL backend.\n\n> **Note:** Throws an error if called when using the Redis backend.\n\n### getRedisClient\n\n```ts\ngetRedisClient(): Redis\n```\n\nReturns the `ioredis` client instance. Only available when using the Redis backend.\n\n> **Note:** Throws an error if called when using the PostgreSQL backend."
+    "content": "## Initialization\n\n### initJobQueue\n\n```ts\ninitJobQueue(config: JobQueueConfig): JobQueue\n```\n\nInitializes the job queue system with the provided configuration. The `JobQueueConfig` is a discriminated union -- you provide either a PostgreSQL or Redis configuration.\n\n#### PostgresJobQueueConfig\n\nProvide either `databaseConfig` (the library creates a pool) or `pool` (bring your own `pg.Pool`). At least one must be set.\n\n```ts\ninterface PostgresJobQueueConfig {\n  backend?: 'postgres'; // Optional, defaults to 'postgres'\n  databaseConfig?: {\n    connectionString?: string;\n    host?: string;\n    port?: number;\n    database?: string;\n    user?: string;\n    password?: string;\n    ssl?: DatabaseSSLConfig;\n  };\n  pool?: import('pg').Pool; // Bring your own pool\n  verbose?: boolean;\n}\n```\n\n#### RedisJobQueueConfig\n\nProvide either `redisConfig` (the library creates an ioredis client) or `client` (bring your own). At least one must be set.\n\n```ts\ninterface RedisJobQueueConfig {\n  backend: 'redis'; // Required\n  redisConfig?: {\n    url?: string;\n    host?: string;\n    port?: number;\n    password?: string;\n    db?: number;\n    tls?: RedisTLSConfig;\n    keyPrefix?: string; // Default: 'dq:'\n  };\n  client?: unknown; // Bring your own ioredis client\n  keyPrefix?: string; // Key prefix when using external client (default: 'dq:')\n  verbose?: boolean;\n}\n```\n\n#### JobQueueConfig\n\n```ts\ntype JobQueueConfig = PostgresJobQueueConfig | RedisJobQueueConfig;\n```\n\n#### DatabaseSSLConfig\n\n```ts\ninterface DatabaseSSLConfig {\n  ca?: string;\n  cert?: string;\n  key?: string;\n  rejectUnauthorized?: boolean;\n}\n```\n\n- `ca` - Client certificate authority (CA) as PEM string or file path. If the value starts with 'file://', it will be loaded from file, otherwise treated as PEM string.\n- `cert` - Client certificate as PEM string or file path. If the value starts with 'file://', it will be loaded from file, otherwise treated as PEM string.\n- `key` - Client private key as PEM string or file path. If the value starts with 'file://', it will be loaded from file, otherwise treated as PEM string.\n- `rejectUnauthorized` - Whether to reject unauthorized certificates (default: true)\n\n#### RedisTLSConfig\n\n```ts\ninterface RedisTLSConfig {\n  ca?: string;\n  cert?: string;\n  key?: string;\n  rejectUnauthorized?: boolean;\n}\n```\n\n---\n\n## Adding Jobs\n\n### addJob\n\n```ts\naddJob(job: JobOptions, options?: AddJobOptions): Promise<number>\n```\n\nAdds a job to the queue. Returns the job ID.\n\n#### JobOptions\n\n```ts\ninterface JobOptions {\n  jobType: string;\n  payload: any;\n  maxAttempts?: number;\n  priority?: number;\n  runAt?: Date | null;\n  timeoutMs?: number;\n  tags?: string[];\n  idempotencyKey?: string;\n  retryDelay?: number; // Base delay between retries in seconds (default: 60)\n  retryBackoff?: boolean; // Use exponential backoff (default: true)\n  retryDelayMax?: number; // Max delay cap in seconds (default: none)\n  group?: { id: string; tier?: string }; // Optional group for global concurrency limits\n}\n```\n\n- `retryDelay` - Base delay between retries in seconds. When `retryBackoff` is true, this is the base for exponential backoff (`retryDelay * 2^attempts`). When false, retries use this fixed delay. Default: `60`.\n- `retryBackoff` - Whether to use exponential backoff. When true, delay doubles with each attempt and includes jitter. Default: `true`.\n- `retryDelayMax` - Maximum delay cap in seconds. Only meaningful when `retryBackoff` is true. No limit when omitted.\n- `group` - Optional grouping metadata. Use `group.id` to enforce global per-group limits with `ProcessorOptions.groupConcurrency`. `group.tier` is reserved for future policies.\n\n#### AddJobOptions\n\n```ts\ninterface AddJobOptions {\n  db?: DatabaseClient;\n}\n```\n\n- `db` — An external database client (e.g., a `pg.PoolClient` inside a transaction). When provided, the INSERT runs on this client instead of the internal pool. **PostgreSQL only.** Throws if used with the Redis backend.\n\n### addJobs\n\n```ts\naddJobs(jobs: JobOptions[], options?: AddJobOptions): Promise<number[]>\n```\n\nAdds multiple jobs to the queue in a single operation. More efficient than calling `addJob` in a loop because it batches the INSERT into a single database round-trip (PostgreSQL) or a single atomic Lua script (Redis).\n\nReturns an array of job IDs in the same order as the input array.\n\nEach job can independently have its own `priority`, `runAt`, `tags`, `idempotencyKey`, and other options. Idempotency keys are handled per-job — duplicates resolve to the existing job's ID without creating a new row.\n\nPassing an empty array returns `[]` immediately without touching the database.\n\n```ts\nconst jobIds = await jobQueue.addJobs([\n  {\n    jobType: 'email',\n    payload: { to: 'a@example.com', subject: 'Hi', body: '...' },\n  },\n  {\n    jobType: 'email',\n    payload: { to: 'b@example.com', subject: 'Hi', body: '...' },\n    priority: 10,\n  },\n  {\n    jobType: 'report',\n    payload: { reportId: '123', userId: '456' },\n    tags: ['monthly'],\n  },\n]);\n// jobIds = [1, 2, 3]\n```\n\nThe `{ db }` option works the same as `addJob` — pass a transactional client to batch-insert within an existing transaction (PostgreSQL only).\n\n#### DatabaseClient\n\n```ts\ninterface DatabaseClient {\n  query(\n    text: string,\n    values?: any[],\n  ): Promise<{ rows: any[]; rowCount: number | null }>;\n}\n```\n\nAny object matching this interface works — `pg.Pool`, `pg.PoolClient`, `pg.Client`, or ORM query runners that expose a raw `query()` method.\n\n---\n\n## Retrieving Jobs\n\n### getJob\n\n```ts\ngetJob(id: number): Promise<JobRecord | null>\n```\n\nRetrieves a job by its ID.\n\n### getJobs\n\n```ts\ngetJobs(\n  filters?: {\n    jobType?: string;\n    priority?: number;\n    runAt?: Date | { gt?: Date; gte?: Date; lt?: Date; lte?: Date; eq?: Date };\n    tags?: { values: string[]; mode?: 'all' | 'any' | 'none' | 'exact' };\n  },\n  limit?: number,\n  offset?: number\n): Promise<JobRecord[]>\n```\n\nRetrieves jobs matching the provided filters, with optional pagination.\n\n### getJobsByStatus\n\n```ts\ngetJobsByStatus(status: string, limit?: number, offset?: number): Promise<JobRecord[]>\n```\n\nRetrieves jobs by their status, with pagination.\n\n### getAllJobs\n\n```ts\ngetAllJobs(limit?: number, offset?: number): Promise<JobRecord[]>\n```\n\nRetrieves all jobs, with optional pagination.\n\n### getJobsByTags\n\n```ts\ngetJobsByTags(tags: string[], mode?: TagQueryMode, limit?: number, offset?: number): Promise<JobRecord[]>\n```\n\nRetrieves jobs by tag(s).\n\n---\n\n## Managing Jobs\n\n### retryJob\n\n```ts\nretryJob(jobId: number): Promise<void>\n```\n\nRetries a job given its ID.\n\n### cancelJob\n\n```ts\ncancelJob(jobId: number): Promise<void>\n```\n\nCancels a job given its ID.\n\n### editJob\n\n```ts\neditJob(jobId: number, updates: EditJobOptions): Promise<void>\n```\n\nEdits a pending job given its ID. Only works for jobs with status 'pending'. Silently fails for other statuses (processing, completed, failed, cancelled).\n\n#### EditJobOptions\n\n```ts\ninterface EditJobOptions {\n  payload?: any;\n  maxAttempts?: number;\n  priority?: number;\n  runAt?: Date | null;\n  timeoutMs?: number;\n  tags?: string[];\n  retryDelay?: number | null;\n  retryBackoff?: boolean | null;\n  retryDelayMax?: number | null;\n}\n```\n\nAll fields are optional - only provided fields will be updated. Note that `jobType` cannot be changed. Set retry fields to `null` to revert to legacy default behavior.\n\n#### Example\n\n```ts\n// Edit a pending job's payload and priority\nawait jobQueue.editJob(jobId, {\n  payload: { to: 'newemail@example.com', subject: 'Updated' },\n  priority: 10,\n});\n\n// Edit only the scheduled run time\nawait jobQueue.editJob(jobId, {\n  runAt: new Date(Date.now() + 60000), // Run in 1 minute\n});\n\n// Edit multiple fields at once\nawait jobQueue.editJob(jobId, {\n  payload: { to: 'updated@example.com' },\n  priority: 5,\n  maxAttempts: 10,\n  timeoutMs: 30000,\n  tags: ['urgent', 'priority'],\n});\n```\n\n### editAllPendingJobs\n\n```ts\neditAllPendingJobs(\n  filters?: {\n    jobType?: string;\n    priority?: number;\n    runAt?: Date | { gt?: Date; gte?: Date; lt?: Date; lte?: Date; eq?: Date };\n    tags?: { values: string[]; mode?: 'all' | 'any' | 'none' | 'exact' };\n  },\n  updates: EditJobOptions\n): Promise<number>\n```\n\nEdits all pending jobs that match the filters. Only works for jobs with status 'pending'. Non-pending jobs are not affected. Returns the number of jobs that were edited.\n\n#### Parameters\n\n- `filters` (optional): Filters to select which jobs to edit. If not provided, all pending jobs are edited.\n  - `jobType`: Filter by job type\n  - `priority`: Filter by priority\n  - `runAt`: Filter by scheduled run time (supports `gt`, `gte`, `lt`, `lte`, `eq` operators or exact Date match)\n  - `tags`: Filter by tags with mode ('all', 'any', 'none', 'exact')\n- `updates`: The fields to update (same as `EditJobOptions`). All fields are optional - only provided fields will be updated.\n\n#### Returns\n\nThe number of jobs that were successfully edited.\n\n#### Examples\n\n```ts\n// Edit all pending jobs\nconst editedCount = await jobQueue.editAllPendingJobs(undefined, {\n  priority: 10,\n});\n\n// Edit all pending email jobs\nconst editedCount = await jobQueue.editAllPendingJobs(\n  { jobType: 'email' },\n  {\n    priority: 5,\n  },\n);\n\n// Edit all pending jobs with 'urgent' tag\nconst editedCount = await jobQueue.editAllPendingJobs(\n  { tags: { values: ['urgent'], mode: 'any' } },\n  {\n    priority: 10,\n    maxAttempts: 5,\n  },\n);\n\n// Edit all pending jobs scheduled in the future\nconst editedCount = await jobQueue.editAllPendingJobs(\n  { runAt: { gte: new Date() } },\n  {\n    priority: 10,\n  },\n);\n\n// Edit with combined filters\nconst editedCount = await jobQueue.editAllPendingJobs(\n  {\n    jobType: 'email',\n    tags: { values: ['urgent'], mode: 'any' },\n  },\n  {\n    priority: 10,\n    maxAttempts: 5,\n  },\n);\n```\n\n**Note:** Only pending jobs are edited. Jobs with other statuses (processing, completed, failed, cancelled) are not affected. Edit events are recorded for each affected job, just like single job edits.\n\n### cancelAllUpcomingJobs\n\n```ts\ncancelAllUpcomingJobs(filters?: {\n  jobType?: string;\n  priority?: number;\n  runAt?: Date | { gt?: Date; gte?: Date; lt?: Date; lte?: Date; eq?: Date };\n  tags?: { values: string[]; mode?: 'all' | 'any' | 'none' | 'exact' };\n}): Promise<number>\n```\n\nCancels all upcoming jobs that match the filters. Returns the number of jobs cancelled.\n\n### cleanupOldJobs\n\n```ts\ncleanupOldJobs(daysToKeep?: number): Promise<number>\n```\n\nCleans up jobs older than the specified number of days. Returns the number of jobs removed.\n\n### reclaimStuckJobs\n\n```ts\nreclaimStuckJobs(maxProcessingTimeMinutes?: number): Promise<number>\n```\n\nReclaims jobs stuck in 'processing' for too long. Returns the number of jobs reclaimed. If a job has a `timeoutMs` that is longer than the `maxProcessingTimeMinutes` threshold, the job's own timeout is used instead, preventing premature reclamation of long-running jobs.\n\n---\n\n## Job Events\n\n### getJobEvents\n\n```ts\ngetJobEvents(jobId: number): Promise<JobEvent[]>\n```\n\nRetrieves the job events for a job.\n\n#### JobEvent\n\n```ts\ninterface JobEvent {\n  id: number;\n  jobId: number;\n  eventType: JobEventType;\n  createdAt: Date;\n  metadata: any;\n}\n```\n\n#### JobEventType\n\n```ts\nenum JobEventType {\n  Added = 'added',\n  Processing = 'processing',\n  Completed = 'completed',\n  Failed = 'failed',\n  Cancelled = 'cancelled',\n  Retried = 'retried',\n  Edited = 'edited',\n}\n```\n\n---\n\n## Event Hooks\n\nDataQueue emits real-time events for job lifecycle transitions. Register listeners using `on`, `once`, `off`, and `removeAllListeners`. Works identically with both PostgreSQL and Redis backends.\n\n### QueueEventMap\n\n```ts\ninterface QueueEventMap {\n  'job:added': { jobId: number; jobType: string };\n  'job:processing': { jobId: number; jobType: string };\n  'job:completed': { jobId: number; jobType: string };\n  'job:failed': {\n    jobId: number;\n    jobType: string;\n    error: Error;\n    willRetry: boolean;\n  };\n  'job:cancelled': { jobId: number };\n  'job:retried': { jobId: number };\n  'job:waiting': { jobId: number; jobType: string };\n  'job:progress': { jobId: number; progress: number };\n  error: Error;\n}\n```\n\n### on\n\n```ts\non(event: QueueEventName, listener: (data) => void): void\n```\n\nRegister a listener that fires every time the event is emitted.\n\n### once\n\n```ts\nonce(event: QueueEventName, listener: (data) => void): void\n```\n\nRegister a one-time listener that auto-removes after the first invocation.\n\n### off\n\n```ts\noff(event: QueueEventName, listener: (data) => void): void\n```\n\nRemove a previously registered listener. Pass the exact function reference used with `on` or `once`.\n\n### removeAllListeners\n\n```ts\nremoveAllListeners(event?: QueueEventName): void\n```\n\nRemove all listeners for a specific event, or all listeners for all events when called without arguments.\n\nSee [Event Hooks](/usage/event-hooks) for detailed usage examples.\n\n---\n\n## Processing Jobs\n\n### createProcessor\n\n```ts\ncreateProcessor(\n  handlers: JobHandlers,\n  options?: ProcessorOptions\n): Processor\n```\n\nCreates a job processor with the provided handlers and options.\n\n#### ProcessorOptions\n\n```ts\ninterface ProcessorOptions {\n  workerId?: string;\n  batchSize?: number;\n  concurrency?: number;\n  groupConcurrency?: number;\n  pollInterval?: number;\n  onError?: (error: Error) => void;\n  verbose?: boolean;\n  jobType?: string | string[];\n}\n```\n\n- `groupConcurrency` - Optional global per-group concurrency limit (positive integer). Applies only to jobs with `group.id`; ungrouped jobs are unaffected.\n\n---\n\n## Background Supervisor\n\n### createSupervisor\n\n```ts\ncreateSupervisor(options?: SupervisorOptions): Supervisor\n```\n\nCreates a background supervisor that automatically runs maintenance tasks on a configurable interval: reclaiming stuck jobs, cleaning up old completed jobs/events, and expiring timed-out waitpoint tokens.\n\n#### SupervisorOptions\n\n```ts\ninterface SupervisorOptions {\n  intervalMs?: number; // default: 60000\n  stuckJobsTimeoutMinutes?: number; // default: 10\n  cleanupJobsDaysToKeep?: number; // default: 30 (0 to disable)\n  cleanupEventsDaysToKeep?: number; // default: 30 (0 to disable)\n  cleanupBatchSize?: number; // default: 1000\n  reclaimStuckJobs?: boolean; // default: true\n  expireTimedOutTokens?: boolean; // default: true\n  onError?: (error: Error) => void; // default: console.error\n  verbose?: boolean;\n}\n```\n\n#### Supervisor\n\n```ts\ninterface Supervisor {\n  start(): Promise<SupervisorRunResult>;\n  startInBackground(): void;\n  stop(): void;\n  stopAndDrain(timeoutMs?: number): Promise<void>;\n  isRunning(): boolean;\n}\n```\n\n- `start()` runs all tasks once and returns the results (serverless-friendly).\n- `startInBackground()` starts a background loop that runs every `intervalMs`.\n- `stopAndDrain()` stops the loop and waits for the current run to finish.\n\n#### SupervisorRunResult\n\n```ts\ninterface SupervisorRunResult {\n  reclaimedJobs: number;\n  cleanedUpJobs: number;\n  cleanedUpEvents: number;\n  expiredTokens: number;\n}\n```\n\nSee [Long-Running Server](/usage/long-running-server#background-supervisor) for usage examples.\n\n---\n\n## Accessing the Underlying Client\n\n### getPool\n\n```ts\ngetPool(): Pool\n```\n\nReturns the PostgreSQL connection pool instance. Only available when using the PostgreSQL backend.\n\n> **Note:** Throws an error if called when using the Redis backend.\n\n### getRedisClient\n\n```ts\ngetRedisClient(): Redis\n```\n\nReturns the `ioredis` client instance. Only available when using the Redis backend.\n\n> **Note:** Throws an error if called when using the PostgreSQL backend."
   },
   {
     "slug": "api/job-record",
@@ -51,7 +51,7 @@
     "slug": "api/processor",
     "title": "Processor",
     "description": "",
-    "content": "The `Processor` interface represents a job processor that can process jobs from the queue, either in the background or synchronously.\n\n## Creating a processor\n\nCreate a processor by calling `createProcessor` on the queue.\n\n```ts\nconst jobQueue = getJobQueue();\nconst processor = queue.createProcessor(handlers, options);\n```\n\n### ProcessorOptions\n\n```ts\ninterface ProcessorOptions {\n  workerId?: string;\n  batchSize?: number;\n  concurrency?: number;\n  pollInterval?: number;\n  onError?: (error: Error) => void;\n  verbose?: boolean;\n  jobType?: string | string[];\n}\n```\n\n## Methods\n\n### startInBackground\n\n```ts\nstartInBackground(): void\n```\n\nStart the job processor in the background. This will run continuously and process jobs as they become available. It polls for new jobs every `pollInterval` milliseconds (default: 5 seconds).\n\n### stop\n\n```ts\nstop(): void\n```\n\nStop the job processor that runs in the background. Does not wait for in-flight jobs to finish.\n\n### stopAndDrain\n\n```ts\nstopAndDrain(timeoutMs?: number): Promise<void>\n```\n\nStop the processor and wait for the current in-flight batch to finish before resolving. Accepts an optional timeout in milliseconds (default: `30000`). If the batch does not complete within the timeout, the promise resolves anyway so your process is not stuck indefinitely. Useful for graceful shutdown (e.g., SIGTERM handling). See [Long-Running Server](/usage/long-running-server) for a full example.\n\n### isRunning\n\n```ts\nisRunning(): boolean\n```\n\nCheck if the job processor is running.\n\n### start\n\n```ts\nstart(): Promise<number>\n```\n\nStart the job processor synchronously. This will process jobs immediately and then stop. Returns the number of jobs processed."
+    "content": "The `Processor` interface represents a job processor that can process jobs from the queue, either in the background or synchronously.\n\n## Creating a processor\n\nCreate a processor by calling `createProcessor` on the queue.\n\n```ts\nconst jobQueue = getJobQueue();\nconst processor = queue.createProcessor(handlers, options);\n```\n\n### ProcessorOptions\n\n```ts\ninterface ProcessorOptions {\n  workerId?: string;\n  batchSize?: number;\n  concurrency?: number;\n  groupConcurrency?: number;\n  pollInterval?: number;\n  onError?: (error: Error) => void;\n  verbose?: boolean;\n  jobType?: string | string[];\n}\n```\n\n- `groupConcurrency` sets a global per-group concurrency cap across all workers/instances for jobs with `group.id`.\n- Must be a positive integer when provided.\n- Jobs without `group.id` are not affected.\n\n## Methods\n\n### startInBackground\n\n```ts\nstartInBackground(): void\n```\n\nStart the job processor in the background. This will run continuously and process jobs as they become available. It polls for new jobs every `pollInterval` milliseconds (default: 5 seconds).\n\n### stop\n\n```ts\nstop(): void\n```\n\nStop the job processor that runs in the background. Does not wait for in-flight jobs to finish.\n\n### stopAndDrain\n\n```ts\nstopAndDrain(timeoutMs?: number): Promise<void>\n```\n\nStop the processor and wait for the current in-flight batch to finish before resolving. Accepts an optional timeout in milliseconds (default: `30000`). If the batch does not complete within the timeout, the promise resolves anyway so your process is not stuck indefinitely. Useful for graceful shutdown (e.g., SIGTERM handling). See [Long-Running Server](/usage/long-running-server) for a full example.\n\n### isRunning\n\n```ts\nisRunning(): boolean\n```\n\nCheck if the job processor is running.\n\n### start\n\n```ts\nstart(): Promise<number>\n```\n\nStart the job processor synchronously. This will process jobs immediately and then stop. Returns the number of jobs processed."
   },
   {
     "slug": "api/tags",
@@ -117,7 +117,7 @@
     "slug": "intro/comparison",
     "title": "Comparison",
     "description": "How DataQueue compares to BullMQ and Trigger.dev",
-    "content": "Choosing a job queue depends on your stack, infrastructure preferences, and the features you need. Here is a side-by-side comparison of **DataQueue**, **BullMQ**, and **Trigger.dev**.\n\n| Feature                 | DataQueue                                       | BullMQ                                      | Trigger.dev                             |\n| ----------------------- | ----------------------------------------------- | ------------------------------------------- | --------------------------------------- |\n| **Backend**             | PostgreSQL or Redis                             | Redis only                                  | Cloud or self-hosted (Postgres + Redis) |\n| **Type Safety**         | Full generic `PayloadMap`                       | Basic types                                 | Full TypeScript tasks                   |\n| **Scheduling**          | `runAt`, Cron                                   | Cron, delayed, recurring                    | Cron, delayed                           |\n| **Retries**             | Exponential backoff, configurable `maxAttempts` | Exponential backoff, custom strategies, DLQ | Auto retries, bulk replay, DLQ          |\n| **Priority**            | Integer priority                                | Priority levels                             | Queue-based priority                    |\n| **Concurrency Control** | `batchSize` + `concurrency`                     | Built-in                                    | Per-task + shared limits                |\n| **Rate Limiting**       | -                                               | Yes                                         | Via concurrency limits                  |\n| **Job Flows / DAGs**    | -                                               | Parent-child flows                          | Workflows                               |\n| **Dashboard**           | Built-in Next.js package                        | Third-party (Bull Board, etc.)              | Built-in web dashboard                  |\n| **Wait / Pause Jobs**   | `waitFor`, `waitUntil`, token system            | -                                           | Durable execution                       |\n| **Human-in-the-Loop**   | Token system                                    | -                                           | Yes                                     |\n| **Progress Tracking**   | Yes (0-100%)                                    | Yes                                         | Yes (realtime)                          |\n| **Serverless-First**    | Yes                                             | No (needs long-running process)             | Yes (cloud)                             |\n| **Self-Hosted**         | Yes                                             | Yes (your Redis)                            | Yes (containers)                        |\n| **Cloud Option**        | -                                               | -                                           | Yes                                     |\n| **License**             | MIT                                             | MIT                                         | Apache-2.0                              |\n| **Pricing**             | Free (OSS)                                      | Free (OSS)                                  | Free tier + paid plans                  |\n| **Infrastructure**      | Your own Postgres or Redis                      | Your own Redis                              | Their cloud or your infra               |\n\n## Where DataQueue shines\n\n- **Serverless-first** — designed from the ground up for Vercel, AWS Lambda, and other serverless platforms. No long-running process required.\n- **Use your existing database** — back your queue with PostgreSQL or Redis. No additional infrastructure to provision or pay for.\n- **Wait and token system** — pause jobs with `waitFor`, `waitUntil`, or token-based waits for human-in-the-loop workflows, all within a single handler function.\n- **Type-safe PayloadMap** — a generic `PayloadMap` gives you compile-time validation of every job type and its payload, catching bugs before they reach production.\n- **Built-in Next.js dashboard** — add a full admin UI to your Next.js app with a single route file. No separate service to deploy."
+    "content": "Choosing a job queue depends on your stack, infrastructure preferences, and the features you need. Here is a side-by-side comparison of **DataQueue**, **BullMQ**, and **Trigger.dev**.\n\n| Feature                 | DataQueue                                               | BullMQ                                      | Trigger.dev                             |\n| ----------------------- | ------------------------------------------------------- | ------------------------------------------- | --------------------------------------- |\n| **Backend**             | PostgreSQL or Redis                                     | Redis only                                  | Cloud or self-hosted (Postgres + Redis) |\n| **Type Safety**         | Full generic `PayloadMap`                               | Basic types                                 | Full TypeScript tasks                   |\n| **Scheduling**          | `runAt`, Cron                                           | Cron, delayed, recurring                    | Cron, delayed                           |\n| **Retries**             | Exponential backoff, configurable `maxAttempts`         | Exponential backoff, custom strategies, DLQ | Auto retries, bulk replay, DLQ          |\n| **Priority**            | Integer priority                                        | Priority levels                             | Queue-based priority                    |\n| **Concurrency Control** | `batchSize` + `concurrency` + global `groupConcurrency` | Built-in                                    | Per-task + shared limits                |\n| **Rate Limiting**       | -                                                       | Yes                                         | Via concurrency limits                  |\n| **Job Flows / DAGs**    | -                                                       | Parent-child flows                          | Workflows                               |\n| **Dashboard**           | Built-in Next.js package                                | Third-party (Bull Board, etc.)              | Built-in web dashboard                  |\n| **Wait / Pause Jobs**   | `waitFor`, `waitUntil`, token system                    | -                                           | Durable execution                       |\n| **Human-in-the-Loop**   | Token system                                            | -                                           | Yes                                     |\n| **Progress Tracking**   | Yes (0-100%)                                            | Yes                                         | Yes (realtime)                          |\n| **Serverless-First**    | Yes                                                     | No (needs long-running process)             | Yes (cloud)                             |\n| **Self-Hosted**         | Yes                                                     | Yes (your Redis)                            | Yes (containers)                        |\n| **Cloud Option**        | -                                                       | -                                           | Yes                                     |\n| **License**             | MIT                                                     | MIT                                         | Apache-2.0                              |\n| **Pricing**             | Free (OSS)                                              | Free (OSS)                                  | Free tier + paid plans                  |\n| **Infrastructure**      | Your own Postgres or Redis                              | Your own Redis                              | Their cloud or your infra               |\n\n## Where DataQueue shines\n\n- **Serverless-first** — designed from the ground up for Vercel, AWS Lambda, and other serverless platforms. No long-running process required.\n- **Use your existing database** — back your queue with PostgreSQL or Redis. No additional infrastructure to provision or pay for.\n- **Wait and token system** — pause jobs with `waitFor`, `waitUntil`, or token-based waits for human-in-the-loop workflows, all within a single handler function.\n- **Type-safe PayloadMap** — a generic `PayloadMap` gives you compile-time validation of every job type and its payload, catching bugs before they reach production.\n- **Built-in Next.js dashboard** — add a full admin UI to your Next.js app with a single route file. No separate service to deploy."
   },
   {
     "slug": "intro",
@@ -249,7 +249,7 @@
     "slug": "usage/process-jobs",
     "title": "Process Jobs",
     "description": "",
-    "content": "So far, we haven't actually performed any jobs—we've only added them to the queue. Now, let's process those jobs.\n\n> **Note:** This page covers processing jobs in a **serverless environment** using\n  cron-triggered API routes. If you're running a long-lived server (Express,\n  Fastify, etc.), see [Long-Running Server](/usage/long-running-server).\n\nIn a serverless environment, we can't have a long-running process that constantly monitors and processes the queue.\n\nInstead, we create an API endpoint that checks the queue and processes jobs in batches. This endpoint is then triggered by a cron job. For example, you can create an API endpoint at `app/api/cron/process` to process jobs in batches:\n\n```typescript title=\"@/app/api/cron/process.ts\"\nimport { jobHandlers } from '@/lib/job-handler';\nimport { getJobQueue } from '@/lib/queue';\nimport { NextResponse } from 'next/server';\n\nexport async function GET(request: Request) {\n  // Secure the cron route: https://vercel.com/docs/cron-jobs/manage-cron-jobs#securing-cron-jobs\n  const authHeader = request.headers.get('authorization');\n  if (authHeader !== `Bearer ${process.env.CRON_SECRET}`) {\n    return NextResponse.json({ message: 'Unauthorized' }, { status: 401 });\n  }\n\n  try {const jobQueue = getJobQueue();\n\n    // Control how many jobs are processed in parallel per batch using the `concurrency` option.\n    // For example, to process up to 3 jobs in parallel per batch:\n    const processor = jobQueue.createProcessor(jobHandlers, {\n      workerId: `cron-${Date.now()}`,\n      batchSize: 10, // up to 10 jobs per batch\n      concurrency: 3, // up to 3 jobs processed in parallel\n      verbose: true,\n    });\n\n    const processed = await processor.start();\n\n    return NextResponse.json({\n      message: 'Job processing completed',\n      processed,\n    });\n  } catch (error) {\n    console.error('Error processing jobs:', error);\n    return NextResponse.json(\n      { message: 'Failed to process jobs' },\n      { status: 500 },\n    );\n  }\n}\n```\n\nIn the example above, we use the `createProcessor` method to create a processor. When you call the processor's `start` function, it processes jobs in the queue up to the `batchSize` limit.\n\n### Batch Size\n\nServerless platforms like Vercel limit how long a function can run. If you set `batchSize` too high, the function might run too long and get killed. Choose a `batchSize` that fits your use case.\n\nYou can also process only certain job types by setting the `jobType` option. If a job type is more resource-intensive, use a lower `batchSize` for that type.\n\nFor example, you can define two endpoints: one for low-resource jobs and another for high-resource jobs, each with different `batchSize` and `concurrency` values.\n\n### Concurrency\n\nSome jobs are resource-intensive, like image processing, LLM calls, or calling a rate-limited external service. In these cases, set the `concurrency` option to control how many jobs run in parallel per batch.\n\nThe default is `3`. Set it to `1` to process jobs one at a time. Use a lower value to avoid exhausting resources in constrained environments.\n\n### Triggering the Processor via Cron\n\nDefining an endpoint isn't enough—you need to trigger it regularly. For example, use Vercel cron to trigger the endpoint every minute by adding this to your `vercel.json`:\n\n```json title=\"vercel.json\"\n{\n  \"$schema\": \"https://openapi.vercel.sh/vercel.json\",\n  \"crons\": [\n    {\n      \"path\": \"/api/cron/process\",\n      \"schedule\": \"* * * * *\"\n    }\n  ]\n}\n```\n\nFor Vercel cron, set the `CRON_SECRET` environment variable, as it's sent in the `authorization` header. If you use a different cron service, set the `authorization` header to the value of `CRON_SECRET`:\n\n```\nAuthorization: Bearer <VALUE_OF_CRON_SECRET>\n```\n\nDuring development, you can create a small script to run the cron job continuously in the background. For example, you can create a `cron.sh` file like [this one](https://github.com/nicnocquee/dataqueue/blob/main/apps/demo/cron.sh), then add it to your `package.json` scripts:\n\n```json title=\"package.json\"\n{\n  \"scripts\": {\n    \"cron\": \"bash cron.sh\"\n  }\n}\n```\n\nThen, you can run the cron job by running `pnpm cron` from the apps/demo directory."
+    "content": "So far, we haven't actually performed any jobs—we've only added them to the queue. Now, let's process those jobs.\n\n> **Note:** This page covers processing jobs in a **serverless environment** using\n  cron-triggered API routes. If you're running a long-lived server (Express,\n  Fastify, etc.), see [Long-Running Server](/usage/long-running-server).\n\nIn a serverless environment, we can't have a long-running process that constantly monitors and processes the queue.\n\nInstead, we create an API endpoint that checks the queue and processes jobs in batches. This endpoint is then triggered by a cron job. For example, you can create an API endpoint at `app/api/cron/process` to process jobs in batches:\n\n```typescript title=\"@/app/api/cron/process.ts\"\nimport { jobHandlers } from '@/lib/job-handler';\nimport { getJobQueue } from '@/lib/queue';\nimport { NextResponse } from 'next/server';\n\nexport async function GET(request: Request) {\n  // Secure the cron route: https://vercel.com/docs/cron-jobs/manage-cron-jobs#securing-cron-jobs\n  const authHeader = request.headers.get('authorization');\n  if (authHeader !== `Bearer ${process.env.CRON_SECRET}`) {\n    return NextResponse.json({ message: 'Unauthorized' }, { status: 401 });\n  }\n\n  try {const jobQueue = getJobQueue();\n\n    // Control how many jobs are processed in parallel per batch using the `concurrency` option.\n    // For example, to process up to 3 jobs in parallel per batch:\n    const processor = jobQueue.createProcessor(jobHandlers, {\n      workerId: `cron-${Date.now()}`,\n      batchSize: 10, // up to 10 jobs per batch\n      concurrency: 3, // up to 3 jobs processed in parallel\n      groupConcurrency: 2, // optional: max 2 concurrent jobs per group.id globally\n      verbose: true,\n    });\n\n    const processed = await processor.start();\n\n    return NextResponse.json({\n      message: 'Job processing completed',\n      processed,\n    });\n  } catch (error) {\n    console.error('Error processing jobs:', error);\n    return NextResponse.json(\n      { message: 'Failed to process jobs' },\n      { status: 500 },\n    );\n  }\n}\n```\n\nIn the example above, we use the `createProcessor` method to create a processor. When you call the processor's `start` function, it processes jobs in the queue up to the `batchSize` limit.\n\n### Batch Size\n\nServerless platforms like Vercel limit how long a function can run. If you set `batchSize` too high, the function might run too long and get killed. Choose a `batchSize` that fits your use case.\n\nYou can also process only certain job types by setting the `jobType` option. If a job type is more resource-intensive, use a lower `batchSize` for that type.\n\nFor example, you can define two endpoints: one for low-resource jobs and another for high-resource jobs, each with different `batchSize` and `concurrency` values.\n\n### Concurrency\n\nSome jobs are resource-intensive, like image processing, LLM calls, or calling a rate-limited external service. In these cases, set the `concurrency` option to control how many jobs run in parallel per batch.\n\nThe default is `3`. Set it to `1` to process jobs one at a time. Use a lower value to avoid exhausting resources in constrained environments.\n\n### Group-Based Concurrency\n\nWhen jobs should be limited per tenant/customer/account across all workers, set `groupConcurrency` on the processor and provide `group.id` when enqueuing jobs:\n\n```typescript\nawait jobQueue.addJob({\n  jobType: 'send_email',\n  payload: { to: 'user@example.com' },\n  group: { id: 'tenant_123' },\n});\n\nconst processor = jobQueue.createProcessor(jobHandlers, {\n  batchSize: 20,\n  concurrency: 10,\n  groupConcurrency: 2,\n});\n```\n\nWith this configuration, at most 2 jobs from the same `group.id` run at once globally, while ungrouped jobs continue to flow normally.\n\n### Triggering the Processor via Cron\n\nDefining an endpoint isn't enough—you need to trigger it regularly. For example, use Vercel cron to trigger the endpoint every minute by adding this to your `vercel.json`:\n\n```json title=\"vercel.json\"\n{\n  \"$schema\": \"https://openapi.vercel.sh/vercel.json\",\n  \"crons\": [\n    {\n      \"path\": \"/api/cron/process\",\n      \"schedule\": \"* * * * *\"\n    }\n  ]\n}\n```\n\nFor Vercel cron, set the `CRON_SECRET` environment variable, as it's sent in the `authorization` header. If you use a different cron service, set the `authorization` header to the value of `CRON_SECRET`:\n\n```\nAuthorization: Bearer <VALUE_OF_CRON_SECRET>\n```\n\nDuring development, you can create a small script to run the cron job continuously in the background. For example, you can create a `cron.sh` file like [this one](https://github.com/nicnocquee/dataqueue/blob/main/apps/demo/cron.sh), then add it to your `package.json` scripts:\n\n```json title=\"package.json\"\n{\n  \"scripts\": {\n    \"cron\": \"bash cron.sh\"\n  }\n}\n```\n\nThen, you can run the cron job by running `pnpm cron` from the apps/demo directory."
   },
   {
     "slug": "usage/progress-tracking",
@@ -279,7 +279,7 @@
     "slug": "usage/scaling",
     "title": "Scaling to Thousands of Jobs",
     "description": "",
-    "content": "DataQueue is designed to handle high-volume workloads out of the box. This page covers how to tune your setup for thousands (or more) of concurrent jobs, whether you're using PostgreSQL or Redis.\n\n## How Throughput Works\n\nWhen a processor runs, it follows this cycle:\n\n1. **Claim** a batch of ready jobs from the database (atomically, using `FOR UPDATE SKIP LOCKED` in PostgreSQL or Lua scripts in Redis).\n2. **Process** up to `concurrency` jobs in parallel from that batch.\n3. **Repeat** immediately if the batch was full, or wait `pollInterval` before checking again.\n\nThe theoretical maximum throughput of a single processor instance is:\n\n```\nthroughput = batchSize / (avgJobDuration + pollInterval)\n```\n\nFor example, with `batchSize: 20`, `concurrency: 10`, `pollInterval: 2000ms`, and jobs averaging 500ms each, a single processor can handle roughly **10 jobs/second** (600/minute).\n\n> **Note:** When a full batch is returned, the processor immediately polls again without\n  waiting. This means it can drain backlogs much faster than the formula\n  suggests during peak load.\n\n## Tuning Processor Settings\n\n### Batch Size\n\n`batchSize` controls how many jobs are claimed per polling cycle.\n\n| Environment                 | Recommended | Why                                    |\n| --------------------------- | ----------- | -------------------------------------- |\n| Serverless (Vercel, Lambda) | 1-10        | Functions have execution time limits   |\n| Long-running server         | 10-50       | Larger batches reduce polling overhead |\n| High-throughput worker      | 50-100      | Maximizes throughput per poll cycle    |\n\n### Concurrency\n\n`concurrency` controls how many jobs from the batch run in parallel.\n\n- **CPU-bound jobs** (image processing, compression): keep concurrency low (1-4) to avoid CPU saturation.\n- **IO-bound jobs** (API calls, email sending): higher concurrency (5-20) works well since jobs spend most time waiting.\n- **Rate-limited APIs**: match concurrency to the API's rate limit to avoid throttling.\n\n### Poll Interval\n\n`pollInterval` controls how often the processor checks for new jobs when idle.\n\n- **1000ms**: Low latency, higher database load. Good for real-time workloads.\n- **5000ms** (default): Balanced. Good for most use cases.\n- **10000-30000ms**: Gentle on the database. Use when latency tolerance is high.\n\n```typescript\nconst processor = jobQueue.createProcessor(jobHandlers, {\n  batchSize: 30,\n  concurrency: 10,\n  pollInterval: 2000,\n});\n```\n\n## Horizontal Scaling with Multiple Workers\n\nDataQueue supports running **multiple processor instances** simultaneously -- on the same server, across multiple servers, or in separate containers. No coordination is needed between workers.\n\n### How It Works\n\n- Each processor gets a unique `workerId` (auto-generated, or set manually).\n- PostgreSQL uses `FOR UPDATE SKIP LOCKED` to ensure no two workers claim the same job.\n- Redis uses atomic Lua scripts for the same guarantee.\n- Workers can safely run on different machines pointing at the same database.\n\n### Example: Multiple Workers\n\n```typescript\n// Worker 1 (e.g., on server A)\nconst processor1 = jobQueue.createProcessor(jobHandlers, {\n  workerId: 'worker-a',\n  batchSize: 20,\n  concurrency: 5,\n});\nprocessor1.startInBackground();\n\n// Worker 2 (e.g., on server B)\nconst processor2 = jobQueue.createProcessor(jobHandlers, {\n  workerId: 'worker-b',\n  batchSize: 20,\n  concurrency: 5,\n});\nprocessor2.startInBackground();\n```\n\n### Specialized Workers\n\nUse the `jobType` filter to create workers dedicated to specific job types. This lets you scale different workloads independently:\n\n```typescript\n// Fast worker for lightweight jobs\nconst emailWorker = jobQueue.createProcessor(jobHandlers, {\n  jobType: 'email',\n  batchSize: 50,\n  concurrency: 20,\n  pollInterval: 1000,\n});\n\n// Slow worker for heavy jobs\nconst reportWorker = jobQueue.createProcessor(jobHandlers, {\n  jobType: 'report',\n  batchSize: 5,\n  concurrency: 1,\n  pollInterval: 5000,\n});\n```\n\n## PostgreSQL Scaling\n\n### Connection Pool\n\nEach processor instance uses database connections from the pool. The default `max` is 10, which works for a single processor. If you run multiple processors in the same process, increase the pool size:\n\n```typescript\nconst jobQueue = initJobQueue({\n  databaseConfig: {\n    connectionString: process.env.DATABASE_URL,\n    max: 20, // increase for multiple processors\n  },\n});\n```\n\n> **Note:** If you run processors on separate servers, each has its own pool. The total\n  connections across all servers should stay within your database's\n  `max_connections` setting (typically 100 for managed databases like Neon or\n  Supabase).\n\n### Table Maintenance\n\nAs completed jobs accumulate, the `job_queue` table grows. This doesn't affect processing speed (claim queries use partial indexes that only cover active jobs), but it does increase storage and slow down full-table queries like `getJobs()`.\n\n**Run cleanup regularly:**\n\n```typescript\n// Delete completed jobs older than 30 days\nawait jobQueue.cleanupOldJobs(30);\n\n// Delete old job events\nawait jobQueue.cleanupOldJobEvents(30);\n```\n\nCleanup operations are batched internally (1000 rows at a time) so they won't lock the table or timeout even with hundreds of thousands of old jobs.\n\n**Run reclaim regularly:**\n\n```typescript\n// Reclaim jobs stuck in 'processing' for more than 10 minutes\nawait jobQueue.reclaimStuckJobs(10);\n```\n\nThis recovers jobs from workers that crashed or timed out.\n\n### Monitoring Table Size\n\nYou can monitor the `job_queue` table size with this query:\n\n```sql\nSELECT\n  pg_size_pretty(pg_total_relation_size('job_queue')) AS total_size,\n  (SELECT count(*) FROM job_queue WHERE status = 'pending') AS pending,\n  (SELECT count(*) FROM job_queue WHERE status = 'processing') AS processing,\n  (SELECT count(*) FROM job_queue WHERE status = 'completed') AS completed,\n  (SELECT count(*) FROM job_queue WHERE status = 'failed') AS failed;\n```\n\n### Performance Indexes\n\nDataQueue includes optimized partial indexes out of the box:\n\n- `idx_job_queue_claimable` -- speeds up job claiming (pending jobs by priority).\n- `idx_job_queue_failed_retry` -- speeds up retry scheduling.\n- `idx_job_queue_stuck` -- speeds up stuck job reclamation.\n- `idx_job_queue_cleanup` -- speeds up cleanup of old completed jobs.\n\nThese are created automatically when you run migrations.\n\n## Redis Scaling\n\n### Memory Usage\n\nRedis stores everything in memory. Estimate memory usage as:\n\n| Jobs    | Approximate Memory |\n| ------- | ------------------ |\n| 1,000   | 1-2 MB             |\n| 10,000  | 10-20 MB           |\n| 100,000 | 100-200 MB         |\n\nThis assumes payloads under 1 KB each. Larger payloads increase memory proportionally.\n\n### Key Best Practices\n\n- **Enable persistence**: Use AOF (Append Only File) for durability. Without it, a Redis restart loses all jobs.\n- **Use `keyPrefix`** to isolate multiple queues in the same Redis instance:\n\n```typescript\nconst jobQueue = initJobQueue({\n  backend: 'redis',\n  redisConfig: {\n    url: process.env.REDIS_URL,\n    keyPrefix: 'myapp:jobs:',\n  },\n});\n```\n\n- **Run cleanup regularly** to free memory from completed jobs. Like PostgreSQL, Redis cleanup is batched internally using cursor-based scanning, so it's safe to run even with a large number of completed jobs.\n\n## Payload Best Practices\n\nKeep payloads small. Store references (IDs, URLs) rather than full data:\n\n```typescript\n// Good: small payload with reference\nawait jobQueue.addJob({\n  jobType: 'processImage',\n  payload: { imageId: 'img_abc123', bucket: 's3://uploads' },\n});\n\n// Avoid: large payload with embedded data\nawait jobQueue.addJob({\n  jobType: 'processImage',\n  payload: { imageData: '<base64 string of 5MB image>' }, // too large\n});\n```\n\nA good target is **under 10 KB per payload**. This keeps database queries fast and Redis memory predictable.\n\n## Example: Processing 10,000 Jobs per Hour\n\nHere's a configuration that can comfortably process ~10,000 jobs per hour, assuming each job takes about 1 second:\n\n```typescript title=\"worker.ts\"\nimport { initJobQueue } from '@nicnocquee/dataqueue';\nimport { jobHandlers } from './job-handlers';\n\nconst jobQueue = initJobQueue({\n  databaseConfig: {\n    connectionString: process.env.DATABASE_URL,\n    max: 15, // room for processor + maintenance queries\n  },\n});\n\nconst processor = jobQueue.createProcessor(jobHandlers, {\n  batchSize: 30,\n  concurrency: 10, // 10 jobs in parallel\n  pollInterval: 2000,\n  onError: (error) => {\n    console.error('Processor error:', error);\n  },\n});\n\nprocessor.startInBackground();\n\n// Maintenance\nsetInterval(\n  async () => {\n    try {\n      await jobQueue.reclaimStuckJobs(10);\n    } catch (e) {\n      console.error(e);\n    }\n  },\n  10 * 60 * 1000,\n);\n\nsetInterval(\n  async () => {\n    try {\n      await jobQueue.cleanupOldJobs(30);\n      await jobQueue.cleanupOldJobEvents(30);\n    } catch (e) {\n      console.error(e);\n    }\n  },\n  24 * 60 * 60 * 1000,\n);\n\n// Graceful shutdown\nprocess.on('SIGTERM', async () => {\n  await processor.stopAndDrain(30000);\n  jobQueue.getPool().end();\n  process.exit(0);\n});\n```\n\nWith 10 concurrent jobs and ~1s each, a single worker handles roughly **10 jobs/second** = **36,000 jobs/hour**. For higher throughput, add more worker instances.\n\n## Quick Reference\n\n| Scale               | Workers | batchSize | concurrency | pollInterval | Notes                            |\n| ------------------- | ------- | --------- | ----------- | ------------ | -------------------------------- |\n| < 100 jobs/hour     | 1       | 10        | 3           | 5000ms       | Default settings work fine       |\n| 100-1,000/hour      | 1       | 20        | 5           | 3000ms       | Single worker is sufficient      |\n| 1,000-10,000/hour   | 1-2     | 30        | 10          | 2000ms       | Add a second worker if needed    |\n| 10,000-100,000/hour | 2-5     | 50        | 15          | 1000ms       | Multiple workers recommended     |\n| 100,000+/hour       | 5+      | 50-100    | 20          | 1000ms       | Specialized workers per job type |"
+    "content": "DataQueue is designed to handle high-volume workloads out of the box. This page covers how to tune your setup for thousands (or more) of concurrent jobs, whether you're using PostgreSQL or Redis.\n\n## How Throughput Works\n\nWhen a processor runs, it follows this cycle:\n\n1. **Claim** a batch of ready jobs from the database (atomically, using `FOR UPDATE SKIP LOCKED` in PostgreSQL or Lua scripts in Redis).\n2. **Process** up to `concurrency` jobs in parallel from that batch.\n3. **Repeat** immediately if the batch was full, or wait `pollInterval` before checking again.\n\nThe theoretical maximum throughput of a single processor instance is:\n\n```\nthroughput = batchSize / (avgJobDuration + pollInterval)\n```\n\nFor example, with `batchSize: 20`, `concurrency: 10`, `pollInterval: 2000ms`, and jobs averaging 500ms each, a single processor can handle roughly **10 jobs/second** (600/minute).\n\n> **Note:** When a full batch is returned, the processor immediately polls again without\n  waiting. This means it can drain backlogs much faster than the formula\n  suggests during peak load.\n\n## Tuning Processor Settings\n\n### Batch Size\n\n`batchSize` controls how many jobs are claimed per polling cycle.\n\n| Environment                 | Recommended | Why                                    |\n| --------------------------- | ----------- | -------------------------------------- |\n| Serverless (Vercel, Lambda) | 1-10        | Functions have execution time limits   |\n| Long-running server         | 10-50       | Larger batches reduce polling overhead |\n| High-throughput worker      | 50-100      | Maximizes throughput per poll cycle    |\n\n### Concurrency\n\n`concurrency` controls how many jobs from the batch run in parallel.\n\n- **CPU-bound jobs** (image processing, compression): keep concurrency low (1-4) to avoid CPU saturation.\n- **IO-bound jobs** (API calls, email sending): higher concurrency (5-20) works well since jobs spend most time waiting.\n- **Rate-limited APIs**: match concurrency to the API's rate limit to avoid throttling.\n\n### Group Concurrency\n\n`groupConcurrency` caps how many jobs with the same `group.id` can be in `processing` at the same time across all worker instances.\n\n- Use this for multi-tenant fairness and per-account rate protection.\n- It is global coordination (PostgreSQL + Redis), not per-process.\n- Ungrouped jobs are unaffected.\n\n```typescript\nawait jobQueue.addJob({\n  jobType: 'send_email',\n  payload: { to: 'user@example.com' },\n  group: { id: 'tenant_abc' },\n});\n\nconst processor = jobQueue.createProcessor(jobHandlers, {\n  batchSize: 30,\n  concurrency: 10,\n  groupConcurrency: 2,\n  pollInterval: 2000,\n});\n```\n\n### Poll Interval\n\n`pollInterval` controls how often the processor checks for new jobs when idle.\n\n- **1000ms**: Low latency, higher database load. Good for real-time workloads.\n- **5000ms** (default): Balanced. Good for most use cases.\n- **10000-30000ms**: Gentle on the database. Use when latency tolerance is high.\n\n```typescript\nconst processor = jobQueue.createProcessor(jobHandlers, {\n  batchSize: 30,\n  concurrency: 10,\n  pollInterval: 2000,\n});\n```\n\n## Horizontal Scaling with Multiple Workers\n\nDataQueue supports running **multiple processor instances** simultaneously -- on the same server, across multiple servers, or in separate containers. No coordination is needed between workers.\n\n### How It Works\n\n- Each processor gets a unique `workerId` (auto-generated, or set manually).\n- PostgreSQL uses `FOR UPDATE SKIP LOCKED` to ensure no two workers claim the same job.\n- Redis uses atomic Lua scripts for the same guarantee.\n- Workers can safely run on different machines pointing at the same database.\n\n### Example: Multiple Workers\n\n```typescript\n// Worker 1 (e.g., on server A)\nconst processor1 = jobQueue.createProcessor(jobHandlers, {\n  workerId: 'worker-a',\n  batchSize: 20,\n  concurrency: 5,\n});\nprocessor1.startInBackground();\n\n// Worker 2 (e.g., on server B)\nconst processor2 = jobQueue.createProcessor(jobHandlers, {\n  workerId: 'worker-b',\n  batchSize: 20,\n  concurrency: 5,\n});\nprocessor2.startInBackground();\n```\n\n### Specialized Workers\n\nUse the `jobType` filter to create workers dedicated to specific job types. This lets you scale different workloads independently:\n\n```typescript\n// Fast worker for lightweight jobs\nconst emailWorker = jobQueue.createProcessor(jobHandlers, {\n  jobType: 'email',\n  batchSize: 50,\n  concurrency: 20,\n  pollInterval: 1000,\n});\n\n// Slow worker for heavy jobs\nconst reportWorker = jobQueue.createProcessor(jobHandlers, {\n  jobType: 'report',\n  batchSize: 5,\n  concurrency: 1,\n  pollInterval: 5000,\n});\n```\n\n## PostgreSQL Scaling\n\n### Connection Pool\n\nEach processor instance uses database connections from the pool. The default `max` is 10, which works for a single processor. If you run multiple processors in the same process, increase the pool size:\n\n```typescript\nconst jobQueue = initJobQueue({\n  databaseConfig: {\n    connectionString: process.env.DATABASE_URL,\n    max: 20, // increase for multiple processors\n  },\n});\n```\n\n> **Note:** If you run processors on separate servers, each has its own pool. The total\n  connections across all servers should stay within your database's\n  `max_connections` setting (typically 100 for managed databases like Neon or\n  Supabase).\n\n### Table Maintenance\n\nAs completed jobs accumulate, the `job_queue` table grows. This doesn't affect processing speed (claim queries use partial indexes that only cover active jobs), but it does increase storage and slow down full-table queries like `getJobs()`.\n\n**Run cleanup regularly:**\n\n```typescript\n// Delete completed jobs older than 30 days\nawait jobQueue.cleanupOldJobs(30);\n\n// Delete old job events\nawait jobQueue.cleanupOldJobEvents(30);\n```\n\nCleanup operations are batched internally (1000 rows at a time) so they won't lock the table or timeout even with hundreds of thousands of old jobs.\n\n**Run reclaim regularly:**\n\n```typescript\n// Reclaim jobs stuck in 'processing' for more than 10 minutes\nawait jobQueue.reclaimStuckJobs(10);\n```\n\nThis recovers jobs from workers that crashed or timed out.\n\n### Monitoring Table Size\n\nYou can monitor the `job_queue` table size with this query:\n\n```sql\nSELECT\n  pg_size_pretty(pg_total_relation_size('job_queue')) AS total_size,\n  (SELECT count(*) FROM job_queue WHERE status = 'pending') AS pending,\n  (SELECT count(*) FROM job_queue WHERE status = 'processing') AS processing,\n  (SELECT count(*) FROM job_queue WHERE status = 'completed') AS completed,\n  (SELECT count(*) FROM job_queue WHERE status = 'failed') AS failed;\n```\n\n### Performance Indexes\n\nDataQueue includes optimized partial indexes out of the box:\n\n- `idx_job_queue_claimable` -- speeds up job claiming (pending jobs by priority).\n- `idx_job_queue_failed_retry` -- speeds up retry scheduling.\n- `idx_job_queue_stuck` -- speeds up stuck job reclamation.\n- `idx_job_queue_cleanup` -- speeds up cleanup of old completed jobs.\n\nThese are created automatically when you run migrations.\n\n## Redis Scaling\n\n### Memory Usage\n\nRedis stores everything in memory. Estimate memory usage as:\n\n| Jobs    | Approximate Memory |\n| ------- | ------------------ |\n| 1,000   | 1-2 MB             |\n| 10,000  | 10-20 MB           |\n| 100,000 | 100-200 MB         |\n\nThis assumes payloads under 1 KB each. Larger payloads increase memory proportionally.\n\n### Key Best Practices\n\n- **Enable persistence**: Use AOF (Append Only File) for durability. Without it, a Redis restart loses all jobs.\n- **Use `keyPrefix`** to isolate multiple queues in the same Redis instance:\n\n```typescript\nconst jobQueue = initJobQueue({\n  backend: 'redis',\n  redisConfig: {\n    url: process.env.REDIS_URL,\n    keyPrefix: 'myapp:jobs:',\n  },\n});\n```\n\n- **Run cleanup regularly** to free memory from completed jobs. Like PostgreSQL, Redis cleanup is batched internally using cursor-based scanning, so it's safe to run even with a large number of completed jobs.\n\n## Payload Best Practices\n\nKeep payloads small. Store references (IDs, URLs) rather than full data:\n\n```typescript\n// Good: small payload with reference\nawait jobQueue.addJob({\n  jobType: 'processImage',\n  payload: { imageId: 'img_abc123', bucket: 's3://uploads' },\n});\n\n// Avoid: large payload with embedded data\nawait jobQueue.addJob({\n  jobType: 'processImage',\n  payload: { imageData: '<base64 string of 5MB image>' }, // too large\n});\n```\n\nA good target is **under 10 KB per payload**. This keeps database queries fast and Redis memory predictable.\n\n## Example: Processing 10,000 Jobs per Hour\n\nHere's a configuration that can comfortably process ~10,000 jobs per hour, assuming each job takes about 1 second:\n\n```typescript title=\"worker.ts\"\nimport { initJobQueue } from '@nicnocquee/dataqueue';\nimport { jobHandlers } from './job-handlers';\n\nconst jobQueue = initJobQueue({\n  databaseConfig: {\n    connectionString: process.env.DATABASE_URL,\n    max: 15, // room for processor + maintenance queries\n  },\n});\n\nconst processor = jobQueue.createProcessor(jobHandlers, {\n  batchSize: 30,\n  concurrency: 10, // 10 jobs in parallel\n  pollInterval: 2000,\n  onError: (error) => {\n    console.error('Processor error:', error);\n  },\n});\n\nprocessor.startInBackground();\n\n// Maintenance\nsetInterval(\n  async () => {\n    try {\n      await jobQueue.reclaimStuckJobs(10);\n    } catch (e) {\n      console.error(e);\n    }\n  },\n  10 * 60 * 1000,\n);\n\nsetInterval(\n  async () => {\n    try {\n      await jobQueue.cleanupOldJobs(30);\n      await jobQueue.cleanupOldJobEvents(30);\n    } catch (e) {\n      console.error(e);\n    }\n  },\n  24 * 60 * 60 * 1000,\n);\n\n// Graceful shutdown\nprocess.on('SIGTERM', async () => {\n  await processor.stopAndDrain(30000);\n  jobQueue.getPool().end();\n  process.exit(0);\n});\n```\n\nWith 10 concurrent jobs and ~1s each, a single worker handles roughly **10 jobs/second** = **36,000 jobs/hour**. For higher throughput, add more worker instances.\n\n## Quick Reference\n\n| Scale               | Workers | batchSize | concurrency | pollInterval | Notes                            |\n| ------------------- | ------- | --------- | ----------- | ------------ | -------------------------------- |\n| < 100 jobs/hour     | 1       | 10        | 3           | 5000ms       | Default settings work fine       |\n| 100-1,000/hour      | 1       | 20        | 5           | 3000ms       | Single worker is sufficient      |\n| 1,000-10,000/hour   | 1-2     | 30        | 10          | 2000ms       | Add a second worker if needed    |\n| 10,000-100,000/hour | 2-5     | 50        | 15          | 1000ms       | Multiple workers recommended     |\n| 100,000+/hour       | 5+      | 50-100    | 20          | 1000ms       | Specialized workers per job type |"
   },
   {
     "slug": "usage/wait",

package/ai/rules/advanced.md

@@ -137,6 +137,7 @@ Events: `job:added`, `job:processing`, `job:completed`, `job:failed` (with `will
 ## Scaling
 
 - Increase `batchSize` and `concurrency` for higher throughput.
+- Use `group: { id }` on jobs with `groupConcurrency` on processors when you need global per-tenant/per-account fairness.
 - Run multiple processor instances with unique `workerId` values — `FOR UPDATE SKIP LOCKED` (PostgreSQL) or Lua scripts (Redis) prevent double-claiming.
 - Use `jobType` filter for specialized workers.
 - Use `createSupervisor()` to automate maintenance (reclaim stuck jobs, cleanup, token expiry). Safe to run across multiple instances.

package/ai/rules/basic.md

@@ -86,7 +86,7 @@ const ids = await queue.addJobs([
 // ids[i] corresponds to the i-th input job
 ```
 
-Both support `idempotencyKey`, `priority`, `runAt`, `tags`, and `{ db }` for transactional inserts (PostgreSQL only).
+Both support `idempotencyKey`, `priority`, `runAt`, `tags`, optional `group: { id, tier? }`, and `{ db }` for transactional inserts (PostgreSQL only).
 
 ## Handlers
 
@@ -113,6 +113,7 @@ Handler signature: `(payload: T, signal: AbortSignal, ctx: JobContext) => Promis
 const processor = queue.createProcessor(handlers, {
   batchSize: 10,
   concurrency: 3,
+  groupConcurrency: 2, // optional global cap per group.id
 });
 await processor.start();
 ```

package/ai/skills/dataqueue-advanced/SKILL.md

@@ -239,6 +239,28 @@ await queue.cancelAllUpcomingJobs({
 
 Tag query modes: `'exact'`, `'all'`, `'any'`, `'none'`.
 
+## Group-Based Concurrency
+
+Use job `group.id` plus processor `groupConcurrency` to enforce a global cap per group across all workers/instances (PostgreSQL and Redis).
+
+```typescript
+await queue.addJob({
+  jobType: 'email',
+  payload: {
+    /* ... */
+  },
+  group: { id: 'tenant_abc', tier: 'gold' }, // tier is optional/reserved
+});
+
+const processor = queue.createProcessor(handlers, {
+  batchSize: 20,
+  concurrency: 10,
+  groupConcurrency: 2,
+});
+```
+
+Ungrouped jobs are unaffected by `groupConcurrency`.
+
 ## Idempotency
 
 ```typescript

package/ai/skills/dataqueue-core/SKILL.md

@@ -114,6 +114,7 @@ const jobId = await queue.addJob({
   runAt: new Date(Date.now() + 5000),
   tags: ['welcome'],
   idempotencyKey: 'welcome-user-123',
+  group: { id: 'tenant_123' }, // optional: for global per-group concurrency limits
 });
 ```
 
@@ -190,6 +191,7 @@ await queue.addJob({
 const processor = queue.createProcessor(handlers, {
   batchSize: 10,
   concurrency: 3,
+  groupConcurrency: 2, // optional global cap per group.id across all workers
 });
 const processed = await processor.start();
 ```