@nicnocquee/dataqueue 1.34.0 → 1.35.0-beta.20260224110011

package/ai/rules/advanced.md

@@ -78,12 +78,69 @@ await queue.addJob({
 
 Returns existing job ID if key already exists. Key persists until `cleanupOldJobs` removes the job.
 
+## Transactional Job Creation (PostgreSQL Only)
+
+Pass a `pg.PoolClient` inside a transaction via the `{ db }` option to enqueue a job atomically with other writes:
+
+```typescript
+const client = await pool.connect();
+await client.query('BEGIN');
+await client.query('INSERT INTO users (email) VALUES ($1)', [email]);
+await queue.addJob(
+  {
+    jobType: 'send_email',
+    payload: { to: email, subject: 'Welcome!', body: '...' },
+  },
+  { db: client },
+);
+await client.query('COMMIT');
+client.release();
+```
+
+If the transaction rolls back, the job and its event are never persisted. The `db` option accepts any object with a `.query(text, values)` method matching `pg`'s signature. Using `{ db }` with the Redis backend throws an error.
+
+## Retry Strategy
+
+```typescript
+await queue.addJob({
+  jobType: 'email',
+  payload,
+  retryDelay: 10, // base 10s
+  retryBackoff: true, // exponential (default)
+  retryDelayMax: 300, // cap at 5 min
+});
+```
+
+- `retryBackoff: false` — fixed delay of `retryDelay` seconds.
+- `retryBackoff: true` (default) — `retryDelay * 2^attempts` with jitter, capped by `retryDelayMax`.
+- No config — legacy `2^attempts * 60s` formula (backward compatible).
+- Cron schedules propagate retry config to enqueued jobs.
+
+## Event Hooks
+
+Subscribe to real-time lifecycle events via `on`, `once`, `off`, `removeAllListeners`. Works with both Postgres and Redis.
+
+```typescript
+queue.on('job:completed', ({ jobId, jobType }) => {
+  metrics.increment('job.completed', { jobType });
+});
+queue.on('job:failed', ({ jobId, jobType, error, willRetry }) => {
+  if (!willRetry) alertOps(`Permanent failure: ${jobId}`);
+});
+queue.on('error', (error) => Sentry.captureException(error));
+```
+
+Events: `job:added`, `job:processing`, `job:completed`, `job:failed` (with `willRetry`), `job:cancelled`, `job:retried`, `job:waiting`, `job:progress`, `job:output`, `error`.
+
+`error` events fire alongside `onError` callbacks in `ProcessorOptions` / `SupervisorOptions` — both mechanisms work independently.
+
 ## Scaling
 
 - Increase `batchSize` and `concurrency` for higher throughput.
+- Use `group: { id }` on jobs with `groupConcurrency` on processors when you need global per-tenant/per-account fairness.
 - Run multiple processor instances with unique `workerId` values — `FOR UPDATE SKIP LOCKED` (PostgreSQL) or Lua scripts (Redis) prevent double-claiming.
 - Use `jobType` filter for specialized workers.
-- Call `cleanupOldJobs` and `reclaimStuckJobs` on intervals.
+- Use `createSupervisor()` to automate maintenance (reclaim stuck jobs, cleanup, token expiry). Safe to run across multiple instances.
 
 ## Progress Tracking
 
@@ -92,3 +149,23 @@ await ctx.setProgress(50); // 0–100, persisted to DB
 ```
 
 Read via `queue.getJob(id)` (`progress` field) or React SDK's `useJob` hook.
+
+## Job Output
+
+Store results via `ctx.setOutput(data)` or by returning a value from the handler:
+
+```typescript
+// Option 1: return a value
+const handler = async (payload, signal, ctx) => {
+  const result = await doWork(payload);
+  return { url: result.downloadUrl };
+};
+
+// Option 2: ctx.setOutput (takes precedence over return value)
+const handler = async (payload, signal, ctx) => {
+  const result = await doWork(payload);
+  await ctx.setOutput({ url: result.downloadUrl });
+};
+```
+
+Read via `queue.getJob(id)` (`output` field) or React SDK's `useJob` hook (`output` property).

package/ai/rules/basic.md

@@ -40,6 +40,54 @@ export const getJobQueue = () => {
 
 For Redis, set `backend: 'redis'` and use `redisConfig` with `url` or `host`/`port`/`password`. Install `ioredis` as a peer dependency.
 
+### Bring Your Own Pool / Client
+
+Pass an existing `pg.Pool` or `ioredis` client instead of connection config:
+
+```typescript
+import { Pool } from 'pg';
+const pool = new Pool({ connectionString: process.env.DATABASE_URL });
+jobQueue = initJobQueue<JobPayloadMap>({ pool });
+```
+
+```typescript
+import IORedis from 'ioredis';
+const redis = new IORedis(process.env.REDIS_URL);
+jobQueue = initJobQueue<JobPayloadMap>({
+  backend: 'redis',
+  client: redis,
+  keyPrefix: 'myapp:',
+});
+```
+
+The library will **not** close externally provided connections on shutdown.
+
+## Adding Jobs
+
+Use `addJob` for a single job, `addJobs` for bulk inserts (single DB round-trip).
+
+```typescript
+const id = await queue.addJob({
+  jobType: 'send_email',
+  payload: { to: 'a@x.com', subject: 'Hi', body: '...' },
+});
+
+const ids = await queue.addJobs([
+  {
+    jobType: 'send_email',
+    payload: { to: 'a@x.com', subject: 'Hi', body: '...' },
+  },
+  {
+    jobType: 'send_email',
+    payload: { to: 'b@x.com', subject: 'Hi', body: '...' },
+    priority: 10,
+  },
+]);
+// ids[i] corresponds to the i-th input job
+```
+
+Both support `idempotencyKey`, `priority`, `runAt`, `tags`, optional `group: { id, tier? }`, and `{ db }` for transactional inserts (PostgreSQL only).
+
 ## Handlers
 
 Type handlers as `JobHandlers<PayloadMap>` so TypeScript enforces a handler for every job type.
@@ -65,26 +113,48 @@ Handler signature: `(payload: T, signal: AbortSignal, ctx: JobContext) => Promis
 const processor = queue.createProcessor(handlers, {
   batchSize: 10,
   concurrency: 3,
+  groupConcurrency: 2, // optional global cap per group.id
 });
 await processor.start();
 ```
 
-**Long-running** — call `processor.startInBackground()` which polls continuously:
+**Long-running** — call `processor.startInBackground()` which polls continuously, and `createSupervisor()` to automate maintenance:
 
 ```typescript
 processor.startInBackground();
+
+const supervisor = queue.createSupervisor({
+  intervalMs: 60_000,
+  stuckJobsTimeoutMinutes: 10,
+  cleanupJobsDaysToKeep: 30,
+});
+supervisor.startInBackground();
+
 process.on('SIGTERM', async () => {
-  await processor.stopAndDrain(30000);
+  await Promise.all([
+    processor.stopAndDrain(30000),
+    supervisor.stopAndDrain(30000),
+  ]);
   queue.getPool().end(); // or queue.getRedisClient().quit() for Redis
   process.exit(0);
 });
 ```
 
+## Retry Configuration
+
+Control retry behavior per-job with optional fields on `addJob`:
+
+- `retryDelay` (seconds, default 60) — base delay between retries.
+- `retryBackoff` (boolean, default true) — enable exponential backoff with jitter.
+- `retryDelayMax` (seconds, optional) — cap the maximum delay.
+
+When none are set, the legacy `2^attempts * 60s` formula is used.
+
 ## Common Mistakes
 
 1. Creating `initJobQueue` per request — use a singleton.
 2. Missing handler for a job type — fails with `NoHandler`. Type as `JobHandlers<PayloadMap>`.
 3. Not checking `signal.aborted` in long handlers — timed-out jobs keep running.
-4. Forgetting `reclaimStuckJobs()` — crashed workers leave jobs stuck.
+4. Skipping maintenance — use `createSupervisor()` to automate reclaim, cleanup, and token expiry. Without it, stuck jobs and old data accumulate.
 5. Skipping migrations (PostgreSQL) — run `dataqueue-cli migrate` first. Redis needs none.
 6. Using `stop()` instead of `stopAndDrain()` — leaves in-flight jobs stuck.

package/ai/rules/react-dashboard.md

@@ -10,7 +10,7 @@ Install: `npm install @nicnocquee/dataqueue-react` (requires React 18+).
 'use client';
 import { useJob } from '@nicnocquee/dataqueue-react';
 
-const { status, progress, data, isLoading, error } = useJob(jobId, {
+const { status, progress, output, data, isLoading, error } = useJob(jobId, {
   fetcher: (id) =>
     fetch(`/api/jobs/${id}`)
       .then((r) => r.json())
@@ -81,3 +81,7 @@ Wrap handlers with your auth middleware before exporting GET/POST.
 ## Progress Tracking
 
 Use `ctx.setProgress(percent)` in handlers (0–100). The value appears in `useJob`'s `progress` field and the dashboard detail view.
+
+## Job Output
+
+Store results via `ctx.setOutput(data)` or by returning a value from the handler. The value appears in `useJob`'s `output` field and the dashboard detail view. If both are used, `ctx.setOutput()` takes precedence.

package/ai/skills/dataqueue-advanced/SKILL.md

@@ -170,6 +170,56 @@ await queue.addJob({
 - Handler must be serializable (no closures over external variables).
 - `prolong`, `onTimeout`, `ctx.run`, waits are NOT available.
 
+## Event Hooks
+
+Subscribe to real-time job lifecycle events. Works identically with PostgreSQL and Redis.
+
+```typescript
+const queue = initJobQueue<MyPayloadMap>(config);
+
+queue.on('job:completed', ({ jobId, jobType }) => {
+  console.log(`Job ${jobId} (${jobType}) completed`);
+});
+
+queue.on('job:failed', ({ jobId, jobType, error, willRetry }) => {
+  console.error(`Job ${jobId} failed: ${error.message}`);
+  if (!willRetry) {
+    alertOps(`Permanent failure for job ${jobId}`);
+  }
+});
+
+queue.on('error', (error) => {
+  Sentry.captureException(error);
+});
+```
+
+### Available events
+
+| Event            | Payload                                |
+| ---------------- | -------------------------------------- |
+| `job:added`      | `{ jobId, jobType }`                   |
+| `job:processing` | `{ jobId, jobType }`                   |
+| `job:completed`  | `{ jobId, jobType }`                   |
+| `job:failed`     | `{ jobId, jobType, error, willRetry }` |
+| `job:cancelled`  | `{ jobId }`                            |
+| `job:retried`    | `{ jobId }`                            |
+| `job:waiting`    | `{ jobId, jobType }`                   |
+| `job:progress`   | `{ jobId, progress }`                  |
+| `error`          | `Error`                                |
+
+### Listener management
+
+```typescript
+const listener = ({ jobId }) => console.log(jobId);
+queue.on('job:completed', listener);
+queue.off('job:completed', listener);
+queue.once('job:added', ({ jobId }) => console.log('First job:', jobId));
+queue.removeAllListeners('job:completed');
+queue.removeAllListeners(); // all events
+```
+
+The `error` event fires alongside `onError` callbacks in `ProcessorOptions` and `SupervisorOptions` -- both mechanisms work independently.
+
 ## Tags
 
 ```typescript
@@ -189,6 +239,28 @@ await queue.cancelAllUpcomingJobs({
 
 Tag query modes: `'exact'`, `'all'`, `'any'`, `'none'`.
 
+## Group-Based Concurrency
+
+Use job `group.id` plus processor `groupConcurrency` to enforce a global cap per group across all workers/instances (PostgreSQL and Redis).
+
+```typescript
+await queue.addJob({
+  jobType: 'email',
+  payload: {
+    /* ... */
+  },
+  group: { id: 'tenant_abc', tier: 'gold' }, // tier is optional/reserved
+});
+
+const processor = queue.createProcessor(handlers, {
+  batchSize: 20,
+  concurrency: 10,
+  groupConcurrency: 2,
+});
+```
+
+Ungrouped jobs are unaffected by `groupConcurrency`.
+
 ## Idempotency
 
 ```typescript
@@ -201,8 +273,117 @@ const jobId = await queue.addJob({
 
 If a job with the same key exists, returns the existing job ID. Key is unique across all statuses until `cleanupOldJobs` removes it.
 
+## Transactional Job Creation (PostgreSQL Only)
+
+Insert a job within an existing database transaction so the job is enqueued **atomically** with other writes:
+
+```typescript
+import { Pool } from 'pg';
+
+const pool = new Pool({ connectionString: process.env.DATABASE_URL });
+
+async function registerUser(email: string, name: string) {
+  const client = await pool.connect();
+  try {
+    await client.query('BEGIN');
+
+    await client.query('INSERT INTO users (email, name) VALUES ($1, $2)', [
+      email,
+      name,
+    ]);
+
+    const queue = getJobQueue();
+    await queue.addJob(
+      {
+        jobType: 'send_email',
+        payload: { to: email, subject: 'Welcome!', body: `Hi ${name}!` },
+      },
+      { db: client },
+    );
+
+    await client.query('COMMIT');
+  } catch (error) {
+    await client.query('ROLLBACK');
+    throw error;
+  } finally {
+    client.release();
+  }
+}
+```
+
+The `db` option accepts any object matching `DatabaseClient { query(text, values): Promise<{ rows, rowCount }> }` — works with `pg.PoolClient`, `pg.Client`, or compatible ORM query runners.
+
+The job event (`'added'`) is also inserted within the same transaction.
+
+## Retry Strategy
+
+Configure how failed jobs are retried with `retryDelay`, `retryBackoff`, and `retryDelayMax`.
+
+### Fixed delay
+
+```typescript
+await queue.addJob({
+  jobType: 'email',
+  payload: {
+    /* ... */
+  },
+  maxAttempts: 5,
+  retryDelay: 30, // 30 seconds between each retry
+  retryBackoff: false,
+});
+```
+
+### Exponential backoff with cap
+
+```typescript
+await queue.addJob({
+  jobType: 'email',
+  payload: {
+    /* ... */
+  },
+  maxAttempts: 10,
+  retryDelay: 5, // base: 5 seconds
+  retryBackoff: true, // default — delay doubles each attempt with jitter
+  retryDelayMax: 300, // never wait more than 5 minutes
+});
+```
+
+### Cron schedules with retry config
+
+```typescript
+await queue.addCronJob({
+  scheduleName: 'daily-sync',
+  cronExpression: '0 * * * *',
+  jobType: 'sync',
+  payload: { source: 'api' },
+  retryDelay: 60,
+  retryBackoff: true,
+  retryDelayMax: 600,
+});
+```
+
+Every job enqueued by the schedule inherits the retry settings.
+
+### Default behavior
+
+When no retry options are set, the legacy formula `2^attempts * 60 seconds` is used. This is fully backward compatible.
+
 ## Maintenance
 
+Use `createSupervisor()` to automate all maintenance tasks in a long-running server:
+
+```typescript
+const supervisor = queue.createSupervisor({
+  intervalMs: 60_000,
+  stuckJobsTimeoutMinutes: 10,
+  cleanupJobsDaysToKeep: 30,
+  cleanupEventsDaysToKeep: 30,
+});
+supervisor.startInBackground();
+```
+
+For serverless or one-off scripts, call `supervisor.start()` (runs once) or use the manual methods:
+
 ```typescript
 await queue.reclaimStuckJobs(10); // reclaim jobs stuck > 10 min
 await queue.cleanupOldJobs(30); // delete completed jobs > 30 days

package/ai/skills/dataqueue-core/SKILL.md

@@ -38,7 +38,8 @@ export const jobHandlers: JobHandlers<JobPayloadMap> = {
   },
   generate_report: async (payload, signal) => {
     if (signal.aborted) return;
-    await generateReport(payload.reportId, payload.userId);
+    const url = await generateReport(payload.reportId, payload.userId);
+    return { url }; // stored as job output, readable via getJob()
   },
 };
 ```
@@ -79,6 +80,30 @@ jobQueue = initJobQueue<JobPayloadMap>({
 });
 ```
 
+### Bring Your Own Pool / Client
+
+You can pass an existing `pg.Pool` or `ioredis` client instead of connection config:
+
+```typescript
+import { Pool } from 'pg';
+const pool = new Pool({ connectionString: process.env.DATABASE_URL });
+
+jobQueue = initJobQueue<JobPayloadMap>({ pool });
+```
+
+```typescript
+import IORedis from 'ioredis';
+const redis = new IORedis(process.env.REDIS_URL);
+
+jobQueue = initJobQueue<JobPayloadMap>({
+  backend: 'redis',
+  client: redis,
+  keyPrefix: 'myapp:',
+});
+```
+
+When you provide your own pool/client, the library will **not** close it on shutdown — you manage its lifecycle.
+
 ## Step 4: Add Jobs
 
 ```typescript
@@ -89,9 +114,75 @@ const jobId = await queue.addJob({
   runAt: new Date(Date.now() + 5000),
   tags: ['welcome'],
   idempotencyKey: 'welcome-user-123',
+  group: { id: 'tenant_123' }, // optional: for global per-group concurrency limits
 });
 ```
 
+### Batch Insert
+
+Use `addJobs` to insert many jobs in a single database round-trip. Returns IDs in the same order as the input array.
+
+```typescript
+const jobIds = await queue.addJobs([
+  {
+    jobType: 'send_email',
+    payload: { to: 'a@example.com', subject: 'Hi', body: '...' },
+  },
+  {
+    jobType: 'send_email',
+    payload: { to: 'b@example.com', subject: 'Hi', body: '...' },
+    priority: 10,
+  },
+  {
+    jobType: 'generate_report',
+    payload: { reportId: '1', userId: '2' },
+    tags: ['monthly'],
+  },
+]);
+```
+
+Each job can independently have its own `idempotencyKey`, `priority`, `runAt`, `tags`, etc. The `{ db }` transactional option is also supported (PostgreSQL only).
+
+### Transactional Job Creation (PostgreSQL only)
+
+Pass an external `pg.PoolClient` inside a transaction via `{ db: client }`:
+
+```typescript
+const client = await pool.connect();
+await client.query('BEGIN');
+await client.query('INSERT INTO users (email) VALUES ($1)', [email]);
+await queue.addJob(
+  {
+    jobType: 'send_email',
+    payload: { to: email, subject: 'Welcome!', body: '...' },
+  },
+  { db: client },
+);
+await client.query('COMMIT');
+client.release();
+```
+
+If the transaction rolls back, the job is never enqueued.
+
+### Retry configuration
+
+Control retry behavior per-job with `retryDelay`, `retryBackoff`, and `retryDelayMax`:
+
+```typescript
+await queue.addJob({
+  jobType: 'send_email',
+  payload: { to: 'user@example.com', subject: 'Hi', body: 'Hello' },
+  maxAttempts: 5,
+  retryDelay: 10, // base delay: 10 seconds
+  retryBackoff: true, // exponential backoff (default)
+  retryDelayMax: 300, // cap at 5 minutes
+});
+```
+
+- **Fixed delay**: set `retryBackoff: false` for constant delay between retries.
+- **Exponential backoff** (default): delay doubles each attempt with jitter.
+- **Default**: when no retry options are set, legacy `2^attempts * 60s` is used.
+
 ## Step 5: Process Jobs
 
 ### Serverless (one-shot)
@@ -100,6 +191,7 @@ const jobId = await queue.addJob({
 const processor = queue.createProcessor(handlers, {
   batchSize: 10,
   concurrency: 3,
+  groupConcurrency: 2, // optional global cap per group.id across all workers
 });
 const processed = await processor.start();
 ```
@@ -114,8 +206,20 @@ const processor = queue.createProcessor(handlers, {
 });
 processor.startInBackground();
 
+// Automate maintenance (reclaim stuck jobs, cleanup old data, expire tokens)
+const supervisor = queue.createSupervisor({
+  intervalMs: 60_000,
+  stuckJobsTimeoutMinutes: 10,
+  cleanupJobsDaysToKeep: 30,
+  cleanupEventsDaysToKeep: 30,
+});
+supervisor.startInBackground();
+
 process.on('SIGTERM', async () => {
-  await processor.stopAndDrain(30000);
+  await Promise.all([
+    processor.stopAndDrain(30000),
+    supervisor.stopAndDrain(30000),
+  ]);
   queue.getPool().end();
   process.exit(0);
 });
@@ -126,6 +230,8 @@ process.on('SIGTERM', async () => {
 1. **Creating a new queue per request** — always use a singleton. Each `initJobQueue` creates a DB pool.
 2. **Missing handler for a job type** — the job fails with `FailureReason.NoHandler`. Let TypeScript enforce completeness by typing handlers as `JobHandlers<PayloadMap>`.
 3. **Not checking `signal.aborted`** — timed-out jobs keep running in the background. Always check the signal in long-running handlers.
-4. **Forgetting `reclaimStuckJobs`** — crashed workers leave jobs stuck in `processing`. Call `reclaimStuckJobs()` periodically.
+4. **Skipping maintenance** — use `createSupervisor()` to automate reclaiming stuck jobs, cleaning up old data, and expiring tokens. Without it, crashed workers leave jobs stuck in `processing` and tables grow unbounded.
 5. **Forgetting to run migrations** — PostgreSQL requires `dataqueue-cli migrate` before use. Redis needs no migrations.
 6. **Not calling `stopAndDrain` on shutdown** — use `stopAndDrain()` (not `stop()`) for graceful shutdown to avoid stuck jobs.
+7. **Forgetting to commit/rollback when using `db` option** — the `addJob` INSERT sits in an open transaction. If you never `COMMIT` or `ROLLBACK`, the connection leaks and the job is invisible to other sessions.
+8. **Using `db` option with Redis** — transactional job creation is PostgreSQL only. The Redis backend throws if `db` is provided.

package/ai/skills/dataqueue-react/SKILL.md

@@ -102,13 +102,14 @@ export async function GET(
 
 ### useJob Return Value
 
-| Field       | Type                | Description                     |
-| ----------- | ------------------- | ------------------------------- |
-| `data`      | `JobData \| null`   | Latest job data from fetcher    |
-| `status`    | `JobStatus \| null` | Current job status              |
-| `progress`  | `number \| null`    | Progress percentage (0–100)     |
-| `isLoading` | `boolean`           | True until first fetch resolves |
-| `error`     | `Error \| null`     | Last fetch error                |
+| Field       | Type                | Description                                           |
+| ----------- | ------------------- | ----------------------------------------------------- |
+| `data`      | `JobData \| null`   | Latest job data from fetcher                          |
+| `status`    | `JobStatus \| null` | Current job status                                    |
+| `progress`  | `number \| null`    | Progress percentage (0–100)                           |
+| `output`    | `unknown \| null`   | Handler output from `ctx.setOutput()` or return value |
+| `isLoading` | `boolean`           | True until first fetch resolves                       |
+| `error`     | `Error \| null`     | Last fetch error                                      |
 
 ## Dashboard — @nicnocquee/dataqueue-dashboard
 
@@ -187,3 +188,14 @@ const handler = async (payload, signal, ctx) => {
   }
 };
 ```
+
+### Job Output from Handlers
+
+Store results via `ctx.setOutput(data)` or by returning a value from the handler. Exposed via `getJob()` (`output` field) and the `useJob` hook's `output` property. If both are used, `ctx.setOutput()` takes precedence.
+
+```typescript
+const handler = async (payload, signal, ctx) => {
+  const result = await doWork(payload);
+  return { url: result.downloadUrl }; // stored as output
+};
+```