@nicnocquee/dataqueue 1.24.0 → 1.26.0-beta.20260223195940

package/ai/rules/advanced.md

@@ -0,0 +1,132 @@
+# DataQueue — Advanced Rules
+
+## Step Memoization (ctx.run)
+
+Wrap side-effectful work in `ctx.run(stepName, fn)` for durability. Cached results replay on re-invocation after a wait.
+
+```typescript
+const data = await ctx.run('fetch', async () => fetchFromAPI(url));
+await ctx.waitFor({ hours: 1 });
+await ctx.run('notify', async () => sendNotification(data));
+```
+
+Step names must be unique within a handler and stable across deployments.
+
+## Waits
+
+- `ctx.waitFor({ hours: 24 })` — pause for a duration (seconds, minutes, hours, days, weeks, months, years).
+- `ctx.waitUntil(date)` — pause until a specific date.
+- `ctx.waitForToken(tokenId)` — pause until an external actor completes the token.
+
+Waiting jobs release their worker lock and concurrency slot. They consume no resources.
+
+Wait calls use a positional counter internally. Do not add/remove waits conditionally between re-invocations.
+
+## Token System
+
+```typescript
+const token = await ctx.createToken({ timeout: '48h', tags: ['approval'] });
+const result = await ctx.waitForToken<{ approved: boolean }>(token.id);
+if (result.ok) {
+  /* result.output.approved */
+}
+```
+
+Complete externally: `await queue.completeToken(tokenId, { approved: true })`.
+Expire timed-out tokens: `await queue.expireTimedOutTokens()`.
+
+## Cron Scheduling
+
+```typescript
+await queue.addCronJob({
+  scheduleName: 'daily-cleanup',
+  cronExpression: '0 2 * * *',
+  jobType: 'cleanup',
+  payload: { days: 30 },
+  timezone: 'UTC',
+  allowOverlap: false,
+});
+```
+
+The processor auto-enqueues due cron jobs before each batch. Manage with `pauseCronJob`, `resumeCronJob`, `editCronJob`, `removeCronJob`, `listCronJobs`.
+
+## Timeout Management
+
+- `ctx.prolong(ms)` — proactively reset deadline. `ctx.prolong()` resets to original `timeoutMs`.
+- `ctx.onTimeout(() => ms)` — reactive; return ms to extend, or nothing to let timeout proceed.
+- `forceKillOnTimeout: true` — terminates handler via Worker Thread. Requires Node.js, serializable handler, and disables `ctx.run`/waits/`prolong`/`onTimeout`.
+
+## Tags and Filtering
+
+```typescript
+await queue.addJob({ jobType: 'email', payload, tags: ['welcome', 'user'] });
+const jobs = await queue.getJobsByTags(['welcome'], 'any');
+await queue.cancelAllUpcomingJobs({ tags: { values: ['user'], mode: 'all' } });
+```
+
+Modes: `exact` (exact set), `all` (superset), `any` (intersection), `none` (exclusion).
+
+## Idempotency
+
+```typescript
+await queue.addJob({
+  jobType: 'email',
+  payload,
+  idempotencyKey: `welcome-${userId}`,
+});
+```
+
+Returns existing job ID if key already exists. Key persists until `cleanupOldJobs` removes the job.
+
+## Transactional Job Creation (PostgreSQL Only)
+
+Pass a `pg.PoolClient` inside a transaction via the `{ db }` option to enqueue a job atomically with other writes:
+
+```typescript
+const client = await pool.connect();
+await client.query('BEGIN');
+await client.query('INSERT INTO users (email) VALUES ($1)', [email]);
+await queue.addJob(
+  {
+    jobType: 'send_email',
+    payload: { to: email, subject: 'Welcome!', body: '...' },
+  },
+  { db: client },
+);
+await client.query('COMMIT');
+client.release();
+```
+
+If the transaction rolls back, the job and its event are never persisted. The `db` option accepts any object with a `.query(text, values)` method matching `pg`'s signature. Using `{ db }` with the Redis backend throws an error.
+
+## Retry Strategy
+
+```typescript
+await queue.addJob({
+  jobType: 'email',
+  payload,
+  retryDelay: 10, // base 10s
+  retryBackoff: true, // exponential (default)
+  retryDelayMax: 300, // cap at 5 min
+});
+```
+
+- `retryBackoff: false` — fixed delay of `retryDelay` seconds.
+- `retryBackoff: true` (default) — `retryDelay * 2^attempts` with jitter, capped by `retryDelayMax`.
+- No config — legacy `2^attempts * 60s` formula (backward compatible).
+- Cron schedules propagate retry config to enqueued jobs.
+
+## Scaling
+
+- Increase `batchSize` and `concurrency` for higher throughput.
+- Run multiple processor instances with unique `workerId` values — `FOR UPDATE SKIP LOCKED` (PostgreSQL) or Lua scripts (Redis) prevent double-claiming.
+- Use `jobType` filter for specialized workers.
+- Use `createSupervisor()` to automate maintenance (reclaim stuck jobs, cleanup, token expiry). Safe to run across multiple instances.
+
+## Progress Tracking
+
+```typescript
+await ctx.setProgress(50); // 0–100, persisted to DB
+```
+
+Read via `queue.getJob(id)` (`progress` field) or React SDK's `useJob` hook.

package/ai/rules/basic.md

@@ -0,0 +1,159 @@
+# DataQueue — Basic Rules
+
+## Imports
+
+Always import from `@nicnocquee/dataqueue`. There is no subpath like `/v2` or `/v3`.
+
+```typescript
+import { initJobQueue, JobHandlers } from '@nicnocquee/dataqueue';
+```
+
+## PayloadMap Pattern
+
+Define an object type where keys are job type strings and values are payload shapes. This powers type-safe `addJob`, `createProcessor`, and handler completeness checking.
+
+```typescript
+type JobPayloadMap = {
+  send_email: { to: string; subject: string; body: string };
+  generate_report: { reportId: string; userId: string };
+};
+```
+
+## Initialization (Singleton)
+
+Never call `initJobQueue` per request — each call creates a new database connection pool. Use a module-level singleton:
+
+```typescript
+import { initJobQueue } from '@nicnocquee/dataqueue';
+
+let jobQueue: ReturnType<typeof initJobQueue<JobPayloadMap>> | null = null;
+
+export const getJobQueue = () => {
+  if (!jobQueue) {
+    jobQueue = initJobQueue<JobPayloadMap>({
+      databaseConfig: { connectionString: process.env.PG_DATAQUEUE_DATABASE },
+    });
+  }
+  return jobQueue;
+};
+```
+
+For Redis, set `backend: 'redis'` and use `redisConfig` with `url` or `host`/`port`/`password`. Install `ioredis` as a peer dependency.
+
+### Bring Your Own Pool / Client
+
+Pass an existing `pg.Pool` or `ioredis` client instead of connection config:
+
+```typescript
+import { Pool } from 'pg';
+const pool = new Pool({ connectionString: process.env.DATABASE_URL });
+jobQueue = initJobQueue<JobPayloadMap>({ pool });
+```
+
+```typescript
+import IORedis from 'ioredis';
+const redis = new IORedis(process.env.REDIS_URL);
+jobQueue = initJobQueue<JobPayloadMap>({
+  backend: 'redis',
+  client: redis,
+  keyPrefix: 'myapp:',
+});
+```
+
+The library will **not** close externally provided connections on shutdown.
+
+## Adding Jobs
+
+Use `addJob` for a single job, `addJobs` for bulk inserts (single DB round-trip).
+
+```typescript
+const id = await queue.addJob({
+  jobType: 'send_email',
+  payload: { to: 'a@x.com', subject: 'Hi', body: '...' },
+});
+
+const ids = await queue.addJobs([
+  {
+    jobType: 'send_email',
+    payload: { to: 'a@x.com', subject: 'Hi', body: '...' },
+  },
+  {
+    jobType: 'send_email',
+    payload: { to: 'b@x.com', subject: 'Hi', body: '...' },
+    priority: 10,
+  },
+]);
+// ids[i] corresponds to the i-th input job
+```
+
+Both support `idempotencyKey`, `priority`, `runAt`, `tags`, and `{ db }` for transactional inserts (PostgreSQL only).
+
+## Handlers
+
+Type handlers as `JobHandlers<PayloadMap>` so TypeScript enforces a handler for every job type.
+
+```typescript
+export const jobHandlers: JobHandlers<JobPayloadMap> = {
+  send_email: async (payload, signal, ctx) => {
+    await sendEmail(payload.to, payload.subject, payload.body);
+  },
+  generate_report: async (payload) => {
+    await generateReport(payload.reportId, payload.userId);
+  },
+};
+```
+
+Handler signature: `(payload: T, signal: AbortSignal, ctx: JobContext) => Promise<void>`. You can omit arguments you don't need.
+
+## Processing
+
+**Serverless** — call `processor.start()` which processes one batch and stops:
+
+```typescript
+const processor = queue.createProcessor(handlers, {
+  batchSize: 10,
+  concurrency: 3,
+});
+await processor.start();
+```
+
+**Long-running** — call `processor.startInBackground()` which polls continuously, and `createSupervisor()` to automate maintenance:
+
+```typescript
+processor.startInBackground();
+
+const supervisor = queue.createSupervisor({
+  intervalMs: 60_000,
+  stuckJobsTimeoutMinutes: 10,
+  cleanupJobsDaysToKeep: 30,
+});
+supervisor.startInBackground();
+
+process.on('SIGTERM', async () => {
+  await Promise.all([
+    processor.stopAndDrain(30000),
+    supervisor.stopAndDrain(30000),
+  ]);
+  queue.getPool().end(); // or queue.getRedisClient().quit() for Redis
+  process.exit(0);
+});
+```
+
+## Retry Configuration
+
+Control retry behavior per-job with optional fields on `addJob`:
+
+- `retryDelay` (seconds, default 60) — base delay between retries.
+- `retryBackoff` (boolean, default true) — enable exponential backoff with jitter.
+- `retryDelayMax` (seconds, optional) — cap the maximum delay.
+
+When none are set, the legacy `2^attempts * 60s` formula is used.
+
+## Common Mistakes
+
+1. Creating `initJobQueue` per request — use a singleton.
+2. Missing handler for a job type — fails with `NoHandler`. Type as `JobHandlers<PayloadMap>`.
+3. Not checking `signal.aborted` in long handlers — timed-out jobs keep running.
+4. Skipping maintenance — use `createSupervisor()` to automate reclaim, cleanup, and token expiry. Without it, stuck jobs and old data accumulate.
+5. Skipping migrations (PostgreSQL) — run `dataqueue-cli migrate` first. Redis needs none.
+6. Using `stop()` instead of `stopAndDrain()` — leaves in-flight jobs stuck.

package/ai/rules/react-dashboard.md

@@ -0,0 +1,83 @@
+# DataQueue — React & Dashboard Rules
+
+## React SDK (@nicnocquee/dataqueue-react)
+
+Install: `npm install @nicnocquee/dataqueue-react` (requires React 18+).
+
+### useJob Hook
+
+```tsx
+'use client';
+import { useJob } from '@nicnocquee/dataqueue-react';
+
+const { status, progress, data, isLoading, error } = useJob(jobId, {
+  fetcher: (id) =>
+    fetch(`/api/jobs/${id}`)
+      .then((r) => r.json())
+      .then((d) => d.job),
+  pollingInterval: 1000,
+  onComplete: (job) => {
+    /* job completed */
+  },
+  onFailed: (job) => {
+    /* job failed */
+  },
+});
+```
+
+Polling auto-stops on terminal statuses (`completed`, `failed`, `cancelled`).
+
+### DataqueueProvider
+
+Wrap app in `DataqueueProvider` to share `fetcher` and `pollingInterval`:
+
+```tsx
+<DataqueueProvider fetcher={fetcher} pollingInterval={2000}>
+  {children}
+</DataqueueProvider>
+```
+
+### API Route (Next.js)
+
+```typescript
+// app/api/jobs/[id]/route.ts
+export async function GET(
+  _req: Request,
+  { params }: { params: Promise<{ id: string }> },
+) {
+  const { id } = await params;
+  const job = await getJobQueue().getJob(Number(id));
+  if (!job) return NextResponse.json({ error: 'Not found' }, { status: 404 });
+  return NextResponse.json({ job });
+}
+```
+
+## Dashboard (@nicnocquee/dataqueue-dashboard)
+
+Install: `npm install @nicnocquee/dataqueue-dashboard`.
+
+### Setup (Next.js App Router)
+
+```typescript
+// app/admin/dataqueue/[[...path]]/route.ts
+import { createDataqueueDashboard } from '@nicnocquee/dataqueue-dashboard/next';
+import { getJobQueue, jobHandlers } from '@/lib/queue';
+
+const { GET, POST } = createDataqueueDashboard({
+  jobQueue: getJobQueue(),
+  jobHandlers,
+  basePath: '/admin/dataqueue',
+});
+
+export { GET, POST };
+```
+
+`basePath` must match the route directory path.
+
+### Protection
+
+Wrap handlers with your auth middleware before exporting GET/POST.
+
+## Progress Tracking
+
+Use `ctx.setProgress(percent)` in handlers (0–100). The value appears in `useJob`'s `progress` field and the dashboard detail view.

package/ai/skills/dataqueue-advanced/SKILL.md

@@ -0,0 +1,320 @@
+---
+name: dataqueue-advanced
+description: Advanced DataQueue patterns — step memoization, waits, tokens, cron, timeouts, tags, idempotency.
+---
+
+# DataQueue Advanced Patterns
+
+## Step Memoization with ctx.run()
+
+Wrap side-effectful work in `ctx.run(stepName, fn)`. Results are cached in the database — when the handler re-runs after a wait, completed steps replay from cache without re-executing.
+
+```typescript
+const handler = async (payload, signal, ctx) => {
+  const data = await ctx.run('fetch-data', async () => {
+    return await fetchFromAPI(payload.url);
+  });
+
+  await ctx.run('send-notification', async () => {
+    await notify(data.userId, data.message);
+  });
+};
+```
+
+**Rules:**
+
+- Step names must be unique within a handler.
+- Step names must be stable across deployments while jobs are waiting.
+- Step order must not change conditionally between re-invocations.
+
+## Time-Based Waits
+
+### waitFor (duration)
+
+```typescript
+const handler = async (payload, signal, ctx) => {
+  await ctx.run('step-1', async () => {
+    /* ... */
+  });
+  await ctx.waitFor({ hours: 24 });
+  await ctx.run('step-2', async () => {
+    /* ... */
+  });
+};
+```
+
+Duration fields: `seconds`, `minutes`, `hours`, `days`, `weeks`, `months`, `years` (additive).
+
+### waitUntil (date)
+
+```typescript
+await ctx.waitUntil(new Date('2025-03-01T09:00:00Z'));
+```
+
+### How waits work internally
+
+1. Handler throws a `WaitSignal` internally.
+2. Job moves to `'waiting'` status — worker lock is released.
+3. After the wait expires, job becomes `'pending'` again.
+4. Handler re-runs from top; `ctx.run()` replays cached steps.
+
+Waiting jobs are idle — they hold no lock, no concurrency slot, no resources.
+
+## Token-Based Waits (Human-in-the-Loop)
+
+Create a token, send it to an external actor, and wait for them to complete it.
+
+```typescript
+const handler = async (payload, signal, ctx) => {
+  const token = await ctx.run('create-token', async () => {
+    return await ctx.createToken({ timeout: '48h', tags: ['approval'] });
+  });
+
+  await ctx.run('notify', async () => {
+    await sendSlack(`Approve: ${token.id}`);
+  });
+
+  const result = await ctx.waitForToken<{ approved: boolean }>(token.id);
+  if (result.ok) {
+    await ctx.run('process', async () => {
+      if (result.output.approved) await approve(payload.id);
+    });
+  }
+};
+```
+
+Complete tokens externally:
+
+```typescript
+await queue.completeToken(tokenId, { approved: true });
+```
+
+Expire timed-out tokens periodically:
+
+```typescript
+await queue.expireTimedOutTokens();
+```
+
+## Cron Scheduling
+
+```typescript
+const cronId = await queue.addCronJob({
+  scheduleName: 'daily-report',
+  cronExpression: '0 9 * * *',
+  jobType: 'generate_report',
+  payload: { reportId: 'daily', userId: 'system' },
+  timezone: 'America/New_York',
+  allowOverlap: false,
+});
+```
+
+The processor automatically enqueues due cron jobs before each batch — no manual triggering needed.
+
+Manage schedules:
+
+```typescript
+await queue.pauseCronJob(cronId);
+await queue.resumeCronJob(cronId);
+await queue.editCronJob(cronId, { cronExpression: '0 */2 * * *' });
+await queue.removeCronJob(cronId);
+const schedules = await queue.listCronJobs('active');
+```
+
+## Timeout Management
+
+### Proactive — ctx.prolong()
+
+```typescript
+const handler = async (payload, signal, ctx) => {
+  ctx.prolong(60_000); // set deadline to 60s from now
+  await doHeavyWork();
+  ctx.prolong(); // reset to original timeoutMs
+};
+```
+
+### Reactive — ctx.onTimeout()
+
+```typescript
+const handler = async (payload, signal, ctx) => {
+  let step = 0;
+  ctx.onTimeout(() => {
+    if (step < 3) return 30_000; // extend 30s
+  });
+  step = 1;
+  await doStep1();
+  step = 2;
+  await doStep2();
+  step = 3;
+  await doStep3();
+};
+```
+
+Both update `locked_at` in the DB, preventing premature reclamation.
+
+### Force Kill on Timeout
+
+```typescript
+await queue.addJob({
+  jobType: 'task',
+  payload: {
+    /* ... */
+  },
+  timeoutMs: 5000,
+  forceKillOnTimeout: true,
+});
+```
+
+**Limitations of forceKillOnTimeout:**
+
+- Requires Node.js (not Bun).
+- Handler must be serializable (no closures over external variables).
+- `prolong`, `onTimeout`, `ctx.run`, waits are NOT available.
+
+## Tags
+
+```typescript
+await queue.addJob({
+  jobType: 'email',
+  payload: {
+    /* ... */
+  },
+  tags: ['welcome', 'onboarding'],
+});
+
+const jobs = await queue.getJobsByTags(['welcome'], 'any');
+await queue.cancelAllUpcomingJobs({
+  tags: { values: ['onboarding'], mode: 'all' },
+});
+```
+
+Tag query modes: `'exact'`, `'all'`, `'any'`, `'none'`.
+
+## Idempotency
+
+```typescript
+const jobId = await queue.addJob({
+  jobType: 'email',
+  payload: { to: 'user@example.com', subject: 'Welcome', body: '...' },
+  idempotencyKey: `welcome-${userId}`,
+});
+```
+
+If a job with the same key exists, returns the existing job ID. Key is unique across all statuses until `cleanupOldJobs` removes it.
+
+## Transactional Job Creation (PostgreSQL Only)
+
+Insert a job within an existing database transaction so the job is enqueued **atomically** with other writes:
+
+```typescript
+import { Pool } from 'pg';
+
+const pool = new Pool({ connectionString: process.env.DATABASE_URL });
+
+async function registerUser(email: string, name: string) {
+  const client = await pool.connect();
+  try {
+    await client.query('BEGIN');
+
+    await client.query('INSERT INTO users (email, name) VALUES ($1, $2)', [
+      email,
+      name,
+    ]);
+
+    const queue = getJobQueue();
+    await queue.addJob(
+      {
+        jobType: 'send_email',
+        payload: { to: email, subject: 'Welcome!', body: `Hi ${name}!` },
+      },
+      { db: client },
+    );
+
+    await client.query('COMMIT');
+  } catch (error) {
+    await client.query('ROLLBACK');
+    throw error;
+  } finally {
+    client.release();
+  }
+}
+```
+
+The `db` option accepts any object matching `DatabaseClient { query(text, values): Promise<{ rows, rowCount }> }` — works with `pg.PoolClient`, `pg.Client`, or compatible ORM query runners.
+
+The job event (`'added'`) is also inserted within the same transaction.
+
+## Retry Strategy
+
+Configure how failed jobs are retried with `retryDelay`, `retryBackoff`, and `retryDelayMax`.
+
+### Fixed delay
+
+```typescript
+await queue.addJob({
+  jobType: 'email',
+  payload: {
+    /* ... */
+  },
+  maxAttempts: 5,
+  retryDelay: 30, // 30 seconds between each retry
+  retryBackoff: false,
+});
+```
+
+### Exponential backoff with cap
+
+```typescript
+await queue.addJob({
+  jobType: 'email',
+  payload: {
+    /* ... */
+  },
+  maxAttempts: 10,
+  retryDelay: 5, // base: 5 seconds
+  retryBackoff: true, // default — delay doubles each attempt with jitter
+  retryDelayMax: 300, // never wait more than 5 minutes
+});
+```
+
+### Cron schedules with retry config
+
+```typescript
+await queue.addCronJob({
+  scheduleName: 'daily-sync',
+  cronExpression: '0 * * * *',
+  jobType: 'sync',
+  payload: { source: 'api' },
+  retryDelay: 60,
+  retryBackoff: true,
+  retryDelayMax: 600,
+});
+```
+
+Every job enqueued by the schedule inherits the retry settings.
+
+### Default behavior
+
+When no retry options are set, the legacy formula `2^attempts * 60 seconds` is used. This is fully backward compatible.
+
+## Maintenance
+
+Use `createSupervisor()` to automate all maintenance tasks in a long-running server:
+
+```typescript
+const supervisor = queue.createSupervisor({
+  intervalMs: 60_000,
+  stuckJobsTimeoutMinutes: 10,
+  cleanupJobsDaysToKeep: 30,
+  cleanupEventsDaysToKeep: 30,
+});
+supervisor.startInBackground();
+```
+
+For serverless or one-off scripts, call `supervisor.start()` (runs once) or use the manual methods:
+
+```typescript
+await queue.reclaimStuckJobs(10); // reclaim jobs stuck > 10 min
+await queue.cleanupOldJobs(30); // delete completed jobs > 30 days
+await queue.cleanupOldJobEvents(30); // delete old events > 30 days
+await queue.expireTimedOutTokens(); // expire overdue tokens
+```