@boopkit/cap-background-jobs 0.1.0

package/capability.json

@@ -0,0 +1,39 @@
+{
+  "name": "background-jobs",
+  "description": "Background job processing with RabbitMQ via CloudAMQP",
+  "type": "application",
+  "package": "@boopkit/cap-background-jobs",
+  "requires": ["services"],
+  "dependencies": ["amqplib@^0.10"],
+  "devDependencies": ["@types/amqplib@^0.10"],
+  "env": [
+    { "name": "CLOUDAMQP_URL", "description": "CloudAMQP connection URL (amqp://...)", "required": "boot" },
+    { "name": "QUEUE_PREFIX", "description": "Queue name prefix for environment isolation (e.g. dev/mike)" },
+    { "name": "MAX_MESSAGE_RETRIES", "description": "Maximum retry attempts before dead-lettering (default: 5)" }
+  ],
+  "scripts": {
+    "worker": { "cmd": "ts-node src/worker.ts", "description": "Start the background job worker process" },
+    "scheduler": { "cmd": "ts-node src/scheduler.ts", "description": "Publish trigger messages to job queues (run via Heroku Scheduler)" }
+  },
+  "files": [
+    "src/scheduler.ts",
+    "src/worker.ts",
+    "src/services/message-queue.service.ts",
+    "src/controllers/example-job.controller.ts",
+    "tests/services/message-queue.service.test.ts",
+    "tests/controllers/example-job.controller.test.ts",
+    "reference/background-jobs.ref.md"
+  ],
+  "skills": ["design.background-jobs.skill.md"],
+  "hooks": [],
+  "postInstall": {
+    "prerequisites": ["CloudAMQP addon provisioned (or CLOUDAMQP_URL set)"],
+    "steps": [
+      "Add CLOUDAMQP_URL to .env",
+      "Set QUEUE_PREFIX for local dev isolation (e.g. dev/yourname)",
+      "Add worker entry to Procfile: worker: npm run worker",
+      "Configure Heroku Scheduler to run npm run scheduler on desired interval"
+    ],
+    "verify": "npm run worker"
+  }
+}

package/package.json

@@ -0,0 +1,33 @@
+{
+  "name": "@boopkit/cap-background-jobs",
+  "version": "0.1.0",
+  "description": "Background job processing with RabbitMQ for boopkit-scaffolded projects",
+  "main": "dist/index.js",
+  "types": "dist/index.d.ts",
+  "files": [
+    "dist/",
+    "scaffold/",
+    "skills/",
+    "capability.json"
+  ],
+  "boopkit": {
+    "capability": true,
+    "scaffoldDir": "scaffold",
+    "skillsDir": "skills",
+    "manifest": "capability.json"
+  },
+  "scripts": {
+    "build": "tsc -p tsconfig.json"
+  },
+  "dependencies": {
+    "amqplib": "^0.10.0"
+  },
+  "devDependencies": {
+    "@types/amqplib": "^0.10.0",
+    "typescript": "^5.9.3"
+  },
+  "license": "ISC",
+  "publishConfig": {
+    "access": "public"
+  }
+}

package/scaffold/reference/background-jobs.ref.md

@@ -0,0 +1,160 @@
+# Background Jobs Infrastructure
+
+Async job processing with RabbitMQ via CloudAMQP: publish messages from any process, consume them in a dedicated worker with retry and dead-letter handling.
+
+**Related skills**: [design.background-jobs.skill.md](../skills/design.background-jobs.skill.md)
+
+---
+
+## Process Model
+
+```
+Web (src/server.ts)          Scheduler (src/scheduler.ts)
+  │ publish()                   │ publish() → exit
+  ▼                             ▼
+┌─────────────────────────────────────┐
+│         CloudAMQP (RabbitMQ)        │
+│  ┌──────────┐ ┌────────┐ ┌───────┐ │
+│  │ main     │ │ .retry │ │.failed│ │
+│  │ queue    │ │ queue  │ │ (DLQ) │ │
+│  └────┬─────┘ └───┬────┘ └───────┘ │
+└───────┼────────────┼────────────────┘
+        ▼            │
+Worker (src/worker.ts)
+  │ listen() → controller.onMessage()
+  │   true  → ACK
+  │   false → retry queue (backoff TTL) ──┘
+  │   throw → retry or DLQ after max retries
+```
+
+Three separate OS processes share one MQ service:
+
+| Process | Entry point | Role | Lifecycle |
+|---------|------------|------|-----------|
+| **Web** | `src/server.ts` | Publishes messages as a side effect of HTTP requests | Long-running |
+| **Worker** | `src/worker.ts` | Consumes messages via registered controllers | Long-running |
+| **Scheduler** | `src/scheduler.ts` | Publishes trigger messages, then exits | One-shot (cron) |
+
+---
+
+## Queue Topology
+
+Each queue registered via `listen()` creates three RabbitMQ queues:
+
+| Queue | Name pattern | Purpose |
+|-------|-------------|---------|
+| Main | `{QUEUE_PREFIX}/{name}` | Messages waiting for consumption |
+| Retry | `{QUEUE_PREFIX}/{name}.retry` | Failed messages waiting for TTL-based redelivery |
+| DLQ | `{QUEUE_PREFIX}/{name}.failed` | Messages that exceeded max retries |
+
+Each queue has a corresponding direct exchange for routing. Retry messages dead-letter back to the main exchange after TTL expiry.
+
+**Retry behavior:** Exponential backoff starting at 4s, doubling per retry, capped at 60s. Retry count tracked in `x-retry-count` header. After `MAX_MESSAGE_RETRIES` (default 5), messages move to the DLQ.
+
+---
+
+## Connection Lifecycle
+
+- **Publish connection**: Lazy — created on first `publish()` call, reused across subsequent calls. Auto-reconnects on error by clearing the cached reference.
+- **Listener connections**: One per `listen()` call — created at registration time. Each gets its own channel.
+- **Shutdown**: `close()` tears down all connections (publish + listeners).
+
+The web process only opens an AMQP connection if it actually publishes a message.
+
+---
+
+## Consumer Registration
+
+The worker registers consumers via a static array in `worker.ts`:
+
+```typescript
+import * as exampleJobController from './controllers/example-job.controller';
+
+const consumers = [
+  { queue: exampleJobController.QUEUE_NAME, handler: exampleJobController.onMessage },
+];
+```
+
+To add a new consumer: write a controller, add an entry to this array.
+
+---
+
+## ACK Modes
+
+### Auto-ACK (default)
+
+Handler returns a boolean:
+
+```typescript
+export async function onMessage(msg: unknown): Promise<boolean> {
+  // return true  → ACK (success or permanent failure)
+  // return false → retry with backoff
+  // throw        → retry, then DLQ after max retries
+}
+```
+
+### Manual-ACK
+
+Handler receives guarded `ack()`/`nack()` functions:
+
+```typescript
+await MessageQueueService.listen('queue', async (msg, ack, nack) => {
+  // ack()  → ACK the message
+  // nack() → retry with backoff / DLQ
+  // First call wins — subsequent calls are no-ops
+  // Resolving without calling either → treated as failure
+}, { manualAck: true });
+```
+
+### Prefetch
+
+Limits concurrent message delivery per consumer for backpressure control:
+
+```typescript
+await MessageQueueService.listen('queue', handler, { prefetch: 10 });
+```
+
+---
+
+## Environment Configuration
+
+| Variable | Required | Description |
+|----------|----------|-------------|
+| `CLOUDAMQP_URL` | Yes (boot) | CloudAMQP connection URL (`amqp://...`) |
+| `QUEUE_PREFIX` | No | Queue name prefix for environment isolation (e.g. `dev/mike`) |
+| `MAX_MESSAGE_RETRIES` | No | Max retry attempts before dead-lettering (default: 5) |
+
+---
+
+## Heroku Setup
+
+1. **CloudAMQP addon**: `heroku addons:create cloudamqp` — sets `CLOUDAMQP_URL` automatically
+2. **Procfile**: Add `worker: npm run worker`
+3. **Heroku Scheduler**: Configure `npm run scheduler` on the desired interval
+4. **Scale worker**: `heroku ps:scale worker=1`
+
+---
+
+## Graceful Shutdown
+
+The worker handles SIGINT/SIGTERM:
+
+1. Stop accepting new messages
+2. Drain in-flight messages (30s timeout — matches Heroku's SIGTERM grace period)
+3. Close all AMQP connections
+4. `process.exit(0)`
+
+If drain exceeds 30s, the worker force-exits with code 1.
+
+---
+
+## Key Files
+
+| File | Purpose |
+|------|---------|
+| `src/worker.ts` | Worker entry point — boots services, registers consumers, handles shutdown |
+| `src/scheduler.ts` | Scheduler entry point — publishes trigger messages, exits |
+| `src/services/message-queue.service.ts` | MQ service — publish, listen (auto/manual ACK), topology, retry/DLQ |
+| `src/controllers/example-job.controller.ts` | Example consumer — copyable template for new job handlers |
+| `tests/services/message-queue.service.test.ts` | MQ service unit tests |
+| `tests/controllers/example-job.controller.test.ts` | Example controller unit tests |

package/scaffold/src/controllers/example-job.controller.ts

@@ -0,0 +1,64 @@
+/**
+ * Example Job Controller — copyable template for MQ consumer handlers.
+ *
+ * Demonstrates the auto-ACK pattern: parse payload, call a service,
+ * return true (ACK) or false (retry). Copy this file and replace the
+ * handler body with your business logic.
+ *
+ * Error classification:
+ *   return true  → ACK (success, or permanent failure — don't retry)
+ *   return false → retry with exponential backoff
+ *   uncaught throw → retry, then DLQ after max retries
+ */
+
+/** Queue name constant — used by worker.ts to register this consumer. */
+export const QUEUE_NAME = 'example-job';
+
+/**
+ * Handler for 'example-job' queue messages (auto-ACK mode).
+ *
+ * @param msg - Parsed message payload (JSON already deserialized by MQ service)
+ * @returns true to ACK, false to retry
+ */
+export async function onMessage(msg: unknown): Promise<boolean> {
+  // --- Parse: validate the payload shape ---
+  let payload: { task: string; data?: Record<string, unknown> };
+  try {
+    payload = validatePayload(msg);
+  } catch (err) {
+    // Poison message — bad shape, retrying won't help
+    console.error('[example-job] invalid payload — ACKing to prevent retry loop', {
+      error: (err as Error).message,
+    });
+    return true;
+  }
+
+  // --- Call: delegate to a service ---
+  try {
+    // Replace this with your actual service call, e.g.:
+    //   await services.myService.process(payload.task, payload.data);
+    console.log('[example-job] processed:', payload.task);
+    return true;
+  } catch (err) {
+    // Transient error — retry with backoff
+    console.error('[example-job] processing failed — retrying', {
+      error: (err as Error).message,
+    });
+    return false;
+  }
+}
+
+// ---------------------------------------------------------------------------
+// Internal helpers
+// ---------------------------------------------------------------------------
+
+function validatePayload(msg: unknown): { task: string; data?: Record<string, unknown> } {
+  if (typeof msg !== 'object' || msg === null || !('task' in msg)) {
+    throw new Error('payload must be an object with a "task" field');
+  }
+  const obj = msg as Record<string, unknown>;
+  if (typeof obj.task !== 'string') {
+    throw new Error('"task" must be a string');
+  }
+  return obj as { task: string; data?: Record<string, unknown> };
+}

package/scaffold/src/scheduler.ts

@@ -0,0 +1,39 @@
+/** @boopkit/cap-background-jobs — Scheduler entry point for triggering background jobs. */
+
+import { loadServices } from '@services';
+import MessageQueueService from './services/message-queue.service';
+
+// Static job registration — add new {queue, payload} entries here
+const jobs = [
+  {
+    queue: 'example-job',
+    payload: {
+      task: 'scheduled-run',
+      data: {
+        triggeredBy: 'scheduler',
+        triggeredAt: new Date().toISOString(),
+      },
+    },
+  },
+];
+
+async function run() {
+  await loadServices();
+
+  for (const job of jobs) {
+    const ok = await MessageQueueService.publish(job.queue, job.payload);
+    if (ok) {
+      console.log(`[scheduler] published to "${job.queue}": ${job.payload.task}`);
+    } else {
+      console.error(`[scheduler] failed to publish to "${job.queue}"`);
+    }
+  }
+
+  await MessageQueueService.close();
+  process.exit(0);
+}
+
+run().catch((err) => {
+  console.error('[scheduler] failed:', err);
+  process.exit(1);
+});

package/scaffold/src/services/message-queue.service.ts

@@ -0,0 +1,316 @@
+/** @boopkit/cap-background-jobs — RabbitMQ message queue service. */
+
+import amqplib from 'amqplib';
+import type { ChannelModel, Channel, ConsumeMessage } from 'amqplib';
+
+const QUEUE_PREFIX = process.env.QUEUE_PREFIX ?? '';
+const MAX_RETRIES = parseInt(process.env.MAX_MESSAGE_RETRIES ?? '5', 10);
+const BACKOFF_BASE_MS = 4000;
+const BACKOFF_MAX_MS = 60000;
+
+/** All open connections tracked for shutdown cleanup. */
+const connections: ChannelModel[] = [];
+
+/** Lazy-initialized publish connection and channel. */
+let publishConnection: ChannelModel | null = null;
+let publishChannel: Channel | null = null;
+
+const MessageQueueService = {
+  name: 'message-queue',
+  init,
+  publish,
+  listen,
+  close,
+};
+
+export default MessageQueueService;
+
+/**
+ * Validate that CLOUDAMQP_URL is set. Does NOT open a connection —
+ * publish connection is lazy (first publish call), listener connections
+ * are created at listen() time.
+ */
+function init(): void {
+  if (!process.env.CLOUDAMQP_URL) {
+    throw new Error(
+      'E: CLOUDAMQP_URL is not set. ' +
+      'R: The message queue service requires a CloudAMQP connection URL. ' +
+      'R: Set CLOUDAMQP_URL in your environment (e.g. .env file).',
+    );
+  }
+}
+
+/**
+ * Publish a message to the given queue.
+ * Lazy-initializes a persistent connection on first call, reuses it
+ * across subsequent calls. Returns true on success, false on failure.
+ */
+async function publish(queue: string, payload: unknown): Promise<boolean> {
+  const queueName = prefixedName(queue);
+
+  try {
+    const channel = await getPublishChannel();
+    await channel.assertQueue(queueName, { durable: true });
+
+    const body = Buffer.from(JSON.stringify(payload));
+    const ok = channel.sendToQueue(queueName, body, {
+      persistent: true,
+      messageId: crypto.randomUUID(),
+    });
+
+    if (!ok) {
+      // Channel buffer full — wait for drain
+      await new Promise<void>((resolve) => channel.once('drain', resolve));
+    }
+
+    return true;
+  } catch (err) {
+    console.error(`[message-queue] publish error on "${queue}":`, err);
+    resetPublishConnection();
+    return false;
+  }
+}
+
+/** Options for listen(). */
+interface ListenOpts {
+  /** When true, handler receives guarded ack/nack functions instead of returning boolean. */
+  manualAck?: boolean;
+  /** Channel prefetch count — limits concurrent message delivery for backpressure. */
+  prefetch?: number;
+}
+
+/**
+ * Listen on a queue with auto-ACK or manual-ACK semantics.
+ *
+ * Asserts full queue topology on registration:
+ *   - Main queue: `{prefix}/{name}`
+ *   - Retry queue: `{prefix}/{name}.retry` (TTL redelivery to main)
+ *   - DLQ: `{prefix}/{name}.failed` (dead-letter exchange)
+ *
+ * **Auto-ACK (default):** Handler returns true → ACK, false → retry with backoff,
+ * throws → NACK to DLQ after max retries.
+ *
+ * **Manual-ACK (opts.manualAck = true):** Handler receives guarded ack()/nack()
+ * functions. First call wins — subsequent calls are no-ops. If handler resolves
+ * or throws without settling → retry/DLQ after max retries.
+ */
+// eslint-disable-next-line @typescript-eslint/no-explicit-any
+async function listen(queue: string, handler: (...args: any[]) => Promise<any>, opts?: ListenOpts): Promise<void> {
+  const connection = await amqplib.connect(process.env.CLOUDAMQP_URL!);
+  connections.push(connection);
+
+  const channel = await connection.createChannel();
+  const { mainQueue, retryQueue, failedQueue } = await assertTopology(channel, queue);
+
+  if (opts?.prefetch != null) {
+    await channel.prefetch(opts.prefetch);
+  }
+
+  await channel.consume(mainQueue, async (raw) => {
+    if (!raw) return;
+
+    const retryCount = getRetryCount(raw);
+    let parsed: unknown;
+
+    try {
+      parsed = JSON.parse(raw.content.toString());
+    } catch {
+      // Poison message — ACK to prevent infinite redelivery
+      console.error(`[message-queue] unparseable message on "${queue}", sending to DLQ`);
+      channel.nack(raw, false, false);
+      return;
+    }
+
+    if (opts?.manualAck) {
+      // Manual-ACK: handler receives guarded ack/nack
+      let settled = false;
+
+      const guardedAck = (): void => {
+        if (settled) {
+          console.warn(`[message-queue] duplicate ack on "${queue}" — ignored`);
+          return;
+        }
+        settled = true;
+        channel.ack(raw);
+      };
+
+      const guardedNack = async (): Promise<void> => {
+        if (settled) {
+          console.warn(`[message-queue] duplicate nack on "${queue}" — ignored`);
+          return;
+        }
+        settled = true;
+        await retryOrDLQ(channel, raw, retryCount, retryQueue, failedQueue);
+      };
+
+      try {
+        await (handler as (msg: unknown, ack: () => void, nack: () => void) => Promise<void>)(parsed, guardedAck, guardedNack);
+
+        if (!settled) {
+          console.warn(`[message-queue] handler resolved without settling on "${queue}" — treating as failure`);
+          await retryOrDLQ(channel, raw, retryCount, retryQueue, failedQueue);
+        }
+      } catch (err) {
+        console.error(`[message-queue] handler threw on "${queue}":`, err);
+        if (!settled) {
+          await retryOrDLQ(channel, raw, retryCount, retryQueue, failedQueue);
+        }
+      }
+    } else {
+      // Auto-ACK: handler returns boolean
+      try {
+        const ok = await (handler as (msg: unknown) => Promise<boolean>)(parsed);
+
+        if (ok) {
+          channel.ack(raw);
+        } else {
+          await retryOrDLQ(channel, raw, retryCount, retryQueue, failedQueue);
+        }
+      } catch (err) {
+        console.error(`[message-queue] handler threw on "${queue}":`, err);
+        await retryOrDLQ(channel, raw, retryCount, retryQueue, failedQueue);
+      }
+    }
+  }, { noAck: false });
+}
+
+/** Close all tracked connections (publish + listeners). */
+async function close(): Promise<void> {
+  resetPublishConnection();
+
+  const closing = connections.map(async (conn) => {
+    try {
+      await conn.close();
+    } catch {
+      // Already closed — ignore
+    }
+  });
+  await Promise.all(closing);
+  connections.length = 0;
+}
+
+// ---------------------------------------------------------------------------
+// Internal helpers
+// ---------------------------------------------------------------------------
+
+function prefixedName(name: string): string {
+  return QUEUE_PREFIX ? `${QUEUE_PREFIX}/${name}` : name;
+}
+
+/** Lazy-create or return the existing publish channel. */
+async function getPublishChannel(): Promise<Channel> {
+  if (publishChannel) return publishChannel;
+
+  const conn = await amqplib.connect(process.env.CLOUDAMQP_URL!);
+  connections.push(conn);
+  publishConnection = conn;
+
+  conn.on('error', () => resetPublishConnection());
+  conn.on('close', () => resetPublishConnection());
+
+  publishChannel = await conn.createChannel();
+  return publishChannel;
+}
+
+/** Clear cached publish connection so the next call reconnects. */
+function resetPublishConnection(): void {
+  if (publishConnection) {
+    publishConnection.close().catch(() => { /* already closing */ });
+    const idx = connections.indexOf(publishConnection);
+    if (idx !== -1) connections.splice(idx, 1);
+  }
+  publishConnection = null;
+  publishChannel = null;
+}
+
+/**
+ * Assert main + retry + DLQ queues for a given queue name.
+ *
+ * Topology:
+ *   main ──(dead-letter)──▸ {name}.failed  (via exchange)
+ *   retry ──(TTL expiry)──▸ main           (via dead-letter back to main exchange)
+ */
+async function assertTopology(
+  channel: Channel,
+  queue: string,
+): Promise<{ mainQueue: string; retryQueue: string; failedQueue: string }> {
+  const mainQueue = prefixedName(queue);
+  const retryQueue = `${mainQueue}.retry`;
+  const failedQueue = `${mainQueue}.failed`;
+
+  const mainExchange = `${mainQueue}.exchange`;
+  const retryExchange = `${retryQueue}.exchange`;
+  const failedExchange = `${failedQueue}.exchange`;
+
+  // Exchanges
+  await channel.assertExchange(mainExchange, 'direct', { durable: true });
+  await channel.assertExchange(retryExchange, 'direct', { durable: true });
+  await channel.assertExchange(failedExchange, 'direct', { durable: true });
+
+  // Main queue — dead-letters to failed exchange.
+  // Note: the DLX here is primarily exercised for poison messages (unparseable
+  // JSON) which are nacked without requeue. Normal failures go through
+  // retryOrDLQ() which manually publishes to the retry/failed queue and acks.
+  await channel.assertQueue(mainQueue, {
+    durable: true,
+    deadLetterExchange: failedExchange,
+  });
+  await channel.bindQueue(mainQueue, mainExchange, '');
+
+  // Retry queue — messages TTL then dead-letter back to main exchange
+  await channel.assertQueue(retryQueue, {
+    durable: true,
+    deadLetterExchange: mainExchange,
+  });
+  await channel.bindQueue(retryQueue, retryExchange, '');
+
+  // Failed (DLQ) — terminal resting place
+  await channel.assertQueue(failedQueue, { durable: true });
+  await channel.bindQueue(failedQueue, failedExchange, '');
+
+  return { mainQueue, retryQueue, failedQueue };
+}
+
+function getRetryCount(msg: ConsumeMessage): number {
+  const headers = msg.properties.headers ?? {};
+  return typeof headers['x-retry-count'] === 'number' ? headers['x-retry-count'] : 0;
+}
+
+function computeBackoff(retryCount: number): number {
+  return Math.min(BACKOFF_BASE_MS * Math.pow(2, retryCount), BACKOFF_MAX_MS);
+}
+
+/**
+ * Retry a message (publish to retry queue with incremented count and TTL)
+ * or send to DLQ if max retries exceeded.
+ */
+async function retryOrDLQ(
+  channel: Channel,
+  raw: ConsumeMessage,
+  retryCount: number,
+  retryQueue: string,
+  failedQueue: string,
+): Promise<void> {
+  if (retryCount >= MAX_RETRIES) {
+    console.error(
+      `[message-queue] max retries (${MAX_RETRIES}) exceeded, moving to DLQ: ${failedQueue}`,
+    );
+    channel.publish('', failedQueue, raw.content, {
+      persistent: true,
+      headers: { ...raw.properties.headers, 'x-retry-count': retryCount },
+    });
+    channel.ack(raw);
+    return;
+  }
+
+  const ttl = computeBackoff(retryCount);
+  channel.publish('', retryQueue, raw.content, {
+    persistent: true,
+    expiration: String(ttl),
+    headers: {
+      ...raw.properties.headers,
+      'x-retry-count': retryCount + 1,
+    },
+  });
+  channel.ack(raw);
+}

package/scaffold/src/worker.ts

@@ -0,0 +1,58 @@
+/** @boopkit/cap-background-jobs — Worker entry point for background job processing. */
+
+import { loadServices } from '@services';
+import MessageQueueService from './services/message-queue.service';
+import * as exampleJobController from './controllers/example-job.controller';
+
+// Static consumer registration — add new {queue, handler} entries here
+const consumers = [
+  { queue: exampleJobController.QUEUE_NAME, handler: exampleJobController.onMessage },
+];
+
+const DRAIN_TIMEOUT_MS = 30_000;
+let shuttingDown = false;
+
+async function boot() {
+  await loadServices();
+
+  for (const consumer of consumers) {
+    await MessageQueueService.listen(consumer.queue, consumer.handler);
+  }
+
+  console.log(`[worker] listening on ${consumers.length} queue(s)`);
+}
+
+async function shutdown(signal: string) {
+  if (shuttingDown) return;
+  shuttingDown = true;
+
+  console.log(`[worker] ${signal} received — starting graceful shutdown`);
+
+  // Drain in-flight messages with timeout
+  const drainTimer = setTimeout(() => {
+    console.error('[worker] drain timeout exceeded — forcing exit');
+    process.exit(1);
+  }, DRAIN_TIMEOUT_MS);
+
+  try {
+    await MessageQueueService.close();
+    clearTimeout(drainTimer);
+    console.log('[worker] shutdown complete');
+    process.exit(0);
+  } catch (err) {
+    clearTimeout(drainTimer);
+    console.error('[worker] error during shutdown:', err);
+    process.exit(1);
+  }
+}
+
+process.on('SIGINT', () => shutdown('SIGINT'));
+process.on('SIGTERM', () => shutdown('SIGTERM'));
+
+boot().catch((err) => {
+  console.error('[worker] boot failed:', err);
+  process.exit(1);
+});
+
+// Keep the worker process alive
+process.stdin.resume();

package/scaffold/tests/controllers/example-job.controller.test.ts

@@ -0,0 +1,50 @@
+import { onMessage, QUEUE_NAME } from '../../src/controllers/example-job.controller';
+
+describe('example-job controller', () => {
+  let consoleSpy: jest.SpyInstance;
+
+  beforeEach(() => {
+    consoleSpy = jest.spyOn(console, 'log').mockImplementation();
+    jest.spyOn(console, 'error').mockImplementation();
+  });
+
+  afterEach(() => {
+    jest.restoreAllMocks();
+  });
+
+  it('exports QUEUE_NAME', () => {
+    expect(QUEUE_NAME).toBe('example-job');
+  });
+
+  it('ACKs a valid payload', async () => {
+    const result = await onMessage({ task: 'send-email', data: { to: 'a@b.com' } });
+    expect(result).toBe(true);
+    expect(consoleSpy).toHaveBeenCalledWith('[example-job] processed:', 'send-email');
+  });
+
+  it('ACKs payload with no optional data field', async () => {
+    const result = await onMessage({ task: 'cleanup' });
+    expect(result).toBe(true);
+  });
+
+  // Poison message: bad shape → ACK (don't retry)
+  it('ACKs when payload is not an object', async () => {
+    const result = await onMessage('not-an-object');
+    expect(result).toBe(true);
+  });
+
+  it('ACKs when payload is null', async () => {
+    const result = await onMessage(null);
+    expect(result).toBe(true);
+  });
+
+  it('ACKs when payload is missing "task" field', async () => {
+    const result = await onMessage({ foo: 'bar' });
+    expect(result).toBe(true);
+  });
+
+  it('ACKs when "task" is not a string', async () => {
+    const result = await onMessage({ task: 123 });
+    expect(result).toBe(true);
+  });
+});

package/scaffold/tests/services/message-queue.service.test.ts

@@ -0,0 +1,419 @@
+import type { ConsumeMessage } from 'amqplib';
+
+// Mock amqplib before importing the service
+const mockChannel: Record<string, jest.Mock> = {
+  assertExchange: jest.fn().mockResolvedValue(undefined),
+  assertQueue: jest.fn().mockResolvedValue({ queue: 'test' }),
+  bindQueue: jest.fn().mockResolvedValue(undefined),
+  sendToQueue: jest.fn().mockReturnValue(true),
+  publish: jest.fn().mockReturnValue(true),
+  consume: jest.fn().mockResolvedValue({ consumerTag: 'test-tag' }),
+  prefetch: jest.fn().mockResolvedValue(undefined),
+  ack: jest.fn(),
+  nack: jest.fn(),
+  once: jest.fn(),
+};
+
+const mockConnection: Record<string, jest.Mock> = {
+  createChannel: jest.fn().mockResolvedValue(mockChannel),
+  close: jest.fn().mockResolvedValue(undefined),
+  on: jest.fn(),
+};
+
+jest.mock('amqplib', () => ({
+  connect: jest.fn().mockResolvedValue(mockConnection),
+}));
+
+import amqplib from 'amqplib';
+import MessageQueueService from '../../src/services/message-queue.service';
+
+describe('MessageQueueService', () => {
+  const ORIGINAL_ENV = process.env;
+
+  beforeEach(() => {
+    jest.clearAllMocks();
+    process.env = { ...ORIGINAL_ENV, CLOUDAMQP_URL: 'amqp://test:test@localhost/test' };
+  });
+
+  afterEach(() => {
+    process.env = ORIGINAL_ENV;
+  });
+
+  describe('init()', () => {
+    it('succeeds when CLOUDAMQP_URL is set', () => {
+      expect(() => MessageQueueService.init()).not.toThrow();
+    });
+
+    it('throws when CLOUDAMQP_URL is missing', () => {
+      delete process.env.CLOUDAMQP_URL;
+      expect(() => MessageQueueService.init()).toThrow('CLOUDAMQP_URL is not set');
+    });
+  });
+
+  describe('publish()', () => {
+    it('connects lazily on first publish', async () => {
+      const result = await MessageQueueService.publish('test-queue', { foo: 'bar' });
+
+      expect(result).toBe(true);
+      expect(amqplib.connect).toHaveBeenCalledWith(process.env.CLOUDAMQP_URL);
+      expect(mockConnection.createChannel).toHaveBeenCalled();
+    });
+
+    it('publishes with persistent flag and messageId', async () => {
+      await MessageQueueService.publish('test-queue', { data: 1 });
+
+      expect(mockChannel.sendToQueue).toHaveBeenCalledWith(
+        'test-queue',
+        expect.any(Buffer),
+        expect.objectContaining({
+          persistent: true,
+          messageId: expect.any(String),
+        }),
+      );
+    });
+
+    it('serializes payload as JSON', async () => {
+      const payload = { key: 'value', count: 42 };
+      await MessageQueueService.publish('test-queue', payload);
+
+      const sentBuffer = (mockChannel.sendToQueue as jest.Mock).mock.calls[0][1];
+      expect(JSON.parse(sentBuffer.toString())).toEqual(payload);
+    });
+
+    it('returns false on error', async () => {
+      (mockChannel.sendToQueue as jest.Mock).mockImplementationOnce(() => {
+        throw new Error('channel closed');
+      });
+
+      const result = await MessageQueueService.publish('test-queue', {});
+      expect(result).toBe(false);
+    });
+
+    it('waits for drain when channel buffer is full', async () => {
+      (mockChannel.sendToQueue as jest.Mock).mockReturnValueOnce(false);
+      (mockChannel.once as jest.Mock).mockImplementationOnce(
+        (_event: string, cb: () => void) => { cb(); },
+      );
+
+      const result = await MessageQueueService.publish('test-queue', {});
+
+      expect(result).toBe(true);
+      expect(mockChannel.once).toHaveBeenCalledWith('drain', expect.any(Function));
+    });
+
+    // Note: QUEUE_PREFIX is captured once at module load time via const.
+    // It cannot be tested by setting process.env after import. The prefix
+    // behavior is verified through listen() topology assertions instead.
+  });
+
+  describe('listen()', () => {
+    it('creates a new connection per listener', async () => {
+      await MessageQueueService.listen('jobs', async () => true);
+
+      expect(amqplib.connect).toHaveBeenCalledWith(process.env.CLOUDAMQP_URL);
+      expect(mockConnection.createChannel).toHaveBeenCalled();
+    });
+
+    it('asserts full queue topology (main + retry + DLQ)', async () => {
+      await MessageQueueService.listen('jobs', async () => true);
+
+      // Exchanges
+      expect(mockChannel.assertExchange).toHaveBeenCalledWith(
+        'jobs.exchange', 'direct', { durable: true },
+      );
+      expect(mockChannel.assertExchange).toHaveBeenCalledWith(
+        'jobs.retry.exchange', 'direct', { durable: true },
+      );
+      expect(mockChannel.assertExchange).toHaveBeenCalledWith(
+        'jobs.failed.exchange', 'direct', { durable: true },
+      );
+
+      // Queues
+      expect(mockChannel.assertQueue).toHaveBeenCalledWith(
+        'jobs',
+        expect.objectContaining({ durable: true, deadLetterExchange: 'jobs.failed.exchange' }),
+      );
+      expect(mockChannel.assertQueue).toHaveBeenCalledWith(
+        'jobs.retry',
+        expect.objectContaining({ durable: true, deadLetterExchange: 'jobs.exchange' }),
+      );
+      expect(mockChannel.assertQueue).toHaveBeenCalledWith(
+        'jobs.failed',
+        expect.objectContaining({ durable: true }),
+      );
+    });
+
+    it('sets up a consumer on the main queue', async () => {
+      await MessageQueueService.listen('jobs', async () => true);
+      expect(mockChannel.consume).toHaveBeenCalledWith('jobs', expect.any(Function), { noAck: false });
+    });
+
+    describe('message dispatch', () => {
+      let consumeCallback: (msg: ConsumeMessage | null) => Promise<void>;
+
+      beforeEach(async () => {
+        (mockChannel.consume as jest.Mock).mockImplementationOnce(
+          (_queue: string, cb: (msg: ConsumeMessage | null) => Promise<void>) => {
+            consumeCallback = cb;
+            return Promise.resolve({ consumerTag: 'tag' });
+          },
+        );
+        await MessageQueueService.listen('jobs', async (msg) => {
+          if ((msg as any).fail) return false;
+          if ((msg as any).throw) throw new Error('boom');
+          return true;
+        });
+      });
+
+      function makeMessage(payload: unknown, retryCount = 0): ConsumeMessage {
+        return {
+          content: Buffer.from(JSON.stringify(payload)),
+          properties: {
+            headers: { 'x-retry-count': retryCount },
+          },
+          fields: { deliveryTag: 1, redelivered: false, exchange: '', routingKey: '' },
+        } as unknown as ConsumeMessage;
+      }
+
+      it('ACKs when handler returns true', async () => {
+        await consumeCallback(makeMessage({ ok: true }));
+        expect(mockChannel.ack).toHaveBeenCalled();
+      });
+
+      it('retries when handler returns false', async () => {
+        await consumeCallback(makeMessage({ fail: true }));
+
+        // Should publish to retry queue with incremented retry count
+        expect(mockChannel.publish).toHaveBeenCalledWith(
+          '',
+          'jobs.retry',
+          expect.any(Buffer),
+          expect.objectContaining({
+            persistent: true,
+            expiration: '4000', // 4000 * 2^0
+            headers: expect.objectContaining({ 'x-retry-count': 1 }),
+          }),
+        );
+        expect(mockChannel.ack).toHaveBeenCalled();
+      });
+
+      it('sends to DLQ after max retries', async () => {
+        await consumeCallback(makeMessage({ fail: true }, 5));
+
+        expect(mockChannel.publish).toHaveBeenCalledWith(
+          '',
+          'jobs.failed',
+          expect.any(Buffer),
+          expect.objectContaining({
+            persistent: true,
+            headers: expect.objectContaining({ 'x-retry-count': 5 }),
+          }),
+        );
+        expect(mockChannel.ack).toHaveBeenCalled();
+      });
+
+      it('retries on handler throw (below max retries)', async () => {
+        await consumeCallback(makeMessage({ throw: true }, 2));
+
+        expect(mockChannel.publish).toHaveBeenCalledWith(
+          '',
+          'jobs.retry',
+          expect.any(Buffer),
+          expect.objectContaining({
+            expiration: '16000', // 4000 * 2^2
+            headers: expect.objectContaining({ 'x-retry-count': 3 }),
+          }),
+        );
+      });
+
+      it('ignores null messages', async () => {
+        await consumeCallback(null as unknown as ConsumeMessage);
+        expect(mockChannel.ack).not.toHaveBeenCalled();
+        expect(mockChannel.nack).not.toHaveBeenCalled();
+      });
+
+      it('NACKs unparseable messages (poison)', async () => {
+        const poisonMsg = {
+          content: Buffer.from('not-json{{{'),
+          properties: { headers: {} },
+          fields: { deliveryTag: 1, redelivered: false, exchange: '', routingKey: '' },
+        } as unknown as ConsumeMessage;
+
+        await consumeCallback(poisonMsg);
+        expect(mockChannel.nack).toHaveBeenCalledWith(poisonMsg, false, false);
+      });
+    });
+  });
+
+  describe('listen() manual-ACK mode', () => {
+    let consumeCallback: (msg: ConsumeMessage | null) => Promise<void>;
+
+    function makeMessage(payload: unknown, retryCount = 0): ConsumeMessage {
+      return {
+        content: Buffer.from(JSON.stringify(payload)),
+        properties: {
+          headers: { 'x-retry-count': retryCount },
+        },
+        fields: { deliveryTag: 1, redelivered: false, exchange: '', routingKey: '' },
+      } as unknown as ConsumeMessage;
+    }
+
+    function setupManualAckListener(
+      handler: (msg: unknown, ack: () => void, nack: () => void) => Promise<void>,
+    ) {
+      (mockChannel.consume as jest.Mock).mockImplementationOnce(
+        (_queue: string, cb: (msg: ConsumeMessage | null) => Promise<void>) => {
+          consumeCallback = cb;
+          return Promise.resolve({ consumerTag: 'tag' });
+        },
+      );
+      return MessageQueueService.listen('jobs', handler, { manualAck: true });
+    }
+
+    it('handler receives ack and nack functions', async () => {
+      let receivedAck: (() => void) | undefined;
+      let receivedNack: (() => void) | undefined;
+
+      await setupManualAckListener(async (_msg, ack, nack) => {
+        receivedAck = ack;
+        receivedNack = nack;
+        ack();
+      });
+      await consumeCallback(makeMessage({ data: 1 }));
+
+      expect(typeof receivedAck).toBe('function');
+      expect(typeof receivedNack).toBe('function');
+      expect(mockChannel.ack).toHaveBeenCalled();
+    });
+
+    it('guarded ack — second call is no-op', async () => {
+      await setupManualAckListener(async (_msg, ack) => {
+        ack();
+        ack(); // second call — should be no-op
+      });
+      await consumeCallback(makeMessage({ data: 1 }));
+
+      expect(mockChannel.ack).toHaveBeenCalledTimes(1);
+    });
+
+    it('guarded nack — second call is no-op', async () => {
+      await setupManualAckListener(async (_msg, _ack, nack) => {
+        nack();
+        nack(); // second call — should be no-op
+      });
+      await consumeCallback(makeMessage({ data: 1 }));
+
+      // retryOrDLQ publishes to retry queue then acks — only one round
+      expect(mockChannel.publish).toHaveBeenCalledTimes(1);
+      expect(mockChannel.ack).toHaveBeenCalledTimes(1);
+    });
+
+    it('ack then nack — nack is no-op (first call wins)', async () => {
+      await setupManualAckListener(async (_msg, ack, nack) => {
+        ack();
+        nack(); // should be no-op since ack already settled
+      });
+      await consumeCallback(makeMessage({ data: 1 }));
+
+      expect(mockChannel.ack).toHaveBeenCalledTimes(1);
+      expect(mockChannel.publish).not.toHaveBeenCalled(); // no retry
+    });
+
+    it('handler throws without settling — retries via DLQ path', async () => {
+      await setupManualAckListener(async () => {
+        throw new Error('boom');
+      });
+      await consumeCallback(makeMessage({ data: 1 }));
+
+      // retryOrDLQ should have been called (publishes to retry queue)
+      expect(mockChannel.publish).toHaveBeenCalledWith(
+        '',
+        'jobs.retry',
+        expect.any(Buffer),
+        expect.objectContaining({
+          persistent: true,
+          headers: expect.objectContaining({ 'x-retry-count': 1 }),
+        }),
+      );
+      expect(mockChannel.ack).toHaveBeenCalled();
+    });
+
+    it('handler resolves without settling — treated as failure', async () => {
+      await setupManualAckListener(async () => {
+        // deliberately not calling ack or nack
+      });
+      await consumeCallback(makeMessage({ data: 1 }));
+
+      // retryOrDLQ should have been called
+      expect(mockChannel.publish).toHaveBeenCalledWith(
+        '',
+        'jobs.retry',
+        expect.any(Buffer),
+        expect.objectContaining({
+          headers: expect.objectContaining({ 'x-retry-count': 1 }),
+        }),
+      );
+      expect(mockChannel.ack).toHaveBeenCalled();
+    });
+
+    it('handler throws after acking — no double retry', async () => {
+      await setupManualAckListener(async (_msg, ack) => {
+        ack();
+        throw new Error('post-ack error');
+      });
+      await consumeCallback(makeMessage({ data: 1 }));
+
+      // ack was called but no retry should happen (already settled)
+      expect(mockChannel.ack).toHaveBeenCalledTimes(1);
+      expect(mockChannel.publish).not.toHaveBeenCalled();
+    });
+  });
+
+  describe('listen() prefetch', () => {
+    it('sets channel.prefetch when opts.prefetch is provided', async () => {
+      await MessageQueueService.listen('jobs', async () => true, { prefetch: 10 });
+      expect(mockChannel.prefetch).toHaveBeenCalledWith(10);
+    });
+
+    it('does not call prefetch when opts.prefetch is not provided', async () => {
+      await MessageQueueService.listen('jobs', async () => true);
+      expect(mockChannel.prefetch).not.toHaveBeenCalled();
+    });
+
+    it('sets prefetch before consuming', async () => {
+      const callOrder: string[] = [];
+      (mockChannel.prefetch as jest.Mock).mockImplementation(() => {
+        callOrder.push('prefetch');
+        return Promise.resolve(undefined);
+      });
+      (mockChannel.consume as jest.Mock).mockImplementation(() => {
+        callOrder.push('consume');
+        return Promise.resolve({ consumerTag: 'tag' });
+      });
+
+      await MessageQueueService.listen('jobs', async () => true, { prefetch: 5 });
+
+      expect(callOrder).toEqual(['prefetch', 'consume']);
+    });
+  });
+
+  describe('close()', () => {
+    it('closes all tracked connections', async () => {
+      // Create a listener connection
+      await MessageQueueService.listen('q', async () => true);
+
+      await MessageQueueService.close();
+      expect(mockConnection.close).toHaveBeenCalled();
+    });
+  });
+
+  describe('service shape', () => {
+    it('exports the required service interface', () => {
+      expect(MessageQueueService.name).toBe('message-queue');
+      expect(typeof MessageQueueService.init).toBe('function');
+      expect(typeof MessageQueueService.publish).toBe('function');
+      expect(typeof MessageQueueService.listen).toBe('function');
+      expect(typeof MessageQueueService.close).toBe('function');
+    });
+  });
+});

package/skills/design.background-jobs.skill.md

@@ -0,0 +1,145 @@
+# Background Jobs Design
+When you need to add a background job consumer, publish messages, or work with the message queue infrastructure, follow these principles.
+
+Assumes: controller design principles from [design.controllers.skill.md](../../cap-services/skills/design.controllers.skill.md) (services capability). MQ controllers follow the same thin-dispatch pattern — parse, call service, return result.
+
+## Adding a new consumer
+
+Create a controller, register it in the worker. Two files to touch:
+
+### 1. Write the controller
+
+Copy `src/controllers/example-job.controller.ts` and replace the handler body:
+
+```typescript
+export const QUEUE_NAME = 'my-job';
+
+export async function onMessage(msg: unknown): Promise<boolean> {
+  const payload = validatePayload(msg);
+
+  try {
+    await services.myService.process(payload);
+    return true;   // ACK
+  } catch (err) {
+    console.error('[my-job] failed:', err);
+    return false;  // retry with backoff
+  }
+}
+```
+
+**Error classification:**
+- `return true` — Success, or permanent failure that retrying won't fix (bad input, missing entity)
+- `return false` — Transient failure worth retrying (timeout, rate limit, temporary unavailability)
+- `throw` — Unexpected error — retries, then DLQ after max attempts
+
+### 2. Register in worker.ts
+
+```typescript
+import * as myJobController from './controllers/my-job.controller';
+
+const consumers = [
+  { queue: exampleJobController.QUEUE_NAME, handler: exampleJobController.onMessage },
+  { queue: myJobController.QUEUE_NAME, handler: myJobController.onMessage },
+];
+```
+
+That's it. The MQ service creates the queue topology (main + retry + DLQ) automatically on first listen.
+
+## Publishing messages
+
+Use `MessageQueueService.publish(queue, payload)` from any process:
+
+```typescript
+import MessageQueueService from './services/message-queue.service';
+
+await MessageQueueService.publish('my-job', {
+  task: 'process-upload',
+  data: { fileId: 'abc123', userId: 'usr_456' },
+});
+```
+
+### Where to publish from
+
+| Source | When to use | Example |
+|--------|------------|---------|
+| **Service** (in web process) | Side effect of a user action | User uploads file → publish processing job |
+| **Scheduler** | Periodic triggers on a cron schedule | Nightly cleanup, hourly sync |
+| **Controller** (in worker) | Follow-up work from a completed job | Job A completes → publish Job B |
+
+### Adding a scheduled job
+
+Add an entry to the `jobs` array in `src/scheduler.ts`:
+
+```typescript
+const jobs = [
+  { queue: 'example-job', payload: { task: 'scheduled-run', data: { ... } } },
+  { queue: 'my-job',      payload: { task: 'nightly-sync', data: { ... } } },
+];
+```
+
+The scheduler publishes all jobs and exits. Configure Heroku Scheduler to run `npm run scheduler` at the desired interval.
+
+## Auto-ACK vs manual-ACK
+
+**Use auto-ACK (default)** for most consumers. The handler returns `true`/`false` and the MQ service handles ACK/retry automatically. Simpler and safer.
+
+**Use manual-ACK** when you need to:
+- ACK after a batch of work completes (buffered batch pattern)
+- Conditionally ACK based on external state that changes during processing
+- Coordinate ACK with a transaction or external system commit
+
+```typescript
+await MessageQueueService.listen('batch-job', async (msg, ack, nack) => {
+  try {
+    await processBatch(msg);
+    ack();  // explicit ACK after batch commit
+  } catch (err) {
+    nack(); // explicit retry/DLQ
+  }
+}, { manualAck: true });
+```
+
+**Guard rules:**
+- First call to `ack()` or `nack()` wins — subsequent calls are silent no-ops
+- If the handler resolves without calling either → treated as failure (retry/DLQ)
+- If the handler throws after acking → no double-retry (already settled)
+
+## Prefetch and backpressure
+
+Set `prefetch` to limit how many messages RabbitMQ delivers to a consumer before it ACKs:
+
+```typescript
+await MessageQueueService.listen('heavy-job', handler, { prefetch: 5 });
+```
+
+Use this for consumers that do expensive work (API calls, file processing) to prevent the worker from being overwhelmed. Without prefetch, RabbitMQ delivers all available messages as fast as the consumer can accept them.
+
+## Retry model
+
+Failed messages follow exponential backoff:
+
+```
+Attempt 1: 4s wait   (4000 * 2^0)
+Attempt 2: 8s wait   (4000 * 2^1)
+Attempt 3: 16s wait  (4000 * 2^2)
+Attempt 4: 32s wait  (4000 * 2^3)
+Attempt 5: 60s wait  (capped at 60s)
+→ DLQ after MAX_MESSAGE_RETRIES (default 5)
+```
+
+Messages in the DLQ (`{queue}.failed`) sit until manually inspected or reprocessed. Use the CloudAMQP management UI to inspect DLQ contents.
+
+## Queue naming
+
+Queues are prefixed with `QUEUE_PREFIX` for environment isolation:
+
+- Production: `QUEUE_PREFIX=` (empty) → `my-job`
+- Dev: `QUEUE_PREFIX=dev/mike` → `dev/mike/my-job`
+
+Set `QUEUE_PREFIX` in `.env` to avoid cross-talk between developers sharing a CloudAMQP instance.
+
+---
+
+See also:
+- [reference/background-jobs.ref.md](../reference/background-jobs.ref.md) for infrastructure topology and environment setup
+- [reference/message-queue-patterns.ref.md](../../profiles/web-app/reference/message-queue-patterns.ref.md) for advanced patterns (buffered batch ACK, connection lifecycle)