@jgamaraalv/ts-dev-kit 1.0.0

package/skills/bullmq/SKILL.md

@@ -0,0 +1,225 @@
+---
+name: bullmq
+description: "BullMQ queue system reference for Redis-backed job queues, workers, flows, and schedulers. Use when: (1) creating queues and workers with BullMQ, (2) adding jobs (delayed, prioritized, repeatable, deduplicated), (3) setting up FlowProducer parent-child job hierarchies, (4) configuring retry strategies, rate limiting, or concurrency, (5) implementing job schedulers with cron/interval patterns, (6) preparing BullMQ for production (graceful shutdown, Redis config, monitoring), or (7) debugging stalled jobs or connection issues"
+---
+
+# BullMQ
+
+Redis-backed queue system for Node.js. Four core classes: `Queue`, `Worker`, `QueueEvents`, `FlowProducer`.
+
+## Table of Contents
+
+- [Install](#install)
+- [Quick Start](#quick-start)
+- [Connections](#connections)
+- [Queue](#queue)
+- [Worker](#worker)
+- [TypeScript Generics](#typescript-generics)
+- [Events](#events)
+- [Job Lifecycle States](#job-lifecycle-states)
+- [Advanced Topics](#advanced-topics)
+
+## Install
+
+`yarn add bullmq` — requires Redis 5.0+ with `maxmemory-policy=noeviction`.
+
+## Quick Start
+
+```ts
+import { Queue, Worker, QueueEvents } from "bullmq";
+
+// --- Producer ---
+const queue = new Queue("my-queue", {
+  connection: { host: "localhost", port: 6379 },
+});
+
+await queue.add("job-name", { foo: "bar" });
+
+// --- Consumer ---
+const worker = new Worker(
+  "my-queue",
+  async (job) => {
+    // process job
+    await job.updateProgress(50);
+    return { result: "done" };
+  },
+  { connection: { host: "localhost", port: 6379 } },
+);
+
+worker.on("completed", (job, returnvalue) => {
+  console.log(`${job.id} completed with`, returnvalue);
+});
+
+worker.on("failed", (job, err) => {
+  console.error(`${job.id} failed with`, err.message);
+});
+
+// IMPORTANT: always attach an error handler
+worker.on("error", (err) => {
+  console.error(err);
+});
+
+// --- Global event listener (all workers) ---
+const queueEvents = new QueueEvents("my-queue", {
+  connection: { host: "localhost", port: 6379 },
+});
+
+queueEvents.on("completed", ({ jobId, returnvalue }) => {
+  console.log(`Job ${jobId} completed`);
+});
+
+queueEvents.on("failed", ({ jobId, failedReason }) => {
+  console.error(`Job ${jobId} failed: ${failedReason}`);
+});
+```
+
+## Connections
+
+BullMQ uses ioredis internally. Pass `connection` options or an existing ioredis instance.
+
+```ts
+import { Queue, Worker } from "bullmq";
+import { Redis } from "ioredis";
+
+// Option 1: connection config (new connection per instance)
+const queue = new Queue("q", {
+  connection: { host: "redis.example.com", port: 6379 },
+});
+
+// Option 2: reuse ioredis instance (Queue and multiple Queues can share)
+const connection = new Redis();
+const q1 = new Queue("q1", { connection });
+const q2 = new Queue("q2", { connection });
+
+// Option 3: reuse for Workers (BullMQ internally duplicates for blocking)
+const workerConn = new Redis({ maxRetriesPerRequest: null });
+const w1 = new Worker("q1", async (job) => {}, { connection: workerConn });
+```
+
+**Critical rules:**
+
+- Workers REQUIRE `maxRetriesPerRequest: null` on the ioredis instance. BullMQ enforces this and will warn/throw if not set.
+- Do NOT use ioredis `keyPrefix` option — use BullMQ's `prefix` option instead.
+- `QueueEvents` cannot share connections (uses blocking Redis commands).
+- Redis MUST have `maxmemory-policy=noeviction`.
+
+## Queue
+
+```ts
+const queue = new Queue("paint", { connection });
+
+// Add a job
+await queue.add("job-name", { color: "red" });
+
+// Add with options
+await queue.add(
+  "job-name",
+  { color: "blue" },
+  {
+    delay: 5000, // wait 5s before processing
+    priority: 1, // lower = higher priority (0 is highest, max 2^21)
+    attempts: 3, // retry up to 3 times
+    backoff: { type: "exponential", delay: 1000 },
+    removeOnComplete: true, // or { count: 100 } to keep last 100
+    removeOnFail: 1000, // keep last 1000 failed jobs
+  },
+);
+
+// Add bulk
+await queue.addBulk([
+  { name: "job1", data: { x: 1 } },
+  { name: "job2", data: { x: 2 }, opts: { priority: 1 } },
+]);
+
+// Queue operations
+await queue.pause();
+await queue.resume();
+await queue.obliterate({ force: true }); // remove all data
+await queue.close();
+```
+
+## Worker
+
+```ts
+const worker = new Worker<MyData, MyReturn>(
+  "paint",
+  async (job) => {
+    await job.updateProgress(42);
+    return { cost: 100 };
+  },
+  {
+    connection,
+    concurrency: 5, // process 5 jobs concurrently
+    autorun: false, // don't start immediately
+  },
+);
+
+worker.run(); // start when ready
+
+// Update concurrency at runtime
+worker.concurrency = 10;
+```
+
+**Processor receives 3 args:** `(job, token?, signal?)` — signal is an `AbortSignal` for cancellation support.
+
+## TypeScript Generics
+
+```ts
+interface JobData {
+  color: string;
+}
+interface JobReturn {
+  cost: number;
+}
+
+const queue = new Queue<JobData, JobReturn>("paint");
+const worker = new Worker<JobData, JobReturn>("paint", async (job) => {
+  // job.data is typed as JobData
+  return { cost: 100 }; // must match JobReturn
+});
+```
+
+## Events
+
+**Worker events** (local to that worker instance):
+
+| Event       | Callback signature                   |
+| ----------- | ------------------------------------ |
+| `completed` | `(job, returnvalue)`                 |
+| `failed`    | `(job \| undefined, error, prev)`    |
+| `progress`  | `(job, progress: number \| object)`  |
+| `drained`   | `()` — queue is empty                |
+| `error`     | `(error)` — MUST attach this handler |
+
+**QueueEvents** (global, all workers, uses Redis Streams):
+
+| Event          | Callback signature                                |
+| -------------- | ------------------------------------------------- |
+| `completed`    | `({ jobId, returnvalue })`                        |
+| `failed`       | `({ jobId, failedReason })`                       |
+| `progress`     | `({ jobId, data })`                               |
+| `waiting`      | `({ jobId })`                                     |
+| `active`       | `({ jobId, prev })`                               |
+| `delayed`      | `({ jobId, delay })`                              |
+| `deduplicated` | `({ jobId, deduplicationId, deduplicatedJobId })` |
+
+Event stream is auto-trimmed (~10,000 events). Configure via `streams.events.maxLen`.
+
+## Job Lifecycle States
+
+```
+add() → wait / prioritized / delayed
+         ↓
+       active → completed
+         ↓
+       failed → (retry) → wait/delayed
+```
+
+With FlowProducer: jobs can also be in `waiting-children` state until all children complete.
+
+## Advanced Topics
+
+- **Job types and options** (delayed, prioritized, deduplication, repeatable): See [references/job-types-and-options.md](references/job-types-and-options.md)
+- **Flows and schedulers** (FlowProducer, parent-child, job schedulers, cron): See [references/flows-and-schedulers.md](references/flows-and-schedulers.md)
+- **Patterns** (step jobs, idempotent, throttle, manual rate-limit): See [references/patterns.md](references/patterns.md)
+- **Production** (shutdown, Redis config, retries, backoff, monitoring): See [references/production.md](references/production.md)

package/skills/bullmq/references/flows-and-schedulers.md

@@ -0,0 +1,186 @@
+# Flows and Job Schedulers
+
+## Table of Contents
+
+- [FlowProducer — Parent-Child Relationships](#flowproducer--parent-child-relationships)
+- [Job Schedulers (v5.16.0+)](#job-schedulers-v5160)
+
+## FlowProducer — Parent-Child Relationships
+
+A parent job waits in `waiting-children` state until all its children complete. Children can be in different queues.
+
+```ts
+import { FlowProducer } from "bullmq";
+
+const flowProducer = new FlowProducer({ connection });
+
+const flow = await flowProducer.add({
+  name: "renovate-interior",
+  queueName: "renovate",
+  children: [
+    { name: "paint", data: { place: "ceiling" }, queueName: "steps" },
+    { name: "paint", data: { place: "walls" }, queueName: "steps" },
+    { name: "fix", data: { place: "floor" }, queueName: "steps" },
+  ],
+});
+```
+
+This atomically adds 4 jobs. The parent (`renovate`) processes only after all 3 children complete.
+
+### Accessing Children Results
+
+```ts
+const renovateWorker = new Worker("renovate", async (job) => {
+  const childrenValues = await job.getChildrenValues();
+  // { "bull:steps:paint:1": 2500, "bull:steps:fix:3": 1750 }
+  const total = Object.values(childrenValues).reduce((a, b) => a + b, 0);
+  return total;
+});
+```
+
+### Serial Chains (Deep Hierarchies)
+
+Nest children to create sequential execution:
+
+```ts
+await flowProducer.add({
+  name: "car",
+  data: { step: "engine" },
+  queueName: "assembly",
+  children: [
+    {
+      name: "car",
+      data: { step: "wheels" },
+      queueName: "assembly",
+      children: [{ name: "car", data: { step: "chassis" }, queueName: "assembly" }],
+    },
+  ],
+});
+// Processing order: chassis → wheels → engine
+```
+
+### Flow Getters
+
+```ts
+// Get direct children
+const deps = await job.getDependencies();
+
+// Paginated by type
+const { processed, nextProcessedCursor } = await job.getDependencies({
+  processed: { count: 5, cursor: 0 },
+});
+
+// Count by type
+const { failed, ignored, processed, unprocessed } = await job.getDependenciesCount();
+
+// Check state
+const state = await job.getState(); // "waiting-children"
+```
+
+### Queue Options for Flows
+
+```ts
+await flowProducer.add(flowDefinition, {
+  queuesOptions: {
+    assembly: {
+      defaultJobOptions: { removeOnComplete: true },
+    },
+  },
+});
+```
+
+### Flow Job Removal
+
+- Removing a **parent** removes all its children.
+- Removing a **child** removes the parent's dependency on it. If it was the last child, the parent completes.
+- If any job to be removed is **locked** (being processed), the removal throws an exception.
+
+```ts
+await job.remove();
+// or
+await queue.remove(jobId);
+```
+
+### Add Flows in Bulk
+
+```ts
+await flowProducer.addBulk([
+  { name: "flow1", queueName: "q", children: [...] },
+  { name: "flow2", queueName: "q", children: [...] },
+]);
+```
+
+## Job Schedulers (v5.16.0+)
+
+Job Schedulers replace legacy "repeatable jobs". They act as factories producing jobs on a schedule.
+
+### Fixed Interval
+
+```ts
+const firstJob = await queue.upsertJobScheduler("my-scheduler", {
+  every: 1000, // every 1 second
+});
+```
+
+### Cron Pattern
+
+```ts
+await queue.upsertJobScheduler(
+  "daily-report",
+  { pattern: "0 15 3 * * *" }, // daily at 3:15 AM
+  {
+    name: "generate-report",
+    data: { type: "daily" },
+    opts: { attempts: 5, backoff: 3, removeOnFail: 1000 },
+  },
+);
+```
+
+### Key Behaviors
+
+- **Upsert semantics**: calling `upsertJobScheduler` with the same ID updates the existing scheduler (no duplicates).
+- **Production rate**: a new job is only produced when the previous one starts processing. If the queue is busy, jobs may arrive less frequently than the interval.
+- **Job ID**: jobs get auto-generated IDs — you cannot use custom job IDs. Use `name` to discriminate.
+- There is always one associated job in "delayed" status.
+
+### Manage Schedulers
+
+```ts
+// List all schedulers
+const schedulers = await queue.getJobSchedulers(0, 100);
+
+// Remove a scheduler
+await queue.removeJobScheduler("my-scheduler");
+```
+
+### Repeat Strategies
+
+Built-in: `every` (fixed interval) and `pattern` (cron). You can define custom strategies:
+
+```ts
+const queue = new Queue("q", {
+  settings: {
+    repeatStrategy: (millis, opts) => {
+      // Return next execution timestamp
+      return Date.now() + customDelay;
+    },
+  },
+});
+```
+
+### Repeat Options
+
+```ts
+await queue.upsertJobScheduler("scheduler-id", {
+  pattern: "0 * * * *", // cron expression
+  // or
+  every: 5000, // fixed ms interval
+
+  // Optional:
+  limit: 100, // max times to repeat
+  startDate: new Date("2025-01-01"),
+  endDate: new Date("2025-12-31"),
+  tz: "America/Sao_Paulo", // timezone for cron
+  immediately: true, // produce first job immediately
+});
+```

package/skills/bullmq/references/job-types-and-options.md

@@ -0,0 +1,163 @@
+# Job Types and Options
+
+## Table of Contents
+
+- [Delayed Jobs](#delayed-jobs)
+- [Prioritized Jobs](#prioritized-jobs)
+- [FIFO and LIFO](#fifo-and-lifo)
+- [Custom Job IDs](#custom-job-ids)
+- [Job Data](#job-data)
+- [Deduplication](#deduplication)
+- [Repeatable Jobs (Legacy)](#repeatable-jobs-legacy)
+- [Auto-Removal Options](#auto-removal-options)
+- [Returning Job Data](#returning-job-data)
+
+## Delayed Jobs
+
+```ts
+await queue.add("email", { to: "user@example.com" }, { delay: 60000 }); // 1 min delay
+```
+
+Jobs wait in the "delayed" set, then move to "wait" (or "prioritized") when the delay expires. No `QueueScheduler` needed since BullMQ 2.0.
+
+## Prioritized Jobs
+
+Priority range: `0` (highest) to `2^21` (lowest). Default is `0`. Follows Unix nice convention — higher number = lower priority.
+
+```ts
+await queue.add("urgent", data, { priority: 1 });
+await queue.add("normal", data, { priority: 100 });
+await queue.add("low", data, { priority: 2097152 }); // 2^21
+```
+
+Prioritized jobs go into a dedicated set and are picked before regular "wait" jobs.
+
+## FIFO and LIFO
+
+Default is FIFO. Use `{ lifo: true }` to place jobs at the head of the waiting list.
+
+## Custom Job IDs
+
+```ts
+await queue.add("job", data, { jobId: "my-unique-id" });
+```
+
+If a job with the same ID exists and is not yet removed, the `add` call is ignored. Useful for idempotency. Do NOT use colons `:` in job IDs (used as internal separator).
+
+## Job Data
+
+Keep job data small — it's stored in Redis. For large payloads, store data externally and pass a reference.
+
+```ts
+// Update data during processing
+await job.updateData({ ...job.data, step: "next" });
+
+// Get current data
+const data = job.data;
+```
+
+## Deduplication
+
+Three modes, all use a `deduplication` option with an `id`:
+
+### Simple Mode — deduplicate until job completes/fails
+
+```ts
+await queue.add("process", data, {
+  deduplication: { id: "unique-key" },
+});
+```
+
+Subsequent adds with same dedup ID are ignored while the job is active.
+
+### Throttle Mode — deduplicate for a TTL period
+
+```ts
+await queue.add("process", data, {
+  deduplication: { id: "unique-key", ttl: 5000 },
+});
+```
+
+Ignores duplicate adds for 5 seconds regardless of job state.
+
+### Debounce Mode — replace previous job, reset TTL
+
+```ts
+await queue.add("process", data, {
+  deduplication: { id: "unique-key", ttl: 5000, extend: true, replace: true },
+  delay: 5000,
+});
+```
+
+Replaces the previous pending job with new data. Only the last added job is processed.
+
+### Deduplication API
+
+```ts
+// Check which job holds the dedup key
+const jobId = await queue.getDeduplicationJobId("unique-key");
+
+// Remove dedup key manually
+await queue.removeDeduplicationKey("unique-key");
+// Or from the job itself
+const removed = await job.removeDeduplicationKey();
+```
+
+Listen for dedup events:
+
+```ts
+queueEvents.on("deduplicated", ({ jobId, deduplicationId, deduplicatedJobId }) => {
+  console.log(`Job ${deduplicatedJobId} was deduplicated`);
+});
+```
+
+## Repeatable Jobs (Legacy)
+
+Prefer Job Schedulers (`upsertJobScheduler`) for new code. Legacy `repeat` option still works:
+
+```ts
+await queue.add("report", data, {
+  repeat: { every: 60000 }, // every 60s
+  // or
+  repeat: { pattern: "0 * * * *" }, // cron: every hour
+});
+```
+
+## Auto-Removal Options
+
+Control how many completed/failed jobs to keep:
+
+```ts
+await queue.add("job", data, {
+  // removeOnComplete: true,           // Option A: remove immediately
+  // removeOnComplete: { count: 100 }, // Option B: keep last 100
+  removeOnComplete: { age: 3600 }, // Option C: keep for 1 hour
+  removeOnFail: { count: 500 }, // keep last 500 failed
+});
+```
+
+Set defaults for all jobs:
+
+```ts
+const queue = new Queue("q", {
+  defaultJobOptions: {
+    removeOnComplete: { count: 200 },
+    removeOnFail: { count: 1000 },
+  },
+});
+```
+
+## Returning Job Data
+
+The value returned from the processor is stored as the job's `returnvalue`:
+
+```ts
+// In the worker
+async (job) => {
+  return { processedAt: Date.now() };
+};
+
+// Retrieve later
+const job = await Job.fromId(queue, jobId);
+console.log(job.returnvalue); // { processedAt: ... }
+```