@igniter-js/jobs 0.1.1 → 0.1.12

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (42) hide show
  1. package/AGENTS.md +1118 -96
  2. package/CHANGELOG.md +8 -0
  3. package/README.md +2146 -93
  4. package/dist/{adapter-PiDCQWQd.d.mts → adapter-CXZxomI9.d.mts} +2 -2
  5. package/dist/{adapter-PiDCQWQd.d.ts → adapter-CXZxomI9.d.ts} +2 -2
  6. package/dist/adapters/bullmq.adapter.d.mts +2 -2
  7. package/dist/adapters/bullmq.adapter.d.ts +2 -2
  8. package/dist/adapters/bullmq.adapter.js +2 -2
  9. package/dist/adapters/bullmq.adapter.js.map +1 -1
  10. package/dist/adapters/bullmq.adapter.mjs +1 -1
  11. package/dist/adapters/bullmq.adapter.mjs.map +1 -1
  12. package/dist/adapters/index.d.mts +140 -2
  13. package/dist/adapters/index.d.ts +140 -2
  14. package/dist/adapters/index.js +864 -31
  15. package/dist/adapters/index.js.map +1 -1
  16. package/dist/adapters/index.mjs +863 -31
  17. package/dist/adapters/index.mjs.map +1 -1
  18. package/dist/adapters/memory.adapter.d.mts +2 -2
  19. package/dist/adapters/memory.adapter.d.ts +2 -2
  20. package/dist/adapters/memory.adapter.js +122 -30
  21. package/dist/adapters/memory.adapter.js.map +1 -1
  22. package/dist/adapters/memory.adapter.mjs +121 -29
  23. package/dist/adapters/memory.adapter.mjs.map +1 -1
  24. package/dist/index.d.mts +452 -342
  25. package/dist/index.d.ts +452 -342
  26. package/dist/index.js +1923 -1002
  27. package/dist/index.js.map +1 -1
  28. package/dist/index.mjs +1921 -1001
  29. package/dist/index.mjs.map +1 -1
  30. package/dist/shim.d.mts +36 -0
  31. package/dist/shim.d.ts +36 -0
  32. package/dist/shim.js +75 -0
  33. package/dist/shim.js.map +1 -0
  34. package/dist/shim.mjs +67 -0
  35. package/dist/shim.mjs.map +1 -0
  36. package/dist/telemetry/index.d.mts +281 -0
  37. package/dist/telemetry/index.d.ts +281 -0
  38. package/dist/telemetry/index.js +97 -0
  39. package/dist/telemetry/index.js.map +1 -0
  40. package/dist/telemetry/index.mjs +95 -0
  41. package/dist/telemetry/index.mjs.map +1 -0
  42. package/package.json +44 -11
package/AGENTS.md CHANGED
@@ -1,131 +1,1153 @@
1
1
  # AGENTS.md - @igniter-js/jobs
2
2
 
3
- > **Last Updated:** 2025-12-15
4
- > **Version:** 0.0.1 (bootstrap)
3
+ > **Last Updated:** 2026-01-29
4
+ > **Version:** 1.0.0-alpha.0
5
+ > **Goal:** This document serves as the complete operational manual for Code Agents maintaining and consuming the @igniter-js/jobs package.
5
6
 
6
- This file is the operations manual for AI agents maintaining `@igniter-js/jobs`. Follow these instructions before changing any code.
7
+ ---
7
8
 
8
- ## 1. Purpose
9
+ ## 1. Package Vision & Context
9
10
 
10
- `@igniter-js/jobs` delivers a fluent builder API for background jobs, workers, and queue management. It mirrors the DX of `@igniter-js/store` and `@igniter-js/telemetry`, wraps adapter implementations (BullMQ first, memory for tests), and exposes typed events, scopes, and management APIs.
11
+ `@igniter-js/jobs` is the definitive background processing and task scheduling solution for the Igniter.js ecosystem. It transforms the often-unpredictable world of distributed queues into a type-safe, observable, and highly structured domain that feels like a native extension of the application's business logic.
11
12
 
12
- ## 2. Directory Map
13
+ ### Why This Package Exists
13
14
 
15
+ In modern distributed systems, background processing is usually the first point of failure and the last point of visibility. Traditional queue libraries often suffer from:
16
+
17
+ - **Type Erasure:** Job payloads are treated as `any` or raw JSON, leading to "poison pill" jobs that crash workers after being enqueued from a different version of the app.
18
+ - **Context Isolation:** Accessing core resources like database connections, mailers, or configurations inside a worker usually requires brittle global variables or complex dependency injection hacks.
19
+ - **Observability Gaps:** Tracking a job from the moment it's enqueued in an API request to its final completion on a remote worker often requires manual, inconsistent instrumentation.
20
+ - **Multi-Tenant Leakage:** In SaaS environments, it's dangerously easy for a job intended for Organization A to leak context or data into Organization B's processing loop.
21
+
22
+ ### The Igniter.js Solution
23
+
24
+ `@igniter-js/jobs` solves these problems by enforcing a strict **Infrastructure-as-Code** approach to background tasks:
25
+
26
+ 1. **End-to-End Type Safety:** By using TypeScript's powerful inference engine, the package ensures that the `input` passed to `.dispatch()` is exactly what the `handler` expects.
27
+ 2. **Resource-Aware Context:** The `withContext()` factory ensures that every job execution starts with a fresh, validated set of resources, exactly like a standard API request.
28
+ 3. **Multi-Platform Support (Adapters):** Whether using **BullMQ** for massive production scale or an **In-Memory** adapter for unit tests, the business logic remains identical.
29
+ 4. **Native Observability:** Telemetry and internal event publishing are "baked in," providing immediate visibility into enqueuing, execution times, failures, and retries without writing a single line of logging code.
30
+
31
+ ---
32
+
33
+ ## I. MAINTAINER GUIDE (Internal Architecture)
34
+
35
+ ### 2. FileSystem Topology (Maintenance)
36
+
37
+ Maintainers must respect the following directory structure and responsibilities:
38
+
39
+ - **`src/builders/`**: The configuration factory.
40
+ - `main.builder.ts`: `IgniterJobsBuilder`. Manages immutable state accumulation and generic type narrowing for context and queues.
41
+ - `queue.builder.ts`: `IgniterQueueBuilder`. Fluent API for defining jobs and crons within a queue.
42
+ - `worker.builder.ts`: `IgniterWorkerBuilder`. Configures worker concurrency and lifecycle handlers.
43
+ - **`src/core/`**: The runtime heart.
44
+ - `manager.ts`: `IgniterJobsManager`. Implements the proxy-based API and the handler-wrapping logic.
45
+ - `queue.ts`: `IgniterQueue` facade for the queue builder.
46
+ - **`src/adapters/`**: The infrastructure boundary.
47
+ - `memory.adapter.ts`: Production-grade in-memory implementation for tests (non-persistent).
48
+ - `sqlite.adapter.ts`: SQLite-based persistent adapter for local/desktop/CLI environments.
49
+ - `bullmq.adapter.ts`: Reference to external `@igniter-js/adapter-bullmq` for production Redis-based queues.
50
+ - **`src/errors/`**: Resiliency definitions.
51
+ - `jobs.error.ts`: `IgniterJobsError`. Extends `IgniterError` with metadata-rich payloads and authoritative error codes.
52
+ - **`src/telemetry/`**: Observability registry.
53
+ - `index.ts`: defines Zod-validated telemetry events for the package.
54
+ - **`src/types/`**: Pure contract definitions.
55
+ - `adapter.ts`: `IgniterJobsAdapter` interface.
56
+ - `runtime.ts`: Accessor interfaces for the proxy API.
57
+ - `job.ts`: authoritative types for job definitions, contexts, and hooks.
58
+ - **`src/utils/`**: Static helper library.
59
+ - `events.utils.ts`: Pub/sub event helpers and channel composition.
60
+ - `id-generator.ts`: Deterministic ID utilities for adapters.
61
+ - `prefix.ts`: Consistency logic for queue names and event channels.
62
+ - `scope.ts`: Scope merging and extraction utilities.
63
+ - `telemetry.ts`: Telemetry helper utilities.
64
+ - `validation.ts`: Runtime schema validation utilities.
65
+
66
+ ### 3. Architecture Deep-Dive
67
+
68
+ #### 3.1 The Handler-Wrapper Pattern
69
+
70
+ The core reliability of the package comes from the fact that user handlers are never called directly by the adapter. Instead, `core/manager.ts` wraps every handler in a sophisticated pipeline:
71
+
72
+ 1. **Telemetry Injection:** A "Started" event is emitted before the handler runs.
73
+ 2. **Context Resolution:** The `contextFactory` provided in the builder is invoked to create the execution environment.
74
+ 3. **Metadata Unpacking:** The scope (tenant ID) is extracted from the job metadata and injected into the context.
75
+ 4. **Schema Enforcement:** If an `input` schema is defined, the payload is validated _inside the worker_ before being passed to the handler.
76
+ 5. **Lifecycle Management:** The wrapper catches errors, determines if it was the final attempt, and emits the corresponding "Success" or "Failure" telemetry.
77
+
78
+ #### 3.2 Proxy-Based Accessors
79
+
80
+ The `IgniterJobsRuntime` uses a dynamic property accessor pattern. When you call `jobs.email.sendWelcome`, you aren't accessing a hardcoded property. Instead, the runtime maps the `email` queue and `sendWelcome` job from its internal configuration. This allows for a fluent, "discoverable" API that is 100% type-safe without needing code generation.
81
+
82
+ ### 4. Operational Flow Mapping (Pipelines)
83
+
84
+ #### 4.1 Method: `IgniterJobsBuilder.build()`
85
+
86
+ 1. **Argument Validation:** Adapter, Service, Environment, and Context Factory must be present.
87
+ 2. **Internal Logic:** Builds `IgniterJobsConfig` from builder state.
88
+ 3. **Runtime Creation:** Instantiates `IgniterJobsManager` and calls `.toRuntime()`.
89
+ 4. **Adapter Registration:** `IgniterJobsManager.ensureRegistered()` registers jobs and crons on the adapter.
90
+ 5. **Result Formatting:** Returns a proxy runtime with queue/job accessors.
91
+
92
+ #### 4.2 Method: `job.dispatch(params)`
93
+
94
+ 1. **Argument Validation:** Validates `input` against job schema.
95
+ 2. **Telemetry (Enqueued):** Emits `igniter.jobs.job.enqueued` when telemetry is configured.
96
+ 3. **Internal Logic:** Calls `IgniterJobsScopeUtils.mergeMetadataWithScope`.
97
+ 4. **Adapter Call:** Calls `adapter.dispatch()`.
98
+ 5. **Telemetry (Success):** Emits success event with `jobId`.
99
+
100
+ #### 4.3 Method: `worker.start()`
101
+
102
+ 1. **Internal Logic:** Consolidates queues, handlers, and concurrency.
103
+ 2. **Adapter Call:** Calls `adapter.createWorker()`.
104
+ 3. **Result Formatting:** Returns a worker handle for control.
105
+
106
+ #### 4.4 Method: `queue.pause()`
107
+
108
+ 1. **Internal Logic:** Resolves queue name.
109
+ 2. **Adapter Call:** Calls `adapter.pauseQueue()`.
110
+
111
+ #### 4.5 Method: `job.get(id).retrieve()`
112
+
113
+ 1. **Internal Logic:** Resolves queue and job name.
114
+ 2. **Adapter Call:** Calls `adapter.getJob()`.
115
+ 3. **Result Formatting:** Maps adapter result to `JobSearchResult`.
116
+
117
+ #### 4.6 Method: `job.get(id).retry()`
118
+
119
+ 1. **Adapter Call:** Calls `adapter.retryJob()`.
120
+ 2. **Result:** Adapter handles retry; telemetry emission depends on adapter implementation.
121
+
122
+ ### 5. Dependency & Type Graph
123
+
124
+ The package is designed for maximum portability with minimal dependency creep.
125
+
126
+ - **`@igniter-js/common`**: For base `IgniterError`, `IgniterLogger`, and the `StandardSchemaV1` interface.
127
+ - **`@igniter-js/telemetry`**: Optional peer dependency for operational monitoring.
128
+
129
+ Type Flow:
130
+ `IgniterJobsBuilder` -> `IgniterJobsConfig` -> `IgniterJobsRuntime` -> `QueueAccessor` -> `JobAccessor`
131
+
132
+ ### 6. Maintenance Checklist
133
+
134
+ 1. **Parity Check:** If you add a new feature to the `IgniterJobsAdapter` interface, ensure it is implemented in `memory.adapter.ts`.
135
+ 2. **Telemetry Audit:** Job lifecycle emits telemetry in `core/manager.ts`. Queue/worker telemetry depends on adapter behavior.
136
+ 3. **Error Propagation:** Ensure that errors thrown in user hooks (`onSuccess`, `onFailure`) do not crash the worker loop.
137
+ 4. **Inference Test:** Run `npm run typecheck` and verify that the `main.builder.spec.ts` can still autocomplete queue names.
138
+
139
+ ### 7. Maintainer Troubleshooting
140
+
141
+ #### Issue: Context factory is called too many times
142
+
143
+ - **Check:** Is the adapter caching the context factory result per job execution?
144
+ - **Fix:** Move factory invocation to the wrapper logic in `core/manager.ts`.
145
+
146
+ #### Issue: Telemetry attributes are missing
147
+
148
+ - **Check:** Are the attributes prefixed with `ctx.job.` or `ctx.worker.`?
149
+ - **Fix:** Follow the schema in `telemetry/index.ts`.
150
+
151
+ ---
152
+
153
+ ## II. CONSUMER GUIDE (Developer Manual)
154
+
155
+ ### 8. Distribution Anatomy (Consumption)
156
+
157
+ The package provides organized subpath exports for optimized bundling:
158
+
159
+ - **`@igniter-js/jobs`**: The main entry point. Use for `IgniterJobs` and `IgniterQueue`.
160
+ - **`@igniter-js/jobs/adapters`**: Built-in adapters (Memory, SQLite, BullMQ wrapper).
161
+ - **`@igniter-js/jobs/telemetry`**: Telemetry registry for registration.
162
+
163
+ ### 9. Quick Start & Common Patterns
164
+
165
+ #### Pattern: The Singleton Instance
166
+
167
+ Setup your jobs in a central `src/services/jobs.ts` file:
168
+
169
+ ```typescript
170
+ import { IgniterJobs } from "@igniter-js/jobs";
171
+ import { IgniterJobsBullMQAdapter } from "@igniter-js/jobs/adapters";
172
+
173
+ export const jobs = IgniterJobs.create()
174
+ .withAdapter(IgniterJobsBullMQAdapter.create({ redis }))
175
+ .withService("api")
176
+ .withEnvironment("production")
177
+ .withContext(async () => ({ db }))
178
+ .addQueue(myQueue)
179
+ .build();
180
+ ```
181
+
182
+ #### Pattern: Scheduled Retries
183
+
184
+ Prevent permanent failures in external API calls:
185
+
186
+ ```typescript
187
+ .addJob("sync", {
188
+ attempts: 5,
189
+ delay: 5000,
190
+ handler: async () => { /* ... */ }
191
+ })
192
+ ```
193
+
194
+ ### 10. Real-World Use Case Library
195
+
196
+ #### Case 1: E-commerce Order Expiry
197
+
198
+ - **Scenario:** Cancel unpaid orders after 1 hour.
199
+ - **Implementation:** `jobs.orders.cancel.schedule({ input: { id }, delay: 3600000 })`.
200
+
201
+ #### Case 2: Fintech Nightly Reconciliation
202
+
203
+ - **Scenario:** Verify 1M transactions against bank API.
204
+ - **Implementation:** Cron job at 3 AM. Dispatches batch jobs with 100 transactions each.
205
+
206
+ #### Case 3: Social Media Media Transcoding
207
+
208
+ - **Scenario:** Generate 5 video qualities after upload.
209
+ - **Implementation:** High-priority queue for 360p, low-priority for 4k.
210
+
211
+ #### Case 4: Healthcare Appointment Reminders
212
+
213
+ - **Scenario:** Send SMS 24h before appointment.
214
+ - **Implementation:** `.schedule({ at: reminderDate })`.
215
+
216
+ #### Case 5: SaaS Multi-Tenant CSV Import
217
+
218
+ - **Scenario:** Process 100k row CSV.
219
+ - **Implementation:** `job.progress(percent)` every 100 rows for real-time progress bar.
220
+
221
+ ### 11. Domain-Specific Guidance
222
+
223
+ - **Financial Systems:** Always use `attempts` and `backoff`. Implement `onFailure` to alert engineering.
224
+ - **High-Volume Caching:** Use `delay` to batch invalidation events.
225
+ - **AI Processing:** Set concurrency to match your GPU/API limits.
226
+
227
+ ### 12. Best Practices & Anti-Patterns
228
+
229
+ | Practice | Why? | Example |
230
+ | :------------------- | :------------------- | :-------------------------------------- |
231
+ | ✅ Small Payloads | Reduces overhead. | `input: { id: '123' }` |
232
+ | ✅ Context Injection | Keeps handlers pure. | `withContext(() => ({ db }))` |
233
+ | ✅ Idempotency | Jobs WILL retry. | Check status before taking action. |
234
+ | ❌ Sync I/O | Kills throughput. | `readFileSync(...)` |
235
+ | ❌ Global DB | Untestable. | `import { db } from './db'` in handler. |
236
+
237
+ ---
238
+
239
+ ## III. TECHNICAL REFERENCE & RESILIENCE
240
+
241
+ ### 13. Exhaustive API Reference
242
+
243
+ | Class/Method | Parameters | Returns | Description |
244
+ | :--------------------- | :--------------- | :---------------- | :--------------------- |
245
+ | `IgniterJobs.create()` | - | `Builder` | Starts the fluent API. |
246
+ | `dispatch()` | `params` | `Promise<string>` | Enqueues a job. |
247
+ | `schedule()` | `params` | `Promise<string>` | Schedules a job. |
248
+ | `search()` | `target, filter` | `Promise<T[]>` | Management API. |
249
+ | `worker.create()` | - | `WorkerBuilder` | Configures a worker. |
250
+
251
+ ### 14. Telemetry & Observability Registry
252
+
253
+ | Event Name | Group | Meaning |
254
+ | :--------------- | :----- | :---------------- |
255
+ | `job.enqueued` | job | Task accepted. |
256
+ | `job.started` | job | Task started. |
257
+ | `job.completed` | job | Task finished. |
258
+ | `job.failed` | job | Task threw error. |
259
+ | `worker.started` | worker | Worker is online. |
260
+
261
+ ### 15. Troubleshooting & Error Code Library
262
+
263
+ #### JOBS_ADAPTER_REQUIRED
264
+
265
+ - **Context:** `build()`.
266
+ - **Cause:** No adapter provided.
267
+ - **Solution:** Call `.withAdapter()`.
268
+
269
+ #### JOBS_SERVICE_REQUIRED
270
+
271
+ - **Context:** `build()`.
272
+ - **Cause:** No service name.
273
+ - **Solution:** Call `.withService()`.
274
+
275
+ #### JOBS_CONTEXT_REQUIRED
276
+
277
+ - **Context:** `build()`.
278
+ - **Cause:** No context factory.
279
+ - **Solution:** Call `.withContext()`.
280
+
281
+ #### JOBS_CONFIGURATION_INVALID
282
+
283
+ - **Context:** Various.
284
+ - **Cause:** Missing environment or invalid scope.
285
+ - **Solution:** Follow builder hints.
286
+
287
+ #### JOBS_QUEUE_NOT_FOUND
288
+
289
+ - **Context:** Worker creation.
290
+ - **Cause:** Wrong queue name.
291
+ - **Solution:** Check `addQueue()` vs `worker.addQueue()`.
292
+
293
+ #### JOBS_QUEUE_DUPLICATE
294
+
295
+ - **Context:** Initialization.
296
+ - **Cause:** Same name twice.
297
+ - **Solution:** Use unique names.
298
+
299
+ #### JOBS_INVALID_DEFINITION
300
+
301
+ - **Context:** Job registration.
302
+ - **Cause:** Malformed job object.
303
+ - **Solution:** Ensure `handler` exists.
304
+
305
+ #### JOBS_HANDLER_REQUIRED
306
+
307
+ - **Context:** Job registration.
308
+ - **Cause:** Handler is null.
309
+ - **Solution:** Provide async function.
310
+
311
+ #### JOBS_DUPLICATE_JOB
312
+
313
+ - **Context:** Queue building.
314
+ - **Cause:** Two jobs with same name.
315
+ - **Solution:** Rename one.
316
+
317
+ #### JOBS_NOT_FOUND
318
+
319
+ - **Context:** `get(id)`.
320
+ - **Cause:** Invalid ID.
321
+ - **Solution:** Verify ID or check retention.
322
+
323
+ #### JOBS_NOT_REGISTERED
324
+
325
+ - **Context:** Worker execution.
326
+ - **Cause:** Worker doesn't know job.
327
+ - **Solution:** Check runtime config parity.
328
+
329
+ #### JOBS_EXECUTION_FAILED
330
+
331
+ - **Context:** Worker.
332
+ - **Cause:** Handler threw error.
333
+ - **Solution:** Check business logic.
334
+
335
+ #### JOBS_TIMEOUT
336
+
337
+ - **Context:** Worker.
338
+ - **Cause:** Job took too long.
339
+ - **Solution:** Increase `timeout` in job opts.
340
+
341
+ #### JOBS_CONTEXT_FACTORY_FAILED
342
+
343
+ - **Context:** Worker.
344
+ - **Cause:** Factory threw error.
345
+ - **Solution:** Check DB/resource health.
346
+
347
+ #### JOBS_VALIDATION_FAILED
348
+
349
+ - **Context:** `dispatch`.
350
+ - **Cause:** Schema mismatch.
351
+ - **Solution:** Correct the input.
352
+
353
+ #### JOBS_INVALID_INPUT
354
+
355
+ - **Context:** Runtime.
356
+ - **Cause:** Malformed data.
357
+ - **Solution:** Validate early.
358
+
359
+ #### JOBS_INVALID_CRON
360
+
361
+ - **Context:** Registration.
362
+ - **Cause:** Bad syntax.
363
+ - **Solution:** Check cron expression.
364
+
365
+ #### JOBS_INVALID_SCHEDULE
366
+
367
+ - **Context:** `schedule`.
368
+ - **Cause:** Past date.
369
+ - **Solution:** Use future date.
370
+
371
+ #### JOBS_SCOPE_ALREADY_DEFINED
372
+
373
+ - **Context:** Builder.
374
+ - **Cause:** Two scopes added.
375
+ - **Solution:** Only one allowed.
376
+
377
+ #### JOBS_WORKER_FAILED
378
+
379
+ - **Context:** Worker.
380
+ - **Cause:** Connection failure.
381
+ - **Solution:** Check Redis health.
382
+
383
+ #### JOBS_ADAPTER_ERROR
384
+
385
+ - **Context:** Generic.
386
+ - **Cause:** Backend failure.
387
+ - **Solution:** Check Redis/Adapter logs.
388
+
389
+ #### JOBS_ADAPTER_CONNECTION_FAILED
390
+
391
+ - **Context:** Generic.
392
+ - **Cause:** Redis down.
393
+ - **Solution:** Restore connectivity.
394
+
395
+ #### JOBS_SUBSCRIBE_FAILED
396
+
397
+ - **Context:** `subscribe`.
398
+ - **Cause:** Pub/Sub failure.
399
+ - **Solution:** Check Redis ACLs.
400
+
401
+ ---
402
+
403
+ ## IV. ADAPTERS REFERENCE
404
+
405
+ The `@igniter-js/jobs` package supports multiple adapters for different use cases and environments. This section provides comprehensive documentation for each adapter.
406
+
407
+ ### 16. Adapter Architecture Overview
408
+
409
+ All adapters implement the `IgniterJobsAdapter` interface defined in `src/types/adapter.ts`. This interface ensures consistent behavior across different storage backends.
410
+
411
+ ```
412
+ ┌─────────────────────────────────────────────────────────────────┐
413
+ │ IgniterJobsAdapter Interface │
414
+ ├─────────────────────────────────────────────────────────────────┤
415
+ │ Job Management │
416
+ │ ├── registerJob(queue, name, definition) │
417
+ │ ├── registerCron(queue, name, definition) │
418
+ │ ├── dispatch(params) → jobId │
419
+ │ ├── schedule(params) → jobId │
420
+ │ ├── getJob(id) → job | null │
421
+ │ ├── getJobState(id) → status | null │
422
+ │ ├── getJobLogs(id) → logs[] │
423
+ │ ├── getJobProgress(id) → number │
424
+ │ ├── removeJob(id) │
425
+ │ ├── retryJob(id) │
426
+ │ ├── promoteJob(id) │
427
+ │ └── moveJobToFailed(id, error) │
428
+ ├─────────────────────────────────────────────────────────────────┤
429
+ │ Queue Management │
430
+ │ ├── listQueues() → queue[] │
431
+ │ ├── getQueueInfo(name) → info | null │
432
+ │ ├── getQueueJobCounts(name) → counts │
433
+ │ ├── pauseQueue(name) │
434
+ │ ├── resumeQueue(name) │
435
+ │ ├── drainQueue(name) → removed │
436
+ │ ├── cleanQueue(name, options) → cleaned │
437
+ │ ├── obliterateQueue(name, options) │
438
+ │ └── retryAllInQueue(name) → retried │
439
+ ├─────────────────────────────────────────────────────────────────┤
440
+ │ Worker Management │
441
+ │ ├── createWorker(config) → WorkerHandle │
442
+ │ ├── getWorkers() → Map<id, worker> │
443
+ │ └── searchWorkers(filter) → workers[] │
444
+ ├─────────────────────────────────────────────────────────────────┤
445
+ │ Search & Discovery │
446
+ │ ├── searchJobs(filter) → jobs[] │
447
+ │ └── searchQueues(filter) → queues[] │
448
+ ├─────────────────────────────────────────────────────────────────┤
449
+ │ Pub/Sub │
450
+ │ ├── publishEvent(channel, data) │
451
+ │ └── subscribeEvent(channel, handler) → unsubscribe │
452
+ ├─────────────────────────────────────────────────────────────────┤
453
+ │ Lifecycle │
454
+ │ └── shutdown() │
455
+ └─────────────────────────────────────────────────────────────────┘
456
+ ```
457
+
458
+ ### 17. Adapter Comparison Matrix
459
+
460
+ | Feature | Memory | SQLite | BullMQ |
461
+ |---------|--------|--------|--------|
462
+ | **Persistence** | ❌ In-memory only | ✅ File-based | ✅ Redis |
463
+ | **Multi-process** | ❌ Single process | ⚠️ Single process (WAL) | ✅ Distributed |
464
+ | **Installation** | Built-in | `better-sqlite3` | `bullmq` + Redis |
465
+ | **Use Case** | Tests, demos | Desktop, CLI, MCP | Production, scale |
466
+ | **Job Limits** | Memory-bound | Disk-bound | Redis-bound |
467
+ | **Worker Mechanism** | Immediate | Polling-based | Event-driven |
468
+ | **Cron Jobs** | ⚠️ Basic | ⚠️ Basic | ✅ Full |
469
+ | **Distributed Locking** | ❌ | ❌ | ✅ |
470
+ | **Rate Limiting** | ⚠️ Basic | ⚠️ Basic | ✅ Advanced |
471
+
472
+ ---
473
+
474
+ ### 18. Memory Adapter (Testing & Development)
475
+
476
+ The `IgniterJobsMemoryAdapter` provides a full-featured in-memory implementation perfect for unit tests and quick development.
477
+
478
+ #### 18.1 When to Use
479
+
480
+ - **Unit Tests:** Fast, isolated, no external dependencies.
481
+ - **Integration Tests:** When you need predictable job behavior.
482
+ - **Development:** Quick experimentation without Redis.
483
+ - **CI/CD Pipelines:** Zero-config test environments.
484
+
485
+ #### 18.2 Installation
486
+
487
+ No additional dependencies required. The adapter is built into `@igniter-js/jobs`.
488
+
489
+ #### 18.3 Basic Usage
490
+
491
+ ```typescript
492
+ import { IgniterJobs } from "@igniter-js/jobs";
493
+ import { IgniterJobsMemoryAdapter } from "@igniter-js/jobs/adapters";
494
+
495
+ const adapter = IgniterJobsMemoryAdapter.create();
496
+
497
+ const jobs = IgniterJobs.create()
498
+ .withAdapter(adapter)
499
+ .withService("test")
500
+ .withContext(async () => ({ db: mockDb }))
501
+ .addQueue(emailQueue)
502
+ .build();
503
+ ```
504
+
505
+ #### 18.4 Configuration Options
506
+
507
+ ```typescript
508
+ interface IgniterJobsMemoryAdapterOptions {
509
+ /** Maximum jobs to retain in history (default: 1000) */
510
+ maxJobHistory?: number;
511
+ }
14
512
  ```
15
- packages/jobs/
16
- ├── src/
17
- │ ├── adapters/ # Adapter implementations + barrel
18
- │ ├── builders/ # Fluent builders (jobs, queue, worker)
19
- │ ├── core/ # Runtime classes & proxies
20
- │ ├── errors/ # IgniterJobsError hierarchy
21
- │ ├── types/ # Public type surface (adapter/config/events/queue/worker)
22
- │ └── utils/ # Static utility classes (id/prefix helpers)
23
- ├── AGENTS.md
24
- # AGENTS.md - @igniter-js/jobs
25
513
 
26
- > **Last Updated:** 2025-12-16
27
- > **Version:** 0.0.1 (bootstrap)
514
+ #### 18.5 Testing Patterns
28
515
 
29
- Operations manual for AI agents maintaining `@igniter-js/jobs`. Read this before changing code.
516
+ ```typescript
517
+ // Access internal state for assertions
518
+ const adapter = IgniterJobsMemoryAdapter.create();
30
519
 
31
- ## 1. Mission & Scope
520
+ // Dispatch a job
521
+ const jobId = await adapter.dispatch({
522
+ queue: "email",
523
+ jobName: "send",
524
+ input: { to: "test@example.com" },
525
+ });
32
526
 
33
- `@igniter-js/jobs` provides a fluent, type-safe API for defining queues, jobs, workers, and scheduling. It mirrors `@igniter-js/store` and `@igniter-js/telemetry` patterns, wraps adapters (BullMQ primary, memory for tests), and emits typed telemetry for observability. Scope includes: builders, runtime, adapters, telemetry glue, and docs/tests for everything public.
527
+ // Verify job was created
528
+ const job = await adapter.getJob(jobId);
529
+ expect(job?.status).toBe("waiting");
34
530
 
35
- ## 2. Directory Map & Responsibilities
531
+ // Simulate worker processing
532
+ const worker = await adapter.createWorker({
533
+ queues: ["email"],
534
+ concurrency: 1,
535
+ });
36
536
 
537
+ // Wait for processing
538
+ await new Promise((r) => setTimeout(r, 100));
539
+
540
+ // Verify completion
541
+ const completed = await adapter.getJob(jobId);
542
+ expect(completed?.status).toBe("completed");
543
+
544
+ // Cleanup
545
+ await worker.close();
546
+ ```
547
+
548
+ #### 18.6 Limitations
549
+
550
+ - **Non-persistent:** All data lost on process restart.
551
+ - **Single-process:** Cannot share state across workers.
552
+ - **Memory-bound:** Large job volumes may cause OOM.
553
+
554
+ ---
555
+
556
+ ### 19. SQLite Adapter (Desktop, CLI, MCP Servers)
557
+
558
+ The `IgniterJobsSQLiteAdapter` provides persistent job storage using SQLite, ideal for local environments where Redis is impractical.
559
+
560
+ #### 19.1 When to Use
561
+
562
+ - **Desktop Applications:** Tauri, Electron apps needing background tasks.
563
+ - **CLI Tools:** Long-running commands with progress tracking.
564
+ - **MCP Servers:** Model Context Protocol servers with job queues.
565
+ - **Edge Deployments:** Single-node deployments.
566
+ - **Local Development:** When you want jobs to survive restarts.
567
+
568
+ #### 19.2 Installation
569
+
570
+ ```bash
571
+ npm install better-sqlite3
572
+ # or
573
+ bun add better-sqlite3
574
+ ```
575
+
576
+ The `better-sqlite3` package is an optional peer dependency.
577
+
578
+ #### 19.3 Basic Usage
579
+
580
+ ```typescript
581
+ import { IgniterJobs } from "@igniter-js/jobs";
582
+ import { IgniterJobsSQLiteAdapter } from "@igniter-js/jobs/adapters";
583
+
584
+ // File-based (persistent)
585
+ const adapter = IgniterJobsSQLiteAdapter.create({
586
+ path: "./jobs.sqlite",
587
+ });
588
+
589
+ // In-memory (for tests)
590
+ const memoryAdapter = IgniterJobsSQLiteAdapter.create({
591
+ path: ":memory:",
592
+ });
593
+
594
+ const jobs = IgniterJobs.create()
595
+ .withAdapter(adapter)
596
+ .withService("desktop-app")
597
+ .withContext(async () => ({ settings: loadSettings() }))
598
+ .addQueue(syncQueue)
599
+ .build();
600
+ ```
601
+
602
+ #### 19.4 Configuration Options
603
+
604
+ ```typescript
605
+ interface IgniterJobsSQLiteAdapterOptions {
606
+ /**
607
+ * Path to the SQLite database file.
608
+ * Use ":memory:" for in-memory database.
609
+ * @example "./data/jobs.sqlite"
610
+ */
611
+ path: string;
612
+
613
+ /**
614
+ * Polling interval in milliseconds for workers to check for new jobs.
615
+ * Lower values = more responsive, higher CPU usage.
616
+ * @default 500
617
+ */
618
+ pollingInterval?: number;
619
+
620
+ /**
621
+ * Enable WAL (Write-Ahead Logging) mode for better concurrent performance.
622
+ * Recommended for production use.
623
+ * @default true
624
+ */
625
+ enableWAL?: boolean;
626
+ }
627
+ ```
628
+
629
+ #### 19.5 Database Schema
630
+
631
+ The adapter automatically creates and manages the following tables:
632
+
633
+ ```sql
634
+ -- Main jobs table
635
+ CREATE TABLE IF NOT EXISTS jobs (
636
+ id TEXT PRIMARY KEY,
637
+ queue TEXT NOT NULL,
638
+ name TEXT NOT NULL,
639
+ input TEXT, -- JSON serialized
640
+ result TEXT, -- JSON serialized
641
+ error TEXT,
642
+ status TEXT NOT NULL DEFAULT 'waiting',
643
+ progress INTEGER DEFAULT 0,
644
+ priority INTEGER DEFAULT 0,
645
+ attempts_made INTEGER DEFAULT 0,
646
+ max_attempts INTEGER DEFAULT 1,
647
+ delay_until INTEGER, -- Unix timestamp
648
+ metadata TEXT, -- JSON serialized
649
+ scope_type TEXT,
650
+ scope_id TEXT,
651
+ created_at INTEGER NOT NULL,
652
+ started_at INTEGER,
653
+ completed_at INTEGER
654
+ );
655
+
656
+ -- Indexes for efficient queries
657
+ CREATE INDEX IF NOT EXISTS idx_jobs_queue_status ON jobs(queue, status);
658
+ CREATE INDEX IF NOT EXISTS idx_jobs_status ON jobs(status);
659
+ CREATE INDEX IF NOT EXISTS idx_jobs_priority ON jobs(priority DESC);
660
+ CREATE INDEX IF NOT EXISTS idx_jobs_delay ON jobs(delay_until);
661
+
662
+ -- Job logs table
663
+ CREATE TABLE IF NOT EXISTS job_logs (
664
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
665
+ job_id TEXT NOT NULL,
666
+ level TEXT NOT NULL,
667
+ message TEXT NOT NULL,
668
+ timestamp INTEGER NOT NULL,
669
+ FOREIGN KEY (job_id) REFERENCES jobs(id) ON DELETE CASCADE
670
+ );
671
+
672
+ -- Paused queues tracking
673
+ CREATE TABLE IF NOT EXISTS paused_queues (
674
+ name TEXT PRIMARY KEY,
675
+ paused_at INTEGER NOT NULL
676
+ );
677
+ ```
678
+
679
+ #### 19.6 Worker Polling Mechanism
680
+
681
+ Unlike event-driven adapters (BullMQ), the SQLite adapter uses a polling-based approach:
682
+
683
+ ```
684
+ ┌─────────────────────────────────────────────────────────────────┐
685
+ │ Worker Polling Loop │
686
+ ├─────────────────────────────────────────────────────────────────┤
687
+ │ │
688
+ │ ┌─────────┐ ┌─────────────┐ ┌─────────────┐ │
689
+ │ │ Wait │────▶│ Check Jobs │────▶│ Process Job │ │
690
+ │ │ Polling │ │ Available? │ │ (if any) │ │
691
+ │ │ Interval│ └─────────────┘ └─────────────┘ │
692
+ │ └─────────┘ │ │ │
693
+ │ ▲ │ No jobs │ Done │
694
+ │ │ │ ▼ │
695
+ │ └────────────────┴───────────────────┘ │
696
+ │ │
697
+ │ Query Priority: │
698
+ │ 1. Delayed jobs ready to run (delay_until <= NOW) │
699
+ │ 2. Waiting jobs, ordered by priority DESC │
700
+ │ 3. Retry jobs (failed but attempts < max_attempts) │
701
+ │ │
702
+ └─────────────────────────────────────────────────────────────────┘
703
+ ```
704
+
705
+ #### 19.7 Complete Example
706
+
707
+ ```typescript
708
+ import { IgniterJobsSQLiteAdapter } from "@igniter-js/jobs/adapters";
709
+
710
+ async function main() {
711
+ // 1. Create adapter
712
+ const adapter = IgniterJobsSQLiteAdapter.create({
713
+ path: "./my-app-jobs.sqlite",
714
+ pollingInterval: 500,
715
+ enableWAL: true,
716
+ });
717
+
718
+ // 2. Register jobs
719
+ adapter.registerJob("sync", "upload", {
720
+ handler: async ({ input }) => {
721
+ const { fileId } = input as { fileId: string };
722
+ // Simulate upload
723
+ await uploadFile(fileId);
724
+ return { uploaded: true };
725
+ },
726
+ onProgress: async ({ progress }) => {
727
+ console.log(`Upload progress: ${progress}%`);
728
+ },
729
+ });
730
+
731
+ // 3. Create worker
732
+ const worker = await adapter.createWorker({
733
+ queues: ["sync"],
734
+ concurrency: 2,
735
+ handlers: {
736
+ onActive: ({ job }) => console.log(`Processing: ${job.id}`),
737
+ onSuccess: ({ job }) => console.log(`Completed: ${job.id}`),
738
+ onFailure: ({ job, error }) => console.error(`Failed: ${job.id}`, error),
739
+ },
740
+ });
741
+
742
+ // 4. Dispatch jobs
743
+ const jobId = await adapter.dispatch({
744
+ queue: "sync",
745
+ jobName: "upload",
746
+ input: { fileId: "file_123" },
747
+ priority: 10,
748
+ });
749
+
750
+ console.log(`Dispatched job: ${jobId}`);
751
+
752
+ // 5. Monitor status
753
+ const counts = await adapter.getQueueJobCounts("sync");
754
+ console.log(`Queue status:`, counts);
755
+
756
+ // 6. Graceful shutdown
757
+ process.on("SIGINT", async () => {
758
+ await worker.close();
759
+ await adapter.shutdown();
760
+ process.exit(0);
761
+ });
762
+ }
763
+
764
+ main().catch(console.error);
37
765
  ```
38
- packages/jobs/
39
- ├── src/
40
- │ ├── adapters/ # Adapter impls + barrel (bullmq, memory); capability parity with @igniter-js/adapter-bullmq
41
- │ ├── builders/ # Fluent builders: queue, worker, job, cron definitions
42
- │ ├── core/ # Runtime classes (queue runtime, worker runtime, proxies)
43
- │ ├── errors/ # IgniterJobsError hierarchy
44
- │ ├── telemetry/ # Event descriptors + emit helpers (if present)
45
- │ ├── types/ # Public surface (config, adapter contracts, events, queue/worker types)
46
- │ └── utils/ # Static helpers (ids, prefixes, validation)
47
- ├── AGENTS.md # You are here
48
- ├── CHANGELOG.md # Release notes
49
- ├── README.md # User-facing docs
50
- ├── package.json # Package metadata
51
- ├── tsconfig.json # TS build config
52
- ├── tsup.config.ts # Build outputs
53
- └── vitest.config.ts # Test config
766
+
767
+ #### 19.8 Persistence & Recovery
768
+
769
+ Jobs are automatically persisted and survive process restarts:
770
+
771
+ ```typescript
772
+ // Session 1: Dispatch job and exit
773
+ const adapter = IgniterJobsSQLiteAdapter.create({ path: "./jobs.sqlite" });
774
+ await adapter.dispatch({
775
+ queue: "emails",
776
+ jobName: "send",
777
+ input: { to: "user@example.com" },
778
+ });
779
+ await adapter.shutdown();
780
+ // Process exits
781
+
782
+ // Session 2: Jobs are still there
783
+ const adapter2 = IgniterJobsSQLiteAdapter.create({ path: "./jobs.sqlite" });
784
+ const jobs = await adapter2.searchJobs({ queue: "emails", status: ["waiting"] });
785
+ console.log(jobs); // Shows the pending job from session 1
54
786
  ```
55
787
 
56
- ## 3. Architecture Overview
788
+ #### 19.9 Limitations
57
789
 
58
- - **Builders (immutable):** `.create()` chain setters → `.build()`. Must copy state, never mutate in-place. Maintain type accumulation for scopes/options like `@igniter-js/store`.
59
- - **Runtime flow (happy path):**
60
- 1) User defines queue/worker via builders.
61
- 2) Adapter (BullMQ) wires queues/workers; memory adapter is for tests.
62
- 3) Job enqueue → adapter schedules → worker consumes → lifecycle events emit (internal + telemetry).
63
- 4) Errors bubble as `IgniterJobsError` with codes; runtime logs/telemetry should not crash workers.
64
- - **Telemetry:** Optional peer; events emitted via lightweight helper (fire-and-forget). Event names `igniter.jobs.<group>.<event>`; attributes prefixed with `ctx.`. No actors; single scope (e.g., queue/workspace) only.
65
- - **Scheduling:** `queue.addCron()` produces repeatable jobs; advanced scheduling metadata (skip weekends/business hours) attaches via `metadata.advancedScheduling`. BullMQ repeat options must be preserved.
790
+ - **Single Process:** WAL mode helps, but true concurrent writes from multiple processes may cause issues.
791
+ - **Polling Overhead:** Unlike event-driven systems, there's CPU overhead from polling.
792
+ - **No Distributed Locking:** Cannot guarantee exactly-once processing across processes.
793
+ - **Limited Cron:** Basic cron support without advanced scheduling features.
66
794
 
67
- ## 4. Design Rules (must-keep)
795
+ ---
68
796
 
69
- - **Prefixing:** Public classes/types start with `IgniterJobs` (singular `IgniterJob` when appropriate). Utils are static classes.
70
- - **Type Safety:** No implicit `any` on public API. Use `Prettify` helpers for flattened types. Align with `packages/core/src/types/jobs.interface.ts` and keep parity.
71
- - **Adapters:** `IgniterJobsAdapter` defines capabilities. BullMQ adapter must keep: rate limiting, scheduling (cron/repeat), retries/backoff, job metadata, progress, bulk ops, management APIs (pause/resume/drain/clean/obliterate), hooks/callbacks. Memory adapter is test-only.
72
- - **Docs:** TSDoc (English) with examples/edge cases for every exported symbol. Keep README + AGENTS + CHANGELOG in sync with behavior.
73
- - **Utils:** Keep shared helpers in `src/utils`; add tests. Prefer pure functions or static classes.
74
- - **Telemetry:** Optional dependency; interface lives in types to avoid hard dependency. Telemetry errors must never fail job execution.
797
+ ### 20. BullMQ Adapter (Production Scale)
75
798
 
76
- ## 5. Component Map (what to touch)
799
+ The `IgniterJobsBullMQAdapter` (external package) provides Redis-based job queuing for production environments.
77
800
 
78
- - **src/builders/**: Queue builder, worker builder, job definition builder. Ensure immutability and type accumulation. Provide `.withTelemetry()`, `.withAdapter()`, and scheduling helpers.
79
- - **src/core/**: Runtime wrappers that orchestrate builders + adapters. Handles lifecycle hooks, logging/telemetry, error translation.
80
- - **src/adapters/**: Adapter implementations + barrel. BullMQ mirrors `@igniter-js/adapter-bullmq`; memory adapter minimal but API-compatible for tests.
81
- - **src/errors/**: `IgniterJobsError` codes. Add new codes here first; keep messages actionable.
82
- - **src/telemetry/**: Event descriptors + emit helpers. Keep names stable and attributes prefixed `ctx.`.
83
- - **src/types/**: Single source of truth for public API (config, events, adapters, queue/worker types). Update before implementation.
84
- - **src/utils/**: ID generation, prefix handling, validation, backoff helpers. Pure and tested.
801
+ #### 20.1 When to Use
85
802
 
86
- ## 6. Telemetry Events (contract)
803
+ - **Production Deployments:** High availability and reliability.
804
+ - **Distributed Systems:** Multiple workers across nodes.
805
+ - **High Throughput:** Thousands of jobs per second.
806
+ - **Advanced Features:** Rate limiting, priorities, delayed jobs, cron.
87
807
 
88
- - **Namespace:** `igniter.jobs`.
89
- - **Job events:** `enqueued`, `scheduled`, `started`, `completed`, `failed`, `progress`, `retrying`.
90
- - **Worker events:** `started`, `stopped`, `idle`, `paused`, `resumed`.
91
- - **Queue events:** `paused`, `resumed`, `drained`, `cleaned`, `obliterated`.
92
- - **Attributes:** always include `ctx.job.id`, `ctx.job.queue`, plus duration/progress where applicable. Add `ctx.worker.id` for worker lifecycle.
93
- - **Behavior:** Emission is best-effort; never throw. Keep helper tolerant (swallow telemetry errors, log if logger available).
808
+ #### 20.2 Installation
94
809
 
95
- ## 7. Testing Strategy
810
+ ```bash
811
+ npm install @igniter-js/adapter-bullmq bullmq ioredis
812
+ ```
96
813
 
97
- - `npm test --filter @igniter-js/jobs`
98
- - `bun test` (if bun is configured in scripts)
99
- - `npx tsc --noEmit` for types
814
+ #### 20.3 Basic Usage
100
815
 
101
- ## 8. Build & Tooling
816
+ ```typescript
817
+ import { IgniterJobs } from "@igniter-js/jobs";
818
+ import { IgniterJobsBullMQAdapter } from "@igniter-js/adapter-bullmq";
819
+ import Redis from "ioredis";
820
+
821
+ const redis = new Redis(process.env.REDIS_URL);
822
+
823
+ const adapter = IgniterJobsBullMQAdapter.create({ redis });
824
+
825
+ const jobs = IgniterJobs.create()
826
+ .withAdapter(adapter)
827
+ .withService("api-server")
828
+ .withContext(async () => ({ db, cache }))
829
+ .addQueue(emailQueue)
830
+ .addQueue(analyticsQueue)
831
+ .build();
832
+ ```
102
833
 
103
- - **Build:** `bun run build` or `npm run build --filter @igniter-js/jobs` (tsup). Outputs ESM + CJS with dts.
104
- - **Config:** `tsconfig.json` extends shared base; `tsup.config.ts` declares multiple entries if needed; keep barrels aligned.
105
- - **External deps:** BullMQ/Redis via adapter; avoid pulling heavy deps into core runtime. Telemetry is optional peer.
834
+ #### 20.4 Reference
106
835
 
107
- ## 9. Workflow Checklist (per change)
836
+ See `@igniter-js/adapter-bullmq` package documentation for full configuration options.
108
837
 
109
- 1. Read this AGENTS + `README.md` + relevant code before editing.
110
- 2. Update types (public surface) first; then implementation; then tests.
111
- 3. Preserve BullMQ feature parity; log any intentional deltas in CHANGELOG.
112
- 4. Keep telemetry names/attributes stable; update descriptors + tests if changed.
113
- 5. Add/adjust TSDoc for every exported symbol and public function.
114
- 6. Run tests and typecheck (`npm test --filter @igniter-js/jobs`, `npx tsc --noEmit`).
115
- 7. Update README + AGENTS + CHANGELOG when behavior or API changes.
838
+ ---
116
839
 
117
- ## 10. Troubleshooting Notes
840
+ ### 21. Implementing Custom Adapters
841
+
842
+ To create a custom adapter, implement the `IgniterJobsAdapter` interface:
843
+
844
+ ```typescript
845
+ import type {
846
+ IgniterJobsAdapter,
847
+ IgniterJobsDispatchParams,
848
+ IgniterJobsWorkerConfig,
849
+ IgniterJobsWorkerHandle,
850
+ IgniterJobSearchResult,
851
+ IgniterJobsQueueInfo,
852
+ } from "@igniter-js/jobs";
853
+
854
+ export class CustomAdapter implements IgniterJobsAdapter {
855
+ // ... implement all required methods
856
+ }
857
+ ```
858
+
859
+ #### 21.1 Required Method Signatures
860
+
861
+ See `src/types/adapter.ts` for the complete interface definition.
862
+
863
+ #### 21.2 Testing Custom Adapters
864
+
865
+ Use the existing test suites as a reference:
866
+ - `src/adapters/memory.adapter.spec.ts`
867
+ - `src/adapters/sqlite.adapter.spec.ts`
868
+
869
+ ---
870
+
871
+ ## V. EXAMPLES & USE CASES
872
+
873
+ ### 22. Example Directory Structure
874
+
875
+ The `examples/` directory contains runnable examples:
876
+
877
+ ```
878
+ packages/jobs/examples/
879
+ ├── README.md # Example documentation
880
+ └── sqlite-example.ts # Complete SQLite adapter demo
881
+ ```
882
+
883
+ ### 23. Running the SQLite Example
884
+
885
+ ```bash
886
+ cd packages/jobs
887
+ npx tsx examples/sqlite-example.ts
888
+ ```
889
+
890
+ This example demonstrates:
891
+ - Creating an SQLite adapter with file persistence
892
+ - Registering multiple job types
893
+ - Priority-based job processing
894
+ - Delayed job scheduling
895
+ - Worker metrics and lifecycle handlers
896
+ - Queue status monitoring
897
+ - Graceful shutdown
898
+
899
+ ---
900
+
901
+ ## VI. MIGRATION GUIDE
902
+
903
+ ### 24. Migrating from Memory to SQLite
904
+
905
+ For development to production-like local testing:
906
+
907
+ ```typescript
908
+ // Before: Memory adapter
909
+ import { IgniterJobsMemoryAdapter } from "@igniter-js/jobs/adapters";
910
+ const adapter = IgniterJobsMemoryAdapter.create();
911
+
912
+ // After: SQLite adapter
913
+ import { IgniterJobsSQLiteAdapter } from "@igniter-js/jobs/adapters";
914
+ const adapter = IgniterJobsSQLiteAdapter.create({
915
+ path: "./dev-jobs.sqlite",
916
+ });
917
+
918
+ // Rest of the code remains identical!
919
+ ```
920
+
921
+ ### 25. Migrating from SQLite to BullMQ
922
+
923
+ For local development to production:
924
+
925
+ ```typescript
926
+ // Before: SQLite adapter
927
+ import { IgniterJobsSQLiteAdapter } from "@igniter-js/jobs/adapters";
928
+ const adapter = IgniterJobsSQLiteAdapter.create({
929
+ path: "./jobs.sqlite",
930
+ });
931
+
932
+ // After: BullMQ adapter
933
+ import { IgniterJobsBullMQAdapter } from "@igniter-js/adapter-bullmq";
934
+ const adapter = IgniterJobsBullMQAdapter.create({
935
+ redis: new Redis(process.env.REDIS_URL),
936
+ });
937
+
938
+ // Rest of the code remains identical!
939
+ ```
940
+
941
+ ---
942
+
943
+ ## VII. MAINTENANCE CHANGELOG
944
+
945
+ ### Version 0.2.0 (2026-01-17)
946
+
947
+ - **Added:** `IgniterJobsSQLiteAdapter` for persistent local job queues
948
+ - **Added:** SQLite adapter comprehensive test suite (71 tests)
949
+ - **Added:** `examples/` directory with runnable examples
950
+ - **Added:** `better-sqlite3` as optional peer dependency
951
+ - **Updated:** Adapter exports to include SQLite adapter
952
+ - **Updated:** AGENTS.md with comprehensive adapter documentation
953
+
954
+ ### Version 0.1.2 (2025-12-23)
955
+
956
+ - Initial stable release
957
+ - Memory adapter for testing
958
+ - BullMQ adapter reference
959
+
960
+ ---
961
+
962
+ ## VIII. FAQ & TROUBLESHOOTING
963
+
964
+ ### 26. Frequently Asked Questions
965
+
966
+ #### Q: Which adapter should I choose?
967
+
968
+ | Environment | Recommended Adapter |
969
+ |-------------|---------------------|
970
+ | Unit/Integration Tests | Memory Adapter |
971
+ | Desktop App (Tauri/Electron) | SQLite Adapter |
972
+ | CLI Tool | SQLite Adapter |
973
+ | MCP Server | SQLite Adapter |
974
+ | Local Development | SQLite Adapter |
975
+ | Production (Single Node) | SQLite or BullMQ |
976
+ | Production (Multi Node) | BullMQ Adapter |
977
+ | Serverless | BullMQ Adapter |
978
+
979
+ #### Q: Can I switch adapters without changing business logic?
980
+
981
+ **Yes!** All adapters implement the same `IgniterJobsAdapter` interface. Your job definitions, handlers, and queue configurations remain identical.
982
+
983
+ #### Q: How do I handle job failures?
984
+
985
+ ```typescript
986
+ adapter.registerJob("email", "send", {
987
+ handler: async ({ input }) => {
988
+ // Your logic here
989
+ },
990
+ attempts: 3, // Retry up to 3 times
991
+ onFailure: async ({ error, isFinalAttempt }) => {
992
+ if (isFinalAttempt) {
993
+ // Send alert, log to monitoring, etc.
994
+ await alertOps(`Job failed permanently: ${error.message}`);
995
+ }
996
+ },
997
+ });
998
+ ```
999
+
1000
+ #### Q: How do I track job progress?
1001
+
1002
+ ```typescript
1003
+ adapter.registerJob("import", "csv", {
1004
+ handler: async ({ input, job }) => {
1005
+ const rows = await parseCSV(input.file);
1006
+
1007
+ for (let i = 0; i < rows.length; i++) {
1008
+ await processRow(rows[i]);
1009
+
1010
+ // Update progress
1011
+ await adapter.updateJobProgress(job.id, Math.floor((i / rows.length) * 100));
1012
+ }
1013
+
1014
+ return { processed: rows.length };
1015
+ },
1016
+ onProgress: async ({ progress, message }) => {
1017
+ console.log(`Progress: ${progress}%`);
1018
+ },
1019
+ });
1020
+ ```
1021
+
1022
+ #### Q: Can SQLite handle high throughput?
1023
+
1024
+ SQLite with WAL mode can handle moderate throughput (100-1000 jobs/minute) on a single machine. For higher throughput or distributed processing, use BullMQ.
1025
+
1026
+ ### 27. Common Issues & Solutions
1027
+
1028
+ #### Issue: "Cannot find module 'better-sqlite3'"
1029
+
1030
+ **Cause:** The `better-sqlite3` peer dependency is not installed.
1031
+
1032
+ **Solution:**
1033
+ ```bash
1034
+ npm install better-sqlite3
1035
+ # If compilation fails on macOS:
1036
+ npm install better-sqlite3 --build-from-source
1037
+ ```
1038
+
1039
+ #### Issue: Jobs stuck in "active" status after restart
1040
+
1041
+ **Cause:** Process crashed while job was processing.
1042
+
1043
+ **Solution:**
1044
+ ```typescript
1045
+ // On startup, reset stale active jobs
1046
+ const staleJobs = await adapter.searchJobs({
1047
+ queue: "my-queue",
1048
+ status: ["active"],
1049
+ });
1050
+
1051
+ for (const job of staleJobs) {
1052
+ await adapter.retryJob(job.id);
1053
+ }
1054
+ ```
1055
+
1056
+ #### Issue: Worker not processing jobs
1057
+
1058
+ **Cause:** Worker queues don't match job queue names.
1059
+
1060
+ **Solution:**
1061
+ ```typescript
1062
+ // Ensure queue names match
1063
+ adapter.registerJob("email", "send", { ... }); // Queue: "email"
1064
+
1065
+ // Worker must listen to "email" queue
1066
+ const worker = await adapter.createWorker({
1067
+ queues: ["email"], // Must include "email"
1068
+ concurrency: 1,
1069
+ });
1070
+ ```
1071
+
1072
+ #### Issue: SQLite database locked
1073
+
1074
+ **Cause:** Multiple processes trying to write simultaneously.
1075
+
1076
+ **Solution:**
1077
+ - Use WAL mode (enabled by default)
1078
+ - Ensure only one process writes at a time
1079
+ - Consider BullMQ for multi-process scenarios
1080
+
1081
+ ---
1082
+
1083
+ ## IX. PERFORMANCE CONSIDERATIONS
1084
+
1085
+ ### 28. SQLite Adapter Performance Tips
1086
+
1087
+ 1. **Use WAL Mode:** Always enable WAL mode for better read/write concurrency.
1088
+
1089
+ 2. **Tune Polling Interval:**
1090
+ - Lower interval (100ms) = more responsive, higher CPU
1091
+ - Higher interval (1000ms) = less responsive, lower CPU
1092
+
1093
+ 3. **Batch Operations:** When dispatching many jobs, consider batching:
1094
+ ```typescript
1095
+ const jobIds = [];
1096
+ for (const item of items) {
1097
+ jobIds.push(await adapter.dispatch({
1098
+ queue: "process",
1099
+ jobName: "item",
1100
+ input: item,
1101
+ }));
1102
+ }
1103
+ ```
1104
+
1105
+ 4. **Clean Old Jobs:** Periodically clean completed/failed jobs:
1106
+ ```typescript
1107
+ await adapter.cleanQueue("email", {
1108
+ status: ["completed", "failed"],
1109
+ olderThan: 7 * 24 * 60 * 60 * 1000, // 7 days
1110
+ });
1111
+ ```
1112
+
1113
+ 5. **Index Usage:** The adapter creates optimized indexes for common queries.
1114
+
1115
+ ### 29. Memory Considerations
1116
+
1117
+ - **Memory Adapter:** All jobs stored in RAM; limit maxJobHistory.
1118
+ - **SQLite Adapter:** Disk-based; memory usage scales with active jobs.
1119
+ - **BullMQ Adapter:** Redis memory; configure maxmemory policy.
1120
+
1121
+ ---
1122
+
1123
+ ## X. SECURITY BEST PRACTICES
1124
+
1125
+ ### 30. Input Validation
1126
+
1127
+ Always validate job inputs using schemas:
1128
+
1129
+ ```typescript
1130
+ import { z } from "zod";
1131
+
1132
+ const emailQueue = IgniterQueue.create("email")
1133
+ .addJob("send", {
1134
+ input: z.object({
1135
+ to: z.string().email(),
1136
+ subject: z.string().max(200),
1137
+ body: z.string(),
1138
+ }),
1139
+ handler: async ({ input }) => {
1140
+ // input is validated and typed
1141
+ },
1142
+ })
1143
+ .build();
1144
+ ```
118
1145
 
119
- - **Telemetry failures:** Should be swallowed; if tests fail, ensure helper mocks errors without throwing.
120
- - **Cron/repeat drift:** Verify repeat options passed untouched to BullMQ; advanced scheduling metadata must survive serialization.
121
- - **Type regressions:** Compare against `packages/core/src/types/jobs.interface.ts`. No `any` leakage in public exports.
122
- - **Adapter parity gaps:** Cross-check with `packages/adapter-bullmq/src/bullmq.adapter.ts` for missing capabilities.
123
- - **Flaky integration tests:** Prefer Redis-mock/stub; isolate time-based tests with fake timers.
1146
+ ### 31. Sensitive Data Handling
124
1147
 
125
- ## 11. References
1148
+ - **Never log raw inputs** containing passwords, tokens, or PII.
1149
+ - **Use job IDs** for reference rather than storing sensitive data in jobs.
1150
+ - **Encrypt at rest** if storing sensitive metadata.
126
1151
 
127
- - `packages/core/src/types/jobs.interface.ts` — canonical job interfaces.
128
- - `packages/adapter-bullmq/src/bullmq.adapter.ts` — legacy feature set to mirror.
129
- - `packages/store`, `packages/telemetry` — builder/test patterns and redaction/sampling ideas.
1152
+ ---
130
1153
 
131
- Stay consistent, type-safe, and keep docs/tests in lockstep with the code.