@igniter-js/jobs 0.1.0 → 0.1.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +1006 -410
- package/CHANGELOG.md +13 -0
- package/README.md +2092 -277
- package/dist/adapter-CXZxomI9.d.mts +529 -0
- package/dist/adapter-CXZxomI9.d.ts +529 -0
- package/dist/adapters/bullmq.adapter.d.mts +56 -111
- package/dist/adapters/bullmq.adapter.d.ts +56 -111
- package/dist/adapters/bullmq.adapter.js +432 -530
- package/dist/adapters/bullmq.adapter.js.map +1 -1
- package/dist/adapters/bullmq.adapter.mjs +432 -530
- package/dist/adapters/bullmq.adapter.mjs.map +1 -1
- package/dist/adapters/index.d.mts +142 -4
- package/dist/adapters/index.d.ts +142 -4
- package/dist/adapters/index.js +1701 -939
- package/dist/adapters/index.js.map +1 -1
- package/dist/adapters/index.mjs +1699 -938
- package/dist/adapters/index.mjs.map +1 -1
- package/dist/adapters/memory.adapter.d.mts +53 -99
- package/dist/adapters/memory.adapter.d.ts +53 -99
- package/dist/adapters/memory.adapter.js +554 -464
- package/dist/adapters/memory.adapter.js.map +1 -1
- package/dist/adapters/memory.adapter.mjs +554 -464
- package/dist/adapters/memory.adapter.mjs.map +1 -1
- package/dist/index.d.mts +524 -887
- package/dist/index.d.ts +524 -887
- package/dist/index.js +2917 -860
- package/dist/index.js.map +1 -1
- package/dist/index.mjs +2914 -860
- package/dist/index.mjs.map +1 -1
- package/dist/shim.d.mts +36 -0
- package/dist/shim.d.ts +36 -0
- package/dist/shim.js +75 -0
- package/dist/shim.js.map +1 -0
- package/dist/shim.mjs +67 -0
- package/dist/shim.mjs.map +1 -0
- package/dist/telemetry/index.d.mts +281 -0
- package/dist/telemetry/index.d.ts +281 -0
- package/dist/telemetry/index.js +97 -0
- package/dist/telemetry/index.js.map +1 -0
- package/dist/telemetry/index.mjs +95 -0
- package/dist/telemetry/index.mjs.map +1 -0
- package/package.json +71 -20
- package/dist/adapter-CcQCatSa.d.mts +0 -1411
- package/dist/adapter-CcQCatSa.d.ts +0 -1411
package/AGENTS.md
CHANGED
|
@@ -1,557 +1,1153 @@
|
|
|
1
|
-
# @igniter-js/jobs
|
|
1
|
+
# AGENTS.md - @igniter-js/jobs
|
|
2
2
|
|
|
3
|
-
> **Last Updated:**
|
|
4
|
-
> **Version:** 0.
|
|
3
|
+
> **Last Updated:** 2026-01-29
|
|
4
|
+
> **Version:** 1.0.0-alpha.0
|
|
5
|
+
> **Goal:** This document serves as the complete operational manual for Code Agents maintaining and consuming the @igniter-js/jobs package.
|
|
5
6
|
|
|
6
|
-
|
|
7
|
+
---
|
|
7
8
|
|
|
8
|
-
|
|
9
|
+
## 1. Package Vision & Context
|
|
9
10
|
|
|
10
|
-
`@igniter-js/jobs`
|
|
11
|
+
`@igniter-js/jobs` is the definitive background processing and task scheduling solution for the Igniter.js ecosystem. It transforms the often-unpredictable world of distributed queues into a type-safe, observable, and highly structured domain that feels like a native extension of the application's business logic.
|
|
11
12
|
|
|
12
|
-
###
|
|
13
|
+
### Why This Package Exists
|
|
13
14
|
|
|
14
|
-
|
|
15
|
-
|---------|-------------|
|
|
16
|
-
| **Queue** | Named container for jobs (e.g., `email`, `notifications`) |
|
|
17
|
-
| **Job** | Unit of work with typed payload and handler |
|
|
18
|
-
| **Cron Job** | Scheduled job with pattern or interval |
|
|
19
|
-
| **Worker** | Process that executes jobs from queues |
|
|
20
|
-
| **Scope** | Multi-tenant context (e.g., organization, workspace) |
|
|
21
|
-
| **Actor** | Entity that triggered the job (e.g., user, system) |
|
|
22
|
-
| **Adapter** | Implementation for queue backend (BullMQ, Memory) |
|
|
15
|
+
In modern distributed systems, background processing is usually the first point of failure and the last point of visibility. Traditional queue libraries often suffer from:
|
|
23
16
|
|
|
24
|
-
|
|
17
|
+
- **Type Erasure:** Job payloads are treated as `any` or raw JSON, leading to "poison pill" jobs that crash workers after being enqueued from a different version of the app.
|
|
18
|
+
- **Context Isolation:** Accessing core resources like database connections, mailers, or configurations inside a worker usually requires brittle global variables or complex dependency injection hacks.
|
|
19
|
+
- **Observability Gaps:** Tracking a job from the moment it's enqueued in an API request to its final completion on a remote worker often requires manual, inconsistent instrumentation.
|
|
20
|
+
- **Multi-Tenant Leakage:** In SaaS environments, it's dangerously easy for a job intended for Organization A to leak context or data into Organization B's processing loop.
|
|
25
21
|
|
|
26
|
-
|
|
27
|
-
// Main entry point
|
|
28
|
-
import { IgniterJobs, IgniterQueue, IgniterJobsError } from '@igniter-js/jobs'
|
|
22
|
+
### The Igniter.js Solution
|
|
29
23
|
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
24
|
+
`@igniter-js/jobs` solves these problems by enforcing a strict **Infrastructure-as-Code** approach to background tasks:
|
|
25
|
+
|
|
26
|
+
1. **End-to-End Type Safety:** By using TypeScript's powerful inference engine, the package ensures that the `input` passed to `.dispatch()` is exactly what the `handler` expects.
|
|
27
|
+
2. **Resource-Aware Context:** The `withContext()` factory ensures that every job execution starts with a fresh, validated set of resources, exactly like a standard API request.
|
|
28
|
+
3. **Multi-Platform Support (Adapters):** Whether using **BullMQ** for massive production scale or an **In-Memory** adapter for unit tests, the business logic remains identical.
|
|
29
|
+
4. **Native Observability:** Telemetry and internal event publishing are "baked in," providing immediate visibility into enqueuing, execution times, failures, and retries without writing a single line of logging code.
|
|
33
30
|
|
|
34
31
|
---
|
|
35
32
|
|
|
36
|
-
##
|
|
33
|
+
## I. MAINTAINER GUIDE (Internal Architecture)
|
|
37
34
|
|
|
38
|
-
###
|
|
35
|
+
### 2. FileSystem Topology (Maintenance)
|
|
39
36
|
|
|
40
|
-
|
|
41
|
-
packages/jobs/
|
|
42
|
-
├── src/
|
|
43
|
-
│ ├── index.ts # Main exports
|
|
44
|
-
│ ├── types/
|
|
45
|
-
│ │ ├── index.ts # Type exports
|
|
46
|
-
│ │ ├── scope.ts # Scope/Actor types
|
|
47
|
-
│ │ ├── job.ts # Job definition types
|
|
48
|
-
│ │ ├── queue.ts # Queue config types
|
|
49
|
-
│ │ ├── events.ts # Event types
|
|
50
|
-
│ │ ├── adapter.ts # Adapter interface
|
|
51
|
-
│ │ ├── config.ts # Builder state types
|
|
52
|
-
│ │ └── worker.ts # Worker types
|
|
53
|
-
│ ├── errors/
|
|
54
|
-
│ │ ├── index.ts # Error exports
|
|
55
|
-
│ │ └── igniter-jobs.error.ts # Error class and codes
|
|
56
|
-
│ ├── builders/
|
|
57
|
-
│ │ ├── index.ts # Builder exports
|
|
58
|
-
│ │ ├── igniter-queue.builder.ts # Queue builder
|
|
59
|
-
│ │ ├── igniter-jobs.builder.ts # Jobs builder
|
|
60
|
-
│ │ └── igniter-worker.builder.ts # Worker builder
|
|
61
|
-
│ ├── core/
|
|
62
|
-
│ │ ├── index.ts # Core exports
|
|
63
|
-
│ │ └── igniter-jobs.ts # Runtime class
|
|
64
|
-
│ └── adapters/
|
|
65
|
-
│ ├── index.ts # Adapter exports
|
|
66
|
-
│ ├── bullmq.adapter.ts # BullMQ implementation
|
|
67
|
-
│ └── memory.adapter.ts # Memory implementation
|
|
68
|
-
├── tests/
|
|
69
|
-
│ ├── igniter-queue.builder.test.ts
|
|
70
|
-
│ ├── igniter-jobs.builder.test.ts
|
|
71
|
-
│ ├── igniter-jobs.runtime.test.ts
|
|
72
|
-
│ └── memory.adapter.test.ts
|
|
73
|
-
├── package.json
|
|
74
|
-
├── tsconfig.json
|
|
75
|
-
├── tsup.config.ts
|
|
76
|
-
├── vitest.config.ts
|
|
77
|
-
├── README.md
|
|
78
|
-
└── AGENTS.md # This file
|
|
79
|
-
```
|
|
37
|
+
Maintainers must respect the following directory structure and responsibilities:
|
|
80
38
|
|
|
81
|
-
|
|
39
|
+
- **`src/builders/`**: The configuration factory.
|
|
40
|
+
- `main.builder.ts`: `IgniterJobsBuilder`. Manages immutable state accumulation and generic type narrowing for context and queues.
|
|
41
|
+
- `queue.builder.ts`: `IgniterQueueBuilder`. Fluent API for defining jobs and crons within a queue.
|
|
42
|
+
- `worker.builder.ts`: `IgniterWorkerBuilder`. Configures worker concurrency and lifecycle handlers.
|
|
43
|
+
- **`src/core/`**: The runtime heart.
|
|
44
|
+
- `manager.ts`: `IgniterJobsManager`. Implements the proxy-based API and the handler-wrapping logic.
|
|
45
|
+
- `queue.ts`: `IgniterQueue` facade for the queue builder.
|
|
46
|
+
- **`src/adapters/`**: The infrastructure boundary.
|
|
47
|
+
- `memory.adapter.ts`: Production-grade in-memory implementation for tests (non-persistent).
|
|
48
|
+
- `sqlite.adapter.ts`: SQLite-based persistent adapter for local/desktop/CLI environments.
|
|
49
|
+
- `bullmq.adapter.ts`: Reference to external `@igniter-js/adapter-bullmq` for production Redis-based queues.
|
|
50
|
+
- **`src/errors/`**: Resiliency definitions.
|
|
51
|
+
- `jobs.error.ts`: `IgniterJobsError`. Extends `IgniterError` with metadata-rich payloads and authoritative error codes.
|
|
52
|
+
- **`src/telemetry/`**: Observability registry.
|
|
53
|
+
- `index.ts`: defines Zod-validated telemetry events for the package.
|
|
54
|
+
- **`src/types/`**: Pure contract definitions.
|
|
55
|
+
- `adapter.ts`: `IgniterJobsAdapter` interface.
|
|
56
|
+
- `runtime.ts`: Accessor interfaces for the proxy API.
|
|
57
|
+
- `job.ts`: authoritative types for job definitions, contexts, and hooks.
|
|
58
|
+
- **`src/utils/`**: Static helper library.
|
|
59
|
+
- `events.utils.ts`: Pub/sub event helpers and channel composition.
|
|
60
|
+
- `id-generator.ts`: Deterministic ID utilities for adapters.
|
|
61
|
+
- `prefix.ts`: Consistency logic for queue names and event channels.
|
|
62
|
+
- `scope.ts`: Scope merging and extraction utilities.
|
|
63
|
+
- `telemetry.ts`: Telemetry helper utilities.
|
|
64
|
+
- `validation.ts`: Runtime schema validation utilities.
|
|
82
65
|
|
|
83
|
-
|
|
84
|
-
1. IgniterQueue.create('name')
|
|
85
|
-
└─► addJob() / addCron()
|
|
86
|
-
└─► build() → IgniterQueueBuiltConfig
|
|
87
|
-
|
|
88
|
-
2. IgniterJobs.create()
|
|
89
|
-
└─► withAdapter()
|
|
90
|
-
└─► addQueue()
|
|
91
|
-
└─► addScope() / addActor()
|
|
92
|
-
└─► withDefaults()
|
|
93
|
-
└─► build() → IgniterJobsRuntime (with Proxy)
|
|
94
|
-
|
|
95
|
-
3. jobs.queueName.jobName.dispatch(data)
|
|
96
|
-
└─► Proxy intercepts access
|
|
97
|
-
└─► Returns typed dispatch function
|
|
98
|
-
```
|
|
66
|
+
### 3. Architecture Deep-Dive
|
|
99
67
|
|
|
100
|
-
|
|
68
|
+
#### 3.1 The Handler-Wrapper Pattern
|
|
101
69
|
|
|
102
|
-
The
|
|
70
|
+
The core reliability of the package comes from the fact that user handlers are never called directly by the adapter. Instead, `core/manager.ts` wraps every handler in a sophisticated pipeline:
|
|
103
71
|
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
72
|
+
1. **Telemetry Injection:** A "Started" event is emitted before the handler runs.
|
|
73
|
+
2. **Context Resolution:** The `contextFactory` provided in the builder is invoked to create the execution environment.
|
|
74
|
+
3. **Metadata Unpacking:** The scope (tenant ID) is extracted from the job metadata and injected into the context.
|
|
75
|
+
4. **Schema Enforcement:** If an `input` schema is defined, the payload is validated _inside the worker_ before being passed to the handler.
|
|
76
|
+
5. **Lifecycle Management:** The wrapper catches errors, determines if it was the final attempt, and emits the corresponding "Success" or "Failure" telemetry.
|
|
77
|
+
|
|
78
|
+
#### 3.2 Proxy-Based Accessors
|
|
79
|
+
|
|
80
|
+
The `IgniterJobsRuntime` uses a dynamic property accessor pattern. When you call `jobs.email.sendWelcome`, you aren't accessing a hardcoded property. Instead, the runtime maps the `email` queue and `sendWelcome` job from its internal configuration. This allows for a fluent, "discoverable" API that is 100% type-safe without needing code generation.
|
|
81
|
+
|
|
82
|
+
### 4. Operational Flow Mapping (Pipelines)
|
|
83
|
+
|
|
84
|
+
#### 4.1 Method: `IgniterJobsBuilder.build()`
|
|
85
|
+
|
|
86
|
+
1. **Argument Validation:** Adapter, Service, Environment, and Context Factory must be present.
|
|
87
|
+
2. **Internal Logic:** Builds `IgniterJobsConfig` from builder state.
|
|
88
|
+
3. **Runtime Creation:** Instantiates `IgniterJobsManager` and calls `.toRuntime()`.
|
|
89
|
+
4. **Adapter Registration:** `IgniterJobsManager.ensureRegistered()` registers jobs and crons on the adapter.
|
|
90
|
+
5. **Result Formatting:** Returns a proxy runtime with queue/job accessors.
|
|
91
|
+
|
|
92
|
+
#### 4.2 Method: `job.dispatch(params)`
|
|
93
|
+
|
|
94
|
+
1. **Argument Validation:** Validates `input` against job schema.
|
|
95
|
+
2. **Telemetry (Enqueued):** Emits `igniter.jobs.job.enqueued` when telemetry is configured.
|
|
96
|
+
3. **Internal Logic:** Calls `IgniterJobsScopeUtils.mergeMetadataWithScope`.
|
|
97
|
+
4. **Adapter Call:** Calls `adapter.dispatch()`.
|
|
98
|
+
5. **Telemetry (Success):** Emits success event with `jobId`.
|
|
99
|
+
|
|
100
|
+
#### 4.3 Method: `worker.start()`
|
|
101
|
+
|
|
102
|
+
1. **Internal Logic:** Consolidates queues, handlers, and concurrency.
|
|
103
|
+
2. **Adapter Call:** Calls `adapter.createWorker()`.
|
|
104
|
+
3. **Result Formatting:** Returns a worker handle for control.
|
|
105
|
+
|
|
106
|
+
#### 4.4 Method: `queue.pause()`
|
|
107
|
+
|
|
108
|
+
1. **Internal Logic:** Resolves queue name.
|
|
109
|
+
2. **Adapter Call:** Calls `adapter.pauseQueue()`.
|
|
110
|
+
|
|
111
|
+
#### 4.5 Method: `job.get(id).retrieve()`
|
|
112
|
+
|
|
113
|
+
1. **Internal Logic:** Resolves queue and job name.
|
|
114
|
+
2. **Adapter Call:** Calls `adapter.getJob()`.
|
|
115
|
+
3. **Result Formatting:** Maps adapter result to `JobSearchResult`.
|
|
116
|
+
|
|
117
|
+
#### 4.6 Method: `job.get(id).retry()`
|
|
118
|
+
|
|
119
|
+
1. **Adapter Call:** Calls `adapter.retryJob()`.
|
|
120
|
+
2. **Result:** Adapter handles retry; telemetry emission depends on adapter implementation.
|
|
121
|
+
|
|
122
|
+
### 5. Dependency & Type Graph
|
|
123
|
+
|
|
124
|
+
The package is designed for maximum portability with minimal dependency creep.
|
|
125
|
+
|
|
126
|
+
- **`@igniter-js/common`**: For base `IgniterError`, `IgniterLogger`, and the `StandardSchemaV1` interface.
|
|
127
|
+
- **`@igniter-js/telemetry`**: Optional peer dependency for operational monitoring.
|
|
128
|
+
|
|
129
|
+
Type Flow:
|
|
130
|
+
`IgniterJobsBuilder` -> `IgniterJobsConfig` -> `IgniterJobsRuntime` -> `QueueAccessor` -> `JobAccessor`
|
|
131
|
+
|
|
132
|
+
### 6. Maintenance Checklist
|
|
133
|
+
|
|
134
|
+
1. **Parity Check:** If you add a new feature to the `IgniterJobsAdapter` interface, ensure it is implemented in `memory.adapter.ts`.
|
|
135
|
+
2. **Telemetry Audit:** Job lifecycle emits telemetry in `core/manager.ts`. Queue/worker telemetry depends on adapter behavior.
|
|
136
|
+
3. **Error Propagation:** Ensure that errors thrown in user hooks (`onSuccess`, `onFailure`) do not crash the worker loop.
|
|
137
|
+
4. **Inference Test:** Run `npm run typecheck` and verify that the `main.builder.spec.ts` can still autocomplete queue names.
|
|
138
|
+
|
|
139
|
+
### 7. Maintainer Troubleshooting
|
|
140
|
+
|
|
141
|
+
#### Issue: Context factory is called too many times
|
|
142
|
+
|
|
143
|
+
- **Check:** Is the adapter caching the context factory result per job execution?
|
|
144
|
+
- **Fix:** Move factory invocation to the wrapper logic in `core/manager.ts`.
|
|
145
|
+
|
|
146
|
+
#### Issue: Telemetry attributes are missing
|
|
147
|
+
|
|
148
|
+
- **Check:** Are the attributes prefixed with `ctx.job.` or `ctx.worker.`?
|
|
149
|
+
- **Fix:** Follow the schema in `telemetry/index.ts`.
|
|
112
150
|
|
|
113
151
|
---
|
|
114
152
|
|
|
115
|
-
##
|
|
153
|
+
## II. CONSUMER GUIDE (Developer Manual)
|
|
154
|
+
|
|
155
|
+
### 8. Distribution Anatomy (Consumption)
|
|
156
|
+
|
|
157
|
+
The package provides organized subpath exports for optimized bundling:
|
|
116
158
|
|
|
117
|
-
|
|
159
|
+
- **`@igniter-js/jobs`**: The main entry point. Use for `IgniterJobs` and `IgniterQueue`.
|
|
160
|
+
- **`@igniter-js/jobs/adapters`**: Built-in adapters (Memory, SQLite, BullMQ wrapper).
|
|
161
|
+
- **`@igniter-js/jobs/telemetry`**: Telemetry registry for registration.
|
|
118
162
|
|
|
119
|
-
|
|
163
|
+
### 9. Quick Start & Common Patterns
|
|
120
164
|
|
|
121
|
-
|
|
165
|
+
#### Pattern: The Singleton Instance
|
|
122
166
|
|
|
123
|
-
|
|
124
|
-
- `create(name, options?)` - Initialize builder
|
|
125
|
-
- `addJob(name, config)` - Add job with schema and handler
|
|
126
|
-
- `addCron(name, config)` - Add cron job
|
|
127
|
-
- `build()` - Return frozen config
|
|
167
|
+
Setup your jobs in a central `src/services/jobs.ts` file:
|
|
128
168
|
|
|
129
|
-
**Type Flow:**
|
|
130
169
|
```typescript
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
170
|
+
import { IgniterJobs } from "@igniter-js/jobs";
|
|
171
|
+
import { IgniterJobsBullMQAdapter } from "@igniter-js/jobs/adapters";
|
|
172
|
+
|
|
173
|
+
export const jobs = IgniterJobs.create()
|
|
174
|
+
.withAdapter(IgniterJobsBullMQAdapter.create({ redis }))
|
|
175
|
+
.withService("api")
|
|
176
|
+
.withEnvironment("production")
|
|
177
|
+
.withContext(async () => ({ db }))
|
|
178
|
+
.addQueue(myQueue)
|
|
179
|
+
.build();
|
|
136
180
|
```
|
|
137
181
|
|
|
138
|
-
|
|
182
|
+
#### Pattern: Scheduled Retries
|
|
183
|
+
|
|
184
|
+
Prevent permanent failures in external API calls:
|
|
185
|
+
|
|
186
|
+
```typescript
|
|
187
|
+
.addJob("sync", {
|
|
188
|
+
attempts: 5,
|
|
189
|
+
delay: 5000,
|
|
190
|
+
handler: async () => { /* ... */ }
|
|
191
|
+
})
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
### 10. Real-World Use Case Library
|
|
195
|
+
|
|
196
|
+
#### Case 1: E-commerce Order Expiry
|
|
139
197
|
|
|
140
|
-
**
|
|
198
|
+
- **Scenario:** Cancel unpaid orders after 1 hour.
|
|
199
|
+
- **Implementation:** `jobs.orders.cancel.schedule({ input: { id }, delay: 3600000 })`.
|
|
141
200
|
|
|
142
|
-
|
|
201
|
+
#### Case 2: Fintech Nightly Reconciliation
|
|
143
202
|
|
|
144
|
-
**
|
|
145
|
-
-
|
|
146
|
-
- `withAdapter(adapter)` - Set queue adapter (required)
|
|
147
|
-
- `withContext(factory)` - Set context factory for handlers
|
|
148
|
-
- `addQueue(queue)` - Add queue configuration
|
|
149
|
-
- `addScope(name, config)` - Define scope type
|
|
150
|
-
- `addActor(name, config)` - Define actor type
|
|
151
|
-
- `withTelemetry(adapter)` - Enable tracing
|
|
152
|
-
- `withDefaults(options)` - Default job options
|
|
153
|
-
- `build()` - Return runtime instance
|
|
203
|
+
- **Scenario:** Verify 1M transactions against bank API.
|
|
204
|
+
- **Implementation:** Cron job at 3 AM. Dispatches batch jobs with 100 transactions each.
|
|
154
205
|
|
|
155
|
-
|
|
206
|
+
#### Case 3: Social Media Media Transcoding
|
|
156
207
|
|
|
157
|
-
**
|
|
208
|
+
- **Scenario:** Generate 5 video qualities after upload.
|
|
209
|
+
- **Implementation:** High-priority queue for 360p, low-priority for 4k.
|
|
158
210
|
|
|
159
|
-
|
|
211
|
+
#### Case 4: Healthcare Appointment Reminders
|
|
160
212
|
|
|
161
|
-
**
|
|
162
|
-
-
|
|
163
|
-
- `job` - Job management methods
|
|
164
|
-
- `queue` - Queue management methods
|
|
165
|
-
- `subscribe` - Event subscription
|
|
166
|
-
- `search` - Search methods
|
|
167
|
-
- `worker` - Worker builder access
|
|
168
|
-
- `shutdown()` - Graceful shutdown
|
|
213
|
+
- **Scenario:** Send SMS 24h before appointment.
|
|
214
|
+
- **Implementation:** `.schedule({ at: reminderDate })`.
|
|
169
215
|
|
|
170
|
-
|
|
216
|
+
#### Case 5: SaaS Multi-Tenant CSV Import
|
|
171
217
|
|
|
172
|
-
**
|
|
173
|
-
-
|
|
174
|
-
- Supports all BullMQ features
|
|
175
|
-
- Lazy loads BullMQ module
|
|
218
|
+
- **Scenario:** Process 100k row CSV.
|
|
219
|
+
- **Implementation:** `job.progress(percent)` every 100 rows for real-time progress bar.
|
|
176
220
|
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
-
|
|
180
|
-
-
|
|
221
|
+
### 11. Domain-Specific Guidance
|
|
222
|
+
|
|
223
|
+
- **Financial Systems:** Always use `attempts` and `backoff`. Implement `onFailure` to alert engineering.
|
|
224
|
+
- **High-Volume Caching:** Use `delay` to batch invalidation events.
|
|
225
|
+
- **AI Processing:** Set concurrency to match your GPU/API limits.
|
|
226
|
+
|
|
227
|
+
### 12. Best Practices & Anti-Patterns
|
|
228
|
+
|
|
229
|
+
| Practice | Why? | Example |
|
|
230
|
+
| :------------------- | :------------------- | :-------------------------------------- |
|
|
231
|
+
| ✅ Small Payloads | Reduces overhead. | `input: { id: '123' }` |
|
|
232
|
+
| ✅ Context Injection | Keeps handlers pure. | `withContext(() => ({ db }))` |
|
|
233
|
+
| ✅ Idempotency | Jobs WILL retry. | Check status before taking action. |
|
|
234
|
+
| ❌ Sync I/O | Kills throughput. | `readFileSync(...)` |
|
|
235
|
+
| ❌ Global DB | Untestable. | `import { db } from './db'` in handler. |
|
|
181
236
|
|
|
182
237
|
---
|
|
183
238
|
|
|
184
|
-
##
|
|
239
|
+
## III. TECHNICAL REFERENCE & RESILIENCE
|
|
185
240
|
|
|
186
|
-
###
|
|
241
|
+
### 13. Exhaustive API Reference
|
|
187
242
|
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
243
|
+
| Class/Method | Parameters | Returns | Description |
|
|
244
|
+
| :--------------------- | :--------------- | :---------------- | :--------------------- |
|
|
245
|
+
| `IgniterJobs.create()` | - | `Builder` | Starts the fluent API. |
|
|
246
|
+
| `dispatch()` | `params` | `Promise<string>` | Enqueues a job. |
|
|
247
|
+
| `schedule()` | `params` | `Promise<string>` | Schedules a job. |
|
|
248
|
+
| `search()` | `target, filter` | `Promise<T[]>` | Management API. |
|
|
249
|
+
| `worker.create()` | - | `WorkerBuilder` | Configures a worker. |
|
|
195
250
|
|
|
196
|
-
|
|
197
|
-
interface IgniterJobHandlerContext<TData, TContext> {
|
|
198
|
-
data: TData
|
|
199
|
-
context: TContext
|
|
200
|
-
job: { id: string; name: string; queue: string; attempt: number }
|
|
201
|
-
log: (level, message) => Promise<void>
|
|
202
|
-
updateProgress: (progress) => Promise<void>
|
|
203
|
-
scope?: IgniterJobScopeEntry
|
|
204
|
-
actor?: IgniterJobActorEntry
|
|
205
|
-
}
|
|
251
|
+
### 14. Telemetry & Observability Registry
|
|
206
252
|
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
scope?: { type: string; id: string }
|
|
215
|
-
actor?: { type: string; id: string }
|
|
216
|
-
}
|
|
217
|
-
```
|
|
253
|
+
| Event Name | Group | Meaning |
|
|
254
|
+
| :--------------- | :----- | :---------------- |
|
|
255
|
+
| `job.enqueued` | job | Task accepted. |
|
|
256
|
+
| `job.started` | job | Task started. |
|
|
257
|
+
| `job.completed` | job | Task finished. |
|
|
258
|
+
| `job.failed` | job | Task threw error. |
|
|
259
|
+
| `worker.started` | worker | Worker is online. |
|
|
218
260
|
|
|
219
|
-
###
|
|
261
|
+
### 15. Troubleshooting & Error Code Library
|
|
220
262
|
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
263
|
+
#### JOBS_ADAPTER_REQUIRED
|
|
264
|
+
|
|
265
|
+
- **Context:** `build()`.
|
|
266
|
+
- **Cause:** No adapter provided.
|
|
267
|
+
- **Solution:** Call `.withAdapter()`.
|
|
268
|
+
|
|
269
|
+
#### JOBS_SERVICE_REQUIRED
|
|
270
|
+
|
|
271
|
+
- **Context:** `build()`.
|
|
272
|
+
- **Cause:** No service name.
|
|
273
|
+
- **Solution:** Call `.withService()`.
|
|
274
|
+
|
|
275
|
+
#### JOBS_CONTEXT_REQUIRED
|
|
276
|
+
|
|
277
|
+
- **Context:** `build()`.
|
|
278
|
+
- **Cause:** No context factory.
|
|
279
|
+
- **Solution:** Call `.withContext()`.
|
|
280
|
+
|
|
281
|
+
#### JOBS_CONFIGURATION_INVALID
|
|
225
282
|
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
283
|
+
- **Context:** Various.
|
|
284
|
+
- **Cause:** Missing environment or invalid scope.
|
|
285
|
+
- **Solution:** Follow builder hints.
|
|
229
286
|
|
|
230
|
-
|
|
231
|
-
.addJob('sendWelcome', { schema, handler })
|
|
232
|
-
// Jobs type: { sendWelcome: { data: { email: string } } }
|
|
287
|
+
#### JOBS_QUEUE_NOT_FOUND
|
|
233
288
|
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
289
|
+
- **Context:** Worker creation.
|
|
290
|
+
- **Cause:** Wrong queue name.
|
|
291
|
+
- **Solution:** Check `addQueue()` vs `worker.addQueue()`.
|
|
292
|
+
|
|
293
|
+
#### JOBS_QUEUE_DUPLICATE
|
|
294
|
+
|
|
295
|
+
- **Context:** Initialization.
|
|
296
|
+
- **Cause:** Same name twice.
|
|
297
|
+
- **Solution:** Use unique names.
|
|
298
|
+
|
|
299
|
+
#### JOBS_INVALID_DEFINITION
|
|
300
|
+
|
|
301
|
+
- **Context:** Job registration.
|
|
302
|
+
- **Cause:** Malformed job object.
|
|
303
|
+
- **Solution:** Ensure `handler` exists.
|
|
304
|
+
|
|
305
|
+
#### JOBS_HANDLER_REQUIRED
|
|
306
|
+
|
|
307
|
+
- **Context:** Job registration.
|
|
308
|
+
- **Cause:** Handler is null.
|
|
309
|
+
- **Solution:** Provide async function.
|
|
310
|
+
|
|
311
|
+
#### JOBS_DUPLICATE_JOB
|
|
312
|
+
|
|
313
|
+
- **Context:** Queue building.
|
|
314
|
+
- **Cause:** Two jobs with same name.
|
|
315
|
+
- **Solution:** Rename one.
|
|
316
|
+
|
|
317
|
+
#### JOBS_NOT_FOUND
|
|
318
|
+
|
|
319
|
+
- **Context:** `get(id)`.
|
|
320
|
+
- **Cause:** Invalid ID.
|
|
321
|
+
- **Solution:** Verify ID or check retention.
|
|
322
|
+
|
|
323
|
+
#### JOBS_NOT_REGISTERED
|
|
324
|
+
|
|
325
|
+
- **Context:** Worker execution.
|
|
326
|
+
- **Cause:** Worker doesn't know job.
|
|
327
|
+
- **Solution:** Check runtime config parity.
|
|
328
|
+
|
|
329
|
+
#### JOBS_EXECUTION_FAILED
|
|
330
|
+
|
|
331
|
+
- **Context:** Worker.
|
|
332
|
+
- **Cause:** Handler threw error.
|
|
333
|
+
- **Solution:** Check business logic.
|
|
334
|
+
|
|
335
|
+
#### JOBS_TIMEOUT
|
|
336
|
+
|
|
337
|
+
- **Context:** Worker.
|
|
338
|
+
- **Cause:** Job took too long.
|
|
339
|
+
- **Solution:** Increase `timeout` in job opts.
|
|
340
|
+
|
|
341
|
+
#### JOBS_CONTEXT_FACTORY_FAILED
|
|
342
|
+
|
|
343
|
+
- **Context:** Worker.
|
|
344
|
+
- **Cause:** Factory threw error.
|
|
345
|
+
- **Solution:** Check DB/resource health.
|
|
346
|
+
|
|
347
|
+
#### JOBS_VALIDATION_FAILED
|
|
348
|
+
|
|
349
|
+
- **Context:** `dispatch`.
|
|
350
|
+
- **Cause:** Schema mismatch.
|
|
351
|
+
- **Solution:** Correct the input.
|
|
352
|
+
|
|
353
|
+
#### JOBS_INVALID_INPUT
|
|
354
|
+
|
|
355
|
+
- **Context:** Runtime.
|
|
356
|
+
- **Cause:** Malformed data.
|
|
357
|
+
- **Solution:** Validate early.
|
|
358
|
+
|
|
359
|
+
#### JOBS_INVALID_CRON
|
|
360
|
+
|
|
361
|
+
- **Context:** Registration.
|
|
362
|
+
- **Cause:** Bad syntax.
|
|
363
|
+
- **Solution:** Check cron expression.
|
|
364
|
+
|
|
365
|
+
#### JOBS_INVALID_SCHEDULE
|
|
366
|
+
|
|
367
|
+
- **Context:** `schedule`.
|
|
368
|
+
- **Cause:** Past date.
|
|
369
|
+
- **Solution:** Use future date.
|
|
370
|
+
|
|
371
|
+
#### JOBS_SCOPE_ALREADY_DEFINED
|
|
372
|
+
|
|
373
|
+
- **Context:** Builder.
|
|
374
|
+
- **Cause:** Two scopes added.
|
|
375
|
+
- **Solution:** Only one allowed.
|
|
376
|
+
|
|
377
|
+
#### JOBS_WORKER_FAILED
|
|
378
|
+
|
|
379
|
+
- **Context:** Worker.
|
|
380
|
+
- **Cause:** Connection failure.
|
|
381
|
+
- **Solution:** Check Redis health.
|
|
382
|
+
|
|
383
|
+
#### JOBS_ADAPTER_ERROR
|
|
384
|
+
|
|
385
|
+
- **Context:** Generic.
|
|
386
|
+
- **Cause:** Backend failure.
|
|
387
|
+
- **Solution:** Check Redis/Adapter logs.
|
|
388
|
+
|
|
389
|
+
#### JOBS_ADAPTER_CONNECTION_FAILED
|
|
390
|
+
|
|
391
|
+
- **Context:** Generic.
|
|
392
|
+
- **Cause:** Redis down.
|
|
393
|
+
- **Solution:** Restore connectivity.
|
|
394
|
+
|
|
395
|
+
#### JOBS_SUBSCRIBE_FAILED
|
|
396
|
+
|
|
397
|
+
- **Context:** `subscribe`.
|
|
398
|
+
- **Cause:** Pub/Sub failure.
|
|
399
|
+
- **Solution:** Check Redis ACLs.
|
|
400
|
+
|
|
401
|
+
---
|
|
402
|
+
|
|
403
|
+
## IV. ADAPTERS REFERENCE
|
|
404
|
+
|
|
405
|
+
The `@igniter-js/jobs` package supports multiple adapters for different use cases and environments. This section provides comprehensive documentation for each adapter.
|
|
406
|
+
|
|
407
|
+
### 16. Adapter Architecture Overview
|
|
408
|
+
|
|
409
|
+
All adapters implement the `IgniterJobsAdapter` interface defined in `src/types/adapter.ts`. This interface ensures consistent behavior across different storage backends.
|
|
410
|
+
|
|
411
|
+
```
|
|
412
|
+
┌─────────────────────────────────────────────────────────────────┐
|
|
413
|
+
│ IgniterJobsAdapter Interface │
|
|
414
|
+
├─────────────────────────────────────────────────────────────────┤
|
|
415
|
+
│ Job Management │
|
|
416
|
+
│ ├── registerJob(queue, name, definition) │
|
|
417
|
+
│ ├── registerCron(queue, name, definition) │
|
|
418
|
+
│ ├── dispatch(params) → jobId │
|
|
419
|
+
│ ├── schedule(params) → jobId │
|
|
420
|
+
│ ├── getJob(id) → job | null │
|
|
421
|
+
│ ├── getJobState(id) → status | null │
|
|
422
|
+
│ ├── getJobLogs(id) → logs[] │
|
|
423
|
+
│ ├── getJobProgress(id) → number │
|
|
424
|
+
│ ├── removeJob(id) │
|
|
425
|
+
│ ├── retryJob(id) │
|
|
426
|
+
│ ├── promoteJob(id) │
|
|
427
|
+
│ └── moveJobToFailed(id, error) │
|
|
428
|
+
├─────────────────────────────────────────────────────────────────┤
|
|
429
|
+
│ Queue Management │
|
|
430
|
+
│ ├── listQueues() → queue[] │
|
|
431
|
+
│ ├── getQueueInfo(name) → info | null │
|
|
432
|
+
│ ├── getQueueJobCounts(name) → counts │
|
|
433
|
+
│ ├── pauseQueue(name) │
|
|
434
|
+
│ ├── resumeQueue(name) │
|
|
435
|
+
│ ├── drainQueue(name) → removed │
|
|
436
|
+
│ ├── cleanQueue(name, options) → cleaned │
|
|
437
|
+
│ ├── obliterateQueue(name, options) │
|
|
438
|
+
│ └── retryAllInQueue(name) → retried │
|
|
439
|
+
├─────────────────────────────────────────────────────────────────┤
|
|
440
|
+
│ Worker Management │
|
|
441
|
+
│ ├── createWorker(config) → WorkerHandle │
|
|
442
|
+
│ ├── getWorkers() → Map<id, worker> │
|
|
443
|
+
│ └── searchWorkers(filter) → workers[] │
|
|
444
|
+
├─────────────────────────────────────────────────────────────────┤
|
|
445
|
+
│ Search & Discovery │
|
|
446
|
+
│ ├── searchJobs(filter) → jobs[] │
|
|
447
|
+
│ └── searchQueues(filter) → queues[] │
|
|
448
|
+
├─────────────────────────────────────────────────────────────────┤
|
|
449
|
+
│ Pub/Sub │
|
|
450
|
+
│ ├── publishEvent(channel, data) │
|
|
451
|
+
│ └── subscribeEvent(channel, handler) → unsubscribe │
|
|
452
|
+
├─────────────────────────────────────────────────────────────────┤
|
|
453
|
+
│ Lifecycle │
|
|
454
|
+
│ └── shutdown() │
|
|
455
|
+
└─────────────────────────────────────────────────────────────────┘
|
|
237
456
|
```
|
|
238
457
|
|
|
458
|
+
### 17. Adapter Comparison Matrix
|
|
459
|
+
|
|
460
|
+
| Feature | Memory | SQLite | BullMQ |
|
|
461
|
+
|---------|--------|--------|--------|
|
|
462
|
+
| **Persistence** | ❌ In-memory only | ✅ File-based | ✅ Redis |
|
|
463
|
+
| **Multi-process** | ❌ Single process | ⚠️ Single process (WAL) | ✅ Distributed |
|
|
464
|
+
| **Installation** | Built-in | `better-sqlite3` | `bullmq` + Redis |
|
|
465
|
+
| **Use Case** | Tests, demos | Desktop, CLI, MCP | Production, scale |
|
|
466
|
+
| **Job Limits** | Memory-bound | Disk-bound | Redis-bound |
|
|
467
|
+
| **Worker Mechanism** | Immediate | Polling-based | Event-driven |
|
|
468
|
+
| **Cron Jobs** | ⚠️ Basic | ⚠️ Basic | ✅ Full |
|
|
469
|
+
| **Distributed Locking** | ❌ | ❌ | ✅ |
|
|
470
|
+
| **Rate Limiting** | ⚠️ Basic | ⚠️ Basic | ✅ Advanced |
|
|
471
|
+
|
|
239
472
|
---
|
|
240
473
|
|
|
241
|
-
|
|
474
|
+
### 18. Memory Adapter (Testing & Development)
|
|
242
475
|
|
|
243
|
-
|
|
476
|
+
The `IgniterJobsMemoryAdapter` provides a full-featured in-memory implementation perfect for unit tests and quick development.
|
|
244
477
|
|
|
245
|
-
|
|
246
|
-
- `JOBS_MISSING_ADAPTER` - No adapter provided
|
|
247
|
-
- `JOBS_INVALID_QUEUE_NAME` - Invalid queue name
|
|
248
|
-
- `JOBS_INVALID_JOB_NAME` - Invalid job name
|
|
249
|
-
- `JOBS_DUPLICATE_QUEUE` - Queue already registered
|
|
250
|
-
- `JOBS_DUPLICATE_JOB` - Job already registered
|
|
478
|
+
#### 18.1 When to Use
|
|
251
479
|
|
|
252
|
-
**
|
|
253
|
-
-
|
|
254
|
-
-
|
|
255
|
-
-
|
|
480
|
+
- **Unit Tests:** Fast, isolated, no external dependencies.
|
|
481
|
+
- **Integration Tests:** When you need predictable job behavior.
|
|
482
|
+
- **Development:** Quick experimentation without Redis.
|
|
483
|
+
- **CI/CD Pipelines:** Zero-config test environments.
|
|
256
484
|
|
|
257
|
-
|
|
258
|
-
- `JOBS_JOB_NOT_FOUND` - Job doesn't exist
|
|
259
|
-
- `JOBS_JOB_VALIDATION_FAILED` - Payload validation failed
|
|
260
|
-
- `JOBS_JOB_HANDLER_FAILED` - Handler threw error
|
|
261
|
-
- `JOBS_JOB_TIMEOUT` - Job exceeded timeout
|
|
485
|
+
#### 18.2 Installation
|
|
262
486
|
|
|
263
|
-
|
|
264
|
-
- `JOBS_DISPATCH_FAILED` - Dispatch operation failed
|
|
265
|
-
- `JOBS_SCHEDULE_FAILED` - Schedule operation failed
|
|
266
|
-
- `JOBS_INVALID_DISPATCH_PAYLOAD` - Invalid payload
|
|
487
|
+
No additional dependencies required. The adapter is built into `@igniter-js/jobs`.
|
|
267
488
|
|
|
268
|
-
|
|
489
|
+
#### 18.3 Basic Usage
|
|
269
490
|
|
|
270
491
|
```typescript
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
492
|
+
import { IgniterJobs } from "@igniter-js/jobs";
|
|
493
|
+
import { IgniterJobsMemoryAdapter } from "@igniter-js/jobs/adapters";
|
|
494
|
+
|
|
495
|
+
const adapter = IgniterJobsMemoryAdapter.create();
|
|
496
|
+
|
|
497
|
+
const jobs = IgniterJobs.create()
|
|
498
|
+
.withAdapter(adapter)
|
|
499
|
+
.withService("test")
|
|
500
|
+
.withContext(async () => ({ db: mockDb }))
|
|
501
|
+
.addQueue(emailQueue)
|
|
502
|
+
.build();
|
|
277
503
|
```
|
|
278
504
|
|
|
279
|
-
|
|
505
|
+
#### 18.4 Configuration Options
|
|
280
506
|
|
|
281
|
-
|
|
507
|
+
```typescript
|
|
508
|
+
interface IgniterJobsMemoryAdapterOptions {
|
|
509
|
+
/** Maximum jobs to retain in history (default: 1000) */
|
|
510
|
+
maxJobHistory?: number;
|
|
511
|
+
}
|
|
512
|
+
```
|
|
282
513
|
|
|
283
|
-
|
|
514
|
+
#### 18.5 Testing Patterns
|
|
284
515
|
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
|
|
516
|
+
```typescript
|
|
517
|
+
// Access internal state for assertions
|
|
518
|
+
const adapter = IgniterJobsMemoryAdapter.create();
|
|
519
|
+
|
|
520
|
+
// Dispatch a job
|
|
521
|
+
const jobId = await adapter.dispatch({
|
|
522
|
+
queue: "email",
|
|
523
|
+
jobName: "send",
|
|
524
|
+
input: { to: "test@example.com" },
|
|
525
|
+
});
|
|
526
|
+
|
|
527
|
+
// Verify job was created
|
|
528
|
+
const job = await adapter.getJob(jobId);
|
|
529
|
+
expect(job?.status).toBe("waiting");
|
|
530
|
+
|
|
531
|
+
// Simulate worker processing
|
|
532
|
+
const worker = await adapter.createWorker({
|
|
533
|
+
queues: ["email"],
|
|
534
|
+
concurrency: 1,
|
|
535
|
+
});
|
|
536
|
+
|
|
537
|
+
// Wait for processing
|
|
538
|
+
await new Promise((r) => setTimeout(r, 100));
|
|
539
|
+
|
|
540
|
+
// Verify completion
|
|
541
|
+
const completed = await adapter.getJob(jobId);
|
|
542
|
+
expect(completed?.status).toBe("completed");
|
|
543
|
+
|
|
544
|
+
// Cleanup
|
|
545
|
+
await worker.close();
|
|
546
|
+
```
|
|
289
547
|
|
|
290
|
-
|
|
291
|
-
- Add method to builder class
|
|
292
|
-
- Update state type if needed
|
|
293
|
-
- Maintain fluent return type
|
|
548
|
+
#### 18.6 Limitations
|
|
294
549
|
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
550
|
+
- **Non-persistent:** All data lost on process restart.
|
|
551
|
+
- **Single-process:** Cannot share state across workers.
|
|
552
|
+
- **Memory-bound:** Large job volumes may cause OOM.
|
|
298
553
|
|
|
299
|
-
|
|
300
|
-
- Add method to `IgniterJobsAdapter`
|
|
301
|
-
- Implement in BullMQ adapter
|
|
302
|
-
- Implement in Memory adapter
|
|
554
|
+
---
|
|
303
555
|
|
|
304
|
-
|
|
305
|
-
- Unit test for builder method
|
|
306
|
-
- Integration test for runtime
|
|
307
|
-
- Adapter test for implementation
|
|
556
|
+
### 19. SQLite Adapter (Desktop, CLI, MCP Servers)
|
|
308
557
|
|
|
309
|
-
|
|
558
|
+
The `IgniterJobsSQLiteAdapter` provides persistent job storage using SQLite, ideal for local environments where Redis is impractical.
|
|
559
|
+
|
|
560
|
+
#### 19.1 When to Use
|
|
561
|
+
|
|
562
|
+
- **Desktop Applications:** Tauri, Electron apps needing background tasks.
|
|
563
|
+
- **CLI Tools:** Long-running commands with progress tracking.
|
|
564
|
+
- **MCP Servers:** Model Context Protocol servers with job queues.
|
|
565
|
+
- **Edge Deployments:** Single-node deployments.
|
|
566
|
+
- **Local Development:** When you want jobs to survive restarts.
|
|
567
|
+
|
|
568
|
+
#### 19.2 Installation
|
|
310
569
|
|
|
311
570
|
```bash
|
|
312
|
-
|
|
313
|
-
|
|
571
|
+
npm install better-sqlite3
|
|
572
|
+
# or
|
|
573
|
+
bun add better-sqlite3
|
|
574
|
+
```
|
|
575
|
+
|
|
576
|
+
The `better-sqlite3` package is an optional peer dependency.
|
|
577
|
+
|
|
578
|
+
#### 19.3 Basic Usage
|
|
579
|
+
|
|
580
|
+
```typescript
|
|
581
|
+
import { IgniterJobs } from "@igniter-js/jobs";
|
|
582
|
+
import { IgniterJobsSQLiteAdapter } from "@igniter-js/jobs/adapters";
|
|
314
583
|
|
|
315
|
-
|
|
316
|
-
|
|
584
|
+
// File-based (persistent)
|
|
585
|
+
const adapter = IgniterJobsSQLiteAdapter.create({
|
|
586
|
+
path: "./jobs.sqlite",
|
|
587
|
+
});
|
|
317
588
|
|
|
318
|
-
|
|
319
|
-
|
|
589
|
+
// In-memory (for tests)
|
|
590
|
+
const memoryAdapter = IgniterJobsSQLiteAdapter.create({
|
|
591
|
+
path: ":memory:",
|
|
592
|
+
});
|
|
593
|
+
|
|
594
|
+
const jobs = IgniterJobs.create()
|
|
595
|
+
.withAdapter(adapter)
|
|
596
|
+
.withService("desktop-app")
|
|
597
|
+
.withContext(async () => ({ settings: loadSettings() }))
|
|
598
|
+
.addQueue(syncQueue)
|
|
599
|
+
.build();
|
|
320
600
|
```
|
|
321
601
|
|
|
322
|
-
|
|
323
|
-
- `tests/igniter-queue.builder.test.ts` - Queue builder
|
|
324
|
-
- `tests/igniter-jobs.builder.test.ts` - Jobs builder
|
|
325
|
-
- `tests/igniter-jobs.runtime.test.ts` - Integration
|
|
326
|
-
- `tests/memory.adapter.test.ts` - Memory adapter
|
|
602
|
+
#### 19.4 Configuration Options
|
|
327
603
|
|
|
328
|
-
|
|
604
|
+
```typescript
|
|
605
|
+
interface IgniterJobsSQLiteAdapterOptions {
|
|
606
|
+
/**
|
|
607
|
+
* Path to the SQLite database file.
|
|
608
|
+
* Use ":memory:" for in-memory database.
|
|
609
|
+
* @example "./data/jobs.sqlite"
|
|
610
|
+
*/
|
|
611
|
+
path: string;
|
|
612
|
+
|
|
613
|
+
/**
|
|
614
|
+
* Polling interval in milliseconds for workers to check for new jobs.
|
|
615
|
+
* Lower values = more responsive, higher CPU usage.
|
|
616
|
+
* @default 500
|
|
617
|
+
*/
|
|
618
|
+
pollingInterval?: number;
|
|
619
|
+
|
|
620
|
+
/**
|
|
621
|
+
* Enable WAL (Write-Ahead Logging) mode for better concurrent performance.
|
|
622
|
+
* Recommended for production use.
|
|
623
|
+
* @default true
|
|
624
|
+
*/
|
|
625
|
+
enableWAL?: boolean;
|
|
626
|
+
}
|
|
627
|
+
```
|
|
329
628
|
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
629
|
+
#### 19.5 Database Schema
|
|
630
|
+
|
|
631
|
+
The adapter automatically creates and manages the following tables:
|
|
632
|
+
|
|
633
|
+
```sql
|
|
634
|
+
-- Main jobs table
|
|
635
|
+
CREATE TABLE IF NOT EXISTS jobs (
|
|
636
|
+
id TEXT PRIMARY KEY,
|
|
637
|
+
queue TEXT NOT NULL,
|
|
638
|
+
name TEXT NOT NULL,
|
|
639
|
+
input TEXT, -- JSON serialized
|
|
640
|
+
result TEXT, -- JSON serialized
|
|
641
|
+
error TEXT,
|
|
642
|
+
status TEXT NOT NULL DEFAULT 'waiting',
|
|
643
|
+
progress INTEGER DEFAULT 0,
|
|
644
|
+
priority INTEGER DEFAULT 0,
|
|
645
|
+
attempts_made INTEGER DEFAULT 0,
|
|
646
|
+
max_attempts INTEGER DEFAULT 1,
|
|
647
|
+
delay_until INTEGER, -- Unix timestamp
|
|
648
|
+
metadata TEXT, -- JSON serialized
|
|
649
|
+
scope_type TEXT,
|
|
650
|
+
scope_id TEXT,
|
|
651
|
+
created_at INTEGER NOT NULL,
|
|
652
|
+
started_at INTEGER,
|
|
653
|
+
completed_at INTEGER
|
|
654
|
+
);
|
|
655
|
+
|
|
656
|
+
-- Indexes for efficient queries
|
|
657
|
+
CREATE INDEX IF NOT EXISTS idx_jobs_queue_status ON jobs(queue, status);
|
|
658
|
+
CREATE INDEX IF NOT EXISTS idx_jobs_status ON jobs(status);
|
|
659
|
+
CREATE INDEX IF NOT EXISTS idx_jobs_priority ON jobs(priority DESC);
|
|
660
|
+
CREATE INDEX IF NOT EXISTS idx_jobs_delay ON jobs(delay_until);
|
|
661
|
+
|
|
662
|
+
-- Job logs table
|
|
663
|
+
CREATE TABLE IF NOT EXISTS job_logs (
|
|
664
|
+
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
665
|
+
job_id TEXT NOT NULL,
|
|
666
|
+
level TEXT NOT NULL,
|
|
667
|
+
message TEXT NOT NULL,
|
|
668
|
+
timestamp INTEGER NOT NULL,
|
|
669
|
+
FOREIGN KEY (job_id) REFERENCES jobs(id) ON DELETE CASCADE
|
|
670
|
+
);
|
|
671
|
+
|
|
672
|
+
-- Paused queues tracking
|
|
673
|
+
CREATE TABLE IF NOT EXISTS paused_queues (
|
|
674
|
+
name TEXT PRIMARY KEY,
|
|
675
|
+
paused_at INTEGER NOT NULL
|
|
676
|
+
);
|
|
677
|
+
```
|
|
335
678
|
|
|
336
|
-
|
|
679
|
+
#### 19.6 Worker Polling Mechanism
|
|
337
680
|
|
|
338
|
-
|
|
681
|
+
Unlike event-driven adapters (BullMQ), the SQLite adapter uses a polling-based approach:
|
|
339
682
|
|
|
340
|
-
|
|
683
|
+
```
|
|
684
|
+
┌─────────────────────────────────────────────────────────────────┐
|
|
685
|
+
│ Worker Polling Loop │
|
|
686
|
+
├─────────────────────────────────────────────────────────────────┤
|
|
687
|
+
│ │
|
|
688
|
+
│ ┌─────────┐ ┌─────────────┐ ┌─────────────┐ │
|
|
689
|
+
│ │ Wait │────▶│ Check Jobs │────▶│ Process Job │ │
|
|
690
|
+
│ │ Polling │ │ Available? │ │ (if any) │ │
|
|
691
|
+
│ │ Interval│ └─────────────┘ └─────────────┘ │
|
|
692
|
+
│ └─────────┘ │ │ │
|
|
693
|
+
│ ▲ │ No jobs │ Done │
|
|
694
|
+
│ │ │ ▼ │
|
|
695
|
+
│ └────────────────┴───────────────────┘ │
|
|
696
|
+
│ │
|
|
697
|
+
│ Query Priority: │
|
|
698
|
+
│ 1. Delayed jobs ready to run (delay_until <= NOW) │
|
|
699
|
+
│ 2. Waiting jobs, ordered by priority DESC │
|
|
700
|
+
│ 3. Retry jobs (failed but attempts < max_attempts) │
|
|
701
|
+
│ │
|
|
702
|
+
└─────────────────────────────────────────────────────────────────┘
|
|
703
|
+
```
|
|
341
704
|
|
|
342
|
-
|
|
705
|
+
#### 19.7 Complete Example
|
|
343
706
|
|
|
344
707
|
```typescript
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
|
|
359
|
-
|
|
360
|
-
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
|
|
364
|
-
|
|
365
|
-
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
|
|
370
|
-
|
|
371
|
-
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
|
|
377
|
-
|
|
378
|
-
|
|
379
|
-
//
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
|
|
386
|
-
|
|
387
|
-
|
|
708
|
+
import { IgniterJobsSQLiteAdapter } from "@igniter-js/jobs/adapters";
|
|
709
|
+
|
|
710
|
+
async function main() {
|
|
711
|
+
// 1. Create adapter
|
|
712
|
+
const adapter = IgniterJobsSQLiteAdapter.create({
|
|
713
|
+
path: "./my-app-jobs.sqlite",
|
|
714
|
+
pollingInterval: 500,
|
|
715
|
+
enableWAL: true,
|
|
716
|
+
});
|
|
717
|
+
|
|
718
|
+
// 2. Register jobs
|
|
719
|
+
adapter.registerJob("sync", "upload", {
|
|
720
|
+
handler: async ({ input }) => {
|
|
721
|
+
const { fileId } = input as { fileId: string };
|
|
722
|
+
// Simulate upload
|
|
723
|
+
await uploadFile(fileId);
|
|
724
|
+
return { uploaded: true };
|
|
725
|
+
},
|
|
726
|
+
onProgress: async ({ progress }) => {
|
|
727
|
+
console.log(`Upload progress: ${progress}%`);
|
|
728
|
+
},
|
|
729
|
+
});
|
|
730
|
+
|
|
731
|
+
// 3. Create worker
|
|
732
|
+
const worker = await adapter.createWorker({
|
|
733
|
+
queues: ["sync"],
|
|
734
|
+
concurrency: 2,
|
|
735
|
+
handlers: {
|
|
736
|
+
onActive: ({ job }) => console.log(`Processing: ${job.id}`),
|
|
737
|
+
onSuccess: ({ job }) => console.log(`Completed: ${job.id}`),
|
|
738
|
+
onFailure: ({ job, error }) => console.error(`Failed: ${job.id}`, error),
|
|
739
|
+
},
|
|
740
|
+
});
|
|
741
|
+
|
|
742
|
+
// 4. Dispatch jobs
|
|
743
|
+
const jobId = await adapter.dispatch({
|
|
744
|
+
queue: "sync",
|
|
745
|
+
jobName: "upload",
|
|
746
|
+
input: { fileId: "file_123" },
|
|
747
|
+
priority: 10,
|
|
748
|
+
});
|
|
749
|
+
|
|
750
|
+
console.log(`Dispatched job: ${jobId}`);
|
|
751
|
+
|
|
752
|
+
// 5. Monitor status
|
|
753
|
+
const counts = await adapter.getQueueJobCounts("sync");
|
|
754
|
+
console.log(`Queue status:`, counts);
|
|
755
|
+
|
|
756
|
+
// 6. Graceful shutdown
|
|
757
|
+
process.on("SIGINT", async () => {
|
|
758
|
+
await worker.close();
|
|
759
|
+
await adapter.shutdown();
|
|
760
|
+
process.exit(0);
|
|
761
|
+
});
|
|
388
762
|
}
|
|
763
|
+
|
|
764
|
+
main().catch(console.error);
|
|
389
765
|
```
|
|
390
766
|
|
|
767
|
+
#### 19.8 Persistence & Recovery
|
|
768
|
+
|
|
769
|
+
Jobs are automatically persisted and survive process restarts:
|
|
770
|
+
|
|
771
|
+
```typescript
|
|
772
|
+
// Session 1: Dispatch job and exit
|
|
773
|
+
const adapter = IgniterJobsSQLiteAdapter.create({ path: "./jobs.sqlite" });
|
|
774
|
+
await adapter.dispatch({
|
|
775
|
+
queue: "emails",
|
|
776
|
+
jobName: "send",
|
|
777
|
+
input: { to: "user@example.com" },
|
|
778
|
+
});
|
|
779
|
+
await adapter.shutdown();
|
|
780
|
+
// Process exits
|
|
781
|
+
|
|
782
|
+
// Session 2: Jobs are still there
|
|
783
|
+
const adapter2 = IgniterJobsSQLiteAdapter.create({ path: "./jobs.sqlite" });
|
|
784
|
+
const jobs = await adapter2.searchJobs({ queue: "emails", status: ["waiting"] });
|
|
785
|
+
console.log(jobs); // Shows the pending job from session 1
|
|
786
|
+
```
|
|
787
|
+
|
|
788
|
+
#### 19.9 Limitations
|
|
789
|
+
|
|
790
|
+
- **Single Process:** WAL mode helps, but true concurrent writes from multiple processes may cause issues.
|
|
791
|
+
- **Polling Overhead:** Unlike event-driven systems, there's CPU overhead from polling.
|
|
792
|
+
- **No Distributed Locking:** Cannot guarantee exactly-once processing across processes.
|
|
793
|
+
- **Limited Cron:** Basic cron support without advanced scheduling features.
|
|
794
|
+
|
|
391
795
|
---
|
|
392
796
|
|
|
393
|
-
|
|
797
|
+
### 20. BullMQ Adapter (Production Scale)
|
|
798
|
+
|
|
799
|
+
The `IgniterJobsBullMQAdapter` (external package) provides Redis-based job queuing for production environments.
|
|
394
800
|
|
|
395
|
-
|
|
801
|
+
#### 20.1 When to Use
|
|
802
|
+
|
|
803
|
+
- **Production Deployments:** High availability and reliability.
|
|
804
|
+
- **Distributed Systems:** Multiple workers across nodes.
|
|
805
|
+
- **High Throughput:** Thousands of jobs per second.
|
|
806
|
+
- **Advanced Features:** Rate limiting, priorities, delayed jobs, cron.
|
|
807
|
+
|
|
808
|
+
#### 20.2 Installation
|
|
809
|
+
|
|
810
|
+
```bash
|
|
811
|
+
npm install @igniter-js/adapter-bullmq bullmq ioredis
|
|
812
|
+
```
|
|
813
|
+
|
|
814
|
+
#### 20.3 Basic Usage
|
|
396
815
|
|
|
397
816
|
```typescript
|
|
817
|
+
import { IgniterJobs } from "@igniter-js/jobs";
|
|
818
|
+
import { IgniterJobsBullMQAdapter } from "@igniter-js/adapter-bullmq";
|
|
819
|
+
import Redis from "ioredis";
|
|
820
|
+
|
|
821
|
+
const redis = new Redis(process.env.REDIS_URL);
|
|
822
|
+
|
|
823
|
+
const adapter = IgniterJobsBullMQAdapter.create({ redis });
|
|
824
|
+
|
|
398
825
|
const jobs = IgniterJobs.create()
|
|
399
826
|
.withAdapter(adapter)
|
|
827
|
+
.withService("api-server")
|
|
828
|
+
.withContext(async () => ({ db, cache }))
|
|
400
829
|
.addQueue(emailQueue)
|
|
401
|
-
.
|
|
402
|
-
|
|
403
|
-
})
|
|
404
|
-
.addActor('user', {
|
|
405
|
-
resolver: (id) => db.user.findUnique({ where: { id } }),
|
|
406
|
-
})
|
|
407
|
-
.build()
|
|
408
|
-
|
|
409
|
-
// Dispatch with scope/actor
|
|
410
|
-
await jobs.email.sendWelcome.dispatch(
|
|
411
|
-
{ email: 'test@example.com' },
|
|
412
|
-
{
|
|
413
|
-
scope: { type: 'organization', id: 'org-123' },
|
|
414
|
-
actor: { type: 'user', id: 'user-456' },
|
|
415
|
-
}
|
|
416
|
-
)
|
|
830
|
+
.addQueue(analyticsQueue)
|
|
831
|
+
.build();
|
|
417
832
|
```
|
|
418
833
|
|
|
419
|
-
|
|
834
|
+
#### 20.4 Reference
|
|
835
|
+
|
|
836
|
+
See `@igniter-js/adapter-bullmq` package documentation for full configuration options.
|
|
837
|
+
|
|
838
|
+
---
|
|
839
|
+
|
|
840
|
+
### 21. Implementing Custom Adapters
|
|
841
|
+
|
|
842
|
+
To create a custom adapter, implement the `IgniterJobsAdapter` interface:
|
|
420
843
|
|
|
421
844
|
```typescript
|
|
422
|
-
|
|
423
|
-
|
|
424
|
-
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
|
|
845
|
+
import type {
|
|
846
|
+
IgniterJobsAdapter,
|
|
847
|
+
IgniterJobsDispatchParams,
|
|
848
|
+
IgniterJobsWorkerConfig,
|
|
849
|
+
IgniterJobsWorkerHandle,
|
|
850
|
+
IgniterJobSearchResult,
|
|
851
|
+
IgniterJobsQueueInfo,
|
|
852
|
+
} from "@igniter-js/jobs";
|
|
853
|
+
|
|
854
|
+
export class CustomAdapter implements IgniterJobsAdapter {
|
|
855
|
+
// ... implement all required methods
|
|
856
|
+
}
|
|
857
|
+
```
|
|
428
858
|
|
|
429
|
-
|
|
859
|
+
#### 21.1 Required Method Signatures
|
|
430
860
|
|
|
431
|
-
|
|
432
|
-
await log('info', 'Halfway done')
|
|
861
|
+
See `src/types/adapter.ts` for the complete interface definition.
|
|
433
862
|
|
|
434
|
-
|
|
863
|
+
#### 21.2 Testing Custom Adapters
|
|
435
864
|
|
|
436
|
-
|
|
437
|
-
|
|
438
|
-
|
|
439
|
-
|
|
440
|
-
|
|
865
|
+
Use the existing test suites as a reference:
|
|
866
|
+
- `src/adapters/memory.adapter.spec.ts`
|
|
867
|
+
- `src/adapters/sqlite.adapter.spec.ts`
|
|
868
|
+
|
|
869
|
+
---
|
|
870
|
+
|
|
871
|
+
## V. EXAMPLES & USE CASES
|
|
872
|
+
|
|
873
|
+
### 22. Example Directory Structure
|
|
874
|
+
|
|
875
|
+
The `examples/` directory contains runnable examples:
|
|
876
|
+
|
|
877
|
+
```
|
|
878
|
+
packages/jobs/examples/
|
|
879
|
+
├── README.md # Example documentation
|
|
880
|
+
└── sqlite-example.ts # Complete SQLite adapter demo
|
|
881
|
+
```
|
|
882
|
+
|
|
883
|
+
### 23. Running the SQLite Example
|
|
884
|
+
|
|
885
|
+
```bash
|
|
886
|
+
cd packages/jobs
|
|
887
|
+
npx tsx examples/sqlite-example.ts
|
|
441
888
|
```
|
|
442
889
|
|
|
443
|
-
|
|
890
|
+
This example demonstrates:
|
|
891
|
+
- Creating an SQLite adapter with file persistence
|
|
892
|
+
- Registering multiple job types
|
|
893
|
+
- Priority-based job processing
|
|
894
|
+
- Delayed job scheduling
|
|
895
|
+
- Worker metrics and lifecycle handlers
|
|
896
|
+
- Queue status monitoring
|
|
897
|
+
- Graceful shutdown
|
|
898
|
+
|
|
899
|
+
---
|
|
900
|
+
|
|
901
|
+
## VI. MIGRATION GUIDE
|
|
902
|
+
|
|
903
|
+
### 24. Migrating from Memory to SQLite
|
|
904
|
+
|
|
905
|
+
For development to production-like local testing:
|
|
444
906
|
|
|
445
907
|
```typescript
|
|
446
|
-
//
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
at: new Date('2025-12-25T00:00:00Z'),
|
|
450
|
-
})
|
|
908
|
+
// Before: Memory adapter
|
|
909
|
+
import { IgniterJobsMemoryAdapter } from "@igniter-js/jobs/adapters";
|
|
910
|
+
const adapter = IgniterJobsMemoryAdapter.create();
|
|
451
911
|
|
|
452
|
-
//
|
|
453
|
-
|
|
454
|
-
|
|
455
|
-
|
|
456
|
-
})
|
|
912
|
+
// After: SQLite adapter
|
|
913
|
+
import { IgniterJobsSQLiteAdapter } from "@igniter-js/jobs/adapters";
|
|
914
|
+
const adapter = IgniterJobsSQLiteAdapter.create({
|
|
915
|
+
path: "./dev-jobs.sqlite",
|
|
916
|
+
});
|
|
917
|
+
|
|
918
|
+
// Rest of the code remains identical!
|
|
457
919
|
```
|
|
458
920
|
|
|
459
|
-
###
|
|
921
|
+
### 25. Migrating from SQLite to BullMQ
|
|
922
|
+
|
|
923
|
+
For local development to production:
|
|
460
924
|
|
|
461
925
|
```typescript
|
|
462
|
-
|
|
463
|
-
|
|
464
|
-
|
|
465
|
-
.
|
|
466
|
-
|
|
467
|
-
|
|
468
|
-
|
|
469
|
-
|
|
470
|
-
|
|
471
|
-
.
|
|
926
|
+
// Before: SQLite adapter
|
|
927
|
+
import { IgniterJobsSQLiteAdapter } from "@igniter-js/jobs/adapters";
|
|
928
|
+
const adapter = IgniterJobsSQLiteAdapter.create({
|
|
929
|
+
path: "./jobs.sqlite",
|
|
930
|
+
});
|
|
931
|
+
|
|
932
|
+
// After: BullMQ adapter
|
|
933
|
+
import { IgniterJobsBullMQAdapter } from "@igniter-js/adapter-bullmq";
|
|
934
|
+
const adapter = IgniterJobsBullMQAdapter.create({
|
|
935
|
+
redis: new Redis(process.env.REDIS_URL),
|
|
936
|
+
});
|
|
937
|
+
|
|
938
|
+
// Rest of the code remains identical!
|
|
472
939
|
```
|
|
473
940
|
|
|
474
941
|
---
|
|
475
942
|
|
|
476
|
-
##
|
|
943
|
+
## VII. MAINTENANCE CHANGELOG
|
|
477
944
|
|
|
478
|
-
### 0.
|
|
945
|
+
### Version 0.2.0 (2026-01-17)
|
|
479
946
|
|
|
480
|
-
-
|
|
481
|
-
-
|
|
482
|
-
-
|
|
483
|
-
-
|
|
484
|
-
-
|
|
485
|
-
-
|
|
486
|
-
|
|
487
|
-
|
|
488
|
-
|
|
489
|
-
-
|
|
947
|
+
- **Added:** `IgniterJobsSQLiteAdapter` for persistent local job queues
|
|
948
|
+
- **Added:** SQLite adapter comprehensive test suite (71 tests)
|
|
949
|
+
- **Added:** `examples/` directory with runnable examples
|
|
950
|
+
- **Added:** `better-sqlite3` as optional peer dependency
|
|
951
|
+
- **Updated:** Adapter exports to include SQLite adapter
|
|
952
|
+
- **Updated:** AGENTS.md with comprehensive adapter documentation
|
|
953
|
+
|
|
954
|
+
### Version 0.1.2 (2025-12-23)
|
|
955
|
+
|
|
956
|
+
- Initial stable release
|
|
957
|
+
- Memory adapter for testing
|
|
958
|
+
- BullMQ adapter reference
|
|
490
959
|
|
|
491
960
|
---
|
|
492
961
|
|
|
493
|
-
##
|
|
962
|
+
## VIII. FAQ & TROUBLESHOOTING
|
|
963
|
+
|
|
964
|
+
### 26. Frequently Asked Questions
|
|
494
965
|
|
|
495
|
-
|
|
966
|
+
#### Q: Which adapter should I choose?
|
|
496
967
|
|
|
497
|
-
|
|
498
|
-
|
|
499
|
-
|
|
968
|
+
| Environment | Recommended Adapter |
|
|
969
|
+
|-------------|---------------------|
|
|
970
|
+
| Unit/Integration Tests | Memory Adapter |
|
|
971
|
+
| Desktop App (Tauri/Electron) | SQLite Adapter |
|
|
972
|
+
| CLI Tool | SQLite Adapter |
|
|
973
|
+
| MCP Server | SQLite Adapter |
|
|
974
|
+
| Local Development | SQLite Adapter |
|
|
975
|
+
| Production (Single Node) | SQLite or BullMQ |
|
|
976
|
+
| Production (Multi Node) | BullMQ Adapter |
|
|
977
|
+
| Serverless | BullMQ Adapter |
|
|
500
978
|
|
|
501
|
-
|
|
502
|
-
- Queue must be registered with `addQueue()`
|
|
503
|
-
- Check queue name spelling
|
|
979
|
+
#### Q: Can I switch adapters without changing business logic?
|
|
504
980
|
|
|
505
|
-
**
|
|
506
|
-
- Payload doesn't match Zod schema
|
|
507
|
-
- Check `details.issues` for validation errors
|
|
981
|
+
**Yes!** All adapters implement the same `IgniterJobsAdapter` interface. Your job definitions, handlers, and queue configurations remain identical.
|
|
508
982
|
|
|
509
|
-
|
|
510
|
-
- Ensure queue is added before build
|
|
511
|
-
- Check job name exists in queue
|
|
512
|
-
- Verify payload matches schema
|
|
983
|
+
#### Q: How do I handle job failures?
|
|
513
984
|
|
|
514
|
-
|
|
985
|
+
```typescript
|
|
986
|
+
adapter.registerJob("email", "send", {
|
|
987
|
+
handler: async ({ input }) => {
|
|
988
|
+
// Your logic here
|
|
989
|
+
},
|
|
990
|
+
attempts: 3, // Retry up to 3 times
|
|
991
|
+
onFailure: async ({ error, isFinalAttempt }) => {
|
|
992
|
+
if (isFinalAttempt) {
|
|
993
|
+
// Send alert, log to monitoring, etc.
|
|
994
|
+
await alertOps(`Job failed permanently: ${error.message}`);
|
|
995
|
+
}
|
|
996
|
+
},
|
|
997
|
+
});
|
|
998
|
+
```
|
|
999
|
+
|
|
1000
|
+
#### Q: How do I track job progress?
|
|
515
1001
|
|
|
516
1002
|
```typescript
|
|
517
|
-
|
|
518
|
-
|
|
1003
|
+
adapter.registerJob("import", "csv", {
|
|
1004
|
+
handler: async ({ input, job }) => {
|
|
1005
|
+
const rows = await parseCSV(input.file);
|
|
1006
|
+
|
|
1007
|
+
for (let i = 0; i < rows.length; i++) {
|
|
1008
|
+
await processRow(rows[i]);
|
|
1009
|
+
|
|
1010
|
+
// Update progress
|
|
1011
|
+
await adapter.updateJobProgress(job.id, Math.floor((i / rows.length) * 100));
|
|
1012
|
+
}
|
|
1013
|
+
|
|
1014
|
+
return { processed: rows.length };
|
|
1015
|
+
},
|
|
1016
|
+
onProgress: async ({ progress, message }) => {
|
|
1017
|
+
console.log(`Progress: ${progress}%`);
|
|
1018
|
+
},
|
|
1019
|
+
});
|
|
1020
|
+
```
|
|
519
1021
|
|
|
520
|
-
|
|
521
|
-
const state = await jobs.job.state('queue', 'job-id')
|
|
522
|
-
console.log('Job state:', state)
|
|
1022
|
+
#### Q: Can SQLite handle high throughput?
|
|
523
1023
|
|
|
524
|
-
|
|
525
|
-
const logs = await jobs.job.logs('queue', 'job-id')
|
|
526
|
-
console.log('Job logs:', logs)
|
|
1024
|
+
SQLite with WAL mode can handle moderate throughput (100-1000 jobs/minute) on a single machine. For higher throughput or distributed processing, use BullMQ.
|
|
527
1025
|
|
|
528
|
-
|
|
529
|
-
|
|
530
|
-
|
|
531
|
-
|
|
1026
|
+
### 27. Common Issues & Solutions
|
|
1027
|
+
|
|
1028
|
+
#### Issue: "Cannot find module 'better-sqlite3'"
|
|
1029
|
+
|
|
1030
|
+
**Cause:** The `better-sqlite3` peer dependency is not installed.
|
|
1031
|
+
|
|
1032
|
+
**Solution:**
|
|
1033
|
+
```bash
|
|
1034
|
+
npm install better-sqlite3
|
|
1035
|
+
# If compilation fails on macOS:
|
|
1036
|
+
npm install better-sqlite3 --build-from-source
|
|
1037
|
+
```
|
|
1038
|
+
|
|
1039
|
+
#### Issue: Jobs stuck in "active" status after restart
|
|
1040
|
+
|
|
1041
|
+
**Cause:** Process crashed while job was processing.
|
|
1042
|
+
|
|
1043
|
+
**Solution:**
|
|
1044
|
+
```typescript
|
|
1045
|
+
// On startup, reset stale active jobs
|
|
1046
|
+
const staleJobs = await adapter.searchJobs({
|
|
1047
|
+
queue: "my-queue",
|
|
1048
|
+
status: ["active"],
|
|
1049
|
+
});
|
|
1050
|
+
|
|
1051
|
+
for (const job of staleJobs) {
|
|
1052
|
+
await adapter.retryJob(job.id);
|
|
1053
|
+
}
|
|
532
1054
|
```
|
|
533
1055
|
|
|
1056
|
+
#### Issue: Worker not processing jobs
|
|
1057
|
+
|
|
1058
|
+
**Cause:** Worker queues don't match job queue names.
|
|
1059
|
+
|
|
1060
|
+
**Solution:**
|
|
1061
|
+
```typescript
|
|
1062
|
+
// Ensure queue names match
|
|
1063
|
+
adapter.registerJob("email", "send", { ... }); // Queue: "email"
|
|
1064
|
+
|
|
1065
|
+
// Worker must listen to "email" queue
|
|
1066
|
+
const worker = await adapter.createWorker({
|
|
1067
|
+
queues: ["email"], // Must include "email"
|
|
1068
|
+
concurrency: 1,
|
|
1069
|
+
});
|
|
1070
|
+
```
|
|
1071
|
+
|
|
1072
|
+
#### Issue: SQLite database locked
|
|
1073
|
+
|
|
1074
|
+
**Cause:** Multiple processes trying to write simultaneously.
|
|
1075
|
+
|
|
1076
|
+
**Solution:**
|
|
1077
|
+
- Use WAL mode (enabled by default)
|
|
1078
|
+
- Ensure only one process writes at a time
|
|
1079
|
+
- Consider BullMQ for multi-process scenarios
|
|
1080
|
+
|
|
534
1081
|
---
|
|
535
1082
|
|
|
536
|
-
##
|
|
1083
|
+
## IX. PERFORMANCE CONSIDERATIONS
|
|
537
1084
|
|
|
538
|
-
###
|
|
1085
|
+
### 28. SQLite Adapter Performance Tips
|
|
539
1086
|
|
|
540
|
-
|
|
541
|
-
|
|
542
|
-
|
|
543
|
-
|
|
544
|
-
|
|
545
|
-
|
|
1087
|
+
1. **Use WAL Mode:** Always enable WAL mode for better read/write concurrency.
|
|
1088
|
+
|
|
1089
|
+
2. **Tune Polling Interval:**
|
|
1090
|
+
- Lower interval (100ms) = more responsive, higher CPU
|
|
1091
|
+
- Higher interval (1000ms) = less responsive, lower CPU
|
|
1092
|
+
|
|
1093
|
+
3. **Batch Operations:** When dispatching many jobs, consider batching:
|
|
1094
|
+
```typescript
|
|
1095
|
+
const jobIds = [];
|
|
1096
|
+
for (const item of items) {
|
|
1097
|
+
jobIds.push(await adapter.dispatch({
|
|
1098
|
+
queue: "process",
|
|
1099
|
+
jobName: "item",
|
|
1100
|
+
input: item,
|
|
1101
|
+
}));
|
|
1102
|
+
}
|
|
1103
|
+
```
|
|
1104
|
+
|
|
1105
|
+
4. **Clean Old Jobs:** Periodically clean completed/failed jobs:
|
|
1106
|
+
```typescript
|
|
1107
|
+
await adapter.cleanQueue("email", {
|
|
1108
|
+
status: ["completed", "failed"],
|
|
1109
|
+
olderThan: 7 * 24 * 60 * 60 * 1000, // 7 days
|
|
1110
|
+
});
|
|
1111
|
+
```
|
|
1112
|
+
|
|
1113
|
+
5. **Index Usage:** The adapter creates optimized indexes for common queries.
|
|
1114
|
+
|
|
1115
|
+
### 29. Memory Considerations
|
|
1116
|
+
|
|
1117
|
+
- **Memory Adapter:** All jobs stored in RAM; limit maxJobHistory.
|
|
1118
|
+
- **SQLite Adapter:** Disk-based; memory usage scales with active jobs.
|
|
1119
|
+
- **BullMQ Adapter:** Redis memory; configure maxmemory policy.
|
|
1120
|
+
|
|
1121
|
+
---
|
|
1122
|
+
|
|
1123
|
+
## X. SECURITY BEST PRACTICES
|
|
1124
|
+
|
|
1125
|
+
### 30. Input Validation
|
|
1126
|
+
|
|
1127
|
+
Always validate job inputs using schemas:
|
|
1128
|
+
|
|
1129
|
+
```typescript
|
|
1130
|
+
import { z } from "zod";
|
|
1131
|
+
|
|
1132
|
+
const emailQueue = IgniterQueue.create("email")
|
|
1133
|
+
.addJob("send", {
|
|
1134
|
+
input: z.object({
|
|
1135
|
+
to: z.string().email(),
|
|
1136
|
+
subject: z.string().max(200),
|
|
1137
|
+
body: z.string(),
|
|
1138
|
+
}),
|
|
1139
|
+
handler: async ({ input }) => {
|
|
1140
|
+
// input is validated and typed
|
|
1141
|
+
},
|
|
1142
|
+
})
|
|
1143
|
+
.build();
|
|
1144
|
+
```
|
|
546
1145
|
|
|
547
|
-
###
|
|
1146
|
+
### 31. Sensitive Data Handling
|
|
548
1147
|
|
|
549
|
-
|
|
550
|
-
|
|
551
|
-
|
|
552
|
-
| `vitest` | `^1.0.0` | Testing framework |
|
|
553
|
-
| `tsup` | `^8.0.0` | Build tool |
|
|
1148
|
+
- **Never log raw inputs** containing passwords, tokens, or PII.
|
|
1149
|
+
- **Use job IDs** for reference rather than storing sensitive data in jobs.
|
|
1150
|
+
- **Encrypt at rest** if storing sensitive metadata.
|
|
554
1151
|
|
|
555
1152
|
---
|
|
556
1153
|
|
|
557
|
-
**Remember:** Always follow existing patterns. When in doubt, check how similar features are implemented in `@igniter-js/store` or `@igniter-js/core`.
|