@iikareem/litequeue 1.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 liteQueue Contributors
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md ADDED
@@ -0,0 +1,413 @@
1
+ <div align="center">
2
+
3
+ # liteQueue
4
+
5
+ **A persistent, zero-infrastructure task queue for Node.js — powered by SQLite.**
6
+
7
+ [![GitHub](https://img.shields.io/badge/GitHub-iikareem/liteQueue-181717?logo=github)](https://github.com/iikareem/liteQueue)
8
+ [![node](https://img.shields.io/badge/node-%3E%3D18.0.0-339933?logo=nodedotjs)](https://nodejs.org)
9
+ [![license](https://img.shields.io/badge/license-MIT-blue)](./LICENSE)
10
+ [![TypeScript](https://img.shields.io/badge/TypeScript-ready-3178C6?logo=typescript)](https://www.typescriptlang.org)
11
+ [![npm](https://img.shields.io/badge/npm-v1.0.1-CB3837?logo=npm)](https://www.npmjs.com/package/@iikareem/litequeue)
12
+
13
+ Delayed scheduling · Atomic job locking · Exponential backoff · CPU thread isolation
14
+
15
+ **No Redis. No Docker. No infrastructure.**
16
+
17
+ </div>
18
+
19
+ ---
20
+
21
+ ## Why liteQueue?
22
+
23
+ Most apps don't need Redis. They need a reliable way to run background jobs without spinning up external services, managing connections, or paying for more infrastructure.
24
+
25
+ | Feature | External Redis/Infra | liteQueue (SQLite) |
26
+ | :--- | :--- | :--- |
27
+ | **Visibility** | No visibility into what's running. | You can inspect, pause, and retry jobs. |
28
+ | **Control** | Wait for the next poll cycle. | Trigger what is available to run right now. |
29
+ | **Performance** | No insight into execution time. | See exactly how much time each job takes. |
30
+ | **History** | Jobs are gone once processed. | Full history of completed and failed jobs. |
31
+
32
+ **Your data stays local.** Because liteQueue uses SQLite, all job data is stored on your local disk rather than being sent over a network. This eliminates:
33
+ - **Network Latency:** No round-trips to an external database.
34
+ - **TLS Overhead:** No encryption/decryption cycles for every job enqueue.
35
+ - **Connection Complexity:** No connection pooling or TCP handshake failures.
36
+
37
+ Every external dependency adds a new failure domain. liteQueue eliminates all of it — your queue runs in-process. No TCP connections, no serialization hops, no dropped connections to retry.
38
+
39
+ liteQueue uses SQLite as a persistent state machine. Jobs survive crashes, restarts, and deploys. Workers are isolated. Retries are automatic. And the entire thing is a single `npm install`.
40
+
41
+ ```
42
+ Your App
43
+
44
+ ├── queue.register('send-email', handler) ← runs once at boot
45
+
46
+ └── sendEmail({ to: 'user@example.com' }) ← runs on demand, anywhere
47
+
48
+
49
+ SQLite (WAL mode)
50
+
51
+ ┌─────┴─────────┐
52
+ │ │
53
+ pending → processing → completed
54
+ └→ failed (after retries exhausted)
55
+ ```
56
+
57
+ ---
58
+
59
+ ## Install
60
+
61
+ ```bash
62
+ npm install @iikareem/litequeue
63
+ ```
64
+
65
+ **Requirements:** Node.js ≥ 18.0.0
66
+
67
+ ---
68
+
69
+ ## Quick Start
70
+
71
+ ```typescript
72
+ import { LiteQ } from '@iikareem/litequeue';
73
+
74
+ const queue = new LiteQ({ storagePath: './jobs.db' });
75
+
76
+ // Register an I/O handler — returns a typed enqueuer function
77
+ const sendEmail = queue.register<{ to: string; subject: string }>(
78
+ 'send-email',
79
+ async (job) => {
80
+ await mailer.send(job.data.to, job.data.subject);
81
+ return { sent: true };
82
+ }
83
+ );
84
+
85
+ // Register a CPU handler — runs in a dedicated worker thread
86
+ const generatePdf = queue.register('generate-pdf', './workers/pdf-worker.js');
87
+
88
+ // Start the polling engine
89
+ await queue.start();
90
+
91
+ // Call the enqueuer from anywhere — fully typed, no raw strings
92
+ await sendEmail({ to: 'user@example.com', subject: 'Welcome!' });
93
+ await generatePdf({ orderId: 'ord_123' });
94
+ ```
95
+
96
+ The string `'send-email'` is written **once** inside `register()`. The returned function is typed to your payload — no magic strings, no mismatches.
97
+
98
+ ---
99
+
100
+ ## How It Works
101
+
102
+ ### The Job Lifecycle
103
+
104
+ | Step | What Happens |
105
+ |---|---|
106
+ | **Register** | `queue.register()` stores the handler and determines the execution type: **I/O** (function arg) or **CPU** (string path arg). Returns a typed enqueuer. |
107
+ | **Enqueue** | Calling the enqueuer writes a row to SQLite with status `'pending'` and the execution type (`'io'` or `'worker'`). |
108
+ | **Poll** | `queue.start()` polls every `pollInterval` ms, running `tryClaimIo()` and `tryClaimCpu()` independently. |
109
+ | **Claim — I/O** | `tryClaimIo()` claims jobs where `type = 'io'`, gated by the `concurrency` counter. The handler runs directly on the main thread. |
110
+ | **Claim — CPU** | `tryClaimCpu()` claims jobs where `type = 'worker'`, gated by the pool's `canAccept` (idle worker or room to spawn). Dispatched to a generic worker thread. |
111
+ | **Success** | Status shifts to `'completed'`. |
112
+ | **Failure** | Status returns to `'pending'`, attempts increment, `run_at` bumps with exponential backoff. |
113
+ | **Dead** | After `max_retries` exhausted, status shifts to `'failed'`. |
114
+
115
+ I/O and CPU jobs never block each other — they use independent concurrency controls.
116
+
117
+ ### The Generic Worker Pool
118
+
119
+ CPU-bound jobs run in a pool of reusable worker threads. The pool is not coupled to handler paths — any idle worker can dynamically import and run any handler module.
120
+
121
+ ```
122
+ pool.execute(handlerPath, job)
123
+
124
+ ├── idle worker? → dispatch(worker, handlerPath, job)
125
+
126
+ ├── room to spawn? → spawn() → dispatch(newWorker, handlerPath, job)
127
+
128
+ └── busy + at maxWorkers → queue internally → (dispatched when a worker frees)
129
+ ```
130
+
131
+ When a job completes, the worker is marked idle and the pool drains its internal queue. Excess idle workers above `minWorkers` are automatically trimmed.
132
+
133
+ ### Delivery Guarantee & Crash Recovery
134
+
135
+ liteQueue provides **at least once** delivery — a job will always run, but in rare cases (crash after execution, before the success is committed) it may retry. This means your handlers must be **idempotent**: running the same job twice should produce the same result as running it once.
136
+
137
+ On restart, any job stuck in `'processing'` beyond `jobTimeout` is returned to `'pending'` and retried. **No job is ever silently lost.**
138
+
139
+ ---
140
+
141
+ ## API
142
+
143
+ ### Initialization
144
+
145
+ ```typescript
146
+ import { LiteQ } from '@iikareem/litequeue';
147
+
148
+ const queue = new LiteQ({
149
+ storagePath: './data/jobs.db', // or ':memory:' for tests
150
+ concurrency: 4, // max concurrent I/O jobs (default: 1)
151
+ pollInterval: 500, // ms between DB polls (default: 500)
152
+ jobTimeout: 60_000, // ms before a stuck job is released (default: 60000)
153
+ minWorkers: 2, // min idle worker threads kept alive (default: 1)
154
+ maxWorkers: 4, // max concurrent worker threads (default: 4)
155
+ });
156
+ ```
157
+
158
+ | Option | Type | Default | Description |
159
+ |---|---|---|---|
160
+ | `storagePath` | `string` | — | SQLite file path (`':memory:'` for tests) |
161
+ | `concurrency` | `number` | `1` | Max concurrent I/O jobs on the main thread |
162
+ | `pollInterval` | `number` | `500` | ms between DB polls |
163
+ | `jobTimeout` | `number` | `60000` | ms before a stuck `'processing'` job is released |
164
+ | `minWorkers` | `number` | `1` | Min idle workers to keep alive |
165
+ | `maxWorkers` | `number` | `4` | Max concurrent worker threads |
166
+
167
+ ---
168
+
169
+ ### `queue.register()` — Register a Handler
170
+
171
+ Registers a handler and returns a **typed enqueuer function**. The job type string lives only here — never repeated.
172
+
173
+ #### I/O Bound (async callback)
174
+
175
+ Use for state-changing operations that **must** be durable: sending emails, SMS, processing payments, or triggering webhooks. These are tasks where you need at-least-once delivery and built-in retry logic to ensure the work is eventually completed without being "lost" or manually retried.
176
+
177
+ Because liteQueue is **at least once**, a job may retry if a crash happens after execution but before the success is committed. Pass an **idempotency key** (e.g. `job.id`) to the external provider — it skips the work if it already saw that key.
178
+
179
+ ```typescript
180
+ const sendEmail = queue.register<{ email: string; templateId: string }>(
181
+ 'send-transactional-email',
182
+ async (job) => {
183
+ // Use job.id as the idempotency key — the provider
184
+ // guarantees it only processes this once.
185
+ await emailProvider.send(job.data.email, job.data.templateId, {
186
+ idempotencyKey: job.id,
187
+ });
188
+ return { sent: true };
189
+ }
190
+ );
191
+ ```
192
+
193
+ Avoid using this for simple, read-only `fetch` calls that don't require persistence or crash recovery.
194
+
195
+ `register()` detects the function argument and marks this job with `type = 'io'` in the DB. Concurrency is managed by the `concurrency` counter — these run on the main thread.
196
+
197
+ #### CPU Bound (worker thread)
198
+
199
+ Pass a file path instead of a callback. liteQueue detects the string, resolves it to an absolute path, and marks the job with `type = 'worker'`. It runs in the generic worker pool, keeping the main event loop unblocked.
200
+
201
+ Write a **handler module** — a file that exports a default async function. No `worker_threads` API needed.
202
+
203
+ ```typescript
204
+ const generatePdf = queue.register('generate-pdf', './workers/pdf-worker.js');
205
+ ```
206
+
207
+ ```typescript
208
+ // workers/pdf-worker.js — runs in an isolated CPU thread
209
+ export default async function (job) {
210
+ const url = await buildAndUploadPdf(job.data);
211
+ return { url };
212
+ }
213
+ ```
214
+
215
+ Handler modules are dynamically imported by liteQueue's generic worker. Any idle thread can run any handler — the pool is not coupled to paths. Throw inside the handler and the error automatically propagates to liteQueue's retry logic.
216
+
217
+ **The `job` object passed to your handler:**
218
+
219
+ | Property | Type | Description |
220
+ |---|---|---|
221
+ | `job.id` | `string` | UUID |
222
+ | `job.taskType` | `string` | The registered type name |
223
+ | `job.data` | `T` | Your typed payload |
224
+ | `job.attempts` | `number` | How many times this job has run |
225
+ | `job.maxRetries` | `number` | Max attempts before permanent failure |
226
+
227
+ ---
228
+
229
+ ### Enqueueing Jobs
230
+
231
+ Use the function returned by `register()`.
232
+
233
+ ```typescript
234
+ // Immediate
235
+ await sendEmail({ email: 'user@domain.com', templateId: 'welcome_v2' });
236
+
237
+ // Delayed — run 1 hour from now
238
+ await checkTrialExpiry({ userId: 'usr_9011' }, { delay: 60 * 60 * 1000 });
239
+
240
+ // With custom retry config — retries at 1s → 2s → 4s → 8s → 16s
241
+ await syncLedger({ transactionId: 'ch_3Mv1' }, { maxRetries: 5 });
242
+
243
+ // With priority — runs before lower-priority jobs
244
+ await sendAlert(data, { priority: 100 });
245
+ ```
246
+
247
+ **Options:**
248
+
249
+ | Option | Type | Default | Description |
250
+ |---|---|---|---|
251
+ | `delay` | `number` | `0` | Milliseconds before the job becomes eligible |
252
+ | `maxRetries` | `number` | `3` | Max retry attempts before permanent failure |
253
+ | `priority` | `number` | `10` | Higher value = runs first |
254
+
255
+ ---
256
+
257
+ ### Lifecycle Methods
258
+
259
+ ```typescript
260
+ await queue.start(); // Begin polling — call once at boot
261
+
262
+ await queue.stop(); // Graceful shutdown — drains pool, finishes in-flight jobs
263
+
264
+ const stats = await queue.stats();
265
+ // { pending: 3, processing: 1, completed: 142, failed: 2, total: 148 }
266
+
267
+ await queue.purge({ olderThan: 7 * 24 * 60 * 60 * 1000 });
268
+ // Removes completed/failed jobs older than 7 days
269
+ ```
270
+
271
+ ---
272
+
273
+ ## Recommended Project Setup
274
+
275
+ ```typescript
276
+ // queue.ts — create the instance once
277
+ import { LiteQ } from '@iikareem/litequeue';
278
+ export const queue = new LiteQ({ storagePath: './jobs.db' });
279
+ ```
280
+
281
+ ```typescript
282
+ // jobs/index.ts — register all handlers, export enqueuers
283
+ import { queue } from '../queue.js';
284
+
285
+ export const sendEmail = queue.register<{ to: string }>(
286
+ 'send-email',
287
+ async (job) => { /* ... */ }
288
+ );
289
+
290
+ export const generateReport = queue.register(
291
+ 'generate-report',
292
+ './workers/report.js'
293
+ );
294
+ ```
295
+
296
+ ```typescript
297
+ // main.ts — boot
298
+ import { queue } from './queue.js';
299
+ import './jobs/index.js'; // registers all handlers
300
+
301
+ await queue.start();
302
+ ```
303
+
304
+ ```typescript
305
+ // anywhere in your app
306
+ import { sendEmail } from './jobs/index.js';
307
+
308
+ await sendEmail({ to: 'user@example.com' });
309
+ ```
310
+
311
+ ---
312
+
313
+ ## Comparison
314
+
315
+ | | liteQueue | BullMQ | Bee-Queue |
316
+ |---|---|---|---|
317
+ | Infrastructure required | **None** | Redis | Redis |
318
+ | Persistent jobs | ✅ | ✅ | ❌ |
319
+ | Survives crashes | ✅ | ✅ | ❌ |
320
+ | CPU thread isolation | ✅ | ❌ | ❌ |
321
+ | Delayed scheduling | ✅ | ✅ | ✅ |
322
+ | Exponential backoff | ✅ | ✅ | ✅ |
323
+ | Zero runtime deps | ✅ | ❌ | ❌ |
324
+ | TypeScript built-in | ✅ | ✅ | ❌ |
325
+ | Multi-machine workers | ❌ | ✅ | ✅ |
326
+
327
+ **liteQueue is the right choice when** you want BullMQ-level reliability without operating Redis. If you need workers across multiple machines, use BullMQ.
328
+
329
+ ---
330
+
331
+ ## Database Internals
332
+
333
+ liteQueue configures SQLite on startup for maximum concurrency and durability:
334
+
335
+ ```sql
336
+ PRAGMA journal_mode = WAL; -- concurrent readers, single writer
337
+ PRAGMA busy_timeout = 5000; -- wait up to 5s on write contention
338
+ PRAGMA synchronous = NORMAL; -- crash-safe without full fsync overhead
339
+ ```
340
+
341
+ The `type` column distinguishes I/O jobs (main thread) from CPU jobs (worker thread), so each claim path queries only its own job type.
342
+
343
+ ```sql
344
+ CREATE TABLE liteQueue_jobs (
345
+ id TEXT PRIMARY KEY,
346
+ name TEXT NOT NULL, -- job type name: 'send-email', 'resize-image', etc.
347
+ type TEXT NOT NULL, -- execution type: 'io' or 'worker'
348
+ payload TEXT NOT NULL,
349
+ status TEXT NOT NULL DEFAULT 'pending',
350
+ attempts INTEGER DEFAULT 0,
351
+ max_retries INTEGER DEFAULT 3,
352
+ priority INTEGER DEFAULT 10,
353
+ run_at INTEGER NOT NULL, -- epoch ms, eligible run time
354
+ locked_at INTEGER, -- set when status = 'processing'
355
+ error_log TEXT
356
+ );
357
+
358
+ -- Prevents full table scans during high-frequency polling
359
+ CREATE INDEX IF NOT EXISTS idx_liteQueue_polling
360
+ ON liteQueue_jobs (status, type, run_at, priority DESC);
361
+ ```
362
+
363
+ ---
364
+
365
+ ## Roadmap
366
+
367
+ ### ✅ v1.0 — Core Engine
368
+ SQLite WAL persistence · Atomic job locking · I/O + CPU concurrency separation · Generic worker pool with minWorkers/maxWorkers lifecycle · Exponential backoff · Delayed scheduling · Priority queues · Graceful shutdown · Handler modules (no `worker_threads` boilerplate)
369
+
370
+ ### 🔜 v1.1 — Scheduled Jobs
371
+ ```typescript
372
+ queue.schedule('0 0 * * *', 'cleanup-sessions', {});
373
+ ```
374
+
375
+ ### 🔜 v1.2 — Built-in Idempotency
376
+ ```typescript
377
+ await sendSms(data, {
378
+ uniqueKey: 'sms-verify-+1234567890',
379
+ uniqueWithin: 5 * 60 * 1000,
380
+ });
381
+ ```
382
+
383
+ ### 🔜 v2.0 — Observability Dashboard
384
+ Zero-config HTTP dashboard for queue observability — no external UI framework.
385
+
386
+ ---
387
+
388
+ ## FAQ
389
+
390
+ **Does this work with multiple processes?**
391
+ Yes. WAL mode supports concurrent readers and `BEGIN IMMEDIATE TRANSACTION` ensures no two processes ever claim the same job, even across separate OS processes on the same machine.
392
+
393
+ **What happens if my app crashes mid-job?**
394
+ Any job stuck in `'processing'` beyond `jobTimeout` is automatically returned to `'pending'` on the next restart. Because the success may or may not have been committed before the crash, liteQueue provides **at least once** delivery — your handler should use an idempotency key (e.g. `job.id`) to detect and skip duplicates.
395
+
396
+ **When should I use BullMQ instead?**
397
+ When you need workers distributed across multiple machines, or throughput above tens of thousands of jobs per second. liteQueue is intentionally scoped to single-node deployments.
398
+
399
+ **Is TypeScript required?**
400
+ No — works with plain JavaScript too. TypeScript types are bundled; no separate `@types` package needed.
401
+
402
+ ---
403
+
404
+ ## Links
405
+
406
+ - **Source:** [github.com/iikareem/liteQueue](https://github.com/iikareem/liteQueue)
407
+ - **Issues:** [github.com/iikareem/liteQueue/issues](https://github.com/iikareem/liteQueue/issues)
408
+
409
+ ---
410
+
411
+ ## License
412
+
413
+ MIT © liteQueue Contributors
package/dist/db.d.ts ADDED
@@ -0,0 +1,43 @@
1
+ export type ExecType = 'io' | 'worker';
2
+ interface EnqueueRow {
3
+ id: string;
4
+ name: string;
5
+ type: ExecType;
6
+ payload: string;
7
+ runAt: number;
8
+ maxRetries: number;
9
+ priority: number;
10
+ }
11
+ export interface ClaimedJob {
12
+ id: string;
13
+ name: string;
14
+ type: ExecType;
15
+ payload: string;
16
+ attempts: number;
17
+ maxRetries: number;
18
+ }
19
+ export declare class DB {
20
+ private db;
21
+ private readonly enqueueStmt;
22
+ private readonly selectNextStmt;
23
+ private readonly lockJobStmt;
24
+ private readonly lockCrashedJobStmt;
25
+ private readonly completeStmt;
26
+ private readonly failStmt;
27
+ private readonly retryStmt;
28
+ constructor(storagePath: string);
29
+ initialize(): void;
30
+ enqueue(row: EnqueueRow): void;
31
+ claimNext(now: number, timeout: number, type: ExecType): ClaimedJob | null;
32
+ complete(jobId: string, completedAt: number): void;
33
+ fail(jobId: string, errorLog: string, completedAt: number): void;
34
+ retry(jobId: string, nextRunAt: number): void;
35
+ stats(): {
36
+ status: string;
37
+ count: number;
38
+ }[];
39
+ purge(before: number): void;
40
+ close(): void;
41
+ }
42
+ export {};
43
+ //# sourceMappingURL=db.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"db.d.ts","sourceRoot":"","sources":["../src/db.ts"],"names":[],"mappings":"AAEA,MAAM,MAAM,QAAQ,GAAG,IAAI,GAAG,QAAQ,CAAC;AAEvC,UAAU,UAAU;IAChB,EAAE,EAAE,MAAM,CAAC;IACX,IAAI,EAAE,MAAM,CAAC;IACb,IAAI,EAAE,QAAQ,CAAC;IACf,OAAO,EAAE,MAAM,CAAC;IAChB,KAAK,EAAE,MAAM,CAAC;IACd,UAAU,EAAE,MAAM,CAAC;IACnB,QAAQ,EAAE,MAAM,CAAC;CACpB;AAkBD,MAAM,WAAW,UAAU;IACvB,EAAE,EAAE,MAAM,CAAC;IACX,IAAI,EAAE,MAAM,CAAC;IACb,IAAI,EAAE,QAAQ,CAAC;IACf,OAAO,EAAE,MAAM,CAAC;IAChB,QAAQ,EAAE,MAAM,CAAC;IACjB,UAAU,EAAE,MAAM,CAAC;CACtB;AAsED,qBAAa,EAAE;IACX,OAAO,CAAC,EAAE,CAAoB;IAC9B,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAC;IAC7B,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAC;IAChC,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAC;IAC7B,OAAO,CAAC,QAAQ,CAAC,kBAAkB,CAAC;IACpC,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAC;IAC9B,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAC;IAC1B,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAC;gBAEf,WAAW,EAAE,MAAM;IA6D/B,UAAU,IAAI,IAAI;IAOlB,OAAO,CAAC,GAAG,EAAE,UAAU,GAAG,IAAI;IAI9B,SAAS,CAAC,GAAG,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,EAAE,IAAI,EAAE,QAAQ,GAAG,UAAU,GAAG,IAAI;IAiB1E,QAAQ,CAAC,KAAK,EAAE,MAAM,EAAE,WAAW,EAAE,MAAM,GAAG,IAAI;IAIlD,IAAI,CAAC,KAAK,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,EAAE,WAAW,EAAE,MAAM,GAAG,IAAI;IAIhE,KAAK,CAAC,KAAK,EAAE,MAAM,EAAE,SAAS,EAAE,MAAM,GAAG,IAAI;IAI7C,KAAK,IAAI;QAAE,MAAM,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,MAAM,CAAA;KAAE,EAAE;IAM5C,KAAK,CAAC,MAAM,EAAE,MAAM,GAAG,IAAI;IAM3B,KAAK,IAAI,IAAI;CAGhB"}
package/dist/db.js ADDED
@@ -0,0 +1,176 @@
1
+ import Database from 'better-sqlite3';
2
+ const SCHEMA = `
3
+ DROP TABLE IF EXISTS liteQueue_jobs;
4
+
5
+ CREATE TABLE IF NOT EXISTS liteQueue_jobs
6
+ (
7
+ id
8
+ TEXT
9
+ PRIMARY
10
+ KEY,
11
+ name
12
+ TEXT
13
+ NOT
14
+ NULL,
15
+ type
16
+ TEXT
17
+ NOT
18
+ NULL,
19
+ payload
20
+ TEXT
21
+ NOT
22
+ NULL,
23
+ status
24
+ TEXT
25
+ NOT
26
+ NULL
27
+ DEFAULT
28
+ 'pending',
29
+ attempts
30
+ INTEGER
31
+ DEFAULT
32
+ 0,
33
+ max_retries
34
+ INTEGER
35
+ DEFAULT
36
+ 3,
37
+ priority
38
+ INTEGER
39
+ DEFAULT
40
+ 10,
41
+ run_at
42
+ INTEGER
43
+ NOT
44
+ NULL,
45
+ locked_at
46
+ INTEGER,
47
+ started_at
48
+ INTEGER,
49
+ completed_at
50
+ INTEGER,
51
+ error_log
52
+ TEXT
53
+ );
54
+
55
+ CREATE INDEX IF NOT EXISTS idx_liteQueue_polling
56
+ ON liteQueue_jobs (status, type, run_at, priority DESC);
57
+ `;
58
+ function toClaimedJob(row) {
59
+ return {
60
+ id: row.id,
61
+ name: row.name,
62
+ type: row.type,
63
+ payload: row.payload,
64
+ attempts: row.attempts,
65
+ maxRetries: row.max_retries,
66
+ };
67
+ }
68
+ export class DB {
69
+ db;
70
+ enqueueStmt;
71
+ selectNextStmt;
72
+ lockJobStmt;
73
+ lockCrashedJobStmt;
74
+ completeStmt;
75
+ failStmt;
76
+ retryStmt;
77
+ constructor(storagePath) {
78
+ this.db = new Database(storagePath);
79
+ this.initialize();
80
+ this.enqueueStmt = this.db.prepare(`
81
+ INSERT INTO liteQueue_jobs (id, name, type, payload, run_at, max_retries, priority)
82
+ VALUES (@id, @name, @type, @payload, @runAt, @maxRetries, @priority)
83
+ `);
84
+ this.selectNextStmt = this.db.prepare(`
85
+ SELECT *
86
+ FROM liteQueue_jobs
87
+ WHERE (status = 'pending' AND run_at <= ? AND type = ?)
88
+ OR (status = 'processing' AND started_at + ? <= ? AND type = ? AND attempts < max_retries)
89
+ ORDER BY priority DESC, run_at ASC LIMIT 1
90
+ `);
91
+ this.lockJobStmt = this.db.prepare(`
92
+ UPDATE liteQueue_jobs
93
+ SET status = 'processing',
94
+ locked_at = ?,
95
+ started_at = ?
96
+ WHERE id = ?
97
+ `);
98
+ this.lockCrashedJobStmt = this.db.prepare(`
99
+ UPDATE liteQueue_jobs
100
+ SET locked_at = ?,
101
+ started_at = ?,
102
+ attempts = ?
103
+ WHERE id = ?
104
+ `);
105
+ this.completeStmt = this.db.prepare(`
106
+ UPDATE liteQueue_jobs
107
+ SET status = 'completed',
108
+ completed_at = ?
109
+ WHERE id = ?
110
+ `);
111
+ this.failStmt = this.db.prepare(`
112
+ UPDATE liteQueue_jobs
113
+ SET status = 'failed',
114
+ error_log = ?,
115
+ completed_at = ?
116
+ WHERE id = ?
117
+ `);
118
+ this.retryStmt = this.db.prepare(`
119
+ UPDATE liteQueue_jobs
120
+ SET status = 'pending',
121
+ attempts = attempts + 1,
122
+ locked_at = NULL,
123
+ started_at = NULL,
124
+ completed_at = NULL,
125
+ run_at = ?
126
+ WHERE id = ?
127
+ `);
128
+ }
129
+ initialize() {
130
+ this.db.pragma('journal_mode = WAL');
131
+ this.db.pragma('busy_timeout = 5000');
132
+ this.db.pragma('synchronous = NORMAL');
133
+ this.db.exec(SCHEMA);
134
+ }
135
+ enqueue(row) {
136
+ this.enqueueStmt.run(row);
137
+ }
138
+ claimNext(now, timeout, type) {
139
+ const claim = this.db.transaction(() => {
140
+ const row = this.selectNextStmt.get(now, type, timeout, now, type);
141
+ if (!row)
142
+ return null;
143
+ if (row.status === 'processing') {
144
+ this.lockCrashedJobStmt.run(now, now, row.attempts + 1, row.id);
145
+ }
146
+ else {
147
+ this.lockJobStmt.run(now, now, row.id);
148
+ }
149
+ return row;
150
+ })();
151
+ return claim ? toClaimedJob(claim) : null;
152
+ }
153
+ complete(jobId, completedAt) {
154
+ this.completeStmt.run(completedAt, jobId);
155
+ }
156
+ fail(jobId, errorLog, completedAt) {
157
+ this.failStmt.run(errorLog, completedAt, jobId);
158
+ }
159
+ retry(jobId, nextRunAt) {
160
+ this.retryStmt.run(nextRunAt, jobId);
161
+ }
162
+ stats() {
163
+ return this.db
164
+ .prepare(`SELECT status, COUNT(*) AS count FROM liteQueue_jobs GROUP BY status`)
165
+ .all();
166
+ }
167
+ purge(before) {
168
+ this.db
169
+ .prepare(`DELETE FROM liteQueue_jobs WHERE status IN ('completed', 'failed') AND completed_at < ?`)
170
+ .run(before);
171
+ }
172
+ close() {
173
+ this.db.close();
174
+ }
175
+ }
176
+ //# sourceMappingURL=db.js.map
package/dist/db.js.map ADDED
@@ -0,0 +1 @@
1
+ {"version":3,"file":"db.js","sourceRoot":"","sources":["../src/db.ts"],"names":[],"mappings":"AAAA,OAAO,QAAQ,MAAM,gBAAgB,CAAC;AAuCtC,MAAM,MAAM,GAAG;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;CAuDd,CAAC;AAEF,SAAS,YAAY,CAAC,GAAU;IAC5B,OAAO;QACH,EAAE,EAAE,GAAG,CAAC,EAAE;QACV,IAAI,EAAE,GAAG,CAAC,IAAI;QACd,IAAI,EAAE,GAAG,CAAC,IAAgB;QAC1B,OAAO,EAAE,GAAG,CAAC,OAAO;QACpB,QAAQ,EAAE,GAAG,CAAC,QAAQ;QACtB,UAAU,EAAE,GAAG,CAAC,WAAW;KAC9B,CAAC;AACN,CAAC;AAED,MAAM,OAAO,EAAE;IACH,EAAE,CAAoB;IACb,WAAW,CAAC;IACZ,cAAc,CAAC;IACf,WAAW,CAAC;IACZ,kBAAkB,CAAC;IACnB,YAAY,CAAC;IACb,QAAQ,CAAC;IACT,SAAS,CAAC;IAE3B,YAAY,WAAmB;QAC3B,IAAI,CAAC,EAAE,GAAG,IAAI,QAAQ,CAAC,WAAW,CAAC,CAAC;QACpC,IAAI,CAAC,UAAU,EAAE,CAAC;QAElB,IAAI,CAAC,WAAW,GAAG,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC;;;SAGlC,CAAC,CAAC;QAEH,IAAI,CAAC,cAAc,GAAG,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC;;;;;;SAMrC,CAAC,CAAC;QAEH,IAAI,CAAC,WAAW,GAAG,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC;;;;;;SAMlC,CAAC,CAAC;QAGH,IAAI,CAAC,kBAAkB,GAAG,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC;;;;;;SAMzC,CAAC,CAAC;QAEH,IAAI,CAAC,YAAY,GAAG,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC;;;;;SAKnC,CAAC,CAAC;QAEH,IAAI,CAAC,QAAQ,GAAG,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC;;;;;;SAM/B,CAAC,CAAC;QAEH,IAAI,CAAC,SAAS,GAAG,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC;;;;;;;;;SAShC,CAAC,CAAC;IACP,CAAC;IAED,UAAU;QACN,IAAI,CAAC,EAAE,CAAC,MAAM,CAAC,oBAAoB,CAAC,CAAC;QACrC,IAAI,CAAC,EAAE,CAAC,MAAM,CAAC,qBAAqB,CAAC,CAAC;QACtC,IAAI,CAAC,EAAE,CAAC,MAAM,CAAC,sBAAsB,CAAC,CAAC;QACvC,IAAI,CAAC,EAAE,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;IACzB,CAAC;IAED,OAAO,CAAC,GAAe;QACnB,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;IAC9B,CAAC;IAED,SAAS,CAAC,GAAW,EAAE,OAAe,EAAE,IAAc;QAClD,MAAM,KAAK,GAAG,IAAI,CAAC,EAAE,CAAC,WAAW,CAAC,GAAG,EAAE;YACnC,MAAM,GAAG,GAAG,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,GAAG,EAAE,IAAI,EAAE,OAAO,EAAE,GAAG,EAAE,IAAI,CAAsB,CAAC;YACxF,IAAI,CAAC,GAAG;gBAAE,OAAO,IAAI,CAAC;YAEtB,IAAI,GAAG,CAAC,MAAM,KAAK,YAAY,EAAE,CAAC;gBAC9B,IAAI,CAAC,kBAAkB,CAAC,GAAG,CAAC,GAAG,EAAE,GAAG,EAAE,GAAG,CAAC,QAAQ,GAAG,CAAC,EAAE,GAAG,CAAC,EAAE,CAAC,CAAC;YACpE,CAAC;iBAAM,CAAC;gBACJ,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,GAAG,EAAE,GAAG,EAAE,GAAG,CAAC,EAAE,CAAC,CAAC;YAC3C,CAAC;YAED,OAAO,GAAG,CAAC;QACf,CAAC,CAAC,EAAE,CAAC;QAEL,OAAO,KAAK,CAAC,CAAC,CAAC,YAAY,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;IAC9C,CAAC;IAED,QAAQ,CAAC,KAAa,EAAE,WAAmB;QACvC,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,WAAW,EAAE,KAAK,CAAC,CAAC;IAC9C,CAAC;IAED,IAAI,CAAC,KAAa,EAAE,QAAgB,EAAE,WAAmB;QACrD,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,QAAQ,EAAE,WAAW,EAAE,KAAK,CAAC,CAAC;IACpD,CAAC;IAED,KAAK,CAAC,KAAa,EAAE,SAAiB;QAClC,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;IACzC,CAAC;IAED,KAAK;QACD,OAAO,IAAI,CAAC,EAAE;aACT,OAAO,CAAC,sEAAsE,CAAC;aAC/E,GAAG,EAAyC,CAAC;IACtD,CAAC;IAED,KAAK,CAAC,MAAc;QAChB,IAAI,CAAC,EAAE;aACF,OAAO,CAAC,yFAAyF,CAAC;aAClG,GAAG,CAAC,MAAM,CAAC,CAAC;IACrB,CAAC;IAED,KAAK;QACD,IAAI,CAAC,EAAE,CAAC,KAAK,EAAE,CAAC;IACpB,CAAC;CACJ"}
@@ -0,0 +1,18 @@
1
+ import { parentPort } from 'node:worker_threads';
2
+
3
+ parentPort.on('message', async ({ handlerPath, job }) => {
4
+ try {
5
+ const mod = await import(handlerPath);
6
+ const fn = mod.default;
7
+ if (typeof fn !== 'function') {
8
+ throw new Error(`Handler at "${handlerPath}" must export a default function`);
9
+ }
10
+ await fn(job);
11
+ parentPort.postMessage({ status: 'success' });
12
+ } catch (err) {
13
+ parentPort.postMessage({
14
+ status: 'error',
15
+ error: err instanceof Error ? err.message : String(err),
16
+ });
17
+ }
18
+ });
@@ -0,0 +1,3 @@
1
+ export { LiteQ } from './queue.js';
2
+ export type { LiteQOptions, Job, Enqueuer, EnqueueOptions, QueueStats, JobHandler, } from './types.js';
3
+ //# sourceMappingURL=index.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAC,KAAK,EAAC,MAAM,YAAY,CAAC;AACjC,YAAY,EACR,YAAY,EACZ,GAAG,EACH,QAAQ,EACR,cAAc,EACd,UAAU,EACV,UAAU,GACb,MAAM,YAAY,CAAC"}
package/dist/index.js ADDED
@@ -0,0 +1,2 @@
1
+ export { LiteQ } from './queue.js';
2
+ //# sourceMappingURL=index.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAC,KAAK,EAAC,MAAM,YAAY,CAAC"}
@@ -0,0 +1,27 @@
1
+ import type { Enqueuer, JobHandler, LiteQOptions, PurgeOptions, QueueStats } from './types.js';
2
+ export declare class LiteQ {
3
+ private readonly db;
4
+ private readonly handlers;
5
+ private readonly concurrency;
6
+ private readonly pollInterval;
7
+ private readonly jobTimeout;
8
+ private readonly pool;
9
+ private activeIo;
10
+ private timer;
11
+ constructor(options: LiteQOptions);
12
+ register<T>(type: string, handler: JobHandler<T>): Enqueuer<T>;
13
+ register(type: string, workerPath: string): Enqueuer<unknown>;
14
+ start(): Promise<void>;
15
+ stop(): Promise<void>;
16
+ stats(): Promise<QueueStats>;
17
+ purge(options: PurgeOptions): Promise<void>;
18
+ private enqueue;
19
+ private tick;
20
+ private tryClaimIo;
21
+ private tryClaimCpu;
22
+ private runJob;
23
+ private executeHandler;
24
+ private handleJobFailure;
25
+ private withTimeout;
26
+ }
27
+ //# sourceMappingURL=queue.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"queue.d.ts","sourceRoot":"","sources":["../src/queue.ts"],"names":[],"mappings":"AAKA,OAAO,KAAK,EAAiB,QAAQ,EAAO,UAAU,EAAE,YAAY,EAAE,YAAY,EAAE,UAAU,EAAE,MAAM,YAAY,CAAC;AAWnH,qBAAa,KAAK;IACd,OAAO,CAAC,QAAQ,CAAC,EAAE,CAAK;IACxB,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAA0C;IACnE,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAS;IACrC,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAS;IACtC,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAS;IACpC,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAa;IAClC,OAAO,CAAC,QAAQ,CAAK;IACrB,OAAO,CAAC,KAAK,CAA+C;gBAEhD,OAAO,EAAE,YAAY;IAYjC,QAAQ,CAAC,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,UAAU,CAAC,CAAC,CAAC,GAAG,QAAQ,CAAC,CAAC,CAAC;IAC9D,QAAQ,CAAC,IAAI,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,GAAG,QAAQ,CAAC,OAAO,CAAC;IAavD,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;IAItB,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC;IAQrB,KAAK,IAAI,OAAO,CAAC,UAAU,CAAC;IAkB5B,KAAK,CAAC,OAAO,EAAE,YAAY,GAAG,OAAO,CAAC,IAAI,CAAC;YAInC,OAAO;IA8BrB,OAAO,CAAC,IAAI;IAKZ,OAAO,CAAC,UAAU;IAUlB,OAAO,CAAC,WAAW;YASL,MAAM;IAqBpB,OAAO,CAAC,cAAc;IAOtB,OAAO,CAAC,gBAAgB;IAYxB,OAAO,CAAC,WAAW;CActB"}
package/dist/queue.js ADDED
@@ -0,0 +1,177 @@
1
+ import { randomUUID } from 'node:crypto';
2
+ import os from 'node:os';
3
+ import { resolve } from 'node:path';
4
+ import { DB } from './db.js';
5
+ import { WorkerPool } from './worker-pool.js';
6
+ const CPU_COUNT = os.cpus().length;
7
+ const DEFAULT_CONCURRENCY = 1;
8
+ const DEFAULT_POLL_INTERVAL = 500;
9
+ const DEFAULT_JOB_TIMEOUT = 60_000;
10
+ const DEFAULT_MIN_WORKERS = 1;
11
+ const DEFAULT_MAX_RETRIES = 3;
12
+ const DEFAULT_PRIORITY = 10;
13
+ export class LiteQ {
14
+ db;
15
+ handlers = new Map();
16
+ concurrency;
17
+ pollInterval;
18
+ jobTimeout;
19
+ pool;
20
+ activeIo = 0;
21
+ timer = null;
22
+ constructor(options) {
23
+ this.db = new DB(options.storagePath);
24
+ this.concurrency = options.concurrency ?? DEFAULT_CONCURRENCY;
25
+ this.pollInterval = options.pollInterval ?? DEFAULT_POLL_INTERVAL;
26
+ this.jobTimeout = options.jobTimeout ?? DEFAULT_JOB_TIMEOUT;
27
+ const minWorkers = options.minWorkers ?? DEFAULT_MIN_WORKERS;
28
+ const maxWorkers = resolveMaxWorkers(options.maxWorkers);
29
+ this.pool = new WorkerPool(minWorkers, maxWorkers);
30
+ }
31
+ register(type, handlerOrPath) {
32
+ if (typeof handlerOrPath === 'string') {
33
+ this.handlers.set(type, resolve(handlerOrPath));
34
+ return (data, options) => this.enqueue(type, data, options, 'worker');
35
+ }
36
+ this.handlers.set(type, handlerOrPath);
37
+ return (data, options) => this.enqueue(type, data, options, 'io');
38
+ }
39
+ async start() {
40
+ this.timer = setInterval(() => this.tick(), this.pollInterval);
41
+ }
42
+ async stop() {
43
+ if (this.timer) {
44
+ clearInterval(this.timer);
45
+ this.timer = null;
46
+ }
47
+ await this.pool.stop();
48
+ }
49
+ async stats() {
50
+ const rows = this.db.stats();
51
+ let total = 0;
52
+ const stats = { pending: 0, processing: 0, completed: 0, failed: 0, total: 0 };
53
+ for (const row of rows) {
54
+ const count = Number(row.count);
55
+ if (row.status === 'pending')
56
+ stats.pending = count;
57
+ else if (row.status === 'processing')
58
+ stats.processing = count;
59
+ else if (row.status === 'completed')
60
+ stats.completed = count;
61
+ else if (row.status === 'failed')
62
+ stats.failed = count;
63
+ total += count;
64
+ }
65
+ stats.total = total;
66
+ return stats;
67
+ }
68
+ async purge(options) {
69
+ this.db.purge(options.olderThan);
70
+ }
71
+ async enqueue(name, data, options, execType) {
72
+ const id = randomUUID();
73
+ const maxRetries = options?.maxRetries ?? DEFAULT_MAX_RETRIES;
74
+ const runAt = Date.now() + (options?.delay ?? 0);
75
+ this.db.enqueue({
76
+ id,
77
+ name,
78
+ type: execType,
79
+ payload: JSON.stringify(data),
80
+ runAt,
81
+ maxRetries,
82
+ priority: options?.priority ?? DEFAULT_PRIORITY,
83
+ });
84
+ return {
85
+ id,
86
+ taskType: name,
87
+ data,
88
+ attempts: 0,
89
+ maxRetries,
90
+ status: 'pending',
91
+ };
92
+ }
93
+ tick() {
94
+ this.tryClaimIo();
95
+ this.tryClaimCpu();
96
+ }
97
+ tryClaimIo() {
98
+ if (this.activeIo >= this.concurrency)
99
+ return;
100
+ const claimed = this.db.claimNext(Date.now(), this.jobTimeout, 'io');
101
+ if (!claimed)
102
+ return;
103
+ this.activeIo++;
104
+ this.runJob(claimed).finally(() => this.activeIo--);
105
+ }
106
+ tryClaimCpu() {
107
+ if (!this.pool.canAccept)
108
+ return;
109
+ const claimed = this.db.claimNext(Date.now(), this.jobTimeout, 'worker');
110
+ if (!claimed)
111
+ return;
112
+ this.runJob(claimed);
113
+ }
114
+ async runJob(claimed) {
115
+ const handler = this.handlers.get(claimed.name);
116
+ if (!handler) {
117
+ this.db.fail(claimed.id, `No handler registered for job type: ${claimed.name}`, Date.now());
118
+ return;
119
+ }
120
+ const job = toJob(claimed, JSON.parse(claimed.payload));
121
+ try {
122
+ await this.withTimeout(this.executeHandler(handler, job), this.jobTimeout);
123
+ this.db.complete(claimed.id, Date.now());
124
+ }
125
+ catch (err) {
126
+ this.handleJobFailure(claimed, err);
127
+ }
128
+ }
129
+ executeHandler(handler, job) {
130
+ if (typeof handler === 'function') {
131
+ return handler(job);
132
+ }
133
+ return this.pool.execute(handler, job);
134
+ }
135
+ handleJobFailure(claimed, err) {
136
+ const errorMsg = err instanceof Error ? err.message : String(err);
137
+ if (claimed.attempts < claimed.maxRetries) {
138
+ const backoffMs = Math.pow(2, claimed.attempts) * 1000;
139
+ this.db.retry(claimed.id, Date.now() + backoffMs);
140
+ return;
141
+ }
142
+ this.db.fail(claimed.id, errorMsg, Date.now());
143
+ }
144
+ withTimeout(promise, ms) {
145
+ let timeoutId;
146
+ const timeoutPromise = new Promise((_, reject) => {
147
+ timeoutId = setTimeout(() => reject(new Error(`Job timed out after ${ms}ms`)), ms);
148
+ });
149
+ return Promise.race([promise, timeoutPromise]).finally(() => {
150
+ clearTimeout(timeoutId);
151
+ });
152
+ }
153
+ }
154
+ function toJob(claimed, data) {
155
+ return {
156
+ id: claimed.id,
157
+ taskType: claimed.name,
158
+ data,
159
+ attempts: claimed.attempts,
160
+ maxRetries: claimed.maxRetries,
161
+ status: 'processing',
162
+ };
163
+ }
164
+ function resolveMaxWorkers(requested) {
165
+ const osReserved = Math.max(1, CPU_COUNT - 1);
166
+ if (requested === undefined) {
167
+ return Math.max(1, Math.floor(osReserved / 2));
168
+ }
169
+ if (requested > CPU_COUNT) {
170
+ throw new Error(`Your machine only has ${CPU_COUNT} cores, so you cannot use ${requested} workers.`);
171
+ }
172
+ if (requested > osReserved) {
173
+ console.warn(`Warning: Setting workers to ${requested} leaves no room for the OS. Recommended max is ${osReserved}.`);
174
+ }
175
+ return requested;
176
+ }
177
+ //# sourceMappingURL=queue.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"queue.js","sourceRoot":"","sources":["../src/queue.ts"],"names":[],"mappings":"AAAA,OAAO,EAAC,UAAU,EAAC,MAAM,aAAa,CAAC;AACvC,OAAO,EAAE,MAAM,SAAS,CAAC;AACzB,OAAO,EAAC,OAAO,EAAC,MAAM,WAAW,CAAC;AAElC,OAAO,EAAC,EAAE,EAAC,MAAM,SAAS,CAAC;AAE3B,OAAO,EAAC,UAAU,EAAC,MAAM,kBAAkB,CAAC;AAE5C,MAAM,SAAS,GAAG,EAAE,CAAC,IAAI,EAAE,CAAC,MAAM,CAAC;AACnC,MAAM,mBAAmB,GAAG,CAAC,CAAC;AAC9B,MAAM,qBAAqB,GAAG,GAAG,CAAC;AAClC,MAAM,mBAAmB,GAAG,MAAM,CAAC;AACnC,MAAM,mBAAmB,GAAG,CAAC,CAAC;AAC9B,MAAM,mBAAmB,GAAG,CAAC,CAAC;AAC9B,MAAM,gBAAgB,GAAG,EAAE,CAAC;AAE5B,MAAM,OAAO,KAAK;IACG,EAAE,CAAK;IACP,QAAQ,GAAG,IAAI,GAAG,EAA+B,CAAC;IAClD,WAAW,CAAS;IACpB,YAAY,CAAS;IACrB,UAAU,CAAS;IACnB,IAAI,CAAa;IAC1B,QAAQ,GAAG,CAAC,CAAC;IACb,KAAK,GAA0C,IAAI,CAAC;IAE5D,YAAY,OAAqB;QAC7B,IAAI,CAAC,EAAE,GAAG,IAAI,EAAE,CAAC,OAAO,CAAC,WAAW,CAAC,CAAC;QAEtC,IAAI,CAAC,WAAW,GAAG,OAAO,CAAC,WAAW,IAAI,mBAAmB,CAAC;QAC9D,IAAI,CAAC,YAAY,GAAG,OAAO,CAAC,YAAY,IAAI,qBAAqB,CAAC;QAClE,IAAI,CAAC,UAAU,GAAG,OAAO,CAAC,UAAU,IAAI,mBAAmB,CAAC;QAE5D,MAAM,UAAU,GAAG,OAAO,CAAC,UAAU,IAAI,mBAAmB,CAAC;QAC7D,MAAM,UAAU,GAAG,iBAAiB,CAAC,OAAO,CAAC,UAAU,CAAC,CAAC;QACzD,IAAI,CAAC,IAAI,GAAG,IAAI,UAAU,CAAC,UAAU,EAAE,UAAU,CAAC,CAAC;IACvD,CAAC;IAID,QAAQ,CAAI,IAAY,EAAE,aAAqC;QAC3D,IAAI,OAAO,aAAa,KAAK,QAAQ,EAAE,CAAC;YACpC,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,IAAI,EAAE,OAAO,CAAC,aAAa,CAAC,CAAC,CAAC;YAChD,OAAO,CAAC,IAAO,EAAE,OAAwB,EAAE,EAAE,CACzC,IAAI,CAAC,OAAO,CAAC,IAAI,EAAE,IAAI,EAAE,OAAO,EAAE,QAAQ,CAAC,CAAC;QACpD,CAAC;QAED,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,IAAI,EAAE,aAA2B,CAAC,CAAC;QACrD,OAAO,CAAC,IAAO,EAAE,OAAwB,EAAE,EAAE,CACzC,IAAI,CAAC,OAAO,CAAC,IAAI,EAAE,IAAI,EAAE,OAAO,EAAE,IAAI,CAAC,CAAC;IAChD,CAAC;IAED,KAAK,CAAC,KAAK;QACP,IAAI,CAAC,KAAK,GAAG,WAAW,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,IAAI,EAAE,EAAE,IAAI,CAAC,YAAY,CAAC,CAAC;IACnE,CAAC;IAED,KAAK,CAAC,IAAI;QACN,IAAI,IAAI,CAAC,KAAK,EAAE,CAAC;YACb,aAAa,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;YAC1B,IAAI,CAAC,KAAK,GAAG,IAAI,CAAC;QACtB,CAAC;QACD,MAAM,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,CAAC;IAC3B,CAAC;IAED,KAAK,CAAC,KAAK;QACP,MAAM,IAAI,GAAG,IAAI,CAAC,EAAE,CAAC,KAAK,EAAE,CAAC;QAC7B,IAAI,KAAK,GAAG,CAAC,CAAC;QACd,MAAM,KAAK,GAAe,EAAC,OAAO,EAAE,CAAC,EAAE,UAAU,EAAE,CAAC,EAAE,SAAS,EAAE,CAAC,EAAE,MAAM,EAAE,CAAC,EAAE,KAAK,EAAE,CAAC,EAAC,CAAC;QAEzF,KAAK,MAAM,GAAG,IAAI,IAAI,EAAE,CAAC;YACrB,MAAM,KAAK,GAAG,MAAM,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC;YAChC,IAAI,GAAG,CAAC,MAAM,KAAK,SAAS;gBAAE,KAAK,CAAC,OAAO,GAAG,KAAK,CAAC;iBAC/C,IAAI,GAAG,CAAC,MAAM,KAAK,YAAY;gBAAE,KAAK,CAAC,UAAU,GAAG,KAAK,CAAC;iBAC1D,IAAI,GAAG,CAAC,MAAM,KAAK,WAAW;gBAAE,KAAK,CAAC,SAAS,GAAG,KAAK,CAAC;iBACxD,IAAI,GAAG,CAAC,MAAM,KAAK,QAAQ;gBAAE,KAAK,CAAC,MAAM,GAAG,KAAK,CAAC;YACvD,KAAK,IAAI,KAAK,CAAC;QACnB,CAAC;QAED,KAAK,CAAC,KAAK,GAAG,KAAK,CAAC;QACpB,OAAO,KAAK,CAAC;IACjB,CAAC;IAED,KAAK,CAAC,KAAK,CAAC,OAAqB;QAC7B,IAAI,CAAC,EAAE,CAAC,KAAK,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC;IACrC,CAAC;IAEO,KAAK,CAAC,OAAO,CACjB,IAAY,EACZ,IAAO,EACP,OAAmC,EACnC,QAAkB;QAElB,MAAM,EAAE,GAAG,UAAU,EAAE,CAAC;QACxB,MAAM,UAAU,GAAG,OAAO,EAAE,UAAU,IAAI,mBAAmB,CAAC;QAC9D,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,EAAE,GAAG,CAAC,OAAO,EAAE,KAAK,IAAI,CAAC,CAAC,CAAC;QAEjD,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC;YACZ,EAAE;YACF,IAAI;YACJ,IAAI,EAAE,QAAQ;YACd,OAAO,EAAE,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC;YAC7B,KAAK;YACL,UAAU;YACV,QAAQ,EAAE,OAAO,EAAE,QAAQ,IAAI,gBAAgB;SAClD,CAAC,CAAC;QAEH,OAAO;YACH,EAAE;YACF,QAAQ,EAAE,IAAI;YACd,IAAI;YACJ,QAAQ,EAAE,CAAC;YACX,UAAU;YACV,MAAM,EAAE,SAAS;SACpB,CAAC;IACN,CAAC;IAEO,IAAI;QACR,IAAI,CAAC,UAAU,EAAE,CAAC;QAClB,IAAI,CAAC,WAAW,EAAE,CAAC;IACvB,CAAC;IAEO,UAAU;QACd,IAAI,IAAI,CAAC,QAAQ,IAAI,IAAI,CAAC,WAAW;YAAE,OAAO;QAE9C,MAAM,OAAO,GAAG,IAAI,CAAC,EAAE,CAAC,SAAS,CAAC,IAAI,CAAC,GAAG,EAAE,EAAE,IAAI,CAAC,UAAU,EAAE,IAAI,CAAC,CAAC;QACrE,IAAI,CAAC,OAAO;YAAE,OAAO;QAErB,IAAI,CAAC,QAAQ,EAAE,CAAC;QAChB,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,OAAO,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC;IACxD,CAAC;IAEO,WAAW;QACf,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,SAAS;YAAE,OAAO;QAEjC,MAAM,OAAO,GAAG,IAAI,CAAC,EAAE,CAAC,SAAS,CAAC,IAAI,CAAC,GAAG,EAAE,EAAE,IAAI,CAAC,UAAU,EAAE,QAAQ,CAAC,CAAC;QACzE,IAAI,CAAC,OAAO;YAAE,OAAO;QAErB,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;IACzB,CAAC;IAEO,KAAK,CAAC,MAAM,CAAC,OAAmB;QACpC,MAAM,OAAO,GAAG,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC;QAChD,IAAI,CAAC,OAAO,EAAE,CAAC;YACX,IAAI,CAAC,EAAE,CAAC,IAAI,CACR,OAAO,CAAC,EAAE,EACV,uCAAuC,OAAO,CAAC,IAAI,EAAE,EACrD,IAAI,CAAC,GAAG,EAAE,CACb,CAAC;YACF,OAAO;QACX,CAAC;QAED,MAAM,GAAG,GAAG,KAAK,CAAC,OAAO,EAAE,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC,CAAC;QAExD,IAAI,CAAC;YACD,MAAM,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,cAAc,CAAC,OAAO,EAAE,GAAG,CAAC,EAAE,IAAI,CAAC,UAAU,CAAC,CAAC;YAC3E,IAAI,CAAC,EAAE,CAAC,QAAQ,CAAC,OAAO,CAAC,EAAE,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;QAC7C,CAAC;QAAC,OAAO,GAAG,EAAE,CAAC;YACX,IAAI,CAAC,gBAAgB,CAAC,OAAO,EAAE,GAAG,CAAC,CAAC;QACxC,CAAC;IACL,CAAC;IAEO,cAAc,CAAC,OAA4B,EAAE,GAAQ;QACzD,IAAI,OAAO,OAAO,KAAK,UAAU,EAAE,CAAC;YAChC,OAAO,OAAO,CAAC,GAAG,CAAC,CAAC;QACxB,CAAC;QACD,OAAO,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,OAAO,EAAE,GAAG,CAAC,CAAC;IAC3C,CAAC;IAEO,gBAAgB,CAAC,OAAmB,EAAE,GAAY;QACtD,MAAM,QAAQ,GAAG,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;QAElE,IAAI,OAAO,CAAC,QAAQ,GAAG,OAAO,CAAC,UAAU,EAAE,CAAC;YACxC,MAAM,SAAS,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,OAAO,CAAC,QAAQ,CAAC,GAAG,IAAI,CAAC;YACvD,IAAI,CAAC,EAAE,CAAC,KAAK,CAAC,OAAO,CAAC,EAAE,EAAE,IAAI,CAAC,GAAG,EAAE,GAAG,SAAS,CAAC,CAAC;YAClD,OAAO;QACX,CAAC;QAED,IAAI,CAAC,EAAE,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE,EAAE,QAAQ,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;IACnD,CAAC;IAEO,WAAW,CAAI,OAAmB,EAAE,EAAU;QAClD,IAAI,SAAwC,CAAC;QAE7C,MAAM,cAAc,GAAG,IAAI,OAAO,CAAQ,CAAC,CAAC,EAAE,MAAM,EAAE,EAAE;YACpD,SAAS,GAAG,UAAU,CAClB,GAAG,EAAE,CAAC,MAAM,CAAC,IAAI,KAAK,CAAC,uBAAuB,EAAE,IAAI,CAAC,CAAC,EACtD,EAAE,CACL,CAAC;QACN,CAAC,CAAC,CAAC;QAEH,OAAO,OAAO,CAAC,IAAI,CAAC,CAAC,OAAO,EAAE,cAAc,CAAC,CAAC,CAAC,OAAO,CAAC,GAAG,EAAE;YACxD,YAAY,CAAC,SAAS,CAAC,CAAC;QAC5B,CAAC,CAAC,CAAC;IACP,CAAC;CACJ;AAED,SAAS,KAAK,CAAC,OAAmB,EAAE,IAAa;IAC7C,OAAO;QACH,EAAE,EAAE,OAAO,CAAC,EAAE;QACd,QAAQ,EAAE,OAAO,CAAC,IAAI;QACtB,IAAI;QACJ,QAAQ,EAAE,OAAO,CAAC,QAAQ;QAC1B,UAAU,EAAE,OAAO,CAAC,UAAU;QAC9B,MAAM,EAAE,YAAY;KACvB,CAAC;AACN,CAAC;AAED,SAAS,iBAAiB,CAAC,SAAkB;IACzC,MAAM,UAAU,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,SAAS,GAAG,CAAC,CAAC,CAAC;IAE9C,IAAI,SAAS,KAAK,SAAS,EAAE,CAAC;QAC1B,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,UAAU,GAAG,CAAC,CAAC,CAAC,CAAC;IACnD,CAAC;IAED,IAAI,SAAS,GAAG,SAAS,EAAE,CAAC;QACxB,MAAM,IAAI,KAAK,CACX,yBAAyB,SAAS,6BAA6B,SAAS,WAAW,CACtF,CAAC;IACN,CAAC;IAED,IAAI,SAAS,GAAG,UAAU,EAAE,CAAC;QACzB,OAAO,CAAC,IAAI,CACR,+BAA+B,SAAS,kDAAkD,UAAU,GAAG,CAC1G,CAAC;IACN,CAAC;IAED,OAAO,SAAS,CAAC;AACrB,CAAC"}
@@ -0,0 +1,49 @@
1
+ export type JobStatus = 'pending' | 'processing' | 'completed' | 'failed';
2
+ export interface BackoffConfig {
3
+ type: 'exponential' | 'fixed';
4
+ delay: number;
5
+ }
6
+ export interface EnqueueOptions {
7
+ delay?: number;
8
+ maxRetries?: number;
9
+ priority?: number;
10
+ backoff?: BackoffConfig;
11
+ }
12
+ export interface Job<T = unknown> {
13
+ id: string;
14
+ taskType: string;
15
+ data: T;
16
+ attempts: number;
17
+ maxRetries: number;
18
+ status: JobStatus;
19
+ result?: unknown;
20
+ errorLog?: string;
21
+ }
22
+ export type Enqueuer<T = unknown> = (data: T, options?: EnqueueOptions) => Promise<Job<T>>;
23
+ export interface QueueStats {
24
+ pending: number;
25
+ processing: number;
26
+ completed: number;
27
+ failed: number;
28
+ total: number;
29
+ }
30
+ export interface PurgeOptions {
31
+ olderThan: number;
32
+ }
33
+ export interface LiteQOptions {
34
+ storagePath: string;
35
+ concurrency?: number;
36
+ pollInterval?: number;
37
+ jobTimeout?: number;
38
+ minWorkers?: number;
39
+ maxWorkers?: number;
40
+ }
41
+ export type JobHandler<T = unknown> = (job: Job<T>) => Promise<unknown>;
42
+ export interface LiteQEvents {
43
+ 'job:success': [job: Job];
44
+ 'job:failed': [job: Job, errorLog: string];
45
+ 'job:retry': [job: Job];
46
+ 'queue:start': [];
47
+ 'queue:stop': [];
48
+ }
49
+ //# sourceMappingURL=types.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../src/types.ts"],"names":[],"mappings":"AAAA,MAAM,MAAM,SAAS,GAAG,SAAS,GAAG,YAAY,GAAG,WAAW,GAAG,QAAQ,CAAC;AAE1E,MAAM,WAAW,aAAa;IAC1B,IAAI,EAAE,aAAa,GAAG,OAAO,CAAC;IAC9B,KAAK,EAAE,MAAM,CAAC;CACjB;AAED,MAAM,WAAW,cAAc;IAC3B,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,OAAO,CAAC,EAAE,aAAa,CAAC;CAC3B;AAED,MAAM,WAAW,GAAG,CAAC,CAAC,GAAG,OAAO;IAC5B,EAAE,EAAE,MAAM,CAAC;IACX,QAAQ,EAAE,MAAM,CAAC;IACjB,IAAI,EAAE,CAAC,CAAC;IACR,QAAQ,EAAE,MAAM,CAAC;IACjB,UAAU,EAAE,MAAM,CAAC;IACnB,MAAM,EAAE,SAAS,CAAC;IAClB,MAAM,CAAC,EAAE,OAAO,CAAC;IACjB,QAAQ,CAAC,EAAE,MAAM,CAAC;CACrB;AAED,MAAM,MAAM,QAAQ,CAAC,CAAC,GAAG,OAAO,IAAI,CAAC,IAAI,EAAE,CAAC,EAAE,OAAO,CAAC,EAAE,cAAc,KAAK,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC;AAE3F,MAAM,WAAW,UAAU;IACvB,OAAO,EAAE,MAAM,CAAC;IAChB,UAAU,EAAE,MAAM,CAAC;IACnB,SAAS,EAAE,MAAM,CAAC;IAClB,MAAM,EAAE,MAAM,CAAC;IACf,KAAK,EAAE,MAAM,CAAC;CACjB;AAED,MAAM,WAAW,YAAY;IACzB,SAAS,EAAE,MAAM,CAAC;CACrB;AAED,MAAM,WAAW,YAAY;IACzB,WAAW,EAAE,MAAM,CAAC;IACpB,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,UAAU,CAAC,EAAE,MAAM,CAAC;CACvB;AAED,MAAM,MAAM,UAAU,CAAC,CAAC,GAAG,OAAO,IAAI,CAAC,GAAG,EAAE,GAAG,CAAC,CAAC,CAAC,KAAK,OAAO,CAAC,OAAO,CAAC,CAAC;AAExE,MAAM,WAAW,WAAW;IACxB,aAAa,EAAE,CAAC,GAAG,EAAE,GAAG,CAAC,CAAC;IAC1B,YAAY,EAAE,CAAC,GAAG,EAAE,GAAG,EAAE,QAAQ,EAAE,MAAM,CAAC,CAAC;IAC3C,WAAW,EAAE,CAAC,GAAG,EAAE,GAAG,CAAC,CAAC;IACxB,aAAa,EAAE,EAAE,CAAC;IAClB,YAAY,EAAE,EAAE,CAAC;CACpB"}
package/dist/types.js ADDED
@@ -0,0 +1,2 @@
1
+ export {};
2
+ //# sourceMappingURL=types.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"types.js","sourceRoot":"","sources":["../src/types.ts"],"names":[],"mappings":""}
@@ -0,0 +1,22 @@
1
+ import type { Job } from './types.js';
2
+ export declare class WorkerPool {
3
+ private minWorkers;
4
+ private maxWorkers;
5
+ private workers;
6
+ private queue;
7
+ private stopping;
8
+ constructor(minWorkers: number, maxWorkers: number);
9
+ get canAccept(): boolean;
10
+ execute(handlerPath: string, job: Job): Promise<void>;
11
+ stop(): Promise<void>;
12
+ private findIdle;
13
+ private acquireWorker;
14
+ private spawn;
15
+ private dispatch;
16
+ private processQueue;
17
+ private trimIdle;
18
+ private remove;
19
+ private clearPending;
20
+ private rejectPending;
21
+ }
22
+ //# sourceMappingURL=worker-pool.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"worker-pool.d.ts","sourceRoot":"","sources":["../src/worker-pool.ts"],"names":[],"mappings":"AAEA,OAAO,KAAK,EAAC,GAAG,EAAC,MAAM,YAAY,CAAC;AA2BpC,qBAAa,UAAU;IAMf,OAAO,CAAC,UAAU;IAClB,OAAO,CAAC,UAAU;IANtB,OAAO,CAAC,OAAO,CAAoB;IACnC,OAAO,CAAC,KAAK,CAAmB;IAChC,OAAO,CAAC,QAAQ,CAAS;gBAGb,UAAU,EAAE,MAAM,EAClB,UAAU,EAAE,MAAM;IAG9B,IAAI,SAAS,IAAI,OAAO,CAEvB;IAED,OAAO,CAAC,WAAW,EAAE,MAAM,EAAE,GAAG,EAAE,GAAG,GAAG,OAAO,CAAC,IAAI,CAAC;IAU/C,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC;IAgB3B,OAAO,CAAC,QAAQ;IAIhB,OAAO,CAAC,aAAa;IAOrB,OAAO,CAAC,KAAK;IAqCb,OAAO,CAAC,QAAQ;IAShB,OAAO,CAAC,YAAY;IAoBpB,OAAO,CAAC,QAAQ;IAWhB,OAAO,CAAC,MAAM;IAKd,OAAO,CAAC,YAAY;IAKpB,OAAO,CAAC,aAAa;CAKxB"}
@@ -0,0 +1,135 @@
1
+ import { Worker } from 'node:worker_threads';
2
+ import { fileURLToPath } from 'node:url';
3
+ const GENERIC_WORKER = fileURLToPath(new URL('./generic-worker.js', import.meta.url));
4
+ const POOL_STOPPED = new Error('Worker pool stopped');
5
+ export class WorkerPool {
6
+ minWorkers;
7
+ maxWorkers;
8
+ workers = [];
9
+ queue = [];
10
+ stopping = false;
11
+ constructor(minWorkers, maxWorkers) {
12
+ this.minWorkers = minWorkers;
13
+ this.maxWorkers = maxWorkers;
14
+ }
15
+ get canAccept() {
16
+ return this.findIdle() !== undefined || this.workers.length < this.maxWorkers;
17
+ }
18
+ execute(handlerPath, job) {
19
+ const worker = this.acquireWorker();
20
+ if (!worker) {
21
+ return new Promise((resolve, reject) => {
22
+ this.queue.push({ handlerPath, job, resolve, reject });
23
+ });
24
+ }
25
+ return this.dispatch(worker, handlerPath, job);
26
+ }
27
+ async stop() {
28
+ this.stopping = true;
29
+ for (const entry of this.workers) {
30
+ this.rejectPending(entry, POOL_STOPPED);
31
+ }
32
+ for (const queued of this.queue) {
33
+ queued.reject(POOL_STOPPED);
34
+ }
35
+ this.queue = [];
36
+ const workers = this.workers.splice(0);
37
+ await Promise.all(workers.map((entry) => entry.worker.terminate()));
38
+ }
39
+ findIdle() {
40
+ return this.workers.find((worker) => !worker.busy);
41
+ }
42
+ acquireWorker() {
43
+ const idle = this.findIdle();
44
+ if (idle)
45
+ return idle;
46
+ if (this.workers.length < this.maxWorkers)
47
+ return this.spawn();
48
+ return undefined;
49
+ }
50
+ spawn() {
51
+ const worker = new Worker(GENERIC_WORKER);
52
+ const entry = { worker, busy: false, resolve: null, reject: null };
53
+ worker.on('message', (msg) => {
54
+ if (entry.reject) {
55
+ if (msg.status === 'error') {
56
+ entry.reject(new Error(msg.error ?? 'Unknown worker error'));
57
+ }
58
+ else {
59
+ entry.resolve();
60
+ }
61
+ this.clearPending(entry);
62
+ }
63
+ entry.busy = false;
64
+ this.processQueue();
65
+ });
66
+ worker.on('error', (err) => {
67
+ this.rejectPending(entry, err);
68
+ entry.busy = false;
69
+ this.remove(entry);
70
+ this.processQueue();
71
+ });
72
+ worker.on('exit', (code) => {
73
+ if (code !== 0) {
74
+ this.rejectPending(entry, new Error(`Worker exited with code ${code}`));
75
+ }
76
+ entry.busy = false;
77
+ this.remove(entry);
78
+ this.processQueue();
79
+ });
80
+ this.workers.push(entry);
81
+ return entry;
82
+ }
83
+ dispatch(entry, handlerPath, job) {
84
+ entry.busy = true;
85
+ return new Promise((resolve, reject) => {
86
+ entry.resolve = resolve;
87
+ entry.reject = reject;
88
+ entry.worker.postMessage({ handlerPath, job });
89
+ });
90
+ }
91
+ processQueue() {
92
+ if (this.stopping)
93
+ return;
94
+ const remaining = [];
95
+ for (const queued of this.queue) {
96
+ const worker = this.acquireWorker();
97
+ if (worker) {
98
+ this.dispatch(worker, queued.handlerPath, queued.job)
99
+ .then(queued.resolve)
100
+ .catch(queued.reject);
101
+ }
102
+ else {
103
+ remaining.push(queued);
104
+ }
105
+ }
106
+ this.queue = remaining;
107
+ this.trimIdle();
108
+ }
109
+ trimIdle() {
110
+ const idle = this.workers.filter((worker) => !worker.busy);
111
+ const excess = idle.length - this.minWorkers;
112
+ if (excess <= 0)
113
+ return;
114
+ for (const entry of idle.slice(0, excess)) {
115
+ this.remove(entry);
116
+ entry.worker.terminate();
117
+ }
118
+ }
119
+ remove(entry) {
120
+ const index = this.workers.indexOf(entry);
121
+ if (index >= 0)
122
+ this.workers.splice(index, 1);
123
+ }
124
+ clearPending(entry) {
125
+ entry.resolve = null;
126
+ entry.reject = null;
127
+ }
128
+ rejectPending(entry, err) {
129
+ if (!entry.reject)
130
+ return;
131
+ entry.reject(err);
132
+ this.clearPending(entry);
133
+ }
134
+ }
135
+ //# sourceMappingURL=worker-pool.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"worker-pool.js","sourceRoot":"","sources":["../src/worker-pool.ts"],"names":[],"mappings":"AAAA,OAAO,EAAC,MAAM,EAAC,MAAM,qBAAqB,CAAC;AAC3C,OAAO,EAAC,aAAa,EAAC,MAAM,UAAU,CAAC;AAsBvC,MAAM,cAAc,GAAG,aAAa,CAChC,IAAI,GAAG,CAAC,qBAAqB,EAAE,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAClD,CAAC;AAEF,MAAM,YAAY,GAAG,IAAI,KAAK,CAAC,qBAAqB,CAAC,CAAC;AAEtD,MAAM,OAAO,UAAU;IAMP;IACA;IANJ,OAAO,GAAiB,EAAE,CAAC;IAC3B,KAAK,GAAgB,EAAE,CAAC;IACxB,QAAQ,GAAG,KAAK,CAAC;IAEzB,YACY,UAAkB,EAClB,UAAkB;QADlB,eAAU,GAAV,UAAU,CAAQ;QAClB,eAAU,GAAV,UAAU,CAAQ;IAC3B,CAAC;IAEJ,IAAI,SAAS;QACT,OAAO,IAAI,CAAC,QAAQ,EAAE,KAAK,SAAS,IAAI,IAAI,CAAC,OAAO,CAAC,MAAM,GAAG,IAAI,CAAC,UAAU,CAAC;IAClF,CAAC;IAED,OAAO,CAAC,WAAmB,EAAE,GAAQ;QACjC,MAAM,MAAM,GAAG,IAAI,CAAC,aAAa,EAAE,CAAC;QACpC,IAAI,CAAC,MAAM,EAAE,CAAC;YACV,OAAO,IAAI,OAAO,CAAC,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE;gBACnC,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,EAAC,WAAW,EAAE,GAAG,EAAE,OAAO,EAAE,MAAM,EAAC,CAAC,CAAC;YACzD,CAAC,CAAC,CAAC;QACP,CAAC;QACD,OAAO,IAAI,CAAC,QAAQ,CAAC,MAAM,EAAE,WAAW,EAAE,GAAG,CAAC,CAAC;IACnD,CAAC;IAED,KAAK,CAAC,IAAI;QACN,IAAI,CAAC,QAAQ,GAAG,IAAI,CAAC;QAErB,KAAK,MAAM,KAAK,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;YAC/B,IAAI,CAAC,aAAa,CAAC,KAAK,EAAE,YAAY,CAAC,CAAC;QAC5C,CAAC;QAED,KAAK,MAAM,MAAM,IAAI,IAAI,CAAC,KAAK,EAAE,CAAC;YAC9B,MAAM,CAAC,MAAM,CAAC,YAAY,CAAC,CAAC;QAChC,CAAC;QACD,IAAI,CAAC,KAAK,GAAG,EAAE,CAAC;QAEhB,MAAM,OAAO,GAAG,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC;QACvC,MAAM,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,KAAK,CAAC,MAAM,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC;IACxE,CAAC;IAEO,QAAQ;QACZ,OAAO,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC;IACvD,CAAC;IAEO,aAAa;QACjB,MAAM,IAAI,GAAG,IAAI,CAAC,QAAQ,EAAE,CAAC;QAC7B,IAAI,IAAI;YAAE,OAAO,IAAI,CAAC;QACtB,IAAI,IAAI,CAAC,OAAO,CAAC,MAAM,GAAG,IAAI,CAAC,UAAU;YAAE,OAAO,IAAI,CAAC,KAAK,EAAE,CAAC;QAC/D,OAAO,SAAS,CAAC;IACrB,CAAC;IAEO,KAAK;QACT,MAAM,MAAM,GAAG,IAAI,MAAM,CAAC,cAAc,CAAC,CAAC;QAC1C,MAAM,KAAK,GAAe,EAAC,MAAM,EAAE,IAAI,EAAE,KAAK,EAAE,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,IAAI,EAAC,CAAC;QAE7E,MAAM,CAAC,EAAE,CAAC,SAAS,EAAE,CAAC,GAAkB,EAAE,EAAE;YACxC,IAAI,KAAK,CAAC,MAAM,EAAE,CAAC;gBACf,IAAI,GAAG,CAAC,MAAM,KAAK,OAAO,EAAE,CAAC;oBACzB,KAAK,CAAC,MAAM,CAAC,IAAI,KAAK,CAAC,GAAG,CAAC,KAAK,IAAI,sBAAsB,CAAC,CAAC,CAAC;gBACjE,CAAC;qBAAM,CAAC;oBACJ,KAAK,CAAC,OAAQ,EAAE,CAAC;gBACrB,CAAC;gBACD,IAAI,CAAC,YAAY,CAAC,KAAK,CAAC,CAAC;YAC7B,CAAC;YACD,KAAK,CAAC,IAAI,GAAG,KAAK,CAAC;YACnB,IAAI,CAAC,YAAY,EAAE,CAAC;QACxB,CAAC,CAAC,CAAC;QAEH,MAAM,CAAC,EAAE,CAAC,OAAO,EAAE,CAAC,GAAG,EAAE,EAAE;YACvB,IAAI,CAAC,aAAa,CAAC,KAAK,EAAE,GAAG,CAAC,CAAC;YAC/B,KAAK,CAAC,IAAI,GAAG,KAAK,CAAC;YACnB,IAAI,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;YACnB,IAAI,CAAC,YAAY,EAAE,CAAC;QACxB,CAAC,CAAC,CAAC;QAEH,MAAM,CAAC,EAAE,CAAC,MAAM,EAAE,CAAC,IAAI,EAAE,EAAE;YACvB,IAAI,IAAI,KAAK,CAAC,EAAE,CAAC;gBACb,IAAI,CAAC,aAAa,CAAC,KAAK,EAAE,IAAI,KAAK,CAAC,2BAA2B,IAAI,EAAE,CAAC,CAAC,CAAC;YAC5E,CAAC;YACD,KAAK,CAAC,IAAI,GAAG,KAAK,CAAC;YACnB,IAAI,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;YACnB,IAAI,CAAC,YAAY,EAAE,CAAC;QACxB,CAAC,CAAC,CAAC;QAEH,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;QACzB,OAAO,KAAK,CAAC;IACjB,CAAC;IAEO,QAAQ,CAAC,KAAiB,EAAE,WAAmB,EAAE,GAAQ;QAC7D,KAAK,CAAC,IAAI,GAAG,IAAI,CAAC;QAClB,OAAO,IAAI,OAAO,CAAC,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE;YACnC,KAAK,CAAC,OAAO,GAAG,OAAO,CAAC;YACxB,KAAK,CAAC,MAAM,GAAG,MAAM,CAAC;YACtB,KAAK,CAAC,MAAM,CAAC,WAAW,CAAC,EAAC,WAAW,EAAE,GAAG,EAAC,CAAC,CAAC;QACjD,CAAC,CAAC,CAAC;IACP,CAAC;IAEO,YAAY;QAChB,IAAI,IAAI,CAAC,QAAQ;YAAE,OAAO;QAE1B,MAAM,SAAS,GAAgB,EAAE,CAAC;QAElC,KAAK,MAAM,MAAM,IAAI,IAAI,CAAC,KAAK,EAAE,CAAC;YAC9B,MAAM,MAAM,GAAG,IAAI,CAAC,aAAa,EAAE,CAAC;YACpC,IAAI,MAAM,EAAE,CAAC;gBACT,IAAI,CAAC,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC,WAAW,EAAE,MAAM,CAAC,GAAG,CAAC;qBAChD,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC;qBACpB,KAAK,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;YAC9B,CAAC;iBAAM,CAAC;gBACJ,SAAS,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;YAC3B,CAAC;QACL,CAAC;QAED,IAAI,CAAC,KAAK,GAAG,SAAS,CAAC;QACvB,IAAI,CAAC,QAAQ,EAAE,CAAC;IACpB,CAAC;IAEO,QAAQ;QACZ,MAAM,IAAI,GAAG,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC;QAC3D,MAAM,MAAM,GAAG,IAAI,CAAC,MAAM,GAAG,IAAI,CAAC,UAAU,CAAC;QAC7C,IAAI,MAAM,IAAI,CAAC;YAAE,OAAO;QAExB,KAAK,MAAM,KAAK,IAAI,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,MAAM,CAAC,EAAE,CAAC;YACxC,IAAI,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;YACnB,KAAK,CAAC,MAAM,CAAC,SAAS,EAAE,CAAC;QAC7B,CAAC;IACL,CAAC;IAEO,MAAM,CAAC,KAAiB;QAC5B,MAAM,KAAK,GAAG,IAAI,CAAC,OAAO,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC;QAC1C,IAAI,KAAK,IAAI,CAAC;YAAE,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC;IAClD,CAAC;IAEO,YAAY,CAAC,KAAiB;QAClC,KAAK,CAAC,OAAO,GAAG,IAAI,CAAC;QACrB,KAAK,CAAC,MAAM,GAAG,IAAI,CAAC;IACxB,CAAC;IAEO,aAAa,CAAC,KAAiB,EAAE,GAAY;QACjD,IAAI,CAAC,KAAK,CAAC,MAAM;YAAE,OAAO;QAC1B,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;QAClB,IAAI,CAAC,YAAY,CAAC,KAAK,CAAC,CAAC;IAC7B,CAAC;CACJ"}
package/package.json ADDED
@@ -0,0 +1,37 @@
1
+ {
2
+ "name": "@iikareem/litequeue",
3
+ "version": "1.0.1",
4
+ "description": "Zero-dependency, serverless, high-performance task queue engine powered by embedded SQLite in WAL mode",
5
+ "license": "MIT",
6
+ "type": "module",
7
+ "main": "./dist/index.js",
8
+ "types": "./dist/index.d.ts",
9
+ "files": [
10
+ "dist",
11
+ "README.md",
12
+ "LICENSE"
13
+ ],
14
+ "engines": {
15
+ "node": ">=18.0.0"
16
+ },
17
+ "scripts": {
18
+ "build": "tsc && cp src/generic-worker.js dist/generic-worker.js",
19
+ "prebuild": "rm -rf dist",
20
+ "typecheck": "tsc --noEmit",
21
+ "test": "node --test",
22
+ "manual": "tsx src/manual.ts",
23
+ "prepublishOnly": "npm run build"
24
+ },
25
+ "devDependencies": {
26
+ "@types/better-sqlite3": "^7.6.13",
27
+ "tsx": "^4.22.4",
28
+ "typescript": "^5.6.0",
29
+ "vitest": "^4.1.8"
30
+ },
31
+ "publishConfig": {
32
+ "access": "public"
33
+ },
34
+ "dependencies": {
35
+ "better-sqlite3": "^12.10.0"
36
+ }
37
+ }