@sidequest/engine 1.0.0-next.13 → 1.0.0-next.15
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +65 -0
- package/dist/engine.cjs +7 -7
- package/dist/engine.cjs.map +1 -1
- package/dist/engine.js +8 -8
- package/dist/engine.js.map +1 -1
- package/dist/execution/executor-manager.cjs +23 -19
- package/dist/execution/executor-manager.cjs.map +1 -1
- package/dist/execution/executor-manager.d.ts +4 -7
- package/dist/execution/executor-manager.js +24 -20
- package/dist/execution/executor-manager.js.map +1 -1
- package/dist/index.cjs +2 -0
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.ts +1 -0
- package/dist/index.js +1 -0
- package/dist/index.js.map +1 -1
- package/dist/job/job-builder.cjs +51 -9
- package/dist/job/job-builder.cjs.map +1 -1
- package/dist/job/job-builder.d.ts +21 -0
- package/dist/job/job-builder.js +52 -10
- package/dist/job/job-builder.js.map +1 -1
- package/dist/job/job-transitioner.cjs +1 -1
- package/dist/job/job-transitioner.cjs.map +1 -1
- package/dist/job/job-transitioner.js +1 -1
- package/dist/job/job-transitioner.js.map +1 -1
- package/dist/job/job.cjs +1 -1
- package/dist/job/job.cjs.map +1 -1
- package/dist/job/job.js +1 -1
- package/dist/job/job.js.map +1 -1
- package/dist/shared-runner/runner-pool.cjs +8 -6
- package/dist/shared-runner/runner-pool.cjs.map +1 -1
- package/dist/shared-runner/runner-pool.d.ts +4 -2
- package/dist/shared-runner/runner-pool.js +8 -6
- package/dist/shared-runner/runner-pool.js.map +1 -1
- package/dist/shared-runner/runner.cjs +30 -2
- package/dist/shared-runner/runner.cjs.map +1 -1
- package/dist/shared-runner/runner.d.ts +20 -3
- package/dist/shared-runner/runner.js +30 -3
- package/dist/shared-runner/runner.js.map +1 -1
- package/dist/utils/import.cjs +13 -0
- package/dist/utils/import.cjs.map +1 -0
- package/dist/utils/import.d.ts +15 -0
- package/dist/utils/import.js +11 -0
- package/dist/utils/import.js.map +1 -0
- package/dist/workers/main.cjs +4 -4
- package/dist/workers/main.cjs.map +1 -1
- package/dist/workers/main.js +4 -4
- package/dist/workers/main.js.map +1 -1
- package/package.json +6 -3
package/README.md
ADDED
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
# @sidequest/engine
|
|
2
|
+
|
|
3
|
+
Core job processing engine for the [Sidequest](https://github.com/sidequestjs/sidequest) job processing system.
|
|
4
|
+
|
|
5
|
+
## Summary
|
|
6
|
+
|
|
7
|
+
This package contains the core job processing engine that powers Sidequest.js. It manages job execution, worker threads, queue processing, background maintenance routines, and graceful shutdown handling. The engine is responsible for orchestrating all aspects of job processing including job claiming, execution, state transitions, error handling, and cleanup operations.
|
|
8
|
+
|
|
9
|
+
**⚠️ Important:** This package is not intended for direct use by end users. It is a core internal package that is used by the main `sidequest` package. Always use the main package instead:
|
|
10
|
+
|
|
11
|
+
```typescript
|
|
12
|
+
// ❌ Don't do this
|
|
13
|
+
import { Engine } from "@sidequest/engine";
|
|
14
|
+
|
|
15
|
+
// ✅ Do this instead
|
|
16
|
+
import { Sidequest } from "sidequest";
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
The engine package provides:
|
|
20
|
+
|
|
21
|
+
- **Job Processing Engine** - Core orchestration of job execution with worker thread management
|
|
22
|
+
- **Queue Management** - Queue configuration, priority handling, and concurrency control
|
|
23
|
+
- **Worker Thread Pool** - Scalable worker thread management using Piscina for job isolation
|
|
24
|
+
- **Background Routines** - Automated cleanup of finished jobs and stale job recovery
|
|
25
|
+
- **Graceful Shutdown** - Clean termination with job completion and resource cleanup
|
|
26
|
+
- **Job Lifecycle Management** - Complete job state transitions from waiting to completion
|
|
27
|
+
- **Error Handling** - Comprehensive error capture, retry logic, and failure management
|
|
28
|
+
- **Performance Optimization** - Efficient job claiming, batching, and resource utilization
|
|
29
|
+
- **Monitoring Integration** - Metrics and logging for job processing insights
|
|
30
|
+
|
|
31
|
+
The engine is designed to work seamlessly with all Sidequest backends (PostgreSQL, MySQL, SQLite, MongoDB) and provides the foundation for reliable, scalable job processing in production environments.
|
|
32
|
+
|
|
33
|
+
## Documentation
|
|
34
|
+
|
|
35
|
+
For complete configuration options, advanced usage, and architectural details, visit:
|
|
36
|
+
|
|
37
|
+
**[https://docs.sidequestjs.com/engine/](https://docs.sidequestjs.com/engine/)**
|
|
38
|
+
|
|
39
|
+
The documentation covers:
|
|
40
|
+
|
|
41
|
+
- **Engine Configuration** - Backend setup, queue configuration, and performance tuning
|
|
42
|
+
- **Starting the Engine** - Initialization, startup options, and lifecycle management
|
|
43
|
+
- **Queue Management** - Queue creation, priority settings, and concurrency control
|
|
44
|
+
- **Job Processing** - Worker thread management, execution flow, and error handling
|
|
45
|
+
- **Background Routines** - Cleanup operations, stale job recovery, and maintenance
|
|
46
|
+
- **Graceful Shutdown** - Clean termination procedures and resource management
|
|
47
|
+
- **Performance Tuning** - Thread pool sizing, concurrency limits, and optimization
|
|
48
|
+
- **Monitoring** - Logging, metrics, and health check integration
|
|
49
|
+
- **Troubleshooting** - Common issues, debugging techniques, and best practices
|
|
50
|
+
|
|
51
|
+
### Core Components
|
|
52
|
+
|
|
53
|
+
The engine consists of several key components:
|
|
54
|
+
|
|
55
|
+
- **`Engine`** - Main orchestrator class managing all job processing operations
|
|
56
|
+
- **`JobBuilder`** - Factory for creating and configuring job instances
|
|
57
|
+
- **`JobTransitioner`** - Handles job state transitions and lifecycle management
|
|
58
|
+
- **`QueueManager`** - Manages queue configurations and processing priorities
|
|
59
|
+
- **`WorkerPool`** - Thread pool management for isolated job execution
|
|
60
|
+
- **`RoutineManager`** - Background maintenance and cleanup operations
|
|
61
|
+
- **`SharedRunner`** - Coordination between main thread and worker threads
|
|
62
|
+
|
|
63
|
+
## License
|
|
64
|
+
|
|
65
|
+
LGPL-3.0-or-later
|
package/dist/engine.cjs
CHANGED
|
@@ -81,19 +81,14 @@ class Engine {
|
|
|
81
81
|
state: config?.queueDefaults?.state ?? backend.QUEUE_FALLBACK.state,
|
|
82
82
|
},
|
|
83
83
|
};
|
|
84
|
+
core.logger("Engine").debug(`Configuring Sidequest engine: ${JSON.stringify(this.config)}`);
|
|
84
85
|
if (this.config.logger) {
|
|
85
86
|
core.configureLogger(this.config.logger);
|
|
86
87
|
}
|
|
87
|
-
|
|
88
|
-
this.backend = await backend.createBackendFromDriver(this.config.backend);
|
|
88
|
+
this.backend = new backend.LazyBackend(this.config.backend);
|
|
89
89
|
if (!this.config.skipMigration) {
|
|
90
90
|
await this.backend.migrate();
|
|
91
91
|
}
|
|
92
|
-
if (this.config.queues) {
|
|
93
|
-
for (const queue of this.config.queues) {
|
|
94
|
-
await grantQueueConfig.grantQueueConfig(this.backend, queue, this.config.queueDefaults, true);
|
|
95
|
-
}
|
|
96
|
-
}
|
|
97
92
|
return this.config;
|
|
98
93
|
}
|
|
99
94
|
/**
|
|
@@ -103,6 +98,11 @@ class Engine {
|
|
|
103
98
|
async start(config) {
|
|
104
99
|
await this.configure(config);
|
|
105
100
|
core.logger("Engine").info(`Starting Sidequest using backend ${this.config.backend.driver}`);
|
|
101
|
+
if (this.config.queues) {
|
|
102
|
+
for (const queue of this.config.queues) {
|
|
103
|
+
await grantQueueConfig.grantQueueConfig(this.backend, queue, this.config.queueDefaults, true);
|
|
104
|
+
}
|
|
105
|
+
}
|
|
106
106
|
return new Promise((resolve, reject) => {
|
|
107
107
|
const timeout = setTimeout(() => {
|
|
108
108
|
reject(new Error("Timeout on starting sidequest fork!"));
|
package/dist/engine.cjs.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"engine.cjs","sources":["../src/engine.ts"],"sourcesContent":[null],"names":["logger","cpus","MISC_FALLBACK","JOB_BUILDER_FALLBACK","QUEUE_FALLBACK","configureLogger","
|
|
1
|
+
{"version":3,"file":"engine.cjs","sources":["../src/engine.ts"],"sourcesContent":[null],"names":["logger","cpus","MISC_FALLBACK","JOB_BUILDER_FALLBACK","QUEUE_FALLBACK","configureLogger","LazyBackend","grantQueueConfig","fork","gracefulShutdown","clearGracefulShutdown","JobBuilder"],"mappings":";;;;;;;;;;;;AAWA,MAAM,UAAU,GAAG,IAAI,CAAC,OAAO,CAAC,SAAmB,EAAE,SAAS,EAAE,SAAS,CAAC;AA8D1E;;AAEG;MACU,MAAM,CAAA;AACjB;;;AAGG;AACK,IAAA,OAAO;AAEf;;;;AAIG;AACK,IAAA,MAAM;AAEd;;;AAGG;AACK,IAAA,UAAU;AAElB;;;AAGG;IACK,YAAY,GAAG,KAAK;AAE5B;;;;AAIG;IACH,MAAM,SAAS,CAAC,MAAqB,EAAA;AACnC,QAAA,IAAI,IAAI,CAAC,MAAM,EAAE;YACfA,WAAM,CAAC,QAAQ,CAAC,CAAC,KAAK,CAAC,8BAA8B,CAAC;YACtD,OAAO,IAAI,CAAC,MAAM;QACpB;QACA,IAAI,CAAC,MAAM,GAAG;AACZ,YAAA,MAAM,EAAE,MAAM,EAAE,MAAM,IAAI,EAAE;AAC5B,YAAA,OAAO,EAAE;AACP,gBAAA,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,IAAI,2BAA2B;AAC9D,gBAAA,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,IAAI,oBAAoB;AACxD,aAAA;AACD,YAAA,8BAA8B,EAAE,MAAM,EAAE,8BAA8B,IAAI,EAAE;AAC5E,YAAA,4BAA4B,EAAE,MAAM,EAAE,4BAA4B,IAAI,EAAE,GAAG,EAAE,GAAG,EAAE,GAAG,EAAE,GAAG,IAAI;AAC9F,YAAA,2BAA2B,EAAE,MAAM,EAAE,2BAA2B,IAAI,EAAE;AACtE,YAAA,iBAAiB,EAAE,MAAM,EAAE,iBAAiB,IAAI,EAAE;AAClD,YAAA,aAAa,EAAE,MAAM,EAAE,aAAa,IAAI,KAAK;AAC7C,YAAA,MAAM,EAAE;AACN,gBAAA,KAAK,EAAE,MAAM,EAAE,MAAM,EAAE,KAAK,IAAI,MAAM;AACtC,gBAAA,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,IAAI,IAAI,KAAK;AACpC,aAAA;AACD,YAAA,gBAAgB,EAAE,MAAM,EAAE,gBAAgB,IAAI,IAAI;YAClD,UAAU,EAAE,MAAM,EAAE,UAAU,IAAIC,OAAI,EAAE,CAAC,MAAM;YAC/C,UAAU,EAAE,MAAM,EAAE,UAAU,IAAIA,OAAI,EAAE,CAAC,MAAM,GAAG,CAAC;YACnD,0BAA0B,EAAE,MAAM,EAAE,0BAA0B,IAAIC,qBAAa,CAAC,UAAU;YAC1F,4BAA4B,EAAE,MAAM,EAAE,4BAA4B,IAAIA,qBAAa,CAAC,YAAY;AAChG,YAAA,WAAW,EAAE;gBACX,KAAK,EAAE,MAAM,EAAE,WAAW,EAAE,KAAK,IAAIC,8BAAoB,CAAC,KAAM;gBAChE,WAAW,EAAE,MAAM,EAAE,WAAW,EAAE,WAAW,IAAIA,8BAAoB,CAAC,WAAY;;;AAGlF,gBAAA,WAAW,EAAE,MAAM,EAAE,WAAW,EAAE,WAAW;gBAC7C,OAAO,EAAE,MAAM,EAAE,WAAW,EAAE,OAAO,IAAIA,8BAAoB,CAAC,OAAQ;gBACtE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,UAAU,IAAIA,8BAAoB,CAAC,UAAW;AAChF,aAAA;AACD,YAAA,aAAa,EAAE;gBACb,WAAW,EAAE,MAAM,EAAE,aAAa,EAAE,WAAW,IAAIC,sBAAc,CAAC,WAAW;gBAC7E,QAAQ,EAAE,MAAM,EAAE,aAAa,EAAE,QAAQ,IAAIA,sBAAc,CAAC,QAAQ;gBACpE,KAAK,EAAE,MAAM,EAAE,aAAa,EAAE,KAAK,IAAIA,sBAAc,CAAC,KAAK;AAC5D,aAAA;SACF;AAED,QAAAJ,WAAM,CAAC,QAAQ,CAAC,CAAC,KAAK,CAAC,CAAA,8BAAA,EAAiC,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,MAAM,CAAC,CAAA,CAAE,CAAC;AAEtF,QAAA,IAAI,IAAI,CAAC,MAAM,CAAC,MAAM,EAAE;AACtB,YAAAK,oBAAe,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC;QACrC;AAEA,QAAA,IAAI,CAAC,OAAO,GAAG,IAAIC,mBAAW,CAAC,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC;AACnD,QAAA,IAAI,CAAC,IAAI,CAAC,MAAM,CAAC,aAAa,EAAE;AAC9B,YAAA,MAAM,IAAI,CAAC,OAAO,CAAC,OAAO,EAAE;QAC9B;QAEA,OAAO,IAAI,CAAC,MAAM;IACpB;AAEA;;;AAGG;IACH,MAAM,KAAK,CAAC,MAAoB,EAAA;AAC9B,QAAA,MAAM,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC;AAE5B,QAAAN,WAAM,CAAC,QAAQ,CAAC,CAAC,IAAI,CAAC,CAAA,iCAAA,EAAoC,IAAI,CAAC,MAAO,CAAC,OAAO,CAAC,MAAM,CAAA,CAAE,CAAC;AAExF,QAAA,IAAI,IAAI,CAAC,MAAO,CAAC,MAAM,EAAE;YACvB,KAAK,MAAM,KAAK,IAAI,IAAI,CAAC,MAAO,CAAC,MAAM,EAAE;AACvC,gBAAA,MAAMO,iCAAgB,CAAC,IAAI,CAAC,OAAQ,EAAE,KAAK,EAAE,IAAI,CAAC,MAAO,CAAC,aAAa,EAAE,IAAI,CAAC;YAChF;QACF;QAEA,OAAO,IAAI,OAAO,CAAC,CAAC,OAAO,EAAE,MAAM,KAAI;AACrC,YAAA,MAAM,OAAO,GAAG,UAAU,CAAC,MAAK;AAC9B,gBAAA,MAAM,CAAC,IAAI,KAAK,CAAC,qCAAqC,CAAC,CAAC;YAC1D,CAAC,EAAE,IAAI,CAAC;AAER,YAAA,IAAI,CAAC,IAAI,CAAC,UAAU,EAAE;gBACpB,MAAM,SAAS,GAAG,MAAK;oBACrBP,WAAM,CAAC,QAAQ,CAAC,CAAC,KAAK,CAAC,yBAAyB,CAAC;AACjD,oBAAA,IAAI,CAAC,UAAU,GAAGQ,kBAAI,CAAC,UAAU,CAAC;AAClC,oBAAAR,WAAM,CAAC,QAAQ,CAAC,CAAC,KAAK,CAAC,CAAA,YAAA,EAAe,IAAI,CAAC,UAAU,CAAC,GAAG,CAAA,CAAE,CAAC;oBAC5D,IAAI,CAAC,UAAU,CAAC,EAAE,CAAC,SAAS,EAAE,CAAC,GAAG,KAAI;AACpC,wBAAA,IAAI,GAAG,KAAK,OAAO,EAAE;4BACnBA,WAAM,CAAC,QAAQ,CAAC,CAAC,KAAK,CAAC,sBAAsB,CAAC;AAC9C,4BAAA,IAAI,CAAC,UAAU,EAAE,IAAI,CAAC,EAAE,IAAI,EAAE,OAAO,EAAE,eAAe,EAAE,IAAI,CAAC,MAAO,EAAE,CAAC;4BACvE,YAAY,CAAC,OAAO,CAAC;AACrB,4BAAA,OAAO,EAAE;wBACX;AACF,oBAAA,CAAC,CAAC;oBAEF,IAAI,CAAC,UAAU,CAAC,EAAE,CAAC,MAAM,EAAE,MAAK;AAC9B,wBAAA,IAAI,CAAC,IAAI,CAAC,YAAY,EAAE;4BACtBA,WAAM,CAAC,QAAQ,CAAC,CAAC,KAAK,CAAC,wCAAwC,CAAC;AAChE,4BAAA,SAAS,EAAE;wBACb;AACF,oBAAA,CAAC,CAAC;AACJ,gBAAA,CAAC;AAED,gBAAA,SAAS,EAAE;AACX,gBAAAS,yBAAgB,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,QAAQ,EAAE,IAAI,CAAC,MAAO,CAAC,gBAAgB,CAAC;YAClF;AACF,QAAA,CAAC,CAAC;IACJ;AAEA;;;AAGG;IACH,SAAS,GAAA;QACP,OAAO,IAAI,CAAC,MAAM;IACpB;AAEA;;;AAGG;IACH,UAAU,GAAA;QACR,OAAO,IAAI,CAAC,OAAO;IACrB;AAEA;;AAEG;AACH,IAAA,MAAM,KAAK,GAAA;AACT,QAAA,IAAI,CAAC,IAAI,CAAC,YAAY,EAAE;AACtB,YAAA,IAAI,CAAC,YAAY,GAAG,IAAI;YACxBT,WAAM,CAAC,QAAQ,CAAC,CAAC,KAAK,CAAC,6BAA6B,CAAC;AACrD,YAAA,IAAI,IAAI,CAAC,UAAU,EAAE;gBACnB,MAAM,OAAO,GAAG,IAAI,OAAO,CAAC,CAAC,OAAO,KAAI;oBACtC,IAAI,CAAC,UAAW,CAAC,EAAE,CAAC,MAAM,EAAE,OAAO,CAAC;AACtC,gBAAA,CAAC,CAAC;gBACF,IAAI,CAAC,UAAU,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,UAAU,EAAE,CAAC;AAC1C,gBAAA,MAAM,OAAO;YACf;AACA,YAAA,MAAM,IAAI,CAAC,OAAO,EAAE,KAAK,EAAE;AAC3B,YAAA,IAAI,CAAC,MAAM,GAAG,SAAS;AACvB,YAAA,IAAI,CAAC,OAAO,GAAG,SAAS;AACxB,YAAA,IAAI,CAAC,UAAU,GAAG,SAAS;;;AAG3B,YAAAU,8BAAqB,EAAE;YACvBV,WAAM,CAAC,QAAQ,CAAC,CAAC,KAAK,CAAC,0BAA0B,CAAC;AAClD,YAAA,IAAI,CAAC,YAAY,GAAG,KAAK;QAC3B;IACF;AAEA;;;;AAIG;AACH,IAAA,KAAK,CAAyB,QAAW,EAAA;QACvC,IAAI,CAAC,IAAI,CAAC,MAAM,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE;AACjC,YAAA,MAAM,IAAI,KAAK,CAAC,yEAAyE,CAAC;QAC5F;AACA,QAAA,IAAI,IAAI,CAAC,YAAY,EAAE;AACrB,YAAA,MAAM,IAAI,KAAK,CAAC,4CAA4C,CAAC;QAC/D;AACA,QAAAA,WAAM,CAAC,QAAQ,CAAC,CAAC,KAAK,CAAC,CAAA,wBAAA,EAA2B,QAAQ,CAAC,IAAI,CAAA,CAAE,CAAC;QAClE,OAAO,IAAIW,qBAAU,CAAC,IAAI,CAAC,OAAO,EAAE,QAAQ,EAAE;AAC5C,YAAA,GAAG,IAAI,CAAC,MAAM,CAAC,WAAW;;;YAG1B,WAAW,EAAE,IAAI,CAAC,MAAM,CAAC,WAAW,CAAC,WAAW,IAAIR,8BAAoB,CAAC,WAAY;AACtF,SAAA,CAAC;IACJ;AACD;;;;"}
|
package/dist/engine.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { QUEUE_FALLBACK, MISC_FALLBACK,
|
|
1
|
+
import { QUEUE_FALLBACK, MISC_FALLBACK, LazyBackend } from '@sidequest/backend';
|
|
2
2
|
import { logger, configureLogger } from '@sidequest/core';
|
|
3
3
|
import { fork } from 'child_process';
|
|
4
4
|
import { cpus } from 'os';
|
|
@@ -79,19 +79,14 @@ class Engine {
|
|
|
79
79
|
state: config?.queueDefaults?.state ?? QUEUE_FALLBACK.state,
|
|
80
80
|
},
|
|
81
81
|
};
|
|
82
|
+
logger("Engine").debug(`Configuring Sidequest engine: ${JSON.stringify(this.config)}`);
|
|
82
83
|
if (this.config.logger) {
|
|
83
84
|
configureLogger(this.config.logger);
|
|
84
85
|
}
|
|
85
|
-
|
|
86
|
-
this.backend = await createBackendFromDriver(this.config.backend);
|
|
86
|
+
this.backend = new LazyBackend(this.config.backend);
|
|
87
87
|
if (!this.config.skipMigration) {
|
|
88
88
|
await this.backend.migrate();
|
|
89
89
|
}
|
|
90
|
-
if (this.config.queues) {
|
|
91
|
-
for (const queue of this.config.queues) {
|
|
92
|
-
await grantQueueConfig(this.backend, queue, this.config.queueDefaults, true);
|
|
93
|
-
}
|
|
94
|
-
}
|
|
95
90
|
return this.config;
|
|
96
91
|
}
|
|
97
92
|
/**
|
|
@@ -101,6 +96,11 @@ class Engine {
|
|
|
101
96
|
async start(config) {
|
|
102
97
|
await this.configure(config);
|
|
103
98
|
logger("Engine").info(`Starting Sidequest using backend ${this.config.backend.driver}`);
|
|
99
|
+
if (this.config.queues) {
|
|
100
|
+
for (const queue of this.config.queues) {
|
|
101
|
+
await grantQueueConfig(this.backend, queue, this.config.queueDefaults, true);
|
|
102
|
+
}
|
|
103
|
+
}
|
|
104
104
|
return new Promise((resolve, reject) => {
|
|
105
105
|
const timeout = setTimeout(() => {
|
|
106
106
|
reject(new Error("Timeout on starting sidequest fork!"));
|
package/dist/engine.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"engine.js","sources":["../src/engine.ts"],"sourcesContent":[null],"names":[],"mappings":";;;;;;;;;;
|
|
1
|
+
{"version":3,"file":"engine.js","sources":["../src/engine.ts"],"sourcesContent":[null],"names":[],"mappings":";;;;;;;;;;AAWA,MAAM,UAAU,GAAG,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,OAAO,EAAE,SAAS,EAAE,SAAS,CAAC;AA8D1E;;AAEG;MACU,MAAM,CAAA;AACjB;;;AAGG;AACK,IAAA,OAAO;AAEf;;;;AAIG;AACK,IAAA,MAAM;AAEd;;;AAGG;AACK,IAAA,UAAU;AAElB;;;AAGG;IACK,YAAY,GAAG,KAAK;AAE5B;;;;AAIG;IACH,MAAM,SAAS,CAAC,MAAqB,EAAA;AACnC,QAAA,IAAI,IAAI,CAAC,MAAM,EAAE;YACf,MAAM,CAAC,QAAQ,CAAC,CAAC,KAAK,CAAC,8BAA8B,CAAC;YACtD,OAAO,IAAI,CAAC,MAAM;QACpB;QACA,IAAI,CAAC,MAAM,GAAG;AACZ,YAAA,MAAM,EAAE,MAAM,EAAE,MAAM,IAAI,EAAE;AAC5B,YAAA,OAAO,EAAE;AACP,gBAAA,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,IAAI,2BAA2B;AAC9D,gBAAA,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,IAAI,oBAAoB;AACxD,aAAA;AACD,YAAA,8BAA8B,EAAE,MAAM,EAAE,8BAA8B,IAAI,EAAE;AAC5E,YAAA,4BAA4B,EAAE,MAAM,EAAE,4BAA4B,IAAI,EAAE,GAAG,EAAE,GAAG,EAAE,GAAG,EAAE,GAAG,IAAI;AAC9F,YAAA,2BAA2B,EAAE,MAAM,EAAE,2BAA2B,IAAI,EAAE;AACtE,YAAA,iBAAiB,EAAE,MAAM,EAAE,iBAAiB,IAAI,EAAE;AAClD,YAAA,aAAa,EAAE,MAAM,EAAE,aAAa,IAAI,KAAK;AAC7C,YAAA,MAAM,EAAE;AACN,gBAAA,KAAK,EAAE,MAAM,EAAE,MAAM,EAAE,KAAK,IAAI,MAAM;AACtC,gBAAA,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,IAAI,IAAI,KAAK;AACpC,aAAA;AACD,YAAA,gBAAgB,EAAE,MAAM,EAAE,gBAAgB,IAAI,IAAI;YAClD,UAAU,EAAE,MAAM,EAAE,UAAU,IAAI,IAAI,EAAE,CAAC,MAAM;YAC/C,UAAU,EAAE,MAAM,EAAE,UAAU,IAAI,IAAI,EAAE,CAAC,MAAM,GAAG,CAAC;YACnD,0BAA0B,EAAE,MAAM,EAAE,0BAA0B,IAAI,aAAa,CAAC,UAAU;YAC1F,4BAA4B,EAAE,MAAM,EAAE,4BAA4B,IAAI,aAAa,CAAC,YAAY;AAChG,YAAA,WAAW,EAAE;gBACX,KAAK,EAAE,MAAM,EAAE,WAAW,EAAE,KAAK,IAAI,oBAAoB,CAAC,KAAM;gBAChE,WAAW,EAAE,MAAM,EAAE,WAAW,EAAE,WAAW,IAAI,oBAAoB,CAAC,WAAY;;;AAGlF,gBAAA,WAAW,EAAE,MAAM,EAAE,WAAW,EAAE,WAAW;gBAC7C,OAAO,EAAE,MAAM,EAAE,WAAW,EAAE,OAAO,IAAI,oBAAoB,CAAC,OAAQ;gBACtE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,UAAU,IAAI,oBAAoB,CAAC,UAAW;AAChF,aAAA;AACD,YAAA,aAAa,EAAE;gBACb,WAAW,EAAE,MAAM,EAAE,aAAa,EAAE,WAAW,IAAI,cAAc,CAAC,WAAW;gBAC7E,QAAQ,EAAE,MAAM,EAAE,aAAa,EAAE,QAAQ,IAAI,cAAc,CAAC,QAAQ;gBACpE,KAAK,EAAE,MAAM,EAAE,aAAa,EAAE,KAAK,IAAI,cAAc,CAAC,KAAK;AAC5D,aAAA;SACF;AAED,QAAA,MAAM,CAAC,QAAQ,CAAC,CAAC,KAAK,CAAC,CAAA,8BAAA,EAAiC,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,MAAM,CAAC,CAAA,CAAE,CAAC;AAEtF,QAAA,IAAI,IAAI,CAAC,MAAM,CAAC,MAAM,EAAE;AACtB,YAAA,eAAe,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC;QACrC;AAEA,QAAA,IAAI,CAAC,OAAO,GAAG,IAAI,WAAW,CAAC,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC;AACnD,QAAA,IAAI,CAAC,IAAI,CAAC,MAAM,CAAC,aAAa,EAAE;AAC9B,YAAA,MAAM,IAAI,CAAC,OAAO,CAAC,OAAO,EAAE;QAC9B;QAEA,OAAO,IAAI,CAAC,MAAM;IACpB;AAEA;;;AAGG;IACH,MAAM,KAAK,CAAC,MAAoB,EAAA;AAC9B,QAAA,MAAM,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC;AAE5B,QAAA,MAAM,CAAC,QAAQ,CAAC,CAAC,IAAI,CAAC,CAAA,iCAAA,EAAoC,IAAI,CAAC,MAAO,CAAC,OAAO,CAAC,MAAM,CAAA,CAAE,CAAC;AAExF,QAAA,IAAI,IAAI,CAAC,MAAO,CAAC,MAAM,EAAE;YACvB,KAAK,MAAM,KAAK,IAAI,IAAI,CAAC,MAAO,CAAC,MAAM,EAAE;AACvC,gBAAA,MAAM,gBAAgB,CAAC,IAAI,CAAC,OAAQ,EAAE,KAAK,EAAE,IAAI,CAAC,MAAO,CAAC,aAAa,EAAE,IAAI,CAAC;YAChF;QACF;QAEA,OAAO,IAAI,OAAO,CAAC,CAAC,OAAO,EAAE,MAAM,KAAI;AACrC,YAAA,MAAM,OAAO,GAAG,UAAU,CAAC,MAAK;AAC9B,gBAAA,MAAM,CAAC,IAAI,KAAK,CAAC,qCAAqC,CAAC,CAAC;YAC1D,CAAC,EAAE,IAAI,CAAC;AAER,YAAA,IAAI,CAAC,IAAI,CAAC,UAAU,EAAE;gBACpB,MAAM,SAAS,GAAG,MAAK;oBACrB,MAAM,CAAC,QAAQ,CAAC,CAAC,KAAK,CAAC,yBAAyB,CAAC;AACjD,oBAAA,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC,UAAU,CAAC;AAClC,oBAAA,MAAM,CAAC,QAAQ,CAAC,CAAC,KAAK,CAAC,CAAA,YAAA,EAAe,IAAI,CAAC,UAAU,CAAC,GAAG,CAAA,CAAE,CAAC;oBAC5D,IAAI,CAAC,UAAU,CAAC,EAAE,CAAC,SAAS,EAAE,CAAC,GAAG,KAAI;AACpC,wBAAA,IAAI,GAAG,KAAK,OAAO,EAAE;4BACnB,MAAM,CAAC,QAAQ,CAAC,CAAC,KAAK,CAAC,sBAAsB,CAAC;AAC9C,4BAAA,IAAI,CAAC,UAAU,EAAE,IAAI,CAAC,EAAE,IAAI,EAAE,OAAO,EAAE,eAAe,EAAE,IAAI,CAAC,MAAO,EAAE,CAAC;4BACvE,YAAY,CAAC,OAAO,CAAC;AACrB,4BAAA,OAAO,EAAE;wBACX;AACF,oBAAA,CAAC,CAAC;oBAEF,IAAI,CAAC,UAAU,CAAC,EAAE,CAAC,MAAM,EAAE,MAAK;AAC9B,wBAAA,IAAI,CAAC,IAAI,CAAC,YAAY,EAAE;4BACtB,MAAM,CAAC,QAAQ,CAAC,CAAC,KAAK,CAAC,wCAAwC,CAAC;AAChE,4BAAA,SAAS,EAAE;wBACb;AACF,oBAAA,CAAC,CAAC;AACJ,gBAAA,CAAC;AAED,gBAAA,SAAS,EAAE;AACX,gBAAA,gBAAgB,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,QAAQ,EAAE,IAAI,CAAC,MAAO,CAAC,gBAAgB,CAAC;YAClF;AACF,QAAA,CAAC,CAAC;IACJ;AAEA;;;AAGG;IACH,SAAS,GAAA;QACP,OAAO,IAAI,CAAC,MAAM;IACpB;AAEA;;;AAGG;IACH,UAAU,GAAA;QACR,OAAO,IAAI,CAAC,OAAO;IACrB;AAEA;;AAEG;AACH,IAAA,MAAM,KAAK,GAAA;AACT,QAAA,IAAI,CAAC,IAAI,CAAC,YAAY,EAAE;AACtB,YAAA,IAAI,CAAC,YAAY,GAAG,IAAI;YACxB,MAAM,CAAC,QAAQ,CAAC,CAAC,KAAK,CAAC,6BAA6B,CAAC;AACrD,YAAA,IAAI,IAAI,CAAC,UAAU,EAAE;gBACnB,MAAM,OAAO,GAAG,IAAI,OAAO,CAAC,CAAC,OAAO,KAAI;oBACtC,IAAI,CAAC,UAAW,CAAC,EAAE,CAAC,MAAM,EAAE,OAAO,CAAC;AACtC,gBAAA,CAAC,CAAC;gBACF,IAAI,CAAC,UAAU,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,UAAU,EAAE,CAAC;AAC1C,gBAAA,MAAM,OAAO;YACf;AACA,YAAA,MAAM,IAAI,CAAC,OAAO,EAAE,KAAK,EAAE;AAC3B,YAAA,IAAI,CAAC,MAAM,GAAG,SAAS;AACvB,YAAA,IAAI,CAAC,OAAO,GAAG,SAAS;AACxB,YAAA,IAAI,CAAC,UAAU,GAAG,SAAS;;;AAG3B,YAAA,qBAAqB,EAAE;YACvB,MAAM,CAAC,QAAQ,CAAC,CAAC,KAAK,CAAC,0BAA0B,CAAC;AAClD,YAAA,IAAI,CAAC,YAAY,GAAG,KAAK;QAC3B;IACF;AAEA;;;;AAIG;AACH,IAAA,KAAK,CAAyB,QAAW,EAAA;QACvC,IAAI,CAAC,IAAI,CAAC,MAAM,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE;AACjC,YAAA,MAAM,IAAI,KAAK,CAAC,yEAAyE,CAAC;QAC5F;AACA,QAAA,IAAI,IAAI,CAAC,YAAY,EAAE;AACrB,YAAA,MAAM,IAAI,KAAK,CAAC,4CAA4C,CAAC;QAC/D;AACA,QAAA,MAAM,CAAC,QAAQ,CAAC,CAAC,KAAK,CAAC,CAAA,wBAAA,EAA2B,QAAQ,CAAC,IAAI,CAAA,CAAE,CAAC;QAClE,OAAO,IAAI,UAAU,CAAC,IAAI,CAAC,OAAO,EAAE,QAAQ,EAAE;AAC5C,YAAA,GAAG,IAAI,CAAC,MAAM,CAAC,WAAW;;;YAG1B,WAAW,EAAE,IAAI,CAAC,MAAM,CAAC,WAAW,CAAC,WAAW,IAAI,oBAAoB,CAAC,WAAY;AACtF,SAAA,CAAC;IACJ;AACD;;;;"}
|
|
@@ -10,27 +10,21 @@ var runnerPool = require('../shared-runner/runner-pool.cjs');
|
|
|
10
10
|
*/
|
|
11
11
|
class ExecutorManager {
|
|
12
12
|
backend;
|
|
13
|
-
|
|
14
|
-
minThreads;
|
|
15
|
-
maxThreads;
|
|
13
|
+
nonNullConfig;
|
|
16
14
|
activeByQueue;
|
|
17
15
|
activeJobs;
|
|
18
16
|
runnerPool;
|
|
19
17
|
/**
|
|
20
18
|
* Creates a new ExecutorManager.
|
|
21
19
|
* @param backend The backend instance.
|
|
22
|
-
* @param
|
|
23
|
-
* @param minThreads Minimum number of worker threads to use.
|
|
24
|
-
* @param maxThreads Maximum number of worker threads to use.
|
|
20
|
+
* @param nonNullConfig The non-nullable engine configuration.
|
|
25
21
|
*/
|
|
26
|
-
constructor(backend,
|
|
22
|
+
constructor(backend, nonNullConfig) {
|
|
27
23
|
this.backend = backend;
|
|
28
|
-
this.
|
|
29
|
-
this.minThreads = minThreads;
|
|
30
|
-
this.maxThreads = maxThreads;
|
|
24
|
+
this.nonNullConfig = nonNullConfig;
|
|
31
25
|
this.activeByQueue = {};
|
|
32
26
|
this.activeJobs = new Set();
|
|
33
|
-
this.runnerPool = new runnerPool.RunnerPool(this.
|
|
27
|
+
this.runnerPool = new runnerPool.RunnerPool(this.nonNullConfig);
|
|
34
28
|
}
|
|
35
29
|
/**
|
|
36
30
|
* Gets the number of available slots for a given queue.
|
|
@@ -54,7 +48,7 @@ class ExecutorManager {
|
|
|
54
48
|
* @returns The number of available slots.
|
|
55
49
|
*/
|
|
56
50
|
availableSlotsGlobal() {
|
|
57
|
-
const limit = this.maxConcurrentJobs;
|
|
51
|
+
const limit = this.nonNullConfig.maxConcurrentJobs;
|
|
58
52
|
const availableSlots = limit - this.activeJobs.size;
|
|
59
53
|
if (availableSlots < 0) {
|
|
60
54
|
return 0;
|
|
@@ -88,10 +82,10 @@ class ExecutorManager {
|
|
|
88
82
|
job = await jobTransitioner.JobTransitioner.apply(this.backend, job, new core.RunTransition());
|
|
89
83
|
const signal = new EventEmitter();
|
|
90
84
|
let isRunning = true;
|
|
91
|
-
const
|
|
85
|
+
const cancellationCheck = async () => {
|
|
92
86
|
while (isRunning) {
|
|
93
87
|
const watchedJob = await this.backend.getJob(job.id);
|
|
94
|
-
if (watchedJob
|
|
88
|
+
if (watchedJob.state === "canceled") {
|
|
95
89
|
core.logger("Executor Manager").debug(`Emitting abort signal for job ${job.id}`);
|
|
96
90
|
signal.emit("abort");
|
|
97
91
|
isRunning = false;
|
|
@@ -100,10 +94,20 @@ class ExecutorManager {
|
|
|
100
94
|
await new Promise((r) => setTimeout(r, 1000));
|
|
101
95
|
}
|
|
102
96
|
};
|
|
103
|
-
void
|
|
97
|
+
void cancellationCheck();
|
|
104
98
|
try {
|
|
105
99
|
core.logger("Executor Manager").debug(`Running job ${job.id} in queue ${queueConfig.name}`);
|
|
106
|
-
const
|
|
100
|
+
const runPromise = this.runnerPool.run(job, signal);
|
|
101
|
+
if (job.timeout) {
|
|
102
|
+
void new Promise(() => {
|
|
103
|
+
setTimeout(() => {
|
|
104
|
+
core.logger("Executor Manager").debug(`Job ${job.id} timed out after ${job.timeout}ms, aborting.`);
|
|
105
|
+
signal.emit("abort");
|
|
106
|
+
void jobTransitioner.JobTransitioner.apply(this.backend, job, new core.RetryTransition(`Job timed out after ${job.timeout}ms`));
|
|
107
|
+
}, job.timeout);
|
|
108
|
+
});
|
|
109
|
+
}
|
|
110
|
+
const result = await runPromise;
|
|
107
111
|
isRunning = false;
|
|
108
112
|
core.logger("Executor Manager").debug(`Job ${job.id} completed with result: ${JSON.stringify(result)}`);
|
|
109
113
|
const transition = core.JobTransitionFactory.create(result);
|
|
@@ -113,11 +117,11 @@ class ExecutorManager {
|
|
|
113
117
|
isRunning = false;
|
|
114
118
|
const err = error;
|
|
115
119
|
if (err.message === "The task has been aborted") {
|
|
116
|
-
core.logger("Executor Manager").debug(`Job ${job.id} was
|
|
120
|
+
core.logger("Executor Manager").debug(`Job ${job.id} was aborted`);
|
|
117
121
|
}
|
|
118
122
|
else {
|
|
119
|
-
core.logger("Executor Manager").error(`
|
|
120
|
-
|
|
123
|
+
core.logger("Executor Manager").error(`Unhandled error while executing job ${job.id}: ${err.message}`);
|
|
124
|
+
await jobTransitioner.JobTransitioner.apply(this.backend, job, new core.RetryTransition(err));
|
|
121
125
|
}
|
|
122
126
|
}
|
|
123
127
|
finally {
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"executor-manager.cjs","sources":["../../src/execution/executor-manager.ts"],"sourcesContent":[null],"names":["RunnerPool","logger","JobTransitioner","SnoozeTransition","RunTransition","JobTransitionFactory"],"mappings":";;;;;;;
|
|
1
|
+
{"version":3,"file":"executor-manager.cjs","sources":["../../src/execution/executor-manager.ts"],"sourcesContent":[null],"names":["RunnerPool","logger","JobTransitioner","SnoozeTransition","RunTransition","RetryTransition","JobTransitionFactory"],"mappings":";;;;;;;AAeA;;AAEG;MACU,eAAe,CAAA;AAWhB,IAAA,OAAA;AACA,IAAA,aAAA;AAXF,IAAA,aAAa;AACb,IAAA,UAAU;AACV,IAAA,UAAU;AAElB;;;;AAIG;IACH,WAAA,CACU,OAAgB,EAChB,aAAsC,EAAA;QADtC,IAAA,CAAA,OAAO,GAAP,OAAO;QACP,IAAA,CAAA,aAAa,GAAb,aAAa;AAErB,QAAA,IAAI,CAAC,aAAa,GAAG,EAAE;AACvB,QAAA,IAAI,CAAC,UAAU,GAAG,IAAI,GAAG,EAAE;QAC3B,IAAI,CAAC,UAAU,GAAG,IAAIA,qBAAU,CAAC,IAAI,CAAC,aAAa,CAAC;IACtD;AAEA;;;;AAIG;AACH,IAAA,qBAAqB,CAAC,WAAwB,EAAA;QAC5C,IAAI,CAAC,IAAI,CAAC,aAAa,CAAC,WAAW,CAAC,IAAI,CAAC,EAAE;YACzC,IAAI,CAAC,aAAa,CAAC,WAAW,CAAC,IAAI,CAAC,GAAG,IAAI,GAAG,EAAE;QAClD;QACA,MAAM,UAAU,GAAG,IAAI,CAAC,aAAa,CAAC,WAAW,CAAC,IAAI,CAAC;AACvD,QAAA,MAAM,KAAK,GAAG,WAAW,CAAC,WAAW,IAAI,EAAE;AAE3C,QAAA,MAAM,cAAc,GAAG,KAAK,GAAG,UAAU,CAAC,IAAI;AAC9C,QAAA,IAAI,cAAc,GAAG,CAAC,EAAE;AACtB,YAAA,OAAO,CAAC;QACV;AACA,QAAA,OAAO,cAAc;IACvB;AAEA;;;AAGG;IACH,oBAAoB,GAAA;AAClB,QAAA,MAAM,KAAK,GAAG,IAAI,CAAC,aAAa,CAAC,iBAAiB;QAClD,MAAM,cAAc,GAAG,KAAK,GAAG,IAAI,CAAC,UAAU,CAAC,IAAI;AACnD,QAAA,IAAI,cAAc,GAAG,CAAC,EAAE;AACtB,YAAA,OAAO,CAAC;QACV;AACA,QAAA,OAAO,cAAc;IACvB;AAEA;;;AAGG;IACH,kBAAkB,GAAA;AAChB,QAAA,OAAO,IAAI,CAAC,UAAU,CAAC,IAAI;IAC7B;AAEA;;;;AAIG;AACH,IAAA,MAAM,OAAO,CAAC,WAAwB,EAAE,GAAY,EAAA;AAClD,QAAAC,WAAM,CAAC,kBAAkB,CAAC,CAAC,KAAK,CAAC,CAAA,eAAA,EAAkB,GAAG,CAAC,EAAE,2BAA2B,WAAW,CAAC,IAAI,CAAA,CAAE,CAAC;QACvG,IAAI,CAAC,IAAI,CAAC,aAAa,CAAC,WAAW,CAAC,IAAI,CAAC,EAAE;YACzC,IAAI,CAAC,aAAa,CAAC,WAAW,CAAC,IAAI,CAAC,GAAG,IAAI,GAAG,EAAE;QAClD;AAEA,QAAA,IAAI,IAAI,CAAC,qBAAqB,CAAC,WAAW,CAAC,IAAI,CAAC,IAAI,IAAI,CAAC,oBAAoB,EAAE,IAAI,CAAC,EAAE;AACpF,YAAAA,WAAM,CAAC,kBAAkB,CAAC,CAAC,KAAK,CAAC,CAAA,2BAAA,EAA8B,GAAG,CAAC,EAAE,aAAa,WAAW,CAAC,IAAI,CAAA,CAAE,CAAC;AACrG,YAAA,MAAMC,+BAAe,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,EAAE,GAAG,EAAE,IAAIC,qBAAgB,CAAC,CAAC,CAAC,CAAC;YACvE;QACF;AAEA,QAAA,IAAI,CAAC,aAAa,CAAC,WAAW,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC;QAChD,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC;AAE3B,QAAA,GAAG,GAAG,MAAMD,+BAAe,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,EAAE,GAAG,EAAE,IAAIE,kBAAa,EAAE,CAAC;AAEzE,QAAA,MAAM,MAAM,GAAG,IAAI,YAAY,EAAE;QACjC,IAAI,SAAS,GAAG,IAAI;AACpB,QAAA,MAAM,iBAAiB,GAAG,YAAW;YACnC,OAAO,SAAS,EAAE;AAChB,gBAAA,MAAM,UAAU,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,GAAG,CAAC,EAAE,CAAC;AACpD,gBAAA,IAAI,UAAW,CAAC,KAAK,KAAK,UAAU,EAAE;AACpC,oBAAAH,WAAM,CAAC,kBAAkB,CAAC,CAAC,KAAK,CAAC,CAAA,8BAAA,EAAiC,GAAG,CAAC,EAAE,CAAA,CAAE,CAAC;AAC3E,oBAAA,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC;oBACpB,SAAS,GAAG,KAAK;oBACjB;gBACF;AACA,gBAAA,MAAM,IAAI,OAAO,CAAC,CAAC,CAAC,KAAK,UAAU,CAAC,CAAC,EAAE,IAAI,CAAC,CAAC;YAC/C;AACF,QAAA,CAAC;QACD,KAAK,iBAAiB,EAAE;AAExB,QAAA,IAAI;AACF,YAAAA,WAAM,CAAC,kBAAkB,CAAC,CAAC,KAAK,CAAC,CAAA,YAAA,EAAe,GAAG,CAAC,EAAE,aAAa,WAAW,CAAC,IAAI,CAAA,CAAE,CAAC;AAEtF,YAAA,MAAM,UAAU,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,GAAG,EAAE,MAAM,CAAC;AAEnD,YAAA,IAAI,GAAG,CAAC,OAAO,EAAE;AACf,gBAAA,KAAK,IAAI,OAAO,CAAC,MAAK;oBACpB,UAAU,CAAC,MAAK;AACd,wBAAAA,WAAM,CAAC,kBAAkB,CAAC,CAAC,KAAK,CAAC,CAAA,IAAA,EAAO,GAAG,CAAC,EAAE,oBAAoB,GAAG,CAAC,OAAO,CAAA,aAAA,CAAe,CAAC;AAC7F,wBAAA,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC;wBACpB,KAAKC,+BAAe,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,EAAE,GAAG,EAAE,IAAIG,oBAAe,CAAC,uBAAuB,GAAG,CAAC,OAAO,CAAA,EAAA,CAAI,CAAC,CAAC;AAC5G,oBAAA,CAAC,EAAE,GAAG,CAAC,OAAQ,CAAC;AAClB,gBAAA,CAAC,CAAC;YACJ;AAEA,YAAA,MAAM,MAAM,GAAG,MAAM,UAAU;YAE/B,SAAS,GAAG,KAAK;AACjB,YAAAJ,WAAM,CAAC,kBAAkB,CAAC,CAAC,KAAK,CAAC,OAAO,GAAG,CAAC,EAAE,CAAA,wBAAA,EAA2B,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,CAAA,CAAE,CAAC;YAClG,MAAM,UAAU,GAAGK,yBAAoB,CAAC,MAAM,CAAC,MAAM,CAAC;AACtD,YAAA,MAAMJ,+BAAe,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,EAAE,GAAG,EAAE,UAAU,CAAC;QAC5D;QAAE,OAAO,KAAc,EAAE;YACvB,SAAS,GAAG,KAAK;YACjB,MAAM,GAAG,GAAG,KAAc;AAC1B,YAAA,IAAI,GAAG,CAAC,OAAO,KAAK,2BAA2B,EAAE;AAC/C,gBAAAD,WAAM,CAAC,kBAAkB,CAAC,CAAC,KAAK,CAAC,CAAA,IAAA,EAAO,GAAG,CAAC,EAAE,CAAA,YAAA,CAAc,CAAC;YAC/D;iBAAO;AACL,gBAAAA,WAAM,CAAC,kBAAkB,CAAC,CAAC,KAAK,CAAC,CAAA,oCAAA,EAAuC,GAAG,CAAC,EAAE,KAAK,GAAG,CAAC,OAAO,CAAA,CAAE,CAAC;AACjG,gBAAA,MAAMC,+BAAe,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,EAAE,GAAG,EAAE,IAAIG,oBAAe,CAAC,GAAG,CAAC,CAAC;YAC1E;QACF;gBAAU;YACR,SAAS,GAAG,KAAK;AACjB,YAAA,IAAI,CAAC,aAAa,CAAC,WAAW,CAAC,IAAI,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,EAAE,CAAC;YACnD,IAAI,CAAC,UAAU,CAAC,MAAM,CAAC,GAAG,CAAC,EAAE,CAAC;QAChC;IACF;AAEA;;AAEG;AACH,IAAA,MAAM,OAAO,GAAA;AACX,QAAA,MAAM,IAAI,OAAO,CAAO,CAAC,OAAO,KAAI;AAClC,YAAA,MAAM,SAAS,GAAG,YAAW;AAC3B,gBAAA,IAAI,IAAI,CAAC,kBAAkB,EAAE,KAAK,CAAC,EAAE;oBACnCJ,WAAM,CAAC,iBAAiB,CAAC,CAAC,IAAI,CAAC,mDAAmD,CAAC;AACnF,oBAAA,MAAM,IAAI,CAAC,UAAU,CAAC,OAAO,EAAE;AAC/B,oBAAA,OAAO,EAAE;gBACX;qBAAO;AACL,oBAAAA,WAAM,CAAC,iBAAiB,CAAC,CAAC,IAAI,CAAC,CAAA,YAAA,EAAe,IAAI,CAAC,kBAAkB,EAAE,CAAA,yBAAA,CAA2B,CAAC;oBACnG,UAAU,CAAC,MAAM,KAAK,SAAS,EAAE,EAAE,IAAI,CAAC;gBAC1C;AACF,YAAA,CAAC;YAED,KAAK,SAAS,EAAE;AAClB,QAAA,CAAC,CAAC;IACJ;AACD;;;;"}
|
|
@@ -1,25 +1,22 @@
|
|
|
1
1
|
import { Backend } from '@sidequest/backend';
|
|
2
2
|
import { QueueConfig, JobData } from '@sidequest/core';
|
|
3
|
+
import { NonNullableEngineConfig } from '../engine.js';
|
|
3
4
|
|
|
4
5
|
/**
|
|
5
6
|
* Manages job execution and worker concurrency for Sidequest.
|
|
6
7
|
*/
|
|
7
8
|
declare class ExecutorManager {
|
|
8
9
|
private backend;
|
|
9
|
-
private
|
|
10
|
-
private minThreads;
|
|
11
|
-
private maxThreads;
|
|
10
|
+
private nonNullConfig;
|
|
12
11
|
private activeByQueue;
|
|
13
12
|
private activeJobs;
|
|
14
13
|
private runnerPool;
|
|
15
14
|
/**
|
|
16
15
|
* Creates a new ExecutorManager.
|
|
17
16
|
* @param backend The backend instance.
|
|
18
|
-
* @param
|
|
19
|
-
* @param minThreads Minimum number of worker threads to use.
|
|
20
|
-
* @param maxThreads Maximum number of worker threads to use.
|
|
17
|
+
* @param nonNullConfig The non-nullable engine configuration.
|
|
21
18
|
*/
|
|
22
|
-
constructor(backend: Backend,
|
|
19
|
+
constructor(backend: Backend, nonNullConfig: NonNullableEngineConfig);
|
|
23
20
|
/**
|
|
24
21
|
* Gets the number of available slots for a given queue.
|
|
25
22
|
* @param queueConfig The queue configuration.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { logger, SnoozeTransition, RunTransition, JobTransitionFactory } from '@sidequest/core';
|
|
1
|
+
import { logger, SnoozeTransition, RunTransition, RetryTransition, JobTransitionFactory } from '@sidequest/core';
|
|
2
2
|
import EventEmitter from 'events';
|
|
3
3
|
import { JobTransitioner } from '../job/job-transitioner.js';
|
|
4
4
|
import { RunnerPool } from '../shared-runner/runner-pool.js';
|
|
@@ -8,27 +8,21 @@ import { RunnerPool } from '../shared-runner/runner-pool.js';
|
|
|
8
8
|
*/
|
|
9
9
|
class ExecutorManager {
|
|
10
10
|
backend;
|
|
11
|
-
|
|
12
|
-
minThreads;
|
|
13
|
-
maxThreads;
|
|
11
|
+
nonNullConfig;
|
|
14
12
|
activeByQueue;
|
|
15
13
|
activeJobs;
|
|
16
14
|
runnerPool;
|
|
17
15
|
/**
|
|
18
16
|
* Creates a new ExecutorManager.
|
|
19
17
|
* @param backend The backend instance.
|
|
20
|
-
* @param
|
|
21
|
-
* @param minThreads Minimum number of worker threads to use.
|
|
22
|
-
* @param maxThreads Maximum number of worker threads to use.
|
|
18
|
+
* @param nonNullConfig The non-nullable engine configuration.
|
|
23
19
|
*/
|
|
24
|
-
constructor(backend,
|
|
20
|
+
constructor(backend, nonNullConfig) {
|
|
25
21
|
this.backend = backend;
|
|
26
|
-
this.
|
|
27
|
-
this.minThreads = minThreads;
|
|
28
|
-
this.maxThreads = maxThreads;
|
|
22
|
+
this.nonNullConfig = nonNullConfig;
|
|
29
23
|
this.activeByQueue = {};
|
|
30
24
|
this.activeJobs = new Set();
|
|
31
|
-
this.runnerPool = new RunnerPool(this.
|
|
25
|
+
this.runnerPool = new RunnerPool(this.nonNullConfig);
|
|
32
26
|
}
|
|
33
27
|
/**
|
|
34
28
|
* Gets the number of available slots for a given queue.
|
|
@@ -52,7 +46,7 @@ class ExecutorManager {
|
|
|
52
46
|
* @returns The number of available slots.
|
|
53
47
|
*/
|
|
54
48
|
availableSlotsGlobal() {
|
|
55
|
-
const limit = this.maxConcurrentJobs;
|
|
49
|
+
const limit = this.nonNullConfig.maxConcurrentJobs;
|
|
56
50
|
const availableSlots = limit - this.activeJobs.size;
|
|
57
51
|
if (availableSlots < 0) {
|
|
58
52
|
return 0;
|
|
@@ -86,10 +80,10 @@ class ExecutorManager {
|
|
|
86
80
|
job = await JobTransitioner.apply(this.backend, job, new RunTransition());
|
|
87
81
|
const signal = new EventEmitter();
|
|
88
82
|
let isRunning = true;
|
|
89
|
-
const
|
|
83
|
+
const cancellationCheck = async () => {
|
|
90
84
|
while (isRunning) {
|
|
91
85
|
const watchedJob = await this.backend.getJob(job.id);
|
|
92
|
-
if (watchedJob
|
|
86
|
+
if (watchedJob.state === "canceled") {
|
|
93
87
|
logger("Executor Manager").debug(`Emitting abort signal for job ${job.id}`);
|
|
94
88
|
signal.emit("abort");
|
|
95
89
|
isRunning = false;
|
|
@@ -98,10 +92,20 @@ class ExecutorManager {
|
|
|
98
92
|
await new Promise((r) => setTimeout(r, 1000));
|
|
99
93
|
}
|
|
100
94
|
};
|
|
101
|
-
void
|
|
95
|
+
void cancellationCheck();
|
|
102
96
|
try {
|
|
103
97
|
logger("Executor Manager").debug(`Running job ${job.id} in queue ${queueConfig.name}`);
|
|
104
|
-
const
|
|
98
|
+
const runPromise = this.runnerPool.run(job, signal);
|
|
99
|
+
if (job.timeout) {
|
|
100
|
+
void new Promise(() => {
|
|
101
|
+
setTimeout(() => {
|
|
102
|
+
logger("Executor Manager").debug(`Job ${job.id} timed out after ${job.timeout}ms, aborting.`);
|
|
103
|
+
signal.emit("abort");
|
|
104
|
+
void JobTransitioner.apply(this.backend, job, new RetryTransition(`Job timed out after ${job.timeout}ms`));
|
|
105
|
+
}, job.timeout);
|
|
106
|
+
});
|
|
107
|
+
}
|
|
108
|
+
const result = await runPromise;
|
|
105
109
|
isRunning = false;
|
|
106
110
|
logger("Executor Manager").debug(`Job ${job.id} completed with result: ${JSON.stringify(result)}`);
|
|
107
111
|
const transition = JobTransitionFactory.create(result);
|
|
@@ -111,11 +115,11 @@ class ExecutorManager {
|
|
|
111
115
|
isRunning = false;
|
|
112
116
|
const err = error;
|
|
113
117
|
if (err.message === "The task has been aborted") {
|
|
114
|
-
logger("Executor Manager").debug(`Job ${job.id} was
|
|
118
|
+
logger("Executor Manager").debug(`Job ${job.id} was aborted`);
|
|
115
119
|
}
|
|
116
120
|
else {
|
|
117
|
-
logger("Executor Manager").error(`
|
|
118
|
-
|
|
121
|
+
logger("Executor Manager").error(`Unhandled error while executing job ${job.id}: ${err.message}`);
|
|
122
|
+
await JobTransitioner.apply(this.backend, job, new RetryTransition(err));
|
|
119
123
|
}
|
|
120
124
|
}
|
|
121
125
|
finally {
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"executor-manager.js","sources":["../../src/execution/executor-manager.ts"],"sourcesContent":[null],"names":[],"mappings":";;;;;
|
|
1
|
+
{"version":3,"file":"executor-manager.js","sources":["../../src/execution/executor-manager.ts"],"sourcesContent":[null],"names":[],"mappings":";;;;;AAeA;;AAEG;MACU,eAAe,CAAA;AAWhB,IAAA,OAAA;AACA,IAAA,aAAA;AAXF,IAAA,aAAa;AACb,IAAA,UAAU;AACV,IAAA,UAAU;AAElB;;;;AAIG;IACH,WAAA,CACU,OAAgB,EAChB,aAAsC,EAAA;QADtC,IAAA,CAAA,OAAO,GAAP,OAAO;QACP,IAAA,CAAA,aAAa,GAAb,aAAa;AAErB,QAAA,IAAI,CAAC,aAAa,GAAG,EAAE;AACvB,QAAA,IAAI,CAAC,UAAU,GAAG,IAAI,GAAG,EAAE;QAC3B,IAAI,CAAC,UAAU,GAAG,IAAI,UAAU,CAAC,IAAI,CAAC,aAAa,CAAC;IACtD;AAEA;;;;AAIG;AACH,IAAA,qBAAqB,CAAC,WAAwB,EAAA;QAC5C,IAAI,CAAC,IAAI,CAAC,aAAa,CAAC,WAAW,CAAC,IAAI,CAAC,EAAE;YACzC,IAAI,CAAC,aAAa,CAAC,WAAW,CAAC,IAAI,CAAC,GAAG,IAAI,GAAG,EAAE;QAClD;QACA,MAAM,UAAU,GAAG,IAAI,CAAC,aAAa,CAAC,WAAW,CAAC,IAAI,CAAC;AACvD,QAAA,MAAM,KAAK,GAAG,WAAW,CAAC,WAAW,IAAI,EAAE;AAE3C,QAAA,MAAM,cAAc,GAAG,KAAK,GAAG,UAAU,CAAC,IAAI;AAC9C,QAAA,IAAI,cAAc,GAAG,CAAC,EAAE;AACtB,YAAA,OAAO,CAAC;QACV;AACA,QAAA,OAAO,cAAc;IACvB;AAEA;;;AAGG;IACH,oBAAoB,GAAA;AAClB,QAAA,MAAM,KAAK,GAAG,IAAI,CAAC,aAAa,CAAC,iBAAiB;QAClD,MAAM,cAAc,GAAG,KAAK,GAAG,IAAI,CAAC,UAAU,CAAC,IAAI;AACnD,QAAA,IAAI,cAAc,GAAG,CAAC,EAAE;AACtB,YAAA,OAAO,CAAC;QACV;AACA,QAAA,OAAO,cAAc;IACvB;AAEA;;;AAGG;IACH,kBAAkB,GAAA;AAChB,QAAA,OAAO,IAAI,CAAC,UAAU,CAAC,IAAI;IAC7B;AAEA;;;;AAIG;AACH,IAAA,MAAM,OAAO,CAAC,WAAwB,EAAE,GAAY,EAAA;AAClD,QAAA,MAAM,CAAC,kBAAkB,CAAC,CAAC,KAAK,CAAC,CAAA,eAAA,EAAkB,GAAG,CAAC,EAAE,2BAA2B,WAAW,CAAC,IAAI,CAAA,CAAE,CAAC;QACvG,IAAI,CAAC,IAAI,CAAC,aAAa,CAAC,WAAW,CAAC,IAAI,CAAC,EAAE;YACzC,IAAI,CAAC,aAAa,CAAC,WAAW,CAAC,IAAI,CAAC,GAAG,IAAI,GAAG,EAAE;QAClD;AAEA,QAAA,IAAI,IAAI,CAAC,qBAAqB,CAAC,WAAW,CAAC,IAAI,CAAC,IAAI,IAAI,CAAC,oBAAoB,EAAE,IAAI,CAAC,EAAE;AACpF,YAAA,MAAM,CAAC,kBAAkB,CAAC,CAAC,KAAK,CAAC,CAAA,2BAAA,EAA8B,GAAG,CAAC,EAAE,aAAa,WAAW,CAAC,IAAI,CAAA,CAAE,CAAC;AACrG,YAAA,MAAM,eAAe,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,EAAE,GAAG,EAAE,IAAI,gBAAgB,CAAC,CAAC,CAAC,CAAC;YACvE;QACF;AAEA,QAAA,IAAI,CAAC,aAAa,CAAC,WAAW,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC;QAChD,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC;AAE3B,QAAA,GAAG,GAAG,MAAM,eAAe,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,EAAE,GAAG,EAAE,IAAI,aAAa,EAAE,CAAC;AAEzE,QAAA,MAAM,MAAM,GAAG,IAAI,YAAY,EAAE;QACjC,IAAI,SAAS,GAAG,IAAI;AACpB,QAAA,MAAM,iBAAiB,GAAG,YAAW;YACnC,OAAO,SAAS,EAAE;AAChB,gBAAA,MAAM,UAAU,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,GAAG,CAAC,EAAE,CAAC;AACpD,gBAAA,IAAI,UAAW,CAAC,KAAK,KAAK,UAAU,EAAE;AACpC,oBAAA,MAAM,CAAC,kBAAkB,CAAC,CAAC,KAAK,CAAC,CAAA,8BAAA,EAAiC,GAAG,CAAC,EAAE,CAAA,CAAE,CAAC;AAC3E,oBAAA,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC;oBACpB,SAAS,GAAG,KAAK;oBACjB;gBACF;AACA,gBAAA,MAAM,IAAI,OAAO,CAAC,CAAC,CAAC,KAAK,UAAU,CAAC,CAAC,EAAE,IAAI,CAAC,CAAC;YAC/C;AACF,QAAA,CAAC;QACD,KAAK,iBAAiB,EAAE;AAExB,QAAA,IAAI;AACF,YAAA,MAAM,CAAC,kBAAkB,CAAC,CAAC,KAAK,CAAC,CAAA,YAAA,EAAe,GAAG,CAAC,EAAE,aAAa,WAAW,CAAC,IAAI,CAAA,CAAE,CAAC;AAEtF,YAAA,MAAM,UAAU,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,GAAG,EAAE,MAAM,CAAC;AAEnD,YAAA,IAAI,GAAG,CAAC,OAAO,EAAE;AACf,gBAAA,KAAK,IAAI,OAAO,CAAC,MAAK;oBACpB,UAAU,CAAC,MAAK;AACd,wBAAA,MAAM,CAAC,kBAAkB,CAAC,CAAC,KAAK,CAAC,CAAA,IAAA,EAAO,GAAG,CAAC,EAAE,oBAAoB,GAAG,CAAC,OAAO,CAAA,aAAA,CAAe,CAAC;AAC7F,wBAAA,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC;wBACpB,KAAK,eAAe,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,EAAE,GAAG,EAAE,IAAI,eAAe,CAAC,uBAAuB,GAAG,CAAC,OAAO,CAAA,EAAA,CAAI,CAAC,CAAC;AAC5G,oBAAA,CAAC,EAAE,GAAG,CAAC,OAAQ,CAAC;AAClB,gBAAA,CAAC,CAAC;YACJ;AAEA,YAAA,MAAM,MAAM,GAAG,MAAM,UAAU;YAE/B,SAAS,GAAG,KAAK;AACjB,YAAA,MAAM,CAAC,kBAAkB,CAAC,CAAC,KAAK,CAAC,OAAO,GAAG,CAAC,EAAE,CAAA,wBAAA,EAA2B,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,CAAA,CAAE,CAAC;YAClG,MAAM,UAAU,GAAG,oBAAoB,CAAC,MAAM,CAAC,MAAM,CAAC;AACtD,YAAA,MAAM,eAAe,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,EAAE,GAAG,EAAE,UAAU,CAAC;QAC5D;QAAE,OAAO,KAAc,EAAE;YACvB,SAAS,GAAG,KAAK;YACjB,MAAM,GAAG,GAAG,KAAc;AAC1B,YAAA,IAAI,GAAG,CAAC,OAAO,KAAK,2BAA2B,EAAE;AAC/C,gBAAA,MAAM,CAAC,kBAAkB,CAAC,CAAC,KAAK,CAAC,CAAA,IAAA,EAAO,GAAG,CAAC,EAAE,CAAA,YAAA,CAAc,CAAC;YAC/D;iBAAO;AACL,gBAAA,MAAM,CAAC,kBAAkB,CAAC,CAAC,KAAK,CAAC,CAAA,oCAAA,EAAuC,GAAG,CAAC,EAAE,KAAK,GAAG,CAAC,OAAO,CAAA,CAAE,CAAC;AACjG,gBAAA,MAAM,eAAe,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,EAAE,GAAG,EAAE,IAAI,eAAe,CAAC,GAAG,CAAC,CAAC;YAC1E;QACF;gBAAU;YACR,SAAS,GAAG,KAAK;AACjB,YAAA,IAAI,CAAC,aAAa,CAAC,WAAW,CAAC,IAAI,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,EAAE,CAAC;YACnD,IAAI,CAAC,UAAU,CAAC,MAAM,CAAC,GAAG,CAAC,EAAE,CAAC;QAChC;IACF;AAEA;;AAEG;AACH,IAAA,MAAM,OAAO,GAAA;AACX,QAAA,MAAM,IAAI,OAAO,CAAO,CAAC,OAAO,KAAI;AAClC,YAAA,MAAM,SAAS,GAAG,YAAW;AAC3B,gBAAA,IAAI,IAAI,CAAC,kBAAkB,EAAE,KAAK,CAAC,EAAE;oBACnC,MAAM,CAAC,iBAAiB,CAAC,CAAC,IAAI,CAAC,mDAAmD,CAAC;AACnF,oBAAA,MAAM,IAAI,CAAC,UAAU,CAAC,OAAO,EAAE;AAC/B,oBAAA,OAAO,EAAE;gBACX;qBAAO;AACL,oBAAA,MAAM,CAAC,iBAAiB,CAAC,CAAC,IAAI,CAAC,CAAA,YAAA,EAAe,IAAI,CAAC,kBAAkB,EAAE,CAAA,yBAAA,CAA2B,CAAC;oBACnG,UAAU,CAAC,MAAM,KAAK,SAAS,EAAE,EAAE,IAAI,CAAC;gBAC1C;AACF,YAAA,CAAC;YAED,KAAK,SAAS,EAAE;AAClB,QAAA,CAAC,CAAC;IACJ;AACD;;;;"}
|
package/dist/index.cjs
CHANGED
|
@@ -13,6 +13,7 @@ var cleanupFinishedJob = require('./routines/cleanup-finished-job.cjs');
|
|
|
13
13
|
var releaseStaleJobs = require('./routines/release-stale-jobs.cjs');
|
|
14
14
|
var runner = require('./shared-runner/runner.cjs');
|
|
15
15
|
var runnerPool = require('./shared-runner/runner-pool.cjs');
|
|
16
|
+
var _import = require('./utils/import.cjs');
|
|
16
17
|
var shutdown = require('./utils/shutdown.cjs');
|
|
17
18
|
var main = require('./workers/main.cjs');
|
|
18
19
|
|
|
@@ -32,6 +33,7 @@ exports.cleanupFinishedJobs = cleanupFinishedJob.cleanupFinishedJobs;
|
|
|
32
33
|
exports.releaseStaleJobs = releaseStaleJobs.releaseStaleJobs;
|
|
33
34
|
exports.run = runner;
|
|
34
35
|
exports.RunnerPool = runnerPool.RunnerPool;
|
|
36
|
+
exports.importSidequest = _import.importSidequest;
|
|
35
37
|
exports.clearGracefulShutdown = shutdown.clearGracefulShutdown;
|
|
36
38
|
exports.gracefulShutdown = shutdown.gracefulShutdown;
|
|
37
39
|
exports.MainWorker = main.MainWorker;
|
package/dist/index.cjs.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.cjs","sources":[],"sourcesContent":[],"names":[],"mappings":"
|
|
1
|
+
{"version":3,"file":"index.cjs","sources":[],"sourcesContent":[],"names":[],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;"}
|
package/dist/index.d.ts
CHANGED
|
@@ -12,5 +12,6 @@ export { releaseStaleJobs } from './routines/release-stale-jobs.js';
|
|
|
12
12
|
import * as runner from './shared-runner/runner.js';
|
|
13
13
|
export { runner as run };
|
|
14
14
|
export { RunnerPool } from './shared-runner/runner-pool.js';
|
|
15
|
+
export { importSidequest } from './utils/import.js';
|
|
15
16
|
export { clearGracefulShutdown, gracefulShutdown } from './utils/shutdown.js';
|
|
16
17
|
export { MainWorker } from './workers/main.js';
|
package/dist/index.js
CHANGED
|
@@ -12,6 +12,7 @@ export { releaseStaleJobs } from './routines/release-stale-jobs.js';
|
|
|
12
12
|
import * as runner from './shared-runner/runner.js';
|
|
13
13
|
export { runner as run };
|
|
14
14
|
export { RunnerPool } from './shared-runner/runner-pool.js';
|
|
15
|
+
export { importSidequest } from './utils/import.js';
|
|
15
16
|
export { clearGracefulShutdown, gracefulShutdown } from './utils/shutdown.js';
|
|
16
17
|
export { MainWorker } from './workers/main.js';
|
|
17
18
|
//# sourceMappingURL=index.js.map
|
package/dist/index.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.js","sources":[],"sourcesContent":[],"names":[],"mappings":"
|
|
1
|
+
{"version":3,"file":"index.js","sources":[],"sourcesContent":[],"names":[],"mappings":";;;;;;;;;;;;;;;;"}
|
package/dist/job/job-builder.cjs
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
'use strict';
|
|
2
2
|
|
|
3
3
|
var core = require('@sidequest/core');
|
|
4
|
+
var nodeCron = require('node-cron');
|
|
4
5
|
var constants = require('./constants.cjs');
|
|
5
6
|
|
|
6
7
|
/**
|
|
@@ -114,12 +115,7 @@ class JobBuilder {
|
|
|
114
115
|
this.jobAvailableAt = value;
|
|
115
116
|
return this;
|
|
116
117
|
}
|
|
117
|
-
|
|
118
|
-
* Enqueues the job with the specified arguments.
|
|
119
|
-
* @param args Arguments to pass to the job's run method.
|
|
120
|
-
* @returns A promise resolving to the created job data.
|
|
121
|
-
*/
|
|
122
|
-
async enqueue(...args) {
|
|
118
|
+
async build(...args) {
|
|
123
119
|
const job = new this.JobClass(...this.constructorArgs);
|
|
124
120
|
await job.ready();
|
|
125
121
|
if (!job.script) {
|
|
@@ -138,15 +134,61 @@ class JobBuilder {
|
|
|
138
134
|
timeout: this.jobTimeout,
|
|
139
135
|
uniqueness_config: this.uniquenessConfig,
|
|
140
136
|
};
|
|
141
|
-
core.logger("JobBuilder").debug(`Enqueuing job ${job.className} with args: ${JSON.stringify(args)}
|
|
142
|
-
and constructor args: ${JSON.stringify(this.constructorArgs)}`);
|
|
143
137
|
if (this.uniquenessConfig) {
|
|
144
138
|
const uniqueness = core.UniquenessFactory.create(this.uniquenessConfig);
|
|
145
139
|
jobData.unique_digest = uniqueness.digest(jobData);
|
|
146
|
-
core.logger("JobBuilder").debug(`Job ${
|
|
140
|
+
core.logger("JobBuilder").debug(`Job ${jobData.class} uniqueness digest: ${jobData.unique_digest}`);
|
|
147
141
|
}
|
|
142
|
+
return jobData;
|
|
143
|
+
}
|
|
144
|
+
/**
|
|
145
|
+
* Enqueues the job with the specified arguments.
|
|
146
|
+
* @param args Arguments to pass to the job's run method.
|
|
147
|
+
* @returns A promise resolving to the created job data.
|
|
148
|
+
*/
|
|
149
|
+
async enqueue(...args) {
|
|
150
|
+
const jobData = await this.build(...args);
|
|
151
|
+
core.logger("JobBuilder").debug(`Enqueuing job ${jobData.class} with args: ${JSON.stringify(args)}
|
|
152
|
+
and constructor args: ${JSON.stringify(this.constructorArgs)}`);
|
|
148
153
|
return this.backend.createNewJob(jobData);
|
|
149
154
|
}
|
|
155
|
+
/**
|
|
156
|
+
* Registers a recurring schedule to enqueue the job automatically based on a cron expression.
|
|
157
|
+
*
|
|
158
|
+
* This sets up an in-memory schedule that enqueues the job with the provided arguments
|
|
159
|
+
* every time the cron expression is triggered.
|
|
160
|
+
*
|
|
161
|
+
* @remarks
|
|
162
|
+
* - The schedule is **not persisted** to any database. It will be lost if the process restarts and must be re-registered at startup.
|
|
163
|
+
* - You must call this method during application initialization to ensure the job is scheduled correctly.
|
|
164
|
+
* - Uses node-cron’s `noOverlap: true` option to prevent concurrent executions.
|
|
165
|
+
*
|
|
166
|
+
* @param cronExpression - A valid cron expression (node-cron compatible) that defines when the job should be enqueued.
|
|
167
|
+
* @param args - Arguments to be passed to the job’s `run` method on each scheduled execution.
|
|
168
|
+
*
|
|
169
|
+
* @returns The underlying `ScheduledTask` instance created by node-cron.
|
|
170
|
+
*
|
|
171
|
+
* @throws {Error} If the cron expression is invalid.
|
|
172
|
+
*/
|
|
173
|
+
async schedule(cronExpression, ...args) {
|
|
174
|
+
if (!nodeCron.validate(cronExpression)) {
|
|
175
|
+
throw new Error(`Invalid cron expression ${cronExpression}`);
|
|
176
|
+
}
|
|
177
|
+
// Build the job data using the provided arguments,
|
|
178
|
+
// this ensures the scheduled state is going to be respected in cases where the builder was reused.
|
|
179
|
+
// Includes class name, queue, timeout, uniqueness, etc.
|
|
180
|
+
const jobData = await this.build(...args);
|
|
181
|
+
// Freeze the job data to prevent future modifications.
|
|
182
|
+
// Ensures the same payload is used on every scheduled execution.
|
|
183
|
+
Object.freeze(jobData);
|
|
184
|
+
core.logger("JobBuilder").debug(`Scheduling job ${jobData.class} with cron: "${cronExpression}", args: ${JSON.stringify(args)}, ` +
|
|
185
|
+
`constructor args: ${JSON.stringify(this.constructorArgs)}`);
|
|
186
|
+
return nodeCron.schedule(cronExpression, async () => {
|
|
187
|
+
const newJobData = Object.assign({}, jobData);
|
|
188
|
+
core.logger("JobBuilder").debug(`Cron triggered for job ${newJobData.class} at ${newJobData.available_at.toISOString()} with args: ${JSON.stringify(args)}`);
|
|
189
|
+
return this.backend.createNewJob(jobData);
|
|
190
|
+
}, { noOverlap: true });
|
|
191
|
+
}
|
|
150
192
|
}
|
|
151
193
|
|
|
152
194
|
exports.JobBuilder = JobBuilder;
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"job-builder.cjs","sources":["../../src/job/job-builder.ts"],"sourcesContent":[null],"names":["JOB_BUILDER_FALLBACK","
|
|
1
|
+
{"version":3,"file":"job-builder.cjs","sources":["../../src/job/job-builder.ts"],"sourcesContent":[null],"names":["JOB_BUILDER_FALLBACK","UniquenessFactory","logger"],"mappings":";;;;;;AA0DA;;;AAGG;MACU,UAAU,CAAA;AAaX,IAAA,OAAA;AACA,IAAA,QAAA;AACA,IAAA,QAAA;AAdF,IAAA,eAAe;AACf,IAAA,SAAS;AACT,IAAA,UAAU;AACV,IAAA,gBAAgB;AAChB,IAAA,cAAc;AACd,IAAA,cAAc;AAEtB;;;AAGG;AACH,IAAA,WAAA,CACU,OAAgB,EAChB,QAAW,EACX,QAA6B,EAAA;QAF7B,IAAA,CAAA,OAAO,GAAP,OAAO;QACP,IAAA,CAAA,QAAQ,GAAR,QAAQ;QACR,IAAA,CAAA,QAAQ,GAAR,QAAQ;AAEhB,QAAA,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,QAAQ,EAAE,KAAK,IAAIA,8BAAoB,CAAC,KAAM,CAAC;AAC/D,QAAA,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,QAAQ,EAAE,WAAW,IAAIA,8BAAoB,CAAC,WAAY,CAAC;AACjF,QAAA,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,QAAQ,EAAE,WAAW,IAAIA,8BAAoB,CAAC,WAAY,CAAC;AACjF,QAAA,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,QAAQ,EAAE,OAAO,IAAIA,8BAAoB,CAAC,OAAQ,CAAC;AACrE,QAAA,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,QAAQ,EAAE,UAAU,IAAIA,8BAAoB,CAAC,UAAW,CAAC;QAC1E,IAAI,CAAC,IAAI,CAAC,GAAIA,8BAAoB,CAAC,eAAuD,CAAC;IAC7F;AAEA;;;;AAIG;IACH,IAAI,CAAC,GAAG,IAA8B,EAAA;AACpC,QAAA,IAAI,CAAC,eAAe,GAAG,IAAI;AAC3B,QAAA,OAAO,IAAI;IACb;AAEA;;;;AAIG;AACH,IAAA,KAAK,CAAC,KAAa,EAAA;AACjB,QAAA,IAAI,CAAC,SAAS,GAAG,KAAK;AACtB,QAAA,OAAO,IAAI;IACb;AAEA;;;;AAIG;AACH,IAAA,OAAO,CAAC,EAAU,EAAA;AAChB,QAAA,IAAI,CAAC,UAAU,GAAG,EAAE;AACpB,QAAA,OAAO,IAAI;IACb;AAEA;;;;;;;;;AASG;AACH,IAAA,MAAM,CAAC,KAAsB,EAAA;AAC3B,QAAA,IAAI,OAAO,KAAK,KAAK,SAAS,EAAE;YAC9B,IAAI,KAAK,EAAE;AACT,gBAAA,MAAM,MAAM,GAAmB;AAC7B,oBAAA,IAAI,EAAE,WAAW;AACjB,oBAAA,QAAQ,EAAE,KAAK;iBAChB;AACD,gBAAA,IAAI,CAAC,gBAAgB,GAAG,MAAM;YAChC;iBAAO;AACL,gBAAA,IAAI,CAAC,gBAAgB,GAAG,SAAS,CAAC;YACpC;QACF;aAAO;AACL,YAAA,IAAI,KAAK,CAAC,MAAM,EAAE;gBAChB,IAAI,CAAC,gBAAgB,GAAG;AACtB,oBAAA,IAAI,EAAE,cAAc;oBACpB,MAAM,EAAE,KAAK,CAAC,MAAM;oBACpB,QAAQ,EAAE,KAAK,CAAC,QAAQ;iBACJ;YACxB;iBAAO;AACL,gBAAA,IAAI,CAAC,gBAAgB,GAAG,EAAE,IAAI,EAAE,WAAW,EAAE,QAAQ,EAAE,KAAK,CAAC,QAAQ,EAAoB;YAC3F;QACF;AACA,QAAA,OAAO,IAAI;IACb;AAEA;;;;AAIG;AACH,IAAA,WAAW,CAAC,KAAa,EAAA;AACvB,QAAA,IAAI,CAAC,cAAc,GAAG,KAAK;AAC3B,QAAA,OAAO,IAAI;IACb;AAEA;;;;AAIG;AACH,IAAA,WAAW,CAAC,KAAW,EAAA;AACrB,QAAA,IAAI,CAAC,cAAc,GAAG,KAAK;AAC3B,QAAA,OAAO,IAAI;IACb;AAEQ,IAAA,MAAM,KAAK,CAAC,GAAG,IAAwC,EAAA;AAC7D,QAAA,MAAM,GAAG,GAAG,IAAI,IAAI,CAAC,QAAQ,CAAC,GAAG,IAAI,CAAC,eAAgB,CAAC;AAEvD,QAAA,MAAM,GAAG,CAAC,KAAK,EAAE;AAEjB,QAAA,IAAI,CAAC,GAAG,CAAC,MAAM,EAAE;YACf,MAAM,IAAI,KAAK,CAAC,CAAA,sBAAA,EAAyB,GAAG,CAAC,SAAS,CAAA,8BAAA,CAAgC,CAAC;QACzF;AAEA,QAAA,MAAM,OAAO,GAAe;YAC1B,KAAK,EAAE,IAAI,CAAC,SAAU;YACtB,MAAM,EAAE,GAAG,CAAC,MAAM;YAClB,KAAK,EAAE,GAAG,CAAC,SAAS;AACpB,YAAA,KAAK,EAAE,SAAS;YAChB,IAAI;YACJ,gBAAgB,EAAE,IAAI,CAAC,eAAgB;AACvC,YAAA,OAAO,EAAE,CAAC;YACV,YAAY,EAAE,IAAI,CAAC,cAAe;YAClC,YAAY,EAAE,IAAI,CAAC,cAAe;YAClC,OAAO,EAAE,IAAI,CAAC,UAAW;YACzB,iBAAiB,EAAE,IAAI,CAAC,gBAAiB;SAC1C;AAED,QAAA,IAAI,IAAI,CAAC,gBAAgB,EAAE;YACzB,MAAM,UAAU,GAAGC,sBAAiB,CAAC,MAAM,CAAC,IAAI,CAAC,gBAAgB,CAAC;YAClE,OAAO,CAAC,aAAa,GAAG,UAAU,CAAC,MAAM,CAAC,OAAkB,CAAC;AAC7D,YAAAC,WAAM,CAAC,YAAY,CAAC,CAAC,KAAK,CAAC,CAAA,IAAA,EAAO,OAAO,CAAC,KAAK,uBAAuB,OAAO,CAAC,aAAa,CAAA,CAAE,CAAC;QAChG;AAEA,QAAA,OAAO,OAAO;IAChB;AAEA;;;;AAIG;AACH,IAAA,MAAM,OAAO,CAAC,GAAG,IAAwC,EAAA;QACvD,MAAM,OAAO,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC;AAEzC,QAAAA,WAAM,CAAC,YAAY,CAAC,CAAC,KAAK,CACxB,CAAA,cAAA,EAAiB,OAAO,CAAC,KAAK,eAAe,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC;8BACzC,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,eAAe,CAAC,CAAA,CAAE,CAC/D;QACD,OAAO,IAAI,CAAC,OAAO,CAAC,YAAY,CAAC,OAAO,CAAC;IAC3C;AAEA;;;;;;;;;;;;;;;;;AAiBG;AACH,IAAA,MAAM,QAAQ,CAAC,cAAsB,EAAE,GAAG,IAAwC,EAAA;QAChF,IAAI,CAAC,QAAQ,CAAC,QAAQ,CAAC,cAAc,CAAC,EAAE;AACtC,YAAA,MAAM,IAAI,KAAK,CAAC,2BAA2B,cAAc,CAAA,CAAE,CAAC;QAC9D;;;;QAKA,MAAM,OAAO,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC;;;AAIzC,QAAA,MAAM,CAAC,MAAM,CAAC,OAAO,CAAC;AAEtB,QAAAA,WAAM,CAAC,YAAY,CAAC,CAAC,KAAK,CACxB,kBAAkB,OAAO,CAAC,KAAK,CAAA,aAAA,EAAgB,cAAc,YAAY,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,CAAA,EAAA,CAAI;YAC/F,CAAA,kBAAA,EAAqB,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,eAAe,CAAC,CAAA,CAAE,CAC9D;QAED,OAAO,QAAQ,CAAC,QAAQ,CACtB,cAAc,EACd,YAAW;YACT,MAAM,UAAU,GAAe,MAAM,CAAC,MAAM,CAAC,EAAE,EAAE,OAAO,CAAC;YACzDA,WAAM,CAAC,YAAY,CAAC,CAAC,KAAK,CACxB,CAAA,uBAAA,EAA0B,UAAU,CAAC,KAAK,CAAA,IAAA,EAAO,UAAU,CAAC,YAAa,CAAC,WAAW,EAAE,CAAA,YAAA,EAAe,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,CAAA,CAAE,CAC7H;YACD,OAAO,IAAI,CAAC,OAAO,CAAC,YAAY,CAAC,OAAO,CAAC;AAC3C,QAAA,CAAC,EACD,EAAE,SAAS,EAAE,IAAI,EAAE,CACpB;IACH;AACD;;;;"}
|