@memberjunction/queue 4.0.0 → 4.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +129 -262
  2. package/package.json +7 -7
package/README.md CHANGED
@@ -1,17 +1,31 @@
1
1
  # @memberjunction/queue
2
2
 
3
- A flexible queue management system for MemberJunction applications that enables background task processing, job scheduling, and asynchronous execution with database persistence.
3
+ A server-side queue management framework for MemberJunction applications that provides database-backed task persistence, concurrent processing, and heartbeat monitoring.
4
4
 
5
5
  ## Overview
6
6
 
7
- The `@memberjunction/queue` package provides a robust framework for implementing persistent queues in MemberJunction applications. It offers:
8
-
9
- - Database-backed task persistence
10
- - Automatic queue creation and management
11
- - Concurrent task processing with configurable limits
12
- - Heartbeat monitoring for process health
13
- - Type-safe task definitions
14
- - Extensible queue implementations
7
+ The `@memberjunction/queue` package delivers a robust queuing system for background task processing. It manages task lifecycle from creation through execution, with automatic queue provisioning, configurable concurrency limits, and process-level health tracking.
8
+
9
+ ```mermaid
10
+ graph TD
11
+ A["QueueManager<br/>(Singleton)"] --> B["QueueBase<br/>(Abstract)"]
12
+ B --> C["AIActionQueue"]
13
+ B --> D["EntityAIActionQueue"]
14
+ B --> E["Custom Queue<br/>(Your Implementation)"]
15
+
16
+ A --> F["Queue Types<br/>(Database Metadata)"]
17
+ A --> G["Queue Records<br/>(Process Tracking)"]
18
+ B --> H["TaskBase<br/>(Individual Tasks)"]
19
+
20
+ style A fill:#2d6a9f,stroke:#1a4971,color:#fff
21
+ style B fill:#7c5295,stroke:#563a6b,color:#fff
22
+ style C fill:#2d8659,stroke:#1a5c3a,color:#fff
23
+ style D fill:#2d8659,stroke:#1a5c3a,color:#fff
24
+ style E fill:#b8762f,stroke:#8a5722,color:#fff
25
+ style F fill:#2d6a9f,stroke:#1a4971,color:#fff
26
+ style G fill:#2d6a9f,stroke:#1a4971,color:#fff
27
+ style H fill:#7c5295,stroke:#563a6b,color:#fff
28
+ ```
15
29
 
16
30
  ## Installation
17
31
 
@@ -19,319 +33,172 @@ The `@memberjunction/queue` package provides a robust framework for implementing
19
33
  npm install @memberjunction/queue
20
34
  ```
21
35
 
22
- ## Dependencies
23
-
24
- This package requires the following MemberJunction packages:
25
-
26
- - `@memberjunction/core` - Core functionality and entity management
27
- - `@memberjunction/global` - Global utilities and class registration
28
- - `@memberjunction/core-entities` - Entity type definitions
29
- - `@memberjunction/ai` - AI functionality (for AI-related queues)
30
- - `@memberjunction/aiengine` - AI Engine integration
31
-
32
- Additional dependencies:
33
- - `uuid` - For generating unique identifiers
34
-
35
- ## Core Components
36
-
37
- ### TaskBase
36
+ ## Architecture
38
37
 
39
- The `TaskBase` class represents an individual task in a queue:
38
+ ### Task Lifecycle
40
39
 
41
- ```typescript
42
- export class TaskBase {
43
- constructor(
44
- taskRecord: QueueTaskEntity,
45
- data: any,
46
- options: TaskOptions
47
- )
48
-
49
- // Properties
50
- ID: string // Unique task identifier
51
- Status: TaskStatus // Current task status
52
- Data: any // Task payload data
53
- Options: TaskOptions // Task configuration
54
- TaskRecord: QueueTaskEntity // Database entity
55
- }
40
+ ```mermaid
41
+ stateDiagram-v2
42
+ [*] --> Pending: Task Created
43
+ Pending --> InProgress: Queue Picks Up
44
+ InProgress --> Complete: ProcessTask Succeeds
45
+ InProgress --> Failed: ProcessTask Fails
46
+ Pending --> Cancelled: External Cancel
56
47
  ```
57
48
 
58
- ### TaskStatus
59
-
60
- Available task statuses:
61
-
62
- ```typescript
63
- export const TaskStatus = {
64
- Pending: 'Pending',
65
- InProgress: 'InProgress',
66
- Complete: 'Complete',
67
- Failed: 'Failed',
68
- Cancelled: 'Cancelled',
69
- } as const;
49
+ ### Processing Flow
50
+
51
+ ```mermaid
52
+ sequenceDiagram
53
+ participant Client
54
+ participant QM as QueueManager
55
+ participant QB as QueueBase
56
+ participant DB as Database
57
+
58
+ Client->>QM: AddTask(type, data, options)
59
+ QM->>QM: Find or create queue for type
60
+ QM->>DB: Save QueueTask record (Pending)
61
+ QM->>QB: AddTask(taskBase)
62
+ QB->>QB: ProcessTasks() loop (250ms interval)
63
+ QB->>QB: Check concurrency (max 3 tasks)
64
+ QB->>QB: StartTask(task)
65
+ QB->>QB: ProcessTask(task) [abstract]
66
+ QB->>DB: Update QueueTask status
67
+ QB-->>Client: TaskResult
70
68
  ```
71
69
 
72
- ### QueueBase
73
-
74
- The abstract `QueueBase` class serves as the foundation for all queue implementations:
75
-
76
- ```typescript
77
- export abstract class QueueBase {
78
- constructor(
79
- QueueRecord: QueueEntity,
80
- QueueTypeID: string,
81
- ContextUser: UserInfo
82
- )
83
-
84
- // Public methods
85
- AddTask(task: TaskBase): boolean
86
- FindTask(ID: string): TaskBase
87
-
88
- // Protected abstract method to implement
89
- protected abstract ProcessTask(
90
- task: TaskBase,
91
- contextUser: UserInfo
92
- ): Promise<TaskResult>
93
- }
94
- ```
70
+ ## Core Components
95
71
 
96
72
  ### QueueManager
97
73
 
98
- The `QueueManager` is a singleton that manages all active queues:
74
+ The `QueueManager` is a singleton that coordinates all active queues. It auto-creates queue instances per type and captures process-level metadata (PID, hostname, network interfaces) for monitoring.
99
75
 
100
76
  ```typescript
101
- export class QueueManager {
102
- // Singleton access
103
- static get Instance(): QueueManager
104
-
105
- // Static methods
106
- static async Config(contextUser: UserInfo): Promise<void>
107
- static async AddTask(
108
- QueueType: string,
109
- data: any,
110
- options: any,
111
- contextUser: UserInfo
112
- ): Promise<TaskBase | undefined>
113
-
114
- // Instance methods
115
- async AddTask(
116
- QueueTypeID: string,
117
- data: any,
118
- options: any,
119
- contextUser: UserInfo
120
- ): Promise<TaskBase | undefined>
121
- }
122
- ```
77
+ import { QueueManager } from '@memberjunction/queue';
123
78
 
124
- ### TaskResult
79
+ // Initialize (typically at application startup)
80
+ await QueueManager.Config(contextUser);
125
81
 
126
- Structure returned by task processing:
82
+ // Add a task by queue type name
83
+ const task = await QueueManager.AddTask(
84
+ 'Email Notification',
85
+ { recipient: 'user@example.com', subject: 'Welcome' },
86
+ { priority: 1 },
87
+ contextUser
88
+ );
127
89
 
128
- ```typescript
129
- export class TaskResult {
130
- success: boolean // Whether task completed successfully
131
- userMessage: string // User-friendly message
132
- output: any // Task output data
133
- exception: any // Exception details if failed
90
+ if (task) {
91
+ console.log(`Task created: ${task.ID}`);
134
92
  }
135
93
  ```
136
94
 
137
- ## Usage Examples
138
-
139
- ### Basic Queue Implementation
95
+ ### QueueBase
140
96
 
141
- Create a custom queue by extending `QueueBase`:
97
+ Abstract base class for all queue implementations. Subclasses implement `ProcessTask()` to define task execution logic.
142
98
 
143
99
  ```typescript
144
100
  import { QueueBase, TaskBase, TaskResult } from '@memberjunction/queue';
145
101
  import { RegisterClass } from '@memberjunction/global';
146
102
  import { UserInfo } from '@memberjunction/core';
147
103
 
148
- // Register your queue with a specific queue type name
149
104
  @RegisterClass(QueueBase, 'Email Notification')
150
105
  export class EmailNotificationQueue extends QueueBase {
151
106
  protected async ProcessTask(
152
- task: TaskBase,
107
+ task: TaskBase,
153
108
  contextUser: UserInfo
154
109
  ): Promise<TaskResult> {
155
- try {
156
- // Extract task data
157
- const { recipient, subject, body } = task.Data;
158
-
159
- // Implement your email sending logic here
160
- console.log(`Sending email to ${recipient}`);
161
- console.log(`Subject: ${subject}`);
162
-
163
- // Simulate email sending
164
- await this.sendEmail(recipient, subject, body);
165
-
166
- // Return success result
167
- return {
168
- success: true,
169
- userMessage: 'Email sent successfully',
170
- output: { sentAt: new Date() },
171
- exception: null
172
- };
173
- } catch (error) {
174
- // Return failure result
175
- return {
176
- success: false,
177
- userMessage: `Failed to send email: ${error.message}`,
178
- output: null,
179
- exception: error
180
- };
181
- }
182
- }
183
-
184
- private async sendEmail(to: string, subject: string, body: string) {
185
- // Your email service integration here
110
+ const { recipient, subject, body } = task.Data;
111
+ await sendEmail(recipient, subject, body);
112
+ return {
113
+ success: true,
114
+ userMessage: 'Email sent successfully',
115
+ output: { sentAt: new Date() },
116
+ exception: null
117
+ };
186
118
  }
187
119
  }
188
120
  ```
189
121
 
190
- ### Adding Tasks to Queue
122
+ ### TaskBase
191
123
 
192
- ```typescript
193
- import { QueueManager } from '@memberjunction/queue';
194
- import { UserInfo } from '@memberjunction/core';
124
+ Represents an individual task with its payload, options, and database-backed record.
195
125
 
196
- // Initialize queue manager (typically done once at app startup)
197
- await QueueManager.Config(contextUser);
126
+ | Property | Type | Description |
127
+ |----------|------|-------------|
128
+ | `ID` | `string` | Unique task identifier from database |
129
+ | `Status` | `TaskStatus` | Current status (Pending, InProgress, Complete, Failed, Cancelled) |
130
+ | `Data` | `object` | Task payload data |
131
+ | `Options` | `TaskOptions` | Configuration (e.g., priority) |
132
+ | `TaskRecord` | `QueueTaskEntity` | Underlying database entity |
198
133
 
199
- // Add a task using queue type name
200
- const task = await QueueManager.AddTask(
201
- 'Email Notification', // Queue type name
202
- { // Task data
203
- recipient: 'user@example.com',
204
- subject: 'Welcome to MemberJunction',
205
- body: 'Thank you for joining!'
206
- },
207
- { // Task options
208
- priority: 1
209
- },
210
- contextUser
211
- );
134
+ ### TaskResult
212
135
 
213
- if (task) {
214
- console.log(`Task created with ID: ${task.ID}`);
215
- }
216
- ```
136
+ Returned by `ProcessTask()` to communicate outcome.
217
137
 
218
- ### AI Action Queue Example
138
+ | Property | Type | Description |
139
+ |----------|------|-------------|
140
+ | `success` | `boolean` | Whether the task completed successfully |
141
+ | `userMessage` | `string` | Human-readable result message |
142
+ | `output` | `object` | Task output data |
143
+ | `exception` | `object` | Error details if failed |
219
144
 
220
- The package includes built-in queues for AI operations:
145
+ ## Built-in Queues
221
146
 
222
- ```typescript
223
- import { AIActionQueue, EntityAIActionQueue } from '@memberjunction/queue';
147
+ ### AIActionQueue
148
+
149
+ Processes AI actions through the MemberJunction AI Engine.
224
150
 
225
- // These queues are automatically registered and available for use
226
- // Add an AI action task
227
- const aiTask = await QueueManager.AddTask(
151
+ ```typescript
152
+ const task = await QueueManager.AddTask(
228
153
  'AI Action',
229
- {
230
- actionName: 'GenerateText',
231
- prompt: 'Write a product description',
232
- parameters: { maxTokens: 100 }
233
- },
154
+ { actionName: 'GenerateText', prompt: 'Summarize this document' },
234
155
  {},
235
156
  contextUser
236
157
  );
158
+ ```
237
159
 
238
- // Add an entity-specific AI action
239
- const entityAITask = await QueueManager.AddTask(
160
+ ### EntityAIActionQueue
161
+
162
+ Processes entity-specific AI actions.
163
+
164
+ ```typescript
165
+ const task = await QueueManager.AddTask(
240
166
  'Entity AI Action',
241
- {
242
- entityName: 'Products',
243
- entityID: 123,
244
- actionName: 'GenerateDescription'
245
- },
167
+ { entityName: 'Products', entityID: '123', actionName: 'GenerateDescription' },
246
168
  {},
247
169
  contextUser
248
170
  );
249
171
  ```
250
172
 
251
- ## Database Schema
252
-
253
- The queue system requires the following database tables:
254
-
255
- ### Queue Types Table (`__mj.QueueType`)
256
- Stores definitions of different queue types (e.g., "Email Notification", "AI Action")
257
-
258
- ### Queues Table (`__mj.Queue`)
259
- Tracks active queue instances with process information:
260
- - Queue type reference
261
- - Process details (PID, platform, hostname)
262
- - Heartbeat timestamp
263
- - Network information
264
-
265
- ### Queue Tasks Table (`__mj.QueueTask`)
266
- Stores individual tasks:
267
- - Queue reference
268
- - Task status
269
- - Task data (JSON)
270
- - Task options (JSON)
271
- - Output and error information
272
-
273
- ## Process Management
274
-
275
- The QueueManager automatically captures process information for monitoring:
276
- - Process ID (PID)
277
- - Platform and version
278
- - Working directory
279
- - Network interfaces
280
- - Operating system details
281
- - User information
282
- - Heartbeat timestamps
283
-
284
- This information helps track queue health and enables failover scenarios.
285
-
286
173
  ## Configuration
287
174
 
288
- Queue behavior can be configured through the implementation:
175
+ Queue behavior is controlled through constructor parameters:
289
176
 
290
- ```typescript
291
- export class CustomQueue extends QueueBase {
292
- private _maxTasks = 5; // Maximum concurrent tasks
293
- private _checkInterval = 500; // Check interval in milliseconds
294
-
295
- // Override these values in your constructor
296
- constructor(QueueRecord: QueueEntity, QueueTypeID: string, ContextUser: UserInfo) {
297
- super(QueueRecord, QueueTypeID, ContextUser);
298
- // Customize queue behavior
299
- this._maxTasks = 10;
300
- this._checkInterval = 1000;
301
- }
302
- }
303
- ```
304
-
305
- ## Best Practices
306
-
307
- 1. **Task Data Structure**: Keep task data serializable as JSON
308
- 2. **Error Handling**: Always return proper TaskResult with error details
309
- 3. **Queue Registration**: Use `@RegisterClass` decorator for automatic registration
310
- 4. **Idempotency**: Design tasks to be safely retryable
311
- 5. **Resource Cleanup**: Clean up resources in finally blocks
312
- 6. **Monitoring**: Check heartbeat timestamps for queue health
313
-
314
- ## Integration with MemberJunction
177
+ | Parameter | Default | Description |
178
+ |-----------|---------|-------------|
179
+ | `_maxTasks` | `3` | Maximum concurrent tasks per queue |
180
+ | `_checkInterval` | `250` | Polling interval in milliseconds |
315
181
 
316
- The queue system integrates seamlessly with:
317
- - **Entity System**: Use entities for task data and processing
318
- - **User Context**: All operations respect user permissions
319
- - **Global Registry**: Automatic queue discovery via class registration
320
- - **AI Engine**: Built-in support for AI task processing
182
+ ## Database Schema
321
183
 
322
- ## Build & Development
184
+ The queue system persists state across three tables:
323
185
 
324
- ```bash
325
- # Build the package
326
- npm run build
186
+ | Table | Purpose |
187
+ |-------|---------|
188
+ | `__mj.QueueType` | Defines available queue types |
189
+ | `__mj.Queue` | Tracks active queue instances with process info and heartbeat |
190
+ | `__mj.QueueTask` | Stores individual tasks with status, data, and output |
327
191
 
328
- # Development mode with auto-reload
329
- npm run start
192
+ ## Dependencies
330
193
 
331
- # TypeScript compilation only
332
- npm run build
333
- ```
194
+ | Package | Purpose |
195
+ |---------|---------|
196
+ | `@memberjunction/core` | Entity management and metadata |
197
+ | `@memberjunction/global` | Class registration and global state |
198
+ | `@memberjunction/core-entities` | Queue and task entity types |
199
+ | `@memberjunction/ai` | AI functionality for built-in queues |
200
+ | `@memberjunction/aiengine` | AI Engine integration |
334
201
 
335
202
  ## License
336
203
 
337
- ISC
204
+ ISC
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "@memberjunction/queue",
3
3
  "type": "module",
4
- "version": "4.0.0",
4
+ "version": "4.2.0",
5
5
  "description": "MemberJunction: Queue Library for managing server side queues",
6
6
  "main": "dist/index.js",
7
7
  "types": "dist/index.d.ts",
@@ -20,12 +20,12 @@
20
20
  "typescript": "^5.9.3"
21
21
  },
22
22
  "dependencies": {
23
- "@memberjunction/ai": "4.0.0",
24
- "@memberjunction/ai-provider-bundle": "4.0.0",
25
- "@memberjunction/aiengine": "4.0.0",
26
- "@memberjunction/core": "4.0.0",
27
- "@memberjunction/global": "4.0.0",
28
- "@memberjunction/core-entities": "4.0.0",
23
+ "@memberjunction/ai": "4.2.0",
24
+ "@memberjunction/ai-provider-bundle": "4.2.0",
25
+ "@memberjunction/aiengine": "4.2.0",
26
+ "@memberjunction/core": "4.2.0",
27
+ "@memberjunction/global": "4.2.0",
28
+ "@memberjunction/core-entities": "4.2.0",
29
29
  "@types/uuid": "^11.0.0",
30
30
  "uuid": "^13.0.0"
31
31
  },