multi-tasks 3.1.0 → 3.1.2

package/README.md

@@ -1,6 +1,6 @@
 # multi-tasks
 
-Multi-tasks is a toolkit to manage long-term and large-scale parallel computing tasks. It manages progress and tasks based on the file system, which means that even if the host crashes, tasks can be resumed based on file records.
+Multi-tasks is a toolkit to manage long-term and large-scale parallel computing tasks. Progress and tasks are stored on the file system, so tasks can be resumed even if the host crashes.
 
 **Zero dependencies:** multi-tasks has no runtime dependencies at all — it is built entirely on Node.js built-in modules (`cluster`, `fs`, `path`, `os`), so installing it adds nothing extra to your `node_modules`.
 
@@ -12,27 +12,36 @@ npm install multi-tasks
 
 ### API:
 
-- `multiTasks(config)` — run tasks; auto-resumes if the task folder already exists or `initialTasks` points to one.
-- `multiTasks.start(config)` — always start a fresh run.
-- `multiTasks.resume(config)` — resume an interrupted run.
-- `multiTasks.restart(config)` — resume and also retry failed tasks (needs `config.taskFolder`).
-- `helper.createNewTasks(tasks)` — create new tasks dynamically while processing.
-- `helper.emit(event, payload, option?)` — broadcast an event to all workers (relayed by the master, best-effort); `option.includingMe` defaults to `true` — pass `{includingMe: false}` to exclude the sender. Payload must be JSON-serializable.
-- `helper.setListener(event, handler)` — listen for broadcast events; one handler per event per worker process (re-registering replaces it), so calling it inside `processTask` is always safe; `handler(payload, meta)` gets `meta.fromWorkerId` and `meta.fromWorkerPid`. Broadcasts are runtime-only messages — not persisted, not replayed on resume.
-- `helper.emitSys(event, payload?)` — send a system event to the master (consumed by the master itself, not relayed). Built-in event: `'TERMINATE_ALL_WORKERS'` force-kills all workers and exits the master; any other event goes to `config.setSysListener`.
-- Config options:
-  - `initialTasks` — array of task objects, or a folder path string to resume from.
-  - `processTask(task, helper)` — required; return a value or a Promise.
-  - `taskRootFolder` — root directory where progress and result files are stored.
-  - `taskId` — task folder name under `taskRootFolder`.
-  - `taskFolder` — full task folder path, required by `restart`.
-  - `numberOfWorkers` — how many worker processes run in parallel; a number, or a percentage string of CPU cores like `"50%"` (default).
-  - `taskTimeout` — optional, milliseconds; an overdue task fails with a timeout error.
-  - `maxTaskRetries` — optional, max times a failed task is auto-retried.
-  - `autoCloseAfterCompletion` — set `false` if you create new tasks dynamically.
-  - `shouldTerminate(info)` — return `true` to terminate the whole process.
-  - `setSysListener(event, payload, meta)` — optional, handle custom system events (sent via `helper.emitSys`) on the master; `meta` has `fromWorkerId` and `fromWorkerPid`.
-  - `onFinish(report)` — called once after all workers are done.
+The API has three parts: the entry functions, the config options, and the `helper` object injected into `processTask`.
+
+**Entry functions:**
+
+- **`multiTasks(config)`** — run tasks; auto-resumes if the task folder already exists or `initialTasks` points to one.
+- **`multiTasks.start(config)`** — always start a fresh run.
+- **`multiTasks.resume(config)`** — resume an interrupted run.
+- **`multiTasks.restart(config)`** — resume and also retry failed tasks (needs `config.taskFolder`).
+
+**Config options:**
+
+- **`initialTasks`** — array of task objects, or a folder path string to resume from.
+- **`processTask(task, helper)`** — required; return a value or a Promise.
+- **`taskRootFolder`** — root directory where progress and result files are stored.
+- **`taskId`** — task folder name under `taskRootFolder`.
+- **`taskFolder`** — full task folder path, required by `restart`.
+- **`numberOfWorkers`** — how many worker processes run in parallel; a number, or a percentage string of CPU cores like `"50%"` (default).
+- **`taskTimeout`** — optional, milliseconds; an overdue task fails with a timeout error.
+- **`maxTaskRetries`** — optional, max times a failed task is auto-retried.
+- **`autoCloseAfterCompletion`** — set `false` if you create new tasks dynamically.
+- **`shouldTerminate(info)`** — return `true` to terminate the whole process.
+- **`setSysListener(event, payload, meta)`** — optional, handle custom system events (sent via `helper.emitSys`) on the master; `meta` has `fromWorkerId` and `fromWorkerPid`.
+- **`onFinish(report)`** — called once after all workers are done.
+
+**Task helper** (the `helper` object passed as the second argument of `processTask`):
+
+- **`helper.createNewTasks(tasks)`** — create new tasks dynamically while processing.
+- **`helper.emit(event, payload, option?)`** — broadcast an event to all workers (relayed by the master, best-effort); `option.includingMe` defaults to `true` — pass `{includingMe: false}` to exclude the sender. Payload must be JSON-serializable.
+- **`helper.setListener(event, handler)`** — listen for broadcast events; one handler per event per worker process (re-registering replaces it), so calling it inside `processTask` is always safe; `handler(payload, meta)` gets `meta.fromWorkerId` and `meta.fromWorkerPid`. Broadcasts are runtime-only messages — not persisted, not replayed on resume.
+- **`helper.emitSys(event, payload?)`** — send a system event to the master (consumed by the master itself, not relayed). Built-in event: `'TERMINATE_ALL_WORKERS'` force-kills all workers and exits the master; any other event goes to `config.setSysListener`.
 
 ### How to use:
 
@@ -40,7 +49,7 @@ npm install multi-tasks
 //see examples/example0
 let multiTasks = require('multi-tasks').multiTasks;
 
-//Step1, create your tasks that need to be executed simultaneously as an array.
+//Step1, create the tasks to run in parallel as an array.
 let alltasks = [];
 for(let i=0;i<50;i++){
     alltasks.push({
@@ -49,7 +58,7 @@ for(let i=0;i<50;i++){
     });
 };
 
-//Step2, provide a function to process a certain sub-task and return the result data
+//Step2, provide a function that processes each sub-task and returns the result
 let processTask = (task, helper)=>{
     let {taskCount} = task;//get your task data
 
@@ -63,12 +72,12 @@ let processTask = (task, helper)=>{
 multiTasks({
     initialTasks: alltasks, 
     processTask, 
-    taskRootFolder: `../examples-tmp-data/example0`, //a directory to store progress and results files, you can check the progress here
+    taskRootFolder: `../examples-tmp-data/example0`, //a directory to store progress and result files; you can check the progress here
     taskId: 'my-task',
     numberOfWorkers: 3, //how many workers are working in parallel
     //taskTimeout: 30000, //optional, in milliseconds, an overdue task is treated as failed with a timeout error
     //maxTaskRetries: 2, //optional, auto-retry a failed task (worker crash, processTask error, or timeout); retried tasks go back to the queue
-    //autoCloseAfterCompletion: true, //if you have dynamically generated new tasks, put this as false
+    //autoCloseAfterCompletion: true, //if you have dynamically generated new tasks, set this to false
     shouldTerminate:(info)=>{
         //return true if you need to terminate the whole process
     },
@@ -118,21 +127,26 @@ let processTask = (task, helper)=>{
 let processTask = (task, helper)=>{
     let {taskCount} = task;
 
-    //This is the demo of exceptions/errors, they will be captured and saved in the results/errors folder
+    //Demo of exceptions/errors: they are captured and saved in the results/errors folder
     if(taskCount===3) throw 'exception';
     if(taskCount===4) return Promise.reject({err:'a test error'});//use Promise.reject method
-    if(taskCount===5) aaa =  bbb;//this undefined exception will be captured by multi-tasks
+    if(taskCount===5) aaa =  bbb;//this ReferenceError will be captured by multi-tasks
 
     return {data:'succ'};
 }
 
-//Example5, auto-retry a failed task:
-//          when a task fails - its worker process crashes, processTask throws
-//          or rejects, or it exceeds taskTimeout - it is sent back to the
-//          queue and retried, at most maxTaskRetries times; when the limit
-//          is reached it lands in finished_with_errors (crash:
-//          {type:'worker_crash'}, timeout: {type:'timeout'}, processTask
-//          error: the original error data)
+//Example4, timeout: a task that does not finish within taskTimeout is
+//          treated as failed with a timeout error
+multiTasks({
+    initialTasks: alltasks,
+    processTask,
+    taskRootFolder: `../examples-tmp-data/example-timeout`,
+    taskId: 'my-task',
+    numberOfWorkers: 3,
+    taskTimeout: 30000,
+});
+
+//Example5, auto-retry a failed task with maxTaskRetries
 multiTasks({
     initialTasks: alltasks,
     processTask,
@@ -142,13 +156,7 @@ multiTasks({
     maxTaskRetries: 2,
 });
 
-//Example6, broadcast events between workers:
-//          a worker emits an event and the master relays it to every worker
-//          (best-effort, runtime only - not persisted, not replayed on
-//          resume); by default the sender also receives its own event, pass
-//          {includingMe: false} to exclude it; each event keeps only one
-//          listener per worker process, so calling setListener on every
-//          task is safe
+//Example6, broadcast events between workers with helper.emit/helper.setListener
 multiTasks({
     initialTasks: alltasks,
     taskRootFolder: `../examples-tmp-data/example-broadcast`,
@@ -163,12 +171,7 @@ multiTasks({
     },
 });
 
-//Example7, system events from a worker to the master:
-//          helper.emitSys sends a system event that the master consumes
-//          itself (not relayed to workers); the built-in event
-//          TERMINATE_ALL_WORKERS force-kills all workers and exits the
-//          master (unfinished tasks stay for resume); any other event is
-//          passed to config.setSysListener on the master
+//Example7, system events from a worker to the master with helper.emitSys
 multiTasks({
     initialTasks: alltasks,
     taskRootFolder: `../examples-tmp-data/example-sysevent`,
@@ -191,12 +194,12 @@ multiTasks({
 
 ### Resuming
 
-Sometimes the task execution is interrupted due to some reasons (such as power outage), you can resume the execution like this
+If the execution is interrupted (e.g. a power outage), resume it like this:
 
 ```javascript
 
 multiTasks({
-    initialTasks: `/myworks/my_scan_tasks/`, //Point 'initialTasks' to the interrupted task directory, multi-tasks will read the tasks in the 'new' folder and initialize them to 'initialTasks' and then continue execution
+    initialTasks: `/myworks/my_scan_tasks/`, //point 'initialTasks' to the interrupted task directory; multi-tasks reads the tasks in its 'new' folder and continues execution
     ...
     ...//Other configurations remain unchanged
     ...
@@ -204,12 +207,10 @@ multiTasks({
 
 ```
 
-### Testing:
-
-The usage patterns documented above are covered by automated tests (unit tests in `test/`, end-to-end tests in `teste2e/`). The e2e suite runs the API in real child processes with a mixed workload — successful tasks (both promise and non-promise results), planned failures (rejected promises and thrown exceptions), dynamically created tasks via `helper.createNewTasks`, and workers that crash randomly — then resumes repeatedly and verifies that every task lands exactly one result in `results/succ` or `results/errors`.
-
 ### Changelog:
 
+- 3.1.2 Update README
+- 3.1.1 Fix readme documentation
 - 3.1.0 Support worker broadcast ('helper.emit' and 'helper.setListener') and system events ('helper.emitSys' and 'setSysListener', with built-in 'TERMINATE_ALL_WORKERS'); default numberOfWorkers is now "50%" of CPU cores (was core count minus 1)
 - 3.0.4 numberOfWorkers accepts a percentage string of CPU cores, e.g. "50%"
 - 3.0.3 Support 'maxTaskRetries'
@@ -253,4 +254,4 @@ The usage patterns documented above are covered by automated tests (unit tests i
 
 ### License:
 
-[MIT](https://opensource.org/license/MIT) (see [LICENSE](LICENSE))
+[MIT](https://opensource.org/license/MIT) 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "multi-tasks",
-  "version": "3.1.0",
+  "version": "3.1.2",
   "description": "Multi-process task scheduling based on Node.js cluster, with crash resume and failed-task restart support",
   "main": "index.js",
   "files": [