multi-tasks 3.1.1 → 3.1.3

package/README.md

@@ -1,6 +1,6 @@
 # multi-tasks
 
-Multi-tasks is a toolkit to manage long-term and large-scale parallel computing tasks. It manages progress and tasks based on the file system, which means that even if the host crashes, tasks can be resumed based on file records.
+Multi-tasks is a toolkit to manage long-term and large-scale parallel computing tasks. Progress and tasks are stored on the file system, so tasks can be resumed even if the host crashes.
 
 **Zero dependencies:** multi-tasks has no runtime dependencies at all — it is built entirely on Node.js built-in modules (`cluster`, `fs`, `path`, `os`), so installing it adds nothing extra to your `node_modules`.
 
@@ -31,10 +31,10 @@ The API has three parts: the entry functions, the config options, and the `helpe
 - **`numberOfWorkers`** — how many worker processes run in parallel; a number, or a percentage string of CPU cores like `"50%"` (default).
 - **`taskTimeout`** — optional, milliseconds; an overdue task fails with a timeout error.
 - **`maxTaskRetries`** — optional, max times a failed task is auto-retried.
-- **`autoCloseAfterCompletion`** — set `false` if you create new tasks dynamically.
+- **`autoCloseAfterCompletion`** — set `true` to kill workers and exit the process when no tasks are left; default is `false`, i.e. workers stay alive waiting for dynamically created tasks.
 - **`shouldTerminate(info)`** — return `true` to terminate the whole process.
 - **`setSysListener(event, payload, meta)`** — optional, handle custom system events (sent via `helper.emitSys`) on the master; `meta` has `fromWorkerId` and `fromWorkerPid`.
-- **`onFinish(report)`** — called once after all workers are done.
+- **`onFinish(report)`** — called once after all workers exit (only when `autoCloseAfterCompletion` is `true`); `report` is an empty object for now.
 
 **Task helper** (the `helper` object passed as the second argument of `processTask`):
 
@@ -49,7 +49,7 @@ The API has three parts: the entry functions, the config options, and the `helpe
 //see examples/example0
 let multiTasks = require('multi-tasks').multiTasks;
 
-//Step1, create your tasks that need to be executed simultaneously as an array.
+//Step1, create the tasks to run in parallel as an array.
 let alltasks = [];
 for(let i=0;i<50;i++){
     alltasks.push({
@@ -58,7 +58,7 @@ for(let i=0;i<50;i++){
     });
 };
 
-//Step2, provide a function to process a certain sub-task and return the result data
+//Step2, provide a function that processes each sub-task and returns the result
 let processTask = (task, helper)=>{
     let {taskCount} = task;//get your task data
 
@@ -72,12 +72,12 @@ let processTask = (task, helper)=>{
 multiTasks({
     initialTasks: alltasks, 
     processTask, 
-    taskRootFolder: `../examples-tmp-data/example0`, //a directory to store progress and results files, you can check the progress here
+    taskRootFolder: `../examples-tmp-data/example0`, //a directory to store progress and result files; you can check the progress here
     taskId: 'my-task',
     numberOfWorkers: 3, //how many workers are working in parallel
     //taskTimeout: 30000, //optional, in milliseconds, an overdue task is treated as failed with a timeout error
     //maxTaskRetries: 2, //optional, auto-retry a failed task (worker crash, processTask error, or timeout); retried tasks go back to the queue
-    //autoCloseAfterCompletion: true, //if you have dynamically generated new tasks, put this as false
+    //autoCloseAfterCompletion: true, //set true to exit the process when all tasks are done; keep it false (default) if you create new tasks dynamically
     shouldTerminate:(info)=>{
         //return true if you need to terminate the whole process
     },
@@ -91,6 +91,8 @@ multiTasks({
 ### More about the task processing function
 
 ```javascript
+//Examples 1-3 below are independent variants of processTask, use one at a time.
+
 //Example1, return a promise for async processes,
 //          you can return a promise or a non-promise result, 
 //          all result data can be found in the results/succ folder
@@ -127,21 +129,26 @@ let processTask = (task, helper)=>{
 let processTask = (task, helper)=>{
     let {taskCount} = task;
 
-    //This is the demo of exceptions/errors, they will be captured and saved in the results/errors folder
+    //Demo of exceptions/errors: they are captured and saved in the results/errors folder
     if(taskCount===3) throw 'exception';
     if(taskCount===4) return Promise.reject({err:'a test error'});//use Promise.reject method
-    if(taskCount===5) aaa =  bbb;//this undefined exception will be captured by multi-tasks
+    if(taskCount===5) aaa =  bbb;//this ReferenceError will be captured by multi-tasks
 
     return {data:'succ'};
 }
 
-//Example5, auto-retry a failed task:
-//          when a task fails - its worker process crashes, processTask throws
-//          or rejects, or it exceeds taskTimeout - it is sent back to the
-//          queue and retried, at most maxTaskRetries times; when the limit
-//          is reached it lands in finished_with_errors (crash:
-//          {type:'worker_crash'}, timeout: {type:'timeout'}, processTask
-//          error: the original error data)
+//Example4, timeout: a task that does not finish within taskTimeout is
+//          treated as failed with a timeout error
+multiTasks({
+    initialTasks: alltasks,
+    processTask,
+    taskRootFolder: `../examples-tmp-data/example-timeout`,
+    taskId: 'my-task',
+    numberOfWorkers: 3,
+    taskTimeout: 30000,
+});
+
+//Example5, auto-retry a failed task with maxTaskRetries
 multiTasks({
     initialTasks: alltasks,
     processTask,
@@ -151,13 +158,7 @@ multiTasks({
     maxTaskRetries: 2,
 });
 
-//Example6, broadcast events between workers:
-//          a worker emits an event and the master relays it to every worker
-//          (best-effort, runtime only - not persisted, not replayed on
-//          resume); by default the sender also receives its own event, pass
-//          {includingMe: false} to exclude it; each event keeps only one
-//          listener per worker process, so calling setListener on every
-//          task is safe
+//Example6, broadcast events between workers with helper.emit/helper.setListener
 multiTasks({
     initialTasks: alltasks,
     taskRootFolder: `../examples-tmp-data/example-broadcast`,
@@ -172,12 +173,7 @@ multiTasks({
     },
 });
 
-//Example7, system events from a worker to the master:
-//          helper.emitSys sends a system event that the master consumes
-//          itself (not relayed to workers); the built-in event
-//          TERMINATE_ALL_WORKERS force-kills all workers and exits the
-//          master (unfinished tasks stay for resume); any other event is
-//          passed to config.setSysListener on the master
+//Example7, system events from a worker to the master with helper.emitSys
 multiTasks({
     initialTasks: alltasks,
     taskRootFolder: `../examples-tmp-data/example-sysevent`,
@@ -200,12 +196,12 @@ multiTasks({
 
 ### Resuming
 
-Sometimes the task execution is interrupted due to some reasons (such as power outage), you can resume the execution like this
+If the execution is interrupted (e.g. a power outage), resume it like this:
 
 ```javascript
 
 multiTasks({
-    initialTasks: `/myworks/my_scan_tasks/`, //Point 'initialTasks' to the interrupted task directory, multi-tasks will read the tasks in the 'new' folder and initialize them to 'initialTasks' and then continue execution
+    initialTasks: `/myworks/my_scan_tasks/`, //point 'initialTasks' to the interrupted task directory; multi-tasks reads the tasks in its 'new' folder and continues execution
     ...
     ...//Other configurations remain unchanged
     ...
@@ -213,12 +209,10 @@ multiTasks({
 
 ```
 
-### Testing:
-
-The usage patterns documented above are covered by automated tests (unit tests in `test/`, end-to-end tests in `teste2e/`). The e2e suite runs the API in real child processes with a mixed workload — successful tasks (both promise and non-promise results), planned failures (rejected promises and thrown exceptions), dynamically created tasks via `helper.createNewTasks`, and workers that crash randomly — then resumes repeatedly and verifies that every task lands exactly one result in `results/succ` or `results/errors`.
-
 ### Changelog:
 
+- 3.1.3 Update README
+- 3.1.2 Update README
 - 3.1.1 Fix readme documentation
 - 3.1.0 Support worker broadcast ('helper.emit' and 'helper.setListener') and system events ('helper.emitSys' and 'setSysListener', with built-in 'TERMINATE_ALL_WORKERS'); default numberOfWorkers is now "50%" of CPU cores (was core count minus 1)
 - 3.0.4 numberOfWorkers accepts a percentage string of CPU cores, e.g. "50%"
@@ -263,4 +257,4 @@ The usage patterns documented above are covered by automated tests (unit tests i
 
 ### License:
 
-[MIT](https://opensource.org/license/MIT) (see [LICENSE](LICENSE))
+[MIT](https://opensource.org/license/MIT) 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "multi-tasks",
-  "version": "3.1.1",
+  "version": "3.1.3",
   "description": "Multi-process task scheduling based on Node.js cluster, with crash resume and failed-task restart support",
   "main": "index.js",
   "files": [