multi-tasks 3.2.0 → 3.2.2

package/README.md

@@ -1,8 +1,6 @@
 # multi-tasks
 
-Multi-tasks is a toolkit to manage long-term and large-scale parallel computing tasks. Progress and tasks are stored on the file system, so tasks can be resumed even if the host crashes.
-
-**Zero dependencies:** multi-tasks has no runtime dependencies at all — it is built entirely on Node.js built-in modules (`cluster`, `fs`, `path`, `os`), so installing it adds nothing extra to your `node_modules`.
+Multi-tasks is a toolkit to manage long-term and large-scale parallel computing tasks, with zero dependencies. Progress and tasks are stored on the file system, so tasks can be resumed even if the host crashes.
 
 ### Install:
 
@@ -12,59 +10,23 @@ npm install multi-tasks
 
 Tip: monitor running tasks in real time — see [Real-Time Monitoring](#real-time-monitoring).
 
-### API:
-
-The API has three parts: the entry functions, the config options, and the `helper` object injected into `processTask`.
-
-**Entry functions:**
-
-- **`multiTasks(config)`** — run tasks; auto-resumes if the task folder already exists or `initialTasks` points to one.
-- **`multiTasks.resume(config)`** — usually you don't need this: the simplest way to resume is to just run `multiTasks(config)` again, it auto-resumes when the task folder already exists. This method resumes an interrupted run from the task folder alone; pass a config object with `taskFolder` (the full task folder path). The whole config (including `processTask`) is restored from `task_config.json` in that folder; any field you pass overrides the snapshot, and overrides are written back into it.
-- **`multiTasks.retry_fails(config)`** — same as `resume`, and also retries the failed tasks.
-- **`multiTasks.restart(config)`** — wipe the task folder's progress and re-run from scratch; pass a config object with `taskFolder`. The config (including `processTask` and `initialTasks`) is restored from `task_config.json`; any field you pass overrides the snapshot, and overrides are written back into it. **All previous progress and results are deleted.**
-
-**Config options:**
-
-- **`initialTasks`** — array of task objects, or a folder path string to resume from.
-- **`processTask(task, helper)`** — required; return a value or a Promise.
-- **`taskRootFolder`** — the task run folder (named by `taskId`) is created under this directory.
-- **`taskId`** — task folder name under `taskRootFolder`.
-- **`numberOfWorkers`** — how many worker processes run in parallel; a number, or a percentage string of CPU cores like `"50%"` (default).
-- **`taskTimeout`** — optional, milliseconds; an overdue task fails with a timeout error.
-- **`maxTaskRetries`** — optional, max times a failed task is auto-retried.
-- **`progressBar`** — `true` by default: shows a live single-line progress bar in the terminal (set `false` to disable); logs (including your own `console.*` calls) always go to `<taskFolder>/.sys/run.log`, and print to the console as well only when the bar is off or stdout is not a TTY.
-- **`autoCloseAfterCompletion`** — set `true` to kill workers and exit the process when no tasks are left; default is `false`, i.e. workers stay alive waiting for dynamically created tasks.
-- **`shouldTerminate(info)`** — return `true` to terminate the whole process.
-- **`setSysListener(event, payload, meta)`** — optional, handle custom system events (sent via `helper.emitSys`) on the master; `meta` has `fromWorkerId` and `fromWorkerPid`.
-- **`onFinish(report, helper)`** — called once after all workers exit (only when `autoCloseAfterCompletion` is `true`). `report` summarizes the run: `{ taskId, taskFolder, startTimestamp, endTimestamp, cost, total, succeeded, failed, failedTasks, workers }` — `cost` is the run duration in ms (a resumed run measures only the current round); `failedTasks` lists the failed tasks' subids; `workers` is `{ count, crashed }` — the configured worker count and how many workers crashed and were replaced. `helper` lets you walk every persisted task result:
-  - **`helper.foreachSuccResult((count, task, result) => {})`** — iterate all succeeded results (`results/succ`); `count` is a 0-based index, `task` is the original task object (`null` if its task file is missing or unparseable), `result` is the value returned by `processTask` (`undefined` when the task returned nothing).
-  - **`helper.foreachErrorResult((count, task, err) => {})`** — iterate all failed results (`results/errors`); `err` is the persisted error object as-is (`{ex}` / `{subTask, exception}` / `{type: 'timeout', ...}` / `{type: 'worker_crash', ...}`).
-  - Iteration follows filesystem order (not sorted). `onFinish` runs synchronously at the very end and the master process exits shortly after it returns — don't `await` inside it.
-
-**Other options:**
-
-- **`taskFolder`** — full task folder path; only used by `resume`/`retry_fails`/`restart`.
-
-**Task helper** (the `helper` object passed as the second argument of `processTask`):
-
-- **`helper.createNewTasks(tasks)`** — create new tasks dynamically while processing.
-- **`helper.emit(event, payload, option?)`** — broadcast an event to all workers (relayed by the master, best-effort); `option.includingMe` defaults to `true` — pass `{includingMe: false}` to exclude the sender. Payload must be JSON-serializable.
-- **`helper.setListener(event, handler)`** — listen for broadcast events; one handler per event per worker process (re-registering replaces it), so calling it inside `processTask` is always safe; `handler(payload, meta)` gets `meta.fromWorkerId` and `meta.fromWorkerPid`. Broadcasts are runtime-only messages — not persisted, not replayed on resume.
-- **`helper.emitSys(event, payload?)`** — send a system event to the master (consumed by the master itself, not relayed). Built-in event: `'TERMINATE_ALL_WORKERS'` force-kills all workers and exits the master; any other event goes to `config.setSysListener`.
-
 ### How to use:
 
 ```javascript
 //see examples/example0
 let multiTasks = require('multi-tasks').multiTasks;
 
-//Step1, create the tasks to run in parallel as an array.
-let alltasks = [];
-for(let i=0;i<50;i++){
-    alltasks.push({
-        name: `task-${i}`,
-        data: `This prop is for a subtask`
-    });
+//Step1, provide the tasks as a function returning them (or a Promise of them);
+//it runs only once, in the master process.
+let initialTasks = async ()=>{
+    let alltasks = [];
+    for(let i=0;i<50;i++){
+        alltasks.push({
+            name: `task-${i}`,
+            data: `This prop is for a subtask`
+        });
+    };
+    return alltasks;
 };
 
 //Step2, provide a function that processes each sub-task and returns the result
@@ -79,7 +41,7 @@ let processTask = (task, helper)=>{
 
 //Step3, run!
 multiTasks({
-    initialTasks: alltasks, 
+    initialTasks, //accepts array, function, or function returning a promise
     processTask, 
     taskRootFolder: `../examples-tmp-data/example0`, //a directory to store progress and result files; you can check the progress here
     taskId: 'my-task',
@@ -215,6 +177,27 @@ multiTasks({
 
 ```
 
+### Performance: runs only once, in the master process
+
+Every worker process re-runs your entire entry script — so top-level task-generation code (building a huge array, scanning folders, querying a database, ...) runs once per worker. Wrap it in a function for performance: it is called **only once, in the master process**, and workers never call it. `initialTasks` accepts three forms — pick one:
+
+```javascript
+
+//Example8, the three forms of initialTasks
+
+//form 1: a plain array — assign it directly
+multiTasks({ initialTasks: alltasks, processTask, taskRootFolder, taskId });
+
+//form 2: a function returning the array
+multiTasks({ initialTasks: () => fetchTasksFromSomewhere(), processTask, taskRootFolder, taskId });
+
+//form 3: a function returning a Promise of the array
+multiTasks({ initialTasks: async () => await fetchTasksFromDb(), processTask, taskRootFolder, taskId });
+
+```
+
+The (resolved) value must be an array, otherwise the master exits with an error. The resulting array — not the function — is stored in `task_config.json`, so the self-contained-function note under Resuming does not apply to it.
+
 ### Resuming
 
 If the execution is interrupted (e.g. a power outage), the simplest way to resume is to just re-run the same code — `multiTasks(config)` detects the existing task folder and auto-resumes, re-running only the stuck tasks:
@@ -288,8 +271,50 @@ Quick start (requires multi-tasks >=3.2.0): once your multi-tasks run has starte
 
 For installation, command-line options, and more, see the [multi-tasks-monitor](https://www.npmjs.com/package/multi-tasks-monitor) page.
 
+### API:
+
+The API has three parts: the entry functions, the config options, and the `helper` object injected into `processTask`.
+
+**Entry functions:**
+
+- **`multiTasks(config)`** — run tasks; auto-resumes if the task folder already exists or `initialTasks` points to one.
+- **`multiTasks.resume(config)`** — usually you don't need this: the simplest way to resume is to just run `multiTasks(config)` again, it auto-resumes when the task folder already exists. This method resumes an interrupted run from the task folder alone; pass a config object with `taskFolder` (the full task folder path). The whole config (including `processTask`) is restored from `task_config.json` in that folder; any field you pass overrides the snapshot, and overrides are written back into it.
+- **`multiTasks.retry_fails(config)`** — same as `resume`, and also retries the failed tasks.
+- **`multiTasks.restart(config)`** — wipe the task folder's progress and re-run from scratch; pass a config object with `taskFolder`. The config (including `processTask` and `initialTasks`) is restored from `task_config.json`; any field you pass overrides the snapshot, and overrides are written back into it. **All previous progress and results are deleted.**
+
+**Config options:**
+
+- **`initialTasks`** — array of task objects, a function returning the array (or a Promise of it — called only once, in the master process; worker processes never call it; see [Performance: runs only once, in the master process](#performance-runs-only-once-in-the-master-process)), or a folder path string to resume from.
+- **`processTask(task, helper)`** — required; return a value or a Promise.
+- **`taskRootFolder`** — the task run folder (named by `taskId`) is created under this directory.
+- **`taskId`** — task folder name under `taskRootFolder`.
+- **`numberOfWorkers`** — how many worker processes run in parallel; a number, or a percentage string of CPU cores like `"50%"` (default).
+- **`taskTimeout`** — optional, milliseconds; an overdue task fails with a timeout error.
+- **`maxTaskRetries`** — optional, max times a failed task is auto-retried.
+- **`progressBar`** — `true` by default: shows a live single-line progress bar in the terminal (set `false` to disable); logs (including your own `console.*` calls) always go to `<taskFolder>/.sys/run.log`, and print to the console as well only when the bar is off or stdout is not a TTY.
+- **`autoCloseAfterCompletion`** — set `true` to kill workers and exit the process when no tasks are left; default is `false`, i.e. workers stay alive waiting for dynamically created tasks.
+- **`shouldTerminate(info)`** — return `true` to terminate the whole process.
+- **`setSysListener(event, payload, meta)`** — optional, handle custom system events (sent via `helper.emitSys`) on the master; `meta` has `fromWorkerId` and `fromWorkerPid`.
+- **`onFinish(report, helper)`** — called once after all workers exit (only when `autoCloseAfterCompletion` is `true`). `report` summarizes the run: `{ taskId, taskFolder, startTimestamp, endTimestamp, cost, total, succeeded, failed, failedTasks, workers }` — `cost` is the run duration in ms (a resumed run measures only the current round); `failedTasks` lists the failed tasks' subids; `workers` is `{ count, crashed }` — the configured worker count and how many workers crashed and were replaced. `helper` lets you walk every persisted task result:
+  - **`helper.foreachSuccResult((count, task, result) => {})`** — iterate all succeeded results (`results/succ`); `count` is a 0-based index, `task` is the original task object (`null` if its task file is missing or unparseable), `result` is the value returned by `processTask` (`undefined` when the task returned nothing).
+  - **`helper.foreachErrorResult((count, task, err) => {})`** — iterate all failed results (`results/errors`); `err` is the persisted error object as-is (`{ex}` / `{subTask, exception}` / `{type: 'timeout', ...}` / `{type: 'worker_crash', ...}`).
+  - Iteration follows filesystem order (not sorted). `onFinish` runs synchronously at the very end and the master process exits shortly after it returns — don't `await` inside it.
+
+**Other options:**
+
+- **`taskFolder`** — full task folder path; only used by `resume`/`retry_fails`/`restart`.
+
+**Task helper** (the `helper` object passed as the second argument of `processTask`):
+
+- **`helper.createNewTasks(tasks)`** — create new tasks dynamically while processing.
+- **`helper.emit(event, payload, option?)`** — broadcast an event to all workers (relayed by the master, best-effort); `option.includingMe` defaults to `true` — pass `{includingMe: false}` to exclude the sender. Payload must be JSON-serializable.
+- **`helper.setListener(event, handler)`** — listen for broadcast events; one handler per event per worker process (re-registering replaces it), so calling it inside `processTask` is always safe; `handler(payload, meta)` gets `meta.fromWorkerId` and `meta.fromWorkerPid`. Broadcasts are runtime-only messages — not persisted, not replayed on resume.
+- **`helper.emitSys(event, payload?)`** — send a system event to the master (consumed by the master itself, not relayed). Built-in event: `'TERMINATE_ALL_WORKERS'` force-kills all workers and exits the master; any other event goes to `config.setSysListener`.
+
 ### Changelog:
 
+- 3.2.2 initialTasks accepts a function (sync or async) that is called only once, in the master process
+- 3.2.1 Fix retry bug: user-defined taskCount/index no longer overwritten on task retries
 - 3.2.0 Add progressBar (on by default); console output diverted to run.log; new startup banner
 - 3.1.9 Print the multi-tasks-monitor command on startup
 - 3.1.8 Small refinements

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "multi-tasks",
-  "version": "3.2.0",
+  "version": "3.2.2",
   "description": "Multi-process task scheduling based on Node.js cluster, with crash resume and failed-task restart support",
   "main": "index.js",
   "files": [

package/workers/Multithread.js

@@ -341,8 +341,10 @@ function subWorker(config){
         if(message.subTask){//new task
             let {config, subTask, startTimestamp, taskCount} = message;
             TaskMgr.load(config);
-            subTask.index = taskCount;//for v1 backward
-            subTask.taskCount = taskCount;
+            //用户任务自带的 index/taskCount 保留原值(重试/续跑后任务身份不变),
+            //未提供时才填全局派发计数器(v1 兼容)
+            if(typeof subTask.index === 'undefined') subTask.index = taskCount;
+            if(typeof subTask.taskCount === 'undefined') subTask.taskCount = taskCount;
             subTask.workerId = worker.id;
             subTask.workerPid = process.pid;
             runLog.write(`[Task Start]: "${subTask.subid}"`, `pid=${process.pid}`, getDateTimeTxt());
@@ -353,6 +355,7 @@ function subWorker(config){
                 startTimestamp
             };
             runTask(config, subTask, subTaskLog).then((tasklog)=>{
+                if(tasklog.retried) return;//重试任务已回 new 等重跑, 未完成, 不打 [Task Finished]
                 let {cost, startTimestamp, endTimestamp} = tasklog;
 
                 runLog.write(`[Task Finished] "${config.masterTaskId}"."${subTask.subid}".`, getDateTimeTxt(), `cost=${cost}ms`)
@@ -422,7 +425,8 @@ function runTask(config, subTask, subTaskLog){
                 resolve({
                     startTimestamp,
                     endTimestamp,
-                    cost
+                    cost,
+                    retried: true
                 });
                 return;
             };
@@ -434,7 +438,8 @@ function runTask(config, subTask, subTaskLog){
                 resolve({
                     startTimestamp,
                     endTimestamp,
-                    cost
+                    cost,
+                    retried: false
                 });
             });   
         });

package/workers/index.js

@@ -1,5 +1,6 @@
 const fs = require('fs');
 const pathUtil = require('path');
+const cluster = require('cluster');
 const Multithread = require('./Multithread');
 const backwardv1 = require('../utils/backward-v1');
 const { FUNCTION_CONFIG_KEYS, reviveConfig } = require('../utils/serializeFns');
@@ -52,19 +53,58 @@ const _computeSysLogFile = (config)=>{
     };
 };
 
+//initialTasks 支持函数形式(惰性生成): 只在 master 进程求值一次, worker 进程重跑入口脚本时跳过,
+//防生成逻辑(扫目录/查库等初始化脚本)在每个 worker 里各执行一遍。
+//函数可同步返回数组, 也可返回 Promise(resolve 值同样必须是数组)——同步路径保持同步 throw,
+//Promise 路径返回 then 链交给 _proceedAfterEval 等待。求值在入口统一完成(先于 _mergeSnapshotConfig
+//与 Multithread.start): 求值后的数组才会参与合并并写入 task_config.json(函数本身会被
+//JSON.stringify 丢弃, 不进存档、不参与 revive)。
+const _evalInitialTasks = (config)=>{
+    if(typeof config.initialTasks !== 'function') return null;
+    if(!(cluster.isPrimary || cluster.isMaster)) return null;
+    let tasks = config.initialTasks();
+    let applyTasks = (resolved)=>{
+        if(!Array.isArray(resolved)){
+            throw new Error('FATAL: initialTasks function should return an array!');
+        };
+        config.initialTasks = resolved;
+    };
+    if(tasks && typeof tasks.then === 'function'){//Promise 形式: 异步求值, resolve 后同样校验数组(假设真 Promise, 即 .then 返回 Promise; 非标准 thenable 不支持)
+        return tasks.then(applyTasks);
+    };
+    applyTasks(tasks);
+    return null;
+};
+
+//同步路径(pending 为 null)直接执行 proceed, 保持既有同步 throw 语义;
+//Promise 路径等 resolve 后执行, resolve 非数组/reject/proceed 抛错时打 FATAL 并 exit(1)
+//(与 init 异步失败一致; Promise 路径只发生在 master, 无同步 throw 对象)。
+const _proceedAfterEval = (pending, proceed)=>{
+    if(!pending){
+        proceed();
+        return;
+    };
+    return pending.then(proceed).catch((e)=>{
+        console.error('[FATAL] failed to evaluate initialTasks:', e);
+        process.exit(1);
+    });
+};
+
 const start = (config)=>{
     config = backwardv1.v1tov2(config);
+    return _proceedAfterEval(_evalInitialTasks(config), ()=>{
 
-    config.taskRootFolder = _toAbsPath(config.taskRootFolder);
-    config.taskFolder = _toAbsPath(config.taskFolder);
-    _computeSysLogFile(config);
+        config.taskRootFolder = _toAbsPath(config.taskRootFolder);
+        config.taskFolder = _toAbsPath(config.taskFolder);
+        _computeSysLogFile(config);
 
-    Multithread.start(config);
+        Multithread.start(config);
+    });
 };
 const resume = (config)=>{
     config = _normalizeResumeArg('resume', config);
-    config.__resume = true;
-
+    //先校验/推导 taskFolder 再求值 initialTasks(与 retry_fails/restart 一致):
+    //缺目录时同步 throw fail fast, 不让可能昂贵的生成逻辑白跑一遍
     config.taskRootFolder = _toAbsPath(config.taskRootFolder);
     config.taskFolder = _toAbsPath(config.taskFolder);
     if(!config.taskFolder){
@@ -75,10 +115,12 @@ const resume = (config)=>{
     };
     config.taskRootFolder = pathUtil.dirname(config.taskFolder);
     config.taskId = pathUtil.basename(config.taskFolder);
-    _computeSysLogFile(config);
-
-    config = _mergeSnapshotConfig(config);
-    Multithread.start(config);
+    config.__resume = true;
+    return _proceedAfterEval(_evalInitialTasks(config), ()=>{
+        _computeSysLogFile(config);
+        config = _mergeSnapshotConfig(config);
+        Multithread.start(config);
+    });
 };
 const retry_fails = (config)=>{
     config = _normalizeResumeArg('retry_fails', config);
@@ -91,9 +133,11 @@ const retry_fails = (config)=>{
     config.taskId = pathUtil.basename(taskFolder);
     config.__resume = true;
     config.__retryFails = true;
-    _computeSysLogFile(config);
-    config = _mergeSnapshotConfig(config);
-    Multithread.start(config);
+    return _proceedAfterEval(_evalInitialTasks(config), ()=>{
+        _computeSysLogFile(config);
+        config = _mergeSnapshotConfig(config);
+        Multithread.start(config);
+    });
 };
 const restart = (config)=>{
     config = _normalizeResumeArg('restart', config);
@@ -105,9 +149,11 @@ const restart = (config)=>{
     config.taskRootFolder = pathUtil.dirname(taskFolder);
     config.taskId = pathUtil.basename(taskFolder);
     config.__restart = true;
-    _computeSysLogFile(config);
-    config = _mergeSnapshotConfig(config);
-    Multithread.start(config);
+    return _proceedAfterEval(_evalInitialTasks(config), ()=>{
+        _computeSysLogFile(config);
+        config = _mergeSnapshotConfig(config);
+        Multithread.start(config);
+    });
 };
 
 const multiTasks = (config)=>{
@@ -126,7 +172,7 @@ const multiTasks = (config)=>{
                 config.taskRootFolder = pathUtil.dirname(folder);
             }
         }
-        if(!isResume) throw new Error('FATAL: initialTasks should be an array or a task root folder!');
+        if(!isResume) throw new Error('FATAL: initialTasks should be an array, a function or a task root folder!');
     }else{
         let taskFolder = pathUtil.resolve(taskRootFolder, taskId);
         if(fs.existsSync(taskFolder)){
@@ -135,9 +181,9 @@ const multiTasks = (config)=>{
     }
 
     if(isResume){
-        resume(config);
+        return resume(config);
     }else{
-        start(config);
+        return start(config);
     }
 
 };