multi-tasks 3.2.0 → 3.2.1

package/README.md

@@ -12,46 +12,6 @@ npm install multi-tasks
 
 Tip: monitor running tasks in real time — see [Real-Time Monitoring](#real-time-monitoring).
 
-### API:
-
-The API has three parts: the entry functions, the config options, and the `helper` object injected into `processTask`.
-
-**Entry functions:**
-
-- **`multiTasks(config)`** — run tasks; auto-resumes if the task folder already exists or `initialTasks` points to one.
-- **`multiTasks.resume(config)`** — usually you don't need this: the simplest way to resume is to just run `multiTasks(config)` again, it auto-resumes when the task folder already exists. This method resumes an interrupted run from the task folder alone; pass a config object with `taskFolder` (the full task folder path). The whole config (including `processTask`) is restored from `task_config.json` in that folder; any field you pass overrides the snapshot, and overrides are written back into it.
-- **`multiTasks.retry_fails(config)`** — same as `resume`, and also retries the failed tasks.
-- **`multiTasks.restart(config)`** — wipe the task folder's progress and re-run from scratch; pass a config object with `taskFolder`. The config (including `processTask` and `initialTasks`) is restored from `task_config.json`; any field you pass overrides the snapshot, and overrides are written back into it. **All previous progress and results are deleted.**
-
-**Config options:**
-
-- **`initialTasks`** — array of task objects, or a folder path string to resume from.
-- **`processTask(task, helper)`** — required; return a value or a Promise.
-- **`taskRootFolder`** — the task run folder (named by `taskId`) is created under this directory.
-- **`taskId`** — task folder name under `taskRootFolder`.
-- **`numberOfWorkers`** — how many worker processes run in parallel; a number, or a percentage string of CPU cores like `"50%"` (default).
-- **`taskTimeout`** — optional, milliseconds; an overdue task fails with a timeout error.
-- **`maxTaskRetries`** — optional, max times a failed task is auto-retried.
-- **`progressBar`** — `true` by default: shows a live single-line progress bar in the terminal (set `false` to disable); logs (including your own `console.*` calls) always go to `<taskFolder>/.sys/run.log`, and print to the console as well only when the bar is off or stdout is not a TTY.
-- **`autoCloseAfterCompletion`** — set `true` to kill workers and exit the process when no tasks are left; default is `false`, i.e. workers stay alive waiting for dynamically created tasks.
-- **`shouldTerminate(info)`** — return `true` to terminate the whole process.
-- **`setSysListener(event, payload, meta)`** — optional, handle custom system events (sent via `helper.emitSys`) on the master; `meta` has `fromWorkerId` and `fromWorkerPid`.
-- **`onFinish(report, helper)`** — called once after all workers exit (only when `autoCloseAfterCompletion` is `true`). `report` summarizes the run: `{ taskId, taskFolder, startTimestamp, endTimestamp, cost, total, succeeded, failed, failedTasks, workers }` — `cost` is the run duration in ms (a resumed run measures only the current round); `failedTasks` lists the failed tasks' subids; `workers` is `{ count, crashed }` — the configured worker count and how many workers crashed and were replaced. `helper` lets you walk every persisted task result:
-  - **`helper.foreachSuccResult((count, task, result) => {})`** — iterate all succeeded results (`results/succ`); `count` is a 0-based index, `task` is the original task object (`null` if its task file is missing or unparseable), `result` is the value returned by `processTask` (`undefined` when the task returned nothing).
-  - **`helper.foreachErrorResult((count, task, err) => {})`** — iterate all failed results (`results/errors`); `err` is the persisted error object as-is (`{ex}` / `{subTask, exception}` / `{type: 'timeout', ...}` / `{type: 'worker_crash', ...}`).
-  - Iteration follows filesystem order (not sorted). `onFinish` runs synchronously at the very end and the master process exits shortly after it returns — don't `await` inside it.
-
-**Other options:**
-
-- **`taskFolder`** — full task folder path; only used by `resume`/`retry_fails`/`restart`.
-
-**Task helper** (the `helper` object passed as the second argument of `processTask`):
-
-- **`helper.createNewTasks(tasks)`** — create new tasks dynamically while processing.
-- **`helper.emit(event, payload, option?)`** — broadcast an event to all workers (relayed by the master, best-effort); `option.includingMe` defaults to `true` — pass `{includingMe: false}` to exclude the sender. Payload must be JSON-serializable.
-- **`helper.setListener(event, handler)`** — listen for broadcast events; one handler per event per worker process (re-registering replaces it), so calling it inside `processTask` is always safe; `handler(payload, meta)` gets `meta.fromWorkerId` and `meta.fromWorkerPid`. Broadcasts are runtime-only messages — not persisted, not replayed on resume.
-- **`helper.emitSys(event, payload?)`** — send a system event to the master (consumed by the master itself, not relayed). Built-in event: `'TERMINATE_ALL_WORKERS'` force-kills all workers and exits the master; any other event goes to `config.setSysListener`.
-
 ### How to use:
 
 ```javascript
@@ -288,8 +248,49 @@ Quick start (requires multi-tasks >=3.2.0): once your multi-tasks run has starte
 
 For installation, command-line options, and more, see the [multi-tasks-monitor](https://www.npmjs.com/package/multi-tasks-monitor) page.
 
+### API:
+
+The API has three parts: the entry functions, the config options, and the `helper` object injected into `processTask`.
+
+**Entry functions:**
+
+- **`multiTasks(config)`** — run tasks; auto-resumes if the task folder already exists or `initialTasks` points to one.
+- **`multiTasks.resume(config)`** — usually you don't need this: the simplest way to resume is to just run `multiTasks(config)` again, it auto-resumes when the task folder already exists. This method resumes an interrupted run from the task folder alone; pass a config object with `taskFolder` (the full task folder path). The whole config (including `processTask`) is restored from `task_config.json` in that folder; any field you pass overrides the snapshot, and overrides are written back into it.
+- **`multiTasks.retry_fails(config)`** — same as `resume`, and also retries the failed tasks.
+- **`multiTasks.restart(config)`** — wipe the task folder's progress and re-run from scratch; pass a config object with `taskFolder`. The config (including `processTask` and `initialTasks`) is restored from `task_config.json`; any field you pass overrides the snapshot, and overrides are written back into it. **All previous progress and results are deleted.**
+
+**Config options:**
+
+- **`initialTasks`** — array of task objects, or a folder path string to resume from.
+- **`processTask(task, helper)`** — required; return a value or a Promise.
+- **`taskRootFolder`** — the task run folder (named by `taskId`) is created under this directory.
+- **`taskId`** — task folder name under `taskRootFolder`.
+- **`numberOfWorkers`** — how many worker processes run in parallel; a number, or a percentage string of CPU cores like `"50%"` (default).
+- **`taskTimeout`** — optional, milliseconds; an overdue task fails with a timeout error.
+- **`maxTaskRetries`** — optional, max times a failed task is auto-retried.
+- **`progressBar`** — `true` by default: shows a live single-line progress bar in the terminal (set `false` to disable); logs (including your own `console.*` calls) always go to `<taskFolder>/.sys/run.log`, and print to the console as well only when the bar is off or stdout is not a TTY.
+- **`autoCloseAfterCompletion`** — set `true` to kill workers and exit the process when no tasks are left; default is `false`, i.e. workers stay alive waiting for dynamically created tasks.
+- **`shouldTerminate(info)`** — return `true` to terminate the whole process.
+- **`setSysListener(event, payload, meta)`** — optional, handle custom system events (sent via `helper.emitSys`) on the master; `meta` has `fromWorkerId` and `fromWorkerPid`.
+- **`onFinish(report, helper)`** — called once after all workers exit (only when `autoCloseAfterCompletion` is `true`). `report` summarizes the run: `{ taskId, taskFolder, startTimestamp, endTimestamp, cost, total, succeeded, failed, failedTasks, workers }` — `cost` is the run duration in ms (a resumed run measures only the current round); `failedTasks` lists the failed tasks' subids; `workers` is `{ count, crashed }` — the configured worker count and how many workers crashed and were replaced. `helper` lets you walk every persisted task result:
+  - **`helper.foreachSuccResult((count, task, result) => {})`** — iterate all succeeded results (`results/succ`); `count` is a 0-based index, `task` is the original task object (`null` if its task file is missing or unparseable), `result` is the value returned by `processTask` (`undefined` when the task returned nothing).
+  - **`helper.foreachErrorResult((count, task, err) => {})`** — iterate all failed results (`results/errors`); `err` is the persisted error object as-is (`{ex}` / `{subTask, exception}` / `{type: 'timeout', ...}` / `{type: 'worker_crash', ...}`).
+  - Iteration follows filesystem order (not sorted). `onFinish` runs synchronously at the very end and the master process exits shortly after it returns — don't `await` inside it.
+
+**Other options:**
+
+- **`taskFolder`** — full task folder path; only used by `resume`/`retry_fails`/`restart`.
+
+**Task helper** (the `helper` object passed as the second argument of `processTask`):
+
+- **`helper.createNewTasks(tasks)`** — create new tasks dynamically while processing.
+- **`helper.emit(event, payload, option?)`** — broadcast an event to all workers (relayed by the master, best-effort); `option.includingMe` defaults to `true` — pass `{includingMe: false}` to exclude the sender. Payload must be JSON-serializable.
+- **`helper.setListener(event, handler)`** — listen for broadcast events; one handler per event per worker process (re-registering replaces it), so calling it inside `processTask` is always safe; `handler(payload, meta)` gets `meta.fromWorkerId` and `meta.fromWorkerPid`. Broadcasts are runtime-only messages — not persisted, not replayed on resume.
+- **`helper.emitSys(event, payload?)`** — send a system event to the master (consumed by the master itself, not relayed). Built-in event: `'TERMINATE_ALL_WORKERS'` force-kills all workers and exits the master; any other event goes to `config.setSysListener`.
+
 ### Changelog:
 
+- 3.2.1 Fix retry bug: user-defined taskCount/index no longer overwritten on task retries
 - 3.2.0 Add progressBar (on by default); console output diverted to run.log; new startup banner
 - 3.1.9 Print the multi-tasks-monitor command on startup
 - 3.1.8 Small refinements

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "multi-tasks",
-  "version": "3.2.0",
+  "version": "3.2.1",
   "description": "Multi-process task scheduling based on Node.js cluster, with crash resume and failed-task restart support",
   "main": "index.js",
   "files": [

package/workers/Multithread.js

@@ -341,8 +341,10 @@ function subWorker(config){
         if(message.subTask){//new task
             let {config, subTask, startTimestamp, taskCount} = message;
             TaskMgr.load(config);
-            subTask.index = taskCount;//for v1 backward
-            subTask.taskCount = taskCount;
+            //用户任务自带的 index/taskCount 保留原值(重试/续跑后任务身份不变),
+            //未提供时才填全局派发计数器(v1 兼容)
+            if(typeof subTask.index === 'undefined') subTask.index = taskCount;
+            if(typeof subTask.taskCount === 'undefined') subTask.taskCount = taskCount;
             subTask.workerId = worker.id;
             subTask.workerPid = process.pid;
             runLog.write(`[Task Start]: "${subTask.subid}"`, `pid=${process.pid}`, getDateTimeTxt());
@@ -353,6 +355,7 @@ function subWorker(config){
                 startTimestamp
             };
             runTask(config, subTask, subTaskLog).then((tasklog)=>{
+                if(tasklog.retried) return;//重试任务已回 new 等重跑, 未完成, 不打 [Task Finished]
                 let {cost, startTimestamp, endTimestamp} = tasklog;
 
                 runLog.write(`[Task Finished] "${config.masterTaskId}"."${subTask.subid}".`, getDateTimeTxt(), `cost=${cost}ms`)
@@ -422,7 +425,8 @@ function runTask(config, subTask, subTaskLog){
                 resolve({
                     startTimestamp,
                     endTimestamp,
-                    cost
+                    cost,
+                    retried: true
                 });
                 return;
             };
@@ -434,7 +438,8 @@ function runTask(config, subTask, subTaskLog){
                 resolve({
                     startTimestamp,
                     endTimestamp,
-                    cost
+                    cost,
+                    retried: false
                 });
             });   
         });