@yemi33/minions 0.1.2448 → 0.1.2449
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bin/cli-api-client.js +1 -1
- package/bin/install-internal-minions.js +1382 -44
- package/bin/install-layout.js +150 -0
- package/bin/minions.js +460 -167
- package/dashboard/docs/typography.md +65 -12
- package/dashboard/js/command-center.js +58 -4
- package/dashboard/js/detail-panel.js +36 -0
- package/dashboard/js/memory-panel.js +59 -12
- package/dashboard/js/qa.js +179 -20
- package/dashboard/js/refresh.js +148 -12
- package/dashboard/js/render-dispatch.js +3 -4
- package/dashboard/js/render-inbox.js +2 -2
- package/dashboard/js/render-other.js +3 -3
- package/dashboard/js/render-pipelines.js +14 -0
- package/dashboard/js/render-plans.js +57 -9
- package/dashboard/js/render-prd.js +132 -23
- package/dashboard/js/render-prs.js +195 -166
- package/dashboard/js/render-schedules.js +63 -3
- package/dashboard/js/render-utils.js +3 -3
- package/dashboard/js/render-watches.js +19 -3
- package/dashboard/js/render-work-items.js +238 -30
- package/dashboard/js/settings.js +205 -54
- package/dashboard/js/utils.js +51 -1
- package/dashboard/pages/home.html +1 -1
- package/dashboard/pages/qa.html +1 -16
- package/dashboard/pages/work.html +40 -0
- package/dashboard/shared/cc-limits.js +79 -0
- package/dashboard/shared/pr-filters.js +21 -38
- package/dashboard/shared/project-git-summary.js +1 -1
- package/dashboard/shared/record-filters.js +169 -0
- package/dashboard/shared/watches-source.js +1 -1
- package/dashboard/shared/welcome-popup.js +1 -1
- package/dashboard/shared/wi-filters.js +302 -0
- package/dashboard/slim/body.html +1 -0
- package/dashboard/slim/js/command-send.js +26 -0
- package/dashboard/slim/js/modals-tiles.js +380 -39
- package/dashboard/slim/js/status.js +13 -21
- package/dashboard/slim/layout.html +1 -0
- package/dashboard/slim/panel-bootstrap.js +6 -2
- package/dashboard/slim/styles.css +38 -0
- package/dashboard/styles.css +159 -55
- package/dashboard-build.js +13 -2
- package/dashboard.js +956 -423
- package/docs/README.md +11 -6
- package/docs/api-errors.md +2 -2
- package/docs/architecture-review-2026-07-09.md +1 -1
- package/docs/architecture.excalidraw +2 -2
- package/docs/auto-discovery.md +18 -9
- package/docs/branch-derivation.md +4 -4
- package/docs/capture-demos.js +39 -2
- package/docs/ci-runner-canary.md +123 -0
- package/docs/claude-md-propagation.md +2 -2
- package/docs/cloud-agent-dispatch.md +204 -0
- package/docs/command-center.md +7 -7
- package/docs/completion-reports.md +43 -20
- package/docs/constants.md +10 -3
- package/docs/constellation-bridge.md +134 -6
- package/docs/constellation-style-telemetry.md +4 -4
- package/docs/contracts/capability-protocol.v1.json +165 -0
- package/docs/cooldown-merge-semantics.md +12 -12
- package/docs/copilot-cli-schema.md +7 -7
- package/docs/cross-repo-plans.md +10 -10
- package/docs/dead-code-audit-retractions.md +5 -5
- package/docs/default-branch-ci.md +173 -0
- package/docs/deprecated.json +31 -31
- package/docs/design-inbox-entries-schema.md +3 -3
- package/docs/design-language.md +1051 -0
- package/docs/design-state-storage.md +11 -11
- package/docs/diagnostics-crash-reports.md +9 -9
- package/docs/diagnostics-memory.md +5 -5
- package/docs/documentation-audit-2026-07-09.md +7 -7
- package/docs/engine-restart.md +90 -5
- package/docs/harness-mode.md +1 -1
- package/docs/internal-install.md +338 -39
- package/docs/kb-dedup-duplicate-pair-investigation.md +5 -5
- package/docs/kb-pr3223-cascade-archiving.md +1 -1
- package/docs/kb-pr696-merge-conflict-docs.md +6 -6
- package/docs/kb-sweep.md +35 -35
- package/docs/keep-processes.md +1 -1
- package/docs/live-checkout-mode.md +30 -30
- package/docs/managed-spawn.md +18 -14
- package/docs/named-agents.md +7 -7
- package/docs/plan-lifecycle.md +69 -2
- package/docs/pr-author-identity.md +114 -0
- package/docs/pr-auto-fix-dispatch.md +19 -4
- package/docs/pr-comment-followup.md +6 -6
- package/docs/pr-review-fix-loop.md +59 -10
- package/docs/process-termination.md +40 -0
- package/docs/proposals/repo-pool-for-live-checkout.md +13 -13
- package/docs/qa-runbook-lifecycle.md +367 -17
- package/docs/qa-runbooks.md +3 -3
- package/docs/rfc-completion-json.md +18 -18
- package/docs/runtime-adapters.md +26 -21
- package/docs/security.md +6 -6
- package/docs/self-improvement.md +4 -4
- package/docs/shared-lifecycle-module-map.md +473 -472
- package/docs/skills.md +52 -3
- package/docs/slim-ux/concepts.md +121 -116
- package/docs/specs/agent-configurability.md +18 -18
- package/docs/specs/agent-rename.md +18 -18
- package/docs/team-memory.md +38 -21
- package/docs/timeouts-and-liveness.md +118 -10
- package/docs/tutorials/01-install-and-connect.md +1 -1
- package/docs/watches.md +40 -39
- package/docs/workspace-manifests.md +4 -4
- package/docs/worktree-lifecycle.md +293 -14
- package/engine/README.md +46 -0
- package/engine/{ado-comment.js → ado/comment.js} +8 -8
- package/engine/{ado-git-auth.js → ado/git-auth.js} +4 -4
- package/engine/{ado.js → ado/index.js} +417 -63
- package/engine/{ado-status.js → ado/status.js} +6 -8
- package/engine/{ado-token.js → ado/token.js} +1 -1
- package/engine/{acp-transport.js → agents/acp-transport.js} +62 -22
- package/engine/{agent-worker-pool.js → agents/agent-worker-pool.js} +17 -8
- package/engine/{cc-worker-pool.js → agents/cc-worker-pool.js} +16 -6
- package/engine/{claude-md-context.js → agents/claude-md-context.js} +5 -5
- package/engine/{harness-context.js → agents/harness-context.js} +5 -5
- package/engine/{harness.js → agents/harness.js} +3 -3
- package/engine/{llm.js → agents/llm.js} +18 -14
- package/engine/{model-discovery.js → agents/model-discovery.js} +2 -2
- package/engine/{playbook.js → agents/playbook.js} +155 -22
- package/engine/{pooled-agent-process.js → agents/pooled-agent-process.js} +14 -12
- package/engine/{preflight.js → agents/preflight.js} +29 -10
- package/engine/{spawn-agent.js → agents/spawn-agent.js} +25 -14
- package/engine/{spawn-phase-watchdog.js → agents/spawn-phase-watchdog.js} +16 -7
- package/engine/{steering.js → agents/steering.js} +5 -5
- package/engine/{tools-inventory.js → agents/tools-inventory.js} +2 -2
- package/engine/{agent-api-validation.js → api/agent-api-validation.js} +2 -2
- package/engine/{api-validation.js → api/api-validation.js} +1 -1
- package/engine/api/bridge.js +787 -0
- package/engine/{cc-api-validation.js → api/cc-api-validation.js} +1 -1
- package/engine/api/companion.js +560 -0
- package/engine/{content-api-validation.js → api/content-api-validation.js} +2 -2
- package/engine/{pr-issue-validation.js → api/pr-issue-validation.js} +33 -6
- package/engine/{settings-validation.js → api/settings-validation.js} +32 -4
- package/engine/api-contracts/agent-content.js +4 -4
- package/engine/api-contracts/capability-manifest.js +236 -0
- package/engine/api-contracts/capability-protocol.js +333 -0
- package/engine/api-contracts/cc-ops.js +1 -1
- package/engine/api-contracts/config-runtime.js +5 -0
- package/engine/api-contracts/core.js +28 -1
- package/engine/api-contracts/index.js +100 -0
- package/engine/api-contracts/orchestration.js +18 -5
- package/engine/api-contracts/pull-requests.js +37 -6
- package/engine/api-contracts/qa-process.js +29 -6
- package/engine/api-contracts/work-plan-prd.js +21 -1
- package/engine/cloud/contract.js +212 -0
- package/engine/cloud/index.js +159 -0
- package/engine/{execution-model.js → core/execution-model.js} +1 -1
- package/engine/{features.js → core/features.js} +4 -4
- package/engine/{operator-identity.js → core/operator-identity.js} +1 -1
- package/engine/{queries.js → core/queries.js} +201 -36
- package/engine/{safe-expr.js → core/safe-expr.js} +1 -1
- package/engine/{shared.js → core/shared.js} +1637 -175
- package/engine/{stdio-timestamps.js → core/stdio-timestamps.js} +1 -1
- package/engine/{untrusted-fence.js → core/untrusted-fence.js} +3 -3
- package/engine/db/index.js +11 -2
- package/engine/db/migrations/002-dispatches.js +3 -3
- package/engine/db/migrations/003-work-items.js +1 -1
- package/engine/db/migrations/004-pull-requests.js +1 -1
- package/engine/db/migrations/006-metrics.js +1 -1
- package/engine/db/migrations/007-watches.js +2 -2
- package/engine/db/migrations/008-small-state.js +1 -1
- package/engine/db/migrations/009-qa.js +1 -1
- package/engine/db/migrations/010-pr-links.js +1 -1
- package/engine/db/migrations/011-remaining-state.js +1 -1
- package/engine/db/migrations/012-steering-deliveries.js +2 -2
- package/engine/db/migrations/013-backfill-broken-note-links.js +1 -1
- package/engine/db/migrations/014-pr-fix-target-prefs.js +2 -2
- package/engine/db/migrations/015-plans-prds.js +0 -0
- package/engine/db/migrations/018-sql-only-cutover.js +2 -2
- package/engine/db/migrations/021-archived-work-items.js +1 -1
- package/engine/db/migrations/022-global-cc-session.js +1 -1
- package/engine/db/migrations/023-engine-state.js +1 -1
- package/engine/db/migrations/025-malformed-work-item-phantoms.js +1 -1
- package/engine/db/migrations/027-review-learning-lifecycle.js +1 -1
- package/engine/db/migrations/029-repair-reused-versions.js +20 -0
- package/engine/db/migrations/031-pr-author-identity.js +137 -0
- package/engine/{consolidation.js → memory/consolidation.js} +6 -6
- package/engine/{kb-sweep-runner.js → memory/kb-sweep-runner.js} +2 -2
- package/engine/{kb-sweep.js → memory/kb-sweep.js} +9 -7
- package/engine/{memory-retrieval.js → memory/memory-retrieval.js} +46 -4
- package/engine/{memory-store.js → memory/memory-store.js} +3 -3
- package/engine/{promotion.js → memory/promotion.js} +3 -3
- package/engine/{review-learning-backfill.js → memory/review-learning-backfill.js} +6 -6
- package/engine/{review-learning.js → memory/review-learning.js} +10 -5
- package/engine/{diagnostics-memory.js → observability/diagnostics-memory.js} +1 -1
- package/engine/{logs-store.js → observability/logs-store.js} +5 -5
- package/engine/{metrics-store.js → observability/metrics-store.js} +4 -4
- package/engine/{check-status.js → operations/check-status.js} +3 -3
- package/engine/{cli.js → operations/cli.js} +271 -113
- package/engine/{distribution.js → operations/distribution.js} +5 -6
- package/engine/{cleanup.js → orchestration/cleanup.js} +72 -45
- package/engine/{cooldown.js → orchestration/cooldown.js} +5 -5
- package/engine/{dispatch-events.js → orchestration/dispatch-events.js} +2 -2
- package/engine/{dispatch.js → orchestration/dispatch.js} +129 -36
- package/engine/orchestration/failed-scheduled-cleanup.js +274 -0
- package/engine/{lifecycle.js → orchestration/lifecycle.js} +198 -90
- package/engine/{meeting.js → orchestration/meeting.js} +6 -16
- package/engine/{pipeline.js → orchestration/pipeline.js} +12 -12
- package/engine/{pre-dispatch-eval.js → orchestration/pre-dispatch-eval.js} +10 -9
- package/engine/{routing.js → orchestration/routing.js} +3 -3
- package/engine/{schedule-bootstrap.js → orchestration/schedule-bootstrap.js} +4 -4
- package/engine/{scheduler.js → orchestration/scheduler.js} +38 -8
- package/engine/{timeout.js → orchestration/timeout.js} +158 -109
- package/engine/{db-events.js → persistence/db-events.js} +2 -2
- package/engine/{dispatch-store.js → persistence/dispatch-store.js} +7 -7
- package/engine/{inbox-store.js → persistence/inbox-store.js} +2 -2
- package/engine/{note-link-backfill.js → persistence/note-link-backfill.js} +4 -4
- package/engine/{pr-fix-target-store.js → persistence/pr-fix-target-store.js} +8 -8
- package/engine/{pull-requests-store.js → persistence/pull-requests-store.js} +21 -7
- package/engine/{small-state-store.js → persistence/small-state-store.js} +31 -31
- package/engine/persistence/state-operations.js +350 -0
- package/engine/{steering-store.js → persistence/steering-store.js} +6 -6
- package/engine/{issues.js → planning/issues.js} +2 -2
- package/engine/{plan-prd-validation.js → planning/plan-prd-validation.js} +8 -2
- package/engine/planning/prd-result-sidecar.js +190 -0
- package/engine/{prd-store.js → planning/prd-store.js} +17 -17
- package/engine/{project-discovery.js → planning/project-discovery.js} +5 -5
- package/engine/{projects.js → planning/projects.js} +10 -10
- package/engine/{resolve-area.js → planning/resolve-area.js} +1 -1
- package/engine/{work-item-validation.js → planning/work-item-validation.js} +39 -3
- package/engine/{work-items-store.js → planning/work-items-store.js} +29 -21
- package/engine/{keep-process-sweep.js → processes/keep-process-sweep.js} +57 -17
- package/engine/{managed-spawn-launcher.js → processes/managed-spawn-launcher.js} +3 -3
- package/engine/{managed-spawn.js → processes/managed-spawn.js} +97 -46
- package/engine/{process-utils.js → processes/process-utils.js} +599 -55
- package/engine/{abandoned-pr-reconciliation.js → providers/abandoned-pr-reconciliation.js} +17 -7
- package/engine/{comment-classifier.js → providers/comment-classifier.js} +85 -17
- package/engine/{comment-format.js → providers/comment-format.js} +5 -5
- package/engine/{gh-comment.js → providers/gh-comment.js} +15 -15
- package/engine/{gh-token.js → providers/gh-token.js} +4 -4
- package/engine/{github.js → providers/github.js} +131 -54
- package/engine/{pr-action.js → providers/pr-action.js} +13 -12
- package/engine/{pr-clone-keep.js → providers/pr-clone-keep.js} +7 -7
- package/engine/{pr-devbox.js → providers/pr-devbox.js} +6 -6
- package/engine/{pr-fix-target.js → providers/pr-fix-target.js} +13 -13
- package/engine/{pr-remote-patch.js → providers/pr-remote-patch.js} +4 -4
- package/engine/{pr-resolve.js → providers/pr-resolve.js} +7 -7
- package/engine/{pr-temp-clone.js → providers/pr-temp-clone.js} +5 -5
- package/engine/{pr-track.js → providers/pr-track.js} +11 -13
- package/engine/{shared-branch-pr-reconcile.js → providers/shared-branch-pr-reconcile.js} +4 -4
- package/engine/qa/auto-prd-qa.js +313 -0
- package/engine/{qa-from-prd.js → qa/from-prd.js} +42 -12
- package/engine/qa/prd-session.js +240 -0
- package/engine/{qa-process-validation.js → qa/process-validation.js} +14 -9
- package/engine/{qa-runbooks.js → qa/runbooks.js} +1 -1
- package/engine/{qa-runs.js → qa/runs.js} +286 -15
- package/engine/{qa-sessions.js → qa/sessions.js} +595 -49
- package/engine/qa/visual-journey.js +654 -0
- package/engine/{qa-runners.js → qa-runners/index.js} +7 -7
- package/engine/qa-runners/maestro.js +3 -3
- package/engine/qa-runners/playwright.js +2 -2
- package/engine/{restart-health.js → recovery/restart-health.js} +48 -4
- package/engine/recovery/stop-stack.js +607 -0
- package/engine/{supervisor.js → recovery/supervisor.js} +105 -175
- package/engine/{watchdog.js → recovery/watchdog.js} +136 -13
- package/engine/runtimes/claude.js +14 -12
- package/engine/runtimes/codex.js +8 -6
- package/engine/runtimes/copilot.js +17 -16
- package/engine/{watch-actions.js → watches/actions.js} +13 -13
- package/engine/{watches.js → watches/index.js} +43 -32
- package/engine/{watches-store.js → watches/store.js} +4 -4
- package/engine/{create-pr-worktree.js → worktrees/create-pr.js} +1 -1
- package/engine/{worktree-gc.js → worktrees/gc.js} +70 -22
- package/engine/worktrees/inventory.js +671 -0
- package/engine/{live-checkout.js → worktrees/live-checkout.js} +4 -4
- package/engine/{worktree-pool.js → worktrees/pool.js} +2 -2
- package/engine/{worktree-preflight.js → worktrees/preflight.js} +1 -0
- package/engine/worktrees/quarantine-refs.js +173 -0
- package/engine.js +1137 -208
- package/minions.js +147 -77
- package/package.json +10 -6
- package/playbooks/_pr-description-audit.md +110 -78
- package/playbooks/build-fix-complex.md +2 -0
- package/playbooks/fix.md +16 -12
- package/playbooks/implement-shared.md +2 -0
- package/playbooks/implement.md +19 -20
- package/playbooks/plan-to-prd.md +18 -3
- package/playbooks/qa-session-draft.md +136 -1
- package/playbooks/qa-session-execute.md +80 -2
- package/playbooks/qa-session-setup.md +17 -1
- package/playbooks/qa-validate.md +1 -1
- package/playbooks/setup.md +2 -0
- package/playbooks/shared-rules.md +25 -32
- package/playbooks/templates/followup-dispatch.md +4 -3
- package/playbooks/verify.md +1 -1
- package/prompts/cc-system.md +19 -27
- package/watch-plugins/README.md +92 -0
- package/watch-plugins/ado-author-prs.js +336 -0
- package/watch-plugins/gh-author-prs.js +375 -0
- package/watch-plugins/http.js +474 -0
- package/watch-plugins/teams-channel.js +869 -0
- package/docs/dev-composite-workflow.md +0 -101
- package/docs/pr-screenshots/pr-886/after-single-header.png +0 -0
- package/docs/pr-screenshots/pr-886/before-duplicate-header.png +0 -0
- package/docs/pr-screenshots/pr-895/01-cancellation-reason-detail.png +0 -0
- package/docs/pr-screenshots/pr-899/worker-pool-worktrees-AFTER.png +0 -0
- package/docs/pr-screenshots/pr-899/worker-pool-worktrees-BEFORE.png +0 -0
- package/docs/pr-screenshots/pr-901/projects-tab-default.png +0 -0
- package/docs/pr-screenshots/pr-901/projects-tab-fmf-selected.png +0 -0
- package/docs/pr-screenshots/pr-916/model-picker-AFTER-crop.png +0 -0
- package/docs/pr-screenshots/pr-916/model-picker-AFTER.png +0 -0
- package/docs/pr-screenshots/pr-916/model-picker-BEFORE-crop.png +0 -0
- package/docs/pr-screenshots/pr-916/model-picker-BEFORE.png +0 -0
- package/docs/pr-screenshots/pr-916/model-picker-dropdown-AFTER.png +0 -0
- package/docs/pr-screenshots/pr-979/auto-fix-pane-AFTER.png +0 -0
- package/docs/pr-screenshots/pr-979/auto-fix-pane-BEFORE.png +0 -0
- package/docs/pr-screenshots/pr-985/pr-column-em-dash-AFTER.png +0 -0
- package/docs/pr-screenshots/pr-985/pr-column-em-dash-BEFORE.png +0 -0
- package/docs/visual-evidence-ci.md +0 -103
- package/engine/bridge.js +0 -379
- package/engine/quarantine-refs.js +0 -103
- package/engine/state-operations.js +0 -178
- /package/engine/{steering-constraints.js → agents/steering-constraints.js} +0 -0
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
> Author: Rebecca (Architect) | Date: 2026-04-07 | Status: **Implemented**
|
|
4
4
|
|
|
5
|
-
> **Implementation status:** Migrations 001–024 implement the full SQLite cutover and final compatibility cleanup. `engine/state.db` is the sole runtime-state authority; migration 018 records the cutover, migrations 020–024 absorb the remaining scoped/archive/session state and one-time PRD collision repair. Historical migrations retain one-time JSON import support. Normal runtime never references retired mirrors. `minions state check`, `backup`, and `export` provide operations support; explicit offline recovery import is available through `minions state import-legacy-json <dir> --confirm`. The remaining sections are retained as historical design rationale.
|
|
5
|
+
> **Implementation status:** Migrations 001–024 implement the full SQLite cutover and final compatibility cleanup. `engine/state.db` is the sole runtime-state authority; migration 018 records the cutover, migrations 020–024 absorb the remaining scoped/archive/session state and one-time PRD collision repair. Historical migrations retain one-time JSON import support. Normal runtime never references retired mirrors. `minions state check`, `backup`, `summary`, `verify-backup`, and `export` provide operations support; explicit offline recovery import is available through `minions state import-legacy-json <dir> --confirm`. The remaining sections are retained as historical design rationale.
|
|
6
6
|
|
|
7
7
|
## Executive Summary
|
|
8
8
|
|
|
@@ -30,11 +30,11 @@ Minions previously persisted runtime state as flat JSON files guarded by file-lo
|
|
|
30
30
|
|
|
31
31
|
**Total live state:** ~1.8 MB across 9+ JSON files.
|
|
32
32
|
|
|
33
|
-
(source: `engine/shared.js` `mutateJsonFileLocked` (~L1542) for locking, `engine/queries.js` for paths, live file sizes from `ls -la engine/*.json`)
|
|
33
|
+
(source: `engine/core/shared.js` `mutateJsonFileLocked` (~L1542) for locking, `engine/core/queries.js` for paths, live file sizes from `ls -la engine/*.json`)
|
|
34
34
|
|
|
35
35
|
### 1.2 Concurrency Model
|
|
36
36
|
|
|
37
|
-
All mutations go through `mutateJsonFileLocked()` (source: `engine/shared.js` ~L1542):
|
|
37
|
+
All mutations go through `mutateJsonFileLocked()` (source: `engine/core/shared.js` ~L1542):
|
|
38
38
|
|
|
39
39
|
```
|
|
40
40
|
acquire .lock file (exclusive create via fs.openSync 'wx')
|
|
@@ -46,12 +46,12 @@ release .lock file
|
|
|
46
46
|
```
|
|
47
47
|
|
|
48
48
|
Key properties:
|
|
49
|
-
- **Synchronous blocking** — `withFileLock` spins with `sleepMs(25)` until lock acquired or 5s timeout (source: `engine/shared.js` `withFileLock` ~L1329)
|
|
49
|
+
- **Synchronous blocking** — `withFileLock` spins with `sleepMs(25)` until lock acquired or 5s timeout (source: `engine/core/shared.js` `withFileLock` ~L1329)
|
|
50
50
|
- **Whole-file granularity** — updating one field in one work item rewrites all 180 items (370 KB)
|
|
51
|
-
- **Stale lock recovery** — locks older than 5 min (`LOCK_STALE_MS = 300_000`) are force-removed; holders that recorded a `{pid, ts, token}` payload are kept alive past the threshold while `process.kill(pid, 0)` succeeds, with a hard last-resort cap at 5×LOCK_STALE_MS (source: `engine/shared.js`, P-b7d4e8f2)
|
|
52
|
-
- **Identity-checked reap and release** — every acquisition stamps a unique `token` into the lock payload. Reaping never unlinks the lock path directly: the reaper re-observes the same token after a short grace delay, takes custody with an atomic `rename`, and verifies the quarantined file is the acquisition it judged stale (restoring it otherwise). Release likewise unlinks only while the on-disk token is still ours. Removing a lock by path alone is a TOCTOU that lets a second writer into the critical section and silently loses one update (source: `engine/shared.js` `withFileLock`, W-ms2nbtkj00016ffc)
|
|
51
|
+
- **Stale lock recovery** — locks older than 5 min (`LOCK_STALE_MS = 300_000`) are force-removed; holders that recorded a `{pid, ts, token}` payload are kept alive past the threshold while `process.kill(pid, 0)` succeeds, with a hard last-resort cap at 5×LOCK_STALE_MS (source: `engine/core/shared.js`, P-b7d4e8f2)
|
|
52
|
+
- **Identity-checked reap and release** — every acquisition stamps a unique `token` into the lock payload. Reaping never unlinks the lock path directly: the reaper re-observes the same token after a short grace delay, takes custody with an atomic `rename`, and verifies the quarantined file is the acquisition it judged stale (restoring it otherwise). Release likewise unlinks only while the on-disk token is still ours. Removing a lock by path alone is a TOCTOU that lets a second writer into the critical section and silently loses one update (source: `engine/core/shared.js` `withFileLock`, W-ms2nbtkj00016ffc)
|
|
53
53
|
- **Compromised sections fail loudly** — if the lock is gone or re-owned when the critical section ends, `withFileLock` throws `ELOCKCOMPROMISED`. `mutateJsonFileLocked` / `mutateTextFileLocked` first re-apply the mutation against the current on-disk state (up to `LOCK_COMPROMISE_REDOS`) when their own write did not survive, so a raced update is recovered rather than dropped
|
|
54
|
-
- **Read caching** — only `dispatch.json` has a 2s TTL cache (source: `engine/queries.js`)
|
|
54
|
+
- **Read caching** — only `dispatch.json` has a 2s TTL cache (source: `engine/core/queries.js`)
|
|
55
55
|
|
|
56
56
|
### 1.3 Read vs Write Ratio
|
|
57
57
|
|
|
@@ -228,13 +228,13 @@ A creative zero-dep option: use append-only JSON Lines files as the write log, w
|
|
|
228
228
|
|
|
229
229
|
Stay with files. Fix the two highest-pain issues immediately:
|
|
230
230
|
|
|
231
|
-
1. **Cap `cooldowns.json`** — Add a cleanup sweep that deletes entries older than 7 days. This file is 511 KB with 125 keys, most of which are stale. Implement in `engine/cooldown.js` cleanup function. (source: `engine/cooldown.js`)
|
|
231
|
+
1. **Cap `cooldowns.json`** — Add a cleanup sweep that deletes entries older than 7 days. This file is 511 KB with 125 keys, most of which are stale. Implement in `engine/orchestration/cooldown.js` cleanup function. (source: `engine/orchestration/cooldown.js`)
|
|
232
232
|
|
|
233
|
-
2. **Cap `dispatch.json` completed array** more aggressively — Currently capped at 100 entries (source: `engine/dispatch.js:112-114`). Reduce to 50 or archive to `dispatch/completed/` directory. The 380 KB file is mostly completed entries.
|
|
233
|
+
2. **Cap `dispatch.json` completed array** more aggressively — Currently capped at 100 entries (source: `engine/orchestration/dispatch.js:112-114`). Reduce to 50 or archive to `dispatch/completed/` directory. The 380 KB file is mostly completed entries.
|
|
234
234
|
|
|
235
|
-
3. **Add read caches to `work-items.json` and `pull-requests.json`** — Same 2s TTL pattern as dispatch.json (source: `engine/queries.js:82-91`). These are read 8+ times per tick but only written 1-2 times.
|
|
235
|
+
3. **Add read caches to `work-items.json` and `pull-requests.json`** — Same 2s TTL pattern as dispatch.json (source: `engine/core/queries.js:82-91`). These are read 8+ times per tick but only written 1-2 times.
|
|
236
236
|
|
|
237
|
-
4. **Convert `log.json` to append-only JSONL** — Eliminates the parse-entire-file-to-append pattern in `_flushLogBuffer()` (source: `engine/shared.js` `_flushLogBuffer` ~L499). Log rotation becomes `readFile → keep last 2000 lines → writeFile` instead of `parse JSON array → splice → stringify → write`.
|
|
237
|
+
4. **Convert `log.json` to append-only JSONL** — Eliminates the parse-entire-file-to-append pattern in `_flushLogBuffer()` (source: `engine/core/shared.js` `_flushLogBuffer` ~L499). Log rotation becomes `readFile → keep last 2000 lines → writeFile` instead of `parse JSON array → splice → stringify → write`.
|
|
238
238
|
|
|
239
239
|
### Phase 2: `node:sqlite` Migration (When API stabilizes — estimated Node 26 LTS)
|
|
240
240
|
|
|
@@ -12,7 +12,7 @@ crash of `node.exe` would have vanished without a trace.
|
|
|
12
12
|
## What this does
|
|
13
13
|
|
|
14
14
|
Every spawn of `engine.js` — via `minions start`/`restart` (`bin/minions.js`)
|
|
15
|
-
**and** the supervisor's respawn-on-death path (`engine/supervisor.js`) —
|
|
15
|
+
**and** the supervisor's respawn-on-death path (`engine/recovery/supervisor.js`) —
|
|
16
16
|
now gets Node's own diagnostic-report flags folded into `NODE_OPTIONS`:
|
|
17
17
|
|
|
18
18
|
```
|
|
@@ -44,7 +44,7 @@ existing `engine/diagnostics/` directory used for heap snapshots and
|
|
|
44
44
|
CPU/heap profiles (see [docs/diagnostics-memory.md](diagnostics-memory.md)
|
|
45
45
|
§3–4), in its own subfolder so retention sweeps don't collide. The directory
|
|
46
46
|
is gitignored and created on first spawn if missing
|
|
47
|
-
(`getEngineCrashDiagnosticsEnv`, [`engine/shared.js`](../engine/shared.js)).
|
|
47
|
+
(`getEngineCrashDiagnosticsEnv`, [`engine/core/shared.js`](../engine/core/shared.js)).
|
|
48
48
|
|
|
49
49
|
## Config + opt-out
|
|
50
50
|
|
|
@@ -67,12 +67,12 @@ is gitignored and created on first spawn if missing
|
|
|
67
67
|
after editing `enabled`, since it only takes effect at the next spawn.
|
|
68
68
|
|
|
69
69
|
Both live in `ENGINE_DEFAULTS.crashDiagnostics`
|
|
70
|
-
([`engine/shared.js`](../engine/shared.js)).
|
|
70
|
+
([`engine/core/shared.js`](../engine/core/shared.js)).
|
|
71
71
|
|
|
72
72
|
## Retention / cleanup
|
|
73
73
|
|
|
74
|
-
Reports are pruned by `pruneCrashDiagnosticsReports` (`engine/shared.js`),
|
|
75
|
-
called from `engine/cleanup.js#runCleanup` — the same periodic cleanup pass
|
|
74
|
+
Reports are pruned by `pruneCrashDiagnosticsReports` (`engine/core/shared.js`),
|
|
75
|
+
called from `engine/orchestration/cleanup.js#runCleanup` — the same periodic cleanup pass
|
|
76
76
|
that already runs every `ENGINE_DEFAULTS.cleanupEvery` ticks (default 60
|
|
77
77
|
ticks ≈ 10 min at the default 10 s tick interval). It keeps the newest
|
|
78
78
|
`retainCount` files by mtime and deletes the rest, mirroring the existing
|
|
@@ -82,7 +82,7 @@ swallowed and retried on the next cleanup pass.
|
|
|
82
82
|
|
|
83
83
|
## Idempotency across the CLI → supervisor respawn chain
|
|
84
84
|
|
|
85
|
-
Both `bin/minions.js` (initial spawn) and `engine/supervisor.js` (respawn on
|
|
85
|
+
Both `bin/minions.js` (initial spawn) and `engine/recovery/supervisor.js` (respawn on
|
|
86
86
|
death) call the same `getEngineCrashDiagnosticsEnv(config)` helper. It checks
|
|
87
87
|
whether the caller's `NODE_OPTIONS` already contains `--diagnostic-dir=`
|
|
88
88
|
(which a spawned child inherits from its parent's env) and returns `{}`
|
|
@@ -119,13 +119,13 @@ a second or two, without killing the process.
|
|
|
119
119
|
|
|
120
120
|
## Related modules and tests
|
|
121
121
|
|
|
122
|
-
- Module: [`engine/shared.js`](../engine/shared.js) — `CRASH_REPORTS_DIR`,
|
|
122
|
+
- Module: [`engine/core/shared.js`](../engine/core/shared.js) — `CRASH_REPORTS_DIR`,
|
|
123
123
|
`getEngineCrashDiagnosticsEnv`, `pruneCrashDiagnosticsReports`,
|
|
124
124
|
`ENGINE_DEFAULTS.crashDiagnostics`.
|
|
125
125
|
- Spawn wiring: [`bin/minions.js`](../bin/minions.js)
|
|
126
126
|
(`spawnFullStackAndVerify`, the post-install/upgrade auto-start path) and
|
|
127
|
-
[`engine/supervisor.js`](../engine/supervisor.js) (`spawnEngine`).
|
|
128
|
-
- Cleanup wiring: [`engine/cleanup.js`](../engine/cleanup.js) `runCleanup`.
|
|
127
|
+
[`engine/recovery/supervisor.js`](../engine/recovery/supervisor.js) (`spawnEngine`).
|
|
128
|
+
- Cleanup wiring: [`engine/orchestration/cleanup.js`](../engine/orchestration/cleanup.js) `runCleanup`.
|
|
129
129
|
- Tests: [`test/unit/crash-diagnostics.test.js`](../test/unit/crash-diagnostics.test.js).
|
|
130
130
|
- Related: [docs/diagnostics-memory.md](diagnostics-memory.md) (memory/GC/heap
|
|
131
131
|
observability — the sibling diagnostics surface this doc's directory lives
|
|
@@ -75,7 +75,7 @@ curl -s http://localhost:7331/api/diagnostics/memory | jq
|
|
|
75
75
|
```
|
|
76
76
|
|
|
77
77
|
Field reference (every value originates from
|
|
78
|
-
[`engine/diagnostics-memory.js`](../engine/diagnostics-memory.js)
|
|
78
|
+
[`engine/observability/diagnostics-memory.js`](../engine/observability/diagnostics-memory.js)
|
|
79
79
|
`sampleSelf()`):
|
|
80
80
|
|
|
81
81
|
| Field | Source | Units |
|
|
@@ -107,7 +107,7 @@ In-memory ring buffer of samples. Two backing rings:
|
|
|
107
107
|
`DIAGNOSTICS_MEMORY_SAMPLE_INTERVAL_MS = 60 000` ms,
|
|
108
108
|
[`dashboard.js:122`](../dashboard.js)). Capacity is
|
|
109
109
|
`RING_BUFFER_CAP = 1440` samples (≈ 24 h at 60 s cadence,
|
|
110
|
-
[`engine/diagnostics-memory.js:28`](../engine/diagnostics-memory.js)).
|
|
110
|
+
[`engine/observability/diagnostics-memory.js:28`](../engine/observability/diagnostics-memory.js)).
|
|
111
111
|
- `process=engine` — the dashboard's **polled accumulation** of the
|
|
112
112
|
engine's sidecar. Engine.js only persists the latest sample to
|
|
113
113
|
`engine/diagnostics-memory.json`, so engine-side history is rebuilt by
|
|
@@ -148,7 +148,7 @@ A 400 is returned when `process` is missing or not exactly one of
|
|
|
148
148
|
|
|
149
149
|
Every `ENGINE_DEFAULTS.memoryBaselineEveryTicks` ticks (default `6`,
|
|
150
150
|
defined at
|
|
151
|
-
[`engine/shared.js:3092`](../engine/shared.js)), the engine emits one
|
|
151
|
+
[`engine/core/shared.js:3092`](../engine/core/shared.js)), the engine emits one
|
|
152
152
|
structured log line and writes the latest sample to the sidecar. At the
|
|
153
153
|
default `tickInterval: 10` the cadence is ≈ 60 s.
|
|
154
154
|
|
|
@@ -421,7 +421,7 @@ they sit in `notes/inbox/` with predictable slugs:
|
|
|
421
421
|
| 2 — heap dominators + leak candidates | `P-f6a7b8c9` | `notes/inbox/ripley-memory-audit-phase-2-heap-dominators*.md` | `knowledge/architecture/2026-06-12-ripley-memory-audit-phase-2-heap-dominators.md` |
|
|
422
422
|
| 3 — CPU + allocation hot frames | `P-a7b8c9d0` | `notes/inbox/ripley-memory-audit-phase-3-cpu-and-allocations*.md` | `knowledge/architecture/2026-06-12-ripley-memory-audit-phase-3-cpu-and-allocations.md` |
|
|
423
423
|
|
|
424
|
-
Use [`engine/kb-sweep.js`](../engine/kb-sweep.js) (see
|
|
424
|
+
Use [`engine/memory/kb-sweep.js`](../engine/memory/kb-sweep.js) (see
|
|
425
425
|
[`docs/kb-sweep.md`](kb-sweep.md)) or the dashboard's "Sweep" button to
|
|
426
426
|
trigger promotion. The cheap-win PR items downstream of the explores
|
|
427
427
|
(`P-b8c9d0e1`, `P-c9d0e1f2`, `P-d0e1f2a3`) consume the prioritized lists
|
|
@@ -430,7 +430,7 @@ re-deriving the same findings.
|
|
|
430
430
|
|
|
431
431
|
## Related modules and tests
|
|
432
432
|
|
|
433
|
-
- Module: [`engine/diagnostics-memory.js`](../engine/diagnostics-memory.js) — sampler + ring buffer + GC observer.
|
|
433
|
+
- Module: [`engine/observability/diagnostics-memory.js`](../engine/observability/diagnostics-memory.js) — sampler + ring buffer + GC observer.
|
|
434
434
|
- Engine wiring: [`engine.js`](../engine.js) `emitMemoryBaseline` + `processHeapSnapshotRequest`.
|
|
435
435
|
- Dashboard wiring: [`dashboard.js`](../dashboard.js) `handleDiagnosticsMemory`, `handleDiagnosticsMemoryHistory`, `handleDiagnosticsHeapSnapshot`.
|
|
436
436
|
- Frontend: [`dashboard/pages/engine-memory-panel.html`](../dashboard/pages/engine-memory-panel.html) + [`dashboard/js/memory-panel.js`](../dashboard/js/memory-panel.js).
|
|
@@ -12,13 +12,13 @@ canonical data, or misunderstand routing and dashboard behavior.
|
|
|
12
12
|
| Stale claim | Correction | Source of truth |
|
|
13
13
|
|---|---|---|
|
|
14
14
|
| Node.js 18+ was sufficient | Node.js 22.5+ is required for `node:sqlite` | `package.json:68` |
|
|
15
|
-
| Agent dispatch was described as Claude-only | Copilot is the default fleet runtime; Claude and Codex remain supported | `engine/shared.js:3188`, `engine/runtimes/index.js` |
|
|
15
|
+
| Agent dispatch was described as Claude-only | Copilot is the default fleet runtime; Claude and Codex remain supported | `engine/core/shared.js:3188`, `engine/runtimes/index.js` |
|
|
16
16
|
| Root and per-project JSON trackers were described as primary state | `engine/state.db` is the sole runtime-state authority; retired trackers are migration inputs only | `engine/db/migrations/018-sql-only-cutover.js`, `CLAUDE.md` "State storage" |
|
|
17
|
-
| `minions preflight` was documented as a public command | `minions doctor` is public; lighter checks run internally during initialization and startup | `bin/minions.js:1207-1264`, `engine/cli.js:232-255` |
|
|
18
|
-
| Git repository host default was documented as ADO | The fallback host is GitHub | `engine/projects.js:450`, `engine/projects.js:510` |
|
|
19
|
-
| `WORK_TYPE` documentation omitted complex build fixes | Added `BUILD_FIX_COMPLEX` | `engine/shared.js:4057-4081`, `routing.md:10-28` |
|
|
20
|
-
| Command Center sessions were described as a JSON-file store | Per-tab and global sessions are stored only in SQLite | `engine/small-state-store.js`, `engine/db/migrations/022-global-cc-session.js` |
|
|
21
|
-
| Watches were described as checking every three ticks and persisting to JSON | The default cadence is 18 ticks; persistence routes through the SQL watch store | `engine/shared.js` `watchPollEvery`, `engine/watches
|
|
17
|
+
| `minions preflight` was documented as a public command | `minions doctor` is public; lighter checks run internally during initialization and startup | `bin/minions.js:1207-1264`, `engine/operations/cli.js:232-255` |
|
|
18
|
+
| Git repository host default was documented as ADO | The fallback host is GitHub | `engine/planning/projects.js:450`, `engine/planning/projects.js:510` |
|
|
19
|
+
| `WORK_TYPE` documentation omitted complex build fixes | Added `BUILD_FIX_COMPLEX` | `engine/core/shared.js:4057-4081`, `routing.md:10-28` |
|
|
20
|
+
| Command Center sessions were described as a JSON-file store | Per-tab and global sessions are stored only in SQLite | `engine/persistence/small-state-store.js`, `engine/db/migrations/022-global-cc-session.js` |
|
|
21
|
+
| Watches were described as checking every three ticks and persisting to JSON | The default cadence is 18 ticks; persistence routes through the SQL watch store | `engine/core/shared.js` `watchPollEvery`, `engine/watches/store.js` |
|
|
22
22
|
| Two local documentation links targeted files that are not shipped | Removed the runtime `notes.md` link and the stale managed-spawn plan link | `docs/README.md`, `docs/managed-spawn.md` |
|
|
23
23
|
|
|
24
24
|
## New tutorial coverage
|
|
@@ -38,6 +38,6 @@ The tutorial track under [`docs/tutorials/`](tutorials/README.md) now covers:
|
|
|
38
38
|
|
|
39
39
|
Reference docs remain intentionally detailed. Tutorials should link to those
|
|
40
40
|
references rather than duplicate full schemas. Future sweeps should continue to
|
|
41
|
-
verify CLI examples against `bin/minions.js` and `engine/cli.js`, dashboard
|
|
41
|
+
verify CLI examples against `bin/minions.js` and `engine/operations/cli.js`, dashboard
|
|
42
42
|
routes against `dashboard.js`, work types against `routing.md`, and state
|
|
43
43
|
storage claims against the SQL store modules.
|
package/docs/engine-restart.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Engine Restart & Agent Survival
|
|
2
2
|
|
|
3
|
-
> Last verified: 2026-07-
|
|
3
|
+
> Last verified: 2026-07-29 against `bin/minions.js`, `engine/operations/cli.js` startup reattachment, `engine/orchestration/timeout.js`, `engine/recovery/supervisor.js`, `engine/recovery/restart-health.js`, and `ENGINE_DEFAULTS`.
|
|
4
4
|
|
|
5
5
|
## The Problem
|
|
6
6
|
|
|
@@ -41,7 +41,9 @@ Configurable via `config.json`:
|
|
|
41
41
|
### 2. Process-Based Liveness
|
|
42
42
|
|
|
43
43
|
At startup, each non-pooled active dispatch reads its PID file, verifies that
|
|
44
|
-
the process is alive
|
|
44
|
+
the process is alive, verifies its command line still belongs to an agent
|
|
45
|
+
runtime (protecting against PID reuse), and confirms its persisted execution
|
|
46
|
+
path is still safe, restores
|
|
45
47
|
the matching session id when available, and repopulates `activeProcesses` with
|
|
46
48
|
`reattached: true`. A recent `live-output.log` can provide temporary
|
|
47
49
|
unknown-PID liveness when no PID file survived.
|
|
@@ -63,7 +65,9 @@ when the drain deadline expires it is cancelled and terminalized as a
|
|
|
63
65
|
retryable timeout before the old engine exits.
|
|
64
66
|
|
|
65
67
|
Unexpected crashes remain fail-safe: persisted `item.pooled` dispatches never
|
|
66
|
-
trust the worker PID and are reaped through the
|
|
68
|
+
trust the worker PID and are reaped through the restart-recovery path on startup.
|
|
69
|
+
Restart recovery has its own bounded counter, so host/daemon failures do not
|
|
70
|
+
consume the semantic agent retry or per-agent reassignment budgets.
|
|
67
71
|
|
|
68
72
|
### 4. Stop Warning
|
|
69
73
|
|
|
@@ -79,16 +83,38 @@ To kill them now, run: node engine.js kill
|
|
|
79
83
|
|
|
80
84
|
### 5. Exponential Backoff on Failures
|
|
81
85
|
|
|
82
|
-
If an agent is killed as an orphan and the work item retries, cooldowns use exponential backoff (2^failures, max 8x) to prevent spam-retrying broken tasks.
|
|
86
|
+
If an agent is killed as an orphan and the work item retries, cooldowns use exponential backoff (2^failures, max 8x) to prevent spam-retrying broken tasks. Confirmed restart losses use `_infrastructureRetryCount` (bounded by `ENGINE_DEFAULTS.maxRetries`) instead of `_retryCount`.
|
|
83
87
|
|
|
84
88
|
### 6. Crash-Loop Counter + Alert (W-mr2azk6i000y06e0)
|
|
85
89
|
|
|
86
|
-
The engine.js *process itself* (not an agent) can crash and get auto-respawned by three independent mechanisms: `engine/supervisor.js#checkEngine` (dead-PID), `engine/supervisor.js#checkEngineHung` (stale-heartbeat), and `engine/watchdog.js` (OS-scheduled external recovery); `dashboard.js`'s in-process 30s watchdog also auto-restarts on a dead PID (the user-triggered Restart Engine button is excluded). Each successful respawn calls `shared.recordEngineRespawn(source)`, which appends `{ts, source}` to `control.json`'s `engineRespawns[]` (rolling window, default `CRASH_LOOP_WINDOW_MS` = 10 min). Once respawns within the window reach `CRASH_LOOP_THRESHOLD` (default 3), `control.crashLoopAlert` is set (dashboard-visible via `/api/status`) and a deduped (one-per-day) `notes/inbox/` alert is written; the alert clears automatically once the window rolls past the incident. Override the window/threshold via `MINIONS_CRASH_LOOP_WINDOW_MS` / `MINIONS_CRASH_LOOP_THRESHOLD` env vars.
|
|
90
|
+
The engine.js *process itself* (not an agent) can crash and get auto-respawned by three independent mechanisms: `engine/recovery/supervisor.js#checkEngine` (dead-PID), `engine/recovery/supervisor.js#checkEngineHung` (stale-heartbeat), and `engine/recovery/watchdog.js` (OS-scheduled external recovery); `dashboard.js`'s in-process 30s watchdog also auto-restarts on a dead PID (the user-triggered Restart Engine button is excluded). Each successful respawn calls `shared.recordEngineRespawn(source)`, which appends `{ts, source}` to `control.json`'s `engineRespawns[]` (rolling window, default `CRASH_LOOP_WINDOW_MS` = 10 min). Once respawns within the window reach `CRASH_LOOP_THRESHOLD` (default 3), `control.crashLoopAlert` is set (dashboard-visible via `/api/status`) and a deduped (one-per-day) `notes/inbox/` alert is written; the alert clears automatically once the window rolls past the incident. Override the window/threshold via `MINIONS_CRASH_LOOP_WINDOW_MS` / `MINIONS_CRASH_LOOP_THRESHOLD` env vars.
|
|
91
|
+
|
|
92
|
+
On Windows, the scheduled watchdog permits parallel one-shot ticks and caps each
|
|
93
|
+
instance at two minutes, so a stranded launcher cannot block future recovery
|
|
94
|
+
fires. Source checkouts pin the task to `<MINIONS_HOME>/bin/minions.js`; packaged
|
|
95
|
+
installs fall back to the installed CLI executable.
|
|
96
|
+
|
|
97
|
+
Partial state (only engine or dashboard alive) is debounced for 90 seconds.
|
|
98
|
+
After that grace, the OS watchdog starts only the missing component; it never
|
|
99
|
+
tears down the healthy component with `minions restart`. Both-dead state still
|
|
100
|
+
triggers `minions start` immediately. Windows dashboard PID probing also fails
|
|
101
|
+
open: a slow or timed-out `tasklist` command is indeterminate, not evidence
|
|
102
|
+
that the engine died.
|
|
87
103
|
|
|
88
104
|
### 7. Exclusive Restart Topology
|
|
89
105
|
|
|
90
106
|
Managed dashboard spawns must bind the requested port; automatic fallback remains available only for standalone `node dashboard.js` use. Before a supervisor respawn, every matching stale daemon is killed and confirmed dead. If any PID survives, the supervisor skips the replacement and retries on its next tick instead of creating a second engine or a dashboard on another port.
|
|
91
107
|
|
|
108
|
+
Daemon ownership is exact: cleanup matches the normalized script token under
|
|
109
|
+
the selected runtime root, never a loose `engine.js`/`dashboard.js` substring
|
|
110
|
+
or an unverified port owner. The supervisor also treats a recent 15s engine
|
|
111
|
+
heartbeat as positive liveness evidence when its shell-based PID probe fails,
|
|
112
|
+
and daemon replacement kills only the daemon PID—not its detached cold-agent
|
|
113
|
+
or managed-service descendants. The OS-scheduled watchdog applies the same
|
|
114
|
+
fresh-heartbeat veto before turning simultaneous probe failures into an
|
|
115
|
+
immediate `minions start`, and it stops trusting an existing PID once the
|
|
116
|
+
recorded engine heartbeat is stale.
|
|
117
|
+
|
|
92
118
|
`minions restart` first verifies that its exact engine and dashboard PIDs are alive and that the dashboard owns the requested-port beacon. It starts the supervisor only after that cold-start phase succeeds, preventing the supervisor's first watchdog tick from replacing a legitimately slow, not-yet-listening dashboard. It then verifies ownership of the exact engine, dashboard, and supervisor PIDs and rejects duplicate daemon processes.
|
|
93
119
|
|
|
94
120
|
A failed verification tears down the partial stack and leaves `stop-intent.json` set so watchdogs cannot race another replacement into the failed topology. Dashboard failures also print the bounded `engine/dashboard-stdio.log` output written since that spawn, or the log path when the process produced no output, instead of returning only a dead-PID summary.
|
|
@@ -97,6 +123,65 @@ PID-bearing file locks are reaped immediately when their recorded owner is dead.
|
|
|
97
123
|
|
|
98
124
|
Dashboard hot polling avoids recurring event-loop stalls: freshness-directory walks are coalesced briefly, plan scans are cached for 30 seconds and processed in cooperative batches, and quarantine-ref Git reads run asynchronously in parallel.
|
|
99
125
|
|
|
126
|
+
## Stopping the Runtime: `minions stop` vs `minions stop --all`
|
|
127
|
+
|
|
128
|
+
Stopping is deliberately two commands, because "ask the engine to shut down" and
|
|
129
|
+
"quiesce the whole runtime" are different contracts.
|
|
130
|
+
|
|
131
|
+
| Command | What it does | Blocks until |
|
|
132
|
+
|---------|--------------|--------------|
|
|
133
|
+
| `minions stop` | Delegates to `engine.js stop`: writes stop-intent, stamps the shutdown request into `control.json`, returns. The dashboard and supervisor are untouched. | Nothing — it returns as soon as shutdown is *requested*. |
|
|
134
|
+
| `minions stop --all` | Ordered whole-stack teardown: stop-intent → supervisor → dashboard → graceful engine drain → identity-verified reap → late-respawn sweep → verification. | The daemons being verified down. It reports the `engine/state.db-shm` state but does not wait for it. |
|
|
135
|
+
| `minions stop --all --wait` | The same teardown, and then waits for `engine/state.db-shm` to be released. | The database handles being released, or the budget expiring. |
|
|
136
|
+
|
|
137
|
+
`--timeout <ms>` sets the budget for the whole sequence (default 60000, matching
|
|
138
|
+
the internal installer's quiescence window).
|
|
139
|
+
|
|
140
|
+
Bare `minions stop` is unchanged and will stay that way: recovery paths, operator
|
|
141
|
+
scripts, and the engine's own callers depend on "request shutdown and return".
|
|
142
|
+
The whole-stack behavior is strictly opt-in.
|
|
143
|
+
|
|
144
|
+
**Why `--wait` exists.** The dashboard holds `engine/state.db` open independently
|
|
145
|
+
of the engine, so after an engine-only stop the WAL index (`engine/state.db-shm`)
|
|
146
|
+
is still present and non-empty and every quiescence probe correctly stays red.
|
|
147
|
+
Anything that must touch the database file itself — the internal installer's
|
|
148
|
+
pre-migration gate, a manual backup/restore — needs the release proven, not
|
|
149
|
+
assumed.
|
|
150
|
+
|
|
151
|
+
### Exit codes
|
|
152
|
+
|
|
153
|
+
`minions stop --all` reports its result through the exit code, so callers branch
|
|
154
|
+
without scraping stdout (`STOP_STACK_EXIT` in `engine/recovery/stop-stack.js`):
|
|
155
|
+
|
|
156
|
+
| Code | Meaning | Operator action |
|
|
157
|
+
|------|---------|-----------------|
|
|
158
|
+
| `0` | Stopped. Nothing holds the runtime root. | None. |
|
|
159
|
+
| `3` | Did not quiesce within the budget — a daemon is still alive, or (with `--wait`) the WAL index is still held. | Retry, widen `--timeout`, or find the remaining holder. |
|
|
160
|
+
| `4` | Refused: a live PID could not be proven to belong to this runtime root, so nothing was signalled. | Inspect the reported PID; Minions will not kill a process it cannot verify. |
|
|
161
|
+
|
|
162
|
+
`4` is deliberately distinct from `3`: when identity cannot be proven, nothing was
|
|
163
|
+
force-killed at all, and telling the operator to widen a budget would be wrong.
|
|
164
|
+
|
|
165
|
+
### One implementation, four other callers
|
|
166
|
+
|
|
167
|
+
The sequence itself lives in `engine/recovery/stop-stack.js#stopRuntimeStack`.
|
|
168
|
+
`minions restart`, `minions nuke`, `minions uninstall`, and the `init` upgrade
|
|
169
|
+
path all call it through `stopWholeRuntimeStack()` in `bin/minions.js` — there is
|
|
170
|
+
no second copy of the teardown in the tree.
|
|
171
|
+
|
|
172
|
+
Those four pass `waitForRelease: false`: they either replace the stack
|
|
173
|
+
immediately or delete the runtime root, so blocking on the WAL index would add
|
|
174
|
+
latency without adding safety. They still get the same ordered teardown, the same
|
|
175
|
+
identity-verified termination, the same late-respawn sweep, and the same
|
|
176
|
+
structured verdict.
|
|
177
|
+
|
|
178
|
+
`restart` keeps its own budget (`engine.shutdownTimeout + 5000`) and its own
|
|
179
|
+
gates on top: it aborts if a *live daemon of ours* survived the teardown, but not
|
|
180
|
+
when a holder is merely a stale PID file pointing at a foreign process
|
|
181
|
+
(`holderIsLiveDaemon`) — healing that state is exactly what `restart` is for. The
|
|
182
|
+
dashboard port gate, the beacon clearing, and the post-spawn topology
|
|
183
|
+
verification are unchanged.
|
|
184
|
+
|
|
100
185
|
## Safe Restart Pattern
|
|
101
186
|
|
|
102
187
|
```bash
|
package/docs/harness-mode.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# Tri-Agent Harness Mode
|
|
2
2
|
|
|
3
3
|
> Status: opt-in feature flag on scheduled tasks (`harness_mode: "tri_agent"`).
|
|
4
|
-
> Shipped: W-mq07a9gf000jbc2b. Module: [`engine/harness.js`](../engine/harness.js).
|
|
4
|
+
> Shipped: W-mq07a9gf000jbc2b. Module: [`engine/agents/harness.js`](../engine/agents/harness.js).
|
|
5
5
|
|
|
6
6
|
## What it is
|
|
7
7
|
|