@apralabs/apra-fleet 0.4.0 → 0.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +145 -7
- package/dist/agents/backlog-groomer.md +17 -1
- package/dist/agents/ci-watcher.md +13 -1
- package/dist/agents/deployer.md +15 -3
- package/dist/agents/doer.md +35 -1
- package/dist/agents/harvester.md +22 -3
- package/dist/agents/integ-test-runner.md +18 -2
- package/dist/agents/kb-reconciler.md +216 -0
- package/dist/agents/plan-reviewer.md +14 -1
- package/dist/agents/planner.md +88 -5
- package/dist/agents/regression-test-runner.md +19 -1
- package/dist/agents/reviewer.md +104 -3
- package/dist/agents/schemas/doer-output.json +63 -3
- package/dist/agents/schemas/harvester-output.json +59 -2
- package/dist/agents/schemas/integ-test-runner-output.json +2 -2
- package/dist/agents/schemas/kb-reconciler-output.json +87 -0
- package/dist/agents/schemas/planner-output.json +65 -0
- package/dist/agents/schemas/reviewer-output.json +105 -8
- package/dist/cli/install.d.ts +1 -0
- package/dist/cli/install.d.ts.map +1 -1
- package/dist/cli/install.js +136 -5
- package/dist/cli/install.js.map +1 -1
- package/dist/cli/join.js +1 -1
- package/dist/cli/join.js.map +1 -1
- package/dist/cli/kb-commit.d.ts +12 -0
- package/dist/cli/kb-commit.d.ts.map +1 -0
- package/dist/cli/kb-commit.js +53 -0
- package/dist/cli/kb-commit.js.map +1 -0
- package/dist/cli/kb-directives.d.ts +17 -0
- package/dist/cli/kb-directives.d.ts.map +1 -0
- package/dist/cli/kb-directives.js +146 -0
- package/dist/cli/kb-directives.js.map +1 -0
- package/dist/cli/kb-import.d.ts +13 -0
- package/dist/cli/kb-import.d.ts.map +1 -0
- package/dist/cli/kb-import.js +56 -0
- package/dist/cli/kb-import.js.map +1 -0
- package/dist/cli/uninstall.d.ts.map +1 -1
- package/dist/cli/uninstall.js +11 -0
- package/dist/cli/uninstall.js.map +1 -1
- package/dist/cli/watch.d.ts +53 -4
- package/dist/cli/watch.d.ts.map +1 -1
- package/dist/cli/watch.js +118 -17
- package/dist/cli/watch.js.map +1 -1
- package/dist/commands/kb-server.d.ts +8 -0
- package/dist/commands/kb-server.d.ts.map +1 -0
- package/dist/commands/kb-server.js +221 -0
- package/dist/commands/kb-server.js.map +1 -0
- package/dist/index.js +77 -2
- package/dist/index.js.map +1 -1
- package/dist/os/linux.d.ts +1 -0
- package/dist/os/linux.d.ts.map +1 -1
- package/dist/os/linux.js +18 -1
- package/dist/os/linux.js.map +1 -1
- package/dist/os/os-commands.d.ts +5 -0
- package/dist/os/os-commands.d.ts.map +1 -1
- package/dist/os/windows.d.ts +35 -0
- package/dist/os/windows.d.ts.map +1 -1
- package/dist/os/windows.js +67 -31
- package/dist/os/windows.js.map +1 -1
- package/dist/providers/agy.d.ts +12 -1
- package/dist/providers/agy.d.ts.map +1 -1
- package/dist/providers/agy.js +134 -9
- package/dist/providers/agy.js.map +1 -1
- package/dist/providers/claude.d.ts +11 -1
- package/dist/providers/claude.d.ts.map +1 -1
- package/dist/providers/claude.js +47 -19
- package/dist/providers/claude.js.map +1 -1
- package/dist/providers/codex.d.ts +11 -1
- package/dist/providers/codex.d.ts.map +1 -1
- package/dist/providers/codex.js +29 -6
- package/dist/providers/codex.js.map +1 -1
- package/dist/providers/copilot.d.ts +11 -1
- package/dist/providers/copilot.d.ts.map +1 -1
- package/dist/providers/copilot.js +26 -6
- package/dist/providers/copilot.js.map +1 -1
- package/dist/providers/email/headers.d.ts +15 -0
- package/dist/providers/email/headers.d.ts.map +1 -0
- package/dist/providers/email/headers.js +42 -0
- package/dist/providers/email/headers.js.map +1 -0
- package/dist/providers/email/index.d.ts +18 -0
- package/dist/providers/email/index.d.ts.map +1 -0
- package/dist/providers/email/index.js +60 -0
- package/dist/providers/email/index.js.map +1 -0
- package/dist/providers/email/provider.d.ts +49 -0
- package/dist/providers/email/provider.d.ts.map +1 -0
- package/dist/providers/email/provider.js +2 -0
- package/dist/providers/email/provider.js.map +1 -0
- package/dist/providers/email/sendgrid.d.ts +12 -0
- package/dist/providers/email/sendgrid.d.ts.map +1 -0
- package/dist/providers/email/sendgrid.js +67 -0
- package/dist/providers/email/sendgrid.js.map +1 -0
- package/dist/providers/email/smtp.d.ts +65 -0
- package/dist/providers/email/smtp.d.ts.map +1 -0
- package/dist/providers/email/smtp.js +373 -0
- package/dist/providers/email/smtp.js.map +1 -0
- package/dist/providers/gemini.d.ts +11 -1
- package/dist/providers/gemini.d.ts.map +1 -1
- package/dist/providers/gemini.js +38 -4
- package/dist/providers/gemini.js.map +1 -1
- package/dist/providers/none.d.ts +11 -1
- package/dist/providers/none.d.ts.map +1 -1
- package/dist/providers/none.js +21 -0
- package/dist/providers/none.js.map +1 -1
- package/dist/providers/opencode.d.ts +11 -1
- package/dist/providers/opencode.d.ts.map +1 -1
- package/dist/providers/opencode.js +55 -4
- package/dist/providers/opencode.js.map +1 -1
- package/dist/providers/provider.d.ts +77 -0
- package/dist/providers/provider.d.ts.map +1 -1
- package/dist/providers/provider.js +44 -0
- package/dist/providers/provider.js.map +1 -1
- package/dist/services/agent-provisioner.d.ts.map +1 -1
- package/dist/services/agent-provisioner.js +3 -6
- package/dist/services/agent-provisioner.js.map +1 -1
- package/dist/services/cloud/lifecycle.d.ts.map +1 -1
- package/dist/services/cloud/lifecycle.js +3 -0
- package/dist/services/cloud/lifecycle.js.map +1 -1
- package/dist/services/cloud/task-wrapper.d.ts +18 -0
- package/dist/services/cloud/task-wrapper.d.ts.map +1 -1
- package/dist/services/cloud/task-wrapper.js +157 -0
- package/dist/services/cloud/task-wrapper.js.map +1 -1
- package/dist/services/github-app.d.ts.map +1 -1
- package/dist/services/github-app.js +11 -2
- package/dist/services/github-app.js.map +1 -1
- package/dist/services/knowledge/audn.d.ts +37 -0
- package/dist/services/knowledge/audn.d.ts.map +1 -0
- package/dist/services/knowledge/audn.js +219 -0
- package/dist/services/knowledge/audn.js.map +1 -0
- package/dist/services/knowledge/file-hash.d.ts +11 -0
- package/dist/services/knowledge/file-hash.d.ts.map +1 -0
- package/dist/services/knowledge/file-hash.js +123 -0
- package/dist/services/knowledge/file-hash.js.map +1 -0
- package/dist/services/knowledge/http-provider.d.ts +46 -0
- package/dist/services/knowledge/http-provider.d.ts.map +1 -0
- package/dist/services/knowledge/http-provider.js +254 -0
- package/dist/services/knowledge/http-provider.js.map +1 -0
- package/dist/services/knowledge/kb-providers.d.ts +15 -0
- package/dist/services/knowledge/kb-providers.d.ts.map +1 -0
- package/dist/services/knowledge/kb-providers.js +92 -0
- package/dist/services/knowledge/kb-providers.js.map +1 -0
- package/dist/services/knowledge/kb-scope-input.d.ts +5 -0
- package/dist/services/knowledge/kb-scope-input.d.ts.map +1 -0
- package/dist/services/knowledge/kb-scope-input.js +11 -0
- package/dist/services/knowledge/kb-scope-input.js.map +1 -0
- package/dist/services/knowledge/kb-service.d.ts +3 -0
- package/dist/services/knowledge/kb-service.d.ts.map +1 -0
- package/dist/services/knowledge/kb-service.js +2 -0
- package/dist/services/knowledge/kb-service.js.map +1 -0
- package/dist/services/knowledge/path-validation.d.ts +2 -0
- package/dist/services/knowledge/path-validation.d.ts.map +1 -0
- package/dist/services/knowledge/path-validation.js +13 -0
- package/dist/services/knowledge/path-validation.js.map +1 -0
- package/dist/services/knowledge/pre-init.d.ts +14 -0
- package/dist/services/knowledge/pre-init.d.ts.map +1 -0
- package/dist/services/knowledge/pre-init.js +147 -0
- package/dist/services/knowledge/pre-init.js.map +1 -0
- package/dist/services/knowledge/project-slug.d.ts +2 -0
- package/dist/services/knowledge/project-slug.d.ts.map +1 -0
- package/dist/services/knowledge/project-slug.js +49 -0
- package/dist/services/knowledge/project-slug.js.map +1 -0
- package/dist/services/knowledge/sqlite-provider.d.ts +147 -0
- package/dist/services/knowledge/sqlite-provider.d.ts.map +1 -0
- package/dist/services/knowledge/sqlite-provider.js +1646 -0
- package/dist/services/knowledge/sqlite-provider.js.map +1 -0
- package/dist/services/knowledge/types.d.ts +163 -0
- package/dist/services/knowledge/types.d.ts.map +1 -0
- package/dist/services/knowledge/types.js +9 -0
- package/dist/services/knowledge/types.js.map +1 -0
- package/dist/services/member-home.d.ts +89 -0
- package/dist/services/member-home.d.ts.map +1 -0
- package/dist/services/member-home.js +256 -0
- package/dist/services/member-home.js.map +1 -0
- package/dist/services/member-remote-url.d.ts +34 -0
- package/dist/services/member-remote-url.d.ts.map +1 -0
- package/dist/services/member-remote-url.js +39 -0
- package/dist/services/member-remote-url.js.map +1 -0
- package/dist/services/orphan-recovery.d.ts +5 -2
- package/dist/services/orphan-recovery.d.ts.map +1 -1
- package/dist/services/orphan-recovery.js +19 -8
- package/dist/services/orphan-recovery.js.map +1 -1
- package/dist/services/preflight-check.d.ts +56 -0
- package/dist/services/preflight-check.d.ts.map +1 -0
- package/dist/services/preflight-check.js +277 -0
- package/dist/services/preflight-check.js.map +1 -0
- package/dist/services/stall/log-path-resolver.d.ts +11 -8
- package/dist/services/stall/log-path-resolver.d.ts.map +1 -1
- package/dist/services/stall/log-path-resolver.js +17 -40
- package/dist/services/stall/log-path-resolver.js.map +1 -1
- package/dist/services/stall/stall-detector.d.ts +32 -0
- package/dist/services/stall/stall-detector.d.ts.map +1 -1
- package/dist/services/stall/stall-detector.js +149 -6
- package/dist/services/stall/stall-detector.js.map +1 -1
- package/dist/services/stall/stall-poller.d.ts +34 -0
- package/dist/services/stall/stall-poller.d.ts.map +1 -1
- package/dist/services/stall/stall-poller.js +134 -2
- package/dist/services/stall/stall-poller.js.map +1 -1
- package/dist/services/strategy.d.ts.map +1 -1
- package/dist/services/strategy.js +2 -2
- package/dist/services/strategy.js.map +1 -1
- package/dist/services/tool-registry.d.ts.map +1 -1
- package/dist/services/tool-registry.js +87 -1
- package/dist/services/tool-registry.js.map +1 -1
- package/dist/services/vcs/github.d.ts.map +1 -1
- package/dist/services/vcs/github.js +32 -1
- package/dist/services/vcs/github.js.map +1 -1
- package/dist/services/version-check.d.ts +22 -0
- package/dist/services/version-check.d.ts.map +1 -0
- package/dist/services/version-check.js +103 -0
- package/dist/services/version-check.js.map +1 -0
- package/dist/skills/fleet-sprint-cli/SKILL.md +13 -2
- package/dist/skills/pm/beads.md +6 -0
- package/dist/skills/pm/fleet-addendum.md +6 -0
- package/dist/skills/pm/index.md +56 -0
- package/dist/skills/pm/kb-reconcile.md +132 -0
- package/dist/skills/pm/kb-review.md +109 -0
- package/dist/tools/check-status.d.ts +47 -0
- package/dist/tools/check-status.d.ts.map +1 -1
- package/dist/tools/check-status.js +242 -0
- package/dist/tools/check-status.js.map +1 -1
- package/dist/tools/child-id-allocator.d.ts +2 -2
- package/dist/tools/code-intelligence-codebase-memory.d.ts +11 -0
- package/dist/tools/code-intelligence-codebase-memory.d.ts.map +1 -0
- package/dist/tools/code-intelligence-codebase-memory.js +270 -0
- package/dist/tools/code-intelligence-codebase-memory.js.map +1 -0
- package/dist/tools/code-intelligence-freshness.d.ts +2 -0
- package/dist/tools/code-intelligence-freshness.d.ts.map +1 -0
- package/dist/tools/code-intelligence-freshness.js +26 -0
- package/dist/tools/code-intelligence-freshness.js.map +1 -0
- package/dist/tools/code-intelligence-gitnexus.d.ts +14 -0
- package/dist/tools/code-intelligence-gitnexus.d.ts.map +1 -0
- package/dist/tools/code-intelligence-gitnexus.js +475 -0
- package/dist/tools/code-intelligence-gitnexus.js.map +1 -0
- package/dist/tools/code-intelligence-kb-enrich.d.ts +2 -0
- package/dist/tools/code-intelligence-kb-enrich.d.ts.map +1 -0
- package/dist/tools/code-intelligence-kb-enrich.js +65 -0
- package/dist/tools/code-intelligence-kb-enrich.js.map +1 -0
- package/dist/tools/code-intelligence-reindex.d.ts +7 -0
- package/dist/tools/code-intelligence-reindex.d.ts.map +1 -0
- package/dist/tools/code-intelligence-reindex.js +106 -0
- package/dist/tools/code-intelligence-reindex.js.map +1 -0
- package/dist/tools/code-intelligence-telemetry.d.ts +5 -0
- package/dist/tools/code-intelligence-telemetry.d.ts.map +1 -0
- package/dist/tools/code-intelligence-telemetry.js +52 -0
- package/dist/tools/code-intelligence-telemetry.js.map +1 -0
- package/dist/tools/code-intelligence-tests.d.ts +2 -0
- package/dist/tools/code-intelligence-tests.d.ts.map +1 -0
- package/dist/tools/code-intelligence-tests.js +24 -0
- package/dist/tools/code-intelligence-tests.js.map +1 -0
- package/dist/tools/code-intelligence.d.ts +114 -0
- package/dist/tools/code-intelligence.d.ts.map +1 -0
- package/dist/tools/code-intelligence.js +124 -0
- package/dist/tools/code-intelligence.js.map +1 -0
- package/dist/tools/compose-permissions.d.ts +26 -0
- package/dist/tools/compose-permissions.d.ts.map +1 -1
- package/dist/tools/compose-permissions.js +244 -33
- package/dist/tools/compose-permissions.js.map +1 -1
- package/dist/tools/execute-command.d.ts +2 -0
- package/dist/tools/execute-command.d.ts.map +1 -1
- package/dist/tools/execute-command.js +86 -16
- package/dist/tools/execute-command.js.map +1 -1
- package/dist/tools/execute-prompt.d.ts +13 -1
- package/dist/tools/execute-prompt.d.ts.map +1 -1
- package/dist/tools/execute-prompt.js +257 -45
- package/dist/tools/execute-prompt.js.map +1 -1
- package/dist/tools/kb-capture.d.ts +53 -0
- package/dist/tools/kb-capture.d.ts.map +1 -0
- package/dist/tools/kb-capture.js +130 -0
- package/dist/tools/kb-capture.js.map +1 -0
- package/dist/tools/kb-context.d.ts +17 -0
- package/dist/tools/kb-context.d.ts.map +1 -0
- package/dist/tools/kb-context.js +29 -0
- package/dist/tools/kb-context.js.map +1 -0
- package/dist/tools/kb-export.d.ts +19 -0
- package/dist/tools/kb-export.d.ts.map +1 -0
- package/dist/tools/kb-export.js +344 -0
- package/dist/tools/kb-export.js.map +1 -0
- package/dist/tools/kb-feedback.d.ts +23 -0
- package/dist/tools/kb-feedback.d.ts.map +1 -0
- package/dist/tools/kb-feedback.js +43 -0
- package/dist/tools/kb-feedback.js.map +1 -0
- package/dist/tools/kb-freshness-sweep.d.ts +14 -0
- package/dist/tools/kb-freshness-sweep.d.ts.map +1 -0
- package/dist/tools/kb-freshness-sweep.js +29 -0
- package/dist/tools/kb-freshness-sweep.js.map +1 -0
- package/dist/tools/kb-harvest.d.ts +20 -0
- package/dist/tools/kb-harvest.d.ts.map +1 -0
- package/dist/tools/kb-harvest.js +143 -0
- package/dist/tools/kb-harvest.js.map +1 -0
- package/dist/tools/kb-import.d.ts +44 -0
- package/dist/tools/kb-import.d.ts.map +1 -0
- package/dist/tools/kb-import.js +239 -0
- package/dist/tools/kb-import.js.map +1 -0
- package/dist/tools/kb-invalidate.d.ts +19 -0
- package/dist/tools/kb-invalidate.d.ts.map +1 -0
- package/dist/tools/kb-invalidate.js +34 -0
- package/dist/tools/kb-invalidate.js.map +1 -0
- package/dist/tools/kb-list.d.ts +32 -0
- package/dist/tools/kb-list.d.ts.map +1 -0
- package/dist/tools/kb-list.js +43 -0
- package/dist/tools/kb-list.js.map +1 -0
- package/dist/tools/kb-promote.d.ts +20 -0
- package/dist/tools/kb-promote.d.ts.map +1 -0
- package/dist/tools/kb-promote.js +20 -0
- package/dist/tools/kb-promote.js.map +1 -0
- package/dist/tools/kb-query.d.ts +35 -0
- package/dist/tools/kb-query.d.ts.map +1 -0
- package/dist/tools/kb-query.js +111 -0
- package/dist/tools/kb-query.js.map +1 -0
- package/dist/tools/kb-reconcile-prefilter.d.ts +14 -0
- package/dist/tools/kb-reconcile-prefilter.d.ts.map +1 -0
- package/dist/tools/kb-reconcile-prefilter.js +26 -0
- package/dist/tools/kb-reconcile-prefilter.js.map +1 -0
- package/dist/tools/kb-resolve-contradiction.d.ts +23 -0
- package/dist/tools/kb-resolve-contradiction.d.ts.map +1 -0
- package/dist/tools/kb-resolve-contradiction.js +37 -0
- package/dist/tools/kb-resolve-contradiction.js.map +1 -0
- package/dist/tools/kb-session-prime.d.ts +26 -0
- package/dist/tools/kb-session-prime.d.ts.map +1 -0
- package/dist/tools/kb-session-prime.js +392 -0
- package/dist/tools/kb-session-prime.js.map +1 -0
- package/dist/tools/kb-setup.d.ts +20 -0
- package/dist/tools/kb-setup.d.ts.map +1 -0
- package/dist/tools/kb-setup.js +49 -0
- package/dist/tools/kb-setup.js.map +1 -0
- package/dist/tools/kb-stats.d.ts +20 -0
- package/dist/tools/kb-stats.d.ts.map +1 -0
- package/dist/tools/kb-stats.js +115 -0
- package/dist/tools/kb-stats.js.map +1 -0
- package/dist/tools/member-detail.d.ts.map +1 -1
- package/dist/tools/member-detail.js +8 -0
- package/dist/tools/member-detail.js.map +1 -1
- package/dist/tools/monitor-task.d.ts.map +1 -1
- package/dist/tools/monitor-task.js +21 -3
- package/dist/tools/monitor-task.js.map +1 -1
- package/dist/tools/provision-auth.d.ts.map +1 -1
- package/dist/tools/provision-auth.js +10 -3
- package/dist/tools/provision-auth.js.map +1 -1
- package/dist/tools/register-member.d.ts +3 -0
- package/dist/tools/register-member.d.ts.map +1 -1
- package/dist/tools/register-member.js +30 -1
- package/dist/tools/register-member.js.map +1 -1
- package/dist/tools/remove-member.d.ts.map +1 -1
- package/dist/tools/remove-member.js +17 -3
- package/dist/tools/remove-member.js.map +1 -1
- package/dist/tools/report-status.d.ts.map +1 -1
- package/dist/tools/report-status.js +2 -1
- package/dist/tools/report-status.js.map +1 -1
- package/dist/tools/send-email.d.ts +69 -0
- package/dist/tools/send-email.d.ts.map +1 -0
- package/dist/tools/send-email.js +177 -0
- package/dist/tools/send-email.js.map +1 -0
- package/dist/tools/setup-ssh-key.d.ts.map +1 -1
- package/dist/tools/setup-ssh-key.js +2 -0
- package/dist/tools/setup-ssh-key.js.map +1 -1
- package/dist/tools/update-member.d.ts +3 -0
- package/dist/tools/update-member.d.ts.map +1 -1
- package/dist/tools/update-member.js +25 -1
- package/dist/tools/update-member.js.map +1 -1
- package/dist/types.d.ts +1 -0
- package/dist/types.d.ts.map +1 -1
- package/dist/utils/prompt-errors.js +1 -1
- package/dist/utils/prompt-errors.js.map +1 -1
- package/dist/utils/session-caller.d.ts +28 -0
- package/dist/utils/session-caller.d.ts.map +1 -0
- package/dist/utils/session-caller.js +24 -0
- package/dist/utils/session-caller.js.map +1 -0
- package/dist/utils/work-folder-validation.d.ts +39 -0
- package/dist/utils/work-folder-validation.d.ts.map +1 -0
- package/dist/utils/work-folder-validation.js +60 -0
- package/dist/utils/work-folder-validation.js.map +1 -0
- package/dist/workflows/auto-sprint.js +546 -6
- package/package.json +7 -4
- package/packages/apra-fleet-client/src/client/api.mjs +118 -4
- package/packages/apra-fleet-se/apra-pm/agents/backlog-groomer.md +17 -1
- package/packages/apra-fleet-se/apra-pm/agents/ci-watcher.md +13 -1
- package/packages/apra-fleet-se/apra-pm/agents/deployer.md +15 -3
- package/packages/apra-fleet-se/apra-pm/agents/doer.md +35 -1
- package/packages/apra-fleet-se/apra-pm/agents/harvester.md +22 -3
- package/packages/apra-fleet-se/apra-pm/agents/integ-test-runner.md +18 -2
- package/packages/apra-fleet-se/apra-pm/agents/kb-reconciler.md +216 -0
- package/packages/apra-fleet-se/apra-pm/agents/plan-reviewer.md +14 -1
- package/packages/apra-fleet-se/apra-pm/agents/planner.md +88 -5
- package/packages/apra-fleet-se/apra-pm/agents/regression-test-runner.md +19 -1
- package/packages/apra-fleet-se/apra-pm/agents/reviewer.md +104 -3
- package/packages/apra-fleet-se/apra-pm/agents/schemas/doer-output.json +63 -3
- package/packages/apra-fleet-se/apra-pm/agents/schemas/harvester-output.json +59 -2
- package/packages/apra-fleet-se/apra-pm/agents/schemas/integ-test-runner-output.json +2 -2
- package/packages/apra-fleet-se/apra-pm/agents/schemas/kb-reconciler-output.json +87 -0
- package/packages/apra-fleet-se/apra-pm/agents/schemas/planner-output.json +65 -0
- package/packages/apra-fleet-se/apra-pm/agents/schemas/reviewer-output.json +105 -8
- package/packages/apra-fleet-se/apra-pm/skills/pm/beads.md +6 -0
- package/packages/apra-fleet-se/apra-pm/skills/pm/fleet-addendum.md +6 -0
- package/packages/apra-fleet-se/apra-pm/skills/pm/index.md +56 -0
- package/packages/apra-fleet-se/apra-pm/skills/pm/kb-reconcile.md +132 -0
- package/packages/apra-fleet-se/apra-pm/skills/pm/kb-review.md +109 -0
- package/packages/apra-fleet-se/bin/cli.mjs +7 -0
- package/packages/apra-fleet-se/bin/serve.mjs +14 -2
- package/packages/apra-fleet-se/fleet-sprint/contracts.mjs +31 -5
- package/packages/apra-fleet-se/fleet-sprint/docs/README.md +1 -1
- package/packages/apra-fleet-se/fleet-sprint/docs/coach-role-design.md +23 -151
- package/packages/apra-fleet-se/fleet-sprint/docs/dolt-manual-recovery-verified.md +256 -0
- package/packages/apra-fleet-se/fleet-sprint/docs/dolt-sync-redesign.md +1482 -0
- package/packages/apra-fleet-se/fleet-sprint/docs/escalate-to-llm-design.md +908 -0
- package/packages/apra-fleet-se/fleet-sprint/docs/shared-orchestrator-reservation-design.md +399 -0
- package/packages/apra-fleet-se/fleet-sprint/dolt-settle.mjs +972 -0
- package/packages/apra-fleet-se/fleet-sprint/dolt-sync.mjs +167 -101
- package/packages/apra-fleet-se/fleet-sprint/errors.mjs +50 -6
- package/packages/apra-fleet-se/fleet-sprint/runner.js +1416 -116
- package/packages/apra-fleet-se/fleet-sprint/skills/fleet-sprint-cli/SKILL.md +13 -2
- package/packages/apra-fleet-se/fleet-sprint/skills/fleet-supervisor/SKILL.md +350 -0
- package/packages/apra-fleet-se/fleet-sprint/sprint-progress.mjs +101 -0
- package/packages/apra-fleet-se/fleet-sprint/vcs-providers/github.mjs +30 -17
- package/packages/apra-fleet-se/fleet-sprint/viewer-extensions.mjs +62 -2
- package/packages/apra-fleet-se/package.json +1 -0
- package/packages/apra-fleet-workflow/src/viewer/index.mjs +127 -13
- package/packages/apra-fleet-workflow/src/viewer/run-title.mjs +60 -0
- package/packages/apra-fleet-workflow/src/workflow/index.mjs +8 -4
- package/skills/fleet/knowledge-agent.md +66 -0
- package/skills/fleet/permissions.md +28 -4
- package/skills/fleet/profiles/base-dev.json +10 -1
- package/skills/fleet/profiles/base-reviewer.json +8 -1
- package/skills/fleet/profiles/tag-kb-reconciler.json +20 -0
- package/version.json +1 -1
- package/packages/apra-fleet-se/fleet-sprint/docs/dolt-tier2-runbook.md +0 -127
- package/packages/apra-fleet-se/fleet-sprint/dolt-recovery-path-b.mjs +0 -339
- package/packages/apra-fleet-se/fleet-sprint/dolt-recovery-tier2.mjs +0 -360
- package/packages/apra-fleet-se/fleet-sprint/dolt-recovery.mjs +0 -363
|
@@ -0,0 +1,908 @@
|
|
|
1
|
+
# sprint-doctor: an escalateToLLM() mechanism for fleet-sprint
|
|
2
|
+
|
|
3
|
+
Design doc. Source path: `packages/apra-fleet-se/fleet-sprint/docs/escalate-to-llm-design.md`
|
|
4
|
+
Status: design -- not yet implemented. Last updated: 2026-08-10.
|
|
5
|
+
|
|
6
|
+
**sprint-doctor's job in one line:** turn a sick sprint into either a healthy sprint (bounded
|
|
7
|
+
auto-repair, then resume) or a cleanly dead one (abort with a diagnosis report and, where needed,
|
|
8
|
+
a specific human referral) -- never leave it wedged, silently burning tokens, or ambiguous.
|
|
9
|
+
|
|
10
|
+
> **Supersedes `coach-role-design.md`** (the 2026-07-20 "coach role: just-in-time error
|
|
11
|
+
> resolution" design note, now reduced to a pointer here). sprint-doctor is the same core idea --
|
|
12
|
+
> an in-workflow LLM layer for failures the deterministic engine did not foresee, positioned
|
|
13
|
+
> strictly AFTER typed handlers/ladders, failing loudly when it cannot fix -- carried further.
|
|
14
|
+
> Ideas incorporated directly from coach, credited inline where they appear: the three-layer
|
|
15
|
+
> error-handling hierarchy and red-state interception taxonomy (section 1), the
|
|
16
|
+
> same-class-twice fingerprint cap and single-shot/no-recursion rules (section 6), the
|
|
17
|
+
> rescue-WIP-to-branch salvage action (section 2.5), interventions as first-class final-review
|
|
18
|
+
> evidence and deterministic test stubbing (section 6), and the consent-gated, sanitized
|
|
19
|
+
> engine-flaw telemetry flow (section 4.4). Where the two designs differed, this doc's answer
|
|
20
|
+
> stands: decision/action separation with a zero-tool consult instead of a tool-bearing coach
|
|
21
|
+
> agent (section 2.1), an explicit environment-vs-engine-vs-task-shape classification coach did
|
|
22
|
+
> not have (section 3), a data-driven symptom/remedy registry resolving coach's open
|
|
23
|
+
> "self-priming knowledge base" question (section 4), and pattern-driven stagnation triggers on
|
|
24
|
+
> top of coach's event-driven ones (section 1).
|
|
25
|
+
|
|
26
|
+
---
|
|
27
|
+
|
|
28
|
+
## 0. Grounding: the two incidents and what the code does today
|
|
29
|
+
|
|
30
|
+
**Incident 1 (blunt sprint-wide kill).** `runner.js` stall detection: `STALL_CYCLE_LIMIT = 2`
|
|
31
|
+
(runner.js:5976), high-water-mark progress score = closed count + monotone verify-routed set
|
|
32
|
+
(runner.js:8180-8190), and at `staleCycles >= STALL_CYCLE_LIMIT` it throws `StalledSprintError`
|
|
33
|
+
(runner.js:8192-8227, class in errors.mjs:139-156). The throw is deliberately never caught inside
|
|
34
|
+
the cycle loop (errors.mjs:126-128); `isTypedAbortError()` (runner.js:4200-4210) routes it to
|
|
35
|
+
`finalizeAbort()`'s branch push + `[ABORTED]` PR. There is no diagnosis step between "no new
|
|
36
|
+
high-water mark twice" and "kill the whole sprint" -- no attempt to defer one bead, swap a member,
|
|
37
|
+
or distinguish a broken verifier from genuine out-of-work.
|
|
38
|
+
|
|
39
|
+
**Incident 2 (12 identical stalls on one bead).** Server-side, a confirmed stall kills the remote
|
|
40
|
+
pid and fails the dispatch with `details.reason: 'stalled'` (src/tools/execute-prompt.ts:1378-1385,
|
|
41
|
+
threshold in src/services/stall/stall-detector.ts:96, default `STALL_THRESHOLD_MS` 120s). Engine-side,
|
|
42
|
+
`'stalled'` is classified as an infra dispatch failure (`INFRA_DISPATCH_REASONS`, errors.mjs:427-438).
|
|
43
|
+
The doer catch path (runner.js:7217-7331) retries a generic throw exactly once in-cycle
|
|
44
|
+
(runner.js:7314-7330), attributes the failure per bead (7333-7377), and re-lanes the bead next
|
|
45
|
+
round/cycle -- where the identical dispatch is rebuilt and stalls again. Nothing counts *per-bead
|
|
46
|
+
failure streaks across cycles*, so as long as anything else in scope keeps setting new high-water
|
|
47
|
+
marks (or Plan/Review rounds churn), the same bead can burn premium dispatches indefinitely. A human
|
|
48
|
+
eventually ran `bd defer <bead> --reason ...` by hand -- `bd defer --reason` is a stock beads verb
|
|
49
|
+
(verified against `bd defer --help`), so the manual fix was always expressible in the orchestrator's
|
|
50
|
+
own vocabulary. ~15.8M output tokens were burned first.
|
|
51
|
+
|
|
52
|
+
**A caution the design must honor:** incident 2's root cause is NOT established.
|
|
53
|
+
"The bead was too big" (task shape) and "the member's remote Claude CLI session / environment was
|
|
54
|
+
wedged" (environment) are *competing hypotheses* the engine never tested -- e.g. by re-dispatching
|
|
55
|
+
the same bead on a different member. sprint-doctor's triage (section 3) exists precisely to
|
|
56
|
+
discriminate these instead of assuming either.
|
|
57
|
+
|
|
58
|
+
**Existing bounded-recovery precedents this design copies structurally:**
|
|
59
|
+
|
|
60
|
+
- *VCS-auth self-heal* (`runGitStep()`, runner.js:701-729; heal callback runner.js:1928-1937;
|
|
61
|
+
provision runner.js:1616-1659): classify failure from tool output; a one-shot boolean latch
|
|
62
|
+
(`authHealAttempted`) guarantees the heal fires at most once; heal must positively confirm
|
|
63
|
+
success ("a failed provision must never... report success", runner.js:1654-1656); exactly one
|
|
64
|
+
retry of the original operation; every transition logged with a distinguishing prefix.
|
|
65
|
+
- *Dolt conflict ladder (historical)*: the scripted-rung, agent-dispatch-armed-with-a-runbook
|
|
66
|
+
ladder this bullet used to cite (`dolt-recovery.mjs` et al.) was retired and deleted on branch
|
|
67
|
+
`fix/dolt-settle-recovery`. It is replaced by the deterministic `settleDoltConflicts()`
|
|
68
|
+
(`fleet-sprint/dolt-settle.mjs`), which resolves every row-level conflict shape with zero LLM
|
|
69
|
+
dispatch. Dolt/beads-sync wedging is explicitly out of scope for sprint-doctor -- see the
|
|
70
|
+
registry exclusion note in section 4 (symptom/remedy registry) and
|
|
71
|
+
`fleet-sprint/docs/dolt-sync-redesign.md` (Parts 2, 7.4, 8.4).
|
|
72
|
+
- *LLM-auth self-heal* on non-retryable doer failures: one `provision_llm_auth`-backed heal + one
|
|
73
|
+
bounded retry (runner.js:7290-7313).
|
|
74
|
+
- *Structured-verdict contracts*: every role dispatch passes an ajv schema from `contracts.mjs`
|
|
75
|
+
(schemas vendored under `apra-pm/agents/schemas/<role>-output.json`, resolved at
|
|
76
|
+
contracts.mjs:630-637, validated at 706-713), and the runner -- not the agent -- applies verdict
|
|
77
|
+
side effects ("NEVER touch beads yourself... the orchestrator applies your newTasks and
|
|
78
|
+
reopenIds", runner.js:3955-3957). sprint-doctor is a new role in exactly this mold.
|
|
79
|
+
|
|
80
|
+
**Engine integration surface** (packages/apra-fleet-workflow): phases are *imperative labels*, not
|
|
81
|
+
registered objects -- `phase(title)`/`group(title)` set async-local state and emit events
|
|
82
|
+
(index.mjs:510-550); the dashboard subscribes to those events (viewer/index.mjs:1052-1124), so a
|
|
83
|
+
new `phase('Sprint Doctor C3')` appears on the dashboard with zero viewer changes. There is no
|
|
84
|
+
middleware/phase-registry to extend; the correct integration is plain code at chosen points inside
|
|
85
|
+
`runSprintCycle()` (runner.js:4636) plus `publishState('sprint-doctor', ...)` for structured
|
|
86
|
+
dashboard data (index.mjs:553, viewer/index.mjs:1119).
|
|
87
|
+
|
|
88
|
+
---
|
|
89
|
+
|
|
90
|
+
## 1. Trigger / observation layer
|
|
91
|
+
|
|
92
|
+
### 1.1 What to record: a SprintHealthLedger
|
|
93
|
+
|
|
94
|
+
A small in-memory (plus JSONL-appended, for post-mortem) ledger owned by `runSprintCycle()`,
|
|
95
|
+
populated at the dispatch catch sites that already exist -- no new instrumentation planes:
|
|
96
|
+
|
|
97
|
+
```
|
|
98
|
+
recordDispatch({
|
|
99
|
+
cycle, phaseLabel, role, member, beadIds[], // position
|
|
100
|
+
ok, reason, // reason = err.details.reason when present
|
|
101
|
+
errorSignature, // normalized: reason + first line of message,
|
|
102
|
+
// ids/numbers/paths stripped
|
|
103
|
+
durationS, tier, costUsd, // from activity/pricing data already tracked
|
|
104
|
+
})
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
Record points (all existing catch/outcome sites):
|
|
108
|
+
- doer streak attribution (runner.js:7333-7377, `streakOutcomes`),
|
|
109
|
+
- planner retry ladder (runner.js:6244-6280),
|
|
110
|
+
- reviewer / plan-reviewer degraded rounds (runner.js:5468-5497, 6339-6420),
|
|
111
|
+
- deploy / integ-test infra failures (runner.js:7975-7999),
|
|
112
|
+
- the cycle-evaluation counters that already exist: `staleCycles`, `highWaterClosedCount`,
|
|
113
|
+
`closedCountHistory` (runner.js:5976-5979), `reopenCounts` (5987), `verifyDispatchAttempts/
|
|
114
|
+
Closures` (6014-6015), `verifyEverIds` (6002).
|
|
115
|
+
|
|
116
|
+
The `errorSignature` normalization is the load-bearing part: "identical signature repeated" vs
|
|
117
|
+
"different failures each time" is the primary environment-vs-engine discriminator (section 3).
|
|
118
|
+
|
|
119
|
+
### 1.2 Triggers (all must be cheap boolean checks over the ledger)
|
|
120
|
+
|
|
121
|
+
| id | Trigger | Default threshold | Catches | Why not lower/higher |
|
|
122
|
+
|----|---------|-------------------|---------|----------------------|
|
|
123
|
+
| T1 | Same bead(set) accumulates consecutive *infra-reason* dispatch failures (`INFRA_DISPATCH_REASONS` + `watchdog_timeout`) with zero closures, counted across cycles | 3 attempts | Incident 2 at attempt 3 instead of 12 | 1-2 would fire on transient blips the existing retry-once already absorbs; the existing in-cycle retry means 3 attempts ~= 2 cycles of failure |
|
|
124
|
+
| T2 | Same member accumulates consecutive infra failures across >= 2 *distinct* beads | 3 failures | A wedged member (hung CLI, dead session, disk) poisoning unrelated work | Requires bead diversity so a single hard bead can't impersonate a sick member |
|
|
125
|
+
| T3 | Cycle stagnation: `staleCycles` reaches `STALL_CYCLE_LIMIT` (runner.js:8192) | interpose once, before the throw | Incident 1: consult before killing the sprint | Reuses the tuned existing detector; doctor gets exactly one chance, then the original abort fires |
|
|
126
|
+
| T4 | Spend-without-progress backstop: `budget.spent()` delta since the last new high-water mark exceeds `doctor_spend_trigger_usd` | min(15% of `budget.total`, $25); $25 flat when budget unset | Token burn that outruns cycle counters (the 15.8M-token shape) | Uses the existing budget plumbing (`context.budget`, BudgetExceededError check in workflow index.mjs:760-765) |
|
|
127
|
+
| T5 | Unforeseen red state: a typed handler/retry ladder exhausted itself on a failure class with NO recognized signature and no registry match -- unrecognized dispatch reasons after existing retries, sync-bracket failures the dolt/git ladders could not close, deploy/integ/harvester phase crashes | single-shot: one consult per red event | The failures nobody predicted -- the deterministic engine's blind spots | Event-driven, not pattern-driven: firing once immediately is correct because layer-1 handling has already been exhausted by definition |
|
|
128
|
+
|
|
129
|
+
T1-T4 are pattern-driven (repetition/stagnation); T5 is event-driven and comes from the earlier
|
|
130
|
+
coach design, whose error-handling hierarchy this layer preserves verbatim: (1) typed handlers
|
|
131
|
+
and ladders -- deterministic, free, NEVER preempted by the doctor; (2) doctor -- JIT judgment,
|
|
132
|
+
tokens, invoked only when layer 1 has no match or its handling failed; (3) fail loudly --
|
|
133
|
+
if the doctor cannot resolve or its caps are hit, the normal failure path proceeds unchanged,
|
|
134
|
+
and the doctor never suppresses a failure it could not fix. Coach's red-state interception
|
|
135
|
+
taxonomy (streak failure after ladder exhaustion, unrecognized dispatch reasons, sync failures
|
|
136
|
+
after Tier-2, phase-level crashes, contract violations after built-in retry, member
|
|
137
|
+
unreachable, auth expiry mid-run, and workflow-level aborts -- the last as evidence collector
|
|
138
|
+
even when unresolvable) is the seed list of T5 interception points, wired through the same H1
|
|
139
|
+
record-only sites plus the terminal catch in `main()` for the evidence-collector case.
|
|
140
|
+
|
|
141
|
+
False-positive guards:
|
|
142
|
+
- No trigger except T1 may fire in a cycle whose `progressScore` set a new high-water mark
|
|
143
|
+
(a slow-but-progressing sprint is left alone).
|
|
144
|
+
- T1 additionally requires all attempts in the streak be infra-reason failures OR share one
|
|
145
|
+
`errorSignature` -- a bead failing three times with *different, substantive* errors (schema
|
|
146
|
+
invalid, then review reopen, then test failure) is the develop/review loop working, not a stall.
|
|
147
|
+
- All four triggers are debounced by the consult ledger: a (trigger, bead-or-member) pair that
|
|
148
|
+
already produced a consult this sprint cannot re-fire unless the doctor's prescribed action was
|
|
149
|
+
executed and a new failure occurred *after* it.
|
|
150
|
+
|
|
151
|
+
### 1.3 Hook points (cheapest given the engine's shape)
|
|
152
|
+
|
|
153
|
+
Since phases are imperative (no registry -- workflow index.mjs:510-550), hooks are code:
|
|
154
|
+
|
|
155
|
+
- **H1 (record-only):** inside the existing catch/attribution sites listed in 1.1. No dispatching
|
|
156
|
+
from inside `parallel()` branches -- H1 only sets a `pendingConsult` flag with its evidence.
|
|
157
|
+
(Dispatching the doctor from inside a parallel doer branch would interleave with sibling
|
|
158
|
+
streaks and violate the per-branch async-local phase store, index.mjs:1435-1471.)
|
|
159
|
+
- **H2 (consult point):** Cycle Evaluation (runner.js:8110-8238), immediately after the fresh
|
|
160
|
+
`closedBeadsNow` read and progress-score update -- the one serialized, fresh-state point per
|
|
161
|
+
cycle. Runs under `phase('Sprint Doctor C<n>')` so it is dashboard-visible for free.
|
|
162
|
+
- **H3 (abort interposition, T3):** replace the direct `throw new StalledSprintError(...)` at
|
|
163
|
+
runner.js:8219 with: consult once -> apply action -> allow at most one further cycle; if the
|
|
164
|
+
next evaluation still sets no new high-water mark, throw the original `StalledSprintError`
|
|
165
|
+
*with the doctor's verdict attached in `details`* so the abort report explains what was tried.
|
|
166
|
+
- **H4 (mid-cycle fast path for T1/T2 only):** after Develop's streak outcomes are folded
|
|
167
|
+
(post-`parallel()`, around runner.js:7498 before Review), so a bead on its 3rd consecutive
|
|
168
|
+
infra failure does not wait for the cycle boundary while Review/Deploy/IntegTest burn more
|
|
169
|
+
dispatches on a doomed cycle. Same consult machinery, same caps.
|
|
170
|
+
|
|
171
|
+
---
|
|
172
|
+
|
|
173
|
+
## 2. sprint-doctor phase contract
|
|
174
|
+
|
|
175
|
+
### 2.1 Decision architecture: doctor decides, orchestrator does
|
|
176
|
+
|
|
177
|
+
The single most important boundary decision: **the doctor's in-sprint decision call gets ZERO
|
|
178
|
+
tools.** The runner pre-assembles all evidence into the prompt; the doctor returns a
|
|
179
|
+
schema-validated verdict; the runner executes the chosen action through its own existing verbs.
|
|
180
|
+
This mirrors the reviewer contract (verdict data -> orchestrator applies, runner.js:3955-3957) and
|
|
181
|
+
(historically) the retired Tier-2 dolt ladder's posture of never letting a dispatch self-judge
|
|
182
|
+
success. Dolt/beads-sync wedging is no longer part of this precedent at all -- it is excluded from
|
|
183
|
+
sprint-doctor's scope entirely (section 4) and resolved deterministically by
|
|
184
|
+
`settleDoltConflicts()` (`fleet-sprint/dolt-settle.mjs`).
|
|
185
|
+
|
|
186
|
+
Why zero tools rather than a restricted toolset:
|
|
187
|
+
- It is the only enforcement that is structural rather than configurational. The fleet has a
|
|
188
|
+
member-harness permission mechanism (`compose_permissions`, src/tools/compose-permissions.ts:15-22,
|
|
189
|
+
writing profiles like `skills/fleet/profiles/base-reviewer.json` onto the member's own
|
|
190
|
+
`.claude/settings.local.json`) -- but `execute_command` itself has **no allowlist** (arbitrary
|
|
191
|
+
shell string, src/tools/execute-command.ts:26-34), and mutating a member's permission profile
|
|
192
|
+
mid-sprint for one dispatch would leak into every subsequent dispatch on that member. A
|
|
193
|
+
tool-less dispatch cannot edit fleet-sprint source, cannot edit the target project, cannot shell
|
|
194
|
+
anything, no matter what its prompt is tricked into wanting.
|
|
195
|
+
- It converts the prompt-injection surface (member log tails are attacker-influenceable text) from
|
|
196
|
+
"possible arbitrary action" into "possibly skewed choice among N enumerated actions", which the
|
|
197
|
+
engine further bounds (section 6).
|
|
198
|
+
|
|
199
|
+
Mechanically: the doctor is dispatched like every other role -- `agent()` -> `execute_prompt` at
|
|
200
|
+
tier `'premium'` -- as a new persona (`sprint-doctor.md` vendored beside the other role personas
|
|
201
|
+
in `apra-pm/agents/`, schema beside them in `apra-pm/agents/schemas/sprint-doctor-output.json`,
|
|
202
|
+
re-exported through contracts.mjs's `resolveOutputSchema` pattern, contracts.mjs:630-637). The
|
|
203
|
+
persona's frontmatter declares NO tools; `max_turns` small; the engine's schema-repair loop
|
|
204
|
+
(index.mjs:759-1079, `schemaRetries` default 2) is the only retry it gets.
|
|
205
|
+
|
|
206
|
+
**Probe round (bounded):** some diagnoses need one more observation. Rather than granting tools,
|
|
207
|
+
the verdict schema includes an optional `probes[]` request drawn from a closed enum; the runner
|
|
208
|
+
executes them and re-dispatches the doctor once (max ONE probe round per consult) with results
|
|
209
|
+
appended. Probe enum (all runner-executed, all read-only except the last):
|
|
210
|
+
|
|
211
|
+
```
|
|
212
|
+
member_cli_version <member> -> execute_command: provider CLI --version
|
|
213
|
+
member_workspace_state <member> -> execute_command: git status --porcelain && git log -1
|
|
214
|
+
member_disk_free <member> -> execute_command: platform-appropriate disk-free
|
|
215
|
+
member_session_state <member> -> fleet_status / member_detail snapshot
|
|
216
|
+
bd_show <id...> -> orchestrator-side bd show --json
|
|
217
|
+
log_tail <which> <kb> -> more of a log the runner already holds
|
|
218
|
+
cross_member_redispatch <beadId> -> NOT a probe; must be requested as the
|
|
219
|
+
retry_different_member ACTION (it costs a full dispatch)
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
### 2.2 Inputs (assembled by the runner, schema-checked via the validateRoleInput pattern, contracts.mjs:818+)
|
|
223
|
+
|
|
224
|
+
- Sprint position: branch, base, goal, cycle/`max_cycles`, current phase label, members + roleMap,
|
|
225
|
+
budget `{total, spent}` (runner.js:2356-2362 arg plumbing).
|
|
226
|
+
- Trigger record: which of T1-T5 fired, with its evidence rows from the health ledger.
|
|
227
|
+
- The triggering bead(s): full `bd show --json`, plus scope summary counts (open/in_progress/
|
|
228
|
+
blocked/deferred/closed at goal priority -- same reads the Cycle Evaluation already does,
|
|
229
|
+
runner.js:8134-8153).
|
|
230
|
+
- Dispatch/stall history for this sprint: the ledger rows for the bead and member involved, plus
|
|
231
|
+
per-member aggregate failure counts and `errorSignature` frequency table across ALL members
|
|
232
|
+
(this cross-member table is what makes environment-vs-engine triage possible).
|
|
233
|
+
- Log tails: last N KB of the runner's own sprint log; last N KB of the relevant member-side
|
|
234
|
+
dispatch output the server already captured. Sizes capped (default 16KB each) to bound cost.
|
|
235
|
+
- Doctor's own consult history this sprint (verdicts + outcomes) -- required for the no-repeat
|
|
236
|
+
circuit-breaker and so it never re-prescribes a failed remedy blind.
|
|
237
|
+
- The symptom/remedy registry (section 4) -- injected as data, not baked into the prompt text.
|
|
238
|
+
|
|
239
|
+
### 2.3 Output contract (ajv schema; enum-closed; engine logs `evidence` verbatim)
|
|
240
|
+
|
|
241
|
+
```jsonc
|
|
242
|
+
{
|
|
243
|
+
"classification": "ENVIRONMENT | ENGINE_FLAW | TASK_SHAPE | UNCLEAR",
|
|
244
|
+
"confidence": "high | medium | low",
|
|
245
|
+
"evidence": ["..."], // short factual bullets; logged verbatim to sprint log
|
|
246
|
+
"matchedRegistryEntry": "id | null", // which symptom entry (section 4) matched, if any
|
|
247
|
+
|
|
248
|
+
"action": {
|
|
249
|
+
"kind": "repair_environment_then_retry | retry_same | retry_different_member |
|
|
250
|
+
swap_model_tier | defer_bead | reduce_scope_and_continue |
|
|
251
|
+
abort_sprint | pause_for_human",
|
|
252
|
+
// kind-specific params, each independently schema-bounded:
|
|
253
|
+
"repairs": ["reprovision_llm_auth", ...], // closed enum, section 4 registry ids
|
|
254
|
+
"member": "name", // must be in the sprint's member list
|
|
255
|
+
"tier": "cheap|standard|premium",
|
|
256
|
+
"timeoutMultiplier": 1.5, // capped at 2 in schema
|
|
257
|
+
"beadIds": ["..."], "reason": "...",
|
|
258
|
+
"salvageWip": false // opt-in rescue-branch commit of uncommitted
|
|
259
|
+
}, // member WIP before defer/abort (section 2.5)
|
|
260
|
+
|
|
261
|
+
"probes": ["member_session_state <m>", ...], // optional; triggers the single probe round
|
|
262
|
+
// INSTEAD of an action (round 2 must act)
|
|
263
|
+
|
|
264
|
+
"engineFlawReport": { // required when classification=ENGINE_FLAW
|
|
265
|
+
"symptom": "...", "suspectedComponent": "...",
|
|
266
|
+
"reproEvidence": ["..."], "proposedBeadTitle": "..."
|
|
267
|
+
},
|
|
268
|
+
|
|
269
|
+
"proposedRegistryEntry": { ... }, // optional, section 4.3 -- logged for human
|
|
270
|
+
// review, NEVER auto-applied
|
|
271
|
+
|
|
272
|
+
"humanActionRequired": { // optional; REQUIRED when action is
|
|
273
|
+
"summary": "...", // pause_for_human or abort_sprint with
|
|
274
|
+
"suggestedCommands": ["..."], // classification ENGINE_FLAW/UNCLEAR
|
|
275
|
+
"relevantFiles": ["..."], "relevantBeadIds": ["..."],
|
|
276
|
+
"whyBeyondBounds": "..."
|
|
277
|
+
},
|
|
278
|
+
|
|
279
|
+
"notes": "..."
|
|
280
|
+
}
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
Contract rules the engine enforces (not the prompt):
|
|
284
|
+
- Exactly one of `action` / `probes` per response; a probe-round response must contain `action`.
|
|
285
|
+
- Every mutating consequence is executed by the runner from these fields. The doctor never runs
|
|
286
|
+
`bd`, never touches git, never calls MCP tools -- it has none.
|
|
287
|
+
- A schema-invalid response after the repair loop => the consult FAILS and the engine falls back
|
|
288
|
+
to pre-doctor behavior (section 6, strictly-additive rule).
|
|
289
|
+
|
|
290
|
+
### 2.4 The humanActionRequired referral block
|
|
291
|
+
|
|
292
|
+
Whenever the correct fix is outside sprint-doctor's own permission boundary -- an engine code fix,
|
|
293
|
+
a credential type it cannot self-provision (`AUTH_DENIED` in the VCS taxonomy is explicitly
|
|
294
|
+
"an operator must grant access", errors.mjs:29-32), OS/SSH-level intervention on a member, or a
|
|
295
|
+
genuine judgment call -- the verdict must carry a *referral*, written the way a doctor writes one:
|
|
296
|
+
specific, actionable, and honest about why it is being handed off. Vague "escalate to human" is a
|
|
297
|
+
schema-legal-but-rejected shape: the engine treats a `pause_for_human`/ENGINE_FLAW verdict with an
|
|
298
|
+
empty or command-free `humanActionRequired` the same way `isReviewerContractViolation()` treats an
|
|
299
|
+
actionless CHANGES_NEEDED (runner.js:3552-3557) -- one re-ask, then the consult fails.
|
|
300
|
+
|
|
301
|
+
**Example A -- environment case beyond bounds (member needs OS-level attention):**
|
|
302
|
+
|
|
303
|
+
```jsonc
|
|
304
|
+
{
|
|
305
|
+
"classification": "ENVIRONMENT", "confidence": "high",
|
|
306
|
+
"evidence": [
|
|
307
|
+
"5 infra failures on member 'mac-01' across 3 unrelated beads (T2), signatures all 'stalled'",
|
|
308
|
+
"probe member_cli_version timed out after 120s; member_disk_free never returned",
|
|
309
|
+
"same beads dispatched to 'linux-02' completed normally this cycle"
|
|
310
|
+
],
|
|
311
|
+
"matchedRegistryEntry": "hung-remote-session",
|
|
312
|
+
"action": { "kind": "pause_for_human", "reason": "member 'mac-01' unresponsive below the CLI layer; remaining work continues on other members" },
|
|
313
|
+
"humanActionRequired": {
|
|
314
|
+
"summary": "Member 'mac-01' does not respond to even trivial execute_command probes; the fleet server's kill/stop verbs cannot reach whatever is wedged (likely the SSH channel or the host itself).",
|
|
315
|
+
"suggestedCommands": [
|
|
316
|
+
"ssh <mac-01 host> 'pkill -f <provider CLI binary>; uptime; df -h'",
|
|
317
|
+
"apra-fleet exec --member mac-01 -- <provider CLI> --version # after the ssh-side kill, to confirm the channel recovered"
|
|
318
|
+
],
|
|
319
|
+
"relevantFiles": [], "relevantBeadIds": [],
|
|
320
|
+
"whyBeyondBounds": "Doctor repairs are limited to fleet MCP verbs (stop_prompt, member_reservation force_release, provision_*); all were attempted or unreachable. Host-level access requires operator SSH."
|
|
321
|
+
},
|
|
322
|
+
"notes": "Sprint can continue at reduced parallelism; nothing suggests a bead or engine problem."
|
|
323
|
+
}
|
|
324
|
+
```
|
|
325
|
+
|
|
326
|
+
**Example B -- real engine bug (code fix needed; doctor must not touch source):**
|
|
327
|
+
|
|
328
|
+
```jsonc
|
|
329
|
+
{
|
|
330
|
+
"classification": "ENGINE_FLAW", "confidence": "medium",
|
|
331
|
+
"evidence": [
|
|
332
|
+
"3 'stalled' kills (T1) on one bead, all on premium-tier dispatches; stall threshold is a flat 120s regardless of tier",
|
|
333
|
+
"member-side output shows steady token streaming right up to each kill -- long single-turn reasoning, not silence",
|
|
334
|
+
"cross-member retry reproduced the identical kill on a second member, ruling out a member-environment cause"
|
|
335
|
+
],
|
|
336
|
+
"matchedRegistryEntry": null,
|
|
337
|
+
"action": { "kind": "retry_same", "timeoutMultiplier": 2, "reason": "mitigation only; root cause is engine-side" },
|
|
338
|
+
"engineFlawReport": {
|
|
339
|
+
"symptom": "Confirmed-stall detector kills healthy premium-tier dispatches whose single-turn latency exceeds the flat stall threshold",
|
|
340
|
+
"suspectedComponent": "stall threshold configuration (server-side stall detector) vs model-tier-aware dispatch budgets",
|
|
341
|
+
"reproEvidence": ["ledger rows #14,#17,#21: identical kill at ~120s of streaming output"],
|
|
342
|
+
"proposedBeadTitle": "sprint-doctor finding: stall threshold not tier-aware; premium dispatches killed mid-stream"
|
|
343
|
+
},
|
|
344
|
+
"humanActionRequired": {
|
|
345
|
+
"summary": "If the doubled timeout retry also dies, this needs an engine change: make the stall threshold tier- or dispatch-aware. Doctor cannot and must not patch engine source.",
|
|
346
|
+
"suggestedCommands": [
|
|
347
|
+
"grep -n STALL_THRESHOLD <fleet server source> # confirm the flat default",
|
|
348
|
+
"file the proposedBeadTitle above in the ENGINE's own tracker (not this project's)"
|
|
349
|
+
],
|
|
350
|
+
"relevantFiles": ["<engine stall-detector module>"],
|
|
351
|
+
"relevantBeadIds": ["<the stalled bead id>"],
|
|
352
|
+
"whyBeyondBounds": "Source modification of fleet-sprint/fleet server is outside the doctor's action set by design; only defer/mitigate/report are permitted."
|
|
353
|
+
},
|
|
354
|
+
"notes": "Bead itself looks healthy; do not defer it yet -- mitigation may complete it."
|
|
355
|
+
}
|
|
356
|
+
```
|
|
357
|
+
|
|
358
|
+
(Note example B's file references are deliberately generic in the *skill/prompt* -- the doctor
|
|
359
|
+
names components from evidence it was shown, not from hardcoded engine paths; see section 7.)
|
|
360
|
+
|
|
361
|
+
### 2.5 Action executor: what the runner does with each verdict
|
|
362
|
+
|
|
363
|
+
| action.kind | Executor mapping (existing verbs only) | Bound |
|
|
364
|
+
|---|---|---|
|
|
365
|
+
| repair_environment_then_retry | Registry remedies (section 4): `provision_llm_auth` / `provisionVcsAuthForMember()` (runner.js:1616-1659) / `stop_prompt` + `memberSessionGuard.killIfAlive` (used at runner.js:7260) / `member_reservation force_release` (src/tools/member-reservation.ts:17-27) / `git fetch` via `command()`. **Dolt/beads-sync wedging is explicitly excluded from this remedy set** -- `settleDoltConflicts()` (`fleet-sprint/dolt-settle.mjs`) is the terminal, non-LLM answer for that failure class; see section 4 | each remedy latched once per member per sprint (the `authHealAttempted` pattern, runner.js:703); exactly one post-repair retry of the failed dispatch |
|
|
366
|
+
| retry_same | one redispatch with `timeout_s` scaled by `timeoutMultiplier` (<= 2 -- matches the integ-test 2x ceiling precedent in the stabilization log, and open bead apra-fleet-aw8's escalate-timeout-on-retry recommendation) | once per bead per sprint |
|
|
367
|
+
| retry_different_member | re-lane the bead via existing streak re-lane, with a doctor-set member exclusion respected by streak assignment (member selection machinery runner.js:4890-4926) | once per bead; doubles as the environment-vs-task-shape probe |
|
|
368
|
+
| swap_model_tier | runner-side per-bead tier override map consulted where the bead's `metadata.model` tier is read (runner.js:2664-2665); no `bd update` of the bead's stored metadata (keeps the DB as the planner wrote it) | one swap per bead |
|
|
369
|
+
| defer_bead / reduce_scope_and_continue | `bd defer <id> --reason "<reason>"` via orchestrator `command()`. **Implementation trap, called out now:** `NOT_DONE_STATUSES` includes `deferred` (runner.js:284), so a deferred bead still counts as open-at-goal in the completion check (runner.js:8134-8138) and contributes nothing to `progressScore` -- a naive defer would wedge the sprint at "can never complete" and read as continued stagnation. The executor must (a) add doctor-deferred ids to an exclusion set filtered out of `openAtGoal` exactly as `decomposedParentIds` already is (runner.js:8138), and (b) credit them to the progress score via a monotone set, the `verifyEverIds` pattern (runner.js:6002, 8180) | max `doctor_max_defers` (default 3) beads per sprint; each defer's reason is the doctor's `evidence` + `reason`, appended to the bead's notes by `bd defer --reason` itself |
|
|
370
|
+
| abort_sprint | throw new `SprintDoctorAbortError` (extends WorkflowError, added to `isTypedAbortError()` runner.js:4200-4210) so the existing terminal machinery produces the record + `[ABORTED]` PR, with the doctor's full verdict in `details` and in the PR body | terminal by definition |
|
|
371
|
+
| pause_for_human | A THIN caller of the generic engine pause primitive (section 2.6): the runner calls `workflow.requestPause(reason)` at the consult point and the engine's own gate does the rest -- the consult point sits at Cycle Evaluation with all sync brackets closed, so the pause takes effect at the very next activity entry. The paused card carries the doctor's `humanActionRequired` block via `publishState('sprint-doctor', ...)` (dashboard sees it through the state event, viewer/index.mjs:1119). Member reservations are released while paused and re-acquired on resume (section 2.6.4). A human resumes via the viewer/supervisor Resume control | **No timeout, no auto-abort.** Paused is a safe, durable, indefinitely-holdable state (rationale in 2.6.4); "never wedged" means never *ambiguous* -- a paused sprint is loudly, visibly paused with a written referral, not silently dead |
|
|
372
|
+
|
|
373
|
+
Salvage pre-step (from coach's action allowlist): before executing `defer_bead`,
|
|
374
|
+
`reduce_scope_and_continue`, or `abort_sprint`, the executor MAY -- when the verdict sets
|
|
375
|
+
`salvageWip: true` and the failed member's tree has uncommitted changes -- commit that WIP to a
|
|
376
|
+
clearly-named rescue branch (`rescue/<sprint-branch>/<beadId>-<timestamp>`) and push it, never
|
|
377
|
+
touching the sprint branch itself. This upgrades the "failed-attempt debris is discarded by
|
|
378
|
+
design" status quo (section 2.6.3) to "discarded but recoverable" for exactly the cases where a
|
|
379
|
+
human may want to inspect what the stuck attempt actually did. Coach's hard prohibitions apply
|
|
380
|
+
verbatim to this executor path: never force-push, never delete branches/files, never mutate
|
|
381
|
+
bead verdicts, never edit code content.
|
|
382
|
+
|
|
383
|
+
### 2.6 Pause/resume as a workflow-engine primitive (generic; fleet-sprint is only a client)
|
|
384
|
+
|
|
385
|
+
Ownership decision: pause/resume is implemented ONCE, in `packages/apra-fleet-workflow`, usable
|
|
386
|
+
by any workflow script -- NOT as fleet-sprint checkpoint calls sprinkled through runner.js.
|
|
387
|
+
Because fleet-sprint's cycles and rounds are themselves just script loops over the engine's
|
|
388
|
+
`agent()`/`command()` activities under `phase()`/`group()` labels (runner.js registers nothing;
|
|
389
|
+
it merely calls the primitives -- see section 0), a single engine-level gate gives cycles,
|
|
390
|
+
rounds, and every future workflow script pause support with zero script cooperation.
|
|
391
|
+
sprint-doctor's `pause_for_human` is a thin caller of this primitive, not its owner.
|
|
392
|
+
|
|
393
|
+
Scope note: this is deliberately NOT open bead `apra-fleet-9ub`'s full journaled
|
|
394
|
+
`interrupt()` (LangGraph-style suspend-with-resume-value, `'paused'` as a resumable terminal
|
|
395
|
+
status, journal replay via `computeActivityKey`/`completedByKey`, journal.mjs:106-221, off by
|
|
396
|
+
default per journal.mjs:14-19). 9ub stays open for same-run resume across process restarts; the
|
|
397
|
+
gate designed here is the in-process mechanism it would later journal.
|
|
398
|
+
|
|
399
|
+
#### 2.6.1 The gate: engine-owned, at activity entry
|
|
400
|
+
|
|
401
|
+
New primitives on `FleetWorkflow`, siblings of `requestStop()` (index.mjs:499-503):
|
|
402
|
+
|
|
403
|
+
- `requestPause(reason)` -- sets a pause-pending flag + reason, emits `'pause:requested'`.
|
|
404
|
+
Touches NO AbortController: nothing in flight is ever rejected by a pause.
|
|
405
|
+
- An internal `await this._pauseGate()` at the top of `agent()` (with the budget check,
|
|
406
|
+
index.mjs:760-765) and at `command()`'s equivalent entry. Both methods are async and always
|
|
407
|
+
awaited by callers -- that is precisely what makes them valid park points. When no pause is
|
|
408
|
+
pending the gate is one flag read; when one is pending (and the guard permits, 2.6.3) it emits
|
|
409
|
+
`'paused'`, flips run state, and parks until `requestResume(value?)` resolves it (emitting
|
|
410
|
+
`'resumed'`). Events flow to dashboards exactly like the existing `'phase'`/`'state'`/`'log'`
|
|
411
|
+
events (viewer/index.mjs:1052-1124).
|
|
412
|
+
|
|
413
|
+
Answering the "start of every phase AND every activity" framing precisely: `phase()`/`group()`
|
|
414
|
+
are *synchronous label-setters* (index.mjs:510-550) invoked unawaited at every runner.js call
|
|
415
|
+
site (e.g. 6133, 6804, 7733, 8973) -- making them awaitable gates would be an API break for
|
|
416
|
+
every existing script. But behaviorally the start of a phase IS its first activity: nothing a
|
|
417
|
+
phase does before its first `agent()`/`command()` call has side effects beyond labels/logs. So
|
|
418
|
+
one gate at activity entry, whose `'paused'` event reports the current phase/group from the
|
|
419
|
+
async-local run store (`_currentPhase()`/`_currentGroup()`, index.mjs:432-442), delivers full
|
|
420
|
+
"phase-start + activity-start" coverage from a single engine-owned function. `parallel()`
|
|
421
|
+
branches each park at their own next gated call (branch-forked stores, index.mjs:1435-1471);
|
|
422
|
+
the engine declares the run paused when a pause is pending and its in-flight activity count
|
|
423
|
+
(the `activity:start`/`activity:end` accounting it already emits) reaches zero.
|
|
424
|
+
`requestStop()` on a paused/pausing run still works: the parked gate promise also rejects on
|
|
425
|
+
the run's abort controller, so `CancelledError` unwinds normally.
|
|
426
|
+
|
|
427
|
+
#### 2.6.2 Why an await gate and not a thrown pause signal
|
|
428
|
+
|
|
429
|
+
The rejected alternative: throw a `PauseRequested` signal at request time (or inject it into the
|
|
430
|
+
next `agent()`/`command()` call), caught only by the engine. Any script-VISIBLE exception is
|
|
431
|
+
disqualified by runner.js's own error handling: its broad catch blocks swallow or misroute
|
|
432
|
+
anything not on their curated lists (doer generic retry arm runner.js:7314-7330, planner ladder
|
|
433
|
+
6244-6280, `runGitStep()` 701-729; historically also the now-deleted dolt ladder composition), and
|
|
434
|
+
the one exception that must pierce them today -- `CancelledError` -- survives only via explicit
|
|
435
|
+
guards at every layer (index.mjs:1100-1108, runner.js:4187-4189, 4201, 4227). A second
|
|
436
|
+
must-pierce exception type would mean auditing every broad catch in a 9400-line file forever.
|
|
437
|
+
The engine-internal await gate has no such problem: it throws nothing, so script catch blocks
|
|
438
|
+
cannot see an exception that never exists -- the script observes only a slow-returning
|
|
439
|
+
`agent()`/`command()` call.
|
|
440
|
+
|
|
441
|
+
One real hazard remains for the gate, arriving through timers instead of catches:
|
|
442
|
+
script-side deadlines racing composites that contain gated calls. Concretely,
|
|
443
|
+
`withDispatchWatchdog()` races an already-started dispatch promise against
|
|
444
|
+
`timeoutS + grace` with a deliberately non-unref'd timer (runner.js:4511-4548), and the planner
|
|
445
|
+
dispatch sites nest it around `agent()` inside `withGitSync` brackets (runner.js:6178, 6182,
|
|
446
|
+
6676). A gate that parked inside such a raced promise would burn paused wall-clock against the
|
|
447
|
+
timer, fire a spurious `watchdog_timeout`, and the generic retry arms would misroute the pause
|
|
448
|
+
as a dispatch failure -- the old failure mode through a different door. The clean-barrier guard
|
|
449
|
+
in 2.6.3 closes this structurally: the gate never parks while a sync bracket is open, and every
|
|
450
|
+
watchdog-raced composite in runner.js lives within a bracket. One mechanism resolves both the
|
|
451
|
+
timer race and the durable-state problem below.
|
|
452
|
+
|
|
453
|
+
#### 2.6.3 Clean-state invariant: member-agnostic resume by design
|
|
454
|
+
|
|
455
|
+
The subtle gap: "between activities" is not the same as "clean". Inside a
|
|
456
|
+
`withGitSync` bracket (runner.js:4999; ordered teardown in `syncMemberAfterOrdered`,
|
|
457
|
+
runner.js:1066), the member holds committed-but-unpushed git commits and unpushed dolt
|
|
458
|
+
mutations until the post-dispatch G-push/D-push teardown runs. An in-process, same-member
|
|
459
|
+
resume is safe anywhere (the parked call resumes and the bracket completes) -- but a
|
|
460
|
+
different-member or relaunch-grade resume from a mid-bracket pause would lose that unpublished
|
|
461
|
+
work outright. That is data loss, not a pull-the-branch inconvenience.
|
|
462
|
+
|
|
463
|
+
**Invariant adopted (brief's candidates (a)+(b) together):** a pause only *takes effect* at an
|
|
464
|
+
activity entry where no sync bracket is open -- i.e. every piece of member state the engine
|
|
465
|
+
considers real is committed AND pushed (git to origin, dolt to the shared remote). A pause
|
|
466
|
+
requested mid-bracket stays pending, with the UI showing "pausing...", until the first clean
|
|
467
|
+
activity entry -- the identical deferral UX already specified for the human-initiated case, now
|
|
468
|
+
serving both triggers.
|
|
469
|
+
|
|
470
|
+
Mechanism: the engine exposes `setPauseGuard(fn)` -- optional, one registration per run. With a
|
|
471
|
+
pause pending, each gated entry consults the guard and defers parking while it returns false.
|
|
472
|
+
fleet-sprint's ENTIRE pause awareness is registering a guard backed by an open-bracket counter
|
|
473
|
+
incremented/decremented at `withGitSync` entry/exit and around the orchestrator's own
|
|
474
|
+
`DoltSync.syncBefore/After` brackets (runner.js:8124). No checkpoint calls, no other runner.js
|
|
475
|
+
changes. Scripts that register no guard get pause-at-any-activity-entry -- correct for
|
|
476
|
+
single-member or stateless workflows.
|
|
477
|
+
|
|
478
|
+
Sweep for states the invariant cannot cover (brief's candidate (c)):
|
|
479
|
+
- *Uncommitted WIP during an agent turn:* unreachable by pause -- the gate never interrupts an
|
|
480
|
+
in-flight activity; a member's tree only holds mid-turn WIP while its dispatch is running.
|
|
481
|
+
- *Uncommitted debris from failed attempts:* can exist between activities, but it is already
|
|
482
|
+
discarded-by-design state -- retries run fresh `resume:false` sessions that redo the work
|
|
483
|
+
(confirmed behavior recorded in bead apra-fleet-aw8), and pre-dispatch brackets re-sync the
|
|
484
|
+
workspace. The invariant deliberately protects exactly the state the engine treats as real:
|
|
485
|
+
committed+pushed. Nothing new is lost that today's retry path does not already discard.
|
|
486
|
+
- *Member-local runtime state after Deploy:* the one genuine partial-coverage case. A pause
|
|
487
|
+
between Deploy and Integ Test (the old CP2 boundary, runner.js:7815) is git/dolt-clean, but
|
|
488
|
+
the deployed instance lives on the deploy member. Verdict: NOT a non-pausable region --
|
|
489
|
+
same-member in-process resume is fully safe there; a relaunch-grade or different-member
|
|
490
|
+
resume simply re-runs Deploy (deploy.md-driven, re-runnable by its own phase contract, and
|
|
491
|
+
the cycle re-derives from pushed state). Documented, not forbidden.
|
|
492
|
+
|
|
493
|
+
Comparison with the obvious simpler alternative -- a fixed list of hand-placed checkpoint calls
|
|
494
|
+
in runner.js (cycle top runner.js:6074, post-Deploy 7815, post-evaluation 8238,
|
|
495
|
+
pre-Finalization 8397, pre-Publish-PR 8973, the same boundary list bead 9ub sketched): every
|
|
496
|
+
one of those points is a zero-open-bracket boundary, so all of them remain reachable pause
|
|
497
|
+
points under the guard -- they are simply members of the guard's larger effective set, which
|
|
498
|
+
also includes between-round and between-streak activity entries. The guard therefore strictly
|
|
499
|
+
dominates the hand-placed list: better worst-case pause latency ("next clean activity entry"
|
|
500
|
+
instead of "end of cycle"), no script-author burden, and no risk of a forgotten checkpoint in a
|
|
501
|
+
future phase. The fixed list survives only as a *description* of where pauses typically land.
|
|
502
|
+
|
|
503
|
+
**Resume is member-agnostic by design, and re-sync is unconditional.** On resume -- same member
|
|
504
|
+
or different, in-process or relaunch -- the resume path re-syncs every re-acquired member as a
|
|
505
|
+
first-class step, not a defensive one: `git fetch` + the ensure-branch probe
|
|
506
|
+
(`decideEnsureBranchAction`, runner.js:4583-4634) + `bd dolt pull`. This is *sufficient*, not
|
|
507
|
+
merely prudent, precisely because the invariant guarantees there was never local-only state to
|
|
508
|
+
lose at a pause-effective point. v1's in-process resume still prefers and requires the SAME
|
|
509
|
+
member set (member identities are baked into in-flight closures and the roleMap
|
|
510
|
+
specialist/generalist math, runner.js:4890-4926) and fails cleanly back to "still paused",
|
|
511
|
+
naming the unavailable member(s), when one cannot be re-reserved. The invariant's payoff is
|
|
512
|
+
that the coarse escape hatch -- relaunching the sprint with ANY member set -- is now provably
|
|
513
|
+
lossless rather than best-effort, and 9ub's future same-run resume inherits the same guarantee.
|
|
514
|
+
|
|
515
|
+
#### 2.6.4 Paused is a safe, indefinite state -- what bounds it instead of a timeout
|
|
516
|
+
|
|
517
|
+
Deliberate decision: there is NO pause timeout and NO auto-abort. A pause that degrades to a
|
|
518
|
+
destructive action on human response time is a ticking bomb -- the human may be asleep, which is
|
|
519
|
+
the product's own value proposition. What actually needs bounding, and what doesn't:
|
|
520
|
+
|
|
521
|
+
- **Member reservations (the real starvation concern).** The supervisor reserves each member for
|
|
522
|
+
the sprint at launch, and Stop = `POST /api/reservations/:sprintId/force-release` (kills child +
|
|
523
|
+
releases, dashboard.mjs:279-282). A paused sprint holding reservations indefinitely would starve
|
|
524
|
+
every other sprint wanting those members. So: on entering `'paused'`, the runner releases each
|
|
525
|
+
member via `member_reservation action:'release'` (owner-checked, member-reservation.ts:54-68);
|
|
526
|
+
on resume it re-reserves (action `'reserve'`). Reserve is idempotent for the same sprint id
|
|
527
|
+
(member-reservation.ts:49-51) and fails CLEANLY when another sprint took the member in the
|
|
528
|
+
interim ("already reserved by <owner>", member-reservation.ts:42-44) -- in that case **resume
|
|
529
|
+
fails with a clear message naming the unavailable member(s) and the sprint STAYS paused**; it
|
|
530
|
+
never silently proceeds on a substitute member (member identity carries work folders, sessions,
|
|
531
|
+
and credentials). The operator retries resume later, or relaunches the sprint -- which the
|
|
532
|
+
clean-state invariant (2.6.3) now makes provably lossless. Either way, every re-acquired
|
|
533
|
+
member goes through the unconditional resume re-sync of 2.6.3 before any dispatch, since
|
|
534
|
+
another sprint may have legitimately used the member meanwhile.
|
|
535
|
+
- **Cost: nothing to bound.** A paused run dispatches nothing and spends nothing --
|
|
536
|
+
`budget.spent()` is flat while parked; the only footprint is the idle runner process and its
|
|
537
|
+
viewer port. This is unlike the doctor's own consult loop, which does real premium dispatches
|
|
538
|
+
and therefore does carry hard caps (section 6).
|
|
539
|
+
- **Visibility, not force.** The bound on an indefinite pause is that it is *loud*: the viewer
|
|
540
|
+
and supervisor both show "paused since <time>" with the reason/referral card (2.6.5), and the
|
|
541
|
+
supervisor watchdog must be taught that a live-pid paused run is `PAUSED`, not stall-suspect --
|
|
542
|
+
it already only classifies and never remediates (watchdog.mjs:36-38), so this is a state label,
|
|
543
|
+
not a behavior change. One genuinely useful *informational* indicator: the sprint branch ages
|
|
544
|
+
against base while paused, so the paused card should show "base has moved N commits since
|
|
545
|
+
pause" -- information for the human, never a trigger for automatic action.
|
|
546
|
+
- **Process death while paused.** Keeping the process alive while paused is cheap (a parked
|
|
547
|
+
promise), and v1 relies on it: same-run in-place resume requires the live process. If the
|
|
548
|
+
process dies anyway, this is NOT catastrophic and NOT new: 9ub's own investigation confirmed
|
|
549
|
+
the system is already resumable at full-process-restart granularity -- beads and git are the
|
|
550
|
+
durable source of truth, and relaunching the identical invocation skips closed work (at the
|
|
551
|
+
cost of re-running Plan). True same-run resume across a process restart is exactly the
|
|
552
|
+
journaled part of 9ub (journal.mjs replay is already scheduler-independent across `parallel()`
|
|
553
|
+
interleavings, journal.mjs:60-100, but wiring `'paused'` as a resumable terminal status is the
|
|
554
|
+
deferred design work) -- explicitly out of scope here, explicitly NOT foreclosed.
|
|
555
|
+
|
|
556
|
+
Net: an indefinitely-paused sprint with released reservations, zero spend, and a loud
|
|
557
|
+
"paused since <t>" card is a genuinely safe resting state. No forced resolution, full stop.
|
|
558
|
+
|
|
559
|
+
#### 2.6.5 Viewer / supervisor surfacing
|
|
560
|
+
|
|
561
|
+
- **Per-run viewer** (the natural sibling of Stop): a Pause button beside the existing Save/Stop
|
|
562
|
+
buttons in `HTML_TEMPLATE` (viewer/index.mjs:186-187), backed by `POST /pause` and
|
|
563
|
+
`POST /resume` routes beside `POST /stop` (viewer/index.mjs:1253-1288; the stop route calls
|
|
564
|
+
`workflow.requestStop(...)` at 1278 -- pause calls `workflow.requestPause(...)`, resume calls
|
|
565
|
+
`workflow.requestResume()`). Button state machine driven by the engine events the viewer
|
|
566
|
+
already consumes: Pause -> (on `'pause:requested'`) disabled "Pausing... (at next clean point)"
|
|
567
|
+
-> (on `'paused'`) "Paused since <t>" badge + Resume button + the reason/referral card rendered
|
|
568
|
+
from the `sprint-doctor` state namespace when the pause was doctor-forced.
|
|
569
|
+
- **Supervisor Sprint Stack**: Pause/Resume buttons as siblings of the existing per-row
|
|
570
|
+
Stop/Restart buttons (dashboard.mjs:128-137), implemented in the same
|
|
571
|
+
source-string-script pattern as `SPRINT_STOP_SCRIPT`/`SPRINT_RESTART_SCRIPT`
|
|
572
|
+
(dashboard.mjs:248-437). Crucially Pause does NOT reuse Stop's force-release route (which
|
|
573
|
+
kills the child, dashboard.mjs:279-282): it proxies to the child run's own viewer `/pause`
|
|
574
|
+
(the supervisor already tracks each sprint's viewer port for the live-view link) or an
|
|
575
|
+
equivalent `POST /api/sprints/:id/pause` forwarding route. The row's status cell gains
|
|
576
|
+
`pausing`/`paused since <t>` states; watchdog labels the run `PAUSED` per 2.6.4. These are
|
|
577
|
+
viewer/supervisor HTTP routes, not MCP tool surface -- no `apra-fleet-client` sync is
|
|
578
|
+
triggered (per the repo rule that client sync tracks `src/tools/*`).
|
|
579
|
+
|
|
580
|
+
---
|
|
581
|
+
|
|
582
|
+
## 3. Environment-vs-engine triage (first-class)
|
|
583
|
+
|
|
584
|
+
Every consult classifies the incident BEFORE choosing an action, and the classification is an
|
|
585
|
+
explicit output field. Buckets:
|
|
586
|
+
|
|
587
|
+
1. **ENVIRONMENT** -- engine logic fine; execution environment broken: stale/expired VCS or LLM
|
|
588
|
+
credentials, wedged reservation, dead/hung remote session, stale dolt/beads clone, disk/network
|
|
589
|
+
flakiness, missing not-yet-synced branch, member CLI version drift.
|
|
590
|
+
2. **ENGINE_FLAW** -- a real fleet-sprint bug/gap: bad retry logic, a mutex not wired on some
|
|
591
|
+
path, stall threshold too aggressive for a tier, a phase invariant violated.
|
|
592
|
+
3. **TASK_SHAPE** -- the bead itself is too large/ambiguous/mis-specified for any doer to land
|
|
593
|
+
(the hypothesis the incident-2 human acted on).
|
|
594
|
+
4. **UNCLEAR** -- evidence insufficient; handled by the cheap-first policy below.
|
|
595
|
+
|
|
596
|
+
### 3.1 Discriminating signals
|
|
597
|
+
|
|
598
|
+
| Signal | Points to | Rationale |
|
|
599
|
+
|---|---|---|
|
|
600
|
+
| Identical `errorSignature` across unrelated beads AND multiple members | ENVIRONMENT (shared substrate: credentials, dolt remote, network) -- unless the signature names an engine invariant (e.g. a sync-bracket or mutex error string), then ENGINE_FLAW | Bead content varies, member varies, failure doesn't => cause is in what they share |
|
|
601
|
+
| Failures confined to ONE member, across >= 2 distinct beads (T2) | ENVIRONMENT (that member): hung CLI session, disk, version drift | Work varies, host doesn't |
|
|
602
|
+
| Failures confined to ONE bead, reproduced on a SECOND member (`retry_different_member` probe) | TASK_SHAPE, or ENGINE_FLAW if the failure mode is mechanical (e.g. killed mid-stream at a fixed time offset) | Host varies, work doesn't |
|
|
603
|
+
| `details.reason` in {'auth'} or VCS kind in {AUTH_EXPIRED} | ENVIRONMENT with a known registry remedy | Already classified by errors.mjs:363-394 / the VCS taxonomy (errors.mjs:25-76) |
|
|
604
|
+
| 'stalled'/'watchdog_timeout' ONLY on premium-tier dispatches, member-independent, with output streaming up to the kill | ENGINE_FLAW (threshold vs tier mismatch) | The stall detector measures inactivity; a slow-but-streaming or long-single-turn premium call is a known blind spot (stabilization log Issue 32: `claude -p` prints nothing until turn completion) |
|
|
605
|
+
| Fixed-time-offset deaths (~300s, ~120s, ~3600s) matching known knobs | ENGINE_FLAW or config, not TASK_SHAPE | Timeouts kill at their configured value; content-caused failures land at varied offsets |
|
|
606
|
+
| `bd show` of the bead: no acceptance criteria, XL scope markers, prior reopen-thrash (`reopenCounts` > REOPEN_THRASH_LIMIT, runner.js:5986-5995) | TASK_SHAPE | The bead was already oscillating before it started stalling |
|
|
607
|
+
|
|
608
|
+
### 3.2 Policy for UNCLEAR: cheap, reversible, environment-first
|
|
609
|
+
|
|
610
|
+
Environment repairs are cheap, latched, and reversible-or-idempotent; defer/abort destroys sprint
|
|
611
|
+
momentum. So the doctrine, encoded in the skill and the registry ordering:
|
|
612
|
+
|
|
613
|
+
1. Attempt matching registry remedies (section 4) -- at most one pass, each remedy latched.
|
|
614
|
+
2. If no registry match or remedies verified-failed: ONE `retry_different_member` (the
|
|
615
|
+
discriminating probe -- it converts UNCLEAR into ENVIRONMENT-on-that-member vs
|
|
616
|
+
TASK_SHAPE/ENGINE_FLAW with a single data point).
|
|
617
|
+
3. Still failing: treat as bucket 2/3 -- `defer_bead` with a written reason (exactly the human's
|
|
618
|
+
incident-2 remedy, now automatic and evidenced) and continue reduced scope; or `abort_sprint`
|
|
619
|
+
when the bead IS the sprint's whole scope; plus `engineFlawReport` when mechanical evidence
|
|
620
|
+
points engine-ward; plus `humanActionRequired` whenever the residual fix is beyond bounds.
|
|
621
|
+
|
|
622
|
+
### 3.3 Incident replays under this design
|
|
623
|
+
|
|
624
|
+
- *Incident 2:* T1 fires at attempt 3. Probe round: `member_session_state` + `member_cli_version`
|
|
625
|
+
on the stalling member. If the member is wedged -> registry `hung-remote-session` remedy
|
|
626
|
+
(stop_prompt/kill + one redispatch); if healthy -> `retry_different_member`; if reproduced ->
|
|
627
|
+
`defer_bead` with the evidence trail as the reason + TASK_SHAPE (or ENGINE_FLAW if the kills are
|
|
628
|
+
mechanical). Worst case ~5 dispatches and one premium consult instead of 12 stalls + manual
|
|
629
|
+
archaeology.
|
|
630
|
+
- *Incident 1:* T3 interposes one consult before `StalledSprintError`. If the verify-routed-never-
|
|
631
|
+
closed pattern is present (the detector already computes it, runner.js:8209-8217), the doctor
|
|
632
|
+
can distinguish "verifier broken" (ENVIRONMENT/ENGINE_FLAW: check the integ playbook probe
|
|
633
|
+
output, deploy failures list) from "genuinely out of work" and choose defer/reduce-scope/abort
|
|
634
|
+
with a report instead of a bare kill.
|
|
635
|
+
|
|
636
|
+
---
|
|
637
|
+
|
|
638
|
+
## 4. Symptom/remedy registry (extensible by data, not redesign)
|
|
639
|
+
|
|
640
|
+
### 4.1 Shape
|
|
641
|
+
|
|
642
|
+
A data file, versioned with fleet-sprint, separate from all logic:
|
|
643
|
+
`packages/apra-fleet-se/fleet-sprint/doctor-registry.mjs` (a frozen exported array -- .mjs rather
|
|
644
|
+
than JSON so entries can carry regexes and doc comments, exactly like the taxonomy tables already
|
|
645
|
+
in errors.mjs; still pure data, no behavior). Each entry:
|
|
646
|
+
|
|
647
|
+
```js
|
|
648
|
+
{
|
|
649
|
+
id: 'stale-vcs-credential',
|
|
650
|
+
classification: 'ENVIRONMENT',
|
|
651
|
+
detect: { // ALL matchers are mechanical
|
|
652
|
+
reasons: ['auth'], // err.details.reason values
|
|
653
|
+
signatureRe: /authentication failed|not logged in/i,
|
|
654
|
+
scope: 'member', // member | bead | fleet
|
|
655
|
+
},
|
|
656
|
+
remedy: {
|
|
657
|
+
verb: 'reprovision_vcs_auth', // executor enum, section 2.5
|
|
658
|
+
latch: 'once-per-member-per-sprint',
|
|
659
|
+
},
|
|
660
|
+
verify: { // MUST pass before resuming
|
|
661
|
+
kind: 'redispatch-original', // re-run the failed operation once
|
|
662
|
+
},
|
|
663
|
+
fallback: 'human', // 'retry-once' | 'human' | 'defer' | 'escalate-unclear'
|
|
664
|
+
humanReferralTemplate: '...', // seeds humanActionRequired when fallback=human
|
|
665
|
+
}
|
|
666
|
+
```
|
|
667
|
+
|
|
668
|
+
Four mandatory fields per entry -- detection signature, remedy, verification, fallback -- so
|
|
669
|
+
adding case #6 is a data change reviewed like any other, not prompt engineering or a code
|
|
670
|
+
restructure. The executor (section 2.5) is the only code that knows how to run each `remedy.verb`;
|
|
671
|
+
new verbs DO require code (they must be implemented against real MCP tools), but new *symptoms*
|
|
672
|
+
using existing verbs are pure data.
|
|
673
|
+
|
|
674
|
+
### 4.2 Seed entries (today's known cases)
|
|
675
|
+
|
|
676
|
+
| id | detect (signature/state) | remedy (verb) | verify | fallback |
|
|
677
|
+
|---|---|---|---|---|
|
|
678
|
+
| stale-llm-credential | reason 'auth'; `/authentication failed|not logged in/i` (AUTH_DISPATCH_RE, errors.mjs:382) | reprovision_llm_auth (existing heal, runner.js:7296-7308) | redispatch-original once | human (referral: run provider login on member; note local members use host OAuth and only interactive login fixes them) |
|
|
679
|
+
| stale-vcs-credential | VCS kind AUTH_EXPIRED (errors.mjs:25-28) | reprovision_vcs_auth (runner.js:1616-1659) | re-run failed git step once (runGitStep already does this) | human (AUTH_DENIED explicitly needs an operator grant, errors.mjs:29-32) |
|
|
680
|
+
| wedged-reservation | execute_prompt rejected: member reserved by a sprint id whose pid/ledger entry is dead (reservation enforcement in execute-prompt.ts; ledger liveness via supervisor watchdog states) | member_reservation force_release (src/tools/member-reservation.ts:17-27) | reserve succeeds for THIS sprint | human (never force-release a reservation whose owner is alive -- referral includes the owning sprint id and its watchdog status) |
|
|
681
|
+
| hung-remote-session | T2 pattern on one member; probes time out or `member_session_state` shows a stuck in-flight prompt | stop_prompt + kill session (memberSessionGuard.killIfAlive pattern, runner.js:7260), then redispatch | trivial probe (`CLI --version`) returns within timeout | human (example A referral: host-level ssh) |
|
|
682
|
+
| stale-dolt-clone | **OUT OF SCOPE for sprint-doctor.** Dolt/beads-sync wedging is excluded from LLM-escalatable scope entirely: `settleDoltConflicts()` (`fleet-sprint/dolt-settle.mjs`) resolves every row-level conflict shape deterministically, with zero LLM dispatch, wired at both divergence terminals in `dolt-sync.mjs`. An LLM escalation here would reintroduce the no-guaranteed-rollback recovery class the redesign eliminated. See `fleet-sprint/docs/dolt-sync-redesign.md` (Parts 2, 7.4, 8.4). | n/a -- not a registry entry | n/a | n/a |
|
|
683
|
+
| branch-not-synced-false-alarm | git fetch fails with `/couldn't find remote ref/i` on a branch another member just pushed (the one benign fetch-failure signature, runner.js:4588) | refetch_branch: bounded wait (30s) + one `git fetch` retry | ref present | retry-once, then human |
|
|
684
|
+
| member-cli-version-drift | `member_cli_version` probe differs from fleet baseline / known-bad version | none in v1 (update_llm_cli exists as an MCP tool but mid-sprint CLI upgrades are riskier than the disease) | n/a | human (referral: run update_llm_cli between sprints) |
|
|
685
|
+
|
|
686
|
+
### 4.3 Growing the registry from real incidents
|
|
687
|
+
|
|
688
|
+
The doctor MAY return `proposedRegistryEntry` (same shape as 4.1) when it diagnoses a novel,
|
|
689
|
+
recurring-looking symptom. The engine NEVER applies it -- it is appended to a
|
|
690
|
+
`doctor-proposals.jsonl` artifact beside the run state and surfaced in the harvest analysis
|
|
691
|
+
(`buildAnalysisText`, runner.js:3971-4029, gains a doctor section) and in the post-mortem skill
|
|
692
|
+
output. A human reviews and lands it as a normal PR to the registry file. This is the same
|
|
693
|
+
trust boundary as reviewer `newTasks`: propose in data, apply through the owner. (The registry
|
|
694
|
+
also resolves the coach design's open question 4 -- "should reports feed a local knowledge base
|
|
695
|
+
consulted on later invocations?" -- affirmatively, as reviewed data rather than free-form
|
|
696
|
+
self-priming: consults read the registry, proposals feed it through a human.)
|
|
697
|
+
|
|
698
|
+
### 4.4 Consent-gated upstreaming of engine-flaw reports (from coach)
|
|
699
|
+
|
|
700
|
+
Section 2.3's `engineFlawReport` stays local by default (artifact + harvest analysis + PR-body
|
|
701
|
+
block, section 7). The coach design solved the next step -- getting real field telemetry to the
|
|
702
|
+
ENGINE's maintainers without silent uploads -- and its flow is adopted whole:
|
|
703
|
+
|
|
704
|
+
- **Sanitize FIRST, locally, always** -- before the report is even written to disk: absolute
|
|
705
|
+
paths -> stable placeholders (`<work-folder>/...`, `<home>/...`); hostnames/usernames/IPs/
|
|
706
|
+
emails -> redacted tokens; anything matching secret patterns (keys, tokens, `{{secure.*}}`
|
|
707
|
+
values, env dumps) -> stripped outright, never placeholdered; target-repo identifiers (repo
|
|
708
|
+
name, branch names, bead titles) -> anonymized to `<target-repo>`/`<sprint-branch>` by
|
|
709
|
+
default, with an opt-in tier for internal users to keep them; code snippets excluded by
|
|
710
|
+
default, error text truncated to the minimal reproducing lines. A stable ANONYMOUS install id
|
|
711
|
+
(random UUID, never derived from machine identity) enables cross-report dedup without
|
|
712
|
+
identifying the user.
|
|
713
|
+
- **Consent modes** (fleet config, e.g. `doctor.telemetry`): `never` | `ask` (default) |
|
|
714
|
+
`always`. Under `ask`, reports sit in the run's artifacts and the sprint-end surfaces (PR
|
|
715
|
+
body section + dashboard card) show the exact sanitized text with a pre-filled
|
|
716
|
+
new-issue URL against the engine's configured upstream tracker -- one click, human-in-the-loop
|
|
717
|
+
by construction, nothing uploads silently, zero token handling. Under `always` (standing
|
|
718
|
+
consent: explicit config, per-install, revocable, recorded in the report footer), reports
|
|
719
|
+
auto-file via the user's own `gh` auth or a credential-store handle. Under `never`, local
|
|
720
|
+
only.
|
|
721
|
+
- **Dedup by fingerprint**: the report carries the normalized `errorSignature` hash; upstream,
|
|
722
|
+
a matching fingerprint appends an occurrence comment instead of opening a duplicate issue.
|
|
723
|
+
- **Generalization fix over the original coach sketch**: coach hardcoded the apra-fleet GitHub
|
|
724
|
+
issues URL as the upstream target. That is the dogfood leak this product must not ship --
|
|
725
|
+
the upstream tracker (owner/repo or full new-issue URL template) MUST come from fleet/engine
|
|
726
|
+
config with no default pointing anywhere; telemetry is simply disabled until it is set.
|
|
727
|
+
|
|
728
|
+
---
|
|
729
|
+
|
|
730
|
+
## 5. Model tier, frequency, and cost
|
|
731
|
+
|
|
732
|
+
**Tier: `premium`.** The consult is a low-frequency judgment call over ambiguous, mixed evidence --
|
|
733
|
+
exactly the class this engine already reserves premium for (planner / plan-reviewer / reviewer are
|
|
734
|
+
pinned `'premium'` with the vendor-contract note, runner.js:86-105; `FIXED_ROLE_TIER` at 102-116).
|
|
735
|
+
Add `'sprint-doctor': 'premium'` to that map. Tier resolution stays per-member via the registered
|
|
736
|
+
`model_tiers` / `resolveModelForTier()` (src/tools/execute-prompt.ts:236-243), so the design stays
|
|
737
|
+
mixed-provider-correct: `premium` means "this member's best reasoning model", never a hardcoded
|
|
738
|
+
model id.
|
|
739
|
+
|
|
740
|
+
**Frequency:** zero on a healthy sprint (all four triggers require sustained no-progress or
|
|
741
|
+
repeated infra failure). Expected 0-2 consults on a troubled sprint; hard cap
|
|
742
|
+
`doctor_max_consults` (default 3) + optional probe round each => at most 6 premium dispatches per
|
|
743
|
+
sprint, each single-turn with a capped (~16KB x 2 logs + ledger + bead JSON, roughly 30-60k input
|
|
744
|
+
tokens) context and a small JSON output. Order-of-magnitude: low single-digit USD per consult on
|
|
745
|
+
current premium pricing -- vs incident 2's ~15.8M output tokens burned by NOT having it. Doctor
|
|
746
|
+
spend flows through the existing `budget.spent()` accounting, so a user's `--budget` ceiling
|
|
747
|
+
bounds it like every other dispatch (`BudgetExceededError`, workflow index.mjs:760-765).
|
|
748
|
+
|
|
749
|
+
---
|
|
750
|
+
|
|
751
|
+
## 6. Failure modes of sprint-doctor itself, and its own circuit-breakers
|
|
752
|
+
|
|
753
|
+
| Failure | Bound |
|
|
754
|
+
|---|---|
|
|
755
|
+
| Doctor dispatch itself stalls/fails/returns garbage | **Strictly-additive rule:** any consult failure (dispatch error, schema-repair exhaustion, watchdog fire -- `withDispatchWatchdog` applies to it like everything, runner.js:4527-4549) causes the engine to behave exactly as it does today: T3 falls through to the original `StalledSprintError` throw; T1/T2/T4 log the failed consult and let existing re-lane/stall machinery proceed. The doctor can never make an outcome worse than the status quo. |
|
|
756
|
+
| Repeated bad calls | `doctor_max_consults` (3) per sprint; no-repeat rule enforced by the ENGINE (a (beadId, action.kind) pair executes at most once; a verdict prescribing an already-tried-and-failed action is overridden to the defer/abort path); the doctor's own consult history is in its input so a compliant model self-corrects first. |
|
|
757
|
+
| Cost runaway | Consult cap + probe cap (1) + capped input assembly + inclusion in `budget.spent()`. |
|
|
758
|
+
| Oscillation with the stall detector | Doctor actions never touch `staleCycles`/`highWaterClosedCount` directly; only genuine `progressScore` increases (and the doctor-deferred monotone credit set, section 2.5) reset staleness. T3 grants at most ONE extra cycle ever. |
|
|
759
|
+
| Environment repair loops | Every registry remedy latched once per member per sprint (the `authHealAttempted` pattern); every remedy has a mandatory `verify` step -- an unverified remedy counts as failed and consumes the latch (the "failed provision must never report success" rule, runner.js:1654-1656). |
|
|
760
|
+
| pause_for_human wedges | No timeout, no auto-abort -- by design, a pause must never degrade to a destructive action on human response time. What makes indefinite pause safe instead: reservations released on pause / cleanly re-acquired on resume, zero spend while parked, and loud "paused since <t>" surfacing in viewer + supervisor with the watchdog taught a PAUSED state (section 2.6.4). "Never wedged" is satisfied by *unambiguity*: paused is an explicit, visible, resumable state with a written referral -- the failure mode being designed away is a sprint that is silently dead, not one that is visibly waiting for its human. |
|
|
761
|
+
| Prompt injection via member logs | Zero-tool dispatch: injected text can at most skew a choice among enumerated actions, each of which is independently bounded above; `humanActionRequired.suggestedCommands` are DISPLAYED to the operator, never executed by the engine. |
|
|
762
|
+
| Doctor edits source (the forbidden move) | Structurally impossible in-sprint: no tools. In post-mortem skill mode (section 8) the invoking agent has whatever tools its harness grants -- the skill text states the boundary, and the recommended harness profile for post-mortem runs is the existing read-mostly reviewer profile (`skills/fleet/profiles/base-reviewer.json` via `compose_permissions role:'reviewer'`), which permits no source writes. |
|
|
763
|
+
| Same failure class healed over and over ("healing becomes a home") | Coach's same-class-twice policy, adopted: an error class (by `errorSignature` fingerprint) may be doctor-remedied at most `doctor_max_per_class` times per sprint (default 2, complementing the per-(bead, action) no-repeat rule); after that the class must fail loudly and demand an engine fix via the 4.4 report path. Healing is a bridge, never a home. |
|
|
764
|
+
| Recursive escalation (the doctor's own machinery fails and something tries to doctor THAT) | Coach's no-recursion rule, adopted and enforced by construction: doctor consults/repairs are engine-internal steps, not ledger-recorded dispatch outcomes, so they can never satisfy T1-T5; a failed consult routes ONLY to the strictly-additive fallback above. The doctor is never doctored. |
|
|
765
|
+
| Interventions hidden from quality signals | Every consult and every executed remedy is a first-class event in sprint state, threaded into the Final Review's evidence and the harvest analysis (the buildAnalysisText doctor section, runner.js:3971-4029) -- a PASS with 12 doctor saves must read differently from a clean PASS (coach's no-silent-saves rule). |
|
|
766
|
+
| Nondeterminism poisons the golden/mock test suites | Doctor outcomes are recorded in replayable form and the mock-sprint harness / golden-transcript suites stub them deterministically (same posture the suites already take toward every other dispatch), so adding the doctor cannot destabilize the regression baseline. |
|
|
767
|
+
|
|
768
|
+
---
|
|
769
|
+
|
|
770
|
+
## 7. Generalization audit (must work for ANY target project)
|
|
771
|
+
|
|
772
|
+
- **No bead-prefix assumptions:** all bead ids flow from the sprint's own scope reads
|
|
773
|
+
(`bdListScoped`, `bd show`); signatures strip ids during normalization.
|
|
774
|
+
- **No apra-fleet paths:** doctor inputs are runner-owned artifacts (its own log, ledger, run
|
|
775
|
+
state) plus member probes phrased against the member's registered work folder and provider CLI --
|
|
776
|
+
the same abstractions every existing phase uses. The generic playbook filenames fleet-sprint
|
|
777
|
+
already contracts on (`deploy.md`, `integ-test-playbook.md`, `regression-test-playbook.md`) are
|
|
778
|
+
product contract, not apra-fleet specifics.
|
|
779
|
+
- **Provider/model agnostic:** tiers only ('cheap|standard|premium'), resolved per member
|
|
780
|
+
(execute-prompt.ts:236-243, provider `modelForTier` implementations across all adapters).
|
|
781
|
+
- **Engine-flaw routing is the one real leak risk:** engine bugs belong in the ENGINE's tracker,
|
|
782
|
+
not the target project's beads DB. Therefore `engineFlawReport` is emitted as an artifact +
|
|
783
|
+
harvest-analysis section + PR-body block, and NEVER auto-filed as a bead anywhere. Upstreaming
|
|
784
|
+
goes exclusively through the section 4.4 consent flow (default: sanitized report + one-click
|
|
785
|
+
pre-filled issue against a config-supplied upstream tracker; no default target, no silent
|
|
786
|
+
uploads). This keeps the [fleet-sprint-product-vs-dogfood] separation: nothing target-specific
|
|
787
|
+
leaks into the engine, and nothing engine-specific pollutes a customer's DB.
|
|
788
|
+
- **Registry entries must stay generic:** detection signatures key off the engine's own structured
|
|
789
|
+
reasons and neutral taxonomies (errors.mjs), never off a target project's log text. Review rule
|
|
790
|
+
for new entries: if a signature quotes project-specific strings, it is rejected as a
|
|
791
|
+
target-side concern.
|
|
792
|
+
- **All thresholds are args** (validated alongside the existing arg contract, runner.js:2264-2448):
|
|
793
|
+
`doctor_enabled` (default true), `doctor_streak_limit`, `doctor_spend_trigger_usd`,
|
|
794
|
+
`doctor_max_consults`, `doctor_max_defers`, `doctor_max_per_class`; the telemetry mode and
|
|
795
|
+
upstream tracker live in fleet config (`doctor.telemetry`, section 4.4), not per-sprint args.
|
|
796
|
+
- **No MCP tool schema changes** are required for v1 (the doctor consumes existing verbs; the
|
|
797
|
+
pause/resume surface is viewer/supervisor HTTP routes plus engine primitives, not `src/tools/*`),
|
|
798
|
+
so no `packages/apra-fleet-client` sync is triggered.
|
|
799
|
+
|
|
800
|
+
---
|
|
801
|
+
|
|
802
|
+
## 8. Packaging as a skill: `sprint-doctor` (dual purpose)
|
|
803
|
+
|
|
804
|
+
**Location and distribution:** `packages/apra-fleet-se/fleet-sprint/skills/sprint-doctor/SKILL.md`,
|
|
805
|
+
beside `fleet-sprint-cli` (currently the only skill there, a single ~104-line SKILL.md with
|
|
806
|
+
name+description frontmatter). Vendored exactly like fleet-sprint-cli: add the directory to
|
|
807
|
+
`scripts/dist-pm.mjs` (:60-94, npm dist copy) and `scripts/gen-sea-config.mjs` (:104-110, SEA
|
|
808
|
+
binary asset) so npm-global and single-binary installs both ship it, provider-agnostic.
|
|
809
|
+
|
|
810
|
+
**Skill shape (SKILL.md sections):**
|
|
811
|
+
1. Frontmatter: `name: sprint-doctor`; description covering both invocation modes and the trigger
|
|
812
|
+
phrases ("sprint stalled", "sprint burning tokens", "post-mortem sprint logs", "recurring
|
|
813
|
+
sprint failure").
|
|
814
|
+
2. *Triage doctrine* -- the section-3 decision tree and signal table, written generically.
|
|
815
|
+
3. *Registry* -- points at `doctor-registry.mjs` as the live symptom/remedy table; documents the
|
|
816
|
+
4-field entry contract and the proposal-review flow (section 4.3). The skill never inlines a
|
|
817
|
+
copy of the table (single source of truth).
|
|
818
|
+
4. *Output contract* -- the section-2.3 schema, including classification, humanActionRequired
|
|
819
|
+
referral rules, and the "referral must be actionable" bar with examples A/B.
|
|
820
|
+
5. *In-sprint mode* -- states that the engine assembles inputs and executes actions; the skill's
|
|
821
|
+
doctrine text is what the runner embeds into the doctor persona's prompt (historically the same
|
|
822
|
+
pattern as the now-retired Tier-2 dolt dispatch being armed with a runbook -- dolt is no longer
|
|
823
|
+
part of this precedent; see section 4). The persona markdown references the skill doctrine so
|
|
824
|
+
the two cannot drift.
|
|
825
|
+
6. *Post-mortem mode* -- operator/agent-invoked over a BATCH of runs.
|
|
826
|
+
|
|
827
|
+
**How the two modes differ:**
|
|
828
|
+
|
|
829
|
+
| | In-sprint (engine-invoked) | Post-mortem (operator/agent-invoked) |
|
|
830
|
+
|---|---|---|
|
|
831
|
+
| Input | One incident: ledger excerpt, log tails, bead JSON -- pre-assembled by the runner | Many runs: `old_runs/<runId>.json` terminal states, runner logs, `doctor-consults.jsonl` + `doctor-proposals.jsonl` artifacts across sprints |
|
|
832
|
+
| Tools | None (zero-tool dispatch) | Read-only file access via the invoking harness; recommended under the reviewer permission profile |
|
|
833
|
+
| Output | Enum-bounded verdict, schema-enforced; engine executes | Free-form findings report: signature clusters, recurrence counts across sprints, ENVIRONMENT-vs-ENGINE_FLAW classification per cluster, proposed registry entries, proposed ENGINE-tracker bead titles with evidence quotes |
|
|
834
|
+
| Purpose | Unstick or cleanly kill THIS sprint | Mine REAL recurring engine flaws ("mutex not wired on path X", "stall threshold not tier-aware") vs one-off operational noise; feed the registry and the engine backlog |
|
|
835
|
+
| Actions executed | Yes, by the runner, bounded | Never -- purely advisory |
|
|
836
|
+
|
|
837
|
+
The JSONL artifacts the in-sprint mode writes (`doctor-consults`, `doctor-proposals`) are exactly
|
|
838
|
+
the corpus the post-mortem mode mines -- the two modes form a loop: incidents -> consults ->
|
|
839
|
+
post-mortem clustering -> registry entries + engine beads -> fewer future consults.
|
|
840
|
+
|
|
841
|
+
---
|
|
842
|
+
|
|
843
|
+
## 9. Recommended beads to file (titles + scope only; not created here)
|
|
844
|
+
|
|
845
|
+
1. **sprint-doctor: dispatch-health ledger + trigger layer (T1-T5)** -- Add the SprintHealthLedger
|
|
846
|
+
recording at the existing catch/outcome sites, errorSignature normalization, the five triggers
|
|
847
|
+
(T1-T4 pattern-driven, T5 single-shot red-state events per the coach taxonomy) with
|
|
848
|
+
arg-configurable thresholds, and the H1-H4 hook points. No LLM call yet; triggers just log.
|
|
849
|
+
Includes the JSONL artifact for post-mortem use.
|
|
850
|
+
2. **sprint-doctor: verdict schema, premium consult dispatch, and action executor** -- New role
|
|
851
|
+
persona + `sprint-doctor-output.json` schema through the contracts.mjs pattern; zero-tool
|
|
852
|
+
premium dispatch with one probe round; executor mapping each action.kind to existing verbs,
|
|
853
|
+
including the doctor-deferred exclusion/credit fix for `NOT_DONE_STATUSES` containing
|
|
854
|
+
`deferred` (runner.js:284, 8134-8138). Depends on 1.
|
|
855
|
+
3. **sprint-doctor: symptom/remedy registry with seed entries** -- `doctor-registry.mjs` with the
|
|
856
|
+
seven seed entries (section 4.2), the 4-field entry contract, per-remedy latches + mandatory
|
|
857
|
+
verify steps, and the `proposedRegistryEntry` capture-to-artifact flow. Depends on 2.
|
|
858
|
+
4. **sprint-doctor: cycle-stall interposition + SprintDoctorAbortError** -- T3 wiring: consult once
|
|
859
|
+
before the runner.js:8219 throw, one doctored extra cycle, then the original abort with doctor
|
|
860
|
+
details attached; new typed error added to `isTypedAbortError()` so abort reports/PR bodies
|
|
861
|
+
carry the diagnosis. Depends on 2.
|
|
862
|
+
|
|
863
|
+
*Framework beads (generic pause/resume, packages/apra-fleet-workflow -- usable by any workflow
|
|
864
|
+
script, independent of sprint-doctor):*
|
|
865
|
+
|
|
866
|
+
5. **workflow engine: pause/resume primitive with activity-entry gate + pause guard hook** --
|
|
867
|
+
`requestPause`/`requestResume` on FleetWorkflow as siblings of `requestStop()`
|
|
868
|
+
(index.mjs:499-503); the internal gate at `agent()`/`command()` entry (beside the budget
|
|
869
|
+
check, index.mjs:760-765); `'pause:requested'`/`'paused'`/`'resumed'` events carrying
|
|
870
|
+
phase/group labels from the run store; `setPauseGuard(fn)` deferral; paused-state declaration
|
|
871
|
+
at zero in-flight activities; gate promise rejects on `requestStop()`. Deliberately NOT
|
|
872
|
+
apra-fleet-9ub's journaled interrupt(); note on 9ub that this gate is its future journal
|
|
873
|
+
point.
|
|
874
|
+
6. **workflow viewer: Pause/Pausing/Paused/Resume UX** -- `POST /pause` / `POST /resume` routes
|
|
875
|
+
beside `POST /stop` (viewer/index.mjs:1253-1288), buttons beside Save/Stop
|
|
876
|
+
(viewer/index.mjs:186-187), button state machine driven by the new events, "paused since <t>"
|
|
877
|
+
badge + reason card. Generic to every workflow run, not fleet-sprint-specific. Depends on 5.
|
|
878
|
+
7. **supervisor: Pause/Resume controls + PAUSED watchdog state** -- Sprint Stack row buttons as
|
|
879
|
+
siblings of Stop/Restart (dashboard.mjs:128-137) in the same script pattern
|
|
880
|
+
(dashboard.mjs:248-437), proxying to the child viewer's pause/resume rather than the
|
|
881
|
+
kill+force-release route Stop uses; watchdog classifies a live-pid paused run as PAUSED;
|
|
882
|
+
base-drift indicator in the row. Depends on 6.
|
|
883
|
+
8. **fleet-sprint: clean-state pause guard + reservation handling + unconditional resume
|
|
884
|
+
re-sync** -- Register the open-bracket pause guard (counter at `withGitSync` entry/exit,
|
|
885
|
+
runner.js:4999, and around `DoltSync.syncBefore/After`, runner.js:8124); reservation
|
|
886
|
+
release-on-pause / owner-checked re-reserve-on-resume with clean resume failure naming
|
|
887
|
+
unavailable members (member-reservation.ts:38-68); the unconditional resume re-sync (git
|
|
888
|
+
fetch + `decideEnsureBranchAction` probe + `bd dolt pull`) on every re-acquired member.
|
|
889
|
+
Depends on 5. This is runner.js's ONLY pause-awareness.
|
|
890
|
+
|
|
891
|
+
*sprint-doctor beads (continued):*
|
|
892
|
+
|
|
893
|
+
9. **sprint-doctor: wire pause_for_human to the engine pause primitive** -- The thin caller:
|
|
894
|
+
`requestPause(reason)` at the consult point plus the referral card published via
|
|
895
|
+
`publishState('sprint-doctor', ...)` for the paused-state UI. Depends on 2 and 5 (and
|
|
896
|
+
benefits from 8's reservation handling, but does not own it).
|
|
897
|
+
10. **sprint-doctor skill (dual-mode) + dist wiring** -- SKILL.md per section 8, persona
|
|
898
|
+
references doctrine, additions to `scripts/dist-pm.mjs` and `scripts/gen-sea-config.mjs`;
|
|
899
|
+
post-mortem procedure over old_runs + doctor JSONL artifacts. Depends on 3 (registry exists
|
|
900
|
+
to reference).
|
|
901
|
+
11. **sprint-doctor: consent-gated engine-flaw telemetry** -- The section 4.4 flow: local
|
|
902
|
+
sanitization spec, anonymous install id, `doctor.telemetry` config (`never`/`ask`/`always`)
|
|
903
|
+
with a config-supplied upstream tracker and no default target, pre-filled new-issue URL in
|
|
904
|
+
the PR body + dashboard card, fingerprint dedup. Depends on 2 (report schema exists).
|
|
905
|
+
12. **docs: sprint-doctor phase + pause/resume semantics in fleet-sprint README + diagram** --
|
|
906
|
+
Document the new phase, the triage buckets, the registry contract, the clean-state pause
|
|
907
|
+
invariant and member-agnostic resume, and the never-wedged invariant in
|
|
908
|
+
`fleet-sprint/docs/README.md` and `fleet-sprint-diagram.md`. Depends on 4.
|