@hongmaple0820/scale-engine 0.25.0 → 0.27.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (171) hide show
  1. package/LICENSE +15 -15
  2. package/README.en.md +384 -346
  3. package/README.md +564 -529
  4. package/dist/adapters/AiderAdapter.js +52 -52
  5. package/dist/adapters/AntigravityAdapter.d.ts +4 -0
  6. package/dist/adapters/AntigravityAdapter.js +21 -0
  7. package/dist/adapters/AntigravityAdapter.js.map +1 -0
  8. package/dist/adapters/ClaudeCodeAdapter.d.ts +4 -1
  9. package/dist/adapters/ClaudeCodeAdapter.js +34 -34
  10. package/dist/adapters/ClaudeCodeAdapter.js.map +1 -1
  11. package/dist/adapters/ClineAdapter.d.ts +4 -0
  12. package/dist/adapters/ClineAdapter.js +20 -0
  13. package/dist/adapters/ClineAdapter.js.map +1 -0
  14. package/dist/adapters/CodexAdapter.js +28 -28
  15. package/dist/adapters/CursorAdapter.js +26 -26
  16. package/dist/adapters/DeepSeekTuiAdapter.js +97 -97
  17. package/dist/adapters/DoubaoAdapter.js +33 -33
  18. package/dist/adapters/GeminiAdapter.js +26 -26
  19. package/dist/adapters/GenericProjectAgentAdapter.d.ts +29 -0
  20. package/dist/adapters/GenericProjectAgentAdapter.js +204 -0
  21. package/dist/adapters/GenericProjectAgentAdapter.js.map +1 -0
  22. package/dist/adapters/HermesAdapter.js +26 -26
  23. package/dist/adapters/JCodeAdapter.d.ts +4 -0
  24. package/dist/adapters/JCodeAdapter.js +19 -0
  25. package/dist/adapters/JCodeAdapter.js.map +1 -0
  26. package/dist/adapters/KiloCodeAdapter.d.ts +4 -0
  27. package/dist/adapters/KiloCodeAdapter.js +20 -0
  28. package/dist/adapters/KiloCodeAdapter.js.map +1 -0
  29. package/dist/adapters/KimiAdapter.js +32 -32
  30. package/dist/adapters/KiroAdapter.js +26 -26
  31. package/dist/adapters/OpenClawAdapter.js +26 -26
  32. package/dist/adapters/OpenCodeAdapter.js +26 -26
  33. package/dist/adapters/QCoderAdapter.js +26 -26
  34. package/dist/adapters/QoderAdapter.d.ts +4 -0
  35. package/dist/adapters/QoderAdapter.js +21 -0
  36. package/dist/adapters/QoderAdapter.js.map +1 -0
  37. package/dist/adapters/TraeAdapter.js +26 -26
  38. package/dist/adapters/VSCAdapter.js +26 -26
  39. package/dist/adapters/WindsurfAdapter.js +32 -32
  40. package/dist/adapters/WorkBuddyAdapter.js +26 -26
  41. package/dist/adapters/index.d.ts +5 -0
  42. package/dist/adapters/index.js +15 -0
  43. package/dist/adapters/index.js.map +1 -1
  44. package/dist/api/cli.js +190 -56
  45. package/dist/api/cli.js.map +1 -1
  46. package/dist/api/doctor.js +10 -3
  47. package/dist/api/doctor.js.map +1 -1
  48. package/dist/api/quickstart.js +7 -1
  49. package/dist/api/quickstart.js.map +1 -1
  50. package/dist/artifact/sqliteStore.js +89 -89
  51. package/dist/artifact/types.d.ts +1 -1
  52. package/dist/cli/phaseCommands.js +53 -53
  53. package/dist/cli/phaseCommands.js.map +1 -1
  54. package/dist/context/AntiPatternRegistry.js +20 -20
  55. package/dist/context/ContextBudget.d.ts +14 -0
  56. package/dist/context/ContextBudget.js +50 -14
  57. package/dist/context/ContextBudget.js.map +1 -1
  58. package/dist/context/ContextBuilder.js +155 -155
  59. package/dist/context/ContextCompiler.d.ts +34 -0
  60. package/dist/context/ContextCompiler.js +120 -0
  61. package/dist/context/ContextCompiler.js.map +1 -0
  62. package/dist/eval/WorkflowEval.js +4 -6
  63. package/dist/eval/WorkflowEval.js.map +1 -1
  64. package/dist/evolution/EvolutionEngine.js +31 -31
  65. package/dist/evolution/EvolutionEvaluator.d.ts +2 -0
  66. package/dist/evolution/EvolutionEvaluator.js +7 -1
  67. package/dist/evolution/EvolutionEvaluator.js.map +1 -1
  68. package/dist/fsm/FSMAgentBridge.js +11 -11
  69. package/dist/governance/GovernanceRoi.d.ts +6 -1
  70. package/dist/governance/GovernanceRoi.js +32 -0
  71. package/dist/governance/GovernanceRoi.js.map +1 -1
  72. package/dist/guardrails/DependencyAuditor.js +38 -0
  73. package/dist/guardrails/DependencyAuditor.js.map +1 -1
  74. package/dist/hooks/HookGeneratorEnhanced.js +218 -218
  75. package/dist/index.d.ts +2 -1
  76. package/dist/index.js +3 -2
  77. package/dist/index.js.map +1 -1
  78. package/dist/knowledge/SQLiteKnowledgeBase.js +28 -28
  79. package/dist/memory/MemoryBrain.js +52 -52
  80. package/dist/output/GovernanceDashboard.js +44 -44
  81. package/dist/output/HTMLArtifactLayer.js +31 -31
  82. package/dist/prompts/VibeTemplateGallery.js +121 -121
  83. package/dist/runtime/AiOsRuntime.d.ts +53 -0
  84. package/dist/runtime/AiOsRuntime.js +142 -0
  85. package/dist/runtime/AiOsRuntime.js.map +1 -0
  86. package/dist/runtime/index.d.ts +1 -0
  87. package/dist/runtime/index.js +1 -0
  88. package/dist/runtime/index.js.map +1 -1
  89. package/dist/skills/SkillDiscovery.js +12 -1
  90. package/dist/skills/SkillDiscovery.js.map +1 -1
  91. package/dist/skills/routing/SkillPlanner.js +128 -40
  92. package/dist/skills/routing/SkillPlanner.js.map +1 -1
  93. package/dist/skills/routing/SkillRoutingTypes.d.ts +17 -0
  94. package/dist/tools/SafeCommandRunner.d.ts +16 -0
  95. package/dist/tools/SafeCommandRunner.js +83 -0
  96. package/dist/tools/SafeCommandRunner.js.map +1 -0
  97. package/dist/workflow/EngineeringStandards.js +62 -62
  98. package/dist/workflow/GovernanceTemplatePacks.d.ts +1 -1
  99. package/dist/workflow/GovernanceTemplatePacks.js +1990 -162
  100. package/dist/workflow/GovernanceTemplatePacks.js.map +1 -1
  101. package/dist/workflow/GovernanceTemplates.d.ts +2 -0
  102. package/dist/workflow/GovernanceTemplates.js +1012 -1001
  103. package/dist/workflow/GovernanceTemplates.js.map +1 -1
  104. package/dist/workflow/ResourceGovernance.js +16 -16
  105. package/dist/workflow/TaskArtifactScaffolder.js +10 -10
  106. package/dist/workflow/UpgradeManager.d.ts +3 -2
  107. package/dist/workflow/UpgradeManager.js +134 -49
  108. package/dist/workflow/UpgradeManager.js.map +1 -1
  109. package/dist/workflow/WorkspaceTopology.js +18 -15
  110. package/dist/workflow/WorkspaceTopology.js.map +1 -1
  111. package/dist/workflow/gates/GateSystem.js +3 -9
  112. package/dist/workflow/gates/GateSystem.js.map +1 -1
  113. package/docs/ACTIVE_SECURITY_VISUAL_GATES.md +87 -87
  114. package/docs/AI_ENGINEERING_OS_POSITIONING.md +462 -0
  115. package/docs/BACKGROUND_HUNTER.md +62 -62
  116. package/docs/CODE_INTELLIGENCE.md +138 -138
  117. package/docs/CONTEXT_BUDGET.md +155 -113
  118. package/docs/DEPENDENCY_AUDIT.md +118 -89
  119. package/docs/EVOLUTION_SHADOW_MODE.md +63 -63
  120. package/docs/EXTERNAL_REFERENCES.md +63 -58
  121. package/docs/GITLAB_FLOW.md +125 -125
  122. package/docs/GOVERNANCE_DASHBOARD.md +85 -85
  123. package/docs/MEMORY_BRAIN.md +104 -104
  124. package/docs/MEMORY_FABRIC.md +136 -134
  125. package/docs/README.md +102 -92
  126. package/docs/RUNTIME_EVIDENCE.md +101 -101
  127. package/docs/SKILL-REPOSITORY.md +57 -57
  128. package/docs/SKILL_RADAR.md +135 -122
  129. package/docs/THIRD_PARTY_SKILLS.md +57 -57
  130. package/docs/WORKFLOW_EVAL.md +151 -151
  131. package/docs/guides/DEVELOPMENT_WORKFLOW.md +80 -0
  132. package/docs/guides/GETTING_STARTED.md +50 -0
  133. package/docs/start/README.md +78 -72
  134. package/docs/start/agent-governance-demo.md +107 -107
  135. package/docs/start/quickstart.md +137 -127
  136. package/docs/start/workflow-upgrade.md +32 -8
  137. package/docs/workflow/README.md +67 -0
  138. package/docs/workflow/node-library.md +52 -0
  139. package/docs/workflow/templates/api-contract.md +29 -0
  140. package/docs/workflow/templates/architecture-review.md +23 -0
  141. package/docs/workflow/templates/db-change-plan.md +20 -0
  142. package/docs/workflow/templates/docs-impact.md +17 -0
  143. package/docs/workflow/templates/e2e-plan.md +20 -0
  144. package/docs/workflow/templates/explore.md +16 -0
  145. package/docs/workflow/templates/github-actions-scale-preflight.yml +32 -0
  146. package/docs/workflow/templates/mini-prd.md +16 -0
  147. package/docs/workflow/templates/plan.md +37 -0
  148. package/docs/workflow/templates/pre-push-scale-preflight.sh +8 -0
  149. package/docs/workflow/templates/product-smoke.md +61 -0
  150. package/docs/workflow/templates/reality-check.md +28 -0
  151. package/docs/workflow/templates/resource-cleanup.md +17 -0
  152. package/docs/workflow/templates/resource-impact.md +25 -0
  153. package/docs/workflow/templates/review.md +12 -0
  154. package/docs/workflow/templates/runtime.md +23 -0
  155. package/docs/workflow/templates/security-review.md +26 -0
  156. package/docs/workflow/templates/skill-evidence.md +33 -0
  157. package/docs/workflow/templates/skill-plan.md +39 -0
  158. package/docs/workflow/templates/spec.md +17 -0
  159. package/docs/workflow/templates/standards-impact.md +28 -0
  160. package/docs/workflow/templates/summary.md +16 -0
  161. package/docs/workflow/templates/tasks.md +8 -0
  162. package/docs/workflow/templates/ui-spec.md +29 -0
  163. package/docs/workflow/templates/verification.md +20 -0
  164. package/docs/workflow/templates/visual-review.md +20 -0
  165. package/examples/demo-projects/agent-governance-demo/CONTEXT.md +14 -14
  166. package/examples/demo-projects/agent-governance-demo/README.md +48 -48
  167. package/examples/demo-projects/agent-governance-demo/docs/CONTEXT-MAP.md +14 -14
  168. package/examples/demo-projects/agent-governance-demo/package.json +22 -21
  169. package/examples/demo-projects/agent-governance-demo/src/oauth-state.ts +39 -39
  170. package/examples/demo-projects/agent-governance-demo/tests/oauth-state.test.ts +52 -52
  171. package/package.json +95 -78
@@ -1,57 +1,57 @@
1
- # Third-Party Skills and External References
2
-
3
- This document records external skill projects that SCALE may learn from, recommend, or integrate with. It is a governance boundary, not a vendoring manifest. The complete cross-repo inventory is maintained in [External Reference Inventory](EXTERNAL_REFERENCES.md).
4
-
5
- ## Policy
6
-
7
- - Do not vendor third-party skill code, images, logos, examples, or marketing copy unless the license review explicitly allows redistribution.
8
- - Preserve upstream license text, copyright notices, NOTICE files, source URL, and source revision before any vendored or modified redistribution.
9
- - Mark modified files and document what changed from upstream.
10
- - Treat optional external services as review-required until privacy, retention, credential, and delete boundaries are reviewed.
11
- - `scale skill doctor --supply-chain` must include license, attribution, script, and pinned-revision checks for third-party skills.
12
- - Community skills start as `review-required`; promotion requires real installation evidence and a recorded safety decision.
13
-
14
- ## Highlighted External References
15
-
16
- | Project | License | Upstream | SCALE usage | Redistribution status |
17
- | --- | --- | --- | --- | --- |
18
- | Planning with Files | MIT | [OthmanAdi/planning-with-files](https://github.com/OthmanAdi/planning-with-files) | Adapt concepts for file-backed plans, findings, progress logs, active-plan routing, and plan attestation. | Not vendored. |
19
- | agentmemory | Apache-2.0 | [rohitg00/agentmemory](https://github.com/rohitg00/agentmemory) | Optional external memory provider via REST or MCP for teams that need cross-agent persistent memory beyond local SCALE Memory Brain. | Not vendored. |
20
- | GBrain | MIT | [garrytan/gbrain](https://github.com/garrytan/gbrain) | Optional graph memory provider for brain repos, hybrid search, entity relationships, MCP, and background maintenance. | Not vendored. |
21
-
22
- Other referenced skills, MCP servers, CLIs, discovery candidates, and adapter targets are listed in [External Reference Inventory](EXTERNAL_REFERENCES.md). Unknown licenses stay `review-required`; do not treat a repository link as redistribution permission.
23
-
24
- ## Acknowledgements
25
-
26
- SCALE acknowledges these upstream projects and contributors:
27
-
28
- - `OthmanAdi/planning-with-files`, Copyright (c) 2026 Ahmad Adi.
29
- - `rohitg00/agentmemory` and its upstream contributors.
30
- - `garrytan/gbrain` and its upstream contributors.
31
- - All upstream projects listed in [External Reference Inventory](EXTERNAL_REFERENCES.md) according to their licenses and contribution histories.
32
-
33
- The current SCALE implementation records these projects as external references or adapted concepts. It does not copy their source code into this repository.
34
-
35
- ## Vendoring Checklist
36
-
37
- If SCALE later vendors or modifies any third-party skill, the change must include:
38
-
39
- 1. Full upstream license text in the distributed package.
40
- 2. Upstream copyright and NOTICE material.
41
- 3. Source repository URL and pinned revision.
42
- 4. Modification notes for every copied or changed file.
43
- 5. Tests or doctor checks proving the attribution metadata is present.
44
- 6. README and generated skill repository documentation updates.
45
-
46
- ## Runtime Boundaries
47
-
48
- External memory providers must not be enabled silently. Before use, record:
49
-
50
- - provider endpoint and health check evidence
51
- - project data scope
52
- - credential boundary
53
- - retention and deletion policy
54
- - whether data leaves the local machine or team-controlled infrastructure
55
- - whether provider writes are disabled, candidate-only, or explicitly enabled
56
-
57
- External planning skills must not replace SCALE task evidence. They can improve the plan artifact shape, but final delivery still requires verification output, changed-file evidence, and explicit unverified-risk notes.
1
+ # Third-Party Skills and External References
2
+
3
+ This document records external skill projects that SCALE may learn from, recommend, or integrate with. It is a governance boundary, not a vendoring manifest. The complete cross-repo inventory is maintained in [External Reference Inventory](EXTERNAL_REFERENCES.md).
4
+
5
+ ## Policy
6
+
7
+ - Do not vendor third-party skill code, images, logos, examples, or marketing copy unless the license review explicitly allows redistribution.
8
+ - Preserve upstream license text, copyright notices, NOTICE files, source URL, and source revision before any vendored or modified redistribution.
9
+ - Mark modified files and document what changed from upstream.
10
+ - Treat optional external services as review-required until privacy, retention, credential, and delete boundaries are reviewed.
11
+ - `scale skill doctor --supply-chain` must include license, attribution, script, and pinned-revision checks for third-party skills.
12
+ - Community skills start as `review-required`; promotion requires real installation evidence and a recorded safety decision.
13
+
14
+ ## Highlighted External References
15
+
16
+ | Project | License | Upstream | SCALE usage | Redistribution status |
17
+ | --- | --- | --- | --- | --- |
18
+ | Planning with Files | MIT | [OthmanAdi/planning-with-files](https://github.com/OthmanAdi/planning-with-files) | Adapt concepts for file-backed plans, findings, progress logs, active-plan routing, and plan attestation. | Not vendored. |
19
+ | agentmemory | Apache-2.0 | [rohitg00/agentmemory](https://github.com/rohitg00/agentmemory) | Optional external memory provider via REST or MCP for teams that need cross-agent persistent memory beyond local SCALE Memory Brain. | Not vendored. |
20
+ | GBrain | MIT | [garrytan/gbrain](https://github.com/garrytan/gbrain) | Optional graph memory provider for brain repos, hybrid search, entity relationships, MCP, and background maintenance. | Not vendored. |
21
+
22
+ Other referenced skills, MCP servers, CLIs, discovery candidates, and adapter targets are listed in [External Reference Inventory](EXTERNAL_REFERENCES.md). Unknown licenses stay `review-required`; do not treat a repository link as redistribution permission.
23
+
24
+ ## Acknowledgements
25
+
26
+ SCALE acknowledges these upstream projects and contributors:
27
+
28
+ - `OthmanAdi/planning-with-files`, Copyright (c) 2026 Ahmad Adi.
29
+ - `rohitg00/agentmemory` and its upstream contributors.
30
+ - `garrytan/gbrain` and its upstream contributors.
31
+ - All upstream projects listed in [External Reference Inventory](EXTERNAL_REFERENCES.md) according to their licenses and contribution histories.
32
+
33
+ The current SCALE implementation records these projects as external references or adapted concepts. It does not copy their source code into this repository.
34
+
35
+ ## Vendoring Checklist
36
+
37
+ If SCALE later vendors or modifies any third-party skill, the change must include:
38
+
39
+ 1. Full upstream license text in the distributed package.
40
+ 2. Upstream copyright and NOTICE material.
41
+ 3. Source repository URL and pinned revision.
42
+ 4. Modification notes for every copied or changed file.
43
+ 5. Tests or doctor checks proving the attribution metadata is present.
44
+ 6. README and generated skill repository documentation updates.
45
+
46
+ ## Runtime Boundaries
47
+
48
+ External memory providers must not be enabled silently. Before use, record:
49
+
50
+ - provider endpoint and health check evidence
51
+ - project data scope
52
+ - credential boundary
53
+ - retention and deletion policy
54
+ - whether data leaves the local machine or team-controlled infrastructure
55
+ - whether provider writes are disabled, candidate-only, or explicitly enabled
56
+
57
+ External planning skills must not replace SCALE task evidence. They can improve the plan artifact shape, but final delivery still requires verification output, changed-file evidence, and explicit unverified-risk notes.
@@ -1,151 +1,151 @@
1
- # Workflow Eval Harness
2
-
3
- Status: implemented baseline
4
- Since: v0.22 development branch
5
-
6
- Workflow Eval Harness 用来证明工作流是否真的提升了 Agent 的工程交付质量,而不是只依赖主观感觉。它会运行轻量 eval suite,记录 pass@k、修复迭代、工具调用、token 估算、人类纠偏次数,并在失败时保留 Failure Replay。
7
-
8
- ## Commands
9
-
10
- 初始化默认基线套件:
11
-
12
- ```bash
13
- scale eval init
14
- scale eval init --suite workflow-baseline --json
15
- ```
16
-
17
- 运行套件:
18
-
19
- ```bash
20
- scale eval run --suite workflow-baseline
21
- scale eval run --suite workflow-baseline --json
22
- ```
23
-
24
- 对比两次运行:
25
-
26
- ```bash
27
- scale eval compare --baseline <run-id> --candidate <run-id>
28
- scale eval compare --baseline <run-id> --candidate <run-id> --json
29
- ```
30
-
31
- 生成 Markdown 报告:
32
-
33
- ```bash
34
- scale eval report --run <run-id>
35
- scale eval report --run <run-id> --output docs/worklog/eval-report.md
36
- ```
37
-
38
- 查看和提升失败重放:
39
-
40
- ```bash
41
- scale eval failures --since 30d
42
- scale eval replay <failure-id>
43
- scale eval replay --task-id <task-id>
44
- scale eval promote-failure <failure-id>
45
- ```
46
-
47
- ## Failure Replay To Memory
48
-
49
- Failure Replay is local eval evidence first. When a failure pattern is useful for future work, ingest it into Memory Brain as an `incident` candidate:
50
-
51
- ```bash
52
- scale memory ingest --from failure --failure-id <failure-id>
53
- scale memory query "missing verification evidence"
54
- scale memory promote <memory-node-id>
55
- ```
56
-
57
- This does not auto-change standards or hooks. It only makes the failure queryable and evidence-backed so repeated mistakes can be promoted deliberately after review.
58
-
59
- ## Storage
60
-
61
- ```text
62
- .scale/evals/
63
- ├── suites/
64
- ├── runs/
65
- ├── failures/
66
- └── improvements/
67
- ```
68
-
69
- These files are local runtime evidence by default. Commit only curated summaries or intentional benchmark fixtures.
70
-
71
- ## Suite Shape
72
-
73
- ```json
74
- {
75
- "version": "1.0",
76
- "id": "workflow-baseline",
77
- "name": "SCALE workflow baseline",
78
- "cases": [
79
- {
80
- "id": "governance-command-smoke",
81
- "type": "bugfix",
82
- "title": "Command evidence smoke",
83
- "task": "Verify that a local command can produce concrete eval evidence.",
84
- "phase": "verify",
85
- "successCriteria": ["command exits 0"],
86
- "attempts": [
87
- {
88
- "id": "attempt-1",
89
- "command": "node -e \"console.log('scale-eval-ok')\"",
90
- "expectedExitCode": 0,
91
- "outputContains": "scale-eval-ok"
92
- }
93
- ]
94
- }
95
- ]
96
- }
97
- ```
98
-
99
- ## Metrics
100
-
101
- | Metric | Meaning |
102
- | --- | --- |
103
- | `passAt1Rate` | 一次完整尝试就通过的比例 |
104
- | `passAt3Rate` | 三次以内通过的比例 |
105
- | `averageFixIterations` | 首次失败后的平均修复循环 |
106
- | `totalToolCalls` | eval attempts 数量,可近似衡量工具调用成本 |
107
- | `estimatedTokens` | task 与输出摘要的估算 token 成本 |
108
- | `humanCorrections` | 人类纠偏次数 |
109
- | `failureReplayCount` | 失败重放记录数量 |
110
-
111
- ## Failure Replay
112
-
113
- 失败不只记录最终失败状态,还会保存:
114
-
115
- - task and success criteria
116
- - phase
117
- - wrong turn
118
- - evidence
119
- - correction
120
- - prevention
121
- - replay command
122
- - redaction status
123
-
124
- Failure category 当前包括:
125
-
126
- - `wrong-exploration-path`
127
- - `hallucinated-project-fact`
128
- - `missing-codegraph-or-graph-fallback`
129
- - `over-broad-context-load`
130
- - `bad-skill-recommendation`
131
- - `missing-verification-evidence`
132
- - `failed-security-or-resource-gate`
133
- - `human-correction-after-agent-confidence`
134
- - `command-failure`
135
- - `unknown`
136
-
137
- `scale eval promote-failure` 会把失败重放提升为 improvement candidate,但不会自动修改项目规范。是否进入长期标准仍需要人工或后续 review 确认。
138
-
139
- ## Governance Use
140
-
141
- - v0.22 的默认 suite 是轻量 smoke baseline,用来验证 eval 管线可运行。
142
- - 真实项目应逐步增加 bugfix、feature、security、frontend、release、resource 类型案例。
143
- - Failure Replay 应与 Resource Governance 配合:默认本地保留,只有总结、基准或明确要长期维护的案例才提交。
144
- - Workflow Eval 的数据可以进入后续 Governance ROI,用来判断某个治理模块是否真的减少 rework、tool calls、token 或人类纠偏。
145
-
146
- ## Policy
147
-
148
- - 不允许用 eval 通过率替代真实项目验证。
149
- - 失败记录中的命令输出会做基础脱敏,但仍应避免把敏感原始日志写入 suite。
150
- - 低成本 smoke suite 可以频繁运行;重型项目 suite 应按需运行。
151
- - 没有 eval 证据时,不应宣称工作流能力已经提升。
1
+ # Workflow Eval Harness
2
+
3
+ Status: implemented baseline
4
+ Since: v0.22 development branch
5
+
6
+ Workflow Eval Harness 用来证明工作流是否真的提升了 Agent 的工程交付质量,而不是只依赖主观感觉。它会运行轻量 eval suite,记录 pass@k、修复迭代、工具调用、token 估算、人类纠偏次数,并在失败时保留 Failure Replay。
7
+
8
+ ## Commands
9
+
10
+ 初始化默认基线套件:
11
+
12
+ ```bash
13
+ scale eval init
14
+ scale eval init --suite workflow-baseline --json
15
+ ```
16
+
17
+ 运行套件:
18
+
19
+ ```bash
20
+ scale eval run --suite workflow-baseline
21
+ scale eval run --suite workflow-baseline --json
22
+ ```
23
+
24
+ 对比两次运行:
25
+
26
+ ```bash
27
+ scale eval compare --baseline <run-id> --candidate <run-id>
28
+ scale eval compare --baseline <run-id> --candidate <run-id> --json
29
+ ```
30
+
31
+ 生成 Markdown 报告:
32
+
33
+ ```bash
34
+ scale eval report --run <run-id>
35
+ scale eval report --run <run-id> --output docs/worklog/eval-report.md
36
+ ```
37
+
38
+ 查看和提升失败重放:
39
+
40
+ ```bash
41
+ scale eval failures --since 30d
42
+ scale eval replay <failure-id>
43
+ scale eval replay --task-id <task-id>
44
+ scale eval promote-failure <failure-id>
45
+ ```
46
+
47
+ ## Failure Replay To Memory
48
+
49
+ Failure Replay is local eval evidence first. When a failure pattern is useful for future work, ingest it into Memory Brain as an `incident` candidate:
50
+
51
+ ```bash
52
+ scale memory ingest --from failure --failure-id <failure-id>
53
+ scale memory query "missing verification evidence"
54
+ scale memory promote <memory-node-id>
55
+ ```
56
+
57
+ This does not auto-change standards or hooks. It only makes the failure queryable and evidence-backed so repeated mistakes can be promoted deliberately after review.
58
+
59
+ ## Storage
60
+
61
+ ```text
62
+ .scale/evals/
63
+ ├── suites/
64
+ ├── runs/
65
+ ├── failures/
66
+ └── improvements/
67
+ ```
68
+
69
+ These files are local runtime evidence by default. Commit only curated summaries or intentional benchmark fixtures.
70
+
71
+ ## Suite Shape
72
+
73
+ ```json
74
+ {
75
+ "version": "1.0",
76
+ "id": "workflow-baseline",
77
+ "name": "SCALE workflow baseline",
78
+ "cases": [
79
+ {
80
+ "id": "governance-command-smoke",
81
+ "type": "bugfix",
82
+ "title": "Command evidence smoke",
83
+ "task": "Verify that a local command can produce concrete eval evidence.",
84
+ "phase": "verify",
85
+ "successCriteria": ["command exits 0"],
86
+ "attempts": [
87
+ {
88
+ "id": "attempt-1",
89
+ "command": "node -e \"console.log('scale-eval-ok')\"",
90
+ "expectedExitCode": 0,
91
+ "outputContains": "scale-eval-ok"
92
+ }
93
+ ]
94
+ }
95
+ ]
96
+ }
97
+ ```
98
+
99
+ ## Metrics
100
+
101
+ | Metric | Meaning |
102
+ | --- | --- |
103
+ | `passAt1Rate` | 一次完整尝试就通过的比例 |
104
+ | `passAt3Rate` | 三次以内通过的比例 |
105
+ | `averageFixIterations` | 首次失败后的平均修复循环 |
106
+ | `totalToolCalls` | eval attempts 数量,可近似衡量工具调用成本 |
107
+ | `estimatedTokens` | task 与输出摘要的估算 token 成本 |
108
+ | `humanCorrections` | 人类纠偏次数 |
109
+ | `failureReplayCount` | 失败重放记录数量 |
110
+
111
+ ## Failure Replay
112
+
113
+ 失败不只记录最终失败状态,还会保存:
114
+
115
+ - task and success criteria
116
+ - phase
117
+ - wrong turn
118
+ - evidence
119
+ - correction
120
+ - prevention
121
+ - replay command
122
+ - redaction status
123
+
124
+ Failure category 当前包括:
125
+
126
+ - `wrong-exploration-path`
127
+ - `hallucinated-project-fact`
128
+ - `missing-codegraph-or-graph-fallback`
129
+ - `over-broad-context-load`
130
+ - `bad-skill-recommendation`
131
+ - `missing-verification-evidence`
132
+ - `failed-security-or-resource-gate`
133
+ - `human-correction-after-agent-confidence`
134
+ - `command-failure`
135
+ - `unknown`
136
+
137
+ `scale eval promote-failure` 会把失败重放提升为 improvement candidate,但不会自动修改项目规范。是否进入长期标准仍需要人工或后续 review 确认。
138
+
139
+ ## Governance Use
140
+
141
+ - v0.22 的默认 suite 是轻量 smoke baseline,用来验证 eval 管线可运行。
142
+ - 真实项目应逐步增加 bugfix、feature、security、frontend、release、resource 类型案例。
143
+ - Failure Replay 应与 Resource Governance 配合:默认本地保留,只有总结、基准或明确要长期维护的案例才提交。
144
+ - Workflow Eval 的数据可以进入后续 Governance ROI,用来判断某个治理模块是否真的减少 rework、tool calls、token 或人类纠偏。
145
+
146
+ ## Policy
147
+
148
+ - 不允许用 eval 通过率替代真实项目验证。
149
+ - 失败记录中的命令输出会做基础脱敏,但仍应避免把敏感原始日志写入 suite。
150
+ - 低成本 smoke suite 可以频繁运行;重型项目 suite 应按需运行。
151
+ - 没有 eval 证据时,不应宣称工作流能力已经提升。
@@ -0,0 +1,80 @@
1
+ # SCALE Engine 开发工作流
2
+
3
+ 这份文档说明日常如何在 `scale-engine` 仓库里按最新工程化工作流工作。
4
+
5
+ ## 标准闭环
6
+
7
+ ```text
8
+ 探索 -> 规划 -> 执行 -> 验证 -> 沉淀
9
+ ```
10
+
11
+ ## 1. 探索
12
+
13
+ 目标:先弄清真实仓库状态,再动手。
14
+
15
+ ```bash
16
+ make new-task NAME=task-slug LEVEL=M
17
+ make plan NAME=task-slug LEVEL=M
18
+ make explore FILES='AGENTS.md CLAUDE.md README.md package.json src/api/cli.ts' MSG='main contradiction'
19
+ make gate-workflow
20
+ ```
21
+
22
+ 最低要求:
23
+
24
+ - 至少读 3 个相关文件。
25
+ - 写清主矛盾,而不是只列文件名。
26
+ - 对不确定项明确标出,不靠猜。
27
+
28
+ ## 2. 规划
29
+
30
+ 在 `.planning/tasks/<task>/plan.md` 里至少补齐这些信息:
31
+
32
+ - scope / boundary
33
+ - acceptance criteria
34
+ - exception / failure path
35
+ - rollback / fallback
36
+ - verification commands
37
+
38
+ 如果任务改动发布、权限、安全、凭据、npm 发版或破坏性行为,按 `CRITICAL` 处理。
39
+
40
+ ## 3. 执行
41
+
42
+ 原则:
43
+
44
+ - 最小必要修改。
45
+ - 优先复用现有脚本和 `npm` 命令,不再发明第二套命令。
46
+ - 改 `src/` 行为时,原则上同步改 `tests/`,否则会被 G3 拦下。
47
+
48
+ ## 4. 验证
49
+
50
+ 推荐顺序:
51
+
52
+ ```bash
53
+ make gate-quality
54
+ make verify PROFILE=default
55
+ git diff --check
56
+ ```
57
+
58
+ 其中:
59
+
60
+ - `G4` 验证 workflow 脚本本身可解析。
61
+ - `G5` 运行 `lint + typecheck + test + build`。
62
+ - `G6` 检查任务证据和 diff hygiene。
63
+ - `G7` 是安全面,默认走 `npm audit --audit-level=high`。
64
+ - `G8` 检查 Markdown 与工作流文档的基础卫生。
65
+
66
+ ## 5. 沉淀
67
+
68
+ 应该留下:
69
+
70
+ - `verification.md`
71
+ - `review.md`
72
+ - `summary.md`
73
+ - 必要的长期规则文档更新
74
+
75
+ 不应该留下:
76
+
77
+ - 临时日志
78
+ - worktree 状态
79
+ - 截图、trace、缓存
80
+ - 只对一次任务有意义的中间文件
@@ -0,0 +1,50 @@
1
+ # SCALE Engine 仓库上手
2
+
3
+ 这份文档面向要开发 `scale-engine` 仓库本身的人,不是面向安装 CLI 的最终用户。
4
+
5
+ ## 15 分钟路径
6
+
7
+ 1. 先读根目录 [README.md](../../README.md)。
8
+ 2. 跑本仓库 workflow 预检:
9
+
10
+ ```bash
11
+ make preflight
12
+ ```
13
+
14
+ 3. 看当前可用验证面:
15
+
16
+ ```bash
17
+ make verify-list
18
+ ```
19
+
20
+ 4. 建一个任务骨架并记录探索:
21
+
22
+ ```bash
23
+ make new-task NAME=example LEVEL=M
24
+ make plan NAME=example LEVEL=M
25
+ make explore FILES='AGENTS.md CLAUDE.md README.md package.json' MSG='main contradiction'
26
+ make gate-workflow
27
+ ```
28
+
29
+ 5. 做完改动后跑质量面:
30
+
31
+ ```bash
32
+ make gate-quality
33
+ make verify PROFILE=default
34
+ git diff --check
35
+ ```
36
+
37
+ ## 你应该看到什么
38
+
39
+ - `.scale/workspace.json` 明确了 `dev -> master` 的仓库分支策略。
40
+ - `.agent/project.json` 定义了本仓库的 Node/TypeScript 验证命令。
41
+ - `scripts/gates/*` 和 `scripts/workflow/*` 不是说明文档,而是可执行入口。
42
+ - `.planning/tasks/<date>-<task>/` 用于任务级证据,不再把临时过程写进 `docs/`。
43
+
44
+ ## 常见误区
45
+
46
+ - `make gate-workflow` 通过,不代表代码质量通过。
47
+ - `make gate-quality` 通过,也不代表你已经记录了风险、回滚和未验证项。
48
+ - `G8` 会检查改动过的 Markdown 和工作流文档卫生,不替代业务验证。
49
+ - `--dry-run` 只能证明入口存在,不能写成“测试通过”。
50
+ - 不要把 `.claude/worktrees/`、`.agent/state/`、日志或截图提交进仓库。