solmate-skills 2.0.12 → 2.0.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +9 -0
- package/CLAUDE.md +13 -3
- package/README.md +91 -5
- package/USAGE.md +118 -12
- package/bin/cli.js +171 -1
- package/bin/harness-artifact.js +621 -0
- package/bin/harness-artifact.test.js +520 -0
- package/bin/harness-check.js +373 -0
- package/bin/harness-check.test.js +159 -0
- package/bin/test.js +2 -0
- package/docs/01_Concept_Design/01_AGENT_HARNESS_REQUIREMENTS_ANALYSIS.md +162 -0
- package/docs/03_Technical_Specs/01_AGENT_HARNESS_ARCHITECTURE.md +376 -0
- package/docs/03_Technical_Specs/assets/01_agent_harness_data_flow.svg +94 -0
- package/docs/04_Logic_Progress/00_BACKLOG.md +339 -0
- package/docs/04_Logic_Progress/01_EXECUTION_PLAN.md +160 -0
- package/docs/04_Logic_Progress/03_DECISION_LOG.md +98 -0
- package/docs/05_QA_Validation/01_TEST_SCENARIOS.md +313 -0
- package/docs/05_QA_Validation/02_AGENT_HARNESS_DESIGN_REVIEW.md +100 -0
- package/docs/05_QA_Validation/03_AGENT_HARNESS_CONTRACT_QA.md +96 -0
- package/manage-collaboration/SKILL.md +2 -0
- package/package.json +6 -4
- package/rules-docs/SKILL.md +15 -1
- package/rules-product/SKILL.md +30 -6
- package/rules-product/agents/openai.yaml +2 -2
- package/rules-workflow/SKILL.md +32 -7
- package/rules-workflow/adapters/claude/solmate-context-reader.md +17 -0
- package/rules-workflow/adapters/claude/solmate-implementer.md +20 -0
- package/rules-workflow/adapters/claude/solmate-verifier.md +21 -0
- package/rules-workflow/agents/openai.yaml +2 -2
- package/rules-workflow/resources/agent-harness-contract.md +187 -0
- package/rules-workflow/resources/agent-harness-v1.schema.json +432 -0
- package/verify-docs/SKILL.md +33 -6
- package/verify-docs/agents/openai.yaml +2 -2
- package/verify-implementation/SKILL.md +14 -2
- package/verify-implementation/agents/openai.yaml +2 -2
- package/verify-skills/SKILL.md +27 -0
|
@@ -0,0 +1,313 @@
|
|
|
1
|
+
# Agent Harness Test Scenarios
|
|
2
|
+
> Created: 2026-07-17 01:04
|
|
3
|
+
> Last Updated: 2026-07-17 01:53
|
|
4
|
+
|
|
5
|
+
## 1. Purpose
|
|
6
|
+
|
|
7
|
+
This document defines design-stage acceptance and future implementation scenarios for the enhanced Solmate agent harness. A scenario is not considered passing until independent evidence identifies the command, fixture, result, and relevant requirement.
|
|
8
|
+
|
|
9
|
+
## 2. Global Rubric Validation
|
|
10
|
+
|
|
11
|
+
| Criterion | Status | Evidence / Design Response |
|
|
12
|
+
|---|---|---|
|
|
13
|
+
| Functionality | Pass at design stage | Requirements, architecture, state transitions, messages, errors, execution plan, and tests are defined; implementation remains pending |
|
|
14
|
+
| Potential Impact | Pass at design stage | The design targets the recurring failures of unread context, improvised workflows, conflicting agents, and unsupported completion claims |
|
|
15
|
+
| Novelty | Pass at design stage | Solmate combines role/topology design with machine-enforced document and independent-verification Receipts rather than relying on personas alone |
|
|
16
|
+
| UX | Pass at design stage | Minimal topology is the default; optional specialists and warning rollout limit procedural and latency overhead |
|
|
17
|
+
| Open-source | Pass at design stage | Runtime-neutral contracts, standard JSON/JSONL, and thin adapters allow reuse without a proprietary orchestration service |
|
|
18
|
+
| Business Plan | Pass at design stage | Reliable Claude/Codex workflow behavior strengthens the package's value and reduces support cost from inconsistent agent execution |
|
|
19
|
+
|
|
20
|
+
Implementation-stage status must replace these design-stage statements with executable evidence.
|
|
21
|
+
|
|
22
|
+
## 3. Requirement Traceability Matrix
|
|
23
|
+
|
|
24
|
+
| Requirement | Architecture Coverage | Primary Scenario | Design Status |
|
|
25
|
+
|---|---|---|---|
|
|
26
|
+
| FR-001 | Artifact Model, Receipt Integration | QA-REQ-001 | Covered |
|
|
27
|
+
| FR-002 | Role Catalog, Activation Rules | QA-ROLE-001 | Covered |
|
|
28
|
+
| FR-003 | Persona Contract, Permissions | QA-ROLE-002 | Covered |
|
|
29
|
+
| FR-004 | Receipt Integration, Compatibility | QA-COMPAT-001 | Covered |
|
|
30
|
+
| FR-005 | Optional Specialists, State Machine | QA-DESIGN-001 | Covered |
|
|
31
|
+
| FR-006 | Message Protocol | QA-SCHEMA-001, QA-MSG-001, QA-MSG-002 | Covered |
|
|
32
|
+
| FR-007 | State Machine, Error Handling | QA-STATE-001, QA-STATE-002 | Covered |
|
|
33
|
+
| FR-008 | Permissions, QA Role | QA-QA-001, QA-QA-002 | Covered |
|
|
34
|
+
| FR-009 | Error Handling | QA-ERR-001 through QA-ERR-006 | Covered |
|
|
35
|
+
| FR-010 | Runtime Adapters | QA-ADAPTER-001 | Covered |
|
|
36
|
+
| FR-011 | Execution Workspace | QA-RECOVERY-001, QA-RECOVERY-002 | Covered |
|
|
37
|
+
| FR-012 | Requirements Traceability | QA-TRACE-001 | Covered |
|
|
38
|
+
| FR-013 | Topology Selection | QA-TOPO-001 through QA-TOPO-003 | Covered |
|
|
39
|
+
| FR-014 | Compatibility And Maintenance | QA-DRIFT-001 | Covered |
|
|
40
|
+
| FR-015 | Warning Pilot | QA-PILOT-001 | Covered |
|
|
41
|
+
|
|
42
|
+
### 3.1 Non-Functional Requirement Traceability
|
|
43
|
+
|
|
44
|
+
| Requirement | Primary Scenario | Design Status |
|
|
45
|
+
|---|---|---|
|
|
46
|
+
| NFR-001 | QA-SEC-001, QA-ERR-006 | Covered |
|
|
47
|
+
| NFR-002 | QA-STATE-001, QA-RECOVERY-002 | Covered |
|
|
48
|
+
| NFR-003 | QA-ADAPTER-001 | Covered |
|
|
49
|
+
| NFR-004 | QA-ROLE-001, QA-TOPO-001 | Covered |
|
|
50
|
+
| NFR-005 | QA-COST-001, QA-PILOT-001 | Covered |
|
|
51
|
+
| NFR-006 | QA-SCHEMA-001, QA-MSG-001, QA-STATE-002 | Covered |
|
|
52
|
+
| NFR-007 | QA-COMPAT-001 | Covered |
|
|
53
|
+
| NFR-008 | QA-QA-001, QA-QA-002 | Covered |
|
|
54
|
+
| NFR-009 | QA-RECOVERY-001, QA-RECOVERY-002 | Covered |
|
|
55
|
+
| NFR-010 | QA-LICENSE-001 | Covered |
|
|
56
|
+
|
|
57
|
+
## 4. Requirements And Context Scenarios
|
|
58
|
+
|
|
59
|
+
### QA-REQ-001 - Requirements Artifact Completeness
|
|
60
|
+
|
|
61
|
+
- **Given**: A new feature request with product behavior and implementation expectations.
|
|
62
|
+
- **When**: Requirements Analyst prepares the canonical analysis.
|
|
63
|
+
- **Then**: The document contains user need, FR, NFR, included/excluded scope, acceptance criteria, assumptions, unresolved questions, risks, and related-document links.
|
|
64
|
+
- **Failure**: Implementation begins from chat text without the required artifact or explicit small-task skip decision.
|
|
65
|
+
- **Evidence**: Parsed document fields and Coordinator acceptance event.
|
|
66
|
+
|
|
67
|
+
### QA-CONTEXT-001 - Linked Document Read Proof
|
|
68
|
+
|
|
69
|
+
- **Given**: A code task links Concept, Technical, and QA documents.
|
|
70
|
+
- **When**: Context Reader returns a Context Receipt.
|
|
71
|
+
- **Then**: Every linked path exists and appears in `Required References Read`; constraints and conflicts are present.
|
|
72
|
+
- **Failure**: Any required document is missing, unread, or silently omitted.
|
|
73
|
+
- **Evidence**: Existing strict `preflight` command plus manifest references.
|
|
74
|
+
|
|
75
|
+
### QA-CONTEXT-002 - Conflicting Documents
|
|
76
|
+
|
|
77
|
+
- **Given**: Two linked documents define incompatible behavior.
|
|
78
|
+
- **When**: Context Reader compares them.
|
|
79
|
+
- **Then**: State becomes `BLOCKED_CONTEXT`; both sources are preserved and Coordinator routes a decision.
|
|
80
|
+
- **Failure**: Context Reader chooses one silently or Implementer starts work.
|
|
81
|
+
|
|
82
|
+
## 5. Role And Design Scenarios
|
|
83
|
+
|
|
84
|
+
### QA-ROLE-001 - Minimal Task Avoids Optional Specialists
|
|
85
|
+
|
|
86
|
+
- **Given**: A small, single-module change with complete requirements and no public contract or state-machine change.
|
|
87
|
+
- **When**: Coordinator selects topology.
|
|
88
|
+
- **Then**: Core linear path is used; Requirements Analyst, Architect, Logic Builder, Security Inspector, and Release Guardian are recorded as `Skipped - reason` where considered.
|
|
89
|
+
- **Failure**: Full specialist team is spawned only because multiple roles are available.
|
|
90
|
+
|
|
91
|
+
### QA-ROLE-002 - Persona Contract Completeness
|
|
92
|
+
|
|
93
|
+
- **Given**: A runtime agent definition.
|
|
94
|
+
- **When**: Persona validation runs.
|
|
95
|
+
- **Then**: All persona fields are present, permissions match role responsibility, and denied tools are explicit for read-only roles.
|
|
96
|
+
- **Failure**: A narrative persona has no non-goals, output contract, escalation, completion evidence, or write scope.
|
|
97
|
+
|
|
98
|
+
### QA-DESIGN-001 - Architect And Logic Builder Activation
|
|
99
|
+
|
|
100
|
+
- **Given**: A feature changes an API contract and introduces three business states with failure transitions.
|
|
101
|
+
- **When**: Coordinator evaluates specialists.
|
|
102
|
+
- **Then**: Architect and Logic Builder are both activated, with non-overlapping outputs and linked Design Receipt.
|
|
103
|
+
- **Failure**: Implementer invents the API or state behavior during coding.
|
|
104
|
+
|
|
105
|
+
## 6. Message And State Scenarios
|
|
106
|
+
|
|
107
|
+
### QA-SCHEMA-001 - Versioned Artifact Schema Selection
|
|
108
|
+
|
|
109
|
+
- **Given**: A v1 manifest, message, or individual event object.
|
|
110
|
+
- **When**: A standard Draft 2020-12 validator or the Solmate CLI selects the artifact contract.
|
|
111
|
+
- **Then**: Top-level `oneOf` selects exactly one artifact shape, `schema_version: 1` is required, and every non-empty JSONL line validates as an event before ordered-log checks run.
|
|
112
|
+
- **Failure**: The top-level schema accepts an unconstrained object, an unsupported version passes, or malformed JSONL is treated as a contract warning instead of an operational error.
|
|
113
|
+
|
|
114
|
+
### QA-MSG-001 - Valid Structured Handoff
|
|
115
|
+
|
|
116
|
+
- **Given**: Architect completes an approved Architecture Decision Packet.
|
|
117
|
+
- **When**: A `HANDOFF` message is emitted.
|
|
118
|
+
- **Then**: Schema version, message/task IDs, attempt, timestamp, sender, recipient, status, summary, requirement/artifact/decision refs, blockers, next action, and retry count validate.
|
|
119
|
+
- **Failure**: A free-form “done” message is accepted without an artifact.
|
|
120
|
+
|
|
121
|
+
### QA-MSG-002 - Direct Peer Decision Is Rejected
|
|
122
|
+
|
|
123
|
+
- **Given**: Frontend Implementer asks Backend Implementer to change an accepted API response.
|
|
124
|
+
- **When**: Backend replies directly with a scope-changing decision.
|
|
125
|
+
- **Then**: Direct peer messages are limited to `STATUS` or `QUESTION` with `INFO` or `PENDING`; PASS, FAIL, BLOCKED, decision, rework, and completion claims route through Coordinator.
|
|
126
|
+
- **Failure**: The API and UI drift without an updated requirement or decision.
|
|
127
|
+
|
|
128
|
+
### QA-STATE-001 - Legal Transition Sequence
|
|
129
|
+
|
|
130
|
+
- **Given**: All required artifacts pass.
|
|
131
|
+
- **When**: Coordinator advances the task.
|
|
132
|
+
- **Then**: Every state follows a legal transition and records reason, actor, timestamp, and artifacts.
|
|
133
|
+
- **Evidence**: Ordered `events.jsonl` fixture.
|
|
134
|
+
|
|
135
|
+
### QA-STATE-002 - Completion Bypass Is Rejected
|
|
136
|
+
|
|
137
|
+
- **Given**: A Change Receipt exists but independent verification has not passed.
|
|
138
|
+
- **When**: A role requests `CHANGE_READY -> COMPLETE`.
|
|
139
|
+
- **Then**: The transition is rejected as a contract error.
|
|
140
|
+
- **Failure**: PR, merge, publish, or deploy is reported ready.
|
|
141
|
+
|
|
142
|
+
## 7. Error And Recovery Scenarios
|
|
143
|
+
|
|
144
|
+
### QA-ERR-001 - Context Error Does Not Retry
|
|
145
|
+
|
|
146
|
+
- **Given**: A required technical document is missing.
|
|
147
|
+
- **Then**: State becomes `BLOCKED_CONTEXT`; no automatic retry or implementation occurs.
|
|
148
|
+
|
|
149
|
+
### QA-ERR-002 - Transient Tool Error Retries Once
|
|
150
|
+
|
|
151
|
+
- **Given**: An otherwise valid non-destructive command fails from a temporary runtime interruption.
|
|
152
|
+
- **Then**: One retry occurs in a new recorded attempt with unchanged scope.
|
|
153
|
+
- **Failure**: Retry changes requirements, deletes prior evidence, or loops indefinitely.
|
|
154
|
+
|
|
155
|
+
### QA-ERR-003 - Repeated Transient Failure Blocks Dependency
|
|
156
|
+
|
|
157
|
+
- **Given**: The allowed retry fails again.
|
|
158
|
+
- **Then**: State becomes `BLOCKED_DEPENDENCY` and reports attempts, evidence, and recovery condition.
|
|
159
|
+
|
|
160
|
+
### QA-ERR-004 - Verification Failure Routes Rework
|
|
161
|
+
|
|
162
|
+
- **Given**: QA finds a failed acceptance criterion.
|
|
163
|
+
- **Then**: QA emits a finding; Coordinator sets `REWORK`; Implementer creates a new attempt and Change Receipt.
|
|
164
|
+
- **Failure**: QA modifies source files or marks its own fix as verified.
|
|
165
|
+
|
|
166
|
+
### QA-ERR-005 - Rework Loop Limit
|
|
167
|
+
|
|
168
|
+
- **Given**: Two implementation rework attempts fail the same criterion.
|
|
169
|
+
- **Then**: Coordinator pauses further changes and requests a user decision with the repeated failure pattern.
|
|
170
|
+
- **Failure**: A third automatic loop starts.
|
|
171
|
+
|
|
172
|
+
### QA-ERR-006 - Critical Artifact Failure Cannot Degrade Silently
|
|
173
|
+
|
|
174
|
+
- **Given**: Architect, Implementer, or QA fails without producing a required critical artifact.
|
|
175
|
+
- **Then**: Code/deploy task remains blocked; final output explicitly names the missing artifact.
|
|
176
|
+
- **Failure**: Remaining partial results are used to report completion.
|
|
177
|
+
|
|
178
|
+
### QA-RECOVERY-001 - Interrupted Task Resume
|
|
179
|
+
|
|
180
|
+
- **Given**: The active agent stops after valid requirements and context events.
|
|
181
|
+
- **When**: The task resumes.
|
|
182
|
+
- **Then**: Runtime loads the latest valid manifest and events, creates or resumes the legal attempt, and does not repeat approved work unnecessarily.
|
|
183
|
+
|
|
184
|
+
### QA-RECOVERY-002 - Attempt Preservation
|
|
185
|
+
|
|
186
|
+
- **Given**: Attempt 1 fails and attempt 2 begins.
|
|
187
|
+
- **Then**: Attempt 1 artifacts and failure evidence remain immutable and manifest points to attempt 2 as active.
|
|
188
|
+
|
|
189
|
+
## 8. Independent QA And Boundary Scenarios
|
|
190
|
+
|
|
191
|
+
### QA-QA-001 - QA Write Attempt Is Denied
|
|
192
|
+
|
|
193
|
+
- **Given**: QA Inspector identifies a code defect.
|
|
194
|
+
- **When**: It attempts to use Edit or Write on implementation files.
|
|
195
|
+
- **Then**: Runtime permission or contract validation rejects the operation and records the violation.
|
|
196
|
+
|
|
197
|
+
### QA-QA-002 - Implementer Cannot Self-Verify
|
|
198
|
+
|
|
199
|
+
- **Given**: Implementer reports basic checks as passing.
|
|
200
|
+
- **Then**: Change Receipt is accepted as a handoff but cannot satisfy Verification Receipt or completion.
|
|
201
|
+
|
|
202
|
+
### QA-BOUNDARY-001 - Producer And Consumer Contract Comparison
|
|
203
|
+
|
|
204
|
+
- **Given**: A producer returns a structured response and a consumer declares its expected type.
|
|
205
|
+
- **Then**: QA reads both sides, compares shape, optionality, naming, and failure behavior, and links evidence.
|
|
206
|
+
- **Failure**: QA checks only that both files exist or only that compilation passes.
|
|
207
|
+
|
|
208
|
+
### QA-BOUNDARY-002 - State Definition And Mutation Comparison
|
|
209
|
+
|
|
210
|
+
- **Given**: A Logic Contract defines legal transitions.
|
|
211
|
+
- **Then**: QA compares every implementation mutation and UI branch against reachable states and identifies unauthorized or dead transitions.
|
|
212
|
+
|
|
213
|
+
### QA-TRACE-001 - End-To-End Requirement Evidence
|
|
214
|
+
|
|
215
|
+
- **Given**: A P0 functional requirement is marked complete.
|
|
216
|
+
- **When**: Traceability validation runs.
|
|
217
|
+
- **Then**: The requirement links to its approved Requirements Analysis, required Architecture or Logic artifact, Change Receipt or diff, and independent QA evidence.
|
|
218
|
+
- **Failure**: A requirement is marked `Pass` from implementation claims alone, or a required design artifact is absent without `Skipped - reason`.
|
|
219
|
+
|
|
220
|
+
## 9. Topology And Adapter Scenarios
|
|
221
|
+
|
|
222
|
+
### QA-TOPO-001 - Read-Only Fan-Out/Fan-In
|
|
223
|
+
|
|
224
|
+
- **Given**: Four independent document groups require analysis.
|
|
225
|
+
- **Then**: Read-only agents run in parallel, write no shared source, and Coordinator integrates artifact references and conflicts.
|
|
226
|
+
|
|
227
|
+
### QA-TOPO-002 - Parallel Write Ownership
|
|
228
|
+
|
|
229
|
+
- **Given**: Frontend and Backend work can proceed in parallel.
|
|
230
|
+
- **Then**: Exclusive paths are canonical project-relative paths; aliases and overlapping literal or recursive scopes are rejected, while shared contracts are frozen or owned serially.
|
|
231
|
+
- **Failure**: Both agents modify the same shared file without coordination.
|
|
232
|
+
|
|
233
|
+
### QA-TOPO-003 - Hierarchy Depth Limit
|
|
234
|
+
|
|
235
|
+
- **Given**: A proposed topology delegates more than two levels deep.
|
|
236
|
+
- **Then**: Coordinator flattens or rejects the topology unless the user approves a documented exception.
|
|
237
|
+
|
|
238
|
+
### QA-ADAPTER-001 - Claude And Codex Contract Parity
|
|
239
|
+
|
|
240
|
+
- **Given**: The same fixture task runs through both adapters.
|
|
241
|
+
- **Then**: Role activation, canonical states, required artifact refs, failure classification, and Receipt fields are semantically equivalent.
|
|
242
|
+
- **Allowed Difference**: Native tool names, peer-message support, concurrency, and resume capability.
|
|
243
|
+
|
|
244
|
+
### QA-ADAPTER-002 - Degraded Independent Agent Capability
|
|
245
|
+
|
|
246
|
+
- **Given**: A runtime cannot provide an independent QA execution context.
|
|
247
|
+
- **Then**: State becomes `DEGRADED`; user approval is required before any completion claim and the limitation remains visible.
|
|
248
|
+
|
|
249
|
+
## 10. Compatibility, Drift, And Security Scenarios
|
|
250
|
+
|
|
251
|
+
### QA-COMPAT-001 - Existing Receipt Fixtures Remain Valid
|
|
252
|
+
|
|
253
|
+
- **Given**: Current valid Context and Verification Receipt fixtures.
|
|
254
|
+
- **When**: The enhanced validator runs in compatibility mode.
|
|
255
|
+
- **Then**: Existing commands and exit codes remain valid; new fields are warning-only until migration.
|
|
256
|
+
|
|
257
|
+
### QA-DRIFT-001 - Role And Adapter Drift Detection
|
|
258
|
+
|
|
259
|
+
- **Given**: A role is renamed or a runtime adapter omits a mandatory persona field.
|
|
260
|
+
- **Then**: Package verification identifies the stale reference, duplicate role, or missing adapter mapping.
|
|
261
|
+
|
|
262
|
+
### QA-COST-001 - Agent And Model Cost Is Proportional
|
|
263
|
+
|
|
264
|
+
- **Given**: Equivalent small, medium, and complex task fixtures.
|
|
265
|
+
- **Then**: The small fixture uses the minimal role path; optional specialists and stronger model tiers appear only with a recorded task need.
|
|
266
|
+
- **Failure**: Every task uses the full team or globally forces the most expensive model tier.
|
|
267
|
+
|
|
268
|
+
### QA-LICENSE-001 - External Reference Attribution And Originality
|
|
269
|
+
|
|
270
|
+
- **Given**: A new contract or persona is informed by an external open-source project.
|
|
271
|
+
- **Then**: Source references and applicable license obligations are reviewed, while Solmate text and schemas are independently authored unless copied material is explicitly attributed and licensed.
|
|
272
|
+
- **Failure**: Substantial external text is copied without notice, source, or license review.
|
|
273
|
+
|
|
274
|
+
### QA-SEC-001 - Sensitive Workspace Content Rejected
|
|
275
|
+
|
|
276
|
+
- **Given**: A message or evidence artifact contains a credential, secret token, or prohibited raw personal data.
|
|
277
|
+
- **Then**: Persistence is rejected or redacted according to explicit policy and the task reports the blocked evidence.
|
|
278
|
+
- **Failure**: Sensitive content is written to a commit-ready workspace.
|
|
279
|
+
|
|
280
|
+
## 11. Pilot And Release Scenarios
|
|
281
|
+
|
|
282
|
+
### QA-PILOT-001 - Five-Task Warning Calibration
|
|
283
|
+
|
|
284
|
+
- **Given**: Five representative real code/deploy tasks.
|
|
285
|
+
- **Then**: Each records selected topology, activated/skipped roles, missing fields, false positives, retries, rework, user overrides, runtime gaps, and escaped findings.
|
|
286
|
+
- **Exit**: User reviews the pilot and explicitly approves or rejects blocking mode.
|
|
287
|
+
|
|
288
|
+
### QA-RELEASE-001 - Package And Documentation Agreement
|
|
289
|
+
|
|
290
|
+
- **Given**: A proposed npm release.
|
|
291
|
+
- **Then**: Tests, package dry run, installed adapters, schema files, README, USAGE, version, changelog, rollback, and migration guidance agree.
|
|
292
|
+
- **Failure**: Documentation claims behavior absent from the packaged files.
|
|
293
|
+
|
|
294
|
+
## 12. Design Document Verification Checklist
|
|
295
|
+
|
|
296
|
+
- [ ] All Markdown files use required Created and Last Updated metadata.
|
|
297
|
+
- [ ] All document filenames use two-digit numbering and uppercase snake case where applicable.
|
|
298
|
+
- [ ] Related Documents use valid relative links with relationship descriptions.
|
|
299
|
+
- [ ] Architecture embeds the data-flow SVG from the required assets directory.
|
|
300
|
+
- [ ] Every P0 FR has architecture coverage and a QA scenario.
|
|
301
|
+
- [ ] Backlog code/deploy tasks contain all Related fields and pending Receipt contracts.
|
|
302
|
+
- [ ] UI and HTML Preview fields use reasoned N/A rather than blank values.
|
|
303
|
+
- [ ] Decision Log records alternatives and reconsideration conditions.
|
|
304
|
+
- [ ] No implementation or release is represented as approved by these documents.
|
|
305
|
+
- [ ] External reference concepts are independently expressed and not copied verbatim.
|
|
306
|
+
|
|
307
|
+
## 13. Related Documents
|
|
308
|
+
|
|
309
|
+
- **Concept_Design**: [Agent Harness Requirements Analysis](../01_Concept_Design/01_AGENT_HARNESS_REQUIREMENTS_ANALYSIS.md) - requirement IDs and approved scope
|
|
310
|
+
- **Technical_Specs**: [Agent Harness Architecture](../03_Technical_Specs/01_AGENT_HARNESS_ARCHITECTURE.md) - roles, states, messages, errors, artifacts, and adapters under test
|
|
311
|
+
- **Logic_Progress**: [Backlog](../04_Logic_Progress/00_BACKLOG.md) - task-level acceptance and Receipt status
|
|
312
|
+
- **Logic_Progress**: [Execution Plan](../04_Logic_Progress/01_EXECUTION_PLAN.md) - staged implementation, pilot, and release exits
|
|
313
|
+
- **Logic_Progress**: [Decision Log](../04_Logic_Progress/03_DECISION_LOG.md) - approved design decisions verified by these scenarios
|
|
@@ -0,0 +1,100 @@
|
|
|
1
|
+
# Agent Harness Design Review
|
|
2
|
+
> Created: 2026-07-17 01:15
|
|
3
|
+
> Last Updated: 2026-07-17 01:22
|
|
4
|
+
|
|
5
|
+
## 1. Review Scope
|
|
6
|
+
|
|
7
|
+
This report records structural and traceability verification of the enhanced Agent Harness design baseline. It does not verify future CLI, schema, adapter, orchestration, release, or runtime behavior.
|
|
8
|
+
|
|
9
|
+
Reviewed artifacts:
|
|
10
|
+
|
|
11
|
+
- Requirements Analysis
|
|
12
|
+
- Architecture and data-flow SVG
|
|
13
|
+
- Backlog
|
|
14
|
+
- Execution Plan
|
|
15
|
+
- Decision Log
|
|
16
|
+
- Test Scenarios
|
|
17
|
+
|
|
18
|
+
## 2. Verification Results
|
|
19
|
+
|
|
20
|
+
| Check | Result | Evidence |
|
|
21
|
+
|---|---|---|
|
|
22
|
+
| Markdown metadata | Pass | Seven design Markdown files contain Created and Last Updated metadata |
|
|
23
|
+
| File naming | Pass | Markdown filenames use two-digit uppercase snake case |
|
|
24
|
+
| Relative document links | Pass | All local Markdown links resolve to existing files |
|
|
25
|
+
| SVG validity | Pass | `xmllint --noout` accepts the data-flow SVG |
|
|
26
|
+
| SVG visual rendering | Pass | A full 1600x900 Chrome render has no clipping or text/connector overlap after correction |
|
|
27
|
+
| Backlog Context Lock shape | Pass for current docs task | All Related fields are present; UI and HTML Preview use reasoned N/A |
|
|
28
|
+
| Requirement traceability | Pass | 15 FR and 10 NFR entries have QA trace rows |
|
|
29
|
+
| Scenario references | Pass | 35 referenced QA scenario IDs have corresponding definitions |
|
|
30
|
+
| Existing package tests | Pass | `npm test` reports `harness checks ok` |
|
|
31
|
+
| Whitespace / patch integrity | Pass | `git diff --check` returns no error |
|
|
32
|
+
| User design review | Pending | The user has not yet reviewed the completed artifacts |
|
|
33
|
+
|
|
34
|
+
## 3. Commands And Results
|
|
35
|
+
|
|
36
|
+
| Command / Method | Result | Summary |
|
|
37
|
+
|---|---|---|
|
|
38
|
+
| `git diff --check` | Pass | No whitespace errors |
|
|
39
|
+
| Metadata and naming shell checks | Pass | Required metadata and filename conventions found |
|
|
40
|
+
| Relative-link Node check | Pass | Six Markdown files checked before this report was added; all local targets resolved |
|
|
41
|
+
| Requirement trace Node check | Pass | 25 unique FR/NFR IDs have QA trace rows |
|
|
42
|
+
| Scenario-reference Node check | Pass | 35 scenario definitions cover every referenced scenario ID |
|
|
43
|
+
| `xmllint --noout docs/03_Technical_Specs/assets/01_agent_harness_data_flow.svg` | Pass | SVG is well-formed XML |
|
|
44
|
+
| Chrome headless 1600x900 SVG render and visual inspection | Pass | Full diagram framing, labels, and connectors are visible without overlap |
|
|
45
|
+
| `node bin/cli.js preflight TASK-HARNESS-001 --strict --backlog docs/04_Logic_Progress/00_BACKLOG.md` | Advisory Pass | Docs work type is correctly treated as advisory |
|
|
46
|
+
| `npm test` | Pass | Existing harness test suite remains green |
|
|
47
|
+
|
|
48
|
+
## 4. Findings And Corrections
|
|
49
|
+
|
|
50
|
+
### DR-001 - Missing Traceability Scenario
|
|
51
|
+
|
|
52
|
+
- **Initial Finding**: The traceability matrix referenced `QA-TRACE-001`, but no matching scenario section existed.
|
|
53
|
+
- **Correction**: Added the end-to-end requirement evidence scenario.
|
|
54
|
+
- **Status**: Resolved and rechecked.
|
|
55
|
+
|
|
56
|
+
### DR-002 - Non-Functional Requirements Were Not Explicitly Mapped
|
|
57
|
+
|
|
58
|
+
- **Initial Finding**: NFR concerns appeared across scenarios, but no NFR traceability table proved complete coverage.
|
|
59
|
+
- **Correction**: Added a 10-row NFR matrix and dedicated cost and external-reference originality scenarios.
|
|
60
|
+
- **Status**: Resolved and rechecked.
|
|
61
|
+
|
|
62
|
+
### DR-003 - Data-Flow Connector Overlapped A Role Boundary
|
|
63
|
+
|
|
64
|
+
- **Initial Finding**: The Coordinator-to-document connector crossed the Release Guardian boundary in the first full SVG render.
|
|
65
|
+
- **Correction**: Shifted the Release Guardian node and adjusted adjacent connector endpoints without changing the architecture.
|
|
66
|
+
- **Status**: Resolved in a second 1600x900 render and visually rechecked.
|
|
67
|
+
|
|
68
|
+
## 5. Global Rubric Validation
|
|
69
|
+
|
|
70
|
+
| Criterion | Result | Evidence |
|
|
71
|
+
|---|---|---|
|
|
72
|
+
| Functionality | Pass at design stage | Workflow contracts and failure paths are specified and structurally verified |
|
|
73
|
+
| Potential Impact | Pass at design stage | Design addresses context omission, improvised delegation, agent conflict, and unsupported completion |
|
|
74
|
+
| Novelty | Pass at design stage | Team architecture is combined with machine-enforced document and independent-verification evidence |
|
|
75
|
+
| UX | Pass at design stage | Minimal topology and conditional specialists avoid mandatory full-team overhead |
|
|
76
|
+
| Open-source | Pass at design stage | Runtime-neutral contracts use standard formats and independently authored text |
|
|
77
|
+
| Business Plan | Pass at design stage | More reliable installed behavior can reduce support burden and improve package trust |
|
|
78
|
+
|
|
79
|
+
## 6. Residual Risks
|
|
80
|
+
|
|
81
|
+
- Runtime capability details must be reverified against current official Claude Code and Codex behavior before adapter implementation.
|
|
82
|
+
- Structured schemas and legal transitions are design-only until automated validators exist.
|
|
83
|
+
- Read-only permissions and exclusive write ownership require runtime and fixture evidence.
|
|
84
|
+
- Cost and agent-count benefits require the five-task warning pilot.
|
|
85
|
+
- The user must review this baseline before Phase B code work can be proposed.
|
|
86
|
+
|
|
87
|
+
## 7. Review Conclusion
|
|
88
|
+
|
|
89
|
+
**Status: Pass for structural design review.**
|
|
90
|
+
|
|
91
|
+
The design baseline is internally linked, traceable, and consistent with the approved direction. It is not an implementation approval and does not authorize code changes, blocking mode, merge, publish, or deployment.
|
|
92
|
+
|
|
93
|
+
## 8. Related Documents
|
|
94
|
+
|
|
95
|
+
- **Concept_Design**: [Agent Harness Requirements Analysis](../01_Concept_Design/01_AGENT_HARNESS_REQUIREMENTS_ANALYSIS.md) - reviewed requirement source
|
|
96
|
+
- **Technical_Specs**: [Agent Harness Architecture](../03_Technical_Specs/01_AGENT_HARNESS_ARCHITECTURE.md) - reviewed technical contract
|
|
97
|
+
- **Logic_Progress**: [Backlog](../04_Logic_Progress/00_BACKLOG.md) - current task and verification summary
|
|
98
|
+
- **Logic_Progress**: [Execution Plan](../04_Logic_Progress/01_EXECUTION_PLAN.md) - future unapproved delivery phases
|
|
99
|
+
- **Logic_Progress**: [Decision Log](../04_Logic_Progress/03_DECISION_LOG.md) - reviewed approved decisions
|
|
100
|
+
- **QA_Validation**: [Agent Harness Test Scenarios](./01_TEST_SCENARIOS.md) - reviewed scenario and traceability catalog
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
# Agent Harness Contract Implementation QA
|
|
2
|
+
> Created: 2026-07-17 02:39
|
|
3
|
+
> Last Updated: 2026-07-17 02:44
|
|
4
|
+
|
|
5
|
+
## 1. Scope
|
|
6
|
+
|
|
7
|
+
This report verifies `TASK-HARNESS-002`: the opt-in v1 manifest, message, and event contracts; semantic validation; deterministic CLI exits; existing Receipt compatibility; package inclusion; and documentation synchronization.
|
|
8
|
+
|
|
9
|
+
It does not verify specialist persona adapters, runtime orchestration, persistent recovery, the five-task warning pilot, blocking rollout, merge, publish, or deployment.
|
|
10
|
+
|
|
11
|
+
## 2. Flow Status
|
|
12
|
+
|
|
13
|
+
```text
|
|
14
|
+
[Flow]
|
|
15
|
+
Current: Phase 5 - Quality Verification
|
|
16
|
+
Gate: Quality + Minimal Implementation + Independent Agent Harness Verification - PASS
|
|
17
|
+
Completed: Approved requirements, Context Receipt, implementation, local checks, independent QA
|
|
18
|
+
Next: Phase 6 - Commit and handoff
|
|
19
|
+
Required confirmation: Separate approval for TASK-HARNESS-003, PR, merge, publish, or deploy
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
## 3. Verification Results
|
|
23
|
+
|
|
24
|
+
| Area | Result | Evidence |
|
|
25
|
+
|---|:---:|---|
|
|
26
|
+
| Versioned schema | Pass | Draft 2020-12 schema contains disjoint manifest, message, and event contracts under top-level `oneOf` |
|
|
27
|
+
| Structured parsing | Pass | JSON and JSONL parse errors are separated from contract failures |
|
|
28
|
+
| State transitions | Pass | Legal, illegal, blocked-resume, rework, cancellation, sequence, attempt, and final-state fixtures |
|
|
29
|
+
| Role and ownership | Pass | Core activation, skipped reason, read-only ownership, canonical paths, recursive overlap, and alias fixtures |
|
|
30
|
+
| Message authority | Pass | Direct peer traffic is limited to `STATUS` / `QUESTION` with `INFO` / `PENDING`; Coordinator routing is enforced |
|
|
31
|
+
| Evidence gates | Pass | Required Receipt and event evidence blocks incomplete `COMPLETE` transitions |
|
|
32
|
+
| Input safety | Pass | Path traversal, URL, backslash, wildcard, non-string, null, prototype-named, and invalid timestamp fixtures |
|
|
33
|
+
| Exit behavior | Pass | Warning contract failure `0`; blocking contract failure `1`; operational parse/file/input error `2` |
|
|
34
|
+
| Compatibility | Pass | Existing `preflight`, `verify`, Receipt fixture text, and custom Claude agent preservation remain valid |
|
|
35
|
+
| Dependency scope | Pass | No runtime or development dependency was added |
|
|
36
|
+
| Independent QA | Pass | Final read-only QA Inspector reported no P0, P1, or P2 findings |
|
|
37
|
+
|
|
38
|
+
## 4. Commands And Results
|
|
39
|
+
|
|
40
|
+
| Command / Method | Result | Summary |
|
|
41
|
+
|---|:---:|---|
|
|
42
|
+
| `npm test` | Pass | `harness checks ok`; `harness artifact checks ok` |
|
|
43
|
+
| `node --check` on changed/new JavaScript | Pass | CLI, validator, runner, and fixture syntax valid |
|
|
44
|
+
| `node bin/cli.js preflight TASK-HARNESS-002 --strict` | Pass | Context and related-document evidence accepted |
|
|
45
|
+
| Node 16 `bin/test.js` | Pass | Both suites pass on an older ARM-compatible runtime |
|
|
46
|
+
| Acorn `--ecma2020` parse | Pass | New validator uses Node 14-compatible syntax |
|
|
47
|
+
| Ajv Draft 2020 compile | Pass | External validator compiles the top-level v1 schema |
|
|
48
|
+
| Ajv validation of `{}` | Expected reject | Top-level `oneOf` rejects an unconstrained artifact |
|
|
49
|
+
| `npm pack --dry-run --json` | Pass | 454 files; validator, tests, schema, contract, and architecture included |
|
|
50
|
+
| Document metadata and relative links | Pass | All current `docs/**/*.md` files validate |
|
|
51
|
+
| Requirement and scenario trace | Pass | 25 requirements and 36 scenario IDs resolve |
|
|
52
|
+
| Secret-pattern scan | Pass | No credential pattern found in changed implementation and documentation |
|
|
53
|
+
| `git diff --check` | Pass | No whitespace or patch-integrity errors |
|
|
54
|
+
|
|
55
|
+
## 5. Independent Findings And Corrections
|
|
56
|
+
|
|
57
|
+
Independent read-only QA runs found and the Implementer corrected the following before the final PASS:
|
|
58
|
+
|
|
59
|
+
1. Canonicalized exclusive ownership paths and rejected `./`, duplicate separators, parent traversal, and ambiguous recursive aliases.
|
|
60
|
+
2. Rejected direct peer PASS, FAIL, BLOCKED, decision, rework, and completion status.
|
|
61
|
+
3. Added top-level `oneOf` so the schema is useful to a standard Draft 2020-12 validator.
|
|
62
|
+
4. Kept parseable invalid values as contract failures instead of internal operational exceptions.
|
|
63
|
+
5. Made v1 attempt increments legal only for `REWORK -> IMPLEMENTING`.
|
|
64
|
+
6. Replaced `Date.parse()` normalization with a schema-aligned Solmate canonical timestamp profile.
|
|
65
|
+
7. Rejected inherited `constructor` and `__proto__` names under `additionalProperties: false`.
|
|
66
|
+
8. Corrected the architecture description so `events.jsonl` contains state events and messages remain attempt artifacts.
|
|
67
|
+
|
|
68
|
+
After each correction a new read-only QA Inspector reviewed the updated files. The final run returned **No findings** and **Verification Receipt Status: PASS** without editing files.
|
|
69
|
+
|
|
70
|
+
## 6. Minimal Implementation Review
|
|
71
|
+
|
|
72
|
+
- **YAGNI**: No persona runtime, orchestration server, database, queue, recovery engine, rollout automation, or release behavior was added.
|
|
73
|
+
- **KISS**: One canonical JSON Schema file and one Node standard-library validator module implement the approved contract.
|
|
74
|
+
- **DRY**: State, role, evidence, and schema definitions remain centralized; CLI and runtime adapters reference the contract.
|
|
75
|
+
- **Safety**: Validation, independent QA, deterministic failures, path safety, evidence preservation, and compatibility checks were retained.
|
|
76
|
+
|
|
77
|
+
## 7. Unrun Checks And Residual Risks
|
|
78
|
+
|
|
79
|
+
- Actual Node 14 execution was unavailable because npm has no matching `node-bin-darwin-arm64@14.21.3`; Node 16 execution and ECMAScript 2020 static parsing passed.
|
|
80
|
+
- Ajv CLI compiled the schema and exercised top-level rejection without `ajv-formats`; calendar and canonical timestamp semantics are additionally covered by the package validator tests.
|
|
81
|
+
- Claude/Codex persona parity, runtime permission enforcement, orchestration recovery, warning-pilot metrics, and release behavior belong to later tasks.
|
|
82
|
+
- Package version remains `2.0.12`; no version bump, PR, merge, publish, or deploy was authorized in this task.
|
|
83
|
+
|
|
84
|
+
## 8. Conclusion
|
|
85
|
+
|
|
86
|
+
**Status: PASS for TASK-HARNESS-002.**
|
|
87
|
+
|
|
88
|
+
The v1 structured contract and machine validator satisfy the approved acceptance criteria while preserving current Receipt behavior. Later phases remain separately gated.
|
|
89
|
+
|
|
90
|
+
## 9. Related Documents
|
|
91
|
+
|
|
92
|
+
- **Concept_Design**: [Agent Harness Requirements Analysis](../01_Concept_Design/01_AGENT_HARNESS_REQUIREMENTS_ANALYSIS.md) - approved requirements and compatibility constraints
|
|
93
|
+
- **Technical_Specs**: [Agent Harness Architecture](../03_Technical_Specs/01_AGENT_HARNESS_ARCHITECTURE.md) - implemented schema and semantic contract
|
|
94
|
+
- **Logic_Progress**: [Backlog](../04_Logic_Progress/00_BACKLOG.md) - completed task Receipts
|
|
95
|
+
- **Logic_Progress**: [Execution Plan](../04_Logic_Progress/01_EXECUTION_PLAN.md) - Phase B completion and later phases
|
|
96
|
+
- **QA_Validation**: [Agent Harness Test Scenarios](./01_TEST_SCENARIOS.md) - traceability and planned scenario catalog
|
|
@@ -33,6 +33,8 @@ The AI MUST check for and follow these documents if they exist in the repository
|
|
|
33
33
|
Apply the "Subagent Strategy" to keep the main context clean:
|
|
34
34
|
- **Delegation**: Offload research/analysis to Browser/Thinking tools.
|
|
35
35
|
- **Aggregation**: Return only summarized, high-value insights to the main chat.
|
|
36
|
+
- **Canonical Contract**: For code and deploy tasks, use `rules-workflow/resources/agent-harness-contract.md` for Coordinator, Context, Implementation, Verification roles and handoffs.
|
|
37
|
+
- **Independence**: Context and Verification agents are read-only. Verification findings return to the Implementation Agent instead of being fixed by the verifier.
|
|
36
38
|
|
|
37
39
|
### 3.2. 자가개선 루프 (Self-Improvement)
|
|
38
40
|
- Maintain `tasks/lessons.md` to capture patterns and avoid repeating mistakes.
|
package/package.json
CHANGED
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "solmate-skills",
|
|
3
|
-
"version": "2.0.
|
|
4
|
-
"description": "AI workflow skills for
|
|
3
|
+
"version": "2.0.13",
|
|
4
|
+
"description": "AI agent harness and workflow skills for context-aware planning, implementation, verification, docs, and QA",
|
|
5
5
|
"main": "bin/cli.js",
|
|
6
6
|
"bin": {
|
|
7
7
|
"solmate-skills": "bin/cli.js"
|
|
8
8
|
},
|
|
9
9
|
"scripts": {
|
|
10
|
-
"test": "
|
|
10
|
+
"test": "node bin/test.js"
|
|
11
11
|
},
|
|
12
12
|
"keywords": [
|
|
13
13
|
"ai-skills",
|
|
@@ -22,7 +22,9 @@
|
|
|
22
22
|
"qa",
|
|
23
23
|
"solmate",
|
|
24
24
|
"skills",
|
|
25
|
-
"automation"
|
|
25
|
+
"automation",
|
|
26
|
+
"agent-harness",
|
|
27
|
+
"multi-agent"
|
|
26
28
|
],
|
|
27
29
|
"author": "",
|
|
28
30
|
"license": "ISC",
|
package/rules-docs/SKILL.md
CHANGED
|
@@ -89,12 +89,26 @@ docs/02_UI_Screens/previews/
|
|
|
89
89
|
- `Related HTML Preview`
|
|
90
90
|
- `Related Technical Docs`
|
|
91
91
|
- `Related QA Docs`
|
|
92
|
+
- `Work Type`
|
|
92
93
|
- `Implementation Preconditions`
|
|
94
|
+
- `Context Receipt`
|
|
95
|
+
- `Change Receipt`
|
|
96
|
+
- `Verification Receipt`
|
|
93
97
|
- `Acceptance Criteria`
|
|
94
98
|
- `Document Sync Check`
|
|
95
99
|
|
|
96
100
|
각 Related 필드는 상대 경로 링크와 관계 설명을 포함해야 한다. 관련 문서나 HTML Preview가 없으면 빈칸으로 두지 않고 `N/A - 사유`를 명시한다. 단, 구현 판단에 필요한 Concept, UI, HTML Preview, Technical Spec, QA 기준이 아직 문서화되지 않은 경우에는 백로그만으로 구현을 시작할 수 없으며, 먼저 문서 작성 또는 보완 여부를 사용자에게 확인한다.
|
|
97
101
|
|
|
102
|
+
`Work Type`은 `code`, `deploy`, `docs`, `prototype` 중 하나를 사용한다. `code`와 `deploy` 작업은 다음 Agent Harness 조건을 추가로 지킨다.
|
|
103
|
+
|
|
104
|
+
- Context Agent가 모든 Related 링크를 읽고 `Context Receipt`에 기록한다.
|
|
105
|
+
- warning 기간에는 Context Receipt 발견 사항과 사용자 확인을 기록한다. blocking 전환 후에는 PASS Context Receipt 없이는 구현을 시작하지 않는다.
|
|
106
|
+
- Implementation Agent는 `Change Receipt`를 남기지만 이를 독립 검증으로 간주하지 않는다.
|
|
107
|
+
- 읽기 전용 Verification Agent가 `Verification Receipt`를 작성한다.
|
|
108
|
+
- warning 기간에는 Verification Receipt 발견 사항과 사용자 확인을 기록한다. blocking 전환 후에는 PASS Verification Receipt와 QA 문서 또는 PR 상세 근거 링크 없이는 Done 처리하지 않는다.
|
|
109
|
+
|
|
110
|
+
Receipt 형식과 런타임별 역할은 `rules-workflow/resources/agent-harness-contract.md`를 정본으로 따른다.
|
|
111
|
+
|
|
98
112
|
### 메타데이터 (필수)
|
|
99
113
|
|
|
100
114
|
모든 문서 최상단:
|
|
@@ -133,7 +147,7 @@ docs/02_UI_Screens/previews/
|
|
|
133
147
|
|
|
134
148
|
**Logic_Progress**: Concept_Design + UI_Screens + Technical_Specs (DB, API) + QA_Validation
|
|
135
149
|
|
|
136
|
-
**Backlog Items**: 각 항목 단위로 Concept_Design + UI_Screens + HTML Preview + Technical_Specs + QA_Validation 링크를 포함한다.
|
|
150
|
+
**Backlog Items**: 각 항목 단위로 Concept_Design + UI_Screens + HTML Preview + Technical_Specs + QA_Validation 링크를 포함한다. `code`와 `deploy` 작업은 구현자가 아니라 읽기 전용 Context Agent가 먼저 링크된 문서를 확인하고 Context Receipt를 제출해야 한다.
|
|
137
151
|
|
|
138
152
|
**QA_Validation**: 모든 상위 레이어 참조
|
|
139
153
|
|