orchestr8 2.5.0 → 2.6.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (54) hide show
  1. package/.blueprint/agents/AGENT_BA_CASS.md +42 -19
  2. package/.blueprint/agents/AGENT_DEVELOPER_CODEY.md +42 -38
  3. package/.blueprint/agents/AGENT_SPECIFICATION_ALEX.md +45 -0
  4. package/.blueprint/agents/AGENT_TESTER_NIGEL.md +42 -21
  5. package/.blueprint/features/feature_adaptive-retry/FEATURE_SPEC.md +239 -0
  6. package/.blueprint/features/feature_adaptive-retry/IMPLEMENTATION_PLAN.md +48 -0
  7. package/.blueprint/features/feature_adaptive-retry/story-prompt-modification.md +85 -0
  8. package/.blueprint/features/feature_adaptive-retry/story-retry-config.md +89 -0
  9. package/.blueprint/features/feature_adaptive-retry/story-should-retry.md +98 -0
  10. package/.blueprint/features/feature_adaptive-retry/story-strategy-recommendation.md +85 -0
  11. package/.blueprint/features/feature_agent-guardrails/FEATURE_SPEC.md +328 -0
  12. package/.blueprint/features/feature_agent-guardrails/IMPLEMENTATION_PLAN.md +90 -0
  13. package/.blueprint/features/feature_agent-guardrails/story-citation-requirements.md +50 -0
  14. package/.blueprint/features/feature_agent-guardrails/story-confidentiality.md +50 -0
  15. package/.blueprint/features/feature_agent-guardrails/story-escalation-protocol.md +55 -0
  16. package/.blueprint/features/feature_agent-guardrails/story-source-restrictions.md +50 -0
  17. package/.blueprint/features/feature_feedback-loop/FEATURE_SPEC.md +347 -0
  18. package/.blueprint/features/feature_feedback-loop/IMPLEMENTATION_PLAN.md +71 -0
  19. package/.blueprint/features/feature_feedback-loop/story-feedback-collection.md +63 -0
  20. package/.blueprint/features/feature_feedback-loop/story-feedback-config.md +61 -0
  21. package/.blueprint/features/feature_feedback-loop/story-feedback-insights.md +63 -0
  22. package/.blueprint/features/feature_feedback-loop/story-quality-gates.md +57 -0
  23. package/.blueprint/features/feature_pipeline-history/FEATURE_SPEC.md +239 -0
  24. package/.blueprint/features/feature_pipeline-history/IMPLEMENTATION_PLAN.md +71 -0
  25. package/.blueprint/features/feature_pipeline-history/story-clear-history.md +73 -0
  26. package/.blueprint/features/feature_pipeline-history/story-display-history.md +75 -0
  27. package/.blueprint/features/feature_pipeline-history/story-record-execution.md +76 -0
  28. package/.blueprint/features/feature_pipeline-history/story-show-statistics.md +85 -0
  29. package/.blueprint/features/feature_pipeline-insights/FEATURE_SPEC.md +288 -0
  30. package/.blueprint/features/feature_pipeline-insights/IMPLEMENTATION_PLAN.md +65 -0
  31. package/.blueprint/features/feature_pipeline-insights/story-anomaly-detection.md +71 -0
  32. package/.blueprint/features/feature_pipeline-insights/story-bottleneck-analysis.md +75 -0
  33. package/.blueprint/features/feature_pipeline-insights/story-failure-patterns.md +75 -0
  34. package/.blueprint/features/feature_pipeline-insights/story-json-output.md +75 -0
  35. package/.blueprint/features/feature_pipeline-insights/story-trend-analysis.md +78 -0
  36. package/.blueprint/features/feature_validate-command/FEATURE_SPEC.md +209 -0
  37. package/.blueprint/features/feature_validate-command/IMPLEMENTATION_PLAN.md +59 -0
  38. package/.blueprint/features/feature_validate-command/story-failure-output.md +61 -0
  39. package/.blueprint/features/feature_validate-command/story-node-version-check.md +52 -0
  40. package/.blueprint/features/feature_validate-command/story-run-validation.md +59 -0
  41. package/.blueprint/features/feature_validate-command/story-success-output.md +50 -0
  42. package/.blueprint/system_specification/SYSTEM_SPEC.md +248 -0
  43. package/README.md +182 -38
  44. package/SKILL.md +333 -23
  45. package/bin/cli.js +128 -20
  46. package/package.json +2 -2
  47. package/src/feedback.js +171 -0
  48. package/src/history.js +306 -0
  49. package/src/index.js +57 -2
  50. package/src/init.js +2 -6
  51. package/src/insights.js +504 -0
  52. package/src/retry.js +274 -0
  53. package/src/validate.js +172 -0
  54. package/src/skills.js +0 -93
@@ -0,0 +1,50 @@
1
+ # Story — Citation Requirements
2
+
3
+ ## User story
4
+
5
+ As a user of orchestr8, I want all agent assertions to cite their source files so that I can trace claims back to authoritative inputs and verify correctness.
6
+
7
+ ---
8
+
9
+ ## Context / scope
10
+
11
+ - Applies to all agents: Alex, Cass, Nigel, Codey
12
+ - Per FEATURE_SPEC.md section 6, Rule 2: "All assertions about requirements, behaviour, or domain knowledge must cite their source"
13
+ - Per SYSTEM_SPEC.md section 8: "Traceability" is a cross-cutting concern
14
+
15
+ ---
16
+
17
+ ## Acceptance criteria
18
+
19
+ **AC-1 — Standard citation format used**
20
+ - Given an agent makes an assertion about requirements or behaviour,
21
+ - When the assertion is included in output,
22
+ - Then the assertion includes a citation in one of the standard formats: "Per [filename]: [claim]" or "[filename:section] states...".
23
+
24
+ **AC-2 — Section-level citations where feasible**
25
+ - Given an agent cites a source file with multiple sections,
26
+ - When the assertion relates to a specific section,
27
+ - Then the citation includes the section reference (e.g., "FEATURE_SPEC.md section 6" or "story-login.md:AC-3").
28
+
29
+ **AC-3 — Assumptions distinguished from cited facts**
30
+ - Given an agent includes both cited facts and assumptions in output,
31
+ - When the output is reviewed,
32
+ - Then assumptions are clearly labelled (e.g., "ASSUMPTION:" prefix) and distinguishable from cited assertions.
33
+
34
+ **AC-4 — Domain knowledge grounded in business context**
35
+ - Given an agent makes assertions about business or domain concepts,
36
+ - When those concepts are defined in `.business_context/` files,
37
+ - Then the assertion cites the specific business context file (e.g., ".business_context/glossary.md defines...").
38
+
39
+ **AC-5 — Traceable chain maintained**
40
+ - Given a downstream artifact (story, test, code) references upstream artifacts,
41
+ - When the reference is made,
42
+ - Then it includes sufficient citation to trace back to the source (story cites feature spec, test cites story, implementation cites test).
43
+
44
+ ---
45
+
46
+ ## Out of scope
47
+
48
+ - Automated citation validation tooling
49
+ - Link checking or broken reference detection
50
+ - Citation index or cross-reference reports
@@ -0,0 +1,50 @@
1
+ # Story — Confidentiality
2
+
3
+ ## User story
4
+
5
+ As a user of orchestr8, I want agents to treat business context and project content as confidential so that sensitive information is not leaked in outputs or exposed to external services.
6
+
7
+ ---
8
+
9
+ ## Context / scope
10
+
11
+ - Applies to all agents: Alex, Cass, Nigel, Codey
12
+ - Per FEATURE_SPEC.md section 6, Rule 3: "Agents must treat `.business_context/` content as confidential and prevent data leakage"
13
+ - Per FEATURE_SPEC.md section 9, Open Question 3: "treat all project content as confidential by default"
14
+
15
+ ---
16
+
17
+ ## Acceptance criteria
18
+
19
+ **AC-1 — Business context not exposed verbatim**
20
+ - Given an agent processes content from `.business_context/` directory,
21
+ - When the agent produces output artifacts,
22
+ - Then the output does not reproduce confidential details verbatim; instead it uses generic descriptions or summarises appropriately.
23
+
24
+ **AC-2 — External entities not referenced by name**
25
+ - Given an agent is aware of external projects, companies, or implementations from business context,
26
+ - When producing output,
27
+ - Then the agent does not reference these external entities by name in specifications, stories, or code comments.
28
+
29
+ **AC-3 — No external service exposure**
30
+ - Given an agent is processing project content,
31
+ - When external services would expose project data (e.g., external APIs, logging services),
32
+ - Then the agent does not use such services and keeps all processing local to the pipeline.
33
+
34
+ **AC-4 — Self-contained outputs**
35
+ - Given an agent produces an output artifact,
36
+ - When the artifact is reviewed in isolation,
37
+ - Then it does not require access to confidential business context to be understood; it is self-contained.
38
+
39
+ **AC-5 — Confidentiality conflict escalation**
40
+ - Given an output would require exposing confidential details,
41
+ - When no acceptable redaction or generalisation is possible,
42
+ - Then the agent flags the conflict and escalates to the user for guidance before proceeding.
43
+
44
+ ---
45
+
46
+ ## Out of scope
47
+
48
+ - Encryption or access control mechanisms for business context files
49
+ - Audit logging of confidential data access
50
+ - Data loss prevention (DLP) tooling integration
@@ -0,0 +1,55 @@
1
+ # Story — Escalation Protocol
2
+
3
+ ## User story
4
+
5
+ As a user of orchestr8, I want agents to escalate to me when they encounter ambiguity or missing information so that I can provide guidance rather than agents guessing incorrectly.
6
+
7
+ ---
8
+
9
+ ## Context / scope
10
+
11
+ - Applies to all agents: Alex, Cass, Nigel, Codey
12
+ - Per FEATURE_SPEC.md section 6, Rule 5: "Agents must escalate to the user under defined conditions rather than proceeding with guesses"
13
+ - Per SYSTEM_SPEC.md section 7: "No silent changes: Agents flag deviations; do not silently alter specifications"
14
+
15
+ ---
16
+
17
+ ## Acceptance criteria
18
+
19
+ **AC-1 — Escalation on missing critical information**
20
+ - Given information required for a task is not present in provided inputs,
21
+ - When the information cannot be safely assumed (domain-specific, policy-related, or high-impact),
22
+ - Then the agent escalates to the user with a clear question explaining what information is needed and why.
23
+
24
+ **AC-2 — Escalation on significant ambiguity**
25
+ - Given inputs are ambiguous in a way that significantly affects output,
26
+ - When multiple interpretations are possible,
27
+ - Then the agent lists the possible interpretations and asks the user to clarify before proceeding.
28
+
29
+ **AC-3 — Escalation on source conflict**
30
+ - Given different input sources contain conflicting information,
31
+ - When the conflict cannot be resolved by recency or specificity rules,
32
+ - Then the agent flags the conflict, cites both sources, and asks the user to resolve.
33
+
34
+ **AC-4 — Escalation on confidentiality concern**
35
+ - Given a task would require violating confidentiality constraints,
36
+ - When no acceptable alternative approach exists,
37
+ - Then the agent explains the concern and requests user guidance.
38
+
39
+ **AC-5 — Explicit assumptions when escalation not warranted**
40
+ - Given information is missing but a safe default exists,
41
+ - When the assumption is low-risk and reversible,
42
+ - Then the agent may proceed with an explicit assumption labelled as "ASSUMPTION:" or "NOTE: Assuming..." rather than escalating.
43
+
44
+ **AC-6 — Anti-hallucination preference**
45
+ - Given an agent lacks information to answer a question,
46
+ - When asked to produce content,
47
+ - Then the agent states "This information is not available in the provided inputs" rather than inventing an answer.
48
+
49
+ ---
50
+
51
+ ## Out of scope
52
+
53
+ - Automated escalation routing or ticketing
54
+ - Escalation SLAs or response time requirements
55
+ - Multi-user escalation workflows
@@ -0,0 +1,50 @@
1
+ # Story — Source Restrictions
2
+
3
+ ## User story
4
+
5
+ As a user of orchestr8, I want agents to use only information from explicitly provided inputs so that outputs are grounded in authoritative sources and not hallucinated from training data or external content.
6
+
7
+ ---
8
+
9
+ ## Context / scope
10
+
11
+ - Applies to all agents: Alex, Cass, Nigel, Codey
12
+ - This is a behavioural constraint, not a runtime enforcement mechanism
13
+ - Per FEATURE_SPEC.md section 6, Rule 1: "Agents must use ONLY information from explicitly provided inputs"
14
+
15
+ ---
16
+
17
+ ## Acceptance criteria
18
+
19
+ **AC-1 — Allowed sources are used**
20
+ - Given an agent is processing a task,
21
+ - When the agent produces output,
22
+ - Then the output is derived exclusively from allowed sources: system specification, feature specifications, user stories, test artifacts, implementation code, business context, templates, and agent specifications.
23
+
24
+ **AC-2 — Prohibited sources are not used**
25
+ - Given an agent is processing a task,
26
+ - When the agent encounters a gap in provided inputs,
27
+ - Then the agent does not reference social media, forums, blog posts, web searches, external APIs, or external project implementations to fill that gap.
28
+
29
+ **AC-3 — Training data not used for domain facts**
30
+ - Given an agent needs domain-specific information (business rules, project requirements),
31
+ - When that information is not present in provided inputs,
32
+ - Then the agent does not use its training data to invent or assume domain facts; instead it flags the gap explicitly.
33
+
34
+ **AC-4 — External references not introduced**
35
+ - Given an agent is writing output artifacts,
36
+ - When the output references patterns, examples, or implementations,
37
+ - Then those references are to files within the project, not to external companies, products, or open-source projects.
38
+
39
+ **AC-5 — Gap handling triggers assumption or escalation**
40
+ - Given required information is not available in allowed sources,
41
+ - When the agent cannot proceed without this information,
42
+ - Then the agent either declares an explicit assumption (labelled as such) or escalates to the user.
43
+
44
+ ---
45
+
46
+ ## Out of scope
47
+
48
+ - Runtime validation or filtering of agent outputs
49
+ - Technical enforcement mechanisms (content blockers, API restrictions)
50
+ - Monitoring infrastructure for source compliance
@@ -0,0 +1,347 @@
1
+ # Feature Specification — Agent Feedback Loop
2
+
3
+ ## 1. Feature Intent
4
+ **Why this feature exists.**
5
+
6
+ - **Problem being addressed:** The orchestr8 pipeline executes sequentially but lacks intra-stage feedback. Agents cannot assess the quality of upstream artifacts, leading to silent propagation of poor-quality specifications, stories, or tests through the pipeline.
7
+ - **User need:** Developers want visibility into how each agent perceives the quality of inputs from previous stages. When quality is low, the pipeline should pause for human review rather than proceeding with flawed inputs.
8
+ - **System alignment:** Per SYSTEM_SPEC.md:Section 7 (Governing Rules), agents are expected to "flag deviations" and "not silently alter specifications". This feature operationalises that principle by requiring explicit quality assessment at each stage boundary.
9
+
10
+ > This feature introduces a quality feedback mechanism that integrates with existing history, insights, and retry modules to create a closed-loop quality system.
11
+
12
+ ---
13
+
14
+ ## 2. Scope
15
+
16
+ ### In Scope
17
+
18
+ - Feedback collection schema and data structure for agent assessments
19
+ - Feedback capture at each stage boundary (Cass on Alex, Nigel on Cass, Codey on Nigel)
20
+ - Quality gate logic: pause pipeline if feedback rating falls below threshold
21
+ - Configuration management for feedback thresholds (`.claude/feedback-config.json`)
22
+ - Storage of feedback in pipeline history entries
23
+ - Insights extension: correlation analysis between feedback scores and outcomes
24
+ - Retry integration: mapping feedback issues to retry strategies
25
+ - CLI commands for feedback configuration and analysis
26
+
27
+ ### Out of Scope
28
+
29
+ - Feedback from Alex (no prior stage to assess within the pipeline)
30
+ - Automatic remediation based on feedback (human review required)
31
+ - Cross-pipeline feedback aggregation (each run is independent)
32
+ - Natural language feedback parsing (structured schema only)
33
+ - Feedback on auto-commit stage (no agent assessment)
34
+
35
+ ---
36
+
37
+ ## 3. Actors Involved
38
+
39
+ ### Human User
40
+
41
+ - **Can do:** View feedback thresholds; modify threshold configuration; view feedback correlation insights; review and approve paused pipelines
42
+ - **Cannot do:** Directly inject feedback into history; bypass quality gates without explicit action
43
+
44
+ ### Cass (Story Writer Agent)
45
+
46
+ - **Can do:** Provide feedback on Alex's feature specification before writing stories
47
+ - **Feedback target:** Feature specification quality, completeness, clarity
48
+
49
+ ### Nigel (Tester Agent)
50
+
51
+ - **Can do:** Provide feedback on Cass's user stories before writing tests
52
+ - **Feedback target:** Story quality, acceptance criteria testability, scope clarity
53
+
54
+ ### Codey (Developer Agent)
55
+
56
+ - **Can do:** Provide feedback on Nigel's test specification before planning/implementing
57
+ - **Feedback target:** Test coverage, implementation feasibility, specification clarity
58
+
59
+ ### Pipeline Orchestrator (SKILL.md implementation)
60
+
61
+ - **Can do:** Collect feedback from agents; evaluate against thresholds; persist to history; trigger quality gates
62
+ - **Cannot do:** Override human decisions on paused pipelines
63
+
64
+ ### History Module (src/history.js)
65
+
66
+ - **Extended by:** New `feedback` field in stage entries
67
+ - **Maintains:** Backward compatibility with existing entries (no feedback = null)
68
+
69
+ ### Insights Module (src/insights.js)
70
+
71
+ - **Extended by:** New `--feedback` analysis mode
72
+ - **Provides:** Calibration scoring, issue pattern correlation
73
+
74
+ ### Retry Module (src/retry.js)
75
+
76
+ - **Extended by:** Feedback-informed strategy selection
77
+ - **Consumes:** Issue patterns from feedback to recommend targeted strategies
78
+
79
+ ---
80
+
81
+ ## 4. Behaviour Overview
82
+
83
+ ### Happy Path: Feedback Collection and Proceeding
84
+
85
+ 1. Alex completes feature specification
86
+ 2. Orchestrator spawns Cass with explicit instruction to provide feedback on Alex's output
87
+ 3. Cass writes feedback object (rating, confidence, issues, recommendation) to designated output
88
+ 4. Orchestrator reads feedback and evaluates against configured threshold (default: 3.0)
89
+ 5. Rating >= threshold: pipeline proceeds, feedback stored in history
90
+ 6. Cass proceeds to write user stories
91
+ 7. Pattern repeats for Nigel (feedback on Cass) and Codey (feedback on Nigel)
92
+ 8. On completion, all feedback is persisted in history entry
93
+
94
+ ### Alternative: Quality Gate Triggers Pause
95
+
96
+ 1. Agent provides feedback with rating < configured threshold
97
+ 2. Agent's recommendation is "pause" or "revise"
98
+ 3. Orchestrator pauses pipeline before current agent's main work
99
+ 4. User is prompted: "Quality gate triggered. {Agent} rated previous stage {rating}/5. Issues: {issues}. (review/proceed/abort)"
100
+ 5. User can review upstream artifacts, request revision, or proceed anyway
101
+ 6. Decision and feedback are recorded in history
102
+
103
+ ### Alternative: Dynamic Threshold Adjustment
104
+
105
+ 1. User runs `orchestr8 insights --feedback` after sufficient runs
106
+ 2. Insights module calculates agent calibration (how predictive is their feedback of actual outcomes)
107
+ 3. User runs `orchestr8 feedback-config set minRating 3.5` to adjust threshold based on data
108
+ 4. Future runs use updated threshold
109
+
110
+ ### Alternative: Retry with Feedback-Informed Strategy
111
+
112
+ 1. Pipeline fails at a stage (e.g., Codey cannot implement)
113
+ 2. Retry module examines feedback chain for the failed run
114
+ 3. Feedback issues are mapped to strategies:
115
+ - "missing-error-handling" → `add-context`
116
+ - "too-complex" → `simplify-prompt`
117
+ - "too-many-stories" → `reduce-stories`
118
+ 4. Recommended strategy reflects feedback analysis
119
+ 5. User accepts or chooses alternative
120
+
121
+ ---
122
+
123
+ ## 5. State & Lifecycle Interactions
124
+
125
+ ### States Entered
126
+
127
+ - **feedback_pending:** After upstream agent completes, before downstream agent provides feedback
128
+ - **quality_gate_paused:** When feedback triggers quality gate (rating < threshold)
129
+
130
+ ### States Exited
131
+
132
+ - **feedback_pending → in_progress:** When feedback is recorded and threshold is met
133
+ - **quality_gate_paused → in_progress:** When user chooses to proceed
134
+ - **quality_gate_paused → paused:** When user requests review/revision
135
+
136
+ ### States Modified
137
+
138
+ - Pipeline history entries gain `stages[].feedback` field
139
+ - Queue entries may include temporary feedback data during execution
140
+
141
+ ### Lifecycle Classification
142
+
143
+ - **State-creating:** Creates feedback_pending state at each stage boundary
144
+ - **State-constraining:** Quality gates can block progression
145
+ - **State-transitioning:** Moves between feedback states based on ratings
146
+
147
+ ---
148
+
149
+ ## 6. Rules & Decision Logic
150
+
151
+ ### Rule 1: Feedback Schema Validation
152
+
153
+ - **Description:** All feedback must conform to the defined schema
154
+ - **Inputs:** Agent feedback output
155
+ - **Outputs:** Validated feedback object or validation error
156
+ - **Type:** Deterministic
157
+
158
+ ```json
159
+ {
160
+ "about": "alex|cass|nigel",
161
+ "rating": 1-5,
162
+ "confidence": 0.0-1.0,
163
+ "issues": ["issue-code", ...],
164
+ "recommendation": "proceed|pause|revise"
165
+ }
166
+ ```
167
+
168
+ ### Rule 2: Quality Gate Evaluation
169
+
170
+ - **Description:** Compare feedback rating against threshold to determine if pipeline should pause
171
+ - **Inputs:** Feedback rating, configured threshold, recommendation
172
+ - **Outputs:** Boolean (shouldPause)
173
+ - **Type:** Deterministic
174
+
175
+ ```
176
+ shouldPause = (rating < minRatingThreshold) OR (recommendation === "pause")
177
+ ```
178
+
179
+ ### Rule 3: Issue-to-Strategy Mapping
180
+
181
+ - **Description:** Map feedback issue codes to retry strategies
182
+ - **Inputs:** List of issue codes from feedback chain
183
+ - **Outputs:** Prioritised list of recommended strategies
184
+ - **Type:** Deterministic with configurable mappings
185
+
186
+ Default mappings:
187
+ | Issue Code | Strategy |
188
+ |------------|----------|
189
+ | `missing-error-handling` | `add-context` |
190
+ | `unclear-scope` | `simplify-prompt` |
191
+ | `too-complex` | `simplify-prompt` |
192
+ | `too-many-stories` | `reduce-stories` |
193
+ | `untestable-criteria` | `simplify-tests` |
194
+ | `missing-edge-cases` | `add-context` |
195
+
196
+ ### Rule 4: Agent Calibration Calculation
197
+
198
+ - **Description:** Measure correlation between agent feedback and eventual pipeline outcomes
199
+ - **Inputs:** Historical feedback ratings, pipeline outcomes (success/failed)
200
+ - **Outputs:** Calibration score per agent (0.0 = uncorrelated, 1.0 = perfect predictor)
201
+ - **Type:** Deterministic (statistical calculation)
202
+
203
+ ```
204
+ calibration[agent] = correlation(feedback_ratings[agent], outcome_success_binary)
205
+ ```
206
+
207
+ ### Rule 5: Threshold Recommendation
208
+
209
+ - **Description:** Suggest optimal threshold based on historical data
210
+ - **Inputs:** All feedback/outcome pairs, desired false positive/negative balance
211
+ - **Outputs:** Recommended minRating threshold
212
+ - **Type:** Deterministic
213
+
214
+ ---
215
+
216
+ ## 7. Dependencies
217
+
218
+ ### System Components
219
+
220
+ - **src/history.js:** Extended to store feedback in history entries
221
+ - New field: `stages[stage].feedback` containing feedback object
222
+ - Backward compatible: existing entries without feedback are valid
223
+
224
+ - **src/insights.js:** Extended with feedback analysis functions
225
+ - New function: `analyzeFeedbackCorrelation(history)`
226
+ - New CLI flag: `--feedback` for feedback-specific analysis
227
+
228
+ - **src/retry.js:** Extended with feedback-informed strategy selection
229
+ - New function: `mapIssuesToStrategies(issues, config)`
230
+ - Modified: `shouldRetry()` to consider feedback chain
231
+
232
+ - **bin/cli.js:** New command registration
233
+ - `orchestr8 feedback-config` (view)
234
+ - `orchestr8 feedback-config set <key> <value>`
235
+ - `orchestr8 insights --feedback`
236
+
237
+ ### File Dependencies
238
+
239
+ - **`.claude/feedback-config.json`:** Configuration storage
240
+ - **`.claude/pipeline-history.json`:** Extended schema for feedback storage
241
+
242
+ ### Agent Specification Dependencies
243
+
244
+ - Agent prompts (in SKILL.md or agent specs) must include feedback collection instructions
245
+ - Feedback schema must be communicated to agents in their task prompts
246
+
247
+ ---
248
+
249
+ ## 8. Non-Functional Considerations
250
+
251
+ ### Performance
252
+
253
+ - Feedback collection adds one structured output per stage (minimal overhead)
254
+ - Quality gate evaluation is O(1) comparison
255
+ - Calibration calculation is O(n) over history entries
256
+
257
+ ### Resilience
258
+
259
+ - If feedback collection fails, pipeline proceeds with warning (degraded mode)
260
+ - Missing feedback in history is treated as neutral (no quality gate effect)
261
+ - Invalid feedback schema triggers warning but does not block pipeline
262
+
263
+ ### Audit/Logging
264
+
265
+ - All feedback is persisted in history for retrospective analysis
266
+ - Quality gate decisions are logged with timestamp and user action
267
+
268
+ ### Security
269
+
270
+ - Feedback file is gitignored (contains project-specific assessments)
271
+ - No sensitive data expected in feedback (ratings, codes, recommendations only)
272
+
273
+ ---
274
+
275
+ ## 9. Assumptions & Open Questions
276
+
277
+ ### Assumptions
278
+
279
+ - ASSUMPTION: Agents can reliably produce structured feedback in the specified schema
280
+ - ASSUMPTION: Feedback ratings are comparable across agents and runs
281
+ - ASSUMPTION: Issue codes will emerge from practice and can be standardised iteratively
282
+ - ASSUMPTION: Correlation analysis requires 10+ completed runs for meaningful results
283
+
284
+ ### Open Questions
285
+
286
+ - Should feedback influence agent prompts proactively (not just on retry)?
287
+ - How should conflicting feedback (high rating but "pause" recommendation) be handled?
288
+ - Should there be feedback severity levels (blocking vs advisory)?
289
+ - What feedback should Codey provide about Codey-plan (within same agent)?
290
+
291
+ ---
292
+
293
+ ## 10. Impact on System Specification
294
+
295
+ ### Reinforces Existing Assumptions
296
+
297
+ - Per SYSTEM_SPEC.md:Section 7, agents must "flag deviations" - feedback formalises this
298
+ - Per SYSTEM_SPEC.md:Section 8, failure handling already supports pause/review - quality gates extend this
299
+
300
+ ### Stretches Existing Assumptions
301
+
302
+ - History module shifts from pure observability to operational dependency (also noted in adaptive-retry)
303
+ - Agent boundaries are subtly extended: agents now assess peer outputs, not just produce artifacts
304
+ - Pipeline flow gains conditional branches (quality gates) beyond explicit `--pause-after`
305
+
306
+ ### Potential Contradiction
307
+
308
+ The system spec states pipelines are "sequential" (Section 7). Quality gates introduce conditional pauses that may feel like interruptions. However, this is consistent with the existing `--pause-after` mechanism and does not fundamentally alter sequence.
309
+
310
+ **Flagged for consideration:** Should SYSTEM_SPEC.md:Section 6 be updated to explicitly acknowledge quality gates as a pipeline flow modifier?
311
+
312
+ ---
313
+
314
+ ## 11. Handover to BA (Cass)
315
+
316
+ ### Story Themes
317
+
318
+ 1. **Feedback Collection:** Agents provide structured feedback on upstream artifacts
319
+ 2. **Quality Gates:** Pipeline pauses when feedback indicates quality concerns
320
+ 3. **Configuration Management:** User can view and modify feedback thresholds
321
+ 4. **History Integration:** Feedback is stored in pipeline history entries
322
+ 5. **Insights Extension:** Feedback correlation analysis and calibration scoring
323
+ 6. **Retry Integration:** Feedback issues inform retry strategy selection
324
+
325
+ ### Expected Story Boundaries
326
+
327
+ - Feedback schema definition and validation as foundational story
328
+ - Quality gate logic as separate story (depends on schema)
329
+ - CLI configuration commands as separate story (parallel track)
330
+ - History integration as separate story (depends on schema)
331
+ - Insights extension as separate story (depends on history integration)
332
+ - Retry integration as separate story (depends on insights correlation)
333
+
334
+ ### Areas Needing Careful Story Framing
335
+
336
+ - Feedback collection happens *within* agent execution; must not disrupt agent focus
337
+ - Quality gate UX: user prompt must clearly explain situation and options
338
+ - Issue code taxonomy: start with small set, plan for iterative expansion
339
+ - Calibration display: statistical concepts must be presented accessibly
340
+
341
+ ---
342
+
343
+ ## 12. Change Log (Feature-Level)
344
+
345
+ | Date | Change | Reason | Raised By |
346
+ |------------|---------------------------------------|---------------------------------|-----------|
347
+ | 2026-02-24 | Initial feature specification created | Feature request for agent feedback system | Alex |
@@ -0,0 +1,71 @@
1
+ # Implementation Plan — Feedback Loop Feature
2
+
3
+ ## Summary
4
+
5
+ This feature adds a quality feedback mechanism where downstream agents (Cass, Nigel, Codey) assess upstream artifacts before proceeding. Implementation requires a new `src/feedback.js` module for schema validation, quality gate logic, configuration management, and insights analysis. The existing `src/history.js`, `src/insights.js`, and `src/retry.js` modules need extensions to store, analyze, and act on feedback data.
6
+
7
+ ---
8
+
9
+ ## Files to Create/Modify
10
+
11
+ | Path | Action | Purpose |
12
+ |------|--------|---------|
13
+ | `src/feedback.js` | Create | Core feedback logic: validation, quality gates, config management |
14
+ | `src/history.js` | Modify | Add `storeStageFeedback()` and extend entry schema |
15
+ | `src/insights.js` | Modify | Add feedback analysis: `analyzeFeedbackCorrelation()`, calibration |
16
+ | `src/retry.js` | Modify | Add `mapIssuesToStrategies()` for feedback-informed retries |
17
+ | `bin/cli.js` | Modify | Register `feedback-config` command and `insights --feedback` flag |
18
+ | `src/index.js` | Modify | Export feedback module |
19
+
20
+ ---
21
+
22
+ ## Implementation Steps
23
+
24
+ 1. **Create `src/feedback.js` with schema validation** — Implement `validateFeedback()` per FEATURE_SPEC.md:Rule 1. Schema: `{about, rating, confidence, issues, recommendation}`. Return `{valid, errors}`.
25
+
26
+ 2. **Add quality gate logic to `src/feedback.js`** — Implement `shouldPause(feedback, config)` per FEATURE_SPEC.md:Rule 2. Returns true if `rating < minRatingThreshold` OR `recommendation === "pause"`.
27
+
28
+ 3. **Add config management to `src/feedback.js`** — Implement `getDefaultConfig()`, `readConfig()`, `writeConfig()`, `setConfigValue()`. Default: `{minRatingThreshold: 3.0, enabled: true, issueMappings: {...}}`.
29
+
30
+ 4. **Extend `src/history.js`** — Add `storeStageFeedback(slug, stage, feedback)` to persist feedback at `stages[stage].feedback`. Ensure backward compatibility (missing feedback = null).
31
+
32
+ 5. **Add calibration calculation to `src/insights.js`** — Implement `calculateCalibration(agent, history)` per FEATURE_SPEC.md:Rule 4. Return null if <10 runs with feedback, else correlation score 0-1.
33
+
34
+ 6. **Add issue correlation to `src/insights.js`** — Implement `correlateIssues(history)` to map issue codes to failure rates. Return `{issueCode: failureCorrelation}`.
35
+
36
+ 7. **Add threshold recommendation to `src/insights.js`** — Implement `recommendThreshold(history)` to suggest optimal minRatingThreshold based on historical data.
37
+
38
+ 8. **Extend `src/retry.js`** — Add `mapIssuesToStrategies(issues, config)` using default mappings from FEATURE_SPEC.md:Rule 3.
39
+
40
+ 9. **Register CLI commands in `bin/cli.js`** — Add `feedback-config` (view), `feedback-config set <key> <value>`, and `--feedback` flag to `insights` command.
41
+
42
+ 10. **Wire exports in `src/index.js`** — Export feedback module for orchestrator integration.
43
+
44
+ ---
45
+
46
+ ## Key Functions
47
+
48
+ **src/feedback.js:**
49
+ - `validateFeedback(feedback)` — Schema validation, returns `{valid, errors}`
50
+ - `shouldPause(feedback, config)` — Quality gate evaluation
51
+ - `getDefaultConfig()` / `readConfig()` / `writeConfig(config)` — Config I/O
52
+ - `setConfigValue(key, value)` — CLI config setter with validation
53
+ - `displayConfig()` — Pretty-print current config
54
+
55
+ **src/insights.js (new):**
56
+ - `calculateCalibration(agent, history)` — Agent calibration score
57
+ - `correlateIssues(history)` — Issue-to-failure correlation
58
+ - `recommendThreshold(history)` — Optimal threshold suggestion
59
+ - `displayFeedbackInsights(options)` — CLI output for `--feedback`
60
+
61
+ **src/retry.js (new):**
62
+ - `mapIssuesToStrategies(issues, config)` — Feedback-informed strategy selection
63
+
64
+ ---
65
+
66
+ ## Risks/Questions
67
+
68
+ - **Agent prompt integration**: Feedback collection requires agent prompts (in SKILL.md) to include feedback instructions. This is orchestrator-level work outside core modules.
69
+ - **Calibration metric**: Tests use simple accuracy (predicted vs actual). May need Pearson correlation for better calibration measure in production.
70
+ - **Issue taxonomy**: Starting with 6 issue codes per FEATURE_SPEC.md:Rule 3. Plan for iterative expansion.
71
+ - **Conflicting signals**: Per FEATURE_SPEC.md:Section 9, high rating + "pause" recommendation is unresolved. Recommend: recommendation takes precedence.