shipwright-cli 3.1.0 → 3.3.0

package/docs/research/BACKLOG_QUICK_REFERENCE.md

@@ -0,0 +1,352 @@
+# Shipwright Backlog: Quick Reference (20-Item Priority List)
+
+## At-a-Glance Priority Matrix
+
+| Priority | ID  | Feature                                              | Impact   | Effort   | ROI             | Category      |
+| -------- | --- | ---------------------------------------------------- | -------- | -------- | --------------- | ------------- |
+| 🔴 P0    | #1  | Semantic trajectory analysis + convergence detection | 🟢🟢🟢   | 🟡🟡     | **EXCEPTIONAL** | Loop Patterns |
+| 🔴 P0    | #2  | Intent Specification Engine (business → outcomes)    | 🟢🟢🟢🟢 | 🔴🔴🔴   | **EXCEPTIONAL** | Dark Factory  |
+| 🔴 P0    | #3  | Vulnerability Reward Model + online RL               | 🟢🟢🟢   | 🟡🟡     | **EXCEPTIONAL** | RL/Security   |
+| 🔴 P0    | #5  | Speculative Cascade Model Routing                    | 🟢🟢🟢🟢 | 🟡🟡     | **VERY HIGH**   | Cost          |
+| 🟡 P1    | #4  | Episodic Memory Layer                                | 🟢🟢🟢   | 🔴🔴🔴   | **HIGH**        | Memory        |
+| 🟡 P1    | #6  | Mutation Testing Feedback Loop                       | 🟢🟢🟢   | 🟡🟡     | **HIGH**        | Testing       |
+| 🟡 P1    | #7  | CI Repair Agent                                      | 🟢🟢🟢   | 🔴🔴🔴   | **HIGH**        | Self-Healing  |
+| 🟡 P1    | #8  | LLM-as-a-Judge validation                            | 🟢🟢     | 🟡🟡     | **HIGH**        | Quality       |
+| 🟢 P2    | #9  | Explicit Conflict Detection + DAG Scheduling         | 🟢🟢     | 🟡🟡     | **MEDIUM**      | Multi-Agent   |
+| 🟢 P2    | #10 | Intelligent Reasoning Budget Allocation              | 🟢🟢     | 🟡🟡     | **MEDIUM**      | Reasoning     |
+| 🟢 P2    | #11 | Formal Verification Integration (Dafny/Lean)         | 🟢🟢     | 🔴🔴🔴🔴 | **MEDIUM**      | Verification  |
+| 🟢 P2    | #12 | Active Context Compression + Semantic Memory         | 🟢🟢🟢   | 🔴🔴🔴   | **MEDIUM**      | Memory        |
+| 🟢 P2    | #13 | Multi-Pass Mutation Generation (LLM-based)           | 🟢🟢     | 🔴🔴🔴   | **MEDIUM**      | Testing       |
+| 🟢 P2    | #14 | Anomaly Detection + Predictive Repair                | 🟢🟢     | 🔴🔴🔴   | **MEDIUM**      | Self-Healing  |
+| 🟢 P2    | #15 | Cross-Repo Fleet Learning                            | 🟢🟢     | 🔴🔴🔴   | **MEDIUM**      | Memory/Fleet  |
+| 🟢 P3    | #16 | Quorum-Based Merge Decisions                         | 🟢       | 🟡       | **LOW**         | Quality       |
+| 🟢 P3    | #17 | Privacy-Hardening Mutations                          | 🟢       | 🔴🔴     | **LOW**         | Compliance    |
+| 🟢 P3    | #18 | Dependency-Aware Task Scheduling (DAG)               | 🟢       | 🟡       | **LOW**         | Multi-Agent   |
+| 🟢 P3    | #19 | Symbol Caching + Semantic Search                     | 🟢       | 🟡       | **LOW**         | Performance   |
+| 🟢 P3    | #20 | WebSocket Real-Time Loop Monitoring                  | 🟢       | 🟡       | **LOW**         | Observability |
+
+---
+
+## PHASE 1 (Weeks 1-4): Convergence & Cost
+
+### #1 Semantic Trajectory Analysis + Convergence Detection
+
+**What it does:** Tracks embedding-space distance of consecutive agent outputs; detects stuck (contractive) vs wandering (exploratory) loops
+
+**Why it matters:**
+
+- Current: Hard iteration limit (5 iterations) wastes compute on stuck loops
+- SOTA: Geometric Dynamics paper (arxiv 2512.10350) shows regime detection enables early exit
+- Impact: 25-40% iteration waste reduction
+
+**How to implement:**
+
+1. On each loop iteration: encode agent output to embedding space (use Claude's embeddings)
+2. Compute cosine distance to previous iteration's embedding
+3. Track distance trend (contracting = converging, diverging = exploring)
+4. Early exit if contracting + distance < threshold
+5. Escalate to longer thinking if diverging unbounded
+
+**Effort:** Medium (embedding integration, vector math, tracking state)  
+**Blocking:** Nothing (can implement in isolation)  
+**Files to modify:** `sw-loop.sh`, `sw-convergence-test.sh`
+
+---
+
+### #5 Speculative Cascade Model Routing
+
+**What it does:** Try Haiku first (short timeout), escalate to Sonnet → Opus on failure
+
+**Why it matters:**
+
+- Current: Pick model upfront (per `--effort` flag), no escalation
+- SOTA: Google Speculative Cascades paper; 30-60% cost reduction on median tasks
+- Impact: 40-60% cost reduction while maintaining quality
+
+**How to implement:**
+
+1. Build failure prediction model: (query_type, difficulty) → success_rate on Haiku
+2. For new query: estimate difficulty via embedding similarity
+3. Route to Haiku with timeout (e.g., 30s)
+4. If timeout/failure (tests fail), cascade to Sonnet, then Opus
+5. Track cascade effectiveness per query type in memory
+
+**Effort:** Medium (timeout management, cascade orchestration, tracking)  
+**Blocking:** Nothing  
+**Files to modify:** `sw-model-router.sh`, `sw-loop.sh`, new: `sw-cascade-router.sh`
+
+---
+
+## PHASE 2 (Weeks 5-8): Security & Testing
+
+### #3 Vulnerability Reward Model + Online RL Hardening
+
+**What it does:** Add security signals (detected vulnerabilities, CWE patterns) to reward model; enable vulnerability-aware RL
+
+**Why it matters:**
+
+- Current: Reward signals are functional-only (test pass, coverage)
+- SOTA: Meta's SecCoderX, Anthropic's security research
+- Impact: 30-40% security issue reduction; compliance-ready code
+
+**How to implement:**
+
+1. Integrate lightweight SAST (e.g., Semgrep, bandit, Trivy)
+2. Run on generated code; extract (vulnerability_count, cwe_classes)
+3. Add to reward signal as negative reward: reward -= vulnerability_count \* weight
+4. Store effective security fixes in episodic memory
+5. Fine-tune on secure code examples
+
+**Effort:** Medium (scanner integration, signal weighting, RL loop)  
+**Blocking:** Nothing  
+**Files to modify:** `sw-reward-aggregator.sh`, `sw-rl-optimizer.sh`, new: `sw-security-reward.sh`
+
+---
+
+### #6 Mutation Testing Feedback Loop
+
+**What it does:** Validate test quality by checking % of mutants killed; regenerate tests if score low
+
+**Why it matters:**
+
+- Current: Coverage metrics inflated; 45% of LLM-generated tests are ineffective
+- SOTA: Meta ACH, MutGen papers show mutation feedback improves test quality
+- Impact: 30-40% better test effectiveness; catches subtle bugs
+
+**How to implement:**
+
+1. After test generation: run mutation tool (Major, PIT) on code
+2. Run generated tests against mutants; compute mutation_score = killed / total
+3. If score < threshold (e.g., 80%): add feedback to testgen prompt
+4. Regenerate tests with mutation feedback
+5. Store effective test patterns for reuse
+
+**Effort:** Medium (mutation tool integration, feedback loop)  
+**Blocking:** Nothing  
+**Files to modify:** `sw-testgen.sh`, new: `sw-mutation-validator.sh`
+
+---
+
+### #13 Multi-Pass Mutation Generation (LLM-based)
+
+**What it does:** Use LLM to generate diverse mutants (not just rule-based); Meta-style compliance
+
+**Why it matters:**
+
+- Current: Traditional mutation tools (Major) have limited operators
+- SOTA: GPT-4o/DeepSeek-R1 generate 57 different AST node types vs 2 for rules
+- Impact: Better mutation diversity; more confident test validation
+
+**How to implement:**
+
+1. Take source code + list of mutation types
+2. Prompt LLM: "Generate N mutants that change behavior but keep syntax valid"
+3. Validate mutants compile + are distinct from originals
+4. Run tests; track mutation score
+5. Feed back into testgen loop if coverage is low
+
+**Effort:** High (prompt engineering, mutation validation)  
+**Blocking:** Nothing  
+**Files to modify:** new: `sw-llm-mutant-generator.sh`
+
+---
+
+## PHASE 3 (Weeks 9-12): Memory & Self-Healing
+
+### #4 Episodic Memory Layer
+
+**What it does:** Store complete execution traces (inputs, actions, outcomes); enable case-based reasoning
+
+**Why it matters:**
+
+- Current: Memory is pattern-based ("when X fails, do Y")
+- SOTA: Mem0, EM-LLM, MemRL papers show episodic learning 20-35% faster
+- Impact: Case-based analogy; long-horizon self-improvement
+
+**How to implement:**
+
+1. On each pipeline run: capture episode JSON (inputs, agent_actions, outputs, duration, cost, test_results)
+2. Store in episodic DB (SQLite + JSON or Postgres)
+3. Query: "Find 3 similar past episodes" (via embedding similarity)
+4. Inject case as few-shot examples into new agent prompts
+5. Active compression: every 10 episodes, consolidate → semantic facts
+
+**Effort:** High (episode storage, retrieval, compression)  
+**Blocking:** Nothing  
+**Files to modify:** `sw-memory.sh`, new: `sw-episodic-memory.sh`
+
+---
+
+### #7 CI Repair Agent
+
+**What it does:** When test/check fails, spawn repair agent to diagnose & fix root cause
+
+**Why it matters:**
+
+- Current: Retries on failure; no diagnosis
+- SOTA: Pipeline Doctor pattern (2026 AIOps trend); 67% MTTR drop
+- Impact: 50% fewer retries; faster merge times
+
+**How to implement:**
+
+1. Detect test/check failure (via CI logs)
+2. Classify failure: timeout, race condition, assertion, resource, flaky
+3. Spawn repair agent with failure context (logs, git diff, error)
+4. Agent proposes fix (increase timeout, add sync, skip flaky test, etc.)
+5. Re-run test; if passes, commit repair
+6. Track effective repairs in memory
+
+**Effort:** High (log parsing, classification, repair proposals, commit management)  
+**Blocking:** Nothing  
+**Files to modify:** `sw-ci.sh`, new: `sw-repair-agent.sh`
+
+---
+
+### #8 LLM-as-a-Judge Validation
+
+**What it does:** Secondary model evaluates primary agent output; triggers repair if needed
+
+**Why it matters:**
+
+- Current: Quality gates are rule-based (coverage > X%, no ASan)
+- SOTA: 2026 standard design pattern for agentic systems
+- Impact: 10-15% fewer merge regressions; catches issues rules miss
+
+**How to implement:**
+
+1. After primary agent completes task: send code + acceptance criteria to Judge model
+2. Judge evaluates: "Does this code meet requirements? Any issues?"
+3. If Judge flags issues: auto-trigger repair agent or escalate
+4. Log Judge decisions for learning
+5. Track Judge accuracy (via post-merge bug rates)
+
+**Effort:** Medium (prompt engineering, logic orchestration)  
+**Blocking:** Nothing  
+**Files to modify:** `sw-quality.sh`, new: `sw-judge.sh`
+
+---
+
+## TIER 2 Items (Brief Summary)
+
+| #   | Feature                               | Quick Implementation Path                                                      |
+| --- | ------------------------------------- | ------------------------------------------------------------------------------ |
+| #2  | Intent Specification Engine           | Research phase; build DSL for constraints; integrate formal spec generation    |
+| #9  | Conflict Detection + DAG              | Track file locks per agent; build task DAG scheduler; merge conflict resolver  |
+| #10 | Reasoning Budget Allocation           | Track thinking cost vs outcome; build (task_type, complexity) → tokens lookup  |
+| #11 | Formal Verification (Dafny/Lean)      | Integrate theorem prover APIs; generate specs; gate merge on proof completion  |
+| #12 | Active Context Compression            | EM-LLM approach: Bayesian surprise + graph refinement for episode boundaries   |
+| #14 | Anomaly Detection + Predictive Repair | Time-series analysis on logs; ML model for failure prediction; repair triggers |
+| #15 | Cross-Repo Fleet Learning             | Share patterns via fleet event bus; rank patterns by repo similarity           |
+
+---
+
+## Implementation Checklist
+
+### PHASE 1 (Target: 2 weeks per item)
+
+- [ ] #1 Semantic trajectory analysis
+  - [ ] Embedding integration
+  - [ ] Distance tracking + regime classification
+  - [ ] Early exit logic
+  - [ ] Tests + monitoring
+- [ ] #5 Speculative cascade routing
+  - [ ] Failure prediction model
+  - [ ] Cascade orchestration
+  - [ ] Timeout management
+  - [ ] Tracking + learning
+
+### PHASE 2 (Target: 1.5-2 weeks per item)
+
+- [ ] #3 Vulnerability reward model
+- [ ] #6 Mutation testing loop
+- [ ] #13 LLM-based mutants
+
+### PHASE 3 (Target: 2-3 weeks per item)
+
+- [ ] #4 Episodic memory layer
+- [ ] #7 CI repair agent
+- [ ] #8 LLM-as-a-Judge
+
+---
+
+## Success Metrics (Post-Implementation)
+
+| Feature             | Metric                              | Target  | Current  |
+| ------------------- | ----------------------------------- | ------- | -------- |
+| #1 Loop convergence | Iteration waste reduction           | -25-40% | Baseline |
+| #5 Cascade routing  | Cost reduction on median tasks      | -40-60% | Baseline |
+| #3 Security rewards | Bug reduction                       | -30-40% | Current  |
+| #6 Mutation testing | Test effectiveness (mutation score) | >80%    | ~60%     |
+| #4 Episodic memory  | Solution time on similar tasks      | -20-35% | Baseline |
+| #7 CI repair        | Retry cycles                        | -50%    | Baseline |
+| Overall             | Pipeline success rate               | >85%    | ~77%     |
+
+---
+
+## Dependencies & Blocking Relationships
+
+```
+#1 (trajectory) ─────┐
+                     ├──→ #5 (cascade) ──→ Cost optimization ✓
+                     │
+#2 (intent)  [research phase; no immediate blocks]
+
+#3 (vulnerability) ──┐
+#6 (mutations)       ├──→ Security + Testing quality
+#13 (LLM mutants) ───┘
+
+#4 (episodic) ───────┐
+#12 (compression) ───┤
+#15 (fleet learning) ┤ All feed each other; can implement in parallel
+                     └──→ Long-horizon learning
+
+#7 (CI repair) ──┐
+#8 (judge)       └──→ Quality gates
+
+No critical blocking path: all items can start immediately with risk.
+Recommend: Start #1 + #5 in week 1, #3 + #6 in week 5, #4 + #7 in week 9.
+```
+
+---
+
+## Cost-Benefit Analysis
+
+### Immediate ROI (Phase 1-2, Weeks 1-8)
+
+**Investment:**
+
+- 2 engineers × 8 weeks @ $200K/year = ~$60K engineering cost
+- Compute for research + prototyping = ~$5K
+
+**Returns (Annual):**
+
+- Cost reduction via cascade: 40-60% savings on compute (current $50K/month → $20-30K) = **$240-360K/year**
+- Faster iteration: 30% speedup on 200 pipelines/month × $5/pipeline = **$30K/year**
+- Security improvement: 30-40% fewer CVEs → reduced incident response = **$50K+ saved**
+
+**Total Annual ROI: $320-440K on $65K investment = 5-7x**
+
+### Long-Term ROI (Phase 3 + Beyond, Weeks 9-26)
+
+**Additional returns:**
+
+- Episodic memory: 20-35% faster solutions × 200 pipelines = **$50-85K/year**
+- Self-healing CI: 50% fewer retries = **$30K/year** (fewer human reviews)
+- Fleet learning: 20% faster on new projects = **$40K/year**
+
+**Total Long-Term ROI: $440-555K on $120K investment = 3-4x**
+
+---
+
+## Next Steps
+
+1. **This week:** Review [CUTTING_EDGE_RESEARCH_2026.md](./CUTTING_EDGE_RESEARCH_2026.md) for full details on each feature
+2. **Next week:** Spike on #1 (trajectory analysis) — prototype embedding-space distance tracking
+3. **Following week:** Begin #5 (cascade routing) and #3 (vulnerability rewards) in parallel
+4. **Week 4+:** Ramp up to PHASE 2 items as Phase 1 items ship
+
+---
+
+**Generated:** April 4, 2026  
+**Total research effort:** 50+ sources, 25+ papers, 8 research areas  
+**Full report:** See CUTTING_EDGE_RESEARCH_2026.md (comprehensive analysis)