@thierrynakoa/fire-flow 10.0.0

package/skills-library/methodology/CONFIDENCE_GATED_EXECUTION.md

@@ -0,0 +1,243 @@
+# Confidence-Gated Execution — Adaptive Autonomy Through Uncertainty Estimation
+
+## The Problem
+
+AI agents treat all tasks with the same level of autonomy. Whether the agent is implementing a well-understood CRUD endpoint or modifying a security-critical authentication flow, it proceeds with the same approach. This leads to:
+
+- Overconfident execution on unfamiliar tasks (mistakes that could have been prevented by asking)
+- Underconfident pausing on familiar tasks (unnecessary user interruptions)
+- No learning signal for when to be cautious vs. autonomous
+
+### Why It Was Hard
+
+- Confidence is subjective — agents tend to default to "high confidence" (optimism bias)
+- Need concrete, measurable signals rather than vibes
+- Gate behavior must be lightweight (can't add 5 minutes of analysis before every action)
+- Must integrate with existing execution flows without disrupting them
+- Progressive autonomy requires tracking outcomes over time
+
+### Impact
+
+- Without confidence gates: 73.5% error recovery rate
+- With self-evaluation (which confidence gates enable): 95% error recovery rate
+- Users report 20%→40% auto-approve rate increase over 750 sessions when agents demonstrate appropriate caution
+
+---
+
+## The Solution
+
+### Root Cause
+
+Agents lack a structured way to estimate "how sure am I?" before acting. The SAUP framework (ACL 2025) shows that propagating uncertainty through reasoning steps significantly improves decision quality.
+
+### Signal-Based Confidence Scoring
+
+Instead of asking "how confident are you?" (subjective), compute confidence from concrete signals:
+
+```
+confidence = 50 (baseline)
+
+Positive signals (increase confidence):
+  + Matching skill found in library:      +20
+  + Similar reflection exists:            +15
+  + Tests available to verify:            +25
+  + Familiar technology/framework:        +15
+  + Clear, unambiguous requirements:      +15
+
+Negative signals (decrease confidence):
+  - Unfamiliar framework/library:         -20
+  - No tests available to verify:         -15
+  - Ambiguous/incomplete requirements:    -20
+  - Security-sensitive change:            -10
+  - Destructive operation:                -15
+```
+
+### Three Confidence Levels
+
+```
+HIGH (>80%): Proceed Autonomously
+  → Execute task directly
+  → Run Self-Judge after completion
+  → Log confidence in summary
+
+MEDIUM (50-80%): Proceed with Extra Validation
+  → Search reflections for similar scenarios
+  → Search skills library for guidance
+  → Run Self-Judge BEFORE and AFTER
+  → Log uncertainty reason for future learning
+
+LOW (<50%): Pause and Escalate
+  → Search Context7 for current library docs
+  → Check if this is outside trained domain
+  → Ask user for guidance before proceeding
+  → Create checkpoint before attempting
+  → Log what made confidence low
+```
+
+### Integration Points
+
+**In fire-3-execute (executor context injection):**
+```xml
+<confidence_gates>
+Before each plan task, estimate confidence using signal scoring.
+Log confidence level and signals in RECORD.md.
+Gate behavior: HIGH=proceed, MEDIUM=extra-validation, LOW=escalate.
+</confidence_gates>
+```
+
+**In fire-loop (recitation block):**
+```markdown
+## Confidence Check (v5.0)
+- Score: {0-100} — {HIGH/MEDIUM/LOW}
+- Signals: {what raised or lowered confidence}
+- Action: {proceed / extra-validation / escalate}
+```
+
+**In RECORD.md (execution log):**
+```yaml
+confidence_log:
+  - task: 1
+    score: 85
+    level: HIGH
+    signals: [skill_match, tests_available, familiar_tech]
+  - task: 3
+    score: 45
+    level: LOW
+    signals: [unfamiliar_framework, no_tests, ambiguous_requirements]
+    action: "Asked user for clarification on WebSocket auth approach"
+```
+
+### Code Example: Confidence Estimation
+
+```python
+# Conceptual implementation (adapt to your agent framework)
+
+def estimate_confidence(task, skills_library, reflections, test_suite):
+    score = 50  # baseline
+    signals = []
+
+    # Check skill library
+    matching_skills = skills_library.search(task.description)
+    if matching_skills:
+        score += 20
+        signals.append("skill_match")
+
+    # Check reflections
+    matching_reflections = reflections.search(task.description)
+    if matching_reflections:
+        score += 15
+        signals.append("reflection_match")
+
+    # Check test availability
+    if test_suite.has_tests_for(task.affected_files):
+        score += 25
+        signals.append("tests_available")
+    else:
+        score -= 15
+        signals.append("no_tests")
+
+    # Check technology familiarity
+    if task.technology in KNOWN_TECHNOLOGIES:
+        score += 15
+        signals.append("familiar_tech")
+    else:
+        score -= 20
+        signals.append("unfamiliar_framework")
+
+    # Check requirement clarity
+    if task.has_clear_acceptance_criteria:
+        score += 15
+        signals.append("clear_requirements")
+    else:
+        score -= 20
+        signals.append("ambiguous_requirements")
+
+    # Security/destructive checks
+    if task.is_security_sensitive:
+        score -= 10
+        signals.append("security_sensitive")
+    if task.is_destructive:
+        score -= 15
+        signals.append("destructive_operation")
+
+    # Clamp to 0-100
+    score = max(0, min(100, score))
+
+    level = "HIGH" if score > 80 else "MEDIUM" if score >= 50 else "LOW"
+    return score, level, signals
+```
+
+---
+
+## Testing the Fix
+
+### Scenario Tests
+
+| Task | Expected Score | Expected Level | Expected Action |
+|------|---------------|----------------|-----------------|
+| Add CRUD endpoint (known stack, tests exist, skill found) | 90+ | HIGH | Proceed |
+| Implement WebSocket auth (unfamiliar, no tests, no skill) | 30-40 | LOW | Escalate |
+| Fix CSS layout (familiar, no tests, clear requirement) | 65-75 | MEDIUM | Extra validation |
+| Delete user data migration (familiar, tests, destructive) | 55-65 | MEDIUM | Extra validation |
+| Integrate new payment provider (unfamiliar, security) | 25-35 | LOW | Escalate |
+
+### Verification
+
+1. Execute tasks of varying familiarity
+2. Verify HIGH-confidence tasks proceed without interruption
+3. Verify MEDIUM-confidence tasks trigger reflection/skill search
+4. Verify LOW-confidence tasks pause for user input
+5. Check confidence log in RECORD.md matches observed behavior
+
+---
+
+## Prevention
+
+- Calibrate signals based on actual outcomes (if LOW-confidence tasks consistently succeed, adjust weights)
+- Don't let confidence become a checkbox (the score should reflect genuine uncertainty)
+- Review confidence logs periodically to identify systematic biases
+- Track confidence vs. outcome correlation to improve scoring
+
+---
+
+## Related Patterns
+
+- [AGENT_SELF_IMPROVEMENT_LOOP](./AGENT_SELF_IMPROVEMENT_LOOP.md) - Confidence is upgrade #6 of the loop
+- [REFLEXION_MEMORY_PATTERN](./REFLEXION_MEMORY_PATTERN.md) - Reflections feed confidence scoring (+15)
+- [SELF_QUESTIONING_TASK_GENERATION](./SELF_QUESTIONING_TASK_GENERATION.md) - Self-Judge runs at MEDIUM+ confidence
+- [CONFIDENCE_ANNOTATION_PATTERN](./CONFIDENCE_ANNOTATION_PATTERN.md) - Related annotation approach
+
+---
+
+## Common Mistakes to Avoid
+
+- Setting baseline too high (70+) — makes everything look "confident"
+- Ignoring negative signals — agents naturally want to proceed
+- Treating confidence gates as hard blockers — they're advisory, agent can override with justification
+- Not logging confidence scores — you need the data to calibrate over time
+- Applying same weights to all task types — security tasks should weight "security_sensitive" more heavily
+- Making confidence estimation take >30 seconds — speed is critical for adoption
+
+---
+
+## Resources
+
+- SAUP (ACL 2025): Uncertainty propagation through reasoning steps
+- Anthropic measurement: Progressive autonomy 20%→40% over 750 sessions
+- Snorkel AI: 95% vs 73.5% error recovery with self-evaluation
+- Dominion Flow implementation: `fire-3-execute.md` confidence_gates, `fire-loop.md` confidence check
+
+---
+
+## Time to Implement
+
+**2-3 hours** — Add confidence scoring to executor context, integrate into loop recitation, add RECORD.md logging
+
+## Difficulty Level
+
+Stars: 3/5 — The signal scoring is simple. The challenge is **calibration**: getting the weights right so the agent isn't overconfident or over-cautious. Requires tracking outcomes over multiple sessions.
+
+---
+
+**Author Notes:**
+The most counterintuitive finding: agents that ask for help more often perform better overall. LOW-confidence escalation isn't a failure — it's the agent saying "I know what I don't know." The 95% error recovery rate comes not from avoiding errors, but from knowing when to pause and seek guidance. Confidence gates formalize the difference between "I know this" and "I'm guessing" — and that distinction is worth a 21.5 percentage point improvement in recovery rate.

package/skills-library/methodology/EVIDENCE_BASED_VALIDATION.md

@@ -0,0 +1,308 @@
+# Evidence-Based Validation - Verification Before Completion
+
+## The Problem
+
+AI agents (and humans) often claim work is "done" without actually verifying it works. This leads to:
+- "Tests pass" claims without running tests
+- "Bug fixed" assertions without reproduction verification
+- Premature completion claims that waste time on rework
+
+### Why It Was Hard
+
+- Pressure to deliver quickly encourages shortcuts
+- Verification feels redundant after writing code
+- Confidence in one's work creates blind spots
+- No systematic enforcement of "prove it works"
+
+### Impact
+
+- False completion claims waste reviewer time
+- Bugs reach production that should have been caught
+- Trust erodes when "done" doesn't mean "done"
+- Rework costs exceed original implementation time
+
+---
+
+## The Solution
+
+**Evidence Before Assertion** - Never claim something works without captured proof.
+
+### Core Principle
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                                                                 │
+│  WRONG: "The tests pass" → then run tests                      │
+│  RIGHT: Run tests → "The tests pass (output: 45/45)"           │
+│                                                                 │
+│  WRONG: "I fixed the bug" → hope it works                      │
+│  RIGHT: Reproduce → fix → verify fix → "Bug fixed (proof)"     │
+│                                                                 │
+└─────────────────────────────────────────────────────────────────┘
+```
+
+### The Verification Protocol
+
+#### Step 1: Identify Claims
+
+Before declaring work complete, list all implicit claims:
+
+```markdown
+## Claims to Verify
+
+| # | Implicit Claim | Evidence Required |
+|---|----------------|-------------------|
+| 1 | Code compiles | Build output with exit code 0 |
+| 2 | Tests pass | Test runner output showing pass count |
+| 3 | Feature works | Demo or API test with expected response |
+| 4 | No regressions | Full test suite output |
+| 5 | Types correct | TypeScript compiler output |
+```
+
+#### Step 2: Execute Verification Commands
+
+Run ALL commands and capture output:
+
+```bash
+# Build check
+echo "=== BUILD CHECK ==="
+npm run build 2>&1
+echo "Exit code: $?"
+
+# Test check
+echo "=== TEST CHECK ==="
+npm run test 2>&1
+echo "Exit code: $?"
+
+# Type check
+echo "=== TYPE CHECK ==="
+npm run typecheck 2>&1
+echo "Exit code: $?"
+
+# Lint check
+echo "=== LINT CHECK ==="
+npm run lint 2>&1
+echo "Exit code: $?"
+```
+
+#### Step 3: Document Results with Evidence
+
+```markdown
+## Verification Results
+
+### Build Check
+**Command:** `npm run build`
+**Exit Code:** 0
+**Output:**
+```
+> project@1.0.0 build
+> tsc && vite build
+
+vite v5.0.0 building for production...
+✓ 234 modules transformed.
+dist/index.html          0.45 kB │ gzip:  0.29 kB
+dist/assets/index.js   145.67 kB │ gzip: 47.23 kB
+✓ built in 2.34s
+```
+**Verdict:** PASS
+
+### Test Check
+**Command:** `npm run test`
+**Exit Code:** 0
+**Output:**
+```
+PASS src/auth/login.test.ts
+PASS src/api/users.test.ts
+PASS src/utils/helpers.test.ts
+
+Test Suites: 3 passed, 3 total
+Tests:       45 passed, 45 total
+Time:        3.42s
+```
+**Verdict:** PASS (45/45 tests)
+```
+
+#### Step 4: Honesty Protocol
+
+Before claiming complete, answer honestly:
+
+```markdown
+## Honesty Check
+
+### Question 1: Did I actually run these commands?
+- [x] Yes, all commands executed in this session
+- [ ] No, I assumed based on previous runs
+
+### Question 2: Am I interpreting output honestly?
+- [x] Output clearly shows success
+- [ ] Output is ambiguous, I'm assuming
+
+### Question 3: What am I NOT checking?
+- [ ] Nothing unchecked
+- [x] E2E tests not run (documented limitation)
+
+### Question 4: Would a skeptic be convinced?
+- [x] Yes, evidence is clear
+- [ ] No, more verification needed
+```
+
+#### Step 5: Final Verdict
+
+```markdown
+## Double-Check Summary
+
+| Check | Command | Result | Evidence |
+|-------|---------|--------|----------|
+| Build | `npm run build` | PASS | Exit 0, no errors |
+| Tests | `npm run test` | PASS | 45/45 passing |
+| Types | `npm run typecheck` | PASS | 0 errors |
+| Lint | `npm run lint` | PASS | 0 warnings |
+
+**STATUS:** VERIFIED
+**Confidence:** HIGH
+
+You may now claim this work is complete.
+```
+
+---
+
+## Implementation
+
+### Verification Command Template
+
+```javascript
+async function doubleCheck(checks = ['build', 'test', 'typecheck', 'lint']) {
+  const results = {};
+
+  for (const check of checks) {
+    const command = COMMANDS[check];
+    console.log(`=== ${check.toUpperCase()} CHECK ===`);
+
+    const { stdout, stderr, exitCode } = await exec(command);
+
+    results[check] = {
+      command,
+      exitCode,
+      output: stdout + stderr,
+      verdict: exitCode === 0 ? 'PASS' : 'FAIL'
+    };
+
+    console.log(`Exit code: ${exitCode}`);
+    console.log(stdout);
+  }
+
+  return results;
+}
+
+const COMMANDS = {
+  build: 'npm run build',
+  test: 'npm run test',
+  typecheck: 'npm run typecheck',
+  lint: 'npm run lint',
+  coverage: 'npm run test:coverage',
+  security: 'npm audit',
+  e2e: 'npm run test:e2e'
+};
+```
+
+### Anti-Pattern Detection
+
+```javascript
+// BAD: Claiming without evidence
+function reviewCode() {
+  // ... look at code ...
+  return "Tests pass"; // NO EVIDENCE!
+}
+
+// GOOD: Evidence-based claim
+async function reviewCode() {
+  const output = await exec('npm run test');
+  return `Tests pass (${output.passCount}/${output.totalCount})`;
+}
+```
+
+---
+
+## Testing the Pattern
+
+### Before (No Verification)
+```
+Claim: "All tests pass"
+Reality: 3 tests failing
+Result: Bug reaches production
+Cost: 4 hours debugging + hotfix
+```
+
+### After (Evidence-Based)
+```
+Claim: "All tests pass"
+Evidence: Test output shows 42/45 passing
+Reality: 3 tests failing (caught immediately)
+Result: Fixed before merge
+Cost: 15 minutes
+```
+
+---
+
+## Prevention
+
+### When to Use Evidence-Based Validation
+
+- **Always:** Before claiming any work is complete
+- **Always:** Before creating a PR
+- **Always:** Before merging to main
+- **Always:** After fixing bugs (verify the fix)
+
+### Verification Depth Levels
+
+| Depth | Checks | Use Case |
+|-------|--------|----------|
+| Quick | build, lint | Minor changes |
+| Standard | build, test, types, lint | Normal PRs |
+| Deep | All + coverage + security + E2E | Production releases |
+
+---
+
+## Related Patterns
+
+- [Multi-Perspective Code Review](./MULTI_PERSPECTIVE_CODE_REVIEW.md)
+- [Honesty Protocols](./HONESTY_PROTOCOLS.md)
+- [60-Point Validation Checklist](./VALIDATION_CHECKLIST.md)
+
+---
+
+## Common Mistakes to Avoid
+
+- **Skipping verification to save time** - Rework costs more
+- **Assuming previous run is still valid** - Always re-run
+- **Ignoring warnings** - Warnings become errors
+- **Partial verification** - Run ALL relevant checks
+- **Trusting memory** - Capture actual output
+
+---
+
+## Resources
+
+- [superpowers:verification-before-completion](https://github.com/anthropics/claude-code-plugins)
+- [Test-Driven Development patterns](./TDD_PATTERNS.md)
+- [Continuous Integration best practices](../deployment-security/CI_CD_PATTERNS.md)
+
+---
+
+## Time to Implement
+
+**Per verification:** 2-5 minutes
+**ROI:** Prevents 1-4 hour debugging sessions
+
+## Difficulty Level
+
+⭐ (1/5) - Simple to implement, requires discipline
+
+---
+
+**Author Notes:**
+This pattern seems obvious but is violated constantly. The key insight is that **verification must be mandatory, not optional**. By requiring captured output as evidence, you eliminate the possibility of false claims.
+
+The honesty protocol questions force reflection before completion. Question 3 ("What am I NOT checking?") is particularly powerful - it surfaces blind spots before they become problems.
+
+**Implementation in Dominion Flow:** Available via `/fire-double-check` command.