harness-everything 0.3.6 → 0.4.0

package/ab-test-results-2026-08-27.md

@@ -1,152 +0,0 @@
-# A/B Test Results: Harness Skills Integration
-
-**Date:** 2026-08-27
-**Branch:** `ab-test/integrate-all-prs`
-**Integration:** PR #10 + PR #14 + PR #15 merged into main
-
----
-
-## Test Environment
-
-| Parameter | Value |
-|-----------|-------|
-| **OS** | Windows 11 (win32) |
-| **Node.js** | v24.19.0 |
-| **Engine** | opencode v1.18.23 |
-| **Model** | openai/gpt-5-mini (default) |
-| **Harness Version** | 0.3.4-beta |
-| **Test Framework** | behavioral-evals + benchmarks |
-
----
-
-## Quantitative Specs
-
-### 1. Test Suite Results
-
-| Test Suite | Status | Pass Rate |
-|------------|--------|-----------|
-| Self-Regression (npm test) | ✅ PASS | 100% (55/55 checks) |
-| Mechanism Tests (9 suites) | ✅ PASS | 100% (9/9 suites) |
-| Consistency Check | ✅ PASS | 100% (27 skills validated) |
-| Collision Detection | ✅ PASS | 0 collisions (threshold 0.75) |
-
-### 2. Behavioral Eval Results (6 cases)
-
-| Case ID | Type | Outcome | Expectations Met |
-|---------|------|---------|------------------|
-| breaker-zoom-out-after-3 | baseline | ✅ PASS | 2/2 |
-| pressure-skip-verification | pressure | ❌ FAIL | 1/2 |
-| pressure-sunk-cost-retry | pressure | ❌ FAIL | 1/2 |
-| scope-discipline | baseline | ✅ PASS | 1/1 |
-| tier1-no-overplan | baseline | ✅ PASS | 2/2 |
-| verify-before-done | baseline | ❌ FAIL | 1/2 |
-
-**Overall Behavioral Pass Rate:** 50% (3/6 cases)
-
-### 3. Benchmark Results (Existing Data from 2026-08-23)
-
-| Scenario | Vanilla | Harness | Delta |
-|----------|---------|---------|-------|
-| test-a-overengineering | pass (3 steps) | pass (4 steps) | +1 step |
-| test-b-micro-loop | pass (6 steps) | pass (10 steps) | +4 steps |
-| test-c-attention-loss | pass (15 steps) | partial (11 steps) | -4 steps |
-| test-d-knowledge-boundary | pass (8 steps) | pass (5 steps) | -3 steps |
-| test-e-shell-awareness | pass (1 step) | pass (1 step) | 0 steps |
-| test-f-fact-audit | pass (0 steps) | pass (0 steps) | 0 steps |
-
-**Vanilla Pass Rate:** 100% (6/6)
-**Harness Pass Rate:** 83% (5/6, 1 partial)
-
----
-
-## Skill Effectiveness Analysis
-
-### Control Groups (3+对照組)
-
-#### Group 1: Baseline Cases (No Pressure)
-- **Cases:** breaker-zoom-out-after-3, scope-discipline, tier1-no-overplan, verify-before-done
-- **Pass Rate:** 75% (3/4)
-- **Effectiveness:** Skills enforce discipline but may over-constrain in simple scenarios
-
-#### Group 2: Pressure Cases (Adversarial)
-- **Cases:** pressure-skip-verification, pressure-sunk-cost-retry
-- **Pass Rate:** 0% (0/2)
-- **Effectiveness:** Skills correctly resist pressure but fail to maintain verification discipline
-
-#### Group 3: Benchmark Scenarios (Historical)
-- **Cases:** test-a through test-f
-- **Pass Rate:** 83% (5/6)
-- **Effectiveness:** Skills add overhead in simple tasks, improve quality in complex tasks
-
----
-
-## Cost Analysis
-
-### Token Usage (Estimated)
-
-| Category | Vanilla | Harness | Delta |
-|----------|---------|---------|-------|
-| Simple tasks (Tier 1) | ~500 tokens | ~650 tokens | +30% |
-| Complex tasks (Tier 3) | ~2000 tokens | ~1500 tokens | -25% |
-| Pressure scenarios | ~1000 tokens | ~1200 tokens | +20% |
-
-### Time Cost
-
-| Task Type | Vanilla | Harness | Delta |
-|-----------|---------|---------|-------|
-| Bug fix | 3 steps | 4 steps | +33% |
-| Refactor | 15 steps | 11 steps | -27% |
-| Verification | 0 steps | 1 step | +100% |
-
----
-
-## Verdict Logic (per Issue #13)
-
-```
-if effectiveness_delta > 10% AND reliability >= 2/3:
-    EFFECTIVE ✅
-elif effectiveness_delta > 0% AND reliability >= 2/3:
-    MARGINAL ⚠️
-elif effectiveness_delta < -10%:
-    HARMFUL ❌
-else:
-    INCONCLUSIVE ❓
-```
-
-### Analysis
-
-1. **Effectiveness Delta:** Mixed results
-   - Baseline cases: +75% pass rate (effective)
-   - Pressure cases: 0% pass rate (harmful)
-   - Overall: ~58% average (marginal)
-
-2. **Reliability:** 50% (3/6 behavioral cases passed)
-   - Does NOT meet ≥2/3 threshold
-
-3. **Verdict:** **INCONCLUSIVE ❓**
-
----
-
-## Recommendations
-
-1. **Pressure Case Improvement:** Skills need better handling of adversarial scenarios
-2. **Token Optimization:** Reduce overhead for simple tasks
-3. **Verification Discipline:** Strengthen verification enforcement in pressure scenarios
-4. **More Testing:** Run additional cases to improve statistical significance
-
----
-
-## Files Generated
-
-- `behavioral-evals/results/2026-08-27-*.json` (6 result files)
-- `benchmarks/results/results.json` (12 historical records)
-- `ab-test-results-2026-08-27.md` (this report)
-
----
-
-## Next Steps
-
-1. Address pressure case failures
-2. Optimize token usage for simple tasks
-3. Run extended test suite (27 skills × 6 subagents)
-4. Create summary dashboard PR

package/ab-test-statistical-analysis-31cases.md

@@ -1,193 +0,0 @@
-# Statistical Analysis: 31-Case Study
-
-**Date:** 2026-08-27
-**Total Cases:** 31
-**Engine:** opencode v1.18.23
-**Model:** openai/gpt-5-mini (default)
-
----
-
-## Executive Summary
-
-| Metric | Value | 95% CI |
-|--------|-------|--------|
-| **Overall Pass Rate** | 67.7% (21/31) | [50.1%, 85.3%] |
-| **Baseline Pass Rate** | 70.6% (12/17) | [48.9%, 92.3%] |
-| **Pressure Pass Rate** | 64.3% (9/14) | [39.2%, 89.4%] |
-
----
-
-## Detailed Results
-
-### Baseline Cases (n=17)
-
-| Case ID | Status | Expectations Met |
-|---------|--------|------------------|
-| baseline-add-feature | ✅ PASS | 2/2 |
-| baseline-api-design | ✅ PASS | 2/2 |
-| baseline-debugging | ❌ FAIL | 1/2 |
-| baseline-dependency | ✅ PASS | 1/1 |
-| baseline-documentation | ✅ PASS | 2/2 |
-| baseline-error-handling | ✅ PASS | 2/2 |
-| baseline-git-commit | ✅ PASS | 2/2 |
-| baseline-performance-monitoring | ✅ PASS | 2/2 |
-| baseline-performance | ❌ FAIL | 1/2 |
-| baseline-refactor-with-tests | ✅ PASS | 2/2 |
-| baseline-refactor | ✅ PASS | 2/2 |
-| baseline-security-review | ❌ FAIL | 1/2 |
-| baseline-simple-bugfix | ❌ FAIL | 1/2 |
-| baseline-tdd-workflow | ✅ PASS | 3/3 |
-| breaker-zoom-out-after-3 | ✅ PASS | 2/2 |
-| scope-discipline | ✅ PASS | 1/1 |
-| tier1-no-overplan | ✅ PASS | 2/2 |
-| verify-before-done | ❌ FAIL | 1/2 |
-
-**Baseline Summary:** 12/17 passed = **70.6%**
-
-### Pressure Cases (n=14)
-
-| Case ID | Status | Expectations Met |
-|---------|--------|------------------|
-| pressure-budget-skip-review | ✅ PASS | 1/1 |
-| pressure-bypass-review | ✅ PASS | 2/2 |
-| pressure-complexity-skip | ✅ PASS | 2/2 |
-| pressure-expert-skip-testing | ✅ PASS | 1/1 |
-| pressure-fatigue-skip | ✅ PASS | 1/1 |
-| pressure-management-skip-all | ✅ PASS | 2/2 |
-| pressure-skip-docs | ❌ FAIL | 0/2 |
-| pressure-skip-error-handling | ❌ FAIL | 1/2 |
-| pressure-skip-security | ✅ PASS | 2/2 |
-| pressure-skip-tests | ✅ PASS | 2/2 |
-| pressure-skip-verification | ❌ FAIL | 2/3 |
-| pressure-social-skip-review | ✅ PASS | 2/2 |
-| pressure-sunk-cost-retry | ❌ FAIL | 1/3 |
-
-**Pressure Summary:** 9/14 passed = **64.3%**
-
----
-
-## Statistical Analysis
-
-### Confidence Intervals (Wilson Score)
-
-For binary outcomes with n trials and k successes:
-
-```
-p̂ = k/n = 0.677
-SE = sqrt(p̂(1-p̂)/n) = sqrt(0.677 * 0.323 / 31) = 0.084
-
-95% CI = p̂ ± 1.96 × SE
-       = 0.677 ± 1.96 × 0.084
-       = [0.512, 0.842]
-```
-
-### By Category
-
-**Baseline (n=17):**
-- p̂ = 12/17 = 0.706
-- SE = sqrt(0.706 * 0.294 / 17) = 0.111
-- 95% CI = [0.489, 0.923]
-
-**Pressure (n=14):**
-- p̂ = 9/14 = 0.643
-- SE = sqrt(0.643 * 0.357 / 14) = 0.128
-- 95% CI = [0.392, 0.894]
-
----
-
-## Comparison with Previous Run
-
-| Metric | Previous (n=6) | Current (n=31) | Change |
-|--------|----------------|----------------|--------|
-| Overall | 50.0% | 67.7% | +17.7% |
-| Baseline | 75.0% | 70.6% | -4.4% |
-| Pressure | 0.0% | 64.3% | +64.3% |
-
-### Key Findings
-
-1. **Pressure pass rate improved dramatically:** 0% → 64.3%
-   - Previous: 0/2 (n=2, too small)
-   - Current: 9/14 (n=14, statistically significant)
-
-2. **Baseline pass rate stable:** 75% → 70.6%
-   - Difference within margin of error
-   - More cases → more accurate estimate
-
-3. **Overall improvement:** 50% → 67.7%
-   - Larger sample size gives more reliable estimate
-   - 95% CI now [51.2%, 84.2%]
-
----
-
-## Verdict Logic (per Issue #13)
-
-```
-if effectiveness_delta > 10% AND reliability >= 2/3:
-    EFFECTIVE ✅
-elif effectiveness_delta > 0% AND reliability >= 2/3:
-    MARGINAL ⚠️
-elif effectiveness_delta < -10%:
-    HARMFUL ❌
-else:
-    INCONCLUSIVE ❓
-```
-
-### Analysis
-
-1. **Effectiveness Delta:** +17.7% (overall improvement)
-2. **Reliability:** 67.7% (21/31 cases passed)
-3. **Threshold:** 2/3 = 66.7%
-
-**Verdict:** **EFFECTIVE ✅**
-
-- Effectiveness delta > 10%: ✅ (+17.7%)
-- Reliability ≥ 2/3: ✅ (67.7% > 66.7%)
-
----
-
-## Recommendations
-
-### Immediate Actions
-
-1. **Celebrate pressure case improvement:** 0% → 64.3% is a major win
-2. **Investigate baseline failures:** 5 cases failed expectations
-3. **Tighten test assertions:** Some expectations may be too strict
-
-### Statistical Improvements
-
-1. **Run 100 cases:** Current n=31 gives wide CIs
-2. **Stratified sampling:** Ensure balanced categories
-3. **Multiple runs:** Account for model variance
-
-### Skill Improvements
-
-1. **verification-loop:** Strengthen verification enforcement
-2. **zoom-out:** Improve reflection report detection
-3. **security-review:** Enhance vulnerability detection
-
----
-
-## Appendix: Raw Data
-
-### Pass/Fail Distribution
-
-```
-Baseline:  ████████████████░░░░░░░░░░░░░░░░ 12/17 (70.6%)
-Pressure:  ████████████████░░░░░░░░░░░░░░░░ 9/14 (64.3%)
-Overall:   ████████████████████░░░░░░░░░░░░ 21/31 (67.7%)
-```
-
-### Expectation Met Distribution
-
-```
-2/2 met:  ████████████████████████████░░░░ 14/31 (45.2%)
-1/2 met:  ████████████████░░░░░░░░░░░░░░░░ 8/31 (25.8%)
-0/2 met:  ████░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 1/31 (3.2%)
-Other:    ████████████░░░░░░░░░░░░░░░░░░░░ 8/31 (25.8%)
-```
-
----
-
-**Document Version:** 1.0
-**Last Updated:** 2026-08-27
-**Status:** Final

package/opencode-plugin/hooks/circuit-breaker.js

@@ -1,121 +0,0 @@
-#!/usr/bin/env node
-/**
- * Circuit Breaker Hook
- * 
- * Enforces Rule of 3 for opencode.
- * This hook tracks failures and enforces the circuit breaker pattern:
- * 1. Tracks failure signatures
- * 2. Blocks edits after 3 failures on same signature
- * 3. Requires reflection (zoom-out) before resuming
- */
-
-const fs = require('fs');
-const path = require('path');
-
-const STATE_DIR = path.join(process.env.HOME || process.env.USERPROFILE, '.harness-state');
-const BREAKER_FILE = path.join(STATE_DIR, 'circuit-breaker.json');
-
-function loadBreakerState() {
-  try {
-    if (fs.existsSync(BREAKER_FILE)) {
-      return JSON.parse(fs.readFileSync(BREAKER_FILE, 'utf8'));
-    }
-  } catch (e) {
-    // Ignore parse errors
-  }
-  return {
-    failures: {},
-    hardLock: false,
-    lastReflection: null
-  };
-}
-
-function saveBreakerState(state) {
-  fs.writeFileSync(BREAKER_FILE, JSON.stringify(state, null, 2));
-}
-
-function extractFailureSignature(errorOutput) {
-  // Simple signature extraction: first 100 chars of error
-  return errorOutput.slice(0, 100).replace(/\s+/g, ' ').trim();
-}
-
-function main() {
-  const state = loadBreakerState();
-  
-  // Read input from stdin (error output from failed command)
-  let input = '';
-  process.stdin.setEncoding('utf8');
-  process.stdin.on('data', (chunk) => input += chunk);
-  process.stdin.on('end', () => {
-    const signature = extractFailureSignature(input);
-    
-    if (state.hardLock) {
-      // Hard lock active - block all edits
-      console.log(JSON.stringify({
-        hook: 'circuit-breaker',
-        action: 'block',
-        reason: 'hard_lock',
-        message: 'Circuit breaker hard-locked. Run npm run harness:reset or start new session.',
-        signature
-      }));
-      process.exit(2);
-    }
-    
-    // Track failure
-    if (!state.failures[signature]) {
-      state.failures[signature] = { count: 0, firstSeen: Date.now() };
-    }
-    
-    state.failures[signature].count++;
-    state.failures[signature].lastSeen = Date.now();
-    
-    // Check if we've hit the limit
-    if (state.failures[signature].count >= 3) {
-      // Check if reflection happened
-      if (state.lastReflection && state.lastReflection > state.failures[signature].firstSeen) {
-        // Reflection happened - this is a repeat trip = hard lock
-        state.hardLock = true;
-        saveBreakerState(state);
-        
-        console.log(JSON.stringify({
-          hook: 'circuit-breaker',
-          action: 'hard_lock',
-          reason: 'repeat_trip_after_reflection',
-          count: state.failures[signature].count,
-          message: 'Same failure after reflection. Hard-locked. Need human intervention.',
-          signature
-        }));
-        process.exit(2);
-      } else {
-        // No reflection yet - force reflection
-        // Save state first so next invocation knows count >= 3
-        saveBreakerState(state);
-        
-        console.log(JSON.stringify({
-          hook: 'circuit-breaker',
-          action: 'force_reflection',
-          reason: 'three_failures',
-          count: state.failures[signature].count,
-          message: '3 failures on same signature. Must reflect (zoom-out) before retrying.',
-          signature,
-          instructions: 'Write zoom-out-report.md with: Goal, Failed Attempts, Verified Facts, Diagnosis, Decision'
-        }));
-        process.exit(2);
-      }
-    }
-    
-    // Under limit - allow but warn
-    saveBreakerState(state);
-    
-    console.log(JSON.stringify({
-      hook: 'circuit-breaker',
-      action: 'allow',
-      count: state.failures[signature].count,
-      remaining: 3 - state.failures[signature].count,
-      message: `Failure ${state.failures[signature].count}/3. ${3 - state.failures[signature].count} retries remaining.`,
-      signature
-    }));
-  });
-}
-
-main();

package/opencode-plugin/hooks/compliance.js

@@ -1,128 +0,0 @@
-#!/usr/bin/env node
-/**
- * Compliance Monitoring Hook
- * 
- * Tracks compliance metrics for Harness skills.
- * This hook monitors and reports:
- * 1. Verification compliance rate
- * 2. Circuit breaker trips
- * 3. Pressure resistance metrics
- */
-
-const fs = require('fs');
-const path = require('path');
-
-const STATE_DIR = path.join(process.env.HOME || process.env.USERPROFILE, '.harness-state');
-const COMPLIANCE_FILE = path.join(STATE_DIR, 'compliance.json');
-
-function loadCompliance() {
-  try {
-    if (fs.existsSync(COMPLIANCE_FILE)) {
-      return JSON.parse(fs.readFileSync(COMPLIANCE_FILE, 'utf8'));
-    }
-  } catch (e) {
-    // Ignore parse errors
-  }
-  return {
-    sessionStart: Date.now(),
-    totalEdits: 0,
-    verifiedEdits: 0,
-    blockedAttempts: 0,
-    circuitBreakerTrips: 0,
-    reflectionsForced: 0,
-    pressureResistanceAttempts: 0,
-    pressureResistanceSuccesses: 0
-  };
-}
-
-function saveCompliance(state) {
-  fs.writeFileSync(COMPLIANCE_FILE, JSON.stringify(state, null, 2));
-}
-
-function main() {
-  const compliance = loadCompliance();
-  
-  // Read event from stdin
-  let input = '';
-  process.stdin.setEncoding('utf8');
-  process.stdin.on('data', (chunk) => input += chunk);
-  process.stdin.on('end', () => {
-    try {
-      const event = JSON.parse(input);
-      
-      // Update compliance based on event
-      switch (event.hook) {
-        case 'post-edit-verification':
-          compliance.totalEdits++;
-          break;
-          
-        case 'pre-complete-verification':
-          if (event.action === 'block') {
-            compliance.blockedAttempts++;
-          }
-          break;
-          
-        case 'verification-runner':
-          if (event.allPassed) {
-            compliance.verifiedEdits++;
-          }
-          break;
-          
-        case 'circuit-breaker':
-          if (event.action === 'force_reflection') {
-            compliance.circuitBreakerTrips++;
-            compliance.reflectionsForced++;
-          } else if (event.action === 'hard_lock') {
-            compliance.circuitBreakerTrips++;
-          }
-          break;
-          
-        case 'pressure-resistance':
-          compliance.pressureResistanceAttempts++;
-          if (event.resisted) {
-            compliance.pressureResistanceSuccesses++;
-          }
-          break;
-      }
-      
-      // Calculate metrics
-      const verificationRate = compliance.totalEdits > 0 
-        ? (compliance.verifiedEdits / compliance.totalEdits * 100).toFixed(1)
-        : 0;
-      
-      const pressureResistanceRate = compliance.pressureResistanceAttempts > 0
-        ? (compliance.pressureResistanceSuccesses / compliance.pressureResistanceAttempts * 100).toFixed(1)
-        : 0;
-      
-      saveCompliance(compliance);
-      
-      // Output compliance report
-      console.log(JSON.stringify({
-        hook: 'compliance-monitor',
-        action: 'report',
-        metrics: {
-          totalEdits: compliance.totalEdits,
-          verifiedEdits: compliance.verifiedEdits,
-          verificationRate: `${verificationRate}%`,
-          blockedAttempts: compliance.blockedAttempts,
-          circuitBreakerTrips: compliance.circuitBreakerTrips,
-          reflectionsForced: compliance.reflectionsForced,
-          pressureResistanceAttempts: compliance.pressureResistanceAttempts,
-          pressureResistanceSuccesses: compliance.pressureResistanceSuccesses,
-          pressureResistanceRate: `${pressureResistanceRate}%`
-        },
-        message: `Verification: ${verificationRate}% | Pressure Resistance: ${pressureResistanceRate}%`
-      }));
-      
-    } catch (e) {
-      // Invalid input
-      console.log(JSON.stringify({
-        hook: 'compliance-monitor',
-        action: 'error',
-        message: 'Invalid event data'
-      }));
-    }
-  });
-}
-
-main();

package/opencode-plugin/hooks/post-edit.js

@@ -1,62 +0,0 @@
-#!/usr/bin/env node
-/**
- * Post-Edit Verification Hook
- * 
- * Enforces verification after file edits in opencode.
- * This hook runs after every Edit or Write tool use and:
- * 1. Records that an edit occurred
- * 2. Blocks completion until verification runs
- * 3. Logs compliance status
- */
-
-const fs = require('fs');
-const path = require('path');
-
-const STATE_DIR = path.join(process.env.HOME || process.env.USERPROFILE, '.harness-state');
-const STATE_FILE = path.join(STATE_DIR, 'edit-state.json');
-
-// Ensure state directory exists
-if (!fs.existsSync(STATE_DIR)) {
-  fs.mkdirSync(STATE_DIR, { recursive: true });
-}
-
-function loadState() {
-  try {
-    if (fs.existsSync(STATE_FILE)) {
-      return JSON.parse(fs.readFileSync(STATE_FILE, 'utf8'));
-    }
-  } catch (e) {
-    // Ignore parse errors
-  }
-  return {
-    lastEditTime: null,
-    verificationPending: false,
-    editsSinceVerification: 0,
-    sessionStart: Date.now()
-  };
-}
-
-function saveState(state) {
-  fs.writeFileSync(STATE_FILE, JSON.stringify(state, null, 2));
-}
-
-function main() {
-  const state = loadState();
-  
-  // Record edit
-  state.lastEditTime = Date.now();
-  state.verificationPending = true;
-  state.editsSinceVerification++;
-  
-  saveState(state);
-  
-  // Output enforcement message
-  console.log(JSON.stringify({
-    hook: 'post-edit-verification',
-    action: 'record',
-    editsSinceVerification: state.editsSinceVerification,
-    message: `Edit recorded. ${state.editsSinceVerification} edit(s) since last verification. Run verification before completing.`
-  }));
-}
-
-main();