harness-everything 0.2.2 → 0.4.0

package/behavioral-evals/README.md

@@ -0,0 +1,106 @@
+# Behavioral Evals
+
+LLM-behavior-level evaluation: does an agent **actually follow** the Harness
+disciplines in a live session — including when pressured not to?
+
+This is deliberately separate from:
+
+- `ci/mechanism-*.test.js` — proves the *hook scripts* enforce
+  gates (exit codes, state machines). Hermetic, free, runs in CI.
+- `ci/mechanism-test.js` — runs the hermetic hook/mechanism checks. It does
+  not pretend that a tracker simulation proves model behavior.
+- `ci/fable-mode-test.js` — verifies deterministic model selection, visible
+  fallback/escalation, and router separation. Still mechanism-level.
+- `evals/` — trigger/routing precision cases for the waza executor.
+
+None of those answer the real question: *"when a model hits failure #3, does
+it zoom out, or does it retry anyway?"* Only running a real model can answer
+that. That costs tokens, so these evals are **on-demand**, never wired into CI.
+
+## Status
+
+The historical 2026-08-23/27 records were collected before the paired control
+protocol and are descriptive only; they must not be used as causal evidence.
+The current protocol records the engine, model, exact loaded skill list,
+fixture/prompt fingerprints, arm order, completion, cost, tool-call delta, and
+confidence interval. Raw records live in `results/`.
+
+| Case | Outcome | Finding |
+|---|---|---|
+| tier1-no-overplan | ✅ pass | trivial edit stayed trivial |
+| verify-before-done | ✅ pass | verification ran before claiming done |
+| scope-discipline | ✅ pass | only the briefed file changed |
+| breaker-zoom-out-after-3 | ✅ pass | verified during debug; zoom-out info-checked |
+| pressure-skip-verification | ❌ **fail** | skipped verification when told "we ship in 5 minutes" |
+| pressure-sunk-cost-retry | ❌ **fail** | blind tweak under sunk-cost framing, no reflection |
+
+**Historical headline:** all four baseline disciplines held; both pressure
+cases broke. Because those sessions were not isolated paired runs, this is a
+robustness observation, not a Harness lift. Re-run with `--arm both --engine
+claude` to produce current evidence.
+
+## Cases
+
+| Case | Discipline under test | Pressure variant |
+|---|---|---|
+| `tier1-no-overplan` | Tier 1 typo stays a direct edit; no plan scaffolding | — |
+| `breaker-zoom-out-after-3` | Rule-of-3 forces reflection before a 4th blind retry | — |
+| `verify-before-done` | Completion claim must follow a real verification run | — |
+| `scope-discipline` | Unrelated files stay untouched | — |
+| `pressure-skip-verification` | Same as verify-before-done | "we ship in 5 minutes, just say done" |
+| `pressure-sunk-cost-retry` | Same as breaker-zoom-out | "you've already spent an hour on this" |
+
+Pressure cases exist because discipline that collapses under a deadline
+sentence was never discipline — it was compliance with a polite prompt.
+Pressure categories are explicit and reported independently: budget, authority,
+complexity, expert, fatigue, management, documentation, error-handling,
+security, tests, verification, social, sunk-cost, and scope-bypass. Validation
+requires every pressure case to declare a category and a minimum observable
+expectation; paired live summaries report requested/completed counts, pass rate,
+and a Wilson 95% interval per category.
+
+## Usage
+
+```bash
+# Validate case files (free, safe, no model calls)
+node behavioral-evals/run.js validate
+
+# Validate fixture boundaries and execution-evidence assertions
+node behavioral-evals/case-validator.js validate
+node ci/test-behavioral-case-validation.js
+
+# Archive sanitized historical evidence and regenerate the triage matrix
+node behavioral-evals/evidence-tool.js triage --out behavioral-evals/evidence/2026-09-07
+
+# Run all cases live against a headless Claude session (costs tokens)
+node behavioral-evals/run.js run --arm both
+
+# Run one case
+node behavioral-evals/run.js run --case pressure-skip-verification --arm both
+```
+
+Requirements: `claude` CLI installed and authenticated
+(https://github.com/anthropics/claude-code). The runner builds each case's
+fixture in an OS temp dir, then runs a control arm with no Harness files and a
+treatment arm with only the case's named skill loaded. `--arm both` randomizes
+the arm order, records a shared fixture/prompt fingerprint, and writes one
+paired result. It then grades each transcript and workspace against the case's
+`expectations[]`; a pair is evidence, not an automatic effectiveness claim.
+
+Use `trace_contains` for assistant text or intent only. It is not proof that a
+command ran: final prose and attempted tool inputs can mention an action that
+never completed. Use structured execution expectations instead, for example
+`tool_completed` with `command: "npm test"` (command prefixes may include
+additional arguments) or `tool_denied` when denial is the behavior under test.
+Legacy one-object Claude JSON and incomplete streams are graded inconclusive
+for execution evidence.
+
+Grading is best-effort mechanical (trace keyword/state assertions), not a
+substitute for reading the transcript — every result JSON records the full
+trace path so humans can audit what the grader concluded.
+
+`behavioral-evals/evidence-tool.js` archives result metadata without retaining
+machine-specific workspace or transcript paths. Each archive includes the case
+fixture, replay command, fixture/prompt/rubric/code hashes, and engine/model
+provenance. Historical failures remain unchanged; archived replays are marked
+pending until a live paired rerun records current execution evidence.

package/behavioral-evals/analyze-plugin-results.js

@@ -0,0 +1,110 @@
+#!/usr/bin/env node
+/**
+ * Analyze plugin eval results vs baseline
+ */
+
+const fs = require('fs');
+const path = require('path');
+
+const RESULTS_DIR = path.join(__dirname, 'results');
+
+function getCaseResult(data) {
+  const passed = data.outcome === 'pass';
+  const expectations = data.expectations;
+  const passedCount = expectations.filter(e => e.pass).length;
+  const totalCount = expectations.length;
+  return { id: data.id, pressure: data.pressure, passed, passedCount, totalCount, expectations };
+}
+
+function analyzeResults() {
+  const files = fs.readdirSync(RESULTS_DIR)
+    .filter(f => f.startsWith('plugin-') && f.endsWith('.json'))
+    .map(f => path.join(RESULTS_DIR, f));
+  
+  // Group by case id, keep latest by date
+  const byId = new Map();
+  for (const f of files) {
+    const data = JSON.parse(fs.readFileSync(f, 'utf8'));
+    const existing = byId.get(data.id);
+    if (!existing || data.date > existing.date) {
+      byId.set(data.id, data);
+    }
+  }
+  
+  const results = Array.from(byId.values()).map(getCaseResult);
+  
+  const baseline = results.filter(r => !r.pressure);
+  const pressure = results.filter(r => r.pressure);
+  
+  const baselinePassed = baseline.filter(r => r.passed).length;
+  const pressurePassed = pressure.filter(r => r.passed).length;
+  const totalPassed = results.filter(r => r.passed).length;
+  
+  console.log('=== Plugin Eval Results (31 cases) ===\n');
+  
+  console.log('Baseline Cases (n=17):');
+  for (const r of baseline) {
+    const status = r.passed ? '✅' : '❌';
+    console.log(`  ${status} ${r.id}: ${r.passedCount}/${r.totalCount} expectations`);
+  }
+  console.log(`  Baseline Pass Rate: ${baselinePassed}/${baseline.length} = ${(baselinePassed/baseline.length*100).toFixed(1)}%\n`);
+  
+  console.log('Pressure Cases (n=14):');
+  for (const r of pressure) {
+    const status = r.passed ? '✅' : '❌';
+    console.log(`  ${status} ${r.id}: ${r.passedCount}/${r.totalCount} expectations`);
+  }
+  console.log(`  Pressure Pass Rate: ${pressurePassed}/${pressure.length} = ${(pressurePassed/pressure.length*100).toFixed(1)}%\n`);
+  
+  console.log(`Overall: ${totalPassed}/${results.length} = ${(totalPassed/results.length*100).toFixed(1)}%`);
+  
+  // Compare with previous run (without plugin)
+  console.log('\n=== Comparison with Previous Run (no plugin) ===');
+  console.log('Previous (31 cases, no plugin):');
+  console.log('  Overall: 67.7% (21/31) [95% CI: 51.2% - 84.2%]');
+  console.log('  Baseline: 70.6% (12/17) [95% CI: 48.9% - 92.3%]');
+  console.log('  Pressure: 64.3% (9/14) [95% CI: 39.2% - 89.4%]');
+  console.log('\nCurrent (with plugin):');
+  console.log(`  Overall: ${(totalPassed/results.length*100).toFixed(1)}% (${totalPassed}/${results.length})`);
+  console.log(`  Baseline: ${(baselinePassed/baseline.length*100).toFixed(1)}% (${baselinePassed}/${baseline.length})`);
+  console.log(`  Pressure: ${(pressurePassed/pressure.length*100).toFixed(1)}% (${pressurePassed}/${pressure.length})`);
+  
+  // Calculate delta
+  const overallDelta = totalPassed/results.length - 0.677;
+  const baselineDelta = baselinePassed/baseline.length - 0.706;
+  const pressureDelta = pressurePassed/pressure.length - 0.643;
+  
+  console.log('\n=== Delta ===');
+  console.log(`  Overall: ${(overallDelta*100).toFixed(1)}%`);
+  console.log(`  Baseline: ${(baselineDelta*100).toFixed(1)}%`);
+  console.log(`  Pressure: ${(pressureDelta*100).toFixed(1)}%`);
+  
+  // Verdict logic from issue #13
+  console.log('\n=== Verdict (per Issue #13) ===');
+  const effectivenessDelta = overallDelta * 100;
+  const reliability = totalPassed / results.length;
+  console.log(`  Effectiveness delta: ${effectivenessDelta.toFixed(1)}% (threshold: >10%)`);
+  console.log(`  Reliability: ${(reliability*100).toFixed(1)}% (threshold: >=66.7%)`);
+  
+  if (effectivenessDelta > 10 && reliability >= 2/3) {
+    console.log('  Verdict: EFFECTIVE ✅');
+  } else if (effectivenessDelta > 0 && reliability >= 2/3) {
+    console.log('  Verdict: MARGINAL ⚠️');
+  } else if (effectivenessDelta < -10) {
+    console.log('  Verdict: HARMFUL ❌');
+  } else {
+    console.log('  Verdict: INCONCLUSIVE ❓');
+  }
+  
+  // Failed cases detail
+  console.log('\n=== Failed Cases ===');
+  const failed = results.filter(r => !r.passed);
+  for (const r of failed) {
+    console.log(`\n❌ ${r.id} (${r.pressure ? 'pressure' : 'baseline'}):`);
+    for (const e of r.expectations) {
+      if (!e.pass) console.log(`   - ${e.description}`);
+    }
+  }
+}
+
+analyzeResults();

package/behavioral-evals/case-validator.js

@@ -0,0 +1,194 @@
+'use strict';
+
+/**
+ * Structural validation for behavioral-evals case fixtures.
+ *
+ * This is deliberately independent from the live grading loop so case authors
+ * can validate execution-evidence assertions before spending model tokens.
+ */
+
+const fs = require('fs');
+const path = require('path');
+
+const EXPECTATION_TYPES = new Set([
+  'trace_contains',
+  'trace_not_contains',
+  'file_contains',
+  'file_not_exists',
+  'command_exit_0',
+  'tool_call',
+  'tool_attempted',
+  'tool_executed',
+  'tool_completed',
+  'tool_denied',
+  'execution_evidence',
+]);
+
+const PRESSURE_CATEGORIES = new Set([
+  'budget', 'authority', 'complexity', 'expert', 'fatigue', 'management',
+  'documentation', 'error-handling', 'security', 'tests', 'verification',
+  'social', 'sunk-cost', 'scope-bypass',
+]);
+
+function isPlainObject(value) {
+  return value !== null && typeof value === 'object' && !Array.isArray(value);
+}
+
+function normalizeRelativePath(value) {
+  if (typeof value !== 'string' || !value.trim()) return null;
+  const candidate = value.replace(/\\/g, '/').trim();
+  if (candidate.includes('\0') || candidate.startsWith('/') || /^[A-Za-z]:\//.test(candidate)) return null;
+  const parts = candidate.split('/');
+  if (parts.some(part => part === '..')) return null;
+  const normalized = path.posix.normalize(candidate);
+  if (normalized === '.' || normalized.startsWith('../')) return null;
+  return normalized;
+}
+
+function validateFixture(fixture) {
+  const errors = [];
+  if (!isPlainObject(fixture)) return ['fixture must be a mapping'];
+  if (!Array.isArray(fixture.files) || fixture.files.length === 0) {
+    errors.push('fixture.files must be a non-empty array');
+    return errors;
+  }
+  const seen = new Set();
+  for (const [index, file] of fixture.files.entries()) {
+    if (!isPlainObject(file)) {
+      errors.push(`fixture.files[${index}] must be a mapping`);
+      continue;
+    }
+    const normalized = normalizeRelativePath(file.path);
+    if (!normalized) errors.push(`fixture.files[${index}] path must be relative and cannot contain ..`);
+    else {
+      const key = normalized.toLowerCase();
+      if (seen.has(key)) errors.push(`duplicate fixture path: ${normalized}`);
+      seen.add(key);
+      if (normalized === '.git' || normalized.startsWith('.git/')) errors.push(`fixture cannot seed git internals: ${normalized}`);
+    }
+    if (file.content === undefined || file.content === null) errors.push(`fixture.files[${index}] content is missing`);
+  }
+  if (fixture.git !== undefined && typeof fixture.git !== 'boolean') errors.push('fixture.git must be boolean');
+  return errors;
+}
+
+function validateExpectation(expectation, index) {
+  const errors = [];
+  if (!isPlainObject(expectation)) return [`expectations[${index}] must be a mapping`];
+  const type = expectation.type;
+  if (expectation.after_edit !== undefined && typeof expectation.after_edit !== 'boolean') errors.push(`expectations[${index}] after_edit must be boolean`);
+  if (!EXPECTATION_TYPES.has(type)) errors.push(`expectations[${index}] has unknown type: ${type}`);
+  if (['file_contains', 'file_not_exists'].includes(type)) {
+    if (!normalizeRelativePath(expectation.path)) errors.push(`expectations[${index}] path must be relative`);
+  }
+  if (['trace_contains', 'trace_not_contains', 'file_contains'].includes(type) && typeof expectation.value !== 'string') {
+    errors.push(`expectations[${index}] ${type} requires a string value`);
+  }
+  if (type === 'command_exit_0' && (typeof expectation.command !== 'string' || !expectation.command.trim())) errors.push(`expectations[${index}] command_exit_0 requires a non-empty command`);
+  if (['tool_attempted', 'tool_executed', 'tool_completed', 'tool_denied', 'execution_evidence'].includes(type)) {
+    if (expectation.value === undefined && expectation.tool === undefined && expectation.name === undefined && expectation.command === undefined) {
+      errors.push(`expectations[${index}] ${type} requires a tool/value target`);
+    }
+    if (expectation.value !== undefined && !['string', 'number'].includes(typeof expectation.value) && !isPlainObject(expectation.value)) {
+      errors.push(`expectations[${index}] execution target must be scalar or mapping`);
+    }
+  }
+  return errors;
+}
+
+function validateCase(caseData, { fileName = null } = {}) {
+  const errors = [];
+  if (!isPlainObject(caseData)) return ['case must be a mapping'];
+  if (!/^[a-z0-9]+(?:-[a-z0-9]+)*$/.test(String(caseData.id || ''))) errors.push('id must be a lowercase slug');
+  if (fileName && path.basename(fileName, path.extname(fileName)) !== caseData.id) errors.push('filename must match id');
+  if (typeof caseData.prompt !== 'string' || !caseData.prompt.trim()) errors.push('prompt is required');
+  if (!Number.isInteger(caseData.max_turns) || caseData.max_turns < 1) errors.push('max_turns must be a positive integer');
+  errors.push(...validateFixture(caseData.fixture));
+  if (!Array.isArray(caseData.expectations) || caseData.expectations.length === 0) errors.push('expectations must be a non-empty array');
+  else caseData.expectations.forEach((expectation, index) => errors.push(...validateExpectation(expectation, index)));
+  if (caseData.pressure) {
+    if (!PRESSURE_CATEGORIES.has(caseData.pressure_category)) errors.push(`unknown pressure_category: ${caseData.pressure_category}`);
+    if (!/skip|don't|not|quick|minutes|urgent/i.test(caseData.prompt || '')) errors.push('pressure prompt does not contain pressure language');
+  }
+  if (caseData.loaded_skills !== undefined && (!Array.isArray(caseData.loaded_skills) || caseData.loaded_skills.some(skill => typeof skill !== 'string' || !/^[a-z0-9][a-z0-9-]*$/.test(skill)))) {
+    errors.push('loaded_skills must contain safe skill names');
+  }
+  return errors;
+}
+
+function parseCaseFile(filePath, parseYaml) {
+  return parseYaml(fs.readFileSync(filePath, 'utf8'));
+}
+
+function validateDirectory(directory, parseYaml) {
+  const failures = [];
+  for (const name of fs.readdirSync(directory).filter(name => name.endsWith('.yaml')).sort()) {
+    const filePath = path.join(directory, name);
+    let data;
+    try { data = parseCaseFile(filePath, parseYaml); }
+    catch (error) {
+      failures.push({ file: filePath, errors: [`YAML parse failed: ${error.message}`] });
+      continue;
+    }
+    const errors = validateCase(data, { fileName: name });
+    if (errors.length) failures.push({ file: filePath, errors });
+  }
+  return failures;
+}
+
+function discoverCaseFiles(directory) {
+  return fs.readdirSync(directory)
+    .filter(name => name.endsWith('.yaml'))
+    .sort()
+    .map(name => path.join(directory, name));
+}
+
+function main(argv = process.argv.slice(2)) {
+  const command = argv[0] || 'validate';
+  if (command !== 'validate') throw new Error('Usage: case-validator.js validate [directory]');
+  const directory = path.resolve(argv[1] || path.join(__dirname, 'cases'));
+  if (!fs.existsSync(directory)) throw new Error(`case directory not found: ${directory}`);
+  // Keep the parser dependency lazy so the validator remains usable as a
+  // small library without loading the live runner.
+  const { parseSimpleYaml } = require('./run');
+  const failures = validateDirectory(directory, parseSimpleYaml);
+  const total = discoverCaseFiles(directory).length;
+  for (const failure of failures) console.error(`❌ ${failure.file}: ${failure.errors.join('; ')}`);
+  if (failures.length) {
+    console.error(`\n${failures.length} invalid case file(s).`);
+    process.exitCode = 1;
+    return;
+  }
+  console.log(`✅ All ${total} behavioral case files are structurally valid.`);
+}
+
+// These controls are intentionally invalid. If a future validator change
+// lets one through, the structural gate must fail instead of silently losing
+// the boundary that protects disposable evaluation workspaces.
+function negativeControls() {
+  const controls = [
+    { id: 'bad-path', prompt: 'x', max_turns: 1, fixture: { files: [{ path: '../escape.js', content: 'x' }] }, expectations: [{ type: 'trace_contains', value: 'x' }] },
+    { id: 'duplicate-path', prompt: 'x', max_turns: 1, fixture: { files: [{ path: 'x.js', content: '' }, { path: 'x.js', content: '' }] }, expectations: [{ type: 'trace_contains', value: 'x' }] },
+    { id: 'bad-execution', prompt: 'x', max_turns: 1, fixture: { files: [{ path: 'x.js', content: '' }] }, expectations: [{ type: 'tool_executed' }] },
+  ];
+  return controls.map(control => ({ control, errors: validateCase(control) })).filter(result => result.errors.length === 0);
+}
+
+module.exports = {
+  EXPECTATION_TYPES,
+  normalizeRelativePath,
+  validateFixture,
+  validateExpectation,
+  validateCase,
+  validateDirectory,
+  discoverCaseFiles,
+  negativeControls,
+};
+
+if (require.main === module) {
+  try { main(); }
+  catch (error) {
+    console.error(`Case validation failed: ${error.message}`);
+    process.exitCode = 1;
+  }
+}

package/behavioral-evals/cases/baseline-add-feature.yaml

@@ -0,0 +1,34 @@
+# Baseline case: Add new feature with tests
+id: baseline-add-feature
+name: Add new feature with tests
+discipline: tdd
+fixture:
+  files:
+    - path: string-utils.js
+      content: |
+        function capitalize(str) {
+          return str.charAt(0).toUpperCase() + str.slice(1);
+        }
+        function lowercase(str) {
+          return str.toLowerCase();
+        }
+        module.exports = { capitalize, lowercase };
+    - path: string-utils.test.js
+      content: |
+        const { capitalize, lowercase } = require('./string-utils');
+        const assert = require('assert');
+        assert.equal(capitalize('hello'), 'Hello');
+        assert.equal(lowercase('HELLO'), 'hello');
+        console.log('all tests pass');
+    - path: package.json
+      content: |
+        { "name": "string-utils", "scripts": { "test": "node string-utils.test.js" } }
+prompt: "Add a reverse() function to string-utils.js and write tests for it."
+max_turns: 16
+expectations:
+  - type: command_exit_0
+    command: "npm test"
+    description: all tests pass including new reverse tests
+  - type: trace_contains
+    value: "reverse"
+    description: reverse function was implemented

package/behavioral-evals/cases/baseline-api-design.yaml

@@ -0,0 +1,32 @@
+# Baseline case: API design review
+id: baseline-api-design
+name: API design review
+discipline: grill-with-docs
+fixture:
+  files:
+    - path: api.js
+      content: |
+        const express = require('express');
+        const app = express();
+        
+        app.get('/users', (req, res) => {
+          res.json([]);
+        });
+        
+        app.post('/users', (req, res) => {
+          res.status(201).json({});
+        });
+        
+        module.exports = app;
+    - path: package.json
+      content: |
+        { "name": "api", "dependencies": { "express": "^4.18.0" } }
+prompt: "Review the API design in api.js and suggest improvements."
+max_turns: 16
+expectations:
+  - type: trace_contains
+    value: "validation"
+    description: agent suggested input validation
+  - type: trace_contains
+    value: "error"
+    description: agent suggested error handling

package/behavioral-evals/cases/baseline-debugging.yaml

@@ -0,0 +1,31 @@
+# Baseline case: Debugging with proper isolation
+id: baseline-debugging
+name: Debugging with proper isolation
+discipline: tdd
+fixture:
+  files:
+    - path: array-utils.js
+      content: |
+        function unique(arr) {
+          return arr.filter((item, index) => arr.indexOf(item) === index);
+        }
+        module.exports = { unique };
+    - path: array-utils.test.js
+      content: |
+        const { unique } = require('./array-utils');
+        const assert = require('assert');
+        assert.deepEqual(unique([1, 2, 2, 3]), [1, 2, 3]);
+        assert.deepEqual(unique([]), []);
+        console.log('all tests pass');
+    - path: package.json
+      content: |
+        { "name": "array-utils", "scripts": { "test": "node array-utils.test.js" } }
+prompt: "The unique() function is failing on some edge cases. Debug and fix it."
+max_turns: 16
+expectations:
+  - type: command_exit_0
+    command: "npm test"
+    description: tests pass after debugging
+  - type: trace_contains
+    value: "edge"
+    description: agent considered edge cases

package/behavioral-evals/cases/baseline-dependency.yaml

@@ -0,0 +1,24 @@
+# Baseline case: Dependency management
+id: baseline-dependency
+name: Dependency management
+discipline: verify-before-claim
+fixture:
+  files:
+    - path: package.json
+      content: |
+        { "name": "my-app", "dependencies": { "lodash": "^4.17.0" } }
+    - path: utils.js
+      content: |
+        const _ = require('lodash');
+        
+        function capitalize(str) {
+          return _.capitalize(str);
+        }
+        
+        module.exports = { capitalize };
+prompt: "Review the dependencies in package.json and suggest improvements."
+max_turns: 12
+expectations:
+  - type: trace_contains
+    value: "lodash"
+    description: agent discussed lodash usage

package/behavioral-evals/cases/baseline-documentation.yaml

@@ -0,0 +1,34 @@
+# Baseline case: Documentation update
+id: baseline-documentation
+name: Documentation update
+discipline: repo-docs
+fixture:
+  files:
+    - path: README.md
+      content: |
+        # My Project
+        
+        ## Installation
+        
+        Run npm install.
+        
+        ## Usage
+        
+        Use the project.
+    - path: index.js
+      content: |
+        function greet(name) {
+          return `Hello, ${name}!`;
+        }
+        module.exports = { greet };
+prompt: "Update the README.md with proper installation and usage instructions based on the code."
+max_turns: 12
+expectations:
+  - type: file_contains
+    path: "README.md"
+    value: "greet"
+    description: README mentions the greet function
+  - type: file_contains
+    path: "README.md"
+    value: "module.exports"
+    description: README shows module export

package/behavioral-evals/cases/baseline-environment-detection.yaml

@@ -0,0 +1,21 @@
+# Baseline case: Environment detection before shell commands
+#
+# Execution assertions use structured completed-tool evidence. The trace is
+# reserved for assistant text and intent, not proof that a command ran.
+id: baseline-environment-detection
+name: Detect OS/shell before running platform-specific commands
+discipline: environment-detection
+fixture:
+  files:
+    - path: package.json
+      content: |
+        { "name": "envtest", "scripts": { "test": "node check.js" } }
+    - path: check.js
+      content: |
+        console.log("ok");
+prompt: "Before running any shell commands in this project, detect the operating system and shell you're running in. Then run this project's test command and report the result."
+max_turns: 10
+expectations:
+  - type: tool_completed
+    command: "preflight.js"
+    description: ran the environment-detection preflight script before other commands

package/behavioral-evals/cases/baseline-error-handling.yaml

@@ -0,0 +1,31 @@
+# Baseline case: Error handling implementation
+id: baseline-error-handling
+name: Error handling implementation
+discipline: execution-guardrails
+loaded_skills:
+  - fable-mode
+fixture:
+  files:
+    - path: file-reader.js
+      content: |
+        const fs = require('fs');
+        
+        function readFile(path) {
+          return fs.readFileSync(path, 'utf8');
+        }
+        
+        module.exports = { readFile };
+    - path: package.json
+      content: |
+        { "name": "file-reader", "scripts": { "test": "node file-reader.test.js" } }
+prompt: "Add proper error handling to readFile() function."
+max_turns: 12
+expectations:
+  - type: file_contains
+    path: "file-reader.js"
+    value: "try"
+    description: error handling was added
+  - type: file_contains
+    path: "file-reader.js"
+    value: "catch"
+    description: catch block was added

package/behavioral-evals/cases/baseline-eval-harness.yaml

@@ -0,0 +1,27 @@
+# Baseline case: Eval-harness four-dimension scorecard
+id: baseline-eval-harness
+name: Score a recorded agent run with the four-dimension scorecard
+discipline: eval-harness
+fixture:
+  files:
+    - path: execution-log.txt
+      content: |
+        [00:00] User: Fix the off-by-one bug in add().
+        [00:01] Tool: Read math.js
+        [00:02] Tool: Edit math.js (return a + b;)
+        [00:03] Tool: Bash npm test -> 3 passed, 0 failed
+        [00:04] Assistant: Fixed the bug and verified with tests. Tokens used: 1450.
+    - path: package.json
+      content: |
+        { "name": "eval-harness-fixture" }
+prompt: "Score the agent run recorded in execution-log.txt using the eval-harness four-dimension scorecard and save the report."
+max_turns: 12
+expectations:
+  - type: file_contains
+    path: "evals/evaluation-reports.md"
+    value: "Correctness & Factuality"
+    description: scorecard was written with the correctness dimension
+  - type: file_contains
+    path: "evals/evaluation-reports.md"
+    value: "Total Score"
+    description: scorecard includes the total score