agent-eval-harness 0.1.0

package/CHANGELOG.md

@@ -0,0 +1,41 @@
+# Changelog
+
+All notable changes to `agent-eval-harness` documented here.
+Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
+Version policy: [SemVer](https://semver.org/) — major for breaking config/CLI changes, minor for new suites or check types, patch for bug fixes and tightened audits.
+
+## [0.1.0] — 2026-05-09
+
+Initial extraction from the in-tree harness at `~/.claude/agents/_evals/`.
+
+### Added
+
+- Library API at `index.js` — `loadConfig`, `loadAgents`, `loadCases`, `staticSuite`, `schemaSuite`, `fenceAudit`, `routingSuite`, `overlapAudit`, `spawnSuite`, `extractJson`, plus `text` helpers (stem, tokenize, jaccard, IDF, weighted-recall).
+- CLI binary at `cli.js` — `agent-eval [--config=<path>] [--threshold=<n>] [--strict] [--static|--schema|--routing|--spawn|--all] [--verbose] [--json] [--init]`.
+- Config loader at `lib/config.js` — JSON config with env-var override (`AGENT_EVAL_CONFIG`), defaults baked in, paths resolved relative to config file.
+- `--init` scaffolder copying a working sample project (1 agent, 1 schema, 2 routing cases, 1 fixture) to cwd. Sample passes `--threshold=1.0`.
+- README.md with quick-start, suite documentation, schema/case/fixture file shapes, and exit-code table.
+
+### Suites
+
+- **Static** (8 checks/agent): name, description length, tools count, tools whitelist, filename match, trigger phrasing, scope discipline, no recursion.
+- **Schema** (1 check/agent + fence audit): return contract section + JSON shape declaration; flags ` ```json ` fences inside Return Contract sections (provoke mimicry on some LLMs).
+- **Routing** (N cases + overlap audit): IDF-weighted recall ranks the right agent first with positive margin; flags description pairs ≥0.20 Jaccard.
+- **Spawn** (M schema-bound agents): fixture parses as JSON, required fields present, types match, enums valid.
+
+### Strict mode
+
+`--strict` and `--threshold>=1.0` promote informational fence/overlap/no-fixture audits to blocking score-affecting checks.
+
+### Known limitations
+
+- Spawn suite is a recorded-fixture contract test, not a live spawn — Node can't invoke the Claude Code Task tool from a script. Fixtures must be captured manually (or by a separate spawn pipeline) and dropped into `fixtures/<name>.txt`.
+- Stemmer is English-only. Non-English agent descriptions need a custom tokenizer.
+- IDF is computed over agent descriptions, not a real corpus — small-N effects above ~30 agents may shift token weights.
+
+## [Unreleased]
+
+Planned:
+- Live-spawn integration so the spawn suite can regenerate fixtures itself when an `--update-fixtures` flag is set (requires harness-side runner — out of scope for v0.1).
+- Pluggable tokenizer for non-English agent definitions.
+- Per-suite threshold (e.g. accept 95% routing + 100% static).

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Yorkis Estevez
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,95 @@
+# agent-eval-harness
+
+Static + schema + routing + spawn-fixture eval harness for `*.md` subagent definitions.
+
+Drop a directory of agents (Claude Code subagents, or any markdown-frontmatter agents) into a project, point the harness at it, and get a 100-point lint that catches description bloat, scope drift, fence-mimicry traps, low routing margin, and schema regressions before they ship.
+
+## Quick start
+
+```bash
+# In a fresh directory
+node /path/to/agent-eval-harness/cli.js --init
+node /path/to/agent-eval-harness/cli.js --threshold=1.0
+```
+
+`--init` scaffolds `agents/`, `_evals/`, and an `agent-eval.config.json`. The sample agent passes 100% out of the box — copy its shape for your own.
+
+## What it checks
+
+| Suite | Per-agent | What it catches |
+|---|---|---|
+| **static** | 8 checks | missing name/description, too many tools, invalid tool names, file-name mismatch, missing trigger ("use when..."), no Scope/Hard-rules section, agent calls itself recursively |
+| **schema** | 1 check + fence audit | no Return Contract section, no JSON shape declared, ` ```json ` fence inside Return Contract (provokes mimicry in some LLMs) |
+| **routing** | N cases + overlap audit | wrong agent ranks first for a given prompt, zero margin between top two, two descriptions overlap ≥0.20 Jaccard |
+| **spawn** | M schema-bound agents | fixture in `fixtures/<name>.txt` doesn't parse as JSON, missing required fields, type mismatches, enum violations |
+
+Strict mode (`--strict` or `--threshold=1.0`) promotes the fence audit, overlap audit, and missing-fixture from informational to blocking.
+
+## Config
+
+`agent-eval.config.json` — paths and thresholds. Resolved relative to the config file's directory:
+
+```json
+{
+  "agentSourceDir": "./agents",
+  "fixturesDir": "./_evals/fixtures",
+  "schemasFile": "./_evals/schemas.json",
+  "casesFile": "./_evals/cases.jsonl",
+  "validTools": ["Read", "Write", "Edit", "Bash", "Grep", "Glob", "WebSearch", "WebFetch", "NotebookEdit", "Task"],
+  "minDescriptionChars": 40,
+  "maxTools": 5,
+  "defaultThreshold": 0.85
+}
+```
+
+Override paths via `--config=path/to/cfg.json` or `AGENT_EVAL_CONFIG` env var.
+
+## Library use
+
+```js
+const { loadConfig, loadAgents, staticSuite, schemaSuite, routingSuite, spawnSuite } = require('agent-eval-harness');
+
+const config = loadConfig({ configPath: './agent-eval.config.json' });
+const agents = loadAgents(config.agentSourceDir);
+const results = staticSuite(agents, config);
+// ... render however you want
+```
+
+## Schema file shape
+
+```json
+{
+  "<agent-name>": {
+    "required": ["field1", "field2"],
+    "types": { "field1": "string", "field2": "number" },
+    "enums": { "field1": ["ok", "error"] },
+    "nested": {
+      "field2": { "required": ["sub1"], "types": { "sub1": "string" } }
+    }
+  }
+}
+```
+
+## Cases file shape (cases.jsonl)
+
+One JSON object per line:
+
+```jsonl
+{"id":"case-1","prompt":"some user prompt","expect_agent":"agent-name"}
+```
+
+## Fixture file shape
+
+`fixtures/<agent-name>.txt` — a real recorded response from spawning the agent. Can include surrounding prose or fences; the harness extracts the JSON. Aim for one fixture per schema-bound agent.
+
+## Exit codes
+
+| Code | Meaning |
+|---|---|
+| 0 | Score ≥ threshold |
+| 1 | Runtime error (config missing, file not found) |
+| 2 | Score below threshold |
+
+## License
+
+MIT.

package/cli.js

@@ -0,0 +1,212 @@
+#!/usr/bin/env node
+'use strict';
+
+// agent-eval CLI.
+//
+//   agent-eval                              # all suites against ./agent-eval.config.json
+//   agent-eval --config=path/to/cfg.json
+//   agent-eval --static                     # only the static suite
+//   agent-eval --routing                    # only routing
+//   agent-eval --schema                     # only schema (+ fence audit)
+//   agent-eval --spawn                      # only spawn-fixture
+//   agent-eval --threshold=1.0              # auto-on --strict at 1.0
+//   agent-eval --strict                     # promote informational audits to blocking
+//   agent-eval --verbose
+//   agent-eval --json                       # also emit JSON failure dump
+//   agent-eval --init                       # scaffold a sample project in cwd
+//
+// Exit 0 if score >= threshold (default 0.85), else exit 2.
+
+const fs = require('node:fs');
+const path = require('node:path');
+const {
+  loadConfig, loadAgents, loadCases,
+  staticSuite, schemaSuite, fenceAudit,
+  routingSuite, overlapAudit,
+  spawnSuite,
+} = require('./index');
+
+function parseArgs(argv) {
+  const args = argv.slice(2);
+  const opts = {
+    configPath: null,
+    threshold: null,
+    verbose: args.includes('--verbose'),
+    json: args.includes('--json'),
+    strict: args.includes('--strict'),
+    init: args.includes('--init'),
+    suites: [],
+  };
+  for (const a of args) {
+    if (a.startsWith('--config=')) opts.configPath = a.slice('--config='.length);
+    else if (a.startsWith('--threshold=')) opts.threshold = parseFloat(a.slice('--threshold='.length));
+    else if (a === '--static') opts.suites.push('static');
+    else if (a === '--schema') opts.suites.push('schema');
+    else if (a === '--routing') opts.suites.push('routing');
+    else if (a === '--spawn') opts.suites.push('spawn');
+    else if (a === '--all') opts.suites = ['static', 'schema', 'routing', 'spawn'];
+    else if (a === '--help' || a === '-h') opts.help = true;
+  }
+  if (opts.suites.length === 0) opts.suites = ['static', 'schema', 'routing', 'spawn'];
+  return opts;
+}
+
+function help() {
+  process.stdout.write([
+    'agent-eval — static + schema + routing + spawn-fixture eval for *.md subagents',
+    '',
+    'Usage:',
+    '  agent-eval                       # all suites',
+    '  agent-eval --config=path.json    # explicit config file',
+    '  agent-eval --static              # one suite',
+    '  agent-eval --threshold=1.0       # auto-strict at 1.0',
+    '  agent-eval --init                # scaffold a sample project',
+    '',
+    'Config: looks for ./agent-eval.config.json by default.',
+    'Exit codes: 0=score>=threshold, 2=below threshold, 1=runtime error',
+    '',
+  ].join('\n'));
+}
+
+function runInit(cwd = process.cwd()) {
+  const src = path.join(__dirname, 'init');
+  const targets = ['agent-eval.config.json', 'agents/sample-agent.md', '_evals/cases.jsonl', '_evals/schemas.json', '_evals/fixtures/sample-agent.txt'];
+  for (const rel of targets) {
+    const from = path.join(src, rel);
+    const to = path.join(cwd, rel);
+    if (fs.existsSync(to)) {
+      console.log(`  exists  ${rel} (skipped)`);
+      continue;
+    }
+    fs.mkdirSync(path.dirname(to), { recursive: true });
+    fs.copyFileSync(from, to);
+    console.log(`  created ${rel}`);
+  }
+  console.log('\nNext: run `agent-eval --threshold=1.0` to verify the sample passes.');
+}
+
+function main() {
+  const opts = parseArgs(process.argv);
+  if (opts.help) return help();
+  if (opts.init) return runInit();
+
+  const config = loadConfig({ configPath: opts.configPath });
+  const threshold = opts.threshold ?? config.defaultThreshold ?? 0.85;
+  const strict = opts.strict || threshold >= 1.0;
+  const verbose = opts.verbose;
+
+  if (config.__configPath) {
+    console.log(`config: ${path.relative(process.cwd(), config.__configPath)}`);
+  } else {
+    console.log('config: (defaults — no agent-eval.config.json found)');
+  }
+
+  const agents = loadAgents(config.agentSourceDir);
+  const cases = loadCases(config.casesFile);
+
+  let totalChecks = 0, totalPass = 0;
+  const failures = [];
+
+  if (opts.suites.includes('static')) {
+    console.log(`\n=== Static suite (${agents.length} agents) ===`);
+    const out = staticSuite(agents, config);
+    for (const r of out) {
+      const passes = r.checks.filter(c => c.pass).length;
+      const total = r.checks.length;
+      totalChecks += total;
+      totalPass += passes;
+      const fails = r.checks.filter(c => !c.pass);
+      if (fails.length === 0) {
+        if (verbose) console.log(`  ✓ ${r.agent} (${passes}/${total})`);
+      } else {
+        console.log(`  ✗ ${r.agent} (${passes}/${total}) — ${fails.map(c => c.name + (c.detail ? ':' + c.detail : '')).join(', ')}`);
+        failures.push({ suite: 'static', agent: r.agent, fails });
+      }
+    }
+  }
+
+  if (opts.suites.includes('schema')) {
+    console.log(`\n=== Schema suite (${agents.length} agents) ===`);
+    const out = schemaSuite(agents);
+    for (const r of out) {
+      totalChecks++;
+      if (r.pass) {
+        totalPass++;
+        if (verbose) console.log(`  ✓ ${r.agent}`);
+      } else {
+        console.log(`  ✗ ${r.agent} — ${r.detail}`);
+        failures.push({ suite: 'schema', agent: r.agent, detail: r.detail });
+      }
+    }
+    const fencers = fenceAudit(agents);
+    if (fencers.length) {
+      const tag = strict ? 'BLOCKING' : 'informational';
+      console.log(`\n=== Fence audit (${tag} — provokes \`\`\`json mimicry) ===`);
+      for (const name of fencers) console.log(`  ${strict ? '✗' : '⚠'} ${name} uses a \`\`\`json fence in schema example`);
+      if (strict) {
+        totalChecks += fencers.length;
+        for (const name of fencers) failures.push({ suite: 'fence', agent: name });
+      }
+    }
+  }
+
+  if (opts.suites.includes('routing')) {
+    console.log(`\n=== Routing suite (${cases.length} cases) ===`);
+    const out = routingSuite(agents, cases);
+    for (const r of out) {
+      totalChecks++;
+      if (r.pass) {
+        totalPass++;
+        if (verbose) console.log(`  ✓ ${r.id}: ${r.got} (margin ${r.margin.toFixed(3)})`);
+      } else {
+        const reason = r.expected !== r.got
+          ? `wrong agent (got ${r.got}, wanted ${r.expected})`
+          : `low margin ${r.margin.toFixed(3)} vs ${r.runner_up}`;
+        console.log(`  ✗ ${r.id} — ${reason}`);
+        failures.push({ suite: 'routing', id: r.id, expected: r.expected, got: r.got, margin: r.margin });
+      }
+    }
+    const pairs = overlapAudit(agents);
+    const overlapTag = strict ? 'BLOCKING' : 'informational';
+    console.log(`\n=== Description overlap (${overlapTag} — pairs ≥0.20 Jaccard) ===`);
+    if (pairs.length === 0) console.log('  no high-overlap pairs');
+    else for (const p of pairs) console.log(`  ${strict ? '✗' : '⚠'} ${p.a} ↔ ${p.b} = ${p.score.toFixed(3)}`);
+    if (strict && pairs.length) {
+      totalChecks += pairs.length;
+      for (const p of pairs) failures.push({ suite: 'overlap', a: p.a, b: p.b, score: p.score });
+    }
+  }
+
+  if (opts.suites.includes('spawn')) {
+    const sp = spawnSuite(config, { strict });
+    console.log(`\n=== Spawn suite (${sp.coveredAgents}/${Object.keys(require(config.schemasFile)).length} agents covered) ===`);
+    for (const r of sp.results) {
+      if (r.pass) {
+        if (verbose) console.log(`  ✓ ${r.agent}`);
+      } else {
+        const detail = r.reason === 'schema' ? r.errors.join('; ')
+                     : r.reason === 'extract-json' ? r.detail
+                     : r.reason;
+        console.log(`  ✗ ${r.agent} — ${detail}`);
+        failures.push({ suite: 'spawn', agent: r.agent, reason: r.reason, detail });
+      }
+    }
+    if (sp.noFixture.length && !strict) {
+      console.log(`  (no fixture: ${sp.noFixture.join(', ')})`);
+    }
+    totalChecks += sp.totalChecks;
+    totalPass += sp.totalPass;
+  }
+
+  const score = totalChecks ? totalPass / totalChecks : 0;
+  console.log(`\n=== Score: ${totalPass}/${totalChecks} = ${(score * 100).toFixed(1)}% ===`);
+  console.log(`Threshold: ${(threshold * 100).toFixed(0)}%`);
+
+  if (failures.length && opts.json) {
+    console.log('\n' + JSON.stringify({ score, totalPass, totalChecks, failures }, null, 2));
+  }
+
+  process.exit(score >= threshold ? 0 : 2);
+}
+
+if (require.main === module) main();

package/index.js

@@ -0,0 +1,34 @@
+'use strict';
+
+// Public API for agent-eval-harness.
+// Library callers (e.g. a thin run.js wrapper in your own repo) can import
+// suites, supply a config, and render the results however they like.
+
+const { loadConfig, DEFAULTS, DEFAULT_VALID_TOOLS } = require('./lib/config');
+const { parseAgent, loadAgents, loadCases } = require('./lib/agent-loader');
+const { extractJson } = require('./lib/extract-json');
+const text = require('./lib/text');
+const { staticSuite } = require('./suites/static');
+const { schemaSuite, fenceAudit } = require('./suites/schema');
+const { routingSuite, overlapAudit } = require('./suites/routing');
+const { spawnSuite, validateAgainst, loadSchemas, loadFixture } = require('./suites/spawn');
+
+module.exports = {
+  loadConfig,
+  DEFAULTS,
+  DEFAULT_VALID_TOOLS,
+  parseAgent,
+  loadAgents,
+  loadCases,
+  extractJson,
+  text,
+  staticSuite,
+  schemaSuite,
+  fenceAudit,
+  routingSuite,
+  overlapAudit,
+  spawnSuite,
+  validateAgainst,
+  loadSchemas,
+  loadFixture,
+};

package/init/_evals/cases.jsonl

@@ -0,0 +1,2 @@
+{"id":"sample-1","prompt":"count the widgets in this inventory dump","expect_agent":"sample-agent"}
+{"id":"sample-2","prompt":"summarize widget counts by type from the dump file","expect_agent":"sample-agent"}

package/init/_evals/fixtures/sample-agent.txt

@@ -0,0 +1,3 @@
+```json
+{"result":"ok","widget_count":42,"by_type":{"red":10,"blue":20,"green":12},"summary":"42 widgets across 3 types from inventory.dump"}
+```

package/init/_evals/schemas.json

@@ -0,0 +1,14 @@
+{
+  "sample-agent": {
+    "required": ["result", "widget_count", "by_type", "summary"],
+    "types": {
+      "result": "string",
+      "widget_count": "number",
+      "by_type": "object",
+      "summary": "string"
+    },
+    "enums": {
+      "result": ["ok", "rejected"]
+    }
+  }
+}

package/init/agent-eval.config.json

@@ -0,0 +1,10 @@
+{
+  "agentSourceDir": "./agents",
+  "fixturesDir": "./_evals/fixtures",
+  "schemasFile": "./_evals/schemas.json",
+  "casesFile": "./_evals/cases.jsonl",
+  "validTools": ["Read", "Write", "Edit", "Bash", "Grep", "Glob", "WebSearch", "WebFetch", "NotebookEdit", "Task"],
+  "minDescriptionChars": 40,
+  "maxTools": 5,
+  "defaultThreshold": 0.85
+}

package/init/agents/sample-agent.md

@@ -0,0 +1,30 @@
+---
+name: sample-agent
+description: Counts widgets in a hypothetical inventory dump and returns a JSON summary. Use when you need a quick, deterministic widget-count report for a small inventory dump (under 1000 lines). Not for live inventory queries (no DB access) and not for forecasting.
+tools: Read, Grep
+---
+
+You are sample-agent. Given an inventory dump on stdin or in a file, count widgets by type and return a structured JSON summary.
+
+## Scope
+
+- IN: counting widgets in a static dump file under 1000 lines
+- OUT: live inventory queries, forecasting, sales prediction, anything requiring writes
+
+## Hard rules
+
+- Never modify the dump file.
+- Reject dumps over 1000 lines with `result: "rejected", reason: "too large"`.
+
+## Return contract
+
+Schema fields:
+
+- `result` — one of `"ok"`, `"rejected"` (string)
+- `widget_count` — total widgets across all types (number)
+- `by_type` — object mapping widget type to count (object)
+- `summary` — one-sentence human-readable summary (string)
+
+Example (single line, no fence):
+
+`{"result":"ok","widget_count":42,"by_type":{"red":10,"blue":32},"summary":"42 widgets across 2 types"}`

package/lib/agent-loader.js

@@ -0,0 +1,47 @@
+'use strict';
+
+// Loads *.md agents from a directory and parses YAML-frontmatter + body.
+// Agents must have a `---\n...\n---\n` frontmatter block at the top with at
+// minimum a `name:` field. `description:` and `tools:` are commonly required
+// downstream but missing fields are non-fatal here — the static suite reports
+// them as failures.
+
+const fs = require('node:fs');
+const path = require('node:path');
+
+function parseAgent(filePath) {
+  const raw = fs.readFileSync(filePath, 'utf8');
+  const m = raw.match(/^---\n([\s\S]*?)\n---\n([\s\S]*)$/);
+  if (!m) return { error: 'no frontmatter', file: filePath };
+  const fm = {};
+  for (const line of m[1].split('\n')) {
+    const kv = line.match(/^(\w+):\s*(.+)$/);
+    if (kv) fm[kv[1]] = kv[2].trim();
+  }
+  return {
+    file: filePath,
+    name: fm.name,
+    description: fm.description || '',
+    tools: (fm.tools || '').split(',').map(t => t.trim()).filter(Boolean),
+    body: m[2],
+  };
+}
+
+function loadAgents(agentsDir) {
+  if (!fs.existsSync(agentsDir)) {
+    throw new Error(`agentSourceDir not found: ${agentsDir}`);
+  }
+  return fs.readdirSync(agentsDir)
+    .filter(f => f.endsWith('.md'))
+    .map(f => parseAgent(path.join(agentsDir, f)))
+    .filter(a => !a.error);
+}
+
+function loadCases(casesFile) {
+  if (!casesFile || !fs.existsSync(casesFile)) return [];
+  return fs.readFileSync(casesFile, 'utf8')
+    .trim().split('\n').filter(Boolean)
+    .map(l => JSON.parse(l));
+}
+
+module.exports = { parseAgent, loadAgents, loadCases };

package/lib/config.js

@@ -0,0 +1,60 @@
+'use strict';
+
+// Loads agent-eval.config.json with sensible defaults and env-var overrides.
+//
+// Resolution order (highest precedence wins):
+//   1. CLI --config=<path>
+//   2. process.env.AGENT_EVAL_CONFIG
+//   3. ./agent-eval.config.json in cwd
+//   4. Built-in defaults
+//
+// Paths in the config are resolved relative to the config file's directory
+// (or cwd if no file was loaded), so a config can be checked into a repo and
+// stay valid wherever the eval is run from.
+
+const fs = require('node:fs');
+const path = require('node:path');
+
+const DEFAULT_VALID_TOOLS = [
+  'Read', 'Write', 'Edit', 'Bash', 'Grep', 'Glob',
+  'WebSearch', 'WebFetch', 'NotebookEdit', 'Task',
+];
+
+const DEFAULTS = {
+  agentSourceDir: './agents',
+  fixturesDir: './_evals/fixtures',
+  schemasFile: './_evals/schemas.json',
+  casesFile: './_evals/cases.jsonl',
+  validTools: DEFAULT_VALID_TOOLS,
+  scopeSectionPattern: '## (Hard rules|Scope|When to refuse)',
+  triggerPattern: "\\b[Uu]se\\b(?:\\s+(?:this|proactively|always|only|never|right))?\\s+(?:when|whenever|before|after|during|on)\\b",
+  minDescriptionChars: 40,
+  maxTools: 5,
+  defaultThreshold: 0.85,
+};
+
+function loadConfig({ configPath, cwd = process.cwd() } = {}) {
+  const explicit = configPath || process.env.AGENT_EVAL_CONFIG;
+  const candidate = explicit
+    ? path.resolve(cwd, explicit)
+    : path.resolve(cwd, 'agent-eval.config.json');
+
+  let loaded = {};
+  let baseDir = cwd;
+  if (fs.existsSync(candidate)) {
+    loaded = JSON.parse(fs.readFileSync(candidate, 'utf8'));
+    baseDir = path.dirname(candidate);
+  } else if (explicit) {
+    throw new Error(`config file not found: ${candidate}`);
+  }
+
+  const merged = { ...DEFAULTS, ...loaded };
+  for (const key of ['agentSourceDir', 'fixturesDir', 'schemasFile', 'casesFile']) {
+    if (merged[key]) merged[key] = path.resolve(baseDir, merged[key]);
+  }
+  merged.__configPath = fs.existsSync(candidate) ? candidate : null;
+  merged.__baseDir = baseDir;
+  return merged;
+}
+
+module.exports = { loadConfig, DEFAULTS, DEFAULT_VALID_TOOLS };

package/lib/extract-json.js

@@ -0,0 +1,79 @@
+'use strict';
+
+// Extracts the canonical JSON object from a subagent response.
+//
+// Subagents are instructed to emit raw JSON only, but the underlying LLMs
+// sometimes still wrap the JSON in markdown fences or add explanatory prose
+// around it. Callers should run this defensively rather than calling
+// JSON.parse directly.
+//
+// Strategy: find the largest balanced {...} block in the response and parse
+// it. We balance braces while ignoring brace-like characters inside strings.
+
+function extractJson(text) {
+  if (typeof text !== 'string') {
+    throw new TypeError('extractJson expects a string');
+  }
+
+  // Fast path: response is already pure JSON.
+  const trimmed = text.trim();
+  if (trimmed.startsWith('{') && trimmed.endsWith('}')) {
+    try { return JSON.parse(trimmed); } catch (_) { /* fall through */ }
+  }
+
+  // Strip a single ```json ... ``` fence if present (common LLM habit).
+  const fence = trimmed.match(/```(?:json)?\s*\n([\s\S]*?)\n```/i);
+  if (fence) {
+    try { return JSON.parse(fence[1].trim()); } catch (_) { /* fall through */ }
+  }
+
+  // General case: scan for the first top-level balanced {...} block.
+  const start = text.indexOf('{');
+  if (start === -1) throw new Error('extractJson: no `{` in response');
+
+  let depth = 0;
+  let inString = false;
+  let escape = false;
+  for (let i = start; i < text.length; i++) {
+    const ch = text[i];
+    if (escape) { escape = false; continue; }
+    if (ch === '\\' && inString) { escape = true; continue; }
+    if (ch === '"') { inString = !inString; continue; }
+    if (inString) continue;
+    if (ch === '{') depth++;
+    else if (ch === '}') {
+      depth--;
+      if (depth === 0) {
+        const candidate = text.slice(start, i + 1);
+        return JSON.parse(candidate);
+      }
+    }
+  }
+  throw new Error('extractJson: unbalanced braces');
+}
+
+module.exports = { extractJson };
+
+if (require.main === module) {
+  const cases = [
+    ['{"a":1}', { a: 1 }],
+    ['  {"a":1}  ', { a: 1 }],
+    ['Here you go:\n```json\n{"a":1}\n```\nDone.', { a: 1 }],
+    ['noise\n{"nested":{"b":2}}\nmore noise', { nested: { b: 2 } }],
+    ['{"s":"with } brace inside"}', { s: 'with } brace inside' }],
+    ['prose {"a":1} trailing', { a: 1 }],
+  ];
+  let pass = 0;
+  for (const [input, want] of cases) {
+    try {
+      const got = extractJson(input);
+      const ok = JSON.stringify(got) === JSON.stringify(want);
+      if (ok) pass++;
+      else console.log('FAIL:', JSON.stringify(input), '→', JSON.stringify(got));
+    } catch (e) {
+      console.log('FAIL:', JSON.stringify(input), '→', e.message);
+    }
+  }
+  console.log(`extract-json self-test: ${pass}/${cases.length}`);
+  process.exit(pass === cases.length ? 0 : 1);
+}

package/lib/text.js

@@ -0,0 +1,87 @@
+'use strict';
+
+// Tokenization, stemming, Jaccard, IDF, weighted-recall — language-agnostic
+// helpers used by the routing and overlap suites. No filesystem access here;
+// callers pass in plain strings and lists of agents.
+
+const DEFAULT_STOPWORDS = new Set([
+  'a','an','the','is','are','was','were','be','been','being','do','does','did',
+  'have','has','had','this','that','these','those','of','in','on','to','for',
+  'with','at','by','from','about','as','it','its','and','or','but','if','then',
+  'use','when','i','you','your','my','me','we','us','our','make','need','want',
+  'show','give','get','asked','ask','please','can','could','would','should','will',
+  'one','any','some','all','no','not','only','also','just','still','very','more',
+]);
+
+// Pragmatic English morphology stemmer: ies→y, ing→drop+undouble, ed→drop,
+// es→drop with sxzh/o exception, s→drop except ss. Approximates what an LLM
+// classifier handles implicitly.
+function stem(w) {
+  if (w.length > 5 && w.endsWith('ies')) return w.slice(0, -3) + 'y';
+  if (w.length > 5 && w.endsWith('ing')) {
+    let s = w.slice(0, -3);
+    if (s.length > 2 && s[s.length - 1] === s[s.length - 2] && !'aeiou'.includes(s[s.length - 1])) {
+      s = s.slice(0, -1);
+    }
+    return s;
+  }
+  if (w.length > 4 && w.endsWith('ed')) return w.slice(0, -2);
+  if (w.length > 4 && w.endsWith('es')) {
+    const prev = w[w.length - 3];
+    if ('sxzh'.includes(prev) || prev === 'o') return w.slice(0, -2);
+    return w.slice(0, -1);
+  }
+  if (w.length > 3 && w.endsWith('s') && !w.endsWith('ss')) return w.slice(0, -1);
+  return w;
+}
+
+function tokenize(s, stopwords = DEFAULT_STOPWORDS) {
+  return (s || '')
+    .toLowerCase()
+    .replace(/[^a-z0-9 ]+/g, ' ')
+    .split(/\s+/)
+    .filter(w => w.length > 2 && !stopwords.has(w))
+    .map(stem);
+}
+
+function jaccard(a, b) {
+  const sa = new Set(a);
+  const sb = new Set(b);
+  let inter = 0;
+  for (const x of sa) if (sb.has(x)) inter++;
+  const uni = sa.size + sb.size - inter;
+  return uni === 0 ? 0 : inter / uni;
+}
+
+// IDF over a corpus of agent descriptions. Tokens appearing in many agents are
+// common (low signal); rare tokens are discriminating (high signal).
+function buildIDF(agents) {
+  const N = agents.length;
+  const df = new Map();
+  for (const a of agents) {
+    const seen = new Set(tokenize(a.description));
+    for (const t of seen) df.set(t, (df.get(t) || 0) + 1);
+  }
+  const idf = new Map();
+  for (const [t, count] of df) {
+    idf.set(t, Math.log((N + 1) / (count + 1)) + 1);
+  }
+  return idf;
+}
+
+// Of the prompt's discriminative tokens (IDF-weighted), what fraction does the
+// agent description cover? Better proxy for "which agent fits this prompt"
+// than plain Jaccard, because long descriptions aren't penalized for length.
+function weightedRecall(promptTokens, agentTokens, idf) {
+  const sa = new Set(promptTokens);
+  const sb = new Set(agentTokens);
+  let interW = 0, promptW = 0;
+  for (const t of sa) {
+    const w = idf.get(t) ?? 1;
+    promptW += w;
+    if (sb.has(t)) interW += w;
+  }
+  return promptW === 0 ? 0 : interW / promptW;
+}
+
+module.exports = { stem, tokenize, jaccard, buildIDF, weightedRecall, DEFAULT_STOPWORDS };

package/package.json

@@ -0,0 +1,51 @@
+{
+  "name": "agent-eval-harness",
+  "version": "0.1.0",
+  "description": "Static + schema + routing + spawn-fixture eval harness for *.md subagents (Claude Code, etc.). Catches description bloat, fence-mimicry, low routing margin, and schema regressions before they ship.",
+  "main": "index.js",
+  "bin": {
+    "agent-eval": "./cli.js"
+  },
+  "files": [
+    "index.js",
+    "cli.js",
+    "lib/",
+    "suites/",
+    "init/",
+    "README.md",
+    "CHANGELOG.md",
+    "LICENSE"
+  ],
+  "scripts": {
+    "test": "node test.js"
+  },
+  "engines": {
+    "node": ">=18"
+  },
+  "keywords": [
+    "agents",
+    "subagents",
+    "claude-code",
+    "claude",
+    "anthropic",
+    "eval",
+    "lint",
+    "linter",
+    "ai",
+    "llm",
+    "routing"
+  ],
+  "author": "Yorkis Estevez",
+  "license": "MIT",
+  "homepage": "https://github.com/yorkisestevez/agent-eval-harness#readme",
+  "bugs": {
+    "url": "https://github.com/yorkisestevez/agent-eval-harness/issues"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/yorkisestevez/agent-eval-harness.git"
+  },
+  "publishConfig": {
+    "access": "public"
+  }
+}

package/suites/routing.js

@@ -0,0 +1,55 @@
+'use strict';
+
+// Routing suite: scoring proxy for the LLM-based agent picker.
+// For each case (prompt + expected agent), score every agent's description
+// with IDF-weighted recall against the prompt. Pass = expected agent ranks
+// first AND has strictly positive margin over runner-up (a tie at the top
+// could go either way through the LLM picker, so it's treated as a fail).
+//
+// Plus an overlap audit: any pair of agents whose descriptions exceed
+// Jaccard 0.20 is informational by default, BLOCKING under --strict.
+
+const { tokenize, jaccard, buildIDF, weightedRecall } = require('../lib/text');
+
+function routingSuite(agents, cases) {
+  const agentTokens = new Map(agents.map(a => [a.name, tokenize(a.description)]));
+  const idf = buildIDF(agents);
+  const results = [];
+  for (const c of cases) {
+    const promptTokens = tokenize(c.prompt);
+    const scored = agents.map(a => ({
+      agent: a.name,
+      score: weightedRecall(promptTokens, agentTokens.get(a.name), idf),
+    })).sort((x, y) => y.score - x.score);
+    const top = scored[0];
+    const second = scored[1] || { score: 0 };
+    const absMargin = top.score - second.score;
+    const relRatio = second.score > 0 ? top.score / second.score : Infinity;
+    const pass = top.agent === c.expect_agent && absMargin > 0;
+    results.push({
+      id: c.id,
+      expected: c.expect_agent,
+      got: top.agent,
+      score: top.score,
+      margin: absMargin,
+      ratio: relRatio,
+      runner_up: second.agent,
+      pass,
+    });
+  }
+  return results;
+}
+
+function overlapAudit(agents, threshold = 0.20) {
+  const tok = new Map(agents.map(a => [a.name, tokenize(a.description)]));
+  const pairs = [];
+  for (let i = 0; i < agents.length; i++) {
+    for (let j = i + 1; j < agents.length; j++) {
+      const s = jaccard(tok.get(agents[i].name), tok.get(agents[j].name));
+      if (s >= threshold) pairs.push({ a: agents[i].name, b: agents[j].name, score: s });
+    }
+  }
+  return pairs.sort((x, y) => y.score - x.score);
+}
+
+module.exports = { routingSuite, overlapAudit };

package/suites/schema.js

@@ -0,0 +1,48 @@
+'use strict';
+
+// Schema suite: agents declare a structured-output return contract.
+//   - body has a "## Return contract" section OR a "must be JSON" instruction
+//   - body declares the JSON shape in one of three forms: ```json fenced block,
+//     a single-line backticked example with balanced braces, or a Schema-fields
+//     bullet list.
+//
+// Plus a fence audit: agents using ```json INSIDE the Return contract section
+// can provoke fence-mimicry in some LLMs (response wraps the JSON in fences,
+// breaking naive JSON.parse). Informational by default; promoted to BLOCKING
+// under --strict / threshold=1.0.
+
+function schemaSuite(agents) {
+  const results = [];
+  for (const a of agents) {
+    const hasReturnContract =
+      /## Return contract/i.test(a.body) ||
+      /response MUST be (a |the )?JSON/i.test(a.body) ||
+      /Return JSON only:?/i.test(a.body) ||
+      /your final response must be/i.test(a.body) ||
+      /STRICT OUTPUT MODE/i.test(a.body);
+    const hasJsonBlock =
+      /```json[\s\S]*?```/m.test(a.body) ||
+      /`\{[\s\S]*?\}`/m.test(a.body) ||
+      /## Schema fields|Schema fields:/i.test(a.body);
+    results.push({
+      agent: a.name,
+      pass: hasReturnContract && hasJsonBlock,
+      detail: !hasReturnContract ? 'no return-contract section'
+            : !hasJsonBlock ? 'no JSON shape declaration'
+            : 'ok',
+    });
+  }
+  return results;
+}
+
+function fenceAudit(agents) {
+  const offenders = [];
+  for (const a of agents) {
+    const m = a.body.match(/##\s*Return contract[\s\S]*$/i);
+    const tail = m ? m[0] : '';
+    if (/```json[\s\S]*?```/m.test(tail)) offenders.push(a.name);
+  }
+  return offenders;
+}
+
+module.exports = { schemaSuite, fenceAudit };

package/suites/spawn.js

@@ -0,0 +1,94 @@
+'use strict';
+
+// Spawn suite: fixture-driven contract test.
+// For each agent in schemas.json, look for a fixture at fixtures/<name>.txt.
+// Extract JSON, validate every required field, type, and enum against the
+// declared schema. Missing fixture is informational by default; under
+// --strict / threshold=1.0 it counts as a failed check.
+
+const fs = require('node:fs');
+const path = require('node:path');
+const { extractJson } = require('../lib/extract-json');
+
+function loadSchemas(schemasFile) {
+  if (!fs.existsSync(schemasFile)) return {};
+  return JSON.parse(fs.readFileSync(schemasFile, 'utf8'));
+}
+
+function loadFixture(fixturesDir, agentName) {
+  const p = path.join(fixturesDir, `${agentName}.txt`);
+  if (!fs.existsSync(p)) return null;
+  return fs.readFileSync(p, 'utf8');
+}
+
+function jsType(v) {
+  if (Array.isArray(v)) return 'array';
+  if (v === null) return 'null';
+  return typeof v;
+}
+
+function validateAgainst(obj, schema, prefix = '') {
+  const errors = [];
+  for (const k of schema.required || []) {
+    if (!(k in obj)) errors.push(`missing required field: ${prefix}${k}`);
+  }
+  for (const [k, expected] of Object.entries(schema.types || {})) {
+    if (!(k in obj)) continue;
+    const actual = jsType(obj[k]);
+    if (actual !== expected && !(expected === 'string' && actual === 'null')) {
+      errors.push(`type mismatch: ${prefix}${k} expected ${expected}, got ${actual}`);
+    }
+  }
+  for (const [k, allowed] of Object.entries(schema.enums || {})) {
+    if (!(k in obj)) continue;
+    if (!allowed.includes(obj[k])) {
+      errors.push(`enum violation: ${prefix}${k}="${obj[k]}" not in [${allowed.join('|')}]`);
+    }
+  }
+  for (const [k, sub] of Object.entries(schema.nested || {})) {
+    if (!(k in obj)) continue;
+    if (jsType(obj[k]) !== 'object') continue;
+    errors.push(...validateAgainst(obj[k], sub, `${prefix}${k}.`));
+  }
+  return errors;
+}
+
+function spawnSuite(config, { strict = false } = {}) {
+  const schemas = loadSchemas(config.schemasFile);
+  const targets = Object.keys(schemas);
+
+  const out = { results: [], noFixture: [], totalChecks: 0, totalPass: 0, coveredAgents: 0 };
+  for (const name of targets) {
+    const schema = schemas[name];
+    const fixture = loadFixture(config.fixturesDir, name);
+    if (!fixture) {
+      out.noFixture.push(name);
+      if (strict) {
+        out.totalChecks++;
+        out.results.push({ agent: name, pass: false, reason: 'no-fixture' });
+      }
+      continue;
+    }
+    out.coveredAgents++;
+    out.totalChecks++;
+
+    let parsed;
+    try {
+      parsed = extractJson(fixture);
+    } catch (e) {
+      out.results.push({ agent: name, pass: false, reason: 'extract-json', detail: e.message });
+      continue;
+    }
+
+    const errors = validateAgainst(parsed, schema);
+    if (errors.length === 0) {
+      out.totalPass++;
+      out.results.push({ agent: name, pass: true });
+    } else {
+      out.results.push({ agent: name, pass: false, reason: 'schema', errors });
+    }
+  }
+  return out;
+}
+
+module.exports = { spawnSuite, validateAgainst, loadSchemas, loadFixture };

package/suites/static.js

@@ -0,0 +1,45 @@
+'use strict';
+
+// Static suite: per-agent linting of the *.md frontmatter + body.
+//   - has-name             frontmatter has `name:`
+//   - has-description      description >= minDescriptionChars
+//   - tools<=N             tools count between 1 and config.maxTools
+//   - tools-valid          every tool in config.validTools[]
+//   - filename-matches     basename(file) === name
+//   - description-trigger  description names a "use when X" condition
+//   - has-scope-discipline body has a Scope/Hard rules/When to refuse section
+//   - no-recursion         body doesn't itself call subagents
+
+const path = require('node:path');
+
+function staticSuite(agents, config) {
+  const validTools = new Set(config.validTools);
+  const triggerRe = new RegExp(config.triggerPattern, 'i');
+  const scopeRe = new RegExp(config.scopeSectionPattern, 'i');
+  const minDesc = config.minDescriptionChars;
+  const maxTools = config.maxTools;
+
+  const results = [];
+  for (const a of agents) {
+    const checks = [];
+    checks.push({ name: 'has-name', pass: !!a.name });
+    checks.push({ name: 'has-description', pass: a.description.length >= minDesc });
+    checks.push({ name: `tools<=${maxTools}`, pass: a.tools.length <= maxTools && a.tools.length >= 1 });
+    checks.push({
+      name: 'tools-valid',
+      pass: a.tools.every(t => validTools.has(t)),
+      detail: a.tools.filter(t => !validTools.has(t)).join(','),
+    });
+    checks.push({
+      name: 'filename-matches-name',
+      pass: path.basename(a.file, '.md') === a.name,
+    });
+    checks.push({ name: 'description-uses-when', pass: triggerRe.test(a.description) });
+    checks.push({ name: 'has-scope-discipline', pass: scopeRe.test(a.body) });
+    checks.push({ name: 'no-recursion', pass: !/(subagent_type|Task\s*\(|Agent\s*\()/i.test(a.body) });
+    results.push({ agent: a.name, checks });
+  }
+  return results;
+}
+
+module.exports = { staticSuite };