naive-framework 0.0.0-stage → 0.1.0

package/evidence/benchmarks/JEV_VS_REASONING_MODEL_2026-10-27.md

@@ -0,0 +1,68 @@
+---
+type: research-note
+source: hybrid
+created: 2026-10-27T06:05:00Z
+ai-model: claude-opus-5.5
+authority: advisory
+status: DRAFT
+tags: [everything-ecosystem, mad, jev, typesafe, benchmark, scoring]
+description: "First head-to-head of Jev (TypeSafe jev-1.13.0) and a reasoning model (Claude) as MAD's methodology scorer, on two cases: assignment A-001 and the golden due-date case. Small sample; read as a first look, not a verdict."
+---
+
+# Jev versus a reasoning model as MAD's scorer: first look
+
+Both scorers answer the same typed questions from `ee recommend` (whole fit, part fit, and whether each fits-when and unfit-when condition holds); the same scoring rule (v2) turns the answers into a ranking. Jev ran three times per case; the reasoning model once, by hand (`--answers`).
+
+## Results
+
+**A-001, Product Anatomy runner JSON output** (a one-flag change to a small CLI, with tests).
+
+| Methodology | Reasoning model | Jev (3 runs) |
+| --- | --- | --- |
+| small change | 1st, 0.70 | 5th, 0.36 |
+| spec-first | 2nd, 0.64 | 11th, 0.14 |
+| contract-first | 3rd, 0.34 | 6th, 0.32 |
+| spec-anchored | 4th, 0.26 | 2nd, 0.44 |
+| V-model (candidate) | 9th, 0.05 | **1st, 0.62** |
+
+**Golden case, due-date calculator** (rules as data, high cost of error, high verifiability). The golden expectation (SD-20) is spec-as-source, TDD and design by contract near the top, and neither spec-anchored prose nor a spike first.
+
+| Methodology | Reasoning model | Jev (3 runs) |
+| --- | --- | --- |
+| design by contract | **1st, 0.63** | 3rd, 0.57 |
+| V-model (candidate) | 2nd, 0.44 | **1st, 0.72** |
+| contract-first | 3rd, 0.41 | 7th, 0.36 |
+| spec-anchored | 4th, 0.37 | 2nd, 0.60 |
+| BDD | lower | 4th, 0.52 |
+
+| | Reasoning model | Jev |
+| --- | --- | --- |
+| Run-to-run spread | not measured (one run) | at most 0.05, mostly 0.01 |
+| Cost per recommendation | the session's own tokens | about $0.0034 for 3 runs of 20 methodologies |
+| Reasons | written with each answer | none; reasons come from which conditions passed |
+| Setup | none | API key; project description sent to TypeSafe |
+
+## What it shows
+
+1. **Jev is very consistent.** Three runs moved scores by 0.01 to 0.05. That is its real strength over hand scoring.
+2. **Jev ranks V-model first in both cases, including a one-flag CLI change.** It answers each condition literally ("requirements are stable enough to write a test specification" holds for a JSON flag) and the catalog gives it nothing to weigh proportion against: V-model's unfit-when list has no "small, low-risk change" condition (spiral's has one). This is a catalog gap first; any scorer reading the catalog literally will make it.
+3. **Jev reads conditions without context.** For A-001 it says both "tiny changes" (0.66) and "intent that must stay true for months" (0.77) hold, which pushes spec-first to 11th.
+4. **The golden case cannot be fully tested today.** TDD and spec-as-source are in `re-check` lifecycle, which `ee recommend` skips, so the two methods the golden case expects near the top were not scored by either side. Design by contract, the third, is in both top three.
+5. **The reasoning model's answers are not blind.** It proposed A-001 before Jev ran, and the golden expectation is written in the case file it read. Its agreement with the expectation is weak evidence.
+
+## After the catalog fix (same day)
+
+Torian chose to fix the catalog before re-scoring. Proportion conditions were added to V-model, waterfall, the agile-waterfall hybrid, Scrum and spec-anchored (Boehm and Turner's home grounds; the Scrum Guide), and TDD and spec-as-source were re-checked and returned to `listed`. Jev was re-run three times on each case (`2026-10-27-golden.jev.after-catalog-fix.json` beside this note; A-001's run is in the register).
+
+| Case | Jev before | Jev after |
+| --- | --- | --- |
+| A-001 | V-model 0.62, then spec-anchored, design by contract | **TDD 0.79 alone in band 1**; spec-anchored 0.45; V-model fell to 8th (its new proportion condition held at 0.75) |
+| Golden | V-model 0.72, spec-anchored 0.60, design by contract 0.57 | V-model 0.73; band 2 holds spec-anchored, design by contract, TDD, BDD and spec-as-source (0.60 to 0.50) |
+
+On the golden case all three expected methods are now in the second band, behind V-model. For a high-stakes, auditable calculator, V-model first is defensible. Jev's spread stayed at 0.01 to 0.03. The two disagreements in the first run came mostly from the catalog, not the scorer.
+
+## What would make the comparison fair
+
+- Fix the catalog first: add a proportion condition to V-model and the other heavyweight candidates, and finish the re-check of TDD and spec-as-source.
+- Score held-out cases whose expected ranking the reasoning model has not seen, and repeat the reasoning model three times too.
+- Then compare agreement with the expected rankings, spread, cost and time.

package/evidence/capabilities/agent-capabilities.json

@@ -0,0 +1,86 @@
+{
+  "updated": "2026-10-27",
+  "note": "Starts small and honest: entries are 'unknown' until a source or our own eval says otherwise. Re-check entries with a url when they are older than six months; the research moves fast.",
+  "capabilities": [
+    {
+      "id": "implement-ready-handoff",
+      "task": "Implement a software change from a handoff that passed its readiness check, to its done criteria",
+      "area": "execution",
+      "status": "can-with-review",
+      "basis": [
+        {
+          "title": "Our own handoff runs (anatomy runner fixes, JSON output, site refresh)",
+          "eval": "instance/ventures/everything-ecosystem/evidence/results/anatomy-runner-fixes-20261027052719.json"
+        },
+        {
+          "title": "Measuring AI Ability to Complete Long Software Tasks (METR)",
+          "url": "https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/",
+          "checked": "2026-10-27"
+        }
+      ],
+      "checked": "2026-10-27",
+      "notes": "Reliability falls with task length; METR measures a 50% time horizon that has doubled about every seven months. Keep handoffs small and the conformance checks objective."
+    },
+    {
+      "id": "draft-records-from-sources",
+      "task": "Draft venture records (schemas, templates, references) from cited sources",
+      "area": "knowledge",
+      "status": "can-with-review",
+      "basis": [
+        {
+          "title": "Our own sessions building this repository",
+          "eval": "instance/ventures/everything-ecosystem/evidence/results/methodology-site-refresh-20261027054211.json"
+        }
+      ],
+      "checked": "2026-10-27",
+      "notes": "Every reference URL is fetched and dated before it is cited."
+    },
+    {
+      "id": "sign-or-commit-the-venture",
+      "task": "Sign, file or agree to legal terms, ownership changes, spending or public commitments",
+      "area": "governance",
+      "status": "not-allowed",
+      "basis": [
+        {
+          "decision_class": "ownership-or-control"
+        },
+        {
+          "decision_class": "legal-or-terms"
+        },
+        {
+          "decision_class": "spend-money"
+        },
+        {
+          "decision_class": "public-commitment"
+        }
+      ],
+      "checked": "2026-10-27"
+    },
+    {
+      "id": "customer-interviews",
+      "task": "Conduct problem interviews with people in the target segment",
+      "area": "purpose",
+      "status": "unknown",
+      "basis": [],
+      "checked": "2026-10-27",
+      "notes": "Agents can draft the script and synthesise notes; whether people talk to an agent as candidly as to a founder is untested here."
+    },
+    {
+      "id": "source-candidates",
+      "task": "Find and screen candidates for a role (cofounder, hire, contractor, investor)",
+      "area": "governance",
+      "status": "unknown",
+      "basis": [],
+      "checked": "2026-10-27",
+      "notes": "Agents can write the brief and search public profiles; outreach and judgement of fit stay with a person until an eval says otherwise."
+    },
+    {
+      "id": "sales-conversations",
+      "task": "Hold sales conversations with buyers",
+      "area": "purpose",
+      "status": "unknown",
+      "basis": [],
+      "checked": "2026-10-27"
+    }
+  ]
+}