@draig/lexis-two 1.0.0 → 1.0.3

package/benchmarks/render-opencode-go-report.js

@@ -1,28 +0,0 @@
-#!/usr/bin/env node
-/**
- * Sync aggregated benchmark data for the Astro site.
- *
- * Usage:
- *   node benchmarks/render-opencode-go-report.js
- *   node benchmarks/render-opencode-go-report.js benchmarks/results/opencode-go-2026-06-16.json
- */
-
-const fs = require('fs');
-const path = require('path');
-const { aggregateOpencodeGo, findLatestJson } = require('./lib/aggregate-opencode-go.js');
-
-const SITE_DATA = path.join(__dirname, '..', 'site', 'src', 'data', 'opencode-go-benchmark.json');
-
-function main() {
-  const input = process.argv[2] ? path.resolve(process.argv[2]) : findLatestJson();
-  const data = JSON.parse(fs.readFileSync(input, 'utf8'));
-  const chart = aggregateOpencodeGo(data);
-
-  fs.mkdirSync(path.dirname(SITE_DATA), { recursive: true });
-  fs.writeFileSync(SITE_DATA, `${JSON.stringify(chart, null, 2)}\n`, 'utf8');
-
-  console.log(`Site data → ${SITE_DATA}`);
-  console.log('Preview charts: npm run site:dev → http://localhost:4321/benchmarks/');
-}
-
-main();

package/benchmarks/results/2026-06-15-llama3.2-local.md

@@ -1,76 +0,0 @@
-# Local model benchmark: llama3.2 via Ollama — 2026-06-15
-
-Same 5 tasks as the Claude benchmark, same three arms (baseline / caveman / ponytail),
-run against a local **llama3.2:latest** (3.2B, Q4_K_M) via Ollama on a Windows 11 machine.
-Tooling: `benchmarks/benchmark-local.py` (no promptfoo needed).
-
-> **Updated 2026-06-15:** the LOC counter now counts bare, unfenced code. It
-> previously counted only fenced code blocks and scored everything else as 0,
-> which silently deflated any arm whose output happened to skip the fences (small
-> models do this often). Numbers below use the corrected counter at n=5 median.
-> Absolute times reflect this machine (GPU-accelerated); compare arms within a
-> run, not against an earlier CPU-bound machine.
-
-## Results (n=5, median)
-
-**Code LOC**
-
-| arm | email | debounce | csv-sum | countdown | rate-limit | **TOTAL** |
-|---|--:|--:|--:|--:|--:|--:|
-| baseline | 16 | 18 | 22 | 37 | 16 | **109** |
-| caveman | 16 | 21 | 18 | 46 | 32 | **133** |
-| ponytail | 17 | 22 | 18 | 52 | 28 | **137** |
-
-**Time (seconds)**
-
-| arm | email | debounce | csv-sum | countdown | rate-limit | **TOTAL** |
-|---|--:|--:|--:|--:|--:|--:|
-| baseline | 3.1 | 3.7 | 3.6 | 4.2 | 4.8 | **19.4** |
-| caveman | 4.1 | 4.2 | 3.6 | 4.4 | 4.8 | **21.1** |
-| ponytail | 4.1 | 4.2 | 3.8 | 4.8 | 4.9 | **21.8** |
-
-## Key findings
-
-**On llama3.2 the LOC effect is inside the noise floor.** At temperature 0.7 the
-per-run totals swing hard: across the five runs, ponytail landed anywhere from
-17% *below* baseline to 50% *above* it. The n=5 median came out +26%; a separate
-n=3 median came out −17%. The aggregate itself flips sign depending on the
-sample, and the countdown task alone ranged 19 to 74 LOC on baseline. There is no
-stable LOC reduction to report.
-
-**Ponytail does not transfer to llama3.2.** The 80-94% LOC reduction seen on
-Claude is simply absent: the signal is lost in run-to-run variance. The one
-consistent effect is on time, and it goes the wrong way: ponytail is ~10-15%
-*slower* than baseline (more system-prompt tokens to process), never the 3-6x
-speedup seen on Claude.
-
-**Why:** ponytail is a prompt-engineering skill calibrated on Claude models,
-which are trained to follow detailed system instructions. A 3.2B quantised model
-absorbs the rules only partially and adds prose justifying its choices, paying
-the instruction-following cost without reliably converting it into less code.
-
-## Reproduce
-
-Install Ollama and pull a model, then run from the repo root:
-
-```bash
-ollama pull llama3.2
-python benchmarks/benchmark-local.py --model llama3.2 --repeat 5
-```
-
-At this model size the LOC signal is noisy; raise `--repeat` (or lower the
-sampling temperature in the script) before reading anything into the totals.
-
-Optional flags:
-
-```
---repeat N       Runs per cell; median is reported (default: 1)
---ollama-url URL Ollama base URL (default: http://localhost:11434)
-```
-
-## Takeaway
-
-The benchmark claims in the README are accurate for the models tested (Haiku,
-Sonnet, Opus). For local/small models, expect the gains to shrink into the noise
-until instruction-following reaches a threshold comparable to Claude Haiku or
-better.

package/benchmarks/results/2026-06-16-opencode-go.md

@@ -1,56 +0,0 @@
-# Lexis-Two benchmark — OpenCode Go (2026-06-16)
-
-Provider: [OpenCode Go](https://opencode.ai/docs/go/).
-Repeat: 3 per cell. Temperature: 1.
-
-## Kimi K2.6 (`kimi-k2.6`)
-
-Repeat: 3. Arms: baseline, lexis-two.
-
-**Code LOC (median)**
-
-| arm | email | debounce | csv-sum | countdown | rate-limit | TOTAL | correct |
-| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
-| baseline | 46 | 63 | 18 | 413 | 62 | 602 | 12/15 |
-| lexis-two | 13 | 10 | 4 | 13 | 23 | 63 | 12/15 |
-
-**lexis-two vs baseline (median total LOC):** 90% less code.
-
-## DeepSeek V4 Pro (`deepseek-v4-pro`)
-
-Repeat: 3. Arms: baseline, lexis-two.
-
-**Code LOC (median)**
-
-| arm | email | debounce | csv-sum | countdown | rate-limit | TOTAL | correct |
-| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
-| baseline | 36 | 61 | 25 | 113 | 53 | 288 | 14/15 |
-| lexis-two | 9 | 12 | 4 | 12 | 20 | 57 | 13/15 |
-
-**lexis-two vs baseline (median total LOC):** 80% less code.
-
-## Qwen3.7 Max (`qwen3.7-max`)
-
-Repeat: 3. Arms: baseline, lexis-two.
-
-**Code LOC (median)**
-
-| arm | email | debounce | csv-sum | countdown | rate-limit | TOTAL | correct |
-| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
-| baseline | 39 | 48 | 19 | 124 | 40 | 270 | 12/15 |
-| lexis-two | 14 | 9 | 4 | 10 | 17 | 54 | 13/15 |
-
-**lexis-two vs baseline (median total LOC):** 80% less code.
-
-## MiniMax M3 (`minimax-m3`)
-
-Repeat: 3. Arms: baseline, lexis-two.
-
-**Code LOC (median)**
-
-| arm | email | debounce | csv-sum | countdown | rate-limit | TOTAL | correct |
-| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
-| baseline | 55 | 66 | 33 | 112 | 59 | 325 | 11/15 |
-| lexis-two | 9 | 10 | 4 | 18 | 15 | 56 | 15/15 |
-
-**lexis-two vs baseline (median total LOC):** 83% less code.

package/benchmarks/results/opencode-go-2026-06-16-report.html

@@ -1,226 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
-  <meta charset="utf-8" />
-  <meta name="viewport" content="width=device-width, initial-scale=1" />
-  <title>Lexis-Two benchmark — OpenCode Go (2026-06-16)</title>
-  <script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.1/dist/chart.umd.min.js"></script>
-  <style>
-    :root {
-      --bg: #0b0d0c;
-      --panel: #121614;
-      --border: #1e2420;
-      --text: #e8ece9;
-      --muted: #8a948d;
-      --accent: #7cba8a;
-      --baseline: #c97a7a;
-    }
-    * { box-sizing: border-box; }
-    body {
-      margin: 0;
-      font-family: system-ui, sans-serif;
-      background: var(--bg);
-      color: var(--text);
-      line-height: 1.5;
-    }
-    .wrap { max-width: 1100px; margin: 0 auto; padding: 2rem 1rem 3rem; }
-    h1 { font-size: 1.75rem; margin: 0 0 0.25rem; }
-    .sub { color: var(--muted); margin-bottom: 2rem; font-size: 0.95rem; }
-    .grid {
-      display: grid;
-      gap: 1.25rem;
-      grid-template-columns: repeat(auto-fit, minmax(320px, 1fr));
-    }
-    .card {
-      background: var(--panel);
-      border: 1px solid var(--border);
-      border-radius: 0.5rem;
-      padding: 1rem 1rem 0.5rem;
-    }
-    .card h2 {
-      font-size: 0.85rem;
-      text-transform: uppercase;
-      letter-spacing: 0.06em;
-      color: var(--muted);
-      margin: 0 0 0.75rem;
-    }
-    .card canvas { max-height: 280px; }
-    .wide { grid-column: 1 / -1; }
-    table {
-      width: 100%;
-      border-collapse: collapse;
-      font-size: 0.875rem;
-    }
-    th, td {
-      padding: 0.5rem 0.75rem;
-      border-bottom: 1px solid var(--border);
-      text-align: right;
-    }
-    th:first-child, td:first-child { text-align: left; }
-    th { color: var(--muted); font-weight: 600; }
-    .good { color: var(--accent); }
-    footer { margin-top: 2rem; color: var(--muted); font-size: 0.8rem; }
-    a { color: var(--accent); }
-  </style>
-</head>
-<body>
-  <div class="wrap">
-    <h1>Lexis-Two × OpenCode Go</h1>
-    <p class="sub">
-      Source: <code>opencode-go-2026-06-16.json</code> · 3 runs/cell · median LOC ·
-      arms: baseline, lexis-two
-    </p>
-
-    <div class="grid">
-      <div class="card wide">
-        <h2>Total code LOC (median, 5 tasks)</h2>
-        <canvas id="chart-total-loc"></canvas>
-      </div>
-      <div class="card">
-        <h2>LOC reduction vs baseline</h2>
-        <canvas id="chart-reduction"></canvas>
-      </div>
-      <div class="card">
-        <h2>Wall time (median total seconds)</h2>
-        <canvas id="chart-time"></canvas>
-      </div>
-      <div class="card wide">
-        <h2>LOC by task — lexis-two arm</h2>
-        <canvas id="chart-by-task"></canvas>
-      </div>
-      <div class="card wide">
-        <h2>Summary table</h2>
-        <table id="summary-table">
-          <thead>
-            <tr>
-              <th>Model</th>
-              <th>Baseline LOC</th>
-              <th>Lexis-Two LOC</th>
-              <th>Reduction</th>
-              <th>Correct (lexis)</th>
-            </tr>
-          </thead>
-          <tbody></tbody>
-        </table>
-      </div>
-    </div>
-
-    <footer>
-      Regenerate: <code>node benchmarks/render-opencode-go-report.js</code>
-    </footer>
-  </div>
-
-  <script>
-    const DATA = {"date":"2026-06-16","repeat":3,"models":[{"id":"kimi-k2.6","locByArmTask":{"baseline":{"email":56,"debounce":71,"csv-sum":19,"countdown":366,"rate-limit":49},"lexis-two":{"email":10,"debounce":5,"csv-sum":4,"countdown":16,"rate-limit":14}},"timeByArmTask":{"baseline":{"email":12.973,"debounce":14.098,"csv-sum":5.08,"countdown":21.987,"rate-limit":16.836},"lexis-two":{"email":16.23,"debounce":8.363,"csv-sum":9.846,"countdown":18.389,"rate-limit":16.189}},"correctByArm":{"baseline":{"pass":11,"total":15},"lexis-two":{"pass":10,"total":15}},"totals":{"baselineLoc":561,"lexisLoc":49,"reductionPct":91,"baselineTimeSec":71,"lexisTimeSec":69}},{"id":"deepseek-v4-pro","locByArmTask":{"baseline":{"email":60,"debounce":39,"csv-sum":26,"countdown":90,"rate-limit":66},"lexis-two":{"email":7,"debounce":7,"csv-sum":4,"countdown":10,"rate-limit":18}},"timeByArmTask":{"baseline":{"email":35.051,"debounce":14.509,"csv-sum":13.422,"countdown":46.762,"rate-limit":32.401},"lexis-two":{"email":27.456,"debounce":16.982,"csv-sum":24.931,"countdown":36.191,"rate-limit":39.952}},"correctByArm":{"baseline":{"pass":13,"total":15},"lexis-two":{"pass":12,"total":15}},"totals":{"baselineLoc":281,"lexisLoc":46,"reductionPct":84,"baselineTimeSec":142.1,"lexisTimeSec":145.5}},{"id":"qwen3.7-max","locByArmTask":{"baseline":{"email":34,"debounce":44,"csv-sum":17,"countdown":128,"rate-limit":47},"lexis-two":{"email":7,"debounce":5,"csv-sum":3,"countdown":10,"rate-limit":13}},"timeByArmTask":{"baseline":{"email":43.542,"debounce":17.58,"csv-sum":25.876,"countdown":32.586,"rate-limit":41.602},"lexis-two":{"email":39.323,"debounce":28.275,"csv-sum":23.769,"countdown":28.775,"rate-limit":40.786}},"correctByArm":{"baseline":{"pass":14,"total":15},"lexis-two":{"pass":11,"total":15}},"totals":{"baselineLoc":270,"lexisLoc":38,"reductionPct":86,"baselineTimeSec":161.2,"lexisTimeSec":160.9}},{"id":"minimax-m3","locByArmTask":{"baseline":{"email":62,"debounce":58,"csv-sum":43,"countdown":117,"rate-limit":76},"lexis-two":{"email":12,"debounce":6,"csv-sum":2,"countdown":13,"rate-limit":14}},"timeByArmTask":{"baseline":{"email":18.398,"debounce":14.916,"csv-sum":10.237,"countdown":19.516,"rate-limit":22.131},"lexis-two":{"email":8.002,"debounce":2.393,"csv-sum":2.423,"countdown":2.753,"rate-limit":4.209}},"correctByArm":{"baseline":{"pass":12,"total":15},"lexis-two":{"pass":9,"total":15}},"totals":{"baselineLoc":356,"lexisLoc":47,"reductionPct":87,"baselineTimeSec":85.2,"lexisTimeSec":19.8}}],"tasks":["email","debounce","csv-sum","countdown","rate-limit"],"arms":["baseline","lexis-two"]};
-    const labels = DATA.models.map((m) => m.id.replace('kimi-k2.6','Kimi K2.6').replace('deepseek-v4-pro','DeepSeek V4').replace('qwen3.7-max','Qwen3.7 Max').replace('minimax-m3','MiniMax M3'));
-
-    Chart.defaults.color = '#8a948d';
-    Chart.defaults.borderColor = '#1e2420';
-    Chart.defaults.font.family = 'system-ui, sans-serif';
-
-    new Chart(document.getElementById('chart-total-loc'), {
-      type: 'bar',
-      data: {
-        labels,
-        datasets: [
-          {
-            label: 'baseline',
-            data: DATA.models.map((m) => m.totals.baselineLoc),
-            backgroundColor: '#c97a7a',
-          },
-          {
-            label: 'lexis-two',
-            data: DATA.models.map((m) => m.totals.lexisLoc),
-            backgroundColor: '#7cba8a',
-          },
-        ],
-      },
-      options: {
-        responsive: true,
-        plugins: { legend: { position: 'bottom' } },
-        scales: { y: { beginAtZero: true, title: { display: true, text: 'lines of code' } } },
-      },
-    });
-
-    new Chart(document.getElementById('chart-reduction'), {
-      type: 'bar',
-      data: {
-        labels,
-        datasets: [{
-          label: '% less code',
-          data: DATA.models.map((m) => m.totals.reductionPct),
-          backgroundColor: '#7cba8a',
-        }],
-      },
-      options: {
-        indexAxis: 'y',
-        responsive: true,
-        plugins: { legend: { display: false } },
-        scales: {
-          x: { beginAtZero: true, max: 100, ticks: { callback: (v) => v + '%' } },
-        },
-      },
-    });
-
-    new Chart(document.getElementById('chart-time'), {
-      type: 'bar',
-      data: {
-        labels,
-        datasets: [
-          {
-            label: 'baseline',
-            data: DATA.models.map((m) => m.totals.baselineTimeSec),
-            backgroundColor: '#c97a7a',
-          },
-          {
-            label: 'lexis-two',
-            data: DATA.models.map((m) => m.totals.lexisTimeSec),
-            backgroundColor: '#7cba8a',
-          },
-        ],
-      },
-      options: {
-        responsive: true,
-        plugins: { legend: { position: 'bottom' } },
-        scales: { y: { beginAtZero: true, title: { display: true, text: 'seconds' } } },
-      },
-    });
-
-    const taskLabels = DATA.tasks;
-    const taskColors = ['#7cba8a', '#5a9a6a', '#9fd4a8', '#4a7356', '#3d5f48'];
-    new Chart(document.getElementById('chart-by-task'), {
-      type: 'bar',
-      data: {
-        labels,
-        datasets: taskLabels.map((taskId, i) => ({
-          label: taskId,
-          data: DATA.models.map((m) => m.locByArmTask['lexis-two'][taskId]),
-          backgroundColor: taskColors[i % taskColors.length],
-        })),
-      },
-      options: {
-        responsive: true,
-        plugins: { legend: { position: 'bottom' } },
-        scales: {
-          x: { stacked: false },
-          y: { beginAtZero: true, title: { display: true, text: 'LOC (lexis-two)' } },
-        },
-      },
-    });
-
-    const tbody = document.querySelector('#summary-table tbody');
-    for (const m of DATA.models) {
-      const c = m.correctByArm['lexis-two'];
-      const tr = document.createElement('tr');
-      tr.innerHTML = `
-        <td>${m.id}</td>
-        <td>${m.totals.baselineLoc}</td>
-        <td class="good">${m.totals.lexisLoc}</td>
-        <td class="good">${m.totals.reductionPct}%</td>
-        <td>${c.pass}/${c.total}</td>
-      `;
-      tbody.appendChild(tr);
-    }
-  </script>
-</body>
-</html>