@draig/lexis-two 1.0.0

package/benchmarks/README.md

@@ -0,0 +1,114 @@
+# Benchmark
+
+Five everyday tasks, **baseline vs lexis-two** (`skills/lexis-two/SKILL.md`). Code LOC from fenced blocks; correctness gate via `correctness.js`.
+
+## OpenCode Go (recommended)
+
+Uses the same models you run in OpenCode today. Default set in `opencode-go-models.json`:
+
+| Model ID | Name | Transport |
+| -------- | ---- | --------- |
+| `kimi-k2.6` | Kimi K2.6 | OpenAI `/chat/completions` |
+| `deepseek-v4-pro` | DeepSeek V4 Pro | OpenAI `/chat/completions` |
+| `qwen3.7-max` | Qwen3.7 Max | Anthropic `/messages` |
+| `minimax-m3` | MiniMax M3 | Anthropic `/messages` |
+
+[OpenCode Go docs](https://opencode.ai/docs/go/) — subscribe, copy API key, `/connect` in TUI.
+
+### Run
+
+```bash
+cp .env.example .env          # OPENCODE_API_KEY=...
+node benchmarks/benchmark-opencode-go.js --repeat 3 --write-md
+```
+
+Single model:
+
+```bash
+node benchmarks/benchmark-opencode-go.js --model kimi-k2.6 --repeat 10 --write-md
+```
+
+Subset:
+
+```bash
+node benchmarks/benchmark-opencode-go.js --models kimi-k2.6,deepseek-v4-pro --repeat 5
+```
+
+Optional third arm:
+
+```bash
+node benchmarks/benchmark-opencode-go.js --caveman --repeat 3
+```
+
+Outputs:
+
+- `benchmarks/results/opencode-go-YYYY-MM-DD.json` — full responses + usage
+- `benchmarks/results/YYYY-MM-DD-opencode-go.md` — summary tables (with `--write-md`)
+- `site/src/data/opencode-go-benchmark.json` — chart data for the Astro site (`npm run benchmark:report`)
+- Live charts: `https://lexis-two.excelso.xyz/benchmarks/` (after `npm run site:build` + deploy)
+
+**Publish site/README metrics only from a committed `results/*-opencode-go.md` for the lexis-two arm.**
+
+### Adding models later (Gemini, Claude, OpenAI)
+
+1. Add entry to `opencode-go-models.json` (or a new `providers/*.json` when you split harnesses).
+2. Set `transport`: `openai-chat` or `anthropic-messages` per [Go endpoints](https://opencode.ai/docs/go/#endpoints).
+3. Re-run and commit a new `results/` file.
+
+---
+
+## Legacy: Claude via promptfoo
+
+Historical ponytail-arm results (2026-06-13) — **not Lexis-Two**. `arms/ponytail.js` now reads `skills/lexis-two/SKILL.md` until the promptfoo labels are renamed.
+
+Requires `ANTHROPIC_API_KEY` and **Node.js ≥ 22.22.0**:
+
+```bash
+npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml --repeat 10
+npx promptfoo@latest view
+```
+
+### Median results (10 runs, 2026-06-13) — ponytail arm only
+
+| arm | Haiku LOC | Sonnet LOC | Opus LOC |
+| --- | --------: | ---------: | -------: |
+| baseline | 518 | 693 | 256 |
+| caveman | 116 | 120 | 67 |
+| ponytail | 39 | 44 | 51 |
+
+Versus baseline, ponytail wrote **80-94% less code** in that run. Do not cite as Lexis-Two until re-run via `benchmark-opencode-go.js`.
+
+---
+
+## Local models (Ollama)
+
+```bash
+python benchmarks/benchmark-local.py --model llama3.2 --repeat 3
+```
+
+Arms: baseline, caveman, lexis-two.
+
+---
+
+## Tasks
+
+Email validator, JS debounce, CSV sum, React countdown, FastAPI rate-limit (`promptfooconfig.yaml`).
+
+## Metrics
+
+| File | Metric | Behavior |
+| ---- | ------ | -------- |
+| `loc.js` | `loc` | Line count (measurement) |
+| `correctness.js` | `correct` | Gate — broken one-liners fail |
+
+### Prerequisites
+
+- **OpenCode Go:** Node.js 18+, `OPENCODE_API_KEY`
+- **promptfoo:** Node.js ≥ 22.22.0, `ANTHROPIC_API_KEY`, Python 3 for correctness checks
+- **Ollama:** Python 3, local Ollama
+
+## Notes
+
+- Go usage limits apply ($12/5h etc.) — use `--delay-ms` and start with `--repeat 3`.
+- `qwen3.7-max` and `minimax-m3` use Anthropic `/messages` with header **`x-api-key`** (not `Authorization: Bearer`).
+- Real sessions inject the skill once (cached); benchmark re-sends the full skill each call — pessimistic on tokens.

package/benchmarks/arms/baseline.js

@@ -0,0 +1,2 @@
+// Baseline arm: no skill, just the task.
+module.exports = ({ vars }) => [{ role: 'user', content: vars.task }];

package/benchmarks/arms/caveman-SKILL.md

@@ -0,0 +1,67 @@
+---
+name: caveman
+description: >
+  Ultra-compressed communication mode. Cuts token usage ~75% by speaking like caveman
+  while keeping full technical accuracy. Supports intensity levels: lite, full (default), ultra,
+  wenyan-lite, wenyan-full, wenyan-ultra.
+  Use when user says "caveman mode", "talk like caveman", "use caveman", "less tokens",
+  "be brief", or invokes /caveman. Also auto-triggers when token efficiency is requested.
+---
+
+Respond terse like smart caveman. All technical substance stay. Only fluff die.
+
+## Persistence
+
+ACTIVE EVERY RESPONSE. No revert after many turns. No filler drift. Still active if unsure. Off only: "stop caveman" / "normal mode".
+
+Default: **full**. Switch: `/caveman lite|full|ultra`.
+
+## Rules
+
+Drop: articles (a/an/the), filler (just/really/basically/actually/simply), pleasantries (sure/certainly/of course/happy to), hedging. Fragments OK. Short synonyms (big not extensive, fix not "implement a solution for"). Technical terms exact. Code blocks unchanged. Errors quoted exact.
+
+Pattern: `[thing] [action] [reason]. [next step].`
+
+Not: "Sure! I'd be happy to help you with that. The issue you're experiencing is likely caused by..."
+Yes: "Bug in auth middleware. Token expiry check use `<` not `<=`. Fix:"
+
+## Intensity
+
+| Level | What change |
+|-------|------------|
+| **lite** | No filler/hedging. Keep articles + full sentences. Professional but tight |
+| **full** | Drop articles, fragments OK, short synonyms. Classic caveman |
+| **ultra** | Abbreviate (DB/auth/config/req/res/fn/impl), strip conjunctions, arrows for causality (X → Y), one word when one word enough |
+| **wenyan-lite** | Semi-classical. Drop filler/hedging but keep grammar structure, classical register |
+| **wenyan-full** | Maximum classical terseness. Fully 文言文. 80-90% character reduction. Classical sentence patterns, verbs precede objects, subjects often omitted, classical particles (之/乃/為/其) |
+| **wenyan-ultra** | Extreme abbreviation while keeping classical Chinese feel. Maximum compression, ultra terse |
+
+Example — "Why React component re-render?"
+- lite: "Your component re-renders because you create a new object reference each render. Wrap it in `useMemo`."
+- full: "New object ref each render. Inline object prop = new ref = re-render. Wrap in `useMemo`."
+- ultra: "Inline obj prop → new ref → re-render. `useMemo`."
+- wenyan-lite: "組件頻重繪，以每繪新生對象參照故。以 useMemo 包之。"
+- wenyan-full: "物出新參照，致重繪。useMemo .Wrap之。"
+- wenyan-ultra: "新參照→重繪。useMemo Wrap。"
+
+Example — "Explain database connection pooling."
+- lite: "Connection pooling reuses open connections instead of creating new ones per request. Avoids repeated handshake overhead."
+- full: "Pool reuse open DB connections. No new connection per request. Skip handshake overhead."
+- ultra: "Pool = reuse DB conn. Skip handshake → fast under load."
+- wenyan-full: "池reuse open connection。不每req新開。skip handshake overhead。"
+- wenyan-ultra: "池reuse conn。skip handshake → fast。"
+
+## Auto-Clarity
+
+Drop caveman for: security warnings, irreversible action confirmations, multi-step sequences where fragment order risks misread, user asks to clarify or repeats question. Resume caveman after clear part done.
+
+Example — destructive op:
+> **Warning:** This will permanently delete all rows in the `users` table and cannot be undone.
+> ```sql
+> DROP TABLE users;
+> ```
+> Caveman resume. Verify backup exist first.
+
+## Boundaries
+
+Code/commits/PRs: write normal. "stop caveman" or "normal mode": revert. Level persist until changed or session end.

package/benchmarks/arms/caveman.js

@@ -0,0 +1,8 @@
+// Caveman arm: caveman SKILL.md (full) as the system prompt.
+const fs = require('fs');
+const path = require('path');
+const system = fs.readFileSync(path.join(__dirname, 'caveman-SKILL.md'), 'utf8');
+module.exports = ({ vars }) => [
+  { role: 'system', content: system },
+  { role: 'user', content: vars.task },
+];

package/benchmarks/arms/lexis-two.js

@@ -0,0 +1,10 @@
+// Lexis-Two arm: repo skill (full) as the system prompt.
+const fs = require('fs');
+const path = require('path');
+
+const system = fs.readFileSync(
+  path.join(__dirname, '..', '..', 'skills', 'lexis-two', 'SKILL.md'),
+  'utf8',
+);
+
+module.exports = { system };

package/benchmarks/arms/ponytail.js

@@ -0,0 +1,6 @@
+// Ponytail arm (legacy label) — reads Lexis-Two skill until promptfoo config is renamed.
+const { system } = require('./lexis-two');
+module.exports = ({ vars }) => [
+  { role: 'system', content: system },
+  { role: 'user', content: vars.task },
+];

package/benchmarks/behavior.js

@@ -0,0 +1,58 @@
+// Behavior gate: does the ponytail ruleset actually PRODUCE its refined
+// behaviors, not just carry the text? One check per probe (vars.probe), each
+// targeting a rule that a field review (rcstack, phases 0-8) showed mattered:
+//   hardware     - "hardware is never the spec ideal, leave the calibration knob"
+//   explanation  - "explanation the user explicitly asked for is not debt"
+//   onecheck     - "lazy code without its check is unfinished"
+//
+// Heuristic graders, same spirit as loc.js / correctness.js. The graders
+// themselves are proven by tests/behavior.test.js (RED/GREEN, no API key).
+//
+// Metric: `behavior` (1 = behavior present, 0 = absent).
+
+function codeOf(text) {
+  return [...String(text || '').matchAll(/```[\w-]*\n([\s\S]*?)```/g)].map((m) => m[1]).join('\n');
+}
+
+function proseOf(text) {
+  return String(text || '').replace(/```[\s\S]*?```/g, ' ').replace(/\s+/g, ' ').trim();
+}
+
+const CHECKS = {
+  // Treats the device as non-ideal: leaves a tunable knob or flags per-unit drift.
+  // A passing mention of "calibration" is not enough; it must be actionable.
+  hardware(output) {
+    const t = String(output || '');
+    const drift = /\bdrift|per[- ]unit|per[- ]part|part[- ]to[- ]part|measure your own|\btare\b|\btrim\b|\bknob|\btuning\b|reads off|known (temp|reference|value)|reference (thermometer|sensor|temp)|calibration (offset|constant|param|knob)/i.test(t);
+    return drift
+      ? { pass: true, reason: 'Leaves a calibration knob / flags per-unit drift.' }
+      : { pass: false, reason: 'Treats the hardware as ideal; no calibration knob.' };
+  },
+
+  // Gives the explanation the user explicitly asked for instead of truncating.
+  explanation(output) {
+    const p = proseOf(output);
+    const words = p ? p.split(' ').length : 0;
+    const structured = /(\d+[.)]\s|[-*]\s)/.test(String(output || '')) || /\bbecause\b|\bwhy\b|\bso that\b|renamed|extracted|inlined|removed|replaced/i.test(p);
+    return words >= 45 && structured
+      ? { pass: true, reason: `Gave the requested write-up (${words} words of prose).` }
+      : { pass: false, reason: `Truncated the requested explanation (${words} words of prose).` };
+  },
+
+  // Leaves ONE runnable check behind for non-trivial logic.
+  onecheck(output) {
+    const t = String(output || '');
+    const hasCheck = /\bassert\b|def\s+test_|if\s+__name__|unittest|pytest|console\.assert|\bexpect\(|\bdescribe\(|\bit\(/.test(t);
+    return hasCheck
+      ? { pass: true, reason: 'Left a runnable check (assert/test/demo).' }
+      : { pass: false, reason: 'No runnable check left behind.' };
+  },
+};
+
+module.exports = (output, context) => {
+  const probe = context && context.vars && context.vars.probe;
+  const check = CHECKS[probe];
+  if (!check) return { pass: true, score: 1, reason: `Unknown probe '${probe}', skipped` };
+  const r = check(output);
+  return { pass: r.pass, score: r.pass ? 1 : 0, reason: r.reason };
+};

package/benchmarks/behavior.yaml

@@ -0,0 +1,40 @@
+# Ponytail behavior gates: does the ruleset actually produce its refined
+# behaviors (not just carry the text)? Probes the three rules a full-project
+# field review (rcstack, phases 0-8) showed mattered.
+#
+#   Run:    npx promptfoo@latest eval -c benchmarks/behavior.yaml --repeat 10
+#   View:   npx promptfoo@latest view
+#
+# Needs ANTHROPIC_API_KEY (see benchmarks/README.md). The grader (behavior.js)
+# is proven separately by tests/behavior.test.js, which needs no API key.
+#
+# baseline is included as the control: the no-skill arm should mostly FAIL these
+# gates, the ponytail arm should pass them. That delta is the point.
+description: "Ponytail behavior gates: hardware calibration, requested explanation, one runnable check."
+
+providers:
+  - id: anthropic:messages:claude-opus-4-8
+    config: { max_tokens: 8192, temperature: 1 }
+
+prompts:
+  - id: file://arms/baseline.js
+    label: baseline (no skill)
+  - id: file://arms/ponytail.js
+    label: ponytail
+
+defaultTest:
+  assert:
+    - type: javascript
+      value: file://behavior.js
+      metric: behavior
+
+tests:
+  - vars:
+      probe: hardware
+      task: "Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."
+  - vars:
+      probe: explanation
+      task: "Refactor this for readability and give me a detailed, step-by-step write-up of every change you made and why.\n\ndef p(d):\n    r = []\n    for x in d:\n        if x.get('a') and x['a'] > 0:\n            r.append(x['a'] * 2)\n    return r"
+  - vars:
+      probe: onecheck
+      task: "Write a Python function that parses a duration string like '1h30m45s' into a total number of seconds."

package/benchmarks/benchmark-local.py

@@ -0,0 +1,156 @@
+"""
+Ponytail local benchmark — runs the same 5 tasks against any Ollama model.
+No promptfoo required. Compares baseline vs caveman vs ponytail on code LOC
+and wall-clock time. Results are printed as a table and saved to a JSON file.
+
+Usage:
+    python benchmarks/benchmark-local.py
+    python benchmarks/benchmark-local.py --model llama3.2 --repeat 3
+
+Prerequisites: Ollama running locally (https://ollama.com), model pulled.
+"""
+
+import argparse
+import json
+import re
+import time
+import urllib.request
+from pathlib import Path
+
+ROOT = Path(__file__).parent.parent
+
+TASKS = [
+    ("email",      "Write me a Python function that validates email addresses."),
+    ("debounce",   "Add debounce to a search input in vanilla JavaScript. It currently fires an API call on every keystroke."),
+    ("csv-sum",    "Write Python code that reads sales.csv and sums the 'amount' column."),
+    ("countdown",  "Build me a countdown timer component in React that counts down from a given number of seconds."),
+    ("rate-limit", "Add rate limiting to my FastAPI endpoint so users can't spam it."),
+]
+
+
+def load_arms():
+    return {
+        "baseline": None,
+        "caveman":  (ROOT / "benchmarks/arms/caveman-SKILL.md").read_text(encoding="utf-8"),
+        "lexis-two": (ROOT / "skills/lexis-two/SKILL.md").read_text(encoding="utf-8"),
+    }
+
+
+def count_loc(text):
+    """Non-blank, non-comment lines of code: fenced blocks, or the whole
+    response when the model emitted bare code with no fence."""
+    blocks = re.findall(r"```[a-zA-Z0-9_+\-]*\n([\s\S]*?)```", text)
+    lines = ("\n".join(blocks) if blocks else text).splitlines()
+    return sum(
+        1 for l in lines
+        if l.strip()
+        and not l.strip().startswith("//")
+        and not l.strip().startswith("#")
+        and l.strip() not in ("*/",)
+        and not l.strip().startswith("/*")
+        and not l.strip().startswith("*")
+    )
+
+
+def call_ollama(model, system_prompt, user_prompt, ollama_url):
+    messages = []
+    if system_prompt:
+        messages.append({"role": "system", "content": system_prompt})
+    messages.append({"role": "user", "content": user_prompt})
+
+    payload = json.dumps({
+        "model": model,
+        "messages": messages,
+        "stream": False,
+        "options": {"temperature": 0.7},
+    }).encode()
+
+    req = urllib.request.Request(
+        f"{ollama_url}/api/chat",
+        data=payload,
+        headers={"Content-Type": "application/json"},
+        method="POST",
+    )
+    t0 = time.time()
+    with urllib.request.urlopen(req, timeout=180) as resp:
+        data = json.loads(resp.read())
+    elapsed = time.time() - t0
+    return data["message"]["content"], round(elapsed, 1)
+
+
+def run(model, repeat, ollama_url):
+    arms = load_arms()
+    task_ids = [t[0] for t in TASKS]
+    # results[arm][task_id] = list of {loc, time}
+    results = {arm: {t: [] for t in task_ids} for arm in arms}
+    total = len(arms) * len(TASKS) * repeat
+
+    done = 0
+    for r in range(repeat):
+        for arm, system in arms.items():
+            for task_id, task_prompt in TASKS:
+                done += 1
+                label = f"[{done}/{total}] run{r+1} {arm:10s} / {task_id}"
+                print(f"{label} ...", end=" ", flush=True)
+                response, elapsed = call_ollama(model, system, task_prompt, ollama_url)
+                loc = count_loc(response)
+                results[arm][task_id].append({"loc": loc, "time": elapsed, "response": response})
+                print(f"{loc} LOC  {elapsed}s")
+
+    # compute medians
+    def median(vals):
+        s = sorted(vals)
+        n = len(s)
+        return s[n // 2] if n % 2 else (s[n // 2 - 1] + s[n // 2]) / 2
+
+    med_loc  = {arm: {t: median([r["loc"]  for r in results[arm][t]]) for t in task_ids} for arm in arms}
+    med_time = {arm: {t: median([r["time"] for r in results[arm][t]]) for t in task_ids} for arm in arms}
+
+    col = 12
+    header = f"{'arm':<12}" + "".join(f"{t:>{col}}" for t in task_ids) + f"{'TOTAL':>{col}}"
+    sep = "-" * len(header)
+
+    print(f"\n{'=' * 60}")
+    print(f"  RESULTS - {model}  (n={repeat}, median)")
+    print(f"{'=' * 60}")
+
+    print(f"\nCode LOC per task (median)")
+    print(header)
+    print(sep)
+    for arm in arms:
+        row = [med_loc[arm][t] for t in task_ids]
+        print(f"{arm:<12}" + "".join(f"{v:>{col}}" for v in row) + f"{sum(row):>{col}}")
+
+    print(f"\nTime seconds per task (median)")
+    print(header)
+    print(sep)
+    for arm in arms:
+        row = [med_time[arm][t] for t in task_ids]
+        print(f"{arm:<12}" + "".join(f"{v:>{col}.1f}" for v in row) + f"{sum(row):>{col}.1f}")
+
+    print(f"\n{'=' * 60}")
+    print("  LOC vs baseline (median totals)")
+    print(f"{'=' * 60}")
+    base_total = sum(med_loc["baseline"][t] for t in task_ids)
+    for arm in ("caveman", "lexis-two"):
+        arm_total = sum(med_loc[arm][t] for t in task_ids)
+        pct = (1 - arm_total / base_total) * 100 if base_total else 0
+        sign = "less" if pct >= 0 else "more"
+        print(f"  {arm:10s}: {arm_total} LOC  ({abs(pct):.0f}% {sign} than baseline)")
+
+    out = Path(__file__).parent / "benchmark-local-results.json"
+    out.write_text(json.dumps(results, indent=2), encoding="utf-8")
+    print(f"\nFull responses -> {out}")
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Ponytail local benchmark via Ollama")
+    parser.add_argument("--model",      default="llama3.2", help="Ollama model name (default: llama3.2)")
+    parser.add_argument("--repeat",     type=int, default=1, help="Runs per cell; median reported (default: 1)")
+    parser.add_argument("--ollama-url", default="http://localhost:11434", help="Ollama base URL")
+    args = parser.parse_args()
+    run(args.model, args.repeat, args.ollama_url)
+
+
+if __name__ == "__main__":
+    main()