lcode-cli 0.3.0__tar.gz → 0.3.1__tar.gz

{lcode_cli-0.3.0 → lcode_cli-0.3.1}/.github/ISSUE_TEMPLATE/model_request.yml

@@ -35,6 +35,11 @@ body:
       description: Context size used, tokens/s, whether tool calls (reading, editing, running commands) worked reliably, and anything odd.
     validations:
       required: true
+  - type: textarea
+    id: bench
+    attributes:
+      label: Output of `lcode bench <model> --markdown`
+      description: Optional, but the most useful thing you can add. It takes a few minutes; see https://nasser1941.github.io/lcode/bench/
   - type: textarea
     id: doctor
     attributes:

{lcode_cli-0.3.0 → lcode_cli-0.3.1}/CHANGELOG.md

@@ -6,6 +6,18 @@ All notable changes to lcode are documented here. The format follows
 
 ## [Unreleased]
 
+## [0.3.1] - 2026-09-30
+
+### Added
+
+- `lcode bench` scores models on this machine with eight small coding tasks: fixing a bug, finding
+  code, writing a script from a spec, renaming across files, a one-line edit in a long file,
+  recovering from failing commands, fixing a function to match its spec and adding a feature across
+  files. Each task runs in a temporary folder and has an automatic check, some with hidden tests.
+  It reports tasks passed, time, generation and prompt speed, tool-call errors and memory use.
+  `lcode bench model-a model-b` compares models, `--json` saves the results (a documented, versioned
+  format) and `--markdown` prints a table for a model test report.
+
 ## [0.3.0] - 2026-09-30
 
 ### Added
@@ -128,7 +140,8 @@ First public release.
 - `AGENTS.md` project instructions and `/init` to generate them.
 - One-line installer for Linux and macOS.
 
-[Unreleased]: https://github.com/nasser1941/lcode/compare/v0.3.0...HEAD
+[Unreleased]: https://github.com/nasser1941/lcode/compare/v0.3.1...HEAD
+[0.3.1]: https://github.com/nasser1941/lcode/compare/v0.3.0...v0.3.1
 [0.3.0]: https://github.com/nasser1941/lcode/compare/v0.2.2...v0.3.0
 [0.2.2]: https://github.com/nasser1941/lcode/compare/v0.2.1...v0.2.2
 [0.2.1]: https://github.com/nasser1941/lcode/compare/v0.2.0...v0.2.1

{lcode_cli-0.3.0 → lcode_cli-0.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: lcode-cli
-Version: 0.3.0
+Version: 0.3.1
 Summary: A local-first terminal coding agent powered by open-weight models on your own GPU or Mac, via Ollama.
 Project-URL: Homepage, https://nasser1941.github.io/lcode/
 Project-URL: Documentation, https://nasser1941.github.io/lcode/
@@ -73,7 +73,8 @@ through [Ollama](https://ollama.com), and when it needs current information it c
   plans multi-step tasks with a visible todo list.
 - **Long context.** 256K tokens with the default model (1M with Nemotron). You choose the window.
 - **Hardware-aware.** Detects your NVIDIA GPU or Apple Silicon Mac and picks the best model and the
-  largest context that fits: `lcode setup` does it in one step.
+  largest context that fits: `lcode setup` does it in one step. `lcode bench` compares models on
+  real coding tasks on your machine.
 - **Safe by default.** Every edit is shown as a diff and every command that isn't read-only needs
   your approval. `auto-edit` and `yolo` modes when you want speed.
 - **Undo.** lcode saves a checkpoint before the model changes files; `/undo` takes back the last
@@ -125,6 +126,7 @@ lcode
 | `lcode -c` / `lcode --resume` | continue the last session here / pick a saved session from a list |
 | `lcode --model qwen3.5-9b --context 128k` | pick a model and context window for this session |
 | `lcode models` / `lcode doctor` | what fits this machine / check the installation |
+| `lcode bench qwen3.6-35b qwen3.5-9b` | compare models on small coding tasks on this machine |
 | `/undo`, `/rewind` | take back the last request's file changes, or go back further |
 | `/rename`, `/resume` | name the current session, resume a saved one |
 | `/model`, `/context`, `/compact`, `/help` | switch model, resize the context window (pick from a list), summarize, list commands |
@@ -170,7 +172,6 @@ fills up. Details: [How it works](https://nasser1941.github.io/lcode/how-it-work
 
 See the pinned [Roadmap issue](https://github.com/nasser1941/lcode/issues/10). Next up:
 
-- [`lcode bench`: score models on your own hardware](https://github.com/nasser1941/lcode/issues/5)
 - [MCP tool servers](https://github.com/nasser1941/lcode/issues/6)
 - [Homebrew formula](https://github.com/nasser1941/lcode/issues/8) and an [optional sandbox for shell commands](https://github.com/nasser1941/lcode/issues/9)
 - More models in the catalog, and test reports from your hardware ([report one](https://github.com/nasser1941/lcode/issues/new?template=model_request.yml))

{lcode_cli-0.3.0 → lcode_cli-0.3.1}/README.md

@@ -40,7 +40,8 @@ through [Ollama](https://ollama.com), and when it needs current information it c
   plans multi-step tasks with a visible todo list.
 - **Long context.** 256K tokens with the default model (1M with Nemotron). You choose the window.
 - **Hardware-aware.** Detects your NVIDIA GPU or Apple Silicon Mac and picks the best model and the
-  largest context that fits: `lcode setup` does it in one step.
+  largest context that fits: `lcode setup` does it in one step. `lcode bench` compares models on
+  real coding tasks on your machine.
 - **Safe by default.** Every edit is shown as a diff and every command that isn't read-only needs
   your approval. `auto-edit` and `yolo` modes when you want speed.
 - **Undo.** lcode saves a checkpoint before the model changes files; `/undo` takes back the last
@@ -92,6 +93,7 @@ lcode
 | `lcode -c` / `lcode --resume` | continue the last session here / pick a saved session from a list |
 | `lcode --model qwen3.5-9b --context 128k` | pick a model and context window for this session |
 | `lcode models` / `lcode doctor` | what fits this machine / check the installation |
+| `lcode bench qwen3.6-35b qwen3.5-9b` | compare models on small coding tasks on this machine |
 | `/undo`, `/rewind` | take back the last request's file changes, or go back further |
 | `/rename`, `/resume` | name the current session, resume a saved one |
 | `/model`, `/context`, `/compact`, `/help` | switch model, resize the context window (pick from a list), summarize, list commands |
@@ -137,7 +139,6 @@ fills up. Details: [How it works](https://nasser1941.github.io/lcode/how-it-work
 
 See the pinned [Roadmap issue](https://github.com/nasser1941/lcode/issues/10). Next up:
 
-- [`lcode bench`: score models on your own hardware](https://github.com/nasser1941/lcode/issues/5)
 - [MCP tool servers](https://github.com/nasser1941/lcode/issues/6)
 - [Homebrew formula](https://github.com/nasser1941/lcode/issues/8) and an [optional sandbox for shell commands](https://github.com/nasser1941/lcode/issues/9)
 - More models in the catalog, and test reports from your hardware ([report one](https://github.com/nasser1941/lcode/issues/new?template=model_request.yml))

lcode_cli-0.3.1/docs/bench.md

@@ -0,0 +1,172 @@
+# Benchmarking models
+
+`lcode bench` answers the most common question about local models: *which one works best for
+coding on my machine?* It runs a fixed set of small coding tasks against one or more models and
+reports how many they solved, how fast they ran and how much memory they used.
+
+```bash
+lcode bench                           # the configured model
+lcode bench qwen3.6-35b qwen3.5-9b    # compare models
+lcode bench --tasks fix-bug,rename    # run some of the tasks
+lcode bench --json results.json       # also save machine-readable results
+lcode bench --markdown                # also print a table to paste into a test report
+```
+
+```text
+lcode bench · NVIDIA GeForce RTX 4080 Laptop GPU (12 GB VRAM), 31 GB RAM · Ollama 0.32.15
+
+lcode-qwen3.6-35b · 32K context · reasoning on · 8 tasks
+  Loaded in 16s · 22.3 GB, 45% on the GPU · reads prompts at 855 tok/s
+  ✓ fix-bug       Fix a bug so the tests pass         22s  the tests pass
+  ✓ find-code     Answer with file:line                9s  next_wait at backoff.py:13
+  ✓ write-script  Write a script from a spec          13s  correct output, also on data it hadn't seen
+  ✓ rename        Rename a function across files      26s  its tests and extra checks pass
+  ✓ precise-edit  One-line edit in a long file        10s  changed only service_087's timeout
+  ✓ recover       Recover from failing commands       30s  used --source and dropped the invalid line
+  ✓ spec-fix      Fix a function to match its spec    55s  follows the whole docstring
+  ✓ feature       Add a feature across files          71s  priorities work, including old todos
+```
+
+At the end it prints a table comparing the models (see [results](#results-on-a-12-gb-laptop) below).
+
+A model usually takes 4 to 12 minutes; each task stops after 5 minutes (`--timeout`).
+
+## The tasks
+
+Every task runs in a new temporary folder, with every permission granted (`yolo` mode) and web
+access off, and ends with an automatic check. The folders are deleted afterwards unless you pass
+`--keep`, which also keeps each task's transcript (`lcode-bench.log`).
+
+| Task | The model has to | Passes when |
+|---|---|---|
+| `fix-bug` | Find and fix a bug so the failing unit tests pass | The tests pass and the test file is unchanged |
+| `find-code` | Answer a question about a small codebase | It names the right function and cites its `file:line` |
+| `write-script` | Write a script from a spec and run it | The script prints the right output, also for data it never saw |
+| `rename` | Rename a function across several files and tests | The old name is gone and extra hidden tests pass |
+| `precise-edit` | Change one value in a 900-line file of near-identical entries | Exactly that one line changed |
+| `recover` | Run a command that fails twice (a removed option, a bad input line) and work around it | The report is correct and only the bad line was dropped |
+| `spec-fix` | Fix a function from a one-line bug report so it does everything its docstring says | Hidden tests of every rule in the docstring pass, edge cases included |
+| `feature` | Add priorities to a small todo app (a new option, sorting, a marker in the output) | Its tests pass, and a scripted session gives the exact expected output, including for todos saved before the change |
+
+The first six tasks are everyday work that good models get right; the last two need careful
+reading and are where smaller models tend to slip. The checks are deterministic and can't be passed
+by hard-coding: for example, `write-script` reruns the script on new data, and `rename`, `spec-fix`
+and `feature` are checked with tests the model never saw.
+
+## What's measured
+
+| | |
+|---|---|
+| **Passed** | Tasks solved within the time limit |
+| **Time** | Total time for the tasks (loading the model is reported separately) |
+| **Generation** | Output tokens per second, across all the model's responses |
+| **Prompt reading** | Prompt tokens per second: how fast the model reads files and tool output. Measured per model with fresh 4,000-token prompts (the faster of two readings, since the first one after loading includes warm-up), because during the tasks Ollama serves most of each request from its prompt cache |
+| **Tool-call errors** | Tool calls that failed (wrong arguments, editing without reading first, unmatched edits) or that the model wrote as invalid JSON |
+| **Memory** | The model's memory use and how much of it is on the GPU, from Ollama |
+
+Results depend on the context window, so `lcode bench` uses **32K** by default on every machine,
+which makes results comparable. When comparing models, each one is unloaded before the next starts
+so every model gets the whole GPU; lcode warns if other models are already loaded. Use `--context 128k` to measure a larger window, and `--no-think`
+to measure a model without reasoning.
+
+## Results on a 12 GB laptop
+
+RTX 4080 Laptop GPU (12 GB) with 31 GB of RAM, Ollama 0.32.15, 32K context, reasoning on:
+
+| | `qwen3.6-35b` | `qwen3.5-9b` | `qwen3.5-4b` |
+|---|---|---|---|
+| fix-bug | ✓ 22s | ✓ 16s | ✓ 11s |
+| find-code | ✓ 9s | ✓ 6s | ✓ 5s |
+| write-script | ✓ 13s | ✓ 44s | ✗ 18s |
+| rename | ✓ 26s | ✓ 28s | ✓ 30s |
+| precise-edit | ✓ 10s | ✓ 12s | ✓ 8s |
+| recover | ✓ 30s | ✗ 29s | ✓ 24s |
+| spec-fix | ✓ 55s | ✓ 4m 40s | ✗ 2m 57s |
+| feature | ✓ 1m 11s | ✗ 5m 00s | ✓ 47s |
+| **Passed** | **8/8** | **6/8** | **6/8** |
+| Time | 3m 57s | 11m 55s | 5m 20s |
+| Generation | 71.4 tok/s | 60.1 tok/s | 91.3 tok/s |
+| Prompt reading | 855 tok/s | 3,507 tok/s | 5,186 tok/s |
+| Tool-call errors | 0 | 9 | 1 |
+| Memory | 22.3 GB, 45% on GPU | 6.6 GB, 100% on GPU | 4.2 GB, 100% on GPU |
+
+The small models fit entirely on the GPU and read prompts much faster, but the larger mixture-of-
+experts model solves more and needs fewer attempts. The small models' failures were real mistakes:
+copying `read_file`'s line numbers into a data file, ignoring negative numbers, missing an edge case
+of the spec, and running out of time.
+
+Models sample their answers, so a task near the edge of a model's ability can pass in one run and
+fail in the next (`qwen3.6-35b` scored 7/8 in an earlier run). Compare totals rather than single
+tasks, and run twice before drawing conclusions.
+
+## Sharing results
+
+Results from other machines, especially Apple Silicon Macs and other GPUs, decide which models lcode
+recommends. Run
+
+```bash
+lcode bench qwen3.6-35b --markdown
+```
+
+and paste the table into a
+[model test report](https://github.com/nasser1941/lcode/issues/new?template=model_request.yml).
+
+## JSON format
+
+`--json FILE` writes one document per run. The schema is versioned: fields may be added, but
+existing fields keep their meaning until `schema` changes.
+
+```json
+{
+  "schema": 1,
+  "lcode": "0.3.1",
+  "ollama": "0.32.15",
+  "date": "2026-09-30T19:09:09+02:00",
+  "hardware": {
+    "os": "linux", "cpu": "13th Gen Intel(R) Core(TM) i9-13980HX", "ram_gib": 30.96,
+    "gpu": "NVIDIA GeForce RTX 4080 Laptop GPU", "vram_gib": 11.99, "unified": false,
+    "description": "NVIDIA GeForce RTX 4080 Laptop GPU (12 GB VRAM), 31 GB RAM"
+  },
+  "tasks": [{"id": "fix-bug", "title": "Fix a bug so the tests pass"}, …],
+  "runs": [
+    {
+      "model": "qwen3.6-35b",
+      "ollama_model": "lcode-qwen3.6-35b",
+      "context": 32768,
+      "num_batch": null,
+      "think": true,
+      "load_seconds": 16.2,
+      "prompt_tps": 854.6,
+      "memory_gb": 22.3,
+      "gpu_percent": 45,
+      "passed": 8,
+      "total": 8,
+      "seconds": 237.3,
+      "generation_tps": 71.4,
+      "tool_errors": 0,
+      "usage": {"requests": 56, "prompt_tokens": 221522, "prompt_ns": 73377987000,
+                "output_tokens": 11490, "output_ns": 161017657999},
+      "tasks": [
+        {"id": "fix-bug", "passed": true, "seconds": 21.9, "detail": "the tests pass",
+         "steps": 5, "tool_errors": 0, "error": null},
+        …
+      ],
+      "error": null,
+      "interrupted": false
+    }
+  ]
+}
+```
+
+| Field | Meaning |
+|---|---|
+| `runs[].model` | The name you gave (catalog key or Ollama tag); `ollama_model` is the model that ran |
+| `runs[].context`, `num_batch`, `think` | The settings used. lcode lowers the batch size or context if the GPU runs out of memory, and reports what it ended up using |
+| `runs[].memory_gb`, `gpu_percent` | From Ollama's `/api/ps`; `null` if unavailable |
+| `runs[].generation_tps` | Output tokens per second across the tasks; `null` if Ollama reported no timings |
+| `runs[].prompt_tps` | Prompt tokens per second for a fresh 4,000-token prompt (the faster of two readings); `null` if it couldn't be measured |
+| `runs[].usage` | Totals over the tasks as Ollama reports them. `prompt_tokens` includes tokens served from the prompt cache |
+| `runs[].tasks[].steps` | Model responses in the task |
+| `runs[].tasks[].error` | Why the task stopped early (`timed out after 300s`, an Ollama error), otherwise `null` |
+| `runs[].error` | Why the model didn't run at all (for example, not installed), otherwise `null` |
+| `runs[].interrupted` | You pressed ++ctrl+c++; the results cover the tasks that finished |

{lcode_cli-0.3.0 → lcode_cli-0.3.1}/docs/how-it-works.md

@@ -75,6 +75,7 @@ the Ollama server you configure, it only contacts the web when the model searche
 | `agent.py` | System prompt, streaming, tool loop, compaction, sessions |
 | `tools.py` | Tool schemas and implementations |
 | `permissions.py` | Approval prompts and the read-only allowlist |
+| `bench.py` | `lcode bench`: the benchmark tasks, their checks and the reports |
 | `checkpoints.py` | Snapshots before the model changes files, for `/undo` and `/rewind` |
 | `catalog.py`, `models.toml` | Model catalog and memory estimates |
 | `hardware.py` | NVIDIA / Apple Silicon detection |

{lcode_cli-0.3.0 → lcode_cli-0.3.1}/docs/models.md

@@ -17,8 +17,8 @@ size, and it works with any other Ollama model that supports tool calling.
 | `qwen3.5-4b` | Qwen3.5 4B | dense | 3.4 GB | 256K | tested |
 
 *Tested* means the model passed lcode's two acceptance tasks end to end (see
-[test results](#test-results)). Results on other hardware are very welcome: please
-[report how a model does](https://github.com/nasser1941/lcode/issues/new?template=model_request.yml).
+[test results](#test-results)). Results on other hardware are very welcome: run
+[`lcode bench`](bench.md) and [report how a model does](https://github.com/nasser1941/lcode/issues/new?template=model_request.yml).
 
 Run `lcode models` to see the same list with what fits on **your** machine:
 

{lcode_cli-0.3.0 → lcode_cli-0.3.1}/docs/quickstart.md

@@ -12,7 +12,7 @@ lcode
 
 ```text
 ╭──────────────────────────────────────────────╮
-│ lcode v0.3.0 — local coding agent            │
+│ lcode v0.3.1 — local coding agent            │
 │                                              │
 │ model    lcode-qwen3.6-35b                   │
 │ context  256K tokens                         │

{lcode_cli-0.3.0 → lcode_cli-0.3.1}/docs/usage.md

@@ -8,6 +8,7 @@ lcode setup [model]          # pick, download and configure a model
 lcode models                 # the model catalog and what fits this machine
 lcode doctor                 # check the installation (include this in bug reports)
 lcode config [set|unset]     # show or change settings
+lcode bench [models]         # score models on small coding tasks on this machine
 ```
 
 | Option | Meaning |

{lcode_cli-0.3.0 → lcode_cli-0.3.1}/mkdocs.yml

@@ -93,6 +93,7 @@ nav:
   - Guide:
       - Using lcode: usage.md
       - Models & context windows: models.md
+      - Benchmarking models: bench.md
       - Configuration: configuration.md
   - Reference:
       - How it works: how-it-works.md

{lcode_cli-0.3.0 → lcode_cli-0.3.1}/src/lcode/__init__.py

@@ -1,3 +1,3 @@
 """lcode — a local-first terminal coding agent powered by open-weight models via Ollama."""
 
-__version__ = "0.3.0"
+__version__ = "0.3.1"

{lcode_cli-0.3.0 → lcode_cli-0.3.1}/src/lcode/agent.py

@@ -154,6 +154,7 @@ class Agent:
         self.session_title = ""
         self.ctx_used = 0
         self.last_speed = 0.0
+        self.usage = {"requests": 0, "prompt_tokens": 0, "prompt_ns": 0, "output_tokens": 0, "output_ns": 0}
         self.messages: list[dict] = []
         self.reset()
 
@@ -390,6 +391,11 @@ class Agent:
             message["tool_calls"] = tool_calls
         self.messages.append(message)
         if final:
+            self.usage["requests"] += 1
+            self.usage["prompt_tokens"] += final.get("prompt_eval_count", 0)
+            self.usage["prompt_ns"] += final.get("prompt_eval_duration", 0)
+            self.usage["output_tokens"] += final.get("eval_count", 0)
+            self.usage["output_ns"] += final.get("eval_duration", 0)
             self.ctx_used = final.get("prompt_eval_count", 0) + final.get("eval_count", 0)
             seconds = final.get("eval_duration", 0) / 1e9
             self.last_speed = final.get("eval_count", 0) / seconds if seconds else 0.0