rulelab-cli 0.1.2__tar.gz

rulelab_cli-0.1.2/.gitignore

@@ -0,0 +1,12 @@
+__pycache__/
+*.py[cod]
+.venv/
+dist/
+build/
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+.env
+rulelab-results/
+rulelab-demo*/
+.DS_Store

rulelab_cli-0.1.2/CHANGELOG.md

@@ -0,0 +1,42 @@
+# Changelog
+
+All notable changes to this project are documented here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses
+[Semantic Versioning](https://semver.org/).
+
+## [0.1.2] - 2026-11-04
+
+### Changed
+- Published on PyPI as `rulelab-cli` (the name `rulelab` is too close to an existing project). The command is still `rulelab`; `rulelab-cli` works too, so `uvx rulelab-cli` runs it.
+- Depends on `commitarena==0.1.1` from PyPI instead of its GitHub tag.
+
+### Added
+- Claude Code plugin (`/plugin install rulelab --marketplace diegoperezg7/rulelab`) with the `ablate-agent-rules` skill.
+
+## [0.1.1] - 2026-11-03
+
+### Fixed
+- Report: numbers follow the active language (ES `1.953`, `0,23 US$`; EN `1,953`, `0.23 USD`).
+- Report: the cost-projection footnote is fully translated.
+- Report, terminal summary and docs: the estimated tokens per run (file resent on every call) and the measured
+  difference against no file are shown as two separate, named figures.
+
+## [0.1.0] - 2026-11-03
+
+### Added
+- `rulelab split`: split AGENTS.md / CLAUDE.md into rules by section or by bullet.
+- `rulelab run`: leave-one-out experiment (no file, full file, full file minus each rule) on real
+  tasks mined from a repository's history, through the commitarena engine (no-network Docker
+  sandbox), 3 repeats by default.
+- Per-rule verdict (helps / no effect / hurts / inconclusive) from a two-level bootstrap interval,
+  tokens per rule and projected monthly cost.
+- Plan and cost estimate before running; `--max-cost` (default 1 USD) checked before every run,
+  partial results kept, `--resume` to continue.
+- Static HTML report (EN/ES, light/dark, mobile) with the annotated file.
+- `--concurrency` to run several agent attempts at once (each in its own sandbox).
+- `rulelab demo`: a recorded real run on 4 tasks of the commitarena demo set (ministral-14b, 64 runs).
+
+### Notes
+- rulelab depends on `commitarena` from its GitHub tag (`v0.1.1`). PyPI rejects direct URL
+  dependencies, so that dependency moves to the PyPI release once commitarena is published there,
+  and rulelab is published to PyPI after that.

rulelab_cli-0.1.2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Diego Pérez
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

rulelab_cli-0.1.2/PKG-INFO

@@ -0,0 +1,260 @@
+Metadata-Version: 2.5
+Name: rulelab-cli
+Version: 0.1.2
+Summary: Find out which lines of your AGENTS.md earn their tokens. Remove one rule at a time and measure success and cost on real tasks.
+Project-URL: Homepage, https://diegoperezg7.github.io/rulelab/
+Project-URL: Repository, https://github.com/diegoperezg7/rulelab
+Project-URL: Issues, https://github.com/diegoperezg7/rulelab/issues
+Author: Diego Pérez
+License-Expression: MIT
+License-File: LICENSE
+Keywords: ablation,agents,agents-md,claude-md,coding-agents,evaluation,llm
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Requires-Dist: commitarena==0.1.1
+Requires-Dist: jinja2==3.1.6
+Requires-Dist: rich==15.0.0
+Requires-Dist: typer==0.27.3
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <img src="docs/assets/banner.svg" alt="rulelab" width="560">
+</p>
+
+<p align="center">
+  <a href="https://github.com/diegoperezg7/rulelab/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/diegoperezg7/rulelab/actions/workflows/ci.yml/badge.svg"></a>
+  <a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-5d51d1"></a>
+  <a href="https://github.com/diegoperezg7/rulelab/releases"><img alt="Release" src="https://img.shields.io/github/v/release/diegoperezg7/rulelab?color=5d51d1"></a>
+  <a href="https://github.com/diegoperezg7/rulelab/stargazers"><img alt="Stars" src="https://img.shields.io/github/stars/diegoperezg7/rulelab?style=flat&color=5d51d1"></a>
+</p>
+
+<p align="center"><b>Find out which lines of your AGENTS.md earn their tokens.</b><br>
+Remove one rule at a time and measure success and cost on real tasks.</p>
+
+<p align="center"><a href="README.es.md">Leer en español</a> · <a href="https://diegoperezg7.github.io/rulelab/">Example report</a></p>
+
+<p align="center"><img src="docs/assets/demo.gif" alt="rulelab split and rulelab demo in a terminal" width="820"></p>
+
+<p align="center"><a href="https://diegoperezg7.github.io/rulelab/"><img src="docs/assets/report-light.png" alt="rulelab HTML report" width="820"></a></p>
+
+## Why
+
+Every line in `AGENTS.md` or `CLAUDE.md` is sent to the model on every call. Is it worth it?
+
+The best evidence so far says "often not". Gloaguen et al. (ETH Zurich and LogicStar.ai,
+[arXiv 2602.11988](https://arxiv.org/abs/2602.11988)) ran Claude Code, Codex and Qwen Code on
+SWE-bench Lite and on CTXbench (138 tasks from 12 repositories) and found that context files
+**do not generally improve task success rates** while raising inference cost **by over 20% on
+average**:
+
+| Context file | Success | Inference cost |
+| --- | --- | --- |
+| Written by an LLM | −0.5% (SWE-bench Lite), −2% (CTXbench) | +20%, +23% |
+| Written by developers | +2.4% (not significant) | up to +19% |
+
+They also found that repository overviews do not help. That is the average over many repos.
+Your file, your repo and your model are what matter to you, and the paper measures whole files,
+not the individual rules you would actually delete.
+
+rulelab runs that experiment on **your** repo, **one rule at a time**.
+
+| | What it measures | Tasks |
+| --- | --- | --- |
+| [fernforge/ruleprof](https://github.com/fernforge/ruleprof) (0★) | Removes rules one at a time; checks whether the agent **obeys** each rule | Toy tasks |
+| [sx4im/skillcheck](https://github.com/sx4im/skillcheck) (29★) | Blind A/B test of the **whole file** against no file | Your prompts |
+| **rulelab** | Removes rules one at a time; measures **task success and cost** per rule | Real bug fixes from your repo's history, scored by its own tests |
+
+Star counts checked with the GitHub API on 3 Nov 2026.
+
+## Quickstart
+
+See a recorded real run (no API key, no Docker, under a minute):
+
+```bash
+uvx rulelab-cli demo --open
+```
+
+or from a clone:
+
+```bash
+git clone https://github.com/diegoperezg7/rulelab && cd rulelab && uv run rulelab demo --open
+```
+
+The PyPI package is `rulelab-cli` (the name `rulelab` is too close to an existing project there); the command is `rulelab`. Install it with `uv tool install rulelab-cli` or `pipx install rulelab-cli`.
+
+Then run it on your own file and repo (needs Docker, and Ollama or an API key):
+
+```bash
+rulelab split AGENTS.md                                   # see how the file is split, free
+rulelab run AGENTS.md --repo . --dry-run                  # plan and cost estimate, free
+rulelab run AGENTS.md --repo . --model ollama/qwen3:4b    # local model, 0 USD
+rulelab run AGENTS.md --repo . --model mistral/ministral-14b-latest --price 0.2,0.2 -j 3
+```
+
+## Example output
+
+```text
+$ rulelab demo
+Recorded run · real results from rulelab run on 2026-11-03 · rulelab 0.1.0 · Darwin arm64 (Apple M4, 16 GB RAM)
+AGENTS.md · model mistral/ministral-14b-latest · 4 tasks × 2 repeats × 8 configs · 64 runs · 0.7376 USD
+full file 2/8 · no file 1/8 · file effect +12 pp [−21, +39]
+file tokens per run, estimated if resent on every call: ≈4,305 · measured difference vs no file: +776 [−15,405, +17,616]
+ rule                    ≈tok   effect (pp, 95% CI)   verdict        USD/month
+ R1 Repository overview    88        +12 [−21, +41]   inconclusive        0.23
+ R2 Running tests          47          0 [−26, +25]   inconclusive        0.12
+ R3 How to fix a bug       56        +12 [−20, +40]   inconclusive        0.15
+ R4 Lexer and keywords     68          0 [−26, +27]   inconclusive        0.18
+ R5 Code style             34          0 [−26, +27]   inconclusive        0.09
+ R6 Communication          30          0 [−26, +26]   inconclusive        0.08
+Small sample: most rules are inconclusive. [...]
+```
+
+**What this recorded run says, plainly.** The demo is a real `rulelab run` of a 6-rule
+[AGENTS.md](examples/demo/AGENTS.md) (it includes a repository overview, as in the paper) on the 4
+most recent tasks of commitarena's own demo set: bug fixes from
+[sqlparse](https://github.com/andialbrecht/sqlparse), scored by sqlparse's tests
+([tasks.json](examples/demo/tasks.json)). Model `mistral/ministral-14b-latest`, 15 agent steps,
+2 repeats, commitarena 0.1.1: 64 agent runs, 27 minutes, 0.74 USD. The model was picked with a
+4-run probe because it passed some tasks but not all (1 of 4), so the baseline is neither 0% nor
+100% and an effect could show up.
+
+13 of 64 runs passed. The full file passed 2/8 and no file 1/8 (+12 pp, interval −21 to +39).
+Removing the repository overview or the bug-fixing rule dropped success from 2/8 to 1/8, but with
+4 tasks and 2 repeats that is one run's difference: **every rule is inconclusive**, and the report
+says so instead of forcing a verdict. What the run does pin down is the price: the file is ≈328
+tokens; resent on each of ~13 model calls it would add an estimated ≈4,305 input tokens per run. The difference actually measured in this run (full file vs no file) was +776 input tokens per run, with a wide interval, because agent trajectories vary far more than that. To get a verdict,
+run more tasks and repeats on your own repo.
+
+An earlier, larger run (144 runs with codestral on an older commitarena build) solved 1 of 144
+tasks and is not published as a result; it did prove that the run survives engine errors, budget
+stops and interruptions (`--resume`).
+
+The full report, with the annotated file, is published at
+**[diegoperezg7.github.io/rulelab](https://diegoperezg7.github.io/rulelab/)** (EN/ES, light/dark).
+Screenshots: [light](docs/assets/report-light.png) · [dark](docs/assets/report-dark.png) ·
+[mobile](docs/assets/report-mobile.png).
+
+## How it works
+
+```mermaid
+flowchart LR
+  F[AGENTS.md / CLAUDE.md] --> S[split into rules<br/>by section or bullet]
+  S --> C["configurations:<br/>no file · full file · full minus R1 … Rn"]
+  R[your repo] --> M[commitarena mine:<br/>real bug-fix commits with tests]
+  C --> X{{each config × each task × 3 repeats}}
+  M --> X
+  X --> D[Docker sandbox, no network:<br/>agent edits code, model called from host]
+  D --> T[repo's own tests decide pass/fail]
+  T --> B[bootstrap interval per rule:<br/>helps · no effect · hurts · inconclusive]
+  B --> H[HTML report + results.json]
+```
+
+1. **Split.** `--granularity section` (default) makes each heading a rule; `bullet` makes each
+   top-level bullet or paragraph a rule. A lone `# Title` line is kept in every configuration.
+   Check the split with `rulelab split`.
+2. **Configurations.** No file, the full file, and the full file without each rule
+   (leave-one-out). With *n* rules that is *n* + 2 configurations.
+3. **Tasks.** Real bug fixes from your git history, mined by
+   [commitarena](https://github.com/diegoperezg7/commitarena): commits that add a test that fails
+   before the fix. The agent gets the failing tests, never the commit message.
+4. **Runs.** Every configuration runs every task `--repeats` times (default 3) in the same
+   no-network Docker sandbox (2 CPU, 2 GB). The file is written at the repo root and appended to
+   the system prompt (`--inject both|repo|prompt`). The repo's own AGENTS.md/CLAUDE.md is removed
+   first, so "no file" really means none.
+5. **Verdict.** Effect of a rule = success with the full file − success without it. The 95%
+   interval comes from a two-level bootstrap (tasks resampled; each per-task rate drawn from a
+   Beta(s+½, f+½) posterior, so 3/3 vs 3/3 does not look like certainty). **Helps** if the whole
+   interval is above 0, **hurts** if below, **no effect** if it fits inside ±10 pp (`--margin`),
+   **inconclusive** otherwise.
+6. **Cost.** Tokens per rule (≈ characters / 4), the measured input-token difference per run, and
+   a monthly projection: rule tokens × model calls per run × sessions per month × input price.
+
+### Read the result honestly
+
+With 6 tasks and 3 repeats, only large effects can be told apart from noise, so **most rules
+will come out inconclusive**. The report says so in plain words instead of forcing a verdict.
+"Inconclusive" is not "useless": it means this experiment cannot tell. Add tasks (`--max-tasks`)
+or repeats before deleting a rule because of it.
+
+## Configuration
+
+| Option | Default | What it does |
+| --- | --- | --- |
+| `--repo` | | Git repo (path or URL) to mine tasks from |
+| `--tasks-file` | | Reuse a `tasks.json` from `commitarena mine` |
+| `--only` | all | Comma-separated task ids |
+| `--max-tasks` | 6 | Tasks to mine with `--repo` |
+| `--model`, `-m` | `ollama/qwen3:4b` | `ollama/…`, `mistral/…`, `openrouter/…`, `openai/…` |
+| `--granularity`, `-g` | `section` | `section` or `bullet` |
+| `--repeats`, `-n` | 3 | Runs per task and configuration |
+| `--max-cost` | 1.0 | Hard budget in USD, checked before every run. Local models count as 0 |
+| `--inject` | `both` | Put the file in the repo root, the system prompt, or both |
+| `--max-steps` | 30 | Model calls per run |
+| `--concurrency`, `-j` | 1 | Agent runs at the same time, each in its own sandbox |
+| `--price` | | `IN,OUT` USD per 1M tokens for models without a known price |
+| `--price-source` | | Where that price comes from; shown in the report |
+| `--sessions-per-month` | 1000 | For the monthly cost projection |
+| `--out`, `-o` | `rulelab-results` | Writes `results.json` (after every run) and `report.html` |
+| `--resume` | | Continue an interrupted or budget-stopped run |
+| `--dry-run` | | Print the plan and estimate only |
+
+Before anything runs, rulelab prints the number of agent runs and the estimated cost. If the
+estimate is over `--max-cost`, it stops and tells you the amount to pass instead. Nothing is spent
+until you allow it.
+
+**Small local models.** Ollama serves models with a 4,096-token context by default. An agent run
+needs far more, and in our test `qwen3:4b` used up that window thinking and never ran a command.
+Start Ollama with a bigger window (for example `OLLAMA_CONTEXT_LENGTH=32768 ollama serve`) before
+using a local model.
+
+**Keys** come only from environment variables: `MISTRAL_API_KEY`, `OPENROUTER_API_KEY`,
+`OPENAI_API_KEY` (see [.env.example](.env.example)). Ollama needs none (`OLLAMA_HOST` to point
+elsewhere). rulelab sends no telemetry.
+
+`rulelab report results.json` rebuilds the report with another `--margin`, `--price-in` or
+`--sessions-per-month` without re-running anything.
+
+## Use it from Claude Code
+
+Install the plugin and ask Claude in plain words; its `ablate-agent-rules` skill splits your file, shows the free cost estimate, runs the experiment only after you approve it and explains which rules help, hurt or cannot be told apart yet.
+
+```text
+/plugin install rulelab --marketplace diegoperezg7/rulelab
+```
+
+The skill tells Claude to ask before any paid run, and `--max-cost` still caps the spend.
+
+## Roadmap
+
+Not in v0.1:
+
+- Synthetic tasks for repos without enough bug-fix history.
+- Adherence checks (did the agent follow the rule?), as a second signal next to success.
+- `.cursor/rules` and skills.
+- Rule interactions (removing pairs or groups).
+- Automatic rewrites of rules that cost more than they help.
+
+## Part of a series
+
+Open source tools that evaluate agents and LLMs on your own data:
+
+- [memarena](https://github.com/diegoperezg7/memarena): Benchmark memory layers on your own conversations.
+- [parsearena](https://github.com/diegoperezg7/parsearena): Compare PDF parsers side by side and their effect on retrieval.
+- [commitarena](https://github.com/diegoperezg7/commitarena): Replay real bug fixes from your git history against any model.
+- [toolbloat](https://github.com/diegoperezg7/toolbloat): Measure the token cost of your MCP stack and tool-choice errors.
+- [doesitfire](https://github.com/diegoperezg7/doesitfire): Test whether your SKILL.md activates in Claude Code, Codex and OpenCode.
+
+## License
+
+[MIT](LICENSE) © 2026 Diego Pérez. The demo tasks contain code and tests from
+[andialbrecht/sqlparse](https://github.com/andialbrecht/sqlparse) (BSD-3-Clause, see
+[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md)), taken from commitarena's demo set; the demo
+AGENTS.md in [`examples/demo`](examples/demo) was written for this project.

rulelab_cli-0.1.2/README.es.md

@@ -0,0 +1,235 @@
+<p align="center">
+  <img src="docs/assets/banner.svg" alt="rulelab" width="560">
+</p>
+
+<p align="center">
+  <a href="https://github.com/diegoperezg7/rulelab/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/diegoperezg7/rulelab/actions/workflows/ci.yml/badge.svg"></a>
+  <a href="LICENSE"><img alt="Licencia: MIT" src="https://img.shields.io/badge/license-MIT-5d51d1"></a>
+  <a href="https://github.com/diegoperezg7/rulelab/releases"><img alt="Release" src="https://img.shields.io/github/v/release/diegoperezg7/rulelab?color=5d51d1"></a>
+  <a href="https://github.com/diegoperezg7/rulelab/stargazers"><img alt="Estrellas" src="https://img.shields.io/github/stars/diegoperezg7/rulelab?style=flat&color=5d51d1"></a>
+</p>
+
+<p align="center"><b>Descubre qué líneas de tu AGENTS.md se ganan sus tokens.</b><br>
+Quita una regla cada vez y mide acierto y coste en tareas reales.</p>
+
+<p align="center"><a href="README.md">Read in English</a> · <a href="https://diegoperezg7.github.io/rulelab/">Informe de ejemplo</a></p>
+
+<p align="center"><img src="docs/assets/demo.gif" alt="rulelab split y rulelab demo en un terminal" width="820"></p>
+
+<p align="center"><a href="https://diegoperezg7.github.io/rulelab/"><img src="docs/assets/report-light.png" alt="Informe HTML de rulelab" width="820"></a></p>
+
+## Por qué
+
+Cada línea de `AGENTS.md` o `CLAUDE.md` se envía al modelo en cada llamada. ¿Merece la pena?
+
+La mejor evidencia hasta ahora dice que a menudo no. Gloaguen et al. (ETH Zúrich y LogicStar.ai,
+[arXiv 2602.11988](https://arxiv.org/abs/2602.11988)) pusieron a Claude Code, Codex y Qwen Code
+sobre SWE-bench Lite y CTXbench (138 tareas de 12 repositorios) y vieron que los ficheros de
+contexto **no mejoran en general el acierto** y suben el coste de inferencia **más de un 20 % de
+media**:
+
+| Fichero de contexto | Acierto | Coste de inferencia |
+| --- | --- | --- |
+| Escrito por un LLM | −0,5 % (SWE-bench Lite), −2 % (CTXbench) | +20 %, +23 % |
+| Escrito por desarrolladores | +2,4 % (no significativo) | hasta +19 % |
+
+También vieron que las descripciones generales del repositorio no ayudan. Eso es la media de
+muchos repos. A ti te importan tu fichero, tu repo y tu modelo, y el paper mide ficheros
+enteros, no las reglas sueltas que de verdad borrarías.
+
+rulelab hace ese experimento sobre **tu** repo, **regla a regla**.
+
+| | Qué mide | Tareas |
+| --- | --- | --- |
+| [fernforge/ruleprof](https://github.com/fernforge/ruleprof) (0★) | Quita reglas de una en una y mira si el agente **obedece** cada regla | De juguete |
+| [sx4im/skillcheck](https://github.com/sx4im/skillcheck) (29★) | A/B a ciegas del **fichero entero** frente a no tenerlo | Tus prompts |
+| **rulelab** | Quita reglas de una en una y mide **acierto y coste** por regla | Arreglos reales del historial de tu repo, puntuados con sus propios tests |
+
+Estrellas comprobadas con la API de GitHub el 3 nov 2026.
+
+## Inicio rápido
+
+Mira una ejecución real grabada (sin API key, sin Docker, en menos de un minuto):
+
+```bash
+uvx rulelab-cli demo --open
+```
+
+o desde un clon:
+
+```bash
+git clone https://github.com/diegoperezg7/rulelab && cd rulelab && uv run rulelab demo --open
+```
+
+En PyPI el paquete se llama `rulelab-cli` (el nombre `rulelab` se parece demasiado a un proyecto que ya existe allí); el comando es `rulelab`. Instálalo con `uv tool install rulelab-cli` o `pipx install rulelab-cli`.
+
+Después, sobre tu propio fichero y repo (necesita Docker, y Ollama o una API key):
+
+```bash
+rulelab split AGENTS.md                                   # cómo se parte el fichero, gratis
+rulelab run AGENTS.md --repo . --dry-run                  # plan y coste estimado, gratis
+rulelab run AGENTS.md --repo . --model ollama/qwen3:4b    # modelo local, 0 USD
+rulelab run AGENTS.md --repo . --model mistral/ministral-14b-latest --price 0.2,0.2 -j 3
+```
+
+## Salida de ejemplo
+
+```text
+$ rulelab demo
+Recorded run · real results from rulelab run on 2026-11-03 · rulelab 0.1.0 · Darwin arm64 (Apple M4, 16 GB RAM)
+AGENTS.md · model mistral/ministral-14b-latest · 4 tasks × 2 repeats × 8 configs · 64 runs · 0.7376 USD
+full file 2/8 · no file 1/8 · file effect +12 pp [−21, +39]
+file tokens per run, estimated if resent on every call: ≈4,305 · measured difference vs no file: +776 [−15,405, +17,616]
+ rule                    ≈tok   effect (pp, 95% CI)   verdict        USD/month
+ R1 Repository overview    88        +12 [−21, +41]   inconclusive        0.23
+ R2 Running tests          47          0 [−26, +25]   inconclusive        0.12
+ R3 How to fix a bug       56        +12 [−20, +40]   inconclusive        0.15
+ R4 Lexer and keywords     68          0 [−26, +27]   inconclusive        0.18
+ R5 Code style             34          0 [−26, +27]   inconclusive        0.09
+ R6 Communication          30          0 [−26, +26]   inconclusive        0.08
+Small sample: most rules are inconclusive. [...]
+```
+
+**Lo que dice esta ejecución grabada, sin adornos.** La demo es un `rulelab run` real de un
+[AGENTS.md](examples/demo/AGENTS.md) de 6 reglas (incluye una descripción general del repositorio,
+como en el paper) sobre las 4 tareas más recientes de la demo de commitarena: arreglos de bugs de
+[sqlparse](https://github.com/andialbrecht/sqlparse), puntuados con los tests de sqlparse
+([tasks.json](examples/demo/tasks.json)). Modelo `mistral/ministral-14b-latest`, 15 pasos de agente,
+2 repeticiones, commitarena 0.1.1: 64 ejecuciones, 27 minutos, 0,74 USD. El modelo se eligió con
+un sondeo de 4 ejecuciones porque resolvía algunas tareas pero no todas (1 de 4), así que la línea
+base no es ni 0 % ni 100 % y un efecto podría aparecer.
+
+Pasaron 13 de 64 ejecuciones. El fichero completo pasó 2/8 y sin fichero 1/8 (+12 pp, intervalo de
+−21 a +39). Quitar la descripción del repositorio o la regla de cómo arreglar un bug bajó el
+acierto de 2/8 a 1/8, pero con 4 tareas y 2 repeticiones eso es una sola ejecución de diferencia:
+**todas las reglas salen no concluyentes**, y el informe lo dice en vez de forzar un veredicto. Lo
+que sí deja claro es el precio: el fichero son ≈328 tokens reenviados en cada una de las ~13
+llamadas al modelo; reenviado en cada una sumaría ≈4.305 tokens de entrada por ejecución estimados. La diferencia medida de verdad en esta ejecución (fichero completo frente a sin fichero) fue de +776 tokens de entrada por ejecución, con un intervalo amplio, porque las trayectorias del agente varían mucho más que eso. Para tener un veredicto, usa más
+tareas y repeticiones sobre tu propio repo.
+
+Una ejecución anterior más grande (144 ejecuciones con codestral sobre una versión previa de
+commitarena) resolvió 1 de 144 tareas y no se publica como resultado; sí demostró que la ejecución
+sobrevive a errores del motor, cortes por presupuesto e interrupciones (`--resume`).
+
+El informe completo, con el fichero anotado, está publicado en
+**[diegoperezg7.github.io/rulelab](https://diegoperezg7.github.io/rulelab/)** (EN/ES, claro/oscuro).
+Capturas: [claro](docs/assets/report-light.png) · [oscuro](docs/assets/report-dark.png) ·
+[móvil](docs/assets/report-mobile.png).
+
+## Cómo funciona
+
+```mermaid
+flowchart LR
+  F[AGENTS.md / CLAUDE.md] --> S[partir en reglas<br/>por sección o viñeta]
+  S --> C["configuraciones:<br/>sin fichero · completo · completo sin R1 … Rn"]
+  R[tu repo] --> M[commitarena mine:<br/>commits reales que arreglan un bug con test]
+  C --> X{{cada config × cada tarea × 3 repeticiones}}
+  M --> X
+  X --> D[sandbox Docker sin red:<br/>el agente edita código, el modelo se llama desde el host]
+  D --> T[los tests del repo deciden si pasa]
+  T --> B[intervalo bootstrap por regla:<br/>ayuda · sin efecto · empeora · no concluyente]
+  B --> H[informe HTML + results.json]
+```
+
+1. **Partir.** `--granularity section` (por defecto) convierte cada encabezado en una regla;
+   `bullet`, cada viñeta de primer nivel o párrafo. Un `# Título` suelto se mantiene en todas las
+   configuraciones. Compruébalo con `rulelab split`.
+2. **Configuraciones.** Sin fichero, fichero completo y fichero completo sin cada regla. Con
+   *n* reglas son *n* + 2 configuraciones.
+3. **Tareas.** Arreglos de bugs reales de tu historial de git, minados por
+   [commitarena](https://github.com/diegoperezg7/commitarena): commits que añaden un test que falla
+   antes del arreglo. El agente recibe los tests que fallan, nunca el mensaje del commit.
+4. **Ejecuciones.** Cada configuración corre cada tarea `--repeats` veces (3 por defecto) en el
+   mismo sandbox Docker sin red (2 CPU, 2 GB). El fichero se escribe en la raíz del repo y se añade
+   al prompt de sistema (`--inject both|repo|prompt`). Antes se quita el AGENTS.md/CLAUDE.md propio
+   del repo, así que «sin fichero» es sin fichero de verdad.
+5. **Veredicto.** Efecto de una regla = acierto con el fichero completo − acierto sin ella. El
+   intervalo del 95 % sale de un bootstrap en dos niveles (se remuestrean tareas y el acierto por
+   tarea se saca de una posterior Beta(s+½, f+½), para que 3/3 frente a 3/3 no parezca una
+   certeza). **Ayuda** si todo el intervalo está por encima de 0, **empeora** si está por debajo,
+   **sin efecto** si cabe dentro de ±10 pp (`--margin`) y **no concluyente** en otro caso.
+6. **Coste.** Tokens por regla (≈ caracteres / 4), la diferencia medida de tokens de entrada por
+   ejecución y una proyección mensual: tokens de la regla × llamadas al modelo por ejecución ×
+   sesiones al mes × precio de entrada.
+
+### Leer el resultado con honestidad
+
+Con 6 tareas y 3 repeticiones solo se distinguen del ruido los efectos grandes, así que **la
+mayoría de reglas saldrán no concluyentes**. El informe lo dice tal cual en vez de forzar un
+veredicto. «No concluyente» no es «inútil»: significa que este experimento no puede decidirlo.
+Añade tareas (`--max-tasks`) o repeticiones antes de borrar una regla por eso.
+
+## Configuración
+
+| Opción | Por defecto | Qué hace |
+| --- | --- | --- |
+| `--repo` | | Repo git (ruta o URL) del que minar tareas |
+| `--tasks-file` | | Reusar un `tasks.json` de `commitarena mine` |
+| `--only` | todas | Ids de tarea separados por comas |
+| `--max-tasks` | 6 | Tareas a minar con `--repo` |
+| `--model`, `-m` | `ollama/qwen3:4b` | `ollama/…`, `mistral/…`, `openrouter/…`, `openai/…` |
+| `--granularity`, `-g` | `section` | `section` o `bullet` |
+| `--repeats`, `-n` | 3 | Ejecuciones por tarea y configuración |
+| `--max-cost` | 1.0 | Tope de gasto en USD, comprobado antes de cada ejecución. Los modelos locales cuentan 0 |
+| `--inject` | `both` | El fichero en la raíz del repo, en el prompt de sistema o en ambos |
+| `--max-steps` | 30 | Llamadas al modelo por ejecución |
+| `--concurrency`, `-j` | 1 | Ejecuciones a la vez, cada una en su sandbox |
+| `--price` | | `IN,OUT` en USD por 1M de tokens para modelos sin precio conocido |
+| `--price-source` | | De dónde sale ese precio; aparece en el informe |
+| `--sessions-per-month` | 1000 | Para la proyección de coste mensual |
+| `--out`, `-o` | `rulelab-results` | Escribe `results.json` (tras cada ejecución) y `report.html` |
+| `--resume` | | Continuar una ejecución interrumpida o parada por presupuesto |
+| `--dry-run` | | Solo imprime el plan y la estimación |
+
+Antes de ejecutar nada, rulelab imprime el número de ejecuciones del agente y el coste estimado.
+Si la estimación supera `--max-cost`, se para y te dice qué cantidad pasar. No se gasta nada
+hasta que lo permites.
+
+**Modelos locales pequeños.** Ollama sirve los modelos con un contexto de 4.096 tokens por
+defecto. Una ejecución del agente necesita mucho más, y en nuestra prueba `qwen3:4b` gastó esa
+ventana pensando y nunca llegó a ejecutar un comando. Arranca Ollama con una ventana mayor (por
+ejemplo `OLLAMA_CONTEXT_LENGTH=32768 ollama serve`) antes de usar un modelo local.
+
+**Las claves** se leen solo de variables de entorno: `MISTRAL_API_KEY`, `OPENROUTER_API_KEY`,
+`OPENAI_API_KEY` (ver [.env.example](.env.example)). Ollama no necesita ninguna (`OLLAMA_HOST`
+para apuntar a otro sitio). rulelab no envía telemetría.
+
+`rulelab report results.json` regenera el informe con otro `--margin`, `--price-in` o
+`--sessions-per-month` sin volver a ejecutar nada.
+
+## Úsalo desde Claude Code
+
+Instala el plugin y pídeselo a Claude con tus palabras; su skill `ablate-agent-rules` divide tu fichero, enseña la estimación de coste (gratis), lanza el experimento solo cuando lo apruebas y explica qué reglas ayudan, cuáles perjudican y cuáles aún no se pueden distinguir.
+
+```text
+/plugin install rulelab --marketplace diegoperezg7/rulelab
+```
+
+La skill indica a Claude que pregunte antes de cualquier ejecución de pago, y `--max-cost` sigue limitando el gasto.
+
+## Hoja de ruta
+
+Fuera de la v0.1:
+
+- Tareas sintéticas para repos sin suficiente historial de arreglos.
+- Comprobación de obediencia (¿siguió el agente la regla?) como segunda señal junto al acierto.
+- `.cursor/rules` y skills.
+- Interacciones entre reglas (quitar parejas o grupos).
+- Reescritura automática de reglas que cuestan más de lo que ayudan.
+
+## Parte de una serie
+
+Herramientas open source para medir agentes y LLMs con tus propios datos:
+
+- [memarena](https://github.com/diegoperezg7/memarena): Compara capas de memoria con tus propias conversaciones.
+- [parsearena](https://github.com/diegoperezg7/parsearena): Compara parsers de PDF lado a lado y su efecto en la recuperación.
+- [commitarena](https://github.com/diegoperezg7/commitarena): Reproduce fixes reales de tu historial de git con cualquier modelo.
+- [toolbloat](https://github.com/diegoperezg7/toolbloat): Mide el coste en tokens de tu pila MCP y los errores al elegir tool.
+- [doesitfire](https://github.com/diegoperezg7/doesitfire): Comprueba si tu SKILL.md se activa en Claude Code, Codex y OpenCode.
+
+## Licencia
+
+[MIT](LICENSE) © 2026 Diego Pérez. Las tareas de la demo contienen código y tests de
+[andialbrecht/sqlparse](https://github.com/andialbrecht/sqlparse) (BSD-3-Clause, ver
+[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md)), tomadas de la demo de commitarena; el AGENTS.md
+de la demo en [`examples/demo`](examples/demo) se escribió para este proyecto.