evalroute 0.6.0__tar.gz

evalroute-0.6.0/.gitignore

@@ -0,0 +1,9 @@
+.venv/
+__pycache__/
+*.pyc
+.DS_Store
+.pytest_cache/
+dist/
+build/
+*.egg-info/
+data/flywheel/labels.jsonl

evalroute-0.6.0/CHANGELOG.md

@@ -0,0 +1,53 @@
+# Changelog
+
+## [0.6.0] - 2026-10-03
+
+The routing core moved out of `hermes-plugin-evalroute` v0.5.1 into this
+library. A move, not a rewrite: no behaviour change. The Hermes plugin
+remains at `keppy/hermes-plugin-evalroute` and becomes a thin adapter over
+the nine contract names in `evalroute/contract.py`.
+
+Moved to the library:
+
+- `tools.py` → split mechanically along existing function boundaries into
+  `evalroute/routing.py` (table load/validate, classifier, facets, route
+  card, `_tool_result`, `_note_route`, `install_routes`) and `evalroute/cli.py`
+  (`setup_cli`, `evalroute_cli`, the workflow epilog).
+- `flywheel.py`, `dispatch.py`, `dataset.py`, `schemas.py`, `adjudicate.py`,
+  `routes_from_report.py`, `routes_from_labels.py` (imports made relative;
+  the path-load + sibling-injection hack in `evalroute_cli` and its
+  `if "fw" not in globals()` guard deleted — `dispatch` is now
+  `from . import dispatch`).
+- `harness/evalroute.py` → `evalroute/harness/tier_a.py`;
+  `runners/hermes-shim.py` → `evalroute/runners/hermes_shim.py`.
+- `data/routes.yaml`, `data/facets.yaml` → `evalroute/data/` as package data
+  (resolved via `importlib.resources`).
+- `examples/artifacts/` (repo root, not package data) and
+  `scripts/publish_dataset.py`.
+- The tests, minus `test_registration.py` and `test_sniff.py`, which stay in
+  the plugin (registration wiring and the Hermes sniff hook are plugin
+  concepts). `sniff.py` and `__init__.py`'s `register(ctx)` stay in the
+  plugin too.
+
+Adapted:
+
+- The `hermes_constants.get_hermes_home` try/except fallback at four sites
+  is now one function, `evalroute.paths.hermes_home()`.
+- `PLUGIN_DIR` anchoring is gone: package data comes from
+  `importlib.resources`, and `_plugin_version()` (which read `plugin.yaml`)
+  becomes the package version — `pyproject.toml` via
+  `importlib.metadata.version("evalroute")`, literal fallback when not
+  installed. The card's provenance line reads `table: bundled
+  (evalroute 0.6.0)`.
+- `evalroute.contract` is new: `CONTRACT_VERSION = 1` and the nine names the
+  plugin relies on (`routing.set_llm_facade`, `routing.evalroute_route`,
+  `routing.handle_route_command`, `cli.setup_cli`, `cli.evalroute_cli`,
+  `flywheel.handle_rate`, `flywheel.on_pre_command`,
+  `flywheel.on_post_llm_call`, `schemas.EVALROUTE_ROUTE`).
+- A standalone `evalroute` console script wraps the same argparse tree, so
+  `evalroute route|dispatch|sync|rate ...` matches `hermes evalroute ...`
+  byte-for-byte.
+
+Kept in the plugin: `__init__.py` (`register(ctx)`), `sniff.py`, `skills/`,
+`plugin.yaml`, `catalog/`, `tests/test_registration.py`,
+`tests/test_sniff.py`.

evalroute-0.6.0/PKG-INFO

@@ -0,0 +1,351 @@
+Metadata-Version: 2.5
+Name: evalroute
+Version: 0.6.0
+Summary: Model routing with a verified-success flywheel: classify tasks, measure arms, feed results back
+License: MIT
+Requires-Python: >=3.11
+Requires-Dist: pyyaml<7,>=6
+Provides-Extra: dev
+Requires-Dist: gonogo-eval<0.4,>=0.3; extra == 'dev'
+Requires-Dist: pytest>=7; extra == 'dev'
+Provides-Extra: hub
+Requires-Dist: huggingface-hub>=0.20; extra == 'hub'
+Description-Content-Type: text/markdown
+
+# evalroute
+
+Route a task to the right **(model, reasoning effort) arm** before you start.
+evalroute is a Python package built around the evalroute procedure — a harness
+that measures **cost per verified success** per task lane (`evalroute/harness/
+tier_a.py`: measure, then serve the results) — and turns that output into a
+route table with provenance on every row: pick the lane, pick the model, pick
+the effort, as data.
+
+The Hermes plugin is a thin adapter over this package, at
+`keppy/hermes-plugin-evalroute` (`/route`, the `evalroute_route` tool, the
+first-turn sniff, and the bundled skill live there; the routing core lives
+here).
+
+## Install
+
+```bash
+pip install evalroute          # runtime: pyyaml only
+pip install "evalroute[hub]"   # + huggingface_hub, for `sync`
+```
+
+## Standalone CLI
+
+The same argparse tree the plugin registers, standalone and byte-identical in
+output to `hermes evalroute ...`:
+
+```bash
+evalroute route --lane routine-coding "fix the failing test"   # route card
+evalroute route --json "read this 80-page spec and summarize"  # JSON envelope
+evalroute dispatch brief.md          # route a brief, spawn hermes chat on that arm,
+                                     # print the rate line (--dry-run prints the argv)
+evalroute sync --status              # which route table is active
+evalroute rate pass --note "why"     # label the last routed task
+evalroute install-routes --dry-run   # route table -> agent.reasoning_overrides
+```
+
+Works with no Hermes installed: the Hermes home falls back to `HERMES_HOME`
+or `~/.hermes`, and the ledger, dataset pins, and effort-override reads all
+honor it.
+
+## The nine-name contract
+
+The plugin adapter relies on exactly these names; everything else in the
+package is private to the library (see `evalroute/contract.py`,
+`CONTRACT_VERSION = 1`):
+
+| contract name | module | what it is |
+| --- | --- | --- |
+| `routing.set_llm_facade` | `evalroute/routing.py` | stash the host LLM facade; `None` disables the fallback |
+| `routing.evalroute_route` | `evalroute/routing.py` | tool handler (`evalroute_route`) |
+| `routing.handle_route_command` | `evalroute/routing.py` | `/route` slash command |
+| `cli.setup_cli` | `evalroute/cli.py` | argparse wiring (`register_cli_command` setup_fn) |
+| `cli.evalroute_cli` | `evalroute/cli.py` | CLI handler |
+| `flywheel.handle_rate` | `evalroute/flywheel.py` | `/rate` pass\|fail\|skip |
+| `flywheel.on_pre_command` | `evalroute/flywheel.py` | `/model` + `/reasoning` observer |
+| `flywheel.on_post_llm_call` | `evalroute/flywheel.py` | last-seen-model diagnostic |
+| `schemas.EVALROUTE_ROUTE` | `evalroute/schemas.py` | tool schema |
+
+## The route table
+
+`evalroute/data/routes.yaml` — one row per lane: `id`, `keywords` (the rule
+layer), `model`, `effort`, `escalation`, `provenance`, `notes`. Lane taxonomy
+is the union of the two source tables (9 lanes); where they disagreed
+(long-doc merged into web-research in one, orchestration only in the other)
+both are kept as distinct lanes. Model ids must match `/model` spelling
+exactly.
+
+The table began with a **2026-09-26 priors snapshot** (public benchmarks,
+many vendor-run). Three lanes now have small measured batches; the others
+remain marked `priors`. Replace those rows only after your own controlled
+data, and keep each row's `provenance` visible.
+
+### Route table: bundled or synced
+
+The table you route against is either the bundled one or a pinned dataset
+revision; the card says which. The dataset is fetched only when you run
+`sync` (never on install, never while routing), it is pinned to a resolved
+revision, and the library routes fully offline without it:
+
+```bash
+evalroute sync --revision <sha>   # pin the published table (default: main)
+evalroute sync --status           # bundled, or dataset @ <sha>
+evalroute sync --clear            # back to the bundled table
+```
+
+`sync` downloads only the `routes/` config of
+[keppy/evalroute-flywheel](https://huggingface.co/datasets/keppy/evalroute-flywheel)
+into `<hermes home>/evalroute/dataset/<sha>/` — never the measured evidence
+(grows over time; leave it on the Hub). It needs `pip install
+huggingface_hub` (the `hub` extra). Routing data lands only under the Hermes
+home, like the ledger.
+
+### Regenerating from measured data
+
+```bash
+# in the harness venv (openai + anthropic; it stays out of the runtime venv)
+python -m evalroute.harness.tier_a run -m models.json -t tasks.jsonl -k 3
+python -m evalroute.harness.tier_a report -o runs.jsonl -t tasks.jsonl -k 3 --csv report.csv
+python -m evalroute.routes_from_report --csv report.csv --runs runs.jsonl \
+    --models models.json --k 3 --out routes.generated.yaml
+```
+
+`report` needs the matching `--tasks` file: it checks each run's prompt/checker
+contract and treats missing declared tasks as incomplete. Without that file it
+prints diagnostics but selects no route. The `report.csv` files under
+`examples/artifacts/` were regenerated from their `runs.jsonl` and
+`tasks.jsonl` with this harness and carry the `complete` column. A legacy CSV
+(no `complete` column) is refused even with `--runs`, since the old winner
+selection may have ignored pending cells; regenerate it the same way.
+
+The harness is packaged so the loop is complete inside one repo: write
+tasksets (deterministic `python` checkers where possible — validate every
+checker against a reference solution before paid runs), run k samples per
+arm, report, then flip the lane's row. `routes_from_report` applies the
+report's own routing rule (coverage-gated lowest all-in $/success), stamps
+`provenance: measured ...` with a gonogo McNemar stamp when the winner and
+runner-up shared cases, preserves each lane's `keywords`/`match_hint`/
+`escalation`/`notes`, and carries unmeasured lanes over verbatim —
+regeneration never silently deletes a route. The tier-a tasksets and runs
+that produced the current measured lanes are under `examples/artifacts/`
+(sets: `tasks.jsonl`; raw run records: `runs.jsonl`).
+
+Inspect the generated YAML before replacing the bundled table. `--models`
+maps harness arm names such as `glm-5.3@high` to `/model` IDs; omission is
+only safe if the CSV already contains routable IDs. `--k` defaults to 3;
+graded sample counts must divide evenly by k, or the generator refuses to
+invent a task count. The harness v2 resume key includes the full task and
+checker spec, model price/config and effective max tokens, plus the judge
+configuration when used. It keeps legacy JSONL reportable, but reporting
+mixed legacy/v2 or multiple prompt/checker versions together fails explicitly.
+
+### The Hermes shim (`api: "cmd"` arms)
+
+To run Hermes itself as an evalroute arm (agentic cells):
+
+```json
+{"name": "hermes-glm@high", "api": "cmd", "effort": "high",
+ "cmd": "python -m evalroute.runners.herbes_shim --model {model} --effort {effort} --prompt {prompt_file}",
+ "model": "z-ai/glm-5.3", "in": 0.91, "out": 2.86, "timeout": 3600}
+```
+
+The shim wraps `hermes -z` (one-shot; tools, memory, AGENTS.md loaded as
+normal; approvals auto-bypassed), reads the usage report (`--usage-file`),
+and emits the contract evalroute expects: the answer on stdout, then one
+JSON last line `{"text": ..., "usage": {"inp", "out", "cache_read"}}`. A
+non-zero hermes exit becomes an `error` field in that line (the harness
+records an error row and retries on the next `run`); the shim itself always
+exits 0. `--system` is prepended to the prompt. `--hermes PATH` overrides
+the executable (tests use this to point at a fake — no real runs).
+
+## Classification: rules first, LLM when weak
+
+The classifier's first layer is deterministic keyword rules over
+`evalroute/data/routes.yaml` — free, no API calls — but rules alone misroute
+paraphrase ("manage life, writing, and researchy tasks" has zero keyword
+signal) and negation ("not usually hard math though" used to count as a
+math hit; the rules now guard negated keywords). So routing is two-layer:
+
+1. **Strong rules** (2+ distinct keyword hits on the winning lane) — trusted
+   outright, no LLM call.
+2. **Weak signal** (0-1 hits) — one structured call via the facade set with
+   `routing.set_llm_facade` (the host's own model and auth; the plugin sets
+   it at register time, but **it consumes tokens and may incur provider
+   charges**). `None` (the default, and what a bare `evalroute` CLI sees)
+   disables the fallback. The LLM judges what the work IS — a description of
+   an assistant's duties routes to `orchestration`, not to whatever nouns
+   appear.
+
+The card always prints which layer decided: `rules match`, `LLM fallback`,
+or `no keyword hit - defaulted`. If the LLM call fails (offline, no facade),
+the weak rules result stands and the card says so. Pin manually with
+`--lane <id>` when you know better.
+
+## Facets: labels with dimensions
+
+A lane is the routing decision; facets are the label. Every route also
+captures the task's shape along three axes, defined in
+`evalroute/data/facets.yaml`:
+
+- **input-shape**: `long-doc` | `interactive`
+- **domain**: `domain-dlml` | `domain-alignment` | `domain-math` | `domain-prose` | `domain-research`
+- **demand-tier**: `tier-routine` | `tier-hard` | `tier-orchestration`
+
+So "audit my RL training plan files" is recorded as `long-doc +
+domain-dlml + tier-hard` — three facts about one task — instead of one
+collapsed lane. The rules layer derives facets from keyword evidence
+(conservative: only lanes that drew hits claim facets); the LLM fallback
+names them semantically in the same structured call.
+
+When a task claims facets on multiple axes, the card describes the
+conjunction. **Facets do not alter the chosen arm**: the lane classifier
+selects the arm; no domain/tier precedence is implemented:
+
+```
+facets: long-doc + domain-dlml (descriptive conjunction; lane chooses arm)
+```
+
+Facet conjunctions aggregate in `routes_from_labels`, so the
+high-dimensional nodes — "how do long-doc x dl-ml tasks fare on arm X?" —
+fill in from daily use without controlled-batch spend.
+
+## Flywheel: labels from daily workflow
+
+The controlled harness is not the only source of data. As you route in daily
+sessions, the library quietly builds an observational dataset:
+
+- **`route`** logs the assignment (lane, recommended arm, method,
+  confidence, facets) to `<hermes home>/evalroute/labels.jsonl` — the
+  task text you typed is the label.
+- **`/model` or `/reasoning` after a route** logs a process-global switch
+  observation. Without a session join it is not a verified route rejection
+  and cannot supply the actual arm for a flip.
+- **`rate pass|fail [--route-id <id>] [--lane <id>] [--model <id> --effort <level>] [--note ...]`**
+  labels the outcome when you finish. Use both arm flags to self-report what
+  actually ran; otherwise the arm stays unknown. `--lane` corrects a lane;
+  `skip` discards that route. Prefer `--route-id` in overlapping sessions.
+- Nothing else is recorded: no response bodies or turn telemetry. Task text,
+  switch arguments and optional notes are recorded. The last-seen model is
+  process-global memory only and is **not** assigned to a route as fact.
+
+The ledger therefore holds two row qualities: an **observed arm** (a
+`/model` or `/reasoning` switch after a route — a process-global candidate,
+not proof) and a **caller-stated arm** (`evalroute dispatch <brief.md>`
+records the spawn arguments as `arm_attribution: explicit_user` on the
+outcome row). `dispatch` is the one-line form of the flywheel: route the
+brief, spawn `hermes chat` on exactly that arm, print the `rate it:` line —
+it still never auto-rates `pass`.
+
+The ledger is **profile-wide**, not session-scoped: command hooks do not
+supply a reliable session ID for route and rate. The card prints a route ID;
+when tasks overlap, select it with `--route-id`. Without it, `rate` consumes
+the latest pending route in that profile, which may be another session's.
+`/model` and `/reasoning` observations are process-global candidates, not
+proof of which model served a given route; only an explicit `rate --model
+... --effort ...` confirms an observational arm. To replace a pinned card,
+use `route --lane <lane> --replace-route-id <old-id> <same task>`; identical
+task text alone never consumes another pending route. Inspect the route ID,
+task and lane before trusting a label.
+
+**Continuing across sessions (turn caps).** A task that outlives its session —
+the turn limit hits, the terminal closes mid-task — is a continuation, not a
+new route. The row being labeled is (task, arm), not (task, session):
+
+- Prefer staying in the session: `continue: <what remains>` gets a fresh
+  iteration budget, and the arm is session-scoped and persists.
+- Otherwise `hermes -c` continues the same conversation, or paste the capped
+  session's final turn as the new session's opener.
+- Never re-`route` the continuation. If the new session is a different task,
+  `rate skip --route-id <id>` clears that specific pending row first if it
+  belongs to you; don't consume another user's pending row.
+- Steer as much as you like. The observed layer is defined as
+  daily-workflow-with-a-human-in-the-loop; a directive continuation is normal
+  operation, and it matches a detailed original prompt better than a bare
+  "continue" — which quietly tests prompt-luck instead of the arm. Measured
+  rows are untouched: they come from fixed-prompt, fresh-context harness cells.
+- Put the methodology in the note: `--note "completed across two sessions
+  (turn cap), directive continuation"`. The label records neither cost nor
+  session boundaries, and a session-spanning pass re-reads the accumulated
+  context at full input price — the note is where that lives.
+
+**Turning labels into route data:** `python -m evalroute.routes_from_labels`
+prints per-lane, per-arm pass rates, lane corrections, and facet conjunction
+outcomes; `--apply` writes `routes.observed.yaml`. Observed rows carry
+honest, weaker provenance:
+
+```
+observed 23 outcomes, same-maintainer observational single-arm, pass 78%, 2026-10-30;
+not independent trials or a controlled comparison
+```
+
+and only where the lane has no `measured` row — observational data can contest
+a priors row, never overwrite a measured one. When gonogo is installed, each
+observed row also carries its decide() verdict, so a row with 3 outcomes reads
+as INSUFFICIENT_EVIDENCE rather than a pass rate someone will trust. The one
+provisional flip threshold is 3 user-confirmed arm failures on the recommended
+arm and 2 user-confirmed wins on another observed arm. This is still
+same-maintainer, non-randomized evidence — review it and verify with paired
+controlled cases before treating it as a quality comparison. The two
+same-arm outcomes in the old snapshot never justify a route flip.
+
+**Publishing the labels.** The live ledger stays local and append-only; do
+not check raw labels into a public repo. Task text and notes can expose
+paths and private project details even without response bodies. Do not claim
+retroactive erasure for any label data once shared.
+
+## What the harness measures
+
+Three lanes measured with the evalroute harness via the Nous inference API —
+10 tasks x 3 samples per arm (four routine-coding, five DL/ML, five
+alignment arms, of which only four alignment arms have graded samples: 119
+graded, 30 judge-pending, and one missing cell). Raw API and judge cost in
+the vendored rows totals **$3.0590051**, excluding verification time across
+routine coding, DL/ML research engineering, and alignment reasoning. Every
+winner was statistically indistinguishable from its runner-up at n=10
+(McNemar, via gonogo) — the empirical paired gap is zero, but the
+conservative interval spans [-33.4%, +33.4%]; $p=1$ is not a population
+equivalence test. All-coverage lanes have a ceiling on this taskset; the
+routes choose cost among observed ties, not quality parity. The alignment
+judge has no blind checker audit; its 30 pending Qwen outputs are excluded
+from the route comparison. No quality claim spans that arm. The historical
+v1 run records omit model IDs; the arm-name-to-ID mapping is the vendored
+`models.json`, not an ID echoed by those records. Routine coding overturned
+the priors' vendor pick: glm-5.3-flash at medium effort covered every task
+at $0.00005/success, 2.4–3.9x cheaper than the V4.1 Flash arms at equal
+coverage.
+
+Every row is `priors`, `observed`, or `measured`. Nothing hypothesis-shaped
+masquerades as a result — the card prints the row's provenance verbatim,
+statistical stamp included.
+
+## Known limitations
+
+- **The rule layer is keywords.** Deterministic, free, and misses
+  paraphrase — which is what the LLM fallback is for. The fallback's
+  quality tracks whatever model the host is on; it costs one small
+  structured call (temp 0, 256 tokens) only when rules are weak.
+- **The library cannot switch the model for you.** `route` prints the card;
+  you run `/model <id>`. Run it before turn 1 — mid-session switches re-read
+  the whole context at full input price.
+- **One effort slot per model id** (`agent.reasoning_overrides`); when a
+  model serves two lanes, `install-routes` keeps the higher effort. A
+  lower-effort lane's card explicitly prints `/reasoning <lane-effort>`
+  after `/model`, because the installed override alone would run the wrong
+  arm.
+- **Provenance is priors until you measure.** Several rows are explicitly
+  untested/unmeasured/contested; the card prints the provenance verbatim so
+  nobody mistakes a hypothesis for a result.
+- **The harness stays out of the runtime venv.** Run it in its own venv with
+  `openai`/`anthropic`; `EVALROUTE_PYTHON` points the runners at that
+  interpreter. The package itself requires nothing beyond `pyyaml`
+  (`huggingface_hub` only for `sync`).
+- **The sniff hook is advisory only** (plugin side): it speaks when the
+  classifier is confident and the session's model disagrees with the lane's
+  route; it never rewrites, blocks, or switches.
+- **`install-routes` needs the Hermes config module to write.** Standalone,
+  writing `agent.reasoning_overrides` works inside the `hermes` process;
+  `--dry-run` works anywhere.