catherd-cli 1.1.1 → 1.2.1

package/README.md

@@ -35,6 +35,8 @@ of work, climbing a ladder only when a cheaper rung falls short.
     (the npm package `opencode-ai` is v1 and is not supported)
   - Claude Code's `claude` CLI 2.1.282 or newer, for headless `claude-code:` rungs
 - Optional: a TypeSafe API key for Jev, in `TYPESAFE_API_KEY` or saved by `catherd init`
+- Optional: a free [Artificial Analysis](https://artificialanalysis.ai) API key for more scores, in
+  `ARTIFICIAL_ANALYSIS_API_KEY` or saved by `catherd init`; its numbers are read for you alone and never shipped
 
 `catherd doctor` checks each backend's version and login and prints the fix for anything missing. The default
 profile runs its workers on Codex; without Codex, doctor's fix also names how to move those roles to a backend
@@ -50,10 +52,11 @@ catherd init
 The npm package is `catherd-cli`; the command it installs is `catherd`. `bunx catherd-cli init` works too: `init`
 installs the global command at its own version (`--no-global` skips it) and says `installing catherd…` before
 it does, though the first `bunx` resolve itself prints nothing for up to half a minute. `init` asks for the
-optional Jev key, writes the default profile and links its Claude agents, lists your backends' models, and ends
-with a readiness report (`--no-input` asks nothing and keeps what exists; `--profile <name>` sets up and activates
-that profile instead of `default`; piped, it reads one answer per line once stdin closes: the Jev key, the
-profile, whether to replace it, each on its own line even when a question is skipped). Then add the plugin to
+optional Jev key and the optional Artificial Analysis key, syncs the public model sources, writes the default
+profile and links its Claude agents, lists your backends' models, and ends with a readiness report (`--no-input`
+asks nothing and keeps what exists; `--profile <name>` sets up and activates that profile instead of `default`;
+piped, it reads one answer per line once stdin closes: the Jev key, the Artificial Analysis key, the profile,
+whether to replace it, each on its own line even when a question is skipped). Then add the plugin to
 Claude Code:
 
 ```sh
@@ -87,7 +90,9 @@ In a terminal:
 | `catherd status [run]`, `catherd watch [--once] [--interval <s>]`                         | Where runs stand, grouped by the Claude Code session that drove them                                |
 | `catherd runs list [--repo <path>]\|show <id> [--debug [--name <n>]]\|cancel <id> <name>` | Past runs, by session; `--debug` adds exit.json and the stderr and event tails, `--name` one role's |
 | `catherd catalog refresh\|list [--backend <b>] [--role <r>] [--text <t>] [--scored]`      | The models catherd can place, filtered                                                              |
+| `catherd catalog sync [--force] [--unmatched]`                                            | Fetches the public model facts and scores now (below); `--unmatched` lists ids no model matched     |
 | `catherd catalog treat-like <rung> <like>`                                                | Scores an unscored rung as a scored one                                                             |
+| `catherd catalog treat-like --suggest <rung>\|--clear <rung>\|--reset`                    | The three nearest stand-ins for a rung; removes one or every mapping of yours                       |
 | `catherd lock [--slots N] -- <cmd>`                                                       | Runs a heavy command behind the machine-wide semaphore, in its own session (no /dev/tty)            |
 | `catherd mcp`                                                                             | The MCP server on stdio; the plugin starts it, you never need to                                    |
 | `catherd capture-fixtures [--backend <b>] [--out <dir>]`                                  | Contributors: records sanitized test fixtures from real runs (see CONTRIBUTING.md)                  |
@@ -99,21 +104,42 @@ on the profile the repo you are in runs on: the one bound to it, else the active
 `~/.local/share/catherd/logs/`, kept for 7 days with secrets redacted. The dashboard takes `--plain` (ASCII, no colour)
 and `--reduced-motion`; `doctor` and `init` take `--plain` for ASCII glyphs too. `NO_COLOR` drops colour, never glyphs.
 
+Model facts and scores also come from public sources: models.dev, OpenRouter, LiteLLM, Arena (LMArena), Vectara's
+hallucination leaderboard and Epoch AI, plus Artificial Analysis when you give `catherd init` a free key. The MCP
+server syncs them in the background when a Claude Code session starts (each source at most every 12 hours, never
+delaying the session), `init` syncs them, and `catherd catalog sync` (or the `catalog_sync` tool) does it on demand.
+Each answer is kept in `~/.local/share/catherd/sources/`; a source that fails keeps its last good answer, and with no
+network and no sync at all catherd routes on the scores it ships. Those are the keyless sources' values, refreshed
+weekly (`catalog/ATTRIBUTION.md` credits each source); Artificial Analysis values are read with your key only and
+never shipped.
+
+A lane's kind and difficulty pick a bar: a threshold on each dimension it spans (`repo_code`, `terminal`,
+`honesty`, `agentic`, `frontend`; `steer` is shown but has no default bar), and the lane starts on the cheapest
+rung that clears them all. A rung with no value on a dimension takes its nearest stand-in's as `inferred`, and
+`profile validate` and `doctor` list it as a "stand-in to confirm": confirm or replace it with `catherd catalog
+treat-like --suggest <rung>`, then `treat-like <rung> <like>`. `route` says, for the rung it picks, each threshold,
+the value used, its confidence and source, and catherd's own runs on it ("12 lanes, 2 climbed, 1 partial"), which
+it shows but never routes on. A bar of your own goes in `~/.config/catherd/catalog.override.json`, per dimension
+(`"bars": { "ui": { "hard": { "frontend": 1700, "honesty": null } } }`: a number sets a threshold, `null` removes
+the default's).
+
 Run data lives in `~/.local/share/catherd/`, config in `~/.config/catherd/` (both follow
 `XDG_*`).
 
 ### Environment variables
 
-| Variable                    | What it does                                                                                          |
-| --------------------------- | ----------------------------------------------------------------------------------------------------- |
-| `TYPESAFE_API_KEY`          | The Jev key, instead of the one `catherd init` saves                                                  |
-| `CATHERD_HOME`              | Puts config and data under `$CATHERD_HOME/config` and `$CATHERD_HOME/data` instead of XDG             |
-| `CATHERD_LOG`               | Log level: `off`, `error`, `warn`, `info` (default) or `debug` (what `--verbose` sets)                |
-| `CATHERD_LOCK_SLOTS`        | `catherd lock`'s slot count when `--slots` is not given (before the profile's `lock.heavy`)           |
-| `CATHERD_REDUCED_MOTION`    | Any value: the dashboard's `--reduced-motion`                                                         |
-| `CATHERD_NO_KITTY`          | Any value: turns off the kitty keyboard protocol in the dashboard, for terminals it breaks            |
-| `CATHERD_CLAUDE_AGENTS_DIR` | Where catherd links its Claude agents (default: `$CLAUDE_CONFIG_DIR/agents`, else `~/.claude/agents`) |
-| `NO_COLOR`                  | Drops colour (the dashboard's and `--help`'s; piped `--help` has none either)                         |
+| Variable                      | What it does                                                                                                |
+| ----------------------------- | ----------------------------------------------------------------------------------------------------------- |
+| `TYPESAFE_API_KEY`            | The Jev key, instead of the one `catherd init` saves                                                        |
+| `ARTIFICIAL_ANALYSIS_API_KEY` | An Artificial Analysis key for `catalog sync`, instead of the one `catherd init` saves                      |
+| `CATHERD_NO_SYNC`             | `1`: no automatic sync of the public sources (at MCP server start and in `init`); `catalog sync` still runs |
+| `CATHERD_HOME`                | Puts config and data under `$CATHERD_HOME/config` and `$CATHERD_HOME/data` instead of XDG                   |
+| `CATHERD_LOG`                 | Log level: `off`, `error`, `warn`, `info` (default) or `debug` (what `--verbose` sets)                      |
+| `CATHERD_LOCK_SLOTS`          | `catherd lock`'s slot count when `--slots` is not given (before the profile's `lock.heavy`)                 |
+| `CATHERD_REDUCED_MOTION`      | Any value: the dashboard's `--reduced-motion`                                                               |
+| `CATHERD_NO_KITTY`            | Any value: turns off the kitty keyboard protocol in the dashboard, for terminals it breaks                  |
+| `CATHERD_CLAUDE_AGENTS_DIR`   | Where catherd links its Claude agents (default: `$CLAUDE_CONFIG_DIR/agents`, else `~/.claude/agents`)       |
+| `NO_COLOR`                    | Drops colour (the dashboard's and `--help`'s; piped `--help` has none either)                               |
 
 For development only: `CATHERD_STORY=1` opens the dashboard's storybook, and `CATHERD_LIVE=1` enables the live
 tests (CONTRIBUTING.md has the rest).

package/catalog/ATTRIBUTION.md

@@ -0,0 +1,36 @@
+# Attribution for the shipped catalog data
+
+`catalog/scores.json` carries model scores from public sources. Each value names its source (`source`), the page
+it came from (`url`), its date and its confidence. The weekly `catalog-refresh` workflow rebuilds the values that
+come from the keyless sources below; the hand-typed values (no `source`) cite their own vendor or benchmark page in
+their `url`.
+
+## Sources whose values are shipped
+
+### Arena (LMArena)
+
+- Data: `lmarena-ai/leaderboard-dataset` on Hugging Face, configs `agent`, `agent_task_outcome_explicit`,
+  `agent_bash_recovery_steps`, `agent_steerability`, `agent_tool_hallucination`, `webdev`
+- License: CC BY 4.0
+- Attribution: Leaderboard data by LMArena, CC BY 4.0, https://huggingface.co/datasets/lmarena-ai/leaderboard-dataset
+
+### Epoch AI benchmarks
+
+- Data: https://epoch.ai/data/benchmark_data.zip (FrontierCode, Terminal-Bench and WebDev Arena tables)
+- License: CC BY 4.0; external tables keep their own license
+- Attribution: Epoch AI, 'Capabilities & benchmarking'. Published online at epoch.ai. Retrieved from 'https://epoch.ai/benchmarks' (CC BY 4.0)
+
+### Vectara hallucination leaderboard
+
+- Data: the README table of https://github.com/vectara/hallucination-leaderboard
+- License: Apache License 2.0
+- Attribution: Hallucination Leaderboard by Vectara (Apache License 2.0), https://github.com/vectara/hallucination-leaderboard
+
+## Sources read at run time only
+
+catherd reads these on the user's machine and ships none of their data: models.dev (MIT License,
+https://models.dev), the OpenRouter API (https://openrouter.ai) and LiteLLM's
+`model_prices_and_context_window.json` (MIT License, https://github.com/BerriAI/litellm).
+
+Artificial Analysis (https://artificialanalysis.ai) is read only with the user's own key, and its values are never
+shipped.

package/catalog/models.json

@@ -20,6 +20,7 @@
       "id": "gpt-6-astra",
       "name": "GPT-6 Astra",
       "vendor": "openai",
+      "releaseDate": "2026-09-04",
       "capabilities": { "toolUse": true, "imageIn": true, "reasoning": true },
       "price": { "input": 10, "cached": 1, "output": 50 },
       "planWeight": 60,
@@ -43,6 +44,7 @@
       "id": "gpt-6-sol",
       "name": "GPT-6 Sol",
       "vendor": "openai",
+      "releaseDate": "2026-09-22",
       "capabilities": { "toolUse": true, "imageIn": true, "reasoning": true },
       "price": { "input": 2, "cached": 0.2, "output": 10 },
       "planWeight": 20,
@@ -66,6 +68,7 @@
       "id": "gpt-6-luna",
       "name": "GPT-6 Luna",
       "vendor": "openai",
+      "releaseDate": "2026-09-22",
       "capabilities": { "toolUse": true, "imageIn": true, "reasoning": true },
       "price": { "input": 0.1, "cached": 0.01, "output": 0.5 },
       "planWeight": 1,
@@ -91,6 +94,7 @@
       "id": "gpt-5.6-sol",
       "name": "GPT-5.6 Sol",
       "vendor": "openai",
+      "releaseDate": "2026-07-09",
       "status": "legacy",
       "capabilities": { "toolUse": true, "imageIn": true, "reasoning": true },
       "price": { "input": 4, "cached": 0.4, "output": 20 },
@@ -113,6 +117,7 @@
       "id": "gpt-5.6-terra",
       "name": "GPT-5.6 Terra",
       "vendor": "openai",
+      "releaseDate": "2026-07-09",
       "status": "legacy",
       "capabilities": { "toolUse": true, "imageIn": true, "reasoning": true },
       "price": { "input": 2, "cached": 0.2, "output": 12 },
@@ -135,6 +140,7 @@
       "id": "gpt-5.6-luna",
       "name": "GPT-5.6 Luna",
       "vendor": "openai",
+      "releaseDate": "2026-07-09",
       "status": "legacy",
       "capabilities": { "toolUse": true, "imageIn": true, "reasoning": true },
       "price": { "input": 0.2, "cached": 0.02, "output": 1.2 },
@@ -162,6 +168,7 @@
       "id": "claude-fable-5-1",
       "name": "Claude Fable 5.1",
       "vendor": "anthropic",
+      "releaseDate": "2026-09-01",
       "capabilities": { "toolUse": true, "imageIn": true, "reasoning": true },
       "price": { "input": 10, "cached": 0.25, "output": 50 },
       "meteredOnPlan": true,
@@ -185,6 +192,7 @@
       "id": "claude-opus-5-5",
       "name": "Claude Opus 5.5",
       "vendor": "anthropic",
+      "releaseDate": "2026-09-22",
       "capabilities": { "toolUse": true, "imageIn": true, "reasoning": true },
       "price": { "input": 4, "cached": 0.2, "output": 20 },
       "on": {
@@ -201,10 +209,31 @@
       },
       "notes": { "effort": "the API default effort is medium" }
     },
+    {
+      "id": "claude-sonnet-5-5",
+      "name": "Claude Sonnet 5.5",
+      "vendor": "anthropic",
+      "releaseDate": "2026-09-28",
+      "capabilities": { "toolUse": true, "imageIn": true, "reasoning": true },
+      "price": { "input": 2, "cached": 0.2, "output": 10 },
+      "on": {
+        "claude-code": {
+          "id": "claude-sonnet-5-5",
+          "efforts": ["low", "medium", "high", "xhigh", "max"],
+          "context": 1000000
+        },
+        "opencode": {
+          "id": "opencode/claude-sonnet-5-5",
+          "efforts": ["low", "medium", "high", "xhigh", "max"],
+          "context": 1000000
+        }
+      }
+    },
     {
       "id": "claude-sonnet-5",
       "name": "Claude Sonnet 5",
       "vendor": "anthropic",
+      "releaseDate": "2026-06-29",
       "capabilities": { "toolUse": true, "imageIn": true, "reasoning": true },
       "price": { "input": 2, "cached": 0.2, "output": 10 },
       "on": {
@@ -224,6 +253,7 @@
       "id": "claude-haiku-4-5",
       "name": "Claude Haiku 4.5",
       "vendor": "anthropic",
+      "releaseDate": "2025-10-15",
       "capabilities": { "toolUse": true, "imageIn": true, "reasoning": true },
       "price": { "input": 1, "cached": 0.1, "output": 5 },
       "on": {