claude-autorouter 0.3.6 → 0.4.0

package/docs/router-performance.md

@@ -0,0 +1,50 @@
+# Synthetic router performance
+
+On October 5, 2026, eight simultaneous identical requests made one evaluator call after coalescing, compared with eight before it. This holds across different agent scopes. Distinct requests still evaluate independently, and pinned tool continuations still evaluate on every request. [The recorded baseline and candidate](router-performance.json) contain three rounds of measurements and their comparison gates.
+
+The baseline was recorded before changing routing or evaluator transport. The final candidate was measured at 10:25 UTC after the compatibility guards and response handling were frozen; the baseline and comparison thresholds were retained unchanged. Both versions used the same synthetic harness: 60 iterations per scenario, concurrency eight, a mocked evaluator with a one-millisecond timer, and a 286,807-byte request containing 300 synthetic tools. No network services, user prompts, provider credentials, model inference, or downloads were involved.
+
+Measurements came from one 16 GiB Apple M4 Mac with ten logical CPUs, Darwin 25.6.0, and Node v26.10.0. The workstation remained in normal use. Its one-minute load average was 2.32 after the baseline and 2.22 after the final candidate; free memory and the remaining load averages are in the JSON. The final run preceded the full regression suite to avoid interference from that workload. No user process names or arguments were collected.
+
+## Results
+
+The table reports the largest p50 and p95 across the three rounds, in milliseconds. Counts exclude setup calls that warm the cache or establish the initial pin.
+
+| Scenario | Before p50 / p95 | After p50 / p95 | Evaluator calls per round, before → after |
+| --- | ---: | ---: | ---: |
+| Small request | 1.331 / 1.646 | 1.261 / 1.396 | 60 → 60 |
+| Large tool catalog | 3.421 / 4.415 | 2.810 / 3.037 | 60 → 60 |
+| Cached request | 0.025 / 0.035 | 0.010 / 0.014 | 0 → 0 |
+| Pinned tool continuation | 1.373 / 1.778 | 1.261 / 1.346 | 60 → 60 |
+| Eight identical concurrent requests | 1.936 / 2.815 | 1.915 / 2.397 | 480 → 60 |
+| Identical requests from eight agents | 1.715 / 2.219 | 1.566 / 1.815 | 480 → 60 |
+| Distinct requests from eight agents | 1.686 / 2.201 | 1.523 / 1.907 | 480 → 480 |
+| Immediate caller cancellation | 0.097 / 0.124 | 0.028 / 0.056 | 60 → 0 |
+
+The evaluator-call reduction is deterministic. Timing differences also reflect timer scheduling, garbage collection, JIT warmup, and background load; these measurements do not establish faster Claude responses or a model inference speedup. Route time includes the synthetic evaluator delay and JSON processing. `policy_overhead_ms` separately reports route timing minus classification timing; it is an approximate measure using the router's rounded telemetry. There is no network or real model latency in either measure.
+
+The JSON includes retained heap/RSS changes, sampled peak heap/RSS changes, and event-loop delay. Sampling runs every millisecond, with explicit garbage collection before and after each workload. It can miss allocations inside one synchronous interval. The event-loop histogram has ten-millisecond resolution, so values near ten milliseconds are the sampling floor. RSS is descriptive because allocator page reuse can dominate differences between short runs.
+
+## Local regression gates
+
+For each scenario, the p95 route-time limit is twice the largest p95 across the three pre-change rounds. Peak sampled heap and p95 event-loop delay use the same baseline-relative margin. For example, the small-request route limit is 3.292 ms, the large-catalog limit is 8.831 ms, and the identical-agent limit is 4.439 ms. These are local regression guards, not portable latency promises or CI timing thresholds. The comparison also requires matching workload parameters, Node version, and hardware, and exactly one evaluator call per group of eight identical requests. All recorded gates pass.
+
+Repeat this process with a new output file before and after a proposed change:
+
+```sh
+node --expose-gc scripts/benchmark-router.mjs baseline /tmp/router-comparison.json
+# Apply the proposed change.
+node --expose-gc scripts/benchmark-router.mjs candidate /tmp/router-comparison.json
+```
+
+The script refuses to replace an existing baseline and exits nonzero when a comparison gate fails. Its results should be reviewed alongside deterministic concurrency/cancellation tests. Actual local-model measurements on 16 GiB and 64 GiB Macs are recorded separately in [the hardware comparison](hardware-comparison.md), including classifier failures and comparability limits. Status-file storage performance is measured separately.
+
+## Bounds and preserved behavior
+
+The router retains at most 256 pending evaluations and 1,024 subscribers. Capacity exhaustion uses the conservative fallback and reports `classifier_error: capacity_exhausted`. Each subscriber can cancel independently; the final cancellation aborts the evaluator, removes its pending entry, and prevents a late answer from entering the cache. Failed work is never cached. Coalescing shares only classification; each caller still applies its own scoped turn, context, and compatibility policy.
+
+Keys retain the full request body, including the requested model/floor, and include the evaluator endpoint, model, credentials, deadline, excerpt budget, confidence policy or keep-alive setting, and a frozen rubric hash. Only their SHA-256 digests are stored as keys; credentials and request contents are not benchmark output. Using a smaller excerpt-only key would need a separate compatibility review.
+
+Both evaluators use the same bounded JSON reader: 64 KiB for decisions and 1 MiB for local-model metadata. Streamed bytes enforce the limit even with missing or inaccurate length headers. A single growable buffer bounds chunk metadata, malformed JSON produces a stable error category, and cancellation never waits on an uncooperative body-cancellation promise. Jev's deadline now includes body reading; Ollama's configured zero deadline remains disabled while caller cancellation and byte limits remain active.
+
+Prior-pin classification reuse was considered and intentionally deferred. The benchmark shows approximately 1.3 ms median total routing time for synthetic pinned turns, while eliminating the evaluation would alter the existing evaluate-per-request behavior. Coalescing removes duplicate simultaneous evaluations without making that semantic change. New human tasks and sequential tool continuations retain their current evaluation and safety checks.

package/docs/status-performance.json

@@ -0,0 +1,363 @@
+{
+  "schema_version": 1,
+  "runs": [
+    {
+      "schema_version": 1,
+      "label": "synchronous-baseline",
+      "recorded_at": "2026-10-05T08:22:55.868Z",
+      "environment": {
+        "node": "v26.10.0",
+        "platform": "darwin",
+        "release": "25.6.0",
+        "arch": "arm64",
+        "cpu": "Apple M4",
+        "logical_cpus": 10,
+        "memory_bytes": 17179869184,
+        "load_average": [
+          3.75537109375,
+          3.2236328125,
+          3.1318359375
+        ],
+        "background_load": "Other workspace activity was not suspended; no process identities or personal data captured."
+      },
+      "method": {
+        "rounds_per_run": 60,
+        "repetitions": 3,
+        "events_per_round": 6,
+        "concurrent_sessions": 20,
+        "producer_interval_ms": 5,
+        "mock_stream_interval_ms": 5,
+        "injection": "One 20ms delay per snapshot write: blocking wait for synchronous writer, timer wait for asynchronous writer. Metadata operations are not delayed.",
+        "note": "Stream timer lateness measures event-loop interference, not a network benchmark. Flush completion may include coalesced subsequent snapshots."
+      },
+      "scenarios": [
+        {
+          "storage": "normal_temporary_directory",
+          "injected_write_delay_ms": 0,
+          "timings": {
+            "update_ms": {
+              "samples": 180,
+              "p50": 0.09,
+              "p95": 0.189,
+              "max": 1.263
+            },
+            "flush_call_ms": {
+              "samples": 180,
+              "p50": 1.07,
+              "p95": 1.955,
+              "max": 2.467
+            },
+            "flush_completion_ms": {
+              "samples": 180,
+              "p50": 1.082,
+              "p95": 2.007,
+              "max": 2.542
+            },
+            "stream_timer_lateness_ms": {
+              "samples": 186,
+              "p50": 0.996,
+              "p95": 2.36,
+              "max": 3.6
+            },
+            "startup_ms": {
+              "samples": 3,
+              "p50": 1.069,
+              "p95": 1.069,
+              "max": 1.25
+            },
+            "close_ms": {
+              "samples": 3,
+              "p50": 0.958,
+              "p95": 0.958,
+              "max": 1.062
+            }
+          },
+          "event_loop_delay_ms_by_run": [
+            {
+              "p50": 1.151,
+              "p95": 2.357,
+              "max": 4.073
+            },
+            {
+              "p50": 1.153,
+              "p95": 2.591,
+              "max": 3.912
+            },
+            {
+              "p50": 1.154,
+              "p95": 2.726,
+              "max": 3.903
+            }
+          ],
+          "snapshot_writes_by_run": [
+            61,
+            61,
+            61
+          ],
+          "rss_peak_bytes": 22137664
+        },
+        {
+          "storage": "injected_slow",
+          "injected_write_delay_ms": 20,
+          "timings": {
+            "update_ms": {
+              "samples": 180,
+              "p50": 0.219,
+              "p95": 0.304,
+              "max": 0.442
+            },
+            "flush_call_ms": {
+              "samples": 180,
+              "p50": 26.068,
+              "p95": 27.621,
+              "max": 28.401
+            },
+            "flush_completion_ms": {
+              "samples": 180,
+              "p50": 26.108,
+              "p95": 27.719,
+              "max": 28.528
+            },
+            "stream_timer_lateness_ms": {
+              "samples": 186,
+              "p50": 26.445,
+              "p95": 28.508,
+              "max": 29.835
+            },
+            "startup_ms": {
+              "samples": 3,
+              "p50": 25.292,
+              "p95": 25.292,
+              "max": 27.452
+            },
+            "close_ms": {
+              "samples": 3,
+              "p50": 1.203,
+              "p95": 1.203,
+              "max": 1.303
+            }
+          },
+          "event_loop_delay_ms_by_run": [
+            {
+              "p50": 1.151,
+              "p95": 27.853,
+              "max": 29.508
+            },
+            {
+              "p50": 1.156,
+              "p95": 28.639,
+              "max": 30.015
+            },
+            {
+              "p50": 1.158,
+              "p95": 28.836,
+              "max": 30.31
+            }
+          ],
+          "snapshot_writes_by_run": [
+            61,
+            61,
+            61
+          ],
+          "rss_peak_bytes": 22678336
+        }
+      ]
+    },
+    {
+      "schema_version": 1,
+      "label": "asynchronous-single-flight",
+      "recorded_at": "2026-10-05T08:28:48.754Z",
+      "environment": {
+        "node": "v26.10.0",
+        "platform": "darwin",
+        "release": "25.6.0",
+        "arch": "arm64",
+        "cpu": "Apple M4",
+        "logical_cpus": 10,
+        "memory_bytes": 17179869184,
+        "load_average": [
+          4.52685546875,
+          3.9130859375,
+          3.46826171875
+        ],
+        "background_load": "Other workspace activity was not suspended; no process identities or personal data captured."
+      },
+      "method": {
+        "rounds_per_run": 60,
+        "repetitions": 3,
+        "events_per_round": 6,
+        "concurrent_sessions": 20,
+        "producer_interval_ms": 5,
+        "mock_stream_interval_ms": 5,
+        "injection": "One 20ms delay per snapshot write: blocking wait for synchronous writer, timer wait for asynchronous writer. Metadata operations are not delayed.",
+        "note": "Stream timer lateness measures event-loop interference, not a network benchmark. Flush completion may include coalesced subsequent snapshots."
+      },
+      "scenarios": [
+        {
+          "storage": "normal_temporary_directory",
+          "injected_write_delay_ms": 0,
+          "timings": {
+            "update_ms": {
+              "samples": 180,
+              "p50": 0.121,
+              "p95": 0.185,
+              "max": 5.922
+            },
+            "flush_call_ms": {
+              "samples": 180,
+              "p50": 0.004,
+              "p95": 0.006,
+              "max": 0.016
+            },
+            "flush_completion_ms": {
+              "samples": 180,
+              "p50": 1.387,
+              "p95": 2.341,
+              "max": 4.168
+            },
+            "stream_timer_lateness_ms": {
+              "samples": 189,
+              "p50": 0.215,
+              "p95": 0.955,
+              "max": 1.381
+            },
+            "startup_ms": {
+              "samples": 3,
+              "p50": 2.103,
+              "p95": 2.103,
+              "max": 3.156
+            },
+            "close_ms": {
+              "samples": 3,
+              "p50": 2.203,
+              "p95": 2.203,
+              "max": 2.358
+            }
+          },
+          "event_loop_delay_ms_by_run": [
+            {
+              "p50": 1.143,
+              "p95": 1.747,
+              "max": 7.68
+            },
+            {
+              "p50": 1.151,
+              "p95": 1.838,
+              "max": 4.184
+            },
+            {
+              "p50": 1.152,
+              "p95": 1.766,
+              "max": 2.052
+            }
+          ],
+          "snapshot_writes_by_run": [
+            61,
+            61,
+            61
+          ],
+          "rss_peak_bytes": 23284496
+        },
+        {
+          "storage": "injected_slow",
+          "injected_write_delay_ms": 20,
+          "timings": {
+            "update_ms": {
+              "samples": 180,
+              "p50": 0.125,
+              "p95": 0.193,
+              "max": 0.28
+            },
+            "flush_call_ms": {
+              "samples": 180,
+              "p50": 0.004,
+              "p95": 0.006,
+              "max": 0.025
+            },
+            "flush_completion_ms": {
+              "samples": 180,
+              "p50": 199.18,
+              "p95": 342.723,
+              "max": 361.781
+            },
+            "stream_timer_lateness_ms": {
+              "samples": 212,
+              "p50": 0.065,
+              "p95": 1.042,
+              "max": 1.299
+            },
+            "startup_ms": {
+              "samples": 3,
+              "p50": 22.754,
+              "p95": 22.754,
+              "max": 23.073
+            },
+            "close_ms": {
+              "samples": 3,
+              "p50": 1.595,
+              "p95": 1.595,
+              "max": 2.428
+            }
+          },
+          "event_loop_delay_ms_by_run": [
+            {
+              "p50": 1.158,
+              "p95": 1.509,
+              "max": 2.224
+            },
+            {
+              "p50": 1.154,
+              "p95": 1.451,
+              "max": 2.001
+            },
+            {
+              "p50": 1.154,
+              "p95": 1.501,
+              "max": 2.061
+            }
+          ],
+          "snapshot_writes_by_run": [
+            17,
+            17,
+            17
+          ],
+          "rss_peak_bytes": 23874368
+        }
+      ],
+      "baseline_comparison": {
+        "baseline_label": "synchronous-baseline",
+        "scope": "Local synthetic regression thresholds derived from this recorded baseline; not portable latency promises or ordinary CI gates.",
+        "gates": [
+          {
+            "name": "slow_storage_stream_p95_ms",
+            "actual": 1.042,
+            "maximum": 7.127,
+            "passed": true,
+            "rationale": "Require at least 75% less interference than the measured blocking-storage baseline."
+          },
+          {
+            "name": "slow_storage_flush_call_p95_ms",
+            "actual": 0.006,
+            "maximum": 2.762,
+            "passed": true,
+            "rationale": "Require at least 90% less synchronous caller time when storage is slow."
+          },
+          {
+            "name": "normal_storage_stream_p95_ms",
+            "actual": 0.955,
+            "maximum": 2.95,
+            "passed": true,
+            "rationale": "Allow 25% baseline headroom for scheduler noise while checking normal-storage regression."
+          },
+          {
+            "name": "normal_storage_update_p95_ms",
+            "actual": 0.185,
+            "maximum": 0.378,
+            "passed": true,
+            "rationale": "Allow twice the small pre-existing normalization/update cost; this is not a disk-latency bound."
+          }
+        ]
+      }
+    }
+  ]
+}

package/docs/status-performance.md

@@ -0,0 +1,44 @@
+# Status persistence benchmark
+
+Status snapshots now use asynchronous atomic replacement with one writer and one dirty flag. Slow storage can delay the status display without blocking request handling or response streaming. Events received during a write are folded into the latest bounded session state; the writer does not queue a snapshot for every event.
+
+The [recorded results](status-performance.json) compare the original synchronous writer, captured before implementation, with the asynchronous writer on an Apple M4, 16 GiB RAM, macOS Darwin 25.6.0, Node 26.10.0. Other workspace activity continued during both runs; load averages and peak process RSS are recorded. These are local synthetic observations, not provider latency or a cross-machine performance claim.
+
+Each storage scenario has three runs of 60 bursts, six telemetry events per burst, 20 rotating sessions, and a 5 ms producer interval. A separate 5 ms timer feeds a local mock stream. The slow-storage case injects 20 ms into each snapshot write: a blocking wait for the original writer and an asynchronous wait for its replacement. It does not delay filesystem metadata operations. The stream measurement is lateness beyond its expected timer interval, not network throughput.
+
+| Measurement, milliseconds unless stated | Normal synchronous | Normal asynchronous | Slow synchronous | Slow asynchronous |
+| --- | ---: | ---: | ---: | ---: |
+| Update burst p50 / p95 | 0.090 / 0.189 | 0.121 / 0.185 | 0.219 / 0.304 | 0.125 / 0.193 |
+| `flush()` caller p50 / p95 | 1.070 / 1.955 | 0.004 / 0.006 | 26.068 / 27.621 | 0.004 / 0.006 |
+| Mock stream lateness p50 / p95 | 0.996 / 2.360 | 0.215 / 0.955 | 26.445 / 28.508 | 0.065 / 1.042 |
+| Snapshot writes per run, including initial file | 61 | 61 | 61 | 17 |
+| Peak process RSS, MiB | 21.11 | 22.21 | 21.63 | 22.77 |
+
+Under injected slow storage, stream lateness p95 fell by 96.3%. The asynchronous `flush()` promise still waits for persistence: its slow-storage completion p95 was 342.723 ms because it drains updates arriving during that write sequence. Request handling calls `update()` and does not await this drain. The JSON includes completion latency, initialization, shutdown, event-loop histograms, and maximum values, including scheduler outliers omitted from the compact table.
+
+## Local regression gates
+
+The harness derives these gates from the recorded synchronous baseline. All four passed on the measured machine:
+
+| Gate | Maximum | Measured |
+| --- | ---: | ---: |
+| Slow-storage stream lateness p95: at least 75% lower | 7.127 ms | 1.042 ms |
+| Slow-storage synchronous flush cost p95: at least 90% lower | 2.762 ms | 0.006 ms |
+| Normal-storage stream lateness p95: baseline plus 25% | 2.950 ms | 0.955 ms |
+| Normal-storage update p95: twice baseline | 0.378 ms | 0.185 ms |
+
+These numerical checks are opt-in and sensitive to host load; they do not run as ordinary CI timing tests. Repeat locally with:
+
+```sh
+node scripts/benchmark-status.mjs --label local-check --check
+```
+
+This appends or replaces that label in `docs/status-performance.json`. `--output PATH` selects a separate report; `--module PATH` can benchmark another checkout's `src/status-state.mjs`. `--check` requires an existing `synchronous-baseline` entry in the selected report. No sockets, inference, credentials, or user session data are used.
+
+## Persistence contract
+
+`createStatusState()` returns immediately. Callers await `ready` before reading `path`; the path becomes available only after the first complete snapshot exists. Initial storage failure or a one-second readiness deadline disables the optional status display. Accepted snapshots use a private directory with mode `0700`, exclusive temporary files with mode `0600`, and atomic rename. Filesystem errors do not reject `ready`, `flush()`, or `close()`.
+
+`close()` stops new updates, waits for the active write and latest accepted state, then removes the private directory. Native filesystem operations cannot be cancelled, so shutdown retains ownership of an outstanding operation even if startup readiness has already timed out. It never removes the directory while a detached writer could recreate a file.
+
+Deterministic tests hold a write unresolved while an entire mock inference stream finishes; 2,000 update bursts produce only one subsequent snapshot. Other tests cover concurrent close calls, late updates, readiness timeout, temporary-file cleanup, recovery after a failed rename, permissions, and foreground/session isolation.

package/package.json

@@ -1,16 +1,58 @@
 {
   "name": "claude-autorouter",
-  "version": "0.3.6",
+  "version": "0.4.0",
   "license": "Apache-2.0",
   "type": "module",
   "description": "A Claude Code model router with Jev and local Ollama System One evaluators",
-  "repository": { "type": "git", "url": "git+https://github.com/frapposelli/claude-autorouter.git" },
-  "bin": { "claude-autorouter": "bin/autorouter.mjs" },
-  "engines": { "node": ">=22" },
-  "os": ["darwin", "linux"],
-  "keywords": ["claude", "claude-code", "model-routing", "jev", "ollama", "nimble", "tev1", "systemone", "cli"],
-  "files": ["bin/*.mjs", "src/*.mjs", "docs/reference.md", "docs/development.md", "docs/releasing.md", "docs/ollama-evaluation.md", ".env.example", "LICENSE"],
-  "publishConfig": { "access": "public", "registry": "https://registry.npmjs.org/" },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/frapposelli/claude-autorouter.git"
+  },
+  "bin": {
+    "claude-autorouter": "bin/autorouter.mjs"
+  },
+  "engines": {
+    "node": ">=22"
+  },
+  "os": [
+    "darwin",
+    "linux"
+  ],
+  "keywords": [
+    "claude",
+    "claude-code",
+    "model-routing",
+    "jev",
+    "ollama",
+    "nimble",
+    "tev1",
+    "systemone",
+    "cli"
+  ],
+  "files": [
+    "bin/*.mjs",
+    "src/*.mjs",
+    "docs/reference.md",
+    "docs/development.md",
+    "docs/releasing.md",
+    "docs/ollama-evaluation.md",
+    ".env.example",
+    "LICENSE",
+    "CONTRIBUTING.md",
+    "docs/router-performance.md",
+    "docs/router-performance.json",
+    "docs/status-performance.md",
+    "docs/status-performance.json",
+    "docs/hardware-benchmark.md",
+    "docs/hardware-results-16gb.md",
+    "docs/hardware-results-16gb.json",
+    "docs/hardware-comparison.md",
+    "docs/hardware-results-64gb.json"
+  ],
+  "publishConfig": {
+    "access": "public",
+    "registry": "https://registry.npmjs.org/"
+  },
   "scripts": {
     "start": "node bin/autorouter.mjs serve",
     "claude": "node bin/autorouter.mjs claude",
@@ -21,6 +63,12 @@
     "eval": "node --env-file=.env scripts/evaluate.mjs",
     "eval:ollama": "node scripts/evaluate-ollama.mjs",
     "test:live": "node --env-file=.env scripts/live-validation.mjs",
-    "check": "node scripts/check.mjs"
+    "check": "node scripts/check.mjs",
+    "benchmark:router": "node --expose-gc scripts/benchmark-router.mjs",
+    "benchmark:status": "node scripts/benchmark-status.mjs"
+  },
+  "devDependencies": {
+    "@types/node": "22.20.5",
+    "typescript": "6.0.3"
   }
 }