claude-token-saver 1.3.1__tar.gz

claude_token_saver-1.3.1/BENCHMARKING.md

@@ -0,0 +1,245 @@
+# Verify integration, then benchmark successful work
+
+## Deterministic context-quality benchmark
+
+Before paid paired-agent trials, run the included 25-task ground-truth selector
+benchmark. It measures relevant-file recall, relevant-symbol recall, and context
+reduction; selection metrics alone do not prove agent success.
+
+```bash
+token-saver evaluate benchmarks/context-quality.json --path . --max-tokens 6000
+```
+
+Each item also reports `symbol_recall_in_expected_files`, which counts a
+symbol only when it was selected from one of the task's expected files. Bare
+`symbol_recall` can be satisfied by a same-named symbol in an unrelated file;
+prefer the scoped figure (or `qualified_symbols` / `symbol_identities`) when
+names are common.
+
+Add project-specific tasks using `query`, `files`, and `symbols`. Keep the
+manifest under version control so ranking changes can be compared reproducibly.
+
+## Multi-repository holdout benchmark
+
+The repository-local selector benchmark is useful for regressions, but because
+Token Saver is developed against this codebase it is not independent evidence of
+generalization. For unseen evaluation, define ground truth before running the
+tool and point one manifest at repositories that were excluded from ranking
+work/tuning. `benchmarks/holdout.example.json` contains the full schema.
+
+Repository paths are resolved relative to the manifest. Pin exact Git commits so
+the corpus cannot move between runs. A publishable holdout also records a freeze
+timestamp and a SHA-256 of the task/repository ground-truth definition:
+
+```json
+{
+  "suite_version": 1,
+  "protocol": {
+    "ground_truth_frozen": true,
+    "development_excluded": true,
+    "frozen_at": "2026-09-17T12:00:00Z",
+    "ground_truth_sha256": "HASH_FROM_COMMAND_BELOW"
+  },
+  "repositories": {
+    "app-a": {"path": "../app-a", "revision": "ACTUAL_COMMIT_SHA"},
+    "app-b": {"path": "../app-b", "revision": "ACTUAL_COMMIT_SHA"}
+  },
+  "tasks": [
+    {
+      "id": "auth-refresh",
+      "repository": "app-a",
+      "query": "session refresh after logout",
+      "files": ["src/auth/session.ts"],
+      "symbols": ["refreshSession"]
+    }
+  ]
+}
+```
+
+After the tasks, expected evidence, and revision pins are final, calculate the
+freeze hash without running retrieval:
+
+```bash
+token-saver evaluate benchmarks/holdout.json --print-ground-truth-hash
+```
+
+Put that value in `protocol.ground_truth_sha256`, commit the manifest, then run:
+
+```bash
+token-saver evaluate benchmarks/holdout.json --require-holdout --max-tokens 6000
+```
+
+`--require-holdout` rejects a manifest unless both protocol flags are true,
+`frozen_at` is present, the SHA-256 still matches the frozen task definition,
+and every pinned repository `HEAD` matches its declared revision. Filesystem
+paths are excluded from the freeze hash so the same manifest can be replicated
+on another machine without changing the benchmark identity. The output includes
+aggregate metrics, per-repository summaries, and the verified hash.
+
+A valid hash proves the evaluated definition did not change after it was frozen;
+it does not by itself prove the labels were independently authored before tuning.
+Preserve manifest history and the task-definition process as audit evidence.
+
+## Paired agent outcomes
+
+Record independently validated baseline and Token Saver runs using the schema in
+`benchmarks/agent-runs.example.json`, then run:
+
+```bash
+token-saver agent-evaluate benchmarks/agent-runs.json
+```
+
+The evaluator requires exactly one run per condition and task. It reports input
+and output tokens, retries, elapsed time, context failures, success rate, and
+tokens per success. It suppresses the reduction headline whenever Token Saver's
+success rate is below baseline.
+
+## Live host validation (not yet executed for this release)
+
+Token Saver can verify installation and its local hook/recovery transport before
+a live host trial:
+
+```bash
+token-saver host-check . --require-ready
+```
+
+This checks project/user hook configuration, probes the host executable version,
+runs a synthetic 500-line Bash response through the real PostToolUse hook, and
+retrieves an omitted middle line from Token Saver's saved original output. That
+is a local transport test; it does **not** prove the host actually feeds
+`hookSpecificOutput.updatedToolOutput` back to the model.
+
+For that final gate, capture a real host debug transcript and supply it explicitly:
+
+```bash
+token-saver host-check . \
+  --live-evidence /path/to/claude-debug.log \
+  --require-live
+```
+
+`live_verified=true` is reported only when the supplied evidence contains both
+the host replacement field and Token Saver's filtered-output recovery marker.
+The manual validation protocol remains:
+
+1. Record `claude --version`, model ID, configuration, and Token Saver version.
+2. Use a disposable project. Install the package and its project hooks. Check
+   that inherited user hooks do not run Token Saver a second time.
+3. Start Claude Code with debugging enabled. Ask it to run a harmless command
+   that prints 500 distinct progress lines. Do not use a command with side effects.
+4. Verify the host accepts `hookSpecificOutput.updatedToolOutput` without a
+   validation error. The model-visible result should include the recovery note,
+   show the shortened head/tail, and preserve the structured Bash fields.
+   Hook stdout alone is not sufficient evidence of host acceptance.
+5. Ask it to retrieve a known omitted middle line using the supplied
+   `token-saver output` command. Verify the exact line without rerunning the
+   original command. Test stderr independently and test a failing Jest-style
+   output containing a test name, stack location and multiline assertion diff.
+6. Test a >220-line source read. A bounded Read must return actual bytes, and a
+   subsequent Edit must succeed. Repeat after `/compact`; prior full-read state
+   must not block access to needed source. Test two independent sessions.
+7. Save the version, debug evidence and outcomes. If the installed host does not
+   support structured replacement, use manual `filter` until upgraded. Do not
+   claim automatic savings based on an ignored hook field.
+
+No paid model call or live Claude Code session was executed in the repair
+workspace. Automated tests cover the documented contract, subprocess transport,
+state isolation, retrieval, and accounting. Live behavior remains a separate gate.
+
+## Paired task protocol
+
+- Choose representative tasks before measuring: bug fixes, refactors, unfamiliar
+  repo navigation, noisy passing tests, and failures requiring deep diagnostics.
+- Use independent fresh worktrees at the same commit and the same exact prompt,
+  model, effort, tools, system instructions, and approval configuration.
+- In baseline, disable all Token Saver hooks (including user-scope hooks). In
+  enabled runs, use the release's hooks. Do not add orientation maps to only one
+  arm unless that is the specific intervention being tested.
+- Randomize condition order and run multiple trials per task. Record cache
+  policy and cold/warm conditions rather than assuming a five-minute TTL.
+- Keep tests/evaluation independent of the agent. Record success, regression
+  checks, elapsed time, retries and any manual intervention. A smaller context
+  that fails the task is not a win.
+- Preserve each run's complete transcripts, including nested subagent
+  transcripts where applicable. Do not reuse a transcript across runs.
+- Keep raw transcripts local; they may contain code or secrets.
+
+`token-saver benchmark` analyzes recorded runs; it does not execute coding agents
+or certify evaluator outcomes. It verifies paired task/trial identity, revision,
+model and prompt metadata, requires measured usage and explicit prices, and
+rejects incomplete or duplicate runs. Metadata must be recorded honestly by the
+runner; the evaluator does not independently attest your checkout or prompt.
+
+## Run manifest
+
+Paths are relative to the manifest file. `success` is determined by independent
+validation, not by the agent's own assertion. Use the actual SHA-256 of the prompt.
+
+```json
+{
+  "runs": [
+    {
+      "task": "fix-user-lookup",
+      "trial": 1,
+      "condition": "baseline",
+      "revision": "ACTUAL_COMMIT",
+      "model": "EXACT_MODEL_ID",
+      "prompt_sha256": "ACTUAL_PROMPT_SHA256",
+      "success": true,
+      "validation": "Independent tests passed; no regressions in required suite",
+      "seconds": 120.5,
+      "transcripts": ["runs/baseline.jsonl"]
+    },
+    {
+      "task": "fix-user-lookup",
+      "trial": 1,
+      "condition": "enabled",
+      "revision": "ACTUAL_COMMIT",
+      "model": "EXACT_MODEL_ID",
+      "prompt_sha256": "ACTUAL_PROMPT_SHA256",
+      "success": true,
+      "validation": "Same independent checks passed",
+      "seconds": 118.2,
+      "transcripts": ["runs/enabled.jsonl"]
+    }
+  ]
+}
+```
+
+The times above demonstrate the schema; they are not measured results.
+
+## Rate file
+
+Create a JSON object mapping each exact model ID to numeric USD-per-million
+rates: `input`, `cache_write_5m`, `cache_write_1h`, `cache_read`, `output`.
+For example, this object is syntactically valid but deliberately uses a fictitious
+model and synthetic prices. **It must not be used to price actual models.**
+
+```json
+{
+  "synthetic-test-model": {
+    "input": 1,
+    "cache_write_5m": 1.25,
+    "cache_write_1h": 2,
+    "cache_read": 0.1,
+    "output": 5
+  }
+}
+```
+
+Only add `cache_write_unknown` when the run's actual cache configuration permits
+an explicit rate for missing TTL details. Missing prices or unexplained cache
+creation prevent a complete cost result.
+
+```bash
+token-saver benchmark runs.json --rates rates.json
+```
+
+The output includes per-condition success rates, total tokens by usage type,
+tool-result counts, repeated reads, elapsed time, total cost, and cost per success.
+Costs from failed attempts are included. A reduced success rate suppresses the
+headline cost-per-success reduction. Inspect per-task outcomes too: aggregate
+success parity does not prove each task retained the same quality.
+
+No confidence interval or statistical significance is claimed. Before publishing
+savings, retain task definitions, repeated trials, model versions, host versions,
+prices, independent checks, and raw results needed to reproduce the claim.