its-magic 0.1.2-35 → 0.1.2-36

package/template/docs/engineering/runbook.md

@@ -315,6 +315,21 @@ Manual override precedence:
 - Profile changes must not disable mandatory gate contracts
   (`/qa`, `/verify-work`, `/release`).
 
+### Token-cost evidence + comparability (US-0080 / DEC-0062)
+
+- **Fresh context**: spawn **new** subagents per `/auto` phase; avoid carrying prior chat
+  reasoning as phase input.
+- **`start-from`**: use **`/auto start-from=<canonical_phase_id>`** when resuming so the
+  schedule intersection matches materialized **`resolved_phase_plan`** (**`DEC-0052`**).
+- **`TOKEN_PROFILE`**: `lean` lowers default automation breadth; does **not** remove
+  isolation, strict-proof, role, or release gates.
+- **Metrics**: append-only **`handoffs/token_cost_runs/<orchestrator_run_id>.md`** (or
+  **`.jsonl`**); copy path into **`token_cost_evidence_ref`** on **`state.md`** checkpoints.
+- **AC-2**: compare **`cache_read_tokens`** only when **`run_class_hash`** matches; else
+  **`TOKEN_COST_RUN_CLASS_MISMATCH`**.
+- **CI/repo checks**: `python scripts/check_token_cost_parity.py --repo .` (manifest-listed
+  active/`template/` pairs); **`tests/run-tests.ps1`** / **`tests/run-tests.sh`** §26M.
+
 Context compaction policy:
 
 - `docs/engineering/state.md` is a compact hot surface for current execution

package/template/docs/engineering/token-cost-parity-manifest.md

@@ -0,0 +1,16 @@
+# Token-cost parity manifest (DEC-0062 §5 / US-0080)
+
+- **version**: `1`
+- **purpose**: Paths below MUST be **byte-identical** between repository root and
+  `template/` after edits (installer parity). CI: `python scripts/check_token_cost_parity.py --repo .`
+
+## Mirrored path pairs (`active` → `template`)
+
+- `.cursor/commands/auto.md` → `template/.cursor/commands/auto.md`
+- `.cursor/commands/execute.md` → `template/.cursor/commands/execute.md`
+- `docs/engineering/auto-orchestration-reference.md` → `template/docs/engineering/auto-orchestration-reference.md`
+- `docs/engineering/token-cost-parity-manifest.md` → `template/docs/engineering/token-cost-parity-manifest.md`
+- `scripts/token_cost_lib.py` → `template/scripts/token_cost_lib.py`
+- `scripts/token_cost_compare.py` → `template/scripts/token_cost_compare.py`
+- `scripts/check_token_cost_parity.py` → `template/scripts/check_token_cost_parity.py`
+- `handoffs/token_cost_runs/README.md` → `template/handoffs/token_cost_runs/README.md`

package/template/handoffs/token_cost_runs/README.md

@@ -0,0 +1,26 @@
+# Token-cost run evidence (`handoffs/token_cost_runs/`)
+
+Append-only artifacts per **`orchestrator_run_id`** (**`DEC-0062`** §3, **`US-0080`**).
+
+## Schema (markdown table)
+
+Each run file SHOULD include:
+
+- `orchestrator_run_id` — stable id for the orchestrated run
+- `run_class_hash` — SHA-256 hex over canonical **`DEC-0062`** §2 JSON object
+- `metric_source` — how numbers were produced (for example `cursor_usage_export`,
+  `fixture`, `unmapped_host`)
+- `recorded_at_utc` — RFC3339 UTC
+- **Run totals** row: `cache_read_tokens`, `input_tokens`, `output_tokens`, and
+  optional `cache_creation_tokens`, `orchestrator_call_estimate`
+- **Per-phase** rows: same metrics plus `phase_id`, `phase_call_count`
+
+`docs/engineering/state.md` checkpoints SHOULD set **`token_cost_evidence_ref`**
+to the primary file path when metrics exist.
+
+## Tooling
+
+- `scripts/token_cost_lib.py` — `run_class_hash`, strict-proof hash helper, AC-2 compare
+- `scripts/token_cost_compare.py <baseline.json> <target.json>` — exit **0** if
+  same `run_class_hash` and ≥ **50%** `cache_read_tokens` reduction vs baseline
+- `python scripts/check_token_cost_parity.py --repo .` — active/`template/` parity

package/template/scripts/check_token_cost_parity.py

@@ -0,0 +1,69 @@
+#!/usr/bin/env python3
+"""Verify active vs template/ bytes match for DEC-0062 token-cost parity manifest paths."""
+
+from __future__ import annotations
+
+import argparse
+import re
+import sys
+from pathlib import Path
+
+_PAIR_RE = re.compile(r"`([^`]+)`\s*→\s*`([^`]+)`")
+
+
+def load_pairs(manifest_text: str) -> list[tuple[Path, Path]]:
+    pairs: list[tuple[Path, Path]] = []
+    for line in manifest_text.splitlines():
+        line = line.strip()
+        if not line.startswith("- "):
+            continue
+        m = _PAIR_RE.search(line)
+        if not m:
+            continue
+        pairs.append((Path(m.group(1)), Path(m.group(2))))
+    return pairs
+
+
+def main() -> int:
+    p = argparse.ArgumentParser(description=__doc__)
+    p.add_argument(
+        "--repo",
+        type=Path,
+        default=Path(__file__).resolve().parent.parent,
+        help="Repository root",
+    )
+    args = p.parse_args()
+    root: Path = args.repo
+    manifest = root / "docs/engineering/token-cost-parity-manifest.md"
+    if not manifest.is_file():
+        print(f"[TOKEN_COST_PARITY_ERROR] missing manifest: {manifest}")
+        return 2
+    text = manifest.read_text(encoding="utf-8")
+    pairs = load_pairs(text)
+    if not pairs:
+        print("[TOKEN_COST_PARITY_ERROR] no path pairs parsed from manifest")
+        return 2
+    failed = False
+    for rel_active, rel_tpl in pairs:
+        a = root / rel_active
+        t = root / rel_tpl
+        if not a.is_file() or not t.is_file():
+            print(f"[TOKEN_COST_PARITY_ERROR] missing file: {rel_active} or {rel_tpl}")
+            failed = True
+            continue
+        ba = a.read_bytes()
+        bt = t.read_bytes()
+        if ba != bt:
+            print(
+                f"[TOKEN_COST_PARITY_ERROR] mismatch: {rel_active} ({len(ba)}b) "
+                f"!= {rel_tpl} ({len(bt)}b)"
+            )
+            failed = True
+    if failed:
+        return 2
+    print("[TOKEN_COST_PARITY_OK]")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

package/template/scripts/token_cost_compare.py

@@ -0,0 +1,40 @@
+#!/usr/bin/env python3
+"""Compare two token-cost run JSON records for AC-2 (same run_class_hash, 50% cache read)."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from pathlib import Path
+
+_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
+if _SCRIPT_DIR not in sys.path:
+    sys.path.insert(0, _SCRIPT_DIR)
+
+import token_cost_lib  # noqa: E402
+
+
+def main() -> int:
+    p = argparse.ArgumentParser(description=__doc__)
+    p.add_argument("baseline", type=Path, help="JSON file with run_class_hash + totals")
+    p.add_argument("target", type=Path, help="JSON file with run_class_hash + totals")
+    p.add_argument(
+        "--reduction-fraction",
+        type=float,
+        default=0.5,
+        help="Required fractional reduction in cache_read_tokens (default 0.5)",
+    )
+    args = p.parse_args()
+    base = json.loads(args.baseline.read_text(encoding="utf-8"))
+    tgt = json.loads(args.target.read_text(encoding="utf-8"))
+    ok, msg = token_cost_lib.compare_cache_read_reduction(
+        base, tgt, reduction_fraction=args.reduction_fraction
+    )
+    print(msg)
+    return 0 if ok else 2
+
+
+if __name__ == "__main__":
+    sys.exit(main())

package/template/scripts/token_cost_lib.py

@@ -0,0 +1,108 @@
+"""Token-cost run class, metrics, and AC-2 comparability helpers (US-0080 / DEC-0062)."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from typing import Any, Mapping
+
+# Canonical metric keys (DEC-0062 §1) — totals / per-phase rows
+METRIC_KEYS = (
+    "cache_read_tokens",
+    "input_tokens",
+    "output_tokens",
+    "phase_call_count",
+    "cache_creation_tokens",
+    "orchestrator_call_estimate",
+)
+
+REQUIRED_TOTAL_KEYS = (
+    "cache_read_tokens",
+    "input_tokens",
+    "output_tokens",
+)
+
+
+def canonical_json_dumps(obj: Mapping[str, Any]) -> str:
+    return json.dumps(obj, sort_keys=True, separators=(",", ":"))
+
+
+def compute_run_class_hash(run_class: Mapping[str, Any]) -> str:
+    """SHA-256 hex (lowercase) of canonical JSON for DEC-0062 §2 tuple."""
+    payload = canonical_json_dumps(run_class)
+    return hashlib.sha256(payload.encode("utf-8")).hexdigest()
+
+
+def compute_strict_proof_hash(
+    orchestrator_run_id: str,
+    runtime_proof_id: str,
+    phase_id: str,
+    role: str,
+    proof_issued_at: str,
+    proof_ttl_seconds: int,
+) -> str:
+    """DEC-0038-style sorted-key JSON SHA-256 over the strict-proof tuple fields."""
+    obj = {
+        "orchestrator_run_id": orchestrator_run_id,
+        "runtime_proof_id": runtime_proof_id,
+        "phase_id": phase_id,
+        "role": role,
+        "proof_issued_at": proof_issued_at,
+        "proof_ttl_seconds": int(proof_ttl_seconds),
+    }
+    return hashlib.sha256(canonical_json_dumps(obj).encode("utf-8")).hexdigest()
+
+
+def validate_run_totals(row: Mapping[str, Any]) -> list[str]:
+    """Return diagnostic strings for invalid totals; empty if OK."""
+    errs: list[str] = []
+    for k in REQUIRED_TOTAL_KEYS:
+        if k not in row:
+            errs.append(f"missing_metric:{k}")
+            continue
+        v = row[k]
+        if not isinstance(v, int) or v < 0:
+            errs.append(f"invalid_non_negative_int:{k}")
+    if "metric_source" not in row or not str(row.get("metric_source", "")).strip():
+        errs.append("missing_metric_source")
+    return errs
+
+
+def compare_cache_read_reduction(
+    baseline: Mapping[str, Any],
+    target: Mapping[str, Any],
+    *,
+    reduction_fraction: float = 0.5,
+) -> tuple[bool, str]:
+    """
+    AC-2: target must achieve >= reduction_fraction lower cache_read_tokens
+    vs baseline when run_class_hash matches. Otherwise TOKEN_COST_RUN_CLASS_MISMATCH.
+    """
+    hb = baseline.get("run_class_hash")
+    ht = target.get("run_class_hash")
+    if not isinstance(hb, str) or not isinstance(ht, str) or hb != ht:
+        return False, "TOKEN_COST_RUN_CLASS_MISMATCH: run_class_hash differs or missing"
+
+    tb = baseline.get("totals") or {}
+    tt = target.get("totals") or {}
+    if not isinstance(tb, Mapping) or not isinstance(tt, Mapping):
+        return False, "TOKEN_COST_RUN_CLASS_MISMATCH: totals object missing"
+
+    br = tb.get("cache_read_tokens")
+    tr = tt.get("cache_read_tokens")
+    if not isinstance(br, int) or not isinstance(tr, int) or br < 0 or tr < 0:
+        return False, "TOKEN_COST_RUN_CLASS_MISMATCH: invalid cache_read_tokens"
+
+    if br == 0:
+        if tr == 0:
+            return True, "ok: baseline and target zero cache_read_tokens"
+        return False, "TOKEN_COST_RUN_CLASS_MISMATCH: baseline zero but target non-zero"
+
+    max_target = int(br * (1.0 - reduction_fraction))
+    if tr > max_target:
+        return (
+            False,
+            f"TOKEN_COST_RUN_CLASS_MISMATCH: cache_read_tokens {tr} exceeds "
+            f"50% reduction threshold (baseline {br}, max_target {max_target})",
+        )
+    return True, "ok: cache_read_tokens meets 50% reduction vs baseline"