cctally 1.64.0 → 1.66.0

package/CHANGELOG.md

@@ -5,6 +5,56 @@ based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+## [1.66.0] - 2026-07-10
+
+### Added
+- Reporting `--json` output now carries a `schemaVersion: 1` key (rendered first) so downstream consumers can version-gate against the payload shape. It is added to every reporting surface that lacked it — `daily`/`daily --instances`/`monthly`/`weekly`/`session`, the Codex `codex-daily`/`codex-monthly`/`codex-weekly`/`codex-session` reports, `blocks`, `forecast`, `report` (`dollar-per-percent`), `project`, `range-cost`, `cache-report`, `percent-breakdown`, `sync-week`, `telemetry`, and the `budget set`/`unset` actions — including their empty and error forms. The already-versioned surfaces (`five-hour-blocks`, `five-hour-breakdown`, `budget` status, `record-credit`, `pricing-check`) are unchanged, and the shipped snake-case `schema_version` surfaces (`diff`, `doctor`, `db status`, `refresh-usage`, `setup`) keep their existing key. Existing keys are unchanged and consumers should tolerate unknown keys (#279).
+- The cache sync now counts JSONL lines it can't parse (malformed) or has to drift-skip (usage/model/timestamp no longer parse), per vendor, and surfaces them on the `cache-sync` done lines, the hook-tick log line, and a new `cctally doctor` "Ingest parse health" check — so a Claude Code / Codex session-format change that would silently affect your cost numbers becomes visible instead of vanishing (#279).
+- `CCTALLY_DEBUG=1` now makes `cctally` print the full Python traceback on stderr when a command crashes (default output is unchanged), and `CCTALLY_DEBUG_LOG=<path>` additionally appends the backend log to a file — so a bug is reportable without editing source (#279).
+- The dashboard now logs server errors (HTTP 500) to the terminal running it, instead of swallowing them silently; routine client errors and the per-request access log stay quiet (#279).
+- `cctally doctor` gains a SQLite integrity check (`PRAGMA quick_check`, run only from the CLI where it's affordable — FAILs on stats.db corruption with safe recovery guidance, WARNs on the re-derivable cache.db) and an informational lock-state check that reports whether a sync lock is currently held (#279).
+- `CCTALLY_PERF_TRACE=1 cctally cache-sync` now traces both the Claude and Codex ingest under one shared `cache-sync` phase tree, with the Codex sync carrying the same flock/discover/walk timing seams as the Claude sync (#279).
+- New cache migration `020_session_entries_physical_unique` adds a `UNIQUE(source_path, line_offset)` backstop to the Claude session cache (matching the Codex table) so an offset-bookkeeping regression can never silently double-count your cost data — a collision fails loudly and rolls back that file instead. It dedups any pre-existing duplicates (keeping the first-ingested row) and runs automatically; `cache.db` is re-derivable, so `cache-sync --rebuild` remains the escape hatch (#279).
+- `cctally pricing-check` now flags embedded-pricing allowlist suppressions that no longer correspond to a real LiteLLM divergence (`staleSuppressions`) or that are past a new optional `expires` cutover date (`expiredSuppressions`) — two additive `--json` arrays and an exit-1 actionable finding — and the weekly pricing-freshness workflow files them on its auto-tracked issue, so a deliberate pricing override can't silently ossify past its stated cutover (#279).
+- Codex pricing now covers the `gpt-5.6` model family (`gpt-5.6`, `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`), so cost for those models is computed from their own published rates instead of falling back to `gpt-5` pricing.
+
+### Changed
+- `cctally project` and `cctally forecast` now exit `2` (was `1`) on their own usage/validation errors — a bad `--weeks`, `--weeks` combined with `--since`/`--until`, `--since > --until` or an unparseable date for `project`; `--json` together with `--status-line` or a bad `--targets` for `forecast` — matching the rest of the cctally-native command family (`diff`, `budget`, `five-hour-*`, `pricing-check`, `doctor`). The ccusage-parity commands (`daily`/`monthly`/`weekly`/`session`/`blocks`/`codex-*`) keep their exit `1` on bad dates for upstream parity. The exit-code convention is now documented in `docs/cli-contract.md` (#279).
+- Release tooling (maintainer): the npm-publish poll now hard-fails on timeout with `--resume` guidance instead of silently reporting success; `--npm-soft-timeout` restores the old exit-0 behavior and marks the final line "published (npm pending verification)".
+- Internal: the record write path (resets-at plausibility, weekly credit / reset-to-zero debounce, 5h in-place credit, the reset-aware HWM clamp + monotonic HWM files, 5h milestone ranges, and the projected-pace alert crossings) and the forecast projection now route their decision logic through pure, unit-tested kernels (`_lib_record.py`, `_lib_credit.py`, `_lib_forecast.py`, `_lib_five_hour.py`); no behavior change (#279).
+- Internal: CLI-architecture consistency pass — `build_parser()` is now a loop over an ordered per-command registration table of builder functions (every subcommand's `--help` output is byte-for-byte unchanged, verified by a recursive sweep across the flat commands, the `claude`/`codex` subgroups, and the hidden commands); the four `_iso_z` UTC-Z serializers collapse to one canonical helper (doctor keeps its deliberately divergent copy); the `_eprint` / `_load_lib` / `_ensure_sibling_loaded` per-module bootstrap copies are normalized and pinned by AST drift-guard tests; and the three deviant thin bash wrappers are aligned to the shared template. No behavior change (#279).
+- Internal: the dashboard server module (`_cctally_dashboard.py`, ~10.5K lines) is split into four consumer-only sibling modules (share, envelope, cache-report, conversation) and its HTTP request routing is now table-driven; every endpoint's response bytes and status codes are byte-identical, verified against a real data corpus (#279).
+- `cctally report --json`'s `generatedAt` timestamp now routes through the same as-of clock as the rest of the report (unchanged in normal use — it still reflects wall-clock when no override is set), making the JSON output reproducible under a pinned clock (#279).
+- Internal: test-infrastructure hardening — every migration in both registries now ships a per-migration golden with an enforced registry-completeness guard and a documented handler-idempotency contract; a new ancient-DB→head chain test covers migration interaction (the recompute gate + the FTS search-split→drop→title sequence); `pytest-timeout` is wired as an optional-by-detection dev dependency with a deliberate-hang canary and `timeout-minutes` on every CI job; the npm shim/postinstall harnesses now assert empty-output emptiness instead of skipping it; and `daily`, `monthly`, and `report` gain golden test harnesses (#279).
+
+### Fixed
+- `cctally setup` now installs the `cctally-budget` symlink (it existed but was never linked); a drift-guard test keeps the wrapper list and the installer in sync.
+- `CCTALLY_ALLOW_PROD_MIGRATION=0`, `CCTALLY_DISABLE_DEV_AUTODETECT=0`, `CCTALLY_DEBUG=0`, and `CCTALLY_DISABLE_UPDATE_CHECK=0` now mean "disabled" — previously any value, including `0`, enabled these flags.
+- A corrupt `stats.db` now produces a one-line diagnosis with recovery guidance and exit 2 instead of a raw traceback.
+- Piping output to a closed reader (`cctally daily | head`) now exits 0 quietly instead of `Error: [Errno 32] Broken pipe`.
+- The dashboard share endpoints now cap request bodies at 64 KiB and the dashboard server sets a 60s socket timeout (slow-loris hardening); server-sent-event streams treat a send timeout as a normal client disconnect.
+- The dashboard `/api/data` endpoint now returns a JSON `500 {"error": "internal error"}` on an internal error instead of dropping the connection with no response, and a server-sent-event stream that fails mid-flight is now logged and closed cleanly instead of dying silently (#279).
+- `cache.db` now opens with `PRAGMA synchronous=NORMAL` (a fully re-derivable database; fewer fsyncs during ingest).
+- Codex resume tracking now persists the ingest iterator's actual dedup watermark (the cumulative token counter it compares against) rather than a reconstructed sum of per-turn deltas, closing a latent double-count/skip on rollouts whose cumulative and per-turn token accounting diverge; healthy sessions are unaffected and need no re-ingest (#279).
+- The direct-JSONL fallback (used when `cache.db` can't be opened) now parses with the same single implementation as the cache path so the two can't drift, and a malformed `costUSD` value in a session line no longer aborts the read — it degrades to the token-derived cost like every other cost path (#279).
+
+## [1.65.0] - 2026-07-09
+
+### Added
+- Opt-in backend performance instrumentation: set `CCTALLY_PERF_TRACE=1` to print a phase-timing trace to stderr on `cache-sync` (and the hidden `tui --render-once` build), and read live backend timings + cache state from the dashboard's loopback-only `/api/debug/backend` endpoint. The trace is off by default and never changes command output — it goes to stderr only, so stdout stays byte-identical. New [docs/backend-performance.md](docs/backend-performance.md) documents the read model, hot paths, and invariants.
+- Reproducible backend benchmark suite (`bin/cctally-bench` + `bin/build-bench-fixtures.py`, a dev/maintainer tool — never shipped): in-process timings for the dashboard snapshot (cold/warm/idle), cache ingest (no-op/delta), the conversation rail/search/find/payload/outline, and the warm reconcile paths, measured against a deterministic synthetic fixture, with a committed advisory baseline and an opt-in `--compare`/`--gate` (never a CI gate) that guard the recent backend performance wins from silent regression. See [bench/README.md](bench/README.md).
+- The opt-in `CCTALLY_PERF_TRACE` backend trace now breaks whole-session conversation assembly into named sub-phases (read, dedup, build, correlate, fold, classify, cost, finalize) under the reader's detail/paginate phases, and the loopback `/api/debug/backend` diagnostic can now surface a conversation-open trace (list/search/outline/find/export/prompts/detail) as well as the data-snapshot build — so a slow transcript open is attributable to a specific stage on a live dashboard; off by default, no command output changes.
+- `cctally-bench --assembly-scan` (dev/maintainer tool): sweeps the cost of assembling a whole conversation across a size ladder against its own isolated fixture and baseline (`bench/baselines/assembly.json`), the measurement behind the recorded decision to defer materializing pre-rendered conversation turns — assembly stays well under a 100 ms visibility budget for all but pathologically large (multi-thousand-turn) sessions, and the reader already caps payload bytes. See [docs/backend-performance.md](docs/backend-performance.md) §5.
+- The conversation browse rail and cross-session search can now be filtered by **model family**: a new "Model" section in the filter popover lists the Opus / Sonnet / Haiku / Fable families that actually appear (each with a session count), and selecting one or more scopes both browsing and search to sessions that used that model — anywhere in the session, including its subagents — with removable chips and "Clear all" just like the other filters (#278).
+
+### Changed
+- Conversation reader no longer re-processes an open conversation on every background refresh — it now recomputes only when the conversation actually grows, cutting idle CPU/battery use for anyone leaving a transcript open (#278).
+- The dashboard now opens instantly on a heavy-history instance: it binds and paints the current-week and forecast panels almost immediately instead of waiting for the full ~2s data aggregation, and the heavier panels (sessions, projects, weekly, monthly, blocks, daily, trend, cache report) hydrate over the live stream showing a brief loading skeleton (respecting reduced-motion) rather than flashing a misleading "no data" / "restart the dashboard" empty state that then fills in (#278).
+- A first-run or long-gap dashboard now fills in progressively as session history is ingested, instead of sitting blank and then snapping to fully-loaded in one jump; a warm returning session is unaffected, and a slow or backgrounded browser tab always converges to the latest data instead of getting stuck on a stale frame (#278).
+
+### Fixed
+- The conversation filter popover now retries once if its filter options (the Project and Model lists) fail to load on a transient hiccup, instead of coming up empty and staying that way until it is reopened (#278).
+
 ## [1.64.0] - 2026-07-07
 
 ### Added

package/bin/_cctally_alerts.py

@@ -44,7 +44,6 @@ from __future__ import annotations
 
 import argparse
 import datetime as dt
-import importlib.util as _ilu
 import os
 import pathlib
 import shutil
@@ -58,6 +57,7 @@ def _load_lib(name: str):
     cached = sys.modules.get(name)
     if cached is not None:
         return cached
+    import importlib.util as _ilu
     p = pathlib.Path(__file__).resolve().parent / f"{name}.py"
     spec = _ilu.spec_from_file_location(name, p)
     mod = _ilu.module_from_spec(spec)

package/bin/_cctally_cache.py

@@ -100,7 +100,6 @@ from __future__ import annotations
 import argparse
 import datetime as dt
 import fcntl
-import importlib.util as _ilu
 import json
 import os
 import pathlib
@@ -151,6 +150,7 @@ def _load_lib(name: str):
     cached = sys.modules.get(name)
     if cached is not None:
         return cached
+    import importlib.util as _ilu
     p = pathlib.Path(__file__).resolve().parent / f"{name}.py"
     spec = _ilu.spec_from_file_location(name, p)
     mod = _ilu.module_from_spec(spec)
@@ -176,6 +176,12 @@ _should_replace = _lib_jsonl._should_replace
 _lib_conversation = _load_lib("_lib_conversation")
 _iter_message_rows = _lib_conversation.iter_message_rows
 
+# Opt-in backend phase-instrumentation collector (issue #276, Session A). Pure
+# stdlib leaf; near-noop when CCTALLY_PERF_TRACE is unset (phase() returns a
+# shared no-op singleton), so the sync_cache seam wraps below cost nothing on
+# the default path.
+_perf = _load_lib("_lib_perf")
+
 # Shared by the fused per-file walk AND backfill_conversation_messages so the
 # column list, placeholders, and tuple order live in ONE place — a column
 # add/reorder can't silently desync the two ingest paths (which would land
@@ -223,7 +229,7 @@ def _conv_row_tuple(m, path_str):
     )
 
 
-def _iter_sync_entries(fh, path_str):
+def _iter_sync_entries(fh, path_str, stats: "IngestStats | None" = None):
     """Fused single-pass sync walker (#138). Yields
     ``(byte_offset, cost_or_None, msgrow_or_None, aititle_or_None)`` for each
     JSONL line from ``fh``'s current position that produces a cost entry, a
@@ -263,11 +269,25 @@ def _iter_sync_entries(fh, path_str):
         stripped = line.strip()
         if not stripped:
             continue
+        # #279 S2 F1: passive parse-health counters over the new-byte span.
+        # lines_seen counts non-blank lines (malformed included).
+        if stats is not None:
+            stats.lines_seen += 1
         try:
             obj = json.loads(stripped)
         except json.JSONDecodeError:
+            if stats is not None:
+                stats.lines_malformed += 1
             continue
         cost = _lib_jsonl.parse_cost_entry(obj, path_str)
+        if cost is None and stats is not None:
+            # Assistant-typed line rejected for a NON-deliberate reason
+            # (schema-drift tripwire; <synthetic>/non-assistant are normal).
+            reason = _lib_jsonl.assistant_skip_reason(obj)
+            if reason is not None:
+                stats.assistant_lines_skipped += 1
+                stats.skip_reasons[reason] = \
+                    stats.skip_reasons.get(reason, 0) + 1
         mrow = _lib_conversation.parse_message_row(obj, offset)
         ai = _lib_conversation.parse_ai_title(obj, offset)
         if cost is not None or mrow is not None or ai is not None:
@@ -296,6 +316,77 @@ clear_conversation_messages = _cctally_db_sib.clear_conversation_messages
 _set_cache_meta = _cctally_db_sib._set_cache_meta
 
 
+_PARSE_HEALTH_SCHEMA = 1
+
+
+def _update_parse_health_meta(
+    conn: sqlite3.Connection,
+    key: str,
+    *,
+    lines_seen: int,
+    lines_malformed: int,
+    lines_skipped: int,
+    skip_reasons: dict,
+    rebuild: bool,
+) -> None:
+    """Anomaly-delta-gated rolling parse-health record (#279 S2 F1 /
+    Codex P1-2). Writes ONLY when (a) this sync's malformed+skipped delta
+    is nonzero, (b) rebuild=True (baseline reset — write fresh from that
+    walk's counters), or (c) the key is absent (first adoption). Healthy
+    steady-state syncs — including the ~1s live-tail targeted ingests —
+    never write, so the cumulative ``lines_seen`` denominator advances
+    only at these writes ("as of the last write"); doctor reasons from
+    counts + recency, never a precise ratio.
+
+    Caller holds the sync flock. Runs at end-of-sync, OUTSIDE every
+    per-file ``[before, after]`` total_changes window, so
+    ``stats.rows_changed`` stays byte-identical (#270); never bumps
+    ``mutation_seq``. Commits its own write (mirrors the walk-complete
+    sentinel's commit discipline). Fail-soft: a corrupt prior value is
+    treated as absent.
+    """
+    anomaly_delta = int(lines_malformed) + int(lines_skipped)
+    now_iso = dt.datetime.now(dt.timezone.utc).isoformat()
+    prior = None
+    try:
+        row = conn.execute(
+            "SELECT value FROM cache_meta WHERE key = ?", (key,)
+        ).fetchone()
+        if row and row[0]:
+            loaded = json.loads(row[0])
+            if isinstance(loaded, dict):
+                prior = loaded
+    except (sqlite3.DatabaseError, ValueError):
+        prior = None
+    if not rebuild and prior is not None and anomaly_delta == 0:
+        return  # steady state: zero-write
+    if rebuild or prior is None:
+        prior = {"lines_seen": 0, "lines_malformed": 0, "lines_skipped": 0,
+                 "reasons": {}, "last_anomaly_at": None, "since": now_iso}
+    record = {
+        "schema": _PARSE_HEALTH_SCHEMA,
+        "lines_seen": int(prior.get("lines_seen", 0) or 0) + int(lines_seen),
+        "lines_malformed": (int(prior.get("lines_malformed", 0) or 0)
+                            + int(lines_malformed)),
+        "lines_skipped": (int(prior.get("lines_skipped", 0) or 0)
+                          + int(lines_skipped)),
+        "last_anomaly_at": (now_iso if anomaly_delta > 0
+                            else prior.get("last_anomaly_at")),
+        "last_write_at": now_iso,
+        "since": prior.get("since") or now_iso,
+    }
+    reasons = dict(prior.get("reasons") or {}) \
+        if isinstance(prior.get("reasons"), dict) else {}
+    for r, n in (skip_reasons or {}).items():
+        reasons[r] = int(reasons.get(r, 0) or 0) + int(n)
+    record["reasons"] = reasons
+    try:
+        _set_cache_meta(conn, key, json.dumps(record, sort_keys=True))
+        conn.commit()
+    except sqlite3.DatabaseError:
+        conn.rollback()  # observability must never fail the sync
+
+
 # === BEGIN MOVED REGIONS ===
 # Path constants APP_DIR / CACHE_DB_PATH / CACHE_LOCK_PATH /
 # CACHE_LOCK_CODEX_PATH live in _cctally_core (promoted 2026-05-22, #84);
@@ -491,6 +582,16 @@ class IngestStats:
     # and are otherwise unaffected.
     files_failed: int = 0
     deferred_reason: "str | None" = None
+    # #279 S2 F1 parse-health counters — passive observers over the new-byte
+    # span this sync walked. lines_seen counts non-blank lines (malformed
+    # included); assistant_lines_skipped counts assistant-typed lines
+    # parse_cost_entry rejected for a NON-deliberate reason (schema-drift
+    # tripwire; `<synthetic>` and non-assistant lines are normal). Reason
+    # vocabulary in _lib_jsonl._classify_cost_entry.
+    lines_seen: int = 0
+    lines_malformed: int = 0
+    assistant_lines_skipped: int = 0
+    skip_reasons: dict = field(default_factory=dict)
 
     @property
     def targeted_clean(self) -> bool:
@@ -857,7 +958,10 @@ def sync_cache(
 
     lock_fh = open(_cctally_core.CACHE_LOCK_PATH, "w")
     try:
-        if not _acquire_cache_flock(lock_fh, timeout=lock_timeout):
+        with _perf.phase("flock") as _p_flock:
+            _acquired = _acquire_cache_flock(lock_fh, timeout=lock_timeout)
+            _p_flock.set_meta(contended=not _acquired)
+        if not _acquired:
             eprint("[cache] sync already in progress; using existing cache")
             stats.lock_contended = True
             return stats
@@ -1044,6 +1148,12 @@ def sync_cache(
         # path, which already cleared the flag and repopulates via the normal
         # walk). A path-less/:memory: conn has no cache_meta only if the schema
         # was never applied; the try/except tolerates that.
+        # #276 perf: bracket the (rare, upgrade-only) backfill/reingest region
+        # as one coarse "backfills" phase. Opened via the context-manager
+        # protocol rather than a ``with`` block so the ~150-line body below is
+        # not reindented. Near-noop when tracing is off (_NULL_PHASE).
+        _p_backfills = _perf.phase("backfills")
+        _p_backfills.__enter__()
         if not rebuild and not targeted:
             try:
                 _pending = conn.execute(
@@ -1191,12 +1301,22 @@ def sync_cache(
             # IGNORE). Touches ONLY conversation_file_touches, never
             # conversation_messages (P1-2).
             _consume_file_touches(conn)
+        _p_backfills.__exit__(None, None, None)
 
-        if targeted:
-            paths = [pathlib.Path(p) for p in only_paths if pathlib.Path(p).is_file()]
-        else:
-            paths = list(_iter_claude_jsonl_files())
-        stats.files_total = len(paths)
+        with _perf.phase("discover") as _p_disc:
+            if targeted:
+                # A requested path that vanished (session rotated/deleted
+                # mid-live-tail) is deliberately DROPPED here without flagging
+                # failure: marking files_failed would wedge the watch loop's
+                # targeted_clean advance forever for a file that will never
+                # return; the orphan-prune path owns its stale rows on the next
+                # full sync. Pinned by tests/test_cache_accepted_behaviors.py
+                # (#279 S3 F4).
+                paths = [pathlib.Path(p) for p in only_paths if pathlib.Path(p).is_file()]
+            else:
+                paths = list(_iter_claude_jsonl_files())
+            stats.files_total = len(paths)
+            _p_disc.set_count(len(paths))
 
         # This SELECT does NOT open an implicit transaction (Python's
         # sqlite3 module only BEGINs on DML). Do NOT add any INSERT/
@@ -1406,6 +1526,12 @@ def sync_cache(
         # zero-write-lock read/parse region, so it adds no DML there.
         touched_sessions: set = set()
 
+        # #276 perf: bracket the fused per-file ingest loop as ONE coarse
+        # "walk" phase (never per-row — Section 2 rule: volume is a count, not
+        # N timed phases). Opened via the context-manager protocol so the hot
+        # loop body below is not reindented; counts recorded after the loop.
+        _p_walk = _perf.phase("walk")
+        _p_walk.__enter__()
         for jp in paths:
             path_str = str(jp)
             # Backfill session_id/project_path for A2 `session` subcommand.
@@ -1460,7 +1586,7 @@ def sync_cache(
                     # walk over the identical span — the "identical span"
                     # invariant is now structural (a single stop point) rather
                     # than a prose-enforced ``>= final_offset`` runtime break.
-                    for offset, cost, mrow, ai in _iter_sync_entries(fh, path_str):
+                    for offset, cost, mrow, ai in _iter_sync_entries(fh, path_str, stats=stats):
                         if cost is not None:
                             entry, msg_id, req_id = cost
                             usage = entry.usage
@@ -1683,6 +1809,13 @@ def sync_cache(
 
         if progress is not None:
             progress(stats)
+        _p_walk.__exit__(None, None, None)
+        _p_walk.set_count(stats.files_processed)
+        _p_walk.set_meta(
+            skipped=stats.files_skipped_unchanged,
+            failed=stats.files_failed,
+            rows=stats.rows_changed,
+        )
 
         # Browse-rail rollup maintenance (single post-walk recompute, under the
         # still-held flock, after every per-file commit and before the
@@ -1698,16 +1831,17 @@ def sync_cache(
         # ~1 session/tick). Both recomputes derive COUNT/MIN/MAX from the same
         # rows the rail's old live aggregate read, so the rollup stays
         # byte-identical to that aggregate.
-        if _conversation_sessions_backfill_pending(conn):
-            _recompute_conversation_sessions(conn)
-            conn.execute(
-                "DELETE FROM cache_meta "
-                "WHERE key='conversation_sessions_backfill_pending'"
-            )
-            conn.commit()
-        elif touched_sessions:
-            _recompute_conversation_sessions(conn, touched_sessions)
-            conn.commit()
+        with _perf.phase("recompute.conversation_sessions"):
+            if _conversation_sessions_backfill_pending(conn):
+                _recompute_conversation_sessions(conn)
+                conn.execute(
+                    "DELETE FROM cache_meta "
+                    "WHERE key='conversation_sessions_backfill_pending'"
+                )
+                conn.commit()
+            elif touched_sessions:
+                _recompute_conversation_sessions(conn, touched_sessions)
+                conn.commit()
 
         # Walk-complete sentinel write (cctally-dev#93, D5a). Still inside the
         # held fcntl lock, before the finally-unlock. Only when the entire walk
@@ -1724,6 +1858,17 @@ def sync_cache(
                 (dt.datetime.now(dt.timezone.utc).isoformat(),),
             )
             conn.commit()
+        # #279 S2 F1: rolling parse-health record. Anomaly-delta-gated so
+        # steady-state (incl. targeted live-tail) syncs stay zero-write;
+        # targeted syncs still accumulate — they ingest real new bytes.
+        _update_parse_health_meta(
+            conn, "parse_health_claude",
+            lines_seen=stats.lines_seen,
+            lines_malformed=stats.lines_malformed,
+            lines_skipped=stats.assistant_lines_skipped,
+            skip_reasons=stats.skip_reasons,
+            rebuild=rebuild,
+        )
         # At-rest hardening (Plan 2, spec §5). Runs here — at the end of the
         # write transaction, while the cache.db.lock flock is still held (so a
         # concurrent writer can't be mid-checkpoint) AND after at least one
@@ -2794,6 +2939,13 @@ class CodexIngestStats:
     # $CODEX_HOME root set (issue #108 — a prior-root purge, not a delta).
     files_pruned: int = 0
     lock_contended: bool = False
+    # #279 S2 F1 parse-health counters — folded from each file's
+    # _CodexIterState after its drain. Same vocabulary as the iterator
+    # (info-non-dict / no-last-token-usage / bad-timestamp / no-session-id).
+    lines_seen: int = 0
+    lines_malformed: int = 0
+    token_events_skipped: int = 0
+    skip_reasons: dict = field(default_factory=dict)
 
 
 def _progress_codex_stderr(stats: CodexIngestStats, *, force: bool = False) -> None:
@@ -2831,7 +2983,10 @@ def sync_codex_cache(
 
     lock_fh = open(_cctally_core.CACHE_LOCK_CODEX_PATH, "w")
     try:
-        if not _acquire_cache_flock(lock_fh, timeout=lock_timeout):
+        with _perf.phase("flock") as _p_flock:
+            _acquired = _acquire_cache_flock(lock_fh, timeout=lock_timeout)
+            _p_flock.set_meta(contended=not _acquired)
+        if not _acquired:
             eprint("[codex-cache] sync already in progress; using existing cache")
             stats.lock_contended = True
             return stats
@@ -2848,8 +3003,10 @@ def sync_codex_cache(
         roots = _cctally()._codex_session_roots()
         # Pure read (glob + is_file only); safe to run before the SELECT and
         # the per-file loop, where no cache.db write lock may be held.
-        paths: list[pathlib.Path] = list(_iter_codex_jsonl_paths(roots))
-        stats.files_total = len(paths)
+        with _perf.phase("discover") as _p_disc:
+            paths: list[pathlib.Path] = list(_iter_codex_jsonl_paths(roots))
+            stats.files_total = len(paths)
+            _p_disc.set_count(len(paths))
 
         # Scope the cache to the CURRENT root set: drop rows ingested under a
         # prior $CODEX_HOME (issue #108). iter_codex_entries() has NO root
@@ -2910,6 +3067,11 @@ def sync_codex_cache(
             )
         }
 
+        # #279 S2 F4: ONE coarse `walk` phase bracketing the per-file loop
+        # (count = files_processed, never per-row — §2 rule). Manual CM so
+        # the loop stays flat, mirroring sync_cache's walk seam.
+        _p_walk = _perf.phase("walk")
+        _p_walk.__enter__()
         for jp in paths:
             path_str = str(jp)
             try:
@@ -2958,15 +3120,19 @@ def sync_codex_cache(
             # ends on a metadata-only tail would lose the terminal
             # session_id/model and the next resume would mis-attribute the
             # first post-resume token_count.
+            # #279 S3 F1: the state is BOTH the seed carrier for the dedup
+            # watermark and the sink the iterator stamps it into. Seeding it
+            # with initial_total_tokens (the prior resume's persisted
+            # cumulative) means iter_state.total_tokens holds the guard's
+            # terminal watermark once the iterator drains — which we persist
+            # directly, replacing the former initial+Σ(per-turn) reconstruction
+            # that could diverge from the true cumulative and double-count/skip
+            # on the next resume.
             iter_state = _CodexIterState(
                 session_id=initial_session_id,
                 model=initial_model,
+                total_tokens=initial_total_tokens,
             )
-            # Track the cumulative `total_token_usage.total_tokens` across this
-            # call. The iterator only yields when the cumulative strictly
-            # advances by the current turn's `last_token_usage.total_tokens`,
-            # so summing the per-turn totals reconstructs the final cumulative.
-            running_total = initial_total_tokens
             yielded_count = 0
             try:
                 with open(jp, "r", encoding="utf-8", errors="replace") as fh:
@@ -2991,9 +3157,18 @@ def sync_codex_cache(
                             entry.reasoning_output_tokens,
                             entry.total_tokens,
                         ))
-                        running_total += int(entry.total_tokens or 0)
                         yielded_count += 1
                     final_offset = fh.tell()
+                    # #279 S2 F1: fold this file's iterator counters into the
+                    # sync-level stats. iter_state is per-file, so += is
+                    # exact. Folded inside the try: an OSError mid-read drops
+                    # this file's partial counters AND its offset advance
+                    # together, so re-walked lines are never double-counted.
+                    stats.lines_seen += iter_state.lines_seen
+                    stats.lines_malformed += iter_state.lines_malformed
+                    stats.token_events_skipped += iter_state.token_events_skipped
+                    for _r, _n in iter_state.skip_reasons.items():
+                        stats.skip_reasons[_r] = stats.skip_reasons.get(_r, 0) + _n
             except OSError as exc:
                 eprint(f"[codex-cache] could not read {jp}: {exc}")
                 continue
@@ -3014,11 +3189,13 @@ def sync_codex_cache(
                 else initial_model
             )
 
-            # Persist the running cumulative if we yielded this call. Otherwise
-            # preserve the prior value — never overwrite with 0, which would
-            # re-enable double-counting on the next resume.
+            # Persist the iterator's stamped cumulative watermark if we yielded
+            # this call (iter_state.total_tokens == the dedup guard's terminal
+            # value by construction, #279 S3 F1). Otherwise preserve the prior
+            # value — never overwrite with 0, which would re-enable
+            # double-counting on the next resume.
             new_last_total_tokens: int | None = (
-                running_total if yielded_count > 0 else prev_total_tokens
+                iter_state.total_tokens if yielded_count > 0 else prev_total_tokens
             )
 
             # Python's sqlite3 module starts an implicit transaction on the
@@ -3070,6 +3247,21 @@ def sync_codex_cache(
 
         if progress is not None:
             progress(stats)
+        _p_walk.__exit__(None, None, None)
+        _p_walk.set_count(stats.files_processed)
+        _p_walk.set_meta(skipped=stats.files_skipped_unchanged,
+                         rows=stats.rows_changed)
+        # #279 S2 F1: rolling parse-health record (codex half). Same
+        # anomaly-delta gate as the Claude tail; SQLite serializes the
+        # cache.db write against a concurrent Claude sync.
+        _update_parse_health_meta(
+            conn, "parse_health_codex",
+            lines_seen=stats.lines_seen,
+            lines_malformed=stats.lines_malformed,
+            lines_skipped=stats.token_events_skipped,
+            skip_reasons=stats.skip_reasons,
+            rebuild=rebuild,
+        )
         return stats
     finally:
         try:
@@ -3322,6 +3514,11 @@ def open_cache_db() -> sqlite3.Connection:
 
     conn.execute("PRAGMA journal_mode=WAL")
     conn.execute("PRAGMA busy_timeout=5000")
+    # Re-derivable DB under WAL: NORMAL risks at most the tail transaction on
+    # power loss, and cache.db can always be rebuilt (cache-sync --rebuild).
+    # Fewer ingest fsyncs than the default FULL. Matches stats.db
+    # (bin/_cctally_core.py open_db). #279 S1 F8.
+    conn.execute("PRAGMA synchronous=NORMAL")
 
     # Apply the shared cache.db schema (cctally-dev#93, D4): Claude tables +
     # indexes, the session_id / project_path column adds on session_files
@@ -3370,6 +3567,10 @@ def cmd_cache_sync(args: argparse.Namespace) -> int:
     default is 'all'.
     """
     source = getattr(args, "source", "all")
+    # #276 perf: clear any prior tree on this thread so a leaked root can't be
+    # flushed, then (below) time the Claude sync_cache call as the "sync_cache"
+    # root phase and flush the tree to stderr when CCTALLY_PERF_TRACE is set.
+    _perf.reset_thread()
     conn = open_cache_db()
 
     # --prune-orphans: fast, targeted cleanup of cache rows whose source
@@ -3418,10 +3619,18 @@ def cmd_cache_sync(args: argparse.Namespace) -> int:
     lt = _REBUILD_LOCK_TIMEOUT_SECONDS if args.rebuild else None
     contended = False
 
+    # #279 S2 F4: one shared `cache-sync` root so a single flushed tree
+    # carries BOTH vendors — with two sequential sync roots,
+    # flush_stderr(current_root()) would show only the last one. Opened
+    # after the --prune-orphans early returns so they can't leak a root.
+    _p_root = _perf.phase("cache-sync")
+    _p_root.__enter__()
+
     if source in ("claude", "all"):
-        stats = sync_cache(
-            conn, progress=_progress_stderr, rebuild=args.rebuild, lock_timeout=lt
-        )
+        with _perf.phase("sync_cache"):
+            stats = sync_cache(
+                conn, progress=_progress_stderr, rebuild=args.rebuild, lock_timeout=lt
+            )
         _progress_stderr(stats, force=True)
         if stats.lock_contended and args.rebuild:
             eprint(
@@ -3434,13 +3643,17 @@ def cmd_cache_sync(args: argparse.Namespace) -> int:
                 f"[cache-sync] claude done: {stats.files_processed} processed, "
                 f"{stats.files_skipped_unchanged} skipped, "
                 f"{stats.files_reset_truncated} reset, "
-                f"{stats.rows_changed} rows changed"
+                f"{stats.rows_changed} rows changed, "
+                f"{stats.lines_malformed} malformed, "
+                f"{stats.assistant_lines_skipped} drift-skipped"
             )
 
     if source in ("codex", "all"):
-        stats = sync_codex_cache(
-            conn, progress=_progress_codex_stderr, rebuild=args.rebuild, lock_timeout=lt
-        )
+        with _perf.phase("sync_codex_cache"):
+            stats = sync_codex_cache(
+                conn, progress=_progress_codex_stderr, rebuild=args.rebuild,
+                lock_timeout=lt,
+            )
         _progress_codex_stderr(stats, force=True)
         if stats.lock_contended and args.rebuild:
             eprint(
@@ -3453,7 +3666,16 @@ def cmd_cache_sync(args: argparse.Namespace) -> int:
                 f"[cache-sync] codex done: {stats.files_processed} processed, "
                 f"{stats.files_skipped_unchanged} skipped, "
                 f"{stats.files_reset_truncated} reset, "
-                f"{stats.rows_changed} rows changed"
+                f"{stats.rows_changed} rows changed, "
+                f"{stats.lines_malformed} malformed, "
+                f"{stats.token_events_skipped} drift-skipped"
             )
 
+    _p_root.__exit__(None, None, None)
+    # #276 perf: when tracing is enabled, flush the completed "cache-sync"
+    # phase tree to stderr (stdout stays byte-identical). No-op when off.
+    # As of #279 S2 F4 the root carries both the sync_cache and
+    # sync_codex_cache children, so one flushed tree covers both vendors.
+    if _perf.enabled():
+        _perf.flush_stderr(_perf.current_root())
     return 1 if contended else 0

package/bin/_cctally_cache_report.py

@@ -1010,7 +1010,7 @@ def _emit_cache_report_json(
         for row_dict in output[top_key]:
             row_dict["totalCost"] = row_dict["cost"]
         output["totals"]["totalCost"] = output["totals"]["cost"]
-    return json.dumps(output, indent=2)
+    return json.dumps(_cctally().stamp_schema_version(output), indent=2)
 
 
 def _build_cache_report_title(args: argparse.Namespace, mode: str) -> str:
@@ -1141,8 +1141,9 @@ def cmd_cache_report(args: argparse.Namespace) -> int:
     if since == until:
         if args.json:
             print(json.dumps(
-                {top_key: [], "totals": None,
-                 "generatedAt": now_utc_iso(now_utc=now_utc)},
+                c.stamp_schema_version(
+                    {top_key: [], "totals": None,
+                     "generatedAt": now_utc_iso(now_utc=now_utc)}),
                 indent=2,
             ))
         else:
@@ -1163,8 +1164,9 @@ def cmd_cache_report(args: argparse.Namespace) -> int:
     if not rows:
         if args.json:
             print(json.dumps(
-                {top_key: [], "totals": None,
-                 "generatedAt": now_utc_iso(now_utc=now_utc)},
+                c.stamp_schema_version(
+                    {top_key: [], "totals": None,
+                     "generatedAt": now_utc_iso(now_utc=now_utc)}),
                 indent=2,
             ))
         else: