cctally 1.64.0 → 1.65.0

package/CHANGELOG.md

@@ -5,6 +5,23 @@ based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+## [1.65.0] - 2026-07-09
+
+### Added
+- Opt-in backend performance instrumentation: set `CCTALLY_PERF_TRACE=1` to print a phase-timing trace to stderr on `cache-sync` (and the hidden `tui --render-once` build), and read live backend timings + cache state from the dashboard's loopback-only `/api/debug/backend` endpoint. The trace is off by default and never changes command output — it goes to stderr only, so stdout stays byte-identical. New [docs/backend-performance.md](docs/backend-performance.md) documents the read model, hot paths, and invariants.
+- Reproducible backend benchmark suite (`bin/cctally-bench` + `bin/build-bench-fixtures.py`, a dev/maintainer tool — never shipped): in-process timings for the dashboard snapshot (cold/warm/idle), cache ingest (no-op/delta), the conversation rail/search/find/payload/outline, and the warm reconcile paths, measured against a deterministic synthetic fixture, with a committed advisory baseline and an opt-in `--compare`/`--gate` (never a CI gate) that guard the recent backend performance wins from silent regression. See [bench/README.md](bench/README.md).
+- The opt-in `CCTALLY_PERF_TRACE` backend trace now breaks whole-session conversation assembly into named sub-phases (read, dedup, build, correlate, fold, classify, cost, finalize) under the reader's detail/paginate phases, and the loopback `/api/debug/backend` diagnostic can now surface a conversation-open trace (list/search/outline/find/export/prompts/detail) as well as the data-snapshot build — so a slow transcript open is attributable to a specific stage on a live dashboard; off by default, no command output changes.
+- `cctally-bench --assembly-scan` (dev/maintainer tool): sweeps the cost of assembling a whole conversation across a size ladder against its own isolated fixture and baseline (`bench/baselines/assembly.json`), the measurement behind the recorded decision to defer materializing pre-rendered conversation turns — assembly stays well under a 100 ms visibility budget for all but pathologically large (multi-thousand-turn) sessions, and the reader already caps payload bytes. See [docs/backend-performance.md](docs/backend-performance.md) §5.
+- The conversation browse rail and cross-session search can now be filtered by **model family**: a new "Model" section in the filter popover lists the Opus / Sonnet / Haiku / Fable families that actually appear (each with a session count), and selecting one or more scopes both browsing and search to sessions that used that model — anywhere in the session, including its subagents — with removable chips and "Clear all" just like the other filters (#278).
+
+### Changed
+- Conversation reader no longer re-processes an open conversation on every background refresh — it now recomputes only when the conversation actually grows, cutting idle CPU/battery use for anyone leaving a transcript open (#278).
+- The dashboard now opens instantly on a heavy-history instance: it binds and paints the current-week and forecast panels almost immediately instead of waiting for the full ~2s data aggregation, and the heavier panels (sessions, projects, weekly, monthly, blocks, daily, trend, cache report) hydrate over the live stream showing a brief loading skeleton (respecting reduced-motion) rather than flashing a misleading "no data" / "restart the dashboard" empty state that then fills in (#278).
+- A first-run or long-gap dashboard now fills in progressively as session history is ingested, instead of sitting blank and then snapping to fully-loaded in one jump; a warm returning session is unaffected, and a slow or backgrounded browser tab always converges to the latest data instead of getting stuck on a stale frame (#278).
+
+### Fixed
+- The conversation filter popover now retries once if its filter options (the Project and Model lists) fail to load on a transient hiccup, instead of coming up empty and staying that way until it is reopened (#278).
+
 ## [1.64.0] - 2026-07-07
 
 ### Added

package/bin/_cctally_cache.py

@@ -176,6 +176,12 @@ _should_replace = _lib_jsonl._should_replace
 _lib_conversation = _load_lib("_lib_conversation")
 _iter_message_rows = _lib_conversation.iter_message_rows
 
+# Opt-in backend phase-instrumentation collector (issue #276, Session A). Pure
+# stdlib leaf; near-noop when CCTALLY_PERF_TRACE is unset (phase() returns a
+# shared no-op singleton), so the sync_cache seam wraps below cost nothing on
+# the default path.
+_perf = _load_lib("_lib_perf")
+
 # Shared by the fused per-file walk AND backfill_conversation_messages so the
 # column list, placeholders, and tuple order live in ONE place — a column
 # add/reorder can't silently desync the two ingest paths (which would land
@@ -857,7 +863,10 @@ def sync_cache(
 
     lock_fh = open(_cctally_core.CACHE_LOCK_PATH, "w")
     try:
-        if not _acquire_cache_flock(lock_fh, timeout=lock_timeout):
+        with _perf.phase("flock") as _p_flock:
+            _acquired = _acquire_cache_flock(lock_fh, timeout=lock_timeout)
+            _p_flock.set_meta(contended=not _acquired)
+        if not _acquired:
             eprint("[cache] sync already in progress; using existing cache")
             stats.lock_contended = True
             return stats
@@ -1044,6 +1053,12 @@ def sync_cache(
         # path, which already cleared the flag and repopulates via the normal
         # walk). A path-less/:memory: conn has no cache_meta only if the schema
         # was never applied; the try/except tolerates that.
+        # #276 perf: bracket the (rare, upgrade-only) backfill/reingest region
+        # as one coarse "backfills" phase. Opened via the context-manager
+        # protocol rather than a ``with`` block so the ~150-line body below is
+        # not reindented. Near-noop when tracing is off (_NULL_PHASE).
+        _p_backfills = _perf.phase("backfills")
+        _p_backfills.__enter__()
         if not rebuild and not targeted:
             try:
                 _pending = conn.execute(
@@ -1191,12 +1206,15 @@ def sync_cache(
             # IGNORE). Touches ONLY conversation_file_touches, never
             # conversation_messages (P1-2).
             _consume_file_touches(conn)
+        _p_backfills.__exit__(None, None, None)
 
-        if targeted:
-            paths = [pathlib.Path(p) for p in only_paths if pathlib.Path(p).is_file()]
-        else:
-            paths = list(_iter_claude_jsonl_files())
-        stats.files_total = len(paths)
+        with _perf.phase("discover") as _p_disc:
+            if targeted:
+                paths = [pathlib.Path(p) for p in only_paths if pathlib.Path(p).is_file()]
+            else:
+                paths = list(_iter_claude_jsonl_files())
+            stats.files_total = len(paths)
+            _p_disc.set_count(len(paths))
 
         # This SELECT does NOT open an implicit transaction (Python's
         # sqlite3 module only BEGINs on DML). Do NOT add any INSERT/
@@ -1406,6 +1424,12 @@ def sync_cache(
         # zero-write-lock read/parse region, so it adds no DML there.
         touched_sessions: set = set()
 
+        # #276 perf: bracket the fused per-file ingest loop as ONE coarse
+        # "walk" phase (never per-row — Section 2 rule: volume is a count, not
+        # N timed phases). Opened via the context-manager protocol so the hot
+        # loop body below is not reindented; counts recorded after the loop.
+        _p_walk = _perf.phase("walk")
+        _p_walk.__enter__()
         for jp in paths:
             path_str = str(jp)
             # Backfill session_id/project_path for A2 `session` subcommand.
@@ -1683,6 +1707,13 @@ def sync_cache(
 
         if progress is not None:
             progress(stats)
+        _p_walk.__exit__(None, None, None)
+        _p_walk.set_count(stats.files_processed)
+        _p_walk.set_meta(
+            skipped=stats.files_skipped_unchanged,
+            failed=stats.files_failed,
+            rows=stats.rows_changed,
+        )
 
         # Browse-rail rollup maintenance (single post-walk recompute, under the
         # still-held flock, after every per-file commit and before the
@@ -1698,16 +1729,17 @@ def sync_cache(
         # ~1 session/tick). Both recomputes derive COUNT/MIN/MAX from the same
         # rows the rail's old live aggregate read, so the rollup stays
         # byte-identical to that aggregate.
-        if _conversation_sessions_backfill_pending(conn):
-            _recompute_conversation_sessions(conn)
-            conn.execute(
-                "DELETE FROM cache_meta "
-                "WHERE key='conversation_sessions_backfill_pending'"
-            )
-            conn.commit()
-        elif touched_sessions:
-            _recompute_conversation_sessions(conn, touched_sessions)
-            conn.commit()
+        with _perf.phase("recompute.conversation_sessions"):
+            if _conversation_sessions_backfill_pending(conn):
+                _recompute_conversation_sessions(conn)
+                conn.execute(
+                    "DELETE FROM cache_meta "
+                    "WHERE key='conversation_sessions_backfill_pending'"
+                )
+                conn.commit()
+            elif touched_sessions:
+                _recompute_conversation_sessions(conn, touched_sessions)
+                conn.commit()
 
         # Walk-complete sentinel write (cctally-dev#93, D5a). Still inside the
         # held fcntl lock, before the finally-unlock. Only when the entire walk
@@ -3370,6 +3402,10 @@ def cmd_cache_sync(args: argparse.Namespace) -> int:
     default is 'all'.
     """
     source = getattr(args, "source", "all")
+    # #276 perf: clear any prior tree on this thread so a leaked root can't be
+    # flushed, then (below) time the Claude sync_cache call as the "sync_cache"
+    # root phase and flush the tree to stderr when CCTALLY_PERF_TRACE is set.
+    _perf.reset_thread()
     conn = open_cache_db()
 
     # --prune-orphans: fast, targeted cleanup of cache rows whose source
@@ -3419,9 +3455,10 @@ def cmd_cache_sync(args: argparse.Namespace) -> int:
     contended = False
 
     if source in ("claude", "all"):
-        stats = sync_cache(
-            conn, progress=_progress_stderr, rebuild=args.rebuild, lock_timeout=lt
-        )
+        with _perf.phase("sync_cache"):
+            stats = sync_cache(
+                conn, progress=_progress_stderr, rebuild=args.rebuild, lock_timeout=lt
+            )
         _progress_stderr(stats, force=True)
         if stats.lock_contended and args.rebuild:
             eprint(
@@ -3456,4 +3493,8 @@ def cmd_cache_sync(args: argparse.Namespace) -> int:
                 f"{stats.rows_changed} rows changed"
             )
 
+    # #276 perf: when tracing is enabled, flush the completed "sync_cache"
+    # phase tree to stderr (stdout stays byte-identical). No-op when off.
+    if _perf.enabled():
+        _perf.flush_stderr(_perf.current_root())
     return 1 if contended else 0