cctally 1.63.0 → 1.65.0

package/CHANGELOG.md

@@ -5,6 +5,28 @@ based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+## [1.65.0] - 2026-07-09
+
+### Added
+- Opt-in backend performance instrumentation: set `CCTALLY_PERF_TRACE=1` to print a phase-timing trace to stderr on `cache-sync` (and the hidden `tui --render-once` build), and read live backend timings + cache state from the dashboard's loopback-only `/api/debug/backend` endpoint. The trace is off by default and never changes command output — it goes to stderr only, so stdout stays byte-identical. New [docs/backend-performance.md](docs/backend-performance.md) documents the read model, hot paths, and invariants.
+- Reproducible backend benchmark suite (`bin/cctally-bench` + `bin/build-bench-fixtures.py`, a dev/maintainer tool — never shipped): in-process timings for the dashboard snapshot (cold/warm/idle), cache ingest (no-op/delta), the conversation rail/search/find/payload/outline, and the warm reconcile paths, measured against a deterministic synthetic fixture, with a committed advisory baseline and an opt-in `--compare`/`--gate` (never a CI gate) that guard the recent backend performance wins from silent regression. See [bench/README.md](bench/README.md).
+- The opt-in `CCTALLY_PERF_TRACE` backend trace now breaks whole-session conversation assembly into named sub-phases (read, dedup, build, correlate, fold, classify, cost, finalize) under the reader's detail/paginate phases, and the loopback `/api/debug/backend` diagnostic can now surface a conversation-open trace (list/search/outline/find/export/prompts/detail) as well as the data-snapshot build — so a slow transcript open is attributable to a specific stage on a live dashboard; off by default, no command output changes.
+- `cctally-bench --assembly-scan` (dev/maintainer tool): sweeps the cost of assembling a whole conversation across a size ladder against its own isolated fixture and baseline (`bench/baselines/assembly.json`), the measurement behind the recorded decision to defer materializing pre-rendered conversation turns — assembly stays well under a 100 ms visibility budget for all but pathologically large (multi-thousand-turn) sessions, and the reader already caps payload bytes. See [docs/backend-performance.md](docs/backend-performance.md) §5.
+- The conversation browse rail and cross-session search can now be filtered by **model family**: a new "Model" section in the filter popover lists the Opus / Sonnet / Haiku / Fable families that actually appear (each with a session count), and selecting one or more scopes both browsing and search to sessions that used that model — anywhere in the session, including its subagents — with removable chips and "Clear all" just like the other filters (#278).
+
+### Changed
+- Conversation reader no longer re-processes an open conversation on every background refresh — it now recomputes only when the conversation actually grows, cutting idle CPU/battery use for anyone leaving a transcript open (#278).
+- The dashboard now opens instantly on a heavy-history instance: it binds and paints the current-week and forecast panels almost immediately instead of waiting for the full ~2s data aggregation, and the heavier panels (sessions, projects, weekly, monthly, blocks, daily, trend, cache report) hydrate over the live stream showing a brief loading skeleton (respecting reduced-motion) rather than flashing a misleading "no data" / "restart the dashboard" empty state that then fills in (#278).
+- A first-run or long-gap dashboard now fills in progressively as session history is ingested, instead of sitting blank and then snapping to fully-loaded in one jump; a warm returning session is unaffected, and a slow or backgrounded browser tab always converges to the latest data instead of getting stuck on a stale frame (#278).
+
+### Fixed
+- The conversation filter popover now retries once if its filter options (the Project and Model lists) fail to load on a transient hiccup, instead of coming up empty and staying that way until it is reopened (#278).
+
+## [1.64.0] - 2026-07-07
+
+### Added
+- cctally now sends an anonymous, opt-out install-count beat at most once a day so the project can gauge how many people actually use it. It is deliberately minimal and private: the only thing transmitted is a one-way token that rotates every month (derived from a random local id that never leaves your machine and cannot be recovered from the token), plus the cctally version and a coarse operating-system family (macos/linux/windows/other) — no IP, username, file paths, project names, or usage data — and nothing leaves your machine for at least 24 hours after first run, so you always have a window to opt out first. Run `cctally telemetry` to see the current state, why it is on or off, and exactly what would be sent; `cctally telemetry off` disables it (`cctally telemetry on` re-enables), `cctally telemetry reset` rotates your local id, and `cctally doctor` shows a read-only telemetry line. It is also disabled by `cctally config set telemetry.enabled false`, `CCTALLY_DISABLE_TELEMETRY=1`, the `DO_NOT_TRACK` convention, and automatically in development checkouts. Full transparency page — token construction, retention, and an honest threat model — is in [docs/telemetry.md](docs/telemetry.md).
+
 ## [1.63.0] - 2026-07-07
 
 ### Added

package/README.md

@@ -154,6 +154,12 @@ If you also use OpenAI's Codex CLI, `cctally codex daily / monthly / session` ar
 
 `cctally doctor` gives you a read-only health check - install, hooks, sign-in, database, data freshness, pricing, and safety - and the same status shows up in the dashboard. `cctally pricing-check` warns you when the built-in model pricing is getting stale, `cctally db` manages the local database, and `cctally update` keeps your install current. The local cache is always safe to delete or rebuild.
 
+## Privacy & telemetry
+
+cctally sends an anonymous, opt-out **install-count beat** — at most once a day — so the project can gauge how many people actually use it (npm and GitHub numbers are drowned in bots and mirrors). The entire payload is a one-way token that rotates every month (derived from a random local id that never leaves your machine and can't be recovered from the token), the cctally version, and a coarse OS family (`macos`/`linux`/`windows`/`other`). No identity, file paths, prompts, usage data, or stored IP — ever. Nothing leaves your machine for at least 24 hours after first run, so you always have a window to opt out first.
+
+Turn it off any time with `cctally telemetry off` (or `cctally config set telemetry.enabled false`, `CCTALLY_DISABLE_TELEMETRY=1`, or the community-standard `DO_NOT_TRACK=1`); it's also off automatically in dev checkouts. Run `cctally telemetry` to see the current state and exactly what would be sent. The full transparency page — token construction, retention, and an honest threat model — is [docs/telemetry.md](docs/telemetry.md).
+
 ## A faster, local-first ccusage
 
 `cctally` started as a local replacement for [`ccusage`](https://github.com/ryoppippi/ccusage) and stays compatible at the level of common flows: `cctally claude <cmd>` is a drop-in for `ccusage claude <cmd>` (and `cctally codex <cmd>` for `ccusage codex <cmd>`), with the flat forms (`cctally daily`, `cctally codex-daily`, …) kept as aliases. Paste your ccusage commands verbatim - then reach for the dashboard, forecast, trend, conversation viewer, and alerts that ccusage doesn't have.
@@ -176,6 +182,7 @@ It's also fast. Pricing is built in and computed in-process from a local SQLite
 - [Configuration](docs/configuration.md): `config.json` shape and week-start rules.
 - [Architecture](docs/architecture.md): data flow, caches, week boundaries.
 - [Runtime data](docs/runtime-data.md): what lives in `~/.local/share/cctally/`.
+- [Telemetry](docs/telemetry.md): the anonymous install-count beat, in full — and how to opt out.
 - [Command reference](docs/commands/): one page per subcommand.
 
 ## License

package/bin/_cctally_cache.py

@@ -176,6 +176,12 @@ _should_replace = _lib_jsonl._should_replace
 _lib_conversation = _load_lib("_lib_conversation")
 _iter_message_rows = _lib_conversation.iter_message_rows
 
+# Opt-in backend phase-instrumentation collector (issue #276, Session A). Pure
+# stdlib leaf; near-noop when CCTALLY_PERF_TRACE is unset (phase() returns a
+# shared no-op singleton), so the sync_cache seam wraps below cost nothing on
+# the default path.
+_perf = _load_lib("_lib_perf")
+
 # Shared by the fused per-file walk AND backfill_conversation_messages so the
 # column list, placeholders, and tuple order live in ONE place — a column
 # add/reorder can't silently desync the two ingest paths (which would land
@@ -857,7 +863,10 @@ def sync_cache(
 
     lock_fh = open(_cctally_core.CACHE_LOCK_PATH, "w")
     try:
-        if not _acquire_cache_flock(lock_fh, timeout=lock_timeout):
+        with _perf.phase("flock") as _p_flock:
+            _acquired = _acquire_cache_flock(lock_fh, timeout=lock_timeout)
+            _p_flock.set_meta(contended=not _acquired)
+        if not _acquired:
             eprint("[cache] sync already in progress; using existing cache")
             stats.lock_contended = True
             return stats
@@ -1044,6 +1053,12 @@ def sync_cache(
         # path, which already cleared the flag and repopulates via the normal
         # walk). A path-less/:memory: conn has no cache_meta only if the schema
         # was never applied; the try/except tolerates that.
+        # #276 perf: bracket the (rare, upgrade-only) backfill/reingest region
+        # as one coarse "backfills" phase. Opened via the context-manager
+        # protocol rather than a ``with`` block so the ~150-line body below is
+        # not reindented. Near-noop when tracing is off (_NULL_PHASE).
+        _p_backfills = _perf.phase("backfills")
+        _p_backfills.__enter__()
         if not rebuild and not targeted:
             try:
                 _pending = conn.execute(
@@ -1191,12 +1206,15 @@ def sync_cache(
             # IGNORE). Touches ONLY conversation_file_touches, never
             # conversation_messages (P1-2).
             _consume_file_touches(conn)
+        _p_backfills.__exit__(None, None, None)
 
-        if targeted:
-            paths = [pathlib.Path(p) for p in only_paths if pathlib.Path(p).is_file()]
-        else:
-            paths = list(_iter_claude_jsonl_files())
-        stats.files_total = len(paths)
+        with _perf.phase("discover") as _p_disc:
+            if targeted:
+                paths = [pathlib.Path(p) for p in only_paths if pathlib.Path(p).is_file()]
+            else:
+                paths = list(_iter_claude_jsonl_files())
+            stats.files_total = len(paths)
+            _p_disc.set_count(len(paths))
 
         # This SELECT does NOT open an implicit transaction (Python's
         # sqlite3 module only BEGINs on DML). Do NOT add any INSERT/
@@ -1406,6 +1424,12 @@ def sync_cache(
         # zero-write-lock read/parse region, so it adds no DML there.
         touched_sessions: set = set()
 
+        # #276 perf: bracket the fused per-file ingest loop as ONE coarse
+        # "walk" phase (never per-row — Section 2 rule: volume is a count, not
+        # N timed phases). Opened via the context-manager protocol so the hot
+        # loop body below is not reindented; counts recorded after the loop.
+        _p_walk = _perf.phase("walk")
+        _p_walk.__enter__()
         for jp in paths:
             path_str = str(jp)
             # Backfill session_id/project_path for A2 `session` subcommand.
@@ -1683,6 +1707,13 @@ def sync_cache(
 
         if progress is not None:
             progress(stats)
+        _p_walk.__exit__(None, None, None)
+        _p_walk.set_count(stats.files_processed)
+        _p_walk.set_meta(
+            skipped=stats.files_skipped_unchanged,
+            failed=stats.files_failed,
+            rows=stats.rows_changed,
+        )
 
         # Browse-rail rollup maintenance (single post-walk recompute, under the
         # still-held flock, after every per-file commit and before the
@@ -1698,16 +1729,17 @@ def sync_cache(
         # ~1 session/tick). Both recomputes derive COUNT/MIN/MAX from the same
         # rows the rail's old live aggregate read, so the rollup stays
         # byte-identical to that aggregate.
-        if _conversation_sessions_backfill_pending(conn):
-            _recompute_conversation_sessions(conn)
-            conn.execute(
-                "DELETE FROM cache_meta "
-                "WHERE key='conversation_sessions_backfill_pending'"
-            )
-            conn.commit()
-        elif touched_sessions:
-            _recompute_conversation_sessions(conn, touched_sessions)
-            conn.commit()
+        with _perf.phase("recompute.conversation_sessions"):
+            if _conversation_sessions_backfill_pending(conn):
+                _recompute_conversation_sessions(conn)
+                conn.execute(
+                    "DELETE FROM cache_meta "
+                    "WHERE key='conversation_sessions_backfill_pending'"
+                )
+                conn.commit()
+            elif touched_sessions:
+                _recompute_conversation_sessions(conn, touched_sessions)
+                conn.commit()
 
         # Walk-complete sentinel write (cctally-dev#93, D5a). Still inside the
         # held fcntl lock, before the finally-unlock. Only when the entire walk
@@ -3370,6 +3402,10 @@ def cmd_cache_sync(args: argparse.Namespace) -> int:
     default is 'all'.
     """
     source = getattr(args, "source", "all")
+    # #276 perf: clear any prior tree on this thread so a leaked root can't be
+    # flushed, then (below) time the Claude sync_cache call as the "sync_cache"
+    # root phase and flush the tree to stderr when CCTALLY_PERF_TRACE is set.
+    _perf.reset_thread()
     conn = open_cache_db()
 
     # --prune-orphans: fast, targeted cleanup of cache rows whose source
@@ -3419,9 +3455,10 @@ def cmd_cache_sync(args: argparse.Namespace) -> int:
     contended = False
 
     if source in ("claude", "all"):
-        stats = sync_cache(
-            conn, progress=_progress_stderr, rebuild=args.rebuild, lock_timeout=lt
-        )
+        with _perf.phase("sync_cache"):
+            stats = sync_cache(
+                conn, progress=_progress_stderr, rebuild=args.rebuild, lock_timeout=lt
+            )
         _progress_stderr(stats, force=True)
         if stats.lock_contended and args.rebuild:
             eprint(
@@ -3456,4 +3493,8 @@ def cmd_cache_sync(args: argparse.Namespace) -> int:
                 f"{stats.rows_changed} rows changed"
             )
 
+    # #276 perf: when tracing is enabled, flush the completed "sync_cache"
+    # phase tree to stderr (stdout stays byte-identical). No-op when off.
+    if _perf.enabled():
+        _perf.flush_stderr(_perf.current_root())
     return 1 if contended else 0

package/bin/_cctally_config.py

@@ -318,6 +318,7 @@ ALLOWED_CONFIG_KEYS = (
     "budget.projects",
     "budget.project_alerts_enabled",
     "budget.codex",
+    "telemetry.enabled",
 )
 
 
@@ -540,6 +541,25 @@ def _config_known_value(config: dict, key: str) -> "object":
             except ValueError:
                 return True
         return True
+    if key == "telemetry.enabled":
+        # Boolean opt-OUT (anonymous install-count telemetry, spec 2026-07-07).
+        # Default TRUE — absence is ON. A hand-edited junk value surfaces the
+        # True default. Mirrors dashboard.live_tail exactly, under a top-level
+        # `telemetry` block instead of `dashboard`.
+        block = config.get("telemetry") if isinstance(config, dict) else None
+        if not isinstance(block, dict):
+            block = {}
+        stored = block.get("enabled")
+        if stored is None:
+            return True
+        if isinstance(stored, bool):
+            return stored
+        if isinstance(stored, str):
+            try:
+                return c._normalize_alerts_enabled_value(stored)
+            except ValueError:
+                return True
+        return True
     if key in ("update.check.enabled", "update.check.ttl_hours"):
         # Defaults mirror `_is_update_check_due` (True / 24 hours).
         # Hand-edited junk surfaces as the default — matches dashboard.bind.
@@ -1006,6 +1026,40 @@ def _cmd_config_set(args: argparse.Namespace) -> int:
         else:
             print(f"dashboard.live_tail={'true' if canonical else 'false'}")
         return 0
+    if key == "telemetry.enabled":
+        # Anonymous install-count telemetry opt-out (spec 2026-07-07). Mirror
+        # dashboard.live_tail exactly: validate the bool first, then
+        # read-modify-write under config_writer_lock with _load_config_unlocked
+        # (load_config under the lock self-deadlocks). Preserves any sibling
+        # telemetry.* keys. Re-message the shared normalizer's ValueError with
+        # the actual key name.
+        try:
+            canonical = c._normalize_alerts_enabled_value(raw)
+        except ValueError:
+            print(
+                f"cctally: invalid boolean value for telemetry.enabled: "
+                f"{raw!r} (expected true|false|yes|no|1|0|on|off)",
+                file=sys.stderr,
+            )
+            return 2
+        with config_writer_lock():
+            config = _load_config_unlocked()
+            existing = config.get("telemetry")
+            if existing is not None and not isinstance(existing, dict):
+                print(
+                    "cctally: telemetry config error: telemetry must be an object",
+                    file=sys.stderr,
+                )
+                return 2
+            block = dict(existing or {})
+            block["enabled"] = canonical
+            config["telemetry"] = block
+            save_config(config)
+        if getattr(args, "emit_json", False):
+            print(json.dumps({"telemetry": {"enabled": canonical}}, indent=2))
+        else:
+            print(f"telemetry.enabled={'true' if canonical else 'false'}")
+        return 0
     if key in (
         "statusline.visual_burn_rate",
         "statusline.cost_source",
@@ -1411,6 +1465,20 @@ def _cmd_config_unset(args: argparse.Namespace) -> int:
                 save_config(config)
             # idempotent: silent on missing key
         return 0
+    if key == "telemetry.enabled":
+        # Mirror the dashboard.live_tail unset branch: drop only the enabled
+        # leaf; if the telemetry block ends up empty, drop the parent too.
+        # Unsetting restores the True (opt-out) default at read time.
+        with config_writer_lock():
+            config = _load_config_unlocked()
+            block = config.get("telemetry")
+            if isinstance(block, dict) and "enabled" in block:
+                del block["enabled"]
+                if not block:
+                    config.pop("telemetry", None)
+                save_config(config)
+            # idempotent: silent on missing key
+        return 0
     if key in (
         "statusline.visual_burn_rate",
         "statusline.cost_source",

package/bin/_cctally_core.py

@@ -69,6 +69,8 @@ def _init_paths_from_env() -> None:
     global UPDATE_LOCK_PATH, UPDATE_LOG_PATH, UPDATE_LOG_ROTATED_PATH
     global UPDATE_CHECK_LAST_FETCH_PATH, CLAUDE_SETTINGS_PATH
     global CLAUDE_PROJECTS_DIR
+    global TELEMETRY_INSTALL_ID_PATH, TELEMETRY_LAST_BEAT_PATH
+    global TELEMETRY_NOTICE_SHOWN_PATH, TELEMETRY_FIRST_SEEN_PATH
 
     home = pathlib.Path.home()
 
@@ -124,6 +126,14 @@ def _init_paths_from_env() -> None:
     UPDATE_LOG_ROTATED_PATH = APP_DIR / "update.log.1"
     UPDATE_CHECK_LAST_FETCH_PATH = APP_DIR / "update-check.last-fetch"
 
+    # Anonymous install-count telemetry markers (see spec 2026-07-07).
+    # All four derive from APP_DIR and are re-bound here so a redirected
+    # APP_DIR (tests, dev-instance isolation) carries them along.
+    TELEMETRY_INSTALL_ID_PATH = APP_DIR / "install_id"
+    TELEMETRY_LAST_BEAT_PATH = APP_DIR / "telemetry.last-beat"
+    TELEMETRY_NOTICE_SHOWN_PATH = APP_DIR / "telemetry.notice-shown"
+    TELEMETRY_FIRST_SEEN_PATH = APP_DIR / "telemetry.first-seen"
+
     CLAUDE_SETTINGS_PATH = home / ".claude" / "settings.json"
 
     # Claude session JSONL root. Production path is `~/.claude/projects`;
@@ -189,6 +199,25 @@ def _real_prod_data_dir() -> pathlib.Path:
 _init_paths_from_env()
 
 
+# === Telemetry constants (non-path; see spec 2026-07-07) =============
+#
+# These are static (not APP_DIR-derived) so they live outside
+# `_init_paths_from_env()`. The kernel `bin/_cctally_telemetry.py` reads
+# them at call time via its `_core()` accessor.
+#
+# Public, non-secret domain-separation constant folded into the monthly
+# rotating token (SHA-256, truncated to 32 hex). It only namespaces
+# cctally's token from any other consumer of the same install_id — it is
+# NOT a secret and leaking it discloses nothing about the install.
+TELEMETRY_PEPPER = "cctally-install-count-v1"
+# Default beat endpoint; overridable for tests via CCTALLY_TELEMETRY_ENDPOINT.
+TELEMETRY_ENDPOINT_DEFAULT = "https://cctally-telemetry.cctally.workers.dev/beat"
+# Send at most one beat per this many seconds (mtime-gated on the beat marker).
+TELEMETRY_BEAT_THROTTLE_SECONDS = 24 * 3600
+# Wait this long after first eligibility before sending the first beat.
+TELEMETRY_FIRST_BEAT_GRACE_SECONDS = 24 * 3600
+
+
 def _resolve_claude_projects_dirs() -> list[pathlib.Path]:
     """Return Claude Code projects dirs that exist on disk, env-aware.