cctally 1.69.3 → 1.71.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -5,6 +5,29 @@ based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
5
5
 
6
6
  ## [Unreleased]
7
7
 
8
+ ## [1.71.0] - 2026-07-17
9
+
10
+ ### Added
11
+ - Transcript retention: the dashboard's background sync thread now runs a throttled (at most once every 24 hours) prune of conversation transcripts older than the new `conversation.retention_days` config key (default 180; set to `off`/`0` to keep transcripts forever), bounding `cache.db`'s otherwise-unbounded growth. Only transcript rows are pruned — cost/usage history and Codex analytics metadata are untouched and everything pruned is re-derivable from the JSONL. Eligibility is decided per conversation from the authoritative message rows (a conversation with any recent activity is kept whole), never a rollup. (#313)
12
+ - `cctally cache-sync --prune-conversations` prunes >retention-day transcripts on demand and reports the rows removed per provider. (#313)
13
+ - `cctally db vacuum [--db {cache,stats,all}]` reclaims the disk space a prune freed (SQLite `VACUUM`); it is never automatic, runs under an exclusive lock so a running dashboard makes it fail promptly rather than race, and refuses when free disk is below ~2x the file size. (#313)
14
+
15
+ ### Fixed
16
+ - Dashboard CPU peg: `cctally dashboard` no longer pegs a CPU core under sustained active use on large caches. The Codex quota-projection reconcile — which ran unconditionally on every dashboard tick and every `codex-*` command, re-loading the entire quota history (~2.9s) — now short-circuits to an O(1) no-op when the Codex physical state is unchanged and the stored projection is intact; `codex-daily` with no fresh Codex data drops from ~3.2s to well under a second. The dashboard sync thread also caps its CPU duty at 50% of one core via a work-proportional cooldown, so a slow rebuild can no longer keep the thread busy back-to-back. (#313)
17
+ - Dashboard: the #294 S5 source selector no longer causes the mobile topbar to scroll sideways on the narrowest phones. The active-source status chip sits in the non-shrinkable action cluster, and adding it pushed that cluster past a 320px-wide viewport; it now hides at ≤360px (freshness is still on the global sync chip and a degraded source stays flagged in the selector and panels), and both the source selector and its status chip drop out of the condensed one-row header once you scroll, matching how the workspace switcher and doctor chip already collapse there. (#294)
18
+
19
+ ## [1.70.0] - 2026-07-17
20
+
21
+ ### Added
22
+ - `cctally setup` now adds `statusLine.refreshInterval: 30` to a cctally-pointing Claude Code `statusLine` block that lacks one, so status-line-fed usage keeps recording on a 30-second timer while a coordinator session waits on a long-running subagent (Claude Code's event-driven status-line updates go quiet then). Ownership is add-when-absent / never-mutate / never-remove: setup never creates a `statusLine` block, never changes a `refreshInterval` you set yourself, and `--uninstall` leaves it. Surfaced in install / `--status` / `--dry-run` (text plus the `--json` `statusline_refresh {state, value, action}` object). (#311)
23
+ - `cctally doctor` gains a `hooks.statusline_refresh_interval` check that WARNs when a recognized cctally `statusLine` command is missing its `refreshInterval`, and is OK (with a per-state summary) otherwise. (#311)
24
+
25
+ ### Changed
26
+ - Retuned the status-line usage-persist throttle from 60s to 25s so it sits below the new 30s `statusLine.refreshInterval` timer; a 60/60 pairing beat-frequency-throttled every other tick and oscillated the effective cadence between 60 and 120 seconds. (#311)
27
+
28
+ ### Fixed
29
+ - Status-line usage persistence now skips sessions running a bracketed model variant such as `claude-opus-4-8[1m]` (the 1M-context variant), whose `rate_limits` describe a separate account usage pool. Persisting those poisoned the default-pool snapshots — the reset-aware high-water-mark clamp latched the foreign value and froze all subsequent genuine writes. The guard is persist-only, so a `[1m]` session still renders its own true pool numbers on the status line. (#311)
30
+
8
31
  ## [1.69.3] - 2026-07-17
9
32
 
10
33
  ### Fixed
@@ -23,6 +46,7 @@ based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
23
46
  ## [1.69.0] - 2026-07-17
24
47
 
25
48
  ### Added
49
+ - Dashboard source selector and source-aware UX (#294 S5): a Header Claude/Codex/All radiogroup (cycle with `v`) drives every panel, modal, alert filter, and share default at once, rendering each provider's native vocabulary (Codex token counters and quota windows, no `$ / 1%` or subscription-week copy) and hiding — never zero-filling — panels a source doesn't publish, with a Help-overlay note pointing at where the equivalent lives (Codex forecast/trend → the quota panel, cache-report → the hero counters). Alerts and toasts read the per-source projections (a Codex budget alert can't double-toast), the alert settings regroup into global-notifier / Claude / Codex controls with a CLI pointer for Codex quota rules, a per-source status chip surfaces freshness and degraded warnings, and every share render, composer section, preset, and history row carries and displays the source it was captured under — a mid-flow selector switch never restamps an open share flow, and `all` composes provider-labelled sections rather than a blended snapshot. The Claude-only user's data and flows are unchanged except that new share artifacts and history now say "Claude".
26
50
  - Dashboard source-aware backend read model (#294 S4): `/api/data` now appends
27
51
  immutable Claude, Codex, and labelled all-source states; provider-owned
28
52
  session/project/quota details use opaque qualified routes backed by bounded
@@ -725,6 +725,13 @@ def _clear_codex_derived_rows(conn: sqlite3.Connection) -> None:
725
725
  conn.execute("DELETE FROM codex_conversation_events")
726
726
  conn.execute("DELETE FROM codex_session_files")
727
727
  conn.execute("DELETE FROM codex_source_roots")
728
+ # F3: this clears the Codex physical quota state, so any stored
729
+ # quota-projection certificate would become stale-valid (its cache
730
+ # sequence unchanged) and let the reconcile short-circuit skip over
731
+ # now-deleted data. Invalidate it in the same transaction.
732
+ conn.execute(
733
+ "DELETE FROM cache_meta WHERE key='codex_quota_projection_certificate'"
734
+ )
728
735
 
729
736
 
730
737
  def _bump_codex_physical_mutation_seq(conn: sqlite3.Connection) -> None:
@@ -1388,6 +1395,30 @@ def _bump_mutation_seq(conn: sqlite3.Connection) -> int:
1388
1395
  return int(row[0])
1389
1396
 
1390
1397
 
1398
+ def _force_retention_prune_after_replay(conn: sqlite3.Connection) -> None:
1399
+ """#313 P3 (F9): run an UNTHROTTLED transcript retention prune after a
1400
+ from-zero replay (a ``--rebuild`` or a truncation/requalification re-ingest,
1401
+ both of which replay from offset 0 and restore >retention-day rows the
1402
+ throttled prune already trimmed). Best-effort — a prune failure must never
1403
+ break a sync. The caller invokes this only after the sync released its
1404
+ provider flock, so the orchestrator can re-acquire it. No-op when retention
1405
+ is disabled (``conversation.retention_days`` 0)."""
1406
+ try:
1407
+ import _lib_conversation_retention as retention
1408
+ from _cctally_config import resolve_retention_days
1409
+ retention_days = resolve_retention_days(_cctally().load_config())
1410
+ if retention_days <= 0:
1411
+ return
1412
+ retention._maybe_prune_conversation_retention(
1413
+ conn,
1414
+ now_utc=dt.datetime.now(dt.timezone.utc),
1415
+ retention_days=retention_days,
1416
+ force=True,
1417
+ )
1418
+ except Exception:
1419
+ pass
1420
+
1421
+
1391
1422
  def sync_cache(
1392
1423
  conn: sqlite3.Connection,
1393
1424
  *,
@@ -2340,13 +2371,20 @@ def sync_cache(
2340
2371
  # and the flock is still held, so the short busy_timeout keeps it from
2341
2372
  # stalling the lock under heavy-reader contention.
2342
2373
  _maybe_truncate_wal(conn, _cctally_core.CACHE_DB_PATH)
2343
- return stats
2374
+ # #313 P3 (F9): a rebuild or a truncation escalation replays from offset
2375
+ # 0 and restores >retention-day transcript rows. Force an unthrottled
2376
+ # prune AFTER the flock releases below (early lock-contended / deferred
2377
+ # returns above never reach here, so a no-op sync does not prune).
2378
+ did_from_zero_replay = rebuild or stats.files_reset_truncated > 0
2344
2379
  finally:
2345
2380
  try:
2346
2381
  fcntl.flock(lock_fh, fcntl.LOCK_UN)
2347
2382
  except OSError:
2348
2383
  pass
2349
2384
  lock_fh.close()
2385
+ if did_from_zero_replay:
2386
+ _force_retention_prune_after_replay(conn)
2387
+ return stats
2350
2388
 
2351
2389
 
2352
2390
  def backfill_conversation_messages(conn: sqlite3.Connection) -> int:
@@ -3491,6 +3529,17 @@ def sync_codex_cache(
3491
3529
  """
3492
3530
  stats = CodexIngestStats()
3493
3531
  project_after_unlock = False
3532
+ did_from_zero_replay = False
3533
+ # #313 P1 review (F4/F1): when the CACHE certificate is current we cannot
3534
+ # yet decide whether to skip the reconcile — reconcile's own short-circuit
3535
+ # ALSO requires the stats-side quota_projection_state signatures to match
3536
+ # (F1: stats.db can be wiped/recovered while cache.db persists). That
3537
+ # cross-DB read must happen AFTER the Codex flock releases (see the design
3538
+ # comment near the reconcile trigger below), so capture the material for the
3539
+ # deferred stats-side check here. ``None`` means "no deferred check pending"
3540
+ # (the seq-advanced / no-roots / stale-cert branches decide immediately).
3541
+ deferred_cert_roots: "set[str] | None" = None
3542
+ deferred_cert_sigs: "dict[str, str] | None" = None
3494
3543
  c = _cctally()
3495
3544
  _cctally_core.APP_DIR.mkdir(parents=True, exist_ok=True)
3496
3545
  _cctally_core.CACHE_LOCK_CODEX_PATH.touch()
@@ -3505,6 +3554,18 @@ def sync_codex_cache(
3505
3554
  stats.lock_contended = True
3506
3555
  return stats
3507
3556
 
3557
+ # F4 (#313): the reconcile trigger gate is "did the Codex physical
3558
+ # mutation sequence advance during this sync", NOT rows_changed —
3559
+ # rows_changed counts only inserted accounting rows and misses
3560
+ # quota-only / metadata-only / prune-only batches that bump the
3561
+ # sequence. Capture the baseline before any clear/prune/ingest bump.
3562
+ from _cctally_quota import (
3563
+ codex_physical_mutation_seq,
3564
+ load_codex_quota_projection_certificate,
3565
+ _cache_root_keys,
3566
+ )
3567
+ seq_before = codex_physical_mutation_seq(conn)
3568
+
3508
3569
  if rebuild:
3509
3570
  # Clear INSIDE the lock — see sync_cache() for the full
3510
3571
  # rationale. Done before the existing SELECT so delta
@@ -3892,7 +3953,56 @@ def sync_codex_cache(
3892
3953
  # Codex cache flock in ``finally`` below. cache.db and stats.db are not
3893
3954
  # cross-database atomic: after this committed ingest, a projection
3894
3955
  # interruption is repaired by the next full reconciliation.
3895
- project_after_unlock = True
3956
+ #
3957
+ # F4 (#313): reconcile when the physical mutation sequence advanced this
3958
+ # sync (a genuine quota/metadata/prune change — NOT rows_changed, which
3959
+ # misses quota-only batches). A pure no-op with an already-coherent
3960
+ # certificate skips even the O(1) reconcile call.
3961
+ #
3962
+ # A no-op sync must STILL reconcile when there are Codex roots but the
3963
+ # projection certificate is missing/stale at the current sequence — a
3964
+ # lost/failed certificate write (best-effort I/O; can fail under a
3965
+ # cache.db lock storm) leaves the dashboard's Codex source "unavailable"
3966
+ # and is recovered by the next unchanged-file sync re-stamping the
3967
+ # certificate. Claude-only users (no Codex roots) always skip.
3968
+ #
3969
+ # F1 (#313 P1 review): even when the CACHE certificate looks current, the
3970
+ # STATS-side quota_projection_state may have been independently
3971
+ # wiped/recovered (this user has a documented stats.db corruption
3972
+ # history). The cache cert alone does NOT prove stats.db still holds the
3973
+ # projection, so the skip decision on that branch is DEFERRED to the
3974
+ # post-flock stats-side signature check below — making the gate's
3975
+ # skip-condition identical to reconcile's own short-circuit-condition.
3976
+ cur_seq = codex_physical_mutation_seq(conn)
3977
+ if cur_seq != seq_before:
3978
+ project_after_unlock = True
3979
+ else:
3980
+ active_roots = _cache_root_keys(conn)
3981
+ if not active_roots:
3982
+ project_after_unlock = False
3983
+ else:
3984
+ certificate = load_codex_quota_projection_certificate(conn)
3985
+ certificate_current = (
3986
+ certificate is not None
3987
+ and certificate[0] == cur_seq
3988
+ and active_roots <= set(certificate[1])
3989
+ )
3990
+ if certificate_current:
3991
+ # The CACHE certificate is current, but that alone does NOT
3992
+ # prove stats.db still holds the projection (F1). Defer the
3993
+ # stats-side signature match until AFTER the Codex flock
3994
+ # releases below — only then may the reconcile be skipped.
3995
+ project_after_unlock = False
3996
+ deferred_cert_roots = set(active_roots)
3997
+ assert certificate is not None
3998
+ deferred_cert_sigs = dict(certificate[1])
3999
+ else:
4000
+ project_after_unlock = True
4001
+ # #313 P3 (F9): a Codex rebuild or a truncation/requalification
4002
+ # re-ingest replays from offset 0 and restores >retention-day
4003
+ # codex_conversation_events. Force an unthrottled prune after the flock
4004
+ # releases (below), so restored old rows don't persist for up to 24h.
4005
+ did_from_zero_replay = rebuild or stats.files_reset_truncated > 0
3896
4006
  finally:
3897
4007
  try:
3898
4008
  fcntl.flock(lock_fh, fcntl.LOCK_UN)
@@ -3900,9 +4010,29 @@ def sync_codex_cache(
3900
4010
  pass
3901
4011
  lock_fh.close()
3902
4012
 
4013
+ if deferred_cert_roots is not None:
4014
+ # F1: the gate's skip-condition must be IDENTICAL to
4015
+ # reconcile_codex_quota_projection's own short-circuit-condition, which
4016
+ # additionally requires the stats-side quota_projection_state signatures
4017
+ # to match the cache certificate for every active root. Read stats.db
4018
+ # HERE — after the Codex flock released in the ``finally`` above — so the
4019
+ # cross-DB read never widens the Codex flock's critical section (the
4020
+ # design invariant documented near the reconcile trigger). A wiped/stale
4021
+ # stats projection (mismatch) forces the reconcile so it self-heals.
4022
+ from _cctally_quota import _stats_projection_signatures_match
4023
+ stats_conn = _cctally_core.open_db()
4024
+ try:
4025
+ if not _stats_projection_signatures_match(
4026
+ stats_conn, deferred_cert_roots, deferred_cert_sigs or {}
4027
+ ):
4028
+ project_after_unlock = True
4029
+ finally:
4030
+ stats_conn.close()
3903
4031
  if project_after_unlock:
3904
4032
  from _cctally_quota import reconcile_codex_quota_projection
3905
4033
  reconcile_codex_quota_projection()
4034
+ if did_from_zero_replay:
4035
+ _force_retention_prune_after_replay(conn)
3906
4036
  return stats
3907
4037
 
3908
4038
 
@@ -4253,6 +4383,41 @@ def cmd_cache_sync(args: argparse.Namespace) -> int:
4253
4383
  )
4254
4384
  return 0
4255
4385
 
4386
+ # --prune-conversations: on-demand, UNTHROTTLED transcript retention prune
4387
+ # (#313 P3). Removes >retention-day conversation transcripts (re-derivable
4388
+ # from JSONL). Reclaim the freed disk space with `cctally db vacuum`.
4389
+ if getattr(args, "prune_conversations", False):
4390
+ from _cctally_config import resolve_retention_days
4391
+ import _lib_conversation_retention as retention
4392
+ retention_days = resolve_retention_days(_cctally().load_config())
4393
+ if retention_days <= 0:
4394
+ eprint(
4395
+ "[cache-sync] transcript retention is disabled "
4396
+ "(conversation.retention_days=0); nothing pruned."
4397
+ )
4398
+ return 0
4399
+ result = retention._maybe_prune_conversation_retention(
4400
+ conn,
4401
+ now_utc=dt.datetime.now(dt.timezone.utc),
4402
+ retention_days=retention_days,
4403
+ force=True,
4404
+ )
4405
+ if result is None:
4406
+ eprint(
4407
+ "[cache-sync] prune-conversations skipped: another process "
4408
+ "holds the maintenance or a provider lock; retry shortly."
4409
+ )
4410
+ return 1
4411
+ eprint(
4412
+ f"[cache-sync] pruned transcripts older than {retention_days}d: "
4413
+ f"claude {result.claude_sessions} session(s) / "
4414
+ f"{result.claude_messages} message(s), "
4415
+ f"codex {result.codex_conversations} conversation(s) / "
4416
+ f"{result.codex_events} event(s). "
4417
+ f"Run `cctally db vacuum` to reclaim the freed space."
4418
+ )
4419
+ return 0
4420
+
4256
4421
  # Note: when --rebuild is set we delegate the DELETE to sync_cache /
4257
4422
  # sync_codex_cache, which execute it AFTER acquiring the flock. A
4258
4423
  # pre-sync DELETE here would wipe the cache even when the subsequent
@@ -327,11 +327,76 @@ ALLOWED_CONFIG_KEYS = (
327
327
  "budget.codex.alert_thresholds",
328
328
  "budget.codex.projected_enabled",
329
329
  "telemetry.enabled",
330
+ "conversation.retention_days",
330
331
  )
331
332
 
332
333
 
333
334
  _CODEX_BUDGET_LEAF_PREFIX = "budget.codex."
334
335
 
336
+ _DEFAULT_CONVERSATION_RETENTION_DAYS = 180
337
+
338
+
339
+ def _validate_retention_days_value(raw: object) -> int:
340
+ """Validate a ``config set`` value for ``conversation.retention_days`` (F8).
341
+
342
+ Accept a non-negative integer; ``off`` / ``0`` canonicalize to integer 0
343
+ (disabled — keep transcripts forever). Reject booleans, negatives, and
344
+ non-numeric input by raising ``ValueError`` (the caller maps that to exit 2).
345
+ """
346
+ if isinstance(raw, bool):
347
+ raise ValueError(
348
+ "conversation.retention_days must be a non-negative integer or 'off'"
349
+ )
350
+ if isinstance(raw, str):
351
+ text = raw.strip().lower()
352
+ if text == "off":
353
+ return 0
354
+ try:
355
+ value = int(text, 10)
356
+ except (TypeError, ValueError):
357
+ raise ValueError(
358
+ "conversation.retention_days must be a non-negative integer "
359
+ f"or 'off', got {raw!r}"
360
+ )
361
+ elif isinstance(raw, int):
362
+ value = raw
363
+ else:
364
+ raise ValueError(
365
+ "conversation.retention_days must be a non-negative integer or 'off'"
366
+ )
367
+ if value < 0:
368
+ raise ValueError(
369
+ "conversation.retention_days must be >= 0 (use 'off' or 0 to disable)"
370
+ )
371
+ return value
372
+
373
+
374
+ def resolve_retention_days(config: dict) -> int:
375
+ """Effective conversation transcript retention in days (F8).
376
+
377
+ Default is ``180``; ``0`` disables retention (keep forever). Any malformed
378
+ persisted value (non-int, boolean, negative, un-parseable string) degrades
379
+ to the safe 180 default rather than raising.
380
+ """
381
+ default = _DEFAULT_CONVERSATION_RETENTION_DAYS
382
+ block = config.get("conversation") if isinstance(config, dict) else None
383
+ if not isinstance(block, dict):
384
+ return default
385
+ stored = block.get("retention_days")
386
+ if stored is None:
387
+ return default
388
+ if isinstance(stored, bool):
389
+ return default
390
+ if isinstance(stored, int):
391
+ return stored if stored >= 0 else default
392
+ if isinstance(stored, str):
393
+ try:
394
+ value = int(stored.strip(), 10)
395
+ except (TypeError, ValueError):
396
+ return default
397
+ return value if value >= 0 else default
398
+ return default
399
+
335
400
 
336
401
  def _config_codex_leaf_value(config: dict, key: str) -> object:
337
402
  """Read one nested Codex budget leaf with its public default.
@@ -794,6 +859,10 @@ def _config_known_value(config: dict, key: str) -> "object":
794
859
  except ValueError:
795
860
  return True
796
861
  return True
862
+ if key == "conversation.retention_days":
863
+ # Effective transcript-retention window in days (default 180; 0 = keep
864
+ # forever). Malformed persisted data surfaces the safe default (F8).
865
+ return resolve_retention_days(config)
797
866
  if key in ("update.check.enabled", "update.check.ttl_hours"):
798
867
  # Defaults mirror `_is_update_check_due` (True / 24 hours).
799
868
  # Hand-edited junk surfaces as the default — matches dashboard.bind.
@@ -1422,6 +1491,32 @@ def _cmd_config_set(args: argparse.Namespace) -> int:
1422
1491
  rendered = str(normalized)
1423
1492
  print(f"{key}={rendered}")
1424
1493
  return 0
1494
+ if key == "conversation.retention_days":
1495
+ # Validate first; rejection short-circuits before lock acquisition (F8).
1496
+ try:
1497
+ normalized = _validate_retention_days_value(raw)
1498
+ except ValueError as exc:
1499
+ print(f"cctally: {exc}", file=sys.stderr)
1500
+ return 2
1501
+ with config_writer_lock():
1502
+ config = _load_config_unlocked()
1503
+ existing = config.get("conversation")
1504
+ if existing is not None and not isinstance(existing, dict):
1505
+ print(
1506
+ "cctally: conversation config error: conversation must be "
1507
+ "an object",
1508
+ file=sys.stderr,
1509
+ )
1510
+ return 2
1511
+ block = dict(existing or {})
1512
+ block["retention_days"] = normalized
1513
+ config["conversation"] = block
1514
+ save_config(config)
1515
+ if getattr(args, "emit_json", False):
1516
+ print(json.dumps({"conversation": {"retention_days": normalized}}, indent=2))
1517
+ else:
1518
+ print(f"{key}={normalized}")
1519
+ return 0
1425
1520
  if key in ("update.check.enabled", "update.check.ttl_hours"):
1426
1521
  # Validate first; rejection short-circuits before lock acquisition.
1427
1522
  if key == "update.check.enabled":
@@ -1862,6 +1957,19 @@ def _cmd_config_unset(args: argparse.Namespace) -> int:
1862
1957
  save_config(config)
1863
1958
  # idempotent: silent on missing key
1864
1959
  return 0
1960
+ if key == "conversation.retention_days":
1961
+ # Mirror the display.tz branch: drop the leaf; if the `conversation`
1962
+ # block ends up empty, drop it too. Next get resolves the 180 default.
1963
+ with config_writer_lock():
1964
+ config = _load_config_unlocked()
1965
+ block = config.get("conversation")
1966
+ if isinstance(block, dict) and "retention_days" in block:
1967
+ del block["retention_days"]
1968
+ if not block:
1969
+ config.pop("conversation", None)
1970
+ save_config(config)
1971
+ # idempotent: silent on missing key
1972
+ return 0
1865
1973
  if key in ("update.check.enabled", "update.check.ttl_hours"):
1866
1974
  # Mirror the dashboard.bind branch: drop the leaf, then prune
1867
1975
  # empty `check` and empty `update` so config.json stays tidy.
@@ -61,7 +61,8 @@ def _init_paths_from_env() -> None:
61
61
  """
62
62
  global APP_DIR, LEGACY_APP_DIR, LOG_DIR, DEV_MODE
63
63
  global DB_PATH, CACHE_DB_PATH
64
- global CACHE_LOCK_PATH, CACHE_LOCK_CODEX_PATH, CONFIG_LOCK_PATH
64
+ global CACHE_LOCK_PATH, CACHE_LOCK_CODEX_PATH, CACHE_LOCK_MAINTENANCE_PATH
65
+ global CONFIG_LOCK_PATH
65
66
  global CONFIG_PATH, MIGRATION_ERROR_LOG_PATH, CHANGELOG_PATH
66
67
  global HOOK_TICK_LOG_DIR, HOOK_TICK_LOG_PATH, HOOK_TICK_LOG_ROTATED_PATH
67
68
  global HOOK_TICK_THROTTLE_PATH, HOOK_TICK_THROTTLE_LOCK_PATH
@@ -100,6 +101,10 @@ def _init_paths_from_env() -> None:
100
101
 
101
102
  CACHE_LOCK_PATH = APP_DIR / "cache.db.lock"
102
103
  CACHE_LOCK_CODEX_PATH = APP_DIR / "cache.db.codex.lock"
104
+ # #313 P3 (F7): dedicated maintenance flock serializing the transcript
105
+ # retention prune across processes, held ABOVE the two provider flocks so a
106
+ # rebuild/reingest cannot land between candidate selection and deletion.
107
+ CACHE_LOCK_MAINTENANCE_PATH = APP_DIR / "cache.db.maintenance.lock"
103
108
  CONFIG_LOCK_PATH = APP_DIR / "config.json.lock"
104
109
 
105
110
  CONFIG_PATH = APP_DIR / "config.json"
@@ -243,13 +248,37 @@ def _real_prod_data_dir() -> pathlib.Path:
243
248
  #
244
249
  # STATUSLINE_PERSIST_THROTTLE_SECONDS: min seconds between statusline
245
250
  # persist attempts (keyed off STATUSLINE_OBSERVE_MARKER_PATH liveness).
251
+ # Retuned 60.0 -> 25.0 for #311: the setup-managed statusLine.refreshInterval
252
+ # timer fires the statusline on a 30s cadence, and the throttle MUST be
253
+ # strictly LESS than that interval. When interval <= throttle, phase jitter
254
+ # produces beat-frequency skips (a tick at 59.9s of marker age is throttled;
255
+ # the next persist waits ~120s), so a 60/60 pairing oscillates 60<->120s.
256
+ # With interval 30 > throttle 25, every tick whose predecessor's record
257
+ # completed promptly (within interval - throttle = 5s) passes the gate.
258
+ # CADENCE QUALIFIER: the marker is touched at persist COMPLETION, so a tick
259
+ # observes marker age = 30 - d where d is the previous record's duration;
260
+ # if d > 5s (the record kernel can run cmd_sync_week JSONL scans) that tick
261
+ # throttles and the cycle degrades to skip-one-tick (~60s), self-correcting
262
+ # on the next tick. Exact-30s is NOT an invariant (no attempt-start marker
263
+ # exists — one would let a hung/failed record claim liveness and suppress
264
+ # the OAuth backfill). Still strictly better than the 60/60 beat's 60-120s.
246
265
  # OAUTH_BACKFILL_STALE_SECONDS: the OAuth poll only backfills once the
247
266
  # observation marker is at least this stale (i.e. the statusline has
248
267
  # NOT fed recently). Strictly greater than the persist throttle so the
249
- # statusline is the primary writer and OAuth only covers its absence.
268
+ # statusline is the primary writer and OAuth only covers its absence
269
+ # (300 > 25 still holds).
250
270
  # OAUTH_BACKOFF_BASE_SECONDS / OAUTH_BACKOFF_CAP_SECONDS: the headerless
251
271
  # exponential 429 backoff (base * 2**consecutive_429, capped).
252
- STATUSLINE_PERSIST_THROTTLE_SECONDS = 60.0
272
+ STATUSLINE_PERSIST_THROTTLE_SECONDS = 25.0
273
+ # STATUSLINE_REFRESH_INTERVAL_DEFAULT (#311): the value `cctally setup`
274
+ # writes into Claude Code's settings.json `statusLine.refreshInterval` when a
275
+ # recognized cctally statusLine block lacks one. Claude Code re-runs the
276
+ # statusline command on this fixed timer "in addition to the event-driven
277
+ # updates", which keeps the usage-persistence feeder ticking while a parent
278
+ # session waits on a long subagent (event-driven updates go quiet then). MUST
279
+ # exceed STATUSLINE_PERSIST_THROTTLE_SECONDS (30 > 25) — see the pairing rule
280
+ # above. Add-when-absent only; a user-set value is never mutated.
281
+ STATUSLINE_REFRESH_INTERVAL_DEFAULT = 30
253
282
  OAUTH_BACKFILL_STALE_SECONDS = 300.0
254
283
  OAUTH_BACKOFF_BASE_SECONDS = 60.0
255
284
  OAUTH_BACKOFF_CAP_SECONDS = 3600.0
@@ -1068,6 +1068,86 @@ def _load_recorded_five_hour_windows(*args, **kwargs):
1068
1068
  return sys.modules["cctally"]._load_recorded_five_hour_windows(*args, **kwargs)
1069
1069
 
1070
1070
 
1071
+ def _next_deadline(t0: float, interval: float, work: float) -> float:
1072
+ """Monotonic deadline for the next sync iteration (#313 P2 / F10).
1073
+
1074
+ The iteration's work ended at ``t0 + work``; cool down from there for
1075
+ ``max(interval, work)``. So the period is ``work + max(interval, work)``:
1076
+
1077
+ - work >= interval → period = ``2 * work`` → CPU duty capped at 50% of one
1078
+ core, scale-independent (a slow rebuild on a huge cache can never peg a
1079
+ full core — the exact #313 symptom).
1080
+ - work < interval → period = ``work + interval`` — byte-identical to the
1081
+ prior loop (rebuild then a fixed ``interval`` sleep), so normal-cadence
1082
+ behavior and its goldens are unchanged.
1083
+
1084
+ NOTE (spec deviation): the spec §3 pseudocode's ``deadline = t0 + max(
1085
+ interval, work)`` cools down from ``t0`` (before the work), which for
1086
+ work >= interval yields a ZERO cooldown (period = work, 100% duty) — it does
1087
+ not meet the spec's own stated "period = 2*work / duty <= 50%" property or
1088
+ §1's "<= 50% of one core" success criterion, and would leave the peg unfixed
1089
+ (in fact worse than the old work+interval cadence). This formula cools down
1090
+ from the work's END, which is what those properties require.
1091
+ """
1092
+ return (t0 + work) + max(interval, work)
1093
+
1094
+
1095
+ def _dashboard_sync_loop(
1096
+ *,
1097
+ stop,
1098
+ interval: float,
1099
+ run_iteration,
1100
+ take_sync_request,
1101
+ monotonic=time.monotonic,
1102
+ sleep=time.sleep,
1103
+ ) -> None:
1104
+ """Run the dashboard's automatic sync loop with a work-proportional cooldown.
1105
+
1106
+ ``run_iteration`` performs one whole automatic iteration (rebuild plus any
1107
+ orphan self-heal / retention maintenance the thread does), so its measured
1108
+ duration — not just the rebuild — drives the deadline (F10). The manual
1109
+ ``POST /api/sync`` refresh is a separate synchronous path under ``sync_lock``
1110
+ and is unaffected; ``take_sync_request`` is the TUI force-refresh flag that
1111
+ breaks the cooldown early.
1112
+ """
1113
+ while not stop.is_set():
1114
+ t0 = monotonic()
1115
+ run_iteration()
1116
+ work = monotonic() - t0
1117
+ deadline = _next_deadline(t0, interval, work)
1118
+ while not stop.is_set() and monotonic() < deadline:
1119
+ if take_sync_request():
1120
+ break
1121
+ sleep(min(0.1, max(0.0, deadline - monotonic())))
1122
+
1123
+
1124
+ def _dashboard_maybe_prune_retention() -> None:
1125
+ """#313 P3 (F7): throttled transcript retention prune driven from the
1126
+ dashboard sync thread. Runs once per iteration INSIDE the measured cooldown
1127
+ work (Task 6), so its cost paces the deadline. Opens a dedicated cache
1128
+ connection (no provider flock, no open transaction) for the orchestrator.
1129
+ No-op when retention is disabled; never raises (a prune failure must not
1130
+ crash the sync thread)."""
1131
+ try:
1132
+ c = sys.modules["cctally"]
1133
+ from _cctally_config import resolve_retention_days
1134
+ import _lib_conversation_retention as retention
1135
+ retention_days = resolve_retention_days(c.load_config())
1136
+ if retention_days <= 0:
1137
+ return
1138
+ conn = c.open_cache_db()
1139
+ try:
1140
+ retention._maybe_prune_conversation_retention(
1141
+ conn,
1142
+ now_utc=dt.datetime.now(dt.timezone.utc),
1143
+ retention_days=retention_days,
1144
+ )
1145
+ finally:
1146
+ conn.close()
1147
+ except Exception:
1148
+ pass
1149
+
1150
+
1071
1151
  def _make_run_sync_now(*args, **kwargs):
1072
1152
  return sys.modules["cctally"]._make_run_sync_now(*args, **kwargs)
1073
1153
 
@@ -6623,23 +6703,38 @@ def cmd_dashboard(args: argparse.Namespace) -> int:
6623
6703
 
6624
6704
  class _DashboardSyncThread(_c_for_subclass._TuiSyncThread):
6625
6705
  def _run(self) -> None:
6626
- last_heal = _time.monotonic()
6627
- while not self._stop.is_set():
6706
+ last_heal = [_time.monotonic()]
6707
+
6708
+ def run_iteration() -> None:
6628
6709
  _run_sync_now(skip_sync=self._skip_sync)
6629
6710
  # Self-heal removed-worktree orphans on a ~60s cadence (far
6630
6711
  # rarer than the sync tick — a deleted worktree is not urgent).
6631
6712
  # Non-blocking on the flock, so a contended tick just retries
6632
- # next cadence; gated off under --no-sync.
6713
+ # next cadence; gated off under --no-sync. Runs INSIDE the
6714
+ # measured iteration so its cost counts toward the cooldown
6715
+ # deadline (#313 P2 / F10).
6633
6716
  if (not self._skip_sync
6634
- and _time.monotonic() - last_heal >= 60.0):
6635
- last_heal = _time.monotonic()
6717
+ and _time.monotonic() - last_heal[0] >= 60.0):
6718
+ last_heal[0] = _time.monotonic()
6636
6719
  _dashboard_self_heal_orphans(skip_sync=self._skip_sync)
6637
- for _ in range(int(max(1, self._interval * 10))):
6638
- if self._stop.is_set():
6639
- return
6640
- if self._ref.take_sync_request():
6641
- break
6642
- _time.sleep(0.1)
6720
+ # #313 P3 (F7): throttled transcript retention prune, inside the
6721
+ # measured iteration so its cost is in the cooldown work. Gated
6722
+ # off under --no-sync (frozen mode makes no cache writes).
6723
+ if not self._skip_sync:
6724
+ _dashboard_maybe_prune_retention()
6725
+
6726
+ # Work-proportional cooldown (F10): sleep to t0 + max(interval, work)
6727
+ # so a slow rebuild cannot peg a full core. The manual POST /api/sync
6728
+ # refresh runs synchronously under sync_lock, independent of this
6729
+ # cooldown, so a user force-refresh is always immediate.
6730
+ _dashboard_sync_loop(
6731
+ stop=self._stop,
6732
+ interval=self._interval,
6733
+ run_iteration=run_iteration,
6734
+ take_sync_request=self._ref.take_sync_request,
6735
+ monotonic=_time.monotonic,
6736
+ sleep=_time.sleep,
6737
+ )
6643
6738
 
6644
6739
  sync_thread = (
6645
6740
  None if args.no_sync