cctally 1.69.3 → 1.71.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +24 -0
- package/bin/_cctally_cache.py +167 -2
- package/bin/_cctally_config.py +108 -0
- package/bin/_cctally_core.py +32 -3
- package/bin/_cctally_dashboard.py +106 -11
- package/bin/_cctally_db.py +159 -0
- package/bin/_cctally_doctor.py +10 -0
- package/bin/_cctally_parser.py +20 -0
- package/bin/_cctally_quota.py +58 -6
- package/bin/_cctally_setup.py +320 -0
- package/bin/_cctally_statusline.py +11 -0
- package/bin/_lib_doctor.py +48 -0
- package/bin/_lib_statusline.py +26 -0
- package/bin/cctally +10 -0
- package/dashboard/static/assets/index-BWadAHxC.js +80 -0
- package/dashboard/static/assets/index-D2nwo_ln.css +1 -0
- package/dashboard/static/dashboard.html +2 -2
- package/package.json +1 -1
- package/dashboard/static/assets/index-CBbErI-P.js +0 -80
- package/dashboard/static/assets/index-kDDVOLa_.css +0 -1
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,29 @@ based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
|
|
5
5
|
|
|
6
6
|
## [Unreleased]
|
|
7
7
|
|
|
8
|
+
## [1.71.0] - 2026-07-17
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
- Transcript retention: the dashboard's background sync thread now runs a throttled (at most once every 24 hours) prune of conversation transcripts older than the new `conversation.retention_days` config key (default 180; set to `off`/`0` to keep transcripts forever), bounding `cache.db`'s otherwise-unbounded growth. Only transcript rows are pruned — cost/usage history and Codex analytics metadata are untouched and everything pruned is re-derivable from the JSONL. Eligibility is decided per conversation from the authoritative message rows (a conversation with any recent activity is kept whole), never a rollup. (#313)
|
|
12
|
+
- `cctally cache-sync --prune-conversations` prunes >retention-day transcripts on demand and reports the rows removed per provider. (#313)
|
|
13
|
+
- `cctally db vacuum [--db {cache,stats,all}]` reclaims the disk space a prune freed (SQLite `VACUUM`); it is never automatic, runs under an exclusive lock so a running dashboard makes it fail promptly rather than race, and refuses when free disk is below ~2x the file size. (#313)
|
|
14
|
+
|
|
15
|
+
### Fixed
|
|
16
|
+
- Dashboard CPU peg: `cctally dashboard` no longer pegs a CPU core under sustained active use on large caches. The Codex quota-projection reconcile — which ran unconditionally on every dashboard tick and every `codex-*` command, re-loading the entire quota history (~2.9s) — now short-circuits to an O(1) no-op when the Codex physical state is unchanged and the stored projection is intact; `codex-daily` with no fresh Codex data drops from ~3.2s to well under a second. The dashboard sync thread also caps its CPU duty at 50% of one core via a work-proportional cooldown, so a slow rebuild can no longer keep the thread busy back-to-back. (#313)
|
|
17
|
+
- Dashboard: the #294 S5 source selector no longer causes the mobile topbar to scroll sideways on the narrowest phones. The active-source status chip sits in the non-shrinkable action cluster, and adding it pushed that cluster past a 320px-wide viewport; it now hides at ≤360px (freshness is still on the global sync chip and a degraded source stays flagged in the selector and panels), and both the source selector and its status chip drop out of the condensed one-row header once you scroll, matching how the workspace switcher and doctor chip already collapse there. (#294)
|
|
18
|
+
|
|
19
|
+
## [1.70.0] - 2026-07-17
|
|
20
|
+
|
|
21
|
+
### Added
|
|
22
|
+
- `cctally setup` now adds `statusLine.refreshInterval: 30` to a cctally-pointing Claude Code `statusLine` block that lacks one, so status-line-fed usage keeps recording on a 30-second timer while a coordinator session waits on a long-running subagent (Claude Code's event-driven status-line updates go quiet then). Ownership is add-when-absent / never-mutate / never-remove: setup never creates a `statusLine` block, never changes a `refreshInterval` you set yourself, and `--uninstall` leaves it. Surfaced in install / `--status` / `--dry-run` (text plus the `--json` `statusline_refresh {state, value, action}` object). (#311)
|
|
23
|
+
- `cctally doctor` gains a `hooks.statusline_refresh_interval` check that WARNs when a recognized cctally `statusLine` command is missing its `refreshInterval`, and is OK (with a per-state summary) otherwise. (#311)
|
|
24
|
+
|
|
25
|
+
### Changed
|
|
26
|
+
- Retuned the status-line usage-persist throttle from 60s to 25s so it sits below the new 30s `statusLine.refreshInterval` timer; a 60/60 pairing beat-frequency-throttled every other tick and oscillated the effective cadence between 60 and 120 seconds. (#311)
|
|
27
|
+
|
|
28
|
+
### Fixed
|
|
29
|
+
- Status-line usage persistence now skips sessions running a bracketed model variant such as `claude-opus-4-8[1m]` (the 1M-context variant), whose `rate_limits` describe a separate account usage pool. Persisting those poisoned the default-pool snapshots — the reset-aware high-water-mark clamp latched the foreign value and froze all subsequent genuine writes. The guard is persist-only, so a `[1m]` session still renders its own true pool numbers on the status line. (#311)
|
|
30
|
+
|
|
8
31
|
## [1.69.3] - 2026-07-17
|
|
9
32
|
|
|
10
33
|
### Fixed
|
|
@@ -23,6 +46,7 @@ based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
|
|
23
46
|
## [1.69.0] - 2026-07-17
|
|
24
47
|
|
|
25
48
|
### Added
|
|
49
|
+
- Dashboard source selector and source-aware UX (#294 S5): a Header Claude/Codex/All radiogroup (cycle with `v`) drives every panel, modal, alert filter, and share default at once, rendering each provider's native vocabulary (Codex token counters and quota windows, no `$ / 1%` or subscription-week copy) and hiding — never zero-filling — panels a source doesn't publish, with a Help-overlay note pointing at where the equivalent lives (Codex forecast/trend → the quota panel, cache-report → the hero counters). Alerts and toasts read the per-source projections (a Codex budget alert can't double-toast), the alert settings regroup into global-notifier / Claude / Codex controls with a CLI pointer for Codex quota rules, a per-source status chip surfaces freshness and degraded warnings, and every share render, composer section, preset, and history row carries and displays the source it was captured under — a mid-flow selector switch never restamps an open share flow, and `all` composes provider-labelled sections rather than a blended snapshot. The Claude-only user's data and flows are unchanged except that new share artifacts and history now say "Claude".
|
|
26
50
|
- Dashboard source-aware backend read model (#294 S4): `/api/data` now appends
|
|
27
51
|
immutable Claude, Codex, and labelled all-source states; provider-owned
|
|
28
52
|
session/project/quota details use opaque qualified routes backed by bounded
|
package/bin/_cctally_cache.py
CHANGED
|
@@ -725,6 +725,13 @@ def _clear_codex_derived_rows(conn: sqlite3.Connection) -> None:
|
|
|
725
725
|
conn.execute("DELETE FROM codex_conversation_events")
|
|
726
726
|
conn.execute("DELETE FROM codex_session_files")
|
|
727
727
|
conn.execute("DELETE FROM codex_source_roots")
|
|
728
|
+
# F3: this clears the Codex physical quota state, so any stored
|
|
729
|
+
# quota-projection certificate would become stale-valid (its cache
|
|
730
|
+
# sequence unchanged) and let the reconcile short-circuit skip over
|
|
731
|
+
# now-deleted data. Invalidate it in the same transaction.
|
|
732
|
+
conn.execute(
|
|
733
|
+
"DELETE FROM cache_meta WHERE key='codex_quota_projection_certificate'"
|
|
734
|
+
)
|
|
728
735
|
|
|
729
736
|
|
|
730
737
|
def _bump_codex_physical_mutation_seq(conn: sqlite3.Connection) -> None:
|
|
@@ -1388,6 +1395,30 @@ def _bump_mutation_seq(conn: sqlite3.Connection) -> int:
|
|
|
1388
1395
|
return int(row[0])
|
|
1389
1396
|
|
|
1390
1397
|
|
|
1398
|
+
def _force_retention_prune_after_replay(conn: sqlite3.Connection) -> None:
|
|
1399
|
+
"""#313 P3 (F9): run an UNTHROTTLED transcript retention prune after a
|
|
1400
|
+
from-zero replay (a ``--rebuild`` or a truncation/requalification re-ingest,
|
|
1401
|
+
both of which replay from offset 0 and restore >retention-day rows the
|
|
1402
|
+
throttled prune already trimmed). Best-effort — a prune failure must never
|
|
1403
|
+
break a sync. The caller invokes this only after the sync released its
|
|
1404
|
+
provider flock, so the orchestrator can re-acquire it. No-op when retention
|
|
1405
|
+
is disabled (``conversation.retention_days`` 0)."""
|
|
1406
|
+
try:
|
|
1407
|
+
import _lib_conversation_retention as retention
|
|
1408
|
+
from _cctally_config import resolve_retention_days
|
|
1409
|
+
retention_days = resolve_retention_days(_cctally().load_config())
|
|
1410
|
+
if retention_days <= 0:
|
|
1411
|
+
return
|
|
1412
|
+
retention._maybe_prune_conversation_retention(
|
|
1413
|
+
conn,
|
|
1414
|
+
now_utc=dt.datetime.now(dt.timezone.utc),
|
|
1415
|
+
retention_days=retention_days,
|
|
1416
|
+
force=True,
|
|
1417
|
+
)
|
|
1418
|
+
except Exception:
|
|
1419
|
+
pass
|
|
1420
|
+
|
|
1421
|
+
|
|
1391
1422
|
def sync_cache(
|
|
1392
1423
|
conn: sqlite3.Connection,
|
|
1393
1424
|
*,
|
|
@@ -2340,13 +2371,20 @@ def sync_cache(
|
|
|
2340
2371
|
# and the flock is still held, so the short busy_timeout keeps it from
|
|
2341
2372
|
# stalling the lock under heavy-reader contention.
|
|
2342
2373
|
_maybe_truncate_wal(conn, _cctally_core.CACHE_DB_PATH)
|
|
2343
|
-
|
|
2374
|
+
# #313 P3 (F9): a rebuild or a truncation escalation replays from offset
|
|
2375
|
+
# 0 and restores >retention-day transcript rows. Force an unthrottled
|
|
2376
|
+
# prune AFTER the flock releases below (early lock-contended / deferred
|
|
2377
|
+
# returns above never reach here, so a no-op sync does not prune).
|
|
2378
|
+
did_from_zero_replay = rebuild or stats.files_reset_truncated > 0
|
|
2344
2379
|
finally:
|
|
2345
2380
|
try:
|
|
2346
2381
|
fcntl.flock(lock_fh, fcntl.LOCK_UN)
|
|
2347
2382
|
except OSError:
|
|
2348
2383
|
pass
|
|
2349
2384
|
lock_fh.close()
|
|
2385
|
+
if did_from_zero_replay:
|
|
2386
|
+
_force_retention_prune_after_replay(conn)
|
|
2387
|
+
return stats
|
|
2350
2388
|
|
|
2351
2389
|
|
|
2352
2390
|
def backfill_conversation_messages(conn: sqlite3.Connection) -> int:
|
|
@@ -3491,6 +3529,17 @@ def sync_codex_cache(
|
|
|
3491
3529
|
"""
|
|
3492
3530
|
stats = CodexIngestStats()
|
|
3493
3531
|
project_after_unlock = False
|
|
3532
|
+
did_from_zero_replay = False
|
|
3533
|
+
# #313 P1 review (F4/F1): when the CACHE certificate is current we cannot
|
|
3534
|
+
# yet decide whether to skip the reconcile — reconcile's own short-circuit
|
|
3535
|
+
# ALSO requires the stats-side quota_projection_state signatures to match
|
|
3536
|
+
# (F1: stats.db can be wiped/recovered while cache.db persists). That
|
|
3537
|
+
# cross-DB read must happen AFTER the Codex flock releases (see the design
|
|
3538
|
+
# comment near the reconcile trigger below), so capture the material for the
|
|
3539
|
+
# deferred stats-side check here. ``None`` means "no deferred check pending"
|
|
3540
|
+
# (the seq-advanced / no-roots / stale-cert branches decide immediately).
|
|
3541
|
+
deferred_cert_roots: "set[str] | None" = None
|
|
3542
|
+
deferred_cert_sigs: "dict[str, str] | None" = None
|
|
3494
3543
|
c = _cctally()
|
|
3495
3544
|
_cctally_core.APP_DIR.mkdir(parents=True, exist_ok=True)
|
|
3496
3545
|
_cctally_core.CACHE_LOCK_CODEX_PATH.touch()
|
|
@@ -3505,6 +3554,18 @@ def sync_codex_cache(
|
|
|
3505
3554
|
stats.lock_contended = True
|
|
3506
3555
|
return stats
|
|
3507
3556
|
|
|
3557
|
+
# F4 (#313): the reconcile trigger gate is "did the Codex physical
|
|
3558
|
+
# mutation sequence advance during this sync", NOT rows_changed —
|
|
3559
|
+
# rows_changed counts only inserted accounting rows and misses
|
|
3560
|
+
# quota-only / metadata-only / prune-only batches that bump the
|
|
3561
|
+
# sequence. Capture the baseline before any clear/prune/ingest bump.
|
|
3562
|
+
from _cctally_quota import (
|
|
3563
|
+
codex_physical_mutation_seq,
|
|
3564
|
+
load_codex_quota_projection_certificate,
|
|
3565
|
+
_cache_root_keys,
|
|
3566
|
+
)
|
|
3567
|
+
seq_before = codex_physical_mutation_seq(conn)
|
|
3568
|
+
|
|
3508
3569
|
if rebuild:
|
|
3509
3570
|
# Clear INSIDE the lock — see sync_cache() for the full
|
|
3510
3571
|
# rationale. Done before the existing SELECT so delta
|
|
@@ -3892,7 +3953,56 @@ def sync_codex_cache(
|
|
|
3892
3953
|
# Codex cache flock in ``finally`` below. cache.db and stats.db are not
|
|
3893
3954
|
# cross-database atomic: after this committed ingest, a projection
|
|
3894
3955
|
# interruption is repaired by the next full reconciliation.
|
|
3895
|
-
|
|
3956
|
+
#
|
|
3957
|
+
# F4 (#313): reconcile when the physical mutation sequence advanced this
|
|
3958
|
+
# sync (a genuine quota/metadata/prune change — NOT rows_changed, which
|
|
3959
|
+
# misses quota-only batches). A pure no-op with an already-coherent
|
|
3960
|
+
# certificate skips even the O(1) reconcile call.
|
|
3961
|
+
#
|
|
3962
|
+
# A no-op sync must STILL reconcile when there are Codex roots but the
|
|
3963
|
+
# projection certificate is missing/stale at the current sequence — a
|
|
3964
|
+
# lost/failed certificate write (best-effort I/O; can fail under a
|
|
3965
|
+
# cache.db lock storm) leaves the dashboard's Codex source "unavailable"
|
|
3966
|
+
# and is recovered by the next unchanged-file sync re-stamping the
|
|
3967
|
+
# certificate. Claude-only users (no Codex roots) always skip.
|
|
3968
|
+
#
|
|
3969
|
+
# F1 (#313 P1 review): even when the CACHE certificate looks current, the
|
|
3970
|
+
# STATS-side quota_projection_state may have been independently
|
|
3971
|
+
# wiped/recovered (this user has a documented stats.db corruption
|
|
3972
|
+
# history). The cache cert alone does NOT prove stats.db still holds the
|
|
3973
|
+
# projection, so the skip decision on that branch is DEFERRED to the
|
|
3974
|
+
# post-flock stats-side signature check below — making the gate's
|
|
3975
|
+
# skip-condition identical to reconcile's own short-circuit-condition.
|
|
3976
|
+
cur_seq = codex_physical_mutation_seq(conn)
|
|
3977
|
+
if cur_seq != seq_before:
|
|
3978
|
+
project_after_unlock = True
|
|
3979
|
+
else:
|
|
3980
|
+
active_roots = _cache_root_keys(conn)
|
|
3981
|
+
if not active_roots:
|
|
3982
|
+
project_after_unlock = False
|
|
3983
|
+
else:
|
|
3984
|
+
certificate = load_codex_quota_projection_certificate(conn)
|
|
3985
|
+
certificate_current = (
|
|
3986
|
+
certificate is not None
|
|
3987
|
+
and certificate[0] == cur_seq
|
|
3988
|
+
and active_roots <= set(certificate[1])
|
|
3989
|
+
)
|
|
3990
|
+
if certificate_current:
|
|
3991
|
+
# The CACHE certificate is current, but that alone does NOT
|
|
3992
|
+
# prove stats.db still holds the projection (F1). Defer the
|
|
3993
|
+
# stats-side signature match until AFTER the Codex flock
|
|
3994
|
+
# releases below — only then may the reconcile be skipped.
|
|
3995
|
+
project_after_unlock = False
|
|
3996
|
+
deferred_cert_roots = set(active_roots)
|
|
3997
|
+
assert certificate is not None
|
|
3998
|
+
deferred_cert_sigs = dict(certificate[1])
|
|
3999
|
+
else:
|
|
4000
|
+
project_after_unlock = True
|
|
4001
|
+
# #313 P3 (F9): a Codex rebuild or a truncation/requalification
|
|
4002
|
+
# re-ingest replays from offset 0 and restores >retention-day
|
|
4003
|
+
# codex_conversation_events. Force an unthrottled prune after the flock
|
|
4004
|
+
# releases (below), so restored old rows don't persist for up to 24h.
|
|
4005
|
+
did_from_zero_replay = rebuild or stats.files_reset_truncated > 0
|
|
3896
4006
|
finally:
|
|
3897
4007
|
try:
|
|
3898
4008
|
fcntl.flock(lock_fh, fcntl.LOCK_UN)
|
|
@@ -3900,9 +4010,29 @@ def sync_codex_cache(
|
|
|
3900
4010
|
pass
|
|
3901
4011
|
lock_fh.close()
|
|
3902
4012
|
|
|
4013
|
+
if deferred_cert_roots is not None:
|
|
4014
|
+
# F1: the gate's skip-condition must be IDENTICAL to
|
|
4015
|
+
# reconcile_codex_quota_projection's own short-circuit-condition, which
|
|
4016
|
+
# additionally requires the stats-side quota_projection_state signatures
|
|
4017
|
+
# to match the cache certificate for every active root. Read stats.db
|
|
4018
|
+
# HERE — after the Codex flock released in the ``finally`` above — so the
|
|
4019
|
+
# cross-DB read never widens the Codex flock's critical section (the
|
|
4020
|
+
# design invariant documented near the reconcile trigger). A wiped/stale
|
|
4021
|
+
# stats projection (mismatch) forces the reconcile so it self-heals.
|
|
4022
|
+
from _cctally_quota import _stats_projection_signatures_match
|
|
4023
|
+
stats_conn = _cctally_core.open_db()
|
|
4024
|
+
try:
|
|
4025
|
+
if not _stats_projection_signatures_match(
|
|
4026
|
+
stats_conn, deferred_cert_roots, deferred_cert_sigs or {}
|
|
4027
|
+
):
|
|
4028
|
+
project_after_unlock = True
|
|
4029
|
+
finally:
|
|
4030
|
+
stats_conn.close()
|
|
3903
4031
|
if project_after_unlock:
|
|
3904
4032
|
from _cctally_quota import reconcile_codex_quota_projection
|
|
3905
4033
|
reconcile_codex_quota_projection()
|
|
4034
|
+
if did_from_zero_replay:
|
|
4035
|
+
_force_retention_prune_after_replay(conn)
|
|
3906
4036
|
return stats
|
|
3907
4037
|
|
|
3908
4038
|
|
|
@@ -4253,6 +4383,41 @@ def cmd_cache_sync(args: argparse.Namespace) -> int:
|
|
|
4253
4383
|
)
|
|
4254
4384
|
return 0
|
|
4255
4385
|
|
|
4386
|
+
# --prune-conversations: on-demand, UNTHROTTLED transcript retention prune
|
|
4387
|
+
# (#313 P3). Removes >retention-day conversation transcripts (re-derivable
|
|
4388
|
+
# from JSONL). Reclaim the freed disk space with `cctally db vacuum`.
|
|
4389
|
+
if getattr(args, "prune_conversations", False):
|
|
4390
|
+
from _cctally_config import resolve_retention_days
|
|
4391
|
+
import _lib_conversation_retention as retention
|
|
4392
|
+
retention_days = resolve_retention_days(_cctally().load_config())
|
|
4393
|
+
if retention_days <= 0:
|
|
4394
|
+
eprint(
|
|
4395
|
+
"[cache-sync] transcript retention is disabled "
|
|
4396
|
+
"(conversation.retention_days=0); nothing pruned."
|
|
4397
|
+
)
|
|
4398
|
+
return 0
|
|
4399
|
+
result = retention._maybe_prune_conversation_retention(
|
|
4400
|
+
conn,
|
|
4401
|
+
now_utc=dt.datetime.now(dt.timezone.utc),
|
|
4402
|
+
retention_days=retention_days,
|
|
4403
|
+
force=True,
|
|
4404
|
+
)
|
|
4405
|
+
if result is None:
|
|
4406
|
+
eprint(
|
|
4407
|
+
"[cache-sync] prune-conversations skipped: another process "
|
|
4408
|
+
"holds the maintenance or a provider lock; retry shortly."
|
|
4409
|
+
)
|
|
4410
|
+
return 1
|
|
4411
|
+
eprint(
|
|
4412
|
+
f"[cache-sync] pruned transcripts older than {retention_days}d: "
|
|
4413
|
+
f"claude {result.claude_sessions} session(s) / "
|
|
4414
|
+
f"{result.claude_messages} message(s), "
|
|
4415
|
+
f"codex {result.codex_conversations} conversation(s) / "
|
|
4416
|
+
f"{result.codex_events} event(s). "
|
|
4417
|
+
f"Run `cctally db vacuum` to reclaim the freed space."
|
|
4418
|
+
)
|
|
4419
|
+
return 0
|
|
4420
|
+
|
|
4256
4421
|
# Note: when --rebuild is set we delegate the DELETE to sync_cache /
|
|
4257
4422
|
# sync_codex_cache, which execute it AFTER acquiring the flock. A
|
|
4258
4423
|
# pre-sync DELETE here would wipe the cache even when the subsequent
|
package/bin/_cctally_config.py
CHANGED
|
@@ -327,11 +327,76 @@ ALLOWED_CONFIG_KEYS = (
|
|
|
327
327
|
"budget.codex.alert_thresholds",
|
|
328
328
|
"budget.codex.projected_enabled",
|
|
329
329
|
"telemetry.enabled",
|
|
330
|
+
"conversation.retention_days",
|
|
330
331
|
)
|
|
331
332
|
|
|
332
333
|
|
|
333
334
|
_CODEX_BUDGET_LEAF_PREFIX = "budget.codex."
|
|
334
335
|
|
|
336
|
+
_DEFAULT_CONVERSATION_RETENTION_DAYS = 180
|
|
337
|
+
|
|
338
|
+
|
|
339
|
+
def _validate_retention_days_value(raw: object) -> int:
|
|
340
|
+
"""Validate a ``config set`` value for ``conversation.retention_days`` (F8).
|
|
341
|
+
|
|
342
|
+
Accept a non-negative integer; ``off`` / ``0`` canonicalize to integer 0
|
|
343
|
+
(disabled — keep transcripts forever). Reject booleans, negatives, and
|
|
344
|
+
non-numeric input by raising ``ValueError`` (the caller maps that to exit 2).
|
|
345
|
+
"""
|
|
346
|
+
if isinstance(raw, bool):
|
|
347
|
+
raise ValueError(
|
|
348
|
+
"conversation.retention_days must be a non-negative integer or 'off'"
|
|
349
|
+
)
|
|
350
|
+
if isinstance(raw, str):
|
|
351
|
+
text = raw.strip().lower()
|
|
352
|
+
if text == "off":
|
|
353
|
+
return 0
|
|
354
|
+
try:
|
|
355
|
+
value = int(text, 10)
|
|
356
|
+
except (TypeError, ValueError):
|
|
357
|
+
raise ValueError(
|
|
358
|
+
"conversation.retention_days must be a non-negative integer "
|
|
359
|
+
f"or 'off', got {raw!r}"
|
|
360
|
+
)
|
|
361
|
+
elif isinstance(raw, int):
|
|
362
|
+
value = raw
|
|
363
|
+
else:
|
|
364
|
+
raise ValueError(
|
|
365
|
+
"conversation.retention_days must be a non-negative integer or 'off'"
|
|
366
|
+
)
|
|
367
|
+
if value < 0:
|
|
368
|
+
raise ValueError(
|
|
369
|
+
"conversation.retention_days must be >= 0 (use 'off' or 0 to disable)"
|
|
370
|
+
)
|
|
371
|
+
return value
|
|
372
|
+
|
|
373
|
+
|
|
374
|
+
def resolve_retention_days(config: dict) -> int:
|
|
375
|
+
"""Effective conversation transcript retention in days (F8).
|
|
376
|
+
|
|
377
|
+
Default is ``180``; ``0`` disables retention (keep forever). Any malformed
|
|
378
|
+
persisted value (non-int, boolean, negative, un-parseable string) degrades
|
|
379
|
+
to the safe 180 default rather than raising.
|
|
380
|
+
"""
|
|
381
|
+
default = _DEFAULT_CONVERSATION_RETENTION_DAYS
|
|
382
|
+
block = config.get("conversation") if isinstance(config, dict) else None
|
|
383
|
+
if not isinstance(block, dict):
|
|
384
|
+
return default
|
|
385
|
+
stored = block.get("retention_days")
|
|
386
|
+
if stored is None:
|
|
387
|
+
return default
|
|
388
|
+
if isinstance(stored, bool):
|
|
389
|
+
return default
|
|
390
|
+
if isinstance(stored, int):
|
|
391
|
+
return stored if stored >= 0 else default
|
|
392
|
+
if isinstance(stored, str):
|
|
393
|
+
try:
|
|
394
|
+
value = int(stored.strip(), 10)
|
|
395
|
+
except (TypeError, ValueError):
|
|
396
|
+
return default
|
|
397
|
+
return value if value >= 0 else default
|
|
398
|
+
return default
|
|
399
|
+
|
|
335
400
|
|
|
336
401
|
def _config_codex_leaf_value(config: dict, key: str) -> object:
|
|
337
402
|
"""Read one nested Codex budget leaf with its public default.
|
|
@@ -794,6 +859,10 @@ def _config_known_value(config: dict, key: str) -> "object":
|
|
|
794
859
|
except ValueError:
|
|
795
860
|
return True
|
|
796
861
|
return True
|
|
862
|
+
if key == "conversation.retention_days":
|
|
863
|
+
# Effective transcript-retention window in days (default 180; 0 = keep
|
|
864
|
+
# forever). Malformed persisted data surfaces the safe default (F8).
|
|
865
|
+
return resolve_retention_days(config)
|
|
797
866
|
if key in ("update.check.enabled", "update.check.ttl_hours"):
|
|
798
867
|
# Defaults mirror `_is_update_check_due` (True / 24 hours).
|
|
799
868
|
# Hand-edited junk surfaces as the default — matches dashboard.bind.
|
|
@@ -1422,6 +1491,32 @@ def _cmd_config_set(args: argparse.Namespace) -> int:
|
|
|
1422
1491
|
rendered = str(normalized)
|
|
1423
1492
|
print(f"{key}={rendered}")
|
|
1424
1493
|
return 0
|
|
1494
|
+
if key == "conversation.retention_days":
|
|
1495
|
+
# Validate first; rejection short-circuits before lock acquisition (F8).
|
|
1496
|
+
try:
|
|
1497
|
+
normalized = _validate_retention_days_value(raw)
|
|
1498
|
+
except ValueError as exc:
|
|
1499
|
+
print(f"cctally: {exc}", file=sys.stderr)
|
|
1500
|
+
return 2
|
|
1501
|
+
with config_writer_lock():
|
|
1502
|
+
config = _load_config_unlocked()
|
|
1503
|
+
existing = config.get("conversation")
|
|
1504
|
+
if existing is not None and not isinstance(existing, dict):
|
|
1505
|
+
print(
|
|
1506
|
+
"cctally: conversation config error: conversation must be "
|
|
1507
|
+
"an object",
|
|
1508
|
+
file=sys.stderr,
|
|
1509
|
+
)
|
|
1510
|
+
return 2
|
|
1511
|
+
block = dict(existing or {})
|
|
1512
|
+
block["retention_days"] = normalized
|
|
1513
|
+
config["conversation"] = block
|
|
1514
|
+
save_config(config)
|
|
1515
|
+
if getattr(args, "emit_json", False):
|
|
1516
|
+
print(json.dumps({"conversation": {"retention_days": normalized}}, indent=2))
|
|
1517
|
+
else:
|
|
1518
|
+
print(f"{key}={normalized}")
|
|
1519
|
+
return 0
|
|
1425
1520
|
if key in ("update.check.enabled", "update.check.ttl_hours"):
|
|
1426
1521
|
# Validate first; rejection short-circuits before lock acquisition.
|
|
1427
1522
|
if key == "update.check.enabled":
|
|
@@ -1862,6 +1957,19 @@ def _cmd_config_unset(args: argparse.Namespace) -> int:
|
|
|
1862
1957
|
save_config(config)
|
|
1863
1958
|
# idempotent: silent on missing key
|
|
1864
1959
|
return 0
|
|
1960
|
+
if key == "conversation.retention_days":
|
|
1961
|
+
# Mirror the display.tz branch: drop the leaf; if the `conversation`
|
|
1962
|
+
# block ends up empty, drop it too. Next get resolves the 180 default.
|
|
1963
|
+
with config_writer_lock():
|
|
1964
|
+
config = _load_config_unlocked()
|
|
1965
|
+
block = config.get("conversation")
|
|
1966
|
+
if isinstance(block, dict) and "retention_days" in block:
|
|
1967
|
+
del block["retention_days"]
|
|
1968
|
+
if not block:
|
|
1969
|
+
config.pop("conversation", None)
|
|
1970
|
+
save_config(config)
|
|
1971
|
+
# idempotent: silent on missing key
|
|
1972
|
+
return 0
|
|
1865
1973
|
if key in ("update.check.enabled", "update.check.ttl_hours"):
|
|
1866
1974
|
# Mirror the dashboard.bind branch: drop the leaf, then prune
|
|
1867
1975
|
# empty `check` and empty `update` so config.json stays tidy.
|
package/bin/_cctally_core.py
CHANGED
|
@@ -61,7 +61,8 @@ def _init_paths_from_env() -> None:
|
|
|
61
61
|
"""
|
|
62
62
|
global APP_DIR, LEGACY_APP_DIR, LOG_DIR, DEV_MODE
|
|
63
63
|
global DB_PATH, CACHE_DB_PATH
|
|
64
|
-
global CACHE_LOCK_PATH, CACHE_LOCK_CODEX_PATH,
|
|
64
|
+
global CACHE_LOCK_PATH, CACHE_LOCK_CODEX_PATH, CACHE_LOCK_MAINTENANCE_PATH
|
|
65
|
+
global CONFIG_LOCK_PATH
|
|
65
66
|
global CONFIG_PATH, MIGRATION_ERROR_LOG_PATH, CHANGELOG_PATH
|
|
66
67
|
global HOOK_TICK_LOG_DIR, HOOK_TICK_LOG_PATH, HOOK_TICK_LOG_ROTATED_PATH
|
|
67
68
|
global HOOK_TICK_THROTTLE_PATH, HOOK_TICK_THROTTLE_LOCK_PATH
|
|
@@ -100,6 +101,10 @@ def _init_paths_from_env() -> None:
|
|
|
100
101
|
|
|
101
102
|
CACHE_LOCK_PATH = APP_DIR / "cache.db.lock"
|
|
102
103
|
CACHE_LOCK_CODEX_PATH = APP_DIR / "cache.db.codex.lock"
|
|
104
|
+
# #313 P3 (F7): dedicated maintenance flock serializing the transcript
|
|
105
|
+
# retention prune across processes, held ABOVE the two provider flocks so a
|
|
106
|
+
# rebuild/reingest cannot land between candidate selection and deletion.
|
|
107
|
+
CACHE_LOCK_MAINTENANCE_PATH = APP_DIR / "cache.db.maintenance.lock"
|
|
103
108
|
CONFIG_LOCK_PATH = APP_DIR / "config.json.lock"
|
|
104
109
|
|
|
105
110
|
CONFIG_PATH = APP_DIR / "config.json"
|
|
@@ -243,13 +248,37 @@ def _real_prod_data_dir() -> pathlib.Path:
|
|
|
243
248
|
#
|
|
244
249
|
# STATUSLINE_PERSIST_THROTTLE_SECONDS: min seconds between statusline
|
|
245
250
|
# persist attempts (keyed off STATUSLINE_OBSERVE_MARKER_PATH liveness).
|
|
251
|
+
# Retuned 60.0 -> 25.0 for #311: the setup-managed statusLine.refreshInterval
|
|
252
|
+
# timer fires the statusline on a 30s cadence, and the throttle MUST be
|
|
253
|
+
# strictly LESS than that interval. When interval <= throttle, phase jitter
|
|
254
|
+
# produces beat-frequency skips (a tick at 59.9s of marker age is throttled;
|
|
255
|
+
# the next persist waits ~120s), so a 60/60 pairing oscillates 60<->120s.
|
|
256
|
+
# With interval 30 > throttle 25, every tick whose predecessor's record
|
|
257
|
+
# completed promptly (within interval - throttle = 5s) passes the gate.
|
|
258
|
+
# CADENCE QUALIFIER: the marker is touched at persist COMPLETION, so a tick
|
|
259
|
+
# observes marker age = 30 - d where d is the previous record's duration;
|
|
260
|
+
# if d > 5s (the record kernel can run cmd_sync_week JSONL scans) that tick
|
|
261
|
+
# throttles and the cycle degrades to skip-one-tick (~60s), self-correcting
|
|
262
|
+
# on the next tick. Exact-30s is NOT an invariant (no attempt-start marker
|
|
263
|
+
# exists — one would let a hung/failed record claim liveness and suppress
|
|
264
|
+
# the OAuth backfill). Still strictly better than the 60/60 beat's 60-120s.
|
|
246
265
|
# OAUTH_BACKFILL_STALE_SECONDS: the OAuth poll only backfills once the
|
|
247
266
|
# observation marker is at least this stale (i.e. the statusline has
|
|
248
267
|
# NOT fed recently). Strictly greater than the persist throttle so the
|
|
249
|
-
# statusline is the primary writer and OAuth only covers its absence
|
|
268
|
+
# statusline is the primary writer and OAuth only covers its absence
|
|
269
|
+
# (300 > 25 still holds).
|
|
250
270
|
# OAUTH_BACKOFF_BASE_SECONDS / OAUTH_BACKOFF_CAP_SECONDS: the headerless
|
|
251
271
|
# exponential 429 backoff (base * 2**consecutive_429, capped).
|
|
252
|
-
STATUSLINE_PERSIST_THROTTLE_SECONDS =
|
|
272
|
+
STATUSLINE_PERSIST_THROTTLE_SECONDS = 25.0
|
|
273
|
+
# STATUSLINE_REFRESH_INTERVAL_DEFAULT (#311): the value `cctally setup`
|
|
274
|
+
# writes into Claude Code's settings.json `statusLine.refreshInterval` when a
|
|
275
|
+
# recognized cctally statusLine block lacks one. Claude Code re-runs the
|
|
276
|
+
# statusline command on this fixed timer "in addition to the event-driven
|
|
277
|
+
# updates", which keeps the usage-persistence feeder ticking while a parent
|
|
278
|
+
# session waits on a long subagent (event-driven updates go quiet then). MUST
|
|
279
|
+
# exceed STATUSLINE_PERSIST_THROTTLE_SECONDS (30 > 25) — see the pairing rule
|
|
280
|
+
# above. Add-when-absent only; a user-set value is never mutated.
|
|
281
|
+
STATUSLINE_REFRESH_INTERVAL_DEFAULT = 30
|
|
253
282
|
OAUTH_BACKFILL_STALE_SECONDS = 300.0
|
|
254
283
|
OAUTH_BACKOFF_BASE_SECONDS = 60.0
|
|
255
284
|
OAUTH_BACKOFF_CAP_SECONDS = 3600.0
|
|
@@ -1068,6 +1068,86 @@ def _load_recorded_five_hour_windows(*args, **kwargs):
|
|
|
1068
1068
|
return sys.modules["cctally"]._load_recorded_five_hour_windows(*args, **kwargs)
|
|
1069
1069
|
|
|
1070
1070
|
|
|
1071
|
+
def _next_deadline(t0: float, interval: float, work: float) -> float:
|
|
1072
|
+
"""Monotonic deadline for the next sync iteration (#313 P2 / F10).
|
|
1073
|
+
|
|
1074
|
+
The iteration's work ended at ``t0 + work``; cool down from there for
|
|
1075
|
+
``max(interval, work)``. So the period is ``work + max(interval, work)``:
|
|
1076
|
+
|
|
1077
|
+
- work >= interval → period = ``2 * work`` → CPU duty capped at 50% of one
|
|
1078
|
+
core, scale-independent (a slow rebuild on a huge cache can never peg a
|
|
1079
|
+
full core — the exact #313 symptom).
|
|
1080
|
+
- work < interval → period = ``work + interval`` — byte-identical to the
|
|
1081
|
+
prior loop (rebuild then a fixed ``interval`` sleep), so normal-cadence
|
|
1082
|
+
behavior and its goldens are unchanged.
|
|
1083
|
+
|
|
1084
|
+
NOTE (spec deviation): the spec §3 pseudocode's ``deadline = t0 + max(
|
|
1085
|
+
interval, work)`` cools down from ``t0`` (before the work), which for
|
|
1086
|
+
work >= interval yields a ZERO cooldown (period = work, 100% duty) — it does
|
|
1087
|
+
not meet the spec's own stated "period = 2*work / duty <= 50%" property or
|
|
1088
|
+
§1's "<= 50% of one core" success criterion, and would leave the peg unfixed
|
|
1089
|
+
(in fact worse than the old work+interval cadence). This formula cools down
|
|
1090
|
+
from the work's END, which is what those properties require.
|
|
1091
|
+
"""
|
|
1092
|
+
return (t0 + work) + max(interval, work)
|
|
1093
|
+
|
|
1094
|
+
|
|
1095
|
+
def _dashboard_sync_loop(
|
|
1096
|
+
*,
|
|
1097
|
+
stop,
|
|
1098
|
+
interval: float,
|
|
1099
|
+
run_iteration,
|
|
1100
|
+
take_sync_request,
|
|
1101
|
+
monotonic=time.monotonic,
|
|
1102
|
+
sleep=time.sleep,
|
|
1103
|
+
) -> None:
|
|
1104
|
+
"""Run the dashboard's automatic sync loop with a work-proportional cooldown.
|
|
1105
|
+
|
|
1106
|
+
``run_iteration`` performs one whole automatic iteration (rebuild plus any
|
|
1107
|
+
orphan self-heal / retention maintenance the thread does), so its measured
|
|
1108
|
+
duration — not just the rebuild — drives the deadline (F10). The manual
|
|
1109
|
+
``POST /api/sync`` refresh is a separate synchronous path under ``sync_lock``
|
|
1110
|
+
and is unaffected; ``take_sync_request`` is the TUI force-refresh flag that
|
|
1111
|
+
breaks the cooldown early.
|
|
1112
|
+
"""
|
|
1113
|
+
while not stop.is_set():
|
|
1114
|
+
t0 = monotonic()
|
|
1115
|
+
run_iteration()
|
|
1116
|
+
work = monotonic() - t0
|
|
1117
|
+
deadline = _next_deadline(t0, interval, work)
|
|
1118
|
+
while not stop.is_set() and monotonic() < deadline:
|
|
1119
|
+
if take_sync_request():
|
|
1120
|
+
break
|
|
1121
|
+
sleep(min(0.1, max(0.0, deadline - monotonic())))
|
|
1122
|
+
|
|
1123
|
+
|
|
1124
|
+
def _dashboard_maybe_prune_retention() -> None:
|
|
1125
|
+
"""#313 P3 (F7): throttled transcript retention prune driven from the
|
|
1126
|
+
dashboard sync thread. Runs once per iteration INSIDE the measured cooldown
|
|
1127
|
+
work (Task 6), so its cost paces the deadline. Opens a dedicated cache
|
|
1128
|
+
connection (no provider flock, no open transaction) for the orchestrator.
|
|
1129
|
+
No-op when retention is disabled; never raises (a prune failure must not
|
|
1130
|
+
crash the sync thread)."""
|
|
1131
|
+
try:
|
|
1132
|
+
c = sys.modules["cctally"]
|
|
1133
|
+
from _cctally_config import resolve_retention_days
|
|
1134
|
+
import _lib_conversation_retention as retention
|
|
1135
|
+
retention_days = resolve_retention_days(c.load_config())
|
|
1136
|
+
if retention_days <= 0:
|
|
1137
|
+
return
|
|
1138
|
+
conn = c.open_cache_db()
|
|
1139
|
+
try:
|
|
1140
|
+
retention._maybe_prune_conversation_retention(
|
|
1141
|
+
conn,
|
|
1142
|
+
now_utc=dt.datetime.now(dt.timezone.utc),
|
|
1143
|
+
retention_days=retention_days,
|
|
1144
|
+
)
|
|
1145
|
+
finally:
|
|
1146
|
+
conn.close()
|
|
1147
|
+
except Exception:
|
|
1148
|
+
pass
|
|
1149
|
+
|
|
1150
|
+
|
|
1071
1151
|
def _make_run_sync_now(*args, **kwargs):
|
|
1072
1152
|
return sys.modules["cctally"]._make_run_sync_now(*args, **kwargs)
|
|
1073
1153
|
|
|
@@ -6623,23 +6703,38 @@ def cmd_dashboard(args: argparse.Namespace) -> int:
|
|
|
6623
6703
|
|
|
6624
6704
|
class _DashboardSyncThread(_c_for_subclass._TuiSyncThread):
|
|
6625
6705
|
def _run(self) -> None:
|
|
6626
|
-
last_heal = _time.monotonic()
|
|
6627
|
-
|
|
6706
|
+
last_heal = [_time.monotonic()]
|
|
6707
|
+
|
|
6708
|
+
def run_iteration() -> None:
|
|
6628
6709
|
_run_sync_now(skip_sync=self._skip_sync)
|
|
6629
6710
|
# Self-heal removed-worktree orphans on a ~60s cadence (far
|
|
6630
6711
|
# rarer than the sync tick — a deleted worktree is not urgent).
|
|
6631
6712
|
# Non-blocking on the flock, so a contended tick just retries
|
|
6632
|
-
# next cadence; gated off under --no-sync.
|
|
6713
|
+
# next cadence; gated off under --no-sync. Runs INSIDE the
|
|
6714
|
+
# measured iteration so its cost counts toward the cooldown
|
|
6715
|
+
# deadline (#313 P2 / F10).
|
|
6633
6716
|
if (not self._skip_sync
|
|
6634
|
-
and _time.monotonic() - last_heal >= 60.0):
|
|
6635
|
-
last_heal = _time.monotonic()
|
|
6717
|
+
and _time.monotonic() - last_heal[0] >= 60.0):
|
|
6718
|
+
last_heal[0] = _time.monotonic()
|
|
6636
6719
|
_dashboard_self_heal_orphans(skip_sync=self._skip_sync)
|
|
6637
|
-
|
|
6638
|
-
|
|
6639
|
-
|
|
6640
|
-
|
|
6641
|
-
|
|
6642
|
-
|
|
6720
|
+
# #313 P3 (F7): throttled transcript retention prune, inside the
|
|
6721
|
+
# measured iteration so its cost is in the cooldown work. Gated
|
|
6722
|
+
# off under --no-sync (frozen mode makes no cache writes).
|
|
6723
|
+
if not self._skip_sync:
|
|
6724
|
+
_dashboard_maybe_prune_retention()
|
|
6725
|
+
|
|
6726
|
+
# Work-proportional cooldown (F10): sleep to t0 + max(interval, work)
|
|
6727
|
+
# so a slow rebuild cannot peg a full core. The manual POST /api/sync
|
|
6728
|
+
# refresh runs synchronously under sync_lock, independent of this
|
|
6729
|
+
# cooldown, so a user force-refresh is always immediate.
|
|
6730
|
+
_dashboard_sync_loop(
|
|
6731
|
+
stop=self._stop,
|
|
6732
|
+
interval=self._interval,
|
|
6733
|
+
run_iteration=run_iteration,
|
|
6734
|
+
take_sync_request=self._ref.take_sync_request,
|
|
6735
|
+
monotonic=_time.monotonic,
|
|
6736
|
+
sleep=_time.sleep,
|
|
6737
|
+
)
|
|
6643
6738
|
|
|
6644
6739
|
sync_thread = (
|
|
6645
6740
|
None if args.no_sync
|