cctally 1.92.0 → 1.92.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +21 -1
- package/README.md +4 -2
- package/bin/_cctally_cache.py +44 -4
- package/bin/_cctally_core.py +56 -7
- package/bin/_cctally_dashboard.py +71 -14
- package/bin/_cctally_dashboard_conversation.py +8 -4
- package/bin/_cctally_db.py +111 -11
- package/bin/_cctally_journal.py +1102 -102
- package/bin/_cctally_parser.py +20 -0
- package/bin/_cctally_quota.py +2 -2
- package/bin/_cctally_statusline.py +6 -6
- package/bin/_cctally_store.py +783 -67
- package/bin/_cctally_tui.py +54 -6
- package/bin/_lib_codex_conversation_query.py +30 -5
- package/bin/_lib_codex_find_projection.py +147 -0
- package/bin/_lib_conversation_dispatch.py +15 -1
- package/bin/_lib_conversation_query.py +62 -2
- package/bin/_lib_journal_router.py +234 -0
- package/bin/_lib_stats_publish.py +243 -0
- package/bin/cctally +12 -3
- package/dashboard/static/assets/{index-BEzzJtUd.js → index-Dat-mza6.js} +51 -51
- package/dashboard/static/dashboard.html +1 -1
- package/package.json +3 -1
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,26 @@ based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
|
|
5
5
|
|
|
6
6
|
## [Unreleased]
|
|
7
7
|
|
|
8
|
+
## [1.92.2] - 2026-08-06
|
|
9
|
+
|
|
10
|
+
### Changed
|
|
11
|
+
- Rebuilding the stats index now uses about half the memory it used to. On a 1.7 GB journal the rebuild's peak memory dropped from 9.0 GB to 4.6 GB, and the memory Python itself holds dropped from 6.5 GB to 2.1 GB. The rebuild used to read the whole journal twice — once to hold every raw line and once to hold every parsed record — and then read it a third time from the beginning to find one operator record; it now reads each byte at most once and keeps only the records it actually replays. Reading and decoding the journal got faster (12.2 s to 9.5 s), the separate search for that operator record disappeared entirely (6.3 s to none), and a whole rebuild finished in 62.9 s instead of 68.9 s. Nothing about the resulting index changed: the same records are selected, the same rows are written, and the same corrections win. One thing did get worse, and you may notice it: while a rebuild replays Codex quota history it now blocks other commands from writing the Codex cache for longer than before — about 38% longer on a run whose data is already in memory (16.7 s to 23.0 s) and about 18% longer on a cold one (41.6 s to 48.9 s) — because the quota records are now decoded while that lock is held instead of before it is taken (#496).
|
|
12
|
+
|
|
13
|
+
### Fixed
|
|
14
|
+
- A readable but structurally damaged stats index no longer traps an upgrade or recovery in an endless rebuild loop. Version 1.92.1 could copy a healthy rebuilt generation into a file that still contained unreferenced pages, preserve those pages because no schema object named them, fail its post-publication integrity check, and leave the dashboard at `server sync error`; CLI block reports then lost their authoritative reset anchors and marked every block approximate. Publication now checks the destination before mutating it and uses the validated replacement path when the existing file fails integrity, while healthy files keep the in-place transactional path (#496).
|
|
15
|
+
|
|
16
|
+
## [1.92.1] - 2026-08-06
|
|
17
|
+
|
|
18
|
+
### Changed
|
|
19
|
+
- A rebuilt stats index is now published into the existing database file inside a single transaction, instead of being written to a scratch file that is then renamed over the live one. Reports and the dashboard keep working while a rebuild publishes, rather than being refused because a connection is open, and a reader that is mid-query keeps seeing the data it started with. Because nothing is destroyed, `cctally db rebuild --db stats` no longer leaves a quarantined copy of the previous index and no longer prints the line saying it did; use `cctally db backup --db stats` when you want a snapshot beforehand. If a published index then fails its own validation, the refusal explains that no copy of the previous index was kept, instead of directing you to a quarantine directory that was never created. A database the tool cannot open at all is still replaced the old way, with the previous copy preserved. This upgrade rebuilds the stats index once (#496).
|
|
20
|
+
- Recovering a corrupt stats index no longer makes you wait for it. Previously the command that first met the damage — a status line render, a dashboard start, any report — replayed the whole journal on its own thread before returning. That recovery now runs in the background: the command that found the damage writes the evidence, schedules the rebuild, tells you where the evidence is and that a rebuild is running, and exits without replacing anything. The dashboard and the terminal dashboard show a "rebuilding" frame and fill in on a later refresh, and `cctally db rebuild --db stats` is still the command to run when you want to wait for it. Repeated detections while a rebuild is in flight collapse into that one rebuild rather than starting more. A recovery also no longer runs unless a second check confirms the index really is damaged, and every recovery is now recorded in a small private log of the last 50 events, so a fault that keeps coming back is visible instead of only being repaired. That log does not report whether any recorded value changed, because the rebuild has no way to know (#496).
|
|
21
|
+
|
|
22
|
+
### Fixed
|
|
23
|
+
- The Codex Current Cycle modal no longer presents its native-quota label as a second provider identity, keeping Codex on its single app-wide identity accent while preserving the modal's section and state colors (#498).
|
|
24
|
+
- Account-filtered Codex conversation lists now retain each conversation's project grouping and badge instead of relabelling every scoped row as `(unassigned)` (#497).
|
|
25
|
+
- Conversation permalinks now keep the selected conversation visible and marked current even when it falls beyond the rail's first 50 rows, without changing the saved search and filter behavior of browser history traversal (#501).
|
|
26
|
+
- Conversation Viewer find now counts and reaches text inside Codex injected-context, skill, command, compaction, and notification bodies. Selecting a match opens only its owning disclosure and marks the exact visible text, including provider-tagged context prose and rendered context diffs; existing stores rebuild the disposable find projection in bounded resumable batches (#499).
|
|
27
|
+
|
|
8
28
|
## [1.92.0] - 2026-08-05
|
|
9
29
|
|
|
10
30
|
### Added
|
|
@@ -22,7 +42,7 @@ based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
|
|
22
42
|
- Stats damage characterization now detects a table or index whose root page holds the wrong kind of b-tree page. The previous check accepted any valid page type, so the damage shape that actually occurs — a table rooted at an index leaf — was reported as healthy (#496).
|
|
23
43
|
- A detached Codex quota verification can no longer make the following hook tick acknowledge success without evaluating its budget alerts. If the verifier still owns the stats ingest lock, the tick remains due and retries as soon as that lock is released (#461).
|
|
24
44
|
- Cache Report no longer warns that net cache spend is negative when savings and write costs cancel to an effectively zero floating-point residue (#471).
|
|
25
|
-
- Fix `cctally setup` npm installs repeatedly adding duplicate Codex Stop/SubagentStop hooks instead of reconciling them (#458).
|
|
45
|
+
- Fix `cctally setup` npm installs repeatedly adding duplicate Codex Stop/SubagentStop hooks instead of reconciling them (#458; reported independently by @darlingm, who also supplied a regression patch).
|
|
26
46
|
- Cache Report charts now announce one consistent window size to screen readers and identify unobserved days as a measured subset (#469).
|
|
27
47
|
- Codex quota rows restored from the journal or repaired during migration now invalidate cached dashboard and projection state immediately, instead of waiting for an unrelated later Codex mutation (#457).
|
|
28
48
|
- Codex file-path search now indexes the dict-shaped patch events the provider actually records, and upgrades backfill retained history so existing conversations become searchable by touched file (#489).
|
package/README.md
CHANGED
|
@@ -30,9 +30,11 @@ Your Claude Code plan meters you with a percentage that creeps up all week. ccta
|
|
|
30
30
|
</p>
|
|
31
31
|
|
|
32
32
|
<!-- cctally:latest-stable:begin -->
|
|
33
|
-
**Latest stable: v1.
|
|
33
|
+
**Latest stable: v1.92.0** (2026-08-05)
|
|
34
34
|
|
|
35
|
-
-
|
|
35
|
+
- Add per-run bearer authentication for non-loopback dashboards, with fragment-to-HttpOnly-cookie browser bootstrap and a restart-only Settings/CLI opt-out (#282).
|
|
36
|
+
- Add account-scoped conversation browsing, search, reading, export, and permalinks to the dashboard and `cctally transcript`, while preserving the existing all-account and single-account output shapes (#347).
|
|
37
|
+
- Long Codex conversations now open with two concurrent data requests instead of issuing a third detail request just to repeat totals already available from the outline. On the same 3,733-row conversation, the first painted row improved from 784 ms to 421, 429 ms while live-tail streaming remained connected (#477).
|
|
36
38
|
<!-- cctally:latest-stable:end -->
|
|
37
39
|
|
|
38
40
|
## Quick start
|
package/bin/_cctally_cache.py
CHANGED
|
@@ -127,6 +127,7 @@ def _cctally():
|
|
|
127
127
|
# for ``eprint`` is deleted.
|
|
128
128
|
import _cctally_core
|
|
129
129
|
from _cctally_core import eprint
|
|
130
|
+
from _lib_codex_find_projection import CODEX_FIND_PROJECTION_VERSION
|
|
130
131
|
from _lib_source_identity import source_root_key
|
|
131
132
|
# #416 spec §4.2: the pure tolerance-anchored reset kernel. `_lib_quota` imports
|
|
132
133
|
# only `_lib_accounts` (a stdlib leaf), so binding it here is circular-safe.
|
|
@@ -2076,7 +2077,8 @@ def run_codex_find_projection_backfill(
|
|
|
2076
2077
|
if pending is None:
|
|
2077
2078
|
complete = conn.execute(
|
|
2078
2079
|
"SELECT 1 FROM cache_meta "
|
|
2079
|
-
"WHERE key='codex_find_projection_complete_version' AND value
|
|
2080
|
+
"WHERE key='codex_find_projection_complete_version' AND value=?",
|
|
2081
|
+
(str(CODEX_FIND_PROJECTION_VERSION),),
|
|
2080
2082
|
).fetchone() is not None
|
|
2081
2083
|
return {"processed": 0, "complete": complete}
|
|
2082
2084
|
row = conn.execute(
|
|
@@ -2113,13 +2115,17 @@ def run_codex_find_projection_backfill(
|
|
|
2113
2115
|
complete = remaining is None
|
|
2114
2116
|
if complete:
|
|
2115
2117
|
conn.execute(
|
|
2116
|
-
"INSERT OR REPLACE INTO cache_meta(key,value) VALUES"
|
|
2117
|
-
|
|
2118
|
+
"INSERT OR REPLACE INTO cache_meta(key,value) VALUES(?,?)",
|
|
2119
|
+
(
|
|
2120
|
+
"codex_find_projection_complete_version",
|
|
2121
|
+
str(CODEX_FIND_PROJECTION_VERSION),
|
|
2122
|
+
),
|
|
2118
2123
|
)
|
|
2119
2124
|
conn.execute(
|
|
2120
2125
|
"DELETE FROM cache_meta WHERE key IN "
|
|
2121
2126
|
"('codex_find_projection_backfill_pending',"
|
|
2122
|
-
" 'codex_find_projection_backfill_cursor'
|
|
2127
|
+
" 'codex_find_projection_backfill_cursor',"
|
|
2128
|
+
" 'codex_find_projection_backfill_version')"
|
|
2123
2129
|
)
|
|
2124
2130
|
conn.commit()
|
|
2125
2131
|
return {"processed": len(selected), "complete": complete}
|
|
@@ -8699,7 +8705,41 @@ def scope_conversations_db_to_account(
|
|
|
8699
8705
|
)
|
|
8700
8706
|
if row[0]
|
|
8701
8707
|
}
|
|
8708
|
+
safe_project_attribution: dict[str, tuple[str | None, str | None]] = {}
|
|
8709
|
+
for conversation_key in codex_keys:
|
|
8710
|
+
persisted = conn.execute(
|
|
8711
|
+
"SELECT project_key,project_label "
|
|
8712
|
+
"FROM main.codex_conversation_rollups WHERE conversation_key=?",
|
|
8713
|
+
(conversation_key,),
|
|
8714
|
+
).fetchone()
|
|
8715
|
+
if persisted is not None:
|
|
8716
|
+
safe_project_attribution[conversation_key] = persisted
|
|
8717
|
+
continue
|
|
8718
|
+
thread = conn.execute(
|
|
8719
|
+
"SELECT source_root_key,cwd,git_json "
|
|
8720
|
+
"FROM cache_db.codex_conversation_threads WHERE conversation_key=?",
|
|
8721
|
+
(conversation_key,),
|
|
8722
|
+
).fetchone()
|
|
8723
|
+
if thread is not None:
|
|
8724
|
+
safe_project_attribution[conversation_key] = (
|
|
8725
|
+
_codex_conversation_project_attribution(*thread)
|
|
8726
|
+
)
|
|
8702
8727
|
_recompute_codex_rollups(conn, codex_keys)
|
|
8728
|
+
# Project identity is safe conversation-level enrichment: it is already
|
|
8729
|
+
# visible on the unqualified rail and contains only an opaque key plus the
|
|
8730
|
+
# derived display label. Preserve those two fields for conversations that
|
|
8731
|
+
# survived the physical-row account predicate, without exposing the
|
|
8732
|
+
# conversation-level cwd/git, source-root path, title, or thread topology
|
|
8733
|
+
# that produced them (#497).
|
|
8734
|
+
conn.executemany(
|
|
8735
|
+
"UPDATE codex_conversation_rollups SET project_key=?,project_label=? "
|
|
8736
|
+
"WHERE conversation_key=?",
|
|
8737
|
+
[
|
|
8738
|
+
(project_key, project_label, conversation_key)
|
|
8739
|
+
for conversation_key, (project_key, project_label)
|
|
8740
|
+
in safe_project_attribution.items()
|
|
8741
|
+
],
|
|
8742
|
+
)
|
|
8703
8743
|
# TEMP rollup writes open a transaction. Close it before a long-lived
|
|
8704
8744
|
# account-scoped SSE reader starts watching so later provider-writer commits
|
|
8705
8745
|
# are visible to the dynamic leaf views and never contend on this setup work.
|
package/bin/_cctally_core.py
CHANGED
|
@@ -367,7 +367,15 @@ STATS_WAL_SIZE_LIMIT_BYTES = 16 * 1024 * 1024 # 16777216
|
|
|
367
367
|
# 1006 -> 1007 (#460): scheduled quota-alert ownership. Adds the per-root
|
|
368
368
|
# future-capture schedule that lets a matured boundary widen to its owning root
|
|
369
369
|
# instead of deferring forever on a quiet hook-only install.
|
|
370
|
-
|
|
370
|
+
# 1007 -> 1008 (#496 S3): in-place transactional publication. Adds
|
|
371
|
+
# `stats_publication_stamp`, the publication identity written inside the
|
|
372
|
+
# publication transaction. It replaces the marker's `scratchPath` crash
|
|
373
|
+
# discriminator, which an in-place publish inverts because it attaches the
|
|
374
|
+
# scratch read-only and leaves it on disk whether the transaction committed or
|
|
375
|
+
# rolled back. A stats schema change is an epoch bump and never a migration;
|
|
376
|
+
# the 13-migration registry stays frozen. Each install pays one rebuild on
|
|
377
|
+
# upgrade, deferred to the background worker by #453.
|
|
378
|
+
STATS_INDEX_EPOCH = 1008
|
|
371
379
|
LEGACY_STATS_HEAD = 13
|
|
372
380
|
|
|
373
381
|
#: #496 S1 F1. A NEW branch, for a state that cannot occur before the
|
|
@@ -384,6 +392,21 @@ STATS_PUBLICATION_FAILED_MSG = (
|
|
|
384
392
|
"logs/. Recovery: run `cctally db rebuild --db stats`."
|
|
385
393
|
)
|
|
386
394
|
|
|
395
|
+
#: #496 S3. The text above describes PHYSICAL replacement, which is now the
|
|
396
|
+
#: fallback. In-place publication drops the live generation and installs the
|
|
397
|
+
#: scratch's inside one transaction, so it preserves nothing and allocates no
|
|
398
|
+
#: quarantine directory — and the sentence about a preserved predecessor would
|
|
399
|
+
#: send a user whose index is already known bad to a directory that does not
|
|
400
|
+
#: exist. Selected by the mechanism the publication marker records.
|
|
401
|
+
STATS_PUBLICATION_FAILED_IN_PLACE_MSG = (
|
|
402
|
+
"stats.db published a rebuilt index that then FAILED validation, so the "
|
|
403
|
+
"live index is known bad and cctally refuses to use it. path: {path}. "
|
|
404
|
+
"The rebuild record naming the failing check is at {record}. This index "
|
|
405
|
+
"was published in place, so no copy of the previous index was kept; every "
|
|
406
|
+
"row it holds is derived from the append-only journal, which the "
|
|
407
|
+
"publication did not touch. Recovery: run `cctally db rebuild --db stats`."
|
|
408
|
+
)
|
|
409
|
+
|
|
387
410
|
|
|
388
411
|
# === Telemetry constants (non-path; see spec 2026-07-07) =============
|
|
389
412
|
#
|
|
@@ -722,18 +745,27 @@ def ensure_dirs() -> None:
|
|
|
722
745
|
# before returning does not, and must not — see `stats_open_guarded`):
|
|
723
746
|
# bin/_cctally_journal.py _acquire_maintenance_{shared,exclusive} / _release
|
|
724
747
|
# bin/_cctally_store.py _heal_flock_blocking, reached through
|
|
725
|
-
# _acquire_stats_maintenance_reentrant by the
|
|
726
|
-
#
|
|
748
|
+
# _acquire_stats_maintenance_reentrant by the epoch
|
|
749
|
+
# resolver
|
|
750
|
+
# bin/_cctally_store.py _acquire_stats_maintenance_for_heal, the corruption
|
|
751
|
+
# heal's ownership-first BOUNDED acquire (#496 S3)
|
|
727
752
|
# bin/_cctally_db.py cmd_db_rebuild, _acquire_db_admin_writer_flocks
|
|
728
753
|
# (db skip / db unskip), _cmd_db_repair_exclusive,
|
|
729
754
|
# _vacuum_one_db
|
|
730
755
|
# bin/_cctally_rederive.py _rederive_locks
|
|
756
|
+
# bin/_cctally_store.py stats_open_guarded's interrupted-rebuild-recovery
|
|
757
|
+
# branch, which upgrades to EXCLUSIVE and then calls
|
|
758
|
+
# rebuild_stats_index (#496 S3)
|
|
731
759
|
# Adding another acquisition site without noting it here reintroduces the hang.
|
|
732
760
|
#
|
|
733
|
-
# The opener (`_cctally_store.stats_open_guarded`) takes the lock SHARED
|
|
734
|
-
# releases it before handing the connection back, so
|
|
735
|
-
# note a hold — but it DOES consult
|
|
736
|
-
# acquire entirely when this context
|
|
761
|
+
# The opener (`_cctally_store.stats_open_guarded`) takes the lock SHARED around
|
|
762
|
+
# an ordinary open and releases it before handing the connection back, so that
|
|
763
|
+
# acquire deliberately does NOT note a hold — but it DOES consult
|
|
764
|
+
# `holds_stats_maintenance()` to skip the acquire entirely when this context
|
|
765
|
+
# already owns the exclusive side. Its interrupted-rebuild-recovery branch is
|
|
766
|
+
# the exception: that one upgrades to EXCLUSIVE and holds it across a rebuild,
|
|
767
|
+
# whose in-place publisher opens the destination through `stats_open_guarded`
|
|
768
|
+
# again, so it notes the hold like every other exclusive site.
|
|
737
769
|
|
|
738
770
|
_STATS_MAINTENANCE_HELD = contextvars.ContextVar(
|
|
739
771
|
"cctally_stats_maintenance_held", default=0
|
|
@@ -2475,6 +2507,23 @@ def open_db(*, _target_path=None) -> sqlite3.Connection:
|
|
|
2475
2507
|
"kind TEXT NOT NULL, "
|
|
2476
2508
|
"violation_json TEXT NOT NULL)"
|
|
2477
2509
|
)
|
|
2510
|
+
# In-place publication identity (#496 S3 §5). An in-place publish
|
|
2511
|
+
# attaches the scratch read-only and detaches it, so the scratch
|
|
2512
|
+
# survives commit and rollback identically and the publication marker's
|
|
2513
|
+
# `scratchPath` proxy inverts. This row is written INSIDE the
|
|
2514
|
+
# publication transaction, so it commits atomically with the content
|
|
2515
|
+
# and the `user_version` it describes and a crash before the commit
|
|
2516
|
+
# rolls it back. The opener compares it against the marker's
|
|
2517
|
+
# `recordPath` and knows without inference whether that publication
|
|
2518
|
+
# committed. Holds at most one row; single-row-ness is deliberately not
|
|
2519
|
+
# enforced structurally, because a duplicated row is one of the states
|
|
2520
|
+
# that must resolve INDETERMINATE rather than be made impossible.
|
|
2521
|
+
conn.execute(
|
|
2522
|
+
"CREATE TABLE IF NOT EXISTS stats_publication_stamp ("
|
|
2523
|
+
"record_path TEXT NOT NULL, "
|
|
2524
|
+
"started_at_utc TEXT NOT NULL, "
|
|
2525
|
+
"stamped_at_utc TEXT NOT NULL)"
|
|
2526
|
+
)
|
|
2478
2527
|
|
|
2479
2528
|
# §6.2 backfill gate (Task 8): stamp the one-shot marker AFTER the three
|
|
2480
2529
|
# open-time backfills ran, so the next open skips them (and their probes)
|
|
@@ -4353,12 +4353,40 @@ def _debug_source_state_wire(bundle, source: str) -> dict:
|
|
|
4353
4353
|
}
|
|
4354
4354
|
|
|
4355
4355
|
|
|
4356
|
-
def
|
|
4356
|
+
def _debug_stats_fault(exc: BaseException, leg: str) -> dict:
|
|
4357
|
+
"""The wire form of a stats attribution these debug reads may report.
|
|
4358
|
+
|
|
4359
|
+
#496 S3 §8 records a DELIBERATE deviation from F16's literal wording. F16
|
|
4360
|
+
asks that every path meeting a corrupt index produce a typed attribution
|
|
4361
|
+
AND reach the heal. The two debug helpers deliver the attribution WITHOUT
|
|
4362
|
+
the heal: they are short-lived diagnostic reads that bypass the corruption
|
|
4363
|
+
boundary for cost, and making a debug endpoint able to trigger a rebuild is
|
|
4364
|
+
a worse outcome than making it honest. The dashboard's main build path
|
|
4365
|
+
already reaches the heal, so a fault attributed here is healed on the next
|
|
4366
|
+
tick.
|
|
4367
|
+
|
|
4368
|
+
The shape is exactly the mapping `_sync_failure_envelope`'s attribution
|
|
4369
|
+
reader already accepts, so this travels on the established vocabulary
|
|
4370
|
+
rather than inventing a second one. No database probe is needed: the
|
|
4371
|
+
connection these helpers were opening is the stats index by construction.
|
|
4372
|
+
"""
|
|
4373
|
+
return {
|
|
4374
|
+
"leg": leg,
|
|
4375
|
+
"database": "stats",
|
|
4376
|
+
"corruption": bool(_cctally()._is_sqlite_corruption_error(exc)),
|
|
4377
|
+
}
|
|
4378
|
+
|
|
4379
|
+
|
|
4380
|
+
def _debug_source_counts(cache_conn, bundle, *, faults=None) -> dict:
|
|
4357
4381
|
"""Bounded, source-owned counts and opaque state for the debug endpoint.
|
|
4358
4382
|
|
|
4359
4383
|
Every table and predicate is fixed here. This deliberately reports no
|
|
4360
4384
|
values from rows: roots, paths, logical limits, conversation IDs, and
|
|
4361
4385
|
project labels never cross the diagnostic boundary.
|
|
4386
|
+
|
|
4387
|
+
``faults`` collects `_debug_stats_fault` mappings for a stats read this
|
|
4388
|
+
diagnostic could not complete, so the endpoint reports the fault instead of
|
|
4389
|
+
silently degrading a stats failure into a missing count.
|
|
4362
4390
|
"""
|
|
4363
4391
|
result = {
|
|
4364
4392
|
source: _debug_source_state_wire(bundle, source)
|
|
@@ -4388,8 +4416,9 @@ def _debug_source_counts(cache_conn, bundle) -> dict:
|
|
|
4388
4416
|
result[source]["tables"][table] = int(row[0])
|
|
4389
4417
|
except sqlite3.Error:
|
|
4390
4418
|
pass
|
|
4391
|
-
except sqlite3.Error:
|
|
4392
|
-
|
|
4419
|
+
except sqlite3.Error as exc:
|
|
4420
|
+
if faults is not None:
|
|
4421
|
+
faults.append(_debug_stats_fault(exc, "debug-source-counts"))
|
|
4393
4422
|
finally:
|
|
4394
4423
|
if stats_conn is not None:
|
|
4395
4424
|
stats_conn.close()
|
|
@@ -4419,7 +4448,7 @@ def _stats_ro_guarded():
|
|
|
4419
4448
|
)
|
|
4420
4449
|
|
|
4421
4450
|
|
|
4422
|
-
def _debug_cache_state(cache_conn) -> dict:
|
|
4451
|
+
def _debug_cache_state(cache_conn, *, faults=None) -> dict:
|
|
4423
4452
|
"""On-demand signature legs + pending-reingest flags + generation.
|
|
4424
4453
|
|
|
4425
4454
|
The signature legs are the canonical ``compute_signature`` fields (ints /
|
|
@@ -4433,8 +4462,10 @@ def _debug_cache_state(cache_conn) -> dict:
|
|
|
4433
4462
|
stats_conn = None
|
|
4434
4463
|
try:
|
|
4435
4464
|
stats_conn = _stats_ro_guarded()
|
|
4436
|
-
except sqlite3.Error:
|
|
4465
|
+
except sqlite3.Error as exc:
|
|
4437
4466
|
stats_conn = None
|
|
4467
|
+
if faults is not None:
|
|
4468
|
+
faults.append(_debug_stats_fault(exc, "debug-cache-state"))
|
|
4438
4469
|
try:
|
|
4439
4470
|
if stats_conn is not None:
|
|
4440
4471
|
sig = sc.compute_signature(
|
|
@@ -5061,13 +5092,16 @@ class DashboardHTTPHandler(BaseHTTPRequestHandler):
|
|
|
5061
5092
|
source_bundle = self.snapshot_ref.get().source_bundle
|
|
5062
5093
|
except Exception: # noqa: BLE001 -- diagnostics fail closed.
|
|
5063
5094
|
source_bundle = None
|
|
5064
|
-
|
|
5095
|
+
stats_faults: list[dict] = []
|
|
5096
|
+
sources = _debug_source_counts(None, source_bundle, faults=stats_faults)
|
|
5065
5097
|
try:
|
|
5066
5098
|
conn = open_cache_db()
|
|
5067
5099
|
try:
|
|
5068
5100
|
dataset = _debug_cache_table_counts(conn)
|
|
5069
|
-
cache_state = _debug_cache_state(conn)
|
|
5070
|
-
sources = _debug_source_counts(
|
|
5101
|
+
cache_state = _debug_cache_state(conn, faults=stats_faults)
|
|
5102
|
+
sources = _debug_source_counts(
|
|
5103
|
+
conn, source_bundle, faults=stats_faults
|
|
5104
|
+
)
|
|
5071
5105
|
finally:
|
|
5072
5106
|
conn.close()
|
|
5073
5107
|
except Exception: # noqa: BLE001 -- a diagnostic must not expose raw errors.
|
|
@@ -5080,6 +5114,11 @@ class DashboardHTTPHandler(BaseHTTPRequestHandler):
|
|
|
5080
5114
|
"phases": (last or {}).get("phases"),
|
|
5081
5115
|
"cache_state": cache_state,
|
|
5082
5116
|
"sources": sources,
|
|
5117
|
+
# Additive, and named rather than folded into `cache_state`: a
|
|
5118
|
+
# stats fault is not cache state, and #496 S3 §8 exists because a
|
|
5119
|
+
# stats failure reported as a cache one sends the user to
|
|
5120
|
+
# `cctally cache-sync --rebuild`.
|
|
5121
|
+
"stats_faults": stats_faults,
|
|
5083
5122
|
}
|
|
5084
5123
|
if body["phases"] is None:
|
|
5085
5124
|
body["note"] = "tracing_disabled"
|
|
@@ -7033,15 +7072,28 @@ def _dashboard_initial_snapshot(args, *, pinned_now, display_tz_pref_override):
|
|
|
7033
7072
|
display_tz_pref_override=display_tz_pref_override,
|
|
7034
7073
|
stats_heal_attempted=False,
|
|
7035
7074
|
)
|
|
7036
|
-
except c.
|
|
7037
|
-
return
|
|
7075
|
+
except c.StatsRebuildDeferred as exc:
|
|
7076
|
+
return _dashboard_stats_deferred_snapshot(
|
|
7038
7077
|
args,
|
|
7039
7078
|
pinned_now=pinned_now,
|
|
7040
7079
|
exc=exc,
|
|
7041
7080
|
)
|
|
7042
7081
|
except tui._StatsSnapshotCorruption as fault:
|
|
7043
7082
|
# The once-builder's finally has closed the cheap-seed stats handle.
|
|
7044
|
-
|
|
7083
|
+
#
|
|
7084
|
+
# #496 S3 §6: this call is made from INSIDE an `except` handler, so a
|
|
7085
|
+
# `BaseException` raised by the heal would escape past the sibling
|
|
7086
|
+
# `except` above and fail dashboard startup instead of hydrating it.
|
|
7087
|
+
# The shared deferral base is therefore caught AT the heal-call
|
|
7088
|
+
# boundary, and the correctly typed degraded frame is built directly.
|
|
7089
|
+
try:
|
|
7090
|
+
tui._tui_heal_post_query_stats(fault.cause)
|
|
7091
|
+
except c.StatsRebuildDeferred as deferred:
|
|
7092
|
+
return _dashboard_stats_deferred_snapshot(
|
|
7093
|
+
args,
|
|
7094
|
+
pinned_now=pinned_now,
|
|
7095
|
+
exc=deferred,
|
|
7096
|
+
)
|
|
7045
7097
|
return _dashboard_initial_snapshot_once(
|
|
7046
7098
|
args,
|
|
7047
7099
|
pinned_now=pinned_now,
|
|
@@ -7050,8 +7102,13 @@ def _dashboard_initial_snapshot(args, *, pinned_now, display_tz_pref_override):
|
|
|
7050
7102
|
)
|
|
7051
7103
|
|
|
7052
7104
|
|
|
7053
|
-
def
|
|
7054
|
-
"""Bind promptly with a typed degraded frame while
|
|
7105
|
+
def _dashboard_stats_deferred_snapshot(args, *, pinned_now, exc):
|
|
7106
|
+
"""Bind promptly with a typed degraded frame while a rebuild runs detached.
|
|
7107
|
+
|
|
7108
|
+
Serves both deferral classes and keeps them distinct: a wrong EPOCH is a
|
|
7109
|
+
readable index, while a deferred corruption heal is an index that could not
|
|
7110
|
+
be read, and the attribution must say which (#496 S3 §6).
|
|
7111
|
+
"""
|
|
7055
7112
|
|
|
7056
7113
|
import time as _time
|
|
7057
7114
|
|
|
@@ -7085,7 +7142,7 @@ def _dashboard_stats_epoch_deferred_snapshot(args, *, pinned_now, exc):
|
|
|
7085
7142
|
tui.SyncFailureAttribution(
|
|
7086
7143
|
leg="stats-open",
|
|
7087
7144
|
database="stats",
|
|
7088
|
-
corruption=
|
|
7145
|
+
corruption=isinstance(exc, c.StatsHealDeferred),
|
|
7089
7146
|
),
|
|
7090
7147
|
),
|
|
7091
7148
|
"doctor_payload": doctor_payload,
|
|
@@ -223,12 +223,12 @@ def _conversation_query_impl():
|
|
|
223
223
|
# are deliberately absent — they are meaningless on a collection route, so they
|
|
224
224
|
# fall to "genuinely unknown → ignored".
|
|
225
225
|
_RECOGNIZED_CONVERSATION_PARAMS = (
|
|
226
|
-
"source", "account", "project_key", "model", "limit", "cursor", "q", "kind",
|
|
226
|
+
"source", "account", "project_key", "model", "limit", "cursor", "selected", "q", "kind",
|
|
227
227
|
"sort", "offset", "date_from", "date_to", "projects",
|
|
228
228
|
"cost_min", "cost_max", "rebuild_min", "models",
|
|
229
229
|
)
|
|
230
230
|
_QUALIFIED_BROWSE_ACCEPTED = (
|
|
231
|
-
"source", "account", "project_key", "model", "limit", "cursor")
|
|
231
|
+
"source", "account", "project_key", "model", "limit", "cursor", "selected")
|
|
232
232
|
_QUALIFIED_SEARCH_ACCEPTED = (
|
|
233
233
|
"source", "account", "q", "kind", "limit", "cursor")
|
|
234
234
|
_QUALIFIED_FACETS_ACCEPTED = ("source", "account")
|
|
@@ -364,12 +364,14 @@ def _handle_qualified_browse(handler, qs_raw, source):
|
|
|
364
364
|
return
|
|
365
365
|
project_key = (parsed.get("project_key", [None])[0] or None)
|
|
366
366
|
model = (parsed.get("model", [None])[0] or None)
|
|
367
|
+
selected = (parsed.get("selected", [None])[0] or None)
|
|
367
368
|
speed = _resolve_effective_speed()
|
|
368
369
|
disp = _conversation_dispatch()
|
|
369
370
|
ok, body = handler._run_conversation_query(
|
|
370
371
|
lambda conn: disp.neutral_browse(
|
|
371
372
|
conn, source=source, effective_speed=speed,
|
|
372
|
-
project_key=project_key, model=model, limit=limit, cursor=cursor
|
|
373
|
+
project_key=project_key, model=model, limit=limit, cursor=cursor,
|
|
374
|
+
selected=selected),
|
|
373
375
|
"/api/conversations")
|
|
374
376
|
if not ok:
|
|
375
377
|
return
|
|
@@ -604,12 +606,14 @@ def _handle_get_conversations_impl(handler) -> None:
|
|
|
604
606
|
sort = _qs_str(q, "sort", "recent")
|
|
605
607
|
limit = _qs_int(q, "limit", 50)
|
|
606
608
|
offset = _qs_int(q, "offset", 0)
|
|
609
|
+
selected = _qs_str(q, "selected", None)
|
|
607
610
|
filters = handler._parse_conversation_filters(q)
|
|
608
611
|
if filters is None:
|
|
609
612
|
return # a 400 has already been sent
|
|
610
613
|
ok, body = handler._run_conversation_query(
|
|
611
614
|
lambda conn: handler._conversation_query().list_conversations(
|
|
612
|
-
conn, sort=sort, limit=limit, offset=offset,
|
|
615
|
+
conn, sort=sort, limit=limit, offset=offset, selected=selected,
|
|
616
|
+
**filters),
|
|
613
617
|
"/api/conversations")
|
|
614
618
|
if not ok:
|
|
615
619
|
return
|
package/bin/_cctally_db.py
CHANGED
|
@@ -99,6 +99,7 @@ from _cctally_core import (
|
|
|
99
99
|
# the migration gate-defer diagnostic routes through _lib_log so
|
|
100
100
|
# CCTALLY_DEBUG verbosity is decided in one place.
|
|
101
101
|
import _lib_log
|
|
102
|
+
from _lib_codex_find_projection import CODEX_FIND_PROJECTION_VERSION
|
|
102
103
|
|
|
103
104
|
|
|
104
105
|
# Production cache dispatchers hold maintenance-exclusive + the global cache
|
|
@@ -312,31 +313,84 @@ class StatsEpochMismatchError(sqlite3.DatabaseError):
|
|
|
312
313
|
DB failure; ``main()`` maps it to a staged exit 3."""
|
|
313
314
|
|
|
314
315
|
|
|
315
|
-
class
|
|
316
|
-
"""A
|
|
316
|
+
class StatsRebuildDeferred(BaseException):
|
|
317
|
+
"""A stats.db rebuild this caller must not perform inline runs detached.
|
|
318
|
+
|
|
319
|
+
The shared parent of both deferral signals (#496 S3 §6), so every catch
|
|
320
|
+
site is widened ONCE rather than growing a second name each time a rebuild
|
|
321
|
+
class is detached. ``outcome`` records whether this caller spawned the
|
|
322
|
+
worker, observed an existing attempt, or could not spawn it.
|
|
317
323
|
|
|
318
|
-
Ordinary callers must not read the schema-incompatible old index or pay
|
|
319
|
-
whole-journal replay latency inline. ``outcome`` records whether this
|
|
320
|
-
caller spawned the worker, observed an existing attempt, or could not
|
|
321
|
-
spawn it; ``main()`` maps every case to prompt retry guidance and exit 3.
|
|
322
324
|
This deliberately derives directly from ``BaseException``: reporting
|
|
323
325
|
kernels contain many broad ``Exception`` / ``sqlite3.DatabaseError``
|
|
324
326
|
fallbacks that turn missing optional data into ``n/a``. Swallowing this
|
|
325
327
|
control signal there would publish a misleading partial report. The CLI
|
|
326
328
|
boundary, dashboard, and statusline catch it explicitly; ``finally``
|
|
327
329
|
cleanup still runs normally.
|
|
330
|
+
|
|
331
|
+
The two subclasses are NOT interchangeable at the degrade sites: the epoch
|
|
332
|
+
path describes a readable index at the wrong version and reports
|
|
333
|
+
``corruption=False``, while a deferred heal describes an index that could
|
|
334
|
+
not be read and must report ``corruption=True``. Flattening them makes the
|
|
335
|
+
dashboard name the wrong fault.
|
|
328
336
|
"""
|
|
329
337
|
|
|
330
|
-
def __init__(self, outcome: str) -> None:
|
|
338
|
+
def __init__(self, outcome: str, message: str) -> None:
|
|
331
339
|
self.outcome = str(outcome)
|
|
340
|
+
super().__init__(message)
|
|
341
|
+
|
|
342
|
+
|
|
343
|
+
class StatsEpochRebuildDeferred(StatsRebuildDeferred):
|
|
344
|
+
"""A readable wrong-epoch stats index is rebuilding out of process.
|
|
345
|
+
|
|
346
|
+
Ordinary callers must not read the schema-incompatible old index or pay
|
|
347
|
+
whole-journal replay latency inline; ``main()`` maps every case to prompt
|
|
348
|
+
retry guidance and exit 3.
|
|
349
|
+
"""
|
|
350
|
+
|
|
351
|
+
def __init__(self, outcome: str) -> None:
|
|
352
|
+
outcome = str(outcome)
|
|
332
353
|
message = (
|
|
333
354
|
"could not start the stats.db index epoch rebuild; retry this "
|
|
334
355
|
"command shortly"
|
|
335
|
-
if
|
|
356
|
+
if outcome == "failed"
|
|
336
357
|
else "stats.db index epoch rebuild is running in the background; "
|
|
337
358
|
"retry shortly"
|
|
338
359
|
)
|
|
339
|
-
super().__init__(message)
|
|
360
|
+
super().__init__(outcome, message)
|
|
361
|
+
|
|
362
|
+
|
|
363
|
+
class StatsHealDeferred(StatsRebuildDeferred):
|
|
364
|
+
"""A corrupt stats index is being rebuilt by the detached heal worker.
|
|
365
|
+
|
|
366
|
+
Replaces the inline heal's return value on the path where it used to
|
|
367
|
+
quarantine and rebuild while the caller waited (#496 S3 §6). The caller
|
|
368
|
+
degrades where it already degrades; it never blocks on the rebuild.
|
|
369
|
+
|
|
370
|
+
``heal_id`` correlates this signal with the durable heal event the hook
|
|
371
|
+
recorded at detection, and ``forensics_path`` is the absolute bundle the
|
|
372
|
+
user was told about. Both are attributes rather than message text so a
|
|
373
|
+
consumer can use them without parsing.
|
|
374
|
+
"""
|
|
375
|
+
|
|
376
|
+
def __init__(
|
|
377
|
+
self,
|
|
378
|
+
outcome: str,
|
|
379
|
+
*,
|
|
380
|
+
heal_id: "str | None" = None,
|
|
381
|
+
forensics_path: "str | None" = None,
|
|
382
|
+
) -> None:
|
|
383
|
+
outcome = str(outcome)
|
|
384
|
+
self.heal_id = heal_id
|
|
385
|
+
self.forensics_path = forensics_path
|
|
386
|
+
message = (
|
|
387
|
+
"could not start the stats.db corruption rebuild; retry this "
|
|
388
|
+
"command shortly"
|
|
389
|
+
if outcome == "failed"
|
|
390
|
+
else "stats.db corruption rebuild is running in the background; "
|
|
391
|
+
"retry shortly"
|
|
392
|
+
)
|
|
393
|
+
super().__init__(outcome, message)
|
|
340
394
|
|
|
341
395
|
|
|
342
396
|
_SQLITE_CORRUPTION_MESSAGES = (
|
|
@@ -4407,7 +4461,11 @@ def _apply_codex_find_projection_schema(conn: sqlite3.Connection) -> None:
|
|
|
4407
4461
|
if has_state is None and conn.execute(
|
|
4408
4462
|
"SELECT 1 FROM codex_conversation_messages LIMIT 1"
|
|
4409
4463
|
).fetchone() is None:
|
|
4410
|
-
_set_cache_meta(
|
|
4464
|
+
_set_cache_meta(
|
|
4465
|
+
conn,
|
|
4466
|
+
"codex_find_projection_complete_version",
|
|
4467
|
+
str(CODEX_FIND_PROJECTION_VERSION),
|
|
4468
|
+
)
|
|
4411
4469
|
_set_cache_meta(conn, "codex_find_projection_generation", "0")
|
|
4412
4470
|
|
|
4413
4471
|
|
|
@@ -4561,7 +4619,8 @@ def _conv_004_codex_find_projection(conn: sqlite3.Connection) -> None:
|
|
|
4561
4619
|
_apply_codex_find_projection_schema(conn)
|
|
4562
4620
|
complete = conn.execute(
|
|
4563
4621
|
"SELECT 1 FROM cache_meta "
|
|
4564
|
-
"WHERE key='codex_find_projection_complete_version' AND value
|
|
4622
|
+
"WHERE key='codex_find_projection_complete_version' AND value=?",
|
|
4623
|
+
(str(CODEX_FIND_PROJECTION_VERSION),),
|
|
4565
4624
|
).fetchone()
|
|
4566
4625
|
if complete is None:
|
|
4567
4626
|
_set_cache_meta(conn, "codex_find_projection_backfill_pending", "1")
|
|
@@ -4569,6 +4628,11 @@ def _conv_004_codex_find_projection(conn: sqlite3.Connection) -> None:
|
|
|
4569
4628
|
"INSERT OR IGNORE INTO cache_meta(key,value) VALUES"
|
|
4570
4629
|
"('codex_find_projection_backfill_cursor','0')"
|
|
4571
4630
|
)
|
|
4631
|
+
_set_cache_meta(
|
|
4632
|
+
conn,
|
|
4633
|
+
"codex_find_projection_backfill_version",
|
|
4634
|
+
str(CODEX_FIND_PROJECTION_VERSION),
|
|
4635
|
+
)
|
|
4572
4636
|
_set_cache_meta(conn, "codex_find_projection_generation", "0")
|
|
4573
4637
|
conn.commit()
|
|
4574
4638
|
finally:
|
|
@@ -4678,6 +4742,42 @@ def _conv_006_backfill_codex_file_touches(conn: sqlite3.Connection) -> None:
|
|
|
4678
4742
|
_release_cache_db_writer_flocks(held)
|
|
4679
4743
|
|
|
4680
4744
|
|
|
4745
|
+
@conversations_migration("007_codex_find_projection_v2_meta")
|
|
4746
|
+
def _conv_007_codex_find_projection_v2_meta(conn: sqlite3.Connection) -> None:
|
|
4747
|
+
"""Arm a bounded v2 rebuild so retained visible meta bodies become findable.
|
|
4748
|
+
|
|
4749
|
+
The projection is disposable. Preserve a markerless retry's v2 cursor, but
|
|
4750
|
+
restart a v1/incompletely-versioned backfill at zero so no conversation
|
|
4751
|
+
already passed by that older walk keeps stale rows.
|
|
4752
|
+
"""
|
|
4753
|
+
held = _acquire_conversations_db_codex_provider_flock(
|
|
4754
|
+
conn, migration="conversations 007 Codex find projection v2 meta")
|
|
4755
|
+
try:
|
|
4756
|
+
target = str(CODEX_FIND_PROJECTION_VERSION)
|
|
4757
|
+
complete = conn.execute(
|
|
4758
|
+
"SELECT 1 FROM cache_meta "
|
|
4759
|
+
"WHERE key='codex_find_projection_complete_version' AND value=?",
|
|
4760
|
+
(target,),
|
|
4761
|
+
).fetchone()
|
|
4762
|
+
if complete is not None:
|
|
4763
|
+
return
|
|
4764
|
+
pending_version = conn.execute(
|
|
4765
|
+
"SELECT value FROM cache_meta "
|
|
4766
|
+
"WHERE key='codex_find_projection_backfill_version'"
|
|
4767
|
+
).fetchone()
|
|
4768
|
+
_set_cache_meta(conn, "codex_find_projection_backfill_pending", "1")
|
|
4769
|
+
if pending_version is None or pending_version[0] != target:
|
|
4770
|
+
_set_cache_meta(conn, "codex_find_projection_backfill_cursor", "0")
|
|
4771
|
+
_set_cache_meta(conn, "codex_find_projection_backfill_version", target)
|
|
4772
|
+
conn.execute(
|
|
4773
|
+
"DELETE FROM cache_meta "
|
|
4774
|
+
"WHERE key='codex_find_projection_complete_version'"
|
|
4775
|
+
)
|
|
4776
|
+
conn.commit()
|
|
4777
|
+
finally:
|
|
4778
|
+
_release_cache_db_writer_flocks(held)
|
|
4779
|
+
|
|
4780
|
+
|
|
4681
4781
|
# #177 S6: the consolidated multi-column external-content FTS5 table that
|
|
4682
4782
|
# replaces the old conversation_fts(text) + conversation_fts_aux(search_aux)
|
|
4683
4783
|
# pair. The three column names MUST match the conversation_messages columns BY
|