cctally 1.92.2 → 1.93.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (36) hide show
  1. package/CHANGELOG.md +43 -0
  2. package/bin/_cctally_cache.py +354 -0
  3. package/bin/_cctally_core.py +180 -3
  4. package/bin/_cctally_dashboard.py +71 -1
  5. package/bin/_cctally_dashboard_envelope.py +28 -2
  6. package/bin/_cctally_dashboard_share.py +75 -19
  7. package/bin/_cctally_dashboard_sources.py +12 -0
  8. package/bin/_cctally_db.py +89 -1
  9. package/bin/_cctally_doctor.py +31 -0
  10. package/bin/_cctally_forecast.py +4 -2
  11. package/bin/_cctally_journal.py +3482 -258
  12. package/bin/_cctally_journal_repair.py +123 -32
  13. package/bin/_cctally_milestone_history.py +4 -1
  14. package/bin/_cctally_project.py +8 -6
  15. package/bin/_cctally_quota.py +420 -20
  16. package/bin/_cctally_rederive.py +57 -23
  17. package/bin/_cctally_reporting.py +8 -6
  18. package/bin/_cctally_share.py +74 -37
  19. package/bin/_cctally_source_analytics.py +6 -8
  20. package/bin/_cctally_store.py +13 -2
  21. package/bin/_cctally_tui.py +53 -0
  22. package/bin/_lib_cache_coverage.py +547 -0
  23. package/bin/_lib_doctor.py +54 -2
  24. package/bin/_lib_journal.py +235 -95
  25. package/bin/_lib_journal_router.py +21 -0
  26. package/bin/_lib_segment_summary.py +374 -0
  27. package/bin/_lib_selector_state.py +959 -0
  28. package/bin/_lib_share.py +1073 -165
  29. package/bin/_lib_share_templates.py +35 -11
  30. package/bin/_lib_stats_wal.py +327 -0
  31. package/bin/_lib_view_models.py +2 -1
  32. package/dashboard/static/assets/index-DwWJOYxd.css +1 -0
  33. package/dashboard/static/assets/{index-Dat-mza6.js → index-HlIK7k8Q.js} +47 -47
  34. package/dashboard/static/dashboard.html +2 -2
  35. package/package.json +5 -1
  36. package/dashboard/static/assets/index-DnWdv8um.css +0 -1
package/CHANGELOG.md CHANGED
@@ -5,6 +5,49 @@ based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
5
5
 
6
6
  ## [Unreleased]
7
7
 
8
+ ## [1.93.0] - 2026-08-07
9
+
10
+ ### Added
11
+ - The dashboard's share modal now states, above the preview, what the export will contain. With "Anonymize project names on export" checked it reads "Preview shows real names. Export is anonymized."; unchecked it reads "Export will show real project names." On a report that contains no project names at all — Trend and Forecast, and any other report whose data has none — it instead reads "This export contains no project names.", which is true whichever way the checkbox is set. The line is always present and changes text rather than appearing and disappearing, so a screen reader announces the new state when you toggle the checkbox. The preview pane itself still always shows real names, which is what makes the line necessary: until now the checkbox changed nothing you could see (#503).
12
+
13
+ ### Changed
14
+ - A share artifact that would disclose an identifier the documentation promises it will not is now **refused** rather than produced. The check runs over the finished document and looks for session ids and other canonical UUIDs, conversation identity keys, absolute paths and `~`/`$HOME` expansions, email addresses, source-root fingerprints, quota identity keys, and credential shapes. On the command line the refusal prints `cctally: refused to write a share artifact — share artifact would disclose: <what it found> (<the offending value>)` and exits 3, and no file is written; in the dashboard it surfaces as an export error. Naming the value matters because one known case has no way around it: if a project directory is itself named as a bare hexadecimal token or a UUID, showing real names in a report is refused, and the message tells you which directory to rename. This replaces redacting and continuing, which hid the underlying defect (#503).
15
+ - Anonymized session charts now label each bar with its cost rank and project (`1 · project-1`) instead of the session id. The chart previously printed session ids in plain sight while anonymizing a field that was never rendered, so a `sessions` artifact could carry fifteen session ids and claim to be anonymized (#503).
16
+ - Revealing project names now shows a disambiguated basename everywhere, on the dashboard as well as the command line. Two projects named `app` under different parents render as `app (work)` and `app (personal)` rather than collapsing into one indistinguishable label — and, once anonymized, into one `project-N` (#503).
17
+ - A composed multi-panel document now uses one set of anonymous names across all of its sections, so `project-1` means the same project everywhere in the document. Previously each section was numbered on its own and the same name could denote different projects in different sections (#503).
18
+ - When the dashboard refuses a share export, its log now records only what kind of identifier was found, not the identifier itself. Because the refusal message names the offending value so you can find it, that value — which may be an absolute path, a session id or an email address — was being written to the dashboard log as well, and a log is a plausible thing to paste into a bug report. The message shown on screen is unchanged, and none of this detail was ever sent to a browser (#503).
19
+ - Rebuilding the stats index no longer re-reads the journal files whose contents cctally has already proved are in the cache. A rebuild reads every byte of the journal on every pass, and on a large install most of those bytes are Codex quota observations sitting in older files that never change and that contribute nothing new to the index. When the record described above says the cache has already caught up with one of those files, the rebuild now skips reading it and uses the counts it recorded the last time it read it. A file is skipped only when it is not the newest one, when what cctally recorded about it still matches the file on disk exactly — same size, same last complete line, same identity — when it contains no record type the index folds, and when the catch-up record covers it. Anything else, including a file that was still being written to when it was last read, is read normally, and the resulting index is identical either way. If the cache changes while the rebuild is running — an ordinary Codex session is enough — it reads the skipped files after all and replays them in the order they appear in the journal; if it cannot read one of them in full — including a file that ends earlier than cctally recorded, which reading it reports no error for — it still publishes the index but does not record the cache as caught up. `cctally db rebuild --db stats --json` reports which files were skipped and, for each one that was not, why (#496).
20
+
21
+ ### Fixed
22
+ - Keep the multi-account dashboard header on one desktop row when the live sync age changes between seconds and minutes, while preserving the intentional stacked layout on phone-width screens (#496).
23
+ - The `anonymized:` field in exported Markdown now reports whether the export was anonymized, rather than being guessed from what the labels look like. A chart-only artifact such as the Sessions visual stamped `anonymized: false` onto an artifact that had in fact been anonymized, and a project genuinely named `project-1` was reported as anonymized when it was not (#503).
24
+ - The dashboard's render and compose endpoints now anonymize when the request omits the anonymization field. `POST /api/share/render` previously defaulted that absent field to revealing real names, and the Projects panel's subtitle separately claimed "real projects" about contents that had been anonymized (#503).
25
+ - Restore generated agent instruction validation and keep repository instruction chains within their enforced size budget (#515).
26
+ - Prevent stats index corruption when an in-place rebuild overlaps an idle reader by leaving live WAL/SHM sidecars under SQLite's ownership; corruption recovery also preserves a mismatched WAL-index generation without checkpointing it into the main database (#514).
27
+
28
+ ## [1.92.3] - 2026-08-06
29
+
30
+ ### Changed
31
+ - `cctally db journal-repair` and `cctally db rederive` now read the journal once per pass instead of two to four times, which lowers their memory use and start-up time on large journals. Their output is unchanged (#496).
32
+ - The stats index now records what a rebuild worked out about correction batches, instead of throwing that away and working it out again on the next command. This upgrade rebuilds the stats index once. Nothing about the reports changes (#496).
33
+ - Rebuilding the stats index no longer replays Codex quota history into a cache that already holds it. Until now every rebuild re-materialized every retained quota observation into `cache.db` while holding both cache write locks, which is why version 1.92.2 reported that lock being held for 23.0 seconds on a 1.7 GB journal, and why other commands could report `database is locked` during a rebuild. cctally now records how far the cache has already caught up with the journal and, when that record still describes the journal and cache in front of it, skips the replay and takes no cache write lock at all. Measured on a 300,000-line journal carrying 276,538 quota observations: a rebuild whose cache already holds those observations held the cache write locks for 5.2 seconds without the record and 0.0 seconds with it, and finished in 12.6 seconds instead of 17.8. Those are the two figures to compare, because both describe the same full cache. A first rebuild into an empty cache still pays 14.2 seconds of lock hold to insert the rows for the first time, and the record does not change that — it removes the repeated re-materialization on every rebuild afterwards, not the initial one. The saving also assumes nothing wrote to the journal or the Codex cache between the two rebuilds; any such write discards the record and the next rebuild replays. The record is deliberately narrow: it says only that everything in the journal has reached the cache, never that the cache is otherwise correct, and anything that deletes cached quota rows discards it — including the Codex cache-rebuild migration, which clears every cached Codex quota row and can be re-run through `cctally db skip` and `cctally db unskip` — so the next rebuild replays. A record written by a version of cctally that materialized journal records differently is discarded rather than carried forward, and the record never claims a journal position past the last record the pass actually read, so a journal file that was still being written to is never counted as caught up. A rebuild that has to replay still does, and `cctally db rebuild --db stats --json` now reports which of the two happened (#496).
34
+ - A rebuild that does have to replay Codex quota history no longer holds the cache write locks for the whole replay. It now works in bounded pieces — each capped at 8 MB of journal bytes and 20,000 observations — and releases both locks between them, so a status line, a report or a Codex sync can write while a rebuild is running instead of waiting for it or reporting `database is locked`. Each piece is decoded only when its turn comes, so the memory saving version 1.92.2 delivered is unaffected. Because the locks are genuinely released, another process can now clear the Codex cache midway through a rebuild's replay; when that happens the rebuild notices and starts its replay again from the beginning rather than continuing over the gap, so it never records that the cache has caught up with the journal when it has not. Restarting is always safe, because replaying an observation that is already stored changes nothing. A rebuild will restart at most three times before reporting that its cache recovery is incomplete, so a process that keeps clearing the cache cannot keep a rebuild running indefinitely. A rebuild also restarts when the set of Codex source roots changes underneath it, which is the second signal that another process cleared the cache, and it reports its cache recovery as incomplete rather than complete whenever it finished the replay but could not record how far the cache had caught up (#496).
35
+ - A rebuild whose Codex cache recovery could not finish no longer serves the partial quota picture that recovery would have completed. The stats index's quota view is worked out from the Codex cache, so a rebuild that could not finish restoring that cache publishes a quota view covering only part of your history. The rebuild still publishes, still exits 0 and still prints the same success line — refusing to publish would leave a corrupt index in place, which is worse — but it now records inside the published index that its quota view is incomplete, and every command that reads that view refuses to serve it until it has been reconciled rather than showing you an understated one. The next command you run finishes the interrupted cache recovery and rewrites the quota view before answering, so the refusal is normally invisible; when it cannot finish — another process is mid-ingest, say — it leaves the mark in place and tries again on the following command rather than clearing it. `cctally db rebuild --db stats --json` now reports the two outcomes as separate fields, `publication` and `cacheRecovery`, so a script cannot read one as the other (#496).
36
+ - Reading a correction record no longer makes cctally read the whole journal. Until now, an ingest pass that met one re-read every journal file from the beginning to work out which corrections applied, on a 1.7 GB journal every time; a report or a dashboard refresh paid that cost only when it triggered an ingest. It now continues from what the last rebuild recorded and reads only the records it has not seen yet, and it loads only the stored rows that delta actually names rather than every one of them. The result is identical: the same corrections win, an incomplete or structurally invalid batch stays inert, and anything cctally cannot decide from the recorded state falls back to the full read rather than guessing. One case is handled explicitly instead of being carried forward: when a later record invalidates a correction batch that had already been applied, cctally rebuilds the index through exactly that record, so the problem is resolved once rather than being reported again on every following command. A pass that cannot use the recorded state — because another process held the database for a moment, say — now catches up on the next pass instead of leaving the shortcut switched off until the next full rebuild, and `cctally db rebuild --db stats --json` reports when it replaced an index that had fallen behind in that way. That catch-up is bounded: past a fixed number of journal bytes a pass falls back to the full read rather than re-reading a range that keeps growing, and the same JSON reports when it did. Recorded problems with a correction batch are also withdrawn when a later record resolves them, so `cctally doctor` no longer reports a problem, and no longer prints a `cctally db journal-repair` command for it, once the journal itself no longer shows one (#496).
37
+
38
+ ### Fixed
39
+ - `cctally doctor` now reports when the stats index's quota view is marked incomplete, and names `cctally cache-sync` as the way to reconcile it. Only two things clear that mark — a reconciliation, which only `cctally cache-sync` and the dashboard perform, and a later rebuild whose cache recovery completed — so until now the mark could stay set indefinitely with every Codex quota surface showing empty or stale figures and nothing anywhere stating the cause or the remedy. The new report is read-only, never reconciles anything, and is a warning rather than a failure, because the index is valid and no data is lost (#496).
40
+ - Every refusal to serve an incomplete quota view now names `cctally cache-sync` as the way to reconcile it, instead of saying only that the view is incomplete. In the terminal dashboard the refusal is also reported as its own cause rather than being folded into a generic database failure, which previously left the Codex source panel blank on a first run with nothing on screen explaining why (#496).
41
+ - The web dashboard now shows that refusal too. Its status chip reads "quota view reconciling" and names `cctally cache-sync`, where before it read "server sync error" with no suggested action and the sentence naming the remedy reached only the server's log. The two dashboard endpoints that can meet the refusal — a quota cycle's detail and a quota block's detail — now answer with a distinct response naming the same command, instead of reporting a server failure or an unavailable feature for a view that is simply being rebuilt (#496).
42
+ - A corrupt stats index is no longer reported as an incomplete quota view. The check that reads the incompleteness mark cannot read anything at all from a corrupt index, and it was treating that as its own case, which both stated the wrong cause and stopped the corruption ever reaching cctally's repair path. A corrupt index is now passed on to that path unchanged, and no quota figure is served either way (#496, #407).
43
+ - Publishing a rebuilt stats index no longer deletes the write-ahead log files of the database other cctally processes still have open, which could make those processes read stale figures or fail outright. Deleting those files is outside what SQLite supports while any connection is open, and two things had to happen together before the damage showed: something had to write to the database after the deletion, and the affected process had to have read from it before. Measured on two test machines: when the later write came from the same process, the open connection failed with a disk I/O error, and when it came from a separate process, the open connection — and any connection opened afterwards — kept reporting the figures from before that write, with no error at all. Reports, the dashboard and the status line all keep connections open across a rebuild, so both outcomes were reachable in ordinary use. The deletion is gone. Those files are removed by SQLite itself when the last connection closes, so nothing accumulates; what remains straight after a rebuild is an emptied log (#516, #496).
44
+ - The check that refuses to serve an incomplete quota view no longer treats a failure to read the incompleteness mark as proof the mark is clear. It previously ignored every database error while reading it, on the reasoning that any current index carries the mark's table; a connection can instead fail its reads for an unrelated reason, and was then told the quota view was complete when it was not. Only a genuinely absent table — an index predating the mark — is still read as complete, and absence is now established by looking the table up rather than by matching the wording of an error message; every other failure refuses the read. The separate decision to *attempt* a reconciliation still declines on any such failure, because a connection that cannot read the mark cannot safely start one either (#496).
45
+ - Finishing an interrupted Codex cache recovery no longer makes an ordinary command read the whole journal. The mark a rebuild leaves when its cache recovery could not finish used to be acted on by every command that opened the stats index, including each status line render and each background hook, and each of those read the journal from the beginning before it could try. Only `cctally cache-sync` and the dashboard now attempt that recovery; every other command reads the mark and moves on. An attempt that cannot finish is not repeated for five minutes rather than being retried on the next command, and an attempt that could not take the Codex cache write locks now stops before reading the journal rather than after. The messages the recovery used to print on an ordinary command are gone, and the ones a rebuild prints are unchanged (#496).
46
+ - A rebuild whose Codex cache recovery could not finish no longer clears its own incompleteness mark when `cache.db` is missing. With no cache to read from there is nothing to rewrite the quota view from, so the mark stayed correct and clearing it served the partial view it was there to refuse (#496).
47
+ - A rebuild now holds its read view of `cache.db` for a fraction of a second instead of about 28 seconds. That view blocks other processes from compacting the Codex cache's write-ahead log while it is open, which is the file version 1.87.0 added `cctally db checkpoint` for. `cctally db rebuild --db stats --json` now also reports the two rebuild phases that window spans, so a `database is locked` report during a rebuild is diagnosable (#496, #297).
48
+ - A cctally process that was interrupted just before a month boundary could append to last month's journal file after this month's had already started, leaving the journal out of order. Both writers now re-check which file is current at the moment they take the write lock, and refuse the write rather than appending out of order; a Codex quota capture that hits the refusal is retried on the next sync instead of being dropped (#511).
49
+ - A crash while cctally was first converting its database to the journal format could leave a duplicate copy of the conversion file behind, and every later rebuild then read both copies. On one real install two such duplicates accounted for 366 MB, about a fifth of the journal. The conversion now reuses the most recent conversion file when that file is byte for byte what it was about to write, instead of writing a second copy; when the most recent file differs in any way it still writes its own. Existing duplicates are left untouched, because the journal is never rewritten (#496).
50
+
8
51
  ## [1.92.2] - 2026-08-06
9
52
 
10
53
  ### Changed
@@ -132,6 +132,9 @@ from _lib_source_identity import source_root_key
132
132
  # #416 spec §4.2: the pure tolerance-anchored reset kernel. `_lib_quota` imports
133
133
  # only `_lib_accounts` (a stdlib leaf), so binding it here is circular-safe.
134
134
  import _lib_quota
135
+ # #496 S5b §4: the journal-to-cache coverage certificate kernel. Stdlib-only,
136
+ # so it is circular-safe for the same reason.
137
+ import _lib_cache_coverage
135
138
 
136
139
 
137
140
  # Module-level back-ref shims for the three out-of-scope JSONL/project
@@ -1775,6 +1778,14 @@ def _delete_codex_file_derived_rows(
1775
1778
  "DELETE FROM codex_session_files WHERE path = ?" + root_clause,
1776
1779
  params,
1777
1780
  )
1781
+ # #496 S5b §4.3: this deletes `quota_window_snapshots` rows the journal still
1782
+ # retains, so any stored coverage certificate would assert coverage for
1783
+ # durable observations whose materialization was just removed — the
1784
+ # `journal ⊃ cache` direction the certificate exists to exclude. Invalidate
1785
+ # in the same transaction as the deletes, exactly as `_clear_codex_derived_
1786
+ # rows` does for the projection certificate. This covers `reset_file=True`
1787
+ # in `_write_codex_file_batch` and every other caller alike.
1788
+ _invalidate_codex_journal_coverage_certificate(conn)
1778
1789
 
1779
1790
 
1780
1791
  def _clear_codex_derived_rows(conn: sqlite3.Connection) -> bool:
@@ -1808,6 +1819,10 @@ def _clear_codex_derived_rows(conn: sqlite3.Connection) -> bool:
1808
1819
  "SELECT 1 FROM codex_conversation_rollups LIMIT 1",
1809
1820
  "SELECT 1 FROM cache_meta "
1810
1821
  "WHERE key='codex_quota_projection_certificate' LIMIT 1",
1822
+ "SELECT 1 FROM cache_meta "
1823
+ f"WHERE key='{_lib_cache_coverage.CERTIFICATE_KEY}' LIMIT 1",
1824
+ "SELECT 1 FROM cache_meta "
1825
+ f"WHERE key='{_lib_cache_coverage.PROGRESS_KEY}' LIMIT 1",
1811
1826
  )
1812
1827
  )
1813
1828
  conn.execute("DELETE FROM codex_session_entries")
@@ -1824,9 +1839,334 @@ def _clear_codex_derived_rows(conn: sqlite3.Connection) -> bool:
1824
1839
  conn.execute(
1825
1840
  "DELETE FROM cache_meta WHERE key='codex_quota_projection_certificate'"
1826
1841
  )
1842
+ # #496 S5b §4.3, for the identical reason one layer up: this clears the
1843
+ # physical quota state the COVERAGE certificate describes, so leaving it
1844
+ # would make it stale-valid and let a rebuild's fast path skip a replay the
1845
+ # cache now needs. The two certificates have non-overlapping authority —
1846
+ # coverage binds journal-to-cache, projection binds cache-to-stats — and
1847
+ # neither may satisfy the other's gate, so both are deleted here.
1848
+ _invalidate_codex_journal_coverage_certificate(conn)
1827
1849
  return state_changed
1828
1850
 
1829
1851
 
1852
+ #: The families the #496 S5b coverage certificate describes: the cache tables a
1853
+ #: journal record materializes into. `tests/test_cache_coverage_496_s5b.py`
1854
+ #: scans `bin/` for DML against exactly these names, so widening the certificate
1855
+ #: to a further family means adding it here and re-running that guard.
1856
+ COVERAGE_CACHE_FAMILIES: "tuple[str, ...]" = (
1857
+ "quota_window_snapshots",
1858
+ "codex_file_accounts",
1859
+ "codex_file_incarnations",
1860
+ )
1861
+
1862
+ #: Every path that mutates or materializes those families, mapped to the ONE
1863
+ #: action it takes on the certificate (spec §4.3). `prohibited` is the default
1864
+ #: for anything absent: a writer nobody assigned an action to would leave the
1865
+ #: certificate stale-valid, which is the failure the certificate exists to
1866
+ #: exclude.
1867
+ #:
1868
+ #: `advance` may only ever move an EXTENDABLE predecessor forward, and it bumps
1869
+ #: `codex_physical_mutation_seq` in the same transaction. `mint` is the separate
1870
+ #: action for establishing a certificate where none existed, which the rebuild's
1871
+ #: recovery pass alone may do because only that pass reads the journal; it makes
1872
+ #: no sequence bump. Keeping the two words apart is what stops the inventory
1873
+ #: describing a bump the minting path does not make.
1874
+ #:
1875
+ #: The keys are `module.function`, and the static guard asserts that every
1876
+ #: function in `bin/` whose own body issues DML against a covered family appears
1877
+ #: here. Two kinds of key are NOT leaves and therefore cannot be scanned for:
1878
+ #: the parenthesized `(reset_file=True)` variant, which distinguishes two
1879
+ #: branches of one function, and the two composite transaction owners that
1880
+ #: delegate their DML to `_apply_quota_records` and `_apply_file_account_records`
1881
+ #: while owning the certificate decision. The guard names both sets explicitly so
1882
+ #: a key nobody can reach cannot be added either.
1883
+ COVERAGE_WRITER_ACTIONS: "dict[str, str]" = {
1884
+ # Ordinary rollout-walk batch: the observations it writes were journaled
1885
+ # (best-effort) immediately before, so the cache stays a superset and
1886
+ # coverage is never BROKEN here.
1887
+ #
1888
+ # DELIBERATE DEVIATION from spec §4.3, which assigns `advance`. Advancing
1889
+ # requires proving that the journal delta between the prior covered boundary
1890
+ # and now contains only records this batch applied, and this function runs
1891
+ # once per rollout file inside one flock hold — a journal range read per file
1892
+ # is the per-file cost issue #297 exists about.
1893
+ #
1894
+ # `preserve` is safe rather than merely cheaper, and the reason is the
1895
+ # UNCONDITIONAL `_bump_codex_physical_mutation_seq` this function makes in
1896
+ # the same transaction as its rows: that bump alone moves the sequence the
1897
+ # certificate is bound to, so any stored certificate goes invalid on the
1898
+ # sequence axis whatever else happened. The journal append is NOT the
1899
+ # reason, and stating it as one would be false: `_append_codex_quota_obs` is
1900
+ # best-effort and swallows every exception, so a batch can write cache rows
1901
+ # while growing no segment at all. A stale certificate is possible; a
1902
+ # stale-VALID one is not. `tests/test_quota_journal.py` pins the sequence
1903
+ # bump, because a future change making it conditional would silently turn
1904
+ # this path into a stale-valid one with a green suite. Coverage is then
1905
+ # re-established by `_cache_applier`, which consumes those same journal
1906
+ # records on the next status-line tick and owns a contiguous batch it can
1907
+ # prove.
1908
+ "_cctally_cache._write_codex_file_batch": "preserve",
1909
+ # `reset_file=True` runs `_delete_codex_file_derived_rows`, which drops
1910
+ # `quota_window_snapshots` rows the journal still retains. Advancing there
1911
+ # would assert coverage for durable observations whose materialization was
1912
+ # just deleted — the `journal ⊃ cache` direction the certificate exists to
1913
+ # exclude. File resets are rare, so paying a recovery pass afterwards is
1914
+ # cheap and unambiguous.
1915
+ "_cctally_cache._write_codex_file_batch(reset_file=True)": "invalidate",
1916
+ "_cctally_cache._clear_codex_derived_rows": "invalidate",
1917
+ "_cctally_cache._delete_codex_file_derived_rows": "invalidate",
1918
+ # `UPDATE quota_window_snapshots SET canonical_resets_at_utc = ?` over rows
1919
+ # already stored. It changes a row VALUE and never the SET of materialized
1920
+ # journal records, and the certificate deliberately promises nothing about
1921
+ # values (§4.1) — `canonical_resets_at_utc` is not even journaled, because
1922
+ # it is a property of the observation's population rather than of the
1923
+ # observation.
1924
+ "_cctally_cache.CodexResetAnchorResolver.apply_pending_merges": "preserve",
1925
+ # The two attribution-map leaves. Both materialize a decision that was
1926
+ # journaled fail-closed before the call, and neither deletes, so no caller
1927
+ # can reach them in a way that breaks coverage. The action belongs to the
1928
+ # transaction owner above them, not here.
1929
+ "_cctally_cache.record_codex_file_account": "preserve",
1930
+ "_cctally_cache.set_codex_file_incarnation": "preserve",
1931
+ # The two journal-to-cache appliers. Same reasoning: INSERT OR IGNORE on the
1932
+ # natural key, no deletes, and the certificate decision belongs to the
1933
+ # composite that owns their transaction.
1934
+ "_cctally_journal._apply_quota_records": "preserve",
1935
+ "_cctally_journal._apply_file_account_records": "preserve",
1936
+ # Two branches, and only the additive one leaves the covered families
1937
+ # untouched. `authoritative=False` replays journaled decisions into
1938
+ # `codex_file_accounts` with an idempotent upsert and writes no quota row at
1939
+ # all, so the coverage statement is unaffected.
1940
+ #
1941
+ # `authoritative=True` runs `DELETE FROM codex_file_accounts`, which IS a
1942
+ # covered family. `preserve` holds there for a reason that lives in the
1943
+ # caller rather than in this function, so it is written down here instead of
1944
+ # left as an ordering nobody stated: `sync_codex_cache` passes
1945
+ # `authoritative=bool(rebuild)`, and that same `rebuild` flag already ran
1946
+ # `_clear_codex_derived_rows` — which invalidates the certificate and the
1947
+ # progress record — and committed, before this call. The certificate is
1948
+ # therefore already gone when the delete runs, and the clear-then-replay
1949
+ # re-derives the whole map from `since=None` to the journal high water. A
1950
+ # second `authoritative=True` caller, or a reordering inside
1951
+ # `sync_codex_cache`, would break that silently, and the static scanner
1952
+ # cannot catch it because this key is already in the inventory with a green
1953
+ # label. `rehydrate_codex_file_accounts` therefore checks the invariant
1954
+ # itself and raises `CoverageInvariantViolation` rather than relying on it.
1955
+ "_cctally_journal.rehydrate_codex_file_accounts": "preserve",
1956
+ # Spec §4.3's migrations row, enumerated rather than named — and enumerated
1957
+ # by the action each one TAKES, not by the action the spec's one-line row
1958
+ # assumed. Only `_024` deletes rows the journal still retains, and it is the
1959
+ # only one that invalidates. `_038_codex_session_files_ingest_complete` is
1960
+ # deliberately ABSENT: the review that asked for this row listed it, but it
1961
+ # adds a column to `codex_session_files`, which is not a covered family —
1962
+ # which is why the guard scans rather than trusts a hand list.
1963
+ #
1964
+ # `DELETE FROM quota_window_snapshots WHERE source = 'codex'` with NO
1965
+ # sequence bump. Left standing, the certificate would be stale-VALID over a
1966
+ # cache holding zero Codex quota rows.
1967
+ "_cctally_db._024_codex_fused_ingest_rebuild": "invalidate",
1968
+ # `UPDATE quota_window_snapshots SET observed_model = ?`, and it bumps
1969
+ # `codex_physical_mutation_seq` in the same transaction whenever it changes
1970
+ # a row. The certificate is left standing and goes invalid on the SEQUENCE
1971
+ # axis, which is a different mechanism from deleting it but the same
1972
+ # outcome, so `preserve` is the honest label for what this path does.
1973
+ "_cctally_db._028_split_conversation_store": "preserve",
1974
+ # Both rewrite `canonical_resets_at_utc` on rows already stored. That is a
1975
+ # row VALUE and never the SET of materialized journal records, and §4.1
1976
+ # excludes values from the promise explicitly — `canonical_resets_at_utc` is
1977
+ # not even journaled, because it is a property of the observation's
1978
+ # population rather than of the observation.
1979
+ "_cctally_db._032_codex_canonical_reset_anchor": "preserve",
1980
+ "_cctally_db._033_codex_reset_anchor_component_closure": "preserve",
1981
+ # `UPDATE quota_window_snapshots SET observed_model = ?` over stored rows.
1982
+ # A value again, and the same §4.1 exclusion applies.
1983
+ "_cctally_db.backfill_codex_quota_observed_model": "preserve",
1984
+ # The composite transaction owners.
1985
+ "_cctally_journal._cache_applier": "advance",
1986
+ # `mint`, not `advance`. §4.3 defines advancing as including a
1987
+ # `codex_physical_mutation_seq` bump, and the rebuild leg never bumps: it
1988
+ # ESTABLISHES a certificate over a prefix it has just read from the journal
1989
+ # rather than extending a predecessor's claim, and the mint reads the
1990
+ # sequence inside its own transaction so the stored value is current
1991
+ # whatever any other writer did. Labelling it `advance` described behaviour
1992
+ # this path does not have.
1993
+ "_cctally_journal._rebuild_quota_cache_leg_raw": "mint",
1994
+ }
1995
+
1996
+ #: The inventory keys that name no scannable function body, with the reason each
1997
+ #: one cannot be found by scanning for DML. The guard asserts this set exactly,
1998
+ #: so an unreachable key cannot be parked here either.
1999
+ COVERAGE_NON_LEAF_ACTIONS: "frozenset[str]" = frozenset({
2000
+ # One branch of a function that is itself in the inventory.
2001
+ "_cctally_cache._write_codex_file_batch(reset_file=True)",
2002
+ # Transaction owners whose DML lives in the two appliers they call.
2003
+ "_cctally_journal._cache_applier",
2004
+ "_cctally_journal._rebuild_quota_cache_leg_raw",
2005
+ })
2006
+
2007
+
2008
+ def _advance_codex_journal_coverage(
2009
+ conn: sqlite3.Connection, *, prior: "dict | None", covered, applied_through,
2010
+ pinned_vector, allow_mint: bool = False,
2011
+ ) -> bool:
2012
+ """Store the advanced certificate in the CALLER's open transaction.
2013
+
2014
+ Returns whether it advanced. `prior` must be the certificate the caller read
2015
+ and validated before opening the transaction: a writer may only move a valid
2016
+ predecessor forward, and an absent one answers False rather than minting,
2017
+ because establishing coverage requires reading the journal and only the
2018
+ rebuild's recovery pass does that.
2019
+
2020
+ ``allow_mint`` is that pass's exemption. It is the one caller that has just
2021
+ read the journal prefix it is about to certify, so it may establish coverage
2022
+ where none existed. Every other writer leaves False.
2023
+
2024
+ The physical-mutation sequence is read HERE rather than passed in, because
2025
+ the caller bumps it in this same transaction and the certificate must carry
2026
+ the post-bump value. Reading it before the bump would store a certificate
2027
+ that `certificate_is_valid` rejects on its first use.
2028
+ """
2029
+ if prior is None and not allow_mint:
2030
+ return False
2031
+ if allow_mint and prior is None:
2032
+ # The mint stores over whatever is present, so it reads what is present
2033
+ # first and refuses to move it backward. Today no other writer can be
2034
+ # inside this transaction — `cmd_db_rebuild` holds the ingest lock
2035
+ # exclusively — but that safety comes from a lock in another module, and
2036
+ # the mint is what has to hold it.
2037
+ stored = load_codex_journal_coverage_certificate(conn)
2038
+ if _lib_cache_coverage.applied_through_regresses(
2039
+ stored, applied_through, pinned_vector
2040
+ ):
2041
+ return False
2042
+ row = conn.execute(
2043
+ "SELECT value FROM cache_meta WHERE key='codex_physical_mutation_seq'"
2044
+ ).fetchone()
2045
+ try:
2046
+ physical_seq = 0 if row is None or row[0] is None else int(row[0])
2047
+ except (TypeError, ValueError):
2048
+ return False
2049
+ try:
2050
+ certificate = _lib_cache_coverage.advance(
2051
+ prior, covered=covered, applied_through=applied_through,
2052
+ pinned_vector=pinned_vector, physical_seq=physical_seq)
2053
+ except (TypeError, ValueError):
2054
+ # The covered boundary is outside the pinned vector, which means the
2055
+ # journal moved under this writer. Leaving the prior certificate is the
2056
+ # safe direction: it describes a smaller prefix, and its identity root
2057
+ # no longer matches, so the next rebuild replays.
2058
+ return False
2059
+ _store_codex_journal_coverage_certificate(conn, certificate)
2060
+ return True
2061
+
2062
+
2063
+ def load_codex_journal_coverage_certificate(
2064
+ conn: sqlite3.Connection,
2065
+ ) -> "dict | None":
2066
+ """The stored coverage certificate, or None when absent or unreadable.
2067
+
2068
+ Unreadable answers None rather than raising, because every degraded state
2069
+ here falls back to a full replay silently (spec §6.3).
2070
+ """
2071
+ try:
2072
+ row = conn.execute(
2073
+ "SELECT value FROM cache_meta WHERE key = ?",
2074
+ (_lib_cache_coverage.CERTIFICATE_KEY,),
2075
+ ).fetchone()
2076
+ except sqlite3.Error:
2077
+ return None
2078
+ if row is None or row[0] is None:
2079
+ return None
2080
+ try:
2081
+ payload = json.loads(row[0])
2082
+ except (TypeError, ValueError):
2083
+ return None
2084
+ return payload if isinstance(payload, dict) else None
2085
+
2086
+
2087
+ def _store_codex_journal_coverage_certificate(
2088
+ conn: sqlite3.Connection, certificate: dict
2089
+ ) -> None:
2090
+ """Write the certificate inside the CALLER's open transaction.
2091
+
2092
+ It must commit with the rows it describes and with the
2093
+ `codex_physical_mutation_seq` bump, so a rollback leaves the prior
2094
+ certificate unchanged even when the journal appends survived — which is the
2095
+ safe direction.
2096
+ """
2097
+ conn.execute(
2098
+ "INSERT INTO cache_meta(key, value) VALUES (?, ?) "
2099
+ "ON CONFLICT(key) DO UPDATE SET value = excluded.value",
2100
+ (
2101
+ _lib_cache_coverage.CERTIFICATE_KEY,
2102
+ json.dumps(certificate, separators=(",", ":"), sort_keys=True),
2103
+ ),
2104
+ )
2105
+
2106
+
2107
+ def load_codex_recovery_progress(conn: sqlite3.Connection) -> "dict | None":
2108
+ """A recovery pass's stored progress, or None when absent or unreadable."""
2109
+ try:
2110
+ row = conn.execute(
2111
+ "SELECT value FROM cache_meta WHERE key = ?",
2112
+ (_lib_cache_coverage.PROGRESS_KEY,),
2113
+ ).fetchone()
2114
+ except sqlite3.Error:
2115
+ return None
2116
+ if row is None or row[0] is None:
2117
+ return None
2118
+ try:
2119
+ payload = json.loads(row[0])
2120
+ except (TypeError, ValueError):
2121
+ return None
2122
+ return payload if isinstance(payload, dict) else None
2123
+
2124
+
2125
+ def _store_codex_recovery_progress(
2126
+ conn: sqlite3.Connection, progress: dict,
2127
+ ) -> bool:
2128
+ """Advance progress under the monotonic compare-and-swap, in this txn.
2129
+
2130
+ Returns whether it was written. Refusing rather than overwriting is what
2131
+ keeps an older worker from moving a newer pass's checkpoint backwards.
2132
+ """
2133
+ stored = load_codex_recovery_progress(conn)
2134
+ if not _lib_cache_coverage.progress_supersedes(stored, progress):
2135
+ return False
2136
+ conn.execute(
2137
+ "INSERT INTO cache_meta(key, value) VALUES (?, ?) "
2138
+ "ON CONFLICT(key) DO UPDATE SET value = excluded.value",
2139
+ (
2140
+ _lib_cache_coverage.PROGRESS_KEY,
2141
+ json.dumps(progress, separators=(",", ":"), sort_keys=True),
2142
+ ),
2143
+ )
2144
+ return True
2145
+
2146
+
2147
+ def _invalidate_codex_journal_coverage_certificate(
2148
+ conn: sqlite3.Connection,
2149
+ ) -> None:
2150
+ """Drop the certificate AND any in-flight recovery progress, in this txn.
2151
+
2152
+ Every destructive or partial mutation of the covered families takes this
2153
+ branch. Re-establishing coverage is recovery's job.
2154
+
2155
+ The two deletes are ONE function rather than two calls a writer could get
2156
+ half right. Spec §4.5 requires that every destructive clear delete the
2157
+ progress record in the same transaction as the certificate, because a
2158
+ recovery pass that resumed from an in-memory cursor over cleared state would
2159
+ eventually mint a certificate claiming coverage the cache does not have.
2160
+ Structuring it as a single call is what makes "in the same transaction"
2161
+ impossible to violate by omission.
2162
+ """
2163
+ conn.execute(
2164
+ "DELETE FROM cache_meta WHERE key IN (?, ?)",
2165
+ (_lib_cache_coverage.CERTIFICATE_KEY,
2166
+ _lib_cache_coverage.PROGRESS_KEY),
2167
+ )
2168
+
2169
+
1830
2170
  def _bump_codex_physical_mutation_seq(conn: sqlite3.Connection) -> None:
1831
2171
  """Advance the shared Codex physical-state invalidation token in this txn.
1832
2172
 
@@ -2362,6 +2702,14 @@ def _append_codex_quota_obs(quota_rows: list) -> None:
2362
2702
  "individual_limit_json": individual_limit_json,
2363
2703
  "reached_type": reached_type, "observed_model": observed_model,
2364
2704
  }), dedupe_codex_quota=True)
2705
+ except _jr.JournalAppendTargetStale:
2706
+ # Retryable, NOT best-effort (#511, #496 S5b §2.4). Swallowing this
2707
+ # one lets the file offset advance past bytes whose observation was
2708
+ # never journaled, and the rollout JSONL those bytes came from
2709
+ # evaporates — the exact data-loss hole the quota journal exists to
2710
+ # close. Re-raising leaves the offset where it was, so the next sync
2711
+ # re-reads and re-appends the same bytes.
2712
+ raise
2365
2713
  except Exception as exc: # best-effort; a journal append must not break sync
2366
2714
  eprint(f"[codex-cache] quota obs journal append failed: {exc}")
2367
2715
 
@@ -10534,6 +10882,12 @@ def cmd_cache_sync(args: argparse.Namespace) -> int:
10534
10882
  default is 'all'.
10535
10883
  """
10536
10884
  source = getattr(args, "source", "all")
10885
+ # #496 S5b §4.7: this command is the documented remedy for a cache that is
10886
+ # behind its journal, so it is one of the contexts allowed to resume a
10887
+ # gated quota-projection recovery when it opens stats.db. The interactive
10888
+ # render paths (`statusline`, every hook tick) never arm it — the
10889
+ # resumption reads the whole journal.
10890
+ _cctally_core.enable_quota_projection_reconciliation()
10537
10891
  # #276 perf: clear any prior tree on this thread so a leaked root can't be
10538
10892
  # flushed, then (below) time the Claude sync_cache call as the "sync_cache"
10539
10893
  # root phase and flush the tree to stderr when CCTALLY_PERF_TRACE is set.