cctally 1.92.3 → 1.93.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +37 -2
- package/bin/_cctally_cache.py +354 -0
- package/bin/_cctally_core.py +180 -3
- package/bin/_cctally_dashboard.py +71 -1
- package/bin/_cctally_dashboard_envelope.py +28 -2
- package/bin/_cctally_dashboard_share.py +75 -19
- package/bin/_cctally_dashboard_sources.py +12 -0
- package/bin/_cctally_db.py +89 -1
- package/bin/_cctally_doctor.py +31 -0
- package/bin/_cctally_forecast.py +4 -2
- package/bin/_cctally_journal.py +3326 -215
- package/bin/_cctally_milestone_history.py +4 -1
- package/bin/_cctally_project.py +8 -6
- package/bin/_cctally_quota.py +420 -20
- package/bin/_cctally_reporting.py +8 -6
- package/bin/_cctally_share.py +74 -37
- package/bin/_cctally_source_analytics.py +6 -8
- package/bin/_cctally_store.py +13 -2
- package/bin/_cctally_tui.py +53 -0
- package/bin/_lib_cache_coverage.py +547 -0
- package/bin/_lib_doctor.py +54 -2
- package/bin/_lib_journal.py +195 -95
- package/bin/_lib_journal_router.py +21 -0
- package/bin/_lib_segment_summary.py +374 -0
- package/bin/_lib_selector_state.py +959 -0
- package/bin/_lib_share.py +1073 -165
- package/bin/_lib_share_templates.py +35 -11
- package/bin/_lib_stats_wal.py +327 -0
- package/bin/_lib_view_models.py +2 -1
- package/dashboard/static/assets/index-DwWJOYxd.css +1 -0
- package/dashboard/static/assets/{index-Dat-mza6.js → index-HlIK7k8Q.js} +47 -47
- package/dashboard/static/dashboard.html +2 -2
- package/package.json +5 -1
- package/dashboard/static/assets/index-DnWdv8um.css +0 -1
package/CHANGELOG.md
CHANGED
|
@@ -5,12 +5,47 @@ based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
|
|
5
5
|
|
|
6
6
|
## [Unreleased]
|
|
7
7
|
|
|
8
|
-
## [1.
|
|
8
|
+
## [1.93.0] - 2026-08-07
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
- The dashboard's share modal now states, above the preview, what the export will contain. With "Anonymize project names on export" checked it reads "Preview shows real names. Export is anonymized."; unchecked it reads "Export will show real project names." On a report that contains no project names at all — Trend and Forecast, and any other report whose data has none — it instead reads "This export contains no project names.", which is true whichever way the checkbox is set. The line is always present and changes text rather than appearing and disappearing, so a screen reader announces the new state when you toggle the checkbox. The preview pane itself still always shows real names, which is what makes the line necessary: until now the checkbox changed nothing you could see (#503).
|
|
9
12
|
|
|
10
13
|
### Changed
|
|
11
|
-
- `
|
|
14
|
+
- A share artifact that would disclose an identifier the documentation promises it will not is now **refused** rather than produced. The check runs over the finished document and looks for session ids and other canonical UUIDs, conversation identity keys, absolute paths and `~`/`$HOME` expansions, email addresses, source-root fingerprints, quota identity keys, and credential shapes. On the command line the refusal prints `cctally: refused to write a share artifact — share artifact would disclose: <what it found> (<the offending value>)` and exits 3, and no file is written; in the dashboard it surfaces as an export error. Naming the value matters because one known case has no way around it: if a project directory is itself named as a bare hexadecimal token or a UUID, showing real names in a report is refused, and the message tells you which directory to rename. This replaces redacting and continuing, which hid the underlying defect (#503).
|
|
15
|
+
- Anonymized session charts now label each bar with its cost rank and project (`1 · project-1`) instead of the session id. The chart previously printed session ids in plain sight while anonymizing a field that was never rendered, so a `sessions` artifact could carry fifteen session ids and claim to be anonymized (#503).
|
|
16
|
+
- Revealing project names now shows a disambiguated basename everywhere, on the dashboard as well as the command line. Two projects named `app` under different parents render as `app (work)` and `app (personal)` rather than collapsing into one indistinguishable label — and, once anonymized, into one `project-N` (#503).
|
|
17
|
+
- A composed multi-panel document now uses one set of anonymous names across all of its sections, so `project-1` means the same project everywhere in the document. Previously each section was numbered on its own and the same name could denote different projects in different sections (#503).
|
|
18
|
+
- When the dashboard refuses a share export, its log now records only what kind of identifier was found, not the identifier itself. Because the refusal message names the offending value so you can find it, that value — which may be an absolute path, a session id or an email address — was being written to the dashboard log as well, and a log is a plausible thing to paste into a bug report. The message shown on screen is unchanged, and none of this detail was ever sent to a browser (#503).
|
|
19
|
+
- Rebuilding the stats index no longer re-reads the journal files whose contents cctally has already proved are in the cache. A rebuild reads every byte of the journal on every pass, and on a large install most of those bytes are Codex quota observations sitting in older files that never change and that contribute nothing new to the index. When the record described above says the cache has already caught up with one of those files, the rebuild now skips reading it and uses the counts it recorded the last time it read it. A file is skipped only when it is not the newest one, when what cctally recorded about it still matches the file on disk exactly — same size, same last complete line, same identity — when it contains no record type the index folds, and when the catch-up record covers it. Anything else, including a file that was still being written to when it was last read, is read normally, and the resulting index is identical either way. If the cache changes while the rebuild is running — an ordinary Codex session is enough — it reads the skipped files after all and replays them in the order they appear in the journal; if it cannot read one of them in full — including a file that ends earlier than cctally recorded, which reading it reports no error for — it still publishes the index but does not record the cache as caught up. `cctally db rebuild --db stats --json` reports which files were skipped and, for each one that was not, why (#496).
|
|
12
20
|
|
|
13
21
|
### Fixed
|
|
22
|
+
- Keep the multi-account dashboard header on one desktop row when the live sync age changes between seconds and minutes, while preserving the intentional stacked layout on phone-width screens (#496).
|
|
23
|
+
- The `anonymized:` field in exported Markdown now reports whether the export was anonymized, rather than being guessed from what the labels look like. A chart-only artifact such as the Sessions visual stamped `anonymized: false` onto an artifact that had in fact been anonymized, and a project genuinely named `project-1` was reported as anonymized when it was not (#503).
|
|
24
|
+
- The dashboard's render and compose endpoints now anonymize when the request omits the anonymization field. `POST /api/share/render` previously defaulted that absent field to revealing real names, and the Projects panel's subtitle separately claimed "real projects" about contents that had been anonymized (#503).
|
|
25
|
+
- Restore generated agent instruction validation and keep repository instruction chains within their enforced size budget (#515).
|
|
26
|
+
- Prevent stats index corruption when an in-place rebuild overlaps an idle reader by leaving live WAL/SHM sidecars under SQLite's ownership; corruption recovery also preserves a mismatched WAL-index generation without checkpointing it into the main database (#514).
|
|
27
|
+
|
|
28
|
+
## [1.92.3] - 2026-08-06
|
|
29
|
+
|
|
30
|
+
### Changed
|
|
31
|
+
- `cctally db journal-repair` and `cctally db rederive` now read the journal once per pass instead of two to four times, which lowers their memory use and start-up time on large journals. Their output is unchanged (#496).
|
|
32
|
+
- The stats index now records what a rebuild worked out about correction batches, instead of throwing that away and working it out again on the next command. This upgrade rebuilds the stats index once. Nothing about the reports changes (#496).
|
|
33
|
+
- Rebuilding the stats index no longer replays Codex quota history into a cache that already holds it. Until now every rebuild re-materialized every retained quota observation into `cache.db` while holding both cache write locks, which is why version 1.92.2 reported that lock being held for 23.0 seconds on a 1.7 GB journal, and why other commands could report `database is locked` during a rebuild. cctally now records how far the cache has already caught up with the journal and, when that record still describes the journal and cache in front of it, skips the replay and takes no cache write lock at all. Measured on a 300,000-line journal carrying 276,538 quota observations: a rebuild whose cache already holds those observations held the cache write locks for 5.2 seconds without the record and 0.0 seconds with it, and finished in 12.6 seconds instead of 17.8. Those are the two figures to compare, because both describe the same full cache. A first rebuild into an empty cache still pays 14.2 seconds of lock hold to insert the rows for the first time, and the record does not change that — it removes the repeated re-materialization on every rebuild afterwards, not the initial one. The saving also assumes nothing wrote to the journal or the Codex cache between the two rebuilds; any such write discards the record and the next rebuild replays. The record is deliberately narrow: it says only that everything in the journal has reached the cache, never that the cache is otherwise correct, and anything that deletes cached quota rows discards it — including the Codex cache-rebuild migration, which clears every cached Codex quota row and can be re-run through `cctally db skip` and `cctally db unskip` — so the next rebuild replays. A record written by a version of cctally that materialized journal records differently is discarded rather than carried forward, and the record never claims a journal position past the last record the pass actually read, so a journal file that was still being written to is never counted as caught up. A rebuild that has to replay still does, and `cctally db rebuild --db stats --json` now reports which of the two happened (#496).
|
|
34
|
+
- A rebuild that does have to replay Codex quota history no longer holds the cache write locks for the whole replay. It now works in bounded pieces — each capped at 8 MB of journal bytes and 20,000 observations — and releases both locks between them, so a status line, a report or a Codex sync can write while a rebuild is running instead of waiting for it or reporting `database is locked`. Each piece is decoded only when its turn comes, so the memory saving version 1.92.2 delivered is unaffected. Because the locks are genuinely released, another process can now clear the Codex cache midway through a rebuild's replay; when that happens the rebuild notices and starts its replay again from the beginning rather than continuing over the gap, so it never records that the cache has caught up with the journal when it has not. Restarting is always safe, because replaying an observation that is already stored changes nothing. A rebuild will restart at most three times before reporting that its cache recovery is incomplete, so a process that keeps clearing the cache cannot keep a rebuild running indefinitely. A rebuild also restarts when the set of Codex source roots changes underneath it, which is the second signal that another process cleared the cache, and it reports its cache recovery as incomplete rather than complete whenever it finished the replay but could not record how far the cache had caught up (#496).
|
|
35
|
+
- A rebuild whose Codex cache recovery could not finish no longer serves the partial quota picture that recovery would have completed. The stats index's quota view is worked out from the Codex cache, so a rebuild that could not finish restoring that cache publishes a quota view covering only part of your history. The rebuild still publishes, still exits 0 and still prints the same success line — refusing to publish would leave a corrupt index in place, which is worse — but it now records inside the published index that its quota view is incomplete, and every command that reads that view refuses to serve it until it has been reconciled rather than showing you an understated one. The next command you run finishes the interrupted cache recovery and rewrites the quota view before answering, so the refusal is normally invisible; when it cannot finish — another process is mid-ingest, say — it leaves the mark in place and tries again on the following command rather than clearing it. `cctally db rebuild --db stats --json` now reports the two outcomes as separate fields, `publication` and `cacheRecovery`, so a script cannot read one as the other (#496).
|
|
36
|
+
- Reading a correction record no longer makes cctally read the whole journal. Until now, an ingest pass that met one re-read every journal file from the beginning to work out which corrections applied, on a 1.7 GB journal every time; a report or a dashboard refresh paid that cost only when it triggered an ingest. It now continues from what the last rebuild recorded and reads only the records it has not seen yet, and it loads only the stored rows that delta actually names rather than every one of them. The result is identical: the same corrections win, an incomplete or structurally invalid batch stays inert, and anything cctally cannot decide from the recorded state falls back to the full read rather than guessing. One case is handled explicitly instead of being carried forward: when a later record invalidates a correction batch that had already been applied, cctally rebuilds the index through exactly that record, so the problem is resolved once rather than being reported again on every following command. A pass that cannot use the recorded state — because another process held the database for a moment, say — now catches up on the next pass instead of leaving the shortcut switched off until the next full rebuild, and `cctally db rebuild --db stats --json` reports when it replaced an index that had fallen behind in that way. That catch-up is bounded: past a fixed number of journal bytes a pass falls back to the full read rather than re-reading a range that keeps growing, and the same JSON reports when it did. Recorded problems with a correction batch are also withdrawn when a later record resolves them, so `cctally doctor` no longer reports a problem, and no longer prints a `cctally db journal-repair` command for it, once the journal itself no longer shows one (#496).
|
|
37
|
+
|
|
38
|
+
### Fixed
|
|
39
|
+
- `cctally doctor` now reports when the stats index's quota view is marked incomplete, and names `cctally cache-sync` as the way to reconcile it. Only two things clear that mark — a reconciliation, which only `cctally cache-sync` and the dashboard perform, and a later rebuild whose cache recovery completed — so until now the mark could stay set indefinitely with every Codex quota surface showing empty or stale figures and nothing anywhere stating the cause or the remedy. The new report is read-only, never reconciles anything, and is a warning rather than a failure, because the index is valid and no data is lost (#496).
|
|
40
|
+
- Every refusal to serve an incomplete quota view now names `cctally cache-sync` as the way to reconcile it, instead of saying only that the view is incomplete. In the terminal dashboard the refusal is also reported as its own cause rather than being folded into a generic database failure, which previously left the Codex source panel blank on a first run with nothing on screen explaining why (#496).
|
|
41
|
+
- The web dashboard now shows that refusal too. Its status chip reads "quota view reconciling" and names `cctally cache-sync`, where before it read "server sync error" with no suggested action and the sentence naming the remedy reached only the server's log. The two dashboard endpoints that can meet the refusal — a quota cycle's detail and a quota block's detail — now answer with a distinct response naming the same command, instead of reporting a server failure or an unavailable feature for a view that is simply being rebuilt (#496).
|
|
42
|
+
- A corrupt stats index is no longer reported as an incomplete quota view. The check that reads the incompleteness mark cannot read anything at all from a corrupt index, and it was treating that as its own case, which both stated the wrong cause and stopped the corruption ever reaching cctally's repair path. A corrupt index is now passed on to that path unchanged, and no quota figure is served either way (#496, #407).
|
|
43
|
+
- Publishing a rebuilt stats index no longer deletes the write-ahead log files of the database other cctally processes still have open, which could make those processes read stale figures or fail outright. Deleting those files is outside what SQLite supports while any connection is open, and two things had to happen together before the damage showed: something had to write to the database after the deletion, and the affected process had to have read from it before. Measured on two test machines: when the later write came from the same process, the open connection failed with a disk I/O error, and when it came from a separate process, the open connection — and any connection opened afterwards — kept reporting the figures from before that write, with no error at all. Reports, the dashboard and the status line all keep connections open across a rebuild, so both outcomes were reachable in ordinary use. The deletion is gone. Those files are removed by SQLite itself when the last connection closes, so nothing accumulates; what remains straight after a rebuild is an emptied log (#516, #496).
|
|
44
|
+
- The check that refuses to serve an incomplete quota view no longer treats a failure to read the incompleteness mark as proof the mark is clear. It previously ignored every database error while reading it, on the reasoning that any current index carries the mark's table; a connection can instead fail its reads for an unrelated reason, and was then told the quota view was complete when it was not. Only a genuinely absent table — an index predating the mark — is still read as complete, and absence is now established by looking the table up rather than by matching the wording of an error message; every other failure refuses the read. The separate decision to *attempt* a reconciliation still declines on any such failure, because a connection that cannot read the mark cannot safely start one either (#496).
|
|
45
|
+
- Finishing an interrupted Codex cache recovery no longer makes an ordinary command read the whole journal. The mark a rebuild leaves when its cache recovery could not finish used to be acted on by every command that opened the stats index, including each status line render and each background hook, and each of those read the journal from the beginning before it could try. Only `cctally cache-sync` and the dashboard now attempt that recovery; every other command reads the mark and moves on. An attempt that cannot finish is not repeated for five minutes rather than being retried on the next command, and an attempt that could not take the Codex cache write locks now stops before reading the journal rather than after. The messages the recovery used to print on an ordinary command are gone, and the ones a rebuild prints are unchanged (#496).
|
|
46
|
+
- A rebuild whose Codex cache recovery could not finish no longer clears its own incompleteness mark when `cache.db` is missing. With no cache to read from there is nothing to rewrite the quota view from, so the mark stayed correct and clearing it served the partial view it was there to refuse (#496).
|
|
47
|
+
- A rebuild now holds its read view of `cache.db` for a fraction of a second instead of about 28 seconds. That view blocks other processes from compacting the Codex cache's write-ahead log while it is open, which is the file version 1.87.0 added `cctally db checkpoint` for. `cctally db rebuild --db stats --json` now also reports the two rebuild phases that window spans, so a `database is locked` report during a rebuild is diagnosable (#496, #297).
|
|
48
|
+
- A cctally process that was interrupted just before a month boundary could append to last month's journal file after this month's had already started, leaving the journal out of order. Both writers now re-check which file is current at the moment they take the write lock, and refuse the write rather than appending out of order; a Codex quota capture that hits the refusal is retried on the next sync instead of being dropped (#511).
|
|
14
49
|
- A crash while cctally was first converting its database to the journal format could leave a duplicate copy of the conversion file behind, and every later rebuild then read both copies. On one real install two such duplicates accounted for 366 MB, about a fifth of the journal. The conversion now reuses the most recent conversion file when that file is byte for byte what it was about to write, instead of writing a second copy; when the most recent file differs in any way it still writes its own. Existing duplicates are left untouched, because the journal is never rewritten (#496).
|
|
15
50
|
|
|
16
51
|
## [1.92.2] - 2026-08-06
|
package/bin/_cctally_cache.py
CHANGED
|
@@ -132,6 +132,9 @@ from _lib_source_identity import source_root_key
|
|
|
132
132
|
# #416 spec §4.2: the pure tolerance-anchored reset kernel. `_lib_quota` imports
|
|
133
133
|
# only `_lib_accounts` (a stdlib leaf), so binding it here is circular-safe.
|
|
134
134
|
import _lib_quota
|
|
135
|
+
# #496 S5b §4: the journal-to-cache coverage certificate kernel. Stdlib-only,
|
|
136
|
+
# so it is circular-safe for the same reason.
|
|
137
|
+
import _lib_cache_coverage
|
|
135
138
|
|
|
136
139
|
|
|
137
140
|
# Module-level back-ref shims for the three out-of-scope JSONL/project
|
|
@@ -1775,6 +1778,14 @@ def _delete_codex_file_derived_rows(
|
|
|
1775
1778
|
"DELETE FROM codex_session_files WHERE path = ?" + root_clause,
|
|
1776
1779
|
params,
|
|
1777
1780
|
)
|
|
1781
|
+
# #496 S5b §4.3: this deletes `quota_window_snapshots` rows the journal still
|
|
1782
|
+
# retains, so any stored coverage certificate would assert coverage for
|
|
1783
|
+
# durable observations whose materialization was just removed — the
|
|
1784
|
+
# `journal ⊃ cache` direction the certificate exists to exclude. Invalidate
|
|
1785
|
+
# in the same transaction as the deletes, exactly as `_clear_codex_derived_
|
|
1786
|
+
# rows` does for the projection certificate. This covers `reset_file=True`
|
|
1787
|
+
# in `_write_codex_file_batch` and every other caller alike.
|
|
1788
|
+
_invalidate_codex_journal_coverage_certificate(conn)
|
|
1778
1789
|
|
|
1779
1790
|
|
|
1780
1791
|
def _clear_codex_derived_rows(conn: sqlite3.Connection) -> bool:
|
|
@@ -1808,6 +1819,10 @@ def _clear_codex_derived_rows(conn: sqlite3.Connection) -> bool:
|
|
|
1808
1819
|
"SELECT 1 FROM codex_conversation_rollups LIMIT 1",
|
|
1809
1820
|
"SELECT 1 FROM cache_meta "
|
|
1810
1821
|
"WHERE key='codex_quota_projection_certificate' LIMIT 1",
|
|
1822
|
+
"SELECT 1 FROM cache_meta "
|
|
1823
|
+
f"WHERE key='{_lib_cache_coverage.CERTIFICATE_KEY}' LIMIT 1",
|
|
1824
|
+
"SELECT 1 FROM cache_meta "
|
|
1825
|
+
f"WHERE key='{_lib_cache_coverage.PROGRESS_KEY}' LIMIT 1",
|
|
1811
1826
|
)
|
|
1812
1827
|
)
|
|
1813
1828
|
conn.execute("DELETE FROM codex_session_entries")
|
|
@@ -1824,9 +1839,334 @@ def _clear_codex_derived_rows(conn: sqlite3.Connection) -> bool:
|
|
|
1824
1839
|
conn.execute(
|
|
1825
1840
|
"DELETE FROM cache_meta WHERE key='codex_quota_projection_certificate'"
|
|
1826
1841
|
)
|
|
1842
|
+
# #496 S5b §4.3, for the identical reason one layer up: this clears the
|
|
1843
|
+
# physical quota state the COVERAGE certificate describes, so leaving it
|
|
1844
|
+
# would make it stale-valid and let a rebuild's fast path skip a replay the
|
|
1845
|
+
# cache now needs. The two certificates have non-overlapping authority —
|
|
1846
|
+
# coverage binds journal-to-cache, projection binds cache-to-stats — and
|
|
1847
|
+
# neither may satisfy the other's gate, so both are deleted here.
|
|
1848
|
+
_invalidate_codex_journal_coverage_certificate(conn)
|
|
1827
1849
|
return state_changed
|
|
1828
1850
|
|
|
1829
1851
|
|
|
1852
|
+
#: The families the #496 S5b coverage certificate describes: the cache tables a
|
|
1853
|
+
#: journal record materializes into. `tests/test_cache_coverage_496_s5b.py`
|
|
1854
|
+
#: scans `bin/` for DML against exactly these names, so widening the certificate
|
|
1855
|
+
#: to a further family means adding it here and re-running that guard.
|
|
1856
|
+
COVERAGE_CACHE_FAMILIES: "tuple[str, ...]" = (
|
|
1857
|
+
"quota_window_snapshots",
|
|
1858
|
+
"codex_file_accounts",
|
|
1859
|
+
"codex_file_incarnations",
|
|
1860
|
+
)
|
|
1861
|
+
|
|
1862
|
+
#: Every path that mutates or materializes those families, mapped to the ONE
|
|
1863
|
+
#: action it takes on the certificate (spec §4.3). `prohibited` is the default
|
|
1864
|
+
#: for anything absent: a writer nobody assigned an action to would leave the
|
|
1865
|
+
#: certificate stale-valid, which is the failure the certificate exists to
|
|
1866
|
+
#: exclude.
|
|
1867
|
+
#:
|
|
1868
|
+
#: `advance` may only ever move an EXTENDABLE predecessor forward, and it bumps
|
|
1869
|
+
#: `codex_physical_mutation_seq` in the same transaction. `mint` is the separate
|
|
1870
|
+
#: action for establishing a certificate where none existed, which the rebuild's
|
|
1871
|
+
#: recovery pass alone may do because only that pass reads the journal; it makes
|
|
1872
|
+
#: no sequence bump. Keeping the two words apart is what stops the inventory
|
|
1873
|
+
#: describing a bump the minting path does not make.
|
|
1874
|
+
#:
|
|
1875
|
+
#: The keys are `module.function`, and the static guard asserts that every
|
|
1876
|
+
#: function in `bin/` whose own body issues DML against a covered family appears
|
|
1877
|
+
#: here. Two kinds of key are NOT leaves and therefore cannot be scanned for:
|
|
1878
|
+
#: the parenthesized `(reset_file=True)` variant, which distinguishes two
|
|
1879
|
+
#: branches of one function, and the two composite transaction owners that
|
|
1880
|
+
#: delegate their DML to `_apply_quota_records` and `_apply_file_account_records`
|
|
1881
|
+
#: while owning the certificate decision. The guard names both sets explicitly so
|
|
1882
|
+
#: a key nobody can reach cannot be added either.
|
|
1883
|
+
COVERAGE_WRITER_ACTIONS: "dict[str, str]" = {
|
|
1884
|
+
# Ordinary rollout-walk batch: the observations it writes were journaled
|
|
1885
|
+
# (best-effort) immediately before, so the cache stays a superset and
|
|
1886
|
+
# coverage is never BROKEN here.
|
|
1887
|
+
#
|
|
1888
|
+
# DELIBERATE DEVIATION from spec §4.3, which assigns `advance`. Advancing
|
|
1889
|
+
# requires proving that the journal delta between the prior covered boundary
|
|
1890
|
+
# and now contains only records this batch applied, and this function runs
|
|
1891
|
+
# once per rollout file inside one flock hold — a journal range read per file
|
|
1892
|
+
# is the per-file cost issue #297 exists about.
|
|
1893
|
+
#
|
|
1894
|
+
# `preserve` is safe rather than merely cheaper, and the reason is the
|
|
1895
|
+
# UNCONDITIONAL `_bump_codex_physical_mutation_seq` this function makes in
|
|
1896
|
+
# the same transaction as its rows: that bump alone moves the sequence the
|
|
1897
|
+
# certificate is bound to, so any stored certificate goes invalid on the
|
|
1898
|
+
# sequence axis whatever else happened. The journal append is NOT the
|
|
1899
|
+
# reason, and stating it as one would be false: `_append_codex_quota_obs` is
|
|
1900
|
+
# best-effort and swallows every exception, so a batch can write cache rows
|
|
1901
|
+
# while growing no segment at all. A stale certificate is possible; a
|
|
1902
|
+
# stale-VALID one is not. `tests/test_quota_journal.py` pins the sequence
|
|
1903
|
+
# bump, because a future change making it conditional would silently turn
|
|
1904
|
+
# this path into a stale-valid one with a green suite. Coverage is then
|
|
1905
|
+
# re-established by `_cache_applier`, which consumes those same journal
|
|
1906
|
+
# records on the next status-line tick and owns a contiguous batch it can
|
|
1907
|
+
# prove.
|
|
1908
|
+
"_cctally_cache._write_codex_file_batch": "preserve",
|
|
1909
|
+
# `reset_file=True` runs `_delete_codex_file_derived_rows`, which drops
|
|
1910
|
+
# `quota_window_snapshots` rows the journal still retains. Advancing there
|
|
1911
|
+
# would assert coverage for durable observations whose materialization was
|
|
1912
|
+
# just deleted — the `journal ⊃ cache` direction the certificate exists to
|
|
1913
|
+
# exclude. File resets are rare, so paying a recovery pass afterwards is
|
|
1914
|
+
# cheap and unambiguous.
|
|
1915
|
+
"_cctally_cache._write_codex_file_batch(reset_file=True)": "invalidate",
|
|
1916
|
+
"_cctally_cache._clear_codex_derived_rows": "invalidate",
|
|
1917
|
+
"_cctally_cache._delete_codex_file_derived_rows": "invalidate",
|
|
1918
|
+
# `UPDATE quota_window_snapshots SET canonical_resets_at_utc = ?` over rows
|
|
1919
|
+
# already stored. It changes a row VALUE and never the SET of materialized
|
|
1920
|
+
# journal records, and the certificate deliberately promises nothing about
|
|
1921
|
+
# values (§4.1) — `canonical_resets_at_utc` is not even journaled, because
|
|
1922
|
+
# it is a property of the observation's population rather than of the
|
|
1923
|
+
# observation.
|
|
1924
|
+
"_cctally_cache.CodexResetAnchorResolver.apply_pending_merges": "preserve",
|
|
1925
|
+
# The two attribution-map leaves. Both materialize a decision that was
|
|
1926
|
+
# journaled fail-closed before the call, and neither deletes, so no caller
|
|
1927
|
+
# can reach them in a way that breaks coverage. The action belongs to the
|
|
1928
|
+
# transaction owner above them, not here.
|
|
1929
|
+
"_cctally_cache.record_codex_file_account": "preserve",
|
|
1930
|
+
"_cctally_cache.set_codex_file_incarnation": "preserve",
|
|
1931
|
+
# The two journal-to-cache appliers. Same reasoning: INSERT OR IGNORE on the
|
|
1932
|
+
# natural key, no deletes, and the certificate decision belongs to the
|
|
1933
|
+
# composite that owns their transaction.
|
|
1934
|
+
"_cctally_journal._apply_quota_records": "preserve",
|
|
1935
|
+
"_cctally_journal._apply_file_account_records": "preserve",
|
|
1936
|
+
# Two branches, and only the additive one leaves the covered families
|
|
1937
|
+
# untouched. `authoritative=False` replays journaled decisions into
|
|
1938
|
+
# `codex_file_accounts` with an idempotent upsert and writes no quota row at
|
|
1939
|
+
# all, so the coverage statement is unaffected.
|
|
1940
|
+
#
|
|
1941
|
+
# `authoritative=True` runs `DELETE FROM codex_file_accounts`, which IS a
|
|
1942
|
+
# covered family. `preserve` holds there for a reason that lives in the
|
|
1943
|
+
# caller rather than in this function, so it is written down here instead of
|
|
1944
|
+
# left as an ordering nobody stated: `sync_codex_cache` passes
|
|
1945
|
+
# `authoritative=bool(rebuild)`, and that same `rebuild` flag already ran
|
|
1946
|
+
# `_clear_codex_derived_rows` — which invalidates the certificate and the
|
|
1947
|
+
# progress record — and committed, before this call. The certificate is
|
|
1948
|
+
# therefore already gone when the delete runs, and the clear-then-replay
|
|
1949
|
+
# re-derives the whole map from `since=None` to the journal high water. A
|
|
1950
|
+
# second `authoritative=True` caller, or a reordering inside
|
|
1951
|
+
# `sync_codex_cache`, would break that silently, and the static scanner
|
|
1952
|
+
# cannot catch it because this key is already in the inventory with a green
|
|
1953
|
+
# label. `rehydrate_codex_file_accounts` therefore checks the invariant
|
|
1954
|
+
# itself and raises `CoverageInvariantViolation` rather than relying on it.
|
|
1955
|
+
"_cctally_journal.rehydrate_codex_file_accounts": "preserve",
|
|
1956
|
+
# Spec §4.3's migrations row, enumerated rather than named — and enumerated
|
|
1957
|
+
# by the action each one TAKES, not by the action the spec's one-line row
|
|
1958
|
+
# assumed. Only `_024` deletes rows the journal still retains, and it is the
|
|
1959
|
+
# only one that invalidates. `_038_codex_session_files_ingest_complete` is
|
|
1960
|
+
# deliberately ABSENT: the review that asked for this row listed it, but it
|
|
1961
|
+
# adds a column to `codex_session_files`, which is not a covered family —
|
|
1962
|
+
# which is why the guard scans rather than trusts a hand list.
|
|
1963
|
+
#
|
|
1964
|
+
# `DELETE FROM quota_window_snapshots WHERE source = 'codex'` with NO
|
|
1965
|
+
# sequence bump. Left standing, the certificate would be stale-VALID over a
|
|
1966
|
+
# cache holding zero Codex quota rows.
|
|
1967
|
+
"_cctally_db._024_codex_fused_ingest_rebuild": "invalidate",
|
|
1968
|
+
# `UPDATE quota_window_snapshots SET observed_model = ?`, and it bumps
|
|
1969
|
+
# `codex_physical_mutation_seq` in the same transaction whenever it changes
|
|
1970
|
+
# a row. The certificate is left standing and goes invalid on the SEQUENCE
|
|
1971
|
+
# axis, which is a different mechanism from deleting it but the same
|
|
1972
|
+
# outcome, so `preserve` is the honest label for what this path does.
|
|
1973
|
+
"_cctally_db._028_split_conversation_store": "preserve",
|
|
1974
|
+
# Both rewrite `canonical_resets_at_utc` on rows already stored. That is a
|
|
1975
|
+
# row VALUE and never the SET of materialized journal records, and §4.1
|
|
1976
|
+
# excludes values from the promise explicitly — `canonical_resets_at_utc` is
|
|
1977
|
+
# not even journaled, because it is a property of the observation's
|
|
1978
|
+
# population rather than of the observation.
|
|
1979
|
+
"_cctally_db._032_codex_canonical_reset_anchor": "preserve",
|
|
1980
|
+
"_cctally_db._033_codex_reset_anchor_component_closure": "preserve",
|
|
1981
|
+
# `UPDATE quota_window_snapshots SET observed_model = ?` over stored rows.
|
|
1982
|
+
# A value again, and the same §4.1 exclusion applies.
|
|
1983
|
+
"_cctally_db.backfill_codex_quota_observed_model": "preserve",
|
|
1984
|
+
# The composite transaction owners.
|
|
1985
|
+
"_cctally_journal._cache_applier": "advance",
|
|
1986
|
+
# `mint`, not `advance`. §4.3 defines advancing as including a
|
|
1987
|
+
# `codex_physical_mutation_seq` bump, and the rebuild leg never bumps: it
|
|
1988
|
+
# ESTABLISHES a certificate over a prefix it has just read from the journal
|
|
1989
|
+
# rather than extending a predecessor's claim, and the mint reads the
|
|
1990
|
+
# sequence inside its own transaction so the stored value is current
|
|
1991
|
+
# whatever any other writer did. Labelling it `advance` described behaviour
|
|
1992
|
+
# this path does not have.
|
|
1993
|
+
"_cctally_journal._rebuild_quota_cache_leg_raw": "mint",
|
|
1994
|
+
}
|
|
1995
|
+
|
|
1996
|
+
#: The inventory keys that name no scannable function body, with the reason each
|
|
1997
|
+
#: one cannot be found by scanning for DML. The guard asserts this set exactly,
|
|
1998
|
+
#: so an unreachable key cannot be parked here either.
|
|
1999
|
+
COVERAGE_NON_LEAF_ACTIONS: "frozenset[str]" = frozenset({
|
|
2000
|
+
# One branch of a function that is itself in the inventory.
|
|
2001
|
+
"_cctally_cache._write_codex_file_batch(reset_file=True)",
|
|
2002
|
+
# Transaction owners whose DML lives in the two appliers they call.
|
|
2003
|
+
"_cctally_journal._cache_applier",
|
|
2004
|
+
"_cctally_journal._rebuild_quota_cache_leg_raw",
|
|
2005
|
+
})
|
|
2006
|
+
|
|
2007
|
+
|
|
2008
|
+
def _advance_codex_journal_coverage(
|
|
2009
|
+
conn: sqlite3.Connection, *, prior: "dict | None", covered, applied_through,
|
|
2010
|
+
pinned_vector, allow_mint: bool = False,
|
|
2011
|
+
) -> bool:
|
|
2012
|
+
"""Store the advanced certificate in the CALLER's open transaction.
|
|
2013
|
+
|
|
2014
|
+
Returns whether it advanced. `prior` must be the certificate the caller read
|
|
2015
|
+
and validated before opening the transaction: a writer may only move a valid
|
|
2016
|
+
predecessor forward, and an absent one answers False rather than minting,
|
|
2017
|
+
because establishing coverage requires reading the journal and only the
|
|
2018
|
+
rebuild's recovery pass does that.
|
|
2019
|
+
|
|
2020
|
+
``allow_mint`` is that pass's exemption. It is the one caller that has just
|
|
2021
|
+
read the journal prefix it is about to certify, so it may establish coverage
|
|
2022
|
+
where none existed. Every other writer leaves False.
|
|
2023
|
+
|
|
2024
|
+
The physical-mutation sequence is read HERE rather than passed in, because
|
|
2025
|
+
the caller bumps it in this same transaction and the certificate must carry
|
|
2026
|
+
the post-bump value. Reading it before the bump would store a certificate
|
|
2027
|
+
that `certificate_is_valid` rejects on its first use.
|
|
2028
|
+
"""
|
|
2029
|
+
if prior is None and not allow_mint:
|
|
2030
|
+
return False
|
|
2031
|
+
if allow_mint and prior is None:
|
|
2032
|
+
# The mint stores over whatever is present, so it reads what is present
|
|
2033
|
+
# first and refuses to move it backward. Today no other writer can be
|
|
2034
|
+
# inside this transaction — `cmd_db_rebuild` holds the ingest lock
|
|
2035
|
+
# exclusively — but that safety comes from a lock in another module, and
|
|
2036
|
+
# the mint is what has to hold it.
|
|
2037
|
+
stored = load_codex_journal_coverage_certificate(conn)
|
|
2038
|
+
if _lib_cache_coverage.applied_through_regresses(
|
|
2039
|
+
stored, applied_through, pinned_vector
|
|
2040
|
+
):
|
|
2041
|
+
return False
|
|
2042
|
+
row = conn.execute(
|
|
2043
|
+
"SELECT value FROM cache_meta WHERE key='codex_physical_mutation_seq'"
|
|
2044
|
+
).fetchone()
|
|
2045
|
+
try:
|
|
2046
|
+
physical_seq = 0 if row is None or row[0] is None else int(row[0])
|
|
2047
|
+
except (TypeError, ValueError):
|
|
2048
|
+
return False
|
|
2049
|
+
try:
|
|
2050
|
+
certificate = _lib_cache_coverage.advance(
|
|
2051
|
+
prior, covered=covered, applied_through=applied_through,
|
|
2052
|
+
pinned_vector=pinned_vector, physical_seq=physical_seq)
|
|
2053
|
+
except (TypeError, ValueError):
|
|
2054
|
+
# The covered boundary is outside the pinned vector, which means the
|
|
2055
|
+
# journal moved under this writer. Leaving the prior certificate is the
|
|
2056
|
+
# safe direction: it describes a smaller prefix, and its identity root
|
|
2057
|
+
# no longer matches, so the next rebuild replays.
|
|
2058
|
+
return False
|
|
2059
|
+
_store_codex_journal_coverage_certificate(conn, certificate)
|
|
2060
|
+
return True
|
|
2061
|
+
|
|
2062
|
+
|
|
2063
|
+
def load_codex_journal_coverage_certificate(
|
|
2064
|
+
conn: sqlite3.Connection,
|
|
2065
|
+
) -> "dict | None":
|
|
2066
|
+
"""The stored coverage certificate, or None when absent or unreadable.
|
|
2067
|
+
|
|
2068
|
+
Unreadable answers None rather than raising, because every degraded state
|
|
2069
|
+
here falls back to a full replay silently (spec §6.3).
|
|
2070
|
+
"""
|
|
2071
|
+
try:
|
|
2072
|
+
row = conn.execute(
|
|
2073
|
+
"SELECT value FROM cache_meta WHERE key = ?",
|
|
2074
|
+
(_lib_cache_coverage.CERTIFICATE_KEY,),
|
|
2075
|
+
).fetchone()
|
|
2076
|
+
except sqlite3.Error:
|
|
2077
|
+
return None
|
|
2078
|
+
if row is None or row[0] is None:
|
|
2079
|
+
return None
|
|
2080
|
+
try:
|
|
2081
|
+
payload = json.loads(row[0])
|
|
2082
|
+
except (TypeError, ValueError):
|
|
2083
|
+
return None
|
|
2084
|
+
return payload if isinstance(payload, dict) else None
|
|
2085
|
+
|
|
2086
|
+
|
|
2087
|
+
def _store_codex_journal_coverage_certificate(
|
|
2088
|
+
conn: sqlite3.Connection, certificate: dict
|
|
2089
|
+
) -> None:
|
|
2090
|
+
"""Write the certificate inside the CALLER's open transaction.
|
|
2091
|
+
|
|
2092
|
+
It must commit with the rows it describes and with the
|
|
2093
|
+
`codex_physical_mutation_seq` bump, so a rollback leaves the prior
|
|
2094
|
+
certificate unchanged even when the journal appends survived — which is the
|
|
2095
|
+
safe direction.
|
|
2096
|
+
"""
|
|
2097
|
+
conn.execute(
|
|
2098
|
+
"INSERT INTO cache_meta(key, value) VALUES (?, ?) "
|
|
2099
|
+
"ON CONFLICT(key) DO UPDATE SET value = excluded.value",
|
|
2100
|
+
(
|
|
2101
|
+
_lib_cache_coverage.CERTIFICATE_KEY,
|
|
2102
|
+
json.dumps(certificate, separators=(",", ":"), sort_keys=True),
|
|
2103
|
+
),
|
|
2104
|
+
)
|
|
2105
|
+
|
|
2106
|
+
|
|
2107
|
+
def load_codex_recovery_progress(conn: sqlite3.Connection) -> "dict | None":
|
|
2108
|
+
"""A recovery pass's stored progress, or None when absent or unreadable."""
|
|
2109
|
+
try:
|
|
2110
|
+
row = conn.execute(
|
|
2111
|
+
"SELECT value FROM cache_meta WHERE key = ?",
|
|
2112
|
+
(_lib_cache_coverage.PROGRESS_KEY,),
|
|
2113
|
+
).fetchone()
|
|
2114
|
+
except sqlite3.Error:
|
|
2115
|
+
return None
|
|
2116
|
+
if row is None or row[0] is None:
|
|
2117
|
+
return None
|
|
2118
|
+
try:
|
|
2119
|
+
payload = json.loads(row[0])
|
|
2120
|
+
except (TypeError, ValueError):
|
|
2121
|
+
return None
|
|
2122
|
+
return payload if isinstance(payload, dict) else None
|
|
2123
|
+
|
|
2124
|
+
|
|
2125
|
+
def _store_codex_recovery_progress(
|
|
2126
|
+
conn: sqlite3.Connection, progress: dict,
|
|
2127
|
+
) -> bool:
|
|
2128
|
+
"""Advance progress under the monotonic compare-and-swap, in this txn.
|
|
2129
|
+
|
|
2130
|
+
Returns whether it was written. Refusing rather than overwriting is what
|
|
2131
|
+
keeps an older worker from moving a newer pass's checkpoint backwards.
|
|
2132
|
+
"""
|
|
2133
|
+
stored = load_codex_recovery_progress(conn)
|
|
2134
|
+
if not _lib_cache_coverage.progress_supersedes(stored, progress):
|
|
2135
|
+
return False
|
|
2136
|
+
conn.execute(
|
|
2137
|
+
"INSERT INTO cache_meta(key, value) VALUES (?, ?) "
|
|
2138
|
+
"ON CONFLICT(key) DO UPDATE SET value = excluded.value",
|
|
2139
|
+
(
|
|
2140
|
+
_lib_cache_coverage.PROGRESS_KEY,
|
|
2141
|
+
json.dumps(progress, separators=(",", ":"), sort_keys=True),
|
|
2142
|
+
),
|
|
2143
|
+
)
|
|
2144
|
+
return True
|
|
2145
|
+
|
|
2146
|
+
|
|
2147
|
+
def _invalidate_codex_journal_coverage_certificate(
|
|
2148
|
+
conn: sqlite3.Connection,
|
|
2149
|
+
) -> None:
|
|
2150
|
+
"""Drop the certificate AND any in-flight recovery progress, in this txn.
|
|
2151
|
+
|
|
2152
|
+
Every destructive or partial mutation of the covered families takes this
|
|
2153
|
+
branch. Re-establishing coverage is recovery's job.
|
|
2154
|
+
|
|
2155
|
+
The two deletes are ONE function rather than two calls a writer could get
|
|
2156
|
+
half right. Spec §4.5 requires that every destructive clear delete the
|
|
2157
|
+
progress record in the same transaction as the certificate, because a
|
|
2158
|
+
recovery pass that resumed from an in-memory cursor over cleared state would
|
|
2159
|
+
eventually mint a certificate claiming coverage the cache does not have.
|
|
2160
|
+
Structuring it as a single call is what makes "in the same transaction"
|
|
2161
|
+
impossible to violate by omission.
|
|
2162
|
+
"""
|
|
2163
|
+
conn.execute(
|
|
2164
|
+
"DELETE FROM cache_meta WHERE key IN (?, ?)",
|
|
2165
|
+
(_lib_cache_coverage.CERTIFICATE_KEY,
|
|
2166
|
+
_lib_cache_coverage.PROGRESS_KEY),
|
|
2167
|
+
)
|
|
2168
|
+
|
|
2169
|
+
|
|
1830
2170
|
def _bump_codex_physical_mutation_seq(conn: sqlite3.Connection) -> None:
|
|
1831
2171
|
"""Advance the shared Codex physical-state invalidation token in this txn.
|
|
1832
2172
|
|
|
@@ -2362,6 +2702,14 @@ def _append_codex_quota_obs(quota_rows: list) -> None:
|
|
|
2362
2702
|
"individual_limit_json": individual_limit_json,
|
|
2363
2703
|
"reached_type": reached_type, "observed_model": observed_model,
|
|
2364
2704
|
}), dedupe_codex_quota=True)
|
|
2705
|
+
except _jr.JournalAppendTargetStale:
|
|
2706
|
+
# Retryable, NOT best-effort (#511, #496 S5b §2.4). Swallowing this
|
|
2707
|
+
# one lets the file offset advance past bytes whose observation was
|
|
2708
|
+
# never journaled, and the rollout JSONL those bytes came from
|
|
2709
|
+
# evaporates — the exact data-loss hole the quota journal exists to
|
|
2710
|
+
# close. Re-raising leaves the offset where it was, so the next sync
|
|
2711
|
+
# re-reads and re-appends the same bytes.
|
|
2712
|
+
raise
|
|
2365
2713
|
except Exception as exc: # best-effort; a journal append must not break sync
|
|
2366
2714
|
eprint(f"[codex-cache] quota obs journal append failed: {exc}")
|
|
2367
2715
|
|
|
@@ -10534,6 +10882,12 @@ def cmd_cache_sync(args: argparse.Namespace) -> int:
|
|
|
10534
10882
|
default is 'all'.
|
|
10535
10883
|
"""
|
|
10536
10884
|
source = getattr(args, "source", "all")
|
|
10885
|
+
# #496 S5b §4.7: this command is the documented remedy for a cache that is
|
|
10886
|
+
# behind its journal, so it is one of the contexts allowed to resume a
|
|
10887
|
+
# gated quota-projection recovery when it opens stats.db. The interactive
|
|
10888
|
+
# render paths (`statusline`, every hook tick) never arm it — the
|
|
10889
|
+
# resumption reads the whole journal.
|
|
10890
|
+
_cctally_core.enable_quota_projection_reconciliation()
|
|
10537
10891
|
# #276 perf: clear any prior tree on this thread so a leaked root can't be
|
|
10538
10892
|
# flushed, then (below) time the Claude sync_cache call as the "sync_cache"
|
|
10539
10893
|
# root phase and flush the tree to stderr when CCTALLY_PERF_TRACE is set.
|