cctally 1.85.1 → 1.87.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +16 -0
- package/bin/_cctally_cache.py +388 -35
- package/bin/_cctally_dashboard_sources.py +66 -33
- package/bin/_cctally_db.py +117 -1
- package/bin/_cctally_doctor.py +16 -1
- package/bin/_cctally_journal.py +5 -0
- package/bin/_cctally_quota.py +103 -11
- package/bin/_cctally_record.py +32 -6
- package/bin/_lib_aggregators.py +3 -0
- package/bin/_lib_codex_account_adoption.py +173 -0
- package/bin/_lib_doctor.py +28 -0
- package/bin/_lib_quota.py +170 -6
- package/dashboard/static/assets/index-B0ZCsoxI.css +1 -0
- package/dashboard/static/assets/index-Bvp8mxtz.js +92 -0
- package/dashboard/static/dashboard.html +2 -2
- package/package.json +2 -1
- package/dashboard/static/assets/index-DlVVJeS4.js +0 -92
- package/dashboard/static/assets/index-OYBkyglj.css +0 -1
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,22 @@ based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
|
|
5
5
|
|
|
6
6
|
## [Unreleased]
|
|
7
7
|
|
|
8
|
+
## [1.87.0] - 2026-07-30
|
|
9
|
+
|
|
10
|
+
### Fixed
|
|
11
|
+
- A Codex cycle's per-percent milestone ladder no longer reports `$0.00` for a crossing whose spend is real. cctally already recognised which account a quota window belongs to even when individual samples arrived unlabelled, but the dollars behind those percentages were attributed separately, per rollout file — and for usage recorded before cctally began keeping durable per-file attribution there was no decision to apply, so that spend belonged to nobody. The crossing was confidently the account's while its cost read as zero, the cycle hero under-reported spend by the same amount, and roughly a seventh of one Pro account's weekly Codex spend sat in the `unattributed` bucket instead of on its own card. Codex spend recorded inside an account-level weekly window is now attributed to that window's account whenever the window names exactly one, so the ladder, the cycle hero, the per-account cards, the Codex budget and the CLI all agree. Nothing is guessed: spend that no window can name is left alone, so is spend claimed by two windows naming different accounts, and separate model pools such as GPT-5.3-Codex-Spark are never treated as account quota. Existing history is repaired once on upgrade, and no usage is ever counted twice.
|
|
12
|
+
|
|
13
|
+
## [1.86.0] - 2026-07-30
|
|
14
|
+
|
|
15
|
+
### Fixed
|
|
16
|
+
- Each Claude, Codex, or future-provider logical model release now receives a distinct, accessible dashboard color that stays consistent across cards and modals, while dated and capacity-qualified variants of the same release share one color.
|
|
17
|
+
- Codex quota windows whose reported reset drifts through a chain of nearby values now converge into one physical window. The original upgrade grouped each sample only against already-settled anchors, so real history could leave one connected window split across multiple percentages, forecasts, milestone ladders and alert identities even though every adjacent reset remained within the intended ten-minute tolerance. A follow-up cache migration repairs already-upgraded history without changing the raw provider evidence, live ingest applies the same transitive rule, and terminal alert evidence follows exact retained members even when an endpoint is more than ten minutes from the winning first observation. Release cuts now also refuse every newly registered migration until a dated real-production-copy verification is tracked; `--ack-migrations` cannot bypass that gate. (#425)
|
|
18
|
+
- Pooled Codex weekly rows now name every account whose near-identical reset boundary contributed to the row. The Weekly and Trend panels and their expanded tables no longer show one account's maximum percentage without disclosing the other account blended into it; focused and single-account views remain unchanged. Three dashboard tests also now reject malformed trailing text instead of checking only a correct-looking prefix. (#424)
|
|
19
|
+
- The dashboard now keeps account focus consistent across every account-sensitive surface. Claude's merged and All-provider headlines no longer borrow one account's quota, while a focused Claude account uses only the percentage, reset and spend fields its own card actually provides. Focused keyboard shortcuts open scoped sessions and blocks, merged Codex block rows always expose their account in visible and accessible text, and shared-root quota forecasts no longer join through the account-blind parent. Account selection announcements now name the selected state correctly, long account labels remain identifiable on 320px screens, and small non-zero dollar values retain their cents instead of rounding to `$0`. (#423)
|
|
20
|
+
- A weekly percent milestone is no longer recorded carrying the cost of an earlier crossing. Recording a crossing first refreshes the week's cost, but when a concurrent cache ingest made that account-scoped refresh fail closed, cctally kept going and stamped the last stored cost onto the new percentage — so the Current Week per-percent ladder showed two consecutive percentages at an identical cumulative dollar figure, a `$0.00` marginal and a `$/1%` that never happened. Because milestone rows are written once, that was permanent. The crossing is now skipped instead and recorded by the next observation with its real cost, so nothing is lost; the same fail-closed condition on a reset-affected week also no longer aborts the whole ingest cycle over a transient lock.
|
|
21
|
+
- Codex quota percentages now appear only for windows the server still marks live. The Current Cycle modal no longer turns a retained, already-reset history row into a current percentage or `$ / 1%`; the All-accounts cycle table derives each live percentage and reset from that account's own quota window, including a live unattributed bucket whose totals-only card intentionally blanks those fields. Decorated parent envelopes also stop exposing one representative account's cycle-history index, and Forecast panels now preserve the same one-decimal percentage shown by their expanded views. (#422)
|
|
22
|
+
- Codex cache sync now stops before reading rollout files when its durable account-attribution decisions cannot be replayed, preventing a transient journal/cache failure from reassigning usage to the account currently signed in. Doctor also reports retained quota rows that lack their canonical reset anchor and points to the safe Codex cache rebuild. (#421)
|
|
23
|
+
|
|
8
24
|
## [1.85.1] - 2026-07-29
|
|
9
25
|
|
|
10
26
|
### Fixed
|
package/bin/_cctally_cache.py
CHANGED
|
@@ -192,6 +192,11 @@ _iter_message_rows = _lib_conversation.iter_message_rows
|
|
|
192
192
|
# so it loads at module-load time alongside _lib_conversation.
|
|
193
193
|
_lib_codex_conversation = _load_lib("_lib_codex_conversation")
|
|
194
194
|
|
|
195
|
+
# Window-scoped spend adoption's decision kernel (2026-07-30 spec). Pure
|
|
196
|
+
# stdlib leaf, same shape as `_lib_codex_pools`, so it loads here rather than
|
|
197
|
+
# through a bare import that would depend on ``bin/`` being on ``sys.path``.
|
|
198
|
+
_lib_codex_account_adoption = _load_lib("_lib_codex_account_adoption")
|
|
199
|
+
|
|
195
200
|
# Opt-in backend phase-instrumentation collector (issue #276, Session A). Pure
|
|
196
201
|
# stdlib leaf; near-noop when CCTALLY_PERF_TRACE is unset (phase() returns a
|
|
197
202
|
# shared no-op singleton), so the sync_cache seam wraps below cost nothing on
|
|
@@ -1054,7 +1059,11 @@ class CodexResetAnchorResolver:
|
|
|
1054
1059
|
|
|
1055
1060
|
def __init__(self, conn: sqlite3.Connection):
|
|
1056
1061
|
self._conn = conn
|
|
1057
|
-
self._groups: "dict[tuple[str, str, str, object], _lib_quota.
|
|
1062
|
+
self._groups: "dict[tuple[str, str, str, object], _lib_quota.ResetAnchorComponents]" = {}
|
|
1063
|
+
self._pending_merges: list[
|
|
1064
|
+
tuple[tuple[str, str, str, object], str, dt.datetime,
|
|
1065
|
+
tuple[dt.datetime, ...]]
|
|
1066
|
+
] = []
|
|
1058
1067
|
self._seed_failed = False
|
|
1059
1068
|
|
|
1060
1069
|
@staticmethod
|
|
@@ -1068,34 +1077,31 @@ class CodexResetAnchorResolver:
|
|
|
1068
1077
|
_lib_jsonl.snap_codex_window_minutes(window_minutes),
|
|
1069
1078
|
)
|
|
1070
1079
|
|
|
1071
|
-
def
|
|
1080
|
+
def _components_for(
|
|
1072
1081
|
self, group, logical_limit_key: str,
|
|
1073
|
-
) -> "_lib_quota.
|
|
1074
|
-
"""The group's
|
|
1075
|
-
|
|
1076
|
-
|
|
1077
|
-
|
|
1078
|
-
|
|
1079
|
-
walk by `cache-sync --rebuild`. A linear scan per observation makes both
|
|
1080
|
-
quadratic in the group's anchor count (~1,750 a year for a 5h window).
|
|
1082
|
+
) -> "_lib_quota.ResetAnchorComponents":
|
|
1083
|
+
"""The group's raw-reset components, ordered by deterministic ingest.
|
|
1084
|
+
|
|
1085
|
+
Raw values, not only stored anchors, are required to recover transitive
|
|
1086
|
+
chain membership. Ordering by rollout byte identity preserves the
|
|
1087
|
+
original first-sight winner while making a rebuild reproducible.
|
|
1081
1088
|
"""
|
|
1082
|
-
|
|
1083
|
-
if
|
|
1084
|
-
return
|
|
1085
|
-
|
|
1089
|
+
components = self._groups.get(group)
|
|
1090
|
+
if components is not None:
|
|
1091
|
+
return components
|
|
1092
|
+
components = _lib_quota.ResetAnchorComponents()
|
|
1086
1093
|
if not self._seed_failed:
|
|
1087
1094
|
candidates = _lib_jsonl.codex_snap_equivalent_limit_keys(
|
|
1088
1095
|
str(logical_limit_key))
|
|
1089
1096
|
placeholders = ",".join("?" for _ in candidates)
|
|
1090
1097
|
try:
|
|
1091
1098
|
rows = self._conn.execute(
|
|
1092
|
-
"SELECT
|
|
1099
|
+
"SELECT source_path, line_offset, id, resets_at_utc "
|
|
1093
1100
|
"FROM quota_window_snapshots "
|
|
1094
1101
|
"WHERE source = 'codex' AND source_root_key = ? "
|
|
1095
1102
|
" AND observed_slot = ? "
|
|
1096
1103
|
f" AND logical_limit_key IN ({placeholders}) "
|
|
1097
|
-
"
|
|
1098
|
-
"ORDER BY canonical_resets_at_utc",
|
|
1104
|
+
"ORDER BY source_path, line_offset, id",
|
|
1099
1105
|
(group[0], group[1], *candidates),
|
|
1100
1106
|
).fetchall()
|
|
1101
1107
|
except sqlite3.DatabaseError:
|
|
@@ -1105,15 +1111,66 @@ class CodexResetAnchorResolver:
|
|
|
1105
1111
|
self._seed_failed = True
|
|
1106
1112
|
rows = []
|
|
1107
1113
|
for row in rows:
|
|
1108
|
-
parsed = _parse_anchor_iso(row[
|
|
1114
|
+
parsed = _parse_anchor_iso(row[3])
|
|
1109
1115
|
if parsed is not None:
|
|
1110
|
-
|
|
1111
|
-
|
|
1112
|
-
|
|
1116
|
+
components.add(
|
|
1117
|
+
parsed,
|
|
1118
|
+
order_key=(
|
|
1119
|
+
str(row[0]), int(row[1]), int(row[2])),
|
|
1120
|
+
)
|
|
1121
|
+
self._groups[group] = components
|
|
1122
|
+
return components
|
|
1123
|
+
|
|
1124
|
+
def _merge_stored_anchors(
|
|
1125
|
+
self, group, logical_limit_key: str, winner: dt.datetime,
|
|
1126
|
+
retired: tuple[dt.datetime, ...],
|
|
1127
|
+
) -> None:
|
|
1128
|
+
if not retired:
|
|
1129
|
+
return
|
|
1130
|
+
self._pending_merges.append(
|
|
1131
|
+
(group, str(logical_limit_key), winner, retired))
|
|
1132
|
+
|
|
1133
|
+
def apply_pending_merges(self) -> None:
|
|
1134
|
+
"""Apply queued component merges inside the caller's transaction.
|
|
1135
|
+
|
|
1136
|
+
Resolution happens while the direct walk is still buffering a file.
|
|
1137
|
+
Deferring DML until `_write_codex_file_batch` keeps retired-anchor
|
|
1138
|
+
updates atomic with that file's rows and cursor; its one rollback/retry
|
|
1139
|
+
can safely reapply the unchanged queue.
|
|
1140
|
+
"""
|
|
1141
|
+
for group, logical_limit_key, winner, retired in self._pending_merges:
|
|
1142
|
+
candidates = _lib_jsonl.codex_snap_equivalent_limit_keys(
|
|
1143
|
+
logical_limit_key)
|
|
1144
|
+
key_placeholders = ",".join("?" for _ in candidates)
|
|
1145
|
+
retired_text = tuple(
|
|
1146
|
+
_codex_anchor_iso(value) for value in retired)
|
|
1147
|
+
retired_placeholders = ",".join("?" for _ in retired_text)
|
|
1148
|
+
self._conn.execute(
|
|
1149
|
+
"UPDATE quota_window_snapshots "
|
|
1150
|
+
"SET canonical_resets_at_utc = ? "
|
|
1151
|
+
"WHERE source = 'codex' AND source_root_key = ? "
|
|
1152
|
+
" AND observed_slot = ? "
|
|
1153
|
+
f" AND logical_limit_key IN ({key_placeholders}) "
|
|
1154
|
+
f" AND canonical_resets_at_utc IN ({retired_placeholders})",
|
|
1155
|
+
(
|
|
1156
|
+
_codex_anchor_iso(winner), group[0], group[1],
|
|
1157
|
+
*candidates, *retired_text,
|
|
1158
|
+
),
|
|
1159
|
+
)
|
|
1160
|
+
|
|
1161
|
+
def mark_file_committed(self) -> None:
|
|
1162
|
+
self._pending_merges.clear()
|
|
1163
|
+
|
|
1164
|
+
def discard_uncommitted_file(self) -> None:
|
|
1165
|
+
"""Forget buffered evidence after its file did not commit."""
|
|
1166
|
+
self._groups.clear()
|
|
1167
|
+
self._pending_merges.clear()
|
|
1113
1168
|
|
|
1114
1169
|
def resolve(
|
|
1115
1170
|
self, *, source_root_key: str, observed_slot: str,
|
|
1116
1171
|
logical_limit_key: str, window_minutes: object, resets_at_utc: object,
|
|
1172
|
+
source_path: "str | None" = None,
|
|
1173
|
+
line_offset: "int | None" = None,
|
|
1117
1174
|
) -> "str | None":
|
|
1118
1175
|
"""The canonical anchor for one observation, as stored TEXT.
|
|
1119
1176
|
|
|
@@ -1126,10 +1183,17 @@ class CodexResetAnchorResolver:
|
|
|
1126
1183
|
return None
|
|
1127
1184
|
group = self.group_key(
|
|
1128
1185
|
source_root_key, observed_slot, logical_limit_key, window_minutes)
|
|
1129
|
-
|
|
1130
|
-
|
|
1131
|
-
if
|
|
1132
|
-
|
|
1186
|
+
components = self._components_for(group, logical_limit_key)
|
|
1187
|
+
order_key = None
|
|
1188
|
+
if (
|
|
1189
|
+
isinstance(source_path, str)
|
|
1190
|
+
and isinstance(line_offset, int)
|
|
1191
|
+
and not isinstance(line_offset, bool)
|
|
1192
|
+
):
|
|
1193
|
+
order_key = (source_path, line_offset, -1)
|
|
1194
|
+
chosen, retired = components.add(raw, order_key=order_key)
|
|
1195
|
+
self._merge_stored_anchors(
|
|
1196
|
+
group, logical_limit_key, chosen, retired)
|
|
1133
1197
|
# ALWAYS re-serialized through the canonical UTC form, established or
|
|
1134
1198
|
# joined alike: two spellings of one instant ("…Z" vs "…+00:00") must
|
|
1135
1199
|
# never mint two anchors for one cluster, and a row seeded by anything
|
|
@@ -1137,6 +1201,25 @@ class CodexResetAnchorResolver:
|
|
|
1137
1201
|
# spell it either way.
|
|
1138
1202
|
return _codex_anchor_iso(chosen)
|
|
1139
1203
|
|
|
1204
|
+
def normalize_quota_rows(self, quota_rows: list[tuple[Any, ...]]) -> None:
|
|
1205
|
+
"""Converge buffered direct-walk rows after a later bridge observation.
|
|
1206
|
+
|
|
1207
|
+
The journal applier inserts one row immediately after resolution, so a
|
|
1208
|
+
component merge can update earlier rows in SQLite. The direct rollout
|
|
1209
|
+
walk buffers a whole file before its first DML; this final pass applies
|
|
1210
|
+
the same union result to those not-yet-inserted tuples.
|
|
1211
|
+
"""
|
|
1212
|
+
for index, row in enumerate(quota_rows):
|
|
1213
|
+
raw = _parse_anchor_iso(row[11])
|
|
1214
|
+
if raw is None:
|
|
1215
|
+
continue
|
|
1216
|
+
group = self.group_key(row[1], row[5], row[6], row[9])
|
|
1217
|
+
components = self._groups.get(group)
|
|
1218
|
+
if components is None:
|
|
1219
|
+
continue
|
|
1220
|
+
quota_rows[index] = row[:-1] + (
|
|
1221
|
+
_codex_anchor_iso(components.canonical(raw)),)
|
|
1222
|
+
|
|
1140
1223
|
|
|
1141
1224
|
def _codex_anchor_iso(value: dt.datetime) -> str:
|
|
1142
1225
|
return value.astimezone(dt.timezone.utc).isoformat().replace("+00:00", "Z")
|
|
@@ -1272,7 +1355,9 @@ def _discover_codex_files_with_roots() -> list[CodexDiscoveredFile]:
|
|
|
1272
1355
|
discovered: list[CodexDiscoveredFile] = []
|
|
1273
1356
|
seen: set[pathlib.Path] = set()
|
|
1274
1357
|
for root in _codex_provider_roots():
|
|
1275
|
-
for candidate in
|
|
1358
|
+
for candidate in sorted(
|
|
1359
|
+
root.walk_root.glob("**/*.jsonl"), key=lambda path: str(path)
|
|
1360
|
+
):
|
|
1276
1361
|
if not candidate.is_file():
|
|
1277
1362
|
continue
|
|
1278
1363
|
physical_path = _canonical_codex_path(candidate)
|
|
@@ -1905,6 +1990,7 @@ def _write_codex_file_batch(
|
|
|
1905
1990
|
file_identity: "str | None" = None,
|
|
1906
1991
|
incarnation: "int | None" = None,
|
|
1907
1992
|
file_account_decision: "tuple[int, str | None] | None" = None,
|
|
1993
|
+
anchor_resolver: "CodexResetAnchorResolver | None" = None,
|
|
1908
1994
|
) -> int:
|
|
1909
1995
|
"""Write one fully-buffered Codex file atomically and return entry changes.
|
|
1910
1996
|
|
|
@@ -1960,6 +2046,8 @@ def _write_codex_file_batch(
|
|
|
1960
2046
|
)
|
|
1961
2047
|
rows_changed = conn.total_changes - before
|
|
1962
2048
|
if quota_rows:
|
|
2049
|
+
if anchor_resolver is not None:
|
|
2050
|
+
anchor_resolver.apply_pending_merges()
|
|
1963
2051
|
conn.executemany(
|
|
1964
2052
|
"""INSERT OR IGNORE INTO quota_window_snapshots
|
|
1965
2053
|
(source, source_root_key, source_path, line_offset,
|
|
@@ -2027,7 +2115,7 @@ def _iter_codex_jsonl_paths(roots: list[pathlib.Path]) -> Iterator[pathlib.Path]
|
|
|
2027
2115
|
"""
|
|
2028
2116
|
seen: set[pathlib.Path] = set()
|
|
2029
2117
|
for root in roots:
|
|
2030
|
-
for jp in root.glob("**/*.jsonl"):
|
|
2118
|
+
for jp in sorted(root.glob("**/*.jsonl"), key=lambda path: str(path)):
|
|
2031
2119
|
# Dedup on the RESOLVED path, not the raw spelling. A symlinked
|
|
2032
2120
|
# $CODEX_HOME root or an alias entry (`.../.codex`,
|
|
2033
2121
|
# `.../sub/../.codex`) can glob the same physical file under
|
|
@@ -4729,6 +4817,22 @@ def _progress_codex_stderr(stats: CodexIngestStats, *, force: bool = False) -> N
|
|
|
4729
4817
|
)
|
|
4730
4818
|
|
|
4731
4819
|
|
|
4820
|
+
def _extend_codex_touched_span(
|
|
4821
|
+
spans: "dict[str, tuple[dt.datetime, dt.datetime]]",
|
|
4822
|
+
source_root_key: object,
|
|
4823
|
+
moment: "dt.datetime | None",
|
|
4824
|
+
) -> None:
|
|
4825
|
+
"""Widen one root's touched instant span in place."""
|
|
4826
|
+
if not source_root_key or moment is None:
|
|
4827
|
+
return
|
|
4828
|
+
key = str(source_root_key)
|
|
4829
|
+
current = spans.get(key)
|
|
4830
|
+
if current is None:
|
|
4831
|
+
spans[key] = (moment, moment)
|
|
4832
|
+
else:
|
|
4833
|
+
spans[key] = (min(current[0], moment), max(current[1], moment))
|
|
4834
|
+
|
|
4835
|
+
|
|
4732
4836
|
def sync_codex_cache(
|
|
4733
4837
|
conn: sqlite3.Connection,
|
|
4734
4838
|
*,
|
|
@@ -4753,6 +4857,13 @@ def sync_codex_cache(
|
|
|
4753
4857
|
"""
|
|
4754
4858
|
stats = CodexIngestStats()
|
|
4755
4859
|
project_after_unlock = False
|
|
4860
|
+
# Per-root instant span this sync wrote — accounting-row timestamps AND
|
|
4861
|
+
# canonical window resets. It bounds the end-of-sync spend-adoption pass to
|
|
4862
|
+
# the windows this sync could have changed; an unchanged tree leaves it empty
|
|
4863
|
+
# and the pass does no SQL at all. A rebuild deliberately passes ``None``
|
|
4864
|
+
# instead (full re-derivation restores the unattributed state, so the repair
|
|
4865
|
+
# has to re-run over everything).
|
|
4866
|
+
adoption_spans: "dict[str, tuple[dt.datetime, dt.datetime]]" = {}
|
|
4756
4867
|
# #313 P1 review (F4/F1): when the CACHE certificate is current we cannot
|
|
4757
4868
|
# yet decide whether to skip the reconcile — reconcile's own short-circuit
|
|
4758
4869
|
# ALSO requires the stats-side quota_projection_state signatures to match
|
|
@@ -4845,10 +4956,13 @@ def sync_codex_cache(
|
|
|
4845
4956
|
conn.commit()
|
|
4846
4957
|
eprint("[cache-sync] rebuild: cleared Codex cached entries")
|
|
4847
4958
|
# #416 spec §3.4: rehydrate the attribution map from the journal BEFORE
|
|
4848
|
-
# the walk.
|
|
4849
|
-
#
|
|
4850
|
-
#
|
|
4851
|
-
#
|
|
4959
|
+
# the walk. This cursor is intentionally distinct from the stats ingest
|
|
4960
|
+
# cursor that drives `_cache_applier`: a recreated/rebuilt cache.db may
|
|
4961
|
+
# have an empty map while stats is already at journal high-water, and
|
|
4962
|
+
# invoking stats ingest here would either reverse the total lock order
|
|
4963
|
+
# (inside the cache flocks) or leave an append-before-lock race (outside
|
|
4964
|
+
# them). The private cache-map cursor is therefore the only safe witness
|
|
4965
|
+
# that every durable decision visible to this locked walk was replayed.
|
|
4852
4966
|
#
|
|
4853
4967
|
# Deliberately NOT rebuild-only. Every production Codex call site syncs
|
|
4854
4968
|
# with rebuild=False, and the corruption auto-heal recreates the cache.db
|
|
@@ -4921,9 +5035,11 @@ def sync_codex_cache(
|
|
|
4921
5035
|
_jr._report_file_account_conflicts(_declined)
|
|
4922
5036
|
except Exception as exc:
|
|
4923
5037
|
conn.rollback()
|
|
5038
|
+
stats.deferred_reason = "attribution_rehydration"
|
|
4924
5039
|
eprint(
|
|
4925
5040
|
"[cache-sync] could not rehydrate Codex attribution "
|
|
4926
|
-
f"decisions: {exc};
|
|
5041
|
+
f"decisions: {exc}; deferring the Codex walk")
|
|
5042
|
+
return stats
|
|
4927
5043
|
|
|
4928
5044
|
# Pure read (glob + is_file only); safe to run before the SELECT and
|
|
4929
5045
|
# the per-file loop, where no cache.db write lock may be held. Targeted
|
|
@@ -5186,9 +5302,14 @@ def sync_codex_cache(
|
|
|
5186
5302
|
# The guard is the whole safety argument: `delta_append` means
|
|
5187
5303
|
# `start_offset` is this file's ingest watermark, and the second
|
|
5188
5304
|
# condition means the new range starts strictly beyond every
|
|
5189
|
-
# decided range.
|
|
5190
|
-
#
|
|
5191
|
-
# a
|
|
5305
|
+
# decided range. Today the term is algebraically redundant:
|
|
5306
|
+
# every non-delta branch sets `start_offset = 0`, while every
|
|
5307
|
+
# decided range starts at a non-negative offset, so the strict
|
|
5308
|
+
# comparison alone implies a delta append. Keep the explicit
|
|
5309
|
+
# term as belt-and-suspenders: it pins the semantic permission
|
|
5310
|
+
# to consult auth.json if a future branch changes the offsets.
|
|
5311
|
+
# Auth can therefore mint a range only for bytes NOBODY has
|
|
5312
|
+
# attributed yet; it never re-decides covered bytes.
|
|
5192
5313
|
if delta_append and start_offset > account_ranges[-1][0]:
|
|
5193
5314
|
root_account = _live_root_account()
|
|
5194
5315
|
if root_account.status == "torn":
|
|
@@ -5345,6 +5466,8 @@ def sync_codex_cache(
|
|
|
5345
5466
|
logical_limit_key=quota.logical_limit_key,
|
|
5346
5467
|
window_minutes=quota.window_minutes,
|
|
5347
5468
|
resets_at_utc=quota.resets_at_utc,
|
|
5469
|
+
source_path=quota.source_path,
|
|
5470
|
+
line_offset=quota.line_offset,
|
|
5348
5471
|
),
|
|
5349
5472
|
))
|
|
5350
5473
|
if (thread := emission.thread) is not None and (
|
|
@@ -5394,6 +5517,7 @@ def sync_codex_cache(
|
|
|
5394
5517
|
stats.skip_reasons[_r] = stats.skip_reasons.get(_r, 0) + _n
|
|
5395
5518
|
except OSError as exc:
|
|
5396
5519
|
eprint(f"[codex-cache] could not read {jp}: {exc}")
|
|
5520
|
+
anchor_resolver.discard_uncommitted_file()
|
|
5397
5521
|
if targeted:
|
|
5398
5522
|
stats.files_failed += 1 # §5.1 I/O decline → call dirty
|
|
5399
5523
|
continue
|
|
@@ -5468,6 +5592,7 @@ def sync_codex_cache(
|
|
|
5468
5592
|
f"[codex-cache] attribution decision journal append "
|
|
5469
5593
|
f"failed for {jp}: {exc}; deferring the file")
|
|
5470
5594
|
stats.files_failed += 1
|
|
5595
|
+
anchor_resolver.discard_uncommitted_file()
|
|
5471
5596
|
continue
|
|
5472
5597
|
|
|
5473
5598
|
# Task 7 Item 1: journal the Codex quota observations BEFORE the cache
|
|
@@ -5477,6 +5602,7 @@ def sync_codex_cache(
|
|
|
5477
5602
|
# (idempotent at the QUOTA_APPLIER natural key) rather than losing the
|
|
5478
5603
|
# observation. Appended once here, not inside the retry loop, so a DB
|
|
5479
5604
|
# retry never double-journals.
|
|
5605
|
+
anchor_resolver.normalize_quota_rows(quota_rows)
|
|
5480
5606
|
_append_codex_quota_obs(quota_rows)
|
|
5481
5607
|
|
|
5482
5608
|
# Every derived row above was buffered before the first DML. A
|
|
@@ -5514,6 +5640,7 @@ def sync_codex_cache(
|
|
|
5514
5640
|
file_identity=file_identity,
|
|
5515
5641
|
incarnation=incarnation,
|
|
5516
5642
|
file_account_decision=pending_decision,
|
|
5643
|
+
anchor_resolver=anchor_resolver,
|
|
5517
5644
|
)
|
|
5518
5645
|
except sqlite3.DatabaseError as exc:
|
|
5519
5646
|
conn.rollback()
|
|
@@ -5543,7 +5670,26 @@ def sync_codex_cache(
|
|
|
5543
5670
|
# walk count. Targeted mode already depended on this signal;
|
|
5544
5671
|
# explicit rebuild now uses it to reject partial success too.
|
|
5545
5672
|
stats.files_failed += 1
|
|
5673
|
+
anchor_resolver.discard_uncommitted_file()
|
|
5546
5674
|
continue
|
|
5675
|
+
anchor_resolver.mark_file_committed()
|
|
5676
|
+
|
|
5677
|
+
if not rebuild:
|
|
5678
|
+
# Accounting timestamps share one producer spelling, so the
|
|
5679
|
+
# lexicographic extremes ARE the chronological ones and only two
|
|
5680
|
+
# rows need parsing. Quota anchors are few per file, so they are
|
|
5681
|
+
# parsed individually.
|
|
5682
|
+
if accounting_rows:
|
|
5683
|
+
for _extreme in (
|
|
5684
|
+
min(_r[2] for _r in accounting_rows),
|
|
5685
|
+
max(_r[2] for _r in accounting_rows),
|
|
5686
|
+
):
|
|
5687
|
+
_extend_codex_touched_span(
|
|
5688
|
+
adoption_spans, discovered.source_root_key,
|
|
5689
|
+
_parse_anchor_iso(_extreme))
|
|
5690
|
+
for _qrow in quota_rows:
|
|
5691
|
+
_extend_codex_touched_span(
|
|
5692
|
+
adoption_spans, _qrow[1], _parse_anchor_iso(_qrow[17]))
|
|
5547
5693
|
|
|
5548
5694
|
# Private test seam (§5.1 post-preflight late-shrink race): fires
|
|
5549
5695
|
# after each file's successful commit, so a race test can shrink a
|
|
@@ -5597,6 +5743,31 @@ def sync_codex_cache(
|
|
|
5597
5743
|
conn.execute("DELETE FROM cache_meta WHERE key = ?",
|
|
5598
5744
|
("codex_torn_auth_deferred",))
|
|
5599
5745
|
conn.commit()
|
|
5746
|
+
# Window-scoped spend adoption (spec
|
|
5747
|
+
# docs/superpowers/specs/2026-07-30-codex-window-scoped-spend-adoption.md).
|
|
5748
|
+
# Runs AFTER the walk committed and while both cache writer flocks are
|
|
5749
|
+
# still held, so the observation evidence and the accounting rows it
|
|
5750
|
+
# stamps are the same committed generation. Cache-only — no stats.db read
|
|
5751
|
+
# — so the lock-order law is untouched. A failure here is never fatal:
|
|
5752
|
+
# the stamp is fully re-derivable, so the next sync (or the migration)
|
|
5753
|
+
# repeats it.
|
|
5754
|
+
try:
|
|
5755
|
+
adopted = apply_codex_window_spend_adoption(
|
|
5756
|
+
conn, touched=None if rebuild else adoption_spans)
|
|
5757
|
+
conn.commit()
|
|
5758
|
+
# Terse, and silent on zero: a rebuild re-derives every row and so
|
|
5759
|
+
# legitimately re-stamps the same population each time, which would
|
|
5760
|
+
# otherwise read as a recurring anomaly rather than convergence.
|
|
5761
|
+
if adopted:
|
|
5762
|
+
eprint(f"[cache-sync] attributed {adopted} Codex row(s) "
|
|
5763
|
+
"from quota windows")
|
|
5764
|
+
except sqlite3.DatabaseError as exc:
|
|
5765
|
+
conn.rollback()
|
|
5766
|
+
if _cctally_db_sib._is_sqlite_corruption_error(exc):
|
|
5767
|
+
# Classified family corruption belongs to the shared recovery
|
|
5768
|
+
# boundary, never to a best-effort local except.
|
|
5769
|
+
raise
|
|
5770
|
+
eprint(f"[cache-sync] could not adopt Codex window spend: {exc}")
|
|
5600
5771
|
# Codex creates/extends cache.db sidecars independently of Claude's
|
|
5601
5772
|
# sync path. Harden them while both cache flocks are still held and
|
|
5602
5773
|
# after all Codex writes, before the optional checkpoint can rotate a
|
|
@@ -5693,6 +5864,188 @@ def sync_codex_cache(
|
|
|
5693
5864
|
return stats
|
|
5694
5865
|
|
|
5695
5866
|
|
|
5867
|
+
_CODEX_ACCOUNT_WEEK = dt.timedelta(
|
|
5868
|
+
minutes=_lib_codex_account_adoption.ACCOUNT_WEEKLY_WINDOW_MINUTES)
|
|
5869
|
+
|
|
5870
|
+
|
|
5871
|
+
def apply_codex_window_spend_adoption(
|
|
5872
|
+
conn: sqlite3.Connection,
|
|
5873
|
+
*,
|
|
5874
|
+
touched: "dict[str, tuple[dt.datetime, dt.datetime]] | None" = None,
|
|
5875
|
+
) -> int:
|
|
5876
|
+
"""Stamp window-derived attribution onto unattributed Codex spend.
|
|
5877
|
+
|
|
5878
|
+
The I/O half of ``_lib_codex_account_adoption``: read the folded window
|
|
5879
|
+
evidence and the candidate rows, hand both to the pure kernel, write back the
|
|
5880
|
+
plan it returns. Cache-only by construction — the window's identified
|
|
5881
|
+
accounts come from ``load_codex_quota_observations`` (which already runs
|
|
5882
|
+
``adopt_unidentified_observations``) and the nominal range is derived from
|
|
5883
|
+
the canonical reset, so no stats.db read is involved and the lock-order law
|
|
5884
|
+
is untouched. The caller owns the transaction and the commit.
|
|
5885
|
+
|
|
5886
|
+
``touched`` maps ``source_root_key`` to the ``(low, high)`` instant span this
|
|
5887
|
+
sync wrote — the timestamps of the accounting rows AND the canonical resets
|
|
5888
|
+
of the quota rows. ``None`` runs the pass over all history (``cache-sync
|
|
5889
|
+
--rebuild`` and the one-time migration); an EMPTY map is a no-op that issues
|
|
5890
|
+
NO SQL AT ALL, which is what keeps a quiescent hook tick free.
|
|
5891
|
+
|
|
5892
|
+
A bounded pass must reach the SAME verdict the unbounded one would, because
|
|
5893
|
+
the stamp is one-way (``NULL`` -> key, never back) and an incremental sync
|
|
5894
|
+
followed by a later rebuild would otherwise disagree. That needs the loaded
|
|
5895
|
+
window set to be a SUPERSET of the windows that can claim any candidate the
|
|
5896
|
+
scan offers, so the two bounds are derived together: windows are loaded for
|
|
5897
|
+
resets in ``[low - 7d, high + 7d]``, and candidates are clamped to
|
|
5898
|
+
``[low - 7d, high]``. Every window claiming an instant ``t`` in that
|
|
5899
|
+
candidate span has its reset in ``(t, t + 7d]``, which the window bound
|
|
5900
|
+
contains — so no window can claim a scanned row unseen. The candidate span
|
|
5901
|
+
still covers everything this sync could have changed: the rows it wrote lie
|
|
5902
|
+
in ``[low, high]``, and a window whose reset it wrote lies in ``[low, high]``
|
|
5903
|
+
too, so that window's whole nominal range lies in ``[low - 7d, high)``.
|
|
5904
|
+
|
|
5905
|
+
Idempotent and re-runnable: ``codex_session_entries`` is fully re-derived on
|
|
5906
|
+
every rebuild, so the pass must re-stamp afterwards, and re-running over an
|
|
5907
|
+
already-stamped cache writes nothing because an identified row is never a
|
|
5908
|
+
candidate. Returns the number of rows actually stamped.
|
|
5909
|
+
"""
|
|
5910
|
+
roots: "set[str] | None" = None
|
|
5911
|
+
reset_bounds: "tuple[dt.datetime, dt.datetime] | None" = None
|
|
5912
|
+
candidate_bounds: "tuple[dt.datetime, dt.datetime] | None" = None
|
|
5913
|
+
if touched is not None:
|
|
5914
|
+
spans = {
|
|
5915
|
+
str(root): span for root, span in touched.items()
|
|
5916
|
+
if root and span is not None
|
|
5917
|
+
}
|
|
5918
|
+
# Before any SQL: an unchanged tree must cost this pass nothing.
|
|
5919
|
+
if not spans:
|
|
5920
|
+
return 0
|
|
5921
|
+
roots = set(spans)
|
|
5922
|
+
low = min(span[0] for span in spans.values())
|
|
5923
|
+
high = max(span[1] for span in spans.values())
|
|
5924
|
+
reset_bounds = (low - _CODEX_ACCOUNT_WEEK, high + _CODEX_ACCOUNT_WEEK)
|
|
5925
|
+
candidate_bounds = (low - _CODEX_ACCOUNT_WEEK, high)
|
|
5926
|
+
|
|
5927
|
+
from _cctally_quota import load_codex_quota_observations
|
|
5928
|
+
|
|
5929
|
+
# `_load_lib`, not a bare import: this module is loadable in isolation, where
|
|
5930
|
+
# `bin/` may not be on `sys.path` (see the module docstring).
|
|
5931
|
+
_lib_accounts = _load_lib("_lib_accounts")
|
|
5932
|
+
is_model_scoped_codex_quota = _load_lib(
|
|
5933
|
+
"_lib_codex_pools").is_model_scoped_codex_quota
|
|
5934
|
+
adopt = _lib_codex_account_adoption
|
|
5935
|
+
try:
|
|
5936
|
+
columns = {
|
|
5937
|
+
str(row[1]) for row in conn.execute(
|
|
5938
|
+
"PRAGMA table_info(codex_session_entries)")
|
|
5939
|
+
}
|
|
5940
|
+
except sqlite3.DatabaseError:
|
|
5941
|
+
return 0
|
|
5942
|
+
if not {"account_key", "source_root_key", "timestamp_utc"} <= columns:
|
|
5943
|
+
return 0
|
|
5944
|
+
|
|
5945
|
+
try:
|
|
5946
|
+
observations = load_codex_quota_observations(
|
|
5947
|
+
source_root_keys=roots, cache_conn=conn,
|
|
5948
|
+
canonical_resets_between=reset_bounds,
|
|
5949
|
+
)
|
|
5950
|
+
except sqlite3.DatabaseError:
|
|
5951
|
+
return 0
|
|
5952
|
+
|
|
5953
|
+
# Group on the SAME key the observation fold groups on
|
|
5954
|
+
# (`_lib_quota._physical_window_key`) — the account is deliberately excluded
|
|
5955
|
+
# from it, which is precisely what makes a window able to name an account for
|
|
5956
|
+
# rows that carry none.
|
|
5957
|
+
buckets: "dict[tuple, dict]" = {}
|
|
5958
|
+
for observation in observations:
|
|
5959
|
+
identity = observation.identity
|
|
5960
|
+
bucket = buckets.get(key := _lib_quota._physical_window_key(observation))
|
|
5961
|
+
if bucket is None:
|
|
5962
|
+
bucket = buckets[key] = {
|
|
5963
|
+
"root": identity.source_root_key,
|
|
5964
|
+
"minutes": identity.window_minutes,
|
|
5965
|
+
"reset": observation.canonical_resets_at,
|
|
5966
|
+
"accounts": set(),
|
|
5967
|
+
"model_scoped": False,
|
|
5968
|
+
}
|
|
5969
|
+
if identity.account_key != _lib_accounts.UNATTRIBUTED:
|
|
5970
|
+
bucket["accounts"].add(identity.account_key)
|
|
5971
|
+
# `limit_name` is compare=False on the identity, so the label can differ
|
|
5972
|
+
# across one group's observations; ANY Spark evidence demotes the whole
|
|
5973
|
+
# window out of account weekly quota (#373). That direction only ever
|
|
5974
|
+
# withholds a stamp, never invents one.
|
|
5975
|
+
if is_model_scoped_codex_quota(
|
|
5976
|
+
identity.logical_limit_key, identity.limit_name):
|
|
5977
|
+
bucket["model_scoped"] = True
|
|
5978
|
+
|
|
5979
|
+
windows: "list[object]" = []
|
|
5980
|
+
root_ranges: "dict[str, list[tuple[dt.datetime, dt.datetime]]]" = {}
|
|
5981
|
+
for bucket in buckets.values():
|
|
5982
|
+
window = adopt.SpendAdoptionWindow(
|
|
5983
|
+
source_root_key=bucket["root"],
|
|
5984
|
+
window_minutes=bucket["minutes"],
|
|
5985
|
+
canonical_resets_at=bucket["reset"],
|
|
5986
|
+
identified_accounts=frozenset(bucket["accounts"]),
|
|
5987
|
+
model_scoped=bucket["model_scoped"],
|
|
5988
|
+
)
|
|
5989
|
+
if not window.in_scope:
|
|
5990
|
+
continue
|
|
5991
|
+
windows.append(window)
|
|
5992
|
+
root_ranges.setdefault(window.source_root_key, []).append(
|
|
5993
|
+
(window.nominal_start_at, window.canonical_resets_at))
|
|
5994
|
+
if not windows:
|
|
5995
|
+
return 0
|
|
5996
|
+
|
|
5997
|
+
# SQL bounds the scan to a coarse per-root union of the candidate windows,
|
|
5998
|
+
# clamped to the span the loaded window set provably covers (see the
|
|
5999
|
+
# docstring); exact half-open containment stays in the kernel. `unixepoch`
|
|
6000
|
+
# deliberately accepts both retained spellings (`Z` and `+00:00`) — the
|
|
6001
|
+
# accounting rows are written with the offset form, the quota rows with `Z`.
|
|
6002
|
+
# Both comparisons are INCLUSIVE on the truncated second: `unixepoch` drops
|
|
6003
|
+
# any sub-second fraction, so an exclusive upper bound would discard rows in
|
|
6004
|
+
# the reset's final second if a canonical anchor ever carried one. Admitting
|
|
6005
|
+
# that second here is free — the kernel re-tests containment exactly.
|
|
6006
|
+
candidates = []
|
|
6007
|
+
for root, spans_for_root in root_ranges.items():
|
|
6008
|
+
window_low = min(span[0] for span in spans_for_root)
|
|
6009
|
+
window_high = max(span[1] for span in spans_for_root)
|
|
6010
|
+
if candidate_bounds is not None:
|
|
6011
|
+
window_low = max(window_low, candidate_bounds[0])
|
|
6012
|
+
window_high = min(window_high, candidate_bounds[1])
|
|
6013
|
+
if window_low > window_high:
|
|
6014
|
+
continue
|
|
6015
|
+
for row in conn.execute(
|
|
6016
|
+
"SELECT id, timestamp_utc FROM codex_session_entries "
|
|
6017
|
+
" WHERE source_root_key = ? "
|
|
6018
|
+
" AND (account_key IS NULL OR account_key = '' "
|
|
6019
|
+
" OR account_key = ?) "
|
|
6020
|
+
" AND unixepoch(timestamp_utc) >= unixepoch(?) "
|
|
6021
|
+
" AND unixepoch(timestamp_utc) <= unixepoch(?)",
|
|
6022
|
+
(root, _lib_accounts.UNATTRIBUTED,
|
|
6023
|
+
_codex_anchor_iso(window_low), _codex_anchor_iso(window_high)),
|
|
6024
|
+
):
|
|
6025
|
+
timestamp = _parse_anchor_iso(row[1])
|
|
6026
|
+
if timestamp is None:
|
|
6027
|
+
continue
|
|
6028
|
+
candidates.append(adopt.SpendAdoptionCandidate(
|
|
6029
|
+
entry_id=int(row[0]), source_root_key=root,
|
|
6030
|
+
timestamp=timestamp, account_key=None,
|
|
6031
|
+
))
|
|
6032
|
+
if not candidates:
|
|
6033
|
+
return 0
|
|
6034
|
+
|
|
6035
|
+
plan = adopt.build_spend_adoption_plan(windows, candidates)
|
|
6036
|
+
if not plan:
|
|
6037
|
+
return 0
|
|
6038
|
+
before = conn.total_changes
|
|
6039
|
+
conn.executemany(
|
|
6040
|
+
"UPDATE codex_session_entries SET account_key = ? "
|
|
6041
|
+
" WHERE id = ? AND (account_key IS NULL OR account_key = '' "
|
|
6042
|
+
" OR account_key = ?)",
|
|
6043
|
+
[(stamp.account_key, stamp.entry_id, _lib_accounts.UNATTRIBUTED)
|
|
6044
|
+
for stamp in plan],
|
|
6045
|
+
)
|
|
6046
|
+
return conn.total_changes - before
|
|
6047
|
+
|
|
6048
|
+
|
|
5696
6049
|
def iter_codex_entries(
|
|
5697
6050
|
conn: sqlite3.Connection,
|
|
5698
6051
|
range_start: dt.datetime,
|