tledger 0.3.0 → 0.4.0

package/docs/durable-ledger-operations.md

@@ -0,0 +1,198 @@
+# Durable ledger operations
+
+Token Ledger keeps two local artifacts with different responsibilities:
+
+- `token-ledger-ledger.sqlite` is the durable, deduplicated source of truth.
+- `token-ledger-snapshot-v3.json.gz` is a bounded report cache that can be
+  regenerated from the ledger and currently available Codex sources.
+
+The durable database, writer guard, WAL, journal, and shared-memory files live
+only in the app-owned private `~/.token-ledger` directory. Snapshot exports may
+be written elsewhere, but output selection never relocates SQLite state.
+
+Deleting or replacing the report cache must not be treated as deleting ledger
+history. Conversely, a successfully parsed report cache is not proof that its
+revision matches the current ledger.
+
+## Health signals
+
+Generated snapshots expose the durable schema version, revision, quota-identity
+contract, hashed Codex home identity, retention horizons,
+`legacySnapshotStatus`, and separate legacy-quota diagnostics under
+`metadata.durableLedger`. Coverage also includes source-state counts,
+`sourceIncomplete`, compacted and migrated bucket counts, and the same legacy
+statuses.
+
+The main legacy statuses are:
+
+- `migrated`: the legacy v3 snapshot had a provable collection scope and a
+  matching hashed Codex-home identity, so its history was imported as
+  estimated compacted rows.
+- `collection-scope-unverified`: the snapshot did not prove whether it included
+  archived sources or a time cutoff.
+- `codex-home-unverified`: the snapshot did not carry a verifiable hashed Codex
+  home identity.
+- `codex-home-mismatch`: the snapshot belonged to a different Codex home.
+
+The last three statuses mean legacy history was deliberately excluded. Exact
+rollout collection still proceeds, and PNG reports show `LEGACY HISTORY
+SKIPPED` so the omission is not silent.
+
+Legacy quota rows have a narrower gate than usage history. They migrate only
+when the source snapshot explicitly declares the current quota-identity
+contract. `legacyQuotaStatus` reports `migrated`, `not-present`,
+`skipped-contract-unverified`, `skipped-contract-mismatch`, or
+`skipped-invalid`. A whole-snapshot provenance-gate failure instead reports
+`skipped-<legacySnapshotStatus>`, such as
+`skipped-collection-scope-unverified`, `skipped-codex-home-unverified`, or
+`skipped-codex-home-mismatch`. `legacyQuotaRowsSkipped` records the excluded
+array length; zero with `skipped-invalid` means the malformed non-array shape
+had no reliable row count. Markerless, older, or semantically invalid quota
+rows are never guessed or re-keyed; safe usage and thread history still
+migrate.
+
+A missing legacy snapshot is a completed absence check. By contrast, an
+existing malformed usage/thread history, unreadable, oversized, non-regular,
+or non-v3 snapshot
+stops with `ERR_DURABLE_LEDGER_LEGACY_SNAPSHOT`. That failure does not advance
+the ledger revision, mark the one-shot migration as checked, or publish a new
+cache. Preserve the unreadable artifact in a private backup, then repair or
+replace it with the matching legacy v3 snapshot and retry. If migration is
+intentionally declined, move the preserved artifact out of the configured
+snapshot path; the next successful refresh records that absence as checked.
+
+`sourceIncomplete` means a previously observed source is missing, tombstoned,
+truncated, or replaced. It is a provenance warning, not proof that SQLite is
+corrupt. A true append keeps stable provenance; a larger same-inode rewrite is
+classified as a replacement rather than an append.
+
+A source scan with malformed JSON or an invalid token record is evidence-only:
+its coverage counters and source watermark are retained, but none of its token,
+quota, tool, position, ownership, origin, or thread operations can change the
+last complete durable interpretation. The affected file's usage is excluded
+from totals until it scans cleanly. To recover, fix or remove the malformed
+line, then refresh. If that scan also detected a replacement, the source state
+records a pending reconciliation. The bit survives lifecycle moves, missing or
+tombstoned state, and truncation, then clears atomically only after a clean
+complete scan reconciles every source-owned membership and position.
+
+Quota labels are stored separately as explicit, timestamped source evidence.
+Malformed quota scans cannot update that evidence. Missing or tombstoned
+sources are ineligible at read time, and archived-source labels are ineligible
+under `--no-archived`; observation rows are never destructively relabeled.
+
+## Schema and privacy upgrades
+
+Schema-v1 preview ledgers with reconstructable migration scope upgrade to v2
+inside one SQLite transaction. The upgrade removes stored working directories,
+Git remotes, and raw source values, then compacts the database so those bytes do
+not remain in free pages. New v2 writes store no raw versions of those values.
+
+Schema v3 keeps the durable source-position index bounded: `event_key` in
+`source_event_positions` is now a 64-character SHA-256 digest, while the full
+event key remains in the observation or compaction-membership row that owns it.
+Existing v2 ledgers rewrite those position values once on first write. New
+ledgers use SQLite incremental auto-vacuum, and successful writes reclaim a
+large freelist when it crosses both an absolute and percentage threshold;
+small or busy databases defer maintenance to a later write.
+
+During importer commits, durable usage and tool rows are materialized through a
+synchronous SQLite cursor into the disk-backed usage spool. The public
+`readDurableLedger()` result remains array-shaped for callers, but the refresh
+hot path retains only the current row and one compacted row's membership
+iterator at a time. Legacy migrated buckets are the bounded exception because
+their residuals must remain available while current observations are compared.
+Quota and thread metadata continue to use their existing collection paths; a
+future change can stream those independently if their retention grows enough
+to justify it.
+
+The quota-identity contract upgrades separately. Opening a markerless ledger or
+the prior `codex-limit-id-v1` contract for a write transaction discards all
+pre-contract exact and migrated quota rows with their memberships, then records
+the current contract atomically. Only a subsequent clean source scan can
+repopulate canonical quota history. Rollback restores both the prior rows and
+prior marker. An unknown or future contract stops before mutation with
+`ERR_DURABLE_LEDGER_QUOTA_CONTRACT`; do not downgrade or rewrite that ledger.
+
+Quota identity has one irreducible local boundary. Codex rollout
+`RateLimitSnapshot` records carry a provider limit id, but no ChatGPT user or
+account id. Token Ledger therefore separates quota pools by the canonical
+provider limit id only within one account identity per `CODEX_HOME`; an omitted
+or blank id is the default `codex` pool. Limit names and plan labels are display
+metadata, never identity. Reusing one `CODEX_HOME` and durable ledger across
+different ChatGPT users can stitch equal `codex` ids into one quota timeline;
+use separate Codex homes and ledger locations when account isolation matters.
+
+An early schema-v1 ledger that contains migrated history without a
+reconstructable scope stops with `ERR_DURABLE_LEDGER_MIGRATION_SCOPE`. That
+database is left unchanged because guessing its scope could either double-count
+or erase usage.
+
+## Crash safety and non-destructive repair
+
+1. Stop concurrent Token Ledger refreshes and copy the ledger plus any SQLite
+   `-wal` and `-shm` sidecars to a private backup location.
+2. For `ERR_DURABLE_LEDGER_CODEX_HOME`, select the Codex home that originally
+   created the ledger. Do not rebind the ledger to a different home; use a
+   separate OS user profile when account-level isolation is required.
+3. For `ERR_DURABLE_LEDGER_MIGRATION_SCOPE`, keep the preview ledger as a
+   backup. Recreate the private app-owned ledger and rebuild exact history from
+   the matching Codex sources. Import a legacy snapshot only when its collection
+   scope and Codex-home fingerprint are both known.
+4. For a stale or oversized report cache, preserve the ledger and retry with a
+   compressed output, `--since`, or `--no-archived`. Removing only the cache is
+   safe when a rebuild is desired.
+5. Treat missing or rewritten rollout history as incomplete provenance. Do not
+   delete the ledger to make that warning disappear.
+
+Snapshot encoding is staged in a private temporary file. Source watermarks are
+validated twice before the SQLite commit and once immediately after it. The
+SQLite commit atomically writes a complete candidate revision. That candidate
+remains pending until the post-commit source check succeeds and its recovery
+marker is cleared. Coordinated readers use the same writer guard, so they
+never observe an in-progress transaction.
+
+The staged report cache is published only after post-commit source validation
+succeeds and the pending marker is cleared. If sources changed after the final
+pre-commit check, the candidate is reverted through its undo log, the staged
+cache is discarded, and collection retries from the new source inventory.
+An interrupted validation or failed marker-clear transaction also leaves a
+pending candidate that the next writer reverts before collecting again.
+Observations from such an incomplete refresh are not accepted history; their
+recovery requires the source files to remain available.
+
+A crash before commit is rolled back by SQLite. A crash after the candidate
+commit but before finalization can leave that candidate visible to a direct
+ledger reader, but it remains provisional and will be unwound on refresh.
+After finalization, a cache-publication failure leaves the accepted ledger
+ahead of the prior cache. The cache's revision mismatch forces an automatic
+refresh instead of labeling it verified-current. Successfully finalized
+history remains available when its source files disappear.
+
+Before staging a replacement cache, Token Ledger also removes a
+same-destination temporary file only when it is an ordinary, single-link file
+owned by the current user and its recorded process is demonstrably gone. No
+ledger-sized baseline or restore copy is created during refresh.
+
+## Repeatable refresh benchmark
+
+The benchmark can generate actual old token-count events, prove that the
+durable ledger compacted them, verify the durable token total and revision, and
+measure cold plus warm refreshes:
+
+```bash
+for events in 100 1000 5000 10000; do
+  npm run benchmark:refresh -- \
+    --files 24 \
+    --token-events "$events" \
+    --warm-runs 3 \
+    --event-age-days 4000
+done
+```
+
+Record the operating system, CPU, memory, Node and npm versions with the result.
+Use `warmMedianWallTimeMs` for the steady-state refresh comparison and retain
+`coldWallTimeMs` separately. A valid compaction run has nonzero
+`durableCompactedBuckets`, a `durableRevision` equal to `warmRuns`, the expected
+`durableTotalTokens`, and zero parse errors. Running several benchmarks in
+parallel invalidates timing comparisons.

package/docs/release-notes-0.4.0.md

@@ -0,0 +1,41 @@
+# Token Ledger 0.4.0
+
+## Highlights
+
+- One-day PNG reports now show hourly token usage and cache efficiency, with
+  a partial current hour and support for daylight-saving transitions.
+- Add `--private` to a report to replace project names with ranked anonymous
+  labels. Usage figures stay visible, the source snapshot stays unchanged,
+  and the default output filename ends in `-private.png`.
+- Recognize Astra in usage reports. Models without bundled prices remain
+  explicitly unrated in purchased-credit and API-equivalent estimates.
+- Keep model names and token amounts readable in narrow bars, with floating
+  labels for small segments and captions for small model-mix shares.
+- Show unrecorded fast-mode status separately from confirmed fast usage.
+
+## Correctness and reliability
+
+- Allocate compacted usage across local calendar and hourly boundaries, retain
+  persisted call counts, and compare equivalent partial reporting periods.
+- Keep report cutoffs fixed during collection and disclose allocated estimates.
+- Repair token timestamps that precede their own recorded turn, preserve token
+  totals, and mark the replacement time as estimated.
+- Reject impossible quota-window timestamps so valid meter history remains
+  visible, and keep meter attribution separate from daily token precision.
+- Correct detailed-call coverage for total-only records while retaining valid
+  partial coverage in compacted history.
+- Preserve competing snapshot replacements during publication checks.
+- Redact local project paths during legacy migration and report materialization,
+  and redact local paths in CLI and collector diagnostics.
+- Reject malformed quota readings, handle fragmented terminal key sequences,
+  and tolerate XML-invalid label characters during PNG encoding.
+- Improve cross-month labels, narrow chart spacing, peak-hour labels, and
+  fallback to token bars when observed drain is unavailable.
+- Preserve persisted thread-title updates and isolate benchmark state from the
+  live ledger.
+
+## Release checks and requirements
+
+Requires Node.js 22.13 or newer. CI and prepublish checks include regular and
+stress tests, lint, and offline installation of the packed artifact with CLI
+and PNG smoke tests. Token Ledger remains a local-only CLI.

package/lib/token-ledger-calendar.mjs

@@ -0,0 +1,225 @@
+const DAY_MS = 24 * 60 * 60 * 1_000;
+const BOUNDARY_SEARCH_RADIUS_MS = 3 * DAY_MS;
+const BOUNDARY_SAMPLE_STEP_MS = 6 * 60 * 60 * 1_000;
+const BOUNDARY_PROBE_STEP_MS = 60 * 60 * 1_000;
+const MAX_BOUNDARY_PROBES = 72;
+const CALENDAR_DATE_PATTERN = /^\d{4}-\d{2}-\d{2}$/;
+
+function dateStringFromParts(year, month, day) {
+  return [year, month, day]
+    .map((value, index) => String(value).padStart(index === 0 ? 4 : 2, "0"))
+    .join("-");
+}
+
+function parseCalendarDate(dateString) {
+  if (!CALENDAR_DATE_PATTERN.test(dateString)) {
+    throw new Error(`Invalid calendar date: ${dateString}`);
+  }
+  const [year, month, day] = dateString.split("-").map(Number);
+  const check = new Date(0);
+  check.setUTCFullYear(year, month - 1, day);
+  check.setUTCHours(0, 0, 0, 0);
+  if (
+    check.getUTCFullYear() !== year ||
+    check.getUTCMonth() + 1 !== month ||
+    check.getUTCDate() !== day
+  ) {
+    throw new Error(`Invalid calendar date: ${dateString}`);
+  }
+  return { year, month, day };
+}
+
+function calendarDate(dateString) {
+  const { year, month, day } = parseCalendarDate(dateString);
+  const date = new Date(0);
+  date.setUTCFullYear(year, month - 1, day);
+  date.setUTCHours(0, 0, 0, 0);
+  return date;
+}
+
+function utcMilliseconds(year, month, day, hour, minute, second) {
+  const date = new Date(0);
+  date.setUTCFullYear(year, month - 1, day);
+  date.setUTCHours(hour, minute, second, 0);
+  return date.getTime();
+}
+
+function numericDateTimeParts(timestampMs, formatter) {
+  const parts = formatter.formatToParts(new Date(timestampMs));
+  return Object.fromEntries(
+    parts
+      .filter((part) => part.type !== "literal" && part.type !== "timeZoneName")
+      .map((part) => [part.type, Number(part.value)]),
+  );
+}
+
+function timeZoneOffsetMs(timestampMs, formatter) {
+  const values = numericDateTimeParts(timestampMs, formatter);
+  const instantSecondMs = Math.floor(timestampMs / 1_000) * 1_000;
+  return (
+    utcMilliseconds(
+      values.year,
+      values.month,
+      values.day,
+      values.hour,
+      values.minute,
+      values.second,
+    ) - instantSecondMs
+  );
+}
+
+function localDateStringWithFormatter(timestampMs, formatter) {
+  const values = numericDateTimeParts(timestampMs, formatter);
+  return dateStringFromParts(values.year, values.month, values.day);
+}
+
+function findEarliestLocalDateInstant(candidateMs, dateString, formatter) {
+  let high = candidateMs;
+  let low = high - BOUNDARY_PROBE_STEP_MS;
+  let probes = 0;
+  while (
+    localDateStringWithFormatter(low, formatter) === dateString &&
+    probes < MAX_BOUNDARY_PROBES
+  ) {
+    high = low;
+    low -= BOUNDARY_PROBE_STEP_MS;
+    probes += 1;
+  }
+  if (localDateStringWithFormatter(low, formatter) === dateString) {
+    throw new Error(`Could not find the start of local date ${dateString}.`);
+  }
+  while (high - low > 1) {
+    const middle = Math.floor((low + high) / 2);
+    if (localDateStringWithFormatter(middle, formatter) === dateString) {
+      high = middle;
+    } else {
+      low = middle;
+    }
+  }
+  return high;
+}
+
+function candidateOffsets(utcGuess, formatter) {
+  // Keep both sides of a nearby offset transition available. A fixed-point
+  // midnight guess alone can land in the preceding local date when midnight
+  // was skipped.
+  const offsets = new Set();
+  for (
+    let delta = -BOUNDARY_SEARCH_RADIUS_MS;
+    delta <= BOUNDARY_SEARCH_RADIUS_MS;
+    delta += BOUNDARY_SAMPLE_STEP_MS
+  ) {
+    offsets.add(timeZoneOffsetMs(utcGuess + delta, formatter));
+  }
+
+  let probe = utcGuess;
+  for (let index = 0; index < 8; index += 1) {
+    const offset = timeZoneOffsetMs(probe, formatter);
+    offsets.add(offset);
+    const next = utcGuess - offset;
+    if (next === probe) break;
+    probe = next;
+  }
+  return offsets;
+}
+
+export function createTimeZoneFormatter(timeZone) {
+  return new Intl.DateTimeFormat("en-US", {
+    timeZone,
+    timeZoneName: "longOffset",
+    year: "numeric",
+    month: "2-digit",
+    day: "2-digit",
+    hour: "2-digit",
+    minute: "2-digit",
+    second: "2-digit",
+    hourCycle: "h23",
+  });
+}
+
+export function validateTimeZone(timeZone) {
+  try {
+    createTimeZoneFormatter(timeZone);
+  } catch {
+    throw new Error(`Unknown IANA timezone: ${timeZone}`);
+  }
+}
+
+export function shiftCalendarDate(dateString, amount) {
+  const { year, month, day } = parseCalendarDate(dateString);
+  const delta = Number(amount);
+  if (!Number.isSafeInteger(delta)) {
+    throw new Error(`Calendar date shift must be an integer: ${amount}`);
+  }
+  const date = new Date(0);
+  date.setUTCFullYear(year, month - 1, day + delta);
+  date.setUTCHours(0, 0, 0, 0);
+  return dateStringFromParts(
+    date.getUTCFullYear(),
+    date.getUTCMonth() + 1,
+    date.getUTCDate(),
+  );
+}
+
+export function localDateString(timestampMs, timeZone, formatter) {
+  const dateFormatter = formatter ?? createTimeZoneFormatter(timeZone);
+  return localDateStringWithFormatter(timestampMs, dateFormatter);
+}
+
+export function todayInTimeZone(timeZone, formatter) {
+  return localDateString(Date.now(), timeZone, formatter);
+}
+
+// The boundary is the first valid instant belonging to the requested local
+// date. If a time-zone transition skips the entire date, the boundary is
+// collapsed onto the next representable date, so the skipped date is empty.
+// When midnight is repeated, the earliest occurrence is selected.
+export function localDateBoundary(dateString, timeZone, formatter) {
+  parseCalendarDate(dateString);
+  const dateFormatter = formatter ?? createTimeZoneFormatter(timeZone);
+  let boundaryDateString = dateString;
+
+  for (let skippedDays = 0; skippedDays <= 370; skippedDays += 1) {
+    const utcGuess = calendarDate(boundaryDateString).getTime();
+    const starts = [];
+    for (const offset of candidateOffsets(utcGuess, dateFormatter)) {
+      const candidateMs = utcGuess - offset;
+      if (
+        localDateStringWithFormatter(candidateMs, dateFormatter) !==
+        boundaryDateString
+      ) {
+        continue;
+      }
+      starts.push(
+        findEarliestLocalDateInstant(
+          candidateMs,
+          boundaryDateString,
+          dateFormatter,
+        ),
+      );
+    }
+    if (starts.length > 0) return new Date(Math.min(...starts));
+    boundaryDateString = shiftCalendarDate(boundaryDateString, 1);
+  }
+
+  throw new Error(`Could not resolve a local calendar boundary near ${dateString}.`);
+}
+
+export function formatCalendarDate(dateString, options) {
+  return new Intl.DateTimeFormat("en-US", {
+    ...options,
+    timeZone: "UTC",
+  }).format(calendarDate(dateString));
+}
+
+export function calendarDateParts(dateString, options) {
+  const parts = new Intl.DateTimeFormat("en-US", {
+    ...options,
+    timeZone: "UTC",
+  }).formatToParts(calendarDate(dateString));
+  return Object.fromEntries(
+    parts
+      .filter((part) => part.type !== "literal")
+      .map((part) => [part.type, part.value]),
+  );
+}

package/lib/token-ledger-collection.mjs

@@ -0,0 +1,100 @@
+const ISO_TIMESTAMP_PATTERN =
+  /^(?<year>\d{4})-(?<month>\d{2})-(?<day>\d{2})T(?<hour>\d{2}):(?<minute>\d{2})(?::(?<second>\d{2})(?:\.(?<fraction>\d+))?)?(?<timeZone>Z|[+-]\d{2}:\d{2})$/;
+
+function invalidCollectionSince() {
+  return new Error("--since requires a valid ISO timestamp.");
+}
+
+export function normalizeCollectionSince(value) {
+  if (value === null || value === undefined) return null;
+  if (value instanceof Date) {
+    if (!Number.isFinite(value.getTime())) throw invalidCollectionSince();
+    return value.toISOString();
+  }
+
+  const match = ISO_TIMESTAMP_PATTERN.exec(value);
+  if (!match) throw invalidCollectionSince();
+  const date = new Date(value);
+  if (!Number.isFinite(date.getTime())) throw invalidCollectionSince();
+
+  const components = match.groups;
+  const year = Number(components.year);
+  const month = Number(components.month);
+  const day = Number(components.day);
+  const hour = Number(components.hour);
+  const minute = Number(components.minute);
+  const second = components.second ? Number(components.second) : 0;
+  const milliseconds = components.fraction
+    ? Number(components.fraction.slice(0, 3).padEnd(3, "0"))
+    : 0;
+  const timeZone = components.timeZone;
+  const offsetMinutes =
+    timeZone === "Z"
+      ? 0
+      : (timeZone[0] === "-" ? -1 : 1) *
+        (Number(timeZone.slice(1, 3)) * 60 + Number(timeZone.slice(4, 6)));
+  const localDate = new Date(date.getTime() + offsetMinutes * 60 * 1_000);
+  if (
+    localDate.getUTCFullYear() !== year ||
+    localDate.getUTCMonth() + 1 !== month ||
+    localDate.getUTCDate() !== day ||
+    localDate.getUTCHours() !== hour ||
+    localDate.getUTCMinutes() !== minute ||
+    localDate.getUTCSeconds() !== second ||
+    localDate.getUTCMilliseconds() !== milliseconds
+  ) {
+    throw invalidCollectionSince();
+  }
+  return date.toISOString();
+}
+
+export function collectionScope(options = {}) {
+  return {
+    since: normalizeCollectionSince(options.since),
+    includeArchived: options.includeArchived !== false,
+  };
+}
+
+export function snapshotCollectionScope(snapshot = {}) {
+  const collection = snapshot?.provenance?.collection;
+  if (!collection) return null;
+  if (
+    !Object.prototype.hasOwnProperty.call(collection, "since") ||
+    (collection.includeArchived !== true && collection.includeArchived !== false)
+  ) {
+    return null;
+  }
+  let since;
+  try {
+    since = normalizeCollectionSince(collection.since);
+  } catch {
+    return null;
+  }
+  if (since !== collection.since) return null;
+  return { since, includeArchived: collection.includeArchived };
+}
+
+export function snapshotMatchesCollectionScope(snapshot, scope) {
+  const actual = snapshotCollectionScope(snapshot);
+  return Boolean(
+    actual &&
+      actual.since === scope.since &&
+      actual.includeArchived === scope.includeArchived,
+  );
+}
+
+export function historyScopeLabel(snapshot = {}) {
+  const scope = snapshotCollectionScope(snapshot);
+  if (!scope || (scope.since === null && scope.includeArchived)) return null;
+
+  const details = [];
+  if (scope.since !== null) details.push(`before ${scope.since}`);
+  if (!scope.includeArchived) details.push("archived sessions excluded");
+  return `TRUNCATED HISTORY${details.length ? ` · ${details.join(" · ")}` : ""}`;
+}
+
+export function snapshotCollectionCutoffMs(snapshot = {}) {
+  const scope = snapshotCollectionScope(snapshot);
+  if (!scope?.since) return null;
+  return Date.parse(scope.since);
+}