tokenol 0.3.2__tar.gz → 0.4.0__tar.gz

{tokenol-0.3.2 → tokenol-0.4.0}/.gitignore

@@ -1,4 +1,5 @@
 _local/
+.worktrees/
 *_PLAN.md
 *.plan.md
 RESUME.md

{tokenol-0.3.2 → tokenol-0.4.0}/CHANGELOG.md

@@ -4,6 +4,61 @@ All notable changes to tokenol are documented here. The format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and versions follow
 [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 0.4.0 — 2026-05-03
+
+### Features
+- **Persistent history that survives JSONL deletion** (opt-in via
+  `--persist`). `tokenol serve --persist` backs the live in-memory
+  dashboard with a single-file DuckDB store at `~/.tokenol/history.duckdb`
+  (override via `TOKENOL_HISTORY_PATH`). On startup the store seeds the
+  in-memory hot tier so cold start is bounded by `hot_window_days`
+  (default 90), not by total history length. Each tick parses only JSONLs
+  whose `mtime_ns` exceeds the per-session high-water mark — typically
+  just today's active files — and appends derived turns to both memory and
+  a background batch flush (every 30 s or 100 turns). Deleting a JSONL no
+  longer drops its data from the dashboard; the affected sessions are
+  marked `archived=True` and continue to render every quantitative panel.
+  Only the per-turn modal's content snippets (user prompt, assistant
+  preview, tool-call list) become unavailable for archived sessions, in
+  line with the privacy intent of the deletion. Default off — `tokenol
+  serve` without `--persist` matches the v0.3.2 resource profile
+  byte-for-byte (no `import duckdb`, no `~/.tokenol/` directory, no extra
+  steady RSS).
+- `Preferences.hot_window_days` (default `90`, accepted range `1..3650`),
+  exposed via the existing `/api/prefs` endpoint. Takes effect on next
+  startup.
+- `Session.archived: bool` field surfaced through `/api/session/{id}` and
+  `/api/session/{id}/turn/{idx}`; the session-detail page renders an
+  amber "Archived — text snippets unavailable" badge and hides the
+  per-turn snippet block when the flag is set.
+- `tokenol.persistence.forget_handoff` — a pidfile + atomic request-file
+  handshake so a future `tokenol forget` CLI (PR 2) can apply deletions
+  to a live serve within one tick, without requiring a restart.
+
+### Changes
+- `duckdb` moved from a core dependency to the new `[persist]` optional
+  extras group. Default `pip install tokenol` no longer pulls the DuckDB
+  binary wheel (~30 MB saved). Users who pass `--persist` install with
+  `pip install 'tokenol[persist]'`.
+- `build_snapshot_full` now accepts optional `history_store` and
+  `flush_queue` arguments. When neither is supplied the legacy whole-corpus
+  derivation path is used unchanged, so CLI report commands and any
+  existing test that constructs a bare `ParseCache` keep working.
+- Default mode prints a yellow `WARNING` at startup if it finds an existing
+  `~/.tokenol/history.duckdb` (or `TOKENOL_HISTORY_PATH`), prompting the
+  user to pass `--persist` if they want to use it (rather than silently
+  ignoring the file).
+- `select_edge_paths` now tracks per-file `mtime_ns` instead of comparing
+  filesystem mtime to turn timestamps — fixes a freshness bug in the
+  store-backed snapshot path where backdated turn timestamps could silently
+  exclude files from re-parse.
+
+### Notes
+- See `docs/superpowers/specs/2026-05-03-opt-in-persistence-design.md` for
+  the gating-and-extras design.
+- See `docs/superpowers/specs/2026-05-02-persistent-history-design.md` for
+  the underlying store design.
+
 ## [0.3.2] — 2026-04-28
 
 ### Fixed

{tokenol-0.3.2 → tokenol-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tokenol
-Version: 0.3.2
+Version: 0.4.0
 Summary: Audit Claude Code JSONL session logs: track cost, cache health, context blow-ups, and 5h-window pressure.
 Project-URL: Homepage, https://github.com/farhanferoz/tokenol
 Project-URL: Issues, https://github.com/farhanferoz/tokenol/issues
@@ -15,7 +15,6 @@ Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Topic :: Utilities
 Requires-Python: >=3.10
-Requires-Dist: duckdb>=0.10
 Requires-Dist: rich>=13
 Requires-Dist: typer>=0.12
 Provides-Extra: dev
@@ -25,6 +24,8 @@ Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
 Requires-Dist: pytest-cov; extra == 'dev'
 Requires-Dist: pytest>=8; extra == 'dev'
 Requires-Dist: ruff>=0.4; extra == 'dev'
+Provides-Extra: persist
+Requires-Dist: duckdb>=0.10; extra == 'persist'
 Provides-Extra: serve
 Requires-Dist: fastapi>=0.111; extra == 'serve'
 Requires-Dist: uvicorn[standard]>=0.29; extra == 'serve'
@@ -66,9 +67,9 @@ Project page — cache efficiency trend, verdict distribution, top turns:
 
 ## Install
 
-```bash
-pipx install tokenol
-```
+    pip install tokenol                        # CLI commands (daily, sessions, projects, ...)
+    pip install 'tokenol[serve]'               # adds the live dashboard (tokenol serve)
+    pip install 'tokenol[serve,persist]'       # adds DuckDB-backed history that survives JSONL deletion
 
 Requires Python 3.10+. See [tokenol on PyPI](https://pypi.org/project/tokenol/).
 
@@ -152,6 +153,31 @@ The dashboard updates via SSE as Claude Code writes events to disk. The server g
 
 If SSE delivery silently stalls (browser tab throttling, extension hooks, long-lived `EventSource` quirks), the client self-heals: it polls `/api/snapshot` every 30 s as a backstop, force-reconnects on tab-visibility return, and runs a 90 s staleness watchdog. `/api/snapshot` reuses the broadcaster's cached payload while an SSE group is live, so the backstop costs only a JSON serialize. Hover the live-status dot for a "last update Ns ago" indicator.
 
+### Persistent history (opt-in)
+
+By default, `tokenol serve` parses your `~/.claude*/projects/**/*.jsonl` files into an in-memory model on each restart — fast, but the dashboard loses any session whose JSONL has been deleted or rotated.
+
+Pass `--persist` to enable a DuckDB-backed history store at `~/.tokenol/history.duckdb` (override with `TOKENOL_HISTORY_PATH`). The store contains **no message content** — only token counts, costs, models, timestamps, tool counts, and session metadata, comparable to a billing receipt. With persistence on:
+
+- **Deleting a JSONL no longer drops it from the dashboard.** Quantitative panels render as before; only the per-turn modal's verbatim content snippets become unavailable, indicated by an "Archived — text snippets unavailable" badge. Metrics survive; words don't (matching the privacy intent of the deletion).
+- **Restart picks up where you left off.** A background flusher batches writes (every 30 s or 100 turns, whichever first) and force-drains on graceful shutdown. The JSONLs remain the durable substrate — a process crash mid-flush loses nothing because the next start re-derives the missing window from the JSONLs (idempotent on `message.id:requestId`).
+- **Cold start stays bounded.** The hot tier loads only the last `hot_window_days` of turns (default 90, tunable via `/api/prefs`); older history is read on demand from the warm tier.
+
+Measured cost on the author's full `~/.claude*` corpus (~1820 files, ~2 GB of JSONLs, page cache cold both runs):
+
+| | Default mode | `--persist` first start | `--persist` subsequent starts |
+|---|---|---|---|
+| Time to first paint | ~5 s | ~12 s | ~12 s |
+| Wall to settle | ~5 s | ~4 min (one-time backfill) | <30 s |
+| Steady RSS | ~250 MiB | — | ~750 MiB |
+| Durable disk | 0 | ~40 MB after backfill | grows incrementally |
+
+Requires the persist extras (`pip install 'tokenol[serve,persist]'`).
+
+See `docs/superpowers/specs/2026-05-03-opt-in-persistence-design.md` for design rationale and `docs/superpowers/specs/2026-05-02-persistent-history-design.md` for the underlying store design.
+
+### Main dashboard
+
 Main page layout (top to bottom):
 
 | Panel | What it shows |

{tokenol-0.3.2 → tokenol-0.4.0}/README.md

@@ -33,9 +33,9 @@ Project page — cache efficiency trend, verdict distribution, top turns:
 
 ## Install
 
-```bash
-pipx install tokenol
-```
+    pip install tokenol                        # CLI commands (daily, sessions, projects, ...)
+    pip install 'tokenol[serve]'               # adds the live dashboard (tokenol serve)
+    pip install 'tokenol[serve,persist]'       # adds DuckDB-backed history that survives JSONL deletion
 
 Requires Python 3.10+. See [tokenol on PyPI](https://pypi.org/project/tokenol/).
 
@@ -119,6 +119,31 @@ The dashboard updates via SSE as Claude Code writes events to disk. The server g
 
 If SSE delivery silently stalls (browser tab throttling, extension hooks, long-lived `EventSource` quirks), the client self-heals: it polls `/api/snapshot` every 30 s as a backstop, force-reconnects on tab-visibility return, and runs a 90 s staleness watchdog. `/api/snapshot` reuses the broadcaster's cached payload while an SSE group is live, so the backstop costs only a JSON serialize. Hover the live-status dot for a "last update Ns ago" indicator.
 
+### Persistent history (opt-in)
+
+By default, `tokenol serve` parses your `~/.claude*/projects/**/*.jsonl` files into an in-memory model on each restart — fast, but the dashboard loses any session whose JSONL has been deleted or rotated.
+
+Pass `--persist` to enable a DuckDB-backed history store at `~/.tokenol/history.duckdb` (override with `TOKENOL_HISTORY_PATH`). The store contains **no message content** — only token counts, costs, models, timestamps, tool counts, and session metadata, comparable to a billing receipt. With persistence on:
+
+- **Deleting a JSONL no longer drops it from the dashboard.** Quantitative panels render as before; only the per-turn modal's verbatim content snippets become unavailable, indicated by an "Archived — text snippets unavailable" badge. Metrics survive; words don't (matching the privacy intent of the deletion).
+- **Restart picks up where you left off.** A background flusher batches writes (every 30 s or 100 turns, whichever first) and force-drains on graceful shutdown. The JSONLs remain the durable substrate — a process crash mid-flush loses nothing because the next start re-derives the missing window from the JSONLs (idempotent on `message.id:requestId`).
+- **Cold start stays bounded.** The hot tier loads only the last `hot_window_days` of turns (default 90, tunable via `/api/prefs`); older history is read on demand from the warm tier.
+
+Measured cost on the author's full `~/.claude*` corpus (~1820 files, ~2 GB of JSONLs, page cache cold both runs):
+
+| | Default mode | `--persist` first start | `--persist` subsequent starts |
+|---|---|---|---|
+| Time to first paint | ~5 s | ~12 s | ~12 s |
+| Wall to settle | ~5 s | ~4 min (one-time backfill) | <30 s |
+| Steady RSS | ~250 MiB | — | ~750 MiB |
+| Durable disk | 0 | ~40 MB after backfill | grows incrementally |
+
+Requires the persist extras (`pip install 'tokenol[serve,persist]'`).
+
+See `docs/superpowers/specs/2026-05-03-opt-in-persistence-design.md` for design rationale and `docs/superpowers/specs/2026-05-02-persistent-history-design.md` for the underlying store design.
+
+### Main dashboard
+
 Main page layout (top to bottom):
 
 | Panel | What it shows |