tokenol 0.3.1__tar.gz → 0.3.2__tar.gz

{tokenol-0.3.1 → tokenol-0.3.2}/CHANGELOG.md

@@ -4,6 +4,20 @@ All notable changes to tokenol are documented here. The format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and versions follow
 [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.3.2] — 2026-04-28
+
+### Fixed
+- **Dashboard fallback endpoints no longer freeze on stale turn counts.**
+  The 0.3.1 `/api/snapshot` fast-path stopped refreshing
+  `app.state.snapshot_result`, so every endpoint that fell back to it
+  (`/api/hourly`, `/api/daily`, `/api/models`, `/api/recent`, `/api/session/*`,
+  `/api/project/*`, `/api/breakdown/*`, `/api/search`, `/api/model/*`,
+  `/api/tool/*`) silently froze at first-page-load turn counts. Symptom: the
+  HIT% panel showed live turns but switching to $/KW, CTX, Cache Reuse, Output,
+  or Cost rendered stale numbers. Fix: `SnapshotBroadcaster` now exposes its
+  freshest `SnapshotResult`, and fallback endpoints prefer it over the
+  app-level cache.
+
 ## [0.3.1] — 2026-04-27
 
 ### Fixed

{tokenol-0.3.1 → tokenol-0.3.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tokenol
-Version: 0.3.1
+Version: 0.3.2
 Summary: Audit Claude Code JSONL session logs: track cost, cache health, context blow-ups, and 5h-window pressure.
 Project-URL: Homepage, https://github.com/farhanferoz/tokenol
 Project-URL: Issues, https://github.com/farhanferoz/tokenol/issues
@@ -150,6 +150,8 @@ tokenol serve --open
 
 The dashboard updates via SSE as Claude Code writes events to disk. The server gates rebuilds on JSONL file changes — when no files have changed, it idles at near-zero CPU and forces a refresh at most once a minute (so time-windowed panels like Recent Activity don't drift more than ~60 s from wall clock). Multiple browser tabs share a single producer, so opening more tabs does not multiply server cost.
 
+If SSE delivery silently stalls (browser tab throttling, extension hooks, long-lived `EventSource` quirks), the client self-heals: it polls `/api/snapshot` every 30 s as a backstop, force-reconnects on tab-visibility return, and runs a 90 s staleness watchdog. `/api/snapshot` reuses the broadcaster's cached payload while an SSE group is live, so the backstop costs only a JSON serialize. Hover the live-status dot for a "last update Ns ago" indicator.
+
 Main page layout (top to bottom):
 
 | Panel | What it shows |

{tokenol-0.3.1 → tokenol-0.3.2}/README.md

@@ -117,6 +117,8 @@ tokenol serve --open
 
 The dashboard updates via SSE as Claude Code writes events to disk. The server gates rebuilds on JSONL file changes — when no files have changed, it idles at near-zero CPU and forces a refresh at most once a minute (so time-windowed panels like Recent Activity don't drift more than ~60 s from wall clock). Multiple browser tabs share a single producer, so opening more tabs does not multiply server cost.
 
+If SSE delivery silently stalls (browser tab throttling, extension hooks, long-lived `EventSource` quirks), the client self-heals: it polls `/api/snapshot` every 30 s as a backstop, force-reconnects on tab-visibility return, and runs a 90 s staleness watchdog. `/api/snapshot` reuses the broadcaster's cached payload while an SSE group is live, so the backstop costs only a JSON serialize. Hover the live-status dot for a "last update Ns ago" indicator.
+
 Main page layout (top to bottom):
 
 | Panel | What it shows |

{tokenol-0.3.1 → tokenol-0.3.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "tokenol"
-version = "0.3.1"
+version = "0.3.2"
 description = "Audit Claude Code JSONL session logs: track cost, cache health, context blow-ups, and 5h-window pressure."
 readme = "README.md"
 license = { text = "MIT" }

{tokenol-0.3.1 → tokenol-0.3.2}/src/tokenol/serve/app.py

@@ -113,6 +113,16 @@ def _build_and_cache_snapshot(request: Request, period: str = "today") -> Snapsh
     return result
 
 
+def _current_snapshot_result(request: Request) -> SnapshotResult:
+    """Prefer the broadcaster's latest build; fall back to the app cache or a fresh build."""
+    broadcaster = getattr(request.app.state, "broadcaster", None)
+    if broadcaster is not None:
+        latest = broadcaster.latest_result()
+        if latest is not None:
+            return latest
+    return request.app.state.snapshot_result or _build_and_cache_snapshot(request)
+
+
 def create_app(
     config: ServerConfig | None = None,
     prefs_path: Path | None = None,
@@ -193,7 +203,7 @@ def create_app(
 
     @app.get("/api/session/{session_id}")
     async def api_session_detail(session_id: str, request: Request):
-        result = request.app.state.snapshot_result or _build_and_cache_snapshot(request)
+        result = _current_snapshot_result(request)
         session = next((s for s in result.sessions if s.session_id == session_id), None)
         if session is None:
             raise HTTPException(status_code=404, detail="Session not found")
@@ -201,7 +211,7 @@ def create_app(
 
     @app.get("/api/session/{session_id}/turn/{turn_idx}")
     async def api_turn_detail(session_id: str, turn_idx: int, request: Request):
-        result = request.app.state.snapshot_result or _build_and_cache_snapshot(request)
+        result = _current_snapshot_result(request)
         session = next((s for s in result.sessions if s.session_id == session_id), None)
         if session is None:
             raise HTTPException(status_code=404, detail="Session not found")
@@ -218,7 +228,7 @@ def create_app(
             raise HTTPException(status_code=400, detail="Invalid cwd encoding") from None
         if range not in ("1d", "7d", "14d", "30d", "all"):
             raise HTTPException(status_code=400, detail="Invalid range — use 1d, 7d, 14d, 30d, or all")
-        result = request.app.state.snapshot_result or _build_and_cache_snapshot(request)
+        result = _current_snapshot_result(request)
         detail = await asyncio.get_running_loop().run_in_executor(
             None,
             lambda: build_project_detail(cwd, result.sessions, range_key=range),
@@ -235,7 +245,7 @@ def create_app(
             raise HTTPException(status_code=400, detail="Invalid date format — use YYYY-MM-DD") from None
         if parsed_date > date.today():
             raise HTTPException(status_code=400, detail="Future dates not supported")
-        result = request.app.state.snapshot_result or _build_and_cache_snapshot(request)
+        result = _current_snapshot_result(request)
         detail = await asyncio.get_running_loop().run_in_executor(
             None,
             lambda: build_day_detail(parsed_date, result.turns, result.sessions),
@@ -272,7 +282,7 @@ def create_app(
             parsed = date.fromisoformat(target_date)
         except ValueError:
             raise HTTPException(status_code=400, detail="Invalid date — use YYYY-MM-DD") from None
-        result = request.app.state.snapshot_result or _build_and_cache_snapshot(request)
+        result = _current_snapshot_result(request)
         return JSONResponse(build_hourly_panel(parsed, result.turns, result.sessions, metric, project, model))
 
     @app.get("/api/daily")
@@ -289,7 +299,7 @@ def create_app(
             raise HTTPException(status_code=400, detail=f"Unknown metric — valid: {sorted(VALID_METRICS)}")
         if range not in ("7d", "30d", "90d", "all"):
             raise HTTPException(status_code=400, detail="range must be 7d, 30d, 90d, or all")
-        result = request.app.state.snapshot_result or _build_and_cache_snapshot(request)
+        result = _current_snapshot_result(request)
         # Fall back silently to the longest available window when the requested range
         # exceeds the data we have — return 200 with a `note` so the UI can caption it.
         # Returning 400 here forced clients to special-case "policy" failures even though
@@ -318,7 +328,7 @@ def create_app(
 
     @app.get("/api/models")
     async def api_models(request: Request, range: str = "today"):
-        result = request.app.state.snapshot_result or _build_and_cache_snapshot(request)
+        result = _current_snapshot_result(request)
         since = range_since(range, date.today())
         turns = [t for t in result.turns if t.timestamp.date() >= since] if since else result.turns
         return JSONResponse(build_models_panel(turns, range))
@@ -327,7 +337,7 @@ def create_app(
     async def api_recent(request: Request, window: str = "60m"):
         if window not in _WINDOW_MINUTES:
             raise HTTPException(status_code=400, detail=f"window must be one of: {list(_WINDOW_MINUTES)}")
-        result = request.app.state.snapshot_result or _build_and_cache_snapshot(request)
+        result = _current_snapshot_result(request)
         now = datetime.now(tz=timezone.utc)
         return JSONResponse(build_recent_activity_panel(
             result.turns, result.sessions, now, _WINDOW_MINUTES[window]
@@ -335,7 +345,7 @@ def create_app(
 
     @app.get("/api/model/{name}")
     async def api_model_detail(name: str, request: Request):
-        result = request.app.state.snapshot_result or _build_and_cache_snapshot(request)
+        result = _current_snapshot_result(request)
         detail = build_model_detail(name, result.turns, result.sessions)
         if detail is None:
             raise HTTPException(status_code=404, detail="Model not found")
@@ -343,7 +353,7 @@ def create_app(
 
     @app.get("/api/tool/{name}")
     async def api_tool_detail(name: str, request: Request):
-        result = request.app.state.snapshot_result or _build_and_cache_snapshot(request)
+        result = _current_snapshot_result(request)
         detail = build_tool_detail(name, result.turns, result.sessions)
         if detail is None:
             raise HTTPException(status_code=404, detail="Tool not found")
@@ -353,7 +363,7 @@ def create_app(
     async def api_search(request: Request, q: str = ""):
         if not q.strip():
             return JSONResponse({"hits": [], "query": q})
-        result = request.app.state.snapshot_result or _build_and_cache_snapshot(request)
+        result = _current_snapshot_result(request)
         return JSONResponse(build_search_results(q, result.turns, result.sessions))
 
     @app.get("/api/prefs")
@@ -402,7 +412,7 @@ def create_app(
     @app.get("/api/breakdown/summary")
     async def api_breakdown_summary(request: Request, range: str = "30d"):
         _validate_breakdown_range(range)
-        result = request.app.state.snapshot_result or _build_and_cache_snapshot(request)
+        result = _current_snapshot_result(request)
         since = range_since(range, date.today()) if range != "all" else None
         if since is None:
             turns = list(result.turns)
@@ -429,7 +439,7 @@ def create_app(
     @app.get("/api/breakdown/daily-tokens")
     async def api_breakdown_daily_tokens(request: Request, range: str = "30d"):
         _validate_breakdown_range(range)
-        result = request.app.state.snapshot_result or _build_and_cache_snapshot(request)
+        result = _current_snapshot_result(request)
         since = range_since(range, date.today()) if range != "all" else None
         if since is None:
             turns = list(result.turns)
@@ -456,7 +466,7 @@ def create_app(
     @app.get("/api/breakdown/by-project")
     async def api_breakdown_by_project(request: Request, range: str = "30d"):
         _validate_breakdown_range(range)
-        result = request.app.state.snapshot_result or _build_and_cache_snapshot(request)
+        result = _current_snapshot_result(request)
         since = range_since(range, date.today()) if range != "all" else None
 
         cwd_by_sid = _grouped_cwd_by_sid(result.sessions)
@@ -484,7 +494,7 @@ def create_app(
     @app.get("/api/breakdown/by-model")
     async def api_breakdown_by_model(request: Request, range: str = "30d"):
         _validate_breakdown_range(range)
-        result = request.app.state.snapshot_result or _build_and_cache_snapshot(request)
+        result = _current_snapshot_result(request)
         since = range_since(range, date.today()) if range != "all" else None
 
         buckets = _bucket_turns(
@@ -508,7 +518,7 @@ def create_app(
     @app.get("/api/breakdown/tools")
     async def api_breakdown_tools(request: Request, range: str = "30d"):
         _validate_breakdown_range(range)
-        result = request.app.state.snapshot_result or _build_and_cache_snapshot(request)
+        result = _current_snapshot_result(request)
         since = range_since(range, date.today()) if range != "all" else None
 
         total: Counter[str] = Counter()

{tokenol-0.3.1 → tokenol-0.3.2}/src/tokenol/serve/streaming.py

@@ -177,6 +177,8 @@ class SnapshotBroadcaster:
         self._heartbeat_s = heartbeat_s
         self._groups: dict[str, _Group] = {}
         self._lock = asyncio.Lock()
+        # turns/sessions are period-agnostic, so any producer's freshest result serves all readers.
+        self._latest_result: SnapshotResult | None = None
 
     def _compute_active_keys(self) -> frozenset[tuple[str, int, int]]:
         return compute_active_keys(self._all_projects)
@@ -190,8 +192,12 @@ class SnapshotBroadcaster:
             period=period,
             thresholds=self._get_thresholds(),
         )
+        self._latest_result = result
         return result.payload
 
+    def latest_result(self) -> SnapshotResult | None:
+        return self._latest_result
+
     async def subscribe(self, period: str) -> AsyncGenerator[str, None]:
         sub = _Subscriber()
         async with self._lock:

{tokenol-0.3.1 → tokenol-0.3.2}/tests/test_serve_app.py

@@ -1052,3 +1052,91 @@ async def test_api_tool_detail_404_on_unknown(tmp_path: Path) -> None:
         async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
             resp = await client.get("/api/tool/NoSuchTool")
     assert resp.status_code == 404
+
+
+@pytest.mark.asyncio
+async def test_api_hourly_reflects_broadcaster_freshness(tmp_path: Path) -> None:
+    """Regression: /api/hourly must track the broadcaster, not stale app.state.snapshot_result."""
+    import asyncio
+    from datetime import datetime, timezone
+
+    from httpx import ASGITransport, AsyncClient
+
+    today = datetime.now(tz=timezone.utc).date()
+    target_date = today.isoformat()
+
+    def _turn_line(uuid: str, hour: int, minute: int) -> bytes:
+        ts = f"{target_date}T{hour:02d}:{minute:02d}:00Z"
+        return (
+            b'{"type":"assistant","timestamp":"' + ts.encode() + b'",'
+            b'"sessionId":"sess-001","requestId":"req-' + uuid.encode() + b'",'
+            b'"uuid":"' + uuid.encode() + b'","isSidechain":false,'
+            b'"model":"claude-opus-4-7",'
+            b'"message":{"id":"msg-' + uuid.encode() + b'","role":"assistant",'
+            b'"stop_reason":"end_turn",'
+            b'"usage":{"input_tokens":10,"output_tokens":20,'
+            b'"cache_read_input_tokens":0,"cache_creation_input_tokens":0}}}\n'
+        )
+
+    sess_path = tmp_path / "projects" / "sess-001.jsonl"
+    sess_path.parent.mkdir(parents=True)
+    sess_path.write_bytes(_turn_line("evt-001", 10, 0) + _turn_line("evt-002", 10, 5))
+
+    with _mock_dirs(tmp_path):
+        app = create_app(ServerConfig())
+        bc = app.state.broadcaster
+        # Force the producer to rebuild on every mtime change with no heartbeat delay.
+        bc._get_tick_seconds = lambda: 0
+        bc._heartbeat_s = 0.0
+
+        agen = bc.subscribe("today").__aiter__()
+        try:
+            await asyncio.wait_for(agen.__anext__(), timeout=5.0)
+
+            async with AsyncClient(
+                transport=ASGITransport(app=app), base_url="http://test"
+            ) as client:
+                resp = await client.get(
+                    f"/api/hourly/{target_date}?metric=cost_per_kw"
+                )
+                turns_before = sum(
+                    p["turns"] for s in resp.json()["series"] for p in s["points"]
+                )
+                assert turns_before == 2, (
+                    f"fixture has 2 assistant turns, /api/hourly saw {turns_before}"
+                )
+
+            sess_path.write_bytes(sess_path.read_bytes() + _turn_line("evt-003", 10, 30))
+
+            # Poll /api/snapshot as the sync barrier — it exercises the same fast-path
+            # the fix targets, without reaching into broadcaster internals.
+            async with AsyncClient(
+                transport=ASGITransport(app=app), base_url="http://test"
+            ) as client:
+                snapshot_turns = 0
+                for _ in range(200):
+                    resp = await client.get("/api/snapshot?period=today")
+                    snapshot_turns = sum(
+                        s["turns"]
+                        for s in resp.json()["hourly_today"]["series"]
+                    )
+                    if snapshot_turns >= 3:
+                        break
+                    await asyncio.sleep(0.02)
+                assert snapshot_turns == 3, (
+                    "broadcaster did not pick up appended turn within 4 s "
+                    f"(snapshot still reports {snapshot_turns} turns)"
+                )
+
+                resp = await client.get(
+                    f"/api/hourly/{target_date}?metric=cost_per_kw"
+                )
+                turns_after = sum(
+                    p["turns"] for s in resp.json()["series"] for p in s["points"]
+                )
+                assert turns_after == 3, (
+                    "/api/hourly returned stale turns "
+                    f"({turns_after}); expected 3 after appending one turn"
+                )
+        finally:
+            await agen.aclose()