windhover 0.7.0__tar.gz → 0.8.0__tar.gz

windhover-0.8.0/.github/workflows/ci.yml

@@ -0,0 +1,19 @@
+name: ci
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: pip install -e .[dev]
+      - run: pytest tests/ -q

{windhover-0.7.0 → windhover-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: windhover
-Version: 0.7.0
+Version: 0.8.0
 Summary: Self-hosted, mobile-friendly observability for LangGraph — traces, run history, cost, and a living graph view.
 Project-URL: Repository, https://github.com/justfeltlikerunning/windhover
 Author: justfeltlikerunning
@@ -19,6 +19,11 @@ Description-Content-Type: text/markdown
 
 <p align="center"><img src="docs/logo.svg" width="112" alt="Windhover — a hovering kestrel"></p>
 <h1 align="center">Windhover</h1>
+<p align="center">
+  <a href="https://pypi.org/project/windhover/"><img src="https://img.shields.io/pypi/v/windhover" alt="PyPI"></a>
+  <a href="https://github.com/justfeltlikerunning/windhover/actions/workflows/ci.yml"><img src="https://github.com/justfeltlikerunning/windhover/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
+  <img src="https://img.shields.io/badge/license-MIT-8E2434" alt="MIT">
+</p>
 
 > *Windhover* — the old poetic name for the kestrel, the falcon that hangs motionless
 > in the wind, watching everything below. This tool does the same for your agent graphs.
@@ -43,8 +48,8 @@ your own app. No LangSmith account, no cloud tunnel, no fragile websocket. HTTP
 
 ## Quick start
 ```bash
-pip install fastapi uvicorn langgraph
-WINDHOVER_GRAPH=windhover.demo_graph:graph python -m windhover.server   # -> :8090
+pip install windhover langgraph
+WINDHOVER_GRAPH=windhover.demo_graph:graph windhover   # -> :8090
 ```
 Open `http://<host>:8090`. **New run** (input pre-filled from the graph's schema) →
 watch it execute → **Runs** for history, span trees, and replay → **Stats** for cost/latency.
@@ -85,11 +90,27 @@ graph.invoke(input, config={
   status/tag/session filters, bookmarks, pagination, CSV/JSON export.
 - **Sessions** — group runs into threads/batches; roll-up tokens, cost, errors.
 - **Scores** — attach numeric evals to runs (API or UI): eval harnesses, LLM-as-judge, human review.
+- **Live tail** — open a running run and watch its spans arrive in real time.
+- **Time-travel** — checkpointed graphs get a per-thread checkpoint browser: state, writes,
+  and next-nodes at every superstep (`get_state_history`).
+- **Run diff** — compare any two runs node-by-node: identical vs differing outputs,
+  duration and token deltas.
+- **Datasets / batch eval** — store golden input sets, run the graph over them, and get an
+  `expected_match` score per item (see Datasets on the Stats page).
 - **Run history + replay** — SQLite; runs persist even if the browser closes (worker thread).
 - **Living graph** — file watcher re-extracts topology in a subprocess and pushes it to the UI.
 - **Dashboards** — runs/tokens per day, per-model usage and latency, per-node latency, error rate.
 - **Mobile-first PWA**, light/dark. Fully local (FastAPI + Cytoscape.js).
 
+## Datasets API
+```bash
+curl -X POST :8090/api/datasets -H 'Content-Type: application/json' -d '{
+  "name": "golden", "items": [
+    {"input": {"n": 2},  "expected": 6},
+    {"input": {"n": 40}, "expected": "big"}]}'
+curl -X POST :8090/api/datasets/golden/run   # -> runs land in an eval:golden:<ts> session
+```
+
 ## Scores API
 ```bash
 curl -X POST :8090/api/runs/RUN_ID/scores -H 'Content-Type: application/json' \
@@ -99,7 +120,9 @@ curl -X POST :8090/api/runs/RUN_ID/scores -H 'Content-Type: application/json' \
 ## Config (env)
 `WINDHOVER_GRAPH` (module:attr; unset = ingest-only) · `WINDHOVER_GRAPH_DIR` · `WINDHOVER_DB`
 · `WINDHOVER_HOST`/`WINDHOVER_PORT` (0.0.0.0/8090) · `WINDHOVER_WATCH` (1) · `WINDHOVER_PRICING`
-· `WINDHOVER_RETENTION_DAYS` (0 = keep forever; else prune older runs on startup + every 6h).
+· `WINDHOVER_RETENTION_DAYS` (0 = keep forever; else prune older runs on startup + every 6h)
+· `WINDHOVER_TOKEN` (set to require `Authorization: Bearer <token>` — or `?token=` — on all
+`/api` routes; the UI prompts once and remembers it).
 Edit `windhover/pricing.json` for your models' $/1M rates (unknown model → cost null).
 
 ## Notes

{windhover-0.7.0 → windhover-0.8.0}/README.md

@@ -1,5 +1,10 @@
 <p align="center"><img src="docs/logo.svg" width="112" alt="Windhover — a hovering kestrel"></p>
 <h1 align="center">Windhover</h1>
+<p align="center">
+  <a href="https://pypi.org/project/windhover/"><img src="https://img.shields.io/pypi/v/windhover" alt="PyPI"></a>
+  <a href="https://github.com/justfeltlikerunning/windhover/actions/workflows/ci.yml"><img src="https://github.com/justfeltlikerunning/windhover/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
+  <img src="https://img.shields.io/badge/license-MIT-8E2434" alt="MIT">
+</p>
 
 > *Windhover* — the old poetic name for the kestrel, the falcon that hangs motionless
 > in the wind, watching everything below. This tool does the same for your agent graphs.
@@ -24,8 +29,8 @@ your own app. No LangSmith account, no cloud tunnel, no fragile websocket. HTTP
 
 ## Quick start
 ```bash
-pip install fastapi uvicorn langgraph
-WINDHOVER_GRAPH=windhover.demo_graph:graph python -m windhover.server   # -> :8090
+pip install windhover langgraph
+WINDHOVER_GRAPH=windhover.demo_graph:graph windhover   # -> :8090
 ```
 Open `http://<host>:8090`. **New run** (input pre-filled from the graph's schema) →
 watch it execute → **Runs** for history, span trees, and replay → **Stats** for cost/latency.
@@ -66,11 +71,27 @@ graph.invoke(input, config={
   status/tag/session filters, bookmarks, pagination, CSV/JSON export.
 - **Sessions** — group runs into threads/batches; roll-up tokens, cost, errors.
 - **Scores** — attach numeric evals to runs (API or UI): eval harnesses, LLM-as-judge, human review.
+- **Live tail** — open a running run and watch its spans arrive in real time.
+- **Time-travel** — checkpointed graphs get a per-thread checkpoint browser: state, writes,
+  and next-nodes at every superstep (`get_state_history`).
+- **Run diff** — compare any two runs node-by-node: identical vs differing outputs,
+  duration and token deltas.
+- **Datasets / batch eval** — store golden input sets, run the graph over them, and get an
+  `expected_match` score per item (see Datasets on the Stats page).
 - **Run history + replay** — SQLite; runs persist even if the browser closes (worker thread).
 - **Living graph** — file watcher re-extracts topology in a subprocess and pushes it to the UI.
 - **Dashboards** — runs/tokens per day, per-model usage and latency, per-node latency, error rate.
 - **Mobile-first PWA**, light/dark. Fully local (FastAPI + Cytoscape.js).
 
+## Datasets API
+```bash
+curl -X POST :8090/api/datasets -H 'Content-Type: application/json' -d '{
+  "name": "golden", "items": [
+    {"input": {"n": 2},  "expected": 6},
+    {"input": {"n": 40}, "expected": "big"}]}'
+curl -X POST :8090/api/datasets/golden/run   # -> runs land in an eval:golden:<ts> session
+```
+
 ## Scores API
 ```bash
 curl -X POST :8090/api/runs/RUN_ID/scores -H 'Content-Type: application/json' \
@@ -80,7 +101,9 @@ curl -X POST :8090/api/runs/RUN_ID/scores -H 'Content-Type: application/json' \
 ## Config (env)
 `WINDHOVER_GRAPH` (module:attr; unset = ingest-only) · `WINDHOVER_GRAPH_DIR` · `WINDHOVER_DB`
 · `WINDHOVER_HOST`/`WINDHOVER_PORT` (0.0.0.0/8090) · `WINDHOVER_WATCH` (1) · `WINDHOVER_PRICING`
-· `WINDHOVER_RETENTION_DAYS` (0 = keep forever; else prune older runs on startup + every 6h).
+· `WINDHOVER_RETENTION_DAYS` (0 = keep forever; else prune older runs on startup + every 6h)
+· `WINDHOVER_TOKEN` (set to require `Authorization: Bearer <token>` — or `?token=` — on all
+`/api` routes; the UI prompts once and remembers it).
 Edit `windhover/pricing.json` for your models' $/1M rates (unknown model → cost null).
 
 ## Notes

{windhover-0.7.0 → windhover-0.8.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "windhover"
-version = "0.7.0"
+version = "0.8.0"
 description = "Self-hosted, mobile-friendly observability for LangGraph — traces, run history, cost, and a living graph view."
 readme = "README.md"
 requires-python = ">=3.10"

{windhover-0.7.0 → windhover-0.8.0}/tests/test_smoke.py

@@ -244,6 +244,61 @@ def test_xray_topology():
     print("xray topology OK —", len(plain["nodes"]), "->", len(xray["nodes"]), "nodes")
 
 
+def test_auth_check():
+    os.environ.setdefault("WINDHOVER_DB", tempfile.mktemp(suffix=".db"))
+    from windhover.server import _auth_ok
+    assert _auth_ok("", "/api/runs", "", "")                     # no token configured -> open
+    assert _auth_ok("s3cret", "/", "", "")                       # UI shell never gated
+    assert _auth_ok("s3cret", "/static/x.js", "", "")
+    assert not _auth_ok("s3cret", "/api/runs", "", "")           # gated without creds
+    assert _auth_ok("s3cret", "/api/runs", "Bearer s3cret", "")
+    assert _auth_ok("s3cret", "/api/runs", "bearer  s3cret", "") # case/space tolerant
+    assert _auth_ok("s3cret", "/api/events", "", "s3cret")       # query token (SSE)
+    assert not _auth_ok("s3cret", "/api/runs", "Bearer wrong", "")
+    print("auth check OK")
+
+
+def test_thread_capture_and_history():
+    from typing import TypedDict
+    from langgraph.graph import StateGraph, START, END
+    from langgraph.checkpoint.memory import MemorySaver
+    p = tempfile.mktemp(suffix=".db")
+    s = Store(p)
+
+    class St(TypedDict):
+        x: int
+    g = StateGraph(St)
+    g.add_node("inc", lambda st: {"x": st["x"] + 1})
+    g.add_edge(START, "inc"); g.add_edge("inc", END)
+    app = g.compile(checkpointer=MemorySaver())
+    tr = SpanBuilder(db_sink(s), run_name="tt")
+    app.invoke({"x": 1}, config={"callbacks": [tr],
+                                 "configurable": {"thread_id": "thread-9"}})
+    time.sleep(.1)
+    d = s.run_detail(tr.run_id)
+    assert d["thread_id"] == "thread-9", d["thread_id"]
+    steps = list(app.get_state_history({"configurable": {"thread_id": "thread-9"}}))
+    assert len(steps) >= 2   # input checkpoint + node step
+    print("thread capture + history OK —", len(steps), "checkpoints")
+
+
+def test_datasets_and_scoring():
+    os.environ.setdefault("WINDHOVER_DB", tempfile.mktemp(suffix=".db"))
+    from windhover.server import _expected_score
+    p = tempfile.mktemp(suffix=".db")
+    s = Store(p)
+    ds = s.add_dataset("golden", [{"input": {"n": 2}, "expected": 6},
+                                  {"input": {"n": 5}}])
+    assert s.datasets()[0]["n_items"] == 2
+    assert s.dataset("golden")["id"] == ds["id"]
+    assert _expected_score(6, {"n": 6}) == 1.0
+    assert _expected_score(7, {"n": 6}) == 0.0
+    assert _expected_score("flattened", {"answer": "tiers were flattened in Q3"}) == 1.0
+    assert _expected_score("missing", {"answer": "nope"}) == 0.0
+    assert s.delete_dataset("golden") and not s.datasets()
+    print("datasets + scoring OK")
+
+
 if __name__ == "__main__":
     test_cost(); print("cost OK")
     test_store_roundtrip()
@@ -256,4 +311,7 @@ if __name__ == "__main__":
     test_retriever_spans()
     test_interrupt_status()
     test_xray_topology()
+    test_auth_check()
+    test_thread_capture_and_history()
+    test_datasets_and_scoring()
     print("ALL SMOKE TESTS PASSED")

{windhover-0.7.0 → windhover-0.8.0}/windhover/config.py

@@ -17,6 +17,7 @@ class Config:
     watch: bool               # live-topology file watcher
     pricing_path: str
     retention_days: int       # 0 = keep runs forever
+    token: str                # WINDHOVER_TOKEN: require Bearer/query token on /api ("" = open)
 
     @classmethod
     def from_env(cls) -> "Config":
@@ -29,4 +30,5 @@ class Config:
             watch=os.environ.get("WINDHOVER_WATCH", "1") not in ("0", "false", "no"),
             pricing_path=os.environ.get("WINDHOVER_PRICING", str(PKG_DIR / "pricing.json")),
             retention_days=int(os.environ.get("WINDHOVER_RETENTION_DAYS", "0") or 0),
+            token=os.environ.get("WINDHOVER_TOKEN", ""),
         )

{windhover-0.7.0 → windhover-0.8.0}/windhover/demo_graph.py

@@ -45,4 +45,8 @@ _g.add_edge("sign", "summarize")
 _g.add_edge("magnitude", "summarize")
 _g.add_edge("summarize", END)
 
-graph = _g.compile()
+try:  # checkpointer makes the Time-travel view demoable; optional dependency surface
+    from langgraph.checkpoint.memory import MemorySaver
+    graph = _g.compile(checkpointer=MemorySaver())
+except Exception:
+    graph = _g.compile()

{windhover-0.7.0 → windhover-0.8.0}/windhover/server.py

@@ -15,7 +15,7 @@ from fastapi.staticfiles import StaticFiles
 
 from .config import Config
 from .store import Store
-from .tracer import SpanBuilder, db_sink, apply_to_store
+from .tracer import SpanBuilder, db_sink, apply_to_store, _trunc
 
 cfg = Config.from_env()
 store = Store(cfg.db_path)
@@ -30,6 +30,27 @@ if cfg.graph_ref:
 app = FastAPI(title="Windhover")
 
 
+def _auth_ok(token: str, path: str, auth_header: str, query_token: str) -> bool:
+    """True when the request may proceed. Only /api is gated; static/UI stay
+    open (they contain no data — every payload comes through /api)."""
+    if not token or not path.startswith("/api"):
+        return True
+    supplied = (auth_header or "").strip()
+    if supplied.lower().startswith("bearer "):
+        supplied = supplied[7:].strip()
+    return supplied == token or (query_token or "") == token
+
+
+@app.middleware("http")
+async def _auth_middleware(request: Request, call_next):
+    if not _auth_ok(cfg.token, request.url.path,
+                    request.headers.get("authorization", ""),
+                    request.query_params.get("token", "")):
+        return JSONResponse({"error": "unauthorized — supply WINDHOVER_TOKEN as "
+                            "Bearer header or ?token="}, 401)
+    return await call_next(request)
+
+
 # ---- topology manager (subprocess extraction, mtime-cached) ---------------
 class Topo:
     def __init__(self):
@@ -152,14 +173,20 @@ async def api_run(request: Request):
         payload = {}
     session = payload.pop("_session", None)
     tags = payload.pop("_tags", None)
+    thread = payload.pop("_thread", None)
     tracer = SpanBuilder(db_sink(store), run_name=cfg.graph_ref, session=session, tags=tags)
     run_id = tracer.run_id
+    config = {"callbacks": [tracer]}
+    if thread or getattr(graph, "checkpointer", None) is not None:
+        # a checkpointed graph needs a thread; default to the run id so
+        # time-travel works out of the box
+        config["configurable"] = {"thread_id": thread or run_id}
     q: "queue.Queue" = queue.Queue()
 
     def worker():
         try:
             interrupted = False
-            for update in graph.stream(payload, config={"callbacks": [tracer]},
+            for update in graph.stream(payload, config=config,
                                        stream_mode="updates"):
                 for node in update:
                     if node == "__interrupt__":
@@ -287,6 +314,99 @@ def api_node_source(name: str):
                         "(external-only run, or a runnable inspect can't trace)"}, 404)
 
 
+@app.get("/api/threads/{thread_id}/history")
+def api_thread_history(thread_id: str, limit: int = 80):
+    """Time-travel: LangGraph checkpoint history for a thread (local graph
+    with a checkpointer only)."""
+    if graph is None or getattr(graph, "checkpointer", None) is None:
+        return JSONResponse({"error": "no local graph with a checkpointer"}, 404)
+    steps = []
+    try:
+        for st in graph.get_state_history({"configurable": {"thread_id": thread_id}}):
+            md = st.metadata or {}
+            steps.append({
+                "checkpoint_id": ((st.config or {}).get("configurable") or {}).get("checkpoint_id"),
+                "step": md.get("step"),
+                "source": md.get("source"),
+                "writes": _trunc(md.get("writes"), 2000) if md.get("writes") is not None else None,
+                "next": list(st.next or []),
+                "values": _trunc(st.values, 3000),
+                "created_at": str(getattr(st, "created_at", "") or "") or None,
+            })
+            if len(steps) >= limit:
+                break
+    except Exception as e:
+        return JSONResponse({"error": str(e)}, 500)
+    return JSONResponse({"thread_id": thread_id, "steps": steps})
+
+
+@app.get("/api/datasets")
+def api_datasets():
+    return JSONResponse([{k: d[k] for k in ("id", "name", "n_items", "created_ms")}
+                         for d in store.datasets()])
+
+
+@app.post("/api/datasets")
+async def api_dataset_add(request: Request):
+    body = await request.json()
+    name, items = body.get("name"), body.get("items")
+    if not name or not isinstance(items, list) or not items:
+        return JSONResponse({"error": "requires name and a non-empty items list "
+                            "[{input: {...}, expected?: ...}]"}, 400)
+    for it in items:
+        if not isinstance(it, dict) or "input" not in it:
+            return JSONResponse({"error": "every item needs an input object"}, 400)
+    ds = store.add_dataset(str(name), items)
+    return JSONResponse({"id": ds["id"], "name": ds["name"], "n_items": len(items)})
+
+
+@app.delete("/api/datasets/{ds_id}")
+def api_dataset_delete(ds_id: str):
+    return JSONResponse({"ok": store.delete_dataset(ds_id)})
+
+
+def _expected_score(expected, output) -> float:
+    """Naive match: exact for scalars, substring against the JSON otherwise."""
+    try:
+        if isinstance(expected, (int, float, bool)):
+            return 1.0 if any(v == expected for v in
+                              (output.values() if isinstance(output, dict) else [output])) else 0.0
+        return 1.0 if str(expected) in json.dumps(output, default=str) else 0.0
+    except Exception:
+        return 0.0
+
+
+@app.post("/api/datasets/{ds_id}/run")
+def api_dataset_run(ds_id: str):
+    """Batch-eval: run the local graph over every dataset item; expected values
+    become an expected_match score on each run. Fire-and-forget worker."""
+    if graph is None:
+        return JSONResponse({"error": "no local graph (WINDHOVER_GRAPH unset)"}, 400)
+    ds = store.dataset(ds_id)
+    if not ds:
+        return JSONResponse({"error": "dataset not found"}, 404)
+    session = f"eval:{ds['name']}:{int(time.time())}"
+
+    def worker():
+        for i, item in enumerate(ds["items"]):
+            tracer = SpanBuilder(db_sink(store), run_name=cfg.graph_ref,
+                                 session=session, tags=[f"dataset:{ds['name']}"])
+            config = {"callbacks": [tracer]}
+            if getattr(graph, "checkpointer", None) is not None:
+                config["configurable"] = {"thread_id": tracer.run_id}
+            try:
+                out = graph.invoke(dict(item["input"]), config=config)
+            except Exception:
+                continue  # tracer already recorded the error run
+            if "expected" in item:
+                store.add_score(tracer.run_id, "expected_match",
+                                _expected_score(item["expected"], out),
+                                comment=f"item {i}", source="dataset")
+
+    threading.Thread(target=worker, daemon=True).start()
+    return JSONResponse({"ok": True, "session": session, "items": len(ds["items"])})
+
+
 @app.get("/api/stats")
 def api_stats(days: int = 30):
     return JSONResponse(store.stats(days=max(1, min(days, 365))))

{windhover-0.7.0 → windhover-0.8.0}/windhover/static/index.html

@@ -130,6 +130,73 @@ td.r,th.r{text-align:right}
 .empty{padding:56px 20px;text-align:center;color:var(--muted)}
 .empty .t{font-weight:600;color:var(--text2);margin-bottom:4px}
 
+/* ---------- time-travel (checkpoint history) ---------- */
+window.showHistory=async(tid)=>{
+  let d=null; try{ const r=await fetch('/api/threads/'+encodeURIComponent(tid)+'/history');
+    d=await r.json(); if(!r.ok) return toast(d.error||'history unavailable'); }catch(e){ return toast('history unavailable'); }
+  const steps=d.steps||[];
+  $('#d-title').textContent='thread '+tid.slice(0,10)+' · time-travel';
+  const st=$('#d-status'); st.className='status done'; $('#d-status-t').textContent=steps.length+' checkpoints';
+  $('#d-meta').innerHTML='<span>newest first — every LangGraph checkpoint for this thread</span>';
+  $('#d-body').innerHTML=steps.map((s,i)=>{
+    const did='tt'+i;
+    const writes=s.writes&&typeof s.writes==='object'?Object.keys(s.writes):[];
+    return `<div>
+      <div class="tr-row flat" onclick="tog('${did}',this)">
+        <div class="tr-main"><span class="caret">▶</span>
+          <span class="chip node">step ${s.step??'—'}</span>
+          <span class="tr-name">${esc(s.source||'')}</span>
+          ${writes.map(w=>`<span class="tagchip" style="cursor:default">${esc(w)}</span>`).join('')}
+          ${(s.next||[]).length?`<span class="mono" style="color:var(--muted);font-size:11px">→ ${esc((s.next||[]).join(', '))}</span>`:''}</div>
+        <div class="tr-meta">${s.created_at?esc(String(s.created_at).slice(11,19)):''}</div>
+      </div>
+      <div class="tr-detail" id="${did}">
+        ${s.writes!=null?kv('writes',s.writes):''}
+        ${kv('state at this checkpoint',s.values)}
+      </div></div>`;
+  }).join('')||'<div class="empty">No checkpoints recorded for this thread</div>';
+  $('#scrim').classList.add('on'); $('#drawer').classList.add('on');
+};
+
+/* ---------- run diff ---------- */
+window.pickDiff=async(id)=>{
+  if(!window.__diffA||window.__diffA===id){ window.__diffA=id; toast('Compare: pick a second run (open it, hit compare)'); return; }
+  const a=window.__diffA; window.__diffA=null;
+  const [ra,rb]=await Promise.all([fetch('/api/runs/'+a).then(r=>r.json()),
+                                   fetch('/api/runs/'+id).then(r=>r.json())]);
+  const tops=r=>r.spans.filter(s=>!s.parent_id&&s.type==='node');
+  const A=tops(ra),B=tops(rb);
+  const names=[...new Set([...A.map(s=>s.name),...B.map(s=>s.name)])];
+  $('#d-title').textContent='diff · '+a.slice(0,8)+' ↔ '+id.slice(0,8);
+  const st=$('#d-status'); st.className='status done'; $('#d-status-t').textContent='compare';
+  const delta=(x,y,f)=>{const va=x?.[f],vb=y?.[f]; if(va==null&&vb==null)return '';
+    return `<span class="mono" style="font-size:11px;color:var(--muted)">${va??'—'} → ${vb??'—'}</span>`;};
+  $('#d-meta').innerHTML=[
+    `<span>A <b class="mono">${esc(ra.graph||'')} ${a.slice(0,8)}</b></span>`,
+    `<span>B <b class="mono">${esc(rb.graph||'')} ${id.slice(0,8)}</b></span>`,
+    `<span>duration ${delta(ra,rb,'duration_ms')}</span>`,
+    `<span>tokens ${delta(ra,rb,'total_tokens')}</span>`].join('');
+  $('#d-body').innerHTML=names.map((n,i)=>{
+    const sa=A.find(s=>s.name===n),sb=B.find(s=>s.name===n);
+    const ja=JSON.stringify(sa?.output??null),jb=JSON.stringify(sb?.output??null);
+    const same=ja===jb;
+    const did='df'+i;
+    return `<div>
+      <div class="tr-row flat" onclick="tog('${did}',this)">
+        <div class="tr-main"><span class="caret">▶</span><span class="chip node">${esc(n)}</span>
+          <span class="tr-name" style="color:${same?'var(--muted)':'var(--err)'}">${same?'≡ identical output':'≠ outputs differ'}</span></div>
+        <div class="tr-meta">${sa?.dur_ms??'—'} / ${sb?.dur_ms??'—'} ms</div>
+      </div>
+      <div class="tr-detail" id="${did}">
+        <div class="diffgrid">
+          <div class="kv"><div class="k">A · ${esc(n)}</div><pre>${esc(JSON.stringify(sa?.output??'(absent)',null,2))}</pre></div>
+          <div class="kv"><div class="k">B · ${esc(n)}</div><pre>${esc(JSON.stringify(sb?.output??'(absent)',null,2))}</pre></div>
+        </div>
+      </div></div>`;
+  }).join('');
+  $('#scrim').classList.add('on'); $('#drawer').classList.add('on');
+};
+
 /* ---------- stats ---------- */
 .metrics{display:grid;grid-template-columns:repeat(auto-fit,minmax(160px,1fr));gap:12px;margin-bottom:16px}
 .metric{background:var(--surface);border:1px solid var(--border);border-radius:var(--r-lg);box-shadow:var(--shadow);padding:14px 16px}
@@ -185,6 +252,8 @@ td.r,th.r{text-align:right}
 .code .errline{background:var(--err-weak)}
 .code .cl.errline{box-shadow:inset 3px 0 0 var(--err);padding-left:6px;margin-left:-6px}
 .errhint{font:11px var(--mono);color:var(--err);margin-top:3px;max-width:400px;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}
+.diffgrid{display:grid;grid-template-columns:1fr 1fr;gap:10px}
+@media(max-width:840px){.diffgrid{grid-template-columns:1fr}}
 .kv{margin:8px 0}
 .kv .k{display:flex;align-items:center;gap:8px;font-size:11px;font-weight:700;letter-spacing:.05em;text-transform:uppercase;color:var(--muted);margin-bottom:4px}
 .copy{font-size:10.5px;color:var(--accent-text);background:var(--accent-weak);border-radius:5px;padding:1px 7px}
@@ -317,7 +386,7 @@ th{font-family:var(--mono);letter-spacing:.11em;font-size:10.5px}
       <button data-v="sessions">${ICON.layers} Sessions</button>
       <button data-v="stats">${ICON.chart} Stats</button>
     </nav>
-    <div class="side-foot"><span class="live-dot" id="live-dot" title="live topology"></span><span id="foot-note">watching</span><span class="spacer"></span><span class="mono" id="ver">v0.7</span></div>
+    <div class="side-foot"><span class="live-dot" id="live-dot" title="live topology"></span><span id="foot-note">watching</span><span class="spacer"></span><span class="mono" id="ver">v0.8</span></div>
   </aside>
 
   <main>
@@ -429,6 +498,16 @@ const ICON = {
 document.body.innerHTML = document.body.innerHTML.replaceAll(/\$\{ICON\.(\w+)\}/g,(m,k)=>ICON[k]||'');
 
 const $=s=>document.querySelector(s), $$=s=>[...document.querySelectorAll(s)];
+/* optional WINDHOVER_TOKEN auth: attach Bearer to /api calls; prompt once on 401 */
+const _fetch=window.fetch.bind(window);
+window.whToken=()=>localStorage.getItem('wh-token')||'';
+window.fetch=(u,o={})=>{ const t=whToken();
+  if(t&&String(u).startsWith('/api')) o={...o,headers:{...(o.headers||{}),'Authorization':'Bearer '+t}};
+  return _fetch(u,o).then(r=>{ if(r.status===401&&String(u).startsWith('/api')&&!window.__askedTok){
+      window.__askedTok=true;
+      const v=window.prompt('This Windhover requires an access token (WINDHOVER_TOKEN):');
+      if(v){ localStorage.setItem('wh-token',v.trim()); location.reload(); } }
+    return r; }); };
 let cy=null, VIEW='graph';
 
 /* ---------- theme ---------- */
@@ -563,7 +642,7 @@ async function openNode(name){
   $('#d-body').innerHTML=html;
   $('#scrim').classList.add('on'); $('#drawer').classList.add('on');
 }
-function liveTopology(){ try{ const es=new EventSource('/api/events');
+function liveTopology(){ try{ const es=new EventSource('/api/events'+(whToken()?'?token='+encodeURIComponent(whToken()):''));
   es.addEventListener('topology',async()=>{ await loadGraph(true); toast('Graph definition changed — view updated'); });
   es.onerror=()=>{ es.close(); $('#foot-note').textContent='reconnecting…'; setTimeout(liveTopology,5000); };
   es.onopen=()=>{ $('#foot-note').textContent='watching'; };
@@ -576,7 +655,7 @@ function switchView(v){ VIEW=v;
   $('#crumb').textContent=v[0].toUpperCase()+v.slice(1);
   for(const k of ['graph','runs','sessions','stats'])
     $('#view-'+k).style.display=v===k?'':'none';
-  if(v==='runs')loadRuns(); if(v==='sessions')loadSessions(); if(v==='stats')loadStats();
+  if(v==='runs')loadRuns(); if(v==='sessions')loadSessions(); if(v==='stats'){loadStats();loadDatasets();}
   if(cy&&v==='graph')setTimeout(()=>cy.resize(),40);
 }
 $$('.nav button,.bottom-nav button').forEach(b=>b.onclick=()=>switchView(b.dataset.v));
@@ -654,7 +733,7 @@ window.toggleBook=async(id,b)=>{
   await fetch('/api/runs/'+id,{method:'PATCH',headers:{'Content-Type':'application/json'},
     body:JSON.stringify({bookmarked:!!b})});
   loadRuns();};
-window.exportRuns=fmt=>{const p=runParams();p.set('format',fmt);location='/api/export?'+p;};
+window.exportRuns=fmt=>{const p=runParams();p.set('format',fmt);if(whToken())p.set('token',whToken());location='/api/export?'+p;};
 let rqT; $('#rq').oninput=e=>{clearTimeout(rqT);rqT=setTimeout(()=>{RF.q=e.target.value.trim();RF.offset=0;loadRuns();},300);};
 $('#rstatus').onchange=e=>{RF.status=e.target.value;RF.offset=0;loadRuns();};
 $('#rbook').onclick=()=>{RF.bookmarked=!RF.bookmarked;$('#rbook').classList.toggle('on',RF.bookmarked);RF.offset=0;loadRuns();};
@@ -689,6 +768,8 @@ async function openRun(id){
     r.total_tokens?`<span><b class="num">${fmtk(r.total_tokens)}</b> tokens</span>`:'',
     r.cost_usd!=null?`<span><b class="num">$${r.cost_usd.toFixed(5)}</b> cost</span>`:'',
     r.session?`<span>session <b class="mono">${esc(r.session)}</b></span>`:'',
+    r.thread_id?`<span>thread <b class="mono">${esc(r.thread_id)}</b> <button class="copy" onclick="showHistory('${esc(r.thread_id)}')">time-travel</button></span>`:'',
+    `<span><button class="copy" onclick="pickDiff('${id}')">compare</button></span>`,
     (r.tags||[]).length?`<span>${r.tags.map(t=>`<span class="tagchip">${esc(t)}</span>`).join('')}</span>`:'',
     `<span>${new Date(r.started_ms).toLocaleString()}</span>`,
   ].filter(Boolean).join('');
@@ -714,8 +795,15 @@ async function openRun(id){
       <button class="btn ghost" onclick="addScore('${id}')">Add score</button>
     </div>`;
   html += `<div style="margin-top:6px"><button class="btn ghost" onclick='replay(${JSON.stringify(r.spans.filter(s=>s.type==="node").map(s=>({n:s.name,e:s.status==="error"})))})'>⟲ Replay on graph</button></div>`;
+  const openIds=[...document.querySelectorAll('#d-body .tr-detail')]
+    .filter(e=>e.style.display==='block').map(e=>e.id);
   $('#d-body').innerHTML=html;
+  openIds.forEach(i=>{const e=document.getElementById(i); if(e){e.style.display='block';
+    const row=e.previousElementSibling; row&&row.querySelector('.caret')?.classList.add('open');}});
   $('#scrim').classList.add('on'); $('#drawer').classList.add('on');
+  clearTimeout(window.__liveT);
+  if(r.status==='running')
+    window.__liveT=setTimeout(()=>{ if($('#drawer').classList.contains('on')&&window.__run&&window.__run.id===id) openRun(id); },1500);
 }
 window.addScore=async(id)=>{
   const name=$('#sc-name').value.trim(), value=parseFloat($('#sc-val').value);
@@ -779,6 +867,73 @@ window.showSpanSource=async(sid)=>{
                   :'<pre>no source available for this node (external run, or inspect could not trace it)</pre>';
 };
 
+/* ---------- time-travel (checkpoint history) ---------- */
+window.showHistory=async(tid)=>{
+  let d=null; try{ const r=await fetch('/api/threads/'+encodeURIComponent(tid)+'/history');
+    d=await r.json(); if(!r.ok) return toast(d.error||'history unavailable'); }catch(e){ return toast('history unavailable'); }
+  const steps=d.steps||[];
+  $('#d-title').textContent='thread '+tid.slice(0,10)+' · time-travel';
+  const st=$('#d-status'); st.className='status done'; $('#d-status-t').textContent=steps.length+' checkpoints';
+  $('#d-meta').innerHTML='<span>newest first — every LangGraph checkpoint for this thread</span>';
+  $('#d-body').innerHTML=steps.map((s,i)=>{
+    const did='tt'+i;
+    const writes=s.writes&&typeof s.writes==='object'?Object.keys(s.writes):[];
+    return `<div>
+      <div class="tr-row flat" onclick="tog('${did}',this)">
+        <div class="tr-main"><span class="caret">▶</span>
+          <span class="chip node">step ${s.step??'—'}</span>
+          <span class="tr-name">${esc(s.source||'')}</span>
+          ${writes.map(w=>`<span class="tagchip" style="cursor:default">${esc(w)}</span>`).join('')}
+          ${(s.next||[]).length?`<span class="mono" style="color:var(--muted);font-size:11px">→ ${esc((s.next||[]).join(', '))}</span>`:''}</div>
+        <div class="tr-meta">${s.created_at?esc(String(s.created_at).slice(11,19)):''}</div>
+      </div>
+      <div class="tr-detail" id="${did}">
+        ${s.writes!=null?kv('writes',s.writes):''}
+        ${kv('state at this checkpoint',s.values)}
+      </div></div>`;
+  }).join('')||'<div class="empty">No checkpoints recorded for this thread</div>';
+  $('#scrim').classList.add('on'); $('#drawer').classList.add('on');
+};
+
+/* ---------- run diff ---------- */
+window.pickDiff=async(id)=>{
+  if(!window.__diffA||window.__diffA===id){ window.__diffA=id; toast('Compare: pick a second run (open it, hit compare)'); return; }
+  const a=window.__diffA; window.__diffA=null;
+  const [ra,rb]=await Promise.all([fetch('/api/runs/'+a).then(r=>r.json()),
+                                   fetch('/api/runs/'+id).then(r=>r.json())]);
+  const tops=r=>r.spans.filter(s=>!s.parent_id&&s.type==='node');
+  const A=tops(ra),B=tops(rb);
+  const names=[...new Set([...A.map(s=>s.name),...B.map(s=>s.name)])];
+  $('#d-title').textContent='diff · '+a.slice(0,8)+' ↔ '+id.slice(0,8);
+  const st=$('#d-status'); st.className='status done'; $('#d-status-t').textContent='compare';
+  const delta=(x,y,f)=>{const va=x?.[f],vb=y?.[f]; if(va==null&&vb==null)return '';
+    return `<span class="mono" style="font-size:11px;color:var(--muted)">${va??'—'} → ${vb??'—'}</span>`;};
+  $('#d-meta').innerHTML=[
+    `<span>A <b class="mono">${esc(ra.graph||'')} ${a.slice(0,8)}</b></span>`,
+    `<span>B <b class="mono">${esc(rb.graph||'')} ${id.slice(0,8)}</b></span>`,
+    `<span>duration ${delta(ra,rb,'duration_ms')}</span>`,
+    `<span>tokens ${delta(ra,rb,'total_tokens')}</span>`].join('');
+  $('#d-body').innerHTML=names.map((n,i)=>{
+    const sa=A.find(s=>s.name===n),sb=B.find(s=>s.name===n);
+    const ja=JSON.stringify(sa?.output??null),jb=JSON.stringify(sb?.output??null);
+    const same=ja===jb;
+    const did='df'+i;
+    return `<div>
+      <div class="tr-row flat" onclick="tog('${did}',this)">
+        <div class="tr-main"><span class="caret">▶</span><span class="chip node">${esc(n)}</span>
+          <span class="tr-name" style="color:${same?'var(--muted)':'var(--err)'}">${same?'≡ identical output':'≠ outputs differ'}</span></div>
+        <div class="tr-meta">${sa?.dur_ms??'—'} / ${sb?.dur_ms??'—'} ms</div>
+      </div>
+      <div class="tr-detail" id="${did}">
+        <div class="diffgrid">
+          <div class="kv"><div class="k">A · ${esc(n)}</div><pre>${esc(JSON.stringify(sa?.output??'(absent)',null,2))}</pre></div>
+          <div class="kv"><div class="k">B · ${esc(n)}</div><pre>${esc(JSON.stringify(sb?.output??'(absent)',null,2))}</pre></div>
+        </div>
+      </div></div>`;
+  }).join('');
+  $('#scrim').classList.add('on'); $('#drawer').classList.add('on');
+};
+
 /* ---------- stats ---------- */
 function chartCard(title,daily,val,errKey){
   const max=Math.max(1,...daily.map(d=>val(d)||0));
@@ -818,6 +973,8 @@ async function loadStats(){
         <td class="r num">${m.completion_tokens!=null?fmtk(m.completion_tokens):'—'}</td>
         <td class="r num">${m.cost!=null?'$'+m.cost.toFixed(4):'—'}</td>
         <td class="r num mono">${fmtms(Math.round(m.avg_ms||0))}</td></tr>`).join('')}</tbody></table></div></div>`:''}
+    <div class="page-head"><h1 style="font-size:14px">Datasets — batch eval</h1></div>
+    <div class="card" style="margin-bottom:14px"><div id="ds-wrap" style="padding:6px 8px"><div class="empty" style="padding:14px">Loading…</div></div></div>
     <div class="page-head"><h1 style="font-size:14px">Average latency per node</h1></div>
     <div class="card">${(s.per_node||[]).map(n=>`<div class="latrow"><div class="latname">${esc(n.name)}</div>
       <div class="lattrack"><div class="latfill" style="width:${(n.avg_ms||0)/max*100}%"></div></div>
@@ -832,6 +989,24 @@ function fmtk(n){ return n<1000?String(n):(n/1000).toFixed(1)+'k'; }
 function ago(ms){ const s=(Date.now()-ms)/1000; if(s<60)return Math.round(s)+'s ago'; if(s<3600)return Math.round(s/60)+'m ago'; if(s<86400)return Math.round(s/3600)+'h ago'; return Math.round(s/86400)+'d ago'; }
 let tT; function toast(m){ const t=$('#toast'); t.textContent=m; t.classList.add('show'); clearTimeout(tT); tT=setTimeout(()=>t.classList.remove('show'),2600); }
 
+async function loadDatasets(){
+  const w=document.getElementById('ds-wrap'); if(!w) return;
+  let ds=[]; try{ ds=await(await fetch('/api/datasets')).json(); }catch(e){}
+  w.innerHTML=ds.length?ds.map(d=>`<div class="scorerow">
+      <span class="chip node">${esc(d.name)}</span>
+      <span style="color:var(--muted);font-size:12px">${d.n_items} items</span>
+      <span class="del" title="delete" onclick="delDataset('${d.id}')">×</span>
+      <button class="btn ghost" style="margin-left:auto;padding:4px 12px" onclick="runDataset('${d.id}','${esc(d.name)}')">Run eval</button>
+    </div>`).join('')
+    :`<div class="empty" style="padding:14px"><div class="t">No datasets yet</div>POST <span class="mono">/api/datasets</span> with <span class="mono">{name, items:[{input:{…}, expected:…}]}</span> — runs land in a session with an <span class="mono">expected_match</span> score per item.</div>`;
+}
+window.runDataset=async(id,name)=>{
+  const r=await fetch('/api/datasets/'+id+'/run',{method:'POST'});
+  const d=await r.json();
+  if(!r.ok) return toast(d.error||'eval failed to start');
+  toast('Eval started: '+d.items+' items → session '+d.session);
+};
+window.delDataset=async(id)=>{ await fetch('/api/datasets/'+id,{method:'DELETE'}); loadDatasets(); };
 loadGraph(); liveTopology();
 /* deep links: #runs #sessions #stats #run=<id> */
 (function(){ const h=location.hash.slice(1);

{windhover-0.7.0 → windhover-0.8.0}/windhover/store.py

@@ -13,7 +13,7 @@ from __future__ import annotations
 import json, os, sqlite3, threading, time, uuid
 from typing import Any, Optional
 
-SCHEMA_VERSION = 4
+SCHEMA_VERSION = 5
 _lock = threading.Lock()
 
 
@@ -56,6 +56,8 @@ class Store:
             CREATE TABLE IF NOT EXISTS scores(
               id TEXT PRIMARY KEY, run_id TEXT, name TEXT, value REAL,
               comment TEXT, source TEXT, created_ms INTEGER);
+            CREATE TABLE IF NOT EXISTS datasets(
+              id TEXT PRIMARY KEY, name TEXT UNIQUE, items TEXT, created_ms INTEGER);
             CREATE INDEX IF NOT EXISTS ix_spans_run ON spans(run_id, seq);
             CREATE INDEX IF NOT EXISTS ix_runs_started ON runs(started_ms DESC);
             CREATE INDEX IF NOT EXISTS ix_runs_session ON runs(session);
@@ -64,6 +66,8 @@ class Store:
             cols = [r[1] for r in c.execute("PRAGMA table_info(runs)").fetchall()]
             if "bookmarked" not in cols:
                 c.execute("ALTER TABLE runs ADD COLUMN bookmarked INTEGER DEFAULT 0")
+            if "thread_id" not in cols:
+                c.execute("ALTER TABLE runs ADD COLUMN thread_id TEXT")
             try:
                 c.execute("SELECT count(*) FROM json_each('[1]')")
                 self.has_json1 = True
@@ -118,11 +122,12 @@ class Store:
     def open_run(self, run: dict) -> None:
         with _lock, self._conn() as c:
             c.execute("""INSERT OR REPLACE INTO runs
-              (id,graph,source,status,session,tags,input,started_ms)
-              VALUES(?,?,?,?,?,?,?,?)""",
+              (id,graph,source,status,session,tags,input,started_ms,thread_id)
+              VALUES(?,?,?,?,?,?,?,?,?)""",
               (run["id"], run.get("graph"), run.get("source", "ui"), "running",
                run.get("session"), json.dumps(run.get("tags")),
-               json.dumps(run.get("input"), default=str), run["started_ms"]))
+               json.dumps(run.get("input"), default=str), run["started_ms"],
+               run.get("thread_id")))
 
     def add_span(self, s: dict) -> None:
         with _lock, self._conn() as c:
@@ -190,6 +195,36 @@ class Store:
         with _lock, self._conn() as c:
             return c.execute("DELETE FROM scores WHERE id=?", (score_id,)).rowcount > 0
 
+    # ---- datasets -----------------------------------------------------------
+    def add_dataset(self, name: str, items: list) -> dict:
+        ds = {"id": uuid.uuid4().hex[:12], "name": str(name),
+              "items": items, "created_ms": int(time.time() * 1000)}
+        with _lock, self._conn() as c:
+            c.execute("""INSERT OR REPLACE INTO datasets(id,name,items,created_ms)
+                         VALUES(?,?,?,?)""",
+                      (ds["id"], ds["name"], json.dumps(items, default=str), ds["created_ms"]))
+        return ds
+
+    def datasets(self) -> list[dict]:
+        with _lock, self._conn() as c:
+            rows = [dict(r) for r in c.execute(
+                "SELECT id,name,items,created_ms FROM datasets ORDER BY created_ms DESC").fetchall()]
+        for r in rows:
+            r["items"] = json.loads(r["items"]) if r.get("items") else []
+            r["n_items"] = len(r["items"])
+        return rows
+
+    def dataset(self, ds_id: str):
+        for d in self.datasets():
+            if d["id"] == ds_id or d["name"] == ds_id:
+                return d
+        return None
+
+    def delete_dataset(self, ds_id: str) -> bool:
+        with _lock, self._conn() as c:
+            return c.execute("DELETE FROM datasets WHERE id=? OR name=?",
+                             (ds_id, ds_id)).rowcount > 0
+
     def prune(self, days: int) -> dict:
         """Delete runs (and their spans/scores/index rows) older than `days`."""
         cutoff = int((time.time() - days * 86400) * 1000)

{windhover-0.7.0 → windhover-0.8.0}/windhover/tracer.py

@@ -179,7 +179,8 @@ class SpanBuilder(BaseCallbackHandler):
             self.tags = merged or None
             self._emit({"kind": "run_open", "run_id": self.run_id, "graph": self.run_name,
                         "input": _trunc(inputs), "started_ms": int(self._t0 * 1000),
-                        "session": self.session, "tags": self.tags})
+                        "session": self.session, "tags": self.tags,
+                        "thread_id": md.get("thread_id")})
         if node:
             sid = uuid.uuid4().hex[:12]
             self._span_of[run_id] = sid
@@ -295,7 +296,8 @@ def apply_to_store(store, ev: dict, source: str = "ui") -> None:
     if kind == "run_open":
         store.open_run({"id": ev["run_id"], "graph": ev.get("graph"), "source": source,
                         "session": ev.get("session"), "tags": ev.get("tags"),
-                        "input": ev.get("input"), "started_ms": ev["started_ms"]})
+                        "input": ev.get("input"), "started_ms": ev["started_ms"],
+                        "thread_id": ev.get("thread_id")})
     elif kind == "span":
         store.add_span(ev)
     elif kind == "run_close":