jobgpumonitor-server 0.4.0__tar.gz

jobgpumonitor_server-0.4.0/.github/workflows/ci.yml

@@ -0,0 +1,19 @@
+name: CI
+on:
+  push:
+    branches: [main]
+  pull_request:
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python: ["3.9", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
+      - run: pip install ruff && ruff check src tests
+      - run: pip install -e ".[dev]"
+      - run: pytest -q -p no:warnings

jobgpumonitor_server-0.4.0/.github/workflows/release.yml

@@ -0,0 +1,47 @@
+name: Release
+
+# Tag a commit `vX.Y.Z` (matching the version in pyproject.toml) to build and publish to PyPI.
+# Uses PyPI trusted publishing: register this repo/workflow once at
+# https://pypi.org/manage/account/publishing/ (no API token needed).
+
+on:
+  push:
+    tags: ["v*"]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install build
+      - name: check tag matches package version
+        run: |
+          v=$(python -c "import tomllib;print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])")
+          test "v$v" = "${GITHUB_REF_NAME}" || { echo "tag ${GITHUB_REF_NAME} != v$v"; exit 1; }
+      - run: python -m build
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+      contents: write
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+      - uses: pypa/gh-action-pypi-publish@release/v1
+      - name: GitHub release
+        uses: softprops/action-gh-release@v2
+        with:
+          files: dist/*
+          generate_release_notes: true

jobgpumonitor_server-0.4.0/.gitignore

@@ -0,0 +1,14 @@
+__pycache__/
+*.pyc
+*.egg-info/
+.venv/
+dist/
+build/
+.pytest_cache/
+.ruff_cache/
+.DS_Store
+*.service
+*.db
+*.sqlite*
+update.bat
+!deploy/*.service

jobgpumonitor_server-0.4.0/PKG-INFO

@@ -0,0 +1,104 @@
+Metadata-Version: 2.5
+Name: jobgpumonitor-server
+Version: 0.4.0
+Summary: Consumer for jobgpumonitor events: run state, alert rules, notifications (ntfy, Telegram, webhook) and a small API.
+Project-URL: Homepage, https://github.com/marcpinet/jobgpumonitor-server
+Author: Marc Pinet
+License-Expression: MIT
+Keywords: gpu,hpc,monitoring,notifications,ntfy,oar,slurm
+Requires-Python: >=3.9
+Provides-Extra: api
+Requires-Dist: fastapi>=0.110; extra == 'api'
+Requires-Dist: uvicorn>=0.27; extra == 'api'
+Provides-Extra: dev
+Requires-Dist: fastapi>=0.110; extra == 'dev'
+Requires-Dist: httpx>=0.27; extra == 'dev'
+Requires-Dist: jobgpumonitor>=0.1.0; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Requires-Dist: tomli>=2; (python_version < '3.11') and extra == 'dev'
+Provides-Extra: toml
+Requires-Dist: tomli>=2; (python_version < '3.11') and extra == 'toml'
+Description-Content-Type: text/markdown
+
+# jobgpumonitor-server
+
+The consumer side of [jobgpumonitor](https://github.com/marcpinet/jobgpumonitor): reads the
+JSONL events written by the emitter and the scheduler probe, keeps one document per run,
+sends notifications (ntfy, Telegram, webhook) and serves a small read-only API.
+
+```
+jobgpumonitor (in the job) ──┐
+                             ├──> $JGM_DIR/runs/**.jsonl ──> jgmd serve ──> ntfy / Telegram / webhook
+jgm scheduler (login node) ──┘                                   └──> SQLite ──> HTTP API
+```
+
+## Quick start (login node)
+
+```bash
+pip install "jobgpumonitor-server[api]"
+jgmd init                         # writes ~/.config/jgm-server/config.toml
+$EDITOR ~/.config/jgm-server/config.toml   # put an ntfy topic or a Telegram bot in [notify.*]
+jgmd notify-test                  # phone should buzz
+jgmd serve --api                  # keep it in tmux / systemd --user
+```
+
+Then, still on the login node, `jgm scheduler` from the emitter package so that OOM,
+time-outs, preemptions and the queue are reported too.
+
+## What you get notified about
+
+| Alert | When |
+|---|---|
+| 🚀 started | the job leaves the queue (with node, GPUs, time queued, time limit) |
+| ✅ finished | clean end: duration, last metrics, GPU utilisation and idle share, peak memory |
+| ❌ failed / OOM / timed out / cancelled / preempted | with the traceback or the scheduler's reason, exit code, last stderr lines, path of the `.out` file |
+| ⚠️ correction | the scheduler's verdict contradicts what the process reported |
+| ⏱ will not finish in time | tqdm ETA overshoots the job deadline by more than 2 minutes |
+| 🧠 memory | above 90 % of the cgroup / requested memory |
+| 🥱 GPU idle | every visible GPU under 5 % for 15 minutes |
+| 💀 stopped reporting | no heartbeat for 3 intervals while the scheduler still says RUNNING (15 min when nothing at all arrives from the cluster: the agent or the proxy is then the likelier culprit) |
+| 🔄 reporting again | heartbeats are back after a 💀; the next silence is reported again |
+| ⚠️ correction | the scheduler's real verdict arrives after an `UNKNOWN_ENDED` (accounting was down) |
+
+Every alert fires once per run and is recorded, so restarting the server never re-sends.
+A channel that fails (ntfy down, no network) gets the alert again later, with backoff, for
+up to 6 hours; the other channels are not sent it twice. Raw events are kept
+`keep_events_days` after they are received (not after they were emitted).
+
+## API
+
+`jgmd serve --api` (needs the `[api]` extra) exposes on `127.0.0.1:21834`:
+
+```
+GET /health
+GET /runs?phase=running
+GET /runs/<cluster>/<job>/<restart>
+GET /runs/<cluster>/<job>/<restart>/events?after=0&types=metric.log,progress.update
+GET /runs/<cluster>/<job>/<restart>/stream          # server-sent events
+GET /runs/<cluster>/<job>/<restart>/logs?stream=stdout   # the job's .out/.err, live
+GET /alerts
+```
+
+Interactive docs at `/docs`. From your laptop: `ssh -L 21834:localhost:21834 cluster`.
+
+## CLI
+
+```
+jgmd serve [--once] [--api]    ingest, rules, notifications
+jgmd runs [--phase running]    table of runs
+jgmd show <run_id | job id>    full run document (--events N for raw events)
+jgmd alerts                    what was sent, and through which channel
+jgmd notify-test               send a test message
+jgmd init                      write the example config
+```
+
+Configuration lives in `~/.config/jgm-server/config.toml` (see `jgmd init`); `JGMD_DIRS`,
+`JGMD_NTFY_TOPIC`, `JGMD_TELEGRAM_TOKEN` / `JGMD_TELEGRAM_CHAT_ID`, `JGMD_WEBHOOK_URL` work
+without a file.
+
+## Development
+
+```bash
+uv venv && uv pip install -e ".[dev]" && uv run pytest
+```

jobgpumonitor_server-0.4.0/README.md

@@ -0,0 +1,81 @@
+# jobgpumonitor-server
+
+The consumer side of [jobgpumonitor](https://github.com/marcpinet/jobgpumonitor): reads the
+JSONL events written by the emitter and the scheduler probe, keeps one document per run,
+sends notifications (ntfy, Telegram, webhook) and serves a small read-only API.
+
+```
+jobgpumonitor (in the job) ──┐
+                             ├──> $JGM_DIR/runs/**.jsonl ──> jgmd serve ──> ntfy / Telegram / webhook
+jgm scheduler (login node) ──┘                                   └──> SQLite ──> HTTP API
+```
+
+## Quick start (login node)
+
+```bash
+pip install "jobgpumonitor-server[api]"
+jgmd init                         # writes ~/.config/jgm-server/config.toml
+$EDITOR ~/.config/jgm-server/config.toml   # put an ntfy topic or a Telegram bot in [notify.*]
+jgmd notify-test                  # phone should buzz
+jgmd serve --api                  # keep it in tmux / systemd --user
+```
+
+Then, still on the login node, `jgm scheduler` from the emitter package so that OOM,
+time-outs, preemptions and the queue are reported too.
+
+## What you get notified about
+
+| Alert | When |
+|---|---|
+| 🚀 started | the job leaves the queue (with node, GPUs, time queued, time limit) |
+| ✅ finished | clean end: duration, last metrics, GPU utilisation and idle share, peak memory |
+| ❌ failed / OOM / timed out / cancelled / preempted | with the traceback or the scheduler's reason, exit code, last stderr lines, path of the `.out` file |
+| ⚠️ correction | the scheduler's verdict contradicts what the process reported |
+| ⏱ will not finish in time | tqdm ETA overshoots the job deadline by more than 2 minutes |
+| 🧠 memory | above 90 % of the cgroup / requested memory |
+| 🥱 GPU idle | every visible GPU under 5 % for 15 minutes |
+| 💀 stopped reporting | no heartbeat for 3 intervals while the scheduler still says RUNNING (15 min when nothing at all arrives from the cluster: the agent or the proxy is then the likelier culprit) |
+| 🔄 reporting again | heartbeats are back after a 💀; the next silence is reported again |
+| ⚠️ correction | the scheduler's real verdict arrives after an `UNKNOWN_ENDED` (accounting was down) |
+
+Every alert fires once per run and is recorded, so restarting the server never re-sends.
+A channel that fails (ntfy down, no network) gets the alert again later, with backoff, for
+up to 6 hours; the other channels are not sent it twice. Raw events are kept
+`keep_events_days` after they are received (not after they were emitted).
+
+## API
+
+`jgmd serve --api` (needs the `[api]` extra) exposes on `127.0.0.1:21834`:
+
+```
+GET /health
+GET /runs?phase=running
+GET /runs/<cluster>/<job>/<restart>
+GET /runs/<cluster>/<job>/<restart>/events?after=0&types=metric.log,progress.update
+GET /runs/<cluster>/<job>/<restart>/stream          # server-sent events
+GET /runs/<cluster>/<job>/<restart>/logs?stream=stdout   # the job's .out/.err, live
+GET /alerts
+```
+
+Interactive docs at `/docs`. From your laptop: `ssh -L 21834:localhost:21834 cluster`.
+
+## CLI
+
+```
+jgmd serve [--once] [--api]    ingest, rules, notifications
+jgmd runs [--phase running]    table of runs
+jgmd show <run_id | job id>    full run document (--events N for raw events)
+jgmd alerts                    what was sent, and through which channel
+jgmd notify-test               send a test message
+jgmd init                      write the example config
+```
+
+Configuration lives in `~/.config/jgm-server/config.toml` (see `jgmd init`); `JGMD_DIRS`,
+`JGMD_NTFY_TOPIC`, `JGMD_TELEGRAM_TOKEN` / `JGMD_TELEGRAM_CHAT_ID`, `JGMD_WEBHOOK_URL` work
+without a file.
+
+## Development
+
+```bash
+uv venv && uv pip install -e ".[dev]" && uv run pytest
+```

jobgpumonitor_server-0.4.0/deploy/README.md

@@ -0,0 +1,8 @@
+# Deploying on a Linux box (systemd)
+
+1. Copy `jobgpumonitor-server.example.service` to `../jobgpumonitor-server.service`, replace `USER`.
+2. Copy `update.example.bat` to `../update.bat`, set `HOST`. Both copies are gitignored.
+3. Run `update.bat` from Windows (or the same ssh lines from any shell). It pulls, installs into `.venv`,
+   installs the unit and (re)starts the service. The API listens on `127.0.0.1:21834`; put your
+   reverse proxy in front of it.
+4. Configuration lives in `~/.config/jgm-server/config.toml` on the host (`jgmd init` writes an example).

jobgpumonitor_server-0.4.0/deploy/jobgpumonitor-server.example.service

@@ -0,0 +1,15 @@
+[Unit]
+Description=jobgpumonitor server (run state, alerts, API)
+After=network.target
+
+[Service]
+User=USER
+Type=simple
+WorkingDirectory=/home/USER/jobgpumonitor-server/
+ExecStart=/home/USER/jobgpumonitor-server/.venv/bin/jgmd serve --api
+Restart=always
+RestartSec=5
+Environment=PYTHONUNBUFFERED=1
+
+[Install]
+WantedBy=multi-user.target

jobgpumonitor_server-0.4.0/deploy/update.example.bat

@@ -0,0 +1,11 @@
+@echo off
+:: Deploy to a Linux host over SSH (edit HOST). update.bat itself is gitignored: copy this file.
+set HOST=USER@your-host.example
+ssh %HOST% "cd jobgpumonitor-server && git pull"
+ssh %HOST% "sudo systemctl stop jobgpumonitor-server"
+ssh %HOST% "cd jobgpumonitor-server && [ ! -d .venv ] && python3 -m venv .venv"
+ssh %HOST% "cd jobgpumonitor-server && source .venv/bin/activate && pip install --upgrade pip && pip install -e '.[api,toml]'"
+scp jobgpumonitor-server.service %HOST%:jobgpumonitor-server/
+ssh %HOST% "sudo cp jobgpumonitor-server/jobgpumonitor-server.service /etc/systemd/system/"
+ssh %HOST% "sudo systemctl enable jobgpumonitor-server && sudo systemctl daemon-reload && sudo systemctl start jobgpumonitor-server"
+ssh %HOST% "sleep 2 && systemctl is-active jobgpumonitor-server && curl -s http://127.0.0.1:21834/health"

jobgpumonitor_server-0.4.0/pyproject.toml

@@ -0,0 +1,42 @@
+[build-system]
+requires = ["hatchling>=1.25"]
+build-backend = "hatchling.build"
+
+[project]
+name = "jobgpumonitor-server"
+version = "0.4.0"
+description = "Consumer for jobgpumonitor events: run state, alert rules, notifications (ntfy, Telegram, webhook) and a small API."
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.9"
+authors = [{ name = "Marc Pinet" }]
+keywords = ["slurm", "oar", "hpc", "gpu", "monitoring", "notifications", "ntfy"]
+dependencies = []
+
+[project.optional-dependencies]
+toml = ["tomli>=2; python_version < '3.11'"]
+api = ["fastapi>=0.110", "uvicorn>=0.27"]
+dev = ["pytest>=8", "ruff>=0.6", "jobgpumonitor>=0.1.0", "httpx>=0.27", "fastapi>=0.110", "tomli>=2; python_version < '3.11'"]
+
+[project.scripts]
+jgmd = "jobgpumonitor_server.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/marcpinet/jobgpumonitor-server"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/jobgpumonitor_server"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-q"
+
+[tool.ruff]
+line-length = 110
+target-version = "py39"
+src = ["src", "tests"]
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B"]
+ignore = ["E501", "UP006", "UP035", "UP007", "UP045", "B008"]
+

jobgpumonitor_server-0.4.0/src/jobgpumonitor_server/__init__.py

@@ -0,0 +1,3 @@
+"""jobgpumonitor-server: consume jobgpumonitor events, keep run state, alert, expose an API."""
+
+__version__ = "0.4.0"

jobgpumonitor_server-0.4.0/src/jobgpumonitor_server/api.py

@@ -0,0 +1,187 @@
+"""Read-only HTTP API over the store (optional: ``pip install "jobgpumonitor-server[api]"``).
+
+    GET /health
+    GET /runs?phase=running&cluster=…&limit=100
+    GET /runs/{run_id}                 (run_id contains slashes: /runs/marcel-c3/8224458/0)
+    GET /runs/{run_id}/events?after=0&limit=1000&types=run.heartbeat,metric.log
+    GET /runs/{run_id}/stream          server-sent events, new events as they are ingested
+    GET /alerts?limit=100
+    GET /runs/{run_id}/logs[?stream=stdout&tail=200000]   live stdout/stderr rebuilt from log.chunk events
+    POST /ingest                       JSON list of events (gzip ok), bearer = ingest_token
+"""
+
+import asyncio
+import gzip
+import hmac
+import io
+import json
+import os
+import re
+import time
+import zlib
+from typing import Any, Dict, List, Optional
+
+from .store import Store
+
+
+def append_events(base_dir: str, events: List[Any]) -> "tuple[int, int]":
+    """Append well-formed events to ``<base_dir>/runs/<run_id>/<emitter>.jsonl``. Returns (accepted, rejected)."""
+    accepted = rejected = 0
+    handles: Dict[str, Any] = {}
+    try:
+        for ev in events:
+            if not (isinstance(ev, dict) and isinstance(ev.get("run_id"), str) and isinstance(ev.get("emitter"), str)
+                    and isinstance(ev.get("type"), str) and isinstance(ev.get("data"), dict)
+                    and _RUN_ID_RE.match(ev["run_id"]) and _EMITTER_RE.match(ev["emitter"])
+                    and not any(p in (".", "..") for p in [*ev["run_id"].split("/"), ev["emitter"]])):
+                rejected += 1
+                continue
+            path = os.path.join(base_dir, "runs", *ev["run_id"].split("/"), ev["emitter"] + ".jsonl")
+            fh = handles.get(path)
+            if fh is None:
+                os.makedirs(os.path.dirname(path), exist_ok=True)
+                fh = handles[path] = open(path, "a", encoding="utf-8")
+            fh.write(json.dumps(ev, separators=(",", ":"), ensure_ascii=False) + "\n")
+            accepted += 1
+    finally:
+        for fh in handles.values():
+            fh.flush()
+            fh.close()
+    return accepted, rejected
+
+
+def create_app(store: Store, token: str = "", prefix: str = "", ingest_token: str = "", ingest_dir: str = "") -> Any:
+    """Build the app. With ``prefix`` (e.g. ``/jgm``) every route is served under that path,
+    which lets a reverse proxy expose the API as a sub-path of an existing host."""
+    app = _build_app(store, token, ingest_token, ingest_dir)
+    prefix = "/" + prefix.strip("/") if prefix and prefix.strip("/") else ""
+    if not prefix:
+        return app
+    from fastapi import FastAPI
+
+    outer = FastAPI(docs_url=None, redoc_url=None, openapi_url=None)
+    outer.mount(prefix, app)
+    return outer
+
+
+_RUN_ID_RE = re.compile(r"^[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+/[0-9]+$")
+_EMITTER_RE = re.compile(r"^[A-Za-z0-9_.\-]+$")
+_MAX_BODY = 32 * 1024 * 1024
+#: Once decompressed (``jgm forward`` sends at most ~4 MB of JSONL per request; a gzip bomb
+#: inflates ~1000x).
+_MAX_INFLATED = 64 * 1024 * 1024
+
+
+def _gunzip(raw: bytes, limit: int) -> bytes:
+    """Decompress, reading at most ``limit + 1`` bytes: memory stays bounded whatever the ratio."""
+    with gzip.GzipFile(fileobj=io.BytesIO(raw)) as gz:
+        return gz.read(limit + 1)
+
+
+def _build_app(store: Store, token: str = "", ingest_token: str = "", ingest_dir: str = "") -> Any:
+    try:
+        from fastapi import Depends, FastAPI, HTTPException, Query, Request
+        from fastapi.responses import StreamingResponse
+    except ImportError as e:  # pragma: no cover
+        raise SystemExit('the API needs fastapi and uvicorn: pip install "jobgpumonitor-server[api]"') from e
+
+    app = FastAPI(title="jobgpumonitor-server", version="0.1.0")
+
+    async def auth(request: Request) -> None:
+        if not token:
+            return
+        got = request.headers.get("authorization") or ""
+        if not hmac.compare_digest(got.encode(), f"Bearer {token}".encode()):
+            raise HTTPException(status_code=401, detail="bad token", headers={"WWW-Authenticate": "Bearer"})
+
+    async def ingest_auth(request: Request) -> None:
+        if not ingest_token:
+            raise HTTPException(status_code=503, detail="ingest disabled: set [api] ingest_token")
+        got = request.headers.get("authorization") or ""
+        if not hmac.compare_digest(got.encode(), f"Bearer {ingest_token}".encode()):
+            raise HTTPException(status_code=401, detail="bad ingest token", headers={"WWW-Authenticate": "Bearer"})
+
+    @app.post("/ingest", dependencies=[Depends(ingest_auth)])
+    async def ingest(request: Request) -> Dict[str, Any]:
+        """Accept a JSON list of events (optionally gzip-encoded) and append them to the local
+        event directory, where the engine picks them up exactly like locally written files."""
+        try:
+            declared = int(request.headers.get("content-length") or 0)
+        except ValueError:
+            declared = 0
+        if declared > _MAX_BODY:
+            raise HTTPException(status_code=413, detail="body too large")
+        raw = await request.body()
+        if len(raw) > _MAX_BODY:
+            raise HTTPException(status_code=413, detail="body too large")
+        if request.headers.get("content-encoding", "").lower() == "gzip":
+            try:
+                raw = _gunzip(raw, _MAX_INFLATED)
+            except (OSError, EOFError, zlib.error) as e:
+                raise HTTPException(status_code=400, detail="bad gzip") from e
+            if len(raw) > _MAX_INFLATED:
+                raise HTTPException(status_code=413, detail="decompressed body too large")
+        try:
+            events = json.loads(raw.decode("utf-8"))
+        except (ValueError, UnicodeDecodeError) as e:
+            raise HTTPException(status_code=400, detail="body must be a JSON list of events") from e
+        if not isinstance(events, list):
+            raise HTTPException(status_code=400, detail="body must be a JSON list of events")
+        accepted, rejected = append_events(ingest_dir, events)
+        return {"accepted": accepted, "rejected": rejected}
+
+    @app.api_route("/health", methods=["GET", "HEAD"])
+    def health() -> Dict[str, Any]:
+        return {"ok": True, "ts": time.time(), "active": len(store.active_runs())}
+
+    @app.get("/runs", dependencies=[Depends(auth)])
+    def runs(phase: Optional[str] = None, cluster: Optional[str] = None, limit: int = Query(100, le=1000)) -> List[Dict[str, Any]]:
+        return store.list_runs(phase=phase, limit=limit, cluster=cluster)
+
+    @app.get("/alerts", dependencies=[Depends(auth)])
+    def alerts(limit: int = Query(100, le=1000), run_id: Optional[str] = None) -> List[Dict[str, Any]]:
+        return store.list_alerts(limit=limit, run_id=run_id)
+
+    @app.get("/runs/{run_id:path}/events", dependencies=[Depends(auth)])
+    def events(run_id: str, after: int = 0, limit: int = Query(1000, le=10000), types: Optional[str] = None) -> List[Dict[str, Any]]:
+        return store.events(run_id, after_id=after, limit=limit, types=types.split(",") if types else None)
+
+    @app.get("/runs/{run_id:path}/logs", dependencies=[Depends(auth)])
+    def logs(run_id: str, stream: Optional[str] = None, tail: int = Query(200_000, le=4_000_000)) -> Any:
+        """Without ``stream``: the streams available. With it: the (tail of the) reconstructed file."""
+        if not stream:
+            return store.log_streams(run_id)
+        doc = store.get_log(run_id, stream, tail=tail)
+        if doc is None:
+            raise HTTPException(status_code=404, detail="no such log")
+        return doc
+
+    @app.get("/runs/{run_id:path}/stream", dependencies=[Depends(auth)])
+    async def stream(run_id: str, after: int = 0) -> Any:
+        async def gen():
+            last = after
+            while True:
+                batch = store.events(run_id, after_id=last, limit=500)
+                for e in batch:
+                    last = e["_id"]
+                    yield f"id: {last}\nevent: {e['type']}\ndata: {json.dumps(e, separators=(',', ':'))}\n\n"
+                if not batch:
+                    yield ": keepalive\n\n"
+                    await asyncio.sleep(2)
+
+        return StreamingResponse(gen(), media_type="text/event-stream")
+
+    @app.get("/runs/{run_id:path}", dependencies=[Depends(auth)])
+    def run(run_id: str) -> Dict[str, Any]:
+        doc = store.get_run(run_id)
+        if doc is None:
+            raise HTTPException(status_code=404, detail="unknown run")
+        return doc
+
+    return app
+
+
+def serve_api(store: Store, host: str, port: int, token: str = "", prefix: str = "", ingest_token: str = "", ingest_dir: str = "") -> None:  # pragma: no cover
+    import uvicorn
+
+    uvicorn.run(create_app(store, token, prefix, ingest_token, ingest_dir), host=host, port=port, log_level="warning")