tkati-dashboard 0.4.0__tar.gz

tkati_dashboard-0.4.0/PKG-INFO

@@ -0,0 +1,172 @@
+Metadata-Version: 2.4
+Name: tkati-dashboard
+Version: 0.4.0
+Summary: Local web dashboard that renders a serialized tkati dataflow directory as a graph
+Requires-Python: >=3.13
+Description-Content-Type: text/markdown
+Requires-Dist: tkati-core==0.4.0
+Requires-Dist: fastapi>=0.115.0
+Requires-Dist: uvicorn>=0.34.0
+Requires-Dist: confluent-kafka>=2.11.0
+Requires-Dist: orjson>=3.9.0
+Requires-Dist: pyyaml>=6.0
+
+# tkati-dashboard — dataflow graph viewer
+
+Reads a serialized tkati dataflow directory (see
+[docs/dataflow-serialization.md](../../docs/dataflow-serialization.md)) and serves a local web page
+rendering it as a graph — no live runtime process required, no manifest to maintain, just a
+directory of `*.json`/`*.yaml`/`*.yml` fragment files (freely mixable — both encodings merge into
+the same graph identically).
+
+## Usage
+
+```sh
+tkati-dashboard path/to/dataflow-dir
+```
+
+Then open `http://127.0.0.1:8000/` in a browser. Pass more than one directory (or
+`--flows-root`, see [Multiple flows](#multiple-flows) below) to observe several dataflows from
+one dashboard instead of just this one. The page fetches the list of flows from `/api/flows`,
+then the selected one's graph from `/api/flows/{id}/graph`, and renders it with
+[React Flow](https://reactflow.dev), laid out by [dagre](https://github.com/dagrejs/dagre) using
+each node's real measured box (from its label text, via canvas measurement — no DOM mount needed)
+rather than a flat grid, so a node with a long `broker`/`topic` string pushes its neighbors aside
+instead of overlapping them. The graph always lays out left-to-right.
+
+A node consuming a stream with a consumer group doesn't get that edge's `group`/`lag` as a
+floating label — it gets its own stacked row inside the *consuming* node instead, right below the
+node's own header, one row per such input (so a fan-in node like a sessionizer joining two topics
+shows two rows, each fed by its own arrow landing directly on its row). An edge with nothing extra
+to show beyond its `kind` (no consumer group) stays a plain, unlabeled-beyond-`kind` line into the
+node's header. Source/sink nodes (`kafka-topic`, `clickhouse-table`) and processing nodes are
+colored differently. Click a node to fill the always-visible inspector panel on the right — the
+whole node box glows blue while selected, and every edge into or out of it (in the graph itself)
+turns the same blue and draws on top of any edge it crosses, so what a node actually talks to is
+easy to trace at a glance — with its full connection/config/schema details, plus every stream
+edge touching it and that edge's live consumer lag; drag the panel's left edge to
+resize it (the width is remembered across reloads).
+
+### Try it with the bundled example
+
+[`examples/simple-pipeline`](examples/simple-pipeline) is the smallest interesting dataflow: two
+Kafka topics and one processing node (`raw-events` → `dedup` → `deduped-events`). Its `connection`
+blocks point at `localhost:9092`, so if you have a broker there, seed it with sample events first:
+
+```sh
+uv run python packages/tkati-dashboard/examples/simple-pipeline/seed_kafka.py
+uv run tkati-dashboard packages/tkati-dashboard/examples/simple-pipeline
+```
+
+Then open <http://127.0.0.1:8000/>, click the `raw-events` or `deduped-events` node, and click
+"Load latest events" in the side panel to see the real messages the seed script just produced
+(`raw-events` includes two intentional duplicate `event_id`s so you can see what the `dedup` node
+in between is for).
+
+For a bigger graph exercising fragment merging across `topics.json`, `tables.json`, `nodes.json`,
+and `edges.json` (two raw topics → dedup → a sessionize/enrich node → a ClickHouse table, plus a
+side branch straight to another table), see [`examples/analytics-pipeline`](examples/analytics-pipeline)
+— its topics aren't seeded with data, so "Latest events" there will error unless you produce to
+them yourself.
+
+Since `examples/` itself contains both as sibling directories, it also doubles as a ready-made
+`--flows-root` demo — `tkati-dashboard --flows-root packages/tkati-dashboard/examples` serves both
+as separate flows, switchable from the ☰ menu (see [Multiple flows](#multiple-flows)).
+
+Options:
+
+- `--host` (default `127.0.0.1`)
+- `--port` (default `8000`)
+- `--flows-root DIR` — auto-discover flows (see below); repeatable
+
+A dataflow directory is re-read on every request to its `/api/flows/{id}/graph`, so editing the
+fragments and refreshing the browser picks up the change without restarting the server.
+
+## Multiple flows
+
+A real deployment usually runs more than one dataflow per environment, and one dashboard
+instance can observe all of them at once instead of one directory per instance:
+
+```sh
+# name each flow directory explicitly (flow id = each directory's own basename)
+tkati-dashboard path/to/orders-flow path/to/clicks-flow
+
+# or point at a parent directory and let every fragment-containing subdirectory become a flow
+tkati-dashboard --flows-root path/to/env/flows
+```
+
+The two forms can be combined, and `--flows-root` is repeatable. Either way, a flow's id is its
+directory's own basename — passing two directories with the same name is a startup error asking
+you to rename one of them. Unlike the explicit directories, a `--flows-root`'s subdirectories are
+rescanned on every request to `/api/flows`, so adding or removing a flow directory there shows up
+without restarting the server; a subdirectory with no fragments in it (a stray `README`, say) is
+silently skipped rather than treated as an error.
+
+With more than one flow configured, the graph page gets a ☰ menu in its top-left corner naming the
+currently selected flow — click it to switch to another one. It stays collapsed to that single
+button otherwise, so the canvas keeps its space once a flow is picked.
+The current flow is also reflected in the page's URL as `?flow=<id>`, so a link to a specific
+flow's view can be bookmarked or shared. With only one flow configured (the common case, and the
+default if you pass a single directory as before), the menu doesn't appear at all — the page looks
+exactly as it always has.
+
+## Node panel
+
+The panel on the right is always there, like an inspector rather than a popup — it shows a
+placeholder until you click a node, then shows that node's full metadata: connection settings,
+`config`, and `schema` (field → type). Clicking empty canvas clears the selection back to the
+placeholder. Every section has a clickable header to collapse/expand it — a collapsed section
+stays mounted, just hidden, so collapsing a live-fetching section and reopening it doesn't
+re-fetch. For a `kafka-topic` node, the panel also fetches two more, live views:
+
+- `GET /api/flows/{flow_id}/nodes/{id}/snapshot` connects to `connection.broker`/`connection.topic` and shows the
+  most recent messages on that topic (newest last), parsed as JSON, using a throwaway consumer
+  group that never commits offsets. It only fetches on demand — click "Load latest events" — and
+  a "↻ Refresh" button afterward pulls a fresh batch on request rather than automatically. Each
+  message renders as its own pretty-printed JSON block (fields reordered to match `schema`, when
+  there is one) rather than a table — real messages commonly carry 5-20 fields, too many to lay
+  out sensibly as table columns in a side panel — with a small header showing that message's
+  Kafka-level `partition`, `offset`, and timestamp, alongside its parsed body.
+- `GET /api/flows/{flow_id}/nodes/{id}/topic-stats` shows the topic's partitioning and replication (per-partition
+  leader/replicas/in-sync-replicas, flagging any under-replicated partition) and its topic-level
+  config — `retention.ms`, `retention.bytes`, `cleanup.policy`, `segment.bytes`,
+  `compression.type`, `max.message.bytes` — via `tkati_dashboard.topic_stats`, marking each value
+  that differs from the broker default. Partition/replica info comes from the same topic metadata
+  lookup as the snapshot/lag features (`_kafka_metadata.py`); the config comes from a separate
+  read-only `AdminClient.describe_configs()` call.
+
+These are two of the places `tkati-dashboard` talks to a live broker rather than just the
+serialized directory — best-effort conveniences for the panel, not something the graph view itself
+depends on: a broker that's unreachable, or a topic that doesn't exist, shows an inline error in
+that section instead of breaking the page.
+
+## Consumer lag
+
+For every stream edge whose `consumer.group_id` is set and whose source is a `kafka-topic`, the
+page also fetches `GET /api/flows/{flow_id}/nodes/{topic_id}/consumer-lag?group_id=...` and shows the result in
+two places: the `group`/`lag` lines in that edge's stacked row inside the consuming node (see
+above), and, for either node that edge touches, the inspector panel's "Consumer lag" section
+(`← other-node (group_id)` for an edge consumed by the selected node, `→ other-node (group_id)`
+for one where it's the topic being consumed). This is another place `tkati-dashboard` talks to a
+live broker: it looks up `group_id`'s committed offset with `Consumer.committed()` and compares
+it to the topic's high watermark — it never subscribes or polls as that group, so it can't join
+it, trigger a rebalance, or otherwise disturb a real pipeline's consumer. A group that has never
+committed an offset is reported as fully behind (lag = the topic's full size); an unreachable
+broker shows `lag: n/a` instead of failing the page.
+
+Lag is time-sensitive, so it doesn't just get fetched once. A ☰-style control in the canvas's
+top-right corner (shown whenever the graph has at least one such edge) works like Grafana's
+refresh picker: a "↻" button re-fetches every visible edge's lag immediately, and a dropdown next
+to it sets an auto-refresh interval (Off/5s/15s/30s/1m/5m, persisted across reloads, paused while
+the browser tab isn't visible). The inspector's "Consumer lag" section additionally has its own
+per-row "↻" to refresh just the one edge you're looking at, without waiting for the next tick or
+refreshing every other edge in the graph.
+
+## Validation
+
+`tkati_dashboard.dataflow.load_dataflow` enforces the rules from the serialization doc: the
+directory must contain at least one `*.json`/`*.yaml`/`*.yml` fragment, node ids must be unique
+(or identically redefined) across fragments, edges must reference existing nodes, and a node's `schema`, when
+present, must use field types known to `tkati_core.type_mapping` — `schema` itself is always
+optional, since it isn't always on hand for a real-world node. A validation failure surfaces as
+an HTTP 422 with the error message, shown inline on the page instead of a blank graph.

tkati_dashboard-0.4.0/README.md

@@ -0,0 +1,159 @@
+# tkati-dashboard — dataflow graph viewer
+
+Reads a serialized tkati dataflow directory (see
+[docs/dataflow-serialization.md](../../docs/dataflow-serialization.md)) and serves a local web page
+rendering it as a graph — no live runtime process required, no manifest to maintain, just a
+directory of `*.json`/`*.yaml`/`*.yml` fragment files (freely mixable — both encodings merge into
+the same graph identically).
+
+## Usage
+
+```sh
+tkati-dashboard path/to/dataflow-dir
+```
+
+Then open `http://127.0.0.1:8000/` in a browser. Pass more than one directory (or
+`--flows-root`, see [Multiple flows](#multiple-flows) below) to observe several dataflows from
+one dashboard instead of just this one. The page fetches the list of flows from `/api/flows`,
+then the selected one's graph from `/api/flows/{id}/graph`, and renders it with
+[React Flow](https://reactflow.dev), laid out by [dagre](https://github.com/dagrejs/dagre) using
+each node's real measured box (from its label text, via canvas measurement — no DOM mount needed)
+rather than a flat grid, so a node with a long `broker`/`topic` string pushes its neighbors aside
+instead of overlapping them. The graph always lays out left-to-right.
+
+A node consuming a stream with a consumer group doesn't get that edge's `group`/`lag` as a
+floating label — it gets its own stacked row inside the *consuming* node instead, right below the
+node's own header, one row per such input (so a fan-in node like a sessionizer joining two topics
+shows two rows, each fed by its own arrow landing directly on its row). An edge with nothing extra
+to show beyond its `kind` (no consumer group) stays a plain, unlabeled-beyond-`kind` line into the
+node's header. Source/sink nodes (`kafka-topic`, `clickhouse-table`) and processing nodes are
+colored differently. Click a node to fill the always-visible inspector panel on the right — the
+whole node box glows blue while selected, and every edge into or out of it (in the graph itself)
+turns the same blue and draws on top of any edge it crosses, so what a node actually talks to is
+easy to trace at a glance — with its full connection/config/schema details, plus every stream
+edge touching it and that edge's live consumer lag; drag the panel's left edge to
+resize it (the width is remembered across reloads).
+
+### Try it with the bundled example
+
+[`examples/simple-pipeline`](examples/simple-pipeline) is the smallest interesting dataflow: two
+Kafka topics and one processing node (`raw-events` → `dedup` → `deduped-events`). Its `connection`
+blocks point at `localhost:9092`, so if you have a broker there, seed it with sample events first:
+
+```sh
+uv run python packages/tkati-dashboard/examples/simple-pipeline/seed_kafka.py
+uv run tkati-dashboard packages/tkati-dashboard/examples/simple-pipeline
+```
+
+Then open <http://127.0.0.1:8000/>, click the `raw-events` or `deduped-events` node, and click
+"Load latest events" in the side panel to see the real messages the seed script just produced
+(`raw-events` includes two intentional duplicate `event_id`s so you can see what the `dedup` node
+in between is for).
+
+For a bigger graph exercising fragment merging across `topics.json`, `tables.json`, `nodes.json`,
+and `edges.json` (two raw topics → dedup → a sessionize/enrich node → a ClickHouse table, plus a
+side branch straight to another table), see [`examples/analytics-pipeline`](examples/analytics-pipeline)
+— its topics aren't seeded with data, so "Latest events" there will error unless you produce to
+them yourself.
+
+Since `examples/` itself contains both as sibling directories, it also doubles as a ready-made
+`--flows-root` demo — `tkati-dashboard --flows-root packages/tkati-dashboard/examples` serves both
+as separate flows, switchable from the ☰ menu (see [Multiple flows](#multiple-flows)).
+
+Options:
+
+- `--host` (default `127.0.0.1`)
+- `--port` (default `8000`)
+- `--flows-root DIR` — auto-discover flows (see below); repeatable
+
+A dataflow directory is re-read on every request to its `/api/flows/{id}/graph`, so editing the
+fragments and refreshing the browser picks up the change without restarting the server.
+
+## Multiple flows
+
+A real deployment usually runs more than one dataflow per environment, and one dashboard
+instance can observe all of them at once instead of one directory per instance:
+
+```sh
+# name each flow directory explicitly (flow id = each directory's own basename)
+tkati-dashboard path/to/orders-flow path/to/clicks-flow
+
+# or point at a parent directory and let every fragment-containing subdirectory become a flow
+tkati-dashboard --flows-root path/to/env/flows
+```
+
+The two forms can be combined, and `--flows-root` is repeatable. Either way, a flow's id is its
+directory's own basename — passing two directories with the same name is a startup error asking
+you to rename one of them. Unlike the explicit directories, a `--flows-root`'s subdirectories are
+rescanned on every request to `/api/flows`, so adding or removing a flow directory there shows up
+without restarting the server; a subdirectory with no fragments in it (a stray `README`, say) is
+silently skipped rather than treated as an error.
+
+With more than one flow configured, the graph page gets a ☰ menu in its top-left corner naming the
+currently selected flow — click it to switch to another one. It stays collapsed to that single
+button otherwise, so the canvas keeps its space once a flow is picked.
+The current flow is also reflected in the page's URL as `?flow=<id>`, so a link to a specific
+flow's view can be bookmarked or shared. With only one flow configured (the common case, and the
+default if you pass a single directory as before), the menu doesn't appear at all — the page looks
+exactly as it always has.
+
+## Node panel
+
+The panel on the right is always there, like an inspector rather than a popup — it shows a
+placeholder until you click a node, then shows that node's full metadata: connection settings,
+`config`, and `schema` (field → type). Clicking empty canvas clears the selection back to the
+placeholder. Every section has a clickable header to collapse/expand it — a collapsed section
+stays mounted, just hidden, so collapsing a live-fetching section and reopening it doesn't
+re-fetch. For a `kafka-topic` node, the panel also fetches two more, live views:
+
+- `GET /api/flows/{flow_id}/nodes/{id}/snapshot` connects to `connection.broker`/`connection.topic` and shows the
+  most recent messages on that topic (newest last), parsed as JSON, using a throwaway consumer
+  group that never commits offsets. It only fetches on demand — click "Load latest events" — and
+  a "↻ Refresh" button afterward pulls a fresh batch on request rather than automatically. Each
+  message renders as its own pretty-printed JSON block (fields reordered to match `schema`, when
+  there is one) rather than a table — real messages commonly carry 5-20 fields, too many to lay
+  out sensibly as table columns in a side panel — with a small header showing that message's
+  Kafka-level `partition`, `offset`, and timestamp, alongside its parsed body.
+- `GET /api/flows/{flow_id}/nodes/{id}/topic-stats` shows the topic's partitioning and replication (per-partition
+  leader/replicas/in-sync-replicas, flagging any under-replicated partition) and its topic-level
+  config — `retention.ms`, `retention.bytes`, `cleanup.policy`, `segment.bytes`,
+  `compression.type`, `max.message.bytes` — via `tkati_dashboard.topic_stats`, marking each value
+  that differs from the broker default. Partition/replica info comes from the same topic metadata
+  lookup as the snapshot/lag features (`_kafka_metadata.py`); the config comes from a separate
+  read-only `AdminClient.describe_configs()` call.
+
+These are two of the places `tkati-dashboard` talks to a live broker rather than just the
+serialized directory — best-effort conveniences for the panel, not something the graph view itself
+depends on: a broker that's unreachable, or a topic that doesn't exist, shows an inline error in
+that section instead of breaking the page.
+
+## Consumer lag
+
+For every stream edge whose `consumer.group_id` is set and whose source is a `kafka-topic`, the
+page also fetches `GET /api/flows/{flow_id}/nodes/{topic_id}/consumer-lag?group_id=...` and shows the result in
+two places: the `group`/`lag` lines in that edge's stacked row inside the consuming node (see
+above), and, for either node that edge touches, the inspector panel's "Consumer lag" section
+(`← other-node (group_id)` for an edge consumed by the selected node, `→ other-node (group_id)`
+for one where it's the topic being consumed). This is another place `tkati-dashboard` talks to a
+live broker: it looks up `group_id`'s committed offset with `Consumer.committed()` and compares
+it to the topic's high watermark — it never subscribes or polls as that group, so it can't join
+it, trigger a rebalance, or otherwise disturb a real pipeline's consumer. A group that has never
+committed an offset is reported as fully behind (lag = the topic's full size); an unreachable
+broker shows `lag: n/a` instead of failing the page.
+
+Lag is time-sensitive, so it doesn't just get fetched once. A ☰-style control in the canvas's
+top-right corner (shown whenever the graph has at least one such edge) works like Grafana's
+refresh picker: a "↻" button re-fetches every visible edge's lag immediately, and a dropdown next
+to it sets an auto-refresh interval (Off/5s/15s/30s/1m/5m, persisted across reloads, paused while
+the browser tab isn't visible). The inspector's "Consumer lag" section additionally has its own
+per-row "↻" to refresh just the one edge you're looking at, without waiting for the next tick or
+refreshing every other edge in the graph.
+
+## Validation
+
+`tkati_dashboard.dataflow.load_dataflow` enforces the rules from the serialization doc: the
+directory must contain at least one `*.json`/`*.yaml`/`*.yml` fragment, node ids must be unique
+(or identically redefined) across fragments, edges must reference existing nodes, and a node's `schema`, when
+present, must use field types known to `tkati_core.type_mapping` — `schema` itself is always
+optional, since it isn't always on hand for a real-world node. A validation failure surfaces as
+an HTTP 422 with the error message, shown inline on the page instead of a blank graph.

tkati_dashboard-0.4.0/pyproject.toml

@@ -0,0 +1,38 @@
+[project]
+name = "tkati-dashboard"
+version = "0.4.0"
+description = "Local web dashboard that renders a serialized tkati dataflow directory as a graph"
+readme = "README.md"
+requires-python = ">=3.13"
+dependencies = [
+    "tkati-core==0.4.0",
+    "fastapi>=0.115.0",
+    "uvicorn>=0.34.0",
+    "confluent-kafka>=2.11.0",
+    "orjson>=3.9.0",
+    "pyyaml>=6.0",
+]
+
+[project.scripts]
+tkati-dashboard = "tkati_dashboard.main:main"
+
+[dependency-groups]
+dev = ["pytest>=9.0.1", "httpx>=0.28.0"]
+
+[build-system]
+requires = ["setuptools", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["tkati_dashboard*"]
+
+[tool.setuptools.package-data]
+tkati_dashboard = ["static/*.html"]
+
+[tool.uv]
+package = true
+
+[tool.uv-workspace-codegen]
+generate = true
+template_type = ["test", "publish"]

tkati_dashboard-0.4.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

tkati_dashboard-0.4.0/src/tkati_dashboard/__init__.py

@@ -0,0 +1,9 @@
+from tkati_dashboard.app import create_app
+from tkati_dashboard.dataflow import Dataflow, DataflowValidationError, load_dataflow
+
+__all__ = [
+    "Dataflow",
+    "DataflowValidationError",
+    "create_app",
+    "load_dataflow",
+]

tkati_dashboard-0.4.0/src/tkati_dashboard/__main__.py

@@ -0,0 +1,4 @@
+from .main import main
+
+if __name__ == "__main__":
+    main()

tkati_dashboard-0.4.0/src/tkati_dashboard/_kafka_metadata.py

@@ -0,0 +1,31 @@
+"""Shared "resolve a live topic's metadata" helpers for snapshot.py, lag.py, and topic_stats.py."""
+
+from typing import Any
+
+from confluent_kafka import Consumer
+
+
+def resolve_topic_metadata(
+    consumer: Consumer, broker: str, topic: str, timeout_sec: float
+) -> Any:
+    """Return `topic`'s TopicMetadata (partitions, each with leader/replicas/isrs), or raise
+    RuntimeError with a message naming `broker`/`topic` if the broker is unreachable or the
+    topic doesn't exist.
+    """
+    try:
+        metadata = consumer.list_topics(topic, timeout=timeout_sec)
+    except Exception as e:
+        raise RuntimeError(f"Could not reach broker {broker!r}: {e}") from e
+
+    topic_metadata = metadata.topics.get(topic)
+    if topic_metadata is None or topic_metadata.error is not None:
+        raise RuntimeError(f"Topic {topic!r} not found on {broker!r}")
+
+    return topic_metadata
+
+
+def resolve_partitions(
+    consumer: Consumer, broker: str, topic: str, timeout_sec: float
+) -> list[int]:
+    """Return the partition ids of `topic`. See resolve_topic_metadata for error behavior."""
+    return list(resolve_topic_metadata(consumer, broker, topic, timeout_sec).partitions)

tkati_dashboard-0.4.0/src/tkati_dashboard/app.py

@@ -0,0 +1,155 @@
+from collections.abc import Callable
+from pathlib import Path
+from typing import Any
+
+from fastapi import FastAPI, HTTPException
+from fastapi.responses import FileResponse
+
+from tkati_dashboard import lag, snapshot, topic_stats
+from tkati_dashboard.dataflow import (
+    SOURCE_SINK_TYPES,
+    DataflowValidationError,
+    NodeDef,
+    load_dataflow,
+)
+from tkati_dashboard.flows import FlowConfigError
+
+STATIC_DIR = Path(__file__).parent / "static"
+
+
+def _list_flows(list_flows: Callable[[], dict[str, Path]]) -> dict[str, Path]:
+    try:
+        return list_flows()
+    except FlowConfigError as e:
+        raise HTTPException(status_code=500, detail=str(e)) from e
+
+
+def _resolve_flow_dir(list_flows: Callable[[], dict[str, Path]], flow_id: str) -> Path:
+    directory = _list_flows(list_flows).get(flow_id)
+    if directory is None:
+        raise HTTPException(status_code=404, detail=f"Unknown flow {flow_id!r}")
+    return directory
+
+
+def _graph_json(directory: Path) -> dict[str, Any]:
+    dataflow = load_dataflow(directory)
+
+    nodes = [
+        {
+            "id": node_id,
+            "label": node.name or node_id,
+            "type": node.type,
+            "group": "source-sink"
+            if node.type in SOURCE_SINK_TYPES
+            else "processing-node",
+            "schema": node.schema,
+            "connection": node.connection,
+            "config": node.config,
+        }
+        for node_id, node in dataflow.nodes.items()
+    ]
+    edges = [
+        {
+            "from": edge.from_,
+            "to": edge.to,
+            "kind": edge.kind,
+            "consumer": edge.consumer,
+        }
+        for edge in dataflow.edges
+    ]
+    return {"name": dataflow.name, "nodes": nodes, "edges": edges}
+
+
+def _get_node(directory: Path, node_id: str) -> NodeDef:
+    dataflow = load_dataflow(directory)
+    node = dataflow.nodes.get(node_id)
+    if node is None:
+        raise HTTPException(status_code=404, detail=f"Unknown node {node_id!r}")
+    return node
+
+
+def _require_kafka_connection(
+    node_id: str, node: NodeDef, feature: str
+) -> tuple[str, str]:
+    """Common gate for the two live-Kafka endpoints: node must be a kafka-topic with a
+    broker/topic to connect to. Returns (broker, topic) or raises HTTPException."""
+    if node.type != "kafka-topic":
+        raise HTTPException(
+            status_code=404,
+            detail=f"No {feature} available for node type {node.type!r}",
+        )
+    connection = node.connection or {}
+    broker, topic = connection.get("broker"), connection.get("topic")
+    if not broker or not topic:
+        raise HTTPException(
+            status_code=422,
+            detail=f"Node {node_id!r} is missing connection.broker/connection.topic",
+        )
+    return broker, topic
+
+
+def create_app(list_flows: Callable[[], dict[str, Path]]) -> FastAPI:
+    app = FastAPI(title="tkati-dashboard")
+
+    @app.get("/")
+    def index() -> FileResponse:
+        return FileResponse(STATIC_DIR / "index.html")
+
+    @app.get("/api/flows")
+    def flows() -> list[dict[str, str]]:
+        resolved = _list_flows(list_flows)
+        return [{"id": flow_id, "name": flow_id} for flow_id in sorted(resolved, key=str.lower)]
+
+    @app.get("/api/flows/{flow_id}/graph")
+    def graph(flow_id: str) -> dict[str, Any]:
+        directory = _resolve_flow_dir(list_flows, flow_id)
+        try:
+            return _graph_json(directory)
+        except DataflowValidationError as e:
+            raise HTTPException(status_code=422, detail=str(e)) from e
+
+    @app.get("/api/flows/{flow_id}/nodes/{node_id}/snapshot")
+    def node_snapshot(flow_id: str, node_id: str) -> dict[str, Any]:
+        directory = _resolve_flow_dir(list_flows, flow_id)
+        try:
+            node = _get_node(directory, node_id)
+        except DataflowValidationError as e:
+            raise HTTPException(status_code=422, detail=str(e)) from e
+        broker, topic = _require_kafka_connection(node_id, node, "live snapshot")
+
+        try:
+            events = snapshot.fetch_kafka_snapshot(broker, topic)
+        except snapshot.SnapshotError as e:
+            raise HTTPException(status_code=502, detail=str(e)) from e
+
+        return {"events": events}
+
+    @app.get("/api/flows/{flow_id}/nodes/{node_id}/consumer-lag")
+    def node_consumer_lag(flow_id: str, node_id: str, group_id: str) -> dict[str, Any]:
+        directory = _resolve_flow_dir(list_flows, flow_id)
+        try:
+            node = _get_node(directory, node_id)
+        except DataflowValidationError as e:
+            raise HTTPException(status_code=422, detail=str(e)) from e
+        broker, topic = _require_kafka_connection(node_id, node, "consumer lag")
+
+        try:
+            return lag.fetch_consumer_lag(broker, topic, group_id)
+        except lag.LagError as e:
+            raise HTTPException(status_code=502, detail=str(e)) from e
+
+    @app.get("/api/flows/{flow_id}/nodes/{node_id}/topic-stats")
+    def node_topic_stats(flow_id: str, node_id: str) -> dict[str, Any]:
+        directory = _resolve_flow_dir(list_flows, flow_id)
+        try:
+            node = _get_node(directory, node_id)
+        except DataflowValidationError as e:
+            raise HTTPException(status_code=422, detail=str(e)) from e
+        broker, topic = _require_kafka_connection(node_id, node, "topic stats")
+
+        try:
+            return topic_stats.fetch_topic_stats(broker, topic)
+        except topic_stats.TopicStatsError as e:
+            raise HTTPException(status_code=502, detail=str(e)) from e
+
+    return app