tkati-dashboard 0.4.0a1__tar.gz

tkati_dashboard-0.4.0a1/PKG-INFO

@@ -0,0 +1,92 @@
+Metadata-Version: 2.4
+Name: tkati-dashboard
+Version: 0.4.0a1
+Summary: Local web dashboard that renders a serialized tkati dataflow directory as a graph
+Requires-Python: >=3.13
+Description-Content-Type: text/markdown
+Requires-Dist: tkati-core==0.4.0a1
+Requires-Dist: fastapi>=0.115.0
+Requires-Dist: uvicorn>=0.34.0
+Requires-Dist: confluent-kafka>=2.11.0
+Requires-Dist: orjson>=3.9.0
+
+# tkati-dashboard — dataflow graph viewer
+
+Reads a serialized tkati dataflow directory (see
+[docs/dataflow-serialization.md](../../docs/dataflow-serialization.md)) and serves a local web page
+rendering it as a graph — no live runtime process required, no manifest to maintain, just a
+directory of `*.json` fragment files.
+
+## Usage
+
+```sh
+tkati-dashboard path/to/dataflow-dir
+```
+
+Then open `http://127.0.0.1:8000/` in a browser. The page fetches `/api/graph` and renders it
+top-to-bottom with [React Flow](https://reactflow.dev); source/sink nodes (`kafka-topic`,
+`clickhouse-table`) and processing nodes are colored differently, and stream edges are labeled by
+`kind` plus, for a Kafka consumer edge, its `group_id` and live lag. Click a node to open a side
+panel with its full connection/config/schema details.
+
+### Try it with the bundled example
+
+[`examples/simple-pipeline`](examples/simple-pipeline) is the smallest interesting dataflow: two
+Kafka topics and one processing node (`raw-events` → `dedup` → `deduped-events`). Its `connection`
+blocks point at `localhost:9092`, so if you have a broker there, seed it with sample events first:
+
+```sh
+uv run python packages/tkati-dashboard/examples/simple-pipeline/seed_kafka.py
+uv run tkati-dashboard packages/tkati-dashboard/examples/simple-pipeline
+```
+
+Then open <http://127.0.0.1:8000/> and click the `raw-events` or `deduped-events` node — the
+"Latest events" section in the side panel shows the real messages the seed script just produced
+(`raw-events` includes two intentional duplicate `event_id`s so you can see what the `dedup` node
+in between is for).
+
+For a bigger graph exercising fragment merging across `topics.json`, `tables.json`, `nodes.json`,
+and `edges.json` (two raw topics → dedup → a sessionize/enrich node → a ClickHouse table, plus a
+side branch straight to another table), see [`examples/analytics-pipeline`](examples/analytics-pipeline)
+— its topics aren't seeded with data, so "Latest events" there will error unless you produce to
+them yourself.
+
+Options:
+
+- `--host` (default `127.0.0.1`)
+- `--port` (default `8000`)
+
+The dataflow directory is re-read on every request to `/api/graph`, so editing the fragments and
+refreshing the browser picks up the change without restarting the server.
+
+## Node panel
+
+Clicking a node opens a side panel with its full metadata: connection settings, `config`, and
+`schema` (field → type). For a `kafka-topic` node, the panel also fetches
+`GET /api/nodes/{id}/snapshot`, which connects live to `connection.broker`/`connection.topic` and
+shows the most recent messages on that topic (newest last), parsed as JSON. This is one of two
+places `tkati-dashboard` talks to a live broker rather than just the serialized directory — it's a
+best-effort convenience for the panel, not something the graph view itself depends on: a broker
+that's unreachable, or a topic that doesn't exist, shows an inline error in that section instead of
+breaking the page. It uses a throwaway consumer group and never commits offsets, so it never
+interferes with a real pipeline's consumers.
+
+## Consumer lag
+
+For every stream edge whose `consumer.group_id` is set and whose source is a `kafka-topic`, the
+page also fetches `GET /api/nodes/{topic_id}/consumer-lag?group_id=...` and appends the result to
+the edge's label (e.g. `stream · group: orders-dedup · lag: 4`). This is the other place
+`tkati-dashboard` talks to a live broker: it looks up `group_id`'s committed offset with
+`Consumer.committed()` and compares it to the topic's high watermark — it never subscribes or
+polls as that group, so it can't join it, trigger a rebalance, or otherwise disturb a real
+pipeline's consumer. A group that has never committed an offset is reported as fully behind (lag
+= the topic's full size); an unreachable broker shows `lag: n/a` on that edge instead of failing
+the page.
+
+## Validation
+
+`tkati_dashboard.dataflow.load_dataflow` enforces the rules from the serialization doc: the
+directory must contain at least one `*.json` fragment, node ids must be unique (or identically
+redefined) across fragments, edges must reference existing nodes, and source/sink schemas must use
+field types known to `tkati_core.type_mapping`. A validation failure surfaces as an HTTP 422 with
+the error message, shown inline on the page instead of a blank graph.

tkati_dashboard-0.4.0a1/README.md

@@ -0,0 +1,80 @@
+# tkati-dashboard — dataflow graph viewer
+
+Reads a serialized tkati dataflow directory (see
+[docs/dataflow-serialization.md](../../docs/dataflow-serialization.md)) and serves a local web page
+rendering it as a graph — no live runtime process required, no manifest to maintain, just a
+directory of `*.json` fragment files.
+
+## Usage
+
+```sh
+tkati-dashboard path/to/dataflow-dir
+```
+
+Then open `http://127.0.0.1:8000/` in a browser. The page fetches `/api/graph` and renders it
+top-to-bottom with [React Flow](https://reactflow.dev); source/sink nodes (`kafka-topic`,
+`clickhouse-table`) and processing nodes are colored differently, and stream edges are labeled by
+`kind` plus, for a Kafka consumer edge, its `group_id` and live lag. Click a node to open a side
+panel with its full connection/config/schema details.
+
+### Try it with the bundled example
+
+[`examples/simple-pipeline`](examples/simple-pipeline) is the smallest interesting dataflow: two
+Kafka topics and one processing node (`raw-events` → `dedup` → `deduped-events`). Its `connection`
+blocks point at `localhost:9092`, so if you have a broker there, seed it with sample events first:
+
+```sh
+uv run python packages/tkati-dashboard/examples/simple-pipeline/seed_kafka.py
+uv run tkati-dashboard packages/tkati-dashboard/examples/simple-pipeline
+```
+
+Then open <http://127.0.0.1:8000/> and click the `raw-events` or `deduped-events` node — the
+"Latest events" section in the side panel shows the real messages the seed script just produced
+(`raw-events` includes two intentional duplicate `event_id`s so you can see what the `dedup` node
+in between is for).
+
+For a bigger graph exercising fragment merging across `topics.json`, `tables.json`, `nodes.json`,
+and `edges.json` (two raw topics → dedup → a sessionize/enrich node → a ClickHouse table, plus a
+side branch straight to another table), see [`examples/analytics-pipeline`](examples/analytics-pipeline)
+— its topics aren't seeded with data, so "Latest events" there will error unless you produce to
+them yourself.
+
+Options:
+
+- `--host` (default `127.0.0.1`)
+- `--port` (default `8000`)
+
+The dataflow directory is re-read on every request to `/api/graph`, so editing the fragments and
+refreshing the browser picks up the change without restarting the server.
+
+## Node panel
+
+Clicking a node opens a side panel with its full metadata: connection settings, `config`, and
+`schema` (field → type). For a `kafka-topic` node, the panel also fetches
+`GET /api/nodes/{id}/snapshot`, which connects live to `connection.broker`/`connection.topic` and
+shows the most recent messages on that topic (newest last), parsed as JSON. This is one of two
+places `tkati-dashboard` talks to a live broker rather than just the serialized directory — it's a
+best-effort convenience for the panel, not something the graph view itself depends on: a broker
+that's unreachable, or a topic that doesn't exist, shows an inline error in that section instead of
+breaking the page. It uses a throwaway consumer group and never commits offsets, so it never
+interferes with a real pipeline's consumers.
+
+## Consumer lag
+
+For every stream edge whose `consumer.group_id` is set and whose source is a `kafka-topic`, the
+page also fetches `GET /api/nodes/{topic_id}/consumer-lag?group_id=...` and appends the result to
+the edge's label (e.g. `stream · group: orders-dedup · lag: 4`). This is the other place
+`tkati-dashboard` talks to a live broker: it looks up `group_id`'s committed offset with
+`Consumer.committed()` and compares it to the topic's high watermark — it never subscribes or
+polls as that group, so it can't join it, trigger a rebalance, or otherwise disturb a real
+pipeline's consumer. A group that has never committed an offset is reported as fully behind (lag
+= the topic's full size); an unreachable broker shows `lag: n/a` on that edge instead of failing
+the page.
+
+## Validation
+
+`tkati_dashboard.dataflow.load_dataflow` enforces the rules from the serialization doc: the
+directory must contain at least one `*.json` fragment, node ids must be unique (or identically
+redefined) across fragments, edges must reference existing nodes, and source/sink schemas must use
+field types known to `tkati_core.type_mapping`. A validation failure surfaces as an HTTP 422 with
+the error message, shown inline on the page instead of a blank graph.

tkati_dashboard-0.4.0a1/pyproject.toml

@@ -0,0 +1,37 @@
+[project]
+name = "tkati-dashboard"
+version = "0.4.0a1"
+description = "Local web dashboard that renders a serialized tkati dataflow directory as a graph"
+readme = "README.md"
+requires-python = ">=3.13"
+dependencies = [
+    "tkati-core==0.4.0a1",
+    "fastapi>=0.115.0",
+    "uvicorn>=0.34.0",
+    "confluent-kafka>=2.11.0",
+    "orjson>=3.9.0",
+]
+
+[project.scripts]
+tkati-dashboard = "tkati_dashboard.main:main"
+
+[dependency-groups]
+dev = ["pytest>=9.0.1", "httpx>=0.28.0"]
+
+[build-system]
+requires = ["setuptools", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["tkati_dashboard*"]
+
+[tool.setuptools.package-data]
+tkati_dashboard = ["static/*.html"]
+
+[tool.uv]
+package = true
+
+[tool.uv-workspace-codegen]
+generate = true
+template_type = ["test", "publish"]

tkati_dashboard-0.4.0a1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

tkati_dashboard-0.4.0a1/src/tkati_dashboard/__init__.py

@@ -0,0 +1,9 @@
+from tkati_dashboard.app import create_app
+from tkati_dashboard.dataflow import Dataflow, DataflowValidationError, load_dataflow
+
+__all__ = [
+    "Dataflow",
+    "DataflowValidationError",
+    "create_app",
+    "load_dataflow",
+]

tkati_dashboard-0.4.0a1/src/tkati_dashboard/__main__.py

@@ -0,0 +1,4 @@
+from .main import main
+
+if __name__ == "__main__":
+    main()

tkati_dashboard-0.4.0a1/src/tkati_dashboard/_kafka_metadata.py

@@ -0,0 +1,21 @@
+"""Shared "resolve a live topic's partitions" helper for snapshot.py and lag.py."""
+
+from confluent_kafka import Consumer
+
+
+def resolve_partitions(
+    consumer: Consumer, broker: str, topic: str, timeout_sec: float
+) -> list[int]:
+    """Return the partition ids of `topic`, or raise RuntimeError with a message naming
+    `broker`/`topic` if the broker is unreachable or the topic doesn't exist.
+    """
+    try:
+        metadata = consumer.list_topics(topic, timeout=timeout_sec)
+    except Exception as e:
+        raise RuntimeError(f"Could not reach broker {broker!r}: {e}") from e
+
+    topic_metadata = metadata.topics.get(topic)
+    if topic_metadata is None or topic_metadata.error is not None:
+        raise RuntimeError(f"Topic {topic!r} not found on {broker!r}")
+
+    return list(topic_metadata.partitions)

tkati_dashboard-0.4.0a1/src/tkati_dashboard/app.py

@@ -0,0 +1,117 @@
+from pathlib import Path
+from typing import Any
+
+from fastapi import FastAPI, HTTPException
+from fastapi.responses import FileResponse
+
+from tkati_dashboard import lag, snapshot
+from tkati_dashboard.dataflow import (
+    SOURCE_SINK_TYPES,
+    DataflowValidationError,
+    NodeDef,
+    load_dataflow,
+)
+
+STATIC_DIR = Path(__file__).parent / "static"
+
+
+def _graph_json(directory: Path) -> dict[str, Any]:
+    dataflow = load_dataflow(directory)
+
+    nodes = [
+        {
+            "id": node_id,
+            "label": node.name or node_id,
+            "type": node.type,
+            "group": "source-sink"
+            if node.type in SOURCE_SINK_TYPES
+            else "processing-node",
+            "schema": node.schema,
+            "connection": node.connection,
+            "config": node.config,
+        }
+        for node_id, node in dataflow.nodes.items()
+    ]
+    edges = [
+        {
+            "from": edge.from_,
+            "to": edge.to,
+            "kind": edge.kind,
+            "consumer": edge.consumer,
+        }
+        for edge in dataflow.edges
+    ]
+    return {"name": dataflow.name, "nodes": nodes, "edges": edges}
+
+
+def _get_node(directory: Path, node_id: str) -> NodeDef:
+    dataflow = load_dataflow(directory)
+    node = dataflow.nodes.get(node_id)
+    if node is None:
+        raise HTTPException(status_code=404, detail=f"Unknown node {node_id!r}")
+    return node
+
+
+def _require_kafka_connection(
+    node_id: str, node: NodeDef, feature: str
+) -> tuple[str, str]:
+    """Common gate for the two live-Kafka endpoints: node must be a kafka-topic with a
+    broker/topic to connect to. Returns (broker, topic) or raises HTTPException."""
+    if node.type != "kafka-topic":
+        raise HTTPException(
+            status_code=404,
+            detail=f"No {feature} available for node type {node.type!r}",
+        )
+    connection = node.connection or {}
+    broker, topic = connection.get("broker"), connection.get("topic")
+    if not broker or not topic:
+        raise HTTPException(
+            status_code=422,
+            detail=f"Node {node_id!r} is missing connection.broker/connection.topic",
+        )
+    return broker, topic
+
+
+def create_app(dataflow_dir: Path) -> FastAPI:
+    app = FastAPI(title="tkati-dashboard")
+
+    @app.get("/")
+    def index() -> FileResponse:
+        return FileResponse(STATIC_DIR / "index.html")
+
+    @app.get("/api/graph")
+    def graph() -> dict[str, Any]:
+        try:
+            return _graph_json(dataflow_dir)
+        except DataflowValidationError as e:
+            raise HTTPException(status_code=422, detail=str(e)) from e
+
+    @app.get("/api/nodes/{node_id}/snapshot")
+    def node_snapshot(node_id: str) -> dict[str, Any]:
+        try:
+            node = _get_node(dataflow_dir, node_id)
+        except DataflowValidationError as e:
+            raise HTTPException(status_code=422, detail=str(e)) from e
+        broker, topic = _require_kafka_connection(node_id, node, "live snapshot")
+
+        try:
+            events = snapshot.fetch_kafka_snapshot(broker, topic)
+        except snapshot.SnapshotError as e:
+            raise HTTPException(status_code=502, detail=str(e)) from e
+
+        return {"events": events}
+
+    @app.get("/api/nodes/{node_id}/consumer-lag")
+    def node_consumer_lag(node_id: str, group_id: str) -> dict[str, Any]:
+        try:
+            node = _get_node(dataflow_dir, node_id)
+        except DataflowValidationError as e:
+            raise HTTPException(status_code=422, detail=str(e)) from e
+        broker, topic = _require_kafka_connection(node_id, node, "consumer lag")
+
+        try:
+            return lag.fetch_consumer_lag(broker, topic, group_id)
+        except lag.LagError as e:
+            raise HTTPException(status_code=502, detail=str(e)) from e
+
+    return app

tkati_dashboard-0.4.0a1/src/tkati_dashboard/dataflow.py

@@ -0,0 +1,121 @@
+"""Load and validate a serialized tkati dataflow directory.
+
+See docs/dataflow-serialization.md for the format this module implements: a directory of JSON
+fragments merged into one graph of nodes and edges. There is no manifest file — every `*.json`
+file directly inside the directory is a fragment.
+"""
+
+import json
+from pathlib import Path
+from typing import Any
+
+from pydantic import BaseModel, ConfigDict, Field
+from tkati_core.type_mapping import TYPE_MAPPING
+
+# Node types that represent data at rest (as opposed to a processing step) and therefore require
+# a `schema`. Kept as a heuristic, not a closed registry: an unrecognized type is still accepted,
+# it just isn't schema-checked.
+SOURCE_SINK_TYPES = {"kafka-topic", "clickhouse-table"}
+
+
+class DataflowValidationError(ValueError):
+    """A serialized dataflow directory failed validation."""
+
+
+class NodeDef(BaseModel):
+    model_config = ConfigDict(extra="allow")
+
+    type: str
+    name: str | None = None
+    schema: dict[str, str] | None = None
+    connection: dict[str, Any] | None = None
+    config: dict[str, Any] | None = None
+
+
+class EdgeDef(BaseModel):
+    model_config = ConfigDict(populate_by_name=True)
+
+    from_: str = Field(alias="from")
+    to: str
+    kind: str = "stream"
+    consumer: dict[str, Any] | None = None
+
+
+class Dataflow(BaseModel):
+    name: str
+    nodes: dict[str, NodeDef]
+    edges: list[EdgeDef]
+
+
+def _read_json(path: Path) -> dict[str, Any]:
+    try:
+        return json.loads(path.read_text())
+    except FileNotFoundError as e:
+        raise DataflowValidationError(f"Missing dataflow file: {path}") from e
+    except json.JSONDecodeError as e:
+        raise DataflowValidationError(f"Invalid JSON in {path}: {e}") from e
+
+
+def _validate_node(node_id: str, node: NodeDef) -> None:
+    if node.type in SOURCE_SINK_TYPES:
+        if node.schema is None:
+            raise DataflowValidationError(
+                f"Node {node_id!r} of type {node.type!r} needs a schema"
+            )
+        for field_name, field_type in node.schema.items():
+            if field_type not in TYPE_MAPPING:
+                raise DataflowValidationError(
+                    f"Node {node_id!r} field {field_name!r} has unknown schema type {field_type!r}"
+                )
+
+
+def load_dataflow(directory: Path) -> Dataflow:
+    """Read every `*.json` fragment directly inside `directory`, merge, and validate them.
+
+    There is no manifest: any JSON file in the directory is a fragment contributing to the
+    graph. The dataflow's name is the directory's own name.
+    """
+    if not directory.is_dir():
+        raise DataflowValidationError(f"Not a directory: {directory}")
+
+    fragment_paths = sorted(directory.glob("*.json"))
+    if not fragment_paths:
+        raise DataflowValidationError(
+            f"No dataflow fragments (*.json) found in {directory}"
+        )
+
+    nodes: dict[str, NodeDef] = {}
+    node_sources: dict[
+        str, str
+    ] = {}  # node id -> fragment it was first seen in, for error messages
+    edges: list[EdgeDef] = []
+
+    for fragment_path in fragment_paths:
+        fragment = _read_json(fragment_path)
+
+        for node_id, raw_node in fragment.get("nodes", {}).items():
+            node = NodeDef.model_validate(raw_node)
+            if node_id in nodes:
+                if nodes[node_id] != node:
+                    raise DataflowValidationError(
+                        f"Node {node_id!r} is defined differently in "
+                        f"{node_sources[node_id]!r} and {fragment_path.name!r}"
+                    )
+                continue
+            nodes[node_id] = node
+            node_sources[node_id] = fragment_path.name
+
+        for raw_edge in fragment.get("edges", []):
+            edges.append(EdgeDef.model_validate(raw_edge))
+
+    for node_id, node in nodes.items():
+        _validate_node(node_id, node)
+
+    for edge in edges:
+        for endpoint in (edge.from_, edge.to):
+            if endpoint not in nodes:
+                raise DataflowValidationError(
+                    f"Edge {edge.from_!r} -> {edge.to!r} references unknown node {endpoint!r}"
+                )
+
+    return Dataflow(name=directory.name, nodes=nodes, edges=edges)

tkati_dashboard-0.4.0a1/src/tkati_dashboard/lag.py

@@ -0,0 +1,78 @@
+"""Live consumer-group lag lookup for a dataflow edge, for the dashboard's edge labels.
+
+Like tkati_dashboard.snapshot, this is a best-effort, on-demand look at a live broker — it never
+subscribes or polls as the consumer group, only reads its committed offsets via `.committed()`,
+so it can never join the group, trigger a rebalance, or otherwise disrupt a real pipeline.
+"""
+
+from typing import Any
+
+from confluent_kafka import Consumer, TopicPartition
+
+from tkati_dashboard._kafka_metadata import resolve_partitions
+
+DEFAULT_TIMEOUT_SEC = 5.0
+
+
+class LagError(RuntimeError):
+    """Consumer lag could not be computed."""
+
+
+def fetch_consumer_lag(
+    broker: str,
+    topic: str,
+    group_id: str,
+    timeout_sec: float = DEFAULT_TIMEOUT_SEC,
+) -> dict[str, Any]:
+    """Return per-partition and total lag of `group_id` on `topic`.
+
+    A partition `group_id` has never committed an offset on counts as fully behind (lag =
+    partition size), since that's the backlog the group would have to process from scratch.
+    """
+    consumer = Consumer(
+        {
+            "bootstrap.servers": broker,
+            "group.id": group_id,
+            "enable.auto.commit": False,
+        }
+    )
+    try:
+        try:
+            partition_ids = resolve_partitions(consumer, broker, topic, timeout_sec)
+        except RuntimeError as e:
+            raise LagError(str(e)) from e
+
+        if not partition_ids:
+            return {"total_lag": 0, "partitions": []}
+
+        try:
+            committed = consumer.committed(
+                [TopicPartition(topic, pid) for pid in partition_ids],
+                timeout=timeout_sec,
+            )
+        except Exception as e:
+            raise LagError(
+                f"Could not fetch committed offsets for group {group_id!r}: {e}"
+            ) from e
+
+        partitions = []
+        total_lag = 0
+        for tp in committed:
+            low, high = consumer.get_watermark_offsets(
+                TopicPartition(topic, tp.partition), timeout=timeout_sec, cached=False
+            )
+            has_committed = tp.offset is not None and tp.offset >= 0
+            current = tp.offset if has_committed else low
+            partition_lag = max(high - current, 0)
+            total_lag += partition_lag
+            partitions.append(
+                {
+                    "partition": tp.partition,
+                    "committed_offset": tp.offset if has_committed else None,
+                    "high_watermark": high,
+                    "lag": partition_lag,
+                }
+            )
+        return {"total_lag": total_lag, "partitions": partitions}
+    finally:
+        consumer.close()

tkati_dashboard-0.4.0a1/src/tkati_dashboard/main.py

@@ -0,0 +1,29 @@
+import argparse
+from pathlib import Path
+
+import uvicorn
+
+from tkati_dashboard.app import create_app
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        prog="tkati-dashboard",
+        description="Serve a graph view of a serialized tkati dataflow directory",
+    )
+    parser.add_argument(
+        "dataflow_dir",
+        type=Path,
+        help="Path to the dataflow directory (a directory of *.json fragments)",
+    )
+    parser.add_argument("--host", default="127.0.0.1")
+    parser.add_argument("--port", type=int, default=8000)
+    args = parser.parse_args()
+
+    if not args.dataflow_dir.is_dir():
+        parser.error(f"{args.dataflow_dir} is not a directory")
+    if not any(args.dataflow_dir.glob("*.json")):
+        parser.error(f"{args.dataflow_dir} contains no dataflow fragments (*.json)")
+
+    app = create_app(args.dataflow_dir)
+    uvicorn.run(app, host=args.host, port=args.port)