tkati-node-el 0.3.0a3__tar.gz

tkati_node_el-0.3.0a3/PKG-INFO

@@ -0,0 +1,116 @@
+Metadata-Version: 2.4
+Name: tkati-node-el
+Version: 0.3.0a3
+Summary: Generic extract/load node: reads from a configurable input and writes to a configurable output
+Requires-Python: >=3.13
+Description-Content-Type: text/markdown
+Requires-Dist: tkati-core==0.3.0-alpha.3
+Requires-Dist: loguru>=0.7.0
+Requires-Dist: pydantic-settings>=2.11.0
+
+# tkati-node-el — generic extract/load node
+
+Reads batches from a configurable input and writes them to a configurable output. Offsets are committed only after a successful write (at-least-once delivery).
+
+Input and output kinds are selected via the `type` field in each section — pick from whatever `tkati-core` supports. Every backend's settings split a **`connection`** tier (server-specific: how to reach the broker/database) from the resource tier (`topic` for Kafka, `table` for ClickHouse) and, where relevant, a tier local to this reader/writer instance (Kafka's `consumer` settings).
+
+- **Input**: `"kafka"` (JSON or Arrow-batch messages from a Kafka/Redpanda topic)
+- **Output**: `"kafka"` or `"clickhouse"` (native Arrow insert)
+- **DLQ**: same `OutputSettings` shape as `output` — a DLQ can be Kafka or ClickHouse too
+
+## Configuration
+
+Settings are loaded from a TOML file. Set the `SETTINGS_FILE` environment variable to point to it (defaults to `settings.toml`).
+
+```toml
+[input]
+type = "kafka"
+
+[input.connection]
+broker = "redpanda:29092"
+
+[input.topic]
+name   = "traffic_event"
+
+[input.topic.schema]
+uid        = "string"
+time       = "timestamp[ms]"
+traffic_in = "uint32"
+# … other columns
+
+[input.consumer]
+group_id          = "node-el-group"
+batch_size        = 1000
+batch_timeout_sec = 10
+auto_offset_reset = "latest"
+
+[output]
+type             = "clickhouse"
+dlq_split_factor = 10
+
+[output.connection]
+host     = "clickhouse"
+port     = 9000
+user     = "default"
+password = ""
+secure   = false
+
+[output.table]
+database = "default"
+name     = "traffic_event"
+
+[dlq]
+type = "kafka"
+
+[dlq.connection]
+broker = "redpanda:29092"
+
+[dlq.topic]
+name = "node-el-dlq"
+```
+
+A Kafka output instead looks like:
+
+```toml
+[output]
+type = "kafka"
+
+[output.connection]
+broker = "redpanda:29092"
+
+[output.topic]
+name   = "some-other-topic"
+format = "json"        # or "arrow-batch"
+key_column = "uid"     # optional
+```
+
+A ClickHouse DLQ instead looks like:
+
+```toml
+[dlq]
+type = "clickhouse"
+
+[dlq.connection]
+host     = "clickhouse"
+port     = 9000
+user     = "default"
+password = ""
+secure   = false
+
+[dlq.table]
+database = "default"
+name     = "traffic_event_dlq"
+```
+
+## DLQ semantics
+
+DLQ *fallback triggering* is currently only implemented for the `clickhouse` output kind — `KafkaProducer` has no retry/split logic of its own. The DLQ *sink* itself (where isolated bad rows end up) can be Kafka or ClickHouse, independent of the primary output. When a batch insert fails after all retries, the app switches to a recursive fallback to isolate the problematic rows:
+
+1. The failing batch is split into `dlq_split_factor` equal sub-batches and each is retried independently.
+2. If a sub-batch also fails it is split again — this repeats until individual rows are reached.
+3. A single row that ClickHouse still rejects is written to the DLQ sink, preserving the full schema (Arrow IPC `arrow-batch` format for a Kafka DLQ).
+4. After all rows are handled (inserted or DLQ'd), the input offset is committed and the app resumes normal large-batch processing.
+
+`dlq_split_factor` is a setting on the `clickhouse` `[output]` block (see above), not on `[dlq]` — it describes how the primary output retries, independent of where the DLQ sink sends isolated rows. With `dlq_split_factor=10` and a 1 000-row batch this takes at most 3 recursive levels (1000 → 100 → 10 → 1).
+
+**Delivery guarantee: at-least-once.** If the process crashes mid-recursion the uncommitted batch is re-read on restart and re-processed from the beginning, which may produce duplicate rows in the output and duplicate messages in the DLQ.

tkati_node_el-0.3.0a3/README.md

@@ -0,0 +1,106 @@
+# tkati-node-el — generic extract/load node
+
+Reads batches from a configurable input and writes them to a configurable output. Offsets are committed only after a successful write (at-least-once delivery).
+
+Input and output kinds are selected via the `type` field in each section — pick from whatever `tkati-core` supports. Every backend's settings split a **`connection`** tier (server-specific: how to reach the broker/database) from the resource tier (`topic` for Kafka, `table` for ClickHouse) and, where relevant, a tier local to this reader/writer instance (Kafka's `consumer` settings).
+
+- **Input**: `"kafka"` (JSON or Arrow-batch messages from a Kafka/Redpanda topic)
+- **Output**: `"kafka"` or `"clickhouse"` (native Arrow insert)
+- **DLQ**: same `OutputSettings` shape as `output` — a DLQ can be Kafka or ClickHouse too
+
+## Configuration
+
+Settings are loaded from a TOML file. Set the `SETTINGS_FILE` environment variable to point to it (defaults to `settings.toml`).
+
+```toml
+[input]
+type = "kafka"
+
+[input.connection]
+broker = "redpanda:29092"
+
+[input.topic]
+name   = "traffic_event"
+
+[input.topic.schema]
+uid        = "string"
+time       = "timestamp[ms]"
+traffic_in = "uint32"
+# … other columns
+
+[input.consumer]
+group_id          = "node-el-group"
+batch_size        = 1000
+batch_timeout_sec = 10
+auto_offset_reset = "latest"
+
+[output]
+type             = "clickhouse"
+dlq_split_factor = 10
+
+[output.connection]
+host     = "clickhouse"
+port     = 9000
+user     = "default"
+password = ""
+secure   = false
+
+[output.table]
+database = "default"
+name     = "traffic_event"
+
+[dlq]
+type = "kafka"
+
+[dlq.connection]
+broker = "redpanda:29092"
+
+[dlq.topic]
+name = "node-el-dlq"
+```
+
+A Kafka output instead looks like:
+
+```toml
+[output]
+type = "kafka"
+
+[output.connection]
+broker = "redpanda:29092"
+
+[output.topic]
+name   = "some-other-topic"
+format = "json"        # or "arrow-batch"
+key_column = "uid"     # optional
+```
+
+A ClickHouse DLQ instead looks like:
+
+```toml
+[dlq]
+type = "clickhouse"
+
+[dlq.connection]
+host     = "clickhouse"
+port     = 9000
+user     = "default"
+password = ""
+secure   = false
+
+[dlq.table]
+database = "default"
+name     = "traffic_event_dlq"
+```
+
+## DLQ semantics
+
+DLQ *fallback triggering* is currently only implemented for the `clickhouse` output kind — `KafkaProducer` has no retry/split logic of its own. The DLQ *sink* itself (where isolated bad rows end up) can be Kafka or ClickHouse, independent of the primary output. When a batch insert fails after all retries, the app switches to a recursive fallback to isolate the problematic rows:
+
+1. The failing batch is split into `dlq_split_factor` equal sub-batches and each is retried independently.
+2. If a sub-batch also fails it is split again — this repeats until individual rows are reached.
+3. A single row that ClickHouse still rejects is written to the DLQ sink, preserving the full schema (Arrow IPC `arrow-batch` format for a Kafka DLQ).
+4. After all rows are handled (inserted or DLQ'd), the input offset is committed and the app resumes normal large-batch processing.
+
+`dlq_split_factor` is a setting on the `clickhouse` `[output]` block (see above), not on `[dlq]` — it describes how the primary output retries, independent of where the DLQ sink sends isolated rows. With `dlq_split_factor=10` and a 1 000-row batch this takes at most 3 recursive levels (1000 → 100 → 10 → 1).
+
+**Delivery guarantee: at-least-once.** If the process crashes mid-recursion the uncommitted batch is re-read on restart and re-processed from the beginning, which may produce duplicate rows in the output and duplicate messages in the DLQ.

tkati_node_el-0.3.0a3/pyproject.toml

@@ -0,0 +1,32 @@
+[project]
+name = "tkati-node-el"
+version = "0.3.0-alpha.3"
+description = "Generic extract/load node: reads from a configurable input and writes to a configurable output"
+readme = "README.md"
+requires-python = ">=3.13"
+dependencies = [
+    "tkati-core==0.3.0-alpha.3",
+    "loguru>=0.7.0",
+    "pydantic-settings>=2.11.0",
+]
+
+[project.scripts]
+tkati-node-el = "tkati_node_el.main:main"
+
+[dependency-groups]
+dev = ["pytest>=9.0.1", "confluent-kafka>=2.11.0", "clickhouse-connect>=0.11.0"]
+
+[build-system]
+requires = ["setuptools", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["tkati_node_el*"]
+
+[tool.uv]
+package = true
+
+[tool.uv-workspace-codegen]
+generate = true
+template_type = ["test", "publish"]

tkati_node_el-0.3.0a3/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

tkati_node_el-0.3.0a3/src/tkati_node_el/__main__.py

@@ -0,0 +1,4 @@
+from .main import main
+
+if __name__ == "__main__":
+    main()

tkati_node_el-0.3.0a3/src/tkati_node_el/main.py

@@ -0,0 +1,40 @@
+from loguru import logger
+from tkati_core import Consumer, Producer, build_consumer, build_producer
+
+from tkati_node_el.settings import AppSettings
+
+
+def run_one_iteration(
+    consumer: Consumer,
+    producer: Producer,
+    settings: AppSettings,
+) -> None:
+    batch = consumer.read_arrow(
+        num_messages=settings.input.consumer.batch_size,
+        timeout=settings.input.consumer.batch_timeout_sec,
+    )
+    if batch is None:
+        return
+    producer.produce_arrow(batch)
+    consumer.commit()
+    logger.info(f"Produced {len(batch)} rows")
+
+
+def main() -> None:
+    settings = AppSettings()  # type: ignore
+
+    consumer = build_consumer(settings.input)
+
+    dlq_producer: Producer | None = None
+    if settings.dlq is not None:
+        dlq_producer = build_producer(settings.dlq)
+
+    producer = build_producer(settings.output, dlq_producer=dlq_producer)
+
+    try:
+        while True:
+            run_one_iteration(consumer, producer, settings)
+    finally:
+        consumer.close()
+        if dlq_producer is not None:
+            dlq_producer.close()

tkati_node_el-0.3.0a3/src/tkati_node_el/settings.py

@@ -0,0 +1,8 @@
+from tkati_core.settings import TomlBaseSettings
+from tkati_core.settings import InputSettings, OutputSettings
+
+
+class AppSettings(TomlBaseSettings):
+    input: InputSettings
+    output: OutputSettings
+    dlq: OutputSettings | None = None

tkati_node_el-0.3.0a3/src/tkati_node_el.egg-info/PKG-INFO

@@ -0,0 +1,116 @@
+Metadata-Version: 2.4
+Name: tkati-node-el
+Version: 0.3.0a3
+Summary: Generic extract/load node: reads from a configurable input and writes to a configurable output
+Requires-Python: >=3.13
+Description-Content-Type: text/markdown
+Requires-Dist: tkati-core==0.3.0-alpha.3
+Requires-Dist: loguru>=0.7.0
+Requires-Dist: pydantic-settings>=2.11.0
+
+# tkati-node-el — generic extract/load node
+
+Reads batches from a configurable input and writes them to a configurable output. Offsets are committed only after a successful write (at-least-once delivery).
+
+Input and output kinds are selected via the `type` field in each section — pick from whatever `tkati-core` supports. Every backend's settings split a **`connection`** tier (server-specific: how to reach the broker/database) from the resource tier (`topic` for Kafka, `table` for ClickHouse) and, where relevant, a tier local to this reader/writer instance (Kafka's `consumer` settings).
+
+- **Input**: `"kafka"` (JSON or Arrow-batch messages from a Kafka/Redpanda topic)
+- **Output**: `"kafka"` or `"clickhouse"` (native Arrow insert)
+- **DLQ**: same `OutputSettings` shape as `output` — a DLQ can be Kafka or ClickHouse too
+
+## Configuration
+
+Settings are loaded from a TOML file. Set the `SETTINGS_FILE` environment variable to point to it (defaults to `settings.toml`).
+
+```toml
+[input]
+type = "kafka"
+
+[input.connection]
+broker = "redpanda:29092"
+
+[input.topic]
+name   = "traffic_event"
+
+[input.topic.schema]
+uid        = "string"
+time       = "timestamp[ms]"
+traffic_in = "uint32"
+# … other columns
+
+[input.consumer]
+group_id          = "node-el-group"
+batch_size        = 1000
+batch_timeout_sec = 10
+auto_offset_reset = "latest"
+
+[output]
+type             = "clickhouse"
+dlq_split_factor = 10
+
+[output.connection]
+host     = "clickhouse"
+port     = 9000
+user     = "default"
+password = ""
+secure   = false
+
+[output.table]
+database = "default"
+name     = "traffic_event"
+
+[dlq]
+type = "kafka"
+
+[dlq.connection]
+broker = "redpanda:29092"
+
+[dlq.topic]
+name = "node-el-dlq"
+```
+
+A Kafka output instead looks like:
+
+```toml
+[output]
+type = "kafka"
+
+[output.connection]
+broker = "redpanda:29092"
+
+[output.topic]
+name   = "some-other-topic"
+format = "json"        # or "arrow-batch"
+key_column = "uid"     # optional
+```
+
+A ClickHouse DLQ instead looks like:
+
+```toml
+[dlq]
+type = "clickhouse"
+
+[dlq.connection]
+host     = "clickhouse"
+port     = 9000
+user     = "default"
+password = ""
+secure   = false
+
+[dlq.table]
+database = "default"
+name     = "traffic_event_dlq"
+```
+
+## DLQ semantics
+
+DLQ *fallback triggering* is currently only implemented for the `clickhouse` output kind — `KafkaProducer` has no retry/split logic of its own. The DLQ *sink* itself (where isolated bad rows end up) can be Kafka or ClickHouse, independent of the primary output. When a batch insert fails after all retries, the app switches to a recursive fallback to isolate the problematic rows:
+
+1. The failing batch is split into `dlq_split_factor` equal sub-batches and each is retried independently.
+2. If a sub-batch also fails it is split again — this repeats until individual rows are reached.
+3. A single row that ClickHouse still rejects is written to the DLQ sink, preserving the full schema (Arrow IPC `arrow-batch` format for a Kafka DLQ).
+4. After all rows are handled (inserted or DLQ'd), the input offset is committed and the app resumes normal large-batch processing.
+
+`dlq_split_factor` is a setting on the `clickhouse` `[output]` block (see above), not on `[dlq]` — it describes how the primary output retries, independent of where the DLQ sink sends isolated rows. With `dlq_split_factor=10` and a 1 000-row batch this takes at most 3 recursive levels (1000 → 100 → 10 → 1).
+
+**Delivery guarantee: at-least-once.** If the process crashes mid-recursion the uncommitted batch is re-read on restart and re-processed from the beginning, which may produce duplicate rows in the output and duplicate messages in the DLQ.

tkati_node_el-0.3.0a3/src/tkati_node_el.egg-info/SOURCES.txt

@@ -0,0 +1,14 @@
+README.md
+pyproject.toml
+src/tkati_node_el/__init__.py
+src/tkati_node_el/__main__.py
+src/tkati_node_el/main.py
+src/tkati_node_el/py.typed
+src/tkati_node_el/settings.py
+src/tkati_node_el.egg-info/PKG-INFO
+src/tkati_node_el.egg-info/SOURCES.txt
+src/tkati_node_el.egg-info/dependency_links.txt
+src/tkati_node_el.egg-info/entry_points.txt
+src/tkati_node_el.egg-info/requires.txt
+src/tkati_node_el.egg-info/top_level.txt
+tests/test_node_el.py

tkati_node_el-0.3.0a3/src/tkati_node_el.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

tkati_node_el-0.3.0a3/src/tkati_node_el.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+tkati-node-el = tkati_node_el.main:main

tkati_node_el-0.3.0a3/src/tkati_node_el.egg-info/requires.txt

@@ -0,0 +1,3 @@
+tkati-core==0.3.0-alpha.3
+loguru>=0.7.0
+pydantic-settings>=2.11.0

tkati_node_el-0.3.0a3/src/tkati_node_el.egg-info/top_level.txt

@@ -0,0 +1 @@
+tkati_node_el

tkati_node_el-0.3.0a3/tests/test_node_el.py

@@ -0,0 +1,103 @@
+import time
+from unittest.mock import MagicMock
+
+import clickhouse_connect.driver as ch_driver
+import orjson
+import pytest
+from confluent_kafka import Producer
+from tkati_core.clickhouse.producer import ClickhouseProducer
+from tkati_core.clickhouse.settings import ClickHouseOutputSettings
+from tkati_core.kafka.consumer import KafkaConsumer
+from tkati_node_el.main import run_one_iteration
+from tkati_node_el.settings import AppSettings
+
+
+def _make_consumer(test_settings: AppSettings) -> KafkaConsumer:
+    return KafkaConsumer(
+        kafka_config={
+            "bootstrap.servers": test_settings.input.connection.broker,
+            "group.id": test_settings.input.consumer.group_id,
+            "auto.offset.reset": "earliest",
+            "enable.auto.commit": False,
+        },
+        topic_name=test_settings.input.topic.name,
+        input_schema=test_settings.input.topic.schema,
+    )
+
+
+def test_node_el_valid_flow(
+    kafka_producer_and_topic: Producer,
+    ch_client: ch_driver.Client,
+    ch_table: str,
+    mock_dlq_producer: MagicMock,
+    test_settings: AppSettings,
+) -> None:
+    """Produce a valid event to Kafka, run one iteration, verify the row lands in ClickHouse."""
+    event = {
+        "uid": "abc123",
+        "time": int(time.time() * 1000),
+        "package_id": 1,
+        "user_hash": "uhash",
+        "sdk_hash": "shash",
+        "conn_type": "https",
+        "country": "US",
+        "local_ip": "10.0.0.1",
+        "frontend_ip": "1.2.3.4",
+        "dest_addr": "8.8.8.8",
+        "client_ip": "192.168.1.1",
+        "traffic_in": 100,
+        "traffic_out": 200,
+    }
+
+    kafka_producer_and_topic.produce(
+        test_settings.input.topic.name, value=orjson.dumps(event)
+    )
+    kafka_producer_and_topic.flush()
+
+    assert isinstance(test_settings.output, ClickHouseOutputSettings)
+
+    consumer = _make_consumer(test_settings)
+    ch_producer = ClickhouseProducer(
+        ch_client=ch_client,
+        table=ch_table,
+        dlq_producer=mock_dlq_producer,
+    )
+    try:
+        run_one_iteration(consumer, ch_producer, test_settings)
+    finally:
+        consumer.close()
+
+    result = ch_client.query(f"SELECT uid, traffic_in, traffic_out FROM {ch_table}")
+    assert result.result_rows == [("abc123", 100, 200)]
+    mock_dlq_producer.produce_arrow.assert_not_called()
+
+
+def test_node_el_malformed_data(
+    kafka_producer_and_topic: Producer,
+    ch_client: ch_driver.Client,
+    ch_table: str,
+    mock_dlq_producer: MagicMock,
+    test_settings: AppSettings,
+) -> None:
+    """Produce malformed JSON to Kafka; run_one_iteration must raise with 'JSON parse error'."""
+    kafka_producer_and_topic.produce(
+        test_settings.input.topic.name, value=b"not a json object"
+    )
+    kafka_producer_and_topic.flush()
+
+    assert isinstance(test_settings.output, ClickHouseOutputSettings)
+
+    consumer = _make_consumer(test_settings)
+    ch_producer = ClickhouseProducer(
+        ch_client=ch_client,
+        table=ch_table,
+        dlq_producer=mock_dlq_producer,
+    )
+    try:
+        with pytest.raises(Exception, match="JSON parse error"):
+            run_one_iteration(consumer, ch_producer, test_settings)
+    finally:
+        consumer.close()
+
+    result = ch_client.query(f"SELECT count() FROM {ch_table}")
+    assert result.result_rows[0][0] == 0