tkati-node-dedup 0.4.3__tar.gz → 0.4.4__tar.gz

{tkati_node_dedup-0.4.3/src/tkati_node_dedup.egg-info → tkati_node_dedup-0.4.4}/PKG-INFO

@@ -1,10 +1,10 @@
 Metadata-Version: 2.4
 Name: tkati-node-dedup
-Version: 0.4.3
+Version: 0.4.4
 Summary: Kafka-to-Kafka streaming node that deduplicates events by a configurable field within a rolling processing-time window
 Requires-Python: >=3.13
 Description-Content-Type: text/markdown
-Requires-Dist: tkati-core==0.4.3
+Requires-Dist: tkati-core==0.4.4
 Requires-Dist: loguru>=0.7.0
 Requires-Dist: pydantic-settings>=2.11.0
 Requires-Dist: pyarrow>=21.0.0
@@ -85,24 +85,34 @@ Every 10 seconds the node logs where its wall clock went, using
 
 ```
 dedup perf over 10s: 157000 rows in, 153880 out (3120 dropped), 157 iterations (0 input-starved)
-dedup perf: read=4.91s (49%) lookup=0.52s (5%) produce=3.96s (39%) write=0.21s (2%) commit=0.38s (4%)
+dedup perf: poll=4.43s (44%) parse=0.48s (5%) lookup=0.52s (5%) produce=3.96s (39%) write=0.21s (2%) commit=0.38s (4%)
 ```
 
 `dropped` is the rows this node deduplicated away.
 
-* `read` — fetching from Kafka *and* JSON-parsing into Arrow
+* `poll` — fetching message batches from the broker. Mostly broker round trips,
+  but it also includes librdkafka handing each message to Python, which has a
+  floor of roughly 0.8 us/message no matter how fast the broker is
+* `parse` — JSON-decoding those payloads into an Arrow table, and casting to
+  the internal schema
 * `lookup` — encoding keys, resolving in-batch duplicates, querying the store
 * `produce` — serializing and producing, including the blocking `flush`
 * `write` — marking the surviving keys seen
 * `commit` — the synchronous offset commit (and bucket cleanup, which is ~0
   except once an hour when a bucket is destroyed)
 
+`poll` and `parse` come from `tkati-core`'s consumer rather than from this
+node, which splices them in from `CONSUMER_PHASES`. They are split apart
+because their fixes are unrelated: a large `poll` points at batch sizing,
+broker latency or an under-fed topic, while a large `parse` points at the JSON
+decode and is what a faster wire format would address.
+
 Percentages are of the interval, not of each other, so they **do not sum to
 100** — the remainder is time in none of the named phases.
 
 `input-starved` counts iterations where the node drained the topic and waited
-out the batch timeout. Those iterations were not CPU-bound, and because `read`
-blocks for the whole wait, a mostly-starved interval will show `read` at close
+out the batch timeout. Those iterations were not CPU-bound, and because `poll`
+blocks for the whole wait, a mostly-starved interval will show `poll` at close
 to 100% and tells you nothing about whether the node can keep up.
 
 `benchmarks/bench_store.py` A/B tests the store in isolation. It populates in

tkati_node_dedup-0.4.3/PKG-INFO → tkati_node_dedup-0.4.4/README.md

@@ -1,15 +1,3 @@
-Metadata-Version: 2.4
-Name: tkati-node-dedup
-Version: 0.4.3
-Summary: Kafka-to-Kafka streaming node that deduplicates events by a configurable field within a rolling processing-time window
-Requires-Python: >=3.13
-Description-Content-Type: text/markdown
-Requires-Dist: tkati-core==0.4.3
-Requires-Dist: loguru>=0.7.0
-Requires-Dist: pydantic-settings>=2.11.0
-Requires-Dist: pyarrow>=21.0.0
-Requires-Dist: rocksdict>=0.3.0
-
 # tkati-node-dedup — streaming deduplication node
 
 Reads batches from a Kafka input topic, drops events that are duplicates of an
@@ -85,24 +73,34 @@ Every 10 seconds the node logs where its wall clock went, using
 
 ```
 dedup perf over 10s: 157000 rows in, 153880 out (3120 dropped), 157 iterations (0 input-starved)
-dedup perf: read=4.91s (49%) lookup=0.52s (5%) produce=3.96s (39%) write=0.21s (2%) commit=0.38s (4%)
+dedup perf: poll=4.43s (44%) parse=0.48s (5%) lookup=0.52s (5%) produce=3.96s (39%) write=0.21s (2%) commit=0.38s (4%)
 ```
 
 `dropped` is the rows this node deduplicated away.
 
-* `read` — fetching from Kafka *and* JSON-parsing into Arrow
+* `poll` — fetching message batches from the broker. Mostly broker round trips,
+  but it also includes librdkafka handing each message to Python, which has a
+  floor of roughly 0.8 us/message no matter how fast the broker is
+* `parse` — JSON-decoding those payloads into an Arrow table, and casting to
+  the internal schema
 * `lookup` — encoding keys, resolving in-batch duplicates, querying the store
 * `produce` — serializing and producing, including the blocking `flush`
 * `write` — marking the surviving keys seen
 * `commit` — the synchronous offset commit (and bucket cleanup, which is ~0
   except once an hour when a bucket is destroyed)
 
+`poll` and `parse` come from `tkati-core`'s consumer rather than from this
+node, which splices them in from `CONSUMER_PHASES`. They are split apart
+because their fixes are unrelated: a large `poll` points at batch sizing,
+broker latency or an under-fed topic, while a large `parse` points at the JSON
+decode and is what a faster wire format would address.
+
 Percentages are of the interval, not of each other, so they **do not sum to
 100** — the remainder is time in none of the named phases.
 
 `input-starved` counts iterations where the node drained the topic and waited
-out the batch timeout. Those iterations were not CPU-bound, and because `read`
-blocks for the whole wait, a mostly-starved interval will show `read` at close
+out the batch timeout. Those iterations were not CPU-bound, and because `poll`
+blocks for the whole wait, a mostly-starved interval will show `poll` at close
 to 100% and tells you nothing about whether the node can keep up.
 
 `benchmarks/bench_store.py` A/B tests the store in isolation. It populates in

{tkati_node_dedup-0.4.3 → tkati_node_dedup-0.4.4}/pyproject.toml

@@ -1,11 +1,11 @@
 [project]
 name = "tkati-node-dedup"
-version = "0.4.3"
+version = "0.4.4"
 description = "Kafka-to-Kafka streaming node that deduplicates events by a configurable field within a rolling processing-time window"
 readme = "README.md"
 requires-python = ">=3.13"
 dependencies = [
-    "tkati-core==0.4.3",
+    "tkati-core==0.4.4",
     "loguru>=0.7.0",
     "pydantic-settings>=2.11.0",
     "pyarrow>=21.0.0",

{tkati_node_dedup-0.4.3 → tkati_node_dedup-0.4.4}/src/tkati_node_dedup/main.py

@@ -1,15 +1,23 @@
 import pyarrow as pa
 from loguru import logger
-from tkati_core import Consumer, LoopStats, Producer, build_consumer, build_producer
+from tkati_core import (
+    CONSUMER_PHASES,
+    Consumer,
+    LoopStats,
+    Producer,
+    build_consumer,
+    build_producer,
+)
 
 from tkati_node_dedup.settings import AppSettings
 from tkati_node_dedup.store import BucketedDedupStore
 
 # Reported in this order, not sorted by duration: a stable field order is what
-# makes two consecutive log lines comparable at a glance. Lives here rather
-# than in tkati-core because these five names are this node's pipeline —
-# tkati-node-el, for instance, has no lookup or write phase.
-_PHASES = ("read", "lookup", "produce", "write", "commit")
+# makes two consecutive log lines comparable at a glance. The tail lives here
+# rather than in tkati-core because these four names are this node's pipeline —
+# tkati-node-el, for instance, has no lookup or write phase. The head is spliced
+# in from the consumer, which owns the names it times itself against.
+_PHASES = (*CONSUMER_PHASES, "lookup", "produce", "write", "commit")
 
 
 def _new_stats() -> LoopStats:
@@ -57,11 +65,15 @@ def run_one_iteration(
         except Exception:
             logger.exception("dedup store cleanup failed; will retry next iteration")
 
-    with stats.phase("read"):
-        batch = consumer.read_arrow(
-            num_messages=settings.input.consumer.batch_size,
-            timeout=settings.input.consumer.batch_timeout_sec,
-        )
+    # No phase block here: the consumer splits its own time into `poll` and
+    # `parse`. Wrapping it in an umbrella phase as well would double-count that
+    # time, and the percentages are of the interval — they are meant to fall
+    # short of 100%, with the shortfall being genuinely unaccounted work.
+    batch = consumer.read_arrow(
+        num_messages=settings.input.consumer.batch_size,
+        timeout=settings.input.consumer.batch_timeout_sec,
+        stats=stats,
+    )
     stats.iterations += 1
     if batch is None:
         stats.starved_iterations += 1

tkati_node_dedup-0.4.3/README.md → tkati_node_dedup-0.4.4/src/tkati_node_dedup.egg-info/PKG-INFO

@@ -1,3 +1,15 @@
+Metadata-Version: 2.4
+Name: tkati-node-dedup
+Version: 0.4.4
+Summary: Kafka-to-Kafka streaming node that deduplicates events by a configurable field within a rolling processing-time window
+Requires-Python: >=3.13
+Description-Content-Type: text/markdown
+Requires-Dist: tkati-core==0.4.4
+Requires-Dist: loguru>=0.7.0
+Requires-Dist: pydantic-settings>=2.11.0
+Requires-Dist: pyarrow>=21.0.0
+Requires-Dist: rocksdict>=0.3.0
+
 # tkati-node-dedup — streaming deduplication node
 
 Reads batches from a Kafka input topic, drops events that are duplicates of an
@@ -73,24 +85,34 @@ Every 10 seconds the node logs where its wall clock went, using
 
 ```
 dedup perf over 10s: 157000 rows in, 153880 out (3120 dropped), 157 iterations (0 input-starved)
-dedup perf: read=4.91s (49%) lookup=0.52s (5%) produce=3.96s (39%) write=0.21s (2%) commit=0.38s (4%)
+dedup perf: poll=4.43s (44%) parse=0.48s (5%) lookup=0.52s (5%) produce=3.96s (39%) write=0.21s (2%) commit=0.38s (4%)
 ```
 
 `dropped` is the rows this node deduplicated away.
 
-* `read` — fetching from Kafka *and* JSON-parsing into Arrow
+* `poll` — fetching message batches from the broker. Mostly broker round trips,
+  but it also includes librdkafka handing each message to Python, which has a
+  floor of roughly 0.8 us/message no matter how fast the broker is
+* `parse` — JSON-decoding those payloads into an Arrow table, and casting to
+  the internal schema
 * `lookup` — encoding keys, resolving in-batch duplicates, querying the store
 * `produce` — serializing and producing, including the blocking `flush`
 * `write` — marking the surviving keys seen
 * `commit` — the synchronous offset commit (and bucket cleanup, which is ~0
   except once an hour when a bucket is destroyed)
 
+`poll` and `parse` come from `tkati-core`'s consumer rather than from this
+node, which splices them in from `CONSUMER_PHASES`. They are split apart
+because their fixes are unrelated: a large `poll` points at batch sizing,
+broker latency or an under-fed topic, while a large `parse` points at the JSON
+decode and is what a faster wire format would address.
+
 Percentages are of the interval, not of each other, so they **do not sum to
 100** — the remainder is time in none of the named phases.
 
 `input-starved` counts iterations where the node drained the topic and waited
-out the batch timeout. Those iterations were not CPU-bound, and because `read`
-blocks for the whole wait, a mostly-starved interval will show `read` at close
+out the batch timeout. Those iterations were not CPU-bound, and because `poll`
+blocks for the whole wait, a mostly-starved interval will show `poll` at close
 to 100% and tells you nothing about whether the node can keep up.
 
 `benchmarks/bench_store.py` A/B tests the store in isolation. It populates in

{tkati_node_dedup-0.4.3 → tkati_node_dedup-0.4.4}/src/tkati_node_dedup.egg-info/requires.txt

@@ -1,4 +1,4 @@
-tkati-core==0.4.3
+tkati-core==0.4.4
 loguru>=0.7.0
 pydantic-settings>=2.11.0
 pyarrow>=21.0.0