tkati-node-dedup 0.4.1__tar.gz → 0.4.3__tar.gz

tkati_node_dedup-0.4.3/PKG-INFO

@@ -0,0 +1,185 @@
+Metadata-Version: 2.4
+Name: tkati-node-dedup
+Version: 0.4.3
+Summary: Kafka-to-Kafka streaming node that deduplicates events by a configurable field within a rolling processing-time window
+Requires-Python: >=3.13
+Description-Content-Type: text/markdown
+Requires-Dist: tkati-core==0.4.3
+Requires-Dist: loguru>=0.7.0
+Requires-Dist: pydantic-settings>=2.11.0
+Requires-Dist: pyarrow>=21.0.0
+Requires-Dist: rocksdict>=0.3.0
+
+# tkati-node-dedup — streaming deduplication node
+
+Reads batches from a Kafka input topic, drops events that are duplicates of an
+event seen on the same `field` within a rolling processing-time window, and
+writes the deduplicated batch to a configurable output. Duplicate state is
+tracked in an embedded, on-disk RocksDB store local to this process — no
+external dedup service is required.
+
+## Configuration
+
+Settings are loaded from a TOML file. Set the `SETTINGS_FILE` environment
+variable to point to it (defaults to `settings.toml`).
+
+```toml
+[input]
+type = "kafka"
+
+[input.connection]
+broker = "redpanda:29092"
+
+[input.topic]
+name = "raw_event"
+
+[input.topic.schema]
+uid  = "string"
+time = "timestamp[ms]"
+# … other columns
+
+[input.consumer]
+group_id          = "node-dedup-group"
+batch_size        = 1000
+batch_timeout_sec = 10
+auto_offset_reset = "latest"
+
+[output]
+type = "kafka"
+
+[output.connection]
+broker = "redpanda:29092"
+
+[output.topic]
+name = "raw_event_deduped"
+
+[dedup]
+field        = "uid"     # column in input.topic.schema to dedup by
+window_hours = 3         # rolling dedup window
+bucket_hours = 1         # on-disk bucket granularity (effective window is
+                          # window_hours .. window_hours + bucket_hours)
+store_dir    = "/var/lib/tkati-node-dedup/store"
+
+# Optional. Performance tuning for the embedded store; the defaults below are
+# the ones shipped, and were chosen by A/B measurement rather than reasoning
+# (see "Dedup store performance").
+[dedup.rocksdb]
+point_lookup_optimized   = true    # RocksDB bloom filter + block cache
+memtable_bloom_ratio     = 0.02    # 0 disables
+block_cache_mb           = 128
+write_buffer_mb          = 64
+compression              = "none"  # "none" | "lz4" | "zstd" | "snappy"
+disable_wal              = true
+disable_auto_compactions = false   # benchmarking only; see below
+enable_statistics        = false   # costs ~5-10%; diagnosis only
+```
+
+Output and DLQ follow the same `OutputSettings` shape as `tkati-node-el`
+(`"kafka"` or `"clickhouse"`) — see that package's README for the full
+connection/table config shape.
+
+## Dedup store performance
+
+Every 10 seconds the node logs where its wall clock went, using
+`LoopStats` from `tkati-core`:
+
+```
+dedup perf over 10s: 157000 rows in, 153880 out (3120 dropped), 157 iterations (0 input-starved)
+dedup perf: read=4.91s (49%) lookup=0.52s (5%) produce=3.96s (39%) write=0.21s (2%) commit=0.38s (4%)
+```
+
+`dropped` is the rows this node deduplicated away.
+
+* `read` — fetching from Kafka *and* JSON-parsing into Arrow
+* `lookup` — encoding keys, resolving in-batch duplicates, querying the store
+* `produce` — serializing and producing, including the blocking `flush`
+* `write` — marking the surviving keys seen
+* `commit` — the synchronous offset commit (and bucket cleanup, which is ~0
+  except once an hour when a bucket is destroyed)
+
+Percentages are of the interval, not of each other, so they **do not sum to
+100** — the remainder is time in none of the named phases.
+
+`input-starved` counts iterations where the node drained the topic and waited
+out the batch timeout. Those iterations were not CPU-bound, and because `read`
+blocks for the whole wait, a mostly-starved interval will show `read` at close
+to 100% and tells you nothing about whether the node can keep up.
+
+`benchmarks/bench_store.py` A/B tests the store in isolation. It populates in
+one process and measures in a fresh one, because a store that has just been
+written has everything in its memtable and every table-level option looks
+like it does nothing.
+
+**Turning RocksDB's bloom filter on is the single largest win**, because it
+was off: RocksDB has no filter policy by default, so every negative lookup —
+and with rare duplicates nearly every lookup is negative — read data blocks
+out of the SSTs. Enabling it cut data-block reads by ~40x.
+
+**Caveat, and the reason the code looks the way it does:** the obvious API,
+`BlockBasedOptions.set_bloom_filter()`, **does not work in rocksdict 0.3.29**
+(the current release). It writes the filter into the SST files but the read
+path never consults it — `rocksdb.bloom.filter.useful` stays at 0 and the
+data-block read count is identical with the filter on and off. Only the
+all-in-one `Options.optimize_for_point_lookup()` helper works, and calling
+`set_block_based_table_factory()` after it silently undoes it. Recheck this
+if rocksdict is ever upgraded.
+
+Two changes that look obviously right for this workload measured *worse* and
+are deliberately not enabled — don't "fix" them without re-running the
+benchmark:
+
+* **A larger write buffer.** 256MB measured 25% worse on writes and 20% worse
+  on reads than 64MB.
+* **Disabling compaction.** Each bucket is deleted within the hour, so its
+  compaction looks like pure waste — but it bought nothing on writes (2.41 vs
+  2.43 µs/key; compaction runs on background threads and never contended with
+  the write path) while costing 2.2x on reads as L0 files accumulated.
+
+## Delivery & dedup guarantees
+
+**Delivery: at-least-once.** Offsets are committed only after (1) the
+filtered batch is produced and confirmed delivered (`produce_arrow` followed
+by a blocking `flush`), and (2) the surviving keys are recorded in the current
+RocksDB bucket. If the process crashes between steps, the same input batch is
+re-read at restart; because the keys from a completed produce are already
+marked seen, re-processing that batch is a no-op (or reproduces only the
+genuinely-new subset) rather than losing data.
+
+**The dedup store is not crash-durable, by design.** With `disable_wal`
+(the default) writes go to a volatile memtable, so a hard kill can lose up to
+one write buffer's worth of dedup state — those keys stop being recognized as
+seen, and later duplicates of them are forwarded. It can never cause an event
+to be dropped, which is the tradeoff this node makes everywhere: Kafka is the
+source of truth and the committed offset, not RocksDB, is the durability
+boundary. A *graceful* shutdown flushes and loses nothing. Set
+`dedup.rocksdb.disable_wal = false` to trade throughput for crash durability.
+
+**On any internal dedup-store failure — a bucket won't open, a lookup errors,
+a disk I/O error — the node treats the event as NOT a duplicate and forwards
+it.** This node will occasionally forward a duplicate it should have caught,
+but will never silently drop a real event because of dedup-store trouble.
+
+**The window is approximate, not exact.** Because state is bucketed in
+`bucket_hours` increments (default 1h) rather than a true sliding window, the
+effective dedup window is between `window_hours` and
+`window_hours + bucket_hours`. Stale buckets are deleted from disk
+automatically once they fall outside the window — checked once per iteration,
+so state never grows unbounded.
+
+**Bucketing is by processing time**, not any timestamp field in the event
+payload — an event's bucket is when this node handles it, not when it
+happened upstream.
+
+## IMPORTANT: dedup state is local to this process
+
+The RocksDB store lives on local disk at `store_dir` and is **not shared**
+between instances. Running multiple concurrent instances of this node against
+the same input topic (e.g. multiple consumers in the same consumer group, or
+multiple replicas) will **not** dedup correctly across instances unless the
+input is partitioned such that all events sharing a `field` value are always
+routed to the *same* instance (e.g. Kafka partitioning keyed on `field`, one
+node instance per partition or partition subset it exclusively owns). Running
+this node with more parallelism than that will let duplicates leak through
+across instance boundaries. This is a direct consequence of choosing an
+embedded local-file store instead of a shared external one — evaluate this
+tradeoff before scaling this node horizontally.

tkati_node_dedup-0.4.3/README.md

@@ -0,0 +1,173 @@
+# tkati-node-dedup — streaming deduplication node
+
+Reads batches from a Kafka input topic, drops events that are duplicates of an
+event seen on the same `field` within a rolling processing-time window, and
+writes the deduplicated batch to a configurable output. Duplicate state is
+tracked in an embedded, on-disk RocksDB store local to this process — no
+external dedup service is required.
+
+## Configuration
+
+Settings are loaded from a TOML file. Set the `SETTINGS_FILE` environment
+variable to point to it (defaults to `settings.toml`).
+
+```toml
+[input]
+type = "kafka"
+
+[input.connection]
+broker = "redpanda:29092"
+
+[input.topic]
+name = "raw_event"
+
+[input.topic.schema]
+uid  = "string"
+time = "timestamp[ms]"
+# … other columns
+
+[input.consumer]
+group_id          = "node-dedup-group"
+batch_size        = 1000
+batch_timeout_sec = 10
+auto_offset_reset = "latest"
+
+[output]
+type = "kafka"
+
+[output.connection]
+broker = "redpanda:29092"
+
+[output.topic]
+name = "raw_event_deduped"
+
+[dedup]
+field        = "uid"     # column in input.topic.schema to dedup by
+window_hours = 3         # rolling dedup window
+bucket_hours = 1         # on-disk bucket granularity (effective window is
+                          # window_hours .. window_hours + bucket_hours)
+store_dir    = "/var/lib/tkati-node-dedup/store"
+
+# Optional. Performance tuning for the embedded store; the defaults below are
+# the ones shipped, and were chosen by A/B measurement rather than reasoning
+# (see "Dedup store performance").
+[dedup.rocksdb]
+point_lookup_optimized   = true    # RocksDB bloom filter + block cache
+memtable_bloom_ratio     = 0.02    # 0 disables
+block_cache_mb           = 128
+write_buffer_mb          = 64
+compression              = "none"  # "none" | "lz4" | "zstd" | "snappy"
+disable_wal              = true
+disable_auto_compactions = false   # benchmarking only; see below
+enable_statistics        = false   # costs ~5-10%; diagnosis only
+```
+
+Output and DLQ follow the same `OutputSettings` shape as `tkati-node-el`
+(`"kafka"` or `"clickhouse"`) — see that package's README for the full
+connection/table config shape.
+
+## Dedup store performance
+
+Every 10 seconds the node logs where its wall clock went, using
+`LoopStats` from `tkati-core`:
+
+```
+dedup perf over 10s: 157000 rows in, 153880 out (3120 dropped), 157 iterations (0 input-starved)
+dedup perf: read=4.91s (49%) lookup=0.52s (5%) produce=3.96s (39%) write=0.21s (2%) commit=0.38s (4%)
+```
+
+`dropped` is the rows this node deduplicated away.
+
+* `read` — fetching from Kafka *and* JSON-parsing into Arrow
+* `lookup` — encoding keys, resolving in-batch duplicates, querying the store
+* `produce` — serializing and producing, including the blocking `flush`
+* `write` — marking the surviving keys seen
+* `commit` — the synchronous offset commit (and bucket cleanup, which is ~0
+  except once an hour when a bucket is destroyed)
+
+Percentages are of the interval, not of each other, so they **do not sum to
+100** — the remainder is time in none of the named phases.
+
+`input-starved` counts iterations where the node drained the topic and waited
+out the batch timeout. Those iterations were not CPU-bound, and because `read`
+blocks for the whole wait, a mostly-starved interval will show `read` at close
+to 100% and tells you nothing about whether the node can keep up.
+
+`benchmarks/bench_store.py` A/B tests the store in isolation. It populates in
+one process and measures in a fresh one, because a store that has just been
+written has everything in its memtable and every table-level option looks
+like it does nothing.
+
+**Turning RocksDB's bloom filter on is the single largest win**, because it
+was off: RocksDB has no filter policy by default, so every negative lookup —
+and with rare duplicates nearly every lookup is negative — read data blocks
+out of the SSTs. Enabling it cut data-block reads by ~40x.
+
+**Caveat, and the reason the code looks the way it does:** the obvious API,
+`BlockBasedOptions.set_bloom_filter()`, **does not work in rocksdict 0.3.29**
+(the current release). It writes the filter into the SST files but the read
+path never consults it — `rocksdb.bloom.filter.useful` stays at 0 and the
+data-block read count is identical with the filter on and off. Only the
+all-in-one `Options.optimize_for_point_lookup()` helper works, and calling
+`set_block_based_table_factory()` after it silently undoes it. Recheck this
+if rocksdict is ever upgraded.
+
+Two changes that look obviously right for this workload measured *worse* and
+are deliberately not enabled — don't "fix" them without re-running the
+benchmark:
+
+* **A larger write buffer.** 256MB measured 25% worse on writes and 20% worse
+  on reads than 64MB.
+* **Disabling compaction.** Each bucket is deleted within the hour, so its
+  compaction looks like pure waste — but it bought nothing on writes (2.41 vs
+  2.43 µs/key; compaction runs on background threads and never contended with
+  the write path) while costing 2.2x on reads as L0 files accumulated.
+
+## Delivery & dedup guarantees
+
+**Delivery: at-least-once.** Offsets are committed only after (1) the
+filtered batch is produced and confirmed delivered (`produce_arrow` followed
+by a blocking `flush`), and (2) the surviving keys are recorded in the current
+RocksDB bucket. If the process crashes between steps, the same input batch is
+re-read at restart; because the keys from a completed produce are already
+marked seen, re-processing that batch is a no-op (or reproduces only the
+genuinely-new subset) rather than losing data.
+
+**The dedup store is not crash-durable, by design.** With `disable_wal`
+(the default) writes go to a volatile memtable, so a hard kill can lose up to
+one write buffer's worth of dedup state — those keys stop being recognized as
+seen, and later duplicates of them are forwarded. It can never cause an event
+to be dropped, which is the tradeoff this node makes everywhere: Kafka is the
+source of truth and the committed offset, not RocksDB, is the durability
+boundary. A *graceful* shutdown flushes and loses nothing. Set
+`dedup.rocksdb.disable_wal = false` to trade throughput for crash durability.
+
+**On any internal dedup-store failure — a bucket won't open, a lookup errors,
+a disk I/O error — the node treats the event as NOT a duplicate and forwards
+it.** This node will occasionally forward a duplicate it should have caught,
+but will never silently drop a real event because of dedup-store trouble.
+
+**The window is approximate, not exact.** Because state is bucketed in
+`bucket_hours` increments (default 1h) rather than a true sliding window, the
+effective dedup window is between `window_hours` and
+`window_hours + bucket_hours`. Stale buckets are deleted from disk
+automatically once they fall outside the window — checked once per iteration,
+so state never grows unbounded.
+
+**Bucketing is by processing time**, not any timestamp field in the event
+payload — an event's bucket is when this node handles it, not when it
+happened upstream.
+
+## IMPORTANT: dedup state is local to this process
+
+The RocksDB store lives on local disk at `store_dir` and is **not shared**
+between instances. Running multiple concurrent instances of this node against
+the same input topic (e.g. multiple consumers in the same consumer group, or
+multiple replicas) will **not** dedup correctly across instances unless the
+input is partitioned such that all events sharing a `field` value are always
+routed to the *same* instance (e.g. Kafka partitioning keyed on `field`, one
+node instance per partition or partition subset it exclusively owns). Running
+this node with more parallelism than that will let duplicates leak through
+across instance boundaries. This is a direct consequence of choosing an
+embedded local-file store instead of a shared external one — evaluate this
+tradeoff before scaling this node horizontally.

{tkati_node_dedup-0.4.1 → tkati_node_dedup-0.4.3}/pyproject.toml

@@ -1,11 +1,11 @@
 [project]
 name = "tkati-node-dedup"
-version = "0.4.1"
+version = "0.4.3"
 description = "Kafka-to-Kafka streaming node that deduplicates events by a configurable field within a rolling processing-time window"
 readme = "README.md"
 requires-python = ">=3.13"
 dependencies = [
-    "tkati-core==0.4.1",
+    "tkati-core==0.4.3",
     "loguru>=0.7.0",
     "pydantic-settings>=2.11.0",
     "pyarrow>=21.0.0",

{tkati_node_dedup-0.4.1 → tkati_node_dedup-0.4.3}/src/tkati_node_dedup/main.py

@@ -1,23 +1,34 @@
 import pyarrow as pa
 from loguru import logger
-from tkati_core import Consumer, Producer, build_consumer, build_producer
+from tkati_core import Consumer, LoopStats, Producer, build_consumer, build_producer
 
 from tkati_node_dedup.settings import AppSettings
 from tkati_node_dedup.store import BucketedDedupStore
 
+# Reported in this order, not sorted by duration: a stable field order is what
+# makes two consecutive log lines comparable at a glance. Lives here rather
+# than in tkati-core because these five names are this node's pipeline —
+# tkati-node-el, for instance, has no lookup or write phase.
+_PHASES = ("read", "lookup", "produce", "write", "commit")
+
+
+def _new_stats() -> LoopStats:
+    return LoopStats(name="dedup", phases=_PHASES)
+
 
 def _dedupe_batch(
-    batch: pa.Table, field: str, store: BucketedDedupStore
+    batch: pa.Table, field_name: str, store: BucketedDedupStore
 ) -> tuple[pa.Table, list[bytes]]:
     """Filter out rows whose dedup key was already seen (in-batch or in the store).
 
-    Returns (filtered_batch, keys_to_mark_seen). Null values in `field` always
-    pass through and are never added to the store — we can't dedup on nothing.
+    Returns (filtered_batch, keys_to_mark_seen). Null values in `field_name`
+    always pass through and are never added to the store — we can't dedup on
+    nothing.
 
     Encoding and the store lookup are both batched (one pass over the column,
     one RocksDB round trip per open bucket) rather than done per row.
     """
-    keys = store.encode_keys(batch.column(field))
+    keys = store.encode_keys(batch.column(field_name))
     keep_mask, new_keys = store.filter_duplicates(keys)
     filtered = batch.filter(keep_mask)
     return filtered, new_keys
@@ -28,55 +39,81 @@ def run_one_iteration(
     producer: Producer,
     store: BucketedDedupStore,
     settings: AppSettings,
+    stats: LoopStats | None = None,
 ) -> None:
+    stats = stats if stats is not None else _new_stats()
+
     # Runs first, every iteration (even if no batch arrives), and can never
     # raise. Buckets must be fresh *before* the dedupe check below runs —
     # doing this only after commit would leave a just-expired bucket open and
     # checked against for one extra iteration, and an idle node (no messages,
     # read_arrow returns None below) would never clean up at all.
-    try:
-        store.cleanup_expired()
-    except Exception:
-        logger.exception("dedup store cleanup failed; will retry next iteration")
-
-    batch = consumer.read_arrow(
-        num_messages=settings.input.consumer.batch_size,
-        timeout=settings.input.consumer.batch_timeout_sec,
-    )
+    # Timed into the "commit" bucket rather than given a phase of its own:
+    # it is ~0 except once an hour when a bucket is destroyed, so it shows
+    # up as an occasional commit spike instead of a permanent near-zero field.
+    with stats.phase("commit"):
+        try:
+            store.cleanup_expired()
+        except Exception:
+            logger.exception("dedup store cleanup failed; will retry next iteration")
+
+    with stats.phase("read"):
+        batch = consumer.read_arrow(
+            num_messages=settings.input.consumer.batch_size,
+            timeout=settings.input.consumer.batch_timeout_sec,
+        )
+    stats.iterations += 1
     if batch is None:
+        stats.starved_iterations += 1
         return
 
-    field = settings.dedup.field
-    if field not in batch.column_names:
+    # A short batch means the node drained the topic and waited out the batch
+    # timeout — it wasn't CPU-bound, so its timings say nothing about whether
+    # this node can keep up.
+    if len(batch) < settings.input.consumer.batch_size:
+        stats.starved_iterations += 1
+    stats.rows_in += len(batch)
+
+    field_name = settings.dedup.field
+    if field_name not in batch.column_names:
         logger.warning(
-            f"Dedup field '{field}' missing from batch schema; passing batch through unfiltered"
+            f"Dedup field '{field_name}' missing from batch schema; "
+            "passing batch through unfiltered"
         )
         filtered, new_keys = batch, []
     else:
-        filtered, new_keys = _dedupe_batch(batch, field, store)
+        # Includes the batch.filter() call, which is Arrow work rather than a
+        # store lookup — cheap enough not to be worth a sixth phase.
+        with stats.phase("lookup"):
+            filtered, new_keys = _dedupe_batch(batch, field_name, store)
 
     dropped = len(batch) - len(filtered)
+    stats.rows_out += len(filtered)
 
     if len(filtered) > 0:
-        producer.produce_arrow(filtered)
+        with stats.phase("produce"):
+            producer.produce_arrow(filtered)
         # Block until actually delivered before marking anything "seen" or
         # committing. Required here even though tkati-node-el's loop skips it:
         # KafkaProducer.produce_arrow() only enqueues (non-blocking), and
         # marking a key seen before it's durably delivered would risk losing
         # the event permanently on a crash. ClickhouseProducer.flush() is a
         # no-op since its inserts are already synchronous.
-        producer.flush()
+        with stats.phase("produce"):
+            producer.flush()
 
     # Only after a confirmed-successful produce: mark these keys seen.
-    store.add_many(new_keys)
+    with stats.phase("write"):
+        store.add_many(new_keys)
 
     # Only after mark-seen: commit. If we crash before this line, the batch is
     # re-read at restart; those keys are already in the store, so re-processing
     # it drops what was already produced — a harmless duplicate at worst, never
     # a lost event.
-    consumer.commit()
+    with stats.phase("commit"):
+        consumer.commit()
 
-    logger.info(
+    logger.debug(
         f"Batch of {len(batch)} rows: produced {len(filtered)}, "
         f"deduped {dropped} ({len(new_keys)} newly marked seen)"
     )
@@ -97,11 +134,14 @@ def main() -> None:
         root_dir=settings.dedup.store_dir,
         window_hours=settings.dedup.window_hours,
         bucket_hours=settings.dedup.bucket_hours,
+        tuning=settings.dedup.rocksdb,
     )
 
+    stats = _new_stats()
     try:
         while True:
-            run_one_iteration(consumer, producer, store, settings)
+            run_one_iteration(consumer, producer, store, settings, stats)
+            stats.report_if_due()
     finally:
         consumer.close()
         if dlq_producer is not None:

tkati_node_dedup-0.4.3/src/tkati_node_dedup/settings.py

@@ -0,0 +1,89 @@
+from typing import Literal
+
+from pydantic import BaseModel, field_validator
+from tkati_core.settings import InputSettings, OutputSettings, TomlBaseSettings
+
+
+class RocksDBSettings(BaseModel):
+    """Tuning knobs for the embedded dedup store.
+
+    The defaults here are chosen for this node's actual workload — a store
+    whose lookups almost always miss (duplicates are rare) and whose buckets
+    are written for one hour and then deleted — and they differ substantially
+    from RocksDB's own defaults. See `store.py` for the reasoning behind each.
+    """
+
+    # Enables RocksDB's bloom filter (10 bits/key) plus a block cache of
+    # `block_cache_mb` and an in-data-block hash index. There is no
+    # bits-per-key knob: the only bloom-filter API that actually works in
+    # rocksdict is the all-in-one `optimize_for_point_lookup` helper, which
+    # hardcodes 10. See BucketedDedupStore._build_options for the evidence.
+    # Off is the pre-tuning behavior, kept expressible for A/B benchmarking.
+    point_lookup_optimized: bool = True
+
+    # Bloom filter for the memtable, which on the current bucket holds this
+    # hour's keys. Measured against a warm memtable, 0 -> 0.02 takes an
+    # all-miss lookup from 3.23 to 0.85 µs/key at no write cost. Higher
+    # ratios are worse, not better (0.05 -> 1.22, 0.10 -> 1.51): a larger
+    # bloom probes with worse cache locality. 0.02 is also what
+    # optimize_for_point_lookup picks. 0 disables.
+    memtable_bloom_ratio: float = 0.02
+
+    block_cache_mb: int = 128
+
+    # 64MB, not larger. A 256MB buffer measured 25% worse on writes (deeper
+    # skiplist, larger memtable bloom to populate) and 20% worse on reads.
+    write_buffer_mb: int = 64
+
+    # Leave compaction ON. Each bucket is deleted within the hour, so skipping
+    # compaction looks free — but measured, it bought nothing on writes (2.41
+    # vs 2.43 µs/key; compaction runs on background threads and never
+    # contended with the write path) while costing 2.2x on reads (1.37 -> 2.96
+    # µs/key) as L0 files accumulated. Kept as a knob only for benchmarking.
+    disable_auto_compactions: bool = False
+
+    # Snappy is RocksDB's default and costs 2x on reads here (2.84 vs 1.37
+    # µs/key). Dedup keys are high-entropy and barely compress, and the store
+    # is ephemeral, so paying for compression buys little disk and costs real
+    # CPU.
+    compression: Literal["none", "lz4", "zstd", "snappy"] = "none"
+
+    disable_wal: bool = True
+    enable_statistics: bool = False
+
+    @field_validator("block_cache_mb", "write_buffer_mb")
+    @classmethod
+    def _positive_mb(cls, v: int) -> int:
+        if v <= 0:
+            raise ValueError("must be a positive number of megabytes")
+        return v
+
+    @field_validator("memtable_bloom_ratio")
+    @classmethod
+    def _ratio(cls, v: float) -> float:
+        # RocksDB caps this at 0.25 internally; 0 disables the memtable bloom.
+        if not 0.0 <= v < 1.0:
+            raise ValueError("must be in [0.0, 1.0)")
+        return v
+
+
+class DedupSettings(BaseModel):
+    field: str
+    window_hours: int = 3
+    bucket_hours: int = 1
+    store_dir: str = "./dedup_store"
+    rocksdb: RocksDBSettings = RocksDBSettings()
+
+    @field_validator("window_hours", "bucket_hours")
+    @classmethod
+    def _positive(cls, v: int) -> int:
+        if v <= 0:
+            raise ValueError("must be a positive number of hours")
+        return v
+
+
+class AppSettings(TomlBaseSettings):
+    input: InputSettings
+    output: OutputSettings
+    dlq: OutputSettings | None = None
+    dedup: DedupSettings