mq-bridge-py-basic 0.3.8__tar.gz → 0.3.10__tar.gz

mq_bridge_py_basic-0.3.10/.github/dependabot.yml

@@ -0,0 +1,11 @@
+# To get started with Dependabot version updates, you'll need to specify which
+# package ecosystems to update and where the package manifests are located.
+# Please see the documentation for all configuration options:
+# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
+
+version: 2
+updates:
+  - package-ecosystem: "cargo" # See documentation for possible values
+    directory: "/" # Location of package manifests
+    schedule:
+      interval: "weekly"

{mq_bridge_py_basic-0.3.8 → mq_bridge_py_basic-0.3.10}/.github/workflows/ci.yml

@@ -305,3 +305,30 @@ jobs:
       
       - name: Build documentation (deny warnings)
         run: RUSTDOCFLAGS="-D warnings" cargo doc --all-features --no-deps
+
+  # Supply chain: licenses, banned crates, source registries, RustSec advisories
+  supply-chain:
+    name: Supply Chain
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+
+      # Hard gate: an unapproved license, a banned crate or a non-crates.io
+      # source is always a real, actionable problem in this repo.
+      - name: Licenses / bans / sources
+        uses: EmbarkStudios/cargo-deny-action@v2
+        with:
+          command: check bans licenses sources
+
+      # Informational: every current advisory is in a transitive dependency we
+      # cannot upgrade ourselves, and RustSec updates daily. SECURITY.md records
+      # the per-feature reachability analysis; deny.toml carries the reasoning.
+      - name: Advisories (informational)
+        continue-on-error: true
+        uses: EmbarkStudios/cargo-deny-action@v2
+        with:
+          command: check advisories

{mq_bridge_py_basic-0.3.8 → mq_bridge_py_basic-0.3.10}/.gitignore

@@ -48,3 +48,7 @@ tests/**/*.pkcs12
 .pytest_cache/
 python/mq-bridge-py/.venv/
 python/mq-bridge-py/dist/
+
+# Local build setup (shared cargo target dir with mq-bridge-app), not for git
+.cargo/
+.vscode/

mq_bridge_py_basic-0.3.10/CHANGELOG.md

@@ -0,0 +1,76 @@
+# Changelog
+
+All notable changes to `mq-bridge`. Newest first.
+
+## 0.3.10
+
+
+### Changed
+
+- DLQ and output middleware now run around the publish step, not inside the handler, so a
+  handler failure is no longer retried by the output chain. The tradeoff: a handler failure
+  now skips the output middleware entirely, so `dlq` on the output endpoint cannot capture
+  it — only publish failures reach the DLQ. Put a `dlq` on the input endpoint to catch
+  handler failures.
+- Errors carry their full cause chain instead of only the outermost context.
+- `deduplication` on an **output** endpoint is now a startup error instead of a warning and a
+  silent no-op. Move it to the route's input endpoint.
+- `limiter` paces a single `send_batch` by its own message count, so one large batch is no
+  longer a free burst. Sustained throughput is unchanged.
+- A `message_id` that is neither a UUID nor a `u128` is hashed to a stable id rather than
+  making the whole JSON envelope unparseable.
+
+### Performance
+
+- **Kafka consumer**: a long-lived prefetch task reads librdkafka continuously into a bounded
+  channel, instead of rebuilding the stream inside every `receive_batch`. librdkafka only keeps
+  requesting records while its queue is drained, so every pause the pipeline took — a transform,
+  a slow sink — was also a pause in fetching, and the fetch rate collapsed to well under what the
+  broker could serve. Batch offsets are also recorded once per partition rather than once per
+  message, which was O(n²) in the batch and allocated two `CString`s per record.
+  Kafka → transform → file: **192,854 → 824,983 rows/s** on the 1M-row, four-partition
+  benchmark, from 0.35x to 1.51x Arroyo on identical output. A 16,384-message passthrough
+  batch went from 11.5s to 2.2s, and from 10.4s to 0.9s of CPU.
+- **Postgres / sqlx**: `test_before_acquire(false)` on the pool, zero-copy row encoding via
+  a prebuilt `JsonRowSchema`, and prebuilt first/next page queries on the cursor path.
+- **Command handler**: `send_batch` no longer loops per-message `send()` (≈8x on batched routes).
+- **File**: single-pass byte-array decode, faster CSV writes, compression sniffing.
+- **Deduplication**: the two-phase reserve/commit no longer writes to the store twice per
+  message. Reservations are held in memory — sled takes an exclusive file lock on its directory,
+  so a claim only ever has to be visible to this process — leaving one write per message, on
+  commit.
+
+### Fixed
+
+- `insert_query` batch inserts dropped anything but a bare token from the `VALUES` tuple, so
+  `decode(${payload:x}, 'base64')` and casts were silently discarded — a binary (`bytea`)
+  column could not be written. The batch path now keeps the user's SQL and falls back to
+  iterative inserts when the tuple contains an expression.
+- A payload string containing an embedded NUL byte was dropped by the driver and stored as
+  SQL `NULL` while the route reported success. It is now rejected as non-retryable.
+- Database errors were rendered twice, because `sqlx` errors already display their own
+  source. Cause chains now skip a link an earlier one already contains.
+- `cookie_jar`'s `inject_metadata` resolved only a bare stored name, so the namespaced
+  `cookie.<name>` / `value.<name>` spelling that `export_metadata_prefix` reports back
+  injected nothing. Both spellings now work.
+- MQTT publishes are confirmed by PUBACK/PUBCOMP and guarded against session resets, which
+  took chaos-test message loss to zero.
+- Postgres CDC advances its replication slot durably on shutdown instead of leaving the
+  feedback unflushed.
+- `deduplication` could silently drop a message after a crash. The reservation was written to
+  the store *before* the message was processed, so a redelivery arriving within the 5s pending
+  TTL was classified as a duplicate and acked — without ever having been written to the sink.
+  Reservations are now in-memory and die with the process, so the redelivery is reprocessed.
+- A Kafka record with no value — a tombstone, which is ordinary traffic on a compacted topic —
+  failed the whole batch. Its offset was never committed, so the route reconnected onto the same
+  record forever: one tombstone wedged the consumer permanently, and a compacted topic could not
+  be consumed at all. Tombstones now arrive as an empty payload flagged `mqb.kafka.tombstone`.
+- `--drain` on a Kafka source could report success having copied only part of a topic, or none
+  of it. An idle wait was taken to mean "source exhausted", but an idle channel means nothing on
+  its own: before the first fetch lands, or in an ordinary gap between fetches, it looks exactly
+  the same as the end of the data. The shorter the idle timeout the more went missing — at 1ms a
+  1,000,000-row topic landed 0 rows, and the copy still exited successfully. A drain now
+  completes only once every assigned partition has reached the offset it held when the drain
+  began, and lands all 1,000,000 rows at every idle timeout including 0.
+- Drain no longer hangs on an empty source, and reconnect attempts are bounded.
+- CSV reader handles quoted newlines; file endpoints fail fast on an unopenable path.

{mq_bridge_py_basic-0.3.8 → mq_bridge_py_basic-0.3.10}/Cargo.lock

@@ -2976,6 +2976,15 @@ dependencies = [
  "windows-link",
 ]
 
+[[package]]
+name = "libmimalloc-sys"
+version = "0.1.49"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6a45a52f43e1c16f667ccfe4dd8c85b7f7c204fd5e3bf46c5b0db9a5c3c0b8e9"
+dependencies = [
+ "cc",
+]
+
 [[package]]
 name = "libmqm-constants"
 version = "0.8.4"
@@ -3200,6 +3209,15 @@ dependencies = [
  "sketches-ddsketch",
 ]
 
+[[package]]
+name = "mimalloc"
+version = "0.1.52"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2d4139bb28d14ad1facf21d5eb8825051b326e172d216b39f6d31df53cc97862"
+dependencies = [
+ "libmimalloc-sys",
+]
+
 [[package]]
 name = "mime"
 version = "0.3.17"
@@ -3330,7 +3348,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge"
-version = "0.3.8"
+version = "0.3.10"
 dependencies = [
  "aes-gcm",
  "anyhow",
@@ -3411,7 +3429,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge-bindings-common"
-version = "0.3.8"
+version = "0.3.10"
 dependencies = [
  "anyhow",
  "fast-uuid-v7",
@@ -3425,12 +3443,13 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge-node"
-version = "0.3.8"
+version = "0.3.10"
 dependencies = [
  "anyhow",
  "async-trait",
  "bytes",
  "fast-uuid-v7",
+ "mimalloc",
  "mq-bridge",
  "mq-bridge-bindings-common",
  "napi",
@@ -3446,12 +3465,13 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge-py"
-version = "0.3.8"
+version = "0.3.10"
 dependencies = [
  "anyhow",
  "async-trait",
  "bytes",
  "fast-uuid-v7",
+ "mimalloc",
  "mq-bridge",
  "mq-bridge-bindings-common",
  "pyo3",
@@ -3725,9 +3745,9 @@ dependencies = [
 
 [[package]]
 name = "omq-proto"
-version = "0.24.1"
+version = "0.25.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c9335ef21d570294464c491837af5477caddb05cf3c7223af1c4341a127e6035"
+checksum = "856d18f6b58f39153fcf0b574efe3f04362ff44d3a46829eb984c1f9ef2d121c"
 dependencies = [
  "bytes",
  "patricia_tree",
@@ -3739,9 +3759,9 @@ dependencies = [
 
 [[package]]
 name = "omq-tokio"
-version = "0.20.1"
+version = "0.20.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4a9d9c6caa88d29df5b9dc9638ea9f0f33b087ecc0f149b70e61149f97bcdb80"
+checksum = "98aaa33702e7d723de268dd9fe725f93d469afef3f1bedb14cfa7f579f61a7b7"
 dependencies = [
  "arc-swap",
  "bytes",
@@ -7191,9 +7211,9 @@ dependencies = [
 
 [[package]]
 name = "yring"
-version = "0.3.10"
+version = "0.3.11"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "39146d2147cfb9f71b9f071f93f7b5dbcfb28eee528dafcaf5f500f4917cb69f"
+checksum = "f8545becda6379f37e13fbdf5d9de464849bdd1565a995368648f11fe9d96d13"
 dependencies = [
  "loom",
 ]

{mq_bridge_py_basic-0.3.8 → mq_bridge_py_basic-0.3.10}/Cargo.toml

@@ -16,7 +16,7 @@ members = ["bindings-common", "python/mq-bridge-py"]
 resolver = "2"
 
 [workspace.package]
-version = "0.3.8"
+version = "0.3.10"
 
 [workspace.dependencies]
 anyhow = "1.0"
@@ -29,6 +29,12 @@ serde_yaml_ng = "0.10"
 tokio = "1.34"  # `Receiver::recv_many` (HTTP source batch drain) needs 1.34+.
 tracing = "0.1"
 
+[profile.dev.package."*"]
+debug = false
+
+[profile.dev.build-override]
+debug = false
+
 [profile.release]
 lto = "thin"
 strip = "symbols"
@@ -138,7 +144,8 @@ flate2 = { version = "1.1", optional = true }
 lz4_flex = { version = "0.14", optional = true, default-features = false, features = ["frame"] }
 # zstd (bundled libzstd via zstd-sys); Decoder handles concatenated frames like gzip/lz4.
 zstd = { version = "0.13", optional = true }
-base64 = { version = "0.23", optional = true }
+# Non-optional: `CanonicalMessage`'s JSON form encodes binary payloads as base64.
+base64 = { version = "0.23" }
 
 # AEAD ciphers for the `encryption` feature (payload middleware + at-rest file/object).
 chacha20poly1305 = { version = "0.11", optional = true }
@@ -210,7 +217,7 @@ float-roundtrip = ["serde_json/float_roundtrip"]
 # runtime the client is loaded lazily only when an ibm-mq endpoint is opened
 # (clear error if the client isn't installed). Use `ibm-mq-static` instead for
 # static linking.
-full = ["middleware", "mongodb", "kafka", "amqp", "nats", "mqtt", "http", "aws", "zeromq", "redis-streams", "sled", "grpc", "sqlx", "clickhouse", "websocket", "postgres-cdc", "object-store", "rustls-aws-lc", "ibm-mq"]
+full = ["middleware", "mongodb", "kafka", "amqp", "nats", "mqtt", "http", "aws", "zeromq", "zeromq-omq", "redis-streams", "sled", "grpc", "sqlx", "clickhouse", "websocket", "postgres-cdc", "object-store", "rustls-aws-lc", "ibm-mq"]
 
 # Endpoints that build across most OS targets (glibc/musl/Alpine, macOS,
 # Windows, aarch64/armv7) without a special toolchain — no librdkafka, protoc,
@@ -229,7 +236,7 @@ dedup = ["sled"]
 compression = ["dep:flate2", "dep:lz4_flex", "dep:zstd"]
 # AEAD payload encryption: the `encryption` middleware and the at-rest
 # `encryption` field on the file/object_store endpoints.
-encryption = ["dep:chacha20poly1305", "dep:aes-gcm", "dep:base64"]
+encryption = ["dep:chacha20poly1305", "dep:aes-gcm"]
 metrics = ["dep:metrics"]
 test-utils = ["dep:tracing-appender", "dep:tracing-subscriber", "dep:serde_yaml_ng"]
 # Opt-in produce/consume-only isolation benchmarks (throwaway diagnostics, off by default)
@@ -254,9 +261,9 @@ amqp = ["lapin", "rustls", "rustls-pemfile", "url"]
 nats = ["async-nats", "rustls", "rustls-pemfile"]
 mongodb = ["dep:mongodb", "url"]
 mqtt = ["rumqttc", "rustls", "tokio-rustls", "rustls-pemfile", "url"]
-http = ["dep:arc-swap", "hyper", "hyper-util", "hyper-rustls", "http-body-util", "dep:http-body", "rustls", "tokio-rustls", "rustls-pemfile", "webpki-roots", "h2", "dep:flate2", "dep:lz4_flex", "dep:zstd", "dep:base64"]
+http = ["dep:arc-swap", "hyper", "hyper-util", "hyper-rustls", "http-body-util", "dep:http-body", "rustls", "tokio-rustls", "rustls-pemfile", "webpki-roots", "h2", "dep:flate2", "dep:lz4_flex", "dep:zstd"]
 websocket = ["dep:tokio-websockets"]
-aws = ["aws-config", "aws-sdk-sqs", "aws-sdk-sns", "dep:base64"]
+aws = ["aws-config", "aws-sdk-sqs", "aws-sdk-sns"]
 # Runtime-loaded IBM MQ client via dlopen: builds with no SDK present; the IBM
 # redistributable client is only needed at runtime, and only if an ibm-mq
 # endpoint is actually used.
@@ -283,10 +290,12 @@ object-store = ["dep:object_store", "url"]
 # Postgres logical-replication CDC (pgoutput). Reuses the sqlx dependency for the
 # slot-lifecycle control plane; the streaming wire protocol is pgwire-replication.
 # Needs Rust 1.88 (pgwire-replication MSRV), like the amqp feature gate.
-postgres-cdc = ["dep:pgwire-replication", "dep:byteorder", "sqlx", "dep:base64", "url", "dep:percent-encoding"]
+postgres-cdc = ["dep:pgwire-replication", "dep:byteorder", "sqlx", "url", "dep:percent-encoding"]
 
 [dev-dependencies]
 config = { version = "0.15" }
+# `start_paused` — lets a test cross a 30s health window without waiting 30s.
+tokio = { workspace = true, features = ["test-util"] }
 serde_yaml_ng = "0.10"
 tempfile = "3.27"
 chrono = "0.4"

{mq_bridge_py_basic-0.3.8 → mq_bridge_py_basic-0.3.10}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mq-bridge-py-basic
-Version: 0.3.8
+Version: 0.3.10
 Classifier: Development Status :: 3 - Alpha
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3
@@ -232,8 +232,8 @@ that ack each batch individually (NATS JetStream, AMQP, MQTT) accept any order.
 > per source position (Kafka `partition:offset`, NATS `stream_sequence`, AMQP
 > delivery tag) and makes a natural primary key. Source cursor fields are also
 > available in `message.metadata` (`mqb.src.kafka_topic`/`mqb.src.kafka_offset`, `mqb.src.nats_subject`/`mqb.src.nats_stream_sequence`,
-> `mqb.src.amqp_routing_key`/`mqb.src.amqp_delivery_tag`) when you opt in by setting
-> the `MQB_SOURCE_METADATA=1` environment variable (off by default).
+> `mqb.src.amqp_routing_key`/`mqb.src.amqp_delivery_tag`) when its Kafka, NATS, or
+> AMQP source config sets `source_metadata: true` (off by default).
 >
 > **Ack deadlines vs slow loads.** JetStream `AckWait` (default 30s), AMQP
 > prefetch/consumer-timeout and MQTT inflight windows each bound how long a batch

{mq_bridge_py_basic-0.3.8 → mq_bridge_py_basic-0.3.10}/README.md

@@ -6,6 +6,8 @@
 ![Linux](https://img.shields.io/badge/Linux-supported-green?logo=linux)
 ![Windows](https://img.shields.io/badge/Windows-supported-green?logo=windows)
 ![macOS](https://img.shields.io/badge/macOS-supported-green?logo=apple)
+[![Supply chain](https://img.shields.io/badge/supply%20chain-cargo--deny-blue)](deny.toml)
+[![Security](https://img.shields.io/badge/security-advisory%20analysis-brightgreen)](SECURITY.md)
 [![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 
 
@@ -21,7 +23,7 @@ It is not only a forwarder. A route can transform, filter, fan out, retry, rate-
 
 ## Move data from A to B — inside your own service
 
-If you need to move data reliably between systems and you write code (Rust, Python, or Node), `mq-bridge` is a strong default. It is a **library you embed**, not a daemon or control plane you operate.
+If you need to move data or events reliably between systems and you write code (Rust, Python, or Node), `mq-bridge` is a strong default. It is a **library you embed**, not a daemon or control plane you operate.
 
 **Prefer not to write code?** [`mq-bridge-app`](https://github.com/marcomq/mq-bridge-app) runs the exact same engine as a **standalone, zero-code ETL service** configured entirely by **YAML or environment variables** — move data from A to B without writing a line. It ships a **Postman-style UI** to build, send, and inspect messages against a route, and can **import Postman collections and AsyncAPI documents** to scaffold routes and endpoints for you.
 
@@ -33,7 +35,9 @@ If you need to move data reliably between systems and you write code (Rust, Pyth
 *   **TLS everywhere, one config shape**: a single `TlsConfig` block (CA bundle, client cert/key for mTLS, insecure-skip) is reused across transports.
 *   **Self-hosted, no daemon**: generate config in the optional UI, paste it into your code, run it in-process. No hosted control plane, no separate scheduler.
 
-> **Throughput & footprint.** In our own benchmarks, the same engine — driven the zero-code way through [`mq-bridge-app`](https://github.com/marcomq/mq-bridge-app) — sustained ~267,000 rows/s copying 1,000,000 rows from Postgres to JSONL on commodity hardware at **~20 MiB peak RSS**, keeping it well in the range of dedicated data-movement tools. On a CSV→JSONL file conversion (1,000,000 mixed-type rows, ~116 MiB), it sustained **833,333 rows/s** at the same ~20 MiB, about **~43x faster** and **~22x leaner in memory** than Meltano (`tap-csv` → `target-jsonl`, ~19,500 rows/s / ~444 MiB). Full setup, methodology, and the exact parameters are in [`benches/ETL_BENCHMARKS.md`](benches/ETL_BENCHMARKS.md).
+> **Throughput & footprint.** In our own benchmarks, the same engine — driven the zero-code way through [`mq-bridge-app`](https://github.com/marcomq/mq-bridge-app) — on a CSV→JSONL file conversion (1,000,000 mixed-type rows, ~116 MiB) sustained **1,133,786 rows/s** at ~22 MiB, about **~58x faster** and **~20x leaner in memory** than Meltano (`tap-csv` → `target-jsonl`, ~19,500 rows/s / ~444 MiB). Full setup, methodology, and the exact parameters are in [`benches/ETL_BENCHMARKS.md`](benches/ETL_BENCHMARKS.md).
+
+> **Kafka → file.** In a 1,000,000-row relay with the default file format and no transform, the engine was up to **80% faster than Sea Streamer**. The [`mq-bridge-app` benchmark](https://github.com/marcomq/mq-bridge-app/tree/dev/benches/etl) contains the reproducible helper and native file-format caveats.
 
 
 ## Language Bindings
@@ -306,9 +310,14 @@ server-side (default one-hour window) — `insert_deduplication_token` lets you
 mq-bridge does not set one, so rely on `ReplacingMergeTree` for logical dedup and treat block-level
 dedup only as retry-safety.
 
-**Postgres CDC — deterministic id + `postgres.key`.** The `postgres_cdc` source already resumes from a
-durable LSN checkpoint (no re-delivery on clean restart). For the crash window (row written, then a
-crash before the checkpoint flush) it is at-least-once, so make the sink idempotent. Each change event
+**Postgres CDC — deterministic id + `postgres.key`.** The `postgres_cdc` source resumes from the slot's
+durable `confirmed_flush_lsn`. In-band standby feedback is asynchronous and is *not* flushed when the
+stream stops, so the last acks are made durable by the consumer's `Drop`, which stops the stream and
+advances the slot synchronously. Re-delivery is therefore avoided only on a restart that actually runs
+that teardown — a host process that exits without dropping the route (or one on a current-thread Tokio
+runtime, where the blocking advance is skipped) replays everything since the last asynchronous feedback
+tick. Set `checkpoint_store` to a `file://` path for a second, per-ack durable position that survives
+teardown regardless. Treat the source as at-least-once and make the sink idempotent. Each change event
 carries the full row (so the primary key is in the payload), `postgres.lsn` (a monotonic version),
 `postgres.operation`/`schema`/`table`, and — when the table has a primary key / replica identity —
 `postgres.key` (the key value). The event's `message_id` is a stable hash of `schema.table + key +
@@ -359,11 +368,56 @@ orders_upsert_branch:
 duplicate to detect. Left unwrapped by `request`, the tagged message is returned as the route's
 response as usual.
 
+**Files & object storage — `idempotency`.** A filesystem has no unique constraint, so the `file` and
+`object_store` sinks get replay safety a different way: **deterministic names plus covered-range
+recovery**. Set `idempotency: true` and the sink stops writing UUID-named objects. Instead it groups
+each batch into runs of consecutive source positions and writes one immutable part per run, named
+for the range it covers:
+
+```yaml
+kafka_to_s3:
+  input:
+    kafka: { topic: "orders", url: "localhost:9092", source_metadata: true }
+  output:
+    object_store:
+      url: "s3://my-bucket/orders"
+      idempotency: true          # → s3://my-bucket/orders/part-orders-0-0-1023.jsonl
+```
+
+On startup the sink lists what is already there, parses the ranges out of the part names, and drops
+any incoming record whose position falls inside one. Filtering is **per record, not per batch**, so
+batch boundaries are free to differ across restarts — that is what makes it work without a
+checkpoint protocol. Local files stage to a temp name, `fsync`, then `rename` (atomic within a
+filesystem); object stores have no atomic rename, so a single PUT under the final name *is* the
+commit.
+
+This needs a replayable source position, which today means **Kafka** (topic/partition/offset) or
+**`postgres_cdc`** (commit LSN plus in-transaction ordinal). A route with an idempotent output turns
+`source_metadata` on for its input automatically; set it explicitly only when you want the
+`mqb.src.*` keys for something else. Any other input is rejected when the route starts. NATS, AMQP
+and MongoDB CDC also accept `source_metadata` and emit provenance keys, but a subject or routing key
+is not a replayable offset, so they cannot drive an idempotent sink.
+
+For the `file` sink, `idempotency: true` changes what `path` means: it is the directory that receives
+the part files, not the file that is appended to. The sink creates it on startup, so pointing it at an
+existing regular file fails there with "Failed to create idempotent file sink directory".
+
+`compression` and `encryption` work as usual.
+
+Current limits, all of which the sink rejects or logs rather than silently mishandling:
+
+*   No `csv` (each part would need its own header row).
+*   `date_partition` is ignored — parts are written flat under the prefix, since the name already
+    carries the range. Logged at startup.
+*   **One part file per contiguous run per batch.** There is no size-based rolling yet, so a large
+    backfill produces many small files (~1000 per 1M rows at `batch_size: 1024`). For `postgres_cdc`
+    this is per-transaction, so a high-commit-rate stream produces one small file per commit. Rolling
+    would require buffering across batches the route has already acked, which is not safe today.
 
 ### Cloud Object Storage (S3 / GCS / Azure)
 The `object_store` endpoint (alias `s3`) reads and writes cloud object stores — Amazon S3, Google Cloud Storage, Azure Blob, Cloudflare R2, and anything else the [`object_store`](https://crates.io/crates/object_store) crate speaks — behind the same `receive_batch` / `send_batch` API. Enable it with the `object-store` feature. Credentials and backend options are read from the process environment (`AWS_ACCESS_KEY_ID`, `AWS_ENDPOINT`, `AWS_REGION`, `GOOGLE_SERVICE_ACCOUNT`, `AZURE_STORAGE_ACCOUNT`, ...); the URL scheme picks the backend (`s3://`, `gs://`, `az://`).
 
-*   **As a sink**, each flushed batch is encoded with the same file endpoint formats (`normal` JSONL, `json`, `text`, `raw`) and written as **one immutable object** at `<prefix>/[YYYY/MM/DD/]<uuidv7>.<ext>`. Objects are write-once — nothing is appended or mutated. The uuidv7 name already sorts by write time; the optional `date_partition` prefix (on by default, derived from that same id's timestamp) is a readability / lifecycle-rule convenience.
+*   **As a sink**, each flushed batch is encoded with the same file endpoint formats (`normal` JSONL, `json`, `text`, `raw`) and written as **one immutable object** at `<prefix>/[YYYY/MM/DD/]<uuidv7>.<ext>`. Objects are write-once — nothing is appended or mutated. The uuidv7 name already sorts by write time; the optional `date_partition` prefix (on by default, derived from that same id's timestamp) is a readability / lifecycle-rule convenience. This naming applies only while `idempotency` is off — with `idempotency: true` the parts are instead named for the source range they cover and written flat under the prefix, and `date_partition` is ignored (see "Files & object storage — `idempotency`" above).
 *   **As a source**, objects under the prefix are listed in key order, fetched whole, split on the delimiter, and emitted. Progress is a durable cursor holding the last fully-acked object key: set `cursor_id` and an external `checkpoint_store` (`file://`, `s3://`, `postgres://`, `mongodb://`) so a restart resumes without re-emitting. Objects are **never deleted or rewritten** — resume is non-destructive and at-least-once at object granularity (a nacked batch is redelivered; the cursor only advances once an object is fully acked). `csv` is supported on the source only.
 
 ```yaml
@@ -391,6 +445,26 @@ replay_from_s3:
 > always write the `{message_id, payload, metadata}` wrapper, so the message id survives the round
 > trip. Use `format: raw` to write payloads verbatim (bare documents, no wrapper). Applies to both
 > `file` and `object_store`.
+>
+> A **source** on those formats expects that same wrapper, `message_id` included. A line that is
+> valid JSON but not the wrapper — a hand-written fixture with only `payload` and `metadata`, say —
+> is not decomposed: the whole line becomes the payload and the line's own `metadata` is discarded.
+> The reader logs a warning naming the wrapper when this happens. For plain JSON lines, use
+> `format: raw`.
+>
+> **Payload encoding in the wrapper.** A UTF-8 payload is written as a plain JSON string under
+> `payload`. A binary one (compressed, encrypted, Protobuf, …) is base64-encoded under a separate
+> `payload_base64` field; the two are mutually exclusive, as in the
+> [CloudEvents JSON format](https://github.com/cloudevents/spec/blob/main/cloudevents/formats/json-format.md).
+>
+> ```json
+> {"message_id":"019f9b12-d786-7ebe-a7ec-a1aa71bc47ae","payload":"{\"order_id\":7}"}
+> {"message_id":"019f9b12-d78a-7c01-b0f4-2f0f4d6a1c33","payload_base64":"KLUv/SBOAQAA"}
+> ```
+>
+> Sources still read the older byte-array form (`"payload":[123,34,…]`), so existing files keep
+> working. Only the reverse is a break: a **binary** record written by this version is not readable
+> by an older mq-bridge. Text records are compatible in both directions.
 
 ### Response Endpoint
 The `response` output endpoint sends a reply back to the original requester. This is useful for synchronous request-reply flows, for example HTTP-to-NATS-to-HTTP. Use `response: {}` as the output endpoint configuration.