mq-bridge-py-basic 0.3.9__tar.gz → 0.3.10__tar.gz

mq_bridge_py_basic-0.3.10/.github/dependabot.yml

@@ -0,0 +1,11 @@
+# To get started with Dependabot version updates, you'll need to specify which
+# package ecosystems to update and where the package manifests are located.
+# Please see the documentation for all configuration options:
+# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
+
+version: 2
+updates:
+  - package-ecosystem: "cargo" # See documentation for possible values
+    directory: "/" # Location of package manifests
+    schedule:
+      interval: "weekly"

{mq_bridge_py_basic-0.3.9 → mq_bridge_py_basic-0.3.10}/.github/workflows/ci.yml

@@ -305,3 +305,30 @@ jobs:
       
       - name: Build documentation (deny warnings)
         run: RUSTDOCFLAGS="-D warnings" cargo doc --all-features --no-deps
+
+  # Supply chain: licenses, banned crates, source registries, RustSec advisories
+  supply-chain:
+    name: Supply Chain
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+
+      # Hard gate: an unapproved license, a banned crate or a non-crates.io
+      # source is always a real, actionable problem in this repo.
+      - name: Licenses / bans / sources
+        uses: EmbarkStudios/cargo-deny-action@v2
+        with:
+          command: check bans licenses sources
+
+      # Informational: every current advisory is in a transitive dependency we
+      # cannot upgrade ourselves, and RustSec updates daily. SECURITY.md records
+      # the per-feature reachability analysis; deny.toml carries the reasoning.
+      - name: Advisories (informational)
+        continue-on-error: true
+        uses: EmbarkStudios/cargo-deny-action@v2
+        with:
+          command: check advisories

mq_bridge_py_basic-0.3.10/CHANGELOG.md

@@ -0,0 +1,76 @@
+# Changelog
+
+All notable changes to `mq-bridge`. Newest first.
+
+## 0.3.10
+
+
+### Changed
+
+- DLQ and output middleware now run around the publish step, not inside the handler, so a
+  handler failure is no longer retried by the output chain. The tradeoff: a handler failure
+  now skips the output middleware entirely, so `dlq` on the output endpoint cannot capture
+  it — only publish failures reach the DLQ. Put a `dlq` on the input endpoint to catch
+  handler failures.
+- Errors carry their full cause chain instead of only the outermost context.
+- `deduplication` on an **output** endpoint is now a startup error instead of a warning and a
+  silent no-op. Move it to the route's input endpoint.
+- `limiter` paces a single `send_batch` by its own message count, so one large batch is no
+  longer a free burst. Sustained throughput is unchanged.
+- A `message_id` that is neither a UUID nor a `u128` is hashed to a stable id rather than
+  making the whole JSON envelope unparseable.
+
+### Performance
+
+- **Kafka consumer**: a long-lived prefetch task reads librdkafka continuously into a bounded
+  channel, instead of rebuilding the stream inside every `receive_batch`. librdkafka only keeps
+  requesting records while its queue is drained, so every pause the pipeline took — a transform,
+  a slow sink — was also a pause in fetching, and the fetch rate collapsed to well under what the
+  broker could serve. Batch offsets are also recorded once per partition rather than once per
+  message, which was O(n²) in the batch and allocated two `CString`s per record.
+  Kafka → transform → file: **192,854 → 824,983 rows/s** on the 1M-row, four-partition
+  benchmark, from 0.35x to 1.51x Arroyo on identical output. A 16,384-message passthrough
+  batch went from 11.5s to 2.2s, and from 10.4s to 0.9s of CPU.
+- **Postgres / sqlx**: `test_before_acquire(false)` on the pool, zero-copy row encoding via
+  a prebuilt `JsonRowSchema`, and prebuilt first/next page queries on the cursor path.
+- **Command handler**: `send_batch` no longer loops per-message `send()` (≈8x on batched routes).
+- **File**: single-pass byte-array decode, faster CSV writes, compression sniffing.
+- **Deduplication**: the two-phase reserve/commit no longer writes to the store twice per
+  message. Reservations are held in memory — sled takes an exclusive file lock on its directory,
+  so a claim only ever has to be visible to this process — leaving one write per message, on
+  commit.
+
+### Fixed
+
+- `insert_query` batch inserts dropped anything but a bare token from the `VALUES` tuple, so
+  `decode(${payload:x}, 'base64')` and casts were silently discarded — a binary (`bytea`)
+  column could not be written. The batch path now keeps the user's SQL and falls back to
+  iterative inserts when the tuple contains an expression.
+- A payload string containing an embedded NUL byte was dropped by the driver and stored as
+  SQL `NULL` while the route reported success. It is now rejected as non-retryable.
+- Database errors were rendered twice, because `sqlx` errors already display their own
+  source. Cause chains now skip a link an earlier one already contains.
+- `cookie_jar`'s `inject_metadata` resolved only a bare stored name, so the namespaced
+  `cookie.<name>` / `value.<name>` spelling that `export_metadata_prefix` reports back
+  injected nothing. Both spellings now work.
+- MQTT publishes are confirmed by PUBACK/PUBCOMP and guarded against session resets, which
+  took chaos-test message loss to zero.
+- Postgres CDC advances its replication slot durably on shutdown instead of leaving the
+  feedback unflushed.
+- `deduplication` could silently drop a message after a crash. The reservation was written to
+  the store *before* the message was processed, so a redelivery arriving within the 5s pending
+  TTL was classified as a duplicate and acked — without ever having been written to the sink.
+  Reservations are now in-memory and die with the process, so the redelivery is reprocessed.
+- A Kafka record with no value — a tombstone, which is ordinary traffic on a compacted topic —
+  failed the whole batch. Its offset was never committed, so the route reconnected onto the same
+  record forever: one tombstone wedged the consumer permanently, and a compacted topic could not
+  be consumed at all. Tombstones now arrive as an empty payload flagged `mqb.kafka.tombstone`.
+- `--drain` on a Kafka source could report success having copied only part of a topic, or none
+  of it. An idle wait was taken to mean "source exhausted", but an idle channel means nothing on
+  its own: before the first fetch lands, or in an ordinary gap between fetches, it looks exactly
+  the same as the end of the data. The shorter the idle timeout the more went missing — at 1ms a
+  1,000,000-row topic landed 0 rows, and the copy still exited successfully. A drain now
+  completes only once every assigned partition has reached the offset it held when the drain
+  began, and lands all 1,000,000 rows at every idle timeout including 0.
+- Drain no longer hangs on an empty source, and reconnect attempts are bounded.
+- CSV reader handles quoted newlines; file endpoints fail fast on an unopenable path.

{mq_bridge_py_basic-0.3.9 → mq_bridge_py_basic-0.3.10}/Cargo.lock

@@ -3348,7 +3348,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge"
-version = "0.3.9"
+version = "0.3.10"
 dependencies = [
  "aes-gcm",
  "anyhow",
@@ -3429,7 +3429,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge-bindings-common"
-version = "0.3.9"
+version = "0.3.10"
 dependencies = [
  "anyhow",
  "fast-uuid-v7",
@@ -3443,7 +3443,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge-node"
-version = "0.3.9"
+version = "0.3.10"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -3465,7 +3465,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge-py"
-version = "0.3.9"
+version = "0.3.10"
 dependencies = [
  "anyhow",
  "async-trait",

{mq_bridge_py_basic-0.3.9 → mq_bridge_py_basic-0.3.10}/Cargo.toml

@@ -16,7 +16,7 @@ members = ["bindings-common", "python/mq-bridge-py"]
 resolver = "2"
 
 [workspace.package]
-version = "0.3.9"
+version = "0.3.10"
 
 [workspace.dependencies]
 anyhow = "1.0"
@@ -29,6 +29,12 @@ serde_yaml_ng = "0.10"
 tokio = "1.34"  # `Receiver::recv_many` (HTTP source batch drain) needs 1.34+.
 tracing = "0.1"
 
+[profile.dev.package."*"]
+debug = false
+
+[profile.dev.build-override]
+debug = false
+
 [profile.release]
 lto = "thin"
 strip = "symbols"
@@ -138,7 +144,8 @@ flate2 = { version = "1.1", optional = true }
 lz4_flex = { version = "0.14", optional = true, default-features = false, features = ["frame"] }
 # zstd (bundled libzstd via zstd-sys); Decoder handles concatenated frames like gzip/lz4.
 zstd = { version = "0.13", optional = true }
-base64 = { version = "0.23", optional = true }
+# Non-optional: `CanonicalMessage`'s JSON form encodes binary payloads as base64.
+base64 = { version = "0.23" }
 
 # AEAD ciphers for the `encryption` feature (payload middleware + at-rest file/object).
 chacha20poly1305 = { version = "0.11", optional = true }
@@ -229,7 +236,7 @@ dedup = ["sled"]
 compression = ["dep:flate2", "dep:lz4_flex", "dep:zstd"]
 # AEAD payload encryption: the `encryption` middleware and the at-rest
 # `encryption` field on the file/object_store endpoints.
-encryption = ["dep:chacha20poly1305", "dep:aes-gcm", "dep:base64"]
+encryption = ["dep:chacha20poly1305", "dep:aes-gcm"]
 metrics = ["dep:metrics"]
 test-utils = ["dep:tracing-appender", "dep:tracing-subscriber", "dep:serde_yaml_ng"]
 # Opt-in produce/consume-only isolation benchmarks (throwaway diagnostics, off by default)
@@ -254,9 +261,9 @@ amqp = ["lapin", "rustls", "rustls-pemfile", "url"]
 nats = ["async-nats", "rustls", "rustls-pemfile"]
 mongodb = ["dep:mongodb", "url"]
 mqtt = ["rumqttc", "rustls", "tokio-rustls", "rustls-pemfile", "url"]
-http = ["dep:arc-swap", "hyper", "hyper-util", "hyper-rustls", "http-body-util", "dep:http-body", "rustls", "tokio-rustls", "rustls-pemfile", "webpki-roots", "h2", "dep:flate2", "dep:lz4_flex", "dep:zstd", "dep:base64"]
+http = ["dep:arc-swap", "hyper", "hyper-util", "hyper-rustls", "http-body-util", "dep:http-body", "rustls", "tokio-rustls", "rustls-pemfile", "webpki-roots", "h2", "dep:flate2", "dep:lz4_flex", "dep:zstd"]
 websocket = ["dep:tokio-websockets"]
-aws = ["aws-config", "aws-sdk-sqs", "aws-sdk-sns", "dep:base64"]
+aws = ["aws-config", "aws-sdk-sqs", "aws-sdk-sns"]
 # Runtime-loaded IBM MQ client via dlopen: builds with no SDK present; the IBM
 # redistributable client is only needed at runtime, and only if an ibm-mq
 # endpoint is actually used.
@@ -283,10 +290,12 @@ object-store = ["dep:object_store", "url"]
 # Postgres logical-replication CDC (pgoutput). Reuses the sqlx dependency for the
 # slot-lifecycle control plane; the streaming wire protocol is pgwire-replication.
 # Needs Rust 1.88 (pgwire-replication MSRV), like the amqp feature gate.
-postgres-cdc = ["dep:pgwire-replication", "dep:byteorder", "sqlx", "dep:base64", "url", "dep:percent-encoding"]
+postgres-cdc = ["dep:pgwire-replication", "dep:byteorder", "sqlx", "url", "dep:percent-encoding"]
 
 [dev-dependencies]
 config = { version = "0.15" }
+# `start_paused` — lets a test cross a 30s health window without waiting 30s.
+tokio = { workspace = true, features = ["test-util"] }
 serde_yaml_ng = "0.10"
 tempfile = "3.27"
 chrono = "0.4"

{mq_bridge_py_basic-0.3.9 → mq_bridge_py_basic-0.3.10}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mq-bridge-py-basic
-Version: 0.3.9
+Version: 0.3.10
 Classifier: Development Status :: 3 - Alpha
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3
@@ -232,8 +232,8 @@ that ack each batch individually (NATS JetStream, AMQP, MQTT) accept any order.
 > per source position (Kafka `partition:offset`, NATS `stream_sequence`, AMQP
 > delivery tag) and makes a natural primary key. Source cursor fields are also
 > available in `message.metadata` (`mqb.src.kafka_topic`/`mqb.src.kafka_offset`, `mqb.src.nats_subject`/`mqb.src.nats_stream_sequence`,
-> `mqb.src.amqp_routing_key`/`mqb.src.amqp_delivery_tag`) when you opt in by setting
-> the `MQB_SOURCE_METADATA=1` environment variable (off by default).
+> `mqb.src.amqp_routing_key`/`mqb.src.amqp_delivery_tag`) when its Kafka, NATS, or
+> AMQP source config sets `source_metadata: true` (off by default).
 >
 > **Ack deadlines vs slow loads.** JetStream `AckWait` (default 30s), AMQP
 > prefetch/consumer-timeout and MQTT inflight windows each bound how long a batch

{mq_bridge_py_basic-0.3.9 → mq_bridge_py_basic-0.3.10}/README.md

@@ -6,6 +6,8 @@
 ![Linux](https://img.shields.io/badge/Linux-supported-green?logo=linux)
 ![Windows](https://img.shields.io/badge/Windows-supported-green?logo=windows)
 ![macOS](https://img.shields.io/badge/macOS-supported-green?logo=apple)
+[![Supply chain](https://img.shields.io/badge/supply%20chain-cargo--deny-blue)](deny.toml)
+[![Security](https://img.shields.io/badge/security-advisory%20analysis-brightgreen)](SECURITY.md)
 [![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 
 
@@ -21,7 +23,7 @@ It is not only a forwarder. A route can transform, filter, fan out, retry, rate-
 
 ## Move data from A to B — inside your own service
 
-If you need to move data reliably between systems and you write code (Rust, Python, or Node), `mq-bridge` is a strong default. It is a **library you embed**, not a daemon or control plane you operate.
+If you need to move data or events reliably between systems and you write code (Rust, Python, or Node), `mq-bridge` is a strong default. It is a **library you embed**, not a daemon or control plane you operate.
 
 **Prefer not to write code?** [`mq-bridge-app`](https://github.com/marcomq/mq-bridge-app) runs the exact same engine as a **standalone, zero-code ETL service** configured entirely by **YAML or environment variables** — move data from A to B without writing a line. It ships a **Postman-style UI** to build, send, and inspect messages against a route, and can **import Postman collections and AsyncAPI documents** to scaffold routes and endpoints for you.
 
@@ -35,6 +37,8 @@ If you need to move data reliably between systems and you write code (Rust, Pyth
 
 > **Throughput & footprint.** In our own benchmarks, the same engine — driven the zero-code way through [`mq-bridge-app`](https://github.com/marcomq/mq-bridge-app) — on a CSV→JSONL file conversion (1,000,000 mixed-type rows, ~116 MiB) sustained **1,133,786 rows/s** at ~22 MiB, about **~58x faster** and **~20x leaner in memory** than Meltano (`tap-csv` → `target-jsonl`, ~19,500 rows/s / ~444 MiB). Full setup, methodology, and the exact parameters are in [`benches/ETL_BENCHMARKS.md`](benches/ETL_BENCHMARKS.md).
 
+> **Kafka → file.** In a 1,000,000-row relay with the default file format and no transform, the engine was up to **80% faster than Sea Streamer**. The [`mq-bridge-app` benchmark](https://github.com/marcomq/mq-bridge-app/tree/dev/benches/etl) contains the reproducible helper and native file-format caveats.
+
 
 ## Language Bindings
 
@@ -364,11 +368,56 @@ orders_upsert_branch:
 duplicate to detect. Left unwrapped by `request`, the tagged message is returned as the route's
 response as usual.
 
+**Files & object storage — `idempotency`.** A filesystem has no unique constraint, so the `file` and
+`object_store` sinks get replay safety a different way: **deterministic names plus covered-range
+recovery**. Set `idempotency: true` and the sink stops writing UUID-named objects. Instead it groups
+each batch into runs of consecutive source positions and writes one immutable part per run, named
+for the range it covers:
+
+```yaml
+kafka_to_s3:
+  input:
+    kafka: { topic: "orders", url: "localhost:9092", source_metadata: true }
+  output:
+    object_store:
+      url: "s3://my-bucket/orders"
+      idempotency: true          # → s3://my-bucket/orders/part-orders-0-0-1023.jsonl
+```
+
+On startup the sink lists what is already there, parses the ranges out of the part names, and drops
+any incoming record whose position falls inside one. Filtering is **per record, not per batch**, so
+batch boundaries are free to differ across restarts — that is what makes it work without a
+checkpoint protocol. Local files stage to a temp name, `fsync`, then `rename` (atomic within a
+filesystem); object stores have no atomic rename, so a single PUT under the final name *is* the
+commit.
+
+This needs a replayable source position, which today means **Kafka** (topic/partition/offset) or
+**`postgres_cdc`** (commit LSN plus in-transaction ordinal). A route with an idempotent output turns
+`source_metadata` on for its input automatically; set it explicitly only when you want the
+`mqb.src.*` keys for something else. Any other input is rejected when the route starts. NATS, AMQP
+and MongoDB CDC also accept `source_metadata` and emit provenance keys, but a subject or routing key
+is not a replayable offset, so they cannot drive an idempotent sink.
+
+For the `file` sink, `idempotency: true` changes what `path` means: it is the directory that receives
+the part files, not the file that is appended to. The sink creates it on startup, so pointing it at an
+existing regular file fails there with "Failed to create idempotent file sink directory".
+
+`compression` and `encryption` work as usual.
+
+Current limits, all of which the sink rejects or logs rather than silently mishandling:
+
+*   No `csv` (each part would need its own header row).
+*   `date_partition` is ignored — parts are written flat under the prefix, since the name already
+    carries the range. Logged at startup.
+*   **One part file per contiguous run per batch.** There is no size-based rolling yet, so a large
+    backfill produces many small files (~1000 per 1M rows at `batch_size: 1024`). For `postgres_cdc`
+    this is per-transaction, so a high-commit-rate stream produces one small file per commit. Rolling
+    would require buffering across batches the route has already acked, which is not safe today.
 
 ### Cloud Object Storage (S3 / GCS / Azure)
 The `object_store` endpoint (alias `s3`) reads and writes cloud object stores — Amazon S3, Google Cloud Storage, Azure Blob, Cloudflare R2, and anything else the [`object_store`](https://crates.io/crates/object_store) crate speaks — behind the same `receive_batch` / `send_batch` API. Enable it with the `object-store` feature. Credentials and backend options are read from the process environment (`AWS_ACCESS_KEY_ID`, `AWS_ENDPOINT`, `AWS_REGION`, `GOOGLE_SERVICE_ACCOUNT`, `AZURE_STORAGE_ACCOUNT`, ...); the URL scheme picks the backend (`s3://`, `gs://`, `az://`).
 
-*   **As a sink**, each flushed batch is encoded with the same file endpoint formats (`normal` JSONL, `json`, `text`, `raw`) and written as **one immutable object** at `<prefix>/[YYYY/MM/DD/]<uuidv7>.<ext>`. Objects are write-once — nothing is appended or mutated. The uuidv7 name already sorts by write time; the optional `date_partition` prefix (on by default, derived from that same id's timestamp) is a readability / lifecycle-rule convenience.
+*   **As a sink**, each flushed batch is encoded with the same file endpoint formats (`normal` JSONL, `json`, `text`, `raw`) and written as **one immutable object** at `<prefix>/[YYYY/MM/DD/]<uuidv7>.<ext>`. Objects are write-once — nothing is appended or mutated. The uuidv7 name already sorts by write time; the optional `date_partition` prefix (on by default, derived from that same id's timestamp) is a readability / lifecycle-rule convenience. This naming applies only while `idempotency` is off — with `idempotency: true` the parts are instead named for the source range they cover and written flat under the prefix, and `date_partition` is ignored (see "Files & object storage — `idempotency`" above).
 *   **As a source**, objects under the prefix are listed in key order, fetched whole, split on the delimiter, and emitted. Progress is a durable cursor holding the last fully-acked object key: set `cursor_id` and an external `checkpoint_store` (`file://`, `s3://`, `postgres://`, `mongodb://`) so a restart resumes without re-emitting. Objects are **never deleted or rewritten** — resume is non-destructive and at-least-once at object granularity (a nacked batch is redelivered; the cursor only advances once an object is fully acked). `csv` is supported on the source only.
 
 ```yaml
@@ -396,6 +445,26 @@ replay_from_s3:
 > always write the `{message_id, payload, metadata}` wrapper, so the message id survives the round
 > trip. Use `format: raw` to write payloads verbatim (bare documents, no wrapper). Applies to both
 > `file` and `object_store`.
+>
+> A **source** on those formats expects that same wrapper, `message_id` included. A line that is
+> valid JSON but not the wrapper — a hand-written fixture with only `payload` and `metadata`, say —
+> is not decomposed: the whole line becomes the payload and the line's own `metadata` is discarded.
+> The reader logs a warning naming the wrapper when this happens. For plain JSON lines, use
+> `format: raw`.
+>
+> **Payload encoding in the wrapper.** A UTF-8 payload is written as a plain JSON string under
+> `payload`. A binary one (compressed, encrypted, Protobuf, …) is base64-encoded under a separate
+> `payload_base64` field; the two are mutually exclusive, as in the
+> [CloudEvents JSON format](https://github.com/cloudevents/spec/blob/main/cloudevents/formats/json-format.md).
+>
+> ```json
+> {"message_id":"019f9b12-d786-7ebe-a7ec-a1aa71bc47ae","payload":"{\"order_id\":7}"}
+> {"message_id":"019f9b12-d78a-7c01-b0f4-2f0f4d6a1c33","payload_base64":"KLUv/SBOAQAA"}
+> ```
+>
+> Sources still read the older byte-array form (`"payload":[123,34,…]`), so existing files keep
+> working. Only the reverse is a break: a **binary** record written by this version is not readable
+> by an older mq-bridge. Text records are compatible in both directions.
 
 ### Response Endpoint
 The `response` output endpoint sends a reply back to the original requester. This is useful for synchronous request-reply flows, for example HTTP-to-NATS-to-HTTP. Use `response: {}` as the output endpoint configuration.

{mq_bridge_py_basic-0.3.9 → mq_bridge_py_basic-0.3.10}/REFERENCE.md

@@ -43,7 +43,15 @@ Writing with `[compression, encryption]` produces `compress(encrypt(payload))`;
 given that same list would try to decrypt first and fail. The reading route must say
 `[encryption, compression]`. The lists mirror — they are not copied.
 
-> This is asserted by `route::tests::test_dlq_and_retry_batch_integration`,
+**A route handler sits outside every output middleware**, so it runs **once** per message
+and the middlewares act on what it returned. In particular `retry` re-attempts only the
+publish, never the handler — a handler with a side effect fires once however many times the
+sink is retried. The trade: a `dlq` cannot capture a handler failure, only a send failure;
+a handler error propagates to the route and is reported there.
+
+> This is asserted by `route::tests::test_retryable_handler_error_is_not_retried_by_output_middleware`
+> (the handler runs once and `retry` does not re-run it),
+> `route::tests::test_dlq_and_retry_batch_integration`,
 > `middleware::transform::tests::test_rejected_message_reaches_the_dlq_through_the_config_wiring`,
 > and `reference_docs_test::publisher_middleware_wraps_last_entry_outermost`, and is
 > documented on `apply_middlewares_to_publisher` in `src/middleware/mod.rs`.
@@ -88,9 +96,10 @@ middlewares:
 **Putting a middleware on the wrong side behaves in two different ways**, so check the table
 above rather than assuming:
 
-- `deduplication` on an output, and `dlq` / `retry` on an input, log a warning and are
-  skipped. The route still starts.
-- `weak_join` on an output is a **hard startup error** (`Unsupported publisher middleware`).
+- `dlq` / `retry` on an input log a warning and are skipped. The route still starts.
+- `deduplication` and `weak_join` on an output are **hard startup errors**. Deduplication
+  cannot work on the publish side, and silently starting an un-deduplicated route is worse
+  than refusing to start.
 
 A middleware whose feature is not compiled in (`deduplication` without `dedup`, `metrics`
 without `metrics`) is likewise a startup error, not a silent no-op.
@@ -131,16 +140,20 @@ Sends permanently-failed messages to a separate endpoint instead of failing the
 ```
 
 Captures `NonRetryable` failures and `Retryable` ones whose retries are exhausted. Connection
-errors are **not** dead-lettered — they propagate so the route can reconnect. The DLQ endpoint
+errors are **not** dead-lettered — they propagate so the route can reconnect. Nor are handler
+failures: the handler runs outside the middlewares (see [Ordering](#ordering--read-this-before-combining-middleware)),
+so a `dlq` only ever sees what failed on the way to the sink. The DLQ endpoint
 is a full endpoint, so it can itself have middleware. If the DLQ send fails with a connection
 error that error propagates rather than silently dropping the message.
 
 **Without a `dlq` middleware**, a message that fails permanently — a data/type
-error the sink rejects, a poison payload a handler rejects — is logged at `error` level and
+error the sink rejects — is logged at `error` level and
 **dropped**, and the route keeps processing the rest of the batch. `dlq` is the only retention
 mechanism: `retry` alone does not retain a permanently-failed message nor prevent it from being
-dropped — it only re-attempts retryable/connection errors, then hands a still-failing message on
-to be dropped (or to a following `dlq`). This tolerate-and-continue
+dropped — it only re-attempts `Retryable` errors (a connection error is passed straight through
+for the route to reconnect on, not retried), then hands a still-failing message on to be dropped
+(or to a following `dlq`). This is why a sink that fails with a *connection* error never reaches
+its `dlq`, whether it is a route's sole output or one leg of a `fanout`. This tolerate-and-continue
 policy keeps one bad message from halting the whole stream, but it means a *systematic* failure
 (e.g. every row hitting a column-type mismatch) drains the input while committing nothing and
 still ends `completed`. Add a `dlq` to capture the failures for inspection/replay, or watch the
@@ -161,11 +174,17 @@ validation — over a single parse. Input and output.
 | `schema_file` | path to a schema file | – |
 | `coerce` | bool | `true` |
 | `apply_defaults` | bool | `true` |
+| `coerce_empty_as_null` | bool | `false` |
 | `on_error` | `reject` \| `pass_through` | `reject` |
 
 `schema` and `schema_file` are mutually exclusive. A mapping rule is either a bare path
 string or `{ path, default, required }`.
 
+`schema` must be a JSON *object*, not a string containing one. A flat `key=value` middleware
+syntax (such as the `|transform?schema=…` form in a connection URI) can only pass strings, so
+it cannot express `schema` or any mapping rule beyond a bare path — use `schema_file`, or move
+the route into a config file.
+
 ```yaml middleware
 - transform:
     mapping:
@@ -185,6 +204,26 @@ else is ignored, so an existing fuller schema can be used as-is. Coercions are l
 lossless ones: `string → integer`, `string → number`, `string → boolean` (`true`/`false`/`1`/`0`),
 `number → string`.
 
+#### Empty strings
+
+CSV and many SQL exports spell "no value" as an empty string. `coerce_empty_as_null: true`
+reads every `""` the schema visits as `null`, which is then handled like any other null —
+a `nullable` field keeps it, a `default` replaces it:
+
+```yaml middleware
+- transform:
+    coerce_empty_as_null: true
+    schema:
+      type: object
+      properties:
+        note: { type: string, nullable: true }
+        tier: { type: string, default: standard }
+```
+
+`note: ""` arrives as `null` and `tier: ""` as `"standard"`. A field that is neither
+nullable nor defaulted is rejected, naming the coercion. Only fields the schema declares are
+affected; `" "` is not empty.
+
 #### Embedded JSON
 
 A field carrying a JSON document as a string is decoded by `contentMediaType`, following
@@ -311,6 +350,11 @@ Correlates messages by a metadata key and emits them as one joined message. Inpu
     on_timeout: discard
 ```
 
+`group_by` reads message **metadata** only — never the payload. A message that lacks the key
+falls into a shared `"default"` group, so a mistyped key or a source that never sets it joins
+unrelated messages instead of failing. If the value lives in the payload, lift it into metadata
+first (a `transform` mapping, or the source's own metadata options).
+
 Setting `branch_by` switches to branch mode, where `required` overrides `expected_count`.
 On timeout an incomplete group is either emitted partially (`fire`) or dropped (`discard`).
 Messages are acknowledged on receipt, so a crash before the group completes loses the
@@ -506,6 +550,15 @@ Deliberate fault injection for testing recovery paths. Input and output.
 non-retryable one — useful for exercising a `dlq`. Keep `enabled: false` in committed configs
 rather than deleting the block.
 
+On the **input** side, `json_format_error`/`nack` never call the real consumer at all — they
+substitute a synthetic message (or error) on every triggered `receive`. Leaving
+`trigger_on_message` unset means *every* poll is faulted, so the real source is never read and
+`exit_on_empty`/`--drain` never sees the empty batch it waits for — the route runs forever,
+manufacturing synthetic messages. Always set `trigger_on_message` to a specific count when
+testing a drain-mode route with input-side fault injection. And since `dlq`/`retry` on an input
+are no-ops (see above), pair an input-side fault with a real assertion on the *consumer's*
+recovery, not a `dlq`.
+
 The middleware block alone is **not** enough: fault injection is gated per route by
 `allow_fault_injection`, which defaults to `false`. Copying only the snippet above leaves the
 middleware inert (the route logs that it is disabled). A complete, working configuration:

mq_bridge_py_basic-0.3.10/SECURITY.md

@@ -0,0 +1,104 @@
+# Security Policy
+
+## Reporting a vulnerability
+
+Please report suspected vulnerabilities privately via
+[GitHub Security Advisories](https://github.com/marcomq/mq-bridge/security/advisories/new)
+rather than opening a public issue.
+
+## Supply-chain checks
+
+`cargo deny check` runs in CI ([deny.toml](deny.toml)). Licenses, bans, and
+sources are hard-gated; advisories run as an informational step, for the reasons
+explained below.
+
+## Known advisories and why they do not (mostly) affect mq-bridge
+
+Running `cargo audit` against this repository reports findings. **Most of them
+are not reachable in mq-bridge**, and the rest are narrower than the raw output
+suggests. This section records the analysis so you do not have to redo it.
+
+Two structural points come first, because they explain most of the gap between
+the tool output and the real exposure:
+
+1. **`cargo audit` and `cargo deny` read `Cargo.lock`, which is feature-blind.**
+   Almost every transport in this crate sits behind an optional Cargo feature.
+   A finding attributed to, say, the AWS SDK is not compiled at all in a
+   Kafka-only or Postgres-only build. The table below therefore states status
+   *per feature*.
+2. **A crate appearing in the tree is not the same as its vulnerable code being
+   used.** In one case below, a dependency pulls in an affected crate purely to
+   name a type in an error enum, while the actual cryptographic work happens in
+   a different, patched version.
+
+### Current status
+
+| Advisory | Crate / path | Feature | Status |
+| --- | --- | --- | --- |
+| [RUSTSEC-2026-0104](https://rustsec.org/advisories/RUSTSEC-2026-0104) | `rustls-webpki` via `rumqttc`, `aws-sdk-*` | `mqtt`, `aws` | **Not affected** — CRL code path never invoked |
+| [RUSTSEC-2026-0049](https://rustsec.org/advisories/RUSTSEC-2026-0049) | `rustls-webpki 0.102.8` via `rumqttc` | `mqtt` | **Not affected** — CRL code path never invoked |
+| [RUSTSEC-2026-0098](https://rustsec.org/advisories/RUSTSEC-2026-0098) | `rustls-webpki 0.102.8` via `rumqttc` | `mqtt` | **Not affected** — validation runs on patched `0.103.13` |
+| [RUSTSEC-2026-0099](https://rustsec.org/advisories/RUSTSEC-2026-0099) | `rustls-webpki 0.102.8` via `rumqttc` | `mqtt` | **Not affected** — validation runs on patched `0.103.13` |
+| [RUSTSEC-2026-0098](https://rustsec.org/advisories/RUSTSEC-2026-0098) | `rustls-webpki 0.101.7` via `aws-sdk-*` | `aws` | **Reachable**, narrow preconditions — see below |
+| [RUSTSEC-2026-0099](https://rustsec.org/advisories/RUSTSEC-2026-0099) | `rustls-webpki 0.101.7` via `aws-sdk-*` | `aws` | **Reachable**, narrow preconditions — see below |
+| [RUSTSEC-2025-0057](https://rustsec.org/advisories/RUSTSEC-2025-0057) | `fxhash` via `sled 0.34` | `sled`, `dedup` | Unmaintained only, no vulnerability |
+| [RUSTSEC-2024-0384](https://rustsec.org/advisories/RUSTSEC-2024-0384) | `instant` via `sled 0.34` → `parking_lot 0.11` | `sled`, `dedup` | Unmaintained only, no vulnerability |
+| [RUSTSEC-2025-0134](https://rustsec.org/advisories/RUSTSEC-2025-0134) | `rustls-pemfile` (direct + `rumqttc`) | `amqp`, `nats`, `mqtt`, `http` | Unmaintained only, no vulnerability |
+
+Each row applies only to the features listed for it: a build that enables none of
+a row's features does not compile that finding's code. `rustls-pemfile` is the
+broadest — it is pulled in by `amqp`, `nats`, `mqtt`, and `http` alike.
+
+### Detail: the `rustls-webpki` advisories
+
+These reach the dependency tree by two independent paths.
+
+**Path A — `rumqttc 0.25` → `rustls-webpki 0.102.8` (feature `mqtt`).**
+`rumqttc` declares this dependency but uses exactly one item from it: the
+`webpki::Error` variant in its own error enum (`src/tls.rs:42` is the only
+reference to `webpki` in the crate). All certificate validation on the MQTT path
+runs through `tokio-rustls 0.26` → `rustls 0.23` → `rustls-webpki 0.103.13`,
+which is patched for all four advisories. **Not affected.**
+
+**Path B — `aws-config` / `aws-sdk-sqs` / `aws-sdk-sns` →
+`aws-smithy-http-client` → legacy `rustls 0.21` → `rustls-webpki 0.101.7`
+(feature `aws`).** `aws-smithy-http-client` still selects its
+`legacy-rustls-ring` connector, so this is live code on the SQS/SNS path. It
+splits two ways:
+
+- **RUSTSEC-2026-0104 and RUSTSEC-2026-0049 (CRL handling): not affected.** This
+  code executes only when the application supplies a certificate revocation list
+  to a client-certificate verifier. The AWS SDK is an outbound HTTPS client and
+  never configures one. mq-bridge's only `WebPkiClientVerifier` — used for the
+  optional mTLS HTTP server in `src/endpoints/http/mod.rs` — runs on
+  `rustls 0.23` / `rustls-webpki 0.103.13` and is never given a CRL.
+- **RUSTSEC-2026-0098 and RUSTSEC-2026-0099 (name constraints): reachable.**
+  These fire during ordinary server-certificate chain validation, which the SQS
+  and SNS clients perform on every call. Exploitation requires a
+  name-constrained CA present in the trust store that issues certificates
+  outside its constraints — narrow, and not something mq-bridge can trigger or
+  mitigate itself. The fix depends on the AWS SDK migrating off its legacy
+  `rustls 0.21` connector. We do not claim to be unaffected here.
+
+### Detail: the unmaintained crates
+
+None of these carry a known vulnerability; RustSec flags them as unmaintained.
+
+- **`fxhash`, `instant`** arrive through `sled 0.34`, used by the deduplication
+  store. They resolve when `sled` updates its own dependencies.
+- **`rustls-pemfile`** is a direct dependency of mq-bridge as well as of
+  `rumqttc`. It is superseded by pemfile support in `rustls-pki-types`.
+  Migrating the direct dependency is tracked as a follow-up.
+
+### Keeping this current
+
+[deny.toml](deny.toml) carries these advisories as `[advisories].ignore`
+entries, each with its reasoning inline, so suppressions and justifications
+cannot drift apart. When an advisory is added, removed, or changes status,
+update both files together.
+
+The two tools do not report identically. `cargo deny` is configured with
+`unmaintained = "workspace"`, which surfaces unmaintained crates only when we
+depend on them directly — so the transitive `fxhash` and `instant` advisories
+need no entry there. `cargo audit` applies no such filter and does report them.
+Both are covered in the table above so either tool leads to the same answer.