mq-bridge-py-basic 0.3.3__tar.gz → 0.3.4__tar.gz

{mq_bridge_py_basic-0.3.3 → mq_bridge_py_basic-0.3.4}/.github/workflows/benchmark.yml

@@ -179,6 +179,11 @@ jobs:
     name: Collect Benchmark Results
     needs: [benchmark-streams, benchmark-database, benchmark-dockerless]
     runs-on: ubuntu-latest
+    # Serialize gh-pages pushes so overlapping runs don't race the Pages build
+    # (concurrent pushes make GitHub cancel the superseded deployment -> "failed").
+    concurrency:
+      group: gh-pages-publish
+      cancel-in-progress: false
     permissions:
       contents: write
       deployments: write
@@ -248,7 +253,24 @@ jobs:
           tool: 'cargo'
           output-file-path: output.txt
           github-token: ${{ secrets.GITHUB_TOKEN }}
+          # Push on both main and dev so dev commits appear in the benchmark graph.
+          # This mints a gh-pages deployment per run; the next step prunes old ones.
           auto-push: true
           alert-threshold: '40000%'
           comment-on-alert: true
           fail-on-alert: false
+
+      # Keep only the most recent gh-pages deployments so the environment's
+      # deployment list doesn't grow unbounded (one is minted per push above).
+      - name: Prune old gh-pages deployments
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          KEEP: 20
+        run: |
+          gh api "repos/${{ github.repository }}/deployments?environment=github-pages&per_page=100" --paginate -q '.[].id' \
+            | tail -n +$((KEEP + 1)) \
+            | while read -r id; do
+                gh api -X POST "repos/${{ github.repository }}/deployments/$id/statuses" -f state=inactive >/dev/null 2>&1 \
+                  || echo "::warning::failed to mark deployment $id inactive; attempting delete anyway"
+                gh api -X DELETE "repos/${{ github.repository }}/deployments/$id" >/dev/null 2>&1 && echo "pruned $id"
+              done

{mq_bridge_py_basic-0.3.3 → mq_bridge_py_basic-0.3.4}/CONFIGURATION.md

@@ -139,6 +139,22 @@ For example, to set the Kafka topic for the `kafka_to_nats` route:
 ```sh
 export MQB__KAFKA_TO_NATS__INPUT__KAFKA__TOPIC="my-other-topic"
 ```
+#### Postgres CDC example
+
+```yaml
+orders_cdc:
+  input:
+    postgres_cdc:
+      url: "postgres://user:pass@localhost:5432/app"
+      publication: "orders_pub"   # CREATE PUBLICATION orders_pub FOR TABLE orders;
+      slot_name: "mqb_orders"       # created if missing (permanent slot, resumable)
+  output:
+    nats:
+      subject: "orders.changes"
+      url: "nats://localhost:4222"
+```
+
+Each change arrives as a `CanonicalMessage` whose payload is the flat row and whose `postgres.operation` metadata marks the operation — the same convention as MongoDB CDC, so typed handlers work identically across both. The replication transport uses the published [`pgwire-replication`](https://crates.io/crates/pgwire-replication) crate.
 
 ### NATS JetStream Notes
 

{mq_bridge_py_basic-0.3.3 → mq_bridge_py_basic-0.3.4}/Cargo.lock

@@ -218,6 +218,18 @@ dependencies = [
  "tokio",
 ]
 
+[[package]]
+name = "async-compression"
+version = "0.4.42"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e79b3f8a79cccc2898f31920fc69f304859b3bd567490f75ebf51ae1c792a9ac"
+dependencies = [
+ "compression-codecs",
+ "compression-core",
+ "pin-project-lite",
+ "tokio",
+]
+
 [[package]]
 name = "async-executor"
 version = "1.14.0"
@@ -1097,6 +1109,23 @@ dependencies = [
  "tokio-util",
 ]
 
+[[package]]
+name = "compression-codecs"
+version = "0.4.38"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ce2548391e9c1929c21bf6aa2680af86fe4c1b33e6cea9ac1cfeec0bd11218cf"
+dependencies = [
+ "compression-core",
+ "flate2",
+ "memchr",
+]
+
+[[package]]
+name = "compression-core"
+version = "0.4.32"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cc14f565cf027a105f7a44ccf9e5b424348421a1d8952a8fc9d499d313107789"
+
 [[package]]
 name = "concurrent-queue"
 version = "2.5.0"
@@ -2952,6 +2981,12 @@ dependencies = [
  "digest 0.11.3",
 ]
 
+[[package]]
+name = "md5"
+version = "0.7.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "490cc448043f947bae3cbee9c203358d62dbee0db12107a74be5c30ccfd09771"
+
 [[package]]
 name = "memchr"
 version = "2.8.2"
@@ -3119,7 +3154,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge"
-version = "0.3.3"
+version = "0.3.4"
 dependencies = [
  "anyhow",
  "arc-swap",
@@ -3130,6 +3165,7 @@ dependencies = [
  "aws-sdk-sns",
  "aws-sdk-sqs",
  "base64",
+ "byteorder",
  "bytes",
  "chrono",
  "config",
@@ -3138,6 +3174,7 @@ dependencies = [
  "flate2",
  "futures",
  "h2 0.4.14",
+ "hex",
  "http-body 1.0.1",
  "http-body-util",
  "hyper 1.10.1",
@@ -3151,6 +3188,8 @@ dependencies = [
  "mq-bridge",
  "mqi",
  "once_cell",
+ "percent-encoding",
+ "pgwire-replication",
  "prost",
  "protoc-bin-vendored",
  "rand 0.10.1",
@@ -3188,7 +3227,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge-bindings-common"
-version = "0.3.3"
+version = "0.3.4"
 dependencies = [
  "anyhow",
  "fast-uuid-v7",
@@ -3202,7 +3241,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge-node"
-version = "0.3.3"
+version = "0.3.4"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -3223,7 +3262,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge-py"
-version = "0.3.3"
+version = "0.3.4"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -3689,6 +3728,28 @@ dependencies = [
  "indexmap",
 ]
 
+[[package]]
+name = "pgwire-replication"
+version = "0.3.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7eec6334b2466a0163bd91c29e06a25b391c155231934aee80381b99560e240e"
+dependencies = [
+ "anyhow",
+ "base64",
+ "bytes",
+ "hmac 0.12.1",
+ "md5",
+ "rand 0.9.4",
+ "rustls 0.23.40",
+ "sha2 0.10.9",
+ "thiserror 1.0.69",
+ "tokio",
+ "tokio-rustls 0.26.4",
+ "tokio-util",
+ "tracing",
+ "webpki-roots 0.26.11",
+]
+
 [[package]]
 name = "phf"
 version = "0.13.1"
@@ -4496,6 +4557,8 @@ dependencies = [
  "rustls 0.23.40",
  "rustls-pki-types",
  "rustls-platform-verifier",
+ "serde",
+ "serde_urlencoded",
  "sync_wrapper",
  "tokio",
  "tokio-rustls 0.26.4",
@@ -5027,6 +5090,18 @@ dependencies = [
  "serde_core",
 ]
 
+[[package]]
+name = "serde_urlencoded"
+version = "0.7.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd"
+dependencies = [
+ "form_urlencoded",
+ "itoa",
+ "ryu",
+ "serde",
+]
+
 [[package]]
 name = "serde_with"
 version = "3.21.0"
@@ -5948,12 +6023,17 @@ version = "0.6.11"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840"
 dependencies = [
+ "async-compression",
  "bitflags 2.13.0",
  "bytes",
+ "futures-core",
  "futures-util",
  "http 1.4.2",
  "http-body 1.0.1",
+ "http-body-util",
  "pin-project-lite",
+ "tokio",
+ "tokio-util",
  "tower",
  "tower-layer",
  "tower-service",

{mq_bridge_py_basic-0.3.3 → mq_bridge_py_basic-0.3.4}/Cargo.toml

@@ -16,7 +16,7 @@ members = ["bindings-common", "python/mq-bridge-py"]
 resolver = "2"
 
 [workspace.package]
-version = "0.3.3"
+version = "0.3.4"
 
 [workspace.dependencies]
 anyhow = "1.0"
@@ -134,6 +134,11 @@ aws-sdk-sns = { version = "1.93", optional = true, default-features = false, fea
 # SQLx
 sqlx = { version = "0.9", features = ["runtime-tokio", "tls-rustls", "postgres", "mysql", "sqlite", "any"], optional = true }
 
+# ClickHouse — talks the ClickHouse HTTP interface directly (JSONEachRow in/out).
+# reqwest is used rather than the typed `clickhouse` crate because that crate is
+# RowBinary/`Row`-derive only and cannot round-trip arbitrary dynamic JSON payloads.
+reqwest = { version = "0.13", default-features = false, features = ["rustls", "http2", "charset", "query", "gzip"], optional = true }
+
 # ZeroMQ
 zeromq = { version = "0.6", optional = true }
 
@@ -149,6 +154,13 @@ prost = { version = "0.14", optional = true, default-features = false, features
 tokio-stream = { version = "0.1", optional = true, features = ["net"] }
 
 url = { version = "2.5", optional = true }
+percent-encoding = { version = "2.3", optional = true }
+
+# Postgres CDC (logical replication / pgoutput). pgwire-replication is a
+# narrowly-scoped wire-protocol client (COPY-both, LSN/standby, TLS/SCRAM) that
+# deliberately avoids tokio-postgres; byteorder backs the pgoutput byte decoder.
+pgwire-replication = { version = "0.3", optional = true, default-features = false, features = ["tls-rustls", "scram", "md5"] }
+byteorder = { version = "1.5", optional = true }
 
 # Generate JSON schema
 schemars = { version = "1.2", optional = true }
@@ -168,7 +180,7 @@ middleware = ["metrics", "dedup"]
 # runtime the client is loaded lazily only when an ibm-mq endpoint is opened
 # (clear error if the client isn't installed). Use `ibm-mq-static` instead for
 # static linking.
-full = ["middleware", "mongodb", "kafka", "amqp", "nats", "mqtt", "http", "aws", "zeromq", "redis-streams", "sled", "grpc", "sqlx", "websocket", "rustls-aws-lc", "ibm-mq"]
+full = ["middleware", "mongodb", "kafka", "amqp", "nats", "mqtt", "http", "aws", "zeromq", "redis-streams", "sled", "grpc", "sqlx", "clickhouse", "websocket", "postgres-cdc", "rustls-aws-lc", "ibm-mq"]
 
 # Functionality features
 dedup = ["sled"]
@@ -194,7 +206,7 @@ schema = ["dep:schemars"]
 kafka = ["rdkafka"]
 amqp = ["lapin", "rustls", "rustls-pemfile", "url"]
 nats = ["async-nats", "rustls", "rustls-pemfile"]
-mongodb = ["dep:mongodb"]
+mongodb = ["dep:mongodb", "url"]
 mqtt = ["rumqttc", "rustls", "tokio-rustls", "rustls-pemfile", "url"]
 http = ["dep:arc-swap", "hyper", "hyper-util", "hyper-rustls", "http-body-util", "dep:http-body", "rustls", "tokio-rustls", "rustls-pemfile", "webpki-roots", "h2", "dep:flate2", "dep:base64"]
 websocket = ["dep:tokio-websockets"]
@@ -216,6 +228,11 @@ redis-streams = ["dep:redis"]
 # Note: The grpc feature requires protobuf-src which can significantly increase build times.
 grpc = ["tonic", "prost", "tokio-stream", "tonic-prost", "tonic-prost-build", "protoc-bin-vendored", "rustls"]
 sqlx = ["dep:sqlx", "url"]
+clickhouse = ["dep:reqwest", "dep:flate2", "url"]
+# Postgres logical-replication CDC (pgoutput). Reuses the sqlx dependency for the
+# slot-lifecycle control plane; the streaming wire protocol is pgwire-replication.
+# Needs Rust 1.88 (pgwire-replication MSRV), like the amqp feature gate.
+postgres-cdc = ["dep:pgwire-replication", "dep:byteorder", "sqlx", "dep:base64", "url", "dep:percent-encoding"]
 
 [dev-dependencies]
 config = { version = "0.15" }
@@ -223,6 +240,8 @@ serde_yaml_ng = "0.10"
 tempfile = "3.10"
 chrono = "0.4"
 metrics-util = "0.20"
+# pgoutput decoder unit tests decode hex-encoded wire fixtures.
+hex = "0.4"
 criterion = { version = "0.8", features = ["async", "async_tokio"] }
 # Used by the router_bench microbench to model the proposed lock-free route table.
 arc-swap = "1"

{mq_bridge_py_basic-0.3.3 → mq_bridge_py_basic-0.3.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mq-bridge-py-basic
-Version: 0.3.3
+Version: 0.3.4
 Classifier: Development Status :: 3 - Alpha
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3
@@ -113,9 +113,24 @@ schema = config_schema()        # the JSON Schema as a dict
 ```
 
 `config_schema()` is handy for editor validation of YAML configs too — dump it
-to a file and point your `# yaml-language-server: $schema=` line at it. The types
-are regenerated with `uv run python scripts/gen_config_types.py` (a test fails if
-they drift from the schema).
+to a file and point your `# yaml-language-server: $schema=` line at it.
+
+### Regenerating the config types
+
+`mq_bridge/config.pyi` and `mq_bridge/config.py` are **generated — do not edit by
+hand**. Regenerate them whenever the Rust config models change (e.g. adding a
+field or a new endpoint). Because the generator reads the schema from the
+compiled extension, you must rebuild first:
+
+```bash
+# from python/mq-bridge-py/
+uv run maturin develop                              # rebuild the extension
+uv run --no-sync python scripts/gen_config_types.py # regenerate config.pyi / config.py
+```
+
+Then commit the updated `config.pyi`/`config.py`. `tests/test_config_types.py`
+asserts the checked-in output matches the schema, so CI (the "Python package
+smoke test" job) fails if you skip this after a models change.
 
 ## Running a route
 

{mq_bridge_py_basic-0.3.3 → mq_bridge_py_basic-0.3.4}/README.md

@@ -19,6 +19,22 @@
 
 It is not only a forwarder. A route can transform, filter, fan out, retry, rate-limit, deduplicate, or turn a request into a response before the message reaches the next system. The core is built on Tokio and keeps the transport details at the edge, so application code can mostly work with `CanonicalMessage`s and handlers.
 
+## Move data from A to B — inside your own service
+
+If you need to move data reliably between systems and you write code (Rust, Python, or Node), `mq-bridge` is a strong default. It is a **library you embed**, not a daemon or control plane you operate.
+
+**Prefer not to write code?** [`mq-bridge-app`](https://github.com/marcomq/mq-bridge-app) runs the exact same engine as a **standalone, zero-code ETL service** configured entirely by **YAML or environment variables** — move data from A to B without writing a line. It ships a **Postman-style UI** to build, send, and inspect messages against a route, and can **import Postman collections and AsyncAPI documents** to scaffold routes and endpoints for you.
+
+*   **16+ transports, one API**: Kafka, NATS, AMQP (RabbitMQ), MQTT, MongoDB, **Postgres CDC** (logical replication), PostgreSQL / MySQL / SQLite (SQLx), ClickHouse, HTTP, WebSocket, gRPC, ZeroMQ, Redis Streams, AWS SQS/SNS, IBM MQ, files, and in-memory channels — all behind the same `receive_batch` / `send_batch` shape.
+*   **Change Data Capture**: stream row-level changes from **Postgres** (logical replication / `pgoutput`) and **MongoDB** (change streams) as flat rows with an operation marker.
+*   **Restart-safe delivery**: batch-aware ack/nack with commit sequencing for cumulative-ack brokers; the integration suite shows **no data loss during in-flight broker restarts**, including a Postgres CDC restart-safety test.
+*   **Reliability middleware, not a framework**: retries, dead-letter queues, deduplication, rate limiting, and cookie/session persistence wrap any endpoint.
+*   **Polyglot on one engine**: the same Rust core ships as native **Python** and **Node.js** bindings; routing, batching, and broker I/O stay in Rust.
+*   **TLS everywhere, one config shape**: a single `TlsConfig` block (CA bundle, client cert/key for mTLS, insecure-skip) is reused across transports.
+*   **Self-hosted, no daemon**: generate config in the optional UI, paste it into your code, run it in-process. No hosted control plane, no separate scheduler.
+
+> **Throughput.** In our own benchmarks, the same engine — driven the zero-code way through [`mq-bridge-app`](https://github.com/marcomq/mq-bridge-app) — sustained ~303,000 rows/s copying 1,000,000 rows from Postgres to JSONL on commodity hardware, keeping it well in the range of dedicated data-movement tools. On a CSV→JSONL file conversion (1,000,000 mixed-type rows, ~116 MiB), it sustained **833,333 rows/s**, about **~43x faster than Meltano** (`tap-csv` → `target-jsonl`, ~19,500 rows/s). Full setup, methodology, and the exact parameters are in [`benches/ETL_BENCHMARKS.md`](benches/ETL_BENCHMARKS.md).
+
 
 ## Language Bindings
 
@@ -56,7 +72,8 @@ For implementation details and quick start examples for each usage type, see the
 
 ## Features
 
-*   **Supported Backends**: Kafka, NATS, AMQP (RabbitMQ), MQTT, MongoDB, SQL Databases (PostgreSQL, MySQL, SQLite via sqlx), HTTP, WebSocket, ZeroMQ, Redis Streams, Files, AWS (SQS/SNS), IBM MQ, and in-memory channels.
+*   **Supported Backends**: Kafka, NATS, AMQP (RabbitMQ), MQTT, MongoDB, SQL Databases (PostgreSQL, MySQL, SQLite via sqlx), **Postgres CDC** (logical replication), ClickHouse, gRPC, HTTP, WebSocket, ZeroMQ, Redis Streams, Files, AWS (SQS/SNS), IBM MQ, and in-memory channels.
+*   **Change Data Capture**: PostgreSQL logical replication (`postgres_cdc`) and MongoDB change streams, surfaced as flat rows with an `*.operation` marker (`insert`/`update`/`delete`/`truncate`).
     > **Note**: IBM MQ is included in the `full` feature set via the `ibm-mq` feature, which loads the IBM MQ client library at runtime via dlopen — **no IBM SDK is needed to build**. The IBM MQ redistributable client only has to be present at runtime, and only if you actually use an IBM MQ endpoint (it is loaded lazily on first connect). If the client is missing, the affected route fails fast with a non-retryable error instead of reconnecting forever. The loader finds the client via the platform default name, `MQ_INSTALLATION_PATH` (e.g. `/opt/mqm`), or an explicit `MQB_IBM_MQ_LIB` path. To link the client statically at build time instead, use the `ibm-mq-static` feature (requires the IBM MQ SDK). See the [mqi crate](https://crates.io/crates/mqi/) for details.
     >
     > **TLS**: IBM MQ has its own TLS config shape (`IbmTlsConfig`), since the native client doesn't consume PEM files. The field names mirror the generic `TlsConfig` for config parity, but carry MQ-native semantics: `tls.cert_file` (alias `key_repository`) is a CMS key repository path (e.g. `/path/to/tls` for `tls.kdb`), not a PEM file. The repository can either be passwordless (backed by a `.sth` stash file next to the `.kdb`) or password-protected via `tls.cert_password` (alias `key_repository_password`; requires an IBM MQ client/server at 9.3.0.0 or later, which is the capability level `ibm-mq`/`ibm-mq-static` build against; 9.2 is EOL).
@@ -83,31 +100,32 @@ The error handling follows the same idea. Batch publishing can report partial su
 
 What it does not try to be: a domain framework, an actor runtime, or a full stream processor. You can build CQRS-ish flows with it, but the library cares more about transport, routing, and delivery behavior than about prescribing your domain model.
 
-## Status
+### How it compares
 
-This library was created in 2025 and is still fairly new.
+`mq-bridge` overlaps with config-driven ETL / pipeline runners, but comes in two shapes: a **library you embed in your own service**, or [`mq-bridge-app`](https://github.com/marcomq/mq-bridge-app) — the **same engine as a zero-code, config-driven ETL service**. You get the developer-first path and the no-code path from one codebase.
 
-It may still be possible that there are issues with
-- old or very new versions of broker servers
-- specific settings of the brokers
-- subscribe/event and response patterns if those are not available natively
-- NATS without JetStream
-- TLS setups, which are usually non-trivial and have just been tested automatically, but not in detail.
+*   **Choose the `mq-bridge` library** when you write the code that moves the data — in Rust, Python, or Node — and want batching, retries/DLQ/dedup, request-reply, CDC, and TLS behind one small API you run in-process. No separate scheduler, no daemon to operate.
+*   **Choose `mq-bridge-app` for zero-code ETL** when you'd rather move data A→B purely by config: define routes and endpoints in YAML/env, then use its **Postman-style UI** to send test messages and watch them flow — and **import Postman collections or AsyncAPI specs** through the UI to generate the config. Same reliability engine, no build step.
+*   **Choose a dedicated stream processor** when you need windowed joins or aggregations over time.
 
-Automated integration and performance tests cover all supported endpoints, including queue and subscriber modes, request-reply (where supported), and protocol-specific behaviors. See the backend feature table below for details on configuration and protocol support.
+Either way it stays transport-first on purpose: it moves and routes data reliably rather than prescribing a domain model or a platform.
 
-The following table tracks which endpoints are already used actively in other projects for events. Send me a message or create an issue if you use another endpoint in production:
+## Status
 
-| Endpoint  | Manual Test |
-|-----------|:-----------:|
-| Kafka     |     ✅      |
-| MongoDB   |     ✅      |
-| HTTP      |     ✅      |
-| IBM MQ    |     ✅      |
-| Retry Middleware    |     ✅      |
-| DLQ Middleware    |     ✅      |
+`mq-bridge` is young (created in 2025) but its reliability behavior is exercised by an automated integration + performance suite across **every** supported endpoint:
+
+*   Integration and performance tests cover all endpoints in both queue and subscriber modes, request-reply where supported, and protocol-specific behavior.
+*   All endpoints have automated integration tests that showed **no data loss during in-flight broker restarts**; MQTT publish-confirmation was hardened until a chaos test drove in-flight loss to **zero**.
+*   Postgres CDC ships with a **restart-safety integration test**: an un-acked, in-flight batch is redelivered after a database restart with no loss and no gap into already-acknowledged changes.
+*   Throughput is tracked continuously on a public [benchmark dashboard](https://marcomq.github.io/mq-bridge/dev/bench/); ETL/CDC-shaped, like-for-like scenarios and their fixed methodology are documented in [`benches/ETL_BENCHMARKS.md`](benches/ETL_BENCHMARKS.md).
+
+Known rough edges (be deliberate here):
+
+- old or very new broker-server versions, or unusual broker settings
+- subscribe/event and response patterns where the backend has no native equivalent (emulated)
+- NATS without JetStream
+- **TLS**: one unified `TlsConfig` shape is reused across transports and exercised by automated tests; per-backend certificate matrices (self-signed / CA-signed / mTLS / expired-cert rejection) are being expanded — treat non-trivial TLS setups as needing your own verification for now.
 
-All endpoints have automated integration tests and did not show data loss during simple in-flight broker restarts.
 
 ## Test Notes
 
@@ -157,6 +175,7 @@ The table below summarizes the capabilities and configuration for each backend:
 | **MongoDB** | Set `change_stream: true` | Emulated (Metadata) | **Yes** (Unlock) |
 | **MQTT** | Set `clean_session: true` | Emulated (Property) | Eventual (Skip Ack) |
 | **NATS** | Set `subscriber_mode: true` | **Native** (Inbox) | **Yes** (JetStream Nak) |
+| **Postgres CDC** | N/A (streams committed changes) | No | **Yes** (confirmed LSN not advanced) |
 | **Redis Streams** | Set `subscriber_mode: true` | No | Eventual (PEL, un-acked) |
 | **Sled** | Set `delete_after_read: false` | No | **Yes** (Tx Rollback) |
 | **SQLx** | Not supported | No | Eventual (Skip Delete) |
@@ -169,6 +188,15 @@ The table below summarizes the capabilities and configuration for each backend:
     *   **Emulated**: Publishes a new message to a reply destination (specified by the `reply_to` metadata field) carrying a `correlation_id` metadata field.
 *   **Nack Support**: If "Yes", the backend supports explicit negative acknowledgement triggering redelivery. "Eventual" means redelivery depends on timeout or connection drop. "Simulated" is handled in-memory by the bridge.
 
+### Database Sources: Change Capture vs Polling
+Databases have no native pub/sub, so `mq-bridge` reads them as a source in one of two ways:
+
+*   **Change Data Capture (CDC)** — tails the database's own change log, capturing inserts **and** updates/deletes and resuming from a durable log position.
+    *   On **PostgreSQL**, the `postgres_cdc` endpoint streams a logical-replication slot (`pgoutput`). It emits flat JSON rows tagged with `postgres.operation` metadata (`insert`/`update`/`delete`/`truncate`). Acking a batch confirms the LSN back to the server (`standby_status_update`); a nack (or an interrupted run) does **not** advance the confirmed LSN, so replication resumes from the last acknowledged position on reconnect — at-least-once, verified by a restart-safety integration test. Enable with the `postgres-cdc` feature; requires `wal_level = logical` and a publication. See below.
+    *   On **MongoDB**, set `consume: capture_new` to watch an existing collection for changes from now on, or `consume: capture_all` to read the existing documents first and then keep capturing changes (no gap, at-least-once). It emits `insert`/`update`/`replace`/`delete` events tagged with `mongodb.operation` metadata and checkpoints progress under `cursor_id`. These use the change stream (full post-image via `updateLookup`) and need a replica set; on a standalone server `capture_all` falls back to an insert-only read.
+*   **Cursor polling** — pages an existing table by a monotonic `cursor_column` (`WHERE col > $last ORDER BY col ASC`), persisting the last read value under `cursor_id`. Captures **appends only** — updates and deletes are not observed. Available on **SQLx** (PostgreSQL / MySQL / MariaDB / SQLite) and **ClickHouse**; while idle the poll interval backs off exponentially between `polling_interval_ms` and `max_polling_interval_ms`. **SQLite** and **ClickHouse** are polling-only — they have no server-side change log.
+
+
 ### Response Endpoint
 The `response` output endpoint sends a reply back to the original requester. This is useful for synchronous request-reply flows, for example HTTP-to-NATS-to-HTTP. Use `response: {}` as the output endpoint configuration.
 

mq_bridge_py_basic-0.3.4/benches/ETL_BENCHMARKS.md

@@ -0,0 +1,107 @@
+# ETL / CDC-comparable benchmark methodology
+
+This suite exists so that mq-bridge can be evaluated **like-for-like** against
+config-driven ETL/CDC tools for moving Postgres data — not against
+microbenchmarks of unrelated primitives. Credibility is the asset here: every
+number below is reproducible from a pinned workload and a pinned backend
+version, and the methodology is stated up front rather than cherry-picked.
+
+The existing criterion harness (`benches/performance_bench.rs`, published to the
+[benchmark dashboard](https://marcomq.github.io/mq-bridge/dev/bench/)) already
+measures per-backend batched throughput. This document defines the additional
+**ETL/CDC-shaped scenarios** and the fixed parameters they run under.
+
+## Fixed parameters (keep constant across runs and across tools)
+
+| Parameter        | Value                                             |
+| ---------------- | ------------------------------------------------- |
+| Message payload  | 256 B and 4 KiB JSON rows (report both)           |
+| Message count    | 100_000 per run                                   |
+| Batch sizes      | 1 (unbatched) and 128 (batched)                   |
+| Concurrency      | 1 and 4 route workers                             |
+| Postgres         | `postgres:16-alpine`, `wal_level=logical`         |
+| Warm-up          | criterion default + 5_000-message pre-roll        |
+| Machine          | record CPU model, cores, RAM in the published run |
+
+Report the **backend version and the exact parameters next to every number**.
+Never publish a single favourable figure without its methodology row.
+
+## Scenarios
+
+1. **Bulk-insert throughput** — `memory → sqlx(postgres)` sink, measured in
+   rows/second at batch sizes 1 and 128. Answers "how fast can mq-bridge load a
+   table". Uses `docker-compose/postgres_cdc.yml`.
+
+2. **CDC event-to-sink latency** — insert a row into a captured table and
+   measure the time until the corresponding change event is delivered to a
+   `memory` sink through the `postgres_cdc` endpoint (p50 / p95 / p99). Answers
+   "how quickly does a change propagate". Uses `docker-compose/postgres_cdc.yml`.
+
+3. **Batched vs. unbatched throughput** — the same source→sink route at
+   `batch_size: 1` vs `batch_size: 128`, per backend, to quantify the batching
+   lever the README documents. Runs on `memory` (no external service) and on
+   each broker that has a compose file.
+
+## Running
+
+```bash
+# Batched-vs-unbatched (no external services)
+cargo bench --features test-utils --bench performance_bench
+
+# CDC latency + bulk-insert (needs Docker Postgres with logical replication)
+docker compose -f tests/integration/docker-compose/postgres_cdc.yml up -d --wait
+cargo bench --features "test-utils postgres-cdc" --bench performance_bench -- cdc
+docker compose -f tests/integration/docker-compose/postgres_cdc.yml down -v
+```
+
+## Results
+
+The scenarios are run through
+[`mq-bridge-app`](https://github.com/marcomq/mq-bridge-app) (the zero-code, config-driven
+path a real user takes) so the numbers are comparable to how other ETL/CDC tools publish
+theirs. The run brief lives in [`mq-bridge-app-benchmark-prompt.md`](mq-bridge-app-benchmark-prompt.md).
+
+> ⚠️ **Preliminary — measured on battery power.** The number below was measured on a
+> laptop running on battery, where macOS throttles the CPU. It is **not yet authoritative**
+> and is pending a re-run on AC (mains) power before publication. Treat it as a floor, not a
+> confirmed figure.
+
+**Environment:** Apple M1, 8 cores, 8 GB RAM, macOS (battery power) · via `mq-bridge-app` 0.2.6 · postgres:16
+
+### Validated run — bulk copy Postgres → JSONL
+
+| Scenario | Payload | Batch | Concurrency | Source → Sink | Result |
+| --- | --- | --- | --- | --- | --- |
+| Bulk-copy throughput | 7-col mixed-type rows, 1,000,000 rows | 1024 | 4 | postgres (keyset cursor) → file (JSONL) | **303,398 rows/s** (3.296 s) |
+
+Command: `mq-bridge-app copy --from postgres://…?table=…&cursor_column=id --to file://…?format=raw --drain --batch-size 1024 --concurrency 4`, wall-clocked. Requires an index on the cursor column (`CREATE INDEX ON <table>(id)`) — without it the keyset-pagination reader (`WHERE id > $cursor ORDER BY id LIMIT batch`) does a full scan per batch (near-quadratic).
+
+### Validated run — Postgres → JSONL vs. Meltano
+
+Same source table, same Postgres instance, same machine, both one-shot full-table syncs to a local JSONL file — mq-bridge-app's `copy` CLI vs. Meltano's `tap-postgres` → `target-jsonl` (median of 5 timed runs each side).
+
+| Scenario | Payload | Batch | Concurrency | Source → Sink | Result |
+| --- | --- | --- | --- | --- | --- |
+| mq-bridge-app `copy` | 256 B, 100,000 rows | 1024 | 1 | postgres (keyset cursor) → file (JSONL) | **197,628 rows/s** |
+| Meltano (`tap-postgres` → `target-jsonl`) | 256 B, 100,000 rows | default Singer config | — | postgres → JSONL | 16,330 rows/s |
+
+**~12.1x faster** than Meltano in this scenario. Full setup (including the Meltano project config) is in [mq-bridge-app's `benches/etl/README.md`](https://github.com/marcomq/mq-bridge-app/blob/dev/benches/etl/README.md#5--postgres--jsonl-vs-meltano-tap-postgres--target-jsonl).
+
+### Validated run — CSV → JSONL vs. Meltano
+
+Same seeded dataset both sides, same machine, one-shot full-file CSV → local JSONL (one JSON object per input row) — mq-bridge-app's `copy` CLI vs. Meltano's `tap-csv` → `target-jsonl`.
+
+| Scenario | Payload | Batch | Concurrency | Source → Sink | Result |
+| --- | --- | --- | --- | --- | --- |
+| mq-bridge-app `copy` | 7-col mixed-type, 1,000,000 rows (~116 MiB) | 1024 | 1 | file (CSV) → file (JSONL) | **833,333 rows/s** |
+| Meltano (`tap-csv` → `target-jsonl`) | 7-col mixed-type, 1,000,000 rows | default Singer config | — | file (CSV) → JSONL | ~19,500 rows/s |
+
+**~43x faster** than Meltano in this scenario. Full setup is in [mq-bridge-app's `benches/etl/README.md`](https://github.com/marcomq/mq-bridge-app/blob/dev/benches/etl/README.md#6--csv--jsonl-vs-meltano-faucet-streams-headline-benchmark).
+
+## Status
+
+Scenario 3 (batched vs. unbatched) is covered by the existing harness. Scenarios
+1 and 2 (bulk-insert, CDC latency) share the Docker Postgres environment with the
+CDC integration tests (`tests/integration/postgres_cdc.rs`); their criterion
+wiring is added and validated together with that Docker run so the published
+numbers are real, not estimated.