mq-bridge-py-basic 0.3.7__tar.gz → 0.3.8__tar.gz

{mq_bridge_py_basic-0.3.7 → mq_bridge_py_basic-0.3.8}/CLAUDE.md

@@ -3,7 +3,7 @@
 > **Looking for what middleware or structural endpoints exist, and how to configure them?**
 > [REFERENCE.md](REFERENCE.md) is the complete, authoritative list — every middleware
 > (`retry`, `dlq`, `transform`, `deduplication`, `weak_join`, `buffer`, `limiter`, `delay`,
-> `cookie_jar`, `encryption`, `metrics`, `random_panic`, `custom`) and every structural endpoint (`ref`,
+> `cookie_jar`, `encryption`, `compression`, `metrics`, `random_panic`, `custom`) and every structural endpoint (`ref`,
 > `fanout`, `switch`, `request`, `response`, `reader`, `static`, `stream_buffer`, `null`,
 > `custom`), each with its fields, defaults, and a working YAML example. Do not infer these
 > from the enum definitions in `models.rs`; the reference records the behaviour and the

{mq_bridge_py_basic-0.3.7 → mq_bridge_py_basic-0.3.8}/Cargo.lock

@@ -3330,7 +3330,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge"
-version = "0.3.7"
+version = "0.3.8"
 dependencies = [
  "aes-gcm",
  "anyhow",
@@ -3411,7 +3411,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge-bindings-common"
-version = "0.3.7"
+version = "0.3.8"
 dependencies = [
  "anyhow",
  "fast-uuid-v7",
@@ -3425,7 +3425,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge-node"
-version = "0.3.7"
+version = "0.3.8"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -3446,7 +3446,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge-py"
-version = "0.3.7"
+version = "0.3.8"
 dependencies = [
  "anyhow",
  "async-trait",

{mq_bridge_py_basic-0.3.7 → mq_bridge_py_basic-0.3.8}/Cargo.toml

@@ -16,7 +16,7 @@ members = ["bindings-common", "python/mq-bridge-py"]
 resolver = "2"
 
 [workspace.package]
-version = "0.3.7"
+version = "0.3.8"
 
 [workspace.dependencies]
 anyhow = "1.0"

{mq_bridge_py_basic-0.3.7 → mq_bridge_py_basic-0.3.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mq-bridge-py-basic
-Version: 0.3.7
+Version: 0.3.8
 Classifier: Development Status :: 3 - Alpha
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3

{mq_bridge_py_basic-0.3.7 → mq_bridge_py_basic-0.3.8}/REFERENCE.md

@@ -65,10 +65,21 @@ middlewares:
 | [`delay`](#delay) | ✅ | ✅ | – | Fixed delay per receive/send |
 | [`cookie_jar`](#cookie_jar) | ✅ | ✅ | – | Persist HTTP cookies / session values across messages |
 | [`encryption`](#encryption) | ✅ | ✅ | `encryption` | AEAD-encrypt payloads on send, decrypt on receive |
+| [`compression`](#compression) | ✅ | ✅ | `compression` | Compress payloads on send, decompress on receive |
 | [`metrics`](#metrics) | ✅ | ✅ | `metrics` | Emit throughput/latency/error metrics |
 | [`random_panic`](#random_panic) | ✅ | ✅ | – | Fault injection for testing |
 | [`custom`](#custom-middleware) | ✅ | ✅ | – | Your own middleware via a registered factory |
 
+> **Two kinds of compression.** The [`compression`](#compression) *middleware* compresses
+> each message **payload** on any transport and decompresses it on the far side. Separately,
+> the batch `compression` *field* on the `file` and `object_store` endpoints (`none` / `gzip`
+> / `lz4` / `zstd`, same `compression` feature) compresses whole write batches so the file
+> stays decodable with `zcat` / `lz4 -d`. Use the field for CLI-readable data at rest, the
+> middleware for over-the-wire payloads. Don't stack either compression with the
+> [`encryption`](#encryption) middleware on the same route — ciphertext does not compress; for
+> compressed-and-encrypted data at rest use the endpoints' own `compression`/`encryption`
+> fields (compress-then-encrypt per batch).
+
 **Putting a middleware on the wrong side behaves in two different ways**, so check the table
 above rather than assuming:
 
@@ -421,6 +432,33 @@ sniffing the leading magic bytes, so raw compressed bytes are never emitted as m
 > `double precision` column can change the last bit. Build with the `float-roundtrip` feature
 > for bit-exact float parsing across every endpoint (it trades a little parse speed for it).
 
+### `compression`
+
+Compresses each message **payload** on the output side and decompresses it on the input
+side. Metadata and routing keys are untouched. Input and output. Requires the `compression`
+feature.
+
+| Field | Type | Default |
+|---|---|---|
+| `algorithm` | `none` \| `gzip` \| `lz4` \| `zstd` | `zstd` |
+| `max_decompressed_bytes` | integer — reject a payload that decompresses larger than this (bomb guard); consumer side only | unset (no limit) |
+
+```yaml
+- compression: { algorithm: zstd }
+```
+
+Each payload is compressed independently into a single self-contained member, so this works
+over any transport, not just files. `algorithm: none` is a passthrough. A truncated or
+corrupt frame is a **permanent** consumer error (the poison message is not re-read
+indefinitely), as is a payload that exceeds `max_decompressed_bytes`. Put the same
+`algorithm` on both the input and output side of a route.
+
+Unlike the `file` / `object_store` batch `compression` field — which keeps whole write
+batches decodable with `zcat` / `lz4 -d` — this middleware frames per message and is only
+readable through a matching consumer. Do not combine it with the [`encryption`](#encryption)
+middleware (ciphertext does not compress); for compressed-and-encrypted data at rest, use the
+endpoints' own `compression`/`encryption` fields instead.
+
 ### `metrics`
 
 Emits throughput, latency and error metrics for the endpoint. Input and output. Requires the

{mq_bridge_py_basic-0.3.7 → mq_bridge_py_basic-0.3.8}/mq-bridge.schema.json

@@ -389,6 +389,28 @@
         }
       ]
     },
+    "CompressionMiddleware": {
+      "description": "Payload-compression middleware configuration.\n\nCompresses each message payload on the output side and decompresses it on the input\nside; metadata and routing keys are left untouched. Requires the `compression` feature.\nDistinct from the `file`/`object_store` batch `compression` field, which stays\nCLI-decodable at rest — this operates per message on any transport.",
+      "type": "object",
+      "properties": {
+        "algorithm": {
+          "description": "Algorithm: `none`, `gzip`, `lz4`, or `zstd`. Defaults to `zstd`.",
+          "$ref": "#/$defs/Compression",
+          "default": "zstd"
+        },
+        "max_decompressed_bytes": {
+          "description": "Reject a decompressed payload larger than this many bytes (decompression-bomb guard).\nConsumer side only; unset means no limit.",
+          "type": [
+            "integer",
+            "null"
+          ],
+          "format": "uint64",
+          "default": null,
+          "minimum": 0
+        }
+      },
+      "additionalProperties": false
+    },
     "CookieJarMiddleware": {
       "description": "Cookie/session jar middleware configuration.\n\nOptimized for HTTP by default: it can read `cookie` and `set-cookie` metadata,\npersist session cookies, and inject them into later outgoing requests.\n\nThe middleware can also capture arbitrary metadata values into the same session store\nand optionally expose stored values back into message metadata.",
       "type": "object",
@@ -1926,6 +1948,18 @@
             "encryption"
           ]
         },
+        {
+          "type": "object",
+          "properties": {
+            "compression": {
+              "$ref": "#/$defs/CompressionMiddleware"
+            }
+          },
+          "additionalProperties": false,
+          "required": [
+            "compression"
+          ]
+        },
         {
           "type": "object",
           "properties": {

{mq_bridge_py_basic-0.3.7 → mq_bridge_py_basic-0.3.8}/mq_bridge/config.py

@@ -7,6 +7,7 @@ AmqpConfig = dict
 AwsConfig = dict
 BufferMiddleware = dict
 ClickHouseConfig = dict
+CompressionMiddleware = dict
 CookieJarMiddleware = dict
 DeadLetterQueueMiddleware = dict
 DeduplicationMiddleware = dict
@@ -73,6 +74,7 @@ __all__ = [
     "CipherKind",
     "ClickHouseConfig",
     "Compression",
+    "CompressionMiddleware",
     "ConfigDocument",
     "CookieJarMiddleware",
     "DeadLetterQueueMiddleware",

{mq_bridge_py_basic-0.3.7 → mq_bridge_py_basic-0.3.8}/mq_bridge/config.pyi

@@ -63,6 +63,12 @@ class ClickHouseConfig(TypedDict, total=False):
     wait_for_async_insert: Optional[bool]
 
 
+class CompressionMiddleware(TypedDict, total=False):
+    """Payload-compression middleware configuration."""
+    algorithm: Compression
+    max_decompressed_bytes: Optional[int]
+
+
 class CookieJarMiddleware(TypedDict, total=False):
     """Cookie/session jar middleware configuration."""
     capture_metadata_keys: List[str]
@@ -226,6 +232,7 @@ class MetricsMiddleware(TypedDict, total=False):
 class Middleware(TypedDict, total=False):
     """An enumeration of all supported middleware types."""
     buffer: BufferMiddleware
+    compression: CompressionMiddleware
     cookie_jar: CookieJarMiddleware
     custom: Dict[str, Any]
     deduplication: DeduplicationMiddleware

mq_bridge_py_basic-0.3.8/src/middleware/compression.rs

@@ -0,0 +1,314 @@
+//  mq-bridge
+//  © Copyright 2026, by Marco Mengelkoch
+//  Licensed under MIT License, see License file for more details
+//  git clone https://github.com/marcomq/mq-bridge
+//
+//! Two-way payload compression middleware: the publisher side compresses each
+//! message payload into a single self-contained member, the consumer side
+//! decompresses it. Metadata and routing keys stay untouched. `Compression::None`
+//! is a passthrough.
+
+use crate::models::{Compression, CompressionMiddleware};
+use crate::support::compression::{compress_member, decompress_all};
+use crate::traits::{
+    BoxFuture, ConsumerError, MessageConsumer, MessagePublisher, PublisherError, Received,
+    ReceivedBatch, Sent, SentBatch,
+};
+use crate::CanonicalMessage;
+use async_trait::async_trait;
+use std::any::Any;
+
+pub struct CompressionPublisher {
+    inner: Box<dyn MessagePublisher>,
+    algo: Compression,
+}
+
+impl CompressionPublisher {
+    pub fn new(inner: Box<dyn MessagePublisher>, config: &CompressionMiddleware) -> Self {
+        Self {
+            inner,
+            algo: config.algorithm,
+        }
+    }
+
+    fn compress_message(&self, message: &mut CanonicalMessage) -> Result<(), PublisherError> {
+        if self.algo == Compression::None {
+            return Ok(());
+        }
+        let out = compress_member(self.algo, &message.payload)
+            .map_err(|e| PublisherError::NonRetryable(e.into()))?;
+        message.payload = out.into();
+        Ok(())
+    }
+}
+
+#[async_trait]
+impl MessagePublisher for CompressionPublisher {
+    fn on_connect_hook(&self) -> Option<BoxFuture<'_, anyhow::Result<()>>> {
+        self.inner.on_connect_hook()
+    }
+
+    fn on_disconnect_hook(&self) -> Option<BoxFuture<'_, anyhow::Result<()>>> {
+        self.inner.on_disconnect_hook()
+    }
+
+    async fn send(&self, mut message: CanonicalMessage) -> Result<Sent, PublisherError> {
+        self.compress_message(&mut message)?;
+        self.inner.send(message).await
+    }
+
+    async fn send_batch(
+        &self,
+        mut messages: Vec<CanonicalMessage>,
+    ) -> Result<SentBatch, PublisherError> {
+        // Keep the uncompressed payloads: messages surfaced as failed must go back
+        // upstream (retry/dlq) in their original form, or an outer retry would
+        // double-compress them.
+        let originals: std::collections::HashMap<u128, bytes::Bytes> = messages
+            .iter()
+            .map(|m| (m.message_id, m.payload.clone()))
+            .collect();
+        for message in &mut messages {
+            self.compress_message(message)?;
+        }
+        match self.inner.send_batch(messages).await? {
+            SentBatch::Ack => Ok(SentBatch::Ack),
+            SentBatch::Partial {
+                responses,
+                mut failed,
+            } => {
+                for (msg, _) in &mut failed {
+                    if let Some(original) = originals.get(&msg.message_id) {
+                        msg.payload = original.clone();
+                    }
+                }
+                Ok(SentBatch::Partial { responses, failed })
+            }
+        }
+    }
+
+    fn as_any(&self) -> &dyn Any {
+        self
+    }
+}
+
+pub struct CompressionConsumer {
+    inner: Box<dyn MessageConsumer>,
+    algo: Compression,
+    max_bytes: Option<u64>,
+}
+
+impl CompressionConsumer {
+    pub fn new(inner: Box<dyn MessageConsumer>, config: &CompressionMiddleware) -> Self {
+        Self {
+            inner,
+            algo: config.algorithm,
+            max_bytes: config.max_decompressed_bytes,
+        }
+    }
+
+    fn decompress_message(&self, message: &mut CanonicalMessage) -> Result<(), ConsumerError> {
+        if self.algo == Compression::None {
+            return Ok(());
+        }
+        // A malformed/truncated frame will never decode, so it is a permanent failure
+        // rather than a reconnectable one — otherwise the poison message would be
+        // re-read forever.
+        let out = decompress_all(self.algo, &message.payload, self.max_bytes)
+            .map_err(|e| ConsumerError::Permanent(e.into()))?;
+        message.payload = out.into();
+        Ok(())
+    }
+}
+
+#[async_trait]
+impl MessageConsumer for CompressionConsumer {
+    fn set_exit_on_empty(&mut self, exit_on_empty: bool) {
+        self.inner.set_exit_on_empty(exit_on_empty);
+    }
+
+    fn commit_requires_order(&self) -> bool {
+        self.inner.commit_requires_order()
+    }
+
+    fn on_connect_hook(&self) -> Option<BoxFuture<'_, anyhow::Result<()>>> {
+        self.inner.on_connect_hook()
+    }
+
+    fn on_disconnect_hook(&self) -> Option<BoxFuture<'_, anyhow::Result<()>>> {
+        self.inner.on_disconnect_hook()
+    }
+
+    async fn receive(&mut self) -> Result<Received, ConsumerError> {
+        let mut received = self.inner.receive().await?;
+        self.decompress_message(&mut received.message)?;
+        Ok(received)
+    }
+
+    async fn receive_batch(&mut self, max_messages: usize) -> Result<ReceivedBatch, ConsumerError> {
+        let mut batch = self.inner.receive_batch(max_messages).await?;
+        for message in &mut batch.messages {
+            self.decompress_message(message)?;
+        }
+        Ok(batch)
+    }
+
+    fn as_any(&self) -> &dyn Any {
+        self
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::sync::{Arc, Mutex};
+
+    fn config(algo: Compression) -> CompressionMiddleware {
+        CompressionMiddleware {
+            algorithm: algo,
+            ..Default::default()
+        }
+    }
+
+    #[derive(Clone)]
+    struct RecordingPublisher {
+        sent: Arc<Mutex<Vec<CanonicalMessage>>>,
+    }
+
+    #[async_trait]
+    impl MessagePublisher for RecordingPublisher {
+        async fn send_batch(
+            &self,
+            messages: Vec<CanonicalMessage>,
+        ) -> Result<SentBatch, PublisherError> {
+            self.sent.lock().unwrap().extend(messages);
+            Ok(SentBatch::Ack)
+        }
+
+        fn as_any(&self) -> &dyn Any {
+            self
+        }
+    }
+
+    struct MockConsumer {
+        messages: Option<Vec<CanonicalMessage>>,
+    }
+
+    #[async_trait]
+    impl MessageConsumer for MockConsumer {
+        async fn receive_batch(
+            &mut self,
+            _max_messages: usize,
+        ) -> Result<ReceivedBatch, ConsumerError> {
+            Ok(ReceivedBatch {
+                messages: self.messages.take().expect("batch already consumed"),
+                commit: Box::new(|_| Box::pin(async { Ok(()) })),
+            })
+        }
+
+        fn as_any(&self) -> &dyn Any {
+            self
+        }
+    }
+
+    #[tokio::test]
+    async fn publisher_compresses_and_consumer_decompresses() {
+        for algo in [Compression::Gzip, Compression::Lz4, Compression::Zstd] {
+            let sent = Arc::new(Mutex::new(Vec::new()));
+            let publisher = CompressionPublisher::new(
+                Box::new(RecordingPublisher { sent: sent.clone() }),
+                &config(algo),
+            );
+
+            let plaintext = "compress me ".repeat(64);
+            let mut msg = CanonicalMessage::from(plaintext.as_str());
+            msg.metadata.insert("kind".to_string(), "note".to_string());
+            publisher.send_batch(vec![msg]).await.unwrap();
+
+            // The wire payload is compressed (smaller) and metadata stays clear.
+            let wire = sent.lock().unwrap().clone();
+            assert_ne!(
+                wire[0].payload.as_ref(),
+                plaintext.as_bytes(),
+                "algo {algo:?}"
+            );
+            assert!(wire[0].payload.len() < plaintext.len(), "algo {algo:?}");
+            assert_eq!(
+                wire[0].metadata.get("kind").map(|s| s.as_str()),
+                Some("note")
+            );
+
+            let mut consumer = CompressionConsumer::new(
+                Box::new(MockConsumer {
+                    messages: Some(wire),
+                }),
+                &config(algo),
+            );
+            let batch = consumer.receive_batch(10).await.unwrap();
+            assert_eq!(
+                batch.messages[0].payload.as_ref(),
+                plaintext.as_bytes(),
+                "algo {algo:?}"
+            );
+        }
+    }
+
+    #[tokio::test]
+    async fn corrupt_frame_fails_consume() {
+        let sent = Arc::new(Mutex::new(Vec::new()));
+        let publisher = CompressionPublisher::new(
+            Box::new(RecordingPublisher { sent: sent.clone() }),
+            &config(Compression::Gzip),
+        );
+        publisher
+            .send_batch(vec![CanonicalMessage::from("payload payload payload")])
+            .await
+            .unwrap();
+
+        let mut wire = sent.lock().unwrap().clone();
+        let mut corrupt = wire[0].payload.to_vec();
+        *corrupt.last_mut().unwrap() ^= 0xff;
+        wire[0].payload = corrupt.into();
+
+        let mut consumer = CompressionConsumer::new(
+            Box::new(MockConsumer {
+                messages: Some(wire),
+            }),
+            &config(Compression::Gzip),
+        );
+        assert!(matches!(
+            consumer.receive_batch(10).await,
+            Err(ConsumerError::Permanent(_))
+        ));
+    }
+
+    #[tokio::test]
+    async fn decompression_bomb_guard_rejects_oversized() {
+        let sent = Arc::new(Mutex::new(Vec::new()));
+        let publisher = CompressionPublisher::new(
+            Box::new(RecordingPublisher { sent: sent.clone() }),
+            &config(Compression::Zstd),
+        );
+        let big = "x".repeat(64 * 1024);
+        publisher
+            .send_batch(vec![CanonicalMessage::from(big.as_str())])
+            .await
+            .unwrap();
+        let wire = sent.lock().unwrap().clone();
+
+        let cfg = CompressionMiddleware {
+            algorithm: Compression::Zstd,
+            max_decompressed_bytes: Some(1024),
+        };
+        let mut consumer = CompressionConsumer::new(
+            Box::new(MockConsumer {
+                messages: Some(wire),
+            }),
+            &cfg,
+        );
+        assert!(matches!(
+            consumer.receive_batch(10).await,
+            Err(ConsumerError::Permanent(_))
+        ));
+    }
+}

{mq_bridge_py_basic-0.3.7 → mq_bridge_py_basic-0.3.8}/src/middleware/mod.rs

@@ -10,6 +10,8 @@ use anyhow::Result;
 use std::sync::Arc;
 
 mod buffer;
+#[cfg(feature = "compression")]
+mod compression;
 mod cookie_jar;
 #[cfg(feature = "dedup")]
 pub(crate) mod deduplication;
@@ -26,6 +28,8 @@ mod transform;
 mod weak_join;
 
 use buffer::{BufferConsumer, BufferPublisher};
+#[cfg(feature = "compression")]
+use compression::{CompressionConsumer, CompressionPublisher};
 use cookie_jar::{CookieJarConsumer, CookieJarPublisher};
 #[cfg(feature = "dedup")]
 use deduplication::DeduplicationConsumer;
@@ -77,6 +81,8 @@ pub async fn apply_middlewares_to_consumer(
             Middleware::Transform(cfg) => Box::new(TransformConsumer::new(consumer, cfg)?),
             #[cfg(feature = "encryption")]
             Middleware::Encryption(cfg) => Box::new(EncryptionConsumer::new(consumer, cfg)?),
+            #[cfg(feature = "compression")]
+            Middleware::Compression(cfg) => Box::new(CompressionConsumer::new(consumer, cfg)),
             Middleware::Custom { name, config } => {
                 let factory = get_middleware_factory(name).ok_or_else(|| {
                     anyhow::anyhow!("Custom middleware factory '{}' not found", name)
@@ -140,6 +146,8 @@ pub async fn apply_middlewares_to_publisher(
             Middleware::Transform(cfg) => Box::new(TransformPublisher::new(publisher, cfg)?),
             #[cfg(feature = "encryption")]
             Middleware::Encryption(cfg) => Box::new(EncryptionPublisher::new(publisher, cfg)?),
+            #[cfg(feature = "compression")]
+            Middleware::Compression(cfg) => Box::new(CompressionPublisher::new(publisher, cfg)),
             Middleware::Custom { name, config } => {
                 let factory = get_middleware_factory(name).ok_or_else(|| {
                     anyhow::anyhow!("Custom middleware factory '{}' not found", name)