omq-lz4 0.2.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: ade7fc716054707c2e05bade867b01333708de30df98e3acb524f8da1a7d98d4
+  data.tar.gz: 7ae5fade7f0d88eed7f6f7f8875373a044127098d596b9f6bae5554d585b1e90
+SHA512:
+  metadata.gz: 59afc48b58c8c1efac0973bffa702cac3958b0150b4eb51fe9dec82dacc1c9735446d83e58ac6963f0302e13b26a0890b1eaeff5c0d7d846830c561775a848f4
+  data.tar.gz: 17c66cce9f79f1a375a8614e3aa3cc45071277432521ec96d446daede230b1e6e1a71003b88062345c48550c128e524aa10160d6a06c41e6c7597309e138d5f0

data/CHANGELOG.md

@@ -0,0 +1,123 @@
+# Changelog
+
+## 0.2.0 (2026-05-04)
+
+### Changed
+
+- use rlz4 ~> 0.5 (ported from lz4_flex to lz4_sys)
+
+## 0.1.0 (2026-04-23)
+
+### Added
+
+- Gem skeleton: `omq-lz4.gemspec`, `lib/omq/lz4.rb`,
+  `lib/omq/lz4/version.rb`, `lib/omq/lz4/errors.rb`, `Gemfile`,
+  `Rakefile`, `.gitignore`, `LICENSE` (ISC), `README.md`, CI and
+  release GitHub Actions workflows, minitest harness. Depends on
+  `rlz4 ~> 0.3` (block API) and `omq ~> 0.23`.
+- `require "omq/lz4"` succeeds and defines `OMQ::LZ4::VERSION` and
+  `OMQ::LZ4::ProtocolError`. No transport behaviour yet — the
+  `lz4+tcp://` scheme registration lands in a subsequent milestone.
+- **`OMQ::LZ4::Codec`** — pure wire-format encode/decode over Strings,
+  no I/O, no connection state. Transport (M2) calls into these per
+  ZMTP part.
+  - `Codec.encode_part(plaintext, block_codec:, min_size: nil)` — tries
+    compression; falls back to passthrough when compression wouldn't
+    save the 8-byte envelope overhead (LZ4B envelope = 12 bytes,
+    UNCOMPRESSED envelope = 4 bytes). `min_size` defaults to 64 if the
+    codec has a dict installed, else 256.
+  - `Codec.decode_part(wire_bytes, block_codec:, max_size: nil)` —
+    handles UNCOMPRESSED (`00 00 00 00`) and LZ4B (`LZ4B` = `4C 5A 34 42`)
+    sentinels. Rejects `max_size` violations before decoder
+    invocation. Raises `ProtocolError` on malformed input; never
+    segfaults or OOMs.
+  - `Codec.encode_dict_shipment(dict_bytes)` /
+    `decode_dict_shipment(wire_bytes)` — LZ4D (`LZ4D` = `4C 5A 34 44`)
+    shipments. Enforces `1 ≤ dict_size ≤ 8192`.
+  - Dict shipments are routed by the transport layer (not by
+    `decode_part`); `decode_part` raises if it ever sees an LZ4D
+    sentinel.
+- **`OMQ::Transport::Lz4Tcp`** — transport plugin registering the
+  `lz4+tcp://` scheme on `OMQ::Engine.transports`. Mirrors
+  `omq-zstd`'s transport class hierarchy.
+  - `Lz4Tcp.listener(endpoint, engine, dict: nil)`,
+    `Lz4Tcp.dialer(endpoint, engine, dict: nil)`,
+    `Lz4Tcp.validate_endpoint!`. Oversized dicts
+    (> `OMQ::LZ4::Codec::MAX_DICT_SIZE` = 8 KiB) raise
+    `OMQ::LZ4::ProtocolError` at bind/connect time.
+  - `Lz4Connection` — `SimpleDelegator` over the ZMTP connection.
+    Per-connection state: send `BlockCodec` (built with the
+    sender-side dict if any), receive `BlockCodec` (initially no-dict,
+    replaced with a dict-bound codec on receipt of a dict shipment),
+    send-side "dict shipped" flag. Intercepts `#send_message`,
+    `#write_message`, `#write_messages`, and `#receive_message` —
+    the ZMTP handshake still runs uncompressed over raw TCP
+    (`connection_class` returns the default `Protocol::ZMTP::Connection`).
+  - Dict shipment is sent as a single-part ZMTP message on the first
+    outgoing send when a sender-side dict is configured; the receiver
+    consumes it silently (never delivered upstack) and installs a
+    dict-bound `BlockCodec` for subsequent decodes. A second shipment
+    on the same direction raises `ProtocolError`.
+  - Per-message size budget (`engine.options.max_message_size`)
+    enforced in `decode_wire_parts` — total decompressed plaintext
+    across multipart parts is summed and compared to the socket's
+    `max_message_size`. Dict shipments do not count against the budget.
+- Integration test (`test/integration_test.rb`) covers: small (below
+  threshold) and large payloads, multipart messages, dict shipment +
+  subsequent compressed messages, both sides configured with a dict,
+  oversized-dict rejection at bind, and a 100k-message soak (gated
+  behind `OMQ_LZ4_STRESS=1`) checking for live-slot leaks after full
+  GC.
+- **Receiver size-budget enforcement.** `engine.options.max_message_size`
+  bounds the total decompressed size of a ZMTP message, summed across
+  MORE-flag-chained parts. Over-budget messages raise
+  `OMQ::LZ4::ProtocolError` from the transport and close the
+  connection — `receive` surfaces `OMQ::SocketDeadError` on the next
+  call. The budget is enforced *before* `LZ4_decompress_safe` is
+  invoked, using the `decompressed_size` field declared in the LZ4B
+  envelope for compressed parts, or the wire part length for
+  UNCOMPRESSED parts; lying-size inputs are caught by
+  `LZ4_decompress_safe`'s own bounds check. Dictionary shipments do
+  not count against the budget. Integration tests cover single-part
+  and multi-part overruns, and a sub-budget multipart positive case.
+- **Second-shipment rejection.** `Lz4Connection#install_recv_dict!`
+  raises `OMQ::LZ4::ProtocolError` if a second LZ4D shipment arrives
+  on the same direction of the same connection (dictionary is
+  install-once per direction). Unit-tested by driving the private
+  method directly; the transport's outgoing path never ships twice,
+  so the rule primarily guards against a malicious peer.
+- **Dict wrong-id detection: intentionally not implemented.** LZ4
+  block format has no dictionary-id field; the transport ships dict
+  bytes only (LZ4D sentinel + bytes), no id appended. A dict mismatch
+  between peers produces garbage plaintext, not an error. Detect at
+  the application layer if needed. See [OMQ-LZ4.plan](../OMQ-LZ4.plan)
+  Open Question #1.
+- **RFC.md** — wire-format specification (scheme, sentinels, part
+  encoding, dict shipment, receiver budget, security considerations,
+  constants). Mirrors `omq-zstd/RFC.md` structure. Status: Draft.
+- **Benchmarks** in `bench/`:
+  - `codec_micro.rb` — `OMQ::LZ4::Codec.encode_part` / `decode_part`
+    microbench across sizes {64 B, 256 B, 1 KiB, 16 KiB, 1 MiB}, with
+    and without dict. Reports wire size, ratio, compress ns,
+    decompress ns.
+  - `transport_throughput.rb` — end-to-end PUSH → PULL over
+    `lz4+tcp://` on loopback, messages per second and µs per
+    round-trip.
+  - `min_compress_size_sweep.rb` — sweeps input size from 8 B to
+    320 B on Lorem-ipsum-like text, reports at which size
+    compressed + LZ4B envelope first beats plaintext + passthrough
+    envelope, with and without a dict. Used to tune the
+    `MIN_COMPRESS_*` thresholds below.
+  - `head_to_head.rb` — side-by-side microbenchmark (pure codec)
+    and transport throughput of `omq-lz4` vs `omq-zstd`, with and
+    without a shared dictionary. Requires `omq-zstd` + `rzstd` in
+    the Gemfile's `:bench` group. Numbers land in README's
+    "Head-to-head vs omq-zstd" section.
+- **Compression thresholds tuned** from the sweep:
+  - `MIN_COMPRESS_NO_DICT`: 256 → **512 B**. The measured crossover
+    for Lorem-ipsum text is ~312 B; we round well above it so the
+    compressor isn't invoked for marginal wins that real-world (less
+    repetitive) payloads would lose anyway.
+  - `MIN_COMPRESS_WITH_DICT`: 64 → **32 B**. Measured crossover with
+    dict is ~20 B (the dict-reference is shorter than the literal
+    payload); 32 leaves a small gap.

data/LICENSE

@@ -0,0 +1,15 @@
+ISC License
+
+Copyright (c) 2026, Patrik Wenger
+
+Permission to use, copy, modify, and/or distribute this software for any
+purpose with or without fee is hereby granted, provided that the above
+copyright notice and this permission notice appear in all copies.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

data/README.md

@@ -0,0 +1,231 @@
+# omq-lz4
+
+[![Gem Version](https://img.shields.io/gem/v/omq-lz4?color=e9573f)](https://rubygems.org/gems/omq-lz4)
+[![License: ISC](https://img.shields.io/badge/License-ISC-blue.svg)](LICENSE)
+[![Ruby](https://img.shields.io/badge/Ruby-%3E%3D%203.3-CC342D?logo=ruby&logoColor=white)](https://www.ruby-lang.org)
+
+> **Status:** 0.1.0 — first landable release. See
+> [RFC.md](RFC.md) for the wire-format spec and
+> [CHANGELOG.md](CHANGELOG.md) for what's in.
+
+LZ4-compressed TCP transport for [OMQ](https://github.com/paddor/omq),
+complementary to [`omq-zstd`](https://github.com/paddor/omq-zstd).
+Pick `lz4+tcp://` instead of `tcp://` or `zstd+tcp://` when you want
+cheap per-message compression with a small per-connection footprint.
+
+## When to pick `lz4+tcp://` over `zstd+tcp://`
+
+LZ4 has no entropy stage (no Huffman, no FSE), ~16 KiB of encoder state
+per connection, and trades a **worse compression ratio** for
+**~4–8× faster encode** and **~3× less memory per connection**.
+
+| | `zstd+tcp://` | `lz4+tcp://` |
+|---|---|---|
+| Encode, 1 KiB, no dict | ~3 µs | ~0.4 µs |
+| Encode, 1 KiB, with dict | ~3.5 µs | ~0.5 µs |
+| Memory per connection | ~256 KiB | ~16 KiB + dict |
+| Ratio, 1 KiB JSON no dict | ~45% | ~65% |
+| Ratio, 1 KiB JSON with dict | ~20% | ~35% |
+| Auto-trained dictionaries | ✓ | — (user-supplied only) |
+
+Pick `omq-lz4` for CPU- or memory-scarce deployments (edge gateways,
+IoT concentrators, high-fanout scenarios where per-connection state
+matters more than ratio). Pick `omq-zstd` for bandwidth-bound
+deployments where CPU is cheap.
+
+## Install
+
+```ruby
+# Gemfile
+gem "omq-lz4"
+```
+
+```sh
+gem install omq-lz4
+```
+
+## Usage
+
+```ruby
+require "omq"
+require "omq/lz4"
+
+pull = OMQ::PULL.new
+push = OMQ::PUSH.new
+
+uri = pull.bind("lz4+tcp://127.0.0.1:0")
+push.connect(uri.to_s)
+
+push << ["hello, compressed world"]
+pull.receive  # => ["hello, compressed world"]
+```
+
+Both peers must use `lz4+tcp://`. A `tcp://` peer cannot talk to an
+`lz4+tcp://` peer — they speak different transports.
+
+### Dictionary compression
+
+Small messages don't compress well on their own. A shared dictionary
+gives 2–5× better ratios on payloads with a common prefix. Supply a
+user-trained dictionary (LZ4 has no auto-training — use `omq-zstd`
+for that):
+
+```ruby
+dict = File.binread("schema.dict")
+push.connect("lz4+tcp://127.0.0.1:5555", dict: dict)
+```
+
+The sender ships the dictionary to the receiver in-band, prefixed
+with the dictionary sentinel (`4C 5A 34 44`, "LZ4D" in ASCII), on
+the first outgoing message. The receiver installs the dictionary
+and decompresses subsequent messages against it. Dictionary size
+is capped at **8 KiB** — tighter than `omq-zstd`'s 64 KiB cap, to
+let constrained peers accept shipments without allocating tens of
+KB of scratch.
+
+### Compression thresholds
+
+To avoid pessimizing tiny frames, the sender skips compression below:
+
+| Mode            | Threshold |
+|-----------------|-----------|
+| No dictionary   | 512 B     |
+| With dictionary | 32 B      |
+
+Below the threshold the part is sent uncompressed (4-byte zero
+sentinel + plaintext).
+
+### Security limits
+
+The receiver bounds decompression by the socket's `max_message_size`
+(the same knob you'd use on a plain `tcp://` socket). It caps the
+**total decompressed size of all parts in a single message**. A peer
+attempting to send an over-budget message drops the connection —
+`OMQ::SocketDeadError` surfaces on the next `receive`.
+
+Independent of that, the dictionary itself is capped at 8 KiB; a
+larger shipment drops the connection.
+
+See the plan roadmap ([../OMQ-LZ4.plan](../OMQ-LZ4.plan)) for
+history and open questions.
+
+## Performance
+
+Measured on x86_64 scalar, Ruby 4.0 + YJIT, on dict-friendly (repeated
+Lorem ipsum prefix) input.
+
+**`OMQ::LZ4::Codec` (pure encode/decode, no I/O):**
+
+| Input size | No dict encode | Dict encode | No dict decode | Dict decode |
+|------------|---------------:|------------:|---------------:|------------:|
+|       64 B |       ~0.9 µs  |     ~1.0 µs |       ~0.4 µs  |     ~0.6 µs |
+|      256 B |       ~1.1 µs  |     ~0.8 µs |       ~0.4 µs  |     ~0.5 µs |
+|    1 KiB   |       ~1.5 µs  |     ~0.9 µs |       ~0.9 µs  |     ~1.0 µs |
+|   16 KiB   |       ~3.2 µs  |     ~2.4 µs |       ~3.9 µs  |     ~3.0 µs |
+|    1 MiB   |      ~89 µs    |    ~87 µs   |     ~173 µs    |   ~303 µs   |
+
+**End-to-end PUSH → PULL over `lz4+tcp://` (loopback):**
+
+| Message size | Throughput |
+|--------------|-----------:|
+|         64 B |  ~67k msg/s |
+|        256 B |  ~94k msg/s |
+|       1 KiB  |  ~92k msg/s |
+
+Run the benchmarks yourself:
+
+```sh
+OMQ_DEV=1 bundle exec ruby --yjit bench/codec_micro.rb
+OMQ_DEV=1 bundle exec ruby --yjit bench/transport_throughput.rb
+OMQ_DEV=1 bundle exec ruby --yjit bench/head_to_head.rb   # lz4 vs zstd
+```
+
+### Head-to-head vs `omq-zstd` and plain `tcp`
+
+End-to-end PUSH → PULL throughput, Ruby 4.0 + YJIT. Input:
+UUID-sprinkled Lorem ipsum — a fresh UUID between each Lorem
+paragraph. Approximates realistic workloads where a schema
+repeats but values vary (event logs, protobuf records, JSON
+events), so a fraction of every message is mandatorily
+incompressible.
+
+The link between PUSH and PULL is loopback, rate-shaped with
+`tc netem rate Xmbit` on `dev lo` to simulate bandwidth-limited
+networks. `zstd+tcp` shown at level `-3` (default, fast) and
+level `3` (tighter ratio, more CPU).
+
+The table below: plaintext MiB/s (application-level throughput)
+and wire MiB/s (bytes on the socket) at **128 KiB** payload,
+across three bandwidth regimes.
+
+| Link                | Metric   |   tcp | lz4+tcp | zstd -3 | zstd 3 |
+|---------------------|----------|------:|--------:|--------:|-------:|
+| **100 Mbit**        | plain    |  11.8 |     105 |     114 | **197**|
+| (cap ≈ 12 MiB/s)    | wire     |  11.8 |      12 |      12 |    12  |
+|                     | speedup  | 1.00× |   8.89× |   9.70× |**16.74×**|
+| **1 Gbit**          | plain    | 117   |     794 | **900** |    603 |
+| (cap ≈ 125 MiB/s)   | wire     | 117   |      93 |      94 |     36 |
+|                     | speedup  | 1.00× |   6.81× |**7.73×**|  5.17× |
+| **Unlimited loopback** | plain | **1 064** |  869 |    972  |    626 |
+| (kernel-copy-bound) | wire     | 1 064 |      99 |     101 |     37 |
+|                     | speedup  | 1.00× |   0.82× |   0.91× |  0.59× |
+
+Three regimes visible:
+
+- **100 Mbit** — all compressed transports saturate wire at
+  ~12 MiB/s. Plaintext = wire-cap × (1 / compression-ratio). The
+  tighter the ratio, the bigger the win: `zstd 3`'s 3% wire ratio
+  translates to a **~17× throughput multiplier** over plain tcp.
+- **1 Gbit** — compressed transports shift from wire-saturated to
+  CPU-limited. `zstd -3` reaches ~75% of wire cap; `zstd 3` only
+  29% (deep CPU-bound). Both beat plain tcp (which is pinned at
+  the wire cap) by **6–8×**. `zstd 3`'s tighter wire no longer
+  helps — there's no wire saturation to trade CPU for.
+- **Unlimited loopback** — no wire cap. All three are
+  CPU-limited. Plain tcp doesn't pay compression CPU, so **skip
+  compression on loopback**.
+
+Rate-shape your own link to reproduce:
+
+```sh
+sudo tc qdisc add dev lo root netem rate 100mbit  # or 1gbit, 10mbit, etc.
+OMQ_DEV=1 bundle exec ruby --yjit bench/head_to_head.rb
+sudo tc qdisc del dev lo root
+```
+
+Or use a `veth` pair in a network namespace so shaping doesn't
+touch your host's real loopback (see `tc-netem(8)`, `ip-netns(8)`).
+
+Full sweeps (8 sizes from 256 B to 512 KiB) for each regime live
+in `bench/head_to_head.rb` output — run it yourself; the
+headline numbers above are stable across repeats but small sizes
+and very large sizes vary a bit run-to-run.
+
+**Takeaway:**
+
+- Pick **`lz4+tcp://`** for bandwidth-limited links (any real
+  network — even 1 Gbit LAN). 6–9× throughput multiplier over
+  plain `tcp`, minimal memory (~16 KiB/connection), modest CPU.
+  Ties or beats `zstd -3` at 1 Gbit; loses the ratio race to
+  `zstd 3` at 100 Mbit and below.
+- Pick **`zstd+tcp://` (level ≥ 3)** when the wire is the
+  precious resource (≤ 100 Mbit links, WAN, or you're paying for
+  egress). **~17× throughput multiplier at 100 Mbit** for 128 KiB
+  messages is hard to argue with.
+- Pick **plain `tcp://`** when the link is *not* the bottleneck
+  (localhost IPC, loopback, datacenter-fast inter-host
+  connections where the bandwidth ceiling is above the CPU's
+  compress/decompress speed — typically 10+ Gbit), or when the
+  payload is already high-entropy (encrypted, already compressed,
+  random binary) and compression only adds overhead.
+
+## Development
+
+```sh
+OMQ_DEV=1 bundle install
+OMQ_DEV=1 bundle exec rake test
+```
+
+## License
+
+[ISC](LICENSE)

data/lib/omq/lz4/codec.rb

@@ -0,0 +1,185 @@
+# frozen_string_literal: true
+
+require "rlz4"
+
+require_relative "errors"
+
+module OMQ
+  module LZ4
+    # Wire format for the lz4+tcp:// transport, encode/decode over
+    # String input/output. Pure functions — no I/O, no connection state.
+    # Transport (M2) owns the connection state and calls into these
+    # methods per ZMTP part.
+    #
+    # Each wire part begins with a 4-byte sentinel:
+    #
+    #   00 00 00 00   uncompressed plaintext
+    #   4C 5A 34 42   LZ4-compressed block ("LZ4B" in ASCII)
+    #   4C 5A 34 44   dictionary shipment ("LZ4D" in ASCII)
+    #
+    # `decode_part` handles UNCOMPRESSED and LZ4B only. Dictionary
+    # shipments are a transport-layer concern: the transport peeks the
+    # first 4 bytes of each incoming wire part, routes LZ4D to
+    # `decode_dict_shipment`, and never hands a shipment to `decode_part`.
+    module Codec
+      UNCOMPRESSED_SENTINEL = "\x00\x00\x00\x00".b.freeze
+      LZ4B_SENTINEL         = "LZ4B".b.freeze
+      LZ4D_SENTINEL         = "LZ4D".b.freeze
+
+      # Size thresholds below which compression isn't worth attempting.
+      # Empirically tuned on Lorem-ipsum-like input via
+      # bench/min_compress_size_sweep.rb: for block-format LZ4 the
+      # crossover where compressed + 12-byte envelope beats
+      # plaintext + 4-byte passthrough envelope sits at ~312 B without
+      # a dict and ~20 B with one. We round up to 512 / 32 so the
+      # machinery isn't invoked for marginal wins where real-world
+      # (less repetitive) payloads would likely fall back to
+      # passthrough anyway. Below the threshold, `encode_part` emits
+      # UNCOMPRESSED directly without touching the compressor.
+      MIN_COMPRESS_NO_DICT   = 512
+      MIN_COMPRESS_WITH_DICT = 32
+
+      # Maximum dictionary size on the wire. A policy choice, not a
+      # protocol limit; tight enough that constrained peers can accept
+      # dicts without allocating tens of KB of scratch.
+      MAX_DICT_SIZE = 8192
+
+      # Envelope sizes:
+      #   UNCOMPRESSED = 4 (sentinel)
+      #   LZ4B         = 4 (sentinel) + 8 (decompressed_size u64 LE)
+      # => switching from passthrough to compressed costs 8 bytes of
+      # envelope overhead. Compression must save more than that to win.
+      COMPRESSED_ENVELOPE   = 12
+      PASSTHROUGH_ENVELOPE  = 4
+
+      module_function
+
+      # Encode one plaintext part to wire bytes. Tries compression; falls
+      # back to passthrough when compression wouldn't save at least the
+      # envelope overhead.
+      #
+      # `block_codec` is an RLZ4::BlockCodec, optionally constructed with
+      # `dict: bytes`. The codec's dict presence is detected via
+      # `#has_dict?` to pick the min-size threshold.
+      #
+      # `min_size` overrides the default threshold. Nil (the default)
+      # picks `MIN_COMPRESS_NO_DICT` for a no-dict codec and
+      # `MIN_COMPRESS_WITH_DICT` for a dict codec.
+      def encode_part(plaintext, block_codec:, min_size: nil)
+        min_size ||= block_codec.has_dict? ? MIN_COMPRESS_WITH_DICT : MIN_COMPRESS_NO_DICT
+
+        return encode_passthrough(plaintext) if plaintext.bytesize < min_size
+
+        compressed = block_codec.compress(plaintext)
+
+        # Net savings = (plaintext + 4) − (compressed + 12) = plaintext − compressed − 8.
+        # If ≤ 0, passthrough wins (or ties — prefer passthrough: one
+        # fewer u64 for the receiver to parse).
+        if compressed.bytesize + COMPRESSED_ENVELOPE >= plaintext.bytesize + PASSTHROUGH_ENVELOPE
+          encode_passthrough(plaintext)
+        else
+          encode_compressed(plaintext.bytesize, compressed)
+        end
+      end
+
+      # Decode one wire part. Returns a plaintext binary String.
+      #
+      # `max_size` is an optional cap on the decompressed size of this
+      # single part; if the declared (LZ4B) or wire (UNCOMPRESSED)
+      # plaintext size exceeds it, raises ProtocolError before any
+      # decoder invocation.
+      #
+      # Does not handle LZ4D shipments; transport must route those to
+      # `decode_dict_shipment` before calling here.
+      def decode_part(wire_bytes, block_codec:, max_size: nil)
+        if wire_bytes.bytesize < 4
+          raise ProtocolError, "wire part too short (< 4 bytes)"
+        end
+
+        sentinel = wire_bytes.byteslice(0, 4)
+        case sentinel
+        when UNCOMPRESSED_SENTINEL
+          payload = wire_bytes.byteslice(4, wire_bytes.bytesize - 4)
+          check_size!(payload.bytesize, max_size)
+          payload
+        when LZ4B_SENTINEL
+          if wire_bytes.bytesize < 12
+            raise ProtocolError, "LZ4B part too short (< 12 bytes, no room for size field)"
+          end
+          decompressed_size = wire_bytes.byteslice(4, 8).unpack1("Q<")
+          check_size!(decompressed_size, max_size)
+          block = wire_bytes.byteslice(12, wire_bytes.bytesize - 12)
+          begin
+            block_codec.decompress(block, decompressed_size: decompressed_size)
+          rescue RLZ4::DecompressError => e
+            raise ProtocolError, "LZ4B decode failed: #{e.message}"
+          end
+        when LZ4D_SENTINEL
+          # Should not reach decode_part; transport should have routed this.
+          raise ProtocolError,
+            "LZ4D dictionary shipment seen at decode_part (transport should route to decode_dict_shipment)"
+        else
+          raise ProtocolError, "unknown sentinel #{sentinel.unpack1("H*")}"
+        end
+      end
+
+      # Encode a dictionary shipment. Returns wire bytes:
+      #   LZ4D sentinel (4 bytes) || dict bytes (1..8192)
+      #
+      # The shipment is a single-part ZMTP message (MORE flag clear)
+      # from the transport's perspective, but that framing is the
+      # transport's responsibility.
+      def encode_dict_shipment(dict_bytes)
+        validate_dict_size!(dict_bytes.bytesize)
+        LZ4D_SENTINEL + dict_bytes
+      end
+
+      # Decode a dictionary shipment. Returns the dict bytes (without
+      # sentinel). Raises ProtocolError if the sentinel is wrong or the
+      # size is out of the [1, 8192] range.
+      def decode_dict_shipment(wire_bytes)
+        if wire_bytes.bytesize < 4
+          raise ProtocolError, "dict shipment too short (< 4 bytes)"
+        end
+        sentinel = wire_bytes.byteslice(0, 4)
+        unless sentinel == LZ4D_SENTINEL
+          raise ProtocolError,
+            "not a dict shipment (sentinel #{sentinel.unpack1("H*")}, expected 4C5A3444)"
+        end
+        dict = wire_bytes.byteslice(4, wire_bytes.bytesize - 4)
+        validate_dict_size!(dict.bytesize)
+        dict
+      end
+
+      class << self
+        private
+
+        def encode_passthrough(plaintext)
+          UNCOMPRESSED_SENTINEL + plaintext
+        end
+
+
+        def encode_compressed(decompressed_size, compressed)
+          LZ4B_SENTINEL + [decompressed_size].pack("Q<") + compressed
+        end
+
+
+        def check_size!(declared_size, max_size)
+          return unless max_size
+          return if declared_size <= max_size
+
+          raise ProtocolError,
+            "part size #{declared_size} exceeds max_size #{max_size}"
+        end
+
+
+        def validate_dict_size!(size)
+          if size < 1 || size > MAX_DICT_SIZE
+            raise ProtocolError,
+              "dict shipment size #{size} out of range [1, #{MAX_DICT_SIZE}]"
+          end
+        end
+      end
+    end
+  end
+end

data/lib/omq/lz4/errors.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+module OMQ
+  module LZ4
+    # Raised when the peer sends bytes that violate the lz4+tcp wire
+    # format: unknown sentinel, a dictionary shipment that exceeds the
+    # size cap, a second dictionary shipment on the same direction, a
+    # per-message size-budget overrun, or a decoder failure on a
+    # compressed part. The transport closes the connection on any
+    # protocol error — never silently drops the offending part.
+    class ProtocolError < StandardError; end
+  end
+end

data/lib/omq/lz4/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module OMQ
+  module LZ4
+    VERSION = "0.2.0"
+  end
+end

data/lib/omq/lz4.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+# OMQ LZ4+TCP transport — adds lz4+tcp:// endpoint support.
+#
+# Complementary to `omq-zstd`: LZ4 block format has no entropy stage
+# (no Huffman, no FSE), ~16 KiB of encoder state per connection, and
+# trades worse compression ratio for ~4–8× faster encode and ~3× less
+# memory. Pick `lz4+tcp://` for CPU- or memory-scarce deployments where
+# the bandwidth savings of zstd aren't worth the per-message CPU.
+#
+# See RFC.md for the wire format (not yet written — scheme is still
+# under development).
+
+require_relative "lz4/version"
+require_relative "lz4/errors"
+require_relative "lz4/codec"
+require_relative "transport/lz4_tcp"

data/lib/omq/transport/lz4_tcp/connection.rb

@@ -0,0 +1,149 @@
+# frozen_string_literal: true
+
+require "delegate"
+
+require_relative "../../lz4/codec"
+
+module OMQ
+  module Transport
+    module Lz4Tcp
+      # Per-connection state + encode/decode hooks. A SimpleDelegator over
+      # the ZMTP connection so send_message / write_message /
+      # receive_message route through compression, but everything else
+      # (write_command, close, etc.) passes through untouched.
+      class Lz4Connection < SimpleDelegator
+        # @return [Integer, nil] wire bytesize of the last received
+        #   message (sum across parts, compressed sentinel included).
+        attr_reader :last_wire_size_in
+
+        def initialize(conn, send_dict_bytes:, max_message_size:)
+          super(conn)
+          @max_message_size  = max_message_size
+
+          # Per-direction state. The send codec is built at construction
+          # with the send-side dictionary (if any) baked in. The receive
+          # codec starts dict-less; if/when a dict shipment arrives on
+          # this direction, it is replaced with a dict-bound codec.
+          @send_dict_bytes   = send_dict_bytes&.b
+          @send_codec        = build_block_codec(@send_dict_bytes)
+          @send_dict_shipped = @send_dict_bytes.nil?  # nothing to ship => "already shipped"
+
+          @recv_codec        = build_block_codec(nil)
+          @recv_dict_bytes   = nil
+
+          @last_wire_size_in = nil
+        end
+
+
+        def send_message(parts)
+          wire = encode_parts(parts)
+          ship_send_dict!
+          __getobj__.send_message(wire)
+        end
+
+
+        def write_message(parts)
+          wire = encode_parts(parts)
+          ship_send_dict!
+          __getobj__.write_message(wire)
+        end
+
+
+        def write_messages(messages)
+          wires = messages.map { |parts| encode_parts(parts) }
+          ship_send_dict!
+          __getobj__.write_messages(wires)
+        end
+
+
+        def receive_message
+          # Loop: a dict shipment is consumed silently and we read the
+          # next ZMTP message. Only data messages are returned to the
+          # caller. Budget tracking happens inside decode_wire_parts.
+          loop do
+            parts   = __getobj__.receive_message
+            decoded = decode_wire_parts(parts)
+            if decoded
+              @last_wire_size_in = parts.sum(&:bytesize)
+              return decoded
+            end
+          end
+        end
+
+
+        private
+
+
+        def build_block_codec(dict_bytes)
+          if dict_bytes
+            RLZ4::BlockCodec.new(dict: dict_bytes)
+          else
+            RLZ4::BlockCodec.new
+          end
+        end
+
+
+        def encode_parts(parts)
+          parts.map do |pt|
+            bytes = pt.is_a?(String) && pt.encoding == Encoding::BINARY ? pt : pt.to_s.b
+            LZ4::Codec.encode_part(bytes, block_codec: @send_codec)
+          end
+        end
+
+
+        def ship_send_dict!
+          return if @send_dict_shipped
+
+          shipment = LZ4::Codec.encode_dict_shipment(@send_dict_bytes)
+          __getobj__.write_message([shipment])
+          @send_dict_shipped = true
+        end
+
+
+        # Returns an array of plaintext parts, or nil if the whole
+        # ZMTP message was a dict shipment (consumed silently).
+        #
+        # Budget tracking is per-message (sum of decompressed sizes
+        # across parts). Dict shipment parts do not count against the
+        # budget — they aren't messages.
+        def decode_wire_parts(parts)
+          decoded   = []
+          all_dicts = true
+          budget    = @max_message_size
+
+          parts.each do |wire|
+            raise LZ4::ProtocolError, "wire part too short (< 4 bytes)" if wire.bytesize < 4
+
+            sentinel = wire.byteslice(0, 4)
+            if sentinel == LZ4::Codec::LZ4D_SENTINEL
+              install_recv_dict!(wire)
+              next
+            end
+
+            all_dicts = false
+            plaintext = LZ4::Codec.decode_part(wire, block_codec: @recv_codec, max_size: budget)
+            budget -= plaintext.bytesize if budget
+            decoded << plaintext
+          end
+
+          all_dicts ? nil : decoded
+        end
+
+
+        def install_recv_dict!(wire)
+          if @recv_dict_bytes
+            raise LZ4::ProtocolError, "second dictionary shipment on the same direction"
+          end
+
+          dict_bytes = LZ4::Codec.decode_dict_shipment(wire)
+          # Replace the no-dict recv codec with a dict-bound one. The
+          # old codec is GC'd. Per rlz4: "fresh codec when dict is
+          # needed" — BlockCodec's dict is a permanent property.
+          @recv_codec      = RLZ4::BlockCodec.new(dict: dict_bytes)
+          @recv_dict_bytes = dict_bytes
+        end
+
+      end
+    end
+  end
+end

data/lib/omq/transport/lz4_tcp/transport.rb

@@ -0,0 +1,207 @@
+# frozen_string_literal: true
+
+require "socket"
+require "uri"
+require "io/stream"
+require "omq"
+
+require_relative "connection"
+
+module OMQ
+  module Transport
+    module Lz4Tcp
+      SCHEME = "lz4+tcp"
+
+      class << self
+        # Called by OMQ::Engine::ConnectionLifecycle after the ZMTP
+        # handshake completes; we return the default connection class
+        # so that handshake itself runs uncompressed over raw TCP.
+        def connection_class
+          Protocol::ZMTP::Connection
+        end
+
+
+        # Creates a bound lz4+tcp listener.
+        #
+        # @param endpoint [String] e.g. "lz4+tcp://127.0.0.1:0"
+        # @param engine [OMQ::Engine]
+        # @param dict [String, nil] user-supplied dictionary bytes to
+        #   ship on the first outgoing message. If nil, no dict is
+        #   shipped (payloads compress without dict or go plaintext
+        #   below the min-size threshold).
+        # @return [Listener]
+        def listener(endpoint, engine, dict: nil, **)
+          validate_dict!(dict)
+
+          host, port  = parse_endpoint(endpoint)
+          lookup_host = normalize_bind_host(host)
+          servers     = ::Socket.tcp_server_sockets(lookup_host, port)
+
+          if servers.empty?
+            raise ::Socket::ResolutionError, "no addresses for #{host.inspect}"
+          end
+
+          actual_port  = servers.first.local_address.ip_port
+          display_host = host == "*" ? "*" : (lookup_host || "*")
+          host_part    = display_host.include?(":") ? "[#{display_host}]" : display_host
+          resolved     = "#{SCHEME}://#{host_part}:#{actual_port}"
+
+          Listener.new(resolved, servers, actual_port, engine, dict&.b)
+        end
+
+
+        # Creates an lz4+tcp dialer for an endpoint.
+        #
+        # @param endpoint [String]
+        # @param engine [OMQ::Engine]
+        # @param dict [String, nil] user-supplied dictionary bytes.
+        # @return [Dialer]
+        def dialer(endpoint, engine, dict: nil, **)
+          validate_dict!(dict)
+          Dialer.new(endpoint, engine, dict&.b)
+        end
+
+
+        def validate_endpoint!(endpoint)
+          host, _port = parse_endpoint(endpoint)
+          lookup_host = normalize_connect_host(host)
+          Addrinfo.getaddrinfo(lookup_host, nil, nil, :STREAM) if lookup_host
+        end
+
+
+        def parse_endpoint(endpoint)
+          uri = URI.parse(endpoint)
+          [uri.hostname, uri.port]
+        end
+
+
+        def normalize_bind_host(host)
+          case host
+          when "*" then nil
+          when nil, "", "localhost" then TCP.loopback_host
+          else host
+          end
+        end
+
+
+        def normalize_connect_host(host)
+          case host
+          when nil, "", "*", "localhost" then TCP.loopback_host
+          else host
+          end
+        end
+
+
+        def connect_timeout(options)
+          ri = options.reconnect_interval
+          ri = ri.end if ri.is_a?(Range)
+          [ri, 0.5].max
+        end
+
+
+        private
+
+
+        def validate_dict!(dict)
+          return if dict.nil?
+
+          size = dict.bytesize
+          return if size >= 1 && size <= LZ4::Codec::MAX_DICT_SIZE
+
+          raise LZ4::ProtocolError,
+            "dict size #{size} out of range [1, #{LZ4::Codec::MAX_DICT_SIZE}]"
+        end
+      end
+
+
+      # Dialer: outgoing connections.
+      class Dialer
+        attr_reader :endpoint
+
+        def initialize(endpoint, engine, dict_bytes)
+          @endpoint   = endpoint
+          @engine     = engine
+          @dict_bytes = dict_bytes
+        end
+
+
+        def connect
+          host, port = Lz4Tcp.parse_endpoint(@endpoint)
+          host       = Lz4Tcp.normalize_connect_host(host)
+          sock       = ::Socket.tcp(host, port, connect_timeout: Lz4Tcp.connect_timeout(@engine.options))
+
+          TCP.apply_buffer_sizes(sock, @engine.options)
+
+          @engine.handle_connected(IO::Stream::Buffered.wrap(sock), endpoint: @endpoint)
+        rescue
+          sock&.close
+          raise
+        end
+
+
+        def wrap_connection(conn)
+          Lz4Connection.new(
+            conn,
+            send_dict_bytes:  @dict_bytes,
+            max_message_size: @engine.options.max_message_size,
+          )
+        end
+      end
+
+
+      # Listener: bound server accepting incoming connections.
+      class Listener
+        attr_reader :endpoint, :port
+
+        def initialize(endpoint, servers, port, engine, dict_bytes)
+          @endpoint   = endpoint
+          @servers    = servers
+          @port       = port
+          @engine     = engine
+          @dict_bytes = dict_bytes
+          @tasks      = []
+        end
+
+
+        def wrap_connection(conn)
+          Lz4Connection.new(
+            conn,
+            send_dict_bytes:  @dict_bytes,
+            max_message_size: @engine.options.max_message_size,
+          )
+        end
+
+
+        def start_accept_loops(parent_task, &on_accepted)
+          @tasks = @servers.map do |server|
+            parent_task.async(transient: true, annotation: "#{SCHEME} accept #{@endpoint}") do
+              loop do
+                client, _addr = server.accept
+
+                Async::Task.current.defer_stop do
+                  TCP.apply_buffer_sizes(client, @engine.options)
+
+                  stream = IO::Stream::Buffered.wrap(client)
+
+                  on_accepted.call(stream)
+                end
+              end
+            rescue Async::Stop
+            rescue IOError
+            ensure
+              server.close rescue nil
+            end
+          end
+        end
+
+
+        def stop
+          @tasks.each(&:stop)
+          @servers.each { |s| s.close rescue nil }
+        end
+      end
+
+      Engine.transports[SCHEME] = self
+    end
+  end
+end

data/lib/omq/transport/lz4_tcp.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+# OMQ LZ4+TCP transport — adds lz4+tcp:// endpoint support.
+#
+# Usage:
+#   require "omq/transport/lz4_tcp"
+#
+#   push = OMQ::PUSH.new
+#   push.connect("lz4+tcp://127.0.0.1:5555", dict: File.binread("my.dict"))
+#   # or without a dictionary:
+#   push.connect("lz4+tcp://127.0.0.1:5555")
+
+require "omq"
+require "rlz4"
+
+require_relative "../lz4/errors"
+require_relative "../lz4/codec"
+require_relative "lz4_tcp/transport"

metadata

@@ -0,0 +1,80 @@
+--- !ruby/object:Gem::Specification
+name: omq-lz4
+version: !ruby/object:Gem::Version
+  version: 0.2.0
+platform: ruby
+authors:
+- Patrik Wenger
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: omq
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.23'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.23'
+- !ruby/object:Gem::Dependency
+  name: rlz4
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.5'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.5'
+description: 'Adds lz4+tcp:// endpoint support to OMQ with per-part LZ4 block-format
+  compression, bounded decompression, and in-band dictionary shipping. Complementary
+  to omq-zstd: worse ratio, far faster encode, far smaller per-connection footprint.'
+email:
+- paddor@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE
+- README.md
+- lib/omq/lz4.rb
+- lib/omq/lz4/codec.rb
+- lib/omq/lz4/errors.rb
+- lib/omq/lz4/version.rb
+- lib/omq/transport/lz4_tcp.rb
+- lib/omq/transport/lz4_tcp/connection.rb
+- lib/omq/transport/lz4_tcp/transport.rb
+homepage: https://github.com/paddor/omq-lz4
+licenses:
+- ISC
+metadata: {}
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.3'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.6
+specification_version: 4
+summary: LZ4+TCP transport for OMQ
+test_files: []