omq-lz4 0.3.0 → 0.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4217a939f757a07b0e3e2ad1a905d11ea3cd017e54baa2f059a75f304386bdab
-  data.tar.gz: 1d0f4e4076e0cf02291b3f714fe20c721b8e05e0d4d17af4fbc8662e83897319
+  metadata.gz: b14469227b64cb647b42dbb8495754752ea81654449eab28cae78a0778aa36dd
+  data.tar.gz: 344189a2d0c29ec2d46b310f5607d133bdba87a69fb974be6bb582101f76aacf
 SHA512:
-  metadata.gz: a963f09065ef7a019a8b766a00093e5ef84ba9489fed66d3524da5e8a0d8d956d27cd01534926ac1cb3f5c5088439da9e0db5c5f71c0b7b58b7b90db8ccfc8bf
-  data.tar.gz: 86e7b5fe26c080a13a6b5c5c44b0e0a250bc45d650b2a9f76f4a80330209d6e24374b8a6412dd28118cb971619f56d3bede7a5ed937c91c5a80fd520ab17b5ac
+  metadata.gz: c3ef67d35613434faf125317a61dded584c8d268032eb01830e22a809415ab20b2cf198daa8ee7803d04e88b97d4b39bb6042071a7b18bbf8cff03cb6db06cee
+  data.tar.gz: 49ed574d2b93d2e039160514602cf4f90e125edd9c2b3e05033fbbd5ee1dc1eb098373acaed27c54553e9e6977d6488d3f751b85544ad87a6f0498e5e061cad5

data/CHANGELOG.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## [Unreleased]
+
+## 0.3.1 (2026-05-28)
+
+### Changed
+
+- `MIN_COMPRESS_WITH_DICT`: raised from 32 to 128. The previous value
+  was too aggressive; 128 leaves a safer margin above the measured
+  crossover.
+
 ## 0.3.0 (2026-05-11)
 
 ### Added

data/README.md

@@ -4,19 +4,19 @@
 [![License: ISC](https://img.shields.io/badge/License-ISC-blue.svg)](LICENSE)
 [![Ruby](https://img.shields.io/badge/Ruby-%3E%3D%203.3-CC342D?logo=ruby&logoColor=white)](https://www.ruby-lang.org)
 
-> [RFC.md](RFC.md) for the wire-format spec and
-> [CHANGELOG.md](CHANGELOG.md) for what's in.
-
 LZ4-compressed TCP transport for [OMQ](https://github.com/paddor/omq),
 complementary to [`omq-zstd`](https://github.com/paddor/omq-zstd).
 Pick `lz4+tcp://` instead of `tcp://` or `zstd+tcp://` when you want
 cheap per-message compression with a small per-connection footprint.
 
+See [RFC.md](RFC.md) for the wire-format specification and
+[CHANGELOG.md](CHANGELOG.md) for release history.
+
 ## When to pick `lz4+tcp://` over `zstd+tcp://`
 
 LZ4 has no entropy stage (no Huffman, no FSE), ~16 KiB of encoder state
 per connection, and trades a **worse compression ratio** for
-**~4–8× faster encode** and **~3× less memory per connection**.
+**~4-8x faster encode** and **~3x less memory per connection**.
 
 | | `zstd+tcp://` | `lz4+tcp://` |
 |---|---|---|
@@ -25,7 +25,7 @@ per connection, and trades a **worse compression ratio** for
 | Memory per connection | ~256 KiB | ~16 KiB + dict |
 | Ratio, 1 KiB JSON no dict | ~45% | ~65% |
 | Ratio, 1 KiB JSON with dict | ~20% | ~35% |
-| Auto-trained dictionaries | ✓ | — (user-supplied only) |
+| Auto-trained dictionaries | yes | no (user-supplied only) |
 
 Pick `omq-lz4` for CPU- or memory-scarce deployments (edge gateways,
 IoT concentrators, high-fanout scenarios where per-connection state
@@ -60,13 +60,13 @@ pull.receive  # => ["hello, compressed world"]
 ```
 
 Both peers must use `lz4+tcp://`. A `tcp://` peer cannot talk to an
-`lz4+tcp://` peer — they speak different transports.
+`lz4+tcp://` peer. They speak different transports.
 
 ### Dictionary compression
 
 Small messages don't compress well on their own. A shared dictionary
-gives 2–5× better ratios on payloads with a common prefix. Supply a
-user-trained dictionary (LZ4 has no auto-training — use `omq-zstd`
+gives 2-5x better ratios on payloads with a common prefix. Supply a
+user-trained dictionary (LZ4 has no auto-training; use `omq-zstd`
 for that):
 
 ```ruby
@@ -78,9 +78,7 @@ The sender ships the dictionary to the receiver in-band, prefixed
 with the dictionary sentinel (`4C 5A 34 44`, "LZ4D" in ASCII), on
 the first outgoing message. The receiver installs the dictionary
 and decompresses subsequent messages against it. Dictionary size
-is capped at **8 KiB** — tighter than `omq-zstd`'s 64 KiB cap, to
-let constrained peers accept shipments without allocating tens of
-KB of scratch.
+is capped at **8 KiB** (same cap as `omq-zstd`).
 
 ### Compression thresholds
 
@@ -89,7 +87,7 @@ To avoid pessimizing tiny frames, the sender skips compression below:
 | Mode            | Threshold |
 |-----------------|-----------|
 | No dictionary   | 512 B     |
-| With dictionary | 32 B      |
+| With dictionary | 128 B     |
 
 Below the threshold the part is sent uncompressed (4-byte zero
 sentinel + plaintext).
@@ -99,14 +97,54 @@ sentinel + plaintext).
 The receiver bounds decompression by the socket's `max_message_size`
 (the same knob you'd use on a plain `tcp://` socket). It caps the
 **total decompressed size of all parts in a single message**. A peer
-attempting to send an over-budget message drops the connection —
+attempting to send an over-budget message drops the connection.
 `OMQ::SocketDeadError` surfaces on the next `receive`.
 
 Independent of that, the dictionary itself is capped at 8 KiB; a
 larger shipment drops the connection.
 
-See the plan roadmap ([../OMQ-LZ4.plan](../OMQ-LZ4.plan)) for
-history and open questions.
+## Wire format
+
+Every post-handshake ZMTP message part starts with a 4-byte sentinel:
+
+| Sentinel (hex) | ASCII | Meaning |
+|---|---|---|
+| `00 00 00 00` | (none) | Uncompressed plaintext |
+| `4C 5A 34 42` | `LZ4B` | LZ4-compressed single block |
+| `4C 5A 34 4D` | `LZ4M` | LZ4-compressed multi-block |
+| `4C 5A 34 44` | `LZ4D` | Dictionary shipment |
+
+**Single-block** (`LZ4B`): `sentinel (4) || decompressed_size u64 LE (8) || LZ4 block bytes`.
+12-byte envelope. Raw LZ4 block format (no magic, no descriptor, no
+checksum). `decompressed_size` is required because LZ4 block format
+carries no length prefix; the receiver pre-sizes its output buffer.
+
+**Multi-block** (`LZ4M`): same header, followed by a sequence of
+`u32 LE compressed_block_len || LZ4 block bytes` pairs. Each block
+decompresses independently at up to 1 GiB. Used for parts exceeding
+the single-block size cap.
+
+**Dictionary shipment** (`LZ4D`): `sentinel (4) || dict bytes (1..8192)`.
+Single-part ZMTP message consumed by the transport, not delivered
+to the application. At most one per direction per connection.
+
+Any other leading 4 bytes close the connection.
+
+## Constants
+
+| Constant | Value |
+|---|---|
+| Scheme | `lz4+tcp` |
+| Uncompressed sentinel | `00 00 00 00` |
+| Single-block sentinel | `4C 5A 34 42` (`LZ4B`) |
+| Multi-block sentinel | `4C 5A 34 4D` (`LZ4M`) |
+| Dictionary sentinel | `4C 5A 34 44` (`LZ4D`) |
+| LZ4M block size | 1 GiB (`0x40000000`) |
+| Max dictionary size | 8 KiB |
+| Min compress, no dict | 512 B |
+| Min compress, with dict | 128 B |
+| LZ4B envelope | 12 B (4 sentinel + 8 size) |
+| Uncompressed envelope | 4 B (sentinel only) |
 
 ## Performance
 
@@ -123,7 +161,7 @@ Lorem ipsum prefix) input.
 |   16 KiB   |       ~3.2 µs  |     ~2.4 µs |       ~3.9 µs  |     ~3.0 µs |
 |    1 MiB   |      ~89 µs    |    ~87 µs   |     ~173 µs    |   ~303 µs   |
 
-**End-to-end PUSH → PULL over `lz4+tcp://` (loopback):**
+**End-to-end PUSH -> PULL over `lz4+tcp://` (loopback):**
 
 | Message size | Throughput |
 |--------------|-----------:|
@@ -141,8 +179,8 @@ OMQ_DEV=1 bundle exec ruby --yjit bench/head_to_head.rb   # lz4 vs zstd
 
 ### Head-to-head vs `omq-zstd` and plain `tcp`
 
-End-to-end PUSH → PULL throughput, Ruby 4.0 + YJIT. Input:
-UUID-sprinkled Lorem ipsum — a fresh UUID between each Lorem
+End-to-end PUSH -> PULL throughput, Ruby 4.0 + YJIT. Input:
+UUID-sprinkled Lorem ipsum, a fresh UUID between each Lorem
 paragraph. Approximates realistic workloads where a schema
 repeats but values vary (event logs, protobuf records, JSON
 events), so a fraction of every message is mandatorily
@@ -160,27 +198,27 @@ across three bandwidth regimes.
 | Link                | Metric   |   tcp | lz4+tcp | zstd -3 | zstd 3 |
 |---------------------|----------|------:|--------:|--------:|-------:|
 | **100 Mbit**        | plain    |  11.8 |     105 |     114 | **197**|
-| (cap ≈ 12 MiB/s)    | wire     |  11.8 |      12 |      12 |    12  |
-|                     | speedup  | 1.00× |   8.89× |   9.70× |**16.74×**|
+| (cap ~12 MiB/s)     | wire     |  11.8 |      12 |      12 |    12  |
+|                     | speedup  | 1.00x |   8.89x |   9.70x |**16.74x**|
 | **1 Gbit**          | plain    | 117   |     794 | **900** |    603 |
-| (cap ≈ 125 MiB/s)   | wire     | 117   |      93 |      94 |     36 |
-|                     | speedup  | 1.00× |   6.81× |**7.73×**|  5.17× |
+| (cap ~125 MiB/s)    | wire     | 117   |      93 |      94 |     36 |
+|                     | speedup  | 1.00x |   6.81x |**7.73x**|  5.17x |
 | **Unlimited loopback** | plain | **1 064** |  869 |    972  |    626 |
 | (kernel-copy-bound) | wire     | 1 064 |      99 |     101 |     37 |
-|                     | speedup  | 1.00× |   0.82× |   0.91× |  0.59× |
+|                     | speedup  | 1.00x |   0.82x |   0.91x |  0.59x |
 
 Three regimes visible:
 
-- **100 Mbit** — all compressed transports saturate wire at
-  ~12 MiB/s. Plaintext = wire-cap × (1 / compression-ratio). The
+- **100 Mbit**: all compressed transports saturate wire at
+  ~12 MiB/s. Plaintext = wire-cap x (1 / compression-ratio). The
   tighter the ratio, the bigger the win: `zstd 3`'s 3% wire ratio
-  translates to a **~17× throughput multiplier** over plain tcp.
-- **1 Gbit** — compressed transports shift from wire-saturated to
+  translates to a **~17x throughput multiplier** over plain tcp.
+- **1 Gbit**: compressed transports shift from wire-saturated to
   CPU-limited. `zstd -3` reaches ~75% of wire cap; `zstd 3` only
   29% (deep CPU-bound). Both beat plain tcp (which is pinned at
-  the wire cap) by **6–8×**. `zstd 3`'s tighter wire no longer
-  helps — there's no wire saturation to trade CPU for.
-- **Unlimited loopback** — no wire cap. All three are
+  the wire cap) by **6-8x**. `zstd 3`'s tighter wire no longer
+  helps; there's no wire saturation to trade CPU for.
+- **Unlimited loopback**: no wire cap. All three are
   CPU-limited. Plain tcp doesn't pay compression CPU, so **skip
   compression on loopback**.
 
@@ -196,25 +234,25 @@ Or use a `veth` pair in a network namespace so shaping doesn't
 touch your host's real loopback (see `tc-netem(8)`, `ip-netns(8)`).
 
 Full sweeps (8 sizes from 256 B to 512 KiB) for each regime live
-in `bench/head_to_head.rb` output — run it yourself; the
+in `bench/head_to_head.rb` output. Run it yourself; the
 headline numbers above are stable across repeats but small sizes
 and very large sizes vary a bit run-to-run.
 
 **Takeaway:**
 
 - Pick **`lz4+tcp://`** for bandwidth-limited links (any real
-  network — even 1 Gbit LAN). 6–9× throughput multiplier over
+  network, even 1 Gbit LAN). 6-9x throughput multiplier over
   plain `tcp`, minimal memory (~16 KiB/connection), modest CPU.
   Ties or beats `zstd -3` at 1 Gbit; loses the ratio race to
   `zstd 3` at 100 Mbit and below.
-- Pick **`zstd+tcp://` (level ≥ 3)** when the wire is the
-  precious resource (≤ 100 Mbit links, WAN, or you're paying for
-  egress). **~17× throughput multiplier at 100 Mbit** for 128 KiB
-  messages is hard to argue with.
+- Pick **`zstd+tcp://` (level >= 3)** when the wire is the
+  precious resource (100 Mbit links or slower, WAN, or you're
+  paying for egress). **~17x throughput multiplier at 100 Mbit**
+  for 128 KiB messages is hard to argue with.
 - Pick **plain `tcp://`** when the link is *not* the bottleneck
   (localhost IPC, loopback, datacenter-fast inter-host
   connections where the bandwidth ceiling is above the CPU's
-  compress/decompress speed — typically 10+ Gbit), or when the
+  compress/decompress speed, typically 10+ Gbit), or when the
   payload is already high-entropy (encrypted, already compressed,
   random binary) and compression only adds overhead.
 

data/lib/omq/lz4/codec.rb

@@ -41,7 +41,7 @@ module OMQ
       # passthrough anyway. Below the threshold, `encode_part` emits
       # UNCOMPRESSED directly without touching the compressor.
       MIN_COMPRESS_NO_DICT   = 512
-      MIN_COMPRESS_WITH_DICT = 32
+      MIN_COMPRESS_WITH_DICT = 128
 
       # Maximum dictionary size on the wire. A policy choice, not a
       # protocol limit; tight enough that constrained peers can accept

data/lib/omq/lz4/version.rb

@@ -2,6 +2,6 @@
 
 module OMQ
   module LZ4
-    VERSION = "0.3.0"
+    VERSION = "0.3.1"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: omq-lz4
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.3.1
 platform: ruby
 authors:
 - Patrik Wenger
@@ -74,7 +74,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.6
+rubygems_version: 4.0.10
 specification_version: 4
 summary: LZ4+TCP transport for OMQ
 test_files: []