multi_compress 0.3.1 → 0.3.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 404264c1ba462057374ad7a6a2a72565a16e1da607e47b1d3d91b727b4e74351
-  data.tar.gz: d23f9c3af016742a7aef3880209a2e68ba5ed2107c6c3eb3943a30802606c6ac
+  metadata.gz: e6af5cef6e9b15c5b0e11a148b0543377c0ad332e710083950bca7cb3d49e70e
+  data.tar.gz: 6722c74ed13eabcdb3a4a7ca81cd12b9d3f286f11d00e73de759861db198f2a2
 SHA512:
-  metadata.gz: 91091ad2e184758916b7c79aa0ec82a3216f5d87798a7c597b43afe25b8eaa091fd7094c5de71e4eac5e01804f7b9928734514224cd0312edd1f60187d04c91b
-  data.tar.gz: 275044013ecacca6ee300ce8b980c252048f021acc2cf51e169722830add7301dbc4e9f6942c07845ea672c5e1dd9fe8074c7840b155096c3b40b2b1e3d5ebdc
+  metadata.gz: bca66c39d081538682abd1a45862f4d141605f028b3f233ec886d21f3b38c84000956275b73303c00fda5755f37796c3bcd8bb533db32a0f12b272107ed298c1
+  data.tar.gz: 209cebbbbc4be03bd48eea3d59ec1254502db4a8a5e9109b5f03425cc2f05d474ab7f250d2a2ecfaba36f1c6ac9686b159cf0433bb9c50bd84aecffe62ab365e

data/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## [0.3.3]
+
+### Changed
+- Updated vendored native libraries to **zstd 1.5.7** and **brotli 1.2.0**.
+- Updated README / GET_STARTED examples so documented runtime versions match `MultiCompress.version(...)`.
+- Added third-party license notices for the vendored C libraries.
+
+## [0.3.2]
+
+### Changed
+- Micro-optimization
+
 ## [0.3.1]
 
 ### Fixed
@@ -54,9 +66,9 @@
 - Optional `./build.sh --test`, `./build.sh --valgrind`, and `./build.sh --sanitize` modes
 
 ### Changed
-- Vendored **zstd** is pinned to **1.5.2** in the current release line.
-  - This restores reliable `MultiCompress::Zstd.train_dictionary(...)` behavior on tested platforms, including arm64-darwin.
-  - Newer vendored zstd versions are temporarily avoided until the dictionary training regression is understood and fixed upstream.
+- Historical note for the 0.2.2 release line: vendored **zstd** was pinned to **1.5.2**.
+  - This restored reliable `MultiCompress::Zstd.train_dictionary(...)` behavior on tested platforms at the time, including arm64-darwin.
+  - Later releases moved forward after the native-library upgrade path was revalidated.
 
 ### Changed
 - Documentation now matches current runtime behavior:

data/GET_STARTED.md

@@ -400,7 +400,7 @@ end
 
 Dramatically improves compression on small, similar data (JSON APIs, configs, logs).
 
-**Important**: Dictionary training is available for **Zstd** in the current release (vendored zstd **1.5.2**). Brotli dictionaries can be used, but this gem does not expose Brotli training through `train_dictionary`.
+**Important**: Dictionary training is available for **Zstd** in the current release (vendored zstd **1.5.7**). Brotli dictionaries can be used, but this gem does not expose Brotli training through `train_dictionary`.
 
 ### Training Dictionary (Zstd)
 
@@ -677,9 +677,9 @@ MultiCompress.available?(:zstd)    # => true
 MultiCompress.available?(:fake)    # => false
 
 # Get library versions
-MultiCompress.version(:zstd)       # => "1.5.2"
+MultiCompress.version(:zstd)       # => "1.5.7"
 MultiCompress.version(:lz4)        # => "1.10.0"
-MultiCompress.version(:brotli)     # => "1.1.0"
+MultiCompress.version(:brotli)     # => "1.2.0"
 ```
 
 ### Data Integrity

data/README.md

@@ -7,9 +7,9 @@
 Modern compression technology: **zstd**, **lz4**, **brotli** — unified compression platform with native C performance, **fiber-friendly** for modern async Ruby stacks.
 
 Bundled library versions in the current release:
-- **zstd 1.5.2**
+- **zstd 1.5.7**
 - **lz4 1.10.0**
-- **brotli 1.1.0**
+- **brotli 1.2.0**
 
 📖 **[Get Started →](GET_STARTED.md)** — Complete technology overview, algorithms, and implementation details
 
@@ -88,70 +88,79 @@ If `max_output_size:` is omitted, one-shot calls use `MultiCompress.config.max_o
 
 ## Benchmark Results
 
-> **📝 Note on v0.2.0**: Performance numbers below are from the v0.2.0 build with fiber-friendly paths enabled. There is no throughput regression compared to v0.1.2 — the fiber-friendly path is only taken when a `Fiber::Scheduler` is active, and even then the worker-thread overhead is negligible for payloads large enough to benefit.
-
-Performance comparison against Ruby's built-in zlib compression (200 iterations per test):
-
-### 🗜️  COMPRESSION RATIO (%, lower is better)
-```
-┌─────────────────────────────┬─────────┬─────────┬─────────┬─────────┐
-│ Configuration               │  zlib   │   lz4   │  zstd   │ brotli  │
-├─────────────────────────────┼─────────┼─────────┼─────────┼─────────┤
-│ Small JSON (~10KB, GC)      │    9.4% │   16.1% │    6.9% │    6.1% │
-│ Small text (~10KB, GC)      │    3.1% │    4.6% │    3.2% │    2.6% │
-│ Small JSON (~10KB, no GC)   │    9.4% │   16.1% │    6.9% │    6.1% │
-│ Small text (~10KB, no GC)   │    3.1% │    4.6% │    3.2% │    2.6% │
-│ Medium JSON (~370KB, GC)    │    8.5% │   15.7% │    6.7% │    5.5% │
-│ Medium logs (~168KB, GC)    │    8.6% │   17.2% │    5.4% │    3.2% │
-│ Medium JSON (~370KB, no GC) │    8.5% │   15.7% │    6.7% │    5.5% │
-│ Medium logs (~168KB, no GC) │    8.6% │   17.2% │    5.4% │    3.2% │
-│ Large JSON (~1.6MB, GC)     │    8.1% │   15.1% │    6.1% │    5.6% │
-│ Large logs (~600KB, GC)     │    7.6% │   16.0% │    2.8% │    2.1% │
-│ Large JSON (~1.6MB, no GC)  │    8.1% │   15.1% │    6.1% │    5.6% │
-│ Large logs (~600KB, no GC)  │    7.6% │   16.0% │    2.8% │    2.1% │
-└─────────────────────────────┴─────────┴─────────┴─────────┴─────────┘
-```
-
-### ⚡ TOTAL TIME (compress + decompress, ms — lower is faster)
-```
-┌─────────────────────────────┬─────────┬─────────┬─────────┬─────────┐
-│ Configuration               │  zlib   │   lz4   │  zstd   │ brotli  │
-├─────────────────────────────┼─────────┼─────────┼─────────┼─────────┤
-│ Small JSON (~10KB, GC)      │    0.05 │    0.01 │    0.02 │    0.14 │
-│ Small text (~10KB, GC)      │    0.04 │    0.00 │    0.01 │    0.11 │
-│ Small JSON (~10KB, no GC)   │    0.06 │    0.01 │    0.02 │    0.13 │
-│ Small text (~10KB, no GC)   │    0.04 │    0.00 │    0.01 │    0.11 │
-│ Medium JSON (~370KB, GC)    │    2.73 │    0.29 │    0.42 │    2.36 │
-│ Medium logs (~168KB, GC)    │    1.23 │    0.14 │    0.18 │    0.92 │
-│ Medium JSON (~370KB, no GC) │    2.72 │    0.28 │    0.41 │    2.41 │
-│ Medium logs (~168KB, no GC) │    1.26 │    0.13 │    0.18 │    0.96 │
-│ Large JSON (~1.6MB, GC)     │   12.44 │    1.38 │    1.96 │   12.44 │
-│ Large logs (~600KB, GC)     │    4.29 │    0.46 │    0.49 │    2.85 │
-│ Large JSON (~1.6MB, no GC)  │   12.22 │    1.28 │    1.86 │   11.83 │
-│ Large logs (~600KB, no GC)  │    4.39 │    0.42 │    0.44 │    2.86 │
-└─────────────────────────────┴─────────┴─────────┴─────────┴─────────┘
-```
-
-### 📊 SPEEDUP vs ZLIB (higher is better)
-```
-┌─────────────────────────────┬─────────┬─────────┬─────────┬─────────┐
-│ Configuration               │  zlib   │   lz4   │  zstd   │ brotli  │
-├─────────────────────────────┼─────────┼─────────┼─────────┼─────────┤
-│ Small JSON (~10KB, GC)      │   1.00x │   5.00x │   2.50x │   0.36x │
-│ Small text (~10KB, GC)      │   1.00x │     N/A │   4.00x │   0.36x │
-│ Small JSON (~10KB, no GC)   │   1.00x │   6.00x │   3.00x │   0.46x │
-│ Small text (~10KB, no GC)   │   1.00x │     N/A │   4.00x │   0.36x │
-│ Medium JSON (~370KB, GC)    │   1.00x │   9.41x │   6.50x │   1.16x │
-│ Medium logs (~168KB, GC)    │   1.00x │   8.79x │   6.83x │   1.34x │
-│ Medium JSON (~370KB, no GC) │   1.00x │   9.71x │   6.63x │   1.13x │
-│ Medium logs (~168KB, no GC) │   1.00x │   9.69x │   7.00x │   1.31x │
-│ Large JSON (~1.6MB, GC)     │   1.00x │   9.01x │   6.35x │   1.00x │
-│ Large logs (~600KB, GC)     │   1.00x │   9.33x │   8.76x │   1.51x │
-│ Large JSON (~1.6MB, no GC)  │   1.00x │   9.55x │   6.57x │   1.03x │
-│ Large logs (~600KB, no GC)  │   1.00x │  10.45x │   9.98x │   1.53x │
-└─────────────────────────────┴─────────┴─────────┴─────────┴─────────┘
-```
-
+> Benchmark numbers are environment-dependent. The tables below were generated with **MultiCompress 0.3.2** using vendored **zstd 1.5.7**, **lz4 1.10.0**, and **brotli 1.2.0** on **Ruby 3.1.7 / arm64-darwin24**, 200 iterations per test.
+>
+> Use these numbers as relative guidance, not absolute guarantees. In this run, **lz4** is consistently the fastest, **zstd** gives the best speed/ratio balance, and **brotli** gives the smallest output size but is not optimized for speed-first paths.
+
+Performance comparison against Ruby's built-in zlib compression:
+
+### 🗜️ Compression ratio (%, lower is better)
+
+| Configuration | zlib | lz4 | zstd | brotli |
+|---|---:|---:|---:|---:|
+| Small JSON (~10KB, GC) | 9.4% | 16.1% | 6.9% | 6.1% |
+| Small text (~10KB, GC) | 3.1% | 4.6% | 3.2% | 2.6% |
+| Small JSON (~10KB, no GC) | 9.4% | 16.1% | 6.9% | 6.1% |
+| Small text (~10KB, no GC) | 3.1% | 4.6% | 3.2% | 2.6% |
+| Medium JSON (~370KB, GC) | 8.5% | 15.7% | 6.6% | 5.4% |
+| Medium logs (~168KB, GC) | 8.6% | 17.2% | 5.3% | 3.3% |
+| Medium JSON (~370KB, no GC) | 8.5% | 15.7% | 6.6% | 5.4% |
+| Medium logs (~168KB, no GC) | 8.6% | 17.2% | 5.3% | 3.3% |
+| Large JSON (~1.6MB, GC) | 8.1% | 15.1% | 6.1% | 5.6% |
+| Large logs (~600KB, GC) | 7.6% | 16.0% | 2.9% | 2.0% |
+| Large JSON (~1.6MB, no GC) | 8.1% | 15.1% | 6.1% | 5.6% |
+| Large logs (~600KB, no GC) | 7.6% | 16.0% | 2.9% | 2.0% |
+
+### ⚡ Total time (compress + decompress, ms — lower is faster)
+
+| Configuration | zlib | lz4 | zstd | brotli |
+|---|---:|---:|---:|---:|
+| Small JSON (~10KB, GC) | 0.05 | 0.01 | 0.02 | 0.14 |
+| Small text (~10KB, GC) | 0.04 | 0.00 | 0.01 | 0.12 |
+| Small JSON (~10KB, no GC) | 0.05 | 0.01 | 0.02 | 0.14 |
+| Small text (~10KB, no GC) | 0.04 | 0.00 | 0.01 | 0.11 |
+| Medium JSON (~370KB, GC) | 2.60 | 0.29 | 0.41 | 2.45 |
+| Medium logs (~168KB, GC) | 1.28 | 0.13 | 0.17 | 0.96 |
+| Medium JSON (~370KB, no GC) | 2.62 | 0.27 | 0.39 | 2.41 |
+| Medium logs (~168KB, no GC) | 1.19 | 0.13 | 0.17 | 1.08 |
+| Large JSON (~1.6MB, GC) | 11.60 | 1.30 | 1.81 | 11.12 |
+| Large logs (~600KB, GC) | 4.11 | 0.41 | 0.46 | 2.99 |
+| Large JSON (~1.6MB, no GC) | 11.26 | 1.24 | 1.77 | 10.77 |
+| Large logs (~600KB, no GC) | 4.01 | 0.42 | 0.45 | 2.90 |
+
+### 📊 Speedup vs zlib by total time (higher is better)
+
+| Configuration | zlib | lz4 | zstd | brotli |
+|---|---:|---:|---:|---:|
+| Small JSON (~10KB, GC) | 1.00x | 5.00x | 2.50x | 0.36x |
+| Small text (~10KB, GC) | 1.00x | N/A | 4.00x | 0.33x |
+| Small JSON (~10KB, no GC) | 1.00x | 5.00x | 2.50x | 0.36x |
+| Small text (~10KB, no GC) | 1.00x | N/A | 4.00x | 0.36x |
+| Medium JSON (~370KB, GC) | 1.00x | 8.97x | 6.34x | 1.06x |
+| Medium logs (~168KB, GC) | 1.00x | 9.85x | 7.53x | 1.33x |
+| Medium JSON (~370KB, no GC) | 1.00x | 9.70x | 6.72x | 1.09x |
+| Medium logs (~168KB, no GC) | 1.00x | 9.15x | 7.00x | 1.10x |
+| Large JSON (~1.6MB, GC) | 1.00x | 8.92x | 6.41x | 1.04x |
+| Large logs (~600KB, GC) | 1.00x | 10.02x | 8.93x | 1.37x |
+| Large JSON (~1.6MB, no GC) | 1.00x | 9.08x | 6.36x | 1.05x |
+| Large logs (~600KB, no GC) | 1.00x | 9.55x | 8.91x | 1.38x |
+
+### 📏 Compressed size (bytes, lower is better)
+
+| Configuration | zlib | lz4 | zstd | brotli |
+|---|---:|---:|---:|---:|
+| Small JSON (~10KB, GC) | 900 | 1544 | 665 | 583 |
+| Small text (~10KB, GC) | 310 | 461 | 322 | 256 |
+| Small JSON (~10KB, no GC) | 900 | 1544 | 665 | 583 |
+| Small text (~10KB, no GC) | 310 | 461 | 322 | 256 |
+| Medium JSON (~370KB, GC) | 31524 | 57986 | 24557 | 20122 |
+| Medium logs (~168KB, GC) | 14488 | 28950 | 8985 | 5549 |
+| Medium JSON (~370KB, no GC) | 31524 | 57986 | 24557 | 20122 |
+| Medium logs (~168KB, no GC) | 14488 | 28950 | 8985 | 5549 |
+| Large JSON (~1.6MB, GC) | 133275 | 250026 | 100965 | 92591 |
+| Large logs (~600KB, GC) | 45432 | 96130 | 17385 | 12250 |
+| Large JSON (~1.6MB, no GC) | 133275 | 250026 | 100965 | 92591 |
+| Large logs (~600KB, no GC) | 45432 | 96130 | 17385 | 12250 |
 
 **Dependencies for benchmarking:**
 - `memory_profiler` — Memory usage analysis

data/THIRD_PARTY_NOTICES.md

@@ -0,0 +1,24 @@
+# Third-Party Notices
+
+MultiCompress vendors C sources from upstream compression libraries so the gem can build without system-level compression packages. MultiCompress's own Ruby code and extension glue are MIT-licensed, but vendored third-party sources keep their upstream licenses.
+
+## Vendored native libraries
+
+| Library | Vendored version | Vendored source scope | Upstream license used by MultiCompress |
+|---|---:|---|---|
+| Zstandard (`zstd`) | 1.5.7 | `lib/` | BSD side of upstream BSD OR GPLv2 licensing |
+| LZ4 | 1.10.0 | `lib/` | BSD 2-Clause |
+| Brotli | 1.2.0 | `c/common`, `c/enc`, `c/dec`, `c/include` | MIT |
+
+## Notes
+
+- Zstandard changed copyright wording from older Facebook/Yann Collet headers to `Meta Platforms, Inc. and affiliates` in newer releases. This is a copyright-owner notice change, not a prohibition on use.
+- Do not imply that Facebook, Meta, Google, Yann Collet, or upstream contributors endorse MultiCompress.
+- When distributing source or binary builds, keep the vendored source notices and this file in the package.
+- LZ4's repository contains GPL-licensed areas outside `lib/`; MultiCompress vendors only `lib/`, which upstream documents as the BSD 2-Clause integration area.
+
+## Upstream projects
+
+- Zstandard: https://github.com/facebook/zstd
+- LZ4: https://github.com/lz4/lz4
+- Brotli: https://github.com/google/brotli

data/ext/multi_compress/brotli_dec_static_init.c

@@ -0,0 +1,3 @@
+// Wrapper for Brotli 1.2.0 vendored builds.
+
+#include "vendor/brotli/c/dec/static_init.c"

data/ext/multi_compress/brotli_enc_static_init.c

@@ -0,0 +1,3 @@
+// Wrapper for Brotli 1.2.0 vendored builds.
+
+#include "vendor/brotli/c/enc/static_init.c"

data/ext/multi_compress/extconf.rb

@@ -10,6 +10,14 @@ FORCE_VENDORED = arg_config("--force-vendored") ||
 ZSTD_SUBDIRS   = %w[lib/common lib/compress lib/decompress lib/dictBuilder].freeze
 BROTLI_SUBDIRS = %w[c/common c/enc c/dec].freeze
 LZ4_SOURCES    = %w[lz4.c lz4hc.c lz4frame.c].freeze
+BROTLI_STATIC_INIT_SOURCES = %w[
+  c/enc/static_init.c
+  c/dec/static_init.c
+].freeze
+BROTLI_STATIC_INIT_WRAPPERS = %w[
+  brotli_enc_static_init.c
+  brotli_dec_static_init.c
+].freeze
 
 def find_vendor_dir
   candidates = [
@@ -116,7 +124,20 @@ def collect_vendor_sources(zstd_dir, lz4_dir, brotli_dir)
 
   brotli_srcs = BROTLI_SUBDIRS.flat_map { |d| Dir[File.join(brotli_dir, d, "*.c")] }
 
-  zstd_srcs + lz4_srcs + brotli_srcs
+  # Brotli 1.2.0 has both c/enc/static_init.c and c/dec/static_init.c.
+  # mkmf builds object files by basename, so compiling both as static_init.c
+  # makes one object shadow the other. Use unique wrapper translation units
+  # instead; otherwise the decoder static init can be skipped and crash at runtime.
+  brotli_static_init_paths = BROTLI_STATIC_INIT_SOURCES.map do |relative_path|
+    File.join(brotli_dir, relative_path)
+  end
+  brotli_srcs -= brotli_static_init_paths
+
+  brotli_static_init_wrappers = BROTLI_STATIC_INIT_WRAPPERS.map do |file|
+    File.join(__dir__, file)
+  end
+
+  zstd_srcs + lz4_srcs + brotli_srcs + brotli_static_init_wrappers
 end
 
 def add_include_dirs(zstd_dir, lz4_dir, brotli_dir)