multi_compress 0.2.4

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 9faa0c73c6b3926604e7d3477c5a519a8898fd8a08e0f9d71b84c6fce87a815d
+  data.tar.gz: '0818b9fa5ea7cf2958910150185571a9855a1728f44c6f425ac590a2256298a0'
+SHA512:
+  metadata.gz: b94255c0103f3f89ae4099627a61a972b6da00220ace21d9fbd6d0c3f1641cb46d2ae79126c75f738e21ebd5359c80f6bdee11d958200731e5d402e1058d9fad
+  data.tar.gz: c5ce6f3cb3e6c7d1b2fe08666dd916e66646f8077669f9908f61e2de64e311be1f7052345617faa045526c26136f38cc659bfabd92b596e826dd79224a1962fe

data/CHANGELOG.md

@@ -0,0 +1,152 @@
+# Changelog
+
+## [0.2.4]
+
+### Added
+ - Keyword validation
+
+### Fixed
+- Fixed gem packaging so vendored builds work correctly after installation.
+  Previously, installed gems could incorrectly fall back to system libraries because
+  `ext/multi_compress/vendor/.vendored` was not included in the packaged gem.
+- Fixed vendored zstd build path in `extconf.rb`.
+
+## [0.2.3]
+
+### Changed
+ - Constants
+
+## [0.2.2]
+
+### Added
+- New decompression safeguards:
+  - `max_output_size:` on `MultiCompress.decompress`, `MultiCompress::Inflater.new`, and `MultiCompress::Reader.open`
+  - `max_ratio:` with default `1000` and trusted-input opt-out via `nil`
+  - Cumulative streaming output budget enforcement
+  - Dictionary file size cap (`32MB`) for `MultiCompress::Dictionary.load`
+- New docs/code parity and limits tests
+- Linux-only ASAN/UBSAN CI job
+- Local `script/valgrind.sh` helper for developer verification (with Ruby VM suppressions)
+- Local `script/sanitize.sh` helper for reproducing the ASAN/UBSAN CI job locally on Linux
+- Optional `./build.sh --test`, `./build.sh --valgrind`, and `./build.sh --sanitize` modes
+
+### Changed
+- Vendored **zstd** is pinned to **1.5.2** in the current release line.
+  - This restores reliable `MultiCompress::Zstd.train_dictionary(...)` behavior on tested platforms, including arm64-darwin.
+  - Newer vendored zstd versions are temporarily avoided until the dictionary training regression is understood and fixed upstream.
+
+### Changed
+- Documentation now matches current runtime behavior:
+  - Ruby requirement is documented as `>= 3.1.0`
+  - Supported named levels are `:fastest`, `:default`, and `:best`
+  - Zstd `:best` is documented with the current runtime mapping (level 19)
+  - LZ4 is documented as using a custom internal format that is not compatible with the standard `lz4` CLI
+
+### Fixed
+- Corrected dictionary training docs:
+  - `MultiCompress::Zstd.train_dictionary` is documented as supported
+  - `MultiCompress::Brotli.train_dictionary` is documented as unsupported in the current implementation
+
+### Notes
+- The default `max_ratio: 1000` is a user-visible behavior change. Workloads that legitimately expand beyond `1000:1` must pass `max_ratio: nil`.
+
+## [0.2.1] — 2026-04-15
+- Change version zstd "1.5.6" -> "1.5.7".
+- Micro optimization.
+
+## [0.2.0] — 2026-04-15
+
+### Added
+- **Fiber-friendly execution** 🎉: compression and decompression now cooperate with Ruby's `Fiber::Scheduler`, making MultiCompress safe to use under [async](https://github.com/socketry/async), [falcon](https://github.com/socketry/falcon), and any other scheduler-based runtime.
+  - When a scheduler is active, CPU-heavy work is offloaded to a dedicated worker thread via `rb_thread_create`, while the calling fiber is parked with `rb_fiber_scheduler_block`. The scheduler is free to run other fibers (IO, timers, parallel compression) until the worker finishes and calls `rb_fiber_scheduler_unblock`.
+  - Covers **all three algorithms** (`zstd`, `lz4`, `brotli`) and **both API shapes**:
+    - One-shot `MultiCompress.compress` / `MultiCompress.decompress`
+    - Streaming `MultiCompress::Deflater#write` / `MultiCompress::Inflater#write`
+  - No API changes — the fiber-friendly path is selected automatically when a scheduler is detected.
+  - Chunks smaller than 16 KB (`FIBER_STREAM_THRESHOLD`) stay inline to avoid pthread-create overhead for micro-workloads.
+  - Outside of a scheduler, the previous `rb_thread_call_without_gvl` path is used unchanged.
+- Generic `run_via_fiber_worker(scheduler, func, arg)` internal helper — encapsulates the pthread-worker + scheduler-block pattern and is reused across every fiber-friendly code path.
+- New test suite `test/test_fiber_scheduler.rb` verifying scheduler progress during compress/decompress for every algorithm, in both one-shot and streaming modes.
+
+### Changed
+- `fiber_maybe_yield` now actually yields: previously it was a no-op counter; it now calls `Fiber.scheduler.yield` via `rb_funcall` when the byte threshold is crossed. Used for `lz4` streaming paths where pthread-per-block would be overkill.
+- Streaming `Deflater#write` for zstd/brotli uses the fiber-worker path for chunks ≥ 16 KB; the old inline path is kept as a fallback for small chunks and when no scheduler is present.
+- Streaming `Inflater#write` for zstd/brotli similarly routes chunks through the fiber-worker path when possible.
+- LZ4 one-shot decompress loop extracted into `lz4_decompress_all_nogvl` so it can run on the worker thread with the GVL released.
+
+### Performance
+- No regressions for the non-fiber case: all existing GVL-unlocking fast paths are preserved untouched.
+- Under a fiber scheduler: compression no longer starves concurrent IO fibers. Previously, compressing 50 MB with `zstd` level 3 would block every other fiber in the reactor for the entire duration; now the reactor keeps servicing IO, timers, and other compute fibers throughout.
+- Streaming threshold of 16 KB chosen empirically: below that, pthread-create overhead (~20-50μs) eats the gains; at 16 KB+ the fiber-friendly path is a near-free win.
+
+### Upgrading
+No code changes required. If you run under `Async`/`Falcon`/`Fiber::Scheduler`, you'll immediately get non-blocking compression. If you don't, behavior is identical to 0.1.2.
+
+## [0.1.2] — 2026-04-14
+
+### Fixed
+- **LZ4 Streaming Correctness**: Fixed critical bug in LZ4 Inflater that caused incorrect decompression on large data (>1MB)
+  - Removed complex streaming API (`LZ4_decompress_safe_continue`) that had implementation errors
+  - Restored simple, reliable direct decompression (`LZ4_decompress_safe`)
+  - All roundtrip tests now pass correctly
+- **LZ4 Deflater Ring Buffer**: Fixed ring buffer management to prevent overflow
+  - Added proper offset reset when approaching buffer limit
+  - Improved memory safety
+
+### Added
+- **ZSTD Dictionary Training**: Added `MultiCompress::Zstd.train_dictionary(samples, size:)` method
+  - Uses ZDICT API for optimal dictionary generation
+  - Allows creating custom dictionaries for better compression on similar data
+- **Memory Optimization Helpers**:
+  - Added `rb_binary_str_buf_reserve()` for efficient pre-allocation
+  - Added `grow_binary_str()` for safe capacity management
+
+### Changed
+- **Memory Efficiency**: Eliminated double-buffering in compress/decompress operations
+  - ZSTD, LZ4, Brotli now write directly to Ruby strings when unlocking GVL
+  - Reduced memory allocations and improved cache locality
+- **Dictionary Support in Streaming**: Fixed dictionary reloading in `reset()` methods
+  - Dictionaries are now properly saved as instance variables
+  - `deflater.reset` and `inflater.reset` correctly restore dictionary state
+- **Brotli Error Handling**: Added comprehensive error checking for dictionary operations
+  - All Brotli API calls now verify success/failure
+  - More informative error messages
+
+### Performance Notes
+- **LZ4 Streaming**: Slight performance regression on large chunks (128KB: ~40-50% slower decompression)
+  - Trade-off for correctness: reliable decompression is prioritized over peak performance
+  - Small chunks (4-32KB) maintain original performance
+  - Overall impact minimal as streaming typically uses smaller chunks
+- **Memory Usage**: Reduced due to elimination of intermediate buffers
+- **Dictionary Operations**: Faster on repeated use due to improved caching
+
+### Security
+- Improved LZ4 format validation in `detect_algo()`
+  - Now checks for proper tail marker (4 zero bytes)
+  - Better protection against malformed data
+
+## [0.1.1] — Unreleased
+
+### Breaking Changes
+- **REMOVED**: `MultiCompress::Dictionary.train` method - general dictionary training interface removed
+- Dictionary training is now algorithm-specific to clarify capabilities
+
+### Added
+- `MultiCompress::Brotli.train_dictionary` - Brotli-specific dictionary training method
+- Clear indication that only Brotli supports dictionary training in this implementation
+
+### Changed
+- Dictionary training API is now more explicit about which algorithms support it
+- Improved API clarity by making training method algorithm-specific
+
+## [0.1.0] — Unreleased
+
+### Added
+- One-shot compress/decompress for zstd, lz4, brotli
+- Streaming API: Deflater, Inflater
+- IO wrappers: Writer, Reader with block-style open
+- Dictionary API skeleton (zstd, brotli)
+- Auto-detect algorithm by magic bytes and file extension
+- Named compression levels: :fastest, :default, :best
+- CRC32 / Adler32 utility stubs
+- Algorithm info: `.algorithms`, `.available?`, `.version`

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2025 Roman Haydarov
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,177 @@
+# MultiCompress 🗜️
+
+[![Gem Version](https://badge.fury.io/rb/multi_compress.svg)](https://badge.fury.io/rb/multi_compress)
+
+> **Status:** functional, well-tested, and actively evolving. The current release is suitable for real workloads, but the API and implementation details are still being refined in upcoming releases.
+
+Modern compression technology: **zstd**, **lz4**, **brotli** — unified compression platform with native C performance, **fiber-friendly** for modern async Ruby stacks.
+
+Bundled library versions in the current release:
+- **zstd 1.5.2**
+- **lz4 1.10.0**
+- **brotli 1.1.0**
+
+📖 **[Get Started →](GET_STARTED.md)** — Complete technology overview, algorithms, and implementation details
+
+## Technology Overview
+
+**MultiCompress** is a comprehensive compression system that unites three cutting-edge algorithms in a single platform. Modern algorithms are 3–10x faster than traditional zlib while providing superior compression ratios.
+
+| Algorithm | Strength | Best for |
+|-----------|----------|----------|
+| **zstd** | Best speed/ratio balance | Cache, logs, backups |
+| **lz4** | Fastest compress/decompress | IPC, hot cache, real-time |
+| **brotli** | Best ratio for HTTP | Web assets, API responses |
+
+## How It Works
+
+**MultiCompress** packages modern compression algorithms (zstd, lz4, brotli) with their native C libraries, providing a unified interface. The system includes vendored sources of all compression libraries, eliminating external dependencies.
+
+### Key Design Principles
+
+- **Dictionary support**: Runtime dictionary use is supported for zstd and brotli; Zstd dictionary training is available in the current release line
+- **Zero external dependencies**: All C libraries are vendored and compiled
+- **Unified API**: Same interface for all algorithms — just change the `algo:` parameter
+- **Performance first**: Direct bindings to C libraries, minimal overhead
+- **Fiber-friendly**: Compression and decompression cooperate with Ruby's fiber scheduler — safe to use under `async`, `falcon`, or any `Fiber::Scheduler`-based runtime without blocking the event loop. See [GET_STARTED.md](GET_STARTED.md) for details and examples.
+- **Memory efficient**: Streaming support for large datasets, proper resource cleanup
+- **Operationally focused**: Clear errors, comprehensive tests, and streaming support for practical workloads
+
+### Algorithm Auto-Detection
+
+The system can automatically detect compression algorithms when decompressing data:
+
+- **zstd**: Detected by magic bytes `28 B5 2F FD` (little-endian)
+- **lz4**: Detected by internal format header validation (custom internal format, NOT compatible with the standard `lz4` CLI; optional standard frame support may be added in a future release)
+- **brotli**: Requires explicit `algo: :brotli` parameter - no auto-detection
+
+**Important**: Auto-detection only works for ZSTD and LZ4. Brotli data must be decompressed with explicit algorithm specification.
+
+**Security**: Decompression now enforces a default 256MB output cap, cumulative streaming limits, a default ratio guard of 1000:1, and a 32MB dictionary file size cap.
+
+
+## Security limits
+
+Decompression-facing APIs support conservative defaults intended to protect against decompression bombs and accidental resource spikes:
+
+- **Default output cap:** `256MB`
+- **Streaming cumulative cap:** enforced across the lifetime of an `Inflater`/`Reader`
+- **Default ratio guard:** `1000:1`
+- **Trusted-input opt-out:** pass `max_ratio: nil`
+- **Dictionary file size cap:** `32MB` for `MultiCompress::Dictionary.load`
+
+Examples:
+
+```ruby
+MultiCompress.decompress(blob, algo: :zstd, max_output_size: 64 * 1024 * 1024)
+MultiCompress.decompress(blob, algo: :brotli, max_ratio: nil)
+
+MultiCompress::Reader.open("archive.zst", max_output_size: 128 * 1024 * 1024, max_ratio: 500) do |reader|
+  puts reader.read
+end
+```
+
+`max_output_size: nil` keeps the native default cap of `256MB`. `max_ratio: nil` disables the ratio guard for trusted input.
+
+## Algorithm Comparison
+
+| Algorithm | Speed | Ratio | Best Use Case |
+|-----------|-------|--------|---------------|
+| **lz4** | Fastest | Good | Real-time processing, IPC, hot cache paths |
+| **zstd** | Fast | Excellent | General purpose, logs, backups, web APIs |
+| **brotli** | Slower | Best | Static assets, CDN, long-term storage |
+
+
+## Benchmark Results
+
+> **📝 Note on v0.2.0**: Performance numbers below are from the v0.2.0 build with fiber-friendly paths enabled. There is no throughput regression compared to v0.1.2 — the fiber-friendly path is only taken when a `Fiber::Scheduler` is active, and even then the worker-thread overhead is negligible for payloads large enough to benefit.
+
+Performance comparison against Ruby's built-in zlib compression (200 iterations per test):
+
+### 🗜️  COMPRESSION RATIO (%, lower is better)
+```
+┌─────────────────────────────┬─────────┬─────────┬─────────┬─────────┐
+│ Configuration               │  zlib   │   lz4   │  zstd   │ brotli  │
+├─────────────────────────────┼─────────┼─────────┼─────────┼─────────┤
+│ Small JSON (~10KB, GC)      │    9.4% │   16.1% │    6.9% │    6.1% │
+│ Small text (~10KB, GC)      │    3.1% │    4.6% │    3.2% │    2.6% │
+│ Small JSON (~10KB, no GC)   │    9.4% │   16.1% │    6.9% │    6.1% │
+│ Small text (~10KB, no GC)   │    3.1% │    4.6% │    3.2% │    2.6% │
+│ Medium JSON (~370KB, GC)    │    8.5% │   15.7% │    6.7% │    5.5% │
+│ Medium logs (~168KB, GC)    │    8.6% │   17.2% │    5.4% │    3.2% │
+│ Medium JSON (~370KB, no GC) │    8.5% │   15.7% │    6.7% │    5.5% │
+│ Medium logs (~168KB, no GC) │    8.6% │   17.2% │    5.4% │    3.2% │
+│ Large JSON (~1.6MB, GC)     │    8.1% │   15.1% │    6.1% │    5.6% │
+│ Large logs (~600KB, GC)     │    7.6% │   16.0% │    2.8% │    2.1% │
+│ Large JSON (~1.6MB, no GC)  │    8.1% │   15.1% │    6.1% │    5.6% │
+│ Large logs (~600KB, no GC)  │    7.6% │   16.0% │    2.8% │    2.1% │
+└─────────────────────────────┴─────────┴─────────┴─────────┴─────────┘
+```
+
+### ⚡ TOTAL TIME (compress + decompress, ms — lower is faster)
+```
+┌─────────────────────────────┬─────────┬─────────┬─────────┬─────────┐
+│ Configuration               │  zlib   │   lz4   │  zstd   │ brotli  │
+├─────────────────────────────┼─────────┼─────────┼─────────┼─────────┤
+│ Small JSON (~10KB, GC)      │    0.05 │    0.01 │    0.02 │    0.14 │
+│ Small text (~10KB, GC)      │    0.04 │    0.00 │    0.01 │    0.11 │
+│ Small JSON (~10KB, no GC)   │    0.06 │    0.01 │    0.02 │    0.13 │
+│ Small text (~10KB, no GC)   │    0.04 │    0.00 │    0.01 │    0.11 │
+│ Medium JSON (~370KB, GC)    │    2.73 │    0.29 │    0.42 │    2.36 │
+│ Medium logs (~168KB, GC)    │    1.23 │    0.14 │    0.18 │    0.92 │
+│ Medium JSON (~370KB, no GC) │    2.72 │    0.28 │    0.41 │    2.41 │
+│ Medium logs (~168KB, no GC) │    1.26 │    0.13 │    0.18 │    0.96 │
+│ Large JSON (~1.6MB, GC)     │   12.44 │    1.38 │    1.96 │   12.44 │
+│ Large logs (~600KB, GC)     │    4.29 │    0.46 │    0.49 │    2.85 │
+│ Large JSON (~1.6MB, no GC)  │   12.22 │    1.28 │    1.86 │   11.83 │
+│ Large logs (~600KB, no GC)  │    4.39 │    0.42 │    0.44 │    2.86 │
+└─────────────────────────────┴─────────┴─────────┴─────────┴─────────┘
+```
+
+### 📊 SPEEDUP vs ZLIB (higher is better)
+```
+┌─────────────────────────────┬─────────┬─────────┬─────────┬─────────┐
+│ Configuration               │  zlib   │   lz4   │  zstd   │ brotli  │
+├─────────────────────────────┼─────────┼─────────┼─────────┼─────────┤
+│ Small JSON (~10KB, GC)      │   1.00x │   5.00x │   2.50x │   0.36x │
+│ Small text (~10KB, GC)      │   1.00x │     N/A │   4.00x │   0.36x │
+│ Small JSON (~10KB, no GC)   │   1.00x │   6.00x │   3.00x │   0.46x │
+│ Small text (~10KB, no GC)   │   1.00x │     N/A │   4.00x │   0.36x │
+│ Medium JSON (~370KB, GC)    │   1.00x │   9.41x │   6.50x │   1.16x │
+│ Medium logs (~168KB, GC)    │   1.00x │   8.79x │   6.83x │   1.34x │
+│ Medium JSON (~370KB, no GC) │   1.00x │   9.71x │   6.63x │   1.13x │
+│ Medium logs (~168KB, no GC) │   1.00x │   9.69x │   7.00x │   1.31x │
+│ Large JSON (~1.6MB, GC)     │   1.00x │   9.01x │   6.35x │   1.00x │
+│ Large logs (~600KB, GC)     │   1.00x │   9.33x │   8.76x │   1.51x │
+│ Large JSON (~1.6MB, no GC)  │   1.00x │   9.55x │   6.57x │   1.03x │
+│ Large logs (~600KB, no GC)  │   1.00x │  10.45x │   9.98x │   1.53x │
+└─────────────────────────────┴─────────┴─────────┴─────────┴─────────┘
+```
+
+
+**Dependencies for benchmarking:**
+- `memory_profiler` — Memory usage analysis
+- `benchmark-ips` — Iterations per second benchmarking
+
+Or use the build script:
+
+```bash
+./build.sh
+```
+
+## Requirements
+
+- Ruby >= 3.1.0
+- C compiler (gcc, clang)
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b feature/my-feature`)
+3. Commit your changes (`git commit -am 'Add feature'`)
+4. Push to the branch (`git push origin feature/my-feature`)
+5. Create a Pull Request
+
+## License
+
+MIT — see [LICENSE.txt](LICENSE.txt).

data/ext/multi_compress/extconf.rb

@@ -0,0 +1,190 @@
+# frozen_string_literal: true
+
+require "mkmf"
+
+USE_SYSTEM = arg_config("--use-system-libraries") ||
+             ENV["COMPRESS_USE_SYSTEM_LIBRARIES"]
+FORCE_VENDORED = arg_config("--force-vendored") ||
+                 ENV["COMPRESS_FORCE_VENDORED"]
+
+ZSTD_SUBDIRS   = %w[lib/common lib/compress lib/decompress lib/dictBuilder].freeze
+BROTLI_SUBDIRS = %w[c/common c/enc c/dec].freeze
+LZ4_SOURCES    = %w[lz4.c lz4hc.c lz4frame.c].freeze
+
+def find_vendor_dir
+  candidates = [
+    File.join(__dir__, "vendor"),
+    File.join(__dir__, "..", "..", "..", "..", "ext", "multi_compress", "vendor"),
+    File.expand_path("../../ext/multi_compress/vendor", __dir__),
+    File.join(Dir.pwd, "ext", "multi_compress", "vendor"),
+  ]
+
+  dir = __dir__
+  6.times do
+    candidates << File.join(dir, "ext", "multi_compress", "vendor")
+    dir = File.dirname(dir)
+  end
+
+  candidates.find { |path| File.exist?(File.join(path, ".vendored")) }
+            &.then { |path| File.expand_path(path) }
+end
+
+def find_compress_c_dir
+  candidates = [
+    __dir__,
+    File.join(__dir__, "..", "..", "..", "..", "ext", "multi_compress"),
+  ]
+
+  dir = __dir__
+  6.times do
+    candidates << File.join(dir, "ext", "multi_compress")
+    dir = File.dirname(dir)
+  end
+
+  candidates.find { |path| File.exist?(File.join(path, "multi_compress.c")) }
+            &.then { |path| File.expand_path(path) } || __dir__
+end
+
+def configure_system_libraries
+  puts "Building with SYSTEM libraries"
+
+  configure_homebrew if RUBY_PLATFORM.include?("darwin")
+
+  require_system_lib("zstd",   header: "zstd.h",           func: "ZSTD_compress")
+  require_system_lib("lz4",    header: "lz4.h",            func: "LZ4_compress_default")
+  require_system_lib("brotli", header: "brotli/encode.h",  func: "BrotliEncoderCreateInstance", lib: "brotlienc")
+
+  have_header("zdict.h")
+  have_header("lz4hc.h")
+  have_library("lz4", "LZ4_compress_HC")
+  have_header("brotli/decode.h")
+  have_library("brotlidec", "BrotliDecoderCreateInstance")
+end
+
+def configure_homebrew
+  dir_config("homebrew", "/opt/homebrew")
+  $CPPFLAGS += " -I/opt/homebrew/include"
+  $LDFLAGS  += " -L/opt/homebrew/lib"
+end
+
+def require_system_lib(name, header:, func:, lib: name)
+  return if have_header(header) && have_library(lib, func)
+
+  abort "Missing #{name}. Install: apt install lib#{name}-dev / brew install #{name}"
+end
+
+def configure_vendored_libraries(vendor_dir)
+  versions = File.read(File.join(vendor_dir, ".vendored"))
+  puts "Building with VENDORED libraries from #{vendor_dir}"
+  puts "  forced vendored mode enabled" if FORCE_VENDORED
+  puts "  #{versions.tr("\n", ", ")}"
+
+  zstd_dir   = File.join(vendor_dir, "zstd")
+  lz4_dir    = File.join(vendor_dir, "lz4")
+  brotli_dir = File.join(vendor_dir, "brotli")
+
+  all_vendor_srcs = collect_vendor_sources(zstd_dir, lz4_dir, brotli_dir)
+
+  puts "  #{all_vendor_srcs.length} vendored C files"
+
+  add_include_dirs(zstd_dir, lz4_dir, brotli_dir)
+  $CPPFLAGS += " -DZSTD_DISABLE_ASM"
+
+  vpath_dirs = build_vpath_dirs(zstd_dir, lz4_dir, brotli_dir)
+
+  deduplicate_sources!(all_vendor_srcs)
+
+  compress_c_dir = find_compress_c_dir
+
+  $srcs  = ["multi_compress.c"] + all_vendor_srcs.map { |s| File.basename(s) }
+  $VPATH = [compress_c_dir] + vpath_dirs
+
+  $warnflags = ""
+
+  vpath_dirs
+end
+
+def collect_vendor_sources(zstd_dir, lz4_dir, brotli_dir)
+  zstd_srcs = ZSTD_SUBDIRS.flat_map { |d| Dir[File.join(zstd_dir, d, "*.c")] }
+
+  lz4_srcs = LZ4_SOURCES.filter_map do |f|
+    path = File.join(lz4_dir, "lib", f)
+    path if File.exist?(path)
+  end
+
+  brotli_srcs = BROTLI_SUBDIRS.flat_map { |d| Dir[File.join(brotli_dir, d, "*.c")] }
+
+  zstd_srcs + lz4_srcs + brotli_srcs
+end
+
+def add_include_dirs(zstd_dir, lz4_dir, brotli_dir)
+  [
+    File.join(zstd_dir, "lib"),
+    File.join(zstd_dir, "lib", "common"),
+    File.join(lz4_dir, "lib"),
+    File.join(brotli_dir, "c", "include"),
+  ].each { |d| $CPPFLAGS += " -I#{d}" }
+end
+
+def build_vpath_dirs(zstd_dir, lz4_dir, brotli_dir)
+  ZSTD_SUBDIRS.map { |d| File.join(zstd_dir, d) } +
+    [File.join(lz4_dir, "lib")] +
+    BROTLI_SUBDIRS.map { |d| File.join(brotli_dir, d) }
+end
+
+def deduplicate_sources!(sources)
+  seen = {}
+
+  sources.reject! do |src|
+    basename = File.basename(src)
+    duplicate = seen.key?(basename)
+    puts "  SKIP duplicate: #{src}" if duplicate
+    seen[basename] = true
+    duplicate
+  end
+end
+
+def patch_makefile_vpath!(vpath_dirs)
+  makefile = File.read("Makefile")
+  return if makefile.include?("# vendored vpath")
+
+  vpath_lines = vpath_dirs.map { |d| "vpath %.c #{d}" }.join("\n")
+
+  makefile.sub!(/^(VPATH\s*=.*)$/m) { "#{Regexp.last_match(1)}\n# vendored vpath\n#{vpath_lines}" }
+  File.write("Makefile", makefile)
+  puts "  Patched Makefile with #{vpath_dirs.length} VPATH entries"
+end
+
+# --- Main ---
+
+VENDOR_DIR = find_vendor_dir
+VENDORED   = !VENDOR_DIR.nil?
+
+if FORCE_VENDORED && !VENDORED
+  abort "COMPRESS_FORCE_VENDORED is set, but ext/multi_compress/vendor/.vendored was not found"
+end
+
+if FORCE_VENDORED
+  vpath_dirs = configure_vendored_libraries(VENDOR_DIR)
+elsif USE_SYSTEM || !VENDORED
+  configure_system_libraries
+  vpath_dirs = nil
+else
+  vpath_dirs = configure_vendored_libraries(VENDOR_DIR)
+end
+
+$CFLAGS += " -O3"
+$CFLAGS += " -DXXH_NAMESPACE=MULTICOMPRESS_"
+
+case RUBY_PLATFORM
+when /x86_64|amd64|aarch64|arm64/
+  $CFLAGS += " -DBROTLI_BUILD_LITTLE_ENDIAN"
+end
+
+have_header("ruby/fiber/scheduler.h")
+
+have_library("pthread") unless RUBY_PLATFORM.include?("darwin")
+
+create_makefile("multi_compress/multi_compress")
+
+patch_makefile_vpath!(vpath_dirs) if VENDORED && !USE_SYSTEM && vpath_dirs