multi_compress 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: a1afb05d5b335b931383cb6d0cb8102c93609d4e24e8d4a13fb66ab4c8cbd1e9
+  data.tar.gz: 945e4407133c1208b86c896be35c5c664568a7f2f8a7b452c13b00caa09700b3
+SHA512:
+  metadata.gz: 3908089b848102a6cbc4b254fc08612fef8f448f663df8445e988c2ba10dbbb8db6b118c5c00cc9cac663b6e92771138f7f35ebd7c2078bffbf3d55ab4ad492c
+  data.tar.gz: 6f8df288d73f7ee0bcd4a6b31bc3ac46a7317b5d917fa0b999fd51baf9c06a8acea25f0f89d76b9e4cd8ae3a6e182fdfafc554585fd09c6dd360e9b76ccc977b

data/CHANGELOG.md

@@ -0,0 +1,13 @@
+# Changelog
+
+## [0.1.0] — Unreleased
+
+### Added
+- One-shot compress/decompress for zstd, lz4, brotli
+- Streaming API: Deflater, Inflater
+- IO wrappers: Writer, Reader with block-style open
+- Dictionary API skeleton (zstd, brotli)
+- Auto-detect algorithm by magic bytes and file extension
+- Named compression levels: :fastest, :default, :best
+- CRC32 / Adler32 utility stubs
+- Algorithm info: `.algorithms`, `.available?`, `.version`

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2025 Roman Haydarov
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,116 @@
+# MultiCompress 🗜️
+
+[![Gem Version](https://badge.fury.io/rb/multi_compress.svg)](https://badge.fury.io/rb/multi_compress)
+
+> **⚠️ PROOF OF CONCEPT** — This is a proof-of-concept implementation. While functional and well-tested, additional features and optimizations are planned before the production-ready release.
+
+Modern compression technology: **zstd**, **lz4**, **brotli** — unified compression platform with native C performance.
+
+📖 **[Get Started →](GET_STARTED.md)** — Complete technology overview, algorithms, and implementation details
+
+## Technology Overview
+
+**MultiCompress** is a comprehensive compression system that unites three cutting-edge algorithms in a single platform. Modern algorithms are 3–10x faster than traditional zlib while providing superior compression ratios.
+
+| Algorithm | Strength | Best for |
+|-----------|----------|----------|
+| **zstd** | Best speed/ratio balance | Cache, logs, backups |
+| **lz4** | Fastest compress/decompress | IPC, hot cache, real-time |
+| **brotli** | Best ratio for HTTP | Web assets, API responses |
+
+## How It Works
+
+**MultiCompress** packages modern compression algorithms (zstd, lz4, brotli) with their native C libraries, providing a unified interface. The system includes vendored sources of all compression libraries, eliminating external dependencies.
+
+### Key Design Principles
+
+- **Zero external dependencies**: All C libraries are vendored and compiled
+- **Unified API**: Same interface for all algorithms — just change the `algo:` parameter  
+- **Performance first**: Direct bindings to C libraries, minimal overhead
+- **Memory efficient**: Streaming support for large datasets, proper resource cleanup
+- **Production ready**: Battle-tested error handling, comprehensive test coverage
+
+### Algorithm Auto-Detection
+
+The system can automatically detect compression algorithms when decompressing data:
+
+- **zstd**: Detected by magic bytes `28 B5 2F FD` (little-endian)
+- **lz4**: Detected by internal format header validation (custom format, NOT compatible with lz4 CLI)
+- **brotli**: Requires explicit `algo: :brotli` parameter - no auto-detection
+
+**Important**: Auto-detection only works for ZSTD and LZ4. Brotli data must be decompressed with explicit algorithm specification.
+
+**Security**: All decompression operations have a built-in 256MB size limit to prevent decompression bomb attacks.
+
+## Algorithm Comparison
+
+| Algorithm | Speed | Ratio | Best Use Case |
+|-----------|-------|--------|---------------|
+| **lz4** | Fastest | Good | Real-time processing, IPC, hot cache paths |
+| **zstd** | Fast | Excellent | General purpose, logs, backups, web APIs |
+| **brotli** | Slower | Best | Static assets, CDN, long-term storage |
+
+
+## Benchmark Results
+
+Performance comparison against Ruby's built-in zlib compression:
+
+### 🗜️  COMPRESSION RATIO (%, lower is better)
+```
+┌─────────────────────────────┬─────────┬─────────┬─────────┬─────────┐
+│ Configuration               │  zlib   │   lz4   │  zstd   │ brotli  │
+├─────────────────────────────┼─────────┼─────────┼─────────┼─────────┤
+│ Small JSON (small, GC)      │    8.8% │   16.2% │    8.2% │    6.5% │
+│ Small text (small, GC)      │   11.6% │   20.0% │   13.1% │   11.5% │
+│ Small JSON (small, no GC)   │    8.8% │   16.2% │    8.2% │    6.5% │
+│ Small text (small, no GC)   │   11.6% │   20.0% │   13.1% │   11.5% │
+│ Medium JSON (medium, GC)    │    8.3% │   15.5% │    8.9% │    7.1% │
+│ Medium logs (medium, GC)    │   15.9% │   26.7% │   17.9% │   15.5% │
+│ Medium JSON (medium, no GC) │    8.3% │   15.5% │    8.9% │    7.1% │
+│ Medium logs (medium, no GC) │   15.9% │   26.7% │   17.9% │   15.5% │
+└─────────────────────────────┴─────────┴─────────┴─────────┴─────────┘
+```
+
+### ⚡ PROCESSING SPEED (average time in milliseconds, lower is faster)
+```
+┌─────────────────────────────┬─────────┬─────────┬─────────┬─────────┐
+│ Configuration               │  zlib   │   lz4   │  zstd   │ brotli  │
+├─────────────────────────────┼─────────┼─────────┼─────────┼─────────┤
+│ Small JSON (small, GC)      │    0.68 │    0.20 │    0.30 │    1.64 │
+│ Small text (small, GC)      │    2.35 │    0.68 │    1.19 │    6.57 │
+│ Small JSON (small, no GC)   │    0.69 │    0.19 │    0.30 │    1.72 │
+│ Small text (small, no GC)   │    2.32 │    0.67 │    1.13 │    6.45 │
+│ Medium JSON (medium, GC)    │   21.89 │    6.24 │    8.58 │   49.56 │
+│ Medium logs (medium, GC)    │   45.91 │   11.55 │   22.47 │  131.84 │
+│ Medium JSON (medium, no GC) │   22.90 │    6.07 │    8.67 │   52.25 │
+│ Medium logs (medium, no GC) │   44.00 │   10.82 │   21.56 │  125.21 │
+└─────────────────────────────┴─────────┴─────────┴─────────┴─────────┘
+```
+
+
+**Dependencies for benchmarking:**
+- `memory_profiler` — Memory usage analysis
+- `benchmark-ips` — Iterations per second benchmarking
+
+Or use the build script:
+
+```bash
+./build.sh
+```
+
+## Requirements
+
+- Ruby >= 2.7.0
+- C compiler (gcc, clang)
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b feature/my-feature`)
+3. Commit your changes (`git commit -am 'Add feature'`)
+4. Push to the branch (`git push origin feature/my-feature`)
+5. Create a Pull Request
+
+## License
+
+MIT — see [LICENSE.txt](LICENSE.txt).

data/ext/multi_compress/extconf.rb

@@ -0,0 +1,171 @@
+# frozen_string_literal: true
+
+require "mkmf"
+
+USE_SYSTEM = arg_config("--use-system-libraries") ||
+             ENV["COMPRESS_USE_SYSTEM_LIBRARIES"]
+
+ZSTD_SUBDIRS   = %w[lib/common lib/multi_compress lib/decompress lib/dictBuilder].freeze
+BROTLI_SUBDIRS = %w[c/common c/enc c/dec].freeze
+LZ4_SOURCES    = %w[lz4.c lz4hc.c lz4frame.c].freeze
+
+def find_vendor_dir
+  candidates = [
+    File.join(__dir__, "vendor"),
+    File.join(__dir__, "..", "..", "..", "..", "ext", "multi_compress", "vendor"),
+    File.expand_path("../../ext/multi_compress/vendor", __dir__),
+    File.join(Dir.pwd, "ext", "multi_compress", "vendor"),
+  ]
+
+  dir = __dir__
+  6.times do
+    candidates << File.join(dir, "ext", "multi_compress", "vendor")
+    dir = File.dirname(dir)
+  end
+
+  candidates.find { |path| File.exist?(File.join(path, ".vendored")) }
+            &.then { |path| File.expand_path(path) }
+end
+
+def find_compress_c_dir
+  candidates = [
+    __dir__,
+    File.join(__dir__, "..", "..", "..", "..", "ext", "multi_compress"),
+  ]
+
+  dir = __dir__
+  6.times do
+    candidates << File.join(dir, "ext", "multi_compress")
+    dir = File.dirname(dir)
+  end
+
+  candidates.find { |path| File.exist?(File.join(path, "multi_compress.c")) }
+            &.then { |path| File.expand_path(path) } || __dir__
+end
+
+def configure_system_libraries
+  puts "Building with SYSTEM libraries"
+
+  configure_homebrew if RUBY_PLATFORM.include?("darwin")
+
+  require_system_lib("zstd",   header: "zstd.h",           func: "ZSTD_compress")
+  require_system_lib("lz4",    header: "lz4.h",            func: "LZ4_compress_default")
+  require_system_lib("brotli", header: "brotli/encode.h",  func: "BrotliEncoderCreateInstance", lib: "brotlienc")
+
+  have_header("zdict.h")
+  have_header("lz4hc.h")
+  have_library("lz4", "LZ4_compress_HC")
+  have_header("brotli/decode.h")
+  have_library("brotlidec", "BrotliDecoderCreateInstance")
+end
+
+def configure_homebrew
+  dir_config("homebrew", "/opt/homebrew")
+  $CPPFLAGS += " -I/opt/homebrew/include"
+  $LDFLAGS  += " -L/opt/homebrew/lib"
+end
+
+def require_system_lib(name, header:, func:, lib: name)
+  return if have_header(header) && have_library(lib, func)
+
+  abort "Missing #{name}. Install: apt install lib#{name}-dev / brew install #{name}"
+end
+
+def configure_vendored_libraries(vendor_dir)
+  versions = File.read(File.join(vendor_dir, ".vendored"))
+  puts "Building with VENDORED libraries from #{vendor_dir}"
+  puts "  #{versions.tr("\n", ", ")}"
+
+  zstd_dir   = File.join(vendor_dir, "zstd")
+  lz4_dir    = File.join(vendor_dir, "lz4")
+  brotli_dir = File.join(vendor_dir, "brotli")
+
+  all_vendor_srcs = collect_vendor_sources(zstd_dir, lz4_dir, brotli_dir)
+
+  puts "  #{all_vendor_srcs.length} vendored C files"
+
+  add_include_dirs(zstd_dir, lz4_dir, brotli_dir)
+
+  vpath_dirs = build_vpath_dirs(zstd_dir, lz4_dir, brotli_dir)
+
+  deduplicate_sources!(all_vendor_srcs)
+
+  compress_c_dir = find_compress_c_dir
+
+  $srcs  = ["multi_compress.c"] + all_vendor_srcs.map { |s| File.basename(s) }
+  $VPATH = [compress_c_dir] + vpath_dirs
+
+  $warnflags = ""
+
+  vpath_dirs
+end
+
+def collect_vendor_sources(zstd_dir, lz4_dir, brotli_dir)
+  zstd_srcs = ZSTD_SUBDIRS.flat_map { |d| Dir[File.join(zstd_dir, d, "*.c")] }
+
+  lz4_srcs = LZ4_SOURCES.filter_map do |f|
+    path = File.join(lz4_dir, "lib", f)
+    path if File.exist?(path)
+  end
+
+  brotli_srcs = BROTLI_SUBDIRS.flat_map { |d| Dir[File.join(brotli_dir, d, "*.c")] }
+
+  zstd_srcs + lz4_srcs + brotli_srcs
+end
+
+def add_include_dirs(zstd_dir, lz4_dir, brotli_dir)
+  [
+    File.join(zstd_dir, "lib"),
+    File.join(zstd_dir, "lib", "common"),
+    File.join(lz4_dir, "lib"),
+    File.join(brotli_dir, "c", "include"),
+  ].each { |d| $CPPFLAGS += " -I#{d}" }
+end
+
+def build_vpath_dirs(zstd_dir, lz4_dir, brotli_dir)
+  ZSTD_SUBDIRS.map { |d| File.join(zstd_dir, d) } +
+    [File.join(lz4_dir, "lib")] +
+    BROTLI_SUBDIRS.map { |d| File.join(brotli_dir, d) }
+end
+
+def deduplicate_sources!(sources)
+  seen = {}
+
+  sources.reject! do |src|
+    basename = File.basename(src)
+    duplicate = seen.key?(basename)
+    puts "  SKIP duplicate: #{src}" if duplicate
+    seen[basename] = true
+    duplicate
+  end
+end
+
+def patch_makefile_vpath!(vpath_dirs)
+  makefile = File.read("Makefile")
+  return if makefile.include?("# vendored vpath")
+
+  vpath_lines = vpath_dirs.map { |d| "vpath %.c #{d}" }.join("\n")
+
+  makefile.sub!(/^(VPATH\s*=.*)$/m) { "#{Regexp.last_match(1)}\n# vendored vpath\n#{vpath_lines}" }
+  File.write("Makefile", makefile)
+  puts "  Patched Makefile with #{vpath_dirs.length} VPATH entries"
+end
+
+# --- Main ---
+
+VENDOR_DIR = find_vendor_dir
+VENDORED   = !VENDOR_DIR.nil?
+
+if USE_SYSTEM || !VENDORED
+  configure_system_libraries
+  vpath_dirs = nil
+else
+  vpath_dirs = configure_vendored_libraries(VENDOR_DIR)
+end
+
+$CFLAGS += " -O2"
+$CFLAGS += " -DXXH_NAMESPACE=MULTICOMPRESS_"
+
+create_makefile("multi_compress/multi_compress")
+
+patch_makefile_vpath!(vpath_dirs) if VENDORED && !USE_SYSTEM && vpath_dirs