deflate-ruby 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 68bc8b3a391756a9e043136b9b124086964806fcbb5683810181f4fcd19442ab
+  data.tar.gz: 17442db0cf381e96a52c0c198efdbf762b360c7b5decb552e82d1e8b72f0adec
+SHA512:
+  metadata.gz: 46dbf853b6d7a0632253f167a2f792a69cc85c6623f8eaa77f4f96f6153c7f42af13e4f8c7efb3675245f2303899c80825e12aa3d94cdfbd03ace89d646bc9f8
+  data.tar.gz: 00fffef0a4c970467a3ffd9501f843af740d280f181af56255defb0189c82f5f0552133a3b3f27f54e322bfff0cbe860ab31771028c469f9d4ed58f23ba34a08

data/CLAUDE.md

@@ -0,0 +1,138 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+DeflateRuby is a Ruby gem that wraps the libdeflate C library to provide fast compression and decompression for DEFLATE, zlib, and gzip formats. The gem is built as a C extension that directly integrates libdeflate source files.
+
+## Architecture
+
+### Core Components
+
+1. **C Extension** (`ext/deflate_ruby/`)
+   - `deflate_ruby.c`: Main C extension wrapper implementing Ruby bindings for libdeflate
+   - `extconf.rb`: Extension configuration that compiles libdeflate source files along with the wrapper
+   - `libdeflate/`: Complete libdeflate source tree (v1.21) including all platform-specific optimizations
+
+2. **Ruby Library** (`lib/deflate_ruby/`)
+   - `deflate_ruby.rb`: High-level Ruby API providing file-based compression/decompression methods
+   - `version.rb`: Gem version constant
+   - `deflate_ruby.so`: Compiled C extension (generated during build)
+
+3. **Test Suite** (`test/`)
+   - `deflate_test.rb`: Tests for raw DEFLATE format
+   - `zlib_test.rb`: Tests for zlib format including stdlib interoperability
+   - `gzip_test.rb`: Tests for gzip format including stdlib interoperability
+
+### Key Design Decisions
+
+- **Embedded libdeflate**: The libdeflate source is included directly in `ext/deflate_ruby/libdeflate/` and compiled as part of the extension. This ensures consistent behavior across platforms and avoids external dependencies.
+
+- **Buffer Management**: Decompression uses an adaptive buffer strategy starting at 100x the compressed size and doubling up to 10 attempts. This handles highly compressible data (like repetitive text) that can expand significantly.
+
+- **Dual API**: The gem provides both string-based methods (e.g., `deflate_compress(data)`) and file-based methods (e.g., `deflate_compress_file(input, output)`) for flexibility.
+
+## Common Development Commands
+
+### Building the Extension
+
+```bash
+# Initial setup
+cd ext/deflate_ruby
+ruby extconf.rb
+make
+
+# Copy compiled extension to lib directory
+cp deflate_ruby.so ../../lib/deflate_ruby/
+
+# Or use rake
+cd ../..
+rake compile
+```
+
+### Running Tests
+
+```bash
+# Run all tests
+rake test
+
+# Run specific test file
+ruby -Ilib:test test/deflate_test.rb
+ruby -Ilib:test test/zlib_test.rb
+ruby -Ilib:test test/gzip_test.rb
+
+# Run single test method
+ruby -Ilib:test test/deflate_test.rb -n test_deflate_compress_decompress_roundtrip
+```
+
+### Cleaning Build Artifacts
+
+```bash
+cd ext/deflate_ruby
+make clean
+
+# Or use rake
+cd ../..
+rake clean
+```
+
+## Important Implementation Details
+
+### C Extension (ext/deflate_ruby/deflate_ruby.c)
+
+- **Error Handling**: Uses custom `DeflateRuby::Error` exception class for all compression/decompression errors
+- **Memory Management**: All libdeflate compressor/decompressor objects are properly freed after use
+- **Compression Levels**: Supports levels 1-12 (libdeflate extends beyond standard 1-9)
+- **Buffer Allocation**: Uses Ruby's string buffer API (`rb_str_buf_new`, `rb_str_resize`) for efficient memory handling
+
+### Extension Configuration (ext/deflate_ruby/extconf.rb)
+
+- Compiles all libdeflate source files from `lib/*.c` and architecture-specific subdirectories (`lib/x86/*.c`, `lib/arm/*.c`)
+- Uses `-O2` optimization and `-std=c99` for C99 compatibility
+- Sets up include paths to find `libdeflate.h`
+
+### Test Data Characteristics
+
+- Tests use moderately large data (1000x repetitions) to see meaningful compression level differences
+- For gzip tests, binary data is prepended to text to make compression level differences more apparent
+- Compression level tests compare level 1 vs level 6 (not higher levels) as very high levels can sometimes produce larger output on small, highly compressible data
+
+## libdeflate Integration
+
+The libdeflate library (https://github.com/ebiggers/libdeflate) is a heavily optimized compression library that:
+
+- Provides faster compression/decompression than zlib for whole-buffer operations
+- Includes CPU-specific optimizations (x86 SSE/AVX, ARM NEON)
+- Supports DEFLATE, zlib, and gzip formats
+- Is designed for non-streaming (whole-buffer) use cases
+
+**Version**: Currently using libdeflate v1.21
+
+**Updating libdeflate**: To update to a new version:
+1. Download new release from https://github.com/ebiggers/libdeflate/releases
+2. Extract to `ext/deflate_ruby/libdeflate/`
+3. Test thoroughly as API may have changed
+4. Update version reference in this file
+
+## Testing Philosophy
+
+This gem follows test-driven development (TDD):
+
+1. Tests were written first, defining expected behavior
+2. Implementation follows the test specifications
+3. All three formats (deflate, zlib, gzip) have comprehensive test coverage including:
+   - Round-trip compression/decompression
+   - Multiple compression levels
+   - Empty data handling
+   - Invalid data error handling
+   - Binary data support
+   - Interoperability with Ruby's stdlib Zlib
+
+## Compression Format Notes
+
+- **DEFLATE**: Raw compressed stream with no wrapper or checksum
+- **zlib**: DEFLATE stream with zlib header and Adler-32 checksum trailer
+- **gzip**: DEFLATE stream with gzip header (includes magic bytes 0x1f 0x8b) and CRC32 checksum trailer
+
+The gem's zlib and gzip implementations are fully interoperable with Ruby's standard library and command-line tools.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,117 @@
+# DeflateRuby
+
+Fast compression and decompression for Ruby using [libdeflate](https://github.com/ebiggers/libdeflate).
+
+DeflateRuby provides Ruby bindings for libdeflate, a heavily optimized library for DEFLATE, zlib, and gzip compression and decompression. It's significantly faster than Ruby's built-in Zlib for whole-buffer compression/decompression operations.
+
+## Installation
+
+### From RubyGems.org
+
+Install the gem directly:
+
+```bash
+gem install deflate-ruby
+```
+
+Or add this line to your application's Gemfile:
+
+```ruby
+gem 'deflate-ruby'
+```
+
+And then execute:
+
+```bash
+bundle install
+```
+
+### From Source
+
+To build and install from source:
+
+```bash
+git clone https://github.com/deflate-ruby/deflate-ruby.git
+cd deflate-ruby
+gem build deflate-ruby.gemspec
+gem install deflate-ruby-0.1.0.gem
+```
+
+## Usage
+
+DeflateRuby supports three compression formats: DEFLATE (raw), zlib, and gzip.
+
+### String Compression/Decompression
+
+```ruby
+require 'deflate_ruby'
+
+data = "Hello, World!" * 100
+
+# DEFLATE
+compressed = DeflateRuby.deflate_compress(data, 6)  # level 1-12
+decompressed = DeflateRuby.deflate_decompress(compressed)
+
+# Zlib
+compressed = DeflateRuby.zlib_compress(data, 6)
+decompressed = DeflateRuby.zlib_decompress(compressed)
+
+# Gzip
+compressed = DeflateRuby.gzip_compress(data, 6)
+decompressed = DeflateRuby.gzip_decompress(compressed)
+```
+
+### File Compression/Decompression
+
+```ruby
+# DEFLATE
+DeflateRuby.deflate_compress_file("input.txt", "output.deflate", level: 9)
+DeflateRuby.deflate_decompress_file("output.deflate", "decompressed.txt")
+
+# Zlib
+DeflateRuby.zlib_compress_file("input.txt", "output.zlib", level: 6)
+DeflateRuby.zlib_decompress_file("output.zlib", "decompressed.txt")
+
+# Gzip
+DeflateRuby.gzip_compress_file("input.txt", "output.gz", level: 6)
+DeflateRuby.gzip_decompress_file("output.gz", "decompressed.txt")
+```
+
+### Compression Levels
+
+DeflateRuby supports compression levels from 1 to 12:
+- Levels 1-9: Standard compression levels (1 = fastest, 9 = best compression)
+- Levels 10-12: Enhanced compression using minimum-cost-path algorithms
+
+Default level is 6, which provides a good balance between speed and compression ratio.
+
+## Compatibility
+
+DeflateRuby's zlib and gzip formats are fully compatible with Ruby's built-in Zlib library and standard command-line tools like `gzip`.
+
+## Performance
+
+libdeflate is optimized for modern CPUs and can be significantly faster than zlib, especially for compression. It's designed for whole-buffer operations (not streaming), making it ideal for:
+
+- Compressing/decompressing entire files
+- Working with in-memory data
+- Batch compression operations
+- High-throughput scenarios
+
+## Development
+
+After checking out the repo, run:
+
+```bash
+bundle install
+rake compile
+rake test
+```
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](LICENSE.txt).
+
+## Credits
+
+This gem wraps [libdeflate](https://github.com/ebiggers/libdeflate) by Eric Biggers.

data/ext/deflate_ruby/deflate_ruby.c

@@ -0,0 +1,301 @@
+#include <ruby.h>
+#include "libdeflate.h"
+#include <string.h>
+
+static VALUE rb_mDeflateRuby;
+static VALUE rb_eDeflateError;
+
+// Helper function to raise errors
+static void raise_deflate_error(const char *message) {
+    rb_raise(rb_eDeflateError, "%s", message);
+}
+
+// DEFLATE compression
+static VALUE rb_deflate_compress(int argc, VALUE *argv, VALUE self) {
+    VALUE data, level_val;
+    rb_scan_args(argc, argv, "11", &data, &level_val);
+
+    Check_Type(data, T_STRING);
+
+    int level = NIL_P(level_val) ? 6 : NUM2INT(level_val);
+    if (level < 1 || level > 12) {
+        rb_raise(rb_eArgError, "compression level must be between 1 and 12");
+    }
+
+    const char *in_data = RSTRING_PTR(data);
+    size_t in_size = RSTRING_LEN(data);
+
+    struct libdeflate_compressor *compressor = libdeflate_alloc_compressor(level);
+    if (!compressor) {
+        raise_deflate_error("Failed to allocate compressor");
+    }
+
+    size_t max_out_size = libdeflate_deflate_compress_bound(compressor, in_size);
+    VALUE out_str = rb_str_buf_new(max_out_size);
+
+    size_t actual_size = libdeflate_deflate_compress(
+        compressor,
+        in_data,
+        in_size,
+        RSTRING_PTR(out_str),
+        max_out_size
+    );
+
+    libdeflate_free_compressor(compressor);
+
+    if (actual_size == 0) {
+        raise_deflate_error("Compression failed");
+    }
+
+    rb_str_set_len(out_str, actual_size);
+    return out_str;
+}
+
+// DEFLATE decompression
+static VALUE rb_deflate_decompress(VALUE self, VALUE data) {
+    Check_Type(data, T_STRING);
+
+    const char *in_data = RSTRING_PTR(data);
+    size_t in_size = RSTRING_LEN(data);
+
+    struct libdeflate_decompressor *decompressor = libdeflate_alloc_decompressor();
+    if (!decompressor) {
+        raise_deflate_error("Failed to allocate decompressor");
+    }
+
+    // Start with a reasonable buffer size and grow if needed
+    size_t out_size = in_size * 100;  // Start larger for highly compressible data
+    VALUE out_str = rb_str_buf_new(out_size);
+    size_t actual_size;
+
+    enum libdeflate_result result;
+
+    // Try decompression, increase buffer if needed
+    for (int attempts = 0; attempts < 10; attempts++) {
+        result = libdeflate_deflate_decompress(
+            decompressor,
+            in_data,
+            in_size,
+            RSTRING_PTR(out_str),
+            out_size,
+            &actual_size
+        );
+
+        if (result == LIBDEFLATE_SUCCESS) {
+            break;
+        } else if (result == LIBDEFLATE_INSUFFICIENT_SPACE) {
+            out_size *= 2;
+            rb_str_resize(out_str, out_size);
+        } else {
+            libdeflate_free_decompressor(decompressor);
+            raise_deflate_error("Decompression failed: invalid or corrupted data");
+        }
+    }
+
+    libdeflate_free_decompressor(decompressor);
+
+    if (result != LIBDEFLATE_SUCCESS) {
+        raise_deflate_error("Decompression failed: output buffer too small");
+    }
+
+    rb_str_set_len(out_str, actual_size);
+    return out_str;
+}
+
+// ZLIB compression
+static VALUE rb_zlib_compress(int argc, VALUE *argv, VALUE self) {
+    VALUE data, level_val;
+    rb_scan_args(argc, argv, "11", &data, &level_val);
+
+    Check_Type(data, T_STRING);
+
+    int level = NIL_P(level_val) ? 6 : NUM2INT(level_val);
+    if (level < 1 || level > 12) {
+        rb_raise(rb_eArgError, "compression level must be between 1 and 12");
+    }
+
+    const char *in_data = RSTRING_PTR(data);
+    size_t in_size = RSTRING_LEN(data);
+
+    struct libdeflate_compressor *compressor = libdeflate_alloc_compressor(level);
+    if (!compressor) {
+        raise_deflate_error("Failed to allocate compressor");
+    }
+
+    size_t max_out_size = libdeflate_zlib_compress_bound(compressor, in_size);
+    VALUE out_str = rb_str_buf_new(max_out_size);
+
+    size_t actual_size = libdeflate_zlib_compress(
+        compressor,
+        in_data,
+        in_size,
+        RSTRING_PTR(out_str),
+        max_out_size
+    );
+
+    libdeflate_free_compressor(compressor);
+
+    if (actual_size == 0) {
+        raise_deflate_error("Compression failed");
+    }
+
+    rb_str_set_len(out_str, actual_size);
+    return out_str;
+}
+
+// ZLIB decompression
+static VALUE rb_zlib_decompress(VALUE self, VALUE data) {
+    Check_Type(data, T_STRING);
+
+    const char *in_data = RSTRING_PTR(data);
+    size_t in_size = RSTRING_LEN(data);
+
+    struct libdeflate_decompressor *decompressor = libdeflate_alloc_decompressor();
+    if (!decompressor) {
+        raise_deflate_error("Failed to allocate decompressor");
+    }
+
+    size_t out_size = in_size * 100;  // Start larger for highly compressible data
+    VALUE out_str = rb_str_buf_new(out_size);
+    size_t actual_size;
+
+    enum libdeflate_result result;
+
+    for (int attempts = 0; attempts < 10; attempts++) {
+        result = libdeflate_zlib_decompress(
+            decompressor,
+            in_data,
+            in_size,
+            RSTRING_PTR(out_str),
+            out_size,
+            &actual_size
+        );
+
+        if (result == LIBDEFLATE_SUCCESS) {
+            break;
+        } else if (result == LIBDEFLATE_INSUFFICIENT_SPACE) {
+            out_size *= 2;
+            rb_str_resize(out_str, out_size);
+        } else {
+            libdeflate_free_decompressor(decompressor);
+            raise_deflate_error("Decompression failed: invalid or corrupted zlib data");
+        }
+    }
+
+    libdeflate_free_decompressor(decompressor);
+
+    if (result != LIBDEFLATE_SUCCESS) {
+        raise_deflate_error("Decompression failed: output buffer too small");
+    }
+
+    rb_str_set_len(out_str, actual_size);
+    return out_str;
+}
+
+// GZIP compression
+static VALUE rb_gzip_compress(int argc, VALUE *argv, VALUE self) {
+    VALUE data, level_val;
+    rb_scan_args(argc, argv, "11", &data, &level_val);
+
+    Check_Type(data, T_STRING);
+
+    int level = NIL_P(level_val) ? 6 : NUM2INT(level_val);
+    if (level < 1 || level > 12) {
+        rb_raise(rb_eArgError, "compression level must be between 1 and 12");
+    }
+
+    const char *in_data = RSTRING_PTR(data);
+    size_t in_size = RSTRING_LEN(data);
+
+    struct libdeflate_compressor *compressor = libdeflate_alloc_compressor(level);
+    if (!compressor) {
+        raise_deflate_error("Failed to allocate compressor");
+    }
+
+    size_t max_out_size = libdeflate_gzip_compress_bound(compressor, in_size);
+    VALUE out_str = rb_str_buf_new(max_out_size);
+
+    size_t actual_size = libdeflate_gzip_compress(
+        compressor,
+        in_data,
+        in_size,
+        RSTRING_PTR(out_str),
+        max_out_size
+    );
+
+    libdeflate_free_compressor(compressor);
+
+    if (actual_size == 0) {
+        raise_deflate_error("Compression failed");
+    }
+
+    rb_str_set_len(out_str, actual_size);
+    return out_str;
+}
+
+// GZIP decompression
+static VALUE rb_gzip_decompress(VALUE self, VALUE data) {
+    Check_Type(data, T_STRING);
+
+    const char *in_data = RSTRING_PTR(data);
+    size_t in_size = RSTRING_LEN(data);
+
+    struct libdeflate_decompressor *decompressor = libdeflate_alloc_decompressor();
+    if (!decompressor) {
+        raise_deflate_error("Failed to allocate decompressor");
+    }
+
+    size_t out_size = in_size * 100;  // Start larger for highly compressible data
+    VALUE out_str = rb_str_buf_new(out_size);
+    size_t actual_size;
+
+    enum libdeflate_result result;
+
+    for (int attempts = 0; attempts < 10; attempts++) {
+        result = libdeflate_gzip_decompress(
+            decompressor,
+            in_data,
+            in_size,
+            RSTRING_PTR(out_str),
+            out_size,
+            &actual_size
+        );
+
+        if (result == LIBDEFLATE_SUCCESS) {
+            break;
+        } else if (result == LIBDEFLATE_INSUFFICIENT_SPACE) {
+            out_size *= 2;
+            rb_str_resize(out_str, out_size);
+        } else {
+            libdeflate_free_decompressor(decompressor);
+            raise_deflate_error("Decompression failed: invalid or corrupted gzip data");
+        }
+    }
+
+    libdeflate_free_decompressor(decompressor);
+
+    if (result != LIBDEFLATE_SUCCESS) {
+        raise_deflate_error("Decompression failed: output buffer too small");
+    }
+
+    rb_str_set_len(out_str, actual_size);
+    return out_str;
+}
+
+// Initialize the extension
+void Init_deflate_ruby(void) {
+    rb_mDeflateRuby = rb_define_module("DeflateRuby");
+    rb_eDeflateError = rb_define_class_under(rb_mDeflateRuby, "Error", rb_eStandardError);
+
+    // DEFLATE methods
+    rb_define_module_function(rb_mDeflateRuby, "deflate_compress", rb_deflate_compress, -1);
+    rb_define_module_function(rb_mDeflateRuby, "deflate_decompress", rb_deflate_decompress, 1);
+
+    // ZLIB methods
+    rb_define_module_function(rb_mDeflateRuby, "zlib_compress", rb_zlib_compress, -1);
+    rb_define_module_function(rb_mDeflateRuby, "zlib_decompress", rb_zlib_decompress, 1);
+
+    // GZIP methods
+    rb_define_module_function(rb_mDeflateRuby, "gzip_compress", rb_gzip_compress, -1);
+    rb_define_module_function(rb_mDeflateRuby, "gzip_decompress", rb_gzip_decompress, 1);
+}

data/ext/deflate_ruby/extconf.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+require "mkmf"
+
+# Add libdeflate source directory to the include path
+$INCFLAGS << " -I$(srcdir)/libdeflate"
+$CFLAGS << " -O2 -std=c99"
+
+# Define source files to compile
+libdeflate_sources = %w[
+  libdeflate/lib/deflate_compress.c
+  libdeflate/lib/deflate_decompress.c
+  libdeflate/lib/zlib_compress.c
+  libdeflate/lib/zlib_decompress.c
+  libdeflate/lib/gzip_compress.c
+  libdeflate/lib/gzip_decompress.c
+  libdeflate/lib/adler32.c
+  libdeflate/lib/crc32.c
+  libdeflate/lib/utils.c
+]
+
+# Add CPU architecture-specific files
+arch_dirs = Dir.glob("libdeflate/lib/*/").select { |d| File.directory?(d) }
+arch_dirs.each do |dir|
+  Dir.glob("#{dir}*.c").each do |source|
+    libdeflate_sources << source
+  end
+end
+
+# Set object files for libdeflate
+$objs = libdeflate_sources.map { |src| src.sub(/\.c$/, ".o") }
+$objs << "deflate_ruby.o"
+
+create_makefile("deflate_ruby/deflate_ruby")