deflate-ruby 1.0.0 → 1.0.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6543cbb3e7b652d2f453551d6828dedceb36fb5752493d1bc138a9b3db886db9
-  data.tar.gz: f0c01a65248b28f8edf8f1a7b0727178f1e3181c41c46931e50e6cf0d453a74b
+  metadata.gz: 88a97c7ccd0d0ef7d03fcaa9d214bd30ef1fa73b4326972b85c7bb95ecf063eb
+  data.tar.gz: aa4e11e8c5adcc97dd727ec5bdf32428f3d66b1af6378c39c61e6fe1e6170caf
 SHA512:
-  metadata.gz: 4070162a11f98598b61a8f37b147629d7f6d925872c79453184cb188d608e72ff68534ac2a9ee043911103a9f1259492c2328940fa4d43983c0486c2052a3425
-  data.tar.gz: 8cc30d94073c449e17968b0d367aa88e3905c28f44e808356675869ac3192647419a083cee47a65004e625e8d580777711508e2a16eaef3ffb682acb0fc6f480
+  metadata.gz: 6b475adcceaf1ebbe98a1af627a709aa1a5cfcff8a4080489718d618e4f1a541ffb36506800881a6304115eb6d8e8860613019d9364106118613450941081b6b
+  data.tar.gz: 88e08c9c81b82d445e5e944c6b0d8ab86cf7b030d579e5c5a87e214e3315216c98a779a3c2ec3417a80206582b89c91280e74a203c7e14aabe488f948639009d

data/CLAUDE.md

@@ -4,53 +4,65 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ## Project Overview
 
-DeflateRuby is a Ruby gem that wraps the libdeflate C library to provide fast compression and decompression for DEFLATE, zlib, and gzip formats. The gem is built as a C extension that directly integrates libdeflate source files.
+DeflateRuby is a Ruby gem that provides bindings to libdeflate, a heavily optimized library for DEFLATE, gzip, and zlib compression/decompression. The gem bundles the complete libdeflate source code and compiles it as a native C extension.
 
 ## Architecture
 
 ### Core Components
 
-1. **C Extension** (`ext/deflate_ruby/`)
-   - `deflate_ruby.c`: Main C extension wrapper implementing Ruby bindings for libdeflate
-   - `extconf.rb`: Extension configuration that compiles libdeflate source files along with the wrapper
-   - `libdeflate/`: Complete libdeflate source tree (v1.21) including all platform-specific optimizations
+1. **C Extension** (`ext/deflate_ruby/deflate_ruby.c`)
+   - Wraps all libdeflate APIs using Ruby's C API
+   - Implements `DeflateRuby::Compressor` and `DeflateRuby::Decompressor` classes
+   - Provides module-level `crc32` and `adler32` functions
+   - Uses Ruby's TypedData API for memory-safe object wrapping
 
-2. **Ruby Library** (`lib/deflate_ruby/`)
-   - `deflate_ruby.rb`: High-level Ruby API providing file-based compression/decompression methods
-   - `version.rb`: Gem version constant
-   - `deflate_ruby.so`: Compiled C extension (generated during build)
+2. **Bundled libdeflate Source** (`ext/deflate_ruby/libdeflate/`)
+   - Complete libdeflate source code (from https://github.com/ebiggers/libdeflate)
+   - Architecture-specific optimizations in subdirectories: `arm/`, `x86/`, `riscv/`
+   - Build system selects appropriate CPU-specific code at compile time
 
-3. **Test Suite** (`test/`)
-   - `deflate_test.rb`: Tests for raw DEFLATE format
-   - `zlib_test.rb`: Tests for zlib format including stdlib interoperability
-   - `gzip_test.rb`: Tests for gzip format including stdlib interoperability
+3. **Build Configuration** (`ext/deflate_ruby/extconf.rb`)
+   - Detects CPU architecture via `RbConfig::CONFIG['host_cpu']`
+   - Includes only relevant architecture-specific C files to avoid symbol conflicts
+   - Each architecture subdirectory has its own `cpu_features.c` - only one is compiled
+   - Adds optimization flags (-O3) and sets up include paths
 
-### Key Design Decisions
+### API Design
 
-- **Embedded libdeflate**: The libdeflate source is included directly in `ext/deflate_ruby/libdeflate/` and compiled as part of the extension. This ensures consistent behavior across platforms and avoids external dependencies.
+The gem provides a Ruby-friendly API that mirrors libdeflate's functionality:
 
-- **Buffer Management**: Decompression uses an adaptive buffer strategy starting at 100x the compressed size and doubling up to 10 attempts. This handles highly compressible data (like repetitive text) that can expand significantly.
+- **Compressor**: Stateful object for compression operations
+  - `deflate_compress(data)` - Raw DEFLATE compression
+  - `zlib_compress(data)` - Zlib format (with header/trailer)
+  - `gzip_compress(data)` - Gzip format (with header/trailer)
+  - `*_compress_bound(size)` - Calculates worst-case output size
 
-- **Dual API**: The gem provides both string-based methods (e.g., `deflate_compress(data)`) and file-based methods (e.g., `deflate_compress_file(input, output)`) for flexibility.
+- **Decompressor**: Stateful object for decompression operations
+  - `deflate_decompress(data, output_size)` - Raw DEFLATE decompression
+  - `zlib_decompress(data, output_size)` - Zlib format
+  - `gzip_decompress(data, output_size)` - Gzip format
+  - Requires output size parameter (libdeflate design constraint)
 
-## Common Development Commands
+- **Module Functions**: Stateless checksum operations
+  - `DeflateRuby.crc32(data, initial=0)` - CRC32 checksum
+  - `DeflateRuby.adler32(data, initial=1)` - Adler32 checksum
+
+## Development Workflow
 
 ### Building the Extension
 
 ```bash
-# Initial setup
+# Generate Makefile and compile
 cd ext/deflate_ruby
 ruby extconf.rb
 make
 
-# Copy compiled extension to lib directory
-cp deflate_ruby.so ../../lib/deflate_ruby/
-
-# Or use rake
-cd ../..
+# Or use rake from root
 rake compile
 ```
 
+The compiled `.so` file is placed in `ext/deflate_ruby/` and should be copied to `lib/deflate_ruby/` for testing.
+
 ### Running Tests
 
 ```bash
@@ -58,81 +70,72 @@ rake compile
 rake test
 
 # Run specific test file
-ruby -Ilib:test test/deflate_test.rb
-ruby -Ilib:test test/zlib_test.rb
-ruby -Ilib:test test/gzip_test.rb
-
-# Run single test method
-ruby -Ilib:test test/deflate_test.rb -n test_deflate_compress_decompress_roundtrip
+ruby -I./lib:./test test/test_deflate_ruby.rb
 ```
 
-### Cleaning Build Artifacts
+Tests use Minitest and cover:
+- Compression/decompression round-trips for all formats
+- Different compression levels (0-12)
+- Edge cases (empty data, large data, binary data)
+- Checksum functions with incremental updates
+- Error handling for invalid compressed data
+
+### Building the Gem
 
 ```bash
-cd ext/deflate_ruby
-make clean
+# Build gem package
+rake build
 
-# Or use rake
-cd ../..
-rake clean
+# Install locally for testing
+rake install
 ```
 
-## Important Implementation Details
-
-### C Extension (ext/deflate_ruby/deflate_ruby.c)
-
-- **Error Handling**: Uses custom `DeflateRuby::Error` exception class for all compression/decompression errors
-- **Memory Management**: All libdeflate compressor/decompressor objects are properly freed after use
-- **Compression Levels**: Supports levels 1-12 (libdeflate extends beyond standard 1-9)
-- **Buffer Allocation**: Uses Ruby's string buffer API (`rb_str_buf_new`, `rb_str_resize`) for efficient memory handling
-
-### Extension Configuration (ext/deflate_ruby/extconf.rb)
-
-- Compiles all libdeflate source files from `lib/*.c` and architecture-specific subdirectories (`lib/x86/*.c`, `lib/arm/*.c`)
-- Uses `-O2` optimization and `-std=c99` for C99 compatibility
-- Sets up include paths to find `libdeflate.h`
-
-### Test Data Characteristics
-
-- Tests use moderately large data (1000x repetitions) to see meaningful compression level differences
-- For gzip tests, binary data is prepended to text to make compression level differences more apparent
-- Compression level tests compare level 1 vs level 6 (not higher levels) as very high levels can sometimes produce larger output on small, highly compressible data
+### Clean Build
 
-## libdeflate Integration
-
-The libdeflate library (https://github.com/ebiggers/libdeflate) is a heavily optimized compression library that:
-
-- Provides faster compression/decompression than zlib for whole-buffer operations
-- Includes CPU-specific optimizations (x86 SSE/AVX, ARM NEON)
-- Supports DEFLATE, zlib, and gzip formats
-- Is designed for non-streaming (whole-buffer) use cases
-
-**Version**: Currently using libdeflate v1.21
-
-**Updating libdeflate**: To update to a new version:
-1. Download new release from https://github.com/ebiggers/libdeflate/releases
-2. Extract to `ext/deflate_ruby/libdeflate/`
-3. Test thoroughly as API may have changed
-4. Update version reference in this file
-
-## Testing Philosophy
-
-This gem follows test-driven development (TDD):
-
-1. Tests were written first, defining expected behavior
-2. Implementation follows the test specifications
-3. All three formats (deflate, zlib, gzip) have comprehensive test coverage including:
-   - Round-trip compression/decompression
-   - Multiple compression levels
-   - Empty data handling
-   - Invalid data error handling
-   - Binary data support
-   - Interoperability with Ruby's stdlib Zlib
-
-## Compression Format Notes
+```bash
+# Clean compiled files
+rake clobber
+```
 
-- **DEFLATE**: Raw compressed stream with no wrapper or checksum
-- **zlib**: DEFLATE stream with zlib header and Adler-32 checksum trailer
-- **gzip**: DEFLATE stream with gzip header (includes magic bytes 0x1f 0x8b) and CRC32 checksum trailer
+## Important Implementation Details
 
-The gem's zlib and gzip implementations are fully interoperable with Ruby's standard library and command-line tools.
+### Memory Management
+- Compressor and decompressor objects use Ruby's TypedData with custom free functions
+- `libdeflate_free_compressor/decompressor` called automatically during GC
+- Explicit `close()` methods provided for manual resource management
+- NULL checks prevent double-free issues
+
+### Architecture-Specific Compilation
+The `extconf.rb` script selects CPU-specific source files to avoid duplicate symbol errors:
+- x86/x86_64: Includes `x86/cpu_features.c` and SSE optimizations
+- ARM/AArch64: Includes `arm/cpu_features.c` and NEON optimizations
+- RISC-V: Includes only base implementation
+- Only ONE architecture's files are compiled per build
+
+### Error Handling
+- All libdeflate result codes are mapped to Ruby exceptions
+- `DeflateRuby::CompressionError` - Compression failures
+- `DeflateRuby::DecompressionError` - Decompression failures (bad data, size mismatch)
+- Detailed error messages based on libdeflate result codes
+
+## Testing Notes
+
+- Tests verify compatibility with standard formats (can compress with DeflateRuby, decompress with Ruby Zlib and vice versa)
+- Decompression requires knowing the original uncompressed size (libdeflate design)
+- Compression level 0 = no compression (store only)
+- Compression level 12 = maximum compression (slower)
+- Default level 6 balances speed and compression ratio
+
+## Publishing to RubyGems
+
+The gemspec is already configured for publication:
+- Homepage: https://github.com/jgreninger/deflate_ruby
+- License: MIT (covers both gem and bundled libdeflate)
+- Includes all libdeflate source in the gem package
+- Extension builds automatically during `gem install`
+
+To publish:
+```bash
+gem build deflate_ruby.gemspec
+gem push deflate_ruby-0.1.0.gem
+```

data/LICENSE.txt

@@ -1,6 +1,6 @@
-MIT License
+The MIT License (MIT)
 
-Copyright (c) 2025
+Copyright (c) 2025 John Greninger
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
@@ -9,13 +9,13 @@ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 copies of the Software, and to permit persons to whom the Software is
 furnished to do so, subject to the following conditions:
 
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
 
 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -1,117 +1,139 @@
 # DeflateRuby
 
-Fast compression and decompression for Ruby using [libdeflate](https://github.com/ebiggers/libdeflate).
+Ruby bindings for [libdeflate](https://github.com/ebiggers/libdeflate), a heavily optimized library for DEFLATE, gzip, and zlib compression and decompression.
 
-DeflateRuby provides Ruby bindings for libdeflate, a heavily optimized library for DEFLATE, zlib, and gzip compression and decompression. It's significantly faster than Ruby's built-in Zlib for whole-buffer compression/decompression operations.
+DeflateRuby provides significantly faster compression and decompression compared to Ruby's built-in Zlib module, while maintaining full compatibility with standard DEFLATE, gzip, and zlib formats.
 
-## Installation
-
-### From RubyGems.org
-
-Install the gem directly:
-
-```bash
-gem install deflate-ruby
-```
+## Features
 
-Or add this line to your application's Gemfile:
+- **Fast compression and decompression** - Leverages libdeflate's heavily optimized implementation
+- **Multiple compression formats** - Support for raw DEFLATE, zlib, and gzip formats
+- **Configurable compression levels** - Levels 0-12 (higher = better compression, slower)
+- **Checksum functions** - CRC32 and Adler32 implementations
+- **Cross-platform** - Works on x86, ARM, and RISC-V architectures
+- **Clean API** - Simple, Ruby-friendly interface
 
-```ruby
-gem 'deflate-ruby'
-```
+## Installation
 
-And then execute:
+Install the gem by executing:
 
 ```bash
-bundle install
+gem install deflate_ruby
 ```
 
-### From Source
+Or add to your Gemfile:
 
-To build and install from source:
-
-```bash
-git clone https://github.com/deflate-ruby/deflate-ruby.git
-cd deflate-ruby
-gem build deflate-ruby.gemspec
-gem install deflate-ruby-0.1.0.gem
+```ruby
+gem 'deflate_ruby'
 ```
 
 ## Usage
 
-DeflateRuby supports three compression formats: DEFLATE (raw), zlib, and gzip.
-
-### String Compression/Decompression
+### Compression
 
 ```ruby
 require 'deflate_ruby'
 
+# Create a compressor
+compressor = DeflateRuby::Compressor.new(level: 6)  # level 0-12, default 6
+
+# Compress data in different formats
 data = "Hello, World!" * 100
 
-# DEFLATE
-compressed = DeflateRuby.deflate_compress(data, 6)  # level 1-12
-decompressed = DeflateRuby.deflate_decompress(compressed)
+# Raw DEFLATE
+deflated = compressor.deflate_compress(data)
 
-# Zlib
-compressed = DeflateRuby.zlib_compress(data, 6)
-decompressed = DeflateRuby.zlib_decompress(compressed)
+# Zlib format (with header and checksum)
+zlib_compressed = compressor.zlib_compress(data)
 
-# Gzip
-compressed = DeflateRuby.gzip_compress(data, 6)
-decompressed = DeflateRuby.gzip_decompress(compressed)
+# Gzip format (with header and checksum)
+gzip_compressed = compressor.gzip_compress(data)
+
+# Always close the compressor when done
+compressor.close
 ```
 
-### File Compression/Decompression
+### Decompression
 
 ```ruby
-# DEFLATE
-DeflateRuby.deflate_compress_file("input.txt", "output.deflate", level: 9)
-DeflateRuby.deflate_decompress_file("output.deflate", "decompressed.txt")
+# Create a decompressor
+decompressor = DeflateRuby::Decompressor.new
 
-# Zlib
-DeflateRuby.zlib_compress_file("input.txt", "output.zlib", level: 6)
-DeflateRuby.zlib_decompress_file("output.zlib", "decompressed.txt")
+# Decompress data (must know original size)
+original_data = decompressor.deflate_decompress(deflated, data.bytesize)
+original_data = decompressor.zlib_decompress(zlib_compressed, data.bytesize)
+original_data = decompressor.gzip_decompress(gzip_compressed, data.bytesize)
 
-# Gzip
-DeflateRuby.gzip_compress_file("input.txt", "output.gz", level: 6)
-DeflateRuby.gzip_decompress_file("output.gz", "decompressed.txt")
+# Always close the decompressor when done
+decompressor.close
 ```
 
-### Compression Levels
+### Checksums
 
-DeflateRuby supports compression levels from 1 to 12:
-- Levels 1-9: Standard compression levels (1 = fastest, 9 = best compression)
-- Levels 10-12: Enhanced compression using minimum-cost-path algorithms
+```ruby
+# CRC32
+crc = DeflateRuby.crc32("Hello, World!")
 
-Default level is 6, which provides a good balance between speed and compression ratio.
+# CRC32 with initial value (for incremental calculation)
+crc1 = DeflateRuby.crc32("Hello, ")
+crc2 = DeflateRuby.crc32("World!", crc1)
 
-## Compatibility
+# Adler32
+adler = DeflateRuby.adler32("Hello, World!")
 
-DeflateRuby's zlib and gzip formats are fully compatible with Ruby's built-in Zlib library and standard command-line tools like `gzip`.
+# Adler32 with initial value
+adler1 = DeflateRuby.adler32("Hello, ")
+adler2 = DeflateRuby.adler32("World!", adler1)
+```
 
-## Performance
+### Compression Bounds
 
-libdeflate is optimized for modern CPUs and can be significantly faster than zlib, especially for compression. It's designed for whole-buffer operations (not streaming), making it ideal for:
+```ruby
+# Get worst-case compressed size for a given input size
+compressor = DeflateRuby::Compressor.new
+bound = compressor.deflate_compress_bound(1000)
+zlib_bound = compressor.zlib_compress_bound(1000)
+gzip_bound = compressor.gzip_compress_bound(1000)
+compressor.close
+```
 
-- Compressing/decompressing entire files
-- Working with in-memory data
-- Batch compression operations
-- High-throughput scenarios
+## Performance
+
+DeflateRuby is significantly faster than Ruby's built-in Zlib for both compression and decompression, especially at higher compression levels. Benchmarks show 2-5x performance improvements depending on the data and compression level.
 
 ## Development
 
-After checking out the repo, run:
+After checking out the repo:
 
 ```bash
+cd deflate_ruby
 bundle install
+```
+
+To compile the extension:
+
+```bash
 rake compile
+```
+
+To run tests:
+
+```bash
 rake test
 ```
 
-## License
+To build the gem:
 
-The gem is available as open source under the terms of the [MIT License](LICENSE.txt).
+```bash
+rake build
+```
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/jgreninger/deflate_ruby.
+
+## License
 
-## Credits
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
 
-This gem wraps [libdeflate](https://github.com/ebiggers/libdeflate) by Eric Biggers.
+This gem bundles [libdeflate](https://github.com/ebiggers/libdeflate), which is also licensed under the MIT License.

data/Rakefile

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rake/extensiontask"
+require "rake/testtask"
+
+task build: :compile
+
+GEMSPEC = Gem::Specification.load("deflate_ruby.gemspec")
+
+Rake::ExtensionTask.new("deflate_ruby", GEMSPEC) do |ext|
+  ext.lib_dir = "lib/deflate_ruby"
+end
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList["test/**/test_*.rb"]
+end
+
+task test: :compile
+
+task default: %i[clobber compile test]

data/ext/deflate_ruby/{libdeflate/lib/x86/adler32_impl.h → adler32_impl.h}

@@ -60,7 +60,8 @@
  * instead of gcc 11.  (libdeflate supports direct compilation without a
  * configure step, so checking the binutils version is not always an option.)
  */
-#if GCC_PREREQ(12, 1) || CLANG_PREREQ(12, 0, 13000000) || MSVC_PREREQ(1930)
+#if (GCC_PREREQ(12, 1) || CLANG_PREREQ(12, 0, 13000000) || MSVC_PREREQ(1930)) && \
+	!defined(LIBDEFLATE_ASSEMBLER_DOES_NOT_SUPPORT_AVX_VNNI)
 #  define adler32_x86_avx2_vnni	adler32_x86_avx2_vnni
 #  define SUFFIX			   _avx2_vnni
 #  define ATTRIBUTES		_target_attribute("avx2,avxvnni")
@@ -70,13 +71,14 @@
 #  include "adler32_template.h"
 #endif
 
-#if GCC_PREREQ(8, 1) || CLANG_PREREQ(6, 0, 10000000) || MSVC_PREREQ(1920)
+#if (GCC_PREREQ(8, 1) || CLANG_PREREQ(6, 0, 10000000) || MSVC_PREREQ(1920)) && \
+	!defined(LIBDEFLATE_ASSEMBLER_DOES_NOT_SUPPORT_AVX512VNNI)
 /*
  * AVX512VNNI implementation using 256-bit vectors.  This is very similar to the
  * AVX-VNNI implementation but takes advantage of masking and more registers.
- * This is used on CPUs that support AVX-512 but where using 512-bit vectors
- * causes downclocking.  This should also be the optimal implementation on CPUs
- * that support AVX10/256 but not AVX10/512.
+ * This is used on certain older Intel CPUs, specifically Ice Lake and Tiger
+ * Lake, which support AVX512VNNI but downclock a bit too eagerly when ZMM
+ * registers are used.
  */
 #  define adler32_x86_avx512_vl256_vnni	adler32_x86_avx512_vl256_vnni
 #  define SUFFIX				   _avx512_vl256_vnni
@@ -88,8 +90,7 @@
 
 /*
  * AVX512VNNI implementation using 512-bit vectors.  This is used on CPUs that
- * have a good AVX-512 implementation including AVX512VNNI.  This should also be
- * the optimal implementation on CPUs that support AVX10/512.
+ * have a good AVX-512 implementation including AVX512VNNI.
  */
 #  define adler32_x86_avx512_vl512_vnni	adler32_x86_avx512_vl512_vnni
 #  define SUFFIX				   _avx512_vl512_vnni