zstd-native-ruby 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 19d7981beca0bcd924a81c39307f30568884526f736e944382acf264db1e28c1
+  data.tar.gz: 53d08808dedfc4a2702b3192cb69db12adca1a3545e1bfacae8d339df472320c
+SHA512:
+  metadata.gz: 44b3b749107dc141c7038884d0362f81fa8ffe0ef8a50cfe19969de456da6a5e20271b49eb896a9626f37b02219dd289eece6ce366b93d9f9598508c900225b1
+  data.tar.gz: 39b970609a14482d29d8278c8928b5b893be0cb4cb03b5d3250e5a1d663ce3338ee1e155f8ad39924f66b227099203bc3cff8d487889c84744860c852846d9b1

data/CLAUDE.md

@@ -0,0 +1,101 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+zstd-ruby is a Ruby gem that provides bindings to the Facebook Zstandard (zstd) compression library. The gem bundles the native zstd C library and exposes compression/decompression functionality to Ruby applications.
+
+## Architecture
+
+### Native Extension Structure
+- **ext/zstd_ruby/**: Contains C extension code that interfaces with the zstd library
+  - Wraps zstd C API for compression and decompression
+  - Handles Ruby<->C type conversions and memory management
+  - Uses extconf.rb to generate the Makefile for compilation
+- **lib/**: Ruby wrapper code that provides the high-level API
+  - Main entry point is `lib/zstd_ruby.rb`
+  - Requires the compiled native extension
+  - Provides user-friendly Ruby API on top of C bindings
+
+### Bundled Library
+- **vendor/zstd/**: Contains the Facebook zstd library source code
+  - Bundled to ensure consistent behavior across platforms
+  - Version should be tracked and updated periodically
+  - Library is compiled as part of gem installation
+
+### API Design
+The gem should support:
+- File-based compression: input filename → output filename
+- File-based decompression: input filename → output filename
+- Compression level parameter (passed to zstd library during compression)
+
+## Development Commands
+
+### Setup
+```bash
+bundle install          # Install dependencies
+rake compile           # Compile the native extension
+```
+
+### Testing
+```bash
+rake test              # Run full test suite
+ruby test/test_*.rb    # Run a specific test file
+```
+
+### Building
+```bash
+rake build             # Build the gem package
+gem install pkg/*.gem  # Install locally built gem
+```
+
+### Release
+```bash
+rake release           # Build and push to RubyGems.org
+```
+
+## Native Extension Development
+
+### Compilation
+The native extension must be compiled before running tests or using the gem in development:
+```bash
+rake compile
+```
+
+After making changes to C code in ext/, recompile before testing.
+
+### Debugging
+- Use `printf` debugging in C code for troubleshooting
+- Check `mkmf.log` in ext/zstd_ruby/ if compilation fails
+- Ensure proper memory management (allocate/free) to prevent leaks
+
+## Testing Approach
+
+This gem was developed using Test-Driven Development (TDD):
+1. Tests were written first to define expected behavior
+2. Implementation follows the test specifications
+3. All tests must pass before considering features complete
+
+Test files cover:
+- Basic compression/decompression operations
+- Various compression levels
+- Edge cases (empty files, large files, invalid inputs)
+- Error handling
+
+## Important Notes
+
+### Platform Compatibility
+- The gem includes native extensions that must compile on installation
+- Supports common platforms: Linux, macOS, Windows
+- The bundled zstd library ensures consistent behavior across platforms
+
+### Memory Management
+- C extension must properly manage memory allocation/deallocation
+- Use Ruby's garbage collector integration appropriately
+- Clean up temporary buffers after compression/decompression
+
+### Compression Levels
+- zstd supports compression levels typically from 1 (fastest) to 22 (best compression)
+- Level 3 is default if not specified
+- Validate compression level input in Ruby layer before passing to C

data/LICENSE

@@ -0,0 +1,19 @@
+MIT License
+
+Copyright (c) 2025
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF NOTICE SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,47 @@
+# zstd-native-ruby
+
+Ruby bindings for Facebook's Zstandard (zstd) compression library with bundled native library.
+
+This gem bundles the zstd C library, so you don't need to install zstd separately on your system. It provides native performance with simple Ruby API for file compression and decompression.
+
+## Installation
+
+Install from RubyGems:
+
+```bash
+gem install zstd-native-ruby
+```
+
+Or add to your Gemfile:
+
+```ruby
+gem 'zstd-native-ruby'
+```
+
+## Usage
+
+```ruby
+require 'zstd_ruby'
+
+# Compress a file
+ZstdRuby.compress_file('input.txt', 'output.txt.zst', level: 3)
+
+# Decompress a file
+ZstdRuby.decompress_file('output.txt.zst', 'decompressed.txt')
+```
+
+## Compression Levels
+
+Zstd supports compression levels from 1 (fastest) to 22 (best compression). The default level is 3.
+
+## Development
+
+```bash
+bundle install
+rake compile
+rake test
+```
+
+## License
+
+MIT License

data/ext/zstd_ruby/extconf.rb

@@ -0,0 +1,35 @@
+require "mkmf"
+
+# Get the path to the bundled zstd library
+zstd_path = File.expand_path("../../../vendor/zstd", __FILE__)
+zstd_lib_path = File.join(zstd_path, "lib")
+
+# Add include directories for zstd headers
+$INCFLAGS << " -I#{zstd_lib_path}"
+$INCFLAGS << " -I#{File.join(zstd_lib_path, 'common')}"
+$INCFLAGS << " -I#{File.join(zstd_lib_path, 'compress')}"
+$INCFLAGS << " -I#{File.join(zstd_lib_path, 'decompress')}"
+
+# Set compilation flags
+$CFLAGS << " -O3 -std=c99 -DZSTD_MULTITHREAD -DZSTD_DISABLE_ASM"
+
+# Get list of zstd source files
+zstd_sources = []
+zstd_sources += Dir[File.join(zstd_lib_path, "common/*.c")]
+zstd_sources += Dir[File.join(zstd_lib_path, "compress/*.c")]
+zstd_sources += Dir[File.join(zstd_lib_path, "decompress/*.c")]
+
+# Filter out legacy and deprecated files
+zstd_sources.reject! { |f| f.include?("legacy") || f.include?("deprecated") }
+
+# Add zstd source files to be compiled
+$srcs = ["zstd_ruby.c"] + zstd_sources
+
+# Set additional object files
+$objs = $srcs.map { |s| s.sub(/\.c$/, ".o") }
+
+# Threading support
+have_library("pthread")
+
+# Create the Makefile
+create_makefile("zstd_ruby/zstd_ruby")

data/ext/zstd_ruby/zstd_ruby.c

@@ -0,0 +1,221 @@
+#include <ruby.h>
+#include <zstd.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+static VALUE mZstdRuby;
+
+#define BUFFER_SIZE (128 * 1024) // 128KB buffer
+
+/*
+ * Compress a file using zstd
+ *
+ * @param input_path [String] path to input file
+ * @param output_path [String] path to output file
+ * @param level [Integer] compression level (1-22)
+ * @return [Boolean] true on success
+ */
+static VALUE zstd_compress_file(VALUE self, VALUE input_path, VALUE output_path, VALUE level) {
+    Check_Type(input_path, T_STRING);
+    Check_Type(output_path, T_STRING);
+    Check_Type(level, T_FIXNUM);
+
+    const char *input_path_str = StringValueCStr(input_path);
+    const char *output_path_str = StringValueCStr(output_path);
+    int compression_level = FIX2INT(level);
+
+    FILE *input_file = fopen(input_path_str, "rb");
+    if (!input_file) {
+        rb_raise(rb_eRuntimeError, "Cannot open input file: %s", input_path_str);
+        return Qfalse;
+    }
+
+    FILE *output_file = fopen(output_path_str, "wb");
+    if (!output_file) {
+        fclose(input_file);
+        rb_raise(rb_eRuntimeError, "Cannot open output file: %s", output_path_str);
+        return Qfalse;
+    }
+
+    // Create compression context
+    ZSTD_CCtx *cctx = ZSTD_createCCtx();
+    if (!cctx) {
+        fclose(input_file);
+        fclose(output_file);
+        rb_raise(rb_eRuntimeError, "Failed to create compression context");
+        return Qfalse;
+    }
+
+    // Set compression level
+    ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, compression_level);
+
+    // Allocate buffers
+    void *input_buffer = malloc(BUFFER_SIZE);
+    size_t output_buffer_size = ZSTD_compressBound(BUFFER_SIZE);
+    void *output_buffer = malloc(output_buffer_size);
+
+    if (!input_buffer || !output_buffer) {
+        free(input_buffer);
+        free(output_buffer);
+        ZSTD_freeCCtx(cctx);
+        fclose(input_file);
+        fclose(output_file);
+        rb_raise(rb_eRuntimeError, "Memory allocation failed");
+        return Qfalse;
+    }
+
+    // Read and compress file in chunks
+    size_t bytes_read;
+
+    while ((bytes_read = fread(input_buffer, 1, BUFFER_SIZE, input_file)) > 0) {
+        ZSTD_inBuffer input = {input_buffer, bytes_read, 0};
+
+        // Process all input
+        while (input.pos < input.size) {
+            ZSTD_outBuffer output = {output_buffer, output_buffer_size, 0};
+            size_t remaining = ZSTD_compressStream2(cctx, &output, &input, ZSTD_e_continue);
+
+            if (ZSTD_isError(remaining)) {
+                free(input_buffer);
+                free(output_buffer);
+                ZSTD_freeCCtx(cctx);
+                fclose(input_file);
+                fclose(output_file);
+                rb_raise(rb_eRuntimeError, "Compression failed: %s", ZSTD_getErrorName(remaining));
+                return Qfalse;
+            }
+
+            fwrite(output_buffer, 1, output.pos, output_file);
+        }
+    }
+
+    // Finalize the compression stream
+    ZSTD_inBuffer empty_input = {NULL, 0, 0};
+    int finished = 0;
+    while (!finished) {
+        ZSTD_outBuffer output = {output_buffer, output_buffer_size, 0};
+        size_t remaining = ZSTD_compressStream2(cctx, &output, &empty_input, ZSTD_e_end);
+
+        if (ZSTD_isError(remaining)) {
+            free(input_buffer);
+            free(output_buffer);
+            ZSTD_freeCCtx(cctx);
+            fclose(input_file);
+            fclose(output_file);
+            rb_raise(rb_eRuntimeError, "Compression finalization failed: %s", ZSTD_getErrorName(remaining));
+            return Qfalse;
+        }
+
+        fwrite(output_buffer, 1, output.pos, output_file);
+        finished = (remaining == 0);
+    }
+
+    // Cleanup
+    free(input_buffer);
+    free(output_buffer);
+    ZSTD_freeCCtx(cctx);
+    fclose(input_file);
+    fclose(output_file);
+
+    return Qtrue;
+}
+
+/*
+ * Decompress a file using zstd
+ *
+ * @param input_path [String] path to compressed input file
+ * @param output_path [String] path to output file
+ * @return [Boolean] true on success
+ */
+static VALUE zstd_decompress_file(VALUE self, VALUE input_path, VALUE output_path) {
+    Check_Type(input_path, T_STRING);
+    Check_Type(output_path, T_STRING);
+
+    const char *input_path_str = StringValueCStr(input_path);
+    const char *output_path_str = StringValueCStr(output_path);
+
+    FILE *input_file = fopen(input_path_str, "rb");
+    if (!input_file) {
+        rb_raise(rb_eRuntimeError, "Cannot open input file: %s", input_path_str);
+        return Qfalse;
+    }
+
+    FILE *output_file = fopen(output_path_str, "wb");
+    if (!output_file) {
+        fclose(input_file);
+        rb_raise(rb_eRuntimeError, "Cannot open output file: %s", output_path_str);
+        return Qfalse;
+    }
+
+    // Create decompression context
+    ZSTD_DCtx *dctx = ZSTD_createDCtx();
+    if (!dctx) {
+        fclose(input_file);
+        fclose(output_file);
+        rb_raise(rb_eRuntimeError, "Failed to create decompression context");
+        return Qfalse;
+    }
+
+    // Allocate buffers
+    size_t input_buffer_size = ZSTD_DStreamInSize();
+    void *input_buffer = malloc(input_buffer_size);
+    size_t output_buffer_size = ZSTD_DStreamOutSize();
+    void *output_buffer = malloc(output_buffer_size);
+
+    if (!input_buffer || !output_buffer) {
+        free(input_buffer);
+        free(output_buffer);
+        ZSTD_freeDCtx(dctx);
+        fclose(input_file);
+        fclose(output_file);
+        rb_raise(rb_eRuntimeError, "Memory allocation failed");
+        return Qfalse;
+    }
+
+    // Read and decompress file
+    size_t bytes_read;
+    size_t last_result = 0;
+
+    while ((bytes_read = fread(input_buffer, 1, input_buffer_size, input_file)) > 0) {
+        ZSTD_inBuffer input = {input_buffer, bytes_read, 0};
+
+        while (input.pos < input.size) {
+            ZSTD_outBuffer output = {output_buffer, output_buffer_size, 0};
+
+            size_t result = ZSTD_decompressStream(dctx, &output, &input);
+
+            if (ZSTD_isError(result)) {
+                free(input_buffer);
+                free(output_buffer);
+                ZSTD_freeDCtx(dctx);
+                fclose(input_file);
+                fclose(output_file);
+                rb_raise(rb_eRuntimeError, "Decompression failed: %s", ZSTD_getErrorName(result));
+                return Qfalse;
+            }
+
+            fwrite(output_buffer, 1, output.pos, output_file);
+            last_result = result;
+        }
+    }
+
+    // Cleanup
+    free(input_buffer);
+    free(output_buffer);
+    ZSTD_freeDCtx(dctx);
+    fclose(input_file);
+    fclose(output_file);
+
+    return Qtrue;
+}
+
+/*
+ * Initialize the ZstdRuby module
+ */
+void Init_zstd_ruby(void) {
+    mZstdRuby = rb_define_module("ZstdRubyNative");
+
+    rb_define_singleton_method(mZstdRuby, "compress_file_native", zstd_compress_file, 3);
+    rb_define_singleton_method(mZstdRuby, "decompress_file_native", zstd_decompress_file, 2);
+}

data/lib/zstd_ruby.rb

@@ -0,0 +1,68 @@
+require "zstd_ruby/zstd_ruby"
+
+module ZstdRuby
+  # Default compression level
+  DEFAULT_COMPRESSION_LEVEL = 3
+
+  # Valid compression level range
+  MIN_COMPRESSION_LEVEL = 1
+  MAX_COMPRESSION_LEVEL = 22
+
+  class << self
+    #
+    # Compress a file using zstd compression
+    #
+    # @param input_path [String] path to the input file to compress
+    # @param output_path [String] path where the compressed file will be written
+    # @param level [Integer] compression level (1-22, default: 3)
+    #   - Level 1: fastest compression, lower ratio
+    #   - Level 3: default, good balance
+    #   - Level 22: best compression, slowest
+    # @return [Boolean] true on success
+    # @raise [ArgumentError] if compression level is invalid
+    # @raise [RuntimeError] if file operations fail
+    #
+    # @example
+    #   ZstdRuby.compress_file('input.txt', 'output.txt.zst')
+    #   ZstdRuby.compress_file('input.txt', 'output.txt.zst', level: 9)
+    #
+    def compress_file(input_path, output_path, level: DEFAULT_COMPRESSION_LEVEL)
+      # Validate compression level
+      unless level.is_a?(Integer)
+        raise ArgumentError, "Compression level must be an integer"
+      end
+
+      unless level >= MIN_COMPRESSION_LEVEL && level <= MAX_COMPRESSION_LEVEL
+        raise ArgumentError,
+              "Compression level must be between #{MIN_COMPRESSION_LEVEL} and #{MAX_COMPRESSION_LEVEL}"
+      end
+
+      # Convert paths to strings
+      input_path = input_path.to_s
+      output_path = output_path.to_s
+
+      # Call native extension
+      ZstdRubyNative.compress_file_native(input_path, output_path, level)
+    end
+
+    #
+    # Decompress a zstd-compressed file
+    #
+    # @param input_path [String] path to the compressed input file
+    # @param output_path [String] path where the decompressed file will be written
+    # @return [Boolean] true on success
+    # @raise [RuntimeError] if file operations or decompression fail
+    #
+    # @example
+    #   ZstdRuby.decompress_file('input.txt.zst', 'output.txt')
+    #
+    def decompress_file(input_path, output_path)
+      # Convert paths to strings
+      input_path = input_path.to_s
+      output_path = output_path.to_s
+
+      # Call native extension
+      ZstdRubyNative.decompress_file_native(input_path, output_path)
+    end
+  end
+end