vibe_zstd 1.1.0 → 1.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2e4a4b0e94bb623793a619286308a6e459ecf3e5b23b6ddf668b8fe1063b4bef
-  data.tar.gz: 98878819233bc19e8d89f63c0f6b610b7c3177ac2d1d062897cf01e25162d2e8
+  metadata.gz: 5eb8d0ac2293ee84d588b9eb237b0c3b4623989c4db8692ad1ab325fd8047695
+  data.tar.gz: 1732b409b13fd9e99fecb7ebf8d026f9b6d0119012641c96e66faa3d51c0f5ce
 SHA512:
-  metadata.gz: 35135f9a6c458690982d7b17494ca6eec5439a901d210226a839be685d533c7c387b2046985fa603c05771bdf7baa2db3cba08faa97da2ac256a38b659f6afa3
-  data.tar.gz: b739b0318dcc3f88852b974705c2c2e989e96e063def67525d990b96553c2da0dbe6959c741024aaa4c1e88d357969dc6f33580550216e936b098bf910286ea2
+  metadata.gz: 65cc971c6400d69adaca95b0c520c8bd8ec22648bdb57218f5ac833bd0a63db6908c953b4fca127753f2257f8502b34b89eb383c38a8f315431e4051ff7a9401
+  data.tar.gz: 836b8a52c28f8cd0d79ed4a8aa80a286f51dda58e1e483320f4544b2a63ded6e83cdccd3bfea122d267c9a44cb43dabbd2c12e69aeb69162d7d4f4ee7ecb60c3

data/CHANGELOG.md

@@ -7,6 +7,47 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.3.0] - 2026-06-11
+
+### Security
+- Fixed use-after-free: `CompressWriter` and `DecompressReader` now retain their dictionary object. Previously only the raw `ZSTD_CDict*`/`ZSTD_DDict*` pointer was stored, so a dictionary passed as `dict:` without the caller holding their own reference could be garbage-collected while the stream still used it.
+- `DCtx#decompress` now raises `RuntimeError` ("Truncated frame") when an unknown-content-size frame ends mid-stream, instead of silently returning partial output. The known-size path already rejected truncated input; the streaming path now matches.
+- Dictionary training (`train_dict`, `train_dict_cover`, `train_dict_fast_cover`, `finalize_dictionary`) no longer crashes or risks a heap overflow when samples are non-String objects responding to `to_str`, or when a malicious `to_str` mutates other samples mid-validation. Converted samples are retained and the copy is capacity-checked.
+- Source strings are now locked (`rb_str_locktmp`) while the GVL is released during `CCtx#compress`, `DCtx#decompress`, and across `CompressWriter#write`'s IO calls, preventing a use-after-free read if another thread (or re-entrant IO code) mutates the string mid-operation. Unlocking is async-exception-safe via `rb_ensure`.
+- `DecompressReader` snapshots each input chunk (`rb_str_new_frozen`), so IOs that reuse/mutate the buffer string they return can no longer invalidate the decoder's input pointer between reads.
+- `DecompressReader#read` raises `TypeError` when the underlying IO's `read` returns a non-String (non-`to_str`-able) object, instead of crashing the VM.
+- `CCtx.new` / `DCtx.new` raise `ArgumentError` on non-Symbol keyword keys (e.g. `CCtx.new("level" => 3)`) instead of hitting undefined behavior.
+
+### Fixed
+- `DCtx#decompress` (unknown-size path): the C output buffer and per-call dictionary reference are now released via `rb_ensure` on every exit path, so an async exception (e.g. `Timeout`) can no longer leak the buffer or leave the dictionary referenced on the context.
+- `DecompressReader#read(0)` returns `""` without latching EOF, matching IO semantics. Previously it returned `nil` and marked the stream finished.
+- `DecompressReader#gets` no longer mixes character indexes with byte sizes, fixing line splitting with multibyte separators.
+- Build: added `ext/vibe_zstd/depend` so editing the split implementation files (`cctx.c`, `dctx.c`, `dict.c`, `streaming.c`, `frames.c`) or project headers triggers recompilation of the extension.
+
+### Changed
+- `DecompressReader#read(n)` caps its initial allocation (~128KB) and grows geometrically up to `n`, instead of preallocating the full requested size up front (`read(1_000_000_000)` on a small stream no longer allocates 1GB).
+- `VibeZstd::ThreadLocal` uses true thread-local storage (`Thread#thread_variable_get/set`) instead of fiber-local `Thread.current[]`, so fiber-based servers (Falcon, async) reuse one context pool per OS thread rather than churning a fresh pool per fiber.
+- README: prominent warning recommending `max_decompressed_size` when decompressing untrusted input.
+
+## [1.2.0] - 2026-06-06
+
+### Added
+- `DCtx#format` / `#format=` (`ZSTD_d_format`) and magicless-format decompression. Frames produced with `format: 1` (`ZSTD_f_zstd1_magicless`) can now be decompressed by setting `format: 1` on the decompression side.
+- Opt-in decompressed-size limit on `DCtx#decompress`, configurable per-call (`max_decompressed_size:` / `max_size:`), per-instance (`DCtx#max_decompressed_size=`, alias `max_size=`), and as a class default (`DCtx.default_max_decompressed_size=`). Resolved per-call → instance → class → unlimited. Exceeding the limit raises `VibeZstd::DecompressedSizeExceeded` (a subclass of `VibeZstd::Error`). Off by default, preserving existing behavior.
+- `VibeZstd.compress` / `VibeZstd.decompress` now accept context (sticky) parameters as keyword arguments (e.g. `checksum_flag:`, `window_log:`, `workers:`, `format:`), applying them to a fresh context. Per-call options are still passed to the operation.
+
+### Fixed
+- `CCtx#compress` now honors parameters configured on the context (`compression_level`, `checksum_flag`, `window_log`, `workers`, `format`, etc.). It previously used `ZSTD_compressCCtx`, which ignores all sticky parameters, so context configuration was silently discarded and one-shot compression always ran at the default level.
+- `DCtx#decompress` now applies the dictionary on the unknown-content-size path. Dictionary frames produced by `CompressWriter` (which never pledges a size) previously failed to decompress with "Dictionary mismatch".
+- `VibeZstd.read_skippable_frame` caps its allocation to the bytes actually present instead of trusting the frame's content-size header, preventing a tiny truncated input from forcing a multi-gigabyte allocation.
+- Passing an unknown keyword to `VibeZstd.compress` / `VibeZstd.decompress` now raises `NoMethodError` instead of being silently ignored.
+
+## [1.1.1] - 2026-03-25
+
+### Fixed
+- Fix `RuntimeError: can't set length of shared string` in `CompressWriter` when writing to File IO on Ruby 3.3+ caused by COW buffer sharing during `IO#write`
+- Fix vendored zstd build flags (`-DZSTD_MULTITHREAD`, `-DXXH_NAMESPACE`, `-DZSTD_LEGACY_SUPPORT`) not propagating to compiled sources, restoring multithreaded compression support (`workers`, `rsyncable` parameters)
+
 ## [1.1.0] - 2026-03-02
 
 ### Added
@@ -48,6 +89,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Thread pool support for parallel compression
 - Memory-efficient API for large files
 
+[1.2.0]: https://github.com/kreynolds/vibe_zstd/compare/v1.1.1...v1.2.0
+[1.1.1]: https://github.com/kreynolds/vibe_zstd/compare/v1.1.0...v1.1.1
 [1.1.0]: https://github.com/kreynolds/vibe_zstd/compare/v1.0.2...v1.1.0
 [1.0.2]: https://github.com/kreynolds/vibe_zstd/compare/v1.0.1...v1.0.2
 [1.0.1]: https://github.com/kreynolds/vibe_zstd/compare/v1.0.0...v1.0.1

data/README.md

@@ -134,6 +134,33 @@ decompressed = VibeZstd.decompress(compressed)
 compressed = VibeZstd.compress(data, level: 5)
 ```
 
+The convenience methods accept the same options as `CCtx#compress` /
+`DCtx#decompress`, plus any context (sticky) parameter, which is applied to the
+internally-created context:
+
+```ruby
+# Per-call options (level, dict, pledged_size / dict, initial_capacity, max_size)
+compressed = VibeZstd.compress(data, level: 9, dict: cdict)
+decompressed = VibeZstd.decompress(compressed, dict: ddict, max_size: 10 * 1024 * 1024)
+
+# Context parameters work too (checksum_flag, window_log, workers, format, ...)
+compressed = VibeZstd.compress(data, checksum_flag: true, window_log: 20)
+
+# An unknown keyword raises NoMethodError instead of being silently ignored.
+```
+
+> [!IMPORTANT]
+> **Decompressing untrusted input?** Always set an output-size limit. By default
+> there is no cap, so a tiny malicious frame can demand an enormous allocation
+> (a "decompression bomb"):
+>
+> ```ruby
+> VibeZstd.decompress(untrusted, max_decompressed_size: 50 * 1024 * 1024)
+> ```
+>
+> See [Limiting Decompressed Size](#limiting-decompressed-size) for per-instance
+> and process-wide defaults.
+
 ### Using Contexts (Recommended)
 
 For multiple operations, create reusable contexts:
@@ -550,6 +577,50 @@ VibeZstd::DCtx.default_initial_capacity = nil
 - **Large data (> 1MB)**: Set to `1_048_576` or higher
 - **Known-size frames**: Not applicable (size read from frame header)
 
+#### Limiting Decompressed Size
+
+When decompressing untrusted data, an attacker-controlled frame can declare or
+expand to an enormous output (a "decompression bomb"). Set an opt-in output-size
+limit; exceeding it raises `VibeZstd::DecompressedSizeExceeded` (a subclass of
+`VibeZstd::Error`). It is off by default, so existing behavior is unchanged.
+
+```ruby
+# Per call (alias: max_size)
+VibeZstd::DCtx.new.decompress(data, max_decompressed_size: 50 * 1024 * 1024)
+
+# Per instance
+dctx = VibeZstd::DCtx.new(max_decompressed_size: 50 * 1024 * 1024)
+
+# Class default for all new instances
+VibeZstd::DCtx.default_max_decompressed_size = 100 * 1024 * 1024
+VibeZstd::DCtx.default_max_decompressed_size = nil  # unlimited again
+
+# Resolution order: per-call → instance → class default → unlimited
+begin
+  dctx.decompress(untrusted)
+rescue VibeZstd::DecompressedSizeExceeded => e
+  warn "rejected oversized payload: #{e.message}"
+end
+```
+
+For frames with a known content size the limit is checked against the declared
+size *before* allocating; for unknown-size frames the output buffer never grows
+past the limit. This complements `window_log_max`, which bounds decoder *window*
+memory but not total output size.
+
+#### Magicless Frames
+
+Frames compressed with `format: 1` (`ZSTD_f_zstd1_magicless`) omit the 4-byte
+magic number. Decompress them by setting the same format on the decompression
+side:
+
+```ruby
+compressed = VibeZstd.compress(data, format: 1)        # CCtx format parameter
+original = VibeZstd.decompress(compressed, format: 1)  # DCtx#format / #format=
+```
+
+A magicless `DCtx` cannot read ordinary (magic-prefixed) frames, and vice versa.
+
 ### Memory Estimation
 
 Estimate memory usage before creating contexts:
@@ -810,8 +881,9 @@ end
 ### Module Methods
 
 ```ruby
-VibeZstd.compress(data, level: nil, dict: nil)
-VibeZstd.decompress(data, dict: nil)
+# Per-call options plus any context (sticky) parameter as a keyword.
+VibeZstd.compress(data, level: nil, dict: nil, pledged_size: nil, **ctx_params)
+VibeZstd.decompress(data, dict: nil, initial_capacity: nil, max_decompressed_size: nil, **ctx_params)
 VibeZstd.frame_content_size(data)
 VibeZstd.compress_bound(size)
 VibeZstd.train_dict(samples, max_dict_size: 112640)
@@ -839,6 +911,7 @@ cctx.content_size_flag = 1
 cctx.compression_level = 9
 cctx.window_log = 20
 cctx.workers = 4
+cctx.format = 1   # ZSTD_f_zstd1_magicless (omit the 4-byte magic number)
 # ... and many more
 
 # Class methods
@@ -850,13 +923,16 @@ VibeZstd::CCtx.estimate_memory(level)
 
 ```ruby
 dctx = VibeZstd::DCtx.new(**params)
-dctx.decompress(data, dict: nil, initial_capacity: nil)
+dctx.decompress(data, dict: nil, initial_capacity: nil, max_decompressed_size: nil)
 dctx.use_prefix(prefix_data)
 dctx.initial_capacity = 1_048_576
 dctx.window_log_max = 20
+dctx.max_decompressed_size = 50 * 1024 * 1024  # alias: max_size; raises DecompressedSizeExceeded
+dctx.format = 1                                # ZSTD_d_format (magicless frames)
 
 # Class methods
 VibeZstd::DCtx.default_initial_capacity = value
+VibeZstd::DCtx.default_max_decompressed_size = value  # 0/nil = unlimited
 VibeZstd::DCtx.parameter_bounds(param)
 VibeZstd::DCtx.frame_content_size(data)
 VibeZstd::DCtx.estimate_memory

data/ext/vibe_zstd/cctx.c

@@ -7,6 +7,11 @@ extern rb_data_type_t vibe_zstd_cctx_type;
 // Helper to set CCtx parameter from Ruby keyword argument
 static int
 vibe_zstd_cctx_init_param_iter(VALUE key, VALUE value, VALUE self) {
+    // Reject non-Symbol keys early; SYM2ID on a non-Symbol is undefined behavior.
+    if (!SYMBOL_P(key)) {
+        rb_raise(rb_eArgError, "keyword key must be a Symbol, got %"PRIsVALUE, rb_inspect(key));
+    }
+
     // Build the setter method name: key + "="
     const char* key_str = rb_id2name(SYM2ID(key));
     char setter[256];
@@ -45,35 +50,40 @@ vibe_zstd_cctx_estimate_memory(VALUE self, VALUE level) {
 // Releasing the GVL allows other Ruby threads to run during CPU-intensive compression.
 typedef struct {
     ZSTD_CCtx* cctx;
-    ZSTD_CDict* cdict;
     const void* src;
     size_t srcSize;
     void* dst;
     size_t dstCapacity;
-    int compressionLevel;
     size_t result;
 } compress_args;
 
 // Compress without holding Ruby's GVL
 // Called via rb_thread_call_without_gvl to allow parallel Ruby thread execution
-// during CPU-intensive compression operations
+// during CPU-intensive compression operations.
+//
+// Uses ZSTD_compress2 (the advanced one-shot API) so that "sticky" parameters
+// configured on the context (compression_level, checksum_flag, window_log,
+// workers, long_distance_matching, etc.) are honored. The legacy ZSTD_compressCCtx
+// silently ignores all sticky parameters, which made context configuration a no-op.
 static void*
 compress_without_gvl(void* arg) {
     compress_args* args = arg;
-    if (args->cdict) {
-        args->result = ZSTD_compress_usingCDict(args->cctx, args->dst, args->dstCapacity, args->src, args->srcSize, args->cdict);
-    } else {
-        args->result = ZSTD_compressCCtx(args->cctx, args->dst, args->dstCapacity, args->src, args->srcSize, args->compressionLevel);
-    }
+    args->result = ZSTD_compress2(args->cctx, args->dst, args->dstCapacity, args->src, args->srcSize);
     return NULL;
 }
 
 // CCtx compress - Compress data using this context
 //
-// Supports per-operation parameters via keyword arguments:
-// - level: Compression level (overrides context setting for this operation)
-// - dict: CDict to use for compression
-// - pledged_size: Expected input size for optimization (optional)
+// Honors all parameters configured on the context (sticky parameters), e.g.
+// compression_level, checksum_flag, window_log, workers, etc.
+//
+// Supports per-operation overrides via keyword arguments:
+// - level: Compression level for this call only (restored afterward)
+// - dict: CDict to use for this call only (un-referenced afterward)
+// - pledged_size: Expected input size (enforced; resets after the frame)
+//
+// Per-call overrides are applied around the compression and then restored so
+// repeated one-shot calls on the same context remain independent.
 //
 // Uses ZSTD_compressBound to allocate worst-case output buffer size,
 // which is the recommended approach for one-shot compression.
@@ -86,19 +96,20 @@ vibe_zstd_cctx_compress(int argc, VALUE* argv, VALUE self) {
     TypedData_Get_Struct(self, vibe_zstd_cctx, &vibe_zstd_cctx_type, cctx);
     StringValue(data);
 
-    // Extract keyword arguments
-    int lvl = ZSTD_defaultCLevel();
+    // Extract keyword arguments (all optional, all per-call overrides)
+    int has_level = 0;
+    int lvl = 0;
     ZSTD_CDict* cdict = NULL;
+    int has_pledged = 0;
     unsigned long long pledged_size = ZSTD_CONTENTSIZE_UNKNOWN;
 
     if (!NIL_P(options)) {
-        // Handle level keyword argument
         VALUE level_val = rb_hash_aref(options, ID2SYM(rb_intern("level")));
         if (!NIL_P(level_val)) {
+            has_level = 1;
             lvl = NUM2INT(level_val);
         }
 
-        // Handle dict keyword argument
         VALUE dict_val = rb_hash_aref(options, ID2SYM(rb_intern("dict")));
         if (!NIL_P(dict_val)) {
             vibe_zstd_cdict* cdict_struct;
@@ -106,18 +117,44 @@ vibe_zstd_cctx_compress(int argc, VALUE* argv, VALUE self) {
             cdict = cdict_struct->cdict;
         }
 
-        // Handle pledged_size keyword argument
         VALUE pledged_size_val = rb_hash_aref(options, ID2SYM(rb_intern("pledged_size")));
         if (!NIL_P(pledged_size_val)) {
+            has_pledged = 1;
             pledged_size = NUM2ULL(pledged_size_val);
         }
     }
 
-    // Set pledged size if provided
-    if (pledged_size != ZSTD_CONTENTSIZE_UNKNOWN) {
-        size_t result = ZSTD_CCtx_setPledgedSrcSize(cctx->cctx, pledged_size);
-        if (ZSTD_isError(result)) {
-            rb_raise(rb_eRuntimeError, "Failed to set pledged_size %llu: %s", pledged_size, ZSTD_getErrorName(result));
+    // Apply per-call compression level override without permanently mutating the
+    // context's configured level. The previous value is captured and restored.
+    int prev_level = 0;
+    if (has_level) {
+        size_t gp = ZSTD_CCtx_getParameter(cctx->cctx, ZSTD_c_compressionLevel, &prev_level);
+        if (ZSTD_isError(gp)) {
+            rb_raise(rb_eRuntimeError, "Failed to read compression level: %s", ZSTD_getErrorName(gp));
+        }
+        size_t sp = ZSTD_CCtx_setParameter(cctx->cctx, ZSTD_c_compressionLevel, lvl);
+        if (ZSTD_isError(sp)) {
+            rb_raise(rb_eArgError, "Invalid level %d: %s", lvl, ZSTD_getErrorName(sp));
+        }
+    }
+
+    // Reference a per-call dictionary; un-referenced after compression so the
+    // context returns to no-dictionary mode for subsequent calls.
+    if (cdict) {
+        size_t rc = ZSTD_CCtx_refCDict(cctx->cctx, cdict);
+        if (ZSTD_isError(rc)) {
+            if (has_level) ZSTD_CCtx_setParameter(cctx->cctx, ZSTD_c_compressionLevel, prev_level);
+            rb_raise(rb_eRuntimeError, "Failed to set dictionary: %s", ZSTD_getErrorName(rc));
+        }
+    }
+
+    // Set pledged size if provided (resets to UNKNOWN automatically after the frame)
+    if (has_pledged) {
+        size_t sps = ZSTD_CCtx_setPledgedSrcSize(cctx->cctx, pledged_size);
+        if (ZSTD_isError(sps)) {
+            if (cdict) ZSTD_CCtx_refCDict(cctx->cctx, NULL);
+            if (has_level) ZSTD_CCtx_setParameter(cctx->cctx, ZSTD_c_compressionLevel, prev_level);
+            rb_raise(rb_eRuntimeError, "Failed to set pledged_size %llu: %s", pledged_size, ZSTD_getErrorName(sps));
         }
     }
 
@@ -126,15 +163,24 @@ vibe_zstd_cctx_compress(int argc, VALUE* argv, VALUE self) {
     VALUE result_str = rb_str_new(NULL, dstCapacity);
     compress_args args = {
         .cctx = cctx->cctx,
-        .cdict = cdict,
         .src = RSTRING_PTR(data),
         .srcSize = srcSize,
         .dst = RSTRING_PTR(result_str),
         .dstCapacity = dstCapacity,
-        .compressionLevel = lvl,
         .result = 0
     };
-    rb_thread_call_without_gvl(compress_without_gvl, &args, NULL, NULL);
+    // Lock the source string for the duration of the GVL-released compression.
+    // Without this, another Ruby thread holding the same String object could
+    // modify or reallocate it while compression reads from its buffer, causing
+    // a use-after-free read.  The helper unlocks via rb_ensure so the string
+    // is never left permanently locked, even if an async exception (e.g.
+    // Timeout, Thread#raise) is delivered when the GVL is reacquired.
+    vibe_zstd_nogvl_with_str_locked(compress_without_gvl, &args, data);
+
+    // Restore context state so repeated one-shot calls remain independent.
+    if (cdict) ZSTD_CCtx_refCDict(cctx->cctx, NULL);
+    if (has_level) ZSTD_CCtx_setParameter(cctx->cctx, ZSTD_c_compressionLevel, prev_level);
+
     if (ZSTD_isError(args.result)) {
         rb_raise(rb_eRuntimeError, "Compression failed: %s", ZSTD_getErrorName(args.result));
     }