multi_compress 0.2.4 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +20 -2
- data/GET_STARTED.md +863 -0
- data/README.md +15 -19
- data/ext/multi_compress/extconf.rb +2 -0
- data/ext/multi_compress/multi_compress.c +630 -220
- data/lib/multi_compress/version.rb +1 -1
- data/lib/multi_compress.rb +104 -21
- metadata +2 -1
data/README.md
CHANGED
|
@@ -15,7 +15,7 @@ Bundled library versions in the current release:
|
|
|
15
15
|
|
|
16
16
|
## Technology Overview
|
|
17
17
|
|
|
18
|
-
**MultiCompress** is a comprehensive compression system that unites three
|
|
18
|
+
**MultiCompress** is a comprehensive compression system that unites three modern algorithms in a single platform. Compared with zlib, results depend on the dataset and algorithm choice: LZ4 is usually chosen for speed, Zstd for balance, and Brotli for ratio.
|
|
19
19
|
|
|
20
20
|
| Algorithm | Strength | Best for |
|
|
21
21
|
|-----------|----------|----------|
|
|
@@ -42,36 +42,40 @@ Bundled library versions in the current release:
|
|
|
42
42
|
The system can automatically detect compression algorithms when decompressing data:
|
|
43
43
|
|
|
44
44
|
- **zstd**: Detected by magic bytes `28 B5 2F FD` (little-endian)
|
|
45
|
-
- **lz4**: Detected by internal format header validation (custom internal format, NOT compatible with the standard `lz4` CLI
|
|
45
|
+
- **lz4**: Detected by internal format header validation (custom internal format, NOT compatible with the standard `lz4` CLI)
|
|
46
46
|
- **brotli**: Requires explicit `algo: :brotli` parameter - no auto-detection
|
|
47
47
|
|
|
48
48
|
**Important**: Auto-detection only works for ZSTD and LZ4. Brotli data must be decompressed with explicit algorithm specification.
|
|
49
49
|
|
|
50
|
-
**Security**: Decompression now enforces a default
|
|
50
|
+
**Security**: Decompression now enforces a default 512MB one-shot output cap, a default 2GB cumulative streaming cap, and a 32MB dictionary file size cap.
|
|
51
51
|
|
|
52
52
|
|
|
53
53
|
## Security limits
|
|
54
54
|
|
|
55
|
-
Decompression-facing APIs
|
|
55
|
+
Decompression-facing APIs now use separate size defaults for one-shot and streaming paths:
|
|
56
56
|
|
|
57
|
-
- **
|
|
58
|
-
- **Streaming cumulative cap:**
|
|
59
|
-
- **
|
|
60
|
-
- **
|
|
57
|
+
- **One-shot default output cap:** `512MB`
|
|
58
|
+
- **Streaming cumulative cap:** `2GB` across the lifetime of an `Inflater`/`Reader`
|
|
59
|
+
- **Global configuration:** `MultiCompress.configure`
|
|
60
|
+
- **Per-call override:** `max_output_size:`
|
|
61
61
|
- **Dictionary file size cap:** `32MB` for `MultiCompress::Dictionary.load`
|
|
62
62
|
|
|
63
63
|
Examples:
|
|
64
64
|
|
|
65
65
|
```ruby
|
|
66
|
+
MultiCompress.configure do |config|
|
|
67
|
+
config.max_output_size = 512 * 1024 * 1024
|
|
68
|
+
config.streaming_max_output_size = 2 * 1024 * 1024 * 1024
|
|
69
|
+
end
|
|
70
|
+
|
|
66
71
|
MultiCompress.decompress(blob, algo: :zstd, max_output_size: 64 * 1024 * 1024)
|
|
67
|
-
MultiCompress.decompress(blob, algo: :brotli, max_ratio: nil)
|
|
68
72
|
|
|
69
|
-
MultiCompress::Reader.open("archive.zst", max_output_size: 128 * 1024 * 1024
|
|
73
|
+
MultiCompress::Reader.open("archive.zst", max_output_size: 128 * 1024 * 1024) do |reader|
|
|
70
74
|
puts reader.read
|
|
71
75
|
end
|
|
72
76
|
```
|
|
73
77
|
|
|
74
|
-
`max_output_size
|
|
78
|
+
If `max_output_size:` is omitted, one-shot calls use `MultiCompress.config.max_output_size` and streaming calls use `MultiCompress.config.streaming_max_output_size`.
|
|
75
79
|
|
|
76
80
|
## Algorithm Comparison
|
|
77
81
|
|
|
@@ -164,14 +168,6 @@ Or use the build script:
|
|
|
164
168
|
- Ruby >= 3.1.0
|
|
165
169
|
- C compiler (gcc, clang)
|
|
166
170
|
|
|
167
|
-
## Contributing
|
|
168
|
-
|
|
169
|
-
1. Fork it
|
|
170
|
-
2. Create your feature branch (`git checkout -b feature/my-feature`)
|
|
171
|
-
3. Commit your changes (`git commit -am 'Add feature'`)
|
|
172
|
-
4. Push to the branch (`git push origin feature/my-feature`)
|
|
173
|
-
5. Create a Pull Request
|
|
174
|
-
|
|
175
171
|
## License
|
|
176
172
|
|
|
177
173
|
MIT — see [LICENSE.txt](LICENSE.txt).
|
|
@@ -56,7 +56,9 @@ def configure_system_libraries
|
|
|
56
56
|
|
|
57
57
|
have_header("zdict.h")
|
|
58
58
|
have_header("lz4hc.h")
|
|
59
|
+
have_header("lz4frame.h")
|
|
59
60
|
have_library("lz4", "LZ4_compress_HC")
|
|
61
|
+
have_library("lz4", "LZ4F_compressFrame")
|
|
60
62
|
have_header("brotli/decode.h")
|
|
61
63
|
have_library("brotlidec", "BrotliDecoderCreateInstance")
|
|
62
64
|
end
|