omnizip 0.3.35 → 0.3.37
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +67 -0
- data/docs/getting-started/basic-usage.adoc +14 -22
- data/docs/guides/advanced-features/encryption.adoc +16 -39
- data/docs/guides/advanced-features/index.adoc +16 -6
- data/docs/guides/advanced-features/parallel-processing.adoc +63 -165
- data/docs/guides/advanced-features/progress-tracking.adoc +65 -261
- data/docs/guides/advanced-features/streaming.adoc +43 -113
- data/docs/guides/compression-algorithms/bzip2.adoc +2 -1
- data/docs/guides/compression-algorithms/deflate.adoc +1 -7
- data/docs/guides/compression-algorithms/lzma.adoc +10 -14
- data/docs/guides/compression-algorithms/lzma2.adoc +34 -117
- data/docs/guides/compression-algorithms/ppmd.adoc +24 -27
- data/docs/guides/compression-algorithms/zstandard.adoc +85 -150
- data/docs/troubleshooting/index.adoc +7 -5
- data/lib/omnizip/algorithms/zstandard.rb +2 -3
- data/lib/omnizip/archive/builder.rb +48 -0
- data/lib/omnizip/archive/reader_session.rb +67 -0
- data/lib/omnizip/archive.rb +86 -0
- data/lib/omnizip/archive_handler.rb +1 -0
- data/lib/omnizip/archive_handlers/rar_handler.rb +50 -0
- data/lib/omnizip/archive_handlers/seven_zip_handler.rb +6 -2
- data/lib/omnizip/archive_handlers/tar_handler.rb +4 -0
- data/lib/omnizip/archive_handlers/zip_handler.rb +25 -2
- data/lib/omnizip/archive_handlers.rb +1 -0
- data/lib/omnizip/convenience.rb +91 -11
- data/lib/omnizip/formats/rar/block_parser.rb +8 -12
- data/lib/omnizip/formats/rar/constants.rb +3 -1
- data/lib/omnizip/formats/rar/header.rb +16 -6
- data/lib/omnizip/formats/rar/reader.rb +6 -2
- data/lib/omnizip/version.rb +1 -1
- data/lib/omnizip.rb +1 -0
- metadata +6 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: dbb80e9d06123c538d5823fd0263b38c8a1a386ce0ba107f1aefd6320b2f3618
|
|
4
|
+
data.tar.gz: 53a147f76a0290e7b8dc211c2040cb090799f196fbc57c24cd4ff44290ef53f1
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 8f1fae0da5256104bc0ce6871ebf6f85ec4e60865d513004a55c30389eca1dd3ba47bc3d94c6c0f419717930933920428eb3755e2a3f237014094012638953b7
|
|
7
|
+
data.tar.gz: 27eb53ffd94c880bfa429202cd8b9bdc9094e83ccc382a96479a6dd0c8f090997081f41d5b3463c6f63bb7e9d06fff0dc7b44c45425c0bdd85cc503998559763
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,73 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
### Added
|
|
11
|
+
- `Omnizip::Archive` — the block-style facade the guides have
|
|
12
|
+
documented since the docs were written (it never existed; ~50
|
|
13
|
+
examples across 17 files were dead). `Archive.create(path,
|
|
14
|
+
format:, **options)` yields a Builder with `add_file(path,
|
|
15
|
+
archive_path:)`, `add_directory(dir)` (tree stored under its own
|
|
16
|
+
name), and `add_data(name, content)`; `Archive.open(path,
|
|
17
|
+
password:)` yields (or returns) a session with `entries`
|
|
18
|
+
(name/size/directory?/mtime), `each_entry`, `read`, `extract`,
|
|
19
|
+
`extract_all`, `extract_matching`. 7z options (`level:`,
|
|
20
|
+
`password:`, `encrypt_headers:`, `algorithm:`, `filters:`,
|
|
21
|
+
`solid:`, `dict_size:`) pass through to the writer;
|
|
22
|
+
`ENV['OMNIZIP_PASSWORD']` backs `open`. Passwords are refused
|
|
23
|
+
loudly for zip/tar (no encryption support) instead of silently
|
|
24
|
+
producing unprotected archives.
|
|
25
|
+
- `.zst` routing in the convenience API: `compress_file` to `.zst`
|
|
26
|
+
previously wrote a mislabeled ZIP; it now writes a real Zstandard
|
|
27
|
+
frame (validated against the system zstd CLI in both directions),
|
|
28
|
+
and `decompress_file` reads them.
|
|
29
|
+
- Read-only RAR routing: `extract_archive`/`list_archive`/
|
|
30
|
+
`read_from_archive`/`Archive.open` operate on `.rar` files through
|
|
31
|
+
a new read-only `ArchiveHandlers::RarHandler`; creation keeps
|
|
32
|
+
raising truthfully. `resolve_archive_format` is now
|
|
33
|
+
operation-aware (`writing:`), so read operations no longer fail
|
|
34
|
+
with "cannot be written" errors.
|
|
35
|
+
- `compress_file`/`compress_directory` forward compression options
|
|
36
|
+
(`algorithm:`, `level:`, ...) to the format writer instead of
|
|
37
|
+
silently dropping them (7z honors them; zip absorbs them —
|
|
38
|
+
per-entry ZIP compression is set through `Zip::OutputStream`).
|
|
39
|
+
|
|
40
|
+
### Fixed
|
|
41
|
+
- RAR5 metadata parsing: the file-header parser ignored the
|
|
42
|
+
optional ExtraAreaSize/DataSize vints, misaligning every field —
|
|
43
|
+
all RAR5 entries listed with size 0 (and one fixture failed to
|
|
44
|
+
parse at all). Field order now matches the RAR 5.0 specification
|
|
45
|
+
(verified against RARLAB fixtures); `compressed_size` is populated
|
|
46
|
+
from DataSize and the directory flag is the correct FileFlags bit
|
|
47
|
+
(0x0001). The main-header parser also reads the ArchiveFlags vint
|
|
48
|
+
it was skipping, so volume/solid detection works. Listing now
|
|
49
|
+
prefers the native parser (full metadata) over the external
|
|
50
|
+
`unrar vb` listing (names only, sizes hardcoded to 0).
|
|
51
|
+
- `compress_directory` to a single-file stream extension (`.gz`,
|
|
52
|
+
`.lzma`, ...) silently wrote a ZIP under the foreign name; it now
|
|
53
|
+
raises `UnsupportedFormatError` explaining the mismatch.
|
|
54
|
+
|
|
55
|
+
### Changed
|
|
56
|
+
- Docs: every example across the guides now runs verbatim against
|
|
57
|
+
the real API. Fictional APIs removed or replaced
|
|
58
|
+
(`Omnizip::OutputStream`/`InputStream`, `compress_files`,
|
|
59
|
+
`train_dictionary`, `Archive#compression=` setters, `tar_bzip2`,
|
|
60
|
+
ZIP password attributes, `CorruptArchiveError`,
|
|
61
|
+
`UnsupportedEncryptionError`); fabricated parallel-processing and
|
|
62
|
+
benchmark sections rewritten as truthful process-level
|
|
63
|
+
parallelism; the zstandard guide now centers on the real
|
|
64
|
+
`.zst`/ZIP-entry/dictionary APIs (and no longer claims a native
|
|
65
|
+
zstd binding).
|
|
66
|
+
|
|
67
|
+
## [0.3.36] - 2026-08-28
|
|
68
|
+
|
|
69
|
+
### Fixed
|
|
70
|
+
- `add_to_archive`/`remove_from_archive` errors name the resolved
|
|
71
|
+
format (`Format :seven_zip does not support adding entries`)
|
|
72
|
+
instead of always blaming `:zip`, the default parameter the caller
|
|
73
|
+
never passed.
|
|
74
|
+
|
|
75
|
+
## [0.3.35] - 2026-08-28
|
|
76
|
+
|
|
10
77
|
### Fixed
|
|
11
78
|
- `Omnizip.list_archive` on `.7z` returned raw Reader objects (the
|
|
12
79
|
SevenZip.open facade discards block return values) and
|
|
@@ -53,12 +53,8 @@ Omnizip.compress_directory('my_folder/', 'backup.zip')
|
|
|
53
53
|
# Create 7z archive
|
|
54
54
|
Omnizip.compress_directory('my_folder/', 'backup.7z', format: :seven_zip)
|
|
55
55
|
|
|
56
|
-
# Add individual files
|
|
57
|
-
Omnizip
|
|
58
|
-
archive.add_file('file1.txt')
|
|
59
|
-
archive.add_file('file2.txt')
|
|
60
|
-
archive.add_directory('folder/')
|
|
61
|
-
end
|
|
56
|
+
# Add individual files: compress one file per call
|
|
57
|
+
Omnizip.compress_file('file1.txt', 'custom.zip')
|
|
62
58
|
----
|
|
63
59
|
|
|
64
60
|
=== Extracting Archives
|
|
@@ -70,10 +66,9 @@ require 'omnizip'
|
|
|
70
66
|
# Extract entire archive
|
|
71
67
|
Omnizip.extract_archive('backup.zip', 'output/')
|
|
72
68
|
|
|
73
|
-
# Extract
|
|
74
|
-
Omnizip
|
|
75
|
-
|
|
76
|
-
end
|
|
69
|
+
# Extract a single entry without a full extraction
|
|
70
|
+
content = Omnizip.read_from_archive('backup.zip', 'notes.txt')
|
|
71
|
+
File.write('output/notes.txt', content)
|
|
77
72
|
----
|
|
78
73
|
|
|
79
74
|
=== Reading Archive Contents
|
|
@@ -82,10 +77,10 @@ end
|
|
|
82
77
|
----
|
|
83
78
|
require 'omnizip'
|
|
84
79
|
|
|
85
|
-
# List all entries
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
puts "#{entry
|
|
80
|
+
# List all entries (names); pass details: true for sizes
|
|
81
|
+
Omnizip.list_archive('backup.zip').each { |name| puts name }
|
|
82
|
+
Omnizip.list_archive('backup.zip', details: true).each do |entry|
|
|
83
|
+
puts "#{entry[:name]}: #{entry[:size]} bytes"
|
|
89
84
|
end
|
|
90
85
|
|
|
91
86
|
# Read specific file without extraction
|
|
@@ -187,13 +182,12 @@ end
|
|
|
187
182
|
----
|
|
188
183
|
require 'omnizip'
|
|
189
184
|
|
|
190
|
-
# Verify archive integrity
|
|
185
|
+
# Verify archive integrity: reading the archive's central
|
|
186
|
+
# directory raises FormatError when the archive is damaged.
|
|
191
187
|
begin
|
|
192
|
-
Omnizip
|
|
193
|
-
archive.verify
|
|
194
|
-
end
|
|
188
|
+
Omnizip.list_archive('archive.zip')
|
|
195
189
|
puts "Archive is valid"
|
|
196
|
-
rescue Omnizip::
|
|
190
|
+
rescue Omnizip::FormatError => e
|
|
197
191
|
puts "Archive is corrupted: #{e.message}"
|
|
198
192
|
end
|
|
199
193
|
----
|
|
@@ -253,9 +247,7 @@ require 'tempfile'
|
|
|
253
247
|
|
|
254
248
|
# Create temporary archive
|
|
255
249
|
Tempfile.create(['backup', '.zip']) do |tmpfile|
|
|
256
|
-
Omnizip
|
|
257
|
-
archive.add_directory('data/')
|
|
258
|
-
end
|
|
250
|
+
Omnizip.compress_directory('data', tmpfile.path)
|
|
259
251
|
|
|
260
252
|
# Use archive...
|
|
261
253
|
puts "Temporary archive: #{tmpfile.path}"
|
|
@@ -12,7 +12,7 @@ Protect sensitive data in your archives with industry-standard AES-256 encryptio
|
|
|
12
12
|
== Benefits
|
|
13
13
|
|
|
14
14
|
* **Strong Security**: AES-256 encryption with secure key derivation
|
|
15
|
-
* **Format
|
|
15
|
+
* **Format**: Encryption available for 7z archives (ZIP encryption is unsupported)
|
|
16
16
|
* **Header Protection**: Encrypt filenames and metadata (7z format)
|
|
17
17
|
* **Password Validation**: Built-in password strength checking
|
|
18
18
|
* **Compatibility**: Works with archives from other tools
|
|
@@ -44,40 +44,20 @@ end
|
|
|
44
44
|
* Optional header encryption (hides filenames)
|
|
45
45
|
* HMAC-SHA-256 authentication
|
|
46
46
|
|
|
47
|
-
===
|
|
47
|
+
=== ZIP Encryption: not supported
|
|
48
48
|
|
|
49
|
-
|
|
49
|
+
Omnizip can neither create nor decrypt encrypted ZIP archives —
|
|
50
|
+
neither WinZip AES nor traditional ZipCrypto. Encrypted 7z is the
|
|
51
|
+
supported path; extract encrypted ZIPs with 7-Zip/unzip.
|
|
50
52
|
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
# Create AES-encrypted ZIP
|
|
54
|
-
Omnizip::Zip::File.open('secure.zip', create: true) do |zip|
|
|
55
|
-
zip.password = 'SecurePassword123'
|
|
56
|
-
zip.encryption = :winzip_aes # AES-256
|
|
57
|
-
zip.add('private.doc', 'files/private.doc')
|
|
58
|
-
end
|
|
59
|
-
----
|
|
60
|
-
|
|
61
|
-
**Features**:
|
|
62
|
-
|
|
63
|
-
* AES-256, AES-192, or AES-128
|
|
64
|
-
* Compatible with WinZip, 7-Zip, PeaZip
|
|
65
|
-
* HMAC-SHA-1 authentication
|
|
66
|
-
* Standard ZIP format
|
|
67
|
-
|
|
68
|
-
=== Traditional ZIP Encryption
|
|
69
|
-
|
|
70
|
-
[WARNING]
|
|
71
|
-
Traditional ZIP encryption (ZipCrypto) is weak and should only be used for compatibility with legacy systems.
|
|
53
|
+
The convenience API raises rather than silently ignoring a
|
|
54
|
+
password for zip:
|
|
72
55
|
|
|
73
56
|
[source,ruby]
|
|
74
57
|
----
|
|
75
|
-
|
|
76
|
-
Omnizip::
|
|
77
|
-
|
|
78
|
-
zip.encryption = :traditional # Weak encryption
|
|
79
|
-
zip.add('file.txt', 'data.txt')
|
|
80
|
-
end
|
|
58
|
+
Omnizip::Archive.create('secure.zip', password: 'x')
|
|
59
|
+
# => Omnizip::UnsupportedFormatError:
|
|
60
|
+
# password is only supported for seven_zip; :zip has no encryption support
|
|
81
61
|
----
|
|
82
62
|
|
|
83
63
|
== Decrypting Archives
|
|
@@ -91,10 +71,8 @@ Omnizip::Archive.open('secure.7z', password: 'MyStr0ngP@ssw0rd') do |archive|
|
|
|
91
71
|
archive.extract_all('decrypted/')
|
|
92
72
|
end
|
|
93
73
|
|
|
94
|
-
#
|
|
95
|
-
|
|
96
|
-
zip.extract_all('output/')
|
|
97
|
-
end
|
|
74
|
+
# ZIP encryption: Omnizip can neither create nor decrypt
|
|
75
|
+
# password-protected ZIP archives; extract them with 7-Zip/unzip
|
|
98
76
|
----
|
|
99
77
|
|
|
100
78
|
=== Password via Environment Variable
|
|
@@ -286,12 +264,11 @@ Some older archives may use unsupported encryption:
|
|
|
286
264
|
[source,ruby]
|
|
287
265
|
----
|
|
288
266
|
begin
|
|
289
|
-
Omnizip::Archive.open('old-archive.
|
|
290
|
-
|
|
267
|
+
Omnizip::Archive.open('old-archive.7z', password: 'wrong') do |archive|
|
|
268
|
+
archive.read('secret.txt')
|
|
291
269
|
end
|
|
292
|
-
rescue Omnizip::
|
|
293
|
-
puts "
|
|
294
|
-
puts "Try opening with 7-Zip or WinZip first"
|
|
270
|
+
rescue Omnizip::FormatError => e
|
|
271
|
+
puts "Archive damaged or password wrong: #{e.message}"
|
|
295
272
|
end
|
|
296
273
|
----
|
|
297
274
|
|
|
@@ -34,12 +34,22 @@ These features can be combined for maximum efficiency:
|
|
|
34
34
|
|
|
35
35
|
[source,ruby]
|
|
36
36
|
----
|
|
37
|
-
#
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
37
|
+
# Streaming writes + a progress tracker
|
|
38
|
+
require 'omnizip/zip/output_stream'
|
|
39
|
+
|
|
40
|
+
files = Dir.glob('large_data/**/*').select { |f| File.file?(f) }
|
|
41
|
+
tracker = Omnizip::Progress.track(
|
|
42
|
+
total_files: files.size,
|
|
43
|
+
total_bytes: files.sum { |f| File.size(f) },
|
|
44
|
+
reporter: :console
|
|
45
|
+
)
|
|
46
|
+
|
|
47
|
+
Omnizip::Zip::OutputStream.open('backup.zip') do |stream|
|
|
48
|
+
files.each do |file|
|
|
49
|
+
stream.put_next_entry(file.delete_prefix('large_data/'))
|
|
50
|
+
stream.write(File.binread(file))
|
|
51
|
+
tracker.update(files: 1, bytes: File.size(file))
|
|
52
|
+
end
|
|
43
53
|
end
|
|
44
54
|
----
|
|
45
55
|
|
|
@@ -7,176 +7,94 @@ grand_parent: Guides
|
|
|
7
7
|
|
|
8
8
|
== Purpose
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
Omnizip's codecs are pure Ruby and single-threaded: a single archive
|
|
11
|
+
is always compressed or extracted by one CPU core. To use multiple
|
|
12
|
+
cores, run independent archive operations in parallel — across
|
|
13
|
+
processes for CPU-bound compression, or across threads for
|
|
14
|
+
I/O-bound extraction of separate archives.
|
|
11
15
|
|
|
12
|
-
==
|
|
13
|
-
|
|
14
|
-
* **2-3x Faster**: Achieve 2-3x speedup on multi-core systems
|
|
15
|
-
* **Better Resource Utilization**: Use all available CPU cores effectively
|
|
16
|
-
* **Reduced Wait Time**: Process large batches of files quickly
|
|
17
|
-
* **Scalable**: Performance improves with more CPU cores
|
|
18
|
-
|
|
19
|
-
== When to Use Parallel Processing
|
|
16
|
+
== When It Helps
|
|
20
17
|
|
|
21
18
|
[cols="1,3"]
|
|
22
19
|
|===
|
|
23
|
-
|Scenario |
|
|
24
|
-
|
|
25
|
-
|Large Archives (>100 files)
|
|
26
|
-
|Distribute file processing across cores
|
|
20
|
+
|Scenario |Approach
|
|
27
21
|
|
|
28
|
-
|
|
|
29
|
-
|
|
|
22
|
+
|Many archives to compress
|
|
23
|
+
|One process per archive
|
|
30
24
|
|
|
31
|
-
|
|
|
32
|
-
|
|
|
25
|
+
|Many archives to extract
|
|
26
|
+
|One thread per archive (I/O releases the GVL)
|
|
33
27
|
|
|
34
|
-
|
|
|
35
|
-
|
|
|
28
|
+
|One large archive
|
|
29
|
+
|No parallelism available; pick an appropriate +level:+
|
|
36
30
|
|===
|
|
37
31
|
|
|
38
|
-
== Parallel Compression
|
|
32
|
+
== Parallel Compression (Processes)
|
|
39
33
|
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
Automatically distribute files across available cores:
|
|
34
|
+
Compression is CPU-bound, so use processes — MRI's GVL prevents
|
|
35
|
+
in-thread parallelism:
|
|
43
36
|
|
|
44
37
|
[source,ruby]
|
|
45
38
|
----
|
|
46
|
-
# Compress
|
|
47
|
-
|
|
48
|
-
'large_project/',
|
|
49
|
-
'backup.zip',
|
|
50
|
-
parallel: true,
|
|
51
|
-
threads: 4 # Or omit to use all cores
|
|
52
|
-
)
|
|
53
|
-
----
|
|
54
|
-
|
|
55
|
-
=== Custom Thread Count
|
|
39
|
+
# Compress each directory in its own process
|
|
40
|
+
dirs = %w[project_a project_b project_c]
|
|
56
41
|
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
# Use specific number of threads
|
|
62
|
-
Omnizip::Archive.create('archive.7z', format: :seven_zip) do |archive|
|
|
63
|
-
archive.add_directory(
|
|
64
|
-
'data/',
|
|
65
|
-
parallel: true,
|
|
66
|
-
threads: 8, # Use 8 worker threads
|
|
67
|
-
chunk_size: 1.megabyte # Process 1MB chunks
|
|
68
|
-
)
|
|
42
|
+
pids = dirs.map do |dir|
|
|
43
|
+
Process.fork do
|
|
44
|
+
Omnizip.compress_directory(dir, "#{dir}.7z", format: :seven_zip)
|
|
45
|
+
end
|
|
69
46
|
end
|
|
47
|
+
pids.each { |pid| Process.wait(pid) }
|
|
70
48
|
----
|
|
71
49
|
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
=== Extract Multiple Files Simultaneously
|
|
75
|
-
|
|
76
|
-
Extract files in parallel for faster extraction:
|
|
77
|
-
|
|
78
|
-
[source,ruby]
|
|
79
|
-
----
|
|
80
|
-
# Extract with parallel processing
|
|
81
|
-
Omnizip.extract_archive(
|
|
82
|
-
'large-archive.zip',
|
|
83
|
-
'output/',
|
|
84
|
-
parallel: true,
|
|
85
|
-
threads: 4
|
|
86
|
-
)
|
|
87
|
-
----
|
|
88
|
-
|
|
89
|
-
=== With Progress Tracking
|
|
90
|
-
|
|
91
|
-
Combine parallel processing with progress monitoring:
|
|
50
|
+
Or drive the CLI from your shell or job scheduler:
|
|
92
51
|
|
|
93
|
-
[source,
|
|
52
|
+
[source,bash]
|
|
94
53
|
----
|
|
95
|
-
|
|
96
|
-
puts "Extracted #{progress.files_processed}/#{progress.total_files} files"
|
|
97
|
-
puts "Using #{progress.active_threads} parallel threads"
|
|
98
|
-
puts "Speed: #{progress.throughput_mb_s} MB/s"
|
|
99
|
-
end
|
|
54
|
+
ls -d project_*/ | xargs -P 8 -I{} omnizip archive create {}.7z {}
|
|
100
55
|
----
|
|
101
56
|
|
|
102
|
-
==
|
|
103
|
-
|
|
104
|
-
[cols="2,1,1,1,1"]
|
|
105
|
-
|===
|
|
106
|
-
|Workload |1 Thread |2 Threads |4 Threads |8 Threads
|
|
107
|
-
|
|
108
|
-
|100 small files
|
|
109
|
-
|10s
|
|
110
|
-
|6s
|
|
111
|
-
|4s
|
|
112
|
-
|3s
|
|
113
|
-
|
|
114
|
-
|10 large files
|
|
115
|
-
|60s
|
|
116
|
-
|35s
|
|
117
|
-
|20s
|
|
118
|
-
|15s
|
|
119
|
-
|
|
120
|
-
|Mixed workload
|
|
121
|
-
|30s
|
|
122
|
-
|18s
|
|
123
|
-
|12s
|
|
124
|
-
|10s
|
|
125
|
-
|===
|
|
126
|
-
|
|
127
|
-
*Note: Performance varies based on CPU, disk speed, and file characteristics.*
|
|
57
|
+
== Parallel Extraction (Threads)
|
|
128
58
|
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
=== Thread Count
|
|
132
|
-
|
|
133
|
-
Control parallelism level:
|
|
59
|
+
Extracting separate archives is I/O-bound, so threads work:
|
|
134
60
|
|
|
135
61
|
[source,ruby]
|
|
136
62
|
----
|
|
137
|
-
|
|
138
|
-
parallel: true
|
|
139
|
-
|
|
140
|
-
# Specific thread count
|
|
141
|
-
parallel: true, threads: 4
|
|
142
|
-
|
|
143
|
-
# Match CPU cores
|
|
144
|
-
parallel: true, threads: Etc.nprocessors
|
|
145
|
-
----
|
|
146
|
-
|
|
147
|
-
=== Chunk Size
|
|
63
|
+
archives = Dir.glob('downloads/*.zip')
|
|
148
64
|
|
|
149
|
-
|
|
65
|
+
threads = archives.map do |archive|
|
|
66
|
+
Thread.new do
|
|
67
|
+
Omnizip.extract_archive(
|
|
68
|
+
archive,
|
|
69
|
+
"extracted/#{File.basename(archive, '.zip')}/"
|
|
70
|
+
)
|
|
71
|
+
end
|
|
72
|
+
end
|
|
150
73
|
|
|
151
|
-
|
|
74
|
+
threads.each(&:join)
|
|
152
75
|
----
|
|
153
|
-
# Smaller chunks = better distribution, more overhead
|
|
154
|
-
chunk_size: 512.kilobytes
|
|
155
|
-
|
|
156
|
-
# Larger chunks = less overhead, less parallel benefit
|
|
157
|
-
chunk_size: 10.megabytes
|
|
158
76
|
|
|
159
|
-
|
|
160
|
-
chunk_size: 1.megabyte
|
|
161
|
-
----
|
|
77
|
+
Note: a single archive's extraction still uses one thread.
|
|
162
78
|
|
|
163
79
|
== Use Cases
|
|
164
80
|
|
|
165
81
|
=== Server Deployments
|
|
166
82
|
|
|
167
|
-
Maximize server hardware utilization:
|
|
168
|
-
|
|
169
83
|
[source,ruby]
|
|
170
84
|
----
|
|
171
|
-
#
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
85
|
+
# One process per site on a multi-core backup server
|
|
86
|
+
sites = Dir.children('/var/www')
|
|
87
|
+
|
|
88
|
+
pids = sites.map do |site|
|
|
89
|
+
Process.fork do
|
|
90
|
+
Omnizip.compress_directory(
|
|
91
|
+
File.join('/var/www', site),
|
|
92
|
+
"/backups/#{site}.7z",
|
|
93
|
+
format: :seven_zip, algorithm: :lzma2, level: 9
|
|
94
|
+
)
|
|
95
|
+
end
|
|
96
|
+
end
|
|
97
|
+
pids.each { |pid| Process.wait(pid) }
|
|
180
98
|
----
|
|
181
99
|
|
|
182
100
|
=== Batch Processing
|
|
@@ -204,43 +122,23 @@ end
|
|
|
204
122
|
|
|
205
123
|
== Best Practices
|
|
206
124
|
|
|
207
|
-
. **
|
|
208
|
-
. **Consider
|
|
209
|
-
. **Monitor
|
|
210
|
-
. **Test
|
|
211
|
-
. **Combine with Streaming**: Use both for maximum efficiency
|
|
125
|
+
. **One process per archive** for compression; threads only help I/O
|
|
126
|
+
. **Consider disk I/O**: more parallelism may not help on slow disks
|
|
127
|
+
. **Monitor memory**: each process/thread adds overhead
|
|
128
|
+
. **Test your workload**: measure the real speedup
|
|
212
129
|
|
|
213
130
|
== Limitations
|
|
214
131
|
|
|
215
|
-
*
|
|
216
|
-
*
|
|
217
|
-
*
|
|
218
|
-
* Very small files have overhead from thread management
|
|
132
|
+
* No intra-archive parallelism: a single archive uses one core
|
|
133
|
+
* MRI's GVL blocks in-thread CPU parallelism — use processes
|
|
134
|
+
* Memory usage grows with process/thread count
|
|
219
135
|
|
|
220
136
|
== Platform Considerations
|
|
221
137
|
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|Platform |Notes
|
|
225
|
-
|
|
226
|
-
|MRI Ruby
|
|
227
|
-
|Limited by GIL, 2-3x speedup typical
|
|
228
|
-
|
|
229
|
-
|JRuby
|
|
230
|
-
|No GIL, better parallel scalability
|
|
231
|
-
|
|
232
|
-
|TruffleRuby
|
|
233
|
-
|Excellent parallel performance
|
|
234
|
-
|
|
235
|
-
|Windows
|
|
236
|
-
|Similar performance to Unix systems
|
|
237
|
-
|
|
238
|
-
|Linux/macOS
|
|
239
|
-
|Full parallel processing support
|
|
240
|
-
|===
|
|
138
|
+
`Process.fork` is unavailable on Windows — use `Process.spawn` with
|
|
139
|
+
the +omnizip+ CLI there, or a job scheduler.
|
|
241
140
|
|
|
242
141
|
== See Also
|
|
243
142
|
|
|
244
|
-
* link:streaming.html[Streaming] -
|
|
245
|
-
* link
|
|
246
|
-
* link:../../compatibility.html[Compatibility] - Format support for parallel processing
|
|
143
|
+
* link:streaming.html[Streaming] - Memory efficiency
|
|
144
|
+
* link:../compression-algorithms/lzma2.html[LZMA2] - Default 7z algorithm
|