omnizip 0.3.36 → 0.3.38
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +135 -9
- data/docs/getting-started/basic-usage.adoc +14 -22
- data/docs/guides/advanced-features/encryption.adoc +16 -39
- data/docs/guides/advanced-features/index.adoc +16 -6
- data/docs/guides/advanced-features/parallel-processing.adoc +63 -165
- data/docs/guides/advanced-features/progress-tracking.adoc +65 -261
- data/docs/guides/advanced-features/streaming.adoc +43 -113
- data/docs/guides/archive-formats/gzip-format.adoc +2 -4
- data/docs/guides/archive-formats/ole-format.adoc +54 -219
- data/docs/guides/archive-formats/rar5.adoc +15 -15
- data/docs/guides/archive-formats/rpm-format.adoc +48 -189
- data/docs/guides/compression-algorithms/bzip2.adoc +2 -1
- data/docs/guides/compression-algorithms/deflate.adoc +1 -7
- data/docs/guides/compression-algorithms/lzma.adoc +10 -14
- data/docs/guides/compression-algorithms/lzma2.adoc +34 -117
- data/docs/guides/compression-algorithms/ppmd.adoc +24 -27
- data/docs/guides/compression-algorithms/zstandard.adoc +85 -150
- data/docs/reference/api/overview.adoc +16 -17
- data/docs/troubleshooting/index.adoc +8 -9
- data/lib/omnizip/algorithms/zstandard.rb +2 -3
- data/lib/omnizip/archive/builder.rb +48 -0
- data/lib/omnizip/archive/reader_session.rb +67 -0
- data/lib/omnizip/archive.rb +86 -0
- data/lib/omnizip/archive_handler.rb +3 -0
- data/lib/omnizip/archive_handlers/cpio_handler.rb +47 -0
- data/lib/omnizip/archive_handlers/iso_handler.rb +54 -0
- data/lib/omnizip/archive_handlers/rar_handler.rb +50 -0
- data/lib/omnizip/archive_handlers/seven_zip_handler.rb +6 -2
- data/lib/omnizip/archive_handlers/tar_handler.rb +4 -0
- data/lib/omnizip/archive_handlers/zip_handler.rb +25 -2
- data/lib/omnizip/archive_handlers.rb +3 -0
- data/lib/omnizip/cli.rb +25 -0
- data/lib/omnizip/convenience.rb +91 -11
- data/lib/omnizip/formats/iso/directory_builder.rb +24 -0
- data/lib/omnizip/formats/iso/reader.rb +2 -2
- data/lib/omnizip/formats/iso/writer.rb +20 -4
- data/lib/omnizip/formats/iso.rb +3 -1
- data/lib/omnizip/formats/rar/block_parser.rb +8 -12
- data/lib/omnizip/formats/rar/constants.rb +3 -1
- data/lib/omnizip/formats/rar/header.rb +16 -6
- data/lib/omnizip/formats/rar/rar5/compression/lzss.rb +5 -3
- data/lib/omnizip/formats/rar/rar5/header.rb +50 -33
- data/lib/omnizip/formats/rar/rar5/multi_volume/volume_writer.rb +17 -34
- data/lib/omnizip/formats/rar/rar5/vint.rb +25 -37
- data/lib/omnizip/formats/rar/rar5/writer.rb +50 -14
- data/lib/omnizip/formats/rar/reader.rb +6 -2
- data/lib/omnizip/formats/rar.rb +5 -3
- data/lib/omnizip/formats/seven_zip/parser.rb +22 -7
- data/lib/omnizip/formats/seven_zip/writer.rb +78 -19
- data/lib/omnizip/version.rb +1 -1
- data/lib/omnizip.rb +1 -0
- metadata +7 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 67142de350c2f3dd29c473268977799ed088cedb0340932cff6db7bc23ac009e
|
|
4
|
+
data.tar.gz: c35fbc3353ca0c20f2ec4f86fa39fff67e6ed95822702a18c9b545d35406749c
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 7f6b4e88bb0a227f7cff47b06e979891e6d5a57288ae141bd15e57a07abe2cef3c915600242bb05406c4b7a13ef9fdd0ab5f11870e41b65f0a95077191741ceb
|
|
7
|
+
data.tar.gz: 3e8497c8eb222b4a139f965ec84f1672d8b12ae92e55986df95d8057c452bcace18a75c54a0464ca696776a33256d2e11ea53811c7d73e3ecb1c0eece06d04ff
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,136 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
### Added
|
|
11
|
+
- `.cpio` and `.iso` read routing: `extract_archive`/`list_archive`/
|
|
12
|
+
`read_from_archive`/`Archive.open` operate on CPIO archives and ISO
|
|
13
|
+
9660 images through new read-only handlers, mirroring the RAR
|
|
14
|
+
routing (creation keeps raising truthfully).
|
|
15
|
+
|
|
16
|
+
### Fixed
|
|
17
|
+
- 7z archives with explicit directory entries (or zero-byte
|
|
18
|
+
entries) were structurally invalid: the writer never emitted the
|
|
19
|
+
kEmptyStream/kEmptyFile properties, hardcoded 0x20 attributes
|
|
20
|
+
over each entry's real attributes, and its kSubStreamsInfo
|
|
21
|
+
digest/size arrays counted directory entries as substreams. 7-Zip
|
|
22
|
+
rejects such archives with "Headers Error" (and our own reader
|
|
23
|
+
crashed extracting directory entries). Verified against 7zz
|
|
24
|
+
ground truth (`-mhc=off` dumps): kEmptyStream/kEmptyFile carry
|
|
25
|
+
RAW bit vectors with no all-defined marker — the parser expected
|
|
26
|
+
that marker everywhere, so it now reads both layouts correctly
|
|
27
|
+
(which also fixes reading 7-Zip-created archives containing empty
|
|
28
|
+
files).
|
|
29
|
+
- RAR5 vint encoding was not the spec encoding: multi-byte values
|
|
30
|
+
(any size >= 128, extra-area sizes, dictionary sizes) were written
|
|
31
|
+
in a byte-swapped form no reader decodes — archives with more
|
|
32
|
+
than ~127 bytes of content were corrupt for unrar AND for our own
|
|
33
|
+
spec-conformant parser. All round-trips now verified against
|
|
34
|
+
unrar, including encrypted STORE, multi-file, and multi-volume
|
|
35
|
+
archives (volume flags were the previously documented "no
|
|
36
|
+
write-side reference" gap: Main archive flags 0x0001/0x0002 and
|
|
37
|
+
end-of-archive 0x0001 are written per the RAR 5.0 specification).
|
|
38
|
+
- RAR5 file headers: a "mystery vint" not in the spec shifted every
|
|
39
|
+
field; the compression method was written into the version bits;
|
|
40
|
+
the file-attribute value chmods extracted files unreadable. Now:
|
|
41
|
+
spec field order, method in bits 8-10 with dictionary bits
|
|
42
|
+
11-15, Unix attribute 0o100644.
|
|
43
|
+
- RAR5 encrypted writing now persists the salt/IV as the file
|
|
44
|
+
encryption extra record (spec type 0x01) — archives it wrote
|
|
45
|
+
before could not be decrypted by anything (unrar reported "All
|
|
46
|
+
OK" only because no CRC was stored).
|
|
47
|
+
- RAR5 `:lzma`/`:lzss` compression advertised method bits for a
|
|
48
|
+
stream that nothing can decode (the encoder is not
|
|
49
|
+
official-RAR-compatible; `unrar` extracts empty files silently).
|
|
50
|
+
`Lzss.available?` is now honest, so writers fall back to STORE
|
|
51
|
+
with a warning; solid mode falls back to independent STORE
|
|
52
|
+
entries.
|
|
53
|
+
- ISO writing: directory records pointed every file at sector 0
|
|
54
|
+
(allocated extents were never copied onto the tree nodes); the
|
|
55
|
+
volume descriptor declared 100 sectors regardless of actual
|
|
56
|
+
size; `add_directory` entries were not marked as directories;
|
|
57
|
+
Rock Ridge and Joliet were advertised by default without being
|
|
58
|
+
implemented (the cloned SVD made 7-Zip read ASCII names as
|
|
59
|
+
UCS-2); `Reader#extract_all` looked entries up by bare name
|
|
60
|
+
instead of full path; `Formats::Iso.list` returned the Reader
|
|
61
|
+
instead of entries. Images now extract byte-identically through
|
|
62
|
+
7zz, including nested directories.
|
|
63
|
+
- CLI: `omnizip <command> --help` failed with an arity error on
|
|
64
|
+
every command (Thor does not route `--help` on subcommands by
|
|
65
|
+
itself). A dispatch hook now prints per-command help.
|
|
66
|
+
- `Omnizip::Formats::Rar::Rar5::MainHeader/FileHeader` were
|
|
67
|
+
unreferenceable unless another file happened to load header.rb
|
|
68
|
+
first (missing autoload entries); three Rar5 autoloads pointed at
|
|
69
|
+
files that never existed.
|
|
70
|
+
|
|
71
|
+
### Changed
|
|
72
|
+
- Docs: the OLE, RPM, GZIP, RAR5 and API-overview guides now use
|
|
73
|
+
the real APIs (`Formats::Ole.list/read/info`, `Formats::Rpm.*`,
|
|
74
|
+
`Gzip.compress_stream`, `Formats::Zip::Reader#read`, correct
|
|
75
|
+
error class names); fictional methods (`open_stream`, `root`,
|
|
76
|
+
`extract_to`, `extract_files`, `extract_payload`,
|
|
77
|
+
`Rar::Rar5::Reader`, scriptlets/signature accessors) removed.
|
|
78
|
+
|
|
79
|
+
## [0.3.37] - 2026-08-28
|
|
80
|
+
|
|
81
|
+
### Added
|
|
82
|
+
- `Omnizip::Archive` — the block-style facade the guides have
|
|
83
|
+
documented since the docs were written (it never existed; ~50
|
|
84
|
+
examples across 17 files were dead). `Archive.create(path,
|
|
85
|
+
format:, **options)` yields a Builder with `add_file(path,
|
|
86
|
+
archive_path:)`, `add_directory(dir)` (tree stored under its own
|
|
87
|
+
name), and `add_data(name, content)`; `Archive.open(path,
|
|
88
|
+
password:)` yields (or returns) a session with `entries`
|
|
89
|
+
(name/size/directory?/mtime), `each_entry`, `read`, `extract`,
|
|
90
|
+
`extract_all`, `extract_matching`. 7z options (`level:`,
|
|
91
|
+
`password:`, `encrypt_headers:`, `algorithm:`, `filters:`,
|
|
92
|
+
`solid:`, `dict_size:`) pass through to the writer;
|
|
93
|
+
`ENV['OMNIZIP_PASSWORD']` backs `open`. Passwords are refused
|
|
94
|
+
loudly for zip/tar (no encryption support) instead of silently
|
|
95
|
+
producing unprotected archives.
|
|
96
|
+
- `.zst` routing in the convenience API: `compress_file` to `.zst`
|
|
97
|
+
previously wrote a mislabeled ZIP; it now writes a real Zstandard
|
|
98
|
+
frame (validated against the system zstd CLI in both directions),
|
|
99
|
+
and `decompress_file` reads them.
|
|
100
|
+
- Read-only RAR routing: `extract_archive`/`list_archive`/
|
|
101
|
+
`read_from_archive`/`Archive.open` operate on `.rar` files through
|
|
102
|
+
a new read-only `ArchiveHandlers::RarHandler`; creation keeps
|
|
103
|
+
raising truthfully. `resolve_archive_format` is now
|
|
104
|
+
operation-aware (`writing:`), so read operations no longer fail
|
|
105
|
+
with "cannot be written" errors.
|
|
106
|
+
- `compress_file`/`compress_directory` forward compression options
|
|
107
|
+
(`algorithm:`, `level:`, ...) to the format writer instead of
|
|
108
|
+
silently dropping them (7z honors them; zip absorbs them —
|
|
109
|
+
per-entry ZIP compression is set through `Zip::OutputStream`).
|
|
110
|
+
|
|
111
|
+
### Fixed
|
|
112
|
+
- RAR5 metadata parsing: the file-header parser ignored the
|
|
113
|
+
optional ExtraAreaSize/DataSize vints, misaligning every field —
|
|
114
|
+
all RAR5 entries listed with size 0 (and one fixture failed to
|
|
115
|
+
parse at all). Field order now matches the RAR 5.0 specification
|
|
116
|
+
(verified against RARLAB fixtures); `compressed_size` is populated
|
|
117
|
+
from DataSize and the directory flag is the correct FileFlags bit
|
|
118
|
+
(0x0001). The main-header parser also reads the ArchiveFlags vint
|
|
119
|
+
it was skipping, so volume/solid detection works. Listing now
|
|
120
|
+
prefers the native parser (full metadata) over the external
|
|
121
|
+
`unrar vb` listing (names only, sizes hardcoded to 0).
|
|
122
|
+
- `compress_directory` to a single-file stream extension (`.gz`,
|
|
123
|
+
`.lzma`, ...) silently wrote a ZIP under the foreign name; it now
|
|
124
|
+
raises `UnsupportedFormatError` explaining the mismatch.
|
|
125
|
+
|
|
126
|
+
### Changed
|
|
127
|
+
- Docs: every example across the guides now runs verbatim against
|
|
128
|
+
the real API. Fictional APIs removed or replaced
|
|
129
|
+
(`Omnizip::OutputStream`/`InputStream`, `compress_files`,
|
|
130
|
+
`train_dictionary`, `Archive#compression=` setters, `tar_bzip2`,
|
|
131
|
+
ZIP password attributes, `CorruptArchiveError`,
|
|
132
|
+
`UnsupportedEncryptionError`); fabricated parallel-processing and
|
|
133
|
+
benchmark sections rewritten as truthful process-level
|
|
134
|
+
parallelism; the zstandard guide now centers on the real
|
|
135
|
+
`.zst`/ZIP-entry/dictionary APIs (and no longer claims a native
|
|
136
|
+
zstd binding).
|
|
137
|
+
|
|
138
|
+
## [0.3.36] - 2026-08-28
|
|
139
|
+
|
|
10
140
|
### Fixed
|
|
11
141
|
- `add_to_archive`/`remove_from_archive` errors name the resolved
|
|
12
142
|
format (`Format :seven_zip does not support adding entries`)
|
|
@@ -84,15 +214,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
84
214
|
been pinned to :copy) and replaced the match-encoding TODO with a
|
|
85
215
|
wire-level round-trip verification of the distance-8/length
|
|
86
216
|
pattern through the LZMA2 decoder. No code under lib/ changes.
|
|
87
|
-
|
|
88
|
-
## [0.3.30] - 2026-08-27
|
|
89
|
-
|
|
90
|
-
### Verified
|
|
91
|
-
- LZIP interop with the real `lzip` CLI (1.26): our members pass
|
|
92
|
-
`lzip -t`/`-dc` and the CLI's members decode through
|
|
93
|
-
`Formats::Lzip` (spec skips where the CLI is absent).
|
|
94
|
-
|
|
95
|
-
### Changed
|
|
96
217
|
- The remaining lib/ TODO markers are resolved as documented design
|
|
97
218
|
decisions rather than dangling work: RAR5 volume extras and
|
|
98
219
|
EndHeader volume flags (no write-side reference exists;
|
|
@@ -106,6 +227,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
106
227
|
the salt/IV header, so archives it writes cannot be decrypted;
|
|
107
228
|
encryption stays read-verified only.
|
|
108
229
|
|
|
230
|
+
### Verified
|
|
231
|
+
- LZIP interop with the real `lzip` CLI (1.26): our members pass
|
|
232
|
+
`lzip -t`/`-dc` and the CLI's members decode through
|
|
233
|
+
`Formats::Lzip` (spec skips where the CLI is absent).
|
|
234
|
+
|
|
109
235
|
## [0.3.28] - 2026-08-26
|
|
110
236
|
|
|
111
237
|
### Added
|
|
@@ -53,12 +53,8 @@ Omnizip.compress_directory('my_folder/', 'backup.zip')
|
|
|
53
53
|
# Create 7z archive
|
|
54
54
|
Omnizip.compress_directory('my_folder/', 'backup.7z', format: :seven_zip)
|
|
55
55
|
|
|
56
|
-
# Add individual files
|
|
57
|
-
Omnizip
|
|
58
|
-
archive.add_file('file1.txt')
|
|
59
|
-
archive.add_file('file2.txt')
|
|
60
|
-
archive.add_directory('folder/')
|
|
61
|
-
end
|
|
56
|
+
# Add individual files: compress one file per call
|
|
57
|
+
Omnizip.compress_file('file1.txt', 'custom.zip')
|
|
62
58
|
----
|
|
63
59
|
|
|
64
60
|
=== Extracting Archives
|
|
@@ -70,10 +66,9 @@ require 'omnizip'
|
|
|
70
66
|
# Extract entire archive
|
|
71
67
|
Omnizip.extract_archive('backup.zip', 'output/')
|
|
72
68
|
|
|
73
|
-
# Extract
|
|
74
|
-
Omnizip
|
|
75
|
-
|
|
76
|
-
end
|
|
69
|
+
# Extract a single entry without a full extraction
|
|
70
|
+
content = Omnizip.read_from_archive('backup.zip', 'notes.txt')
|
|
71
|
+
File.write('output/notes.txt', content)
|
|
77
72
|
----
|
|
78
73
|
|
|
79
74
|
=== Reading Archive Contents
|
|
@@ -82,10 +77,10 @@ end
|
|
|
82
77
|
----
|
|
83
78
|
require 'omnizip'
|
|
84
79
|
|
|
85
|
-
# List all entries
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
puts "#{entry
|
|
80
|
+
# List all entries (names); pass details: true for sizes
|
|
81
|
+
Omnizip.list_archive('backup.zip').each { |name| puts name }
|
|
82
|
+
Omnizip.list_archive('backup.zip', details: true).each do |entry|
|
|
83
|
+
puts "#{entry[:name]}: #{entry[:size]} bytes"
|
|
89
84
|
end
|
|
90
85
|
|
|
91
86
|
# Read specific file without extraction
|
|
@@ -187,13 +182,12 @@ end
|
|
|
187
182
|
----
|
|
188
183
|
require 'omnizip'
|
|
189
184
|
|
|
190
|
-
# Verify archive integrity
|
|
185
|
+
# Verify archive integrity: reading the archive's central
|
|
186
|
+
# directory raises FormatError when the archive is damaged.
|
|
191
187
|
begin
|
|
192
|
-
Omnizip
|
|
193
|
-
archive.verify
|
|
194
|
-
end
|
|
188
|
+
Omnizip.list_archive('archive.zip')
|
|
195
189
|
puts "Archive is valid"
|
|
196
|
-
rescue Omnizip::
|
|
190
|
+
rescue Omnizip::FormatError => e
|
|
197
191
|
puts "Archive is corrupted: #{e.message}"
|
|
198
192
|
end
|
|
199
193
|
----
|
|
@@ -253,9 +247,7 @@ require 'tempfile'
|
|
|
253
247
|
|
|
254
248
|
# Create temporary archive
|
|
255
249
|
Tempfile.create(['backup', '.zip']) do |tmpfile|
|
|
256
|
-
Omnizip
|
|
257
|
-
archive.add_directory('data/')
|
|
258
|
-
end
|
|
250
|
+
Omnizip.compress_directory('data', tmpfile.path)
|
|
259
251
|
|
|
260
252
|
# Use archive...
|
|
261
253
|
puts "Temporary archive: #{tmpfile.path}"
|
|
@@ -12,7 +12,7 @@ Protect sensitive data in your archives with industry-standard AES-256 encryptio
|
|
|
12
12
|
== Benefits
|
|
13
13
|
|
|
14
14
|
* **Strong Security**: AES-256 encryption with secure key derivation
|
|
15
|
-
* **Format
|
|
15
|
+
* **Format**: Encryption available for 7z archives (ZIP encryption is unsupported)
|
|
16
16
|
* **Header Protection**: Encrypt filenames and metadata (7z format)
|
|
17
17
|
* **Password Validation**: Built-in password strength checking
|
|
18
18
|
* **Compatibility**: Works with archives from other tools
|
|
@@ -44,40 +44,20 @@ end
|
|
|
44
44
|
* Optional header encryption (hides filenames)
|
|
45
45
|
* HMAC-SHA-256 authentication
|
|
46
46
|
|
|
47
|
-
===
|
|
47
|
+
=== ZIP Encryption: not supported
|
|
48
48
|
|
|
49
|
-
|
|
49
|
+
Omnizip can neither create nor decrypt encrypted ZIP archives —
|
|
50
|
+
neither WinZip AES nor traditional ZipCrypto. Encrypted 7z is the
|
|
51
|
+
supported path; extract encrypted ZIPs with 7-Zip/unzip.
|
|
50
52
|
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
# Create AES-encrypted ZIP
|
|
54
|
-
Omnizip::Zip::File.open('secure.zip', create: true) do |zip|
|
|
55
|
-
zip.password = 'SecurePassword123'
|
|
56
|
-
zip.encryption = :winzip_aes # AES-256
|
|
57
|
-
zip.add('private.doc', 'files/private.doc')
|
|
58
|
-
end
|
|
59
|
-
----
|
|
60
|
-
|
|
61
|
-
**Features**:
|
|
62
|
-
|
|
63
|
-
* AES-256, AES-192, or AES-128
|
|
64
|
-
* Compatible with WinZip, 7-Zip, PeaZip
|
|
65
|
-
* HMAC-SHA-1 authentication
|
|
66
|
-
* Standard ZIP format
|
|
67
|
-
|
|
68
|
-
=== Traditional ZIP Encryption
|
|
69
|
-
|
|
70
|
-
[WARNING]
|
|
71
|
-
Traditional ZIP encryption (ZipCrypto) is weak and should only be used for compatibility with legacy systems.
|
|
53
|
+
The convenience API raises rather than silently ignoring a
|
|
54
|
+
password for zip:
|
|
72
55
|
|
|
73
56
|
[source,ruby]
|
|
74
57
|
----
|
|
75
|
-
|
|
76
|
-
Omnizip::
|
|
77
|
-
|
|
78
|
-
zip.encryption = :traditional # Weak encryption
|
|
79
|
-
zip.add('file.txt', 'data.txt')
|
|
80
|
-
end
|
|
58
|
+
Omnizip::Archive.create('secure.zip', password: 'x')
|
|
59
|
+
# => Omnizip::UnsupportedFormatError:
|
|
60
|
+
# password is only supported for seven_zip; :zip has no encryption support
|
|
81
61
|
----
|
|
82
62
|
|
|
83
63
|
== Decrypting Archives
|
|
@@ -91,10 +71,8 @@ Omnizip::Archive.open('secure.7z', password: 'MyStr0ngP@ssw0rd') do |archive|
|
|
|
91
71
|
archive.extract_all('decrypted/')
|
|
92
72
|
end
|
|
93
73
|
|
|
94
|
-
#
|
|
95
|
-
|
|
96
|
-
zip.extract_all('output/')
|
|
97
|
-
end
|
|
74
|
+
# ZIP encryption: Omnizip can neither create nor decrypt
|
|
75
|
+
# password-protected ZIP archives; extract them with 7-Zip/unzip
|
|
98
76
|
----
|
|
99
77
|
|
|
100
78
|
=== Password via Environment Variable
|
|
@@ -286,12 +264,11 @@ Some older archives may use unsupported encryption:
|
|
|
286
264
|
[source,ruby]
|
|
287
265
|
----
|
|
288
266
|
begin
|
|
289
|
-
Omnizip::Archive.open('old-archive.
|
|
290
|
-
|
|
267
|
+
Omnizip::Archive.open('old-archive.7z', password: 'wrong') do |archive|
|
|
268
|
+
archive.read('secret.txt')
|
|
291
269
|
end
|
|
292
|
-
rescue Omnizip::
|
|
293
|
-
puts "
|
|
294
|
-
puts "Try opening with 7-Zip or WinZip first"
|
|
270
|
+
rescue Omnizip::FormatError => e
|
|
271
|
+
puts "Archive damaged or password wrong: #{e.message}"
|
|
295
272
|
end
|
|
296
273
|
----
|
|
297
274
|
|
|
@@ -34,12 +34,22 @@ These features can be combined for maximum efficiency:
|
|
|
34
34
|
|
|
35
35
|
[source,ruby]
|
|
36
36
|
----
|
|
37
|
-
#
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
37
|
+
# Streaming writes + a progress tracker
|
|
38
|
+
require 'omnizip/zip/output_stream'
|
|
39
|
+
|
|
40
|
+
files = Dir.glob('large_data/**/*').select { |f| File.file?(f) }
|
|
41
|
+
tracker = Omnizip::Progress.track(
|
|
42
|
+
total_files: files.size,
|
|
43
|
+
total_bytes: files.sum { |f| File.size(f) },
|
|
44
|
+
reporter: :console
|
|
45
|
+
)
|
|
46
|
+
|
|
47
|
+
Omnizip::Zip::OutputStream.open('backup.zip') do |stream|
|
|
48
|
+
files.each do |file|
|
|
49
|
+
stream.put_next_entry(file.delete_prefix('large_data/'))
|
|
50
|
+
stream.write(File.binread(file))
|
|
51
|
+
tracker.update(files: 1, bytes: File.size(file))
|
|
52
|
+
end
|
|
43
53
|
end
|
|
44
54
|
----
|
|
45
55
|
|
|
@@ -7,176 +7,94 @@ grand_parent: Guides
|
|
|
7
7
|
|
|
8
8
|
== Purpose
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
Omnizip's codecs are pure Ruby and single-threaded: a single archive
|
|
11
|
+
is always compressed or extracted by one CPU core. To use multiple
|
|
12
|
+
cores, run independent archive operations in parallel — across
|
|
13
|
+
processes for CPU-bound compression, or across threads for
|
|
14
|
+
I/O-bound extraction of separate archives.
|
|
11
15
|
|
|
12
|
-
==
|
|
13
|
-
|
|
14
|
-
* **2-3x Faster**: Achieve 2-3x speedup on multi-core systems
|
|
15
|
-
* **Better Resource Utilization**: Use all available CPU cores effectively
|
|
16
|
-
* **Reduced Wait Time**: Process large batches of files quickly
|
|
17
|
-
* **Scalable**: Performance improves with more CPU cores
|
|
18
|
-
|
|
19
|
-
== When to Use Parallel Processing
|
|
16
|
+
== When It Helps
|
|
20
17
|
|
|
21
18
|
[cols="1,3"]
|
|
22
19
|
|===
|
|
23
|
-
|Scenario |
|
|
24
|
-
|
|
25
|
-
|Large Archives (>100 files)
|
|
26
|
-
|Distribute file processing across cores
|
|
20
|
+
|Scenario |Approach
|
|
27
21
|
|
|
28
|
-
|
|
|
29
|
-
|
|
|
22
|
+
|Many archives to compress
|
|
23
|
+
|One process per archive
|
|
30
24
|
|
|
31
|
-
|
|
|
32
|
-
|
|
|
25
|
+
|Many archives to extract
|
|
26
|
+
|One thread per archive (I/O releases the GVL)
|
|
33
27
|
|
|
34
|
-
|
|
|
35
|
-
|
|
|
28
|
+
|One large archive
|
|
29
|
+
|No parallelism available; pick an appropriate +level:+
|
|
36
30
|
|===
|
|
37
31
|
|
|
38
|
-
== Parallel Compression
|
|
32
|
+
== Parallel Compression (Processes)
|
|
39
33
|
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
Automatically distribute files across available cores:
|
|
34
|
+
Compression is CPU-bound, so use processes — MRI's GVL prevents
|
|
35
|
+
in-thread parallelism:
|
|
43
36
|
|
|
44
37
|
[source,ruby]
|
|
45
38
|
----
|
|
46
|
-
# Compress
|
|
47
|
-
|
|
48
|
-
'large_project/',
|
|
49
|
-
'backup.zip',
|
|
50
|
-
parallel: true,
|
|
51
|
-
threads: 4 # Or omit to use all cores
|
|
52
|
-
)
|
|
53
|
-
----
|
|
54
|
-
|
|
55
|
-
=== Custom Thread Count
|
|
39
|
+
# Compress each directory in its own process
|
|
40
|
+
dirs = %w[project_a project_b project_c]
|
|
56
41
|
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
# Use specific number of threads
|
|
62
|
-
Omnizip::Archive.create('archive.7z', format: :seven_zip) do |archive|
|
|
63
|
-
archive.add_directory(
|
|
64
|
-
'data/',
|
|
65
|
-
parallel: true,
|
|
66
|
-
threads: 8, # Use 8 worker threads
|
|
67
|
-
chunk_size: 1.megabyte # Process 1MB chunks
|
|
68
|
-
)
|
|
42
|
+
pids = dirs.map do |dir|
|
|
43
|
+
Process.fork do
|
|
44
|
+
Omnizip.compress_directory(dir, "#{dir}.7z", format: :seven_zip)
|
|
45
|
+
end
|
|
69
46
|
end
|
|
47
|
+
pids.each { |pid| Process.wait(pid) }
|
|
70
48
|
----
|
|
71
49
|
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
=== Extract Multiple Files Simultaneously
|
|
75
|
-
|
|
76
|
-
Extract files in parallel for faster extraction:
|
|
77
|
-
|
|
78
|
-
[source,ruby]
|
|
79
|
-
----
|
|
80
|
-
# Extract with parallel processing
|
|
81
|
-
Omnizip.extract_archive(
|
|
82
|
-
'large-archive.zip',
|
|
83
|
-
'output/',
|
|
84
|
-
parallel: true,
|
|
85
|
-
threads: 4
|
|
86
|
-
)
|
|
87
|
-
----
|
|
88
|
-
|
|
89
|
-
=== With Progress Tracking
|
|
90
|
-
|
|
91
|
-
Combine parallel processing with progress monitoring:
|
|
50
|
+
Or drive the CLI from your shell or job scheduler:
|
|
92
51
|
|
|
93
|
-
[source,
|
|
52
|
+
[source,bash]
|
|
94
53
|
----
|
|
95
|
-
|
|
96
|
-
puts "Extracted #{progress.files_processed}/#{progress.total_files} files"
|
|
97
|
-
puts "Using #{progress.active_threads} parallel threads"
|
|
98
|
-
puts "Speed: #{progress.throughput_mb_s} MB/s"
|
|
99
|
-
end
|
|
54
|
+
ls -d project_*/ | xargs -P 8 -I{} omnizip archive create {}.7z {}
|
|
100
55
|
----
|
|
101
56
|
|
|
102
|
-
==
|
|
103
|
-
|
|
104
|
-
[cols="2,1,1,1,1"]
|
|
105
|
-
|===
|
|
106
|
-
|Workload |1 Thread |2 Threads |4 Threads |8 Threads
|
|
107
|
-
|
|
108
|
-
|100 small files
|
|
109
|
-
|10s
|
|
110
|
-
|6s
|
|
111
|
-
|4s
|
|
112
|
-
|3s
|
|
113
|
-
|
|
114
|
-
|10 large files
|
|
115
|
-
|60s
|
|
116
|
-
|35s
|
|
117
|
-
|20s
|
|
118
|
-
|15s
|
|
119
|
-
|
|
120
|
-
|Mixed workload
|
|
121
|
-
|30s
|
|
122
|
-
|18s
|
|
123
|
-
|12s
|
|
124
|
-
|10s
|
|
125
|
-
|===
|
|
126
|
-
|
|
127
|
-
*Note: Performance varies based on CPU, disk speed, and file characteristics.*
|
|
57
|
+
== Parallel Extraction (Threads)
|
|
128
58
|
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
=== Thread Count
|
|
132
|
-
|
|
133
|
-
Control parallelism level:
|
|
59
|
+
Extracting separate archives is I/O-bound, so threads work:
|
|
134
60
|
|
|
135
61
|
[source,ruby]
|
|
136
62
|
----
|
|
137
|
-
|
|
138
|
-
parallel: true
|
|
139
|
-
|
|
140
|
-
# Specific thread count
|
|
141
|
-
parallel: true, threads: 4
|
|
142
|
-
|
|
143
|
-
# Match CPU cores
|
|
144
|
-
parallel: true, threads: Etc.nprocessors
|
|
145
|
-
----
|
|
146
|
-
|
|
147
|
-
=== Chunk Size
|
|
63
|
+
archives = Dir.glob('downloads/*.zip')
|
|
148
64
|
|
|
149
|
-
|
|
65
|
+
threads = archives.map do |archive|
|
|
66
|
+
Thread.new do
|
|
67
|
+
Omnizip.extract_archive(
|
|
68
|
+
archive,
|
|
69
|
+
"extracted/#{File.basename(archive, '.zip')}/"
|
|
70
|
+
)
|
|
71
|
+
end
|
|
72
|
+
end
|
|
150
73
|
|
|
151
|
-
|
|
74
|
+
threads.each(&:join)
|
|
152
75
|
----
|
|
153
|
-
# Smaller chunks = better distribution, more overhead
|
|
154
|
-
chunk_size: 512.kilobytes
|
|
155
|
-
|
|
156
|
-
# Larger chunks = less overhead, less parallel benefit
|
|
157
|
-
chunk_size: 10.megabytes
|
|
158
76
|
|
|
159
|
-
|
|
160
|
-
chunk_size: 1.megabyte
|
|
161
|
-
----
|
|
77
|
+
Note: a single archive's extraction still uses one thread.
|
|
162
78
|
|
|
163
79
|
== Use Cases
|
|
164
80
|
|
|
165
81
|
=== Server Deployments
|
|
166
82
|
|
|
167
|
-
Maximize server hardware utilization:
|
|
168
|
-
|
|
169
83
|
[source,ruby]
|
|
170
84
|
----
|
|
171
|
-
#
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
85
|
+
# One process per site on a multi-core backup server
|
|
86
|
+
sites = Dir.children('/var/www')
|
|
87
|
+
|
|
88
|
+
pids = sites.map do |site|
|
|
89
|
+
Process.fork do
|
|
90
|
+
Omnizip.compress_directory(
|
|
91
|
+
File.join('/var/www', site),
|
|
92
|
+
"/backups/#{site}.7z",
|
|
93
|
+
format: :seven_zip, algorithm: :lzma2, level: 9
|
|
94
|
+
)
|
|
95
|
+
end
|
|
96
|
+
end
|
|
97
|
+
pids.each { |pid| Process.wait(pid) }
|
|
180
98
|
----
|
|
181
99
|
|
|
182
100
|
=== Batch Processing
|
|
@@ -204,43 +122,23 @@ end
|
|
|
204
122
|
|
|
205
123
|
== Best Practices
|
|
206
124
|
|
|
207
|
-
. **
|
|
208
|
-
. **Consider
|
|
209
|
-
. **Monitor
|
|
210
|
-
. **Test
|
|
211
|
-
. **Combine with Streaming**: Use both for maximum efficiency
|
|
125
|
+
. **One process per archive** for compression; threads only help I/O
|
|
126
|
+
. **Consider disk I/O**: more parallelism may not help on slow disks
|
|
127
|
+
. **Monitor memory**: each process/thread adds overhead
|
|
128
|
+
. **Test your workload**: measure the real speedup
|
|
212
129
|
|
|
213
130
|
== Limitations
|
|
214
131
|
|
|
215
|
-
*
|
|
216
|
-
*
|
|
217
|
-
*
|
|
218
|
-
* Very small files have overhead from thread management
|
|
132
|
+
* No intra-archive parallelism: a single archive uses one core
|
|
133
|
+
* MRI's GVL blocks in-thread CPU parallelism — use processes
|
|
134
|
+
* Memory usage grows with process/thread count
|
|
219
135
|
|
|
220
136
|
== Platform Considerations
|
|
221
137
|
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|Platform |Notes
|
|
225
|
-
|
|
226
|
-
|MRI Ruby
|
|
227
|
-
|Limited by GIL, 2-3x speedup typical
|
|
228
|
-
|
|
229
|
-
|JRuby
|
|
230
|
-
|No GIL, better parallel scalability
|
|
231
|
-
|
|
232
|
-
|TruffleRuby
|
|
233
|
-
|Excellent parallel performance
|
|
234
|
-
|
|
235
|
-
|Windows
|
|
236
|
-
|Similar performance to Unix systems
|
|
237
|
-
|
|
238
|
-
|Linux/macOS
|
|
239
|
-
|Full parallel processing support
|
|
240
|
-
|===
|
|
138
|
+
`Process.fork` is unavailable on Windows — use `Process.spawn` with
|
|
139
|
+
the +omnizip+ CLI there, or a job scheduler.
|
|
241
140
|
|
|
242
141
|
== See Also
|
|
243
142
|
|
|
244
|
-
* link:streaming.html[Streaming] -
|
|
245
|
-
* link
|
|
246
|
-
* link:../../compatibility.html[Compatibility] - Format support for parallel processing
|
|
143
|
+
* link:streaming.html[Streaming] - Memory efficiency
|
|
144
|
+
* link:../compression-algorithms/lzma2.html[LZMA2] - Default 7z algorithm
|