ziperu 0.2.1-aarch64-linux-gnu → 0.2.2-aarch64-linux-gnu
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +73 -0
- data/README.md +56 -5
- data/lib/ziperu/3.1/ziperu.so +0 -0
- data/lib/ziperu/3.2/ziperu.so +0 -0
- data/lib/ziperu/3.3/ziperu.so +0 -0
- data/lib/ziperu/3.4/ziperu.so +0 -0
- data/lib/ziperu/4.0/ziperu.so +0 -0
- data/lib/ziperu/entry/source.rb +1 -1
- data/lib/ziperu/entry.rb +8 -1
- data/lib/ziperu/errors.rb +5 -0
- data/lib/ziperu/file.rb +24 -2
- data/lib/ziperu/input_stream.rb +167 -11
- data/lib/ziperu/output_stream.rb +72 -1
- data/lib/ziperu/remote_io.rb +278 -0
- data/lib/ziperu/replace_rubyzip.rb +67 -0
- data/lib/ziperu/version.rb +1 -1
- data/lib/ziperu.rb +4 -0
- metadata +4 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 172d2af88e5afc2c160c2c3a2aed554b58fa99d0f1bae00f7d02483331b96d4d
|
|
4
|
+
data.tar.gz: 9c622be5788b3198c7c60a0037c1f956e765aad226656be3f7d8e48518bfab97
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 3298d688ef83f87ba627cbd7d82e095a599fdcca91320630ed96fa40a72334c048a4ae412ac80c709e7629e80f85a0270ff4e6e4403135f89b96ceebb6c1bfda
|
|
7
|
+
data.tar.gz: e53a1a4cd9472e94fb3f707b21727f0a4bfd576ed5f31788dfb42ede39d4226e9bd685c37d46e58a6b0792257a8983da02f83692ee520f638830ebf5e88eae1a
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,78 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.2.2 (2026-11-11)
|
|
4
|
+
|
|
5
|
+
Fixed:
|
|
6
|
+
|
|
7
|
+
- Duplicate entry names no longer get around the open-time limits. The
|
|
8
|
+
checks for `max_entries`, `max_total_size`, `max_entry_size` and
|
|
9
|
+
`max_compression_ratio` counted only the last record under each name,
|
|
10
|
+
while listing, `InputStream` and `extract_all` reach every one: a
|
|
11
|
+
zip bomb behind a harmless entry of the same name went through. Every
|
|
12
|
+
record now counts, in one pass.
|
|
13
|
+
- A large `read` interrupted (`Timeout`, `Thread#raise`) and then
|
|
14
|
+
rescued no longer loses data: what it had decompressed is returned by
|
|
15
|
+
the next read. After any other failure part-way through an entry,
|
|
16
|
+
further reads raise until `rewind`, instead of returning a short entry
|
|
17
|
+
with no error.
|
|
18
|
+
- `OutputStream.new(io, stream: true)` whose IO raises, with the error
|
|
19
|
+
rescued and writing carried on: the writer refuses every later call
|
|
20
|
+
rather than lose track of which entries the archive holds (which gave
|
|
21
|
+
a `NoMethodError` on close, or an entry silently overwritten), and no
|
|
22
|
+
longer writes again what it had handed the IO before it raised.
|
|
23
|
+
- `File#add`, `add_stored` and `replace` take a `Pathname` as the path it
|
|
24
|
+
is, not an IO (`Rails.root.join(...)` raised `NoMethodError`).
|
|
25
|
+
- On Windows, `File.open(dir, create: true)` and a commit no longer move
|
|
26
|
+
a directory at that path aside to put the archive in its place; the
|
|
27
|
+
commit raises, as rubyzip's does.
|
|
28
|
+
- Adding to an archive that is read-only (or has other hard links)
|
|
29
|
+
rewrites it instead of failing to append in place.
|
|
30
|
+
- Errors that came out as a bare `IOError` are `Ziperu::Error`s: corrupt
|
|
31
|
+
zstd and LZMA data (`DecompressionError`, as for Deflate), a local
|
|
32
|
+
header past the end of the archive or with a bad signature, a forged
|
|
33
|
+
zip64 offset no file can seek to. A NUL in a name raises
|
|
34
|
+
`ArgumentError` from the native `extract_all`, as from Ruby's own file
|
|
35
|
+
calls.
|
|
36
|
+
- `add_files(threads:)` (and the parallel commit) can be interrupted: it
|
|
37
|
+
stopped only once the whole batch, up to 256 MB, was compressed. It
|
|
38
|
+
still adds the batch whole or not at all.
|
|
39
|
+
- A local header that names a different compression method from the
|
|
40
|
+
central directory is refused, rather than decoded one way or the other
|
|
41
|
+
depending on `validate_entry_sizes`.
|
|
42
|
+
- An interrupted `extract_all` removes the file it was writing, so a
|
|
43
|
+
second run is not stopped by `DestinationExistsError`.
|
|
44
|
+
- Directory times restored by `extract_all` are set once the files in
|
|
45
|
+
the directory are written, which would otherwise move them (and every
|
|
46
|
+
directory entry is made before any file, wherever it is listed).
|
|
47
|
+
- An entry kept across a commit that moved it (the one `remove` returns,
|
|
48
|
+
say) still answers `time`, `extra` and the like from what it holds.
|
|
49
|
+
- `InputStream.open("missing.zip")` names the file in its `ENOENT`;
|
|
50
|
+
`File.open(nil)` raises `TypeError`.
|
|
51
|
+
|
|
52
|
+
Added:
|
|
53
|
+
|
|
54
|
+
- `require "ziperu/replace_rubyzip"`: ziperu answers for rubyzip
|
|
55
|
+
(`require "zip"`, `"rubyzip"`, `"zip/..."`) even without Bundler, where
|
|
56
|
+
RubyGems would otherwise load a rubyzip that a dependency asks for.
|
|
57
|
+
- `InputStream.open(pipe, streaming: true)`: a pipe or socket read in one
|
|
58
|
+
pass, local header by local header, in an entry's worth of memory, with
|
|
59
|
+
the limits counted as the bytes come. Entries whose sizes follow their
|
|
60
|
+
data (how streaming writers write) are read when Deflated.
|
|
61
|
+
- `Ziperu::RemoteIO`: a seekable IO over HTTP Range requests (or a block
|
|
62
|
+
that fetches byte ranges), so an archive on S3 or a web server can be
|
|
63
|
+
listed and have entries read without downloading it.
|
|
64
|
+
- `OutputStream#rollback_entry`: takes the entry being written back out
|
|
65
|
+
of the archive, so one whose data could not be had can be skipped. On
|
|
66
|
+
a stream it works until the entry's first bytes are written (the entry
|
|
67
|
+
is begun only then); on a file or seekable IO, at any point.
|
|
68
|
+
- `InputStream#seek` and `pos=` in a Stored entry that is not encrypted
|
|
69
|
+
(`get_input_stream(...).seek`): random access without reading what
|
|
70
|
+
comes before.
|
|
71
|
+
|
|
72
|
+
Changed:
|
|
73
|
+
|
|
74
|
+
- `open_buffer` on a String copies the archive once, not twice.
|
|
75
|
+
|
|
3
76
|
## 0.2.1 (2026-11-10)
|
|
4
77
|
|
|
5
78
|
Fixed:
|
data/README.md
CHANGED
|
@@ -165,6 +165,51 @@ Ziperu::InputStream.open(socket) do |zis| # any IO, no seeking needed
|
|
|
165
165
|
process(entry.name, zis.read)
|
|
166
166
|
end
|
|
167
167
|
end
|
|
168
|
+
|
|
169
|
+
# A pipe or an upload read in one pass, an entry's worth of memory at a
|
|
170
|
+
# time, rather than taken in whole to reach the central directory:
|
|
171
|
+
Ziperu::InputStream.open(s3_object.body, streaming: true) do |zis|
|
|
172
|
+
while (entry = zis.get_next_entry)
|
|
173
|
+
zis.each_line { |line| ... }
|
|
174
|
+
end
|
|
175
|
+
end
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
Read an archive in place on a server, without downloading it: only the
|
|
179
|
+
central directory and the entries read are fetched, with HTTP Range
|
|
180
|
+
requests (S3, GCS, a CDN; or a block that returns byte ranges):
|
|
181
|
+
|
|
182
|
+
```ruby
|
|
183
|
+
io = Ziperu::RemoteIO.new(url) # "https://example.com/dataset.zip"
|
|
184
|
+
Ziperu::File.open_buffer(io, write: false) do |zf|
|
|
185
|
+
zf.extract("docs/guide.txt", "guide.txt") # a few requests, not 5 GB
|
|
186
|
+
end
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
Leave out an entry whose data could not be had, and go on with the rest
|
|
190
|
+
(a download that failed half-way through a streamed zip, say):
|
|
191
|
+
|
|
192
|
+
```ruby
|
|
193
|
+
body = Ziperu::OutputStream.body do |zos|
|
|
194
|
+
files.each do |name, url|
|
|
195
|
+
zos.put_next_entry(name)
|
|
196
|
+
begin
|
|
197
|
+
fetch(url) { |chunk| zos << chunk }
|
|
198
|
+
rescue Net::HTTPError
|
|
199
|
+
zos.rollback_entry # before its first byte on a stream; any time on a file
|
|
200
|
+
end
|
|
201
|
+
end
|
|
202
|
+
end
|
|
203
|
+
[200, { "content-type" => "application/zip" }, body]
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
Seek about in a Stored entry, for readers that want random access (a PDF
|
|
207
|
+
stored in an archive):
|
|
208
|
+
|
|
209
|
+
```ruby
|
|
210
|
+
Ziperu::File.open("scans.zip") do |zf|
|
|
211
|
+
zf.get_input_stream("scan.pdf") { |io| io.seek(-1024, IO::SEEK_END); io.read(1024) }
|
|
212
|
+
end
|
|
168
213
|
```
|
|
169
214
|
|
|
170
215
|
Passwords, both ways:
|
|
@@ -209,11 +254,17 @@ path, so `require "zip"` loads ziperu and those gems run on it; rubyzip
|
|
|
209
254
|
stays installed and unused. Checked with Bundler 2.3, 2.5 and 4.0, in
|
|
210
255
|
either Gemfile order.
|
|
211
256
|
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
257
|
+
Without Bundler, RubyGems loads whichever gem asks for `zip` first. To
|
|
258
|
+
have ziperu answer for rubyzip there too, require this before anything
|
|
259
|
+
else loads it (the top of a script, or `config/boot.rb`):
|
|
260
|
+
|
|
261
|
+
```ruby
|
|
262
|
+
require "ziperu/replace_rubyzip" # every require "zip" / "zip/..." loads ziperu
|
|
263
|
+
```
|
|
264
|
+
|
|
265
|
+
It raises if rubyzip is already loaded, rather than mix the two. Short of
|
|
266
|
+
that, use `require "ziperu"` and `Ziperu::` for your own code and leave
|
|
267
|
+
`Zip::` to rubyzip.
|
|
217
268
|
|
|
218
269
|
ziperu tracks the rubyzip 3.x API (`Ziperu::RUBYZIP_COMPAT_VERSION`), and
|
|
219
270
|
diverges on purpose where rubyzip is lax: `InputStream` reads through the
|
data/lib/ziperu/3.1/ziperu.so
CHANGED
|
Binary file
|
data/lib/ziperu/3.2/ziperu.so
CHANGED
|
Binary file
|
data/lib/ziperu/3.3/ziperu.so
CHANGED
|
Binary file
|
data/lib/ziperu/3.4/ziperu.so
CHANGED
|
Binary file
|
data/lib/ziperu/4.0/ziperu.so
CHANGED
|
Binary file
|
data/lib/ziperu/entry/source.rb
CHANGED
|
@@ -96,7 +96,7 @@ module Ziperu
|
|
|
96
96
|
reopened = @owner.native_archive
|
|
97
97
|
unless still_at_place?(reopened, offset)
|
|
98
98
|
where = @index ? "at index #{@index} of" : 'in'
|
|
99
|
-
raise
|
|
99
|
+
raise EntryMovedError, "entry '#{@raw_name}' is no longer #{where} #{@owner.name}"
|
|
100
100
|
end
|
|
101
101
|
|
|
102
102
|
@native = reopened
|
data/lib/ziperu/entry.rb
CHANGED
|
@@ -86,7 +86,14 @@ module Ziperu
|
|
|
86
86
|
def extra
|
|
87
87
|
if @source.stored? && !@local_extra_merged
|
|
88
88
|
@local_extra_merged = true
|
|
89
|
-
|
|
89
|
+
begin
|
|
90
|
+
parsed_extra.merge(@source.local_extra(@local_header_offset), local: true)
|
|
91
|
+
rescue EntryMovedError
|
|
92
|
+
# Kept across a commit that moved its record (a removed entry,
|
|
93
|
+
# say): the central directory's fields are all there is now,
|
|
94
|
+
# which is what rubyzip reads anyway.
|
|
95
|
+
nil
|
|
96
|
+
end
|
|
90
97
|
# Edits made through this object (`extra.merge`, `extra.delete`)
|
|
91
98
|
# leave no other trace; the bytes are compared at commit.
|
|
92
99
|
@extra_snapshot = extra_bytes
|
data/lib/ziperu/errors.rb
CHANGED
|
@@ -156,6 +156,11 @@ module Ziperu
|
|
|
156
156
|
|
|
157
157
|
# Raised by the Rust core where it lacks the domain object a rubyzip
|
|
158
158
|
# exception wants; the Ruby layer re-raises the rubyzip-shaped error.
|
|
159
|
+
# An entry object outlived a commit that moved its record; what it holds
|
|
160
|
+
# from the central directory is still good, its local header is gone.
|
|
161
|
+
class EntryMovedError < Error # :nodoc:
|
|
162
|
+
end
|
|
163
|
+
|
|
159
164
|
module Native
|
|
160
165
|
class SizeLimitError < Ziperu::Error; end
|
|
161
166
|
class PasswordError < Ziperu::Error; end
|
data/lib/ziperu/file.rb
CHANGED
|
@@ -82,6 +82,11 @@ module Ziperu
|
|
|
82
82
|
# decrypter on `InputStream`; `decrypter:` is accepted here too.
|
|
83
83
|
@password = password || decrypter&.password
|
|
84
84
|
|
|
85
|
+
unless path_or_io.kind_of?(::String) || path_or_io.respond_to?(:to_path) ||
|
|
86
|
+
path_or_io.respond_to?(:read) || path_or_io.respond_to?(:write)
|
|
87
|
+
raise TypeError, "no implicit conversion of #{path_or_io.inspect} into String"
|
|
88
|
+
end
|
|
89
|
+
|
|
85
90
|
@name = if IOExtras.path_like?(path_or_io)
|
|
86
91
|
IOExtras.path_of(path_or_io)
|
|
87
92
|
elsif path_or_io.respond_to?(:path)
|
|
@@ -312,8 +317,9 @@ module Ziperu
|
|
|
312
317
|
compression_level: @compression_level, settings: @settings
|
|
313
318
|
)
|
|
314
319
|
end
|
|
315
|
-
if src_path.respond_to?(:read)
|
|
320
|
+
if src_path.respond_to?(:read) && !IOExtras.path_like?(src_path)
|
|
316
321
|
# An IO (File, StringIO, Tempfile, a socket): streamed at commit.
|
|
322
|
+
# A Pathname has `read` too, but names a file.
|
|
317
323
|
new_entry.gather_fileinfo_from_io(src_path)
|
|
318
324
|
else
|
|
319
325
|
new_entry.gather_fileinfo_from_srcpath(src_path)
|
|
@@ -579,7 +585,7 @@ module Ziperu
|
|
|
579
585
|
# the archive is rewritten.
|
|
580
586
|
def appended_entries
|
|
581
587
|
return nil if @create || @buffer || @native.nil? || @comment_changed || @settings.sort_entries
|
|
582
|
-
return nil unless @settings.append_in_place
|
|
588
|
+
return nil unless @settings.append_in_place && appendable_in_place?
|
|
583
589
|
|
|
584
590
|
all = @cdir.entries
|
|
585
591
|
held = @native.entry_count
|
|
@@ -592,6 +598,18 @@ module Ziperu
|
|
|
592
598
|
end
|
|
593
599
|
private :appended_entries
|
|
594
600
|
|
|
601
|
+
# Whether the archive's file can take an append: one that is read-only
|
|
602
|
+
# can still be replaced (the rewrite renames a new file over it), and
|
|
603
|
+
# one with other hard links must be, or the append would change what
|
|
604
|
+
# they hold too.
|
|
605
|
+
def appendable_in_place?
|
|
606
|
+
stat = ::File.stat(@name)
|
|
607
|
+
stat.file? && stat.writable? && stat.nlink <= 1
|
|
608
|
+
rescue SystemCallError
|
|
609
|
+
false
|
|
610
|
+
end
|
|
611
|
+
private :appendable_in_place?
|
|
612
|
+
|
|
595
613
|
# Writes the archive, with any changes, to `io` and returns it. `io`
|
|
596
614
|
# may be a pipe, socket or response body: entries are streamed to it
|
|
597
615
|
# as they are written, unless the archive holds ZipCrypto entries,
|
|
@@ -835,6 +853,10 @@ module Ziperu
|
|
|
835
853
|
begin
|
|
836
854
|
::File.rename(tmp_filename, name)
|
|
837
855
|
rescue Errno::EACCES
|
|
856
|
+
# A directory (or anything but a file) at the path is not ours to
|
|
857
|
+
# move: the commit fails, as rubyzip's does.
|
|
858
|
+
raise unless ::File.file?(name)
|
|
859
|
+
|
|
838
860
|
# Something else (another Ziperu::File, say) still has the archive
|
|
839
861
|
# open. Renaming an open file is allowed where replacing one is
|
|
840
862
|
# not, so move the old archive aside first; that handle keeps
|
data/lib/ziperu/input_stream.rb
CHANGED
|
@@ -28,8 +28,20 @@ module Ziperu
|
|
|
28
28
|
# `Ziperu.*` setting by its name, for this stream alone (the limits,
|
|
29
29
|
# `validate_entry_sizes:`, `force_entry_names_encoding:`; see
|
|
30
30
|
# Ziperu::Settings).
|
|
31
|
+
#
|
|
32
|
+
# `streaming: true` reads a pipe or socket (an upload, an S3 object's
|
|
33
|
+
# body) in one pass, entry by entry, rather than taking all of it in
|
|
34
|
+
# first to reach the central directory at its end: memory stays at an
|
|
35
|
+
# entry's worth, whatever the archive's size. What the stream gives is
|
|
36
|
+
# what the local headers hold: no comments, no permissions, and an
|
|
37
|
+
# entry whose sizes come after its data has them (`size`, `crc`) only
|
|
38
|
+
# once it has been read to its end. Such an entry must be Deflated,
|
|
39
|
+
# which is how streaming writers write; a Stored one cannot be told
|
|
40
|
+
# apart from what follows it in one pass, and raises. The limits apply
|
|
41
|
+
# to the bytes as they come. A file, a StringIO or a seekable IO is
|
|
42
|
+
# read through its central directory either way. Not in rubyzip.
|
|
31
43
|
def initialize(context, *legacy, offset: 0, decrypter: nil, password: nil,
|
|
32
|
-
enforce_size: nil, metadata_encoding: nil, **opts)
|
|
44
|
+
enforce_size: nil, metadata_encoding: nil, streaming: false, **opts)
|
|
33
45
|
@settings = Settings.take!(opts)
|
|
34
46
|
enforce_size = @settings.validate_entry_sizes if enforce_size.nil?
|
|
35
47
|
@metadata_encoding = metadata_encoding
|
|
@@ -41,6 +53,19 @@ module Ziperu
|
|
|
41
53
|
# AES from ZipCrypto by the entry, so one password serves both.
|
|
42
54
|
decrypter ||= TraditionalDecrypter.new(password) if password
|
|
43
55
|
super(**opts)
|
|
56
|
+
if streaming && self.class.pipe_like?(context)
|
|
57
|
+
IOExtras.binmode(context)
|
|
58
|
+
@stream = Native::StreamReader.open(context, decrypter&.password&.b, enforce_size)
|
|
59
|
+
@stream_entries = 0
|
|
60
|
+
@stream_total = 0
|
|
61
|
+
@decrypter = decrypter
|
|
62
|
+
@enforce_size = enforce_size
|
|
63
|
+
@owns_archive = false
|
|
64
|
+
@archive = nil
|
|
65
|
+
@reader = nil
|
|
66
|
+
@current_entry = nil
|
|
67
|
+
return
|
|
68
|
+
end
|
|
44
69
|
@split = !context.kind_of?(Native::Archive) && self.class.split_archive?(context)
|
|
45
70
|
# An archive handed in (by Entry#get_input_stream) stays open for its
|
|
46
71
|
# owner; one opened here is closed with the stream. A split segment
|
|
@@ -85,6 +110,8 @@ module Ziperu
|
|
|
85
110
|
def close
|
|
86
111
|
@reader&.close
|
|
87
112
|
@reader = nil
|
|
113
|
+
@stream&.close
|
|
114
|
+
@stream = nil
|
|
88
115
|
@archive&.close if @owns_archive
|
|
89
116
|
@archive = nil
|
|
90
117
|
@open_error = nil
|
|
@@ -95,6 +122,8 @@ module Ziperu
|
|
|
95
122
|
def get_next_entry
|
|
96
123
|
raise SplitArchiveError if @split
|
|
97
124
|
check_open!
|
|
125
|
+
return next_stream_entry if @stream
|
|
126
|
+
|
|
98
127
|
return @current_entry = nil if @archive.nil? || @next_index.nil?
|
|
99
128
|
if @next_index >= (@records ? @records.size : @archive.entry_count)
|
|
100
129
|
# Past the last entry there is nothing to read, not the previous
|
|
@@ -154,6 +183,7 @@ module Ziperu
|
|
|
154
183
|
def rewind
|
|
155
184
|
check_open!
|
|
156
185
|
return if @current_entry.nil?
|
|
186
|
+
raise Error, 'an entry read with streaming: true cannot be rewound' if @stream
|
|
157
187
|
|
|
158
188
|
@lineno = 0
|
|
159
189
|
@pos = 0
|
|
@@ -161,6 +191,39 @@ module Ziperu
|
|
|
161
191
|
flush
|
|
162
192
|
end
|
|
163
193
|
|
|
194
|
+
# Moves to byte `offset` of the current entry (from its start, its
|
|
195
|
+
# current position or its end, as `whence` says), as IO#seek does.
|
|
196
|
+
# Only a Stored entry that is not encrypted can be seeked in: its
|
|
197
|
+
# bytes are in the archive as they are. A compressed one raises
|
|
198
|
+
# Ziperu::Error; `rewind` is the way back to its start. Reads after a
|
|
199
|
+
# seek check no CRC, which only reading from the start can do.
|
|
200
|
+
# Not in rubyzip.
|
|
201
|
+
def seek(offset, whence = ::IO::SEEK_SET)
|
|
202
|
+
check_open!
|
|
203
|
+
raise Error, 'no entry to seek in' if @reader.nil?
|
|
204
|
+
raise Error, 'an entry read with streaming: true cannot be seeked in' if @stream
|
|
205
|
+
|
|
206
|
+
target =
|
|
207
|
+
case whence
|
|
208
|
+
when ::IO::SEEK_SET, :SET then offset
|
|
209
|
+
when ::IO::SEEK_CUR, :CUR then pos + offset
|
|
210
|
+
when ::IO::SEEK_END, :END then @current_entry.size + offset
|
|
211
|
+
else raise ArgumentError, "unknown whence: #{whence.inspect}"
|
|
212
|
+
end
|
|
213
|
+
raise Errno::EINVAL, "seek to #{target}" if target.negative?
|
|
214
|
+
|
|
215
|
+
wrap_errors { @reader.seek(target) }
|
|
216
|
+
flush
|
|
217
|
+
@pos = [target, @current_entry.size].min
|
|
218
|
+
0
|
|
219
|
+
end
|
|
220
|
+
|
|
221
|
+
# IO#pos=, through #seek.
|
|
222
|
+
# Not in rubyzip.
|
|
223
|
+
def pos=(offset)
|
|
224
|
+
seek(offset)
|
|
225
|
+
end
|
|
226
|
+
|
|
164
227
|
# Modelled after IO#sysread. Reads up to maxlen bytes from the stream,
|
|
165
228
|
# bypassing the line buffer; returns a string (either a new string or the
|
|
166
229
|
# given out_string).
|
|
@@ -202,20 +265,26 @@ module Ziperu
|
|
|
202
265
|
check_limits(open_native(context, settings), settings)
|
|
203
266
|
end
|
|
204
267
|
|
|
205
|
-
# Ziperu.max_entries / max_total_size
|
|
206
|
-
#
|
|
268
|
+
# Ziperu.max_entries / max_total_size / max_entry_size /
|
|
269
|
+
# max_compression_ratio, checked before any entry is read, over every
|
|
270
|
+
# record in the central directory (several under one name included).
|
|
271
|
+
# An archive that fails is closed again.
|
|
207
272
|
def check_limits(native, settings = ::Ziperu)
|
|
208
|
-
|
|
209
|
-
|
|
273
|
+
limits = [settings.max_entries, settings.max_total_size,
|
|
274
|
+
settings.max_entry_size, settings.max_compression_ratio]
|
|
275
|
+
return native if limits.none?
|
|
276
|
+
|
|
277
|
+
count, total, largest, ratio = native.limits
|
|
278
|
+
if (limit = settings.max_entries) && count > limit
|
|
279
|
+
raise LimitError.new('an entry count', count, limit)
|
|
210
280
|
end
|
|
211
|
-
if (limit = settings.max_total_size) &&
|
|
281
|
+
if (limit = settings.max_total_size) && total > limit
|
|
212
282
|
raise LimitError.new('a declared total size', total, limit)
|
|
213
283
|
end
|
|
214
|
-
if (limit = settings.max_entry_size) &&
|
|
284
|
+
if (limit = settings.max_entry_size) && largest > limit
|
|
215
285
|
raise LimitError.new('an entry with a declared size', largest, limit)
|
|
216
286
|
end
|
|
217
|
-
if (limit = settings.max_compression_ratio) &&
|
|
218
|
-
(ratio = native.max_compression_ratio) > limit
|
|
287
|
+
if (limit = settings.max_compression_ratio) && ratio > limit
|
|
219
288
|
raise LimitError.new('an entry with a compression ratio', ratio.round(1), limit)
|
|
220
289
|
end
|
|
221
290
|
|
|
@@ -229,6 +298,12 @@ module Ziperu
|
|
|
229
298
|
IOExtras.path_like?(context)
|
|
230
299
|
end
|
|
231
300
|
|
|
301
|
+
# A pipe, socket or the like: an IO that can only be read forward.
|
|
302
|
+
def pipe_like?(context) # :nodoc:
|
|
303
|
+
!context.kind_of?(Native::Archive) && !path_like?(context) &&
|
|
304
|
+
!context.kind_of?(::StringIO) && context.respond_to?(:read) && !IOExtras.seekable?(context)
|
|
305
|
+
end
|
|
306
|
+
|
|
232
307
|
# Whether what `context` holds begins with a local file header, the
|
|
233
308
|
# one thing rubyzip's `InputStream` needs to yield entries.
|
|
234
309
|
def archive_like?(context) # :nodoc:
|
|
@@ -251,7 +326,15 @@ module Ziperu
|
|
|
251
326
|
end
|
|
252
327
|
|
|
253
328
|
def open_native(context, settings = ::Ziperu)
|
|
254
|
-
|
|
329
|
+
if path_like?(context)
|
|
330
|
+
path = path_of(context)
|
|
331
|
+
begin
|
|
332
|
+
return Native::Archive.open_path(path)
|
|
333
|
+
rescue Errno::ENOENT
|
|
334
|
+
# The native message names no file; Ruby's own does.
|
|
335
|
+
raise Errno::ENOENT, path
|
|
336
|
+
end
|
|
337
|
+
end
|
|
255
338
|
|
|
256
339
|
# A StringIO's bytes are already in memory: snapshot them once rather
|
|
257
340
|
# than paying a method call per chunk (12x faster per entry).
|
|
@@ -338,6 +421,7 @@ module Ziperu
|
|
|
338
421
|
end
|
|
339
422
|
raise ArgumentError, "negative length #{maxlen} given" if maxlen&.negative?
|
|
340
423
|
return nil if @reader.nil?
|
|
424
|
+
return produce_stream_input(maxlen) if @stream
|
|
341
425
|
|
|
342
426
|
wrap_errors { @reader.read(maxlen) }
|
|
343
427
|
end
|
|
@@ -363,7 +447,79 @@ module Ziperu
|
|
|
363
447
|
# archive (`@open_error` holds why), which yields no entries but is
|
|
364
448
|
# not closed.
|
|
365
449
|
def closed?
|
|
366
|
-
@archive.nil? && @open_error.nil?
|
|
450
|
+
@archive.nil? && @open_error.nil? && @stream.nil?
|
|
451
|
+
end
|
|
452
|
+
|
|
453
|
+
# `get_next_entry` with `streaming: true`: the next local header.
|
|
454
|
+
def next_stream_entry
|
|
455
|
+
meta = wrap_errors { @stream.next_entry }
|
|
456
|
+
@reader = nil
|
|
457
|
+
flush
|
|
458
|
+
return @current_entry = nil unless meta
|
|
459
|
+
|
|
460
|
+
@stream_entries += 1
|
|
461
|
+
if (limit = @settings.max_entries) && @stream_entries > limit
|
|
462
|
+
raise LimitError.new('an entry count', @stream_entries, limit)
|
|
463
|
+
end
|
|
464
|
+
if meta[:size]
|
|
465
|
+
check_stream_entry_size(meta[:size])
|
|
466
|
+
if (limit = @settings.max_compression_ratio) && meta[:size].positive?
|
|
467
|
+
ratio = meta[:compressed_size].zero? ? Float::INFINITY : meta[:size].fdiv(meta[:compressed_size])
|
|
468
|
+
raise LimitError.new('an entry with a compression ratio', ratio.round(1), limit) if ratio > limit
|
|
469
|
+
end
|
|
470
|
+
end
|
|
471
|
+
# Unknown until the entry has been read (see `produce_stream_input`).
|
|
472
|
+
meta[:crc] ||= 0
|
|
473
|
+
meta[:compressed_size] ||= 0
|
|
474
|
+
@current_entry = Entry.from_meta(nil, meta, metadata_encoding: @metadata_encoding, settings: @settings)
|
|
475
|
+
@current_entry.instance_variable_set(:@size, nil) if meta[:size].nil?
|
|
476
|
+
@stream_entry_bytes = 0
|
|
477
|
+
@reader = @stream
|
|
478
|
+
@lineno = 0
|
|
479
|
+
@pos = 0
|
|
480
|
+
@current_entry
|
|
481
|
+
end
|
|
482
|
+
|
|
483
|
+
# Reads from the stream, counting against the limits as the bytes
|
|
484
|
+
# come; nil `maxlen` reads to the end of the entry, in chunks.
|
|
485
|
+
def produce_stream_input(maxlen)
|
|
486
|
+
return stream_chunk(maxlen) if maxlen
|
|
487
|
+
|
|
488
|
+
out = +''.b
|
|
489
|
+
while (chunk = stream_chunk(CHUNK_SIZE))
|
|
490
|
+
out << chunk
|
|
491
|
+
end
|
|
492
|
+
out.empty? ? nil : out
|
|
493
|
+
end
|
|
494
|
+
|
|
495
|
+
def stream_chunk(length)
|
|
496
|
+
chunk = wrap_errors { @stream.read(length) }
|
|
497
|
+
if chunk
|
|
498
|
+
@stream_entry_bytes += chunk.bytesize
|
|
499
|
+
@stream_total += chunk.bytesize
|
|
500
|
+
check_stream_entry_size(@stream_entry_bytes)
|
|
501
|
+
if (limit = @settings.max_total_size) && @stream_total > limit
|
|
502
|
+
raise LimitError.new('a total size', @stream_total, limit)
|
|
503
|
+
end
|
|
504
|
+
end
|
|
505
|
+
learn_stream_trailer if @stream.eof?
|
|
506
|
+
chunk
|
|
507
|
+
end
|
|
508
|
+
|
|
509
|
+
def check_stream_entry_size(size)
|
|
510
|
+
return unless (limit = @settings.max_entry_size) && size > limit
|
|
511
|
+
|
|
512
|
+
raise LimitError.new('an entry with a size', size, limit)
|
|
513
|
+
end
|
|
514
|
+
|
|
515
|
+
# Once a streamed entry has been read to its end: what its data came
|
|
516
|
+
# to, for an entry whose header left it to the descriptor.
|
|
517
|
+
def learn_stream_trailer
|
|
518
|
+
crc, size = @stream.trailer
|
|
519
|
+
return unless crc && !@current_entry.size_known?
|
|
520
|
+
|
|
521
|
+
@current_entry.instance_variable_set(:@size, size)
|
|
522
|
+
@current_entry.instance_variable_set(:@crc, crc)
|
|
367
523
|
end
|
|
368
524
|
|
|
369
525
|
# Re-raise the native core's errors as the rubyzip-shaped ones.
|
data/lib/ziperu/output_stream.rb
CHANGED
|
@@ -82,6 +82,12 @@ module Ziperu
|
|
|
82
82
|
@closed = false
|
|
83
83
|
@current_entry = nil
|
|
84
84
|
@written = []
|
|
85
|
+
# On a stream, a file entry is begun in the native writer only when
|
|
86
|
+
# its first bytes are written (or the next entry, or the end, comes):
|
|
87
|
+
# until then nothing of it has gone out, so `rollback_entry` can
|
|
88
|
+
# still take it back. Its name and options wait here.
|
|
89
|
+
@deferred = nil
|
|
90
|
+
@stream_names = {} if @streamed
|
|
85
91
|
end
|
|
86
92
|
|
|
87
93
|
class << self
|
|
@@ -224,6 +230,42 @@ module Ziperu
|
|
|
224
230
|
@current_entry = new_entry
|
|
225
231
|
end
|
|
226
232
|
|
|
233
|
+
# Takes the entry being written back out of the archive, as if it had
|
|
234
|
+
# never been put: the central directory will not list it, and the
|
|
235
|
+
# next entry goes where it was. For an entry whose data could not be
|
|
236
|
+
# had (a download that failed, say), so the rest of the archive can
|
|
237
|
+
# still be written:
|
|
238
|
+
#
|
|
239
|
+
# files.each do |name, url|
|
|
240
|
+
# zos.put_next_entry(name)
|
|
241
|
+
# begin
|
|
242
|
+
# fetch(url) { |chunk| zos << chunk }
|
|
243
|
+
# rescue Net::HTTPError
|
|
244
|
+
# zos.rollback_entry
|
|
245
|
+
# end
|
|
246
|
+
# end
|
|
247
|
+
#
|
|
248
|
+
# On a stream to an IO that cannot seek (`body`, `write_buffer` to a
|
|
249
|
+
# pipe) bytes already sent cannot be taken back, so there it works
|
|
250
|
+
# until the entry's first bytes are written, and raises after.
|
|
251
|
+
# Returns the entry taken back.
|
|
252
|
+
# Not in rubyzip.
|
|
253
|
+
def rollback_entry
|
|
254
|
+
check_open!
|
|
255
|
+
entry = @current_entry
|
|
256
|
+
raise Error, 'there is no entry being written to take back' if entry.nil? || !@written.last.equal?(entry)
|
|
257
|
+
|
|
258
|
+
if @deferred
|
|
259
|
+
@deferred = nil
|
|
260
|
+
else
|
|
261
|
+
@writer.rollback_entry
|
|
262
|
+
end
|
|
263
|
+
@written.pop
|
|
264
|
+
@stream_names&.delete(entry.name.b)
|
|
265
|
+
@current_entry = nil
|
|
266
|
+
entry
|
|
267
|
+
end
|
|
268
|
+
|
|
227
269
|
# Bytes handed to the IO so far, for a stream to an IO that cannot
|
|
228
270
|
# seek (`write_buffer` to a pipe or socket, `body`): what the consumer
|
|
229
271
|
# has been given, for a progress report alongside
|
|
@@ -320,6 +362,8 @@ module Ziperu
|
|
|
320
362
|
{ dos_date: time.to_binary_dos_date, dos_time: time.to_binary_dos_time,
|
|
321
363
|
unix_perms: entry.unix_perms }
|
|
322
364
|
end
|
|
365
|
+
start_deferred
|
|
366
|
+
@stream_names[(new_name || entry.name).to_s.b] = true if @stream_names
|
|
323
367
|
@writer.raw_copy(entry.archive, entry.index, new_name, touch_opts)
|
|
324
368
|
@written << entry
|
|
325
369
|
@current_entry = nil
|
|
@@ -347,6 +391,7 @@ module Ziperu
|
|
|
347
391
|
raise Error, "cannot write data to directory entry '#{@current_entry.name}'"
|
|
348
392
|
end
|
|
349
393
|
|
|
394
|
+
start_deferred
|
|
350
395
|
@writer.write(data.to_s.b)
|
|
351
396
|
self
|
|
352
397
|
end
|
|
@@ -362,6 +407,7 @@ module Ziperu
|
|
|
362
407
|
|
|
363
408
|
@writer.comment = comment
|
|
364
409
|
end
|
|
410
|
+
start_deferred
|
|
365
411
|
@current_entry = nil
|
|
366
412
|
begin
|
|
367
413
|
bytes, stats = @writer.finish
|
|
@@ -372,6 +418,10 @@ module Ziperu
|
|
|
372
418
|
"the IO given does not allow (#{e.message}); open it for reading and writing"
|
|
373
419
|
end
|
|
374
420
|
unless @aborted
|
|
421
|
+
if stats.size != @written.size
|
|
422
|
+
raise Error, "the archive holds #{stats.size} entries where #{@written.size} were written"
|
|
423
|
+
end
|
|
424
|
+
|
|
375
425
|
stats.each_with_index do |(offset, size, compressed_size, crc), i|
|
|
376
426
|
@written[i].written_as(offset, size, compressed_size, crc)
|
|
377
427
|
end
|
|
@@ -388,19 +438,40 @@ module Ziperu
|
|
|
388
438
|
raise ::Ziperu::CompressionMethodError, entry.compression_method
|
|
389
439
|
end
|
|
390
440
|
|
|
441
|
+
start_deferred
|
|
391
442
|
opts = native_entry_options(entry)
|
|
443
|
+
if @stream_names
|
|
444
|
+
# The native writer would catch a second entry under a name only
|
|
445
|
+
# once it begins it, which on a stream is later (see @deferred).
|
|
446
|
+
raise EntryExistsError.new('put_next_entry', entry.name) if @stream_names.key?(entry.name.b)
|
|
447
|
+
|
|
448
|
+
@stream_names[entry.name.b] = true
|
|
449
|
+
end
|
|
392
450
|
if entry.directory?
|
|
393
451
|
@writer.add_directory(entry.name, opts)
|
|
394
452
|
elsif entry.symlink?
|
|
395
453
|
@writer.add_symlink(entry.name, entry.symlink_target, opts)
|
|
396
454
|
else
|
|
397
455
|
opts.merge!(@encrypter.native_options)
|
|
398
|
-
@
|
|
456
|
+
if @streamed
|
|
457
|
+
@deferred = [entry.name.b, opts]
|
|
458
|
+
else
|
|
459
|
+
@writer.start_entry(entry.name.b, opts)
|
|
460
|
+
end
|
|
399
461
|
end
|
|
400
462
|
rescue Native::DuplicateEntryError
|
|
401
463
|
raise EntryExistsError.new('put_next_entry', entry.name)
|
|
402
464
|
end
|
|
403
465
|
|
|
466
|
+
# Begins in the native writer the entry `init_next_entry` deferred.
|
|
467
|
+
def start_deferred
|
|
468
|
+
name, opts = @deferred
|
|
469
|
+
return unless name
|
|
470
|
+
|
|
471
|
+
@deferred = nil
|
|
472
|
+
@writer.start_entry(name, opts)
|
|
473
|
+
end
|
|
474
|
+
|
|
404
475
|
def check_open!
|
|
405
476
|
raise Error, 'zip stream is closed' if @closed
|
|
406
477
|
end
|
|
@@ -0,0 +1,278 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'net/http'
|
|
4
|
+
require 'uri'
|
|
5
|
+
|
|
6
|
+
module Ziperu
|
|
7
|
+
# A read-only, seekable IO over an archive somewhere else: an HTTP(S)
|
|
8
|
+
# URL that answers Range requests (S3, GCS, a CDN, most file servers),
|
|
9
|
+
# or anything a block can fetch byte ranges from. Opening an archive
|
|
10
|
+
# through it reads only what is needed: the end of the file (its
|
|
11
|
+
# central directory) in one request, then each entry's bytes as they
|
|
12
|
+
# are read, so listing a 5 GB archive or pulling one file out of it
|
|
13
|
+
# costs a few requests and a few megabytes.
|
|
14
|
+
#
|
|
15
|
+
# io = Ziperu::RemoteIO.new("https://example.com/big.zip")
|
|
16
|
+
# Ziperu::File.open_buffer(io, write: false) do |zf|
|
|
17
|
+
# zf.extract("docs/readme.txt", "readme.txt")
|
|
18
|
+
# end
|
|
19
|
+
#
|
|
20
|
+
# S3 refuses HEAD to some credentials, so the size comes from the first
|
|
21
|
+
# ranged GET (a presigned URL works as is). With the AWS SDK instead:
|
|
22
|
+
#
|
|
23
|
+
# s3 = Aws::S3::Client.new
|
|
24
|
+
# size = s3.head_object(bucket: b, key: k).content_length
|
|
25
|
+
# io = Ziperu::RemoteIO.new(size: size) do |offset, length|
|
|
26
|
+
# s3.get_object(bucket: b, key: k, range: "bytes=#{offset}-#{offset + length - 1}").body.read
|
|
27
|
+
# end
|
|
28
|
+
#
|
|
29
|
+
# Reads go in blocks of `block_size` bytes (adjacent missing blocks in
|
|
30
|
+
# one request); the last `cache_blocks` of them are kept. Not thread
|
|
31
|
+
# safe: one RemoteIO per thread.
|
|
32
|
+
#
|
|
33
|
+
# Not in rubyzip.
|
|
34
|
+
class RemoteIO
|
|
35
|
+
# Raised when the server or the block does not give the bytes asked
|
|
36
|
+
# for.
|
|
37
|
+
class Error < ::Ziperu::Error; end
|
|
38
|
+
|
|
39
|
+
DEFAULT_BLOCK_SIZE = 256 * 1024
|
|
40
|
+
DEFAULT_CACHE_BLOCKS = 64
|
|
41
|
+
# The first request asks for this much of the end: the end of central
|
|
42
|
+
# directory record, and for most archives the whole central directory.
|
|
43
|
+
TAIL_SIZE = 64 * 1024
|
|
44
|
+
MAX_REDIRECTS = 5
|
|
45
|
+
|
|
46
|
+
attr_reader :size, :pos, :requests
|
|
47
|
+
alias tell pos
|
|
48
|
+
alias length size
|
|
49
|
+
|
|
50
|
+
# `url` (a String or URI) to fetch over HTTP(S) with `headers` sent on
|
|
51
|
+
# every request; or, with no url, `size:` and a block that returns
|
|
52
|
+
# `length` bytes from `offset`.
|
|
53
|
+
def initialize(url = nil, headers: {}, size: nil, block_size: DEFAULT_BLOCK_SIZE,
|
|
54
|
+
cache_blocks: DEFAULT_CACHE_BLOCKS, open_timeout: 10, read_timeout: 60, &fetch)
|
|
55
|
+
raise ArgumentError, 'give a url or a block, not both' if url && fetch
|
|
56
|
+
raise ArgumentError, 'give a url, or size: and a block' unless url || (fetch && size)
|
|
57
|
+
raise ArgumentError, 'block_size must be positive' unless block_size.positive?
|
|
58
|
+
|
|
59
|
+
@block_size = block_size
|
|
60
|
+
@cache_blocks = [cache_blocks, 1].max
|
|
61
|
+
@blocks = {}
|
|
62
|
+
@tail_start = nil
|
|
63
|
+
@tail = nil
|
|
64
|
+
@pos = 0
|
|
65
|
+
@requests = 0
|
|
66
|
+
@closed = false
|
|
67
|
+
if url
|
|
68
|
+
@uri = URI(url.to_s)
|
|
69
|
+
@headers = headers
|
|
70
|
+
@open_timeout = open_timeout
|
|
71
|
+
@read_timeout = read_timeout
|
|
72
|
+
@size = discover_size
|
|
73
|
+
else
|
|
74
|
+
@fetch = fetch
|
|
75
|
+
@size = Integer(size)
|
|
76
|
+
end
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
def read(length = nil, outbuf = nil)
|
|
80
|
+
check_open
|
|
81
|
+
if length.nil?
|
|
82
|
+
data = take(@size - @pos)
|
|
83
|
+
else
|
|
84
|
+
raise ArgumentError, "negative length #{length} given" if length.negative?
|
|
85
|
+
return +''.b if length.zero?
|
|
86
|
+
return nil if @pos >= @size
|
|
87
|
+
|
|
88
|
+
data = take([length, @size - @pos].min)
|
|
89
|
+
end
|
|
90
|
+
outbuf ? outbuf.replace(data) : data
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
def seek(offset, whence = ::IO::SEEK_SET)
|
|
94
|
+
check_open
|
|
95
|
+
target =
|
|
96
|
+
case whence
|
|
97
|
+
when ::IO::SEEK_SET, :SET then offset
|
|
98
|
+
when ::IO::SEEK_CUR, :CUR then @pos + offset
|
|
99
|
+
when ::IO::SEEK_END, :END then @size + offset
|
|
100
|
+
else raise ArgumentError, "unknown whence: #{whence.inspect}"
|
|
101
|
+
end
|
|
102
|
+
raise Errno::EINVAL, "seek to #{target}" if target.negative?
|
|
103
|
+
|
|
104
|
+
@pos = target
|
|
105
|
+
0
|
|
106
|
+
end
|
|
107
|
+
|
|
108
|
+
def pos=(offset)
|
|
109
|
+
seek(offset)
|
|
110
|
+
end
|
|
111
|
+
|
|
112
|
+
def rewind
|
|
113
|
+
seek(0)
|
|
114
|
+
end
|
|
115
|
+
|
|
116
|
+
def eof?
|
|
117
|
+
@pos >= @size
|
|
118
|
+
end
|
|
119
|
+
alias eof eof?
|
|
120
|
+
|
|
121
|
+
def binmode
|
|
122
|
+
self
|
|
123
|
+
end
|
|
124
|
+
|
|
125
|
+
def close
|
|
126
|
+
@closed = true
|
|
127
|
+
@http&.finish if @http&.started?
|
|
128
|
+
@http = nil
|
|
129
|
+
@blocks.clear
|
|
130
|
+
nil
|
|
131
|
+
end
|
|
132
|
+
|
|
133
|
+
def closed?
|
|
134
|
+
@closed
|
|
135
|
+
end
|
|
136
|
+
|
|
137
|
+
private
|
|
138
|
+
|
|
139
|
+
def check_open
|
|
140
|
+
raise IOError, 'closed stream' if @closed
|
|
141
|
+
end
|
|
142
|
+
|
|
143
|
+
# `count` bytes from the position, which moves past them.
|
|
144
|
+
def take(count)
|
|
145
|
+
return +''.b if count <= 0
|
|
146
|
+
|
|
147
|
+
out = +''.b
|
|
148
|
+
at = @pos
|
|
149
|
+
finish = at + count
|
|
150
|
+
while at < finish
|
|
151
|
+
if @tail_start && at >= @tail_start
|
|
152
|
+
piece = @tail.byteslice(at - @tail_start, finish - at)
|
|
153
|
+
else
|
|
154
|
+
index = at / @block_size
|
|
155
|
+
load_blocks(index, ((finish - 1) / @block_size))
|
|
156
|
+
block = @blocks[index]
|
|
157
|
+
offset = at - (index * @block_size)
|
|
158
|
+
piece = block.byteslice(offset, finish - at)
|
|
159
|
+
# Most recently used last, so the first is the one to drop.
|
|
160
|
+
@blocks[index] = @blocks.delete(index)
|
|
161
|
+
end
|
|
162
|
+
raise Error, "no data at offset #{at}" if piece.nil? || piece.empty?
|
|
163
|
+
|
|
164
|
+
out << piece
|
|
165
|
+
at += piece.bytesize
|
|
166
|
+
end
|
|
167
|
+
@pos = finish
|
|
168
|
+
out
|
|
169
|
+
end
|
|
170
|
+
|
|
171
|
+
# Makes sure blocks `first..last` are cached, fetching each run of
|
|
172
|
+
# missing ones in one request.
|
|
173
|
+
def load_blocks(first, last)
|
|
174
|
+
last = [last, first + @cache_blocks - 1].min
|
|
175
|
+
index = first
|
|
176
|
+
while index <= last
|
|
177
|
+
if @blocks.key?(index)
|
|
178
|
+
index += 1
|
|
179
|
+
next
|
|
180
|
+
end
|
|
181
|
+
run_end = index
|
|
182
|
+
run_end += 1 while run_end < last && !@blocks.key?(run_end + 1)
|
|
183
|
+
offset = index * @block_size
|
|
184
|
+
length = [((run_end + 1) * @block_size), @size].min - offset
|
|
185
|
+
data = fetch(offset, length)
|
|
186
|
+
(index..run_end).each_with_index do |i, n|
|
|
187
|
+
@blocks[i] = data.byteslice(n * @block_size, @block_size)
|
|
188
|
+
end
|
|
189
|
+
@blocks.shift while @blocks.size > @cache_blocks
|
|
190
|
+
index = run_end + 1
|
|
191
|
+
end
|
|
192
|
+
end
|
|
193
|
+
|
|
194
|
+
def fetch(offset, length)
|
|
195
|
+
@requests += 1
|
|
196
|
+
data =
|
|
197
|
+
if @fetch
|
|
198
|
+
@fetch.call(offset, length).to_s.b
|
|
199
|
+
else
|
|
200
|
+
http_range("bytes=#{offset}-#{offset + length - 1}", offset)
|
|
201
|
+
end
|
|
202
|
+
unless data.bytesize == length
|
|
203
|
+
raise Error, "asked for #{length} bytes at offset #{offset}, got #{data.bytesize}"
|
|
204
|
+
end
|
|
205
|
+
|
|
206
|
+
data
|
|
207
|
+
end
|
|
208
|
+
|
|
209
|
+
# The size, from the first request: the last TAIL_SIZE bytes (a
|
|
210
|
+
# suffix range), whose Content-Range carries the total. They are kept,
|
|
211
|
+
# being what opening an archive reads first.
|
|
212
|
+
def discover_size
|
|
213
|
+
@requests += 1
|
|
214
|
+
response = request("bytes=-#{TAIL_SIZE}")
|
|
215
|
+
case response
|
|
216
|
+
when Net::HTTPPartialContent
|
|
217
|
+
total = response['content-range'].to_s[%r{/(\d+)\z}, 1]
|
|
218
|
+
raise Error, "#{@uri} answered without a usable Content-Range" unless total
|
|
219
|
+
|
|
220
|
+
body = response.body.to_s.b
|
|
221
|
+
@tail_start = Integer(total) - body.bytesize
|
|
222
|
+
@tail = body
|
|
223
|
+
Integer(total)
|
|
224
|
+
when Net::HTTPRequestedRangeNotSatisfiable
|
|
225
|
+
# An empty file has no last bytes.
|
|
226
|
+
0
|
|
227
|
+
when Net::HTTPOK
|
|
228
|
+
# The server ignores Range: the whole file came back; keep it.
|
|
229
|
+
body = response.body.to_s.b
|
|
230
|
+
@tail_start = 0
|
|
231
|
+
@tail = body
|
|
232
|
+
body.bytesize
|
|
233
|
+
else
|
|
234
|
+
raise Error, "#{@uri} answered #{response.code} #{response.message}"
|
|
235
|
+
end
|
|
236
|
+
end
|
|
237
|
+
|
|
238
|
+
def http_range(range, offset)
|
|
239
|
+
response = request(range)
|
|
240
|
+
unless response.kind_of?(Net::HTTPPartialContent)
|
|
241
|
+
raise Error, "#{@uri} answered #{response.code} to a Range request at offset #{offset}"
|
|
242
|
+
end
|
|
243
|
+
|
|
244
|
+
response.body.to_s.b
|
|
245
|
+
end
|
|
246
|
+
|
|
247
|
+
# A GET with `Range: range`, following redirects (the URL redirected
|
|
248
|
+
# to is used from then on, as a presigned S3 link redirects once).
|
|
249
|
+
def request(range, redirects = MAX_REDIRECTS)
|
|
250
|
+
http = connection
|
|
251
|
+
get = Net::HTTP::Get.new(@uri)
|
|
252
|
+
@headers.each { |k, v| get[k.to_s] = v.to_s }
|
|
253
|
+
get['range'] = range
|
|
254
|
+
get['accept-encoding'] = 'identity'
|
|
255
|
+
response = http.request(get)
|
|
256
|
+
return response unless response.kind_of?(Net::HTTPRedirection)
|
|
257
|
+
raise Error, "too many redirects from #{@uri}" if redirects.zero?
|
|
258
|
+
|
|
259
|
+
location = URI.join(@uri.to_s, response['location'].to_s)
|
|
260
|
+
if location.host != @uri.host || location.port != @uri.port || location.scheme != @uri.scheme
|
|
261
|
+
@http&.finish if @http&.started?
|
|
262
|
+
@http = nil
|
|
263
|
+
end
|
|
264
|
+
@uri = location
|
|
265
|
+
request(range, redirects - 1)
|
|
266
|
+
end
|
|
267
|
+
|
|
268
|
+
def connection
|
|
269
|
+
return @http if @http&.started?
|
|
270
|
+
|
|
271
|
+
@http = Net::HTTP.new(@uri.host, @uri.port)
|
|
272
|
+
@http.use_ssl = @uri.scheme == 'https'
|
|
273
|
+
@http.open_timeout = @open_timeout
|
|
274
|
+
@http.read_timeout = @read_timeout
|
|
275
|
+
@http.start
|
|
276
|
+
end
|
|
277
|
+
end
|
|
278
|
+
end
|
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# Makes ziperu answer for rubyzip in this process, whatever else is
|
|
4
|
+
# installed:
|
|
5
|
+
#
|
|
6
|
+
# # config/boot.rb, or the first line of a script
|
|
7
|
+
# require "ziperu/replace_rubyzip"
|
|
8
|
+
#
|
|
9
|
+
# From then on `require "zip"`, `require "rubyzip"` and every
|
|
10
|
+
# `require "zip/..."` load ziperu's files, so `Zip` is `Ziperu` and the
|
|
11
|
+
# gems written against rubyzip (roo, caxlsx, rubyXL, ...) run on it.
|
|
12
|
+
#
|
|
13
|
+
# Under Bundler this is what already happens, since ziperu's `lib` comes
|
|
14
|
+
# first on the load path. This file is for everywhere else: a script
|
|
15
|
+
# without Bundler, where RubyGems would activate the rubyzip gem a
|
|
16
|
+
# dependency asks for and put its files ahead of ziperu's on the load
|
|
17
|
+
# path. It must be required before anything loads rubyzip; if rubyzip is
|
|
18
|
+
# already loaded it raises LoadError rather than mix the two.
|
|
19
|
+
#
|
|
20
|
+
# Not in rubyzip.
|
|
21
|
+
|
|
22
|
+
module Ziperu
|
|
23
|
+
# The `require` that sends rubyzip's files to ziperu's. Not in rubyzip.
|
|
24
|
+
module ReplaceRubyzip
|
|
25
|
+
# ziperu's own `lib`, where `zip.rb`, `rubyzip.rb` and `zip/` live.
|
|
26
|
+
LIB = ::File.expand_path('..', __dir__)
|
|
27
|
+
|
|
28
|
+
# The ziperu file that stands in for the rubyzip feature `name`, if
|
|
29
|
+
# there is one.
|
|
30
|
+
def self.substitute(name)
|
|
31
|
+
name = name.to_s
|
|
32
|
+
name = name.delete_suffix('.rb')
|
|
33
|
+
return unless name == 'zip' || name == 'rubyzip' || name.start_with?('zip/')
|
|
34
|
+
|
|
35
|
+
path = ::File.join(LIB, "#{name}.rb")
|
|
36
|
+
path if ::File.file?(path)
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
# Whether rubyzip itself has already been loaded: a `Zip` that is not
|
|
40
|
+
# ziperu's, or one of rubyzip's files among the loaded features.
|
|
41
|
+
def self.rubyzip_loaded?
|
|
42
|
+
return true if defined?(::Zip) && !(defined?(::Ziperu::File) && ::Zip.equal?(::Ziperu))
|
|
43
|
+
|
|
44
|
+
ours = "#{LIB}/"
|
|
45
|
+
$LOADED_FEATURES.any? do |feature|
|
|
46
|
+
!feature.start_with?(ours) && feature.match?(%r{/(zip|rubyzip)\.rb\z|/zip/[^/]+\.rb\z}) &&
|
|
47
|
+
feature.include?('rubyzip')
|
|
48
|
+
end
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
private
|
|
52
|
+
|
|
53
|
+
def require(name)
|
|
54
|
+
path = ReplaceRubyzip.substitute(name)
|
|
55
|
+
path ? super(path) : super
|
|
56
|
+
end
|
|
57
|
+
end
|
|
58
|
+
end
|
|
59
|
+
|
|
60
|
+
if Ziperu::ReplaceRubyzip.rubyzip_loaded?
|
|
61
|
+
raise LoadError, 'rubyzip is already loaded, so ziperu cannot stand in for it: require ' \
|
|
62
|
+
'"ziperu/replace_rubyzip" before anything that loads rubyzip (first thing ' \
|
|
63
|
+
'in config/boot.rb, or at the top of the script)'
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
Kernel.prepend(Ziperu::ReplaceRubyzip)
|
|
67
|
+
require 'zip'
|
data/lib/ziperu/version.rb
CHANGED
data/lib/ziperu.rb
CHANGED
|
@@ -82,6 +82,10 @@ require_relative "ziperu/size_estimator"
|
|
|
82
82
|
# c.unicode_names = true
|
|
83
83
|
# end
|
|
84
84
|
module Ziperu
|
|
85
|
+
# Loaded on first use, so net/http is only required by those who read
|
|
86
|
+
# archives over HTTP.
|
|
87
|
+
autoload :RemoteIO, ::File.expand_path('ziperu/remote_io', __dir__)
|
|
88
|
+
|
|
85
89
|
# Used by the ported Deflater only; kept because rubyzip exposes it.
|
|
86
90
|
ZLIB_FLUSHING_STRATEGY = defined?(JRUBY_VERSION) ? Zlib::SYNC_FLUSH : Zlib::NO_FLUSH
|
|
87
91
|
|
metadata
CHANGED
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: ziperu
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.2.
|
|
4
|
+
version: 0.2.2
|
|
5
5
|
platform: aarch64-linux-gnu
|
|
6
6
|
authors:
|
|
7
7
|
- Maor Caplan
|
|
8
8
|
bindir: bin
|
|
9
9
|
cert_chain: []
|
|
10
|
-
date: 2026-11-
|
|
10
|
+
date: 2026-11-11 00:00:00.000000000 Z
|
|
11
11
|
dependencies: []
|
|
12
12
|
description: Zip reading and writing for Ruby with rubyzip's API under the Ziperu
|
|
13
13
|
namespace, implemented on the Rust `zip` crate via Magnus. `require "zip"` aliases
|
|
@@ -123,6 +123,8 @@ files:
|
|
|
123
123
|
- lib/ziperu/pass_thru_compressor.rb
|
|
124
124
|
- lib/ziperu/pass_thru_decompressor.rb
|
|
125
125
|
- lib/ziperu/rails_streaming.rb
|
|
126
|
+
- lib/ziperu/remote_io.rb
|
|
127
|
+
- lib/ziperu/replace_rubyzip.rb
|
|
126
128
|
- lib/ziperu/settings.rb
|
|
127
129
|
- lib/ziperu/size_estimator.rb
|
|
128
130
|
- lib/ziperu/streamable_directory.rb
|