ziperu 0.2.1-aarch64-linux-gnu → 0.2.2-aarch64-linux-gnu

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: bd39384d03192d315259a5e3a74c4248372f423e1636eea68a1e523fb08e50d8
4
- data.tar.gz: 7fe860a37687fe8b6ab221f8ab530aacc85a6f689b2e31077673a2d4dc6fc488
3
+ metadata.gz: 172d2af88e5afc2c160c2c3a2aed554b58fa99d0f1bae00f7d02483331b96d4d
4
+ data.tar.gz: 9c622be5788b3198c7c60a0037c1f956e765aad226656be3f7d8e48518bfab97
5
5
  SHA512:
6
- metadata.gz: a1e082ca4173854301348351ed0afae0992af17c252434cbe3de48e2ce12496a4d0ab16660e8f6b93d42c4b4544f8f861a1e781250458ba65345550b94626329
7
- data.tar.gz: 8951de42f8545a5ff2372aea892aa6d21218d12f3b317dbd8798d07a17b5ef4ff54c8a1941be5541aea19c3b873343ce9d51c93a44b7be46dd049ba9a816d4df
6
+ metadata.gz: 3298d688ef83f87ba627cbd7d82e095a599fdcca91320630ed96fa40a72334c048a4ae412ac80c709e7629e80f85a0270ff4e6e4403135f89b96ceebb6c1bfda
7
+ data.tar.gz: e53a1a4cd9472e94fb3f707b21727f0a4bfd576ed5f31788dfb42ede39d4226e9bd685c37d46e58a6b0792257a8983da02f83692ee520f638830ebf5e88eae1a
data/CHANGELOG.md CHANGED
@@ -1,5 +1,78 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.2.2 (2026-11-11)
4
+
5
+ Fixed:
6
+
7
+ - Duplicate entry names no longer get around the open-time limits. The
8
+ checks for `max_entries`, `max_total_size`, `max_entry_size` and
9
+ `max_compression_ratio` counted only the last record under each name,
10
+ while listing, `InputStream` and `extract_all` reach every one: a
11
+ zip bomb behind a harmless entry of the same name went through. Every
12
+ record now counts, in one pass.
13
+ - A large `read` interrupted (`Timeout`, `Thread#raise`) and then
14
+ rescued no longer loses data: what it had decompressed is returned by
15
+ the next read. After any other failure part-way through an entry,
16
+ further reads raise until `rewind`, instead of returning a short entry
17
+ with no error.
18
+ - `OutputStream.new(io, stream: true)` whose IO raises, with the error
19
+ rescued and writing carried on: the writer refuses every later call
20
+ rather than lose track of which entries the archive holds (which gave
21
+ a `NoMethodError` on close, or an entry silently overwritten), and no
22
+ longer writes again what it had handed the IO before it raised.
23
+ - `File#add`, `add_stored` and `replace` take a `Pathname` as the path it
24
+ is, not an IO (`Rails.root.join(...)` raised `NoMethodError`).
25
+ - On Windows, `File.open(dir, create: true)` and a commit no longer move
26
+ a directory at that path aside to put the archive in its place; the
27
+ commit raises, as rubyzip's does.
28
+ - Adding to an archive that is read-only (or has other hard links)
29
+ rewrites it instead of failing to append in place.
30
+ - Errors that came out as a bare `IOError` are `Ziperu::Error`s: corrupt
31
+ zstd and LZMA data (`DecompressionError`, as for Deflate), a local
32
+ header past the end of the archive or with a bad signature, a forged
33
+ zip64 offset no file can seek to. A NUL in a name raises
34
+ `ArgumentError` from the native `extract_all`, as from Ruby's own file
35
+ calls.
36
+ - `add_files(threads:)` (and the parallel commit) can be interrupted: it
37
+ stopped only once the whole batch, up to 256 MB, was compressed. It
38
+ still adds the batch whole or not at all.
39
+ - A local header that names a different compression method from the
40
+ central directory is refused, rather than decoded one way or the other
41
+ depending on `validate_entry_sizes`.
42
+ - An interrupted `extract_all` removes the file it was writing, so a
43
+ second run is not stopped by `DestinationExistsError`.
44
+ - Directory times restored by `extract_all` are set once the files in
45
+ the directory are written, which would otherwise move them (and every
46
+ directory entry is made before any file, wherever it is listed).
47
+ - An entry kept across a commit that moved it (the one `remove` returns,
48
+ say) still answers `time`, `extra` and the like from what it holds.
49
+ - `InputStream.open("missing.zip")` names the file in its `ENOENT`;
50
+ `File.open(nil)` raises `TypeError`.
51
+
52
+ Added:
53
+
54
+ - `require "ziperu/replace_rubyzip"`: ziperu answers for rubyzip
55
+ (`require "zip"`, `"rubyzip"`, `"zip/..."`) even without Bundler, where
56
+ RubyGems would otherwise load a rubyzip that a dependency asks for.
57
+ - `InputStream.open(pipe, streaming: true)`: a pipe or socket read in one
58
+ pass, local header by local header, in an entry's worth of memory, with
59
+ the limits counted as the bytes come. Entries whose sizes follow their
60
+ data (how streaming writers write) are read when Deflated.
61
+ - `Ziperu::RemoteIO`: a seekable IO over HTTP Range requests (or a block
62
+ that fetches byte ranges), so an archive on S3 or a web server can be
63
+ listed and have entries read without downloading it.
64
+ - `OutputStream#rollback_entry`: takes the entry being written back out
65
+ of the archive, so one whose data could not be had can be skipped. On
66
+ a stream it works until the entry's first bytes are written (the entry
67
+ is begun only then); on a file or seekable IO, at any point.
68
+ - `InputStream#seek` and `pos=` in a Stored entry that is not encrypted
69
+ (`get_input_stream(...).seek`): random access without reading what
70
+ comes before.
71
+
72
+ Changed:
73
+
74
+ - `open_buffer` on a String copies the archive once, not twice.
75
+
3
76
  ## 0.2.1 (2026-11-10)
4
77
 
5
78
  Fixed:
data/README.md CHANGED
@@ -165,6 +165,51 @@ Ziperu::InputStream.open(socket) do |zis| # any IO, no seeking needed
165
165
  process(entry.name, zis.read)
166
166
  end
167
167
  end
168
+
169
+ # A pipe or an upload read in one pass, an entry's worth of memory at a
170
+ # time, rather than taken in whole to reach the central directory:
171
+ Ziperu::InputStream.open(s3_object.body, streaming: true) do |zis|
172
+ while (entry = zis.get_next_entry)
173
+ zis.each_line { |line| ... }
174
+ end
175
+ end
176
+ ```
177
+
178
+ Read an archive in place on a server, without downloading it: only the
179
+ central directory and the entries read are fetched, with HTTP Range
180
+ requests (S3, GCS, a CDN; or a block that returns byte ranges):
181
+
182
+ ```ruby
183
+ io = Ziperu::RemoteIO.new(url) # "https://example.com/dataset.zip"
184
+ Ziperu::File.open_buffer(io, write: false) do |zf|
185
+ zf.extract("docs/guide.txt", "guide.txt") # a few requests, not 5 GB
186
+ end
187
+ ```
188
+
189
+ Leave out an entry whose data could not be had, and go on with the rest
190
+ (a download that failed half-way through a streamed zip, say):
191
+
192
+ ```ruby
193
+ body = Ziperu::OutputStream.body do |zos|
194
+ files.each do |name, url|
195
+ zos.put_next_entry(name)
196
+ begin
197
+ fetch(url) { |chunk| zos << chunk }
198
+ rescue Net::HTTPError
199
+ zos.rollback_entry # before its first byte on a stream; any time on a file
200
+ end
201
+ end
202
+ end
203
+ [200, { "content-type" => "application/zip" }, body]
204
+ ```
205
+
206
+ Seek about in a Stored entry, for readers that want random access (a PDF
207
+ stored in an archive):
208
+
209
+ ```ruby
210
+ Ziperu::File.open("scans.zip") do |zf|
211
+ zf.get_input_stream("scan.pdf") { |io| io.seek(-1024, IO::SEEK_END); io.read(1024) }
212
+ end
168
213
  ```
169
214
 
170
215
  Passwords, both ways:
@@ -209,11 +254,17 @@ path, so `require "zip"` loads ziperu and those gems run on it; rubyzip
209
254
  stays installed and unused. Checked with Bundler 2.3, 2.5 and 4.0, in
210
255
  either Gemfile order.
211
256
 
212
- > [!NOTE]
213
- > Without Bundler, RubyGems loads whichever gem asks for `zip` first, and
214
- > rubyzip and ziperu cannot both own the `Zip` constant. There, use
215
- > `require "ziperu"` and `Ziperu::` for your own code and leave `Zip::` to
216
- > rubyzip.
257
+ Without Bundler, RubyGems loads whichever gem asks for `zip` first. To
258
+ have ziperu answer for rubyzip there too, require this before anything
259
+ else loads it (the top of a script, or `config/boot.rb`):
260
+
261
+ ```ruby
262
+ require "ziperu/replace_rubyzip" # every require "zip" / "zip/..." loads ziperu
263
+ ```
264
+
265
+ It raises if rubyzip is already loaded, rather than mix the two. Short of
266
+ that, use `require "ziperu"` and `Ziperu::` for your own code and leave
267
+ `Zip::` to rubyzip.
217
268
 
218
269
  ziperu tracks the rubyzip 3.x API (`Ziperu::RUBYZIP_COMPAT_VERSION`), and
219
270
  diverges on purpose where rubyzip is lax: `InputStream` reads through the
Binary file
Binary file
Binary file
Binary file
Binary file
@@ -96,7 +96,7 @@ module Ziperu
96
96
  reopened = @owner.native_archive
97
97
  unless still_at_place?(reopened, offset)
98
98
  where = @index ? "at index #{@index} of" : 'in'
99
- raise Error, "entry '#{@raw_name}' is no longer #{where} #{@owner.name}"
99
+ raise EntryMovedError, "entry '#{@raw_name}' is no longer #{where} #{@owner.name}"
100
100
  end
101
101
 
102
102
  @native = reopened
data/lib/ziperu/entry.rb CHANGED
@@ -86,7 +86,14 @@ module Ziperu
86
86
  def extra
87
87
  if @source.stored? && !@local_extra_merged
88
88
  @local_extra_merged = true
89
- parsed_extra.merge(@source.local_extra(@local_header_offset), local: true)
89
+ begin
90
+ parsed_extra.merge(@source.local_extra(@local_header_offset), local: true)
91
+ rescue EntryMovedError
92
+ # Kept across a commit that moved its record (a removed entry,
93
+ # say): the central directory's fields are all there is now,
94
+ # which is what rubyzip reads anyway.
95
+ nil
96
+ end
90
97
  # Edits made through this object (`extra.merge`, `extra.delete`)
91
98
  # leave no other trace; the bytes are compared at commit.
92
99
  @extra_snapshot = extra_bytes
data/lib/ziperu/errors.rb CHANGED
@@ -156,6 +156,11 @@ module Ziperu
156
156
 
157
157
  # Raised by the Rust core where it lacks the domain object a rubyzip
158
158
  # exception wants; the Ruby layer re-raises the rubyzip-shaped error.
159
+ # An entry object outlived a commit that moved its record; what it holds
160
+ # from the central directory is still good, its local header is gone.
161
+ class EntryMovedError < Error # :nodoc:
162
+ end
163
+
159
164
  module Native
160
165
  class SizeLimitError < Ziperu::Error; end
161
166
  class PasswordError < Ziperu::Error; end
data/lib/ziperu/file.rb CHANGED
@@ -82,6 +82,11 @@ module Ziperu
82
82
  # decrypter on `InputStream`; `decrypter:` is accepted here too.
83
83
  @password = password || decrypter&.password
84
84
 
85
+ unless path_or_io.kind_of?(::String) || path_or_io.respond_to?(:to_path) ||
86
+ path_or_io.respond_to?(:read) || path_or_io.respond_to?(:write)
87
+ raise TypeError, "no implicit conversion of #{path_or_io.inspect} into String"
88
+ end
89
+
85
90
  @name = if IOExtras.path_like?(path_or_io)
86
91
  IOExtras.path_of(path_or_io)
87
92
  elsif path_or_io.respond_to?(:path)
@@ -312,8 +317,9 @@ module Ziperu
312
317
  compression_level: @compression_level, settings: @settings
313
318
  )
314
319
  end
315
- if src_path.respond_to?(:read)
320
+ if src_path.respond_to?(:read) && !IOExtras.path_like?(src_path)
316
321
  # An IO (File, StringIO, Tempfile, a socket): streamed at commit.
322
+ # A Pathname has `read` too, but names a file.
317
323
  new_entry.gather_fileinfo_from_io(src_path)
318
324
  else
319
325
  new_entry.gather_fileinfo_from_srcpath(src_path)
@@ -579,7 +585,7 @@ module Ziperu
579
585
  # the archive is rewritten.
580
586
  def appended_entries
581
587
  return nil if @create || @buffer || @native.nil? || @comment_changed || @settings.sort_entries
582
- return nil unless @settings.append_in_place
588
+ return nil unless @settings.append_in_place && appendable_in_place?
583
589
 
584
590
  all = @cdir.entries
585
591
  held = @native.entry_count
@@ -592,6 +598,18 @@ module Ziperu
592
598
  end
593
599
  private :appended_entries
594
600
 
601
+ # Whether the archive's file can take an append: one that is read-only
602
+ # can still be replaced (the rewrite renames a new file over it), and
603
+ # one with other hard links must be, or the append would change what
604
+ # they hold too.
605
+ def appendable_in_place?
606
+ stat = ::File.stat(@name)
607
+ stat.file? && stat.writable? && stat.nlink <= 1
608
+ rescue SystemCallError
609
+ false
610
+ end
611
+ private :appendable_in_place?
612
+
595
613
  # Writes the archive, with any changes, to `io` and returns it. `io`
596
614
  # may be a pipe, socket or response body: entries are streamed to it
597
615
  # as they are written, unless the archive holds ZipCrypto entries,
@@ -835,6 +853,10 @@ module Ziperu
835
853
  begin
836
854
  ::File.rename(tmp_filename, name)
837
855
  rescue Errno::EACCES
856
+ # A directory (or anything but a file) at the path is not ours to
857
+ # move: the commit fails, as rubyzip's does.
858
+ raise unless ::File.file?(name)
859
+
838
860
  # Something else (another Ziperu::File, say) still has the archive
839
861
  # open. Renaming an open file is allowed where replacing one is
840
862
  # not, so move the old archive aside first; that handle keeps
@@ -28,8 +28,20 @@ module Ziperu
28
28
  # `Ziperu.*` setting by its name, for this stream alone (the limits,
29
29
  # `validate_entry_sizes:`, `force_entry_names_encoding:`; see
30
30
  # Ziperu::Settings).
31
+ #
32
+ # `streaming: true` reads a pipe or socket (an upload, an S3 object's
33
+ # body) in one pass, entry by entry, rather than taking all of it in
34
+ # first to reach the central directory at its end: memory stays at an
35
+ # entry's worth, whatever the archive's size. What the stream gives is
36
+ # what the local headers hold: no comments, no permissions, and an
37
+ # entry whose sizes come after its data has them (`size`, `crc`) only
38
+ # once it has been read to its end. Such an entry must be Deflated,
39
+ # which is how streaming writers write; a Stored one cannot be told
40
+ # apart from what follows it in one pass, and raises. The limits apply
41
+ # to the bytes as they come. A file, a StringIO or a seekable IO is
42
+ # read through its central directory either way. Not in rubyzip.
31
43
  def initialize(context, *legacy, offset: 0, decrypter: nil, password: nil,
32
- enforce_size: nil, metadata_encoding: nil, **opts)
44
+ enforce_size: nil, metadata_encoding: nil, streaming: false, **opts)
33
45
  @settings = Settings.take!(opts)
34
46
  enforce_size = @settings.validate_entry_sizes if enforce_size.nil?
35
47
  @metadata_encoding = metadata_encoding
@@ -41,6 +53,19 @@ module Ziperu
41
53
  # AES from ZipCrypto by the entry, so one password serves both.
42
54
  decrypter ||= TraditionalDecrypter.new(password) if password
43
55
  super(**opts)
56
+ if streaming && self.class.pipe_like?(context)
57
+ IOExtras.binmode(context)
58
+ @stream = Native::StreamReader.open(context, decrypter&.password&.b, enforce_size)
59
+ @stream_entries = 0
60
+ @stream_total = 0
61
+ @decrypter = decrypter
62
+ @enforce_size = enforce_size
63
+ @owns_archive = false
64
+ @archive = nil
65
+ @reader = nil
66
+ @current_entry = nil
67
+ return
68
+ end
44
69
  @split = !context.kind_of?(Native::Archive) && self.class.split_archive?(context)
45
70
  # An archive handed in (by Entry#get_input_stream) stays open for its
46
71
  # owner; one opened here is closed with the stream. A split segment
@@ -85,6 +110,8 @@ module Ziperu
85
110
  def close
86
111
  @reader&.close
87
112
  @reader = nil
113
+ @stream&.close
114
+ @stream = nil
88
115
  @archive&.close if @owns_archive
89
116
  @archive = nil
90
117
  @open_error = nil
@@ -95,6 +122,8 @@ module Ziperu
95
122
  def get_next_entry
96
123
  raise SplitArchiveError if @split
97
124
  check_open!
125
+ return next_stream_entry if @stream
126
+
98
127
  return @current_entry = nil if @archive.nil? || @next_index.nil?
99
128
  if @next_index >= (@records ? @records.size : @archive.entry_count)
100
129
  # Past the last entry there is nothing to read, not the previous
@@ -154,6 +183,7 @@ module Ziperu
154
183
  def rewind
155
184
  check_open!
156
185
  return if @current_entry.nil?
186
+ raise Error, 'an entry read with streaming: true cannot be rewound' if @stream
157
187
 
158
188
  @lineno = 0
159
189
  @pos = 0
@@ -161,6 +191,39 @@ module Ziperu
161
191
  flush
162
192
  end
163
193
 
194
+ # Moves to byte `offset` of the current entry (from its start, its
195
+ # current position or its end, as `whence` says), as IO#seek does.
196
+ # Only a Stored entry that is not encrypted can be seeked in: its
197
+ # bytes are in the archive as they are. A compressed one raises
198
+ # Ziperu::Error; `rewind` is the way back to its start. Reads after a
199
+ # seek check no CRC, which only reading from the start can do.
200
+ # Not in rubyzip.
201
+ def seek(offset, whence = ::IO::SEEK_SET)
202
+ check_open!
203
+ raise Error, 'no entry to seek in' if @reader.nil?
204
+ raise Error, 'an entry read with streaming: true cannot be seeked in' if @stream
205
+
206
+ target =
207
+ case whence
208
+ when ::IO::SEEK_SET, :SET then offset
209
+ when ::IO::SEEK_CUR, :CUR then pos + offset
210
+ when ::IO::SEEK_END, :END then @current_entry.size + offset
211
+ else raise ArgumentError, "unknown whence: #{whence.inspect}"
212
+ end
213
+ raise Errno::EINVAL, "seek to #{target}" if target.negative?
214
+
215
+ wrap_errors { @reader.seek(target) }
216
+ flush
217
+ @pos = [target, @current_entry.size].min
218
+ 0
219
+ end
220
+
221
+ # IO#pos=, through #seek.
222
+ # Not in rubyzip.
223
+ def pos=(offset)
224
+ seek(offset)
225
+ end
226
+
164
227
  # Modelled after IO#sysread. Reads up to maxlen bytes from the stream,
165
228
  # bypassing the line buffer; returns a string (either a new string or the
166
229
  # given out_string).
@@ -202,20 +265,26 @@ module Ziperu
202
265
  check_limits(open_native(context, settings), settings)
203
266
  end
204
267
 
205
- # Ziperu.max_entries / max_total_size, checked before any entry is
206
- # read. An archive that fails is closed again.
268
+ # Ziperu.max_entries / max_total_size / max_entry_size /
269
+ # max_compression_ratio, checked before any entry is read, over every
270
+ # record in the central directory (several under one name included).
271
+ # An archive that fails is closed again.
207
272
  def check_limits(native, settings = ::Ziperu)
208
- if (limit = settings.max_entries) && native.entry_count > limit
209
- raise LimitError.new('an entry count', native.entry_count, limit)
273
+ limits = [settings.max_entries, settings.max_total_size,
274
+ settings.max_entry_size, settings.max_compression_ratio]
275
+ return native if limits.none?
276
+
277
+ count, total, largest, ratio = native.limits
278
+ if (limit = settings.max_entries) && count > limit
279
+ raise LimitError.new('an entry count', count, limit)
210
280
  end
211
- if (limit = settings.max_total_size) && (total = native.total_size) > limit
281
+ if (limit = settings.max_total_size) && total > limit
212
282
  raise LimitError.new('a declared total size', total, limit)
213
283
  end
214
- if (limit = settings.max_entry_size) && (largest = native.max_entry_size) > limit
284
+ if (limit = settings.max_entry_size) && largest > limit
215
285
  raise LimitError.new('an entry with a declared size', largest, limit)
216
286
  end
217
- if (limit = settings.max_compression_ratio) &&
218
- (ratio = native.max_compression_ratio) > limit
287
+ if (limit = settings.max_compression_ratio) && ratio > limit
219
288
  raise LimitError.new('an entry with a compression ratio', ratio.round(1), limit)
220
289
  end
221
290
 
@@ -229,6 +298,12 @@ module Ziperu
229
298
  IOExtras.path_like?(context)
230
299
  end
231
300
 
301
+ # A pipe, socket or the like: an IO that can only be read forward.
302
+ def pipe_like?(context) # :nodoc:
303
+ !context.kind_of?(Native::Archive) && !path_like?(context) &&
304
+ !context.kind_of?(::StringIO) && context.respond_to?(:read) && !IOExtras.seekable?(context)
305
+ end
306
+
232
307
  # Whether what `context` holds begins with a local file header, the
233
308
  # one thing rubyzip's `InputStream` needs to yield entries.
234
309
  def archive_like?(context) # :nodoc:
@@ -251,7 +326,15 @@ module Ziperu
251
326
  end
252
327
 
253
328
  def open_native(context, settings = ::Ziperu)
254
- return Native::Archive.open_path(path_of(context)) if path_like?(context)
329
+ if path_like?(context)
330
+ path = path_of(context)
331
+ begin
332
+ return Native::Archive.open_path(path)
333
+ rescue Errno::ENOENT
334
+ # The native message names no file; Ruby's own does.
335
+ raise Errno::ENOENT, path
336
+ end
337
+ end
255
338
 
256
339
  # A StringIO's bytes are already in memory: snapshot them once rather
257
340
  # than paying a method call per chunk (12x faster per entry).
@@ -338,6 +421,7 @@ module Ziperu
338
421
  end
339
422
  raise ArgumentError, "negative length #{maxlen} given" if maxlen&.negative?
340
423
  return nil if @reader.nil?
424
+ return produce_stream_input(maxlen) if @stream
341
425
 
342
426
  wrap_errors { @reader.read(maxlen) }
343
427
  end
@@ -363,7 +447,79 @@ module Ziperu
363
447
  # archive (`@open_error` holds why), which yields no entries but is
364
448
  # not closed.
365
449
  def closed?
366
- @archive.nil? && @open_error.nil?
450
+ @archive.nil? && @open_error.nil? && @stream.nil?
451
+ end
452
+
453
+ # `get_next_entry` with `streaming: true`: the next local header.
454
+ def next_stream_entry
455
+ meta = wrap_errors { @stream.next_entry }
456
+ @reader = nil
457
+ flush
458
+ return @current_entry = nil unless meta
459
+
460
+ @stream_entries += 1
461
+ if (limit = @settings.max_entries) && @stream_entries > limit
462
+ raise LimitError.new('an entry count', @stream_entries, limit)
463
+ end
464
+ if meta[:size]
465
+ check_stream_entry_size(meta[:size])
466
+ if (limit = @settings.max_compression_ratio) && meta[:size].positive?
467
+ ratio = meta[:compressed_size].zero? ? Float::INFINITY : meta[:size].fdiv(meta[:compressed_size])
468
+ raise LimitError.new('an entry with a compression ratio', ratio.round(1), limit) if ratio > limit
469
+ end
470
+ end
471
+ # Unknown until the entry has been read (see `produce_stream_input`).
472
+ meta[:crc] ||= 0
473
+ meta[:compressed_size] ||= 0
474
+ @current_entry = Entry.from_meta(nil, meta, metadata_encoding: @metadata_encoding, settings: @settings)
475
+ @current_entry.instance_variable_set(:@size, nil) if meta[:size].nil?
476
+ @stream_entry_bytes = 0
477
+ @reader = @stream
478
+ @lineno = 0
479
+ @pos = 0
480
+ @current_entry
481
+ end
482
+
483
+ # Reads from the stream, counting against the limits as the bytes
484
+ # come; nil `maxlen` reads to the end of the entry, in chunks.
485
+ def produce_stream_input(maxlen)
486
+ return stream_chunk(maxlen) if maxlen
487
+
488
+ out = +''.b
489
+ while (chunk = stream_chunk(CHUNK_SIZE))
490
+ out << chunk
491
+ end
492
+ out.empty? ? nil : out
493
+ end
494
+
495
+ def stream_chunk(length)
496
+ chunk = wrap_errors { @stream.read(length) }
497
+ if chunk
498
+ @stream_entry_bytes += chunk.bytesize
499
+ @stream_total += chunk.bytesize
500
+ check_stream_entry_size(@stream_entry_bytes)
501
+ if (limit = @settings.max_total_size) && @stream_total > limit
502
+ raise LimitError.new('a total size', @stream_total, limit)
503
+ end
504
+ end
505
+ learn_stream_trailer if @stream.eof?
506
+ chunk
507
+ end
508
+
509
+ def check_stream_entry_size(size)
510
+ return unless (limit = @settings.max_entry_size) && size > limit
511
+
512
+ raise LimitError.new('an entry with a size', size, limit)
513
+ end
514
+
515
+ # Once a streamed entry has been read to its end: what its data came
516
+ # to, for an entry whose header left it to the descriptor.
517
+ def learn_stream_trailer
518
+ crc, size = @stream.trailer
519
+ return unless crc && !@current_entry.size_known?
520
+
521
+ @current_entry.instance_variable_set(:@size, size)
522
+ @current_entry.instance_variable_set(:@crc, crc)
367
523
  end
368
524
 
369
525
  # Re-raise the native core's errors as the rubyzip-shaped ones.
@@ -82,6 +82,12 @@ module Ziperu
82
82
  @closed = false
83
83
  @current_entry = nil
84
84
  @written = []
85
+ # On a stream, a file entry is begun in the native writer only when
86
+ # its first bytes are written (or the next entry, or the end, comes):
87
+ # until then nothing of it has gone out, so `rollback_entry` can
88
+ # still take it back. Its name and options wait here.
89
+ @deferred = nil
90
+ @stream_names = {} if @streamed
85
91
  end
86
92
 
87
93
  class << self
@@ -224,6 +230,42 @@ module Ziperu
224
230
  @current_entry = new_entry
225
231
  end
226
232
 
233
+ # Takes the entry being written back out of the archive, as if it had
234
+ # never been put: the central directory will not list it, and the
235
+ # next entry goes where it was. For an entry whose data could not be
236
+ # had (a download that failed, say), so the rest of the archive can
237
+ # still be written:
238
+ #
239
+ # files.each do |name, url|
240
+ # zos.put_next_entry(name)
241
+ # begin
242
+ # fetch(url) { |chunk| zos << chunk }
243
+ # rescue Net::HTTPError
244
+ # zos.rollback_entry
245
+ # end
246
+ # end
247
+ #
248
+ # On a stream to an IO that cannot seek (`body`, `write_buffer` to a
249
+ # pipe) bytes already sent cannot be taken back, so there it works
250
+ # until the entry's first bytes are written, and raises after.
251
+ # Returns the entry taken back.
252
+ # Not in rubyzip.
253
+ def rollback_entry
254
+ check_open!
255
+ entry = @current_entry
256
+ raise Error, 'there is no entry being written to take back' if entry.nil? || !@written.last.equal?(entry)
257
+
258
+ if @deferred
259
+ @deferred = nil
260
+ else
261
+ @writer.rollback_entry
262
+ end
263
+ @written.pop
264
+ @stream_names&.delete(entry.name.b)
265
+ @current_entry = nil
266
+ entry
267
+ end
268
+
227
269
  # Bytes handed to the IO so far, for a stream to an IO that cannot
228
270
  # seek (`write_buffer` to a pipe or socket, `body`): what the consumer
229
271
  # has been given, for a progress report alongside
@@ -320,6 +362,8 @@ module Ziperu
320
362
  { dos_date: time.to_binary_dos_date, dos_time: time.to_binary_dos_time,
321
363
  unix_perms: entry.unix_perms }
322
364
  end
365
+ start_deferred
366
+ @stream_names[(new_name || entry.name).to_s.b] = true if @stream_names
323
367
  @writer.raw_copy(entry.archive, entry.index, new_name, touch_opts)
324
368
  @written << entry
325
369
  @current_entry = nil
@@ -347,6 +391,7 @@ module Ziperu
347
391
  raise Error, "cannot write data to directory entry '#{@current_entry.name}'"
348
392
  end
349
393
 
394
+ start_deferred
350
395
  @writer.write(data.to_s.b)
351
396
  self
352
397
  end
@@ -362,6 +407,7 @@ module Ziperu
362
407
 
363
408
  @writer.comment = comment
364
409
  end
410
+ start_deferred
365
411
  @current_entry = nil
366
412
  begin
367
413
  bytes, stats = @writer.finish
@@ -372,6 +418,10 @@ module Ziperu
372
418
  "the IO given does not allow (#{e.message}); open it for reading and writing"
373
419
  end
374
420
  unless @aborted
421
+ if stats.size != @written.size
422
+ raise Error, "the archive holds #{stats.size} entries where #{@written.size} were written"
423
+ end
424
+
375
425
  stats.each_with_index do |(offset, size, compressed_size, crc), i|
376
426
  @written[i].written_as(offset, size, compressed_size, crc)
377
427
  end
@@ -388,19 +438,40 @@ module Ziperu
388
438
  raise ::Ziperu::CompressionMethodError, entry.compression_method
389
439
  end
390
440
 
441
+ start_deferred
391
442
  opts = native_entry_options(entry)
443
+ if @stream_names
444
+ # The native writer would catch a second entry under a name only
445
+ # once it begins it, which on a stream is later (see @deferred).
446
+ raise EntryExistsError.new('put_next_entry', entry.name) if @stream_names.key?(entry.name.b)
447
+
448
+ @stream_names[entry.name.b] = true
449
+ end
392
450
  if entry.directory?
393
451
  @writer.add_directory(entry.name, opts)
394
452
  elsif entry.symlink?
395
453
  @writer.add_symlink(entry.name, entry.symlink_target, opts)
396
454
  else
397
455
  opts.merge!(@encrypter.native_options)
398
- @writer.start_entry(entry.name.b, opts)
456
+ if @streamed
457
+ @deferred = [entry.name.b, opts]
458
+ else
459
+ @writer.start_entry(entry.name.b, opts)
460
+ end
399
461
  end
400
462
  rescue Native::DuplicateEntryError
401
463
  raise EntryExistsError.new('put_next_entry', entry.name)
402
464
  end
403
465
 
466
+ # Begins in the native writer the entry `init_next_entry` deferred.
467
+ def start_deferred
468
+ name, opts = @deferred
469
+ return unless name
470
+
471
+ @deferred = nil
472
+ @writer.start_entry(name, opts)
473
+ end
474
+
404
475
  def check_open!
405
476
  raise Error, 'zip stream is closed' if @closed
406
477
  end
@@ -0,0 +1,278 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'net/http'
4
+ require 'uri'
5
+
6
+ module Ziperu
7
+ # A read-only, seekable IO over an archive somewhere else: an HTTP(S)
8
+ # URL that answers Range requests (S3, GCS, a CDN, most file servers),
9
+ # or anything a block can fetch byte ranges from. Opening an archive
10
+ # through it reads only what is needed: the end of the file (its
11
+ # central directory) in one request, then each entry's bytes as they
12
+ # are read, so listing a 5 GB archive or pulling one file out of it
13
+ # costs a few requests and a few megabytes.
14
+ #
15
+ # io = Ziperu::RemoteIO.new("https://example.com/big.zip")
16
+ # Ziperu::File.open_buffer(io, write: false) do |zf|
17
+ # zf.extract("docs/readme.txt", "readme.txt")
18
+ # end
19
+ #
20
+ # S3 refuses HEAD to some credentials, so the size comes from the first
21
+ # ranged GET (a presigned URL works as is). With the AWS SDK instead:
22
+ #
23
+ # s3 = Aws::S3::Client.new
24
+ # size = s3.head_object(bucket: b, key: k).content_length
25
+ # io = Ziperu::RemoteIO.new(size: size) do |offset, length|
26
+ # s3.get_object(bucket: b, key: k, range: "bytes=#{offset}-#{offset + length - 1}").body.read
27
+ # end
28
+ #
29
+ # Reads go in blocks of `block_size` bytes (adjacent missing blocks in
30
+ # one request); the last `cache_blocks` of them are kept. Not thread
31
+ # safe: one RemoteIO per thread.
32
+ #
33
+ # Not in rubyzip.
34
+ class RemoteIO
35
+ # Raised when the server or the block does not give the bytes asked
36
+ # for.
37
+ class Error < ::Ziperu::Error; end
38
+
39
+ DEFAULT_BLOCK_SIZE = 256 * 1024
40
+ DEFAULT_CACHE_BLOCKS = 64
41
+ # The first request asks for this much of the end: the end of central
42
+ # directory record, and for most archives the whole central directory.
43
+ TAIL_SIZE = 64 * 1024
44
+ MAX_REDIRECTS = 5
45
+
46
+ attr_reader :size, :pos, :requests
47
+ alias tell pos
48
+ alias length size
49
+
50
+ # `url` (a String or URI) to fetch over HTTP(S) with `headers` sent on
51
+ # every request; or, with no url, `size:` and a block that returns
52
+ # `length` bytes from `offset`.
53
+ def initialize(url = nil, headers: {}, size: nil, block_size: DEFAULT_BLOCK_SIZE,
54
+ cache_blocks: DEFAULT_CACHE_BLOCKS, open_timeout: 10, read_timeout: 60, &fetch)
55
+ raise ArgumentError, 'give a url or a block, not both' if url && fetch
56
+ raise ArgumentError, 'give a url, or size: and a block' unless url || (fetch && size)
57
+ raise ArgumentError, 'block_size must be positive' unless block_size.positive?
58
+
59
+ @block_size = block_size
60
+ @cache_blocks = [cache_blocks, 1].max
61
+ @blocks = {}
62
+ @tail_start = nil
63
+ @tail = nil
64
+ @pos = 0
65
+ @requests = 0
66
+ @closed = false
67
+ if url
68
+ @uri = URI(url.to_s)
69
+ @headers = headers
70
+ @open_timeout = open_timeout
71
+ @read_timeout = read_timeout
72
+ @size = discover_size
73
+ else
74
+ @fetch = fetch
75
+ @size = Integer(size)
76
+ end
77
+ end
78
+
79
+ def read(length = nil, outbuf = nil)
80
+ check_open
81
+ if length.nil?
82
+ data = take(@size - @pos)
83
+ else
84
+ raise ArgumentError, "negative length #{length} given" if length.negative?
85
+ return +''.b if length.zero?
86
+ return nil if @pos >= @size
87
+
88
+ data = take([length, @size - @pos].min)
89
+ end
90
+ outbuf ? outbuf.replace(data) : data
91
+ end
92
+
93
+ def seek(offset, whence = ::IO::SEEK_SET)
94
+ check_open
95
+ target =
96
+ case whence
97
+ when ::IO::SEEK_SET, :SET then offset
98
+ when ::IO::SEEK_CUR, :CUR then @pos + offset
99
+ when ::IO::SEEK_END, :END then @size + offset
100
+ else raise ArgumentError, "unknown whence: #{whence.inspect}"
101
+ end
102
+ raise Errno::EINVAL, "seek to #{target}" if target.negative?
103
+
104
+ @pos = target
105
+ 0
106
+ end
107
+
108
+ def pos=(offset)
109
+ seek(offset)
110
+ end
111
+
112
+ def rewind
113
+ seek(0)
114
+ end
115
+
116
+ def eof?
117
+ @pos >= @size
118
+ end
119
+ alias eof eof?
120
+
121
+ def binmode
122
+ self
123
+ end
124
+
125
+ def close
126
+ @closed = true
127
+ @http&.finish if @http&.started?
128
+ @http = nil
129
+ @blocks.clear
130
+ nil
131
+ end
132
+
133
+ def closed?
134
+ @closed
135
+ end
136
+
137
+ private
138
+
139
+ def check_open
140
+ raise IOError, 'closed stream' if @closed
141
+ end
142
+
143
+ # `count` bytes from the position, which moves past them.
144
+ def take(count)
145
+ return +''.b if count <= 0
146
+
147
+ out = +''.b
148
+ at = @pos
149
+ finish = at + count
150
+ while at < finish
151
+ if @tail_start && at >= @tail_start
152
+ piece = @tail.byteslice(at - @tail_start, finish - at)
153
+ else
154
+ index = at / @block_size
155
+ load_blocks(index, ((finish - 1) / @block_size))
156
+ block = @blocks[index]
157
+ offset = at - (index * @block_size)
158
+ piece = block.byteslice(offset, finish - at)
159
+ # Most recently used last, so the first is the one to drop.
160
+ @blocks[index] = @blocks.delete(index)
161
+ end
162
+ raise Error, "no data at offset #{at}" if piece.nil? || piece.empty?
163
+
164
+ out << piece
165
+ at += piece.bytesize
166
+ end
167
+ @pos = finish
168
+ out
169
+ end
170
+
171
+ # Makes sure blocks `first..last` are cached, fetching each run of
172
+ # missing ones in one request.
173
+ def load_blocks(first, last)
174
+ last = [last, first + @cache_blocks - 1].min
175
+ index = first
176
+ while index <= last
177
+ if @blocks.key?(index)
178
+ index += 1
179
+ next
180
+ end
181
+ run_end = index
182
+ run_end += 1 while run_end < last && !@blocks.key?(run_end + 1)
183
+ offset = index * @block_size
184
+ length = [((run_end + 1) * @block_size), @size].min - offset
185
+ data = fetch(offset, length)
186
+ (index..run_end).each_with_index do |i, n|
187
+ @blocks[i] = data.byteslice(n * @block_size, @block_size)
188
+ end
189
+ @blocks.shift while @blocks.size > @cache_blocks
190
+ index = run_end + 1
191
+ end
192
+ end
193
+
194
+ def fetch(offset, length)
195
+ @requests += 1
196
+ data =
197
+ if @fetch
198
+ @fetch.call(offset, length).to_s.b
199
+ else
200
+ http_range("bytes=#{offset}-#{offset + length - 1}", offset)
201
+ end
202
+ unless data.bytesize == length
203
+ raise Error, "asked for #{length} bytes at offset #{offset}, got #{data.bytesize}"
204
+ end
205
+
206
+ data
207
+ end
208
+
209
+ # The size, from the first request: the last TAIL_SIZE bytes (a
210
+ # suffix range), whose Content-Range carries the total. They are kept,
211
+ # being what opening an archive reads first.
212
+ def discover_size
213
+ @requests += 1
214
+ response = request("bytes=-#{TAIL_SIZE}")
215
+ case response
216
+ when Net::HTTPPartialContent
217
+ total = response['content-range'].to_s[%r{/(\d+)\z}, 1]
218
+ raise Error, "#{@uri} answered without a usable Content-Range" unless total
219
+
220
+ body = response.body.to_s.b
221
+ @tail_start = Integer(total) - body.bytesize
222
+ @tail = body
223
+ Integer(total)
224
+ when Net::HTTPRequestedRangeNotSatisfiable
225
+ # An empty file has no last bytes.
226
+ 0
227
+ when Net::HTTPOK
228
+ # The server ignores Range: the whole file came back; keep it.
229
+ body = response.body.to_s.b
230
+ @tail_start = 0
231
+ @tail = body
232
+ body.bytesize
233
+ else
234
+ raise Error, "#{@uri} answered #{response.code} #{response.message}"
235
+ end
236
+ end
237
+
238
+ def http_range(range, offset)
239
+ response = request(range)
240
+ unless response.kind_of?(Net::HTTPPartialContent)
241
+ raise Error, "#{@uri} answered #{response.code} to a Range request at offset #{offset}"
242
+ end
243
+
244
+ response.body.to_s.b
245
+ end
246
+
247
+ # A GET with `Range: range`, following redirects (the URL redirected
248
+ # to is used from then on, as a presigned S3 link redirects once).
249
+ def request(range, redirects = MAX_REDIRECTS)
250
+ http = connection
251
+ get = Net::HTTP::Get.new(@uri)
252
+ @headers.each { |k, v| get[k.to_s] = v.to_s }
253
+ get['range'] = range
254
+ get['accept-encoding'] = 'identity'
255
+ response = http.request(get)
256
+ return response unless response.kind_of?(Net::HTTPRedirection)
257
+ raise Error, "too many redirects from #{@uri}" if redirects.zero?
258
+
259
+ location = URI.join(@uri.to_s, response['location'].to_s)
260
+ if location.host != @uri.host || location.port != @uri.port || location.scheme != @uri.scheme
261
+ @http&.finish if @http&.started?
262
+ @http = nil
263
+ end
264
+ @uri = location
265
+ request(range, redirects - 1)
266
+ end
267
+
268
+ def connection
269
+ return @http if @http&.started?
270
+
271
+ @http = Net::HTTP.new(@uri.host, @uri.port)
272
+ @http.use_ssl = @uri.scheme == 'https'
273
+ @http.open_timeout = @open_timeout
274
+ @http.read_timeout = @read_timeout
275
+ @http.start
276
+ end
277
+ end
278
+ end
@@ -0,0 +1,67 @@
1
+ # frozen_string_literal: true
2
+
3
+ # Makes ziperu answer for rubyzip in this process, whatever else is
4
+ # installed:
5
+ #
6
+ # # config/boot.rb, or the first line of a script
7
+ # require "ziperu/replace_rubyzip"
8
+ #
9
+ # From then on `require "zip"`, `require "rubyzip"` and every
10
+ # `require "zip/..."` load ziperu's files, so `Zip` is `Ziperu` and the
11
+ # gems written against rubyzip (roo, caxlsx, rubyXL, ...) run on it.
12
+ #
13
+ # Under Bundler this is what already happens, since ziperu's `lib` comes
14
+ # first on the load path. This file is for everywhere else: a script
15
+ # without Bundler, where RubyGems would activate the rubyzip gem a
16
+ # dependency asks for and put its files ahead of ziperu's on the load
17
+ # path. It must be required before anything loads rubyzip; if rubyzip is
18
+ # already loaded it raises LoadError rather than mix the two.
19
+ #
20
+ # Not in rubyzip.
21
+
22
+ module Ziperu
23
+ # The `require` that sends rubyzip's files to ziperu's. Not in rubyzip.
24
+ module ReplaceRubyzip
25
+ # ziperu's own `lib`, where `zip.rb`, `rubyzip.rb` and `zip/` live.
26
+ LIB = ::File.expand_path('..', __dir__)
27
+
28
+ # The ziperu file that stands in for the rubyzip feature `name`, if
29
+ # there is one.
30
+ def self.substitute(name)
31
+ name = name.to_s
32
+ name = name.delete_suffix('.rb')
33
+ return unless name == 'zip' || name == 'rubyzip' || name.start_with?('zip/')
34
+
35
+ path = ::File.join(LIB, "#{name}.rb")
36
+ path if ::File.file?(path)
37
+ end
38
+
39
+ # Whether rubyzip itself has already been loaded: a `Zip` that is not
40
+ # ziperu's, or one of rubyzip's files among the loaded features.
41
+ def self.rubyzip_loaded?
42
+ return true if defined?(::Zip) && !(defined?(::Ziperu::File) && ::Zip.equal?(::Ziperu))
43
+
44
+ ours = "#{LIB}/"
45
+ $LOADED_FEATURES.any? do |feature|
46
+ !feature.start_with?(ours) && feature.match?(%r{/(zip|rubyzip)\.rb\z|/zip/[^/]+\.rb\z}) &&
47
+ feature.include?('rubyzip')
48
+ end
49
+ end
50
+
51
+ private
52
+
53
+ def require(name)
54
+ path = ReplaceRubyzip.substitute(name)
55
+ path ? super(path) : super
56
+ end
57
+ end
58
+ end
59
+
60
+ if Ziperu::ReplaceRubyzip.rubyzip_loaded?
61
+ raise LoadError, 'rubyzip is already loaded, so ziperu cannot stand in for it: require ' \
62
+ '"ziperu/replace_rubyzip" before anything that loads rubyzip (first thing ' \
63
+ 'in config/boot.rb, or at the top of the script)'
64
+ end
65
+
66
+ Kernel.prepend(Ziperu::ReplaceRubyzip)
67
+ require 'zip'
@@ -1,7 +1,7 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Ziperu
4
- VERSION = "0.2.1"
4
+ VERSION = "0.2.2"
5
5
  # The rubyzip API this gem tracks.
6
6
  RUBYZIP_COMPAT_VERSION = "3.7"
7
7
  end
data/lib/ziperu.rb CHANGED
@@ -82,6 +82,10 @@ require_relative "ziperu/size_estimator"
82
82
  # c.unicode_names = true
83
83
  # end
84
84
  module Ziperu
85
+ # Loaded on first use, so net/http is only required by those who read
86
+ # archives over HTTP.
87
+ autoload :RemoteIO, ::File.expand_path('ziperu/remote_io', __dir__)
88
+
85
89
  # Used by the ported Deflater only; kept because rubyzip exposes it.
86
90
  ZLIB_FLUSHING_STRATEGY = defined?(JRUBY_VERSION) ? Zlib::SYNC_FLUSH : Zlib::NO_FLUSH
87
91
 
metadata CHANGED
@@ -1,13 +1,13 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: ziperu
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.1
4
+ version: 0.2.2
5
5
  platform: aarch64-linux-gnu
6
6
  authors:
7
7
  - Maor Caplan
8
8
  bindir: bin
9
9
  cert_chain: []
10
- date: 2026-11-10 00:00:00.000000000 Z
10
+ date: 2026-11-11 00:00:00.000000000 Z
11
11
  dependencies: []
12
12
  description: Zip reading and writing for Ruby with rubyzip's API under the Ziperu
13
13
  namespace, implemented on the Rust `zip` crate via Magnus. `require "zip"` aliases
@@ -123,6 +123,8 @@ files:
123
123
  - lib/ziperu/pass_thru_compressor.rb
124
124
  - lib/ziperu/pass_thru_decompressor.rb
125
125
  - lib/ziperu/rails_streaming.rb
126
+ - lib/ziperu/remote_io.rb
127
+ - lib/ziperu/replace_rubyzip.rb
126
128
  - lib/ziperu/settings.rb
127
129
  - lib/ziperu/size_estimator.rb
128
130
  - lib/ziperu/streamable_directory.rb