zip_tricks 5.1.0 → 5.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 983b83d36f88a25db8276a74e45d506f801ac3ecdd64197fd6fbc98bdb39cb03
-  data.tar.gz: c74a8a3936f4afb1f71ad7775025b73ccd04d2db309df8942a20ff64e82cea81
+  metadata.gz: 05e8eea8ecf1ad0b9b9cb132c54dde1abd60dc003afd1e5a1ce989786ce89608
+  data.tar.gz: fbdc2172fc3becefa4dd8720713acb11d838900465ac3cb42de38fec76fd0d90
 SHA512:
-  metadata.gz: 2b6eed391cd16c4077003535b0d7fb52e6930f351b230d31ba68aab2f05905df3987565aad76741083e888a91d054eaff3beab507117088b812b6c95f9ecf4f3
-  data.tar.gz: 9c4ad53ef462d28f8cdffcc439200788cb260e9d1b35ee231aed3b7979630d0c31881db95198a944c375c836678063e0db187fbfb3c2672819ad5e06d1023dc9
+  metadata.gz: 449c59e898d2b54a089b60d7aebe7633d9f65ad64a5ce014a7a44e1683f6d6cec4997f732fd06746edeec96594d5041b68efcf19571ef92c9798acc05ffda7bd
+  data.tar.gz: 5ed26109e12373acfb9866531ef03c4302141bbb8a312f759ff768295c8d7926f650745f3f2624ad652333e57cd5bc5f98f932ace41504aeca45a668ef7a4f4a

data/.rubocop.yml

@@ -11,3 +11,5 @@ Style/GlobalVars:
       - qa/*.rb
       - spec/spec_helper.rb
       - spec/support/zip_inspection.rb
+Layout/IndentHeredoc:
+  Enabled: false

data/.travis.yml

@@ -4,8 +4,5 @@ rvm:
 - jruby-9.0
 sudo: false
 cache: bundler
-matrix:
-  allow_failures:
-    - rvm: jruby-9.0
 script:
   - bundle exec rake

data/CHANGELOG.md

@@ -1,3 +1,39 @@
+## 5.4.0
+
+* Use block form for zlib Deflater calls to conserve memory
+* Do not change string encoding in writer wrappers (avoid extra work)
+* Fix a zlib deflater object being leaked per archived file
+* Speed up streaming CRC32 computation
+* When running tests, assign the port for the Puma server dynamically
+* Reduce string allocations in the block deflate spec
+* Make sure RemoteUncap specs run under JRuby correctly
+* Replace Rails::Live streaming with iterable body streaming to avoid issues with Rails::Live across the board
+* Remove `qa/` directory and scripts, as the tests for the library proper should now be sufficient
+* Fix some documentation and sample code omissions and inconsistencies.
+
+## 5.3.1
+
+* Fix extended timestamp timestamp value encoding. Previously we would use an incorrect encoding for the timestamp value, which would output correct but nonsensical timestamps. The pack specifier is now changed to output the correct value.
+
+## 5.3.0
+
+* Raise in `Streamer#close` when the IO offset of the Streamer does not match the size of the written entries. This is a situation which
+  can occur if one adds the local headers, writes the bodies of the files to the socket/output directly, and forgets to adjust the internal
+  Streamer offset. The unadjusted offset would then produce incorrect values in both the local headers which come after the missing
+  offset adjustment _and_ in the central directory headers. Some ZIP unarchivers are able to recover from this (ones that read
+  files "straight-ahead" but others aren't - if the ZIP unarchiver uses central directory entries it would be using incorrect offsets.
+  Instead of producing an invalid ZIP, raise an exception which explains what happened and how it can be resolved.
+
+## 5.2.0
+
+* Remove `Streamer#add_compressed_entry` and `SizeEstimator#add_compressed_entry`
+
+## 5.1.1
+
+* Fix extended timestamp extra field output. The first bit of the flag would be set instead of the last bit of
+  the flag, which made it impossible for Rubyzip to read the timestamp of the entry - and it would also make
+  the extra field useless for most reading applications.
+
 ## 5.1.0
 
 * Slightly rework `RemoteIO` and `RemoteUncap` and make sure they work correctly by spinning up a test webserver
@@ -25,16 +61,16 @@
 
 ## 4.7.4
 
-* Use a single fixed capacity string in StreamCRC32.from_io to avoid unnecessary allocations
+* Use a single fixed capacity string in `StreamCRC32.from_io` to avoid unnecessary allocations
 * Fix a few tests that were calling out to external binaries
 
 ## 4.7.3
 
-* Fix RemoteUncap#request_object_size to function correctly
+* Fix `RemoteUncap#request_object_size` to function correctly
 
 ## 4.7.2
 
-* Relax bundler dependency so that both 1.x and 2.x are supported cleanly
+* Relax bundler dependency so that both bundler 1.x and 2.x are supported cleanly
 
 ## 4.7.1
 

data/README.md

@@ -24,11 +24,11 @@ to [32 bit sizes.](https://github.com/jruby/jruby/issues/3817)
 
 ## Diving in: send some large CSV reports from Rails
 
-The easiest is to use the Rails' built-in streaming feature:
+The easiest is to include the `ZipTricks::RailsStreaming` module into your
+controller.
 
 ```ruby
 class ZipsController < ActionController::Base
-  include ActionController::Live # required for streaming
   include ZipTricks::RailsStreaming
 
   def download
@@ -49,6 +49,10 @@ class ZipsController < ActionController::Base
 end
 ```
 
+If you want some more conveniences you can also use [zipline](https://github.com/fringd/zipline) which
+will automatically process and stream attachments (Carrierwave, Shrine, ActiveStorage) and remote objects
+via HTTP.
+
 ## Create a ZIP file without size estimation, compress on-the-fly during writes
 
 Basic use case is compressing on the fly. Some data will be buffered by the Zlib deflater, but
@@ -84,7 +88,7 @@ body = ZipTricks::RackBody.new do | zip |
     File.open('novel.txt', 'rb'){|source| IO.copy_stream(source, sink) }
   end
 end
-[200, {'Transfer-Encoding' => 'chunked'}, body]
+[200, {}, body]
 ```
 
 ## Send a ZIP file of known size, with correct headers
@@ -174,9 +178,12 @@ that have not been formally verified (ours hasn't been).
 * Commit and push until you are happy with your contribution.
 * Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
 * Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
-* If you alter the `ZipWriter`, please take the time to run the test in the `qa/` directory. Ensure that the generated (large) files open manually - see README_QA for more.
 
-## Copyright
+## Copyright and license
+
+Copyright (c) 2020 WeTransfer.
 
-Copyright (c) 2019 WeTransfer. `zip_tricks` is distributed under the conditions of the [Hippocratic License](https://firstdonoharm.dev/version/1/2/license.html)
- - See LICENSE.txt for further details.
+`zip_tricks` is distributed under the conditions of the [Hippocratic License](https://firstdonoharm.dev/version/1/2/license.html)
+See LICENSE.txt for further details. If this license is not acceptable for your use case we still maintain the 4.x version tree
+which remains under the MIT license, see https://rubygems.org/gems/zip_tricks/versions for more information.
+Note that we only backport some performance optimizations and crucial bugfixes but not the new features to that tree.

data/lib/zip_tricks/block_write.rb

@@ -1,11 +1,25 @@
 # frozen_string_literal: true
 
-# Stashes a block given by the Rack webserver when calling each() on a body, and calls
-# that block every time it is written to using :<< (shovel). Poses as an IO for rubyzip.
-
+# Acts as a converter between callers which send data to the `#<<` method (such as all the ZipTricks
+# writer methods, which push onto anything), and a given block. Every time `#<<` gets called on the BlockWrite,
+# the block given to the constructor will be called with the same argument. ZipTricks uses this object
+# when integrating with Rack and in the OutputEnumerator. Normally you wouldn't need to use it manually but
+# you always can. BlockWrite will also ensure the binary string encoding is forced onto any string
+# that passes through it.
+#
+# For example, you can create a Rack response body like so:
+#
+#     class MyRackResponse
+#       def each
+#         writer = ZipTricks::BlockWrite.new {|chunk| yield(chunk) }
+#         writer << "Hello" << "world" << "!"
+#       end
+#     end
+#     [200, {}, MyRackResponse.new]
 class ZipTricks::BlockWrite
-  # The block is the block given to each() of the Rack body, or other block you want
-  # to receive the string chunks written by the zip compressor.
+  # Creates a new BlockWrite.
+  #
+  # @param block The block that will be called when this object receives the `<<` message
   def initialize(&block)
     @block = block
   end
@@ -17,26 +31,17 @@ class ZipTricks::BlockWrite
     end
   end
 
-  # Every time this object gets written to, call the Rack body each() block
-  # with the bytes given instead.
+  # Sends a string through to the block stored in the BlockWrite.
+  #
+  # @param buf[String] the string to write. Note that a zero-length String
+  #    will not be forwarded to the block, as it has special meaning when used
+  #    with chunked encoding (it indicates the end of the stream).
+  # @return self
   def <<(buf)
     # Zero-size output has a special meaning  when using chunked encoding
     return if buf.nil? || buf.bytesize.zero?
 
-    # Ensure we ALWAYS write in binary encoding.
-    encoded =
-      if buf.encoding != Encoding::BINARY
-        # If we got a frozen string we can't force_encoding on it
-        begin
-          buf.force_encoding(Encoding::BINARY)
-        rescue
-          buf.dup.force_encoding(Encoding::BINARY)
-        end
-      else
-        buf
-      end
-
-    @block.call(encoded)
+    @block.call(buf.b)
     self
   end
 end

data/lib/zip_tricks/file_reader.rb

@@ -19,7 +19,7 @@ require 'stringio'
 # ## Usage
 #
 #     File.open('zipfile.zip', 'rb') do |f|
-#       entries = FileReader.read_zip_structure(f)
+#       entries = ZipTricks::FileReader.read_zip_structure(io: f)
 #       entries.each do |e|
 #         File.open(e.filename, 'wb') do |extracted_file|
 #           ex = e.extractor_from(f)
@@ -82,7 +82,9 @@ class ZipTricks::FileReader
 
   private_constant :StoredReader, :InflatingReader
 
-  # Represents a file within the ZIP archive being read
+  # Represents a file within the ZIP archive being read. This is different from
+  # the Entry object used in Streamer for ZIP writing, since during writing more
+  # data can be kept in memory for immediate use.
   class ZipEntry
     # @return [Fixnum] bit-packed version signature of the program that made the archive
     attr_accessor :made_by
@@ -279,7 +281,7 @@ class ZipTricks::FileReader
       seek(io, next_local_header_offset) # Seek to the next entry, and raise if seek is impossible
     end
     entries
-  rescue ReadError
+  rescue ReadError, RangeError # RangeError is raised if offset exceeds int32/int64 range
     log do
       'Got a read/seek error after reaching %<cur_offset>d, no more entries can be recovered' %
         {cur_offset: cur_offset}
@@ -363,7 +365,7 @@ class ZipTricks::FileReader
   # (read starting at this offset to get the data).
   #
   # @param io[#seek, #read] an IO-ish object the ZIP file can be read from
-  # @param local_header_offset[Fixnum] absolute offset (0-based) where the
+  # @param local_file_header_offset[Fixnum] absolute offset (0-based) where the
   # local file header is supposed to begin @return [Fixnum] absolute offset
   # (0-based) of where the compressed data begins for this file within the ZIP
   def get_compressed_data_offset(io:, local_file_header_offset:)
@@ -375,7 +377,7 @@ class ZipTricks::FileReader
   # Parse an IO handle to a ZIP archive into an array of Entry objects, reading from the end
   # of the IO object.
   #
-  # @see {#read_zip_structure}
+  # @see #read_zip_structure
   # @param options[Hash] any options the instance method of the same name accepts
   # @return [Array<ZipEntry>] an array of entries within the ZIP being parsed
   def self.read_zip_structure(**options)
@@ -385,7 +387,7 @@ class ZipTricks::FileReader
   # Parse an IO handle to a ZIP archive into an array of Entry objects, reading from the start of
   # the file and parsing local file headers one-by-one
   #
-  # @see {#read_zip_straight_ahead}
+  # @see #read_zip_straight_ahead
   # @param options[Hash] any options the instance method of the same name accepts
   # @return [Array<ZipEntry>] an array of entries within the ZIP being parsed
   def self.read_zip_straight_ahead(**options)

data/lib/zip_tricks/null_writer.rb

@@ -4,7 +4,7 @@
 # write operations, but want to discard the data (like when
 # estimating the size of a ZIP)
 module ZipTricks::NullWriter
-  # @param data[String] the data to write
+  # @param _[String] the data to write
   # @return [self]
   def self.<<(_)
     self

data/lib/zip_tricks/path_set.rb

@@ -32,6 +32,10 @@
 # conflict is avoided. This is not possible to apply to directories, because when one of the
 # path components is reused in multiple filenames it means those entities should end up in
 # the same directory (subdirectory) once the archive is opened.
+#
+# The `PathSet` keeps track of entries as they get added using 2 Sets (cheap presence checks),
+# one for directories and one for files. It will raise a `Conflict` exception if there are
+# files clobbering one another, or in case files collide with directories.
 class ZipTricks::PathSet
   class Conflict < StandardError
   end

data/lib/zip_tricks/rails_streaming.rb

@@ -6,17 +6,16 @@ module ZipTricks::RailsStreaming
   # Opens a {ZipTricks::Streamer} and yields it to the caller. The output of the streamer
   # gets automatically forwarded to the Rails response stream. When the output completes,
   # the Rails response stream is going to be closed automatically.
+  # @param zip_streamer_options[Hash] options that will be passed to the Streamer.
+  #     See {ZipTricks::Streamer#initialize} for the full list of options.
   # @yield [Streamer] the streamer that can be written to
-  def zip_tricks_stream
+  # @return [ZipTricks::OutputEnumerator] The output enumerator assigned to the response body
+  def zip_tricks_stream(**zip_streamer_options, &zip_streaming_blk)
     # Set a reasonable content type
     response.headers['Content-Type'] = 'application/zip'
     # Make sure nginx buffering is suppressed - see https://github.com/WeTransfer/zip_tricks/issues/48
     response.headers['X-Accel-Buffering'] = 'no'
-    # Create a wrapper for the write call that quacks like something you
-    # can << to, used by ZipTricks
-    w = ZipTricks::BlockWrite.new { |chunk| response.stream.write(chunk) }
-    ZipTricks::Streamer.open(w) { |z| yield(z) }
-  ensure
-    response.stream.close
+    response.sending_file = true
+    self.response_body = ZipTricks::OutputEnumerator.new(**zip_streamer_options, &zip_streaming_blk)
   end
 end

data/lib/zip_tricks/size_estimator.rb

@@ -73,9 +73,6 @@ class ZipTricks::SizeEstimator
     self
   end
 
-  # Will be phased out in ZipTricks 5.x
-  alias_method :add_compressed_entry, :add_deflated_entry
-
   # Add an empty directory to the archive.
   #
   # @param dirname [String] the name of the directory

data/lib/zip_tricks/stream_crc32.rb

@@ -27,7 +27,7 @@ class ZipTricks::StreamCRC32
 
   # Creates a new streaming CRC32 calculator
   def initialize
-    @crc = Zlib.crc32('')
+    @crc = Zlib.crc32
   end
 
   # Append data to the CRC32. Updates the contained CRC32 value in place.
@@ -35,7 +35,7 @@ class ZipTricks::StreamCRC32
   # @param blob[String] the string to compute the CRC32 from
   # @return [self]
   def <<(blob)
-    @crc = Zlib.crc32_combine(@crc, Zlib.crc32(blob), blob.bytesize)
+    @crc = Zlib.crc32(blob, @crc)
     self
   end
 

data/lib/zip_tricks/streamer.rb

@@ -91,6 +91,7 @@ class ZipTricks::Streamer
   InvalidOutput = Class.new(ArgumentError)
   Overflow = Class.new(StandardError)
   UnknownMode = Class.new(StandardError)
+  OffsetOutOfSync = Class.new(StandardError)
 
   private_constant :DeflatedWriter, :StoredWriter, :STORED, :DEFLATED
 
@@ -130,7 +131,7 @@ class ZipTricks::Streamer
   #     end
   #
   # @param kwargs_for_new [Hash] keyword arguments for {Streamer.new}
-  # @return [Enumerator] the enumerator you can read bytestrings of the ZIP from using `each`
+  # @return [ZipTricks::OutputEnumerator] the enumerator you can read bytestrings of the ZIP from by calling `each`
   def self.output_enum(**kwargs_for_new, &zip_streamer_block)
     ZipTricks::OutputEnumerator.new(**kwargs_for_new, &zip_streamer_block)
   end
@@ -149,7 +150,6 @@ class ZipTricks::Streamer
     @dedupe_filenames = auto_rename_duplicate_filenames
     @out = ZipTricks::WriteAndTell.new(stream)
     @files = []
-    @local_header_offsets = []
     @path_set = ZipTricks::PathSet.new
     @writer = writer
   end
@@ -213,9 +213,6 @@ class ZipTricks::Streamer
     @out.tell
   end
 
-  # Will be phased out in ZipTricks 5.x
-  alias_method :add_compressed_entry, :add_deflated_entry
-
   # Writes out the local header for an entry (file in the ZIP) that is using
   # the stored storage model (is stored as-is).
   # Once this method is called, the `<<` method has to be called one or more
@@ -363,14 +360,16 @@ class ZipTricks::Streamer
   #
   # @return [Integer] the offset the output IO is at after closing the archive
   def close
+    # Make sure offsets are in order
+    verify_offsets!
+
     # Record the central directory offset, so that it can be written into the EOCD record
     cdir_starts_at = @out.tell
 
     # Write out the central directory entries, one for each file
-    @files.each_with_index do |entry, i|
-      header_loc = @local_header_offsets.fetch(i)
+    @files.each do |entry|
       @writer.write_central_directory_file_header(io: @out,
-                                                  local_file_header_location: header_loc,
+                                                  local_file_header_location: entry.local_header_offset,
                                                   gp_flags: entry.gp_flags,
                                                   storage_mode: entry.storage_mode,
                                                   compressed_size: entry.compressed_size,
@@ -423,15 +422,40 @@ class ZipTricks::Streamer
     last_entry.compressed_size = compressed_size
     last_entry.uncompressed_size = uncompressed_size
 
+    offset_before_data_descriptor = @out.tell
     @writer.write_data_descriptor(io: @out,
                                   crc32: last_entry.crc32,
                                   compressed_size: last_entry.compressed_size,
                                   uncompressed_size: last_entry.uncompressed_size)
+    last_entry.bytes_used_for_data_descriptor = @out.tell - offset_before_data_descriptor
+
     @out.tell
   end
 
   private
 
+  def verify_offsets!
+    # We need to check whether the offsets noted for the entries actually make sense
+    computed_offset = @files.map(&:total_bytes_used).inject(0, &:+)
+    actual_offset = @out.tell
+    if computed_offset != actual_offset
+      message = <<-EMS
+The offset of the Streamer output IO is out of sync with the expected value. All entries written so far,
+including their compressed bodies, local headers and data descriptors, add up to a certain offset,
+but this offset does not match the actual offset of the IO.
+
+Entries add up to #{computed_offset} bytes and the IO is at #{actual_offset} bytes.
+
+This can happen if you write local headers for an entry, write the "body" of the entry directly to the IO
+object which is your destination, but do not adjust the offset known to the Streamer object. To adjust
+the offfset you need to call `Streamer#simulate_write(body_size)` after outputting the entry. Otherwise
+the local header offsets of the entries you write are going to be incorrect and some ZIP applications
+are going to have problems opening your archive.
+EMS
+      raise OffsetOutOfSync, message
+    end
+  end
+
   def add_file_and_write_local_header(
       filename:,
       modification_time:,
@@ -464,16 +488,18 @@ class ZipTricks::Streamer
       uncompressed_size = 0
     end
 
+    local_header_starts_at = @out.tell
+
     e = Entry.new(filename,
                   crc32,
                   compressed_size,
                   uncompressed_size,
                   storage_mode,
                   modification_time,
-                  use_data_descriptor)
-
-    @files << e
-    @local_header_offsets << @out.tell
+                  use_data_descriptor,
+                  _local_file_header_offset = local_header_starts_at,
+                  _bytes_used_for_local_header = 0,
+                  _bytes_used_for_data_descriptor = 0)
 
     @writer.write_local_file_header(io: @out,
                                     gp_flags: e.gp_flags,
@@ -483,6 +509,9 @@ class ZipTricks::Streamer
                                     mtime: e.mtime,
                                     filename: e.filename,
                                     storage_mode: e.storage_mode)
+    e.bytes_used_for_local_header = @out.tell - e.local_header_offset
+
+    @files << e
   end
 
   def remove_backslash(filename)