zip_tricks 5.1.1 → 5.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 311852db3fa83e4f6631fb8ecf563255c0f16a65bf2b3c2f2375f41040eb3661
-  data.tar.gz: e7b307a3b5eb5ed344d20c85d8d5cef40e89e9afbe61ce5dded2f3971b60002a
+  metadata.gz: 50ba2d6a0b5bde1cf51443c7ff4228a7e99a1cc1ac843e3ef23c0d197878a04b
+  data.tar.gz: 5fdb377cc34fd6d9edb4e7932e79ebe28bc2dc91aa71348e386b8b601e4f06f1
 SHA512:
-  metadata.gz: 031da6b3ab1199c8967f7cfbf0116beb9025346b10ae3679391f08c6c3205672842b1f38309efeb710af758c0c28a9c2ed0b5ce784936f3c3bfdb79b3920a3a2
-  data.tar.gz: 5e412daec62f191e4e54cfe8ff60127784264b0c851454db58b95b646af379a219d271e9d6d94aa00beb0d91b986960da07266940cc58764abb2e3e8892372b6
+  metadata.gz: 4c2ae765d7e6c584632b7606d66b970c100b9679cb6b503209be1179bc9582f83727b13154bfce5281dd8dc105f1f5112d2794df7f97ebb987a1e224f67f4840
+  data.tar.gz: e06306028f18fe1eb16abe12c48cd93182d11451298cde6068b9ccb37b78846be469b6bd48848507f38f8796b00281f27391589d20d9163b3845cba4112411c5

data/.rubocop.yml

@@ -11,3 +11,5 @@ Style/GlobalVars:
       - qa/*.rb
       - spec/spec_helper.rb
       - spec/support/zip_inspection.rb
+Layout/IndentHeredoc:
+  Enabled: false

data/.travis.yml

@@ -4,8 +4,5 @@ rvm:
 - jruby-9.0
 sudo: false
 cache: bundler
-matrix:
-  allow_failures:
-    - rvm: jruby-9.0
 script:
   - bundle exec rake

data/CHANGELOG.md

@@ -1,3 +1,39 @@
+## 5.5.0
+
+* In `OutputEnumerator` apply some amount of buffering to be within a UNIX socket size for metatada writes. This
+  speeds up usage with Puma by about 20 percent, as there won't be as many `syswrite` calls on the socket.
+* Make `StoredWriter` and `DeflatedWriter` public constants so that standalone tests can be written for them
+
+## 5.4.0
+
+* Use block form for zlib Deflater calls to conserve memory
+* Do not change string encoding in writer wrappers (avoid extra work)
+* Fix a zlib deflater object being leaked per archived file
+* Speed up streaming CRC32 computation
+* When running tests, assign the port for the Puma server dynamically
+* Reduce string allocations in the block deflate spec
+* Make sure RemoteUncap specs run under JRuby correctly
+* Replace Rails::Live streaming with iterable body streaming to avoid issues with Rails::Live across the board
+* Remove `qa/` directory and scripts, as the tests for the library proper should now be sufficient
+* Fix some documentation and sample code omissions and inconsistencies.
+
+## 5.3.1
+
+* Fix extended timestamp timestamp value encoding. Previously we would use an incorrect encoding for the timestamp value, which would output correct but nonsensical timestamps. The pack specifier is now changed to output the correct value.
+
+## 5.3.0
+
+* Raise in `Streamer#close` when the IO offset of the Streamer does not match the size of the written entries. This is a situation which
+  can occur if one adds the local headers, writes the bodies of the files to the socket/output directly, and forgets to adjust the internal
+  Streamer offset. The unadjusted offset would then produce incorrect values in both the local headers which come after the missing
+  offset adjustment _and_ in the central directory headers. Some ZIP unarchivers are able to recover from this (ones that read
+  files "straight-ahead" but others aren't - if the ZIP unarchiver uses central directory entries it would be using incorrect offsets.
+  Instead of producing an invalid ZIP, raise an exception which explains what happened and how it can be resolved.
+
+## 5.2.0
+
+* Remove `Streamer#add_compressed_entry` and `SizeEstimator#add_compressed_entry`
+
 ## 5.1.1
 
 * Fix extended timestamp extra field output. The first bit of the flag would be set instead of the last bit of

data/README.md

@@ -24,20 +24,20 @@ to [32 bit sizes.](https://github.com/jruby/jruby/issues/3817)
 
 ## Diving in: send some large CSV reports from Rails
 
-The easiest is to use the Rails' built-in streaming feature:
+The easiest is to include the `ZipTricks::RailsStreaming` module into your
+controller.
 
 ```ruby
 class ZipsController < ActionController::Base
-  include ActionController::Live # required for streaming
   include ZipTricks::RailsStreaming
 
   def download
     zip_tricks_stream do |zip|
       zip.write_deflated_file('report1.csv') do |sink|
         CSV(sink) do |csv_write|
-          csv << Person.column_names
+          csv_write << Person.column_names
           Person.all.find_each do |person|
-            csv << person.attributes.values
+            csv_write << person.attributes.values
           end
         end
       end
@@ -49,6 +49,10 @@ class ZipsController < ActionController::Base
 end
 ```
 
+If you want some more conveniences you can also use [zipline](https://github.com/fringd/zipline) which
+will automatically process and stream attachments (Carrierwave, Shrine, ActiveStorage) and remote objects
+via HTTP.
+
 ## Create a ZIP file without size estimation, compress on-the-fly during writes
 
 Basic use case is compressing on the fly. Some data will be buffered by the Zlib deflater, but
@@ -71,12 +75,15 @@ since you do not know how large the compressed data segments are going to be.
 
 ## Send a ZIP from a Rack response
 
-Create a `RackBody` object and give it's constructor a block that adds files.
-The block will only be called when actually sending the response to the client
+To "pull" data from ZipTricks you can create an `OutputEnumerator` object which will yield the binary chunks piece
+by piece, and apply some amount of buffering as well. Since this `OutputEnumerator` responds to `#each` and yields
+Strings it also can (and should!) be used as a Rack response body. Return it to your webserver and you will
+have your ZIP streamed. The block that you give to the `OutputEnumerator` will only start executing once your
+response body starts getting iterated over - when actually sending the response to the client
 (unless you are using a buffering Rack webserver, such as Webrick).
 
 ```ruby
-body = ZipTricks::RackBody.new do | zip |
+body = ZipTricks::Streamer.output_enum do | zip |
   zip.write_stored_file('mov.mp4') do |sink| # Those MPEG4 files do not compress that well
     File.open('mov.mp4', 'rb'){|source| IO.copy_stream(source, sink) }
   end
@@ -84,7 +91,7 @@ body = ZipTricks::RackBody.new do | zip |
     File.open('novel.txt', 'rb'){|source| IO.copy_stream(source, sink) }
   end
 end
-[200, {'Transfer-Encoding' => 'chunked'}, body]
+[200, {}, body]
 ```
 
 ## Send a ZIP file of known size, with correct headers
@@ -123,11 +130,12 @@ ZipTricks::Streamer.open(io) do | zip |
   # Write the local file header first..
   zip.add_stored_entry(filename: "first-file.bin", size: raw_file.size, crc32: raw_file_crc32)
 
-  # then send the actual file contents bypassing the Streamer interface
+  # Adjust the ZIP offsets within the Streamer
+  zip.simulate_write(my_temp_file.size)
+
+  # ...and then send the actual file contents bypassing the Streamer interface
   io.sendfile(my_temp_file)
 
-  # ...and then adjust the ZIP offsets within the Streamer
-  zip.simulate_write(my_temp_file.size)
 end
 ```
 
@@ -174,9 +182,12 @@ that have not been formally verified (ours hasn't been).
 * Commit and push until you are happy with your contribution.
 * Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
 * Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
-* If you alter the `ZipWriter`, please take the time to run the test in the `qa/` directory. Ensure that the generated (large) files open manually - see README_QA for more.
 
-## Copyright
+## Copyright and license
+
+Copyright (c) 2020 WeTransfer.
 
-Copyright (c) 2019 WeTransfer. `zip_tricks` is distributed under the conditions of the [Hippocratic License](https://firstdonoharm.dev/version/1/2/license.html)
- - See LICENSE.txt for further details.
+`zip_tricks` is distributed under the conditions of the [Hippocratic License](https://firstdonoharm.dev/version/1/2/license.html)
+See LICENSE.txt for further details. If this license is not acceptable for your use case we still maintain the 4.x version tree
+which remains under the MIT license, see https://rubygems.org/gems/zip_tricks/versions for more information.
+Note that we only backport some performance optimizations and crucial bugfixes but not the new features to that tree.

data/lib/zip_tricks/block_write.rb

@@ -1,11 +1,25 @@
 # frozen_string_literal: true
 
-# Stashes a block given by the Rack webserver when calling each() on a body, and calls
-# that block every time it is written to using :<< (shovel). Poses as an IO for rubyzip.
-
+# Acts as a converter between callers which send data to the `#<<` method (such as all the ZipTricks
+# writer methods, which push onto anything), and a given block. Every time `#<<` gets called on the BlockWrite,
+# the block given to the constructor will be called with the same argument. ZipTricks uses this object
+# when integrating with Rack and in the OutputEnumerator. Normally you wouldn't need to use it manually but
+# you always can. BlockWrite will also ensure the binary string encoding is forced onto any string
+# that passes through it.
+#
+# For example, you can create a Rack response body like so:
+#
+#     class MyRackResponse
+#       def each
+#         writer = ZipTricks::BlockWrite.new {|chunk| yield(chunk) }
+#         writer << "Hello" << "world" << "!"
+#       end
+#     end
+#     [200, {}, MyRackResponse.new]
 class ZipTricks::BlockWrite
-  # The block is the block given to each() of the Rack body, or other block you want
-  # to receive the string chunks written by the zip compressor.
+  # Creates a new BlockWrite.
+  #
+  # @param block The block that will be called when this object receives the `<<` message
   def initialize(&block)
     @block = block
   end
@@ -17,26 +31,17 @@ class ZipTricks::BlockWrite
     end
   end
 
-  # Every time this object gets written to, call the Rack body each() block
-  # with the bytes given instead.
+  # Sends a string through to the block stored in the BlockWrite.
+  #
+  # @param buf[String] the string to write. Note that a zero-length String
+  #    will not be forwarded to the block, as it has special meaning when used
+  #    with chunked encoding (it indicates the end of the stream).
+  # @return self
   def <<(buf)
     # Zero-size output has a special meaning  when using chunked encoding
     return if buf.nil? || buf.bytesize.zero?
 
-    # Ensure we ALWAYS write in binary encoding.
-    encoded =
-      if buf.encoding != Encoding::BINARY
-        # If we got a frozen string we can't force_encoding on it
-        begin
-          buf.force_encoding(Encoding::BINARY)
-        rescue
-          buf.dup.force_encoding(Encoding::BINARY)
-        end
-      else
-        buf
-      end
-
-    @block.call(encoded)
+    @block.call(buf.b)
     self
   end
 end

data/lib/zip_tricks/file_reader.rb

@@ -19,7 +19,7 @@ require 'stringio'
 # ## Usage
 #
 #     File.open('zipfile.zip', 'rb') do |f|
-#       entries = FileReader.read_zip_structure(f)
+#       entries = ZipTricks::FileReader.read_zip_structure(io: f)
 #       entries.each do |e|
 #         File.open(e.filename, 'wb') do |extracted_file|
 #           ex = e.extractor_from(f)
@@ -82,7 +82,9 @@ class ZipTricks::FileReader
 
   private_constant :StoredReader, :InflatingReader
 
-  # Represents a file within the ZIP archive being read
+  # Represents a file within the ZIP archive being read. This is different from
+  # the Entry object used in Streamer for ZIP writing, since during writing more
+  # data can be kept in memory for immediate use.
   class ZipEntry
     # @return [Fixnum] bit-packed version signature of the program that made the archive
     attr_accessor :made_by
@@ -279,7 +281,7 @@ class ZipTricks::FileReader
       seek(io, next_local_header_offset) # Seek to the next entry, and raise if seek is impossible
     end
     entries
-  rescue ReadError
+  rescue ReadError, RangeError # RangeError is raised if offset exceeds int32/int64 range
     log do
       'Got a read/seek error after reaching %<cur_offset>d, no more entries can be recovered' %
         {cur_offset: cur_offset}
@@ -363,7 +365,7 @@ class ZipTricks::FileReader
   # (read starting at this offset to get the data).
   #
   # @param io[#seek, #read] an IO-ish object the ZIP file can be read from
-  # @param local_header_offset[Fixnum] absolute offset (0-based) where the
+  # @param local_file_header_offset[Fixnum] absolute offset (0-based) where the
   # local file header is supposed to begin @return [Fixnum] absolute offset
   # (0-based) of where the compressed data begins for this file within the ZIP
   def get_compressed_data_offset(io:, local_file_header_offset:)
@@ -375,7 +377,7 @@ class ZipTricks::FileReader
   # Parse an IO handle to a ZIP archive into an array of Entry objects, reading from the end
   # of the IO object.
   #
-  # @see {#read_zip_structure}
+  # @see #read_zip_structure
   # @param options[Hash] any options the instance method of the same name accepts
   # @return [Array<ZipEntry>] an array of entries within the ZIP being parsed
   def self.read_zip_structure(**options)
@@ -385,7 +387,7 @@ class ZipTricks::FileReader
   # Parse an IO handle to a ZIP archive into an array of Entry objects, reading from the start of
   # the file and parsing local file headers one-by-one
   #
-  # @see {#read_zip_straight_ahead}
+  # @see #read_zip_straight_ahead
   # @param options[Hash] any options the instance method of the same name accepts
   # @return [Array<ZipEntry>] an array of entries within the ZIP being parsed
   def self.read_zip_straight_ahead(**options)

data/lib/zip_tricks/null_writer.rb

@@ -4,7 +4,7 @@
 # write operations, but want to discard the data (like when
 # estimating the size of a ZIP)
 module ZipTricks::NullWriter
-  # @param data[String] the data to write
+  # @param _[String] the data to write
   # @return [self]
   def self.<<(_)
     self

data/lib/zip_tricks/output_enumerator.rb

@@ -1,43 +1,64 @@
 # frozen_string_literal: true
 
-# Can be used as a Rack response body directly. Will yield
-# a {ZipTricks::Streamer} for adding entries to the archive and writing
-# zip entry bodies.
+# The output enumerator makes it possible to "pull" from a ZipTricks streamer
+# object instead of having it "push" writes to you. It will "stash" the block which
+# writes the ZIP archive through the streamer, and when you call `each` on the Enumerator
+# it will yield you the bytes the block writes. Since it is an enumerator you can
+# use `next` to take chunks written by the ZipTricks streamer one by one. It can be very
+# convenient when you need to segment your ZIP output into bigger chunks for, say,
+# uploading them to a cloud storage provider such as S3.
+#
+# Another use of the output enumerator is outputting a ZIP archive from Rails or Rack,
+# where an object responding to `each` is required which yields Strings. For instance,
+# you can return a ZIP archive from Rack like so:
+#
+#     iterable_zip_body = ZipTricks::OutputEnumerator.new do | streamer |
+#       streamer.write_deflated_file('big.csv') do |sink|
+#         CSV(sink) do |csv_writer|
+#           csv_writer << Person.column_names
+#           Person.all.find_each do |person|
+#             csv_writer << person.attributes.values
+#           end
+#         end
+#       end
+#     end
+#
+#     [200, {'Content-Type' => 'binary/octet-stream'}, iterable_zip_body]
 class ZipTricks::OutputEnumerator
-  # Prepares a new Rack response body with a Zip output stream.
-  # The block given to the constructor will be called when the response
-  # body will be read by the webserver, and will receive a {ZipTricks::Streamer}
-  # as it's block argument. You can then add entries to the Streamer as usual.
-  # The archive will be automatically closed at the end of the block.
+  DEFAULT_WRITE_BUFFER_SIZE = 64 * 1024
+  # Creates a new OutputEnumerator.
   #
-  #     # Precompute the Content-Length ahead of time
-  #     content_length = ZipTricks::SizeEstimator.estimate do | estimator |
-  #       estimator.add_stored_entry(filename: 'large.tif', size: 1289894)
-  #     end
-  #
-  #     # Prepare the response body.
-  #     # The block will only be called when the
-  #     # response starts to be written.
-  #     body = ZipTricks::OutputEnumerator.new do | streamer |
-  #       streamer.add_stored_entry(filename: 'large.tif', size: 1289894, crc32: 198210)
-  #       streamer << large_file.read(1024*1024) until large_file.eof?
-  #       ...
-  #     end
-  #
-  #     return [200, {'Content-Type' => 'binary/octet-stream',
-  #     'Content-Length' => content_length.to_s}, body]
-  def initialize(**streamer_options, &blk)
+  # @param streamer_options[Hash] options for Streamer, see {ZipTricks::Streamer.new}
+  # @param write_buffer_size[Integer] By default all ZipTricks writes are unbuffered. For output to sockets
+  #     it is beneficial to bulkify those writes so that they are roughly sized to a socket buffer chunk. This
+  #     object will bulkify writes for you in this way (so `each` will yield not on every call to `<<` from the Streamer
+  #     but at block size boundaries or greater). Set it to 0 for unbuffered writes.
+  # @param blk a block that will receive the Streamer object when executing. The block will not be executed
+  #     immediately but only once `each` is called on the OutputEnumerator
+  def initialize(write_buffer_size: DEFAULT_WRITE_BUFFER_SIZE, **streamer_options, &blk)
     @streamer_options = streamer_options.to_h
+    @bufsize = write_buffer_size.to_i
     @archiving_block = blk
   end
 
   # Executes the block given to the constructor with a {ZipTricks::Streamer}
   # and passes each written chunk to the block given to the method. This allows one
-  # to "take" output of the ZIP piecewise.
+  # to "take" output of the ZIP piecewise. If called without a block will return an Enumerator
+  # that you can pull data from using `next`.
+  #
+  # **NOTE** Because the `WriteBuffer` inside this object can reuse the buffer, it is important
+  #    that the `String` that is yielded **either** gets consumed eagerly (written byte-by-byte somewhere, or `#dup`-ed)
+  #    since the write buffer will clear it after your block returns. If you expand this Enumerator
+  #    eagerly into an Array you might notice that a lot of the segments of your ZIP output are
+  #    empty - this means that you need to duplicate them.
+  #
+  # @yield [String] a chunk of the ZIP output in binary encoding
   def each
     if block_given?
       block_write = ZipTricks::BlockWrite.new { |chunk| yield(chunk) }
-      ZipTricks::Streamer.open(block_write, **@streamer_options, &@archiving_block)
+      buffer = ZipTricks::WriteBuffer.new(block_write, @bufsize)
+      ZipTricks::Streamer.open(buffer, **@streamer_options, &@archiving_block)
+      buffer.flush
     else
       enum_for(:each)
     end

data/lib/zip_tricks/path_set.rb

@@ -32,6 +32,10 @@
 # conflict is avoided. This is not possible to apply to directories, because when one of the
 # path components is reused in multiple filenames it means those entities should end up in
 # the same directory (subdirectory) once the archive is opened.
+#
+# The `PathSet` keeps track of entries as they get added using 2 Sets (cheap presence checks),
+# one for directories and one for files. It will raise a `Conflict` exception if there are
+# files clobbering one another, or in case files collide with directories.
 class ZipTricks::PathSet
   class Conflict < StandardError
   end

data/lib/zip_tricks/rails_streaming.rb

@@ -6,17 +6,16 @@ module ZipTricks::RailsStreaming
   # Opens a {ZipTricks::Streamer} and yields it to the caller. The output of the streamer
   # gets automatically forwarded to the Rails response stream. When the output completes,
   # the Rails response stream is going to be closed automatically.
+  # @param zip_streamer_options[Hash] options that will be passed to the Streamer.
+  #     See {ZipTricks::Streamer#initialize} for the full list of options.
   # @yield [Streamer] the streamer that can be written to
-  def zip_tricks_stream
+  # @return [ZipTricks::OutputEnumerator] The output enumerator assigned to the response body
+  def zip_tricks_stream(**zip_streamer_options, &zip_streaming_blk)
     # Set a reasonable content type
     response.headers['Content-Type'] = 'application/zip'
     # Make sure nginx buffering is suppressed - see https://github.com/WeTransfer/zip_tricks/issues/48
     response.headers['X-Accel-Buffering'] = 'no'
-    # Create a wrapper for the write call that quacks like something you
-    # can << to, used by ZipTricks
-    w = ZipTricks::BlockWrite.new { |chunk| response.stream.write(chunk) }
-    ZipTricks::Streamer.open(w) { |z| yield(z) }
-  ensure
-    response.stream.close
+    response.sending_file = true
+    self.response_body = ZipTricks::OutputEnumerator.new(**zip_streamer_options, &zip_streaming_blk)
   end
 end

data/lib/zip_tricks/size_estimator.rb

@@ -73,9 +73,6 @@ class ZipTricks::SizeEstimator
     self
   end
 
-  # Will be phased out in ZipTricks 5.x
-  alias_method :add_compressed_entry, :add_deflated_entry
-
   # Add an empty directory to the archive.
   #
   # @param dirname [String] the name of the directory

data/lib/zip_tricks/stream_crc32.rb

@@ -27,7 +27,7 @@ class ZipTricks::StreamCRC32
 
   # Creates a new streaming CRC32 calculator
   def initialize
-    @crc = Zlib.crc32('')
+    @crc = Zlib.crc32
   end
 
   # Append data to the CRC32. Updates the contained CRC32 value in place.
@@ -35,7 +35,7 @@ class ZipTricks::StreamCRC32
   # @param blob[String] the string to compute the CRC32 from
   # @return [self]
   def <<(blob)
-    @crc = Zlib.crc32_combine(@crc, Zlib.crc32(blob), blob.bytesize)
+    @crc = Zlib.crc32(blob, @crc)
     self
   end