zip_tricks 5.3.1 → 5.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a9e1ad9dd7452add6a2f9187edecdd366699d81fba9260692ce2a96003401eca
-  data.tar.gz: 1f2665b7c3a5c99ce190b5dab76fa9334ac4c6a3307928bab94758acd98bcb56
+  metadata.gz: 05e8eea8ecf1ad0b9b9cb132c54dde1abd60dc003afd1e5a1ce989786ce89608
+  data.tar.gz: fbdc2172fc3becefa4dd8720713acb11d838900465ac3cb42de38fec76fd0d90
 SHA512:
-  metadata.gz: 8864d6d56796e49e2024e46aee7cd1ac1c8ea575d757a6fbe6fd62a6e77db94eef6dc5b80bdbeae5bb932536e889fbdab224642862452f5b1acc0020e8a8c906
-  data.tar.gz: 67bc05db334c9a54804ac460f14f5919b9049913ee75532da090405f9a5b1f7e792ff4678aec0683a8b2a3136a9799ea3dd88aa3c9995491b28e4e6d88c4ac3a
+  metadata.gz: 449c59e898d2b54a089b60d7aebe7633d9f65ad64a5ce014a7a44e1683f6d6cec4997f732fd06746edeec96594d5041b68efcf19571ef92c9798acc05ffda7bd
+  data.tar.gz: 5ed26109e12373acfb9866531ef03c4302141bbb8a312f759ff768295c8d7926f650745f3f2624ad652333e57cd5bc5f98f932ace41504aeca45a668ef7a4f4a

data/.travis.yml

@@ -4,8 +4,5 @@ rvm:
 - jruby-9.0
 sudo: false
 cache: bundler
-matrix:
-  allow_failures:
-    - rvm: jruby-9.0
 script:
   - bundle exec rake

data/CHANGELOG.md

@@ -1,3 +1,16 @@
+## 5.4.0
+
+* Use block form for zlib Deflater calls to conserve memory
+* Do not change string encoding in writer wrappers (avoid extra work)
+* Fix a zlib deflater object being leaked per archived file
+* Speed up streaming CRC32 computation
+* When running tests, assign the port for the Puma server dynamically
+* Reduce string allocations in the block deflate spec
+* Make sure RemoteUncap specs run under JRuby correctly
+* Replace Rails::Live streaming with iterable body streaming to avoid issues with Rails::Live across the board
+* Remove `qa/` directory and scripts, as the tests for the library proper should now be sufficient
+* Fix some documentation and sample code omissions and inconsistencies.
+
 ## 5.3.1
 
 * Fix extended timestamp timestamp value encoding. Previously we would use an incorrect encoding for the timestamp value, which would output correct but nonsensical timestamps. The pack specifier is now changed to output the correct value.

data/README.md

@@ -24,11 +24,11 @@ to [32 bit sizes.](https://github.com/jruby/jruby/issues/3817)
 
 ## Diving in: send some large CSV reports from Rails
 
-The easiest is to use the Rails' built-in streaming feature:
+The easiest is to include the `ZipTricks::RailsStreaming` module into your
+controller.
 
 ```ruby
 class ZipsController < ActionController::Base
-  include ActionController::Live # required for streaming
   include ZipTricks::RailsStreaming
 
   def download
@@ -49,6 +49,10 @@ class ZipsController < ActionController::Base
 end
 ```
 
+If you want some more conveniences you can also use [zipline](https://github.com/fringd/zipline) which
+will automatically process and stream attachments (Carrierwave, Shrine, ActiveStorage) and remote objects
+via HTTP.
+
 ## Create a ZIP file without size estimation, compress on-the-fly during writes
 
 Basic use case is compressing on the fly. Some data will be buffered by the Zlib deflater, but
@@ -84,7 +88,7 @@ body = ZipTricks::RackBody.new do | zip |
     File.open('novel.txt', 'rb'){|source| IO.copy_stream(source, sink) }
   end
 end
-[200, {'Transfer-Encoding' => 'chunked'}, body]
+[200, {}, body]
 ```
 
 ## Send a ZIP file of known size, with correct headers
@@ -174,9 +178,12 @@ that have not been formally verified (ours hasn't been).
 * Commit and push until you are happy with your contribution.
 * Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
 * Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
-* If you alter the `ZipWriter`, please take the time to run the test in the `qa/` directory. Ensure that the generated (large) files open manually - see README_QA for more.
 
-## Copyright
+## Copyright and license
+
+Copyright (c) 2020 WeTransfer.
 
-Copyright (c) 2019 WeTransfer. `zip_tricks` is distributed under the conditions of the [Hippocratic License](https://firstdonoharm.dev/version/1/2/license.html)
- - See LICENSE.txt for further details.
+`zip_tricks` is distributed under the conditions of the [Hippocratic License](https://firstdonoharm.dev/version/1/2/license.html)
+See LICENSE.txt for further details. If this license is not acceptable for your use case we still maintain the 4.x version tree
+which remains under the MIT license, see https://rubygems.org/gems/zip_tricks/versions for more information.
+Note that we only backport some performance optimizations and crucial bugfixes but not the new features to that tree.

data/lib/zip_tricks/block_write.rb

@@ -1,11 +1,25 @@
 # frozen_string_literal: true
 
-# Stashes a block given by the Rack webserver when calling each() on a body, and calls
-# that block every time it is written to using :<< (shovel). Poses as an IO for rubyzip.
-
+# Acts as a converter between callers which send data to the `#<<` method (such as all the ZipTricks
+# writer methods, which push onto anything), and a given block. Every time `#<<` gets called on the BlockWrite,
+# the block given to the constructor will be called with the same argument. ZipTricks uses this object
+# when integrating with Rack and in the OutputEnumerator. Normally you wouldn't need to use it manually but
+# you always can. BlockWrite will also ensure the binary string encoding is forced onto any string
+# that passes through it.
+#
+# For example, you can create a Rack response body like so:
+#
+#     class MyRackResponse
+#       def each
+#         writer = ZipTricks::BlockWrite.new {|chunk| yield(chunk) }
+#         writer << "Hello" << "world" << "!"
+#       end
+#     end
+#     [200, {}, MyRackResponse.new]
 class ZipTricks::BlockWrite
-  # The block is the block given to each() of the Rack body, or other block you want
-  # to receive the string chunks written by the zip compressor.
+  # Creates a new BlockWrite.
+  #
+  # @param block The block that will be called when this object receives the `<<` message
   def initialize(&block)
     @block = block
   end
@@ -17,26 +31,17 @@ class ZipTricks::BlockWrite
     end
   end
 
-  # Every time this object gets written to, call the Rack body each() block
-  # with the bytes given instead.
+  # Sends a string through to the block stored in the BlockWrite.
+  #
+  # @param buf[String] the string to write. Note that a zero-length String
+  #    will not be forwarded to the block, as it has special meaning when used
+  #    with chunked encoding (it indicates the end of the stream).
+  # @return self
   def <<(buf)
     # Zero-size output has a special meaning  when using chunked encoding
     return if buf.nil? || buf.bytesize.zero?
 
-    # Ensure we ALWAYS write in binary encoding.
-    encoded =
-      if buf.encoding != Encoding::BINARY
-        # If we got a frozen string we can't force_encoding on it
-        begin
-          buf.force_encoding(Encoding::BINARY)
-        rescue
-          buf.dup.force_encoding(Encoding::BINARY)
-        end
-      else
-        buf
-      end
-
-    @block.call(encoded)
+    @block.call(buf.b)
     self
   end
 end

data/lib/zip_tricks/file_reader.rb

@@ -19,7 +19,7 @@ require 'stringio'
 # ## Usage
 #
 #     File.open('zipfile.zip', 'rb') do |f|
-#       entries = FileReader.read_zip_structure(f)
+#       entries = ZipTricks::FileReader.read_zip_structure(io: f)
 #       entries.each do |e|
 #         File.open(e.filename, 'wb') do |extracted_file|
 #           ex = e.extractor_from(f)
@@ -281,7 +281,7 @@ class ZipTricks::FileReader
       seek(io, next_local_header_offset) # Seek to the next entry, and raise if seek is impossible
     end
     entries
-  rescue ReadError
+  rescue ReadError, RangeError # RangeError is raised if offset exceeds int32/int64 range
     log do
       'Got a read/seek error after reaching %<cur_offset>d, no more entries can be recovered' %
         {cur_offset: cur_offset}
@@ -365,7 +365,7 @@ class ZipTricks::FileReader
   # (read starting at this offset to get the data).
   #
   # @param io[#seek, #read] an IO-ish object the ZIP file can be read from
-  # @param local_header_offset[Fixnum] absolute offset (0-based) where the
+  # @param local_file_header_offset[Fixnum] absolute offset (0-based) where the
   # local file header is supposed to begin @return [Fixnum] absolute offset
   # (0-based) of where the compressed data begins for this file within the ZIP
   def get_compressed_data_offset(io:, local_file_header_offset:)
@@ -377,7 +377,7 @@ class ZipTricks::FileReader
   # Parse an IO handle to a ZIP archive into an array of Entry objects, reading from the end
   # of the IO object.
   #
-  # @see {#read_zip_structure}
+  # @see #read_zip_structure
   # @param options[Hash] any options the instance method of the same name accepts
   # @return [Array<ZipEntry>] an array of entries within the ZIP being parsed
   def self.read_zip_structure(**options)
@@ -387,7 +387,7 @@ class ZipTricks::FileReader
   # Parse an IO handle to a ZIP archive into an array of Entry objects, reading from the start of
   # the file and parsing local file headers one-by-one
   #
-  # @see {#read_zip_straight_ahead}
+  # @see #read_zip_straight_ahead
   # @param options[Hash] any options the instance method of the same name accepts
   # @return [Array<ZipEntry>] an array of entries within the ZIP being parsed
   def self.read_zip_straight_ahead(**options)

data/lib/zip_tricks/null_writer.rb

@@ -4,7 +4,7 @@
 # write operations, but want to discard the data (like when
 # estimating the size of a ZIP)
 module ZipTricks::NullWriter
-  # @param data[String] the data to write
+  # @param _[String] the data to write
   # @return [self]
   def self.<<(_)
     self

data/lib/zip_tricks/path_set.rb

@@ -32,6 +32,10 @@
 # conflict is avoided. This is not possible to apply to directories, because when one of the
 # path components is reused in multiple filenames it means those entities should end up in
 # the same directory (subdirectory) once the archive is opened.
+#
+# The `PathSet` keeps track of entries as they get added using 2 Sets (cheap presence checks),
+# one for directories and one for files. It will raise a `Conflict` exception if there are
+# files clobbering one another, or in case files collide with directories.
 class ZipTricks::PathSet
   class Conflict < StandardError
   end

data/lib/zip_tricks/rails_streaming.rb

@@ -6,17 +6,16 @@ module ZipTricks::RailsStreaming
   # Opens a {ZipTricks::Streamer} and yields it to the caller. The output of the streamer
   # gets automatically forwarded to the Rails response stream. When the output completes,
   # the Rails response stream is going to be closed automatically.
+  # @param zip_streamer_options[Hash] options that will be passed to the Streamer.
+  #     See {ZipTricks::Streamer#initialize} for the full list of options.
   # @yield [Streamer] the streamer that can be written to
-  def zip_tricks_stream
+  # @return [ZipTricks::OutputEnumerator] The output enumerator assigned to the response body
+  def zip_tricks_stream(**zip_streamer_options, &zip_streaming_blk)
     # Set a reasonable content type
     response.headers['Content-Type'] = 'application/zip'
     # Make sure nginx buffering is suppressed - see https://github.com/WeTransfer/zip_tricks/issues/48
     response.headers['X-Accel-Buffering'] = 'no'
-    # Create a wrapper for the write call that quacks like something you
-    # can << to, used by ZipTricks
-    w = ZipTricks::BlockWrite.new { |chunk| response.stream.write(chunk) }
-    ZipTricks::Streamer.open(w) { |z| yield(z) }
-  ensure
-    response.stream.close
+    response.sending_file = true
+    self.response_body = ZipTricks::OutputEnumerator.new(**zip_streamer_options, &zip_streaming_blk)
   end
 end

data/lib/zip_tricks/stream_crc32.rb

@@ -27,7 +27,7 @@ class ZipTricks::StreamCRC32
 
   # Creates a new streaming CRC32 calculator
   def initialize
-    @crc = Zlib.crc32('')
+    @crc = Zlib.crc32
   end
 
   # Append data to the CRC32. Updates the contained CRC32 value in place.
@@ -35,7 +35,7 @@ class ZipTricks::StreamCRC32
   # @param blob[String] the string to compute the CRC32 from
   # @return [self]
   def <<(blob)
-    @crc = Zlib.crc32_combine(@crc, Zlib.crc32(blob), blob.bytesize)
+    @crc = Zlib.crc32(blob, @crc)
     self
   end
 

data/lib/zip_tricks/streamer.rb

@@ -131,7 +131,7 @@ class ZipTricks::Streamer
   #     end
   #
   # @param kwargs_for_new [Hash] keyword arguments for {Streamer.new}
-  # @return [Enumerator] the enumerator you can read bytestrings of the ZIP from using `each`
+  # @return [ZipTricks::OutputEnumerator] the enumerator you can read bytestrings of the ZIP from by calling `each`
   def self.output_enum(**kwargs_for_new, &zip_streamer_block)
     ZipTricks::OutputEnumerator.new(**kwargs_for_new, &zip_streamer_block)
   end

data/lib/zip_tricks/streamer/deflated_writer.rb

@@ -4,13 +4,6 @@
 # registers data passing through it in a CRC32 checksum calculator. Is made to be completely
 # interchangeable with the StoredWriter in terms of interface.
 class ZipTricks::Streamer::DeflatedWriter
-  # After how many bytes of incoming data the deflater for the
-  # contents must be flushed. This is done to prevent unreasonable
-  # memory use when archiving large files, and to ensure we write to
-  # the socket often enough while still maintaining acceptable
-  # compression
-  FLUSH_EVERY_N_BYTES = 1024 * 1024 * 5
-
   # The amount of bytes we will buffer before computing the intermediate
   # CRC32 checksums. Benchmarks show that the optimum is 64KB (see
   # `bench/buffered_crc32_bench.rb), if that is exceeded Zlib is going
@@ -18,11 +11,9 @@ class ZipTricks::Streamer::DeflatedWriter
   CRC32_BUFFER_SIZE = 64 * 1024
 
   def initialize(io)
-    @compressed_io = ZipTricks::WriteAndTell.new(io)
-    @uncompressed_size = 0
+    @compressed_io = io
     @deflater = ::Zlib::Deflate.new(Zlib::DEFAULT_COMPRESSION, -::Zlib::MAX_WBITS)
     @crc = ZipTricks::WriteBuffer.new(ZipTricks::StreamCRC32.new, CRC32_BUFFER_SIZE)
-    @bytes_since_last_flush = 0
   end
 
   # Writes the given data into the deflater, and flushes the deflater
@@ -31,13 +22,8 @@ class ZipTricks::Streamer::DeflatedWriter
   # @param data[String] data to be written
   # @return self
   def <<(data)
-    @uncompressed_size += data.bytesize
-    @bytes_since_last_flush += data.bytesize
-    @compressed_io << @deflater.deflate(data)
+    @deflater.deflate(data) { |chunk| @compressed_io << chunk }
     @crc << data
-
-    interim_flush
-
     self
   end
 
@@ -45,18 +31,11 @@ class ZipTricks::Streamer::DeflatedWriter
   # compressed data written and the CRC32 checksum. The return value
   # can be directly used as the argument to {Streamer#update_last_entry_and_write_data_descriptor}
   #
-  # @param data[String] data to be written
   # @return [Hash] a hash of `{crc32, compressed_size, uncompressed_size}`
   def finish
     @compressed_io << @deflater.finish until @deflater.finished?
-    {crc32: @crc.to_i, compressed_size: @compressed_io.tell, uncompressed_size: @uncompressed_size}
-  end
-
-  private
-
-  def interim_flush
-    return if @bytes_since_last_flush < FLUSH_EVERY_N_BYTES
-    @compressed_io << @deflater.flush
-    @bytes_since_last_flush = 0
+    {crc32: @crc.to_i, compressed_size: @deflater.total_out, uncompressed_size: @deflater.total_in}
+  ensure
+    @deflater.close
   end
 end

data/lib/zip_tricks/streamer/stored_writer.rb

@@ -28,7 +28,6 @@ class ZipTricks::Streamer::StoredWriter
   # Returns the amount of data written and the CRC32 checksum. The return value
   # can be directly used as the argument to {Streamer#update_last_entry_and_write_data_descriptor}
   #
-  # @param data[String] data to be written
   # @return [Hash] a hash of `{crc32, compressed_size, uncompressed_size}`
   def finish
     {crc32: @crc.to_i, compressed_size: @io.tell, uncompressed_size: @io.tell}

data/lib/zip_tricks/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ZipTricks
-  VERSION = '5.3.1'
+  VERSION = '5.4.0'
 end

data/lib/zip_tricks/write_and_tell.rb

@@ -10,9 +10,8 @@ class ZipTricks::WriteAndTell
 
   def <<(bytes)
     return self if bytes.nil?
-    binary_bytes = binary(bytes)
-    @io << binary_bytes
-    @pos += binary_bytes.bytesize
+    @io << bytes.b
+    @pos += bytes.bytesize
     self
   end
 
@@ -23,13 +22,4 @@ class ZipTricks::WriteAndTell
   def tell
     @pos
   end
-
-  private
-
-  def binary(str)
-    return str if str.encoding == Encoding::BINARY
-    str.force_encoding(Encoding::BINARY)
-  rescue RuntimeError # the string is frozen
-    str.dup.force_encoding(Encoding::BINARY)
-  end
 end

data/lib/zip_tricks/zip_writer.rb

@@ -195,7 +195,7 @@ class ZipTricks::ZipWriter
       [TWO_BYTE_MAX_UINT].pack(C_UINT2)
     else
       [0].pack(C_UINT2)
-          end
+    end
     io << [0].pack(C_UINT2)                                # internal file attributes        2 bytes
 
     # Because the add_empty_directory method will create a directory with a trailing "/",

data/zip_tricks.gemspec

@@ -11,7 +11,7 @@ Gem::Specification.new do |spec|
   spec.licenses       = ['MIT (Hippocratic)']
   spec.summary        = 'Stream out ZIP files from Ruby'
   spec.description    = 'Stream out ZIP files from Ruby'
-  spec.homepage       = 'http://github.com/wetransfer/zip_tricks'
+  spec.homepage       = 'https://github.com/wetransfer/zip_tricks'
 
   # Prevent pushing this gem to RubyGems.org.
   # To allow pushes either set the 'allowed_push_host'

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: zip_tricks
 version: !ruby/object:Gem::Version
-  version: 5.3.1
+  version: 5.4.0
 platform: ruby
 authors:
 - Julik Tarkhanov
@@ -11,7 +11,7 @@ authors:
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2020-06-16 00:00:00.000000000 Z
+date: 2020-11-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -262,17 +262,8 @@ files:
 - lib/zip_tricks/write_and_tell.rb
 - lib/zip_tricks/write_buffer.rb
 - lib/zip_tricks/zip_writer.rb
-- qa/README_QA.md
-- qa/generate_test_files.rb
-- qa/in/VTYL8830.jpg
-- qa/in/war-and-peace.txt
-- qa/support.rb
-- qa/test-report-2016-07-28.txt
-- qa/test-report-2016-12-12.txt
-- qa/test-report-2017-04-2.txt
-- qa/test-report.txt
 - zip_tricks.gemspec
-homepage: http://github.com/wetransfer/zip_tricks
+homepage: https://github.com/wetransfer/zip_tricks
 licenses:
 - MIT (Hippocratic)
 metadata:

data/qa/README_QA.md

@@ -1,16 +0,0 @@
-## Manual testing harness for ZipTricks
-
-These tests will generate **very large** files that test various edge cases of ZIP generation. The idea is to generate
-these files and to then try to open them with the unarchiver applications we support. The workflow is as follows:
-
-
-1. Configure your storage to have `zip_tricks` directory linked into your virtual machines and to be on a fast volume (SSD RAID0 is recommended)
-2. Run `generate_test_files.rb`. This will take some time and produce a number of large ZIP files.
-3. Open them with the following ZIP unarchivers:
-  * A recent version of `zipinfo` with the `-tlhvz` flags - to see the information about the file
-  * ArchiveUtility on OSX
-  * The Unarchiver on OSX
-  * Built-in Explorer on Windows 7
-  * 7Zip 9.20 on Windows 7
-  * Any other unarchivers you consider necessary
-* Write down your observations in `test-report.txt` and, when cutting a release, timestamp a copy of that file.