zip_tricks 4.2.2 → 4.2.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 62ea03396a8a75e72000952d6c5ff69b9d8f0a80
-  data.tar.gz: 6cf0bca2293065a1b5430e73d26a9a0117064adb
+  metadata.gz: cc23010c353f0d1118cdbb38649c1363ffe34b7c
+  data.tar.gz: 8be7f24974d25870fd0b9c6852e0ad8fcff1ff0b
 SHA512:
-  metadata.gz: c5ca0c25882b1770b663b199bc6f00b4763691f51f9bc34cd9594442be37e942738c1fbbcf6eff0d8aac56e99a81310f4e691a32aed3a84647e0a8e3d48aa9f2
-  data.tar.gz: 4d53363180343a0017e4054c2d55b40da0c74213a75a3aaf1e339715a6a3ad5ae9b7150abb79685f9cc9d19822d84e9122a12399a75f5488fccad18e35b31357
+  metadata.gz: 15d547d8f049bd7eb71470632c43ccd836d24908d5e98061421caa0af28f8e1747bcbc993906e112fc2874af8b423d879510be2b6a49d1c346f820058f715e11
+  data.tar.gz: b16df9dcbb072b215f866f83cfd073157d21ca2d7e5f336e0226c509b858925e12149f831d816dbf159d041f0cfab27d16bb934692c66b7eefa5bdc048cbb5e1

data/README.md

@@ -3,17 +3,28 @@
 [![Build Status](https://travis-ci.org/WeTransfer/zip_tricks.svg?branch=master)](https://travis-ci.org/WeTransfer/zip_tricks)
 
 Allows streaming, non-rewinding ZIP file output from Ruby.
-Spiritual successor to [zipline](https://github.com/fringd/zipline)
-Requires Ruby 2.1+ syntax support and a working zlib (all available to jRuby as well).
+
+Initially written and as a spiritual successor to [zipline](https://github.com/fringd/zipline)
+and now proudly powering it under the hood.
 
 Allows you to write a ZIP archive out to a File, Socket, String or Array without having to rewind it at any
 point. Usable for creating very large ZIP archives for immediate sending out to clients, or for writing
 large ZIP archives without memory inflation.
 
-## Create a ZIP file without size estimation, compress on-the-fly)
+zip_tricks currently handles all our zipping needs (millions of ZIP files generated per day), so we are
+pretty confident it is widely compatible with a large number of unarchiving end-user applications.
+
+## Requirements
+
+Ruby 2.1+ syntax support (keyword arguments with defaults) and a working zlib (all available to jRuby as well).
+jRuby might experience problems when using the reader methods due to the argument of `IO#seek` being limited
+to [32 bit sizes.](https://github.com/jruby/jruby/issues/3817)
+
+## Create a ZIP file without size estimation, compress on-the-fly during writes
 
-When you compress on the fly and use data descriptors it is not really possible to compute the file size upfront.
-But it is very likely to yield good compression - especially if you send things like CSV files.
+Basic use case is compressing on the fly. Some data will be buffered by the Zlib deflater, but
+memory inflation is going to be very constrained. Data will be written to destination at fairly regular
+intervals. Deflate compression will work best for things like text files.
 
 ```ruby
 out = my_tempfile # can also be a socket
@@ -26,8 +37,10 @@ ZipTricks::Streamer.open(out) do |zip|
   end
 end
 ```
+Unfortunately with this approach it is impossible to compute the size of the ZIP file being output,
+since you do not know how large the compressed data segments are going to be.
 
-## Send the same ZIP file from a Rack response
+## Send a ZIP from a Rack response
 
 Create a `RackBody` object and give it's constructor a block that adds files.
 The block will only be called when actually sending the response to the client
@@ -50,6 +63,12 @@ end
 Use the `SizeEstimator` to compute the correct size of the resulting archive.
 
 ```ruby
+# Precompute the Content-Length ahead of time
+bytesize = ZipTricks::SizeEstimator.estimate do |z|
+ z.add_stored_entry(filename: 'myfile1.bin', size: 9090821)
+ z.add_stored_entry(filename: 'myfile2.bin', size: 458678)
+end
+
 # Prepare the response body. The block will only be called when the response starts to be written.
 zip_body = ZipTricks::RackBody.new do | zip |
   zip.add_stored_entry(filename: "myfile1.bin", size: 9090821, crc32: 12485)
@@ -58,20 +77,9 @@ zip_body = ZipTricks::RackBody.new do | zip |
   zip << read_file('myfile2.bin')
 end
 
-# Precompute the Content-Length ahead of time
-bytesize = ZipTricks::SizeEstimator.estimate do |z|
- z.add_stored_entry(filename: 'myfile1.bin', size: 9090821)
- z.add_stored_entry(filename: 'myfile2.bin', size: 458678)
-end
-
 [200, {'Content-Length' => bytesize.to_s}, zip_body]
 ```
 
-## Other usage examples
-
-Check out the `examples/` directory at the root of the project. This will give you a good idea
-of various use cases the library supports.
-
 ## Writing ZIP files using the Streamer bypass
 
 You do not have to "feed" all the contents of the files you put in the archive through the Streamer object.
@@ -94,6 +102,11 @@ ZipTricks::Streamer.open(io) do | zip |
 end
 ```
 
+## Other usage examples
+
+Check out the `examples/` directory at the root of the project. This will give you a good idea
+of various use cases the library supports.
+
 ## Computing the CRC32 value of a large file
 
 `BlockCRC32` computes the CRC32 checksum of an IO in a streaming fashion.
@@ -110,6 +123,13 @@ crc.to_i # Returns the actual CRC32 value computed so far
 crc.append(precomputed_crc32, size_of_the_blob_computed_from)
 ```
 
+## Reading ZIP files
+
+The library contains a reader module, play with it to see what is possible. It is not a complete ZIP reader
+but it was designed for a specific purpose (highly-parallel unpacking of remotely stored ZIP files), and
+as such it performs it's function quite well. Please beware of the security implications of using ZIP readers
+that have not been formally verified (ours hasn't been).
+
 ## Contributing to zip_tricks
  
 * Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.
@@ -119,6 +139,7 @@ crc.append(precomputed_crc32, size_of_the_blob_computed_from)
 * Commit and push until you are happy with your contribution.
 * Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
 * Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
+* If you alter the `ZipWriter`, please take the time to run the test in the `testing/` directory and verify the generated files do open. You will need fast storage to run those tests.
 
 ## Copyright
 

data/lib/zip_tricks.rb

@@ -1,5 +1,5 @@
 module ZipTricks
-  VERSION = '4.2.2'
+  VERSION = '4.2.3'
   
   # Require all the sub-components except myself
   Dir.glob(__dir__ + '/**/*.rb').sort.each {|p| require p unless p == __FILE__ }

data/lib/zip_tricks/streamer/deflated_writer.rb

@@ -1,21 +1,29 @@
 class ZipTricks::Streamer::DeflatedWriter
+  # After how many bytes of incoming data the deflater for the
+  # contents must be flushed. This is done to prevent unreasonable
+  # memory use when archiving large files.
+  FLUSH_EVERY_N_BYTES = 1024 * 1024 * 5
+  
   def initialize(io)
     @io = io
     @uncompressed_size = 0
     @started_at = @io.tell
     @crc = ZipTricks::StreamCRC32.new
+    @deflater = ::Zlib::Deflate.new(Zlib::DEFAULT_COMPRESSION, -::Zlib::MAX_WBITS)
     @bytes_since_last_flush = 0
   end
 
   def finish
-    ZipTricks::BlockDeflate.write_terminator(@io)
+    @io << @deflater.finish until @deflater.finished?
     [@crc.to_i, @io.tell - @started_at, @uncompressed_size]
   end
   
   def <<(data)
     @uncompressed_size += data.bytesize
-    @io << ZipTricks::BlockDeflate.deflate_chunk(data)
+    @bytes_since_last_flush += data.bytesize
+    @io << @deflater.deflate(data)
     @crc << data
+    interim_flush
     self
   end
 
@@ -23,4 +31,12 @@ class ZipTricks::Streamer::DeflatedWriter
     self << data
     data.bytesize
   end
-end
+  
+  private
+
+  def interim_flush
+    return if @bytes_since_last_flush < FLUSH_EVERY_N_BYTES
+    @io << @deflater.flush
+    @bytes_since_last_flush = 0
+  end
+end

data/zip_tricks.gemspec

@@ -2,16 +2,16 @@
 # DO NOT EDIT THIS FILE DIRECTLY
 # Instead, edit Jeweler::Tasks in Rakefile, and run 'rake gemspec'
 # -*- encoding: utf-8 -*-
-# stub: zip_tricks 4.2.2 ruby lib
+# stub: zip_tricks 4.2.3 ruby lib
 
 Gem::Specification.new do |s|
   s.name = "zip_tricks"
-  s.version = "4.2.2"
+  s.version = "4.2.3"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.require_paths = ["lib"]
   s.authors = ["Julik Tarkhanov"]
-  s.date = "2016-11-13"
+  s.date = "2016-11-18"
   s.description = "Stream out ZIP files from Ruby"
   s.email = "me@julik.nl"
   s.extra_rdoc_files = [

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: zip_tricks
 version: !ruby/object:Gem::Version
-  version: 4.2.2
+  version: 4.2.3
 platform: ruby
 authors:
 - Julik Tarkhanov
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-11-13 00:00:00.000000000 Z
+date: 2016-11-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rubyzip