zip_kit 6.0.0

data/IMPLEMENTATION_DETAILS.md

@@ -0,0 +1,97 @@
+# Implementation details
+
+The ZipKit streaming implementation is designed around the following requirements:
+
+* Only ahead-writes (no IO seek or rewind)
+* Automatic switching to Zip64 as the files get written (no IO seeks), but not requiring Zip64 support if the archive can do without
+* Make use of the fact that CRC32 checksums and the sizes of the files (compressed _and_ uncompressed) are known upfront
+
+It strives to be compatible with the following unzip programs _at the minimum:_
+
+* OSX - builtin ArchiveUtility (except the Zip64 support when files larger than 4GB are in the archive)
+* OSX - The Unarchiver, at least 3.10.1
+* Windows 7 - built-in Explorer zip browser (except for Unicode filenames which it just doesn't support)
+* Windows 7 - 7Zip 9.20
+
+Below is the list of _specific_ decisions taken when writing the implementation, with an explanation for each.
+We specifically _omit_ a number of things that we could do, but that are not necessary to satisfy our objectives.
+The omissions are _intentional_ since we do not want to have things of which we _assume_ they work, or have things
+that work only for one obscure unarchiver in one obscure case (like WinRAR with chinese filenames).
+
+## Data descriptors (postfix CRC32/file sizes)
+
+Data descriptors permit you to generate "postfix" ZIP files (where you write the local file header without having to
+know the CRC32 and the file size upfront, then write the compressed file data, and only then - once you know what your CRC32,
+compressed and uncompressed sizes are etc. - write them into a data descriptor that follows the file data.
+
+The streamer has optional support for data descriptors. Their use can apparently [ be problematic](https://github.com/thejoshwolfe/yazl/issues/13)
+with the 7Zip version that we want to support, but in our tests everything worked fine.
+
+For more info see https://github.com/thejoshwolfe/yazl#general-purpose-bit-flag
+
+## Zip64 support
+
+Zip64 support switches on _by itself_, automatically, when _any_ of the following conditions is met:
+
+* The start of the central directory lies beyound the 4GB limit
+* The ZIP archive has more than 65535 files added to it
+* Any entry is present whose compressed _or_ uncompressed size is above 4GB
+
+When writing out local file headers, the Zip64 extra field (and related changes to the standard fields) are
+_only_ performed if one of the file sizes is larger than 4GB. Otherwise the Zip64 extra will _only_ be
+written in the central directory entry, but not in the local file header.
+
+This has to do with the fact that otherwise we would write Zip64 extra fields for all local file headers,
+regardless whether the file actually requires Zip64 or not. That might impede some older tools from reading
+the archive, which is a problem you don't want to have if your archive otherwise fits perfectly below all
+the Zip64 thresholds.
+
+To be compatible with Windows7 built-in tools, the Zip64 extra field _must_ be written as _the first_ extra
+field, any other extra fields should come after.
+
+## International filename support and the Info-ZIP extra field
+
+If a diacritic-containing character (such as å) does fit into the DOS-437
+codepage, it should be encodable as such. This would, in theory, let older Windows tools
+decode the filename correctly. However, this kills the filename decoding for the OSX builtin
+archive utility (it assumes the filename to be UTF-8, regardless). So if we allow filenames
+to be encoded in DOS-437, we _potentially_ have support in Windows but we upset everyone on Mac.
+If we just use UTF-8 and set the right EFS bit in general purpose flags, we upset Windows users
+because most of the Windows unarchive tools (at least the builtin ones) do not give a flying eff
+about the EFS support bit being set.
+
+Additionally, if we use Unarchiver on OSX (which is our recommended unpacker for large files),
+it will (very rightfully) ask us how we should decode each filename that does not have the EFS bit,
+but does contain something non-ASCII-decodable. This is horrible UX for users.
+
+So, basically, we have 2 choices, for filenames containing diacritics (for bona-fide UTF-8 you do not
+even get those choices, you _have_ to use UTF-8):
+
+* Make life easier for Windows users by setting stuff to DOS, not care about the standard _and_ make
+  most of Mac users upset
+* Make life easy for Mac users and conform to the standard, and tell Windows users to get a _decent_
+  ZIP unarchiving tool.
+
+We are going with option 2, and this is well-thought-out. Trust me. If you want the crazytown
+filename encoding scheme that is described here http://stackoverflow.com/questions/13261347
+you can try this:
+
+   [Encoding::CP437, Encoding::ISO_8859_1, Encoding::UTF_8]
+
+We don't want no such thing, and sorry Windows users, you are going to need a decent unarchiver
+that honors the standard. Alas, alas.
+
+Additionally, the tests with the unarchivers we _do_ support have shown that including the InfoZIP
+extra field does not actually help any of them recognize the file name correctly. And the use of
+those fields for the UTF-8 filename, per spec, tells us we should not set the EFS bit - which ruins
+the unarchiving for all other solutions. As any other, this decision may be changed in the future.
+
+There are some interesting notes about the Info-ZIP/EFS combination here
+https://commons.apache.org/proper/commons-compress/zip.html
+
+## Directory support
+
+ZIP offers the possibility to store empty directories (folders). The directories that contain files, however, get
+created automatically at unarchive time.  If you store a file, called, say, `docs/item.doc` then the unarchiver will
+automatically create the `docs` directory if it doesn't exist already. So you need to use the directory creation
+methods only if you do not have any files in those directories.

data/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2024 Julik Tarkhanov
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,234 @@
+# zip_kit
+
+Allows streaming, non-rewinding ZIP file output from Ruby.
+
+`zip_kit` is a successor to and continuation of [zip_tricks](https://github.com/WeTransfer/zip_tricks), which
+was inspired by [zipline](https://github.com/fringd/zipline). I am grateful to WeTransfer for allowing me
+to develop zip_tricks and for sharing it with the community.
+
+Allows you to write a ZIP archive out to a `File`, `Socket`, `String` or `Array` without having to rewind it at any
+point. Usable for creating very large ZIP archives for immediate sending out to clients, or for writing
+large ZIP archives without memory inflation.
+
+The original gem (zip_tricks) handled all the zipping needs (millions of ZIP files generated per day),
+for a large file transfer service, so we are pretty confident it is widely compatible with a large number
+of unarchiving end-user applications and is well tested.
+
+## Requirements
+
+Ruby 2.6+ syntax support is required, as well as a a working zlib (all available to jRuby as well).
+
+## Diving in: send some large CSV reports from Rails
+
+The easiest is to include the `ZipKit::RailsStreaming` module into your
+controller. You will then have a `zip_kit_stream` method available which accepts a block:
+
+```ruby
+class ZipsController < ActionController::Base
+  include ZipKit::RailsStreaming
+
+  def download
+    zip_kit_stream do |zip|
+      zip.write_file('report1.csv') do |sink|
+        CSV(sink) do |csv_write|
+          csv_write << Person.column_names
+          Person.all.find_each do |person|
+            csv_write << person.attributes.values
+          end
+        end
+      end
+      zip.write_file('report2.csv') do |sink|
+        ...
+      end
+    end
+  end
+end
+```
+
+The `write_file` method will use some heuristics to determine whether your output file would benefit
+from compression, and pick the appropriate storage mode for the file accordingly.
+
+If you want some more conveniences you can also use [zipline](https://github.com/fringd/zipline) which
+will automatically process and stream attachments (Carrierwave, Shrine, ActiveStorage) and remote objects
+via HTTP.
+
+`RailsStreaming` will *not* use [ActionController::Live](https://api.rubyonrails.org/classes/ActionController/Live.html)
+and the ZIP output will run in the same thread as your main request. Your testing flows (be it minitest or
+RSpec) should work normally with controller actions returning ZIPs.
+
+## Writing into other streaming destinations
+
+Any object that accepts bytes via either `<<` or `write` methods can be a write destination. For example, here
+is how to upload a sizeable ZIP to S3 - the SDK will happily chop your upload into multipart upload parts:
+
+```ruby
+bucket = Aws::S3::Bucket.new("mybucket")
+obj = bucket.object("big.zip")
+obj.upload_stream do |write_stream|
+  ZipKit::Streamer.open(write_stream) do |zip|
+    zip.write_file("large.csv") do |sink|
+      CSV(sink) do |csv|
+        csv << ["Line", "Item"]
+        20_000.times do |n|
+          csv << [n, "Item number #{n}"]
+        end
+      end
+    end
+  end
+end
+```
+
+# Writing through an intermediary object
+
+Any object that writes using either `<<` or `write` can write into a `sink`. For example, you can do streaming
+output with [builder](https://github.com/jimweirich/builder#project-builder)
+
+```ruby
+zip.write_file('report1.csv') do |sink|
+  builder = Builder::XmlMarkup.new(target: sink, indent: 2)
+  builder.people do
+    Person.all.find_each do |person|
+      builder.person(name: person.name)
+    end
+  end
+end
+```
+
+and this output will be compressed and output into the ZIP file on the fly. zip_kit composes with any
+Ruby code that streams its output into a destination.
+
+## Create a ZIP file without size estimation, compress on-the-fly during writes
+
+Basic use case is compressing on the fly. Some data will be buffered by the Zlib deflater, but
+memory inflation is going to be very constrained. Data will be written to destination at fairly regular
+intervals. Deflate compression will work best for things like text files.
+
+```ruby
+out = my_tempfile # can also be a socket
+ZipKit::Streamer.open(out) do |zip|
+  zip.write_file('mov.mp4.txt') do |sink|
+    File.open('mov.mp4', 'rb'){|source| IO.copy_stream(source, sink) }
+  end
+  zip.write_file('long-novel.txt') do |sink|
+    File.open('novel.txt', 'rb'){|source| IO.copy_stream(source, sink) }
+  end
+end
+```
+Unfortunately with this approach it is impossible to compute the size of the ZIP file being output,
+since you do not know how large the compressed data segments are going to be.
+
+## Send a ZIP from a Rack response
+
+zip_kit provides an `OutputEnumerator` object which will yield the binary chunks piece
+by piece, and apply some amount of buffering as well. Make sure to also wrap your `OutputEnumerator` in a chunker
+by calling `#to_chunked` on it. Return it to your webserver and you will have your ZIP streamed!
+The block that you give to the `OutputEnumerator` receive the {ZipKit::Streamer} object and will only
+start executing once your response body starts getting iterated over - when actually sending
+the response to the client (unless you are using a buffering Rack webserver, such as Webrick).
+
+```ruby
+body = ZipKit::OutputEnumerator.new do | zip |
+  zip.write_file('mov.mp4') do |sink|
+    File.open('mov.mp4', 'rb'){|source| IO.copy_stream(source, sink) }
+  end
+  zip.write_file('long-novel.txt') do |sink|
+    File.open('novel.txt', 'rb'){|source| IO.copy_stream(source, sink) }
+  end
+end
+
+headers, streaming_body = body.to_rack_response_headers_and_body(env)
+[200, headers, streaming_body]
+```
+
+## Send a ZIP file of known size, with correct headers
+
+Use the `SizeEstimator` to compute the correct size of the resulting archive.
+
+```ruby
+# Precompute the Content-Length ahead of time
+bytesize = ZipKit::SizeEstimator.estimate do |z|
+ z.add_stored_entry(filename: 'myfile1.bin', size: 9090821)
+ z.add_stored_entry(filename: 'myfile2.bin', size: 458678)
+end
+
+# Prepare the response body. The block will only be called when the response starts to be written.
+zip_body = ZipKit::OutputEnumerator.new do | zip |
+  zip.add_stored_entry(filename: "myfile1.bin", size: 9090821, crc32: 12485)
+  zip << read_file('myfile1.bin')
+  zip.add_stored_entry(filename: "myfile2.bin", size: 458678, crc32: 89568)
+  zip << read_file('myfile2.bin')
+end
+
+headers, streaming_body = body.to_rack_response_headers_and_body(env, content_length: bytesize)
+[200, headers, streaming_body]
+```
+
+## Writing ZIP files using the Streamer bypass
+
+You do not have to "feed" all the contents of the files you put in the archive through the Streamer object.
+If the write destination for your use case is a `Socket` (say, you are writing using Rack hijack) and you know
+the metadata of the file upfront (the CRC32 of the uncompressed file and the sizes), you can write directly
+to that socket using some accelerated writing technique, and only use the Streamer to write out the ZIP metadata.
+
+```ruby
+# io has to be an object that supports #<< or #write()
+ZipKit::Streamer.open(io) do | zip |
+  # raw_file is written "as is" (STORED mode).
+  # Write the local file header first..
+  zip.add_stored_entry(filename: "first-file.bin", size: raw_file.size, crc32: raw_file_crc32)
+
+  # Adjust the ZIP offsets within the Streamer
+  zip.simulate_write(my_temp_file.size)
+
+  # ...and then send the actual file contents bypassing the Streamer interface
+  io.sendfile(my_temp_file)
+end
+```
+
+## Other usage examples
+
+Check out the `examples/` directory at the root of the project. This will give you a good idea
+of various use cases the library supports.
+
+### Computing the CRC32 value of a large file
+
+`BlockCRC32` computes the CRC32 checksum of an IO in a streaming fashion.
+It is slightly more convenient for the purpose than using the raw Zlib library functions.
+
+```ruby
+crc = ZipKit::StreamCRC32.new
+crc << next_chunk_of_data
+...
+
+crc.to_i # Returns the actual CRC32 value computed so far
+...
+# Append a known CRC32 value that has been computed previosuly
+crc.append(precomputed_crc32, size_of_the_blob_computed_from)
+```
+
+You can also compute the CRC32 for an entire IO object if it responds to `#eof?`:
+
+```ruby
+crc = ZipKit::StreamCRC32.from_io(file) # Returns an Integer
+```
+
+### Reading ZIP files
+
+The library contains a reader module, play with it to see what is possible. It is not a complete ZIP reader
+but it was designed for a specific purpose (highly-parallel unpacking of remotely stored ZIP files), and
+as such it performs it's function quite well. Please beware of the security implications of using ZIP readers
+that have not been formally verified (ours hasn't been).
+
+## Contributing to zip_kit
+
+* Check out the latest `main` to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.
+* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it.
+* Fork the project.
+* Start a feature/bugfix branch.
+* Commit and push until you are happy with your contribution.
+* Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
+* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
+
+## Copyright
+
+Copyright (c) 2024 Julik Tarkhanov. See LICENSE.txt for further details.

data/Rakefile

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+require "yard"
+require "rubocop/rake_task"
+require "standard/rake"
+
+task :format do
+  `bundle exec standardrb --fix-unsafely`
+  `bundle exec magic_frozen_string_literal ./lib`
+end
+
+YARD::Rake::YardocTask.new(:doc) do |t|
+  # The dash has to be between the two to "divide" the source files and
+  # miscellaneous documentation files that contain no code
+  t.files = ["lib/**/*.rb", "-", "LICENSE.txt", "IMPLEMENTATION_DETAILS.md"]
+end
+
+RSpec::Core::RakeTask.new(:spec)
+task default: [:spec, :standard]

data/bench/buffered_crc32_bench.rb

@@ -0,0 +1,109 @@
+require "bundler"
+Bundler.setup
+
+require "benchmark"
+require "benchmark/ips"
+require_relative "../lib/zip_kit"
+
+n_bytes = 5 * 1024 * 1024
+r = Random.new
+bytes = (0...n_bytes).map { r.bytes(1) }
+buffer_sizes = [
+  1,
+  256,
+  512,
+  1024,
+  8 * 1024,
+  16 * 1024,
+  32 * 1024,
+  64 * 1024,
+  128 * 1024,
+  256 * 1024,
+  512 * 1024,
+  1024 * 1024,
+  2 * 1024 * 1024
+]
+
+Benchmark.ips do |x|
+  x.config(time: 5, warmup: 2)
+  buffer_sizes.each do |buf_size|
+    x.report "Single-byte <<-writes of #{n_bytes} using a #{buf_size} byte buffer" do
+      crc = ZipKit::WriteBuffer.new(ZipKit::StreamCRC32.new, buf_size)
+      bytes.each { |b| crc << b }
+      crc.to_i
+    end
+  end
+  x.compare!
+end
+
+__END__
+
+Warming up --------------------------------------
+Single-byte <<-writes of 5242880 using a 1 byte buffer
+                         1.000  i/100ms
+Single-byte <<-writes of 5242880 using a 256 byte buffer
+                         1.000  i/100ms
+Single-byte <<-writes of 5242880 using a 512 byte buffer
+                         1.000  i/100ms
+Single-byte <<-writes of 5242880 using a 1024 byte buffer
+                         1.000  i/100ms
+Single-byte <<-writes of 5242880 using a 8192 byte buffer
+                         1.000  i/100ms
+Single-byte <<-writes of 5242880 using a 16384 byte buffer
+                         1.000  i/100ms
+Single-byte <<-writes of 5242880 using a 32768 byte buffer
+                         1.000  i/100ms
+Single-byte <<-writes of 5242880 using a 65536 byte buffer
+                         1.000  i/100ms
+Single-byte <<-writes of 5242880 using a 131072 byte buffer
+                         1.000  i/100ms
+Single-byte <<-writes of 5242880 using a 262144 byte buffer
+                         1.000  i/100ms
+Single-byte <<-writes of 5242880 using a 524288 byte buffer
+                         1.000  i/100ms
+Single-byte <<-writes of 5242880 using a 1048576 byte buffer
+                         1.000  i/100ms
+Single-byte <<-writes of 5242880 using a 2097152 byte buffer
+                         1.000  i/100ms
+Calculating -------------------------------------
+Single-byte <<-writes of 5242880 using a 1 byte buffer
+                          0.054  (± 0.0%) i/s -      1.000  in  18.383019s
+Single-byte <<-writes of 5242880 using a 256 byte buffer
+                          0.121  (± 0.0%) i/s -      1.000  in   8.286061s
+Single-byte <<-writes of 5242880 using a 512 byte buffer
+                          0.124  (± 0.0%) i/s -      1.000  in   8.038112s
+Single-byte <<-writes of 5242880 using a 1024 byte buffer
+                          0.128  (± 0.0%) i/s -      1.000  in   7.828562s
+Single-byte <<-writes of 5242880 using a 8192 byte buffer
+                          0.123  (± 0.0%) i/s -      1.000  in   8.121586s
+Single-byte <<-writes of 5242880 using a 16384 byte buffer
+                          0.127  (± 0.0%) i/s -      1.000  in   7.872240s
+Single-byte <<-writes of 5242880 using a 32768 byte buffer
+                          0.126  (± 0.0%) i/s -      1.000  in   7.911816s
+Single-byte <<-writes of 5242880 using a 65536 byte buffer
+                          0.126  (± 0.0%) i/s -      1.000  in   7.917318s
+Single-byte <<-writes of 5242880 using a 131072 byte buffer
+                          0.127  (± 0.0%) i/s -      1.000  in   7.897223s
+Single-byte <<-writes of 5242880 using a 262144 byte buffer
+                          0.130  (± 0.0%) i/s -      1.000  in   7.675608s
+Single-byte <<-writes of 5242880 using a 524288 byte buffer
+                          0.130  (± 0.0%) i/s -      1.000  in   7.679886s
+Single-byte <<-writes of 5242880 using a 1048576 byte buffer
+                          0.128  (± 0.0%) i/s -      1.000  in   7.788439s
+Single-byte <<-writes of 5242880 using a 2097152 byte buffer
+                          0.128  (± 0.0%) i/s -      1.000  in   7.797839s
+
+Comparison:
+Single-byte <<-writes of 5242880 using a 262144 byte buffer:        0.1 i/s
+Single-byte <<-writes of 5242880 using a 524288 byte buffer:        0.1 i/s - 1.00x  slower
+Single-byte <<-writes of 5242880 using a 1048576 byte buffer:        0.1 i/s - 1.01x  slower
+Single-byte <<-writes of 5242880 using a 2097152 byte buffer:        0.1 i/s - 1.02x  slower
+Single-byte <<-writes of 5242880 using a 1024 byte buffer:        0.1 i/s - 1.02x  slower
+Single-byte <<-writes of 5242880 using a 16384 byte buffer:        0.1 i/s - 1.03x  slower
+Single-byte <<-writes of 5242880 using a 131072 byte buffer:        0.1 i/s - 1.03x  slower
+Single-byte <<-writes of 5242880 using a 32768 byte buffer:        0.1 i/s - 1.03x  slower
+Single-byte <<-writes of 5242880 using a 65536 byte buffer:        0.1 i/s - 1.03x  slower
+Single-byte <<-writes of 5242880 using a 512 byte buffer:        0.1 i/s - 1.05x  slower
+Single-byte <<-writes of 5242880 using a 8192 byte buffer:        0.1 i/s - 1.06x  slower
+Single-byte <<-writes of 5242880 using a 256 byte buffer:        0.1 i/s - 1.08x  slower
+Single-byte <<-writes of 5242880 using a 1 byte buffer:        0.1 i/s - 2.39x  slower

data/examples/archive_size_estimate.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+require_relative "../lib/zip_kit"
+
+# Predict how large a ZIP file is going to be without having access to
+# the actual file contents, but using just the filenames (influences the
+# file size) and the size of the files
+zip_archive_size_in_bytes = ZipKit::SizeEstimator.estimate { |zip|
+  # Pretend we are going to make a ZIP file which contains a few
+  # MP4 files (those do not compress all too well)
+  zip.add_stored_entry(filename: "MOV_1234.MP4", size: 898_090)
+  zip.add_stored_entry(filename: "MOV_1235.MP4", size: 7_855_126)
+}
+
+puts zip_archive_size_in_bytes #=> 8_753_467

data/examples/config.ru

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+require File.dirname(__FILE__) + "/rack_application.rb"
+
+# Demonstrates a Rack app that can offer a ZIP download composed
+# at runtime (see rack_application.rb)
+run ZipDownload.new

data/examples/deferred_write.rb

@@ -0,0 +1,58 @@
+# frozen_string_literal: true
+
+require_relative "../lib/zip_kit"
+
+# Using deferred writes (when you want to "pull" from a Streamer)
+# is also possible with ZipKit.
+#
+# The OutputEnumerator class instead of Streamer is very useful for this
+# particular purpose. It does not start the archiving immediately,
+# but waits instead until you start pulling data out of it.
+#
+# Let's make a OutputEnumerator that writes a few files with random content. Note that when you create
+# that body it does not immediately write the ZIP:
+iterable = ZipKit::Streamer.output_enum { |zip|
+  (1..5).each do |i|
+    zip.write_stored_file("random_%d04d.bin" % i) do |sink|
+      warn "Starting on file #{i}...\n"
+      sink << Random.new.bytes(1024)
+    end
+  end
+}
+
+warn "\n\nOutput using #each"
+
+# Now we can treat the iterable as any Ruby enumerable object, since
+# it supports #each yielding every binary string output by the Streamer.
+# Only when we start using each() will the ZIP start generating. Just using
+# each() like we do here runs the archiving procedure to completion. See how
+# the output of the block within OutputEnumerator is interspersed with the stuff
+# being yielded to each():
+iterable.each do |_binary_string|
+  $stderr << "."
+end
+
+warn "\n\nOutput Enumerator returned from #each"
+
+# We now have output the entire archive, so using each() again
+# will restart the block we gave it. For example, we can user
+# an Enumerator - via enum_for - to "take" chunks of output when
+# we find necessary:
+enum = iterable.each
+15.times do
+  _bin_str = enum.next # Obtain the subsequent chunk of the ZIP
+  $stderr << "*"
+end
+
+# ... or a Fiber
+
+warn "\n\nOutput using a Fiber"
+fib = Fiber.new {
+  iterable.each do |binary_string|
+    $stderr << "•"
+    _next_iteration = Fiber.yield(binary_string)
+  end
+}
+15.times do
+  fib.resume # Process the subsequent chunk of the ZIP
+end

data/examples/parallel_compression_with_block_deflate.rb

@@ -0,0 +1,86 @@
+# frozen_string_literal: true
+
+require_relative "../lib/zip_kit"
+require "tempfile"
+
+# This shows how to perform compression in parallel (a-la pigz, but in a less
+# advanced fashion since the compression tables are not shared - to
+# minimize shared state).
+#
+# When using this approach, compressing a large file can be performed as a
+# map-reduce operation.
+# First you prepare all the data per part of your (potentially very large) file,
+# and then you use the reduce task to combine that data into one linear zip.
+# In this example we will generate threads and collect their return values in
+# the order the threads were launched, which guarantees a consistent reduce.
+#
+# So, let each thread generate a part of the file, and also
+# compute the CRC32 of it. The thread will compress it's own part
+# as well, in an independent deflate segment - the threads do not share
+# anything. You could also multiplex this over multiple processes or
+# even machines.
+threads = (0..12).map {
+  Thread.new do
+    source_tempfile = Tempfile.new "t"
+    source_tempfile.binmode
+
+    # Fill the part with random content
+    12.times { source_tempfile << Random.new.bytes(1 * 1024 * 1024) }
+    source_tempfile.rewind
+
+    # Compute the CRC32 of the source file
+    part_crc = ZipKit::StreamCRC32.from_io(source_tempfile)
+    source_tempfile.rewind
+
+    # Create a compressed part
+    compressed_tempfile = Tempfile.new("tc")
+    compressed_tempfile.binmode
+    ZipKit::BlockDeflate.deflate_in_blocks(source_tempfile,
+      compressed_tempfile)
+
+    source_tempfile.close!
+    # The data that the splicing process needs.
+    [compressed_tempfile, part_crc, source_tempfile.size]
+  end
+}
+
+# Threads return us a tuple with [compressed_tempfile, source_part_size,
+# source_part_crc]
+compressed_tempfiles_and_crc_of_parts = threads.map(&:join).map(&:value)
+
+# Now we need to compute the CRC32 of the _entire_ file, and it has to be
+# the CRC32 of the _source_ file (uncompressed), not of the compressed variant.
+# Handily we know
+entire_file_crc = ZipKit::StreamCRC32.new
+compressed_tempfiles_and_crc_of_parts.each do |_, source_part_crc, source_part_size|
+  entire_file_crc.append(source_part_crc, source_part_size)
+end
+
+# We need to append the the terminator bytes to the end of the last part.
+last_compressed_part = compressed_tempfiles_and_crc_of_parts[-1][0]
+ZipKit::BlockDeflate.write_terminator(last_compressed_part)
+
+# and we need to know how big the deflated segment of the ZIP is going to be, in total.
+# To figure that out we just sum the sizes of the files
+compressed_part_files = compressed_tempfiles_and_crc_of_parts.map(&:first)
+size_of_deflated_segment = compressed_part_files.map(&:size).inject(&:+)
+size_of_uncompressed_file = compressed_tempfiles_and_crc_of_parts.map { |e| e[2] }.inject(&:+)
+
+# And now we can create a ZIP with our compressed file in it's entirety.
+# We use a File as a destination here, but you can also use a socket or a
+# non-rewindable IO. ZipKit never needs to rewind your output, since it is
+# made for streaming.
+output = File.open("zip_created_in_parallel.zip", "wb")
+
+ZipKit::Streamer.open(output) do |zip|
+  zip.add_deflated_entry("parallel.bin",
+    size_of_uncompressed_file,
+    entire_file_crc.to_i,
+    size_of_deflated_segment)
+  compressed_part_files.each do |part_file|
+    part_file.rewind
+    while (blob = part_file.read(2048))
+      zip << blob
+    end
+  end
+end