zip_kit 6.2.1 → 6.2.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f1d33b58f4501d3ddbae7abcab3957fde0549abf734eae72ca1a7ce45601f479
-  data.tar.gz: e9126924e6fe75329237ba551a1a65218676c7d2b3757f4ad91e73eb0bce154e
+  metadata.gz: e1136ebba851638486c9e47150a8d706c49a2bbc0074f457c794582d8ce19089
+  data.tar.gz: 80de3edcb5bc748aaf855a7bf0b1f19439522c8efa4b97754f813fe9413bac2c
 SHA512:
-  metadata.gz: 011e57f856ebe7f625b0bfa5eeb4a240c6c38f2b07ff0434b7e89805516b2d47b6d4230ac404203d00562586909c72d7ea225a7238d47af70fb99ed97b3d50bc
-  data.tar.gz: b68fbaae2e57314c47e7971aeef2150341ba80308c7dee1536718250f5cac01b4b387fe3f73cde5f84fb1ffcd93500343ec451d0fccf9f19b2c0c4a58e74aa2f
+  metadata.gz: 20c5922a4178f2068a4f06388b201bd263f01c387d308c2c6297feba1c05385d601072cae0451d59a4a0b4e1ba1e354a6fa7f622ff1b58daf70947e6991b1e82
+  data.tar.gz: c373972ec6980000b40d1808247759b44f317a7aa3795b406e02005412cf0687e0f2311e5809f011eb1fbc19e6b2b7eb2a6a8f036cafe27a2645f6476cf0c441

data/CHANGELOG.md

@@ -1,3 +1,10 @@
+## 6.2.2
+
+* Make sure "zlib" gets required at the top, as it is used everywhere
+* Improve documentation
+* Make sure `zip_kit_stream` honors the custom `Content-Type` parameter
+* Add a streaming example with Sinatra (and add a Sinatra app to the test harness)
+
 ## 6.2.1
 
 * Make `RailsStreaming` compatible with `ActionController::Live` (previously the response would hang)

data/README.md

@@ -55,9 +55,9 @@ If you want some more conveniences you can also use [zipline](https://github.com
 will automatically process and stream attachments (Carrierwave, Shrine, ActiveStorage) and remote objects
 via HTTP.
 
-`RailsStreaming` will *not* use [ActionController::Live](https://api.rubyonrails.org/classes/ActionController/Live.html)
-and the ZIP output will run in the same thread as your main request. Your testing flows (be it minitest or
-RSpec) should work normally with controller actions returning ZIPs.
+`RailsStreaming` does *not* require [ActionController::Live](https://api.rubyonrails.org/classes/ActionController/Live.html)
+and will stream without it. See {ZipKit::RailsStreaming#zip_kit_stream} for more details on this. You can use it
+together with `Live` just fine if you need to.
 
 ## Writing into streaming destinations
 
@@ -108,11 +108,17 @@ zip.write_file('line_items.csv') do |sink|
 end
 ```
 
-## Create a ZIP file without size estimation, compress on-the-fly during writes
+## Automatic storage mode (stored vs. deflated)
 
-Basic use case is compressing on the fly. Some data will be buffered by the Zlib deflater, but
-memory inflation is going to be very constrained. Data will be written to destination at fairly regular
-intervals. Deflate compression will work best for things like text files. For example, here is how to
+The ZIP file format allows storage in both compressed and raw storage modes. The raw ("stored")
+mode does not require decompression and unarchives faster.
+
+ZipKit will buffer a small amount of output and attempt to compress it using deflate compression.
+If this turns out to be significantly smaller than raw data, it is then going to proceed with
+all further output using deflate compression. Memory use is going to be very modest, but it allows
+you to not have to think about the appropriate storage mode.
+
+Deflate compression will work great for JSONs, CSVs and other text- or text-like formats. For example, here is how to
 output direct to STDOUT (so that you can run `$ ruby archive.rb > file.zip` in your terminal):
 
 ```ruby
@@ -126,8 +132,8 @@ ZipKit::Streamer.open($stdout) do |zip|
 end
 ```
 
-Unfortunately with this approach it is impossible to compute the size of the ZIP file being output,
-since you do not know how large the compressed data segments are going to be.
+If you want to use specific storage modes, use `write_deflated_file` and `write_stored_file` instead of
+`write_file`.
 
 ## Send a ZIP from a Rack response
 
@@ -152,7 +158,11 @@ end
 
 ## Send a ZIP file of known size, with correct headers
 
-Use the `SizeEstimator` to compute the correct size of the resulting archive.
+Sending a file with data descriptors is not always desirable - you don't really know how large your ZIP is going to be.
+If you want to present your users with proper download progress, you would need to set a `Content-Length` header - and
+know ahead of time how large your download is going to be. This can be done with ZipKit, provided you know how large
+the compressed versions of your file are going to be. Use the {ZipKit::SizeEstimator} to do the pre-calculation - it
+is not going to produce any large amounts of output, and will give you a to-the-byte value for your future archive:
 
 ```ruby
 bytesize = ZipKit::SizeEstimator.estimate do |z|

data/examples/sinatra_application.rb

@@ -0,0 +1,16 @@
+require "sinatra/base"
+
+class SinatraApp < Sinatra::Base
+  get "/" do
+    content_type :zip
+    stream do |out|
+      ZipKit::Streamer.open(out) do |z|
+        z.write_file(File.basename(__FILE__)) do |io|
+          File.open(__FILE__, "r") do |f|
+            IO.copy_stream(f, io)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/zip_kit/block_deflate.rb

@@ -1,7 +1,5 @@
 # frozen_string_literal: true
 
-require "zlib"
-
 # Permits Deflate compression in independent blocks. The workflow is as follows:
 #
 # * Run every block to compress through deflate_chunk, remove the header,

data/lib/zip_kit/output_enumerator.rb

@@ -34,6 +34,15 @@ require "time" # for .httpdate
 #
 # to bypass things like `Rack::ETag` and the nginx buffering.
 class ZipKit::OutputEnumerator
+  # With HTTP output it is better to apply a small amount of buffering. While Streamer
+  # output does not buffer at all, the `OutputEnumerator` does as it is going to
+  # be used as a Rack response body. Applying some buffering helps reduce the number
+  # of syscalls for otherwise tiny writes, which relieves the app webserver from
+  # doing too much work managing those writes. While we recommend buffering, the
+  # buffer size is configurable via the constructor - so you can disable buffering
+  # if you really need to. While ZipKit ams not to buffer, in this instance this
+  # buffering is justified. See https://github.com/WeTransfer/zip_tricks/issues/78
+  # for the background on buffering.
   DEFAULT_WRITE_BUFFER_SIZE = 64 * 1024
 
   # Creates a new OutputEnumerator enumerator. The enumerator can be read from using `each`,

data/lib/zip_kit/rails_streaming.rb

@@ -5,14 +5,29 @@ module ZipKit::RailsStreaming
   # Opens a {ZipKit::Streamer} and yields it to the caller. The output of the streamer
   # gets automatically forwarded to the Rails response stream. When the output completes,
   # the Rails response stream is going to be closed automatically.
+  #
+  # Note that there is an important difference in how this method works, depending whether
+  # you use it in a controller which includes `ActionController::Live` vs. one that does not.
+  # With a standard `ActionController` this method will assign a response body, but streaming
+  # will begin when your action method returns. With `ActionController::Live` the streaming
+  # will begin immediately, before the method returns. In all other aspects the method should
+  # stream correctly in both types of controllers.
+  #
+  # If you encounter buffering (streaming does not start for a very long time) you probably
+  # have a piece of Rack middleware in your stack which buffers. Known offenders are `Rack::ContentLength`,
+  # `Rack::MiniProfiler` and `Rack::ETag`. ZipKit will try to work around these but it is not
+  # always possible. If you encounter buffering, examine your middleware stack and try to suss
+  # out whether any middleware might be buffering. You can also try setting `use_chunked_transfer_encoding`
+  # to `true` - this is not recommended but sometimes necessary, for example to bypass `Rack::ContentLength`.
+  #
   # @param filename[String] name of the file for the Content-Disposition header
   # @param type[String] the content type (MIME type) of the archive being output
   # @param use_chunked_transfer_encoding[Boolean] whether to forcibly encode output as chunked. Normally you should not need this.
-  # @param zip_streamer_options[Hash] options that will be passed to the Streamer.
-  #     See {ZipKit::Streamer#initialize} for the full list of options.
+  # @param output_enumerator_options[Hash] options that will be passed to the OutputEnumerator - these include
+  #     options for the Streamer. See {ZipKit::OutputEnumerator#initialize} for the full list of options.
   # @yieldparam [ZipKit::Streamer] the streamer that can be written to
-  # @return [ZipKit::OutputEnumerator] The output enumerator assigned to the response body
-  def zip_kit_stream(filename: "download.zip", type: "application/zip", use_chunked_transfer_encoding: false, **zip_streamer_options, &zip_streaming_blk)
+  # @return [Boolean] always returns true
+  def zip_kit_stream(filename: "download.zip", type: "application/zip", use_chunked_transfer_encoding: false, **output_enumerator_options, &zip_streaming_blk)
     # We want some common headers for file sending. Rails will also set
     # self.sending_file = true for us when we call send_file_headers!
     send_file_headers!(type: type, filename: filename)
@@ -26,38 +41,45 @@ module ZipKit::RailsStreaming
     end
 
     headers = ZipKit::OutputEnumerator.streaming_http_headers
-    response.headers.merge!(headers)
+
+    # Allow Rails headers to override ours. This is important if, for example, a content type gets
+    # set to something else than "application/zip"
+    response.headers.reverse_merge!(headers)
 
     # The output enumerator yields chunks of bytes generated from the Streamer,
-    # with some buffering
-    output_enum = ZipKit::OutputEnumerator.new(**zip_streamer_options, &zip_streaming_blk)
+    # with some buffering. See OutputEnumerator docs for more.
+    rack_zip_body = ZipKit::OutputEnumerator.new(**output_enumerator_options, &zip_streaming_blk)
+
+    # Chunked encoding may be forced if, for example, you _need_ to bypass Rack::ContentLength.
+    # Rack::ContentLength is normally not in a Rails middleware stack, but it might get
+    # introduced unintentionally - for example, "rackup" adds the ContentLength middleware for you.
+    # There is a recommendation to leave the chunked encoding to the app server, so that servers
+    # that support HTTP/2 can use native framing and not have to deal with the chunked encoding,
+    # see https://github.com/julik/zip_kit/issues/7
+    # But it is not to be excluded that a user may need to force the chunked encoding to bypass
+    # some especially pesky Rack middleware that just would not cooperate. Those include
+    # Rack::MiniProfiler and the above-mentioned Rack::ContentLength.
+    if use_chunked_transfer_encoding
+      response.headers["Transfer-Encoding"] = "chunked"
+      rack_zip_body = ZipKit::RackChunkedBody.new(rack_zip_body)
+    end
 
     # Time for some branching, which mostly has to do with the 999 flavours of
     # "how to make both Rails and Rack stream"
     if self.class.ancestors.include?(ActionController::Live)
       # If this controller includes Live it will not work correctly with a Rack
-      # response body assignment - we need to write into the Live output stream instead
+      # response body assignment - the action will just hang. We need to read out the response
+      # body ourselves and write it into the Rails stream.
       begin
-        output_enum.each { |bytes| response.stream.write(bytes) }
+        rack_zip_body.each { |bytes| response.stream.write(bytes) }
       ensure
         response.stream.close
       end
-    elsif use_chunked_transfer_encoding
-      # Chunked encoding may be forced if, for example, you _need_ to bypass Rack::ContentLength.
-      # Rack::ContentLength is normally not in a Rails middleware stack, but it might get
-      # introduced unintentionally - for example, "rackup" adds the ContentLength middleware for you.
-      # There is a recommendation to leave the chunked encoding to the app server, so that servers
-      # that support HTTP/2 can use native framing and not have to deal with the chunked encoding,
-      # see https://github.com/julik/zip_kit/issues/7
-      # But it is not to be excluded that a user may need to force the chunked encoding to bypass
-      # some especially pesky Rack middleware that just would not cooperate. Those include
-      # Rack::MiniProfiler and the above-mentioned Rack::ContentLength.
-      response.headers["Transfer-Encoding"] = "chunked"
-      self.response_body = ZipKit::RackChunkedBody.new(output_enum)
     else
-      # Stream using a Rack body assigned to the ActionController response body, without
-      # doing explicit chunked encoding. See above for the reasoning.
-      self.response_body = output_enum
+      # Stream using a Rack body assigned to the ActionController response body
+      self.response_body = rack_zip_body
     end
+
+    true
   end
 end

data/lib/zip_kit/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ZipKit
-  VERSION = "6.2.1"
+  VERSION = "6.2.2"
 end

data/lib/zip_kit.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require_relative "zip_kit/version"
+require "zlib"
 
 module ZipKit
   autoload :OutputEnumerator, File.dirname(__FILE__) + "/zip_kit/rack_body.rb"

data/rbi/zip_kit.rbi

@@ -1,6 +1,6 @@
 # typed: strong
 module ZipKit
-  VERSION = T.let("6.2.1", T.untyped)
+  VERSION = T.let("6.2.2", T.untyped)
 
   # A ZIP archive contains a flat list of entries. These entries can implicitly
   # create directories when the archive is expanded. For example, an entry with
@@ -1979,25 +1979,39 @@ end, T.untyped)
     # gets automatically forwarded to the Rails response stream. When the output completes,
     # the Rails response stream is going to be closed automatically.
     # 
+    # Note that there is an important difference in how this method works, depending whether
+    # you use it in a controller which includes `ActionController::Live` vs. one that does not.
+    # With a standard `ActionController` this method will assign a response body, but streaming
+    # will begin when your action method returns. With `ActionController::Live` the streaming
+    # will begin immediately, before the method returns. In all other aspects the method should
+    # stream correctly in both types of controllers.
+    # 
+    # If you encounter buffering (streaming does not start for a very long time) you probably
+    # have a piece of Rack middleware in your stack which buffers. Known offenders are `Rack::ContentLength`,
+    # `Rack::MiniProfiler` and `Rack::ETag`. ZipKit will try to work around these but it is not
+    # always possible. If you encounter buffering, examine your middleware stack and try to suss
+    # out whether any middleware might be buffering. You can also try setting `use_chunked_transfer_encoding`
+    # to `true` - this is not recommended but sometimes necessary, for example to bypass `Rack::ContentLength`.
+    # 
     # _@param_ `filename` — name of the file for the Content-Disposition header
     # 
     # _@param_ `type` — the content type (MIME type) of the archive being output
     # 
     # _@param_ `use_chunked_transfer_encoding` — whether to forcibly encode output as chunked. Normally you should not need this.
     # 
-    # _@param_ `zip_streamer_options` — options that will be passed to the Streamer. See {ZipKit::Streamer#initialize} for the full list of options.
+    # _@param_ `output_enumerator_options` — options that will be passed to the OutputEnumerator - these include options for the Streamer. See {ZipKit::OutputEnumerator#initialize} for the full list of options.
     # 
-    # _@return_ — The output enumerator assigned to the response body
+    # _@return_ — always returns true
     sig do
       params(
         filename: String,
         type: String,
         use_chunked_transfer_encoding: T::Boolean,
-        zip_streamer_options: T::Hash[T.untyped, T.untyped],
+        output_enumerator_options: T::Hash[T.untyped, T.untyped],
         zip_streaming_blk: T.proc.params(the: ZipKit::Streamer).void
-      ).returns(ZipKit::OutputEnumerator)
+      ).returns(T::Boolean)
     end
-    def zip_kit_stream(filename: "download.zip", type: "application/zip", use_chunked_transfer_encoding: false, **zip_streamer_options, &zip_streaming_blk); end
+    def zip_kit_stream(filename: "download.zip", type: "application/zip", use_chunked_transfer_encoding: false, **output_enumerator_options, &zip_streaming_blk); end
   end
 
   # The output enumerator makes it possible to "pull" from a ZipKit streamer

data/zip_kit.gemspec

@@ -42,5 +42,6 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency "puma"
   spec.add_development_dependency "actionpack", "~> 5" # For testing RailsStreaming against an actual Rails controller
   spec.add_development_dependency "nokogiri", "~> 1", ">= 1.13" # Rails 5 does by mistake use an older Nokogiri otherwise
+  spec.add_development_dependency "sinatra"
   spec.add_development_dependency "sord"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: zip_kit
 version: !ruby/object:Gem::Version
-  version: 6.2.1
+  version: 6.2.2
 platform: ruby
 authors:
 - Julik Tarkhanov
@@ -12,7 +12,7 @@ authors:
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2024-03-23 00:00:00.000000000 Z
+date: 2024-03-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -250,6 +250,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '1.13'
+- !ruby/object:Gem::Dependency
+  name: sinatra
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: sord
   requirement: !ruby/object:Gem::Requirement
@@ -292,6 +306,7 @@ files:
 - examples/parallel_compression_with_block_deflate.rb
 - examples/rack_application.rb
 - examples/s3_upload.rb
+- examples/sinatra_application.rb
 - lib/zip_kit.rb
 - lib/zip_kit/block_deflate.rb
 - lib/zip_kit/block_write.rb
@@ -343,7 +358,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.3.7
+rubygems_version: 3.1.6
 signing_key:
 specification_version: 4
 summary: Stream out ZIP files from Ruby. Successor to zip_tricks.