tus-server 0.10.2 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: a8418797bd1a0d54cfe083534e48a4b014763cf1
-  data.tar.gz: c56a965e748adbd182700f38d32f8a4a530efee8
+  metadata.gz: 9dfd9cb38d7ef46fe3ac684a7ee3569f66e97c84
+  data.tar.gz: 19b75e3d0c222d47c46432f1fd198ab9e27f56bb
 SHA512:
-  metadata.gz: 338316cbd7d8ae243e40a15bbca5b04070b70b6973ad618da9f7524a58af2586254eb73415ec6830d26bf43d470a13eb975b3cc9cf170a915ec8aed0d2309982
-  data.tar.gz: 053cd94a93e0b2f57914240c45c9ed50494a6191acfb62f2b1c01d4331bc6047a249d4dff65367dc8bc29bef15aad9a4f7ff0b8e1bc85edb997bc96b755b6f79
+  metadata.gz: 48c902ba0c5ff439869a93b3b4fa851fe17644c7a41978b25041f4955b5c0697e9a725ece84a5ee1d294acd7c631ad04b9d4348f327a64c1ca0002549b1db509
+  data.tar.gz: 482543723d79aeb1df06a9fb429d8f39584e4891905e795c32df660f860413ffcac4408892114a85c67e4b1c867e1d6b9462157d6f495c9399d3d641e921f6b8

data/README.md

@@ -19,8 +19,7 @@ gem "tus-server"
 
 Tus-ruby-server provides a `Tus::Server` Roda app, which you can run in your
 `config.ru`. That way you can run `Tus::Server` both as a standalone app or as
-part of your main app (though it's recommended to run it as a standalone app,
-as explained in the "Performance considerations" section of this README).
+part of your main app.
 
 ```rb
 # config.ru
@@ -33,6 +32,9 @@ end
 run YourApp
 ```
 
+While this is the most flexible option, it's not optimal in terms of
+performance; see the [Goliath](#goliath) section for an alternative approach.
+
 Now you can tell your tus client library (e.g. [tus-js-client]) to use this
 endpoint:
 
@@ -40,7 +42,7 @@ endpoint:
 // using tus-js-client
 new tus.Upload(file, {
   endpoint: "http://localhost:9292/files",
-  chunkSize: 5*1024*1024, // 5MB
+  chunkSize: 5*1024*1024, // required unless using Goliath
   // ...
 })
 ```
@@ -49,6 +51,42 @@ After the upload is complete, you'll probably want to attach the uploaded file
 to a database record. [Shrine] is one file attachment library that integrates
 nicely with tus-ruby-server, see [shrine-tus-demo] for an example integration.
 
+### Goliath
+
+[Goliath] is the ideal web server to run tus-ruby-server on, because by
+utilizing [EventMachine] it's asnychronous both in reading the request body and
+writing to the response body, so it's not affected by slow clients. Goliath
+also allows tus-ruby-server to handle interrupted requests, by saving data that
+has been uploaded until the interruption. This means that with Goliath it's
+**not** mandatory for client to chunk the upload into multiple requests in
+order to achieve resumable uploads, which would be the case for most other web
+servers.
+
+Tus-ruby-server ships with Goliath integration, you just need to require it in
+a Ruby file and run that file, and that will automatically start up Goliath.
+
+```rb
+# Gemfile
+gem "tus-server", "~> 1.0"
+gem "goliath"
+gem "async-rack", ">= 0.5.1"
+```
+```rb
+# tus.rb
+require "tus/server/goliath"
+
+# any additional Tus::Server configuration you want to put in here
+```
+```sh
+$ ruby tus.rb --stdout # enable logging
+```
+
+Any options provided after the Ruby file will be passed in to the Goliath
+server, see [this wiki][goliath server options] for all available options that
+Goliath supports. As shown above, running tus-ruby-server on Goliath means you
+have to run it separately from your main app (unless your main app is also on
+Goliath).
+
 ## Storage
 
 ### Filesystem
@@ -207,8 +245,8 @@ Tus::Server.opts[:storage].expire_files(expiration_date)
 ## Download
 
 In addition to implementing the tus protocol, tus-ruby-server also comes with a
-GET endpoint for downloading the uploaded file, which streams the file directly
-from the storage.
+GET endpoint for downloading the uploaded file, which streams the file from the
+storage into the response body.
 
 The endpoint will automatically use the following `Upload-Metadata` values if
 they're available:
@@ -237,88 +275,13 @@ The following checksum algorithms are supported for the `checksum` extension:
 * MD5
 * CRC32
 
-## Performance considerations
-
-### Buffering
-
-When handling file uploads it's important not be be vulnerable to slow-write
-clients. That means you need to make sure that your web/application server
-buffers the request body locally before handing the request to the request
-worker.
-
-If the request body is not buffered and is read directly from the socket when
-it has already reached your Rack application, your application throughput will
-be severly impacted, because the workers will spend majority of their time
-waiting for request body to be read from the socket, and in that time they
-won't be able to serve new requests.
-
-Puma will automatically buffer the whole request body in a Tempfile, before
-fowarding the request to your Rack app. Unicorn and Passenger will not do that,
-so it's highly recommended to put a frontend server like Nginx in front of
-those web servers, and configure it to buffer the request body.
-
-### Chunking
-
-The tus protocol specifies
-
-> The Server SHOULD always attempt to store as much of the received data as possible.
-
-The tus-ruby-server Rack application supports saving partial data for if the
-PATCH request gets interrupted before all data has been sent, but I'm not aware
-of any Rack-compliant web server that will forward interrupted requests to the
-Rack app.
-
-This means that for resumable upload to be possible with tus-ruby-server in
-general, the file must be uploaded in multiple chunks; the client shouldn't
-rely that server will store any data if the PATCH request was interrupted.
-
-```js
-// using tus-js-client
-new tus.Upload(file, {
-  endpoint: "http://localhost:9292/files",
-  chunkSize: 5*1024*1024, // required option
-  // ...
-})
-```
-
-### Downloading
-
-Tus-ruby-server has a download endpoint which streams the uploaded file to the
-client. Unfortunately, with most classic web servers this endpoint will be
-vulnerable to slow-read clients, because the worker is only done once the whole
-response body has been received by the client. Web servers that are not
-vulnerable to slow-read clients include [Goliath]/[Thin] ([EventMachine]) and
-[Reel] ([Celluloid::IO]).
-
-So, depending on your requirements, you might want to avoid displaying the
-uploaded file in the browser (making the user download the file directly from
-the tus server), until it has been moved to a permanent storage. You might also
-want to consider copying finished uploads to permanent storage directly from
-the underlying tus storage, instead of downloading them through the app.
-
 ## Tests
 
 Run tests with
 
-```
-$ rake test
-```
-
-The S3 tests are excluded by default, but you can include them by setting the
-`$S3` environment variable.
-
-```
-$ S3=1 rake test
-```
-
-For running S3 tests you need to create an `.env` with the S3 credentials:
-
 ```sh
-# .env
-S3_BUCKET="..."
-S3_REGION="..."
-S3_ACCESS_KEY_ID="..."
-S3_SECRET_ACCESS_KEY="..."
+$ bundle exec rake test # unit tests
+$ bundle exec cucumber  # acceptance tests
 ```
 
 ## Inspiration
@@ -346,8 +309,6 @@ The tus-ruby-server was inspired by [rubytus].
 [`Aws::S3::Client#initialize`]: http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html#initialize-instance_method
 [`Aws::S3::Client#create_multipart_upload`]: http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html#create_multipart_upload-instance_method
 [Range requests]: https://tools.ietf.org/html/rfc7233
-[EventMachine]: https://github.com/eventmachine/eventmachine
-[Reel]: https://github.com/celluloid/reel
 [Goliath]: https://github.com/postrank-labs/goliath
-[Thin]: https://github.com/macournoyer/thin
-[Celluloid::IO]: https://github.com/celluloid/celluloid-io
+[EventMachine]: https://github.com/eventmachine/eventmachine
+[goliath server options]: https://github.com/postrank-labs/goliath/wiki/Server

data/lib/tus/checksum.rb

@@ -1,9 +1,7 @@
-require "base64"
-require "digest"
-require "zlib"
-
 module Tus
   class Checksum
+    CHUNK_SIZE = 16*1024
+
     attr_reader :algorithm
 
     def self.generate(algorithm, input)
@@ -47,14 +45,21 @@ module Tus
     end
 
     def generate_crc32(io)
-      crc = 0
-      crc = Zlib.crc32(io.read(16*1024, buffer ||= ""), crc) until io.eof?
-      Base64.encode64(crc.to_s)
+      require "zlib"
+      require "base64"
+      crc = Zlib.crc32("")
+      while (data = io.read(CHUNK_SIZE, buffer ||= ""))
+        crc = Zlib.crc32(data, crc)
+      end
+      Base64.strict_encode64(crc.to_s)
     end
 
     def digest(name, io)
+      require "digest"
       digest = Digest.const_get(name).new
-      digest.update(io.read(16*1024, buffer ||= "")) until io.eof?
+      while (data = io.read(CHUNK_SIZE, buffer ||= ""))
+        digest.update(data)
+      end
       digest.base64digest
     end
   end

data/lib/tus/errors.rb

@@ -1,4 +1,5 @@
 module Tus
-  Error = Class.new(StandardError)
-  NotFound = Class.new(Error)
+  Error           = Class.new(StandardError)
+  NotFound        = Class.new(Error)
+  MaxSizeExceeded = Class.new(Error)
 end

data/lib/tus/input.rb

@@ -1,31 +1,33 @@
+require "tus/errors"
+
 module Tus
   class Input
-    def initialize(input)
+    def initialize(input, limit: nil)
       @input = input
-      @bytes_read = 0
+      @limit = limit
+      @pos   = 0
     end
 
-    def read(*args)
-      result = @input.read(*args)
-      @bytes_read += result.bytesize if result.is_a?(String)
-      result
+    def read(length = nil, outbuf = nil)
+      data = @input.read(length, outbuf)
+
+      @pos += data.bytesize if data
+      raise MaxSizeExceeded if @limit && @pos > @limit
+
+      data
+    rescue => exception
+      raise unless exception.class.name == "Unicorn::ClientShutdown"
+      outbuf = outbuf.to_s.clear
+      outbuf unless length
     end
 
-    def eof?
-      @bytes_read == size
+    def pos
+      @pos
     end
 
     def rewind
       @input.rewind
-      @bytes_read = 0
-    end
-
-    def size
-      if defined?(Rack::Lint) && @input.is_a?(Rack::Lint::InputWrapper)
-        @input.instance_variable_get("@input").size
-      else
-        @input.size
-      end
+      @pos = 0
     end
 
     def close

data/lib/tus/server/goliath.rb

@@ -0,0 +1,68 @@
+require "tus/server"
+require "goliath"
+
+class Tus::Server::Goliath < Goliath::API
+  # Called as soon as request headers are parsed.
+  def on_headers(env, headers)
+    # the write end of the pipe is written in #on_body, and the read end is read by Tus::Server
+    env["tus.input-reader"], env["tus.input-writer"] = IO.pipe
+    # use a thread so that request is being processed in parallel
+    env["tus.request-thread"] = Thread.new do
+      call_tus_server env.merge("rack.input" => env["tus.input-reader"])
+    end
+  end
+
+  # Called on each request body chunk received from the client.
+  def on_body(env, data)
+    # append data to the write end of the pipe if open, otherwise do nothing
+    env["tus.input-writer"].write(data) unless env["tus.input-writer"].closed?
+  rescue Errno::EPIPE
+    # read end of the pipe has been closed, so we close the write end as well
+    env["tus.input-writer"].close
+  end
+
+  # Called at the end of the request (after #response is called), but also on
+  # client disconnect (in which case #response isn't called), so we want to do
+  # the same finalization in both methods.
+  def on_close(env)
+    finalize(env)
+  end
+
+  # Called after all the data has been received from the client.
+  def response(env)
+    status, headers, body = finalize(env)
+
+    env[STREAM_START].call(status, headers)
+
+    operation = proc { body.each { |chunk| env.stream_send(chunk) } }
+    callback  = proc { env.stream_close }
+
+    EM.defer(operation, callback) # use an outside thread pool for streaming
+
+    nil
+  end
+
+  private
+
+  # Calls the actual Roda application with the slightly modified env hash.
+  def call_tus_server(env)
+    Tus::Server.call env.merge(
+      "rack.url_scheme" => (env["options"][:ssl] ? "https" : "http"), # https://github.com/postrank-labs/goliath/issues/210
+      "async.callback"  => nil, # prevent Roda from calling EventMachine when streaming
+    )
+  end
+
+  # This method needs to be idempotent, because it can be called twice (on
+  # normal requests both #response and #on_close will be called, and on client
+  # disconnect only #on_close will be called).
+  def finalize(env)
+    # closing the write end of the pipe will mark EOF on the read end
+    env["tus.input-writer"].close unless env["tus.input-writer"].closed?
+    # wait for the request to finish
+    result = env["tus.request-thread"].value
+    # close read end of the pipe, since nothing is going to read from it anymore
+    env["tus.input-reader"].close unless env["tus.input-reader"].closed?
+    # return rack response
+    result
+  end
+end

data/lib/tus/server.rb

@@ -32,7 +32,6 @@ module Tus
     plugin :request_headers
     plugin :not_allowed
     plugin :streaming
-    plugin :error_handler
 
     route do |r|
       if request.headers["X-HTTP-Method-Override"]
@@ -74,6 +73,8 @@ module Tus
           )
 
           if info.concatenation?
+            validate_partial_uploads!(info.partial_uploads)
+
             length = storage.concatenate(uid, info.partial_uploads, info.to_h)
             info["Upload-Length"] = length.to_s
             info["Upload-Offset"] = length.to_s
@@ -102,7 +103,11 @@ module Tus
           no_content!
         end
 
-        info = Tus::Info.new(storage.read_info(uid))
+        begin
+          info = Tus::Info.new(storage.read_info(uid))
+        rescue Tus::NotFound
+          error!(404, "Upload Not Found")
+        end
 
         r.head do
           response.headers.update(info.headers)
@@ -112,23 +117,27 @@ module Tus
         end
 
         r.patch do
-          input = Tus::Input.new(request.body)
-
           if info.defer_length? && request.headers["Upload-Length"]
             validate_upload_length!
 
-            info["Upload-Length"] = request.headers["Upload-Length"]
+            info["Upload-Length"]       = request.headers["Upload-Length"]
             info["Upload-Defer-Length"] = nil
           end
 
+          input = get_input(info)
+
           validate_content_type!
-          validate_content_length!(info.offset, info.length)
-          validate_upload_offset!(info.offset)
+          validate_upload_offset!(info)
+          validate_content_length!(request.content_length.to_i, info) if request.content_length
           validate_upload_checksum!(input) if request.headers["Upload-Checksum"]
 
-          storage.patch_file(uid, input, info.to_h)
+          begin
+            bytes_uploaded = storage.patch_file(uid, input, info.to_h)
+          rescue Tus::MaxSizeExceeded
+            validate_content_length!(input.pos, info)
+          end
 
-          info["Upload-Offset"] = (info.offset + input.size).to_s
+          info["Upload-Offset"] = (info.offset + bytes_uploaded).to_s
           info["Upload-Expires"] = (Time.now + expiration_time).httpdate
 
           if info.offset == info.length # last chunk
@@ -142,13 +151,13 @@ module Tus
         end
 
         r.get do
-          validate_upload_finished!(info.length, info.offset)
+          validate_upload_finished!(info)
           range = handle_range_request!(info.length)
 
           response.headers["Content-Length"] = (range.end - range.begin + 1).to_s
 
           metadata = info.metadata
-          response.headers["Content-Disposition"] = opts[:disposition]
+          response.headers["Content-Disposition"]  = opts[:disposition]
           response.headers["Content-Disposition"] += "; filename=\"#{metadata["filename"]}\"" if metadata["filename"]
           response.headers["Content-Type"] = metadata["content_type"] || "application/octet-stream"
 
@@ -167,9 +176,12 @@ module Tus
       end
     end
 
-    error do |exception|
-      not_found! if exception.is_a?(Tus::NotFound)
-      raise
+    def get_input(info)
+      offset = info.offset
+      total  = info.length || max_size
+      limit  = total - offset if total
+
+      Tus::Input.new(request.body, limit: limit)
     end
 
     def validate_content_type!
@@ -197,29 +209,29 @@ module Tus
       end
     end
 
-    def validate_upload_offset!(current_offset)
+    def validate_upload_offset!(info)
       upload_offset = request.headers["Upload-Offset"]
 
       error!(400, "Missing Upload-Offset header") if upload_offset.to_s == ""
       error!(400, "Invalid Upload-Offset header") if upload_offset =~ /\D/
       error!(400, "Invalid Upload-Offset header") if upload_offset.to_i < 0
 
-      if upload_offset.to_i != current_offset
+      if upload_offset.to_i != info.offset
         error!(409, "Upload-Offset header doesn't match current offset")
       end
     end
 
-    def validate_content_length!(current_offset, length)
-      if length
-        error!(403, "Cannot modify completed upload") if current_offset == length
-        error!(413, "Size of this chunk surpasses Upload-Length") if Integer(request.content_length) + current_offset > length
+    def validate_content_length!(size, info)
+      if info.length
+        error!(403, "Cannot modify completed upload") if info.offset == info.length
+        error!(413, "Size of this chunk surpasses Upload-Length") if info.offset + size > info.length
       elsif max_size
-        error!(413, "Size of this chunk surpasses Tus-Max-Size") if Integer(request.content_length) + current_offset > max_size
+        error!(413, "Size of this chunk surpasses Tus-Max-Size") if info.offset + size > max_size
       end
     end
 
-    def validate_upload_finished!(length, current_offset)
-      error!(403, "Cannot download unfinished upload") unless length && current_offset && length == current_offset
+    def validate_upload_finished!(info)
+      error!(403, "Cannot download unfinished upload") unless info.length == info.offset
     end
 
     def validate_upload_metadata!
@@ -249,6 +261,30 @@ module Tus
       end
     end
 
+    def validate_partial_uploads!(part_uids)
+      queue = Queue.new
+      part_uids.each { |part_uid| queue << part_uid }
+
+      threads = 10.times.map do
+        Thread.new do
+          results = []
+          loop do
+            part_uid = queue.deq(true) rescue break
+            part_info = storage.read_info(part_uid)
+            results << part_info["Upload-Concat"]
+          end
+          results
+        end
+      end
+
+      upload_concat_values = threads.flat_map(&:value)
+      unless upload_concat_values.all? { |value| value == "partial" }
+        error!(400, "One or more uploads were not partial")
+      end
+    rescue Tus::NotFound
+      error!(404, "One or more partial uploads were not found")
+    end
+
     def validate_upload_checksum!(input)
       algorithm, checksum = request.headers["Upload-Checksum"].split(" ")
 
@@ -256,7 +292,7 @@ module Tus
       error!(400, "Invalid Upload-Checksum header") unless SUPPORTED_CHECKSUM_ALGORITHMS.include?(algorithm)
 
       generated_checksum = Tus::Checksum.generate(algorithm, input)
-      error!(460, "Checksum from Upload-Checksum header doesn't match generated") if generated_checksum != checksum
+      error!(460, "Upload-Checksum value doesn't match generated checksum") if generated_checksum != checksum
     end
 
     # "Range" header handling logic copied from Rack::File
@@ -312,10 +348,6 @@ module Tus
       request.halt
     end
 
-    def not_found!(message = "Upload not found")
-      error!(404, message)
-    end
-
     def error!(status, message)
       response.status = status
       response.write(message) unless request.head?

data/lib/tus/storage/filesystem.rb

@@ -49,35 +49,28 @@ module Tus
       end
 
       def patch_file(uid, input, info = {})
-        exists!(uid)
-
         file_path(uid).open("ab") { |file| IO.copy_stream(input, file) }
       end
 
       def read_info(uid)
-        exists!(uid)
+        raise Tus::NotFound if !file_path(uid).exist?
 
         JSON.parse(info_path(uid).binread)
       end
 
       def update_info(uid, info)
-        exists!(uid)
-
         info_path(uid).binwrite(JSON.generate(info))
       end
 
       def get_file(uid, info = {}, range: nil)
-        exists!(uid)
-
         file = file_path(uid).open("rb")
-        range ||= 0..(file.size - 1)
-        length = range.end - range.begin + 1
+        length = range ? range.size : file.size
 
         # Create an Enumerator which will yield chunks of the requested file
         # content, allowing tus server to efficiently stream requested content
         # to the client.
         chunks = Enumerator.new do |yielder|
-          file.seek(range.begin)
+          file.seek(range.begin) if range
           remaining_length = length
 
           while remaining_length > 0
@@ -89,11 +82,7 @@ module Tus
 
         # We return a response object that responds to #each, #length and #close,
         # which the tus server can return directly as the Rack response.
-        Response.new(
-          chunks: chunks,
-          length: length,
-          close:  ->{file.close},
-        )
+        Response.new(chunks: chunks, length: length, close: -> { file.close })
       end
 
       def delete_file(uid, info = {})
@@ -116,12 +105,8 @@ module Tus
         FileUtils.rm_f paths
       end
 
-      def exists!(uid)
-        raise Tus::NotFound if !file_path(uid).exist?
-      end
-
       def file_path(uid)
-        directory.join("#{uid}.file")
+        directory.join("#{uid}")
       end
 
       def info_path(uid)

data/lib/tus/storage/gridfs.rb

@@ -8,6 +8,8 @@ require "digest"
 module Tus
   module Storage
     class Gridfs
+      BATCH_SIZE = 5 * 1024 * 1024
+
       attr_reader :client, :prefix, :bucket, :chunk_size
 
       def initialize(client:, prefix: "fs", chunk_size: 256*1024)
@@ -47,7 +49,10 @@ module Tus
         grid_infos.inject(0) do |offset, grid_info|
           result = chunks_collection
             .find(files_id: grid_info[:_id])
-            .update_many("$set" => {files_id: grid_file.id}, "$inc" => {n: offset})
+            .update_many(
+              "$set" => { files_id: grid_file.id },
+              "$inc" => { n: offset },
+            )
 
           offset += result.modified_count
         end
@@ -60,47 +65,72 @@ module Tus
       end
 
       def patch_file(uid, input, info = {})
-        grid_info = find_grid_info!(uid)
+        grid_info      = files_collection.find(filename: uid).first
+        current_length = grid_info[:length]
+        chunk_size     = grid_info[:chunkSize]
+        bytes_saved    = 0
+
+        bytes_saved += patch_last_chunk(input, grid_info) if current_length % chunk_size != 0
+
+        chunks_enumerator = Enumerator.new do |yielder|
+          while (data = input.read(chunk_size))
+            yielder << data
+          end
+        end
 
-        patch_last_chunk(input, grid_info)
+        chunks_in_batch = (BATCH_SIZE.to_f / chunk_size).ceil
+        chunks_offset   = chunks_collection.count(files_id: grid_info[:_id]) - 1
 
-        grid_chunks = split_into_grid_chunks(input, grid_info)
-        chunks_collection.insert_many(grid_chunks)
-        grid_chunks.each { |grid_chunk| grid_chunk.data.data.clear } # deallocate strings
+        chunks_enumerator.each_slice(chunks_in_batch) do |chunks|
+          grid_chunks = chunks.map do |data|
+            Mongo::Grid::File::Chunk.new(
+              data: BSON::Binary.new(data),
+              files_id: grid_info[:_id],
+              n: chunks_offset += 1,
+            )
+          end
+
+          chunks_collection.insert_many(grid_chunks)
+
+          # Update the total length and refresh the upload date on each update,
+          # which are used in #get_file, #concatenate and #expire_files.
+          files_collection.find(filename: uid).update_one(
+            "$inc" => { length: chunks.map(&:bytesize).inject(0, :+) },
+            "$set" => { uploadDate: Time.now.utc },
+          )
+          bytes_saved += chunks.map(&:bytesize).inject(0, :+)
+
+          chunks.each(&:clear) # deallocate strings
+        end
 
-        # Update the total length and refresh the upload date on each update,
-        # which are used in #get_file, #concatenate and #expire_files.
-        files_collection.find(filename: uid).update_one("$set" => {
-          length:     grid_info[:length] + input.size,
-          uploadDate: Time.now.utc,
-        })
+        bytes_saved
       end
 
       def read_info(uid)
-        grid_info = find_grid_info!(uid)
+        grid_info = files_collection.find(filename: uid).first or raise Tus::NotFound
 
         grid_info[:metadata]
       end
 
       def update_info(uid, info)
-        grid_info = find_grid_info!(uid)
+        grid_info = files_collection.find(filename: uid).first
 
         files_collection.update_one({filename: uid}, {"$set" => {metadata: info}})
       end
 
       def get_file(uid, info = {}, range: nil)
-        grid_info = find_grid_info!(uid)
+        grid_info = files_collection.find(filename: uid).first
 
-        range ||= 0..(grid_info[:length] - 1)
-        length = range.end - range.begin + 1
+        length = range ? range.size : grid_info[:length]
 
-        chunk_start = range.begin / grid_info[:chunkSize]
-        chunk_stop  = range.end   / grid_info[:chunkSize]
+        filter = { files_id: grid_info[:_id] }
 
-        filter = {
-          files_id: grid_info[:_id],
-          n: {"$gte" => chunk_start, "$lte" => chunk_stop}
-        }
+        if range
+          chunk_start = range.begin / grid_info[:chunkSize]
+          chunk_stop  = range.end   / grid_info[:chunkSize]
+
+          filter[:n] = {"$gte" => chunk_start, "$lte" => chunk_stop}
+        end
 
         # Query only the subset of chunks specified by the range query. We
         # cannot use Mongo::FsBucket#open_download_stream here because it
@@ -137,11 +167,7 @@ module Tus
 
         # We return a response object that responds to #each, #length and #close,
         # which the tus server can return directly as the Rack response.
-        Response.new(
-          chunks: chunks,
-          length: length,
-          close:  ->{chunks_view.close_query},
-        )
+        Response.new(chunks: chunks, length: length, close: ->{chunks_view.close_query})
       end
 
       def delete_file(uid, info = {})
@@ -168,29 +194,19 @@ module Tus
         grid_file
       end
 
-      def split_into_grid_chunks(io, grid_info)
-        grid_info[:md5] = Digest::MD5.new # hack for `Chunk.split` updating MD5
-        grid_info = Mongo::Grid::File::Info.new(Mongo::Options::Mapper.transform(grid_info, Mongo::Grid::File::Info::MAPPINGS.invert))
-        offset = chunks_collection.count(files_id: grid_info.id)
-
-        Mongo::Grid::File::Chunk.split(io, grid_info, offset)
-      end
-
       def patch_last_chunk(input, grid_info)
-        if grid_info[:length] % grid_info[:chunkSize] != 0
-          last_chunk = chunks_collection.find(files_id: grid_info[:_id]).sort(n: -1).limit(1).first
-          data = last_chunk[:data].data
-          data << input.read(grid_info[:chunkSize] - data.length)
+        last_chunk = chunks_collection.find(files_id: grid_info[:_id]).sort(n: -1).limit(1).first
+        data = last_chunk[:data].data
+        patch = input.read(grid_info[:chunkSize] - data.bytesize)
+        data << patch
 
-          chunks_collection.find(files_id: grid_info[:_id], n: last_chunk[:n])
-            .update_one("$set" => {data: BSON::Binary.new(data)})
+        chunks_collection.find(files_id: grid_info[:_id], n: last_chunk[:n])
+          .update_one("$set" => { data: BSON::Binary.new(data) })
 
-          data.clear # deallocate string
-        end
-      end
+        files_collection.find(_id: grid_info[:_id])
+          .update_one("$inc" => { length: patch.bytesize })
 
-      def find_grid_info!(uid)
-        files_collection.find(filename: uid).first or raise Tus::NotFound
+        patch.bytesize
       end
 
       def validate_parts!(grid_infos, part_uids)

data/lib/tus/storage/s3.rb

@@ -1,18 +1,19 @@
 require "aws-sdk"
 
 require "tus/info"
-require "tus/checksum"
 require "tus/errors"
 
 require "json"
-require "cgi/util"
+require "cgi"
+require "fiber"
+require "stringio"
 
 Aws.eager_autoload!(services: ["S3"])
 
 module Tus
   module Storage
     class S3
-      MIN_PART_SIZE = 5 * 1024 * 1024
+      MIN_PART_SIZE = 5 * 1024 * 1024 # 5MB is the minimum part size for S3 multipart uploads
 
       attr_reader :client, :bucket, :prefix, :upload_options
 
@@ -20,7 +21,7 @@ module Tus
         resource = Aws::S3::Resource.new(**client_options)
 
         @client         = resource.client
-        @bucket         = resource.bucket(bucket)
+        @bucket         = resource.bucket(bucket) or fail(ArgumentError, "the :bucket option was nil")
         @prefix         = prefix
         @upload_options = upload_options
         @thread_count   = thread_count
@@ -33,8 +34,12 @@ module Tus
         options[:content_type] = tus_info.metadata["content_type"]
 
         if filename = tus_info.metadata["filename"]
+          # Aws-sdk doesn't sign non-ASCII characters correctly, and browsers
+          # will automatically URI-decode filenames.
+          filename = CGI.escape(filename).gsub("+", " ")
+
           options[:content_disposition] ||= "inline"
-          options[:content_disposition]  += "; filename=\"#{CGI.escape(filename).gsub("+", " ")}\""
+          options[:content_disposition]  += "; filename=\"#{filename}\""
         end
 
         multipart_upload = object(uid).initiate_multipart_upload(options)
@@ -51,9 +56,7 @@ module Tus
         objects = part_uids.map { |part_uid| object(part_uid) }
         parts   = copy_parts(objects, multipart_upload)
 
-        parts.each do |part|
-          info["multipart_parts"] << { "part_number" => part[:part_number], "etag" => part[:etag] }
-        end
+        info["multipart_parts"].concat parts
 
         finalize_file(uid, info)
 
@@ -68,21 +71,44 @@ module Tus
       end
 
       def patch_file(uid, input, info = {})
-        upload_id   = info["multipart_id"]
-        part_number = info["multipart_parts"].count + 1
+        tus_info = Tus::Info.new(info)
 
-        multipart_upload = object(uid).multipart_upload(upload_id)
-        multipart_part   = multipart_upload.part(part_number)
-        md5              = Tus::Checksum.new("md5").generate(input)
+        upload_id      = info["multipart_id"]
+        part_offset    = info["multipart_parts"].count
+        bytes_uploaded = 0
 
-        response = multipart_part.upload(body: input, content_md5: md5)
+        jobs = []
+        chunk = StringIO.new(input.read(MIN_PART_SIZE).to_s)
 
-        info["multipart_parts"] << {
-          "part_number" => part_number,
-          "etag"        => response.etag[/"(.+)"/, 1],
-        }
-      rescue Aws::S3::Errors::NoSuchUpload
-        raise Tus::NotFound
+        loop do
+          next_chunk = StringIO.new(input.read(MIN_PART_SIZE).to_s)
+
+          # merge next chunk into previous if it's smaller than minimum chunk size
+          if next_chunk.size < MIN_PART_SIZE
+            chunk = StringIO.new(chunk.string + next_chunk.string)
+            next_chunk.close
+            next_chunk = nil
+          end
+
+          # abort if chunk is smaller than 5MB and is not the last chunk
+          if chunk.size < MIN_PART_SIZE
+            break if (tus_info.length && tus_info.offset) &&
+                     chunk.size + tus_info.offset < tus_info.length
+          end
+
+          thread = upload_part_thread(chunk, uid, upload_id, part_offset += 1)
+          jobs << [thread, chunk]
+
+          chunk = next_chunk or break
+        end
+
+        jobs.each do |thread, body|
+          info["multipart_parts"] << thread.value
+          bytes_uploaded += body.size
+          body.close
+        end
+
+        bytes_uploaded
       end
 
       def finalize_file(uid, info = {})
@@ -92,7 +118,7 @@ module Tus
         end
 
         multipart_upload = object(uid).multipart_upload(upload_id)
-        multipart_upload.complete(multipart_upload: {parts: parts})
+        multipart_upload.complete(multipart_upload: { parts: parts })
 
         info.delete("multipart_id")
         info.delete("multipart_parts")
@@ -110,26 +136,15 @@ module Tus
       end
 
       def get_file(uid, info = {}, range: nil)
-        object = object(uid)
-        range  = "bytes=#{range.begin}-#{range.end}" if range
-
-        raw_chunks = object.enum_for(:get, range: range)
-
-        # Start the request to be notified if the object doesn't exist, and to
-        # get Aws::S3::Object#content_length.
-        first_chunk = raw_chunks.next
+        tus_info = Tus::Info.new(info)
 
-        chunks = Enumerator.new do |yielder|
-          yielder << first_chunk
-          loop { yielder << raw_chunks.next }
-        end
+        length = range ? range.size : tus_info.length
+        range  = "bytes=#{range.begin}-#{range.end}" if range
+        chunks = object(uid).enum_for(:get, range: range)
 
-        Response.new(
-          chunks: chunks,
-          length: object.content_length,
-        )
-      rescue Aws::S3::Errors::NoSuchKey
-        raise Tus::NotFound
+        # We return a response object that responds to #each, #length and #close,
+        # which the tus server can return directly as the Rack response.
+        Response.new(chunks: chunks, length: length)
       end
 
       def delete_file(uid, info = {})
@@ -151,7 +166,9 @@ module Tus
         delete(old_objects)
 
         bucket.multipart_uploads.each do |multipart_upload|
-          next unless multipart_upload.initiated <= expiration_date
+          # no need to check multipart uploads initiated before expiration date
+          next if multipart_upload.initiated > expiration_date
+
           most_recent_part = multipart_upload.parts.sort_by(&:last_modified).last
           if most_recent_part.nil? || most_recent_part.last_modified <= expiration_date
             abort_multipart_upload(multipart_upload)
@@ -161,10 +178,23 @@ module Tus
 
       private
 
+      def upload_part_thread(body, key, upload_id, part_number)
+        Thread.new { upload_part(body, key, upload_id, part_number) }
+      end
+
+      def upload_part(body, key, upload_id, part_number)
+        multipart_upload = object(key).multipart_upload(upload_id)
+        multipart_part   = multipart_upload.part(part_number)
+
+        response = multipart_part.upload(body: body)
+
+        { "part_number" => part_number, "etag" => response.etag }
+      end
+
       def delete(objects)
         # S3 can delete maximum of 1000 objects in a single request
         objects.each_slice(1000) do |objects_batch|
-          delete_params = {objects: objects_batch.map { |object| {key: object.key} }}
+          delete_params = { objects: objects_batch.map { |object| { key: object.key } } }
           bucket.delete_objects(delete: delete_params)
         end
       end
@@ -187,7 +217,7 @@ module Tus
 
         threads = @thread_count.times.map { copy_part_thread(queue) }
 
-        threads.flat_map(&:value).sort_by { |part| part[:part_number] }
+        threads.flat_map(&:value).sort_by { |part| part["part_number"] }
       end
 
       def compute_parts(objects, multipart_upload)
@@ -204,7 +234,6 @@ module Tus
 
       def copy_part_thread(queue)
         Thread.new do
-          Thread.current.abort_on_exception = true
           begin
             results = []
             loop do
@@ -212,9 +241,9 @@ module Tus
               results << copy_part(part)
             end
             results
-          rescue => error
+          rescue
             queue.clear
-            raise error
+            raise
           end
         end
       end
@@ -222,7 +251,7 @@ module Tus
       def copy_part(part)
         response = client.upload_part_copy(part)
 
-        { part_number: part[:part_number], etag: response.copy_part_result.etag }
+        { "part_number" => part[:part_number], "etag" => response.copy_part_result.etag }
       end
 
       def object(key)
@@ -239,12 +268,28 @@ module Tus
           @length
         end
 
-        def each(&block)
-          @chunks.each(&block)
+        def each
+          return enum_for(__method__) unless block_given?
+
+          while (chunk = chunks_fiber.resume)
+            yield chunk
+          end
         end
 
         def close
-          # aws-sdk doesn't provide an API to terminate the HTTP connection
+          chunks_fiber.resume(:close) if chunks_fiber.alive?
+        end
+
+        private
+
+        def chunks_fiber
+          @chunks_fiber ||= Fiber.new do
+            @chunks.each do |chunk|
+              action = Fiber.yield chunk
+              break if action == :close
+            end
+            nil
+          end
         end
       end
     end

data/tus-server.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |gem|
   gem.name         = "tus-server"
-  gem.version      = "0.10.2"
+  gem.version      = "1.0.0"
 
   gem.required_ruby_version = ">= 2.1"
 
@@ -19,7 +19,10 @@ Gem::Specification.new do |gem|
   gem.add_development_dependency "rake", "~> 11.1"
   gem.add_development_dependency "minitest", "~> 5.8"
   gem.add_development_dependency "rack-test_app"
+  gem.add_development_dependency "cucumber"
+  gem.add_development_dependency "unicorn"
   gem.add_development_dependency "mongo"
   gem.add_development_dependency "aws-sdk", "~> 2.0"
-  gem.add_development_dependency "dotenv"
+  gem.add_development_dependency "goliath"
+  gem.add_development_dependency "async-rack", ">= 0.5.1"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: tus-server
 version: !ruby/object:Gem::Version
-  version: 0.10.2
+  version: 1.0.0
 platform: ruby
 authors:
 - Janko Marohnić
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-04-19 00:00:00.000000000 Z
+date: 2017-07-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: roda
@@ -66,6 +66,34 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: cucumber
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: unicorn
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: mongo
   requirement: !ruby/object:Gem::Requirement
@@ -95,7 +123,7 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '2.0'
 - !ruby/object:Gem::Dependency
-  name: dotenv
+  name: goliath
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -108,6 +136,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: async-rack
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.5.1
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.5.1
 description: 
 email:
 - janko.marohnic@gmail.com
@@ -123,6 +165,7 @@ files:
 - lib/tus/info.rb
 - lib/tus/input.rb
 - lib/tus/server.rb
+- lib/tus/server/goliath.rb
 - lib/tus/storage/filesystem.rb
 - lib/tus/storage/gridfs.rb
 - lib/tus/storage/s3.rb
@@ -147,7 +190,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.5.1
+rubygems_version: 2.6.11
 signing_key: 
 specification_version: 4
 summary: Ruby server implementation of tus.io, the open protocol for resumable file