tus-server 2.1.2 → 2.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c0caef3d7cfbadc9b74678f99e687c0fe51bfbbab77acece73e9e1332aaa8661
-  data.tar.gz: 8be7b8e4ef8c2c6a5bc58ca11417c88b743b54252da7a3481f3866a591ef5abf
+  metadata.gz: f9ba14174a689878aa1224bdb2e526d869b888dc85164ec3e9320c8a171bcd73
+  data.tar.gz: 61e227c29273bd00007a96522a16b5f470fc433741999b072bf1c3642e13289b
 SHA512:
-  metadata.gz: 65e6d0496407297529e55063bbb27036d08697a986291dd08299e8994bbbeb6329d15dac4fa6a1137364e126379ae2fcd8c23df5500e0cb8c5e60074265070ae
-  data.tar.gz: b3328632108adeaf10761b3df9ab6ad9e957fe48a4b4ab92cccfaa36221f0c99be142af08ced374cbf1ab460e0b6622f0d964be55636683438b9c473e3533263
+  metadata.gz: a4914ab70cda026beee4de2b830778e43ef32131ddb65f470086d5cfd036f9dd046fafb09deced577d64965d539bf08a24ec44b810a77052822efde595610b04
+  data.tar.gz: 3c3f0d9196654541f5ebee297130c3d216dc1948a8a5c29695dc3251d969af85a101b0e249a889ec140e8f837c1fba1269a977649926d537c02dd5cd249d4e14

data/CHANGELOG.md

@@ -1,3 +1,19 @@
+## 2.2.0 (2018-12-02)
+
+* Add `before_create`, `after_create`, `after_finish`, and `after_terminate` hooks (@janko-m)
+
+* Rename `Tus::Info#concatenation?` to `Tus::Info#final?` (@janko-m)
+
+* Use `Storage#concurrency` for parallelized retrieval of partial uploads in `Upload-Concat` validation (@janko-m)
+
+* Replace `:thread_count` with `:concurrency` in S3 storage (@janko-m)
+
+* Validate that sum of partial uploads doesn't exceed `Tus-Max-Size` on concatenation (@janko-m)
+
+* Drop MRI 2.2 support (@janko-m)
+
+* Accept absolute URLs of partial uploads when creating a final upload (@janko-m)
+
 ## 2.1.2 (2018-10-21)
 
 * Make tus-ruby-server fully work with non-rewindable Rack input (@janko-m)

data/README.md

@@ -18,7 +18,7 @@ gem "tus-server", "~> 2.0"
 
 ## Usage
 
-The gem provides a `Tus::Server` Rack app, which you can mount inside your
+The gem provides a `Tus::Server` [Roda] app, which you can mount inside your
 main application. If you're using Rails, you can mount it in `config/routes.rb`:
 
 ```rb
@@ -46,7 +46,7 @@ endpoint:
 // using tus-js-client
 new tus.Upload(file, {
   endpoint: "/files",
-  chunkSize: 5*1024*1024, // required unless using Goliath
+  chunkSize: 5*1024*1024, // required unless using Falcon
   // ...
 })
 ```
@@ -54,79 +54,73 @@ new tus.Upload(file, {
 By default uploaded files will be stored in the `data/` directory. After the
 upload is complete, you'll probably want to attach the uploaded file to a
 database record. [Shrine] is currently the only file attachment library that
-integrates well with tus-ruby-server, see [this walkthrough][shrine resumable
-walkthrough] that adds resumable uploads from scratch, and for a complete
-example you can check out the [demo app][shrine-tus-demo].
+provides an integration with tus-ruby-server, see [this walkthrough][shrine
+resumable walkthrough] that adds resumable uploads from scratch, and for a
+complete example you can check out the [demo app][shrine-tus-demo].
 
-### Goliath
+### Streaming web server
 
-Running the tus server alongside your main app using classic web servers like
+Running the tus server alongside your main app using popular web servers like
 Puma or Unicorn is probably fine for most cases, however, it does come with a
 few gotchas. First, since these web servers don't accept partial requests
 (request where the request body hasn't been fully received), the tus client
 must be configured to split the upload into multiple requests. Second, since
 web workers are tied for the duration of the request, serving uploaded files
-through the tus server app could significantly impact request throughput, so
-you need to be careful to avoid that.
-
-There is an alternative. [Goliath] is an asychronous web server built on top of
-[EventMachine], which supports streaming requests and streaming responses.
-
-* Asynchronous streaming requests allows the tus server to begin saving
-  uploaded data while it's still being received. If the request is interrupted,
-  the tus server will attempt to save as much of the data that was received so
-  far. This means it's not necessary for the tus client to split the upload
-  into multiple smaller requests.
-
-* Asynchronous streaming responses allows the tus server to stream large files
-  with very small impact to the request throughput.
-
-Since Goliath is web server, to run tus server on it we'll have to run it as a
-standalone web app. It's recommended that you use [goliath-rack_proxy] for
-running your tus server app:
+through the tus server app could significantly impact request throughput; this
+can be avoided by having your frontend server (Nginx) serve the files if using
+`Filesystem` storage, or if you're using a cloud service like S3 having
+download requests redirect to the service file URL.
+
+That being said, there is a ruby web server that addresses these limitations
+– [Falcon]. Falcon is part of the [`async` ecosystem][async], and it utilizes
+non-blocking IO to process requests and responses in a streaming fashion
+without tying up your web workers. This has several benefits for
+`tus-ruby-server`:
+
+* since tus server is called to handle the request as soon as the request
+  headers are received, data from the request body will be uploaded to the
+  configured storage as it's coming in from the client
+
+* your web workers don't get tied up waiting for the client data, because as
+  soon as the server needs to wait for more data from the client, Falcon's
+  reactor switches to processing another request
+
+* if the upload request that's in progress gets interrupted, tus server will be
+  able save data that has been received so far, so it's not necessary for the
+  tus client to split the upload into multiple chunks
+
+* when uploaded files are being downloaded from the tus server, the request
+  throughput won't be impacted by the speed in which the client retrieves the
+  response body, because Falcon's reactor will switch to another request if the
+  client buffer gets full
+
+Falcon provides a Rack adapter and is compatible with Rails, so you can use it
+even if you're mounting `tus-ruby-server` inside your main app, just add
+`falcon` to the Gemfile and run `falcon serve`.
 
 ```rb
 # Gemfile
-gem "tus-server", "~> 2.0"
-gem "goliath-rack_proxy", "~> 1.0"
-```
-```rb
-# tus.rb
-require "tus/server"
-require "goliath/rack_proxy"
-
-# any additional Tus::Server configuration you want to put in here
-
-class GoliathTusServer < Goliath::RackProxy
-  rack_app Tus::Server
-  rewindable_input false # set to true if you're using checksums
-end
+gem "falcon"
 ```
 ```sh
-$ ruby tus.rb --stdout # enable logging
+$ falcon serve # reads from config.ru and starts the server
 ```
 
-This will run the tus server app on the root URL; if you want to run it on some
-path you can use `Rack::Builder`:
+Alternatively, you can run `tus-ruby-server` as a standalone app, by creating
+a separate "rackup" file just for it and pointing Falcon to that rackup file:
 
 ```rb
-class GoliathTusServer < Goliath::RackProxy
-  rack_app Rack::Builder.new {
-    map("/files") { run Tus::Server }
-  }
-  rewindable_input false # set to true if you're using checksums
-end
-```
+# tus.ru
+require "tus/server"
 
-In this case you'll have to configure the tus client to point to the standalone
-Goliath app:
+# ... configure tus-ruby-server ...
 
-```js
-// using tus-js-client
-new tus.Upload(file, {
-  endpoint: "http://localhost:9000/files",
-  // ...
-})
+map "/files" do
+  run Tus::Server
+end
+```
+```sh
+$ falcon serve --config tus.ru
 ```
 
 ## Storage
@@ -258,6 +252,13 @@ can for example change the `:endpoint` to use S3's accelerate host:
 Tus::Storage::S3.new(endpoint: "https://s3-accelerate.amazonaws.com", **options)
 ```
 
+If you're using [concatenation], you can specify the concurrency in which S3
+storage will copy partial uploads to the final upload (defaults to `10`):
+
+```rb
+Tus::Storage::S3.new(concurrency: { concatenation: 20 }, **options)
+```
+
 ### Google Cloud Storage, Microsoft Azure Blob Storage
 
 While tus-ruby-server doesn't currently ship with integrations for Google Cloud
@@ -360,6 +361,84 @@ def delete_file(uid, info = {})            ... end
 def expire_files(expiration_date)          ... end
 ```
 
+## Hooks
+
+You can register code to be executed on the following events:
+
+* `before_create` – before upload has been created
+* `after_create` – before upload has been created
+* `after_finish` – after the last chunk has been stored
+* `after_terminate` – after the upload has been deleted
+
+Each hook also receives two parameters: ID of the upload (`String`) and
+additional information about the upload (`Tus::Info`).
+
+```rb
+Tus::Server.after_finish do |uid, info|
+  uid  #=> "c0b67b04a9eccb4b1202000de628964f"
+  info #=> #<Tus::Info>
+
+  info.length        #=> 10      (Upload-Length)
+  info.offset        #=> 0       (Upload-Offset)
+  info.metadata      #=> {...}   (Upload-Metadata)
+  info.expires       #=> #<Time> (Upload-Expires)
+
+  info.partial?      #=> false   (Upload-Concat)
+  info.final?        #=> false   (Upload-Concat)
+
+  info.defer_length? #=> false   (Upload-Defer-Length)
+end
+```
+
+Because each hook is evaluated inside the `Tus::Server` instance (which is also
+a [Roda] instance), you can access request information and set response status
+and headers:
+
+```rb
+Tus::Server.after_terminate do |uid, info|
+  self     #=> #<Tus::Server> (and #<Roda>)
+  request  #=> #<Roda::Request>
+  response #=> #<Roda::Response>
+end
+```
+
+So, with hooks you could for example add authentication to `Tus::Server`:
+
+```rb
+Tus::Server.before_create do |uid, info|
+  authenticated = Authentication.call(request.headers["Authorization"])
+
+  unless authenticated
+    response.status = 403 # Forbidden
+    response.write("Not authenticated")
+    request.halt # return response now
+  end
+end
+```
+
+If you want to add hooks on more types of events, you can use Roda's
+[hooks][roda hooks] plugin to set `before` or `after` hooks for any request,
+which are also evaluated in context of the `Roda` instance:
+
+```rb
+Tus::Server.plugin :hooks
+Tus::Server.before do
+  # called before each Roda request
+end
+Tus::Server.after do
+  # called after each Roda request
+end
+```
+
+Provided that you're mounting `Tus::Server` inside your main app, any Rack
+middlewares that your main app uses will also be called for requests routed to
+the `Tus::Server`. If you want to add Rack middlewares to `Tus::Server`, you
+can do it by calling `Tus::Server.use`:
+
+```rb
+Tus::Server.use SomeRackMiddleware
+```
+
 ## Maximum size
 
 By default the size of files the tus server will accept is unlimited, but you
@@ -427,6 +506,19 @@ The following checksum algorithms are supported for the `checksum` extension:
 * MD5
 * CRC32
 
+## Concatenation
+
+When validating the `Upload-Concat` header for the final upload,
+tus-ruby-server needs to first fetch info about all partial uploads in order to
+check whether everything is in order. By default this retrieval is parallelized
+with 10 threads, but if you're using S3 storage you can change the
+`:concurrency`, and `Tus::Server` will automatically pick it up for its
+validation:
+
+```rb
+Tus::Storage::S3.new(concurrency: { concatenation: 20 }, **options)
+```
+
 ## Tests
 
 Run tests with
@@ -446,6 +538,8 @@ The tus-ruby-server was inspired by [rubytus] and [tusd].
 
 [MIT](/LICENSE.txt)
 
+[Roda]: https://github.com/jeremyevans/roda
+[roda hooks]: http://roda.jeremyevans.net/rdoc/classes/Roda/RodaPlugins/Hooks.html
 [tus resumable upload protocol]: http://tus.io/
 [tus-js-client]: https://github.com/tus/tus-js-client
 [creation]: http://tus.io/protocols/resumable-upload.html#creation
@@ -463,9 +557,6 @@ The tus-ruby-server was inspired by [rubytus] and [tusd].
 [`Aws::S3::Client#initialize`]: http://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#initialize-instance_method
 [`Aws::S3::Client#create_multipart_upload`]: http://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#create_multipart_upload-instance_method
 [Range requests]: https://tools.ietf.org/html/rfc7233
-[Goliath]: https://github.com/postrank-labs/goliath
-[EventMachine]: https://github.com/eventmachine/eventmachine
-[goliath-rack_proxy]: https://github.com/janko-m/goliath-rack_proxy
 [Rack::Sendfile]: https://www.rubydoc.info/github/rack/rack/master/Rack/Sendfile
 [`Aws::S3::Object#get`]: https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#get-instance_method
 [shrine resumable walkthrough]: https://github.com/shrinerb/shrine/wiki/Adding-Resumable-Uploads
@@ -473,3 +564,5 @@ The tus-ruby-server was inspired by [rubytus] and [tusd].
 [minio gcp]: https://minio.io/gcp.html
 [minio azure]: https://minio.io/azure.html
 [tusd]: https://github.com/tus/tusd
+[Falcon]: https://github.com/socketry/falcon
+[async]: https://github.com/socketry

data/lib/tus/info.rb

@@ -51,12 +51,12 @@ module Tus
       Time.parse(@hash["Upload-Expires"])
     end
 
-    def concatenation?
-      @hash["Upload-Concat"].to_s.start_with?("final")
+    def partial?
+      @hash["Upload-Concat"] == "partial"
     end
 
-    def defer_length?
-      @hash["Upload-Defer-Length"] == "1"
+    def final?
+      @hash["Upload-Concat"].to_s.start_with?("final")
     end
 
     def partial_uploads
@@ -64,6 +64,10 @@ module Tus
       urls.map { |url| url.split("/").last }
     end
 
+    def defer_length?
+      @hash["Upload-Defer-Length"] == "1"
+    end
+
     def remaining_length
       length - offset
     end

data/lib/tus/server.rb

@@ -1,4 +1,5 @@
 # frozen-string-literal: true
+
 require "roda"
 
 require "tus/storage/filesystem"
@@ -24,11 +25,13 @@ module Tus
     ]
     SUPPORTED_CHECKSUM_ALGORITHMS = %w[sha1 sha256 sha384 sha512 md5 crc32]
     RESUMABLE_CONTENT_TYPE = "application/offset+octet-stream"
+    HOOKS = %i[before_create after_create after_finish after_terminate]
 
     opts[:max_size]          = nil
     opts[:expiration_time]   = 7*24*60*60
     opts[:disposition]       = "inline"
     opts[:redirect_download] = nil
+    opts[:hooks]             = {}
 
     plugin :all_verbs
     plugin :default_headers, {"Content-Type" => ""}
@@ -36,6 +39,16 @@ module Tus
     plugin :request_headers
     plugin :not_allowed
 
+    HOOKS.each do |hook|
+      define_singleton_method(hook) do |&block|
+        opts[:hooks][hook] = block
+      end
+
+      define_method(hook) do |*args|
+        instance_exec(*args, &opts[:hooks][hook]) if opts[:hooks][hook]
+      end
+    end
+
     route do |r|
       if request.headers["X-HTTP-Method-Override"]
         request.env["REQUEST_METHOD"] = request.headers["X-HTTP-Method-Override"]
@@ -77,7 +90,9 @@ module Tus
             "Upload-Expires"      => (Time.now + expiration_time).httpdate,
           )
 
-          if info.concatenation?
+          before_create(uid, info)
+
+          if info.final?
             validate_partial_uploads!(info.partial_uploads)
 
             length = storage.concatenate(uid, info.partial_uploads, info.to_h)
@@ -87,8 +102,9 @@ module Tus
             storage.create_file(uid, info.to_h)
           end
 
-          storage.update_info(uid, info.to_h)
+          after_create(uid, info)
 
+          storage.update_info(uid, info.to_h)
           response.headers.update(info.headers)
 
           file_url = "#{request.url.chomp("/")}/#{uid}"
@@ -154,6 +170,8 @@ module Tus
 
           if info.offset == info.length # last chunk
             storage.finalize_file(uid, info.to_h) if storage.respond_to?(:finalize_file)
+
+            after_finish(uid, info)
           end
 
           storage.update_info(uid, info.to_h)
@@ -198,6 +216,8 @@ module Tus
         r.delete do
           storage.delete_file(uid, info.to_h)
 
+          after_terminate(uid, info)
+
           no_content!
         end
       end
@@ -285,34 +305,55 @@ module Tus
       if upload_concat.start_with?("final")
         string = upload_concat.split(";").last
         string.split(" ").each do |url|
-          error!(400, "Invalid Upload-Concat header") if url !~ %r{^#{request.script_name}/\w+$}
+          error!(400, "Invalid Upload-Concat header") if url !~ /#{request.script_name}\/\w+$/
         end
       end
     end
 
     # Validates that each partial upload exists and is marked as one.
     def validate_partial_uploads!(part_uids)
-      queue = Queue.new
-      part_uids.each { |part_uid| queue << part_uid }
+      input = Queue.new
+      part_uids.each { |part_uid| input << part_uid }
+      input.close
 
-      threads = 10.times.map do
+      results = Queue.new
+
+      thread_count   = storage.concurrency[:concatenation] if storage.respond_to?(:concurrency)
+      thread_count ||= 10
+
+      threads = thread_count.times.map do
         Thread.new do
-          results = []
-          loop do
-            part_uid = queue.deq(true) rescue break
-            part_info = storage.read_info(part_uid)
-            results << part_info["Upload-Concat"]
+          begin
+            loop do
+              part_uid = input.pop or break
+              part_info = storage.read_info(part_uid)
+              results << Tus::Info.new(part_info)
+            end
+            nil
+          rescue => error
+            input.clear
+            error
           end
-          results
         end
       end
 
-      upload_concat_values = threads.flat_map(&:value)
-      unless upload_concat_values.all? { |value| value == "partial" }
+      errors = threads.map(&:value).compact
+
+      if errors.any? { |error| error.is_a?(Tus::NotFound) }
+        error!(400, "One or more partial uploads were not found")
+      elsif errors.any?
+        fail errors.first
+      end
+
+      part_infos = Array.new(results.size) { results.pop } # convert Queue into an Array
+
+      unless part_infos.all?(&:partial?)
         error!(400, "One or more uploads were not partial")
       end
-    rescue Tus::NotFound
-      error!(400, "One or more partial uploads were not found")
+
+      if max_size && part_infos.map(&:length).inject(0, :+) > max_size
+        error!(400, "The sum of partial upload lengths exceed Tus-Max-Size")
+      end
     end
 
     def validate_upload_checksum!(input)

data/lib/tus/storage/s3.rb

@@ -16,17 +16,24 @@ module Tus
     class S3
       MIN_PART_SIZE = 5 * 1024 * 1024 # 5MB is the minimum part size for S3 multipart uploads
 
-      attr_reader :client, :bucket, :prefix, :upload_options
+      attr_reader :client, :bucket, :prefix, :upload_options, :concurrency
 
       # Initializes an aws-sdk-s3 client with the given credentials.
-      def initialize(bucket:, prefix: nil, upload_options: {}, thread_count: 10, **client_options)
+      def initialize(bucket:, prefix: nil, upload_options: {}, concurrency: {}, thread_count: nil, **client_options)
+        fail ArgumentError, "the :bucket option was nil" unless bucket
+
+        if thread_count
+          warn "[Tus-Ruby-Server] :thread_count is deprecated and will be removed in the next major version, use :concurrency instead, e.g `concurrency: { concatenation: 20 }`"
+          concurrency[:concatenation] = thread_count
+        end
+
         resource = Aws::S3::Resource.new(**client_options)
 
         @client         = resource.client
-        @bucket         = resource.bucket(bucket) or fail(ArgumentError, "the :bucket option was nil")
+        @bucket         = resource.bucket(bucket)
         @prefix         = prefix
         @upload_options = upload_options
-        @thread_count   = thread_count
+        @concurrency    = concurrency
       end
 
       # Initiates multipart upload for the given upload, and stores its
@@ -122,7 +129,8 @@ module Tus
           end
 
           begin
-            info["multipart_parts"] << upload_part(chunk, uid, upload_id, part_offset += 1)
+            part = upload_part(chunk, uid, upload_id, part_offset += 1)
+            info["multipart_parts"] << part
             bytes_uploaded += chunk.bytesize
           rescue Seahorse::Client::NetworkingError => exception
             warn "ERROR: #{exception.inspect} occurred during upload"
@@ -256,12 +264,21 @@ module Tus
       # given objects into them. It uses a queue and a fixed-size thread pool
       # which consumes that queue.
       def copy_parts(objects, multipart_upload)
-        parts = compute_parts(objects, multipart_upload)
-        queue = parts.inject(Queue.new) { |queue, part| queue << part }
+        parts   = compute_parts(objects, multipart_upload)
+        input   = Queue.new
+        results = Queue.new
+
+        parts.each { |part| input << part }
+        input.close
+
+        thread_count = concurrency[:concatenation] || 10
+        threads = thread_count.times.map { copy_part_thread(input, results) }
 
-        threads = @thread_count.times.map { copy_part_thread(queue) }
+        errors = threads.map(&:value).compact
+        fail errors.first if errors.any?
 
-        threads.flat_map(&:value).sort_by { |part| part["part_number"] }
+        part_results = Array.new(results.size) { results.pop } # convert Queue into an Array
+        part_results.sort_by { |part| part.fetch("part_number") }
       end
 
       # Computes data required for copying objects into new multipart parts.
@@ -279,18 +296,18 @@ module Tus
 
       # Consumes the queue for new multipart part information and issues the
       # copy requests.
-      def copy_part_thread(queue)
+      def copy_part_thread(input, results)
         Thread.new do
           begin
-            results = []
             loop do
-              part = queue.deq(true) rescue break
-              results << copy_part(part)
+              part = input.pop or break
+              part_result = copy_part(part)
+              results << part_result
             end
-            results
-          rescue
-            queue.clear
-            raise
+            nil
+          rescue => error
+            input.clear # clear other work
+            error
           end
         end
       end

data/tus-server.gemspec

@@ -1,8 +1,8 @@
 Gem::Specification.new do |gem|
   gem.name         = "tus-server"
-  gem.version      = "2.1.2"
+  gem.version      = "2.2.0"
 
-  gem.required_ruby_version = ">= 2.2"
+  gem.required_ruby_version = ">= 2.3"
 
   gem.summary      = "Ruby server implementation of tus.io, the open protocol for resumable file uploads."
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: tus-server
 version: !ruby/object:Gem::Version
-  version: 2.1.2
+  version: 2.2.0
 platform: ruby
 authors:
 - Janko Marohnić
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-10-21 00:00:00.000000000 Z
+date: 2018-12-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: roda
@@ -148,7 +148,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '2.2'
+      version: '2.3'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="