tus-server 0.2.0 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: d611f90e1f935c7caa84d040fe50cc3a9c49e6f6
-  data.tar.gz: a4095bfb0a87f0241336ec3ce78cb4f5b4442609
+  metadata.gz: 06a1d11d97acd07210d2292e19e57be4c1c13071
+  data.tar.gz: 3b985f19d49493356931d50fdc1556b00e2c9688
 SHA512:
-  metadata.gz: 12860dfff41aa4178d926618915e7cdbc6198599b139eb5a9dc2117ca4dbfed165fda92ff63620d70389562b222227831f3d7ee070f2b12562bb555424c75462
-  data.tar.gz: db9fcba3e6d4f40dffc90cab9d8bea3921727fb60ebba7cd8926b872ce305cedc9766eadcdeafee2f2e040f394b8555d21dc3b2bbd307a3d5d0e55a17f8ccf20
+  metadata.gz: 8f25c106249b1bad3a8ee3787ee3d93dbff14b2b1453a55c1d74cddab8a195ad0014eed28c0ddf19f15caaa6fde610ddca0fce42ca4739cc1af1e60939d03091
+  data.tar.gz: 2675c30b3548d8a6a00976695152129c9b5dd58349e09c9eb2f8b153a10252b39b4aaf2d3d05777b28fc1e7831b56c9e84f533d55c2e434f017347f7dd1f6d3d

data/README.md

@@ -17,7 +17,10 @@ gem "tus-server"
 
 ## Usage
 
-You can run the `Tus::Server` in your `config.ru`:
+Tus-ruby-server provides a `Tus::Server` Roda app, which you can run in your
+`config.ru`. That way you can run `Tus::Server` both as a standalone app or as
+part of your main app (though it's recommended to run it as a standalone app,
+as explained in the "Performance considerations" section of this README).
 
 ```rb
 # config.ru
@@ -37,88 +40,190 @@ endpoint:
 // using tus-js-client
 new tus.Upload(file, {
   endpoint: "http://localhost:9292/files",
-  chunkSize: 15*1024*1024, // 15 MB
+  chunkSize: 5*1024*1024, // 5MB
   // ...
 })
 ```
 
-After upload is complete, you'll probably want to attach the uploaded files to
-database records. [Shrine] is one file attachment library that supports this,
-see [shrine-tus-demo] on how you can integrate the two.
+After the upload is complete, you'll probably want to attach the uploaded file
+to a database record. [Shrine] is one file attachment library that integrates
+nicely with tus-ruby-server, see [shrine-tus-demo] for an example integration.
 
-### Metadata
+## Storage
 
-As per tus protocol, you can assign custom metadata when creating a file using
-the `Upload-Metadata` header. When retrieving the file via a GET request,
-tus-ruby-server will use
+### Filesystem
 
-* `content_type` -- for setting the `Content-Type` header
-* `filename` -- for setting the `Content-Disposition` header
-
-Both of these are optional, and will be used if available.
-
-### Storage
-
-By default `Tus::Server` saves partial and complete files on the filesystem,
-inside the `data/` directory. You can easily change the directory:
+By default `Tus::Server` stores uploaded files to disk, in the `data/`
+directory. You can configure a different directory:
 
 ```rb
-require "tus/server"
+require "tus/storage/filesystem"
 
 Tus::Server.opts[:storage] = Tus::Storage::Filesystem.new("public/cache")
 ```
 
-The downside of storing files on the filesystem is that it isn't distributed,
-so for resumable uploads to work you have to host the tus application on a
-single server.
+One downside of filesystem storage is that by default it doesn't work if you
+want to run tus-ruby-servers on multiple servers, you'd have to set up a shared
+filesystem between the servers. Another downside is that you have to make sure
+your servers have enough disk space. Also, if you're using Heroku, you cannot
+store files on the filesystem as they won't persist.
+
+All these are reasons why you might store uploaded data on a different storage,
+and luckily tus-ruby-server ships with two more storages.
 
-However, tus-ruby-server also ships with MongoDB [GridFS] storage, which among
-other things is convenient for a multi-server setup. It requires the [Mongo]
-gem:
+### MongoDB GridFS
+
+MongoDB has a specification for storing and retrieving large files, called
+"[GridFS]". Tus-ruby-server ships with `Tus::Storage::Gridfs` that you can
+use, which uses the [Mongo] gem.
 
 ```rb
-gem "mongo"
+gem "mongo", ">= 2.2.2", "< 3"
 ```
 
 ```rb
-require "tus/server"
 require "tus/storage/gridfs"
 
 client = Mongo::Client.new("mongodb://127.0.0.1:27017/mydb")
 Tus::Server.opts[:storage] = Tus::Storage::Gridfs.new(client: client)
 ```
 
-You can also write your own storage, you just need to implement the same
-public interface that `Tus::Storage::Filesystem` and `Tus::Storage::Gridfs` do.
+The Gridfs specification requires that all chunks are of equal size, except the
+last chunk. `Tus::Storage::Gridfs` will by default automatically make the
+Gridfs chunk size equal to the size of the first uploaded chunk. This means
+that all of the uploaded chunks need to be of equal size (except the last
+chunk).
+
+If you don't want the Gridfs chunk size to be equal to the size of the uploaded
+chunks, you can hardcode the chunk size that will be used for all uploads.
+
+```rb
+Tus::Storage::Gridfs.new(client: client, chunk_size: 256*1024) # 256 KB
+```
+
+Just note that in this case the size of each uploaded chunk (except the last
+one) needs to be a multiple of the `:chunk_size`.
+
+### Amazon S3
+
+Amazon S3 is probably one of the most popular services for storing files, and
+tus-ruby-server ships with `Tus::Storage::S3` which utilizes S3's multipart API
+to upload files, and depends on the [aws-sdk] gem.
+
+```rb
+gem "aws-sdk", "~> 2.1"
+```
 
-### Maximum size
+```rb
+require "tus/storage/s3"
+
+Tus::Server.opts[:storage] = Tus::Storage::S3.new(
+  access_key_id:     "abc",
+  secret_access_key: "xyz",
+  region:            "eu-west-1",
+  bucket:            "my-app",
+)
+```
+
+It might seem at first that using a remote storage like Amazon S3 will slow
+down the overall upload, but the time it takes for the client to upload the
+file to the Rack app is in general *much* longer than the time for the server
+to upload that chunk to S3, because of the differences in the Internet
+connection speed between the user's computer and server.
+
+One thing to note is that S3's multipart API requires each chunk except the last
+one to be 5MB or larger, so that is the minimum chunk size that you can specify
+on your tus client if you want to use the S3 storage.
+
+If you want to files to be stored in a certain subdirectory, you can specify
+a `:prefix` in the storage configuration.
+
+```rb
+Tus::Storage::S3.new(prefix: "tus", **options)
+```
+
+You can also specify additional options that will be fowarded to
+[`Aws::S3::Client#create_multipart_upload`] using `:upload_options`.
+
+```rb
+Tus::Storage::S3.new(upload_options: {content_disposition: "attachment"}, **options)
+```
+
+All other options will be forwarded to [`Aws::S3::Client#initialize`], so you
+can for example change the `:endpoint` to use S3's accelerate host:
+
+```rb
+Tus::Storage::S3.new(endpoint: "https://s3-accelerate.amazonaws.com", **options)
+```
+
+### Other storages
+
+If none of these storages suit you, you can write your own, you just need to
+implement the same public interface:
+
+```rb
+def create_file(uid, info = {})            ... end
+def concatenate(uid, part_uids, info = {}) ... end
+def patch_file(uid, io, info = {})         ... end
+def update_info(uid, info)                 ... end
+def read_info(uid)                         ... end
+def get_file(uid, info = {}, range: nil)   ... end
+def delete_file(uid, info = {})            ... end
+def expire_files(expiration_date)          ... end
+```
+
+## Maximum size
 
 By default the maximum size for an uploaded file is 1GB, but you can change
 that:
 
 ```rb
-require "tus/server"
-
 Tus::Server.opts[:max_size] = 5 * 1024*1024*1024 # 5GB
-# or
 Tus::Server.opts[:max_size] = nil                # no limit
 ```
 
-### Expiration
+## Expiration
 
-The expiration date is automatically set on each created file, and is refreshed
-on each PATCH request. By default the expiration date is 1 week from the last
-POST or PATCH request, and the interval of checking expired files is 1 hour,
-but this can be changed:
+Tus-ruby-server automatically adds expiration dates to each uploaded file, and
+refreshes this date on each PATCH request. By default files expire 7 days after
+they were last updated, but you can modify `:expiration_time`:
 
 ```rb
-require "tus/server"
+Tus::Server.opts[:expiration_time] = 2*24*60*60 # 2 days
+```
+
+Tus-ruby-server won't automatically delete expired files, but each storage
+knows how to expire old files, so you just have to set up a recurring task
+that will call `#expire_files`.
+
+```rb
+expiration_date = Time.now.utc - Tus::Server.opts[:expiration_time]
+Tus::Server.opts[:storage].expire_files(expiration_date)
+```
+
+## Download
+
+In addition to implementing the tus protocol, tus-ruby-server also comes with
+an endpoint for downloading the uploaded file, streaming the file directly from
+the storage.
 
-Tus::Server.opts[:expiration_time]     = 14*24*60*60 # 2 weeks
-Tus::Server.opts[:expiration_interval] = 24*60*60    # 1 day
+The endpoint will automatically use the following `Upload-Metadata` values if
+they're available:
+
+* `content_type` -- used in the `Content-Type` response header
+* `filename` -- used in the `Content-Disposition` response header
+
+The `Content-Disposition` header will be set to "inline" by default, but you
+can change it to "attachment" if you want the browser to always force download:
+
+```rb
+Tus::Server.opts[:disposition] = "attachment"
 ```
 
-### Checksum
+The download endpoint supports [Range requests], so you can use the tus
+file URL as `src` in `<video>` and `<audio>` HTML tags.
+
+## Checksum
 
 The following checksum algorithms are supported for the `checksum` extension:
 
@@ -129,25 +234,64 @@ The following checksum algorithms are supported for the `checksum` extension:
 * MD5
 * CRC32
 
-## Limitations
+## Performance considerations
+
+### Buffering
+
+When handling file uploads it's important not be be vulnerable to slow-write
+clients. That means you need to make sure that your web/application server
+buffers the request body locally before handing the request to the request
+worker.
+
+If the request body is not buffered and is read directly from the socket when
+it has already reached your Rack application, your application throughput will
+be severly impacted, because the workers will spend majority of their time
+waiting for request body to be read from the socket, and in that time they
+won't be able to serve new requests.
 
-Since tus-ruby-server is built using a Rack-based web framework (Roda), if a
-PATCH request gets interrupted, none of the received data will be stored. It's
-recommended to configure your client tus library not to use a single PATCH
-request for large files, but rather to split it into multiple chunks. You can
-do that for [tus-js-client] by specifying a maximum chunk size:
+Puma will automatically buffer the whole request body in a Tempfile, before
+fowarding the request to your Rack app. Unicorn and Passenger will not do that,
+so it's highly recommended to put a frontend server like Nginx in front of
+those web servers, and configure it to buffer the request body.
+
+### Chunking
+
+The tus protocol specifies
+
+> The Server SHOULD always attempt to store as much of the received data as possible.
+
+The tus-ruby-server Rack application supports saving partial data for if the
+PATCH request gets interrupted before all data has been sent, but I'm not aware
+of any Rack-compliant web server that will forward interrupted requests to the
+Rack app.
+
+This means that for resumable upload to be possible with tus-ruby-server in
+general, the file must be uploaded in multiple chunks; the client shouldn't
+rely that server will store any data if the PATCH request was interrupted.
 
 ```js
+// using tus-js-client
 new tus.Upload(file, {
   endpoint: "http://localhost:9292/files",
-  chunkSize: 15*1024*1024, // 15 MB
+  chunkSize: 5*1024*1024, // required option
   // ...
 })
 ```
 
-Tus-server also currently doesn't support the `checksum-trailer` extension,
-which would allow sending the checksum header *after* the data has been sent,
-using [trailing headers].
+### Downloading
+
+Tus-ruby-server has a download endpoint which streams the uploaded file to the
+client. Unfortunately, with most classic web servers this endpoint will be
+vulnerable to slow-read clients, because the worker is only done once the whole
+response body has been received by the client. Web servers that are not
+vulnerable to slow-read clients include [Goliath]/[Thin] ([EventMachine]) and
+[Reel] ([Celluloid::IO]).
+
+So, depending on your requirements, you might want to avoid displaying the
+uploaded file in the browser (making the user download the file directly from
+the tus server), until it has been moved to a permanent storage. You might also
+want to consider copying finished uploads to permanent storage directly from
+the underlying tus storage, instead of downloading it through the app.
 
 ## Inspiration
 
@@ -170,3 +314,12 @@ The tus-ruby-server was inspired by [rubytus].
 [Shrine]: https://github.com/janko-m/shrine
 [trailing headers]: https://tools.ietf.org/html/rfc7230#section-4.1.2
 [rubytus]: https://github.com/picocandy/rubytus
+[aws-sdk]: https://github.com/aws/aws-sdk-ruby
+[`Aws::S3::Client#initialize`]: http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html#initialize-instance_method
+[`Aws::S3::Client#create_multipart_upload`]: http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html#create_multipart_upload-instance_method
+[Range requests]: https://tools.ietf.org/html/rfc7233
+[EventMachine]: https://github.com/eventmachine/eventmachine
+[Reel]: https://github.com/celluloid/reel
+[Goliath]: https://github.com/postrank-labs/goliath
+[Thin]: https://github.com/macournoyer/thin
+[Celluloid::IO]: https://github.com/celluloid/celluloid-io

data/lib/tus/checksum.rb

@@ -6,43 +6,56 @@ module Tus
   class Checksum
     attr_reader :algorithm
 
+    def self.generate(algorithm, input)
+      new(algorithm).generate(input)
+    end
+
     def initialize(algorithm)
       @algorithm = algorithm
     end
 
-    def match?(checksum, content)
-      checksum = Base64.decode64(checksum)
-      generate(content) == checksum
+    def match?(checksum, io)
+      generate(io) == checksum
     end
 
-    def generate(content)
-      send("generate_#{algorithm}", content)
+    def generate(io)
+      hash = send("generate_#{algorithm}", io)
+      io.rewind
+      hash
     end
 
     private
 
-    def generate_sha1(content)
-      Digest::SHA1.hexdigest(content)
+    def generate_sha1(io)
+      digest(:SHA1, io)
+    end
+
+    def generate_sha256(io)
+      digest(:SHA256, io)
     end
 
-    def generate_sha256(content)
-      Digest::SHA256.hexdigest(content)
+    def generate_sha384(io)
+      digest(:SHA384, io)
     end
 
-    def generate_sha384(content)
-      Digest::SHA384.hexdigest(content)
+    def generate_sha512(io)
+      digest(:SHA512, io)
     end
 
-    def generate_sha512(content)
-      Digest::SHA512.hexdigest(content)
+    def generate_md5(io)
+      digest(:MD5, io)
     end
 
-    def generate_md5(content)
-      Digest::MD5.hexdigest(content)
+    def generate_crc32(io)
+      crc = nil
+      crc = Zlib.crc32(io.read(16*1024, buffer ||= "").to_s, crc) until io.eof?
+      Base64.encode64(crc.to_s)
     end
 
-    def generate_crc32(content)
-      Zlib.crc32(content).to_s
+    def digest(name, io)
+      digest = Digest.const_get(name).new
+      digest.update(io.read(16*1024, buffer ||= "").to_s) until io.eof?
+      digest.base64digest
     end
   end
 end

data/lib/tus/errors.rb

@@ -0,0 +1,4 @@
+module Tus
+  Error = Class.new(StandardError)
+  NotFound = Class.new(Error)
+end

data/lib/tus/info.rb

@@ -3,6 +3,15 @@ require "time"
 
 module Tus
   class Info
+    HEADERS = %w[
+      Upload-Length
+      Upload-Offset
+      Upload-Defer-Length
+      Upload-Metadata
+      Upload-Concat
+      Upload-Expires
+    ]
+
     def initialize(hash)
       @hash = hash
     end
@@ -16,11 +25,15 @@ module Tus
     end
 
     def to_h
-      @hash.reject { |key, value| value.nil? }
+      @hash
+    end
+
+    def headers
+      @hash.select { |key, value| HEADERS.include?(key) && !value.nil? }
     end
 
     def length
-      Integer(@hash["Upload-Length"])
+      Integer(@hash["Upload-Length"]) if @hash["Upload-Length"]
     end
 
     def offset
@@ -35,7 +48,7 @@ module Tus
       Time.parse(@hash["Upload-Expires"])
     end
 
-    def final_upload?
+    def concatenation?
       @hash["Upload-Concat"].to_s.start_with?("final")
     end