shrine-tus 0.1.2 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: a32c431983fb47c2d8bc8adfc51bedebbdfb5a62
-  data.tar.gz: ee506740601ae2417495245aae069ad1e396ba45
+  metadata.gz: 7a80efdb9caee16a6e25d9d1dde239dc53eccfc4
+  data.tar.gz: 0b793b3e15b7e2da93e6f6b068c3a1898b1ad653
 SHA512:
-  metadata.gz: 4c63dad51c8019a6cf2d603b5c3db7a57bb24b3a384d469034a05551c8a79942e52150159c17779c4cdb1080f950344f1d8ae86ebd9590e99569bb8916efb989
-  data.tar.gz: 4b0e9527a78c8902635046c769e7f8d90f20943b1c566083a605cf1a093e8f041e969490706996be858488f14e68ff0e1c13aa9bec400f616ea8321c42a2d44e
+  metadata.gz: 8ebc50f1e3b67661dfe21010135ed89b427ac0b7fb68123160596c20aedf782d8c297bcd3e8794472e3090b3cbd76ff03230c6436344f72a3e010ee3179a1981
+  data.tar.gz: 9aff64fc286488e699869ec36faac6dac07698a1e9b0bcd5d5128eea6106f98101bed0c2fd14a5f706dcad8b1e16608c7b86aefac4ee8614f7367a55a75dff08

data/README.md

@@ -10,8 +10,15 @@ gem "shrine-tus"
 
 ## Usage
 
-Once the file has been uploaded to `tus-ruby-server`, you want to attach it to
-a database record and move it to permanent Shrine storage.
+When the file is uploaded to the tus server, you'll probably want to attach
+it to a database record and move it to permanent storage. The storage that
+tus-ruby-server uses should be considered temporary, as some uploads might
+never be finished, and tus-ruby-server needs to clear expired uploads.
+
+Shrine-tus provides **three ways** of moving the file uploaded into the tus
+server to permanent storage, which differ in performance and level of
+decoupling. But regardless of the approach you choose, the code for attaching
+the uploaded file works the same:
 
 ```rb
 class VideoUploader < Shrine
@@ -28,9 +35,18 @@ file_data #=> {"id":"http://tus-server.org/68db42638388ae645ab747b36a837a79", "s
 Movie.create(video: file_data)
 ```
 
-The simplest setup is to have Shrine download the file uploaded to
-`tus-ruby-server` through the app, which you can do by setting
-`Shrine::Storage::Tus` as the temporary Shrine storage.
+See [shrine-tus-demo] for an example application that uses shrine-tus.
+
+### Approach A: Downloading through tus server
+
+Conceptionally the simplest setup is to have Shrine download the uploaded file
+through your tus server app. This is also the most decoupled option, because
+your main app can remain completely oblivious to what storage the tus server
+app uses internally, or in which programming language is it written (e.g. you
+could swap it for [tusd] and everything should continue to work the same).
+
+To use this approach, you need to assign `Shrine::Storage::Tus` as your
+temporary storage:
 
 ```rb
 require "shrine/storage/tus"
@@ -47,26 +63,71 @@ class VideoUploader < Shrine
 end
 ```
 
-By default `wget` will be used for downloading, but if you need support for
-partial downloads (e.g. you want to use `restore_cached_data`), you can switch
-to using [Down] for downloads.
+The `Shrine::Storage::Tus` class is a subclass of `Shrine::Storage::Url`, which
+uses [Down] for downloading. By default Down uses Ruby's built-in Net::HTTP as
+the backend for making download requests, however, for large files it might be
+more robust to use `wget` for downloading, as `wget` is able to automatically
+resume the download in case of temporary network failures.
 
 ```rb
-Shrine::Storage::Tus.new(downloader: :down)
+Shrine::Storage::Tus.new(downloader: :wget)
 ```
 
-If you want to Shrine, instead of downloading through the `tus-ruby-server`
-app, to download directly from the tus storage, you can assign the tus storage
-instance to `Shrine::Storage::Tus`:
+### Approach B: Downloading directly from storage
+
+While the appoach **A** is decoupled from the tus server implementation, it
+might not be the most performant depending on the overhead between your main
+app and the tus server app, how well the web server that you use for the tus
+server app handles streaming downloads (whether it blocks the web worker, thus
+affecting the app's request throughput), and whether you have hard request
+timeout limits like on Heroku. Down will also temporarily cache downloaded
+content to disk while copying to the permanent storage, so that adds I/O and
+disk usage.
+
+Instead of downloading through the tus server app, you can download directly
+from the underlying storage that it uses. This requires that you have the tus
+storage configured in your main app (which might already by the case if you're
+running tus server in the same process as your main app), and you can pass that
+storage to `Shrine::Storage::Tus`:
 
 ```rb
 Shrine::Storage::Tus.new(tus_storage: Tus::Server.opts[:storage])
 ```
 
-Finally, if you want to use the same kind of permanent storage as your
-`tus-ruby-server` uses, you can setup your temporary Shrine storage to match
-the one your tus server uses, and load the `tus` plugin which will translate
-the assigned tus URL to the corresponding storage ID.
+### Approach C: Tus storage equals Shrine storage
+
+Approach **B** internally utilizes the common interface of each tus storage
+object for streaming the file uploaded to that storage, via `#each`. However,
+certain Shrine storage classes have optimizations when copying/moving a file
+between two storages of the **same kind**.
+
+So if you want to use the same kind of permanent storage as your tus server
+uses, you can reap those performance benefits. In order to do this, instead of
+using `Shrine::Storage::Tus` as your temporary Shrine storage as we did in
+approaches A and B, we will be using a regular Shrine storage which will match
+the storage that the tus server uses. In other words, your Shrine storage and
+your tus storage would reference the same files. So, if your tus server app is
+configured with either of the following storages:
+
+```rb
+Tus::Server.opts[:storage] = Tus::Storage::Filesystem.new("data")
+Tus::Server.opts[:storage] = Tus::Storage::Gridfs.new(client: mongo, prefix: "tus")
+Tus::Server.opts[:storage] = Tus::Storage::S3.new(prefix: "tus", **s3_options)
+```
+
+Then Shrine should be configured with the corresponding temporary storage:
+
+```rb
+Shrine.storages[:cache] = Shrine::Storage::FileSystem.new("data")
+Shrine.storages[:cache] = Shrine::Storage::Gridfs.new(client: mongo, prefix: "tus")
+Shrine.storages[:cache] = Shrine::Storage::S3.new(prefix: "tus", **s3_options)
+```
+
+In approaches **A** and **B** we didn't need to change the file data received
+from the client, because we were using a subclass of `Shrine::Storage::Url`,
+which accepts the `id` field as a URL. But with this approach the `id` field
+will need to be translated from the tus URL to the correct ID for your
+temporary Shrine storage, using the `tus` plugin that ships with shrine-tus.
 
 ```rb
 Shrine.storages = {
@@ -80,16 +141,35 @@ class VideoUploader < Shrine
 end
 ```
 
-Note that this last approach doesn't use `Shrine::Storage::Tus`, so you don't
-need to add `shrine-url` to your Gemfile.
+These are the performance advantages for each of the official storages:
+
+#### Filesystem
+
+`Shrine::Storage::FileSystem` will have roughly the same performance as in
+option **B**, though it will allocate less memory. However, if you load the
+`moving` plugin, Shrine will execute a `mv` command between the tus storage
+and permanent storage, which executes instantaneously regardless of the
+filesize.
+
+```rb
+Shrine.plugin :moving
+```
+
+#### Mongo GridFS
+
+`Shrine::Storage::Gridfs` will use more efficient copying, resulting in up to
+2x speedup according to my benchmarks.
+
+#### AWS S3
 
-For more details, and an explanation of pros and cons for each of the
-approaches, see [shrine-tus-demo].
+`Shrine::Storage::S3` will issue a single S3 COPY request for files smaller
+than 100MB, while files 100MB or larger will be divided into multiple chunks
+which will be copied individually and in parallel using S3's multipart API.
 
 ## Contributing
 
 ```sh
-$ rake test
+$ bundle exec rake test
 ```
 
 ## License
@@ -100,3 +180,4 @@ $ rake test
 [tus-ruby-server]: https://github.com/janko-m/tus-ruby-server
 [Down]: https://github.com/janko-m/down
 [shrine-tus-demo]: https://github.com/janko-m/shrine-tus-demo
+[tusd]: https://github.com/tus/tusd

data/lib/shrine/plugins/tus.rb

@@ -21,7 +21,7 @@ class Shrine
           tus_uid = tus_url.split("/").last
 
           if defined?(Storage::FileSystem) && storage.is_a?(Storage::FileSystem)
-            "#{tus_uid}.file"
+            tus_uid
           elsif defined?(Storage::Gridfs) && storage.is_a?(Storage::Gridfs)
             grid_info = storage.bucket.find(filename: tus_uid).limit(1).first
             grid_info[:_id].to_s

data/lib/shrine/storage/tus.rb

@@ -7,7 +7,7 @@ class Shrine
     class Tus < Url
       attr_reader :tus_storage
 
-      def initialize(downloader: :wget, tus_storage: nil)
+      def initialize(downloader: :net_http, tus_storage: nil)
         @tus_storage = tus_storage
 
         super(downloader: downloader)
@@ -48,7 +48,7 @@ class Shrine
         Down::ChunkedIO.new(
           size:     response.length,
           chunks:   response.each,
-          on_close: ->{response.close},
+          on_close: response.method(:close),
         )
       end
 

data/shrine-tus.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |gem|
   gem.name          = "shrine-tus"
-  gem.version       = "0.1.2"
+  gem.version       = "1.0.0"
 
   gem.required_ruby_version = ">= 2.1"
 
@@ -14,8 +14,8 @@ Gem::Specification.new do |gem|
   gem.require_path = "lib"
 
   gem.add_dependency "shrine", "~> 2.0"
-  gem.add_dependency "tus-server", "~> 0.10"
-  gem.add_dependency "shrine-url", "~> 0.3"
+  gem.add_dependency "tus-server", "~> 1.0"
+  gem.add_dependency "shrine-url", "~> 1.0"
 
   gem.add_development_dependency "rake"
   gem.add_development_dependency "minitest"

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: shrine-tus
 version: !ruby/object:Gem::Version
-  version: 0.1.2
+  version: 1.0.0
 platform: ruby
 authors:
 - Janko Marohnić
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-06-16 00:00:00.000000000 Z
+date: 2017-07-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: shrine
@@ -30,28 +30,28 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.10'
+        version: '1.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.10'
+        version: '1.0'
 - !ruby/object:Gem::Dependency
   name: shrine-url
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.3'
+        version: '1.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.3'
+        version: '1.0'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement