shrine-tus 0.1.2 → 1.0.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: a32c431983fb47c2d8bc8adfc51bedebbdfb5a62
4
- data.tar.gz: ee506740601ae2417495245aae069ad1e396ba45
3
+ metadata.gz: 7a80efdb9caee16a6e25d9d1dde239dc53eccfc4
4
+ data.tar.gz: 0b793b3e15b7e2da93e6f6b068c3a1898b1ad653
5
5
  SHA512:
6
- metadata.gz: 4c63dad51c8019a6cf2d603b5c3db7a57bb24b3a384d469034a05551c8a79942e52150159c17779c4cdb1080f950344f1d8ae86ebd9590e99569bb8916efb989
7
- data.tar.gz: 4b0e9527a78c8902635046c769e7f8d90f20943b1c566083a605cf1a093e8f041e969490706996be858488f14e68ff0e1c13aa9bec400f616ea8321c42a2d44e
6
+ metadata.gz: 8ebc50f1e3b67661dfe21010135ed89b427ac0b7fb68123160596c20aedf782d8c297bcd3e8794472e3090b3cbd76ff03230c6436344f72a3e010ee3179a1981
7
+ data.tar.gz: 9aff64fc286488e699869ec36faac6dac07698a1e9b0bcd5d5128eea6106f98101bed0c2fd14a5f706dcad8b1e16608c7b86aefac4ee8614f7367a55a75dff08
data/README.md CHANGED
@@ -10,8 +10,15 @@ gem "shrine-tus"
10
10
 
11
11
  ## Usage
12
12
 
13
- Once the file has been uploaded to `tus-ruby-server`, you want to attach it to
14
- a database record and move it to permanent Shrine storage.
13
+ When the file is uploaded to the tus server, you'll probably want to attach
14
+ it to a database record and move it to permanent storage. The storage that
15
+ tus-ruby-server uses should be considered temporary, as some uploads might
16
+ never be finished, and tus-ruby-server needs to clear expired uploads.
17
+
18
+ Shrine-tus provides **three ways** of moving the file uploaded into the tus
19
+ server to permanent storage, which differ in performance and level of
20
+ decoupling. But regardless of the approach you choose, the code for attaching
21
+ the uploaded file works the same:
15
22
 
16
23
  ```rb
17
24
  class VideoUploader < Shrine
@@ -28,9 +35,18 @@ file_data #=> {"id":"http://tus-server.org/68db42638388ae645ab747b36a837a79", "s
28
35
  Movie.create(video: file_data)
29
36
  ```
30
37
 
31
- The simplest setup is to have Shrine download the file uploaded to
32
- `tus-ruby-server` through the app, which you can do by setting
33
- `Shrine::Storage::Tus` as the temporary Shrine storage.
38
+ See [shrine-tus-demo] for an example application that uses shrine-tus.
39
+
40
+ ### Approach A: Downloading through tus server
41
+
42
+ Conceptionally the simplest setup is to have Shrine download the uploaded file
43
+ through your tus server app. This is also the most decoupled option, because
44
+ your main app can remain completely oblivious to what storage the tus server
45
+ app uses internally, or in which programming language is it written (e.g. you
46
+ could swap it for [tusd] and everything should continue to work the same).
47
+
48
+ To use this approach, you need to assign `Shrine::Storage::Tus` as your
49
+ temporary storage:
34
50
 
35
51
  ```rb
36
52
  require "shrine/storage/tus"
@@ -47,26 +63,71 @@ class VideoUploader < Shrine
47
63
  end
48
64
  ```
49
65
 
50
- By default `wget` will be used for downloading, but if you need support for
51
- partial downloads (e.g. you want to use `restore_cached_data`), you can switch
52
- to using [Down] for downloads.
66
+ The `Shrine::Storage::Tus` class is a subclass of `Shrine::Storage::Url`, which
67
+ uses [Down] for downloading. By default Down uses Ruby's built-in Net::HTTP as
68
+ the backend for making download requests, however, for large files it might be
69
+ more robust to use `wget` for downloading, as `wget` is able to automatically
70
+ resume the download in case of temporary network failures.
53
71
 
54
72
  ```rb
55
- Shrine::Storage::Tus.new(downloader: :down)
73
+ Shrine::Storage::Tus.new(downloader: :wget)
56
74
  ```
57
75
 
58
- If you want to Shrine, instead of downloading through the `tus-ruby-server`
59
- app, to download directly from the tus storage, you can assign the tus storage
60
- instance to `Shrine::Storage::Tus`:
76
+ ### Approach B: Downloading directly from storage
77
+
78
+ While the appoach **A** is decoupled from the tus server implementation, it
79
+ might not be the most performant depending on the overhead between your main
80
+ app and the tus server app, how well the web server that you use for the tus
81
+ server app handles streaming downloads (whether it blocks the web worker, thus
82
+ affecting the app's request throughput), and whether you have hard request
83
+ timeout limits like on Heroku. Down will also temporarily cache downloaded
84
+ content to disk while copying to the permanent storage, so that adds I/O and
85
+ disk usage.
86
+
87
+ Instead of downloading through the tus server app, you can download directly
88
+ from the underlying storage that it uses. This requires that you have the tus
89
+ storage configured in your main app (which might already by the case if you're
90
+ running tus server in the same process as your main app), and you can pass that
91
+ storage to `Shrine::Storage::Tus`:
61
92
 
62
93
  ```rb
63
94
  Shrine::Storage::Tus.new(tus_storage: Tus::Server.opts[:storage])
64
95
  ```
65
96
 
66
- Finally, if you want to use the same kind of permanent storage as your
67
- `tus-ruby-server` uses, you can setup your temporary Shrine storage to match
68
- the one your tus server uses, and load the `tus` plugin which will translate
69
- the assigned tus URL to the corresponding storage ID.
97
+ ### Approach C: Tus storage equals Shrine storage
98
+
99
+ Approach **B** internally utilizes the common interface of each tus storage
100
+ object for streaming the file uploaded to that storage, via `#each`. However,
101
+ certain Shrine storage classes have optimizations when copying/moving a file
102
+ between two storages of the **same kind**.
103
+
104
+ So if you want to use the same kind of permanent storage as your tus server
105
+ uses, you can reap those performance benefits. In order to do this, instead of
106
+ using `Shrine::Storage::Tus` as your temporary Shrine storage as we did in
107
+ approaches A and B, we will be using a regular Shrine storage which will match
108
+ the storage that the tus server uses. In other words, your Shrine storage and
109
+ your tus storage would reference the same files. So, if your tus server app is
110
+ configured with either of the following storages:
111
+
112
+ ```rb
113
+ Tus::Server.opts[:storage] = Tus::Storage::Filesystem.new("data")
114
+ Tus::Server.opts[:storage] = Tus::Storage::Gridfs.new(client: mongo, prefix: "tus")
115
+ Tus::Server.opts[:storage] = Tus::Storage::S3.new(prefix: "tus", **s3_options)
116
+ ```
117
+
118
+ Then Shrine should be configured with the corresponding temporary storage:
119
+
120
+ ```rb
121
+ Shrine.storages[:cache] = Shrine::Storage::FileSystem.new("data")
122
+ Shrine.storages[:cache] = Shrine::Storage::Gridfs.new(client: mongo, prefix: "tus")
123
+ Shrine.storages[:cache] = Shrine::Storage::S3.new(prefix: "tus", **s3_options)
124
+ ```
125
+
126
+ In approaches **A** and **B** we didn't need to change the file data received
127
+ from the client, because we were using a subclass of `Shrine::Storage::Url`,
128
+ which accepts the `id` field as a URL. But with this approach the `id` field
129
+ will need to be translated from the tus URL to the correct ID for your
130
+ temporary Shrine storage, using the `tus` plugin that ships with shrine-tus.
70
131
 
71
132
  ```rb
72
133
  Shrine.storages = {
@@ -80,16 +141,35 @@ class VideoUploader < Shrine
80
141
  end
81
142
  ```
82
143
 
83
- Note that this last approach doesn't use `Shrine::Storage::Tus`, so you don't
84
- need to add `shrine-url` to your Gemfile.
144
+ These are the performance advantages for each of the official storages:
145
+
146
+ #### Filesystem
147
+
148
+ `Shrine::Storage::FileSystem` will have roughly the same performance as in
149
+ option **B**, though it will allocate less memory. However, if you load the
150
+ `moving` plugin, Shrine will execute a `mv` command between the tus storage
151
+ and permanent storage, which executes instantaneously regardless of the
152
+ filesize.
153
+
154
+ ```rb
155
+ Shrine.plugin :moving
156
+ ```
157
+
158
+ #### Mongo GridFS
159
+
160
+ `Shrine::Storage::Gridfs` will use more efficient copying, resulting in up to
161
+ 2x speedup according to my benchmarks.
162
+
163
+ #### AWS S3
85
164
 
86
- For more details, and an explanation of pros and cons for each of the
87
- approaches, see [shrine-tus-demo].
165
+ `Shrine::Storage::S3` will issue a single S3 COPY request for files smaller
166
+ than 100MB, while files 100MB or larger will be divided into multiple chunks
167
+ which will be copied individually and in parallel using S3's multipart API.
88
168
 
89
169
  ## Contributing
90
170
 
91
171
  ```sh
92
- $ rake test
172
+ $ bundle exec rake test
93
173
  ```
94
174
 
95
175
  ## License
@@ -100,3 +180,4 @@ $ rake test
100
180
  [tus-ruby-server]: https://github.com/janko-m/tus-ruby-server
101
181
  [Down]: https://github.com/janko-m/down
102
182
  [shrine-tus-demo]: https://github.com/janko-m/shrine-tus-demo
183
+ [tusd]: https://github.com/tus/tusd
@@ -21,7 +21,7 @@ class Shrine
21
21
  tus_uid = tus_url.split("/").last
22
22
 
23
23
  if defined?(Storage::FileSystem) && storage.is_a?(Storage::FileSystem)
24
- "#{tus_uid}.file"
24
+ tus_uid
25
25
  elsif defined?(Storage::Gridfs) && storage.is_a?(Storage::Gridfs)
26
26
  grid_info = storage.bucket.find(filename: tus_uid).limit(1).first
27
27
  grid_info[:_id].to_s
@@ -7,7 +7,7 @@ class Shrine
7
7
  class Tus < Url
8
8
  attr_reader :tus_storage
9
9
 
10
- def initialize(downloader: :wget, tus_storage: nil)
10
+ def initialize(downloader: :net_http, tus_storage: nil)
11
11
  @tus_storage = tus_storage
12
12
 
13
13
  super(downloader: downloader)
@@ -48,7 +48,7 @@ class Shrine
48
48
  Down::ChunkedIO.new(
49
49
  size: response.length,
50
50
  chunks: response.each,
51
- on_close: ->{response.close},
51
+ on_close: response.method(:close),
52
52
  )
53
53
  end
54
54
 
data/shrine-tus.gemspec CHANGED
@@ -1,6 +1,6 @@
1
1
  Gem::Specification.new do |gem|
2
2
  gem.name = "shrine-tus"
3
- gem.version = "0.1.2"
3
+ gem.version = "1.0.0"
4
4
 
5
5
  gem.required_ruby_version = ">= 2.1"
6
6
 
@@ -14,8 +14,8 @@ Gem::Specification.new do |gem|
14
14
  gem.require_path = "lib"
15
15
 
16
16
  gem.add_dependency "shrine", "~> 2.0"
17
- gem.add_dependency "tus-server", "~> 0.10"
18
- gem.add_dependency "shrine-url", "~> 0.3"
17
+ gem.add_dependency "tus-server", "~> 1.0"
18
+ gem.add_dependency "shrine-url", "~> 1.0"
19
19
 
20
20
  gem.add_development_dependency "rake"
21
21
  gem.add_development_dependency "minitest"
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: shrine-tus
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.2
4
+ version: 1.0.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Janko Marohnić
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2017-06-16 00:00:00.000000000 Z
11
+ date: 2017-07-17 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: shrine
@@ -30,28 +30,28 @@ dependencies:
30
30
  requirements:
31
31
  - - "~>"
32
32
  - !ruby/object:Gem::Version
33
- version: '0.10'
33
+ version: '1.0'
34
34
  type: :runtime
35
35
  prerelease: false
36
36
  version_requirements: !ruby/object:Gem::Requirement
37
37
  requirements:
38
38
  - - "~>"
39
39
  - !ruby/object:Gem::Version
40
- version: '0.10'
40
+ version: '1.0'
41
41
  - !ruby/object:Gem::Dependency
42
42
  name: shrine-url
43
43
  requirement: !ruby/object:Gem::Requirement
44
44
  requirements:
45
45
  - - "~>"
46
46
  - !ruby/object:Gem::Version
47
- version: '0.3'
47
+ version: '1.0'
48
48
  type: :runtime
49
49
  prerelease: false
50
50
  version_requirements: !ruby/object:Gem::Requirement
51
51
  requirements:
52
52
  - - "~>"
53
53
  - !ruby/object:Gem::Version
54
- version: '0.3'
54
+ version: '1.0'
55
55
  - !ruby/object:Gem::Dependency
56
56
  name: rake
57
57
  requirement: !ruby/object:Gem::Requirement