tus-server 0.2.0 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +205 -52
- data/lib/tus/checksum.rb +30 -17
- data/lib/tus/errors.rb +4 -0
- data/lib/tus/info.rb +16 -3
- data/lib/tus/input.rb +31 -0
- data/lib/tus/server.rb +96 -77
- data/lib/tus/storage/filesystem.rb +82 -28
- data/lib/tus/storage/gridfs.rb +161 -35
- data/lib/tus/storage/s3.rb +242 -0
- data/tus-server.gemspec +4 -2
- metadata +35 -4
- data/lib/tus/expirator.rb +0 -58
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 06a1d11d97acd07210d2292e19e57be4c1c13071
|
4
|
+
data.tar.gz: 3b985f19d49493356931d50fdc1556b00e2c9688
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 8f25c106249b1bad3a8ee3787ee3d93dbff14b2b1453a55c1d74cddab8a195ad0014eed28c0ddf19f15caaa6fde610ddca0fce42ca4739cc1af1e60939d03091
|
7
|
+
data.tar.gz: 2675c30b3548d8a6a00976695152129c9b5dd58349e09c9eb2f8b153a10252b39b4aaf2d3d05777b28fc1e7831b56c9e84f533d55c2e434f017347f7dd1f6d3d
|
data/README.md
CHANGED
@@ -17,7 +17,10 @@ gem "tus-server"
|
|
17
17
|
|
18
18
|
## Usage
|
19
19
|
|
20
|
-
|
20
|
+
Tus-ruby-server provides a `Tus::Server` Roda app, which you can run in your
|
21
|
+
`config.ru`. That way you can run `Tus::Server` both as a standalone app or as
|
22
|
+
part of your main app (though it's recommended to run it as a standalone app,
|
23
|
+
as explained in the "Performance considerations" section of this README).
|
21
24
|
|
22
25
|
```rb
|
23
26
|
# config.ru
|
@@ -37,88 +40,190 @@ endpoint:
|
|
37
40
|
// using tus-js-client
|
38
41
|
new tus.Upload(file, {
|
39
42
|
endpoint: "http://localhost:9292/files",
|
40
|
-
chunkSize:
|
43
|
+
chunkSize: 5*1024*1024, // 5MB
|
41
44
|
// ...
|
42
45
|
})
|
43
46
|
```
|
44
47
|
|
45
|
-
After upload is complete, you'll probably want to attach the uploaded
|
46
|
-
database
|
47
|
-
see [shrine-tus-demo]
|
48
|
+
After the upload is complete, you'll probably want to attach the uploaded file
|
49
|
+
to a database record. [Shrine] is one file attachment library that integrates
|
50
|
+
nicely with tus-ruby-server, see [shrine-tus-demo] for an example integration.
|
48
51
|
|
49
|
-
|
52
|
+
## Storage
|
50
53
|
|
51
|
-
|
52
|
-
the `Upload-Metadata` header. When retrieving the file via a GET request,
|
53
|
-
tus-ruby-server will use
|
54
|
+
### Filesystem
|
54
55
|
|
55
|
-
|
56
|
-
|
57
|
-
|
58
|
-
Both of these are optional, and will be used if available.
|
59
|
-
|
60
|
-
### Storage
|
61
|
-
|
62
|
-
By default `Tus::Server` saves partial and complete files on the filesystem,
|
63
|
-
inside the `data/` directory. You can easily change the directory:
|
56
|
+
By default `Tus::Server` stores uploaded files to disk, in the `data/`
|
57
|
+
directory. You can configure a different directory:
|
64
58
|
|
65
59
|
```rb
|
66
|
-
require "tus/
|
60
|
+
require "tus/storage/filesystem"
|
67
61
|
|
68
62
|
Tus::Server.opts[:storage] = Tus::Storage::Filesystem.new("public/cache")
|
69
63
|
```
|
70
64
|
|
71
|
-
|
72
|
-
|
73
|
-
|
65
|
+
One downside of filesystem storage is that by default it doesn't work if you
|
66
|
+
want to run tus-ruby-servers on multiple servers, you'd have to set up a shared
|
67
|
+
filesystem between the servers. Another downside is that you have to make sure
|
68
|
+
your servers have enough disk space. Also, if you're using Heroku, you cannot
|
69
|
+
store files on the filesystem as they won't persist.
|
70
|
+
|
71
|
+
All these are reasons why you might store uploaded data on a different storage,
|
72
|
+
and luckily tus-ruby-server ships with two more storages.
|
74
73
|
|
75
|
-
|
76
|
-
|
77
|
-
|
74
|
+
### MongoDB GridFS
|
75
|
+
|
76
|
+
MongoDB has a specification for storing and retrieving large files, called
|
77
|
+
"[GridFS]". Tus-ruby-server ships with `Tus::Storage::Gridfs` that you can
|
78
|
+
use, which uses the [Mongo] gem.
|
78
79
|
|
79
80
|
```rb
|
80
|
-
gem "mongo"
|
81
|
+
gem "mongo", ">= 2.2.2", "< 3"
|
81
82
|
```
|
82
83
|
|
83
84
|
```rb
|
84
|
-
require "tus/server"
|
85
85
|
require "tus/storage/gridfs"
|
86
86
|
|
87
87
|
client = Mongo::Client.new("mongodb://127.0.0.1:27017/mydb")
|
88
88
|
Tus::Server.opts[:storage] = Tus::Storage::Gridfs.new(client: client)
|
89
89
|
```
|
90
90
|
|
91
|
-
|
92
|
-
|
91
|
+
The Gridfs specification requires that all chunks are of equal size, except the
|
92
|
+
last chunk. `Tus::Storage::Gridfs` will by default automatically make the
|
93
|
+
Gridfs chunk size equal to the size of the first uploaded chunk. This means
|
94
|
+
that all of the uploaded chunks need to be of equal size (except the last
|
95
|
+
chunk).
|
96
|
+
|
97
|
+
If you don't want the Gridfs chunk size to be equal to the size of the uploaded
|
98
|
+
chunks, you can hardcode the chunk size that will be used for all uploads.
|
99
|
+
|
100
|
+
```rb
|
101
|
+
Tus::Storage::Gridfs.new(client: client, chunk_size: 256*1024) # 256 KB
|
102
|
+
```
|
103
|
+
|
104
|
+
Just note that in this case the size of each uploaded chunk (except the last
|
105
|
+
one) needs to be a multiple of the `:chunk_size`.
|
106
|
+
|
107
|
+
### Amazon S3
|
108
|
+
|
109
|
+
Amazon S3 is probably one of the most popular services for storing files, and
|
110
|
+
tus-ruby-server ships with `Tus::Storage::S3` which utilizes S3's multipart API
|
111
|
+
to upload files, and depends on the [aws-sdk] gem.
|
112
|
+
|
113
|
+
```rb
|
114
|
+
gem "aws-sdk", "~> 2.1"
|
115
|
+
```
|
93
116
|
|
94
|
-
|
117
|
+
```rb
|
118
|
+
require "tus/storage/s3"
|
119
|
+
|
120
|
+
Tus::Server.opts[:storage] = Tus::Storage::S3.new(
|
121
|
+
access_key_id: "abc",
|
122
|
+
secret_access_key: "xyz",
|
123
|
+
region: "eu-west-1",
|
124
|
+
bucket: "my-app",
|
125
|
+
)
|
126
|
+
```
|
127
|
+
|
128
|
+
It might seem at first that using a remote storage like Amazon S3 will slow
|
129
|
+
down the overall upload, but the time it takes for the client to upload the
|
130
|
+
file to the Rack app is in general *much* longer than the time for the server
|
131
|
+
to upload that chunk to S3, because of the differences in the Internet
|
132
|
+
connection speed between the user's computer and server.
|
133
|
+
|
134
|
+
One thing to note is that S3's multipart API requires each chunk except the last
|
135
|
+
one to be 5MB or larger, so that is the minimum chunk size that you can specify
|
136
|
+
on your tus client if you want to use the S3 storage.
|
137
|
+
|
138
|
+
If you want to files to be stored in a certain subdirectory, you can specify
|
139
|
+
a `:prefix` in the storage configuration.
|
140
|
+
|
141
|
+
```rb
|
142
|
+
Tus::Storage::S3.new(prefix: "tus", **options)
|
143
|
+
```
|
144
|
+
|
145
|
+
You can also specify additional options that will be fowarded to
|
146
|
+
[`Aws::S3::Client#create_multipart_upload`] using `:upload_options`.
|
147
|
+
|
148
|
+
```rb
|
149
|
+
Tus::Storage::S3.new(upload_options: {content_disposition: "attachment"}, **options)
|
150
|
+
```
|
151
|
+
|
152
|
+
All other options will be forwarded to [`Aws::S3::Client#initialize`], so you
|
153
|
+
can for example change the `:endpoint` to use S3's accelerate host:
|
154
|
+
|
155
|
+
```rb
|
156
|
+
Tus::Storage::S3.new(endpoint: "https://s3-accelerate.amazonaws.com", **options)
|
157
|
+
```
|
158
|
+
|
159
|
+
### Other storages
|
160
|
+
|
161
|
+
If none of these storages suit you, you can write your own, you just need to
|
162
|
+
implement the same public interface:
|
163
|
+
|
164
|
+
```rb
|
165
|
+
def create_file(uid, info = {}) ... end
|
166
|
+
def concatenate(uid, part_uids, info = {}) ... end
|
167
|
+
def patch_file(uid, io, info = {}) ... end
|
168
|
+
def update_info(uid, info) ... end
|
169
|
+
def read_info(uid) ... end
|
170
|
+
def get_file(uid, info = {}, range: nil) ... end
|
171
|
+
def delete_file(uid, info = {}) ... end
|
172
|
+
def expire_files(expiration_date) ... end
|
173
|
+
```
|
174
|
+
|
175
|
+
## Maximum size
|
95
176
|
|
96
177
|
By default the maximum size for an uploaded file is 1GB, but you can change
|
97
178
|
that:
|
98
179
|
|
99
180
|
```rb
|
100
|
-
require "tus/server"
|
101
|
-
|
102
181
|
Tus::Server.opts[:max_size] = 5 * 1024*1024*1024 # 5GB
|
103
|
-
# or
|
104
182
|
Tus::Server.opts[:max_size] = nil # no limit
|
105
183
|
```
|
106
184
|
|
107
|
-
|
185
|
+
## Expiration
|
108
186
|
|
109
|
-
|
110
|
-
on each PATCH request. By default
|
111
|
-
|
112
|
-
but this can be changed:
|
187
|
+
Tus-ruby-server automatically adds expiration dates to each uploaded file, and
|
188
|
+
refreshes this date on each PATCH request. By default files expire 7 days after
|
189
|
+
they were last updated, but you can modify `:expiration_time`:
|
113
190
|
|
114
191
|
```rb
|
115
|
-
|
192
|
+
Tus::Server.opts[:expiration_time] = 2*24*60*60 # 2 days
|
193
|
+
```
|
194
|
+
|
195
|
+
Tus-ruby-server won't automatically delete expired files, but each storage
|
196
|
+
knows how to expire old files, so you just have to set up a recurring task
|
197
|
+
that will call `#expire_files`.
|
198
|
+
|
199
|
+
```rb
|
200
|
+
expiration_date = Time.now.utc - Tus::Server.opts[:expiration_time]
|
201
|
+
Tus::Server.opts[:storage].expire_files(expiration_date)
|
202
|
+
```
|
203
|
+
|
204
|
+
## Download
|
205
|
+
|
206
|
+
In addition to implementing the tus protocol, tus-ruby-server also comes with
|
207
|
+
an endpoint for downloading the uploaded file, streaming the file directly from
|
208
|
+
the storage.
|
116
209
|
|
117
|
-
|
118
|
-
|
210
|
+
The endpoint will automatically use the following `Upload-Metadata` values if
|
211
|
+
they're available:
|
212
|
+
|
213
|
+
* `content_type` -- used in the `Content-Type` response header
|
214
|
+
* `filename` -- used in the `Content-Disposition` response header
|
215
|
+
|
216
|
+
The `Content-Disposition` header will be set to "inline" by default, but you
|
217
|
+
can change it to "attachment" if you want the browser to always force download:
|
218
|
+
|
219
|
+
```rb
|
220
|
+
Tus::Server.opts[:disposition] = "attachment"
|
119
221
|
```
|
120
222
|
|
121
|
-
|
223
|
+
The download endpoint supports [Range requests], so you can use the tus
|
224
|
+
file URL as `src` in `<video>` and `<audio>` HTML tags.
|
225
|
+
|
226
|
+
## Checksum
|
122
227
|
|
123
228
|
The following checksum algorithms are supported for the `checksum` extension:
|
124
229
|
|
@@ -129,25 +234,64 @@ The following checksum algorithms are supported for the `checksum` extension:
|
|
129
234
|
* MD5
|
130
235
|
* CRC32
|
131
236
|
|
132
|
-
##
|
237
|
+
## Performance considerations
|
238
|
+
|
239
|
+
### Buffering
|
240
|
+
|
241
|
+
When handling file uploads it's important not be be vulnerable to slow-write
|
242
|
+
clients. That means you need to make sure that your web/application server
|
243
|
+
buffers the request body locally before handing the request to the request
|
244
|
+
worker.
|
245
|
+
|
246
|
+
If the request body is not buffered and is read directly from the socket when
|
247
|
+
it has already reached your Rack application, your application throughput will
|
248
|
+
be severly impacted, because the workers will spend majority of their time
|
249
|
+
waiting for request body to be read from the socket, and in that time they
|
250
|
+
won't be able to serve new requests.
|
133
251
|
|
134
|
-
|
135
|
-
|
136
|
-
recommended to
|
137
|
-
|
138
|
-
|
252
|
+
Puma will automatically buffer the whole request body in a Tempfile, before
|
253
|
+
fowarding the request to your Rack app. Unicorn and Passenger will not do that,
|
254
|
+
so it's highly recommended to put a frontend server like Nginx in front of
|
255
|
+
those web servers, and configure it to buffer the request body.
|
256
|
+
|
257
|
+
### Chunking
|
258
|
+
|
259
|
+
The tus protocol specifies
|
260
|
+
|
261
|
+
> The Server SHOULD always attempt to store as much of the received data as possible.
|
262
|
+
|
263
|
+
The tus-ruby-server Rack application supports saving partial data for if the
|
264
|
+
PATCH request gets interrupted before all data has been sent, but I'm not aware
|
265
|
+
of any Rack-compliant web server that will forward interrupted requests to the
|
266
|
+
Rack app.
|
267
|
+
|
268
|
+
This means that for resumable upload to be possible with tus-ruby-server in
|
269
|
+
general, the file must be uploaded in multiple chunks; the client shouldn't
|
270
|
+
rely that server will store any data if the PATCH request was interrupted.
|
139
271
|
|
140
272
|
```js
|
273
|
+
// using tus-js-client
|
141
274
|
new tus.Upload(file, {
|
142
275
|
endpoint: "http://localhost:9292/files",
|
143
|
-
chunkSize:
|
276
|
+
chunkSize: 5*1024*1024, // required option
|
144
277
|
// ...
|
145
278
|
})
|
146
279
|
```
|
147
280
|
|
148
|
-
|
149
|
-
|
150
|
-
|
281
|
+
### Downloading
|
282
|
+
|
283
|
+
Tus-ruby-server has a download endpoint which streams the uploaded file to the
|
284
|
+
client. Unfortunately, with most classic web servers this endpoint will be
|
285
|
+
vulnerable to slow-read clients, because the worker is only done once the whole
|
286
|
+
response body has been received by the client. Web servers that are not
|
287
|
+
vulnerable to slow-read clients include [Goliath]/[Thin] ([EventMachine]) and
|
288
|
+
[Reel] ([Celluloid::IO]).
|
289
|
+
|
290
|
+
So, depending on your requirements, you might want to avoid displaying the
|
291
|
+
uploaded file in the browser (making the user download the file directly from
|
292
|
+
the tus server), until it has been moved to a permanent storage. You might also
|
293
|
+
want to consider copying finished uploads to permanent storage directly from
|
294
|
+
the underlying tus storage, instead of downloading it through the app.
|
151
295
|
|
152
296
|
## Inspiration
|
153
297
|
|
@@ -170,3 +314,12 @@ The tus-ruby-server was inspired by [rubytus].
|
|
170
314
|
[Shrine]: https://github.com/janko-m/shrine
|
171
315
|
[trailing headers]: https://tools.ietf.org/html/rfc7230#section-4.1.2
|
172
316
|
[rubytus]: https://github.com/picocandy/rubytus
|
317
|
+
[aws-sdk]: https://github.com/aws/aws-sdk-ruby
|
318
|
+
[`Aws::S3::Client#initialize`]: http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html#initialize-instance_method
|
319
|
+
[`Aws::S3::Client#create_multipart_upload`]: http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html#create_multipart_upload-instance_method
|
320
|
+
[Range requests]: https://tools.ietf.org/html/rfc7233
|
321
|
+
[EventMachine]: https://github.com/eventmachine/eventmachine
|
322
|
+
[Reel]: https://github.com/celluloid/reel
|
323
|
+
[Goliath]: https://github.com/postrank-labs/goliath
|
324
|
+
[Thin]: https://github.com/macournoyer/thin
|
325
|
+
[Celluloid::IO]: https://github.com/celluloid/celluloid-io
|
data/lib/tus/checksum.rb
CHANGED
@@ -6,43 +6,56 @@ module Tus
|
|
6
6
|
class Checksum
|
7
7
|
attr_reader :algorithm
|
8
8
|
|
9
|
+
def self.generate(algorithm, input)
|
10
|
+
new(algorithm).generate(input)
|
11
|
+
end
|
12
|
+
|
9
13
|
def initialize(algorithm)
|
10
14
|
@algorithm = algorithm
|
11
15
|
end
|
12
16
|
|
13
|
-
def match?(checksum,
|
14
|
-
|
15
|
-
generate(content) == checksum
|
17
|
+
def match?(checksum, io)
|
18
|
+
generate(io) == checksum
|
16
19
|
end
|
17
20
|
|
18
|
-
def generate(
|
19
|
-
send("generate_#{algorithm}",
|
21
|
+
def generate(io)
|
22
|
+
hash = send("generate_#{algorithm}", io)
|
23
|
+
io.rewind
|
24
|
+
hash
|
20
25
|
end
|
21
26
|
|
22
27
|
private
|
23
28
|
|
24
|
-
def generate_sha1(
|
25
|
-
|
29
|
+
def generate_sha1(io)
|
30
|
+
digest(:SHA1, io)
|
31
|
+
end
|
32
|
+
|
33
|
+
def generate_sha256(io)
|
34
|
+
digest(:SHA256, io)
|
26
35
|
end
|
27
36
|
|
28
|
-
def
|
29
|
-
|
37
|
+
def generate_sha384(io)
|
38
|
+
digest(:SHA384, io)
|
30
39
|
end
|
31
40
|
|
32
|
-
def
|
33
|
-
|
41
|
+
def generate_sha512(io)
|
42
|
+
digest(:SHA512, io)
|
34
43
|
end
|
35
44
|
|
36
|
-
def
|
37
|
-
|
45
|
+
def generate_md5(io)
|
46
|
+
digest(:MD5, io)
|
38
47
|
end
|
39
48
|
|
40
|
-
def
|
41
|
-
|
49
|
+
def generate_crc32(io)
|
50
|
+
crc = nil
|
51
|
+
crc = Zlib.crc32(io.read(16*1024, buffer ||= "").to_s, crc) until io.eof?
|
52
|
+
Base64.encode64(crc.to_s)
|
42
53
|
end
|
43
54
|
|
44
|
-
def
|
45
|
-
|
55
|
+
def digest(name, io)
|
56
|
+
digest = Digest.const_get(name).new
|
57
|
+
digest.update(io.read(16*1024, buffer ||= "").to_s) until io.eof?
|
58
|
+
digest.base64digest
|
46
59
|
end
|
47
60
|
end
|
48
61
|
end
|
data/lib/tus/errors.rb
ADDED
data/lib/tus/info.rb
CHANGED
@@ -3,6 +3,15 @@ require "time"
|
|
3
3
|
|
4
4
|
module Tus
|
5
5
|
class Info
|
6
|
+
HEADERS = %w[
|
7
|
+
Upload-Length
|
8
|
+
Upload-Offset
|
9
|
+
Upload-Defer-Length
|
10
|
+
Upload-Metadata
|
11
|
+
Upload-Concat
|
12
|
+
Upload-Expires
|
13
|
+
]
|
14
|
+
|
6
15
|
def initialize(hash)
|
7
16
|
@hash = hash
|
8
17
|
end
|
@@ -16,11 +25,15 @@ module Tus
|
|
16
25
|
end
|
17
26
|
|
18
27
|
def to_h
|
19
|
-
@hash
|
28
|
+
@hash
|
29
|
+
end
|
30
|
+
|
31
|
+
def headers
|
32
|
+
@hash.select { |key, value| HEADERS.include?(key) && !value.nil? }
|
20
33
|
end
|
21
34
|
|
22
35
|
def length
|
23
|
-
Integer(@hash["Upload-Length"])
|
36
|
+
Integer(@hash["Upload-Length"]) if @hash["Upload-Length"]
|
24
37
|
end
|
25
38
|
|
26
39
|
def offset
|
@@ -35,7 +48,7 @@ module Tus
|
|
35
48
|
Time.parse(@hash["Upload-Expires"])
|
36
49
|
end
|
37
50
|
|
38
|
-
def
|
51
|
+
def concatenation?
|
39
52
|
@hash["Upload-Concat"].to_s.start_with?("final")
|
40
53
|
end
|
41
54
|
|