down 1.0.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/LICENSE.txt +21 -0
- data/README.md +112 -0
- data/down.gemspec +18 -0
- data/lib/down.rb +58 -0
- metadata +91 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 4d5f3b4d397d1d57d3caf236c97eb30802f76dc3
|
4
|
+
data.tar.gz: f8d11e7ec8adf34013c319a91c3a4ca17d0bbc7a
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: aa069b0e4b6626623e9d7beccb2aeaf52c947fc1eacdaedc961093590cf1fe920ef6ae76b1410e26e7109394310053d3ae62104c4c6766d7977f0d0b0e48a7ab
|
7
|
+
data.tar.gz: d077a4afbca667fbe78b9d5563bb0bc0e356cc671c51e4d592d363c0a2f4e24abe1456bd17d130675755504b410b03cbbf644fc63fa286b31fbc48837b4157ed
|
data/LICENSE.txt
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
The MIT License (MIT)
|
2
|
+
|
3
|
+
Copyright (c) 2015 Janko Marohnić
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
7
|
+
in the Software without restriction, including without limitation the rights
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
10
|
+
furnished to do so, subject to the following conditions:
|
11
|
+
|
12
|
+
The above copyright notice and this permission notice shall be included in
|
13
|
+
all copies or substantial portions of the Software.
|
14
|
+
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
21
|
+
THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,112 @@
|
|
1
|
+
# Down
|
2
|
+
|
3
|
+
Down is a wrapper around `open-uri` for safe downloading of remote files.
|
4
|
+
|
5
|
+
## Installation
|
6
|
+
|
7
|
+
```rb
|
8
|
+
gem 'down'
|
9
|
+
```
|
10
|
+
|
11
|
+
## Features
|
12
|
+
|
13
|
+
If you're downloading files from URLs that come from you, then it's probably
|
14
|
+
enough to just use `open-uri`. However, if you're accepting URLs from your
|
15
|
+
users (e.g. through `remote_<avatar>_url` in CarrierWave), then downloading is
|
16
|
+
suddenly not as simple as it appears to be.
|
17
|
+
|
18
|
+
### `StringIO`
|
19
|
+
|
20
|
+
Firstly, you may think that `open-uri` always downloads a file to disk, but
|
21
|
+
that's not true. If the downloaded file has 10 KB or less, `open-uri` actually
|
22
|
+
returns a `StringIO`. In my application I needed that the file is always
|
23
|
+
downloaded to disk. This is a wrong design decision, so Down patches this
|
24
|
+
behaviour and always returns a `Tempfile`.
|
25
|
+
|
26
|
+
### Metadata
|
27
|
+
|
28
|
+
`open-uri` adds some metadata to the returned file, like `#content_type`. Down
|
29
|
+
adds `#original_filename` as well, which is extracted from the URL.
|
30
|
+
|
31
|
+
```rb
|
32
|
+
require "down"
|
33
|
+
tempfile = Down.download("http://example.com/nature.jpg")
|
34
|
+
|
35
|
+
tempfile #=> #<Tempfile:/var/folders/k7/6zx6dx6x7ys3rv3srh0nyfj00000gn/T/20150925-55456-z7vxqz>
|
36
|
+
tempfile.content_type #=> "image/jpeg"
|
37
|
+
tempfile.original_filename #=> "nature.jpg"
|
38
|
+
```
|
39
|
+
|
40
|
+
### Maximum size
|
41
|
+
|
42
|
+
When you're accepting URLs from an outside source, it's a good idea to limit
|
43
|
+
the filesize (because attackers want to give a lot of work to your servers).
|
44
|
+
Down allows you to pass a `:max_size` option:
|
45
|
+
|
46
|
+
```rb
|
47
|
+
Down.download("http://example.com/image.jpg", max_size: 5 * 1024 * 1024) # 5 MB
|
48
|
+
# raises Down::TooLarge
|
49
|
+
```
|
50
|
+
|
51
|
+
What is the advantage over simply checking size after downloading? Well, Down
|
52
|
+
terminates the download very early, as soon as it gets the `Content-Length`
|
53
|
+
header. And if the `Content-Length` header is missing, Down will terminate the
|
54
|
+
download as soon as it receives a chunk which surpasses the maximum size.
|
55
|
+
|
56
|
+
### Progress
|
57
|
+
|
58
|
+
You can also tie into the progress of downloading, if you maybe want to display
|
59
|
+
a progress bar:
|
60
|
+
|
61
|
+
```rb
|
62
|
+
Down.download "http://example.com/image.jpg",
|
63
|
+
progress: ->(size) { ... } # called on each chunk
|
64
|
+
```
|
65
|
+
|
66
|
+
### Download errors
|
67
|
+
|
68
|
+
Firstly, Down encodes unencoded URLs, `open-uri` will for example trip if the
|
69
|
+
URL has a space. There are a lot of ways that a download can fail:
|
70
|
+
|
71
|
+
* URL is really invalid (`URI::InvalidURIError`)
|
72
|
+
* URL is a little bit invalid, e.g. "http:/example.com" (`Errno::ECONNREFUSED`)
|
73
|
+
* Domain wasn't not found (`SocketError`)
|
74
|
+
* Domain was found, but status is 4xx or 5xx (`OpenURI::HTTPError`)
|
75
|
+
* Request went into a redirect loop (`RuntimeError`)
|
76
|
+
* Request timeout out (`Timeout::Error`)
|
77
|
+
|
78
|
+
Down unifies all of these errors, and simply throws `Down::NotFound` error
|
79
|
+
(because this is what actually happened from the outside perspective).
|
80
|
+
|
81
|
+
### Timeout
|
82
|
+
|
83
|
+
You can specify the time after the request will time out:
|
84
|
+
|
85
|
+
```rb
|
86
|
+
Down.download "http://example.com/image.jpg", timeout: 5
|
87
|
+
```
|
88
|
+
|
89
|
+
## Supported Ruby versions
|
90
|
+
|
91
|
+
* MRI 1.9.3
|
92
|
+
* MRI 2.0
|
93
|
+
* MRI 2.1
|
94
|
+
* MRI 2.2
|
95
|
+
* JRuby
|
96
|
+
* Rubinius
|
97
|
+
|
98
|
+
## Development
|
99
|
+
|
100
|
+
```
|
101
|
+
$ rake test
|
102
|
+
```
|
103
|
+
|
104
|
+
If you want to test across Ruby versions and you're using rbenv, run
|
105
|
+
|
106
|
+
```
|
107
|
+
$ bin/test-versions
|
108
|
+
```
|
109
|
+
|
110
|
+
## License
|
111
|
+
|
112
|
+
[MIT](LICENSE.txt)
|
data/down.gemspec
ADDED
@@ -0,0 +1,18 @@
|
|
1
|
+
Gem::Specification.new do |spec|
|
2
|
+
spec.name = "down"
|
3
|
+
spec.version = "1.0.0"
|
4
|
+
spec.authors = ["Janko Marohnić"]
|
5
|
+
spec.email = ["janko.marohnic@gmail.com"]
|
6
|
+
|
7
|
+
spec.summary = "Robust file download from URL using open-uri."
|
8
|
+
spec.description = "Robust file download from URL using open-uri."
|
9
|
+
spec.homepage = "https://github.com/janko-m/down"
|
10
|
+
spec.license = "MIT"
|
11
|
+
|
12
|
+
spec.files = ["README.md", "LICENSE.txt", "down.gemspec", "lib/down.rb"]
|
13
|
+
spec.require_paths = ["lib"]
|
14
|
+
|
15
|
+
spec.add_development_dependency "rake"
|
16
|
+
spec.add_development_dependency "minitest", "~> 5.8"
|
17
|
+
spec.add_development_dependency "webmock"
|
18
|
+
end
|
data/lib/down.rb
ADDED
@@ -0,0 +1,58 @@
|
|
1
|
+
require "open-uri"
|
2
|
+
require "tempfile"
|
3
|
+
require "uri"
|
4
|
+
|
5
|
+
module Down
|
6
|
+
class Error < StandardError; end
|
7
|
+
class TooLarge < Error; end
|
8
|
+
class NotFound < Error; end
|
9
|
+
|
10
|
+
module_function
|
11
|
+
|
12
|
+
def download(url, options = {})
|
13
|
+
url = URI.encode(URI.decode(url))
|
14
|
+
|
15
|
+
downloaded_file = URI(url).open(
|
16
|
+
"User-Agent" => "Down/1.0.0",
|
17
|
+
content_length_proc: proc { |size|
|
18
|
+
raise Down::TooLarge if size && options[:max_size] && size > options[:max_size]
|
19
|
+
},
|
20
|
+
progress_proc: proc { |current_size|
|
21
|
+
raise Down::TooLarge if options[:max_size] && current_size > options[:max_size]
|
22
|
+
options[:progress].call(current_size) if options[:progress]
|
23
|
+
},
|
24
|
+
open_timeout: options[:timeout],
|
25
|
+
)
|
26
|
+
|
27
|
+
# open-uri will return a StringIO instead of a Tempfile if the filesize
|
28
|
+
# is less than 10 KB, so if it happens we convert it back to Tempfile.
|
29
|
+
if downloaded_file.is_a?(StringIO)
|
30
|
+
stringio = downloaded_file
|
31
|
+
downloaded_file = copy_to_tempfile("open-uri", stringio)
|
32
|
+
OpenURI::Meta.init downloaded_file, stringio
|
33
|
+
end
|
34
|
+
|
35
|
+
downloaded_file.extend DownloadedFile
|
36
|
+
downloaded_file
|
37
|
+
|
38
|
+
rescue => error
|
39
|
+
raise if error.instance_of?(RuntimeError) && error.message !~ /redirection/
|
40
|
+
raise if error.is_a?(Down::Error)
|
41
|
+
raise Down::NotFound, error.message
|
42
|
+
end
|
43
|
+
|
44
|
+
def copy_to_tempfile(basename, io)
|
45
|
+
tempfile = Tempfile.new(basename, binmode: true)
|
46
|
+
IO.copy_stream(io, tempfile.path)
|
47
|
+
io.rewind
|
48
|
+
tempfile
|
49
|
+
end
|
50
|
+
|
51
|
+
module DownloadedFile
|
52
|
+
def original_filename
|
53
|
+
path = base_uri.path
|
54
|
+
path = URI.decode(path)
|
55
|
+
File.basename(path) unless path.empty? || path == "/"
|
56
|
+
end
|
57
|
+
end
|
58
|
+
end
|
metadata
ADDED
@@ -0,0 +1,91 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: down
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 1.0.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Janko Marohnić
|
8
|
+
autorequire:
|
9
|
+
bindir: bin
|
10
|
+
cert_chain: []
|
11
|
+
date: 2015-09-25 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: rake
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - ">="
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: '0'
|
20
|
+
type: :development
|
21
|
+
prerelease: false
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - ">="
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: '0'
|
27
|
+
- !ruby/object:Gem::Dependency
|
28
|
+
name: minitest
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - "~>"
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: '5.8'
|
34
|
+
type: :development
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - "~>"
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: '5.8'
|
41
|
+
- !ruby/object:Gem::Dependency
|
42
|
+
name: webmock
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - ">="
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '0'
|
48
|
+
type: :development
|
49
|
+
prerelease: false
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - ">="
|
53
|
+
- !ruby/object:Gem::Version
|
54
|
+
version: '0'
|
55
|
+
description: Robust file download from URL using open-uri.
|
56
|
+
email:
|
57
|
+
- janko.marohnic@gmail.com
|
58
|
+
executables: []
|
59
|
+
extensions: []
|
60
|
+
extra_rdoc_files: []
|
61
|
+
files:
|
62
|
+
- LICENSE.txt
|
63
|
+
- README.md
|
64
|
+
- down.gemspec
|
65
|
+
- lib/down.rb
|
66
|
+
homepage: https://github.com/janko-m/down
|
67
|
+
licenses:
|
68
|
+
- MIT
|
69
|
+
metadata: {}
|
70
|
+
post_install_message:
|
71
|
+
rdoc_options: []
|
72
|
+
require_paths:
|
73
|
+
- lib
|
74
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
75
|
+
requirements:
|
76
|
+
- - ">="
|
77
|
+
- !ruby/object:Gem::Version
|
78
|
+
version: '0'
|
79
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
80
|
+
requirements:
|
81
|
+
- - ">="
|
82
|
+
- !ruby/object:Gem::Version
|
83
|
+
version: '0'
|
84
|
+
requirements: []
|
85
|
+
rubyforge_project:
|
86
|
+
rubygems_version: 2.4.5
|
87
|
+
signing_key:
|
88
|
+
specification_version: 4
|
89
|
+
summary: Robust file download from URL using open-uri.
|
90
|
+
test_files: []
|
91
|
+
has_rdoc:
|