down 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 4d5f3b4d397d1d57d3caf236c97eb30802f76dc3
+  data.tar.gz: f8d11e7ec8adf34013c319a91c3a4ca17d0bbc7a
+SHA512:
+  metadata.gz: aa069b0e4b6626623e9d7beccb2aeaf52c947fc1eacdaedc961093590cf1fe920ef6ae76b1410e26e7109394310053d3ae62104c4c6766d7977f0d0b0e48a7ab
+  data.tar.gz: d077a4afbca667fbe78b9d5563bb0bc0e356cc671c51e4d592d363c0a2f4e24abe1456bd17d130675755504b410b03cbbf644fc63fa286b31fbc48837b4157ed

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2015 Janko Marohnić
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,112 @@
+# Down
+
+Down is a wrapper around `open-uri` for safe downloading of remote files.
+
+## Installation
+
+```rb
+gem 'down'
+```
+
+## Features
+
+If you're downloading files from URLs that come from you, then it's probably
+enough to just use `open-uri`. However, if you're accepting URLs from your
+users (e.g. through `remote_<avatar>_url` in CarrierWave), then downloading is
+suddenly not as simple as it appears to be.
+
+### `StringIO`
+
+Firstly, you may think that `open-uri` always downloads a file to disk, but
+that's not true. If the downloaded file has 10 KB or less, `open-uri` actually
+returns a `StringIO`. In my application I needed that the file is always
+downloaded to disk. This is a wrong design decision, so Down patches this
+behaviour and always returns a `Tempfile`.
+
+### Metadata
+
+`open-uri` adds some metadata to the returned file, like `#content_type`. Down
+adds `#original_filename` as well, which is extracted from the URL.
+
+```rb
+require "down"
+tempfile = Down.download("http://example.com/nature.jpg")
+
+tempfile #=> #<Tempfile:/var/folders/k7/6zx6dx6x7ys3rv3srh0nyfj00000gn/T/20150925-55456-z7vxqz>
+tempfile.content_type #=> "image/jpeg"
+tempfile.original_filename #=> "nature.jpg"
+```
+
+### Maximum size
+
+When you're accepting URLs from an outside source, it's a good idea to limit
+the filesize (because attackers want to give a lot of work to your servers).
+Down allows you to pass a `:max_size` option:
+
+```rb
+Down.download("http://example.com/image.jpg", max_size: 5 * 1024 * 1024) # 5 MB
+# raises Down::TooLarge
+```
+
+What is the advantage over simply checking size after downloading? Well, Down
+terminates the download very early, as soon as it gets the `Content-Length`
+header. And if the `Content-Length` header is missing, Down will terminate the
+download as soon as it receives a chunk which surpasses the maximum size.
+
+### Progress
+
+You can also tie into the progress of downloading, if you maybe want to display
+a progress bar:
+
+```rb
+Down.download "http://example.com/image.jpg",
+  progress: ->(size) { ... } # called on each chunk
+```
+
+### Download errors
+
+Firstly, Down encodes unencoded URLs, `open-uri` will for example trip if the
+URL has a space. There are a lot of ways that a download can fail:
+
+* URL is really invalid (`URI::InvalidURIError`)
+* URL is a little bit invalid, e.g. "http:/example.com" (`Errno::ECONNREFUSED`)
+* Domain wasn't not found (`SocketError`)
+* Domain was found, but status is 4xx or 5xx (`OpenURI::HTTPError`)
+* Request went into a redirect loop (`RuntimeError`)
+* Request timeout out (`Timeout::Error`)
+
+Down unifies all of these errors, and simply throws `Down::NotFound` error
+(because this is what actually happened from the outside perspective).
+
+### Timeout
+
+You can specify the time after the request will time out:
+
+```rb
+Down.download "http://example.com/image.jpg", timeout: 5
+```
+
+## Supported Ruby versions
+
+* MRI 1.9.3
+* MRI 2.0
+* MRI 2.1
+* MRI 2.2
+* JRuby
+* Rubinius
+
+## Development
+
+```
+$ rake test
+```
+
+If you want to test across Ruby versions and you're using rbenv, run
+
+```
+$ bin/test-versions
+```
+
+## License
+
+[MIT](LICENSE.txt)

data/down.gemspec

@@ -0,0 +1,18 @@
+Gem::Specification.new do |spec|
+  spec.name          = "down"
+  spec.version       = "1.0.0"
+  spec.authors       = ["Janko Marohnić"]
+  spec.email         = ["janko.marohnic@gmail.com"]
+
+  spec.summary       = "Robust file download from URL using open-uri."
+  spec.description   = "Robust file download from URL using open-uri."
+  spec.homepage      = "https://github.com/janko-m/down"
+  spec.license       = "MIT"
+
+  spec.files         = ["README.md", "LICENSE.txt", "down.gemspec", "lib/down.rb"]
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "minitest", "~> 5.8"
+  spec.add_development_dependency "webmock"
+end

data/lib/down.rb

@@ -0,0 +1,58 @@
+require "open-uri"
+require "tempfile"
+require "uri"
+
+module Down
+  class Error < StandardError; end
+  class TooLarge < Error; end
+  class NotFound < Error; end
+
+  module_function
+
+  def download(url, options = {})
+    url = URI.encode(URI.decode(url))
+
+    downloaded_file = URI(url).open(
+      "User-Agent" => "Down/1.0.0",
+      content_length_proc: proc { |size|
+        raise Down::TooLarge if size && options[:max_size] && size > options[:max_size]
+      },
+      progress_proc: proc { |current_size|
+        raise Down::TooLarge if options[:max_size] && current_size > options[:max_size]
+        options[:progress].call(current_size) if options[:progress]
+      },
+      open_timeout: options[:timeout],
+    )
+
+    # open-uri will return a StringIO instead of a Tempfile if the filesize
+    # is less than 10 KB, so if it happens we convert it back to Tempfile.
+    if downloaded_file.is_a?(StringIO)
+      stringio = downloaded_file
+      downloaded_file = copy_to_tempfile("open-uri", stringio)
+      OpenURI::Meta.init downloaded_file, stringio
+    end
+
+    downloaded_file.extend DownloadedFile
+    downloaded_file
+
+  rescue => error
+    raise if error.instance_of?(RuntimeError) && error.message !~ /redirection/
+    raise if error.is_a?(Down::Error)
+    raise Down::NotFound, error.message
+  end
+
+  def copy_to_tempfile(basename, io)
+    tempfile = Tempfile.new(basename, binmode: true)
+    IO.copy_stream(io, tempfile.path)
+    io.rewind
+    tempfile
+  end
+
+  module DownloadedFile
+    def original_filename
+      path = base_uri.path
+      path = URI.decode(path)
+      File.basename(path) unless path.empty? || path == "/"
+    end
+  end
+end

metadata

@@ -0,0 +1,91 @@
+--- !ruby/object:Gem::Specification
+name: down
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- Janko Marohnić
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2015-09-25 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.8'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.8'
+- !ruby/object:Gem::Dependency
+  name: webmock
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Robust file download from URL using open-uri.
+email:
+- janko.marohnic@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- LICENSE.txt
+- README.md
+- down.gemspec
+- lib/down.rb
+homepage: https://github.com/janko-m/down
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.5
+signing_key: 
+specification_version: 4
+summary: Robust file download from URL using open-uri.
+test_files: []
+has_rdoc: