down 2.5.1 → 3.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 0b0fd2b712dcfee3b7e3da1189103e57b2f67621
-  data.tar.gz: 9d812256dc29a14cd18d17517f8aef6d360cf37b
+  metadata.gz: c16a220821aeb2a11910334c3598ccb5b823473d
+  data.tar.gz: 21b1426c6169e82627fb445cbe0526456e9c9f19
 SHA512:
-  metadata.gz: 1174fbcef905d92928c24470f2a109fb6b96cdbc02819bfdae719e79959f3da091a76baf599f5ed2821eb42d254a56df43313fc4bcfa36d604a3553155c3e7df
-  data.tar.gz: c88b14d47f42bc78601e4cb97e86ff5b559dd25edd1c833e1638699eb04810e09cde0a77e700458b8be582f3a06a48439bc72b63b79801945523d1d4481a9f49
+  metadata.gz: 97447da54fba5009a0dca7b3013906dc5c0f707620b2ab98bd031f85e0b02180829188bb9df31daeacdcc49138aba024f630a2d9a3c5170d6f4119add90ab2c8
+  data.tar.gz: '09498dc3f7ad7256700f42fb536b76293d6f3825e2b686eb091772b22e366116a346130832ea46d28b739fe76e99f477575e3d003efa852696c8fb492128243e'

data/README.md

@@ -1,15 +1,19 @@
 # Down
 
-Down is a wrapper around [open-uri] standard library for safe downloading of
-remote files.
+Down is a utility tool for streaming, flexible and safe downloading of remote
+files. It can use [open-uri] + `Net::HTTP` or [HTTP.rb] as the backend HTTP
+library.
 
 ## Installation
 
 ```rb
-gem 'down'
+gem "down"
 ```
 
-## Usage
+## Downloading
+
+The primary method is `Down.download`, which downloads the remote file into a
+Tempfile:
 
 ```rb
 require "down"
@@ -17,37 +21,13 @@ tempfile = Down.download("http://example.com/nature.jpg")
 tempfile #=> #<Tempfile:/var/folders/k7/6zx6dx6x7ys3rv3srh0nyfj00000gn/T/20150925-55456-z7vxqz.jpg>
 ```
 
-## Downloading
-
-If you're downloading files from URLs that come from you, then it's probably
-enough to just use `open-uri`. However, if you're accepting URLs from your
-users (e.g. through `remote_<avatar>_url` in CarrierWave), then downloading is
-suddenly not as simple as it appears to be.
-
-### StringIO
-
-Firstly, you may think that `open-uri` always downloads a file to disk, but
-that's not true. If the downloaded file has 10 KB or less, `open-uri` actually
-returns a `StringIO`. In my application I needed that the file is always
-downloaded to disk. This was obviously a wrong design decision from the MRI
-team, so Down patches this behaviour and always returns a `Tempfile`.
-
-### File extension
-
-When using `open-uri` directly, the extension of the remote file is not
-preserved. Down patches that behaviour and preserves the file extension.
-
 ### Metadata
 
-`open-uri` adds some metadata to the returned file, like `#content_type`. Down
-adds `#original_filename` as well, which is extracted from the URL.
+The returned Tempfile has `#content_type` and `#original_filename` attributes
+determined from the response headers:
 
 ```rb
-require "down"
-tempfile = Down.download("http://example.com/nature.jpg")
-
-tempfile #=> #<Tempfile:/var/folders/k7/6zx6dx6x7ys3rv3srh0nyfj00000gn/T/20150925-55456-z7vxqz.jpg>
-tempfile.content_type #=> "image/jpeg"
+tempfile.content_type      #=> "image/jpeg"
 tempfile.original_filename #=> "nature.jpg"
 ```
 
@@ -65,60 +45,47 @@ Down.download("http://example.com/image.jpg", max_size: 5 * 1024 * 1024) # 5 MB
 What is the advantage over simply checking size after downloading? Well, Down
 terminates the download very early, as soon as it gets the `Content-Length`
 header. And if the `Content-Length` header is missing, Down will terminate the
-download as soon as it receives a chunk which surpasses the maximum size.
+download as soon as the downloaded content surpasses the maximum size.
 
-### Redirects
+### Basic authentication
 
-By default open-uri's redirects are turned off, since open-uri doesn't have a
-way to limit maximum number of redirects. Instead Down itself implements
-following redirects, by default allowing maximum of 2 redirects.
+`Down.download` and `Down.open` will automatically detect and apply HTTP basic
+authentication from the URL:
 
 ```rb
-Down.download("http://example.com/image.jpg")                   # 2 redirects allowed
-Down.download("http://example.com/image.jpg", max_redirects: 5) # 5 redirects allowed
-Down.download("http://example.com/image.jpg", max_redirects: 0) # 0 redirects allowed
+Down.download("http://user:password@example.org")
+Down.open("http://user:password@example.org")
 ```
 
 ### Download errors
 
 There are a lot of ways in which a download can fail:
 
-* URL is really invalid (`URI::InvalidURIError`)
-* URL is a little bit invalid, e.g. "http:/example.com" (`Errno::ECONNREFUSED`)
-* Domain was not found (`SocketError`)
-* Domain was found, but status is 4xx or 5xx (`OpenURI::HTTPError`)
-* Request timeout (`Timeout::Error`)
+* Response status was 4xx or 5xx
+* Domain was not found
+* Timeout occurred
+* URL is invalid
+* ...
 
-Down unifies all of these errors into one `Down::NotFound` error (because this
-is what actually happened from the outside perspective). If you want to get the
-actual error raised by open-uri, in Ruby 2.1 or later you can use
+Down attempts to unify all of these exceptions into one `Down::NotFound` error
+(because this is what actually happened from the outside perspective). If you
+want to retrieve the original error raised, in Ruby 2.1+ you can use
 `Exception#cause`:
 
 ```rb
 begin
   Down.download("http://example.com")
-rescue Down::Error => error
-  error.cause #=> #<RuntimeError: HTTP redirection loop: http://example.com>
+rescue Down::Error => exception
+  exception.cause #=> #<Timeout::Error>
 end
 ```
 
-### Additional options
-
-Any additional options will be forwarded to [open-uri], so you can for example
-add basic authentication or a timeout:
-
-```rb
-Down.download "http://example.com/image.jpg",
-  http_basic_authentication: ['john', 'secret'],
-  read_timeout: 5
-```
-
 ## Streaming
 
-Down has the ability to access content of the remote file *as it is being
-downloaded*. The `Down.open` method returns an IO object which represents the
-remote file on the given URL. When you read from it, Down internally downloads
-chunks of the remote file, but only how much is needed.
+Down has the ability to retrieve content of the remote file *as it is being
+downloaded*. The `Down.open` method returns a `Down::ChunkedIO` object which
+represents the remote file on the given URL. When you read from it, Down
+internally downloads chunks of the remote file, but only how much is needed.
 
 ```rb
 remote_file = Down.open("http://example.com/image.jpg")
@@ -126,44 +93,61 @@ remote_file.size # read from the "Content-Length" header
 
 remote_file.read(1024) # downloads and returns first 1 KB
 remote_file.read(1024) # downloads and returns next 1 KB
-remote_file.read       # downloads and returns the rest of the file
 
-remote_file.eof? #=> true
-remote_file.rewind
 remote_file.eof? #=> false
+remote_file.read # downloads and returns the rest of the file content
+remote_file.eof? #=> true
 
 remote_file.close # closes the HTTP connection and deletes the internal Tempfile
 ```
 
-You can also yield chunks directly as they're downloaded:
+### Caching
+
+By default the downloaded content is internally cached into a `Tempfile`, so
+that when you rewind the `Down::ChunkedIO`, it continues reading the cached
+content that it had already retrieved.
 
 ```rb
 remote_file = Down.open("http://example.com/image.jpg")
-remote_file.each_chunk do |chunk|
-  # ...
-end
-remote_file.close
+remote_file.read(1*1024*1024) # downloads, caches, and returns first 1MB
+remote_file.rewind
+remote_file.read(1*1024*1024) # reads the cached content
+remote_file.read(1*1024*1024) # downloads the next 1MB
 ```
 
-You can access the response status and headers of the HTTP request that was made:
+If you want to save on IO calls and on disk usage, and don't need to be able to
+rewind the `Down::ChunkedIO`, you can disable caching downloaded content:
+
+```rb
+Down.open("http://example.com/image.jpg", rewindable: false)
+```
+
+### Yielding chunks
+
+You can also yield chunks directly as they're downloaded via `#each_chunk`, in
+which case the downloaded content is not cached into a file regardless of the
+`:rewindable` option.
 
 ```rb
 remote_file = Down.open("http://example.com/image.jpg")
-remote_file.data[:status]  #=> 200
-remote_file.data[:headers] #=> { ... }
+remote_file.each_chunk { |chunk| ... }
+remote_file.close
 ```
 
-Note that `Down::NotFound` error will automatically be raised if response
-status was 4xx or 5xx.
+### Data
 
-`Down.open` accepts `:ssl_verify_mode` and `:ssl_ca_cert` options with the same
-semantics as in `open-uri`, and any options with String keys will be
-interpreted as request headers.
+You can access the response status and headers of the HTTP request that was made:
 
 ```rb
-Down.open("http://example.com/image.jpg", {"Authorization" => "..."})
+remote_file = Down.open("http://example.com/image.jpg")
+remote_file.data[:status]   #=> 200
+remote_file.data[:headers]  #=> { ... }
+remote_file.data[:response] # returns the response object
 ```
 
+Note that `Down::NotFound` error will automatically be raised if response
+status was 4xx or 5xx.
+
 ### `Down::ChunkedIO`
 
 The `Down.open` method uses `Down::ChunkedIO` internally. However,
@@ -173,9 +157,12 @@ The `Down.open` method uses `Down::ChunkedIO` internally. However,
 Down::ChunkedIO.new(...)
 ```
 
-* `:size` – size of the file, if it's known
-* `:chunks` – an `Enumerator` which returns chunks
-* `:on_close` – called when streaming finishes
+* `:chunks` – an `Enumerator` which retrieves chunks
+* `:size` – size of the file if it's known (returned by `#size`)
+* `:on_close` – called when streaming finishes or IO is closed
+* `:data` - custom data that you want to store (returned by `#data`)
+* `:rewindable` - whether to cache retrieved data into a file (defaults to `true`)
+* `:encoding` - force content to be returned in specified encoding (defaults to ASCII-8BIT)
 
 Here is an example of wrapping streaming MongoDB files:
 
@@ -185,7 +172,7 @@ require "down/chunked_io"
 mongo = Mongo::Client.new(...)
 bucket = mongo.database.fs
 
-content_length = bucket.find(_id: id).first["length"]
+content_length = bucket.find(_id: id).first[:length]
 stream = bucket.open_download_stream(id)
 
 io = Down::ChunkedIO.new(
@@ -195,6 +182,42 @@ io = Down::ChunkedIO.new(
 )
 ```
 
+## open-uri + Net::HTTP
+
+Then [open-uri] + Net::HTTP is the default backend, loaded by requiring `down`
+or `down/net_http`:
+
+```rb
+require "down"
+# or
+require "down/net_http"
+```
+
+`Down.download` is implemented as a wrapper around open-uri, and fixes some of
+open-uri's undesired behaviours:
+
+* open-uri returns `StringIO` for files smaller than 10KB, and `Tempfile`
+  otherwise, but `Down.download` always returns a `Tempfile`
+* open-uri doesn't give any extension to the returned `Tempfile`, but
+  `Down.download` adds the extension from the URL
+* ...
+
+Since open-uri doesn't expose support for partial downloads, `Down.open` is
+implemented using `Net::HTTP` directly.
+
+### Redirects
+
+`Down.download` turns off open-uri's following redirects, as open-uri doesn't
+have a way to limit the maximum number of hops, and implements its own. By
+default maximum of 2 redirects will be followed, but you can change it via the
+`:max_redirects` option:
+
+```rb
+Down.download("http://example.com/image.jpg")                   # 2 redirects allowed
+Down.download("http://example.com/image.jpg", max_redirects: 5) # 5 redirects allowed
+Down.download("http://example.com/image.jpg", max_redirects: 0) # 0 redirects allowed
+```
+
 ### Proxy
 
 Both `Down.download` and `Down.open` support a `:proxy` option, where you can
@@ -202,26 +225,89 @@ specify a URL to an HTTP proxy which should be used when downloading.
 
 ```rb
 Down.download("http://example.com/image.jpg", proxy: "http://proxy.org")
-Down.open("http://example.com/image.jpg", proxy: "http://user:password@proxy.org")
+Down.open("http://example.com/image.jpg",     proxy: "http://user:password@proxy.org")
+```
+
+### Additional options
+
+Any additional options passed to `Down.download` will be forwarded to
+[open-uri], so you can for example add basic authentication or a timeout:
+
+```rb
+Down.download "http://example.com/image.jpg",
+  http_basic_authentication: ['john', 'secret'],
+  read_timeout: 5
+```
+
+`Down.open` accepts `:ssl_verify_mode` and `:ssl_ca_cert` options with the same
+semantics as in open-uri, and any options with String keys will be interpreted
+as request headers, like with open-uri.
+
+```rb
+Down.open("http://example.com/image.jpg", {"Authorization" => "..."})
+```
+
+## HTTP.rb
+
+The [HTTP.rb] backend can be used by requiring `down/http`:
+
+```rb
+gem "http", "~> 2.1"
+gem "down"
+```
+```rb
+require "down/http"
 ```
 
-### Copying to tempfile
+Some features that give the HTTP.rb backend an advantage over open-uri +
+Net::HTTP include:
+
+* Correct URI parsing with [Addressable::URI]
+* Proper support for streaming downloads (`#download` and now reuse `#open`)
+* Proper support for SSL
+* Chaninable HTTP client builder API for setting default options
+* Persistent connections
+* Auto-inflating compressed response bodies
+* ...
 
-Down has another "hidden" utility method, `#copy_to_tempfile`, which creates
-a Tempfile out of the given file. The `#download` method uses it internally,
-but it's also publicly available for direct use:
+### Default client
+
+You can change the default `HTTP::Client` to be used in all download requests
+via `Down::Http.client`:
 
 ```rb
-io # IO object that you want to copy to tempfile
-tempfile = Down.copy_to_tempfile "basename.jpg", io
-tempfile.path #=> "/var/folders/k7/6zx6dx6x7ys3rv3srh0nyfj00000gn/T/down20151116-77262-jgcx65.jpg"
+# reuse Down's default client
+Down::Http.client = Down::Http.client.timeout(read: 3).feature(:auto_inflate)
+Down::Http.client.default_options.merge!(ssl_context: ctx)
+
+# or set a new client
+Down::Http.client = HTTP.via("proxy-hostname.local", 8080)
 ```
 
+### Additional options
+
+All additional options passed to `Down::Download` and `Down.open` will be
+forwarded to `HTTP::Client#request`:
+
+```rb
+Down.download("http://example.org/image.jpg", headers: {"Accept-Encoding" => "gzip"})
+```
+
+If you prefer to add options using the chainable API, you can pass a block:
+
+```rb
+Down.open("http://example.org/image.jpg") do |client|
+  client.timeout(read: 3)
+end
+```
+
+### Thread safety
+
+`Down::Http.client` is stored in a thread-local variable, so using the HTTP.rb
+backend is thread safe.
+
 ## Supported Ruby versions
 
-* MRI 1.9.3
-* MRI 2.0
-* MRI 2.1
 * MRI 2.2
 * MRI 2.3
 * MRI 2.4
@@ -229,14 +315,17 @@ tempfile.path #=> "/var/folders/k7/6zx6dx6x7ys3rv3srh0nyfj00000gn/T/down20151116
 
 ## Development
 
+The test suite runs the http://httpbin.org/ server locally, and uses it to test
+downloads. Httpbin is a Python package which is run with GUnicorn:
+
 ```
-$ rake test
+$ pip install gunicorn httpbin
 ```
 
-If you want to test across Ruby versions and you're using rbenv, run
+Afterwards you can run tests with
 
 ```
-$ bin/test-versions
+$ rake test
 ```
 
 ## License
@@ -244,3 +333,5 @@ $ bin/test-versions
 [MIT](LICENSE.txt)
 
 [open-uri]: http://ruby-doc.org/stdlib-2.3.0/libdoc/open-uri/rdoc/OpenURI.html
+[HTTP.rb]: https://github.com/httprb/http
+[Addressable::URI]: https://github.com/sporkmonger/addressable

data/down.gemspec

@@ -1,21 +1,21 @@
 require File.expand_path("../lib/down/version", __FILE__)
 
 Gem::Specification.new do |spec|
-  spec.name          = "down"
-  spec.version       = Down::VERSION
-  spec.authors       = ["Janko Marohnić"]
-  spec.email         = ["janko.marohnic@gmail.com"]
+  spec.name         = "down"
+  spec.version      = Down::VERSION
 
-  spec.summary       = "Robust streaming downloads using net/http."
-  spec.homepage      = "https://github.com/janko-m/down"
-  spec.license       = "MIT"
+  spec.required_ruby_version = ">= 2.1"
 
-  spec.files         = Dir["README.md", "LICENSE.txt", "*.gemspec", "lib/**/*.rb"]
-  spec.require_paths = ["lib"]
+  spec.summary      = "Robust streaming downloads using net/http."
+  spec.homepage     = "https://github.com/janko-m/down"
+  spec.authors      = ["Janko Marohnić"]
+  spec.email        = ["janko.marohnic@gmail.com"]
+  spec.license      = "MIT"
+
+  spec.files        = Dir["README.md", "LICENSE.txt", "*.gemspec", "lib/**/*.rb"]
+  spec.require_path = "lib"
 
-  spec.add_development_dependency "rake"
   spec.add_development_dependency "minitest", "~> 5.8"
-  spec.add_development_dependency "webmock", "~> 2.3"
-  spec.add_development_dependency "addressable", "< 2.5"
   spec.add_development_dependency "mocha"
+  spec.add_development_dependency "http", "~> 2.1"
 end