metainspector 5.19.0 → 5.20.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5bc16dfdcdd22661ba7bb196d4db8750a5fa4d48d054197982768fa7fac44835
-  data.tar.gz: ce356b1f0f2511069ff96329e775e3581282d68d395c4dada93a9275e5dd4aa6
+  metadata.gz: ba31ea21114c1932ca2dde179a2812f24d429b7420807fec232de65ea1ba824e
+  data.tar.gz: 99188cbec093cfd1d22cdfd3d31bf181cf193e937f6c7b23a61cb6ed39ba1ca2
 SHA512:
-  metadata.gz: 6a669057baeb8107c8ae6940f4323f60664982fa3c8c74ce4f029b434ebfe227436c743a6b6041acbcddfdc2ea7fb38704a6126d3204b18479a8ca81b2b63a0e
-  data.tar.gz: 123f1cbb41fd5492307d96e316b17c91fc4f17d523a12f9fd865be49a3de31d8b932ceae6365bc7e1f6f49fd9721f8626eb9a8b5bbcbf9d86b2e972aedaa1544
+  metadata.gz: bc39e09f57cadd29f60e6a41556878d807a629e99de37b6b64b3ebf090d32aeca63d49b38af436fa16e5fff29e4cf5e927d99df5b37f35385ea30165a8de55e3
+  data.tar.gz: d4a4777631f8c63d238236960148330a3dc7ba0f4ae2ec0b497cc3e03f33a5ca8daca05141dbaaa20a99a54fd933d5a5c9cb7056dd4b84b12c1a515119935574

data/CHANGELOG.md

@@ -1,5 +1,10 @@
 # MetaInspector Changelog
 
+## [Changes in 5.20.0](https://github.com/jaimeiniesta/metainspector/compare/v5.19.0...v5.20.0)
+
+* Security: route image size downloads through Faraday so they respect configured network and redirect controls. Add optional `url_filter:` to validate the initial page URL, HEAD requests, image URLs and every redirect destination before requesting them. Returning `false` or `nil` blocks the request with `MetaInspector::RequestError`; filter exceptions propagate. Only HTTP(S) requests are supported ([GHSA-6vgc-f9v5-8939](https://github.com/jaimeiniesta/metainspector/security/advisories/GHSA-6vgc-f9v5-8939)).
+* Accept `faraday_http_cache` inside `faraday_options`, which used to raise `NameError`. The top-level `faraday_http_cache` option still works but is deprecated: it prints a warning once per process. Passing both raises `ArgumentError` (#373).
+
 ## [Changes in 5.19.0](https://github.com/jaimeiniesta/metainspector/compare/v5.18.1...v5.19.0)
 
 * Security: stop modifying the `faraday_options` hash passed to MetaInspector. When the same hash was reused, its `redirect` options were deleted after the first request, so a redirect `callback` used to block private hosts only ran once and later requests followed any redirect ([GHSA-mm2p-6q2v-7ph5](https://github.com/jaimeiniesta/metainspector/security/advisories/GHSA-mm2p-6q2v-7ph5)). Frozen option hashes no longer raise `FrozenError`.

data/README.md

@@ -341,8 +341,7 @@ And even customize what to do in between each redirect:
 
 ```ruby
 callback = proc do |previous_response, next_request|
-  ip_address = Resolv.getaddress(next_request.url.host)
-  raise 'Invalid address' if IPAddr.new(ip_address).private?
+  puts "Following redirect to #{next_request.url}"
 end
 
 page = MetaInspector.new(url, faraday_options: { redirect: { callback: callback } })
@@ -352,6 +351,63 @@ page = MetaInspector.new(url, faraday_options: { redirect: { callback: callback
 The `faraday_options[:redirect]` hash is passed to the `FollowRedirects` middleware used by `Faraday`, so that we can use all available options.
 Check them [here](https://github.com/lostisland/faraday_middleware/blob/main/lib/faraday_middleware/response/follow_redirects.rb#L44).
 
+### Filtering request URLs
+
+Use `url_filter:` to decide which destinations MetaInspector may request. The filter receives a Ruby `URI` and runs before the initial page request, the optional HEAD request, image size downloads, and every redirect destination. Only HTTP and HTTPS requests are supported.
+
+For example, allow HTTPS requests to two known hosts:
+
+```ruby
+allowed_hosts = %w[example.com images.example.com]
+
+filter = proc do |uri|
+  uri.scheme == 'https' && allowed_hosts.include?(uri.hostname)
+end
+
+page = MetaInspector.new('https://example.com', url_filter: filter)
+page.images.largest # Image requests and their redirects use the same filter.
+```
+
+A truthy return value permits the request. Returning `false` or `nil` raises `MetaInspector::RequestError` before contacting the rejected destination. This also applies when an image request is rejected:
+
+```ruby
+begin
+  page = MetaInspector.new(url, url_filter: filter)
+  page.images.largest
+rescue MetaInspector::RequestError => error
+  warn error.message
+end
+```
+
+Exceptions raised by your filter propagate to the caller. The filter is optional: MetaInspector does not block private IP addresses by default. It applies to outgoing requests, including image downloads from HTML passed with `document:`. Reading image URLs with `images.to_a` or `images.owner_suggested` does not download them or run the filter.
+
+If your application accepts arbitrary hostnames, this example rejects addresses in private, loopback, link-local and unspecified ranges, including IPv4-mapped IPv6 addresses:
+
+```ruby
+require 'ipaddr'
+require 'resolv'
+
+zero_network = IPAddr.new('0.0.0.0/8')
+
+filter = proc do |uri|
+  addresses = Resolv.getaddresses(uri.hostname)
+
+  !addresses.empty? && addresses.all? do |address|
+    ip = IPAddr.new(address)
+    ip = ip.native if ip.ipv4_mapped?
+
+    !(ip.private? || ip.loopback? || ip.link_local? ||
+      ip.to_i.zero? || zero_network.include?(ip))
+  end
+end
+
+page = MetaInspector.new(url, url_filter: filter)
+```
+
+DNS resolution in this example happens separately from the connection, so it cannot prevent DNS rebinding. Use network egress restrictions as well when processing untrusted URLs. The example covers the listed ranges; applications may need additional address or port restrictions.
+
+The existing redirect `callback` keeps its two arguments and runs only on redirects. Move destination validation from that callback to `url_filter:` to cover initial URLs as well as redirects, for both pages and images.
+
 ### Headers
 
 By default, the following headers are set:
@@ -447,7 +503,13 @@ You can disable URL normalization by passing the `normalize_url: false` option.
 
 ### Image downloading
 
-When you ask for the largest image on the page with `page.images.largest`, it will be determined by its height and width attributes on the HTML markup, and also by downloading a small portion of each image using the [fastimage](https://github.com/sdsykes/fastimage) gem. This is really fast as it doesn't download the entire images, normally just the headers of the image files.
+Image downloading is enabled by default. `page.images.with_size` and `page.images.largest` fetch images with missing HTML dimensions; `page.images.best` also does this when there is no preferred metadata image. The page controls these image URLs, which may point to other hosts.
+
+Image requests use Faraday with the page's proxy, TLS, headers and redirect settings, and respect [`url_filter:`](#filtering-request-urls). Downloaded bytes are passed to [fastimage](https://github.com/sdsykes/fastimage) to determine dimensions. Dimensions already present in HTML are preserved without downloading the image.
+
+Custom page headers are preserved for images on the same origin (scheme, host and port). Requests to other origins, including image redirects, send only standard image request headers such as `User-Agent` and `Accept`; they do not receive page credentials or arbitrary custom headers.
+
+Each size lookup stops at a 256 KiB response body budget or a two-second overall timeout, including redirects. If the dimensions cannot be determined within these limits, they remain `[0, 0]`. Streamed image responses are not stored in the HTTP response cache, and failed image requests are not retried.
 
 If you want to disable this, you can specify it like this:
 
@@ -470,13 +532,15 @@ The limit leaves `images.to_a` unchanged, but can affect `images.largest` and th
 
 ### Caching responses
 
-MetaInspector can be configured to use [Faraday::HttpCache](https://github.com/plataformatec/faraday-http-cache) to cache page responses. For that you should pass the `faraday_http_cache` option with at least the `:store` key, for example:
+MetaInspector can be configured to use [Faraday::HttpCache](https://github.com/plataformatec/faraday-http-cache) to cache page responses. For that you should pass the `faraday_http_cache` option inside `faraday_options`, with at least the `:store` key, for example:
 
 ```ruby
 cache = ActiveSupport::Cache.lookup_store(:file_store, '/tmp/cache')
-page = MetaInspector.new('http://example.com', faraday_http_cache: { store: cache })
+page = MetaInspector.new('http://example.com', faraday_options: { faraday_http_cache: { store: cache } })
 ```
 
+The top-level `faraday_http_cache` option still works, but it is deprecated and prints a warning, once per process. Passing it in both places raises an `ArgumentError`.
+
 ## Exception Handling
 
 Web page scraping is tricky, you can expect to find different exceptions during the request of the page or the parsing of its contents. MetaInspector will encapsulate these exceptions on these main errors:

data/lib/meta_inspector/document.rb

@@ -15,7 +15,10 @@ module MetaInspector
     # * normalize_url: true by default
     # * download_images: fetch missing image dimensions. Defaults to true
     # * max_image_downloads: maximum image size lookups, or nil for unlimited
-    # * faraday_options: an optional hash of options to pass to Faraday on the request
+    # * url_filter: callable receiving a URI before each request; false/nil rejects it
+    # * faraday_options: an optional hash of options to pass to Faraday on the request.
+    #   Its faraday_http_cache key configures response caching with Faraday::HttpCache
+    # * faraday_http_cache: deprecated, use faraday_options: { faraday_http_cache: ... }
     def initialize(initial_url, options = {})
       options             = defaults.merge(options)
       @max_image_downloads = options[:max_image_downloads]
@@ -37,8 +40,10 @@ module MetaInspector
       @normalize_url      = options[:normalize_url]
       @faraday_options    = options[:faraday_options]
       @faraday_http_cache = options[:faraday_http_cache]
+      @http_session       = MetaInspector::HttpSession.new(options)
       @url                = MetaInspector::URL.new(initial_url, normalize:          @normalize_url)
-      @request            = MetaInspector::Request.new(@url,    allow_redirections: @allow_redirections,
+      @request            = MetaInspector::Request.new(@url,    http_session: @http_session,
+                                                                allow_redirections: @allow_redirections,
                                                                 head_first:         options[:head_first],
                                                                 allow_non_html_content: @allow_non_html_content,
                                                                 connection_timeout: @connection_timeout,
@@ -49,7 +54,8 @@ module MetaInspector
                                                                 faraday_options:    @faraday_options,
                                                                 faraday_http_cache: @faraday_http_cache) unless @document
       @parser             = MetaInspector::Parser.new(self,     download_images:    @download_images,
-                                                                max_image_downloads: @max_image_downloads)
+                                                                max_image_downloads: @max_image_downloads,
+                                                                image_downloader: MetaInspector::ImageDownloader.new(@http_session, @url, options))
     end
 
     extend Forwardable

data/lib/meta_inspector/http_session.rb

@@ -0,0 +1,65 @@
+require 'http/cookie'
+require 'faraday'
+require 'faraday-cookie_jar'
+require 'faraday-http-cache'
+require 'faraday/encoding'
+require 'faraday/follow_redirects'
+require 'faraday/gzip'
+require 'faraday/retry'
+require_relative 'url_policy'
+
+module MetaInspector
+  # Builds page and binary image transports from the same caller configuration.
+  class HttpSession
+    def initialize(options = {})
+      @allow_redirections = options[:allow_redirections]
+      @retries = options[:retries]
+      @headers = options[:headers]
+      @url_filter = options[:url_filter]
+      if !@url_filter.nil? && !@url_filter.respond_to?(:call)
+        raise ArgumentError, 'url_filter must respond to call'
+      end
+      @cache = cache_option(options)
+      @faraday_options = (options[:faraday_options] || {}).except(:faraday_http_cache)
+    end
+
+    def build(url, image: false, origin: nil)
+      URLPolicy.validate!(url)
+      faraday_options = @faraday_options.merge(url: url)
+      redirects = (faraday_options.delete(:redirect) || {}).dup
+      Faraday.new(faraday_options) do |faraday|
+        unless image
+          faraday.request :retry, max: @retries
+          faraday.request :gzip
+        end
+        if @allow_redirections
+          redirects[:limit] ||= 10
+          faraday.use Faraday::FollowRedirects::Middleware, **redirects
+          faraday.use :cookie_jar
+        end
+        faraday.use URLPolicy, @url_filter, image ? origin : nil
+        if !image && @cache.is_a?(Hash)
+          cache_options = @cache.dup
+          cache_options[:serializer] ||= Marshal
+          faraday.use Faraday::HttpCache, **cache_options
+        end
+        faraday.headers.merge!(@headers || {})
+        faraday.response :encoding unless image
+        faraday.adapter :net_http
+      end
+    end
+
+    private
+
+    def cache_option(options)
+      nested = (options[:faraday_options] || {})[:faraday_http_cache]
+      top_level = options[:faraday_http_cache]
+      return nested if top_level.nil?
+      unless nested.nil?
+        raise ArgumentError, 'Pass faraday_http_cache inside faraday_options or as a top-level option, not both'
+      end
+      Request.warn_faraday_http_cache_deprecation
+      top_level
+    end
+  end
+end

data/lib/meta_inspector/image_downloader.rb

@@ -0,0 +1,53 @@
+require 'stringio'
+require 'fastimage'
+require 'timeout'
+
+module MetaInspector
+  class ImageDownloader
+    MAX_BYTES = 256 * 1024
+    TIMEOUT = 2
+
+    def initialize(session, origin, options = {})
+      @session = session
+      @origin = origin
+      @connection_timeout = [options[:connection_timeout] || TIMEOUT, TIMEOUT].min
+      @read_timeout = [options[:read_timeout] || TIMEOUT, TIMEOUT].min
+    end
+
+    def size(url)
+      bytes = ''.b
+      received = 0
+      status = nil
+      stop = Object.new
+      Timeout.timeout(TIMEOUT) do
+        session = @session.build(url, image: true, origin: @origin.url)
+        catch(stop) do
+          session.get do |request|
+            request.headers['Range'] = "bytes=0-#{MAX_BYTES - 1}"
+            request.headers['Accept-Encoding'] = 'identity'
+            request.options.open_timeout = @connection_timeout
+            request.options.timeout = @read_timeout
+            request.options.on_data = proc do |chunk, _total, env|
+              status = env.status
+              remaining = MAX_BYTES - received
+              received += chunk.bytesize
+              # Count redirect bodies too, across the entire lookup.
+              if (200..299).cover?(status)
+                bytes << chunk.b.byteslice(0, remaining)
+              elsif !(300..399).cover?(status)
+                throw stop
+              end
+              throw stop if received >= MAX_BYTES
+            end
+          end
+        end
+      end
+      FastImage.size(StringIO.new(bytes)) if status && (200..299).cover?(status)
+    rescue URLPolicy::Failure => failure
+      raise failure.original
+    rescue Faraday::TimeoutError, Faraday::ConnectionFailed, Faraday::SSLError,
+           Faraday::FollowRedirects::RedirectLimitReached, URI::InvalidURIError, Timeout::Error
+      nil
+    end
+  end
+end

data/lib/meta_inspector/parser.rb

@@ -15,7 +15,8 @@ module MetaInspector
       @links_parser    = MetaInspector::Parsers::LinksParser.new(self)
       @download_images = options[:download_images]
       @images_parser   = MetaInspector::Parsers::ImagesParser.new(self, download_images: @download_images,
-                                                                      max_image_downloads: options[:max_image_downloads])
+                                                                      max_image_downloads: options[:max_image_downloads],
+                                                                      image_downloader: options[:image_downloader])
       @texts_parser    = MetaInspector::Parsers::TextsParser.new(self)
 
       parsed           # parse early so we can fail early

data/lib/meta_inspector/parsers/images.rb

@@ -1,4 +1,4 @@
-require 'fastimage'
+require_relative '../image_downloader'
 
 module MetaInspector
   module Parsers
@@ -9,6 +9,7 @@ module MetaInspector
       include Enumerable
 
       def initialize(main_parser, options = {})
+        @image_downloader = options[:image_downloader]
         @download_images = options[:download_images]
         @max_image_downloads = options[:max_image_downloads]
         super(main_parser)
@@ -48,7 +49,7 @@ module MetaInspector
             if width.nil? || height.nil?
               if @download_images && (@max_image_downloads.nil? || download_attempts < @max_image_downloads)
                 download_attempts += 1
-                width, height = FastImage.size(url)
+                width, height = @image_downloader.size(url)
               else
                 width, height = [0, 0]
               end

data/lib/meta_inspector/request.rb

@@ -1,12 +1,5 @@
-require 'http/cookie'
-require 'faraday'
-require 'faraday-cookie_jar'
-require 'faraday-http-cache'
-require 'faraday/encoding'
-require 'faraday/follow_redirects'
-require 'faraday/gzip'
-require 'faraday/retry'
 require 'timeout'
+require_relative 'http_session'
 
 module MetaInspector
 
@@ -15,18 +8,15 @@ module MetaInspector
     def initialize(initial_url, options = {})
       @url                = initial_url
 
-      fail MetaInspector::RequestError.new('URL must be HTTP') unless @url.url =~ /http[s]?:\/\//i
+      URLPolicy.validate!(@url.url)
 
       @allow_redirections = options[:allow_redirections]
       @head_first         = options[:head_first]
       @allow_non_html_content = options[:allow_non_html_content]
       @connection_timeout = options[:connection_timeout]
       @read_timeout       = options[:read_timeout]
-      @retries            = options[:retries]
       @encoding           = options[:encoding]
-      @headers            = options[:headers]
-      @faraday_options    = options[:faraday_options] || {}
-      @faraday_http_cache = options[:faraday_http_cache]
+      @http_session       = options[:http_session] || HttpSession.new(options)
 
       response            # request early so we can fail early
     end
@@ -49,6 +39,8 @@ module MetaInspector
 
     def response
       @response ||= fetch
+    rescue URLPolicy::Failure => failure
+      raise failure.original
     rescue Faraday::TimeoutError => e
       raise MetaInspector::TimeoutError.new(e)
     rescue Faraday::ConnectionFailed, Faraday::SSLError, URI::InvalidURIError, Faraday::FollowRedirects::RedirectLimitReached => e
@@ -57,34 +49,20 @@ module MetaInspector
 
     private
 
+    # Once per process: apps fetch many pages with the same options, and a warning
+    # per request would flood their logs
+    def self.warn_faraday_http_cache_deprecation
+      return if @faraday_http_cache_deprecation_warned
+
+      @faraday_http_cache_deprecation_warned = true
+      warn 'DEPRECATION WARNING: the top-level faraday_http_cache option is deprecated and will be removed ' \
+           'in the next major version. Pass it inside faraday_options instead: ' \
+           'faraday_options: { faraday_http_cache: { store: cache } }'
+    end
+
     def fetch
       Timeout::timeout(fatal_timeout) do
-        # Work on copies: the caller may reuse or freeze the hashes it passed in, and
-        # losing :redirect would drop a callback that blocks redirects to private hosts
-        faraday_options          = @faraday_options.merge(:url => url)
-        follow_redirects_options = (faraday_options.delete(:redirect) || {}).dup
-
-        session = Faraday.new(faraday_options) do |faraday|
-          faraday.request :retry, max: @retries
-
-          faraday.request :gzip
-
-          if @allow_redirections
-            follow_redirects_options[:limit] ||= 10
-            faraday.use Faraday::FollowRedirects::Middleware, **follow_redirects_options
-            faraday.use :cookie_jar
-          end
-
-          if @faraday_http_cache.is_a?(Hash)
-            cache_options = @faraday_http_cache.dup
-            cache_options[:serializer] ||= Marshal
-            faraday.use Faraday::HttpCache, **cache_options
-          end
-
-          faraday.headers.merge!(@headers || {})
-          faraday.response :encoding
-          faraday.adapter :net_http
-        end
+        session = @http_session.build(url)
 
         check_head_content_type(session) if @head_first
         response = perform_request(session, :get)

data/lib/meta_inspector/url_policy.rb

@@ -0,0 +1,58 @@
+module MetaInspector
+  # Runs inside redirect handling, before cache access or the network adapter.
+  class URLPolicy < Faraday::Middleware
+    class Failure < StandardError
+      attr_reader :original
+
+      def initialize(original)
+        @original = original
+        super('URL policy failed')
+      end
+    end
+
+    IMAGE_HEADERS = %w[user-agent accept accept-language accept-encoding range].freeze
+
+    def self.validate!(value)
+      uri = URI.parse(value.to_s)
+      unless uri.is_a?(URI::HTTP) && uri.hostname && !uri.hostname.empty?
+        raise MetaInspector::RequestError, 'URL must be HTTP or HTTPS'
+      end
+      uri
+    rescue URI::InvalidURIError => error
+      raise MetaInspector::RequestError.new(error)
+    end
+
+    def initialize(app, filter, image_origin = nil)
+      super(app)
+      @filter = filter
+      @image_origin = image_origin && self.class.validate!(image_origin)
+    end
+
+    def call(env)
+      uri = self.class.validate!(env.url)
+      if @image_origin && origin(uri) != origin(@image_origin)
+        env.request_headers.delete_if { |key, _| !IMAGE_HEADERS.include?(key.downcase) }
+      end
+      check_filter(uri) if @filter
+      @app.call(env)
+    end
+
+    private
+
+    def origin(uri)
+      [uri.scheme.downcase, uri.hostname.downcase, uri.port]
+    end
+
+    def check_filter(uri)
+      # Parsing detaches the strings too: URI#dup alone shares its hostname.
+      begin
+        allowed = @filter.call(URI.parse(uri.to_s).freeze)
+      rescue StandardError => error
+        raise Failure.new(error)
+      end
+      unless allowed
+        raise Failure.new(MetaInspector::RequestError.new('Request blocked by url_filter'))
+      end
+    end
+  end
+end

data/lib/meta_inspector/version.rb

@@ -1,3 +1,3 @@
 module MetaInspector
-  VERSION = '5.19.0'
+  VERSION = '5.20.0'
 end

data/spec/image_download_spec.rb

@@ -0,0 +1,114 @@
+require 'spec_helper'
+require 'socket'
+
+RSpec.describe 'bounded image downloads' do
+  let(:png) { "\x89PNG\r\n\x1a\n".b + [13].pack('N') + 'IHDR' + [37, 29].pack('NN') + "\x08\x02\x00\x00\x00".b }
+
+  it 'inherits proxy and TLS options without modifying caller-owned hashes' do
+    image = 'https://images.example/image.png'
+    stub_request(:get, image).to_return(body: png)
+    ssl = { verify: false }.freeze
+    redirect = { limit: 2 }.freeze
+    options = { proxy: 'http://proxy.example:3128', ssl: ssl, redirect: redirect }.freeze
+    expect(Faraday).to receive(:new).with(hash_including(proxy: options[:proxy], ssl: ssl)).and_call_original
+    page = MetaInspector.new('https://page.example/', document: %(<img src="#{image}">), faraday_options: options)
+    expect(page.images.with_size).to eq([[image, 37, 29]])
+    expect(options).to eq(proxy: 'http://proxy.example:3128', ssl: ssl, redirect: redirect)
+  end
+
+  it 'does not cache streamed image responses' do
+    cache = Class.new(Hash) do
+      def read(key); self[key]; end
+      def write(key, value); self[key] = value; end
+    end.new
+    stub_request(:get, 'https://images.example/image.png').to_return(body: png,
+      headers: { 'Cache-Control' => 'public, max-age=3600' })
+    2.times do
+      page = MetaInspector.new('https://page.example/', document: '<img src="https://images.example/image.png">',
+        faraday_options: { faraday_http_cache: { store: cache } })
+      expect(page.images.with_size.first.last(2)).to eq([37, 29])
+    end
+    expect(cache).to be_empty
+    expect(a_request(:get, 'https://images.example/image.png')).to have_been_made.twice
+  end
+
+  context 'with a real HTTP server' do
+    around do |example|
+      @server = TCPServer.new('127.0.0.1', 0)
+      @url = "http://127.0.0.1:#{@server.addr[1]}/image"
+      # WebMock buffers real responses before yielding to Net::HTTP's block.
+      # Disable its adapter here to exercise actual incremental reads.
+      WebMock.disable!
+      example.run
+    ensure
+      @server&.close
+      @worker&.kill
+      @worker&.join
+      WebMock.enable!
+      WebMock.disable_net_connect!
+    end
+
+    def serve(&block)
+      @worker = Thread.new do
+        socket = @server.accept
+        while (line = socket.gets) && line != "\r\n"; end
+        block.call(socket)
+      rescue Errno::EPIPE, Errno::ECONNRESET, IOError
+        # Reaching the byte/time limit closes the peer connection.
+      ensure
+        socket&.close
+      end
+    end
+
+    it 'bounds binary IO and stops a server that ignores Range before reading its full body' do
+      sent = 0
+      serve do |socket|
+        socket.write("HTTP/1.1 200 OK\r\nContent-Length: 4194304\r\nContent-Type: image/png\r\nConnection: close\r\n\r\n")
+        socket.write(png)
+        1024.times do
+          socket.write('x' * 4096)
+          sent += 4096
+          sleep 0.002
+        end
+      end
+      expect(FastImage).to receive(:size).with(an_instance_of(StringIO)) do |io|
+        expect(io.string.bytesize).to eq(256 * 1024)
+        expect(io.string.encoding).to eq(Encoding::BINARY)
+        [37, 29]
+      end
+      page = MetaInspector.new('http://page.example/', document: %(<img src="#{@url}">))
+      expect(page.images.with_size).to eq([[@url, 37, 29]])
+      expect(sent).to be < 4 * 1024 * 1024
+    end
+
+    it 'also stops oversized redirect bodies before following their location' do
+      sent = 0
+      serve do |socket|
+        socket.write("HTTP/1.1 302 Found\r\nLocation: /next\r\nContent-Length: 4194304\r\nConnection: close\r\n\r\n")
+        1024.times do
+          socket.write('x' * 4096)
+          sent += 4096
+          sleep 0.0005
+        end
+      end
+      page = MetaInspector.new('http://page.example/', document: %(<img src="#{@url}">))
+      expect(page.images.with_size).to eq([[@url, 0, 0]])
+      expect(sent).to be < 4 * 1024 * 1024
+    end
+
+    it 'bounds the entire image request even while response data keeps arriving' do
+      serve do |socket|
+        socket.write("HTTP/1.1 200 OK\r\nContent-Length: 1000000\r\nConnection: close\r\n\r\n")
+        loop do
+          socket.write('x')
+          sleep 0.02
+        end
+      end
+      start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      page = MetaInspector.new('http://page.example/', document: %(<img src="#{@url}">))
+      expect(page.images.with_size).to eq([[@url, 0, 0]])
+      elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - start
+      expect(elapsed).to be < 2.8
+    end
+  end
+end

data/spec/meta_inspector/meta_inspector_spec.rb

@@ -5,14 +5,65 @@ describe MetaInspector do
     expect(MetaInspector.new('http://example.com').class).to eq(MetaInspector::Document)
   end
 
-  it "cache request" do
-    # Creates a memory cache (a Hash that responds to #read, #write and #delete)
-    cache = Hash.new
-    def cache.read(k) self[k]; end
-    def cache.write(k, v) self[k] = v; end
+  describe "response caching" do
+    # A memory cache (a Hash that responds to #read and #write)
+    let(:cache) do
+      store = Hash.new
+      def store.read(k) self[k]; end
+      def store.write(k, v) self[k] = v; end
+      store
+    end
 
-    MetaInspector.new('http://example.com', faraday_http_cache: { store: cache })
+    # The deprecation warning prints once per process, so each example starts as a fresh process would
+    before { MetaInspector::Request.instance_variable_set(:@faraday_http_cache_deprecation_warned, nil) }
 
-    expect(cache.keys).not_to be_empty
+    it "caches responses with faraday_options[:faraday_http_cache]" do
+      MetaInspector.new('http://example.com', faraday_options: { faraday_http_cache: { store: cache } })
+
+      expect(cache.keys).not_to be_empty
+    end
+
+    it "does not warn when the cache is configured inside faraday_options" do
+      expect do
+        MetaInspector.new('http://example.com', faraday_options: { faraday_http_cache: { store: cache } })
+      end.not_to output.to_stderr
+    end
+
+    it "leaves the faraday_options hash it was given unchanged" do
+      cache_options   = { store: cache }
+      faraday_options = { faraday_http_cache: cache_options }
+
+      MetaInspector.new('http://example.com', faraday_options: faraday_options)
+
+      expect(faraday_options).to eq({ faraday_http_cache: { store: cache } })
+      expect(cache_options).to eq({ store: cache })
+    end
+
+    it "still caches responses with the deprecated top-level faraday_http_cache option" do
+      expect do
+        MetaInspector.new('http://example.com', faraday_http_cache: { store: cache })
+      end.to output.to_stderr
+
+      expect(cache.keys).not_to be_empty
+    end
+
+    it "warns that the top-level faraday_http_cache option is deprecated" do
+      expect do
+        MetaInspector.new('http://example.com', faraday_http_cache: { store: cache })
+      end.to output(/DEPRECATION.*faraday_http_cache.*faraday_options/).to_stderr
+    end
+
+    it "prints the deprecation warning only once per process" do
+      expect do
+        3.times { MetaInspector.new('http://example.com', faraday_http_cache: { store: cache }) }
+      end.to output(/\ADEPRECATION WARNING: [^\n]*\n\z/).to_stderr
+    end
+
+    it "raises ArgumentError when the cache is configured in both places" do
+      expect do
+        MetaInspector.new('http://example.com', faraday_http_cache: { store: cache },
+                                                faraday_options:    { faraday_http_cache: { store: cache } })
+      end.to raise_error(ArgumentError, /faraday_http_cache/)
+    end
   end
 end

data/spec/request_spec.rb

@@ -91,7 +91,7 @@ describe MetaInspector::Request do
       def cache.write(k, v) self[k] = v; end
       cache_options = { store: cache }
 
-      MetaInspector::Request.new(url('http://example.com'), faraday_http_cache: cache_options)
+      MetaInspector::Request.new(url('http://example.com'), faraday_options: { faraday_http_cache: cache_options })
 
       expect(cache_options).to eq({ store: cache })
     end

data/spec/url_filter_spec.rb

@@ -0,0 +1,206 @@
+require 'spec_helper'
+
+RSpec.describe 'outbound URL policy' do
+  let(:url) { 'https://public.example/page' }
+  let(:image_url) { 'https://public.example/image.png' }
+  let(:html) { '<img src="/image.png">' }
+  let(:png) { "\x89PNG\r\n\x1a\n".b + [13].pack('N') + 'IHDR' + [37, 29].pack('NN') + "\x08\x02\x00\x00\x00".b }
+
+  before do
+    stub_request(:get, url).to_return(body: html, headers: { 'Content-Type' => 'text/html' })
+    stub_request(:get, image_url).to_return(body: png, headers: { 'Content-Type' => 'image/png; charset=UTF-8' })
+  end
+
+  [false, nil].each do |result|
+    it "blocks the initial GET when the filter returns #{result.inspect}" do
+      expect { MetaInspector.new(url, url_filter: ->(_) { result }) }.to raise_error(MetaInspector::RequestError)
+      expect(a_request(:get, url)).not_to have_been_made
+    end
+  end
+
+  it 'passes a standard Ruby URI and permits truthy results' do
+    seen = []
+    page = MetaInspector.new(url, url_filter: ->(uri) { seen << uri; :allowed })
+    expect(page.images.with_size).to eq([[image_url, 37, 29]])
+    expect(seen.map(&:to_s)).to eq([url, image_url])
+    expect(seen).to all(be_a(URI::HTTP))
+  end
+
+  it 'does not permit mutations of the filter URI to change the request host' do
+    filter = ->(uri) { uri.host.replace('internal.example'); true }
+    expect(MetaInspector.new(url, url_filter: filter).url).to eq(url)
+    expect(a_request(:get, 'https://internal.example/page')).not_to have_been_made
+  end
+
+  [RuntimeError, Faraday::TimeoutError, Faraday::ConnectionFailed, URI::InvalidURIError, Timeout::Error].each do |error_class|
+    [:page, :image].each do |target|
+      it "preserves a filter's #{error_class} on a #{target}" do
+        error = error_class.new('policy failed')
+        filter = ->(_) { raise error }
+        expect do
+          if target == :page
+            MetaInspector.new(url, url_filter: filter)
+          else
+            MetaInspector.new(url, document: html, url_filter: filter).images.with_size
+          end
+        end.to raise_error { |actual| expect(actual).to equal(error) }
+      end
+    end
+  end
+
+  it 'checks HEAD and checks GET again after HEAD' do
+    stub_request(:head, url).to_return(headers: { 'Content-Type' => 'text/html' })
+    calls = 0
+    expect do
+      MetaInspector.new(url, head_first: true, url_filter: ->(_) { calls += 1; calls == 1 })
+    end.to raise_error(MetaInspector::RequestError)
+    expect(a_request(:head, url)).to have_been_made.once
+    expect(a_request(:get, url)).not_to have_been_made
+  end
+
+  it 'blocks HEAD before making a request' do
+    expect { MetaInspector.new(url, head_first: true, url_filter: ->(_) { false }) }.to raise_error(MetaInspector::RequestError)
+    expect(a_request(:head, url)).not_to have_been_made
+  end
+
+  [:head, :get].each do |method|
+    it "checks every #{method} redirect after the existing redirects-only callback" do
+      urls = [url, 'https://public.example/next', 'https://internal.example/private']
+      urls.each_cons(2) { |from, to| stub_request(method, from).to_return(status: 302, headers: { 'Location' => to }) }
+      seen = []
+      redirects = []
+      callback = ->(previous, following) { redirects << [previous.url.to_s, following.url.to_s] }
+      filter = ->(uri) { seen << uri.to_s; uri.host == 'public.example' }
+      expect do
+        MetaInspector.new(url, head_first: method == :head, url_filter: filter,
+                          faraday_options: { redirect: { callback: callback }.freeze }.freeze)
+      end.to raise_error(MetaInspector::RequestError)
+      expect(seen).to eq(urls)
+      expect(redirects).to eq(urls.each_cons(2).to_a)
+      expect(a_request(method, urls.last)).not_to have_been_made
+    end
+  end
+
+  it 'validates redirect destinations changed by the existing callback' do
+    stub_request(:get, url).to_return(status: 302, headers: { 'Location' => '/next' })
+    callback = ->(_, following) { following.url.host = 'internal.example' }
+    expect do
+      MetaInspector.new(url, url_filter: ->(uri) { uri.host == 'public.example' }, faraday_options: { redirect: { callback: callback } })
+    end.to raise_error(MetaInspector::RequestError)
+    expect(a_request(:get, 'https://internal.example/next')).not_to have_been_made
+  end
+
+  it 'runs policy before a cached response can be reused' do
+    cache = Class.new(Hash) do
+      def read(key); self[key]; end
+      def write(key, value); self[key] = value; end
+    end.new
+    stub_request(:get, url).to_return(body: html, headers: { 'Cache-Control' => 'public, max-age=3600' })
+    permitted = true
+    options = { url_filter: ->(_) { permitted }, faraday_options: { faraday_http_cache: { store: cache } } }
+    MetaInspector.new(url, options)
+    expect(cache).not_to be_empty
+    permitted = false
+    expect { MetaInspector.new(url, options) }.to raise_error(MetaInspector::RequestError)
+    expect(a_request(:get, url)).to have_been_made.once
+  end
+
+  it 'blocks images from inline HTML before downloading' do
+    page = MetaInspector.new(url, document: html, url_filter: ->(_) { false })
+    expect { page.images.with_size }.to raise_error(MetaInspector::RequestError)
+    expect(a_request(:get, url)).not_to have_been_made
+    expect(a_request(:get, image_url)).not_to have_been_made
+  end
+
+  it 'checks every image redirect' do
+    stub_request(:get, image_url).to_return(status: 302, headers: { 'Location' => '/second.png' })
+    stub_request(:get, 'https://public.example/second.png').to_return(status: 302, headers: { 'Location' => 'http://internal.example/private' })
+    page = MetaInspector.new(url, document: html, url_filter: ->(uri) { uri.host == 'public.example' })
+    expect { page.images.with_size }.to raise_error(MetaInspector::RequestError)
+    expect(a_request(:get, 'http://internal.example/private')).not_to have_been_made
+  end
+
+  ['file:///etc/passwd', 'ftp://files.example/image', 'data:image/png;base64,AAAA', 'ftp://files.example/http://public.example/'].each do |other|
+    it "rejects non-HTTP network URL #{other}" do
+      expect { MetaInspector.new(other) }.to raise_error(MetaInspector::RequestError)
+      page = MetaInspector.new(url, document: %(<img src="#{other}">))
+      expect { page.images.with_size }.to raise_error(MetaInspector::RequestError)
+    end
+  end
+
+  it 'rejects non-HTTP redirect schemes' do
+    stub_request(:get, url).to_return(status: 302, headers: { 'Location' => 'file:///etc/passwd' })
+    expect { MetaInspector.new(url) }.to raise_error(MetaInspector::RequestError)
+  end
+
+  it 'uses the page headers and redirects-only callback for images' do
+    headers = { 'Authorization' => 'Bearer test', 'User-Agent' => 'Policy app' }.freeze
+    stub_request(:get, image_url).with(headers: headers).to_return(status: 302, headers: { 'Location' => '/final.png' })
+    stub_request(:get, 'https://public.example/final.png').with(headers: headers).to_return(body: png)
+    redirects = []
+    page = MetaInspector.new(url, document: html, headers: headers,
+      faraday_options: { redirect: { callback: ->(old, following) { redirects << [old.url.to_s, following.url.to_s] } } })
+    expect(page.images.with_size).to eq([[image_url, 37, 29]])
+    expect(redirects).to eq([[image_url, 'https://public.example/final.png']])
+  end
+
+  it 'does not follow image redirects when disabled' do
+    stub_request(:get, image_url).to_return(status: 302, headers: { 'Location' => '/final.png' })
+    page = MetaInspector.new(url, document: html, allow_redirections: false)
+    expect(page.images.with_size).to eq([[image_url, 0, 0]])
+    expect(a_request(:get, 'https://public.example/final.png')).not_to have_been_made
+  end
+
+  it 'uses binary IO rather than a URL for image parsing' do
+    expect(FastImage).to receive(:size).with(an_instance_of(StringIO)).and_call_original
+    expect(MetaInspector.new(url, document: html).images.with_size).to eq([[image_url, 37, 29]])
+  end
+
+  [:to_timeout, :connection_failure, :unparseable].each do |failure|
+    it "returns zero dimensions for image #{failure}" do
+      request = stub_request(:get, image_url)
+      case failure
+      when :to_timeout then request.to_timeout
+      when :connection_failure then request.to_raise(Faraday::ConnectionFailed)
+      when :unparseable then request.to_return(body: 'not an image')
+      end
+      expect(MetaInspector.new(url, document: html).images.with_size).to eq([[image_url, 0, 0]])
+      expect(request).to have_been_requested.once
+    end
+  end
+  it 'rejects a non-callable filter even for inline HTML' do
+    expect { MetaInspector.new(url, document: html, url_filter: false) }.to raise_error(ArgumentError, /url_filter/)
+  end
+
+  [
+    'https://other.example/image.png',
+    'http://public.example/image.png',
+    'https://public.example:8443/image.png'
+  ].each do |external|
+    it "does not send custom page headers to a different image origin #{external}" do
+      secret_headers = { 'Authorization' => 'Bearer secret', 'Cookie' => 'session=secret', 'X-Api-Key' => 'secret' }
+      stub_request(:get, external).with do |request|
+        expect(request.headers.keys).not_to include('Authorization', 'Cookie', 'X-Api-Key')
+        true
+      end.to_return(body: png)
+      page = MetaInspector.new(url, document: %(<img src="#{external}">),
+        headers: secret_headers.merge('User-Agent' => 'Policy app'),
+        faraday_options: { headers: { 'X-Faraday-Secret' => 'secret' } })
+      expect(page.images.with_size).to eq([[external, 37, 29]])
+      expect(a_request(:get, external).with { |r| r.headers.key?('X-Faraday-Secret') }).not_to have_been_made
+    end
+  end
+
+  it 'retains custom headers on the page origin and strips them on image redirects' do
+    secrets = { 'Authorization' => 'Bearer secret', 'Cookie' => 'session=secret', 'X-Api-Key' => 'secret' }
+    external = 'https://other.example/image.png'
+    stub_request(:get, image_url).with(headers: secrets).to_return(status: 302, headers: { 'Location' => external })
+    stub_request(:get, external).with do |request|
+      expect(request.headers.keys).not_to include('Authorization', 'Cookie', 'X-Api-Key')
+      true
+    end.to_return(body: png)
+    page = MetaInspector.new(url, document: html, headers: secrets)
+    expect(page.images.with_size).to eq([[image_url, 37, 29]])
+  end
+
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: metainspector
 version: !ruby/object:Gem::Version
-  version: 5.19.0
+  version: 5.20.0
 platform: ruby
 authors:
 - Jaime Iniesta
@@ -326,6 +326,8 @@ files:
 - lib/meta_inspector.rb
 - lib/meta_inspector/document.rb
 - lib/meta_inspector/errors.rb
+- lib/meta_inspector/http_session.rb
+- lib/meta_inspector/image_downloader.rb
 - lib/meta_inspector/parser.rb
 - lib/meta_inspector/parsers/base.rb
 - lib/meta_inspector/parsers/head_links.rb
@@ -335,6 +337,7 @@ files:
 - lib/meta_inspector/parsers/texts.rb
 - lib/meta_inspector/request.rb
 - lib/meta_inspector/url.rb
+- lib/meta_inspector/url_policy.rb
 - lib/meta_inspector/version.rb
 - lib/metainspector.rb
 - meta_inspector.gemspec
@@ -406,6 +409,7 @@ files:
 - spec/fixtures/wordpress_site.response
 - spec/fixtures/youtube.response
 - spec/fixtures/youtube_short_title.response
+- spec/image_download_spec.rb
 - spec/meta_inspector/head_first_spec.rb
 - spec/meta_inspector/head_links_spec.rb
 - spec/meta_inspector/images_spec.rb
@@ -417,6 +421,7 @@ files:
 - spec/parser_spec.rb
 - spec/request_spec.rb
 - spec/spec_helper.rb
+- spec/url_filter_spec.rb
 - spec/url_spec.rb
 homepage: https://github.com/jaimeiniesta/metainspector
 licenses: