metainspector 5.18.1 → 5.19.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c74590fe15ada0cba938c3a9c4a0d167fcb10bd6159615751652e6376dec7110
-  data.tar.gz: d92a0ba92d9d7c6b1fa3404cdebc001a49080cbeba419766f6d60a2b96e1b582
+  metadata.gz: 5bc16dfdcdd22661ba7bb196d4db8750a5fa4d48d054197982768fa7fac44835
+  data.tar.gz: ce356b1f0f2511069ff96329e775e3581282d68d395c4dada93a9275e5dd4aa6
 SHA512:
-  metadata.gz: 2f0d12a22b1288aae2ca671ca5aa0516dad4aa3590b32be9337a626f5741d2c07d8050c26f013fa7a6478f83733e372425b0ce0ad450ee7d8f2d8a64f4efdbd9
-  data.tar.gz: 4a3662b35d9e4b5485dc00cd8101d216a086e70c08b152a0dc1c89045f7f179031b8597ccf92b106792f604b340aa49907de8dc6ca9758c6755f32b55fd6a109
+  metadata.gz: 6a669057baeb8107c8ae6940f4323f60664982fa3c8c74ce4f029b434ebfe227436c743a6b6041acbcddfdc2ea7fb38704a6126d3204b18479a8ca81b2b63a0e
+  data.tar.gz: 123f1cbb41fd5492307d96e316b17c91fc4f17d523a12f9fd865be49a3de31d8b932ceae6365bc7e1f6f49fd9721f8626eb9a8b5bbcbf9d86b2e972aedaa1544

data/.gitignore

@@ -9,3 +9,4 @@ pkg/*
 .rubocop_todo.yml
 .rubocop.yml
 .tool-versions
+docs/

data/CHANGELOG.md

@@ -1,5 +1,15 @@
 # MetaInspector Changelog
 
+## [Changes in 5.19.0](https://github.com/jaimeiniesta/metainspector/compare/v5.18.1...v5.19.0)
+
+* Security: stop modifying the `faraday_options` hash passed to MetaInspector. When the same hash was reused, its `redirect` options were deleted after the first request, so a redirect `callback` used to block private hosts only ran once and later requests followed any redirect ([GHSA-mm2p-6q2v-7ph5](https://github.com/jaimeiniesta/metainspector/security/advisories/GHSA-mm2p-6q2v-7ph5)). Frozen option hashes no longer raise `FrozenError`.
+* Add `language` to expose the root HTML `lang` attribute, trimming surrounding whitespace and returning `nil` for missing or blank values. Include it in `to_hash` (#121).
+* Add opt-in `max_image_downloads` to limit image size lookup attempts while preserving dimensions already available in HTML (#242).
+* Add `parsely-title`, `parsely-author`, and `parsely-image-url` fallbacks to `best_title`, `best_author`, and `images.owner_suggested` / `images.best`, preserving existing preferred metadata sources (#214).
+* Resolve relative base hrefs against the full document URL, preserving nested paths and non-default ports (#240).
+* Skip invalid image URLs in `images.with_size`, preventing `images.best` from failing when FastImage receives a nil URL ([#521](https://github.com/jaimeiniesta/metainspector/issues/521), follow-up to [#359](https://github.com/jaimeiniesta/metainspector/issues/359)).
+* Add opt-in `head_first: true` to check content type with HEAD and reject non-HTML files before downloading them (#221).
+
 ## [Changes in 5.18.1](https://github.com/jaimeiniesta/metainspector/compare/v5.18.0...v5.18.1)
 
 * Require nokogiri 1.19.5, which fixes three low-severity security advisories.

data/README.md

@@ -84,6 +84,7 @@ page.feeds               # Get rss or atom links in meta data fields as array of
 ### Texts
 
 ```ruby
+page.language            # declared language from the root <html lang="..."> attribute
 page.title               # title of the page from the head section, as string
 page.best_title          # best title of the page, from a selection of candidates
 page.author              # author of the page from the meta author tag
@@ -98,6 +99,20 @@ page.h5                  # returns h5 text array
 page.h6                  # returns h6 text array
 ```
 
+`language` trims surrounding whitespace and preserves the declared value, including
+capitalization and regional tags such as `en-US`. It returns `nil` when the `lang`
+attribute is missing, empty, or whitespace-only. It does not infer language from
+the page content or other attributes. The value is also included in `to_hash`
+under the `"language"` key.
+
+`best_title` selects the first non-empty value from the standard `title` meta tag,
+`og:title`, the head `<title>`, a body `<title>`, the first `<h1>`, and finally
+`parsely-title`.
+
+`best_author` selects the first non-empty value from the standard `author` meta tag,
+the first link with `rel="author"`, the first `<address>`, `twitter:creator`, and
+finally `parsely-author`. Both methods trim and collapse whitespace in their results.
+
 ### Links
 
 ```ruby
@@ -114,10 +129,15 @@ page.links.external      # every external link found on the page as an absolute
 ```ruby
 page.images              # enumerable collection, with every img found on the page as an absolute URL
 page.images.with_size    # a sorted array (by descending area) of [image_url, width, height]
-page.images.best         # Most relevant image, if defined with the og:image or twitter:image metatags. Fallback to the first page.images array element
+page.images.best         # Most relevant image from og:image, twitter:image, or parsely-image-url; falls back to page.images.largest
 page.images.favicon      # absolute URL to the favicon
 ```
 
+`images.owner_suggested` tries `og:image`, `twitter:image`, and `parsely-image-url`
+in that order, skipping empty or whitespace-only values. Relative image URLs are
+resolved against the document's base URL. `images.best` uses this suggested image
+before falling back to the largest suitable image on the page.
+
 ### Meta tags
 
 When it comes to meta tags, you have several options:
@@ -385,6 +405,28 @@ page.content_type  # "image/png"
 page.description   # will return a garbled string
 ```
 
+### Check content type before downloading
+
+To avoid downloading large non-HTML files, enable `head_first`:
+
+```ruby
+page = MetaInspector.new('http://example.com/page', head_first: true)
+```
+
+This sends an HTTP HEAD request before GET. If a successful HEAD response declares
+a content type other than `text/html`, MetaInspector raises `MetaInspector::NonHtmlError`
+without downloading the body. Content-type parameters such as `charset` are ignored.
+With `allow_non_html_content: true`, the download proceeds regardless of content type.
+
+If HEAD omits the content type or returns an unsuccessful status (including 405 or
+501 for servers that do not support HEAD), MetaInspector continues with GET and its
+usual content-type check. HEAD timeouts and connection errors use the same exception
+handling as GET. The final page data, response headers and status come from GET.
+
+This option is off by default because it adds a request and not every server
+supports HEAD. Both requests use the configured headers, timeouts, retries and
+redirect settings. Passing a `document` directly makes no HTTP requests.
+
 ### URL Normalization
 
 By default, URLs are normalized using the Addressable gem. For example:
@@ -413,6 +455,19 @@ If you want to disable this, you can specify it like this:
 page = MetaInspector.new('http://example.com', download_images: false)
 ```
 
+To limit how many images are downloaded to determine their dimensions, use `max_image_downloads`:
+
+```ruby
+page = MetaInspector.new('http://example.com', max_image_downloads: 10)
+page.images.with_size
+```
+
+The default is `nil` (unlimited). The option accepts a non-negative integer or `nil`; `0` disables image downloads. `download_images: false` always disables downloads, regardless of the limit.
+
+The limit counts image size lookup attempts in HTML order, after excluding unparseable URLs and duplicates. Images with both HTML dimensions already present do not consume the limit, and those dimensions are preserved even after the limit is reached. Failed lookups count toward the limit. Images with incomplete dimensions that are not downloaded remain in `images.with_size` with dimensions `[0, 0]`, matching the behavior when downloads are disabled.
+
+The limit leaves `images.to_a` unchanged, but can affect `images.largest` and the largest-image fallback in `images.best`, since images with unknown dimensions cannot be ranked by size. Size lookup results are cached per document, so repeated calls reuse the same results. This limits size lookup attempts, not elapsed time or HTTP requests made while following image redirects.
+
 ### Caching responses
 
 MetaInspector can be configured to use [Faraday::HttpCache](https://github.com/plataformatec/faraday-http-cache) to cache page responses. For that you should pass the `faraday_http_cache` option with at least the `:store` key, for example:

data/lib/meta_inspector/document.rb

@@ -9,12 +9,20 @@ module MetaInspector
     # * read_timeout: defaults to 20 seconds
     # * retries: defaults to 3 times
     # * allow_redirections: when true, follow HTTP redirects. Defaults to true
+    # * head_first: check content type with HEAD before GET. Defaults to false
     # * document: the html of the url as a string
     # * headers: object containing custom headers for the request
     # * normalize_url: true by default
+    # * download_images: fetch missing image dimensions. Defaults to true
+    # * max_image_downloads: maximum image size lookups, or nil for unlimited
     # * faraday_options: an optional hash of options to pass to Faraday on the request
     def initialize(initial_url, options = {})
       options             = defaults.merge(options)
+      @max_image_downloads = options[:max_image_downloads]
+      unless @max_image_downloads.nil? || (@max_image_downloads.is_a?(Integer) && @max_image_downloads >= 0)
+        raise ArgumentError, 'max_image_downloads must be a non-negative Integer or nil'
+      end
+
       @connection_timeout = options[:connection_timeout]
       @read_timeout       = options[:read_timeout]
       @retries            = options[:retries]
@@ -31,6 +39,8 @@ module MetaInspector
       @faraday_http_cache = options[:faraday_http_cache]
       @url                = MetaInspector::URL.new(initial_url, normalize:          @normalize_url)
       @request            = MetaInspector::Request.new(@url,    allow_redirections: @allow_redirections,
+                                                                head_first:         options[:head_first],
+                                                                allow_non_html_content: @allow_non_html_content,
                                                                 connection_timeout: @connection_timeout,
                                                                 read_timeout:       @read_timeout,
                                                                 retries:            @retries,
@@ -38,7 +48,8 @@ module MetaInspector
                                                                 headers:            @headers,
                                                                 faraday_options:    @faraday_options,
                                                                 faraday_http_cache: @faraday_http_cache) unless @document
-      @parser             = MetaInspector::Parser.new(self,     download_images:    @download_images)
+      @parser             = MetaInspector::Parser.new(self,     download_images:    @download_images,
+                                                                max_image_downloads: @max_image_downloads)
     end
 
     extend Forwardable
@@ -47,7 +58,7 @@ module MetaInspector
 
     delegate [:content_type, :response]               => :@request
 
-    delegate [:parsed, :title, :best_title, :author, :best_author,
+    delegate [:parsed, :language, :title, :best_title, :author, :best_author,
               :h1, :h2, :h3, :h4, :h5, :h6, :description, :best_description, :links,
               :images, :feeds, :feed, :charset, :meta_tags,
               :meta_tag, :meta, :favicon,
@@ -60,6 +71,7 @@ module MetaInspector
         'scheme'           => scheme,
         'host'             => host,
         'root_url'         => root_url,
+        'language'         => language,
         'title'            => title,
         'best_title'       => best_title,
         'author'           => author,
@@ -101,8 +113,10 @@ module MetaInspector
                                   },
         :allow_redirections     => true,
         :allow_non_html_content => false,
+        :head_first             => false,
         :normalize_url          => true,
-        :download_images        => true }
+        :download_images        => true,
+        :max_image_downloads    => nil }
     end
 
     def default_user_agent

data/lib/meta_inspector/parser.rb

@@ -14,7 +14,8 @@ module MetaInspector
       @meta_tag_parser = MetaInspector::Parsers::MetaTagsParser.new(self)
       @links_parser    = MetaInspector::Parsers::LinksParser.new(self)
       @download_images = options[:download_images]
-      @images_parser   = MetaInspector::Parsers::ImagesParser.new(self, download_images: @download_images)
+      @images_parser   = MetaInspector::Parsers::ImagesParser.new(self, download_images: @download_images,
+                                                                      max_image_downloads: options[:max_image_downloads])
       @texts_parser    = MetaInspector::Parsers::TextsParser.new(self)
 
       parsed           # parse early so we can fail early
@@ -26,7 +27,7 @@ module MetaInspector
     delegate [:head_links, :stylesheets, :canonicals, :feeds, :feed]                       => :@head_links_parser
     delegate [:links, :base_url]                                                           => :@links_parser
     delegate :images                                                                       => :@images_parser
-    delegate [:title, :best_title, :author, :best_author, :description, :best_description, 
+    delegate [:language, :title, :best_title, :author, :best_author, :description, :best_description,
               :h1, :h2, :h3, :h4, :h5, :h6]                                                => :@texts_parser
 
     # Returns the whole parsed document

data/lib/meta_inspector/parsers/images.rb

@@ -10,6 +10,7 @@ module MetaInspector
 
       def initialize(main_parser, options = {})
         @download_images = options[:download_images]
+        @max_image_downloads = options[:max_image_downloads]
         super(main_parser)
       end
 
@@ -17,7 +18,7 @@ module MetaInspector
         self
       end
 
-      # Returns either the Facebook Open Graph image, twitter suggested image or
+      # Returns the Facebook Open Graph image, Twitter image, Parsely image or
       # the largest image in the image collection
       def best
         owner_suggested || largest
@@ -26,9 +27,10 @@ module MetaInspector
       # Returns the parsed image from Facebook's open graph property tags
       # Most major websites now define this property and is usually relevant
       # See doc at http://developers.facebook.com/docs/opengraph/
-      # If none found, tries with Twitter image
+      # If none found, tries with Twitter image, then Parsely image
       def owner_suggested
-        suggested_img = content_of(meta['og:image']) || content_of(meta['twitter:image'])
+        suggested_img = content_of(meta['og:image']) || content_of(meta['twitter:image']) ||
+                        content_of(meta['parsely-image-url'])
         URL.absolutify(suggested_img, base_url, normalize: false) if suggested_img
       end
 
@@ -39,17 +41,19 @@ module MetaInspector
           imgs_with_size = img_nodes.map do |img_node|
             [URL.absolutify(img_node['src'], base_url, normalize: false), img_node['width'], img_node['height']]
           end
+          imgs_with_size.reject! { |url, width, height| url.nil? || url.empty? }
           imgs_with_size.uniq! { |url, width, height| url }
-          if @download_images
-            imgs_with_size.map! do |url, width, height|
-              width, height = FastImage.size(url) if width.nil? || height.nil?
-              [url, width.to_i, height.to_i]
-            end
-          else
-            imgs_with_size.map! do |url, width, height|
-              width, height = [0, 0] if width.nil? || height.nil?
-              [url, width.to_i, height.to_i]
+          download_attempts = 0
+          imgs_with_size.map! do |url, width, height|
+            if width.nil? || height.nil?
+              if @download_images && (@max_image_downloads.nil? || download_attempts < @max_image_downloads)
+                download_attempts += 1
+                width, height = FastImage.size(url)
+              else
+                width, height = [0, 0]
+              end
             end
+            [url, width.to_i, height.to_i]
           end
           imgs_with_size.sort_by { |url, width, height| -(width.to_i * height.to_i) }
         end
@@ -93,7 +97,7 @@ module MetaInspector
       end
 
       def content_of(content)
-        return nil if content.nil? || content.empty?
+        return nil if content.nil? || content.strip.empty?
         content
       end
     end

data/lib/meta_inspector/parsers/links.rb

@@ -46,8 +46,9 @@ module MetaInspector
       # Returns the base url to absolutify relative links.
       # This can be the one set on a <base> tag,
       # or the url of the document if no <base> tag was found.
+      # Relative base hrefs are resolved against the document URL.
       def base_url
-        current_base_href = base_href.to_s.strip.empty? ? nil : URL.absolutify(base_href, URL.new(url).root_url)
+        current_base_href = base_href.to_s.strip.empty? ? nil : URL.absolutify(base_href, url)
         current_base_href || url
       end
 

data/lib/meta_inspector/parsers/texts.rb

@@ -3,6 +3,12 @@ module MetaInspector
     class TextsParser < Base
       delegate [:parsed, :meta] => :@main_parser
 
+      # Returns the declared HTML language, or nil when missing or blank
+      def language
+        value = parsed.at_css('html')&.[]('lang')&.strip
+        value unless value.to_s.empty?
+      end
+
       # Returns the parsed document title, from the content of the <title> tag
       # within the <head> section.
       def title
@@ -42,12 +48,13 @@ module MetaInspector
         @author ||= meta['author']
       end
 
-      # An author getter that returns the first non-nil description
+      # An author getter that returns the first non-empty author
       # from the following candidates:
-      # - the standard meta description
+      # - the standard meta author
       # - a link with the relational attribute "author"
       # - address tag which may contain the author
       # - the twitter:creator meta tag for the username
+      # - the parsely-author meta tag
       def best_author
         @best_author ||= find_best_author
       end
@@ -80,7 +87,8 @@ module MetaInspector
             meta['og:title'],
             parsed.css('head title'),
             parsed.css('body title'),
-            parsed.css('h1').first
+            parsed.css('h1').first,
+            meta['parsely-title']
         ]
         candidates.flatten!
         candidates.compact!
@@ -94,7 +102,8 @@ module MetaInspector
           meta['author'],
           parsed.css('a[rel="author"]').first,
           parsed.css('address').first,
-          meta['twitter:creator']
+          meta['twitter:creator'],
+          meta['parsely-author']
         ]
         candidates.flatten!
         candidates.compact!

data/lib/meta_inspector/request.rb

@@ -18,6 +18,8 @@ module MetaInspector
       fail MetaInspector::RequestError.new('URL must be HTTP') unless @url.url =~ /http[s]?:\/\//i
 
       @allow_redirections = options[:allow_redirections]
+      @head_first         = options[:head_first]
+      @allow_non_html_content = options[:allow_non_html_content]
       @connection_timeout = options[:connection_timeout]
       @read_timeout       = options[:read_timeout]
       @retries            = options[:retries]
@@ -42,8 +44,7 @@ module MetaInspector
     end
 
     def content_type
-      return nil if response.headers['content-type'].nil?
-      response.headers['content-type'].split(';')[0] if response
+      response_content_type(response)
     end
 
     def response
@@ -58,10 +59,12 @@ module MetaInspector
 
     def fetch
       Timeout::timeout(fatal_timeout) do
-        @faraday_options.merge!(:url => url)
-        follow_redirects_options = @faraday_options.delete(:redirect) || {}
+        # Work on copies: the caller may reuse or freeze the hashes it passed in, and
+        # losing :redirect would drop a callback that blocks redirects to private hosts
+        faraday_options          = @faraday_options.merge(:url => url)
+        follow_redirects_options = (faraday_options.delete(:redirect) || {}).dup
 
-        session = Faraday.new(@faraday_options) do |faraday|
+        session = Faraday.new(faraday_options) do |faraday|
           faraday.request :retry, max: @retries
 
           faraday.request :gzip
@@ -73,8 +76,9 @@ module MetaInspector
           end
 
           if @faraday_http_cache.is_a?(Hash)
-            @faraday_http_cache[:serializer] ||= Marshal
-            faraday.use Faraday::HttpCache, **@faraday_http_cache
+            cache_options = @faraday_http_cache.dup
+            cache_options[:serializer] ||= Marshal
+            faraday.use Faraday::HttpCache, **cache_options
           end
 
           faraday.headers.merge!(@headers || {})
@@ -82,10 +86,8 @@ module MetaInspector
           faraday.adapter :net_http
         end
 
-        response = session.get do |req|
-          req.options.timeout      = @connection_timeout
-          req.options.open_timeout = @read_timeout
-        end
+        check_head_content_type(session) if @head_first
+        response = perform_request(session, :get)
 
         if @allow_redirections
           @url.url = response.env.url.to_s
@@ -97,6 +99,27 @@ module MetaInspector
       raise MetaInspector::TimeoutError.new(e)
     end
 
+    def check_head_content_type(session)
+      head_response = perform_request(session, :head)
+      return unless head_response.success?
+
+      type = response_content_type(head_response)
+      if !@allow_non_html_content && !type.nil? && type != 'text/html'
+        fail MetaInspector::NonHtmlError.new "The url provided contains #{type} content instead of text/html content"
+      end
+    end
+
+    def response_content_type(response)
+      response.headers['content-type']&.split(';')&.first
+    end
+
+    def perform_request(session, method)
+      session.public_send(method) do |req|
+        req.options.timeout      = @connection_timeout
+        req.options.open_timeout = @read_timeout
+      end
+    end
+
     # Timeouts when connecting / reading a request are handled by Faraday, but in the
     # case of URLs that respond with streaming, Faraday will never return. In that case,
     # we'll resort to our own timeout

data/lib/meta_inspector/version.rb

@@ -1,3 +1,3 @@
 module MetaInspector
-  VERSION = '5.18.1'
+  VERSION = '5.19.0'
 end

data/spec/document_spec.rb

@@ -1,6 +1,18 @@
 require 'spec_helper'
 
 describe MetaInspector::Document do
+  describe 'max_image_downloads option' do
+    [-1, 1.5, '1', true, false].each do |limit|
+      it "should reject #{limit.inspect} before fetching the document" do
+        expect do
+          MetaInspector::Document.new('http://example.com/', max_image_downloads: limit)
+        end.to raise_error(ArgumentError, 'max_image_downloads must be a non-negative Integer or nil')
+
+        expect(WebMock).not_to have_requested(:get, 'http://example.com/')
+      end
+    end
+  end
+
   describe 'passing the contents of the document as html' do
     let(:doc) { MetaInspector::Document.new('http://cnn.com/', :document => "<html><head><title>Hello From Passed Html</title><a href='/hello'>Hello link</a></head><body></body></html>") }
 
@@ -17,6 +29,13 @@ describe MetaInspector::Document do
     expect(MetaInspector::Document.new('http://pagerankalert.com').to_s.class).to eq(String)
   end
 
+  it 'should include a nil language in the hash when the document has no declared language' do
+    stub_request(:get, 'http://example.com/').to_return(body: '<html><body></body></html>', headers: { 'Content-Type' => 'text/html' })
+    doc = MetaInspector::Document.new('http://example.com/')
+
+    expect(doc.to_hash).to include('language' => nil)
+  end
+
   it "should return a Hash with all the values set" do
     doc = MetaInspector::Document.new('http://pagerankalert.com')
     expect(doc.to_hash).to eq({
@@ -24,6 +43,7 @@ describe MetaInspector::Document do
                             "scheme"          => "http",
                             "host"            => "pagerankalert.com",
                             "root_url"        => "http://pagerankalert.com/",
+                            "language"        => "en",
                             "title"           => "PageRankAlert.com :: Track your PageRank changes & receive alerts",
                             "best_title"      => "PageRankAlert.com :: Track your PageRank changes & receive alerts",
                             "author"          => nil,

data/spec/meta_inspector/head_first_spec.rb

@@ -0,0 +1,195 @@
+require 'spec_helper'
+
+describe MetaInspector, 'head_first option' do
+  let(:page_url) { 'http://example.com/head-first' }
+  let(:html) { '<html><head><title>Downloaded page</title></head></html>' }
+  let!(:get_request) do
+    stub_request(:get, page_url).to_return(body: html, headers: { 'Content-Type' => 'text/html' })
+  end
+
+  it 'does not send HEAD by default' do
+    expect(MetaInspector.new(page_url).title).to eq('Downloaded page')
+
+    expect(a_request(:head, page_url)).not_to have_been_made
+    expect(get_request).to have_been_requested.once
+  end
+
+  it 'does not send HEAD when explicitly disabled' do
+    expect(MetaInspector.new(page_url, head_first: false).title).to eq('Downloaded page')
+
+    expect(a_request(:head, page_url)).not_to have_been_made
+  end
+
+  it 'sends HEAD before GET for HTML and exposes the GET response' do
+    requests = []
+    head_request = stub_request(:head, page_url).to_return do
+      requests << :head
+      { headers: { 'Content-Type' => 'text/html; charset=UTF-8', 'X-Response' => 'head' } }
+    end
+    stub_request(:get, page_url).to_return do
+      requests << :get
+      { body: html, headers: { 'Content-Type' => 'text/html', 'X-Response' => 'get' } }
+    end
+
+    page = MetaInspector.new(page_url, head_first: true)
+
+    expect(page.title).to eq('Downloaded page')
+    expect(page.content_type).to eq('text/html')
+    expect(page.response.headers['x-response']).to eq('get')
+    expect(page.to_s).to eq(html)
+    expect(head_request).to have_been_requested.once
+    expect(requests).to eq([:head, :get])
+  end
+
+  it 'rejects non-HTML content without sending GET' do
+    head_request = stub_request(:head, page_url)
+      .to_return(headers: { 'Content-Type' => 'application/pdf; version=1.7' })
+
+    expect do
+      MetaInspector.new(page_url, head_first: true)
+    end.to raise_error(MetaInspector::NonHtmlError,
+      'The url provided contains application/pdf content instead of text/html content')
+
+    expect(head_request).to have_been_requested.once
+    expect(get_request).not_to have_been_requested
+  end
+
+  it 'downloads non-HTML content when explicitly allowed' do
+    head_request = stub_request(:head, page_url).to_return(headers: { 'Content-Type' => 'text/plain' })
+    stub_request(:get, page_url).to_return(body: 'Plain text', headers: { 'Content-Type' => 'text/plain' })
+
+    page = MetaInspector.new(page_url, head_first: true, allow_non_html_content: true)
+
+    expect(page.to_s).to eq('Plain text')
+    expect(page.content_type).to eq('text/plain')
+    expect(head_request).to have_been_requested.once
+    expect(get_request).to have_been_requested.once
+  end
+
+  it 'continues to GET when HEAD has no content type' do
+    head_request = stub_request(:head, page_url).to_return(status: 200)
+
+    expect(MetaInspector.new(page_url, head_first: true).title).to eq('Downloaded page')
+
+    expect(head_request).to have_been_requested.once
+    expect(get_request).to have_been_requested.once
+  end
+
+  [403, 404, 405, 501].each do |status|
+    it "continues to GET when HEAD returns #{status}" do
+      head_request = stub_request(:head, page_url)
+        .to_return(status: status, headers: { 'Content-Type' => 'text/plain' })
+
+      expect(MetaInspector.new(page_url, head_first: true).title).to eq('Downloaded page')
+
+      expect(head_request).to have_been_requested.once
+      expect(get_request).to have_been_requested.once
+    end
+  end
+
+  it 'still rejects a non-HTML GET response when HEAD reports HTML' do
+    head_request = stub_request(:head, page_url).to_return(headers: { 'Content-Type' => 'text/html' })
+    stub_request(:get, page_url).to_return(headers: { 'Content-Type' => 'application/pdf' })
+
+    expect do
+      MetaInspector.new(page_url, head_first: true)
+    end.to raise_error(MetaInspector::NonHtmlError)
+
+    expect(head_request).to have_been_requested.once
+    expect(get_request).to have_been_requested.once
+  end
+
+  it 'does not send requests for a supplied document' do
+    expect(MetaInspector.new(page_url, head_first: true, document: html).title).to eq('Downloaded page')
+
+    expect(a_request(:head, page_url)).not_to have_been_made
+    expect(get_request).not_to have_been_requested
+  end
+
+  it 'sends custom headers on both requests' do
+    headers = { 'User-Agent' => 'Custom agent', 'Authorization' => 'Bearer test-token' }
+    head_request = stub_request(:head, page_url).with(headers: headers)
+      .to_return(headers: { 'Content-Type' => 'text/html' })
+    get_with_headers = stub_request(:get, page_url).with(headers: headers).to_return(body: html)
+
+    expect(MetaInspector.new(page_url, head_first: true, headers: headers).title).to eq('Downloaded page')
+
+    expect(head_request).to have_been_requested.once
+    expect(get_with_headers).to have_been_requested.once
+  end
+
+  it 'follows HEAD redirects to reject non-HTML content before GET' do
+    target = 'http://example.com/large.pdf'
+    stub_request(:head, page_url).to_return(status: 302, headers: { 'Location' => target })
+    target_request = stub_request(:head, target).to_return(headers: { 'Content-Type' => 'application/pdf' })
+
+    expect do
+      MetaInspector.new(page_url, head_first: true)
+    end.to raise_error(MetaInspector::NonHtmlError)
+
+    expect(target_request).to have_been_requested.once
+    expect(a_request(:get, /example.com/)).not_to have_been_made
+  end
+
+  it 'follows GET redirects independently of HEAD and exposes the final GET URL' do
+    head_target = 'http://example.com/head-target'
+    get_target = 'http://example.com/get-target'
+    head_request = stub_request(:head, page_url).to_return(status: 302, headers: { 'Location' => head_target })
+    stub_request(:head, head_target).to_return(headers: { 'Content-Type' => 'text/html' })
+    stub_request(:get, page_url).to_return(status: 302, headers: { 'Location' => get_target })
+    final_request = stub_request(:get, get_target).to_return(body: html, headers: { 'Content-Type' => 'text/html' })
+
+    page = MetaInspector.new(page_url, head_first: true)
+
+    expect(page.title).to eq('Downloaded page')
+    expect(page.url).to eq(get_target)
+    expect(head_request).to have_been_requested.once
+    expect(final_request).to have_been_requested.once
+    expect(a_request(:get, head_target)).not_to have_been_made
+  end
+
+  it 'does not follow HEAD redirects when redirections are disabled' do
+    head_request = stub_request(:head, page_url)
+      .to_return(status: 302, headers: { 'Location' => 'http://example.com/target', 'Content-Type' => 'text/plain' })
+
+    page = MetaInspector.new(page_url, head_first: true, allow_redirections: false)
+
+    expect(page.title).to eq('Downloaded page')
+    expect(page.url).to eq(page_url)
+    expect(head_request).to have_been_requested.once
+    expect(a_request(:head, 'http://example.com/target')).not_to have_been_made
+  end
+
+  it 'keeps cookies from HEAD for the GET request' do
+    head_request = stub_request(:head, page_url)
+      .to_return(headers: { 'Content-Type' => 'text/html', 'Set-Cookie' => 'session=checked; path=/' })
+    get_with_cookie = stub_request(:get, page_url).with(headers: { 'Cookie' => 'session=checked' }).to_return(body: html)
+
+    expect(MetaInspector.new(page_url, head_first: true).title).to eq('Downloaded page')
+
+    expect(head_request).to have_been_requested.once
+    expect(get_with_cookie).to have_been_requested.once
+  end
+
+  it 'wraps HEAD timeouts without sending GET' do
+    head_request = stub_request(:head, page_url).to_raise(Net::ReadTimeout)
+
+    expect do
+      MetaInspector.new(page_url, head_first: true, retries: 0)
+    end.to raise_error(MetaInspector::TimeoutError)
+
+    expect(head_request).to have_been_requested.once
+    expect(get_request).not_to have_been_requested
+  end
+
+  it 'wraps HEAD connection failures without sending GET' do
+    head_request = stub_request(:head, page_url).to_raise(Faraday::ConnectionFailed.new('connection failed'))
+
+    expect do
+      MetaInspector.new(page_url, head_first: true, retries: 0)
+    end.to raise_error(MetaInspector::RequestError)
+
+    expect(head_request).to have_been_requested.once
+    expect(get_request).not_to have_been_requested
+  end
+end

data/spec/meta_inspector/images_spec.rb

@@ -70,6 +70,31 @@ describe MetaInspector do
   end
 
   describe "images.best" do
+    it 'should prefer the Parsely image over the largest image' do
+      html = '<meta name="parsely-image-url" content="/parsely.jpg"><img src="/largest.jpg" width="100" height="100">'
+      page = MetaInspector.new('http://example.com/', document: html)
+
+      expect(page.images.best).to eq('http://example.com/parsely.jpg')
+    end
+
+    ['', '  &#10;  '].each do |value|
+      it "should fall back to the largest image when the Parsely image is empty (#{value.inspect})" do
+        html = %(<meta name="parsely-image-url" content="#{value}"><img src="/largest.jpg" width="100" height="100">)
+        page = MetaInspector.new('http://example.com/', document: html)
+
+        expect(page.images.best).to eq('http://example.com/largest.jpg')
+      end
+    end
+
+    it "should find a valid image when another image URL cannot be parsed" do
+      html = '<html><body><img src="http://exa mple.com/x.png"><img src="/100x100"></body></html>'
+      page = MetaInspector.new('http://example.com/', document: html)
+
+      allow(FastImage).to receive(:size).and_call_original
+      expect(FastImage).not_to receive(:size).with(nil)
+      expect(page.images.best).to eq("http://example.com/100x100")
+    end
+
     it "should find the og image" do
       page = MetaInspector.new('http://www.theonion.com/articles/apple-claims-new-iphone-only-visible-to-most-loyal,2772/')
 
@@ -103,6 +128,44 @@ describe MetaInspector do
   end
 
   describe "images.owner_suggested" do
+    it 'should find the Parsely image when no existing metadata is present' do
+      html = '<meta name="parsely-image-url" content="https://images.example.com/parsely.jpg?b=2&amp;a=1">'
+      page = MetaInspector.new('http://example.com/', document: html)
+
+      expect(page.images.owner_suggested).to eq('https://images.example.com/parsely.jpg?b=2&a=1')
+    end
+
+    it 'should resolve a relative Parsely image against the document base URL' do
+      html = '<base href="/assets/"><meta name="parsely-image-url" content="parsely.jpg">'
+      page = MetaInspector.new('http://example.com/articles/page', document: html)
+
+      expect(page.images.owner_suggested).to eq('http://example.com/assets/parsely.jpg')
+    end
+
+    it 'should use the Parsely image when existing metadata is empty' do
+      html = '<meta property="og:image" content=""><meta name="twitter:image" content=""><meta name="parsely-image-url" content="/parsely.jpg">'
+      page = MetaInspector.new('http://example.com/', document: html)
+
+      expect(page.images.owner_suggested).to eq('http://example.com/parsely.jpg')
+    end
+
+    ['og:image', 'twitter:image'].each do |candidate|
+      it "should prefer #{candidate} over the Parsely image" do
+        html = %(<meta name="parsely-image-url" content="/parsely.jpg"><meta property="#{candidate}" content="/existing.jpg">)
+        page = MetaInspector.new('http://example.com/', document: html)
+
+        expect(page.images.owner_suggested).to eq('http://example.com/existing.jpg')
+      end
+    end
+
+    ['', '  &#10;  '].each do |value|
+      it "should ignore an empty Parsely image (#{value.inspect})" do
+        page = MetaInspector.new('http://example.com/', document: %(<meta name="parsely-image-url" content="#{value}">))
+
+        expect(page.images.owner_suggested).to be_nil
+      end
+    end
+
     it "should find the og image" do
       page = MetaInspector.new('http://www.theonion.com/articles/apple-claims-new-iphone-only-visible-to-most-loyal,2772/')
 
@@ -135,6 +198,176 @@ describe MetaInspector do
   end
 
   describe "images.with_size" do
+    describe "max_image_downloads" do
+      let(:html) do
+        '<img src="/10x10"><img src="/100x100"><img src="/remaining.jpg">'
+      end
+
+      before do
+        stub_request(:get, 'http://example.com/remaining.jpg').to_return(status: 404)
+      end
+
+      it "should download only the first images needing dimensions in document order" do
+        page = MetaInspector.new('http://example.com/', document: html, max_image_downloads: 2)
+
+        expect(page.images.with_size).to eq([
+          ['http://example.com/100x100', 100, 100],
+          ['http://example.com/10x10', 10, 10],
+          ['http://example.com/remaining.jpg', 0, 0]
+        ])
+        expect(WebMock).to have_requested(:get, 'http://example.com/10x10').once
+        expect(WebMock).to have_requested(:get, 'http://example.com/100x100').once
+        expect(WebMock).not_to have_requested(:get, 'http://example.com/remaining.jpg')
+      end
+
+      it "should preserve HTML dimensions without consuming the download budget" do
+        html = '<img src="/known.jpg" width="200" height="200"><img src="/10x10"><img src="/100x100"><img src="/later.jpg" width="300" height="300">'
+        page = MetaInspector.new('http://example.com/', document: html, max_image_downloads: 1)
+
+        expect(page.images.with_size).to eq([
+          ['http://example.com/later.jpg', 300, 300],
+          ['http://example.com/known.jpg', 200, 200],
+          ['http://example.com/10x10', 10, 10],
+          ['http://example.com/100x100', 0, 0]
+        ])
+        expect(WebMock).to have_requested(:get, 'http://example.com/10x10').once
+        expect(WebMock).not_to have_requested(:get, 'http://example.com/100x100')
+      end
+
+      it "should exclude invalid and duplicate URLs before consuming the budget" do
+        html = '<img src="http://exa mple.com/broken.jpg"><img src="/10x10"><img src="http://example.com/10x10"><img src="/100x100"><img src="/remaining.jpg">'
+        page = MetaInspector.new('http://example.com/', document: html, max_image_downloads: 2)
+
+        expect(page.images.with_size).to eq([
+          ['http://example.com/100x100', 100, 100],
+          ['http://example.com/10x10', 10, 10],
+          ['http://example.com/remaining.jpg', 0, 0]
+        ])
+        expect(WebMock).to have_requested(:get, 'http://example.com/10x10').once
+        expect(WebMock).to have_requested(:get, 'http://example.com/100x100').once
+        expect(WebMock).not_to have_requested(:get, 'http://example.com/remaining.jpg')
+      end
+
+      it "should treat an image missing one HTML dimension as needing a download" do
+        html = '<img src="/10x10" width="500"><img src="/100x100" height="500">'
+        page = MetaInspector.new('http://example.com/', document: html, max_image_downloads: 1)
+
+        expect(page.images.with_size).to eq([
+          ['http://example.com/10x10', 10, 10],
+          ['http://example.com/100x100', 0, 0]
+        ])
+        expect(WebMock).to have_requested(:get, 'http://example.com/10x10').once
+        expect(WebMock).not_to have_requested(:get, 'http://example.com/100x100')
+      end
+
+      it "should disable downloads with a zero limit while preserving HTML dimensions" do
+        html = '<img src="/10x10"><img src="/known.jpg" width="200" height="100">'
+        page = MetaInspector.new('http://example.com/', document: html, max_image_downloads: 0)
+
+        expect(page.images.with_size).to eq([
+          ['http://example.com/known.jpg', 200, 100],
+          ['http://example.com/10x10', 0, 0]
+        ])
+        expect(WebMock).not_to have_requested(:get, 'http://example.com/10x10')
+      end
+
+      it "should respect download_images: false even with a positive limit" do
+        page = MetaInspector.new('http://example.com/', document: html, download_images: false, max_image_downloads: 2)
+
+        expect(page.images.with_size).to eq([
+          ['http://example.com/10x10', 0, 0],
+          ['http://example.com/100x100', 0, 0],
+          ['http://example.com/remaining.jpg', 0, 0]
+        ])
+        expect(WebMock).not_to have_requested(:get, %r{http://example.com/})
+      end
+
+      [nil, 10].each do |limit|
+        it "should download every candidate when the limit is #{limit.inspect}" do
+          page = MetaInspector.new('http://example.com/', document: html, max_image_downloads: limit)
+
+          expect(page.images.with_size).to eq([
+            ['http://example.com/100x100', 100, 100],
+            ['http://example.com/10x10', 10, 10],
+            ['http://example.com/remaining.jpg', 0, 0]
+          ])
+          expect(WebMock).to have_requested(:get, 'http://example.com/10x10').once
+          expect(WebMock).to have_requested(:get, 'http://example.com/100x100').once
+          expect(WebMock).to have_requested(:get, 'http://example.com/remaining.jpg').once
+        end
+      end
+
+      [:not_found, :invalid_image, :timeout].each do |failure|
+        it "should count a #{failure} image attempt toward the limit" do
+          request = stub_request(:get, 'http://example.com/broken.jpg')
+          case failure
+          when :not_found then request.to_return(status: 404)
+          when :invalid_image then request.to_return(body: 'not an image')
+          when :timeout then request.to_timeout
+          end
+          html = '<img src="/broken.jpg"><img src="/100x100">'
+          page = MetaInspector.new('http://example.com/', document: html, max_image_downloads: 1)
+
+          expect(page.images.with_size).to eq([
+            ['http://example.com/broken.jpg', 0, 0],
+            ['http://example.com/100x100', 0, 0]
+          ])
+          expect(request).to have_been_requested.once
+          expect(WebMock).not_to have_requested(:get, 'http://example.com/100x100')
+        end
+      end
+
+      it "should reuse the limited results for repeated size and best-image queries" do
+        page = MetaInspector.new('http://example.com/', document: html, max_image_downloads: 1)
+
+        2.times do
+          expect(page.images.with_size.first).to eq(['http://example.com/10x10', 10, 10])
+          expect(page.images.largest).to eq('http://example.com/10x10')
+          expect(page.images.best).to eq('http://example.com/10x10')
+        end
+        expect(WebMock).to have_requested(:get, 'http://example.com/10x10').once
+        expect(WebMock).not_to have_requested(:get, 'http://example.com/100x100')
+      end
+
+      it "should keep the budget local to each document" do
+        2.times do
+          page = MetaInspector.new('http://example.com/', document: html, max_image_downloads: 1)
+
+          expect(page.images.with_size.first).to eq(['http://example.com/10x10', 10, 10])
+        end
+        expect(WebMock).to have_requested(:get, 'http://example.com/10x10').twice
+        expect(WebMock).not_to have_requested(:get, 'http://example.com/100x100')
+      end
+
+      it "should keep all image URLs without downloading them when enumerating images" do
+        page = MetaInspector.new('http://example.com/', document: html, max_image_downloads: 1)
+
+        expect(page.images.to_a).to eq([
+          'http://example.com/10x10',
+          'http://example.com/100x100',
+          'http://example.com/remaining.jpg'
+        ])
+        expect(WebMock).not_to have_requested(:get, %r{http://example.com/})
+      end
+    end
+
+    [true, false].each do |download_images|
+      it "should ignore unparseable image URLs with download_images: #{download_images}" do
+        html = '<html><body><img src="http://exa mple.com/x.png"><img src="/valid.png" width="100" height="100"></body></html>'
+        page = MetaInspector.new('https://example.com/', document: html, download_images: download_images)
+
+        expect(FastImage).not_to receive(:size).with(nil)
+        expect(page.images.with_size).to eq([["https://example.com/valid.png", 100, 100]])
+      end
+    end
+
+    it "should ignore unparseable image URLs even when HTML sizes are present" do
+      html = '<html><body><img src="http://exa mple.com/x.png" width="100" height="100"></body></html>'
+      page = MetaInspector.new('https://example.com/', document: html)
+
+      expect(page.images.with_size).to eq([])
+    end
+
     it "should return sorted by area array of [img_url, width, height] using html sizes" do
       page = MetaInspector.new('http://example.com/largest_image_in_html')
 

data/spec/meta_inspector/links_spec.rb

@@ -174,6 +174,28 @@ describe MetaInspector do
   end
 
   describe 'Relative links with relative base' do
+    [
+      ['assets/', 'https://example.com/posts/article', 'https://example.com/posts/assets/next.html'],
+      ['../assets/', 'https://example.com/posts/2026/article', 'https://example.com/posts/assets/next.html'],
+      ['assets/base.html', 'https://example.com/posts/article', 'https://example.com/posts/assets/next.html'],
+      ['assets/', 'https://example.com/posts/article/', 'https://example.com/posts/article/assets/next.html'],
+      ['/assets/', 'https://example.com:8443/posts/article', 'https://example.com:8443/assets/next.html']
+    ].each do |base_href, url, expected_link|
+      it "resolves base #{base_href.inspect} against #{url}" do
+        html = %(<html><head><base href="#{base_href}"></head><body><a href="next.html">Next</a></body></html>)
+        m = MetaInspector.new(url, document: html)
+
+        expect(m.links.all).to eq([expected_link])
+      end
+    end
+
+    it 'resolves image URLs against a document-relative base in a nested directory' do
+      html = '<html><head><base href="assets/"></head><body><img src="image.jpg"></body></html>'
+      m = MetaInspector.new('https://example.com/posts/article', document: html)
+
+      expect(m.images.to_a).to eq(['https://example.com/posts/assets/image.jpg'])
+    end
+
     it 'should get the relative links with relative directory base' do
       m = MetaInspector.new('http://relativewithrelativebase.com/relativedir')
       expect(m.links.all).to eq(['http://relativewithrelativebase.com/other/about',

data/spec/meta_inspector/redirections_spec.rb

@@ -52,22 +52,36 @@ describe MetaInspector do
     end
 
     context "when there is a callback to be ran between redirects that blocks redirections to private IP addresses" do
-      it "raises an exception" do
-        stub_request(:get, "https://www.facebook.com/")
-          .to_return(:status => 302,
-                     :headers => { "Location" => "http://10.0.0.0/" })
-
-        redirect_options = {
+      let(:redirect_options) do
+        {
           callback: proc do |_previous_response, next_request|
             ip_address = Resolv.getaddress(next_request.url.host)
             raise PrivateIPAddressError if IPAddr.new(ip_address).private?
           end
         }
+      end
+
+      before do
+        stub_request(:get, "https://www.facebook.com/")
+          .to_return(:status => 302,
+                     :headers => { "Location" => "http://10.0.0.0/" })
+      end
 
+      it "raises an exception" do
         expect {
           MetaInspector.new("https://www.facebook.com/", faraday_options: { redirect: redirect_options })
         }.to raise_error PrivateIPAddressError
       end
+
+      it "keeps running the callback when the same faraday_options are reused" do
+        faraday_options = { redirect: redirect_options }
+
+        2.times do
+          expect {
+            MetaInspector.new("https://www.facebook.com/", faraday_options: faraday_options)
+          }.to raise_error PrivateIPAddressError
+        end
+      end
     end
   end
 

data/spec/meta_inspector/texts_spec.rb

@@ -1,6 +1,50 @@
 require 'spec_helper'
 
 describe MetaInspector do
+  describe '#language' do
+    ['en', 'en-US', 'zh-Hant-TW'].each do |language|
+      it "should return the declared language #{language.inspect}" do
+        page = MetaInspector.new('http://example.com/', document: %(<html lang="#{language}"><body></body></html>))
+
+        expect(page.language).to eq(language)
+      end
+    end
+
+    it 'should strip surrounding whitespace from the declared language' do
+      page = MetaInspector.new('http://example.com/', document: "<html lang=\" \t en-US \n \"><body></body></html>")
+
+      expect(page.language).to eq('en-US')
+    end
+
+    [nil, '', " \t\n "].each do |value|
+      it "should return nil for a missing or blank language (#{value.inspect})" do
+        attribute = value.nil? ? '' : %( lang="#{value}")
+        page = MetaInspector.new('http://example.com/', document: "<html#{attribute}><body></body></html>")
+
+        expect(page.language).to be_nil
+      end
+    end
+
+    it 'should read the root language rather than a descendant language' do
+      page = MetaInspector.new('http://example.com/', document: '<html lang="en"><body lang="es"></body></html>')
+
+      expect(page.language).to eq('en')
+    end
+
+    it 'should not infer language from xml:lang, metadata, or descendants' do
+      html = '<html xml:lang="fr"><head><meta http-equiv="content-language" content="de"></head><body lang="es">Hello</body></html>'
+      page = MetaInspector.new('http://example.com/', document: html)
+
+      expect(page.language).to be_nil
+    end
+
+    it 'should return the declared language from a fetched document' do
+      page = MetaInspector.new('http://pagerankalert.com/')
+
+      expect(page.language).to eq('en')
+    end
+  end
+
   it "should get the title from the head section" do
     page = MetaInspector.new('http://example.com')
     expect(page.title).to eq('An example page')
@@ -49,6 +93,42 @@ describe MetaInspector do
   end
 
   describe '#best_title' do
+    it 'should use the Parsely title when no existing candidate is present' do
+      page = MetaInspector.new('http://example.com/', document: '<meta name="parsely-title" content="  A Parsely&#10; title  ">')
+
+      expect(page.best_title).to eq('A Parsely title')
+    end
+
+    it 'should use the Parsely title when existing candidates are empty' do
+      html = '<html><head><meta name="title" content=" "><meta property="og:title" content=""><title> </title><meta name="parsely-title" content="Parsely title"></head><body><h1> </h1></body></html>'
+      page = MetaInspector.new('http://example.com/', document: html)
+
+      expect(page.best_title).to eq('Parsely title')
+    end
+
+    [
+      ['<meta name="title" content="Existing title">', ''],
+      ['<meta property="og:title" content="Existing title">', ''],
+      ['<title>Existing title</title>', ''],
+      ['', '<title>Existing title</title>'],
+      ['', '<h1>Existing title</h1>']
+    ].each do |head, body|
+      it "should prefer #{head + body} over the Parsely title" do
+        html = %(<html><head><meta name="parsely-title" content="Parsely title">#{head}</head><body>#{body}</body></html>)
+        page = MetaInspector.new('http://example.com/', document: html)
+
+        expect(page.best_title).to eq('Existing title')
+      end
+    end
+
+    ['', '  &#10;  '].each do |value|
+      it "should ignore an empty Parsely title (#{value.inspect})" do
+        page = MetaInspector.new('http://example.com/', document: %(<meta name="parsely-title" content="#{value}">))
+
+        expect(page.best_title).to be_nil
+      end
+    end
+
     it "should find 'head title' when that's the only thing" do
       page = MetaInspector.new('http://example.com/title_in_head')
       expect(page.best_title).to eq('This title came from the head')
@@ -105,6 +185,41 @@ describe MetaInspector do
   end
 
   describe "#best_author" do
+    it 'should use the Parsely author when no existing candidate is present' do
+      page = MetaInspector.new('http://example.com/', document: '<meta name="parsely-author" content="  A Parsely&#10; author  ">')
+
+      expect(page.best_author).to eq('A Parsely author')
+    end
+
+    it 'should use the Parsely author when existing candidates are empty' do
+      html = '<meta name="author" content=" "><meta name="twitter:creator" content=""><meta name="parsely-author" content="Parsely author"><a rel="author"> </a><address> </address>'
+      page = MetaInspector.new('http://example.com/', document: html)
+
+      expect(page.best_author).to eq('Parsely author')
+    end
+
+    [
+      '<meta name="author" content="Existing author">',
+      '<a rel="author">Existing author</a>',
+      '<address>Existing author</address>',
+      '<meta name="twitter:creator" content="Existing author">'
+    ].each do |candidate|
+      it "should prefer #{candidate} over the Parsely author" do
+        html = '<meta name="parsely-author" content="Parsely author">' + candidate
+        page = MetaInspector.new('http://example.com/', document: html)
+
+        expect(page.best_author).to eq('Existing author')
+      end
+    end
+
+    ['', '  &#10;  '].each do |value|
+      it "should ignore an empty Parsely author (#{value.inspect})" do
+        page = MetaInspector.new('http://example.com/', document: %(<meta name="parsely-author" content="#{value}">))
+
+        expect(page.best_author).to be_nil
+      end
+    end
+
     it "should return the author meta tag content if present" do
       page = MetaInspector.new('http://example.com/author_in_meta')
 

data/spec/request_spec.rb

@@ -68,6 +68,35 @@ describe MetaInspector::Request do
     end
   end
 
+  describe "options it is given" do
+    it "accepts a frozen faraday_options hash" do
+      expect do
+        MetaInspector::Request.new(url('http://example.com'), faraday_options: { ssl: { verify: true } }.freeze)
+      end.not_to raise_error
+    end
+
+    it "leaves the faraday_options hash unchanged" do
+      callback        = proc {}
+      faraday_options = { redirect: { callback: callback } }
+
+      MetaInspector::Request.new(url('http://example.com'), allow_redirections: true,
+                                                            faraday_options:    faraday_options)
+
+      expect(faraday_options).to eq({ redirect: { callback: callback } })
+    end
+
+    it "leaves the faraday_http_cache hash unchanged" do
+      cache = Hash.new
+      def cache.read(k) self[k]; end
+      def cache.write(k, v) self[k] = v; end
+      cache_options = { store: cache }
+
+      MetaInspector::Request.new(url('http://example.com'), faraday_http_cache: cache_options)
+
+      expect(cache_options).to eq({ store: cache })
+    end
+  end
+
   describe 'exception handling' do
     before(:each) do
       WebMock.allow_net_connect!

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: metainspector
 version: !ruby/object:Gem::Version
-  version: 5.18.1
+  version: 5.19.0
 platform: ruby
 authors:
 - Jaime Iniesta
@@ -406,6 +406,7 @@ files:
 - spec/fixtures/wordpress_site.response
 - spec/fixtures/youtube.response
 - spec/fixtures/youtube_short_title.response
+- spec/meta_inspector/head_first_spec.rb
 - spec/meta_inspector/head_links_spec.rb
 - spec/meta_inspector/images_spec.rb
 - spec/meta_inspector/links_spec.rb