metainspector 5.18.1 → 5.20.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c74590fe15ada0cba938c3a9c4a0d167fcb10bd6159615751652e6376dec7110
-  data.tar.gz: d92a0ba92d9d7c6b1fa3404cdebc001a49080cbeba419766f6d60a2b96e1b582
+  metadata.gz: ba31ea21114c1932ca2dde179a2812f24d429b7420807fec232de65ea1ba824e
+  data.tar.gz: 99188cbec093cfd1d22cdfd3d31bf181cf193e937f6c7b23a61cb6ed39ba1ca2
 SHA512:
-  metadata.gz: 2f0d12a22b1288aae2ca671ca5aa0516dad4aa3590b32be9337a626f5741d2c07d8050c26f013fa7a6478f83733e372425b0ce0ad450ee7d8f2d8a64f4efdbd9
-  data.tar.gz: 4a3662b35d9e4b5485dc00cd8101d216a086e70c08b152a0dc1c89045f7f179031b8597ccf92b106792f604b340aa49907de8dc6ca9758c6755f32b55fd6a109
+  metadata.gz: bc39e09f57cadd29f60e6a41556878d807a629e99de37b6b64b3ebf090d32aeca63d49b38af436fa16e5fff29e4cf5e927d99df5b37f35385ea30165a8de55e3
+  data.tar.gz: d4a4777631f8c63d238236960148330a3dc7ba0f4ae2ec0b497cc3e03f33a5ca8daca05141dbaaa20a99a54fd933d5a5c9cb7056dd4b84b12c1a515119935574

data/.gitignore

@@ -9,3 +9,4 @@ pkg/*
 .rubocop_todo.yml
 .rubocop.yml
 .tool-versions
+docs/

data/CHANGELOG.md

@@ -1,5 +1,20 @@
 # MetaInspector Changelog
 
+## [Changes in 5.20.0](https://github.com/jaimeiniesta/metainspector/compare/v5.19.0...v5.20.0)
+
+* Security: route image size downloads through Faraday so they respect configured network and redirect controls. Add optional `url_filter:` to validate the initial page URL, HEAD requests, image URLs and every redirect destination before requesting them. Returning `false` or `nil` blocks the request with `MetaInspector::RequestError`; filter exceptions propagate. Only HTTP(S) requests are supported ([GHSA-6vgc-f9v5-8939](https://github.com/jaimeiniesta/metainspector/security/advisories/GHSA-6vgc-f9v5-8939)).
+* Accept `faraday_http_cache` inside `faraday_options`, which used to raise `NameError`. The top-level `faraday_http_cache` option still works but is deprecated: it prints a warning once per process. Passing both raises `ArgumentError` (#373).
+
+## [Changes in 5.19.0](https://github.com/jaimeiniesta/metainspector/compare/v5.18.1...v5.19.0)
+
+* Security: stop modifying the `faraday_options` hash passed to MetaInspector. When the same hash was reused, its `redirect` options were deleted after the first request, so a redirect `callback` used to block private hosts only ran once and later requests followed any redirect ([GHSA-mm2p-6q2v-7ph5](https://github.com/jaimeiniesta/metainspector/security/advisories/GHSA-mm2p-6q2v-7ph5)). Frozen option hashes no longer raise `FrozenError`.
+* Add `language` to expose the root HTML `lang` attribute, trimming surrounding whitespace and returning `nil` for missing or blank values. Include it in `to_hash` (#121).
+* Add opt-in `max_image_downloads` to limit image size lookup attempts while preserving dimensions already available in HTML (#242).
+* Add `parsely-title`, `parsely-author`, and `parsely-image-url` fallbacks to `best_title`, `best_author`, and `images.owner_suggested` / `images.best`, preserving existing preferred metadata sources (#214).
+* Resolve relative base hrefs against the full document URL, preserving nested paths and non-default ports (#240).
+* Skip invalid image URLs in `images.with_size`, preventing `images.best` from failing when FastImage receives a nil URL ([#521](https://github.com/jaimeiniesta/metainspector/issues/521), follow-up to [#359](https://github.com/jaimeiniesta/metainspector/issues/359)).
+* Add opt-in `head_first: true` to check content type with HEAD and reject non-HTML files before downloading them (#221).
+
 ## [Changes in 5.18.1](https://github.com/jaimeiniesta/metainspector/compare/v5.18.0...v5.18.1)
 
 * Require nokogiri 1.19.5, which fixes three low-severity security advisories.

data/README.md

@@ -84,6 +84,7 @@ page.feeds               # Get rss or atom links in meta data fields as array of
 ### Texts
 
 ```ruby
+page.language            # declared language from the root <html lang="..."> attribute
 page.title               # title of the page from the head section, as string
 page.best_title          # best title of the page, from a selection of candidates
 page.author              # author of the page from the meta author tag
@@ -98,6 +99,20 @@ page.h5                  # returns h5 text array
 page.h6                  # returns h6 text array
 ```
 
+`language` trims surrounding whitespace and preserves the declared value, including
+capitalization and regional tags such as `en-US`. It returns `nil` when the `lang`
+attribute is missing, empty, or whitespace-only. It does not infer language from
+the page content or other attributes. The value is also included in `to_hash`
+under the `"language"` key.
+
+`best_title` selects the first non-empty value from the standard `title` meta tag,
+`og:title`, the head `<title>`, a body `<title>`, the first `<h1>`, and finally
+`parsely-title`.
+
+`best_author` selects the first non-empty value from the standard `author` meta tag,
+the first link with `rel="author"`, the first `<address>`, `twitter:creator`, and
+finally `parsely-author`. Both methods trim and collapse whitespace in their results.
+
 ### Links
 
 ```ruby
@@ -114,10 +129,15 @@ page.links.external      # every external link found on the page as an absolute
 ```ruby
 page.images              # enumerable collection, with every img found on the page as an absolute URL
 page.images.with_size    # a sorted array (by descending area) of [image_url, width, height]
-page.images.best         # Most relevant image, if defined with the og:image or twitter:image metatags. Fallback to the first page.images array element
+page.images.best         # Most relevant image from og:image, twitter:image, or parsely-image-url; falls back to page.images.largest
 page.images.favicon      # absolute URL to the favicon
 ```
 
+`images.owner_suggested` tries `og:image`, `twitter:image`, and `parsely-image-url`
+in that order, skipping empty or whitespace-only values. Relative image URLs are
+resolved against the document's base URL. `images.best` uses this suggested image
+before falling back to the largest suitable image on the page.
+
 ### Meta tags
 
 When it comes to meta tags, you have several options:
@@ -321,8 +341,7 @@ And even customize what to do in between each redirect:
 
 ```ruby
 callback = proc do |previous_response, next_request|
-  ip_address = Resolv.getaddress(next_request.url.host)
-  raise 'Invalid address' if IPAddr.new(ip_address).private?
+  puts "Following redirect to #{next_request.url}"
 end
 
 page = MetaInspector.new(url, faraday_options: { redirect: { callback: callback } })
@@ -332,6 +351,63 @@ page = MetaInspector.new(url, faraday_options: { redirect: { callback: callback
 The `faraday_options[:redirect]` hash is passed to the `FollowRedirects` middleware used by `Faraday`, so that we can use all available options.
 Check them [here](https://github.com/lostisland/faraday_middleware/blob/main/lib/faraday_middleware/response/follow_redirects.rb#L44).
 
+### Filtering request URLs
+
+Use `url_filter:` to decide which destinations MetaInspector may request. The filter receives a Ruby `URI` and runs before the initial page request, the optional HEAD request, image size downloads, and every redirect destination. Only HTTP and HTTPS requests are supported.
+
+For example, allow HTTPS requests to two known hosts:
+
+```ruby
+allowed_hosts = %w[example.com images.example.com]
+
+filter = proc do |uri|
+  uri.scheme == 'https' && allowed_hosts.include?(uri.hostname)
+end
+
+page = MetaInspector.new('https://example.com', url_filter: filter)
+page.images.largest # Image requests and their redirects use the same filter.
+```
+
+A truthy return value permits the request. Returning `false` or `nil` raises `MetaInspector::RequestError` before contacting the rejected destination. This also applies when an image request is rejected:
+
+```ruby
+begin
+  page = MetaInspector.new(url, url_filter: filter)
+  page.images.largest
+rescue MetaInspector::RequestError => error
+  warn error.message
+end
+```
+
+Exceptions raised by your filter propagate to the caller. The filter is optional: MetaInspector does not block private IP addresses by default. It applies to outgoing requests, including image downloads from HTML passed with `document:`. Reading image URLs with `images.to_a` or `images.owner_suggested` does not download them or run the filter.
+
+If your application accepts arbitrary hostnames, this example rejects addresses in private, loopback, link-local and unspecified ranges, including IPv4-mapped IPv6 addresses:
+
+```ruby
+require 'ipaddr'
+require 'resolv'
+
+zero_network = IPAddr.new('0.0.0.0/8')
+
+filter = proc do |uri|
+  addresses = Resolv.getaddresses(uri.hostname)
+
+  !addresses.empty? && addresses.all? do |address|
+    ip = IPAddr.new(address)
+    ip = ip.native if ip.ipv4_mapped?
+
+    !(ip.private? || ip.loopback? || ip.link_local? ||
+      ip.to_i.zero? || zero_network.include?(ip))
+  end
+end
+
+page = MetaInspector.new(url, url_filter: filter)
+```
+
+DNS resolution in this example happens separately from the connection, so it cannot prevent DNS rebinding. Use network egress restrictions as well when processing untrusted URLs. The example covers the listed ranges; applications may need additional address or port restrictions.
+
+The existing redirect `callback` keeps its two arguments and runs only on redirects. Move destination validation from that callback to `url_filter:` to cover initial URLs as well as redirects, for both pages and images.
+
 ### Headers
 
 By default, the following headers are set:
@@ -385,6 +461,28 @@ page.content_type  # "image/png"
 page.description   # will return a garbled string
 ```
 
+### Check content type before downloading
+
+To avoid downloading large non-HTML files, enable `head_first`:
+
+```ruby
+page = MetaInspector.new('http://example.com/page', head_first: true)
+```
+
+This sends an HTTP HEAD request before GET. If a successful HEAD response declares
+a content type other than `text/html`, MetaInspector raises `MetaInspector::NonHtmlError`
+without downloading the body. Content-type parameters such as `charset` are ignored.
+With `allow_non_html_content: true`, the download proceeds regardless of content type.
+
+If HEAD omits the content type or returns an unsuccessful status (including 405 or
+501 for servers that do not support HEAD), MetaInspector continues with GET and its
+usual content-type check. HEAD timeouts and connection errors use the same exception
+handling as GET. The final page data, response headers and status come from GET.
+
+This option is off by default because it adds a request and not every server
+supports HEAD. Both requests use the configured headers, timeouts, retries and
+redirect settings. Passing a `document` directly makes no HTTP requests.
+
 ### URL Normalization
 
 By default, URLs are normalized using the Addressable gem. For example:
@@ -405,7 +503,13 @@ You can disable URL normalization by passing the `normalize_url: false` option.
 
 ### Image downloading
 
-When you ask for the largest image on the page with `page.images.largest`, it will be determined by its height and width attributes on the HTML markup, and also by downloading a small portion of each image using the [fastimage](https://github.com/sdsykes/fastimage) gem. This is really fast as it doesn't download the entire images, normally just the headers of the image files.
+Image downloading is enabled by default. `page.images.with_size` and `page.images.largest` fetch images with missing HTML dimensions; `page.images.best` also does this when there is no preferred metadata image. The page controls these image URLs, which may point to other hosts.
+
+Image requests use Faraday with the page's proxy, TLS, headers and redirect settings, and respect [`url_filter:`](#filtering-request-urls). Downloaded bytes are passed to [fastimage](https://github.com/sdsykes/fastimage) to determine dimensions. Dimensions already present in HTML are preserved without downloading the image.
+
+Custom page headers are preserved for images on the same origin (scheme, host and port). Requests to other origins, including image redirects, send only standard image request headers such as `User-Agent` and `Accept`; they do not receive page credentials or arbitrary custom headers.
+
+Each size lookup stops at a 256 KiB response body budget or a two-second overall timeout, including redirects. If the dimensions cannot be determined within these limits, they remain `[0, 0]`. Streamed image responses are not stored in the HTTP response cache, and failed image requests are not retried.
 
 If you want to disable this, you can specify it like this:
 
@@ -413,15 +517,30 @@ If you want to disable this, you can specify it like this:
 page = MetaInspector.new('http://example.com', download_images: false)
 ```
 
+To limit how many images are downloaded to determine their dimensions, use `max_image_downloads`:
+
+```ruby
+page = MetaInspector.new('http://example.com', max_image_downloads: 10)
+page.images.with_size
+```
+
+The default is `nil` (unlimited). The option accepts a non-negative integer or `nil`; `0` disables image downloads. `download_images: false` always disables downloads, regardless of the limit.
+
+The limit counts image size lookup attempts in HTML order, after excluding unparseable URLs and duplicates. Images with both HTML dimensions already present do not consume the limit, and those dimensions are preserved even after the limit is reached. Failed lookups count toward the limit. Images with incomplete dimensions that are not downloaded remain in `images.with_size` with dimensions `[0, 0]`, matching the behavior when downloads are disabled.
+
+The limit leaves `images.to_a` unchanged, but can affect `images.largest` and the largest-image fallback in `images.best`, since images with unknown dimensions cannot be ranked by size. Size lookup results are cached per document, so repeated calls reuse the same results. This limits size lookup attempts, not elapsed time or HTTP requests made while following image redirects.
+
 ### Caching responses
 
-MetaInspector can be configured to use [Faraday::HttpCache](https://github.com/plataformatec/faraday-http-cache) to cache page responses. For that you should pass the `faraday_http_cache` option with at least the `:store` key, for example:
+MetaInspector can be configured to use [Faraday::HttpCache](https://github.com/plataformatec/faraday-http-cache) to cache page responses. For that you should pass the `faraday_http_cache` option inside `faraday_options`, with at least the `:store` key, for example:
 
 ```ruby
 cache = ActiveSupport::Cache.lookup_store(:file_store, '/tmp/cache')
-page = MetaInspector.new('http://example.com', faraday_http_cache: { store: cache })
+page = MetaInspector.new('http://example.com', faraday_options: { faraday_http_cache: { store: cache } })
 ```
 
+The top-level `faraday_http_cache` option still works, but it is deprecated and prints a warning, once per process. Passing it in both places raises an `ArgumentError`.
+
 ## Exception Handling
 
 Web page scraping is tricky, you can expect to find different exceptions during the request of the page or the parsing of its contents. MetaInspector will encapsulate these exceptions on these main errors:

data/lib/meta_inspector/document.rb

@@ -9,12 +9,23 @@ module MetaInspector
     # * read_timeout: defaults to 20 seconds
     # * retries: defaults to 3 times
     # * allow_redirections: when true, follow HTTP redirects. Defaults to true
+    # * head_first: check content type with HEAD before GET. Defaults to false
     # * document: the html of the url as a string
     # * headers: object containing custom headers for the request
     # * normalize_url: true by default
-    # * faraday_options: an optional hash of options to pass to Faraday on the request
+    # * download_images: fetch missing image dimensions. Defaults to true
+    # * max_image_downloads: maximum image size lookups, or nil for unlimited
+    # * url_filter: callable receiving a URI before each request; false/nil rejects it
+    # * faraday_options: an optional hash of options to pass to Faraday on the request.
+    #   Its faraday_http_cache key configures response caching with Faraday::HttpCache
+    # * faraday_http_cache: deprecated, use faraday_options: { faraday_http_cache: ... }
     def initialize(initial_url, options = {})
       options             = defaults.merge(options)
+      @max_image_downloads = options[:max_image_downloads]
+      unless @max_image_downloads.nil? || (@max_image_downloads.is_a?(Integer) && @max_image_downloads >= 0)
+        raise ArgumentError, 'max_image_downloads must be a non-negative Integer or nil'
+      end
+
       @connection_timeout = options[:connection_timeout]
       @read_timeout       = options[:read_timeout]
       @retries            = options[:retries]
@@ -29,8 +40,12 @@ module MetaInspector
       @normalize_url      = options[:normalize_url]
       @faraday_options    = options[:faraday_options]
       @faraday_http_cache = options[:faraday_http_cache]
+      @http_session       = MetaInspector::HttpSession.new(options)
       @url                = MetaInspector::URL.new(initial_url, normalize:          @normalize_url)
-      @request            = MetaInspector::Request.new(@url,    allow_redirections: @allow_redirections,
+      @request            = MetaInspector::Request.new(@url,    http_session: @http_session,
+                                                                allow_redirections: @allow_redirections,
+                                                                head_first:         options[:head_first],
+                                                                allow_non_html_content: @allow_non_html_content,
                                                                 connection_timeout: @connection_timeout,
                                                                 read_timeout:       @read_timeout,
                                                                 retries:            @retries,
@@ -38,7 +53,9 @@ module MetaInspector
                                                                 headers:            @headers,
                                                                 faraday_options:    @faraday_options,
                                                                 faraday_http_cache: @faraday_http_cache) unless @document
-      @parser             = MetaInspector::Parser.new(self,     download_images:    @download_images)
+      @parser             = MetaInspector::Parser.new(self,     download_images:    @download_images,
+                                                                max_image_downloads: @max_image_downloads,
+                                                                image_downloader: MetaInspector::ImageDownloader.new(@http_session, @url, options))
     end
 
     extend Forwardable
@@ -47,7 +64,7 @@ module MetaInspector
 
     delegate [:content_type, :response]               => :@request
 
-    delegate [:parsed, :title, :best_title, :author, :best_author,
+    delegate [:parsed, :language, :title, :best_title, :author, :best_author,
               :h1, :h2, :h3, :h4, :h5, :h6, :description, :best_description, :links,
               :images, :feeds, :feed, :charset, :meta_tags,
               :meta_tag, :meta, :favicon,
@@ -60,6 +77,7 @@ module MetaInspector
         'scheme'           => scheme,
         'host'             => host,
         'root_url'         => root_url,
+        'language'         => language,
         'title'            => title,
         'best_title'       => best_title,
         'author'           => author,
@@ -101,8 +119,10 @@ module MetaInspector
                                   },
         :allow_redirections     => true,
         :allow_non_html_content => false,
+        :head_first             => false,
         :normalize_url          => true,
-        :download_images        => true }
+        :download_images        => true,
+        :max_image_downloads    => nil }
     end
 
     def default_user_agent

data/lib/meta_inspector/http_session.rb

@@ -0,0 +1,65 @@
+require 'http/cookie'
+require 'faraday'
+require 'faraday-cookie_jar'
+require 'faraday-http-cache'
+require 'faraday/encoding'
+require 'faraday/follow_redirects'
+require 'faraday/gzip'
+require 'faraday/retry'
+require_relative 'url_policy'
+
+module MetaInspector
+  # Builds page and binary image transports from the same caller configuration.
+  class HttpSession
+    def initialize(options = {})
+      @allow_redirections = options[:allow_redirections]
+      @retries = options[:retries]
+      @headers = options[:headers]
+      @url_filter = options[:url_filter]
+      if !@url_filter.nil? && !@url_filter.respond_to?(:call)
+        raise ArgumentError, 'url_filter must respond to call'
+      end
+      @cache = cache_option(options)
+      @faraday_options = (options[:faraday_options] || {}).except(:faraday_http_cache)
+    end
+
+    def build(url, image: false, origin: nil)
+      URLPolicy.validate!(url)
+      faraday_options = @faraday_options.merge(url: url)
+      redirects = (faraday_options.delete(:redirect) || {}).dup
+      Faraday.new(faraday_options) do |faraday|
+        unless image
+          faraday.request :retry, max: @retries
+          faraday.request :gzip
+        end
+        if @allow_redirections
+          redirects[:limit] ||= 10
+          faraday.use Faraday::FollowRedirects::Middleware, **redirects
+          faraday.use :cookie_jar
+        end
+        faraday.use URLPolicy, @url_filter, image ? origin : nil
+        if !image && @cache.is_a?(Hash)
+          cache_options = @cache.dup
+          cache_options[:serializer] ||= Marshal
+          faraday.use Faraday::HttpCache, **cache_options
+        end
+        faraday.headers.merge!(@headers || {})
+        faraday.response :encoding unless image
+        faraday.adapter :net_http
+      end
+    end
+
+    private
+
+    def cache_option(options)
+      nested = (options[:faraday_options] || {})[:faraday_http_cache]
+      top_level = options[:faraday_http_cache]
+      return nested if top_level.nil?
+      unless nested.nil?
+        raise ArgumentError, 'Pass faraday_http_cache inside faraday_options or as a top-level option, not both'
+      end
+      Request.warn_faraday_http_cache_deprecation
+      top_level
+    end
+  end
+end

data/lib/meta_inspector/image_downloader.rb

@@ -0,0 +1,53 @@
+require 'stringio'
+require 'fastimage'
+require 'timeout'
+
+module MetaInspector
+  class ImageDownloader
+    MAX_BYTES = 256 * 1024
+    TIMEOUT = 2
+
+    def initialize(session, origin, options = {})
+      @session = session
+      @origin = origin
+      @connection_timeout = [options[:connection_timeout] || TIMEOUT, TIMEOUT].min
+      @read_timeout = [options[:read_timeout] || TIMEOUT, TIMEOUT].min
+    end
+
+    def size(url)
+      bytes = ''.b
+      received = 0
+      status = nil
+      stop = Object.new
+      Timeout.timeout(TIMEOUT) do
+        session = @session.build(url, image: true, origin: @origin.url)
+        catch(stop) do
+          session.get do |request|
+            request.headers['Range'] = "bytes=0-#{MAX_BYTES - 1}"
+            request.headers['Accept-Encoding'] = 'identity'
+            request.options.open_timeout = @connection_timeout
+            request.options.timeout = @read_timeout
+            request.options.on_data = proc do |chunk, _total, env|
+              status = env.status
+              remaining = MAX_BYTES - received
+              received += chunk.bytesize
+              # Count redirect bodies too, across the entire lookup.
+              if (200..299).cover?(status)
+                bytes << chunk.b.byteslice(0, remaining)
+              elsif !(300..399).cover?(status)
+                throw stop
+              end
+              throw stop if received >= MAX_BYTES
+            end
+          end
+        end
+      end
+      FastImage.size(StringIO.new(bytes)) if status && (200..299).cover?(status)
+    rescue URLPolicy::Failure => failure
+      raise failure.original
+    rescue Faraday::TimeoutError, Faraday::ConnectionFailed, Faraday::SSLError,
+           Faraday::FollowRedirects::RedirectLimitReached, URI::InvalidURIError, Timeout::Error
+      nil
+    end
+  end
+end

data/lib/meta_inspector/parser.rb

@@ -14,7 +14,9 @@ module MetaInspector
       @meta_tag_parser = MetaInspector::Parsers::MetaTagsParser.new(self)
       @links_parser    = MetaInspector::Parsers::LinksParser.new(self)
       @download_images = options[:download_images]
-      @images_parser   = MetaInspector::Parsers::ImagesParser.new(self, download_images: @download_images)
+      @images_parser   = MetaInspector::Parsers::ImagesParser.new(self, download_images: @download_images,
+                                                                      max_image_downloads: options[:max_image_downloads],
+                                                                      image_downloader: options[:image_downloader])
       @texts_parser    = MetaInspector::Parsers::TextsParser.new(self)
 
       parsed           # parse early so we can fail early
@@ -26,7 +28,7 @@ module MetaInspector
     delegate [:head_links, :stylesheets, :canonicals, :feeds, :feed]                       => :@head_links_parser
     delegate [:links, :base_url]                                                           => :@links_parser
     delegate :images                                                                       => :@images_parser
-    delegate [:title, :best_title, :author, :best_author, :description, :best_description, 
+    delegate [:language, :title, :best_title, :author, :best_author, :description, :best_description,
               :h1, :h2, :h3, :h4, :h5, :h6]                                                => :@texts_parser
 
     # Returns the whole parsed document

data/lib/meta_inspector/parsers/images.rb

@@ -1,4 +1,4 @@
-require 'fastimage'
+require_relative '../image_downloader'
 
 module MetaInspector
   module Parsers
@@ -9,7 +9,9 @@ module MetaInspector
       include Enumerable
 
       def initialize(main_parser, options = {})
+        @image_downloader = options[:image_downloader]
         @download_images = options[:download_images]
+        @max_image_downloads = options[:max_image_downloads]
         super(main_parser)
       end
 
@@ -17,7 +19,7 @@ module MetaInspector
         self
       end
 
-      # Returns either the Facebook Open Graph image, twitter suggested image or
+      # Returns the Facebook Open Graph image, Twitter image, Parsely image or
       # the largest image in the image collection
       def best
         owner_suggested || largest
@@ -26,9 +28,10 @@ module MetaInspector
       # Returns the parsed image from Facebook's open graph property tags
       # Most major websites now define this property and is usually relevant
       # See doc at http://developers.facebook.com/docs/opengraph/
-      # If none found, tries with Twitter image
+      # If none found, tries with Twitter image, then Parsely image
       def owner_suggested
-        suggested_img = content_of(meta['og:image']) || content_of(meta['twitter:image'])
+        suggested_img = content_of(meta['og:image']) || content_of(meta['twitter:image']) ||
+                        content_of(meta['parsely-image-url'])
         URL.absolutify(suggested_img, base_url, normalize: false) if suggested_img
       end
 
@@ -39,17 +42,19 @@ module MetaInspector
           imgs_with_size = img_nodes.map do |img_node|
             [URL.absolutify(img_node['src'], base_url, normalize: false), img_node['width'], img_node['height']]
           end
+          imgs_with_size.reject! { |url, width, height| url.nil? || url.empty? }
           imgs_with_size.uniq! { |url, width, height| url }
-          if @download_images
-            imgs_with_size.map! do |url, width, height|
-              width, height = FastImage.size(url) if width.nil? || height.nil?
-              [url, width.to_i, height.to_i]
-            end
-          else
-            imgs_with_size.map! do |url, width, height|
-              width, height = [0, 0] if width.nil? || height.nil?
-              [url, width.to_i, height.to_i]
+          download_attempts = 0
+          imgs_with_size.map! do |url, width, height|
+            if width.nil? || height.nil?
+              if @download_images && (@max_image_downloads.nil? || download_attempts < @max_image_downloads)
+                download_attempts += 1
+                width, height = @image_downloader.size(url)
+              else
+                width, height = [0, 0]
+              end
             end
+            [url, width.to_i, height.to_i]
           end
           imgs_with_size.sort_by { |url, width, height| -(width.to_i * height.to_i) }
         end
@@ -93,7 +98,7 @@ module MetaInspector
       end
 
       def content_of(content)
-        return nil if content.nil? || content.empty?
+        return nil if content.nil? || content.strip.empty?
         content
       end
     end

data/lib/meta_inspector/parsers/links.rb

@@ -46,8 +46,9 @@ module MetaInspector
       # Returns the base url to absolutify relative links.
       # This can be the one set on a <base> tag,
       # or the url of the document if no <base> tag was found.
+      # Relative base hrefs are resolved against the document URL.
       def base_url
-        current_base_href = base_href.to_s.strip.empty? ? nil : URL.absolutify(base_href, URL.new(url).root_url)
+        current_base_href = base_href.to_s.strip.empty? ? nil : URL.absolutify(base_href, url)
         current_base_href || url
       end
 

data/lib/meta_inspector/parsers/texts.rb

@@ -3,6 +3,12 @@ module MetaInspector
     class TextsParser < Base
       delegate [:parsed, :meta] => :@main_parser
 
+      # Returns the declared HTML language, or nil when missing or blank
+      def language
+        value = parsed.at_css('html')&.[]('lang')&.strip
+        value unless value.to_s.empty?
+      end
+
       # Returns the parsed document title, from the content of the <title> tag
       # within the <head> section.
       def title
@@ -42,12 +48,13 @@ module MetaInspector
         @author ||= meta['author']
       end
 
-      # An author getter that returns the first non-nil description
+      # An author getter that returns the first non-empty author
       # from the following candidates:
-      # - the standard meta description
+      # - the standard meta author
       # - a link with the relational attribute "author"
       # - address tag which may contain the author
       # - the twitter:creator meta tag for the username
+      # - the parsely-author meta tag
       def best_author
         @best_author ||= find_best_author
       end
@@ -80,7 +87,8 @@ module MetaInspector
             meta['og:title'],
             parsed.css('head title'),
             parsed.css('body title'),
-            parsed.css('h1').first
+            parsed.css('h1').first,
+            meta['parsely-title']
         ]
         candidates.flatten!
         candidates.compact!
@@ -94,7 +102,8 @@ module MetaInspector
           meta['author'],
           parsed.css('a[rel="author"]').first,
           parsed.css('address').first,
-          meta['twitter:creator']
+          meta['twitter:creator'],
+          meta['parsely-author']
         ]
         candidates.flatten!
         candidates.compact!