link_thumbnailer 3.3.2

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 281992a6403788c75541c08d69586ab9b85c097e
+  data.tar.gz: 7cd32584b4c3a96b6a54185576597115fdb84764
+SHA512:
+  metadata.gz: d77d1ae2545a92f79363a1e90d19edd61c610080d5497207cf22bdb9afe0d0acb5569aa95d7c16485dc9aa3ff0beac5c900b7cea24278d87d5a0812900f7aa91
+  data.tar.gz: 69c55be441d1581c50e54b3f3652130d8c4f9220047714607d978a993c6be581b28b70cb19a62ad3f27dc09c91455d67f4e83bceff753b451a4d3b3a1e532763

data/.gitignore

@@ -0,0 +1,19 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+.project
+.coveralls.yml

data/.rspec

@@ -0,0 +1,2 @@
+--colour
+--format=documentation

data/.ruby-version

@@ -0,0 +1 @@
+2.4.0

data/.travis.yml

@@ -0,0 +1,5 @@
+bundler_args: --without development
+language: ruby
+rvm:
+  - 2.2.2
+  - 2.4.0

data/CHANGELOG.md

@@ -0,0 +1,334 @@
+# 3.3.2
+
+- Frozen strings https://github.com/gottfrois/link_thumbnailer/pull/125
+
+# 3.3.1
+
+- Gem upgrade (json)
+
+# 3.3.0
+
+- Allows to configure overrided http headers
+
+```ruby
+LinkThumbnailer.configure do |config|
+  config.http_override_headers = { 'Accept-Encoding' => 'none', ... }
+end
+```
+
+# 3.2.1
+
+- Fixes #88
+- Override User-Agent header properly
+- Match xpath nodes if attribute content is present
+- Avoid nil urls in image parser
+
+# 3.2.0
+
+Makes scrapers configurable by allowing to set the scraping strategy:
+
+```ruby
+LinkThumbnailer.configure do |config|
+  config.scrapers = [:opengraph, :default]
+end
+```
+
+`opengraph` use the [Open Graph Protocol](http://ogp.me/).
+`default` use a homemade algorithm
+
+# 3.1.2
+
+Allows to customize ideal description length
+
+Pass the :ideal_description_length option to the Graders::Length initializer to customize
+the ideal description length of a website. In the rails initializer:
+
+```ruby
+LinkThumbnailer.configure do |config|
+  config.graders = [
+    ->(description) { ::LinkThumbnailer::Graders::Length.new(description, ideal_description_length: 500) },
+    ->(description) { ::LinkThumbnailer::Graders::HtmlAttribute.new(description, :class) },
+    ->(description) { ::LinkThumbnailer::Graders::HtmlAttribute.new(description, :id) },
+    ->(description) { ::LinkThumbnailer::Graders::Position.new(description, weigth: 3) },
+    ->(description) { ::LinkThumbnailer::Graders::LinkDensity.new(description) },
+  ]
+end
+```
+
+Will default to `120` characters. More information about how the gem manage to find the best description can be found at
+http://www.codeids.com/2015/06/27/how-to-find-best-description-of-a-website-using-linkthumbnailer/
+
+# 3.1.1
+
+- Upgrade `video_info` gem
+- Fixes https://github.com/gottfrois/link_thumbnailer/issues/69
+
+# 3.1.0
+
+- Fix an issue when image sizes could not be retrieved.
+- Grapers now accepts an optional parameter to customize the weigth of the grader in the probablity computation.
+
+```ruby
+LinkThumbnailer::Graders::Position.new(description, weigth: 3)
+```
+
+Will give a 3 times more weigth to the `Position` grader compare to other graders.
+By default all graders have a weigth of `1` except the above position grader since position should play a bigger role in
+order to find good description candidates.
+
+# 3.0.3
+
+- Fix an issue when dealing with absolute urls. https://github.com/gottfrois/link_thumbnailer/issues/68
+- Fix an issue with http redirection and location header not beeing present. https://github.com/gottfrois/link_thumbnailer/issues/70
+- Rescue and raise custom LinkThumbnailer exceptions. https://github.com/gottfrois/link_thumbnailer/issues/71
+
+# 3.0.2
+
+- Replace FastImage gem dependency by [ImageInfo](https://github.com/gottfrois/image_info) to improve performances when
+fetching multiple images size information. Benchmark shows an order of magnitude improvement response time.
+- Fixes [#57](https://github.com/gottfrois/link_thumbnailer/issues/57)
+
+# 3.0.1
+
+- Remove useless dependencies
+
+# 3.0.0
+
+- Improved description sorting.
+- Refactored how graders work. More information [here](https://github.com/gottfrois/link_thumbnailer/wiki/How-to-build-your-own-Grader%3F)
+
+# 2.6.1
+
+- Fix remove useless dependency
+
+# 2.6.0
+
+- Introduce new `raise_on_invalid_format` option (false by default) to raise `LinkThumbnailer::FormatNotSupported` if http `Content-Type` is invalid. Fixes #61 and #64.
+
+# 2.5.2
+
+- Fix OpenURI::HTTPError exception raised when video_info gem is not able to parse video metadata. Fixes #60.
+
+# 2.5.1
+
+- Implement `Set-Cookie` header between http redirections to set cookies when site requires it. Fixes #55.
+
+# 2.5.0
+
+- Handles seamlessly `og:image` and `og:image:url`
+- Handles seamlessly `og:video` and `og:video:url`
+- Handles `og:video:width` and `og:video:height` for one video only (please create a ticket if you want support for multiple videos/images width & height)
+- Fix calling `as_json` on `website` to return `as_json` representation of videos and images, not just their urls
+- Gem updates and fix rspec deprecation warnings
+
+# 2.4.0
+
+- Handle connection through proxy automatically using the `ENV['HTTP_PROXY']` variable thanks to [taganaka](https://github.com/taganaka).
+
+# 2.3.2
+
+- Fix an issue with vimeo opengraph urls. Fixes [#46](https://github.com/gottfrois/link_thumbnailer/pull/46)
+
+# 2.3.1
+
+- Fix an issue with the link density grader caused by links with image instead of text. Fixes [#45](https://github.com/gottfrois/link_thumbnailer/issues/45)
+
+# 2.3.0
+
+- Add requested favicon scraper [#40](https://github.com/gottfrois/link_thumbnailer/issues/40)
+
+Add `:favicon` to `config.attributes` in LinkThumbnailer initializer:
+
+```ruby
+config.attributes = [:title, :images, :description, :videos, :favicon]
+```
+
+Then
+
+```ruby
+o = LinkThumbnailer.generate('https://github.com')
+o.favicon
+ => "https://github.com/fluidicon.png"
+```
+
+# 2.2.3
+
+- Fixes [#41](https://github.com/gottfrois/link_thumbnailer/issues/41)
+
+# 2.2.2
+
+- Fixes [#41](https://github.com/gottfrois/link_thumbnailer/issues/41)
+
+# 2.2.1
+
+- Fix issue when computing link density ratio
+
+# 2.2.0
+
+- Add support for `og:video`
+- Add support for multiple `og:video` as well
+
+LinkThumbnailer will return the following json for example:
+
+```ruby
+{
+  id:         'x7lni3',
+  src:        'http://www.dailymotion.com/video/x7lni3',
+  size:       [640, 360],
+  duration:   136,
+  provider:   'Dailymotion',
+  embed_code: '<iframe src="//www.dailymotion.com/embed/video/x7lni3" frameborder="0" allowfullscreen="allowfullscreen"></iframe>'
+}
+```
+
+Add `:videos` into your `config/initializers/link_thumbnailer.rb` `attributes` config in order to start scraping videos.
+
+Ex:
+
+```ruby
+config.attributes = [:title, :images, :description, :videos]
+```
+
+# 2.1.0
+
+- Increased `og:image` scraping performance by parsing `og:image:width` and `og:image:height` attribute if specified
+- Introduced `image_stats` option to allow disabling image size and type parsing causing performance issues.
+
+When disabled, size will be `[0, 0]` and type will be `nil`
+
+# 2.0.4
+
+- Fixes [#39](https://github.com/gottfrois/link_thumbnailer/issues/39)
+
+# 2.0.3
+
+- Fixes [#37](https://github.com/gottfrois/link_thumbnailer/issues/37)
+
+# 2.0.2
+
+- Fix couple of issues with `URI` class namespace
+
+# 2.0.1
+
+- Fix issue with image parser (fastimage) when given an URI instance instead of a string
+
+# 2.0.0
+
+- Fully refactored LinkThumbnailer
+- Introduced [Graders](https://github.com/gottfrois/link_thumbnailer/wiki/How-to-build-your-own-Grader%3F)
+- Introduced [Scrapers](https://github.com/gottfrois/link_thumbnailer/wiki/Attributes-option-explained)
+- Ability to score descriptions
+- Ability to fetch multiple `og:image`
+- Fixed memoized run-time options
+- Fixed some website urls not working
+- Refactor ugly code
+- More specs
+- Removed `PreviewsController` since it does not add much value. Simply create your own and use the `to_json` method.
+
+To update from `1.x.x` to `2.x.x` you need to run `rails g link_thumbnailer:install` to get the new configuration file.
+If you used the `PreviewsController` feature, you need to build it yourself since it is not supported anymore.
+
+# 1.1.2
+
+- Fix issue with FastImage URLs [https://github.com/gottfrois/link_thumbnailer/pull/31](https://github.com/gottfrois/link_thumbnailer/pull/31)
+
+# 1.1.1
+
+- Fix route helper not working under rails 4.
+
+# 1.1.0
+
+- Replace RMagick by [FastImage](https://github.com/sdsykes/fastimage)
+- Rename `rmagick_attributes` config into `image_attributes`
+
+# 1.0.9
+
+- Fix issue when Location header used a relative path instead of an absolute path
+- Update gemfile to be more flexible when using Hashie gem
+
+# 1.0.8
+
+- Thanks to [juriglx](https://github.com/juriglx), support for canonical urls
+- Bug fixes
+
+# 1.0.7
+
+- Fix: Issue with preview controller
+
+# 1.0.6
+
+- Fix: Issue when setting `strict` option. Always returning OG representation.
+
+# 1.0.5
+
+- Thanks to [phlegx](https://github.com/phlegx), support for timeout http connection through configurations.
+
+# 1.0.4
+
+- Fix issue #7: nil img was returned when exception is raised. Now skiping nil images in results.
+- Thanks to [phlegx](https://github.com/phlegx), support for SSL and User Agent customization through configurations.
+
+# 1.0.3
+
+- Fix issue #5: Url was incorect in case of HTTP Redirections.
+
+# 1.0.2
+
+- Feature: User can now set options at runtime by passing valid options to ```generate``` method
+- Bug fix when doing ```rails g link_thumbnailer:install``` by explicitly specifying the scope of Rails
+
+# 1.0.1
+
+- Refactor LinkThumbnailer#generate method to have a cleaner code
+
+# 1.0.0
+
+- Update readme
+- Add PreviewController for easy integration with user's app
+- Add link_thumbnailer routes for easy integration with user's app
+- Refactor some code
+- Change 'to_a' method to 'to_hash' in object model
+
+# 0.0.6
+
+- Update readme
+- Add `to_a` to WebImage class
+- Refactor `to_json` for WebImage class
+- Add specs corresponding
+
+# 0.0.5
+
+- Bug fix
+- Remove `require 'rails'` from spec_helper.rb
+- Remove rails dependences (blank? method) in code
+- Spec fix
+
+# 0.0.4
+
+- Add specs for almost all classes
+- Add a method `to_json` for WebImage class to be able to get a usable array of images' attributes
+
+# 0.0.3
+
+- Add specs for LinkThumbnailer class
+- Refactor config system, now using dedicated configuration class
+
+# 0.0.2
+
+- Added Rspec
+- Bug fixes:
+	- Now checking if attribute is blank for LinkThumbnailer::Object.valid? method
+
+# 0.0.1
+
+- LinkThumbnailer::Object
+- LinkThumbnailer::Doc
+- LinkThumbnailer::DocParser
+- LinkThumbnailer::Fetcher
+- LinkThumbnailer::ImgComparator
+- LinkThumbnailer::ImgParser
+- LinkThumbnailer::ImgUrlFilter
+- LinkThumbnailer::Opengraph
+- LinkThumbnailer::WebImage
+- LinkThumbnailer.configure
+- LinkThumbnailer.generate

data/Gemfile

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in link_thumbnailer.gemspec
+gemspec
+
+group :development, :test do
+  gem 'rspec',      '>= 2.14'
+  gem 'webmock',    '>= 1.14'
+  gem 'pry',        '>= 0.9'
+end

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2012 Pierre-Louis Gottfrois
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,210 @@
+# LinkThumbnailer
+
+[![Code Climate](https://codeclimate.com/github/gottfrois/link_thumbnailer.png)](https://codeclimate.com/github/gottfrois/link_thumbnailer)
+[![Build Status](https://travis-ci.org/gottfrois/link_thumbnailer.png?branch=master)](https://travis-ci.org/gottfrois/link_thumbnailer)
+[![Gem Version](https://badge.fury.io/rb/link_thumbnailer.svg)](http://badge.fury.io/rb/link_thumbnailer)
+[![Dependency Status](https://gemnasium.com/gottfrois/link_thumbnailer.svg)](https://gemnasium.com/gottfrois/link_thumbnailer)
+
+Ruby gem generating image thumbnails from a given URL. Rank them and give you back an object containing images and website informations. Works like Facebook link previewer.
+
+Demo Application is [here](http://link-thumbnailer-demo.herokuapp.com/) !
+The source code of the Demo Application is hosted [here](https://github.com/gottfrois/link_thumbnailer_demo)!
+
+**OpenSource** and **Free** API available [here](https://github.com/gottfrois/link_thumbnailer_api) !
+
+## Features
+
+- Dead simple.
+- Support [OpenGraph](http://ogp.me/) protocol.
+- Find and sort images that best represent what the page is about.
+- Find and rate description that best represent what the page is about.
+- Allow for custom class to sort the website descriptions yourself.
+- Support image urls blacklisting (advertisements).
+- Works with and without Rails.
+- Fully customizable.
+- Fully tested.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'link_thumbnailer'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install link_thumbnailer
+
+Run:
+
+	$ rails g link_thumbnailer:install
+
+This will add `link_thumbnailer.rb` to `config/initializers/`.
+
+## Usage
+
+Run `irb` and require the gem:
+
+```ruby
+require 'link_thumbnailer'
+```
+
+The gem handle regular website but also website that use the [Opengraph](http://ogp.me/) protocol.
+
+```ruby
+object = LinkThumbnailer.generate('http://stackoverflow.com')
+ => #<LinkThumbnailer::Models::Website:...>
+
+object.title
+ => "Stack Overflow"
+
+object.favicon
+ => "//cdn.sstatic.net/stackoverflow/img/favicon.ico?v=038622610830"
+
+object.description
+ => "Q&A for professional and enthusiast programmers"
+
+object.images.first.src.to_s
+ => "http://cdn.sstatic.net/stackoverflow/img/apple-touch-icon@2.png?v=fde65a5a78c6"
+```
+
+LinkThumbnailer `generate` method return an instance of `LinkThumbnailer::Models::Website` that respond to `to_json` and `as_json` as you would expect:
+
+```ruby
+object.to_json
+ => "{\"url\":\"http://stackoverflow.com\",\"title\":\"Stack Overflow\",\"description\":\"Q&A for professional and enthusiast programmers\",\"images\":[{\"src\":\"http://cdn.sstatic.net/stackoverflow/img/apple-touch-icon@2.png?v=fde65a5a78c6\",\"size\":[316,316],\"type\":\"png\"}]}"
+```
+
+
+## Configuration
+
+LinkThumbnailer comes with default configuration values. You can change default value by overriding them in a rails initializer:
+
+In `config/initializers/link_thumbnailer.rb`
+
+```ruby
+LinkThumbnailer.configure do |config|
+  # Numbers of redirects before raising an exception when trying to parse given url.
+  #
+  # config.redirect_limit = 3
+
+  # Set user agent
+  #
+  # config.user_agent = 'link_thumbnailer'
+
+  # Enable or disable SSL verification
+  #
+  # config.verify_ssl = true
+
+  # The amount of time in seconds to wait for a connection to be opened.
+  # If the HTTP object cannot open a connection in this many seconds,
+  # it raises a Net::OpenTimeout exception.
+  #
+  # See http://www.ruby-doc.org/stdlib-2.1.1/libdoc/net/http/rdoc/Net/HTTP.html#open_timeout
+  #
+  # config.http_open_timeout = 5
+
+  # List of blacklisted urls you want to skip when searching for images.
+  #
+  # config.blacklist_urls = [
+  #   %r{^http://ad\.doubleclick\.net/},
+  #   %r{^http://b\.scorecardresearch\.com/},
+  #   %r{^http://pixel\.quantserve\.com/},
+  #   %r{^http://s7\.addthis\.com/}
+  # ]
+
+  # List of attributes you want LinkThumbnailer to fetch on a website.
+  #
+  # config.attributes = [:title, :images, :description, :videos, :favicon]
+
+  # List of procedures used to rate the website description. Add you custom class
+  # here. See wiki for more details on how to build your own graders.
+  #
+  # config.graders = [
+  #   ->(description) { ::LinkThumbnailer::Graders::Length.new(description) },
+  #   ->(description) { ::LinkThumbnailer::Graders::HtmlAttribute.new(description, :class) },
+  #   ->(description) { ::LinkThumbnailer::Graders::HtmlAttribute.new(description, :id) },
+  #   ->(description) { ::LinkThumbnailer::Graders::Position.new(description, weight: 3) },
+  #   ->(description) { ::LinkThumbnailer::Graders::LinkDensity.new(description) }
+  # ]
+
+  # Minimum description length for a website.
+  #
+  # config.description_min_length = 25
+
+  # Regex of words considered positive to rate website description.
+  #
+  # config.positive_regex = /article|body|content|entry|hentry|main|page|pagination|post|text|blog|story/i
+
+  # Regex of words considered negative to rate website description.
+  #
+  # config.negative_regex = /combx|comment|com-|contact|foot|footer|footnote|masthead|media|meta|outbrain|promo|related|scroll|shoutbox|sidebar|sponsor|shopping|tags|tool|widget|modal/i
+
+  # Numbers of images to fetch. Fetching too many images will be slow.
+  # Note that LinkThumbnailer will only sort fetched images between each other.
+  # Meaning that they could be a "better" image on the page.
+  #
+  # config.image_limit = 5
+
+  # Whether you want LinkThumbnailer to return image size and type or not.
+  # Setting this value to false will increase performance since for each images, LinkThumbnailer
+  # does not have to fetch its size and type.
+  #
+  # config.image_stats = true
+  #
+  # Whether you want LinkThumbnailer to raise an exception if the Content-Type of the HTTP request
+  # is not an html or xml.
+  #
+  # config.raise_on_invalid_format = false
+  #
+  # Sets number of concurrent http connections that can be opened to fetch images informations such as size and type.
+  #
+  # config.max_concurrency = 20
+
+  # Sets the default encoding.
+  #
+  # config.encoding = 'utf-8'
+end
+```
+
+Or at runtime:
+
+```ruby
+object = LinkThumbnailer.generate('http://stackoverflow.com', redirect_limit: 5, user_agent: 'foo')
+```
+
+Note that runtime options will override default global configuration.
+
+See [Configuration Options Explained](https://github.com/gottfrois/link_thumbnailer/wiki/Configuration-options-explained) for more details on each configuration options.
+
+## Exceptions
+
+LinkThumbnailer defines a list of custom exceptions you may want to rescue in your code. All the following exceptions inherit from `LinkThumbnailer::Exceptions`:
+
+* `RedirectLimit` -- raised when redirection threshold defined in config is reached
+* `BadUriFormat` -- raised when url given is not a valid HTTP url
+* `FormatNotSupported` -- raised when the `Content-Type` of the HTTP request is not supported (not `html`)
+
+You can rescue from any LinkThumbnailer exceptions using the following code:
+
+```ruby
+begin
+  LinkThumbnailer.generate('http://foo.com')
+rescue LinkThumbnailer::Exceptions => e
+  # do something
+end
+```
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Run the specs (`bundle exec rspec spec`)
+4. Commit your changes (`git commit -am 'Added some feature'`)
+5. Push to the branch (`git push origin my-new-feature`)
+6. Create new Pull Request