directlink 0.0.3.1 → 0.0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: d421a30c60a09ebaf198aeb77679a78a61a8521c
-  data.tar.gz: 7acbb0d68a9e150da049bd50afd0bd5a64e61fe1
+  metadata.gz: 31d4684edbeadeddce540e3ca0a1894f5756af86
+  data.tar.gz: f1efd18489faf49d2bc7460f210b53cf7cb0929c
 SHA512:
-  metadata.gz: e1e5a70b474a6261f113304aee2639e10183dc5150017e68f68974b72d3ae4b9b8d41b3a5243883fc20214db6ae51d624a76039b548ccf33a07fe8911f5d1463
-  data.tar.gz: e7ece452fbf63d76a9d155dd53e05d4e702e2f5c9d825a1562206f3ff727ca1e85e910a15814ba646f59cc600f76284206604d91c929e03efccfe5f372496d70
+  metadata.gz: bef3ef81ba91007a8b26d6f45813d67319bd00b3c3bfbc71977d2caa4c40e29f31e8f35a32d2f7af8c1c8bcd8d7a267ef3998bc51912a997462a864f2cf563e0
+  data.tar.gz: 18f45737947862ebc5409b3a20be5d2389aa7e9f85125bd85465c670c7ab488b43ab55c599ac79bdfde7063125b5a891a42336f9c2348ffcd5807707dc908d44

data/README.md

@@ -3,7 +3,7 @@
 
 # gem directlink
 
-This tool converts any sort of image hyperlink (a thumbnail URL, a link to an album, etc.) to a high resolution one. Also it tells the resulting resolution and the image type (format). I wanted such automation often so I made a gem with a binary.
+This tool obtains from any sort of hyperlink (a thumbnail URL, a link to a photo album, a news article, etc.) a directlink(s) to high resolution images at that page. Also it tells the resulting resolution and the image type (format). The gem also includes a binary so you can use it as a CLI.
 
 ## Usage
 
@@ -16,7 +16,7 @@ $ gem install directlink
 $ directlink
 usage: directlink [--debug] [--json] [--github] <link1> <link2> <link3> ...
 ```
-Converts `<img src=` attribute value from any Google web service (current Google regexes are very strict and may often fail -- it is a [defensive programming](https://en.wikipedia.org/wiki/Defensive_programming) practice -- report me your links!) to the largest available:
+Converts `<img src=` attribute value from any Google web service to the largest available:
 ```
 $ directlink //4.bp.blogspot.com/-5kP8ndL0kuM/Wpt82UCqvmI/AAAAAAAAEjI/ZbbZWs0-kgwRXEJ9JEGioR0bm6U8MOkvQCKgBGAs/w530-h278-p/IMG_20171223_093922.jpg
 <= //4.bp.blogspot.com/-5kP8ndL0kuM/Wpt82UCqvmI/AAAAAAAAEjI/ZbbZWs0-kgwRXEJ9JEGioR0bm6U8MOkvQCKgBGAs/w530-h278-p/IMG_20171223_093922.jpg
@@ -77,16 +77,30 @@ $ directlink --json https://imgur.com/a/oacI3gl https://avatars1.githubuserconte
   }
 ]
 ```
-Downloads master:HEAD version of `lib/directlink.rb` from GitHub and uses it once instead of installed one (this is easier than installing gem from repo):
+Downloads `master:HEAD` version of `lib/directlink.rb` from GitHub and uses it once instead of installed one:
 ```
 $ directlink --github https://imgur.com/a/oacI3gl
 ```
-When an image hosting with known API is recognized, the API will be used and you'll have to create app there and provide env vars:
+When an image hosting with known API is recognized, it will try to use the API tokens you've provided as env vars (otherwise it will go "don't give up" mode):
 ```
 $ export IMGUR_CLIENT_ID=0f99cd781...
 $ export FLICKR_API_KEY=dc2bfd348b...
 ```
 
+#### the "don't give up mode"
+
+If the passed link is not the image link or a photo page of a known image hosting, the tool is still able to find the main images that the linked webpage contains (here it found three images in the markdown file):
+```
+$ bundle exec bin/directlink https://github.com/Nakilon/dhash-vips
+<= https://github.com/Nakilon/dhash-vips
+=> https://camo.githubusercontent.com/852607c7f4b604fc3c83b782c4f6983cf488b0d4/68747470733a2f2f73746f726167652e676f6f676c65617069732e636f6d2f64686173682d766970732e6e616b696c6f6e2e70726f2f64686173685f69737375655f6578616d706c652e706e67
+   png 592x366
+=> https://camo.githubusercontent.com/5e354666bac69e32d605dbd45351bfb7d808924b/68747470733a2f2f73746f726167652e676f6f676c65617069732e636f6d2f64686173682d766970732e6e616b696c6f6e2e70726f2f6964686173685f6578616d706c655f696e2e706e67
+   png 773x679
+=> https://camo.githubusercontent.com/5456cc20ae9b20c06792ddd19b533ae36404d8c1/68747470733a2f2f73746f726167652e676f6f676c65617069732e636f6d2f64686173682d766970732e6e616b696c6f6e2e70726f2f6964686173685f6578616d706c655f6f75742e706e67
+   png 1610x800
+```
+
 ### As a library
 
 ```
@@ -110,11 +124,16 @@ Google can serve image in arbitrary resolution so `DirectLink.google` has an opt
 irb> DirectLink.google "//4.bp.blogspot.com/-5kP8ndL0kuM/Wpt82UCqvmI/AAAAAAAAEjI/ZbbZWs0-kgwRXEJ9JEGioR0bm6U8MOkvQCKgBGAs/w530-h278-p/IMG_20171223_093922.jpg", 100
 => "https://4.bp.blogspot.com/-5kP8ndL0kuM/Wpt82UCqvmI/AAAAAAAAEjI/ZbbZWs0-kgwRXEJ9JEGioR0bm6U8MOkvQCKgBGAs/s100/IMG_20171223_093922.jpg"
 ```
-To silent the logger that `DirectLink.imgur` uses:
+To disable the "don't give up" mode (otherwise it consumes time on analyzing all the images on the linked page):
+```
+irb> DirectLink "https://github.com/Nakilon/dhash-vips", nil, true
+# raises FastImage::UnknownImageType
+```
+To silent the STDOUT logger that you may see sometimes:
 ```ruby
 DirectLink.silent = true
 ```
-You also may look into [`bin/directlink`](bin/directlink) for usage example and the list of all possible exceptions.
+You also may look into [`bin/directlink`](bin/directlink) as a library usage example and the list of all possible exceptions.
 
 #### about long retries
 
@@ -125,7 +144,7 @@ NetHTTPUtils.logger.level = Logger::WARN
 ```
 W 180507 102210 : NetHTTPUtils : retrying in 10 seconds because of SocketError 'Failed to open TCP connection to minus.com:80 (getaddrinfo: nodename nor servname provided, or not known)' at: http://minus.com/
 ```
-To make `DirectLink()` respond faster pass an optional argument that specifies the max retry delay as any numeric value. Here we get the exception immediately:
+To make `DirectLink()` respond faster pass an optional argument that specifies the max retry delay. Here we get the exception immediately:
 ```ruby
 DirectLink "http://minus.com/", 0
 ```
@@ -136,7 +155,7 @@ SocketError: Failed to open TCP connection to minus.com:80 (getaddrinfo: nodenam
 ## Notes:
 
 * `module DirectLink` public methods return different sets of properties -- `DirectLink()` unites them
-* the `DirectLink::ErrorAssert` should never happen and you might report it if it does
+* the `ErrorAssert` and `ErrorMissingEnvVar` should never be raised and you might report it if it does
 * style: `@@` and lambdas are used to keep things private
-* this gem is a 2 or 3 libraries merged so don't expect tests to be full and consistent
-* since 500px.com closed their API in June 2018 the gem uses potentially unreliable undocumented methods
+* this gem is a historically 2 or 3 libraries merged -- this is why tests may look awkward
+* 500px.com has discontinued API in June 2018 -- the tool now uses undocumented methods

data/api_tokens_for_travis.sh

@@ -1,3 +1,2 @@
 export IMGUR_CLIENT_ID=0f99cd781c9d0d8
 export FLICKR_API_KEY=dc2bfd348b01bdc5b09d36876dc38f3d
-export _500PX_CONSUMER_KEY=ESkHTUELdcE48bezGfwzSjqVIBVTnNRIPTviTGLv

data/bin/directlink

@@ -72,7 +72,7 @@ rescue NetHTTPUtils::Error,
        SocketError,
        FastImage::UnknownImageType,
        FastImage::ImageFetchFailure,
-       DirectLink::ErrorMissingEnvVar,
+       # DirectLink::ErrorMissingEnvVar,
        # DirectLink::ErrorAssert,
        DirectLink::ErrorNotFound,
        DirectLink::ErrorBadLink => e

data/directlink.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |spec|
   spec.name         = "directlink"
-  spec.version      = "0.0.3.1"
+  spec.version      = "0.0.4.0"
   spec.summary      = "converts any kind of image hyperlink to direct link, type of image and its resolution"
 
   spec.author       = "Victor Maslov aka Nakilon"
@@ -11,6 +11,7 @@ Gem::Specification.new do |spec|
 
   spec.add_dependency "nethttputils", "~>0.2.4.0"
   spec.add_dependency "fastimage", "~>2.1.3"
+  spec.add_dependency "nokogiri"
   spec.add_development_dependency "minitest"
 
   spec.require_path = "lib"

data/lib/directlink.rb

@@ -12,24 +12,28 @@ module DirectLink
     puts str unless Module.nesting.first.silent
   end
 
-  class ErrorMissingEnvVar < RuntimeError ; end
   class ErrorAssert < RuntimeError
     def initialize msg
       super "#{msg} -- consider reporting this issue to GitHub"
     end
   end
-  @@LoggingError = Class.new RuntimeError do
+  logging_error = Class.new RuntimeError do
     def initialize msg
       Module.nesting.first.logger.error msg
       super msg
     end
   end
-  class ErrorNotFound < @@LoggingError ; end
-  class ErrorBadLink < @@LoggingError
+  class ErrorNotFound < logging_error ; end
+  class ErrorBadLink < logging_error
     def initialize link, sure = false
       super "#{link.inspect}#{" -- if you think this link is valid, please report the issue" unless sure}"
     end
   end
+  class ErrorMissingEnvVar < logging_error
+    def initialize msg
+      super "(warning, recommendation) #{msg}"
+    end
+  end
 
 
   def self.google src, width = 0
@@ -202,7 +206,7 @@ end
 
 require "fastimage"
 
-def DirectLink link, max_redirect_resolving_retry_delay = nil
+def DirectLink link, max_redirect_resolving_retry_delay = nil, giveup = false
   begin
     URI link
   rescue URI::InvalidURIError
@@ -234,28 +238,42 @@ def DirectLink link, max_redirect_resolving_retry_delay = nil
   } : {})
   raise NetHTTPUtils::Error.new "", r.code.to_i unless "200" == r.code
   link = r.uri.to_s
+  # why do we resolve redirects before trying the known adapters?
+  #   because they can be hidden behind URL shorteners
+  #   also it can resolve NetHTTPUtils::Error(404) before trying the adapter
+
+  # TODO: get rid of this copypasta, that is caused by that we want to pass urls without schema to this method
+  if %w{ lh3 googleusercontent com } == URI(link).host.split(?.).last(3) ||
+     %w{ bp blogspot com } == URI(link).host.split(?.).last(3)
+    u = DirectLink.google link
+    f = FastImage.new(u, raise_on_failure: true, http_header: {"User-Agent" => "Mozilla"})
+    w, h = f.size
+    return struct.new u, w, h, f.type
+  end
 
-  if %w{ imgur com } == URI(link).host.split(?.).last(2) ||
-     %w{ i imgur com } == URI(link).host.split(?.).last(3) ||
-     %w{ m imgur com } == URI(link).host.split(?.).last(3) ||
-     %w{ www imgur com } == URI(link).host.split(?.).last(3)
+  begin
     imgur = DirectLink.imgur(link).sort_by{ |u, w, h, t| - w * h }.map do |u, w, h, t|
       struct.new u, w, h, t
     end
     # `DirectLink.imgur` return value is always an Array
     return imgur.size == 1 ? imgur.first : imgur
-  end
+  rescue DirectLink::ErrorMissingEnvVar
+  end if %w{ imgur com } == URI(link).host.split(?.).last(2) ||
+         %w{ i imgur com } == URI(link).host.split(?.).last(3) ||
+         %w{ m imgur com } == URI(link).host.split(?.).last(3) ||
+         %w{ www imgur com } == URI(link).host.split(?.).last(3)
 
   if %w{ 500px com } == URI(link).host.split(?.).last(2)
     w, h, u, t = DirectLink._500px(link)
     return struct.new u, w, h, t
   end
 
-  if %w{ www flickr com } == URI(link).host.split(?.).last(3)
+  begin
     w, h, u = DirectLink.flickr(link)
     f = FastImage.new(u, raise_on_failure: true, http_header: {"User-Agent" => "Mozilla"})
     return struct.new u, w, h, f.type
-  end
+  rescue DirectLink::ErrorMissingEnvVar
+  end if %w{ www flickr com } == URI(link).host.split(?.).last(3)
 
   if %w{ wikipedia org } == URI(link).host.split(?.).last(2) ||
      %w{ commons wikimedia org } == URI(link).host.split(?.).last(3)
@@ -265,7 +283,33 @@ def DirectLink link, max_redirect_resolving_retry_delay = nil
     return struct.new u, w, h, f.type
   end
 
-  f = FastImage.new(link, raise_on_failure: true, http_header: {"User-Agent" => "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36"})
-  w, h = f.size
-  struct.new link, w, h, f.type
+  begin
+    f = FastImage.new(link, raise_on_failure: true, http_header: {"User-Agent" => "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36"})
+  rescue FastImage::UnknownImageType
+    raise if giveup
+    require "nokogiri"
+    html = Nokogiri::HTML NetHTTPUtils::request_data link
+    h = {}
+    l = lambda do |node, s = []|
+      node.element_children.flat_map do |child|
+        if "img" == child.node_name
+          begin
+            [[s, (h[child[:src]] = h[child[:src]] || DirectLink(child[:src], nil, true))]]
+          rescue => e
+            []
+          end
+        else
+          l[child, s + [child.node_name]]
+        end
+      end
+    end
+    l[html].group_by(&:first).map{ |k, v| [k.join(?>), v.map(&:last)] }.tap do |result|
+      next unless result.empty?
+      raise unless t = html.at_css "meta[@property='og:image']"
+      return DirectLink t[:content], nil, true
+    end.max_by{ |_, v| v.map{ |i| i.width * i.height }.inject(:+) / v.size }.last
+  else
+    w, h = f.size
+    struct.new link, w, h, f.type
+  end
 end

data/test.rb

@@ -1,4 +1,5 @@
 STDOUT.sync = true
+require "pp"
 
 require "minitest/autorun"
 require "minitest/mock"
@@ -6,6 +7,9 @@ require "minitest/mock"
 # TODO: I'm not sure it's ok that after we started using NetHTTPUtils for redirect resolving
 #       we don't raise `FastImage::ImageFetchFailure` anymore in any test
 
+fail unless ENV.include? "IMGUR_CLIENT_ID"
+fail unless ENV.include? "FLICKR_API_KEY"
+
 require_relative "lib/directlink"
 DirectLink.silent = true
 describe DirectLink do
@@ -408,7 +412,6 @@ describe DirectLink do
     describe "some other tests" do
       [
         ["http://www.aeronautica.difesa.it/organizzazione/REPARTI/divolo/PublishingImages/6%C2%B0%20Stormo/2013-decollo%20al%20tramonto%20REX%201280.jpg", ["http://www.aeronautica.difesa.it/organizzazione/REPARTI/divolo/PublishingImages/6%C2%B0%20Stormo/2013-decollo%20al%20tramonto%20REX%201280.jpg", 1280, 853, :jpeg]],
-        ["http://example.com", FastImage::UnknownImageType, "FastImage::UnknownImageType"],        # we explicitly expect this useless `e.message ` to be sure we know how FastImage behaves
         ["http://minus.com/lkP3hgRJd9npi", SocketError, /nodename nor servname provided, or not known|No address associated with hostname/, 0],
         ["https://i.redd.it/si758zk7r5xz.jpg", NetHTTPUtils::Error, "HTTP error #404 "],
         ["http://www.cutehalloweencostumeideas.org/wp-content/uploads/2017/10/Niagara-Falls_04.jpg", SocketError, /nodename nor servname provided, or not known|Name or service not known/, 0],
@@ -432,6 +435,39 @@ describe DirectLink do
       end
     end
 
+    describe "giving up" do
+      [
+        ["http://example.com",                    FastImage::UnknownImageType],
+        ["https://github.com/Nakilon/dhash-vips", FastImage::UnknownImageType, true],
+        ["https://github.com/Nakilon/dhash-vips", 3],
+        ["http://imgur.com/HQHBBBD",              FastImage::UnknownImageType, true],
+        ["http://imgur.com/HQHBBBD",              "https://i.imgur.com/HQHBBBD.jpg?fb"],
+      ].each_with_index do |(input, expectation, giveup), i|
+        it "##{i + 1}" do
+          t = ENV.delete "IMGUR_CLIENT_ID"
+          begin
+            case expectation
+            when Class
+              e = assert_raises expectation, "for #{input} (giveup = #{giveup})" do
+                DirectLink input, nil, giveup
+              end
+              assert_equal expectation.to_s, e.message, "for #{input} (giveup = #{giveup})"
+            when String
+              result = DirectLink input, nil, giveup
+              assert_equal expectation, result.url, "for #{input} (giveup = #{giveup})"
+            else
+              result = DirectLink input, nil, giveup
+              assert_equal expectation, result.size, ->{
+                "for #{input} (giveup = #{giveup}): #{result.map &:url}"
+              }
+            end
+          ensure
+            ENV["IMGUR_CLIENT_ID"] = t
+          end
+        end
+      end
+    end
+
   end
 
   describe "./bin" do
@@ -468,9 +504,13 @@ describe DirectLink do
       [
         [1, "http://example.com/", "FastImage::UnknownImageType"],
         [1, "http://example.com/404", "NetHTTPUtils::Error: HTTP error #404 "],
-        [1, "http://imgur.com/HQHBBBD", "DirectLink::ErrorMissingEnvVar: define IMGUR_CLIENT_ID env var", " && unset IMGUR_CLIENT_ID"],  # TODO: make similar test for ./lib
+
+        # TODO: a test when the giveup=false fails and reraises the DirectLink::ErrorMissingEnvVar
+        #       maybe put it to ./lib tests
+
         # by design it should be impossible to write a test for DirectLink::ErrorAssert
         [1, "https://flic.kr/p/DirectLinkErrorNotFound", "NetHTTPUtils::Error: HTTP error #404 "],
+
         [1, "https://imgur.com/a/badlinkpattern", "NetHTTPUtils::Error: HTTP error #404 "],
         # TODO: a test that it appends the `exception.cause`
       ].each_with_index do |(expected_exit_code, link, expected_output, unset), i|

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: directlink
 version: !ruby/object:Gem::Version
-  version: 0.0.3.1
+  version: 0.0.4.0
 platform: ruby
 authors:
 - Victor Maslov aka Nakilon
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-08-10 00:00:00.000000000 Z
+date: 2018-08-12 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: nethttputils
@@ -38,6 +38,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: 2.1.3
+- !ruby/object:Gem::Dependency
+  name: nokogiri
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: minitest
   requirement: !ruby/object:Gem::Requirement