google 1.0.3 → 1.0.4

data/LICENSE.md

@@ -0,0 +1,10 @@
+[The MIT License](http://www.opensource.org/licenses/mit-license.php)
+=====================================================================
+
+Copyright (c) 2012 Kerrick Long
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,87 @@
+google
+======
+
+## The power of Google Search in your command line.
+
+The CLI displays results from the [Google Web Search API](https://developers.google.com/web-search/), allows you to page through the results, and choose a page to view, all without leaving the command line.
+
+[Hosted on RubyGems](https://rubygems.org/gems/google).
+
+## Installation
+
+    gem install google # Requires Ruby :)
+
+## Usage Examples
+
+Typing `google --help` will list all the available commands. They are also listed here.
+
+    The google gem is a simple tool to search Gooogle with via a CLI.
+    Usage:
+        google [options] "my search query string here"
+    where [options] are:
+            --page, -p <i>:   Start by showing the <i>th result page. (Default: 1)
+            --size, -s <i>:   Show <i> results on each SERP. Must be between 1 and 8. (Default: 4)
+          --result, -r <i>:   Skip the SERP and show the <i>th result.
+               --lucky, -l:   I'm feeling lucky! Skip the SERP and show the first result. (Alias to --result 1)
+      --no-readability, -e:   Filter the results through readability to get rid of extra content. (Default: true)
+         --no-markdown, -m:   Change the results from raw HTML to markdown. (Default: true)
+             --version, -v:   Print the version and exit.
+                --help, -h:   Show this information and exit.
+
+### I'm Feeling Lucky into Less
+
+    google -l 'how to install nvidia drivers debian squeeze' | less
+
+### Search on a single site
+
+    google 'site:randsinrepose.com NADD'
+
+### Vanity Search
+
+    google '"Kerrick Long" -inurl:kerrick -inurl:kerricklong'
+
+## Features
+
+* You have access to all the [search operators](http://support.google.com/websearch/bin/answer.py?hl=en&answer=136861) you've come to know and love in Google Search.
+
+* If using the `--lucky` or `--result` flags, you can pipe the results into any other unix command.
+
+* Results are filtered through [Readability](http://www.readability.com/), so you only get the relevant content. (Shoutout to [Ruby Readability](https://github.com/iterationlabs/ruby-readability)!)
+
+* Results are shown in [markdown](http://daringfireball.net/projects/markdown/) for a good balance between legibility and document heirarchy. (Shoutout to [reverse_markdown](https://github.com/xijo/reverse_markdown)!)
+
+* Results pages are formatted to look like a Google SERP, including colors, domains, descriptions, and bold search matches. (Shoutout to [formatador](https://github.com/geemus/formatador)!)
+
+* You can pipe a result into another unix command! At the prompt, type the number of the result, followed by a pipe as normal. Note that utilities such as `less` and `more` that need to control the display don't work.
+
+    \> 3 | espeak -a 200 -v en-us
+
+## Supported Ruby Versions
+
+This library has only been tested against Ruby 1.9.3. Other versions might be supported, but they haven't been tested.
+
+## Acknowledgements
+
+This gem is [Powered by Google](http://www.google.com).
+
+### Copyright
+
+Copyright (c) 2012 Kerrick Long. See [LICENSE](https://github.com/Kerrick/google/blob/master/LICENSE.md) for details.
+
+### Dependencies
+
+* [Ruby Readability](https://github.com/iterationlabs/ruby-readability) (gem)
+
+    * [Nokogiri](http://nokogiri.org/)
+
+    * [guess_html_encoding](https://github.com/iterationlabs/guess_html_encoding)
+
+* [JSON Implementation for Ruby](http://flori.github.com/json/) (gem)
+
+* [Formatador](https://github.com/geemus/formatador) (gem)
+
+* [HTML Entities for Ruby](http://htmlentities.rubyforge.org/) (gem)
+
+* [Reverse Markdown](https://github.com/xijo/reverse_markdown) (lib)
+
+* [Trollop](http://trollop.rubyforge.org/) (lib)

data/google.gemspec

@@ -0,0 +1,19 @@
+Gem::Specification.new do |s|
+  s.name = "google"
+  s.version = "1.0.4"
+  s.executables << 'google'
+  s.add_runtime_dependency "json", ["~> 1"]
+  s.add_runtime_dependency "htmlentities", ["~> 4"]
+  s.add_runtime_dependency "formatador", ["~> 0.2"]
+  s.add_runtime_dependency "ruby-readability", ["~> 0.5"]
+  s.date = "2012-05-23"
+  s.summary = "Google Search on the command line"
+  s.description = "A ruby gem to give you the power of Google Search in your command line."
+  s.authors = ["Kerrick Long"]
+  s.email = "me@kerricklong.com"
+  s.files = ["lib/google.rb"]
+  s.files += %w(LICENSE.md README.md google.gemspec)
+  s.files += Dir.glob("lib/**/*.rb")
+  s.files += Dir.glob("bin/**/*")
+  s.homepage = "http://kerrick.github.com/google/"
+end

data/lib/google/display_serp.rb

@@ -0,0 +1,53 @@
+require 'formatador'
+require 'htmlentities'
+
+class Google
+  def display_serp info
+    results           = info[:results]
+    query_strings     = info[:query_strings]
+    coder             = HTMLEntities.new
+    current_page      = results['responseData']['cursor']['currentPageIndex']+1
+    max_result        = query_strings[:start] + query_strings[:rsz]
+    estimated_results = results['responseData']['cursor']['resultCount']
+    result_array      = results['responseData']['results']
+
+    attribution  = "\n"
+    attribution << ' ' * (max_result.to_s.length + 2)
+    attribution << "[yellow]Powered by Google[/]"
+    Formatador.display_line attribution
+
+    result_array.each_with_index do | result, i |
+      this_num = (i + query_strings[:start] + 1).to_s
+
+      serp_title  = "\n#{' ' * (max_result.to_s.length - this_num.length)}"
+      serp_title << "[bold][blue]#{this_num}. "
+      serp_title << "[normal]#{result["titleNoFormatting"]}[/]\n"
+      serp_url    = ' ' * max_result.to_s.length
+      serp_url   << "[green]#{result["url"]}[/]\n"
+      serp_desc   = ' ' * max_result.to_s.length
+      serp_desc  << result["content"].gsub(/<b>/, "[bold]")
+                    .gsub(/<\/b>/, "[/]").squeeze(" ")
+
+      Formatador.display_line coder.decode(
+        Utils::wrap(serp_title, :prefix => max_result.to_s.length + 2)
+      )
+      Formatador.display_line coder.decode(
+        Utils::wrap(serp_url, :prefix => max_result.to_s.length + 2)
+      )
+      Formatador.display_line coder.decode(
+        Utils::wrap(serp_desc, :prefix => max_result.to_s.length + 2)
+      )
+    end
+
+    metadata  = ''
+    metadata << "\n#{' ' * (max_result.to_s.length + 2)}"
+    metadata << "[yellow]Displaying results "
+    metadata << "#{query_strings[:start] + 1} through "
+    metadata << "#{max_result} of "
+    metadata << "#{estimated_results} "
+    metadata << "(Page #{current_page})"
+    Formatador.display_line metadata
+
+    input info, result_array
+  end
+end

data/lib/google/grab.rb

@@ -0,0 +1,26 @@
+require 'open-uri'
+require 'ruby-readability'
+require_relative '../reverse-markdown/reverse_markdown'
+
+class Google
+  def grab url
+    source = open(url).read
+
+    if @opts[:readability]
+      content = Readability::Document.new(source,
+           :tags => %w[div p a pre code h1 h2 h3 h4 h5 h6 blockquote ul ol li],
+           :attributes => %w[href],
+           :remote_empty_nodes => true).content
+    else
+      content = source
+    end
+
+    if @opts[:markdown]
+      output = ReverseMarkdown.new.parse_string(content)
+    else
+      output = content
+    end
+
+    output
+  end
+end

data/lib/google/input.rb

@@ -0,0 +1,51 @@
+require 'formatador'
+
+class Google
+  def input info, result_array
+    prompt  = "\n[yellow]"
+    prompt << "Enter N or P for Next or Previous page, E or Q to quit, "
+    prompt << "or a number to see that result."
+
+    Formatador.display Utils::wrap(prompt)
+
+    Formatador.display "\n[bold]>[/] "
+    choice = STDIN.gets.chomp + ' '
+
+    case
+    when choice[0].downcase == 'e' || choice[0].downcase == 'q'
+      exit
+    when choice[0].downcase == 'n'
+      results = request :q => @query,
+                        :rsz => @opts[:size],
+                        :start => info[:query_strings][:start] + @opts[:size]
+      display_serp results
+    when choice[0].downcase == 'p'
+      if info[:query_strings][:start] < 1
+        Formatador.display Utils::wrap("[yellow]! Already at page one.")
+        input info
+      else
+        results = request :q => @query,
+                          :rsz => @opts[:size],
+                          :start => info[:query_strings][:start] - @opts[:size]
+        display_serp results
+      end
+    when choice[0].match(/\d/)
+      /(\d+)(\s*\|(.*))*/.match(choice) do | str |
+        num = str[1].to_i - 1
+        if info[:query_strings][:start] <= num &&
+           info[:query_strings][:start] + @opts[:size] > num
+          if str[3].nil?
+            view result_array[num]['url']
+          else
+            pipe str[3].strip, result_array[num]['url']
+          end
+        else
+          Formatador.display Utils::wrap("[yellow]! Result not on this page.")
+          input info, result_array
+        end
+      end
+    else
+      input info
+    end
+  end
+end

data/lib/google/pipe-view.rb

@@ -0,0 +1,20 @@
+class Google
+  def pipe command, url
+    # puts "You want to pipe the data."
+    # puts "URL: #{url}"
+    # puts "Command: #{command}"
+    text = grab(url)
+
+    IO.popen(command, 'w+') do | process |
+      process.write(text)
+      process.close_write
+      puts process.read
+    end
+  end
+
+  def view url
+    # puts "You want to view the data."
+    # puts "URL: #{url}"
+    Formatador.display Utils::wrap(grab(url))
+  end
+end

data/lib/google/request.rb

@@ -0,0 +1,20 @@
+require 'open-uri'
+require 'json'
+
+class Google
+  def request query_strings
+    @api_url = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0"
+    @api_url << "&rsz=#{query_strings[:rsz]}"
+    @api_url << "&start=#{query_strings[:start]}"
+    @api_url << "&q=#{query_strings[:q]}"
+
+    results = JSON.parse(open(@api_url).read)
+
+    if results['responseStatus'].to_i != 200
+      Trollop::die Utils::wrap("Google Search API Status " +
+                   "#{results['responseStatus']}. Details:\n" +
+                   "#{results['responseDetails']}\nTry again in a moment")
+    end
+    {results: results, query_strings: query_strings}
+  end
+end

data/lib/google/search.rb

@@ -0,0 +1,26 @@
+require 'uri'
+
+class Google
+
+  def initialize query, opts
+    @query = URI.escape(query)
+    @opts = opts
+
+    @opts[:size] = 8 if opts[:size] > 7
+    @opts[:size] = 1 if opts[:size] <= 1
+  end
+
+  def search
+    if @opts[:result]
+      results = request :q => @query,
+                        :rsz => 1,
+                        :start => @opts[:result]
+      view results[:results]['responseData']['results'][0]['url']
+    else
+      results = request :q => @query,
+                        :rsz => @opts[:size],
+                        :start => ((@opts[:page] - 1) * @opts[:size])
+      display_serp results
+    end
+  end
+end

data/lib/google/utils.rb

@@ -0,0 +1,51 @@
+class Utils
+  # From the trollop rubygem
+  # File lib/trollop.rb, line 644
+  def self.wrap_line str, opts={}
+    prefix = opts[:prefix] || 0
+    width = opts[:width] || (self.width - 1)
+    start = 0
+    ret = []
+    until start > str.length
+      nextt =
+        if start + width >= str.length
+          str.length
+        else
+          x = str.rindex(/\s/, start + width)
+          x = str.index(/\s/, start) if x && x < start
+          x || str.length
+        end
+      ret << (ret.empty? ? "" : " " * prefix) + str[start ... nextt]
+      start = nextt + 1
+    end
+    ret
+  end
+
+  # From the trollop rubygem
+  # File lib/trollop.rb, line 505
+  def self.wrap str, opts={} # :nodoc:
+    if str == ""
+      [""]
+    else
+      str.split("\n").map { |s| self.wrap_line s, opts }.flatten(1).join("\n")
+    end
+  end
+
+  # From the trollop rubygem
+  # File lib/trollop.rb, line 489
+  def self.width #:nodoc:
+    @@width ||= if $stdout.tty?
+      begin
+        require 'curses'
+        Curses::init_screen
+        x = Curses::cols
+        Curses::close_screen
+        x
+      rescue Exception
+        80
+      end
+    else
+      80
+    end
+  end
+end

data/lib/google.rb

@@ -1,11 +1,11 @@
-require 'trollop'
-require 'google/utils'
-require 'google/search'
-require 'google/request'
-require 'google/display_serp'
-require 'google/input'
-require 'google/grab'
-require 'google/pipe-view'
+require_relative 'trollop/lib/trollop'
+require_relative 'google/utils'
+require_relative 'google/search'
+require_relative 'google/request'
+require_relative 'google/display_serp'
+require_relative 'google/input'
+require_relative 'google/grab'
+require_relative 'google/pipe-view'
 
 opts = Trollop::options do
   version "google v1.0.3 (c) 2012 Kerrick Long http://kerrick.github.com/google"

data/lib/reverse-markdown/reverse_markdown.rb

@@ -0,0 +1,297 @@
+require 'rexml/document'
+require 'benchmark'
+include REXML
+include Benchmark
+
+# reverse markdown for ruby
+# author: JO
+# e-mail: xijo@gmx.de
+# date: 14.7.2009
+# version: 0.1
+# license: GPL
+# taken from https://github.com/xijo/reverse-markdown/raw/master/reverse_markdown.rb
+
+# TODO
+# - ol numbering is buggy, in fact doesn't matter for markdown code
+# -
+
+class ReverseMarkdown
+
+  # set basic variables:
+  # - @li_counter: numbering list item (li) tags in an ordered list (ol)
+  # - @links:      hold the links for adding them to the bottom of the @output
+  #                this means 'reference style', please take a look at http://daringfireball.net/projects/markdown/syntax#link
+  # - @outout:     fancy markdown code in here!
+  # - @indent:     control indention level for nested lists
+  # - @errors:     appearing errors, like unknown tags, go into this array
+  def initialize()
+    @li_counter = 0
+    @links = []
+    @output = ""
+    @indent = 0
+    @errors = []
+  end
+
+  # Invokes the HTML parsing by using a string. Returns the markdown code in @output.
+  # To garantuee well-formed xml for REXML a <root> element will be added, but has no effect.
+  # After parsing all elements, the 'reference style'-links will be inserted.
+  def parse_string(string)
+    doc = Document.new("<root>\n"+string+"\n</root>")
+    parse_element(doc.root, :none)
+    insert_links()
+    @output
+  end
+
+  # Parsing an element and its children (recursive) and writing its markdown code to @output
+  # 1. do indent for nested list items
+  # 2. add the markdown opening tag for this element
+  # 3a. if element only contains text, handle it like a text node
+  # 3b. if element is a container handle its children, which may be text- or element nodes
+  # 4. finally add the markdown ending tag for this element
+  def parse_element(element, parent)
+    name = element.name.to_sym
+    # 1.
+    @output << indent() if name.eql?(:li)
+    # 2.
+    @output << opening(element, parent)
+
+    # 3a.
+    if (element.has_text? and element.children.size < 2)
+      @output << text_node(element, parent)
+    end
+
+    # 3b.
+    if element.has_elements?
+      element.children.each do |child|
+        # increase indent if nested list
+        @indent += 1 if element.name=~/(ul|ol)/ and parent.eql?(:li)
+
+        if child.node_type.eql?(:element)
+          parse_element(child, element.name.to_sym)
+        else
+          if parent.eql?(:blockquote)
+            @output << child.to_s.gsub("\n ", "\n>")
+          else
+            @output << child.to_s
+          end
+        end
+
+        # decrease indent if end of nested list
+        @indent -= 1 if element.name=~/(ul|ol)/ and parent.eql?(:li)
+      end
+    end
+
+    # 4.
+    @output << ending(element, parent)
+  end
+
+  # Returns opening markdown tag for the element. Its parent matters sometimes!
+  def opening(type, parent)
+    case type.name.to_sym
+      when :h1
+        "# "
+      when :li
+        parent.eql?(:ul) ? " - " : " "+(@li_counter+=1).to_s+". "
+      when :ol
+        @li_counter = 0
+        ""
+      when :ul
+        ""
+      when :h2
+        "## "
+      when :h3
+        "### "
+      when :h4
+        "#### "
+      when :h5
+        "##### "
+      when :h6
+        "###### "
+      when :em
+        "*"
+      when :strong
+        "**"
+      when :blockquote
+        # remove leading newline
+        type.children.first.value = ""
+        "> "
+      when :code
+        parent.eql?(:pre) ? "    " : "`"
+      when :a
+        "["
+      when :img
+        "!["
+      when :hr
+        "----------\n\n"
+      when :root
+        ""
+      else
+        @errors << "unknown start tag: "+type.name.to_s
+        ""
+    end
+  end
+
+  # Returns the closing markdown tag, like opening()
+  def ending(type, parent)
+    case type.name.to_sym
+      when :h1
+        " #\n\n"
+      when :h2
+        " ##\n\n"
+      when :h3
+        " ###\n\n"
+      when :h4
+        " ####\n\n"
+      when :h5
+        " #####\n\n"
+      when :h6
+        " ######\n\n"
+      when :p
+        parent.eql?(:root) ? "\n\n" : "\n"
+      when :ol
+        parent.eql?(:li) ? "" : "\n"
+      when :ul
+        parent.eql?(:li) ? "" : "\n"
+      when :em
+        "*"
+      when :strong
+        "**"
+      when :li
+        ""
+      when :blockquote
+        ""
+      when :code
+        parent.eql?(:pre) ? "" : "`"
+      when :a
+        @links << type.attribute('href').to_s
+        "][" + @links.size.to_s + "] "
+      when :img
+        @links << type.attribute('src').to_s
+        "" + type.attribute('alt').to_s + "][" + @links.size.to_s + "] "
+        "#{type.attribute('alt')}][#{@links.size}] "
+      when :root
+        ""
+      else
+        @errors << "  unknown end tag: "+type.name.to_s
+        ""
+    end
+  end
+
+  # Perform indent: two space, @indent times - quite simple! :)
+  def indent
+    str = ""
+    @indent.times do
+      str << "  "
+    end
+    str
+  end
+
+  # Return the content of element, which should be just text.
+  # If its a code block to indent of 4 spaces.
+  # For block quotation add a leading '>'
+  def text_node(element, parent)
+    if element.name.to_sym.eql?(:code) and parent.eql?(:pre)
+      element.text.gsub("\n","\n    ") << "\n"
+    elsif parent.eql?(:blockquote)
+      element.text.gsub!("\n ","\n>")
+    else
+      element.text
+    end
+  end
+
+  # Insert the mentioned reference style links.
+  def insert_links
+    @output << "\n"
+    @links.each_index do |index|
+      @output << "  [#{index+1}]: #{@links[index]}\n"
+    end
+  end
+
+  # Print out all errors, that occured and have been written to @errors.
+  def print_errors
+    @errors.each do |error|
+      puts error
+    end
+  end
+
+  # Perform a benchmark on a given string n-times.
+  def speed_benchmark(string, n)
+    initialize()
+    bm(15) do |test|
+      test.report("reverse markdown:")    { n.times do; parse_string(string); initialize(); end; }
+    end
+  end
+
+end
+
+if __FILE__ == $0
+
+  # Example HTML Code for parsing
+  example = <<-EOF
+This text, though not within an element, should also be shown.
+
+<h2>heading 1.1</h2>
+
+<p>text *italic* and **bold**.</p>
+
+<pre><code>text *italic* and **bold**.
+sdfsdff
+sdfsd
+sdf sdfsdf
+</code></pre>
+
+<blockquote>
+  <p>text <em>italic</em> and <strong>bold</strong>. sdfsdff
+  sdfsd sdf sdfsdf</p>
+</blockquote>
+
+<p>asdasd <code>sdfsdfsdf</code> asdad <a href="http://www.bla.de">link text</a></p>
+
+<p><a href="http://www.bla.de">link <strong>text</strong></a></p>
+
+<ol>
+<li>List item</li>
+<li>List <em>item</em>
+<ol><li>List item</li>
+<li>dsfdsf
+<ul><li>dfwe</li>
+<li>dsfsdfsdf</li></ul></li>
+<li>lidsf <img src="http://www.dfgdfg.de/dsf.jpe" alt="item" title="" /></li></ol></li>
+<li>sdfsdfsdf
+<ul><li>sdfsdfsdf</li>
+<li>sdfsdfsdf <strong>sdfsdf</strong></li></ul></li>
+</ol>
+
+<blockquote>
+  <p>Lorem ipsum dolor sit amet, consetetur
+  voluptua. At vero eos et accusam et
+  justo duo dolores et ea rebum. Stet
+  clita kasd gubergren, no sea takimata
+  sanctus est Lorem ipsum dolor sit
+  amet. <em>italic</em></p>
+</blockquote>
+
+<hr />
+
+<blockquote>
+  <p>Lorem ipsum dolor sit amet, consetetur
+  sadipscing elitr, sed diam nonumy
+  eirmod tempor invidunt ut labore et
+  dolore magna aliquyam erat, sed</p>
+</blockquote>
+
+This should also be shown, even if it's not wrapped in an element.
+
+<p>nur ein text! nur eine maschine!</p>
+
+This text should not be invisible!
+EOF
+
+  r = ReverseMarkdown.new
+
+  puts r.parse_string(example)
+
+  #r.print_errors
+
+  #r.speed_benchmark(example, 100)
+end