blekko-search 0.0.1

data/.gitignore

@@ -0,0 +1,50 @@
+# gitignore tips and tricks from https://github.com/github/gitignore
+
+# Ignore bundler config
+/.bundle
+
+# Ignore the default SQLite database.
+/db/*.sqlite3
+
+# Ignore all logfiles and tempfiles.
+/log/*.log
+/tmp
+
+# https://github.com/github/gitignore/blob/master/Ruby.gitignore
+*.gem
+*.rbc
+.bundle
+.config
+coverage
+InstalledFiles
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+
+# YARD artifacts
+.yardoc
+_yardoc
+doc/
+
+
+.DS_Store
+.AppleDouble
+.LSOverride
+Icon
+
+# https://github.com/github/gitignore/blob/master/Global/OSX.gitignore
+# Thumbnails
+._*
+
+# Files that might appear on external disk
+.Spotlight-V100
+.Trashes
+
+# https://github.com/github/gitignore/blob/master/Global/TextMate.gitignore
+*.tmproj
+*.tmproject
+tmtags

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in blekko-search.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2012 Sean Devine
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,141 @@
+# Blekko
+
+Search the Internet (or parts of the Internet!) with ease. This gem is powered by [blekko.com](http://www.blekko.com).
+
+This gem is based on work done on [earmarkd.com](http://www.earmarkd.com) during [RailsRumble 2012](http://railsrumble.com).
+
+## Installation
+
+Add this line to your application's ``Gemfile``:
+
+    gem 'blekko-search'
+
+And then execute:
+
+    $ bundle install
+
+Or install it yourself as:
+
+    $ gem install blekko-search
+
+## Usage
+
+Use this gem for quick searches or to manage groups of slashtags and more complicated search goals.
+
+### Searching
+
+While blekko asks that you [request an API key](http://help.blekko.com/index.php/does-blekko-have-an-api/), you don't need one to start.
+
+    blekko = Blekko.new
+
+While blekko is known for their slashtag based searching, you can search without one:
+
+    results = blekko.search("chicago")
+
+The search method returns an array of ``SearchResult`` instances that expose all blekko attributes (and a few more).
+
+If you want to search using [blekko slashtags](http://blekko.com/tag/show), include a ``:slashtags`` argument in your search.
+
+    results = blekko.search("chicago", slashtags: "/sports")
+
+Blekko allows for a maximum of 100 results per search, but if you'd like more, you can set the ``:total_size`` argument. 
+
+    results = blekko.search("something funny", total_size: 1000)
+
+By default, the searches will be made 100 results at a time, but you can reduce the page size using the ``:page_size`` argument too.
+
+Blekko asks that users of its API limit searches to one per second. This gem doesn't include that, but you may want to implement that feature (or send a pull request) especially if you are multithreading the search.
+
+#### Results
+Each result includes the attributes that blekko provides, plus a couple more:
+
+- ``n_group`` (alias: ``sequence``) = The number of the search result in the overall results from blekko.
+- ``url`` = The url of the result.
+- ``display_url`` = A url formatted for display.
+- ``rss`` = The rss of the result, if available.
+- ``rss_title`` = The title of the rss of the result, if available.
+- ``short_host_url`` = The url of the host of the result.
+- ``short_host`` = The url of the host formatted for display.
+- ``snippet`` (alias: ``abstract``) = A description of the result formatted for display including html
+- ``toplevel`` = Is the result a top level domain?
+- ``url_title`` = The title of the url formatted for display including html
+- ``date`` = The date of the the result's document, if available. Parsed from ``doc_date_iso``.
+- ``address`` = The address of the result, if available (not too often).
+- ``geocluster`` = The geocluster of the result, if avaiable.
+- ``lat`` = The lat of the result, if available.
+- ``lon`` = The lon of the result, if available.
+- ``phone`` = The phone number of the result, if available.
+- ``zip`` = The zip of the result, if available.
+- ``is_robots_banned`` = Does this result ban robots? Almost never available.
+
+### Slashtags
+From ["What is a slashtag?" on blekko.com](http://help.blekko.com/index.php/what-is-a-slashtag/):
+>A slashtag is an easy-to-create custom search engine. It is a tool used to filter search results and helps you to search only high quality sites, without spam or content farms. Slashtags contain a list of websites and when you search with a slashtag, you only search those sites. Some slashtags perform functions such as ordering the results by date.
+
+#### View
+You can view the urls for any slashtag that is public, or for any private slashtag that you have access to if you are logged in.
+
+    slashtag = blekko.slashtag("/sports")
+
+By default, the slashtag will load the urls from blekko if the slashtag exists.
+
+    urls = blekko.slashtag("/sports").urls
+    
+Prevent the slashtag from loading its ``urls`` automatically by setting the ``:eager_load`` argument to ``false``.
+
+    slashtag = blekko.slashtag("/sports", eager_load: false)
+    
+Access the urls currently saved on blekko.com at any point:
+
+    slashtag.saved_urls
+
+#### Create & Edit
+To create or edit a slashtag you'll need to use a blekko instance that has logged in using it's username and password.
+
+    blekko = Blekko.new(username: "derekrose", password: "comeback", api_key: "1")
+
+You will be logged in automatically if you provide all three credentials when you create the Blekko instance. Otherwise, you can call the ``login`` method to authenticate.
+
+    blekko = Blekko.new
+    blekko.username = "derekrose"
+    blekko.password = "comeback"
+    blekko.api_key = "1"
+    blekko.login
+
+Create a slashtag using the same syntax as used for the view method. You can pass in the ``urls`` when you initialize a new instance.
+
+    slashtag = blekko.slashtag("/my/sports", urls: ["http://www.espn.com", "http://http://sportsillustrated.cnn.com"])
+
+You can add urls to an existing slashtag object.
+
+    slashtag = blekko.slashtag("/my/sports")
+    slashtag.urls << "http://www.espn.com"
+
+To save the slashtags to blekko call ``save!``. This will either create a new slashtag or update the existing slashtag.
+
+    slashtag.save!
+
+#### Remove URLS
+
+You can also remove URLs from a slashtag.
+
+    slashtag = blekko.slashtag("/my/sports")
+    slashtag.remove_urls!("http://www.espn.com")
+
+#### Delete
+
+The API does not currently support deleting slashtags. You'll need to login to [blekko.com](http://www.blekko.com) to delete a slashtag from your account.
+
+## Acknowlegements
+Thanks to [blekko.com](http://www.blekko.com) for providing API access to their search engine.
+
+<3 [@barelyknown](http://www.twitter.com/barelyknown)
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request
+6. Thank you :)

data/Rakefile

@@ -0,0 +1 @@
+require "bundler/gem_tasks"

data/blekko-search.gemspec

@@ -0,0 +1,19 @@
+# -*- encoding: utf-8 -*-
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'blekko-search/version'
+
+Gem::Specification.new do |gem|
+  gem.name          = "blekko-search"
+  gem.version       = BlekkoSearch::VERSION
+  gem.authors       = ["Sean Devine"]
+  gem.email         = ["barelyknown@icloud.com"]
+  gem.description   = %q(Search and manage slashtags for blekko.com)
+  gem.summary       = %q(Search and manage slashtags for blekko.com)
+  gem.homepage      = "https://github.com/barelyknown/blekko-search"
+
+  gem.files         = `git ls-files`.split($/)
+  gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
+  gem.require_paths = ["lib"]
+end

data/lib/blekko-search.rb

@@ -0,0 +1,9 @@
+require 'cgi'
+require 'open-uri'
+require 'json'
+require 'net/http'
+require "blekko-search/version"
+require "blekko-search/blekko"
+require "blekko-search/search"
+require "blekko-search/search_result"
+require "blekko-search/slashtag"

data/lib/blekko-search/blekko.rb

@@ -0,0 +1,48 @@
+class Blekko
+  HOST = "www.blekko.com"
+  DEFAULT_MAX_FREQUENCY_PER_SECOND = 1
+      
+  attr_accessor :protocol, :api_key, :max_frequency_per_second, :username, :password, :login_cookie
+      
+  def initialize(args={})
+    @api_key = args[:api_key]
+    @protocol = args[:secure] ? "https://" : "http://"
+    @username = args[:username]
+    @password = args[:password]
+    @max_frequency_per_second = args[:max_frequency_per_second] || DEFAULT_MAX_FREQUENCY_PER_SECOND
+    login if @api_key && @username && @password
+  end
+  
+  def host
+    HOST
+  end
+  
+  def search(query, args={})
+    Blekko::Search.new(self, query, args).search
+  end
+  
+  def slashtag(name, args={})
+    Blekko::Slashtag.new(self, name, args)
+  end
+  
+  def login_uri
+    URI("https://blekko.com/login?u=#{CGI.escape(username)}&p=#{CGI.escape(password)}&auth=#{api_key}")
+  end
+  
+  def headers
+    { 
+      "Cookie" => "A=AfGTtGoSio7Hnc1xiaSrwkJX4ggMfcFiBXufPUPQXZvB5TRe36Q6tPI4woK6SKO8%2Bh8qeD7z0qEk%2B4Ceg5N9HA95UTpznKUvuuEfb04GiwhAlKARpLnp18%2BI6EYQfes1PB0QNnhHwEAC3kLjyJqCZbsxVw8ud4Z6F%2Fbg6BvJj28L;",
+      "User-Agent" => "Ruby"
+    }
+  end
+  
+  def login
+    raise ArgumentError, "Username and password are required" unless username && password
+    Net::HTTP.start(login_uri.host, login_uri.port, use_ssl: true) do |http|
+      response = http.request Net::HTTP::Get.new login_uri.request_uri
+      self.login_cookie = response.get_fields('Set-Cookie').find { |c| c =~ /\AA=/ }
+    end
+  end
+  
+  
+end

data/lib/blekko-search/search.rb

@@ -0,0 +1,67 @@
+class Blekko
+  class Search
+    
+    DEFAULT_PAGE_SIZE = 100
+    DEFAULT_PAGE_NUMBER = 0
+    PREFIX = "/ws/?q="
+    RESPONSE_FORMAT = "/json+/"
+    
+    attr_accessor :query, :slashtags, :results
+    
+    def initialize(blekko, query, args={})
+      args = {page_size: DEFAULT_PAGE_SIZE }.merge(args)
+      @blekko = blekko
+      @query = query
+      @slashtags = *args[:slashtags] || []
+      @page_size = args[:page_size]
+      @total_size = args[:total_size] || @page_size
+    end
+    
+    def results
+      @results ||= []
+    end
+        
+    def search
+      page_number = 0
+      number_of_searches.times do
+        response = JSON.load(open(url(page_number)))
+        if response['RESULT']
+          self.results += response['RESULT'].collect { |r| Blekko::SearchResult.new(r) }
+        else
+          return results
+        end
+        page_number += 1
+      end
+      results[0,@total_size]
+    end
+    
+    def number_of_searches
+      @number_of_searches ||= (@total_size.to_f / @page_size).ceil
+    end
+    
+    def escaped_query
+      CGI.escape(query + " ") + @slashtags.join("+") + "+"
+    end
+        
+    def page_size_param
+      "ps=#{@page_size}"
+    end
+    
+    def page_number_param(page_number)
+      "p=#{page_number}"
+    end
+    
+    def auth_param
+      @blekko.api_key ? "auth=#{@blekko.api_key}" : nil
+    end
+    
+    def params(page_number)
+      [page_size_param, auth_param, page_number_param(page_number)].compact.join("&")
+    end
+    
+    def url(page_number)
+      @blekko.protocol + @blekko.host + PREFIX + escaped_query + RESPONSE_FORMAT + params(page_number)
+    end
+        
+  end
+end

data/lib/blekko-search/search_result.rb

@@ -0,0 +1,25 @@
+class Blekko
+  class SearchResult
+    attr_accessor :n_group, :display_url, :rss, :rss_title, :short_host, :short_host_url,
+                  :snippet, :toplevel, :url, :url_title, :doc_date_iso, :address, :geocluster,
+                  :lat, :lon, :phone, :zip, :is_robots_banned
+                  
+    def initialize(result)
+      result.each do |key, value|
+        send("#{key}=", value) if respond_to? "#{key}="
+      end
+    end
+    
+    def datetime
+      DateTime.parse(doc_date_iso) if doc_date_iso
+    end
+    
+    def toplevel
+      @toplevel == "1" ? true : false
+    end
+    
+    alias_method :sequence, :n_group
+    alias_method :abstract, :snippet
+    
+  end
+end

data/lib/blekko-search/slashtag.rb

@@ -0,0 +1,72 @@
+class Blekko
+  class Slashtag
+    
+    attr_accessor :name, :blekko, :urls
+        
+    def initialize(blekko, name, args={})
+      args = { eager_load: true }.merge(args)
+      @blekko = blekko
+      @name = name
+      @urls = *args[:urls]
+      if args[:eager_load] && !@urls
+        self.urls = saved_urls
+      end
+    end
+    
+    def urls
+      @urls ||= []
+    end
+    
+    def saved_urls
+      url = blekko.protocol + blekko.host + "/tag/view?name=" + CGI.escape(name) + "&format=text&auth=#{blekko.api_key}"
+      lines = open(url).collect { |line| line.strip }
+      unless lines.first.scan(" ").any?
+        lines.collect { |line| line }
+      end
+    end
+        
+    def save!
+      begin
+        if create!.read =~ /already exists/
+          update!
+        end
+        true
+      rescue
+        false      
+      end
+    end
+    
+    def remove_urls!(target_urls)
+      open(remove_url(target_urls), blekko.headers)
+      true
+    end
+
+    
+    def create!
+      open(save_url("create"), blekko.headers)
+    end
+    
+    def update!
+      open(save_url("update"), blekko.headers)      
+    end
+    
+    
+    def save_url(method, target_urls=urls)
+      "https://" + blekko.host + "/tag/add?name=#{name}&submit=#{method}&urls=#{urls.join("%0A")}&auth=#{blekko.api_key}"
+    end
+    
+    def remove_url(target_urls)
+      "https://" + blekko.host + "/tag/edit?submit=1&type=del&name=#{name}&urls=#{target_urls.join("%0A")}&auth=#{blekko.api_key}"
+    end
+    
+    def delete_url
+      "https://blekko.com/tag/delete?submit=1&name=#{name}&auth=#{blekko.api_key}"
+    end
+    
+    def delete!
+      return ArgumentError, "This is not implemented by blekko yet"
+      open(delete_url, blekko.headers)
+    end
+        
+  end
+end

data/lib/blekko-search/version.rb

@@ -0,0 +1,3 @@
+module BlekkoSearch
+  VERSION = "0.0.1"
+end

metadata

@@ -0,0 +1,57 @@
+--- !ruby/object:Gem::Specification
+name: blekko-search
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+  prerelease: 
+platform: ruby
+authors:
+- Sean Devine
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2012-10-19 00:00:00.000000000 Z
+dependencies: []
+description: Search and manage slashtags for blekko.com
+email:
+- barelyknown@icloud.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- blekko-search.gemspec
+- lib/blekko-search.rb
+- lib/blekko-search/blekko.rb
+- lib/blekko-search/search.rb
+- lib/blekko-search/search_result.rb
+- lib/blekko-search/slashtag.rb
+- lib/blekko-search/version.rb
+homepage: https://github.com/barelyknown/blekko-search
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.24
+signing_key: 
+specification_version: 3
+summary: Search and manage slashtags for blekko.com
+test_files: []