google-site-search 0.0.5 → 0.0.7

data/README.rdoc

@@ -10,35 +10,56 @@ In the simplest use case it will query your google site search for a term and su
 
 Add the following to your projects Gemfile.
 
-	gem 'google-site-search', :git => "git@github.com:dvallance/google-site-search.git"
+    gem 'google-site-search', :git => "git@github.com:dvallance/google-site-search.git"
 
 Require the code if necessary (_note:_ some frameworks like rails are set to auto-require gems for you by default)
 
-	require 'google-site-search'
+    require 'google-site-search'
 
 == Usage
 
 The simpliest way to use the gem is by providing just a *search* *query* *term* and your *search* *engine* *unique* *id* code (_e.g._ looks like this +00255077836266642015+:+u-scht7a-8i+ and is located in your google site search control panel) 
 
-	#just assign the query to an object
-	search = GoogleSiteSearch.query(GoogleSiteSearch::UrlBuilder.new("microsoft", "00255077836266642015:u-scht7a-8i")
+    # just assign the query to an object
+    search = GoogleSiteSearch.query(GoogleSiteSearch::UrlBuilder.new("microsoft", "00255077836266642015:u-scht7a-8i")
   
-	#object has search attributes like
-	puts search.next_results_url
-	puts search.previous_results_url
-	puts search.xml
-	puts search.spelling
-	puts search.spelling_url
-
-	#object has an array of each specific result that contains title, description and its link by default
-	search.results.each do |result|
-		puts result.title
-		puts result.description
-		puts result.link
-	end
+    # object has search attributes like
+    puts search.next_results_url
+    puts search.previous_results_url
+    puts search.xml
+    puts search.spelling
+    puts search.spelling_url
+
+    # object has an array of each specific result that contains title, description and its link by default
+    search.results.each do |result|
+      puts result.title
+      puts result.description
+      puts result.link
+    end
 	
 The _query_ method expects a valid url so if you wanted to supply your own you can! However I have created a builder class to help with proper url creation and to help do some of the work for you.
 
+== Multiple Search
+
+Since google only allows a max of 20 returned results I have added a method that will capture up to *n* number of results.
+
+    # the array will be up to 5 search objects if the query actually has that many results.
+    # has soon as a search doesn't have a next_results_url the method stops.
+    array_of_search_results = GoogleSiteSearch.query_multiple(5, *YOUR_URL*)  
+
+== Blocks (and a caching example)
+
+Both the query and query_multiple methods can take a block which executes for each Search object found.
+
+    # here is a rails example using the block 
+    url = GoogleSiteSearch::UrlBuilder.new("microsoft", "00255077836266642015:u-scht7a-8i")
+    @search = Rails.cache.fetch(url) do |search|
+      # I can do something with the search objects.
+      # possibly custom analytics for our searchs?
+      search.url # we can access the object
+    end
+      
+
 == Advanced Usage 
 
 An important requirement for this gem was to be able to use {structured data}[https://developers.google.com/custom-search/docs/structured_data] for:
@@ -49,15 +70,14 @@ Therefore I allow the developer to supply his own "*Results*" class to the query
 
 The default Result class is as follows:
 
-	class Result
-		attr_reader :title, :link, :description
-
-		def initialize(node)
-			@title = node.find_first("T").content
-			@link = node.find_first("UE").content
-			@description = node.find_first("S").content
-		end
-	end
+    class Result
+      attr_reader :title, :link, :description
+        def initialize(node)
+        @title = node.find_first("T").content
+        @link = node.find_first("UE").content
+        @description = node.find_first("S").content
+      end
+    end
 
 As you can see it is very simple. Your class simply needs an initialize method that will recieve an xml node, which it can then do with as it pleases. After it is initialized it is added to the _search.results_ array as shown previously.
 
@@ -67,9 +87,9 @@ See
 
 == Pagination
 
-The google search api actually does the work of pagination for us, supplying the next and previous urls. The urls are relative to \http://www.google.com so I added a _paginate_ method to simplify the call.
+The google search api does the work of pagination for us, supplying the next and previous urls. The urls are relative paths and contain the search engine id parameter. Since this is a security concern I strip out the search engine id when I store them in the Search.next_result_url and Search.previous_result_url methods. This makes them safe to put in links on views and it is why you must supply the search engine id again on the paginate call; so full url can be rebuilt for the query call.
 
-	search2 = GoogleSiteSearch.query(GoogleSiteSearch.paginate(search1.next_results_url))
+    search2 = GoogleSiteSearch.query(GoogleSiteSearch.paginate(search1.next_results_url, "00255077836266642015:u-scht7a-8i"))
 
 == Pagination Simple Example
 
@@ -78,19 +98,19 @@ This works and is fairly straight forward.
 In your controller:
 
     if params[:move]
-      @search = GoogleSiteSearch.query(GoogleSiteSearch.paginate(params[:move]))
+      @search = GoogleSiteSearch.query(GoogleSiteSearch.paginate(params[:move], "00255077836266642015:u-scht7a-8i"))
     else
       @search = GoogleSiteSearch.query(GoogleSiteSearch::UrlBuilder.new("microsoft", "00255077836266642015:u-scht7a-8i", :num => 5))
     end
 
 In your view:
 
-	<% if @search.previous_results_url %>
-		<%= link_to "Previous", search_url(:move => @search.previous_results_url) %>
-	<% end %>
-	<% if @search.next_results_url %>
-		<%= link_to "More", search_url(:move => @search.next_results_url) %>
-	<% end %>
+    <% if @search.previous_results_url %>
+      <%= link_to "Previous", search_url(:move => @search.previous_results_url) %>
+    <% end %>
+    <% if @search.next_results_url %>
+      <%= link_to "More", search_url(:move => @search.next_results_url) %>
+    <% end %>
 
 == Escaping
 
@@ -98,13 +118,11 @@ If you start passing around the url's in parameters you may run into issues if y
 
 View adds escape:
 
-	<%= link_to "Previous", search_url(:move => CGI::escape(@search.previous_results_url)) %>
+    <%= link_to "Previous", search_url(:move => CGI::escape(@search.previous_results_url)) %>
 
 Controller unescapes:
 
-	@search = GoogleSiteSearch.query(GoogleSiteSearch.paginate(CGI::unescape(params[:move])))
-
-
+    @search = GoogleSiteSearch.query(GoogleSiteSearch.paginate(CGI::unescape(params[:move]), "00255077836266642015:u-scht7a-8i"))
 
 == Filtering and Sorting
 
@@ -116,8 +134,8 @@ Google expects filtering to be on the "search query" itself. However I feel my e
 
 From the google reference link above an example filter search query is <b>halloween more:pagemap:document-author:lisamorton</b>
 
-	#using the example above would look like this.
-	search = GoogleSiteSearch.query(GoogleSiteSearch::UrlBuilder.new("halloween", "00255077836266642015:u-scht7a-8i", :filter =>  "more:pagemap:document-author:lisamorton")
+    # using the example above would look like this.
+    search = GoogleSiteSearch.query(GoogleSiteSearch::UrlBuilder.new("halloween", "00255077836266642015:u-scht7a-8i", :filter =>  "more:pagemap:document-author:lisamorton")
 
 === Separate Search Term From Filters
 
@@ -125,20 +143,20 @@ The full "search query" is returned by google's api and stored in the Search obj
 
 To separate the search term from the filter use:
 
-	search_term, filters = GoogleSiteSearch.separate_search_term_from_filters(@search.search_query)
+    search_term, filters = GoogleSiteSearch.separate_search_term_from_filters(@search.search_query)
 
 === Sorting
 
 Sorting would also be done by specifing a *sort* option. 
 
-	search = GoogleSiteSearch.query(GoogleSiteSearch::UrlBuilder.new("halloween", "00255077836266642015:u-scht7a-8i", :filter =>  "more:pagemap:document-author:lisamorton", :sort => "data-sdate")
+    search = GoogleSiteSearch.query(GoogleSiteSearch::UrlBuilder.new("halloween", "00255077836266642015:u-scht7a-8i", :filter =>  "more:pagemap:document-author:lisamorton", :sort => "data-sdate")
 	
 == Other Params
 
 Any <b>[param=value]</b> query string additions you want to add can be assigned like the sorting above. For example to limit the search results return, to 5, would look like...
 
-	#get only 5 search results with the filtering and sorting from above still applyed.
-	search = GoogleSiteSearch.query(GoogleSiteSearch::UrlBuilder.new("halloween more:pagemap:document-author:lisamorton", "00255077836266642015:u-scht7a-8i", :sort => "date-sdate", :num => "5" )
+    # get only 5 search results with the filtering and sorting from above still applyed.
+    search = GoogleSiteSearch.query(GoogleSiteSearch::UrlBuilder.new("halloween more:pagemap:document-author:lisamorton", "00255077836266642015:u-scht7a-8i", :sort => "date-sdate", :num => "5" )
 
 == Author
 

data/google-site-search.gemspec

@@ -16,4 +16,6 @@ Gem::Specification.new do |gem|
   gem.version       = GoogleSiteSearch::VERSION
   gem.add_dependency("activesupport")
   gem.add_dependency("libxml-ruby")
+  gem.add_dependency("rsmaz")
+  gem.add_dependency("rack")
 end

data/lib/google-site-search.rb

@@ -6,10 +6,12 @@ require "google-site-search/version"
 require "google-site-search/url_builder"
 require "google-site-search/search"
 require "google-site-search/result"
-require "timeout"
 require "net/http"
+require "rsmaz"
+require "timeout"
 require "uri"
 require "xml"
+require "rack/utils"
 
 ##
 # A module to help query and parse the google site search api.
@@ -24,9 +26,28 @@ module GoogleSiteSearch
 
   class << self
 
+    # Takes a url, strips out un-required query params, and compresses
+    # a string representation. The intent is to have a small string to
+    # use as a caching key. 
+    def caching_key url
+      params = Rack::Utils.parse_query(URI.parse(url).query)
+      # ei = "Passes on an alphanumeric parameter that decodes the originating SERP where user clicked on a related search". Don't fully understand what it does but it makes my caching less effective.
+      params.delete("ei") 
+      key = params.map{|k,v| k.to_s + v.to_s}.sort.join
+      key.blank? ? nil : RSmaz.compress(key) 
+    end
+
     # Expects the URL returned by Search#next_results_url or Search#previous_results_url.
-    def paginate url
-      GOOGLE_SEARCH_URL + url.to_s
+    def paginate url, search_engine_id
+      raise StandardError, "search_engine_id required" if search_engine_id.blank? 
+      uri = URI.parse(url.to_s)
+      raise StandardError, "url seems to be invalid, parameters expected" if uri.query.blank?
+      if uri.relative?
+        uri.host = "www.google.com"
+        uri.scheme = "http"
+      end
+      uri.query = uri.query += "&cx=#{search_engine_id}"
+      uri.to_s
     end
 
     # See Search - This is a convienence method for creating and querying. 

data/lib/google-site-search/search.rb

@@ -44,6 +44,15 @@ module GoogleSiteSearch
       @result_class = result_class
     end
 
+    def next_results_url  
+      @next_results_url
+    end
+
+    def previous_results_url
+      @previous_results_url
+    end
+
+
 		# Query's Google API, stores the xml and parses values into itself.
     def query
       @xml = GoogleSiteSearch::request_xml(url)
@@ -68,12 +77,21 @@ module GoogleSiteSearch
         @spelling = spelling_node.try(:content) 
         @spelling_q = spelling_node.try(:attributes).try(:[],:q)
         @estimated_results_total = doc.find_first("RES/M").try(:content)
-        @next_results_url = doc.find_first("RES/NB/NU").try(:content)
-        @previous_results_url = doc.find_first("RES/NB/PU").try(:content)
+        @next_results_url = remove_search_engine_id(doc.find_first("RES/NB/NU").try(:content))
+        @previous_results_url = remove_search_engine_id(doc.find_first("RES/NB/PU").try(:content))
         @search_query = doc.find_first("Q").try(:content)
       rescue Exception => e
         raise ParsingError, "#{e.message} URL:[#{@url}] XML:[#{@xml}]"
       end
     end
+
+    def remove_search_engine_id url
+      return nil if url.blank?
+      uri = URI.parse(url)
+      params = Rack::Utils::parse_query(uri.query)
+      params.delete("cx")
+      uri.query = params.map{|k,v| "#{k}=#{v}"}.sort.join("&")
+      uri.to_s
+    end
   end
 end

data/lib/google-site-search/version.rb

@@ -1,3 +1,3 @@
 module GoogleSiteSearch
-	VERSION = "0.0.5"
+	VERSION = "0.0.7"
 end

data/test/data/utf8_results.xml

@@ -35,8 +35,8 @@
   <RES SN="3" EN="4">
     <M>29</M>
     <NB>
-      <PU>/previous</PU>
-      <NU>/next</NU>
+      <PU>/previous?start=20&amp;q=search&amp;cx=my_key</PU>
+      <NU>/next?cx=my_key&amp;q=search&amp;start=1</NU>
     </NB>
     <RG START="1" SIZE="2"/>
     <RG START="1" SIZE="1">    </RG>

data/test/test_google_site_search.rb

@@ -2,10 +2,46 @@ require_relative 'test_helper'
 
 describe GoogleSiteSearch do
 
+
+  describe '.caching_key' do
+    let :sample_url do 
+      "http://domain?q=work&ei=dontshow&do=i"
+    end
+
+    let :becomes do
+      "doiqwork" #query parameter values sorted and concated.
+    end
+
+    it "properly creates a key" do
+      RSmaz.decompress(GoogleSiteSearch.caching_key(sample_url)).must_equal becomes
+    end 
+  
+    it "raises error on bad uri" do
+      -> {GoogleSiteSearch.caching_key(nil)}.must_raise URI::InvalidURIError
+    end
+
+  end
+
   describe '.paginate' do
+
+    let :valid_url do
+      "http://www.valid.com/search?q=mysearch"
+    end
+    
+    it "raises an error if the url has no parameters and is therefore invalid" do
+      url = "http://www.noparameters.com/"
+      -> {GoogleSiteSearch.paginate(url, "my_key")}.must_raise StandardError
+    end
+
+    it "raises an error if a search engine key is nil or blank" do
+      -> {GoogleSiteSearch.paginate(valid_url, "")}.must_raise StandardError 
+      -> {GoogleSiteSearch.paginate(valid_url, nil)}.must_raise StandardError 
+    end
+
     it 'completes a valid url for the relative path supplied' do
-      GoogleSiteSearch.paginate("/some/path").must_equal "http://www.google.com/some/path"
+      GoogleSiteSearch.paginate("/some/path?q=search","my_key").must_equal "http://www.google.com/some/path?q=search&cx=my_key"
     end
+
   end
 
   describe '.relative_path' do

data/test/test_search.rb

@@ -27,13 +27,21 @@ describe Search do
     end
 
     it "contains the next results url" do
-      search.next_results_url.must_equal "/next"
+      search.next_results_url.wont_be :empty?
+    end
+
+    it "next results url removed the search engine id parameter" do
+      search.next_results_url.must_equal "/next?q=search&start=1"
     end
 
     it "contains the previous results url" do
-      search.previous_results_url.must_equal "/previous"
+      search.previous_results_url.wont_be :empty?
     end
 
+    it "next results url removed the search engine id parameter" do
+      search.previous_results_url.must_equal "/previous?q=search&start=20"
+    end
+    
     it "stores the original xml" do
       search.xml.must_equal xml
     end
@@ -54,4 +62,5 @@ describe Search do
       search.spelling_q.must_equal "fake suggestion escaped"
     end
   end
+
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: google-site-search
 version: !ruby/object:Gem::Version
-  version: 0.0.5
+  version: 0.0.7
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-11-21 00:00:00.000000000 Z
+date: 2012-11-25 00:00:00.000000000Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
-  requirement: &22916060 !ruby/object:Gem::Requirement
+  requirement: &11771100 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,10 +21,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *22916060
+  version_requirements: *11771100
 - !ruby/object:Gem::Dependency
   name: libxml-ruby
-  requirement: &22915560 !ruby/object:Gem::Requirement
+  requirement: &11770260 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -32,7 +32,29 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *22915560
+  version_requirements: *11770260
+- !ruby/object:Gem::Dependency
+  name: rsmaz
+  requirement: &11761040 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: *11761040
+- !ruby/object:Gem::Dependency
+  name: rack
+  requirement: &11760240 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: *11760240
 description: A gem to aid in the consumption of the google site search service; querys
   the service, populates a result object and has some related helper methods.
 email:
@@ -70,12 +92,18 @@ required_ruby_version: !ruby/object:Gem::Requirement
   - - ! '>='
     - !ruby/object:Gem::Version
       version: '0'
+      segments:
+      - 0
+      hash: -3596847305660679714
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
   - - ! '>='
     - !ruby/object:Gem::Version
       version: '0'
+      segments:
+      - 0
+      hash: -3596847305660679714
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.10