bento_search 1.4.4 → 1.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: e070ceebebab6e8786050986873ee02f05990db9
-  data.tar.gz: e69e6151f6e2205de884717a1b4fd2cfcce95c40
+  metadata.gz: 8824757e65434f6a003c9748cf68e51cddcc3795
+  data.tar.gz: 081397571dbcb6307fdf9f162f5493e1d75d1e77
 SHA512:
-  metadata.gz: 43541b9f73943bd5d61e648f73e337104df2dc8a8cc01833106a92614a1cbcb14094eaf2e8364fbda1a135f6ed165275745aa3a11e0263b16a3a23b93e995159
-  data.tar.gz: 9d515c11008d35b0bbca0ab20b744818f9a17aabc906990f14715e87c43da9b423cd0a3e22ff3f126e18b12f4a79cf7609a7ea12e1c0f86178fb56c11424c886
+  metadata.gz: b5f11bd2566934fc22409ec97c9aa89d91e10a497948839ce6708e9a7c90bbf1ee1d951d5d2585641aa1dc0afb0fb082ac4cd258d64b25f8589bfda112233fd5
+  data.tar.gz: 32ac0a5ef5be6ec91d2b9bd034e7332d7ab52812743e56aeca97f59832c69fba45a20153102264730e6e18c9d9f97dfb26f4c0d51d1589eaf651fa9d2632823e

data/README.md

@@ -10,34 +10,31 @@ Rails 3.x or 4.x.   ruby 1.9.3+
 ### Goals: To help you
 
 * **Get up and running as quickly as possible** with searching and displaying
-  results from a  third-party service. Solutions to idiosyncracies and
-  undocumented workarounds are encoded in a shared codebase, which abstracts
-  everything to a good, simple code API giving you building blocks to focus
-  on your needs, not the search service's problems.
+  results from a  third-party service. Simple common code API, with idiosyncracies and
+  undocumented workarounds abstracted away.
 * Let you switch out one search service for another in an already built
   application with as little code rewriting as possible. **Avoid vendor lock-in**.
 * Give you the harness to **write adapters for new search services**, without
   having to rewrite common general functionality, just focus on the interface
   with the new API you want to support.
 
-bento_search is focused on use cases for academic libraries, which is mainly
-evidenced by the search engine adapters currently included, and by the
-generalized domain models including fields that matter in our domain (issn,
-vol/issue/page, etc), and some targetted functionality (OpenURL generation).
-But it ought to be useful for more general basic use
-cases too (we include a google site search adapter for instance).
+bento_search is focused on use cases for academic libraries; the shared
+model for search results includes including fields that matter in our domain (issn,
+vol/issue/page, etc), although they ought to have what's needed for general
+basic use too. There is some targetted functionality for academic library/publishing use
+(eg OpenURL generation).
 
 Adapters currently included in bento_search
 
-* Google Books (requires free api key)
-* Scopus (requires license)
-* Serial Solution Summon (requires license)
-* Ex Libris Primo (requires license)
-* EBSCO Discovery Service (requires license)
-* EBSCOHost 'traditional' API (requires license)
-* WorldCat Search (requires OCLC membership to get api key)
-* Google Site Search (requires sign-up for more than 100 searches/day)
-* JournalTOCs (limited support for fetching current articles by ISSN, free but requires registration)
+* [Google Books](https://books.google.com/) (requires free api key)
+* [Scopus](http://www.elsevier.com/solutions/scopus) (requires license)
+* [Serial Solution Summon](http://www.proquest.com/products-services/The-Summon-Service.html) (requires license)
+* [Ex Libris Primo](http://www.exlibrisgroup.com/category/PrimoOverview) (requires license)
+* [EBSCO Discovery Service](https://www.ebscohost.com/discovery) (requires license)
+* [EBSCOHost](https://www.ebscohost.com/) 'traditional' API (requires license)
+* [WorldCat Search](https://www.worldcat.org/) (requires OCLC membership to get api key)
+* [Google Site Search](https://www.google.com/work/search/products/gss.html) (requires sign-up for more than 100 searches/day)
+* [JournalTOCs](http://www.journaltocs.hw.ac.uk/) (limited support for fetching current articles by ISSN, free but requires registration)
 
 
 
@@ -218,6 +215,31 @@ Kaminari's paginate method:
     <%= paginate results.pagination %>
 ~~~~
 
+### Multi-field search
+
+Some search engines support-multi field searching, an engine advertises if it does:
+
+    engine_instance.multi_field_searching? # => `true` or `false`
+
+The bento_search multi-field search feature always combines multiple
+fields with boolean 'and' (intersection). You call a multi-field search
+with a :query hash argument whose value is a hash of search-fields and
+queries:
+
+    engine.search(:query => {
+      :title  => '"Reflections on the History of Debt Resistance"',
+      :author => 'Caffentzis'
+    })
+
+The search field keys can be either semantic_search_field names, or internal
+engine search fields, or a combination. If the key matches a semantic search field
+declared for the engine, that will be preferred.
+
+This can be used to expose a multi-field search to users, and the `bento_field_hash_for`
+helper method might be helpful in creating your UI. But this is also useful for looking
+up known-item citations -- either by author/title, or issn/volume/issue/page, or doi, or
+anything else -- as back-end support for various possible functions. 
+
 ### Concurrent searching
 
 If you're going to search 2 or more search engines at once, you'll want to execute

data/app/models/bento_search/result_item.rb

@@ -224,7 +224,7 @@ module BentoSearch
     # Short summary of item.
     # Mark .html_safe if it includes html -- creator is responsible
     # for making sure html is safely sanitizied and/or stripped,
-    # rails ActionView::Helpers::Sanistize #sanitize and #strip_tags
+    # rails ActionView::Helpers::SanitizeHelper #sanitize and #strip_tags
     # may be helpful.
     serializable_attr_accessor :abstract
 

data/app/models/bento_search/search_engine.rb

@@ -329,11 +329,44 @@ module BentoSearch
       if arguments[:sort]
         arguments[:sort] = arguments[:sort].to_s
       end
+
+      
+      # Multi-field search
+      if arguments[:query].kind_of? Hash
+        # Only if allowed
+        unless self.multi_field_search?
+          raise ArgumentError.new("You supplied a :query as a hash, but this engine (#{self.class}) does not suport multi-search. #{arguments[:query].inspect}")
+        end
+        # Multi-field search incompatible with :search_field or :semantic_search_field
+        if arguments[:search_field].present?
+          raise ArgumentError.new("You supplied a :query as a Hash, but also a :search_field, you can only use one. #{arguments.inspect}")
+        end
+        if arguments[:semantic_search_field].present?
+          raise ArgumentError.new("You supplied a :query as a Hash, but also a :semantic_search_field, you can only use one. #{arguments.inspect}")
+        end
+
+        # translate semantic fields, raising for unfound fields if configured
+        arguments[:query].transform_keys! do |key|
+          new_key = self.semantic_search_map[key.to_s] || key
+          
+          if ( config_arg(arguments, :unrecognized_search_field) == "raise" &&
+              ! self.search_keys.include?(new_key))
+            raise ArgumentError.new("#{self.class.name} does not know about search_field #{new_key}, in query Hash #{arguments[:query]}")
+          end
+
+          new_key
+        end
+
+      end
       
       # translate semantic_search_field to search_field, or raise if
       # can't. 
-      if (semantic = arguments.delete(:semantic_search_field)) && ! semantic.blank?        
-        mapped = self.semantic_search_map[semantic.to_s]
+      if (semantic = arguments.delete(:semantic_search_field)) && ! semantic.blank?
+        semantic = semantic.to_s
+        # Legacy publication_title is now called source_title
+        semantic = "source_title" if semantic == "publication_title"
+
+        mapped = self.semantic_search_map[semantic]
         if config_arg(arguments, :unrecognized_search_field) == "raise" && ! mapped 
           raise ArgumentError.new("#{self.class.name} does not know about :semantic_search_field #{semantic}")
         end
@@ -361,7 +394,7 @@ module BentoSearch
    
     
     protected
-    
+
     # get value of an arg that can be supplied in search args OR config,
     # with search_args over-ridding config. Also normalizes value to_s
     # (for symbols/strings). 

data/app/models/bento_search/search_engine/capabilities.rb

@@ -66,6 +66,20 @@ module BentoSearch::SearchEngine::Capabilities
           end.compact        
         ]
       end
+
+
+      # Engines that support multi-field search should
+      # override to return true. Returns false in base
+      # default implementation. 
+      #
+      # If an engine returns true here, it can receive in :query
+      # a Hash of multiple fields/values. The fields will all be
+      # normalized to internal names before engine receives them. 
+      # The multi-field search is meant to be run as a boolean AND
+      # of all field/values. 
+      def multi_field_search?
+        return false
+      end
       
 
 end

data/app/search_engines/bento_search/doaj_articles_engine.rb

@@ -0,0 +1,279 @@
+require 'httpclient'
+require 'http_client_patch/include_client'
+
+require 'json'
+
+module BentoSearch
+  # DOAJ Articles search. 
+  # https://doaj.org/api/v1/docs
+  #
+  # Phrase searches with double quotes are respected. 
+  #
+  # Supports #get by unique_id feature
+  #
+  class DoajArticlesEngine
+    include BentoSearch::SearchEngine
+    include ActionView::Helpers::SanitizeHelper
+
+
+    class_attribute :http_timeout
+    self.http_timeout = 10
+
+    extend HTTPClientPatch::IncludeClient
+    include_http_client do |client|
+      client.connect_timeout = client.send_timeout = client.receive_timeout = self.http_timeout
+    end
+
+    class_attribute :base_url
+    self.base_url = "https://doaj.org/api/v1/search/articles/"
+
+    def search_implementation(arguments)
+      query_url = args_to_search_url(arguments)
+
+      results = Results.new
+
+      begin
+        Rails.logger.debug("DoajEngine: requesting #{query_url}")
+        response = http_client.get( query_url )
+        json = JSON.parse(response.body)
+      rescue TimeoutError, HTTPClient::TimeoutError,
+             HTTPClient::ConfigurationError, HTTPClient::BadResponseError,
+             JSON::ParserError  => e
+        results.error ||= {}
+        results.error[:exception] = e
+      end
+
+      if ( response.nil? || json.nil? ||
+          (! HTTP::Status.successful? response.status) ||
+          (json && json["error"]))
+
+        results.error ||= {}
+        results.error[:status] = response.status if response
+        results.error[:message] = json["error"] if json["error"]
+
+        return results
+      end
+
+      results.total_items = json["total"]
+
+      (json["results"] || []).each do |item_response|
+        results <<  hash_to_item(item_response)
+      end
+
+      return results
+    end
+
+    def get(unique_id)
+      results = search(unique_id, :search_field => "id")
+
+      raise (results.error[:exception] || StandardError.new(results.error[:message] || results.error[:status])) if results.failed?
+      raise BentoSearch::NotFound.new("For id: #{unique_id}") if results.length == 0
+      raise BentoSearch::TooManyFound.new("For id: #{unique_id}") if results.length > 1
+
+      results.first
+    end
+
+
+    def args_to_search_url(arguments)
+      query = if arguments[:query].kind_of?(Hash)
+        # multi-field query
+        arguments[:query].collect {|field, query| fielded_query(query, field)}.join(" ")
+      else
+        fielded_query(arguments[:query], arguments[:search_field])
+      end
+
+      # We need to escape this for going in a PATH component,
+      # not a query. So space can't be "+", it needs to be "%20",
+      # and indeed DOAJ API does not like "+".
+      # 
+      # But neither CGI.escape nor URI.escape does quite
+      # the right kind of escaping, seems to work out
+      # if we do CGI.escape but then replace '+'
+      # with '%20'
+      escaped_query = CGI.escape(query).gsub('+', '%20')
+      url = self.base_url + escaped_query
+
+      query_args = {}
+
+      if arguments[:per_page]
+        query_args["pageSize"]  = arguments[:per_page]
+      end
+      
+      if arguments[:page]
+        query_args["page"]      = arguments[:page]
+      end
+
+      if arguments[:sort] &&
+          (defn = sort_definitions[arguments[:sort]]) &&
+          (value = defn[:implementation])
+        query_args["sort"] = value
+      end
+
+      query = query_args.to_query
+      url = url + "?" + query if query.present?
+
+      return url
+    end
+
+    # Prepares a DOAJ API (elastic search) query component for 
+    # given textual query in a given field (or default non-fielded search)
+    #
+    # Separates query string into tokens (bare words and phrases),
+    # so they can each be made mandatory for ElasticSearch. Default
+    # DOAJ API makes them all optional, with a very low mm, which
+    # leads to low-precision odd looking results for standard use
+    # cases. 
+    #
+    # Escapes all remaining special characters as literals (not including
+    # double quotes which can be used for phrases, which are respected. )
+    #
+    # Eg:
+    #     fielded_query('apple orange "strawberry banana"', field_name)
+    #     # => '+field_name(+apple +orange +"strawberry banana")'
+    #
+    # The "+" prefixed before field-name is to make sure all separate
+    # fields are also mandatory when doing multi-field searches. It should
+    # make no difference for a single-field search. 
+    def fielded_query(query, field = nil)
+      if field.present?
+        "+#{field}:(#{prepare_mandatory_terms(query)})"
+      else
+        prepare_mandatory_terms(query)
+      end
+    end
+
+    # Takes a query string, prepares an ElasticSearch query
+    # doing what we want: 
+    #   * tokenizes into bare words and double-quoted phrases
+    #   * Escapes other punctuation to be literal not ElasticSearch operator.
+    #     (Does NOT do URI escaping)
+    #   * Makes each token mandatory with an ElasticSearch "+" operator prefixed. 
+    def prepare_mandatory_terms(query)      
+      # use string split with regex to too-cleverly split into space
+      # seperated terms and phrases, keeping phrases as unit.
+      terms = query.split %r{[[:space:]]+|("[^"]+")}
+      # Wound up with some empty strings, get rid of em
+      terms.delete_if {|t| t.blank?}
+
+      terms.collect {|token| "+" + escape_query(token)}.join(" ")
+    end
+
+    # Converts from item found in DOAJ results to BentoSearch::ResultItem
+    def hash_to_item(hash)
+      item = ResultItem.new
+
+      bibjson = hash["bibjson"] || {}
+
+      item.unique_id  = hash["id"]
+
+      # Hard-code to Article, we don't get any format information
+      item.format     = "Article"
+
+      item.title      = bibjson["title"]
+
+
+      item.start_page = bibjson["start_page"]
+      item.end_page   = bibjson["end_page"]
+      
+      item.year       = bibjson["year"]
+      if (year = bibjson["year"].to_i) && (month = bibjson["month"].to_i)
+        if year != 0 && month != 0
+          item.publication_date = Date.new(bibjson["year"].to_i, bibjson["month"].to_i)
+        end
+      end      
+
+      item.abstract   = sanitize(bibjson["abstract"]) if bibjson.has_key?("abstract")
+
+      journal           = bibjson["journal"] || {}
+      item.volume       = journal["volume"]
+      item.issue        = journal["number"]
+      item.source_title = journal["title"]
+      item.publisher    = journal["publisher"]
+      item.language_str = journal["language"].try(:first)
+
+      (bibjson["identifier"] || []).each do |id_hash|
+        case id_hash["type"]
+        when "doi"
+          item.doi = id_hash["id"]
+        when "pissn"
+          item.issn = id_hash["id"]
+        end
+      end
+
+      (bibjson["author"] || []).each do |author_hash|
+        if author_hash.has_key?("name")
+          author = Author.new(:display => author_hash["name"])
+          item.authors << author
+        end
+      end
+
+      # I _think_ DOAJ articles results always only have one link,
+      # and it may always be of type 'fulltext'
+      link_hash             = (bibjson["link"] || []).first
+      if link_hash && link_hash["url"]
+        item.link             = link_hash["url"]
+        item.link_is_fulltext = true if link_hash["type"] == "fulltext"
+      end
+
+      return item
+    end
+
+    # Escape special chars in query, Doaj says it's elastic search,
+    # punctuation that needs to be escaped and how to escape (backslash)
+    # for ES documented here: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html
+    #
+    # We do not escape double quotes, want to allow them for phrases. 
+    #
+    # This method does NOT return URI-escaped, it returns literal, escaped for ES. 
+    def escape_query(q)
+      q.gsub(/([\+\-\=\&\|\>\<\!\(\)\{\}\[\]\^\~\*\?\:\\\/])/) {|m| "\\#{$1}"}
+    end
+
+
+    ###########
+    # BentoBox::SearchEngine API
+    ###########
+
+    def max_per_page
+      100
+    end
+
+    def search_field_definitions
+      { nil                     => {:semantic => :general},
+        "bibjson.title"         => {:semantic => :title},
+        # Using 'exact' seems to produce much better results for
+        # author, don't entirely understand what's up. 
+        "bibjson.author.name"   => {:semantic => :author},
+        "publisher"             => {:semantic => :publisher},
+        "bibjson.subject.term"  => {:semantic => :subject},
+        "bibjson.journal.title" => {:semantic => :source_title},
+        "issn"                  => {:semantic => :issn},
+        "doi"                   => {:semantic => :doi},
+        "bibjson.journal.volume"   => {:semantic => :volume},
+        "bibjson.journal.number"   => {:semantic => :issue},
+        "bibjson.start_page"   => {:semantic => :start_page},
+        "license" => {},
+        "id"      => {}
+      }
+    end
+
+    def multi_field_search?
+      true
+    end
+
+    def sort_definitions
+      # Don't believe DOAJ supports sorting by author
+      {        
+        "relevance" => {:implementation => nil}, # default
+        "title" => {:implementation => "title:asc"},
+        # We don't quite have publication date sorting, but we'll use
+        # created_date from DOAJ
+        "date_desc" => {:implementation => "article.created_date:desc"},
+        "date_asc"  => {:implementation => "article.created_date:asc"},
+        # custom one not previously standardized
+        "publication_name" => {:implementation => "bibjson.journal.title:asc"}
+      }
+    end
+
+  end
+end