pg_search 0.2.2 → 0.3

data/.gitignore

@@ -5,5 +5,4 @@ doc
 .idea
 tags
 *~
-gemfiles/rails2/Gemfile.lock
-gemfiles/rails3/Gemfile.lock
+Gemfile.lock

data/.rspec

@@ -0,0 +1 @@
+--colour

data/.travis.yml

@@ -0,0 +1,2 @@
+before_script:
+  - "psql -c 'create database pg_search_test;' -U postgres >/dev/null"

data/CHANGELOG

@@ -1,11 +1,14 @@
-### 0.2.2
+### 0.3
 
- * Fix a compatibility issue between Ruby 1.8.7 and 1.9.3 when using Rails 2
-   (James Badger)
+ * Drop Active Record 2.0 support.
 
-### 0.2.1
+ * Add PgSearch.multisearch for cross-model searching.
 
- * Backport support for searching against tsvector columns (Kris Hicks)
+ * Fix PostgreSQL warnings about truncated identifiers
+
+ * Support specifying a method of rank normalisation when using tsearch. (Arthur Gunn)
+
+ * Add :any_word option to :tsearch which uses OR between query terms instead of AND. (Fernando Espinosa)
 
 ### 0.2
 

data/Gemfile

@@ -1,5 +1,10 @@
-puts <<-MESSAGE
-This project uses multiple Gemfiles in subdirectories of ./gemfiles.
-The rake tasks automatically install these bundles as necessary. See rake -T.
-MESSAGE
-exit 1
+source "http://rubygems.org"
+
+gemspec
+
+gem "rake"
+gem "rdoc"
+gem "pg"
+gem "rspec", ">=2.4"
+gem "autotest"
+gem "with_model"

data/README.rdoc

@@ -18,16 +18,11 @@ In Gemfile
 
  gem 'pg_search'
 
-=== Rails 2
+=== Other ActiveRecord-based projects
 
-In environment.rb
+In addition to installing and requiring the gem, you may want to include the PgSearch rake tasks in your Rakefile:
 
- config.gem 'pg_search'
-
-In Rakefile
-
- require 'rubygems'
- require 'pg_search/tasks'
+ load "pg_search/tasks.rb"
 
 == USAGE
 
@@ -37,6 +32,94 @@ To add PgSearch to an ActiveRecord model, simply include the PgSearch module.
    include PgSearch
  end
 
+=== Multi-search vs. search scopes
+
+pg_search supports two different techniques for searching, multi-search and search scopes.
+
+The first technique is multi-search, in which records of many different Active Record classes can be mixed together into one global search index across your entire application. Most sites that want to support a generic search page will want to use this feature.
+
+The other technique is search scopes, which allow you to do more advanced searching against only one Active Record class. This is more useful for building things like autocompleters or filtering a list of items in a faceted search.
+
+=== Multi-search
+
+==== Setup
+
+Before using multi-search, you must generate and run a migration to create the pg_search_documents database table.
+
+ $ rake pg_search:migration:multisearch
+ $ rake db:migrate
+
+==== multisearchable
+
+To add a model to the global search index for your application, call multisearchable in its class definition.
+
+ class EpicPoem < ActiveRecord::Base
+   include PgSearch
+   multisearchable :against => [:title, :author]
+ end
+
+ class Flower < ActiveRecord::Base
+   include PgSearch
+   multisearchable :against => :color
+ end
+
+Whenever a record is created, updated, or destroyed, an Active Record callback will fire, leading to the creation of a corresponding PgSearch::Document record in the pg_search_documents table. The :against option can be one or several methods which will be called on the record to generate its search text.
+
+==== Multi-search associations
+
+Two associations are built automatically. On the original record, there is a has_one :pg_search_document association pointing to the PgSearch::Document record, and on the PgSearch::Document record there is a belongs_to :searchable polymorphic association pointing back to the original record.
+
+ odyssey = EpicPoem.create!(:title => "Odyssey", :author => "Homer")
+ search_document = odyssey.pg_search_document #=> PgSearch::Document instance
+ search_document.searchable #=> #<EpicPoem id: 1, title: "Odyssey", author: "Homer">
+
+==== Searching in the global search index
+
+To fetch the PgSearch::Document entries for all of the records that match a given query, use  PgSearch.multisearch.
+
+ odyssey = EpicPoem.create!(:title => "Odyssey", :author => "Homer")
+ rose = Flower.create!(:color => "Red")
+ PgSearch.multisearch("Homer") #=> [#<PgSearch::Document searchable: odyssey>]
+ PgSearch.multisearch("Red") #=> [#<PgSearch::Document searchable: rose>]
+
+==== Chaining method calls onto the results
+
+PgSearch.multisearch returns an ActiveRecord::Relation, just like scopes do, so you can chain scope calls to the end. This works with gems like Kaminari that add scope methods. Just like with regular scopes, the database will only receive SQL requests when necessary.
+
+ PgSearch.multisearch("Bertha").limit(10)
+ PgSearch.multisearch("Juggler").where(:searchable_type => "Occupation")
+ PgSearch.multisearch("Alamo").page(3).per_page(30)
+ PgSearch.mulitsearch("Diagonal").find_each do |document|
+   puts document.searchable.updated_at
+ end
+
+==== Rebuilding search documents for a given class
+
+If you change the :against option on a class, add multisearchable to a class that already has records in the database, or remove multisearchable from a class in order to remove it from the index, you will find that the pg_search_documents table could become out-of-sync with the actual records in your other tables.
+
+The index can also become out-of-sync if you ever modify records in a way that does not trigger Active Record callbacks. For instance, the #update_attribute instance method and the .update_all class method both skip callbacks and directly modify the database.
+
+To remove all of the documents for a given class, you can simply delete all of the PgSearch::Document records.
+
+ PgSearch::Document.delete_all(:searchable_type => "Animal")
+
+Run this Rake task to regenerate all of the documents for a given class.
+
+ $ rake pg_search:multisearch:rebuild CLASS=BlogPost
+
+Currently this is only supported for :against methods that directly map to Active Record attributes. Until that is fixed, you could also manually rebuild all of the documents.
+
+ PgSearch::Document.delete_all(:searchable_type => "Ingredient")
+ Ingredient.find_each { |record| record.update_pg_search_document }
+
+==== Disabling multi-search indexing temporarily
+
+If you have a large bulk operation to perform, such as importing a lot of records from an external source, you might want to speed things up by turning off indexing temporarily. You could then use one of the techniques above to rebuild the search documents off-line.
+
+ PgSearch.disable_multisearch do
+   Movie.import_from_xml_file(File.open("movies.xml"))
+ end
+
 === pg_search_scope
 
 You can use pg_search_scope to build a search scope. The first parameter is a scope name, and the second parameter is an options hash. The only required option is :against, which tells pg_search_scope which column or columns to search against.
@@ -135,11 +218,11 @@ You can pass a Hash into the :associated_against option to search columns on oth
 
 === Searching using different search features
 
-By default, pg_search_scope uses the built-in {PostgreSQL text search}[http://www.postgresql.org/docs/current/static/textsearch-intro.html]. If you pass the :features option to pg_search_scope, you can choose alternative search techniques.
+By default, pg_search_scope uses the built-in {PostgreSQL text search}[http://www.postgresql.org/docs/current/static/textsearch-intro.html]. If you pass the :using option to pg_search_scope, you can choose alternative search techniques.
 
   class Beer < ActiveRecord::Base
     include PgSearch
-    pg_search_scope :against => :name, :features => [:tsearch, :trigram, :dmetaphone]
+    pg_search_scope :search_name, :against => :name, :using => [:tsearch, :trigram, :dmetaphone]
   end
 
 The currently implemented features are
@@ -157,7 +240,7 @@ Each searchable column can be given a weight of "A", "B", "C", or "D". Columns w
 
  class NewsArticle < ActiveRecord::Base
    include PgSearch
-   pg_search_scope :against => {
+   pg_search_scope :search_full_text, :against => {
      :title => 'A',
      :subtitle => 'B',
      :content => 'C'
@@ -168,7 +251,7 @@ You can also pass the weights in as an array of arrays, or any other structure t
 
  class NewsArticle < ActiveRecord::Base
    include PgSearch
-   pg_search_scope :against => [
+   pg_search_scope :search_full_text, :against => [
      [:title, 'A'],
      [:subtitle, 'B'],
      [:content, 'C']
@@ -177,7 +260,7 @@ You can also pass the weights in as an array of arrays, or any other structure t
 
  class NewsArticle < ActiveRecord::Base
    include PgSearch
-   pg_search_scope :against => [
+   pg_search_scope :search_full_text, :against => [
      [:title, 'A'],
      {:subtitle => 'B'},
      :content
@@ -228,15 +311,65 @@ PostgreSQL full text search also support multiple dictionaries for stemming. You
   BoringTweet.kinda_matching("sleeping") # => [sleepy, sleeping, sleeper]
   BoringTweet.literally_matching("sleeping") # => [sleeping]
 
-==== :dmetaphone (Double Metaphone soundalike search)
+===== :normalization
 
-{Double Metaphone}[http://en.wikipedia.org/wiki/Double_Metaphone] is an algorithm for matching words that sound alike even if they are spelled very differently. For example, "Geoff" and "Jeff" sound identical and thus match. Currently, this is not a true double-metaphone, as only the first metaphone is used for searching.
+PostgreSQL supports multiple algorithms for ranking results against queries. For instance, you might want to consider overall document size or the distance between multiple search terms in the original text. This option takes an integer, which is passed directly to PostgreSQL. According to the latest {PostgreSQL documentation}[http://www.postgresql.org/docs/current/static/textsearch-controls.html], the supported algorithms are:
 
-Double Metaphone support is currently available as part of the {fuzzystrmatch contrib package}[http://www.postgresql.org/docs/current/static/fuzzystrmatch.html] that must be installed before this feature can be used. In addition to the contrib package, you must install a utility function into your database. To generate a migration for this, add the following line to your Rakefile:
+ 0 (the default) ignores the document length
+ 1 divides the rank by 1 + the logarithm of the document length
+ 2 divides the rank by the document length
+ 4 divides the rank by the mean harmonic distance between extents
+ 8 divides the rank by the number of unique words in document
+ 16 divides the rank by 1 + the logarithm of the number of unique words in document
+ 32 divides the rank by itself + 1
 
-  include "pg_search/tasks"
+This integer is a bitmask, so if you want to combine algorithms, you can add their numbers together. (e.g. to use algorithms 1, 8, and 32, you would pass 1 + 8 + 32 = 41)
 
-and then run:
+  class BigLongDocument < ActiveRecord::Base
+    include PgSearch
+    pg_search_scope :regular_search,
+                    :against => :text
+
+    pg_search_scope :short_search,
+                    :against => :text,
+                    :using => {
+                      :tsearch => {:normalization => 2}
+                    }
+
+  long = BigLongDocument.create!(:text => "Four score and twenty years ago")
+  short = BigLongDocument.create!(:text => "Four score")
+
+  BigLongDocument.regular_search("four score") #=> [long, short]
+  BigLongDocument.short_search("four score") #=> [short, long]
+
+===== :any_word
+
+Setting this attribute to true will perform a search which will return all models containing any word in the search terms.
+
+  class Number < ActiveRecord::Base
+    include PgSearch
+    pg_search_scope :search_any_word,
+                    :against => :text,
+                    :using => {
+                      :tsearch => {:any_word => true}
+                    }
+
+    pg_search_scope :search_all_words,
+                    :against => :text
+  end
+
+  one = Number.create! :text => 'one'
+  two = Number.create! :text => 'two'
+  three = Number.create! :text => 'three'
+
+  Number.search_any_word('one two three') # => [one, two, three]
+  Number.search_all_words('one two three') # => []
+
+==== :dmetaphone (Double Metaphone soundalike search)
+
+{Double Metaphone}[http://en.wikipedia.org/wiki/Double_Metaphone] is an algorithm for matching words that sound alike even if they are spelled very differently. For example, "Geoff" and "Jeff" sound identical and thus match. Currently, this is not a true double-metaphone, as only the first metaphone is used for searching.
+
+Double Metaphone support is currently available as part of the {fuzzystrmatch contrib package}[http://www.postgresql.org/docs/current/static/fuzzystrmatch.html] that must be installed before this feature can be used. In addition to the contrib package, you must install a utility function into your database. To generate a migration for this, run:
 
   $ rake pg_search:migration:dmetaphone
 
@@ -302,32 +435,6 @@ Ignoring accents uses the {unaccent contrib package}[http://www.postgresql.org/d
   SpanishQuestion.gringo_search("Que") # => [what]
   SpanishQuestion.gringo_search("Cüåñtô") # => [how_many]
 
-=== Using tsvector columns
-
-PostgreSQL allows you the ability to search against a column with type tsvector instead of using an expression; this speeds up searching dramatically as it offloads creation of the tsvector that the tsquery is evaluated against.
-
-To use this functionality you'll need to do a few things:
-
-* Create a column of type tsvector that you'd like to search against. If you want to search using multiple search methods, for example tsearch and dmetaphone, you'll need a column for each.
-* Create a trigger function that will update the column(s) using the expression appropriate for that type of search. See: http://www.postgresql.org/docs/current/static/textsearch-features.html#TEXTSEARCH-UPDATE-TRIGGERS
-* Should you have any pre-existing data in the table, update the newly-created tsvector columns with the expression that your trigger function uses.
-* Add the option to pg_search_scope, e.g:
-
-  pg_search_scope :fast_content_search,
-                  :against => :content,
-                  :using => {
-                    dmetaphone: {
-                      tsvector_column: 'tsvector_content_dmetaphone'
-                    },
-                    tsearch: {
-                      dictionary: 'english',
-                      tsvector_column: 'tsvector_content_tsearch'
-                    }
-                    trigram: {} # trigram does not use tsvectors
-                  }
-
-Please note that the :against column is only used when the tsvector_column is not present for the search type.
-
 == REQUIREMENTS
 
 * ActiveRecord 2 or 3

data/Rakefile

@@ -3,42 +3,27 @@ Bundler::GemHelper.install_tasks
 
 task :default => :spec
 
-environments = %w[rails2 rails3]
-major, minor, revision = RUBY_VERSION.split(".").map{|str| str.to_i }
-
-in_environment = lambda do |environment, command|
-  sh %Q{export BUNDLE_GEMFILE="gemfiles/#{environment}/Gemfile"; bundle update && bundle exec #{command}}
+def bundle_exec(command)
+  sh %Q{bundle update && bundle exec #{command}}
 end
 
-in_all_environments = lambda do |command|
-  environments.each do |environment|
-    next if environment == "rails2" && major == 1 && minor > 8
-    puts "\n---#{environment}---\n"
-    in_environment.call(environment, command)
-  end
-end
-
-desc "Run all specs against ActiveRecord 2 and 3"
+desc "Run all specs"
 task "spec" do
-  in_all_environments.call('rspec spec')
+  bundle_exec("rspec spec")
 end
 
 task "doc" do
-  in_environment.call("rails3", "rspec --format d spec")
+  bundle_exec("rspec --format d spec")
 end
 
-namespace "autotest" do
-  environments.each do |environment|
-    desc "Run autotest in #{environment}"
-    task environment do
-      in_environment.call(environment, 'autotest -s rspec2')
-    end
-  end
+desc "Launch autotest"
+task "autotest" do
+  bundle_exec("autotest -s rspec2")
 end
 
 namespace "doc" do
   desc "Generate README and preview in browser"
   task "readme" do
-    sh "rdoc -c utf8 README.rdoc && open doc/files/README_rdoc.html"
+    sh "rdoc -c utf8 README.rdoc && open doc/README_rdoc.html"
   end
 end

data/lib/pg_search.rb

@@ -1,32 +1,56 @@
 require "active_record"
-require "pg_search/configuration"
-require "pg_search/features"
-require "pg_search/normalizer"
-require "pg_search/scope"
-require "pg_search/scope_options"
-require "pg_search/version"
-#require "pg_search/railtie" if defined?(Rails) && defined?(Rails::Railtie)
+require "active_support/concern"
 
 module PgSearch
-  def self.included(base)
-    base.send(:extend, ClassMethods)
-  end
+  extend ActiveSupport::Concern
 
   module ClassMethods
     def pg_search_scope(name, options)
-      scope = PgSearch::Scope.new(name, self, options)
-      scope_method =
-        if respond_to?(:scope) && !protected_methods.map(&:to_s).include?('scope')
-          :scope # ActiveRecord 3.x
-        else
-          :named_scope # ActiveRecord 2.x
-        end
-
-      send(scope_method, name, scope.to_proc)
+      self.scope(
+        name,
+        PgSearch::Scope.new(name, self, options).to_proc
+      )
+    end
+
+    def multisearchable(options = {})
+      include PgSearch::Multisearchable
+      class_attribute :pg_search_multisearchable_options
+      self.pg_search_multisearchable_options = options
     end
   end
 
-  def rank
-    attributes['pg_search_rank'].to_f
+  module InstanceMethods
+    def rank
+      attributes['pg_search_rank'].to_f
+    end
+  end
+
+  class << self
+    def multisearch(query)
+      PgSearch::Document.search(query)
+    end
+
+    def disable_multisearch
+      Thread.current["PgSearch.enable_multisearch"] = false
+      yield
+    ensure
+      Thread.current["PgSearch.enable_multisearch"] = true
+    end
+
+    def multisearch_enabled?
+      Thread.current.key?("PgSearch.enable_multisearch") ? Thread.current["PgSearch.enable_multisearch"] : true
+    end
   end
 end
+
+require "pg_search/configuration"
+require "pg_search/document"
+require "pg_search/features"
+require "pg_search/multisearch"
+require "pg_search/multisearchable"
+require "pg_search/normalizer"
+require "pg_search/scope"
+require "pg_search/scope_options"
+require "pg_search/version"
+
+require "pg_search/railtie" if defined?(Rails)

data/lib/pg_search/configuration.rb

@@ -10,6 +10,14 @@ module PgSearch
       @model = model
     end
 
+    class << self
+      def alias(*strings)
+        name = Array.wrap(strings).compact.join("_")
+        # By default, PostgreSQL limits names to 32 characters, so we hash and limit to 32 characters.
+        "pg_search_#{Digest::SHA2.hexdigest(name)}"[0,32]
+      end
+    end
+
     def columns
       regular_columns + associated_columns
     end
@@ -49,6 +57,10 @@ module PgSearch
       Array(@options[:using])
     end
 
+    def order_within_rank
+      @options[:order_within_rank]
+    end
+
     private
 
     def default_options
@@ -56,7 +68,7 @@ module PgSearch
     end
 
     def assert_valid_options(options)
-      valid_keys = [:against, :ranked_by, :ignoring, :using, :query, :associated_against]
+      valid_keys = [:against, :ranked_by, :ignoring, :using, :query, :associated_against, :order_within_rank]
       valid_values = {
         :ignoring => [:accents]
       }