dusen 0.2.2 → 0.3.0

data/.gitignore

@@ -5,5 +5,6 @@ pkg
 app_root/log/*
 Gemfile.lock
 spec/shared/app_root/db/*.db
+spec/shared/app_root/config/database.yml
 
 

data/.travis.yml

@@ -7,3 +7,6 @@ script: rake all:bundle all:spec
 notifications:
   email:
     - fail@makandra.de
+branches:
+  only:
+    - master

data/README.md

@@ -1,20 +1,15 @@
-Dusen - Maps Google-like queries to ActiveRecord scopes
-=======================================================
+Dusen [![Build Status](https://secure.travis-ci.org/makandra/dusen.png?branch=master)](https://travis-ci.org/makandra/dusen)
+======
 
-[![Build Status](https://secure.travis-ci.org/makandra/dusen.png?branch=master)](https://travis-ci.org/makandra/dusen)
+Comprehensive search solution for ActiveRecord and MySQL
+--------------------------------------------------------
 
-Dusen gives your ActiveRecord models a DSL to process Google-like queries like:
+Dusen lets you search ActiveRecord model when all you have is MySQL (no Solr, Sphinx, etc.). Here's what Dusen does for you:
 
-    some words
-    "a phrase of words"
-    filetype:pdf
-    a mix of words "and phrases" and qualified:fields
-
-Dusen tokenizes these queries for you and feeds them through simple mappers that
-convert a token to an ActiveRecord scope chain.
-This process is packaged in a class method `.search`:
-
-    Contact.search('makandra software "Ruby on Rails" city:augsburg')
+1. It takes a text query in Google-like syntax (e.g. `some words "a phrase" filetype:pdf`)
+2. It parses the query into individual tokens.
+3. It lets you define simple mappers that convert a token to an ActiveRecord scope chain. Mappers can match tokens using ActiveRecord's `where` or perform full text searches with either [LIKE queries](#processing-full-text-search-queries-with-like-queries) or [FULLTEXT indexes](#processing-full-text-queries-with-fulltext-indexes) (see [performance analysis](https://makandracards.com/makandra/12813-performance-analysis-of-mysql-s-fulltext-indexes-and-like-queries-for-full-text-search)).
+4. It gives your model a method `Model.search('some query')` that performs all of the above and returns an ActiveRecord scope chain.
 
 
 Installation
@@ -28,25 +23,25 @@ Now run `bundle install` and restart your server.
 
 
 
-Processing text queries
------------------------
+Processing full text search queries with LIKE queries
+-----------------------------------------------------
 
 This describes how to define a search syntax that processes queries
-of words and phrases:
+of words and phrases, e.g. `coworking fooville "market ave"`.
+
+Under the hood the search will be performed using [LIKE queries](http://dev.mysql.com/doc/refman/5.0/en/string-comparison-functions.html#operator_like), which are [fast enough](https://makandracards.com/makandra/12813-performance-analysis-of-mysql-s-fulltext-indexes-and-like-queries-for-full-text-search) for medium sized data sets. Once your data outgrows LIKE queries, Dusen lets you [migrate to FULLTEXT indexes](#processing-full-text-queries-with-fulltext-indexes), which perform better but come at some added complexity.
 
-    coworking fooville "market ave"
 
+### Setup and usage
 
 Our example will be a simple address book:
 
     class Contact < ActiveRecord::Base
-
       validates_presence_of :name, :street, :city, :name
-
     end
 
 
-We will now teach `Contact` to process a text query like this:
+In order to teach `Contact` how to process a text query, use the `search_syntax` and `search_by :text` macros:
 
     class Contact < ActiveRecord::Base
 
@@ -54,9 +49,9 @@ We will now teach `Contact` to process a text query like this:
 
       search_syntax do
 
-        search_by :text do |scope, phrase|
+        search_by :text do |scope, phrases|
           columns = [:name, :street, :city, :email]
-          scope.where_like(columns => phrase)
+          scope.where_like(columns => phrases)
         end
 
       end
@@ -64,36 +59,52 @@ We will now teach `Contact` to process a text query like this:
     end
 
 
-Note how you will only ever need to deal with a single token (word or phrase) and return a scope that matches the token.
-Dusen will take care how these scopes will be chained together.
+Dusen will tokenize the query into individual phrases and call the `search_by :text` block with it. The block is expected to return a scope that filters by the given phrases.
 
-If we now call `Contact.search('coworking fooville "market ave"')`
-the block supplied to `search_by` is called once per token:
+If, for example, we call `Contact.search('coworking fooville "market ave"')`
+the block supplied to `search_by :text` is called with the following arguments:
 
-1. `|Contact, 'coworking'|`
-2. `|Contact.where_like(columns => 'coworking'), 'fooville'|`
-3. `|Contact.where_like(columns => 'coworking').where_like(columns => 'fooville'), 'market ave'|`
+    |Contact, ['coworking', 'fooville', 'market ave']|
 
 
 The resulting scope chain is your `Contact` model filtered by
 the given query:
 
      > Contact.search('coworking fooville "market ave"')
-    => Contact.where_like(columns => 'coworking').where_like(columns => 'fooville').where_like(columns => 'market ave')
+    => Contact.where_like([:name, :street, :city, :email] => ['coworking', 'fooville', 'market ave'])
 
+### What where_like does under the hood
 
 Note that `where_like` is an utility method that comes with the Dusen gem.
-It takes one or more column names and a phrase and generates an SQL fragment
-like this:
-
-    contacts.name LIKE "%coworking%" OR contacts.street LIKE "%coworking%" OR contacts.email LIKE "%coworking%" OR contacts.email LIKE "%coworking%"
+It takes one or more column names and one or more phrases and generates an SQL fragment
+that looks roughly like the following:
+
+    ( contacts.name LIKE "%coworking%"    OR 
+      contacts.street LIKE "%coworking%"  OR 
+      contacts.email LIKE "%coworking%"   OR 
+      contacts.email LIKE "%coworking%" ) AND
+    ( contacts.name LIKE "%fooville%"     OR 
+      contacts.street LIKE "%fooville%"   OR 
+      contacts.email LIKE "%fooville%"    OR 
+      contacts.email LIKE "%fooville%" )  AND
+    ( contacts.name LIKE "%market ave%"   OR 
+      contacts.street LIKE "%market ave%" OR 
+      contacts.email LIKE "%market ave%"  OR 
+      contacts.email LIKE "%market ave%" )
 
 
 Processing queries for qualified fields
 ---------------------------------------
 
-Let's give `Contact` a way to explictely search for a contact's email address, without
-going through a full text search. We do this by adding additional `search_by` instructions
+Google supports queries like `filetype:pdf` that filters records by some criteria without performing a full text search. Dusen gives you a simple way to support such search syntax.
+
+### Setup and usage
+
+We now want to process a qualified query like `email:foo@bar.com` to
+explictily search for a contact's email address, without going through
+a full text search.
+
+We can learn this syntax by adding a `search_by :email` instruction
 to our model:
 
     search_syntax do
@@ -102,11 +113,11 @@ to our model:
         ...
       end
 
-     search_by :email do |scope, email|
-       scope.where(:email => email)
-     end
+      search_by :email do |scope, email|
+        scope.where(:email => email)
+      end
 
-   end
+    end
 
 
 The result is this:
@@ -115,17 +126,134 @@ The result is this:
     => Contact.where(:email => 'foo@bar.com')
 
 
-Feel free to combine text tokens and field tokens:
+Note that you can combine text tokens and field tokens:
 
      > Contact.search('fooville email:foo@bar.com')
     => Contact.where_like(columns => 'fooville').where(:email => 'foo@bar.com')
 
 
+Processing full text queries with FULLTEXT indexes
+---------------------------------------------------
+
+### When do I need this?
+
+As your number of records grows larger, you might outgrow a full text implementation that uses LIKE (see [performance analysis](https://makandracards.com/makandra/12813-performance-analysis-of-mysql-s-fulltext-indexes-and-like-queries-for-full-text-search)). For this case Dusen ships with an alternative full text search solution using MySQL FULLTEXT indexes that scale much better.
+
+### Understanding the MyISAM limitation
+
+Using this feature comes at some added complexity so you should first check if search performance is actually a problem for you. If all you have is a few thousand records with a few dozen words each, changes are your views render many times longer than a LIKE query takes to finish. Always measure before optimizing.
+
+Currently stable MySQL versions only allow FULLTEXT indexes on MyISAM tables (this will change in MySQL 5.6). You don't however want to migrate your models to MyISAM tables because of their many limitations (poor crash recovery, no transactions, etc.).
+
+To work around this, Dusen uses a separate MyISAM table `search_texts` to index your searchable text. Each row in your model's table will be shadowed by a corresponding row in `search_texts`. Dusen will automatically create, update and destroy these shadow rows as your model records change.
+
+
+### Setup and usage
+
+First we need to create the `search_texts` table. Since we're on Rails, we will do this using a migration. So enter `rails generate migration CreateSearchText` and use the following code as the migration's content:
+
+    class CreateSearchText < ActiveRecord::Migration
+    
+      def self.up
+        create_table :search_texts, :options => 'ENGINE=MyISAM' do |t|
+          t.integer :source_id
+          t.string :source_type
+          t.boolean :stale
+          t.text :words
+        end
+        add_index :search_texts, [:source_type, :source_id] # for updates
+        add_index :search_texts, [:source_type, :stale] # for refreshs
+        execute 'CREATE FULLTEXT INDEX fulltext_index_words ON search_texts (words)'
+      end
+    
+      def self.down
+        drop_table :search_texts
+      end
+    
+    end
+
+Since we're using some MySQL-specific options we also need to change the format of your `db/schema.rb` from Ruby to SQL (you will get a `db/schema.sql` instead). You can configure this in your `application.rb` (`environment.rb` in Rails 2):
+
+    config.active_record.schema_format = :sql
+
+
+We now need to your model which text to index. We do this using the `search_text` macro and returning the searchable text:
+
+    class Contact < ActiveRecord::Base
+
+      search_syntax
+
+      search_text do
+        [name, street, city, email]
+      end
+
+      end
+
+    end
+
+You can return any object or array of objects. Dusen will stringify the return value and index those words. Note that indexed words do not need to be fields of your model:
+
+    search_text do
+      [email, city, author.screen_name, ('client' if client?)
+    end
+
+You're done! You can now search `Contact` using the same API you used with [LIKE queries](#processing-full-text-search-queries-with-like-queries):
+
+    Contact.search('coworking fooville "market ave"')
+
+Note that you didn't need to teach your model how to process text queries by defining a mapper with `search_by :text { ... }`. The `search_text` macro defines this mapper for you.
+
+Also note that if you migrated an existing table to FULLTEXT search, you need to [build the index the first time](#building-the-index-for-existing-records).
+
+
+### Building the index for existing records
+
+If you migrated an existing table to FULLTEXT search, you must build the index for all existing records:
+
+    Model.all.each(&index_search_text)
+
+You only need to do this once. Dusen will automatically index all further changes to your records.
+
+### Indexing changes in associated records
+
+Dusen lets you index words from associated models. When you do this you need to reindex the indexed model whenever an associated record changes, or else the indexed text will be out of date.
+
+As an example we will associate `Contact` with an `Organization` and make it searchable by the name of her `Organization`:
+
+    class Contact < ActiveRecord::Base
+      
+      belongs_to :organization
+      
+      search_syntax
+
+      search_text do
+        [name, email, organization && organization.name]
+      end
+
+    end
+
+To make sure contacts will reindex when the organization changes its name, use the `part_of_search_text_for` macro:
+
+    class Organization < ActiveRecord::Base
+
+      has_many :contacts
+ 
+      part_of_search_text_for do
+        contacts
+      end
+
+    end
+
+All records returned by `part_of_search_text_for` will be reindexed when the organization is changed or destroyed.
+
 
 Programmatic access without DSL
 -------------------------------
 
-You can use Dusen's functionality without using the ActiveRecord DSL or the search scope. Here are some method calls to get you started:
+You can use Dusen's functionality without using the ActiveRecord DSL or the `search` method.
+**Please note that at this time we cannot yet commit to the API of these internal methods**. So don't get mad when stuff breaks after you update the gem.
+
+Here are some method calls to get you started:
 
     Contact.search_syntax # => #<Dusen::Syntax>
 
@@ -144,14 +272,15 @@ You can use Dusen's functionality without using the ActiveRecord DSL or the sear
 Development
 -----------
 
-Test applications for various Rails versions lives in `spec`. You can run specs from the project root by saying:
-
-  bundle exec rake all:spec
+- Test applications for various Rails versions lives in `spec`.
+- You need to create a MySQL database and put credentials into `spec/shared/app_root/config/database.yml`.
+- You can bundle all test applications by saying `bundle exec rake all:bundle`
+- You can run specs from the project root by saying `bundle exec rake all:spec`.
 
 If you would like to contribute:
 
 - Fork the repository.
-- Push your changes **with specs**.
+- Push your changes **with passing specs**.
 - Send me a pull request.
 
 I'm very eager to keep this gem leightweight and on topic. If you're unsure whether a change would make it into the gem, [talk to me beforehand](mailto:henning.koch@makandra.de).

data/documents/fulltext_vs_like_benchmark/all_records_and_scope/fulltext.csv

@@ -0,0 +1,22 @@
+1000; 0.394796924591064
+2000; 0.391613826751709
+3000; 0.414114084243774
+4000; 0.406150894165039
+5000; 0.426540040969849
+6000; 0.449329986572266
+7000; 0.496056623458862
+8000; 0.480993881225586
+9000; 0.503967409133911
+10000; 0.523442239761353
+11000; 0.53717339515686
+12000; 0.564517946243286
+13000; 0.574517812728882
+14000; 0.585553913116455
+15000; 0.609066152572632
+16000; 0.618757972717285
+17000; 0.672317543029785
+18000; 0.667384099960327
+19000; 0.672960157394409
+20000; 0.696838903427124
+21000; 0.709187746047974
+

data/documents/fulltext_vs_like_benchmark/all_records_and_scope/like.csv

@@ -0,0 +1,22 @@
+1000; 0.0629708099365234
+2000; 0.101030836105347
+3000; 0.15950831413269
+4000; 0.725705051422119
+5000; 0.755747880935669
+6000; 0.791393089294434
+7000; 0.813040456771851
+8000; 0.842384424209595
+9000; 0.880409164428711
+10000; 0.901688318252563
+11000; 0.926919441223145
+12000; 0.963594560623169
+13000; 0.988169240951538
+14000; 1.02734631538391
+15000; 1.0335246181488
+16000; 1.07445472717285
+17000; 1.12085759162903
+18000; 1.12942158699036
+19000; 1.14870473861694
+20000; 1.19215731620789
+21000; 1.22716112136841
+

data/documents/fulltext_vs_like_benchmark/benchmark.rb

@@ -0,0 +1,70 @@
+reload!; Benchmark.measure { puts "..."; Note.search('rails test method').all; puts "DONE." }
+
+
+
+
+
+
+
+
+
+reload!
+
+site = Site.first
+note_scope = site.notes; nil
+note_ids = note_scope.collect_ids; nil
+puts "Benchmarking #{note_ids.size} notes"
+
+batch_size = 1000
+amount = batch_size
+max_amount = note_ids.size
+
+while amount < max_amount
+
+  batch_ids = note_ids[0, amount]
+  batch_scope = note_scope.scoped(:conditions => { :id => batch_ids })
+  
+  times = []
+  25.times do
+    times << (Benchmark.realtime { batch_scope.search('rails test method').count })
+  end
+  
+  time = times.sum.to_f / times.size
+  
+  puts "#{amount}; #{time}"
+
+  amount += batch_size
+
+end
+
+
+#------------------------------------------
+
+
+reload!
+
+Note.delete_all('site_id <> 1')
+
+batch_size = 1000
+amount = 20000
+
+while Note.count >= batch_size
+
+  while Note.count > amount
+    Note.last.destroy
+  end
+
+  times = []
+  25.times do
+    times << (Benchmark.realtime { Note.search('rails test method').count })
+  end
+  
+  time = times.sum.to_f / times.size
+  
+  puts "#{amount}; #{time}"
+
+  amount -= batch_size
+
+end
+
+

data/documents/fulltext_vs_like_benchmark/exact_number_of_records/fulltext.csv

@@ -0,0 +1,22 @@
+20000; 0.226401567459106
+19000; 0.210501832962036
+18000; 0.191384611129761
+17000; 0.188002090454102
+16000; 0.17433009147644
+15000; 0.168737888336182
+14000; 0.156399478912354
+13000; 0.141394910812378
+12000; 0.136331949234009
+11000; 0.129038143157959
+10000; 0.108856143951416
+9000; 0.100853176116943
+8000; 0.0817105197906494
+7000; 0.0714373302459717
+6000; 0.0654478454589844
+5000; 0.0597659683227539
+4000; 0.0383996486663818
+3000; 0.0362687492370605
+2000; 0.0331569194793701
+1000; 0.00965866088867188
+0; 0.00276429176330566
+

data/documents/fulltext_vs_like_benchmark/exact_number_of_records/like.csv

@@ -0,0 +1,21 @@
+Records;LIKE;FULLTEXT
+1000;0.0211521244;0.0096586609
+2000;0.0543338013;0.0331569195
+3000;0.0776431179;0.0362687492
+4000;0.0866408348;0.0383996487
+5000;0.1185267925;0.0597659683
+6000;0.1424144745;0.0654478455
+7000;0.1785503292;0.0714373302
+8000;0.2093483353;0.0817105198
+9000;0.2463747692;0.1008531761
+10000;0.2596423721;0.108856144
+11000;0.2996723843;0.1290381432
+12000;0.3262492657;0.1363319492
+13000;0.3312761688;0.1413949108
+14000;0.3623173046;0.1563994789
+15000;0.3892039013;0.1687378883
+16000;0.4076850605;0.1743300915
+17000;0.4328162861;0.1880020905
+18000;0.4592379665;0.1913846111
+19000;0.489655714;0.210501833
+20000;0.5181032562;0.2264015675

data/dusen.gemspec

@@ -7,7 +7,7 @@ Gem::Specification.new do |s|
   s.authors = ["Henning Koch"]
   s.email = 'henning.koch@makandra.de'
   s.homepage = 'https://github.com/makandra/dusen'
-  s.summary = 'Maps Google-like queries (words, "phrases", qualified:fields) to ActiveRecord scope chains'
+  s.summary = 'Comprehensive full text search for ActiveRecord and MySQL'
   s.description = s.summary
 
   s.files         = `git ls-files`.split("\n")

data/lib/dusen/active_record/base_ext.rb

@@ -0,0 +1,104 @@
+# encoding: utf-8
+
+module Dusen
+  module ActiveRecord
+    module BaseExt
+      module ClassMethods
+
+        def search_syntax(&dsl)
+          @search_syntax ||= Dusen::Syntax.new
+          Dusen::Description.parse_syntax(@search_syntax, &dsl) if dsl
+          unless singleton_class.method_defined?(:search)
+            singleton_class.send(:define_method, :search) do |query_string|
+              @search_syntax.search(self, query_string)
+            end
+          end
+          @search_syntax
+        end
+
+        def search_text?
+          !!@has_search_text
+        end
+
+        def index_search_texts
+          Dusen::ActiveRecord::SearchText.rewrite_all_invalid(self)
+        end
+
+        def search_text(&text)
+
+          @has_search_text = true
+
+          has_one :search_text, :as => :source, :dependent => :destroy, :class_name => '::Dusen::ActiveRecord::SearchText', :inverse_of => :source
+
+          after_create :create_search_text
+
+          after_update :invalidate_search_text
+
+          define_method :index_search_text do
+            new_text = instance_eval(&text)
+            new_text = Array.wrap(new_text).flatten.collect(&:to_s).join(' ').gsub(/\s+/, ' ').strip
+            search_text || build_search_text
+            search_text.update_words!(new_text)
+            true
+          end
+
+          define_method :invalidate_search_text do
+            search_text.invalidate!
+            true
+          end
+
+          private
+
+          define_method :create_search_text do
+            build_search_text(:stale => true)
+            search_text.save!
+          end
+
+          search_syntax do
+            search_by :text do |scope, phrases|
+              Dusen::ActiveRecord::SearchText.match(scope, phrases)
+            end
+          end
+
+        end
+
+        def part_of_search_text_for(&associations)
+          invalidate_associations_method = "invalidate_search_text_for_associated_records"
+
+          before_save invalidate_associations_method
+          before_destroy invalidate_associations_method
+
+          private
+
+          define_method invalidate_associations_method do
+            associated_records = Array.wrap(instance_eval(&associations)).flatten
+            associated_records.each(&:invalidate_search_text)
+            true
+          end
+
+        end
+
+        def where_like(conditions)
+          scope = self
+          conditions.each do |field_or_fields, query|
+            fields = Array(field_or_fields).collect do |field|
+              Util.qualify_column_name(scope, field)
+            end
+            Array.wrap(query).each do |phrase|
+              phrase_with_placeholders = fields.collect { |field| "#{field} LIKE ?" }.join(' OR ')
+              like_expression = Dusen::Util.like_expression(phrase)
+              bindings = [like_expression] * fields.size
+              conditions = [ phrase_with_placeholders, *bindings ]
+              scope = Util.append_scope_conditions(scope, conditions)
+            end
+          end
+          scope
+        end
+
+      end
+    end
+  end
+end
+
+ActiveRecord::Base.send(:extend, Dusen::ActiveRecord::BaseExt::ClassMethods)
+

data/lib/dusen/active_record/search_text.rb

@@ -0,0 +1,50 @@
+module Dusen
+  module ActiveRecord
+    class SearchText < ::ActiveRecord::Base
+
+      self.table_name = 'search_texts'
+
+      belongs_to :source, :polymorphic => true, :inverse_of => :search_text
+
+      def update_words!(words)
+        update_attributes!(:words => words, :stale => false)
+      end
+
+      def invalidate!
+        update_attributes!(:stale => true)
+      end
+
+      def self.for_model(model)
+        Util.append_scope_conditions(scoped({}), :source_type => model.name)
+      end
+
+      def self.invalid
+        scoped(:conditions => { :stale => true })
+      end
+
+      def self.rewrite_all_invalid(model)
+        invalid_index_records = for_model(model).invalid
+        ids = Util.collect_column(invalid_index_records, :source_id)
+        Util.append_scope_conditions(model, :id => ids).each(&:index_search_text)
+      end
+
+      def self.match(model, words)
+        rewrite_all_invalid(model) if model.search_text?
+        Dusen::Util.append_scope_conditions(
+          model,
+          :id => matching_source_ids(model, words)
+        )
+      end
+
+      def self.matching_source_ids(model, words)
+        conditions = [
+          'MATCH (words) AGAINST (? IN BOOLEAN MODE)',
+          Dusen::Util.boolean_fulltext_query(words)
+        ]
+        matching_texts = Dusen::Util.append_scope_conditions(for_model(model), conditions)
+        Dusen::Util.collect_column(matching_texts, :source_id)
+      end
+
+    end
+  end
+end