xapit 0.1.0 → 0.2.0

data/CHANGELOG

@@ -0,0 +1,27 @@
+*0.2.0* (June 12th, 2009)
+
+* sort numeric attributes properly
+
+* adding xapit generator for use with gem install
+
+* adding Xapit.setup/remove_database instead of through Xapit::Config
+
+* adding foundation for adapters to support other ORMs
+
+* facet options which do not narrow down results are ignored
+
+* performance improvements when fetching records and displaying facets
+
+* search conditions can take an array or range of values
+
+  :conditions => { :priority => [2, 5, 7] }
+  :conditions => { :priority => 2..7 }
+
+* adding not_condition option in search method
+
+* fixing spelling suggestions when used with stemming
+
+
+*0.1.0* (May 28th, 2009)
+
+* initial release

data/Manifest

@@ -1,12 +1,18 @@
+CHANGELOG
 features/facets.feature
 features/finding.feature
 features/indexing.feature
+features/sorting.feature
 features/step_definitions/common_steps.rb
 features/step_definitions/xapit_steps.rb
+features/suggestions.feature
 features/support/env.rb
 features/support/xapit_helpers.rb
 init.rb
 install.rb
+lib/xapit/adapters/abstract_adapter.rb
+lib/xapit/adapters/active_record_adapter.rb
+lib/xapit/adapters/data_mapper_adapter.rb
 lib/xapit/collection.rb
 lib/xapit/config.rb
 lib/xapit/facet.rb
@@ -21,8 +27,14 @@ lib/xapit/query.rb
 lib/xapit/query_parsers/abstract_query_parser.rb
 lib/xapit/query_parsers/classic_query_parser.rb
 lib/xapit/query_parsers/simple_query_parser.rb
+lib/xapit/rake_tasks.rb
 lib/xapit.rb
 LICENSE
+Manifest
+rails_generators/xapit/templates/setup_xapit.rb
+rails_generators/xapit/templates/xapit.rake
+rails_generators/xapit/USAGE
+rails_generators/xapit/xapit_generator.rb
 Rakefile
 README.rdoc
 spec/spec_helper.rb
@@ -126,6 +138,8 @@ spec/tmp/xapiandbase/spelling.DB
 spec/tmp/xapiandbase/termlist.baseA
 spec/tmp/xapiandbase/termlist.baseB
 spec/tmp/xapiandbase/termlist.DB
+spec/xapit/adapters/active_record_adapter_spec.rb
+spec/xapit/adapters/data_mapper_adapter_spec.rb
 spec/xapit/collection_spec.rb
 spec/xapit/config_spec.rb
 spec/xapit/facet_blueprint_spec.rb
@@ -157,7 +171,9 @@ tmp/xapiandatabase/spelling.DB
 tmp/xapiandatabase/termlist.baseA
 tmp/xapiandatabase/termlist.baseB
 tmp/xapiandatabase/termlist.DB
+tmp/xapiandatabase/value.baseA
 tmp/xapiandatabase/value.baseB
+tmp/xapiandatabase/value.DB
 tmp/xapiandb/flintlock
 tmp/xapiandb/iamflint
 tmp/xapiandb/postlist.baseA
@@ -175,4 +191,3 @@ tmp/xapiandb/termlist.DB
 tmp/xapiandb/value.baseB
 TODO
 uninstall.rb
-Manifest

data/README.rdoc

@@ -14,7 +14,16 @@ To install as a Rails plugin, run this command.
 
   script/plugin install git://github.com/ryanb/xapit.git
 
-Note: this will soon also be available as a Ruby gem, but not yet.
+Or to install as a gem in Rails first add this to config/environment.rb.
+
+  config.gem 'xapit'
+  
+And then install the gem and run the generator.
+
+  sudo rake gems:install
+  script/generate xapit
+
+Important: only run the generator script on a gem install, not for the plugin.
 
 
 == Setup
@@ -61,9 +70,16 @@ To perform the indexing, run the xapit:index rake task.
 
 It can also be triggered through Ruby code using this command.
 
-  Xapit::Config.remove_database
+  Xapit.remove_database
   Xapit.index_all
 
+You may want to trigger this via a cron job on a recurring schedule (i.e. every day) to update the Xapian database. However it will only take effect after the Rails application is restarted because the Xapian database is stored in memory.
+
+There are two projects in development to help improve this reindexing.
+
+http://github.com/ryanb/xapit-sync/tree/master
+http://github.com/ryanb/xapit-server/tree/master
+
 
 == Search
 
@@ -78,6 +94,12 @@ You can then perform a search on the model.
   # search based on indexed fields
   @articles = Article.search("phone", :conditions => { :category_id => params[:category_id] })
   
+  # search for multiple negative conditions (doesn't match 3, 5, or 8)
+  @articles = Article.search(:not_conditions => { :category_id => [3, 5, 8] })
+  
+  # search for range of conditions by number
+  @articles = Article.search(:conditions => { :released_at => 2.years.ago..Time.now })
+  
   # manually sort based on any number of indexed fields, sort defaults to most relevant
   @articles = Article.search("phone", :order => [:category_id, :id], :descending => true)
   
@@ -140,11 +162,11 @@ You can also list the applied facets along with a remove link.
 
 When installing Xapit as a Rails plugin, an initializer file is automatically created to setup. It looks like this.
 
-  Xapit::Config.setup(:database_path => "#{Rails.root}/db/xapiandb")
+  Xapit.setup(:database_path => "#{Rails.root}/db/xapiandb")
 
 There are many other options you can pass into here. This is a more advanced configuration setting which changes the stemming language, disables spelling, and changes the indexer and parser to a classic variation. The classic ones use Xapian's built-in term generator and query parser instead of the ones offered by Xapit.
 
-  Xapit::Config.setup(
+  Xapit.setup(
     :database_path => "#{Rails.root}/db/external/xapiandb",
     :spelling => false,
     :stemming => "german",
@@ -153,24 +175,16 @@ There are many other options you can pass into here. This is a more advanced con
   )
 
 
-== Outside of Rails
-
-Xapit can be used outside of Rails too. You'll just need to include the membership module:
-
-  class Product # not Active Record
-    include Xapit::Membership
-  
-    xapit # ...
-  end
+== Adapters
 
-This class is expected to respond to "find_each" (Product.find_each) which is used to iterate through the records when indexing and "find" for fetching a record by "id".
+Adapters are used to support multiple ORMs since not everyone uses ActiveRecord. The right adapter is detected automatically so you should not have to do anything for popular ORMs. However if your ORM is not supported then it is very easy to make your own adapter. See AbstractAdapter class for details.
 
 
 == Bug Reports
 
 If you have found a bug to report or a feature to request, please add it to the GitHub issue tracker if it is not there already.
 
-http://github.com/ryanb/xapit/issues/
+http://github.com/ryanb/xapit/issues
 
 
 == Development

data/Rakefile

@@ -2,7 +2,7 @@ require 'rubygems'
 require 'rake'
 require 'echoe'
 
-Echoe.new('xapit', '0.1.0') do |p|
+Echoe.new('xapit', '0.2.0') do |p|
   p.summary        = "Ruby library for interacting with Xapian, a full text search engine."
   p.description    = "Ruby library for interacting with Xapian, a full text search engine."
   p.url            = "http://github.com/ryanb/xapit"

data/features/facets.feature

@@ -44,8 +44,47 @@ Scenario: List Multiple Facets Applied to One Record
     | name       |
     | John, Jack |
     | John       |
+    | Joe, Jack  |
   When I query for ""
   Then I should have the following facets
     | facet | option | count |
-    | Name  | Jack   | 1     |
+    | Name  | Jack   | 2     |
+    | Name  | Joe    | 1     |
     | Name  | John   | 2     |
+
+Scenario: Ignore Facets That Do Not Narrow Down List
+  Given the following indexed records
+    | name       |
+    | John, Jack |
+    | John       |
+  When I query for ""
+  Then I should have the following facets
+    | facet | option | count |
+    | Name  | Jack   | 1     |
+
+Scenario: Query for One Facet
+  Given the following indexed records
+    | name | age |
+    | John | 23  |
+    | Jane | 17  |
+    | Jack | 17  |
+  When I query facets "0c93ee1"
+  Then I should find records named "Jane, Jack"
+
+Scenario: Query for Two Facets
+  Given the following indexed records
+    | name | age |
+    | John | 23  |
+    | Jane | 17  |
+    | Jack | 17  |
+  When I query facets "0c93ee1-078661c"
+  Then I should find records named "Jane"
+
+Scenario: Query for Facets with Keywords
+  Given the following indexed records
+    | name | age |
+    | John | 23  |
+    | Jane | 17  |
+    | Jack | 17  |
+  When I query "Jane" with facets "0c93ee1"
+  Then I should find record named "Jane"

data/features/finding.feature

@@ -36,25 +36,6 @@ Scenario: Query for Page 2
   Then I should find 1 record
   And I should have 3 records total
 
-Scenario: Query for One Facet
-  Given the following indexed records
-    | name | age |
-    | John | 23  |
-    | Jane | 17  |
-    | Jack | 17  |
-  When I query facets "0c93ee1"
-  Then I should find records named "Jane, Jack"
-
-
-Scenario: Query for Two Facets
-  Given the following indexed records
-    | name | age |
-    | John | 23  |
-    | Jane | 17  |
-    | Jack | 17  |
-  When I query facets "0c93ee1-078661c"
-  Then I should find records named "Jane"
-
 Scenario: Query for All Records Class Agnostic
   Given indexed records named "John, Jane"
   When I query for "John" on Xapit
@@ -70,50 +51,25 @@ Scenario: Query Matching Not Query
   When I query for "John NOT Smith"
   Then I should find records named "John Johnson"
 
-Scenario: Query for Facets with Keywords
+Scenario: Unicode characters in search
+  Given indexed records named "über cool, uber hot"
+  When I query for "über"
+  Then I should find records named "über cool"
+
+Scenario: Query Field Not Matching
   Given the following indexed records
     | name | age |
     | John | 23  |
     | Jane | 17  |
     | Jack | 17  |
-  When I query "Jane" with facets "0c93ee1"
-  Then I should find record named "Jane"
-
-Scenario: Query for All Records Sorted by Name
-  Given indexed records named "Zebra, Apple, Banana"
-  When I query "" sorted by name
-  Then I should find records named "Apple, Banana, Zebra"
+  When I query "age" not matching "17"
+  Then I should find records named "John"
 
-Scenario: Query for All Records Sorted by Age then Name
+Scenario: Query Range of Integer
   Given the following indexed records
-    | name   | age |
-    | Banana | 23  |
-    | Zebra  | 17  |
-    | Apple  | 17  |
-  When I query "" sorted by age, name
-  Then I should find records named "Apple, Zebra, Banana"
-
-Scenario: Query for All Records Sorted by Name Descending
-  Given indexed records named "Zebra, Apple, Banana"
-  When I query "" sorted by name descending
-  Then I should find records named "Zebra, Banana, Apple"
-
-Scenario: Spelling suggestion
-  Given indexed records named "Zebra, Apple, Bike"
-  When I query for "zerba bike aple"
-  Then I should have "zebra bike apple" as a spelling suggestion
-
-Scenario: Match similar words with stemming
-  Given indexed records named "flies, fly, glider"
-  When I query for "flying"
-  Then I should find records named "flies, fly"
-
-Scenario: Find similar records
-  Given indexed records named "Jason John Smith, John Doe, Jason Smith, Jacob Johnson"
-  When I query for similar records for "Jason John Smith"
-  Then I should find records named "Jason Smith, John Doe"
-
-Scenario: Unicode characters in search
-  Given indexed records named "über cool, uber hot"
-  When I query for "über"
-  Then I should find records named "über cool"
+    | name | age |
+    | John | 8   |
+    | Jane | 13  |
+    | Jack | 24  |
+  When I query "age" between 8 and 15
+  Then I should find records named "John, Jane"

data/features/sorting.feature

@@ -0,0 +1,29 @@
+Background:
+  Given an empty database at "tmp/xapiandatabase"
+
+Scenario: Query for All Records Sorted by Name
+  Given indexed records named "Zebra, Apple, Banana"
+  When I query "" sorted by name
+  Then I should find records named "Apple, Banana, Zebra"
+
+Scenario: Query for All Records Sorted by Age then Name
+  Given the following indexed records
+    | name   | age |
+    | Banana | 23  |
+    | Zebra  | 17  |
+    | Apple  | 17  |
+  When I query "" sorted by age, name
+  Then I should find records named "Apple, Zebra, Banana"
+
+Scenario: Query for All Records Sorted by Name Descending
+  Given indexed records named "Zebra, Apple, Banana"
+  When I query "" sorted by name descending
+  Then I should find records named "Zebra, Banana, Apple"
+
+Scenario: Query for Records Sorted Numerically
+  Given the following indexed records
+    | name   | age |
+    | Banana | 9   |
+    | Zebra  | 10  |
+  When I query "" sorted by age, name
+  Then I should find records named "Banana, Zebra"

data/features/step_definitions/xapit_steps.rb

@@ -1,10 +1,10 @@
 Given /^I configured the database to be saved at "([^\"]*)"$/ do |path|
-  Xapit::Config.setup(:database_path => File.dirname(__FILE__) + "/../../#{path}")
+  Xapit.setup(:database_path => File.dirname(__FILE__) + "/../../#{path}")
 end
 
 Given /^an empty database at "([^\"]*)"$/ do |path|
-  Xapit::Config.setup(:database_path => File.dirname(__FILE__) + "/../../#{path}")
-  Xapit::Config.remove_database
+  Xapit.setup(:database_path => File.dirname(__FILE__) + "/../../#{path}")
+  Xapit.remove_database
   XapitMember.delete_all
 end
 
@@ -70,6 +70,14 @@ When /^I query "([^\"]*)" matching "([^\"]*)"$/ do |field, value|
   @records = XapitMember.search("", :conditions => { field.to_sym => value })
 end
 
+When /^I query "([^\"]*)" not matching "([^\"]*)"$/ do |field, value|
+  @records = XapitMember.search("", :not_conditions => { field.to_sym => value })
+end
+
+When /^I query "([^\"]*)" between (\d+) and (\d+)$/ do |field, beginning, ending|
+  @records = XapitMember.search("", :conditions => { field.to_sym => beginning.to_i..ending.to_i })
+end
+
 When /^I query page ([0-9]+) at ([0-9]+) per page$/ do |page, per_page|
   @records = XapitMember.search("", :page => page, :per_page => per_page.to_i)
 end

data/features/suggestions.feature

@@ -0,0 +1,17 @@
+Background:
+  Given an empty database at "tmp/xapiandatabase"
+
+Scenario: Spelling suggestion
+  Given indexed records named "Zebra, Apple, Bike"
+  When I query for "zerba bike aple"
+  Then I should have "zebra bike apple" as a spelling suggestion
+
+Scenario: Match similar words with stemming
+  Given indexed records named "flies, fly, glider"
+  When I query for "flying"
+  Then I should find records named "flies, fly"
+
+Scenario: Find similar records
+  Given indexed records named "Jason John Smith, John Doe, Jason Smith, Jacob Johnson"
+  When I query for similar records for "Jason John Smith"
+  Then I should find records named "Jason Smith, John Doe"

data/features/support/xapit_helpers.rb

@@ -1,6 +1,6 @@
 module XapitHelpers
   def create_records(records,  perform_index = true)
-    Xapit::Config.remove_database
+    Xapit.remove_database
     XapitMember.delete_all
     XapitMember.xapit do |index|
       records.first.keys.each do |attribute|

data/install.rb

@@ -1,9 +1,8 @@
-path = "#{Rails.root}/config/initializers/setup_xapit.rb"
-unless File.exist? path
-  puts "Adding setup_xapit.rb initializer."
-  File.open(path, "w") do |f|
-    f.write <<-EOS
-Xapit::Config.setup(:database_path => "\#{Rails.root}/db/xapiandb")
-EOS
-  end
+require 'ftools'
+
+source = File.join(File.dirname(__FILE__), '/rails_generators/xapit/templates/setup_xapit.rb')
+destination = "#{Rails.root}/config/initializers/setup_xapit.rb"
+unless File.exist? destination
+  puts "Adding config/initializers/setup_xapit.rb"
+  File.copy(source, destination)
 end

data/lib/xapit.rb

@@ -4,9 +4,8 @@ require 'xapian'
 
 # Looking for more documentation? A good place to start is Xapit::Membership
 module Xapit
-  
   # Index all membership classes with xapit defined. Delegates to Xapit::IndexBlueprint.
-  # You will likely want to call Xapit::Config.remove_database before this.
+  # You will likely want to call Xapit.remove_database before this.
   def self.index_all(*args, &block)
     IndexBlueprint.index_all(*args, &block)
   end
@@ -21,6 +20,36 @@ module Xapit
   def self.search(*args)
     Collection.new(nil, *args)
   end
+  
+  # Setup configuration options. The following options are supported.
+  # 
+  # <tt>:database_path</tt>:     Where the database is stored.
+  # <tt>:stemming</tt>:          The language to use for stemming, defaults to "english".
+  # <tt>:spelling</tt>:          True or false to enable/disable spelling, defaults to true.
+  # <tt>:indexer</tt>:           Class to handle the indexing, defaults to SimpleIndexer.
+  # <tt>:query_parser</tt>:      Class to handle the parsing, defaults to ClassicQueryParser.
+  # <tt>:breadcrumb_facets</tt>: Use breadcrumb mode for applied facets. See Collection#applied_facet_options for details.
+  #
+  def self.setup(*args)
+    Config.setup(*args)
+  end
+  
+  # Removes the configured database file and clears the stored one in memory.
+  def self.remove_database
+    Config.remove_database
+  end
+  
+  def self.serialize_value(value)
+    if value.kind_of?(Date)
+      Xapian.sortable_serialise(value.to_time.to_i)
+    elsif value.kind_of?(Time)
+      Xapian.sortable_serialise(value.to_i)
+    elsif value.kind_of?(Numeric) || value.to_s =~ /^[0-9]+$/
+      Xapian.sortable_serialise(value.to_f)
+    else
+      value.to_s.downcase
+    end
+  end
 end
 
 require File.dirname(__FILE__) + '/xapit/membership'
@@ -37,3 +66,6 @@ require File.dirname(__FILE__) + '/xapit/query_parsers/classic_query_parser'
 require File.dirname(__FILE__) + '/xapit/indexers/abstract_indexer'
 require File.dirname(__FILE__) + '/xapit/indexers/simple_indexer'
 require File.dirname(__FILE__) + '/xapit/indexers/classic_indexer'
+require File.dirname(__FILE__) + '/xapit/adapters/abstract_adapter'
+require File.dirname(__FILE__) + '/xapit/adapters/active_record_adapter'
+require File.dirname(__FILE__) + '/xapit/adapters/data_mapper_adapter'