xapit 0.2.7 → 0.3.0

data/{CHANGELOG → CHANGELOG.rdoc}

@@ -1,3 +1,8 @@
+*0.3.0* August 17, 2011
+
+* Complete rewrite
+
+
 *0.2.7* (July 7th, 2009)
 
 * support punctuation in wildcard matching
@@ -5,14 +10,14 @@
 * fixing nested search calls
 
 * chain a separate search with "or_search" to find records matching either one
-  
+
   search("foo").or_search(:conditions => { :priority => 3 })
 
 
 *0.2.6* (July 6th, 2009)
 
 * search for field conditions in query string, only supported by ClassicQueryParser
-  
+
   search("age:17")
 
 * check if xapian database exists before removing

data/Gemfile

@@ -0,0 +1,19 @@
+source "http://rubygems.org"
+
+case ENV["MODEL_ADAPTER"]
+when nil, "active_record"
+  gem "sqlite3"
+  gem "activerecord", :require => "active_record"
+  gem "with_model"
+when "data_mapper"
+  gem "dm-core", "~> 1.0.2"
+  gem "dm-sqlite-adapter", "~> 1.0.2"
+  gem "dm-migrations", "~> 1.0.2"
+when "mongoid"
+  gem "bson_ext", "~> 1.1"
+  gem "mongoid", "~> 2.0.0.beta.20"
+else
+  raise "Unknown model adapter: #{ENV["MODEL_ADAPTER"]}"
+end
+
+gemspec

data/LICENSE

@@ -1,5 +1,5 @@
-Copyright (c) 2008 Ryan Bates
- 
+Copyright (c) 2011 Ryan Bates
+
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the
 "Software"), to deal in the Software without restriction, including
@@ -7,10 +7,10 @@ without limitation the rights to use, copy, modify, merge, publish,
 distribute, sublicense, and/or sell copies of the Software, and to
 permit persons to whom the Software is furnished to do so, subject to
 the following conditions:
- 
+
 The above copyright notice and this permission notice shall be
 included in all copies or substantial portions of the Software.
- 
+
 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
 EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
 MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND

data/README.rdoc

@@ -1,119 +1,71 @@
 = Xapit
 
-Xapit (pronounced "zap it") is a high level interface for working with a Xapian database.
-
-Note: This project is early in development and the API is subject to change.
+Xapit (pronounced "zap it") is a Ruby gem for doing full text searching through {Xapian}[http://xapian.org/]. This project was recently rewritten and is not yet ready for production. Please try it out and let me know if there are any issues.
 
 
 == Install
 
-If you haven't already, first install Xapian and the Xapian Bindings for Ruby.
-http://wiki.github.com/Overbryd/acts_as_xapian/installation
+First, install Xapian with the Ruby bindings. The easiest way is through {Homebrew}[http://mxcl.github.com/homebrew/].
+
+  brew install xapian --ruby
 
-To install as a Rails plugin, run this command.
+Note: this is tied to the current version of Ruby you have installed. If you use RVM be certain you're in the correct version. See {Installing Xapian}[https://github.com/ryanb/xapit/wiki/Xapian-Installation] for other methods.
 
-  script/plugin install git://github.com/ryanb/xapit.git
+Next add Xapit to your Gemfile and run the +bundle+ command.
 
-Or to install as a gem in Rails first add this to config/environment.rb.
+  gem "xapit"
 
-  config.gem 'xapit'
-  
-And then install the gem and run the generator.
+Then run the install generator.
 
-  sudo rake gems:install
-  script/generate xapit
+  rails g xapit:install
 
-Important: only run the generator script on a gem install, not for the plugin.
+This creates a {configuration file}[https://github.com/ryanb/xapit/wiki/Configuration] and a rackup file for production use.
 
 
-== Setup
+== Index
 
-Simply call "xapit" in the model and pass a block to define the indexed attributes.
+To make a model searchable you must define an index through the +xapit+ method. Here is an example of the four methods you can call there:
 
   class Article < ActiveRecord::Base
-    xapit do |index|
-      index.text :name, :content
-      index.field :category_id
-      index.facet :author_name, "Author"
-      index.sortable :id, :category_id
+    xapit do
+      text :name, :content
+      field :category_id
+      sortable :id, :created_at
+      facet :author_name, "Author"
     end
   end
 
-First we index "name" and "content" attributes for full text searching. The "category_id" field is indexed for :conditions searching. The "author_name" is indexed as a facet with "Author" being the display name of the facet. See the facets section below for details. Finally the "id" and "category_id" attributes are indexed as sortable attributes so they can be included in the :order option in a search.
-
-Because the indexing happens in Ruby these attributes do no have to be database columns. They can be simple Ruby methods. For example, the "author_name" attribute mentioned above can be defined like this.
-
-  def author_name
-    author.name
-  end
-
-This way you can create a completely custom facet by simply defining your own method. Multiple facet options or field values per record are supported if you return an array.
-
-  def author_names
-    authors.map(&:name) # => ["John", "Bob"]
-  end
-
-Finally, you can pass any find options to the xapit method to determine what gets indexed or improve performance with eager loading or a different batch size.
-
-  xapit(:batch_size => 100, :include => :author, :conditions => { :visible => true })
-
-You can specify a :weight option to give a text attribute more importance. This will cause search terms matching that attribute to have a higher rank. The default weight is 1. Decimal (0.5) weight values are not supported.
-
-  index.text :name, :weight => 10
-
-
-== Index
+This indexes the model to be searched in a variety of ways (shown below). See the {Indexing}[https://github.com/ryanb/xapit/wiki/Indexing] wiki page for details.
 
-To perform the indexing, run the xapit:index rake task.
+The index will automatically be updated when records are added or removed. You can regenerate the index manually to fill it with any existing records.
 
   rake xapit:index
 
-It can also be triggered through Ruby code using this command.
-
-  Xapit.remove_database
-  Xapit.index_all
-
-You may want to trigger this via a cron job on a recurring schedule (i.e. every day) to update the Xapian database. However it will only take effect after the Rails application is restarted because the Xapian database is stored in memory.
-
-There are two projects in development to help improve this reindexing.
-
-* http://github.com/ryanb/xapit-sync/tree/master
-* http://github.com/ryanb/xapit-server/tree/master
-
 
 == Search
 
-You can then perform a search on the model.
-  
-  # perform a simple full text search
+Use the +search+ class method to perform a full text search on the index. This returns a Xapit scope where additional scoping methods can be called similar to Active Record scopes.
+
+  # simple full text search
   @articles = Article.search("phone")
-  
-  # add pagination if you're using will_paginate
-  @articles = Article.search("phone", :per_page => 10, :page => params[:page])
-  
-  # search based on indexed fields
-  @articles = Article.search("phone", :conditions => { :category_id => params[:category_id] })
-  
-  # search for multiple negative conditions (doesn't match 3, 5, or 8)
-  @articles = Article.search(:not_conditions => { :category_id => [3, 5, 8] })
-  
-  # search for range of conditions by number
-  @articles = Article.search(:conditions => { :released_at => 2.years.ago..Time.now })
-  
-  # manually sort based on any number of indexed fields, sort defaults to most relevant
-  @articles = Article.search("phone", :order => [:category_id, :id], :descending => true)
-  
-  # basic boolean matching is supported
+
+  # full text search with basic boolean matching
   @articles = Article.search("phone OR fax NOT email")
 
+  # pagination works with kaminari and will_paginate
+  @articles = Article.search("phone").page(10).per(20)
 
-You can also search all indexed models through Xapit.search.
+  # search based on a specific field
+  @articles = Article.search("phone").where(:category_id => params[:category_id])
 
-  # search all indexed models
-  @records = Xapit.search("phone")
+  # search for multiple negative conditions (doesn't match 3, 5, or 8)
+  @articles = Article.search("phone").not_where(:category_id => [3, 5, 8])
 
+  # search for range of conditions by number
+  @articles = Article.search.where(:released_at => 2.years.ago..Time.now)
 
-== Results
+  # order based on sortable fields, sorting defaults to most relevant
+  @articles = Article.search("phone").order(:created_at, :descending => true)
 
 Simply iterate through the returned set to display the results.
 
@@ -124,15 +76,19 @@ Simply iterate through the returned set to display the results.
 
 The "xapit_relevance" holds a percentage (between 0 and 100) determining how relevant the given document is to the user's search query.
 
+See the {Searching}[https://github.com/ryanb/xapit/wiki/Searching] wiki page for more details.
+
 
 == Spelling
 
-If the searched term isn't found, but it is similar to another term then it will show up as a spelling suggestion.
+Spelling suggestions are available when there is a simlarly indexed term.
 
   <% if @articles.spelling_suggestion %>
-    Did you mean <%= link_to h(@articles.spelling_suggestion), :overwrite_params => { :keywords => @articles.spelling_suggestion } %>?
+    Did you mean <%= link_to @articles.spelling_suggestion, :overwrite_params => { :keywords => @articles.spelling_suggestion } %>?
   <% end %>
 
+Note: the spelling feature is disabled by default in the test environment because it uses an in-memory database.
+
 
 == Facets
 
@@ -146,52 +102,49 @@ Facets allow you to further filter the result set based on certain attributes.
     <% end %>
   <% end %>
 
-The to_param method is defined on option to return an identifier which will be passed through the URL. Use this in the search.
+The facet option is passed in through the URL which you can add to the search.
 
-  Article.search("phone", :facets => params[:facets])
+  Article.search("phone").with_facets(params[:facets])
 
 You can also list the applied facets along with a remove link.
 
   <% for option in @articles.applied_facet_options %>
-    <%=h option.name %>
+    <%= option.name %>
     <%= link_to "remove", :overwrite_params => { :facets => option } %>
   <% end %>
 
+See the {Facets}[https://github.com/ryanb/xapit/wiki/Facets] wiki page for more details.
 
-== Config
 
-When installing Xapit as a Rails plugin, an initializer file is automatically created to setup. It looks like this.
+== Testing
 
-  Xapit.setup(:database_path => "#{Rails.root}/db/xapiandb")
+To use Xapit in the test environment, you will need to reload the configuration before each test. For example, you can do this in RSpec:
 
-There are many other options you can pass into here. This is a more advanced configuration setting which changes the stemming language, disables spelling, and changes the indexer and parser to a classic variation. The classic ones use Xapian's built-in term generator and query parser instead of the ones offered by Xapit.
+  config.before(:each) do
+    Xapit.reload
+  end
 
-  Xapit.setup(
-    :database_path => "#{Rails.root}/db/external/xapiandb",
-    :spelling => false,
-    :stemming => "german",
-    :indexer => ClassicIndexer,
-    :query_parser => ClassicQueryParser
-  )
+Xapit is also disabled by default in the test environment so you can enable it on a per-test basis. This way it doesn't slow down tests which don't need Xapit search.
 
+  Xapit.enable
+  # create records and test searching functionality
 
-== Adapters
+If you have existing records in the database you want to search (maybe loaded by fixtures) you can index those.
 
-Adapters are used to support multiple ORMs since not everyone uses ActiveRecord. The right adapter is detected automatically so you should not have to do anything for popular ORMs. However if your ORM is not supported then it is very easy to make your own adapter. See AbstractAdapter class for details.
+  Xapit.index(Article, Comment) # or whatever models you need
 
+By default, the test environment keeps the Xapian database in memory which makes it much faster and easier to reset. The problem is that the spelling suggestion feature does not work with in-memory databases, so it is disabled in the test environment by default. All of this can be customized in the {configuration file}[https://github.com/ryanb/xapit/wiki/Configuration].
 
-== Bug Reports
 
-If you have found a bug to report or a feature to request, please add it to the GitHub issue tracker if it is not there already.
+== Production
 
-http://github.com/ryanb/xapit/issues
+The default Xapit setup works well in development because there is only one instance of the Rails app running. However in production you will need to move Xapit to a separate process so all of the instances can communicate to it. To do this, start up the rackup file provided. You will likely want to put this behind a server such as Passenger.
 
+  rackup xapit.ru
 
-== Development
+Now when you start up your Rails app in production it will use this server. You can configure the server URL in the {configuration file}[https://github.com/ryanb/xapit/wiki/Configuration].
 
-This project can be found on github at the following URL.
 
-http://github.com/ryanb/xapit
+== Bug Reports
 
-If you would like to contribute to this project, please fork the 
-repository and send me a pull request.
+If you have found a bug to report or a feature to request, please add it to the {GitHub issue tracker}[https://github.com/ryanb/xapit/issues] if it is not there already. Feel free to treat it as a mailing list and discuss new ideas or bring up any confusion you may have.

data/Rakefile

@@ -1,15 +1,16 @@
 require 'rubygems'
 require 'rake'
-require 'echoe'
+require 'cucumber/rake/task'
+require 'rspec/core/rake_task'
 
-Echoe.new('xapit', '0.2.7') do |p|
-  p.summary        = "Ruby library for interacting with Xapian, a full text search engine."
-  p.description    = "Ruby library for interacting with Xapian, a full text search engine."
-  p.url            = "http://github.com/ryanb/xapit"
-  p.author         = 'Ryan Bates'
-  p.email          = "ryan (at) railscasts (dot) com"
-  p.ignore_pattern = ["tmp/**/*", "spec/tmp/**/*"]
-  p.development_dependencies = []
+desc "Run RSpec"
+RSpec::Core::RakeTask.new do |t|
+  t.verbose = false
 end
 
-Dir["#{File.dirname(__FILE__)}/tasks/*.rb"].sort.each { |ext| load ext }
+desc "Run Cucumber"
+Cucumber::Rake::Task.new do |t|
+  t.cucumber_opts = %w[--format progress]
+end
+
+task :default => [:spec, :cucumber]

data/features/facets.feature

@@ -1,90 +1,101 @@
-Background:
-  Given an empty database at "tmp/xapiandatabase"
+Feature: Facets
 
-Scenario: List All Facets
-  Given the following indexed records
-    | name | age |
-    | John | 23  |
-    | John | 17  |
-    | Jack | 17  |
-  When I query for ""
-  Then I should have the following facets
-    | facet | option | count |
-    | Name  | Jack   | 1     |
-    | Name  | John   | 2     |
-    | Age   | 17     | 2     |
-    | Age   | 23     | 1     |
+  Scenario: List All Facets
+    Given the following indexed records
+      | name | age |
+      | John | 23  |
+      | John | 17  |
+      | Jack | 17  |
+    When I query for ""
+    Then I should have the following facets
+      | facet | option | count |
+      | Name  | Jack   | 1     |
+      | Name  | John   | 2     |
+      | Age   | 17     | 2     |
+      | Age   | 23     | 1     |
 
-Scenario: List Matching Facets
-  Given the following indexed records
-    | name | age |
-    | John | 23  |
-    | John | 17  |
-    | Jack | 17  |
-  When I query for "John"
-  Then I should have the following facets
-    | facet | option | count |
-    | Age   | 17     | 1     |
-    | Age   | 23     | 1     |
+  Scenario: List Matching Facets
+    Given the following indexed records
+      | name | age |
+      | John | 23  |
+      | John | 17  |
+      | Jack | 17  |
+    When I query for "John"
+    Then I should have the following facets
+      | facet | option | count |
+      | Age   | 17     | 1     |
+      | Age   | 23     | 1     |
 
-Scenario: List Applied Facets
-  Given the following indexed records
-    | name | age |
-    | John | 23  |
-    | Jane | 17  |
-    | Jack | 17  |
-  When I query facets "0c93ee1-078661c"
-  Then I should have the following applied facets
-    | facet | option |
-    | Age   | 17     |
-    | Name  | Jane   |
+  Scenario: List Multiple Facets Applied to One Record
+    Given the following indexed records
+      | name       |
+      | John, Jack |
+      | John       |
+      | Joe, Jack  |
+    When I query for ""
+    Then I should have the following facets
+      | facet | option | count |
+      | Name  | Jack   | 2     |
+      | Name  | Joe    | 1     |
+      | Name  | John   | 2     |
 
-Scenario: List Multiple Facets Applied to One Record
-  Given the following indexed records
-    | name       |
-    | John, Jack |
-    | John       |
-    | Joe, Jack  |
-  When I query for ""
-  Then I should have the following facets
-    | facet | option | count |
-    | Name  | Jack   | 2     |
-    | Name  | Joe    | 1     |
-    | Name  | John   | 2     |
+  Scenario: Ignore Facets That Do Not Narrow Down List
+    Given the following indexed records
+      | name       |
+      | John, Jack |
+      | John       |
+    When I query for ""
+    Then I should have the following facets
+      | facet | option | count |
+      | Name  | Jack   | 1     |
 
-Scenario: Ignore Facets That Do Not Narrow Down List
-  Given the following indexed records
-    | name       |
-    | John, Jack |
-    | John       |
-  When I query for ""
-  Then I should have the following facets
-    | facet | option | count |
-    | Name  | Jack   | 1     |
+  Scenario: Query for One Facet
+    Given the following indexed records
+      | name | age |
+      | John | 23  |
+      | Jane | 17  |
+      | Jack | 17  |
+    When I query facets "9f33345"
+    Then I should find records named "Jane, Jack"
 
-Scenario: Query for One Facet
-  Given the following indexed records
-    | name | age |
-    | John | 23  |
-    | Jane | 17  |
-    | Jack | 17  |
-  When I query facets "0c93ee1"
-  Then I should find records named "Jane, Jack"
+  Scenario: Query for Two Facets
+    Given the following indexed records
+      | name | age |
+      | John | 23  |
+      | Jane | 17  |
+      | Jack | 17  |
+    When I query facets "9f33345-9a10ff2"
+    Then I should find records named "Jane"
 
-Scenario: Query for Two Facets
-  Given the following indexed records
-    | name | age |
-    | John | 23  |
-    | Jane | 17  |
-    | Jack | 17  |
-  When I query facets "0c93ee1-078661c"
-  Then I should find records named "Jane"
+  Scenario: Query for Facets with Keywords
+    Given the following indexed records
+      | name | age |
+      | John | 23  |
+      | Jane | 17  |
+      | Jack | 17  |
+    When I query "Jane" with facets "9f33345"
+    Then I should find record named "Jane"
 
-Scenario: Query for Facets with Keywords
-  Given the following indexed records
-    | name | age |
-    | John | 23  |
-    | Jane | 17  |
-    | Jack | 17  |
-  When I query "Jane" with facets "0c93ee1"
-  Then I should find record named "Jane"
+  Scenario: List Applied Facets
+    Given the following indexed records
+      | name | age |
+      | John | 23  |
+      | Jane | 17  |
+      | Jack | 17  |
+    When I query facets "9f33345-9a10ff2"
+    Then I should have the following applied facets
+      | facet | option |
+      | Age   | 17     |
+      | Name  | Jane   |
+
+  Scenario: Includes applied facets in facet params
+    Given the following indexed records
+      | name | age |
+      | John | 23  |
+      | Jane | 17  |
+      | Jack | 17  |
+    When I query facets "9f33345"
+    Then I should have the following facets
+      | facet | option | count | param           |
+      | Name  | Jack   | 1     | 9f33345-bff6e04 |
+      | Name  | Jane   | 1     | 9f33345-9a10ff2 |