xapian_db 0.3.1 → 0.3.2

data/CHANGELOG.md

@@ -0,0 +1,33 @@
+##0.3.2 (December 10th, 2010)
+
+Features:
+  - Moved the per_page option from Resultset.paginate to Database.search
+  - Added support for language settings (global and dynamic per object)
+  - Added support for xapian stemmers
+  - Removed the dependency to progressbar (but it is still used if available)
+  - Made the rebuild_xapian_index method silent by default (use :verbose => true to get status info)
+
+##0.3.1 (December 6th, 2010)
+
+Bugfixes:
+
+  - Fixed the gemspec
+
+##0.3.0 (December 4th, 2010)
+
+Features:
+  - Rails integration with configuration file (config/xapian_db.yml) and automatic setup
+
+##0.2.0 (December 1st, 2010)
+
+Features:
+
+  - Blueprint configuration extended
+  - Adapter for Datamapper
+  - Search by attribute names
+  - Search with wildcards
+  - Document attributes can carry anything that is serializable by YAML
+
+##0.1.0 (November 23th, 2010)
+
+Proof of concept, not really useful for real world usage

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2010 Gernot Kogler
+ 
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+ 
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+ 
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,168 @@
+= XapianDb
+
+== What's in the box?
+
+XapianDb is a ruby gem that combines features of nosql databases and fulltext indexing into one piece. The result: Rich documents and very fast queries.
+It is based on {Xapian}[http://xapian.org/], an efficient and powerful indexing library.
+The gem is in very early development and not production ready yet.
+
+XapianDb is inspired by {xapian-fu}[https://github.com/johnl/xapian-fu] and {xapit}[https://github.com/ryanb/xapit].
+Thank you John and Ryan for your great work. It helped me learning to understand the xapian library and I borrowed an idea
+or two from you ;-)
+
+== Why yet another indexing gem?
+
+In the good old days I used {ferret}[https://github.com/dbalmain/ferret] and {acts_as_ferret}[https://github.com/jkraemer/acts_as_ferret]
+as my fulltext indexing solution and everything was fine. But time moved on and Ferret didn't.
+
+So I started to rethink fulltext indexing again. I looked for something that
+
+* is under active development
+* is fast
+* is lightweight and easy to install / deploy
+* is framework and database agnostic and works with pure POROS (plain old ruby objects)
+* is configurable anywhere, not just inside the model classes; I think that index configurations should not be part of the domain model
+* supports document configuration at the class level, not the database level; each class has its own document structure
+* integrates with popular Ruby / Rails ORMs like ActiveRecord or Datamapper through a plugin architecture
+* returns rich document objects that do not necessarily need a database roundtrip to render the search results (but know how to get the underlying object, if needed)
+* updates the index realtime (no scheduled reindexing jobs)
+* supports all major features of a full text indexer, namely wildcards!!
+
+I tried hard but I couldn't find such a thing so I decided to write it, based on the Xapian library.
+
+== Getting started
+
+If you want to use xapian_db in a Rails app, you need Rails 3 or newer.
+
+=== Install Xapian if not already installed
+
+To use xapian_db, make sure you have the Xapian library and ruby bindings installed. At the time of this writing, the newest release of Xapian was 1.2.3. You might
+want to adjust the URLs below to load the most current release of Xapian.
+The example code works for OSX. On linux you might want to use wget instead of curl.
+
+A future release of xapian_db might include the Xapian binaries and make this step obsolete.
+
+==== Install Xapian
+  curl -O http://oligarchy.co.uk/xapian/1.2.3/xapian-core-1.2.3.tar.gz
+  tar xzvf xapian-core-1.2.3.tar.gz
+  cd xapian-core-1.2.3
+  ./configure --prefix=/usr/local
+  make
+  sudo make install
+
+==== Install ruby bindings for Xapian
+  curl -O http://oligarchy.co.uk/xapian/1.2.2/xapian-bindings-1.2.3.tar.gz
+  tar xzvf xapian-bindings-1.2.3.tar.gz
+  cd xapian-bindings-1.2.3
+  ./configure --prefix=/usr/local XAPIAN_CONFIG=/usr/local/bin/xapian-config
+  make
+  sudo make install
+
+The following steps assume that you are using xapian_db within a Rails app. The gem has an
+example in the examples folder that shows how you can use xapian_db without Rails.
+
+=== Configure your databases
+
+Without a config file, xapian_db creates the database in the db folder for development and production
+environments. If you are in the test environment, xapian_db creates an in memory database.
+It assumes you are using ActiveRecord.
+
+You can override these defaults by placing a config file named 'xapian_db.yml' into your config folder. Here's an example:
+
+  # XapianDb configuration
+  defaults: &defaults
+    adapter: datamapper # Avaliable adapters: :active_record, :datamapper
+    language: de        # Global language; can be overridden for specific blueprints
+
+  development:
+    database: db/xapian_db/development
+    <<: *defaults
+
+  test:
+    database: ":memory:" # Use an in memory database for tests
+    <<: *defaults
+
+  production:
+    database: db/xapian_db/production
+    <<: *defaults
+
+=== Configure an index blueprint
+
+In order to get your models indexed, you must configure a document blueprint for each class you want to index:
+
+  XapianDb::DocumentBlueprint.setup(Person) do |blueprint|
+    blueprint.attribute :name, :weight => 10
+    blueprint.attribute :first_name
+  end
+
+The example above assumes that you have a class <code>Person</code> with the methods <code>name</code> and <code>first_name</code>.
+Attributes will get indexed and are stored in the documents. You will be able to access the name and the first name in your search results.
+
+If you want to index additional data but do not need access to it from a search result, use the index method:
+
+  blueprint.index :remarks, :weight => 5
+
+If you config a class that has a language property, e.g.
+
+  class Person
+    attr_reader :language
+  end
+
+you can configure the blueprint to use the language of the object when indexing:
+
+  XapianDb::DocumentBlueprint.setup(Person) do |blueprint|
+    blueprint.language_method :language
+  end
+
+Don't worry if you have languages in your database that are not supported by Xapian. If the language is not supported, XapianDb
+will fall back to the global language configuration or none, if you haven't configured one.
+
+You can place this configuration anywhere, e.g. in an initializer.
+
+=== Update the index
+
+xapian_db injects some helper methods into your configured model classes that update the index automatically
+for you when you create, save or destroy models. If you already have models that should now go into the index,
+use the method <code>rebuild_xapian_index</code>:
+
+  Person.rebuild_xapian_index
+
+=== Query the index
+
+A simple query looks like this:
+
+  results = XapianDb.search("Foo")
+
+You can use wildcards and boolean operators:
+
+  results = XapianDb.search("Fo*" OR "Baz")
+
+You can query attributes:
+
+  results = XapianDb.search("name:Foo")
+
+=== Process the results
+
+<code>XapianDb.search</code> returns a resultset object. You can access the number of hits directly:
+
+  result.size # Very fast, does not load the resulting documents
+
+To access the found documents, get a page from the resultset:
+
+  page = result.paginate                              # Get the first page with 10 documents
+  page = result.paginate(:page => 2, :per_page => 20) # Get the second page page with documents 21-40
+
+Now you can access the documents:
+
+  doc = page.first
+  puts doc.domain_class       # Get the type of the indexed object, e.g. "Person"
+  puts doc.name               # We can access the configured attributes
+  person = doc.indexed_object # Access the object behind this doc (lazy loaded)
+
+
+== What to expect from future releases
+
+* multi language support (spelling correction, stop words)
+* facet support
+* will_paginate support
+* asynchronous index writer based on {resque}[https://github.com/defunkt/resque] for production environments

data/lib/xapian_db/adapters/active_record_adapter.rb

@@ -1,16 +1,27 @@
 # encoding: utf-8
 
-# Adapter for ActiveRecord. To use it, simply set it as the
-# default for any DocumentBlueprint or a specific DocumentBlueprint
-
 module XapianDb
   module Adapters
-     
+
+    # Adapter for ActiveRecord. To use it, configure it like this:
+    #   XapianDb::Config.setup do |config|
+    #     config.adapter :active_record
+    #   end
+    # This adapter does the following:
+    # - adds the instance method <code>xapian_id</code> to an indexed class
+    # - adds the class method <code>rebuild_xapian_index</code> to an indexed class
+    # - adds an after save block to an indexed class to update the index
+    # - adds an after destroy block to an indexed class to update the index
+    # - adds the instance method <code>indexed_object</code> to the module that will be included
+    #   in every found xapian document
+    # @author Gernot Kogler
+
      class ActiveRecordAdapter
 
        class << self
-         
+
          # Implement the class helper methods
+         # @param [Class] klass The class to add the helper methods to
          def add_class_helper_methods_to(klass)
 
            klass.instance_eval do
@@ -18,56 +29,45 @@ module XapianDb
              define_method(:xapian_id) do
                "#{self.class}-#{self.id}"
              end
-                          
+
            end
-         
+
            klass.class_eval do
-             
+
              # add the after save logic
              after_save do
                XapianDb::Config.writer.index(self)
              end
-             
+
              # add the after destroy logic
              after_destroy do
                XapianDb::Config.writer.unindex(self)
              end
 
              # Add a method to reindex all models of this class
-             define_singleton_method(:rebuild_xapian_index) do
-               # db = XapianDb::Adapters::ActiveRecordAdapter.database
-               # # First, delete all docs of this class
-               # db.delete_docs_of_class(klass)
-               # obj_count = klass.count
-               # puts "Reindexing #{obj_count} objects..."
-               # pbar = ProgressBar.new("Status", obj_count)
-               # klass.all.each do |obj|
-               #   doc = @@blueprint.indexer.build_document_for(obj)
-               #   db.store_doc(doc)
-               #   pbar.inc
-               # end
-               # db.commit
-               XapianDb::Config.writer.reindex_class(klass)
+             define_singleton_method(:rebuild_xapian_index) do |options={}|
+               XapianDb::Config.writer.reindex_class(klass, options)
              end
            end
-         
+
          end
-       
-         # Implement the document helper methods
+
+         # Implement the document helper methods on a module
+         # @param [Module] a_module The module to add the helper methods to
          def add_doc_helper_methods_to(a_module)
            a_module.instance_eval do
              # Implement access to the indexed object
              define_method :indexed_object do
-               return @indexed_object unless @indexed_object.nil? 
-               # retrieve the object id from data
+               return @indexed_object unless @indexed_object.nil?
+               # retrieve the class and id from data
                klass_name, id = data.split("-")
                klass = Kernel.const_get(klass_name)
                @indexed_object = klass.find(id.to_i)
              end
            end
-           
+
          end
-       
+
        end
      end
    end

data/lib/xapian_db/adapters/datamapper_adapter.rb

@@ -1,16 +1,26 @@
 # encoding: utf-8
 
-# Adapter for datamapper. To use it, simply set it as the
-# default for any DocumentBlueprint or a specific DocumentBlueprint
-
 module XapianDb
   module Adapters
-     
+
+    # Adapter for Datamapper. To use it, configure it like this:
+    #   XapianDb::Config.setup do |config|
+    #     config.adapter :datamapper
+    #   end
+    # This adapter does the following:
+    # - adds the instance method <code>xapian_id</code> to an indexed class
+    # - adds the class method <code>rebuild_xapian_index</code> to an indexed class
+    # - adds an after save block to an indexed class to update the index
+    # - adds an after destroy block to an indexed class to update the index
+    # - adds the instance method <code>indexed_object</code> to the module that will be included
+    #   in every found xapian document
+    # @author Gernot Kogler
      class DatamapperAdapter
 
        class << self
-         
+
          # Implement the class helper methods
+         # @param [Class] klass The class to add the helper methods to
          def add_class_helper_methods_to(klass)
 
            klass.instance_eval do
@@ -18,44 +28,45 @@ module XapianDb
              define_method(:xapian_id) do
                "#{self.class}-#{self.id}"
              end
-                          
+
            end
-         
+
            klass.class_eval do
-             
+
              # add the after save logic
              after :save do
                XapianDb::Config.writer.index(self)
              end
-             
+
              # add the after destroy logic
              after :destroy do
                XapianDb::Config.writer.unindex(self)
              end
 
              # Add a method to reindex all models of this class
-             define_singleton_method(:rebuild_xapian_index) do
-               XapianDb::Config.writer.reindex_class(self)
+             define_singleton_method(:rebuild_xapian_index) do |options={}|
+               XapianDb::Config.writer.reindex_class(self, options)
              end
            end
-         
+
          end
-       
-         # Implement the document helper methods
+
+         # Implement the document helper methods on a module
+         # @param [Module] a_module The module to add the helper methods to
          def add_doc_helper_methods_to(a_module)
            a_module.instance_eval do
              # Implement access to the indexed object
              define_method :indexed_object do
-               return @indexed_object unless @indexed_object.nil? 
-               # retrieve the object id from data
+               return @indexed_object unless @indexed_object.nil?
+               # retrieve the class and id from data
                klass_name, id = data.split("-")
                klass = Kernel.const_get(klass_name)
                @indexed_object = klass.get(id.to_i)
              end
            end
-           
+
          end
-       
+
        end
      end
    end

data/lib/xapian_db/adapters/generic_adapter.rb

@@ -1,22 +1,30 @@
 # encoding: utf-8
 
-# The generic adapter is a universal adapater that can be used for any
-# ruby class. To use the generic adapter (which is the default),
-# configure the expression that generates a unique key from your objects 
-# using the method 'unique_key'.
 module XapianDb
   module Adapters
-     
+
+    # The generic adapter is a universal adapater that can be used for any
+    # ruby class. To use the generic adapter (which is the default),
+    # configure the expression that generates a unique key from your objects
+    # using the method 'unique_key'.
+    # This adapter does the following:
+    # - adds the instance method <code>xapian_id</code> to an indexed class
+    # @author Gernot Kogler
     class GenericAdapter
 
       class << self
-        
+
         # Define the unique key expression
+        # @example Use the same unique expression like the active record adapter (assuming your objects have an id)
+        #   XapianDb::Adapters::GenericAdapter.unique_key do
+        #     "#{self.class}-#{self.id}"
+        #   end
         def unique_key(&block)
           @unique_key_block = block
         end
-        
+
         # Implement the class helper methods
+        # @param [Class] klass The class to add the helper methods to
         def add_class_helper_methods_to(klass)
           raise "Unique key is not configured for generic adapter!" if @unique_key_block.nil?
           expression = @unique_key_block
@@ -26,16 +34,17 @@ module XapianDb
             end
           end
         end
-        
-        # Implement the document helper methods
+
+        # Implement the document helper methods on a module. So far there are none
+        # @param [Module] a_module The module to add the helper methods to
         def add_doc_helper_methods_to(obj)
           # We have none so far
         end
-        
+
       end
 
     end
-     
+
   end
-   
+
 end

data/lib/xapian_db/config.rb

@@ -1,46 +1,61 @@
 # encoding: utf-8
 
-# Global configuration for XapianDb
-# @author Gernot Kogler
-
 module XapianDb
-  
+
+  # Global configuration for XapianDb
+  # @example A typical configuration might look like this:
+  #   XapianDb::Config.setup do |config|
+  #     config.adapter  :active_record
+  #     config.writer   :direct
+  #     config.database "db/xapian_db"
+  #   end
+  # @author Gernot Kogler
   class Config
 
-    # ---------------------------------------------------------------------------------   
+    # ---------------------------------------------------------------------------------
     # Singleton methods
-    # ---------------------------------------------------------------------------------   
+    # ---------------------------------------------------------------------------------
     class << self
 
+      # Configure global options for XapianDb.
+      # Availabe options:
+      # - adapter (see {XapianDb::Config#adapter})
+      # - writer (see {XapianDb::Config#writer})
+      # - database (see {XapianDb::Config#database})
+      # - language (see {XapianDb::Config#language})
+      # In a Rails app, you can configure XapianDb using a config file. See the README for the details
       def setup(&block)
         @config ||= Config.new
         yield @config if block_given?
       end
-      
+
       # Install delegates for the config instance variables
-      [:database, :adapter, :writer].each do |attr|
+      [:database, :adapter, :writer, :stemmer].each do |attr|
         define_method attr do
           @config.nil? ? nil : @config.instance_variable_get("@_#{attr}")
         end
-      end                
-    end  
+      end
+    end
 
-    # ---------------------------------------------------------------------------------   
+    # ---------------------------------------------------------------------------------
     # DSL methods
-    # ---------------------------------------------------------------------------------   
-    attr_reader :_database, :_adapter, :_writer
-    
-    # Set the database; either pass a path to the file system or
-    # the symbolic name "memory"
+    # ---------------------------------------------------------------------------------
+
+    #
+    attr_reader :_database, :_adapter, :_writer, :_stemmer
+
+    # Set the global database to use
+    # @param [String] path The path to the database. Either apply a file sytem path or :memory
+    #   for an in memory database
     def database(path)
-      
+
       # If the current database is a persistent database, we must release the
       # database and run the garbage collector to remove the write lock
       if @_database.is_a?(XapianDb::PersistentDatabase)
         @_database = nil
         GC.start
       end
-      
+
       if path.to_sym == :memory
         @_database = XapianDb.create_db
       else
@@ -52,31 +67,49 @@ module XapianDb
         end
       end
     end
-    
-    # Define the adapter to use; the following adapters are available:
-    # - :generic
-    # - :active_record
-    # - :datamapper
+
+    # Set the adapter
+    # @param [Symbol] type The adapter type; the following adapters are available:
+    #   - :generic ({XapianDb::Adapters::GenericAdapter})
+    #   - :active_record ({XapianDb::Adapters::ActiveRecordAdapter})
+    #   - :datamapper ({XapianDb::Adapters::DatamapperAdapter})
     def adapter(type)
       # We try to guess the adapter name
       @_adapter = XapianDb::Adapters.const_get("#{camelize(type.to_s)}Adapter")
     end
 
-    # Define the writer to use; the following adapters are available:
-    # - :direct
-    # More to come in a future release :-)
+    # Set the index writer
+    # @param [Symbol] type The writer type; the following adapters are available:
+    #   - :direct ({XapianDb::IndexWriters::DirectWriter})
+    #   More to come in a future release
     def writer(type)
       # We try to guess the writer name
       @_writer = XapianDb::IndexWriters.const_get("#{camelize(type.to_s)}Writer")
     end
-    
+
+    # Set the language
+    # @param [Symbol] lang The language; either apply the english name of the language
+    #   or the two letter IS639 code
+    # @example Use the english name of the language
+    #   XapianDb::Config.setup do |config|
+    #     config.language :german
+    #   end
+    # @example Use the iso code of the language
+    #   XapianDb::Config.setup do |config|
+    #     config.language :de
+    #   end
+    # see http://xapian.org/docs/apidoc/html/classXapian_1_1Stem.html for supported languages
+    def language(lang)
+      @_stemmer = Xapian::Stem.new(lang.to_s)
+    end
+
     private
-    
+
     # TODO: move this to a helper module
     def camelize(string)
       string.split(/[^a-z0-9]/i).map{|w| w.capitalize}.join
     end
-    
+
   end
-  
+
 end