xapian_db 0.3.4 → 0.4.0

data/CHANGELOG.md

@@ -1,3 +1,23 @@
+##0.4.0 (December 15th, 2010)
+
+Features:
+
+  - Simple facets implementation. The only facet supported is the class name of the indexed objects
+  - Support for sorting (only for class searches, not for global searches)
+  - The result of a search can be used with will_paginate
+
+Bugfixes:
+
+  - removed the class scope expression from the spelling suggestion when searching on a class
+  - keys of the attributes and index hashes are now sorted to be compatible with ruby 1.8 (which does
+    not preserve the order of the keys in a hash)
+  - Fixed the problem that blueprint configurations got lost after the first request in the development
+    env (Rails only). You should put your blueprints either into a class that is loaded by Rails or into
+    the file config/xapian_blueprints.rb wich is loaded automatically by XapianDb
+
+**Since the internal structure of the index has changed, you must reindex your objects if you come from an
+earlier version of XapianDb!**
+
 ##0.3.4 (December 14th, 2010)
 
 Features:
@@ -5,6 +25,9 @@ Features:
   - perform searches on indexed classes to scope the search to objects of a specific class
   - specify multiple blueprint attributes and index methods in one statement (without specifying options)
   - use blocks for complex attribute or index specifications
+
+Changes:
+
   - changed the implementation of Resultset.size to get more accurate estimations
   - changed the indexing of active_record or datamapper models when declared as attributes or indexes
     in a blueprint (indexes now all attributes of the object instead of using to_s)

data/README.rdoc

@@ -137,7 +137,8 @@ Use blocks for complex evaluations of attributes or indexed values:
     end
   end
 
-You can place this configuration anywhere, e.g. in an initializer.
+place these configurations either into the corrsepondig class or - I prefer to have the index configurations outside
+the models - into the file config/xapian_blueprints.rb.
 
 === Update the index
 
@@ -157,23 +158,30 @@ In verbose mode, XapianDb will use the progressbar gem if available.
 
 A simple query looks like this:
 
-  results = XapianDb.search("Foo")
+  results = XapianDb.search "Foo"
 
 You can use wildcards and boolean operators:
 
-  results = XapianDb.search("fo* or baz")
+  results = XapianDb.search "fo* or baz"
 
 You can query attributes:
 
-  results = XapianDb.search("name:Foo")
+  results = XapianDb.search "name:Foo"
 
 You can query objects of a specific class:
 
-  results = Person.search("name:Foo")
+  results = Person.search "name:Foo"
 
 If you want to override the default of 10 docs per page, pass the :per_page argument:
 
-  results = Person.search("name:Foo", :per_page => 20)
+  results = Person.search "name:Foo", :per_page => 20
+
+On class queries you can specifiy order options:
+
+  results = Person.search "name:Foo", :order => :first_name
+  results = Person.search "Fo*", :order => [:name, :first_name], :sort_decending => true
+
+Please note that the order option is not avaliable for global searches (XapianDb.search...)
 
 === Process the results
 
@@ -189,8 +197,8 @@ If you use a persistent database, the resultset may contain a spelling correctio
 
 To access the found documents, get a page from the resultset:
 
-  page = result.paginate            # Get the first page
-  page = result.paginate :page => 2 # Get the second page
+  page = results.paginate            # Get the first page
+  page = results.paginate :page => 2 # Get the second page
 
 Now you can access the documents:
 
@@ -199,9 +207,25 @@ Now you can access the documents:
   puts doc.name               # We can access the configured attributes
   person = doc.indexed_object # Access the object behind this doc (lazy loaded)
 
+Use a search result with will_paginate in a view:
+
+  <%= will_paginate @results %>
+
+=== Facets
+
+If you want to implement a simple drilldown for your searches, you can use a facets query:
+
+  search_expression = "Foo"
+  facets = XapianDb.facets(search_expression)
+  facets.each do |klass, count|
+    puts "#{klass.name}: #{count} hits"
+
+    # This is how you would get all documents for the facet
+    # doc = klass.search search_expression
+  end
+
+Facet support in XapianDb is very limited. The only available facet is the class of the indexed objects. In many cases that's all that's needed. Therefore, it is very likely that I won't add more options for facets (since I'm not a fan of facets anyway). However, if you desperately need advanced facets, let me know. Or - even better - send me a pull request with a nice implementation ;-)
 
 == What to expect from future releases
 
-* facet support
-* will_paginate support
 * asynchronous index writer based on {resque}[https://github.com/defunkt/resque] for production environments

data/lib/xapian_db/adapters/base_adapter.rb

@@ -19,8 +19,27 @@ module XapianDb
            klass.class_eval do
 
              # Add a method to search models of this class
-             define_singleton_method(:search) do |expression|
-               XapianDb.database.search "indexed_class:#{klass.name.downcase} and (#{expression})"
+             # Options:
+             # - :order          (Array<Symbol>) Accepts an array of attribute names for sorting
+             # - :sort_decending (Boolean)       Allows to reverse the sorting
+             define_singleton_method(:search) do |expression, options={}|
+               options = {:sort_decending => false}.merge options
+               class_scope = "indexed_class:#{klass.name.downcase}"
+
+               if options[:order]
+                 attr_names             = [options[:order]].flatten
+                 blueprint              = XapianDb::DocumentBlueprint.blueprint_for klass
+                 sort_indices           = attr_names.map {|attr_name| blueprint.value_index_for(attr_name)}
+                 options[:sort_indices] = attr_names.map {|attr_name| blueprint.value_index_for(attr_name)}
+               end
+               result = XapianDb.database.search "#{class_scope} and (#{expression})", options
+
+               # Remove the class scope from the spelling suggestion (if any)
+               unless result.spelling_suggestion.empty?
+                 scope_length = "#{class_scope} and (".size
+                 result.spelling_suggestion = result.spelling_suggestion.slice scope_length..-2
+               end
+               result
              end
 
            end

data/lib/xapian_db/database.rb

@@ -44,6 +44,10 @@ module XapianDb
     # @param [String] expression A valid search expression.
     # @param [Hash] options
     # @option options [Integer] :per_page (10) How many docs per page?
+    # @option options [Array<Integer>] :sort_indices (nil) An array of attribute indices to sort by. This
+    #   option is used internally by the search method implemented on configured classes. Do not use it
+    #   directly unless
+    #   you know what you do
     # @example Simple Query
     #   resultset = db.search("foo")
     # @example Wildcard Query
@@ -54,16 +58,45 @@ module XapianDb
     #   resultset = db.search("name:foo")
     # @return [XapianDb::Resultset] The resultset
     def search(expression, options={})
-      opts          = {:per_page => 10}.merge(options)
+      opts          = {:per_page => 10, :sort_decending => false}.merge(options)
       @query_parser ||= QueryParser.new(self)
       query         = @query_parser.parse(expression)
       enquiry       = Xapian::Enquire.new(reader)
       enquiry.query = query
+
+      if opts[:sort_indices]
+        raise ArgumentError.new("Sorting is available for class scoped searches only") unless expression =~ /^indexed_class:/
+        sorter = Xapian::MultiValueSorter.new
+        options[:sort_indices].each do |index|
+          sorter.add(index, opts[:sort_decending])
+        end
+        enquiry.set_sort_by_key_then_relevance(sorter)
+      end
+
       opts[:spelling_suggestion] = @query_parser.spelling_suggestion
       opts[:db_size]             = self.size
       Resultset.new(enquiry, opts)
     end
 
+    # A very simple implementation of facets limited to the class facets.
+    # @param [String] expression A valid search expression (see {#search} for examples).
+    # @return [Hash<Class, Integer>] A hash containing the classes and the hits per class
+    def facets(expression)
+      @query_parser        ||= QueryParser.new(self)
+      query                = @query_parser.parse(expression)
+      enquiry              = Xapian::Enquire.new(reader)
+      enquiry.query        = query
+      enquiry.collapse_key = 0 # Value 0 always contains the class name
+      facets = {}
+      enquiry.mset(0, self.size).matches.each do |match|
+        class_name = match.document.values[0].value
+        # We must add 1 to the collapse_count since collapse_count means
+        # "how many other matches are there?"
+        facets[Kernel.const_get(class_name)] = match.collapse_count + 1
+      end
+      facets
+    end
+
   end
 
   # In Memory database

data/lib/xapian_db/document_blueprint.rb

@@ -72,10 +72,48 @@ module XapianDb
     # Instance methods
     # ---------------------------------------------------------------------------------
 
+    # Get the names of all configured attributes sorted alphabetically
+    # @return [Array<Symbol>] The names of the attributes
+    def attribute_names
+      @attributes_hash.keys.sort
+    end
+
+    # Get the block associated with an attribute
+    # @param [Symbol] attribute The name of the attribute
+    # @return [Block] The block
+    def block_for_attribute(attribute)
+      @attributes_hash[attribute]
+    end
+
+    # Get the names of all configured index methods sorted alphabetically
+    # @return [Array<Symbol>] The names of the index_methods
+    def indexed_method_names
+      @indexed_methods_hash.keys.sort
+    end
+
+    # Get the options for an indexed method
+    # @param [Symbol] method The name of the method
+    # @return [IndexOptions] The options
+    def options_for_indexed_method(method)
+      @indexed_methods_hash[method]
+    end
+
+    # Return the value index of an attribute. Needed to access the value of an attribute
+    # from a Xapian document.
+    # @param [String, Symbol] attribute_name The name of the attribute
+    # @return [Integer] The value index of the attribute
+    # @raise ArgumentError if the attribute name is unknown
+    def value_index_for(attribute_name)
+      index = attribute_names.index attribute_name.to_sym
+      raise ArgumentError.new("Attribute #{attribute_name} unknown") unless index
+      # We add 1 because value slot 0 is reserved for the class name
+      index + 1
+    end
+
     # Return an array of all configured text methods in this blueprint
     # @return [Array<String>] All searchable prefixes
     def searchable_prefixes
-      @prefixes ||= indexed_methods_hash.keys
+      @prefixes ||= @indexed_methods_hash.keys
     end
 
     # Lazily build and return a module that implements accessors for each field
@@ -91,7 +129,7 @@ module XapianDb
         end
       end
 
-      @attributes_hash.keys.each_with_index do |field, index|
+      @attributes_hash.keys.sort.each_with_index do |field, index|
         @accessors_module.instance_eval do
           define_method field do
             YAML::load(self.values[index+1].value)
@@ -112,15 +150,6 @@ module XapianDb
     # configured class must implement this method.
     attr_reader :lang_method
 
-    # Collection of the configured attribute methods
-    # @return [Array<Symbol>] The names of the configured attribute methods
-    attr_reader :attributes_hash
-
-    # Collection of the configured index methods
-    # @return [Hash<Symbol, IndexOptions>] A hashtable containing all index methods as
-    #   keys and IndexOptions as values
-    attr_reader :indexed_methods_hash
-
     # Set / read a custom adapter.
     # Use this configuration option if you need a specific adapter for an indexed class.
     # If set, it overrides the globally configured adapter (see also {Config#adapter})
@@ -166,8 +195,8 @@ module XapianDb
       self.index(name, opts, &block) if opts[:index]
     end
 
-    # Add list of attributes to the blueprint. Attributes will be stored in the xapian documents an can be
-    # accessed from a search result.
+    # Add a list of attributes to the blueprint. Attributes will be stored in the xapian documents ans
+    # can be accessed from a search result.
     # @param [Array] attributes An array of method names that deliver the values for the attributes
     # @todo Make sure the name does not collide with a method name of Xapian::Document
     def attributes(*attributes)

data/lib/xapian_db/indexer.rb

@@ -35,15 +35,14 @@ module XapianDb
       # We store the class name of the object at position 0
       @xapian_doc.add_value(0, @obj.class.name)
 
-      pos = 1
-      @blueprint.attributes_hash.each do |attribute, block|
+      @blueprint.attribute_names.each do |attribute|
+        block = @blueprint.block_for_attribute attribute
         if block
           value = @obj.instance_eval(&block)
         else
           value = @obj.send(attribute)
         end
-        @xapian_doc.add_value(pos, value.to_yaml)
-        pos += 1
+        @xapian_doc.add_value(@blueprint.value_index_for(attribute), value.to_yaml)
       end
     end
 
@@ -68,7 +67,26 @@ module XapianDb
       @xapian_doc.add_term("C#{@obj.class}")
 
 
-      @blueprint.indexed_methods_hash.each do |method, options|
+      # @blueprint.indexed_methods_hash.keys.sort.each do |method|
+      #   options = @blueprint.indexed_methods_hash[method]
+      #   if options.block
+      #     obj = @obj.instance_eval(&options.block)
+      #   else
+      #     obj = @obj.send(method)
+      #   end
+      #   unless obj.nil?
+      #     values = get_values_to_index_from obj
+      #     values.each do |value|
+      #       # Add value with field name
+      #       term_generator.index_text(value.to_s.downcase, options.weight, "X#{method.upcase}")
+      #       # Add value without field name
+      #       term_generator.index_text(value.to_s.downcase)
+      #     end
+      #   end
+      # end
+
+      @blueprint.indexed_method_names.each do |method|
+        options = @blueprint.options_for_indexed_method method
         if options.block
           obj = @obj.instance_eval(&options.block)
         else
@@ -84,6 +102,7 @@ module XapianDb
           end
         end
       end
+
     end
 
     private

data/lib/xapian_db/railtie.rb

@@ -40,5 +40,11 @@ module XapianDb
 
     end
 
+    config.to_prepare do
+      # Load a blueprint config if there is one
+      blueprints_file_path = "#{Rails.root}/config/xapian_blueprints.rb"
+      load blueprints_file_path if File.exist?(blueprints_file_path)
+    end
+
   end
 end

data/lib/xapian_db/resultset.rb

@@ -4,10 +4,13 @@ module XapianDb
 
   # The resultset encapsulates a Xapian::Query object and allows paged access
   # to the found documents.
+  # The resultset is compatible with will_paginate.
   # @example Process the first page of a resultsest
   #   resultset.paginate(:page => 1, :per_page => 10).each do |doc|
   #     # do something with the xapian document
   #   end
+  # @example Use the resultset and will_paginate in a view
+  #   <%= will_paginate resultset %>
   # @author Gernot Kogler
   class Resultset
 
@@ -15,9 +18,17 @@ module XapianDb
     # @return [Integer]
     attr_reader :size
 
+    # The number of pages
+    # @return [Integer]
+    attr_reader :total_pages
+
+    # The current page
+    # @return [Integer]
+    attr_reader :current_page
+
     # The spelling corrected query (if a language is configured)
     # @return [String]
-    attr_reader :spelling_suggestion
+    attr_accessor :spelling_suggestion
 
     # Constructor
     # @param [Xapian::Enquire] enquiry a Xapian query result (see http://xapian.org/docs/apidoc/html/classXapian_1_1Enquire.html)
@@ -29,15 +40,30 @@ module XapianDb
       # To get more accurate results, we pass the doc count to the mset method
       @size                = enquiry.mset(0, options[:db_size]).matches_estimated
       @spelling_suggestion = options[:spelling_suggestion]
-      @per_page = options[:per_page]
+      @per_page            = options[:per_page]
+      @total_pages         = (@size / @per_page.to_f).ceil
+      @current_page        = 1
     end
 
     # Paginate the result
     # @param [Hash] opts Options for the persistent database
     # @option opts [Integer] :page (1) The page to access
     def paginate(opts={})
-      options = {:page => 1}.merge(opts)
-      build_page(options[:page])
+      options       = {:page => 1}.merge(opts)
+      @current_page = options[:page].to_i
+      build_page(@current_page)
+    end
+
+    # The previous page number
+    # @return [Integer] The number of the previous page or nil, if we are at page 1
+    def previous_page
+      @current_page > 1 ? (@current_page - 1) : nil
+    end
+
+    # The next page number
+    # @return [Integer] The number of the next page or nil, if we are at the last page
+    def next_page
+      @current_page < @total_pages ? (@current_page + 1): nil
     end
 
     private
@@ -45,6 +71,7 @@ module XapianDb
     # Build a page of Xapian documents
     # @return [Array<Xapian::Document>] An array of xapian documents
     def build_page(page)
+      page.nil? ? page = 1 : page = page.to_i
       docs = []
       offset = (page - 1) * @per_page
       return [] if offset > @size

data/lib/xapian_db.rb

@@ -77,6 +77,13 @@ module XapianDb
     XapianDb::Config.database.search(expression)
   end
 
+  # Get facets from the configured database.
+  # See {XapianDb::Database#facets} for options
+  # @return [Hash<Class, Integer>] A hash containing the classes and the hits per class
+  def self.facets(expression)
+    XapianDb::Config.database.facets(expression)
+  end
+
 end
 
 do_not_require = %w(update_stopwords.rb railtie.rb base_adapter.rb)

metadata

@@ -4,9 +4,9 @@ version: !ruby/object:Gem::Version
   prerelease: false
   segments: 
   - 0
-  - 3
   - 4
-  version: 0.3.4
+  - 0
+  version: 0.4.0
 platform: ruby
 authors: 
 - Gernot Kogler
@@ -14,7 +14,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2010-12-13 00:00:00 +01:00
+date: 2010-12-15 00:00:00 +01:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency