xapian-fu 0.2 → 1.0.1

data/README.rdoc

@@ -3,11 +3,33 @@
 XapianFu is a Ruby library for working with
 {Xapian}[http://xapian.org/] databases.  It builds on the GPL licensed
 Xapian Ruby bindings but provides an interface more in-line with "The
-Ruby Way"(tm).
+Ruby Way"(tm) and is considerably easier to use.
 
-== Example
+For example, you can work almost entirely with Hash objects - XapianFu
+will handle converting the Hash keys into Xapian term prefixes when
+indexing and when parsing queries.
 
-Create a database, add 3 documents to it and then search and retrieve 
+It also handles storing and retrieving hash entries as
+Xapian::Document values.  XapianFu basically gives you a persistent
+Hash with full text indexing (and ACID transactions).
+
+== Installation
+
+  sudo gem install xapian-fu
+
+== Documentation
+
+XapianFu::XapianDb is the corner-stone of XapianFu.  A XapianDb
+instance will handle setting up a XapianFu::XapianDocumentsAccessor
+for reading and writing documents from and to a Xapian database.  It
+makes use of XapianFu::QueryParser for parsing and setting up a query.
+
+XapianFu::XapianDoc represents a document retrieved from or to be
+added to a Xapian database.
+
+== Basic usage example
+
+Create a database, add 3 documents to it and then search and retrieve
 them.
 
   db = XapianDb.new(:dir => 'example.db', :create => true,
@@ -15,25 +37,142 @@ them.
   db << { :title => 'Brokeback Mountain', :year => 2005 }
   db << { :title => 'Cold Mountain', :year => 2004 }
   db << { :title => 'Yes Man', :year => 2008 }
+  db.flush
   db.search("mountain").each do |match|
-    puts match.fields[:title]
+    puts match.values[:title]
+  end
+
+== Ordering of results
+
+Create an in-memory database, add 3 documents to it and then search and retrieve
+them in year order.
+
+  db = XapianDb.new(:store => [:title], :sortable => [:year])
+  db << { :title => 'Brokeback Mountain', :year => 2005 }
+  db << { :title => 'Cold Mountain', :year => 2004 }
+  db << { :title => 'Yes Man', :year => 2008 }
+  db.search("mountain", :order => :year)
+
+== will_paginate support
+
+Simple integration with the will_paginate Rails helpers.
+
+  @results = db.search("mountain", :page => 1, :per_page => 5)
+  will_paginate @results
+
+== Transactions support
+
+Ensure that a group of documents are either entirely added to the
+database or not at all - the transaction is aborted if an exception is
+raised inside the block.  The documents only become available to
+searches at the end of the block, when the transaction is committed.
+
+  db = XapianDb.new(:store => [:title, :year], :sortable => [:year])
+  db.transaction do
+    db << { :title => 'Brokeback Mountain', :year => 2005 }
+    db << { :title => 'Cold Mountain', :year => 2004 }
+    db << { :title => 'Yes Man', :year => 2008 }
   end
+  db.search("mountain")
+
+== Complete field definition examples
+
+Fields can be described in more detail using a hash. For example,
+telling XapianFu that a particular field is a Date, Fixnum or Bignum
+will allow very efficient on-disk storage and will ensure the same
+type of object is instantiated when returning those stored values.
+And in the case of Fixnum and Bignum, allows you to order search
+results without worrying about leading zeros.
+
+  db = XapianDb.new(:fields => { 
+                                 :title => { :store => true },
+                                 :released => { :type => Date, :store => true },
+                                 :votes => { :type => Fixnum, :store => true }
+                               })
+  db << { :title => 'Brokeback Mountain', :released => Date.parse('13th January 2006'), :votes => 105302 }
+  db << { :title => 'Cold Mountain, :released => Date.parse('2nd January 2004'), :votes => 45895 }
+  db << { :title => 'Yes Man', :released => Date.parse('26th December 2008'), :votes => 44936 }
+  db.search("mountain", :order => :votes)
+
+== Simple max value queries
+
+Find the document with the highest :year value
+
+  db.documents.max(:year)
+
+== Search examples
+
+Search on particular fields
+
+  db.search("title:mountain year:2005")
+
+Boolean AND (default)
+
+  db.search("ruby AND rails")
+  db.search("ruby rails")
+
+Boolean OR
+
+  db.search("rails OR sinatra")
+  db.search("rails sinatra", :default_op => :or)
+
+Exclude certain terms
+
+  db.search("ruby -rails")
+
+Wildcards
+
+  db.search("xap*")
+
+Phrase searches
+
+  db.search("'someone dropped a steamer in the gene pool'")
+
+And any combinations of the above:
+
+  db.search("(ruby OR sinatra) -rails xap*")
+
+== ActiveRecord Integration
+
+XapianFu always stores the :id field, so you can easily use it with
+something like ActiveRecord to index database records:
+
+  db = XapianDb.new(:dir => 'posts.db', :create => true)
+  Post.all.each { |p| db << p.attributes }
+  docs = db.search("custard")
+  docs.each_with_index { |doc,i| docs[i] = Post.find(doc.id) }
+
+Combine it with the max value search to do batch delta updates by primary key:
+
+  db = XapianDb.new(:dir => 'posts.db')
+  latest_doc = db.documents.max(:id)
+  new_posts = Post.find(:all, :conditions => ['id > ?', lastest_doc.id])
+  new_posts.each { |p| db << p.attributes }
+
+Or by :updated_at field if you prefer:
 
-== ActiveRecord Example
+  db = XapianDb.new(:dir => 'posts.db', :fields => { :updated_at => { :type => Time, :store => true } })
+  last_updated_doc = db.documents.max(:updated_at)
+  new_posts = Post.find(:all, :conditions => ['updated_at >= ?', last_updated_doc.updated_at])
+  new_posts.each { |p| db << p.attributes }
 
-You could use it with something like ActiveRecord to index database 
-records:
+Deleted records won't show up in results but can eventually put your
+result pagination out of whack. So, you'll need to track deletions
+yourself, either with a deleted_at field, some kind of delete log or
+perhaps by reindexing once in a while.
 
-  db = XapianDb.new(:dir => 'posts.db', :create => true,
-                    :store => :id)
-  Post.all.each { db << p.attributes }
-  db.search("custard").collect do |doc|
-    Post.find(doc.id)
+  db = XapianDb.new(:dir => 'posts.db')
+  deleted_posts = Post.find(:all, :conditions => 'deleted_at is not null')
+  deleted_posts.each do |post| 
+    db.documents.delete(post.id)
+    post.destroy
   end
 
 = More Info
 
 Author::  John Leach  (mailto:john@johnleach.co.uk)
 Copyright:: Copyright (c) 2009 John Leach
-License:: GPL v2
+License:: MIT (The Xapian library is GPL)
+Mailing list:: http://rubyforge.org/mailman/listinfo/xapian-fu-discuss
+Web page:: http://johnleach.co.uk/documents/xapian-fu
 Github:: http://github.com/johnl/xapian-fu/tree/master

data/examples/query.rb

@@ -1,16 +1,44 @@
 #!/usr/bin/ruby
 #
+# Example file spider index searcher using XapianFu. Conducts a search
+# on ./spider.db created with spider.rb.
+#
+# --order-by-filesize sorts the results by the file size, largest
+# first. Default is to sort by relevance.
+#
+# All other command line arguments are used as the search query:
+#
+# query.rb --order-by-filesize mammoth -woolley
+#
+# You can limit queries to particular fields:
+#
+# query.rb filename:LICENSE text:BSD
+#
 require 'rubygems'
 require 'benchmark'
 require 'lib/xapian_fu'
 
-query_string = ARGV.join(" ")
-db = XapianFu::XapianDb.new(:dir => 'spider.db')
+order = nil
+reverse = false
+if ARGV.delete('--order-by-filesize')
+  order = :filesize
+  reverse = true
+end
+query = ARGV.join(" ")
+db = XapianFu::XapianDb.new(:dir => 'spider.db', :fields => [:text, :filesize, :filename])
+puts "Xapian Database has #{db.size} docs in total"
+puts "Largest filesize recorded is #{db.documents.max(:filesize).values[:filesize].to_i / 1024}k"
+puts "Searching for '#{query}'"
 results = nil
-bm = Benchmark.measure { results = db.search(query_string) }
-puts "Weight\tFilename"
+bm = Benchmark.measure do
+  results = db.search(query, :order => order, :reverse => reverse)
+end
+puts "Returned #{results.size} of #{results.total_entries} total hits"
+puts "Weight\tFilename\tFilesize"
 results.each do |result|
-  puts "%.2f\t%s" % [result.weight, result.fields[:filename]]
+  filename = result.values[:filename]
+  filesize = result.values[:filesize].to_i / 1024
+  puts "%.2f\t%s\t%ik" % [result.weight, filename, filesize]
 end
-puts "Search took %.5f seconds" %  bm.total
+puts "Search took %.5f seconds" %  bm.real
 

data/examples/spider.rb

@@ -1,28 +1,57 @@
 #!/usr/bin/ruby
+#
+# Example file spider using XapianFu.  Overwrites the index on each run (./spider.db)
+#
+# spider.rb /path/to/index
 
 require 'rubygems'
 require 'benchmark'
 require 'lib/xapian_fu'
 
-db = XapianFu::XapianDb.new(:dir => 'spider.db', :store => :filename, 
+db = XapianFu::XapianDb.new(:dir => 'spider.db', :store => [:filename, :filesize],
                             :overwrite => true)
 
 base_path = ARGV[0] || '.'
 
-docs = 0
+index_queue = [base_path]
+total_file_count = 0
 indexing_time = 0.0
-Dir.glob(File.join(base_path, "/**/*")) do |filename|
-  next unless File.file?(filename)
-  next unless filename =~ /\.(txt|doc|README|c|h|rb|py|note|xml)$/i
-  puts "Indexing #{filename}"
-  text = File.open(filename) { |f| f.read(10 * 1024) }
-  bm = Benchmark.measure do
-    db << XapianFu::XapianDoc.new({:text => text, :filename => filename, 
-                                    :filesize => File.size(filename) })
+STDERR.write "Indexing\n"
+while dir = index_queue.shift
+  STDERR.write " - #{dir}: "
+  file_count = 0
+  file_data = 0
+  Dir.foreach(dir) do |filename|
+    # skip . and ..
+    next if filename =~ /^[.]{1,2}$/
+    filename = File.join(dir, filename)
+    # Put any directories we find onto the queue for indexing
+    if File.directory?(filename)
+      index_queue << filename
+      next
+    end
+    next unless File.file?(filename)
+    next unless filename =~ /(txt|doc|README|c|h|pl|sh|rb|py|note|xml)$/i
+    file_count += 1
+
+    # Read the first 10k of data
+    text = File.open(filename) { |f| f.read(10 * 1024) }
+    file_data += text.size
+    # Index the data, filename and filesize
+    bm = Benchmark.measure do
+      db << {
+        :text => text,
+        :filename => filename,
+        :filesize => File.size(filename)
+      }
+    end
+    indexing_time += bm.real
   end
-  indexing_time += bm.total
-  docs += 1
-  break if docs == 10000
+  STDERR.write("#{file_data / 1024}k in #{file_count} files\n")
+  total_file_count += file_count
 end
-indexing_time += Benchmark.measure { db.flush }.total
-puts "#{docs} docs indexed in #{indexing_time} seconds (#{docs / indexing_time} docs per second)"
+
+files_per_second = (total_file_count / indexing_time).round
+puts "#{total_file_count} files indexed in #{indexing_time.round} seconds (#{files_per_second} per second)"
+flush_time = Benchmark.measure { db.flush }.real
+puts "Flush to disk took #{flush_time.round} seconds"

data/lib/xapian_fu/query_parser.rb

@@ -0,0 +1,179 @@
+module XapianFu #:nodoc:
+
+  # The XapianFu::QueryParser is responsible for building useful
+  # Xapian::QueryParser objects.
+  #
+  # The <tt>:fields</tt> option specifies the fields allowed in the
+  # query.  Settings <tt>:fields => [:name, :city]</tt> would allow
+  # searches such as <tt>"name:john city:Leeds"</tt> (assuming those
+  # fields were in the document when it was added to the database.)
+  # This options takes an array of symbols or strings representing the
+  # field names.
+  #
+  # The <tt>:database</tt> option specifies the XapianFu::Database,
+  # necessary for calculating spelling corrections.  The database's
+  # stemmer, stopper and field list will also be used.
+  #
+  # The <tt>:default_op</tt> option specifies the search operator to
+  # be used when not specified.  It takes the operations <tt>:or</tt>,
+  # <tt>:phrase</tt>, <tt>:and</tt> and <tt>:and_maybe</tt>.  The
+  # default is <tt>:and</tt>.  So for example, with the <tt>:or</tt>
+  # operation, a query <tt>"dog cat rabbit"</tt> will be parsed as
+  # <tt>"dog AND cat AND rabbit"</tt>.
+  #
+  # The <tt>:stemming_strategy</tt> option specifies how terms in the
+  # query should be stemmed.  It accepts <tt>:some</tt>, <tt>:all</tt>
+  # or <tt>:none</tt>.  The default is <tt>:some</tt> which is best
+  # for most situations.  See the Xapian documentation for more
+  # details.
+  #
+  # The <tt>:boolean</tt> option enables or disables boolean
+  # queries. Set to true or false.
+  #
+  # The <tt>:boolean_anycase</tt> option enables or disables
+  # case-insensitive boolean queries.  Set to true or false.
+  #
+  # The <tt>:wildcards</tt> option enables or disables the use of
+  # wildcard terms in queries, such as <tt>"york*"</tt>. Set to true or false.
+  #
+  # The <tt>:lovehate</tt> option enables or disables the use of +/-
+  # operators in queries, such as <tt>"+mickey -mouse"</tt>. Set to true or
+  # false.
+  #
+  # The <tt>:spelling</tt> option enables or disables spelling
+  # correction on queries. Set to true or false. Requires the
+  # <tt>:database</tt> option.
+  #
+  # The <tt>:pure_not</tt> option enables or disables the use of
+  # queries that only exclude terms, such as <tt>"NOT apples"</tt>. Set to true
+  # or false.
+  #
+  class QueryParser #:notnew:
+    
+    # The stemming strategy to use when generating terms from a query.
+    # Defaults to <tt>:some</tt>
+    attr_accessor :stemming_strategy
+    
+    # The default operation when combining search terms.  Defaults to
+    # <tt>:and</tt>
+    attr_accessor :default_op
+    
+    # The database that this query is agains, used for setting up
+    # fields, stemming, stopping and spelling.
+    attr_accessor :database
+
+    def initialize(options = { })
+      @options = {
+        :stemming_strategy => :some,
+        :default_op => :and
+      }.merge(options)
+      self.stemming_strategy = @options[:stemming_strategy]
+      self.default_op = @options[:default_op]
+      self.database = @options[:database]
+    end
+
+    # Parse the given query string and return a Xapian::Query object
+    def parse_query(q)
+      query_parser.parse_query(q, xapian_flags)
+    end
+
+    # Return the query string with any spelling corrections made
+    def corrected_query
+      query_parser.get_corrected_query_string
+    end
+
+    # The current Xapian::QueryParser object
+    def query_parser
+      if @query_parser
+        @query_parser
+      else
+        qp = Xapian::QueryParser.new
+        qp.database = xapian_database if xapian_database
+        qp.stopper = database.stopper if database
+        qp.stemmer = database.stemmer if database
+        qp.default_op = xapian_default_op
+        qp.stemming_strategy = xapian_stemming_strategy
+        fields.each do |name, type|
+          qp.add_prefix(name.to_s.downcase, "X" + name.to_s.upcase)
+        end
+        @query_parser = qp
+      end
+    end
+
+    # The Xapian::QueryParser constant for this parsers stemming strategy
+    def xapian_stemming_strategy
+      case stemming_strategy
+      when :all
+        Xapian::QueryParser::STEM_ALL
+      when :some
+        Xapian::QueryParser::STEM_SOME
+      when :none
+      when false
+      when nil
+        Xapian::QueryParser::STEM_NONE
+      end
+    end
+
+    # Return an array of symbols representing the flags set for this
+    # query parser
+    def flags
+      if @flags
+        @flags
+      else
+        valid_flags = [:boolean, :boolean_anycase, :wildcards, :lovehate, :spelling, :pure_not]
+        @flags = valid_flags.delete_if { |vf| not @options[vf] }
+      end
+    end
+
+    # Return a Xapian::QueryParser flag mask representing the flags
+    # set for this query parser
+    def xapian_flags
+      qflags = 0
+      qflags |= Xapian::QueryParser::FLAG_BOOLEAN if flags.include?(:boolean)
+      qflags |= Xapian::QueryParser::FLAG_BOOLEAN_ANY_CASE if flags.include?(:boolean_anycase)
+      qflags |= Xapian::QueryParser::FLAG_WILDCARD if flags.include?(:wildcards)
+      qflags |= Xapian::QueryParser::FLAG_LOVEHATE if flags.include?(:lovehate)
+      qflags |= Xapian::QueryParser::FLAG_SPELLING_CORRECTION if flags.include?(:spelling)
+      qflags |= Xapian::QueryParser::FLAG_PURE_NOT if flags.include?(:pure_not)
+      qflags
+    end
+
+    # Return a Xapian::Query constant for this query parser's default
+    # operation
+    def xapian_default_op
+      case default_op
+      when :and_maybe
+        Xapian::Query::OP_AND_MAYBE
+      when :or
+        Xapian::Query::OP_OR
+      when :phrase
+        Xapian::Query::OP_PHRASE
+      when :and
+        Xapian::Query::OP_AND
+      end
+    end
+
+    # Return the available Xapian::Database for use in the query
+    # parser
+    def xapian_database
+      if database.is_a? XapianFu::XapianDb
+        database.ro
+      elsif database.is_a? Xapian::Database
+        database
+      else
+        nil
+      end
+    end
+
+    # An array of field names that will be recognised in this query
+    def fields
+      if @options[:fields].is_a? Array
+        @options[:fields]
+      elsif database.is_a? XapianFu::XapianDb
+        database.fields
+      else
+        []
+      end
+    end
+  end
+end

data/lib/xapian_fu/result_set.rb

@@ -0,0 +1,52 @@
+module XapianFu
+  # A XapianFu::ResultSet holds the XapianDoc objects returned from a search.
+  # It acts just like an array but is decorated with useful attributes.
+  class ResultSet < Array
+
+    # The Xapian match set for this search
+    attr_reader :mset
+    attr_reader :current_page, :per_page
+    # The total number of pages of results available for this search
+    attr_reader :total_pages
+    # If any spelling corrections were detected, the full collected query is provided
+    # by :corrected_query, otherwise this is empty.
+    attr_reader :corrected_query
+
+    # nodoc
+    def initialize(options = { })
+      @mset = options[:mset]
+      @current_page = options[:current_page]
+      @per_page = options[:per_page]
+      @corrected_query = options[:corrected_query]
+      concat mset.matches.collect { |m| XapianDoc.new(m) }
+    end
+
+    # The estimated total number of matches this search could return
+    def total_entries
+      mset.matches_estimated
+    end
+
+    # The estimated total number of pages of results this search could return
+    def total_pages
+      (total_entries / per_page.to_f).round
+    end
+
+    # The previous page number, or nil if there are no previous pages available
+    def previous_page
+      p = current_page - 1
+      p == 0 ? nil : p
+    end
+
+    # The next page number, or nil if there are no more more pages available
+    def next_page
+      p = current_page + 1
+      p > total_pages ? nil : p
+    end
+
+    # The offset within the total results of the first result in this page
+    def offset
+      (current_page - 1) * per_page
+    end
+
+  end
+end

data/lib/xapian_fu/stopper_factory.rb

@@ -0,0 +1,40 @@
+module XapianFu
+  class UnsupportedStopperLanguage < XapianFuError ; end
+
+  class StopperFactory
+    @stoppers = { }
+
+    # Return a SimpleStopper loaded with stop words for the given language
+    def self.stopper_for(lang)
+      if lang.is_a? Xapian::Stopper
+        lang
+      else
+        lang = lang.to_s.downcase.strip
+        if @stoppers[lang]
+          @stoppers[lang]
+        else
+          stopper = Xapian::SimpleStopper.new
+          stop_words_for(lang).each { |word| stopper.add(word) }
+          @stoppers[lang] = stopper
+        end
+      end
+    end
+
+    # Return the full path to the stop words file for the given language
+    def self.stop_words_filename(lang)
+      File.join(File.dirname(__FILE__), 'stopwords', lang.to_s.downcase + '.txt')
+    end
+
+    # Read and parse the stop words file for the given language, returning an array of words
+    def self.stop_words_for(lang)
+      raise UnsupportedStopperLanguage, lang.to_s unless File.exists?(stop_words_filename(lang))
+      words = []
+      open(stop_words_filename(lang), "r") do |f|
+        while line = f.readline rescue nil
+          words << line.split(" ", 2).first.downcase.strip  unless line =~ /^ +|^$|^\|/
+        end
+      end
+      words
+    end
+  end
+end

data/lib/xapian_fu/stopwords/README

@@ -0,0 +1,7 @@
+These stopword lists are from the Snowball library which is covered by the BSD License, with Copyright (c) 2001, Dr Martin Porter, and (for the Java developments) Copyright (c) 2002, Richard Boulton.
+
+http://snowball.tartarus.org/
+
+Some have been converted to utf8
+
+curl http://snowball.tartarus.org/algorithms/russian/stop.txt | iconv -f "KOI8-R" -t utf8 > russian.txt