xapian_db 0.4.2 → 0.5.0

data/CHANGELOG.md

@@ -1,3 +1,10 @@
+##0.5.0 (December 19th, 2010)
+
+Features:
+
+  - beanstalk based index writer for production environments (multiple app instances, e.g. mongrel clusters,
+    passenger...)
+
 ##0.4.2 (December 17th, 2010)
 
 Features:

data/README.rdoc

@@ -58,8 +58,10 @@ A future release of xapian_db might include the Xapian binaries and make this st
   make
   sudo make install
 
-The following steps assume that you are using xapian_db within a Rails app. The gem has an
-example in the examples folder that shows how you can use xapian_db without Rails.
+For a first look, look at the examples in the examples folder. There's the simple ruby script basic.rb that shows the basic
+usage of XapianDB without rails. In the basic_rails folder you'll find a very simple Rails app unsing XapianDb.
+
+The following steps assume that you are using xapian_db within a Rails app.
 
 === Configure your databases
 
@@ -220,6 +222,59 @@ If you want to implement a simple drilldown for your searches, you can use a fac
 
 Facet support in XapianDb is very limited. The only available facet is the class of the indexed objects. In many cases that's all that's needed. Therefore, it is very likely that I won't add more options for facets (since I'm not a fan of facets anyway). However, if you desperately need advanced facets, let me know. Or - even better - send me a pull request with a nice implementation ;-)
 
-== What to expect from future releases
+== Production setup
+
+Since Xapian allows only one database instance to write to the index, the default setup of XapianDb will not work
+with multiple app instances trying to write to the same database (you will get lock errors).
+Therefore, XapianDb provides a solution based on beanstalk to overcome this.
+
+=== 1. Install beanstalkd
+
+Make sure you have the {beanstalk daemon}[http://kr.github.com/beanstalkd/] installed
+
+==== OSX
+
+The easiest way is to use macports or homebrew:
+
+  port install beanstalkd
+  brew install beanstalkd
+
+==== Debian (Lenny)
+
+  # Add backports to /etc/apt/sources.list:
+  deb http://ftp.de.debian.org/debian-backports lenny-backports main contrib non-free
+  deb-src http://ftp.de.debian.org/debian-backports lenny-backports main contrib non-free
+
+  sudo apt-get update
+  sudo apt-get -t lenny-backports install libevent-1.4-2
+  sudo apt-get -t lenny-backports install libevent-dev
+  cd /tmp
+  curl http://xph.us/dist/beanstalkd/beanstalkd-1.4.6.tar.gz | tar zx
+  cd beanstalkd-1.4.6/
+  ./configure
+  make
+  sudo make install
+
+=== 2. Add the beanstalk-client gem to your config
+
+  gem 'beanstalk-client' # Add this to your Gemfile
+  bundle install
+
+=== 3. Configure your production environment in config/xapian_db.yml
+
+  production:
+    database: db/xapian_db/production
+    writer:   beanstalk
+    beanstalk_daemon: localhost:11300
+
+=== 4. start the beanstalk daemon
+
+  beanstalk -d
+
+=== 5. start the beanstalk worker from within your Rails app root directory
+
+  rake RAILS_ENV=production xapian_db:beanstalk_worker
+
+<b>Important: Do not start multiple instances of this worker task!</b>
+
 
-* asynchronous index writer based on {resque}[https://github.com/defunkt/resque] for production environments

data/lib/xapian_db.rb

@@ -86,7 +86,7 @@ module XapianDb
 
 end
 
-do_not_require = %w(update_stopwords.rb railtie.rb base_adapter.rb)
+do_not_require = %w(update_stopwords.rb railtie.rb base_adapter.rb beanstalk_writer.rb)
 files = Dir.glob("#{File.dirname(__FILE__)}/**/*.rb").reject{|path| do_not_require.include?(File.basename(path))}
 # Require the base adapter first
 require "#{File.dirname(__FILE__)}/xapian_db/adapters/base_adapter"
@@ -94,3 +94,6 @@ files.each {|file| require file}
 
 # Configure XapianDB if we are in a Rails app
 require File.dirname(__FILE__) + '/xapian_db/railtie' if defined?(Rails)
+
+# Require the beanstalk writer is beanstalk-client is installed
+require File.dirname(__FILE__) + '/xapian_db/index_writers/beanstalk_writer' if Gem.available?('beanstalk-client')

data/lib/xapian_db/config.rb

@@ -35,6 +35,13 @@ module XapianDb
           @config.nil? ? nil : @config.instance_variable_get("@_#{attr}")
         end
       end
+
+      # The beanstalk daemon url
+      define_method :beanstalk_daemon_url do
+        default_url = "localhost:11300"
+        return default_url if @config.nil?
+        @config.instance_variable_get("@_beanstalk_daemon_url") || default_url
+      end
     end
 
     # ---------------------------------------------------------------------------------
@@ -42,7 +49,7 @@ module XapianDb
     # ---------------------------------------------------------------------------------
 
     #
-    attr_reader :_database, :_adapter, :_writer, :_stemmer, :_stopper
+    attr_reader :_database, :_adapter, :_writer, :_beanstalk_daemon, :_stemmer, :_stopper
 
     # Set the global database to use
     # @param [String] path The path to the database. Either apply a file sytem path or :memory
@@ -81,12 +88,18 @@ module XapianDb
     # Set the index writer
     # @param [Symbol] type The writer type; the following adapters are available:
     #   - :direct ({XapianDb::IndexWriters::DirectWriter})
-    #   More to come in a future release
+    #   - :beanstalk ({XapianDb::IndexWriters::BeanstalkWriter})
     def writer(type)
       # We try to guess the writer name
       @_writer = XapianDb::IndexWriters.const_get("#{camelize(type.to_s)}Writer")
     end
 
+    # Set the url and port of the beanstalk daemon
+    # @param [Symbol] url The url of the beanstalk daemon; defaults to localhost:11300
+    def beanstalk_daemon_url(url)
+      @_beanstalk_daemon_url = url
+    end
+
     # Set the language.
     # @param [Symbol] lang The language; apply the two letter ISO639 code for the language
     # @example

data/lib/xapian_db/database.rb

@@ -65,6 +65,8 @@ module XapianDb
       # If we do not have a valid query we return an empty result set
       return Resultset.new(nil, opts) unless query
 
+      Rails.logger.info "Executing XapianDb search: #{expression}" if defined?(Rails)
+
       enquiry       = Xapian::Enquire.new(reader)
       enquiry.query = query
       if opts[:sort_indices]

data/lib/xapian_db/index_writers/beanstalk_worker.rb

@@ -0,0 +1,30 @@
+# encoding: utf-8
+
+module XapianDb
+  module IndexWriters
+
+    # Worker to update the Xapian index; the worker is used in the beanstalk worker rake task
+    # and uses the DirectWriter to do the real work
+    # @author Gernot Kogler
+    class BeanstalkWorker
+
+      def index_task(options)
+        klass = Kernel.const_get options[:class]
+        obj   = klass.respond_to?(:get) ? klass.get(options[:id].to_i) : klass.find(options[:id].to_i)
+        DirectWriter.index obj
+      end
+
+      def unindex_task(options)
+        klass = Kernel.const_get options[:class]
+        obj   = klass.respond_to?(:get) ? klass.get(options[:id].to_i) : klass.find(options[:id].to_i)
+        DirectWriter.unindex obj
+      end
+
+      def reindex_class_task(options)
+        klass = Kernel.const_get options[:class]
+        DirectWriter.reindex_class klass, :verbose => false
+      end
+
+    end
+  end
+end

data/lib/xapian_db/index_writers/beanstalk_writer.rb

@@ -0,0 +1,47 @@
+# encoding: utf-8
+
+# This writer puts reindex requests into a stalker queue. If you want
+# to use this writer, you must install beanstalkd and the stalker gem.
+# This writer can only be used inside a Rails app.
+# See https://github.com/adamwiggins/stalker for more info
+# @author Gernot Kogler
+
+require 'beanstalk-client'
+
+module XapianDb
+  module IndexWriters
+
+    class BeanstalkWriter
+
+      class << self
+
+        # Update an object in the index
+        # @param [Object] obj An instance of a class with a blueprint configuration
+        def index(obj)
+          beanstalk.put({:task => "index_task", :class => obj.class.name, :id => obj.id}.to_yaml)
+        end
+
+        # Remove an object from the index
+        # @param [Object] obj An instance of a class with a blueprint configuration
+        def unindex(obj)
+          beanstalk.put({:task => "unindex_task", :class => obj.class.name, :id => obj.id}.to_yaml)
+        end
+
+        # Reindex all objects of a given class
+        # @param [Class] klass The class to reindex
+        def reindex_class(klass, options={})
+          beanstalk.put({:task => "reindex_class_task", :class => klass.name}.to_yaml)
+        end
+
+        private
+
+        def beanstalk
+          @beanstalk ||= Beanstalk::Pool.new([XapianDb::Config.beanstalk_daemon_url])
+        end
+
+      end
+
+    end
+
+  end
+end

data/lib/xapian_db/railtie.rb

@@ -9,6 +9,11 @@ module XapianDb
   # @author Gernot Kogler
   class Railtie < ::Rails::Railtie
 
+    # Require our rake tasks
+    rake_tasks do
+      load "#{File.dirname(__FILE__)}/../../tasks/beanstalk_worker.rake"
+    end
+
     config.before_configuration do
 
       # Read the database configuration file if there is one
@@ -16,14 +21,16 @@ module XapianDb
       if File.exist?(config_file_path)
         db_config = YAML::load_file config_file_path
         env_config = db_config[Rails.env]
-        database_path = env_config["database"] || ":memory:"
-        adapter       = env_config["adapter"]  || :active_record
-        writer        = env_config["writer"]   || :direct
+        database_path    = env_config["database"] || ":memory:"
+        adapter          = env_config["adapter"]  || :active_record
+        writer           = env_config["writer"]   || :direct
+        beanstalk_daemon = env_config["beanstalk_daemon"]
       else
         # No config file, set the defaults
         Rails.env == "test" ? database_path = ":memory:" : database_path = "db/xapian_db/#{Rails.env}"
-        adapter = :active_record
-        writer  = :direct
+        adapter          = :active_record
+        writer           = :direct
+        beanstalk_daemon = nil
       end
 
       # Do the configuration
@@ -35,12 +42,14 @@ module XapianDb
         end
         config.adapter adapter.to_sym
         config.writer writer.to_sym
+        config.beanstalk_daemon_url beanstalk_daemon
         config.language(env_config["language"]) if env_config && env_config["language"]
       end
 
     end
 
     config.to_prepare do
+
       # Load a blueprint config if there is one
       blueprints_file_path = "#{Rails.root}/config/xapian_blueprints.rb"
       load blueprints_file_path if File.exist?(blueprints_file_path)

data/tasks/beanstalk_worker.rake

@@ -0,0 +1,35 @@
+# encoding: utf-8
+
+# This rake task is built to run within a Rails application and is the
+# backend worker to serialize the index updates to a xapian database.
+# Remember to install beanstalkd and configure the beanstalk-client gem
+# in your Gemfile
+
+require "#{Rails.root}/config/environment"
+require "xapian_db"
+require 'yaml'
+
+namespace :xapian_db do
+
+  desc "Run the beanstalk worker process to update the xapian index"
+  task :beanstalk_worker do
+
+    url = XapianDb::Config.beanstalk_daemon_url
+    beanstalk = Beanstalk::Pool.new([url])
+    worker    = XapianDb::IndexWriters::BeanstalkWorker.new
+    puts "XapianDb beanstalk worker is serving on #{url}..."
+    loop do
+      job = beanstalk.reserve
+      begin
+        params = YAML::load job.body
+        Rails.logger.info "XapianDb beanstalk worker: executing task #{params}"
+        task = params.delete :task
+        worker.send task, params
+      rescue Exception => ex
+        Rails.logger.error "XapianDb beanstalk worker: could not process #{job.body} (#{ex})"
+      end
+      job.delete
+    end
+
+  end
+end

metadata

@@ -4,9 +4,9 @@ version: !ruby/object:Gem::Version
   prerelease: false
   segments: 
   - 0
-  - 4
-  - 2
-  version: 0.4.2
+  - 5
+  - 0
+  version: 0.5.0
 platform: ruby
 authors: 
 - Gernot Kogler
@@ -14,7 +14,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2010-12-17 00:00:00 +01:00
+date: 2010-12-19 00:00:00 +01:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -63,6 +63,8 @@ files:
 - lib/xapian_db/config.rb
 - lib/xapian_db/database.rb
 - lib/xapian_db/document_blueprint.rb
+- lib/xapian_db/index_writers/beanstalk_worker.rb
+- lib/xapian_db/index_writers/beanstalk_writer.rb
 - lib/xapian_db/index_writers/direct_writer.rb
 - lib/xapian_db/indexer.rb
 - lib/xapian_db/query_parser.rb
@@ -86,6 +88,7 @@ files:
 - lib/xapian_db/stopwords/sv.txt
 - lib/xapian_db/stopwords/update_stopwords.rb
 - lib/xapian_db.rb
+- tasks/beanstalk_worker.rake
 - LICENSE
 - README.rdoc
 - CHANGELOG.md