index-tanked 0.1.16 → 0.2.0

data/README.markdown

@@ -0,0 +1,114 @@
+Index Tanked
+============
+
+Index Tanked helps you index and search your data on IndexTank. Index Tanked works with any Ruby class but has additional helpful default behavior when used with Active Record.
+
+***
+
+Install
+--------
+
+If you're using Bundler toss a `gem 'index-tanked'` in your GEMFILE. Otherwise `gem install 'index-tanked'`
+
+Example
+-------
+
+    require 'rubygems'
+    require 'index-tanked'
+
+    class Dog
+      include IndexTanked
+
+      attr_accessor :breed, :flea_count, :name, :behavior_score, :description
+
+      index_tank :index => 'dogs', :url => 'http://example@indextank.com' do
+        doc_id, :doc_id
+        field :breed
+        field :behavior_score, :text => nil
+        field :fleas, :flea_count, :text => lambda { |dog| 'infested' if dog.flea_count > 5 }
+        field :name
+        text :description
+        var 0, 15
+      end
+
+      def doc_id
+        ...
+      end
+    end
+
+
+### What did we just do?
+
+First thing's first. Include IndexTanked in your class. Next up is the index_tank block where we determine what we're going to index. You can pass in the index name and url here, alternatively if you have added a url and index to the configuration you can leave it off here and the configured ones will be used.
+
+ The first thing we define is the doc_id. The doc_id is the ID of your record in IndexTank and you need to be able to generate a unique one for each instance that you'll be indexing. If you're using ActiveRecord you can skip this as it's defined by default, if you're using anything else you'll need to come come up with your own. You could base it on the url that points to the document, or the id used by your data store, etc.
+
+Next up are the fields. When you do a search in IndexTank you can specify which field you are searching like this: `breed:pug`. If you don't specify a field you end up searching a special field called text. By default when you add a field in IndexTanked the value of that field *also* goes into the text field.
+
+Sometimes you don't want that to occur, for instance, assuming that :behavior_score, above, is just a number, it may not make sense to have its value go into the text field since you may have multiple numerical fields and it may not make sense for a search of '5' to return dogs with 5 fleas and dogs with a behavior score of 5. If that is the case then `:text => nil` will prevent the field's value from being added to the text field.
+
+The field method takes three arguments. The first argument is what the field should be called in IndexTank. The second argument is the optional method to retrieve the value for the field. If it's not provided then it is assumed that the first argument is also the method to retrieve its value. This can be a symbol (the name of the method to call), a Proc which will be executed, or just a String / Integer etc which will then be indexed identically for all instances.
+
+If you want something other than the value of the field to be added to the text field you can specify it with :text, in the example above any dog with more than 5 fleas will have the word 'infested' in their text field, allowing them to be found by searching for 'infested'.
+
+The text method takes one argument, which is a value to be *added* to the text field. This does not replace the text field, just adds to it. As above this may be a proc, symbol etc.
+
+The var method adds a variable. See the IndexTank documentation for why you might want to do such a thing.
+
+### What can we do now that we've done that?
+
+#### Instance methods
+**add_to_index_tank** Add your instance to your index on IndexTank.
+
+#### Class Methods
+**add_to_index_tank(doc_id, data, fallback)** This method is called internally by the instance method, the third argument is optional and defaults to true, it determines whether or not your add_to_index_fallback will be called.
+
+**add_to_index_tank_without_fallback(doc_id, data)** Calls the above, passing false to fallback.
+
+**delete_from_index_tank(doc_id, fallback)** Removes the document with the doc_id passed as it's first argument from the index. The second argument is optional and defaults to true, it determines whether or not your delete_from_index_fallback will be called.
+
+**delete_from_index_tank_without_fallback(doc_id)** Calls the above, passing false to fallback.
+
+**search_index_tank(query, options)**
+
+ActiveRecord Example
+--------------------
+
+Configuration
+-------------
+You can optionally configure some things in the `IndexTanked::Configuration` class. e.g.
+
+    IndexTanked::Configuration.index = 'your_index_name'
+
+#### url
+The private IndexTank url that will be used if you don't specify one when you define your index.
+
+#### index
+The index that will be used if you don't specify one when you define your index.
+
+#### search_availability
+Whether or not searching is enabled. This can be a boolean or a proc. This value can also be queried by calling `IndexTanked::Configuration.search_available?`. If a search is attempted while this is false a `SearchingDisabledError` will be raised.
+
+#### index_availability
+Whether or not indexing is enabled. This can be a boolean or a proc. This value can also be queried by calling `IndexTanked::Configuration.index_available?`. If you attempt to add to or delete from the index while this is false an `IndexingDisabledError` will be raised and your index or delete fallback will be triggered if configured.
+
+#### timeout
+Timeout in seconds. If this is configured then when you attempt to add or delete from your index a `TimeoutExceededError` will be raised when the configured time has elapsed. This will trigger your add to index or delete fallback if configured.
+
+### Fallback methods
+These let you define how to handle if if something goes wrong when communicating with IndexTank. For example if you fail to add a record to IndexTank due to a temporary network issue you may want to try again later in a background task. e.g.
+
+    IndexTanked::Configuration.add_to_index_fallback do |information_from_failed_attempt|
+        information_from_failed_attempt[:class].send_later.add_to_index_tank_without_fallback(information_from_failed_attempt[:doc_id], information_from_failed_attempt[:data])
+    end
+
+Note that if you are adding your failures to a worker queue like Delayed Job that has it's own method for retrying failures it is important that you use the _without_fallback version of the method you are backgrounding so that each failures in the background queue don't result in new jobs being added to the queue.
+
+#### add_to_index_fallback
+The block or proc that is executed when an exception happens while attempting to add a record to IndexTank. The hash passed in contains the `:class`, `:data`, `:doc_id` and the `:error` that caused the original attempt to fail.
+
+#### delete_from_index_fallback
+The block or proc that is executed when an exception happens while attempting to remove a record from to IndexTank. The hash passed in contains the `:class`, `:doc_id` and the `:error` that caused the original attempt to fail.
+
+#### missing_activerecord_ids_handler
+This block or proc lets you handle the situation where records that are no longer in your database have been returned in a search from IndexTank. You may, for example, take this opportunity to remove them from the index. The block is passed two arguments, the `model_name` and the `ids`.

data/generators/index-tanked/index_tanked_generator.rb

@@ -0,0 +1,7 @@
+class IndexTankedGenerator < Rails::Generator::Base
+  def manifest
+    record do |m|
+      m.migration_template 'migration.rb', 'db/migrate', :migration_file_name => "create_index_tanked_documents"
+    end
+  end
+end

data/generators/index-tanked/templates/migration.rb

@@ -0,0 +1,19 @@
+class CreateIndexTankedDocuments < ActiveRecord::Migration
+  def self.up
+    create_table :index_tanked_documents, :force => true do |t|
+      t.integer  :record_id                    # id of the record being indexed
+      t.string   :model_name                   # Activerecord Model name
+      t.text     :document                     # document from #document_for_batch_addition
+      t.datetime :locked_at                    # Set when a client is working on this object
+      t.string   :locked_by                    # Who is working on this object (if locked)
+      t.timestamps
+    end
+
+    add_index :index_tanked_documents, :locked_at
+  end
+
+  def self.down
+    remove_index :index_tanked_documents, :locked_at
+    drop_table :index_tanked_documents
+  end
+end

data/lib/generators/index_tanked/index_tanked_generator.rb

@@ -0,0 +1,12 @@
+require 'rails/generators/migration'
+require 'rails/generators/active_record'
+
+class IndexTankedGenerator < Rails::Generators::Base
+  include Rails::Generators::Migration
+  extend ActiveRecord::Generators::Migration
+  source_root File.expand_path('../templates', __FILE__)
+
+  def create_migration_file
+    migration_template 'migration.rb', "db/migrate/create_index_tanked_documents.rb"
+  end
+end

data/lib/generators/index_tanked/templates/migration.rb

@@ -0,0 +1,19 @@
+class CreateIndexTankedDocuments < ActiveRecord::Migration
+  def self.up
+    create_table :index_tanked_documents, :force => true do |t|
+      t.integer  :record_id                    # id of the record being indexed
+      t.string   :model_name                   # Activerecord Model name
+      t.text     :document                     # document from #document_for_batch_addition
+      t.datetime :locked_at                    # Set when a client is working on this object
+      t.string   :locked_by                    # Who is working on this object (if locked)
+      t.timestamps
+    end
+
+    add_index :index_tanked_documents, :locked_at
+  end
+
+  def self.down
+    remove_index :index_tanked_documents, :locked_at
+    drop_table :index_tanked_documents
+  end
+end

data/lib/index-tanked.rb

@@ -24,3 +24,5 @@ require 'index-tanked/active_record_defaults/class_methods'
 require 'index-tanked/active_record_defaults/instance_companion'
 require 'index-tanked/active_record_defaults/instance_methods'
 require 'index-tanked/active_record_defaults/search_result'
+require 'index-tanked/active_record_defaults/queue/document'
+require 'index-tanked/active_record_defaults/queue/worker'

data/lib/index-tanked/active_record_defaults/instance_methods.rb

@@ -13,7 +13,11 @@ module IndexTanked
 
       def add_to_index_tank_after_save(fallback=true)
         if index_tanked.dependencies_changed?
-          add_to_index_tank(fallback)
+          if Configuration.activerecord_queue
+            Document.enqueue(id, self.class.name, index_tanked.document_for_batch_addition)
+          else
+            add_to_index_tank(fallback)
+          end
         end
       end
 

data/lib/index-tanked/active_record_defaults/queue/document.rb

@@ -0,0 +1,90 @@
+module IndexTanked
+  module ActiveRecordDefaults
+    module Queue
+
+      class Document < ActiveRecord::Base
+        set_table_name 'index_tanked_documents'
+
+        def document
+          Marshal.load(Base64.decode64(read_attribute(:document)))
+        end
+
+        def document=(doc)
+          write_attribute(:document, Base64.encode64(Marshal.dump(doc)))
+        end
+
+        def inspect
+          super.sub(/document: \"[^\"\r\n]*\"/, %{document: #{document.inspect}})
+        end
+
+        def self.clear_locks_by_identifier(identifier)
+          locks_cleared = update_all(["locked_by = NULL, locked_at = NULL"],
+                                     ["locked_by = ?", identifier])
+          locks_cleared
+        end
+
+        def self.clear_expired_locks
+          locks_cleared = update_all(["locked_at = NULL, locked_by = NULL"],
+                                     ["age(clock_timestamp() at time zone 'UTC', locked_at) > interval '5 minutes'"])
+          locks_cleared
+        end
+
+        def self.enqueue(record_id, model_name, document_hash)
+          destroy_all(:record_id => record_id, :model_name => model_name)
+          create(:record_id => record_id, :model_name => model_name, :document => document_hash)
+        end
+
+        def self.get_or_update_index_information(model_name)
+          @model_list ||= {}
+          @index_list ||= {}
+          if @model_list[model_name]
+            @model_list[model_name]
+          else
+            class_companion = model_name.constantize.index_tanked
+            url = class_companion.index_tank_url
+            index_name = class_companion.index_name
+            companion_key = update_model_list(:model_name => model_name,
+                                              :url => url,
+                                              :index_name => index_name)
+            update_index_list(companion_key, class_companion)
+
+          end
+          @model_list[model_name]
+        end
+
+        def self.index_tanked(companion_key)
+          @index_list[companion_key]
+        end
+
+        def self.lock_records_for_batch(batch_size, identifier)
+          update_all(["locked_by = ?, locked_at = clock_timestamp() at time zone 'UTC'", identifier],
+                     ["locked_by IS NULL"], :limit => batch_size)
+        end
+
+        def self.partition_documents_by_companion_key(documents)
+          documents.inject({}) do |partitioned_documents, document_record|
+            companion_key = get_or_update_index_information(document_record.model_name)[:companion_key]
+            partitioned_documents[companion_key] ||= []
+            partitioned_documents[companion_key] << document_record.document
+            partitioned_documents
+          end
+        end
+
+        def self.update_index_list(companion_key, class_companion)
+          @index_list[companion_key] = class_companion unless @index_list[companion_key].present?
+        end
+
+        def self.update_model_list(options)
+          @model_list[options[:model_name]] = { :index_tank_url => options[:url],
+                                                :index_name => options[:index_name],
+                                                :companion_key => "#{options[:url]} - #{options[:index_name]}" }
+          @model_list[options[:model_name]][:companion_key]
+        end
+
+      end
+
+    end
+  end
+end
+
+IndexTanked::Document = IndexTanked::ActiveRecordDefaults::Queue::Document

data/lib/index-tanked/active_record_defaults/queue/tasks.rb

@@ -0,0 +1,8 @@
+namespace :index_tanked do
+  namespace :queue do
+    desc "Start an index-tanked queue worker."
+    task :process => :environment do
+      IndexTanked::ActiveRecordDefaults::Queue::Worker.new(:batch_size => ENV['BATCH_SIZE']).start
+    end
+  end
+end

data/lib/index-tanked/active_record_defaults/queue/worker.rb

@@ -0,0 +1,74 @@
+module IndexTanked
+  module ActiveRecordDefaults
+    module Queue
+      class Worker
+        SLEEP = 5
+
+        def initialize(options={})
+          @batch_size = options[:batch_size] || 100
+          @identifier = "host:#{Socket.gethostname} pid:#{Process.pid}" rescue "pid:#{Process.pid}"
+        end
+
+        def start
+          log "Starting IndexTanked Queue"
+
+          trap('TERM') { log 'Exiting...'; $exit = true }
+          trap('INT')  { log 'Exiting...'; $exit = true }
+
+          loop do
+            count = process_documents(@batch_size)
+
+            break if $exit
+
+            if count.zero?
+              sleep(SLEEP)
+            end
+
+            break if $exit
+          end
+        end
+
+        def process_documents(batch_size)
+          Queue::Document.clear_expired_locks
+          number_locked = Queue::Document.lock_records_for_batch(batch_size, @identifier)
+          log("#{number_locked} records locked.")
+          return number_locked if number_locked.zero?
+          begin
+            documents = Queue::Document.find_all_by_locked_by(@identifier)
+            partitioned_documents = Queue::Document.partition_documents_by_companion_key(documents)
+            send_batches_to_indextank(partitioned_documents)
+            documents_deleted = Queue::Document.delete_all(:locked_by => @identifier)
+            log("#{documents_deleted} completed documents removed from queue.")
+            documents_deleted
+          rescue StandardError, Timeout::Error => e
+            handle_error(e)
+            locks_cleared = Queue::Document.clear_locks_by_identifier(@identifier)
+            log("#{locks_cleard} locks cleared")
+            0 # return 0 so it sleeps
+          end
+        end
+
+        def send_batches_to_indextank(partitioned_documents)
+          partitioned_documents.keys.each do |companion_key|
+            index_name = companion_key.split(' - ').last
+            record_count = partitioned_documents[companion_key].size
+            log("#{record_count} document(s) prepared for #{index_name}.")
+            Queue::Document.index_tanked(companion_key).index.batch_insert(partitioned_documents[companion_key])
+          end
+        end
+
+        def handle_error(e)
+          log("something (#{e.class} - #{e.message}) got jacked, unlocking")
+          log e.backtrace
+        end
+
+        def log(message)
+          message = "[Index Tanked Worker: #{@identifier}] - #{message}"
+          puts message
+          RAILS_DEFAULT_LOGGER.info(message)
+        end
+
+      end
+    end
+  end
+end

data/lib/index-tanked/class_companion.rb

@@ -1,6 +1,6 @@
 module IndexTanked
   class ClassCompanion
-    attr_reader :fields, :variables, :texts, :index_name, :doc_id_value
+    attr_reader :fields, :variables, :texts, :index_name, :index_tank_url, :doc_id_value
 
     def initialize(options={})
       @fields         = []

data/lib/index-tanked/configuration.rb

@@ -2,7 +2,7 @@ module IndexTanked
   class Configuration
 
     class << self
-      attr_accessor :url, :index, :search_availability, :index_availability, :timeout
+      attr_accessor :url, :index, :search_availability, :index_availability, :timeout, :activerecord_queue
 
       def self.block_accessor(*fields)
         fields.each do |field|

data/lib/index-tanked/version.rb

@@ -1,3 +1,3 @@
 module IndexTanked
-  GEM_VERSION = '0.1.16'
+  GEM_VERSION = '0.2.0'
 end

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: index-tanked
 version: !ruby/object:Gem::Version 
-  hash: 59
+  hash: 23
   prerelease: false
   segments: 
   - 0
-  - 1
-  - 16
-  version: 0.1.16
+  - 2
+  - 0
+  version: 0.2.0
 platform: ruby
 authors: 
 - Adam Kittelson
@@ -16,7 +16,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2011-04-11 00:00:00 -05:00
+date: 2011-05-27 00:00:00 -05:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -131,10 +131,15 @@ extensions: []
 extra_rdoc_files: []
 
 files: 
+- lib/generators/index_tanked/index_tanked_generator.rb
+- lib/generators/index_tanked/templates/migration.rb
 - lib/index-tanked/active_record_defaults/class_companion.rb
 - lib/index-tanked/active_record_defaults/class_methods.rb
 - lib/index-tanked/active_record_defaults/instance_companion.rb
 - lib/index-tanked/active_record_defaults/instance_methods.rb
+- lib/index-tanked/active_record_defaults/queue/document.rb
+- lib/index-tanked/active_record_defaults/queue/tasks.rb
+- lib/index-tanked/active_record_defaults/queue/worker.rb
 - lib/index-tanked/active_record_defaults/search_result.rb
 - lib/index-tanked/class_companion.rb
 - lib/index-tanked/class_methods.rb
@@ -145,6 +150,8 @@ files:
 - lib/index-tanked/search_result.rb
 - lib/index-tanked/version.rb
 - lib/index-tanked.rb
+- generators/index-tanked/index_tanked_generator.rb
+- generators/index-tanked/templates/migration.rb
 - LICENSE
 - README.markdown
 - Rakefile