ts-datetime-delta 1.0.0

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 Pat Allan
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.textile

@@ -0,0 +1,50 @@
+h1. Datetime Deltas for Thinking Sphinx
+
+h2. Installation
+
+You'll need Thinking Sphinx 1.3.0 or later.
+
+<pre><code>gem install ts-datetime-delta --source http://gemcutter.org</code></pre>
+
+In your @environment.rb@ file, with the rest of your gem dependencies:
+
+<pre><code>config.gem 'ts-datetime-delta',
+  :lib     => 'thinking_sphinx/deltas/datetime_delta'
+  :version => '>= 1.0.0',
+  :source  => 'http://gemcutter.org'</code></pre>
+
+And add the following line to the bottom of your @Rakefile@:
+
+<pre><code>require 'thinking_sphinx/deltas/datetime_delta/tasks'</code></pre>
+
+h2. Usage
+
+For the indexes you want to use this delta approach, make sure you set that up in their @define_index@ blocks.
+
+<pre><code>define_index do
+  # ...
+  
+  set_property :delta => :datetime
+end</code></pre>
+
+If you want to use a column other than @updated_at@, you can specify it using the @:delta_column@ option. The same goes for the threshold, which defaults to one day.
+
+<pre><code>set_property :delta => :datetime,
+  :threshold    => 1.hour,
+  :delta_column => :changed_at</code></pre>
+
+Then, while your Rails application is running, you'll need to run the delta indexing rake task regularly - as often as your threshold, allowing for some time for the indexing to actually happen.
+
+For example, if you're going to run the delta indexing task every hour, I would recommend setting your threshold to 70 minutes.
+
+To ensure this rake task is called regularly, it's best to set it up as a recurring task via cron or similar tools.
+
+<pre><code>rake thinking_sphinx:index:delta</code></pre>
+
+The shorthand version is:
+
+<pre><code>rake ts:in:delta</code></pre>
+
+h2. Copyright
+
+Copyright (c) 2009 Pat Allan, and released under an MIT Licence.

data/features/datetime_deltas.feature

@@ -0,0 +1,66 @@
+Feature: Datetime Delta Indexing
+  In order to have delta indexing on frequently-updated sites
+  Developers
+  Should be able to use an existing datetime column to track changes
+  
+  Scenario: Delta Index should not fire automatically
+    Given Sphinx is running
+    And I am searching on thetas
+    When I search for one
+    Then I should get 1 result
+  
+    When I change the name of theta one to eleven
+    And I wait for Sphinx to catch up
+    And I search for one
+    Then I should get 1 result
+    
+    When I search for eleven
+    Then I should get 0 results
+  
+  Scenario: Delta Index should fire when jobs are run
+    Given Sphinx is running
+    And I am searching on thetas
+    When I search for two
+    Then I should get 1 result
+  
+    When I change the name of theta two to twelve
+    And I wait for Sphinx to catch up
+    And I search for twelve
+    Then I should get 0 results
+    
+    When I index the theta datetime delta
+    And I wait for Sphinx to catch up
+    And I search for twelve
+    Then I should get 1 result
+    
+    When I search for two
+    Then I should get 0 results
+  
+  Scenario: New records should be merged into the core index
+    Given Sphinx is running
+    And I am searching on thetas
+    When I search for thirteen
+    Then I should get 0 results
+    
+    When I create a new theta named thirteen
+    And I search for thirteen
+    Then I should get 0 results
+    
+    When I index the theta datetime delta
+    And I wait for Sphinx to catch up
+    And I search for thirteen
+    Then I should get 1 result
+    
+    When I search for the document id of theta thirteen in the theta_core index
+    Then it should exist
+  
+  Scenario: Deleting records
+    Given Sphinx is running
+    And I am searching on thetas
+    When I search for three
+    Then I should get 1 result
+    
+    When I delete the theta named three
+    And I wait for Sphinx to catch up
+    And I search for three
+    Then I should get 0 results

data/features/step_definitions/common_steps.rb

@@ -0,0 +1,61 @@
+Before do
+  $queries_executed = []
+  
+  @model      = nil
+  @method     = :search
+  @query      = ""
+  @conditions = {}
+  @with       = {}
+  @without    = {}
+  @with_all   = {}
+  @options    = {}
+  @results    = nil
+end
+
+Given "Sphinx is running" do
+  ThinkingSphinx::Configuration.instance.controller.should be_running
+end
+
+Given /^I am searching on (.+)$/ do |model|
+  @model = model.gsub(/\s/, '_').singularize.camelize.constantize
+end
+
+When "I wait for Sphinx to catch up" do
+  sleep(0.25)
+end
+
+When /^I search for (\w+)$/ do |query|
+  @results = nil
+  @query = query
+end
+
+When /^I search for the document id of (\w+) (\w+) in the (\w+) index$/ do |model, name, index|
+  model   = model.gsub(/\s/, '_').camelize.constantize
+  @id     = model.find_by_name(name).sphinx_document_id
+  @index  = index
+end
+
+Then /^I should get (\d+) results?$/ do |count|
+  results.length.should == count.to_i
+end
+
+Then "it should exist" do
+  ThinkingSphinx::Search.search_for_id(@id, @index).should == true
+end
+
+Then "it should not exist" do
+  ThinkingSphinx::Search.search_for_id(@id, @index).should == false
+end
+
+def results
+  @results ||= (@model || ThinkingSphinx).send(
+    @method,
+    @query,
+    @options.merge(
+      :conditions => @conditions,
+      :with       => @with,
+      :without    => @without,
+      :with_all   => @with_all
+    )
+  )
+end

data/features/step_definitions/datetime_delta_steps.rb

@@ -0,0 +1,15 @@
+When /^I index the theta datetime delta$/ do
+  Theta.sphinx_indexes.first.delta_object.delayed_index(Theta)
+end
+
+When /^I change the name of theta (\w+) to (\w+)$/ do |current, replacement|
+  Theta.find_by_name(current).update_attributes(:name => replacement)
+end
+
+When /^I create a new theta named (\w+)$/ do |name|
+  Theta.create(:name => name)
+end
+
+When /^I delete the theta named (\w+)$/ do |name|
+  Theta.find_by_name(name).destroy
+end

data/features/support/database.example.yml

@@ -0,0 +1,3 @@
+username: root
+host: localhost
+password: 

data/features/support/database.yml

@@ -0,0 +1,5 @@
+username: thinking_sphinx
+host: localhost
+password: thinking_sphinx
+pool: 1
+min_messages: warning

data/features/support/db/fixtures/thetas.rb

@@ -0,0 +1,10 @@
+Theta.create :name => "one"
+Theta.create :name => "two"
+Theta.create :name => "three"
+Theta.create :name => "four"
+Theta.create :name => "five"
+Theta.create :name => "six"
+Theta.create :name => "seven"
+Theta.create :name => "eight"
+Theta.create :name => "nine"
+Theta.create :name => "ten"

data/features/support/db/migrations/create_thetas.rb

@@ -0,0 +1,5 @@
+ActiveRecord::Base.connection.create_table :thetas, :force => true do |t|
+  t.column :name,       :string,    :null => false
+  t.column :created_at, :datetime,  :null => false
+  t.column :updated_at, :datetime,  :null => false
+end

data/features/support/env.rb

@@ -0,0 +1,17 @@
+require 'rubygems'
+require 'cucumber'
+require 'spec/expectations'
+require 'fileutils'
+require 'active_record'
+
+$:.unshift File.dirname(__FILE__) + '/../../lib'
+
+require 'cucumber/thinking_sphinx/internal_world'
+
+world = Cucumber::ThinkingSphinx::InternalWorld.new
+world.configure_database
+
+require 'thinking_sphinx'
+require 'thinking_sphinx/deltas/datetime_delta'
+
+world.setup

data/features/support/models/theta.rb

@@ -0,0 +1,7 @@
+class Theta < ActiveRecord::Base
+  define_index do
+    indexes :name, :sortable => true
+    
+    set_property :delta => :datetime, :threshold => 1.hour
+  end
+end

data/lib/thinking_sphinx/deltas/datetime_delta.rb

@@ -0,0 +1,124 @@
+# Datetime Deltas for Thinking Sphinx
+# 
+# This documentation is aimed at those reading the code. If you're looking for
+# a guide to Thinking Sphinx and/or deltas, I recommend you start with the
+# Thinking Sphinx site instead - or the README for this library at the very
+# least.
+# 
+# @author Patrick Allan
+# @see http://ts.freelancing-gods.com Thinking Sphinx
+# 
+class ThinkingSphinx::Deltas::DatetimeDelta < ThinkingSphinx::Deltas::DefaultDelta
+  attr_accessor :column, :threshold
+  
+  # Initialises the Delta object for the given index and settings. All handled
+  # by Thinking Sphinx, so you shouldn't need to call this method yourself in
+  # general day-to-day situations.
+  # 
+  # @example
+  #   ThinkingSphinx::Deltas::DatetimeDelta.new index,
+  #     :delta_column => :updated_at,
+  #     :threshold    => 1.day
+  # 
+  # @param [ThinkingSphinx::Index] index the index using this delta object
+  # @param [Hash] options a hash of options for the index
+  # @option options [Symbol] :delta_column (:updated_at) The column to use for
+  #   tracking when a record has changed. Default to :updated_at.
+  # @option options [Integer] :threshold (1.day) The window of time to store
+  #   changes for, in seconds. Defaults to one day.
+  # 
+  def initialize(index, options = {})
+    @index      = index
+    @column     = options.delete(:delta_column) || :updated_at
+    @threshold  = options.delete(:threshold)    || 1.day
+  end
+  
+  # Does absolutely nothing, beyond returning true. Thinking Sphinx expects
+  # this method, though, and we don't want to use the inherited behaviour from
+  # DefaultDelta.
+  # 
+  # All the real indexing logic is done by the delayed_index method.
+  # 
+  # @param [Class] model the ActiveRecord model to index.
+  # @param [ActiveRecord::Base] instance the instance of the given model that
+  #   has changed. Optional.
+  # @return [Boolean] true
+  # @see #delayed_index
+  # 
+  def index(model, instance = nil)
+    # do nothing
+    true
+  end
+  
+  # Processes the delta index for the given model, and then merges the relevant
+  # core and delta indexes together. By default, the output of these indexer
+  # commands are printed to stdout. If you'd rather it didn't, set
+  # ThinkingSphinx.suppress_delta_output to true.
+  # 
+  # @param [Class] model the ActiveRecord model to index
+  # @return [Boolean] true
+  # 
+  def delayed_index(model)
+    config = ThinkingSphinx::Configuration.instance
+    rotate = ThinkingSphinx.sphinx_running? ? " --rotate" : ""
+    
+    output = `#{config.bin_path}#{config.indexer_binary_name} --config #{config.config_file}#{rotate} #{delta_index_name model}`
+    output += `#{config.bin_path}#{config.indexer_binary_name} --config #{config.config_file}#{rotate} --merge #{core_index_name model} #{delta_index_name model} --merge-dst-range sphinx_deleted 0 0`
+    puts output unless ThinkingSphinx.suppress_delta_output?
+    
+    true
+  end
+  
+  # Toggles the given instance to be flagged as part of the next delta indexing.
+  # For datetime deltas, this means do nothing at all.
+  # 
+  # @param [ActiveRecord::Base] instance the instance to be toggled
+  # 
+  def toggle(instance)
+    # do nothing
+  end
+  
+  # Report whether a given instance is considered toggled (part of the next
+  # delta process). For datetime deltas, this is true if the delta column
+  # (updated_at by default) has a value within the threshold. Otherwise, false
+  # is returned.
+  # 
+  # @param [ActiveRecord::Base] instance the instance to check
+  # @return [Boolean] True if within the threshold window, otherwise false.
+  # 
+  def toggled(instance)
+    instance.send(@column) > @threshold.ago
+  end
+  
+  # Returns the SQL query that resets the model data after a normal index. For
+  # datetime deltas, nothing needs to be done, so this method returns nil.
+  # 
+  # @param [Class] model The ActiveRecord model that is requesting the query
+  # @return [NilClass] Always nil
+  # 
+  def reset_query(model)
+    nil
+  end
+  
+  # A SQL condition (as part of the WHERE clause) that limits the result set to
+  # just the delta data, or all data, depending on whether the toggled argument
+  # is true or not. For datetime deltas, the former value is a check on the 
+  # delta column being within the threshold. In the latter's case, no condition
+  # is needed, so nil is returned.
+  # 
+  # @param [Class] model The ActiveRecord model to generate the SQL condition
+  #   for.
+  # @param [Boolean] toggled Whether the query should request delta documents or
+  #   all documents.
+  # @return [String, NilClass] The SQL condition if the toggled version is
+  #   requested, otherwise nil.
+  # 
+  def clause(model, toggled)
+    if toggled
+      "#{model.quoted_table_name}.#{model.connection.quote_column_name(@column.to_s)}" +
+      " > #{adapter.time_difference(@threshold)}"
+    else
+      nil
+    end
+  end
+end

data/lib/thinking_sphinx/deltas/datetime_delta/tasks.rb

@@ -0,0 +1,23 @@
+namespace :thinking_sphinx do
+  namespace :index do
+    desc "Index Thinking Sphinx datetime delta indexes"
+    task :delta => :app_env do
+      ThinkingSphinx.indexed_models.select { |model|
+        model.constantize.sphinx_indexes.any? { |index| index.delta? }
+      }.each do |model|
+        model.constantize.sphinx_indexes.select { |index|
+          index.delta? && index.delta_object.respond_to?(:delayed_index)
+        }.each { |index|
+          index.delta_object.delayed_index(index.model)
+        }
+      end
+    end
+  end
+end
+
+namespace :ts do
+  namespace :in do
+    desc "Index Thinking Sphinx datetime delta indexes"
+    task :delta => "thinking_sphinx:index:delta"
+  end
+end

data/spec/spec.opts

@@ -0,0 +1 @@
+--color

data/spec/spec_helper.rb

@@ -0,0 +1,12 @@
+$LOAD_PATH.unshift File.join(File.dirname(__FILE__), '..', 'lib')
+
+require 'rubygems'
+require 'spec'
+require 'spec/autorun'
+
+require 'thinking_sphinx'
+require 'thinking_sphinx/deltas/delayed_delta'
+
+Spec::Runner.configure do |config|
+  #
+end

data/spec/thinking_sphinx/deltas/datetime_delta_spec.rb

@@ -0,0 +1,130 @@
+require 'spec/spec_helper'
+
+describe ThinkingSphinx::Deltas::DatetimeDelta do
+  before :each do
+    @datetime_delta = ThinkingSphinx::Deltas::DatetimeDelta.new(
+      stub('index'), {}
+    )
+  end
+  
+  describe '#index' do
+    it "should do nothing to the model" do
+      @datetime_delta.index(stub('model'))
+    end
+    
+    it "should do nothing to the instance, if provided" do
+      @datetime_delta.index(stub('model'), stub('instance'))
+    end
+    
+    it "should make no system calls" do
+      @datetime_delta.stub! :`      => true
+      @datetime_delta.stub! :system => true
+      
+      @datetime_delta.should_not_receive(:`)
+      @datetime_delta.should_not_receive(:system)
+      
+      @datetime_delta.index(stub('model'), stub('instance'))
+    end
+    
+    it "should return true" do
+      @datetime_delta.index(stub('model')).should be_true
+    end
+  end
+  
+  describe '#delayed_index' do
+    before :each do
+      @model = stub('foo')
+      @model.stub!(:name => 'foo')
+      @model.stub!(:source_of_sphinx_index => @model)
+      
+      ThinkingSphinx.suppress_delta_output = false
+      
+      @datetime_delta.stub! :`    => ""
+      @datetime_delta.stub! :puts => nil
+    end
+    
+    it "should process the delta index for the given model" do
+      @datetime_delta.should_receive(:`).
+        with('indexer --config /config/development.sphinx.conf foo_delta')
+      
+      @datetime_delta.delayed_index(@model)
+    end
+    
+    it "should merge the core and delta indexes for the given model" do
+      @datetime_delta.should_receive(:`).with('indexer --config /config/development.sphinx.conf --merge foo_core foo_delta --merge-dst-range sphinx_deleted 0 0')
+      
+      @datetime_delta.delayed_index(@model)
+    end
+    
+    it "should include --rotate if Sphinx is running" do
+      ThinkingSphinx.stub!(:sphinx_running? => true)
+      @datetime_delta.should_receive(:`) do |command|
+        command.should match(/\s--rotate\s/)
+      end
+      
+      @datetime_delta.delayed_index(@model)
+    end
+    
+    it "should output the details by default" do
+      @datetime_delta.should_receive(:puts)
+      
+      @datetime_delta.delayed_index(@model)
+    end
+    
+    it "should hide the details if suppressing delta output" do
+      ThinkingSphinx.suppress_delta_output = true
+      @datetime_delta.should_not_receive(:puts)
+      
+      @datetime_delta.delayed_index(@model)
+    end
+  end
+  
+  describe '#toggle' do
+    it "should do nothing to the instance" do
+      @datetime_delta.toggle(stub('instance'))
+    end
+  end
+  
+  describe '#toggled' do
+    it "should return true if the column value is more recent than the threshold" do
+      instance = stub('instance', :updated_at => 20.minutes.ago)
+      @datetime_delta.threshold = 30.minutes
+      
+      @datetime_delta.toggled(instance).should be_true
+    end
+    
+    it "should return false if the column value is older than the threshold" do
+      instance = stub('instance', :updated_at => 30.minutes.ago)
+      @datetime_delta.threshold = 20.minutes
+      
+      @datetime_delta.toggled(instance).should be_false
+    end
+  end
+  
+  describe '#reset_query' do
+    it "should be nil" do
+      @datetime_delta.reset_query(@model).should be_nil
+    end
+  end
+  
+  describe '#clause' do
+    before :each do
+      @model = stub('model', :connection => stub('connection'))
+      @model.stub!(:quoted_table_name => '`foo`')
+      @model.connection.stub!(:quote_column_name => '`updated_at`')
+      
+      @datetime_delta.stub!(
+        :adapter => stub('adapter', :time_difference => 'time_difference')
+      )
+    end
+    
+    it "should return nil if not for the toggled results" do
+      @datetime_delta.clause(@model, false).should be_nil
+    end
+    
+    it "should return only records within the threshold" do
+      @datetime_delta.clause(@model, true).
+        should == '`foo`.`updated_at` > time_difference'
+    end
+  end
+end

metadata

@@ -0,0 +1,99 @@
+--- !ruby/object:Gem::Specification 
+name: ts-datetime-delta
+version: !ruby/object:Gem::Version 
+  version: 1.0.0
+platform: ruby
+authors: 
+- Pat Allan
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2009-11-03 00:00:00 +11:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: rspec
+  type: :development
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 1.2.9
+    version: 
+- !ruby/object:Gem::Dependency 
+  name: yard
+  type: :development
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: "0"
+    version: 
+- !ruby/object:Gem::Dependency 
+  name: cucumber
+  type: :development
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: "0"
+    version: 
+description: Manage delta indexes via datetime columns for Thinking Sphinx
+email: pat@freelancing-gods.com
+executables: []
+
+extensions: []
+
+extra_rdoc_files: 
+- LICENSE
+- README.textile
+files: 
+- LICENSE
+- README.textile
+- lib/thinking_sphinx/deltas/datetime_delta.rb
+- lib/thinking_sphinx/deltas/datetime_delta/tasks.rb
+has_rdoc: true
+homepage: http://github.com/freelancing-god/ts-datetime-delta
+licenses: []
+
+post_install_message: 
+rdoc_options: 
+- --charset=UTF-8
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.3.5
+signing_key: 
+specification_version: 3
+summary: Thinking Sphinx - Datetime Deltas
+test_files: 
+- features/datetime_deltas.feature
+- features/step_definitions/common_steps.rb
+- features/step_definitions/datetime_delta_steps.rb
+- features/support/database.example.yml
+- features/support/database.yml
+- features/support/db/fixtures/thetas.rb
+- features/support/db/migrations/create_thetas.rb
+- features/support/env.rb
+- features/support/models/theta.rb
+- spec/spec.opts
+- spec/spec_helper.rb
+- spec/thinking_sphinx/deltas/datetime_delta_spec.rb