ts-delayed-delta 1.0.0
Sign up to get free protection for your applications and to get access to all the features.
- data/LICENSE +20 -0
- data/README.textile +50 -0
- data/features/delayed_deltas.feature +37 -0
- data/features/step_definitions/common_steps.rb +48 -0
- data/features/step_definitions/delayed_delta_steps.rb +7 -0
- data/features/support/database.example.yml +3 -0
- data/features/support/database.yml +5 -0
- data/features/support/db/fixtures/delayed_betas.rb +10 -0
- data/features/support/db/migrations/create_delayed_betas.rb +17 -0
- data/features/support/env.rb +17 -0
- data/features/support/models/delayed_beta.rb +7 -0
- data/lib/thinking_sphinx/deltas/delayed_delta.rb +64 -0
- data/lib/thinking_sphinx/deltas/delayed_delta/delta_job.rb +27 -0
- data/lib/thinking_sphinx/deltas/delayed_delta/flag_as_deleted_job.rb +38 -0
- data/lib/thinking_sphinx/deltas/delayed_delta/job.rb +54 -0
- data/lib/thinking_sphinx/deltas/delayed_delta/tasks.rb +20 -0
- data/spec/spec.opts +1 -0
- data/spec/spec_helper.rb +12 -0
- data/spec/thinking_sphinx/deltas/delayed_delta/delta_job_spec.rb +35 -0
- data/spec/thinking_sphinx/deltas/delayed_delta/flag_as_deleted_job_spec.rb +70 -0
- data/spec/thinking_sphinx/deltas/delayed_delta/job_spec.rb +52 -0
- data/spec/thinking_sphinx/deltas/delayed_delta_spec.rb +126 -0
- metadata +115 -0
data/LICENSE
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
Copyright (c) 2009 Pat Allan
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
4
|
+
a copy of this software and associated documentation files (the
|
5
|
+
"Software"), to deal in the Software without restriction, including
|
6
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
7
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
8
|
+
permit persons to whom the Software is furnished to do so, subject to
|
9
|
+
the following conditions:
|
10
|
+
|
11
|
+
The above copyright notice and this permission notice shall be
|
12
|
+
included in all copies or substantial portions of the Software.
|
13
|
+
|
14
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
15
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
16
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
17
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
18
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
19
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
20
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.textile
ADDED
@@ -0,0 +1,50 @@
|
|
1
|
+
h1. Delayed Deltas for Thinking Sphinx
|
2
|
+
|
3
|
+
h2. Installation
|
4
|
+
|
5
|
+
You'll need Thinking Sphinx 1.3.0 or later, and Delayed Job as well. The latter is flagged as a dependency.
|
6
|
+
|
7
|
+
<pre><code>gem install ts-delayed-delta --source http://gemcutter.org</code></pre>
|
8
|
+
|
9
|
+
In your @environment.rb@ file, with the rest of your gem dependencies:
|
10
|
+
|
11
|
+
<pre><code>config.gem 'ts-delayed-delta',
|
12
|
+
:lib => 'thinking_sphinx/deltas/delayed_delta'
|
13
|
+
:version => '>= 1.0.0',
|
14
|
+
:source => 'http://gemcutter.org'</code></pre>
|
15
|
+
|
16
|
+
And add the following line to the bottom of your @Rakefile@:
|
17
|
+
|
18
|
+
<pre><code>require 'thinking_sphinx/deltas/delayed_delta/tasks'</code></pre>
|
19
|
+
|
20
|
+
If this is your first time running Delayed Job, then you're going to need the jobs table migration as well:
|
21
|
+
|
22
|
+
<pre><code>script/generate delayed_job</code></pre>
|
23
|
+
|
24
|
+
For the indexes you want to use this delta approach, make sure you set that up in their @define_index@ blocks.
|
25
|
+
|
26
|
+
<pre><code>define_index do
|
27
|
+
# ...
|
28
|
+
|
29
|
+
set_property :delta => :delayed
|
30
|
+
end</code></pre>
|
31
|
+
|
32
|
+
If you've never used delta indexes before, you'll want to add the boolean column named delta to each model that is using the approach.
|
33
|
+
|
34
|
+
<pre><code>def self.up
|
35
|
+
add_column :articles, :delta, :boolean, :default => true, :null => false
|
36
|
+
end</code></pre>
|
37
|
+
|
38
|
+
h2. Usage
|
39
|
+
|
40
|
+
Once you've got it all set up, all you need to do is make sure that the delayed job process is running - either by Delayed Job's built-in approach, or Thinking Sphinx's custom rake task:
|
41
|
+
|
42
|
+
<pre><code>rake thinking_sphinx:delayed_delta</code></pre>
|
43
|
+
|
44
|
+
There's also a short name for the same task, to save your fingers some effort:
|
45
|
+
|
46
|
+
<pre><code>rake ts:dd</code></pre>
|
47
|
+
|
48
|
+
h2. Copyright
|
49
|
+
|
50
|
+
Copyright (c) 2009 Pat Allan, and released under an MIT Licence.
|
@@ -0,0 +1,37 @@
|
|
1
|
+
Feature: Delayed Delta Indexing
|
2
|
+
In order to have delta indexing on frequently-updated sites
|
3
|
+
Developers
|
4
|
+
Should be able to use delayed_job to handle delta indexes to lower system load
|
5
|
+
|
6
|
+
Scenario: Delta Index should not fire automatically
|
7
|
+
Given Sphinx is running
|
8
|
+
And I am searching on delayed betas
|
9
|
+
When I search for one
|
10
|
+
Then I should get 1 result
|
11
|
+
|
12
|
+
When I change the name of delayed beta one to eleven
|
13
|
+
And I wait for Sphinx to catch up
|
14
|
+
And I search for one
|
15
|
+
Then I should get 1 result
|
16
|
+
|
17
|
+
When I search for eleven
|
18
|
+
Then I should get 0 results
|
19
|
+
|
20
|
+
Scenario: Delta Index should fire when jobs are run
|
21
|
+
Given Sphinx is running
|
22
|
+
And I am searching on delayed betas
|
23
|
+
When I search for one
|
24
|
+
Then I should get 1 result
|
25
|
+
|
26
|
+
When I change the name of delayed beta two to twelve
|
27
|
+
And I wait for Sphinx to catch up
|
28
|
+
And I search for twelve
|
29
|
+
Then I should get 0 results
|
30
|
+
|
31
|
+
When I run the delayed jobs
|
32
|
+
And I wait for Sphinx to catch up
|
33
|
+
And I search for twelve
|
34
|
+
Then I should get 1 result
|
35
|
+
|
36
|
+
When I search for two
|
37
|
+
Then I should get 0 results
|
@@ -0,0 +1,48 @@
|
|
1
|
+
Before do
|
2
|
+
$queries_executed = []
|
3
|
+
ThinkingSphinx::Deltas::Job.cancel_thinking_sphinx_jobs
|
4
|
+
|
5
|
+
@model = nil
|
6
|
+
@method = :search
|
7
|
+
@query = ""
|
8
|
+
@conditions = {}
|
9
|
+
@with = {}
|
10
|
+
@without = {}
|
11
|
+
@with_all = {}
|
12
|
+
@options = {}
|
13
|
+
@results = nil
|
14
|
+
end
|
15
|
+
|
16
|
+
Given "Sphinx is running" do
|
17
|
+
ThinkingSphinx::Configuration.instance.controller.should be_running
|
18
|
+
end
|
19
|
+
|
20
|
+
Given /^I am searching on (.+)$/ do |model|
|
21
|
+
@model = model.gsub(/\s/, '_').singularize.camelize.constantize
|
22
|
+
end
|
23
|
+
|
24
|
+
When "I wait for Sphinx to catch up" do
|
25
|
+
sleep(0.25)
|
26
|
+
end
|
27
|
+
|
28
|
+
When /^I search for (\w+)$/ do |query|
|
29
|
+
@results = nil
|
30
|
+
@query = query
|
31
|
+
end
|
32
|
+
|
33
|
+
Then /^I should get (\d+) results?$/ do |count|
|
34
|
+
results.length.should == count.to_i
|
35
|
+
end
|
36
|
+
|
37
|
+
def results
|
38
|
+
@results ||= (@model || ThinkingSphinx).send(
|
39
|
+
@method,
|
40
|
+
@query,
|
41
|
+
@options.merge(
|
42
|
+
:conditions => @conditions,
|
43
|
+
:with => @with,
|
44
|
+
:without => @without,
|
45
|
+
:with_all => @with_all
|
46
|
+
)
|
47
|
+
)
|
48
|
+
end
|
@@ -0,0 +1,10 @@
|
|
1
|
+
DelayedBeta.create :name => "one"
|
2
|
+
DelayedBeta.create :name => "two"
|
3
|
+
DelayedBeta.create :name => "three"
|
4
|
+
DelayedBeta.create :name => "four"
|
5
|
+
DelayedBeta.create :name => "five"
|
6
|
+
DelayedBeta.create :name => "six"
|
7
|
+
DelayedBeta.create :name => "seven"
|
8
|
+
DelayedBeta.create :name => "eight"
|
9
|
+
DelayedBeta.create :name => "nine"
|
10
|
+
DelayedBeta.create :name => "ten"
|
@@ -0,0 +1,17 @@
|
|
1
|
+
ActiveRecord::Base.connection.create_table :delayed_betas, :force => true do |t|
|
2
|
+
t.column :name, :string, :null => false
|
3
|
+
t.column :delta, :boolean, :null => false, :default => false
|
4
|
+
end
|
5
|
+
|
6
|
+
ActiveRecord::Base.connection.create_table :delayed_jobs, :force => true do |t|
|
7
|
+
t.column :priority, :integer, :default => 0
|
8
|
+
t.column :attempts, :integer, :default => 0
|
9
|
+
t.column :handler, :text
|
10
|
+
t.column :last_error, :string
|
11
|
+
t.column :run_at, :datetime
|
12
|
+
t.column :locked_at, :datetime
|
13
|
+
t.column :failed_at, :datetime
|
14
|
+
t.column :locked_by, :string
|
15
|
+
t.column :created_at, :datetime
|
16
|
+
t.column :updated_at, :datetime
|
17
|
+
end
|
@@ -0,0 +1,17 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'cucumber'
|
3
|
+
require 'spec/expectations'
|
4
|
+
require 'fileutils'
|
5
|
+
require 'active_record'
|
6
|
+
|
7
|
+
$:.unshift File.dirname(__FILE__) + '/../../lib'
|
8
|
+
|
9
|
+
require 'cucumber/thinking_sphinx/internal_world'
|
10
|
+
|
11
|
+
world = Cucumber::ThinkingSphinx::InternalWorld.new
|
12
|
+
world.configure_database
|
13
|
+
|
14
|
+
require 'thinking_sphinx'
|
15
|
+
require 'thinking_sphinx/deltas/delayed_delta'
|
16
|
+
|
17
|
+
world.setup
|
@@ -0,0 +1,64 @@
|
|
1
|
+
require 'delayed_job'
|
2
|
+
|
3
|
+
require 'thinking_sphinx/deltas/delayed_delta/delta_job'
|
4
|
+
require 'thinking_sphinx/deltas/delayed_delta/flag_as_deleted_job'
|
5
|
+
require 'thinking_sphinx/deltas/delayed_delta/job'
|
6
|
+
|
7
|
+
# Delayed Deltas for Thinking Sphinx, using Delayed Job.
|
8
|
+
#
|
9
|
+
# This documentation is aimed at those reading the code. If you're looking for
|
10
|
+
# a guide to Thinking Sphinx and/or deltas, I recommend you start with the
|
11
|
+
# Thinking Sphinx site instead - or the README for this library at the very
|
12
|
+
# least.
|
13
|
+
#
|
14
|
+
# @author Patrick Allan
|
15
|
+
# @see http://ts.freelancing-gods.com Thinking Sphinx
|
16
|
+
#
|
17
|
+
class ThinkingSphinx::Deltas::DelayedDelta < ThinkingSphinx::Deltas::DefaultDelta
|
18
|
+
|
19
|
+
# Adds a job to the queue for processing the given model's delta index. A job
|
20
|
+
# for hiding the instance in the core index is also created, if an instance is
|
21
|
+
# provided.
|
22
|
+
#
|
23
|
+
# Neither job will be queued if updates or deltas are disabled, or if the
|
24
|
+
# instance (when given) is not toggled to be in the delta index. The first two
|
25
|
+
# options are controlled via ThinkingSphinx.updates_enabled? and
|
26
|
+
# ThinkingSphinx.deltas_enabled?.
|
27
|
+
#
|
28
|
+
# @param [Class] model the ActiveRecord model to index.
|
29
|
+
# @param [ActiveRecord::Base] instance the instance of the given model that
|
30
|
+
# has changed. Optional.
|
31
|
+
# @return [Boolean] true
|
32
|
+
#
|
33
|
+
def index(model, instance = nil)
|
34
|
+
return true if skip? instance
|
35
|
+
|
36
|
+
ThinkingSphinx::Deltas::Job.enqueue(
|
37
|
+
ThinkingSphinx::Deltas::DeltaJob.new(delta_index_name(model)),
|
38
|
+
ThinkingSphinx::Configuration.instance.delayed_job_priority
|
39
|
+
)
|
40
|
+
|
41
|
+
Delayed::Job.enqueue(
|
42
|
+
ThinkingSphinx::Deltas::FlagAsDeletedJob.new(
|
43
|
+
core_index_name(model), instance.sphinx_document_id
|
44
|
+
),
|
45
|
+
ThinkingSphinx::Configuration.instance.delayed_job_priority
|
46
|
+
) if instance
|
47
|
+
|
48
|
+
true
|
49
|
+
end
|
50
|
+
|
51
|
+
private
|
52
|
+
|
53
|
+
# Checks whether jobs should be enqueued. Only true if updates and deltas are
|
54
|
+
# enabled, and the instance (if there is one) is toggled.
|
55
|
+
#
|
56
|
+
# @param [ActiveRecord::Base, NilClass] instance
|
57
|
+
# @return [Boolean]
|
58
|
+
#
|
59
|
+
def skip?(instance)
|
60
|
+
!ThinkingSphinx.updates_enabled? ||
|
61
|
+
!ThinkingSphinx.deltas_enabled? ||
|
62
|
+
(instance && !toggled(instance))
|
63
|
+
end
|
64
|
+
end
|
@@ -0,0 +1,27 @@
|
|
1
|
+
# A simple job class that processes a given index.
|
2
|
+
#
|
3
|
+
class ThinkingSphinx::Deltas::DeltaJob
|
4
|
+
attr_accessor :index
|
5
|
+
|
6
|
+
# Initialises the object with an index name.
|
7
|
+
#
|
8
|
+
# @param [String] index the name of the Sphinx index
|
9
|
+
#
|
10
|
+
def initialize(index)
|
11
|
+
@index = index
|
12
|
+
end
|
13
|
+
|
14
|
+
# Runs Sphinx's indexer tool to process the index. Currently assumes Sphinx is
|
15
|
+
# running.
|
16
|
+
#
|
17
|
+
# @return [Boolean] true
|
18
|
+
#
|
19
|
+
def perform
|
20
|
+
config = ThinkingSphinx::Configuration.instance
|
21
|
+
|
22
|
+
output = `#{config.bin_path}#{config.indexer_binary_name} --config #{config.config_file} --rotate #{index}`
|
23
|
+
puts output unless ThinkingSphinx.suppress_delta_output?
|
24
|
+
|
25
|
+
true
|
26
|
+
end
|
27
|
+
end
|
@@ -0,0 +1,38 @@
|
|
1
|
+
# A simple job for flagging a specified Sphinx document in a given index as
|
2
|
+
# 'deleted'.
|
3
|
+
#
|
4
|
+
class ThinkingSphinx::Deltas::FlagAsDeletedJob
|
5
|
+
attr_accessor :index, :document_id
|
6
|
+
|
7
|
+
# Initialises the object with an index name and document id. Please note that
|
8
|
+
# the document id is Sphinx's unique identifier, and will almost certainly not
|
9
|
+
# be the model instance's primary key value.
|
10
|
+
#
|
11
|
+
# @param [String] index The index name
|
12
|
+
# @param [Integer] document_id The document id
|
13
|
+
#
|
14
|
+
def initialize(index, document_id)
|
15
|
+
@index, @document_id = index, document_id
|
16
|
+
end
|
17
|
+
|
18
|
+
# Updates the sphinx_deleted attribute for the given document, setting the
|
19
|
+
# value to 1 (true). This is not a special attribute in Sphinx, but is used
|
20
|
+
# by Thinking Sphinx to ignore deleted values between full re-indexing. It's
|
21
|
+
# particularly useful in this situation to avoid old values in the core index
|
22
|
+
# and just use the new values in the delta index as a reference point.
|
23
|
+
#
|
24
|
+
# @return [Boolean] true
|
25
|
+
#
|
26
|
+
def perform
|
27
|
+
config = ThinkingSphinx::Configuration.instance
|
28
|
+
|
29
|
+
config.client.update(
|
30
|
+
@index,
|
31
|
+
['sphinx_deleted'],
|
32
|
+
{@document_id => [1]}
|
33
|
+
) if ThinkingSphinx.sphinx_running? &&
|
34
|
+
ThinkingSphinx::Search.search_for_id(@document_id, @index)
|
35
|
+
|
36
|
+
true
|
37
|
+
end
|
38
|
+
end
|
@@ -0,0 +1,54 @@
|
|
1
|
+
# A custom job model, subclassed from Delayed::Job. The two things it does
|
2
|
+
# differently is that it checks for duplicate tasks before enqueuing them, and
|
3
|
+
# provides the option to remove all Delayed Delta jobs from the queue.
|
4
|
+
#
|
5
|
+
# As such, this class should not be used for any other tasks.
|
6
|
+
#
|
7
|
+
class ThinkingSphinx::Deltas::Job < Delayed::Job
|
8
|
+
# Adds a job to the queue, if it doesn't already exist. This is to ensure
|
9
|
+
# multiple indexing requests for the same delta index don't get added, as the
|
10
|
+
# index only needs to be processed once.
|
11
|
+
#
|
12
|
+
# Because indexing jobs are all the same object, with a single instance
|
13
|
+
# variable (the index name), they all get serialised to the same YAML value.
|
14
|
+
#
|
15
|
+
# @param [Object] object The job, which must respond to the #perform method.
|
16
|
+
# @param [Integer] priority (0)
|
17
|
+
#
|
18
|
+
def self.enqueue(object, priority = 0)
|
19
|
+
Delayed::Job.enqueue(object, priority) unless duplicates_exist(object)
|
20
|
+
end
|
21
|
+
|
22
|
+
# Remove all Thinking Sphinx/Delayed Delta jobs from the queue. If the
|
23
|
+
# delayed_jobs table does not exist, this method will do nothing.
|
24
|
+
#
|
25
|
+
def self.cancel_thinking_sphinx_jobs
|
26
|
+
if connection.tables.include?("delayed_jobs")
|
27
|
+
delete_all("handler LIKE '--- !ruby/object:ThinkingSphinx::Deltas::%'")
|
28
|
+
end
|
29
|
+
end
|
30
|
+
|
31
|
+
# This is to stop ActiveRecord complaining about a missing database when
|
32
|
+
# running specs (otherwise printing failure messages raises confusing stack
|
33
|
+
# traces).
|
34
|
+
#
|
35
|
+
def self.inspect
|
36
|
+
"Job"
|
37
|
+
end
|
38
|
+
|
39
|
+
private
|
40
|
+
|
41
|
+
# Checks whether a given job already exists in the queue.
|
42
|
+
#
|
43
|
+
# @param [Object] object The job
|
44
|
+
# @return [Boolean] True if a duplicate of the job already exists in the queue
|
45
|
+
#
|
46
|
+
def self.duplicates_exist(object)
|
47
|
+
count(
|
48
|
+
:conditions => {
|
49
|
+
:handler => object.to_yaml,
|
50
|
+
:locked_at => nil
|
51
|
+
}
|
52
|
+
) > 0
|
53
|
+
end
|
54
|
+
end
|
@@ -0,0 +1,20 @@
|
|
1
|
+
namespace :thinking_sphinx do
|
2
|
+
task :index do
|
3
|
+
ThinkingSphinx::Deltas::Job.cancel_thinking_sphinx_jobs
|
4
|
+
end
|
5
|
+
|
6
|
+
desc "Process stored delta index requests"
|
7
|
+
task :delayed_delta => :app_env do
|
8
|
+
require 'delayed/worker'
|
9
|
+
|
10
|
+
Delayed::Worker.new(
|
11
|
+
:min_priority => ENV['MIN_PRIORITY'],
|
12
|
+
:max_priority => ENV['MAX_PRIORITY']
|
13
|
+
).start
|
14
|
+
end
|
15
|
+
end
|
16
|
+
|
17
|
+
namespace :ts do
|
18
|
+
desc "Process stored delta index requests"
|
19
|
+
task :dd => "thinking_sphinx:delayed_delta"
|
20
|
+
end
|
data/spec/spec.opts
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
--color
|
data/spec/spec_helper.rb
ADDED
@@ -0,0 +1,12 @@
|
|
1
|
+
$LOAD_PATH.unshift File.join(File.dirname(__FILE__), '..', 'lib')
|
2
|
+
|
3
|
+
require 'rubygems'
|
4
|
+
require 'spec'
|
5
|
+
require 'spec/autorun'
|
6
|
+
|
7
|
+
require 'thinking_sphinx'
|
8
|
+
require 'thinking_sphinx/deltas/delayed_delta'
|
9
|
+
|
10
|
+
Spec::Runner.configure do |config|
|
11
|
+
#
|
12
|
+
end
|
@@ -0,0 +1,35 @@
|
|
1
|
+
require 'spec/spec_helper'
|
2
|
+
|
3
|
+
describe ThinkingSphinx::Deltas::DeltaJob do
|
4
|
+
describe '#perform' do
|
5
|
+
before :each do
|
6
|
+
ThinkingSphinx.suppress_delta_output = false
|
7
|
+
|
8
|
+
@delta_job = ThinkingSphinx::Deltas::DeltaJob.new('foo_core')
|
9
|
+
@delta_job.stub! :` => true
|
10
|
+
@delta_job.stub! :puts => nil
|
11
|
+
end
|
12
|
+
|
13
|
+
it "should output the delta indexing by default" do
|
14
|
+
@delta_job.should_receive(:puts)
|
15
|
+
|
16
|
+
@delta_job.perform
|
17
|
+
end
|
18
|
+
|
19
|
+
it "should not output the delta indexing if requested" do
|
20
|
+
ThinkingSphinx.suppress_delta_output = true
|
21
|
+
@delta_job.should_not_receive(:puts)
|
22
|
+
|
23
|
+
@delta_job.perform
|
24
|
+
end
|
25
|
+
|
26
|
+
it "should process just the requested index" do
|
27
|
+
@delta_job.should_receive(:`) do |command|
|
28
|
+
command.should match(/foo_core/)
|
29
|
+
command.should_not match(/--all/)
|
30
|
+
end
|
31
|
+
|
32
|
+
@delta_job.perform
|
33
|
+
end
|
34
|
+
end
|
35
|
+
end
|
@@ -0,0 +1,70 @@
|
|
1
|
+
require 'spec/spec_helper'
|
2
|
+
|
3
|
+
describe ThinkingSphinx::Deltas::FlagAsDeletedJob do
|
4
|
+
describe '#perform' do
|
5
|
+
before :each do
|
6
|
+
ThinkingSphinx.updates_enabled = true
|
7
|
+
@client = stub('client', :update => true)
|
8
|
+
|
9
|
+
ThinkingSphinx::Configuration.instance.stub!(:client => @client)
|
10
|
+
ThinkingSphinx::Search.stub!(:search_for_id => true)
|
11
|
+
ThinkingSphinx.stub!(:sphinx_running? => true)
|
12
|
+
|
13
|
+
@job = ThinkingSphinx::Deltas::FlagAsDeletedJob.new('foo_core', 12)
|
14
|
+
end
|
15
|
+
|
16
|
+
it "should not update if Sphinx isn't running" do
|
17
|
+
ThinkingSphinx.stub!(:sphinx_running? => false)
|
18
|
+
@client.should_not_receive(:update)
|
19
|
+
|
20
|
+
@job.perform
|
21
|
+
end
|
22
|
+
|
23
|
+
it "should not update if the document isn't in the index" do
|
24
|
+
ThinkingSphinx::Search.stub!(:search_for_id => false)
|
25
|
+
@client.should_not_receive(:update)
|
26
|
+
|
27
|
+
@job.perform
|
28
|
+
end
|
29
|
+
|
30
|
+
it "should update the specified index" do
|
31
|
+
@client.should_receive(:update) do |index, attributes, values|
|
32
|
+
index.should == 'foo_core'
|
33
|
+
end
|
34
|
+
|
35
|
+
@job.perform
|
36
|
+
end
|
37
|
+
|
38
|
+
it "should update the sphinx_deleted attribute" do
|
39
|
+
@client.should_receive(:update) do |index, attributes, values|
|
40
|
+
attributes.should == ['sphinx_deleted']
|
41
|
+
end
|
42
|
+
|
43
|
+
@job.perform
|
44
|
+
end
|
45
|
+
|
46
|
+
it "should set sphinx_deleted for the given document to true" do
|
47
|
+
@client.should_receive(:update) do |index, attributes, values|
|
48
|
+
values[12].should == [1]
|
49
|
+
end
|
50
|
+
|
51
|
+
@job.perform
|
52
|
+
end
|
53
|
+
|
54
|
+
it "should check for the existence of the document in the specified index" do
|
55
|
+
ThinkingSphinx::Search.should_receive(:search_for_id) do |id, index|
|
56
|
+
index.should == 'foo_core'
|
57
|
+
end
|
58
|
+
|
59
|
+
@job.perform
|
60
|
+
end
|
61
|
+
|
62
|
+
it "should check for the existence of the given document id" do
|
63
|
+
ThinkingSphinx::Search.should_receive(:search_for_id) do |id, index|
|
64
|
+
id.should == 12
|
65
|
+
end
|
66
|
+
|
67
|
+
@job.perform
|
68
|
+
end
|
69
|
+
end
|
70
|
+
end
|
@@ -0,0 +1,52 @@
|
|
1
|
+
require 'spec/spec_helper'
|
2
|
+
|
3
|
+
describe ThinkingSphinx::Deltas::Job do
|
4
|
+
describe '.enqueue' do
|
5
|
+
before :each do
|
6
|
+
ThinkingSphinx::Deltas::Job.stub!(:count => 0)
|
7
|
+
end
|
8
|
+
|
9
|
+
it "should enqueue if there's no existing jobs for the same index" do
|
10
|
+
Delayed::Job.should_receive(:enqueue)
|
11
|
+
|
12
|
+
ThinkingSphinx::Deltas::Job.enqueue(stub('object'))
|
13
|
+
end
|
14
|
+
|
15
|
+
it "should not enqueue the job if there's an existing job already" do
|
16
|
+
ThinkingSphinx::Deltas::Job.stub!(:count => 1)
|
17
|
+
Delayed::Job.should_not_receive(:enqueue)
|
18
|
+
|
19
|
+
ThinkingSphinx::Deltas::Job.enqueue(stub('object'))
|
20
|
+
end
|
21
|
+
end
|
22
|
+
|
23
|
+
describe '.cancel_thinking_sphinx_jobs' do
|
24
|
+
before :each do
|
25
|
+
ThinkingSphinx::Deltas::Job.stub!(:connection, stub('connection'))
|
26
|
+
ThinkingSphinx::Deltas::Job.stub!(:delete_all => true)
|
27
|
+
end
|
28
|
+
|
29
|
+
it "should not delete any rows if the delayed_jobs table does not exist" do
|
30
|
+
ThinkingSphinx::Deltas::Job.connection.stub!(:tables => [])
|
31
|
+
ThinkingSphinx::Deltas::Job.should_not_receive(:delete_all)
|
32
|
+
|
33
|
+
ThinkingSphinx::Deltas::Job.cancel_thinking_sphinx_jobs
|
34
|
+
end
|
35
|
+
|
36
|
+
it "should delete rows if the delayed_jobs table does exist" do
|
37
|
+
ThinkingSphinx::Deltas::Job.connection.stub!(:tables => ['delayed_jobs'])
|
38
|
+
ThinkingSphinx::Deltas::Job.should_receive(:delete_all)
|
39
|
+
|
40
|
+
ThinkingSphinx::Deltas::Job.cancel_thinking_sphinx_jobs
|
41
|
+
end
|
42
|
+
|
43
|
+
it "should delete only Thinking Sphinx jobs" do
|
44
|
+
ThinkingSphinx::Deltas::Job.connection.stub!(:tables => ['delayed_jobs'])
|
45
|
+
ThinkingSphinx::Deltas::Job.should_receive(:delete_all) do |sql|
|
46
|
+
sql.should match(/handler LIKE '--- !ruby\/object:ThinkingSphinx::Deltas::\%'/)
|
47
|
+
end
|
48
|
+
|
49
|
+
ThinkingSphinx::Deltas::Job.cancel_thinking_sphinx_jobs
|
50
|
+
end
|
51
|
+
end
|
52
|
+
end
|
@@ -0,0 +1,126 @@
|
|
1
|
+
require 'spec/spec_helper'
|
2
|
+
|
3
|
+
describe ThinkingSphinx::Deltas::DelayedDelta do
|
4
|
+
describe '#index' do
|
5
|
+
before :each do
|
6
|
+
ThinkingSphinx.updates_enabled = true
|
7
|
+
ThinkingSphinx.deltas_enabled = true
|
8
|
+
ThinkingSphinx::Configuration.instance.delayed_job_priority = 2
|
9
|
+
|
10
|
+
ThinkingSphinx::Deltas::Job.stub!(:enqueue => true)
|
11
|
+
Delayed::Job.stub!(:enqueue => true)
|
12
|
+
|
13
|
+
@delayed_delta = ThinkingSphinx::Deltas::DelayedDelta.new(
|
14
|
+
stub('instance'), {}
|
15
|
+
)
|
16
|
+
@delayed_delta.stub!(:toggled => true)
|
17
|
+
|
18
|
+
@model = stub('foo')
|
19
|
+
@model.stub!(:name => 'foo')
|
20
|
+
@model.stub!(:source_of_sphinx_index => @model)
|
21
|
+
|
22
|
+
@instance = stub('instance')
|
23
|
+
@instance.stub!(:sphinx_document_id => 42)
|
24
|
+
end
|
25
|
+
|
26
|
+
context 'updates disabled' do
|
27
|
+
before :each do
|
28
|
+
ThinkingSphinx.updates_enabled = false
|
29
|
+
end
|
30
|
+
|
31
|
+
it "should not enqueue a delta job" do
|
32
|
+
ThinkingSphinx::Deltas::Job.should_not_receive(:enqueue)
|
33
|
+
|
34
|
+
@delayed_delta.index(@model)
|
35
|
+
end
|
36
|
+
|
37
|
+
it "should not enqueue a flag as deleted job" do
|
38
|
+
Delayed::Job.should_not_receive(:enqueue)
|
39
|
+
|
40
|
+
@delayed_delta.index(@model)
|
41
|
+
end
|
42
|
+
end
|
43
|
+
|
44
|
+
context 'deltas disabled' do
|
45
|
+
before :each do
|
46
|
+
ThinkingSphinx.deltas_enabled = false
|
47
|
+
end
|
48
|
+
|
49
|
+
it "should not enqueue a delta job" do
|
50
|
+
ThinkingSphinx::Deltas::Job.should_not_receive(:enqueue)
|
51
|
+
|
52
|
+
@delayed_delta.index(@model)
|
53
|
+
end
|
54
|
+
|
55
|
+
it "should not enqueue a flag as deleted job" do
|
56
|
+
Delayed::Job.should_not_receive(:enqueue)
|
57
|
+
|
58
|
+
@delayed_delta.index(@model)
|
59
|
+
end
|
60
|
+
end
|
61
|
+
|
62
|
+
context "instance isn't toggled" do
|
63
|
+
before :each do
|
64
|
+
@delayed_delta.stub!(:toggled => false)
|
65
|
+
end
|
66
|
+
|
67
|
+
it "should not enqueue a delta job" do
|
68
|
+
ThinkingSphinx::Deltas::Job.should_not_receive(:enqueue)
|
69
|
+
|
70
|
+
@delayed_delta.index(@model, @instance)
|
71
|
+
end
|
72
|
+
|
73
|
+
it "should not enqueue a flag as deleted job" do
|
74
|
+
Delayed::Job.should_not_receive(:enqueue)
|
75
|
+
|
76
|
+
@delayed_delta.index(@model, @instance)
|
77
|
+
end
|
78
|
+
end
|
79
|
+
|
80
|
+
it "should enqueue a delta job for the appropriate index" do
|
81
|
+
ThinkingSphinx::Deltas::Job.should_receive(:enqueue) do |job, priority|
|
82
|
+
job.index.should == 'foo_delta'
|
83
|
+
end
|
84
|
+
|
85
|
+
@delayed_delta.index(@model)
|
86
|
+
end
|
87
|
+
|
88
|
+
it "should use the defined priority for the delta job" do
|
89
|
+
ThinkingSphinx::Deltas::Job.should_receive(:enqueue) do |job, priority|
|
90
|
+
priority.should == 2
|
91
|
+
end
|
92
|
+
|
93
|
+
@delayed_delta.index(@model)
|
94
|
+
end
|
95
|
+
|
96
|
+
it "should enqueue a flag-as-deleted job for the appropriate index" do
|
97
|
+
Delayed::Job.should_receive(:enqueue) do |job, priority|
|
98
|
+
job.index.should == 'foo_core'
|
99
|
+
end
|
100
|
+
|
101
|
+
@delayed_delta.index(@model, @instance)
|
102
|
+
end
|
103
|
+
|
104
|
+
it "should enqueue a flag-as-deleted job for the appropriate id" do
|
105
|
+
Delayed::Job.should_receive(:enqueue) do |job, priority|
|
106
|
+
job.document_id.should == 42
|
107
|
+
end
|
108
|
+
|
109
|
+
@delayed_delta.index(@model, @instance)
|
110
|
+
end
|
111
|
+
|
112
|
+
it "should use the defined priority for the flag-as-deleted job" do
|
113
|
+
Delayed::Job.should_receive(:enqueue) do |job, priority|
|
114
|
+
priority.should == 2
|
115
|
+
end
|
116
|
+
|
117
|
+
@delayed_delta.index(@model, @instance)
|
118
|
+
end
|
119
|
+
|
120
|
+
it "should not enqueue a flag-as-deleted job if no instance is provided" do
|
121
|
+
Delayed::Job.should_not_receive(:enqueue)
|
122
|
+
|
123
|
+
@delayed_delta.index(@model)
|
124
|
+
end
|
125
|
+
end
|
126
|
+
end
|
metadata
ADDED
@@ -0,0 +1,115 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: ts-delayed-delta
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 1.0.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Pat Allan
|
8
|
+
autorequire:
|
9
|
+
bindir: bin
|
10
|
+
cert_chain: []
|
11
|
+
|
12
|
+
date: 2009-11-03 00:00:00 +11:00
|
13
|
+
default_executable:
|
14
|
+
dependencies:
|
15
|
+
- !ruby/object:Gem::Dependency
|
16
|
+
name: delayed_job
|
17
|
+
type: :runtime
|
18
|
+
version_requirement:
|
19
|
+
version_requirements: !ruby/object:Gem::Requirement
|
20
|
+
requirements:
|
21
|
+
- - ">="
|
22
|
+
- !ruby/object:Gem::Version
|
23
|
+
version: 1.8.4
|
24
|
+
version:
|
25
|
+
- !ruby/object:Gem::Dependency
|
26
|
+
name: rspec
|
27
|
+
type: :development
|
28
|
+
version_requirement:
|
29
|
+
version_requirements: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - ">="
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: 1.2.9
|
34
|
+
version:
|
35
|
+
- !ruby/object:Gem::Dependency
|
36
|
+
name: yard
|
37
|
+
type: :development
|
38
|
+
version_requirement:
|
39
|
+
version_requirements: !ruby/object:Gem::Requirement
|
40
|
+
requirements:
|
41
|
+
- - ">="
|
42
|
+
- !ruby/object:Gem::Version
|
43
|
+
version: "0"
|
44
|
+
version:
|
45
|
+
- !ruby/object:Gem::Dependency
|
46
|
+
name: cucumber
|
47
|
+
type: :development
|
48
|
+
version_requirement:
|
49
|
+
version_requirements: !ruby/object:Gem::Requirement
|
50
|
+
requirements:
|
51
|
+
- - ">="
|
52
|
+
- !ruby/object:Gem::Version
|
53
|
+
version: "0"
|
54
|
+
version:
|
55
|
+
description: Manage delta indexes via Delayed Job for Thinking Sphinx
|
56
|
+
email: pat@freelancing-gods.com
|
57
|
+
executables: []
|
58
|
+
|
59
|
+
extensions: []
|
60
|
+
|
61
|
+
extra_rdoc_files:
|
62
|
+
- LICENSE
|
63
|
+
- README.textile
|
64
|
+
files:
|
65
|
+
- LICENSE
|
66
|
+
- README.textile
|
67
|
+
- lib/thinking_sphinx/deltas/delayed_delta.rb
|
68
|
+
- lib/thinking_sphinx/deltas/delayed_delta/delta_job.rb
|
69
|
+
- lib/thinking_sphinx/deltas/delayed_delta/flag_as_deleted_job.rb
|
70
|
+
- lib/thinking_sphinx/deltas/delayed_delta/job.rb
|
71
|
+
- lib/thinking_sphinx/deltas/delayed_delta/tasks.rb
|
72
|
+
has_rdoc: true
|
73
|
+
homepage: http://github.com/freelancing-god/ts-delayed-delta
|
74
|
+
licenses: []
|
75
|
+
|
76
|
+
post_install_message:
|
77
|
+
rdoc_options:
|
78
|
+
- --charset=UTF-8
|
79
|
+
require_paths:
|
80
|
+
- lib
|
81
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
82
|
+
requirements:
|
83
|
+
- - ">="
|
84
|
+
- !ruby/object:Gem::Version
|
85
|
+
version: "0"
|
86
|
+
version:
|
87
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
88
|
+
requirements:
|
89
|
+
- - ">="
|
90
|
+
- !ruby/object:Gem::Version
|
91
|
+
version: "0"
|
92
|
+
version:
|
93
|
+
requirements: []
|
94
|
+
|
95
|
+
rubyforge_project:
|
96
|
+
rubygems_version: 1.3.5
|
97
|
+
signing_key:
|
98
|
+
specification_version: 3
|
99
|
+
summary: Thinking Sphinx - Delayed Deltas
|
100
|
+
test_files:
|
101
|
+
- features/delayed_deltas.feature
|
102
|
+
- features/step_definitions/common_steps.rb
|
103
|
+
- features/step_definitions/delayed_delta_steps.rb
|
104
|
+
- features/support/database.example.yml
|
105
|
+
- features/support/database.yml
|
106
|
+
- features/support/db/fixtures/delayed_betas.rb
|
107
|
+
- features/support/db/migrations/create_delayed_betas.rb
|
108
|
+
- features/support/env.rb
|
109
|
+
- features/support/models/delayed_beta.rb
|
110
|
+
- spec/spec.opts
|
111
|
+
- spec/spec_helper.rb
|
112
|
+
- spec/thinking_sphinx/deltas/delayed_delta/delta_job_spec.rb
|
113
|
+
- spec/thinking_sphinx/deltas/delayed_delta/flag_as_deleted_job_spec.rb
|
114
|
+
- spec/thinking_sphinx/deltas/delayed_delta/job_spec.rb
|
115
|
+
- spec/thinking_sphinx/deltas/delayed_delta_spec.rb
|