ts-resque-delta 1.2.4 → 2.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 2584321ee1b15d855877ef9fb3dc0b4d0719c21e
+  data.tar.gz: fc74627b7a3265a8cafe125ab59585c8de2e2a0c
+SHA512:
+  metadata.gz: 7380a6e12204a7e6ea243c6c2f6726f637e7125b37b1d09fb88ab93169e78820127b1d28dc07bcff23e0b4199f117a1ef0a147664c2fc64188c026884d1c0dff
+  data.tar.gz: a00d91e7bddb444c493fc4fb7ae74167b0b7af598e9d78f5a8790aafd53cfcc730439b81e99520658c1f657eccd015acce30bb6e0ce7b931b793f529f170ef33

data/.gitignore

@@ -1,4 +1,5 @@
 *.gem
+*.lock
 .bundle
 pkg/*
 *~
@@ -16,4 +17,3 @@ tmp
 /bin
 .rvmrc
 .rbenv-version
-Gemfile.lock

data/.travis.yml

@@ -1,18 +1,18 @@
 language: ruby
-before_install: gem update --system
-before_script:
-  - mysql -e 'create database thinking_sphinx;'
 rvm:
-  - 1.8.7
-  - ree
-  - 1.9.2
   - 1.9.3
+before_install:
+  - gem update --system
+  - gem install bundler --no-ri --no-rdoc
+  - bundle install
+before_script:
+  - "mysql -e 'create database thinking_sphinx;' > /dev/null"
+  - "psql -c 'create database thinking_sphinx;' -U postgres > /dev/null"
+services:
+  - redis-server
+env:
+  - SPHINX_BIN=/usr/local/sphinx-2.0.9/bin/ SPHINX_VERSION=2.0.9 DATABASE=mysql2
+  - SPHINX_BIN=/usr/local/sphinx-2.0.9/bin/ SPHINX_VERSION=2.0.9 DATABASE=postgresql
 gemfile:
-  - gemfiles/activerecord2.gemfile
-  - gemfiles/activerecord3.gemfile
-matrix:
-  exclude:
-    - rvm: 1.9.2
-      gemfile: gemfiles/activerecord2.gemfile
-    - rvm: 1.9.3
-      gemfile: gemfiles/activerecord2.gemfile
+  - gemfiles/binary.gemfile
+  - gemfiles/sphinxql.gemfile

data/Appraisals

@@ -1,9 +1,7 @@
-appraise "activerecord2" do
-  gem "activerecord", "~> 2.3.14"
-  gem "mysql2", "~> 0.2.7"
+appraise 'binary' do
+  gem 'thinking-sphinx', '~> 2.1.0'
 end
 
-appraise "activerecord3" do
-  gem "activerecord", "~> 3.2.9"
-  gem "mysql2", "~> 0.3.11"
+appraise 'sphinxql' do
+  gem 'thinking-sphinx', '~> 3.0.3'
 end

data/Gemfile

@@ -1,4 +1,4 @@
-source "http://rubygems.org"
+source "https://rubygems.org"
 
 # Specify your gem's dependencies in ts-resque-delta.gemspec
 gemspec

data/README.markdown

@@ -7,67 +7,56 @@ Delayed Deltas for Thinking Sphinx (with Resque)
 
 Installation
 ------------
-This gem depends on the following gems: _thinking-sphinx_, _resque_, and
-_resque-lock-timeout_.
 
-    gem install ts-resque-delta
+This gem depends on the following gems: `thinking-sphinx` and `resque`.
 
-Add _ts-resque-delta_ to your **Gemfile** file, with the rest of your gem
-dependencies:
+Currently, you'll need Thinking Sphinx v1.5.0 (for Rails 2), v2.1.0
+(for Rails 3), or - ideally - v3.0.3 or newer (for Rails 3.1 onwards). If you're
+on a version of Thinking Sphinx that's too old, you better go upgrade - but
+otherwise, add `ts-resque-delta` to your `Gemfile` file with the rest of your
+gem dependencies:
 
-    gem 'ts-resque-delta', '1.1.1'
+    gem 'ts-resque-delta', '~> 2.0.0'
 
-If you're using Rails 3, the rake tasks will automatically be loaded by Rails.
-If you're using Rails 2, add the following line to your **Rakefile**:
+Add the delta property to index definition. If you're using Thinking Sphinx v3,
+then it'll look something like this:
 
-    require 'thinking_sphinx/deltas/resque_delta/tasks'
+    ThinkingSphinx::Index.define(:article,
+      :with  => :active_record,
+      :delta => ThinkingSphinx::Deltas::ResqueDelta
+    ) do
+      # fields and attributes and so on...
+    end
 
-Add the delta property to each `define_index` block:
+But if you're still using v1.5 or v2.1, you'll want the following:
 
     define_index do
-      # ...
-      set_property :delta => ThinkingSphinx::Deltas::ResqueDelta
-    end
+      # fields and attributes and so on...
 
-If you've never used delta indexes before, you'll want to add the boolean
-column named `:delta` to each model's table (note, you must set the `:default`
-value to `true`):
-
-    def self.up
-      add_column :foos, :delta, :boolean, :default => true, :null => false
+      set_property :delta => ThinkingSphinx::Deltas::ResqueDelta
     end
 
-Also, I highly recommend adding a MySQL index to the table of any model using
-delta indexes. The Sphinx indexer uses `WHERE table.delta = 1` whenever the
-delta indexer runs and `... = 0` whenever the normal indexer runs. Having the
-MySQL index on the delta column will generally be a win:
+If you've never used delta indexes before, you'll need to add the boolean
+column named `:delta` to each table for indexed models. A database index for
+that column is also recommended.
 
-    def self.up
-      # ...
-      add_index :foos, :delta
+    def change
+      add_column :articles, :delta, :boolean, :default => true, :null => false
+      add_index  :articles, :delta
     end
 
 Usage
 -----
+
 Once you've got it all set up, all you need to do is make sure that the Resque
-worker is running. You can do this by specifying the `:ts_delta` queue when
+worker is running. You can do this by specifying the `ts_delta` queue when
 running Resque:
 
     QUEUE=ts_delta,other_queues rake resque:work
 
-Additionally, ts-resque-delta will wrap thinking-sphinx's
-`thinking_sphinx:index` and `thinking_sphinx:reindex` tasks with
-`thinking_sphinx:lock_deltas` and `thinking_sphinx:unlock_deltas`. This will
-prevent the delta indexer from running at the same time as the main indexer.
-
-Finally, ts-resque-delta also provides a rake task called
-`thinking_sphinx:smart_index` (or `ts:si` for short). This task, instead of
-locking all the delta indexes at once while the main indexer runs, will lock
-each delta index independently and sequentially. Thay way, your delta indexer
-can run while the main indexer is processing large core indexes.
-
 Contributors (for ts-resque-delta)
 -----------------------------------
+
 * [Aaron Gibralter](https://github.com/agibralter)
 * [Ryan Schlesinger](https://github.com/ryansch) (Locking/`smart_index`)
 * [Pat Allan](https://github.com/freelancing-god) (FlyingSphinx support)
@@ -75,6 +64,7 @@ Contributors (for ts-resque-delta)
 
 Original Contributors (for ts-delayed-delta)
 --------------------------------------------
+
 * [Pat Allan](https://github.com/freelancing-god)
 * [Ryan Schlesinger](https://github.com/ryansch) (Allowing installs as a plugin)
 * [Maximilian Schulz](https://max.jungeelite.de) (Ensuring compatibility with Bundler)
@@ -83,4 +73,6 @@ Original Contributors (for ts-delayed-delta)
 
 Copyright
 ---------
-Copyright (c) 2011 Aaron Gibralter, and released under an MIT Licence.
+
+Copyright (c) 2011-2014 Aaron Gibralter and Pat Allan, and released under an MIT
+Licence.

data/Rakefile

@@ -1,17 +1,10 @@
-#!/usr/bin/env rake
-require "bundler/gem_tasks"
-require 'appraisal'
-require 'rspec/core/rake_task'
-require 'cucumber'
-require 'cucumber/rake/task'
+require 'bundler/setup'
 
-RSpec::Core::RakeTask.new(:spec) do |t|
-  t.rspec_opts = ["-c", "--format progress"]
-end
+Bundler::GemHelper.install_tasks
 
-Cucumber::Rake::Task.new(:features) do |t|
-end
+require 'appraisal'
+require 'rspec/core/rake_task'
 
-task :all_tests => [:spec, :features]
+RSpec::Core::RakeTask.new
 
-task :default => :all_tests
+task :default => :spec

data/gemfiles/binary.gemfile

@@ -0,0 +1,7 @@
+# This file was generated by Appraisal
+
+source "https://rubygems.org"
+
+gem "thinking-sphinx", "~> 2.1.0"
+
+gemspec :path=>"../"

data/gemfiles/sphinxql.gemfile

@@ -0,0 +1,7 @@
+# This file was generated by Appraisal
+
+source "https://rubygems.org"
+
+gem "thinking-sphinx", "~> 3.0.3"
+
+gemspec :path=>"../"

data/lib/thinking_sphinx/deltas/resque_delta.rb

@@ -1,9 +1,6 @@
 require 'resque'
 require 'thinking_sphinx'
 
-require 'thinking_sphinx/deltas/resque_delta/flag_as_deleted_set'
-require 'thinking_sphinx/deltas/resque_delta/index_utils'
-
 # Delayed Deltas for Thinking Sphinx, using Resque.
 #
 # This documentation is aimed at those reading the code. If you're looking for
@@ -15,104 +12,97 @@ require 'thinking_sphinx/deltas/resque_delta/index_utils'
 # @see http://ts.freelancing-gods.com Thinking Sphinx
 #
 class ThinkingSphinx::Deltas::ResqueDelta < ThinkingSphinx::Deltas::DefaultDelta
+  JOB_PREFIX = 'ts-delta'
+
   def self.job_types
     [
-      ThinkingSphinx::Deltas::ResqueDelta::DeltaJob
+      ThinkingSphinx::Deltas::ResqueDelta::DeltaJob,
+      ThinkingSphinx::Deltas::ResqueDelta::FlagAsDeletedJob
     ]
   end
-  
-  def self.job_prefix
-    'ts-delta'
-  end
-  
+
   # LTRIM + LPOP deletes all items from the Resque queue without loading it
   # into client memory (unlike Resque.dequeue).
   # WARNING: This will clear ALL jobs in any queue used by a ResqueDelta job.
   # If you're sharing a queue with other jobs they'll be deleted!
-  def self.clear_thinking_sphinx_queues
-    job_types.collect { |c| c.instance_variable_get(:@queue) }.uniq.each do |q|
-      Resque.redis.ltrim("queue:#{q}", 0, 0)
-      Resque.redis.lpop("queue:#{q}")
+  def self.cancel_jobs
+    job_types.collect { |klass|
+      klass.instance_variable_get :@queue
+    }.uniq.each do |queue|
+      Resque.redis.ltrim "queue:#{queue}", 0, 0
+      Resque.redis.lpop  "queue:#{queue}"
     end
   end
 
-  # Clear both the resque queues and any other state maintained in redis
-  def self.clear!
-    self.clear_thinking_sphinx_queues
-
-    FlagAsDeletedSet.clear_all!
-  end
-
-  # Use simplistic locking.  We're assuming that the user won't run more than one
-  # `rake ts:si` or `rake ts:in` task at a time.
-  def self.lock(index_name)
-    Resque.redis.set("#{job_prefix}:index:#{index_name}:locked", 'true')
-  end
+  module Binary
+    # Adds a job to the queue for processing the given model's delta index. A
+    # job for hiding the instance in the core index is also created, if an
+    # instance is provided.
+    #
+    # Neither job will be queued if updates or deltas are disabled, or if the
+    # instance (when given) is not toggled to be in the delta index. The first
+    # two options are controlled via ThinkingSphinx.updates_enabled? and
+    # ThinkingSphinx.deltas_enabled?.
+    #
+    # @param [Class] model the ActiveRecord model to index.
+    # @param [ActiveRecord::Base] instance the instance of the given model that
+    #   has changed. Optional.
+    # @return [Boolean] true
+    #
+    def index(model, instance = nil)
+      return true if skip? instance
 
-  def self.unlock(index_name)
-    Resque.redis.del("#{job_prefix}:index:#{index_name}:locked")
-  end
+      Resque.enqueue ThinkingSphinx::Deltas::ResqueDelta::DeltaJob,
+        model.delta_index_names
 
-  def self.locked?(index_name)
-    Resque.redis.get("#{job_prefix}:index:#{index_name}:locked") == 'true'
-  end
+      Resque.enqueue(
+        ThinkingSphinx::Deltas::ResqueDelta::FlagAsDeletedJob,
+        model.core_index_names, instance.sphinx_document_id
+      ) if instance
 
-  def self.prepare_for_core_index(index_name)
-    core = "#{index_name}_core"
-    delta = "#{index_name}_delta"
+      true
+    end
 
-    FlagAsDeletedSet.clear!(core)
+    private
 
-    #clear delta jobs
-    # dequeue is fast for jobs with arguments
-    Resque.dequeue(ThinkingSphinx::Deltas::ResqueDelta::DeltaJob, delta)
+    # Checks whether jobs should be enqueued. Only true if updates and deltas are
+    # enabled, and the instance (if there is one) is toggled.
+    #
+    # @param [ActiveRecord::Base, NilClass] instance
+    # @return [Boolean]
+    #
+    def skip?(instance)
+      !ThinkingSphinx.updates_enabled? ||
+      !ThinkingSphinx.deltas_enabled?  ||
+      (instance && !toggled(instance))
+    end
   end
 
-  # Adds a job to the queue for processing the given model's delta index. A job
-  # for hiding the instance in the core index is also created, if an instance is
-  # provided.
-  #
-  # Neither job will be queued if updates or deltas are disabled, or if the 
-  # instance (when given) is not toggled to be in the delta index. The first two
-  # options are controlled via ThinkingSphinx.updates_enabled? and
-  # ThinkingSphinx.deltas_enabled?.
-  #
-  # @param [Class] model the ActiveRecord model to index.
-  # @param [ActiveRecord::Base] instance the instance of the given model that
-  #   has changed. Optional.
-  # @return [Boolean] true
-  #
-  def index(model, instance = nil)
-    return true if skip?(instance)
-    model.delta_index_names.each do |delta|
-      next if self.class.locked?(delta)
+  module SphinxQL
+    def delete(index, instance)
       Resque.enqueue(
-        ThinkingSphinx::Deltas::ResqueDelta::DeltaJob,
-        delta
+        ThinkingSphinx::Deltas::ResqueDelta::FlagAsDeletedJob,
+        index.name, index.document_id_for_key(instance.id)
       )
     end
-    if instance
-      model.core_index_names.each do |core|
-        FlagAsDeletedSet.add(core, instance.sphinx_document_id)
-      end
+
+    def index(index)
+      Resque.enqueue ThinkingSphinx::Deltas::ResqueDelta::DeltaJob, index.name
     end
-    true
   end
 
-  private
-
-  # Checks whether jobs should be enqueued. Only true if updates and deltas are
-  # enabled, and the instance (if there is one) is toggled.
-  #
-  # @param [ActiveRecord::Base, NilClass] instance
-  # @return [Boolean]
-  #
-  def skip?(instance)
-    !ThinkingSphinx.updates_enabled? ||
-    !ThinkingSphinx.deltas_enabled?  ||
-    (instance && !toggled(instance))
+  if [:delayed_job_priority, 'delayed_job_priority'].any? { |method|
+    ThinkingSphinx::Configuration.instance_methods.include?(method)
+  }
+    include Binary
+  else
+    include SphinxQL
   end
 end
 
+require 'thinking_sphinx/deltas/resque_delta/flag_as_deleted_job'
 require 'thinking_sphinx/deltas/resque_delta/delta_job'
-require 'thinking_sphinx/deltas/resque_delta/core_index'
+
+ThinkingSphinx.before_index_hooks << Proc.new {
+  ThinkingSphinx::Deltas::ResqueDelta.cancel_jobs
+}

data/lib/thinking_sphinx/deltas/resque_delta/delta_job.rb

@@ -1,12 +1,7 @@
-require 'resque-lock-timeout'
-
 # A simple job class that processes a given index.
 #
 class ThinkingSphinx::Deltas::ResqueDelta::DeltaJob
-
-  extend Resque::Plugins::LockTimeout
   @queue = :ts_delta
-  @lock_timeout = 240
 
   # Runs Sphinx's indexer tool to process the index. Currently assumes Sphinx
   # is running.
@@ -14,113 +9,6 @@ class ThinkingSphinx::Deltas::ResqueDelta::DeltaJob
   # @param [String] index the name of the Sphinx index
   #
   def self.perform(index)
-    return if skip?(index)
-
-    config = ThinkingSphinx::Configuration.instance
-
-    # Delta Index
-    output = `#{config.bin_path}#{config.indexer_binary_name} --config #{config.config_file} --rotate #{index}`
-    puts output unless ThinkingSphinx.suppress_delta_output?
-
-    # Flag As Deleted
-    return unless ThinkingSphinx.sphinx_running?
-
-    index = ThinkingSphinx::Deltas::ResqueDelta::IndexUtils.delta_to_core(index)
-
-    # Get the document ids we've saved
-    flag_as_deleted_ids = ThinkingSphinx::Deltas::ResqueDelta::FlagAsDeletedSet.processing_members(index)
-
-    unless flag_as_deleted_ids.empty?
-      # Filter out the ids that aren't present in sphinx
-      flag_as_deleted_ids = filter_flag_as_deleted_ids(flag_as_deleted_ids, index)
-
-      unless flag_as_deleted_ids.empty?
-        # Each hash element should be of the form { id => [1] }
-        flag_hash = Hash[*flag_as_deleted_ids.collect {|id| [id, [1]] }.flatten(1)]
-
-        config.client.update(index, ['sphinx_deleted'], flag_hash)
-      end
-    end
-  end
-
-  # Try again later if lock is in use.
-  def self.lock_failed(*args)
-    Resque.enqueue(self, *args)
-  end
-
-  # Run only one DeltaJob at a time regardless of index.
-  #def self.identifier(*args)
-    #nil
-  #end
-
-  # This allows us to have a concurrency safe version of ts-delayed-delta's
-  # duplicates_exist:
-  #
-  # http://github.com/freelancing-god/ts-delayed-delta/blob/master/lib/thinkin
-  # g_sphinx/deltas/delayed_delta/job.rb#L47
-  #
-  # The name of this method ensures that it runs within around_perform_lock.
-  #
-  # We've leveraged resque-lock-timeout to ensure that only one DeltaJob is
-  # running at a time. Now, this around filter essentially ensures that only
-  # one DeltaJob of each index type can sit at the queue at once. If the queue
-  # has more than one, lrem will clear the rest off.
-  #
-  def self.around_perform_lock1(*args)
-    # Remove all other instances of this job (with the same args) from the
-    # queue. Uses LREM (http://code.google.com/p/redis/wiki/LremCommand) which
-    # takes the form: "LREM key count value" and if count == 0 removes all
-    # instances of value from the list.
-    redis_job_value = Resque.encode(:class => self.to_s, :args => args)
-    Resque.redis.lrem("queue:#{@queue}", 0, redis_job_value)
-
-    # Grab the subset of flag as deleted document ids to work on
-    core_index = ThinkingSphinx::Deltas::ResqueDelta::IndexUtils.delta_to_core(*args)
-    ThinkingSphinx::Deltas::ResqueDelta::FlagAsDeletedSet.get_subset_for_processing(core_index)
-
-    yield
-
-    # Clear processing set
-    ThinkingSphinx::Deltas::ResqueDelta::FlagAsDeletedSet.clear_processing(core_index)
-  end
-
-  protected
-
-  def self.skip?(index)
-    ThinkingSphinx::Deltas::ResqueDelta.locked?(index)
-  end
-
-  def self.filter_flag_as_deleted_ids(ids, index)
-    search_results = []
-    partition_ids(ids, 4096) do |subset|
-      search_results += ThinkingSphinx.search_for_ids(
-        :with => {:@id => subset}, :index => index
-      ).results[:matches].collect { |match| match[:doc] }
-    end
-
-    search_results
-  end
-
-  def self.partition_ids(ids, n)
-    if n > 0 && n < ids.size
-      result = []
-      max_subarray_size = n - 1
-      i = j = 0
-      while i < ids.size && j < ids.size
-        j = i + max_subarray_size
-        result << ids.slice(i..j)
-        i += n
-      end
-    else
-      result = ids
-    end
-
-    if block_given?
-      result.each do |ary|
-        yield ary
-      end
-    end
-
-    result
+    ThinkingSphinx::Deltas::IndexJob.new(index).perform
   end
 end