searchkick 2.0.4 → 2.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 275674a7d9ef11e2faba8426a3a85172d84cf16c
-  data.tar.gz: 3134c16ed33df803187e4b8b75dd3f8167745e62
+  metadata.gz: 95d822316595ca2806bc9f2e3e44644ee2ab5fd8
+  data.tar.gz: 27f86dc98a28378ee489b5adf7bfd11be054baf1
 SHA512:
-  metadata.gz: bf4ab3f4fb5557051f9e5bbc25fd9aaf26f3c5403e473e86f6108f5418a53cc3ee391ec0cf204cc717ca0df7f96ba5e83deeb42bb725bc6025f6258c067d9c7d
-  data.tar.gz: dba920c1264cf007eb5a4759c83f3bf0269a8b36bf381b4e4f6c8b23089abfb1a90c373a3ae6853f4145bae751b1181f7bcd77fb06cd839cd45c8e350c0f22ea
+  metadata.gz: fb18990e391306eb6d3f9a6193b8c14acf8cb22691f0a470d7c47b40c6146a58bc7babc6250378dfadf86b85f5f6846bd79f5302969b54de632df77aa7a55809
+  data.tar.gz: 0983c46e8eda2d402cc632a9849e97fca61f3036cfa96378ff81921dc6901416671b6192a184a23dc04615534422f79a7f00c2694c3b9cb46d7b38f7a2410345

data/CHANGELOG.md

@@ -1,3 +1,8 @@
+## 2.1.0
+
+- Background reindexing and queues are officially supported
+- Log updates and deletes
+
 ## 2.0.4
 
 - Added support for queuing updates [experimental]

data/README.md

@@ -29,7 +29,18 @@ Plus:
 
 **Searchkick 2.0 was just released!** See [notable changes](#200).
 
-## Get Started
+## Contents
+
+- [Getting Started](#getting-started)
+- [Querying](#querying)
+- [Indexing](#indexing)
+- [Aggregations](#aggregations)
+- [Deployment](#deployment)
+- [Performance](#performance)
+- [Elasticsearch DSL](#advanced)
+- [Reference](#reference)
+
+## Getting Started
 
 [Install Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html). For Homebrew, use:
 
@@ -73,7 +84,7 @@ end
 
 Searchkick supports the complete [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html). As your search becomes more advanced, we recommend you use the [Elasticsearch DSL](#advanced) for maximum flexibility.
 
-### Queries
+## Querying
 
 Query like SQL
 
@@ -381,7 +392,7 @@ And use:
 Product.search "🍨🍰", emoji: true
 ```
 
-### Indexing
+## Indexing
 
 Control what data is indexed with the `search_data` method. Call `Product.reindex` after changing this method.
 
@@ -425,6 +436,8 @@ If a reindex is interrupted, you can resume it with:
 Product.reindex(resume: true)
 ```
 
+For large data sets, try [parallel reindexing](#parallel-reindexing).
+
 ### To Reindex, or Not to Reindex
 
 #### Reindex
@@ -439,7 +452,7 @@ Product.reindex(resume: true)
 
 ### Stay Synced
 
-There are three strategies for keeping the index synced with your database.
+There are four strategies for keeping the index synced with your database.
 
 1. Immediate (default)
 
@@ -457,7 +470,11 @@ There are three strategies for keeping the index synced with your database.
 
   And [install Active Job](https://github.com/ankane/activejob_backport) for Rails 4.1 and below. Jobs are added to a queue named `searchkick`.
 
-3. Manual
+3. Queuing
+
+  Push ids of records that need updated to a queue and reindex in the background in batches. This is more performant than the asynchronous method, which updates records individually. See [how to set up](#queuing).
+
+4. Manual
 
   Turn off automatic syncing
 
@@ -543,6 +560,8 @@ Reindex and set up a cron job to add new conversions daily.
 rake searchkick:reindex CLASS=Product
 ```
 
+**Note:** For a more performant (but more advanced) approach, check out [performant conversions](#performant-conversions).
+
 ### Personalized Results
 
 Order results differently for each user. For example, show a user’s previously purchased products before other results.
@@ -962,23 +981,23 @@ Product.search("soap", explain: true).response
 See how Elasticsearch tokenizes your queries with:
 
 ```ruby
-Product.searchkick_index.tokens("Dish Washer Soap", analyzer: "searchkick_index")
+Product.search_index.tokens("Dish Washer Soap", analyzer: "searchkick_index")
 # ["dish", "dishwash", "washer", "washersoap", "soap"]
 
-Product.searchkick_index.tokens("dishwasher soap", analyzer: "searchkick_search")
+Product.search_index.tokens("dishwasher soap", analyzer: "searchkick_search")
 # ["dishwashersoap"] - no match
 
-Product.searchkick_index.tokens("dishwasher soap", analyzer: "searchkick_search2")
+Product.search_index.tokens("dishwasher soap", analyzer: "searchkick_search2")
 # ["dishwash", "soap"] - match!!
 ```
 
 Partial matches
 
 ```ruby
-Product.searchkick_index.tokens("San Diego", analyzer: "searchkick_word_start_index")
+Product.search_index.tokens("San Diego", analyzer: "searchkick_word_start_index")
 # ["s", "sa", "san", "d", "di", "die", "dieg", "diego"]
 
-Product.searchkick_index.tokens("dieg", analyzer: "searchkick_word_search")
+Product.search_index.tokens("dieg", analyzer: "searchkick_word_search")
 # ["dieg"] - match!!
 ```
 
@@ -1136,29 +1155,7 @@ class Product < ActiveRecord::Base
 end
 ```
 
-### Routing
-
-Searchkick supports [Elasticsearch’s routing feature](https://www.elastic.co/blog/customizing-your-document-routing), which can significantly speed up searches.
-
-```ruby
-class Business < ActiveRecord::Base
-  searchkick routing: true
-
-  def search_routing
-    city_id
-  end
-end
-```
-
-Reindex and search with:
-
-```ruby
-Business.search "ice cream", routing: params[:city_id]
-```
-
-## Large Data Sets
-
-### Background Reindexing [experimental, ActiveRecord only]
+### Parallel Reindexing
 
 For large data sets, you can use background jobs to parallelize reindexing.
 
@@ -1170,7 +1167,7 @@ Product.reindex(async: true)
 Once the jobs complete, promote the new index with:
 
 ```ruby
-Product.searchkick_index.promote(index_name)
+Product.search_index.promote(index_name)
 ```
 
 You can optionally track the status with Redis:
@@ -1185,9 +1182,9 @@ And use:
 Searchkick.reindex_status(index_name)
 ```
 
-### Queues [master, experimental]
+### Queuing
 
-You can also queue updates and do them in bulk for better performance. First, set up Redis in an initializer.
+Push ids of records needing reindexed to a queue and reindex in bulk for better performance. First, set up Redis in an initializer.
 
 ```ruby
 Searchkick.redis = Redis.new
@@ -1210,14 +1207,121 @@ Searchkick::ProcessQueueJob.perform_later(class_name: "Product")
 You can check the queue length with:
 
 ```ruby
-Product.searchkick_index.reindex_queue.length
+Product.search_index.reindex_queue.length
 ```
 
 For more tips, check out [Keeping Elasticsearch in Sync](https://www.elastic.co/blog/found-keeping-elasticsearch-in-sync).
 
+### Routing
+
+Searchkick supports [Elasticsearch’s routing feature](https://www.elastic.co/blog/customizing-your-document-routing), which can significantly speed up searches.
+
+```ruby
+class Business < ActiveRecord::Base
+  searchkick routing: true
+
+  def search_routing
+    city_id
+  end
+end
+```
+
+Reindex and search with:
+
+```ruby
+Business.search "ice cream", routing: params[:city_id]
+```
+
+### Partial Reindexing
+
+Reindex a subset of attributes to reduce time spent generating search data and cut down on network traffic.
+
+```ruby
+class Product < ActiveRecord::Base
+  def search_data
+    {
+      name: name
+    }.merge(search_prices)
+  end
+
+  def search_prices
+    {
+      price: price,
+      sale_price: sale_price
+    }
+  end
+end
+```
+
+And use:
+
+```ruby
+Product.reindex(:search_prices)
+```
+
+### Performant Conversions
+
+Split out conversions into a separate method so you can use partial reindexing, and cache conversions to prevent N+1 queries. Be sure to use a centralized cache store like Memcached or Redis.
+
+```ruby
+class Product < ActiveRecord::Base
+  def search_data
+    {
+      name: name
+    }.merge(search_conversions)
+  end
+
+  def search_conversions
+    {
+      conversions: Rails.cache.read("search_conversions:#{self.class.name}:#{id}") || {}
+    }
+  end
+end
+```
+
+Create a job to update the cache and reindex records with new conversions.
+
+```ruby
+class ReindexConversionsJob < ActiveJob::Base
+  def perform(class_name)
+    # get records that have a recent conversion
+    recently_converted_ids =
+      Searchjoy::Search.where("convertable_type = ? AND converted_at > ?", class_name, 1.day.ago)
+      .order(:convertable_id).uniq.pluck(:convertable_id)
+
+    # split into groups
+    recently_converted_ids.in_groups_of(1000, false) do |ids|
+      # fetch conversions and group by record
+      conversions_by_record = {}
+      conversions =
+        Searchjoy::Search.where(convertable_id: ids, convertable_type: class_name)
+        .group(:convertable_id, :query).uniq.count(:user_id)
+
+      conversions.each do |(id, query), count|
+        (conversions_by_record[id] ||= {})[query] = count
+      end
+
+      # write to cache
+      conversions_by_record.each do |id, conversions|
+        Rails.cache.write("search_conversions:#{class_name}:#{id}", conversions)
+      end
+
+      # partial reindex
+      class_name.constantize.where(id: ids).reindex(:search_conversions)
+    end
+  end
+end
+```
+
+Run the job with:
+
+```ruby
+ReindexConversionsJob.perform_later("Product")
+```
+
 ## Advanced
 
-Prefer to use the [Elasticsearch DSL](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-queries.html) but still want awesome features like zero-downtime reindexing?
+Searchkick makes it easy to use the Elasticsearch DSL on its own.
 
 ### Advanced Mapping
 
@@ -1334,25 +1438,10 @@ Reindex associations
 store.products.reindex
 ```
 
-Reindex a subset of attributes (partial reindex)
-
-```ruby
-class Product < ActiveRecord::Base
-  def search_prices
-    {
-      price: price,
-      sale_price: sale_price
-    }
-  end
-end
-
-Product.reindex(:search_prices)
-```
-
 Remove old indices
 
 ```ruby
-Product.searchkick_index.clean_indices
+Product.search_index.clean_indices
 ```
 
 Use custom settings

data/lib/searchkick/bulk_reindex_job.rb

@@ -6,7 +6,12 @@ module Searchkick
       klass = class_name.constantize
       index = index_name ? Searchkick::Index.new(index_name) : klass.searchkick_index
       record_ids ||= min_id..max_id
-      index.import_scope(Searchkick.load_records(klass, record_ids), method_name: method_name, batch: true, batch_id: batch_id)
+      index.import_scope(
+        Searchkick.load_records(klass, record_ids),
+        method_name: method_name,
+        batch: true,
+        batch_id: batch_id
+      )
     end
   end
 end

data/lib/searchkick/index.rb

@@ -248,37 +248,14 @@ module Searchkick
     end
 
     def import_scope(scope, resume: false, method_name: nil, async: false, batch: false, batch_id: nil, full: false)
-      batch_size = @options[:batch_size] || 1000
-
       # use scope for import
       scope = scope.search_import if scope.respond_to?(:search_import)
 
       if batch
         import_or_update scope.to_a, method_name, async
-        Searchkick.redis.srem(batches_key, batch_id) if batch_id && Searchkick.redis
+        redis.srem(batches_key, batch_id) if batch_id && redis
       elsif full && async
-        if scope.respond_to?(:primary_key)
-          # TODO expire Redis key
-          primary_key = scope.primary_key
-          starting_id = scope.minimum(primary_key) || 0
-          max_id = scope.maximum(primary_key) || 0
-          batches_count = ((max_id - starting_id + 1) / batch_size.to_f).ceil
-
-          batches_count.times do |i|
-            batch_id = i + 1
-            min_id = starting_id + (i * batch_size)
-            Searchkick::BulkReindexJob.perform_later(
-              class_name: scope.model_name.name,
-              min_id: min_id,
-              max_id: min_id + batch_size - 1,
-              index_name: name,
-              batch_id: batch_id
-            )
-            Searchkick.redis.sadd(batches_key, batch_id) if Searchkick.redis
-          end
-        else
-          raise Searchkick::Error, "async option only supported for ActiveRecord"
-        end
+        full_reindex_async(scope)
       elsif scope.respond_to?(:find_in_batches)
         if resume
           # use total docs instead of max id since there's not a great way
@@ -294,23 +271,14 @@ module Searchkick
           import_or_update batch, method_name, async
         end
       else
-        # https://github.com/karmi/tire/blob/master/lib/tire/model/import.rb
-        # use cursor for Mongoid
-        items = []
-        # TODO add resume
-        scope.all.each do |item|
-          items << item
-          if items.length == batch_size
-            import_or_update items, method_name, async
-            items = []
-          end
+        each_batch(scope) do |items|
+          import_or_update items, method_name, async
         end
-        import_or_update items, method_name, async
       end
     end
 
     def batches_left
-      Searchkick.redis.scard(batches_key) if Searchkick.redis
+      redis.scard(batches_key) if redis
     end
 
     # other
@@ -432,21 +400,85 @@ module Searchkick
             method_name: method_name ? method_name.to_s : nil
           )
         else
-          retries = 0
           records = records.select(&:should_index?)
-          begin
-            method_name ? bulk_update(records, method_name) : import(records)
-          rescue Faraday::ClientError => e
-            if retries < 1
-              retries += 1
-              retry
+          if records.any?
+            with_retries do
+              method_name ? bulk_update(records, method_name) : import(records)
             end
-            raise e
           end
         end
       end
     end
 
+    def full_reindex_async(scope)
+      if scope.respond_to?(:primary_key)
+        # TODO expire Redis key
+        primary_key = scope.primary_key
+        starting_id = scope.minimum(primary_key) || 0
+        max_id = scope.maximum(primary_key) || 0
+        batches_count = ((max_id - starting_id + 1) / batch_size.to_f).ceil
+
+        batches_count.times do |i|
+          batch_id = i + 1
+          min_id = starting_id + (i * batch_size)
+          bulk_reindex_job scope, batch_id, min_id: min_id, max_id: min_id + batch_size - 1
+        end
+      else
+        batch_id = 1
+        # TODO remove any eager loading
+        scope = scope.only(:_id) if scope.respond_to?(:only)
+        each_batch(scope) do |items|
+          bulk_reindex_job scope, batch_id, record_ids: items.map { |i| i.id.to_s }
+          batch_id += 1
+        end
+      end
+    end
+
+    def each_batch(scope)
+      # https://github.com/karmi/tire/blob/master/lib/tire/model/import.rb
+      # use cursor for Mongoid
+      items = []
+      scope.all.each do |item|
+        items << item
+        if items.length == batch_size
+          yield items
+          items = []
+        end
+      end
+      yield items if items.any?
+    end
+
+    def bulk_reindex_job(scope, batch_id, options)
+      Searchkick::BulkReindexJob.perform_later({
+        class_name: scope.model_name.name,
+        index_name: name,
+        batch_id: batch_id
+      }.merge(options))
+      redis.sadd(batches_key, batch_id) if redis
+    end
+
+    def batch_size
+      @batch_size ||= @options[:batch_size] || 1000
+    end
+
+    def with_retries
+      retries = 0
+
+      begin
+        yield
+      rescue Faraday::ClientError => e
+        if retries < 1
+          retries += 1
+          retry
+        end
+        raise e
+      end
+    end
+
+    def redis
+      Searchkick.redis
+    end
+
     def batches_key
       "searchkick:reindex:#{name}:batches"
     end

data/lib/searchkick/logging.rb

@@ -51,8 +51,44 @@ module Searchkick
           name: "#{records.first.searchkick_klass.name} Import",
           count: records.size
         }
-        ActiveSupport::Notifications.instrument("request.searchkick", event) do
-          super(records)
+        if Searchkick.callbacks_value == :bulk
+          super
+        else
+          ActiveSupport::Notifications.instrument("request.searchkick", event) do
+            super
+          end
+        end
+      end
+    end
+
+    def bulk_update(records, *args)
+      if records.any?
+        event = {
+          name: "#{records.first.searchkick_klass.name} Update",
+          count: records.size
+        }
+        if Searchkick.callbacks_value == :bulk
+          super
+        else
+          ActiveSupport::Notifications.instrument("request.searchkick", event) do
+            super
+          end
+        end
+      end
+    end
+
+    def bulk_delete(records)
+      if records.any?
+        event = {
+          name: "#{records.first.searchkick_klass.name} Delete",
+          count: records.size
+        }
+        if Searchkick.callbacks_value == :bulk
+          super
+        else
+          ActiveSupport::Notifications.instrument("request.searchkick", event) do
+            super
+          end
         end
       end
     end

data/lib/searchkick/process_batch_job.rb

@@ -15,8 +15,8 @@ module Searchkick
       # bulk reindex
       index = klass.searchkick_index
       Searchkick.callbacks(:bulk) do
-        index.bulk_index(records)
-        index.bulk_delete(delete_records)
+        index.bulk_index(records) if records.any?
+        index.bulk_delete(delete_records) if delete_records.any?
       end
     end
   end

data/lib/searchkick/process_queue_job.rb

@@ -5,7 +5,7 @@ module Searchkick
     def perform(class_name:)
       model = class_name.constantize
 
-      limit = 1000
+      limit = model.searchkick_index.options[:batch_size] || 1000
       record_ids = Searchkick::ReindexQueue.new(model.searchkick_index.name).reserve(limit: limit)
       if record_ids.any?
         Searchkick::ProcessBatchJob.perform_later(

data/lib/searchkick/reindex_queue.rb

@@ -4,6 +4,8 @@ module Searchkick
 
     def initialize(name)
       @name = name
+
+      raise Searchkick::Error, "Searchkick.redis not set" unless redis
     end
 
     def push(record_id)

data/lib/searchkick/version.rb

@@ -1,3 +1,3 @@
 module Searchkick
-  VERSION = "2.0.4"
+  VERSION = "2.1.0"
 end

data/test/callbacks_test.rb

@@ -35,7 +35,7 @@ class CallbacksTest < Minitest::Test
       store_names ["Product A", "Product B"]
     end
     Product.searchkick_index.refresh
-    assert_search "product", [], load: false
+    assert_search "product", [], load: false, conversions: false
     assert_equal 2, reindex_queue.length
 
     Searchkick::ProcessQueueJob.perform_later(class_name: "Product")

data/test/gemfiles/mongoid5.gemfile

@@ -4,4 +4,4 @@ source 'https://rubygems.org'
 gemspec path: "../../"
 
 gem "mongoid", "~> 5.0.0"
-gem "activejob_backport"
+gem "activejob"

data/test/index_test.rb

@@ -87,7 +87,7 @@ class IndexTest < Minitest::Test
 
   def test_remove_blank_id
     store_names ["Product A"]
-    Product.searchkick_index.remove(OpenStruct.new)
+    Product.searchkick_index.remove(Product.new)
     assert_search "product", ["Product A"]
   ensure
     Product.reindex

data/test/reindex_test.rb

@@ -1,12 +1,9 @@
 require_relative "test_helper"
 
 class ReindexTest < Minitest::Test
-  def setup
+  def test_scoped
     skip if nobrainer?
-    super
-  end
 
-  def test_scoped
     store_names ["Product A"]
     Searchkick.callbacks(false) do
       store_names ["Product B", "Product C"]
@@ -16,6 +13,8 @@ class ReindexTest < Minitest::Test
   end
 
   def test_associations
+    skip if nobrainer?
+
     store_names ["Product A"]
     store = Store.create!(name: "Test")
     Product.create!(name: "Product B", store_id: store.id)
@@ -24,13 +23,13 @@ class ReindexTest < Minitest::Test
   end
 
   def test_async
-    skip unless defined?(ActiveJob) && defined?(ActiveRecord)
+    skip if !defined?(ActiveJob)
 
     Searchkick.callbacks(false) do
       store_names ["Product A"]
     end
     reindex = Product.reindex(async: true)
-    assert_search "product", []
+    assert_search "product", [], conversions: false
 
     index = Searchkick::Index.new(reindex[:index_name])
     index.refresh

data/test/routing_test.rb

@@ -10,4 +10,14 @@ class RoutingTest < Minitest::Test
     index_options = Store.searchkick_index.index_options
     assert_equal index_options[:mappings][:_default_][:_routing], required: true
   end
+
+  def test_routing_correct_node
+    store_names ["Dollar Tree"], Store
+    assert_search "dollar", ["Dollar Tree"], {routing: "Dollar Tree"}, Store
+  end
+
+  def test_routing_incorrect_node
+    store_names ["Dollar Tree"], Store
+    assert_search "dollar", ["Dollar Tree"], {routing: "Boom"}, Store
+  end
 end

data/test/test_helper.rb

@@ -25,7 +25,7 @@ if defined?(ActiveJob)
   ActiveJob::Base.queue_adapter = :inline
 end
 
-ActiveSupport::LogSubscriber.logger = Logger.new(STDOUT) if ENV["NOTIFICATIONS"]
+ActiveSupport::LogSubscriber.logger = ActiveSupport::Logger.new(STDOUT) if ENV["NOTIFICATIONS"]
 
 def elasticsearch_below50?
   Searchkick.server_below?("5.0.0-alpha1")

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: searchkick
 version: !ruby/object:Gem::Version
-  version: 2.0.4
+  version: 2.1.0
 platform: ruby
 authors:
 - Andrew Kane
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-01-15 00:00:00.000000000 Z
+date: 2017-01-16 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activemodel
@@ -189,7 +189,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.6.8
+rubygems_version: 2.5.1
 signing_key: 
 specification_version: 4
 summary: Searchkick learns what your users are looking for. As more people search,