searchkick 4.4.2 → 4.5.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 782d732ce3dca45ba3654f3afa2ee565fc396543179dbdc87a0ee7418a151a3c
-  data.tar.gz: 2752fb40094f068a97e77009634fccf7a1433182294835ad077a66632d43bcb6
+  metadata.gz: 44f4a5255a208b7d24aa1b6d668808d3aff83a9ebb9f07a1a1a83fd9e3845738
+  data.tar.gz: 0ac096c02151a04a00751de140520547570b9c24fa2fccffd9d4761637b85531
 SHA512:
-  metadata.gz: b1125b09caaf20b0dcc8aa7c55755e257338499dda6d09427e139f9167d9e5b2e9971657b830352babe8100c510236412fbaa5a6edf9617c8eefcbe7aff4d2c7
-  data.tar.gz: b111187e5e1d05f19120f850833a491405727811fde6b31465f6021dffdd01b44e55c6e9a19b930246a4a92671ddda62aa65f9fa6fb6793342a5581009961985
+  metadata.gz: ff25ecdd74852fdbca33909d37445adbb42aa76785df75d9f1d9903d1dbee5712507d30a654dfaf829b9b6c0db0148cf48fbb4a257e3decf69e6c316c40b66fc
+  data.tar.gz: e07d8ddd9dbcf661db01948a7f20ad0b91d6427adddbbe987d571011983046650bd21726f74a2d0d11caaf9d49cd83b70a92738dc1cb21346e2c74009a1c494c

data/CHANGELOG.md

@@ -1,3 +1,22 @@
+## 4.5.1 (2021-08-03)
+
+- Improved performance of reindex queue
+
+## 4.5.0 (2021-06-07)
+
+- Added experimental support for OpenSearch
+- Added support for synonyms in Japanese
+
+## 4.4.4 (2021-03-12)
+
+- Fixed `too_long_frame_exception` with `scroll` method
+- Fixed multi-word emoji tokenization
+
+## 4.4.3 (2021-02-25)
+
+- Added support for Hunspell
+- Fixed warning about accessing system indices
+
 ## 4.4.2 (2020-11-23)
 
 - Added `missing_records` method to results

data/README.md

@@ -20,7 +20,7 @@ Plus:
 - autocomplete
 - “Did you mean” suggestions
 - supports many languages
-- works with ActiveRecord, Mongoid, and NoBrainer
+- works with Active Record, Mongoid, and NoBrainer
 
 Check out [Searchjoy](https://github.com/ankane/searchjoy) for analytics and [Autosuggest](https://github.com/ankane/autosuggest) for query suggestions
 
@@ -45,11 +45,11 @@ Check out [Searchjoy](https://github.com/ankane/searchjoy) for analytics and [Au
 
 ## Getting Started
 
-[Install Elasticsearch](https://www.elastic.co/downloads/elasticsearch). For Homebrew, use:
+Install [Elasticsearch](https://www.elastic.co/downloads/elasticsearch) or [OpenSearch](https://opensearch.org/downloads.html) (OpenSearch support is experimental). For Homebrew, use:
 
 ```sh
-brew install elasticsearch
-brew services start elasticsearch
+brew install elasticsearch # or opensearch
+brew services start elasticsearch # or opensearch
 ```
 
 Add this line to your application’s Gemfile:
@@ -58,7 +58,7 @@ Add this line to your application’s Gemfile:
 gem 'searchkick'
 ```
 
-The latest version works with Elasticsearch 6 and 7. For Elasticsearch 5, use version 3.1.3 and [this readme](https://github.com/ankane/searchkick/blob/v3.1.3/README.md).
+The latest version works with Elasticsearch 6 and 7 and OpenSearch 1. For Elasticsearch 5, use version 3.1.3 and [this readme](https://github.com/ankane/searchkick/blob/v3.1.3/README.md).
 
 Add searchkick to models you want to search.
 
@@ -176,7 +176,7 @@ Get the full response from Elasticsearch
 results.response
 ```
 
-**Note:** By default, Elasticsearch [limits paging](#deep-paging-master) to the first 10,000 results for performance. With Elasticsearch 7, this applies to the total count as well.
+**Note:** By default, Elasticsearch [limits paging](#deep-paging) to the first 10,000 results for performance. With Elasticsearch 7, this applies to the total count as well.
 
 ### Boosting
 
@@ -209,7 +209,7 @@ boost_by_recency: {created_at: {scale: "7d", decay: 0.5}}
 
 You can also boost by:
 
-- [Conversions](#keep-getting-better)
+- [Conversions](#intelligent-search)
 - [Distance](#boost-by-distance)
 
 ### Get Everything
@@ -311,7 +311,7 @@ class Product < ApplicationRecord
 end
 ```
 
-See the [list of stemmers](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-stemmer-tokenfilter.html). A few languages require plugins:
+See the [list of languages](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-stemmer-tokenfilter.html#analysis-stemmer-tokenfilter-configure-parms). A few languages require plugins:
 
 - `chinese` - [analysis-ik plugin](https://github.com/medcl/elasticsearch-analysis-ik)
 - `chinese2` - [analysis-smartcn plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-smartcn.html)
@@ -322,6 +322,14 @@ See the [list of stemmers](https://www.elastic.co/guide/en/elasticsearch/referen
 - `ukrainian` - [analysis-ukrainian plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-ukrainian.html)
 - `vietnamese` - [analysis-vietnamese plugin](https://github.com/duydo/elasticsearch-analysis-vietnamese)
 
+You can also use a Hunspell dictionary for stemming.
+
+```ruby
+class Product < ApplicationRecord
+  searchkick stemmer: {type: "hunspell", locale: "en_US"}
+end
+```
+
 Disable stemming with:
 
 ```ruby
@@ -641,7 +649,7 @@ class Product < ApplicationRecord
   def search_data
     {
       name: name,
-      conversions: searches.group(:query).uniq.count(:user_id)
+      conversions: searches.group(:query).distinct.count(:user_id)
       # {"ice cream" => 234, "chocolate" => 67, "cream" => 2}
     }
   end
@@ -1204,12 +1212,18 @@ FactoryBot.create(:product, :some_trait, :reindex, some_attribute: "foo")
 
 ### GitHub Actions
 
-Check out [setup-elasticsearch](https://github.com/ankane/setup-elasticsearch) for an easy way to install Elasticsearch.
+Check out [setup-elasticsearch](https://github.com/ankane/setup-elasticsearch) for an easy way to install Elasticsearch:
 
 ```yml
     - uses: ankane/setup-elasticsearch@v1
 ```
 
+And [setup-opensearch](https://github.com/ankane/setup-opensearch) for an easy way to install OpenSearch:
+
+```yml
+    - uses: ankane/setup-opensearch@v1
+```
+
 ## Deployment
 
 Searchkick uses `ENV["ELASTICSEARCH_URL"]` for the Elasticsearch server. This defaults to `http://localhost:9200`.
@@ -1217,7 +1231,7 @@ Searchkick uses `ENV["ELASTICSEARCH_URL"]` for the Elasticsearch server. This de
 - [Elastic Cloud](#elastic-cloud)
 - [Heroku](#heroku)
 - [Amazon Elasticsearch Service](#amazon-elasticsearch-service)
-- [Self-Hosted and Other](#other)
+- [Self-Hosted and Other](#self-hosted-and-other)
 
 ### Elastic Cloud
 
@@ -1469,7 +1483,7 @@ Product.search_index.promote(index_name, update_refresh_interval: true)
 
 ### Queuing
 
-Push ids of records needing reindexed to a queue and reindex in bulk for better performance. First, set up Redis in an initializer. We recommend using [connection_pool](https://github.com/mperham/connection_pool).
+Push ids of records needing reindexing to a queue and reindex in bulk for better performance. First, set up Redis in an initializer. We recommend using [connection_pool](https://github.com/mperham/connection_pool).
 
 ```ruby
 Searchkick.redis = ConnectionPool.new { Redis.new }
@@ -1572,14 +1586,14 @@ class ReindexConversionsJob < ApplicationJob
     # get records that have a recent conversion
     recently_converted_ids =
       Searchjoy::Search.where("convertable_type = ? AND converted_at > ?", class_name, 1.day.ago)
-      .order(:convertable_id).uniq.pluck(:convertable_id)
+      .order(:convertable_id).distinct.pluck(:convertable_id)
 
     # split into groups
     recently_converted_ids.in_groups_of(1000, false) do |ids|
       # fetch conversions
       conversions =
         Searchjoy::Search.where(convertable_id: ids, convertable_type: class_name)
-        .group(:convertable_id, :query).uniq.count(:user_id)
+        .group(:convertable_id, :query).distinct.count(:user_id)
 
       # group conversions by record
       conversions_by_record = {}
@@ -1831,7 +1845,7 @@ class Product < ApplicationRecord
   def search_data
     {
       name: name,
-      unique_user_conversions: searches.group(:query).uniq.count(:user_id),
+      unique_user_conversions: searches.group(:query).distinct.count(:user_id),
       # {"ice cream" => 234, "chocolate" => 67, "cream" => 2}
       total_conversions: searches.group(:query).count
       # {"ice cream" => 412, "chocolate" => 117, "cream" => 6}

data/lib/searchkick.rb

@@ -74,11 +74,24 @@ module Searchkick
     (defined?(@search_timeout) && @search_timeout) || timeout
   end
 
+  # private
+  def self.server_info
+    @server_info ||= client.info
+  end
+
   def self.server_version
-    @server_version ||= client.info["version"]["number"]
+    @server_version ||= server_info["version"]["number"]
+  end
+
+  def self.opensearch?
+    unless defined?(@opensearch)
+      @opensearch = server_info["version"]["distribution"] == "opensearch"
+    end
+    @opensearch
   end
 
   def self.server_below?(version)
+    server_version = opensearch? ? "7.10.2" : self.server_version
     Gem::Version.new(server_version.split("-")[0]) < Gem::Version.new(version.split("-")[0])
   end
 

data/lib/searchkick/index.rb

@@ -105,7 +105,7 @@ module Searchkick
       indices =
         begin
           if client.indices.respond_to?(:get_alias)
-            client.indices.get_alias
+            client.indices.get_alias(index: "#{name}*")
           else
             client.indices.get_aliases
           end
@@ -161,6 +161,7 @@ module Searchkick
       RecordData.new(self, record).document_type
     end
 
+    # TODO use like: [{_index: ..., _id: ...}] in Searchkick 5
     def similar_record(record, **options)
       like_text = retrieve(record).to_hash
         .keep_if { |k, _| !options[:fields] || options[:fields].map(&:to_s).include?(k) }

data/lib/searchkick/index_options.rb

@@ -153,6 +153,7 @@ module Searchkick
         }
       }
 
+      raise ArgumentError, "Can't pass both language and stemmer" if options[:stemmer] && language
       update_language(settings, language)
       update_stemming(settings)
 
@@ -234,6 +235,27 @@ module Searchkick
             type: "kuromoji"
           }
         )
+      when "japanese2"
+        analyzer = {
+          type: "custom",
+          tokenizer: "kuromoji_tokenizer",
+          filter: [
+            "kuromoji_baseform",
+            "kuromoji_part_of_speech",
+            "cjk_width",
+            "ja_stop",
+            "searchkick_stemmer",
+            "lowercase"
+          ]
+        }
+        settings[:analysis][:analyzer].merge!(
+          default_analyzer => analyzer.deep_dup,
+          searchkick_search: analyzer.deep_dup,
+          searchkick_search2: analyzer.deep_dup
+        )
+        settings[:analysis][:filter][:searchkick_stemmer] = {
+          type: "kuromoji_stemmer"
+        }
       when "korean"
         settings[:analysis][:analyzer].merge!(
           default_analyzer => {
@@ -286,6 +308,18 @@ module Searchkick
     end
 
     def update_stemming(settings)
+      if options[:stemmer]
+        stemmer = options[:stemmer]
+        # could also support snowball and stemmer
+        case stemmer[:type]
+        when "hunspell"
+          # supports all token filter options
+          settings[:analysis][:filter][:searchkick_stemmer] = stemmer
+        else
+          raise ArgumentError, "Unknown stemmer: #{stemmer[:type]}"
+        end
+      end
+
       stem = options[:stem]
 
       # language analyzer used
@@ -499,8 +533,18 @@ module Searchkick
         end
         settings[:analysis][:filter][:searchkick_synonym_graph] = synonym_graph
 
-        [:searchkick_search2, :searchkick_word_search].each do |analyzer|
-          settings[:analysis][:analyzer][analyzer][:filter].insert(2, "searchkick_synonym_graph")
+        if options[:language] == "japanese2"
+          [:searchkick_search, :searchkick_search2].each do |analyzer|
+            settings[:analysis][:analyzer][analyzer][:filter].insert(4, "searchkick_synonym_graph")
+          end
+        else
+          [:searchkick_search2, :searchkick_word_search].each do |analyzer|
+            unless settings[:analysis][:analyzer][analyzer].key?(:filter)
+              raise Searchkick::Error, "Search synonyms are not supported yet for language"
+            end
+
+            settings[:analysis][:analyzer][analyzer][:filter].insert(2, "searchkick_synonym_graph")
+          end
         end
       end
     end

data/lib/searchkick/model.rb

@@ -6,7 +6,7 @@ module Searchkick
       unknown_keywords = options.keys - [:_all, :_type, :batch_size, :callbacks, :case_sensitive, :conversions, :deep_paging, :default_fields,
         :filterable, :geo_shape, :highlight, :ignore_above, :index_name, :index_prefix, :inheritance, :language,
         :locations, :mappings, :match, :merge_mappings, :routing, :searchable, :search_synonyms, :settings, :similarity,
-        :special_characters, :stem, :stem_conversions, :stem_exclusion, :stemmer_override, :suggest, :synonyms, :text_end,
+        :special_characters, :stem, :stemmer, :stem_conversions, :stem_exclusion, :stemmer_override, :suggest, :synonyms, :text_end,
         :text_middle, :text_start, :word, :wordnet, :word_end, :word_middle, :word_start]
       raise ArgumentError, "unknown keywords: #{unknown_keywords.join(", ")}" if unknown_keywords.any?
 

data/lib/searchkick/process_queue_job.rb

@@ -11,7 +11,7 @@ module Searchkick
         if record_ids.any?
           batch_options = {
             class_name: class_name,
-            record_ids: record_ids,
+            record_ids: record_ids.uniq,
             index_name: index_name
           }
 

data/lib/searchkick/query.rb

@@ -25,7 +25,7 @@ module Searchkick
       term = term.to_s
 
       if options[:emoji]
-        term = EmojiParser.parse_unicode(term) { |e| " #{e.name} " }.strip
+        term = EmojiParser.parse_unicode(term) { |e| " #{e.name.tr('_', ' ')} " }.strip
       end
 
       @klass = klass
@@ -353,8 +353,8 @@ module Searchkick
               shared_options[:cutoff_frequency] = 0.001 unless operator.to_s == "and" || field_misspellings == false || (!below73? && !track_total_hits?)
               qs << shared_options.merge(analyzer: "searchkick_search")
 
-              # searchkick_search and searchkick_search2 are the same for ukrainian
-              unless %w(japanese korean polish ukrainian vietnamese).include?(searchkick_options[:language])
+              # searchkick_search and searchkick_search2 are the same for some languages
+              unless %w(japanese japanese2 korean polish ukrainian vietnamese).include?(searchkick_options[:language])
                 qs << shared_options.merge(analyzer: "searchkick_search2")
               end
               exclude_analyzer = "searchkick_search2"
@@ -864,10 +864,11 @@ module Searchkick
       }
     end
 
-    # TODO id transformation for arrays
     def set_order(payload)
       order = options[:order].is_a?(Enumerable) ? options[:order] : {options[:order] => :asc}
       id_field = :_id
+      # TODO no longer map id to _id in Searchkick 5
+      # since sorting on _id is deprecated in Elasticsearch
       payload[:sort] = order.is_a?(Array) ? order : Hash[order.map { |k, v| [k.to_s == "id" ? id_field : k, v] }]
     end
 

data/lib/searchkick/reindex_queue.rb

@@ -14,11 +14,17 @@ module Searchkick
 
     # TODO use reliable queuing
     def reserve(limit: 1000)
-      record_ids = Set.new
-      while record_ids.size < limit && (record_id = Searchkick.with_redis { |r| r.rpop(redis_key) })
-        record_ids << record_id
+      if supports_rpop_with_count?
+        Searchkick.with_redis { |r| r.call("rpop", redis_key, limit) }
+      else
+        record_ids = []
+        Searchkick.with_redis do |r|
+          while record_ids.size < limit && (record_id = r.rpop(redis_key))
+            record_ids << record_id
+          end
+        end
+        record_ids
       end
-      record_ids.to_a
     end
 
     def clear
@@ -34,5 +40,13 @@ module Searchkick
     def redis_key
       "searchkick:reindex_queue:#{name}"
     end
+
+    def supports_rpop_with_count?
+      redis_version >= Gem::Version.new("6.2")
+    end
+
+    def redis_version
+      @redis_version ||= Searchkick.with_redis { |r| Gem::Version.new(r.info["redis_version"]) }
+    end
   end
 end

data/lib/searchkick/results.rb

@@ -188,14 +188,9 @@ module Searchkick
 
         records.clear_scroll
       else
-        params = {
-          scroll: options[:scroll],
-          scroll_id: scroll_id
-        }
-
         begin
           # TODO Active Support notifications for this scroll call
-          Searchkick::Results.new(@klass, Searchkick.client.scroll(params), @options)
+          Searchkick::Results.new(@klass, Searchkick.client.scroll(scroll: options[:scroll], body: {scroll_id: scroll_id}), @options)
         rescue Elasticsearch::Transport::Transport::Errors::NotFound => e
           if e.class.to_s =~ /NotFound/ && e.message =~ /search_context_missing_exception/i
             raise Searchkick::Error, "Scroll id has expired"
@@ -236,7 +231,7 @@ module Searchkick
                 index_alias = index.split("_")[0..-2].join("_")
                 Array((options[:index_mapping] || {})[index_alias])
               end
-            raise Searchkick::Error, "Unknown model for index: #{index}" unless models.any?
+            raise Searchkick::Error, "Unknown model for index: #{index}. Pass the `models` option to the search method." unless models.any?
             index_models[index] = models
           end
 

data/lib/searchkick/version.rb

@@ -1,3 +1,3 @@
 module Searchkick
-  VERSION = "4.4.2"
+  VERSION = "4.5.1"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: searchkick
 version: !ruby/object:Gem::Version
-  version: 4.4.2
+  version: 4.5.1
 platform: ruby
 authors:
 - Andrew Kane
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-11-24 00:00:00.000000000 Z
+date: 2021-08-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activemodel
@@ -102,7 +102,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.1.4
+rubygems_version: 3.2.22
 signing_key:
 specification_version: 4
 summary: Intelligent search made easy with Rails and Elasticsearch