sunspot_offline 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: e59f5f34c118a37d3f16bda51c7cc1cc855e517498e57f343223a242fefc2d56
+  data.tar.gz: bfe500c3473cf63d82ae692a2a33ac6cbcb66f3f24dc33ef49a0d5e99df28b92
+SHA512:
+  metadata.gz: 5c94a3922fa1e5e143d1c93cd2df9f73d3918c43f3883caa9ae02d276a141408ed84758024c4313c2d36cf478ac77f5b7d0fe676a7d7858b9142088f03d439d1
+  data.tar.gz: 99a6f082841d494989de73c77ed47c21fe2bb1220d659eaff924e933698d3b73935535bbe6e7b481c7f162b6465e2854f2fd0c1b484d9591a7af3d0c2236581e

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright 2018 Sergey Glukhov
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,68 @@
+# Sunspot Offline
+Simple fail over method into your [sunspot_rails](https://github.com/sunspot/sunspot) + [sidekiq](https://github.com/mperham/sidekiq) setup
+
+[![Build Status](https://travis-ci.org/anjlab/sunspot_offline.svg?branch=master)](https://travis-ci.org/anjlab/sunspot_offline)
+[![Gem Version](https://badge.fury.io/rb/sunspot_offline.svg)](https://badge.fury.io/rb/sunspot_offline)
+
+## Why?
+
+Because Solr sometimes fails, it happens. It might be a maintenance work you have to do or just Out-Of-Memory problems.
+If you are running search-sensitive Rails app, you have to deal with it.
+This gem was developed to postpone your index tasks automatically into a sidekiq queue if Solr engine becomes unavailable
+
+## Rails Support
+
+Rails 5.0 and higher
+
+## Installation
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'sunspot_offline'
+```
+
+And then execute:
+```bash
+$ bundle
+```
+
+Or install it yourself as:
+```bash
+$ gem install sunspot_offline
+```
+
+## Configuration
+
+By default gem is configured to use built-in workers which will auto-retry indexing in 1 hour is Solr goes offline.
+This all is configurable, preferable via initializer:
+
+```ruby
+# config/initializers/sunspot_offline.rb
+  
+SunspotOffline.configure do |config|
+  config.retry_delay = 1.hour
+end
+```
+
+### Valid configuration options
+
+|Option         | Type| Default value|                      |
+|---------------|-----|--------------|----------------------|
+| index_job | Class| [bundled worker class](lib/sunspot_offline/sidekiq/index_worker.rb) | Sidekiq worker which will retry **saving new documents** to Solr. Accepts 2 arguments: ActiveRecord class and id (numbers, arrays, and hashes are all valid for this argument) |
+| removal_job | Class| [bundled worker class](lib/sunspot_offline/sidekiq/removal_worker.rb) | Sidekiq worker which will retry **removing existing documents** from Solr. Accepts same set of arguments as `index_job`.
+| retry_delay | Duration | 1 hour | Delay in which sidekiq will attempt to run `index_job` or `removal_job` |
+| default_queue | String | 'default' | Sidekiq's named queue to use |
+| solr_error_callback | Proc(exception) | `nil` | A proc which will be executed if Solr is detected to be offline |
+| filter_sidekiq_job_callback | Proc(job_class) | `nil` | Since some Solr indexing might be happening inside yours sidekiq jobs they dont need to have a custom fail over, sidekiq is able to retry failures by itself. Using this proc you can configure which jobs could be filtered out:<br>`->(job_name) { job_name == 'MySolrIndexJob' }`. In this sample `MySolrIndexJob` wont be applied for a fail over  |
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Run tests (`bundle exec appraisal install && bundle exec appraisal rake`)
+5. Push to the branch (`git push origin my-new-feature`)
+6. Create new Pull Request
+
+## License
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,8 @@
+require 'rake'
+require 'rspec/core/rake_task'
+require 'rubocop/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+RuboCop::RakeTask.new
+
+task default: %i[rubocop spec]

data/lib/sunspot_offline/rails/railtie.rb

@@ -0,0 +1,22 @@
+require 'sunspot'
+require 'sunspot_offline/sunspot/abstract_search'
+require 'sunspot_offline/rsolr/client'
+require 'sunspot_offline/sidekiq/current_job_middleware'
+
+module SunspotOffline
+  module Rails
+    class Railtie < ::Rails::Railtie
+      initializer 'sunspot_offline.init', before: 'sunspot_rails.init' do
+        ::Sunspot::Search::AbstractSearch.prepend SunspotOffline::Sunspot::AbstractSearch
+
+        ::RSolr::Client.prepend SunspotOffline::RSolr::Client
+
+        ::Sidekiq.configure_server do |config|
+          config.server_middleware do |chain|
+            chain.add SunspotOffline::Sidekiq::CurrentJobMiddleware
+          end
+        end
+      end
+    end
+  end
+end

data/lib/sunspot_offline/rsolr/client.rb

@@ -0,0 +1,53 @@
+module SunspotOffline
+  module RSolr
+    module Client
+      def add(documents, opts = {})
+        wrap_request(
+          deletion: false,
+          documents: -> { group_documents(documents.map { |doc| doc.field_by_name(:id).value }) }
+        ) { super }
+      end
+
+      def delete_by_id(id, opts = {})
+        wrap_request(deletion: true, documents: -> { group_documents(Array(id)) }) { super }
+      end
+
+      def delete_by_query(query, opts = {})
+        wrap_request(deletion: true, documents: -> { removal_documents(query) }) { super }
+      end
+
+      private
+
+      def wrap_request(deletion:, documents:)
+        yield
+      rescue StandardError => ex
+        raise ex if raise_exception?
+
+        job = deletion ? SunspotOffline.configuration.removal_job : SunspotOffline.configuration.index_job
+        perform_at = Time.zone.now + SunspotOffline.configuration.retry_delay
+        documents.call.each { |klass, list| job.perform_at(perform_at, klass, list) }
+        SunspotOffline.on_solr_error(ex)
+      end
+
+      def raise_exception?
+        sidekiq_job = SunspotOffline::Sidekiq::CurrentJobMiddleware.get
+        !sidekiq_job.nil? && (fail_over_job?(sidekiq_job) || SunspotOffline.filter_sidekiq_job?(sidekiq_job))
+      end
+
+      def fail_over_job?(job_name)
+        [SunspotOffline.configuration.index_job.name, SunspotOffline.configuration.removal_job.name].include?(job_name)
+      end
+
+      def group_documents(documents)
+        documents
+          .map { |id_text| id_text.split(' ') }
+          .group_by(&:first)
+          .map { |klass, list| [klass, list.map(&:last)] }
+      end
+
+      def removal_documents(query)
+        [query == '*:*' ? [nil, nil] : [query.split(':').last, nil]]
+      end
+    end
+  end
+end

data/lib/sunspot_offline/sidekiq/current_job_middleware.rb

@@ -0,0 +1,22 @@
+require 'request_store'
+
+module SunspotOffline
+  module Sidekiq
+    class CurrentJobMiddleware
+      STORE_KEY = 'Sidekiq.CurrentJob'
+
+      def initialize(*); end
+
+      def call(worker, _job, _queue)
+        RequestStore[STORE_KEY] = worker.class.name
+        yield
+      ensure
+        RequestStore[STORE_KEY] = nil
+      end
+
+      def self.get
+        RequestStore[STORE_KEY]
+      end
+    end
+  end
+end

data/lib/sunspot_offline/sidekiq/index_worker.rb

@@ -0,0 +1,22 @@
+require 'sidekiq'
+
+module SunspotOffline
+  module Sidekiq
+    class IndexWorker
+      include ::Sidekiq::Worker
+
+      def perform(model_class, scope)
+        klass = model_class.classify.constantize
+        if klass.searchable?
+          if scope.is_a?(Array)
+            scope.each_slice(1000) { |slice| Sunspot.index(klass.where(id: slice)) }
+          elsif scope.is_a?(Hash)
+            klass.atomic_update(scope.with_indifferent_access)
+          else
+            klass.where(id: scope).take&.index
+          end
+        end
+      end
+    end
+  end
+end

data/lib/sunspot_offline/sidekiq/removal_worker.rb

@@ -0,0 +1,17 @@
+require 'sidekiq'
+
+module SunspotOffline
+  module Sidekiq
+    class RemovalWorker
+      include ::Sidekiq::Worker
+
+      def perform(model_class, ids)
+        if ids.present?
+          Sunspot.remove_by_id(model_class, Array(ids))
+        else
+          Sunspot.remove_all(model_class)
+        end
+      end
+    end
+  end
+end

data/lib/sunspot_offline/sunspot/abstract_search.rb

@@ -0,0 +1,24 @@
+module SunspotOffline
+  module Sunspot
+    module AbstractSearch
+      def execute
+        super
+      rescue RSolr::Error::ConnectionRefused, RSolr::Error::Http => ex
+        @solr_result = {
+          'response' => {
+            'numFound' => -1,
+            'start' => 0,
+            'docs' => []
+          },
+          'facet_counts' => {
+            'facet_queries' => {},
+            'facet_fields' => {}
+          },
+          'grouped' => {}
+        }
+        SunspotOffline.on_solr_error(ex)
+        self
+      end
+    end
+  end
+end

data/lib/sunspot_offline/version.rb

@@ -0,0 +1,3 @@
+module SunspotOffline
+  VERSION = '0.1.0'
+end

data/lib/sunspot_offline.rb

@@ -0,0 +1,37 @@
+require 'ostruct'
+require 'sunspot_offline/version'
+require 'sunspot_offline/rails/railtie'
+require 'sunspot_offline/sidekiq/index_worker'
+require 'sunspot_offline/sidekiq/removal_worker'
+
+module SunspotOffline
+  class << self
+    def configuration
+      @configuration ||= OpenStruct.new(
+        retry_delay: 1.hour,
+        solr_error_callback: ->(_exception) {},
+        filter_sidekiq_job_callback: ->(_job) { false }, # some Sidekiq jobs are allowed to fail and retry on their own
+        index_job: Sidekiq::IndexWorker, # Sidekiq job which will handle index retries
+        removal_job: Sidekiq::RemovalWorker, # Sidekiq job which will handle removal retries
+        default_queue: 'default'
+      )
+    end
+
+    def configure
+      if block_given?
+        yield(configuration)
+        [SunspotOffline::Sidekiq::IndexWorker, SunspotOffline::Sidekiq::RemovalWorker].each do |worker|
+          worker.sidekiq_options[:queue] = configuration.default_queue
+        end
+      end
+    end
+
+    def on_solr_error(exception)
+      configuration.solr_error_callback.call(exception) if configuration.solr_error_callback
+    end
+
+    def filter_sidekiq_job?(job_class_name)
+      configuration.filter_sidekiq_job_callback.call(job_class_name) if configuration.filter_sidekiq_job_callback
+    end
+  end
+end

metadata

@@ -0,0 +1,212 @@
+--- !ruby/object:Gem::Specification
+name: sunspot_offline
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Sergey Glukhov
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2018-09-19 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: appraisal
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.1'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.1'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.16'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.16'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.6'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.6'
+- !ruby/object:Gem::Dependency
+  name: rubocop
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 0.58.2
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 0.58.2
+- !ruby/object:Gem::Dependency
+  name: solr_wrapper
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: sqlite3
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+- !ruby/object:Gem::Dependency
+  name: rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '5'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '5'
+- !ruby/object:Gem::Dependency
+  name: request_store
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.4'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.4'
+- !ruby/object:Gem::Dependency
+  name: sidekiq
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '4'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '4'
+- !ruby/object:Gem::Dependency
+  name: sunspot_rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.3'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.3'
+description: Because Solr sometimes fails, it happens. It might be a maintenance work
+  you have to do or just Out-Of-Memory problems. If you are running search-sensitive
+  Rails app, you have to deal with it.This gem was developed to postpone your index
+  tasks automatically into a sidekiq queue if Solr engine becomes unavailable
+email:
+- sergey.glukhov@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- MIT-LICENSE
+- README.md
+- Rakefile
+- lib/sunspot_offline.rb
+- lib/sunspot_offline/rails/railtie.rb
+- lib/sunspot_offline/rsolr/client.rb
+- lib/sunspot_offline/sidekiq/current_job_middleware.rb
+- lib/sunspot_offline/sidekiq/index_worker.rb
+- lib/sunspot_offline/sidekiq/removal_worker.rb
+- lib/sunspot_offline/sunspot/abstract_search.rb
+- lib/sunspot_offline/version.rb
+homepage: https://github.com/anjlab/sunspot_offline
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.7.6
+signing_key: 
+specification_version: 4
+summary: Simple failover method into your sunspot_rails + sidekiq setup
+test_files: []