lifesaver 0.0.1 → 0.1.0

data/.rspec

@@ -0,0 +1 @@
+--color

data/.travis.yml

@@ -4,4 +4,6 @@ rvm:
   - 2.0.0
 services:
   - redis-server
-  - elasticsearch
+  - elasticsearch
+env:
+  - RUN_COVERALLS=true

data/CHANGELOG.md

@@ -1,3 +1,10 @@
+## v0.1.0
+
+* Added config
+* Decoupled Indexing and Notification
+* Notification uses eager loading to avoid N+1 queries during graph traversal
+
+
 ## v0.0.1
 
 * initial release

data/Gemfile

@@ -6,3 +6,7 @@ gemspec
 group :test do
   gem 'coveralls', require: false
 end
+
+group :development, :test do
+  gem 'pry', require: false
+end

data/README.md

@@ -1,17 +1,18 @@
 # Lifesaver
 
 [![Build Status](https://travis-ci.org/paulnsorensen/lifesaver.png?branch=master)](https://travis-ci.org/paulnsorensen/lifesaver)
+[![Gem Version](https://badge.fury.io/rb/lifesaver.png)](http://badge.fury.io/rb/lifesaver)
 [![Dependency Status](https://gemnasium.com/paulnsorensen/lifesaver.png)](https://gemnasium.com/paulnsorensen/lifesaver)
 [![Coverage Status](https://coveralls.io/repos/paulnsorensen/lifesaver/badge.png)](https://coveralls.io/r/paulnsorensen/lifesaver)
 [![Code Climate](https://codeclimate.com/github/paulnsorensen/lifesaver.png)](https://codeclimate.com/github/paulnsorensen/lifesaver)
 
-Indexes your ActiveRecord models in [elasticsearch](https://github.com/elasticsearch/elasticsearch) asynchronously by making use of [tire](https://github.com/karmi/tire) and [resque](https://github.com/resque/resque) (hence the name: resque + tire = lifesaver). Using lifesaver, you can easily control when or if to reindex your model depending on your context. Lifesaver also provides the ability to traverse ActiveRecord associations to trigger the index updates of related models.
+Asynchronously sends your ActiveRecord models for reindexing in [elasticsearch](https://github.com/elasticsearch/elasticsearch) by making use of [tire](https://github.com/karmi/tire) and [resque](https://github.com/resque/resque) (hence the name: resque + tire = lifesaver). Lifesaver also provides the ability to traverse ActiveRecord associations to trigger the index updates of related models.
 
 ## Installation
 
 Add this line to your application's Gemfile:
 
-    gem 'lifesaver', git: "git://github.com/paulnsorensen/lifesaver.git"
+    gem 'lifesaver'
 
 And then execute:
 
@@ -34,8 +35,8 @@ Replaces the tire callbacks in your models
     end
 ```
 
-#### Configurable Behavior
-You can decided when or if the index gets updated at all based on your current situation. Lifesaver exposes two methods (`supress_indexing`, `unsuppress_indexing`) that set a model's indexing behavior until that model is saved.
+#### Configuring Indexing Behavior
+You can suppress index updates on a per-model basis or globally using `Lifesaver.suppress_indexing` (to turn suppression back off, you would use `Lifesaver.unsuppress_indexing`). Lifesaver exposes two instance methods on the model level (`supress_indexing`, `unsuppress_indexing`) that set a model's indexing behavior for life of that instance.
 
 ```ruby
     class ArticlesController < ApplicationController
@@ -47,7 +48,7 @@ You can decided when or if the index gets updated at all based on your current s
         @article.suppress_indexing
 
         @article.save!
-        
+
         # Not neccessary but if saved
         # after this following call,
         # this article would reindex
@@ -72,8 +73,30 @@ Lifesaver can trigger other models to reindex if you have nested models in your
     end
 ```
 
+## Integration with Tire
+Lifesaver will not execute any `<after|before>_update_elasticsearch_index` callback hooks. Lifesaver also does not currently support percolation.
+
 ## Integration with Resque
-You will see two new queues: `lifesaver_indexing` and `lifesaver_notification`
+You will see two new queues: `lifesaver_indexing` and `lifesaver_notification`. The queue names are configurable.
+
+## Testing
+
+In your spec_helper, you should place something similar to the following to make sure Lifesaver isn't spawning up indexing jobs unless you want it to.
+
+```ruby
+    config.before(:each) do
+      Lifesaver.suppress_indexing
+    end
+```
+
+Then, when your tests need Lifesaver to run, you should make sure you unsuppress indexing in a `before` block. You may also want to run [Resque inline](http://robots.thoughtbot.com/process-jobs-inline-when-running-acceptance-tests).
+
+```ruby
+  describe 'some test' do
+    before { Lifesaver.unsuppress_indexing }
+    # tests go here
+  end
+```
 
 ## Contributing
 
@@ -84,10 +107,7 @@ You will see two new queues: `lifesaver_indexing` and `lifesaver_notification`
 5. Create new Pull Request
 
 ## TODO
-+ specify which fields will trigger indexing changes
-+ configuration options
-+ bulk indexing
-+ resque-scheduler to provide `delay_indexing` and `enqueues_indexing after: 30.minutes, on: :save`
-+ unsuppress indexing after save
-+ sidekiq support
-+ prepare for new elasticsearch library
+Please visit TODO page [here](https://github.com/paulnsorensen/lifesaver/wiki/TODO)
+
+
+[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/paulnsorensen/lifesaver/trend.png)](https://bitdeli.com/free "Bitdeli Badge")

data/lib/lifesaver.rb

@@ -1,24 +1,47 @@
 require 'resque-loner'
-require "lifesaver/version"
-require "lifesaver/marshal"
-require "lifesaver/index_graph"
-require "lifesaver/model_additions"
-require "lifesaver/index_worker"
-require "lifesaver/visitor_worker"
-require "lifesaver/railtie" if defined? Rails
+require 'lifesaver/version'
+require 'lifesaver/config'
+require 'lifesaver/serialized_model'
+require 'lifesaver/indexing/model_additions'
+require 'lifesaver/indexing/enqueuer'
+require 'lifesaver/indexing/indexer'
+require 'lifesaver/notification/notifiable_associations'
+require 'lifesaver/notification/model_additions'
+require 'lifesaver/notification/eager_loader'
+require 'lifesaver/notification/traversal_queue'
+require 'lifesaver/notification/indexing_graph'
+require 'lifesaver/notification/enqueuer'
+require 'lifesaver/index_worker'
+require 'lifesaver/visitor_worker'
+require 'lifesaver/railtie' if defined? Rails
 
 module Lifesaver
+  extend self
+
   @@suppress_indexing = false
 
-  def self.suppress_indexing
+  def suppress_indexing
     @@suppress_indexing = true
   end
 
-  def self.unsuppress_indexing
+  def unsuppress_indexing
     @@suppress_indexing = false
   end
 
-  def self.indexing_suppressed?
+  def indexing_suppressed?
     @@suppress_indexing
   end
-end
+
+  def config=(options = {})
+    @config = Config.new(options)
+  end
+
+  def config
+    @config ||= Config.new
+  end
+
+  def configure
+    yield config
+  end
+
+end

data/lib/lifesaver/config.rb

@@ -0,0 +1,16 @@
+module Lifesaver
+  # A container for configuration parameters
+  class Config
+    attr_accessor :notification_queue, :indexing_queue
+
+    def initialize(options = {})
+
+      @notification_queue = :lifesaver_notification
+      @indexing_queue = :lifesaver_indexing
+
+      options.each do |key, value|
+        public_send("#{key}=", value)
+      end
+    end
+  end
+end

data/lib/lifesaver/index_worker.rb

@@ -1,13 +1,13 @@
 class Lifesaver::IndexWorker
   include ::Resque::Plugins::UniqueJob
-  @queue = :lifesaver_indexing
-  def self.perform(class_name, id, action)
-    klass = class_name.to_s.classify.constantize
-    case action.to_sym
-    when :update
-      klass.find(id).update_index if klass.exists?(id)
-    when :destroy
-      klass.index.remove({type: klass.document_type, id: id})
-    end
+
+  def self.queue; Lifesaver.config.indexing_queue end
+
+  def self.perform(class_name, model_id, operation)
+    Lifesaver::Indexing::Indexer.new(
+                                     class_name: class_name,
+                                     model_id: model_id,
+                                     operation: operation
+                                    ).perform
   end
-end
+end

data/lib/lifesaver/indexing/enqueuer.rb

@@ -0,0 +1,37 @@
+module Lifesaver
+  module Indexing
+    class Enqueuer
+      def initialize(args)
+        @model = args[:model]
+        @operation = args[:operation]
+      end
+
+      def enqueue
+        if should_enqueue?(model)
+          ::Resque.enqueue(
+                           Lifesaver::IndexWorker,
+                           class_name,
+                           model_id,
+                           operation
+                          )
+        end
+      end
+
+      private
+
+      attr_reader :model, :operation
+
+      def should_enqueue?(model)
+        model.should_index?
+      end
+
+      def class_name
+        model.class.name.underscore.to_sym
+      end
+
+      def model_id
+        model.id
+      end
+    end
+  end
+end

data/lib/lifesaver/indexing/indexer.rb

@@ -0,0 +1,40 @@
+module Lifesaver
+  module Indexing
+    class Indexer
+      def initialize(args)
+        @class_name = args[:class_name]
+        @model_id = args[:model_id]
+        @operation = args[:operation].to_sym
+      end
+
+      def perform
+        case operation
+        when :update
+          store
+        when :destroy
+          remove
+        end
+      end
+
+      private
+
+      attr_reader :class_name, :model_id, :operation
+
+      def index
+        @index ||= Tire.index(klass.index_name)
+      end
+
+      def klass
+        @klass ||= class_name.to_s.classify.constantize
+      end
+
+      def store
+        index.store(klass.find(model_id)) if klass.exists?(model_id)
+      end
+
+      def remove
+        index.remove(type: klass.document_type, id: model_id)
+      end
+    end
+  end
+end

data/lib/lifesaver/indexing/model_additions.rb

@@ -0,0 +1,55 @@
+module Lifesaver
+  module Indexing
+    module ModelAdditions
+      module ClassMethods
+        def enqueues_indexing
+          indexing_callbacks
+        end
+
+        private
+
+        def indexing_callbacks(options = {})
+          # after_commit?
+          after_save do
+            send :enqueue_indexing, options.merge(operation: :update)
+          end
+          after_destroy do
+            send :enqueue_indexing, options.merge(operation: :destroy)
+          end
+        end
+      end
+
+      def self.included(base)
+        base.extend(ClassMethods)
+      end
+
+      def has_index?
+        self.respond_to?(:tire)
+      end
+
+      def should_index?
+        has_index? && !suppress_indexing?
+      end
+
+      def suppress_indexing
+        @indexing_suppressed = true
+      end
+
+      def unsuppress_indexing
+        @indexing_suppressed = false
+      end
+
+      private
+
+      def enqueue_indexing(options)
+        operation = options[:operation]
+        Lifesaver::Indexing::Enqueuer.new(model: self,
+                                          operation: operation).enqueue
+      end
+
+      def suppress_indexing?
+        Lifesaver.indexing_suppressed? || @indexing_suppressed || false
+      end
+    end
+  end
+end

data/lib/lifesaver/notification/eager_loader.rb

@@ -0,0 +1,48 @@
+module Lifesaver
+  module Notification
+    class EagerLoader
+      def initialize
+        @models_to_load = {}
+        @loaded_models = {}
+      end
+
+      def add_model(class_name, id)
+        return if model_previously_added?(class_name, id)
+        models_to_load[class_name] ||= []
+        models_to_load[class_name] << id
+        mark_model_added(class_name, id)
+      end
+
+      def load
+        models = []
+        models_to_load.each do |class_name, ids|
+          klass = class_name.classify.constantize
+          models |= load_associations(klass, ids)
+          models_to_load.delete(class_name)
+        end
+        models
+      end
+
+      def empty?
+        @models_to_load.empty?
+      end
+
+      private
+
+      attr_accessor :models_to_load, :loaded_models
+
+      def load_associations(klass, ids)
+        klass.load_with_notifiable_associations(ids)
+      end
+
+      def model_previously_added?(class_name, id)
+        loaded_models[class_name] && loaded_models[class_name][id]
+      end
+
+      def mark_model_added(class_name, id)
+        loaded_models[class_name] ||= {}
+        loaded_models[class_name][id] = true
+      end
+    end
+  end
+end

data/lib/lifesaver/notification/enqueuer.rb

@@ -0,0 +1,24 @@
+module Lifesaver
+  module Notification
+    class Enqueuer
+      def initialize(models)
+        @serialized_models = models
+      end
+
+      def enqueue
+        if should_enqueue?
+          ::Resque.enqueue(Lifesaver::VisitorWorker, serialized_models)
+        end
+      end
+
+      private
+
+      attr_accessor :serialized_models
+
+      def should_enqueue?
+        !serialized_models.empty? && !Lifesaver.indexing_suppressed?
+      end
+
+    end
+  end
+end

data/lib/lifesaver/notification/indexing_graph.rb

@@ -0,0 +1,91 @@
+module Lifesaver
+  module Notification
+    class IndexingGraph
+      def initialize
+        @queue = Lifesaver::Notification::TraversalQueue.new
+        @loader = Lifesaver::Notification::EagerLoader.new
+        @models_to_index = []
+      end
+
+      def initialize_models(serialized_models)
+
+        serialized_models.each do |model_hash|
+          model = Lifesaver::SerializedModel.new
+          model.class_name = model_hash['class_name']
+          model.id = model_hash['id']
+          add_model_to_loader(model.class_name, model.id)
+        end
+      end
+
+      def generate
+        loop do
+          if queue_full?
+            model = pop_model
+            models_to_index << model if model_should_be_indexed?(model)
+            add_unvisited_associations(model)
+          elsif loader_full?
+            load_into_queue
+          else
+            break
+          end
+        end
+        models_to_index
+      end
+
+      private
+
+      attr_accessor :queue, :loader, :models_to_index
+
+      def loader_full?
+        !loader.empty?
+      end
+
+      def queue_full?
+        !queue.empty?
+      end
+
+      def pop_model
+        queue.pop
+      end
+
+      def push_model(model)
+        queue << model
+      end
+
+      def add_model_to_loader(class_name, id)
+        loader.add_model(class_name, id)
+      end
+
+      def load_models
+        loader.load
+      end
+
+      def load_into_queue
+        load_models.each { |model| queue << model }
+      end
+
+      def model_should_be_indexed?(model)
+        model.has_index?
+      end
+
+      def load_associations_for_model(model)
+        model.associations_to_notify
+      end
+
+      def model_needs_to_notify?(model)
+        model.needs_to_notify?
+      end
+
+      def add_unvisited_associations(model)
+        models = load_associations_for_model(model)
+        models.each do |m|
+          if model_needs_to_notify?(model)
+            add_model_to_loader(m.class.name, m.id)
+          else
+            push_model(model)
+          end
+        end
+      end
+    end
+  end
+end