predictor 1.0.0 → 2.0.0.rc1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 23c921a064f6dcb321d1948e051545298616f3b7
-  data.tar.gz: 501a6132f7ea81fa316faf5a5f14a7b0d28afcdb
+  metadata.gz: 29f61606a156b3a6132dc9212f3d027492285d2e
+  data.tar.gz: acf97ff88f34ca518536e18b7753fe27e69956ec
 SHA512:
-  metadata.gz: 0b14e50f6df801912204a8a312124eba2e44888e8c5b8b1e8f3fe51808b0b3cad2e794b55633fb747cb60189d782312864c49c5105d03de5e1992f240a3528d8
-  data.tar.gz: 850b7d299e0f3ce4352fb3cd3423e96d598f83a72b5a0cf2d0cf27734f25e8951f2834cb40444931820721dce74bb4f029a82081765abb4069fa25d9a31fa04c
+  metadata.gz: cd25db9f133ee47f44703e8e6cab7d0daed59b9a035abfac75d733d6891d0c312981f76875b0278a9b66285c3f374650b01fe4466f45b51c0d42685dafae4783
+  data.tar.gz: 4b01b5f70cdb3d4de6267e72502b50e8c8c7210f3d13855477b9d1dd9a4556b57787bca2827ddc7ac3060c915f032a2b32a50b5feb13709b9ecd5a86167c092c

data/Changelog.md

@@ -0,0 +1,13 @@
+=======
+Predictor Changelog
+=========
+2.0.0 (2014-03-07)
+---------------------
+**Rewrite of 1.0.0 and contains several breaking changes!**
+
+Version 1.0.0 (which really should have been 0.0.1) contained several issues that made compatability with v2 not worth the trouble. This includes:
+* In v1, similarities were cached per input_matrix, and Predictor::Base utilized those caches when determining similarities and predictions. This quickly ate up Redis memory with even a semi-large dataset, as each input_matrix had a significant memory requirement. v2 caches similarities at the root (Recommender::Base), which means you can add any number of input matrices with little impact on memory usage.
+* Added the ability to limit the number of items stored in the similarity cache (via the 'limit_similarities_to' option). Now that similarities are cached at the root, this is possible and can greatly help memory usage.
+* Removed bang methods from input_matrix (add_set!, and_single!, etc). These called process! for you previously, but since the cache is no longer kept at the input_matrix level, process! has to be called at the root (Recommender::Base)
+* Bug fix: Fixed bug where a call to delete_item! on the input matrix didn't update the similarity cache.
+* Other minor fixes.

data/Gemfile

@@ -1,4 +1,4 @@
-source :rubygems
+source 'https://rubygems.org'
 
 gem "redis"
 

data/README.md

@@ -2,7 +2,7 @@
 Predictor
 =========
 
-Fast and efficient recommendations and predictions using Ruby & Redis. Used in production over at [Pathgather](http://pathgather.com) to recommend content to users.
+Fast and efficient recommendations and predictions using Ruby & Redis. Developed by and used at [Pathgather](http://pathgather.com) to generate course similarities and content recommendations to users.
 
 ![](https://www.codeship.io/projects/5aeeedf0-6053-0131-2319-5ede98f174ff/status)
 
@@ -13,6 +13,10 @@ Originally forked and based on [Recommendify](https://github.com/paulasmuth/reco
 
 At the moment, Predictor uses the [Jaccard index](http://en.wikipedia.org/wiki/Jaccard_index) to determine similarities between items. There are other ways to do this, which we intend to implement eventually, but if you want to beat us to the punch, pull requests are quite welcome :)
 
+Notice
+---------------------
+This is the readme for Predictor 2.0, which contains a few breaking changes from 1.0. The 1.0 readme can be found [here](https://github.com/Pathgather/predictor/blob/master/docs/READMEv1.md). See below on how to upgrade to 2.0
+
 Installation
 ---------------------
 ```ruby
@@ -29,9 +33,10 @@ First step is to configure Predictor with your Redis instance.
 # in config/initializers/predictor.rb
 Predictor.redis = Redis.new(:url => ENV["PREDICTOR_REDIS"])
 
-# Or, to improve performance, add hiredis as your driver (you'll need to install the hiredis gem first
+# Or, to improve performance, add hiredis as your driver (you'll need to install the hiredis gem first)
 Predictor.redis = Redis.new(:url => ENV["PREDICTOR_REDIS"], :driver => :hiredis)
 ```
+
 Inputting Data
 ---------------------
 Create a class and include the Predictor::Base module. Define an input_matrix for each relationship you'd like to keep track of. This can be anything you think is a significant metric for the item: page views, purchases, categories the item belongs to, etc.
@@ -51,6 +56,7 @@ Below, we're building a recommender to recommend courses based off of:
 class CourseRecommender
   include Predictor::Base
 
+  limit_similarities_to 500   # Optional, but if specified, Predictor only caches the top x similarities for an item at any given time. Can greatly help with efficient use of Redis memory
   input_matrix :users, weight: 3.0
   input_matrix :tags, weight: 2.0
   input_matrix :topics, weight: 1.0
@@ -62,37 +68,21 @@ Now, we just need to update our matrices when courses are created, users take a
 recommender = CourseRecommender.new
 
 # Add a single course to topic-1's items. If topic-1 already exists as a set ID, this just adds course-1 to the set
-recommender.topics.add_single!("topic-1", "course-1")
-
-# If your matrix is quite large, add_single! could take some time, as it must calculate the similarity scores
-# for course-1 across all other courses. If this is the case, use add_single and process the item at a more
-# convenient time, perhaps in a background job
-recommender.topics.add_single("topic-1", "course-1")
-recommender.topics.process_item!("course-1")
-
-# Add an array of courses to tag-1. Again, these will simply be added to tag-1's existing set, if it exists.
-# If not, the tag-1 set will be initialized with course-1 and course-2
-recommender.tags.add_set!("tag-1", ["course-1", "course-2"])
+recommender.add_to_matrix!(:topics, "topic-1", "course-1")
 
-# Or, just add the set and process whenever you like
-recommender.tags.add_set("tag-1", ["course-1", "course-2"])
-["course-1", "course-2"].each { |course| recommender.topics.process_item!(course) }
+# If your dataset is even remotely large, add_to_matrix! could take some time, as it must calculate the similarity scores
+# for course-1 and other courses that share a set with course-1. If this is the case, use add_to_matrix and
+# process the items at a more convenient time, perhaps in a background job
+recommender.topics.add_to_set("topic-1", "course-1", "course-2") # Same as recommender.add_to_matrix(:topics, "topic-1", "course-1", "course-2")
+recommender.process_items!("course-1", "course-2")
 ```
 
-As noted above, it's important to remember that if you don't use the bang methods (add_set! and add_single!), you'll need to manually update your similarities (the bang methods will likely suffice for most use cases though). You can do so a variety of ways.
-* If you want to simply update the similarities for a single item in a specific matrix:
-  ````
-  recommender.matrix.process_item!(item)
-  ````
-* If you want to update the similarities for all items in a specific matrix:
-  ````
-  recommender.matrix.process!
-  ````
-* If you want to update the similarities for a single item in all matrices:
+As noted above, it's important to remember that if you don't use the bang method 'add_to_matrix!', you'll need to manually update your similarities. If your dataset is even remotely large, you'll probably want to do this:
+* If you want to update the similarities for certain item(s):
   ````
-  recommender.process_item!(item)
+  recommender.process_items!(item1, item2, etc)
   ````
-* If you want to update all similarities in all matrices:
+* If you want to update all similarities for all items:
   ````
   recommender.process!
   ````
@@ -100,6 +90,9 @@ As noted above, it's important to remember that if you don't use the bang method
 Retrieving Similarities and Recommendations
 ---------------------
 Now that your matrices have been initialized with several relationships, you can start generating similarities and recommendations! First, let's start with similarities, which will use the weights we specify on each matrix to determine which courses share the most in common with a given course.
+
+![Course Alternative](http://pathgather.github.io/predictor/images/course-alts.png)
+
 ```ruby
 recommender = CourseRecommender.new
 
@@ -117,6 +110,9 @@ recommender.similarities_for("course-1", exclusion_set: ["course-2"])
 ```
 
 The above examples are great for situations like "Users that viewed this also liked ...", but what if you wanted to recommend courses to a user based on the courses they've already taken? Not a problem!
+
+![Course Recommendations](http://pathgather.github.io/predictor/images/suggested.png)
+
 ```ruby
 recommender = CourseRecommender.new
 
@@ -129,18 +125,18 @@ recommender.predictions_for("user-1", matrix_label: :users)
 # Paginate too!
 recommender.predictions_for("user-1", matrix_label: :users, offset: 10, limit: 10)
 
-# Gimme some scores and ignore user-2....that user-2 is one sketchy fella
-recommender.predictions_for("user-1", matrix_label: :users, with_scores: true, exclusion_set: ["user-2"])
+# Gimme some scores and ignore course-2....that course-2 is one sketchy fella
+recommender.predictions_for("user-1", matrix_label: :users, with_scores: true, exclusion_set: ["course-2"])
 ```
 
 Deleting Items
 ---------------------
-If your data is deleted from your persistent storage, you certainly don't want to recommend that data to a user. To ensure that doesn't happen, simply call delete_item! on the individual matrix or recommender as a whole:
+If your data is deleted from your persistent storage, you certainly don't want to recommend it to a user. To ensure that doesn't happen, simply call delete_from_matrix! with the individual matrix or delete_item! if the item is completely gone:
 ```ruby
 recommender = CourseRecommender.new
 
 # User removed course-1 from topic-1, but course-1 still exists
-recommender.topics.delete_item!("course-1")
+recommender.delete_from_matrix!(:topics, "course-1")
 
 # course-1 was permanently deleted
 recommender.delete_item!("course-1")
@@ -149,6 +145,53 @@ recommender.delete_item!("course-1")
 recommender.clean!
 ```
 
+Limiting Similarities
+---------------------
+By default, Predictor caches all similarities for all items, with no limit. That means if you have 10,000 items, and each item is somehow related to the other, we'll have 10,000 sets each with 9,999 items. That's going to use Redis' memory quite quickly. To limit this, specify the limit_similarities_to option.
+```ruby
+class CourseRecommender
+  include Predictor::Base
+
+  limit_similarities_to 500
+  input_matrix :users, weight: 3.0
+  input_matrix :tags, weight: 2.0
+  input_matrix :topics, weight: 1.0
+end
+```
+
+This can really save a ton of memory. Just remember though, predictions fetched with the predictions_for call utilzes the similarity caches, so if you're using predictions_for, make sure you set the limit high enough so that intelligent predictions can be generated. If you aren't using predictions and are just using similarities, then feel free to set this to the maximum number of similarities you'd possibly want to show!
+
+Upgrading from 1.0 to 2.0
+---------------------
+As mentioned, 2.0.0 is quite a bit different than 1.0.0, so simply upgrading with no changes likely won't work. My apologies for this. I promise this won't happen in future releases, as I'm much more confident in this Predictor release than the last. Anywho, upgrading really shouldn't be that much of a pain if you follow these steps:
+
+* Change predictor.matrix.add_set! and predictor.matrix.add_single! calls to predictor.add_to_matrix!. For example:
+```ruby
+# Change
+predictor.topics.add_single!("topic-1", "course-1")
+# to
+predictor.add_to_matrix!(:topics, "topic-1", "course-1")
+
+# Change
+predictor.tags.add_set!("tag-1", ["course-1", "course-2"])
+# to
+predictor.add_to_matrix!(:tags, "tag-1", "course-1", "course-2")
+```
+* Change predictor.matrix.process! or predictor.matrix.process_item! calls to just predictor.process! or predictor.process_items!
+```ruby
+# Change
+predictor.topics.process_item!("course-1")
+# to
+predictor.process_items!("course-1")
+```
+* Change predictor.matrix.delete_item! calls to predictor.delete_from_matrix!. This will update similarities too, so you may want to queue this to run in a background job.
+```ruby
+# Change
+predictor.topics.delete_item!("course-1")
+# to delete_from_matrix! if you want to update similarities to account for the deleted item (in v1, this was a bug and didn't occur)
+predictor.delete_from_matrix!(:topics, "course-1")
+```
+
 Problems? Issues? Want to help out?
 ---------------------
 Just submit a Gihub issue or pull request! We'd love to have you help out, as the most common library to use for this need, Recommendify, was last updated 2 years ago. We'll be sure to keep this maintained, but we could certainly use your help!

data/docs/READMEv1.md

@@ -0,0 +1,206 @@
+=======
+Predictor
+=========
+
+Fast and efficient recommendations and predictions using Ruby & Redis. Used in production over at [Pathgather](http://pathgather.com) to generate course similarities and content recommendations to users.
+
+![](https://www.codeship.io/projects/5aeeedf0-6053-0131-2319-5ede98f174ff/status)
+
+Originally forked and based on [Recommendify](https://github.com/paulasmuth/recommendify) by Paul Asmuth, so a huge thanks to him for his contributions to Recommendify. Predictor has been almost completely rewritten to
+* Be much, much more performant and efficient by using Redis for most logic.
+* Provide item similarities such as "Users that read this book also read ..."
+* Provide personalized predictions based on a user's past history, such as "You read these 10 books, so you might also like to read ..."
+
+At the moment, Predictor uses the [Jaccard index](http://en.wikipedia.org/wiki/Jaccard_index) to determine similarities between items. There are other ways to do this, which we intend to implement eventually, but if you want to beat us to the punch, pull requests are quite welcome :)
+
+Installation
+---------------------
+```ruby
+gem install predictor
+````
+or in your Gemfile:
+````
+gem 'predictor'
+```
+Getting Started
+---------------------
+First step is to configure Predictor with your Redis instance.
+```ruby
+# in config/initializers/predictor.rb
+Predictor.redis = Redis.new(:url => ENV["PREDICTOR_REDIS"])
+
+# Or, to improve performance, add hiredis as your driver (you'll need to install the hiredis gem first)
+Predictor.redis = Redis.new(:url => ENV["PREDICTOR_REDIS"], :driver => :hiredis)
+```
+Inputting Data
+---------------------
+Create a class and include the Predictor::Base module. Define an input_matrix for each relationship you'd like to keep track of. This can be anything you think is a significant metric for the item: page views, purchases, categories the item belongs to, etc.
+
+Below, we're building a recommender to recommend courses based off of:
+* Users that have taken a course. If 2 courses were taken by the same user, this is 3 times as important to us than if the courses share the same topic. This will lead to sets like:
+  * "user1" -> "course-1", "course-3",
+  * "user2" -> "course-1", "course-4"
+* Tags and their courses. This will lead to sets like:
+  * "rails" -> "course-1", "course-2",
+  * "microeconomics" -> "course-3", "course-4"
+* Topics and their courses. This will lead to sets like:
+  * "computer science" -> "course-1", "course-2",
+  * "economics and finance" -> "course-3", "course-4"
+
+```ruby
+class CourseRecommender
+  include Predictor::Base
+
+  input_matrix :users, weight: 3.0
+  input_matrix :tags, weight: 2.0
+  input_matrix :topics, weight: 1.0
+end
+```
+
+Now, we just need to update our matrices when courses are created, users take a course, topics are changed, etc:
+```ruby
+recommender = CourseRecommender.new
+
+# Add a single course to topic-1's items. If topic-1 already exists as a set ID, this just adds course-1 to the set
+recommender.topics.add_single!("topic-1", "course-1")
+
+# If your matrix is quite large, add_single! could take some time, as it must calculate the similarity scores
+# for course-1 across all other courses. If this is the case, use add_single and process the item at a more
+# convenient time, perhaps in a background job
+recommender.topics.add_single("topic-1", "course-1")
+recommender.topics.process_item!("course-1")
+
+# Add an array of courses to tag-1. Again, these will simply be added to tag-1's existing set, if it exists.
+# If not, the tag-1 set will be initialized with course-1 and course-2
+recommender.tags.add_set!("tag-1", ["course-1", "course-2"])
+
+# Or, just add the set and process whenever you like
+recommender.tags.add_set("tag-1", ["course-1", "course-2"])
+["course-1", "course-2"].each { |course| recommender.topics.process_item!(course) }
+```
+
+As noted above, it's important to remember that if you don't use the bang methods (add_set! and add_single!), you'll need to manually update your similarities (the bang methods will likely suffice for most use cases though). You can do so a variety of ways.
+* If you want to simply update the similarities for a single item in a specific matrix:
+  ````
+  recommender.matrix.process_item!(item)
+  ````
+* If you want to update the similarities for all items in a specific matrix:
+  ````
+  recommender.matrix.process!
+  ````
+* If you want to update the similarities for a single item in all matrices:
+  ````
+  recommender.process_item!(item)
+  ````
+* If you want to update all similarities in all matrices:
+  ````
+  recommender.process!
+  ````
+
+Retrieving Similarities and Recommendations
+---------------------
+Now that your matrices have been initialized with several relationships, you can start generating similarities and recommendations! First, let's start with similarities, which will use the weights we specify on each matrix to determine which courses share the most in common with a given course.
+
+![Course Alternative](http://pathgather.github.io/predictor/images/course-alts.png)
+
+```ruby
+recommender = CourseRecommender.new
+
+# Return all similarities for course-1 (ordered by most similar to least).
+recommender.similarities_for("course-1")
+
+# Need to paginate? Not a problem! Specify an offset and a limit
+recommender.similarities_for("course-1", offset: 10, limit: 10) # Gets similarities 11-20
+
+# Want scores?
+recommender.similarities_for("course-1", with_scores: true)
+
+# Want to ignore a certain set of courses in similarities?
+recommender.similarities_for("course-1", exclusion_set: ["course-2"])
+```
+
+The above examples are great for situations like "Users that viewed this also liked ...", but what if you wanted to recommend courses to a user based on the courses they've already taken? Not a problem!
+
+![Course Recommendations](http://pathgather.github.io/predictor/images/suggested.png)
+
+```ruby
+recommender = CourseRecommender.new
+
+# User has taken course-1 and course-2. Let's see what else they might like...
+recommender.predictions_for(item_set: ["course-1", "course-2"])
+
+# Already have the set you need stored in an input matrix? In our case, we do (the users matrix stores the courses a user has taken), so we can just do:
+recommender.predictions_for("user-1", matrix_label: :users)
+
+# Paginate too!
+recommender.predictions_for("user-1", matrix_label: :users, offset: 10, limit: 10)
+
+# Gimme some scores and ignore user-2....that user-2 is one sketchy fella
+recommender.predictions_for("user-1", matrix_label: :users, with_scores: true, exclusion_set: ["user-2"])
+```
+
+Deleting Items
+---------------------
+If your data is deleted from your persistent storage, you certainly don't want to recommend that data to a user. To ensure that doesn't happen, simply call delete_item! on the individual matrix or recommender as a whole:
+```ruby
+recommender = CourseRecommender.new
+
+# User removed course-1 from topic-1, but course-1 still exists
+recommender.topics.delete_item!("course-1")
+
+# course-1 was permanently deleted
+recommender.delete_item!("course-1")
+
+# Something crazy has happened, so let's just start fresh and wipe out all previously stored similarities:
+recommender.clean!
+```
+
+Memory Management
+---------------------
+Predictor works by caching the similarities for each item in each matrix, then computing overall similarities off those caches. With an even semi-large dataset, this can really eat up Redis's memory. To limit the number of similarities cached in each matrix, specify a similarity_limit option when defining the matrix.
+```ruby
+class CourseRecommender
+  include Predictor::Base
+
+  input_matrix :users, weight: 3.0, similarity_limit: 300
+  input_matrix :tags, weight: 2.0, similarity_limit: 300
+  input_matrix :topics, weight: 1.0, similarity_limit: 300
+end
+```
+
+This will ensure that only the top 300 similarities for each item are cached in each matrix. This can greatly reduce your memory usage, and if you're just using Predictor for scenarios where you maybe show the top 5 or so similar items, then this can be hugely helpful. But note, **don't set similarity_limit to 5 in that case**. This simply limits the similarities cached in each matrix, but does not limit the similarities for an item across all matrices. That is computed (and can be limited) on the fly, and uses the similarity cache in each matrix. So, you need a large enough cache in each matrix to determine an intelligent similarity list across all matrices.
+
+*Note*: This is a bit of a hack, and there are most certainly other ways to improve Predictor's memory usage for large datasets, but each appear to require a more significant change than the trivial implementation of similarity_limit above. PRs are quite welcome that experiment with these other ways :)
+
+Oh, and if you decide to tinker with your limit to try and find a sweet spot, I added a helpful method to ensure limits are obeyed to avoid regenerating all similarities. Of course, this only helps if you are decreasing the limit. If you're increasing it, you'll need to process similarities all over.
+```ruby
+recommender.users.ensure_similarity_limit_is_obeyed!  # Remove similarities that disobey our current limit
+recommender.tags.ensure_similarity_limit_is_obeyed!
+recommender.topics.ensure_similarity_limit_is_obeyed!
+```
+
+Problems? Issues? Want to help out?
+---------------------
+Just submit a Gihub issue or pull request! We'd love to have you help out, as the most common library to use for this need, Recommendify, was last updated 2 years ago. We'll be sure to keep this maintained, but we could certainly use your help!
+
+The MIT License (MIT)
+---------------------
+Copyright (c) 2014 Pathgather
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+

data/lib/predictor/base.rb

@@ -9,6 +9,14 @@ module Predictor::Base
       @matrices[key] = opts
     end
 
+    def limit_similarities_to(val)
+      @similarity_limit = val
+    end
+
+    def similarity_limit
+      @similarity_limit
+    end
+
     def input_matrices=(val)
       @matrices = val
     end
@@ -29,6 +37,10 @@ module Predictor::Base
     "predictor"
   end
 
+  def similarity_limit
+    self.class.similarity_limit
+  end
+
   def redis_key(*append)
     ([redis_prefix] + append).flatten.compact.join(":")
   end
@@ -46,70 +58,60 @@ module Predictor::Base
   end
 
   def all_items
-    Predictor.redis.sunion input_matrices.map{|k,m| m.redis_key(:all_items)}
+    Predictor.redis.smembers(redis_key(:all_items))
   end
 
-  def item_score(item, normalize)
-    if normalize
-      similarities = similarities_for(item, with_scores: true)
-      unless similarities.empty?
-        similarities.map{|x,y| y}.reduce(:+)
-      else
-        1
-      end
-    else
-      1
+  def add_to_matrix(matrix, set, *items)
+    items = items.flatten if items.count == 1 && items[0].is_a?(Array)  # Old syntax
+    input_matrices[matrix].add_to_set(set, *items)
+  end
+
+  def add_to_matrix!(matrix, set, *items)
+    items = items.flatten if items.count == 1 && items[0].is_a?(Array)  # Old syntax
+    add_to_matrix(matrix, set, *items)
+    process_items!(*items)
+  end
+
+  def related_items(item)
+    keys = []
+    input_matrices.each do |key, matrix|
+      sets = Predictor.redis.smembers(matrix.redis_key(:sets, item))
+      keys.concat(sets.map { |set| matrix.redis_key(:items, set) })
     end
+
+    keys.empty? ? [] : (Predictor.redis.sunion(keys) - [item])
   end
 
-  def predictions_for(set_id=nil, item_set: nil, matrix_label: nil, with_scores: false, normalize: true, offset: 0, limit: -1, exclusion_set: [])
-    fail "item_set or matrix_label and set_id is required" unless item_set || (matrix_label && set_id)
-    redis = Predictor.redis
+  def predictions_for(set=nil, item_set: nil, matrix_label: nil, with_scores: false, offset: 0, limit: -1, exclusion_set: [])
+    fail "item_set or matrix_label and set is required" unless item_set || (matrix_label && set)
 
     if matrix_label
       matrix = input_matrices[matrix_label]
-      item_set = redis.smembers(matrix.redis_key(:items, set_id))
-    end
-
-    item_keys = item_set.map do |item|
-      input_matrices.map{ |k,m| m.redis_key(:similarities, item) }
-    end.flatten
-
-    item_weights = item_set.map do |item|
-      score = item_score(item, normalize)
-      input_matrices.map{|k, m| m.weight/score }
-    end.flatten
-
-    unless item_keys.empty?
-      predictions = nil
-      redis.multi do |multi|
-        multi.zunionstore 'temp', item_keys, weights: item_weights
-        multi.zrem 'temp', item_set
-        multi.zrem 'temp', exclusion_set if exclusion_set.length > 0
-        predictions = multi.zrevrange 'temp', offset, limit == -1 ? limit : offset + (limit - 1), with_scores: with_scores
-        multi.del 'temp'
-      end
-      return predictions.value
-    else
-      return []
+      item_set = Predictor.redis.smembers(matrix.redis_key(:items, set))
+    end
+
+    item_keys = item_set.map { |item| redis_key(:similarities, item) }
+    return [] if item_keys.empty?
+    predictions = nil
+    Predictor.redis.multi do |multi|
+      multi.zunionstore 'temp', item_keys
+      multi.zrem 'temp', item_set
+      multi.zrem 'temp', exclusion_set if exclusion_set.length > 0
+      predictions = multi.zrevrange 'temp', offset, limit == -1 ? limit : offset + (limit - 1), with_scores: with_scores
+      multi.del 'temp'
     end
+    predictions.value
   end
 
   def similarities_for(item, with_scores: false, offset: 0, limit: -1, exclusion_set: [])
-    keys = input_matrices.map{ |k,m| m.redis_key(:similarities, item) }
-    weights = input_matrices.map{ |k,m| m.weight }
     neighbors = nil
-    unless keys.empty?
-      Predictor.redis.multi do |multi|
-        multi.zunionstore 'temp', keys, weights: weights
-        multi.zrem 'temp', exclusion_set if exclusion_set.length > 0
-        neighbors = multi.zrevrange('temp', offset, limit == -1 ? limit : offset + (limit - 1), with_scores: with_scores)
-        multi.del 'temp'
-      end
-      return neighbors.value
-    else
-      return []
+    Predictor.redis.multi do |multi|
+      multi.zunionstore 'temp', [1, redis_key(:similarities, item)]
+      multi.zrem 'temp', exclusion_set if exclusion_set.length > 0
+      neighbors = multi.zrevrange('temp', offset, limit == -1 ? limit : offset + (limit - 1), with_scores: with_scores)
+      multi.del 'temp'
     end
+    return neighbors.value
   end
 
   def sets_for(item)
@@ -117,32 +119,98 @@ module Predictor::Base
     Predictor.redis.sunion keys
   end
 
-  def process!
-    input_matrices.each do |k,m|
-      m.process!
+  def process_item!(item)
+    process_items!(item)  # Old method
+  end
+
+  def process_items!(*items)
+    items = items.flatten if items.count == 1 && items[0].is_a?(Array)  # Old syntax
+    items.each do |item|
+      related_items(item).each{ |related_item| cache_similarity(item, related_item) }
     end
     return self
   end
 
-  def process_item!(item)
-    input_matrices.each do |k,m|
-      m.process_item!(item)
-    end
+  def process!
+    process_items!(*all_items)
     return self
   end
 
-  def delete_item!(item_id)
+  def delete_from_matrix!(matrix, item)
+    # Deleting from a specific matrix, so get related_items, delete, then update the similarity of those related_items
+    items = related_items(item)
+    input_matrices[matrix].delete_item(item)
+    items.each { |related_item| cache_similarity(item, related_item) }
+    return self
+  end
+
+  def delete_item!(item)
+    Predictor.redis.srem(redis_key(:all_items), item)
+    Predictor.redis.watch(redis_key(:similarities, item)) do
+      items = related_items(item)
+      Predictor.redis.multi do |multi|
+        items.each do |related_item|
+          multi.zrem(redis_key(:similarities, related_item), item)
+        end
+        multi.del redis_key(:similarities, item)
+      end
+    end
+
     input_matrices.each do |k,m|
-      m.delete_item!(item_id)
+      m.delete_item(item)
     end
     return self
   end
 
   def clean!
-    # now only flushes the keys for the instantiated recommender
     keys = Predictor.redis.keys("#{self.redis_prefix}:*")
     unless keys.empty?
       Predictor.redis.del(keys)
     end
   end
+
+  def ensure_similarity_limit_is_obeyed!
+    if similarity_limit
+      items = all_items
+      Predictor.redis.multi do |multi|
+        items.each do |item|
+          multi.zremrangebyrank(redis_key(:similarities, item), 0, -(similarity_limit))
+        end
+      end
+    end
+  end
+
+  private
+
+  def cache_similarity(item1, item2)
+    score = 0
+    input_matrices.each do |key, matrix|
+      score += (matrix.calculate_jaccard(item1, item2) * matrix.weight)
+    end
+    if score > 0
+      add_similarity_if_necessary(item1, item2, score)
+      add_similarity_if_necessary(item2, item1, score)
+    else
+      Predictor.redis.multi do |multi|
+        multi.zrem(redis_key(:similarities, item1), item2)
+        multi.zrem(redis_key(:similarities, item2), item1)
+      end
+    end
+  end
+
+  def add_similarity_if_necessary(item, similarity, score)
+    store = true
+    key = redis_key(:similarities, item)
+    if similarity_limit
+      if Predictor.redis.zrank(key, similarity).nil? && Predictor.redis.zcard(key) >= similarity_limit
+        # Similarity is not already stored and we are at limit of similarities
+        lowest_scored_item = Predictor.redis.zrangebyscore(key, "0", "+inf", limit: [0, 1], with_scores: true)
+        unless lowest_scored_item.empty?
+          # If score is less than or equal to the lowest score, don't store it. Otherwise, make room by removing the lowest scored similarity
+          score <= lowest_scored_item[0][1] ? store = false : Predictor.redis.zrem(key, lowest_scored_item[0][0])
+        end
+      end
+    end
+    Predictor.redis.zadd(key, score, similarity) if store
+  end
 end