predictor 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 23c921a064f6dcb321d1948e051545298616f3b7
+  data.tar.gz: 501a6132f7ea81fa316faf5a5f14a7b0d28afcdb
+SHA512:
+  metadata.gz: 0b14e50f6df801912204a8a312124eba2e44888e8c5b8b1e8f3fe51808b0b3cad2e794b55633fb747cb60189d782312864c49c5105d03de5e1992f240a3528d8
+  data.tar.gz: 850b7d299e0f3ce4352fb3cd3423e96d598f83a72b5a0cf2d0cf27734f25e8951f2834cb40444931820721dce74bb4f029a82081765abb4069fa25d9a31fa04c

data/Gemfile

@@ -0,0 +1,10 @@
+source :rubygems
+
+gem "redis"
+
+group :development do
+  gem "rake"
+  gem "rspec"
+  gem "yard"
+  gem "pry"
+end

data/LICENSE

@@ -0,0 +1,20 @@
+The MIT License (MIT)
+
+Copyright (c) 2014 Pathgather
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,176 @@
+=======
+Predictor
+=========
+
+Fast and efficient recommendations and predictions using Ruby & Redis. Used in production over at [Pathgather](http://pathgather.com) to recommend content to users.
+
+![](https://www.codeship.io/projects/5aeeedf0-6053-0131-2319-5ede98f174ff/status)
+
+Originally forked and based on [Recommendify](https://github.com/paulasmuth/recommendify) by Paul Asmuth, so a huge thanks to him for his contributions to Recommendify. Predictor has been almost completely rewritten to
+* Be much, much more performant and efficient by using Redis for most logic.
+* Provide item similarities such as "Users that read this book also read ..."
+* Provide personalized predictions based on a user's past history, such as "You read these 10 books, so you might also like to read ..."
+
+At the moment, Predictor uses the [Jaccard index](http://en.wikipedia.org/wiki/Jaccard_index) to determine similarities between items. There are other ways to do this, which we intend to implement eventually, but if you want to beat us to the punch, pull requests are quite welcome :)
+
+Installation
+---------------------
+```ruby
+gem install predictor
+````
+or in your Gemfile:
+````
+gem 'predictor'
+```
+Getting Started
+---------------------
+First step is to configure Predictor with your Redis instance.
+```ruby
+# in config/initializers/predictor.rb
+Predictor.redis = Redis.new(:url => ENV["PREDICTOR_REDIS"])
+
+# Or, to improve performance, add hiredis as your driver (you'll need to install the hiredis gem first
+Predictor.redis = Redis.new(:url => ENV["PREDICTOR_REDIS"], :driver => :hiredis)
+```
+Inputting Data
+---------------------
+Create a class and include the Predictor::Base module. Define an input_matrix for each relationship you'd like to keep track of. This can be anything you think is a significant metric for the item: page views, purchases, categories the item belongs to, etc.
+
+Below, we're building a recommender to recommend courses based off of:
+* Users that have taken a course. If 2 courses were taken by the same user, this is 3 times as important to us than if the courses share the same topic. This will lead to sets like:
+  * "user1" -> "course-1", "course-3",
+  * "user2" -> "course-1", "course-4"
+* Tags and their courses. This will lead to sets like:
+  * "rails" -> "course-1", "course-2",
+  * "microeconomics" -> "course-3", "course-4"
+* Topics and their courses. This will lead to sets like:
+  * "computer science" -> "course-1", "course-2",
+  * "economics and finance" -> "course-3", "course-4"
+
+```ruby
+class CourseRecommender
+  include Predictor::Base
+
+  input_matrix :users, weight: 3.0
+  input_matrix :tags, weight: 2.0
+  input_matrix :topics, weight: 1.0
+end
+```
+
+Now, we just need to update our matrices when courses are created, users take a course, topics are changed, etc:
+```ruby
+recommender = CourseRecommender.new
+
+# Add a single course to topic-1's items. If topic-1 already exists as a set ID, this just adds course-1 to the set
+recommender.topics.add_single!("topic-1", "course-1")
+
+# If your matrix is quite large, add_single! could take some time, as it must calculate the similarity scores
+# for course-1 across all other courses. If this is the case, use add_single and process the item at a more
+# convenient time, perhaps in a background job
+recommender.topics.add_single("topic-1", "course-1")
+recommender.topics.process_item!("course-1")
+
+# Add an array of courses to tag-1. Again, these will simply be added to tag-1's existing set, if it exists.
+# If not, the tag-1 set will be initialized with course-1 and course-2
+recommender.tags.add_set!("tag-1", ["course-1", "course-2"])
+
+# Or, just add the set and process whenever you like
+recommender.tags.add_set("tag-1", ["course-1", "course-2"])
+["course-1", "course-2"].each { |course| recommender.topics.process_item!(course) }
+```
+
+As noted above, it's important to remember that if you don't use the bang methods (add_set! and add_single!), you'll need to manually update your similarities (the bang methods will likely suffice for most use cases though). You can do so a variety of ways.
+* If you want to simply update the similarities for a single item in a specific matrix:
+  ````
+  recommender.matrix.process_item!(item)
+  ````
+* If you want to update the similarities for all items in a specific matrix:
+  ````
+  recommender.matrix.process!
+  ````
+* If you want to update the similarities for a single item in all matrices:
+  ````
+  recommender.process_item!(item)
+  ````
+* If you want to update all similarities in all matrices:
+  ````
+  recommender.process!
+  ````
+
+Retrieving Similarities and Recommendations
+---------------------
+Now that your matrices have been initialized with several relationships, you can start generating similarities and recommendations! First, let's start with similarities, which will use the weights we specify on each matrix to determine which courses share the most in common with a given course.
+```ruby
+recommender = CourseRecommender.new
+
+# Return all similarities for course-1 (ordered by most similar to least).
+recommender.similarities_for("course-1")
+
+# Need to paginate? Not a problem! Specify an offset and a limit
+recommender.similarities_for("course-1", offset: 10, limit: 10) # Gets similarities 11-20
+
+# Want scores?
+recommender.similarities_for("course-1", with_scores: true)
+
+# Want to ignore a certain set of courses in similarities?
+recommender.similarities_for("course-1", exclusion_set: ["course-2"])
+```
+
+The above examples are great for situations like "Users that viewed this also liked ...", but what if you wanted to recommend courses to a user based on the courses they've already taken? Not a problem!
+```ruby
+recommender = CourseRecommender.new
+
+# User has taken course-1 and course-2. Let's see what else they might like...
+recommender.predictions_for(item_set: ["course-1", "course-2"])
+
+# Already have the set you need stored in an input matrix? In our case, we do (the users matrix stores the courses a user has taken), so we can just do:
+recommender.predictions_for("user-1", matrix_label: :users)
+
+# Paginate too!
+recommender.predictions_for("user-1", matrix_label: :users, offset: 10, limit: 10)
+
+# Gimme some scores and ignore user-2....that user-2 is one sketchy fella
+recommender.predictions_for("user-1", matrix_label: :users, with_scores: true, exclusion_set: ["user-2"])
+```
+
+Deleting Items
+---------------------
+If your data is deleted from your persistent storage, you certainly don't want to recommend that data to a user. To ensure that doesn't happen, simply call delete_item! on the individual matrix or recommender as a whole:
+```ruby
+recommender = CourseRecommender.new
+
+# User removed course-1 from topic-1, but course-1 still exists
+recommender.topics.delete_item!("course-1")
+
+# course-1 was permanently deleted
+recommender.delete_item!("course-1")
+
+# Something crazy has happened, so let's just start fresh and wipe out all previously stored similarities:
+recommender.clean!
+```
+
+Problems? Issues? Want to help out?
+---------------------
+Just submit a Gihub issue or pull request! We'd love to have you help out, as the most common library to use for this need, Recommendify, was last updated 2 years ago. We'll be sure to keep this maintained, but we could certainly use your help!
+
+The MIT License (MIT)
+---------------------
+Copyright (c) 2014 Pathgather
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+

data/Rakefile

@@ -0,0 +1,12 @@
+require "rubygems"
+require "rspec"
+require 'rspec/core/rake_task'
+require "yard"
+
+desc "Run all examples"
+task RSpec::Core::RakeTask.new('spec')
+
+task :default => "spec"
+
+desc "Generate documentation"
+task YARD::Rake::YardocTask.new

data/lib/predictor.rb

@@ -0,0 +1,4 @@
+require "predictor/version"
+require "predictor/predictor"
+require "predictor/input_matrix"
+require "predictor/base"

data/lib/predictor/base.rb

@@ -0,0 +1,148 @@
+module Predictor::Base
+  def self.included(base)
+    base.extend(ClassMethods)
+  end
+
+  module ClassMethods
+    def input_matrix(key, opts={})
+      @matrices ||= {}
+      @matrices[key] = opts
+    end
+
+    def input_matrices=(val)
+      @matrices = val
+    end
+
+    def input_matrices
+      @matrices
+    end
+  end
+
+  def input_matrices
+    @input_matrices ||= Hash[self.class.input_matrices.map{ |key, opts|
+      opts.merge!(:key => key, :redis_prefix => redis_prefix)
+      [ key, Predictor::InputMatrix.new(opts) ]
+    }]
+  end
+
+  def redis_prefix
+    "predictor"
+  end
+
+  def redis_key(*append)
+    ([redis_prefix] + append).flatten.compact.join(":")
+  end
+
+  def method_missing(method, *args)
+    if input_matrices.has_key?(method)
+      input_matrices[method]
+    else
+      raise NoMethodError.new(method.to_s)
+    end
+  end
+
+  def respond_to?(method)
+    input_matrices.has_key?(method) ? true : super
+  end
+
+  def all_items
+    Predictor.redis.sunion input_matrices.map{|k,m| m.redis_key(:all_items)}
+  end
+
+  def item_score(item, normalize)
+    if normalize
+      similarities = similarities_for(item, with_scores: true)
+      unless similarities.empty?
+        similarities.map{|x,y| y}.reduce(:+)
+      else
+        1
+      end
+    else
+      1
+    end
+  end
+
+  def predictions_for(set_id=nil, item_set: nil, matrix_label: nil, with_scores: false, normalize: true, offset: 0, limit: -1, exclusion_set: [])
+    fail "item_set or matrix_label and set_id is required" unless item_set || (matrix_label && set_id)
+    redis = Predictor.redis
+
+    if matrix_label
+      matrix = input_matrices[matrix_label]
+      item_set = redis.smembers(matrix.redis_key(:items, set_id))
+    end
+
+    item_keys = item_set.map do |item|
+      input_matrices.map{ |k,m| m.redis_key(:similarities, item) }
+    end.flatten
+
+    item_weights = item_set.map do |item|
+      score = item_score(item, normalize)
+      input_matrices.map{|k, m| m.weight/score }
+    end.flatten
+
+    unless item_keys.empty?
+      predictions = nil
+      redis.multi do |multi|
+        multi.zunionstore 'temp', item_keys, weights: item_weights
+        multi.zrem 'temp', item_set
+        multi.zrem 'temp', exclusion_set if exclusion_set.length > 0
+        predictions = multi.zrevrange 'temp', offset, limit == -1 ? limit : offset + (limit - 1), with_scores: with_scores
+        multi.del 'temp'
+      end
+      return predictions.value
+    else
+      return []
+    end
+  end
+
+  def similarities_for(item, with_scores: false, offset: 0, limit: -1, exclusion_set: [])
+    keys = input_matrices.map{ |k,m| m.redis_key(:similarities, item) }
+    weights = input_matrices.map{ |k,m| m.weight }
+    neighbors = nil
+    unless keys.empty?
+      Predictor.redis.multi do |multi|
+        multi.zunionstore 'temp', keys, weights: weights
+        multi.zrem 'temp', exclusion_set if exclusion_set.length > 0
+        neighbors = multi.zrevrange('temp', offset, limit == -1 ? limit : offset + (limit - 1), with_scores: with_scores)
+        multi.del 'temp'
+      end
+      return neighbors.value
+    else
+      return []
+    end
+  end
+
+  def sets_for(item)
+    keys = input_matrices.map{ |k,m| m.redis_key(:sets, item) }
+    Predictor.redis.sunion keys
+  end
+
+  def process!
+    input_matrices.each do |k,m|
+      m.process!
+    end
+    return self
+  end
+
+  def process_item!(item)
+    input_matrices.each do |k,m|
+      m.process_item!(item)
+    end
+    return self
+  end
+
+  def delete_item!(item_id)
+    input_matrices.each do |k,m|
+      m.delete_item!(item_id)
+    end
+    return self
+  end
+
+  def clean!
+    # now only flushes the keys for the instantiated recommender
+    keys = Predictor.redis.keys("#{self.redis_prefix}:*")
+    unless keys.empty?
+      Predictor.redis.del(keys)
+    end
+  end
+end

data/lib/predictor/input_matrix.rb

@@ -0,0 +1,138 @@
+class Predictor::InputMatrix
+  def initialize(opts)
+    @opts = opts
+  end
+
+  def redis_key(*append)
+    ([@opts.fetch(:redis_prefix), @opts.fetch(:key)] + append).flatten.compact.join(":")
+  end
+
+  def weight
+    (@opts[:weight] || 1).to_f
+  end
+
+  def add_set(set_id, item_ids)
+    Predictor.redis.multi do
+      item_ids.each { |item| add_single_nomulti(set_id, item) }
+    end
+  end
+
+  def add_set!(set_id, item_ids)
+    add_set(set_id, item_ids)
+    item_ids.each { |item_id| process_item!(item_id) }
+  end
+
+  def add_single(set_id, item_id)
+    Predictor.redis.multi do
+      add_single_nomulti(set_id, item_id)
+    end
+  end
+
+  def add_single!(set_id, item_id)
+    add_single(set_id, item_id)
+    process_item!(item_id)
+  end
+
+  def all_items
+    Predictor.redis.smembers(redis_key(:all_items))
+  end
+
+  def items_for(set)
+    Predictor.redis.smembers redis_key(:items, set)
+  end
+
+  def sets_for(item)
+    Predictor.redis.sunion redis_key(:sets, item)
+  end
+
+  def related_items(item_id)
+    sets = Predictor.redis.smembers(redis_key(:sets, item_id))
+    keys = sets.map { |set| redis_key(:items, set) }
+    if keys.length > 0
+      Predictor.redis.sunion(keys) - [item_id]
+    else
+      []
+    end
+  end
+
+  def similarity(item1, item2)
+    Predictor.redis.zscore(redis_key(:similarities, item1), item2)
+  end
+
+  # calculate all similarities to other items in the matrix for item1
+  def similarities_for(item1, with_scores: false, offset: 0, limit: -1)
+    Predictor.redis.zrevrange(redis_key(:similarities, item1), offset, limit == -1 ? limit : offset + (limit - 1), with_scores: with_scores)
+  end
+
+  def process_item!(item)
+    cache_similarities_for(item)
+  end
+
+  def process!
+    all_items.each do |item|
+      process_item!(item)
+    end
+  end
+
+  # delete item_id from the matrix
+  def delete_item!(item_id)
+    Predictor.redis.srem(redis_key(:all_items), item_id)
+    Predictor.redis.watch(redis_key(:sets, item_id), redis_key(:similarities, item_id)) do
+      sets = Predictor.redis.smembers(redis_key(:sets, item_id))
+      items = Predictor.redis.zrange(redis_key(:similarities, item_id), 0, -1)
+      Predictor.redis.multi do |multi|
+        sets.each do |set|
+          multi.srem(redis_key(:items, set), item_id)
+        end
+
+        items.each do |item|
+          multi.zrem(redis_key(:similarities, item), item_id)
+        end
+
+        multi.del redis_key(:sets, item_id), redis_key(:similarities, item_id)
+      end
+    end
+  end
+
+  private
+
+  def add_single_nomulti(set_id, item_id)
+    Predictor.redis.sadd(redis_key(:all_items), item_id)
+    Predictor.redis.sadd(redis_key(:items, set_id), item_id)
+    # add the set_id to the item_id's set--inverting the sets
+    Predictor.redis.sadd(redis_key(:sets, item_id), set_id)
+  end
+
+  def cache_similarity(item1, item2)
+    score = calculate_jaccard(item1, item2)
+
+    if score > 0
+      Predictor.redis.multi do |multi|
+        multi.zadd(redis_key(:similarities, item1), score, item2)
+        multi.zadd(redis_key(:similarities, item2), score, item1)
+      end
+    end
+  end
+
+  def cache_similarities_for(item)
+    related_items(item).each do |related_item|
+      cache_similarity(item, related_item)
+    end
+  end
+
+  def calculate_jaccard(item1, item2)
+    x = nil
+    y = nil
+    Predictor.redis.multi do |multi|
+      x = multi.sinterstore 'temp', [redis_key(:sets, item1), redis_key(:sets, item2)]
+      y = multi.sunionstore 'temp', [redis_key(:sets, item1), redis_key(:sets, item2)]
+      multi.del 'temp'
+    end
+
+    if y.value > 0
+      return (x.value.to_f/y.value.to_f)
+    else
+      return 0.0
+    end
+  end
+end

data/lib/predictor/predictor.rb

@@ -0,0 +1,21 @@
+module Predictor
+  @@redis = nil
+
+  def self.redis=(redis)
+    @@redis = redis
+  end
+
+  def self.redis
+    return @@redis unless @@redis.nil?
+    raise "redis not configured! - Predictor.redis = Redis.new"
+  end
+
+  def self.capitalize(str_or_sym)
+  	str = str_or_sym.to_s.each_char.to_a
+  	str.first.upcase + str[1..-1].join("").downcase
+  end
+
+  def self.constantize(klass)
+    Object.module_eval("Predictor::#{klass}", __FILE__, __LINE__)
+  end
+end

data/lib/predictor/version.rb

@@ -0,0 +1,3 @@
+module Predictor
+  VERSION = "1.0.0"
+end

data/predictor.gemspec

@@ -0,0 +1,21 @@
+# -*- encoding: utf-8 -*-
+require File.expand_path('../lib/predictor/version', __FILE__)
+
+Gem::Specification.new do |s|
+  s.name        = "predictor"
+  s.version     = Predictor::VERSION
+  s.platform    = Gem::Platform::RUBY
+  s.authors     = ["Pathgather"]
+  s.email       = ["tech@pathgather.com"]
+  s.homepage    = "https://github.com/Pathgather/predictor"
+  s.description = s.summary = "Fast and efficient recommendations and predictions using Redis"
+  s.licenses    = ["MIT"]
+
+  s.add_dependency "redis", ">= 3.0.0"
+
+  s.add_development_dependency "rspec", "~> 2.8.0"
+
+  s.files         = `git ls-files`.split("\n") - [".gitignore", ".rspec", ".travis.yml"]
+  s.test_files    = `git ls-files -- spec/*`.split("\n")
+  s.require_paths = ["lib"]
+end

data/spec/base_spec.rb

@@ -0,0 +1,224 @@
+require ::File.expand_path('../spec_helper', __FILE__)
+
+describe Predictor::Base do
+  class BaseRecommender
+    include Predictor::Base
+  end
+
+  before(:each) do
+    flush_redis!
+    BaseRecommender.input_matrices = {}
+  end
+
+  describe "configuration" do
+    it "should add an input_matrix by 'key'" do
+      BaseRecommender.input_matrix(:myinput)
+      BaseRecommender.input_matrices.keys.should == [:myinput]
+    end
+
+    it "should retrieve an input_matrix on a new instance" do
+      BaseRecommender.input_matrix(:myinput)
+      sm = BaseRecommender.new
+      lambda{ sm.myinput }.should_not raise_error
+    end
+
+    it "should retrieve an input_matrix on a new instance and correctly overload respond_to?" do
+      BaseRecommender.input_matrix(:myinput)
+      sm = BaseRecommender.new
+      sm.respond_to?(:process!).should be_true
+      sm.respond_to?(:myinput).should be_true
+      sm.respond_to?(:fnord).should be_false
+    end
+
+    it "should retrieve an input_matrix on a new instance and intialize the correct class" do
+      BaseRecommender.input_matrix(:myinput)
+      sm = BaseRecommender.new
+      sm.myinput.should be_a(Predictor::InputMatrix)
+    end
+  end
+
+  describe "process_item!" do
+    it "should call process_item! on each input_matrix" do
+      BaseRecommender.input_matrix(:myfirstinput)
+      BaseRecommender.input_matrix(:mysecondinput)
+      sm = BaseRecommender.new
+      sm.myfirstinput.should_receive(:process_item!).with("fnorditem").and_return([["fooitem",0.5]])
+      sm.mysecondinput.should_receive(:process_item!).with("fnorditem").and_return([["fooitem",0.5]])
+      sm.process_item!("fnorditem")
+    end
+
+    it "should call process_item! on each input_matrix and add all outputs to the similarity matrix" do
+      BaseRecommender.input_matrix(:myfirstinput)
+      BaseRecommender.input_matrix(:mysecondinput)
+      sm = BaseRecommender.new
+      sm.myfirstinput.should_receive(:process_item!).and_return([["fooitem",0.5]])
+      sm.mysecondinput.should_receive(:process_item!).and_return([["fooitem",0.75], ["baritem", 1.0]])
+      sm.process_item!("fnorditem")
+    end
+
+    it "should call process_item! on each input_matrix and add all outputs to the similarity matrix with weight" do
+      BaseRecommender.input_matrix(:myfirstinput, :weight => 4.0)
+      BaseRecommender.input_matrix(:mysecondinput)
+      sm = BaseRecommender.new
+      sm.myfirstinput.should_receive(:process_item!).and_return([["fooitem",0.5]])
+      sm.mysecondinput.should_receive(:process_item!).and_return([["fooitem",0.75], ["baritem", 1.0]])
+      sm.process_item!("fnorditem")
+    end
+  end
+
+  describe "all_items" do
+    it "should retrieve all items from all input matrices" do
+      BaseRecommender.input_matrix(:anotherinput)
+      BaseRecommender.input_matrix(:yetanotherinput)
+      sm = BaseRecommender.new
+      sm.anotherinput.add_set('a', ["foo", "bar"])
+      sm.yetanotherinput.add_set('b', ["fnord", "shmoo"])
+      sm.all_items.length.should == 4
+      sm.all_items.should include("foo", "bar", "fnord", "shmoo")
+    end
+
+    it "should retrieve all items from all input matrices (uniquely)" do
+      BaseRecommender.input_matrix(:anotherinput)
+      BaseRecommender.input_matrix(:yetanotherinput)
+      sm = BaseRecommender.new
+      sm.anotherinput.add_set('a', ["foo", "bar"])
+      sm.yetanotherinput.add_set('b', ["fnord", "bar"])
+      sm.all_items.length.should == 3
+      sm.all_items.should include("foo", "bar", "fnord")
+    end
+  end
+
+  describe "process!" do
+    it "should call process_item for all input_matrix.all_items's" do
+      BaseRecommender.input_matrix(:anotherinput)
+      BaseRecommender.input_matrix(:yetanotherinput)
+      sm = BaseRecommender.new
+      sm.anotherinput.add_set('a', ["foo", "bar"])
+      sm.yetanotherinput.add_set('b', ["fnord", "shmoo"])
+      sm.anotherinput.should_receive(:process!).exactly(1).times
+      sm.yetanotherinput.should_receive(:process!).exactly(1).times
+      sm.process!
+    end
+  end
+
+  describe "predictions_for" do
+    it "returns relevant predictions" do
+      BaseRecommender.input_matrix(:users, weight: 4.0)
+      BaseRecommender.input_matrix(:tags, weight: 1.0)
+      sm = BaseRecommender.new
+      sm.users.add_set('me', ["foo", "bar", "fnord"])
+      sm.users.add_set('not_me', ["foo", "shmoo"])
+      sm.users.add_set('another', ["fnord", "other"])
+      sm.users.add_set('another', ["nada"])
+      sm.tags.add_set('tag1', ["foo", "fnord", "shmoo"])
+      sm.tags.add_set('tag2', ["bar", "shmoo"])
+      sm.tags.add_set('tag3', ["shmoo", "nada"])
+      sm.process!
+      predictions = sm.predictions_for('me', matrix_label: :users)
+      predictions.should == ["shmoo", "other", "nada"]
+      predictions = sm.predictions_for(item_set: ["foo", "bar", "fnord"])
+      predictions.should == ["shmoo", "other", "nada"]
+      predictions = sm.predictions_for('me', matrix_label: :users, offset: 1, limit: 1)
+      predictions.should == ["other"]
+      predictions = sm.predictions_for('me', matrix_label: :users, offset: 1)
+      predictions.should == ["other", "nada"]
+    end
+
+    it "correctly normalizes predictions" do
+      BaseRecommender.input_matrix(:users, weight: 1.0)
+      BaseRecommender.input_matrix(:tags, weight: 2.0)
+      BaseRecommender.input_matrix(:topics, weight: 4.0)
+
+      sm = BaseRecommender.new
+
+      sm.users.add_set('user1', ["c1", "c2", "c4"])
+      sm.users.add_set('user2', ["c3", "c4"])
+      sm.topics.add_set('topic1', ["c1", "c4"])
+      sm.topics.add_set('topic2', ["c2", "c3"])
+      sm.tags.add_set('tag1', ["c1", "c2", "c4"])
+      sm.tags.add_set('tag2', ["c1", "c4"])
+
+      sm.process!
+
+      predictions = sm.predictions_for('user1', matrix_label: :users, with_scores: true, normalize: false)
+      predictions.should eq([["c3", 4.5]])
+      predictions = sm.predictions_for('user2',  matrix_label: :users, with_scores: true, normalize: false)
+      predictions.should eq([["c1", 6.5], ["c2", 5.5]])
+      predictions = sm.predictions_for('user1', matrix_label: :users, with_scores: true, normalize: true)
+      predictions[0][0].should eq("c3")
+      predictions[0][1].should be_within(0.001).of(0.592)
+      predictions = sm.predictions_for('user2', matrix_label: :users, with_scores: true, normalize: true)
+      predictions[0][0].should eq("c2")
+      predictions[0][1].should be_within(0.001).of(1.065)
+      predictions[1][0].should eq("c1")
+      predictions[1][1].should be_within(0.001).of(0.764)
+    end
+  end
+
+  describe "similarities_for(item_id)" do
+    it "should not throw exception for non existing items" do
+      sm = BaseRecommender.new
+      sm.similarities_for("not_existing_item").length.should == 0
+    end
+
+    it "correctly weighs and sums input matrices" do
+      BaseRecommender.input_matrix(:users, weight: 1.0)
+      BaseRecommender.input_matrix(:tags, weight: 2.0)
+      BaseRecommender.input_matrix(:topics, weight: 4.0)
+
+      sm = BaseRecommender.new
+
+      sm.users.add_set('user1', ["c1", "c2", "c4"])
+      sm.users.add_set('user2', ["c3", "c4"])
+      sm.topics.add_set('topic1', ["c1", "c4"])
+      sm.topics.add_set('topic2', ["c2", "c3"])
+      sm.tags.add_set('tag1', ["c1", "c2", "c4"])
+      sm.tags.add_set('tag2', ["c1", "c4"])
+
+      sm.process!
+      sm.similarities_for("c1", with_scores: true).should eq([["c4", 6.5], ["c2", 2.0]])
+      sm.similarities_for("c2", with_scores: true).should eq([["c3", 4.0], ["c1", 2.0], ["c4", 1.5]])
+      sm.similarities_for("c3", with_scores: true).should eq([["c2", 4.0], ["c4", 0.5]])
+      sm.similarities_for("c4", with_scores: true, exclusion_set: ["c3"]).should eq([["c1", 6.5], ["c2", 1.5]])
+    end
+  end
+
+  describe "sets_for" do
+    it "should return all the sets the given item is in" do
+      BaseRecommender.input_matrix(:set1)
+      BaseRecommender.input_matrix(:set2)
+      sm = BaseRecommender.new
+      sm.set1.add_set "item1", ["foo", "bar"]
+      sm.set1.add_set "item2", ["nada", "bar"]
+      sm.set2.add_set "item3", ["bar", "other"]
+      sm.sets_for("bar").length.should == 3
+      sm.sets_for("bar").should include("item1", "item2", "item3")
+      sm.sets_for("other").should == ["item3"]
+    end
+  end
+
+  describe "delete_item!" do
+    it "should call delete_item on each input_matrix" do
+      BaseRecommender.input_matrix(:myfirstinput)
+      BaseRecommender.input_matrix(:mysecondinput)
+      sm = BaseRecommender.new
+      sm.myfirstinput.should_receive(:delete_item!).with("fnorditem")
+      sm.mysecondinput.should_receive(:delete_item!).with("fnorditem")
+      sm.delete_item!("fnorditem")
+    end
+  end
+
+  describe "clean!" do
+    it "should clean out the Redis storage for this Predictor" do
+      BaseRecommender.input_matrix(:set1)
+      BaseRecommender.input_matrix(:set2)
+      sm = BaseRecommender.new
+      sm.set1.add_set "item1", ["foo", "bar"]
+      sm.set1.add_set "item2", ["nada", "bar"]
+      sm.set2.add_set "item3", ["bar", "other"]
+      Predictor.redis.keys("#{sm.redis_prefix}:*").should_not be_empty
+      sm.clean!
+      Predictor.redis.keys("#{sm.redis_prefix}:*").should be_empty
+    end
+  end
+end

data/spec/input_matrix_spec.rb

@@ -0,0 +1,251 @@
+require ::File.expand_path('../spec_helper', __FILE__)
+
+describe Predictor::InputMatrix do
+
+  before(:all) do
+    @matrix = Predictor::InputMatrix.new(:redis_prefix => "predictor-test", :key => "mymatrix")
+  end
+
+  before(:each) do
+    flush_redis!
+  end
+
+  it "should build the correct keys" do
+    @matrix.redis_key.should == "predictor-test:mymatrix"
+  end
+
+  it "should respond to add_set" do
+    @matrix.respond_to?(:add_set).should == true
+  end
+
+  it "should respond to add_single" do
+    @matrix.respond_to?(:add_single).should == true
+  end
+
+  it "should respond to similarities_for" do
+    @matrix.respond_to?(:similarities_for).should == true
+  end
+
+  it "should respond to all_items" do
+    @matrix.respond_to?(:all_items).should == true
+  end
+
+  describe "weight" do
+    it "returns the weight configured or a default of 1" do
+      @matrix.weight.should == 1.0  # default weight
+      matrix = Predictor::InputMatrix.new(redis_prefix: "predictor-test", key: "mymatrix", weight: 5.0)
+      matrix.weight.should == 5.0
+    end
+  end
+
+  describe "add_set" do
+    it "adds each member of the set to the 'all_items' set" do
+      @matrix.all_items.should_not include("foo", "bar", "fnord", "blubb")
+      @matrix.add_set "item1", ["foo", "bar", "fnord", "blubb"]
+      @matrix.all_items.should include("foo", "bar", "fnord", "blubb")
+    end
+
+    it "adds each member of the set to the key's 'sets' set" do
+      @matrix.items_for("item1").should_not include("foo", "bar", "fnord", "blubb")
+      @matrix.add_set "item1", ["foo", "bar", "fnord", "blubb"]
+      @matrix.items_for("item1").should include("foo", "bar", "fnord", "blubb")
+    end
+
+    it "adds the key to each set member's 'items' set" do
+      @matrix.sets_for("foo").should_not include("item1")
+      @matrix.sets_for("bar").should_not include("item1")
+      @matrix.sets_for("fnord").should_not include("item1")
+      @matrix.sets_for("blubb").should_not include("item1")
+      @matrix.add_set "item1", ["foo", "bar", "fnord", "blubb"]
+      @matrix.sets_for("foo").should include("item1")
+      @matrix.sets_for("bar").should include("item1")
+      @matrix.sets_for("fnord").should include("item1")
+      @matrix.sets_for("blubb").should include("item1")
+    end
+  end
+
+  describe "add_set!" do
+    it "calls add_set and process_item! for each item" do
+      @matrix.should_receive(:add_set).with("item1", ["foo", "bar"])
+      @matrix.should_receive(:process_item!).with("foo")
+      @matrix.should_receive(:process_item!).with("bar")
+      @matrix.add_set! "item1", ["foo", "bar"]
+    end
+  end
+
+  describe "add_single" do
+    it "adds the item to the 'all_items' set" do
+      @matrix.all_items.should_not include("foo")
+      @matrix.add_single "item1", "foo"
+      @matrix.all_items.should include("foo")
+    end
+
+    it "adds the item to the key's 'sets' set" do
+      @matrix.items_for("item1").should_not include("foo")
+      @matrix.add_single "item1", "foo"
+      @matrix.items_for("item1").should include("foo")
+    end
+
+    it "adds the key to the item's 'items' set" do
+      @matrix.sets_for("foo").should_not include("item1")
+      @matrix.add_single "item1", "foo"
+      @matrix.sets_for("foo").should include("item1")
+    end
+  end
+
+  describe "add_single!" do
+    it "calls add_single and process_item! for the item" do
+      @matrix.should_receive(:add_single).with("item1", "foo")
+      @matrix.should_receive(:process_item!).with("foo")
+      @matrix.add_single! "item1", "foo"
+    end
+  end
+
+  describe "all_items" do
+    it "returns all items across all sets in the input matrix" do
+      @matrix.add_set "item1", ["foo", "bar", "fnord", "blubb"]
+      @matrix.add_set "item2", ["foo", "bar", "snafu", "nada"]
+      @matrix.add_set "item3", ["nada"]
+      @matrix.all_items.should include("foo", "bar", "fnord", "blubb", "snafu", "nada")
+      @matrix.all_items.length.should == 6
+    end
+  end
+
+  describe "items_for" do
+    it "returns the items in the given set ID" do
+      @matrix.add_set "item1", ["foo", "bar", "fnord", "blubb"]
+      @matrix.items_for("item1").should include("foo", "bar", "fnord", "blubb")
+      @matrix.add_set "item2", ["foo", "bar", "snafu", "nada"]
+      @matrix.items_for("item2").should include("foo", "bar", "snafu", "nada")
+      @matrix.items_for("item1").should_not include("snafu", "nada")
+    end
+  end
+
+  describe "sets_for" do
+    it "returns the set IDs the given item is in" do
+      @matrix.add_set "item1", ["foo", "bar", "fnord", "blubb"]
+      @matrix.add_set "item2", ["foo", "bar", "snafu", "nada"]
+      @matrix.sets_for("foo").should include("item1", "item2")
+      @matrix.sets_for("snafu").should == ["item2"]
+    end
+  end
+
+  describe "related_items" do
+    it "returns the items in sets the given item is also in" do
+      @matrix.add_set "item1", ["foo", "bar", "fnord", "blubb"]
+      @matrix.add_set "item2", ["foo", "bar", "snafu", "nada"]
+      @matrix.add_set "item3", ["nada", "other"]
+      @matrix.related_items("bar").should include("foo", "fnord", "blubb", "snafu", "nada")
+      @matrix.related_items("bar").length.should == 5
+      @matrix.related_items("other").should == ["nada"]
+      @matrix.related_items("snafu").should include("foo", "bar", "nada")
+      @matrix.related_items("snafu").length.should == 3
+    end
+  end
+
+  describe "similarity" do
+    it "should calculate the correct similarity between two items" do
+      add_two_item_test_data!(@matrix)
+      @matrix.process!
+      @matrix.similarity("fnord", "blubb").should == 0.4
+      @matrix.similarity("blubb", "fnord").should == 0.4
+    end
+  end
+
+  describe "similarities_for" do
+    it "should calculate all similarities for an item (1/3)" do
+      add_three_item_test_data!(@matrix)
+      @matrix.process!
+      res = @matrix.similarities_for("fnord", with_scores: true)
+      res.length.should == 2
+      res[0].should == ["shmoo", 0.75]
+      res[1].should == ["blubb", 0.4]
+    end
+
+    it "should calculate all similarities for an item (2/3)" do
+      add_three_item_test_data!(@matrix)
+      @matrix.process!
+      res = @matrix.similarities_for("shmoo", with_scores: true)
+      res.length.should == 2
+      res[0].should == ["fnord", 0.75]
+      res[1].should == ["blubb", 0.2]
+    end
+
+
+    it "should calculate all similarities for an item (3/3)" do
+      add_three_item_test_data!(@matrix)
+      @matrix.process!
+      res = @matrix.similarities_for("blubb", with_scores: true)
+      res.length.should == 2
+      res[0].should == ["fnord", 0.4]
+      res[1].should == ["shmoo", 0.2]
+    end
+  end
+
+  describe "delete_item!" do
+    before do
+      @matrix.add_set "item1", ["foo", "bar", "fnord", "blubb"]
+      @matrix.add_set "item2", ["foo", "bar", "snafu", "nada"]
+      @matrix.add_set "item3", ["nada", "other"]
+      @matrix.process!
+    end
+
+    it "should delete the item from sets it is in" do
+      @matrix.items_for("item1").should include("bar")
+      @matrix.items_for("item2").should include("bar")
+      @matrix.sets_for("bar").should include("item1", "item2")
+      @matrix.delete_item!("bar")
+      @matrix.items_for("item1").should_not include("bar")
+      @matrix.items_for("item2").should_not include("bar")
+      @matrix.sets_for("bar").should be_empty
+    end
+
+    it "should delete the cached similarities for the item" do
+      @matrix.similarities_for("bar").should_not be_empty
+      @matrix.delete_item!("bar")
+      @matrix.similarities_for("bar").should be_empty
+    end
+
+    it "should delete the item from other cached similarities" do
+      @matrix.similarities_for("foo").should include("bar")
+      @matrix.delete_item!("bar")
+      @matrix.similarities_for("foo").should_not include("bar")
+    end
+
+    it "should delete the item from the all_items set" do
+      @matrix.all_items.should include("bar")
+      @matrix.delete_item!("bar")
+      @matrix.all_items.should_not include("bar")
+    end
+  end
+
+  it "should calculate the correct jaccard index" do
+    @matrix.add_set "item1", ["foo", "bar", "fnord", "blubb"]
+    @matrix.add_set "item2", ["bar", "fnord", "shmoo", "snafu"]
+    @matrix.add_set "item3", ["bar", "nada", "snafu"]
+
+    @matrix.send(:calculate_jaccard,
+      "bar",
+      "snafu"
+    ).should == 2.0/3.0
+  end
+
+private
+
+  def add_two_item_test_data!(matrix)
+    matrix.add_set("user42", ["fnord", "blubb"])
+    matrix.add_set("user44", ["blubb"])
+    matrix.add_set("user46", ["fnord"])
+    matrix.add_set("user48", ["fnord", "blubb"])
+    matrix.add_set("user50", ["fnord"])
+  end
+
+  def add_three_item_test_data!(matrix)
+    matrix.add_set("user42", ["fnord", "blubb", "shmoo"])
+    matrix.add_set("user44", ["blubb"])
+    matrix.add_set("user46", ["fnord", "shmoo"])
+    matrix.add_set("user48", ["fnord", "blubb"])
+    matrix.add_set("user50", ["fnord", "shmoo"])
+  end
+
+end

data/spec/predictor_spec.rb

@@ -0,0 +1,15 @@
+require ::File.expand_path('../spec_helper', __FILE__)
+
+describe Predictor do
+
+  it "should store a redis connection" do
+    Predictor.redis = "asd"
+    Predictor.redis.should == "asd"
+  end
+
+  it "should raise an exception if unconfigured redis connection is accessed" do
+    Predictor.redis = nil
+    lambda{ ecommendify.redis }.should raise_error
+  end
+
+end

data/spec/spec_helper.rb

@@ -0,0 +1,40 @@
+require "rspec"
+require "redis"
+require "pry"
+
+require ::File.expand_path('../../lib/predictor', __FILE__)
+
+def flush_redis!
+  Predictor.redis = Redis.new
+  Predictor.redis.keys("predictor-test*").each do |k|
+    Predictor.redis.del(k)
+  end
+end
+
+module Predictor::Base
+
+  def redis_prefix
+    "predictor-test"
+  end
+
+end
+
+
+class TestRecommender
+  include Predictor::Base
+
+  input_matrix :jaccard_one
+
+end
+
+class Predictor::TestInputMatrix
+
+  def initialize(opts)
+  	@opts = opts
+  end
+
+  def method_missing(method, *args)
+  	@opts[method]
+  end
+
+end

metadata

@@ -0,0 +1,91 @@
+--- !ruby/object:Gem::Specification
+name: predictor
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- Pathgather
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2014-01-17 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: redis
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 2.8.0
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 2.8.0
+description: Fast and efficient recommendations and predictions using Redis
+email:
+- tech@pathgather.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- Gemfile
+- LICENSE
+- README.md
+- Rakefile
+- lib/predictor.rb
+- lib/predictor/base.rb
+- lib/predictor/input_matrix.rb
+- lib/predictor/predictor.rb
+- lib/predictor/version.rb
+- predictor.gemspec
+- spec/base_spec.rb
+- spec/input_matrix_spec.rb
+- spec/predictor_spec.rb
+- spec/spec_helper.rb
+homepage: https://github.com/Pathgather/predictor
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.1.11
+signing_key: 
+specification_version: 4
+summary: Fast and efficient recommendations and predictions using Redis
+test_files:
+- spec/base_spec.rb
+- spec/input_matrix_spec.rb
+- spec/predictor_spec.rb
+- spec/spec_helper.rb
+has_rdoc: