disco 0.1.1 → 0.2.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e16ea9f41bc910a0966c4f16f0e48df98f40abb70d2d3b3bd2e8ba2080e57599
-  data.tar.gz: '05490bda394fa0edf02ab33cb16cc65093d5b66b6317671cbdbfc429c3bd196c'
+  metadata.gz: e9b8792d465e2bd894ce9aaa5dabf79dd89e93337d838917c709ac7747b85772
+  data.tar.gz: 9d34a5124dc26f8a2ecb7e2ed3cbf524fe586c37c693d885e668974e24dfaf0a
 SHA512:
-  metadata.gz: b12681372323e4bc323915923f91b5883ea97161e2f2a2846548657b25932ddc7ce364e09a6b932506e15e3cb45ff03c8c2cc22022f3a2cfa62d64bbe77e988d
-  data.tar.gz: ea2d4200bfb4d4aed3481ebda8de4133c7af5bb4c4bb7c65685a726bbf661b2285579a507ea464ed49b3609d5350b45404dd6cf99401664e013870a6dee54b7f
+  metadata.gz: 658b48b75994a295382eb22908d4a5f1825b01bfc26f52428e993802c42f7ebb435a59e7f1262d17400d65eda886f1a4f38edff82cdeda96c2a0ce280602742f
+  data.tar.gz: c9acce77cae8a575c5814456247600367d5fea4eb85a52309e26cac643d107bc03c42c99ce094c2f9ec46a329fd2dfa97d2f58f0020b337feb88b56346630942

data/CHANGELOG.md

@@ -1,3 +1,32 @@
+## 0.2.3 (2020-11-28)
+
+- Added `predict` method
+- Fixed bad recommendations and scores with `user_recs` and explicit feedback
+- Fixed `item_ids` option for `user_recs`
+
+## 0.2.2 (n/a)
+
+- Not available (released by previous gem owner)
+
+## 0.2.1 (2020-10-28)
+
+- Fixed issue with `user_recs` returning rated items
+
+## 0.2.0 (2020-07-31)
+
+- Changed score to always be between -1 and 1 for `item_recs` and `similar_users` (cosine similarity - this makes it easier to understand and consistent with `optimize_item_recs` and `optimize_similar_users`)
+
+## 0.1.3 (2020-06-28)
+
+- Added support for Rover
+- Raise error when missing user or item ids
+- Fixed string keys for Daru data frames
+- `optimize_item_recs` and `optimize_similar_users` methods are no longer experimental
+
+## 0.1.2 (2020-03-26)
+
+- Added experimental `optimize_item_recs` and `optimize_similar_users` methods
+
 ## 0.1.1 (2019-11-14)
 
 - Fixed Rails integration

data/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright (c) 2019 Andrew Kane
+Copyright (c) 2019-2020 Andrew Kane
 
 MIT License
 

data/README.md

@@ -1,12 +1,12 @@
 # Disco
 
-:fire: Collaborative filtering for Ruby
+:fire: Recommendations for Ruby and Rails using collaborative filtering
 
 - Supports user-based and item-based recommendations
 - Works with explicit and implicit feedback
-- Uses matrix factorization
+- Uses high-performance matrix factorization
 
-[![Build Status](https://travis-ci.org/ankane/disco.svg?branch=master)](https://travis-ci.org/ankane/disco)
+[![Build Status](https://github.com/ankane/disco/workflows/build/badge.svg?branch=master)](https://github.com/ankane/disco/actions)
 
 ## Installation
 
@@ -64,10 +64,10 @@ Use the `count` option to specify the number of recommendations (default is 5)
 recommender.user_recs(user_id, count: 3)
 ```
 
-Get predicted ratings for specific items
+Get predicted ratings for specific users and items
 
 ```ruby
-recommender.user_recs(user_id, item_ids: [1, 2, 3])
+recommender.predict([{user_id: 1, item_id: 2}, {user_id: 2, item_id: 4}])
 ```
 
 Get similar users
@@ -101,14 +101,15 @@ recommender.item_recs("Star Wars (1977)")
 ```ruby
 views = Ahoy::Event.
   where(name: "Viewed post").
-  group(:user_id, "properties->>'post_id'") # postgres syntax
+  group(:user_id).
+  group("properties->>'post_id'"). # postgres syntax
   count
 
 data =
   views.map do |(user_id, post_id), count|
     {
       user_id: user_id,
-      post_id: post_id,
+      item_id: post_id,
       value: count
     }
   end
@@ -202,7 +203,7 @@ recommender = Marshal.load(bin)
 
 ## Algorithms
 
-Disco uses matrix factorization.
+Disco uses high-performance matrix factorization.
 
 - For explicit feedback, it uses [stochastic gradient descent](https://www.csie.ntu.edu.tw/~cjlin/papers/libmf/libmf_journal.pdf)
 - For implicit feedback, it uses [coordinate descent](https://www.csie.ntu.edu.tw/~cjlin/papers/one-class-mf/biased-mf-sdm-with-supp.pdf)
@@ -236,15 +237,50 @@ There are a number of ways to deal with this, but here are some common ones:
 - For user-based recommendations, show new users the most popular items.
 - For item-based recommendations, make content-based recommendations with a gem like [tf-idf-similarity](https://github.com/jpmckinney/tf-idf-similarity).
 
-## Daru
+## Data
 
-Disco works with Daru data frames
+Data can be an array of hashes
 
 ```ruby
-data = Daru::DataFrame.from_csv("ratings.csv")
-recommender.fit(data)
+[{user_id: 1, item_id: 1, rating: 5}, {user_id: 2, item_id: 1, rating: 3}]
+```
+
+Or a Rover data frame
+
+```ruby
+Rover.read_csv("ratings.csv")
 ```
 
+Or a Daru data frame
+
+```ruby
+Daru::DataFrame.from_csv("ratings.csv")
+```
+
+## Faster Similarity
+
+If you have a large number of users/items, you can use an approximate nearest neighbors library like [NGT](https://github.com/ankane/ngt) to speed up item-based recommendations and similar users.
+
+Add this line to your application’s Gemfile:
+
+```ruby
+gem 'ngt', '>= 0.3.0'
+```
+
+Speed up item-based recommendations with:
+
+```ruby
+model.optimize_item_recs
+```
+
+Speed up similar users with:
+
+```ruby
+model.optimize_similar_users
+```
+
+This should be called after fitting or loading the model.
+
 ## Reference
 
 Get the global mean
@@ -280,3 +316,12 @@ Everyone is encouraged to help improve this project. Here are a few ways you can
 - Fix bugs and [submit pull requests](https://github.com/ankane/disco/pulls)
 - Write, clarify, or fix documentation
 - Suggest or add new features
+
+To get started with development:
+
+```sh
+git clone https://github.com/ankane/disco.git
+cd disco
+bundle install
+bundle exec rake test
+```

data/lib/disco/data.rb

@@ -36,8 +36,7 @@ module Disco
 
       return dest if File.exist?(dest)
 
-      temp_dir ||= File.dirname(Tempfile.new("disco"))
-      temp_path = "#{temp_dir}/#{Time.now.to_f}" # TODO better name
+      temp_path = "#{Dir.tmpdir}/disco-#{Time.now.to_f}" # TODO better name
 
       digest = Digest::SHA2.new
 

data/lib/disco/recommender.rb

@@ -9,14 +9,8 @@ module Disco
     end
 
     def fit(train_set, validation_set: nil)
-      if defined?(Daru)
-        if train_set.is_a?(Daru::DataFrame)
-          train_set = train_set.to_a[0]
-        end
-        if validation_set.is_a?(Daru::DataFrame)
-          validation_set = validation_set.to_a[0]
-        end
-      end
+      train_set = to_dataset(train_set)
+      validation_set = to_dataset(validation_set) if validation_set
 
       @implicit = !train_set.any? { |v| v[:rating] }
 
@@ -70,17 +64,38 @@ module Disco
 
       @global_mean = model.bias
 
-      # TODO read from LIBMF directly to Numo for performance
-      @user_factors = Numo::DFloat.cast(model.p_factors)
-      @item_factors = Numo::DFloat.cast(model.q_factors)
+      @user_factors = model.p_factors(format: :numo)
+      @item_factors = model.q_factors(format: :numo)
+
+      @user_index = nil
+      @item_index = nil
+    end
+
+    # generates a prediction even if a user has already rated the item
+    def predict(data)
+      data = to_dataset(data)
+
+      u = data.map { |v| @user_map[v[:user_id]] }
+      i = data.map { |v| @item_map[v[:item_id]] }
+
+      new_index = data.each_index.select { |index| u[index].nil? || i[index].nil? }
+      new_index.each do |j|
+        u[j] = 0
+        i[j] = 0
+      end
+
+      predictions = @user_factors[u, true].inner(@item_factors[i, true])
+      predictions.inplace.clip(@min_rating, @max_rating) if @min_rating
+      predictions[new_index] = @global_mean
+      predictions.to_a
     end
 
     def user_recs(user_id, count: 5, item_ids: nil)
+      check_fit
       u = @user_map[user_id]
 
       if u
-        predictions = @global_mean + @item_factors.dot(@user_factors[u, true])
-        predictions.inplace.clip(@min_rating, @max_rating) if @min_rating
+        predictions = @item_factors.inner(@user_factors[u, true])
 
         predictions =
           @item_map.keys.zip(predictions).map do |item_id, pred|
@@ -89,15 +104,24 @@ module Disco
 
         if item_ids
           idx = item_ids.map { |i| @item_map[i] }.compact
-          predictions.values_at(*idx)
+          predictions = predictions.values_at(*idx)
         else
-          @rated[u].keys.each do |i|
+          @rated[u].keys.sort_by { |v| -v }.each do |i|
             predictions.delete_at(i)
           end
         end
 
         predictions.sort_by! { |pred| -pred[:score] } # already sorted by id
         predictions = predictions.first(count) if count && !item_ids
+
+        # clamp *after* sorting
+        # also, only needed for returned predictions
+        if @min_rating
+          predictions.each do |pred|
+            pred[:score] = pred[:score].clamp(@min_rating, @max_rating)
+          end
+        end
+
         predictions
       else
         # no items if user is unknown
@@ -106,17 +130,38 @@ module Disco
       end
     end
 
+    def optimize_similar_items
+      check_fit
+      @item_index = create_index(@item_factors)
+    end
+    alias_method :optimize_item_recs, :optimize_similar_items
+
+    def optimize_similar_users
+      check_fit
+      @user_index = create_index(@user_factors)
+    end
+
     def similar_items(item_id, count: 5)
-      similar(item_id, @item_map, @item_factors, item_norms, count)
+      check_fit
+      similar(item_id, @item_map, @item_factors, item_norms, count, @item_index)
     end
     alias_method :item_recs, :similar_items
 
     def similar_users(user_id, count: 5)
-      similar(user_id, @user_map, @user_factors, user_norms, count)
+      check_fit
+      similar(user_id, @user_map, @user_factors, user_norms, count, @user_index)
     end
 
     private
 
+    def create_index(factors)
+      require "ngt"
+
+      index = Ngt::Index.new(factors.shape[1], distance_type: "Cosine")
+      index.batch_insert(factors)
+      index
+    end
+
     def user_norms
       @user_norms ||= norms(@user_factors)
     end
@@ -126,25 +171,41 @@ module Disco
     end
 
     def norms(factors)
-      norms = Numo::DFloat::Math.sqrt((factors * factors).sum(axis: 1))
+      norms = Numo::SFloat::Math.sqrt((factors * factors).sum(axis: 1))
       norms[norms.eq(0)] = 1e-10 # no zeros
       norms
     end
 
-    def similar(id, map, factors, norms, count)
+    def similar(id, map, factors, norms, count, index)
       i = map[id]
       if i
-        predictions = factors.dot(factors[i, true]) / norms
-
-        predictions =
-          map.keys.zip(predictions).map do |item_id, pred|
-            {item_id: item_id, score: pred}
+        if index && count
+          keys = map.keys
+          result = index.search(factors[i, true], size: count + 1)[1..-1]
+          result.map do |v|
+            {
+              # ids from batch_insert start at 1 instead of 0
+              item_id: keys[v[:id] - 1],
+              # convert cosine distance to cosine similarity
+              score: 1 - v[:distance]
+            }
           end
-
-        predictions.delete_at(i)
-        predictions.sort_by! { |pred| -pred[:score] } # already sorted by id
-        predictions = predictions.first(count) if count
-        predictions
+        else
+          predictions = factors.dot(factors[i, true]) / norms
+
+          predictions =
+            map.keys.zip(predictions).map do |item_id, pred|
+              {item_id: item_id, score: pred}
+            end
+
+          max_score = predictions.delete_at(i)[:score]
+          predictions.sort_by! { |pred| -pred[:score] } # already sorted by id
+          predictions = predictions.first(count) if count
+          # divide by max score to get cosine similarity
+          # only need to do for returned records
+          predictions.each { |pred| pred[:score] /= max_score }
+          predictions
+        end
       else
         []
       end
@@ -154,6 +215,9 @@ module Disco
       user_ids = train_set.map { |v| v[:user_id] }.uniq.sort
       item_ids = train_set.map { |v| v[:item_id] }.uniq.sort
 
+      raise ArgumentError, "Missing user_id" if user_ids.any?(&:nil?)
+      raise ArgumentError, "Missing item_id" if item_ids.any?(&:nil?)
+
       @user_map = user_ids.zip(user_ids.size.times).to_h
       @item_map = item_ids.zip(item_ids.size.times).to_h
     end
@@ -171,6 +235,29 @@ module Disco
       raise ArgumentError, "No training data" if train_set.empty?
     end
 
+    def check_fit
+      raise "Not fit" unless defined?(@implicit)
+    end
+
+    def to_dataset(dataset)
+      if defined?(Rover::DataFrame) && dataset.is_a?(Rover::DataFrame)
+        # convert keys to symbols
+        dataset = dataset.dup
+        dataset.keys.each do |k, v|
+          dataset[k.to_sym] ||= dataset.delete(k)
+        end
+        dataset.to_a
+      elsif defined?(Daru::DataFrame) && dataset.is_a?(Daru::DataFrame)
+        # convert keys to symbols
+        dataset = dataset.dup
+        new_names = dataset.vectors.to_a.map { |k| [k, k.to_sym] }.to_h
+        dataset.rename_vectors!(new_names)
+        dataset.to_a[0]
+      else
+        dataset
+      end
+    end
+
     def marshal_dump
       obj = {
         implicit: @implicit,

data/lib/disco/version.rb

@@ -1,3 +1,3 @@
 module Disco
-  VERSION = "0.1.1"
+  VERSION = "0.2.3"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: disco
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.2.3
 platform: ruby
 authors:
 - Andrew Kane
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2019-11-14 00:00:00.000000000 Z
+date: 2020-11-28 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: libmf
@@ -16,14 +16,14 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 0.1.3
+        version: 0.2.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 0.1.3
+        version: 0.2.0
 - !ruby/object:Gem::Dependency
   name: numo-narray
   requirement: !ruby/object:Gem::Requirement
@@ -38,91 +38,7 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-- !ruby/object:Gem::Dependency
-  name: bundler
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: rake
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: minitest
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '5'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '5'
-- !ruby/object:Gem::Dependency
-  name: activerecord
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: sqlite3
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: daru
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-description: 
+description:
 email: andrew@chartkick.com
 executables: []
 extensions: []
@@ -144,7 +60,7 @@ homepage: https://github.com/ankane/disco
 licenses:
 - MIT
 metadata: {}
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -159,8 +75,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.6
-signing_key: 
+rubygems_version: 3.1.4
+signing_key:
 specification_version: 4
-summary: Collaborative filtering for Ruby
+summary: Recommendations for Ruby and Rails using collaborative filtering
 test_files: []