eps 0.3.1 → 0.3.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a59850fe508d404a023145710505e721f1bfc24935a30a090aee09d179887d3a
-  data.tar.gz: 8218bc5bb63ee5ebbd23a8e9a129bcd76789b1f6bb628d57b015f1d5740183ac
+  metadata.gz: d56573908e892d8d1959d66c7b6f2940f8930a2d0f2dfd5d4da75e2ff7cfdb63
+  data.tar.gz: 9eaf1a06c8c51ba15d9b4468796fc869f2933945494d027b54789304080c5d5b
 SHA512:
-  metadata.gz: db1011e9228763dc0a98e1e57d1c9e18a297d362cea18b33bf8eeffecce853ea49d4273ae4e782a6de2be37711e9e6373810e5517558248489e696b477c0848b
-  data.tar.gz: 6b9f52453be9d2ad7a29a4703508763988447de64a7599c53f9b9d3b0135e105130aba3c2679fed17ea60ba7242b6bd0d3cac9c5c2b796fe93f9009f0bbbcb30
+  metadata.gz: 971dbd2a95a280ed50925df68a29018ba7b3bccb7094b1374923a8ce7d100720202245843e003b26447832e9c1f8285bafcc7692020f5971a56c0a8e89a12afb
+  data.tar.gz: de06585dc75608b0f8c62188cce351987a0cd53f3b12889d4d63de28ed81ae1b143e31f47ac8c53083eeb250e18c5f8b721fff94a378e14203fd8fa90ba3e440

data/CHANGELOG.md

@@ -1,7 +1,30 @@
+## 0.3.6 (2020-06-19)
+
+- Fixed error with text features for LightGBM
+
+## 0.3.5 (2020-06-10)
+
+- Added `learning_rate` option for LightGBM
+- Added support for Numo and Rover
+
+## 0.3.4 (2020-04-05)
+
+- Added `predict_probability` for classification
+
+## 0.3.3 (2020-02-24)
+
+- Fixed errors and incorrect predictions with boolean columns
+- Fixed deprecation warnings in Ruby 2.7
+
+## 0.3.2 (2019-12-08)
+
+- Added support for GSLR
+
 ## 0.3.1 (2019-12-06)
 
 - Added `weight` option for LightGBM and linear regression
 - Added `intercept` option for linear regression
+- Added LightGBM evaluator safety check
 - Fixed `Unknown label` error for LightGBM
 - Fixed error message for unstable solutions with linear regression
 

data/README.md

@@ -4,7 +4,6 @@ Machine learning for Ruby
 
 - Build predictive models quickly and easily
 - Serve models built in Ruby, Python, R, and more
-- No prior knowledge of machine learning required :tada:
 
 Check out [this post](https://ankane.org/rails-meet-data-science) for more info on machine learning with Rails
 
@@ -314,7 +313,7 @@ y = [1, 2, 3]
 Eps::Model.new(x, y)
 ```
 
-Or pass arrays of arrays
+Data can be an array of arrays
 
 ```ruby
 x = [[1, 2], [2, 0], [3, 1]]
@@ -322,18 +321,29 @@ y = [1, 2, 3]
 Eps::Model.new(x, y)
 ```
 
-### Daru
+Or Numo arrays
 
-Eps works well with Daru data frames.
+```ruby
+x = Numo::NArray.cast([[1, 2], [2, 0], [3, 1]])
+y = Numo::NArray.cast([1, 2, 3])
+Eps::Model.new(x, y)
+```
+
+Or a Rover data frame
 
 ```ruby
-df = Daru::DataFrame.from_csv("houses.csv")
+df = Rover.read_csv("houses.csv")
 Eps::Model.new(df, target: "price")
 ```
 
-### CSVs
+Or a Daru data frame
 
-When importing data from CSV files, be sure to convert numeric fields. The `table` method does this automatically.
+```ruby
+df = Daru::DataFrame.from_csv("houses.csv")
+Eps::Model.new(df, target: "price")
+```
+
+When reading CSV files directly, be sure to convert numeric fields. The `table` method does this automatically.
 
 ```ruby
 CSV.table("data.csv").map { |row| row.to_h }
@@ -353,9 +363,23 @@ Eps supports:
 - Linear Regression
 - Naive Bayes
 
+### LightGBM
+
+Pass the learning rate with:
+
+```ruby
+Eps::Model.new(data, learning_rate: 0.01)
+```
+
 ### Linear Regression
 
-To speed up training on large datasets with linear regression, [install GSL](https://www.gnu.org/software/gsl/). With Homebrew, you can use:
+By default, an intercept is included. Disable this with:
+
+```ruby
+Eps::Model.new(data, intercept: false)
+```
+
+To speed up training on large datasets with linear regression, [install GSL](https://github.com/ankane/gslr#gsl-installation). With Homebrew, you can use:
 
 ```sh
 brew install gsl
@@ -364,17 +388,21 @@ brew install gsl
 Then, add this line to your application’s Gemfile:
 
 ```ruby
-gem 'gsl', group: :development
+gem 'gslr', group: :development
 ```
 
 It only needs to be available in environments used to build the model.
 
-By default, an intercept is included. Disable this with:
+## Probability
+
+To get the probability of each category for predictions with classification, use:
 
 ```ruby
-Eps::Model.new(data, intercept: false)
+model.predict_probability(data)
 ```
 
+Naive Bayes is known to produce poor probability estimates, so stick with LightGBM if you need this.
+
 ## Validation Options
 
 Pass your own validation set with:
@@ -410,7 +438,7 @@ The database is another place you can store models. It’s good if you retrain m
 Create an ActiveRecord model to store the predictive model.
 
 ```sh
-rails g model Model key:string:uniq data:text
+rails generate model Model key:string:uniq data:text
 ```
 
 Store the model with:
@@ -520,11 +548,11 @@ Everyone is encouraged to help improve this project. Here are a few ways you can
 - Write, clarify, or fix documentation
 - Suggest or add new features
 
-To get started with development and testing:
+To get started with development:
 
 ```sh
 git clone https://github.com/ankane/eps.git
 cd eps
 bundle install
-rake test
+bundle exec rake test
 ```

data/lib/eps/base_estimator.rb

@@ -2,33 +2,18 @@ module Eps
   class BaseEstimator
     def initialize(data = nil, y = nil, **options)
       @options = options.dup
+      @trained = false
+      @text_encoders = {}
       # TODO better pattern - don't pass most options to train
-      options.delete(:intercept)
       train(data, y, **options) if data
     end
 
     def predict(data)
-      singular = data.is_a?(Hash)
-      data = [data] if singular
-
-      data = Eps::DataFrame.new(data)
-
-      @evaluator.features.each do |k, type|
-        values = data.columns[k]
-        raise ArgumentError, "Missing column: #{k}" if !values
-        column_type = Utils.column_type(values.compact, k) if values
-
-        if !column_type.nil?
-          if (type == "numeric" && column_type != "numeric") || (type != "numeric" && column_type != "categorical")
-            raise ArgumentError, "Bad type for column #{k}: Expected #{type} but got #{column_type}"
-          end
-        end
-        # TODO check for unknown values for categorical features
-      end
-
-      predictions = @evaluator.predict(data)
+      _predict(data, false)
+    end
 
-      singular ? predictions.first : predictions
+    def predict_probability(data)
+      _predict(data, true)
     end
 
     def evaluate(data, y = nil, target: nil, weight: nil)
@@ -48,6 +33,8 @@ module Eps
     end
 
     def summary(extended: false)
+      raise "Summary not available for loaded models" unless @trained
+
       str = String.new("")
 
       if @validation_set
@@ -72,7 +59,31 @@ module Eps
 
     private
 
-    def train(data, y = nil, target: nil, weight: nil, split: nil, validation_set: nil, verbose: nil, text_features: nil, early_stopping: nil)
+    def _predict(data, probabilities)
+      singular = data.is_a?(Hash)
+      data = [data] if singular
+
+      data = Eps::DataFrame.new(data)
+
+      @evaluator.features.each do |k, type|
+        values = data.columns[k]
+        raise ArgumentError, "Missing column: #{k}" if !values
+        column_type = Utils.column_type(values.compact, k) if values
+
+        if !column_type.nil?
+          if (type == "numeric" && column_type != "numeric") || (type != "numeric" && column_type != "categorical")
+            raise ArgumentError, "Bad type for column #{k}: Expected #{type} but got #{column_type}"
+          end
+        end
+        # TODO check for unknown values for categorical features
+      end
+
+      predictions = @evaluator.predict(data, probabilities: probabilities)
+
+      singular ? predictions.first : predictions
+    end
+
+    def train(data, y = nil, target: nil, weight: nil, split: nil, validation_set: nil, text_features: nil, **options)
       data, @target = prep_data(data, y, target, weight)
       @target_type = Utils.column_type(data.label, @target)
 
@@ -164,11 +175,13 @@ module Eps
       raise "No data in validation set" if validation_set && validation_set.empty?
 
       @validation_set = validation_set
-      @evaluator = _train(verbose: verbose, early_stopping: early_stopping)
+      @evaluator = _train(**options)
 
       # reset pmml
       @pmml = nil
 
+      @trained = true
+
       nil
     end
 
@@ -197,29 +210,38 @@ module Eps
       [data, target]
     end
 
-    def prep_text_features(train_set)
-      @text_encoders = {}
+    def prep_text_features(train_set, fit: true)
       @text_features.each do |k, v|
-        # reset vocabulary
-        v.delete(:vocabulary)
+        if fit
+          # reset vocabulary
+          v.delete(:vocabulary)
+
+          # TODO determine max features automatically
+          # start based on number of rows
+          encoder = Eps::TextEncoder.new(**v)
+          counts = encoder.fit(train_set.columns.delete(k))
+        else
+          encoder = @text_encoders[k]
+          counts = encoder.transform(train_set.columns.delete(k))
+        end
 
-        # TODO determine max features automatically
-        # start based on number of rows
-        encoder = Eps::TextEncoder.new(v)
-        counts = encoder.fit(train_set.columns.delete(k))
         encoder.vocabulary.each do |word|
           train_set.columns[[k, word]] = [0] * counts.size
         end
+
         counts.each_with_index do |ci, i|
           ci.each do |word, count|
             word_key = [k, word]
             train_set.columns[word_key][i] = 1 if train_set.columns.key?(word_key)
           end
         end
-        @text_encoders[k] = encoder
 
-        # update vocabulary
-        v[:vocabulary] = encoder.vocabulary
+        if fit
+          @text_encoders[k] = encoder
+
+          # update vocabulary
+          v[:vocabulary] = encoder.vocabulary
+        end
       end
 
       raise "No features left" if train_set.columns.empty?
@@ -233,7 +255,7 @@ module Eps
 
     def check_missing(c, name)
       raise ArgumentError, "Missing column: #{name}" if !c
-      raise ArgumentError, "Missing values in column #{name}" if c.any?(&:nil?)
+      raise ArgumentError, "Missing values in column #{name}" if c.to_a.any?(&:nil?)
     end
 
     def check_missing_value(df)

data/lib/eps/data_frame.rb

@@ -10,7 +10,7 @@ module Eps
         data.columns.each do |k, v|
           @columns[k] = v
         end
-      elsif daru?(data)
+      elsif rover?(data) || daru?(data)
         data.to_h.each do |k, v|
           @columns[k.to_s] = v.to_a
         end
@@ -19,6 +19,8 @@ module Eps
           @columns[k.to_s] = v.to_a
         end
       else
+        data = data.to_a if numo?(data)
+
         if data.any?
           row = data[0]
 
@@ -140,8 +142,16 @@ module Eps
 
     private
 
+    def numo?(x)
+      defined?(Numo::NArray) && x.is_a?(Numo::NArray)
+    end
+
+    def rover?(x)
+      defined?(Rover::DataFrame) && x.is_a?(Rover::DataFrame)
+    end
+
     def daru?(x)
-      defined?(Daru) && x.is_a?(Daru::DataFrame)
+      defined?(Daru::DataFrame) && x.is_a?(Daru::DataFrame)
     end
   end
 end

data/lib/eps/evaluators/lightgbm.rb

@@ -11,19 +11,15 @@ module Eps
         @text_features = text_features
       end
 
-      def predict(data)
+      def predict(data, probabilities: false)
+        raise "Probabilities not supported" if probabilities && @objective == "regression"
+
         rows = data.map(&:to_h)
 
         # sparse matrix
         @text_features.each do |k, v|
-          encoder = TextEncoder.new(v)
-
-          values = data.columns.delete(k)
-          counts = encoder.transform(values)
-
-          encoder.vocabulary.each do |word|
-            data.columns[[k, word]] = [0] * values.size
-          end
+          encoder = TextEncoder.new(**v)
+          counts = encoder.transform(data.columns[k])
 
           counts.each_with_index do |xc, i|
             row = rows[i]
@@ -38,17 +34,28 @@ module Eps
         when "regression"
           sum_trees(rows, @trees)
         when "binary"
-          sum_trees(rows, @trees).map { |s| @labels[sigmoid(s) > 0.5 ? 1 : 0] }
+          prob = sum_trees(rows, @trees).map { |s| sigmoid(s) }
+          if probabilities
+            prob.map { |v| @labels.zip([1 - v, v]).to_h }
+          else
+            prob.map { |v| @labels[v > 0.5 ? 1 : 0] }
+          end
         else
           tree_scores = []
           num_trees = @trees.size / @labels.size
           @trees.each_slice(num_trees).each do |trees|
             tree_scores << sum_trees(rows, trees)
           end
-          data.size.times.map do |i|
+          rows.size.times.map do |i|
             v = tree_scores.map { |s| s[i] }
-            idx = v.map.with_index.max_by { |v2, _| v2 }.last
-            @labels[idx]
+            if probabilities
+              exp = v.map { |vi| Math.exp(vi) }
+              sum = exp.sum
+              @labels.zip(exp.map { |e| e / sum }).to_h
+            else
+              idx = v.map.with_index.max_by { |v2, _| v2 }.last
+              @labels[idx]
+            end
           end
         end
       end
@@ -81,7 +88,7 @@ module Eps
           else
             case node.operator
             when "equal"
-              v == node.value
+              v.to_s == node.value
             when "in"
               node.value.include?(v)
             when "greaterThan"
@@ -109,7 +116,7 @@ module Eps
       end
 
       def sigmoid(x)
-        1.0 / (1 + Math::E**(-x))
+        1.0 / (1 + Math.exp(-x))
       end
     end
   end

data/lib/eps/evaluators/linear_regression.rb

@@ -9,7 +9,9 @@ module Eps
         @text_features = text_features || {}
       end
 
-      def predict(x)
+      def predict(x, probabilities: false)
+        raise "Probabilities not supported" if probabilities
+
         intercept = @coefficients["_intercept"] || 0.0
         scores = [intercept] * x.size
 
@@ -19,10 +21,11 @@ module Eps
           case type
           when "categorical"
             x.columns[k].each_with_index do |xv, i|
-              scores[i] += @coefficients[[k, xv]].to_f
+              # TODO clean up
+              scores[i] += (@coefficients[[k, xv]] || @coefficients[[k, xv.to_s]]).to_f
             end
           when "text"
-            encoder = TextEncoder.new(@text_features[k])
+            encoder = TextEncoder.new(**@text_features[k])
             counts = encoder.transform(x.columns[k])
             coef = {}
             @coefficients.each do |k2, v|

data/lib/eps/evaluators/naive_bayes.rb

@@ -10,14 +10,15 @@ module Eps
         @legacy = legacy
       end
 
-      def predict(x)
+      def predict(x, probabilities: false)
         probs = calculate_class_probabilities(x)
         probs.map do |xp|
-          # convert probabilities
-          # not needed when just returning label
-          # sum = xp.values.map { |v| Math.exp(v) }.sum.to_f
-          # p xp.map { |k, v| [k, Math.exp(v) / sum] }.to_h
-          xp.sort_by { |k, v| [-v, k] }[0][0]
+          if probabilities
+            sum = xp.values.map { |v| Math.exp(v) }.sum.to_f
+            xp.map { |k, v| [k, Math.exp(v) / sum] }.to_h
+          else
+            xp.sort_by { |k, v| [-v, k] }[0][0]
+          end
         end
       end
 
@@ -38,7 +39,8 @@ module Eps
             case type
             when "categorical"
               x.columns[k].each_with_index do |xi, i|
-                vc = probabilities[:conditional][k][xi]
+                # TODO clean this up
+                vc = probabilities[:conditional][k][xi] || probabilities[:conditional][k][xi.to_s]
 
                 # unknown value if not vc
                 if vc

data/lib/eps/lightgbm.rb

@@ -17,7 +17,7 @@ module Eps
       str
     end
 
-    def _train(verbose: nil, early_stopping: nil)
+    def _train(verbose: nil, early_stopping: nil, learning_rate: 0.1)
       train_set = @train_set
       validation_set = @validation_set.dup
       summary_label = train_set.label
@@ -57,10 +57,13 @@ module Eps
 
       # text feature encoding
       prep_text_features(train_set)
-      prep_text_features(validation_set) if validation_set
+      prep_text_features(validation_set, fit: false) if validation_set
 
       # create params
-      params = {objective: objective}
+      params = {
+        objective: objective,
+        learning_rate: learning_rate
+      }
       params[:num_classes] = labels.size if objective == "multiclass"
       if train_set.size < 30
         params[:min_data_in_bin] = 1
@@ -121,25 +124,30 @@ module Eps
     def check_evaluator(objective, labels, booster, booster_set, evaluator, evaluator_set)
       expected = @booster.predict(booster_set.map_rows(&:to_a))
       if objective == "multiclass"
-        expected.map! do |v|
-          labels[v.map.with_index.max_by { |v2, _| v2 }.last]
-        end
+        actual = evaluator.predict(evaluator_set, probabilities: true)
+        # just compare first for now
+        expected.map! { |v| v.first }
+        actual.map! { |v| v.values.first }
       elsif objective == "binary"
-        expected.map! { |v| labels[v >= 0.5 ? 1 : 0] }
+        actual = evaluator.predict(evaluator_set, probabilities: true).map { |v| v.values.last }
+      else
+        actual = evaluator.predict(evaluator_set)
       end
-      actual = evaluator.predict(evaluator_set)
 
-      regression = objective == "regression"
+      regression = objective == "regression" || objective == "binary"
       bad_observations = []
       expected.zip(actual).each_with_index do |(exp, act), i|
-        success = regression ? (act - exp).abs < 0.001 : act == exp
+        success = (act - exp).abs < 0.001
         unless success
           bad_observations << {expected: exp, actual: act, data_point: evaluator_set[i].map(&:itself).first}
         end
       end
 
       if bad_observations.any?
-        raise "Bug detected in evaluator. Please report an issue. Bad data points: #{bad_observations.inspect}"
+        bad_observations.each do |obs|
+          p obs
+        end
+        raise "Bug detected in evaluator. Please report an issue."
       end
     end
 

data/lib/eps/linear_regression.rb

@@ -37,6 +37,7 @@ module Eps
       str
     end
 
+    # TODO use keyword arguments for gsl and intercept in 0.4.0
     def _train(**options)
       raise "Target must be numeric" if @target_type != "numeric"
       check_missing_value(@train_set)
@@ -50,17 +51,35 @@ module Eps
 
       x = data.map_rows(&:to_a)
 
-      intercept = @options.key?(:intercept) ? @options[:intercept] : true
-      if intercept
+      gsl =
+        if options.key?(:gsl)
+          options[:gsl]
+        elsif defined?(GSL)
+          true
+        elsif defined?(GSLR)
+          :gslr
+        else
+          false
+        end
+
+      intercept = options.key?(:intercept) ? options[:intercept] : true
+      if intercept && gsl != :gslr
         data.size.times do |i|
           x[i].unshift(1)
         end
       end
 
-      gsl = options.key?(:gsl) ? options[:gsl] : defined?(GSL)
-
       v3 =
-        if gsl
+        if gsl == :gslr
+          model = GSLR::OLS.new(intercept: intercept)
+          model.fit(x, data.label, weight: data.weight)
+
+          @covariance = model.covariance
+
+          coefficients = model.coefficients.dup
+          coefficients.unshift(model.intercept) if intercept
+          coefficients
+        elsif gsl
           x = GSL::Matrix.alloc(*x)
           y = GSL::Vector.alloc(data.label)
           w = GSL::Vector.alloc(data.weight) if data.weight
@@ -196,7 +215,11 @@ module Eps
 
     def diagonal
       @diagonal ||= begin
-        if covariance.respond_to?(:each)
+        if covariance.is_a?(Array)
+          covariance.size.times.map do |i|
+            covariance[i][i]
+          end
+        elsif covariance.respond_to?(:each)
           d = covariance.each(:diagonal).to_a
           @removed.each do |i|
             d.insert(i, 0)

data/lib/eps/naive_bayes.rb

@@ -17,7 +17,7 @@ module Eps
       str
     end
 
-    def _train(smoothing: 1, **options)
+    def _train(smoothing: 1)
       raise "Target must be strings" if @target_type != "categorical"
       check_missing_value(@train_set)
       check_missing_value(@validation_set) if @validation_set

data/lib/eps/pmml/generator.rb

@@ -210,10 +210,10 @@ module Eps
               probabilities[:conditional].each do |k, v|
                 xml.BayesInput(fieldName: k) do
                   if features[k] == "categorical"
-                    v.sort_by { |k2, _| k2 }.each do |k2, v2|
+                    v.sort_by { |k2, _| k2.to_s }.each do |k2, v2|
                       xml.PairCounts(value: k2) do
                         xml.TargetValueCounts do
-                          v2.sort_by { |k2, _| k2 }.each do |k3, v3|
+                          v2.sort_by { |k2, _| k2.to_s }.each do |k3, v3|
                             xml.TargetValueCount(value: k3, count: v3)
                           end
                         end
@@ -221,7 +221,7 @@ module Eps
                     end
                   else
                     xml.TargetValueStats do
-                      v.sort_by { |k2, _| k2 }.each do |k2, v2|
+                      v.sort_by { |k2, _| k2.to_s }.each do |k2, v2|
                         xml.TargetValueStat(value: k2) do
                           xml.GaussianDistribution(mean: v2[:mean], variance: v2[:stdev]**2)
                         end
@@ -233,7 +233,7 @@ module Eps
             end
             xml.BayesOutput(fieldName: "target") do
               xml.TargetValueCounts do
-                probabilities[:prior].sort_by { |k, _| k }.each do |k, v|
+                probabilities[:prior].sort_by { |k, _| k.to_s }.each do |k, v|
                   xml.TargetValueCount(value: k, count: v)
                 end
               end

data/lib/eps/version.rb

@@ -1,3 +1,3 @@
 module Eps
-  VERSION = "0.3.1"
+  VERSION = "0.3.6"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: eps
 version: !ruby/object:Gem::Version
-  version: 0.3.1
+  version: 0.3.6
 platform: ruby
 authors:
 - Andrew Kane
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-12-06 00:00:00.000000000 Z
+date: 2020-06-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: lightgbm
@@ -80,6 +80,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: numo-narray
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
@@ -94,6 +108,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: rover-df
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: 
 email: andrew@chartkick.com
 executables: []
@@ -143,7 +171,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.3
+rubygems_version: 3.1.2
 signing_key: 
 specification_version: 4
 summary: Machine learning for Ruby. Supports regression (linear regression) and classification