svmkit 0.4.0 → 0.4.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cef050a2ac6b55583414cb3ce9c3678dd6d2d1c8b2be04a249222683e10465e1
-  data.tar.gz: 7c67ab0e90246f1d9b7e5d0bfb19ed76061d0edf17a05014f521b8ef41e41aed
+  metadata.gz: af30c20b06fec51d531364ad9ca1414ce2fe36cdbe61fd8a1a7128c793d67304
+  data.tar.gz: ba87c535aa723ec17334fd6819577dcb51d2d11ccef6adb967f73de1702522f5
 SHA512:
-  metadata.gz: 15341450f3bf3ca49901ae55b507d647468261682c7fdb0b058c21a470c2eec261718b6721ca0e2ad7738cfdabd184128a588d68ad6d079e53c9b1e916efa2b1
-  data.tar.gz: fd562db538be12896c005840e065f867e342691e899b33f0524a4db26da33439bfc174141e022d4de3d805657d09e854a4593b9b05b2d9eb99f6cd41da064a1d
+  metadata.gz: b32efe1dcd924c3e31ad0dc26dfbdcc86b0154b8b8591e58db5364103526b7dc828c46462b5f2dfe81c7c8ee23836ae8d4b81061cdf1ceb4f023c48cc78dd110
+  data.tar.gz: 6f38f301d23b3abc1037e1b0fe620e687da1fe44216a49707b2192d30fd8f2a7cb7690d6365580dda470e6852200db20b540c35947e3b1c54d8f8b5b599b2dc0

data/HISTORY.md

@@ -1,3 +1,10 @@
+# 0.4.1
+- Add class for linear regressor.
+- Add class for SGD optimizer.
+- Add class for RMSProp optimizer.
+- Add class for YellowFin optimizer.
+- Fix to be able to select optimizer on estimators of LineaModel and PolynomialModel.
+
 # 0.4.0
 ## Breaking changes
 

data/README.md

@@ -8,8 +8,8 @@
 SVMKit is a machine learninig library in Ruby.
 SVMKit provides machine learning algorithms with interfaces similar to Scikit-Learn in Python.
 SVMKit currently supports Linear / Kernel Support Vector Machine,
-Logistic Regression, Ridge, Lasso, Factorization Machine, Naive Bayes, Decision Tree, Random Forest,
-K-nearest neighbor classifier, and cross-validation.
+Logistic Regression, Linear Regression, Ridge, Lasso, Factorization Machine,
+Naive Bayes, Decision Tree, Random Forest, K-nearest neighbor classifier, and cross-validation.
 
 ## Installation
 
@@ -29,61 +29,97 @@ Or install it yourself as:
 
 ## Usage
 
-Training phase:
+### Example 1. Pendigits dataset classification
+
+SVMKit provides function loading libsvm format dataset file.
+We start by downloading the pendigits dataset from LIBSVM Data web site.
+
+```bash
+$ wget https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/multiclass/pendigits
+$ wget https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/multiclass/pendigits.t
+```
+
+Training of the classifier with Linear SVM and RBF kernel feature map is the following code.
 
 ```ruby
 require 'svmkit'
 
+# Load the training dataset.
 samples, labels = SVMKit::Dataset.load_libsvm_file('pendigits')
 
-normalizer = SVMKit::Preprocessing::MinMaxScaler.new
-normalized = normalizer.fit_transform(samples)
+# If the features consists only of integers, load_libsvm_file method reads in Numo::Int32 format.
+# As necessary, you should convert sample array to Numo::DFloat format.
+samples = Numo::DFloat.cast(samples)
 
-transformer = SVMKit::KernelApproximation::RBF.new(gamma: 2.0, n_components: 1024, random_seed: 1)
-transformed = transformer.fit_transform(normalized)
+# Map training data to RBF kernel feature space.
+transformer = SVMKit::KernelApproximation::RBF.new(gamma: 0.0001, n_components: 1024, random_seed: 1)
+transformed = transformer.fit_transform(samples)
 
-classifier = SVMKit::LinearModel::SVC.new(reg_param: 1.0, max_iter: 1000, batch_size: 20, random_seed: 1)
+# Train linear SVM classifier.
+classifier = SVMKit::LinearModel::SVC.new(reg_param: 0.0001, max_iter: 1000, batch_size: 50, random_seed: 1)
 classifier.fit(transformed, labels)
 
-File.open('trained_normalizer.dat', 'wb') { |f| f.write(Marshal.dump(normalizer)) }
-File.open('trained_transformer.dat', 'wb') { |f| f.write(Marshal.dump(transformer)) }
-File.open('trained_classifier.dat', 'wb') { |f| f.write(Marshal.dump(classifier)) }
+# Save the model.
+File.open('transformer.dat', 'wb') { |f| f.write(Marshal.dump(transformer)) }
+File.open('classifier.dat', 'wb') { |f| f.write(Marshal.dump(classifier)) }
 ```
 
-Testing phase:
+Classifying testing data with the trained classifier is the following code.
 
 ```ruby
 require 'svmkit'
 
+# Load the testing dataset.
 samples, labels = SVMKit::Dataset.load_libsvm_file('pendigits.t')
+samples = Numo::DFloat.cast(samples)
 
-normalizer = Marshal.load(File.binread('trained_normalizer.dat'))
-transformer = Marshal.load(File.binread('trained_transformer.dat'))
-classifier = Marshal.load(File.binread('trained_classifier.dat'))
+# Load the model.
+transformer = Marshal.load(File.binread('transformer.dat'))
+classifier = Marshal.load(File.binread('classifier.dat'))
+
+# Map testing data to RBF kernel feature space.
+transformed = transformer.transform(samples)
+
+# Classify the testing data and evaluate prediction results.
+puts("Accuracy: %.1f%%" % (100.0 * classifier.score(transformed, labels)))
+
+# Other evaluating approach
+# results = classifier.predict(transformed)
+# evaluator = SVMKit::EvaluationMeasure::Accuracy.new
+# puts("Accuracy: %.1f%%" % (100.0 * evaluator.score(results, labels)))
+```
 
-normalized = normalizer.transform(samples)
-transformed = transformer.transform(normalized)
+Execution of the above scripts result in the following.
 
-puts(sprintf("Accuracy: %.1f%%", 100.0 * classifier.score(transformed, labels)))
+```bash
+$ ruby train.rb
+$ ruby test.rb
+Accuracy: 98.4%
 ```
 
-5-fold cross-validation:
+### Example 2. Cross-validation
 
 ```ruby
 require 'svmkit'
 
+# Load dataset.
 samples, labels = SVMKit::Dataset.load_libsvm_file('pendigits')
+samples = Numo::DFloat.cast(samples)
 
-kernel_svc = SVMKit::KernelMachine::KernelSVC.new(reg_param: 1.0, max_iter: 1000, random_seed: 1)
+# Define the estimator to be evaluated.
+lr = SVMKit::LinearModel::LogisticRegression.new(reg_param: 0.0001, random_seed: 1)
 
+# Define the evaluation measure, splitting strategy, and cross validation.
+ev = SVMKit::EvaluationMeasure::LogLoss.new
 kf = SVMKit::ModelSelection::StratifiedKFold.new(n_splits: 5, shuffle: true, random_seed: 1)
-cv = SVMKit::ModelSelection::CrossValidation.new(estimator: kernel_svc, splitter: kf)
+cv = SVMKit::ModelSelection::CrossValidation.new(estimator: lr, splitter: kf, evaluator: ev)
 
-kernel_mat = SVMKit::PairwiseMetric::rbf_kernel(samples, nil, 0.005)
-report = cv.perform(kernel_mat, labels)
+# Perform 5-cross validation.
+report = cv.perform(samples, labels)
 
-mean_accuracy = report[:test_score].inject(:+) / kf.n_splits
-puts(sprintf("Mean Accuracy: %.1f%%", 100.0 * mean_accuracy))
+# Output result.
+mean_logloss = report[:test_score].inject(:+) / kf.n_splits
+puts("5-CV mean log-loss: %.3f" % mean_logloss)
 ```
 
 ## Development

data/lib/svmkit.rb

@@ -13,11 +13,15 @@ require 'svmkit/base/regressor'
 require 'svmkit/base/transformer'
 require 'svmkit/base/splitter'
 require 'svmkit/base/evaluator'
+require 'svmkit/optimizer/sgd'
+require 'svmkit/optimizer/rmsprop'
 require 'svmkit/optimizer/nadam'
+require 'svmkit/optimizer/yellow_fin'
 require 'svmkit/kernel_approximation/rbf'
 require 'svmkit/linear_model/svc'
 require 'svmkit/linear_model/svr'
 require 'svmkit/linear_model/logistic_regression'
+require 'svmkit/linear_model/linear_regression'
 require 'svmkit/linear_model/ridge'
 require 'svmkit/linear_model/lasso'
 require 'svmkit/kernel_machine/kernel_svc'

data/lib/svmkit/linear_model/lasso.rb

@@ -43,7 +43,7 @@ module SVMKit
       # @param max_iter [Integer] The maximum number of iterations.
       # @param batch_size [Integer] The size of the mini batches.
       # @param optimizer [Optimizer] The optimizer to calculate adaptive learning rate.
-      #   Nadam is selected automatically on current version.
+      #   If nil is given, Nadam is used.
       # @param random_seed [Integer] The seed value using to initialize the random generator.
       def initialize(reg_param: 1.0, fit_bias: false, max_iter: 1000, batch_size: 10, optimizer: nil, random_seed: nil)
         check_params_float(reg_param: reg_param)
@@ -57,6 +57,7 @@ module SVMKit
         @params[:max_iter] = max_iter
         @params[:batch_size] = batch_size
         @params[:optimizer] = optimizer
+        @params[:optimizer] ||= Optimizer::Nadam.new
         @params[:random_seed] = random_seed
         @params[:random_seed] ||= srand
         @weight_vec = nil
@@ -80,11 +81,7 @@ module SVMKit
         if n_outputs > 1
           @weight_vec = Numo::DFloat.zeros(n_outputs, n_features)
           @bias_term = Numo::DFloat.zeros(n_outputs)
-          n_outputs.times do |n|
-            weight, bias = single_fit(x, y[true, n])
-            @weight_vec[n, true] = weight
-            @bias_term[n] = bias
-          end
+          n_outputs.times { |n| @weight_vec[n, true], @bias_term[n] = single_fit(x, y[true, n]) }
         else
           @weight_vec, @bias_term = single_fit(x, y)
         end
@@ -131,8 +128,8 @@ module SVMKit
         weight_vec = Numo::DFloat.zeros(n_features)
         left_weight_vec = Numo::DFloat.zeros(n_features)
         right_weight_vec = Numo::DFloat.zeros(n_features)
-        left_optimizer = Optimizer::Nadam.new
-        right_optimizer = Optimizer::Nadam.new
+        left_optimizer = @params[:optimizer].dup
+        right_optimizer = @params[:optimizer].dup
         # Start optimization.
         @params[:max_iter].times do |_t|
           # Random sampling.

data/lib/svmkit/linear_model/linear_regression.rb

@@ -0,0 +1,159 @@
+# frozen_string_literal: true
+
+require 'svmkit/validation'
+require 'svmkit/base/base_estimator'
+require 'svmkit/base/regressor'
+require 'svmkit/optimizer/nadam'
+
+module SVMKit
+  module LinearModel
+    # LinearRegression is a class that implements ordinary least square linear regression
+    # with mini-batch stochastic gradient descent optimization.
+    #
+    # @example
+    #   estimator =
+    #     SVMKit::LinearModel::LinearRegression.new(max_iter: 1000, batch_size: 20, random_seed: 1)
+    #   estimator.fit(training_samples, traininig_values)
+    #   results = estimator.predict(testing_samples)
+    #
+    class LinearRegression
+      include Base::BaseEstimator
+      include Base::Regressor
+      include Validation
+
+      # Return the weight vector.
+      # @return [Numo::DFloat] (shape: [n_outputs, n_features])
+      attr_reader :weight_vec
+
+      # Return the bias term (a.k.a. intercept).
+      # @return [Numo::DFloat] (shape: [n_outputs])
+      attr_reader :bias_term
+
+      # Return the random generator for random sampling.
+      # @return [Random]
+      attr_reader :rng
+
+      # Create a new ordinary least square linear regressor.
+      #
+      # @param fit_bias [Boolean] The flag indicating whether to fit the bias term.
+      # @param max_iter [Integer] The maximum number of iterations.
+      # @param batch_size [Integer] The size of the mini batches.
+      # @param optimizer [Optimizer] The optimizer to calculate adaptive learning rate.
+      #   If nil is given, Nadam is used.
+      # @param random_seed [Integer] The seed value using to initialize the random generator.
+      def initialize(fit_bias: false, max_iter: 1000, batch_size: 10, optimizer: nil, random_seed: nil)
+        check_params_integer(max_iter: max_iter, batch_size: batch_size)
+        check_params_boolean(fit_bias: fit_bias)
+        check_params_type_or_nil(Integer, random_seed: random_seed)
+        check_params_positive(max_iter: max_iter, batch_size: batch_size)
+        @params = {}
+        @params[:fit_bias] = fit_bias
+        @params[:max_iter] = max_iter
+        @params[:batch_size] = batch_size
+        @params[:optimizer] = optimizer
+        @params[:optimizer] ||= Optimizer::Nadam.new
+        @params[:random_seed] = random_seed
+        @params[:random_seed] ||= srand
+        @weight_vec = nil
+        @bias_term = nil
+        @rng = Random.new(@params[:random_seed])
+      end
+
+      # Fit the model with given training data.
+      #
+      # @param x [Numo::DFloat] (shape: [n_samples, n_features]) The training data to be used for fitting the model.
+      # @param y [Numo::Int32] (shape: [n_samples, n_outputs]) The target values to be used for fitting the model.
+      # @return [LinearRegression] The learned regressor itself.
+      def fit(x, y)
+        check_sample_array(x)
+        check_tvalue_array(y)
+        check_sample_tvalue_size(x, y)
+
+        n_outputs = y.shape[1].nil? ? 1 : y.shape[1]
+        n_features = x.shape[1]
+
+        if n_outputs > 1
+          @weight_vec = Numo::DFloat.zeros(n_outputs, n_features)
+          @bias_term = Numo::DFloat.zeros(n_outputs)
+          n_outputs.times { |n| @weight_vec[n, true], @bias_term[n] = single_fit(x, y[true, n]) }
+        else
+          @weight_vec, @bias_term = single_fit(x, y)
+        end
+
+        self
+      end
+
+      # Predict values for samples.
+      #
+      # @param x [Numo::DFloat] (shape: [n_samples, n_features]) The samples to predict the values.
+      # @return [Numo::DFloat] (shape: [n_samples, n_outputs]) Predicted values per sample.
+      def predict(x)
+        check_sample_array(x)
+        x.dot(@weight_vec.transpose) + @bias_term
+      end
+
+      # Dump marshal data.
+      # @return [Hash] The marshal data about LinearRegression.
+      def marshal_dump
+        { params: @params,
+          weight_vec: @weight_vec,
+          bias_term: @bias_term,
+          rng: @rng }
+      end
+
+      # Load marshal data.
+      # @return [nil]
+      def marshal_load(obj)
+        @params = obj[:params]
+        @weight_vec = obj[:weight_vec]
+        @bias_term = obj[:bias_term]
+        @rng = obj[:rng]
+        nil
+      end
+
+      private
+
+      def single_fit(x, y)
+        # Expand feature vectors for bias term.
+        samples = @params[:fit_bias] ? expand_feature(x) : x
+        # Initialize some variables.
+        n_samples, n_features = samples.shape
+        rand_ids = [*0...n_samples].shuffle(random: @rng)
+        weight_vec = Numo::DFloat.zeros(n_features)
+        optimizer = @params[:optimizer].dup
+        # Start optimization.
+        @params[:max_iter].times do |_t|
+          # Random sampling.
+          subset_ids = rand_ids.shift(@params[:batch_size])
+          rand_ids.concat(subset_ids)
+          data = samples[subset_ids, true]
+          values = y[subset_ids]
+          # Calculate gradients for loss function.
+          loss_grad = loss_gradient(data, values, weight_vec)
+          next if loss_grad.ne(0.0).count.zero?
+          # Update weight.
+          weight_vec = optimizer.call(weight_vec, weight_gradient(loss_grad, data, weight_vec))
+        end
+        split_weight_vec_bias(weight_vec)
+      end
+
+      def loss_gradient(x, y, weight)
+        2.0 * (x.dot(weight) - y)
+      end
+
+      def weight_gradient(loss_grad, data, _weight)
+        (loss_grad.expand_dims(1) * data).mean(0)
+      end
+
+      def expand_feature(x)
+        Numo::NArray.hstack([x, Numo::DFloat.ones([x.shape[0], 1])])
+      end
+
+      def split_weight_vec_bias(weight_vec)
+        weights = @params[:fit_bias] ? weight_vec[0...-1] : weight_vec
+        bias = @params[:fit_bias] ? weight_vec[-1] : 0.0
+        [weights, bias]
+      end
+    end
+  end
+end

data/lib/svmkit/linear_model/logistic_regression.rb

@@ -49,7 +49,7 @@ module SVMKit
       # @param max_iter [Integer] The maximum number of iterations.
       # @param batch_size [Integer] The size of the mini batches.
       # @param optimizer [Optimizer] The optimizer to calculate adaptive learning rate.
-      #   Nadam is selected automatically on current version.
+      #   If nil is given, Nadam is used.
       # @param random_seed [Integer] The seed value using to initialize the random generator.
       def initialize(reg_param: 1.0, fit_bias: false, bias_scale: 1.0,
                      max_iter: 1000, batch_size: 20, optimizer: nil, random_seed: nil)
@@ -65,6 +65,7 @@ module SVMKit
         @params[:max_iter] = max_iter
         @params[:batch_size] = batch_size
         @params[:optimizer] = optimizer
+        @params[:optimizer] ||= Optimizer::Nadam.new
         @params[:random_seed] = random_seed
         @params[:random_seed] ||= srand
         @weight_vec = nil
@@ -175,7 +176,7 @@ module SVMKit
         n_samples, n_features = samples.shape
         rand_ids = [*0...n_samples].shuffle(random: @rng)
         weight_vec = Numo::DFloat.zeros(n_features)
-        optimizer = Optimizer::Nadam.new
+        optimizer = @params[:optimizer].dup
         # Start optimization.
         @params[:max_iter].times do |_t|
           # random sampling

data/lib/svmkit/linear_model/ridge.rb

@@ -39,6 +39,8 @@ module SVMKit
       # @param fit_bias [Boolean] The flag indicating whether to fit the bias term.
       # @param max_iter [Integer] The maximum number of iterations.
       # @param batch_size [Integer] The size of the mini batches.
+      # @param optimizer [Optimizer] The optimizer to calculate adaptive learning rate.
+      #   If nil is given, Nadam is used.
       # @param random_seed [Integer] The seed value using to initialize the random generator.
       def initialize(reg_param: 1.0, fit_bias: false, max_iter: 1000, batch_size: 10, optimizer: nil, random_seed: nil)
         check_params_float(reg_param: reg_param)
@@ -52,6 +54,7 @@ module SVMKit
         @params[:max_iter] = max_iter
         @params[:batch_size] = batch_size
         @params[:optimizer] = optimizer
+        @params[:optimizer] ||= Optimizer::Nadam.new
         @params[:random_seed] = random_seed
         @params[:random_seed] ||= srand
         @weight_vec = nil
@@ -75,11 +78,7 @@ module SVMKit
         if n_outputs > 1
           @weight_vec = Numo::DFloat.zeros(n_outputs, n_features)
           @bias_term = Numo::DFloat.zeros(n_outputs)
-          n_outputs.times do |n|
-            weight, bias = single_fit(x, y[true, n])
-            @weight_vec[n, true] = weight
-            @bias_term[n] = bias
-          end
+          n_outputs.times { |n| @weight_vec[n, true], @bias_term[n] = single_fit(x, y[true, n]) }
         else
           @weight_vec, @bias_term = single_fit(x, y)
         end
@@ -124,7 +123,7 @@ module SVMKit
         n_samples, n_features = samples.shape
         rand_ids = [*0...n_samples].shuffle(random: @rng)
         weight_vec = Numo::DFloat.zeros(n_features)
-        optimizer = Optimizer::Nadam.new
+        optimizer = @params[:optimizer].dup
         # Start optimization.
         @params[:max_iter].times do |_t|
           # Random sampling.

data/lib/svmkit/linear_model/svc.rb

@@ -51,7 +51,7 @@ module SVMKit
       # @param batch_size [Integer] The size of the mini batches.
       # @param probability [Boolean] The flag indicating whether to perform probability estimation.
       # @param optimizer [Optimizer] The optimizer to calculate adaptive learning rate.
-      #   Nadam is selected automatically on current version.
+      #   If nil is given, Nadam is used.
       # @param random_seed [Integer] The seed value using to initialize the random generator.
       def initialize(reg_param: 1.0, fit_bias: false, bias_scale: 1.0,
                      max_iter: 1000, batch_size: 20, probability: false, optimizer: nil, random_seed: nil)
@@ -68,6 +68,7 @@ module SVMKit
         @params[:batch_size] = batch_size
         @params[:probability] = probability
         @params[:optimizer] = optimizer
+        @params[:optimizer] ||= Optimizer::Nadam.new
         @params[:random_seed] = random_seed
         @params[:random_seed] ||= srand
         @weight_vec = nil
@@ -194,7 +195,7 @@ module SVMKit
         n_samples, n_features = samples.shape
         rand_ids = [*0...n_samples].shuffle(random: @rng)
         weight_vec = Numo::DFloat.zeros(n_features)
-        optimizer = Optimizer::Nadam.new
+        optimizer = @params[:optimizer].dup
         # Start optimization.
         @params[:max_iter].times do |_t|
           # random sampling.

data/lib/svmkit/linear_model/svr.rb

@@ -44,7 +44,7 @@ module SVMKit
       # @param max_iter [Integer] The maximum number of iterations.
       # @param batch_size [Integer] The size of the mini batches.
       # @param optimizer [Optimizer] The optimizer to calculate adaptive learning rate.
-      #   Nadam is selected automatically on current version.
+      #   If nil is given, Nadam is used.
       # @param random_seed [Integer] The seed value using to initialize the random generator.
       def initialize(reg_param: 1.0, fit_bias: false, bias_scale: 1.0, epsilon: 0.1,
                      max_iter: 1000, batch_size: 20, optimizer: nil, random_seed: nil)
@@ -62,6 +62,7 @@ module SVMKit
         @params[:max_iter] = max_iter
         @params[:batch_size] = batch_size
         @params[:optimizer] = optimizer
+        @params[:optimizer] ||= Optimizer::Nadam.new
         @params[:random_seed] = random_seed
         @params[:random_seed] ||= srand
         @weight_vec = nil
@@ -85,11 +86,7 @@ module SVMKit
         if n_outputs > 1
           @weight_vec = Numo::DFloat.zeros(n_outputs, n_features)
           @bias_term = Numo::DFloat.zeros(n_outputs)
-          n_outputs.times do |n|
-            weight, bias = single_fit(x, y[true, n])
-            @weight_vec[n, true] = weight
-            @bias_term[n] = bias
-          end
+          n_outputs.times { |n| @weight_vec[n, true], @bias_term[n] = single_fit(x, y[true, n]) }
         else
           @weight_vec, @bias_term = single_fit(x, y)
         end
@@ -134,7 +131,7 @@ module SVMKit
         n_samples, n_features = samples.shape
         rand_ids = [*0...n_samples].shuffle(random: @rng)
         weight_vec = Numo::DFloat.zeros(n_features)
-        optimizer = Optimizer::Nadam.new
+        optimizer = @params[:optimizer].dup
         # Start optimization.
         @params[:max_iter].times do |_t|
           # random sampling

data/lib/svmkit/optimizer/nadam.rb

@@ -1,16 +1,22 @@
 # frozen_string_literal: true
 
 require 'svmkit/validation'
+require 'svmkit/base/base_estimator'
 
 module SVMKit
   # This module consists of the classes that implement optimizers adaptively tuning hyperparameters.
   module Optimizer
     # Nadam is a class that implements Nadam optimizer.
-    # This class is used for internal processes.
+    #
+    # @example
+    #   optimizer = SVMKit::Optimizer::Nadam.new(learning_rate: 0.01, momentum: 0.9, decay1: 0.9, decay2: 0.999)
+    #   estimator = SVMKit::LinearModel::LinearRegression.new(optimizer: optimizer, random_seed: 1)
+    #   estimator.fit(samples, values)
     #
     # *Reference*
     # - T. Dozat, "Incorporating Nesterov Momentum into Adam," Tech. Repo. Stanford University, 2015.
     class Nadam
+      include Base::BaseEstimator
       include Validation
 
       # Create a new optimizer with Nadam
@@ -19,7 +25,6 @@ module SVMKit
       # @param momentum [Float] The initial value of momentum.
       # @param decay1 [Float] The smoothing parameter for the first moment.
       # @param decay2 [Float] The smoothing parameter for the second moment.
-      # @param schedule_decay [Float] The smooting parameter.
       def initialize(learning_rate: 0.01, momentum: 0.9, decay1: 0.9, decay2: 0.999)
         check_params_float(learning_rate: learning_rate, momentum: momentum, decay1: decay1, decay2: decay2)
         check_params_positive(learning_rate: learning_rate, momentum: momentum, decay1: decay1, decay2: decay2)
@@ -59,6 +64,27 @@ module SVMKit
 
         weight - (@params[:learning_rate] / (nm_sec_moment**0.5 + 1e-8)) * ((1 - decay1_curr) * nm_gradient + decay1_next * nm_fst_moment)
       end
+
+      # Dump marshal data.
+      # @return [Hash] The marshal data.
+      def marshal_dump
+        { params: @params,
+          fst_moment: @fst_moment,
+          sec_moment: @sec_moment,
+          decay1_prod: @decay1_prod,
+          iter: @iter }
+      end
+
+      # Load marshal data.
+      # @return [nil]
+      def marshal_load(obj)
+        @params = obj[:params]
+        @fst_moment = obj[:fst_moment]
+        @sec_moment = obj[:sec_moment]
+        @decay1_prod = obj[:decay1_prod]
+        @iter = obj[:iter]
+        nil
+      end
     end
   end
 end

data/lib/svmkit/optimizer/rmsprop.rb

@@ -0,0 +1,69 @@
+# frozen_string_literal: true
+
+require 'svmkit/validation'
+require 'svmkit/base/base_estimator'
+
+module SVMKit
+  module Optimizer
+    # RMSProp is a class that implements RMSProp optimizer.
+    #
+    # @example
+    #   optimizer = SVMKit::Optimizer::RMSProp.new(learning_rate: 0.01, momentum: 0.9, decay: 0.9)
+    #   estimator = SVMKit::LinearModel::LinearRegression.new(optimizer: optimizer, random_seed: 1)
+    #   estimator.fit(samples, values)
+    #
+    # *Reference*
+    # - I. Sutskever, J. Martens, G. Dahl, and G. Hinton, "On the importance of initialization and momentum in deep learning," Proc. ICML' 13, pp. 1139--1147, 2013.
+    # - G. Hinton, N. Srivastava, and K. Swersky, "Lecture 6e rmsprop," Neural Networks for Machine Learning, 2012.
+    class RMSProp
+      include Base::BaseEstimator
+      include Validation
+
+      # Create a new optimizer with RMSProp.
+      #
+      # @param learning_rate [Float] The initial value of learning rate.
+      # @param momentum [Float] The initial value of momentum.
+      # @param decay [Float] The smooting parameter.
+      def initialize(learning_rate: 0.01, momentum: 0.9, decay: 0.9)
+        check_params_float(learning_rate: learning_rate, momentum: momentum, decay: decay)
+        check_params_positive(learning_rate: learning_rate, momentum: momentum, decay: decay)
+        @params = {}
+        @params[:learning_rate] = learning_rate
+        @params[:momentum] = momentum
+        @params[:decay] = decay
+        @moment = nil
+        @update = nil
+      end
+
+      # Calculate the updated weight with RMSProp adaptive learning rate.
+      #
+      # @param weight [Numo::DFloat] (shape: [n_features]) The weight to be updated.
+      # @param gradient [Numo::DFloat] (shape: [n_features]) The gradient for updating the weight.
+      # @return [Numo::DFloat] (shape: [n_feautres]) The updated weight.
+      def call(weight, gradient)
+        @moment ||= Numo::DFloat.zeros(weight.shape[0])
+        @update ||= Numo::DFloat.zeros(weight.shape[0])
+        @moment = @params[:decay] * @moment + (1.0 - @params[:decay]) * gradient**2
+        @update = @params[:momentum] * @update - (@params[:learning_rate] / (@moment**0.5 + 1.0e-8)) * gradient
+        weight + @update
+      end
+
+      # Dump marshal data.
+      # @return [Hash] The marshal data.
+      def marshal_dump
+        { params: @params,
+          moment: @moment,
+          update: @update }
+      end
+
+      # Load marshal data.
+      # @return [nil]
+      def marshal_load(obj)
+        @params = obj[:params]
+        @moment = obj[:moment]
+        @update = obj[:update]
+        nil
+      end
+    end
+  end
+end

data/lib/svmkit/optimizer/sgd.rb

@@ -0,0 +1,65 @@
+# frozen_string_literal: true
+
+require 'svmkit/validation'
+require 'svmkit/base/base_estimator'
+
+module SVMKit
+  module Optimizer
+    # SGD is a class that implements SGD optimizer.
+    #
+    # @example
+    #   optimizer = SVMKit::Optimizer::SGD.new(learning_rate: 0.01, momentum: 0.9, decay: 0.9)
+    #   estimator = SVMKit::LinearModel::LinearRegression.new(optimizer: optimizer, random_seed: 1)
+    #   estimator.fit(samples, values)
+    class SGD
+      include Base::BaseEstimator
+      include Validation
+
+      # Create a new optimizer with SGD.
+      #
+      # @param learning_rate [Float] The initial value of learning rate.
+      # @param momentum [Float] The initial value of momentum.
+      # @param decay [Float] The smooting parameter.
+      def initialize(learning_rate: 0.01, momentum: 0.0, decay: 0.0)
+        check_params_float(learning_rate: learning_rate, momentum: momentum, decay: decay)
+        check_params_positive(learning_rate: learning_rate, momentum: momentum, decay: decay)
+        @params = {}
+        @params[:learning_rate] = learning_rate
+        @params[:momentum] = momentum
+        @params[:decay] = decay
+        @iter = 0
+        @update = nil
+      end
+
+      # Calculate the updated weight with SGD.
+      #
+      # @param weight [Numo::DFloat] (shape: [n_features]) The weight to be updated.
+      # @param gradient [Numo::DFloat] (shape: [n_features]) The gradient for updating the weight.
+      # @return [Numo::DFloat] (shape: [n_feautres]) The updated weight.
+      def call(weight, gradient)
+        @update ||= Numo::DFloat.zeros(weight.shape[0])
+        current_learning_rate = @params[:learning_rate] / (1.0 + @params[:decay] * @iter)
+        @iter += 1
+        @update = @params[:momentum] * @update - current_learning_rate * gradient
+        weight + @update
+      end
+
+      # Dump marshal data.
+      # @return [Hash] The marshal data.
+      def marshal_dump
+        { params: @params,
+          iter: @iter,
+          update: @update }
+      end
+
+      # Load marshal data.
+      # @return [nil]
+      def marshal_load(obj)
+        @params = obj[:params]
+        @iter = obj[:iter]
+        @update = obj[:update]
+        nil
+      end
+    end
+  end
+end

data/lib/svmkit/optimizer/yellow_fin.rb

@@ -0,0 +1,144 @@
+# frozen_string_literal: true
+
+require 'svmkit/validation'
+require 'svmkit/base/base_estimator'
+
+module SVMKit
+  module Optimizer
+    # YellowFin is a class that implements YellowFin optimizer.
+    #
+    # @example
+    #   optimizer = SVMKit::Optimizer::YellowFin.new(learning_rate: 0.01, momentum: 0.9, decay: 0.999, window_width: 20)
+    #   estimator = SVMKit::LinearModel::LinearRegression.new(optimizer: optimizer, random_seed: 1)
+    #   estimator.fit(samples, values)
+    #
+    # *Reference*
+    # - J. Zhang and I. Mitliagkas, "YellowFin and the Art of Momentum Tuning," CoRR abs/1706.03471, 2017.
+    class YellowFin
+      include Base::BaseEstimator
+      include Validation
+
+      # Create a new optimizer with YellowFin.
+      #
+      # @param learning_rate [Float] The initial value of learning rate.
+      # @param momentum [Float] The initial value of momentum.
+      # @param decay [Float] The smooting parameter.
+      # @param window_width [Integer] The sliding window width for searching curvature range.
+      def initialize(learning_rate: 0.01, momentum: 0.9, decay: 0.999, window_width: 20)
+        check_params_float(learning_rate: learning_rate, momentum: momentum, decay: decay)
+        check_params_integer(window_width: window_width)
+        check_params_positive(learning_rate: learning_rate, momentum: momentum, decay: decay, window_width: window_width)
+        @params = {}
+        @params[:learning_rate] = learning_rate
+        @params[:momentum] = momentum
+        @params[:decay] = decay
+        @params[:window_width] = window_width
+        @smth_learning_rate = learning_rate
+        @smth_momentum = momentum
+        @grad_norms = nil
+        @grad_norm_min = 0.0
+        @grad_norm_max = 0.0
+        @grad_mean_sqr = 0.0
+        @grad_mean = 0.0
+        @grad_var = 0.0
+        @grad_norm_mean = 0.0
+        @curve_mean = 0.0
+        @distance_mean = 0.0
+        @update = nil
+      end
+
+      # Calculate the updated weight with adaptive momentum coefficient and learning rate.
+      #
+      # @param weight [Numo::DFloat] (shape: [n_features]) The weight to be updated.
+      # @param gradient [Numo::DFloat] (shape: [n_features]) The gradient for updating the weight.
+      # @return [Numo::DFloat] (shape: [n_feautres]) The updated weight.
+      def call(weight, gradient)
+        @update ||= Numo::DFloat.zeros(weight.shape[0])
+        curvature_range(gradient)
+        gradient_variance(gradient)
+        distance_to_optimum(gradient)
+        @smth_momentum = @params[:decay] * @smth_momentum + (1 - @params[:decay]) * current_momentum
+        @smth_learning_rate = @params[:decay] * @smth_learning_rate + (1 - @params[:decay]) * current_learning_rate
+        @update = @smth_momentum * @update - @smth_learning_rate * gradient
+        weight + @update
+      end
+
+      private
+
+      def current_momentum
+        dr = Math.sqrt(@grad_norm_max / @grad_norm_min + 1.0e-8)
+        [cubic_root**2, ((dr - 1) / (dr + 1))**2].max
+      end
+
+      def current_learning_rate
+        (1.0 - Math.sqrt(@params[:momentum]))**2 / (@grad_norm_min + 1.0e-8)
+      end
+
+      def cubic_root
+        p = (@distance_mean**2 * @grad_norm_min**2) / (2 * @grad_var + 1.0e-8)
+        w3 = (-Math.sqrt(p**2 + 4.fdiv(27) * p**3) - p).fdiv(2)
+        w = (w3 >= 0.0 ? 1 : -1) * w3.abs**1.fdiv(3)
+        y = w - p / (3 * w + 1.0e-8)
+        y + 1
+      end
+
+      def curvature_range(gradient)
+        @grad_norms ||= []
+        @grad_norms.push((gradient**2).sum)
+        @grad_norms.shift(@grad_norms.size - @params[:window_width]) if @grad_norms.size > @params[:window_width]
+        @grad_norm_min = @params[:decay] * @grad_norm_min + (1 - @params[:decay]) * @grad_norms.min
+        @grad_norm_max = @params[:decay] * @grad_norm_max + (1 - @params[:decay]) * @grad_norms.max
+      end
+
+      def gradient_variance(gradient)
+        @grad_mean_sqr = @params[:decay] * @grad_mean_sqr + (1 - @params[:decay]) * gradient**2
+        @grad_mean = @params[:decay] * @grad_mean + (1 - @params[:decay]) * gradient
+        @grad_var = (@grad_mean_sqr - @grad_mean**2).sum
+      end
+
+      def distance_to_optimum(gradient)
+        grad_sqr = (gradient**2).sum
+        @grad_norm_mean = @params[:decay] * @grad_norm_mean + (1 - @params[:decay]) * Math.sqrt(grad_sqr + 1.0e-8)
+        @curve_mean = @params[:decay] * @curve_mean + (1 - @params[:decay]) * grad_sqr
+        @distance_mean = @params[:decay] * @distance_mean + (1 - @params[:decay]) * (@grad_norm_mean / @curve_mean)
+      end
+
+      # Dump marshal data.
+      # @return [Hash] The marshal data.
+      def marshal_dump
+        { params: @params,
+          smth_learning_rate: @smth_learning_rate,
+          smth_momentum: @smth_momentum,
+          grad_norms: @grad_norms,
+          grad_norm_min: @grad_norm_min,
+          grad_norm_max: @grad_norm_max,
+          grad_mean_sqr: @grad_mean_sqr,
+          grad_mean: @grad_mean,
+          grad_var: @grad_var,
+          grad_norm_mean: @grad_norm_mean,
+          curve_mean: @curve_mean,
+          distance_mean: @distance_mean,
+          update: @update }
+      end
+
+      # Load marshal data.
+      # @return [nis]
+      def marshal_load(obj)
+        @params = obj[:params]
+        @smth_learning_rate = obj[:smth_learning_rate]
+        @smth_momentum = obj[:smth_momentum]
+        @grad_norms = obj[:grad_norms]
+        @grad_norm_min = obj[:grad_norm_min]
+        @grad_norm_max = obj[:grad_norm_max]
+        @grad_mean_sqr = obj[:grad_mean_sqr]
+        @grad_mean = obj[:grad_mean]
+        @grad_var = obj[:grad_var]
+        @grad_norm_mean = obj[:grad_norm_mean]
+        @curve_mean = obj[:curve_mean]
+        @distance_mean = obj[:distance_mean]
+        @update = obj[:update]
+        nil
+      end
+    end
+  end
+end

data/lib/svmkit/polynomial_model/factorization_machine_classifier.rb

@@ -21,8 +21,8 @@ module SVMKit
     #   results = estimator.predict(testing_samples)
     #
     # *Reference*
-    # - S. Rendle, "Factorization Machines with libFM," ACM Transactions on Intelligent Systems and Technology, vol. 3 (3), pp. 57:1--57:22, 2012.
-    # - S. Rendle, "Factorization Machines," Proceedings of the 10th IEEE International Conference on Data Mining (ICDM'10), pp. 995--1000, 2010.
+    # - S. Rendle, "Factorization Machines with libFM," ACM TIST, vol. 3 (3), pp. 57:1--57:22, 2012.
+    # - S. Rendle, "Factorization Machines," Proc. ICDM'10, pp. 995--1000, 2010.
     class FactorizationMachineClassifier
       include Base::BaseEstimator
       include Base::Classifier
@@ -57,7 +57,7 @@ module SVMKit
       # @param max_iter [Integer] The maximum number of iterations.
       # @param batch_size [Integer] The size of the mini batches.
       # @param optimizer [Optimizer] The optimizer to calculate adaptive learning rate.
-      #   Nadam is selected automatically on current version.
+      #   If nil is given, Nadam is used.
       # @param random_seed [Integer] The seed value using to initialize the random generator.
       def initialize(n_factors: 2, loss: 'hinge', reg_param_linear: 1.0, reg_param_factor: 1.0,
                      max_iter: 1000, batch_size: 10, optimizer: nil, random_seed: nil)
@@ -76,6 +76,7 @@ module SVMKit
         @params[:max_iter] = max_iter
         @params[:batch_size] = batch_size
         @params[:optimizer] = optimizer
+        @params[:optimizer] ||= Optimizer::Nadam.new
         @params[:random_seed] = random_seed
         @params[:random_seed] ||= srand
         @factor_mat = nil
@@ -105,10 +106,7 @@ module SVMKit
           @bias_term = Numo::DFloat.zeros(n_classes)
           n_classes.times do |n|
             bin_y = Numo::Int32.cast(y.eq(@classes[n])) * 2 - 1
-            factor, weight, bias = binary_fit(x, bin_y)
-            @factor_mat[n, true, true] = factor
-            @weight_vec[n, true] = weight
-            @bias_term[n] = bias
+            @factor_mat[n, true, true], @weight_vec[n, true], @bias_term[n] = binary_fit(x, bin_y)
           end
         else
           negative_label = y.to_a.uniq.min
@@ -194,8 +192,8 @@ module SVMKit
         rand_ids = [*0...n_samples].shuffle(random: @rng)
         weight_vec = Numo::DFloat.zeros(n_features + 1)
         factor_mat = Numo::DFloat.zeros(@params[:n_factors], n_features)
-        weight_optimizer = Optimizer::Nadam.new
-        factor_optimizers = Array.new(@params[:n_factors]) { Optimizer::Nadam.new }
+        weight_optimizer = @params[:optimizer].dup
+        factor_optimizers = Array.new(@params[:n_factors]) { @params[:optimizer].dup }
         # Start optimization.
         @params[:max_iter].times do |_t|
           # Random sampling.

data/lib/svmkit/polynomial_model/factorization_machine_regressor.rb

@@ -19,8 +19,8 @@ module SVMKit
     #   results = estimator.predict(testing_samples)
     #
     # *Reference*
-    # - S. Rendle, "Factorization Machines with libFM," ACM Transactions on Intelligent Systems and Technology, vol. 3 (3), pp. 57:1--57:22, 2012.
-    # - S. Rendle, "Factorization Machines," Proc. the 10th IEEE International Conference on Data Mining (ICDM'10), pp. 995--1000, 2010.
+    # - S. Rendle, "Factorization Machines with libFM," ACM TIST, vol. 3 (3), pp. 57:1--57:22, 2012.
+    # - S. Rendle, "Factorization Machines," Proc. ICDM'10, pp. 995--1000, 2010.
     class FactorizationMachineRegressor
       include Base::BaseEstimator
       include Base::Regressor
@@ -50,7 +50,7 @@ module SVMKit
       # @param max_iter [Integer] The maximum number of iterations.
       # @param batch_size [Integer] The size of the mini batches.
       # @param optimizer [Optimizer] The optimizer to calculate adaptive learning rate.
-      #   Nadam is selected automatically on current version.
+      #   If nil is given, Nadam is used.
       # @param random_seed [Integer] The seed value using to initialize the random generator.
       def initialize(n_factors: 2, reg_param_linear: 1.0, reg_param_factor: 1.0,
                      max_iter: 1000, batch_size: 10, optimizer: nil, random_seed: nil)
@@ -66,6 +66,7 @@ module SVMKit
         @params[:max_iter] = max_iter
         @params[:batch_size] = batch_size
         @params[:optimizer] = optimizer
+        @params[:optimizer] ||= Optimizer::Nadam.new
         @params[:random_seed] = random_seed
         @params[:random_seed] ||= srand
         @factor_mat = nil
@@ -91,12 +92,7 @@ module SVMKit
           @factor_mat = Numo::DFloat.zeros(n_outputs, @params[:n_factors], n_features)
           @weight_vec = Numo::DFloat.zeros(n_outputs, n_features)
           @bias_term = Numo::DFloat.zeros(n_outputs)
-          n_outputs.times do |n|
-            factor, weight, bias = single_fit(x, y[true, n])
-            @factor_mat[n, true, true] = factor
-            @weight_vec[n, true] = weight
-            @bias_term[n] = bias
-          end
+          n_outputs.times { |n| @factor_mat[n, true, true], @weight_vec[n, true], @bias_term[n] = single_fit(x, y[true, n]) }
         else
           @factor_mat, @weight_vec, @bias_term = single_fit(x, y)
         end
@@ -148,8 +144,8 @@ module SVMKit
         rand_ids = [*0...n_samples].shuffle(random: @rng)
         weight_vec = Numo::DFloat.zeros(n_features + 1)
         factor_mat = Numo::DFloat.zeros(@params[:n_factors], n_features)
-        weight_optimizer = Optimizer::Nadam.new
-        factor_optimizers = Array.new(@params[:n_factors]) { Optimizer::Nadam.new }
+        weight_optimizer = @params[:optimizer].dup
+        factor_optimizers = Array.new(@params[:n_factors]) { @params[:optimizer].dup }
         # Start optimization.
         @params[:max_iter].times do |_t|
           # Random sampling.

data/lib/svmkit/version.rb

@@ -3,5 +3,5 @@
 # SVMKit is a machine learning library in Ruby.
 module SVMKit
   # @!visibility private
-  VERSION = '0.4.0'.freeze
+  VERSION = '0.4.1'.freeze
 end

data/svmkit.gemspec

@@ -17,8 +17,8 @@ MSG
 SVMKit is a machine learninig library in Ruby.
 SVMKit provides machine learning algorithms with interfaces similar to Scikit-Learn in Python.
 SVMKit currently supports Linear / Kernel Support Vector Machine,
-Logistic Regression, Ridge, Lasso, Factorization Machine, Naive Bayes, Decision Tree, Random Forest,
-K-nearest neighbor algorithm, and cross-validation.
+Logistic Regression, Linear Regression, Ridge, Lasso, Factorization Machine,
+Naive Bayes, Decision Tree, Random Forest, K-nearest neighbor algorithm, and cross-validation.
 MSG
   spec.homepage      = 'https://github.com/yoshoku/svmkit'
   spec.license       = 'BSD-2-Clause'

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: svmkit
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.4.1
 platform: ruby
 authors:
 - yoshoku
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2018-06-02 00:00:00.000000000 Z
+date: 2018-06-08 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: numo-narray
@@ -84,8 +84,8 @@ description: |
   SVMKit is a machine learninig library in Ruby.
   SVMKit provides machine learning algorithms with interfaces similar to Scikit-Learn in Python.
   SVMKit currently supports Linear / Kernel Support Vector Machine,
-  Logistic Regression, Ridge, Lasso, Factorization Machine, Naive Bayes, Decision Tree, Random Forest,
-  K-nearest neighbor algorithm, and cross-validation.
+  Logistic Regression, Linear Regression, Ridge, Lasso, Factorization Machine,
+  Naive Bayes, Decision Tree, Random Forest, K-nearest neighbor algorithm, and cross-validation.
 email:
 - yoshoku@outlook.com
 executables: []
@@ -128,6 +128,7 @@ files:
 - lib/svmkit/kernel_approximation/rbf.rb
 - lib/svmkit/kernel_machine/kernel_svc.rb
 - lib/svmkit/linear_model/lasso.rb
+- lib/svmkit/linear_model/linear_regression.rb
 - lib/svmkit/linear_model/logistic_regression.rb
 - lib/svmkit/linear_model/ridge.rb
 - lib/svmkit/linear_model/svc.rb
@@ -140,6 +141,9 @@ files:
 - lib/svmkit/nearest_neighbors/k_neighbors_classifier.rb
 - lib/svmkit/nearest_neighbors/k_neighbors_regressor.rb
 - lib/svmkit/optimizer/nadam.rb
+- lib/svmkit/optimizer/rmsprop.rb
+- lib/svmkit/optimizer/sgd.rb
+- lib/svmkit/optimizer/yellow_fin.rb
 - lib/svmkit/pairwise_metric.rb
 - lib/svmkit/polynomial_model/factorization_machine_classifier.rb
 - lib/svmkit/polynomial_model/factorization_machine_regressor.rb