svmkit 0.3.2 → 0.3.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 93ce9c2e79ac158b4a3e988afc547b1891419eb6e6b1845156cf98eaa3cdd578
-  data.tar.gz: 4e677653deebd035cbdcd5c98529b7f4fee6804075ecab1113dbccc0bf9c65ed
+  metadata.gz: 9c0a64cc46c00a252946033b072b4d9498fb4d5cf7131830333483a336c29315
+  data.tar.gz: 1eb9415f08167772764f1eba4e67f6a3479768db75efcc32a8de85276440d41c
 SHA512:
-  metadata.gz: 7518039557e3c991c4a0cc112764198ed6340c8be1fa9c3fb746be21ffbb5518dd35651149cda5aba8ef52a36dfa6b17f47e1335893ae0cd1dfc5776a0e6bf8e
-  data.tar.gz: c062d9c2a7c04be82787a4d76a970855c2dc8ce0d4bf5531b6196b96873f4f8e679ee434af9a56c2ebf56ae9a8adb33387925050eb9b8964da613318f2a0e430
+  metadata.gz: 2f994dad593e5b752c2a062507f849483a9e4dbdd90190313b672c2f8cd9c9ed102b2fc088823665812f44e6b549bc67cab7d16fb545031bae0b57e713c3c3c3
+  data.tar.gz: d6da2f56721b8d264898fea922e2bee016987d898b4169f12bc6963044b69a4952f25d1b75380a93ce1fccd9854bb86e085c18fdf9535e273bd2eb1a328d3b98

data/HISTORY.md

@@ -1,3 +1,9 @@
+# 0.3.3
+- Add class for Ridge regressor.
+- Add class for Lasso regressor.
+- Fix bug on gradient calculation of FactorizationMachineRegressor.
+- Fix some documents.
+
 # 0.3.2
 - Add class for Factorization Machine regressor.
 - Add class for Decision Tree regressor.

data/README.md

@@ -8,7 +8,7 @@
 SVMKit is a machine learninig library in Ruby.
 SVMKit provides machine learning algorithms with interfaces similar to Scikit-Learn in Python.
 SVMKit currently supports Linear / Kernel Support Vector Machine,
-Logistic Regression, Factorization Machine, Naive Bayes, Decision Tree, Random Forest,
+Logistic Regression, Ridge, Lasso, Factorization Machine, Naive Bayes, Decision Tree, Random Forest,
 K-nearest neighbor classifier, and cross-validation.
 
 ## Installation

data/lib/svmkit.rb

@@ -17,6 +17,8 @@ require 'svmkit/kernel_approximation/rbf'
 require 'svmkit/linear_model/svc'
 require 'svmkit/linear_model/svr'
 require 'svmkit/linear_model/logistic_regression'
+require 'svmkit/linear_model/ridge'
+require 'svmkit/linear_model/lasso'
 require 'svmkit/kernel_machine/kernel_svc'
 require 'svmkit/polynomial_model/factorization_machine_classifier'
 require 'svmkit/polynomial_model/factorization_machine_regressor'

data/lib/svmkit/ensemble/random_forest_classifier.rb

@@ -155,7 +155,7 @@ module SVMKit
       end
 
       # Dump marshal data.
-      # @return [Hash] The marshal data about RandomForestClassifier
+      # @return [Hash] The marshal data about RandomForestClassifier.
       def marshal_dump
         { params: @params, estimators: @estimators, classes: @classes,
           feature_importances: @feature_importances, rng: @rng }

data/lib/svmkit/ensemble/random_forest_regressor.rb

@@ -1,6 +1,5 @@
 # frozen_string_literal: true
 
-require 'pp'
 require 'svmkit/validation'
 require 'svmkit/base/base_estimator'
 require 'svmkit/base/regressor'
@@ -119,7 +118,7 @@ module SVMKit
       end
 
       # Dump marshal data.
-      # @return [Hash] The marshal data about RandomForestRegressor
+      # @return [Hash] The marshal data about RandomForestRegressor.
       def marshal_dump
         { params: @params,
           estimators: @estimators,

data/lib/svmkit/linear_model/lasso.rb

@@ -0,0 +1,199 @@
+# frozen_string_literal: true
+
+require 'svmkit/validation'
+require 'svmkit/base/base_estimator'
+require 'svmkit/base/regressor'
+
+module SVMKit
+  module LinearModel
+    # Lasso is a class that implements Lasso Regression
+    # with stochastic gradient descent (SGD) optimization.
+    #
+    # @example
+    #   estimator =
+    #     SVMKit::LinearModel::Lasso.new(reg_param: 0.1, max_iter: 5000, batch_size: 50, random_seed: 1)
+    #   estimator.fit(training_samples, traininig_values)
+    #   results = estimator.predict(testing_samples)
+    #
+    # *Reference*
+    # - S. Shalev-Shwartz and Y. Singer, "Pegasos: Primal Estimated sub-GrAdient SOlver for SVM," Proc. ICML'07, pp. 807--814, 2007.
+    # - L. Bottou, "Large-Scale Machine Learning with Stochastic Gradient Descent," Proc. COMPSTAT'10, pp. 177--186, 2010.
+    # - I. Sutskever, J. Martens, G. Dahl, and G. Hinton, "On the importance of initialization and momentum in deep learning," Proc. ICML'13, pp. 1139--1147, 2013.
+    # - G. Hinton, N. Srivastava, and K. Swersky, "Lecture 6e rmsprop," Neural Networks for Machine Learning, 2012.
+    class Lasso
+      include Base::BaseEstimator
+      include Base::Regressor
+      include Validation
+
+      # Return the weight vector.
+      # @return [Numo::DFloat] (shape: [n_outputs, n_features])
+      attr_reader :weight_vec
+
+      # Return the bias term (a.k.a. intercept).
+      # @return [Numo::DFloat] (shape: [n_outputs])
+      attr_reader :bias_term
+
+      # Return the random generator for random sampling.
+      # @return [Random]
+      attr_reader :rng
+
+      # Create a new Lasso regressor.
+      #
+      # @param reg_param [Float] The regularization parameter.
+      # @param fit_bias [Boolean] The flag indicating whether to fit the bias term.
+      # @param learning_rate [Float] The learning rate for optimization.
+      # @param decay [Float] The discounting factor for RMS prop optimization.
+      # @param momentum [Float] The momentum for optimization.
+      # @param max_iter [Integer] The maximum number of iterations.
+      # @param batch_size [Integer] The size of the mini batches.
+      # @param random_seed [Integer] The seed value using to initialize the random generator.
+      def initialize(reg_param: 1.0, fit_bias: false, learning_rate: 0.01, decay: 0.9, momentum: 0.9,
+                     max_iter: 1000, batch_size: 10, random_seed: nil)
+        check_params_float(reg_param: reg_param,
+                           learning_rate: learning_rate, decay: decay, momentum: momentum)
+        check_params_integer(max_iter: max_iter, batch_size: batch_size)
+        check_params_boolean(fit_bias: fit_bias)
+        check_params_type_or_nil(Integer, random_seed: random_seed)
+        check_params_positive(reg_param: reg_param,
+                              learning_rate: learning_rate, decay: decay, momentum: momentum,
+                              max_iter: max_iter, batch_size: batch_size)
+        @params = {}
+        @params[:reg_param] = reg_param
+        @params[:fit_bias] = fit_bias
+        @params[:learning_rate] = learning_rate
+        @params[:decay] = decay
+        @params[:momentum] = momentum
+        @params[:max_iter] = max_iter
+        @params[:batch_size] = batch_size
+        @params[:random_seed] = random_seed
+        @params[:random_seed] ||= srand
+        @weight_vec = nil
+        @bias_term = nil
+        @rng = Random.new(@params[:random_seed])
+      end
+
+      # Fit the model with given training data.
+      #
+      # @param x [Numo::DFloat] (shape: [n_samples, n_features]) The training data to be used for fitting the model.
+      # @param y [Numo::Int32] (shape: [n_samples, n_outputs]) The target values to be used for fitting the model.
+      # @return [Lasso] The learned regressor itself.
+      def fit(x, y)
+        check_sample_array(x)
+        check_tvalue_array(y)
+        check_sample_tvalue_size(x, y)
+
+        n_outputs = y.shape[1].nil? ? 1 : y.shape[1]
+        _n_samples, n_features = x.shape
+
+        if n_outputs > 1
+          @weight_vec = Numo::DFloat.zeros(n_outputs, n_features)
+          @bias_term = Numo::DFloat.zeros(n_outputs)
+          n_outputs.times do |n|
+            weight, bias = single_fit(x, y[true, n])
+            @weight_vec[n, true] = weight
+            @bias_term[n] = bias
+          end
+        else
+          @weight_vec, @bias_term = single_fit(x, y)
+        end
+
+        self
+      end
+
+      # Predict values for samples.
+      #
+      # @param x [Numo::DFloat] (shape: [n_samples, n_features]) The samples to predict the values.
+      # @return [Numo::DFloat] (shape: [n_samples, n_outputs]) Predicted values per sample.
+      def predict(x)
+        check_sample_array(x)
+        x.dot(@weight_vec.transpose) + @bias_term
+      end
+
+      # Dump marshal data.
+      # @return [Hash] The marshal data about Lasso.
+      def marshal_dump
+        { params: @params,
+          weight_vec: @weight_vec,
+          bias_term: @bias_term,
+          rng: @rng }
+      end
+
+      # Load marshal data.
+      # @return [nil]
+      def marshal_load(obj)
+        @params = obj[:params]
+        @weight_vec = obj[:weight_vec]
+        @bias_term = obj[:bias_term]
+        @rng = obj[:rng]
+        nil
+      end
+
+      private
+
+      def single_fit(x, y)
+        # Expand feature vectors for bias term.
+        samples = @params[:fit_bias] ? expand_feature(x) : x
+        # Initialize some variables.
+        n_samples, n_features = samples.shape
+        rand_ids = [*0...n_samples].shuffle(random: @rng)
+        weight_vec = Numo::DFloat.zeros(n_features)
+        left_weight_vec = Numo::DFloat.zeros(n_features)
+        left_weight_sqrsum = Numo::DFloat.zeros(n_features)
+        left_weight_update = Numo::DFloat.zeros(n_features)
+        right_weight_vec = Numo::DFloat.zeros(n_features)
+        right_weight_sqrsum = Numo::DFloat.zeros(n_features)
+        right_weight_update = Numo::DFloat.zeros(n_features)
+        # Start optimization.
+        @params[:max_iter].times do |_t|
+          # Random sampling.
+          subset_ids = rand_ids.shift(@params[:batch_size])
+          rand_ids.concat(subset_ids)
+          data = samples[subset_ids, true]
+          values = y[subset_ids]
+          # Calculate gradients for loss function.
+          loss_grad = loss_gradient(data, values, weight_vec)
+          next if loss_grad.ne(0.0).count.zero?
+          # Update weight.
+          left_weight_vec, left_weight_sqrsum, left_weight_update =
+            update_weight(left_weight_vec, left_weight_sqrsum, left_weight_update,
+                          left_weight_gradient(loss_grad, data))
+          right_weight_vec, right_weight_sqrsum, right_weight_update =
+            update_weight(right_weight_vec, right_weight_sqrsum, right_weight_update,
+                          right_weight_gradient(loss_grad, data))
+          weight_vec = left_weight_vec - right_weight_vec
+        end
+        split_weight_vec_bias(weight_vec)
+      end
+
+      def loss_gradient(x, y, weight)
+        2.0 * (x.dot(weight) - y)
+      end
+
+      def left_weight_gradient(loss_grad, data)
+        ((@params[:reg_param] + loss_grad).expand_dims(1) * data).mean(0)
+      end
+
+      def right_weight_gradient(loss_grad, data)
+        ((@params[:reg_param] - loss_grad).expand_dims(1) * data).mean(0)
+      end
+
+      def update_weight(weight, sqrsum, update, gr)
+        new_sqrsum = @params[:decay] * sqrsum + (1.0 - @params[:decay]) * gr**2
+        new_update = (@params[:learning_rate] / ((new_sqrsum + 1.0e-8)**0.5)) * gr
+        new_weight = weight - (new_update + @params[:momentum] * update)
+        new_weight = 0.5 * (new_weight + new_weight.abs)
+        [new_weight, new_sqrsum, new_update]
+      end
+
+      def expand_feature(x)
+        Numo::NArray.hstack([x, Numo::DFloat.ones([x.shape[0], 1])])
+      end
+
+      def split_weight_vec_bias(weight_vec)
+        weights = @params[:fit_bias] ? weight_vec[0...-1] : weight_vec
+        bias = @params[:fit_bias] ? weight_vec[-1] : 0.0
+        [weights, bias]
+      end
+    end
+  end
+end

data/lib/svmkit/linear_model/ridge.rb

@@ -0,0 +1,185 @@
+# frozen_string_literal: true
+
+require 'svmkit/validation'
+require 'svmkit/base/base_estimator'
+require 'svmkit/base/regressor'
+
+module SVMKit
+  module LinearModel
+    # Ridge is a class that implements Ridge Regression
+    # with stochastic gradient descent (SGD) optimization.
+    #
+    # @example
+    #   estimator =
+    #     SVMKit::LinearModel::Ridge.new(reg_param: 0.1, max_iter: 5000, batch_size: 50, random_seed: 1)
+    #   estimator.fit(training_samples, traininig_values)
+    #   results = estimator.predict(testing_samples)
+    #
+    # *Reference*
+    # - S. Shalev-Shwartz and Y. Singer, "Pegasos: Primal Estimated sub-GrAdient SOlver for SVM," Proc. ICML'07, pp. 807--814, 2007.
+    # - I. Sutskever, J. Martens, G. Dahl, and G. Hinton, "On the importance of initialization and momentum in deep learning," Proc. ICML'13, pp. 1139--1147, 2013.
+    # - G. Hinton, N. Srivastava, and K. Swersky, "Lecture 6e rmsprop," Neural Networks for Machine Learning, 2012.
+    class Ridge
+      include Base::BaseEstimator
+      include Base::Regressor
+      include Validation
+
+      # Return the weight vector.
+      # @return [Numo::DFloat] (shape: [n_outputs, n_features])
+      attr_reader :weight_vec
+
+      # Return the bias term (a.k.a. intercept).
+      # @return [Numo::DFloat] (shape: [n_outputs])
+      attr_reader :bias_term
+
+      # Return the random generator for random sampling.
+      # @return [Random]
+      attr_reader :rng
+
+      # Create a new Ridge regressor.
+      #
+      # @param reg_param [Float] The regularization parameter.
+      # @param fit_bias [Boolean] The flag indicating whether to fit the bias term.
+      # @param learning_rate [Float] The learning rate for optimization.
+      # @param decay [Float] The discounting factor for RMS prop optimization.
+      # @param momentum [Float] The Nesterov momentum for optimization.
+      # @param max_iter [Integer] The maximum number of iterations.
+      # @param batch_size [Integer] The size of the mini batches.
+      # @param random_seed [Integer] The seed value using to initialize the random generator.
+      def initialize(reg_param: 1.0, fit_bias: false, learning_rate: 0.01, decay: 0.9, momentum: 0.9,
+                     max_iter: 1000, batch_size: 10, random_seed: nil)
+        check_params_float(reg_param: reg_param,
+                           learning_rate: learning_rate, decay: decay, momentum: momentum)
+        check_params_integer(max_iter: max_iter, batch_size: batch_size)
+        check_params_boolean(fit_bias: fit_bias)
+        check_params_type_or_nil(Integer, random_seed: random_seed)
+        check_params_positive(reg_param: reg_param,
+                              learning_rate: learning_rate, decay: decay, momentum: momentum,
+                              max_iter: max_iter, batch_size: batch_size)
+        @params = {}
+        @params[:reg_param] = reg_param
+        @params[:fit_bias] = fit_bias
+        @params[:learning_rate] = learning_rate
+        @params[:decay] = decay
+        @params[:momentum] = momentum
+        @params[:max_iter] = max_iter
+        @params[:batch_size] = batch_size
+        @params[:random_seed] = random_seed
+        @params[:random_seed] ||= srand
+        @weight_vec = nil
+        @bias_term = nil
+        @rng = Random.new(@params[:random_seed])
+      end
+
+      # Fit the model with given training data.
+      #
+      # @param x [Numo::DFloat] (shape: [n_samples, n_features]) The training data to be used for fitting the model.
+      # @param y [Numo::Int32] (shape: [n_samples, n_outputs]) The target values to be used for fitting the model.
+      # @return [Ridge] The learned regressor itself.
+      def fit(x, y)
+        check_sample_array(x)
+        check_tvalue_array(y)
+        check_sample_tvalue_size(x, y)
+
+        n_outputs = y.shape[1].nil? ? 1 : y.shape[1]
+        _n_samples, n_features = x.shape
+
+        if n_outputs > 1
+          @weight_vec = Numo::DFloat.zeros(n_outputs, n_features)
+          @bias_term = Numo::DFloat.zeros(n_outputs)
+          n_outputs.times do |n|
+            weight, bias = single_fit(x, y[true, n])
+            @weight_vec[n, true] = weight
+            @bias_term[n] = bias
+          end
+        else
+          @weight_vec, @bias_term = single_fit(x, y)
+        end
+
+        self
+      end
+
+      # Predict values for samples.
+      #
+      # @param x [Numo::DFloat] (shape: [n_samples, n_features]) The samples to predict the values.
+      # @return [Numo::DFloat] (shape: [n_samples, n_outputs]) Predicted values per sample.
+      def predict(x)
+        check_sample_array(x)
+        x.dot(@weight_vec.transpose) + @bias_term
+      end
+
+      # Dump marshal data.
+      # @return [Hash] The marshal data about Ridge.
+      def marshal_dump
+        { params: @params,
+          weight_vec: @weight_vec,
+          bias_term: @bias_term,
+          rng: @rng }
+      end
+
+      # Load marshal data.
+      # @return [nil]
+      def marshal_load(obj)
+        @params = obj[:params]
+        @weight_vec = obj[:weight_vec]
+        @bias_term = obj[:bias_term]
+        @rng = obj[:rng]
+        nil
+      end
+
+      private
+
+      def single_fit(x, y)
+        # Expand feature vectors for bias term.
+        samples = @params[:fit_bias] ? expand_feature(x) : x
+        # Initialize some variables.
+        n_samples, n_features = samples.shape
+        rand_ids = [*0...n_samples].shuffle(random: @rng)
+        weight_vec = Numo::DFloat.zeros(n_features)
+        weight_sqrsum = Numo::DFloat.zeros(n_features)
+        weight_update = Numo::DFloat.zeros(n_features)
+        # Start optimization.
+        @params[:max_iter].times do |_t|
+          # Random sampling.
+          subset_ids = rand_ids.shift(@params[:batch_size])
+          rand_ids.concat(subset_ids)
+          data = samples[subset_ids, true]
+          values = y[subset_ids]
+          # Calculate gradients for loss function.
+          loss_grad = loss_gradient(data, values, weight_vec - @params[:momentum] * weight_update)
+          next if loss_grad.ne(0.0).count.zero?
+          # Update weight.
+          weight_vec, weight_sqrsum, weight_update =
+            update_weight(weight_vec, weight_sqrsum, weight_update,
+                          weight_gradient(loss_grad, data, weight_vec - @params[:momentum] * weight_update))
+        end
+        split_weight_vec_bias(weight_vec)
+      end
+
+      def loss_gradient(x, y, weight)
+        2.0 * (x.dot(weight) - y)
+      end
+
+      def weight_gradient(loss_grad, data, weight)
+        (loss_grad.expand_dims(1) * data).mean(0) + @params[:reg_param] * weight
+      end
+
+      def update_weight(weight, sqrsum, update, gr)
+        new_sqrsum = @params[:decay] * sqrsum + (1.0 - @params[:decay]) * gr**2
+        new_update = (@params[:learning_rate] / ((new_sqrsum + 1.0e-8)**0.5)) * gr
+        new_weight = weight - (new_update + @params[:momentum] * update)
+        [new_weight, new_sqrsum, new_update]
+      end
+
+      def expand_feature(x)
+        Numo::NArray.hstack([x, Numo::DFloat.ones([x.shape[0], 1])])
+      end
+
+      def split_weight_vec_bias(weight_vec)
+        weights = @params[:fit_bias] ? weight_vec[0...-1] : weight_vec
+        bias = @params[:fit_bias] ? weight_vec[-1] : 0.0
+        [weights, bias]
+      end
+    end
+  end
+end

data/lib/svmkit/linear_model/svr.rb

@@ -21,11 +21,11 @@ module SVMKit
       include Base::BaseEstimator
       include Base::Regressor
 
-      # Return the weight vector for SVC.
+      # Return the weight vector for SVR.
       # @return [Numo::DFloat] (shape: [n_outputs, n_features])
       attr_reader :weight_vec
 
-      # Return the bias term (a.k.a. intercept) for SVC.
+      # Return the bias term (a.k.a. intercept) for SVR.
       # @return [Numo::DFloat] (shape: [n_outputs])
       attr_reader :bias_term
 
@@ -104,7 +104,7 @@ module SVMKit
       end
 
       # Dump marshal data.
-      # @return [Hash] The marshal data about SVC.
+      # @return [Hash] The marshal data about SVR.
       def marshal_dump
         { params: @params,
           weight_vec: @weight_vec,

data/lib/svmkit/polynomial_model/factorization_machine_classifier.rb

@@ -164,7 +164,7 @@ module SVMKit
       end
 
       # Dump marshal data.
-      # @return [Hash] The marshal data about FactorizationMachineClassifier
+      # @return [Hash] The marshal data about FactorizationMachineClassifier.
       def marshal_dump
         { params: @params,
           factor_mat: @factor_mat,

data/lib/svmkit/polynomial_model/factorization_machine_regressor.rb

@@ -134,7 +134,7 @@ module SVMKit
       end
 
       # Dump marshal data.
-      # @return [Hash] The marshal data about FactorizationMachineRegressor
+      # @return [Hash] The marshal data about FactorizationMachineRegressor.
       def marshal_dump
         { params: @params,
           factor_mat: @factor_mat,
@@ -177,7 +177,10 @@ module SVMKit
           data = x[subset_ids, true]
           values = y[subset_ids]
           # Calculate gradients for loss function.
-          loss_grad = loss_gradient(data, values, factor_mat, weight_vec, bias_term)
+          loss_grad = loss_gradient(data, values,
+                                    factor_mat - @params[:momentum] * factor_update,
+                                    weight_vec - @params[:momentum] * weight_update,
+                                    bias_term - @params[:momentum] * bias_update)
           next if loss_grad.ne(0.0).count.zero?
           # Update each parameter.
           bias_term, bias_sqrsum, bias_update =

data/lib/svmkit/version.rb

@@ -3,5 +3,5 @@
 # SVMKit is a machine learning library in Ruby.
 module SVMKit
   # @!visibility private
-  VERSION = '0.3.2'.freeze
+  VERSION = '0.3.3'.freeze
 end

data/svmkit.gemspec

@@ -17,7 +17,7 @@ MSG
 SVMKit is a machine learninig library in Ruby.
 SVMKit provides machine learning algorithms with interfaces similar to Scikit-Learn in Python.
 SVMKit currently supports Linear / Kernel Support Vector Machine,
-Logistic Regression, Factorization Machine, Naive Bayes, Decision Tree, Random Forest,
+Logistic Regression, Ridge, Lasso, Factorization Machine, Naive Bayes, Decision Tree, Random Forest,
 K-nearest neighbor algorithm, and cross-validation.
 MSG
   spec.homepage      = 'https://github.com/yoshoku/svmkit'

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: svmkit
 version: !ruby/object:Gem::Version
-  version: 0.3.2
+  version: 0.3.3
 platform: ruby
 authors:
 - yoshoku
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2018-05-23 00:00:00.000000000 Z
+date: 2018-05-25 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: numo-narray
@@ -84,7 +84,7 @@ description: |
   SVMKit is a machine learninig library in Ruby.
   SVMKit provides machine learning algorithms with interfaces similar to Scikit-Learn in Python.
   SVMKit currently supports Linear / Kernel Support Vector Machine,
-  Logistic Regression, Factorization Machine, Naive Bayes, Decision Tree, Random Forest,
+  Logistic Regression, Ridge, Lasso, Factorization Machine, Naive Bayes, Decision Tree, Random Forest,
   K-nearest neighbor algorithm, and cross-validation.
 email:
 - yoshoku@outlook.com
@@ -127,7 +127,9 @@ files:
 - lib/svmkit/evaluation_measure/recall.rb
 - lib/svmkit/kernel_approximation/rbf.rb
 - lib/svmkit/kernel_machine/kernel_svc.rb
+- lib/svmkit/linear_model/lasso.rb
 - lib/svmkit/linear_model/logistic_regression.rb
+- lib/svmkit/linear_model/ridge.rb
 - lib/svmkit/linear_model/svc.rb
 - lib/svmkit/linear_model/svr.rb
 - lib/svmkit/model_selection/cross_validation.rb