fselector 1.1.0 → 1.2.0

data/ChangeLog

@@ -1,3 +1,11 @@
+2012-05-20	version 1.2.0
+
+  * add KS-Test algorithm for continuous feature
+  * add KS-CCBF algorithm for continuous feature
+  * add J-Measure algorithm for discrete feature
+  * add KL-Divergence algorithm for discrete feature
+  * include the Discretizer module for algorithms requiring data with discrete feature, which allows to deal with continuous feature after discretization. Those algorithms requiring data with continuous feature now do not include the Discretizer module
+  
 2012-05-15	version 1.1.0
 
   * add replace\_by\_median\_value! for replacing missing value with feature median value

data/README.md

@@ -8,8 +8,8 @@ FSelector: a Ruby gem for feature selection and ranking
 **Email**: [need47@gmail.com](mailto:need47@gmail.com)  
 **Copyright**: 2012  
 **License**: MIT License  
-**Latest Version**: 1.1.0  
-**Release Date**: 2012-05-15
+**Latest Version**: 1.2.0  
+**Release Date**: 2012-05-20
 
 Synopsis
 --------
@@ -42,65 +42,69 @@ Feature List
 
 **2. available feature selection/ranking algorithms**
     
-    algorithm                         alias       algo_type   feature_type  applicability
-    --------------------------------------------------------------------------------------------------
-    Accuracy                          Acc         weighting   discrete
-    AccuracyBalanced                  Acc2        weighting   discrete
-    BiNormalSeparation                BNS         weighting   discrete
-    CFS_d                             CFS_d       subset      discrete
-    ChiSquaredTest                    CHI         weighting   discrete
-    CorrelationCoefficient            CC          weighting   discrete
-    DocumentFrequency                 DF          weighting   discrete
-    F1Measure                         F1          weighting   discrete
-    FishersExactTest                  FET         weighting   discrete
-    FastCorrelationBasedFilter        FCBF        subset      discrete
-    GiniIndex                         GI          weighting   discrete
-    GMean                             GM          weighting   discrete
-    GSSCoefficient                    GSS         weighting   discrete
-    InformationGain                   IG          weighting   discrete
-    INTERACT                          INTERACT    subset      discrete
-    LasVegasFilter                    LVF         subset      discrete
-    LasVegasIncremental               LVI         subset      discrete
-    MatthewsCorrelationCoefficient    MCC, PHI    weighting   discrete
-    McNemarsTest                      MNT         weighting   discrete
-    OddsRatio                         OR          weighting   discrete
-    OddsRatioNumerator                ORN         weighting   discrete
-    PhiCoefficient                    Phi         weighting   discrete
-    Power                             Power       weighting   discrete
-    Precision                         Precision   weighting   discrete
-    ProbabilityRatio                  PR          weighting   discrete
-    Random                            Random      weighting   discrete
-    Recall                            Recall      weighting   discrete
-    Relief_d                          Relief_d    weighting   discrete      two-class, no missing data
-    ReliefF_d                         ReliefF_d   weighting   discrete
-    Sensitivity                       SN, Recall  weighting   discrete
-    Specificity                       SP          weighting   discrete
-    SymmetricalUncertainty            SU          weighting   discrete
-    BetweenWithinClassesSumOfSquare   BSS_WSS     weighting   continuous
-    CFS_c                             CFS_c       subset      continuous
-    FTest                             FT          weighting   continuous
-    PMetric                           PM          weighting   continuous    two-class
-    Relief_c                          Relief_c    weighting   continuous    two-class, no missing data
-    ReliefF_c                         ReliefF_c   weighting   continuous
-    TScore                            TS          weighting   continuous    two-class
-    WilcoxonRankSum                   WRS         weighting   continuous    two-class
+    algorithm                        shortcut    algo_type  feature_type          applicability
+    --------------------------------------------------------------------------------------------------------
+    Accuracy                         Acc         weighting  discrete              multi-class
+    AccuracyBalanced                 Acc2        weighting  discrete              multi-class
+    BiNormalSeparation               BNS         weighting  discrete              multi-class
+    CFS_d                            CFS_d       subset     discrete              multi-class
+    ChiSquaredTest                   CHI         weighting  discrete              multi-class
+    CorrelationCoefficient           CC          weighting  discrete              multi-class
+    DocumentFrequency                DF          weighting  discrete              multi-class
+    F1Measure                        F1          weighting  discrete              multi-class
+    FishersExactTest                 FET         weighting  discrete              multi-class
+    FastCorrelationBasedFilter       FCBF        subset     discrete              multi-class
+    GiniIndex                        GI          weighting  discrete              multi-class
+    GMean                            GM          weighting  discrete              multi-class
+    GSSCoefficient                   GSS         weighting  discrete              multi-class
+    InformationGain                  IG          weighting  discrete              multi-class
+    INTERACT                         INTERACT    subset     discrete              multi-class
+    JMeasure                         JM          weighting  discrete              multi-class
+    KLDivergence                     KLD         weighting  discrete              multi-class
+    LasVegasFilter                   LVF         subset     discrete, continuous  multi-class
+    LasVegasIncremental              LVI         subset     discrete, continuous  multi-class
+    MatthewsCorrelationCoefficient   MCC, PHI    weighting  discrete              multi-class
+    McNemarsTest                     MNT         weighting  discrete              multi-class
+    OddsRatio                        OR          weighting  discrete              multi-class
+    OddsRatioNumerator               ORN         weighting  discrete              multi-class
+    PhiCoefficient                   PHI         weighting  discrete              multi-class
+    Power                            Power       weighting  discrete              multi-class
+    Precision                        Precision   weighting  discrete              multi-class
+    ProbabilityRatio                 PR          weighting  discrete              multi-class
+    Random                           Random      weighting  discrete              multi-class
+    Recall                           Recall      weighting  discrete              multi-class
+    Relief_d                         Relief_d    weighting  discrete              two-class, no missing data
+    ReliefF_d                        ReliefF_d   weighting  discrete              multi-class
+    Sensitivity                      SN, Recall  weighting  discrete              multi-class
+    Specificity                      SP          weighting  discrete              multi-class
+    SymmetricalUncertainty           SU          weighting  discrete              multi-class
+    BetweenWithinClassesSumOfSquare  BSS_WSS     weighting  continuous            multi-class
+    CFS_c                            CFS_c       subset     continuous            multi-class
+    FTest                            FT          weighting  continuous            multi-class
+    KS_CCBF                          KS_CCBF     subset     continuous            multi-class
+    KSTest                           KST         weighting  continuous            two-class
+    PMetric                          PM          weighting  continuous            two-class
+    Relief_c                         Relief_c    weighting  continuous            two-class, no missing data
+    ReliefF_c                        ReliefF_c   weighting  continuous            multi-class
+    TScore                           TS          weighting  continuous            two-class
+    WilcoxonRankSum                  WRS         weighting  continuous            two-class
     
   **note for feature selection interface:**   
   there are two types of filter methods, i.e., feature weighting algorithms and feature subset selection algorithms  
   
-  - for weighting type: use either **select\_feature\_by\_rank!** or **select\_feature\_by\_score!**  
+  - for weighting type: use either **select\_feature\_by\_score!** or **select\_feature\_by\_rank!**  
   - for subset type: use **select\_feature!**
 
 **3. feature selection approaches**
 
  - by a single algorithm
  - by multiple algorithms in a tandem manner
- - by multiple algorithms in an ensemble manner (share same feature selection interface as single algorithm)
+ - by multiple algorithms in an ensemble manner (share the same feature selection interface as single algorithm)
  
 **4. availabe normalization and discretization algorithms for continuous feature**
     
     algorithm                         note
-    -------------------------------------------------------------------------------
+    ---------------------------------------------------------------------------------------
     normalize_by_log!                 normalize by logarithmic transformation
     normalize_by_min_max!             normalize by scaling into [min, max]
     normalize_by_zscore!              normalize by converting into zscore
@@ -108,13 +112,13 @@ Feature List
     discretize_by_equal_frequency!    discretize by equal frequency among intervals
     discretize_by_ChiMerge!           discretize by ChiMerge algorithm
     discretize_by_Chi2!               discretize by Chi2 algorithm
-    discretize_by_MID!                discretize by Multi-Interval Discretization
-    discretize_by_TID!                discretize by Three-Interval Discretization
+    discretize_by_MID!                discretize by Multi-Interval Discretization algorithm
+    discretize_by_TID!                discretize by Three-Interval Discretization algorithm
     
 **5. availabe algorithms for replacing missing feature values**
     
     algorithm                         note                                   feature_type                     
-    ---------------------------------------------------------------------------------------------------------
+    ---------------------------------------------------------------------------------------------
     replace_by_fixed_value!           replace by a fixed value               discrete, continuous
     replace_by_mean_value!            replace by mean feature value          continuous
     replace_by_median_value!          replace by median feature value        continuous
@@ -141,8 +145,8 @@ Usage
 
     require 'fselector'
 	
-    # use InformationGain as a feature selection algorithm
-    r1 = FSelector::InformationGain.new
+    # use InformationGain (IG) as a feature selection algorithm
+    r1 = FSelector::IG.new
     
     # read from random data (or csv, libsvm, weka ARFF file)
     # no. of samples: 100
@@ -161,10 +165,10 @@ Usage
     # number of features after feature selection
     puts "  # features (after): "+ r1.get_features.size.to_s
     
-    # you can also use multiple alogirithms in a tandem manner
-    # e.g. use the ChiSquaredTest with Yates' continuity correction
+    # you can also use a second alogirithm for further feature selection
+    # e.g. use the ChiSquaredTest (CHI) with Yates' continuity correction
     # initialize from r1's data
-    r2 = FSelector::ChiSquaredTest.new(:yates, r1.get_data)
+    r2 = FSelector::CHI.new(:yates, r1.get_data)
     
     # number of features before feature selection
     puts "  # features (before): "+ r2.get_features.size.to_s
@@ -216,18 +220,18 @@ Usage
 	
 	
 	# creating an ensemble of feature selectors by using 
-	# two feature selection algorithms (InformationGain and Relief_d). 
+	# two feature selection algorithms: InformationGain (IG) and Relief_d. 
 	# note: can be 2+ algorithms, as long as they are of the same type, 
 	# either feature weighting or feature subset selection algorithms
 	
 	# test for the type of feature weighting algorithms 
-    r1 = FSelector::InformationGain.new
+    r1 = FSelector::IG.new
     r2 = FSelector::Relief_d.new(10)
     
     # an ensemble of two feature selectors
     re = FSelector::EnsembleMultiple.new(r1, r2)
     
-    # read random data
+    # read random discrete data (containing missing value)
     re.data_from_random(100, 2, 15, 3, true)
     
     # replace missing value because Relief_d 
@@ -247,35 +251,28 @@ Usage
     # number of features after feature selection
     puts '  # features (after): ' + re.get_features.size.to_s
     
-**3. normalization and discretization before feature selection**
-
- In addition to the algorithms designed for continuous feature, one
- can apply those deisgned for discrete feature after (optionally
- normalization and) discretization
+**3. feature selection after discretization**
  
     require 'fselector'
     
-    # for continuous feature
-    r1 = FSelector::Relief_c.new
+    # the Information Gain (IG) algorithm requires data with discrete feature
+    r = FSelector::IG.new
     
-    # read the Iris data set (under the test/ directory)
-    r1.data_from_csv('test/iris.csv')
+    # but the Iris data set contains continuous features (under the test/ directory)
+    r.data_from_csv('test/iris.csv')
         
-    # discretization by ChiMerge algorithm at alpha=0.10
-    r1.discretize_by_ChiMerge!(0.10)
-    
-    # apply Fast Correlation-Based Filter (FCBF) algorithm for discrete feature
-    # initialize with discretized data from r1
-    r2 = FSelector::FCBF.new(0.0, r1.get_data)
+    # let's first discretize it by ChiMerge algorithm at alpha=0.10
+    # then perform feature selection as usual
+    r.discretize_by_ChiMerge!(0.10)
     
     # number of features before feature selection
-    puts '  # features (before): ' + r2.get_features.size.to_s
+    puts '  # features (before): ' + r.get_features.size.to_s
     
-    # feature selection
-    r2.select_feature!
+    # select the top-ranked feature
+    r.select_feature_by_rank!('<=1')
     
     # number of features after feature selection
-    puts '  # features (after): ' + r2.get_features.size.to_s
+    puts '  # features (after): ' + r.get_features.size.to_s
 
 **4. see more examples test_*.rb under the test/ directory**
 

data/lib/fselector.rb

@@ -1,12 +1,13 @@
 # access to the statistical routines in R package
 require 'rinruby'
+R.eval 'options(warn = -1)' # suppress R warnings
 
 #
 # FSelector: a Ruby gem for feature selection and ranking
 #
 module FSelector
   # module version
-  VERSION = '1.1.0'
+  VERSION = '1.2.0'
 end
 
 # the root dir of FSelector

data/lib/fselector/algo_base/base.rb

@@ -6,9 +6,8 @@ module FSelector
   # base class for a single feature selection algorithm
   #
   class Base
-    # include FileIO
+    # include module
     include FileIO
-    # include ReplaceMissingValues
     include ReplaceMissingValues
     
     class << self

data/lib/fselector/algo_base/base_Relief.rb

@@ -11,9 +11,6 @@ module FSelector
 # ref: [The Feature Selection Problem: Traditional Methods and a New Algorithm](http://www.aaai.org/Papers/AAAI/1992/AAAI92-020.pdf)
 #
   class BaseRelief < Base
-    # include ReplaceMissingValue module
-    include ReplaceMissingValues
-    
     #
     # intialize from an existing data structure
     #
@@ -23,6 +20,7 @@ module FSelector
     #
     def initialize(m=30, data=nil)
       super(data)
+      
       @m = m || 30 # default 30
     end
     

data/lib/fselector/algo_base/base_ReliefF.rb

@@ -21,6 +21,7 @@ module FSelector
     #
     def initialize(m=30, k=10, data=nil)
       super(data)
+      
       @m = m || 30 # default 30
       @k = k || 10 # default 10
     end

data/lib/fselector/algo_base/base_continuous.rb

@@ -6,10 +6,8 @@ module FSelector
 # base algorithm for continuous feature
 #
   class BaseContinuous < Base
-    # include normalizer
+    # include module
     include Normalizer
-    # include discretizer
-    include Discretizer
     
     # initialize from an existing data structure
     def initialize(data=nil)

data/lib/fselector/algo_base/base_discrete.rb

@@ -29,6 +29,9 @@ module FSelector
 #      P(f'|c') = D/(B+D)
 #
   class BaseDiscrete < Base
+    # include module
+    include Discretizer
+    
     # initialize from an existing data structure
     def initialize(data=nil)
       super(data)

data/lib/fselector/algo_continuous/CFS_c.rb

@@ -9,9 +9,8 @@ module FSelector
 # ref: [Feature Selection for Discrete and Numeric Class Machine Learning](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.30.5673)
 #
   class CFS_c < BaseCFS
-    # include normalizer and discretizer
+    # include module
     include Normalizer
-    include Discretizer
     
     # this algo outputs a subset of feature
     @algo_type = :feature_subset_selection

data/lib/fselector/algo_continuous/{FTest.rb → F-Test.rb}

@@ -3,7 +3,7 @@
 #
 module FSelector
 #
-# F-test (FT) based on F-statistics for continuous feature
+# F-Test (FT) based on F-statistics for continuous feature
 # 
 #           between-group variability
 #     FT = ---------------------------
@@ -29,36 +29,6 @@ module FSelector
     private    
     
     # calculate contribution of each feature (f) across all classes
-    def calc_contribution2(f)
-      a, b, s = 0.0, 0.0, 0.0
-      ybar = get_feature_values(f).mean
-      kz = get_classes.size.to_f
-      sz = get_sample_size.to_f
-      
-      k2ybar = {} # cache
-      each_class do |k|
-        k2ybar[k] = get_feature_values(f, nil, k).mean        
-      end
-      
-      # a
-      each_class do |k|
-        n_k = get_data[k].size.to_f
-        a += n_k * (k2ybar[k] - ybar)**2 / (kz-1)
-      end
-      
-      # b
-      each_sample do |k, s|
-        if s.has_key? f
-          y_ik = s[f]
-          b += (y_ik - k2ybar[k])**2 / (sz-kz)
-        end
-      end
-      
-      s = a/b if not b.zero?
-      
-      set_feature_score(f, :BEST, s)
-    end # calc_contribution
-    
     def calc_contribution(f)
       a, b, s = 0.0, 0.0, 0.0
       ybar = get_feature_values(f).mean

data/lib/fselector/algo_continuous/KS-CCBF.rb

@@ -0,0 +1,125 @@
+#
+# FSelector: a Ruby gem for feature selection and ranking
+#
+module FSelector
+#
+# Kolmogorov-Smirnov Class Correlation-Based Filter (KS-CCBF) for continuous feature
+#
+# ref: [Feature Selection for Supervised Classification: A KolmogorovSmirnov Class Correlation-Based Filter](http://kzi.polsl.pl/~jbiesiada/Infosel/downolad/publikacje/09-Gliwice.pdf)
+#
+  class KS_CCBF < BaseContinuous
+    # include module
+    include Entropy
+    include Discretizer
+    
+    # this algo outputs a subset of feature
+    @algo_type = :feature_subset_selection
+        
+    #
+    # initialize from an existing data structure
+    # 
+    # @param [Float] lamda threshold value [0, 1] to determine feature redundancy
+    #
+    def initialize(lamda=0.2, data=nil)
+      super(data)
+      
+      @lamda = lamda || 0.2
+    end
+    
+    private
+    
+    # INTERACT algorithm
+    def get_feature_subset
+      # make a copy of data, since the discretization method will alter internal data
+      data_bak = get_data_copy
+      
+      # stage 1: calculate SUC coefficient
+      # but let's discretize features first
+      discretize_for_suc
+      
+      # then SUC
+      f2suc = {}
+      cv = get_class_labels
+      each_feature do |f|
+        fv = get_feature_values(f, :include_missing_values)
+        f2suc[f] = get_symmetrical_uncertainty(fv, cv)
+      end
+      
+      # sort feature according to descending order of its SUC
+      subset = f2suc.keys.sort { |x, y| f2suc[y] <=> f2suc[x] }
+      
+      # restore data, note set_data also clear old variables
+      set_data(data_bak)
+      
+      # stage 2: remove redundancy
+      fp = subset.first
+      while fp
+        fq = get_next_element(subset, fp)
+        
+        while fq
+          ks = calc_ks(fp, fq)
+          
+          if ks < @lamda
+            fq_new = get_next_element(subset, fq)
+            subset.delete(fq) # remove fq
+            fq = fq_new
+          else
+            fq = get_next_element(subset, fq)
+          end
+        end
+        
+        fp = get_next_element(subset, fp)
+      end
+      
+      subset      
+    end # get_feature_subset
+    
+    
+    # discretize continuous feature for calculating the SUC, 
+    # which requires discrete features. See Discretizer module 
+    # for available discretization methods. If you want to use 
+    # alternative one, simply override this function
+    def discretize_for_suc
+      discretize_by_ChiMerge!(0.10)
+    end
+    
+    
+    # get the next element of fp
+    def get_next_element(subset, fp)
+      fq = nil
+      
+      idx = subset.index(fp)      
+      if idx and idx < subset.size-1
+        fq = subset[idx+1]
+      end
+      
+      fq
+    end # get_next_element
+    
+    
+    # calculate K-S statistic (relying on R package) among all classes
+    def calc_ks(fp, fq)
+      ks = 0.0
+      
+      each_class do |k|
+        R.sp = get_feature_values(fp, nil, k)
+        R.sq = get_feature_values(fq, nil, k)
+        
+        # K-S test
+        R.eval "ks <- ks.test(sp, sq)$statistic"
+        
+        # pull K-S statistic
+        ks_try = R.ks
+        
+        # record max ks among classes
+        ks = ks_try if ks_try > ks
+      end
+      
+      ks
+    end # calc_ks
+    
+    
+  end # class
+  
+  
+end # module

data/lib/fselector/algo_continuous/KS-Test.rb

@@ -0,0 +1,51 @@
+#
+# FSelector: a Ruby gem for feature selection and ranking
+#
+module FSelector
+#
+# Kolmogorov-Smirnov Test (KST) for continuous feature
+# 
+# @note KST is applicable only to two-class problems, and missing data are ignored
+#
+# for KST (p-value), the smaller, the better, but we intentionally negate it 
+# so that the larger is always the better (consistent with other algorithms).  
+# R equivalent: ks.test
+#
+# ref: [Wikipedia](http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test) and [Feature Extraction, Foundations and Applications](http://www.springer.com/engineering/computational+intelligence+and+complexity/book/978-3-540-35487-1)
+#
+  class KSTest < BaseContinuous
+    # this algo outputs weight for each feature
+    @algo_type = :feature_weighting
+        
+    private    
+    
+    # calculate contribution of each feature (f) across all classes
+    def calc_contribution(f)
+      if not get_classes.size == 2
+        abort "[#{__FILE__}@#{__LINE__}]: \n"+
+              "  suitable only for two-class problem with continuous feature"
+      end
+      
+      # collect data for class 1 and 2, respectively     
+      k1, k2 = get_classes
+      R.s1 = get_feature_values(f, nil, k1) # class 1
+      R.s2 = get_feature_values(f, nil, k2) # class 2
+      
+      # K-S test
+      R.eval "rv <- ks.test(s1, s2)$p.value"
+      
+      # intensionally negate it
+      s = -1.0 * R.rv # pull the p-value from R
+      
+      set_feature_score(f, :BEST, s)
+    end # calc_contribution
+    
+    
+  end # class
+  
+  
+  # shortcut so that you can use FSelector::KST instead of FSelector::KSTest
+  KST = KSTest
+  
+  
+end # module

data/lib/fselector/algo_continuous/ReliefF_c.rb

@@ -10,9 +10,8 @@ module FSelector
 # ref: [Estimating Attributes: Analysis and Extensions of RELIEF](http://www.springerlink.com/content/fp23jh2h0426ww45/)
 #
   class ReliefF_c < BaseReliefF
-    # include normalizer and discretizer
+    # include module
     include Normalizer
-    include Discretizer
     
     # this algo outputs weight for each feature
     @algo_type = :feature_weighting

data/lib/fselector/algo_continuous/Relief_c.rb

@@ -10,9 +10,8 @@ module FSelector
 # ref: [The Feature Selection Problem: Traditional Methods and a New Algorithm](http://www.aaai.org/Papers/AAAI/1992/AAAI92-020.pdf)
 #
   class Relief_c < BaseRelief
-    # include normalizer and discretizer
+    # include module
     include Normalizer
-    include Discretizer
     
     # this algo outputs weight for each feature
     @algo_type = :feature_weighting

data/lib/fselector/algo_continuous/{TScore.rb → T-Score.rb}

@@ -3,7 +3,7 @@
 #
 module FSelector
 #
-# t-score (TS) based on Student's t-test for continuous feature
+# T-Score (TS) based on Student's t-test for continuous feature
 # 
 #                      |u1 - u2|
 #     TS = -------------------------------------

data/lib/fselector/algo_discrete/CFS_d.rb

@@ -9,7 +9,8 @@ module FSelector
 # ref: [Feature Selection for Discrete and Numeric Class Machine Learning](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.30.5673)
 #
   class CFS_d < BaseCFS
-    # include Entropy module
+    # include module
+    include Discretizer
     include Entropy
     
     # this algo outputs a subset of feature

data/lib/fselector/algo_discrete/ChiSquaredTest.rb

@@ -30,6 +30,7 @@ module FSelector
     #
     def initialize(correction=:yates, data=nil)
       super(data)
+      
       @correction = (correction==:yates) ? true : false
     end
      

data/lib/fselector/algo_discrete/FastCorrelationBasedFilter.rb

@@ -9,7 +9,7 @@ module FSelector
 # ref: [Feature Selection for High-Dimensional Data: A Fast Correlation-Based Filter Solution](http://www.hpl.hp.com/conferences/icml2003/papers/144.pdf)
 #
   class FastCorrelationBasedFilter < BaseDiscrete
-    # include Entropy module
+    # include module
     include Entropy
     
     # this algo outputs a subset of feature
@@ -22,6 +22,7 @@ module FSelector
     #
     def initialize(delta=0.0, data=nil)
       super(data)
+      
       @delta = delta || 0.0
     end
     
@@ -108,14 +109,13 @@ module FSelector
     end
     
     
+    # get the next element of fp in subset
     def get_next_element(subset, fp)
       fq = nil
       
-      subset.each_with_index do |v, i|
-        if v == fp and i+1 < subset.size
-          fq = subset[i+1]
-          break
-        end
+      idx = subset.index(fp)      
+      if idx and idx < subset.size-1
+        fq = subset[idx+1]
       end
       
       fq

data/lib/fselector/algo_discrete/{GMean.rb → G-Mean.rb}

@@ -3,7 +3,7 @@
 #
 module FSelector
 #
-# GMean (GM)
+# G-Mean (GM)
 #
 #     GM = sqrt(Sensitivity * Specificity)
 #     

data/lib/fselector/algo_discrete/INTERACT.rb

@@ -9,9 +9,8 @@ module FSelector
 # ref: [Searching for Interacting Features](http://www.public.asu.edu/~huanliu/papers/ijcai07.pdf)
 #
   class INTERACT < BaseDiscrete
-    # include Entropy module
+    # include module
     include Entropy
-    # include Consistency module
     include Consistency
     
     # this algo outputs a subset of feature
@@ -24,13 +23,14 @@ module FSelector
     #
     def initialize(delta=0.0001, data=nil)
       super(data)
+      
       @delta = delta || 0.0001
     end
     
     private
     
     # INTERACT algorithm
-    def get_feature_subset     
+    def get_feature_subset      
       subset, f2su = get_features.dup, {}
       
       # part 1, get symmetrical uncertainty for each feature

data/lib/fselector/algo_discrete/InformationGain.rb

@@ -14,7 +14,7 @@ module FSelector
 # ref: [Using Information Gain to Analyze and Fine Tune the Performance of Supply Chain Trading Agents](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.141.7895)
 #
   class InformationGain < BaseDiscrete
-    # include Entropy module
+    # include module
     include Entropy
 
     # this algo outputs weight for each feature

data/lib/fselector/algo_discrete/J-Measure.rb

@@ -0,0 +1,51 @@
+#
+# FSelector: a Ruby gem for feature selection and ranking
+#
+module FSelector
+#
+# J-Measure (JM) for discrete feature
+#
+#                                                 P(y_j|x_i)
+#     JM = sigma_i P(x_i) sigma_j P(y_j|x_i) log ------------
+#                                                   P(y_j)
+#
+# ref: [Feature Extraction, Foundations and Applications](http://www.springer.com/engineering/computational+intelligence+and+complexity/book/978-3-540-35487-1)
+#
+  class JMeasure < BaseDiscrete
+    # this algo outputs weight for each feature
+    @algo_type = :feature_weighting
+        
+    private    
+    
+    # calculate contribution of each feature (f) across all classes
+    def calc_contribution(f)
+      cv = get_class_labels
+      fv = get_feature_values(f, :include_missing_values)
+      sz = cv.size.to_f # also equal fv.size
+      
+      s = 0.0
+      fv.uniq.each do |x|
+        px = fv.count(x)/sz
+
+        cv.uniq.each do |y|
+          py = cv.count(y)/sz
+          
+          indices = (0...fv.size).to_a.select { |i| fv[i] == x }
+          pyx = cv.values_at(*indices).count(y)/indices.size.to_f
+          
+          s += px * ( pyx * Math.log2(pyx/py) ) if not pyx.zero?
+        end
+      end
+      
+      set_feature_score(f, :BEST, s)
+    end # calc_contribution
+    
+    
+  end # class
+  
+  
+  # shortcut so that you can use FSelector::JM instead of FSelector::JMeasure
+  JM = JMeasure
+  
+  
+end # module

data/lib/fselector/algo_discrete/KL-Divergence.rb

@@ -0,0 +1,65 @@
+#
+# FSelector: a Ruby gem for feature selection and ranking
+#
+module FSelector
+#
+# Kullback-Leibler Divergence (KLD) for discrete feature
+#
+#     w_i = wbar_i / ( -Z * sigma_j ( P(a_ij) logP(a_ij) ) )
+#
+#     where wbar(i) = sigma_j ( P(a_ij) KL(C|a_ij) )
+#
+#           KL(C|a_ij) = sigma_c ( P(c|a_ij) log(P(c|a_ij)/P(c)) )
+#
+#           Z is normalization constant
+#
+# ref: [Calculating Feature Weights in Naive Bayes with Kullback-Leibler Measure](http://ix.cs.uoregon.edu/~dou/research/papers/icdm11_fw.)
+#
+  class KLDivergence < BaseDiscrete
+    # this algo outputs weight for each feature
+    @algo_type = :feature_weighting
+        
+    private    
+    
+    # calculate contribution of each feature (f) across all classes 
+    # note the normalization constant Z is ignored, since we need only 
+    # the relative feature scores
+    def calc_contribution(f)
+      cv = get_class_labels
+      fv = get_feature_values(f, :include_missing_values)
+      sz = cv.size.to_f # also equal fv.size
+      
+      s, w_avg, d = 0.0, 0.0, 0.0
+      
+      fv.uniq.each do |x|
+        px = fv.count(x)/sz
+        d += -1.0 * px * Math.log2(px)
+        
+        kl_x = 0.0
+        
+        cv.uniq.each do |y|
+          py = cv.count(y)/sz
+          
+          indices = (0...fv.size).to_a.select { |i| fv[i] == x }
+          pyx = cv.values_at(*indices).count(y)/indices.size.to_f
+          
+          kl_x += pyx * Math.log2(pyx/py) if not pyx.zero?
+        end
+        
+        w_avg += px * kl_x
+      end
+      
+      s = w_avg / d if not d.zero?
+      
+      set_feature_score(f, :BEST, s)
+    end # calc_contribution
+    
+    
+  end # class
+  
+  
+  # shortcut so that you can use FSelector::KLD instead of FSelector::KLDivergence
+  KLD = KLDivergence
+  
+  
+end # module

data/lib/fselector/algo_discrete/LasVegasFilter.rb

@@ -11,7 +11,7 @@ module FSelector
 # ref: [Review and Evaluation of Feature Selection Algorithms in Synthetic Problems](http://arxiv.org/abs/1101.2320)
 #
   class LasVegasFilter < BaseDiscrete
-    # include Consistency module
+    # include module
     include Consistency
     
     # this algo outputs a subset of feature
@@ -24,13 +24,14 @@ module FSelector
     #
     def initialize(max_iter=100, data=nil)
       super(data)
+      
       @max_iter = max_iter || 100
     end
     
     private
     
     # Las Vegas Filter (LVF) algorithm
-    def get_feature_subset
+    def get_feature_subset  
       inst_cnt = get_instance_count      
       j0 = get_IR_by_count(inst_cnt)
       

data/lib/fselector/algo_discrete/LasVegasIncremental.rb

@@ -9,7 +9,7 @@ module FSelector
 # ref: [Incremental Feature Selection](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.34.8218)
 #
   class LasVegasIncremental < BaseDiscrete
-    # include Consistency module
+    # include module
     include Consistency
     
     # this algo outputs a subset of feature
@@ -23,6 +23,7 @@ module FSelector
     #
     def initialize(max_iter=100, portion=0.10, data=nil)
       super(data)
+      
       @max_iter = max_iter || 100
       @portion = portion || 0.10
     end
@@ -30,7 +31,7 @@ module FSelector
     private    
     
     # Las Vegas Incremental (LVI) algorithm
-    def get_feature_subset
+    def get_feature_subset     
       data = get_data # working dataset
       s0, s1 = portion(data)
       feats = get_features

data/lib/fselector/algo_discrete/McNemarsTest.rb

@@ -25,6 +25,7 @@ module FSelector
     #
     def initialize(correction=:yates, data=nil)
       super(data)
+      
       @correction = (correction==:yates) ? true : false
     end
      

data/lib/fselector/algo_discrete/Power.rb

@@ -24,6 +24,7 @@ module FSelector
     #
     def initialize(k=5, data=nil)
       super(data)
+      
       @k = k || 5
     end
     

data/lib/fselector/algo_discrete/Random.rb

@@ -22,6 +22,7 @@ module FSelector
     #
     def initialize(seed=nil, data=nil)
       super(data)
+      
       srand(seed) if seed
     end
     

data/lib/fselector/algo_discrete/ReliefF_d.rb

@@ -9,6 +9,9 @@ module FSelector
 # ref: [Estimating Attributes: Analysis and Extensions of RELIEF](http://www.springerlink.com/content/fp23jh2h0426ww45/)
 #
   class ReliefF_d < BaseReliefF
+    # include module
+    include Discretizer
+    
     # this algo outputs weight for each feature
     @algo_type = :feature_weighting
        

data/lib/fselector/algo_discrete/Relief_d.rb

@@ -10,6 +10,9 @@ module FSelector
 # ref: [The Feature Selection Problem: Traditional Methods and a New Algorithm](http://www.aaai.org/Papers/AAAI/1992/AAAI92-020.pdf)
 #
   class Relief_d < BaseRelief
+    # include module
+    include Discretizer
+    
     # this algo outputs weight for each feature
     @algo_type = :feature_weighting
     

data/lib/fselector/algo_discrete/SymmetricalUncertainty.rb

@@ -17,7 +17,7 @@ module FSelector
 # ref: [Wikipedia](http://en.wikipedia.org/wiki/Symmetric_uncertainty) and [Robust Feature Selection Using Ensemble Feature Selection Techniques](http://dl.acm.org/citation.cfm?id=1432021)
 #
   class SymmetricalUncertainty < BaseDiscrete
-    # include Entropy module
+    # include module
     include Entropy
 
     # this algo outputs weight for each feature

data/lib/fselector/discretizer.rb

@@ -2,11 +2,10 @@
 # discretize continuous feature
 #
 module Discretizer
-  # include Entropy module
-  include Entropy
-  # include Consistency module
+  # include module
   include Consistency
-
+  include Entropy
+  
   #    
   # discretize by equal-width intervals
   #
@@ -157,7 +156,7 @@ module Discretizer
   #
   # ref: [Chi2: Feature Selection and Discretization of Numeric Attributes](http://sci2s.ugr.es/keel/pdf/specific/congreso/liu1995.pdf)
   #
-  def discretize_by_Chi2!(delta=0.02)
+  def discretize_by_Chi2!(delta=0.02)    
     # degree of freedom equals one less than number of classes     
     df = get_classes.size-1
         
@@ -270,7 +269,7 @@ module Discretizer
   # 
   # ref: [Multi-Interval Discretization of Continuous-Valued Attributes for Classification Learning](http://www.ijcai.org/Past%20Proceedings/IJCAI-93-VOL2/PDF/022.pdf)
   #
-  def discretize_by_MID!
+  def discretize_by_MID!    
     # determine the final boundaries
     f2cp = {} # cut points for each feature
     each_feature do |f|

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fselector
 version: !ruby/object:Gem::Version
-  version: 1.1.0
+  version: 1.2.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-05-15 00:00:00.000000000 Z
+date: 2012-05-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rinruby
-  requirement: &28540080 !ruby/object:Gem::Requirement
+  requirement: &23863908 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,7 +21,7 @@ dependencies:
         version: 2.0.2
   type: :runtime
   prerelease: false
-  version_requirements: *28540080
+  version_requirements: *23863908
 description: FSelector is a Ruby gem that aims to integrate various feature selection/ranking
   algorithms and related functions into one single package. Welcome to contact me
   (need47@gmail.com) if you'd like to contribute your own algorithms or report a bug.
@@ -50,11 +50,13 @@ files:
 - lib/fselector/algo_base/base_ReliefF.rb
 - lib/fselector/algo_continuous/BSS_WSS.rb
 - lib/fselector/algo_continuous/CFS_c.rb
-- lib/fselector/algo_continuous/FTest.rb
-- lib/fselector/algo_continuous/PMetric.rb
+- lib/fselector/algo_continuous/F-Test.rb
+- lib/fselector/algo_continuous/KS-CCBF.rb
+- lib/fselector/algo_continuous/KS-Test.rb
+- lib/fselector/algo_continuous/P-Metric.rb
 - lib/fselector/algo_continuous/ReliefF_c.rb
 - lib/fselector/algo_continuous/Relief_c.rb
-- lib/fselector/algo_continuous/TScore.rb
+- lib/fselector/algo_continuous/T-Score.rb
 - lib/fselector/algo_continuous/WilcoxonRankSum.rb
 - lib/fselector/algo_discrete/Accuracy.rb
 - lib/fselector/algo_discrete/AccuracyBalanced.rb
@@ -66,11 +68,13 @@ files:
 - lib/fselector/algo_discrete/F1Measure.rb
 - lib/fselector/algo_discrete/FastCorrelationBasedFilter.rb
 - lib/fselector/algo_discrete/FishersExactTest.rb
+- lib/fselector/algo_discrete/G-Mean.rb
 - lib/fselector/algo_discrete/GiniIndex.rb
-- lib/fselector/algo_discrete/GMean.rb
 - lib/fselector/algo_discrete/GSSCoefficient.rb
 - lib/fselector/algo_discrete/InformationGain.rb
 - lib/fselector/algo_discrete/INTERACT.rb
+- lib/fselector/algo_discrete/J-Measure.rb
+- lib/fselector/algo_discrete/KL-Divergence.rb
 - lib/fselector/algo_discrete/LasVegasFilter.rb
 - lib/fselector/algo_discrete/LasVegasIncremental.rb
 - lib/fselector/algo_discrete/MatthewsCorrelationCoefficient.rb