statsample 0.9.0 → 0.10.0

data/History.txt

@@ -1,6 +1,25 @@
+=== 0.10.0 / 2010-04-13
+
+<b>API modifications</b>
+* Refactoring of Statsample::Anova module. 
+  * Statsample::Anova::OneWay :implementation of generic ANOVA One-Way, used by Multiple Regression, for example.
+  * Statsample::Anova::OneWayWithVectors: implementation of ANOVA One-Way to test differences of means.
+
+<b>New features</b>
+* New Statsample::Factor::Parallel Analysis, to performs Horn's 'parallel analysis' to a PCA, to adjust for sample bias on retention of components.
+* New Statsample.only_valid_clone and Statsample::Dataset.clone, which allows to create shallow copys of valid vector and datasets. Used by correlation matrix methods to optimize calculations
+* New module Statsample::Summarizable, which add GetText and ReportBuilder support to classes. Better summaries for Vector, Dataset, Crosstab, PrincipalAxis, PCA and Regression::Multiple classes
+
+<b>Optimizations and bug fix</b>
+
+* Refactoring of Statsample::Regression::Multiple classes. Still needs works
+* Bug fix on Statsample::Factor::PCA and Statsample::Factor::PrincipalAxis
+* Bug fix on Statsample::Bivariate::Polychoric.new_with_vectors. Should be defined class method, no instance method.
+* Optimized correlation and covariance matrix. Only calculates the half of matrix and the other half is returned from cache
+* More tests coverage. RCOV Total: 82.51% , Code: 77.83%
+
 === 0.9.0 / 2010-04-04
 * New Statsample::Test::F. Anova::OneWay subclasses it and Regression classes uses it.
-
 === 0.8.2 / 2010-04-01
 * Statsample::PromiseAfter replaced by external package DirtyMemoize [http://rubygems.org/gems/dirty-memoize]
 === 0.8.1 / 2010-03-29

data/Manifest.txt

@@ -11,15 +11,19 @@ data/test_binomial.csv
 data/tetmat_matrix.txt
 data/tetmat_test.txt
 examples/correlation_matrix.rb
+examples/dataset.rb
 examples/dominance_analysis.rb
 examples/dominance_analysis_bootstrap.rb
 examples/levene.rb
 examples/multiple_regression.rb
 examples/multivariate_correlation.rb
+examples/parallel_analysis.rb
+examples/parallel_analysis_tetrachoric.rb
 examples/polychoric.rb
 examples/principal_axis.rb
 examples/t_test.rb
 examples/tetrachoric.rb
+examples/vector.rb
 lib/distribution.rb
 lib/distribution/chisquare.rb
 lib/distribution/f.rb
@@ -43,6 +47,7 @@ lib/statsample/dataset.rb
 lib/statsample/dominanceanalysis.rb
 lib/statsample/dominanceanalysis/bootstrap.rb
 lib/statsample/factor.rb
+lib/statsample/factor/parallelanalysis.rb
 lib/statsample/factor/pca.rb
 lib/statsample/factor/principalaxis.rb
 lib/statsample/factor/rotation.rb
@@ -84,8 +89,10 @@ po/es/statsample.mo
 po/es/statsample.po
 po/statsample.pot
 setup.rb
-test/test_anova.rb
+test/test_anovaoneway.rb
+test/test_anovawithvectors.rb
 test/test_bivariate.rb
+test/test_bivariate_polychoric.rb
 test/test_codification.rb
 test/test_combination.rb
 test/test_crosstab.rb

data/README.txt

@@ -5,18 +5,19 @@ http://ruby-statsample.rubyforge.org/
 
 == DESCRIPTION:
 
-A suite for basic and advanced statistics on Ruby. Tested on Ruby 1.8.7, 1.9.1, 1.9.2 (April, 2010) and JRuby 1.4 (Ruby 1.8.7 compatible) 
+A suite for basic and advanced statistics on Ruby. Tested on Ruby 1.8.7, 1.9.1, 1.9.2 (April, 2010) and JRuby 1.4 (Ruby 1.8.7 compatible).
 
-Includes:
+Include:
 * Descriptive statistics: frequencies, median, mean, standard error, skew, kurtosis (and many others).
 * Imports and exports datasets from and to Excel, CSV and plain text files.
-* Correlations: Pearson's r, Spearman's rank correlation (rho), Tetrachoric, Polychoric
-* Tests: F (Anona One-Way), T, Levene, U-Mannwhitney.
-* Regression: Simple, Multiple, Probit  and Logit
-* Factorial Analysis: Extraction (PCA and Principal Axis) and Rotation (Varimax and relatives)
+* Correlations: Pearson's r, Spearman's rank correlation (rho), Tetrachoric, Polychoric.
+* Anova: generic and vector-based One-way ANOVA
+* Tests: F, T, Levene, U-Mannwhitney.
+* Regression: Simple, Multiple (OLS), Probit  and Logit
+* Factorial Analysis: Extraction (PCA and Principal Axis), Rotation (Varimax, Equimax, Quartimax) and Parallel Analysis, for estimation of number of factors.
 * Dominance Analysis, with multivariate dependent and bootstrap (Azen & Budescu)
 * Sample calculation related formulas
-* Creates reports on text, html and rtf, using ReportBuilder
+* Creates reports on text, html and rtf, using ReportBuilder gem
 
 == FEATURES:
 
@@ -24,6 +25,7 @@ Includes:
   * Statsample::Vector: An extension of an array, with statistical methods like sum, mean and standard deviation
   * Statsample::Dataset: a group of Statsample::Vector, analog to a excel spreadsheet or a dataframe on R. The base of almost all operations on statsample. 
   * Statsample::Multiset: multiple datasets with same fields and type of vectors
+* Anova module provides generic Statsample::Anova::OneWay and vector based Statsample::Anova::OneWayWithVectors
 * Module Statsample::Bivariate provides covariance and pearson, spearman, point biserial, tau a, tau b, gamma, tetrachoric (see Bivariate::Tetrachoric) and polychoric (see Bivariate::Polychoric) correlations. Include methods to create correlation and covariance matrices
 * Multiple types of regression.
   * Simple Regression :  Statsample::Regression::Simple
@@ -38,6 +40,7 @@ Includes:
     * Statsample::Factor::Varimax
     * Statsample::Factor::Equimax
     * Statsample::Factor::Quartimax
+  * Statsample::Factor::ParallelAnalysis performs Horn's 'parallel analysis' to a principal components analysis to adjust for sample bias in the retention of components. 
 * Dominance Analysis. Based on Budescu and Azen papers, Statsample::DominanceAnalysis class can report dominance analysis for a sample, using uni or multivariate dependent variables and DominanceAnalysisBootstrap can execute bootstrap analysis to determine dominance stability, as recomended by  Azen & Budescu (2003) link[http://psycnet.apa.org/journals/met/8/2/129/]. 
 * Module Statsample::Codification, to help to codify open questions
 * Converters to import and export data:
@@ -53,6 +56,7 @@ Includes:
   * Statsample::Test::Levene
   * Statsample::Test::UMannWhitney
   * Statsample::Test::T
+  * Statsample::Test::F  
 * Interfaces to gdchart, gnuplot and SVG::Graph 
 
 

data/Rakefile

@@ -39,11 +39,11 @@ task :makemo do
 end
 
 h=Hoe.spec('statsample') do 
-	self.testlib=:minitest unless RUBY_VERSION<="1.9"
   self.version=Statsample::VERSION
+  self.testlib=:minitest
 	self.rubyforge_name = "ruby-statsample"
 	self.developer('Claudio Bustos', 'clbustos@gmail.com')
-	self.extra_deps << ["spreadsheet","~>0.6.0"] << ["svg-graph", "~>1.0"] << ["reportbuilder", "~>1.0"] << ["minimization", "~>0.1.0"] << ["fastercsv"] << ["dirty-memoize", "~>0.0"]
+	self.extra_deps << ["spreadsheet","~>0.6.0"] << ["svg-graph", "~>1.0"] << ["reportbuilder", "~>1.0"] << ["minimization", "~>0.2.0"] << ["fastercsv"] << ["dirty-memoize", "~>0.0"]
 	self.clean_globs << "test/images/*" << "demo/item_analysis/*" << "demo/Regression"
 	self.need_rdoc=false
 end

data/examples/dataset.rb

@@ -0,0 +1,8 @@
+#!/usr/bin/ruby
+$:.unshift(File.dirname(__FILE__)+'/../lib/')
+require 'statsample'
+a=1000.times.collect {r=rand(5); r==4 ? nil: r;}.to_scale
+b=1000.times.collect {r=rand(5); r==4 ? nil: r;}.to_scale
+
+ds={'a'=>a,'b'=>b}.to_dataset
+puts ds.summary

data/examples/multiple_regression.rb

@@ -1,5 +1,5 @@
 #!/usr/bin/ruby
-$:.unshift(File.dirname(__FILE__)+'/../lib/')
+#$:.unshift(File.dirname(__FILE__)+'/../lib/')
 
 require 'statsample'
 require 'benchmark'

data/examples/parallel_analysis.rb

@@ -0,0 +1,29 @@
+#!/usr/bin/ruby
+$:.unshift(File.dirname(__FILE__)+'/../lib/')
+
+require 'statsample'
+samples=100
+variables=30
+rng = GSL::Rng.alloc()
+f1=samples.times.collect {rng.ugaussian()}.to_scale
+f2=samples.times.collect {rng.ugaussian()}.to_scale
+f3=samples.times.collect {rng.ugaussian()}.to_scale
+
+vectors={}
+
+variables.times do |i|
+  vectors["v#{i}"]=samples.times.collect {|nv| f1[nv]*i+(f2[nv]*(15-i))+((f3[nv]*(30-i))*1.5)*rng.ugaussian()}.to_scale
+end
+ds=vectors.to_dataset
+
+pa=Statsample::Factor::ParallelAnalysis.new(ds, :iterations=>10, :debug=>true)
+pca=Statsample::Factor::PCA.new(Statsample::Bivariate.correlation_matrix(ds))
+rb=ReportBuilder.new(:name=>"Parallel Analysis with simulation") do |g|
+  g.text("There are 3 real factors on data")
+  g.parse_element(pca)
+  g.text("Traditional Kaiser criterion (k>1) returns #{pca.m} factors")
+  g.parse_element(pa)
+  g.text("Parallel Analysis returns #{pa.number_of_factors} factors to preserve")
+end
+
+puts rb.to_text

data/examples/parallel_analysis_tetrachoric.rb

@@ -0,0 +1,30 @@
+#!/usr/bin/ruby
+$:.unshift(File.dirname(__FILE__)+'/../lib/')
+
+require 'statsample'
+samples=1000
+variables=30
+rng = GSL::Rng.alloc()
+f1=samples.times.collect {rng.ugaussian()}.to_scale
+f2=samples.times.collect {rng.ugaussian()}.to_scale
+f3=samples.times.collect {rng.ugaussian()}.to_scale
+
+vectors={}
+
+variables.times do |i|
+  vectors["v#{i}"]=samples.times.collect {|nv|  f1[nv]*(i-30)+f2[nv]*(i+30)+f3[nv]*(i+15) + rng.ugaussian() > 0 ? 1 : 0}.to_scale
+end
+ds=vectors.to_dataset
+
+pa=Statsample::Factor::ParallelAnalysis.new(ds, :iterations=>10, :matrix_method=>:tetrachoric_correlation_matrix, :debug=>true)
+
+pca=Statsample::Factor::PCA.new(Statsample::Bivariate.tetrachoric_correlation_matrix(ds))
+rb=ReportBuilder.new(:name=>"Parallel Analysis with simulation") do |g|
+  g.text("There are 3 real factors on data")
+  g.parse_element(pca)
+  g.text("Traditional Kaiser criterion (k>1) returns #{pca.m} factors")
+  g.parse_element(pa)
+  g.text("Parallel Analysis returns #{pa.number_of_factors} factors to preserve")
+end
+
+puts rb.to_text

data/examples/vector.rb

@@ -0,0 +1,6 @@
+#!/usr/bin/ruby
+$:.unshift(File.dirname(__FILE__)+'/../lib/')
+
+require 'statsample'
+a=1000.times.collect {r=rand(5); r==4 ? nil: r;}.to_scale
+puts a.summary

data/lib/distribution.rb

@@ -8,10 +8,20 @@ require 'statistics2'
 #    Distribution::Normal.p_value(0.95)
 #    => 1.64485364660836
 module Distribution
-    autoload(:ChiSquare, 'distribution/chisquare')
-    autoload(:T, 'distribution/t')
-    autoload(:F, 'distribution/f')
-    autoload(:Normal, 'distribution/normal')
-    autoload(:NormalBivariate, 'distribution/normalbivariate')
-    autoload(:NormalMultivariate, 'distribution/normalmultivariate')
+  begin
+    require 'rbgsl'
+    def self.has_gsl?
+      true
+    end
+  rescue LoadError
+    def self.has_gsl?
+      false
+    end
+  end
+  autoload(:ChiSquare, 'distribution/chisquare')
+  autoload(:T, 'distribution/t')
+  autoload(:F, 'distribution/f')
+  autoload(:Normal, 'distribution/normal')
+  autoload(:NormalBivariate, 'distribution/normalbivariate')
+  # autoload(:NormalMultivariate, 'distribution/normalmultivariate')
 end

data/lib/distribution/normal.rb

@@ -1,25 +1,32 @@
 module Distribution
-    # Calculate cdf and inverse cdf for Normal Distribution.
-    # Uses Statistics2 module
-    module Normal
-      class << self
-        # Return the P-value of the corresponding integral
-        def p_value(pr)
-            Statistics2.pnormaldist(pr)
-        end
-        # Normal cumulative distribution function (cdf).
-        # 
-        # Returns the integral of  normal distribution 
-        # over (-Infty, x].
-        # 
-        def cdf(x)
-            Statistics2.normaldist(x)
-        end
-        # Normal probability density function (pdf)
-        # With x=0 and sigma=1
-        def pdf(x)
-            (1.0/Math::sqrt(2*Math::PI))*Math::exp(-(x**2/2.0))
+  # Calculate cdf and inverse cdf for Normal Distribution.
+  # Uses Statistics2 module
+  module Normal
+    class << self
+      # Return the P-value of the corresponding integral
+      def p_value(pr)
+          Statistics2.pnormaldist(pr)
+      end
+      # Normal cumulative distribution function (cdf).
+      # 
+      # Returns the integral of  normal distribution 
+      # over (-Infty, x].
+      # 
+      def cdf(x)
+        Statistics2.normaldist(x)
+      end
+
+      if false and Distribution.has_gsl?
+        alias :cdf_ruby :cdf 
+        def cdf(x) # :nodoc:
+          GSL::Cdf::gaussian_P(x)
         end
       end
+      # Normal probability density function (pdf)
+      # With x=0 and sigma=1
+      def pdf(x)
+          (1.0/Math::sqrt(2*Math::PI))*Math::exp(-(x**2/2.0))
+      end
     end
+  end
 end

data/lib/distribution/normalbivariate.rb

@@ -120,7 +120,7 @@ module Distribution
         sum
       end
       # Normal cumulative distribution function (cdf) for a given x, y and rho.
-      # Based on Fortran code by Alan Genz
+      # Ported from Fortran code by Alan Genz
       #
       # Original documentation
       #    DOUBLE PRECISION FUNCTION BVND( DH, DK, R )

data/lib/statsample.rb

@@ -112,7 +112,7 @@ module Statsample
       false
     end
   end
-  VERSION = '0.9.0'
+  VERSION = '0.10.0'
   SPLIT_TOKEN = ","
   autoload(:Database, 'statsample/converters')
   autoload(:Anova, 'statsample/anova')
@@ -186,6 +186,16 @@ module Statsample
       ds=Statsample::Dataset.new(h).dup_only_valid
       ds.vectors.values
     end
+    # Cheap version of #only_valid. 
+    # If any vectors have missing_values, return only valid.
+    # If not, return the vectors it self
+    def only_valid_clone(*vs)
+      if vs.any? {|v| v.has_missing_data?}
+        only_valid(*vs)
+      else
+        vs
+      end
+    end
   end  
   
   
@@ -214,7 +224,14 @@ module Statsample
       fp.close
     end        
   end
-    
+  # Provides basic method to generate summaries
+  module Summarizable
+    include GetText
+    bindtextdomain("statsample")
+    def summary(method=:to_text)
+      ReportBuilder.new(:no_title=>true).add(self).send(method)
+    end
+  end
   module STATSAMPLE__ #:nodoc:
   end
 end

data/lib/statsample/anova.rb

@@ -1,11 +1,86 @@
 module Statsample
   module Anova
-    # One Way Anova
+    class << self
+      def oneway(*args)
+        OneWay.new(*args)
+      end      
+      def oneway_with_vectors(*args)
+        OneWayWithVectors.new(*args)
+      end
+    end
+    # = Generic Anova one-way.
+    # You could enter the sum of squares or the mean squares. You
+    # should enter the degrees of freedom for numerator and denominator.
+    # == Usage
+    #  anova=Statsample::Anova::OneWay(:ss_num=>10,:ss_den=>20, :df_num=>2, :df_den=>10, @name=>"ANOVA for....")
+    class OneWay
+      include GetText
+       bindtextdomain("statsample")
+      attr_reader :df_num, :df_den, :ss_num, :ss_den, :ms_num, :ms_den, :ms_total, :df_total, :ss_total
+      # Name of ANOVA Analisys
+      attr_accessor :name
+      attr_accessor :name_denominator
+      attr_accessor :name_numerator
+      def initialize(opts=Hash.new)
+        # First see if sum of squares or mean squares are entered
+        raise ArgumentError, "You should set d.f." unless (opts.has_key? :df_num and opts.has_key? :df_den)
+        @df_num=opts.delete :df_num
+        @df_den=opts.delete :df_den
+        @df_total=@df_num+@df_den
+        if(opts.has_key? :ss_num and opts.has_key? :ss_den)
+          @ss_num = opts.delete :ss_num
+          @ss_den =opts.delete :ss_den
+          @ms_num =@ss_num.quo(@df_num)
+          @ms_den =@ss_den.quo(@df_den) 
+        elsif (opts.has_key? :ms_num and opts.has_key? :ms_den)
+          @ms_num =opts.delete :ms_num
+          @ms_den =opts.delete :ms_den
+          @ss_num =@ms_num * @df_num
+          @ss_den =@ss_den * @df_den
+        end
+        @ss_total=@ss_num+@ss_den
+        @ms_total=@ms_num+@ms_den
+        opts_default={:name=>"ANOVA",
+                      :name_denominator=>"Explained variance",
+                      :name_numerator=>"Unexplained variance"}
+        @opts=opts_default.merge(opts)
+        opts_default.keys.each {|k|
+          send("#{k}=", @opts[k])
+        }
+        @f_object=Statsample::Test::F.new(@ms_num,@ms_den,@df_num,@df_den)
+      end
+      # F value
+      def f
+        @f_object.f
+      end
+      # P-value of F test
+      def probability
+        @f_object.probability
+      end
+      # Summary of Anova analysis
+      def summary
+        ReportBuilder.new(:no_title=>true).add(self).to_text
+      end
+      def report_building(builder) #:nodoc:
+        builder.section(:name=>@name) do |b|
+          report_building_table(b)
+        end
+      end
+      def report_building_table(builder) #:nodoc:
+        builder.table(:name=>_("%s Table") % @name, :header=>%w{source ss df ms f p}.map {|v| _(v)}) do |t|
+          t.row([@name_numerator, sprintf("%0.3f",@ss_num),   @df_num, sprintf("%0.3f",@ms_num),  sprintf("%0.3f",f), sprintf("%0.3f", probability)])
+          t.row([@name_denominator, sprintf("%0.3f",@ss_den),  @df_den, sprintf("%0.3f",@ms_den), "", ""])
+          t.row([_("Total"), sprintf("%0.3f",@ss_total),  @df_total, sprintf("%0.3f",@ms_total),"",""])
+        end
+      end
+
+    end
+    # One Way Anova with vectors
     # Example:
     #   v1=[2,3,4,5,6].to_scale
     #   v2=[3,3,4,5,6].to_scale
     #   v3=[5,3,1,5,6].to_scale
-    #   anova=Statsample::Anova::OneWay.new([v1,v2,v3])
+    #   anova=Statsample::Anova::OneWayWithVectors.new([v1,v2,v3])
     #   anova.f
     #   => 0.0243902439024391
     #   anova.probability
@@ -13,30 +88,43 @@ module Statsample
     #   anova.sst 
     #   => 32.9333333333333
     #
-    class OneWay < Statsample::Test::F
-      def initialize(vectors,opts=Hash.new)
-        @vectors=vectors
-        opts_default={:name=>_("Anova One-Way"), :name_numerator=>"Between Groups", :name_denominator=>"Within Groups"}
-        super(ssbg,sswg, df_bg, df_wg)
+    class OneWayWithVectors < OneWay
+      # Show on summary Levene test
+      attr_accessor :summary_levene
+      # Show on summary descriptives for vectors
+      attr_accessor :summary_descriptives
+      def initialize(*args)
+        if args[0].is_a? Array
+          @vectors=args.shift
+        else
+          @vectors=args.find_all {|v| v.is_a? Statsample::Vector}
+          opts=args.find {|v| v.is_a? Hash}
+        end
+        opts||=Hash.new
+        opts_default={:name=>_("Anova One-Way"), 
+                      :name_numerator=>"Between Groups",
+                      :name_denominator=>"Within Groups",
+                      :summary_descriptives=>false,
+                      :summary_levene=>false}
+        @opts=opts_default.merge(opts).merge(:ss_num=>ssbg, :ss_den=>sswg, :df_num=>df_bg, :df_den=>df_wg)
+        super(@opts)
+      end
+      alias  :sst :ss_total 
+      def levene
+        Statsample::Test.levene(@vectors, :name=>_("Test of Homogeneity of variances (Levene)"))
       end
       # Total mean
-      def mean
+      def total_mean
         sum=@vectors.inject(0){|a,v| a+v.sum}
         sum.quo(n)
       end
-      
-      # Total sum of squares
-      def sst
-        m=mean
-        @vectors.inject(0) {|total,vector| total+vector.ss(m) }
-      end
       # Sum of squares within groups
       def sswg
         @sswg||=@vectors.inject(0) {|total,vector| total+vector.ss }
       end
       # Sum of squares between groups
       def ssbg
-        m=mean
+        m=total_mean
         @vectors.inject(0) do |total,vector|
           total + (vector.mean-m).square * vector.size 
         end
@@ -56,7 +144,21 @@ module Statsample
       def n
           @vectors.inject(0){|a,v| a+v.size}
       end
-      
+      def report_building(builder) # :nodoc:
+        builder.section(:name=>@name) do |s|
+          if summary_descriptives
+            s.table(:name=>_("Descriptives"),:header=>%w{Name N Mean SD Min Max}.map {|v| _(v)}) do |t|
+              @vectors.each do |v|
+                t.row [v.name, v.n_valid, "%0.4f" % v.mean, "%0.4f" %  v.sd, "%0.4f" % v.min, "%0.4f" % v.max]
+              end
+            end
+          end
+          if summary_levene
+            s.parse_element(levene)
+          end
+          report_building_table(s)
+        end
+      end
     end
   end
 end