statsample 0.3.0

data/History.txt

@@ -0,0 +1,79 @@
+=== 0.3.0 / 2009-08-02
+
+* Statsample renamed to Statsample
+* Optimization extension goes to another gem: ruby-statsample-optimization
+
+=== 0.2.0 / 2009-08-01
+
+* One Way Anova on Statsample::Anova::OneWay
+* Dominance Analysis!!!! The one and only reason to develop a Multiple Regression on pure ruby.
+* Multiple Regression on Multiple Regression module. Pairwise (pure ruby) or MultipleRegressionPairwise and Listwise (optimized) on MultipleRegressionAlglib and 
+* New Dataset#to_gsl_matrix, #from_to,#[..],#bootstrap,#vector_missing_values, #vector_count_characters, #each_with_index, #collect_with_index
+* New Vector#box_cox_transformation
+* Module Correlation renamed to Bivariate
+* Some fancy methods and classes to create Summaries
+* Some documentation about Algorithm used on doc_latex
+* Deleted 'distributions' extension. Ruby/GSL has all the pdf and cdf you ever need.
+* Tests work without any dependency. Only nags about missing deps. 
+* Test for MultipleRegression, Anova, Excel, Bivariate.correlation_matrix and many others 
+
+=== 0.1.9 / 2009-05-22
+
+* Class Vector: new method vector_standarized_pop, []=, min,max
+* Class Dataset: global variable $RUBY_SS_ROW stores the row number on each() and related methods. dup() with argument returns a copy of the dataset only for given fields. New methods: standarize, vector_mean, collect, verify,collect_matrix
+* Module Correlation: new methods covariance, t_pearson, t_r, prop_pearson, covariance_matrix, correlation_matrix, correlation_probability_matrix
+* Module SRS: New methods estimation_n0 and estimation_n
+* Module Reliability: new ItemCharacteristicCurve class
+* New HtmlReport class
+* New experimental SPSS Class. 
+* Converters: Module CSV with new options. Added write() method for GGobi module 
+* New Mx exporter (http://www.vcu.edu/mx/)
+* Class SimpleRegression: new methods standard error
+
+* Added tests for regression and reliability, Vector#vector_mean, Dataset#dup (partial)  and Dataset#verify
+
+
+=== 0.1.8 / 2008-12-10
+* Added Regression and Reliability modules
+* Class Vector: added methods vector_standarized, recode, inspect, ranked
+* Class Dataset: added methods vector_by_calculation, vector_sum, filter_field
+* Module Correlation: added methods like spearman, point biserial and tau-b
+* Added tests for Vector#ranked, Vector#vector_standarized,  Vector#sum_of_squared_deviation, Dataset#vector_by_calculation, Dataset#vector_sum, Dataset#filter_field and various test for Correlation module
+* Added demos: item_analysis and sample_test
+
+=== 0.1.7 / 2008-10-1
+* New module for codification
+* ...
+=== 0.1.6 / 2008-09-26
+* New modules for SRS and stratified sampling
+* Statsample::Database for read and write onto databases.
+  You could use Database and CSV on-tandem for mass-editing and reimport
+  of databases
+
+=== 0.1.5 / 2008-08-29
+* New extension statsampleopt for optimizing some functions on Statsample submodules
+* New submodules Correlation and Test
+
+=== 0.1.4 / 2008-08-27
+
+* New extension, with cdf functions for
+  chi-square, t, gamma and normal distributions. 
+  Based on dcdflib (http://www.netlib.org/random/)
+  Also, has a function to calculate the tail for a noncentral T distribution
+
+=== 0.1.3 / 2008-08-22
+
+* Operational versions of Vector, Dataset, Crosstab and Resample
+* Read and write CSV files
+* Calculate chi-square for 2 matrixes
+
+=== 0.1.1 - 0.1.2 / 2008-08-18
+
+* Included several methods on Ruby::Type classes
+* Organized dirs with sow
+
+
+=== 0.1.0 / 2008-08-12
+
+* First version.
+

data/Manifest.txt

@@ -0,0 +1,56 @@
+History.txt
+Manifest.txt
+README.txt
+Rakefile
+bin/statsample
+demo/benchmark.rb
+demo/chi-square.rb
+demo/dice.rb
+demo/distribution_t.rb
+demo/graph.rb
+demo/item_analysis.rb
+demo/mean.rb
+demo/proportion.rb
+demo/sample_test.csv
+demo/strata_proportion.rb
+demo/stratum.rb
+lib/statsample.rb
+lib/statsample/anova.rb
+lib/statsample/bivariate.rb
+lib/statsample/chidistribution.rb
+lib/statsample/codification.rb
+lib/statsample/converters.rb
+lib/statsample/crosstab.rb
+lib/statsample/dataset.rb
+lib/statsample/dominanceanalysis.rb
+lib/statsample/dominanceanalysis/bootstrap.rb
+lib/statsample/graph/gdchart.rb
+lib/statsample/graph/svggraph.rb
+lib/statsample/graph/svgboxplot.rb
+lib/statsample/graph/svghistogram.rb
+lib/statsample/graph/svgscatterplot.rb
+lib/statsample/htmlreport.rb
+lib/statsample/multiset.rb
+lib/statsample/regression.rb
+lib/statsample/reliability.rb
+lib/statsample/resample.rb
+lib/statsample/srs.rb
+lib/statsample/test.rb
+lib/statsample/vector.rb
+lib/spss.rb
+test/_test_chart.rb
+test/test_anova.rb
+test/test_codification.rb
+test/test_crosstab.rb
+test/test_csv.csv
+test/test_csv.rb
+test/test_dataset.rb
+test/test_ggobi.rb
+test/test_multiset.rb
+test/test_regression.rb
+test/test_reliability.rb
+test/test_resample.rb
+test/test_statistics.rb
+test/test_stratified.rb
+test/test_svg_graph.rb
+test/test_vector.rb

data/README.txt

@@ -0,0 +1,77 @@
+= Statsample
+
+* http://rubyforge.org/projects/ruby-statsample/
+* http://code.google.com/p/ruby-statsample/
+
+== DESCRIPTION:
+
+This package allows to process files and databases for statistical purposes, with focus on validation, recodification and estimation of parameters for several types of samples (simple random, stratified and multistage sampling).
+
+== FEATURES:
+
+* Multiple Regression. Listwise analysis optimized with use of Alglib library. Pairwise analysis is executed on pure ruby and reports same values as SPSS
+* Dominance Analysis. Based on Bodescu and Azen papers, DominanceAnalysis? class can report dominance analysis for a sample and DominanceAnalysis?. Bootstrap can execute bootstrap analysis to determine dominance stability, as recomended by Azen & Bodescu (2003). 
+* Classes for Vector, Datasets (set of Vectors) and Multisets (multiple datasets with same fields and type of vectors), and multiple methods to manipulate them
+* Module Codification, to help to codify open questions
+* Converters to and from database and csv files, and to output Mx and GGobi files
+* Module Correlation provides covariance and pearson, spearman, point biserial, tau a, tau b and gamma correlations. Include methods to create correlation and covariance matrices
+* Module Crosstab provides function to create crosstab for categorical data
+* Module HtmlReport provides methods to create a report for scale analysis and matrix correlation
+* Regression module provides linear regression methods
+* Reliability analysis provides functions to analyze scales. Class ItemAnalysis provides statistics like mean, standard deviation for a scale, alpha and standarized alpha, and for each item: mean, correlation with total scale, mean if deleted, alpha is deleted. With HtmlReport, graph the histogram of the scale and the Item Characteristic Curve for each item
+* Module SRS (Simple Random Sampling) provides a lot of functions to estimate standard error for several type of samples
+* Interfaces to gdchart, gnuplot and SVG::Graph 
+
+
+== Example of use:
+
+    # Read a CSV file, using '' and 'error' as missing values and ommiting 1 lines
+    ds=Statsample::CSV.read('resultados_c1.csv',['','error'],1)
+    
+    # Create a new vector (column), calculating the mean of 13 vectors. Accept 1 missing values on one of the vectors
+    
+    indice_constructivismo_becker=ds.vector_mean(%w{fd_2_1 fd_2_2 fd_3_1 fd_3_2 fd_3_3},1)
+    
+    # Add the vector to the dataset
+    
+    ds.add_vector("ind_cons_becker",indice_constructivismo_becker)
+    
+    # Verify data. Vecto 'de_3_sex' must have values 'a' or 'b'. Dataset#verify returns and array with all errors
+    
+    t_sex=create_test("Sex must be a o b",'de_3_sex') {|v| v['de_3_sex']=="a" or v['de_3_sex']=="b")}
+    
+    p ds.verify(t_sexo)
+    
+    
+    # Creates a new dataset, based on the names of vectors
+    
+    ds_software=ds.dup(%w{pe1n1 pe1n2 pe1n3 pe1n4 pe1n5 })
+    
+    # Creates an html report, add a correlation matrix with all the scale vectors and save the report into a file 
+    hr=Statsample::HtmlReport.new(ds_software,"correlations")
+    hr.add_correlation_matrix()
+    hr.save("correlation_matrix.html")
+    
+    
+    # Saves the new dataset
+    Statsample::CSV.write(ds_software,"ds_software.csv",true)
+
+== REQUIREMENTS:
+
+Optional: 
+
+* Plotting: gnuplot and rbgnuplot, SVG::Graph
+* Advanced Statistical: gsl and rb-gsl (http://rb-gsl.rubyforge.org/)
+
+== INSTALL:
+
+gem install ruby-statsample
+
+For optimization on *nix env
+
+gem install ruby-statsample-optimization
+
+
+== LICENSE:
+
+GPL-2

data/Rakefile

@@ -0,0 +1,22 @@
+#!/usr/bin/ruby
+# -*- ruby -*-
+
+require 'rubygems'
+require 'hoe'
+require './lib/statsample'
+
+if File.exists? './local_rakefile.rb'
+	require './local_rakefile'
+end
+	
+Hoe.spec('statsample') do |p|
+	p.version=Statsample::VERSION
+	p.rubyforge_name = "ruby-statsample"
+	p.developer('Claudio Bustos', 'clbustos@gmail.com')
+	p.extra_deps << ["spreadsheet","=0.6.4"] << "svg-graph"
+	p.clean_globs << "test/images/*" 
+	#  p.rdoc_pattern = /^(lib|bin|ext\/distributions)|txt$/
+end
+
+
+# vim: syntax=Ruby

data/bin/statsample

@@ -0,0 +1,2 @@
+#!/usr/bin/ruby
+echo "Nothing today!"

data/demo/benchmark.rb

@@ -0,0 +1,52 @@
+require File.dirname(__FILE__)+'/../lib/statsample.rb'
+require 'benchmark'
+v=(0..10000).collect{|n|
+	r=rand(100)
+    if(r<90)
+    r 
+    else
+        nil
+    end
+}.to_vector
+v.missing_values=[5,10,20]
+v.type=:scale
+
+ n = 300
+ if (false)
+     Benchmark.bm(7) do |bench|
+         bench.report("missing or")   { for i in 1..n; v.each {|x|             !(x.nil? or v.missing_values.include? x) }; end }
+         bench.report("missing and")   { for i in 1..n;v.each {|x|             !x.nil? and !v.missing_values.include? x } ; end }
+    end
+ end
+ if (false)
+     Benchmark.bm(7) do |bench|
+         bench.report("true")   { Statsample::OPTIMIZED=true; for i in 1..n; v.set_valid_data ; end }
+         bench.report("false")   { Statsample::OPTIMIZED=false; for i in 1..n; v.set_valid_data ; end }
+    end
+ end
+
+
+ if (true)
+    Benchmark.bm(7) do |x|
+		x.report("mean")   { for i in 1..n; v.mean; end }
+		x.report("slow_mean")   { for i in 1..n; v.slow_mean; end }
+
+    end
+
+    Benchmark.bm(7) do |x|
+		x.report("variance_sample")   { for i in 1..n; v.variance_sample; end }
+		x.report("variance_slow")   { for i in 1..n; v.slow_variance_sample; end }
+
+    end
+
+
+    Benchmark.bm(7) do |x|
+		
+		x.report("Nominal.frequencies")   { for i in 1..n; v.frequencies; end }
+		x.report("Nominal.frequencies_slow")   { for i in 1..n; v.frequencies_slow; end }
+
+		x.report("_frequencies")   { for i in 1..n; Statsample._frequencies(v.valid_data); end }
+
+    end
+
+end

data/demo/chi-square.rb

@@ -0,0 +1,44 @@
+require File.dirname(__FILE__)+'/../lib/statsample'
+require 'rbgsl'
+require 'statsample/resample'
+require 'statsample/test'
+require 'matrix'
+ideal=Matrix[[30,30,40]]
+tests=10000
+monte=Statsample::Resample.repeat_and_save(tests) {
+	observed=[0,0,0]
+	(1..100).each{|i|
+		r=rand(100)
+		if r<30
+			observed[0]+=1
+		elsif r<60
+			observed[1]+=1
+		else
+			observed[2]+=1
+		end
+	}
+	Statsample::Test::chi_square(Matrix[observed],ideal)
+}
+
+
+
+v=monte.to_vector(:scale)
+
+x=[]
+y=[]
+y2=[]
+y3=[]
+y4=[]
+prev=0
+prev_chi=0
+v.frequencies.sort.each{|k,v1|
+	x.push(k)
+	y.push(prev+v1)
+	prev=prev+v1
+	cdf_chi=GSL::Cdf.chisq_P(k,2)
+	y2.push(cdf_chi)
+	y4.push(prev.quo(tests))
+}
+
+
+GSL::graph(GSL::Vector.alloc(x), GSL::Vector.alloc(y2), GSL::Vector.alloc(y4))

data/demo/dice.rb

@@ -0,0 +1,13 @@
+require File.dirname(__FILE__)+"/../lib/statsample"
+require 'statsample/srs'
+require 'statsample/resample'
+require 'gnuplot'
+
+tests=3000
+# rand a 50%
+monte_with=Statsample::Resample.repeat_and_save(tests) {
+    (1+rand(6))+(1+rand(6))
+}.to_vector(:scale)
+
+p monte_with.mean
+

data/demo/distribution_t.rb

@@ -0,0 +1,95 @@
+#!/usr/bin/ruby
+
+require File.dirname(__FILE__)+"/../lib/statsample"
+require 'statsample/resample'
+require 'gnuplot'
+r = GSL::Rng.alloc(GSL::Rng::TAUS, 1)
+v=[]
+population_size=10000
+population_size.times{|i|
+    v.push(r.ugaussian)
+}
+
+v=v.to_vector(:scale)
+vm=v.mean
+vsd=v.sdp
+puts "Population sd:#{v.sdp}"
+tests=3000
+Gnuplot.open do |gp|
+			Gnuplot::Plot.new( gp ) do |plot|
+			plot.boxwidth("0.9 absolute")
+			plot.xrange("[-3:3]")
+			plot.yrange("[0:1.1]")
+			plot.style("fill  solid 1.00 border -1")
+[2].each {|ss|
+    puts "Sample size:#{ss}"
+    ee=v.sdp.quo(Math::sqrt(ss))
+    puts "SE: #{ee}"
+
+    puts "Expected variance with replacement: #{v.variance_population.quo(ss)*(v.size-1).quo(v.size)}"
+    puts "Expected variance without replacement: #{v.variance_population.quo(ss)*(1-ss.quo(v.size))}"
+    
+	sample_size=ss
+    sds_prom=[]
+    sds_prom_wo=[]
+	monte_wr=Statsample::Resample.repeat_and_save(tests) {
+		sample=v.sample_with_replacement(sample_size)
+        sds_prom.push(sample.sds)
+		sample.mean
+	}
+    monte_wor=Statsample::Resample.repeat_and_save(tests) {
+		sample=v.sample_without_replacement(sample_size)
+        sds_prom_wo.push(sample.sds)
+		sample.mean
+	}
+    xxz=[]
+    xxt=[]
+	xa=[]
+	xy=[]
+	xt=[]
+	xz=[]
+    
+    s_wr=sds_prom.to_vector(:scale).mean
+    s_wor=sds_prom_wo.to_vector(:scale).mean
+
+	mw=monte_wr.to_vector(:scale)
+    mwo=monte_wor.to_vector(:scale)
+    puts "Sample variance with replacement: #{mw.variance_population}"
+    puts "Sample variance without replacement: #{monte_wor.to_vector(:scale).variance_population}"
+    puts "Mean sd estimadet :#{vsd*Math::sqrt(ss-1)}"
+    puts "Mean Sd W/R: #{s_wr}" 
+    puts "Mean Sd WO/R: #{s_wor}" 
+
+	mx=mw.mean
+    er=mw.sds
+
+	prev=0
+	mw.frequencies.sort.each{|x,y|
+        t=(x-vm).quo(s_wr.quo(Math::sqrt(ss))*s_wr.quo(ss-1))
+        z=(x-vm).quo(vsd.quo(Math::sqrt(ss)))
+		xxz.push(z)
+        xxt.push(t)
+		prev+=y
+		xy.push(prev.to_f/tests)
+		xt.push(GSL::Cdf.tdist_P(t, ss-1))
+		xz.push(GSL::Cdf.gaussian_P(z))
+
+	}
+	plot.data << Gnuplot::DataSet.new( [xxt,xy] ) do |ds|
+		ds.with="lines"
+		ds.title = "sim #{sample_size}"
+	end
+	plot.data << Gnuplot::DataSet.new( [xxt,xt] ) do |ds|
+		ds.with="lines"
+		ds.title = "t #{sample_size}"
+	end
+	plot.data << Gnuplot::DataSet.new( [xxz,xz] ) do |ds|
+		ds.with="lines"
+		ds.title = "z"
+	end
+	
+}
+
+end
+end
+