experiment 0.2.0 → 0.3.0

data/Manifest.txt

@@ -1,18 +1,20 @@
 History.txt
 Manifest.txt
-README.rdoc
+README.md
 Rakefile
 lib/experiment.rb
 lib/experiment/config.rb
-lib/experiment/stats.rb
+lib/experiment/stats/descriptive.rb
 lib/experiment/runner.rb
 lib/experiment/generator/readme_template.txt
-lib/experiment/generator/experiment_template.rb
+lib/experiment/generator/experiment_template.rb.txt
 lib/experiment/generator/Rakefile
 lib/experiment/base.rb
 lib/experiment/notify.rb
 lib/experiment/work_server.rb
 lib/experiment/distributed.rb
+lib/experiment/factorial.rb
+lib/experiment/params.rb
 test/test_experiment.rb
 test/test_helper.rb
 bin/experiment

data/{README.rdoc → README.md}

@@ -1,15 +1,12 @@
-= Experiment
-* http://github.com/gampleman/experiment
-
-== What's it about?
+## What's it about?
 
 Experiment is a ruby library and environment for running scientific experiments (eg. AI, GA...), especially good for experiments in optimizing results by variations in algorithm or parameters.
 
-== Installation
+## Installation
 
     $ sudo gem install experiment
 
-== Getting started
+## Getting started
 
 Experiment is modeled after rails and the workflow should be recognizable enough.
 
@@ -21,21 +18,23 @@ This will create several files and directories. We will shortly introduce you to
 
 First off is the `app` directory. This is where a basic implementation of what you mean to do. You can write your code however you want, just make sure the code is well structured - you will be overriding this later in your experiments.
 
-== Setting up an experiment
+## Setting up an experiment
 
 Experiments are set up in the experiments directory. The first thing you need to do is define what consist an experiment in your case. For this open up the file `experiments/experiment.rb`. You will notice that this file contains a bunch of comments and a stub letting you easily understand what to do.
 
-For a typical experiment you will need to do some setup work (eg. initialize your classes, calculate parametres, etc.), run the experiment and maybe do cleanup (remove temp. files).
+For a typical experiment you will need to do some setup work (eg. initialize your classes, calculate parameters, etc.), run the experiment and maybe do cleanup (remove temp. files).
 
-You do all this work in the `run_the_experiment` method. Use the `measure` method to wrap your measurements. These will be autmatically benchmarked and their ouput will be automatically saved to the results directory for further analysis.
+You do all this work in the `run_the_experiment` method. Use the `measure` method to wrap your measurements. These will be automatically benchmarked and their output will be automatically saved to the results directory for further analysis.
 
-The `test_data` method lets you specify an array of data points that you want split for cross-validation (see below). This will be passed to in `run_the_experiment` in the input variable.
+The `data_set` method lets you specify an array of data points that you want split for cross-validation. You can access this in your experiment with `test_data`.
 
 Next you may want to analyze the data you got. For that there is the `analyze_result!` method which has 2 arguments. One is the raw data file that was output by your code and the other is the path to an expected output file (this can be very rich in detail, ideal for confusion matrices and the like). The method should return a hash of summary results (eg. `:total_performance => 16`).
 
 All of this will be also saved to disk and available for later analysis.
 
-== Creating an experimental condition
+More info [on the wiki](https://github.com/gampleman/Experiment/wiki/Designing-your-experiment).
+
+## Creating an experimental condition
 
 Now to get to making different conditions and measuring them. First call
 
@@ -46,7 +45,7 @@ This will create a directory in `experiments` based on the name you provide (in
 
 Also notice that the description you provided is stored as a comment in that file. You can expand your hypothesis as you work on the file and it will be included in your report automatically.
 
-== Running the experiment
+## Running the experiment
 
 Once you make the desired changes you can run the experiment with:
 
@@ -58,66 +57,34 @@ The experimental results and benchmarks will be written to this directory with a
 
 Please notice that you can provide several different conditions to the run command and it will run them sequentially, all with required options.
 
-== Configuration
+More on the [Command Line Interface](https://github.com/gampleman/Experiment/wiki/Command-Line-Interface).
+
+## Configuration
 
 So far we have been talking mainly about variations in the source code of the experiments. But what if you just want to tweak a few parameters? There is always the almighty *Config* class to the rescue. 
 
     Experiment::Config[:my_config_variable] # anywhere in your code
     
-Where does this come from? You have a config directory containing a `config.yaml` file. This file contains several environments. The idea is that you might want to tweak your options differently when running on your laptop then when running on a university supercomputer ;)
-
-`development` is the default environment, you can set any other with the `--env` option.
+You have a config directory containing a `config.yaml` file. This file contains several environments. The idea is that you might want to tweak your options differently when running on your laptop then when running on a university supercomputer. Experiments also have their own config file that override the global.
 
-Notice that also when generating an experimental condition it also gets its own config.yaml file. This file overrides the main config file so you can introduce in condition specific options.
+More info on [the wiki](https://github.com/gampleman/Experiment/wiki/Configuration).
 
-And finally when running an experiment you can use the -o or --options option to override any config you want.
 
-Let me give you an example. Your main config file looks like this:
 
-    environments:
-      development:
-        ref_dir: /Users/kubowo/Desktop/points-vals
-        master_dir: /Users/kubowo/Desktop/points-vals/s014
-        alpha: 0.4
-      compute:
-        ref_dir: /afs/group/DB/points
-        master_dir: /afs/group/DB/points/s145
-        alpha: 0.4
+## Cross Validation
 
-Your `experiments/my_condition/config.yaml` looks like this:
+*Cross validation* (CV) is one of the most crucial research methods in CS and AI. For that reason it is built right in. You specify how many CVs you want to run using the --cv flag and your data is automatically split up for you and the experiment is run for each CV with the appropriate data.
 
-    experiment:
-      development:
-        alpha: 0.5
-      compute:
-        alpha: 0.6
-
-And you run the experiment with
-
-    $ experiment run my_condition --env compute -o "master_dir: /Users/kubowo/Desktop/points-vals/s015"
-    
-Then your final config will look like this:
-
-    { :ref_dir => "/afs/group/DB/points", 
-      :master_dir => "/Users/kubowo/Desktop/points-vals/s015", 
-      :alpha => 0.6 }
-
-Flexible, eh? **NEW** Check out the *get* method. It has features like interpolation and defaults.
-
-== Cross Validation
-
-Cross validation (CV) is one of the most crucial research methods in CS and AI. For that reason it is built right in. You specify how many CVs you want to run using the --cv flag and your data is automatically split up for you and the experiment is run for each CV with the appropriate data.
-
-== Reporting Results
+## Reporting Results
 
     $ experiment report
 
-Surprise, surprise. This will create two files in your `report` directory (BTW, this directory is also meant for you to store your report or paper draft). The first is methods.mmd. This takes all the stuff you wrote in the beginnings of your experimental condition files and creates a multi-markdown (http://fletcherpenney.net/multimarkdown/) file out of them (I chose multi-markdown for it's LaTEX support and also it is directly importable into Scrivener, my writing application of choice, available at http://www.literatureandlatte.com/scrivener.html).  
+Surprise, surprise. This will create two files in your `report` directory (BTW, this directory is also meant for you to store your report or paper draft). The first is methods.mmd. This takes all the stuff you wrote in the beginnings of your experimental condition files and creates a [multi-markdown](http://fletcherpenney.net/multimarkdown/) file out of them (I chose multi-markdown for it's LaTEX support and also it is directly importable into Scrivener, my writing application of choice, available at <http://www.literatureandlatte.com/scrivener.html>).  
 
 The second file created is the `data.csv` file which contains the data from all your experiments. It should be importable to Numbers, Excel even Matlab for further analysis and charting.
 
 
-== Distributed computing support
+## Distributed computing support
 
 Newly this library supports a simple distributed model of running experiments. Setup worker computers with the 
 
@@ -125,8 +92,10 @@ Newly this library supports a simple distributed model of running experiments. S
     
 and then run experiments with --distributed flag.
 
-== Misc
+More details: <https://github.com/gampleman/Experiment/wiki/Distributed-Mode>.
+
+## Misc
 
-So that's pretty much the gist of experiment. There's a few other features (and a few soon to come to a gem near you ;-) Growl notifications are now supported. Turn them of by setting growl_notifications to false in your config file.
+So that's pretty much the gist of experiment. There's a few other features (and a few soon to come to a gem near you ;-) Growl notifications are now supported. Turn them off by setting growl_notifications to false in your config file.
 
-Also check out the RDocs: http://rdoc.info/github/gampleman/Experiment/master/frames
+Also check out the [RDocs](http://rdoc.info/github/gampleman/Experiment/master/frames).

data/Rakefile

@@ -15,7 +15,9 @@ $hoe = Hoe.spec 'experiment' do
   #self.post_install_message = 'PostInstall.txt' # TODO remove if post-install message not required
   self.rubyforge_name       = self.name # TODO this is default value
   #self.extra_deps         = [['ruby-growl','>= 1.0']]
-
+  self.summary = "A framework for running Scientific experiments."
+  self.description = "It provides basic command line tools for simply defining things like cross validations, factorial experimental design and basic statistics. All of this can be run in a distributed manner."
+  #self.homepage = 'https://github.com/gampleman/Experiment'
 end
 
 require 'newgem/tasks'

data/bin/experiment

@@ -21,11 +21,12 @@ class App
     @options.verbose = false
     @options.quiet = false
 		@options.env = :development
-		@options.cv = 5
+		@options.cv = nil#5
 		@options.description = ""
 		@options.opts = ""
     @options.distributed = false
     @options.master = "localhost"
+    @options.summary = false
   end
 
   # Parse options, check arguments, then process the command
@@ -57,15 +58,33 @@ class App
       @opts.on('-V', '--verbose')    { @options.verbose = true }  
       @opts.on('-q', '--quiet')      { @options.quiet = true }
 			@opts.on('-e', '--env [ENV]', [:development, :compute], "Sets the environment to run in.")	{ |v| @options.env = v }
-			@opts.on('-c', '--cv CV', Integer, "The number of cross validations to run.")	{ |v| @options.cv = v }
-      @opts.on('-m', '--description M', String, "Description or hypothesis for the condition being generated.")      { |v| @options.description = v }
-      @opts.on('-o', '--options OPTSTRING', String, "Options to override config with. key1:val1,key2:val2") do |v|
-        @options.opts = v
+			
+			if ARGV.first == 'generate' || ARGV.first == "-h"
+			  @opts.separator ""
+        @opts.separator "Options for `generate`:"
+			  @opts.on('-m', '--description M', String, "Description or hypothesis for the condition being generated.")      { |v| @options.description = v }
+			end
+			
+			if ARGV.first == 'run' || ARGV.first == "-h"
+			  @opts.separator ""
+        @opts.separator "Options for `run`:"
+			  @opts.on('-c', '--cv CV', Integer, "The number of cross validations to run.")	{ |v| @options.cv = v }
+			  @opts.on('-o', '--options OPTSTRING', String, "Options to override or define configuration with.", "format as:  key1:val1,key2:val2") do |v|
+          @options.opts = v
+        end
+        @opts.on('--summary', "After a run of the experiment print out the summary to STDOUT.") { @options.summary = true }
+        @opts.on('-D', '--distributed', "Run with a distributed computing mode.", "This will be the master server/work cue.")	{  @options.distributed = true }
+        @opts.separator "  Overrideable options (defined in config/config.yaml)"
+        @opts.separator "    No options defined." unless Experiment::Config::parsing_for_options(@opts, @options)
       end
-      @opts.on('-D', '--distributed', "Run with a distributed computing mode. This will be the master server/work cue.")	{  @options.distributed = true }
-      @opts.on('-a', '--address MODE', String, "Address to the master machine.")	{ |v| @options.master = v }
-      @opts.parse!(@arguments) #rescue return false
       
+      if ARGV.first == 'worker' || ARGV.first == "-h"
+			  @opts.separator ""
+        @opts.separator "Options for `worker`:"
+        @opts.on('-a', '--address MODE', String, "Address to the master machine.")	{ |v| @options.master = v }
+      end
+      
+      @opts.parse!(@arguments) #rescue return false
       process_options
       true      
     end
@@ -107,7 +126,7 @@ for a new experiment"
       parser = RDoc::Parser.for top_level, File.dirname(__FILE__) + "/../lib/experiment/runner.rb", File.read(File.dirname(__FILE__) + "/../lib/experiment/runner.rb"), opts, stats
       d = parser.scan
       d.modules.first.classes.first.method_list.each do |m|
-        if m.comment != ""
+        unless m.comment == "" || m.name == 'initialize' || m.name == 'new'
           puts "== #{m.name == 'new_project' ? 'new' : m.name}"
           puts m.comment
           puts

data/lib/experiment/base.rb

@@ -1,23 +1,44 @@
 require File.dirname(__FILE__) + "/notify"
-require File.dirname(__FILE__) + "/stats"
+require File.dirname(__FILE__) + "/stats/descriptive"
 require File.dirname(__FILE__) + "/config"
+require File.dirname(__FILE__) + "/params"
 require File.dirname(__FILE__) + "/distributed"
 require 'benchmark'
 require "drb/drb"
+require "yaml"
 
 module Experiment
+  # The base class for defining experimental conditons.
+  # @author Jakub Hampl
+  # @see https://github.com/gampleman/Experiment/wiki/Designing-your-experiment
   class Base
-    
+
     include Distributed
     
-    attr_reader :dir, :current_cv, :cvs
-
-  	def initialize(mode, experiment, options, env)
+    @@cleanup_raw_files = false
+    
+    # The directory in which the results will be written to.  
+    attr_reader :dir
+    # The number of the current cross-validation
+    attr_reader :current_cv
+    # The number of overall cross-validations
+    attr_reader :cvs
+    # The file the program is currently set to output to.
+    # Use this if you want to write additional data.
+    attr_reader :output_file
+    
+    # Called internally by the framewrok
+    # @private
+    # @param [:normal, :master, :slave] mode
+    # @param [String] experiment name
+  	def initialize(mode, experiment, options)
   		@experiment = experiment
+  		@options = options
   		case mode
   		  
   		when :normal
   		  @abm = [] 
+  		  
 		  when :master
 		    @abm = []
 		    extend DRb::DRbUndumped
@@ -25,31 +46,51 @@ module Experiment
 	    when :slave
 		    
   		end
-  		Experiment::Config::load(experiment, options, env)
+  		Experiment::Config::load(experiment, options.opts, options.env)
   		@mode = mode
   	end
   	
+  	# Is the experiment done.
   	def done?
   	  @done
 	  end
   	
-  	
+  	# The default analysis function
+  	# Not terribly useful, better to override
+  	# @abstract Override for your own method analysis.
+  	# @param [String] input file path of results written by `measure` calls.
+  	# @param [String] output file path where to optionally write detailed analysis.
+  	# @return [Hash] Summary of analysis.
+  	def analyze_result!(input, output)
+      YAML::load_file(input)
+    end
+    
+    # Sets up actions to do after the task is completed.
+    #
+    # This will be expanded in the future. Currently the only
+    # possible usage is with :delete_raw_files
+    # @example
+    #   after_completion :delete_raw_files
+    # @param [:delete_raw_files] args If called will delete the raw-*.txt files
+    #   in the {dir} after the experiment successfully completes.
+    def self.after_completion(*args)
+      @@cleanup_raw_files = args.include? :delete_raw_files
+    end
     
-    # runs the whole experiment
+    # runs the whole experiment, called by the framework
+    # @private
   	def normal_run!(cv)
   		@cvs = cv || 1
       @results = {}
   		Notify.started @experiment
       split_up_data
   		write_dir!
-  		specification!
-
   		@cvs.times do |cv_num|
   			@bm = []
   			@current_cv = cv_num
   			File.open(@dir + "/raw-#{cv_num}.txt", "w") do |output|
   			  @ouptut_file = output
-  			    run_the_experiment(@data[cv_num], output)
+  			    run_the_experiment
   			end
   			array_merge @results, analyze_result!(@dir + "/raw-#{cv_num}.txt", @dir + "/analyzed-#{cv_num}.txt")
   			write_performance!
@@ -57,44 +98,113 @@ module Experiment
   		end
   		summarize_performance!
   		summarize_results! @results
+  		specification!
+  		cleanup!
   		Notify.completed @experiment
+  		puts File.read(@dir + "/summary.mmd") if @options.summary
   	end
     
+    # Returns the portion of the {data_set} that corresponds 
+    # to the current cross validation number.
+    # @return [Array]
+    def test_data
+      @data[@current_cv]
+    end
+    
+    # Returns the {data_set} that *without* the {test_data}.
+    # @return [Array]
+    def training_data
+      (@data - test_data).flatten
+    end
     
-    # use this evry time you want to do a measurement.
+    # Use this every time you want to do a measurement.
     # It will be put on the record file and benchmarked
-    # automatically
-    # The weight parameter is used for calculating 
-    # Notify::step. It should be an integer denoting how many 
-    # such measurements you wish to do.
+    # automatically.
+    #
+    # @param [Integer] weight Used for calculating 
+    #   Notify::step. It should be an integer denoting how many 
+    #   such measurements you wish to do.
     def measure(label = "", weight = nil, &block)
       out = ""
       benchmark label do
         out = yield
       end
-      @ouptut_file << out
+      if out.is_a? String
+        @ouptut_file << out
+      else
+        YAML::dump(out, @ouptut_file)
+      end
       Notify::step(@experiment, @current_cv, 1.0/weight) unless weight.nil?
     end
     
     
     # Registers and performs a benchmark which is then 
-    # calculated to the total and everage times
+    # calculated to the total and everage times.
+    # 
+    # A lower-level alternative to measure.
   	def benchmark(label = "", &block)
   	  @bm ||= []
   	  @bm << Benchmark.measure("CV #{@current_cv} #{label}", &block)
   	end
 
   	
-    # Creates the results directory for the current experiment
+    
+    
+    # creates a summary of the results and writes to 'summary.mmd'
+  	def summarize_results!(results)
+  	  File.open(@dir + '/results.yaml', 'w' ) do |out|
+  			YAML.dump(results, out)
+  		end
+  		# create an array of arrays
+  		res = results.keys.map do |key| 
+  		  # calculate stats
+  		  a = results[key]
+  		  if a.all? {|el| el.is_a? Numeric }
+  		    [key] + a + [Stats::mean(a), Stats::standard_deviation(a)]
+		    else
+		      [key] + a + ["--", "--"]
+	      end
+		  end
+		  
+		  ls = results.keys.map{|v| [7, v.to_s.length].max }
+  		
+  		ls = ["Std Deviation".length] + ls
+  		res = header_column + res
+  		res = res.transpose
+  		out = build_table res, ls
+  		File.open(@dir + "/summary.mmd", 'w') do |f|
+  		  f << "## Results for #{@experiment} ##\n\n"
+  		  f << out
+		  end
+  	  #results = results.reduce({}) do |tot, res|
+  	  # cv = res.delete :cv
+  	  # tot.merge Hash[res.to_a.map {|a| ["cv_#{cv}_#{a.first}".to_sym, a.last]}]
+  	  #end
+  	  #FasterCSV.open("./results/all.csv", "a") do |csv|
+  	  #  csv << results.to_a.sort_by{|a| a.first.to_s}.map(&:last)
+  	  #end
+  	end
+
+  	
+  	# A silly method meant to be overriden.
+  	# should return an array, which will be then split up for cross-validating.
+  	# @abstract Override this method to return an array.
+  	def data_set
+  	  (1..cvs).to_a
+  	end
+  	
+  	protected
+  	
+  	# Creates the results directory for the current experiment
   	def write_dir!
   		@dir = "./results/#{@experiment}-cv#{@cvs}-#{Time.now.to_i.to_s[4..9]}"
   		Dir.mkdir @dir
   	end
     
     # Writes a yaml specification of all the options used to run the experiment
-  	def specification!
+  	def specification! all = false
   		File.open(@dir + '/specification.yaml', 'w' ) do |out|
-  			YAML.dump({:name => @experiment, :date => Time.now, :configuration => Experiment::Config.to_h, :cross_validations => @cvs}, out )
+  			YAML.dump({:name => @experiment, :date => Time.now, :configuration => (all ? Experiment::Config.to_hash : Experiment::Config.to_h), :cross_validations => @cvs}, out )
   		end
   	end
     
@@ -120,71 +230,59 @@ module Experiment
   			f << total.format("       Average: "+Benchmark::FMTSTR)
   		end
   	end
-    
-    # creates a summary of the results and writes to 'all.csv'
-  	def summarize_results!(results)
-  	  File.open(@dir + '/results.yaml', 'w' ) do |out|
-  			YAML.dump(results, out)
-  		end
-  		
-  		# create an array of arrays
-  		res = results.keys.map do |key| 
-  		  # calculate stats
-  		  a = results[key]
-  		  [key] + a + [Stats::mean(a), Stats::standard_deviation(a)]
-		  end
-		  
-		  ls = results.keys.map{|v| v.to_s.length }
-  		
-  		ls = ["Standard Deviation".length] + ls
-  		res = [["cv"] + (1..cvs).to_a.map(&:to_s) + ["Mean", "Standard Deviation"]] + res
-  		out = ""
-  		res.transpose.each do |col|
+  	
+  	def build_table(table_data, ls)
+  	  out = ""
+  	  table_data.each_with_index do |col, row_num|
   		  col.each_with_index do |cell, i|
   		    l = ls[i]
   		    out << "| "
   		    if cell.is_a?(String) || cell.is_a?(Symbol)
   		      out << sprintf("%#{l}s", cell)
-		      else
+		      elsif cell.is_a? Numeric
 		        out << sprintf("%#{l}.3f", cell)
+	        else
+	          out << sprintf("%#{l}s", cell.to_s)
 		      end
 		      out << " "
   		  end
+  		  
   		  out << "|\n"
-  		end
-  		File.open(@dir + "/summary.mmd", 'w') do |f|
-  		  f << "## Results for #{@experiment} ##\n\n"
-  		  f << out
-		  end
-  	  #results = results.reduce({}) do |tot, res|
-  	  # cv = res.delete :cv
-  	  # tot.merge Hash[res.to_a.map {|a| ["cv_#{cv}_#{a.first}".to_sym, a.last]}]
-  	  #end
-  	  #FasterCSV.open("./results/all.csv", "a") do |csv|
-  	  #  csv << results.to_a.sort_by{|a| a.first.to_s}.map(&:last)
-  	  #end
-  	end
-  	
-  	def result_line
-  	 " Done\n"
+  		  
+  		  if row_num == 0 || row_num == table_data.length - 3
+  		    col.each_index do |i|
+  		      out << "|" + "-" * (ls[i] + 2)
+		      end
+		      out << "|\n"
+	      end	      
+  		end # each_with_index
+  		
+  		return out
   	end
   	
-  	# A silly method meant to be overriden.
-  	# should return an array, which will be then split up for cross-validating
-  	def test_data
-  	  (1..cvs).to_a
-  	end
   	
   	def split_up_data
   	  @data = []
-  	  test_data.each_with_index do |item, i|
+  	  data_set.each_with_index do |item, i|
   	    @data[i % cvs] ||= []
   	    @data[i % cvs] << item
   	  end
   	  @data
   	end
   	
-  	private
+  	# Performs cleanup tasks
+  	def cleanup!
+  	  if @@cleanup_raw_files
+  	    FileUtils.rm Dir[@dir + "/raw-*.txt"]
+	    end
+  	end
+  	
+  	
+  	
+  	def header_column
+  	  [["cv"] + (1..cvs).to_a.map(&:to_s) + ["Mean", "Std Deviation"]]
+  	end
+  	
   	# Yields a handle to the performance table
   	def performance_f(&block) # just a simple wrapper to make code a little DRYer
   		File.open(@dir+"/performance_table.txt", "a", &block) 
@@ -196,5 +294,7 @@ module Experiment
   	   h1[key] << value
   	  end
 	  end
+
+	  
   end
 end