pest 0.0.0 → 0.1.0

data/README.md

@@ -2,18 +2,10 @@
 
 [![Build Status](https://secure.travis-ci.org/kerinin/pest.png)](http://travis-ci.org/kerinin/pest)
 
+**A concise API focused on painless investigation of data sets**
 
-Pest provides a unified framework for interacting with different probability
-estimation models.  
-
-* Pest tries to be agnostic about the underlying data data structures, 
-so changing libraries (GSL -> Hadoop) is as simple as using a different data source.
-* Pest is designed to create estimators using subsets of larger data sources, and
-transparently constructs estimators to facilitate dynamic querying
-* Implementing custom estimation models is easy, and Pest implements some model
-common ones for you.
-
-Pest abstracts common statstical operations including:
+Pest provides a framework for interacting with different probability
+estimation models. Pest abstracts common statstical operations including:
 
 * Marginal, Joint and Conditional point probability
 * Interval and Cumulative probability
@@ -21,24 +13,34 @@ Pest abstracts common statstical operations including:
 * Mean, Median, Mode, etc
 
 
-## Ruby Install
+**Scalability if you need it**
 
-``` sh
-brew install gnuplot  # This may take awhile...
-cd /usr/local
-git checkout 83ed494 /usr/local/Library/Formula/gsl.rb
-brew install gsl      # Forcing gsl v1.4
+Pest tries to be agnostic about the underlying data data structures, 
+so changing libraries (NArray -> Hadoop) is as simple as using a different data source.
+Pest is designed to create estimators using subsets of larger data sources, and
+transparently constructs estimators to facilitate dynamic querying
 
-bundle install 
-```
+
+**Code structure designed to be extended**
+
+Implementing custom estimation models is easy, and Pest implements some model
+common ones for you.
+
+
+## Install
+
+Add it to your Gemfile and bundle
+
+    gem "pest"
+
+    bundle install 
 
 ## API
 
 ``` ruby
 # Creating Datasets
-test = Pest::DataSet::Hash.new hash                   # Creates a Hash dataset of observations from a hash
-test = Pest::DataSet::Hash.new file                   # Creates a Hash dataset of observations from an IO (Marshalled) 
-train = Pest::DataSet::GSL.new file                   # Creates a GSL dataset from and IO instance
+test = Pest::DataSet::Hash.from_hash hash             # Creates a Hash dataset of observations from a hash
+train = Pest::DataSet::NArray.from_hash hash          # Creates a NArray dataset
 
 # DataSet Variables
 test.variables                                        # hash of Variable instances detected in observation set
@@ -47,13 +49,14 @@ test.v[:foo]                                          # a specific variable
 test.v[:foo] = another_variable                       # explicit declaration
 
 # Creating Estimators
-e = Pest::Estimator::Set::Multinomial.new(test)       # Creates a multinomial estimator for set o
-e = Pest::Estimator::Discrete::Gaussian.new(file)     # Creating an estimator with the DataSet API
+e = Pest::Estimator::Frequency.new(data)              # Frequentist estimator - values treated as unordered set
+e = Pest::Estimator::Multinomial.new(data)            # Multinomial estimator
+e = Pest::Estimator::Gaussian.new(data)               # Gaussian mean/varaince ML estimator
 
 # Descriptive Statistical Properties
-e.mode(:foo)                                          # Mode
-e.mean(:foo)                                          # Mean (discrete & continuous only)
-e.median(:foo)                                        # Median (discrete & continuous only)
+#e.mode(:foo)                                          # Mode
+#e.mean(:foo)                                          # Mean (discrete & continuous only)
+#e.median(:foo)                                        # Median (discrete & continuous only)
 # quantile?
 # variance?
 # deviation?
@@ -65,26 +68,25 @@ e.h(:foo).given(:bar)                                 # Cross entropy of 'foo' :
 e.mutual_information(:foo, :bar)                      # Mutual information of 'foo' and 'bar'
 e.i(:foo, :bar)                                       # Alias
 
-# Estimating Point Probability (Set & Discrete only)
-e.probability(o.variables[:foo])                      # (Set/Discrete only) Estimate the probability of all values of 'foo'
-e.p(:foo)                                             # Same as above, tries to find a variable named 'foo'
-e.p(:foo).in(test)                                    # Estimate the probability of values in dataset 'test'
-e.p(:foo).given(:bar).in(test)                        # Estimate the conditional foo | bar for the values in 'test'
-e.p(:foo, :bar).in(test)                              # Estimate the joint probablity foo AND bar
-e.p(:foo, :bar).given(:baz, :qux).in(test)            # More complex joint & conditional probabilities
-e.p(:foo => 4, :bar => 2).given(:baz => 0)            # Single prediction (implicitly creates dataset)
-e.p(:foo).given(:bar).cache                           # Builds and persists the model for 'foo|bar'
-e.p(:foo).given(:bar).cache('path.csv')               # Persist to a specific path (defaults to tmp)
-
-# Estimating Cumulative & Interval Probability (Discrete & Continuous only)
-e.probability(:foo).greater_than(:bar).in(test)
-e.p(:foo).greater_than(:bar).less_than(:baz).in(test)
-e.p(:foo).gt(:bar).lt(:baz).given(:qux).in(test)
+# Estimating Point Probability
+e.probability(e.variables[:foo] => 1)                 # Estimate the probability that foo=1
+e.p(:foo => 1)                                        # Same as above, tries to find a variable named 'foo'
+e.p(:foo => 1, :bar => 2)                             # Estimate the probability that foo=1 AND bar=2
+e.p(:foo => 1).given(:bar => 2)                       # Estimate the probability that foo=1 given bar=2
+e.p(:foo => 1, :bar => 2).given(:baz => 3, :qux => 4) # Moar
+
+# Batch Point Probability Estimation
+e.batch_probability(:foo).in(test)                    # Estimate the probability of each value in test
+e.batch_p(:foo, :bar).in(test)                        # Joint probability
+e.batch_p(:foo).given(:bar).in(test)                  # Conditional probability
+e.batch_p(:foo, :bar).given(:baz, :qux).in(test)      # Moar
+
+# Estimating Cumulative & Interval Probability
+#e.probability(:foo).greater_than(:bar).in(test)
+#e.p(:foo).greater_than(:bar).less_than(:baz).in(test)
+#e.p(:foo).gt(:bar).lt(:baz).given(:qux).in(test)
 ```
 
-## Working Notes
+## TODO
 
-Do we want variable equality to be name-based?  It may make more sense to allow
-variables named differently in different data sets to be equivalent. And how the
-fuck do we handle variable type?  I'm almost thinking we don't, and let the actual
-estimators take care of type casting
+the builders should validate the variables they're given and throw errors if they're not part of the estimators data

data/VERSION

@@ -1 +1 @@
-0.0.0
+0.1.0

data/lib/pest.rb

@@ -5,7 +5,6 @@ require 'uuidtools'
 require 'csv'
 
 require "pest/version"
-require "pest/variable"
 
 require "pest/function"
 require "pest/function/probability"

data/lib/pest/data_set.rb

@@ -3,13 +3,15 @@ module Pest::DataSet
     base.extend(ClassMethods)
   end
 
-  def variables
-    @variables ||= {}
-  end
+  include Enumerable
+
+  attr_accessor :variables, :data
 
-  def variable_array
-    @variables.values.sort
+  def initialize(variables = Set.new, data = nil)
+    @variables = variables
+    @data = data
   end
+  alias :v :variables
 
   def to_hash(*args)
     raise NotImplementedError
@@ -27,6 +29,44 @@ module Pest::DataSet
     raise NotImplementedError
   end
 
+  def ==(other)
+    variables == other.variables and data == other.data
+  end
+  alias :eql? :==
+
+  def [](*args)
+    raise NotImplementedError
+  end
+
+  def except(start, finish)
+    left = start > 0 ? self[0..start] : nil
+    right = finish < length - 1 ? self[finish..-1] : nil
+    case [left.nil?, right.nil?]
+    when [true, false]
+      right
+    when [false, true]
+      left
+    when [false, false]
+      right + left
+    end
+  end
+
+  def +(other)
+    raise NotImplementedError
+  end
+
+  def pick(*args)
+    raise NotImplementedError
+  end
+
+  def each(&block)
+    raise NotImplementedError
+  end
+
+  def merge(other)
+    raise NotImplementedError
+  end
+
   module ClassMethods
     def from(data_source)
       # Try to translate the data source directly

data/lib/pest/data_set/hash.rb

@@ -3,68 +3,102 @@ class Pest::DataSet::Hash
 
   def self.translators
     {
-      File    => :from_file,
       String  => :from_file,
       Symbol  => :from_file
     }
   end
 
-  def self.from_file(file)
-    file = File.open(file.to_s, 'r') if file.kind_of?(String)
-
-    object = Marshal.restore(file)
-
-    if object.kind_of?(::Hash)
-      self.new(object)
-    else
-      raise "File does not seem to contain valid data"
-    end
+  def self.from_hash(hash)
+    data_set = new
+    data_set.variables += hash.keys
+    data_set.instance_variable_set(:@hash, hash)
+    data_set
   end
 
   attr_reader :variables, :hash
 
-  def initialize(hash)
-    @hash = hash
-    @variables = {}
-    hash.keys().each do |name|
-      @variables[name] = Pest::Variable.new(:name => name)
-    end
+  def initialize(*args)
+    super *args
+    @hash = {}
   end
 
-  def to_hash
-    @hash
+  def data
+    hash.values
   end
 
-  def data_vectors(variables=nil)
-    VectorEnumerable.new(self,variables)
+  def to_hash
+    hash
   end
 
   def length
     @hash.values.first.length
   end
 
-  def save(file=nil)
-    file ||= Tempfile.new('pest_hash_dataset')
-    file = File.open(file, 'w') if file.kind_of?(String)
-    Marshal.dump(@hash, file)
+  def [](*args)
+    unless args.any?
+      raise ArgumentError, "Indices not specified"
+    end
+
+    args.map do |arg|
+      subset = self.class.new
+      subset.variables = self.variables
+      variables.each do |var|
+        subset.hash[var.name] = hash[var.name][arg]
+      end
+      subset
+    end.inject(:+)
+ 
   end
 
-  class VectorEnumerable
-    include Enumerable
+  def +(other)
+    unless other.variables == variables
+      raise ArgumentError, "DataSets have different variables"
+    end
 
-    def initialize(data_set,variables=nil)
-      @data_set = data_set
-      @variables = variables || @data_set.variables
+    union = self.class.new
+    union.variables = variables
+    variables.each do |var|
+      union.hash[var.name] = hash[var.name] + other.hash[var.name]
     end
+    union
+  end
 
-    def [](i)
-      @variables.map {|var| @data_set.hash[var][i]}
+  def pick(*args)
+    unless args.any?
+      raise ArgumentError, "You didn't specify any variables to pick"
     end
 
-    def each
-      @data_set.hash.values.first.each_index do |i|
-        yield @variables.keys.map {|var| @data_set.hash[var][i]}
-      end
+    subset = self.class.new
+    subset.variables += args
+    args.each do |var|
+      raise ArgumentError, "Dataset doesn't include '#{var}'" unless hash.has_key?(var)
+      subset.hash[var] = hash[var]
     end
+    subset
+  end
+
+  def each(&block)
+    (0..length-1).to_a.each do |i| yield variables.map {|var| hash[var][i]} end
+  end
+
+  def dup
+    instance = self.class.new
+    instance.variables = variables.dup
+    instance.instance_variable_set(:@hash, hash.dup)
+    instance
+  end
+
+  def merge(other)
+    dup.merge!(other)
+  end
+
+  def merge!(other)
+    other = self.class.from_hash(other) if other.kind_of?(::Hash)
+    raise ArgumentError, "Lengths must be the same" if other.length != length
+
+    @variables += other.variables
+    hash.merge! other.hash
+
+    self
   end
 end

data/lib/pest/data_set/narray.rb

@@ -1,6 +1,6 @@
 require 'narray'
 
-class Pest::DataSet::NArray < NMatrix
+class Pest::DataSet::NArray
   include Pest::DataSet
 
   def self.translators
@@ -13,86 +13,137 @@ class Pest::DataSet::NArray < NMatrix
   end
 
   def self.from_hash(hash)
-    data_set = to_na(hash.keys.sort.map {|key| hash[key]}) # Ensure the matrix is sorted the same as the variables
-    data_set.variables = {}
-    hash.keys.each do |key|
-      variable = key.kind_of?(Pest::Variable) ? key : Pest::Variable.new(:name => key)
-      data_set.variables[variable.name] = variable
-    end
+    data_set = new(
+      hash.keys.to_set, 
+      NMatrix.to_na(hash.values)
+    )
+    data_set.instance_variable_set(:@variable_array, hash.keys)
     data_set
   end
 
   def self.from_file(file)
-    file = File.open(file.to_s, 'r') if file.kind_of?(String)
-
-    begin
-      variables, matrix = Marshal.restore(file)
-      data_set = to_na(matrix)
-      data_set.variables = variables
-      data_set
-    rescue
-      raise "File does not seem to contain valid data"
-    end
+    from_csv(file)
   end
 
   def self.from_csv(file, args={})
-    args = args.merge({:converters => :all})
-    data = CSV.read(file, args)
-    data_set = to_na(data[1..-1]).transpose
-    data_set.variables = {}
-    data[0].each do |key|
-      variable = key.kind_of?(Pest::Variable) ? key : Pest::Variable.new(:name => key)
-      data_set.variables[variable.name] = variable
-    end
+    args = {:col_sep => "\t", :headers => true, :converters => :all}.merge args
+    csv_data = CSV.read(file, args).map(&:to_hash)
+
+    data_set = new(
+      csv_data.first.keys.to_set,
+      NMatrix.to_na(csv_data.map(&:values)).transpose
+    )
+    # Ensure the ordering matches what's in the CSV
+    data_set.instance_variable_set(:@variable_array, csv_data.first.keys)
     data_set
   end
 
-  attr_accessor :variables
+  attr_reader :variable_array
+
+  def initialize(*args)
+    super *args
+    @variable_array = variables.to_a.sort
+  end
 
   def to_hash
     hash = {}
-    variables.values.each_index do |i|
-      hash[variables.values[i]] = self[true,i].to_a[0]
+    variable_array.each_index do |i|
+      hash[variable_array[i]] = data[true,i].to_a[0]
     end
     hash
   end
 
-  # variables: an array of variables for which each vector should contain values
-  # Order is retained in the returned value
-  def data_vectors(variables=nil)
-    VectorEnumerable.new(self, variables)
+  def length
+    data.shape[0]
   end
 
-  def length
-    shape[0]
+  # Return a subset of the data with the same variables,
+  # but only the vectors specified by i
+  #
+  def [](*args)
+    unless args.any?
+      raise ArgumentError, "Indices not specified"
+    end
+
+    args.map do |arg|
+      subset = self.class.new
+      subset.variables = self.variables
+      subset.data = self.data[arg,true]
+      subset
+    end.inject(:+)
   end
 
-  def save(file=nil)
-    file ||= Tempfile.new('pest_hash_dataset')
-    file = File.open(file, 'w') if file.kind_of?(String)
-    Marshal.dump([variables,to_a], file)
-    file.close
+
+  # Return the union of self and other
+  #
+  def +(other)
+    unless other.variables == variables
+      raise ArgumentError, "DataSets have different variables"
+    end
+
+    union = self.class.new
+    union.variables = variables
+    union.data = NMatrix[*(data.transpose.to_a + other.data.transpose.to_a)].transpose
+    union
   end
 
-  class VectorEnumerable
-    include Enumerable
+  # Return a subset of the data with the same vectors, but only
+  # the variables specified in args
+  #
+  def pick(*args)
+    unless args.any?
+      raise ArgumentError, "You didn't specify any variables to pick"
+    end
 
-    def initialize(data_set, variables = true)
-      @data_set = data_set
-      @variables = variables
-      if @variables.kind_of?(Enumerable)
-        @variables = variables.map {|v| @data_set.variable_array.index(v)}
-      end
+    picked_indices = args.map do |variable|
+      raise ArgumentError, "Dataset doesn't include #{variable}" unless variables.include?(variable)
+      self.variable_array.index(variable)
     end
 
-    def [](i)
-      @data_set[i,@variables].transpose
+    self.class.new(args, self.data[true, picked_indices] )
+  end
+
+  def each(&block)
+    (0..length-1).to_a.each do |i|
+      yield data[i,true].transpose.to_a.first
     end
+  end
 
-    def each
-      (0..@data_set.shape[0]-1).each do |i|
-        yield Array(self[i]).first
-      end
+  def dup
+    instance = self.class.new( variables.dup, data.dup)
+    instance.instance_variable_set(:@variable_array, variable_array)
+    instance
+  end
+
+  def merge(other)
+    dup.merge!(other)
+  end
+
+  def merge!(other)
+    other = self.class.from_hash(other) if other.kind_of?(::Hash)
+    raise ArgumentError, "Lengths must be the same" if other.length != length
+
+    # Merge the variables.  Existing variables should be updated,
+    # new variables should be appended to the hash in the same order
+    # as they appear in other
+    @variable_array += (other.variables - variables).to_a
+    @variables += other.variables
+
+    # Create the new data array, should be the size of the merged variables
+    # by the number of vectors
+    new_data = ::NArray.object(length, variables.length)
+
+    # Copy over the data from self (as if we had extended self.data to the
+    # right to allow for the new data)
+    new_data[true, 0..data.shape[1]-1] = data
+
+    # Merge in other's data, using the indices of other's variables as the
+    # slice keys
+    other.variables.each do |variable|
+      new_data[true, variable_array.index(variable)] = other.pick(variable).to_a.flatten
     end
+
+    self.data = new_data
+    self
   end
 end