daru 0.0.2.3 → 0.0.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: e425d7fab01db79087549701e3e9e95df2b495d1
-  data.tar.gz: f3b186c9405b1fb14fb7b95a2ecde7a291decbb4
+  metadata.gz: 7bee85826c8bd5bb962982278d93ee62b7874d93
+  data.tar.gz: 7e9e66b3282f44888c3d018bfcfe09e3d03ae065
 SHA512:
-  metadata.gz: fd89ad49623169a8caf335746094a38e2613864ec05f059488773285deaff87eaa5a8a68b6f60a2a12e37f5ad29cc800e8cdb7b0fba12eeb3a21b9bb3dbcca69
-  data.tar.gz: 0b1458e1df11590ca858e9350503eef9a29420661ea92d1c73907ee47c33879637260f489bd3451aa9d7774d69e20b639e17ecdfac339c18f710c6ba25955908
+  metadata.gz: de3897032876c4ced80ca9b8ac741e369b09478adbcc0bec1001e38b7c1474e62b2b0d9d2f5d29802830c2540c460b0688b66124a1aac570699424f95c6884c2
+  data.tar.gz: edeb5ae0b7d9523a1fc72ae845e89145b7a462b09c89989f01e614620c27860ea2bb14b020d59d3e9f21851085ff66d4104fe243d4daa57bc0544fcc46b023ba

data/History.txt

@@ -15,3 +15,13 @@
 * Vector objects passed into a DataFrame are now duplicated so that any changes dont affect the original vector.
 * Added an optional opts argument to DataFrame.
 * Sending more fields than vectors in DataFrame will cause addition of nil vectors.
+* Init a DataFrame without having to convert explicitly to vectors.
+
+== 0.0.2.4
+* Initialize dataframe from an array which looks like [{a: 10, b: 20}, {a: 11, b: 12}]. Works for parsed JSON.
+* Over-riding vectors in DataFrame will still preserve order.
+* Any re-assignment of rows in #each_row and #each_row_with_index will reflect in the DataFrame.
+* Added #to_a and #to_json to DataFrame.
+
+== 0.0.3
+* This release is a complete rewrite of the entire gem to accomodate index values.

data/README.md

@@ -23,6 +23,8 @@ daru employs several data structures for storing and manipulating data:
 
 daru data structures can be constructed by using several Ruby classes. These include `Array`, `Hash`, `Matrix`, [NMatrix](https://github.com/SciRuby/nmatrix) and [MDArray](https://github.com/rbotafogo/mdarray). daru brings a uniform API for handling and manipulating data represented in any of the above Ruby classes.
 
+Currently things work as expected for Arrays only. Rest will added over the next few weeks.
+
 ## Testing
 
 Install jruby using `rvm install jruby`, then run `jruby -S gem install mdarray`, followed by `bundle install`. You will need to install `mdarray` manually because of strange gemspec file behaviour. If anyone can automate this then I'd greatly appreciate it! Then run `rspec` in JRuby to test for MDArray functionality.
@@ -33,9 +35,7 @@ Then switch to MRI, do a normal `bundle install` followed by `rspec` for testing
 
 * Automate testing for both MRI and JRuby.
 * Enable creation of DataFrame by only specifying an NMatrix/MDArray in initialize. Vector naming happens automatically (alphabetic) or is specified in an Array.
-* Add support for missing values in vectors.
-* Destructive version #filter\_rows!
-* NMatrix.first should return NMatrix (in vector).
+* Destructive map iterators for DataFrame and Vector.
 * Completely test all functionality for NMatrix and MDArray.
 * Basic Data manipulation and analysis operations: 
     - Different kinds of join operations
@@ -43,12 +43,19 @@ Then switch to MRI, do a normal `bundle install` followed by `rspec` for testing
     - Creation of correlation, covariance matrices
     - Verification of data in a vector
     - Basic vector statistics - mean, median, variance, etc.
-* Add indexing on vectors.
-    - Creation of vector by supplying an index-value hash.
-    - Auto generation of real numbered indices for any vector.
-    - Ability to separately specify index for each element of a vector.
-    - Runtime alteration of index.
-* Indexing on DataFrame.
 * Vector arithmetic - elementwise addition, subtraction, multiplication, division.
 * Transpose a dataframe.
-* Option to express a DataFrame as an NMatrix or MDArray so as to use more efficient storage techniques.
+* Option to express a DataFrame as an NMatrix or MDArray so as to use more efficient storage techniques.
+* Assignment of a column to a single number should set the entire column to that number.
+* == between daru_vector and string/number.
+* Multiple column assignment with []=
+* Creation of DataFrame from Array of Arrays.
+* Multiple value assignment for vectors with []=.
+* Load DataFrame from multiple sources (excel, SQL, etc.).
+* Allow for boolean operations inside #[].
+* Deletion of elements from Vector should only modify the index and leave the vector as it is so that compacting is not needed and things are faster.
+* Add a #sync method which will sync the modified index with the unmodified vector.
+* Ability to reorder the index of a dataframe.
+
+Copyright (c) 2014, Sameer Deshmukh
+All rights reserved

data/Rakefile

@@ -0,0 +1,5 @@
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec

data/daru.gemspec

@@ -24,7 +24,9 @@ Gem::Specification.new do |spec|
   spec.require_paths = ["lib"]
 
   spec.add_development_dependency 'bundler'
+  spec.add_development_dependency 'rake'
   spec.add_development_dependency 'rspec'
+  spec.add_development_dependency 'awesome_print'
   if RUBY_ENGINE != 'jruby'
     spec.add_development_dependency 'nmatrix', '~> 0.1.0.rc5'
   end

data/lib/daru.rb

@@ -1,7 +1,7 @@
 require 'securerandom'
-require 'matrix'
 require 'csv'
 
+require 'daru/index.rb'
 require 'daru/vector.rb'
 require 'daru/dataframe.rb'
 require 'daru/monkeys.rb'

data/lib/daru/dataframe.rb

@@ -1,259 +1,539 @@
+require_relative 'dataframe_by_row.rb'
+require_relative 'dataframe_by_vector.rb'
+require_relative 'io.rb'
+
 module Daru
   class DataFrame
 
-    attr_reader :vectors
-
-    attr_reader :fields
+    class << self
+      def from_csv path, opts={}, &block
+        Daru::IO.from_csv path, opts, &block      
+      end
+    end
 
+    attr_reader :vectors
+    attr_reader :index
+    attr_reader :name
     attr_reader :size
 
-    attr_reader :name
+    # DataFrame basically consists of an Array of Vector objects.
+    # These objects are indexed by row and column by vectors and index Index objects.
+    # Arguments - source, vectors, index, name in that order. Last 3 are optional.
+    def initialize source, *args
+      vectors = args.shift
+      index   = args.shift
+      @name   = args.shift || SecureRandom.uuid
 
-    def initialize source, fields=[], name=SecureRandom.uuid, opts={}
-      @opts = opts
-      set_default_opts
+      @data = []
 
       if source.empty?
-        @vectors = fields.inject({}){ |a,x| a[x]=Daru::Vector.new; a}
+        @vectors = Daru::Index.new vectors
+        @index   = Daru::Index.new index
+
+        create_empty_vectors
       else
-        @vectors = source.inject({}) do |acc, h|
-          acc[h[0]] = h[1].dv.dup
-          acc
+        case source
+        when Array
+          if vectors.nil?
+            @vectors = Daru::Index.new source[0].keys.map(&:to_sym)
+          else
+            @vectors = Daru::Index.new (vectors + (source[0].keys - vectors)).uniq.map(&:to_sym)
+          end
+
+          if index.nil?
+            @index = Daru::Index.new source.size
+          else
+            @index = Daru::Index.new index
+          end
+
+          @vectors.each do |name|
+            v = []
+
+            source.each do |hsh|
+              v << (hsh[name] || hsh[name.to_s])
+            end
+
+            @data << v.dv(name, @index)
+          end
+        when Hash
+          create_vectors_index_with vectors, source
+
+          if all_daru_vectors_in_source? source
+
+            if !index.nil?
+              @index = index.to_index
+            elsif all_vectors_have_equal_indexes? source
+              @index = source.values[0].index.dup
+            else
+              all_indexes = []
+
+              source.each_value do |vector|
+                all_indexes << vector.index.to_a
+              end
+              # sort only if missing indexes detected
+              all_indexes.flatten!.uniq!.sort!
+
+              @index = Daru::Index.new all_indexes
+            end
+
+            @vectors.each do |vector|
+              @data << Daru::Vector.new(vector, [], @index)
+
+              @index.each do |idx|
+                begin
+                  @data[@vectors[vector]][idx] = source[vector][idx]                   
+                rescue IndexError
+                  # If the index is not present in the vector under consideration
+                  # (in source) then an error is raised. Put a nil in that place if
+                  # that is the case.
+                  @data[@vectors[vector]][idx] = nil                  
+                end
+              end
+            end
+          else   
+            index = source.values[0].size if index.nil?
+
+            if index.is_a?(Daru::Index)
+              @index   = index.to_index
+            else
+              @index   = Daru::Index.new index     
+            end
+
+            @vectors.each do |name|
+              @data << source[name].dup.dv(name, @index)
+            end
+          end
+
         end
       end
 
-      @fields = fields.empty? ? source.keys.sort : fields
-      @name   = name
-
-      check_length
-      set_missing_vectors if @vectors.keys.size < @fields.size
-      set_fields_order    if @vectors.keys.sort != @fields.sort
-      set_vector_names
+      set_size
+      validate
     end
 
-    def self.from_csv file, opts={}
-      opts[:col_sep]           ||= ','
-      opts[:headers]           ||= true
-      opts[:converters]        ||= :numeric
-      opts[:header_converters] ||= :symbol
-
-      csv = CSV.open file, 'r', opts
+    def [](*names, axis)
+      if axis == :vector
+        access_vector *names
+      elsif axis == :row
+        access_row *names
+      else
+        raise IndexError, "Expected axis to be row or vector not #{axis}"
+      end
+    end
 
-      yield csv if block_given?
+    def []=(name, axis ,vector)
+      if axis == :vector
+        insert_or_modify_vector name, vector
+      elsif axis == :row        
+        insert_or_modify_row name, vector
+      else
+        raise IndexError, "Expected axis to be row or vector, not #{axis}."
+      end
+    end
 
-      first = true
-      df    = nil
+    def vector
+      Daru::DataFrameByVector.new(self)
+    end
 
-      csv.each do |row|
-        if first
-          df = Daru::DataFrame.new({}, csv.headers)
-          first = false
-        end
+    def row
+      Daru::DataFrameByRow.new(self)
+    end
 
-        df.insert_row row
+    def dup
+      src = {}
+      @vectors.each do |vector|
+        src[vector] = @data[@vectors[vector]]
       end
 
-      df
+      Daru::DataFrame.new src, @vectors.dup, @index.dup, @name
     end
 
-    def column name
-      @vectors[name]
-    end
+    def each_vector(&block)
+      @data.each(&block)
 
-    def delete_vector name
-      @vectors.delete name
-      @fields.delete name
+      self
     end
 
-    alias_method :delete, :delete_vector
+    def each_vector_with_index(&block)
+      @vectors.each do |vector|
+        yield @data[@vectors[vector]], vector
+      end 
 
-    def delete_row index
-      # TODO: Make this work with NMatrix and MDArray
-      raise "Expected index less than size." if index > @size
+      self
+    end
 
-      @fields.each do |field|
-        @vectors[field].delete index
+    def each_row(&block)
+      @index.each do |index|
+        yield access_row(index)
       end
-      puts @vectors
+
+      self
     end
 
-    def filter_rows name=self.name, &block
-      df = DataFrame.new({}, @fields, name)
+    def each_row_with_index(&block)
+      @index.each do |index|
+        yield access_row(index), index
+      end
 
-      self.each_row do |row|
-        keep_row = yield row
+      self
+    end
 
-        df.insert_row(row.values) if keep_row
+    def map_vectors(&block)
+      df = self.dup
+
+      df.each_vector_with_index do |vector, name|
+        df[name, :vector] = yield(vector)
       end
 
       df
     end
 
-    def [] *name
-      unless name[1]
-        return column(name[0])
-      end
-
-      h = {}
-      req_fields = @fields & name
+    def map_vectors_with_index(&block)
+      df = self.dup
 
-      req_fields.each do |f|
-        h[f] = @vectors[f]
+      df.each_vector_with_index do |vector, name|
+        df[name, :vector] = yield(vector, name)
       end
 
-      DataFrame.new h, req_fields, @name
+      df
     end
 
-    def == other
-      @name == other.name and @vectors == other.vectors and 
-      @size == other.size and @fields  == other.fields 
-    end
+    def map_rows(&block)
+      df = self.dup
+
+      df.each_row_with_index do |row, index|
+        df[index, :row] = yield(row)
+      end
 
-    def []= name, vector
-      insert_vector name, vector
+      df
     end
 
-    def row index
-      raise Exception, "Expected index to be within bounds" if index > @size
+    def map_rows_with_index(&block)
+      df = self.dup
 
-      row = {}
-      self.each_vector do |column|
-        row[column.name] = column[index]
+      df.each_row_with_index do |row, index|
+        df[index, :row] = yield(row, index)
       end
 
-      row
+      df
     end
 
-    def has_vector? vector
-      !!@vectors[vector]
+    def delete_vector vector
+      if @vectors.include? vector
+        @data.delete_at @vectors[vector]
+        @vectors = Daru::Index.new @vectors.to_a - [vector]
+      else
+        raise IndexError, "Vector #{vector} does not exist."
+      end
     end
 
-    def each_row(&block)
-      0.upto(@size-1) do |index|
-        yield row(index)
-      end
+    def delete_row index
+      idx = named_index_for index
 
-      self
-    end
+      if @index.include? idx
+        @index = (@index.to_a - [idx]).to_index
 
-    def each_row_with_index(&block)
-      0.upto(@size-1) do |index|
-        yield row(index), index
+        self.each_vector do |vector|
+          vector.delete_at idx
+        end
+      else
+        raise IndexError, "Index #{index} does not exist."
       end
 
-      self
+      set_size
     end
 
-    def each_vector(&block)
-      @fields.each do |field|
-        yield @vectors[field]
-      end
+    def keep_row_if &block
+      @index.each do |index|
+        keep_row = yield access_row(index)
 
-      self
+        delete_row index unless keep_row
+      end
     end
 
-    def each_vector_with_name(&block)
-      @fields.each do |field|
-        yield @vectors[field], field
+    def keep_vector_if &block
+      @vectors.each do |vector|
+        keep_vector = yield @data[@vectors[vector]], vector
+        
+        delete_vector vector unless keep_vector
       end
-
-      self
     end
 
-    def insert_vector name, vector
-      raise Exeception, "Expected vector size to be same as DataFrame\ 
-        size." if vector.size != self.size
-
-      @vectors.merge({name => vector})
-      @fields << name
+    def has_vector? name
+      !!@vectors[name]
     end
 
-    def insert_row row
-      raise Exception, "Expected new row to same as the number of rows \ 
-        in the DataFrame" if row.size != @fields.size
+    # Converts the DataFrame into an array of hashes where key is vector name
+    # and value is the corresponding element.
+    # The 0th index of the array contains the array of hashes while the 1th
+    # index contains the indexes of each row of the dataframe. Each element in
+    # the index array corresponds to its row in the array of hashes, which has
+    # the same index.
+    def to_a
+      arry = [[],[]]
 
-      @fields.each_with_index do |field, index|
-        @vectors[field] << row[index]
+      self.each_row do |row|
+        arry[0] << row.to_hash
       end
 
-      @size += 1
+      arry[1] = @index.to_a
+
+      arry
     end
 
-    def to_html(threshold=15)
-      html = '<table>'
+    def to_json no_index=true
+      if no_index
+        self.to_a[0].to_json
+      else
+        self.to_a.to_json
+      end
+    end
+
+    def to_html threshold=15
+      html = '<table><tr><th></th>'
+
+      @vectors.each { |vector| html += '<th>' + vector.to_s + '</th>' }
 
-      html += '<tr>'
-      @fields.each { |f| html.concat('<td>' + f.to_s + '</td>') }
       html += '</tr>'
 
-      self.each_row_with_index do |row, index|
-        break if index > threshold and index <= @size
+      @index.each_with_index do |index, num|
         html += '<tr>'
-        row.each_value { |val| html.concat('<td>' + val.to_s + '</td>') }
+        html += '<td>' + index.to_s + '</td>'
+
+        self.row[index].each do |element|
+          html += '<td>' + element.to_s + '</td>'
+        end
         html += '</tr>'
-        if index == threshold
+
+        if num > threshold
           html += '<tr>'
-          row.size.times { html.concat('<td>...</td>') }
+          (@vectors + 1).size.times { html += '<td>...</td>' }
           html += '</tr>'
+          break
         end
       end
 
       html += '</table>'
+
+      html
     end
 
     def to_s
       to_html
     end
 
-    def method_missing(name, *args)
+    def inspect spacing=10, threshold=15
+      longest = [@vectors.map(&:to_s).map(&:size).max, 
+                 @index  .map(&:to_s).map(&:size).max,
+                 @data   .map{ |v|  v.map(&:to_s).map(&:size).max }.max].max
+
+      name      = @name || 'nil'
+      content   = ""
+      longest   = spacing if longest > spacing
+      formatter = "\n"
+
+      (@vectors.size + 1).times { formatter += "%#{longest}.#{longest}s " }
+
+      content += "\n#<" + self.class.to_s + ":" + self.object_id.to_s + " @name = " + 
+                    name.to_s + " @size = " + @size.to_s + ">"
+
+      content += sprintf formatter, "" , *@vectors.map(&:to_s)
+
+      row_num = 1
+
+      self.each_row_with_index do |row, index|
+        content += sprintf formatter, index.to_s, *row.to_hash.values.map { |e| (e || 'nil').to_s }
+
+        row_num += 1
+        if row_num > threshold
+          dots = []
+
+          (@vectors.size + 1).times { dots << "..." }
+          content += sprint formatter, *dots
+          break
+        end
+      end
+
+      content += "\n"
+
+      content
+    end
+
+    def == other
+      @index == other.index and @size == other.size and @vectors.all? { |vector|
+                            self[vector, :vector] == other[vector, :vector] }
+    end
+
+    def method_missing(name, *args, &block)
       if md = name.match(/(.+)\=/)
-        insert_vector name[/(.+)\=/].delete("="), args[0]
+        insert_or_modify_vector name[/(.+)\=/].delete("="), args[0]
       elsif self.has_vector? name
-        column name
+        self[name, :vector]
       else
         super(name, *args)
       end
     end
 
    private
-    def check_length
-      size = nil
-
-      @vectors.each_value do |vector|
-        if size.nil?
-          size = vector.size
-        elsif size != vector.size
-          raise Exception, "Expected all vectors to be of the same size. Vector \ 
-            #{vector.name} is of size #{vector.size} and another one of size #{size}"    
+
+    def access_vector *names
+      unless names[1]
+        if @vectors.include? names[0]
+          return @data[@vectors[names[0]]]
+        elsif @vectors.key names[0]
+          return @data[names[0]]
+        else
+          raise IndexError, "Specified index #{names[0]} does not exist."
+        end
+      end
+
+      new_vcs = {}
+
+      names.each do |name|
+        name = name.to_sym unless name.is_a?(Integer)
+
+        new_vcs[name] = @data[@vectors[name]]
+      end
+
+      Daru::DataFrame.new new_vcs, new_vcs.keys, @index, @name
+    end
+
+    def access_row *names
+      unless names[1]
+        row = []
+
+        @vectors.each do |vector|
+          row << @data[@vectors[vector]][names[0]]
+        end
+
+        if @index.include? names[0]
+          name = names[0]
+        elsif @index.key names[0]
+          name = @index.key names[0]
+        else
+          raise IndexError, "Specified row #{names[0]} does not exist."
+        end
+
+        Daru::Vector.new name, row, @vectors
+      else
+        # TODO: Access multiple rows
+      end
+    end
+
+    def insert_or_modify_vector name, vector
+      @vectors = @vectors.re_index(@vectors + name)
+
+      v = nil
+
+      if vector.is_a?(Daru::Vector)
+        v = Daru::Vector.new name, [], @index
+
+        @index.each do |idx|
+          begin
+            v[idx] = vector[idx]
+          rescue IndexError
+            v[idx] = nil
+          end
+        end
+      else
+        raise Exception, "Specified vector of length #{vector.size} cannot be inserted in DataFrame of size #{@size}" if
+          @size != vector.size
+
+        v = vector.dv(name, @index)
+      end
+
+      @data[@vectors[name]] = v
+    end
+
+    def insert_or_modify_row name, vector      
+      if @index.include? name
+        v = vector.dv(name, @vectors)
+
+        @vectors.each do |vector|
+          begin
+            @data[@vectors[vector]][name] = v[vector] 
+          rescue IndexError
+            @data[@vectors[vector]][name] = nil
+          end
+        end
+      else
+        @index = @index.re_index(@index + name)
+        v      = vector.dv(name, @vectors)
+
+        @vectors.each do |vector|
+          begin
+            @data[@vectors[vector]].concat v[vector], name     
+          rescue IndexError
+            @data[@vectors[vector]].concat nil, name
+          end
         end
       end
 
-      @size = size
+      set_size
     end
 
-    def set_fields_order # vectors more than specified fields
-      @fields = @fields & @vectors.keys
-      @fields += @vectors.keys.sort - @fields
+    def create_empty_vectors
+      @vectors.each do |name|
+        @data << Daru::Vector.new(name, [], @index)
+      end
+    end
+
+    def validate_labels
+      raise IndexError, "Expected equal number of vectors for number of Hash pairs" if 
+        @vectors.size != @data.size
+
+      raise IndexError, "Expected number of indexes same as number of rows" if
+        @index.size != @data[0].size
+    end
+
+    def validate_vector_sizes
+      @data.each do |vector|
+        raise IndexError, "Expected vectors with equal length" if vector.size != @size
+      end
+    end
+
+    def validate
+      # TODO: [IMP] when vectors of different dimensions are specified, they should
+      # be inserted into the dataframe by inserting nils wherever necessary.
+      validate_labels
+      validate_vector_sizes
     end
 
-    # Writes names specified in the hash to the actual name of the vector.
-    # Will over-ride any previous name assigned to the vector.
-    def set_vector_names      
-      @fields.each do |name|
-        @vectors[name].name = name
+    def all_daru_vectors_in_source? source
+      source.values.all? do |vector|
+        vector.is_a?(Daru::Vector)
       end
     end
 
-    def set_default_opts
-      # Future proofing
+    def set_size
+      @size = @index.size
+    end
+
+    def named_index_for index
+      if @index.include? index
+        index
+      elsif @index.key index
+        @index.key index
+      else
+        raise IndexError, "Specified index #{index} does not exist."
+      end
+    end
+
+    def create_vectors_index_with vectors, source
+      vectors = source.keys.sort if vectors.nil?
+
+      if vectors.is_a?(Daru::Index)
+        @vectors = vectors.to_index
+      else
+        @vectors = Daru::Index.new (vectors + (source.keys - vectors)).uniq.map(&:to_sym)
+      end
     end
 
-    def set_missing_vectors
-      missing_fields = @fields - @vectors.keys
+    def all_vectors_have_equal_indexes? source
+      index = source.values[0].index
 
-      missing_fields.each do |field|
-        @vectors[field] = ([nil]*@size).dv
-        @fields << field
+      source.all? do |name, vector|
+        index == vector.index
       end
     end
   end