daru 0.0.3 → 0.0.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 7bee85826c8bd5bb962982278d93ee62b7874d93
-  data.tar.gz: 7e9e66b3282f44888c3d018bfcfe09e3d03ae065
+  metadata.gz: e0388674c6bcbd695b0c456529e987cce5124052
+  data.tar.gz: 4433ab2ef63dd8908e12c43937dbb704abccba43
 SHA512:
-  metadata.gz: de3897032876c4ced80ca9b8ac741e369b09478adbcc0bec1001e38b7c1474e62b2b0d9d2f5d29802830c2540c460b0688b66124a1aac570699424f95c6884c2
-  data.tar.gz: edeb5ae0b7d9523a1fc72ae845e89145b7a462b09c89989f01e614620c27860ea2bb14b020d59d3e9f21851085ff66d4104fe243d4daa57bc0544fcc46b023ba
+  metadata.gz: 37c6930a8a54e567e92efaa68c04c0774a33a7c08b6d90f66b9ee3309a09d05713d265c0dc502dd05ad36658ebd992f50df0994f96f84da5b2bff2576d5d78e7
+  data.tar.gz: 9985f10d9d66615a0fce7a949d03765029bb186ec7f69e91b6397d1a9a51bf2849ede0c64303719918bb097e3cff0f32ba52a610a08a0f92803805dc1a298d88

data/README.md

@@ -7,29 +7,280 @@ Data Analysis in RUby
 
 ## Introduction
 
-daru (Data Analysis in RUby) is a library for storage, analysis and manipulation of data. It aims to be the preferred data analysis library for Ruby. 
+daru (Data Analysis in RUby) is a library for storage, analysis and manipulation of data.
 
-Development of daru was started to address the fragmentation of Dataframe-like classes which were created in many ruby gems as per their own needs. 
-
-This creates a hurdle in using these gems together to solve a problem. For example, calculating something in [statsample](https://github.com/clbustos/statsample) and plotting the results in [Nyaplot](https://github.com/domitry/nyaplot).
+Development of daru was started to address the fragmentation of Dataframe-like classes which were created in many ruby gems as per their own needs. daru offers a uniform interface for all sorts of data analysis and manipulation operations and aims to be compatible with all ruby gems involved in any way with data.
 
 daru is heavily inspired by `Statsample::Dataset`, `Nyaplot::DataFrame` and the super-awesome pandas, a very mature solution in Python.
 
-## Data Structures
+daru works with CRuby (1.9.3+) and JRuby and in a few weeks will be completely compatible with NMatrix and MDArray for fast data manipulation using C or Java structures.
+
+## Features
+
+* Data structures:
+    - Vector - A basic 1-D vector.
+    - DataFrame - A 2-D matrix-like structure which is internally composed of named `Vector` classes.
+* Compatible with IRuby notebook.
+* Indexed and named data structures.
+* Flexible and intuitive API for manipulation and analysis of data.
+
+## Usage
+
+daru has been created with keeping extreme ease of use in mind.
+
+The gem consists of two data structures, Vector and DataFrame. Any data in a serial format is a Vector and a table is a DataFrame.
+
+#### Initialization of DataFrame
+
+A data frame can be initialized from the following sources:
+* Hash of indexed vectors: `{ b: Daru::Vector.new(:b, [11,12,13,14,15], [:two, :one, :four, :five, :three]), a: Daru::Vector.new(:a, [1,2,3,4,5], [:two,:one,:three, :four, :five])}`.
+* Array of hashes: `[{a: 1, b: 11}, {a: 2, b: 12}, {a: 3, b: 13},{a: 4, b: 14}, {a: 5, b: 15}]`.
+* Hash of names and Arrays: `{b: [11,12,13,14,15], a: [1,2,3,4,5]}`
+
+The DataFrame constructor takes 4 arguments: source, vectors, indexes and name in that order. The last 3 are optional while the first is mandatory.
+
+A basic DataFrame can be initialized like this:
+
+```ruby
+
+    df = Daru::DataFrame.new({b: [11,12,13,14,15], a: [1,2,3,4,5]}, vectors: [:a, :b], index: [:one, :two, :three, :four, :five])
+    df
+    # => 
+    # # <Daru::DataFrame:87274040 @name = 7308c587-4073-4e7d-b3ca-3679d1dcc946 # @size = 5>
+    #           a     b 
+    #   one     1    11 
+    #   two     2    12 
+    # three     3    13 
+    #  four     4    14 
+    #  five     5    15 
+
+```
+Daru will automatically align the vectors correctly according to the specified index and then create the DataFrame. Thus, elements having the same index will show up in the same row. The indexes will be arranged alphabetically if vectors with unaligned indexes are supplied.
+
+The vectors of the DataFrame will be arranged according to the array specified in the (optional) second argument. Otherwise the vectors are ordered alphabetically.
+
+```ruby
+
+    df = Daru::DataFrame.new({
+        b: [11,12,13,14,15].dv(:b, [:two, :one, :four, :five, :three]), 
+        a:      [1,2,3,4,5].dv(:a, [:two,:one,:three, :four, :five])
+      }, 
+        vectors: [:a, :b]
+      )
+    df
+
+    # => 
+    # #<Daru::DataFrame:87363700 @name = 75ba0a14-8291-48ac-ac30-35017e4d6c5f # @size = 5>
+    #           a     b 
+    #  five     5    14 
+    #  four     4    13 
+    #   one     2    12 
+    # three     3    15 
+    #   two     1    11
+
+```
+
+If an index for the DataFrame is supplied (third argument), then the indexes of the individual vectors will be matched to the DataFrame index. If any of the indexes do not match, nils will be inserted instead:
+
+```ruby
+
+    df = Daru::DataFrame.new({
+        b: [11]                .dv(nil, [:one]), 
+        a: [1,2,3]             .dv(nil, [:one, :two, :three]), 
+        c: [11,22,33,44,55]    .dv(nil, [:one, :two, :three, :four, :five]),
+        d: [49,69,89,99,108,44].dv(nil, [:one, :two, :three, :four, :five, :six])
+      }, vectors: [:a, :b, :c, :d], index: [:one, :two, :three, :four, :five, :six])
+    df
+    # => 
+    # #<Daru::DataFrame:87523270 @name = bda4eb68-afdd-4404-9981-708edab14201  #@size = 6>
+    #           a     b     c     d 
+    #   one     1    11    11    49 
+    #   two     2   nil    22    69 
+    # three     3   nil    33    89 
+    #  four   nil   nil    44    99 
+    #  five   nil   nil    55   108 
+    #   six   nil   nil   nil    44 
+
+```
+
+If some of the supplied vectors do not contain certain indexes that are contained in other vectors, they are added to those vectors and the correspoding elements are set to `nil`.
+
+```ruby
+
+    df = Daru::DataFrame.new({
+             b: [11,12,13,14,15].dv(:b, [:two, :one, :four, :five, :three]), 
+             a: [1,2,3]         .dv(:a, [:two,:one,:three])
+           }, 
+           vectors: [:a, :b]
+         )
+    df
+
+    #  => 
+    # #<Daru::DataFrame:87612510 @name = 1e904c15-e095-4dce-bfdf-c07ee4d6e4a4 # @size = 5>
+    #           a     b 
+    #  five   nil    14 
+    #  four   nil    13 
+    #   one     2    12 
+    # three     3    15 
+    #   two     1    11 
+
+```
+
+#### Initialization of Vector
+
+The `Vector` data structure is also named and indexed. It accepts arguments name, source, index (in that order).
+
+In the simplest case it can be constructed like this:
+
+```ruby
+
+    dv = Daru::Vector.new [1,2,3,4,5], name: ravan, index: [:ek, :don, :teen, :char, :pach]
+    dv
+
+    #  => 
+    # #<Daru::Vector:87630270 @name = ravan @size = 5 >
+    #     ravan
+    #   ek    1
+    #  don    2
+    # teen    3
+    # char    4
+    # pach    5 
+
+```
+
+Initializing a vector with indexes will insert nils in places where elements dont exist:
+
+```ruby
+
+    dv = Daru::Vector.new [1,2,3], name: yoga, index: [0,1,2,3,4]
+    dv
+    #  => 
+    # #<Daru::Vector:87890840 @name = yoga @size = 5 >
+    #   y
+    # 0 1
+    # 1 2
+    # 2 3
+    # 3 nil 
+    # 4 nil 
+
+
+```
+
+#### Basic Selection Operations
+
+Initialize a dataframe:
+
+```ruby
+
+    df = Daru::DataFrame.new({
+        b: [11,12,13,14,15].dv(:b, [:two, :one, :four, :five, :three]), 
+        a:      [1,2,3,4,5].dv(:a, [:two,:one,:three, :four, :five])
+      }, 
+        vectors: [:a, :b]
+      )
+
+    #  => 
+    # #<Daru::DataFrame:87455010 @name = b3d14e23-98c2-4741-a563-92e8f1fd0f13 # @size = 5>
+    #           a     b 
+    #  five     5    14 
+    #  four     4    13 
+    #   one     2    12 
+    # three     3    15 
+    #   two     1    11 
+
+```
+Select a row from a DataFrame:
+
+```ruby
+    
+    df.row[:one]
+
+    #  => 
+    # #<Daru::Vector:87432070 @name = one @size = 2 >
+    #    one
+    #  a  2
+    #  b 12 
+```
+A row or a vector is returned as a `Daru::Vector` object, so any manipulations supported by `Daru::Vector` can be performed on the chosen row as well.
+
+Select a single vector:
+
+```ruby
+    
+    df.vector[:a] # or simply df.a
+
+    #  => 
+    # #<Daru::Vector:87454270 @name = a @size = 5 >
+    #           a
+    #  five     5
+    #  four     4
+    #   one     2
+    # three     3
+    #   two     1
+
+```
+
+Select multiple vectors and return a DataFrame in the specified order:
+
+```ruby
+
+    df.vector[:b, :a]
+    #  =>
+    # #<Daru::DataFrame:87835960 @name = e80902cc-cff9-4b23-9eca-5da36ebc88a8 #   @size = 5>
+    #           b     a 
+    #  five    14     5 
+    #  four    13     4 
+    #   one    12     2 
+    # three    15     3 
+    #   two    11     1 
+
+```
+
+Keep/remove row according to a specified condition:
+
+```ruby
+
+    df = df.filter_rows do |row|
+        row[:a] == 5
+    end
+
+    df
+    #  => 
+    # #<Daru::DataFrame:87455010 @name = b3d14e23-98c2-4741-a563-92e8f1fd0f13 # @size = 1>
+    #         a    b 
+    # five    5   14 
+
+```
+The same can be applied to vectors using `keep_vector_if`.
+
+To iterate over a DataFrame and perform operations on rows or vectors, `#each_row` or `#each_vector` can be used, which works just like `#each` for Ruby Arrays.
+
+To change the values of a row/vector while iterating through the DataFrame, use `map_rows` or `map_vectors`:
+
+```ruby
+
+    df.map_rows do |row|
+        row = row.map { |e| e*e }
+    end
 
-daru employs several data structures for storing and manipulating data:
-* Vector - A basic 1-D vector.
-* DataFrame - A 2-D matrix-like structure which is internally composed of named `Vector` classes.
+    df
 
-daru data structures can be constructed by using several Ruby classes. These include `Array`, `Hash`, `Matrix`, [NMatrix](https://github.com/SciRuby/nmatrix) and [MDArray](https://github.com/rbotafogo/mdarray). daru brings a uniform API for handling and manipulating data represented in any of the above Ruby classes.
+    #  => 
+    # #<Daru::DataFrame:86826830 @name = b092ca5b-7b83-4dbe-a469-124f7f25a568 # @size = 5>
+    #           a     b 
+    #  five    25   196 
+    #  four    16   169 
+    #   one     4   144 
+    # three     9   225 
+    #   two     1   121 
 
-Currently things work as expected for Arrays only. Rest will added over the next few weeks.
+```
 
-## Testing
+Rows/vectors can be deleted using `delete_row` or `delete_vector`.
 
-Install jruby using `rvm install jruby`, then run `jruby -S gem install mdarray`, followed by `bundle install`. You will need to install `mdarray` manually because of strange gemspec file behaviour. If anyone can automate this then I'd greatly appreciate it! Then run `rspec` in JRuby to test for MDArray functionality.
+#### Basic Math Operations
 
-Then switch to MRI, do a normal `bundle install` followed by `rspec` for testing everything else with NMatrix functionality.
+Coming soon!
 
 ## Roadmap
 
@@ -56,6 +307,11 @@ Then switch to MRI, do a normal `bundle install` followed by `rspec` for testing
 * Deletion of elements from Vector should only modify the index and leave the vector as it is so that compacting is not needed and things are faster.
 * Add a #sync method which will sync the modified index with the unmodified vector.
 * Ability to reorder the index of a dataframe.
+* Slicing operations using Range.
+* Create DataFrame by providing rows.
+* Integrate basic plotting with Nyaplot.
+* Filter through a dataframe with filter\_rows or filter\_vectors based on whatever boolean value evaluates to true.
+* Named arguments
 
 Copyright (c) 2014, Sameer Deshmukh
 All rights reserved

data/lib/daru/accessors/array_wrapper.rb

@@ -0,0 +1,8 @@
+module Daru
+  module Accessors
+    # Internal class for wrapping ruby array
+    class ArrayWrapper
+
+    end
+  end
+end

data/lib/daru/accessors/dataframe_by_row.rb

@@ -0,0 +1,17 @@
+module Daru
+  module Accessors
+    class DataFrameByRow
+      def initialize data_frame
+        @data_frame = data_frame
+      end
+
+      def [](*names)
+        @data_frame[*names, :row]
+      end
+
+      def []=(name, vector)
+        @data_frame[name, :row] = vector
+      end
+    end
+  end
+end

data/lib/daru/accessors/dataframe_by_vector.rb

@@ -0,0 +1,17 @@
+module Daru
+  module Accessors
+    class DataFrameByVector
+      def initialize data_frame
+        @data_frame = data_frame
+      end
+
+      def [](*names)
+        @data_frame[*names, :vector]
+      end
+
+      def []=(name, vectors)
+        @data_frame[name, :vector] = vectors
+      end
+    end
+  end
+end

data/lib/daru/accessors/mdarray_wrapper.rb

@@ -0,0 +1,9 @@
+module Daru
+  module Accessors
+
+    # Internal class for wrapping MDArray
+    class MDArrayWrapper
+
+    end
+  end
+end

data/lib/daru/accessors/nmatrix_wrapper.rb

@@ -0,0 +1,9 @@
+module Daru
+  module Accessors
+
+    # Internal class for wrapping NMatrix
+    class NMatrixWrapper
+
+    end
+  end
+end

data/lib/daru/dataframe.rb

@@ -1,10 +1,15 @@
-require_relative 'dataframe_by_row.rb'
-require_relative 'dataframe_by_vector.rb'
-require_relative 'io.rb'
+require_relative 'accessors/dataframe_by_row.rb'
+require_relative 'accessors/dataframe_by_vector.rb'
+require_relative 'math/arithmetic/dataframe.rb'
+require_relative 'math/statistics/dataframe.rb'
+require_relative 'io/io.rb'
 
 module Daru
   class DataFrame
 
+    include Daru::Math::Arithmetic::DataFrame
+    include Daru::Math::Statistics::DataFrame
+
     class << self
       def from_csv path, opts={}, &block
         Daru::IO.from_csv path, opts, &block      
@@ -19,10 +24,10 @@ module Daru
     # DataFrame basically consists of an Array of Vector objects.
     # These objects are indexed by row and column by vectors and index Index objects.
     # Arguments - source, vectors, index, name in that order. Last 3 are optional.
-    def initialize source, *args
-      vectors = args.shift
-      index   = args.shift
-      @name   = args.shift || SecureRandom.uuid
+    def initialize source, opts={}
+      vectors = opts[:vectors]
+      index   = opts[:index]
+      @name   = (opts[:name] || SecureRandom.uuid).to_sym
 
       @data = []
 
@@ -77,7 +82,7 @@ module Daru
             end
 
             @vectors.each do |vector|
-              @data << Daru::Vector.new(vector, [], @index)
+              @data << Daru::Vector.new([], name: vector, index: @index)
 
               @index.each do |idx|
                 begin
@@ -132,11 +137,11 @@ module Daru
     end
 
     def vector
-      Daru::DataFrameByVector.new(self)
+      Daru::Accessors::DataFrameByVector.new(self)
     end
 
     def row
-      Daru::DataFrameByRow.new(self)
+      Daru::Accessors::DataFrameByRow.new(self)
     end
 
     def dup
@@ -145,7 +150,7 @@ module Daru
         src[vector] = @data[@vectors[vector]]
       end
 
-      Daru::DataFrame.new src, @vectors.dup, @index.dup, @name
+      Daru::DataFrame.new src, vectors: @vectors.dup, index: @index.dup, name: @name
     end
 
     def each_vector(&block)
@@ -244,11 +249,17 @@ module Daru
     end
 
     def keep_row_if &block
+      deletion = []
+
       @index.each do |index|
         keep_row = yield access_row(index)
 
-        delete_row index unless keep_row
+        deletion << index unless keep_row
       end
+
+      deletion.each { |idx| 
+        delete_row idx 
+      }
     end
 
     def keep_vector_if &block
@@ -259,6 +270,31 @@ module Daru
       end
     end
 
+    def filter_rows &block
+      df = Daru::DataFrame.new({}, vectors: @vectors.to_a)
+      marked = []
+
+      @index.each do |index|
+        keep_row = yield access_row(index)
+
+        marked << index if keep_row
+      end
+
+      marked.each do |idx|
+        df.row[idx] = self[idx, :row]
+      end
+
+      df
+    end
+
+    def filter_vectors &block
+      df = self.dup
+
+      df.keep_vector_if &block
+
+      df
+    end
+
     def has_vector? name
       !!@vectors[name]
     end
@@ -323,7 +359,8 @@ module Daru
     end
 
     def inspect spacing=10, threshold=15
-      longest = [@vectors.map(&:to_s).map(&:size).max, 
+      longest = [@name.to_s.size,
+                 @vectors.map(&:to_s).map(&:size).max, 
                  @index  .map(&:to_s).map(&:size).max,
                  @data   .map{ |v|  v.map(&:to_s).map(&:size).max }.max].max
 
@@ -395,16 +432,14 @@ module Daru
         new_vcs[name] = @data[@vectors[name]]
       end
 
-      Daru::DataFrame.new new_vcs, new_vcs.keys, @index, @name
+      Daru::DataFrame.new new_vcs, vectors: new_vcs.keys, index: @index, name: @name
     end
 
     def access_row *names
       unless names[1]
         row = []
 
-        @vectors.each do |vector|
-          row << @data[@vectors[vector]][names[0]]
-        end
+        name = nil
 
         if @index.include? names[0]
           name = names[0]
@@ -414,7 +449,11 @@ module Daru
           raise IndexError, "Specified row #{names[0]} does not exist."
         end
 
-        Daru::Vector.new name, row, @vectors
+        @vectors.each do |vector|
+          row << @data[@vectors[vector]][name]
+        end
+
+        Daru::Vector.new row, index: @vectors, name: name
       else
         # TODO: Access multiple rows
       end
@@ -426,7 +465,7 @@ module Daru
       v = nil
 
       if vector.is_a?(Daru::Vector)
-        v = Daru::Vector.new name, [], @index
+        v = Daru::Vector.new [], name: name, index: @index
 
         @index.each do |idx|
           begin
@@ -474,7 +513,7 @@ module Daru
 
     def create_empty_vectors
       @vectors.each do |name|
-        @data << Daru::Vector.new(name, [], @index)
+        @data << Daru::Vector.new([],name: name, index: @index)
       end
     end