daru 0.0.3.1 → 0.0.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: e0388674c6bcbd695b0c456529e987cce5124052
-  data.tar.gz: 4433ab2ef63dd8908e12c43937dbb704abccba43
+  metadata.gz: dc60c1a59b4c112dface4bdd6d0a40cf1bcc499c
+  data.tar.gz: aec8fd874e2e74ded4eeb2cebe8503b5a3cbefbc
 SHA512:
-  metadata.gz: 37c6930a8a54e567e92efaa68c04c0774a33a7c08b6d90f66b9ee3309a09d05713d265c0dc502dd05ad36658ebd992f50df0994f96f84da5b2bff2576d5d78e7
-  data.tar.gz: 9985f10d9d66615a0fce7a949d03765029bb186ec7f69e91b6397d1a9a51bf2849ede0c64303719918bb097e3cff0f32ba52a610a08a0f92803805dc1a298d88
+  metadata.gz: 091f1ce3e0da8b9083dc6f53a9977550def5d10a4f759797ec56cb6b90788aa8d6a7fb8ad1a0e4105c640b5987e0b200877a12441cc8cd6b2614f26141554d9e
+  data.tar.gz: d189373c2196a3ddc10856f4800fee908d9361f83921e68078c971402a3996b02d82c13bb275a26d070518e90626d4e654556d970d91b69179f2899065b31685

data/History.txt

@@ -25,3 +25,19 @@
 
 == 0.0.3
 * This release is a complete rewrite of the entire gem to accomodate index values.
+
+== 0.0.3.1
+* Added aritmetic methods for vector aritmetic by taking the index of values into account.
+
+== 0.0.4
+* Added wrappers for Array, NMatrix and MDArray such that the external implementation is completely transparent of the data type being used internally.
+* Added statistics methods for vectors for ArrayWrapper. These are compatible with statsample methods.
+* Added plotting functions for DataFrame and Vector using Nyaplot.
+* Create a DataFrame by specifying the rows with the ".rows" class method.
+* Create a Vector from a Hash.
+* Call a Vector element by specfying the index name as a method call (method_missing logic).
+* Retrive multiple rows of a DataFrame by specfying a Range or an Array with multiple index names.
+* #head and #tail for DataFrame.
+* #uniq for Vector.
+* #max for Vector can return a Vector object with the index set to the index of the max value.
+* Tonnes of documentation for most methods.

data/README.md

@@ -11,9 +11,9 @@ daru (Data Analysis in RUby) is a library for storage, analysis and manipulation
 
 Development of daru was started to address the fragmentation of Dataframe-like classes which were created in many ruby gems as per their own needs. daru offers a uniform interface for all sorts of data analysis and manipulation operations and aims to be compatible with all ruby gems involved in any way with data.
 
-daru is heavily inspired by `Statsample::Dataset`, `Nyaplot::DataFrame` and the super-awesome pandas, a very mature solution in Python.
+daru is inspired by `Statsample::Dataset` and pandas, a very mature solution in Python.
 
-daru works with CRuby (1.9.3+) and JRuby and in a few weeks will be completely compatible with NMatrix and MDArray for fast data manipulation using C or Java structures.
+daru works with CRuby (1.9.3+) and JRuby.
 
 ## Features
 
@@ -24,7 +24,19 @@ daru works with CRuby (1.9.3+) and JRuby and in a few weeks will be completely c
 * Indexed and named data structures.
 * Flexible and intuitive API for manipulation and analysis of data.
 
-## Usage
+## Notebooks
+
+* [Analysis and plotting of a data set comprising of music listening habits of a last.fm user(iruby notebook)](http://nbviewer.ipython.org/github/v0dro/daru/blob/master/notebooks/intro_with_music_data_.ipynb)
+
+## Blog Posts
+
+* [Data Analysis in RUby: Basic data manipulation and plotting](http://v0dro.github.io/blog/2014/11/25/data-analysis-in-ruby-basic-data-manipulation-and-plotting/)
+
+## Documentation
+
+Docs can be found [here](https://rubygems.org/gems/daru).
+
+## Basic Usage
 
 daru has been created with keeping extreme ease of use in mind.
 
@@ -33,7 +45,7 @@ The gem consists of two data structures, Vector and DataFrame. Any data in a ser
 #### Initialization of DataFrame
 
 A data frame can be initialized from the following sources:
-* Hash of indexed vectors: `{ b: Daru::Vector.new(:b, [11,12,13,14,15], [:two, :one, :four, :five, :three]), a: Daru::Vector.new(:a, [1,2,3,4,5], [:two,:one,:three, :four, :five])}`.
+* Hash of indexed order: `{ b: Daru::Vector.new(:b, [11,12,13,14,15], [:two, :one, :four, :five, :three]), a: Daru::Vector.new(:a, [1,2,3,4,5], [:two,:one,:three, :four, :five])}`.
 * Array of hashes: `[{a: 1, b: 11}, {a: 2, b: 12}, {a: 3, b: 13},{a: 4, b: 14}, {a: 5, b: 15}]`.
 * Hash of names and Arrays: `{b: [11,12,13,14,15], a: [1,2,3,4,5]}`
 
@@ -43,7 +55,7 @@ A basic DataFrame can be initialized like this:
 
 ```ruby
 
-    df = Daru::DataFrame.new({b: [11,12,13,14,15], a: [1,2,3,4,5]}, vectors: [:a, :b], index: [:one, :two, :three, :four, :five])
+    df = Daru::DataFrame.new({b: [11,12,13,14,15], a: [1,2,3,4,5]}, order: [:a, :b], index: [:one, :two, :three, :four, :five])
     df
     # => 
     # # <Daru::DataFrame:87274040 @name = 7308c587-4073-4e7d-b3ca-3679d1dcc946 # @size = 5>
@@ -65,7 +77,7 @@ The vectors of the DataFrame will be arranged according to the array specified i
         b: [11,12,13,14,15].dv(:b, [:two, :one, :four, :five, :three]), 
         a:      [1,2,3,4,5].dv(:a, [:two,:one,:three, :four, :five])
       }, 
-        vectors: [:a, :b]
+        order: [:a, :b]
       )
     df
 
@@ -89,7 +101,7 @@ If an index for the DataFrame is supplied (third argument), then the indexes of
         a: [1,2,3]             .dv(nil, [:one, :two, :three]), 
         c: [11,22,33,44,55]    .dv(nil, [:one, :two, :three, :four, :five]),
         d: [49,69,89,99,108,44].dv(nil, [:one, :two, :three, :four, :five, :six])
-      }, vectors: [:a, :b, :c, :d], index: [:one, :two, :three, :four, :five, :six])
+      }, order: [:a, :b, :c, :d], index: [:one, :two, :three, :four, :five, :six])
     df
     # => 
     # #<Daru::DataFrame:87523270 @name = bda4eb68-afdd-4404-9981-708edab14201  #@size = 6>
@@ -111,7 +123,7 @@ If some of the supplied vectors do not contain certain indexes that are containe
              b: [11,12,13,14,15].dv(:b, [:two, :one, :four, :five, :three]), 
              a: [1,2,3]         .dv(:a, [:two,:one,:three])
            }, 
-           vectors: [:a, :b]
+           order: [:a, :b]
          )
     df
 
@@ -176,7 +188,7 @@ Initialize a dataframe:
         b: [11,12,13,14,15].dv(:b, [:two, :one, :four, :five, :three]), 
         a:      [1,2,3,4,5].dv(:a, [:two,:one,:three, :four, :five])
       }, 
-        vectors: [:a, :b]
+        order: [:a, :b]
       )
 
     #  => 
@@ -203,6 +215,20 @@ Select a row from a DataFrame:
 ```
 A row or a vector is returned as a `Daru::Vector` object, so any manipulations supported by `Daru::Vector` can be performed on the chosen row as well.
 
+Select multiple rows with a Range and get a DataFrame in return:
+
+``` ruby
+
+df.row[1..3] # OR df.row[:four..:three]
+# => 
+#<Daru::DataFrame:85361520 @name = d6582f66-5a55-473e-ba57-cb2ba974da6a @size #= 3>
+#                    a          b 
+#      four          4         13 
+#       one          2         12 
+#     three          3         15 
+
+```
+
 Select a single vector:
 
 ```ruby
@@ -251,16 +277,16 @@ Keep/remove row according to a specified condition:
     # five    5   14 
 
 ```
-The same can be applied to vectors using `keep_vector_if`.
+The same can be applied to vectors using `filter_vectors`.
 
-To iterate over a DataFrame and perform operations on rows or vectors, `#each_row` or `#each_vector` can be used, which works just like `#each` for Ruby Arrays.
+To iterate over a DataFrame and perform operations on rows or vectors, use `#each_row` or `#each_vector`.
 
 To change the values of a row/vector while iterating through the DataFrame, use `map_rows` or `map_vectors`:
 
 ```ruby
 
     df.map_rows do |row|
-        row = row.map { |e| e*e }
+        row = row * row
     end
 
     df
@@ -278,23 +304,48 @@ To change the values of a row/vector while iterating through the DataFrame, use
 
 Rows/vectors can be deleted using `delete_row` or `delete_vector`.
 
-#### Basic Math Operations
+#### Basic Maths Operations
+
+Performing a binary arithmetic operation on two `Daru::Vector` objects will return a `Vector` object in which the operation will be performed on elements of the same index.
+
+```ruby
+
+    dv1 = Daru::Vector.new [1,2,3,4], name: :boozy, index: [:a, :b, :c, :d]
+
+    dv2 = Daru::Vector.new [1,2,3,4], name: :mayer, index: [:e, :f, :b, :d]
+
+    dv1 * dv2
+
+    # #<Daru::Vector:80924700 @name = boozy @size = 2 >
+    #         boozy
+    #      b      6
+    #      d     16
+ 
+```
+
+Arithmetic operators applied on a single Numeric will perform the operation with that number against the entire vector.
+
+#### Statistics Operations
 
-Coming soon!
+Daru::Vector has a whole lot of statistics operations to maintain compatibility with Statsample::Vector. Check the docs for details.
+
+#### Plotting
+
+daru uses [Nyaplot](https://github.com/domitry/nyaplot) for plotting and an example of this can be found in the [notebook](http://nbviewer.ipython.org/github/v0dro/daru/blob/master/notebooks/intro_with_music_data_.ipynb) or [blog post](http://v0dro.github.io/blog/2014/11/25/data-analysis-in-ruby-basic-data-manipulation-and-plotting/).
+
+Head over to the tutorials and notebooks listed above for more examples.
 
 ## Roadmap
 
 * Automate testing for both MRI and JRuby.
 * Enable creation of DataFrame by only specifying an NMatrix/MDArray in initialize. Vector naming happens automatically (alphabetic) or is specified in an Array.
-* Destructive map iterators for DataFrame and Vector.
+* Destructive map iterators for DataFrame.
 * Completely test all functionality for NMatrix and MDArray.
 * Basic Data manipulation and analysis operations: 
     - Different kinds of join operations
     - Dataframe/vector merge
     - Creation of correlation, covariance matrices
     - Verification of data in a vector
-    - Basic vector statistics - mean, median, variance, etc.
-* Vector arithmetic - elementwise addition, subtraction, multiplication, division.
 * Transpose a dataframe.
 * Option to express a DataFrame as an NMatrix or MDArray so as to use more efficient storage techniques.
 * Assignment of a column to a single number should set the entire column to that number.
@@ -303,15 +354,24 @@ Coming soon!
 * Creation of DataFrame from Array of Arrays.
 * Multiple value assignment for vectors with []=.
 * Load DataFrame from multiple sources (excel, SQL, etc.).
-* Allow for boolean operations inside #[].
 * Deletion of elements from Vector should only modify the index and leave the vector as it is so that compacting is not needed and things are faster.
 * Add a #sync method which will sync the modified index with the unmodified vector.
 * Ability to reorder the index of a dataframe.
-* Slicing operations using Range.
-* Create DataFrame by providing rows.
-* Integrate basic plotting with Nyaplot.
-* Filter through a dataframe with filter\_rows or filter\_vectors based on whatever boolean value evaluates to true.
-* Named arguments
+* head/tail for DV.
+* #find\_max function which will evaluate a block and return the row for the value of the block is max.
+* Function to check if a value of a row/vector is within a specified range.
+* Create a new vector in map_rows if any of the already present rows dont match the one assigned in the block.
+* Direct functions to answer something like 'number of something per thousand of something else'.
+* Tests for checking NMatrix resizing
+* Sort while preserving index.
+
+## Contributing
+
+Pick a feature from the Roadmap above or think of your own and send me a Pull Request!
+
+## Acknowledgements
+
+* Thank you [last.fm](http://www.last.fm/) for making user data accessible to the public.
 
 Copyright (c) 2014, Sameer Deshmukh
 All rights reserved

data/daru.gemspec

@@ -6,6 +6,12 @@ require 'version.rb'
 DESCRIPTION = <<MSG
 Daru (Data Analysis in RUby) is a library for storage, analysis and manipulation
 of data.
+
+Daru works with Ruby arrays, NMatrix and MDArray, thus working seamlessly accross
+ruby interpreters, at the same time providing speed for those who need it.
+
+This library is under active development so NMatrix and MDArray support is
+somewhat limited, but should be available soon!
 MSG
 
 Gem::Specification.new do |spec|
@@ -27,6 +33,7 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency 'rake'
   spec.add_development_dependency 'rspec'
   spec.add_development_dependency 'awesome_print'
+  spec.add_development_dependency 'nyaplot'
   if RUBY_ENGINE != 'jruby'
     spec.add_development_dependency 'nmatrix', '~> 0.1.0.rc5'
   end

data/lib/daru/accessors/array_wrapper.rb

@@ -2,7 +2,255 @@ module Daru
   module Accessors
     # Internal class for wrapping ruby array
     class ArrayWrapper
+      module Statistics
 
+        def average_deviation_population m=nil
+          m ||= mean
+          (@vector.inject(0) {|memo, val| val + (val - m).abs }) / n_valid
+        end
+
+        def coefficient_of_variation
+          standard_deviation_sample / mean
+        end
+
+        def count value=false
+          if block_given?
+            @vector.inject(0){ |memo, val| memo += 1 if yield val; memo}
+          else
+            val = frequencies[value]
+            val.nil? ? 0 : val
+          end
+        end
+
+        def factors
+          index = @data.sorted_indices
+          index.reduce([]){|memo, val| memo.push(@data[val]) if memo.last != @data[val]; memo}
+        end # TODO
+
+        def frequencies
+          @vector.inject({}) do |hash, element|
+            hash[element] ||= 0
+            hash[element] += 1
+            hash
+          end
+        end
+
+        def has_missing_data?
+          has_missing_data
+        end
+
+        def kurtosis m=nil
+          m ||= mean
+          fo  = @vector.inject(0){ |a, x| a + ((x - m) ** 4) }
+          fo.quo(@size * standard_deviation_sample(m) ** 4) - 3
+        end
+
+        def mean
+          sum.quo(@size).to_f
+        end
+
+        def median
+          percentile 50
+        end
+
+        def median_absolute_deviation
+          m = median
+          recode {|val| (val - m).abs }.median
+        end
+
+        def mode
+          freqs = frequencies.values
+
+          @vector[freqs.index(freqs.max)]
+        end
+
+        def n_valid
+          @size
+        end
+
+        def percentile percent
+          sorted = @vector.sort
+          v      = (n_valid * percent).quo(100)
+          if v.to_i != v
+            sorted[v.round]
+          else
+            (sorted[(v - 0.5).round].to_f + sorted[(v + 0.5).round]).quo(2)
+          end
+        end
+
+        def product
+          @vector.inject(:*)
+        end
+
+        def max
+          @vector.max
+        end
+
+        def min
+          @vector.min
+        end
+
+        def proportion value=1
+          frequencies[value] / n_valid
+        end
+
+        def proportions
+          len = n_valid
+          frequencies.inject({}) { |hash, arr| hash[arr[0]] = arr[1] / len; hash }
+        end
+
+        def range
+          max - min
+        end
+
+        def ranked
+          sum = 0
+          r = frequencies.sort.inject( {} ) do |memo, val|
+            memo[val[0]] = ((sum + 1) + (sum + val[1])) / 2
+            sum += val[1]
+            memo
+          end
+
+          Daru::Vector.new @vector.map { |e| r[e] }, index: @caller.index,
+            name: @caller.name, dtype: @caller.dtype
+        end
+
+        def recode(&block)
+          @vector.map(&block)
+        end
+
+        def recode!(&block)
+          @vector.map!(&block)
+        end
+
+        # Calculate skewness using (sigma(xi - mean)^3)/((N)*std_dev_sample^3)
+        def skew m=nil
+          m ||= mean
+          th  = @vector.inject(0) { |memo, val| memo + ((val - m)**3) }
+          th.quo (@size * (standard_deviation_sample(m)**3))
+        end
+
+        def standard_deviation_population m=nil
+          m ||= mean
+          Math::sqrt(variance_population(m))
+        end
+
+        def standard_deviation_sample m=nil
+          Math::sqrt(variance_sample(m))
+        end
+
+        def standard_error
+          standard_deviation_sample/(Math::sqrt(@size))
+        end
+
+        def sum_of_squared_deviation
+          (@vector.inject(0) { |a,x| x.square + a } - (sum.square.quo(@size))).to_f
+        end
+
+        def sum_of_squares(m=nil)
+          m ||= mean
+          @vector.inject(0) { |memo, val| memo + (val - m)**2 }
+        end
+
+        def sum
+          @vector.inject(:+)
+        end
+
+        # Sample variance with denominator (N-1)
+        def variance_sample m=nil
+          m ||= self.mean
+
+          sum_of_squares(m).quo(@size - 1)
+        end
+
+        # Population variance with denominator (N)
+        def variance_population m=nil
+          m ||= mean
+
+          sum_of_squares(m).quo(@size).to_f
+        end
+      end # module Statistics
+
+      include Statistics
+      include Enumerable
+
+      def each(&block)
+        @vector.each(&block)
+      end
+
+      def map!(&block)
+        @vector.map!(&block)
+      end
+
+      attr_accessor :size
+      attr_reader   :vector
+      attr_reader   :has_missing_data
+
+      def initialize vector, caller
+        @vector = vector
+        @caller = caller
+
+        set_size
+      end
+
+      def [] index
+        @vector[index]
+      end
+
+      def []= index, value
+        has_missing_data = true if value.nil?
+        @vector[index] = value
+        set_size
+      end
+
+      def == other
+        @vector == other
+      end
+
+      def delete_at index
+        @vector.delete_at index
+        set_size
+      end
+
+      def index key
+        @vector.index key
+      end
+
+      def << element
+        @vector << element
+        set_size
+      end
+
+      def uniq
+        @vector.uniq
+      end
+
+      def to_a
+        @vector
+      end
+
+      def dup
+        ArrayWrapper.new @vector.dup, @caller
+      end
+
+      def coerce dtype
+        case
+        when dtype == Array
+          self
+        when dtype == NMatrix
+          Daru::Accessors::NMatrixWrapper.new @vector, @caller
+        when dtype == MDArray
+          raise NotImplementedError
+        else
+          raise ArgumentError, "Cant coerce to dtype #{dtype}"
+        end
+      end
+
+     private
+
+      def set_size
+        @size = @vector.size
+      end
     end
   end
 end