daru 0.1.0 → 0.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 6e48778067b94afc9f1060d7d6d4212029b421f2
-  data.tar.gz: 5d0ed9cc2fcf70562e0fcf2767c593e1f8fbfa54
+  metadata.gz: 91fd17361715d81a4eda319e4695bc53a44979de
+  data.tar.gz: 8768b7c62619d4f8446e8167a95990ee59553dde
 SHA512:
-  metadata.gz: 778ad55b592865e08388eac0001cdbce6bc01f58fa77ed8e2a2b72e44d8a54fc2d289f241352affd98b6424326416634ef729889163175f9eb64c83e471fb7e2
-  data.tar.gz: c498252daf63597adc0255810d3eb7b60c102ef086117bf451de821faaa8e196933570b06f88e6415b51c1ef6ea2ee6f60afce18b65691483741158954c73d0b
+  metadata.gz: bd7d40bb2e1b7ed2f4ea5a598556e8a58f5bf35c2fcbe895150a21752b4ca223ca184b6c7c6e1c1c30bf753df478578beec1d99aff2c31d4d7cfe2520013c47d
+  data.tar.gz: 264425a5bcd87e2eca1261792d3adf16a497ac20ec1c9c04743cafc84e9f4335525a2c289d36be63b9f88d8fcf19f84cc70824b6ada29ce3105872bc20231d18

data/.build.sh

@@ -6,9 +6,9 @@ gem build nmatrix.gemspec
 gem install nmatrix-0.1.0.gem
 cd ..
 rm -rf nmatrix
-git clone https://github.com/v0dro/gsl-nmatrix
-cd gsl-nmatrix
-gem build gsl-nmatrix.gemspec
-gem install gsl-nmatrix-1.17.gem
-cd ..
-rm -rf gsl-nmatrix
+# git clone https://github.com/v0dro/gsl-nmatrix
+# cd gsl-nmatrix
+# gem build gsl-nmatrix.gemspec
+# gem install gsl-nmatrix-1.17.gem
+# cd ..
+# rm -rf gsl-nmatrix

data/.gitignore

@@ -1,2 +1,4 @@
 *.gem
 Gemfile.lock
+doc/
+.yardoc/

data/CONTRIBUTING.md

@@ -2,9 +2,9 @@
 
 ## Installing daru development dependencies
 
-If you want to run the full rspec suite, you will need the latest unreleased nmatrix and gsl-nmatrix ruby gems. They will released upstream soon but please follow this procedure for now.
+If you want to run the full rspec suite, you will need the latest unreleased nmatrix gem. They will released upstream soon but please follow this procedure for now.
 
-Keep in mind that either nmatrix OR gsl-nmatrix are NOT NECESSARY for using daru. They are just required for an optional speed up.
+Keep in mind that either nmatrix or rb-gsl are NOT NECESSARY for using daru. They are just required for an optional speed up and for running the test suite.
 
 To install dependencies, execute the following commands:
 
@@ -16,7 +16,7 @@ To install dependencies, execute the following commands:
   `sudo apt-get install -y libgsl0-dev r-base r-base-dev`
   `sudo Rscript -e "install.packages(c('Rserve','irr'),,'http://cran.us.r-project.org')"`
 
-Then execute the .build.sh script to clone and install the latest nmatrix and gsl-nmatrix on your system:
+Then execute the [.build.sh script](https://github.com/v0dro/daru/blob/master/.build.sh) to clone and install the latest nmatrix system:
 
   `./.build.sh`
 
@@ -29,3 +29,7 @@ And run the test suite (should be all green with pending tests):
   `bundle exec rspec`
 
 If you have problems installing nmatrix, please consult the [nmatrix installation wiki](https://github.com/SciRuby/nmatrix/wiki/Installation) or the [mailing list](https://groups.google.com/forum/#!forum/sciruby-dev).
+
+## Daru internals
+
+To get an overview of certain internals of daru and their implementation, go over [this blog post](http://v0dro.github.io/blog/2015/08/16/elaboration-on-certain-internals-of-daru/).

data/History.md

@@ -1,3 +1,39 @@
+# 0.1.1
+
+* Enhancements
+    - Added a new class Daru::Offsets for providing a uniform API to jump between dates.
+    - Added benchmarking scripts
+    - Added a new Arel-like querying syntax for Vector and DataFrame. This will allow faster and more intuitive lookup of data than using loops such as filter.
+    - Vector
+        - #concat now compulsorily requires a second index argument.
+        - Added new method #index= to change the index directly.
+        - Added basic functions for rolling statistics - mean, std, count, etc.
+        - Added cumulative sum function.
+        - Added #keep_if.
+        - Added #count_values.
+    - Indexing
+        - Changed Index so that it now accepts all sorts of data (not restricted to only Symbols as it was previously).
+        - Re wrote MultiIndex in levels and labels form so that its faster and more accomodative of different kinds of index levels.
+        - Changed .new to return appropriate index object based on data passed.
+        - Added .from_tuple and .from_array methods to MultiIndex.
+        - Added union and intersection behaviour to Index and MultiIndex.
+        - Added a new index, DateTimeIndex for indexing with time-based data.
+        - Optimized range search for Index.
+    - DataFrame
+        - Removed the DataFrameByVector class and the #vector function. Now only
+        way to access a Vector in a DF is by using the #[] operator.
+        - Added new method #index= and #vectors= for changing row and column indexes directly.
+        - Optimized Vector value setting and retreival.
+        - Added inner, outer, left outer and right outer joins with the #join method.
+        - Added #set_index.
+* Changes
+    - Removed the + operator overload from Index and replaced in with union.
+    - Removed the second 'values' argument from Daru::Index because it's redundant.
+    - Changed behaviour of Vector#reindex and DataFrame#reindex and #reindex_vectors to preserve indexing of original data when possible.
+* Fixes
+    - Fixed DataFrame#delete_row and Vector#delete_if.
+    - Fixed Vector#rename.
+
 # 0.1.0
 
 * Fixes

data/README.md

@@ -19,7 +19,8 @@ Written in pure Ruby so should work with all ruby implementations. Tested with M
 * Data structures:
     - Vector - A basic 1-D vector.
     - DataFrame - A 2-D spreadsheet-like structure for manipulating and storing data sets. This is daru's primary data structure.
-* Compatible with [IRuby notebook](https://github.com/SciRuby/iruby) and [statsample](https://github.com/SciRuby/statsample).
+* Compatible with [IRuby notebook](https://github.com/SciRuby/iruby), [statsample](https://github.com/SciRuby/statsample), [statsample-glm](https://github.com/SciRuby/statsample-glm) and [statsample-timeseries](https://github.com/SciRuby/statsample-timeseries).
+* Support for time series.
 * Singly and hierarchially indexed data structures.
 * Flexible and intuitive API for manipulation and analysis of data.
 * Easy plotting, statistics and arithmetic.
@@ -27,27 +28,41 @@ Written in pure Ruby so should work with all ruby implementations. Tested with M
 * Optional speed and space optimization on MRI with [NMatrix](https://github.com/SciRuby/nmatrix) and GSL.
 * Easy splitting, aggregation and grouping of data.
 * Quickly reducing data with pivot tables for quick data summary.
-* Import and exports dataset from and to Excel, CSV, Databases and plain text files.
+* Import and export data from and to Excel, CSV, SQL Databases and plain text files.
 
 ## Notebooks
 
-### Usage
+#### Notebooks on most use cases
 
+* [Overview of most daru functions](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Data%20Analysis/Daru%20Demo.ipynb)
 * [Basic Creation of Vectors and DataFrame](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Data%20Analysis/Creation%20of%20Vector%20and%20DataFrame.ipynb)
 * [Detailed Usage of Daru::Vector](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Data%20Analysis/Usage%20of%20Vector.ipynb)
 * [Detailed Usage of Daru::DataFrame](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Data%20Analysis/Usage%20of%20DataFrame.ipynb)
 * [Visualizing Data With Daru::DataFrame](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Visualization/Visualizing%20data%20with%20daru%20DataFrame.ipynb)
+* [Searching and combining data in daru](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Data%20Analysis/Searching%20and%20Combining%20Data.ipynb)
 * [Grouping, Splitting and Pivoting Data](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Data%20Analysis/Grouping%2C%20Splitting%20and%20Pivoting.ipynb)
 
+#### Notebooks on Time series
+
+* [Basic Time Series](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Data%20Analysis/Basic%20Time%20Series.ipynb)
+* [Time Series Analysis and Plotting](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Data%20Analysis/Time%20Series%20Functions.ipynb)
+
 ### Case Studies
 
 * [Logistic Regression Analysis with daru and statsample-glm](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Data%20Analysis/Logistic%20Regression%20with%20daru%20and%20statsample-glm.ipynb)
 * [Finding and Plotting most heard artists from a Last.fm dataset](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Data%20Analysis/Finding%20and%20plotting%20the%20most%20heard%20artists%20on%20last%20fm.ipynb)
+* [Analyzing baby names with daru](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Data%20Analysis/Analyzing%20baby%20names/Use%20Case%20-%20Daru%20for%20analyzing%20baby%20names%20data.ipynb)
 
 ## Blog Posts
 
 * [Data Analysis in RUby: Basic data manipulation and plotting](http://v0dro.github.io/blog/2014/11/25/data-analysis-in-ruby-basic-data-manipulation-and-plotting/)
 * [Data Analysis in RUby: Splitting, sorting, aggregating data and data types](http://v0dro.github.io/blog/2015/02/24/data-analysis-in-ruby-part-2/)
+* [Finding and Combining data in daru](http://v0dro.github.io/blog/2015/08/03/finding-and-combining-data-in-daru/)
+
+### Time series
+
+* [Analysis of Time Series in daru](http://v0dro.github.io/blog/2015/07/31/analysis-of-time-series-in-daru/)
+* [Date Offsets in Daru](http://v0dro.github.io/blog/2015/07/27/date-offsets-in-daru/)
 
 ## Documentation
 
@@ -55,32 +70,25 @@ Docs can be found [here](https://rubygems.org/gems/daru).
 
 ## Roadmap
 
-* Automate testing for both MRI and JRuby.
 * Enable creation of DataFrame by only specifying an NMatrix/MDArray in initialize. Vector naming happens automatically (alphabetic) or is specified in an Array.
-* Completely test all functionality for MDArray.
 * Basic Data manipulation and analysis operations: 
     - DF concat
-* Option to express a DataFrame as an NMatrix or MDArray so as to use more efficient storage techniques.
 * Assignment of a column to a single number should set the entire column to that number.
-* == between daru_vector and string/number.
 * Multiple column assignment with []=
 * Multiple value assignment for vectors with []=.
 * #find\_max function which will evaluate a block and return the row for the value of the block is max.
-* Function to check if a value of a row/vector is within a specified range.
-* Create a new vector in map_rows if any of the already present rows dont match the one assigned in the block.
 * Sort by index.
-* Statistics on DataFrame over rows and columns.
-* Cumulative sum.
-* Time series support.
+* Statistics on DataFrame over rows.
 * Calculate percentage change.
 * Have some sample data sets for users to play around with. Should be able to load these from the code itself.
 * Sorting with missing data present.
-* re_index should re establish previous index values in the newly supplied index.
 
 ## Contributing
 
 Pick a feature from the Roadmap or the issue tracker or think of your own and send me a Pull Request!
 
+For details see [CONTRIBUTING](https://github.com/v0dro/daru/blob/master/CONTRIBUTING.md).
+
 ## Acknowledgements
 
 * Google and the Ruby Science Foundation for the Google Summer of Code 2015 grant for further developing daru and integrating it with other ruby gems.

data/Rakefile

@@ -1,6 +1,21 @@
 require 'rspec/core/rake_task'
 require 'bundler/gem_tasks'
 
+lib_folder = File.expand_path("../lib", __FILE__)
 RSpec::Core::RakeTask.new(:spec)
 
-task :default => :spec
+task :default => :spec
+
+task :console do |task|
+  cmd = [ 'irb', "-r '#{lib_folder}/daru.rb'" ]
+  run *cmd
+end
+
+task :pry do |task|
+  cmd = [ 'pry', "-r '#{lib_folder}/daru.rb'" ]
+  run *cmd
+end
+
+def run *cmd
+  sh(cmd.join(" "))
+end

data/benchmarks/TradeoffData.csv

@@ -0,0 +1,65 @@
+"Group","Treatment","Replicate","RelativeFitness"
+"BKB","Tube",1,0.869962555792838
+"BKB","Tube",2,1.00036299125423
+"BKB","Tube",3,0.982935090384188
+"BAC","Tube",1,0.810391635206191
+"BAC","Tube",2,0.795106571577928
+"JDK","Tube",1,0.849203581734814
+"JDK","Tube",2,0.917636977577209
+"JDK","Tube",3,0.905323024195181
+"ETH","Tube",1,0.930820887284085
+"ETH","Tube",2,0.958183317853959
+"ETH","Tube",3,0.91428823286218
+"SWI","Tube",1,0.918513532826773
+"SWI","Tube",2,0.909023164202865
+"SWI","Tube",3,0.928404818223812
+"PPP","Tube",1,0.990500457882474
+"PPP","Tube",2,0.97156464183624
+"PPP","Tube",3,0.948766905642272
+"ECO","Tube",1,0.996870763277447
+"ECO","Tube",2,0.976750424056972
+"ECO","Tube",3,0.939479247244218
+"DOS","Tube",1,0.9461642789306
+"DOS","Tube",2,0.951056617451754
+"DOS","Tube",3,0.939563175900147
+"FIT","Tube",1,0.942215594296048
+"FIT","Tube",2,0.926698027408171
+"FIT","Tube",3,1.00031750759377
+"HHE","Tube",1,0.944414164259624
+"HHE","Tube",2,0.95194592074032
+"HHE","Tube",3,0.922485751593779
+"H2W","Tube",1,0.915304498822462
+"H2W","Tube",2,0.853906227834699
+"H2W","Tube",3,0.938698147417512
+"BKB","Dish",1,1.15254427639376
+"BKB","Dish",2,1.28708514998039
+"BKB","Dish",3,1.50741545998468
+"BAC","Dish",2,1.65223419104399
+"BAC","Dish",3,1.61502230247434
+"JDK","Dish",1,1.5102682623396
+"JDK","Dish",2,1.58270078151532
+"JDK","Dish",3,1.54715152852448
+"ETH","Dish",1,1.61281555981988
+"ETH","Dish",2,1.36445572541848
+"ETH","Dish",3,1.47155233724852
+"SWI","Dish",1,1.36255560722945
+"SWI","Dish",2,1.51569228710501
+"SWI","Dish",3,1.47714121664773
+"PPP","Dish",1,1.50896917413568
+"PPP","Dish",2,1.58138592886817
+"PPP","Dish",3,1.55356839081014
+"ECO","Dish",1,1.69927564777656
+"ECO","Dish",2,1.45605668065038
+"ECO","Dish",3,1.52825762511041
+"DOS","Dish",1,1.59453120031454
+"DOS","Dish",2,1.58132511409296
+"DOS","Dish",3,1.58558683941181
+"FIT","Dish",1,1.01468578343171
+"FIT","Dish",2,0.955220815085047
+"FIT","Dish",3,1.03597298511451
+"HHE","Dish",1,1.49399422139513
+"HHE","Dish",2,1.32980278545457
+"HHE","Dish",3,1.4505220676174
+"H2W","Dish",1,1.52460143511286
+"H2W","Dish",2,1.53958247554139
+"H2W","Dish",3,1.51149897493835

data/benchmarks/dataframe_creation.rb

@@ -0,0 +1,39 @@
+$:.unshift File.expand_path("../../lib", __FILE__)
+
+require 'benchmark'
+require 'daru'
+
+Benchmark.bm do |x|
+  x.report("Create with Arrays and clone") do
+    df = Daru::DataFrame.new({
+      a: 100000.times.map { rand },
+      b: 100000.times.map { rand },
+      c: 100000.times.map { rand }
+    })
+  end
+
+  x.report("Create with Vectors and clone") do
+    df = Daru::DataFrame.new({
+      a: Daru::Vector.new(100000.times.map { rand }),
+      b: Daru::Vector.new(100000.times.map { rand }),
+      c: Daru::Vector.new(100000.times.map { rand })
+    })
+  end
+
+  x.report("Create with Vector and dont clone") do
+    df = Daru::DataFrame.new({
+      a: Daru::Vector.new(100000.times.map { rand }),
+      b: Daru::Vector.new(100000.times.map { rand }),
+      c: Daru::Vector.new(100000.times.map { rand })
+    }, clone: false)
+  end
+
+  x.report("Create by row from Arrays") do
+  end
+end
+
+#                           ===== Benchmarks =====
+#                                       user     system      total        real
+# Create with Arrays and clone       0.940000   0.010000   0.950000 (  0.959851)
+# Create with Vectors and clone      1.950000   0.020000   1.970000 (  1.966835)
+# Create with Vector and dont clone  1.170000   0.000000   1.170000 (  1.177132)

data/benchmarks/group_by.rb

@@ -0,0 +1,32 @@
+$:.unshift File.expand_path("../../lib", __FILE__)
+
+require 'benchmark'
+require 'daru'
+
+data = Daru::DataFrame.from_csv 'TradeoffData.csv'
+
+Benchmark.bm do |x|
+  x.report("Single column grouping") do
+    @single = data.group_by([:Treatment])
+  end
+
+  x.report("Multi-column grouping") do
+    @multi = data.group_by([:Group, :Treatment])
+  end
+
+  x.report("Single mean") do
+    @single.mean
+  end
+
+  x.report("Multi mean") do
+    @multi.mean
+  end
+end
+
+#                    ===== Benchmarks =====
+# 
+#                          user     system      total        real
+# Single column grouping 0.000000   0.000000   0.000000   (0.000356)
+# Multi-column grouping  0.000000   0.000000   0.000000   (0.000958)
+# Single mean            0.000000   0.000000   0.000000   (0.000865)
+# Multi mean             0.000000   0.000000   0.000000   (0.002748)

data/benchmarks/row_access.rb

@@ -0,0 +1,41 @@
+$:.unshift File.expand_path("../../lib", __FILE__)
+
+require 'benchmark'
+require 'daru'
+
+df = Daru::DataFrame.new({
+  a: 100000.times.map { rand },
+  b: 100000.times.map { rand },
+  c: 100000.times.map { rand }
+})
+
+Benchmark.bm do |x|
+  x.report("Access single row") do
+    df.row[50]
+  end
+
+  x.report("Access rows by comma") do
+    df.row[*(5..40000).to_a.shuffle]
+  end
+
+  x.report("Individual rows") do
+    rows = []
+    index = (5..40000).to_a.shuffle
+    index.each do |a|
+      rows << df.row[a].to_a
+    end
+
+    Daru::DataFrame.rows(rows, order: [:a,:b,:c], index: index)
+  end
+
+  x.report("Access rows by range") do
+    df.row[5..40000]
+  end
+end
+
+#                     ==== Benchmarks ====
+#                         user     system      total        real
+# Access single row     0.000000   0.000000   0.000000 (  0.000059)
+# Access rows by comma  1.410000   0.010000   1.420000 (  1.420426)
+# Individual rows       1.480000   0.000000   1.480000 (  1.488531)
+# Access rows by range  1.440000   0.010000   1.450000 (  1.436750)

data/benchmarks/row_assign.rb

@@ -0,0 +1,36 @@
+$:.unshift File.expand_path("../../lib", __FILE__)
+
+require 'benchmark'
+require 'daru'
+
+df = Daru::DataFrame.new({
+  a: 100000.times.map { rand },
+  b: 100000.times.map { rand },
+  c: 100000.times.map { rand }
+})
+
+Benchmark.bm do |x|
+  x.report("Set a single row with Array") do
+    df.row[5] = [55,22,65]
+  end
+
+  x.report("Set a single row with Daru::Vector") do
+    df.row[3456] = Daru::Vector.new([3,54,11], index: [:b,:e,:a])
+  end
+
+  x.report("Create a new row with Array") do
+    df.row[100001] = [34,66,11]
+  end
+
+  x.report("Create a new row with Daru::Vector") do
+    df.row[100005] = Daru::Vector.new([34,66,11], index: [:a,:b,:t])
+  end
+end
+
+#                      ==== Benchmarks ====
+#
+#                                       user     system      total        real
+# Set a single row with Array         0.600000   0.000000   0.600000 (  0.604718)
+# Set a single row with Daru::Vector  0.600000   0.000000   0.600000 (  0.598599)
+# Create a new row with Array         0.840000   0.010000   0.850000 (  0.858349)
+# Create a new row with Daru::Vector  0.950000   0.000000   0.950000 (  0.950725)

data/benchmarks/sorting.rb

@@ -0,0 +1,44 @@
+$:.unshift File.expand_path("../../lib", __FILE__)
+
+require 'benchmark'
+require 'daru'
+
+vector = Daru::Vector.new(10000.times.map.to_a.shuffle)
+df = Daru::DataFrame.new({
+  a: vector,
+  b: vector,
+  c: vector
+})
+Benchmark.bm do |x|
+  x.report("Sort a Vector without any args") do
+    vector.sort
+  end
+
+  x.report("Sort vector in descending order with custom <=> operator") do
+    vector.sort(ascending: false) { |a,b| a.to_s <=> b.to_s }
+  end
+
+  x.report("Sort single column of DataFrame") do
+    df.sort([:a])
+  end
+
+  x.report("Sort two columns of DataFrame") do
+    df.sort([:c,:a])
+  end
+
+  x.report("Sort two columns with custom operators in different orders of DataFrame") do
+    df.sort([:c,:a], ascending: [true, false], 
+      by: { c: lambda { |a,b| a.to_s <=> b.to_s },
+            a: lambda { |a,b| (a+1) <=> (b+1) } })
+  end
+end
+
+# FIXME: MASSIVE SPEEDUP NECESSARY!
+
+#                                         ===== Benchamarks =====
+#                                                                             user      system      total      real
+# Sort a Vector without any args                                           0.130000    0.000000 0.130000    (  0.128006)
+# Sort vector in descending order with custom <=> operator                 0.190000    0.000000 0.190000    (  0.184604)
+# Sort single column of DataFrame                                          2502.450000 0.000000 2502.450000 (2503.808073)
+# Sort two columns of DataFrame                                            0.540000    0.000000 0.540000    (  0.537670)
+# Sort two columns with custom operators in different orders of DataFrame  2084.160000 7.260000 2091.420000 (2092.716603)