daru 0.1.1 → 0.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 91fd17361715d81a4eda319e4695bc53a44979de
-  data.tar.gz: 8768b7c62619d4f8446e8167a95990ee59553dde
+  metadata.gz: ed2a3e2a4cd9fce8d95af6aac9c3db532eed444f
+  data.tar.gz: 90ca6a62ee824d20f72a9f6689c03f27d7667168
 SHA512:
-  metadata.gz: bd7d40bb2e1b7ed2f4ea5a598556e8a58f5bf35c2fcbe895150a21752b4ca223ca184b6c7c6e1c1c30bf753df478578beec1d99aff2c31d4d7cfe2520013c47d
-  data.tar.gz: 264425a5bcd87e2eca1261792d3adf16a497ac20ec1c9c04743cafc84e9f4335525a2c289d36be63b9f88d8fcf19f84cc70824b6ada29ce3105872bc20231d18
+  metadata.gz: e6f3345ef4372e1c45a3d80c0cc61c2b4c72e4c810cfb183f30bfd9285a09639ea39cd0a3597fc63551d7f72398d8d83af4424855018e8a5b2a99274b46625cd
+  data.tar.gz: 65d262b1deec54680a5fdcfecda3530c9fb9450dbd280c18833b655521418ed340f311253221dfd4e018577b063f9d3638d0d600a108c3b98ec5a7cd2dfe98ec

data/.travis.yml

@@ -1,13 +1,11 @@
 language:
   ruby
 
-env:
-  - CPLUS_INCLUDE_PATH=/usr/include/atlas C_INCLUDE_PATH=/usr/include/atlas
-
 rvm:
   - '2.0'
   - '2.1'
   - '2.2'
+  - '2.3.0'
 
 matrix:
   fast_finish:
@@ -17,11 +15,9 @@ script: "bundle exec rspec"
 
 install:
   - gem install bundler
-  - ./.build.sh
   - bundle install
 
 before_install:
   - sudo apt-get update -qq
-  - sudo apt-get install -qq libatlas-base-dev
   - sudo apt-get install -y libgsl0-dev r-base r-base-dev
   - sudo Rscript -e "install.packages(c('Rserve','irr'),,'http://cran.us.r-project.org')"

data/CONTRIBUTING.md

@@ -2,25 +2,16 @@
 
 ## Installing daru development dependencies
 
-If you want to run the full rspec suite, you will need the latest unreleased nmatrix gem. They will released upstream soon but please follow this procedure for now.
-
-Keep in mind that either nmatrix or rb-gsl are NOT NECESSARY for using daru. They are just required for an optional speed up and for running the test suite.
+Either nmatrix or rb-gsl are NOT NECESSARY for using daru. They are just required for an optional speed up and for running the test suite.
 
 To install dependencies, execute the following commands:
 
-  `export CPLUS_INCLUDE_PATH=/usr/include/atlas` 
-  `export C_INCLUDE_PATH=/usr/include/atlas`
   `sudo apt-get update -qq`
-  `sudo apt-get install -qq libatlas-base-dev`
-  `sudo apt-get --purge remove liblapack-dev liblapack3 liblapack3gf`
   `sudo apt-get install -y libgsl0-dev r-base r-base-dev`
   `sudo Rscript -e "install.packages(c('Rserve','irr'),,'http://cran.us.r-project.org')"`
 
-Then execute the [.build.sh script](https://github.com/v0dro/daru/blob/master/.build.sh) to clone and install the latest nmatrix system:
-
-  `./.build.sh`
 
-Then finally install remaining dependencies:
+Then install remaining dependencies:
 
   `bundle install`
 

data/History.md

@@ -1,3 +1,21 @@
+# 0.1.2
+
+* Enhancements
+    - New method `DataFrame.from_activerecord` for importing data sets from ActiveRecord. (by @mrkn)
+    - Better importing of data from SQL databases by extracting that functionality into a separate class called `Daru::IO::SqlDataSource` (by @mrkn).
+    - Faster algorithm for performing inner joins by using the bloomfilter-rb gem. Available only for MRI. (by Peter Tung)
+    - Added exception `SizeError` (by Peter Tung).
+    - Removed outdated dependencies and build scripts, updated existing dependencies.
+    - Ability to sort a Daru::Vector with nils present (by @gnilrets)
+
+* Fixes
+    - Fix column creation for `Dataframe.from_sql` (by @dansbits).
+    - group_by can now be performed on DataFrames with nils (@gnilrets).
+    - Bug fix for DataFrame Vectors not duplicating when calling `DataFrame#dup` (by @gnilrets).
+    - Bug fix when concantenating DataFrames (by @gnilrets)
+    - Handling improper arguments to `Daru::Vector#[]` (by @lokeshh)
+    - Resolve narray conflict by using the latest nmatrix require methods (by @lokeshh)
+
 # 0.1.1
 
 * Enhancements

data/README.md

@@ -1,18 +1,13 @@
-daru
-====
-
-Data Analysis in RUby
+# daru - Data Analysis in RUby
 
 [![Gem Version](https://badge.fury.io/rb/daru.svg)](http://badge.fury.io/rb/daru)
 [![Build Status](https://travis-ci.org/v0dro/daru.svg)](https://travis-ci.org/v0dro/daru)
 
 ## Introduction
 
-daru (Data Analysis in RUby) is a library for storage, analysis, manipulation and visualization of data.
-
-daru is inspired by pandas, a very mature solution in Python.
+daru (Data Analysis in RUby) is a library for storage, analysis, manipulation and visualization of data in Ruby.
 
-Written in pure Ruby so should work with all ruby implementations. Tested with MRI 2.0, 2.1, 2.2.
+daru makes it easy and intuituive to process data predominantly through 2 data structures: `Daru::DataFrame` and `Daru::Vector`. Written in pure Ruby works with all ruby implementations. Tested with MRI 2.0, 2.1, 2.2 and 2.3.
 
 ## Features
 
@@ -28,7 +23,7 @@ Written in pure Ruby so should work with all ruby implementations. Tested with M
 * Optional speed and space optimization on MRI with [NMatrix](https://github.com/SciRuby/nmatrix) and GSL.
 * Easy splitting, aggregation and grouping of data.
 * Quickly reducing data with pivot tables for quick data summary.
-* Import and export data from and to Excel, CSV, SQL Databases and plain text files.
+* Import and export data from and to Excel, CSV, SQL Databases, ActiveRecord and plain text files.
 
 ## Notebooks
 
@@ -64,6 +59,111 @@ Written in pure Ruby so should work with all ruby implementations. Tested with M
 * [Analysis of Time Series in daru](http://v0dro.github.io/blog/2015/07/31/analysis-of-time-series-in-daru/)
 * [Date Offsets in Daru](http://v0dro.github.io/blog/2015/07/27/date-offsets-in-daru/)
 
+## Basic Usage
+
+daru exposes two major data structures: `DataFrame` and `Vector`. The Vector is a basic 1-D structure corresponding to a labelled Array, while the `DataFrame` - daru's primary data structure - is 2-D spreadsheet-like structure for manipulating and storing data sets.
+
+Basic DataFrame intitialization.
+
+``` ruby
+data_frame = Daru::DataFrame.new(
+  {
+    'Beer' => ['Kingfisher', 'Snow', 'Bud Light', 'Tiger Beer', 'Budweiser'],
+    'Gallons sold' => [500, 400, 450, 200, 250]
+  },
+  index: ['India', 'China', 'USA', 'Malaysia', 'Canada']
+)
+data_frame
+```
+![init0](images/init0.png)
+
+
+Load data from CSV files.
+``` ruby
+df = Daru::DataFrame.from_csv('TradeoffData.csv')
+```
+![init1](images/init1.png)
+
+*Basic Data Manipulation*
+
+Selecting rows.
+``` ruby
+data_frame.row['USA']
+```
+![man0](images/man0.png)
+
+Selecting columns.
+``` ruby
+data_frame['Beer']
+```
+![man1](images/man1.png)
+
+A range of rows.
+``` ruby
+data_frame.row['India'..'USA']
+```
+![man2](images/man2.png)
+
+The first 2 rows.
+``` ruby
+data_frame.first(2)
+```
+![man3](images/man3.png)
+
+The last 2 rows.
+``` ruby
+data_frame.last(2)
+```
+![man4](images/man4.png)
+
+Adding a new column.
+``` ruby
+data_frame['Gallons produced'] = [550, 500, 600, 210, 240]
+```
+![man5](images/man5.png)
+
+Creating a new column based on data in other columns.
+``` ruby
+data_frame['Demand supply gap'] = data_frame['Gallons produced'] - data_frame['Gallons sold']
+```
+![man6](images/man6.png)
+
+*Condition based selection*
+
+Selecting countries based on the number of gallons sold in each. We use a syntax similar to that defined by [Arel](https://github.com/rails/arel), i.e. by using the `where` clause.
+``` ruby
+data_frame.where(data_frame['Gallons sold'].lt(300))
+```
+![con0](images/con0.png)
+
+You can pass a combination of boolean operations into the `#where` method and it should work fine:
+``` ruby
+data_frame.where(
+  data_frame['Beer']
+  .in(['Snow', 'Kingfisher','Tiger Beer'])
+  .and(
+    data_frame['Gallons produced'].gt(520).or(data_frame['Gallons produced'].lt(250))
+  )
+)
+```
+![con1](images/con1.png)
+
+*Plotting* 
+
+Daru supports plotting of interactive graphs with [nyaplot](). You can easily create a plot with the `#plot` method. Here we plot the gallons sold on the Y axis and name of the brand on the X axis in a bar graph.
+``` ruby
+data_frame.plot type: :bar, x: 'Beer', y: 'Gallons sold' do |plot, diagram|
+  plot.x_label "Beer"
+  plot.y_label "Gallons Sold"
+  plot.yrange [0,600]
+  plot.width 500
+  plot.height 400
+end
+```
+![plot0](images/plot0.png)
+
+In addition to nyaplot, daru also supports plotting out of the box with [gnuplotrb](https://github.com/SciRuby/gnuplotrb).
+
 ## Documentation
 
 Docs can be found [here](https://rubygems.org/gems/daru).
@@ -71,8 +171,6 @@ Docs can be found [here](https://rubygems.org/gems/daru).
 ## Roadmap
 
 * Enable creation of DataFrame by only specifying an NMatrix/MDArray in initialize. Vector naming happens automatically (alphabetic) or is specified in an Array.
-* Basic Data manipulation and analysis operations: 
-    - DF concat
 * Assignment of a column to a single number should set the entire column to that number.
 * Multiple column assignment with []=
 * Multiple value assignment for vectors with []=.

data/daru.gemspec

@@ -34,7 +34,7 @@ Thank you for installing daru!
   oOOOOOo 
  ,|    oO
 //|     |
-\\|     |
+\\\\|     |
  `|     |
   `-----`
 
@@ -50,17 +50,22 @@ Cheers!
 EOF
 
   spec.add_runtime_dependency 'reportbuilder', '~> 1.4'
-  spec.add_runtime_dependency 'spreadsheet', '~> 1.0.3'
+  spec.add_runtime_dependency 'spreadsheet', '~> 1.1.1'
 
   spec.add_development_dependency 'bundler', '~> 1.10'
-  spec.add_development_dependency 'rake'
+  spec.add_development_dependency 'rake', '~>10.5'
   spec.add_development_dependency 'pry', '~> 0.10'
   spec.add_development_dependency 'pry-byebug'
   spec.add_development_dependency 'rserve-client', '~> 0.3'
-  spec.add_development_dependency 'rspec'
+  spec.add_development_dependency 'rspec', '~> 3.4'
   spec.add_development_dependency 'awesome_print'
   spec.add_development_dependency 'nyaplot', '~> 0.1.5'
-  spec.add_development_dependency 'nmatrix', '~> 0.1.0'
+  spec.add_development_dependency 'nmatrix', '~> 0.2.1'
   spec.add_development_dependency 'distribution', '~> 0.7'
   spec.add_development_dependency 'rb-gsl', '~>1.16'
-end
+  spec.add_development_dependency 'bloomfilter-rb', '~> 2.1'
+  spec.add_development_dependency 'dbd-sqlite3'
+  spec.add_development_dependency 'dbi'
+  spec.add_development_dependency 'activerecord', '~> 4.0'
+  spec.add_development_dependency 'sqlite3'
+end

data/images/README.md

@@ -0,0 +1,5 @@
+# Images
+
+This folder contains images that are being used in the project README to display code examples.
+
+Do not change any names of files.

data/lib/daru.rb

@@ -38,10 +38,12 @@ module Daru
     attr_accessor :lazy_update
     
     def create_has_library(library)
-      define_singleton_method("has_#{library}?") do
-        cv = "@@#{library}"
+      lib_underscore = library.to_s.gsub(/-/, '_')
+      define_singleton_method("has_#{lib_underscore}?") do
+        cv = "@@#{lib_underscore}"
         unless class_variable_defined? cv
           begin
+            library = 'nmatrix/nmatrix' if library == :nmatrix
             require library.to_s
             class_variable_set(cv, true)
           rescue LoadError
@@ -56,6 +58,7 @@ module Daru
   create_has_library :gsl
   create_has_library :nmatrix
   create_has_library :nyaplot
+  create_has_library :'bloomfilter-rb'
 end
 
 autoload :Spreadsheet, 'spreadsheet'

data/lib/daru/core/group_by.rb

@@ -18,7 +18,7 @@ module Daru
         @context = context
         vectors = names.map { |vec| context[vec].to_a }
         tuples  = vectors[0].zip(*vectors[1..-1])
-        keys    = tuples.uniq.sort
+        keys    = tuples.uniq.sort { |a,b| a && b ? a.compact <=> b.compact : a ? 1 : -1 }
 
         keys.each do |key|
           @groups[key] = all_indices_for(tuples, key)
@@ -28,7 +28,7 @@ module Daru
 
       # Get a Daru::Vector of the size of each group.
       def size
-        index = 
+        index =
         if multi_indexed_grouping?
           Daru::MultiIndex.from_tuples @groups.keys
         else
@@ -59,15 +59,15 @@ module Daru
       #     d:   [11 ,22 ,33 ,44   ,55 ,66 ,77 ,88]
       #   })
       #   df.group_by([:a, :b]).head(1)
-      #   # => 
+      #   # =>
       #   # #<Daru::DataFrame:82745170 @name = d7003f75-5eb9-4967-9303-c08dd9160224 @size = 6>
-      #   #                     a          b          c          d 
-      #   #          1        bar        one          2         22 
-      #   #          3        bar      three          1         44 
-      #   #          5        bar        two          6         66 
-      #   #          0        foo        one          1         11 
-      #   #          7        foo      three          8         88 
-      #   #          2        foo        two          3         33  
+      #   #                     a          b          c          d
+      #   #          1        bar        one          2         22
+      #   #          3        bar      three          1         44
+      #   #          5        bar        two          6         66
+      #   #          0        foo        one          1         11
+      #   #          7        foo      three          8         88
+      #   #          2        foo        two          3         33
       def head quantity=5
         select_groups_from :first, quantity
       end
@@ -82,14 +82,14 @@ module Daru
       #     d:   [11 ,22 ,33 ,44   ,55 ,66 ,77 ,88]
       #   })
       #   # df.group_by([:a, :b]).tail(1)
-      #   # => 
+      #   # =>
       #   # #<Daru::DataFrame:82378270 @name = 0623db46-5425-41bd-a843-99baac3d1d9a @size = 6>
-      #   #                     a          b          c          d 
-      #   #          1        bar        one          2         22 
-      #   #          3        bar      three          1         44 
-      #   #          5        bar        two          6         66 
-      #   #          6        foo        one          3         77 
-      #   #          7        foo      three          8         88 
+      #   #                     a          b          c          d
+      #   #          1        bar        one          2         22
+      #   #          3        bar      three          1         44
+      #   #          5        bar        two          6         66
+      #   #          6        foo        one          3         77
+      #   #          7        foo      three          8         88
       #   #          4        foo        two          3         55
       def tail quantity=5
         select_groups_from :last, quantity
@@ -103,15 +103,15 @@ module Daru
       #     c:   [1  ,2  ,3  ,1    ,3  ,6  ,3  ,8],
       #     d:   [11 ,22 ,33 ,44   ,55 ,66 ,77 ,88]
       #   df.group_by([:a, :b]).mean
-      #   # => 
+      #   # =>
       #   # #<Daru::DataFrame:81097450 @name = 0c32983f-3e06-451f-a9c9-051cadfe7371 @size = 6>
-      #   #                         c          d 
-      #   # ["bar", "one"]          2         22 
-      #   # ["bar", "three"]        1         44 
-      #   # ["bar", "two"]          6         66 
-      #   # ["foo", "one"]        2.0       44.0 
-      #   # ["foo", "three"]        8         88 
-      #   # ["foo", "two"]        3.0       44.0 
+      #   #                         c          d
+      #   # ["bar", "one"]          2         22
+      #   # ["bar", "three"]        1         44
+      #   # ["bar", "two"]          6         66
+      #   # ["foo", "one"]        2.0       44.0
+      #   # ["foo", "three"]        8         88
+      #   # ["foo", "two"]        3.0       44.0
       def mean
         apply_method :numeric, :mean
       end
@@ -128,28 +128,28 @@ module Daru
 
       # Count groups, excludes missing values.
       # @example Using count
-      #   df = Daru::DataFrame.new({    
-      #     a: %w{foo bar foo bar   foo bar foo foo},      
-      #     b: %w{one one two three two two one three},      
-      #     c:   [1  ,2  ,3  ,1    ,3  ,6  ,3  ,8],      
-      #     d:   [11 ,22 ,33 ,44   ,55 ,66 ,77 ,88]      
-      #   })        
+      #   df = Daru::DataFrame.new({
+      #     a: %w{foo bar foo bar   foo bar foo foo},
+      #     b: %w{one one two three two two one three},
+      #     c:   [1  ,2  ,3  ,1    ,3  ,6  ,3  ,8],
+      #     d:   [11 ,22 ,33 ,44   ,55 ,66 ,77 ,88]
+      #   })
       #   df.group_by([:a, :b]).count
-      #   # => 
+      #   # =>
       #   # #<Daru::DataFrame:76900210 @name = 7b9cf55d-17f8-48c7-b03a-2586c6e5ec5a @size = 6>
-      #   #                           c          d 
-      #   # ["bar", "one"]            1          1 
-      #   # ["bar", "two"]            1          1 
-      #   # ["bar", "three"]          1          1 
-      #   # ["foo", "one"]            2          2 
-      #   # ["foo", "three"]          1          1 
-      #   # ["foo", "two"]            2          2 
+      #   #                           c          d
+      #   # ["bar", "one"]            1          1
+      #   # ["bar", "two"]            1          1
+      #   # ["bar", "three"]          1          1
+      #   # ["foo", "one"]            2          2
+      #   # ["foo", "three"]          1          1
+      #   # ["foo", "two"]            2          2
       def count
         width = @non_group_vectors.size
         Daru::DataFrame.new([size]*width, order: @non_group_vectors)
       end
 
-      # Calculate sample standard deviation of numeric vector groups, excluding 
+      # Calculate sample standard deviation of numeric vector groups, excluding
       # missing values.
       def std
         apply_method :numeric, :std
@@ -177,9 +177,9 @@ module Daru
       #         d:   [11 ,22 ,33 ,44   ,55 ,66 ,77 ,88]
       #       })
       #   df.group_by([:a, :b]).get_group ['bar','two']
-      #   #=> 
+      #   #=>
       #   ##<Daru::DataFrame:83258980 @name = 687ee3f6-8874-4899-97fa-9b31d84fa1d5 @size = 1>
-      #   #                    a          b          c          d 
+      #   #                    a          b          c          d
       #   #         5        bar        two          6         66
       def get_group group
         indexes   = @groups[group]
@@ -198,7 +198,7 @@ module Daru
           rows, index: @context.index[indexes], order: @context.vectors)
       end
 
-     private 
+     private
 
       def select_groups_from method, quantity
         selection     = @context
@@ -227,7 +227,7 @@ module Daru
               slice = vec[*indexes]
               single_row << (slice.is_a?(Numeric) ? slice : slice.send(method))
             end
-          end 
+          end
 
           rows << single_row
         end
@@ -260,4 +260,4 @@ module Daru
       end
     end
   end
-end
+end