red_amber 0.1.5 → 0.1.8

data/README.md

@@ -1,16 +1,20 @@
 # RedAmber
 
-A simple dataframe library for Ruby (experimental)
+[![Gem Version](https://badge.fury.io/rb/red_amber.svg)](https://badge.fury.io/rb/red_amber)
+[![Ruby](https://github.com/heronshoes/red_amber/actions/workflows/test.yml/badge.svg)](https://github.com/heronshoes/red_amber/actions/workflows/test.yml)
 
-- Powered by [Red Arrow](https://github.com/apache/arrow/tree/master/ruby/red-arrow)
+A simple dataframe library for Ruby (experimental).
+
+- Powered by [Red Arrow](https://github.com/apache/arrow/tree/master/ruby/red-arrow) [![Gitter Chat](https://badges.gitter.im/red-data-tools/en.svg)](https://gitter.im/red-data-tools/en)
 - Inspired by the dataframe library [Rover-df](https://github.com/ankane/rover)
 
 ## Requirements
 
 ```ruby
 gem 'red-arrow',   '>= 8.0.0'
-gem 'red-parquet', '>= 8.0.0' # if you use IO from/to parquet
-gem 'rover-df',    '~> 0.3.0' # if you use IO from/to Rover::DataFrame
+
+gem 'red-parquet', '>= 8.0.0' # Optional, if you use IO from/to parquet
+gem 'rover-df',    '~> 0.3.0' # Optional, if you use IO from/to Rover::DataFrame
 ```
 
 ## Installation
@@ -18,7 +22,8 @@ gem 'rover-df',    '~> 0.3.0' # if you use IO from/to Rover::DataFrame
 Install requirements before you install Red Amber.
 
 - Apache Arrow GLib (>= 8.0.0)
-- Apache Parquet GLib (>= 8.0.0)
+
+- Apache Parquet GLib (>= 8.0.0)  # If you use IO from/to parquet
 
   See [Apache Arrow install document](https://arrow.apache.org/install/).
   
@@ -44,27 +49,28 @@ gem install red_amber
 
 ## `RedAmber::DataFrame`
 
-Represents a set of data in 2D-shape.
+Represents a set of data in 2D-shape. The entity is a Red Arrow's Table object. 
 
 ```ruby
-require 'red_amber'
+require 'red_amber' # require 'red-amber' is also OK.
 require 'datasets-arrow'
 
 arrow = Datasets::Penguins.new.to_arrow
 penguins = RedAmber::DataFrame.new(arrow)
-penguins.tdr
+
 # =>
-RedAmber::DataFrame : 344 x 8 Vectors
-Vectors : 5 numeric, 3 strings
-# key                type   level data_preview
-1 :species           string     3 {"Adelie"=>152, "Chinstrap"=>68, "Gentoo"=>124}
-2 :island            string     3 {"Torgersen"=>52, "Biscoe"=>168, "Dream"=>124}
-3 :bill_length_mm    double   165 [39.1, 39.5, 40.3, nil, 36.7, ... ], 2 nils
-4 :bill_depth_mm     double    81 [18.7, 17.4, 18.0, nil, 19.3, ... ], 2 nils
-5 :flipper_length_mm uint8     56 [181, 186, 195, nil, 193, ... ], 2 nils
-6 :body_mass_g       uint16    95 [3750, 3800, 3250, nil, 3450, ... ], 2 nils
-7 :sex               string     3 {"male"=>168, "female"=>165, nil=>11}
-8 :year              uint16     3 {2007=>110, 2008=>114, 2009=>120}
+#<RedAmber::DataFrame : 344 x 8 Vectors, 0x0000000000013790>
+    species  island    bill_length_mm bill_depth_mm flipper_length_mm ...     year
+    <string> <string>        <double>      <double>           <uint8> ... <uint16>
+  1 Adelie   Torgersen           39.1          18.7               181 ...     2007
+  2 Adelie   Torgersen           39.5          17.4               186 ...     2007
+  3 Adelie   Torgersen           40.3          18.0               195 ...     2007
+  4 Adelie   Torgersen          (nil)         (nil)             (nil) ...     2007
+  5 Adelie   Torgersen           36.7          19.3               193 ...     2007
+  : :        :                      :             :                 : ...        :
+342 Gentoo   Biscoe              50.4          15.7               222 ...     2009
+343 Gentoo   Biscoe              45.2          14.8               212 ...     2009
+344 Gentoo   Biscoe              49.9          16.1               213 ...     2009
 ```
 
 ### DataFrame model
@@ -72,33 +78,179 @@ Vectors : 5 numeric, 3 strings
 
 For example, `DataFrame#pick` accepts keys as an argument and returns a sub DataFrame.
 
+![pick method image](doc/image/dataframe/pick.png)
+
 ```ruby
-df = penguins.pick(:body_mass_g)
+penguins.keys
+# =>
+[:species,                                       
+ :island,                                        
+ :bill_length_mm,
+ :bill_depth_mm,
+ :flipper_length_mm,
+ :body_mass_g,
+ :sex,
+ :year]
+
+df = penguins.pick(:species, :island, :body_mass_g)
+df
+
+# =>
+#<RedAmber::DataFrame : 344 x 3 Vectors, 0x000000000003cc1c>                 
+    species  island    body_mass_g                                           
+    <string> <string>     <uint16>                                           
+  1 Adelie   Torgersen        3750                                           
+  2 Adelie   Torgersen        3800                                           
+  3 Adelie   Torgersen        3250                                           
+  4 Adelie   Torgersen       (nil)                                           
+  5 Adelie   Torgersen        3450                                           
+  : :        :                   :                                           
+342 Gentoo   Biscoe           5750                                           
+343 Gentoo   Biscoe           5200                                           
+344 Gentoo   Biscoe           5400
+```
+
+`DataFrame#drop` drops some columns to create a remainer DataFrame.
+
+![drop method image](doc/image/dataframe/drop.png)
+
+You can specify by keys or a boolean array (same size as n_keys).
+
+```ruby
+# Same as df.drop(:species, :island)
+df = df.drop(true, true, false)
+
 # =>
-#<RedAmber::DataFrame : 344 x 1 Vector, 0x000000000000fa14>
-Vector : 1 numeric
-# key          type  level data_preview
-1 :body_mass_g int64    95 [3750, 3800, 3250, nil, 3450, ... ], 2 nils
+#<RedAmber::DataFrame : 344 x 1 Vector, 0x0000000000048760>
+    body_mass_g                                     
+       <uint16>                                     
+  1        3750                                     
+  2        3800                                     
+  3        3250                                     
+  4       (nil)                                     
+  5        3450                                     
+  :           :                                     
+342        5750                                     
+343        5200                                     
+344        5400
 ```
 
+Arrow data is immutable, so these methods always return an new object.
+
 `DataFrame#assign` creates new variables (column in the table).
 
+![assign method image](doc/image/dataframe/assign.png)
+
 ```ruby
+# New column is created because ':body_mass_kg' is a new key.
 df.assign(:body_mass_kg => df[:body_mass_g] / 1000.0)
+
+# =>
+#<RedAmber::DataFrame : 344 x 2 Vectors, 0x00000000000212f0>
+    body_mass_g body_mass_kg
+       <uint16>     <double>
+  1        3750          3.8
+  2        3800          3.8
+  3        3250          3.3
+  4       (nil)        (nil)
+  5        3450          3.5
+  :           :            :
+342        5750          5.8
+343        5200          5.2
+344        5400          5.4
+```
+
+`DataFrame#slice` selects rows (observations) to create a sub DataFrame.
+
+![slice method image](doc/image/dataframe/slice.png)
+
+```ruby
+# returns 5 rows at the start and 5 rows from the end
+penguins.slice(0...5, -5..-1)
+
 # =>
-#<RedAmber::DataFrame : 344 x 2 Vectors, 0x000000000000fa28>
-Vectors : 2 numeric
-# key           type   level data_preview
-1 :body_mass_g  int64     95 [3750, 3800, 3250, nil, 3450, ... ], 2 nils
-2 :body_mass_kg double    95 [3.75, 3.8, 3.25, nil, 3.45, ... ], 2 nils
+#<RedAmber::DataFrame : 10 x 8 Vectors, 0x0000000000042be4>
+   species  island    bill_length_mm bill_depth_mm flipper_length_mm ...     year
+   <string> <string>        <double>      <double>           <uint8> ... <uint16>
+ 1 Adelie   Torgersen           39.1          18.7               181 ...     2007
+ 2 Adelie   Torgersen           39.5          17.4               186 ...     2007
+ 3 Adelie   Torgersen           40.3          18.0               195 ...     2007
+ 4 Adelie   Torgersen          (nil)         (nil)             (nil) ...     2007
+ 5 Adelie   Torgersen           36.7          19.3               193 ...     2007
+ : :        :                      :             :                 : ...        :
+ 8 Gentoo   Biscoe              50.4          15.7               222 ...     2009
+ 9 Gentoo   Biscoe              45.2          14.8               212 ...     2009
+10 Gentoo   Biscoe              49.9          16.1               213 ...     2009
+```
+
+`DataFrame#remove` rejects rows (observations) to create a remainer DataFrame.
+
+![remove method image](doc/image/dataframe/remove.png)
+
+```ruby
+# penguins[:bill_length_mm] < 40 returns a boolean Vector
+penguins.remove(penguins[:bill_length_mm] < 40)
+
+# =>
+#<RedAmber::DataFrame : 244 x 8 Vectors, 0x000000000007d6f4>
+    species  island    bill_length_mm bill_depth_mm flipper_length_mm ...     year
+    <string> <string>        <double>      <double>           <uint8> ... <uint16>
+  1 Adelie   Torgersen           40.3          18.0               195 ...     2007
+  2 Adelie   Torgersen          (nil)         (nil)             (nil) ...     2007
+  3 Adelie   Torgersen           42.0          20.2               190 ...     2007
+  4 Adelie   Torgersen           41.1          17.6               182 ...     2007
+  5 Adelie   Torgersen           42.5          20.7               197 ...     2007
+  : :        :                      :             :                 : ...        :
+242 Gentoo   Biscoe              50.4          15.7               222 ...     2009
+243 Gentoo   Biscoe              45.2          14.8               212 ...     2009
+244 Gentoo   Biscoe              49.9          16.1               213 ...     2009
 ```
 
 DataFrame manipulating methods like `pick`, `drop`, `slice`, `remove`, `rename` and `assign` accept a block.
 
-This is an exaple to eliminate observations (row in the table) containing nil.
+This example is usage of block to update numeric columns.
+
+```ruby
+df = RedAmber::DataFrame.new(
+  integer: [0, 1, 2, 3, nil],
+  float:   [0.0, 1.1,  2.2, Float::NAN, nil],
+  string:  ['A', 'B', 'C', 'D', nil],
+  boolean: [true, false, true, false, nil])
+df
+
+# =>
+#<RedAmber::DataFrame : 5 x 4 Vectors, 0x000000000003131c>
+  integer    float string   boolean
+  <uint8> <double> <string> <boolean>
+1       0      0.0 A        true
+2       1      1.1 B        false
+3       2      2.2 C        true
+4       3      NaN D        false
+5   (nil)    (nil) (nil)    (nil)
+
+df.assign do
+  vectors.each_with_object({}) do |v, h|
+    h[v.key] = -v if v.numeric?
+  end
+end
+
+# =>
+#<RedAmber::DataFrame : 5 x 4 Vectors, 0x000000000009a1b4>
+  integer    float string   boolean
+  <uint8> <double> <string> <boolean>
+1       0     -0.0 A        true
+2     255     -1.1 B        false
+3     254     -2.2 C        true
+4     253      NaN D        false
+5   (nil)    (nil) (nil)    (nil)
+```
+
+Negate (-@) method of unsigned integer Vector returns complement. 
+
+Next example is to eliminate observations (row in the table) containing nil.
 
 ```ruby
-# remove all observation contains nil
+# remove all observations containing nil
 nil_removed = penguins.remove { vectors.map(&:is_nil).reduce(&:|) }
 nil_removed.tdr
 # =>
@@ -121,12 +273,51 @@ For this frequently needed task, we can do it much simpler.
 penguins.remove_nil # => same result as above
 ```
 
+`DataFrame#group` method can be used for the grouping tasks.
+
+```ruby
+starwars = RedAmber::DataFrame.load(URI("https://vincentarelbundock.github.io/Rdatasets/csv/dplyr/starwars.csv"))
+starwars
+
+# =>
+#<RedAmber::DataFrame : 87 x 12 Vectors, 0x000000000000607c>
+   unnamed1 name            height     mass hair_color skin_color  eye_color ... species
+    <int64> <string>       <int64> <double> <string>   <string>    <string>  ... <string>
+ 1        1 Luke Skywalker     172     77.0 blond      fair        blue      ... Human
+ 2        2 C-3PO              167     75.0 NA         gold        yellow    ... Droid
+ 3        3 R2-D2               96     32.0 NA         white, blue red       ... Droid
+ 4        4 Darth Vader        202    136.0 none       white       yellow    ... Human
+ 5        5 Leia Organa        150     49.0 brown      light       brown     ... Human
+ :        : :                    :        : :          :           :         ... :
+85       85 BB8              (nil)    (nil) none       none        black     ... Droid
+86       86 Captain Phasma   (nil)    (nil) unknown    unknown     unknown   ... NA
+87       87 Padmé Amidala      165     45.0 brown      light       brown     ... Human
+
+grouped = starwars.group(:species) { [count(:species), mean(:height, :mass)] }
+grouped.slice { v(:count) > 1 }
+
+# =>
+#<RedAmber::DataFrame : 9 x 4 Vectors, 0x000000000006e848>
+  species    count mean(height) mean(mass)
+  <string> <int64>     <double>   <double>
+1 Human         35        176.6       82.8
+2 Droid          6        131.2       69.8
+3 Wookiee        2        231.0      124.0
+4 Gungan         3        208.7       74.0
+5 NA             4        181.3       48.0
+: :              :            :          :
+7 Twi'lek        2        179.0       55.0
+8 Mirialan       2        168.0       53.1
+9 Kaminoan       2        221.0       88.0 
+```
+
 See [DataFrame.md](doc/DataFrame.md) for details.
 
 
 ## `RedAmber::Vector`
 
 Class `RedAmber::Vector` represents a series of data in the DataFrame.
+Method `RedAmber::DataFrame#[key]` returns a Vector with the key `key`. 
 
 ```ruby
 penguins[:bill_length_mm]
@@ -137,11 +328,34 @@ penguins[:bill_length_mm]
 
 Vectors accepts some [functional methods from Arrow](https://arrow.apache.org/docs/cpp/compute.html).
 
+This is an element-wise comparison and returns a boolean Vector of same size.
+
+![unary element-wise](doc/image/vector/unary_element_wise.png)
+
+```ruby
+penguins[:bill_length_mm] < 40
+
+# =>
+#<RedAmber::Vector(:boolean, size=344):0x000000000007e7ac>
+[true, true, false, nil, true, true, true, true, true, false, true, true, false, ... ]
+```
+
+Next example returns aggregated result.
+
+![unary aggregation](doc/image/vector/unary_aggregation.png)
+
+```ruby
+penguins[:bill_length_mm].mean
+43.92192982456141
+# =>
+
+```
+
 See [Vector.md](doc/Vector.md) for details.
 
-## TDR concept
+## Jupyter notebook
 
-I named the data frame representation style in the model above as TDR (Transposed DataFrame Representation). See [TDR.md](doc/tdr.md) for details.
+[53 Examples of Red Amber](doc/examples_of_red_amber.ipynb)
 
 ## Development
 

data/Rakefile

@@ -7,6 +7,7 @@ Rake::TestTask.new(:test) do |t|
   t.libs << 'test'
   t.libs << 'lib'
   t.test_files = FileList['test/**/test_*.rb']
+  t.warning = false
 end
 
 require 'rubocop/rake_task'

data/benchmark/csv_load_penguins.yml

@@ -1,11 +1,11 @@
 prelude: |
-  require 'datasets-arrow'
   require 'rover'
   require 'red_amber'
 
   penguins_csv = 'benchmark/cache/penguins.csv'
 
   unless File.exist?(penguins_csv)
+    require 'datasets-arrow'
     arrow = Datasets::Penguins.new.to_arrow
     RedAmber::DataFrame.new(arrow).save(penguins_csv)
   end