red_amber 0.1.3 → 0.1.6

data/README.md

@@ -1,20 +1,29 @@
 # RedAmber
 
-A simple dataframe library for Ruby (experimental)
+A simple dataframe library for Ruby (experimental).
 
 - Powered by [Red Arrow](https://github.com/apache/arrow/tree/master/ruby/red-arrow)
-- Simple API similar to [Rover-df](https://github.com/ankane/rover)
+- Inspired by the dataframe library [Rover-df](https://github.com/ankane/rover)
 
 ## Requirements
 
 ```ruby
-gem 'red-arrow',   '>= 7.0.0'
-gem 'red-parquet', '>= 7.0.0' # if you use IO from/to parquet
+gem 'red-arrow',   '>= 8.0.0'
+gem 'red-parquet', '>= 8.0.0' # if you use IO from/to parquet
 gem 'rover-df',    '~> 0.3.0' # if you use IO from/to Rover::DataFrame
 ```
 
 ## Installation
 
+Install requirements before you install Red Amber.
+
+- Apache Arrow GLib (>= 8.0.0)
+- Apache Parquet GLib (>= 8.0.0)
+
+  See [Apache Arrow install document](https://arrow.apache.org/install/).
+  
+  Minimum installation example for the latest Ubuntu is in the ['Prepare the Apache Arrow' section in ci test](https://github.com/heronshoes/red_amber/blob/master/.github/workflows/test.yml) of Red Amber.
+
 Add this line to your Gemfile:
 
 ```ruby
@@ -23,134 +32,66 @@ gem 'red_amber'
 
 And then execute:
 
-    $ bundle install
+```shell
+bundle install
+```
 
 Or install it yourself as:
 
-    $ gem install red_amber
-
-## `RedAmber::DataFrame`
-
-### Constructors and saving
-
-- [x] `new` from a columnar Hash
-  - `RedAmber::DataFrame.new(x: [1, 2, 3])`
-
-- [x] `new` from a schema (by Hash) and rows (by Array)
-  - `RedAmber::DataFrame.new({:x=>:uint8}, [[1], [2], [3]])`
-
-- [x] `new` from an Arrow::Table
-  - `RedAmber::DataFrame.new(Arrow::Table.new(x: [1, 2, 3]))`
-
-- [x] `new` from a Rover::DataFrame
-  - `RedAmber::DataFrame.new(Rover::DataFrame.new(x: [1, 2, 3]))`
-
-- [x] `load` (class method)
-
-     - [x] from a [`.arrow`, `.arrows`, `.csv`, `.csv.gz`, `.tsv`] file
-       - `RedAmber::DataFrame.load("test/entity/with_header.csv")`
-
-     - [x] from a string buffer
-
-     - [x] from a URI
-       - `RedAmber::DataFrame.load(URI("https://github.com/heronshoes/red_amber/blob/master/test/entity/with_header.csv"))`
-
-     - [x] from a Parquet file
-
-       `red-parquet` gem is required.
-
-  ```ruby
-    require 'parquet'
-    dataframe = RedAmber::DataFrame.load("file.parquet")
-  ```
-
-- [x] `save` (instance method)
-
-     - [x] to a [`.arrow`, `.arrows`, `.csv`, `.csv.gz`, `.tsv`] file
-
-     - [x] to a string buffer
-
-     - [x] to a URI
-
-     - [x] to a Parquet file
-
-       `red-parquet` gem is required.
-
-  ```ruby
-    require 'parquet'
-    dataframe.save("file.parquet")
-  ```
-
-### Properties
-
-- [x] `table`
-
-  Reader of Arrow::Table object inside.
-
-- [x] `n_rows`, `nrow`, `size`, `length`
-  
-  Returns num of rows (data size).
- 
-- [x] `n_columns`, `ncol`, `width`
-  
-  Returns num of columns (num of vectors).
- 
-- [x] `shape`
- 
-  Returns shape in an Array[n_rows, n_cols].
- 
-- [x] `column_names`, `keys`
-  
-  Returns num of column names by an Array.
-
-- [x] `types`
-  
-  Returns types of columns by an Array of Symbols.
-
-- [x] `data_types`
-
-  Returns types of columns by an Array of `Arrow::DataType`.
-
-- [x] `vectors`
-
-  Returns an Array of Vectors.
-
-- [x] `to_h`
-
-  Returns column-oriented data in a Hash.
-
-- [x] `to_a`, `raw_records`
-
-  Returns an array of row-oriented data without header. If you need a column-oriented full array, use `.to_h.to_a`
-
-- [x] `schema`
-
-  Returns column name and data type in a Hash.
+```shell
+gem install red_amber
+```
 
-- [x] `==`
- 
-- [x] `empty?`
+(From v0.1.6)
 
-### Output
+RedAmber uses TDR mode for `#inspect` and `#to_iruby` by default. If you prefer Table mode, please set the environment variable
+`RED_AMBER_OUTPUT_MODE` to `"table"`. See [TDR section](#TDR) for detail.
 
-- [x] `to_s`
+## `RedAmber::DataFrame`
 
-- [ ] summary, describe
+Represents a set of data in 2D-shape. The entity is a Red Arrow's Table object. 
 
-- [x] `to_rover`
+```ruby
+require 'red_amber' # require 'red-amber' is also OK.
+require 'datasets-arrow'
 
-  Returns a `Rover::DataFrame`.
+arrow = Datasets::Penguins.new.to_arrow
+penguins = RedAmber::DataFrame.new(arrow)
+penguins.table
 
-- [x] `inspect(tally_level: 5, max_element: 5)`
+# =>
+#<Arrow::Table:0x111271098 ptr=0x7f9118b3e0b0>
+	species	island	bill_length_mm	bill_depth_mm	flipper_length_mm	body_mass_g	sex	year
+  0	Adelie 	Torgersen	     39.100000	    18.700000	              181	       3750	male	2007
+  1	Adelie 	Torgersen	     39.500000	    17.400000	              186	       3800	female	2007
+  2	Adelie 	Torgersen	     40.300000	    18.000000	              195	       3250	female	2007
+  3	Adelie 	Torgersen	        (null)	       (null)	           (null)	     (null)	(null)	2007
+  4	Adelie 	Torgersen	     36.700000	    19.300000	              193	       3450	female	2007
+  5	Adelie 	Torgersen	     39.300000	    20.600000	              190	       3650	male	2007
+  6	Adelie 	Torgersen	     38.900000	    17.800000	              181	       3625	female	2007
+  7	Adelie 	Torgersen	     39.200000	    19.600000	              195	       4675	male	2007
+  8	Adelie 	Torgersen	     34.100000	    18.100000	              193	       3475	(null)	2007
+  9	Adelie 	Torgersen	     42.000000	    20.200000	              190	       4250	(null)	2007
+...
+334	Gentoo 	Biscoe	     46.200000	    14.100000	              217	       4375	female	2009
+335	Gentoo 	Biscoe	     55.100000	    16.000000	              230	       5850	male	2009
+336	Gentoo 	Biscoe	     44.500000	    15.700000	              217	       4875	(null)	2009
+337	Gentoo 	Biscoe	     48.800000	    16.200000	              222	       6000	male	2009
+338	Gentoo 	Biscoe	     47.200000	    13.700000	              214	       4925	female	2009
+339	Gentoo 	Biscoe	        (null)	       (null)	           (null)	     (null)	(null)	2009
+340	Gentoo 	Biscoe	     46.800000	    14.300000	              215	       4850	female	2009
+341	Gentoo 	Biscoe	     50.400000	    15.700000	              222	       5750	male	2009
+342	Gentoo 	Biscoe	     45.200000	    14.800000	              212	       5200	female	2009
+343	Gentoo 	Biscoe	     49.900000	    16.100000	              213	       5400	male	2009
+```
 
-  Shows some information about self in a transposed style.
+By default, RedAmber shows self by compact transposed style. This unfamiliar style (TDR) is designed for
+the exploratory data processing. It keeps Vectors as row vectors, shows keys and types at a glance, shows levels 
+for the 'factor-like' variables and shows the number of abnormal values like NaN and nil.
 
 ```ruby
-require 'red_amber'
-require 'datasets-arrow'
+penguins
 
-penguins = Datasets::Penguins.new.to_arrow
-RedAmber::DataFrame.new(penguins)
 # =>
 RedAmber::DataFrame : 344 x 8 Vectors
 Vectors : 5 numeric, 3 strings
@@ -165,257 +106,91 @@ Vectors : 5 numeric, 3 strings
 8 :year              uint16     3 {2007=>110, 2008=>114, 2009=>120}
 ```
 
-  - tally_level: max level to use tally mode
-  - max_element: max num of element to show values in each row
-
-### Selecting
+### DataFrame model
+![dataframe model of RedAmber](doc/image/dataframe_model.png)
 
-- [x] Select columns by `[]` as `[key]`, `[keys]`, `[keys[index]]`
-  - Key in a Symbol: `df[:symbol]`
-  - Key in a String: `df["string"]`
-  - Keys in an Array: `df[:symbol1, "string", :symbol2]`
-  - Keys in indeces: `df[df.keys[0]`, `df[df.keys[1,2]]`, `df[df.keys[1..]]`
-  - Keys in a Range:
-    A end-less Range can be used to represent keys.
+For example, `DataFrame#pick` accepts keys as an argument and returns a sub DataFrame.
 
 ```ruby
-hash = {a: [1, 2, 3], b: %w[A B C], c: [1.0, 2, 3]}
-df = RedAmber::DataFrame.new(hash)
-df[:b..:c, "a"]
+df = penguins.pick(:body_mass_g)
 # =>
-RedAmber::DataFrame : 3 x 3 Vectors
-Vectors : 2 numeric, 1 string
-# key type   level data_preview
-1 :b  string     3 ["A", "B", "C"]
-2 :c  double     3 [1.0, 2.0, 3.0]
-3 :a  uint8      3 [1, 2, 3]
+#<RedAmber::DataFrame : 344 x 1 Vector, 0x000000000000fa14>
+Vector : 1 numeric
+# key          type  level data_preview
+1 :body_mass_g int64    95 [3750, 3800, 3250, nil, 3450, ... ], 2 nils
 ```
 
-- [x] Select rows by `[]` as `[index]`, `[range]`, `[array]`
-  - Select a row by index: `df[0]`
-  - Select rows by indeces in a Range: `df[1..2]`
-  - Select rows by indeces in an Array: `df[1, 2]`
-  - Mixed case: `df[2, 0..]`
-
-- [x] Select rows from top or bottom
-
-  `head(n=5)`, `tail(n=5)`, `first(n=1)`, `last(n=1)`
-
-- [ ] slice
-
-### Updating
-
-- [ ] Add a new column
-
-- [ ] Update a single element
-
-- [ ] Update multiple elements
-
-- [ ] Update all elements
-
-- [ ] Update elements matching a condition
+`DataFrame#assign` creates new variables (column in the table).
 
-- [ ] Clamp
-
-- [ ] Delete columns
-
-- [ ] Rename a column
-
-- [ ] Sort rows
-
-- [ ] Clear data
-
-### Treat na data
-
-- [ ] Drop na (NaN, nil)
-
-- [ ] Replace na with value
-
-- [ ] Interpolate na with convolution array
-
-### Combining DataFrames
-
-- [ ] Add rows
-
-- [ ] Add columns
+```ruby
+df.assign(:body_mass_kg => df[:body_mass_g] / 1000.0)
+# =>
+#<RedAmber::DataFrame : 344 x 2 Vectors, 0x000000000000fa28>
+Vectors : 2 numeric
+# key           type   level data_preview
+1 :body_mass_g  int64     95 [3750, 3800, 3250, nil, 3450, ... ], 2 nils
+2 :body_mass_kg double    95 [3.75, 3.8, 3.25, nil, 3.45, ... ], 2 nils
+```
 
-- [ ] Inner join
+DataFrame manipulating methods like `pick`, `drop`, `slice`, `remove`, `rename` and `assign` accept a block.
 
-- [ ] Left join
+This is an exaple to eliminate observations (row in the table) containing nil.
 
-### Encoding
+```ruby
+# remove all observation contains nil
+nil_removed = penguins.remove { vectors.map(&:is_nil).reduce(&:|) }
+nil_removed.tdr
+# =>
+RedAmber::DataFrame : 342 x 8 Vectors
+Vectors : 5 numeric, 3 strings
+# key                type   level data_preview
+1 :species           string     3 {"Adelie"=>151, "Chinstrap"=>68, "Gentoo"=>123}
+2 :island            string     3 {"Torgersen"=>51, "Biscoe"=>167, "Dream"=>124}
+3 :bill_length_mm    double   164 [39.1, 39.5, 40.3, 36.7, 39.3, ... ]
+4 :bill_depth_mm     double    80 [18.7, 17.4, 18.0, 19.3, 20.6, ... ]
+5 :flipper_length_mm int64     55 [181, 186, 195, 193, 190, ... ]
+6 :body_mass_g       int64     94 [3750, 3800, 3250, 3450, 3650, ... ]
+7 :sex               string     3 {"male"=>168, "female"=>165, ""=>9}
+8 :year              int64      3 {2007=>109, 2008=>114, 2009=>119}
+```
 
-- [ ] One-hot encoding
+For this frequently needed task, we can do it much simpler.
 
-### Iteration (not impremented)
+```ruby
+penguins.remove_nil # => same result as above
+```
 
-### Filtering (not impremented)
+See [DataFrame.md](doc/DataFrame.md) for details.
 
 
 ## `RedAmber::Vector`
-### Constructor
-
-- [x] Create from a column in a DataFrame
-
-- [x] New from an Array
-
-### Properties
-
-- [x] `to_s`
-
-- [x] `values`, `to_a`, `entries`
 
-- [x] `size`, `length`, `n_rows`, `nrow`
+Class `RedAmber::Vector` represents a series of data in the DataFrame.
 
-- [x] `type`
-
-- [x] `data_type`
-
-- [ ] `each`
-
-- [ ] `chunked?`
-
-- [ ] `n_chunks`
-
-- [ ] `each_chunk`
-
-- [x] `tally`
-
-- [x] `n_nils`, `n_nans`
-
-  - `n_nulls` is an alias of `n_nils`
-
-- [x] `inspect(limit: 80)`
+```ruby
+penguins[:bill_length_mm]
+# =>
+#<RedAmber::Vector(:double, size=344):0x000000000000f8fc>
+[39.1, 39.5, 40.3, nil, 36.7, 39.3, 38.9, 39.2, 34.1, 42.0, 37.8, 37.8, 41.1, ... ]
+```
 
-  - `limit` sets size limit to display long array.
+Vectors accepts some [functional methods from Arrow](https://arrow.apache.org/docs/cpp/compute.html).
 
-### Functions
-#### Unary aggregations: vector.func => scalar
+See [Vector.md](doc/Vector.md) for details.
 
-| Method    |Boolean|Numeric|String|Options|Remarks|
-| ----------- | --- | --- | --- | --- | --- |
-| ✓ `all`     |  ✓  |     |     | ✓ ScalarAggregate|     |
-| ✓ `any`     |  ✓  |     |     | ✓ ScalarAggregate|     |
-| ✓ `approximate_median`|  |✓|  | ✓ ScalarAggregate| alias `median`|
-| ✓ `count`   |  ✓  |  ✓  |  ✓  | ✓  Count  |     |
-| ✓ `count_distinct`| ✓ | ✓ | ✓ | ✓  Count  |alias `count_uniq`|
-|[ ]`index`   | [ ] | [ ] | [ ] |[ ] Index  |     |
-| ✓ `max`     |  ✓  |  ✓  |  ✓  | ✓ ScalarAggregate|     |
-| ✓ `mean`    |  ✓  |  ✓  |     | ✓ ScalarAggregate|     |
-| ✓ `min`     |  ✓  |  ✓  |  ✓  | ✓ ScalarAggregate|     |
-|[ ]`min_max` | [ ] | [ ] | [ ] |[ ] ScalarAggregate|     |
-|[ ]`mode`    |     | [ ] |     |[ ] Mode    |     |
-| ✓ `product` |  ✓  |  ✓  |     | ✓ ScalarAggregate|     |
-|[ ]`quantile`|     | [ ] |     |[ ] Quantile|     |
-|[ ]`stddev`  |     |  ✓  |     |[ ] Variance|     |
-| ✓ `sum`     |  ✓  |  ✓  |     | ✓ ScalarAggregate|     |
-|[ ]`tdigest` |     | [ ] |     |[ ] TDigest |     |
-|[ ]`variance`|     |  ✓  |     |[ ] Variance|     |
+## TDR
 
+I named the data frame representation style in the model above as TDR (Transposed DataFrame Representation). 
 
-Options can be used as follows.
-See the [document of C++ function](https://arrow.apache.org/docs/cpp/compute.html) for detail.
+This library can be used with both TDR mode and usual Table mode.
+If you set the environment variable `RED_AMBER_OUTPUT_MODE` to `"table"`, output style by `inspect` and `to_iruby` is the Table mode. Other value including nil will output TDR style.
 
+You can switch the mode in Ruby like this.
 ```ruby
-double = RedAmber::Vector.new([1, 0/0.0, -1/0.0, 1/0.0, nil, ""])
-#=>
-#<RedAmber::Vector(:double, size=6):0x000000000000f910>
-[1.0, NaN, -Infinity, Infinity, nil, 0.0]
-
-double.count #=> 5
-double.count(opts: {mode: :only_valid}) #=> 5, default
-double.count(opts: {mode: :only_null}) #=> 1
-double.count(opts: {mode: :all}) #=> 6
-
-boolean = RedAmber::Vector.new([true, true, nil])
-#=>
-#<RedAmber::Vector(:boolean, size=3):0x000000000000f924>
-[true, true, nil]
-
-boolean.all #=> true
-boolean.all(opts: {skip_nulls: true}) #=> true
-boolean.all(opts: {skip_nulls: false}) #=> false
+ENV['RED_AMBER_OUTPUT_STYLE'] = 'table' # => Table mode
 ```
 
-#### Unary element-wise: vector.func => vector
-
-| Method    |Boolean|Numeric|String|Options|Remarks|
-| ------------ | --- | --- | --- | --- | ----- |
-| ✓ `-@`       |     |  ✓  |     |     |as `-vector`|
-| ✓ `negate`   |     |  ✓  |     |     |`-@`   |
-| ✓ `abs`      |     |  ✓  |     |     |       |
-|[ ]`acos`     |     | [ ] |     |     |       |
-|[ ]`asin`     |     | [ ] |     |     |       |
-| ✓ `atan`     |     |  ✓  |     |     |       |
-| ✓ `bit_wise_not`|  | (✓) |     |     |integer only|
-|[ ]`ceil`     |     |  ✓  |     |     |       |
-| ✓ `cos`      |     |  ✓  |     |     |       |
-|[ ]`floor`    |     |  ✓  |     |     |       |
-| ✓ `invert`   |  ✓  |     |     |     |`!`, alias `not`|
-|[ ]`ln`       |     | [ ] |     |     |       |
-|[ ]`log10`    |     | [ ] |     |     |       |
-|[ ]`log1p`    |     | [ ] |     |     |       |
-|[ ]`log2`     |     | [ ] |     |     |       |
-|[ ]`round`    |     | [ ] |     |[ ] Round|       |
-|[ ]`round_to_multiple`| | [ ] | |[ ] RoundToMultiple|       |
-| ✓ `sign`     |     |  ✓  |     |     |       |
-| ✓ `sin`      |     |  ✓  |     |     |       |
-| ✓ `tan`      |     |  ✓  |     |     |       |
-|[ ]`trunc`    |     |  ✓  |     |     |       |
-
-#### Binary element-wise: vector.func(vector) => vector
-
-| Method       |Boolean|Numeric|String|Options|Remarks|
-| ----------------- | --- | --- | --- | --- | ----- |
-| ✓ `add`           |     |  ✓  |     |     | `+`   |
-| ✓ `atan2`         |     |  ✓  |     |     |       |
-| ✓ `and_kleene`    |  ✓  |     |     |     | `&`   |
-| ✓ `and_org   `    |  ✓  |     |     |     |`and` in Red Arrow|
-| ✓ `and_not`       |  ✓  |     |     |     |       |
-| ✓ `and_not_kleene`|  ✓  |     |     |     |       |
-| ✓ `bit_wise_and`  |     | (✓) |     |     |integer only|
-| ✓ `bit_wise_or`   |     | (✓) |     |     |integer only|
-| ✓ `bit_wise_xor`  |     | (✓) |     |     |integer only|
-| ✓ `divide`        |     |  ✓  |     |     | `/`   |
-| ✓ `equal`         |  ✓  |  ✓  |  ✓  |     |`==`, alias `eq`|
-| ✓ `greater`       |  ✓  |  ✓  |  ✓  |     |`>`, alias `gt`|
-| ✓ `greater_equal` |  ✓  |  ✓  |  ✓  |     |`>=`, alias `ge`|
-| ✓ `is_finite`     |     |  ✓  |     |     |       |
-| ✓ `is_inf`        |     |  ✓  |     |     |       |
-| ✓ `is_na`         |  ✓  |  ✓  |  ✓  |     |       |
-| ✓ `is_nan`        |     |  ✓  |     |     |       |
-|[ ]`is_nil`        |  ✓  |  ✓  |  ✓  |[ ] Null|alias `is_null`|
-| ✓ `is_valid`      |  ✓  |  ✓  |  ✓  |     |       |
-| ✓ `less`          |  ✓  |  ✓  |  ✓  |     |`<`, alias `lt`|
-| ✓ `less_equal`    |  ✓  |  ✓  |  ✓  |     |`<=`, alias `le`|
-|[ ]`logb`          |     | [ ] |     |     |       |
-|[ ]`mod`           |     | [ ] |     |     | `%`   |
-| ✓ `multiply`      |     |  ✓  |     |     | `*`   |
-| ✓ `not_equal`     |  ✓  |  ✓  |  ✓  |     |`!=`, alias `ne`|
-| ✓ `or_kleene`     |  ✓  |     |     |     | `\|`  |
-| ✓ `or_org`        |  ✓  |     |     |     |`or` in Red Arrow|
-| ✓ `power`         |     |  ✓  |     |     | `**`  |
-| ✓ `subtract`      |     |  ✓  |     |     | `-`   |
-| ✓ `shift_left`    |     | (✓) |     |     |`<<`, integer only|
-| ✓ `shift_right`   |     | (✓) |     |     |`>>`, integer only|
-| ✓ `xor`           |  ✓  |     |     |     | `^`   |
-
-##### (Not impremented)
-- [ ] sort, sort_index
-- [ ] argmin, argmax
-- [ ] (array functions)
-- [ ] (strings functions)
-- [ ] (temporal functions)
-- [ ] (conditional functions)
-- [ ] (index functions)
-- [ ] (other functions)
-
-### Coerce (not impremented)
-
-### Updating (not impremented)
-
-### DSL in a block for faster calculation ?
-
+For more detail information about TDR, see [TDR.md](doc/tdr.md).
 
 ## Development
 

data/benchmark/csv_load_penguins.yml

@@ -0,0 +1,15 @@
+prelude: |
+  require 'datasets-arrow'
+  require 'rover'
+  require 'red_amber'
+
+  penguins_csv = 'benchmark/cache/penguins.csv'
+
+  unless File.exist?(penguins_csv)
+    arrow = Datasets::Penguins.new.to_arrow
+    RedAmber::DataFrame.new(arrow).save(penguins_csv)
+  end
+
+benchmark:
+  'penguins by Rover': Rover.read_csv(penguins_csv)
+  'penguins by RedAmber': RedAmber::DataFrame.load(penguins_csv)

data/benchmark/drop_nil.yml

@@ -0,0 +1,11 @@
+prelude: |
+  require 'datasets-arrow'
+  require 'red_amber'
+
+  penguins = RedAmber::DataFrame.new(Datasets::Penguins.new.to_arrow)
+
+  def drop_nil(penguins)
+    penguins.remove { vectors.map { |v| v.is_nil} }
+  end
+
+benchmark: drop_nil(penguins)