red_amber 0.1.2 → 0.1.5

data/README.md

@@ -3,18 +3,27 @@
 A simple dataframe library for Ruby (experimental)
 
 - Powered by [Red Arrow](https://github.com/apache/arrow/tree/master/ruby/red-arrow)
-- Simple API similar to [Rover-df](https://github.com/ankane/rover)
+- Inspired by the dataframe library [Rover-df](https://github.com/ankane/rover)
 
 ## Requirements
 
 ```ruby
-gem 'red-arrow',   '>= 7.0.0'
-gem 'red-parquet', '>= 7.0.0' # if you use IO from/to parquet
+gem 'red-arrow',   '>= 8.0.0'
+gem 'red-parquet', '>= 8.0.0' # if you use IO from/to parquet
 gem 'rover-df',    '~> 0.3.0' # if you use IO from/to Rover::DataFrame
 ```
 
 ## Installation
 
+Install requirements before you install Red Amber.
+
+- Apache Arrow GLib (>= 8.0.0)
+- Apache Parquet GLib (>= 8.0.0)
+
+  See [Apache Arrow install document](https://arrow.apache.org/install/).
+  
+  Minimum installation example for the latest Ubuntu is in the ['Prepare the Apache Arrow' section in ci test](https://github.com/heronshoes/red_amber/blob/master/.github/workflows/test.yml) of Red Amber.
+
 Add this line to your Gemfile:
 
 ```ruby
@@ -23,339 +32,116 @@ gem 'red_amber'
 
 And then execute:
 
-    $ bundle install
+```shell
+bundle install
+```
 
 Or install it yourself as:
 
-    $ gem install red_amber
+```shell
+gem install red_amber
+```
 
 ## `RedAmber::DataFrame`
 
-### Constructors and saving
-
-- [x] `new` from a columnar Hash
-  - `RedAmber::DataFrame.new(x: [1, 2, 3])`
-
-- [x] `new` from a schema (by Hash) and rows (by Array)
-  - `RedAmber::DataFrame.new({:x=>:uint8}, [[1], [2], [3]])`
-
-- [x] `new` from an Arrow::Table
-  - `RedAmber::DataFrame.new(Arrow::Table.new(x: [1, 2, 3]))`
-
-- [x] `new` from a Rover::DataFrame
-  - `RedAmber::DataFrame.new(Rover::DataFrame.new(x: [1, 2, 3]))`
-
-- [ ] `load` (class method)
-
-     - [x] from a [`.arrow`, `.arrows`, `.csv`, `.csv.gz`, `.tsv`] file
-       - `RedAmber::DataFrame.load("test/entity/with_header.csv")`
-
-     - [x] from a string buffer
-
-     - [x] from a URI
-       - `RedAmber::DataFrame.load(URI("https://github.com/heronshoes/red_amber/blob/master/test/entity/with_header.csv"))`
-
-     - [ ] from a parquet file
-
-- [ ] `save` (instance method)
-
-     - [x] to a [`.arrow`, `.arrows`, `.csv`, `.csv.gz`, `.tsv`] file
-
-     - [x] to a string buffer
-
-     - [x] to a URI
-
-     - [ ] to a parquet file
-
-### Properties
-
-- [x] `table`
-
-  Reader of Arrow::Table object inside.
-
-- [x] `n_rows`, `nrow`, `size`, `length`
-  
-  Returns num of rows (data size).
- 
-- [x] `n_columns`, `ncol`, `width`
-  
-  Returns num of columns (num of vectors).
- 
-- [x] `shape`
- 
-  Returns shape in an Array[n_rows, n_cols].
- 
-- [x] `column_names`, `keys`
-  
-  Returns num of column names by an Array.
-
-- [x] `types`
-  
-  Returns types of columns by an Array of Symbols.
-
-- [x] `data_types`
-
-  Returns types of columns by an Array of `Arrow::DataType`.
-
-- [x] `vectors`
-
-  Returns an Array of Vectors.
-
-- [x] `to_h`
-
-  Returns column-oriented data in a Hash.
-
-- [x] `to_a`, `raw_records`
+Represents a set of data in 2D-shape.
 
-  Returns an array of row-oriented data without header. If you need a column-oriented full array, use `.to_h.to_a`
-
-- [x] `schema`
-
-  Returns column name and data type in a Hash.
-
-- [x] `==`
- 
-- [x] `empty?`
-
-### Output
-
-- [x] `to_s`
-
-- [ ] summary, describe
-
-- [x] `to_rover`
+```ruby
+require 'red_amber'
+require 'datasets-arrow'
 
-  Returns a `Rover::DataFrame`.
+arrow = Datasets::Penguins.new.to_arrow
+penguins = RedAmber::DataFrame.new(arrow)
+penguins.tdr
+# =>
+RedAmber::DataFrame : 344 x 8 Vectors
+Vectors : 5 numeric, 3 strings
+# key                type   level data_preview
+1 :species           string     3 {"Adelie"=>152, "Chinstrap"=>68, "Gentoo"=>124}
+2 :island            string     3 {"Torgersen"=>52, "Biscoe"=>168, "Dream"=>124}
+3 :bill_length_mm    double   165 [39.1, 39.5, 40.3, nil, 36.7, ... ], 2 nils
+4 :bill_depth_mm     double    81 [18.7, 17.4, 18.0, nil, 19.3, ... ], 2 nils
+5 :flipper_length_mm uint8     56 [181, 186, 195, nil, 193, ... ], 2 nils
+6 :body_mass_g       uint16    95 [3750, 3800, 3250, nil, 3450, ... ], 2 nils
+7 :sex               string     3 {"male"=>168, "female"=>165, nil=>11}
+8 :year              uint16     3 {2007=>110, 2008=>114, 2009=>120}
+```
 
-- [x] `inspect(tally_level: 5, max_element: 5)`
+### DataFrame model
+![dataframe model of RedAmber](doc/image/dataframe_model.png)
 
-  Shows some information about self.
+For example, `DataFrame#pick` accepts keys as an argument and returns a sub DataFrame.
 
 ```ruby
-hash = {a: [1, 2, 3], b: %w[A B C], c: [1.0, 2, 3]}
-RedAmber::DataFrame.new(hash)
+df = penguins.pick(:body_mass_g)
 # =>
-RedAmber::DataFrame : 3 observations(rows) of 3 variables(columns)
-Variables : 2 numeric, 1 string
-# key type   level data_preview
-1 :a  uint8      3 [1, 2, 3]
-2 :b  string     3 [A, B, C]
-3 :c  double     3 [1.0, 2.0, 3.0]
+#<RedAmber::DataFrame : 344 x 1 Vector, 0x000000000000fa14>
+Vector : 1 numeric
+# key          type  level data_preview
+1 :body_mass_g int64    95 [3750, 3800, 3250, nil, 3450, ... ], 2 nils
 ```
 
-  - tally_level: max level to use tally mode
-  - max_element: max num of element to show values in each row
+`DataFrame#assign` creates new variables (column in the table).
 
-### Selecting
-
-- [x] Select columns by `[]` as `[key]`, `[keys]`, `[keys[index]]`
-  - Key in a Symbol: `df[:symbol]`
-  - Key in a String: `df["string"]`
-  - Keys in an Array: `df[:symbol1`, `"string"`, `:symbol2`
-  - Keys in indeces: `df[df.keys[0]`, `df[df.keys[1,2]]`, `df[df.keys[1..]]`
-  - Keys in a Range:
-    A end-less Range can be used to represent keys.
 ```ruby
-hash = {a: [1, 2, 3], b: %w[A B C], c: [1.0, 2, 3]}
-df = RedAmber::DataFrame.new(hash)
-df[:b..:c, "a"]
+df.assign(:body_mass_kg => df[:body_mass_g] / 1000.0)
 # =>
-RedAmber::DataFrame : 3 observations(rows) of 3 variables(columns)
-Variables : 2 numeric, 1 string
-# key type   level data_preview
-1 :b  string     3 [A, B, C]
-2 :c  double     3 [1.0, 2.0, 3.0]
-3 :a  uint8      3 [1, 2, 3]
+#<RedAmber::DataFrame : 344 x 2 Vectors, 0x000000000000fa28>
+Vectors : 2 numeric
+# key           type   level data_preview
+1 :body_mass_g  int64     95 [3750, 3800, 3250, nil, 3450, ... ], 2 nils
+2 :body_mass_kg double    95 [3.75, 3.8, 3.25, nil, 3.45, ... ], 2 nils
 ```
 
-- [x] Select rows by `[]` as `[index]`, `[range]`, `[array]`
-  - Select a row by index: `df[0]`
-  - Select rows by indeces in a Range: `df[1..2]`
-  - Select rows by indeces in an Array: `df[1, 2]`
-  - Mixed case: `df[2, 0..]`
-
-- [x] Select rows from top or bottom
-
-  `head(n=5)`, `tail(n=5)`, `first(n=1)`, `last(n=1)`
-
-- [ ] slice
-
-### Updating
-
-- [ ] Add a new column
-
-- [ ] Update a single element
-
-- [ ] Update multiple elements
-
-- [ ] Update all elements
+DataFrame manipulating methods like `pick`, `drop`, `slice`, `remove`, `rename` and `assign` accept a block.
 
-- [ ] Update elements matching a condition
+This is an exaple to eliminate observations (row in the table) containing nil.
 
-- [ ] Clamp
-
-- [ ] Delete columns
-
-- [ ] Rename a column
-
-- [ ] Sort rows
-
-- [ ] Clear data
-
-### Treat na data
-
-- [ ] Drop na (NaN, nil)
-
-- [ ] Replace na with value
-
-- [ ] Interpolate na with convolution array
-
-### Combining DataFrames
+```ruby
+# remove all observation contains nil
+nil_removed = penguins.remove { vectors.map(&:is_nil).reduce(&:|) }
+nil_removed.tdr
+# =>
+RedAmber::DataFrame : 342 x 8 Vectors
+Vectors : 5 numeric, 3 strings
+# key                type   level data_preview
+1 :species           string     3 {"Adelie"=>151, "Chinstrap"=>68, "Gentoo"=>123}
+2 :island            string     3 {"Torgersen"=>51, "Biscoe"=>167, "Dream"=>124}
+3 :bill_length_mm    double   164 [39.1, 39.5, 40.3, 36.7, 39.3, ... ]
+4 :bill_depth_mm     double    80 [18.7, 17.4, 18.0, 19.3, 20.6, ... ]
+5 :flipper_length_mm int64     55 [181, 186, 195, 193, 190, ... ]
+6 :body_mass_g       int64     94 [3750, 3800, 3250, 3450, 3650, ... ]
+7 :sex               string     3 {"male"=>168, "female"=>165, ""=>9}
+8 :year              int64      3 {2007=>109, 2008=>114, 2009=>119}
+```
 
-- [ ] Add rows
+For this frequently needed task, we can do it much simpler.
 
-- [ ] Add columns
+```ruby
+penguins.remove_nil # => same result as above
+```
 
-- [ ] Inner join
+See [DataFrame.md](doc/DataFrame.md) for details.
 
-- [ ] Left join
 
-### Encoding
+## `RedAmber::Vector`
 
-- [ ] One-hot encoding
+Class `RedAmber::Vector` represents a series of data in the DataFrame.
 
-### Iteration (not impremented)
+```ruby
+penguins[:bill_length_mm]
+# =>
+#<RedAmber::Vector(:double, size=344):0x000000000000f8fc>
+[39.1, 39.5, 40.3, nil, 36.7, 39.3, 38.9, 39.2, 34.1, 42.0, 37.8, 37.8, 41.1, ... ]
+```
 
-### Filtering (not impremented)
+Vectors accepts some [functional methods from Arrow](https://arrow.apache.org/docs/cpp/compute.html).
 
+See [Vector.md](doc/Vector.md) for details.
 
-## `RedAmber::Vector`
-### Constructor
-
-- [x] Create from a column in a DataFrame
-
-- [x] New from an Array
-
-### Properties
-
-- [x] `to_s`
-
-- [x] `values`, `to_a`, `entries`
-
-- [x] `size`, `length`, `n_rows`, `nrow`
-
-- [x] `type`
-
-- [x] `data_type`
-
-- [ ] `each`
-
-- [ ] `chunked?`
-
-- [ ] `n_chunks`
-
-- [ ] `each_chunk`
-
-- [x] `tally`
-
-- [ ] `n_nulls`
-
-### Functions
-#### Unary aggregations: vector.func => Scalar
-
-| Method    |Boolean|Numeric|String|Remarks|
-| ------------ | --- | --- | --- | ----- |
-|[x] `all`     | [x] |     |     |       |
-|[x] `any`     | [x] |     |     |       |
-|[x] `approximate_median`| | [x] |     |     |
-|[x] `count`         | [x] | [x] | [x] |     |
-|[x] `count_distinct`| [x] | [x] | [x] |     |
-|[x] `count_uniq`    | [x] | [x] | [x] |an alias of `count_distinct`|
-|[ ] `index`   |     |     |     |       |
-|[x] `max`     | [x] | [x] | [x] |       |
-|[x] `mean`    | [x] | [x] |     |       |
-|[x] `min`     | [x] | [x] | [x] |       |
-|[ ] `min_max` |     |     |     |       |
-|[ ] `mode`    |     |     |     |       |
-|[x] `product` | [x] | [x] |     |       |
-|[ ] `quantile`|     |     |     |       |
-|[x] `stddev`  |     | [x] |     |       |
-|[x] `sum`     | [x] | [x] |     |       |
-|[ ] `tdigest` |     |     |     |       |
-|[x] `variance`|     | [x] |     |       |
-
-#### Unary element-wise: vector.func => Vector
-
-| Method    |Boolean|Numeric|String|Remarks|
-| ------------ | --- | --- | --- | ----- |
-|[x] `-@`      |     | [x] |     |as `-vector`|
-|[x] `negate`  |     | [x] |     |`-@`   |
-|[x] `abs`     |     | [x] |     |       |
-|[ ] `acos`    |     | [ ] |     |       |
-|[ ] `asin`    |     | [ ] |     |       |
-|[x] `atan`    |     | [x] |     |       |
-|[ ] `ceil`    |     | [x] |     |       |
-|[x] `cos`     |     | [x] |     |       |
-|[ ] `floor`   |     | [x] |     |       |
-|[ ] `ln`      |     | [ ] |     |       |
-|[ ] `log10`   |     | [ ] |     |       |
-|[ ] `log1p`   |     | [ ] |     |       |
-|[ ] `log2`    |     | [ ] |     |       |
-|[x] `sign`    |     | [x] |     |       |
-|[x] `sin`     |     | [x] |     |       |
-|[x] `tan`     |     | [x] |     |       |
-|[ ] `trunc`   |     | [x] |     |       |
-
-#### Binary element-wise: vector.func(vector) => Vector
-
-| Method          |Boolean|Numeric|String|Remarks|
-| ------------------ | --- | --- | --- | ----- |
-|[x] `add`           |     | [x] |     | `+`   |
-|[x] `atan2`         |     | [x] |     |       |
-|[x] `and`           | [x] |     |     |       |
-|[x] `and_kleene`    | [x] |     |     |       |
-|[x] `and_not`       | [x] |     |     |       |
-|[x] `and_not_kleene`| [x] |     |     |       |
-|[x] `bit_wise_and`  |     |([x])|     |`&`, integer only|
-|[ ] `bit_wise_not`  |     |([x])|     |`!`, integer only|
-|[x] `bit_wise_or`   |     |([x])|     |`|`, integer only|
-|[x] `bit_wise_xor`  |     |([x])|     |`^`, integer only|
-|[x] `divide`        |     | [x] |     | `/`   |
-|[x] `equal`         | [x] | [x] | [x] |`==`, alias `eq`|
-|[x] `greater`       | [x] | [x] | [x] |`>`, alias `gt`|
-|[x] `greater_equal` | [x] | [x] | [x] |`>=`, alias `ge`|
-|[x] `less`          | [x] | [x] | [x] |`<`, alias `lt`|
-|[x] `less_equal`    | [x] | [x] | [x] |`<=`, alias `le`|
-|[ ] `logb`          |     | [ ] |     |       |
-|[ ] `mod`           |     | [ ] |     |       |
-|[x] `multiply`      |     | [x] |     | `*`   |
-|[x] `not_equal`     | [x] | [x] | [x] |`!=`, alias `ne`|
-|[x] `or`            | [x] |     |     |       |
-|[x] `or_kleene`     | [x] |     |     |       |
-|[x] `power`         |     | [x] |     | `**`  |
-|[x] `subtract`      |     | [x] |     | `-`   |
-|[x] `shift_left`    |     |([x])|     |`<<`, integer only|
-|[x] `shift_right`   |     |([x])|     |`>>`, integer only|
-|[x] `xor`           | [x] |     |     |       |
-
-##### (Not impremented)
-- [ ] invert, round, round_to_multiple
-- [ ] sort, sort_index
-- [ ] minmax, var, median, quantile
-- [ ] argmin, argmax
-- [ ] (array functions)
-- [ ] (strings functions)
-- [ ] (temporal functions)
-- [ ] (conditional functions)
-- [ ] (index functions)
-- [ ] (other functions)
-
-### Coerce (not impremented)
-
-### Updating (not impremented)
-
-### DSL in a block for faster calculation ?
+## TDR concept
 
+I named the data frame representation style in the model above as TDR (Transposed DataFrame Representation). See [TDR.md](doc/tdr.md) for details.
 
 ## Development
 

data/benchmark/csv_load_penguins.yml

@@ -0,0 +1,15 @@
+prelude: |
+  require 'datasets-arrow'
+  require 'rover'
+  require 'red_amber'
+
+  penguins_csv = 'benchmark/cache/penguins.csv'
+
+  unless File.exist?(penguins_csv)
+    arrow = Datasets::Penguins.new.to_arrow
+    RedAmber::DataFrame.new(arrow).save(penguins_csv)
+  end
+
+benchmark:
+  'penguins by Rover': Rover.read_csv(penguins_csv)
+  'penguins by RedAmber': RedAmber::DataFrame.load(penguins_csv)

data/benchmark/drop_nil.yml

@@ -0,0 +1,11 @@
+prelude: |
+  require 'datasets-arrow'
+  require 'red_amber'
+
+  penguins = RedAmber::DataFrame.new(Datasets::Penguins.new.to_arrow)
+
+  def drop_nil(penguins)
+    penguins.remove { vectors.map { |v| v.is_nil} }
+  end
+
+benchmark: drop_nil(penguins)