red_amber 0.1.6 → 0.1.7

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ae6a6696e0f01ae7d621d11542e203803ba117fc1ee3d286a1444b3c4ac746fc
-  data.tar.gz: 722d4ad538fe4f0c85db4911e773e1f87eb03f47fa63c954529bf04babc55d8c
+  metadata.gz: 88bdd603d8daec1a95c0277ef68857f84346ad7cf95d0ba23a306e6b70567c29
+  data.tar.gz: 40add80cbaa5183ca0e93eadcdcd1fead37015cac1cb2360660002c0b1878255
 SHA512:
-  metadata.gz: 96887abfbdd44330e80a6a97f91597c00706fc99492d086683702f1d3e757331e90fb275e5796a2b7b3228b476f8c7799ab22727411baedb79ae39acafd2d3f0
-  data.tar.gz: c020bba60734fccdeb4a18efecb98260f70fa76c5b8ba2c7f2830d2dac6de66e9776a1fa2ffc5d396e7270b29c5e4dd9737dee70b5dcc47dd3113aaafe4f4d22
+  metadata.gz: d043eea51117ecc48bdc52fa951e24d2618f273eb289a30f5bbb182e1a891763cdd35f6a7c6764f6e0061bddeaaa86b2374de1dc2b48f25a5b6b05c9af83a0e3
+  data.tar.gz: cdbba19750bf71fe99e55bf6c46cb4522018f43563d7a93fdc375987f9388234e4f7e833297fdb6b8dd5a41b5a1bfdbf287ea47663f5f8a90facb56a4c63daef

data/.rubocop.yml

@@ -80,6 +80,7 @@ Metrics/CyclomaticComplexity:
   Exclude:
     - 'lib/red_amber/data_frame_selectable.rb' # Max: 14
     - 'lib/red_amber/vector_updatable.rb' # Max: 14
+    - 'lib/red_amber/data_frame_displayable.rb' # Max: 18
 
 # Max: 10
 Metrics/MethodLength:
@@ -93,6 +94,7 @@ Metrics/ModuleLength:
   Exclude:
     - 'lib/red_amber/data_frame_selectable.rb' # Max: 141
     - 'lib/red_amber/vector_functions.rb' # Max: 114
+    - 'lib/red_amber/data_frame_displayable.rb' # Max: 132
 
 # Max: 8
 Metrics/PerceivedComplexity:
@@ -100,6 +102,7 @@ Metrics/PerceivedComplexity:
   Exclude:
     - 'lib/red_amber/data_frame_selectable.rb' # Max: 14
     - 'lib/red_amber/vector_updatable.rb' # Max: 15
+    - 'lib/red_amber/data_frame_displayable.rb' # Max: 19
 
 Naming/FileName:
   Exclude:

data/CHANGELOG.md

@@ -1,29 +1,55 @@
-## - unreleased
+## [0.1.9] - Unreleased
 
-- Document
-  - YARD support
+- Supports Arrow 9.0.0
 
-- `datasets-red-amber` gem
-- `red-amber` gem
+## [0.1.7] - 2022-07-15 (experimental)
 
-- `Vector#divmod`
-  - Introduce if Arrow's function is ready
+- Bug fixes
+
+  - Remove development dependency for red-dataset-arrow (#47)
+    - To avoid irregular fails in CI test
+    - Add red-datasets to development dependency instead (#49)
 
-## - Unreleased, will be after Arrow 9.0.0 released
+  - Supress useless log in tests (#46)
+    Suppress log of Webrick and iruby.
 
-- `DataFrame`
-  - Introduce `summary` or ``describe`
-    - `Quantile` will be available
+- New features and improvements
+
+  - Use Table mode as default preview mode in `inspect`/`to_s` (#40)
+    - Show examples in documents in Table
+    - Use the word rows/columns
+    - Update images of data processing in Table style
+
+  - Introduce a new Table formatter (#47)
+    - Migrate from the Arrow's formatter
+      - Do not use TAB, format by spaces only.
+      - Align column width with head rows and tail rows.
+      - Show nils.
+      - Show data types.
+    - Refine documents to use new formatter output
+  
+  - Simplify options of Vector functions (#46)
+    Vector functions with options use optional argument opt in previous code.
+
+  - Add `#float?`, `#integer?` to Vector (#46)
+  - Add `#each` to Vector (#47)
+
+  - Introduce class `Group` (#48)
+    - Refine `DataFrame#group` to use class Group
+    - Add methods to Group
+
+  - Move parquet and rover to development dependency (#49)
+  
+  - Refine text in `DataFrame#to_iruby` (#40)
 
-## [0.1.7] - Unreleased, may be 2022-07-10
+  - Add badges in Github site
+    - Gitter badge for Red Data Tools (#42)
+    - Gem version and CI status badge (#45)
 
-- Feedback something to Red Data Tools
-- Support more functions
-- Improve as more performant
-- More examples of frequently needed tasks
+  - Exchange containers in red-amber.rb and red_amber.rb (#47)
+    - Mainly use red_amber by consistency with the folder name
 
-- New `Group` API
-- `DataFrame#join features
+  - Add Jupyter notebook '47 Examples of Red Amber' (#49)
 
 ## [0.1.6] - 2022-06-26 (experimental)
 

data/Gemfile

@@ -7,6 +7,9 @@ gemspec
 group :test do
   gem 'rake'
 
+  gem 'red-parquet', '>= 8.0.0'
+  gem 'rover-df', '~> 0.3.0'
+
   gem 'rubocop'
   gem 'rubocop-performance', require: false
   gem 'rubocop-rake'
@@ -17,5 +20,5 @@ group :test do
   gem 'webrick'
 
   gem 'benchmark_driver'
-  gem 'red-datasets-arrow'
+  gem 'red-datasets'
 end

data/README.md

@@ -1,16 +1,20 @@
 # RedAmber
 
+[![Gem Version](https://badge.fury.io/rb/red_amber.svg)](https://badge.fury.io/rb/red_amber)
+[![Ruby](https://github.com/heronshoes/red_amber/actions/workflows/test.yml/badge.svg)](https://github.com/heronshoes/red_amber/actions/workflows/test.yml)
+
 A simple dataframe library for Ruby (experimental).
 
-- Powered by [Red Arrow](https://github.com/apache/arrow/tree/master/ruby/red-arrow)
+- Powered by [Red Arrow](https://github.com/apache/arrow/tree/master/ruby/red-arrow) [![Gitter Chat](https://badges.gitter.im/red-data-tools/en.svg)](https://gitter.im/red-data-tools/en)
 - Inspired by the dataframe library [Rover-df](https://github.com/ankane/rover)
 
 ## Requirements
 
 ```ruby
 gem 'red-arrow',   '>= 8.0.0'
-gem 'red-parquet', '>= 8.0.0' # if you use IO from/to parquet
-gem 'rover-df',    '~> 0.3.0' # if you use IO from/to Rover::DataFrame
+
+gem 'red-parquet', '>= 8.0.0' # Optional, if you use IO from/to parquet
+gem 'rover-df',    '~> 0.3.0' # Optional, if you use IO from/to Rover::DataFrame
 ```
 
 ## Installation
@@ -18,7 +22,8 @@ gem 'rover-df',    '~> 0.3.0' # if you use IO from/to Rover::DataFrame
 Install requirements before you install Red Amber.
 
 - Apache Arrow GLib (>= 8.0.0)
-- Apache Parquet GLib (>= 8.0.0)
+
+- Apache Parquet GLib (>= 8.0.0)  # If you use IO from/to parquet
 
   See [Apache Arrow install document](https://arrow.apache.org/install/).
   
@@ -42,11 +47,6 @@ Or install it yourself as:
 gem install red_amber
 ```
 
-(From v0.1.6)
-
-RedAmber uses TDR mode for `#inspect` and `#to_iruby` by default. If you prefer Table mode, please set the environment variable
-`RED_AMBER_OUTPUT_MODE` to `"table"`. See [TDR section](#TDR) for detail.
-
 ## `RedAmber::DataFrame`
 
 Represents a set of data in 2D-shape. The entity is a Red Arrow's Table object. 
@@ -56,54 +56,21 @@ require 'red_amber' # require 'red-amber' is also OK.
 require 'datasets-arrow'
 
 arrow = Datasets::Penguins.new.to_arrow
-penguins = RedAmber::DataFrame.new(arrow)
-penguins.table
-
-# =>
-#<Arrow::Table:0x111271098 ptr=0x7f9118b3e0b0>
-	species	island	bill_length_mm	bill_depth_mm	flipper_length_mm	body_mass_g	sex	year
-  0	Adelie 	Torgersen	     39.100000	    18.700000	              181	       3750	male	2007
-  1	Adelie 	Torgersen	     39.500000	    17.400000	              186	       3800	female	2007
-  2	Adelie 	Torgersen	     40.300000	    18.000000	              195	       3250	female	2007
-  3	Adelie 	Torgersen	        (null)	       (null)	           (null)	     (null)	(null)	2007
-  4	Adelie 	Torgersen	     36.700000	    19.300000	              193	       3450	female	2007
-  5	Adelie 	Torgersen	     39.300000	    20.600000	              190	       3650	male	2007
-  6	Adelie 	Torgersen	     38.900000	    17.800000	              181	       3625	female	2007
-  7	Adelie 	Torgersen	     39.200000	    19.600000	              195	       4675	male	2007
-  8	Adelie 	Torgersen	     34.100000	    18.100000	              193	       3475	(null)	2007
-  9	Adelie 	Torgersen	     42.000000	    20.200000	              190	       4250	(null)	2007
-...
-334	Gentoo 	Biscoe	     46.200000	    14.100000	              217	       4375	female	2009
-335	Gentoo 	Biscoe	     55.100000	    16.000000	              230	       5850	male	2009
-336	Gentoo 	Biscoe	     44.500000	    15.700000	              217	       4875	(null)	2009
-337	Gentoo 	Biscoe	     48.800000	    16.200000	              222	       6000	male	2009
-338	Gentoo 	Biscoe	     47.200000	    13.700000	              214	       4925	female	2009
-339	Gentoo 	Biscoe	        (null)	       (null)	           (null)	     (null)	(null)	2009
-340	Gentoo 	Biscoe	     46.800000	    14.300000	              215	       4850	female	2009
-341	Gentoo 	Biscoe	     50.400000	    15.700000	              222	       5750	male	2009
-342	Gentoo 	Biscoe	     45.200000	    14.800000	              212	       5200	female	2009
-343	Gentoo 	Biscoe	     49.900000	    16.100000	              213	       5400	male	2009
-```
-
-By default, RedAmber shows self by compact transposed style. This unfamiliar style (TDR) is designed for
-the exploratory data processing. It keeps Vectors as row vectors, shows keys and types at a glance, shows levels 
-for the 'factor-like' variables and shows the number of abnormal values like NaN and nil.
-
-```ruby
-penguins
+RedAmber::DataFrame.new(arrow)
 
 # =>
-RedAmber::DataFrame : 344 x 8 Vectors
-Vectors : 5 numeric, 3 strings
-# key                type   level data_preview
-1 :species           string     3 {"Adelie"=>152, "Chinstrap"=>68, "Gentoo"=>124}
-2 :island            string     3 {"Torgersen"=>52, "Biscoe"=>168, "Dream"=>124}
-3 :bill_length_mm    double   165 [39.1, 39.5, 40.3, nil, 36.7, ... ], 2 nils
-4 :bill_depth_mm     double    81 [18.7, 17.4, 18.0, nil, 19.3, ... ], 2 nils
-5 :flipper_length_mm uint8     56 [181, 186, 195, nil, 193, ... ], 2 nils
-6 :body_mass_g       uint16    95 [3750, 3800, 3250, nil, 3450, ... ], 2 nils
-7 :sex               string     3 {"male"=>168, "female"=>165, nil=>11}
-8 :year              uint16     3 {2007=>110, 2008=>114, 2009=>120}
+#<RedAmber::DataFrame : 344 x 8 Vectors, 0x0000000000013790>
+    species  island    bill_length_mm bill_depth_mm flipper_length_mm ...     year
+    <string> <string>        <double>      <double>           <uint8> ... <uint16>
+  1 Adelie   Torgersen           39.1          18.7               181 ...     2007
+  2 Adelie   Torgersen           39.5          17.4               186 ...     2007
+  3 Adelie   Torgersen           40.3          18.0               195 ...     2007
+  4 Adelie   Torgersen          (nil)         (nil)             (nil) ...     2007
+  5 Adelie   Torgersen           36.7          19.3               193 ...     2007
+  : :        :                      :             :                 : ...        :
+342 Gentoo   Biscoe              50.4          15.7               222 ...     2009
+343 Gentoo   Biscoe              45.2          14.8               212 ...     2009
+344 Gentoo   Biscoe              49.9          16.1               213 ...     2009
 ```
 
 ### DataFrame model
@@ -113,23 +80,41 @@ For example, `DataFrame#pick` accepts keys as an argument and returns a sub Data
 
 ```ruby
 df = penguins.pick(:body_mass_g)
+df
+
 # =>
-#<RedAmber::DataFrame : 344 x 1 Vector, 0x000000000000fa14>
-Vector : 1 numeric
-# key          type  level data_preview
-1 :body_mass_g int64    95 [3750, 3800, 3250, nil, 3450, ... ], 2 nils
+#<RedAmber::DataFrame : 344 x 1 Vector, 0x0000000000015cc0>
+    body_mass_g
+       <uint16>
+  1        3750
+  2        3800
+  3        3250
+  4       (nil)
+  5        3450
+  :           :
+342        5750
+343        5200
+344        5400
 ```
 
 `DataFrame#assign` creates new variables (column in the table).
 
 ```ruby
 df.assign(:body_mass_kg => df[:body_mass_g] / 1000.0)
+
 # =>
-#<RedAmber::DataFrame : 344 x 2 Vectors, 0x000000000000fa28>
-Vectors : 2 numeric
-# key           type   level data_preview
-1 :body_mass_g  int64     95 [3750, 3800, 3250, nil, 3450, ... ], 2 nils
-2 :body_mass_kg double    95 [3.75, 3.8, 3.25, nil, 3.45, ... ], 2 nils
+#<RedAmber::DataFrame : 344 x 2 Vectors, 0x00000000000212f0>
+    body_mass_g body_mass_kg
+       <uint16>     <double>
+  1        3750          3.8
+  2        3800          3.8
+  3        3250          3.3
+  4       (nil)        (nil)
+  5        3450          3.5
+  :           :            :
+342        5750          5.8
+343        5200          5.2
+344        5400          5.4
 ```
 
 DataFrame manipulating methods like `pick`, `drop`, `slice`, `remove`, `rename` and `assign` accept a block.
@@ -178,19 +163,9 @@ Vectors accepts some [functional methods from Arrow](https://arrow.apache.org/do
 
 See [Vector.md](doc/Vector.md) for details.
 
-## TDR
-
-I named the data frame representation style in the model above as TDR (Transposed DataFrame Representation). 
-
-This library can be used with both TDR mode and usual Table mode.
-If you set the environment variable `RED_AMBER_OUTPUT_MODE` to `"table"`, output style by `inspect` and `to_iruby` is the Table mode. Other value including nil will output TDR style.
-
-You can switch the mode in Ruby like this.
-```ruby
-ENV['RED_AMBER_OUTPUT_STYLE'] = 'table' # => Table mode
-```
+## Jupyter notebook
 
-For more detail information about TDR, see [TDR.md](doc/tdr.md).
+[47 Examples of Red Amber](doc/47_examples_of_red_amber.ipynb)
 
 ## Development
 

data/Rakefile

@@ -7,6 +7,7 @@ Rake::TestTask.new(:test) do |t|
   t.libs << 'test'
   t.libs << 'lib'
   t.test_files = FileList['test/**/test_*.rb']
+  t.warning = false
 end
 
 require 'rubocop/rake_task'

data/benchmark/csv_load_penguins.yml

@@ -1,11 +1,11 @@
 prelude: |
-  require 'datasets-arrow'
   require 'rover'
   require 'red_amber'
 
   penguins_csv = 'benchmark/cache/penguins.csv'
 
   unless File.exist?(penguins_csv)
+    require 'datasets-arrow'
     arrow = Datasets::Penguins.new.to_arrow
     RedAmber::DataFrame.new(arrow).save(penguins_csv)
   end