red_amber 0.1.7 → 0.2.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.rubocop.yml +12 -2
- data/.rubocop_todo.yml +2 -15
- data/.yardopts +1 -0
- data/CHANGELOG.md +164 -2
- data/Gemfile +2 -1
- data/README.md +246 -17
- data/doc/DataFrame.md +392 -129
- data/doc/Vector.md +37 -19
- data/doc/examples_of_red_amber.ipynb +8979 -0
- data/lib/red_amber/data_frame.rb +138 -24
- data/lib/red_amber/data_frame_displayable.rb +35 -18
- data/lib/red_amber/data_frame_reshaping.rb +85 -0
- data/lib/red_amber/data_frame_selectable.rb +53 -9
- data/lib/red_amber/data_frame_variable_operation.rb +130 -50
- data/lib/red_amber/group.rb +29 -27
- data/lib/red_amber/vector.rb +1 -1
- data/lib/red_amber/vector_functions.rb +65 -23
- data/lib/red_amber/vector_selectable.rb +12 -9
- data/lib/red_amber/vector_updatable.rb +22 -1
- data/lib/red_amber/version.rb +1 -1
- data/lib/red_amber.rb +1 -1
- data/red_amber.gemspec +1 -1
- metadata +7 -5
- data/doc/47_examples_of_red_amber.ipynb +0 -4872
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: d239a3fa90e5796fb695f8d3c4995d0a2178ea7c8c2789bed157e688902585cb
|
|
4
|
+
data.tar.gz: 968c02294d24a3dabaa6e5128be0bcfad713e131df15850ac0ceb64c2883dcd0
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: d1c5ffd9650dd8c9e825514cd7e2ff4914690bd731ac262fca6cc17e56c1e312679689351a05fb741dccfb59377214706a8bf6ca6fe3237ca46fb623ae1b9f10
|
|
7
|
+
data.tar.gz: f37c4aff9170cd5105737a9d2b3d827051254dcca6968b697f5ed3a70e1b2c3cb14303e88a9c342870d1447450a538e445d6f3d37de53591d3f6d13b87aebc16
|
data/.rubocop.yml
CHANGED
|
@@ -43,6 +43,11 @@ Lint/BinaryOperatorWithIdenticalOperands:
|
|
|
43
43
|
Exclude:
|
|
44
44
|
- 'test/test_vector_function.rb'
|
|
45
45
|
|
|
46
|
+
# Need for test with empty block
|
|
47
|
+
Lint/EmptyBlock:
|
|
48
|
+
Exclude:
|
|
49
|
+
- 'test/test_group.rb'
|
|
50
|
+
|
|
46
51
|
# Max: 120
|
|
47
52
|
Layout/LineLength:
|
|
48
53
|
Max: 118
|
|
@@ -56,6 +61,7 @@ Metrics/AbcSize:
|
|
|
56
61
|
Max: 30
|
|
57
62
|
Exclude:
|
|
58
63
|
- 'lib/red_amber/data_frame_displayable.rb' # Max: 55
|
|
64
|
+
- 'lib/red_amber/data_frame_reshaping.rb' # Max 40.91
|
|
59
65
|
- 'lib/red_amber/data_frame_selectable.rb' # Max: 51
|
|
60
66
|
- 'lib/red_amber/vector_updatable.rb' # Max: 36
|
|
61
67
|
- 'lib/red_amber/vector_selectable.rb' # Max: 33
|
|
@@ -78,23 +84,27 @@ Metrics/ClassLength:
|
|
|
78
84
|
Metrics/CyclomaticComplexity:
|
|
79
85
|
Max: 12
|
|
80
86
|
Exclude:
|
|
87
|
+
- 'lib/red_amber/data_frame_displayable.rb' # Max: 18
|
|
81
88
|
- 'lib/red_amber/data_frame_selectable.rb' # Max: 14
|
|
89
|
+
- 'lib/red_amber/vector_selectable.rb' # Max: 13
|
|
82
90
|
- 'lib/red_amber/vector_updatable.rb' # Max: 14
|
|
83
|
-
- 'lib/red_amber/data_frame_displayable.rb' # Max: 18
|
|
84
91
|
|
|
85
92
|
# Max: 10
|
|
86
93
|
Metrics/MethodLength:
|
|
87
94
|
Max: 30
|
|
88
95
|
Exclude:
|
|
89
96
|
- 'lib/red_amber/data_frame_displayable.rb' # Max: 33
|
|
97
|
+
- 'lib/red_amber/data_frame_selectable.rb' # Max: 38
|
|
98
|
+
- 'lib/red_amber/data_frame_variable_operation.rb' # Max: 35
|
|
90
99
|
|
|
91
100
|
# Max: 100
|
|
92
101
|
Metrics/ModuleLength:
|
|
93
102
|
Max: 100
|
|
94
103
|
Exclude:
|
|
104
|
+
- 'lib/red_amber/data_frame_displayable.rb' # Max: 132
|
|
95
105
|
- 'lib/red_amber/data_frame_selectable.rb' # Max: 141
|
|
106
|
+
- 'lib/red_amber/data_frame_variable_operation.rb' # Max: 110
|
|
96
107
|
- 'lib/red_amber/vector_functions.rb' # Max: 114
|
|
97
|
-
- 'lib/red_amber/data_frame_displayable.rb' # Max: 132
|
|
98
108
|
|
|
99
109
|
# Max: 8
|
|
100
110
|
Metrics/PerceivedComplexity:
|
data/.rubocop_todo.yml
CHANGED
|
@@ -1,15 +1,2 @@
|
|
|
1
|
-
#
|
|
2
|
-
#
|
|
3
|
-
# on 2022-05-08 02:37:36 UTC using RuboCop version 1.27.0.
|
|
4
|
-
# The point is for the user to remove these configuration records
|
|
5
|
-
# one by one as the offenses are removed from the code base.
|
|
6
|
-
# Note that changes in the inspected code, or installation of new
|
|
7
|
-
# versions of RuboCop, may require this file to be generated again.
|
|
8
|
-
|
|
9
|
-
# Offense count: 1
|
|
10
|
-
# This cop supports unsafe auto-correction (--auto-correct-all).
|
|
11
|
-
# Configuration parameters: EnforcedStyle.
|
|
12
|
-
# SupportedStyles: forbid_for_all_comparison_operators, forbid_for_equality_operators_only, require_for_all_comparison_operators, require_for_equality_operators_only
|
|
13
|
-
Style/YodaCondition:
|
|
14
|
-
Exclude:
|
|
15
|
-
- 'lib/red_amber/data_frame.rb'
|
|
1
|
+
# We will use cops to detect bugs in an early stage
|
|
2
|
+
# Feel free to use .rubocop_todo.yml by --auto-gen-config
|
data/.yardopts
ADDED
|
@@ -0,0 +1 @@
|
|
|
1
|
+
--output-dir doc/yard
|
data/CHANGELOG.md
CHANGED
|
@@ -1,6 +1,168 @@
|
|
|
1
|
-
## [0.1
|
|
1
|
+
## [0.2.1] - 2022-09-07
|
|
2
2
|
|
|
3
|
-
-
|
|
3
|
+
-Bug fixes
|
|
4
|
+
|
|
5
|
+
- Fix `Vector#each` with block (#66)
|
|
6
|
+
`Vector#each` will return value of each element with block.
|
|
7
|
+
|
|
8
|
+
- Fix table format at size == 9 (#67)
|
|
9
|
+
|
|
10
|
+
- Fix to support Vector in `DataFrame#assign` (#77)
|
|
11
|
+
|
|
12
|
+
- Add `assert_delta` functionality for `assert_with_NaN` (#78)
|
|
13
|
+
|
|
14
|
+
- Fix Vector#is_in when self is chunked (#79)
|
|
15
|
+
|
|
16
|
+
- Fix Array type error (uint/int) (#79)
|
|
17
|
+
|
|
18
|
+
- New features and improvements
|
|
19
|
+
|
|
20
|
+
- Refine `DataFrame#indices` method (#67)
|
|
21
|
+
|
|
22
|
+
- Update DataFrame reshaping methods (#73)
|
|
23
|
+
|
|
24
|
+
- Change default option value of DataFrame reshaping
|
|
25
|
+
|
|
26
|
+
- Change the order of import_cars example
|
|
27
|
+
|
|
28
|
+
- Add `DataFrame#method_missing` to get column vector by method (#75)
|
|
29
|
+
|
|
30
|
+
- Add `DataFrame#method_missing` to get column (#75)
|
|
31
|
+
|
|
32
|
+
- Accept both args and block in `DataFrame#assign` (#75)
|
|
33
|
+
|
|
34
|
+
- Accept indices in `DataFrame#pick` and `DataFrame#drop` (#76)
|
|
35
|
+
|
|
36
|
+
- Add `DataFrame#slice_by` method (#77)
|
|
37
|
+
|
|
38
|
+
- Add new Vector functions (#78)
|
|
39
|
+
|
|
40
|
+
- Add inverse trigonometric function for Vector
|
|
41
|
+
- `acos`
|
|
42
|
+
- `asin`
|
|
43
|
+
|
|
44
|
+
- Add logarithmic function for Vector
|
|
45
|
+
- `ln`
|
|
46
|
+
- `log10`
|
|
47
|
+
- `log1p`
|
|
48
|
+
- `log2`
|
|
49
|
+
|
|
50
|
+
- Add binary function `Vector#logb`
|
|
51
|
+
|
|
52
|
+
- Docker image and Jupyter Notebook (Thanks to @mrkn)
|
|
53
|
+
- Add link to RubyData in README
|
|
54
|
+
- Add link to interactive README by Binder
|
|
55
|
+
|
|
56
|
+
- Update Jupyter Notebook `71 examples of RedAmber`
|
|
57
|
+
|
|
58
|
+
|
|
59
|
+
## [0.2.0] - 2022-08-15
|
|
60
|
+
|
|
61
|
+
- Bump version up to 0.2.0
|
|
62
|
+
|
|
63
|
+
- Bug fixes
|
|
64
|
+
|
|
65
|
+
- Fix order of multiple group keys (#55)
|
|
66
|
+
|
|
67
|
+
Only 1 group key comes to left. Other keys remain in right.
|
|
68
|
+
|
|
69
|
+
- Remove optional `require` for rover (#55)
|
|
70
|
+
|
|
71
|
+
Fix DataFrame.new for argument with Rover::DataFrame.
|
|
72
|
+
|
|
73
|
+
- Fix occasional failure in CI (#59)
|
|
74
|
+
|
|
75
|
+
Sometimes the CI test fails. I added -dev dependency
|
|
76
|
+
in Arrow install by apt, not doing in bundler.
|
|
77
|
+
|
|
78
|
+
- Fix calling :take in V#[] (#56)
|
|
79
|
+
|
|
80
|
+
Fixed to call Arrow function :take instead of :array_take in Vector#take_by_vector. This will prevent the error below
|
|
81
|
+
when called with Arrow::ChunkedArray.
|
|
82
|
+
|
|
83
|
+
- Raise error renaming non existing key (#61)
|
|
84
|
+
|
|
85
|
+
Add error when specified key is not exist.
|
|
86
|
+
|
|
87
|
+
- Fix DataFrame#rename #assign by array (#65)
|
|
88
|
+
|
|
89
|
+
- New features and improvements
|
|
90
|
+
|
|
91
|
+
- Support Arrow 9.0.0
|
|
92
|
+
- Upgrade to Arrow 9.0.0 (#59)
|
|
93
|
+
- Add Vector#quantile method (#59)
|
|
94
|
+
Arrow::QuantileOptions has supported in Arrow GLib 9.0.0 (ARROW-16623, Thanks!)
|
|
95
|
+
|
|
96
|
+
- Add Vector#quantiles (#62)
|
|
97
|
+
|
|
98
|
+
- Add DataFrame#each_row (#56)
|
|
99
|
+
- Returns Enumerator if block is not given.
|
|
100
|
+
- Change DataFrame#each_row to return a Hash {key => row} (#63)
|
|
101
|
+
|
|
102
|
+
- Refactor to use pattern match in overloaded parameter parsing (#61)
|
|
103
|
+
- Refine DataFrame.new to use pattern match
|
|
104
|
+
- Use pattern match in DataFrame#assign
|
|
105
|
+
- Use pattern match in DataFrame#rename
|
|
106
|
+
|
|
107
|
+
- Accept Array for renamer/assigner in #rename/#assign (#61)
|
|
108
|
+
- Accept assigner by Arrays in DataFrame#assign
|
|
109
|
+
- Accept renamer pairs by Arrays in DataFrame#rename
|
|
110
|
+
- Add DataFrame#assign_left method
|
|
111
|
+
|
|
112
|
+
- Add summary/describe (#62)
|
|
113
|
+
- Introduce DataFrame#summary(#describe)
|
|
114
|
+
|
|
115
|
+
- Introduce reshaping methods for DataFrame (#64)
|
|
116
|
+
- Introduce DataFrame#transpose method
|
|
117
|
+
- Intorduce DataFrame#to_long method
|
|
118
|
+
- Intorduce DataFrame#to_wide method
|
|
119
|
+
|
|
120
|
+
- Others
|
|
121
|
+
|
|
122
|
+
- Add alias sort_index for array_sort_indices (#59)
|
|
123
|
+
- Enable :width option in DataFrame#to_s (#62)
|
|
124
|
+
- Add options to DataFrame#format_table (#62)
|
|
125
|
+
|
|
126
|
+
- Update Documents
|
|
127
|
+
|
|
128
|
+
- Add Yard doc for some methods
|
|
129
|
+
|
|
130
|
+
- Update Jupyter notebook '61 Examples of Red Amber' (#65)
|
|
131
|
+
|
|
132
|
+
## [0.1.8] - 2022-08-04 (experimental)
|
|
133
|
+
|
|
134
|
+
- Bug fixes
|
|
135
|
+
|
|
136
|
+
- Fix unnamed column in table formatter (#52)
|
|
137
|
+
- Fix DataFrame#key?, DataFrame#key_index when @keys.nil? (#52)
|
|
138
|
+
- Align order of replacer in Vector#replace (#53, resolved #38)
|
|
139
|
+
|
|
140
|
+
- New features and improvements
|
|
141
|
+
|
|
142
|
+
- Refine DataFrame.new for empty arguments (#50)
|
|
143
|
+
- Delete .rubocop_todo.yml for not to use yoda condition (#50)
|
|
144
|
+
|
|
145
|
+
- Refine Group (#52, resolved #28)
|
|
146
|
+
- Refine Group methods creation
|
|
147
|
+
- Make group key at first(left)
|
|
148
|
+
- Show only one group count when same counts
|
|
149
|
+
- Add block acceptability for group
|
|
150
|
+
- Rename empty key to :unnamed in DataFrame.new
|
|
151
|
+
- Rename Group#aggregated_by to #summarize (#54)
|
|
152
|
+
|
|
153
|
+
- Add Vector#shift (#51)
|
|
154
|
+
|
|
155
|
+
- Vector#[] accepts Range as an argument (#51)
|
|
156
|
+
|
|
157
|
+
- Update documents
|
|
158
|
+
|
|
159
|
+
- Add support for yard (#54)
|
|
160
|
+
|
|
161
|
+
- Renew jupyter notebook '53 examples' (#54)
|
|
162
|
+
|
|
163
|
+
- Add more examples and images in README (#52)
|
|
164
|
+
- Add document of group manipulations in README (#52)
|
|
165
|
+
- Renew DF#group document in DataFrame.md (#52)
|
|
4
166
|
|
|
5
167
|
## [0.1.7] - 2022-07-15 (experimental)
|
|
6
168
|
|
data/Gemfile
CHANGED
|
@@ -7,7 +7,7 @@ gemspec
|
|
|
7
7
|
group :test do
|
|
8
8
|
gem 'rake'
|
|
9
9
|
|
|
10
|
-
gem 'red-parquet', '>=
|
|
10
|
+
gem 'red-parquet', '>= 9.0.0'
|
|
11
11
|
gem 'rover-df', '~> 0.3.0'
|
|
12
12
|
|
|
13
13
|
gem 'rubocop'
|
|
@@ -18,6 +18,7 @@ group :test do
|
|
|
18
18
|
gem 'iruby'
|
|
19
19
|
gem 'test-unit'
|
|
20
20
|
gem 'webrick'
|
|
21
|
+
gem 'yard'
|
|
21
22
|
|
|
22
23
|
gem 'benchmark_driver'
|
|
23
24
|
gem 'red-datasets'
|
data/README.md
CHANGED
|
@@ -3,17 +3,23 @@
|
|
|
3
3
|
[](https://badge.fury.io/rb/red_amber)
|
|
4
4
|
[](https://github.com/heronshoes/red_amber/actions/workflows/test.yml)
|
|
5
5
|
|
|
6
|
-
A simple dataframe library for Ruby
|
|
6
|
+
A simple dataframe library for Ruby.
|
|
7
7
|
|
|
8
8
|
- Powered by [Red Arrow](https://github.com/apache/arrow/tree/master/ruby/red-arrow) [](https://gitter.im/red-data-tools/en)
|
|
9
9
|
- Inspired by the dataframe library [Rover-df](https://github.com/ankane/rover)
|
|
10
10
|
|
|
11
11
|
## Requirements
|
|
12
12
|
|
|
13
|
+
Supported Ruby version is >= 2.7.
|
|
14
|
+
|
|
15
|
+
Since v0.2.0, this library uses pattern matching which is an experimental feature in 2.7 . It is usable but a warning message will be shown in 2.7 .
|
|
16
|
+
I recommend Ruby 3 for performance.
|
|
17
|
+
|
|
13
18
|
```ruby
|
|
14
|
-
|
|
19
|
+
# Libraries required
|
|
20
|
+
gem 'red-arrow', '>= 9.0.0'
|
|
15
21
|
|
|
16
|
-
gem 'red-parquet', '>=
|
|
22
|
+
gem 'red-parquet', '>= 9.0.0' # Optional, if you use IO from/to parquet
|
|
17
23
|
gem 'rover-df', '~> 0.3.0' # Optional, if you use IO from/to Rover::DataFrame
|
|
18
24
|
```
|
|
19
25
|
|
|
@@ -21,9 +27,9 @@ gem 'rover-df', '~> 0.3.0' # Optional, if you use IO from/to Rover::DataFrame
|
|
|
21
27
|
|
|
22
28
|
Install requirements before you install Red Amber.
|
|
23
29
|
|
|
24
|
-
- Apache Arrow GLib (>=
|
|
30
|
+
- Apache Arrow GLib (>= 9.0.0)
|
|
25
31
|
|
|
26
|
-
- Apache Parquet GLib (>=
|
|
32
|
+
- Apache Parquet GLib (>= 9.0.0) # If you use IO from/to parquet
|
|
27
33
|
|
|
28
34
|
See [Apache Arrow install document](https://arrow.apache.org/install/).
|
|
29
35
|
|
|
@@ -47,16 +53,27 @@ Or install it yourself as:
|
|
|
47
53
|
gem install red_amber
|
|
48
54
|
```
|
|
49
55
|
|
|
56
|
+
## Docker image and Jupyter Notebook
|
|
57
|
+
|
|
58
|
+
[RubyData Docker Stacks](https://github.com/RubyData/docker-stacks) is available as a ready-to-run Docker image containing Jupyter and useful data tools as well as RedAmber (Thanks to @mrkn).
|
|
59
|
+
|
|
60
|
+
Also you can try the contents of this README interactively by [Binder](https://mybinder.org/v2/gh/RubyData/docker-stacks/master?filepath=red-amber.ipynb).
|
|
61
|
+
[](https://mybinder.org/v2/gh/RubyData/docker-stacks/master?filepath=red-amber.ipynb)
|
|
62
|
+
|
|
63
|
+
|
|
64
|
+
|
|
50
65
|
## `RedAmber::DataFrame`
|
|
51
66
|
|
|
52
|
-
|
|
67
|
+
It represents a set of data in 2D-shape. The entity is a Red Arrow's Table object.
|
|
68
|
+
|
|
69
|
+
|
|
53
70
|
|
|
54
71
|
```ruby
|
|
55
72
|
require 'red_amber' # require 'red-amber' is also OK.
|
|
56
73
|
require 'datasets-arrow'
|
|
57
74
|
|
|
58
75
|
arrow = Datasets::Penguins.new.to_arrow
|
|
59
|
-
RedAmber::DataFrame.new(arrow)
|
|
76
|
+
penguins = RedAmber::DataFrame.new(arrow)
|
|
60
77
|
|
|
61
78
|
# =>
|
|
62
79
|
#<RedAmber::DataFrame : 344 x 8 Vectors, 0x0000000000013790>
|
|
@@ -73,17 +90,52 @@ RedAmber::DataFrame.new(arrow)
|
|
|
73
90
|
344 Gentoo Biscoe 49.9 16.1 213 ... 2009
|
|
74
91
|
```
|
|
75
92
|
|
|
76
|
-
|
|
77
|
-
|
|
93
|
+
For example, `DataFrame#pick` accepts keys as arguments and returns a sub DataFrame.
|
|
78
94
|
|
|
79
|
-
|
|
95
|
+
|
|
80
96
|
|
|
81
97
|
```ruby
|
|
82
|
-
|
|
98
|
+
penguins.keys
|
|
99
|
+
# =>
|
|
100
|
+
[:species,
|
|
101
|
+
:island,
|
|
102
|
+
:bill_length_mm,
|
|
103
|
+
:bill_depth_mm,
|
|
104
|
+
:flipper_length_mm,
|
|
105
|
+
:body_mass_g,
|
|
106
|
+
:sex,
|
|
107
|
+
:year]
|
|
108
|
+
|
|
109
|
+
df = penguins.pick(:species, :island, :body_mass_g)
|
|
83
110
|
df
|
|
84
111
|
|
|
85
112
|
# =>
|
|
86
|
-
#<RedAmber::DataFrame : 344 x
|
|
113
|
+
#<RedAmber::DataFrame : 344 x 3 Vectors, 0x000000000003cc1c>
|
|
114
|
+
species island body_mass_g
|
|
115
|
+
<string> <string> <uint16>
|
|
116
|
+
1 Adelie Torgersen 3750
|
|
117
|
+
2 Adelie Torgersen 3800
|
|
118
|
+
3 Adelie Torgersen 3250
|
|
119
|
+
4 Adelie Torgersen (nil)
|
|
120
|
+
5 Adelie Torgersen 3450
|
|
121
|
+
: : : :
|
|
122
|
+
342 Gentoo Biscoe 5750
|
|
123
|
+
343 Gentoo Biscoe 5200
|
|
124
|
+
344 Gentoo Biscoe 5400
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
`DataFrame#drop` drops some columns to create a remainer DataFrame.
|
|
128
|
+
|
|
129
|
+
|
|
130
|
+
|
|
131
|
+
You can specify by keys or a boolean array of same size as n_keys.
|
|
132
|
+
|
|
133
|
+
```ruby
|
|
134
|
+
# Same as df.drop(:species, :island)
|
|
135
|
+
df = df.drop(true, true, false)
|
|
136
|
+
|
|
137
|
+
# =>
|
|
138
|
+
#<RedAmber::DataFrame : 344 x 1 Vector, 0x0000000000048760>
|
|
87
139
|
body_mass_g
|
|
88
140
|
<uint16>
|
|
89
141
|
1 3750
|
|
@@ -97,9 +149,14 @@ df
|
|
|
97
149
|
344 5400
|
|
98
150
|
```
|
|
99
151
|
|
|
100
|
-
|
|
152
|
+
Arrow data is immutable, so these methods always return an new object.
|
|
153
|
+
|
|
154
|
+
`DataFrame#assign` creates new columns or update existing columns.
|
|
155
|
+
|
|
156
|
+
|
|
101
157
|
|
|
102
158
|
```ruby
|
|
159
|
+
# New column is created because ':body_mass_kg' is a new key.
|
|
103
160
|
df.assign(:body_mass_kg => df[:body_mass_g] / 1000.0)
|
|
104
161
|
|
|
105
162
|
# =>
|
|
@@ -117,14 +174,103 @@ df.assign(:body_mass_kg => df[:body_mass_g] / 1000.0)
|
|
|
117
174
|
344 5400 5.4
|
|
118
175
|
```
|
|
119
176
|
|
|
177
|
+
`DataFrame#slice` selects rows (observations) to create a sub DataFrame.
|
|
178
|
+
|
|
179
|
+
|
|
180
|
+
|
|
181
|
+
```ruby
|
|
182
|
+
# returns 5 rows at the start and 5 rows from the end
|
|
183
|
+
penguins.slice(0...5, -5..-1)
|
|
184
|
+
|
|
185
|
+
# =>
|
|
186
|
+
#<RedAmber::DataFrame : 10 x 8 Vectors, 0x0000000000042be4>
|
|
187
|
+
species island bill_length_mm bill_depth_mm flipper_length_mm ... year
|
|
188
|
+
<string> <string> <double> <double> <uint8> ... <uint16>
|
|
189
|
+
1 Adelie Torgersen 39.1 18.7 181 ... 2007
|
|
190
|
+
2 Adelie Torgersen 39.5 17.4 186 ... 2007
|
|
191
|
+
3 Adelie Torgersen 40.3 18.0 195 ... 2007
|
|
192
|
+
4 Adelie Torgersen (nil) (nil) (nil) ... 2007
|
|
193
|
+
5 Adelie Torgersen 36.7 19.3 193 ... 2007
|
|
194
|
+
: : : : : : ... :
|
|
195
|
+
8 Gentoo Biscoe 50.4 15.7 222 ... 2009
|
|
196
|
+
9 Gentoo Biscoe 45.2 14.8 212 ... 2009
|
|
197
|
+
10 Gentoo Biscoe 49.9 16.1 213 ... 2009
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
`DataFrame#remove` rejects rows (observations) to create a remainer DataFrame.
|
|
201
|
+
|
|
202
|
+
|
|
203
|
+
|
|
204
|
+
```ruby
|
|
205
|
+
# penguins[:bill_length_mm] < 40 returns a boolean Vector
|
|
206
|
+
penguins.remove(penguins[:bill_length_mm] < 40)
|
|
207
|
+
|
|
208
|
+
# =>
|
|
209
|
+
#<RedAmber::DataFrame : 244 x 8 Vectors, 0x000000000007d6f4>
|
|
210
|
+
species island bill_length_mm bill_depth_mm flipper_length_mm ... year
|
|
211
|
+
<string> <string> <double> <double> <uint8> ... <uint16>
|
|
212
|
+
1 Adelie Torgersen 40.3 18.0 195 ... 2007
|
|
213
|
+
2 Adelie Torgersen (nil) (nil) (nil) ... 2007
|
|
214
|
+
3 Adelie Torgersen 42.0 20.2 190 ... 2007
|
|
215
|
+
4 Adelie Torgersen 41.1 17.6 182 ... 2007
|
|
216
|
+
5 Adelie Torgersen 42.5 20.7 197 ... 2007
|
|
217
|
+
: : : : : : ... :
|
|
218
|
+
242 Gentoo Biscoe 50.4 15.7 222 ... 2009
|
|
219
|
+
243 Gentoo Biscoe 45.2 14.8 212 ... 2009
|
|
220
|
+
244 Gentoo Biscoe 49.9 16.1 213 ... 2009
|
|
221
|
+
```
|
|
222
|
+
|
|
120
223
|
DataFrame manipulating methods like `pick`, `drop`, `slice`, `remove`, `rename` and `assign` accept a block.
|
|
121
224
|
|
|
122
|
-
|
|
225
|
+
Previous example is also OK with a block.
|
|
226
|
+
|
|
227
|
+
```ruby
|
|
228
|
+
penguins.remove { bill_length_mm < 40 }
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
Next example is an usage of block to update a column.
|
|
123
232
|
|
|
124
233
|
```ruby
|
|
125
|
-
|
|
234
|
+
df = RedAmber::DataFrame.new(
|
|
235
|
+
integer: [0, 1, 2, 3, nil],
|
|
236
|
+
float: [0.0, 1.1, 2.2, Float::NAN, nil],
|
|
237
|
+
string: ['A', 'B', 'C', 'D', nil],
|
|
238
|
+
boolean: [true, false, true, false, nil])
|
|
239
|
+
df
|
|
240
|
+
|
|
241
|
+
# =>
|
|
242
|
+
#<RedAmber::DataFrame : 5 x 4 Vectors, 0x000000000003131c>
|
|
243
|
+
integer float string boolean
|
|
244
|
+
<uint8> <double> <string> <boolean>
|
|
245
|
+
1 0 0.0 A true
|
|
246
|
+
2 1 1.1 B false
|
|
247
|
+
3 2 2.2 C true
|
|
248
|
+
4 3 NaN D false
|
|
249
|
+
5 (nil) (nil) (nil) (nil)
|
|
250
|
+
|
|
251
|
+
df.assign do
|
|
252
|
+
vectors.select(&:float?).map { |v| [v.key, -v] }
|
|
253
|
+
# => returns [[:float], [-0.0, -1.1, -2.2, NAN, nil]]
|
|
254
|
+
end
|
|
255
|
+
|
|
256
|
+
# =>
|
|
257
|
+
#<RedAmber::DataFrame : 5 x 3 Vectors, 0x00000000000e270c>
|
|
258
|
+
index float string
|
|
259
|
+
<uint8> <double> <string>
|
|
260
|
+
1 0 -0.0 A
|
|
261
|
+
2 1 -1.1 B
|
|
262
|
+
3 2 -2.2 C
|
|
263
|
+
4 3 NaN D
|
|
264
|
+
5 (nil) (nil) (nil)
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
Next example is to eliminate rows containing nil.
|
|
268
|
+
|
|
269
|
+
```ruby
|
|
270
|
+
# remove all observations containing nil
|
|
126
271
|
nil_removed = penguins.remove { vectors.map(&:is_nil).reduce(&:|) }
|
|
127
272
|
nil_removed.tdr
|
|
273
|
+
|
|
128
274
|
# =>
|
|
129
275
|
RedAmber::DataFrame : 342 x 8 Vectors
|
|
130
276
|
Vectors : 5 numeric, 3 strings
|
|
@@ -145,12 +291,66 @@ For this frequently needed task, we can do it much simpler.
|
|
|
145
291
|
penguins.remove_nil # => same result as above
|
|
146
292
|
```
|
|
147
293
|
|
|
148
|
-
|
|
294
|
+
`DataFrame#summary` shows summary statistics in a DataFrame.
|
|
295
|
+
|
|
296
|
+
```ruby
|
|
297
|
+
puts penguins.summary.to_s(width: 82)
|
|
298
|
+
|
|
299
|
+
# =>
|
|
300
|
+
variables count mean std min 25% median 75% max
|
|
301
|
+
<dictionary> <uint16> <double> <double> <double> <double> <double> <double> <double>
|
|
302
|
+
1 bill_length_mm 342 43.92 5.46 32.1 39.23 44.38 48.5 59.6
|
|
303
|
+
2 bill_depth_mm 342 17.15 1.97 13.1 15.6 17.32 18.7 21.5
|
|
304
|
+
3 flipper_length_mm 342 200.92 14.06 172.0 190.0 197.0 213.0 231.0
|
|
305
|
+
4 body_mass_g 342 4201.75 801.95 2700.0 3550.0 4031.5 4750.0 6300.0
|
|
306
|
+
5 year 344 2008.03 0.82 2007.0 2007.0 2008.0 2009.0 2009.0
|
|
307
|
+
```
|
|
308
|
+
|
|
309
|
+
`DataFrame#group` method can be used for the grouping tasks.
|
|
310
|
+
|
|
311
|
+
```ruby
|
|
312
|
+
starwars = RedAmber::DataFrame.load(URI("https://vincentarelbundock.github.io/Rdatasets/csv/dplyr/starwars.csv"))
|
|
313
|
+
starwars
|
|
314
|
+
|
|
315
|
+
# =>
|
|
316
|
+
#<RedAmber::DataFrame : 87 x 12 Vectors, 0x000000000000607c>
|
|
317
|
+
unnamed1 name height mass hair_color skin_color eye_color ... species
|
|
318
|
+
<int64> <string> <int64> <double> <string> <string> <string> ... <string>
|
|
319
|
+
1 1 Luke Skywalker 172 77.0 blond fair blue ... Human
|
|
320
|
+
2 2 C-3PO 167 75.0 NA gold yellow ... Droid
|
|
321
|
+
3 3 R2-D2 96 32.0 NA white, blue red ... Droid
|
|
322
|
+
4 4 Darth Vader 202 136.0 none white yellow ... Human
|
|
323
|
+
5 5 Leia Organa 150 49.0 brown light brown ... Human
|
|
324
|
+
: : : : : : : : ... :
|
|
325
|
+
85 85 BB8 (nil) (nil) none none black ... Droid
|
|
326
|
+
86 86 Captain Phasma (nil) (nil) unknown unknown unknown ... NA
|
|
327
|
+
87 87 Padmé Amidala 165 45.0 brown light brown ... Human
|
|
328
|
+
|
|
329
|
+
starwars.group(:species) { [count(:species), mean(:height, :mass)] }
|
|
330
|
+
.slice { count > 1 }
|
|
331
|
+
|
|
332
|
+
# =>
|
|
333
|
+
#<RedAmber::DataFrame : 9 x 4 Vectors, 0x000000000006e848>
|
|
334
|
+
species count mean(height) mean(mass)
|
|
335
|
+
<string> <int64> <double> <double>
|
|
336
|
+
1 Human 35 176.6 82.8
|
|
337
|
+
2 Droid 6 131.2 69.8
|
|
338
|
+
3 Wookiee 2 231.0 124.0
|
|
339
|
+
4 Gungan 3 208.7 74.0
|
|
340
|
+
5 NA 4 181.3 48.0
|
|
341
|
+
6 Zabrak 2 173.0 80.0
|
|
342
|
+
7 Twi'lek 2 179.0 55.0
|
|
343
|
+
8 Mirialan 2 168.0 53.1
|
|
344
|
+
9 Kaminoan 2 221.0 88.0
|
|
345
|
+
```
|
|
346
|
+
|
|
347
|
+
See [DataFrame.md](doc/DataFrame.md) for other examples and details.
|
|
149
348
|
|
|
150
349
|
|
|
151
350
|
## `RedAmber::Vector`
|
|
152
351
|
|
|
153
352
|
Class `RedAmber::Vector` represents a series of data in the DataFrame.
|
|
353
|
+
Method `RedAmber::DataFrame#[key]` returns a Vector with the key `key`.
|
|
154
354
|
|
|
155
355
|
```ruby
|
|
156
356
|
penguins[:bill_length_mm]
|
|
@@ -161,11 +361,34 @@ penguins[:bill_length_mm]
|
|
|
161
361
|
|
|
162
362
|
Vectors accepts some [functional methods from Arrow](https://arrow.apache.org/docs/cpp/compute.html).
|
|
163
363
|
|
|
364
|
+
This is an element-wise comparison and returns a boolean Vector of same size.
|
|
365
|
+
|
|
366
|
+
|
|
367
|
+
|
|
368
|
+
```ruby
|
|
369
|
+
penguins[:bill_length_mm] < 40
|
|
370
|
+
|
|
371
|
+
# =>
|
|
372
|
+
#<RedAmber::Vector(:boolean, size=344):0x000000000007e7ac>
|
|
373
|
+
[true, true, false, nil, true, true, true, true, true, false, true, true, false, ... ]
|
|
374
|
+
```
|
|
375
|
+
|
|
376
|
+
Next example returns aggregated result.
|
|
377
|
+
|
|
378
|
+
|
|
379
|
+
|
|
380
|
+
```ruby
|
|
381
|
+
penguins[:bill_length_mm].mean
|
|
382
|
+
43.92192982456141
|
|
383
|
+
# =>
|
|
384
|
+
|
|
385
|
+
```
|
|
386
|
+
|
|
164
387
|
See [Vector.md](doc/Vector.md) for details.
|
|
165
388
|
|
|
166
389
|
## Jupyter notebook
|
|
167
390
|
|
|
168
|
-
[
|
|
391
|
+
[71 Examples of Red Amber](doc/examples_of_red_amber.ipynb) shows more examples in jupyter notebook.
|
|
169
392
|
|
|
170
393
|
## Development
|
|
171
394
|
|
|
@@ -176,6 +399,12 @@ bundle install
|
|
|
176
399
|
bundle exec rake test
|
|
177
400
|
```
|
|
178
401
|
|
|
402
|
+
I will appreciate if you could help to improve this project. Here are a few ways you can help:
|
|
403
|
+
|
|
404
|
+
- [Report bugs or suggest new features](https://github.com/heronshoes/red_amber/issues)
|
|
405
|
+
- Fix bugs and [submit pull requests](https://github.com/heronshoes/red_amber/pulls)
|
|
406
|
+
- Write, clarify, or fix documentation
|
|
407
|
+
|
|
179
408
|
## License
|
|
180
409
|
|
|
181
410
|
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
|