fat_table 0.5.2 → 0.5.5
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.org +136 -17
- data/TODO.org +28 -1
- data/lib/ext/array.rb +17 -0
- data/lib/fat_table/column.rb +112 -56
- data/lib/fat_table/convert.rb +20 -10
- data/lib/fat_table/errors.rb +4 -0
- data/lib/fat_table/evaluator.rb +6 -8
- data/lib/fat_table/footer.rb +1 -1
- data/lib/fat_table/formatters/formatter.rb +17 -11
- data/lib/fat_table/formatters/org_formatter.rb +3 -3
- data/lib/fat_table/formatters/term_formatter.rb +3 -3
- data/lib/fat_table/formatters/text_formatter.rb +3 -3
- data/lib/fat_table/table.rb +90 -31
- data/lib/fat_table/version.rb +1 -1
- data/lib/fat_table.rb +16 -16
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 2f391d5e4ad9d7a4dcb303098b90f6fb42379d9df382192c4bacaf169523dab1
|
4
|
+
data.tar.gz: ea75f906fcd164752a3ca2220324ad10d8438562bb31c4846bc6575723e834fa
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 2914c40a497cf24bcdba8868c4a4607ea1bdb0a94fba8caa3d6e31407ff6832621df2a0b4a8a2bb98c3b40080fed6d752f8e936f66a96270fe675e93d5a852d6
|
7
|
+
data.tar.gz: dbc43f4d2bac4a2a19bc7cc90d1fbfbf739a12f0f280a543670af588cfcf2bb0df74d50cfb7ded76b851f5470fd30d4f971659cead5131f9ea15e24b492e5823
|
data/README.org
CHANGED
@@ -25,6 +25,16 @@ The following is for org.
|
|
25
25
|
|
26
26
|
[[https://travis-ci.org/ddoherty03/fat_table.svg?branch=master]]
|
27
27
|
|
28
|
+
* Version
|
29
|
+
#+begin_src ruby :wrap EXAMPLE
|
30
|
+
require 'fat_table'
|
31
|
+
"Current version is: #{FatTable::VERSION}"
|
32
|
+
#+end_src
|
33
|
+
|
34
|
+
#+begin_EXAMPLE
|
35
|
+
Current version is: 0.5.4
|
36
|
+
#+end_EXAMPLE
|
37
|
+
|
28
38
|
* Introduction
|
29
39
|
|
30
40
|
~FatTable~ is a gem that treats tables as a data type. It provides methods for
|
@@ -57,6 +67,7 @@ array of arrays with its ~.to_aoa~ output function will be rendered in an
|
|
57
67
|
org-mode buffer as an org-table, ready for processing by other code blocks.
|
58
68
|
|
59
69
|
* Table of Contents :toc:noexport:
|
70
|
+
- [[#version][Version]]
|
60
71
|
- [[#introduction][Introduction]]
|
61
72
|
- [[#installation][Installation]]
|
62
73
|
- [[#using-in-a-gem][Using in a gem]]
|
@@ -74,6 +85,7 @@ org-mode buffer as an org-table, ready for processing by other code blocks.
|
|
74
85
|
- [[#without-headers][Without Headers]]
|
75
86
|
- [[#with-headers][With Headers]]
|
76
87
|
- [[#forcing-string-type][Forcing String Type]]
|
88
|
+
- [[#designating-tolerant-columns][Designating "Tolerant" Columns]]
|
77
89
|
- [[#from-csv-or-org-mode-files-or-strings][From CSV or Org Mode files or strings]]
|
78
90
|
- [[#from-arrays-of-arrays][From Arrays of Arrays]]
|
79
91
|
- [[#in-ruby-code][In Ruby Code]]
|
@@ -430,11 +442,26 @@ or nil. There are only five permissible types for a ~Column~:
|
|
430
442
|
5. *NilClass* (for the undetermined column type).
|
431
443
|
|
432
444
|
When a ~Table~ is constructed from an external source, all ~Columns~ start out
|
433
|
-
having a type of ~NilClass~, that is, their type is as yet undetermined. When
|
434
|
-
string or object of one of the four determined types is added to a ~Column
|
435
|
-
|
436
|
-
|
437
|
-
|
445
|
+
having a type of ~NilClass~, that is, their type is as yet undetermined. When
|
446
|
+
a string or object of one of the four determined types is added to a ~Column~
|
447
|
+
and it can be converted into one of the permissible types, it fixes the type
|
448
|
+
of the column, and all further items added to the ~Column~ must either be
|
449
|
+
~nil~ (indicating no value) or be capable of being coerced to the column's
|
450
|
+
type. Otherwise, ~FatTable~ raises an exception.
|
451
|
+
|
452
|
+
The strictness of requiring all items to be of the same type can be relaxed by
|
453
|
+
declaring a column to be "tolerant." You can do so when you create the table
|
454
|
+
by adding a tolerant_columns keyword parameter. If a Column is tolerant,
|
455
|
+
~FatTable~ tries to convert new items into a type other than a ~String~ and,
|
456
|
+
if it can do so, sets /that/ as the Column's type. Any later items that
|
457
|
+
cannot be converted into the Column's type are converted to strings. These
|
458
|
+
interloper strings are treated like nils for purposes of sorting and
|
459
|
+
evaluation, but are displayed according to any string formatting on output.
|
460
|
+
See [[*Designating "Tolerant" Columns][Designating "Tolerant" Columns]] below.
|
461
|
+
|
462
|
+
It is also possible to force ~FatTable~ to treat a column as a String type,
|
463
|
+
even its items look like one of the other types. See [[*Forcing String Type][Forcing String Type]]
|
464
|
+
below.
|
438
465
|
|
439
466
|
Items of input must be either one of the permissible ruby objects or strings. If
|
440
467
|
they are strings, ~FatTable~ attempts to parse them as one of the permissible
|
@@ -591,10 +618,18 @@ columns to be created:
|
|
591
618
|
**** Forcing String Type
|
592
619
|
Occasionally, ~FatTable~'s automatic type detection can get in the way and you
|
593
620
|
just want it to treat one or more columns as Strings regardless of their
|
594
|
-
appearance. Think, for example, of zip codes.
|
595
|
-
table, you can
|
596
|
-
|
597
|
-
|
621
|
+
appearance. Think, for example, of zip codes. If headers are given when a
|
622
|
+
table is contructed, you can designate a forced-string column by appending a
|
623
|
+
~!~ to the end of the header. It will not become part of the header, it will
|
624
|
+
just mark it as a forced-string Column.
|
625
|
+
|
626
|
+
#+begin_SRC emacs-lisp :wrap EXAMPLE
|
627
|
+
tab = FatTable.new(:a, 'b', 'C!', :d, :zip!)
|
628
|
+
#+end_SRC
|
629
|
+
|
630
|
+
In addition, at any time after creating a table, you can force the String type
|
631
|
+
on any number of columns with the ~force_string!~ method. When you do so, all
|
632
|
+
exisiting items in the column are converted to strings with the #to_s method.
|
598
633
|
|
599
634
|
#+begin_src ruby :wrap EXAMPLE
|
600
635
|
tab = FatTable.new(:a, 'b', 'C', :d, :zip)
|
@@ -619,8 +654,83 @@ converted to strings with the #to_s method.
|
|
619
654
|
+======+======+============+===+=======+===+
|
620
655
|
#+end_EXAMPLE
|
621
656
|
|
657
|
+
**** Designating "Tolerant" Columns
|
658
|
+
Related to the problem just discussed is the problem of reading files in from
|
659
|
+
the wild where a column may get typed as, say Numeric, but then contain
|
660
|
+
something that can't be parsed as a Numeric. ~FatTable~ raises an exception
|
661
|
+
is such cases, and that may be what you want if you can control the input.
|
662
|
+
But, especially when you cannot do so, it can be helpful to designate one or
|
663
|
+
more columns as "tolerant." This means that when a conversion problem occurs,
|
664
|
+
the column item is retained as a string type in a column that is otherwise of
|
665
|
+
one of the types Numeric, DateTime, or Boolean. Those string items are
|
666
|
+
treated as nils for purposes of sorting or evaluation in a ~select~ method.
|
667
|
+
When formatted, they participate in string formatting directive, but not those
|
668
|
+
for other types.
|
669
|
+
|
670
|
+
All of the table construction methods, allow a keyword parameter,
|
671
|
+
~tolerant_columns~, where you can designate what columns should be convert to
|
672
|
+
String type when conversion to the auto-typed column type is not possible.
|
673
|
+
The parameter should be an array of headers, in either string or symbol form,
|
674
|
+
for which this behavior is desired. In addition, it can be set to the special
|
675
|
+
string '*' or symbol ~:*~ to indicate that all the columns should be made
|
676
|
+
tolerant.
|
677
|
+
|
678
|
+
#+begin_src ruby :wrap EXAMPLE
|
679
|
+
require 'fat_table'
|
680
|
+
tab = FatTable.new(:a, 'b', 'C', :d, :zip, tolerant_columns: [:zip])
|
681
|
+
tab << { a: 1, b: 2, c: "<2017-01-21>", d: 'f', e: '', zip: 18552 }
|
682
|
+
tab << { a: 3.14, b: 2.17, c: '[2016-01-21 Thu]', d: 'Y', e: nil }
|
683
|
+
tab << { zip: '01879--7884' }
|
684
|
+
tab << { zip: '66210' }
|
685
|
+
tab << { zip: '90210' }
|
686
|
+
tab.to_text
|
687
|
+
#+end_src
|
688
|
+
|
689
|
+
#+RESULTS:
|
690
|
+
#+begin_EXAMPLE
|
691
|
+
+======+======+============+===+=============+===+
|
692
|
+
| A | B | C | D | Zip | E |
|
693
|
+
+------+------+------------+---+-------------+---+
|
694
|
+
| 1 | 2 | 2017-01-21 | F | 18552 | |
|
695
|
+
| 3.14 | 2.17 | 2016-01-21 | T | | |
|
696
|
+
| | | | | 01879--7884 | |
|
697
|
+
| | | | | 66210 | |
|
698
|
+
| | | | | 90210 | |
|
699
|
+
+======+======+============+===+=============+===+
|
700
|
+
#+end_EXAMPLE
|
701
|
+
|
702
|
+
Another way to designate a column as tolerant is to end a column you want to
|
703
|
+
designate as tolerant with a ~!~. The ~!~ will be stripped from the header,
|
704
|
+
but it will be marked as tolerant.
|
705
|
+
#+begin_src ruby :wrap EXAMPLE
|
706
|
+
require 'fat_table'
|
707
|
+
tab = FatTable.new(:a, 'b!', 'C', :d, :zip!)
|
708
|
+
tab << { a: 1, b: 2, c: "<2017-01-21>", d: 'f', e: '', zip: 18552 }
|
709
|
+
tab << { a: 3.14, b: 2.17, c: '[2016-01-21 Thu]', d: 'Y', e: nil }
|
710
|
+
tab << { zip: '01879--7884' }
|
711
|
+
tab << { zip: '66210', b: 'Not a Number' }
|
712
|
+
tab << { zip: '90210' }
|
713
|
+
tab.to_text
|
714
|
+
#+end_src
|
715
|
+
|
716
|
+
#+RESULTS:
|
717
|
+
#+begin_EXAMPLE
|
718
|
+
+======+==============+============+===+=============+===+
|
719
|
+
| A | B | C | D | Zip | E |
|
720
|
+
+------+--------------+------------+---+-------------+---+
|
721
|
+
| 1 | 2 | 2017-01-21 | F | 18552 | |
|
722
|
+
| 3.14 | 2.17 | 2016-01-21 | T | | |
|
723
|
+
| | | | | 01879--7884 | |
|
724
|
+
| | Not a Number | | | 66210 | |
|
725
|
+
| | | | | 90210 | |
|
726
|
+
+======+==============+============+===+=============+===+
|
727
|
+
#+end_EXAMPLE
|
728
|
+
|
622
729
|
*** From CSV or Org Mode files or strings
|
623
|
-
Tables can also be read from ~.csv~ files or files containing ~org-mode~
|
730
|
+
Tables can also be read from ~.csv~ files or files containing ~org-mode~
|
731
|
+
tables. Remember that you can make any column tolerant with a
|
732
|
+
~tolerant_columns:~ keyword argument or make them all tolerant by designating
|
733
|
+
the pseudo-column ~:*~ as tolerant.
|
624
734
|
|
625
735
|
In the case of org-mode files, ~FatTable~ skips through the file until it finds
|
626
736
|
a line that look like a table, that is, it begins with any number of spaces
|
@@ -677,8 +787,10 @@ header row, and the headers are converted to symbols as described above.
|
|
677
787
|
|
678
788
|
*** From Arrays of Arrays
|
679
789
|
**** In Ruby Code
|
680
|
-
You can also initialize a table directly from ruby data structures. You can,
|
681
|
-
example, build a table from an array of arrays
|
790
|
+
You can also initialize a table directly from ruby data structures. You can,
|
791
|
+
for example, build a table from an array of arrays. Remember that you can
|
792
|
+
make any column tolerant with a ~tolerant_columns:~ keyword argument or make
|
793
|
+
them all tolerant by designating the pseudo-column ~:*~ as tolerant.
|
682
794
|
|
683
795
|
#+BEGIN_SRC ruby
|
684
796
|
aoa = [
|
@@ -772,8 +884,10 @@ This example illustrates several things:
|
|
772
884
|
|
773
885
|
A second ruby data structure that can be used to initialize a ~FatTable~ table
|
774
886
|
is an array of ruby Hashes. Each hash represents a row of the table, and the
|
775
|
-
headers of the table are taken from the keys of the hashes. Accordingly, all
|
776
|
-
hashes must have the same keys.
|
887
|
+
headers of the table are taken from the keys of the hashes. Accordingly, all
|
888
|
+
the hashes must have the same keys. Remember that you can make any column
|
889
|
+
tolerant with a ~tolerant_columns:~ keyword argument or make them all tolerant
|
890
|
+
by designating the pseudo-column ~:*~ as tolerant.
|
777
891
|
|
778
892
|
This same method can in fact take an array of any objects that can be converted
|
779
893
|
to a Hash with the ~#to_h~ method, so you can use an array of your own objects
|
@@ -862,6 +976,10 @@ The ~.connect~ function need only be called once, and the database handle it
|
|
862
976
|
creates will be used for all subsequent ~.from_sql~ calls until ~.connect~ is
|
863
977
|
called again.
|
864
978
|
|
979
|
+
Remember that you can make any column tolerant with a ~tolerant_columns:~
|
980
|
+
keyword argument or make them all tolerant by designating the pseudo-column
|
981
|
+
~:*~ as tolerant.
|
982
|
+
|
865
983
|
*** Marking Groups in Input
|
866
984
|
**** Manually
|
867
985
|
At any point, you can add a boundary to a table by invokong the
|
@@ -2549,9 +2667,10 @@ the table, but they can be overridden by more specific directives given in a
|
|
2549
2667
|
~format_for~ directive.
|
2550
2668
|
|
2551
2669
|
**** Type and Column priority
|
2552
|
-
A directive based
|
2553
|
-
|
2554
|
-
|
2670
|
+
A directive based the column name overrides any directive based on type. If
|
2671
|
+
any cell has both a type-based formatting and column-based, the column
|
2672
|
+
instructions prevail. In earlier versions the instuctions were "merged" but
|
2673
|
+
that is no longer the case.
|
2555
2674
|
|
2556
2675
|
However, there is a twist. Since the end result of formatting is to convert
|
2557
2676
|
all columns to strings, the formatting directives for the ~String~ type can
|
data/TODO.org
CHANGED
@@ -1,10 +1,37 @@
|
|
1
|
+
|
2
|
+
* TODO Specify Column Widths
|
3
|
+
Allow a formatter to specify column widths. This could be a number of
|
4
|
+
characters, which would be interpreted as a number of "ems" for LaTeX.
|
5
|
+
Cell content larger than the width would be truncated. Any column without a
|
6
|
+
width specified would be set at the width of the longest value in that cell,
|
7
|
+
after initial formatting.
|
8
|
+
|
9
|
+
#+begin_SRC ruby
|
10
|
+
tab.to_text do |f|
|
11
|
+
f.widths(a: 13, b: 30)
|
12
|
+
end
|
13
|
+
#+end_SRC
|
14
|
+
|
15
|
+
Possible enhancements:
|
16
|
+
- specify an overall width and column widths as decimal or fractions, so that
|
17
|
+
a column's width would be that fraction of the overall width.
|
18
|
+
- specify a Range for a width, so that the column would at least min and at
|
19
|
+
most max, otherwise the width of its largest cell.
|
20
|
+
|
1
21
|
* TODO Conversion to Spreadsheets
|
2
22
|
- State "TODO" from [2017-04-21 Fri 10:36]
|
3
23
|
This is a [[https://github.com/westonganger/spreadsheet_architect][gem]] that I can include into the Table model to convert a table into
|
4
24
|
a spread-sheet, or even a sheet in a multi-sheet spreadsheet file.
|
5
25
|
|
6
|
-
* TODO Add
|
26
|
+
* TODO Add Quandl or EODDATA Queries
|
27
|
+
Possible replacements for YQL.
|
28
|
+
|
29
|
+
* CNCL Add from_yql for fetching from Yahoo
|
30
|
+
CLOSED: [2022-01-30 Sun 06:03]
|
7
31
|
- State "TODO" from [2017-04-21 Fri 10:35]
|
32
|
+
|
33
|
+
Cancelled because Yahoo shut down the YQL api service.
|
34
|
+
|
8
35
|
Add a constructor to allow fetching stock data from yql. Perhaps grab all
|
9
36
|
available fields, then allow a select of those of interest.
|
10
37
|
|
data/lib/ext/array.rb
CHANGED
@@ -12,4 +12,21 @@ class Array
|
|
12
12
|
end
|
13
13
|
end
|
14
14
|
end
|
15
|
+
|
16
|
+
def filter_to_type(typ)
|
17
|
+
if typ == 'Boolean'
|
18
|
+
compact.select { |i| i.is_a?(TrueClass) || i.is_a?(FalseClass) }
|
19
|
+
elsif typ == 'DateTime'
|
20
|
+
compact.select { |i| i.is_a?(Date) || i.is_a?(DateTime) || i.is_a?(Time) }
|
21
|
+
.map { |i| i.to_datetime }
|
22
|
+
elsif typ == 'Numeric'
|
23
|
+
compact.select { |i| i.is_a?(Numeric) }
|
24
|
+
elsif typ == 'String'
|
25
|
+
map { |i| i.to_s }
|
26
|
+
elsif typ == 'NilClass'
|
27
|
+
self
|
28
|
+
else
|
29
|
+
raise ArgumentError, "cannot filter_to_type for type '#{typ}'"
|
30
|
+
end
|
31
|
+
end
|
15
32
|
end
|
data/lib/fat_table/column.rb
CHANGED
@@ -83,7 +83,7 @@ module FatTable
|
|
83
83
|
# col.type #=> 'Numeric'
|
84
84
|
# col.header #=> :prices
|
85
85
|
# col.sum #=> 18376.75
|
86
|
-
def initialize(header:, items: [], type: 'NilClass')
|
86
|
+
def initialize(header:, items: [], type: 'NilClass', tolerant: false)
|
87
87
|
@raw_header = header
|
88
88
|
@header =
|
89
89
|
if @raw_header.is_a?(Symbol)
|
@@ -92,6 +92,7 @@ module FatTable
|
|
92
92
|
@raw_header.to_s.as_sym
|
93
93
|
end
|
94
94
|
@type = type
|
95
|
+
@tolerant = tolerant
|
95
96
|
msg = "unknown column type '#{type}"
|
96
97
|
raise UserError, msg unless TYPES.include?(@type.to_s)
|
97
98
|
|
@@ -141,6 +142,14 @@ module FatTable
|
|
141
142
|
|
142
143
|
# :category: Attributes
|
143
144
|
|
145
|
+
# Is this column tolerant of type incompatibilities? If so, the Column
|
146
|
+
# type will be forced to String if an incompatible type is found.
|
147
|
+
def tolerant?
|
148
|
+
@tolerant
|
149
|
+
end
|
150
|
+
|
151
|
+
# :category: Attributes
|
152
|
+
|
144
153
|
# Force the column to have String type and then convert all items to
|
145
154
|
# strings.
|
146
155
|
def force_string!
|
@@ -182,12 +191,15 @@ module FatTable
|
|
182
191
|
|
183
192
|
# :category: Aggregates
|
184
193
|
|
185
|
-
# Return the first non-nil item in the Column
|
194
|
+
# Return the first non-nil item in the Column, or nil if all items are
|
195
|
+
# nil. Works with any Column type.
|
186
196
|
def first
|
197
|
+
return nil if items.all?(&:nil?)
|
198
|
+
|
187
199
|
if type == 'String'
|
188
200
|
items.reject(&:blank?).first
|
189
201
|
else
|
190
|
-
items.
|
202
|
+
items.filter_to_type(type).first
|
191
203
|
end
|
192
204
|
end
|
193
205
|
|
@@ -195,83 +207,94 @@ module FatTable
|
|
195
207
|
|
196
208
|
# Return the last non-nil item in the Column. Works with any Column type.
|
197
209
|
def last
|
210
|
+
return nil if items.all?(&:nil?)
|
211
|
+
|
198
212
|
if type == 'String'
|
199
213
|
items.reject(&:blank?).last
|
200
214
|
else
|
201
|
-
items.
|
215
|
+
items.filter_to_type(type).last
|
202
216
|
end
|
203
217
|
end
|
204
218
|
|
205
219
|
# :category: Aggregates
|
206
220
|
|
207
|
-
# Return a count of the non-nil items in the Column
|
208
|
-
# type.
|
221
|
+
# Return a count of the non-nil items in the Column, or the size of the
|
222
|
+
# column if all items are nil. Works with any Column type.
|
209
223
|
def count
|
224
|
+
return items.size if items.all?(&:nil?)
|
225
|
+
|
210
226
|
if type == 'String'
|
211
227
|
items.reject(&:blank?).count.to_d
|
212
228
|
else
|
213
|
-
items.
|
229
|
+
items.filter_to_type(type).count.to_d
|
214
230
|
end
|
215
231
|
end
|
216
232
|
|
217
233
|
# :category: Aggregates
|
218
234
|
|
219
|
-
# Return the smallest non-nil, non-blank item in the Column
|
220
|
-
# numeric, string, and datetime Columns.
|
235
|
+
# Return the smallest non-nil, non-blank item in the Column, or nil if all
|
236
|
+
# items are nil. Works with numeric, string, and datetime Columns.
|
221
237
|
def min
|
222
238
|
only_with('min', 'NilClass', 'Numeric', 'String', 'DateTime')
|
223
239
|
if type == 'String'
|
224
240
|
items.reject(&:blank?).min
|
225
241
|
else
|
226
|
-
items.
|
242
|
+
items.filter_to_type(type).min
|
227
243
|
end
|
228
244
|
end
|
229
245
|
|
230
246
|
# :category: Aggregates
|
231
247
|
|
232
|
-
# Return the largest non-nil, non-blank item in the Column
|
233
|
-
# numeric, string, and datetime Columns.
|
248
|
+
# Return the largest non-nil, non-blank item in the Column, or nil if all
|
249
|
+
# items are nil. Works with numeric, string, and datetime Columns.
|
234
250
|
def max
|
235
251
|
only_with('max', 'NilClass', 'Numeric', 'String', 'DateTime')
|
236
252
|
if type == 'String'
|
237
253
|
items.reject(&:blank?).max
|
238
254
|
else
|
239
|
-
items.
|
255
|
+
items.filter_to_type(type).max
|
240
256
|
end
|
241
257
|
end
|
242
258
|
|
243
259
|
# :category: Aggregates
|
244
260
|
|
245
|
-
# Return a Range object for the smallest to largest value in the column
|
246
|
-
# Works with numeric, string, and datetime
|
261
|
+
# Return a Range object for the smallest to largest value in the column,
|
262
|
+
# or nil if all items are nil. Works with numeric, string, and datetime
|
263
|
+
# Columns.
|
247
264
|
def range
|
248
265
|
only_with('range', 'NilClass', 'Numeric', 'String', 'DateTime')
|
266
|
+
return nil if items.all?(&:nil?)
|
267
|
+
|
249
268
|
Range.new(min, max)
|
250
269
|
end
|
251
270
|
|
252
271
|
# :category: Aggregates
|
253
272
|
|
254
|
-
# Return the sum of the non-nil items in the Column
|
255
|
-
# string Columns. For a string Column, it
|
256
|
-
# the non-nil items.
|
273
|
+
# Return the sum of the non-nil items in the Column, or 0 if all items are
|
274
|
+
# nil. Works with numeric and string Columns. For a string Column, it
|
275
|
+
# will return the concatenation of the non-nil items.
|
257
276
|
def sum
|
277
|
+
return 0 if type == 'NilClass' || items.all?(&:nil?)
|
278
|
+
|
258
279
|
only_with('sum', 'Numeric', 'String')
|
259
280
|
if type == 'String'
|
260
281
|
items.reject(&:blank?).join(' ')
|
261
282
|
else
|
262
|
-
items.
|
283
|
+
items.filter_to_type(type).sum
|
263
284
|
end
|
264
285
|
end
|
265
286
|
|
266
287
|
# :category: Aggregates
|
267
288
|
|
268
|
-
# Return the average value of the non-nil items in the Column
|
269
|
-
# numeric and datetime Columns. For datetime
|
270
|
-
# to its Julian day number, computes the
|
271
|
-
# average back to a DateTime.
|
289
|
+
# Return the average value of the non-nil items in the Column, or 0 if all
|
290
|
+
# items are nil. Works with numeric and datetime Columns. For datetime
|
291
|
+
# Columns, it converts each date to its Julian day number, computes the
|
292
|
+
# average, and then converts the average back to a DateTime.
|
272
293
|
def avg
|
294
|
+
return 0 if type == 'NilClass' || items.all?(&:nil?)
|
295
|
+
|
273
296
|
only_with('avg', 'DateTime', 'Numeric')
|
274
|
-
itms = items.
|
297
|
+
itms = items.filter_to_type(type)
|
275
298
|
size = itms.size.to_d
|
276
299
|
if type == 'DateTime'
|
277
300
|
avg_jd = itms.map(&:jd).sum / size
|
@@ -284,17 +307,20 @@ module FatTable
|
|
284
307
|
# :category: Aggregates
|
285
308
|
|
286
309
|
# Return the sample variance (the unbiased estimator of the population
|
287
|
-
# variance using a divisor of N-1) as the average squared deviation from
|
288
|
-
# mean, of the non-nil items in the Column
|
289
|
-
# Columns. For datetime Columns, it
|
290
|
-
# number and computes the variance of
|
310
|
+
# variance using a divisor of N-1) as the average squared deviation from
|
311
|
+
# the mean, of the non-nil items in the Column, or 0 if all items are
|
312
|
+
# nil. Works with numeric and datetime Columns. For datetime Columns, it
|
313
|
+
# converts each date to its Julian day number and computes the variance of
|
314
|
+
# those numbers.
|
291
315
|
def var
|
316
|
+
return 0 if type == 'NilClass' || items.all?(&:nil?)
|
317
|
+
|
292
318
|
only_with('var', 'DateTime', 'Numeric')
|
293
319
|
all_items =
|
294
320
|
if type == 'DateTime'
|
295
|
-
items.
|
321
|
+
items.filter_to_type(type).map(&:jd)
|
296
322
|
else
|
297
|
-
items.
|
323
|
+
items.filter_to_type(type)
|
298
324
|
end
|
299
325
|
n = count
|
300
326
|
return BigDecimal('0.0') if n <= 1
|
@@ -310,12 +336,15 @@ module FatTable
|
|
310
336
|
|
311
337
|
# Return the population variance (the biased estimator of the population
|
312
338
|
# variance using a divisor of N) as the average squared deviation from the
|
313
|
-
# mean, of the non-nil items in the Column
|
314
|
-
# Columns. For datetime Columns, it
|
315
|
-
# number and computes the variance of
|
339
|
+
# mean, of the non-nil items in the Column, or 0 if all items are
|
340
|
+
# nil. Works with numeric and datetime Columns. For datetime Columns, it
|
341
|
+
# converts each date to its Julian day number and computes the variance of
|
342
|
+
# those numbers.
|
316
343
|
def pvar
|
344
|
+
return 0 if type == 'NilClass' || items.all?(&:nil?)
|
345
|
+
|
317
346
|
only_with('var', 'DateTime', 'Numeric')
|
318
|
-
n = items.
|
347
|
+
n = items.filter_to_type(type).size.to_d
|
319
348
|
return BigDecimal('0.0') if n <= 1
|
320
349
|
var * ((n - 1) / n)
|
321
350
|
end
|
@@ -324,11 +353,13 @@ module FatTable
|
|
324
353
|
|
325
354
|
# Return the sample standard deviation (the unbiased estimator of the
|
326
355
|
# population standard deviation using a divisor of N-1) as the square root
|
327
|
-
# of the sample variance, of the non-nil items in the Column
|
328
|
-
# numeric and datetime Columns. For datetime
|
329
|
-
# to its Julian day number and computes the
|
330
|
-
# numbers.
|
356
|
+
# of the sample variance, of the non-nil items in the Column, or 0 if all
|
357
|
+
# items are nil. Works with numeric and datetime Columns. For datetime
|
358
|
+
# Columns, it converts each date to its Julian day number and computes the
|
359
|
+
# standard deviation of those numbers.
|
331
360
|
def dev
|
361
|
+
return 0 if type == 'NilClass' || items.all?(&:nil?)
|
362
|
+
|
332
363
|
only_with('dev', 'DateTime', 'Numeric')
|
333
364
|
var.sqrt(20)
|
334
365
|
end
|
@@ -336,12 +367,14 @@ module FatTable
|
|
336
367
|
# :category: Aggregates
|
337
368
|
|
338
369
|
# Return the population standard deviation (the biased estimator of the
|
339
|
-
# population standard deviation using a divisor of N) as the square root
|
340
|
-
# the population variance, of the non-nil items in the Column
|
341
|
-
# numeric and datetime Columns. For datetime
|
342
|
-
# to its Julian day number and computes the
|
343
|
-
# numbers.
|
370
|
+
# population standard deviation using a divisor of N) as the square root
|
371
|
+
# of the population variance, of the non-nil items in the Column, or 0 if
|
372
|
+
# all items are nil. Works with numeric and datetime Columns. For datetime
|
373
|
+
# Columns, it converts each date to its Julian day number and computes the
|
374
|
+
# standard deviation of those numbers.
|
344
375
|
def pdev
|
376
|
+
return 0 if type == 'NilClass' || items.all?(&:nil?)
|
377
|
+
|
345
378
|
only_with('dev', 'DateTime', 'Numeric')
|
346
379
|
Math.sqrt(pvar)
|
347
380
|
end
|
@@ -349,28 +382,35 @@ module FatTable
|
|
349
382
|
# :category: Aggregates
|
350
383
|
|
351
384
|
# Return true if any of the items in the Column are true; otherwise return
|
352
|
-
# false. Works only with boolean Columns.
|
385
|
+
# false, or false if all items are nil. Works only with boolean Columns.
|
353
386
|
def any?
|
387
|
+
return false if type == 'NilClass' || items.all?(&:nil?)
|
388
|
+
|
354
389
|
only_with('any?', 'Boolean')
|
355
|
-
items.
|
390
|
+
items.filter_to_type(type).any?
|
356
391
|
end
|
357
392
|
|
358
393
|
# :category: Aggregates
|
359
394
|
|
360
395
|
# Return true if all of the items in the Column are true; otherwise return
|
361
|
-
# false. Works only with boolean Columns.
|
396
|
+
# false, or false if all items are nil. Works only with boolean Columns.
|
362
397
|
def all?
|
398
|
+
return false if type == 'NilClass' || items.all?(&:nil?)
|
399
|
+
|
363
400
|
only_with('all?', 'Boolean')
|
364
|
-
items.
|
401
|
+
items.filter_to_type(type).all?
|
365
402
|
end
|
366
403
|
|
367
404
|
# :category: Aggregates
|
368
405
|
|
369
|
-
# Return true if none of the items in the Column are true; otherwise
|
370
|
-
# false. Works only with boolean
|
406
|
+
# Return true if none of the items in the Column are true; otherwise
|
407
|
+
# return false, or true if all items are nil. Works only with boolean
|
408
|
+
# Columns.
|
371
409
|
def none?
|
410
|
+
return true if type == 'NilClass' || items.all?(&:nil?)
|
411
|
+
|
372
412
|
only_with('none?', 'Boolean')
|
373
|
-
items.
|
413
|
+
items.filter_to_type(type).none?
|
374
414
|
end
|
375
415
|
|
376
416
|
# :category: Aggregates
|
@@ -378,14 +418,17 @@ module FatTable
|
|
378
418
|
# Return true if precisely one of the items in the Column is true;
|
379
419
|
# otherwise return false. Works only with boolean Columns.
|
380
420
|
def one?
|
421
|
+
return false if type == 'NilClass' || items.all?(&:nil?)
|
422
|
+
|
381
423
|
only_with('one?', 'Boolean')
|
382
|
-
items.
|
424
|
+
items.filter_to_type(type).one?
|
383
425
|
end
|
384
426
|
|
385
427
|
private
|
386
428
|
|
387
429
|
def only_with(agg, *valid_types)
|
388
430
|
return self if valid_types.include?(type)
|
431
|
+
|
389
432
|
msg = "aggregate '#{agg}' cannot be applied to a #{type} column"
|
390
433
|
raise UserError, msg
|
391
434
|
end
|
@@ -400,9 +443,17 @@ module FatTable
|
|
400
443
|
|
401
444
|
# Append +itm+ to end of the Column after converting it to the Column's
|
402
445
|
# type. If the Column's type is still open, i.e. NilClass, attempt to fix
|
403
|
-
# the Column's type based on the type of +itm+ as with Column.new.
|
446
|
+
# the Column's type based on the type of +itm+ as with Column.new. If its
|
447
|
+
# a tolerant column, respond to type errors by converting the column to a
|
448
|
+
# String type.
|
404
449
|
def <<(itm)
|
405
|
-
items <<
|
450
|
+
items << convert_and_set_type(itm)
|
451
|
+
rescue IncompatibleTypeError => ex
|
452
|
+
if tolerant?
|
453
|
+
items << Convert.convert_to_string(itm)
|
454
|
+
else
|
455
|
+
raise ex
|
456
|
+
end
|
406
457
|
end
|
407
458
|
|
408
459
|
# :category: Constructors
|
@@ -418,9 +469,14 @@ module FatTable
|
|
418
469
|
|
419
470
|
private
|
420
471
|
|
421
|
-
def
|
422
|
-
|
423
|
-
|
472
|
+
def convert_and_set_type(val)
|
473
|
+
begin
|
474
|
+
new_val = Convert.convert_to_type(val, type, tolerant: tolerant?)
|
475
|
+
rescue IncompatibleTypeError
|
476
|
+
err_msg = "attempt to add '#{val}' to column '#{header}' already typed as #{type}"
|
477
|
+
raise IncompatibleTypeError, err_msg
|
478
|
+
end
|
479
|
+
if new_val && (type == 'NilClass' || type == 'String')
|
424
480
|
@type =
|
425
481
|
if [true, false].include?(new_val)
|
426
482
|
'Boolean'
|
data/lib/fat_table/convert.rb
CHANGED
@@ -10,7 +10,7 @@ module FatTable
|
|
10
10
|
# determined, raise an error if the val cannot be converted to the Column
|
11
11
|
# type. Otherwise, returns the converted val as an object of the correct
|
12
12
|
# class.
|
13
|
-
def self.convert_to_type(val, type)
|
13
|
+
def self.convert_to_type(val, type, tolerant: false)
|
14
14
|
case type
|
15
15
|
when 'NilClass'
|
16
16
|
if val != false && val.blank?
|
@@ -36,8 +36,7 @@ module FatTable
|
|
36
36
|
else
|
37
37
|
new_val = convert_to_boolean(val)
|
38
38
|
if new_val.nil?
|
39
|
-
|
40
|
-
raise UserError, msg
|
39
|
+
raise IncompatibleTypeError
|
41
40
|
end
|
42
41
|
new_val
|
43
42
|
end
|
@@ -47,8 +46,7 @@ module FatTable
|
|
47
46
|
else
|
48
47
|
new_val = convert_to_date_time(val)
|
49
48
|
if new_val.nil?
|
50
|
-
|
51
|
-
raise UserError, msg
|
49
|
+
raise IncompatibleTypeError
|
52
50
|
end
|
53
51
|
new_val
|
54
52
|
end
|
@@ -58,24 +56,35 @@ module FatTable
|
|
58
56
|
else
|
59
57
|
new_val = convert_to_numeric(val)
|
60
58
|
if new_val.nil?
|
61
|
-
|
62
|
-
raise UserError, msg
|
59
|
+
raise IncompatibleTypeError
|
63
60
|
end
|
64
61
|
new_val
|
65
62
|
end
|
66
63
|
when 'String'
|
67
64
|
if val.nil?
|
68
65
|
nil
|
66
|
+
elsif tolerant
|
67
|
+
# Allow String to upgrade to one of Numeric, DateTime, or Boolean if
|
68
|
+
# possible.
|
69
|
+
if (new_val = convert_to_numeric(val))
|
70
|
+
new_val
|
71
|
+
elsif (new_val = convert_to_date_time(val))
|
72
|
+
new_val
|
73
|
+
elsif (new_val = convert_to_boolean(val))
|
74
|
+
new_val
|
75
|
+
else
|
76
|
+
new_val = convert_to_string(val)
|
77
|
+
end
|
78
|
+
new_val
|
69
79
|
else
|
70
80
|
new_val = convert_to_string(val)
|
71
81
|
if new_val.nil?
|
72
|
-
|
73
|
-
raise UserError, msg
|
82
|
+
raise IncompatibleTypeError
|
74
83
|
end
|
75
84
|
new_val
|
76
85
|
end
|
77
86
|
else
|
78
|
-
raise
|
87
|
+
raise LogicError, "Mysteriously, column has unknown type '#{type}'"
|
79
88
|
end
|
80
89
|
end
|
81
90
|
|
@@ -121,6 +130,7 @@ module FatTable
|
|
121
130
|
return val if val.is_a?(DateTime)
|
122
131
|
return val if val.is_a?(Date)
|
123
132
|
return val.to_datetime if val.is_a?(Time)
|
133
|
+
|
124
134
|
begin
|
125
135
|
str = val.to_s.clean
|
126
136
|
return nil if str.blank?
|
data/lib/fat_table/errors.rb
CHANGED
@@ -9,6 +9,10 @@ module FatTable
|
|
9
9
|
# cannot correct.
|
10
10
|
class LogicError < StandardError; end
|
11
11
|
|
12
|
+
# Raised when attempting to add an incompatible type to an already-typed
|
13
|
+
# Column.
|
14
|
+
class IncompatibleTypeError < UserError; end
|
15
|
+
|
12
16
|
# Raised when an external resource is not available due to caller or
|
13
17
|
# programmer error or some failure of the external resource to be available.
|
14
18
|
class TransientError < StandardError; end
|
data/lib/fat_table/evaluator.rb
CHANGED
@@ -45,17 +45,15 @@ module FatTable
|
|
45
45
|
end
|
46
46
|
|
47
47
|
# Return the result of evaluating +expr+ as a Ruby expression in which the
|
48
|
-
# instance variables set in Evaluator.new and any local variables set in
|
49
|
-
# Hash parameter +locals+ are available to the expression.
|
48
|
+
# instance variables set in Evaluator.new and any local variables set in
|
49
|
+
# the Hash parameter +locals+ are available to the expression. Certain
|
50
|
+
# errors simply return nil as the result. This can happen, for example,
|
51
|
+
# when a string gets into an otherwise numeric column because the column
|
52
|
+
# is set to tolerant.
|
50
53
|
def evaluate(expr = '', locals: {})
|
51
54
|
eval(expr, local_vars(binding, locals))
|
52
55
|
rescue NoMethodError, TypeError => ex
|
53
|
-
|
54
|
-
# Likely one of the locals was nil, so let nil be the result.
|
55
|
-
return nil
|
56
|
-
else
|
57
|
-
raise ex
|
58
|
-
end
|
56
|
+
nil
|
59
57
|
end
|
60
58
|
|
61
59
|
private
|
data/lib/fat_table/footer.rb
CHANGED
@@ -532,7 +532,7 @@ module FatTable
|
|
532
532
|
valid_keys = table.headers + %i[string numeric datetime boolean nil]
|
533
533
|
invalid_keys = (fmts.keys - valid_keys).uniq
|
534
534
|
unless invalid_keys.empty?
|
535
|
-
msg = "invalid #{location} column or type: #{invalid_keys.join(',')}"
|
535
|
+
msg = "invalid #{location} column or type: #{invalid_keys.join(', ')}"
|
536
536
|
raise UserError, msg
|
537
537
|
end
|
538
538
|
|
@@ -562,20 +562,21 @@ module FatTable
|
|
562
562
|
end
|
563
563
|
end
|
564
564
|
|
565
|
-
# Merge in formatting for column h based on the column
|
566
|
-
#
|
565
|
+
# Merge in formatting instructions for column h based on the column
|
566
|
+
# name, or if there is no formatting instructions for the column by
|
567
|
+
# name, merge in the formatting instructions based on the column's
|
568
|
+
# type. Insist on only the string type for the header location.
|
567
569
|
typ = (location == :header ? :string : table.type(h).as_sym)
|
568
570
|
parse_typ_method_name = 'parse_' + typ.to_s + '_fmt'
|
569
|
-
if fmts.key?(typ)
|
570
|
-
# Merge in type-based formatting
|
571
|
-
typ_fmt = send(parse_typ_method_name, fmts[typ]).first
|
572
|
-
format_h = format_h.merge(typ_fmt)
|
573
|
-
end
|
574
571
|
if fmts[h]
|
575
572
|
# Merge in column formatting
|
576
573
|
col_fmt = send(parse_typ_method_name, fmts[h],
|
577
574
|
strict: location != :header).first
|
578
575
|
format_h = format_h.merge(col_fmt)
|
576
|
+
elsif fmts.key?(typ)
|
577
|
+
# Merge in type-based formatting
|
578
|
+
typ_fmt = send(parse_typ_method_name, fmts[typ]).first
|
579
|
+
format_h = format_h.merge(typ_fmt)
|
579
580
|
end
|
580
581
|
|
581
582
|
# Copy :body formatting for column h to :bfirst and :gfirst if they
|
@@ -1003,9 +1004,14 @@ module FatTable
|
|
1003
1004
|
# converted to strings formatted according to the Formatter's formatting
|
1004
1005
|
# directives given in Formatter.format_for or Formatter.format.
|
1005
1006
|
def output
|
1006
|
-
#
|
1007
|
-
#
|
1008
|
-
|
1007
|
+
# If there are neither headers nor any rows in the table, return an
|
1008
|
+
# empty string.
|
1009
|
+
return '' if table.empty? && table.headers.empty?
|
1010
|
+
|
1011
|
+
# This results in a hash of two-element arrays. The key
|
1012
|
+
# is the header and the value is an array of the header and formatted
|
1013
|
+
# header. We do the latter so the structure parallels the structure for
|
1014
|
+
# rows explained next.
|
1009
1015
|
formatted_headers = build_formatted_headers
|
1010
1016
|
|
1011
1017
|
# These produce an array with each element representing a row of the
|
@@ -20,7 +20,7 @@ module FatTable
|
|
20
20
|
end
|
21
21
|
|
22
22
|
def pre_header(widths)
|
23
|
-
result = '|'
|
23
|
+
result = +'|'
|
24
24
|
widths.each_value do |w|
|
25
25
|
result += '-' * (w + 2) + '+'
|
26
26
|
end
|
@@ -53,7 +53,7 @@ module FatTable
|
|
53
53
|
end
|
54
54
|
|
55
55
|
def hline(widths)
|
56
|
-
result = '|'
|
56
|
+
result = +'|'
|
57
57
|
widths.each_value do |w|
|
58
58
|
result += '-' * (w + 2) + '+'
|
59
59
|
end
|
@@ -62,7 +62,7 @@ module FatTable
|
|
62
62
|
end
|
63
63
|
|
64
64
|
def post_footers(widths)
|
65
|
-
result = '|'
|
65
|
+
result = +'|'
|
66
66
|
widths.each_value do |w|
|
67
67
|
result += '-' * (w + 2) + '+'
|
68
68
|
end
|
@@ -221,7 +221,7 @@ module FatTable
|
|
221
221
|
end
|
222
222
|
|
223
223
|
def pre_header(widths)
|
224
|
-
result = upper_left
|
224
|
+
result = +upper_left
|
225
225
|
widths.each_value do |w|
|
226
226
|
result += double_rule * (w + 2) + upper_tee
|
227
227
|
end
|
@@ -255,7 +255,7 @@ module FatTable
|
|
255
255
|
end
|
256
256
|
|
257
257
|
def hline(widths)
|
258
|
-
result = left_tee
|
258
|
+
result = +left_tee
|
259
259
|
widths.each_value do |w|
|
260
260
|
result += horizontal_rule * (w + 2) + single_cross
|
261
261
|
end
|
@@ -289,7 +289,7 @@ module FatTable
|
|
289
289
|
end
|
290
290
|
|
291
291
|
def post_footers(widths)
|
292
|
-
result = lower_left
|
292
|
+
result = +lower_left
|
293
293
|
widths.each_value do |w|
|
294
294
|
result += double_rule * (w + 2) + lower_tee
|
295
295
|
end
|
@@ -16,7 +16,7 @@ module FatTable
|
|
16
16
|
end
|
17
17
|
|
18
18
|
def pre_header(widths)
|
19
|
-
result = '+'
|
19
|
+
result = +'+'
|
20
20
|
widths.each_value do |w|
|
21
21
|
result += '=' * (w + 2) + '+'
|
22
22
|
end
|
@@ -49,7 +49,7 @@ module FatTable
|
|
49
49
|
end
|
50
50
|
|
51
51
|
def hline(widths)
|
52
|
-
result = '+'
|
52
|
+
result = +'+'
|
53
53
|
widths.each_value do |w|
|
54
54
|
result += '-' * (w + 2) + '+'
|
55
55
|
end
|
@@ -82,7 +82,7 @@ module FatTable
|
|
82
82
|
end
|
83
83
|
|
84
84
|
def post_footers(widths)
|
85
|
-
result = '+'
|
85
|
+
result = +'+'
|
86
86
|
widths.each_value do |w|
|
87
87
|
result += '=' * (w + 2) + '+'
|
88
88
|
end
|
data/lib/fat_table/table.rb
CHANGED
@@ -62,19 +62,53 @@ module FatTable
|
|
62
62
|
# method call.
|
63
63
|
attr_accessor :explicit_boundaries
|
64
64
|
|
65
|
+
# An Array of FatTable::Columns that should be tolerant.
|
66
|
+
attr_reader :tolerant_columns
|
67
|
+
|
65
68
|
###########################################################################
|
66
69
|
# Constructors
|
67
70
|
###########################################################################
|
68
71
|
|
69
72
|
# :category: Constructors
|
70
73
|
|
71
|
-
# Return an empty FatTable::Table object.
|
72
|
-
|
74
|
+
# Return an empty FatTable::Table object. Specifying headers is optional.
|
75
|
+
# Any headers ending with a ! are marked as tolerant, in that, if an
|
76
|
+
# incompatible type is added to it, the column is re-typed as a String
|
77
|
+
# column, and construction proceeds. The ! is stripped from the header to
|
78
|
+
# form the column key, though. You can also provide the names of columns
|
79
|
+
# that should be tolerant by using the +tolerant_columns key-word to
|
80
|
+
# provide an array of headers that should be tolerant. The special string
|
81
|
+
# '*' or the symbol :* indicates that all columns should be created
|
82
|
+
# tolerant.
|
83
|
+
def initialize(*heads, tolerant_columns: [])
|
73
84
|
@columns = []
|
74
85
|
@explicit_boundaries = []
|
86
|
+
@tolerant_columns =
|
87
|
+
case tolerant_columns
|
88
|
+
when Array
|
89
|
+
tolerant_columns.map { |h| h.to_s.as_sym }
|
90
|
+
when String
|
91
|
+
if tolerant_columns.strip == '*'
|
92
|
+
['*'.to_sym]
|
93
|
+
else
|
94
|
+
[tolerant_columns.as_sym]
|
95
|
+
end
|
96
|
+
when Symbol
|
97
|
+
if tolerant_columns.to_s.strip == '*'
|
98
|
+
['*'.to_sym]
|
99
|
+
else
|
100
|
+
[tolerant_columns.to_s.as_sym]
|
101
|
+
end
|
102
|
+
else
|
103
|
+
raise ArgumentError, "set tolerant_columns to String, Symbol, or an Array of either"
|
104
|
+
end
|
75
105
|
unless heads.empty?
|
76
106
|
heads.each do |h|
|
77
|
-
|
107
|
+
if h.to_s.end_with?('!') || @tolerant_columns.include?(h)
|
108
|
+
@columns << Column.new(header: h.to_s.sub(/!\s*\z/, ''), type: 'String')
|
109
|
+
else
|
110
|
+
@columns << Column.new(header: h)
|
111
|
+
end
|
78
112
|
end
|
79
113
|
end
|
80
114
|
end
|
@@ -86,7 +120,7 @@ module FatTable
|
|
86
120
|
# though FatTable::Table objects have no instance variables, a class that
|
87
121
|
# inherits from it might.
|
88
122
|
def empty_dup
|
89
|
-
|
123
|
+
dup.__empty!
|
90
124
|
end
|
91
125
|
|
92
126
|
def __empty!
|
@@ -99,9 +133,9 @@ module FatTable
|
|
99
133
|
|
100
134
|
# Construct a Table from the contents of a CSV file named +fname+. Headers
|
101
135
|
# will be taken from the first CSV row and converted to symbols.
|
102
|
-
def self.from_csv_file(fname)
|
136
|
+
def self.from_csv_file(fname, tolerant_columns: [])
|
103
137
|
File.open(fname, 'r') do |io|
|
104
|
-
from_csv_io(io)
|
138
|
+
from_csv_io(io, tolerant_columns: tolerant_columns)
|
105
139
|
end
|
106
140
|
end
|
107
141
|
|
@@ -109,8 +143,8 @@ module FatTable
|
|
109
143
|
|
110
144
|
# Construct a Table from a CSV string +str+, treated in the same manner as
|
111
145
|
# the input from a CSV file in ::from_org_file.
|
112
|
-
def self.from_csv_string(str)
|
113
|
-
from_csv_io(StringIO.new(str))
|
146
|
+
def self.from_csv_string(str, tolerant_columns: [])
|
147
|
+
from_csv_io(StringIO.new(str), tolerant_columns: tolerant_columns)
|
114
148
|
end
|
115
149
|
|
116
150
|
# :category: Constructors
|
@@ -119,9 +153,9 @@ module FatTable
|
|
119
153
|
# file named +fname+. Headers are taken from the first row if the second row
|
120
154
|
# is an hrule. Otherwise, synthetic headers of the form +:col_1+, +:col_2+,
|
121
155
|
# etc. are created.
|
122
|
-
def self.from_org_file(fname)
|
156
|
+
def self.from_org_file(fname, tolerant_columns: [])
|
123
157
|
File.open(fname, 'r') do |io|
|
124
|
-
from_org_io(io)
|
158
|
+
from_org_io(io, tolerant_columns: tolerant_columns)
|
125
159
|
end
|
126
160
|
end
|
127
161
|
|
@@ -129,8 +163,8 @@ module FatTable
|
|
129
163
|
|
130
164
|
# Construct a Table from a string +str+, treated in the same manner as the
|
131
165
|
# contents of an org-mode file in ::from_org_file.
|
132
|
-
def self.from_org_string(str)
|
133
|
-
from_org_io(StringIO.new(str))
|
166
|
+
def self.from_org_string(str, tolerant_columns: [])
|
167
|
+
from_org_io(StringIO.new(str), tolerant_columns: tolerant_columns)
|
134
168
|
end
|
135
169
|
|
136
170
|
# :category: Constructors
|
@@ -149,8 +183,8 @@ module FatTable
|
|
149
183
|
# :hlines no +) org-mode strips all hrules from the table; otherwise (+
|
150
184
|
# HEADER: :hlines yes +) they are indicated with nil elements in the outer
|
151
185
|
# array.
|
152
|
-
def self.from_aoa(aoa, hlines: false)
|
153
|
-
from_array_of_arrays(aoa, hlines: hlines)
|
186
|
+
def self.from_aoa(aoa, hlines: false, tolerant_columns: [])
|
187
|
+
from_array_of_arrays(aoa, hlines: hlines, tolerant_columns: tolerant_columns)
|
154
188
|
end
|
155
189
|
|
156
190
|
# :category: Constructors
|
@@ -160,9 +194,9 @@ module FatTable
|
|
160
194
|
# keys, which, when converted to symbols will become the headers for the
|
161
195
|
# Table. If hlines is set true, mark a group boundary whenever a nil, rather
|
162
196
|
# than a hash appears in the outer array.
|
163
|
-
def self.from_aoh(aoh, hlines: false)
|
197
|
+
def self.from_aoh(aoh, hlines: false, tolerant_columns: [])
|
164
198
|
if aoh.first.respond_to?(:to_h)
|
165
|
-
from_array_of_hashes(aoh, hlines: hlines)
|
199
|
+
from_array_of_hashes(aoh, hlines: hlines, tolerant_columns: tolerant_columns)
|
166
200
|
else
|
167
201
|
raise UserError,
|
168
202
|
"Cannot initialize Table with an array of #{input[0].class}"
|
@@ -181,7 +215,7 @@ module FatTable
|
|
181
215
|
|
182
216
|
# Construct a Table by running a SQL +query+ against the database set up
|
183
217
|
# with FatTable.connect, with the rows of the query result as rows.
|
184
|
-
def self.from_sql(query)
|
218
|
+
def self.from_sql(query, tolerant_columns: [])
|
185
219
|
msg = 'FatTable.db must be set with FatTable.connect'
|
186
220
|
raise UserError, msg if FatTable.db.nil?
|
187
221
|
|
@@ -203,8 +237,8 @@ module FatTable
|
|
203
237
|
# Construct table from an array of hashes or an array of any object that
|
204
238
|
# can respond to #to_h. If an array element is a nil, mark it as a group
|
205
239
|
# boundary in the Table.
|
206
|
-
def from_array_of_hashes(hashes, hlines: false)
|
207
|
-
result = new
|
240
|
+
def from_array_of_hashes(hashes, hlines: false, tolerant_columns: [])
|
241
|
+
result = new(tolerant_columns: tolerant_columns)
|
208
242
|
hashes.each do |hsh|
|
209
243
|
if hsh.nil?
|
210
244
|
unless hlines
|
@@ -232,8 +266,8 @@ module FatTable
|
|
232
266
|
# hlines are stripped from the table, otherwise (:hlines yes) they are
|
233
267
|
# indicated with nil elements in the outer array as expected by this
|
234
268
|
# method when hlines is set true.
|
235
|
-
def from_array_of_arrays(rows, hlines: false)
|
236
|
-
result = new
|
269
|
+
def from_array_of_arrays(rows, hlines: false, tolerant_columns: [])
|
270
|
+
result = new(tolerant_columns: tolerant_columns)
|
237
271
|
headers = []
|
238
272
|
if !hlines
|
239
273
|
# Take the first row as headers
|
@@ -269,8 +303,8 @@ module FatTable
|
|
269
303
|
result
|
270
304
|
end
|
271
305
|
|
272
|
-
def from_csv_io(io)
|
273
|
-
result = new
|
306
|
+
def from_csv_io(io, tolerant_columns: [])
|
307
|
+
result = new(tolerant_columns: tolerant_columns)
|
274
308
|
::CSV.new(io, headers: true, header_converters: :symbol,
|
275
309
|
skip_blanks: true).each do |row|
|
276
310
|
result << row.to_h
|
@@ -283,7 +317,7 @@ module FatTable
|
|
283
317
|
# header row must be marked with an hline (i.e, a row that looks like
|
284
318
|
# '|---+--...--|') and groups of rows may be marked with hlines to
|
285
319
|
# indicate group boundaries.
|
286
|
-
def from_org_io(io)
|
320
|
+
def from_org_io(io, tolerant_columns: [])
|
287
321
|
table_re = /\A\s*\|/
|
288
322
|
hrule_re = /\A\s*\|[-+]+/
|
289
323
|
rows = []
|
@@ -318,7 +352,7 @@ module FatTable
|
|
318
352
|
rows << line.split('|').map(&:clean)
|
319
353
|
end
|
320
354
|
end
|
321
|
-
from_array_of_arrays(rows, hlines: true)
|
355
|
+
from_array_of_arrays(rows, hlines: true, tolerant_columns: tolerant_columns)
|
322
356
|
end
|
323
357
|
end
|
324
358
|
|
@@ -412,6 +446,15 @@ module FatTable
|
|
412
446
|
|
413
447
|
# :category: Attributes
|
414
448
|
|
449
|
+
# Return whether the column with the given head should be made tolerant.
|
450
|
+
def tolerant_col?(h)
|
451
|
+
return true if tolerant_columns.include?(:'*')
|
452
|
+
|
453
|
+
tolerant_columns.include?(h)
|
454
|
+
end
|
455
|
+
|
456
|
+
# :category: Attributes
|
457
|
+
|
415
458
|
# Return the number of rows in the Table.
|
416
459
|
def size
|
417
460
|
return 0 if columns.empty?
|
@@ -571,7 +614,8 @@ module FatTable
|
|
571
614
|
range = group_row_range(k)
|
572
615
|
tab_col = column(col)
|
573
616
|
gitems = tab_col.items[range]
|
574
|
-
cols << Column.new(header: col, items: gitems,
|
617
|
+
cols << Column.new(header: col, items: gitems,
|
618
|
+
type: tab_col.type, tolerant: tab_col.tolerant?)
|
575
619
|
end
|
576
620
|
cols
|
577
621
|
end
|
@@ -735,7 +779,6 @@ module FatTable
|
|
735
779
|
last_key = nil
|
736
780
|
new_rows.each_with_index do |nrow, k|
|
737
781
|
new_tab << nrow
|
738
|
-
# key = nrow.fetch_values(*sort_heads)
|
739
782
|
key = nrow.fetch_values(*key_hash.keys)
|
740
783
|
new_tab.mark_boundary(k - 1) if last_key && key != last_key
|
741
784
|
last_key = key
|
@@ -941,7 +984,12 @@ module FatTable
|
|
941
984
|
expr = expr.to_s
|
942
985
|
result = empty_dup
|
943
986
|
headers.each do |h|
|
944
|
-
col =
|
987
|
+
col =
|
988
|
+
if tolerant_col?(h)
|
989
|
+
Column.new(header: h, tolerant: true)
|
990
|
+
else
|
991
|
+
Column.new(header: h)
|
992
|
+
end
|
945
993
|
result.add_column(col)
|
946
994
|
end
|
947
995
|
ev = Evaluator.new(ivars: { row: 0, group: 0 })
|
@@ -1406,6 +1454,9 @@ module FatTable
|
|
1406
1454
|
|
1407
1455
|
private
|
1408
1456
|
|
1457
|
+
# Collapse a group of rows to a single row by applying the aggregator from
|
1458
|
+
# the +agg_cols+ to the items in that column and the presumably identical
|
1459
|
+
# value in the +grp_cols to those columns.
|
1409
1460
|
def row_from_group(rows, grp_cols, agg_cols)
|
1410
1461
|
new_row = {}
|
1411
1462
|
grp_cols.each do |h|
|
@@ -1440,7 +1491,7 @@ module FatTable
|
|
1440
1491
|
# This column is new, so it needs nil items for all prior rows lest
|
1441
1492
|
# the value be added to a prior row.
|
1442
1493
|
items = Array.new(size, nil)
|
1443
|
-
columns << Column.new(header: h, items: items)
|
1494
|
+
columns << Column.new(header: h, items: items, tolerant: tolerant_col?(h))
|
1444
1495
|
end
|
1445
1496
|
headers.each do |h|
|
1446
1497
|
# NB: This adds a nil if h is not in row.
|
@@ -1660,11 +1711,19 @@ module FatTable
|
|
1660
1711
|
end
|
1661
1712
|
|
1662
1713
|
# The <=> operator cannot handle nils without some help. Treat a nil as
|
1663
|
-
# smaller than any other value, but equal to other nils. The two keys are
|
1664
|
-
# compared with <=>.
|
1714
|
+
# smaller than any other value, but equal to other nils. The two keys are
|
1715
|
+
# assumed to be arrays of values to be compared with <=>. Since
|
1716
|
+
# tolerant_columns permit strings to be mixed in with columns of type
|
1717
|
+
# Numeric, DateTime, and Boolean, treat strings mixed with another type
|
1718
|
+
# the same as nils.
|
1665
1719
|
def compare_with_nils(key1, key2)
|
1666
1720
|
result = nil
|
1667
1721
|
key1.zip(key2) do |k1, k2|
|
1722
|
+
if k1.is_a?(String) && !k2.is_a?(String)
|
1723
|
+
k1 = nil
|
1724
|
+
elsif !k1.is_a?(String) && k2.is_a?(String)
|
1725
|
+
k2 = nil
|
1726
|
+
end
|
1668
1727
|
if k1.nil? && k2.nil?
|
1669
1728
|
result = 0
|
1670
1729
|
next
|
data/lib/fat_table/version.rb
CHANGED
data/lib/fat_table.rb
CHANGED
@@ -61,22 +61,22 @@ module FatTable
|
|
61
61
|
|
62
62
|
# Return an empty FatTable::Table object. You can use FatTable::Table#add_row
|
63
63
|
# or FatTable::Table#add_column to populate the table with data.
|
64
|
-
def self.new(*args)
|
65
|
-
Table.new(*args)
|
64
|
+
def self.new(*args, tolerant_columns: [])
|
65
|
+
Table.new(*args, tolerant_columns: tolerant_columns)
|
66
66
|
end
|
67
67
|
|
68
68
|
# Construct a FatTable::Table from the contents of a CSV file given by the
|
69
69
|
# file name +fname+. Headers will be taken from the first row and converted to
|
70
70
|
# symbols.
|
71
|
-
def self.from_csv_file(fname)
|
72
|
-
Table.from_csv_file(fname)
|
71
|
+
def self.from_csv_file(fname, tolerant_columns: [])
|
72
|
+
Table.from_csv_file(fname, tolerant_columns: tolerant_columns)
|
73
73
|
end
|
74
74
|
|
75
75
|
# Construct a FatTable::Table from the string +str+, treated in the same
|
76
76
|
# manner as if read the input from a CSV file. Headers will be taken from the
|
77
77
|
# first row and converted to symbols.
|
78
|
-
def self.from_csv_string(str)
|
79
|
-
Table.from_csv_string(str)
|
78
|
+
def self.from_csv_string(str, tolerant_columns: [])
|
79
|
+
Table.from_csv_string(str, tolerant_columns: tolerant_columns)
|
80
80
|
end
|
81
81
|
|
82
82
|
# Construct a FatTable::Table from the first table found in the Emacs org-mode
|
@@ -84,8 +84,8 @@ module FatTable
|
|
84
84
|
# is an hline. Otherwise, synthetic headers of the form +:col_1+, +:col_2+,
|
85
85
|
# etc. are created. Any other hlines will be treated as marking a boundary in
|
86
86
|
# the table.
|
87
|
-
def self.from_org_file(fname)
|
88
|
-
Table.from_org_file(fname)
|
87
|
+
def self.from_org_file(fname, tolerant_columns: [])
|
88
|
+
Table.from_org_file(fname, tolerant_columns: tolerant_columns)
|
89
89
|
end
|
90
90
|
|
91
91
|
# Construct a FatTable::Table from the first table found in the string +str+,
|
@@ -93,8 +93,8 @@ module FatTable
|
|
93
93
|
# are taken from the first row if the second row is an hrule. Otherwise,
|
94
94
|
# synthetic headers of the form :col_1, :col_2, etc. are created. Any other
|
95
95
|
# hlines will be treated as marking a boundary in the table.
|
96
|
-
def self.from_org_string(str)
|
97
|
-
Table.from_org_string(str)
|
96
|
+
def self.from_org_string(str, tolerant_columns: [])
|
97
|
+
Table.from_org_string(str, tolerant_columns: tolerant_columns)
|
98
98
|
end
|
99
99
|
|
100
100
|
# Construct a FatTable::Table from the array of arrays +aoa+. By default, with
|
@@ -108,8 +108,8 @@ module FatTable
|
|
108
108
|
# org-mode code blocks, by default (+:hlines no+) all hlines are stripped from
|
109
109
|
# the table, otherwise (+:hlines yes+) they are indicated with nil elements in
|
110
110
|
# the outer array.
|
111
|
-
def self.from_aoa(aoa, hlines: false)
|
112
|
-
Table.from_aoa(aoa, hlines: hlines)
|
111
|
+
def self.from_aoa(aoa, hlines: false, tolerant_columns: [])
|
112
|
+
Table.from_aoa(aoa, hlines: hlines, tolerant_columns: tolerant_columns)
|
113
113
|
end
|
114
114
|
|
115
115
|
# Construct a FatTable::Table from the array of hashes +aoh+, which can be an
|
@@ -117,8 +117,8 @@ module FatTable
|
|
117
117
|
# interpret nil separators as marking boundaries in the new Table. All hashes
|
118
118
|
# must have the same keys, which, converted to symbols, become the headers for
|
119
119
|
# the new Table.
|
120
|
-
def self.from_aoh(aoh, hlines: false)
|
121
|
-
Table.from_aoh(aoh, hlines: hlines)
|
120
|
+
def self.from_aoh(aoh, hlines: false, tolerant_columns: [])
|
121
|
+
Table.from_aoh(aoh, hlines: hlines, tolerant_columns: tolerant_columns)
|
122
122
|
end
|
123
123
|
|
124
124
|
# Construct a FatTable::Table from another FatTable::Table. Inherit any group
|
@@ -130,8 +130,8 @@ module FatTable
|
|
130
130
|
# Construct a Table by running a SQL query against the database set up with
|
131
131
|
# FatTable.connect. Return the Table with the query results as rows and the
|
132
132
|
# headers from the query, converted to symbols, as headers.
|
133
|
-
def self.from_sql(query)
|
134
|
-
Table.from_sql(query)
|
133
|
+
def self.from_sql(query, tolerant_columns: [])
|
134
|
+
Table.from_sql(query, tolerant_columns: tolerant_columns)
|
135
135
|
end
|
136
136
|
|
137
137
|
########################################################################
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fat_table
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.5.
|
4
|
+
version: 0.5.5
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Daniel E. Doherty
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2022-
|
11
|
+
date: 2022-03-25 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|