fat_table 0.5.2 → 0.5.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.org +136 -17
- data/TODO.org +28 -1
- data/lib/ext/array.rb +17 -0
- data/lib/fat_table/column.rb +112 -56
- data/lib/fat_table/convert.rb +20 -10
- data/lib/fat_table/errors.rb +4 -0
- data/lib/fat_table/evaluator.rb +6 -8
- data/lib/fat_table/footer.rb +1 -1
- data/lib/fat_table/formatters/formatter.rb +17 -11
- data/lib/fat_table/formatters/org_formatter.rb +3 -3
- data/lib/fat_table/formatters/term_formatter.rb +3 -3
- data/lib/fat_table/formatters/text_formatter.rb +3 -3
- data/lib/fat_table/table.rb +90 -31
- data/lib/fat_table/version.rb +1 -1
- data/lib/fat_table.rb +16 -16
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 2f391d5e4ad9d7a4dcb303098b90f6fb42379d9df382192c4bacaf169523dab1
|
4
|
+
data.tar.gz: ea75f906fcd164752a3ca2220324ad10d8438562bb31c4846bc6575723e834fa
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 2914c40a497cf24bcdba8868c4a4607ea1bdb0a94fba8caa3d6e31407ff6832621df2a0b4a8a2bb98c3b40080fed6d752f8e936f66a96270fe675e93d5a852d6
|
7
|
+
data.tar.gz: dbc43f4d2bac4a2a19bc7cc90d1fbfbf739a12f0f280a543670af588cfcf2bb0df74d50cfb7ded76b851f5470fd30d4f971659cead5131f9ea15e24b492e5823
|
data/README.org
CHANGED
@@ -25,6 +25,16 @@ The following is for org.
|
|
25
25
|
|
26
26
|
[[https://travis-ci.org/ddoherty03/fat_table.svg?branch=master]]
|
27
27
|
|
28
|
+
* Version
|
29
|
+
#+begin_src ruby :wrap EXAMPLE
|
30
|
+
require 'fat_table'
|
31
|
+
"Current version is: #{FatTable::VERSION}"
|
32
|
+
#+end_src
|
33
|
+
|
34
|
+
#+begin_EXAMPLE
|
35
|
+
Current version is: 0.5.4
|
36
|
+
#+end_EXAMPLE
|
37
|
+
|
28
38
|
* Introduction
|
29
39
|
|
30
40
|
~FatTable~ is a gem that treats tables as a data type. It provides methods for
|
@@ -57,6 +67,7 @@ array of arrays with its ~.to_aoa~ output function will be rendered in an
|
|
57
67
|
org-mode buffer as an org-table, ready for processing by other code blocks.
|
58
68
|
|
59
69
|
* Table of Contents :toc:noexport:
|
70
|
+
- [[#version][Version]]
|
60
71
|
- [[#introduction][Introduction]]
|
61
72
|
- [[#installation][Installation]]
|
62
73
|
- [[#using-in-a-gem][Using in a gem]]
|
@@ -74,6 +85,7 @@ org-mode buffer as an org-table, ready for processing by other code blocks.
|
|
74
85
|
- [[#without-headers][Without Headers]]
|
75
86
|
- [[#with-headers][With Headers]]
|
76
87
|
- [[#forcing-string-type][Forcing String Type]]
|
88
|
+
- [[#designating-tolerant-columns][Designating "Tolerant" Columns]]
|
77
89
|
- [[#from-csv-or-org-mode-files-or-strings][From CSV or Org Mode files or strings]]
|
78
90
|
- [[#from-arrays-of-arrays][From Arrays of Arrays]]
|
79
91
|
- [[#in-ruby-code][In Ruby Code]]
|
@@ -430,11 +442,26 @@ or nil. There are only five permissible types for a ~Column~:
|
|
430
442
|
5. *NilClass* (for the undetermined column type).
|
431
443
|
|
432
444
|
When a ~Table~ is constructed from an external source, all ~Columns~ start out
|
433
|
-
having a type of ~NilClass~, that is, their type is as yet undetermined. When
|
434
|
-
string or object of one of the four determined types is added to a ~Column
|
435
|
-
|
436
|
-
|
437
|
-
|
445
|
+
having a type of ~NilClass~, that is, their type is as yet undetermined. When
|
446
|
+
a string or object of one of the four determined types is added to a ~Column~
|
447
|
+
and it can be converted into one of the permissible types, it fixes the type
|
448
|
+
of the column, and all further items added to the ~Column~ must either be
|
449
|
+
~nil~ (indicating no value) or be capable of being coerced to the column's
|
450
|
+
type. Otherwise, ~FatTable~ raises an exception.
|
451
|
+
|
452
|
+
The strictness of requiring all items to be of the same type can be relaxed by
|
453
|
+
declaring a column to be "tolerant." You can do so when you create the table
|
454
|
+
by adding a tolerant_columns keyword parameter. If a Column is tolerant,
|
455
|
+
~FatTable~ tries to convert new items into a type other than a ~String~ and,
|
456
|
+
if it can do so, sets /that/ as the Column's type. Any later items that
|
457
|
+
cannot be converted into the Column's type are converted to strings. These
|
458
|
+
interloper strings are treated like nils for purposes of sorting and
|
459
|
+
evaluation, but are displayed according to any string formatting on output.
|
460
|
+
See [[*Designating "Tolerant" Columns][Designating "Tolerant" Columns]] below.
|
461
|
+
|
462
|
+
It is also possible to force ~FatTable~ to treat a column as a String type,
|
463
|
+
even its items look like one of the other types. See [[*Forcing String Type][Forcing String Type]]
|
464
|
+
below.
|
438
465
|
|
439
466
|
Items of input must be either one of the permissible ruby objects or strings. If
|
440
467
|
they are strings, ~FatTable~ attempts to parse them as one of the permissible
|
@@ -591,10 +618,18 @@ columns to be created:
|
|
591
618
|
**** Forcing String Type
|
592
619
|
Occasionally, ~FatTable~'s automatic type detection can get in the way and you
|
593
620
|
just want it to treat one or more columns as Strings regardless of their
|
594
|
-
appearance. Think, for example, of zip codes.
|
595
|
-
table, you can
|
596
|
-
|
597
|
-
|
621
|
+
appearance. Think, for example, of zip codes. If headers are given when a
|
622
|
+
table is contructed, you can designate a forced-string column by appending a
|
623
|
+
~!~ to the end of the header. It will not become part of the header, it will
|
624
|
+
just mark it as a forced-string Column.
|
625
|
+
|
626
|
+
#+begin_SRC emacs-lisp :wrap EXAMPLE
|
627
|
+
tab = FatTable.new(:a, 'b', 'C!', :d, :zip!)
|
628
|
+
#+end_SRC
|
629
|
+
|
630
|
+
In addition, at any time after creating a table, you can force the String type
|
631
|
+
on any number of columns with the ~force_string!~ method. When you do so, all
|
632
|
+
exisiting items in the column are converted to strings with the #to_s method.
|
598
633
|
|
599
634
|
#+begin_src ruby :wrap EXAMPLE
|
600
635
|
tab = FatTable.new(:a, 'b', 'C', :d, :zip)
|
@@ -619,8 +654,83 @@ converted to strings with the #to_s method.
|
|
619
654
|
+======+======+============+===+=======+===+
|
620
655
|
#+end_EXAMPLE
|
621
656
|
|
657
|
+
**** Designating "Tolerant" Columns
|
658
|
+
Related to the problem just discussed is the problem of reading files in from
|
659
|
+
the wild where a column may get typed as, say Numeric, but then contain
|
660
|
+
something that can't be parsed as a Numeric. ~FatTable~ raises an exception
|
661
|
+
is such cases, and that may be what you want if you can control the input.
|
662
|
+
But, especially when you cannot do so, it can be helpful to designate one or
|
663
|
+
more columns as "tolerant." This means that when a conversion problem occurs,
|
664
|
+
the column item is retained as a string type in a column that is otherwise of
|
665
|
+
one of the types Numeric, DateTime, or Boolean. Those string items are
|
666
|
+
treated as nils for purposes of sorting or evaluation in a ~select~ method.
|
667
|
+
When formatted, they participate in string formatting directive, but not those
|
668
|
+
for other types.
|
669
|
+
|
670
|
+
All of the table construction methods, allow a keyword parameter,
|
671
|
+
~tolerant_columns~, where you can designate what columns should be convert to
|
672
|
+
String type when conversion to the auto-typed column type is not possible.
|
673
|
+
The parameter should be an array of headers, in either string or symbol form,
|
674
|
+
for which this behavior is desired. In addition, it can be set to the special
|
675
|
+
string '*' or symbol ~:*~ to indicate that all the columns should be made
|
676
|
+
tolerant.
|
677
|
+
|
678
|
+
#+begin_src ruby :wrap EXAMPLE
|
679
|
+
require 'fat_table'
|
680
|
+
tab = FatTable.new(:a, 'b', 'C', :d, :zip, tolerant_columns: [:zip])
|
681
|
+
tab << { a: 1, b: 2, c: "<2017-01-21>", d: 'f', e: '', zip: 18552 }
|
682
|
+
tab << { a: 3.14, b: 2.17, c: '[2016-01-21 Thu]', d: 'Y', e: nil }
|
683
|
+
tab << { zip: '01879--7884' }
|
684
|
+
tab << { zip: '66210' }
|
685
|
+
tab << { zip: '90210' }
|
686
|
+
tab.to_text
|
687
|
+
#+end_src
|
688
|
+
|
689
|
+
#+RESULTS:
|
690
|
+
#+begin_EXAMPLE
|
691
|
+
+======+======+============+===+=============+===+
|
692
|
+
| A | B | C | D | Zip | E |
|
693
|
+
+------+------+------------+---+-------------+---+
|
694
|
+
| 1 | 2 | 2017-01-21 | F | 18552 | |
|
695
|
+
| 3.14 | 2.17 | 2016-01-21 | T | | |
|
696
|
+
| | | | | 01879--7884 | |
|
697
|
+
| | | | | 66210 | |
|
698
|
+
| | | | | 90210 | |
|
699
|
+
+======+======+============+===+=============+===+
|
700
|
+
#+end_EXAMPLE
|
701
|
+
|
702
|
+
Another way to designate a column as tolerant is to end a column you want to
|
703
|
+
designate as tolerant with a ~!~. The ~!~ will be stripped from the header,
|
704
|
+
but it will be marked as tolerant.
|
705
|
+
#+begin_src ruby :wrap EXAMPLE
|
706
|
+
require 'fat_table'
|
707
|
+
tab = FatTable.new(:a, 'b!', 'C', :d, :zip!)
|
708
|
+
tab << { a: 1, b: 2, c: "<2017-01-21>", d: 'f', e: '', zip: 18552 }
|
709
|
+
tab << { a: 3.14, b: 2.17, c: '[2016-01-21 Thu]', d: 'Y', e: nil }
|
710
|
+
tab << { zip: '01879--7884' }
|
711
|
+
tab << { zip: '66210', b: 'Not a Number' }
|
712
|
+
tab << { zip: '90210' }
|
713
|
+
tab.to_text
|
714
|
+
#+end_src
|
715
|
+
|
716
|
+
#+RESULTS:
|
717
|
+
#+begin_EXAMPLE
|
718
|
+
+======+==============+============+===+=============+===+
|
719
|
+
| A | B | C | D | Zip | E |
|
720
|
+
+------+--------------+------------+---+-------------+---+
|
721
|
+
| 1 | 2 | 2017-01-21 | F | 18552 | |
|
722
|
+
| 3.14 | 2.17 | 2016-01-21 | T | | |
|
723
|
+
| | | | | 01879--7884 | |
|
724
|
+
| | Not a Number | | | 66210 | |
|
725
|
+
| | | | | 90210 | |
|
726
|
+
+======+==============+============+===+=============+===+
|
727
|
+
#+end_EXAMPLE
|
728
|
+
|
622
729
|
*** From CSV or Org Mode files or strings
|
623
|
-
Tables can also be read from ~.csv~ files or files containing ~org-mode~
|
730
|
+
Tables can also be read from ~.csv~ files or files containing ~org-mode~
|
731
|
+
tables. Remember that you can make any column tolerant with a
|
732
|
+
~tolerant_columns:~ keyword argument or make them all tolerant by designating
|
733
|
+
the pseudo-column ~:*~ as tolerant.
|
624
734
|
|
625
735
|
In the case of org-mode files, ~FatTable~ skips through the file until it finds
|
626
736
|
a line that look like a table, that is, it begins with any number of spaces
|
@@ -677,8 +787,10 @@ header row, and the headers are converted to symbols as described above.
|
|
677
787
|
|
678
788
|
*** From Arrays of Arrays
|
679
789
|
**** In Ruby Code
|
680
|
-
You can also initialize a table directly from ruby data structures. You can,
|
681
|
-
example, build a table from an array of arrays
|
790
|
+
You can also initialize a table directly from ruby data structures. You can,
|
791
|
+
for example, build a table from an array of arrays. Remember that you can
|
792
|
+
make any column tolerant with a ~tolerant_columns:~ keyword argument or make
|
793
|
+
them all tolerant by designating the pseudo-column ~:*~ as tolerant.
|
682
794
|
|
683
795
|
#+BEGIN_SRC ruby
|
684
796
|
aoa = [
|
@@ -772,8 +884,10 @@ This example illustrates several things:
|
|
772
884
|
|
773
885
|
A second ruby data structure that can be used to initialize a ~FatTable~ table
|
774
886
|
is an array of ruby Hashes. Each hash represents a row of the table, and the
|
775
|
-
headers of the table are taken from the keys of the hashes. Accordingly, all
|
776
|
-
hashes must have the same keys.
|
887
|
+
headers of the table are taken from the keys of the hashes. Accordingly, all
|
888
|
+
the hashes must have the same keys. Remember that you can make any column
|
889
|
+
tolerant with a ~tolerant_columns:~ keyword argument or make them all tolerant
|
890
|
+
by designating the pseudo-column ~:*~ as tolerant.
|
777
891
|
|
778
892
|
This same method can in fact take an array of any objects that can be converted
|
779
893
|
to a Hash with the ~#to_h~ method, so you can use an array of your own objects
|
@@ -862,6 +976,10 @@ The ~.connect~ function need only be called once, and the database handle it
|
|
862
976
|
creates will be used for all subsequent ~.from_sql~ calls until ~.connect~ is
|
863
977
|
called again.
|
864
978
|
|
979
|
+
Remember that you can make any column tolerant with a ~tolerant_columns:~
|
980
|
+
keyword argument or make them all tolerant by designating the pseudo-column
|
981
|
+
~:*~ as tolerant.
|
982
|
+
|
865
983
|
*** Marking Groups in Input
|
866
984
|
**** Manually
|
867
985
|
At any point, you can add a boundary to a table by invokong the
|
@@ -2549,9 +2667,10 @@ the table, but they can be overridden by more specific directives given in a
|
|
2549
2667
|
~format_for~ directive.
|
2550
2668
|
|
2551
2669
|
**** Type and Column priority
|
2552
|
-
A directive based
|
2553
|
-
|
2554
|
-
|
2670
|
+
A directive based the column name overrides any directive based on type. If
|
2671
|
+
any cell has both a type-based formatting and column-based, the column
|
2672
|
+
instructions prevail. In earlier versions the instuctions were "merged" but
|
2673
|
+
that is no longer the case.
|
2555
2674
|
|
2556
2675
|
However, there is a twist. Since the end result of formatting is to convert
|
2557
2676
|
all columns to strings, the formatting directives for the ~String~ type can
|
data/TODO.org
CHANGED
@@ -1,10 +1,37 @@
|
|
1
|
+
|
2
|
+
* TODO Specify Column Widths
|
3
|
+
Allow a formatter to specify column widths. This could be a number of
|
4
|
+
characters, which would be interpreted as a number of "ems" for LaTeX.
|
5
|
+
Cell content larger than the width would be truncated. Any column without a
|
6
|
+
width specified would be set at the width of the longest value in that cell,
|
7
|
+
after initial formatting.
|
8
|
+
|
9
|
+
#+begin_SRC ruby
|
10
|
+
tab.to_text do |f|
|
11
|
+
f.widths(a: 13, b: 30)
|
12
|
+
end
|
13
|
+
#+end_SRC
|
14
|
+
|
15
|
+
Possible enhancements:
|
16
|
+
- specify an overall width and column widths as decimal or fractions, so that
|
17
|
+
a column's width would be that fraction of the overall width.
|
18
|
+
- specify a Range for a width, so that the column would at least min and at
|
19
|
+
most max, otherwise the width of its largest cell.
|
20
|
+
|
1
21
|
* TODO Conversion to Spreadsheets
|
2
22
|
- State "TODO" from [2017-04-21 Fri 10:36]
|
3
23
|
This is a [[https://github.com/westonganger/spreadsheet_architect][gem]] that I can include into the Table model to convert a table into
|
4
24
|
a spread-sheet, or even a sheet in a multi-sheet spreadsheet file.
|
5
25
|
|
6
|
-
* TODO Add
|
26
|
+
* TODO Add Quandl or EODDATA Queries
|
27
|
+
Possible replacements for YQL.
|
28
|
+
|
29
|
+
* CNCL Add from_yql for fetching from Yahoo
|
30
|
+
CLOSED: [2022-01-30 Sun 06:03]
|
7
31
|
- State "TODO" from [2017-04-21 Fri 10:35]
|
32
|
+
|
33
|
+
Cancelled because Yahoo shut down the YQL api service.
|
34
|
+
|
8
35
|
Add a constructor to allow fetching stock data from yql. Perhaps grab all
|
9
36
|
available fields, then allow a select of those of interest.
|
10
37
|
|
data/lib/ext/array.rb
CHANGED
@@ -12,4 +12,21 @@ class Array
|
|
12
12
|
end
|
13
13
|
end
|
14
14
|
end
|
15
|
+
|
16
|
+
def filter_to_type(typ)
|
17
|
+
if typ == 'Boolean'
|
18
|
+
compact.select { |i| i.is_a?(TrueClass) || i.is_a?(FalseClass) }
|
19
|
+
elsif typ == 'DateTime'
|
20
|
+
compact.select { |i| i.is_a?(Date) || i.is_a?(DateTime) || i.is_a?(Time) }
|
21
|
+
.map { |i| i.to_datetime }
|
22
|
+
elsif typ == 'Numeric'
|
23
|
+
compact.select { |i| i.is_a?(Numeric) }
|
24
|
+
elsif typ == 'String'
|
25
|
+
map { |i| i.to_s }
|
26
|
+
elsif typ == 'NilClass'
|
27
|
+
self
|
28
|
+
else
|
29
|
+
raise ArgumentError, "cannot filter_to_type for type '#{typ}'"
|
30
|
+
end
|
31
|
+
end
|
15
32
|
end
|
data/lib/fat_table/column.rb
CHANGED
@@ -83,7 +83,7 @@ module FatTable
|
|
83
83
|
# col.type #=> 'Numeric'
|
84
84
|
# col.header #=> :prices
|
85
85
|
# col.sum #=> 18376.75
|
86
|
-
def initialize(header:, items: [], type: 'NilClass')
|
86
|
+
def initialize(header:, items: [], type: 'NilClass', tolerant: false)
|
87
87
|
@raw_header = header
|
88
88
|
@header =
|
89
89
|
if @raw_header.is_a?(Symbol)
|
@@ -92,6 +92,7 @@ module FatTable
|
|
92
92
|
@raw_header.to_s.as_sym
|
93
93
|
end
|
94
94
|
@type = type
|
95
|
+
@tolerant = tolerant
|
95
96
|
msg = "unknown column type '#{type}"
|
96
97
|
raise UserError, msg unless TYPES.include?(@type.to_s)
|
97
98
|
|
@@ -141,6 +142,14 @@ module FatTable
|
|
141
142
|
|
142
143
|
# :category: Attributes
|
143
144
|
|
145
|
+
# Is this column tolerant of type incompatibilities? If so, the Column
|
146
|
+
# type will be forced to String if an incompatible type is found.
|
147
|
+
def tolerant?
|
148
|
+
@tolerant
|
149
|
+
end
|
150
|
+
|
151
|
+
# :category: Attributes
|
152
|
+
|
144
153
|
# Force the column to have String type and then convert all items to
|
145
154
|
# strings.
|
146
155
|
def force_string!
|
@@ -182,12 +191,15 @@ module FatTable
|
|
182
191
|
|
183
192
|
# :category: Aggregates
|
184
193
|
|
185
|
-
# Return the first non-nil item in the Column
|
194
|
+
# Return the first non-nil item in the Column, or nil if all items are
|
195
|
+
# nil. Works with any Column type.
|
186
196
|
def first
|
197
|
+
return nil if items.all?(&:nil?)
|
198
|
+
|
187
199
|
if type == 'String'
|
188
200
|
items.reject(&:blank?).first
|
189
201
|
else
|
190
|
-
items.
|
202
|
+
items.filter_to_type(type).first
|
191
203
|
end
|
192
204
|
end
|
193
205
|
|
@@ -195,83 +207,94 @@ module FatTable
|
|
195
207
|
|
196
208
|
# Return the last non-nil item in the Column. Works with any Column type.
|
197
209
|
def last
|
210
|
+
return nil if items.all?(&:nil?)
|
211
|
+
|
198
212
|
if type == 'String'
|
199
213
|
items.reject(&:blank?).last
|
200
214
|
else
|
201
|
-
items.
|
215
|
+
items.filter_to_type(type).last
|
202
216
|
end
|
203
217
|
end
|
204
218
|
|
205
219
|
# :category: Aggregates
|
206
220
|
|
207
|
-
# Return a count of the non-nil items in the Column
|
208
|
-
# type.
|
221
|
+
# Return a count of the non-nil items in the Column, or the size of the
|
222
|
+
# column if all items are nil. Works with any Column type.
|
209
223
|
def count
|
224
|
+
return items.size if items.all?(&:nil?)
|
225
|
+
|
210
226
|
if type == 'String'
|
211
227
|
items.reject(&:blank?).count.to_d
|
212
228
|
else
|
213
|
-
items.
|
229
|
+
items.filter_to_type(type).count.to_d
|
214
230
|
end
|
215
231
|
end
|
216
232
|
|
217
233
|
# :category: Aggregates
|
218
234
|
|
219
|
-
# Return the smallest non-nil, non-blank item in the Column
|
220
|
-
# numeric, string, and datetime Columns.
|
235
|
+
# Return the smallest non-nil, non-blank item in the Column, or nil if all
|
236
|
+
# items are nil. Works with numeric, string, and datetime Columns.
|
221
237
|
def min
|
222
238
|
only_with('min', 'NilClass', 'Numeric', 'String', 'DateTime')
|
223
239
|
if type == 'String'
|
224
240
|
items.reject(&:blank?).min
|
225
241
|
else
|
226
|
-
items.
|
242
|
+
items.filter_to_type(type).min
|
227
243
|
end
|
228
244
|
end
|
229
245
|
|
230
246
|
# :category: Aggregates
|
231
247
|
|
232
|
-
# Return the largest non-nil, non-blank item in the Column
|
233
|
-
# numeric, string, and datetime Columns.
|
248
|
+
# Return the largest non-nil, non-blank item in the Column, or nil if all
|
249
|
+
# items are nil. Works with numeric, string, and datetime Columns.
|
234
250
|
def max
|
235
251
|
only_with('max', 'NilClass', 'Numeric', 'String', 'DateTime')
|
236
252
|
if type == 'String'
|
237
253
|
items.reject(&:blank?).max
|
238
254
|
else
|
239
|
-
items.
|
255
|
+
items.filter_to_type(type).max
|
240
256
|
end
|
241
257
|
end
|
242
258
|
|
243
259
|
# :category: Aggregates
|
244
260
|
|
245
|
-
# Return a Range object for the smallest to largest value in the column
|
246
|
-
# Works with numeric, string, and datetime
|
261
|
+
# Return a Range object for the smallest to largest value in the column,
|
262
|
+
# or nil if all items are nil. Works with numeric, string, and datetime
|
263
|
+
# Columns.
|
247
264
|
def range
|
248
265
|
only_with('range', 'NilClass', 'Numeric', 'String', 'DateTime')
|
266
|
+
return nil if items.all?(&:nil?)
|
267
|
+
|
249
268
|
Range.new(min, max)
|
250
269
|
end
|
251
270
|
|
252
271
|
# :category: Aggregates
|
253
272
|
|
254
|
-
# Return the sum of the non-nil items in the Column
|
255
|
-
# string Columns. For a string Column, it
|
256
|
-
# the non-nil items.
|
273
|
+
# Return the sum of the non-nil items in the Column, or 0 if all items are
|
274
|
+
# nil. Works with numeric and string Columns. For a string Column, it
|
275
|
+
# will return the concatenation of the non-nil items.
|
257
276
|
def sum
|
277
|
+
return 0 if type == 'NilClass' || items.all?(&:nil?)
|
278
|
+
|
258
279
|
only_with('sum', 'Numeric', 'String')
|
259
280
|
if type == 'String'
|
260
281
|
items.reject(&:blank?).join(' ')
|
261
282
|
else
|
262
|
-
items.
|
283
|
+
items.filter_to_type(type).sum
|
263
284
|
end
|
264
285
|
end
|
265
286
|
|
266
287
|
# :category: Aggregates
|
267
288
|
|
268
|
-
# Return the average value of the non-nil items in the Column
|
269
|
-
# numeric and datetime Columns. For datetime
|
270
|
-
# to its Julian day number, computes the
|
271
|
-
# average back to a DateTime.
|
289
|
+
# Return the average value of the non-nil items in the Column, or 0 if all
|
290
|
+
# items are nil. Works with numeric and datetime Columns. For datetime
|
291
|
+
# Columns, it converts each date to its Julian day number, computes the
|
292
|
+
# average, and then converts the average back to a DateTime.
|
272
293
|
def avg
|
294
|
+
return 0 if type == 'NilClass' || items.all?(&:nil?)
|
295
|
+
|
273
296
|
only_with('avg', 'DateTime', 'Numeric')
|
274
|
-
itms = items.
|
297
|
+
itms = items.filter_to_type(type)
|
275
298
|
size = itms.size.to_d
|
276
299
|
if type == 'DateTime'
|
277
300
|
avg_jd = itms.map(&:jd).sum / size
|
@@ -284,17 +307,20 @@ module FatTable
|
|
284
307
|
# :category: Aggregates
|
285
308
|
|
286
309
|
# Return the sample variance (the unbiased estimator of the population
|
287
|
-
# variance using a divisor of N-1) as the average squared deviation from
|
288
|
-
# mean, of the non-nil items in the Column
|
289
|
-
# Columns. For datetime Columns, it
|
290
|
-
# number and computes the variance of
|
310
|
+
# variance using a divisor of N-1) as the average squared deviation from
|
311
|
+
# the mean, of the non-nil items in the Column, or 0 if all items are
|
312
|
+
# nil. Works with numeric and datetime Columns. For datetime Columns, it
|
313
|
+
# converts each date to its Julian day number and computes the variance of
|
314
|
+
# those numbers.
|
291
315
|
def var
|
316
|
+
return 0 if type == 'NilClass' || items.all?(&:nil?)
|
317
|
+
|
292
318
|
only_with('var', 'DateTime', 'Numeric')
|
293
319
|
all_items =
|
294
320
|
if type == 'DateTime'
|
295
|
-
items.
|
321
|
+
items.filter_to_type(type).map(&:jd)
|
296
322
|
else
|
297
|
-
items.
|
323
|
+
items.filter_to_type(type)
|
298
324
|
end
|
299
325
|
n = count
|
300
326
|
return BigDecimal('0.0') if n <= 1
|
@@ -310,12 +336,15 @@ module FatTable
|
|
310
336
|
|
311
337
|
# Return the population variance (the biased estimator of the population
|
312
338
|
# variance using a divisor of N) as the average squared deviation from the
|
313
|
-
# mean, of the non-nil items in the Column
|
314
|
-
# Columns. For datetime Columns, it
|
315
|
-
# number and computes the variance of
|
339
|
+
# mean, of the non-nil items in the Column, or 0 if all items are
|
340
|
+
# nil. Works with numeric and datetime Columns. For datetime Columns, it
|
341
|
+
# converts each date to its Julian day number and computes the variance of
|
342
|
+
# those numbers.
|
316
343
|
def pvar
|
344
|
+
return 0 if type == 'NilClass' || items.all?(&:nil?)
|
345
|
+
|
317
346
|
only_with('var', 'DateTime', 'Numeric')
|
318
|
-
n = items.
|
347
|
+
n = items.filter_to_type(type).size.to_d
|
319
348
|
return BigDecimal('0.0') if n <= 1
|
320
349
|
var * ((n - 1) / n)
|
321
350
|
end
|
@@ -324,11 +353,13 @@ module FatTable
|
|
324
353
|
|
325
354
|
# Return the sample standard deviation (the unbiased estimator of the
|
326
355
|
# population standard deviation using a divisor of N-1) as the square root
|
327
|
-
# of the sample variance, of the non-nil items in the Column
|
328
|
-
# numeric and datetime Columns. For datetime
|
329
|
-
# to its Julian day number and computes the
|
330
|
-
# numbers.
|
356
|
+
# of the sample variance, of the non-nil items in the Column, or 0 if all
|
357
|
+
# items are nil. Works with numeric and datetime Columns. For datetime
|
358
|
+
# Columns, it converts each date to its Julian day number and computes the
|
359
|
+
# standard deviation of those numbers.
|
331
360
|
def dev
|
361
|
+
return 0 if type == 'NilClass' || items.all?(&:nil?)
|
362
|
+
|
332
363
|
only_with('dev', 'DateTime', 'Numeric')
|
333
364
|
var.sqrt(20)
|
334
365
|
end
|
@@ -336,12 +367,14 @@ module FatTable
|
|
336
367
|
# :category: Aggregates
|
337
368
|
|
338
369
|
# Return the population standard deviation (the biased estimator of the
|
339
|
-
# population standard deviation using a divisor of N) as the square root
|
340
|
-
# the population variance, of the non-nil items in the Column
|
341
|
-
# numeric and datetime Columns. For datetime
|
342
|
-
# to its Julian day number and computes the
|
343
|
-
# numbers.
|
370
|
+
# population standard deviation using a divisor of N) as the square root
|
371
|
+
# of the population variance, of the non-nil items in the Column, or 0 if
|
372
|
+
# all items are nil. Works with numeric and datetime Columns. For datetime
|
373
|
+
# Columns, it converts each date to its Julian day number and computes the
|
374
|
+
# standard deviation of those numbers.
|
344
375
|
def pdev
|
376
|
+
return 0 if type == 'NilClass' || items.all?(&:nil?)
|
377
|
+
|
345
378
|
only_with('dev', 'DateTime', 'Numeric')
|
346
379
|
Math.sqrt(pvar)
|
347
380
|
end
|
@@ -349,28 +382,35 @@ module FatTable
|
|
349
382
|
# :category: Aggregates
|
350
383
|
|
351
384
|
# Return true if any of the items in the Column are true; otherwise return
|
352
|
-
# false. Works only with boolean Columns.
|
385
|
+
# false, or false if all items are nil. Works only with boolean Columns.
|
353
386
|
def any?
|
387
|
+
return false if type == 'NilClass' || items.all?(&:nil?)
|
388
|
+
|
354
389
|
only_with('any?', 'Boolean')
|
355
|
-
items.
|
390
|
+
items.filter_to_type(type).any?
|
356
391
|
end
|
357
392
|
|
358
393
|
# :category: Aggregates
|
359
394
|
|
360
395
|
# Return true if all of the items in the Column are true; otherwise return
|
361
|
-
# false. Works only with boolean Columns.
|
396
|
+
# false, or false if all items are nil. Works only with boolean Columns.
|
362
397
|
def all?
|
398
|
+
return false if type == 'NilClass' || items.all?(&:nil?)
|
399
|
+
|
363
400
|
only_with('all?', 'Boolean')
|
364
|
-
items.
|
401
|
+
items.filter_to_type(type).all?
|
365
402
|
end
|
366
403
|
|
367
404
|
# :category: Aggregates
|
368
405
|
|
369
|
-
# Return true if none of the items in the Column are true; otherwise
|
370
|
-
# false. Works only with boolean
|
406
|
+
# Return true if none of the items in the Column are true; otherwise
|
407
|
+
# return false, or true if all items are nil. Works only with boolean
|
408
|
+
# Columns.
|
371
409
|
def none?
|
410
|
+
return true if type == 'NilClass' || items.all?(&:nil?)
|
411
|
+
|
372
412
|
only_with('none?', 'Boolean')
|
373
|
-
items.
|
413
|
+
items.filter_to_type(type).none?
|
374
414
|
end
|
375
415
|
|
376
416
|
# :category: Aggregates
|
@@ -378,14 +418,17 @@ module FatTable
|
|
378
418
|
# Return true if precisely one of the items in the Column is true;
|
379
419
|
# otherwise return false. Works only with boolean Columns.
|
380
420
|
def one?
|
421
|
+
return false if type == 'NilClass' || items.all?(&:nil?)
|
422
|
+
|
381
423
|
only_with('one?', 'Boolean')
|
382
|
-
items.
|
424
|
+
items.filter_to_type(type).one?
|
383
425
|
end
|
384
426
|
|
385
427
|
private
|
386
428
|
|
387
429
|
def only_with(agg, *valid_types)
|
388
430
|
return self if valid_types.include?(type)
|
431
|
+
|
389
432
|
msg = "aggregate '#{agg}' cannot be applied to a #{type} column"
|
390
433
|
raise UserError, msg
|
391
434
|
end
|
@@ -400,9 +443,17 @@ module FatTable
|
|
400
443
|
|
401
444
|
# Append +itm+ to end of the Column after converting it to the Column's
|
402
445
|
# type. If the Column's type is still open, i.e. NilClass, attempt to fix
|
403
|
-
# the Column's type based on the type of +itm+ as with Column.new.
|
446
|
+
# the Column's type based on the type of +itm+ as with Column.new. If its
|
447
|
+
# a tolerant column, respond to type errors by converting the column to a
|
448
|
+
# String type.
|
404
449
|
def <<(itm)
|
405
|
-
items <<
|
450
|
+
items << convert_and_set_type(itm)
|
451
|
+
rescue IncompatibleTypeError => ex
|
452
|
+
if tolerant?
|
453
|
+
items << Convert.convert_to_string(itm)
|
454
|
+
else
|
455
|
+
raise ex
|
456
|
+
end
|
406
457
|
end
|
407
458
|
|
408
459
|
# :category: Constructors
|
@@ -418,9 +469,14 @@ module FatTable
|
|
418
469
|
|
419
470
|
private
|
420
471
|
|
421
|
-
def
|
422
|
-
|
423
|
-
|
472
|
+
def convert_and_set_type(val)
|
473
|
+
begin
|
474
|
+
new_val = Convert.convert_to_type(val, type, tolerant: tolerant?)
|
475
|
+
rescue IncompatibleTypeError
|
476
|
+
err_msg = "attempt to add '#{val}' to column '#{header}' already typed as #{type}"
|
477
|
+
raise IncompatibleTypeError, err_msg
|
478
|
+
end
|
479
|
+
if new_val && (type == 'NilClass' || type == 'String')
|
424
480
|
@type =
|
425
481
|
if [true, false].include?(new_val)
|
426
482
|
'Boolean'
|
data/lib/fat_table/convert.rb
CHANGED
@@ -10,7 +10,7 @@ module FatTable
|
|
10
10
|
# determined, raise an error if the val cannot be converted to the Column
|
11
11
|
# type. Otherwise, returns the converted val as an object of the correct
|
12
12
|
# class.
|
13
|
-
def self.convert_to_type(val, type)
|
13
|
+
def self.convert_to_type(val, type, tolerant: false)
|
14
14
|
case type
|
15
15
|
when 'NilClass'
|
16
16
|
if val != false && val.blank?
|
@@ -36,8 +36,7 @@ module FatTable
|
|
36
36
|
else
|
37
37
|
new_val = convert_to_boolean(val)
|
38
38
|
if new_val.nil?
|
39
|
-
|
40
|
-
raise UserError, msg
|
39
|
+
raise IncompatibleTypeError
|
41
40
|
end
|
42
41
|
new_val
|
43
42
|
end
|
@@ -47,8 +46,7 @@ module FatTable
|
|
47
46
|
else
|
48
47
|
new_val = convert_to_date_time(val)
|
49
48
|
if new_val.nil?
|
50
|
-
|
51
|
-
raise UserError, msg
|
49
|
+
raise IncompatibleTypeError
|
52
50
|
end
|
53
51
|
new_val
|
54
52
|
end
|
@@ -58,24 +56,35 @@ module FatTable
|
|
58
56
|
else
|
59
57
|
new_val = convert_to_numeric(val)
|
60
58
|
if new_val.nil?
|
61
|
-
|
62
|
-
raise UserError, msg
|
59
|
+
raise IncompatibleTypeError
|
63
60
|
end
|
64
61
|
new_val
|
65
62
|
end
|
66
63
|
when 'String'
|
67
64
|
if val.nil?
|
68
65
|
nil
|
66
|
+
elsif tolerant
|
67
|
+
# Allow String to upgrade to one of Numeric, DateTime, or Boolean if
|
68
|
+
# possible.
|
69
|
+
if (new_val = convert_to_numeric(val))
|
70
|
+
new_val
|
71
|
+
elsif (new_val = convert_to_date_time(val))
|
72
|
+
new_val
|
73
|
+
elsif (new_val = convert_to_boolean(val))
|
74
|
+
new_val
|
75
|
+
else
|
76
|
+
new_val = convert_to_string(val)
|
77
|
+
end
|
78
|
+
new_val
|
69
79
|
else
|
70
80
|
new_val = convert_to_string(val)
|
71
81
|
if new_val.nil?
|
72
|
-
|
73
|
-
raise UserError, msg
|
82
|
+
raise IncompatibleTypeError
|
74
83
|
end
|
75
84
|
new_val
|
76
85
|
end
|
77
86
|
else
|
78
|
-
raise
|
87
|
+
raise LogicError, "Mysteriously, column has unknown type '#{type}'"
|
79
88
|
end
|
80
89
|
end
|
81
90
|
|
@@ -121,6 +130,7 @@ module FatTable
|
|
121
130
|
return val if val.is_a?(DateTime)
|
122
131
|
return val if val.is_a?(Date)
|
123
132
|
return val.to_datetime if val.is_a?(Time)
|
133
|
+
|
124
134
|
begin
|
125
135
|
str = val.to_s.clean
|
126
136
|
return nil if str.blank?
|
data/lib/fat_table/errors.rb
CHANGED
@@ -9,6 +9,10 @@ module FatTable
|
|
9
9
|
# cannot correct.
|
10
10
|
class LogicError < StandardError; end
|
11
11
|
|
12
|
+
# Raised when attempting to add an incompatible type to an already-typed
|
13
|
+
# Column.
|
14
|
+
class IncompatibleTypeError < UserError; end
|
15
|
+
|
12
16
|
# Raised when an external resource is not available due to caller or
|
13
17
|
# programmer error or some failure of the external resource to be available.
|
14
18
|
class TransientError < StandardError; end
|
data/lib/fat_table/evaluator.rb
CHANGED
@@ -45,17 +45,15 @@ module FatTable
|
|
45
45
|
end
|
46
46
|
|
47
47
|
# Return the result of evaluating +expr+ as a Ruby expression in which the
|
48
|
-
# instance variables set in Evaluator.new and any local variables set in
|
49
|
-
# Hash parameter +locals+ are available to the expression.
|
48
|
+
# instance variables set in Evaluator.new and any local variables set in
|
49
|
+
# the Hash parameter +locals+ are available to the expression. Certain
|
50
|
+
# errors simply return nil as the result. This can happen, for example,
|
51
|
+
# when a string gets into an otherwise numeric column because the column
|
52
|
+
# is set to tolerant.
|
50
53
|
def evaluate(expr = '', locals: {})
|
51
54
|
eval(expr, local_vars(binding, locals))
|
52
55
|
rescue NoMethodError, TypeError => ex
|
53
|
-
|
54
|
-
# Likely one of the locals was nil, so let nil be the result.
|
55
|
-
return nil
|
56
|
-
else
|
57
|
-
raise ex
|
58
|
-
end
|
56
|
+
nil
|
59
57
|
end
|
60
58
|
|
61
59
|
private
|
data/lib/fat_table/footer.rb
CHANGED
@@ -532,7 +532,7 @@ module FatTable
|
|
532
532
|
valid_keys = table.headers + %i[string numeric datetime boolean nil]
|
533
533
|
invalid_keys = (fmts.keys - valid_keys).uniq
|
534
534
|
unless invalid_keys.empty?
|
535
|
-
msg = "invalid #{location} column or type: #{invalid_keys.join(',')}"
|
535
|
+
msg = "invalid #{location} column or type: #{invalid_keys.join(', ')}"
|
536
536
|
raise UserError, msg
|
537
537
|
end
|
538
538
|
|
@@ -562,20 +562,21 @@ module FatTable
|
|
562
562
|
end
|
563
563
|
end
|
564
564
|
|
565
|
-
# Merge in formatting for column h based on the column
|
566
|
-
#
|
565
|
+
# Merge in formatting instructions for column h based on the column
|
566
|
+
# name, or if there is no formatting instructions for the column by
|
567
|
+
# name, merge in the formatting instructions based on the column's
|
568
|
+
# type. Insist on only the string type for the header location.
|
567
569
|
typ = (location == :header ? :string : table.type(h).as_sym)
|
568
570
|
parse_typ_method_name = 'parse_' + typ.to_s + '_fmt'
|
569
|
-
if fmts.key?(typ)
|
570
|
-
# Merge in type-based formatting
|
571
|
-
typ_fmt = send(parse_typ_method_name, fmts[typ]).first
|
572
|
-
format_h = format_h.merge(typ_fmt)
|
573
|
-
end
|
574
571
|
if fmts[h]
|
575
572
|
# Merge in column formatting
|
576
573
|
col_fmt = send(parse_typ_method_name, fmts[h],
|
577
574
|
strict: location != :header).first
|
578
575
|
format_h = format_h.merge(col_fmt)
|
576
|
+
elsif fmts.key?(typ)
|
577
|
+
# Merge in type-based formatting
|
578
|
+
typ_fmt = send(parse_typ_method_name, fmts[typ]).first
|
579
|
+
format_h = format_h.merge(typ_fmt)
|
579
580
|
end
|
580
581
|
|
581
582
|
# Copy :body formatting for column h to :bfirst and :gfirst if they
|
@@ -1003,9 +1004,14 @@ module FatTable
|
|
1003
1004
|
# converted to strings formatted according to the Formatter's formatting
|
1004
1005
|
# directives given in Formatter.format_for or Formatter.format.
|
1005
1006
|
def output
|
1006
|
-
#
|
1007
|
-
#
|
1008
|
-
|
1007
|
+
# If there are neither headers nor any rows in the table, return an
|
1008
|
+
# empty string.
|
1009
|
+
return '' if table.empty? && table.headers.empty?
|
1010
|
+
|
1011
|
+
# This results in a hash of two-element arrays. The key
|
1012
|
+
# is the header and the value is an array of the header and formatted
|
1013
|
+
# header. We do the latter so the structure parallels the structure for
|
1014
|
+
# rows explained next.
|
1009
1015
|
formatted_headers = build_formatted_headers
|
1010
1016
|
|
1011
1017
|
# These produce an array with each element representing a row of the
|
@@ -20,7 +20,7 @@ module FatTable
|
|
20
20
|
end
|
21
21
|
|
22
22
|
def pre_header(widths)
|
23
|
-
result = '|'
|
23
|
+
result = +'|'
|
24
24
|
widths.each_value do |w|
|
25
25
|
result += '-' * (w + 2) + '+'
|
26
26
|
end
|
@@ -53,7 +53,7 @@ module FatTable
|
|
53
53
|
end
|
54
54
|
|
55
55
|
def hline(widths)
|
56
|
-
result = '|'
|
56
|
+
result = +'|'
|
57
57
|
widths.each_value do |w|
|
58
58
|
result += '-' * (w + 2) + '+'
|
59
59
|
end
|
@@ -62,7 +62,7 @@ module FatTable
|
|
62
62
|
end
|
63
63
|
|
64
64
|
def post_footers(widths)
|
65
|
-
result = '|'
|
65
|
+
result = +'|'
|
66
66
|
widths.each_value do |w|
|
67
67
|
result += '-' * (w + 2) + '+'
|
68
68
|
end
|
@@ -221,7 +221,7 @@ module FatTable
|
|
221
221
|
end
|
222
222
|
|
223
223
|
def pre_header(widths)
|
224
|
-
result = upper_left
|
224
|
+
result = +upper_left
|
225
225
|
widths.each_value do |w|
|
226
226
|
result += double_rule * (w + 2) + upper_tee
|
227
227
|
end
|
@@ -255,7 +255,7 @@ module FatTable
|
|
255
255
|
end
|
256
256
|
|
257
257
|
def hline(widths)
|
258
|
-
result = left_tee
|
258
|
+
result = +left_tee
|
259
259
|
widths.each_value do |w|
|
260
260
|
result += horizontal_rule * (w + 2) + single_cross
|
261
261
|
end
|
@@ -289,7 +289,7 @@ module FatTable
|
|
289
289
|
end
|
290
290
|
|
291
291
|
def post_footers(widths)
|
292
|
-
result = lower_left
|
292
|
+
result = +lower_left
|
293
293
|
widths.each_value do |w|
|
294
294
|
result += double_rule * (w + 2) + lower_tee
|
295
295
|
end
|
@@ -16,7 +16,7 @@ module FatTable
|
|
16
16
|
end
|
17
17
|
|
18
18
|
def pre_header(widths)
|
19
|
-
result = '+'
|
19
|
+
result = +'+'
|
20
20
|
widths.each_value do |w|
|
21
21
|
result += '=' * (w + 2) + '+'
|
22
22
|
end
|
@@ -49,7 +49,7 @@ module FatTable
|
|
49
49
|
end
|
50
50
|
|
51
51
|
def hline(widths)
|
52
|
-
result = '+'
|
52
|
+
result = +'+'
|
53
53
|
widths.each_value do |w|
|
54
54
|
result += '-' * (w + 2) + '+'
|
55
55
|
end
|
@@ -82,7 +82,7 @@ module FatTable
|
|
82
82
|
end
|
83
83
|
|
84
84
|
def post_footers(widths)
|
85
|
-
result = '+'
|
85
|
+
result = +'+'
|
86
86
|
widths.each_value do |w|
|
87
87
|
result += '=' * (w + 2) + '+'
|
88
88
|
end
|
data/lib/fat_table/table.rb
CHANGED
@@ -62,19 +62,53 @@ module FatTable
|
|
62
62
|
# method call.
|
63
63
|
attr_accessor :explicit_boundaries
|
64
64
|
|
65
|
+
# An Array of FatTable::Columns that should be tolerant.
|
66
|
+
attr_reader :tolerant_columns
|
67
|
+
|
65
68
|
###########################################################################
|
66
69
|
# Constructors
|
67
70
|
###########################################################################
|
68
71
|
|
69
72
|
# :category: Constructors
|
70
73
|
|
71
|
-
# Return an empty FatTable::Table object.
|
72
|
-
|
74
|
+
# Return an empty FatTable::Table object. Specifying headers is optional.
|
75
|
+
# Any headers ending with a ! are marked as tolerant, in that, if an
|
76
|
+
# incompatible type is added to it, the column is re-typed as a String
|
77
|
+
# column, and construction proceeds. The ! is stripped from the header to
|
78
|
+
# form the column key, though. You can also provide the names of columns
|
79
|
+
# that should be tolerant by using the +tolerant_columns key-word to
|
80
|
+
# provide an array of headers that should be tolerant. The special string
|
81
|
+
# '*' or the symbol :* indicates that all columns should be created
|
82
|
+
# tolerant.
|
83
|
+
def initialize(*heads, tolerant_columns: [])
|
73
84
|
@columns = []
|
74
85
|
@explicit_boundaries = []
|
86
|
+
@tolerant_columns =
|
87
|
+
case tolerant_columns
|
88
|
+
when Array
|
89
|
+
tolerant_columns.map { |h| h.to_s.as_sym }
|
90
|
+
when String
|
91
|
+
if tolerant_columns.strip == '*'
|
92
|
+
['*'.to_sym]
|
93
|
+
else
|
94
|
+
[tolerant_columns.as_sym]
|
95
|
+
end
|
96
|
+
when Symbol
|
97
|
+
if tolerant_columns.to_s.strip == '*'
|
98
|
+
['*'.to_sym]
|
99
|
+
else
|
100
|
+
[tolerant_columns.to_s.as_sym]
|
101
|
+
end
|
102
|
+
else
|
103
|
+
raise ArgumentError, "set tolerant_columns to String, Symbol, or an Array of either"
|
104
|
+
end
|
75
105
|
unless heads.empty?
|
76
106
|
heads.each do |h|
|
77
|
-
|
107
|
+
if h.to_s.end_with?('!') || @tolerant_columns.include?(h)
|
108
|
+
@columns << Column.new(header: h.to_s.sub(/!\s*\z/, ''), type: 'String')
|
109
|
+
else
|
110
|
+
@columns << Column.new(header: h)
|
111
|
+
end
|
78
112
|
end
|
79
113
|
end
|
80
114
|
end
|
@@ -86,7 +120,7 @@ module FatTable
|
|
86
120
|
# though FatTable::Table objects have no instance variables, a class that
|
87
121
|
# inherits from it might.
|
88
122
|
def empty_dup
|
89
|
-
|
123
|
+
dup.__empty!
|
90
124
|
end
|
91
125
|
|
92
126
|
def __empty!
|
@@ -99,9 +133,9 @@ module FatTable
|
|
99
133
|
|
100
134
|
# Construct a Table from the contents of a CSV file named +fname+. Headers
|
101
135
|
# will be taken from the first CSV row and converted to symbols.
|
102
|
-
def self.from_csv_file(fname)
|
136
|
+
def self.from_csv_file(fname, tolerant_columns: [])
|
103
137
|
File.open(fname, 'r') do |io|
|
104
|
-
from_csv_io(io)
|
138
|
+
from_csv_io(io, tolerant_columns: tolerant_columns)
|
105
139
|
end
|
106
140
|
end
|
107
141
|
|
@@ -109,8 +143,8 @@ module FatTable
|
|
109
143
|
|
110
144
|
# Construct a Table from a CSV string +str+, treated in the same manner as
|
111
145
|
# the input from a CSV file in ::from_org_file.
|
112
|
-
def self.from_csv_string(str)
|
113
|
-
from_csv_io(StringIO.new(str))
|
146
|
+
def self.from_csv_string(str, tolerant_columns: [])
|
147
|
+
from_csv_io(StringIO.new(str), tolerant_columns: tolerant_columns)
|
114
148
|
end
|
115
149
|
|
116
150
|
# :category: Constructors
|
@@ -119,9 +153,9 @@ module FatTable
|
|
119
153
|
# file named +fname+. Headers are taken from the first row if the second row
|
120
154
|
# is an hrule. Otherwise, synthetic headers of the form +:col_1+, +:col_2+,
|
121
155
|
# etc. are created.
|
122
|
-
def self.from_org_file(fname)
|
156
|
+
def self.from_org_file(fname, tolerant_columns: [])
|
123
157
|
File.open(fname, 'r') do |io|
|
124
|
-
from_org_io(io)
|
158
|
+
from_org_io(io, tolerant_columns: tolerant_columns)
|
125
159
|
end
|
126
160
|
end
|
127
161
|
|
@@ -129,8 +163,8 @@ module FatTable
|
|
129
163
|
|
130
164
|
# Construct a Table from a string +str+, treated in the same manner as the
|
131
165
|
# contents of an org-mode file in ::from_org_file.
|
132
|
-
def self.from_org_string(str)
|
133
|
-
from_org_io(StringIO.new(str))
|
166
|
+
def self.from_org_string(str, tolerant_columns: [])
|
167
|
+
from_org_io(StringIO.new(str), tolerant_columns: tolerant_columns)
|
134
168
|
end
|
135
169
|
|
136
170
|
# :category: Constructors
|
@@ -149,8 +183,8 @@ module FatTable
|
|
149
183
|
# :hlines no +) org-mode strips all hrules from the table; otherwise (+
|
150
184
|
# HEADER: :hlines yes +) they are indicated with nil elements in the outer
|
151
185
|
# array.
|
152
|
-
def self.from_aoa(aoa, hlines: false)
|
153
|
-
from_array_of_arrays(aoa, hlines: hlines)
|
186
|
+
def self.from_aoa(aoa, hlines: false, tolerant_columns: [])
|
187
|
+
from_array_of_arrays(aoa, hlines: hlines, tolerant_columns: tolerant_columns)
|
154
188
|
end
|
155
189
|
|
156
190
|
# :category: Constructors
|
@@ -160,9 +194,9 @@ module FatTable
|
|
160
194
|
# keys, which, when converted to symbols will become the headers for the
|
161
195
|
# Table. If hlines is set true, mark a group boundary whenever a nil, rather
|
162
196
|
# than a hash appears in the outer array.
|
163
|
-
def self.from_aoh(aoh, hlines: false)
|
197
|
+
def self.from_aoh(aoh, hlines: false, tolerant_columns: [])
|
164
198
|
if aoh.first.respond_to?(:to_h)
|
165
|
-
from_array_of_hashes(aoh, hlines: hlines)
|
199
|
+
from_array_of_hashes(aoh, hlines: hlines, tolerant_columns: tolerant_columns)
|
166
200
|
else
|
167
201
|
raise UserError,
|
168
202
|
"Cannot initialize Table with an array of #{input[0].class}"
|
@@ -181,7 +215,7 @@ module FatTable
|
|
181
215
|
|
182
216
|
# Construct a Table by running a SQL +query+ against the database set up
|
183
217
|
# with FatTable.connect, with the rows of the query result as rows.
|
184
|
-
def self.from_sql(query)
|
218
|
+
def self.from_sql(query, tolerant_columns: [])
|
185
219
|
msg = 'FatTable.db must be set with FatTable.connect'
|
186
220
|
raise UserError, msg if FatTable.db.nil?
|
187
221
|
|
@@ -203,8 +237,8 @@ module FatTable
|
|
203
237
|
# Construct table from an array of hashes or an array of any object that
|
204
238
|
# can respond to #to_h. If an array element is a nil, mark it as a group
|
205
239
|
# boundary in the Table.
|
206
|
-
def from_array_of_hashes(hashes, hlines: false)
|
207
|
-
result = new
|
240
|
+
def from_array_of_hashes(hashes, hlines: false, tolerant_columns: [])
|
241
|
+
result = new(tolerant_columns: tolerant_columns)
|
208
242
|
hashes.each do |hsh|
|
209
243
|
if hsh.nil?
|
210
244
|
unless hlines
|
@@ -232,8 +266,8 @@ module FatTable
|
|
232
266
|
# hlines are stripped from the table, otherwise (:hlines yes) they are
|
233
267
|
# indicated with nil elements in the outer array as expected by this
|
234
268
|
# method when hlines is set true.
|
235
|
-
def from_array_of_arrays(rows, hlines: false)
|
236
|
-
result = new
|
269
|
+
def from_array_of_arrays(rows, hlines: false, tolerant_columns: [])
|
270
|
+
result = new(tolerant_columns: tolerant_columns)
|
237
271
|
headers = []
|
238
272
|
if !hlines
|
239
273
|
# Take the first row as headers
|
@@ -269,8 +303,8 @@ module FatTable
|
|
269
303
|
result
|
270
304
|
end
|
271
305
|
|
272
|
-
def from_csv_io(io)
|
273
|
-
result = new
|
306
|
+
def from_csv_io(io, tolerant_columns: [])
|
307
|
+
result = new(tolerant_columns: tolerant_columns)
|
274
308
|
::CSV.new(io, headers: true, header_converters: :symbol,
|
275
309
|
skip_blanks: true).each do |row|
|
276
310
|
result << row.to_h
|
@@ -283,7 +317,7 @@ module FatTable
|
|
283
317
|
# header row must be marked with an hline (i.e, a row that looks like
|
284
318
|
# '|---+--...--|') and groups of rows may be marked with hlines to
|
285
319
|
# indicate group boundaries.
|
286
|
-
def from_org_io(io)
|
320
|
+
def from_org_io(io, tolerant_columns: [])
|
287
321
|
table_re = /\A\s*\|/
|
288
322
|
hrule_re = /\A\s*\|[-+]+/
|
289
323
|
rows = []
|
@@ -318,7 +352,7 @@ module FatTable
|
|
318
352
|
rows << line.split('|').map(&:clean)
|
319
353
|
end
|
320
354
|
end
|
321
|
-
from_array_of_arrays(rows, hlines: true)
|
355
|
+
from_array_of_arrays(rows, hlines: true, tolerant_columns: tolerant_columns)
|
322
356
|
end
|
323
357
|
end
|
324
358
|
|
@@ -412,6 +446,15 @@ module FatTable
|
|
412
446
|
|
413
447
|
# :category: Attributes
|
414
448
|
|
449
|
+
# Return whether the column with the given head should be made tolerant.
|
450
|
+
def tolerant_col?(h)
|
451
|
+
return true if tolerant_columns.include?(:'*')
|
452
|
+
|
453
|
+
tolerant_columns.include?(h)
|
454
|
+
end
|
455
|
+
|
456
|
+
# :category: Attributes
|
457
|
+
|
415
458
|
# Return the number of rows in the Table.
|
416
459
|
def size
|
417
460
|
return 0 if columns.empty?
|
@@ -571,7 +614,8 @@ module FatTable
|
|
571
614
|
range = group_row_range(k)
|
572
615
|
tab_col = column(col)
|
573
616
|
gitems = tab_col.items[range]
|
574
|
-
cols << Column.new(header: col, items: gitems,
|
617
|
+
cols << Column.new(header: col, items: gitems,
|
618
|
+
type: tab_col.type, tolerant: tab_col.tolerant?)
|
575
619
|
end
|
576
620
|
cols
|
577
621
|
end
|
@@ -735,7 +779,6 @@ module FatTable
|
|
735
779
|
last_key = nil
|
736
780
|
new_rows.each_with_index do |nrow, k|
|
737
781
|
new_tab << nrow
|
738
|
-
# key = nrow.fetch_values(*sort_heads)
|
739
782
|
key = nrow.fetch_values(*key_hash.keys)
|
740
783
|
new_tab.mark_boundary(k - 1) if last_key && key != last_key
|
741
784
|
last_key = key
|
@@ -941,7 +984,12 @@ module FatTable
|
|
941
984
|
expr = expr.to_s
|
942
985
|
result = empty_dup
|
943
986
|
headers.each do |h|
|
944
|
-
col =
|
987
|
+
col =
|
988
|
+
if tolerant_col?(h)
|
989
|
+
Column.new(header: h, tolerant: true)
|
990
|
+
else
|
991
|
+
Column.new(header: h)
|
992
|
+
end
|
945
993
|
result.add_column(col)
|
946
994
|
end
|
947
995
|
ev = Evaluator.new(ivars: { row: 0, group: 0 })
|
@@ -1406,6 +1454,9 @@ module FatTable
|
|
1406
1454
|
|
1407
1455
|
private
|
1408
1456
|
|
1457
|
+
# Collapse a group of rows to a single row by applying the aggregator from
|
1458
|
+
# the +agg_cols+ to the items in that column and the presumably identical
|
1459
|
+
# value in the +grp_cols to those columns.
|
1409
1460
|
def row_from_group(rows, grp_cols, agg_cols)
|
1410
1461
|
new_row = {}
|
1411
1462
|
grp_cols.each do |h|
|
@@ -1440,7 +1491,7 @@ module FatTable
|
|
1440
1491
|
# This column is new, so it needs nil items for all prior rows lest
|
1441
1492
|
# the value be added to a prior row.
|
1442
1493
|
items = Array.new(size, nil)
|
1443
|
-
columns << Column.new(header: h, items: items)
|
1494
|
+
columns << Column.new(header: h, items: items, tolerant: tolerant_col?(h))
|
1444
1495
|
end
|
1445
1496
|
headers.each do |h|
|
1446
1497
|
# NB: This adds a nil if h is not in row.
|
@@ -1660,11 +1711,19 @@ module FatTable
|
|
1660
1711
|
end
|
1661
1712
|
|
1662
1713
|
# The <=> operator cannot handle nils without some help. Treat a nil as
|
1663
|
-
# smaller than any other value, but equal to other nils. The two keys are
|
1664
|
-
# compared with <=>.
|
1714
|
+
# smaller than any other value, but equal to other nils. The two keys are
|
1715
|
+
# assumed to be arrays of values to be compared with <=>. Since
|
1716
|
+
# tolerant_columns permit strings to be mixed in with columns of type
|
1717
|
+
# Numeric, DateTime, and Boolean, treat strings mixed with another type
|
1718
|
+
# the same as nils.
|
1665
1719
|
def compare_with_nils(key1, key2)
|
1666
1720
|
result = nil
|
1667
1721
|
key1.zip(key2) do |k1, k2|
|
1722
|
+
if k1.is_a?(String) && !k2.is_a?(String)
|
1723
|
+
k1 = nil
|
1724
|
+
elsif !k1.is_a?(String) && k2.is_a?(String)
|
1725
|
+
k2 = nil
|
1726
|
+
end
|
1668
1727
|
if k1.nil? && k2.nil?
|
1669
1728
|
result = 0
|
1670
1729
|
next
|
data/lib/fat_table/version.rb
CHANGED
data/lib/fat_table.rb
CHANGED
@@ -61,22 +61,22 @@ module FatTable
|
|
61
61
|
|
62
62
|
# Return an empty FatTable::Table object. You can use FatTable::Table#add_row
|
63
63
|
# or FatTable::Table#add_column to populate the table with data.
|
64
|
-
def self.new(*args)
|
65
|
-
Table.new(*args)
|
64
|
+
def self.new(*args, tolerant_columns: [])
|
65
|
+
Table.new(*args, tolerant_columns: tolerant_columns)
|
66
66
|
end
|
67
67
|
|
68
68
|
# Construct a FatTable::Table from the contents of a CSV file given by the
|
69
69
|
# file name +fname+. Headers will be taken from the first row and converted to
|
70
70
|
# symbols.
|
71
|
-
def self.from_csv_file(fname)
|
72
|
-
Table.from_csv_file(fname)
|
71
|
+
def self.from_csv_file(fname, tolerant_columns: [])
|
72
|
+
Table.from_csv_file(fname, tolerant_columns: tolerant_columns)
|
73
73
|
end
|
74
74
|
|
75
75
|
# Construct a FatTable::Table from the string +str+, treated in the same
|
76
76
|
# manner as if read the input from a CSV file. Headers will be taken from the
|
77
77
|
# first row and converted to symbols.
|
78
|
-
def self.from_csv_string(str)
|
79
|
-
Table.from_csv_string(str)
|
78
|
+
def self.from_csv_string(str, tolerant_columns: [])
|
79
|
+
Table.from_csv_string(str, tolerant_columns: tolerant_columns)
|
80
80
|
end
|
81
81
|
|
82
82
|
# Construct a FatTable::Table from the first table found in the Emacs org-mode
|
@@ -84,8 +84,8 @@ module FatTable
|
|
84
84
|
# is an hline. Otherwise, synthetic headers of the form +:col_1+, +:col_2+,
|
85
85
|
# etc. are created. Any other hlines will be treated as marking a boundary in
|
86
86
|
# the table.
|
87
|
-
def self.from_org_file(fname)
|
88
|
-
Table.from_org_file(fname)
|
87
|
+
def self.from_org_file(fname, tolerant_columns: [])
|
88
|
+
Table.from_org_file(fname, tolerant_columns: tolerant_columns)
|
89
89
|
end
|
90
90
|
|
91
91
|
# Construct a FatTable::Table from the first table found in the string +str+,
|
@@ -93,8 +93,8 @@ module FatTable
|
|
93
93
|
# are taken from the first row if the second row is an hrule. Otherwise,
|
94
94
|
# synthetic headers of the form :col_1, :col_2, etc. are created. Any other
|
95
95
|
# hlines will be treated as marking a boundary in the table.
|
96
|
-
def self.from_org_string(str)
|
97
|
-
Table.from_org_string(str)
|
96
|
+
def self.from_org_string(str, tolerant_columns: [])
|
97
|
+
Table.from_org_string(str, tolerant_columns: tolerant_columns)
|
98
98
|
end
|
99
99
|
|
100
100
|
# Construct a FatTable::Table from the array of arrays +aoa+. By default, with
|
@@ -108,8 +108,8 @@ module FatTable
|
|
108
108
|
# org-mode code blocks, by default (+:hlines no+) all hlines are stripped from
|
109
109
|
# the table, otherwise (+:hlines yes+) they are indicated with nil elements in
|
110
110
|
# the outer array.
|
111
|
-
def self.from_aoa(aoa, hlines: false)
|
112
|
-
Table.from_aoa(aoa, hlines: hlines)
|
111
|
+
def self.from_aoa(aoa, hlines: false, tolerant_columns: [])
|
112
|
+
Table.from_aoa(aoa, hlines: hlines, tolerant_columns: tolerant_columns)
|
113
113
|
end
|
114
114
|
|
115
115
|
# Construct a FatTable::Table from the array of hashes +aoh+, which can be an
|
@@ -117,8 +117,8 @@ module FatTable
|
|
117
117
|
# interpret nil separators as marking boundaries in the new Table. All hashes
|
118
118
|
# must have the same keys, which, converted to symbols, become the headers for
|
119
119
|
# the new Table.
|
120
|
-
def self.from_aoh(aoh, hlines: false)
|
121
|
-
Table.from_aoh(aoh, hlines: hlines)
|
120
|
+
def self.from_aoh(aoh, hlines: false, tolerant_columns: [])
|
121
|
+
Table.from_aoh(aoh, hlines: hlines, tolerant_columns: tolerant_columns)
|
122
122
|
end
|
123
123
|
|
124
124
|
# Construct a FatTable::Table from another FatTable::Table. Inherit any group
|
@@ -130,8 +130,8 @@ module FatTable
|
|
130
130
|
# Construct a Table by running a SQL query against the database set up with
|
131
131
|
# FatTable.connect. Return the Table with the query results as rows and the
|
132
132
|
# headers from the query, converted to symbols, as headers.
|
133
|
-
def self.from_sql(query)
|
134
|
-
Table.from_sql(query)
|
133
|
+
def self.from_sql(query, tolerant_columns: [])
|
134
|
+
Table.from_sql(query, tolerant_columns: tolerant_columns)
|
135
135
|
end
|
136
136
|
|
137
137
|
########################################################################
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fat_table
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.5.
|
4
|
+
version: 0.5.5
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Daniel E. Doherty
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2022-
|
11
|
+
date: 2022-03-25 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|