fat_table 0.5.2 → 0.5.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: f35f99181e39d7569ce7df958ae92742617826e54b8ed2c6352b73c5723a3d44
4
- data.tar.gz: d37acb3f0be27191bafa17ae30b51dfd2db2c4a5d97e6c80fa5399f2618ea98b
3
+ metadata.gz: 2f391d5e4ad9d7a4dcb303098b90f6fb42379d9df382192c4bacaf169523dab1
4
+ data.tar.gz: ea75f906fcd164752a3ca2220324ad10d8438562bb31c4846bc6575723e834fa
5
5
  SHA512:
6
- metadata.gz: eafd2077939c57673358a680490ccbe48218e8b290540790f3526b5945337e53c9366369c255adcc85e01916ad012cbad12132593def15350fd3993304bc5a30
7
- data.tar.gz: e7a542986752ac4f632e7344c49080786e0e52ecbe224a0644a6da7b4ec65dd8ca2b9abe842ee295ad7cf19a7a0ccd0e93b22632716c95533665a21d14b5cc83
6
+ metadata.gz: 2914c40a497cf24bcdba8868c4a4607ea1bdb0a94fba8caa3d6e31407ff6832621df2a0b4a8a2bb98c3b40080fed6d752f8e936f66a96270fe675e93d5a852d6
7
+ data.tar.gz: dbc43f4d2bac4a2a19bc7cc90d1fbfbf739a12f0f280a543670af588cfcf2bb0df74d50cfb7ded76b851f5470fd30d4f971659cead5131f9ea15e24b492e5823
data/README.org CHANGED
@@ -25,6 +25,16 @@ The following is for org.
25
25
 
26
26
  [[https://travis-ci.org/ddoherty03/fat_table.svg?branch=master]]
27
27
 
28
+ * Version
29
+ #+begin_src ruby :wrap EXAMPLE
30
+ require 'fat_table'
31
+ "Current version is: #{FatTable::VERSION}"
32
+ #+end_src
33
+
34
+ #+begin_EXAMPLE
35
+ Current version is: 0.5.4
36
+ #+end_EXAMPLE
37
+
28
38
  * Introduction
29
39
 
30
40
  ~FatTable~ is a gem that treats tables as a data type. It provides methods for
@@ -57,6 +67,7 @@ array of arrays with its ~.to_aoa~ output function will be rendered in an
57
67
  org-mode buffer as an org-table, ready for processing by other code blocks.
58
68
 
59
69
  * Table of Contents :toc:noexport:
70
+ - [[#version][Version]]
60
71
  - [[#introduction][Introduction]]
61
72
  - [[#installation][Installation]]
62
73
  - [[#using-in-a-gem][Using in a gem]]
@@ -74,6 +85,7 @@ org-mode buffer as an org-table, ready for processing by other code blocks.
74
85
  - [[#without-headers][Without Headers]]
75
86
  - [[#with-headers][With Headers]]
76
87
  - [[#forcing-string-type][Forcing String Type]]
88
+ - [[#designating-tolerant-columns][Designating "Tolerant" Columns]]
77
89
  - [[#from-csv-or-org-mode-files-or-strings][From CSV or Org Mode files or strings]]
78
90
  - [[#from-arrays-of-arrays][From Arrays of Arrays]]
79
91
  - [[#in-ruby-code][In Ruby Code]]
@@ -430,11 +442,26 @@ or nil. There are only five permissible types for a ~Column~:
430
442
  5. *NilClass* (for the undetermined column type).
431
443
 
432
444
  When a ~Table~ is constructed from an external source, all ~Columns~ start out
433
- having a type of ~NilClass~, that is, their type is as yet undetermined. When a
434
- string or object of one of the four determined types is added to a ~Column~, it
435
- fixes the type of the column and all further items added to the ~Column~ must
436
- either be ~nil~ (indicating no value) or be capable of being coerced to the
437
- column's type. Otherwise, ~FatTable~ raises an exception.
445
+ having a type of ~NilClass~, that is, their type is as yet undetermined. When
446
+ a string or object of one of the four determined types is added to a ~Column~
447
+ and it can be converted into one of the permissible types, it fixes the type
448
+ of the column, and all further items added to the ~Column~ must either be
449
+ ~nil~ (indicating no value) or be capable of being coerced to the column's
450
+ type. Otherwise, ~FatTable~ raises an exception.
451
+
452
+ The strictness of requiring all items to be of the same type can be relaxed by
453
+ declaring a column to be "tolerant." You can do so when you create the table
454
+ by adding a tolerant_columns keyword parameter. If a Column is tolerant,
455
+ ~FatTable~ tries to convert new items into a type other than a ~String~ and,
456
+ if it can do so, sets /that/ as the Column's type. Any later items that
457
+ cannot be converted into the Column's type are converted to strings. These
458
+ interloper strings are treated like nils for purposes of sorting and
459
+ evaluation, but are displayed according to any string formatting on output.
460
+ See [[*Designating "Tolerant" Columns][Designating "Tolerant" Columns]] below.
461
+
462
+ It is also possible to force ~FatTable~ to treat a column as a String type,
463
+ even its items look like one of the other types. See [[*Forcing String Type][Forcing String Type]]
464
+ below.
438
465
 
439
466
  Items of input must be either one of the permissible ruby objects or strings. If
440
467
  they are strings, ~FatTable~ attempts to parse them as one of the permissible
@@ -591,10 +618,18 @@ columns to be created:
591
618
  **** Forcing String Type
592
619
  Occasionally, ~FatTable~'s automatic type detection can get in the way and you
593
620
  just want it to treat one or more columns as Strings regardless of their
594
- appearance. Think, for example, of zip codes. At any time after creating a
595
- table, you can have it force the String type on any number of columns with the
596
- ~force_string!~ method. When you do so, all exisiting items in the column are
597
- converted to strings with the #to_s method.
621
+ appearance. Think, for example, of zip codes. If headers are given when a
622
+ table is contructed, you can designate a forced-string column by appending a
623
+ ~!~ to the end of the header. It will not become part of the header, it will
624
+ just mark it as a forced-string Column.
625
+
626
+ #+begin_SRC emacs-lisp :wrap EXAMPLE
627
+ tab = FatTable.new(:a, 'b', 'C!', :d, :zip!)
628
+ #+end_SRC
629
+
630
+ In addition, at any time after creating a table, you can force the String type
631
+ on any number of columns with the ~force_string!~ method. When you do so, all
632
+ exisiting items in the column are converted to strings with the #to_s method.
598
633
 
599
634
  #+begin_src ruby :wrap EXAMPLE
600
635
  tab = FatTable.new(:a, 'b', 'C', :d, :zip)
@@ -619,8 +654,83 @@ converted to strings with the #to_s method.
619
654
  +======+======+============+===+=======+===+
620
655
  #+end_EXAMPLE
621
656
 
657
+ **** Designating "Tolerant" Columns
658
+ Related to the problem just discussed is the problem of reading files in from
659
+ the wild where a column may get typed as, say Numeric, but then contain
660
+ something that can't be parsed as a Numeric. ~FatTable~ raises an exception
661
+ is such cases, and that may be what you want if you can control the input.
662
+ But, especially when you cannot do so, it can be helpful to designate one or
663
+ more columns as "tolerant." This means that when a conversion problem occurs,
664
+ the column item is retained as a string type in a column that is otherwise of
665
+ one of the types Numeric, DateTime, or Boolean. Those string items are
666
+ treated as nils for purposes of sorting or evaluation in a ~select~ method.
667
+ When formatted, they participate in string formatting directive, but not those
668
+ for other types.
669
+
670
+ All of the table construction methods, allow a keyword parameter,
671
+ ~tolerant_columns~, where you can designate what columns should be convert to
672
+ String type when conversion to the auto-typed column type is not possible.
673
+ The parameter should be an array of headers, in either string or symbol form,
674
+ for which this behavior is desired. In addition, it can be set to the special
675
+ string '*' or symbol ~:*~ to indicate that all the columns should be made
676
+ tolerant.
677
+
678
+ #+begin_src ruby :wrap EXAMPLE
679
+ require 'fat_table'
680
+ tab = FatTable.new(:a, 'b', 'C', :d, :zip, tolerant_columns: [:zip])
681
+ tab << { a: 1, b: 2, c: "<2017-01-21>", d: 'f', e: '', zip: 18552 }
682
+ tab << { a: 3.14, b: 2.17, c: '[2016-01-21 Thu]', d: 'Y', e: nil }
683
+ tab << { zip: '01879--7884' }
684
+ tab << { zip: '66210' }
685
+ tab << { zip: '90210' }
686
+ tab.to_text
687
+ #+end_src
688
+
689
+ #+RESULTS:
690
+ #+begin_EXAMPLE
691
+ +======+======+============+===+=============+===+
692
+ | A | B | C | D | Zip | E |
693
+ +------+------+------------+---+-------------+---+
694
+ | 1 | 2 | 2017-01-21 | F | 18552 | |
695
+ | 3.14 | 2.17 | 2016-01-21 | T | | |
696
+ | | | | | 01879--7884 | |
697
+ | | | | | 66210 | |
698
+ | | | | | 90210 | |
699
+ +======+======+============+===+=============+===+
700
+ #+end_EXAMPLE
701
+
702
+ Another way to designate a column as tolerant is to end a column you want to
703
+ designate as tolerant with a ~!~. The ~!~ will be stripped from the header,
704
+ but it will be marked as tolerant.
705
+ #+begin_src ruby :wrap EXAMPLE
706
+ require 'fat_table'
707
+ tab = FatTable.new(:a, 'b!', 'C', :d, :zip!)
708
+ tab << { a: 1, b: 2, c: "<2017-01-21>", d: 'f', e: '', zip: 18552 }
709
+ tab << { a: 3.14, b: 2.17, c: '[2016-01-21 Thu]', d: 'Y', e: nil }
710
+ tab << { zip: '01879--7884' }
711
+ tab << { zip: '66210', b: 'Not a Number' }
712
+ tab << { zip: '90210' }
713
+ tab.to_text
714
+ #+end_src
715
+
716
+ #+RESULTS:
717
+ #+begin_EXAMPLE
718
+ +======+==============+============+===+=============+===+
719
+ | A | B | C | D | Zip | E |
720
+ +------+--------------+------------+---+-------------+---+
721
+ | 1 | 2 | 2017-01-21 | F | 18552 | |
722
+ | 3.14 | 2.17 | 2016-01-21 | T | | |
723
+ | | | | | 01879--7884 | |
724
+ | | Not a Number | | | 66210 | |
725
+ | | | | | 90210 | |
726
+ +======+==============+============+===+=============+===+
727
+ #+end_EXAMPLE
728
+
622
729
  *** From CSV or Org Mode files or strings
623
- Tables can also be read from ~.csv~ files or files containing ~org-mode~ tables.
730
+ Tables can also be read from ~.csv~ files or files containing ~org-mode~
731
+ tables. Remember that you can make any column tolerant with a
732
+ ~tolerant_columns:~ keyword argument or make them all tolerant by designating
733
+ the pseudo-column ~:*~ as tolerant.
624
734
 
625
735
  In the case of org-mode files, ~FatTable~ skips through the file until it finds
626
736
  a line that look like a table, that is, it begins with any number of spaces
@@ -677,8 +787,10 @@ header row, and the headers are converted to symbols as described above.
677
787
 
678
788
  *** From Arrays of Arrays
679
789
  **** In Ruby Code
680
- You can also initialize a table directly from ruby data structures. You can, for
681
- example, build a table from an array of arrays:
790
+ You can also initialize a table directly from ruby data structures. You can,
791
+ for example, build a table from an array of arrays. Remember that you can
792
+ make any column tolerant with a ~tolerant_columns:~ keyword argument or make
793
+ them all tolerant by designating the pseudo-column ~:*~ as tolerant.
682
794
 
683
795
  #+BEGIN_SRC ruby
684
796
  aoa = [
@@ -772,8 +884,10 @@ This example illustrates several things:
772
884
 
773
885
  A second ruby data structure that can be used to initialize a ~FatTable~ table
774
886
  is an array of ruby Hashes. Each hash represents a row of the table, and the
775
- headers of the table are taken from the keys of the hashes. Accordingly, all the
776
- hashes must have the same keys.
887
+ headers of the table are taken from the keys of the hashes. Accordingly, all
888
+ the hashes must have the same keys. Remember that you can make any column
889
+ tolerant with a ~tolerant_columns:~ keyword argument or make them all tolerant
890
+ by designating the pseudo-column ~:*~ as tolerant.
777
891
 
778
892
  This same method can in fact take an array of any objects that can be converted
779
893
  to a Hash with the ~#to_h~ method, so you can use an array of your own objects
@@ -862,6 +976,10 @@ The ~.connect~ function need only be called once, and the database handle it
862
976
  creates will be used for all subsequent ~.from_sql~ calls until ~.connect~ is
863
977
  called again.
864
978
 
979
+ Remember that you can make any column tolerant with a ~tolerant_columns:~
980
+ keyword argument or make them all tolerant by designating the pseudo-column
981
+ ~:*~ as tolerant.
982
+
865
983
  *** Marking Groups in Input
866
984
  **** Manually
867
985
  At any point, you can add a boundary to a table by invokong the
@@ -2549,9 +2667,10 @@ the table, but they can be overridden by more specific directives given in a
2549
2667
  ~format_for~ directive.
2550
2668
 
2551
2669
  **** Type and Column priority
2552
- A directive based on type applies to all columns having that type unless
2553
- overridden by a directive specific to a named column; a directive based on a
2554
- column name applies only to cells in that column.
2670
+ A directive based the column name overrides any directive based on type. If
2671
+ any cell has both a type-based formatting and column-based, the column
2672
+ instructions prevail. In earlier versions the instuctions were "merged" but
2673
+ that is no longer the case.
2555
2674
 
2556
2675
  However, there is a twist. Since the end result of formatting is to convert
2557
2676
  all columns to strings, the formatting directives for the ~String~ type can
data/TODO.org CHANGED
@@ -1,10 +1,37 @@
1
+
2
+ * TODO Specify Column Widths
3
+ Allow a formatter to specify column widths. This could be a number of
4
+ characters, which would be interpreted as a number of "ems" for LaTeX.
5
+ Cell content larger than the width would be truncated. Any column without a
6
+ width specified would be set at the width of the longest value in that cell,
7
+ after initial formatting.
8
+
9
+ #+begin_SRC ruby
10
+ tab.to_text do |f|
11
+ f.widths(a: 13, b: 30)
12
+ end
13
+ #+end_SRC
14
+
15
+ Possible enhancements:
16
+ - specify an overall width and column widths as decimal or fractions, so that
17
+ a column's width would be that fraction of the overall width.
18
+ - specify a Range for a width, so that the column would at least min and at
19
+ most max, otherwise the width of its largest cell.
20
+
1
21
  * TODO Conversion to Spreadsheets
2
22
  - State "TODO" from [2017-04-21 Fri 10:36]
3
23
  This is a [[https://github.com/westonganger/spreadsheet_architect][gem]] that I can include into the Table model to convert a table into
4
24
  a spread-sheet, or even a sheet in a multi-sheet spreadsheet file.
5
25
 
6
- * TODO Add from_yql for fetching from Yahoo
26
+ * TODO Add Quandl or EODDATA Queries
27
+ Possible replacements for YQL.
28
+
29
+ * CNCL Add from_yql for fetching from Yahoo
30
+ CLOSED: [2022-01-30 Sun 06:03]
7
31
  - State "TODO" from [2017-04-21 Fri 10:35]
32
+
33
+ Cancelled because Yahoo shut down the YQL api service.
34
+
8
35
  Add a constructor to allow fetching stock data from yql. Perhaps grab all
9
36
  available fields, then allow a select of those of interest.
10
37
 
data/lib/ext/array.rb CHANGED
@@ -12,4 +12,21 @@ class Array
12
12
  end
13
13
  end
14
14
  end
15
+
16
+ def filter_to_type(typ)
17
+ if typ == 'Boolean'
18
+ compact.select { |i| i.is_a?(TrueClass) || i.is_a?(FalseClass) }
19
+ elsif typ == 'DateTime'
20
+ compact.select { |i| i.is_a?(Date) || i.is_a?(DateTime) || i.is_a?(Time) }
21
+ .map { |i| i.to_datetime }
22
+ elsif typ == 'Numeric'
23
+ compact.select { |i| i.is_a?(Numeric) }
24
+ elsif typ == 'String'
25
+ map { |i| i.to_s }
26
+ elsif typ == 'NilClass'
27
+ self
28
+ else
29
+ raise ArgumentError, "cannot filter_to_type for type '#{typ}'"
30
+ end
31
+ end
15
32
  end
@@ -83,7 +83,7 @@ module FatTable
83
83
  # col.type #=> 'Numeric'
84
84
  # col.header #=> :prices
85
85
  # col.sum #=> 18376.75
86
- def initialize(header:, items: [], type: 'NilClass')
86
+ def initialize(header:, items: [], type: 'NilClass', tolerant: false)
87
87
  @raw_header = header
88
88
  @header =
89
89
  if @raw_header.is_a?(Symbol)
@@ -92,6 +92,7 @@ module FatTable
92
92
  @raw_header.to_s.as_sym
93
93
  end
94
94
  @type = type
95
+ @tolerant = tolerant
95
96
  msg = "unknown column type '#{type}"
96
97
  raise UserError, msg unless TYPES.include?(@type.to_s)
97
98
 
@@ -141,6 +142,14 @@ module FatTable
141
142
 
142
143
  # :category: Attributes
143
144
 
145
+ # Is this column tolerant of type incompatibilities? If so, the Column
146
+ # type will be forced to String if an incompatible type is found.
147
+ def tolerant?
148
+ @tolerant
149
+ end
150
+
151
+ # :category: Attributes
152
+
144
153
  # Force the column to have String type and then convert all items to
145
154
  # strings.
146
155
  def force_string!
@@ -182,12 +191,15 @@ module FatTable
182
191
 
183
192
  # :category: Aggregates
184
193
 
185
- # Return the first non-nil item in the Column. Works with any Column type.
194
+ # Return the first non-nil item in the Column, or nil if all items are
195
+ # nil. Works with any Column type.
186
196
  def first
197
+ return nil if items.all?(&:nil?)
198
+
187
199
  if type == 'String'
188
200
  items.reject(&:blank?).first
189
201
  else
190
- items.compact.first
202
+ items.filter_to_type(type).first
191
203
  end
192
204
  end
193
205
 
@@ -195,83 +207,94 @@ module FatTable
195
207
 
196
208
  # Return the last non-nil item in the Column. Works with any Column type.
197
209
  def last
210
+ return nil if items.all?(&:nil?)
211
+
198
212
  if type == 'String'
199
213
  items.reject(&:blank?).last
200
214
  else
201
- items.compact.last
215
+ items.filter_to_type(type).last
202
216
  end
203
217
  end
204
218
 
205
219
  # :category: Aggregates
206
220
 
207
- # Return a count of the non-nil items in the Column. Works with any Column
208
- # type.
221
+ # Return a count of the non-nil items in the Column, or the size of the
222
+ # column if all items are nil. Works with any Column type.
209
223
  def count
224
+ return items.size if items.all?(&:nil?)
225
+
210
226
  if type == 'String'
211
227
  items.reject(&:blank?).count.to_d
212
228
  else
213
- items.compact.count.to_d
229
+ items.filter_to_type(type).count.to_d
214
230
  end
215
231
  end
216
232
 
217
233
  # :category: Aggregates
218
234
 
219
- # Return the smallest non-nil, non-blank item in the Column. Works with
220
- # numeric, string, and datetime Columns.
235
+ # Return the smallest non-nil, non-blank item in the Column, or nil if all
236
+ # items are nil. Works with numeric, string, and datetime Columns.
221
237
  def min
222
238
  only_with('min', 'NilClass', 'Numeric', 'String', 'DateTime')
223
239
  if type == 'String'
224
240
  items.reject(&:blank?).min
225
241
  else
226
- items.compact.min
242
+ items.filter_to_type(type).min
227
243
  end
228
244
  end
229
245
 
230
246
  # :category: Aggregates
231
247
 
232
- # Return the largest non-nil, non-blank item in the Column. Works with
233
- # numeric, string, and datetime Columns.
248
+ # Return the largest non-nil, non-blank item in the Column, or nil if all
249
+ # items are nil. Works with numeric, string, and datetime Columns.
234
250
  def max
235
251
  only_with('max', 'NilClass', 'Numeric', 'String', 'DateTime')
236
252
  if type == 'String'
237
253
  items.reject(&:blank?).max
238
254
  else
239
- items.compact.max
255
+ items.filter_to_type(type).max
240
256
  end
241
257
  end
242
258
 
243
259
  # :category: Aggregates
244
260
 
245
- # Return a Range object for the smallest to largest value in the column.
246
- # Works with numeric, string, and datetime Columns.
261
+ # Return a Range object for the smallest to largest value in the column,
262
+ # or nil if all items are nil. Works with numeric, string, and datetime
263
+ # Columns.
247
264
  def range
248
265
  only_with('range', 'NilClass', 'Numeric', 'String', 'DateTime')
266
+ return nil if items.all?(&:nil?)
267
+
249
268
  Range.new(min, max)
250
269
  end
251
270
 
252
271
  # :category: Aggregates
253
272
 
254
- # Return the sum of the non-nil items in the Column. Works with numeric and
255
- # string Columns. For a string Column, it will return the concatenation of
256
- # the non-nil items.
273
+ # Return the sum of the non-nil items in the Column, or 0 if all items are
274
+ # nil. Works with numeric and string Columns. For a string Column, it
275
+ # will return the concatenation of the non-nil items.
257
276
  def sum
277
+ return 0 if type == 'NilClass' || items.all?(&:nil?)
278
+
258
279
  only_with('sum', 'Numeric', 'String')
259
280
  if type == 'String'
260
281
  items.reject(&:blank?).join(' ')
261
282
  else
262
- items.compact.sum
283
+ items.filter_to_type(type).sum
263
284
  end
264
285
  end
265
286
 
266
287
  # :category: Aggregates
267
288
 
268
- # Return the average value of the non-nil items in the Column. Works with
269
- # numeric and datetime Columns. For datetime Columns, it converts each date
270
- # to its Julian day number, computes the average, and then converts the
271
- # average back to a DateTime.
289
+ # Return the average value of the non-nil items in the Column, or 0 if all
290
+ # items are nil. Works with numeric and datetime Columns. For datetime
291
+ # Columns, it converts each date to its Julian day number, computes the
292
+ # average, and then converts the average back to a DateTime.
272
293
  def avg
294
+ return 0 if type == 'NilClass' || items.all?(&:nil?)
295
+
273
296
  only_with('avg', 'DateTime', 'Numeric')
274
- itms = items.compact
297
+ itms = items.filter_to_type(type)
275
298
  size = itms.size.to_d
276
299
  if type == 'DateTime'
277
300
  avg_jd = itms.map(&:jd).sum / size
@@ -284,17 +307,20 @@ module FatTable
284
307
  # :category: Aggregates
285
308
 
286
309
  # Return the sample variance (the unbiased estimator of the population
287
- # variance using a divisor of N-1) as the average squared deviation from the
288
- # mean, of the non-nil items in the Column. Works with numeric and datetime
289
- # Columns. For datetime Columns, it converts each date to its Julian day
290
- # number and computes the variance of those numbers.
310
+ # variance using a divisor of N-1) as the average squared deviation from
311
+ # the mean, of the non-nil items in the Column, or 0 if all items are
312
+ # nil. Works with numeric and datetime Columns. For datetime Columns, it
313
+ # converts each date to its Julian day number and computes the variance of
314
+ # those numbers.
291
315
  def var
316
+ return 0 if type == 'NilClass' || items.all?(&:nil?)
317
+
292
318
  only_with('var', 'DateTime', 'Numeric')
293
319
  all_items =
294
320
  if type == 'DateTime'
295
- items.compact.map(&:jd)
321
+ items.filter_to_type(type).map(&:jd)
296
322
  else
297
- items.compact
323
+ items.filter_to_type(type)
298
324
  end
299
325
  n = count
300
326
  return BigDecimal('0.0') if n <= 1
@@ -310,12 +336,15 @@ module FatTable
310
336
 
311
337
  # Return the population variance (the biased estimator of the population
312
338
  # variance using a divisor of N) as the average squared deviation from the
313
- # mean, of the non-nil items in the Column. Works with numeric and datetime
314
- # Columns. For datetime Columns, it converts each date to its Julian day
315
- # number and computes the variance of those numbers.
339
+ # mean, of the non-nil items in the Column, or 0 if all items are
340
+ # nil. Works with numeric and datetime Columns. For datetime Columns, it
341
+ # converts each date to its Julian day number and computes the variance of
342
+ # those numbers.
316
343
  def pvar
344
+ return 0 if type == 'NilClass' || items.all?(&:nil?)
345
+
317
346
  only_with('var', 'DateTime', 'Numeric')
318
- n = items.compact.size.to_d
347
+ n = items.filter_to_type(type).size.to_d
319
348
  return BigDecimal('0.0') if n <= 1
320
349
  var * ((n - 1) / n)
321
350
  end
@@ -324,11 +353,13 @@ module FatTable
324
353
 
325
354
  # Return the sample standard deviation (the unbiased estimator of the
326
355
  # population standard deviation using a divisor of N-1) as the square root
327
- # of the sample variance, of the non-nil items in the Column. Works with
328
- # numeric and datetime Columns. For datetime Columns, it converts each date
329
- # to its Julian day number and computes the standard deviation of those
330
- # numbers.
356
+ # of the sample variance, of the non-nil items in the Column, or 0 if all
357
+ # items are nil. Works with numeric and datetime Columns. For datetime
358
+ # Columns, it converts each date to its Julian day number and computes the
359
+ # standard deviation of those numbers.
331
360
  def dev
361
+ return 0 if type == 'NilClass' || items.all?(&:nil?)
362
+
332
363
  only_with('dev', 'DateTime', 'Numeric')
333
364
  var.sqrt(20)
334
365
  end
@@ -336,12 +367,14 @@ module FatTable
336
367
  # :category: Aggregates
337
368
 
338
369
  # Return the population standard deviation (the biased estimator of the
339
- # population standard deviation using a divisor of N) as the square root of
340
- # the population variance, of the non-nil items in the Column. Works with
341
- # numeric and datetime Columns. For datetime Columns, it converts each date
342
- # to its Julian day number and computes the standard deviation of those
343
- # numbers.
370
+ # population standard deviation using a divisor of N) as the square root
371
+ # of the population variance, of the non-nil items in the Column, or 0 if
372
+ # all items are nil. Works with numeric and datetime Columns. For datetime
373
+ # Columns, it converts each date to its Julian day number and computes the
374
+ # standard deviation of those numbers.
344
375
  def pdev
376
+ return 0 if type == 'NilClass' || items.all?(&:nil?)
377
+
345
378
  only_with('dev', 'DateTime', 'Numeric')
346
379
  Math.sqrt(pvar)
347
380
  end
@@ -349,28 +382,35 @@ module FatTable
349
382
  # :category: Aggregates
350
383
 
351
384
  # Return true if any of the items in the Column are true; otherwise return
352
- # false. Works only with boolean Columns.
385
+ # false, or false if all items are nil. Works only with boolean Columns.
353
386
  def any?
387
+ return false if type == 'NilClass' || items.all?(&:nil?)
388
+
354
389
  only_with('any?', 'Boolean')
355
- items.compact.any?
390
+ items.filter_to_type(type).any?
356
391
  end
357
392
 
358
393
  # :category: Aggregates
359
394
 
360
395
  # Return true if all of the items in the Column are true; otherwise return
361
- # false. Works only with boolean Columns.
396
+ # false, or false if all items are nil. Works only with boolean Columns.
362
397
  def all?
398
+ return false if type == 'NilClass' || items.all?(&:nil?)
399
+
363
400
  only_with('all?', 'Boolean')
364
- items.compact.all?
401
+ items.filter_to_type(type).all?
365
402
  end
366
403
 
367
404
  # :category: Aggregates
368
405
 
369
- # Return true if none of the items in the Column are true; otherwise return
370
- # false. Works only with boolean Columns.
406
+ # Return true if none of the items in the Column are true; otherwise
407
+ # return false, or true if all items are nil. Works only with boolean
408
+ # Columns.
371
409
  def none?
410
+ return true if type == 'NilClass' || items.all?(&:nil?)
411
+
372
412
  only_with('none?', 'Boolean')
373
- items.compact.none?
413
+ items.filter_to_type(type).none?
374
414
  end
375
415
 
376
416
  # :category: Aggregates
@@ -378,14 +418,17 @@ module FatTable
378
418
  # Return true if precisely one of the items in the Column is true;
379
419
  # otherwise return false. Works only with boolean Columns.
380
420
  def one?
421
+ return false if type == 'NilClass' || items.all?(&:nil?)
422
+
381
423
  only_with('one?', 'Boolean')
382
- items.compact.one?
424
+ items.filter_to_type(type).one?
383
425
  end
384
426
 
385
427
  private
386
428
 
387
429
  def only_with(agg, *valid_types)
388
430
  return self if valid_types.include?(type)
431
+
389
432
  msg = "aggregate '#{agg}' cannot be applied to a #{type} column"
390
433
  raise UserError, msg
391
434
  end
@@ -400,9 +443,17 @@ module FatTable
400
443
 
401
444
  # Append +itm+ to end of the Column after converting it to the Column's
402
445
  # type. If the Column's type is still open, i.e. NilClass, attempt to fix
403
- # the Column's type based on the type of +itm+ as with Column.new.
446
+ # the Column's type based on the type of +itm+ as with Column.new. If its
447
+ # a tolerant column, respond to type errors by converting the column to a
448
+ # String type.
404
449
  def <<(itm)
405
- items << convert_to_type(itm)
450
+ items << convert_and_set_type(itm)
451
+ rescue IncompatibleTypeError => ex
452
+ if tolerant?
453
+ items << Convert.convert_to_string(itm)
454
+ else
455
+ raise ex
456
+ end
406
457
  end
407
458
 
408
459
  # :category: Constructors
@@ -418,9 +469,14 @@ module FatTable
418
469
 
419
470
  private
420
471
 
421
- def convert_to_type(val)
422
- new_val = Convert.convert_to_type(val, type)
423
- if new_val && type == 'NilClass'
472
+ def convert_and_set_type(val)
473
+ begin
474
+ new_val = Convert.convert_to_type(val, type, tolerant: tolerant?)
475
+ rescue IncompatibleTypeError
476
+ err_msg = "attempt to add '#{val}' to column '#{header}' already typed as #{type}"
477
+ raise IncompatibleTypeError, err_msg
478
+ end
479
+ if new_val && (type == 'NilClass' || type == 'String')
424
480
  @type =
425
481
  if [true, false].include?(new_val)
426
482
  'Boolean'
@@ -10,7 +10,7 @@ module FatTable
10
10
  # determined, raise an error if the val cannot be converted to the Column
11
11
  # type. Otherwise, returns the converted val as an object of the correct
12
12
  # class.
13
- def self.convert_to_type(val, type)
13
+ def self.convert_to_type(val, type, tolerant: false)
14
14
  case type
15
15
  when 'NilClass'
16
16
  if val != false && val.blank?
@@ -36,8 +36,7 @@ module FatTable
36
36
  else
37
37
  new_val = convert_to_boolean(val)
38
38
  if new_val.nil?
39
- msg = "attempt to add '#{val}' to a column already typed as #{type}"
40
- raise UserError, msg
39
+ raise IncompatibleTypeError
41
40
  end
42
41
  new_val
43
42
  end
@@ -47,8 +46,7 @@ module FatTable
47
46
  else
48
47
  new_val = convert_to_date_time(val)
49
48
  if new_val.nil?
50
- msg = "attempt to add '#{val}' to a column already typed as #{type}"
51
- raise UserError, msg
49
+ raise IncompatibleTypeError
52
50
  end
53
51
  new_val
54
52
  end
@@ -58,24 +56,35 @@ module FatTable
58
56
  else
59
57
  new_val = convert_to_numeric(val)
60
58
  if new_val.nil?
61
- msg = "attempt to add '#{val}' to a column already typed as #{type}"
62
- raise UserError, msg
59
+ raise IncompatibleTypeError
63
60
  end
64
61
  new_val
65
62
  end
66
63
  when 'String'
67
64
  if val.nil?
68
65
  nil
66
+ elsif tolerant
67
+ # Allow String to upgrade to one of Numeric, DateTime, or Boolean if
68
+ # possible.
69
+ if (new_val = convert_to_numeric(val))
70
+ new_val
71
+ elsif (new_val = convert_to_date_time(val))
72
+ new_val
73
+ elsif (new_val = convert_to_boolean(val))
74
+ new_val
75
+ else
76
+ new_val = convert_to_string(val)
77
+ end
78
+ new_val
69
79
  else
70
80
  new_val = convert_to_string(val)
71
81
  if new_val.nil?
72
- msg = "attempt to add '#{val}' to a column already typed as #{type}"
73
- raise UserError, msg
82
+ raise IncompatibleTypeError
74
83
  end
75
84
  new_val
76
85
  end
77
86
  else
78
- raise UserError, "Mysteriously, column has unknown type '#{type}'"
87
+ raise LogicError, "Mysteriously, column has unknown type '#{type}'"
79
88
  end
80
89
  end
81
90
 
@@ -121,6 +130,7 @@ module FatTable
121
130
  return val if val.is_a?(DateTime)
122
131
  return val if val.is_a?(Date)
123
132
  return val.to_datetime if val.is_a?(Time)
133
+
124
134
  begin
125
135
  str = val.to_s.clean
126
136
  return nil if str.blank?
@@ -9,6 +9,10 @@ module FatTable
9
9
  # cannot correct.
10
10
  class LogicError < StandardError; end
11
11
 
12
+ # Raised when attempting to add an incompatible type to an already-typed
13
+ # Column.
14
+ class IncompatibleTypeError < UserError; end
15
+
12
16
  # Raised when an external resource is not available due to caller or
13
17
  # programmer error or some failure of the external resource to be available.
14
18
  class TransientError < StandardError; end
@@ -45,17 +45,15 @@ module FatTable
45
45
  end
46
46
 
47
47
  # Return the result of evaluating +expr+ as a Ruby expression in which the
48
- # instance variables set in Evaluator.new and any local variables set in the
49
- # Hash parameter +locals+ are available to the expression.
48
+ # instance variables set in Evaluator.new and any local variables set in
49
+ # the Hash parameter +locals+ are available to the expression. Certain
50
+ # errors simply return nil as the result. This can happen, for example,
51
+ # when a string gets into an otherwise numeric column because the column
52
+ # is set to tolerant.
50
53
  def evaluate(expr = '', locals: {})
51
54
  eval(expr, local_vars(binding, locals))
52
55
  rescue NoMethodError, TypeError => ex
53
- if ex.to_s =~ /for nil:NilClass|nil can't be coerced/
54
- # Likely one of the locals was nil, so let nil be the result.
55
- return nil
56
- else
57
- raise ex
58
- end
56
+ nil
59
57
  end
60
58
 
61
59
  private
@@ -166,7 +166,7 @@ module FatTable
166
166
  when String
167
167
  begin
168
168
  converted_val = Convert.convert_to_type(agg, column.type)
169
- rescue UserError
169
+ rescue UserError, IncompatibleTypeError
170
170
  converted_val = false
171
171
  end
172
172
  if converted_val
@@ -532,7 +532,7 @@ module FatTable
532
532
  valid_keys = table.headers + %i[string numeric datetime boolean nil]
533
533
  invalid_keys = (fmts.keys - valid_keys).uniq
534
534
  unless invalid_keys.empty?
535
- msg = "invalid #{location} column or type: #{invalid_keys.join(',')}"
535
+ msg = "invalid #{location} column or type: #{invalid_keys.join(', ')}"
536
536
  raise UserError, msg
537
537
  end
538
538
 
@@ -562,20 +562,21 @@ module FatTable
562
562
  end
563
563
  end
564
564
 
565
- # Merge in formatting for column h based on the column type, or based
566
- # on the string type for the header location.
565
+ # Merge in formatting instructions for column h based on the column
566
+ # name, or if there is no formatting instructions for the column by
567
+ # name, merge in the formatting instructions based on the column's
568
+ # type. Insist on only the string type for the header location.
567
569
  typ = (location == :header ? :string : table.type(h).as_sym)
568
570
  parse_typ_method_name = 'parse_' + typ.to_s + '_fmt'
569
- if fmts.key?(typ)
570
- # Merge in type-based formatting
571
- typ_fmt = send(parse_typ_method_name, fmts[typ]).first
572
- format_h = format_h.merge(typ_fmt)
573
- end
574
571
  if fmts[h]
575
572
  # Merge in column formatting
576
573
  col_fmt = send(parse_typ_method_name, fmts[h],
577
574
  strict: location != :header).first
578
575
  format_h = format_h.merge(col_fmt)
576
+ elsif fmts.key?(typ)
577
+ # Merge in type-based formatting
578
+ typ_fmt = send(parse_typ_method_name, fmts[typ]).first
579
+ format_h = format_h.merge(typ_fmt)
579
580
  end
580
581
 
581
582
  # Copy :body formatting for column h to :bfirst and :gfirst if they
@@ -1003,9 +1004,14 @@ module FatTable
1003
1004
  # converted to strings formatted according to the Formatter's formatting
1004
1005
  # directives given in Formatter.format_for or Formatter.format.
1005
1006
  def output
1006
- # This results in a hash of two-element arrays. The key is the header and
1007
- # the value is an array of the header and formatted header. We do the
1008
- # latter so the structure parallels the structure for rows explained next.
1007
+ # If there are neither headers nor any rows in the table, return an
1008
+ # empty string.
1009
+ return '' if table.empty? && table.headers.empty?
1010
+
1011
+ # This results in a hash of two-element arrays. The key
1012
+ # is the header and the value is an array of the header and formatted
1013
+ # header. We do the latter so the structure parallels the structure for
1014
+ # rows explained next.
1009
1015
  formatted_headers = build_formatted_headers
1010
1016
 
1011
1017
  # These produce an array with each element representing a row of the
@@ -20,7 +20,7 @@ module FatTable
20
20
  end
21
21
 
22
22
  def pre_header(widths)
23
- result = '|'
23
+ result = +'|'
24
24
  widths.each_value do |w|
25
25
  result += '-' * (w + 2) + '+'
26
26
  end
@@ -53,7 +53,7 @@ module FatTable
53
53
  end
54
54
 
55
55
  def hline(widths)
56
- result = '|'
56
+ result = +'|'
57
57
  widths.each_value do |w|
58
58
  result += '-' * (w + 2) + '+'
59
59
  end
@@ -62,7 +62,7 @@ module FatTable
62
62
  end
63
63
 
64
64
  def post_footers(widths)
65
- result = '|'
65
+ result = +'|'
66
66
  widths.each_value do |w|
67
67
  result += '-' * (w + 2) + '+'
68
68
  end
@@ -221,7 +221,7 @@ module FatTable
221
221
  end
222
222
 
223
223
  def pre_header(widths)
224
- result = upper_left
224
+ result = +upper_left
225
225
  widths.each_value do |w|
226
226
  result += double_rule * (w + 2) + upper_tee
227
227
  end
@@ -255,7 +255,7 @@ module FatTable
255
255
  end
256
256
 
257
257
  def hline(widths)
258
- result = left_tee
258
+ result = +left_tee
259
259
  widths.each_value do |w|
260
260
  result += horizontal_rule * (w + 2) + single_cross
261
261
  end
@@ -289,7 +289,7 @@ module FatTable
289
289
  end
290
290
 
291
291
  def post_footers(widths)
292
- result = lower_left
292
+ result = +lower_left
293
293
  widths.each_value do |w|
294
294
  result += double_rule * (w + 2) + lower_tee
295
295
  end
@@ -16,7 +16,7 @@ module FatTable
16
16
  end
17
17
 
18
18
  def pre_header(widths)
19
- result = '+'
19
+ result = +'+'
20
20
  widths.each_value do |w|
21
21
  result += '=' * (w + 2) + '+'
22
22
  end
@@ -49,7 +49,7 @@ module FatTable
49
49
  end
50
50
 
51
51
  def hline(widths)
52
- result = '+'
52
+ result = +'+'
53
53
  widths.each_value do |w|
54
54
  result += '-' * (w + 2) + '+'
55
55
  end
@@ -82,7 +82,7 @@ module FatTable
82
82
  end
83
83
 
84
84
  def post_footers(widths)
85
- result = '+'
85
+ result = +'+'
86
86
  widths.each_value do |w|
87
87
  result += '=' * (w + 2) + '+'
88
88
  end
@@ -62,19 +62,53 @@ module FatTable
62
62
  # method call.
63
63
  attr_accessor :explicit_boundaries
64
64
 
65
+ # An Array of FatTable::Columns that should be tolerant.
66
+ attr_reader :tolerant_columns
67
+
65
68
  ###########################################################################
66
69
  # Constructors
67
70
  ###########################################################################
68
71
 
69
72
  # :category: Constructors
70
73
 
71
- # Return an empty FatTable::Table object.
72
- def initialize(*heads)
74
+ # Return an empty FatTable::Table object. Specifying headers is optional.
75
+ # Any headers ending with a ! are marked as tolerant, in that, if an
76
+ # incompatible type is added to it, the column is re-typed as a String
77
+ # column, and construction proceeds. The ! is stripped from the header to
78
+ # form the column key, though. You can also provide the names of columns
79
+ # that should be tolerant by using the +tolerant_columns key-word to
80
+ # provide an array of headers that should be tolerant. The special string
81
+ # '*' or the symbol :* indicates that all columns should be created
82
+ # tolerant.
83
+ def initialize(*heads, tolerant_columns: [])
73
84
  @columns = []
74
85
  @explicit_boundaries = []
86
+ @tolerant_columns =
87
+ case tolerant_columns
88
+ when Array
89
+ tolerant_columns.map { |h| h.to_s.as_sym }
90
+ when String
91
+ if tolerant_columns.strip == '*'
92
+ ['*'.to_sym]
93
+ else
94
+ [tolerant_columns.as_sym]
95
+ end
96
+ when Symbol
97
+ if tolerant_columns.to_s.strip == '*'
98
+ ['*'.to_sym]
99
+ else
100
+ [tolerant_columns.to_s.as_sym]
101
+ end
102
+ else
103
+ raise ArgumentError, "set tolerant_columns to String, Symbol, or an Array of either"
104
+ end
75
105
  unless heads.empty?
76
106
  heads.each do |h|
77
- @columns << Column.new(header: h)
107
+ if h.to_s.end_with?('!') || @tolerant_columns.include?(h)
108
+ @columns << Column.new(header: h.to_s.sub(/!\s*\z/, ''), type: 'String')
109
+ else
110
+ @columns << Column.new(header: h)
111
+ end
78
112
  end
79
113
  end
80
114
  end
@@ -86,7 +120,7 @@ module FatTable
86
120
  # though FatTable::Table objects have no instance variables, a class that
87
121
  # inherits from it might.
88
122
  def empty_dup
89
- self.dup.__empty!
123
+ dup.__empty!
90
124
  end
91
125
 
92
126
  def __empty!
@@ -99,9 +133,9 @@ module FatTable
99
133
 
100
134
  # Construct a Table from the contents of a CSV file named +fname+. Headers
101
135
  # will be taken from the first CSV row and converted to symbols.
102
- def self.from_csv_file(fname)
136
+ def self.from_csv_file(fname, tolerant_columns: [])
103
137
  File.open(fname, 'r') do |io|
104
- from_csv_io(io)
138
+ from_csv_io(io, tolerant_columns: tolerant_columns)
105
139
  end
106
140
  end
107
141
 
@@ -109,8 +143,8 @@ module FatTable
109
143
 
110
144
  # Construct a Table from a CSV string +str+, treated in the same manner as
111
145
  # the input from a CSV file in ::from_org_file.
112
- def self.from_csv_string(str)
113
- from_csv_io(StringIO.new(str))
146
+ def self.from_csv_string(str, tolerant_columns: [])
147
+ from_csv_io(StringIO.new(str), tolerant_columns: tolerant_columns)
114
148
  end
115
149
 
116
150
  # :category: Constructors
@@ -119,9 +153,9 @@ module FatTable
119
153
  # file named +fname+. Headers are taken from the first row if the second row
120
154
  # is an hrule. Otherwise, synthetic headers of the form +:col_1+, +:col_2+,
121
155
  # etc. are created.
122
- def self.from_org_file(fname)
156
+ def self.from_org_file(fname, tolerant_columns: [])
123
157
  File.open(fname, 'r') do |io|
124
- from_org_io(io)
158
+ from_org_io(io, tolerant_columns: tolerant_columns)
125
159
  end
126
160
  end
127
161
 
@@ -129,8 +163,8 @@ module FatTable
129
163
 
130
164
  # Construct a Table from a string +str+, treated in the same manner as the
131
165
  # contents of an org-mode file in ::from_org_file.
132
- def self.from_org_string(str)
133
- from_org_io(StringIO.new(str))
166
+ def self.from_org_string(str, tolerant_columns: [])
167
+ from_org_io(StringIO.new(str), tolerant_columns: tolerant_columns)
134
168
  end
135
169
 
136
170
  # :category: Constructors
@@ -149,8 +183,8 @@ module FatTable
149
183
  # :hlines no +) org-mode strips all hrules from the table; otherwise (+
150
184
  # HEADER: :hlines yes +) they are indicated with nil elements in the outer
151
185
  # array.
152
- def self.from_aoa(aoa, hlines: false)
153
- from_array_of_arrays(aoa, hlines: hlines)
186
+ def self.from_aoa(aoa, hlines: false, tolerant_columns: [])
187
+ from_array_of_arrays(aoa, hlines: hlines, tolerant_columns: tolerant_columns)
154
188
  end
155
189
 
156
190
  # :category: Constructors
@@ -160,9 +194,9 @@ module FatTable
160
194
  # keys, which, when converted to symbols will become the headers for the
161
195
  # Table. If hlines is set true, mark a group boundary whenever a nil, rather
162
196
  # than a hash appears in the outer array.
163
- def self.from_aoh(aoh, hlines: false)
197
+ def self.from_aoh(aoh, hlines: false, tolerant_columns: [])
164
198
  if aoh.first.respond_to?(:to_h)
165
- from_array_of_hashes(aoh, hlines: hlines)
199
+ from_array_of_hashes(aoh, hlines: hlines, tolerant_columns: tolerant_columns)
166
200
  else
167
201
  raise UserError,
168
202
  "Cannot initialize Table with an array of #{input[0].class}"
@@ -181,7 +215,7 @@ module FatTable
181
215
 
182
216
  # Construct a Table by running a SQL +query+ against the database set up
183
217
  # with FatTable.connect, with the rows of the query result as rows.
184
- def self.from_sql(query)
218
+ def self.from_sql(query, tolerant_columns: [])
185
219
  msg = 'FatTable.db must be set with FatTable.connect'
186
220
  raise UserError, msg if FatTable.db.nil?
187
221
 
@@ -203,8 +237,8 @@ module FatTable
203
237
  # Construct table from an array of hashes or an array of any object that
204
238
  # can respond to #to_h. If an array element is a nil, mark it as a group
205
239
  # boundary in the Table.
206
- def from_array_of_hashes(hashes, hlines: false)
207
- result = new
240
+ def from_array_of_hashes(hashes, hlines: false, tolerant_columns: [])
241
+ result = new(tolerant_columns: tolerant_columns)
208
242
  hashes.each do |hsh|
209
243
  if hsh.nil?
210
244
  unless hlines
@@ -232,8 +266,8 @@ module FatTable
232
266
  # hlines are stripped from the table, otherwise (:hlines yes) they are
233
267
  # indicated with nil elements in the outer array as expected by this
234
268
  # method when hlines is set true.
235
- def from_array_of_arrays(rows, hlines: false)
236
- result = new
269
+ def from_array_of_arrays(rows, hlines: false, tolerant_columns: [])
270
+ result = new(tolerant_columns: tolerant_columns)
237
271
  headers = []
238
272
  if !hlines
239
273
  # Take the first row as headers
@@ -269,8 +303,8 @@ module FatTable
269
303
  result
270
304
  end
271
305
 
272
- def from_csv_io(io)
273
- result = new
306
+ def from_csv_io(io, tolerant_columns: [])
307
+ result = new(tolerant_columns: tolerant_columns)
274
308
  ::CSV.new(io, headers: true, header_converters: :symbol,
275
309
  skip_blanks: true).each do |row|
276
310
  result << row.to_h
@@ -283,7 +317,7 @@ module FatTable
283
317
  # header row must be marked with an hline (i.e, a row that looks like
284
318
  # '|---+--...--|') and groups of rows may be marked with hlines to
285
319
  # indicate group boundaries.
286
- def from_org_io(io)
320
+ def from_org_io(io, tolerant_columns: [])
287
321
  table_re = /\A\s*\|/
288
322
  hrule_re = /\A\s*\|[-+]+/
289
323
  rows = []
@@ -318,7 +352,7 @@ module FatTable
318
352
  rows << line.split('|').map(&:clean)
319
353
  end
320
354
  end
321
- from_array_of_arrays(rows, hlines: true)
355
+ from_array_of_arrays(rows, hlines: true, tolerant_columns: tolerant_columns)
322
356
  end
323
357
  end
324
358
 
@@ -412,6 +446,15 @@ module FatTable
412
446
 
413
447
  # :category: Attributes
414
448
 
449
+ # Return whether the column with the given head should be made tolerant.
450
+ def tolerant_col?(h)
451
+ return true if tolerant_columns.include?(:'*')
452
+
453
+ tolerant_columns.include?(h)
454
+ end
455
+
456
+ # :category: Attributes
457
+
415
458
  # Return the number of rows in the Table.
416
459
  def size
417
460
  return 0 if columns.empty?
@@ -571,7 +614,8 @@ module FatTable
571
614
  range = group_row_range(k)
572
615
  tab_col = column(col)
573
616
  gitems = tab_col.items[range]
574
- cols << Column.new(header: col, items: gitems, type: tab_col.type)
617
+ cols << Column.new(header: col, items: gitems,
618
+ type: tab_col.type, tolerant: tab_col.tolerant?)
575
619
  end
576
620
  cols
577
621
  end
@@ -735,7 +779,6 @@ module FatTable
735
779
  last_key = nil
736
780
  new_rows.each_with_index do |nrow, k|
737
781
  new_tab << nrow
738
- # key = nrow.fetch_values(*sort_heads)
739
782
  key = nrow.fetch_values(*key_hash.keys)
740
783
  new_tab.mark_boundary(k - 1) if last_key && key != last_key
741
784
  last_key = key
@@ -941,7 +984,12 @@ module FatTable
941
984
  expr = expr.to_s
942
985
  result = empty_dup
943
986
  headers.each do |h|
944
- col = Column.new(header: h)
987
+ col =
988
+ if tolerant_col?(h)
989
+ Column.new(header: h, tolerant: true)
990
+ else
991
+ Column.new(header: h)
992
+ end
945
993
  result.add_column(col)
946
994
  end
947
995
  ev = Evaluator.new(ivars: { row: 0, group: 0 })
@@ -1406,6 +1454,9 @@ module FatTable
1406
1454
 
1407
1455
  private
1408
1456
 
1457
+ # Collapse a group of rows to a single row by applying the aggregator from
1458
+ # the +agg_cols+ to the items in that column and the presumably identical
1459
+ # value in the +grp_cols to those columns.
1409
1460
  def row_from_group(rows, grp_cols, agg_cols)
1410
1461
  new_row = {}
1411
1462
  grp_cols.each do |h|
@@ -1440,7 +1491,7 @@ module FatTable
1440
1491
  # This column is new, so it needs nil items for all prior rows lest
1441
1492
  # the value be added to a prior row.
1442
1493
  items = Array.new(size, nil)
1443
- columns << Column.new(header: h, items: items)
1494
+ columns << Column.new(header: h, items: items, tolerant: tolerant_col?(h))
1444
1495
  end
1445
1496
  headers.each do |h|
1446
1497
  # NB: This adds a nil if h is not in row.
@@ -1660,11 +1711,19 @@ module FatTable
1660
1711
  end
1661
1712
 
1662
1713
  # The <=> operator cannot handle nils without some help. Treat a nil as
1663
- # smaller than any other value, but equal to other nils. The two keys are assumed to be arrays of values to be
1664
- # compared with <=>.
1714
+ # smaller than any other value, but equal to other nils. The two keys are
1715
+ # assumed to be arrays of values to be compared with <=>. Since
1716
+ # tolerant_columns permit strings to be mixed in with columns of type
1717
+ # Numeric, DateTime, and Boolean, treat strings mixed with another type
1718
+ # the same as nils.
1665
1719
  def compare_with_nils(key1, key2)
1666
1720
  result = nil
1667
1721
  key1.zip(key2) do |k1, k2|
1722
+ if k1.is_a?(String) && !k2.is_a?(String)
1723
+ k1 = nil
1724
+ elsif !k1.is_a?(String) && k2.is_a?(String)
1725
+ k2 = nil
1726
+ end
1668
1727
  if k1.nil? && k2.nil?
1669
1728
  result = 0
1670
1729
  next
@@ -2,5 +2,5 @@
2
2
 
3
3
  module FatTable
4
4
  # The current version of FatTable
5
- VERSION = '0.5.2'
5
+ VERSION = '0.5.5'
6
6
  end
data/lib/fat_table.rb CHANGED
@@ -61,22 +61,22 @@ module FatTable
61
61
 
62
62
  # Return an empty FatTable::Table object. You can use FatTable::Table#add_row
63
63
  # or FatTable::Table#add_column to populate the table with data.
64
- def self.new(*args)
65
- Table.new(*args)
64
+ def self.new(*args, tolerant_columns: [])
65
+ Table.new(*args, tolerant_columns: tolerant_columns)
66
66
  end
67
67
 
68
68
  # Construct a FatTable::Table from the contents of a CSV file given by the
69
69
  # file name +fname+. Headers will be taken from the first row and converted to
70
70
  # symbols.
71
- def self.from_csv_file(fname)
72
- Table.from_csv_file(fname)
71
+ def self.from_csv_file(fname, tolerant_columns: [])
72
+ Table.from_csv_file(fname, tolerant_columns: tolerant_columns)
73
73
  end
74
74
 
75
75
  # Construct a FatTable::Table from the string +str+, treated in the same
76
76
  # manner as if read the input from a CSV file. Headers will be taken from the
77
77
  # first row and converted to symbols.
78
- def self.from_csv_string(str)
79
- Table.from_csv_string(str)
78
+ def self.from_csv_string(str, tolerant_columns: [])
79
+ Table.from_csv_string(str, tolerant_columns: tolerant_columns)
80
80
  end
81
81
 
82
82
  # Construct a FatTable::Table from the first table found in the Emacs org-mode
@@ -84,8 +84,8 @@ module FatTable
84
84
  # is an hline. Otherwise, synthetic headers of the form +:col_1+, +:col_2+,
85
85
  # etc. are created. Any other hlines will be treated as marking a boundary in
86
86
  # the table.
87
- def self.from_org_file(fname)
88
- Table.from_org_file(fname)
87
+ def self.from_org_file(fname, tolerant_columns: [])
88
+ Table.from_org_file(fname, tolerant_columns: tolerant_columns)
89
89
  end
90
90
 
91
91
  # Construct a FatTable::Table from the first table found in the string +str+,
@@ -93,8 +93,8 @@ module FatTable
93
93
  # are taken from the first row if the second row is an hrule. Otherwise,
94
94
  # synthetic headers of the form :col_1, :col_2, etc. are created. Any other
95
95
  # hlines will be treated as marking a boundary in the table.
96
- def self.from_org_string(str)
97
- Table.from_org_string(str)
96
+ def self.from_org_string(str, tolerant_columns: [])
97
+ Table.from_org_string(str, tolerant_columns: tolerant_columns)
98
98
  end
99
99
 
100
100
  # Construct a FatTable::Table from the array of arrays +aoa+. By default, with
@@ -108,8 +108,8 @@ module FatTable
108
108
  # org-mode code blocks, by default (+:hlines no+) all hlines are stripped from
109
109
  # the table, otherwise (+:hlines yes+) they are indicated with nil elements in
110
110
  # the outer array.
111
- def self.from_aoa(aoa, hlines: false)
112
- Table.from_aoa(aoa, hlines: hlines)
111
+ def self.from_aoa(aoa, hlines: false, tolerant_columns: [])
112
+ Table.from_aoa(aoa, hlines: hlines, tolerant_columns: tolerant_columns)
113
113
  end
114
114
 
115
115
  # Construct a FatTable::Table from the array of hashes +aoh+, which can be an
@@ -117,8 +117,8 @@ module FatTable
117
117
  # interpret nil separators as marking boundaries in the new Table. All hashes
118
118
  # must have the same keys, which, converted to symbols, become the headers for
119
119
  # the new Table.
120
- def self.from_aoh(aoh, hlines: false)
121
- Table.from_aoh(aoh, hlines: hlines)
120
+ def self.from_aoh(aoh, hlines: false, tolerant_columns: [])
121
+ Table.from_aoh(aoh, hlines: hlines, tolerant_columns: tolerant_columns)
122
122
  end
123
123
 
124
124
  # Construct a FatTable::Table from another FatTable::Table. Inherit any group
@@ -130,8 +130,8 @@ module FatTable
130
130
  # Construct a Table by running a SQL query against the database set up with
131
131
  # FatTable.connect. Return the Table with the query results as rows and the
132
132
  # headers from the query, converted to symbols, as headers.
133
- def self.from_sql(query)
134
- Table.from_sql(query)
133
+ def self.from_sql(query, tolerant_columns: [])
134
+ Table.from_sql(query, tolerant_columns: tolerant_columns)
135
135
  end
136
136
 
137
137
  ########################################################################
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fat_table
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.5.2
4
+ version: 0.5.5
5
5
  platform: ruby
6
6
  authors:
7
7
  - Daniel E. Doherty
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2022-01-22 00:00:00.000000000 Z
11
+ date: 2022-03-25 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler