fat_table 0.5.2 → 0.5.5

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: f35f99181e39d7569ce7df958ae92742617826e54b8ed2c6352b73c5723a3d44
4
- data.tar.gz: d37acb3f0be27191bafa17ae30b51dfd2db2c4a5d97e6c80fa5399f2618ea98b
3
+ metadata.gz: 2f391d5e4ad9d7a4dcb303098b90f6fb42379d9df382192c4bacaf169523dab1
4
+ data.tar.gz: ea75f906fcd164752a3ca2220324ad10d8438562bb31c4846bc6575723e834fa
5
5
  SHA512:
6
- metadata.gz: eafd2077939c57673358a680490ccbe48218e8b290540790f3526b5945337e53c9366369c255adcc85e01916ad012cbad12132593def15350fd3993304bc5a30
7
- data.tar.gz: e7a542986752ac4f632e7344c49080786e0e52ecbe224a0644a6da7b4ec65dd8ca2b9abe842ee295ad7cf19a7a0ccd0e93b22632716c95533665a21d14b5cc83
6
+ metadata.gz: 2914c40a497cf24bcdba8868c4a4607ea1bdb0a94fba8caa3d6e31407ff6832621df2a0b4a8a2bb98c3b40080fed6d752f8e936f66a96270fe675e93d5a852d6
7
+ data.tar.gz: dbc43f4d2bac4a2a19bc7cc90d1fbfbf739a12f0f280a543670af588cfcf2bb0df74d50cfb7ded76b851f5470fd30d4f971659cead5131f9ea15e24b492e5823
data/README.org CHANGED
@@ -25,6 +25,16 @@ The following is for org.
25
25
 
26
26
  [[https://travis-ci.org/ddoherty03/fat_table.svg?branch=master]]
27
27
 
28
+ * Version
29
+ #+begin_src ruby :wrap EXAMPLE
30
+ require 'fat_table'
31
+ "Current version is: #{FatTable::VERSION}"
32
+ #+end_src
33
+
34
+ #+begin_EXAMPLE
35
+ Current version is: 0.5.4
36
+ #+end_EXAMPLE
37
+
28
38
  * Introduction
29
39
 
30
40
  ~FatTable~ is a gem that treats tables as a data type. It provides methods for
@@ -57,6 +67,7 @@ array of arrays with its ~.to_aoa~ output function will be rendered in an
57
67
  org-mode buffer as an org-table, ready for processing by other code blocks.
58
68
 
59
69
  * Table of Contents :toc:noexport:
70
+ - [[#version][Version]]
60
71
  - [[#introduction][Introduction]]
61
72
  - [[#installation][Installation]]
62
73
  - [[#using-in-a-gem][Using in a gem]]
@@ -74,6 +85,7 @@ org-mode buffer as an org-table, ready for processing by other code blocks.
74
85
  - [[#without-headers][Without Headers]]
75
86
  - [[#with-headers][With Headers]]
76
87
  - [[#forcing-string-type][Forcing String Type]]
88
+ - [[#designating-tolerant-columns][Designating "Tolerant" Columns]]
77
89
  - [[#from-csv-or-org-mode-files-or-strings][From CSV or Org Mode files or strings]]
78
90
  - [[#from-arrays-of-arrays][From Arrays of Arrays]]
79
91
  - [[#in-ruby-code][In Ruby Code]]
@@ -430,11 +442,26 @@ or nil. There are only five permissible types for a ~Column~:
430
442
  5. *NilClass* (for the undetermined column type).
431
443
 
432
444
  When a ~Table~ is constructed from an external source, all ~Columns~ start out
433
- having a type of ~NilClass~, that is, their type is as yet undetermined. When a
434
- string or object of one of the four determined types is added to a ~Column~, it
435
- fixes the type of the column and all further items added to the ~Column~ must
436
- either be ~nil~ (indicating no value) or be capable of being coerced to the
437
- column's type. Otherwise, ~FatTable~ raises an exception.
445
+ having a type of ~NilClass~, that is, their type is as yet undetermined. When
446
+ a string or object of one of the four determined types is added to a ~Column~
447
+ and it can be converted into one of the permissible types, it fixes the type
448
+ of the column, and all further items added to the ~Column~ must either be
449
+ ~nil~ (indicating no value) or be capable of being coerced to the column's
450
+ type. Otherwise, ~FatTable~ raises an exception.
451
+
452
+ The strictness of requiring all items to be of the same type can be relaxed by
453
+ declaring a column to be "tolerant." You can do so when you create the table
454
+ by adding a tolerant_columns keyword parameter. If a Column is tolerant,
455
+ ~FatTable~ tries to convert new items into a type other than a ~String~ and,
456
+ if it can do so, sets /that/ as the Column's type. Any later items that
457
+ cannot be converted into the Column's type are converted to strings. These
458
+ interloper strings are treated like nils for purposes of sorting and
459
+ evaluation, but are displayed according to any string formatting on output.
460
+ See [[*Designating "Tolerant" Columns][Designating "Tolerant" Columns]] below.
461
+
462
+ It is also possible to force ~FatTable~ to treat a column as a String type,
463
+ even its items look like one of the other types. See [[*Forcing String Type][Forcing String Type]]
464
+ below.
438
465
 
439
466
  Items of input must be either one of the permissible ruby objects or strings. If
440
467
  they are strings, ~FatTable~ attempts to parse them as one of the permissible
@@ -591,10 +618,18 @@ columns to be created:
591
618
  **** Forcing String Type
592
619
  Occasionally, ~FatTable~'s automatic type detection can get in the way and you
593
620
  just want it to treat one or more columns as Strings regardless of their
594
- appearance. Think, for example, of zip codes. At any time after creating a
595
- table, you can have it force the String type on any number of columns with the
596
- ~force_string!~ method. When you do so, all exisiting items in the column are
597
- converted to strings with the #to_s method.
621
+ appearance. Think, for example, of zip codes. If headers are given when a
622
+ table is contructed, you can designate a forced-string column by appending a
623
+ ~!~ to the end of the header. It will not become part of the header, it will
624
+ just mark it as a forced-string Column.
625
+
626
+ #+begin_SRC emacs-lisp :wrap EXAMPLE
627
+ tab = FatTable.new(:a, 'b', 'C!', :d, :zip!)
628
+ #+end_SRC
629
+
630
+ In addition, at any time after creating a table, you can force the String type
631
+ on any number of columns with the ~force_string!~ method. When you do so, all
632
+ exisiting items in the column are converted to strings with the #to_s method.
598
633
 
599
634
  #+begin_src ruby :wrap EXAMPLE
600
635
  tab = FatTable.new(:a, 'b', 'C', :d, :zip)
@@ -619,8 +654,83 @@ converted to strings with the #to_s method.
619
654
  +======+======+============+===+=======+===+
620
655
  #+end_EXAMPLE
621
656
 
657
+ **** Designating "Tolerant" Columns
658
+ Related to the problem just discussed is the problem of reading files in from
659
+ the wild where a column may get typed as, say Numeric, but then contain
660
+ something that can't be parsed as a Numeric. ~FatTable~ raises an exception
661
+ is such cases, and that may be what you want if you can control the input.
662
+ But, especially when you cannot do so, it can be helpful to designate one or
663
+ more columns as "tolerant." This means that when a conversion problem occurs,
664
+ the column item is retained as a string type in a column that is otherwise of
665
+ one of the types Numeric, DateTime, or Boolean. Those string items are
666
+ treated as nils for purposes of sorting or evaluation in a ~select~ method.
667
+ When formatted, they participate in string formatting directive, but not those
668
+ for other types.
669
+
670
+ All of the table construction methods, allow a keyword parameter,
671
+ ~tolerant_columns~, where you can designate what columns should be convert to
672
+ String type when conversion to the auto-typed column type is not possible.
673
+ The parameter should be an array of headers, in either string or symbol form,
674
+ for which this behavior is desired. In addition, it can be set to the special
675
+ string '*' or symbol ~:*~ to indicate that all the columns should be made
676
+ tolerant.
677
+
678
+ #+begin_src ruby :wrap EXAMPLE
679
+ require 'fat_table'
680
+ tab = FatTable.new(:a, 'b', 'C', :d, :zip, tolerant_columns: [:zip])
681
+ tab << { a: 1, b: 2, c: "<2017-01-21>", d: 'f', e: '', zip: 18552 }
682
+ tab << { a: 3.14, b: 2.17, c: '[2016-01-21 Thu]', d: 'Y', e: nil }
683
+ tab << { zip: '01879--7884' }
684
+ tab << { zip: '66210' }
685
+ tab << { zip: '90210' }
686
+ tab.to_text
687
+ #+end_src
688
+
689
+ #+RESULTS:
690
+ #+begin_EXAMPLE
691
+ +======+======+============+===+=============+===+
692
+ | A | B | C | D | Zip | E |
693
+ +------+------+------------+---+-------------+---+
694
+ | 1 | 2 | 2017-01-21 | F | 18552 | |
695
+ | 3.14 | 2.17 | 2016-01-21 | T | | |
696
+ | | | | | 01879--7884 | |
697
+ | | | | | 66210 | |
698
+ | | | | | 90210 | |
699
+ +======+======+============+===+=============+===+
700
+ #+end_EXAMPLE
701
+
702
+ Another way to designate a column as tolerant is to end a column you want to
703
+ designate as tolerant with a ~!~. The ~!~ will be stripped from the header,
704
+ but it will be marked as tolerant.
705
+ #+begin_src ruby :wrap EXAMPLE
706
+ require 'fat_table'
707
+ tab = FatTable.new(:a, 'b!', 'C', :d, :zip!)
708
+ tab << { a: 1, b: 2, c: "<2017-01-21>", d: 'f', e: '', zip: 18552 }
709
+ tab << { a: 3.14, b: 2.17, c: '[2016-01-21 Thu]', d: 'Y', e: nil }
710
+ tab << { zip: '01879--7884' }
711
+ tab << { zip: '66210', b: 'Not a Number' }
712
+ tab << { zip: '90210' }
713
+ tab.to_text
714
+ #+end_src
715
+
716
+ #+RESULTS:
717
+ #+begin_EXAMPLE
718
+ +======+==============+============+===+=============+===+
719
+ | A | B | C | D | Zip | E |
720
+ +------+--------------+------------+---+-------------+---+
721
+ | 1 | 2 | 2017-01-21 | F | 18552 | |
722
+ | 3.14 | 2.17 | 2016-01-21 | T | | |
723
+ | | | | | 01879--7884 | |
724
+ | | Not a Number | | | 66210 | |
725
+ | | | | | 90210 | |
726
+ +======+==============+============+===+=============+===+
727
+ #+end_EXAMPLE
728
+
622
729
  *** From CSV or Org Mode files or strings
623
- Tables can also be read from ~.csv~ files or files containing ~org-mode~ tables.
730
+ Tables can also be read from ~.csv~ files or files containing ~org-mode~
731
+ tables. Remember that you can make any column tolerant with a
732
+ ~tolerant_columns:~ keyword argument or make them all tolerant by designating
733
+ the pseudo-column ~:*~ as tolerant.
624
734
 
625
735
  In the case of org-mode files, ~FatTable~ skips through the file until it finds
626
736
  a line that look like a table, that is, it begins with any number of spaces
@@ -677,8 +787,10 @@ header row, and the headers are converted to symbols as described above.
677
787
 
678
788
  *** From Arrays of Arrays
679
789
  **** In Ruby Code
680
- You can also initialize a table directly from ruby data structures. You can, for
681
- example, build a table from an array of arrays:
790
+ You can also initialize a table directly from ruby data structures. You can,
791
+ for example, build a table from an array of arrays. Remember that you can
792
+ make any column tolerant with a ~tolerant_columns:~ keyword argument or make
793
+ them all tolerant by designating the pseudo-column ~:*~ as tolerant.
682
794
 
683
795
  #+BEGIN_SRC ruby
684
796
  aoa = [
@@ -772,8 +884,10 @@ This example illustrates several things:
772
884
 
773
885
  A second ruby data structure that can be used to initialize a ~FatTable~ table
774
886
  is an array of ruby Hashes. Each hash represents a row of the table, and the
775
- headers of the table are taken from the keys of the hashes. Accordingly, all the
776
- hashes must have the same keys.
887
+ headers of the table are taken from the keys of the hashes. Accordingly, all
888
+ the hashes must have the same keys. Remember that you can make any column
889
+ tolerant with a ~tolerant_columns:~ keyword argument or make them all tolerant
890
+ by designating the pseudo-column ~:*~ as tolerant.
777
891
 
778
892
  This same method can in fact take an array of any objects that can be converted
779
893
  to a Hash with the ~#to_h~ method, so you can use an array of your own objects
@@ -862,6 +976,10 @@ The ~.connect~ function need only be called once, and the database handle it
862
976
  creates will be used for all subsequent ~.from_sql~ calls until ~.connect~ is
863
977
  called again.
864
978
 
979
+ Remember that you can make any column tolerant with a ~tolerant_columns:~
980
+ keyword argument or make them all tolerant by designating the pseudo-column
981
+ ~:*~ as tolerant.
982
+
865
983
  *** Marking Groups in Input
866
984
  **** Manually
867
985
  At any point, you can add a boundary to a table by invokong the
@@ -2549,9 +2667,10 @@ the table, but they can be overridden by more specific directives given in a
2549
2667
  ~format_for~ directive.
2550
2668
 
2551
2669
  **** Type and Column priority
2552
- A directive based on type applies to all columns having that type unless
2553
- overridden by a directive specific to a named column; a directive based on a
2554
- column name applies only to cells in that column.
2670
+ A directive based the column name overrides any directive based on type. If
2671
+ any cell has both a type-based formatting and column-based, the column
2672
+ instructions prevail. In earlier versions the instuctions were "merged" but
2673
+ that is no longer the case.
2555
2674
 
2556
2675
  However, there is a twist. Since the end result of formatting is to convert
2557
2676
  all columns to strings, the formatting directives for the ~String~ type can
data/TODO.org CHANGED
@@ -1,10 +1,37 @@
1
+
2
+ * TODO Specify Column Widths
3
+ Allow a formatter to specify column widths. This could be a number of
4
+ characters, which would be interpreted as a number of "ems" for LaTeX.
5
+ Cell content larger than the width would be truncated. Any column without a
6
+ width specified would be set at the width of the longest value in that cell,
7
+ after initial formatting.
8
+
9
+ #+begin_SRC ruby
10
+ tab.to_text do |f|
11
+ f.widths(a: 13, b: 30)
12
+ end
13
+ #+end_SRC
14
+
15
+ Possible enhancements:
16
+ - specify an overall width and column widths as decimal or fractions, so that
17
+ a column's width would be that fraction of the overall width.
18
+ - specify a Range for a width, so that the column would at least min and at
19
+ most max, otherwise the width of its largest cell.
20
+
1
21
  * TODO Conversion to Spreadsheets
2
22
  - State "TODO" from [2017-04-21 Fri 10:36]
3
23
  This is a [[https://github.com/westonganger/spreadsheet_architect][gem]] that I can include into the Table model to convert a table into
4
24
  a spread-sheet, or even a sheet in a multi-sheet spreadsheet file.
5
25
 
6
- * TODO Add from_yql for fetching from Yahoo
26
+ * TODO Add Quandl or EODDATA Queries
27
+ Possible replacements for YQL.
28
+
29
+ * CNCL Add from_yql for fetching from Yahoo
30
+ CLOSED: [2022-01-30 Sun 06:03]
7
31
  - State "TODO" from [2017-04-21 Fri 10:35]
32
+
33
+ Cancelled because Yahoo shut down the YQL api service.
34
+
8
35
  Add a constructor to allow fetching stock data from yql. Perhaps grab all
9
36
  available fields, then allow a select of those of interest.
10
37
 
data/lib/ext/array.rb CHANGED
@@ -12,4 +12,21 @@ class Array
12
12
  end
13
13
  end
14
14
  end
15
+
16
+ def filter_to_type(typ)
17
+ if typ == 'Boolean'
18
+ compact.select { |i| i.is_a?(TrueClass) || i.is_a?(FalseClass) }
19
+ elsif typ == 'DateTime'
20
+ compact.select { |i| i.is_a?(Date) || i.is_a?(DateTime) || i.is_a?(Time) }
21
+ .map { |i| i.to_datetime }
22
+ elsif typ == 'Numeric'
23
+ compact.select { |i| i.is_a?(Numeric) }
24
+ elsif typ == 'String'
25
+ map { |i| i.to_s }
26
+ elsif typ == 'NilClass'
27
+ self
28
+ else
29
+ raise ArgumentError, "cannot filter_to_type for type '#{typ}'"
30
+ end
31
+ end
15
32
  end
@@ -83,7 +83,7 @@ module FatTable
83
83
  # col.type #=> 'Numeric'
84
84
  # col.header #=> :prices
85
85
  # col.sum #=> 18376.75
86
- def initialize(header:, items: [], type: 'NilClass')
86
+ def initialize(header:, items: [], type: 'NilClass', tolerant: false)
87
87
  @raw_header = header
88
88
  @header =
89
89
  if @raw_header.is_a?(Symbol)
@@ -92,6 +92,7 @@ module FatTable
92
92
  @raw_header.to_s.as_sym
93
93
  end
94
94
  @type = type
95
+ @tolerant = tolerant
95
96
  msg = "unknown column type '#{type}"
96
97
  raise UserError, msg unless TYPES.include?(@type.to_s)
97
98
 
@@ -141,6 +142,14 @@ module FatTable
141
142
 
142
143
  # :category: Attributes
143
144
 
145
+ # Is this column tolerant of type incompatibilities? If so, the Column
146
+ # type will be forced to String if an incompatible type is found.
147
+ def tolerant?
148
+ @tolerant
149
+ end
150
+
151
+ # :category: Attributes
152
+
144
153
  # Force the column to have String type and then convert all items to
145
154
  # strings.
146
155
  def force_string!
@@ -182,12 +191,15 @@ module FatTable
182
191
 
183
192
  # :category: Aggregates
184
193
 
185
- # Return the first non-nil item in the Column. Works with any Column type.
194
+ # Return the first non-nil item in the Column, or nil if all items are
195
+ # nil. Works with any Column type.
186
196
  def first
197
+ return nil if items.all?(&:nil?)
198
+
187
199
  if type == 'String'
188
200
  items.reject(&:blank?).first
189
201
  else
190
- items.compact.first
202
+ items.filter_to_type(type).first
191
203
  end
192
204
  end
193
205
 
@@ -195,83 +207,94 @@ module FatTable
195
207
 
196
208
  # Return the last non-nil item in the Column. Works with any Column type.
197
209
  def last
210
+ return nil if items.all?(&:nil?)
211
+
198
212
  if type == 'String'
199
213
  items.reject(&:blank?).last
200
214
  else
201
- items.compact.last
215
+ items.filter_to_type(type).last
202
216
  end
203
217
  end
204
218
 
205
219
  # :category: Aggregates
206
220
 
207
- # Return a count of the non-nil items in the Column. Works with any Column
208
- # type.
221
+ # Return a count of the non-nil items in the Column, or the size of the
222
+ # column if all items are nil. Works with any Column type.
209
223
  def count
224
+ return items.size if items.all?(&:nil?)
225
+
210
226
  if type == 'String'
211
227
  items.reject(&:blank?).count.to_d
212
228
  else
213
- items.compact.count.to_d
229
+ items.filter_to_type(type).count.to_d
214
230
  end
215
231
  end
216
232
 
217
233
  # :category: Aggregates
218
234
 
219
- # Return the smallest non-nil, non-blank item in the Column. Works with
220
- # numeric, string, and datetime Columns.
235
+ # Return the smallest non-nil, non-blank item in the Column, or nil if all
236
+ # items are nil. Works with numeric, string, and datetime Columns.
221
237
  def min
222
238
  only_with('min', 'NilClass', 'Numeric', 'String', 'DateTime')
223
239
  if type == 'String'
224
240
  items.reject(&:blank?).min
225
241
  else
226
- items.compact.min
242
+ items.filter_to_type(type).min
227
243
  end
228
244
  end
229
245
 
230
246
  # :category: Aggregates
231
247
 
232
- # Return the largest non-nil, non-blank item in the Column. Works with
233
- # numeric, string, and datetime Columns.
248
+ # Return the largest non-nil, non-blank item in the Column, or nil if all
249
+ # items are nil. Works with numeric, string, and datetime Columns.
234
250
  def max
235
251
  only_with('max', 'NilClass', 'Numeric', 'String', 'DateTime')
236
252
  if type == 'String'
237
253
  items.reject(&:blank?).max
238
254
  else
239
- items.compact.max
255
+ items.filter_to_type(type).max
240
256
  end
241
257
  end
242
258
 
243
259
  # :category: Aggregates
244
260
 
245
- # Return a Range object for the smallest to largest value in the column.
246
- # Works with numeric, string, and datetime Columns.
261
+ # Return a Range object for the smallest to largest value in the column,
262
+ # or nil if all items are nil. Works with numeric, string, and datetime
263
+ # Columns.
247
264
  def range
248
265
  only_with('range', 'NilClass', 'Numeric', 'String', 'DateTime')
266
+ return nil if items.all?(&:nil?)
267
+
249
268
  Range.new(min, max)
250
269
  end
251
270
 
252
271
  # :category: Aggregates
253
272
 
254
- # Return the sum of the non-nil items in the Column. Works with numeric and
255
- # string Columns. For a string Column, it will return the concatenation of
256
- # the non-nil items.
273
+ # Return the sum of the non-nil items in the Column, or 0 if all items are
274
+ # nil. Works with numeric and string Columns. For a string Column, it
275
+ # will return the concatenation of the non-nil items.
257
276
  def sum
277
+ return 0 if type == 'NilClass' || items.all?(&:nil?)
278
+
258
279
  only_with('sum', 'Numeric', 'String')
259
280
  if type == 'String'
260
281
  items.reject(&:blank?).join(' ')
261
282
  else
262
- items.compact.sum
283
+ items.filter_to_type(type).sum
263
284
  end
264
285
  end
265
286
 
266
287
  # :category: Aggregates
267
288
 
268
- # Return the average value of the non-nil items in the Column. Works with
269
- # numeric and datetime Columns. For datetime Columns, it converts each date
270
- # to its Julian day number, computes the average, and then converts the
271
- # average back to a DateTime.
289
+ # Return the average value of the non-nil items in the Column, or 0 if all
290
+ # items are nil. Works with numeric and datetime Columns. For datetime
291
+ # Columns, it converts each date to its Julian day number, computes the
292
+ # average, and then converts the average back to a DateTime.
272
293
  def avg
294
+ return 0 if type == 'NilClass' || items.all?(&:nil?)
295
+
273
296
  only_with('avg', 'DateTime', 'Numeric')
274
- itms = items.compact
297
+ itms = items.filter_to_type(type)
275
298
  size = itms.size.to_d
276
299
  if type == 'DateTime'
277
300
  avg_jd = itms.map(&:jd).sum / size
@@ -284,17 +307,20 @@ module FatTable
284
307
  # :category: Aggregates
285
308
 
286
309
  # Return the sample variance (the unbiased estimator of the population
287
- # variance using a divisor of N-1) as the average squared deviation from the
288
- # mean, of the non-nil items in the Column. Works with numeric and datetime
289
- # Columns. For datetime Columns, it converts each date to its Julian day
290
- # number and computes the variance of those numbers.
310
+ # variance using a divisor of N-1) as the average squared deviation from
311
+ # the mean, of the non-nil items in the Column, or 0 if all items are
312
+ # nil. Works with numeric and datetime Columns. For datetime Columns, it
313
+ # converts each date to its Julian day number and computes the variance of
314
+ # those numbers.
291
315
  def var
316
+ return 0 if type == 'NilClass' || items.all?(&:nil?)
317
+
292
318
  only_with('var', 'DateTime', 'Numeric')
293
319
  all_items =
294
320
  if type == 'DateTime'
295
- items.compact.map(&:jd)
321
+ items.filter_to_type(type).map(&:jd)
296
322
  else
297
- items.compact
323
+ items.filter_to_type(type)
298
324
  end
299
325
  n = count
300
326
  return BigDecimal('0.0') if n <= 1
@@ -310,12 +336,15 @@ module FatTable
310
336
 
311
337
  # Return the population variance (the biased estimator of the population
312
338
  # variance using a divisor of N) as the average squared deviation from the
313
- # mean, of the non-nil items in the Column. Works with numeric and datetime
314
- # Columns. For datetime Columns, it converts each date to its Julian day
315
- # number and computes the variance of those numbers.
339
+ # mean, of the non-nil items in the Column, or 0 if all items are
340
+ # nil. Works with numeric and datetime Columns. For datetime Columns, it
341
+ # converts each date to its Julian day number and computes the variance of
342
+ # those numbers.
316
343
  def pvar
344
+ return 0 if type == 'NilClass' || items.all?(&:nil?)
345
+
317
346
  only_with('var', 'DateTime', 'Numeric')
318
- n = items.compact.size.to_d
347
+ n = items.filter_to_type(type).size.to_d
319
348
  return BigDecimal('0.0') if n <= 1
320
349
  var * ((n - 1) / n)
321
350
  end
@@ -324,11 +353,13 @@ module FatTable
324
353
 
325
354
  # Return the sample standard deviation (the unbiased estimator of the
326
355
  # population standard deviation using a divisor of N-1) as the square root
327
- # of the sample variance, of the non-nil items in the Column. Works with
328
- # numeric and datetime Columns. For datetime Columns, it converts each date
329
- # to its Julian day number and computes the standard deviation of those
330
- # numbers.
356
+ # of the sample variance, of the non-nil items in the Column, or 0 if all
357
+ # items are nil. Works with numeric and datetime Columns. For datetime
358
+ # Columns, it converts each date to its Julian day number and computes the
359
+ # standard deviation of those numbers.
331
360
  def dev
361
+ return 0 if type == 'NilClass' || items.all?(&:nil?)
362
+
332
363
  only_with('dev', 'DateTime', 'Numeric')
333
364
  var.sqrt(20)
334
365
  end
@@ -336,12 +367,14 @@ module FatTable
336
367
  # :category: Aggregates
337
368
 
338
369
  # Return the population standard deviation (the biased estimator of the
339
- # population standard deviation using a divisor of N) as the square root of
340
- # the population variance, of the non-nil items in the Column. Works with
341
- # numeric and datetime Columns. For datetime Columns, it converts each date
342
- # to its Julian day number and computes the standard deviation of those
343
- # numbers.
370
+ # population standard deviation using a divisor of N) as the square root
371
+ # of the population variance, of the non-nil items in the Column, or 0 if
372
+ # all items are nil. Works with numeric and datetime Columns. For datetime
373
+ # Columns, it converts each date to its Julian day number and computes the
374
+ # standard deviation of those numbers.
344
375
  def pdev
376
+ return 0 if type == 'NilClass' || items.all?(&:nil?)
377
+
345
378
  only_with('dev', 'DateTime', 'Numeric')
346
379
  Math.sqrt(pvar)
347
380
  end
@@ -349,28 +382,35 @@ module FatTable
349
382
  # :category: Aggregates
350
383
 
351
384
  # Return true if any of the items in the Column are true; otherwise return
352
- # false. Works only with boolean Columns.
385
+ # false, or false if all items are nil. Works only with boolean Columns.
353
386
  def any?
387
+ return false if type == 'NilClass' || items.all?(&:nil?)
388
+
354
389
  only_with('any?', 'Boolean')
355
- items.compact.any?
390
+ items.filter_to_type(type).any?
356
391
  end
357
392
 
358
393
  # :category: Aggregates
359
394
 
360
395
  # Return true if all of the items in the Column are true; otherwise return
361
- # false. Works only with boolean Columns.
396
+ # false, or false if all items are nil. Works only with boolean Columns.
362
397
  def all?
398
+ return false if type == 'NilClass' || items.all?(&:nil?)
399
+
363
400
  only_with('all?', 'Boolean')
364
- items.compact.all?
401
+ items.filter_to_type(type).all?
365
402
  end
366
403
 
367
404
  # :category: Aggregates
368
405
 
369
- # Return true if none of the items in the Column are true; otherwise return
370
- # false. Works only with boolean Columns.
406
+ # Return true if none of the items in the Column are true; otherwise
407
+ # return false, or true if all items are nil. Works only with boolean
408
+ # Columns.
371
409
  def none?
410
+ return true if type == 'NilClass' || items.all?(&:nil?)
411
+
372
412
  only_with('none?', 'Boolean')
373
- items.compact.none?
413
+ items.filter_to_type(type).none?
374
414
  end
375
415
 
376
416
  # :category: Aggregates
@@ -378,14 +418,17 @@ module FatTable
378
418
  # Return true if precisely one of the items in the Column is true;
379
419
  # otherwise return false. Works only with boolean Columns.
380
420
  def one?
421
+ return false if type == 'NilClass' || items.all?(&:nil?)
422
+
381
423
  only_with('one?', 'Boolean')
382
- items.compact.one?
424
+ items.filter_to_type(type).one?
383
425
  end
384
426
 
385
427
  private
386
428
 
387
429
  def only_with(agg, *valid_types)
388
430
  return self if valid_types.include?(type)
431
+
389
432
  msg = "aggregate '#{agg}' cannot be applied to a #{type} column"
390
433
  raise UserError, msg
391
434
  end
@@ -400,9 +443,17 @@ module FatTable
400
443
 
401
444
  # Append +itm+ to end of the Column after converting it to the Column's
402
445
  # type. If the Column's type is still open, i.e. NilClass, attempt to fix
403
- # the Column's type based on the type of +itm+ as with Column.new.
446
+ # the Column's type based on the type of +itm+ as with Column.new. If its
447
+ # a tolerant column, respond to type errors by converting the column to a
448
+ # String type.
404
449
  def <<(itm)
405
- items << convert_to_type(itm)
450
+ items << convert_and_set_type(itm)
451
+ rescue IncompatibleTypeError => ex
452
+ if tolerant?
453
+ items << Convert.convert_to_string(itm)
454
+ else
455
+ raise ex
456
+ end
406
457
  end
407
458
 
408
459
  # :category: Constructors
@@ -418,9 +469,14 @@ module FatTable
418
469
 
419
470
  private
420
471
 
421
- def convert_to_type(val)
422
- new_val = Convert.convert_to_type(val, type)
423
- if new_val && type == 'NilClass'
472
+ def convert_and_set_type(val)
473
+ begin
474
+ new_val = Convert.convert_to_type(val, type, tolerant: tolerant?)
475
+ rescue IncompatibleTypeError
476
+ err_msg = "attempt to add '#{val}' to column '#{header}' already typed as #{type}"
477
+ raise IncompatibleTypeError, err_msg
478
+ end
479
+ if new_val && (type == 'NilClass' || type == 'String')
424
480
  @type =
425
481
  if [true, false].include?(new_val)
426
482
  'Boolean'
@@ -10,7 +10,7 @@ module FatTable
10
10
  # determined, raise an error if the val cannot be converted to the Column
11
11
  # type. Otherwise, returns the converted val as an object of the correct
12
12
  # class.
13
- def self.convert_to_type(val, type)
13
+ def self.convert_to_type(val, type, tolerant: false)
14
14
  case type
15
15
  when 'NilClass'
16
16
  if val != false && val.blank?
@@ -36,8 +36,7 @@ module FatTable
36
36
  else
37
37
  new_val = convert_to_boolean(val)
38
38
  if new_val.nil?
39
- msg = "attempt to add '#{val}' to a column already typed as #{type}"
40
- raise UserError, msg
39
+ raise IncompatibleTypeError
41
40
  end
42
41
  new_val
43
42
  end
@@ -47,8 +46,7 @@ module FatTable
47
46
  else
48
47
  new_val = convert_to_date_time(val)
49
48
  if new_val.nil?
50
- msg = "attempt to add '#{val}' to a column already typed as #{type}"
51
- raise UserError, msg
49
+ raise IncompatibleTypeError
52
50
  end
53
51
  new_val
54
52
  end
@@ -58,24 +56,35 @@ module FatTable
58
56
  else
59
57
  new_val = convert_to_numeric(val)
60
58
  if new_val.nil?
61
- msg = "attempt to add '#{val}' to a column already typed as #{type}"
62
- raise UserError, msg
59
+ raise IncompatibleTypeError
63
60
  end
64
61
  new_val
65
62
  end
66
63
  when 'String'
67
64
  if val.nil?
68
65
  nil
66
+ elsif tolerant
67
+ # Allow String to upgrade to one of Numeric, DateTime, or Boolean if
68
+ # possible.
69
+ if (new_val = convert_to_numeric(val))
70
+ new_val
71
+ elsif (new_val = convert_to_date_time(val))
72
+ new_val
73
+ elsif (new_val = convert_to_boolean(val))
74
+ new_val
75
+ else
76
+ new_val = convert_to_string(val)
77
+ end
78
+ new_val
69
79
  else
70
80
  new_val = convert_to_string(val)
71
81
  if new_val.nil?
72
- msg = "attempt to add '#{val}' to a column already typed as #{type}"
73
- raise UserError, msg
82
+ raise IncompatibleTypeError
74
83
  end
75
84
  new_val
76
85
  end
77
86
  else
78
- raise UserError, "Mysteriously, column has unknown type '#{type}'"
87
+ raise LogicError, "Mysteriously, column has unknown type '#{type}'"
79
88
  end
80
89
  end
81
90
 
@@ -121,6 +130,7 @@ module FatTable
121
130
  return val if val.is_a?(DateTime)
122
131
  return val if val.is_a?(Date)
123
132
  return val.to_datetime if val.is_a?(Time)
133
+
124
134
  begin
125
135
  str = val.to_s.clean
126
136
  return nil if str.blank?
@@ -9,6 +9,10 @@ module FatTable
9
9
  # cannot correct.
10
10
  class LogicError < StandardError; end
11
11
 
12
+ # Raised when attempting to add an incompatible type to an already-typed
13
+ # Column.
14
+ class IncompatibleTypeError < UserError; end
15
+
12
16
  # Raised when an external resource is not available due to caller or
13
17
  # programmer error or some failure of the external resource to be available.
14
18
  class TransientError < StandardError; end
@@ -45,17 +45,15 @@ module FatTable
45
45
  end
46
46
 
47
47
  # Return the result of evaluating +expr+ as a Ruby expression in which the
48
- # instance variables set in Evaluator.new and any local variables set in the
49
- # Hash parameter +locals+ are available to the expression.
48
+ # instance variables set in Evaluator.new and any local variables set in
49
+ # the Hash parameter +locals+ are available to the expression. Certain
50
+ # errors simply return nil as the result. This can happen, for example,
51
+ # when a string gets into an otherwise numeric column because the column
52
+ # is set to tolerant.
50
53
  def evaluate(expr = '', locals: {})
51
54
  eval(expr, local_vars(binding, locals))
52
55
  rescue NoMethodError, TypeError => ex
53
- if ex.to_s =~ /for nil:NilClass|nil can't be coerced/
54
- # Likely one of the locals was nil, so let nil be the result.
55
- return nil
56
- else
57
- raise ex
58
- end
56
+ nil
59
57
  end
60
58
 
61
59
  private
@@ -166,7 +166,7 @@ module FatTable
166
166
  when String
167
167
  begin
168
168
  converted_val = Convert.convert_to_type(agg, column.type)
169
- rescue UserError
169
+ rescue UserError, IncompatibleTypeError
170
170
  converted_val = false
171
171
  end
172
172
  if converted_val
@@ -532,7 +532,7 @@ module FatTable
532
532
  valid_keys = table.headers + %i[string numeric datetime boolean nil]
533
533
  invalid_keys = (fmts.keys - valid_keys).uniq
534
534
  unless invalid_keys.empty?
535
- msg = "invalid #{location} column or type: #{invalid_keys.join(',')}"
535
+ msg = "invalid #{location} column or type: #{invalid_keys.join(', ')}"
536
536
  raise UserError, msg
537
537
  end
538
538
 
@@ -562,20 +562,21 @@ module FatTable
562
562
  end
563
563
  end
564
564
 
565
- # Merge in formatting for column h based on the column type, or based
566
- # on the string type for the header location.
565
+ # Merge in formatting instructions for column h based on the column
566
+ # name, or if there is no formatting instructions for the column by
567
+ # name, merge in the formatting instructions based on the column's
568
+ # type. Insist on only the string type for the header location.
567
569
  typ = (location == :header ? :string : table.type(h).as_sym)
568
570
  parse_typ_method_name = 'parse_' + typ.to_s + '_fmt'
569
- if fmts.key?(typ)
570
- # Merge in type-based formatting
571
- typ_fmt = send(parse_typ_method_name, fmts[typ]).first
572
- format_h = format_h.merge(typ_fmt)
573
- end
574
571
  if fmts[h]
575
572
  # Merge in column formatting
576
573
  col_fmt = send(parse_typ_method_name, fmts[h],
577
574
  strict: location != :header).first
578
575
  format_h = format_h.merge(col_fmt)
576
+ elsif fmts.key?(typ)
577
+ # Merge in type-based formatting
578
+ typ_fmt = send(parse_typ_method_name, fmts[typ]).first
579
+ format_h = format_h.merge(typ_fmt)
579
580
  end
580
581
 
581
582
  # Copy :body formatting for column h to :bfirst and :gfirst if they
@@ -1003,9 +1004,14 @@ module FatTable
1003
1004
  # converted to strings formatted according to the Formatter's formatting
1004
1005
  # directives given in Formatter.format_for or Formatter.format.
1005
1006
  def output
1006
- # This results in a hash of two-element arrays. The key is the header and
1007
- # the value is an array of the header and formatted header. We do the
1008
- # latter so the structure parallels the structure for rows explained next.
1007
+ # If there are neither headers nor any rows in the table, return an
1008
+ # empty string.
1009
+ return '' if table.empty? && table.headers.empty?
1010
+
1011
+ # This results in a hash of two-element arrays. The key
1012
+ # is the header and the value is an array of the header and formatted
1013
+ # header. We do the latter so the structure parallels the structure for
1014
+ # rows explained next.
1009
1015
  formatted_headers = build_formatted_headers
1010
1016
 
1011
1017
  # These produce an array with each element representing a row of the
@@ -20,7 +20,7 @@ module FatTable
20
20
  end
21
21
 
22
22
  def pre_header(widths)
23
- result = '|'
23
+ result = +'|'
24
24
  widths.each_value do |w|
25
25
  result += '-' * (w + 2) + '+'
26
26
  end
@@ -53,7 +53,7 @@ module FatTable
53
53
  end
54
54
 
55
55
  def hline(widths)
56
- result = '|'
56
+ result = +'|'
57
57
  widths.each_value do |w|
58
58
  result += '-' * (w + 2) + '+'
59
59
  end
@@ -62,7 +62,7 @@ module FatTable
62
62
  end
63
63
 
64
64
  def post_footers(widths)
65
- result = '|'
65
+ result = +'|'
66
66
  widths.each_value do |w|
67
67
  result += '-' * (w + 2) + '+'
68
68
  end
@@ -221,7 +221,7 @@ module FatTable
221
221
  end
222
222
 
223
223
  def pre_header(widths)
224
- result = upper_left
224
+ result = +upper_left
225
225
  widths.each_value do |w|
226
226
  result += double_rule * (w + 2) + upper_tee
227
227
  end
@@ -255,7 +255,7 @@ module FatTable
255
255
  end
256
256
 
257
257
  def hline(widths)
258
- result = left_tee
258
+ result = +left_tee
259
259
  widths.each_value do |w|
260
260
  result += horizontal_rule * (w + 2) + single_cross
261
261
  end
@@ -289,7 +289,7 @@ module FatTable
289
289
  end
290
290
 
291
291
  def post_footers(widths)
292
- result = lower_left
292
+ result = +lower_left
293
293
  widths.each_value do |w|
294
294
  result += double_rule * (w + 2) + lower_tee
295
295
  end
@@ -16,7 +16,7 @@ module FatTable
16
16
  end
17
17
 
18
18
  def pre_header(widths)
19
- result = '+'
19
+ result = +'+'
20
20
  widths.each_value do |w|
21
21
  result += '=' * (w + 2) + '+'
22
22
  end
@@ -49,7 +49,7 @@ module FatTable
49
49
  end
50
50
 
51
51
  def hline(widths)
52
- result = '+'
52
+ result = +'+'
53
53
  widths.each_value do |w|
54
54
  result += '-' * (w + 2) + '+'
55
55
  end
@@ -82,7 +82,7 @@ module FatTable
82
82
  end
83
83
 
84
84
  def post_footers(widths)
85
- result = '+'
85
+ result = +'+'
86
86
  widths.each_value do |w|
87
87
  result += '=' * (w + 2) + '+'
88
88
  end
@@ -62,19 +62,53 @@ module FatTable
62
62
  # method call.
63
63
  attr_accessor :explicit_boundaries
64
64
 
65
+ # An Array of FatTable::Columns that should be tolerant.
66
+ attr_reader :tolerant_columns
67
+
65
68
  ###########################################################################
66
69
  # Constructors
67
70
  ###########################################################################
68
71
 
69
72
  # :category: Constructors
70
73
 
71
- # Return an empty FatTable::Table object.
72
- def initialize(*heads)
74
+ # Return an empty FatTable::Table object. Specifying headers is optional.
75
+ # Any headers ending with a ! are marked as tolerant, in that, if an
76
+ # incompatible type is added to it, the column is re-typed as a String
77
+ # column, and construction proceeds. The ! is stripped from the header to
78
+ # form the column key, though. You can also provide the names of columns
79
+ # that should be tolerant by using the +tolerant_columns key-word to
80
+ # provide an array of headers that should be tolerant. The special string
81
+ # '*' or the symbol :* indicates that all columns should be created
82
+ # tolerant.
83
+ def initialize(*heads, tolerant_columns: [])
73
84
  @columns = []
74
85
  @explicit_boundaries = []
86
+ @tolerant_columns =
87
+ case tolerant_columns
88
+ when Array
89
+ tolerant_columns.map { |h| h.to_s.as_sym }
90
+ when String
91
+ if tolerant_columns.strip == '*'
92
+ ['*'.to_sym]
93
+ else
94
+ [tolerant_columns.as_sym]
95
+ end
96
+ when Symbol
97
+ if tolerant_columns.to_s.strip == '*'
98
+ ['*'.to_sym]
99
+ else
100
+ [tolerant_columns.to_s.as_sym]
101
+ end
102
+ else
103
+ raise ArgumentError, "set tolerant_columns to String, Symbol, or an Array of either"
104
+ end
75
105
  unless heads.empty?
76
106
  heads.each do |h|
77
- @columns << Column.new(header: h)
107
+ if h.to_s.end_with?('!') || @tolerant_columns.include?(h)
108
+ @columns << Column.new(header: h.to_s.sub(/!\s*\z/, ''), type: 'String')
109
+ else
110
+ @columns << Column.new(header: h)
111
+ end
78
112
  end
79
113
  end
80
114
  end
@@ -86,7 +120,7 @@ module FatTable
86
120
  # though FatTable::Table objects have no instance variables, a class that
87
121
  # inherits from it might.
88
122
  def empty_dup
89
- self.dup.__empty!
123
+ dup.__empty!
90
124
  end
91
125
 
92
126
  def __empty!
@@ -99,9 +133,9 @@ module FatTable
99
133
 
100
134
  # Construct a Table from the contents of a CSV file named +fname+. Headers
101
135
  # will be taken from the first CSV row and converted to symbols.
102
- def self.from_csv_file(fname)
136
+ def self.from_csv_file(fname, tolerant_columns: [])
103
137
  File.open(fname, 'r') do |io|
104
- from_csv_io(io)
138
+ from_csv_io(io, tolerant_columns: tolerant_columns)
105
139
  end
106
140
  end
107
141
 
@@ -109,8 +143,8 @@ module FatTable
109
143
 
110
144
  # Construct a Table from a CSV string +str+, treated in the same manner as
111
145
  # the input from a CSV file in ::from_org_file.
112
- def self.from_csv_string(str)
113
- from_csv_io(StringIO.new(str))
146
+ def self.from_csv_string(str, tolerant_columns: [])
147
+ from_csv_io(StringIO.new(str), tolerant_columns: tolerant_columns)
114
148
  end
115
149
 
116
150
  # :category: Constructors
@@ -119,9 +153,9 @@ module FatTable
119
153
  # file named +fname+. Headers are taken from the first row if the second row
120
154
  # is an hrule. Otherwise, synthetic headers of the form +:col_1+, +:col_2+,
121
155
  # etc. are created.
122
- def self.from_org_file(fname)
156
+ def self.from_org_file(fname, tolerant_columns: [])
123
157
  File.open(fname, 'r') do |io|
124
- from_org_io(io)
158
+ from_org_io(io, tolerant_columns: tolerant_columns)
125
159
  end
126
160
  end
127
161
 
@@ -129,8 +163,8 @@ module FatTable
129
163
 
130
164
  # Construct a Table from a string +str+, treated in the same manner as the
131
165
  # contents of an org-mode file in ::from_org_file.
132
- def self.from_org_string(str)
133
- from_org_io(StringIO.new(str))
166
+ def self.from_org_string(str, tolerant_columns: [])
167
+ from_org_io(StringIO.new(str), tolerant_columns: tolerant_columns)
134
168
  end
135
169
 
136
170
  # :category: Constructors
@@ -149,8 +183,8 @@ module FatTable
149
183
  # :hlines no +) org-mode strips all hrules from the table; otherwise (+
150
184
  # HEADER: :hlines yes +) they are indicated with nil elements in the outer
151
185
  # array.
152
- def self.from_aoa(aoa, hlines: false)
153
- from_array_of_arrays(aoa, hlines: hlines)
186
+ def self.from_aoa(aoa, hlines: false, tolerant_columns: [])
187
+ from_array_of_arrays(aoa, hlines: hlines, tolerant_columns: tolerant_columns)
154
188
  end
155
189
 
156
190
  # :category: Constructors
@@ -160,9 +194,9 @@ module FatTable
160
194
  # keys, which, when converted to symbols will become the headers for the
161
195
  # Table. If hlines is set true, mark a group boundary whenever a nil, rather
162
196
  # than a hash appears in the outer array.
163
- def self.from_aoh(aoh, hlines: false)
197
+ def self.from_aoh(aoh, hlines: false, tolerant_columns: [])
164
198
  if aoh.first.respond_to?(:to_h)
165
- from_array_of_hashes(aoh, hlines: hlines)
199
+ from_array_of_hashes(aoh, hlines: hlines, tolerant_columns: tolerant_columns)
166
200
  else
167
201
  raise UserError,
168
202
  "Cannot initialize Table with an array of #{input[0].class}"
@@ -181,7 +215,7 @@ module FatTable
181
215
 
182
216
  # Construct a Table by running a SQL +query+ against the database set up
183
217
  # with FatTable.connect, with the rows of the query result as rows.
184
- def self.from_sql(query)
218
+ def self.from_sql(query, tolerant_columns: [])
185
219
  msg = 'FatTable.db must be set with FatTable.connect'
186
220
  raise UserError, msg if FatTable.db.nil?
187
221
 
@@ -203,8 +237,8 @@ module FatTable
203
237
  # Construct table from an array of hashes or an array of any object that
204
238
  # can respond to #to_h. If an array element is a nil, mark it as a group
205
239
  # boundary in the Table.
206
- def from_array_of_hashes(hashes, hlines: false)
207
- result = new
240
+ def from_array_of_hashes(hashes, hlines: false, tolerant_columns: [])
241
+ result = new(tolerant_columns: tolerant_columns)
208
242
  hashes.each do |hsh|
209
243
  if hsh.nil?
210
244
  unless hlines
@@ -232,8 +266,8 @@ module FatTable
232
266
  # hlines are stripped from the table, otherwise (:hlines yes) they are
233
267
  # indicated with nil elements in the outer array as expected by this
234
268
  # method when hlines is set true.
235
- def from_array_of_arrays(rows, hlines: false)
236
- result = new
269
+ def from_array_of_arrays(rows, hlines: false, tolerant_columns: [])
270
+ result = new(tolerant_columns: tolerant_columns)
237
271
  headers = []
238
272
  if !hlines
239
273
  # Take the first row as headers
@@ -269,8 +303,8 @@ module FatTable
269
303
  result
270
304
  end
271
305
 
272
- def from_csv_io(io)
273
- result = new
306
+ def from_csv_io(io, tolerant_columns: [])
307
+ result = new(tolerant_columns: tolerant_columns)
274
308
  ::CSV.new(io, headers: true, header_converters: :symbol,
275
309
  skip_blanks: true).each do |row|
276
310
  result << row.to_h
@@ -283,7 +317,7 @@ module FatTable
283
317
  # header row must be marked with an hline (i.e, a row that looks like
284
318
  # '|---+--...--|') and groups of rows may be marked with hlines to
285
319
  # indicate group boundaries.
286
- def from_org_io(io)
320
+ def from_org_io(io, tolerant_columns: [])
287
321
  table_re = /\A\s*\|/
288
322
  hrule_re = /\A\s*\|[-+]+/
289
323
  rows = []
@@ -318,7 +352,7 @@ module FatTable
318
352
  rows << line.split('|').map(&:clean)
319
353
  end
320
354
  end
321
- from_array_of_arrays(rows, hlines: true)
355
+ from_array_of_arrays(rows, hlines: true, tolerant_columns: tolerant_columns)
322
356
  end
323
357
  end
324
358
 
@@ -412,6 +446,15 @@ module FatTable
412
446
 
413
447
  # :category: Attributes
414
448
 
449
+ # Return whether the column with the given head should be made tolerant.
450
+ def tolerant_col?(h)
451
+ return true if tolerant_columns.include?(:'*')
452
+
453
+ tolerant_columns.include?(h)
454
+ end
455
+
456
+ # :category: Attributes
457
+
415
458
  # Return the number of rows in the Table.
416
459
  def size
417
460
  return 0 if columns.empty?
@@ -571,7 +614,8 @@ module FatTable
571
614
  range = group_row_range(k)
572
615
  tab_col = column(col)
573
616
  gitems = tab_col.items[range]
574
- cols << Column.new(header: col, items: gitems, type: tab_col.type)
617
+ cols << Column.new(header: col, items: gitems,
618
+ type: tab_col.type, tolerant: tab_col.tolerant?)
575
619
  end
576
620
  cols
577
621
  end
@@ -735,7 +779,6 @@ module FatTable
735
779
  last_key = nil
736
780
  new_rows.each_with_index do |nrow, k|
737
781
  new_tab << nrow
738
- # key = nrow.fetch_values(*sort_heads)
739
782
  key = nrow.fetch_values(*key_hash.keys)
740
783
  new_tab.mark_boundary(k - 1) if last_key && key != last_key
741
784
  last_key = key
@@ -941,7 +984,12 @@ module FatTable
941
984
  expr = expr.to_s
942
985
  result = empty_dup
943
986
  headers.each do |h|
944
- col = Column.new(header: h)
987
+ col =
988
+ if tolerant_col?(h)
989
+ Column.new(header: h, tolerant: true)
990
+ else
991
+ Column.new(header: h)
992
+ end
945
993
  result.add_column(col)
946
994
  end
947
995
  ev = Evaluator.new(ivars: { row: 0, group: 0 })
@@ -1406,6 +1454,9 @@ module FatTable
1406
1454
 
1407
1455
  private
1408
1456
 
1457
+ # Collapse a group of rows to a single row by applying the aggregator from
1458
+ # the +agg_cols+ to the items in that column and the presumably identical
1459
+ # value in the +grp_cols to those columns.
1409
1460
  def row_from_group(rows, grp_cols, agg_cols)
1410
1461
  new_row = {}
1411
1462
  grp_cols.each do |h|
@@ -1440,7 +1491,7 @@ module FatTable
1440
1491
  # This column is new, so it needs nil items for all prior rows lest
1441
1492
  # the value be added to a prior row.
1442
1493
  items = Array.new(size, nil)
1443
- columns << Column.new(header: h, items: items)
1494
+ columns << Column.new(header: h, items: items, tolerant: tolerant_col?(h))
1444
1495
  end
1445
1496
  headers.each do |h|
1446
1497
  # NB: This adds a nil if h is not in row.
@@ -1660,11 +1711,19 @@ module FatTable
1660
1711
  end
1661
1712
 
1662
1713
  # The <=> operator cannot handle nils without some help. Treat a nil as
1663
- # smaller than any other value, but equal to other nils. The two keys are assumed to be arrays of values to be
1664
- # compared with <=>.
1714
+ # smaller than any other value, but equal to other nils. The two keys are
1715
+ # assumed to be arrays of values to be compared with <=>. Since
1716
+ # tolerant_columns permit strings to be mixed in with columns of type
1717
+ # Numeric, DateTime, and Boolean, treat strings mixed with another type
1718
+ # the same as nils.
1665
1719
  def compare_with_nils(key1, key2)
1666
1720
  result = nil
1667
1721
  key1.zip(key2) do |k1, k2|
1722
+ if k1.is_a?(String) && !k2.is_a?(String)
1723
+ k1 = nil
1724
+ elsif !k1.is_a?(String) && k2.is_a?(String)
1725
+ k2 = nil
1726
+ end
1668
1727
  if k1.nil? && k2.nil?
1669
1728
  result = 0
1670
1729
  next
@@ -2,5 +2,5 @@
2
2
 
3
3
  module FatTable
4
4
  # The current version of FatTable
5
- VERSION = '0.5.2'
5
+ VERSION = '0.5.5'
6
6
  end
data/lib/fat_table.rb CHANGED
@@ -61,22 +61,22 @@ module FatTable
61
61
 
62
62
  # Return an empty FatTable::Table object. You can use FatTable::Table#add_row
63
63
  # or FatTable::Table#add_column to populate the table with data.
64
- def self.new(*args)
65
- Table.new(*args)
64
+ def self.new(*args, tolerant_columns: [])
65
+ Table.new(*args, tolerant_columns: tolerant_columns)
66
66
  end
67
67
 
68
68
  # Construct a FatTable::Table from the contents of a CSV file given by the
69
69
  # file name +fname+. Headers will be taken from the first row and converted to
70
70
  # symbols.
71
- def self.from_csv_file(fname)
72
- Table.from_csv_file(fname)
71
+ def self.from_csv_file(fname, tolerant_columns: [])
72
+ Table.from_csv_file(fname, tolerant_columns: tolerant_columns)
73
73
  end
74
74
 
75
75
  # Construct a FatTable::Table from the string +str+, treated in the same
76
76
  # manner as if read the input from a CSV file. Headers will be taken from the
77
77
  # first row and converted to symbols.
78
- def self.from_csv_string(str)
79
- Table.from_csv_string(str)
78
+ def self.from_csv_string(str, tolerant_columns: [])
79
+ Table.from_csv_string(str, tolerant_columns: tolerant_columns)
80
80
  end
81
81
 
82
82
  # Construct a FatTable::Table from the first table found in the Emacs org-mode
@@ -84,8 +84,8 @@ module FatTable
84
84
  # is an hline. Otherwise, synthetic headers of the form +:col_1+, +:col_2+,
85
85
  # etc. are created. Any other hlines will be treated as marking a boundary in
86
86
  # the table.
87
- def self.from_org_file(fname)
88
- Table.from_org_file(fname)
87
+ def self.from_org_file(fname, tolerant_columns: [])
88
+ Table.from_org_file(fname, tolerant_columns: tolerant_columns)
89
89
  end
90
90
 
91
91
  # Construct a FatTable::Table from the first table found in the string +str+,
@@ -93,8 +93,8 @@ module FatTable
93
93
  # are taken from the first row if the second row is an hrule. Otherwise,
94
94
  # synthetic headers of the form :col_1, :col_2, etc. are created. Any other
95
95
  # hlines will be treated as marking a boundary in the table.
96
- def self.from_org_string(str)
97
- Table.from_org_string(str)
96
+ def self.from_org_string(str, tolerant_columns: [])
97
+ Table.from_org_string(str, tolerant_columns: tolerant_columns)
98
98
  end
99
99
 
100
100
  # Construct a FatTable::Table from the array of arrays +aoa+. By default, with
@@ -108,8 +108,8 @@ module FatTable
108
108
  # org-mode code blocks, by default (+:hlines no+) all hlines are stripped from
109
109
  # the table, otherwise (+:hlines yes+) they are indicated with nil elements in
110
110
  # the outer array.
111
- def self.from_aoa(aoa, hlines: false)
112
- Table.from_aoa(aoa, hlines: hlines)
111
+ def self.from_aoa(aoa, hlines: false, tolerant_columns: [])
112
+ Table.from_aoa(aoa, hlines: hlines, tolerant_columns: tolerant_columns)
113
113
  end
114
114
 
115
115
  # Construct a FatTable::Table from the array of hashes +aoh+, which can be an
@@ -117,8 +117,8 @@ module FatTable
117
117
  # interpret nil separators as marking boundaries in the new Table. All hashes
118
118
  # must have the same keys, which, converted to symbols, become the headers for
119
119
  # the new Table.
120
- def self.from_aoh(aoh, hlines: false)
121
- Table.from_aoh(aoh, hlines: hlines)
120
+ def self.from_aoh(aoh, hlines: false, tolerant_columns: [])
121
+ Table.from_aoh(aoh, hlines: hlines, tolerant_columns: tolerant_columns)
122
122
  end
123
123
 
124
124
  # Construct a FatTable::Table from another FatTable::Table. Inherit any group
@@ -130,8 +130,8 @@ module FatTable
130
130
  # Construct a Table by running a SQL query against the database set up with
131
131
  # FatTable.connect. Return the Table with the query results as rows and the
132
132
  # headers from the query, converted to symbols, as headers.
133
- def self.from_sql(query)
134
- Table.from_sql(query)
133
+ def self.from_sql(query, tolerant_columns: [])
134
+ Table.from_sql(query, tolerant_columns: tolerant_columns)
135
135
  end
136
136
 
137
137
  ########################################################################
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fat_table
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.5.2
4
+ version: 0.5.5
5
5
  platform: ruby
6
6
  authors:
7
7
  - Daniel E. Doherty
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2022-01-22 00:00:00.000000000 Z
11
+ date: 2022-03-25 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler