fat_table 0.7.0 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ba8791fd0b89302cb5127adc325afd0a84bee337298955ecb48119805bcff53a
-  data.tar.gz: da6f4923fe75df0707c3fc21f96fef52127188e76f8a9ae1b923ea768689d31e
+  metadata.gz: 6511a69dea56479777956084ceae10b33d6769b4e9e8fcbb87d50140fa5475d4
+  data.tar.gz: b41a1e1a578f1c9f4f4f82ac7fba1413343ec75d8febda7ebb1f12737b9fb6b2
 SHA512:
-  metadata.gz: 7f9b047a93e55f7752173dc7f2c927db2bf45669ede7de1d652b89f1052054f90763c7910b850f84d80ea86a959c82e48f9f9d5b5fe9192ff6b15f9dabb104a7
-  data.tar.gz: 498eb355cce5f5fa8597ad23657d4699cfead201b6bbf5200386f50773c846061199de0373edbe41ff94c79e1a5d90da8c1f9a37d275c580a0a021109c2d8780
+  metadata.gz: fe1da822b9d2ef6995dbfe665f570a3474d955892e062cba559162a07ef43013e9c1f649040fa47529c780aec213a9d74c4ac0684217a608960ebedac9479843
+  data.tar.gz: b4ab503c3f992a471bd72a70822455ac6a50a6a72558242841656d514e2c60c9b51bf7a1d052c66a35ecb1ba4cbce30ac6351d30919aa85f3284f0d9cfad1821

data/README.org

@@ -445,27 +445,49 @@ or nil. There are only five permissible types for a ~Column~:
 4. *String* (for ruby ~String~ objects), or
 5. *NilClass* (for the undetermined column type).
 
-When a ~Table~ is constructed from an external source, all ~Columns~ start out
-having a type of ~NilClass~, that is, their type is as yet undetermined. When
-a string or object of one of the four determined types is added to a ~Column~
-and it can be converted into one of the permissible types, it fixes the type
-of the column, and all further items added to the ~Column~ must either be
-~nil~ (indicating no value) or be capable of being coerced to the column's
-type. Otherwise, ~FatTable~ raises an exception.
+By default, when a ~Table~ is constructed from an external source, all
+~Columns~ start out having a type of ~NilClass~, that is, their type is as yet
+undetermined. When a string or object is added to a ~Column~ and it can be
+converted into one of the permissible types, it fixes the type of the column,
+and all further items added to the ~Column~ must either be ~nil~ (indicating
+no value) or be capable of being coerced to the column's type. Otherwise,
+~FatTable~ raises an ~IncompatibleTypeError~ exception.
+
+*** Type Keywords Arguments
+All of the table constructors allow you to set the type for a column in
+advance by adding keyword arguments to the end of the contructor arguments
+where the keyword is a header symbol and the value is a string designating one
+of the types.  For example, suppose we are constructing a table from a CSV
+file, and we know that one of the columns is labeled 'Start' and another
+'Price'.  We want to require the items in the 'Start' column to be a valid
+date and the items in the 'Price' column to be valid numbers:
+
+#+begin_example
+  FatTable.from_csv_file('data.csv', start: 'date', price: 'num')
+#+end_example
+
+The type string can be anything that starts with 'dat', 'num', 'boo', or
+'str', regardless of case, to designate ~DateTime~, ~Numeric~, ~Boolean~, or
+~String~ types, respectively.  Any other string keeps the type as NilClass,
+that is, it remains open for automatic typing.
 
 The strictness of requiring all items to be of the same type can be relaxed by
-declaring a column to be "tolerant."  You can do so when you create the table
-by adding a tolerant_columns keyword parameter.  If a Column is tolerant,
-~FatTable~ tries to convert new items into a type other than a ~String~ and,
-if it can do so, sets /that/ as the Column's type.  Any later items that
-cannot be converted into the Column's type are converted to strings.  These
-interloper strings are treated like nils for purposes of sorting and
-evaluation, but are displayed according to any string formatting on output.
-See [[*Designating "Tolerant" Columns][Designating "Tolerant" Columns]] below.
-
-It is also possible to force ~FatTable~ to treat a column as a String type,
-even its items look like one of the other types.  See [[*Forcing String Type][Forcing String Type]]
-below.
+declaring a column to be "tolerant."  You can do so by adding a '~' to the end
+of a keyword type specifier in the table constructor.  In the above example,
+if we wanted to allow strings to be mixed up with the numeric prices, we would
+use the following:
+
+#+begin_example
+  FatTable.from_csv_file('data.csv', start: 'date', price: 'num~')
+#+end_example
+
+If a Column is tolerant, ~FatTable~ tries to convert new items into the
+column's specified type, or if the type is still open, to one of ~DateTime~,
+~Numeric~, or ~Boolean~ and then fixing the column's type, or, if it cannot do
+so converts the item into a ~String~ but does not raise an
+~IncompatibleTypeError~ exception.  These interloper strings are treated like
+nils for purposes of sorting and evaluation, but are displayed according to
+any string formatting on output.  See [[*Designating "Tolerant" Columns][Designating "Tolerant" Columns]] below.
 
 Items of input must be either one of the permissible ruby objects or strings. If
 they are strings, ~FatTable~ attempts to parse them as one of the permissible
@@ -626,14 +648,32 @@ columns to be created:
 
 Occasionally, ~FatTable~'s automatic type detection can get in the way and you
 just want it to treat one or more columns as Strings regardless of their
-appearance.  Think, for example, of zip codes.  If headers are given when a
-table is contructed, you can designate a forced-string column by appending a
-~!~ to the end of the header.  It will not become part of the header, it will
-just mark it as a forced-string Column.
+appearance.  Think, for example, of zip codes.  As mentioned above, when a
+table is contructed, you can designate a 'String' type for a column by
+using a keyword parameter.
 
-#+begin_SRC emacs-lisp :wrap EXAMPLE
-  tab = FatTable.new(:a, 'b', 'C!', :d, :zip!)
-#+end_SRC
+#+begin_src ruby :wrap EXAMPLE
+  require 'fat_table'
+  tab = FatTable.new(:a, 'b!', 'C', :d, :zip, zip: 'str')
+  tab << { a: 1, b: 2, c: "<2017-01-21>", d: 'f', e: '', zip: 18552 }
+  tab << { a: 3.14, b: 2.17, c: '[2016-01-21 Thu]', d: 'Y', e: nil }
+  tab << { zip: '01879--7884' }
+  tab << { zip: '66210', b: 'Not a Number' }
+  tab << { zip: '90210' }
+  tab.to_text
+#+end_src
+
+#+begin_EXAMPLE
++===+===+============+===+=============+===+
+| A | B | C          | D | Zip         | E |
++---+---+------------+---+-------------+---+
+| 1 | 2 | 2017-01-21 | F | 18552       |   |
+| 3 | 2 | 2016-01-21 | T |             |   |
+|   |   |            |   | 01879--7884 |   |
+|   |   |            |   | 90210       |   |
+|   |   |            |   |             |   |
++===+===+============+===+=============+===+
+#+end_EXAMPLE
 
 In addition, at any time after creating a table, you can force the String type
 on any number of columns with the ~force_string!~ method.  When you do so, all
@@ -662,84 +702,9 @@ exisiting items in the column are converted to strings with the #to_s method.
 +======+======+============+===+=======+===+
 #+end_EXAMPLE
 
-**** Designating "Tolerant" Columns
-
-Related to the problem just discussed is the problem of reading files in from
-the wild where a column may get typed as, say Numeric, but then contain
-something that can't be parsed as a Numeric.  ~FatTable~ raises an exception
-is such cases, and that may be what you want if you can control the input.
-But, especially when you cannot do so, it can be helpful to designate one or
-more columns as "tolerant."  This means that when a conversion problem occurs,
-the column item is retained as a string type in a column that is otherwise of
-one of the types Numeric, DateTime, or Boolean.  Those string items are
-treated as nils for purposes of sorting or evaluation in a ~select~ method.
-When formatted, they participate in string formatting directive, but not those
-for other types.
-
-All of the table construction methods, allow a keyword parameter,
-~tolerant_columns~, where you can designate what columns should be convert to
-String type when conversion to the auto-typed column type is not possible.
-The parameter should be an array of headers, in either string or symbol form,
-for which this behavior is desired.  In addition, it can be set to the special
-string '*' or symbol ~:*~ to indicate that all the columns should be made
-tolerant.
-
-#+begin_src ruby :wrap EXAMPLE
-  require 'fat_table'
-  tab = FatTable.new(:a, 'b', 'C', :d, :zip, tolerant_columns: [:zip])
-  tab << { a: 1, b: 2, c: "<2017-01-21>", d: 'f', e: '', zip: 18552 }
-  tab << { a: 3.14, b: 2.17, c: '[2016-01-21 Thu]', d: 'Y', e: nil }
-  tab << { zip: '01879--7884' }
-  tab << { zip: '66210' }
-  tab << { zip: '90210' }
-  tab.to_text
-#+end_src
-
-#+RESULTS:
-#+begin_EXAMPLE
-+======+======+============+===+=============+===+
-| A    | B    | C          | D | Zip         | E |
-+------+------+------------+---+-------------+---+
-| 1    | 2    | 2017-01-21 | F | 18552       |   |
-| 3.14 | 2.17 | 2016-01-21 | T |             |   |
-|      |      |            |   | 01879--7884 |   |
-|      |      |            |   | 66210       |   |
-|      |      |            |   | 90210       |   |
-+======+======+============+===+=============+===+
-#+end_EXAMPLE
-
-Another way to designate a column as tolerant is to end a column you want to
-designate as tolerant with a ~!~.  The ~!~ will be stripped from the header,
-but it will be marked as tolerant.
-#+begin_src ruby :wrap EXAMPLE
-  require 'fat_table'
-  tab = FatTable.new(:a, 'b!', 'C', :d, :zip!)
-  tab << { a: 1, b: 2, c: "<2017-01-21>", d: 'f', e: '', zip: 18552 }
-  tab << { a: 3.14, b: 2.17, c: '[2016-01-21 Thu]', d: 'Y', e: nil }
-  tab << { zip: '01879--7884' }
-  tab << { zip: '66210', b: 'Not a Number' }
-  tab << { zip: '90210' }
-  tab.to_text
-#+end_src
-
-#+RESULTS:
-#+begin_EXAMPLE
-+======+==============+============+===+=============+===+
-| A    | B            | C          | D | Zip         | E |
-+------+--------------+------------+---+-------------+---+
-| 1    | 2            | 2017-01-21 | F | 18552       |   |
-| 3.14 | 2.17         | 2016-01-21 | T |             |   |
-|      |              |            |   | 01879--7884 |   |
-|      | Not a Number |            |   | 66210       |   |
-|      |              |            |   | 90210       |   |
-+======+==============+============+===+=============+===+
-#+end_EXAMPLE
-
 *** From CSV or Org Mode files or strings
 Tables can also be read from ~.csv~ files or files containing ~org-mode~
-tables.  Remember that you can make any column tolerant with a
-~tolerant_columns:~ keyword argument or make them all tolerant by designating
-the pseudo-column ~:*~ as tolerant.
+tables.
 
 In the case of org-mode files, ~FatTable~ skips through the file until it finds
 a line that look like a table, that is, it begins with any number of spaces
@@ -799,10 +764,10 @@ header row, and the headers are converted to symbols as described above.
 
 You can also initialize a table directly from ruby data structures. You can,
 for example, build a table from an array of arrays.  Remember that you can
-make any column tolerant with a ~tolerant_columns:~ keyword argument or make
-them all tolerant by designating the pseudo-column ~:*~ as tolerant.
+make any column tolerant with a keyword argument for the column symbol and
+ending it with a '~'.
 
-#+BEGIN_SRC ruby :eval never
+#+BEGIN_SRC ruby
   aoa = [
     ['Ref', 'Date', 'Code', 'Raw', 'Shares', 'Price', 'Info', 'Bool'],
     [1, '2013-05-02', 'P', 795_546.20, 795_546.2, 1.1850, 'ENTITY1', 'T'],
@@ -816,11 +781,27 @@ them all tolerant by designating the pseudo-column ~:*~ as tolerant.
     [13, '2013-05-29', 'S', 13_459.00, 5659.51, 24.7464, 'ENTITY3', 'T'],
     [14, '2013-05-29', 'S', 15_700.00, 6601.85, 24.7790, 'ENTITY3', 'F'],
     [15, '2013-05-29', 'S', 15_900.00, 6685.95, 24.5802, 'ENTITY3', 'T'],
-    [16, '2013-05-30', 'S', 6_679.00, 2808.52, 25.0471, 'ENTITY3', 'T']
-  ]
-  tab = FatTable.from_aoa(aoa)
+    [16, '2013-05-30', 'S', 6_679.00, 2808.52, 25.0471, 'ENTITY3', 'T'] ]
+
+  tab = FatTable.from_aoa(aoa).to_aoa
 #+END_SRC
 
+#+RESULTS:
+| Ref | Date       | Code |    Raw | Shares | Price | Info    | Bool |
+|-----+------------+------+--------+--------+-------+---------+------|
+|   1 | 2013-05-02 | P    | 795546 | 795546 |     1 | ENTITY1 | T    |
+|   2 | 2013-05-02 | P    | 118186 | 118186 |    12 | ENTITY1 | T    |
+|   7 | 2013-05-20 | S    |  12000 |   5046 |    28 | ENTITY3 | F    |
+|   8 | 2013-05-20 | S    |  85000 |  35743 |    28 | ENTITY3 | T    |
+|   9 | 2013-05-20 | S    |  33302 |  14003 |    29 | ENTITY3 | T    |
+|  10 | 2013-05-23 | S    |   8000 |   3364 |    27 | ENTITY3 | T    |
+|  11 | 2013-05-23 | S    |  23054 |   9694 |    27 | ENTITY3 | F    |
+|  12 | 2013-05-23 | S    |  39906 |  16780 |    25 | ENTITY3 | T    |
+|  13 | 2013-05-29 | S    |  13459 |   5660 |    25 | ENTITY3 | T    |
+|  14 | 2013-05-29 | S    |  15700 |   6602 |    25 | ENTITY3 | F    |
+|  15 | 2013-05-29 | S    |  15900 |   6686 |    25 | ENTITY3 | T    |
+|  16 | 2013-05-30 | S    |   6679 |   2809 |    25 | ENTITY3 | T    |
+
 Notice that the values can either be ruby objects, such as the Integer ~85_000~,
 or strings that can be parsed into one of the permissible column types.
 
@@ -898,9 +879,7 @@ This example illustrates several things:
 A second ruby data structure that can be used to initialize a ~FatTable~ table
 is an array of ruby Hashes. Each hash represents a row of the table, and the
 headers of the table are taken from the keys of the hashes. Accordingly, all
-the hashes must have the same keys.  Remember that you can make any column
-tolerant with a ~tolerant_columns:~ keyword argument or make them all tolerant
-by designating the pseudo-column ~:*~ as tolerant.
+the hashes must have the same keys.
 
 This same method can in fact take an array of any objects that can be converted
 to a Hash with the ~#to_h~ method, so you can use an array of your own objects
@@ -989,10 +968,6 @@ The ~.connect~ function need only be called once, and the database handle it
 creates will be used for all subsequent ~.from_sql~ calls until ~.connect~ is
 called again.
 
-Remember that you can make any column tolerant with a ~tolerant_columns:~
-keyword argument or make them all tolerant by designating the pseudo-column
-~:*~ as tolerant.
-
 *** Marking Groups in Input
 **** Manually
 

data/lib/fat_table/column.rb

@@ -29,6 +29,8 @@ module FatTable
     # perhaps after removing nils with +.items.compact+.
     attr_reader :items
 
+    attr_accessor :tolerant
+
     # Valid Column types as strings.
     TYPES = %w[NilClass Boolean DateTime Numeric String].freeze
 

data/lib/fat_table/convert.rb

@@ -63,19 +63,6 @@ module FatTable
       when 'String'
         if val.nil?
           nil
-        elsif tolerant
-          # Allow String to upgrade to one of Numeric, DateTime, or Boolean if
-          # possible.
-          if (new_val = convert_to_numeric(val))
-            new_val
-          elsif (new_val = convert_to_date_time(val))
-            new_val
-          elsif (new_val = convert_to_boolean(val))
-            new_val
-          else
-            new_val = convert_to_string(val)
-          end
-          new_val
         else
           new_val = convert_to_string(val)
           if new_val.nil?

data/lib/fat_table/table.rb

@@ -53,6 +53,11 @@ module FatTable
   class Table
     # An Array of FatTable::Columns that constitute the table.
     attr_reader :columns
+    attr_reader :heads
+
+    # Headers of columns that are to be tolerant when they are built.
+    attr_accessor :tolerant_cols
+    attr_reader :omni_typ, :omni_tol
 
     # Record boundaries set explicitly with mark_boundaries or from reading
     # hlines from input.  When we want to access boundaries, however, we want
@@ -62,80 +67,94 @@ module FatTable
     # method call.
     attr_accessor :explicit_boundaries
 
-    # An Array of FatTable::Columns that should be tolerant.
-    attr_reader :tolerant_columns
-
     ###########################################################################
     # Constructors
-    ###########################################################################
-
+    #
+    #
     # :category: Constructors
-
+    #
     # Return an empty FatTable::Table object.  Specifying headers is optional.
-    # Any headers ending with a ! are marked as tolerant, in that, if an
-    # incompatible type is added to it, the column is re-typed as a String
-    # column, and construction proceeds.  The ! is stripped from the header to
-    # form the column key, though.  You can also provide the names of columns
-    # that should be tolerant by using the +tolerant_columns key-word to
-    # provide an array of headers that should be tolerant.  The special string
-    # '*' or the symbol :* indicates that all columns should be created
-    # tolerant.
-    def initialize(*heads, tolerant_columns: [])
+    # By default, all columns start our as having an "open" type and get
+    # assigned a type based on their contents.  For example, if a column
+    # contains items that can be interpreted as dates, the column gets
+    # assigned a DateTime type.  Other types are Numeric, Boolean, and String.
+    # Once a type is assigned to a column, any non-conforming vaules in that
+    # column raise an IncompatibleType error.  If a column is marked
+    # "tolerant", however, the incompatible item is converted to a string and
+    # allowed to remain in the column without raising an error.  They count as
+    # nils when calculations are performed on the column and paricipate only
+    # in string formatting directives on output.
+    #
+    # Rather than have a column's type determined by content, you can also
+    # specify a column type by providing a type hash, where the key is the
+    # header's name and the value is the desired type.  In that case, any
+    # incompatible type raises an an IncompatibleTypeError unless the column
+    # is also marked tolerant, in which case it gets converted to a string as
+    # discussed above.  If the type name in the types hash ends in a '~', it
+    # is treated as a specifying the given type but marking it as tolerant as
+    # well.  The values in the type hash can be any string or sybol that
+    # starts with  'num', 'dat', 'bool', or 'str' to specify Numeric,
+    # DateTime, Boolean, or String types respectively.
+    def initialize(*heads, **types)
+      @heads = heads.flatten.map(&:as_sym)
+      @types = types
       @columns = []
-      @explicit_boundaries = []
-      @tolerant_columns =
-        case tolerant_columns
-        when Array
-          tolerant_columns.map { |h| h.to_s.as_sym }
-        when String
-          if tolerant_columns.strip == '*'
-            ['*'.to_sym]
-          else
-            [tolerant_columns.as_sym]
-          end
-        when Symbol
-          if tolerant_columns.to_s.strip == '*'
-            ['*'.to_sym]
-          else
-            [tolerant_columns.to_s.as_sym]
-          end
+      @tolerant_cols = []
+      # Check for the special 'omni' key
+      @omni_type = 'NilClass'
+      @omni_tol = false
+      if types.keys.map(&:to_s).include?('omni')
+        # All columns not otherwise included in types should have the type and
+        # tolerance of omni.
+        omni_val = (types['omni'] || types[:omni])
+        @omni_type, @omni_tol = Table.typ_tol(omni_val)
+        # Remove omni from types.
+        types.delete(:omni)
+        types.delete('omni')
+      end
+      # heads += types.keys
+      (heads.flatten + types.keys).uniq.each do |h|
+        if types[h]
+          typ, tol = Table.typ_tol(types[h])
         else
-          raise ArgumentError, "set tolerant_columns to String, Symbol, or an Array of either"
-        end
-      unless heads.empty?
-        heads.each do |h|
-          if h.to_s.end_with?('!') || @tolerant_columns.include?(h)
-            @columns << Column.new(header: h.to_s.sub(/!\s*\z/, ''), type: 'String')
-          else
-            @columns << Column.new(header: h)
-          end
+          typ = @omni_type
+          tol = @omni_tol
         end
+        @tolerant_cols << h.to_s.as_sym if tol
+        @columns << Column.new(header: h.to_s.sub(/~\s*\z/, ''), type: typ,
+                               tolerant: tol)
       end
+      @explicit_boundaries = []
     end
 
     # :category: Constructors
 
-    # Return an empty duplicate of self.  This allows the library to create an
-    # empty table that preserves all the instance variables from self.  Even
-    # though FatTable::Table objects have no instance variables, a class that
-    # inherits from it might.
-    def empty_dup
-      dup.__empty!
-    end
-
-    def __empty!
-      @columns = []
-      @explicit_boundaries = []
-      self
+    # Return an new table based on this Table but with empty columns named by
+    # the result_cols parameter, by default the this Table's columns.  If any
+    # of the result_cols have the same name as an existing column, inherit
+    # that column's type and tolerance.  Also, set any instance variables that
+    # might have been set by a subclass instance.
+    def empty_dup(result_cols = nil)
+      result_cols ||= heads
+      result_types = types.select { |k,_v| result_cols.include?(k) }
+      result = Table.new(result_cols, **result_types)
+      tolerant_cols.each do |h|
+        result.tolerant_cols << h
+        result.column(h).tolerant = true
+      end
+      (instance_variables - result.instance_variables).each do |v|
+        result.instance_variable_set(instance_variable_get(v))
+      end
+      result
     end
 
     # :category: Constructors
 
     # Construct a Table from the contents of a CSV file named +fname+. Headers
     # will be taken from the first CSV row and converted to symbols.
-    def self.from_csv_file(fname, tolerant_columns: [])
+    def self.from_csv_file(fname, **types)
       File.open(fname, 'r') do |io|
-        from_csv_io(io, tolerant_columns: tolerant_columns)
+        from_csv_io(io, **types)
       end
     end
 
@@ -143,8 +162,8 @@ module FatTable
 
     # Construct a Table from a CSV string +str+, treated in the same manner as
     # the input from a CSV file in ::from_org_file.
-    def self.from_csv_string(str, tolerant_columns: [])
-      from_csv_io(StringIO.new(str), tolerant_columns: tolerant_columns)
+    def self.from_csv_string(str, **types)
+      from_csv_io(StringIO.new(str), **types)
     end
 
     # :category: Constructors
@@ -153,9 +172,9 @@ module FatTable
     # file named +fname+. Headers are taken from the first row if the second row
     # is an hrule. Otherwise, synthetic headers of the form +:col_1+, +:col_2+,
     # etc. are created.
-    def self.from_org_file(fname, tolerant_columns: [])
+    def self.from_org_file(fname, **types)
       File.open(fname, 'r') do |io|
-        from_org_io(io, tolerant_columns: tolerant_columns)
+        from_org_io(io, **types)
       end
     end
 
@@ -163,8 +182,8 @@ module FatTable
 
     # Construct a Table from a string +str+, treated in the same manner as the
     # contents of an org-mode file in ::from_org_file.
-    def self.from_org_string(str, tolerant_columns: [])
-      from_org_io(StringIO.new(str), tolerant_columns: tolerant_columns)
+    def self.from_org_string(str, **types)
+      from_org_io(StringIO.new(str), **types)
     end
 
     # :category: Constructors
@@ -183,8 +202,8 @@ module FatTable
     # :hlines no +) org-mode strips all hrules from the table; otherwise (+
     # HEADER: :hlines yes +) they are indicated with nil elements in the outer
     # array.
-    def self.from_aoa(aoa, hlines: false, tolerant_columns: [])
-      from_array_of_arrays(aoa, hlines: hlines, tolerant_columns: tolerant_columns)
+    def self.from_aoa(aoa, hlines: false, **types)
+      from_array_of_arrays(aoa, hlines: hlines, **types)
     end
 
     # :category: Constructors
@@ -194,9 +213,9 @@ module FatTable
     # keys, which, when converted to symbols will become the headers for the
     # Table. If hlines is set true, mark a group boundary whenever a nil, rather
     # than a hash appears in the outer array.
-    def self.from_aoh(aoh, hlines: false, tolerant_columns: [])
+    def self.from_aoh(aoh, hlines: false, **types)
       if aoh.first.respond_to?(:to_h)
-        from_array_of_hashes(aoh, hlines: hlines, tolerant_columns: tolerant_columns)
+        from_array_of_hashes(aoh, hlines: hlines, **types)
       else
         raise UserError,
               "Cannot initialize Table with an array of #{input[0].class}"
@@ -215,7 +234,7 @@ module FatTable
 
     # Construct a Table by running a SQL +query+ against the database set up
     # with FatTable.connect, with the rows of the query result as rows.
-    def self.from_sql(query, tolerant_columns: [])
+    def self.from_sql(query, **types)
       msg = 'FatTable.db must be set with FatTable.connect'
       raise UserError, msg if FatTable.db.nil?
 
@@ -232,13 +251,33 @@ module FatTable
     ############################################################################
 
     class << self
+      # Return [typ, tol] based on the type string, str.
+      def typ_tol(str)
+        tol = str ? str.match?(/~\s*\Z/) : false
+        typ =
+          case str
+          when /\A\s*num/i
+            'Numeric'
+          when /\A\s*boo/i
+            'Boolean'
+          when /\A\s*dat/i
+            'DateTime'
+          when /\A\s*str/i
+            'String'
+          else
+            'NilClass'
+          end
+        [typ, tol]
+      end
+
       private
 
       # Construct table from an array of hashes or an array of any object that
       # can respond to #to_h. If an array element is a nil, mark it as a group
       # boundary in the Table.
-      def from_array_of_hashes(hashes, hlines: false, tolerant_columns: [])
-        result = new(tolerant_columns: tolerant_columns)
+      def from_array_of_hashes(hashes, hlines: false, **types)
+        heads = hashes.first.keys
+        result = new(*heads, **types)
         hashes.each do |hsh|
           if hsh.nil?
             unless hlines
@@ -266,8 +305,7 @@ module FatTable
       # hlines are stripped from the table, otherwise (:hlines yes) they are
       # indicated with nil elements in the outer array as expected by this
       # method when hlines is set true.
-      def from_array_of_arrays(rows, hlines: false, tolerant_columns: [])
-        result = new(tolerant_columns: tolerant_columns)
+      def from_array_of_arrays(rows, hlines: false, **types)
         headers = []
         if !hlines
           # Take the first row as headers
@@ -286,6 +324,7 @@ module FatTable
           headers = (1..rows[0].size).to_a.map { |k| "col_#{k}".as_sym }
           first_data_row = 0
         end
+        result = new(*headers, **types)
         rows[first_data_row..-1].each do |row|
           if row.nil?
             unless hlines
@@ -303,8 +342,8 @@ module FatTable
         result
       end
 
-      def from_csv_io(io, tolerant_columns: [])
-        result = new(tolerant_columns: tolerant_columns)
+      def from_csv_io(io, **types)
+        result = new(**types)
         ::CSV.new(io, headers: true, header_converters: :symbol,
                   skip_blanks: true).each do |row|
           result << row.to_h
@@ -317,7 +356,7 @@ module FatTable
       # header row must be marked with an hline (i.e, a row that looks like
       # '|---+--...--|') and groups of rows may be marked with hlines to
       # indicate group boundaries.
-      def from_org_io(io, tolerant_columns: [])
+      def from_org_io(io, **types)
         table_re = /\A\s*\|/
         hrule_re = /\A\s*\|[-+]+/
         rows = []
@@ -352,7 +391,7 @@ module FatTable
             rows << line.split('|').map(&:clean)
           end
         end
-        from_array_of_arrays(rows, hlines: true, tolerant_columns: tolerant_columns)
+        from_array_of_arrays(rows, hlines: true, **types)
       end
     end
 
@@ -377,12 +416,23 @@ module FatTable
       column(key).type
     end
 
+    # Return the type of the Column with the given +key+ as its
+    # header as a String.
+    def types
+      result = {}
+      headers.each do |h|
+        result[h] = type(h)
+      end
+      result
+    end
+
     # :category: Attributes
 
     # Set the column type for Column with the given +key+ as a String type.
     def force_string!(*keys)
       keys.each do |h|
         raise UserError, "force_string!: #{h} not a column in table" unless column(h)
+
         column(h).force_string!
       end
       self
@@ -428,7 +478,7 @@ module FatTable
     # :category: Attributes
 
     # Return a Hash of the Table's Column header symbols to type strings.
-    def types
+    def col_types
       result = {}
       columns.each do |c|
         result[c.header] = c.type
@@ -445,11 +495,11 @@ module FatTable
 
     # :category: Attributes
 
-    # Return whether the column with the given head should be made tolerant.
+    # Return whether the column with the given head is supposed to be
+    # tolerant.  We can't just look up the Column because it may not be build
+    # yet, as when we do a row-by-row add.
     def tolerant_col?(h)
-      return true if tolerant_columns.include?(:'*')
-
-      tolerant_columns.include?(h)
+      tolerant_cols.include?(h.to_s.as_sym) || self.omni_tol
     end
 
     # :category: Attributes
@@ -809,6 +859,7 @@ module FatTable
       unless expr.is_a?(String)
         raise "must call FatTable::Table\#order_with with a single string expression"
       end
+
       rev = false
       if expr.match?(/\s*!\s*\z/)
         rev = true
@@ -921,8 +972,15 @@ module FatTable
                          before: before_hook,
                          after: after_hook)
       # Compute the new Table from this Table
-      result = empty_dup
+      result_cols =
+        if cols.include?(:omni)
+          (headers + new_cols.keys - [:omni])
+        else
+          (cols + new_cols.keys)
+        end
+      result = empty_dup(result_cols)
       normalize_boundaries
+
       rows.each_with_index do |old_row, old_k|
         # Set the group number in the before hook and run the hook with the
         # local variables set to the row before the new row is evaluated.
@@ -992,15 +1050,6 @@ module FatTable
     def where(expr)
       expr = expr.to_s
       result = empty_dup
-      headers.each do |h|
-        col =
-        if tolerant_col?(h)
-          Column.new(header: h, tolerant: true)
-        else
-          Column.new(header: h)
-        end
-        result.add_column(col)
-      end
       ev = Evaluator.new(ivars: { row: 0, group: 0 })
       rows.each_with_index do |row, k|
         grp = row_index_to_group_index(k)
@@ -1058,10 +1107,13 @@ module FatTable
     # boundaries of the constituent tables. Preserves and adjusts the group
     # boundaries of the constituent table.
     def union_all(other)
-      set_operation(other, :+,
-                    distinct: false,
-                    add_boundaries: true,
-                    inherit_boundaries: true)
+      set_operation(
+        other,
+        :+,
+        distinct: false,
+        add_boundaries: true,
+        inherit_boundaries: true
+      )
     end
 
     # :category: Operators
@@ -1453,7 +1505,8 @@ module FatTable
       groups = sorted_tab.rows.group_by do |r|
         group_cols.map { |k| r[k] }
       end
-      result = empty_dup
+      grp_types = types.select { |k, _v| group_cols.include?(k) }
+      result = Table.new(*group_cols, **grp_types)
       groups.each_pair do |_vals, grp_rows|
         result << row_from_group(grp_rows, group_cols, agg_cols)
       end

data/lib/fat_table/version.rb

@@ -2,5 +2,5 @@
 
 module FatTable
   # The current version of FatTable
-  VERSION = '0.7.0'
+  VERSION = '0.9.0'
 end

data/lib/fat_table.rb

@@ -61,22 +61,22 @@ module FatTable
 
   # Return an empty FatTable::Table object. You can use FatTable::Table#add_row
   # or FatTable::Table#add_column to populate the table with data.
-  def self.new(*args, tolerant_columns: [])
-    Table.new(*args, tolerant_columns: tolerant_columns)
+  def self.new(*args, **types)
+    Table.new(*args, **types)
   end
 
   # Construct a FatTable::Table from the contents of a CSV file given by the
   # file name +fname+. Headers will be taken from the first row and converted to
   # symbols.
-  def self.from_csv_file(fname, tolerant_columns: [])
-    Table.from_csv_file(fname, tolerant_columns: tolerant_columns)
+  def self.from_csv_file(fname, **types)
+    Table.from_csv_file(fname, **types)
   end
 
   # Construct a FatTable::Table from the string +str+, treated in the same
   # manner as if read the input from a CSV file. Headers will be taken from the
   # first row and converted to symbols.
-  def self.from_csv_string(str, tolerant_columns: [])
-    Table.from_csv_string(str, tolerant_columns: tolerant_columns)
+  def self.from_csv_string(str, **types)
+    Table.from_csv_string(str, **types)
   end
 
   # Construct a FatTable::Table from the first table found in the Emacs org-mode
@@ -84,8 +84,8 @@ module FatTable
   # is an hline. Otherwise, synthetic headers of the form +:col_1+, +:col_2+,
   # etc. are created. Any other hlines will be treated as marking a boundary in
   # the table.
-  def self.from_org_file(fname, tolerant_columns: [])
-    Table.from_org_file(fname, tolerant_columns: tolerant_columns)
+  def self.from_org_file(fname, **types)
+    Table.from_org_file(fname, **types)
   end
 
   # Construct a FatTable::Table from the first table found in the string +str+,
@@ -93,8 +93,8 @@ module FatTable
   # are taken from the first row if the second row is an hrule. Otherwise,
   # synthetic headers of the form :col_1, :col_2, etc. are created. Any other
   # hlines will be treated as marking a boundary in the table.
-  def self.from_org_string(str, tolerant_columns: [])
-    Table.from_org_string(str, tolerant_columns: tolerant_columns)
+  def self.from_org_string(str, **types)
+    Table.from_org_string(str, **types)
   end
 
   # Construct a FatTable::Table from the array of arrays +aoa+. By default, with
@@ -108,8 +108,8 @@ module FatTable
   # org-mode code blocks, by default (+:hlines no+) all hlines are stripped from
   # the table, otherwise (+:hlines yes+) they are indicated with nil elements in
   # the outer array.
-  def self.from_aoa(aoa, hlines: false, tolerant_columns: [])
-    Table.from_aoa(aoa, hlines: hlines, tolerant_columns: tolerant_columns)
+  def self.from_aoa(aoa, hlines: false, **types)
+    Table.from_aoa(aoa, hlines: hlines, **types)
   end
 
   # Construct a FatTable::Table from the array of hashes +aoh+, which can be an
@@ -117,8 +117,8 @@ module FatTable
   # interpret nil separators as marking boundaries in the new Table. All hashes
   # must have the same keys, which, converted to symbols, become the headers for
   # the new Table.
-  def self.from_aoh(aoh, hlines: false, tolerant_columns: [])
-    Table.from_aoh(aoh, hlines: hlines, tolerant_columns: tolerant_columns)
+  def self.from_aoh(aoh, hlines: false, **types)
+    Table.from_aoh(aoh, hlines: hlines, **types)
   end
 
   # Construct a FatTable::Table from another FatTable::Table. Inherit any group
@@ -130,8 +130,8 @@ module FatTable
   # Construct a Table by running a SQL query against the database set up with
   # FatTable.connect. Return the Table with the query results as rows and the
   # headers from the query, converted to symbols, as headers.
-  def self.from_sql(query, tolerant_columns: [])
-    Table.from_sql(query, tolerant_columns: tolerant_columns)
+  def self.from_sql(query, **types)
+    Table.from_sql(query, **types)
   end
 
   ########################################################################

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: fat_table
 version: !ruby/object:Gem::Version
-  version: 0.7.0
+  version: 0.9.0
 platform: ruby
 authors:
 - Daniel E. Doherty
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2023-04-02 00:00:00.000000000 Z
+date: 2023-05-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler