fat_table 0.5.3 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fa339300269582eedb57d26138bd3fa2de68d526ed4cc94e4c3a369a67989ddb
-  data.tar.gz: b35cc4785b208b670d6f039de8ef12351ea0533c37d8802fbf89f57763e63330
+  metadata.gz: f348d317c061becae08890ff02cc9c08789787d966f9ed88f3a4ef22b6eddb44
+  data.tar.gz: bf672b4f4f4ab07df465d06904ccebb9b48507177dc88a343eb09bfb0e98bc02
 SHA512:
-  metadata.gz: 49fbab273c609035f9c2a179b0b97cdca067a8a30db754916e7b17d03a435b4d3a4eefb89ddb326ef07c6e5b91632f764ce101de0acbc79fcf6bd876741da4cf
-  data.tar.gz: 76f87e5b342da743ea3e0c2000273de35d081ad4835c2423d6342278ebbced67d3a34bd11526c7f32185ac19e0b225356baa02fbcb8b7b630868739a944f0ecb
+  metadata.gz: c721233a04cc1664e8d8acea3cf677190aa47da30b3db7e1b9f6c13ddd56028a8ecd8b0c71915d0a1703f46fc9a22a1c3a8a3f13d6c80fb07964a22fd864ab7e
+  data.tar.gz: 7877d4529d8a6f2a13ee0a1518730791b4e5ed7dbadd2030367593c95ccdbd5159e711f7ffe69b9b2fa9b4812a6f49d3d1a9fce46642e4454f54ff40d769a4cb

data/README.org

@@ -31,8 +31,9 @@ The following is for org.
   "Current version is: #{FatTable::VERSION}"
 #+end_src
 
+#+RESULTS:
 #+begin_EXAMPLE
-Current version is: 0.5.3
+Current version is: 0.6.0
 #+end_EXAMPLE
 
 * Introduction
@@ -148,6 +149,7 @@ org-mode buffer as an org-table, ready for processing by other code blocks.
       - [[#type-and-column-priority][Type and Column priority]]
     - [[#footers][Footers]]
       - [[#adding-footers][Adding Footers]]
+      - [[#dynamic-labels][Dynamic Labels]]
       - [[#aggregators][Aggregators]]
       - [[#footer-objects][Footer objects]]
       - [[#footer-examples][Footer Examples]]
@@ -442,11 +444,26 @@ or nil. There are only five permissible types for a ~Column~:
 5. *NilClass* (for the undetermined column type).
 
 When a ~Table~ is constructed from an external source, all ~Columns~ start out
-having a type of ~NilClass~, that is, their type is as yet undetermined. When a
-string or object of one of the four determined types is added to a ~Column~, it
-fixes the type of the column and all further items added to the ~Column~ must
-either be ~nil~ (indicating no value) or be capable of being coerced to the
-column's type. Otherwise, ~FatTable~ raises an exception.
+having a type of ~NilClass~, that is, their type is as yet undetermined. When
+a string or object of one of the four determined types is added to a ~Column~
+and it can be converted into one of the permissible types, it fixes the type
+of the column, and all further items added to the ~Column~ must either be
+~nil~ (indicating no value) or be capable of being coerced to the column's
+type. Otherwise, ~FatTable~ raises an exception.
+
+The strictness of requiring all items to be of the same type can be relaxed by
+declaring a column to be "tolerant."  You can do so when you create the table
+by adding a tolerant_columns keyword parameter.  If a Column is tolerant,
+~FatTable~ tries to convert new items into a type other than a ~String~ and,
+if it can do so, sets /that/ as the Column's type.  Any later items that
+cannot be converted into the Column's type are converted to strings.  These
+interloper strings are treated like nils for purposes of sorting and
+evaluation, but are displayed according to any string formatting on output.
+See [[*Designating "Tolerant" Columns][Designating "Tolerant" Columns]] below.
+
+It is also possible to force ~FatTable~ to treat a column as a String type,
+even its items look like one of the other types.  See [[*Forcing String Type][Forcing String Type]]
+below.
 
 Items of input must be either one of the permissible ruby objects or strings. If
 they are strings, ~FatTable~ attempts to parse them as one of the permissible
@@ -519,6 +536,7 @@ You can create an empty table with ~FatTable::Table.new~ or, the shorter form,
 in the added rows determine the names of the headers:
 
 #+BEGIN_SRC ruby :results silent
+  require 'fat_table'
   tab = FatTable.new
   tab << { a: 1, b: 2, c: "<2017-01-21>", d: 'f', e: '' }
   tab << { a: 3.14, b: 2.17, c: '[2016-01-21 Thu]', d: 'Y', e: nil }
@@ -603,10 +621,18 @@ columns to be created:
 **** Forcing String Type
 Occasionally, ~FatTable~'s automatic type detection can get in the way and you
 just want it to treat one or more columns as Strings regardless of their
-appearance.  Think, for example, of zip codes.  At any time after creating a
-table, you can have it force the String type on any number of columns with the
-~force_string!~ method.  When you do so, all exisiting items in the column are
-converted to strings with the #to_s method.
+appearance.  Think, for example, of zip codes.  If headers are given when a
+table is contructed, you can designate a forced-string column by appending a
+~!~ to the end of the header.  It will not become part of the header, it will
+just mark it as a forced-string Column.
+
+#+begin_SRC emacs-lisp :wrap EXAMPLE
+  tab = FatTable.new(:a, 'b', 'C!', :d, :zip!)
+#+end_SRC
+
+In addition, at any time after creating a table, you can force the String type
+on any number of columns with the ~force_string!~ method.  When you do so, all
+exisiting items in the column are converted to strings with the #to_s method.
 
 #+begin_src ruby :wrap EXAMPLE
   tab = FatTable.new(:a, 'b', 'C', :d, :zip)
@@ -638,8 +664,11 @@ something that can't be parsed as a Numeric.  ~FatTable~ raises an exception
 is such cases, and that may be what you want if you can control the input.
 But, especially when you cannot do so, it can be helpful to designate one or
 more columns as "tolerant."  This means that when a conversion problem occurs,
-the column is forced to String type instead of throwing an exception, and the
-table can continue to be read.
+the column item is retained as a string type in a column that is otherwise of
+one of the types Numeric, DateTime, or Boolean.  Those string items are
+treated as nils for purposes of sorting or evaluation in a ~select~ method.
+When formatted, they participate in string formatting directive, but not those
+for other types.
 
 All of the table construction methods, allow a keyword parameter,
 ~tolerant_columns~, where you can designate what columns should be convert to
@@ -1728,8 +1757,6 @@ available in ~ft_console~ as ~@tab_a~ and ~@tab_b~:
     |  2 | Engineering |      2 |
     |  3 | Finance     |      7 |
     EOS
-
-  tab_b = FatTable.from_org_string(tab_b_str)
 #+END_SRC
 
 Here is ~tab_a~:
@@ -2439,7 +2466,6 @@ Finance&
  {:id=>"3", :dept=>"Finance", :emp_id=>"7"}]
 #+end_EXAMPLE
 
-
 *** Formatting Directives
 The formatting methods explained in the next section all take formatting
 directives as strings in which letters and other characters signify what
@@ -2641,9 +2667,10 @@ the table, but they can be overridden by more specific directives given in a
 ~format_for~ directive.
 
 **** Type and Column priority
-A directive based on type applies to all columns having that type unless
-overridden by a directive specific to a named column; a directive based on a
-column name applies only to cells in that column.
+A directive based the column name overrides any directive based on type.  If
+any cell has both a type-based formatting and column-based, the column
+instructions prevail.  In earlier versions the instuctions were "merged" but
+that is no longer the case.
 
 However, there is a twist.  Since the end result of formatting is to convert
 all columns to strings, the formatting directives for the ~String~ type can
@@ -2679,12 +2706,27 @@ but not for other nils, such as in the last row of the ~:join_date~ column.
 
 *** Footers
 **** Adding Footers
-You can call the ~footer,~ ~gfooter, foot, and gfoot~ methods on ~Formatter~
-objects to add footers and group footers. Note that all of these methods
-return a ~Footer~ object that can be accessed to extract the computed values.
-All of these methods return the ~FatTable::Footer~ object so constructed. It
-can be used to access the values and other attributes of the footer
-computed. Their signatures are:
+You can call the ~foot~, ~gfoot~, ~footer,~ or ~gfooter~, methods on
+~Formatter~ objects to add footers and group footers. Note that all of these
+methods return a ~Footer~ object that can be accessed to extract the computed
+values.  All of these methods return the ~FatTable::Footer~ object so
+constructed. It can be used to access the values and other attributes of the
+footer computed. Their signatures are:
+
+- ~foot(label: label, label_col: nil, **agg_cols)~ :: where ~label~ is a label
+  to be placed in the column with header ~label_col~, or, if ommitted, in the
+  first cell of the footer (unless that column is named as one of the
+  ~agg_cols~, in which case the label is ignored), and ~**agg_cols~ is zero or
+  more hash-like parameters with a column symbol as a key and a valid
+  aggregate as the value. This causes a table-wide header to be added at the
+  bottom of the table applying ~agg~, to the ~agg_cols~. A table can have any
+  number of footers attached, and they will appear at the bottom of the output
+  table in the order they are given.
+
+- ~gfoot(label: 'Group Total', label_col: nil, **agg_cols)~ :: where the
+  parameters have the same meaning as for the ~foot~ method, but results in a
+  footer for each group in the table rather than the table as a whole. These
+  will appear in the output table just below each group.
 
 - ~footer(label, *sum_cols, **agg_cols)~ :: where ~label~ is a label to be
   placed in the first cell of the footer (unless that column is named as one
@@ -2697,26 +2739,11 @@ computed. Their signatures are:
   number of footers attached, and they will appear at the bottom of the output
   table in the order they are given.
 
-- ~foot(label, label_col, **agg_cols)~ :: where ~label~ is a label to be
-  placed in the column with header ~label_col~, or, if ommitted, in the first
-  cell of the footer (unless that column is named as one of the ~agg_cols~, in
-  which case the label is ignored), and ~**agg_cols~ is zero or more hash-like
-  parameters with a column symbol as a key and a valid aggregate as the
-  value. This causes a table-wide header to be added at the bottom of the
-  table applying ~agg~, to the ~agg_cols~. A table can have any number of
-  footers attached, and they will appear at the bottom of the output table in
-  the order they are given.
-
 - ~gfooter(label, *sum_cols, **agg_cols)~ :: where the parameters have the
   same meaning as for the ~footer~ method, but results in a footer for each
   group in the table rather than the table as a whole. These will appear in
   the output table just below each group.
 
-- ~gfoot(label, label_col, **agg_cols)~ :: where the parameters have the same
-  meaning as for the ~foot~ method, but results in a footer for each group in
-  the table rather than the table as a whole. These will appear in the output
-  table just below each group.
-
 There are also a number of convenience methods for adding common footers:
 - ~sum_footer(*cols)~ :: Add a footer summing the given columns with the label
      'Total'.
@@ -2735,6 +2762,39 @@ There are also a number of convenience methods for adding common footers:
 - ~max_gfooter(*cols)~ :: Add a group footer showing the maximum for the given
      columns with the label 'Group Maximum'.
 
+**** Dynamic Labels
+Most of the time, you will want a fixed string as the label.  However,
+especially in the case of a group footer, you might want a dynamically
+contructed label.  You can use a proc or lambda for a label, and it will be
+computed for you.  In the case of non-group footers, the proc takes a single
+parameter, the footer object itself.  This allows you to make the label a
+function of other footer values, for example, you could make the label
+include the most recent year from the date column:
+
+#+begin_src ruby
+  fmtr.foot(label: -> (f) { "Average (latest year #{f.column(:date).max.year})" },
+  temp: :avg)
+#+end_src
+
+In the case of a group footer, the lambda or proc may take either one or qtwo parameters.
+If it takes one, the parameter is simply the 0-based number of the group:
+
+#+begin_src ruby
+  fmtr.gfoot(label: -> (k) { "Group #{(k+1).to_roman} Average" }, temp: :avg)
+#+end_src
+This would format the label with a roman numeral (assuming you defined a
+method to do so) for the group number.
+
+If it takes two arguments, the second argument is the footer itself, as with
+non-group footers:
+
+#+begin_src ruby
+    fmtr.gfoot(label: -> (k, f) { "Year #{f.column(:date, k).max.year} Group #{(k+1).to_roman} Average" },
+               temp: :avg)
+#+end_src
+This would add the group's year to label, assuming the :date column of the
+footer's table had the same year for each item in the group.
+
 **** Aggregators
 When adding a footer with the above methods, you can specify an aggregator for
 each column named in the ~agg_cols~ parameter.  There are several candidates
@@ -2751,6 +2811,9 @@ for what you can use for an aggregator:
     In the case of datetime columns, these aggrgators convert the dates to
     julian date numbers, perform the calculation, then convert the result back
     to a datetime object.
+    Apart from the built-in aggrgators, you could define your own by opening the
+    FatTable::Column class and adding a suitable instance method.  In that
+    case, the symbol could also refer to the method you defined.
 - String :: using a string as an aggrgegator can result in:
   + the string being converted to an object matching the type of the column
     (for example, using '$1,888' in a numeric column puts the constant number
@@ -2765,11 +2828,16 @@ for what you can use for an aggregator:
 - A Lambda :: finally, you can provide a lambda for performing arbitrary
   calculations and placing the result in the footer field.  The number of
   arguments the lambda takes can vary:
-  * If the lambda is used in a group footer, it must take a single integer
-    argument that is set to the group number being calculated and /can/ take a
-    second argument for the column symbol in which it appears, or
-  * If the lambda is used in an ordinary footer, it either takes no arguments,
-    or a single argument for the column symbol in which it appears.
+  * If the lambda is used in an ordinary footer column, it can take 0, 1, or 2
+    arguments: (1) the first argument, if given, will be set to the
+    FatTable::Column object for that column and (2) the second argument, if
+    given, will be set to the Footer object itself.
+  * If the lambda is used in a group footer column, it can 0, 1, 2, or 3
+    arguments: (1) the first argument, if given, will be set to the group's
+    0-based index number, (2) the second argument, if given, will be set to a
+    FatTable::Column object consisting of those items in the group's column,
+    and (3) the third argument, if given, will be set to the Footer object
+    itself.
 
 **** Footer objects
 Each of the methods for adding a footer to a ~Formatter~ returns a ~Footer~ object
@@ -2793,7 +2861,6 @@ their computed values.  Here are the accessors available on a
   to the footer value for that column, nil for unused columns.  Use the index
   ~k~ to specify which group to access in the case of a group footer.
 
-
 **** Footer Examples
 As a reminder, here is the table, ~tab_a~ defined earlier:
 
@@ -2954,18 +3021,20 @@ But it can be any type.  Here we pick a lottery winner from the employee ids.
 #+end_EXAMPLE
 
 ***** Lambdas
-Perhaps the most flexible form of aggregator is a lambda form.  They require 2
-or 3 parameters in non-group and group footers, respectively:
-
-- ~->(f, c) {...}~ :: in a normal, non-group footer, you must provide for two
-  paramters: the first, ~f~, will be bound to the footer in which the lambda
-  appears and the second, ~c~, will be bound to the column header to which the
-  lambda is attached.
-- ~->(f, c, k)~ :: in a group footer, you must provide for three paramters:
-  the first, ~f~, will be bound to the footer in which the lambda appears, the
-  second, ~c~, will be bound to the column header to which the lambda is
-  attached, and the third, ~k~ will be bound to the group number of the group
-  being evaluated.
+Perhaps the most flexible form of aggregator is a lambda form.  They can take
+up to 2 or up to 3 parameters in non-group and group footers, respectively:
+
+- ~->(c, f) {...}~ :: in a normal, non-group footer, you may provide for up to
+  two paramters: the first, ~c~, if given, will be bound to the column header
+  to which the lambda is attached and and the second, ~f~, if given will be
+  bound to the footer in which the lambda appears.  A lambda with no
+  parameters can be provided as well if none are needed.
+- ~->(k, c, f)~ :: in a group footer, you may provide for up to three
+  paramters: the the first, ~k~, if provided, will be bound to the group
+  number of the group being evaluated, the second, ~c~, if provided, will be
+  bound to the column header to which the lambda is attached, and the third,
+  ~f~, will be bound to the footer in which the lambda appears. A lambda with
+  no parameters can be provided as well if none are needed.
 
 With the first argument, the footer itself becomes available and with it all
 the things accessible with the footers, including the items in the current
@@ -2976,7 +3045,7 @@ Compute the summ of the squares if the items in the ~:age~ column:
   tab_a.to_text do |f|
     f.format(numeric: '0.0R,', datetime: 'd[%v]D[%v]')
     f.footer('Average', age: :avg, salary: :avg, join_date: :avg)
-    f.footer('SSQ', age: ->(f, c) { sa = f.items(c).map {|x| x * x}.sum; Math.sqrt(sa) })
+    f.footer('SSQ', age: ->(c) { sa = c.items.map {|x| x * x}.sum; Math.sqrt(sa) })
   end
 #+END_SRC
 
@@ -3005,8 +3074,8 @@ summ of the squares if the ages in each group:
   tab_a.order_with('join_date.year').to_text do |f|
     f.format(numeric: '0.0R,', datetime: 'd[%v]D[%v]', sort_key: '0.0~,')
     f.footer('Average', age: :avg, salary: :avg, join_date: :avg)
-    f.gfooter('Group SSQ', age: ->(f, c, k) { sa = f.items(c, k).map {|x| x * x}.sum; Math.sqrt(sa) })
-    f.footer('Total SSQ', age: ->(f, c) { sa = f.items(c).map {|x| x * x}.sum; Math.sqrt(sa) })
+    f.gfooter('Group SSQ', age: ->(k, c, f) { sa = c.items.map {|x| x * x}.sum; Math.sqrt(sa) })
+    f.footer('Total SSQ', age: ->(c, f) { sa = c.items.map {|x| x * x}.sum; Math.sqrt(sa) })
   end
 #+END_SRC
 
@@ -3018,19 +3087,19 @@ summ of the squares if the ages in each group:
 +-----------+-------+-----+------------+--------+-------------+----------+
 | Group SSQ |       |  45 |            |        |             |          |
 +-----------+-------+-----+------------+--------+-------------+----------+
-|         1 | Paul  |  32 | California | 20,000 | 13-JUL-2001 |     2001 |
+|         1 | Paul  |  32 | California | 20,000 | 13-JUL-2001 | 2001     |
 +-----------+-------+-----+------------+--------+-------------+----------+
 | Group SSQ |       |  32 |            |        |             |          |
 +-----------+-------+-----+------------+--------+-------------+----------+
-|         2 | Allen |  25 | Texas      |        | 13-JUL-2005 |     2005 |
-|         8 | Paul  |  24 | Houston    | 20,000 | 13-JUL-2005 |     2005 |
-|         9 | James |  44 | Norway     |  5,000 | 13-JUL-2005 |     2005 |
+|         2 | Allen |  25 | Texas      |        | 13-JUL-2005 | 2005     |
+|         8 | Paul  |  24 | Houston    | 20,000 | 13-JUL-2005 | 2005     |
+|         9 | James |  44 | Norway     |  5,000 | 13-JUL-2005 | 2005     |
 +-----------+-------+-----+------------+--------+-------------+----------+
 | Group SSQ |       |  56 |            |        |             |          |
 +-----------+-------+-----+------------+--------+-------------+----------+
-|         3 | Teddy |  23 | Norway     | 20,000 | 13-DEC-2007 |     2007 |
-|         4 | Mark  |  25 | Rich-Mond  | 65,000 | 13-DEC-2007 |     2007 |
-|         5 | David |  27 | Texas      | 85,000 | 13-DEC-2007 |     2007 |
+|         3 | Teddy |  23 | Norway     | 20,000 | 13-DEC-2007 | 2007     |
+|         4 | Mark  |  25 | Rich-Mond  | 65,000 | 13-DEC-2007 | 2007     |
+|         5 | David |  27 | Texas      | 85,000 | 13-DEC-2007 | 2007     |
 +-----------+-------+-----+------------+--------+-------------+----------+
 | Group SSQ |       |  43 |            |        |             |          |
 +-----------+-------+-----+------------+--------+-------------+----------+

data/TODO.org

@@ -1,10 +1,37 @@
+
+* TODO Specify Column Widths
+Allow a formatter to specify column widths.  This could be a number of
+characters, which would be interpreted as a number of "ems" for LaTeX.
+Cell content larger than the width would be truncated.  Any column without a
+width specified would be set at the width of the longest value in that cell,
+after initial formatting.
+
+#+begin_SRC ruby
+  tab.to_text do |f|
+    f.widths(a: 13, b: 30)
+  end
+#+end_SRC
+
+Possible enhancements:
+- specify an overall width and column widths as decimal or fractions, so that
+  a column's width would be that fraction of the overall width.
+- specify a Range for a width, so that the column would at least min and at
+  most max, otherwise the width of its largest cell.
+
 * TODO Conversion to Spreadsheets
 - State "TODO"       from              [2017-04-21 Fri 10:36]
 This is a [[https://github.com/westonganger/spreadsheet_architect][gem]] that I can include into the Table model to convert a table into
 a spread-sheet, or even a sheet in a multi-sheet spreadsheet file.
 
-* TODO Add from_yql for fetching from Yahoo
+* TODO Add Quandl or EODDATA Queries
+Possible replacements for YQL.
+
+* CNCL Add from_yql for fetching from Yahoo
+CLOSED: [2022-01-30 Sun 06:03]
 - State "TODO"       from              [2017-04-21 Fri 10:35]
+
+Cancelled because Yahoo shut down the YQL api service.
+
 Add a constructor to allow fetching stock data from yql.  Perhaps grab all
 available fields, then allow a select of those of interest.
 

data/lib/ext/array.rb

@@ -12,4 +12,21 @@ class Array
       end
     end
   end
+
+  def filter_to_type(typ)
+    if typ == 'Boolean'
+      compact.select { |i| i.is_a?(TrueClass) || i.is_a?(FalseClass) }
+    elsif typ == 'DateTime'
+      compact.select { |i| i.is_a?(Date) || i.is_a?(DateTime) || i.is_a?(Time) }
+        .map { |i| i.to_datetime }
+    elsif typ == 'Numeric'
+      compact.select { |i| i.is_a?(Numeric) }
+    elsif typ == 'String'
+      map { |i| i.to_s }
+    elsif typ == 'NilClass'
+      self
+    else
+      raise ArgumentError, "cannot filter_to_type for type '#{typ}'"
+    end
+  end
 end