dreader 0.4.1 → 0.4.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2216ea348440adb3c2cff5236ae8b3a0d916e4576002b400b4d3121da650dc8e
-  data.tar.gz: 1d22f900bf6b578ddcbf28ab30cbf835cba6121dd6904f071690939ab7b84d5d
+  metadata.gz: 6d616b9bad780960c105a2c754e392ecf51c0cdaeeb6076c11b4042d7c40e414
+  data.tar.gz: 656507d726a22a24111fc239c0f640dedf90321aab2bcda74a73dc0cf730a83d
 SHA512:
-  metadata.gz: 15a938341dfe5075bad0255fe4c623220c2f4cbd6a3e629410df1f6da58e2554bf7e5b15f17b641a57de45b694254771b9e58911d9c324582f786c8604db35c4
-  data.tar.gz: f02e9653e465e0eedf1d0feac0aecfbb20d7ae5177b51956a917159d4405d786a049d9bb840b8520223c6aeb7239fc1f8b954ac80caa07a322fe5d06f2aeb487
+  metadata.gz: 9ee4f8c9367864ef01aea8d75240a3a93a7bdbcab2411b4e2d4f92df8dc4fa5840e812308a02b8c5da4afc791c358f5eff4849017f63a4d90011ebcdca24e217
+  data.tar.gz: 11571140e63afb0c52b33a010e701e754964150878cba7e395878c2cf2f59cc85eb6b493d5fd05f287d229ec807e8873a45aafe850be6fc30abcbbb2d1cc47fc

data/Changelog.org

@@ -1,3 +1,14 @@
+* Version 0.4.2
+** better error messages for process and check functions
+   dreader now captures exceptions raised by process and check and
+   prints and error message to stdout if an error is found.
+   the exception is then propagated in the standard way.
+** new method bulk_declare
+   bulk_declare allow to easily declare columns which don't need a
+   specific treatment
+** read will now complains if the argument passed is not a hash
+** virtualcols is now accessible (attr_reader)
+** fixed a bug with slice
 * Version 0.4.1
 ** fixed an issue with ~read~: it always required a hash as input
 ** changed syntax of ~debug~, which now accepts a hash as argument

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    dreader (0.4.1)
+    dreader (0.4.2)
       roo
 
 GEM

data/README.md

@@ -46,7 +46,7 @@ Or install it yourself as:
 
 ## Usage
 
-### Declare the file you want to read
+### Declare what file you want to read
 
 Require `dreader` and declare an instance of the `Dreader::Engine` class:
 
@@ -137,6 +137,44 @@ end
 # we are done with our declarations)
 ```
 
+If there are different columns that you want to read and process in
+the same way, you can use the method `bulk_declare`, which accepts a
+hash as input.
+
+For instance:
+
+```ruby
+i.bulk_declare {a: 'A', b: 'B'}
+```
+
+is equivalent to:
+
+```ruby
+i.column :a do
+  colref 'A'
+end
+
+i.column :b do
+  colref 'B'
+end
+```
+
+The method also accepts a code block, which allows to define a common
+`process` function for all columns.  In case, **don't forget to put
+the hash in parentheses, or the Ruby parser won't be able to
+distinguish the hash from the code block.** For instance:
+
+```ruby
+i.bulk_declare({a: 'A', b: 'B'}) do
+  process do |cell|
+    ...
+  end
+end
+```
+
+There is an example of `bulk_declare` in the examples directory:
+([us_cities_bulk_declare.rb](examples/wikipedia_us_cities/us_cities_bulk_declare.rb)).
+
 **Remarks:**
 
 1. the column name can be anything ruby can use as a Hash key.  You
@@ -146,7 +184,7 @@ end
 2. `colref` can be a string (e.g., `'A'`) or an integer, in which case
    the first column is one
  
-3. you need to declare only the columns you want to import.  For
+3. **you need to declare only the columns you want to import.**  For
    instance, we could skip the declaration for column 1, if 'Date of
    Birth' is the only data we want to import
 
@@ -215,9 +253,6 @@ into a `@table` instance variable.
 i.read
 ```
 
-**Read applies all the `column` and `virtual_column` declarations and
-builds a hash with the data read.**
-
 After reading the file we can use `errors` to see whether any of the
 `check` functions failed:
 
@@ -228,6 +263,13 @@ array_of_strings ech do |error_line|
 end
 ```
 
+We can then use `virtual_columns` to process data and generate the
+virtual columns:
+
+```ruby
+i.virtual_columns
+```
+
 Finally we can use the `process` function to execute the `mapping`
 directive to each line read from the file.
 
@@ -290,13 +332,11 @@ i.table
     age:  { value: 31, row_number: 2, col_number: 2, errors: nil } } ]
 ```
 
-## Simplifying the data read
+## Simplifying the hash with the data read
 
 The `Dreader::Util` class provides some functions to simplify and
 restructure the hashes built by `dreader`.
 
-More in details:
-
 `Dreader::Util.simplify hash` simplifies the hash passed as input by
 removing all information but the value and making the value
 accessible directly from the name of the column.
@@ -309,14 +349,35 @@ Dreader::Util.simplify i.table[0]
 `Dreader::Util.slice hash, keys` and `Dreader::Util.slice hash,
 keys`, where `keys` is an arrays of keys, are respectively used to
 select or remove some keys from `hash`.
+
+```ruby
+i.table[0]
+{ name: { value: "John", row_number: 1, col_number: 1, errors: nil },
+  age:  { value: 30, row_number: 1, col_number: 2, errors: nil }}
+
+Dreader::Util.slice i.table[0], :name
+{name: { value: "John", row_number: 1, col_number: 1, errors: nil}
+
+Dreader::Util.clean i.table[0], :name
+{age:  { value: 30, row_number: 1, col_number: 2, errors: nil }
+```  
+
+The methods `slice` and `clean` are more useful when used in
+conjuction with `simplify`:
   
 ```ruby
-Dreader::Util.slice i.table[0], [:age]
+hash = Dreader::Util.simplify i.table[0]
+{name: "John", age: 30}
+
+Dreader::Util.slice hash, [:age]
 {age: 30}
   
-Dreader::Util.clean i.table[0], [:age]
+Dreader::Util.clean hash, [:age]
 {name: "John"}
 ```
+
+Notice that the output produced by `slice` and `simplify` is a has
+which can be used to create an `ActiveRecord` object.
     
 Finally, the `Dreader::Util.restructure` method helps building hashes
 to create

data/examples/wikipedia_us_cities/us_cities_bulk_declare.rb

@@ -0,0 +1,85 @@
+require 'dreader'
+
+# this is the class which will contain all the data we read from the file
+class City
+  [:city, :state, :population, :lat, :lon].each do |var|
+    attr_accessor var
+  end
+
+  def initialize(hash)
+    hash.each do |k, v|
+      self.send("#{k}=", v)
+    end
+  end
+end
+
+importer = Dreader::Engine.new
+
+# read from us_cities.tsv, lines from 2 to 10 (included)
+importer.options do
+  filename "us_cities.tsv"
+  first_row 2
+  last_row  10
+end
+
+# these are the columns for which we only need to specify column and name
+importer.bulk_declare ({city: 2, state: 3, latlon: 11}) do
+  process { |val| val.strip }
+end
+
+# the population column requires more work
+importer.column :population do |col|
+  col.colref 4
+
+  # make "3,000" into 3000 (int)
+  col.process do |value|
+    value.gsub(",", "").to_i
+  end
+
+  col.check do |value|
+    value > 0
+  end
+
+end
+
+cities = []
+
+importer.mapping do |row|
+  # remove all additional information stored in each cell
+  r = Dreader::Util.simplify row
+
+  # make latlon into the lat, lon fields
+  r[:lat], r[:lon] = r[:latlon].split(" ")
+
+  # now r contains something like
+  # {lat: ..., lon: ..., city: ..., state: ..., population: ..., latlon: ...}
+
+  # remove fields which are not understood by the Cities class and
+  # make a new instance
+  cleaned = Dreader::Util.clean r, [:latlon]
+
+  # you must declare an array cities before calling importer.process
+  cities << City.new(cleaned)
+end
+
+# print to stdout what we told dreader to read
+# (useful only for ... debugging!)
+importer.debug n: 10
+
+# check some other features of debug:
+# disable processing and debug (e.g., to analyze the raw data read)
+importer.debug process: false, check: false
+
+# load and process
+importer.load
+cities = []
+importer.process
+
+# output everything to see whether it works
+puts "First ten cities in the US (source Wikipedia)"
+cities.each do |city|
+  [:city, :state, :population, :lat, :lon].each do |var|
+    puts "#{var.to_s.capitalize}: #{city.send(var)}"
+  end
+  puts ""
+end

data/lib/dreader/version.rb

@@ -1,3 +1,3 @@
 module Dreader
-  VERSION = "0.4.1"
+  VERSION = "0.4.2"
 end

data/lib/dreader.rb

@@ -70,8 +70,9 @@ module Dreader
     end
 
     # an alias for Hash.slice
-    def self.slice hash, *keys 
-      hash.slice keys
+    # keys is an array of keys
+    def self.slice hash, keys 
+      hash.slice *keys
     end
 
     # remove all `keys` from `hash`
@@ -102,9 +103,11 @@ module Dreader
     attr_reader :options
     # the specification of the columns to process
     attr_reader :colspec
+    # the specification of the virtual columns
+    attr_reader :virtualcols
     # the data we read
     attr_reader :table
-
+    
     def initialize
       @options = {}
       @colspec = []
@@ -133,6 +136,51 @@ module Dreader
       @colspec << column.to_hash.merge({name: name})
     end
 
+    # bulk declare columns we intend to read
+    #
+    # - hash is a hash in the form { symbolic_name: colref }
+    #
+    # i.bulk_declare {name: 'B', age: 'C'} is equivalent to:
+    # 
+    # i.column :name do
+    #   colref 'B'
+    # end
+    # i.column :age do
+    #   colref 'C'
+    # end
+    #
+    # i.bulk_declare {name: 'B', age: 'C'} do
+    #   process do |cell|
+    #     cell.strip
+    #   end
+    # end
+    #
+    # is equivalent to:
+    #
+    # i.column :name do
+    #   colref 'B'
+    #   process do |cell|
+    #     cell.strip
+    #   end
+    # end
+    # i.column :age do
+    #   colref 'C'
+    #   process do |cell|
+    #     cell.strip
+    #   end
+    # end
+    def bulk_declare hash, &block 
+      hash.keys.each do |key|
+        column = Column.new
+        column.colref hash[key]
+        if block
+          column.instance_eval(&block)
+        end
+        @colspec << column.to_hash.merge({name: key})
+      end
+    end
+
+
     # virtual columns define derived attributes
     # the code specified in the virtual column is executed after reading
     # a row and before applying the mapping function
@@ -165,7 +213,12 @@ module Dreader
     # @return the data read from filename, in the form of an array of
     #         hashes
     def read args = {}
-      hash = @options.merge(args)
+      if args.class == Hash
+        hash = @options.merge(args)
+      else
+        puts "dreader error at #{__callee__}: this function takes a Hash as input"
+        exit
+      end
 
       spreadsheet = Dreader::Engine.open_spreadsheet (hash[:filename])
       sheet = spreadsheet.sheet(hash[:sheet] || 0)
@@ -187,13 +240,23 @@ module Dreader
           r[colname][:row_number] = row_number
           r[colname][:col_number] = colspec[:colref]
 
-          r[colname][:value] = value = colspec[:process] ? colspec[:process].call(cell) : cell
+          begin
+            r[colname][:value] = value = colspec[:process] ? colspec[:process].call(cell) : cell
+          rescue => e
+            puts "dreader error at #{__callee__}: 'process' specification for :#{colname} raised an exception at row #{row_number} (col #{index + 1}, value: #{cell})"
+            raise e
+          end
 
-          if colspec[:check] and not colspec[:check].call(value) then
-            r[colname][:error] = true
-            @errors << "Error: value \"#{cell}\" for #{colname} at row #{row_number} (col #{index + 1}) does not pass the check function"
-          else
-            r[colname][:error] = false
+          begin
+            if colspec[:check] and not colspec[:check].call(value) then
+              r[colname][:error] = true
+              @errors << "dreader error at #{__callee__}: value \"#{cell}\" for #{colname} at row #{row_number} (col #{index + 1}) does not pass the check function"
+            else
+              r[colname][:error] = false
+            end
+          rescue => e
+            puts "dreader error at #{__callee__}: 'check' specification for :#{colname} raised an exception at row #{row_number} (col #{index + 1}, value: #{cell})"
+            raise e
           end
         end
 
@@ -205,10 +268,34 @@ module Dreader
 
     alias_method :load, :read
 
+    # get (processed) row number
+    # 
+    # - row_number is the row to get: index starts at 1.
+    #
+    # get_row(1) get the first line read, that is, the row specified
+    # by `first_row` in `options` (or in read)
+    #
+    # You need to invoke read first
+    def get_row row_number
+      if row_number > @table.size
+        puts "dreader error at #{__callee__}: 'row_number' is out of range (did you invoke read first?)"
+        exit
+      elsif row_number <= 0
+        puts "dreader error at #{__callee__}: 'row_number' is zero or negative (first row is 1)."
+      else
+        @table[row_number - 1]
+      end
+    end
+
     # show to stdout the first `n` records we read from the file given the current
     # configuration
     def debug args = {}
-      hash = @options.merge(args)
+      if args.class == Hash
+        hash = @options.merge(args)
+      else
+        puts "dreader error at #{__callee__}: this function takes a Hash as input"
+        exit
+      end
 
       # apply some defaults, if not defined in the options
       hash[:process] = true if not hash.has_key? :process # shall we apply the process function?
@@ -246,13 +333,23 @@ module Dreader
           checked_str = ""
           
           if hash[:process]
-            processed = colspec[:process] ? colspec[:process].call(cell) : cell
-            processed_str = "processed: '#{processed}' (#{processed.class})"
+            begin
+              processed = colspec[:process] ? colspec[:process].call(cell) : cell
+              processed_str = "processed: '#{processed}' (#{processed.class})"
+            rescue => e
+              puts "dreader error at #{__callee__}: 'check' specification for :#{colname} raised an exception at row #{row_number} (col #{index + 1}, value: #{cell})"
+              raise e
+            end
           end
           if hash[:check]
-            processed = colspec[:process] ? colspec[:process].call(cell) : cell
-            check = colspec[:check] ? colspec[:check].call(processed) : "no check specified"
-            checked_str = "checked: '#{check}'"
+            begin
+              processed = colspec[:process] ? colspec[:process].call(cell) : cell
+              check = colspec[:check] ? colspec[:check].call(processed) : "no check specified"
+              checked_str = "checked: '#{check}'"
+            rescue => e
+              puts "dreader error at #{__callee__}: 'check' specification for #{colname} at row #{row_number} raised an exception (col #{index + 1}, value: #{cell})"
+              raise e
+            end
           end
 
           puts "  #{colname} => orig: '#{cell}' (#{cell.class}) #{processed_str} #{checked_str} (column: '#{colspec[:colref]}')"
@@ -268,13 +365,18 @@ module Dreader
 
     def virtual_columns
       # execute the virtual column specification
-      @virtualcols.each do |virtualcol|
-        @table.each do |r|
-          # add the cell to the table
-          r[virtualcol[:name]] = {
-            value: virtualcol[:process].call(r),
-            virtual: true,
-          }
+      @table.each do |r|
+        @virtualcols.each do |virtualcol|
+          begin
+            # add the cell to the table
+            r[virtualcol[:name]] = {
+              value: virtualcol[:process].call(r),
+              virtual: true,
+            }
+          rescue => e
+            puts "dreader error at #{__callee__}: 'process' specification for :#{virtualcol[:name]} raised an exception at row #{r[r.keys.first][:row_number]}"
+            raise e
+          end
         end
       end
     end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: dreader
 version: !ruby/object:Gem::Version
-  version: 0.4.1
+  version: 0.4.2
 platform: ruby
 authors:
 - Adolfo Villafiorita
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2018-04-11 00:00:00.000000000 Z
+date: 2018-05-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -84,6 +84,7 @@ files:
 - examples/wikipedia_big_us_cities/cities_by_state.ods
 - examples/wikipedia_us_cities/us_cities.rb
 - examples/wikipedia_us_cities/us_cities.tsv
+- examples/wikipedia_us_cities/us_cities_bulk_declare.rb
 - lib/dreader.rb
 - lib/dreader/version.rb
 homepage: http://github.com/avillafiorita/dreader