dreader 0.2.1 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: eff0a237ea9c0a4162696790f1689a868fc8a3efb3075eb0c890ddcc4651b5d0
-  data.tar.gz: f3ef74d2063aa07e5f42e3569563b37837ee3fa8d83e243a546d49d6699ef0ae
+  metadata.gz: b4e929eff1813efc3d2021430773275b213146e00adb01c5ad620bb3a6dfb98b
+  data.tar.gz: 3a9d4ecd7dce0713b29ef54550eb5a026176184798f7ae0bfbe1e634812e912a
 SHA512:
-  metadata.gz: 491a031d343211d988d7687601cacaeecac49770510940c56c46a770eddc26d703d8c1eb6e370deb79e0b536ecf2ebc8fbd5eac751f0354067c2242cb90f6268
-  data.tar.gz: 2ef1c5b00dadeba9f522f874022ec1e88e2409368144e056268155183f2cd5245f64b9e1445e44210cebbba01a24e409cc988651eddc8284080f85d1f216c394
+  metadata.gz: d31972c25fbc073211e9f398133b11d76dd71667cc4f5042a738d37304c9d0be8a88fd61fb446685c5867c9111ad81798d23b31b2017a396082352855a7af16f
+  data.tar.gz: c57d6b9e059c31262f728748c011a2631a91e64365ce72eea0554492bf7512db430dd9600dea80e64dbda50ec2256848cff2733e432b484e272458f6641bad2f

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    dreader (0.2.0)
+    dreader (0.2.1)
       roo
 
 GEM

data/README.md

@@ -46,6 +46,8 @@ Or install it yourself as:
 
 ## Usage
 
+### Declare the file you want to read
+
 Require `dreader` and declare an instance of the `Dreader::Engine` class:
 
 ```ruby
@@ -79,20 +81,28 @@ where:
 * (optional) `sheet` is the sheet name or number to read from. If not
   specified, the first (default) sheet is used
 
-Specify the structure of your file, for the columns you are interested
-to process.  You have to specify a name (used to access data) and a
-column reference (used to read data from the file).
+### Declare the columns you want to read
+
+Declare the columns you want to read by assigning them a name and a
+column reference:
+
+```ruby
+# we will access column A in Ruby code using :name
+i.column :name
+  colref 'A'
+end
+```
 
 You can also specify two ruby blocks, `process` and `check` to
 preprocess data and to check for errors.
 
 For instance, given the following file:
 
-| Name | Surname | Age |
-|------|---------|-----|
-| John | Doe     | 30  |
-| Jane | Doe     | 31  |
-| ...  | ...     | ... |
+| Name             | Date of birth   | 
+|------------------|-----------------|
+| Forest Whitaker  | July 15, 1961   |
+| Daniel Day-Lewis | April 29, 1957  |
+| Sean Penn        | August 17, 1960 |
 
 we could use the following declaration to specify the data to read:
 
@@ -106,19 +116,19 @@ i.column :name do
   end
 end
 
-# we want to access column 3 (Age) using :age
-# :age should be non nil and of length greater than 0
-i.column :age do
-  colref 3
+# we want to access column 2 (Date of birth) using :birthdate
+i.column :birthdate do
+  colref 2
   
   # make sure the column is transformed into an integer
   process do |x|
-    x.to_i
+    Date.parse(x)
   end
   
-  # check age is greater than zero
+  # check age is a date (check is invoked on the value returned
+  # by process)
   check do |x|
-    x > 0
+    x.class == Date
   end
 end 
 
@@ -126,13 +136,51 @@ end
 # we are done with our declarations)
 ```
 
-Notice that `colref` can be a string (e.g., `'A'`) or an integer
-(first column is one).
+**Remarks:**
+
+1. `colref` can be a string (e.g., `'A'`) or an integer, in which case
+   the first column is one
+ 
+2. you need to declare only the columns you want to import.  For
+   instance, we could skip the declaration for column 1, if 'Date of
+   Birth' is the only data we want to import
+
+3. If `process` and `check` are specified, then `check` will receive
+   the result of invoking `process` on the cell value.  This makes
+   sense if process is used to make the cell value more accessible to
+   ruby code (e.g., transforming a string into an integer).
+   
+
+### Add virtual columns, if you want
+
+Sometimes it is convenient to aggregate or otherwise manipulate the
+data read from each row before doing the actual processing.
+
+For instance, we might have a table with dates of birth, while we are
+really interested in the age of people.
+
+In such cases, we can use virtual column. A **virtual column** allows
+one to add a column to the data read.  The value of the column for
+each row is computed using the values of other cells.
+
+Virtual columns are declared similar to columns.  Thus, for instance,
+the following declaration adds an `age` column to each row of the data
+we read from the previous example:
+
+```ruby
+i.virtual_column :age do
+  process do |row|
+    # `compute_birthday` has to be defined
+    compute_birthday(row[:birthdate])
+  end
+end
+```
+
+Virtual columns are, of course, available to the `mapping` directive
+(see below).
 
-Notice also that if `process` and `check` are specified, then `check`
-will receive the result of invoking `process` on the cell value.  This
-makes sense if process is used to make the cell value more accessible
-to ruby code (e.g., transforming a string into an integer).
+
+### Specify how to process data
 
 Finally we can specify how we process lines, using the `mapping`
 directive.  Mapping takes an arbitrary piece of ruby code, which can
@@ -146,22 +194,26 @@ i.mapping do |row|
 end
 ```
 
-Notice that the data read from a line is stored in a hash which uses
-the column names and stores names in the `:value` key.
+Notice that the data read from each row of our input data is stored in
+a hash.  The hash uses column names as the primary key and stores
+the values in the `:value` key.
+
+### Start working with the data
 
-Now we are all set and we can start working with the data.
+We are now all set and we can start working with the data.
 
 First use `read` or `load` (synonyms), to read all data and put it
-into a `@table` instance variable.  This function uses the `column`
-declarations to read data and executes the `process` and `check`
-functions for each cell read.
+into a `@table` instance variable.
 
 ```ruby
 i.read
 ```
 
-We can now use `errors` to see whether any of the `check` functions
-failed:
+Read applies all the `column` and `virtual_column` declarations and
+buils a hash with the data read.
+
+After reading the file we can use `errors` to see whether any of the
+`check` functions failed:
 
 ```ruby
 array_of_strings = i.errors
@@ -170,29 +222,31 @@ array_of_strings ech do |error_line|
 end
 ```
 
-Now we can process the file with the `process` function, which
-executes the `mapping` directive for each line read from the file.
+Finally we can use the `process` function to execute the `mapping`
+directive to each line read from the file.
 
 ```ruby
 i.process
 ```
 
-If you need to perform more complex elaborations on the data, you can
-also directly access all data read, using the `table` method, which
-returns an array of hashes (see next section for the details).
+Look in the examples directory for further details and a couple of
+working examples.
 
-```ruby
-i.table
-```
 
-For further details and a couple of working examples, look in the
-examples directory.
+## Digging deeper
 
+If you need to perform more elaborations on the data which cannot be
+captured with `process` (that is, by processing the data row by row),
+you can also directly access all data read, using the `table` method:
 
-## Digging deeper
+```ruby
+i.read
+i.table
+# an array of hashes (one hash per row)
+```
 
-The `read` method fills a `@table` instance variable with an array of
-hashes.  Each hash represents a line of the file. 
+More in details, the `read` method fills a `@table` instance variable
+with an array of hashes.  Each hash represents a line of the file.
 
 Each hash contains one key per column, following your specification.
 Its value is, in turn, a hash with the following structure:
@@ -206,6 +260,9 @@ Its value is, in turn, a hash with the following structure:
 }
 ```
 
+(Note that virtual columns only store `value` and a Boolean `virtual`,
+which is always `true`.)
+
 Thus, for instance, given the example above:
 
 ```ruby
@@ -274,6 +331,7 @@ i.debug 40, filename # like above, but read from filename
 Another possibility is getting the value of the `@table` variable,
 which contains all the data read.
 
+
 ## Known Limitations
 
 At the moment:

data/examples/age/age.rb

@@ -0,0 +1,39 @@
+require 'dreader'
+
+i = Dreader::Engine.new
+
+i.options do
+  first_row 2
+end
+
+i.column :name do
+  colref 'A'
+end
+
+i.column :birthdate do
+  colref 'B'
+
+  process do |c|
+    Date.parse(c)
+  end
+end
+
+i.virtual_column :age do
+  process do |row|
+    birthdate = row[:birthdate][:value]
+    birthday = Date.new(Date.today.year, birthdate.month, birthdate.day)
+    today = Date.today
+
+    [0, today.year - birthdate.year - (birthday < today ? 1 : 0)].max
+  end
+end
+
+i.mapping do |row|
+  r = Dreader::Util.simplify(row)
+  puts "#{r[:name]} is #{r[:age]} years old (born on #{r[:birthdate]})"
+end
+
+i.read "Birthdays.ods"
+i.virtual_columns
+i.process
+

data/lib/dreader/version.rb

@@ -1,3 +1,3 @@
 module Dreader
-  VERSION = "0.2.1"
+  VERSION = "0.3.0"
 end

data/lib/dreader.rb

@@ -103,11 +103,12 @@ module Dreader
     # the specification of the columns to process
     attr_reader :colspec
     # the data we read
-    attr_reader :array
+    attr_reader :table
 
     def initialize
       @options = {}
       @colspec = []
+      @virtualcols = []
     end
 
     # define a DSL for options
@@ -132,6 +133,19 @@ module Dreader
       @colspec << column.to_hash.merge({name: name})
     end
 
+    # virtual columns define derived attributes
+    # the code specified in the virtual column is executed after reading
+    # a row and before applying the mapping function
+    #
+    # virtual colum declarations are executed in the order in which
+    # they are defined
+    def virtual_column name, &block
+      column = Column.new
+      column.instance_eval &block
+
+      @virtualcols << column.to_hash.merge({name: name})
+    end
+
     # define what we do with each line we read
     # - `block` is the code which takes as input a `row` and processes
     #   `row` is a hash in which each spreadsheet cell is accessible under
@@ -218,6 +232,19 @@ module Dreader
       @errors
     end
 
+    def virtual_columns
+      # execute the virtual column specification
+      @virtualcols.each do |virtualcol|
+        @table.each do |r|
+          # add the cell to the table
+          r[virtualcol[:name]] = {
+            value: virtualcol[:process].call(r),
+            virtual: true,
+          }
+        end
+      end
+    end
+
     # apply the mapping code to the array
     # it makes sense to invoke it only 
     def process

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: dreader
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.3.0
 platform: ruby
 authors:
 - Adolfo Villafiorita
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2018-03-23 00:00:00.000000000 Z
+date: 2018-03-26 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -77,6 +77,8 @@ files:
 - bin/console
 - bin/setup
 - dreader.gemspec
+- examples/age/Birthdays.ods
+- examples/age/age.rb
 - examples/wikipedia_big_us_cities/big_us_cities.rb
 - examples/wikipedia_big_us_cities/cities_by_state.ods
 - examples/wikipedia_us_cities/us_cities.rb