dreader 1.1.1 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 524e55af5bb94cae3f407a1602069549783e935798a638361f3c98e922ffc54d
-  data.tar.gz: 2599e048324ccd233e3fa4a0261e134ced0a3347d27af1e978e635639b6284a8
+  metadata.gz: 3c30be2fe49c6c8ce20d4930c75f1279ac1a92a099f609b1266b14dc61c7cf3c
+  data.tar.gz: 58c735a67c45ef11a180bc6f17892ba912656d346320da1a716caa69661f4695
 SHA512:
-  metadata.gz: e8a78531d96ef35f9272a38daa5327620026dd66c98529710901d4c5994b9e5369308612cc79d1e0998d46dbb9dd6d26c448ddc4265dd536f1e61a4fc50fb885
-  data.tar.gz: de71caed5d3df79d0d456b080a72a0edc035ef4e46906aac3f94295b07d57b956554a9d9e07b4b6195c5c09df0badfa9202122d7b6e0568cf89569b6a2277d28
+  metadata.gz: 7847892dbcf648432a9c51867fd70e1260e956e82ea7cbbad93f92882dd867be6f36f67a0edfbb972e16e12c1364efe92a54bd03d31801770da4b28fac725350
+  data.tar.gz: a717955a2eaa0c406d6fb140daf9cd084c11e0d4710289de34b7757b5fd4f4e920ded4fd1450306e3e42a42278c983e078ff8e53248fe0f5b7a93d09fb8a9d40

data/CHANGELOG.org

@@ -1,5 +1,20 @@
 #+TITLE: Changelog
 
+* Version 1.2.0 - <2023-11-02 Thu>
+** reject declaration
+
+   - A new reject declaration allows to reject some lines.  reject takes as
+     input a row and can predicate over columns and virtual columns.  When
+     true, the corresponding line is discarded.
+
+* Version 1.1.2 - <2023-10-31 Tue>
+** Fixes an issue with the :extension option
+
+   - Fixes a bug related to =:extension= and adds a working example, to test
+     the feature
+   - Changes the extension from a string to a symbol. No initial dot required
+     any longer
+
 * Version 1.1.1 - <2023-10-16 Mon>
 ** Adds option :extension
 

data/README.org

@@ -137,7 +137,8 @@ To write an import function with Dreader:
   and check parsed data
 - Add virtual columns, that is, columns computed from other values
   in the row
-- Specify how to map line. This is where you do the actual work
+- Specify what lines you want to reject, if any
+- Specify how to transform lines. This is where you do the actual work
   (for instance, if you process a file line by line) or put together data for
   processing after the file has been fully read --- see the next step.
 
@@ -165,12 +166,13 @@ Require =dreader= and declare a class which extends =Dreader::Engine=:
   end
 #+END_EXAMPLE
 
-In the class specify parsing option, using the following syntax:
+Specify parsing option in the class, using the following syntax:
 
 #+BEGIN_EXAMPLE ruby
   options do
     filename 'example.ods'
-    extension ".ods"
+    # this optional. Use it when the file does not have an extension
+    extension :ods
 
     sheet 'Sheet 1'
 
@@ -190,10 +192,10 @@ where:
   to supply a filename when loading the file (see =read=, below).  *Use
   =.tsv= for tab-separated files.* 
 - (optional) =extension= overrides or specify the extension of =filename=.
-  Takes as input the extension preceded by a "." (e.g., ".xlsx").  Notice that
-  **value of this option is not appended to filename** (see =read= below).
-  Filename must thus be a valid reference to a file in the file system. This
-  option is useful in one of these two circumstances:
+  Takes as input a symbol (e.g., =:xlsx=).  
+  Notice that **value of this option is not appended to filename** (see =read=
+  below).  Filename must thus be a valid reference to a file in the file
+  system. This option is useful in one of these two circumstances:
   1. When =filename= has no extension
   2. When you want to override the extension of the filename, e.g., to force
      reading a "file.csv" as a tab separated file
@@ -397,6 +399,9 @@ See [[file:examples/wikipedia_us_cities/us_cities_bulk_declare.rb][us_cities_bul
   hash from the code block.
 #+END_NOTES
 
+The data read from each row of our input data is stored in a hash. The hash
+uses column names as the primary key and stores the values in the =:value=
+key.
 
 *** Add virtual columns
 
@@ -426,6 +431,22 @@ Virtual columns are, of course, available to the =mapping= directive
 (see below).
 
 
+*** Specify which lines to reject
+
+You can reject some lines using the =reject= declaration, which is applied row
+by row, can predicate over columns and virtual columns, and has to return a
+Boolean value.
+
+All lines returning a truish value will be be rejected, that is, not stored in
+the =@table= variable (and, consequently, passed to the mapping function).
+
+For instance, the following declaration rejects all lines in which the
+population column is higher than =3_000_000=:
+
+#+begin_src ruby
+  reject { |row| row[:population][:value] > 3_000_000 }
+#+end_src
+
 *** Specify how to process each line
 
 The =mapping= directive specifies what to do with each line read.  The
@@ -441,10 +462,9 @@ value of column =:age= and prints them to standard output
   end
 #+END_EXAMPLE
 
-The data read from each row of our input data is stored in a hash. The hash
-uses column names as the primary key and stores the values in the =:value=
-key.
-
+To invoke the =mapping= declaration on a file, use the =mappings= method,
+which invokes =map= to each row and it stores in the =@table= variable
+whatever value mapping returns.
 
 *** Process data
 
@@ -464,8 +484,8 @@ A typical scenario works as follows:
   # examples:
   # i.read
   # i.read filename: "example.ods"
-  # i.read filename: "example.ods", extension: ".ods"
-  # i.read filename: "example", extension: ".ods" 
+  # i.read filename: "example.ods", extension: :ods
+  # i.read filename: "example", extension: :ods
   # (the line above opens the file "example" as an Open Document Spreasdheet)
   i.read
   
@@ -500,7 +520,13 @@ A typical scenario works as follows:
 (Optionally: check again for errors.)
 
 5. Add your own code to process the data returned after =mappings=, which you
-   can access with =i.table= or =i.data= (synonyms).
+   can assign to a variable (e.g., =returned_data = i.mappings=) or access
+   with =i.table= or =i.data= (synonyms).
+
+#+begin_quote
+Notice that =mappings= does a side effect and invoking the mapping twice in a
+row won't work: you need to reload the file first.
+#+end_quote
 
 Look in the examples directory for further details and a couple of working
 examples.

data/examples/age_csv/Birthdays-TabSeparated.csv

@@ -0,0 +1,13 @@
+Name	Date of birth
+Forest Whitaker	July 15, 1961
+Daniel Day-Lewis	April 29, 1957
+Sean Penn	August 17, 1960
+Jeff Bridges	December 4, 1949
+Colin Firth	September 10, 1960
+Jean Dujardin	June 19, 1972
+Daniel Day-Lewis	April 29, 1957
+Matthew McConaughey	November 4, 1969
+Eddie Redmayne	January 6, 1982
+Leonardo DiCaprio	November 11, 1974
+Casey Affleck	August 12, 1975
+Gary Oldman	March 21, 1958

data/examples/age_csv/Birthdays.csv

@@ -0,0 +1,13 @@
+Name,Date of birth
+Forest Whitaker,"July 15, 1961"
+Daniel Day-Lewis,"April 29, 1957"
+Sean Penn,"August 17, 1960"
+Jeff Bridges,"December 4, 1949"
+Colin Firth,"September 10, 1960"
+Jean Dujardin,"June 19, 1972"
+Daniel Day-Lewis,"April 29, 1957"
+Matthew McConaughey,"November 4, 1969"
+Eddie Redmayne,"January 6, 1982"
+Leonardo DiCaprio,"November 11, 1974"
+Casey Affleck,"August 12, 1975"
+Gary Oldman,"March 21, 1958"

data/examples/age_csv/age.rb

@@ -0,0 +1,55 @@
+require "dreader"
+
+class Reader
+  extend Dreader::Engine
+
+  options do
+    first_row 2
+    debug true
+  end
+
+  column :name do
+    doc "A is the name string"
+    colref 'A'
+  end
+
+  column :birthdate do
+    doc "Birthdate contains a full date (i.e., including the year)"
+    colref 'B'
+
+    process do |c|
+      Date.parse(c)
+    end
+  end
+
+  virtual_column :age do
+    process do |row|
+      birthdate = row[:birthdate][:value]
+      birthday = Date.new(Date.today.year, birthdate.month, birthdate.day)
+      today = Date.today
+
+      [0, today.year - birthdate.year - (birthday < today ? 1 : 0)].max
+    end
+  end
+
+  mapping do |row|
+    r = Dreader::Util.simplify(row)
+    puts "#{r[:name]} is #{r[:age]} years old (born on #{r[:birthdate]})"
+  end
+end
+
+i = Reader
+i.read filename: "Birthdays.csv", mapping: true
+
+i.read filename: "Birthdays-TabSeparated.csv", extension: :tsv, mapping: true
+
+#
+# Here we can do further processing on the data
+#
+File.open("ages.txt", "w") do |file|
+  i.table.each do |row|
+    unless row[:row_errors].any?
+      file.puts "#{row[:name][:value]} #{row[:age][:value]}"
+    end
+  end
+end

data/examples/age_noext/age.rb

@@ -0,0 +1,73 @@
+require "dreader"
+
+class Reader
+  extend Dreader::Engine
+
+  options do
+    first_row 2
+    debug true
+    extension :ods
+  end
+
+  column :name do
+    doc "A is the name string"
+    colref 'A'
+  end
+
+  column :birthdate do
+    doc "Birthdate contains a full date (i.e., including the year)"
+    colref 'B'
+
+    process do |c|
+      Date.parse(c)
+    end
+  end
+
+  virtual_column :age do
+    process do |row|
+      birthdate = row[:birthdate][:value]
+      birthday = Date.new(Date.today.year, birthdate.month, birthdate.day)
+      today = Date.today
+
+      [0, today.year - birthdate.year - (birthday < today ? 1 : 0)].max
+    end
+  end
+
+  mapping do |row|
+    r = Dreader::Util.simplify(row)
+    puts "#{r[:name]} is #{r[:age]} years old (born on #{r[:birthdate]})"
+  end
+end
+
+puts
+puts "*****************************************************************"
+puts "Reading ODS with no extension, using extension set in the options"
+puts "*****************************************************************"
+puts
+
+i = Reader
+i.read filename: "Birthdays"
+i.virtual_columns
+i.mappings
+
+puts
+puts "*****************************************************************"
+puts "Reading XLSX with wrong extension, overriding existing extension"
+puts "*****************************************************************"
+puts
+
+i = Reader
+i.read filename: "Birthdays-xlsx-with-wrong-extension.xls", extension: :xlsx
+i.virtual_columns
+i.mappings
+
+puts
+puts "*****************************************************************"
+puts "Reading XLSX with no extension"
+puts "*****************************************************************"
+puts
+
+i = Reader
+i.read filename: "Birthdays-xlsx", extension: :xlsx
+i.virtual_columns
+i.mappings

data/examples/wikipedia_us_cities/us_cities_reject.rb

@@ -0,0 +1,77 @@
+require 'dreader'
+
+# this is the class which will contain all the data we read from the file
+class City
+  [:city, :state, :population, :lat, :lon].each do |var|
+    attr_accessor var
+  end
+
+  def initialize(hash)
+    hash.each do |k, v|
+      self.send("#{k}=", v)
+    end
+  end
+end
+
+class Importer
+  extend Dreader::Engine
+
+  # read from us_cities.tsv, lines from 2 to 10 (included)
+  options do
+    filename "us_cities.tsv"
+    first_row 2
+    last_row  307
+  end
+
+  # these are the columns for which we only need to specify column and name
+  columns ({city: 2, state: 3, latlon: 11}) do
+    process { |val| val.strip }
+  end
+
+  # the population column requires more work
+  column :population do |col|
+    col.colref 4
+
+    # make "3,000" into 3000 (int)
+    col.process { |value| value.gsub(",", "").to_i }
+
+    # check population is positive
+    col.check { |value| value > 0 }
+  end
+
+  # reject all cities with more than 3M people
+  reject do |row|
+    row[:population][:value] >= 3_000_000
+  end
+
+  mapping do |row|
+    # remove all additional information stored in each cell
+    r = Dreader::Util.simplify row
+
+    # make latlon into the lat, lon fields
+    r[:lat], r[:lon] = r[:latlon].split(" ")
+
+    # now r contains something like
+    # {lat: ..., lon: ..., city: ..., state: ..., population: ..., latlon: ...}
+
+    # remove fields which are not understood by the Cities class and
+    # make a new instance
+    cleaned = Dreader::Util.clean r, [:latlon]
+
+    # you must declare an array cities before calling importer.mapping
+    City.new(cleaned)
+  end
+end
+
+# load and process
+importer = Importer
+importer.load mapping: true, debug: true
+
+# output everything to see whether it works
+puts "First ten cities in the US with less than 3M (source Wikipedia)"
+importer.table.each do |city|
+  [:city, :state, :population, :lat, :lon].each do |var|
+    puts "#{var.to_s.capitalize}: #{city.send(var)}"
+  end
+  puts ""
+end

data/lib/dreader/engine.rb

@@ -21,6 +21,8 @@ module Dreader
     attr_accessor :declared_virtual_columns
     # the mapping rules
     attr_accessor :declared_mapping
+    # the declared filter
+    attr_accessor :declared_reject
     
     # the data we read
     attr_reader :table
@@ -118,6 +120,11 @@ module Dreader
       @declared_virtual_columns << column.to_hash.merge({ name: name })
     end
 
+    # define a filter, which skips some rows
+    def reject(&block)
+      @declared_reject = block
+    end
+
     # define what we do with each line we read
     # - `block` is the code which takes as input a `row` and processes
     #   `row` is a hash in which each spreadsheet cell is accessible under
@@ -187,8 +194,13 @@ module Dreader
         # this has side-effects on r
         virtual_columns_on(r) if options[:virtual] || options[:mapping]
 
+        # check whether the filter would ignore this line
+        # notice that we need to invoke compact to avoid nil being added
+        # to the table
+        next if !options[:ignore_reject] && reject?(r)
+
         options[:mapping] ? mappings_on(r) : r
-      end
+      end.compact
     end
 
     # TODO: PASS A ROW (and not row_number and sheet)
@@ -268,6 +280,7 @@ module Dreader
 
     # Compute virtual columns for, with side effect on row
     def virtual_columns_on(row)
+      @declared_virtual_columns ||= []
       @declared_virtual_columns.each do |virtualcol|
         colname = virtualcol[:name]
         row[colname] = { virtual: true }
@@ -291,13 +304,36 @@ module Dreader
       end
     end
 
-    # apply the mapping code to the array it makes sense to invoke it only
-    # once.
+    # check whether a line has to be rejected
+    def reject?(row)
+      rejected = @declared_reject&.call(row)
+      if rejected
+        @logger.debug "[dreader] row rejected by reject declaration #{row}"
+      end
+    end
+
+    # apply the mapping code to the @table.  Notice that we do a side effect
+    # on @table and, hence, invoking the mapping twice won't work (you need to
+    # reload first).
+    #
+    # the mapping is applied only if it defined and it returns the output of
+    # the mapping.
     #
-    # the mapping is applied only if it defined and it uses map, so that
-    # it can be used functionally
+    # notice also that we do a side-effect on @table.  This is to make the
+    # behavior of
+    #
+    #   i.load mapping: true
+    #   i.table
+    #
+    # and
+    #
+    #   i = load;
+    #   i.mappings
+    #   i.table
+    #
+    # the same
     def mappings
-      @table.map { |row| mappings_on(row) }
+      @table = @table.map { |row| mappings_on(row) }
     end
 
     def mappings_on(row)
@@ -398,28 +434,36 @@ module Dreader
     # list of keys we support in options. We remove them when reading
     # the CSV file
     OPTION_KEYS = %i[
-      filename sheet first_row last_row logger logger_level
+      filename extension sheet first_row last_row
+      logger logger_level
+      debug
     ]
 
     def open_spreadsheet(options)
       filename = options[:filename]
-      ext = options[:extension] || File.extname(filename)
+      # use the extension option or make ".CSV" into :csv
+      extension = options[:extension] || File.extname(filename).downcase[1..-1]&.to_sym
+
+      
+      # TODO: MAKE DEBUG AND LOGGER INTO REAL CLASS VARIABLES OR MAKE LOCAL AND/OR FUNCTIONS
+      @debug = @declared_options.merge(options)[:debug] == true
+      if @debug
+        @logger = options[:logger] || Logger.new($stdout)
+        @logger.debug "[dreader open_spreadsheet] filename: #{filename}"
+        @logger.debug "[dreader open_spreadsheet] extension: #{extension}"
+      end
 
-      case ext
-      when ".csv"
+      case extension
+      when :csv
         csv_options = @declared_options.except(*OPTION_KEYS)
         Roo::CSV.new(filename, csv_options:)
-      when ".tsv"
+      when :tsv
         csv_options = @declared_options.except(*OPTION_KEYS).merge({ col_sep: "\t" })
         Roo::CSV.new(filename, csv_options:)
-      when ".ods"
-        Roo::OpenOffice.new(filename)
-      when ".xls"
-        Roo::Excel.new(filename)
-      when ".xlsx"
-        Roo::Excelx.new(filename)
+      when :ods, :xls, :xlsx
+        Roo::Spreadsheet.open(filename, extension:)
       else
-        raise "Unknown extension: #{ext}"
+        raise "Unknown extension: #{ext}. Use the :extension option."
       end
     end
 

data/lib/dreader/version.rb

@@ -1,3 +1,3 @@
 module Dreader
-  VERSION = "1.1.1"
+  VERSION = "1.2.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: dreader
 version: !ruby/object:Gem::Version
-  version: 1.1.1
+  version: 1.2.0
 platform: ruby
 authors:
 - Adolfo Villafiorita
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2023-10-31 00:00:00.000000000 Z
+date: 2023-11-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: roo
@@ -108,6 +108,13 @@ files:
 - dreader.gemspec
 - examples/age/Birthdays.ods
 - examples/age/age.rb
+- examples/age_csv/Birthdays-TabSeparated.csv
+- examples/age_csv/Birthdays.csv
+- examples/age_csv/age.rb
+- examples/age_noext/Birthdays
+- examples/age_noext/Birthdays-xlsx
+- examples/age_noext/Birthdays-xlsx-with-wrong-extension.xls
+- examples/age_noext/age.rb
 - examples/age_with_multiple_checks/Birthdays.ods
 - examples/age_with_multiple_checks/age_with_multiple_checks.rb
 - examples/local_vars/local_vars.rb
@@ -117,6 +124,7 @@ files:
 - examples/wikipedia_us_cities/us_cities.rb
 - examples/wikipedia_us_cities/us_cities.tsv
 - examples/wikipedia_us_cities/us_cities_bulk_declare.rb
+- examples/wikipedia_us_cities/us_cities_reject.rb
 - lib/dreader.rb
 - lib/dreader/column.rb
 - lib/dreader/engine.rb
@@ -142,7 +150,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.4.10
+rubygems_version: 3.4.21
 signing_key:
 specification_version: 4
 summary: Process and import data from cvs and spreadsheets