dreader 1.1.2 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d59887423bcb1658823534ca3d8a5505899cd70a5f8d23eda235159c590cf364
-  data.tar.gz: b17d71ed8d7db969fcc872a3f01cb6fdfbfaff69e0030ffc304ebc481a15675d
+  metadata.gz: 3c30be2fe49c6c8ce20d4930c75f1279ac1a92a099f609b1266b14dc61c7cf3c
+  data.tar.gz: 58c735a67c45ef11a180bc6f17892ba912656d346320da1a716caa69661f4695
 SHA512:
-  metadata.gz: 1407751e4baa35d4c6644cc9c9ebea7a8185febd2211e6b76c86a3001e8972d58061b6946a178dd007043c897a9943d41971e0868dab2e9e332350a5436bfff1
-  data.tar.gz: da163158d92a1618de0218f5285fe189f3ec5fe78b0d815c6b873574175c817ef4134725ff70a174b0206fb0da1770f9888044b076dbfc8e62b79f22e10930a6
+  metadata.gz: 7847892dbcf648432a9c51867fd70e1260e956e82ea7cbbad93f92882dd867be6f36f67a0edfbb972e16e12c1364efe92a54bd03d31801770da4b28fac725350
+  data.tar.gz: a717955a2eaa0c406d6fb140daf9cd084c11e0d4710289de34b7757b5fd4f4e920ded4fd1450306e3e42a42278c983e078ff8e53248fe0f5b7a93d09fb8a9d40

data/CHANGELOG.org

@@ -1,5 +1,12 @@
 #+TITLE: Changelog
 
+* Version 1.2.0 - <2023-11-02 Thu>
+** reject declaration
+
+   - A new reject declaration allows to reject some lines.  reject takes as
+     input a row and can predicate over columns and virtual columns.  When
+     true, the corresponding line is discarded.
+
 * Version 1.1.2 - <2023-10-31 Tue>
 ** Fixes an issue with the :extension option
 

data/README.org

@@ -137,7 +137,8 @@ To write an import function with Dreader:
   and check parsed data
 - Add virtual columns, that is, columns computed from other values
   in the row
-- Specify how to map line. This is where you do the actual work
+- Specify what lines you want to reject, if any
+- Specify how to transform lines. This is where you do the actual work
   (for instance, if you process a file line by line) or put together data for
   processing after the file has been fully read --- see the next step.
 
@@ -398,6 +399,9 @@ See [[file:examples/wikipedia_us_cities/us_cities_bulk_declare.rb][us_cities_bul
   hash from the code block.
 #+END_NOTES
 
+The data read from each row of our input data is stored in a hash. The hash
+uses column names as the primary key and stores the values in the =:value=
+key.
 
 *** Add virtual columns
 
@@ -427,6 +431,22 @@ Virtual columns are, of course, available to the =mapping= directive
 (see below).
 
 
+*** Specify which lines to reject
+
+You can reject some lines using the =reject= declaration, which is applied row
+by row, can predicate over columns and virtual columns, and has to return a
+Boolean value.
+
+All lines returning a truish value will be be rejected, that is, not stored in
+the =@table= variable (and, consequently, passed to the mapping function).
+
+For instance, the following declaration rejects all lines in which the
+population column is higher than =3_000_000=:
+
+#+begin_src ruby
+  reject { |row| row[:population][:value] > 3_000_000 }
+#+end_src
+
 *** Specify how to process each line
 
 The =mapping= directive specifies what to do with each line read.  The
@@ -442,10 +462,9 @@ value of column =:age= and prints them to standard output
   end
 #+END_EXAMPLE
 
-The data read from each row of our input data is stored in a hash. The hash
-uses column names as the primary key and stores the values in the =:value=
-key.
-
+To invoke the =mapping= declaration on a file, use the =mappings= method,
+which invokes =map= to each row and it stores in the =@table= variable
+whatever value mapping returns.
 
 *** Process data
 
@@ -501,7 +520,13 @@ A typical scenario works as follows:
 (Optionally: check again for errors.)
 
 5. Add your own code to process the data returned after =mappings=, which you
-   can access with =i.table= or =i.data= (synonyms).
+   can assign to a variable (e.g., =returned_data = i.mappings=) or access
+   with =i.table= or =i.data= (synonyms).
+
+#+begin_quote
+Notice that =mappings= does a side effect and invoking the mapping twice in a
+row won't work: you need to reload the file first.
+#+end_quote
 
 Look in the examples directory for further details and a couple of working
 examples.

data/examples/wikipedia_us_cities/us_cities_reject.rb

@@ -0,0 +1,77 @@
+require 'dreader'
+
+# this is the class which will contain all the data we read from the file
+class City
+  [:city, :state, :population, :lat, :lon].each do |var|
+    attr_accessor var
+  end
+
+  def initialize(hash)
+    hash.each do |k, v|
+      self.send("#{k}=", v)
+    end
+  end
+end
+
+class Importer
+  extend Dreader::Engine
+
+  # read from us_cities.tsv, lines from 2 to 10 (included)
+  options do
+    filename "us_cities.tsv"
+    first_row 2
+    last_row  307
+  end
+
+  # these are the columns for which we only need to specify column and name
+  columns ({city: 2, state: 3, latlon: 11}) do
+    process { |val| val.strip }
+  end
+
+  # the population column requires more work
+  column :population do |col|
+    col.colref 4
+
+    # make "3,000" into 3000 (int)
+    col.process { |value| value.gsub(",", "").to_i }
+
+    # check population is positive
+    col.check { |value| value > 0 }
+  end
+
+  # reject all cities with more than 3M people
+  reject do |row|
+    row[:population][:value] >= 3_000_000
+  end
+
+  mapping do |row|
+    # remove all additional information stored in each cell
+    r = Dreader::Util.simplify row
+
+    # make latlon into the lat, lon fields
+    r[:lat], r[:lon] = r[:latlon].split(" ")
+
+    # now r contains something like
+    # {lat: ..., lon: ..., city: ..., state: ..., population: ..., latlon: ...}
+
+    # remove fields which are not understood by the Cities class and
+    # make a new instance
+    cleaned = Dreader::Util.clean r, [:latlon]
+
+    # you must declare an array cities before calling importer.mapping
+    City.new(cleaned)
+  end
+end
+
+# load and process
+importer = Importer
+importer.load mapping: true, debug: true
+
+# output everything to see whether it works
+puts "First ten cities in the US with less than 3M (source Wikipedia)"
+importer.table.each do |city|
+  [:city, :state, :population, :lat, :lon].each do |var|
+    puts "#{var.to_s.capitalize}: #{city.send(var)}"
+  end
+  puts ""
+end

data/lib/dreader/engine.rb

@@ -21,6 +21,8 @@ module Dreader
     attr_accessor :declared_virtual_columns
     # the mapping rules
     attr_accessor :declared_mapping
+    # the declared filter
+    attr_accessor :declared_reject
     
     # the data we read
     attr_reader :table
@@ -118,6 +120,11 @@ module Dreader
       @declared_virtual_columns << column.to_hash.merge({ name: name })
     end
 
+    # define a filter, which skips some rows
+    def reject(&block)
+      @declared_reject = block
+    end
+
     # define what we do with each line we read
     # - `block` is the code which takes as input a `row` and processes
     #   `row` is a hash in which each spreadsheet cell is accessible under
@@ -187,8 +194,13 @@ module Dreader
         # this has side-effects on r
         virtual_columns_on(r) if options[:virtual] || options[:mapping]
 
+        # check whether the filter would ignore this line
+        # notice that we need to invoke compact to avoid nil being added
+        # to the table
+        next if !options[:ignore_reject] && reject?(r)
+
         options[:mapping] ? mappings_on(r) : r
-      end
+      end.compact
     end
 
     # TODO: PASS A ROW (and not row_number and sheet)
@@ -268,6 +280,7 @@ module Dreader
 
     # Compute virtual columns for, with side effect on row
     def virtual_columns_on(row)
+      @declared_virtual_columns ||= []
       @declared_virtual_columns.each do |virtualcol|
         colname = virtualcol[:name]
         row[colname] = { virtual: true }
@@ -291,13 +304,36 @@ module Dreader
       end
     end
 
-    # apply the mapping code to the array it makes sense to invoke it only
-    # once.
+    # check whether a line has to be rejected
+    def reject?(row)
+      rejected = @declared_reject&.call(row)
+      if rejected
+        @logger.debug "[dreader] row rejected by reject declaration #{row}"
+      end
+    end
+
+    # apply the mapping code to the @table.  Notice that we do a side effect
+    # on @table and, hence, invoking the mapping twice won't work (you need to
+    # reload first).
+    #
+    # the mapping is applied only if it defined and it returns the output of
+    # the mapping.
+    #
+    # notice also that we do a side-effect on @table.  This is to make the
+    # behavior of
+    #
+    #   i.load mapping: true
+    #   i.table
+    #
+    # and
+    #
+    #   i = load;
+    #   i.mappings
+    #   i.table
     #
-    # the mapping is applied only if it defined and it uses map, so that
-    # it can be used functionally
+    # the same
     def mappings
-      @table.map { |row| mappings_on(row) }
+      @table = @table.map { |row| mappings_on(row) }
     end
 
     def mappings_on(row)

data/lib/dreader/version.rb

@@ -1,3 +1,3 @@
 module Dreader
-  VERSION = "1.1.2"
+  VERSION = "1.2.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: dreader
 version: !ruby/object:Gem::Version
-  version: 1.1.2
+  version: 1.2.0
 platform: ruby
 authors:
 - Adolfo Villafiorita
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2023-11-01 00:00:00.000000000 Z
+date: 2023-11-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: roo
@@ -124,6 +124,7 @@ files:
 - examples/wikipedia_us_cities/us_cities.rb
 - examples/wikipedia_us_cities/us_cities.tsv
 - examples/wikipedia_us_cities/us_cities_bulk_declare.rb
+- examples/wikipedia_us_cities/us_cities_reject.rb
 - lib/dreader.rb
 - lib/dreader/column.rb
 - lib/dreader/engine.rb
@@ -149,7 +150,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.4.10
+rubygems_version: 3.4.21
 signing_key:
 specification_version: 4
 summary: Process and import data from cvs and spreadsheets