dreader 0.1.2 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: aaf2760d6b154eed0557f71a3485c1ee25b16d378f0e9e39ad4e32f2b179fc6d
-  data.tar.gz: 03c06359c7b7db68585501ff171f020037358ee5cc98bed4fd3047ce68e964f6
+  metadata.gz: a59b3e8b84e9bd26d753db6e25ff23158ecc6e64dc16eb61c37b3f49434f06b2
+  data.tar.gz: a68e6045cce69837e779a85981b2f09feba3c868508abc65e806132786542cde
 SHA512:
-  metadata.gz: eb6bfca19ddb2f7e35ae46fc19c20453b24e85d03f820b692d76743d3a835edc7addc974ff3a0a1e51f7dd90356c27b7ab83675105c850326f9d8d0320798535
-  data.tar.gz: 97736d7edf259fcbfcc0338c5a94cff108a2c2ec2f1e48b3d205714e3391fe7cd5b0a61d6a6a95a15cc6a795295dbb14ee2a32bc480c4c095a31313afbe2f80d
+  metadata.gz: 5c2aae8a2439807827513eca2f20f8c1124849477e48b87133379a702f34157e729b964450b5f2ac3104583ef92a4a21f9022824d0bd6f865e9ed7824455545b
+  data.tar.gz: 9ed0f9b735bd3f1ae75ff46fe1509d37c15f2458e064612cd1909e4be65f11654beaaac2a807d1f7f6ef5962e81925848136bcf80c8eddecafb026b07d5ee5b1

data/.gitignore

@@ -6,3 +6,4 @@
 /pkg/
 /spec/reports/
 /tmp/
+*~

data/Gemfile.lock

@@ -0,0 +1,28 @@
+PATH
+  remote: .
+  specs:
+    dreader (0.2.0)
+      roo
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    mini_portile2 (2.3.0)
+    nokogiri (1.8.2)
+      mini_portile2 (~> 2.3.0)
+    rake (10.5.0)
+    roo (2.7.1)
+      nokogiri (~> 1)
+      rubyzip (~> 1.1, < 2.0.0)
+    rubyzip (1.2.1)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  bundler (~> 1.16)
+  dreader!
+  rake (~> 10.0)
+
+BUNDLED WITH
+   1.16.1

data/README.md

@@ -1,19 +1,32 @@
 # Dreader
 
-A simple DSL built on top of `roo` to read and process tabular data (CSV,
-LibreOffice, Excel).
+A simple DSL built on top of [Roo](https://github.com/roo-rb/roo) to
+read and process tabular data (CSV, LibreOffice, Excel).
 
-Use this gem to specify the structure of some tabular data you want to
-process.  The input data can be in CSV, LibreOffice, and Excel.  Each row can
-then be passed to a block of code you define.
+This gem allows you to:
 
-The gem can thus be used to check, process, import data. We use it to import data into
-Rails application, but the gem can used in any Ruby application.
+1. specify the structure of some tabular data you want to process
+2. debug and check correctness of the data you read
+3. read a file and process it, that is, execute code for each cell and
+   each row of the file
 
-The gem should be relatively easy to use, despite its name: *dread* stands for
-*d*ata *r*eader.
+If your data require elaborations which cannot be performed line by
+line, you can also access all the data read by the gem and manipulate
+it as you need.
+
+The input data can be in CSV (comma or tab separated), LibreOffice,
+and Excel.
+
+We use it to import data into Rails application, but the gem can used
+in any Ruby application.
+
+The gem should be relatively easy to use, despite its name: *dread*
+stands for *d*ata *r*eader.
+
+The gem depends on `roo`, from which we leverage all data
+reading/parsing facilities and which allows us to achieve what we want
+in about 250 lines of code.
 
-The gem depends on `roo`.
 
 ## Installation
 
@@ -33,61 +46,245 @@ Or install it yourself as:
 
 ## Usage
 
-```
+Require `dreader` and declare an instance of the `Dreader::Engine` class:
+
+```ruby
+require 'dreader'
+
 i = Dreader::Engine.new
+```
 
+Specify parsing option, using the following syntax:
+
+```ruby
 i.options do
-  start_at 1
+  filename 'example.ods'
+  
+  sheet 'Sheet 1'
+
+  first_row 1
+  last_row 20
 end
+```
+
+where:
+
+* (optional) `filename` is the file to read.  If not specified, you
+  will have to supply a filename when loading the file.  The extension
+  determines the file type.  **Use `tsv` for tab-separated files.**
+* (optional) `first_row` is the first line to read (use `2` if your
+  file has a header)
+* (optional) `last_row` is the last line to read. If not specified, we
+  will rely on `roo` to determine the last row
+* (optional) `sheet` is the sheet name or number to read from. If not
+  specified, the first (default) sheet is used
+
+Spexcify the structure of your file, for the columns you are interested
+to process.  You have to specify a name (used to access data) and a
+column reference (used to read data from the file).
+
+You can also specify two ruby blocks, `process` and `check` to
+preprocess data and to check for errors.
 
-# first column is called name
+For instance, given the following file:
+
+| Name | Surname | Age |
+| John | Doe     | 30  |
+| Jane | Doe     | 31  |
+| ...  | ...     | ... |
+
+we could use the following declaration to specify the data to read:
+
+```ruby
+# we want to access column 1 using :name
 # :name should be non nil and of length greater than 0
 i.column :name do
+  colref 1
   check do |x|
     x and x.length > 0
   end
 end
 
-# second column is called surname
-# :surname should be non nil and of length greater than 0
-i.column :surname do
+# we want to access column 3 (Age) using :age
+# :age should be non nil and of length greater than 0
+i.column :age do
+  colref 3
+  
+  # make sure the column is transformed into an integer
+  process do |x|
+    x.to_i
+  end
+  
+  # check age is defined and greater than zero
   check do |x|
     x and x.length > 0
   end
-end
+end 
 
-# third column is called birthdate
-# :name should be non nil and of length greater than 0
-i.column :birthdate do
-  check do |x|
-    x.class == Date
-  end
-end
+# we don't care about any other column (and, therefore, 
+# we are done with our declarations)
+```
 
-# fourth column is called age
-# we make it into an integer
-# age should be greater than 0
-i.column :age do
-  process do |r|
-    r.to_i
-  end
+Notice that `colref` can be a string (e.g., `'A'`) or an integer
+(first column is one).
 
-  check do |r|
-    r > 0
-  end
+Notice also that if `process` and `check` are specified, then `check`
+will receive the result of invoking `process` on the cell value.  This
+makes sense if process is used to make the cell value more accessible
+to ruby code (e.g., transforming a string into an integer).
+
+Finally we can specify how we process lines, using the `mapping`
+directive.  Mapping takes an arbitrary piece of ruby code, which can
+reference the fields of a row.
+
+For instance:
+
+```ruby
+i.mapping do |row|
+  puts "#{row[:name][:value]} is #{row[:age][:value]} years old"
 end
+```
+
+Notice that the data read from a line is stored in a hash which uses
+the column names and stores names in the `:value` key.
+
+Now we are all set and we can start working with the data.
+
+First use `read` or `load` (synonyms), to read all data and put it
+into a `@table` instance variable.  This function uses the `column`
+declarations to read data and executes the `process` and `check`
+functions for each cell read.
+
+```ruby
+i.read
+```
+
+We can now use `errors` to see whether any of the `check` functions
+failed:
 
-# when we decide to process the file, for every row
-# we print the value of the name and the value of the surname
-i.mapping do |r|
-  puts "#{r[:name][:value]} #{r[:surname][:value]}"
+```ruby
+array_of_strings = i.errors
+array_of_strings ech do |error_line|
+  puts error_line
 end
+```
+
+Now we can process the file with the `process` function, which
+executes the `mapping` directive for each line read from the file.
 
-i.read "/home/adolfo/Desktop/a.ods"
+```ruby
 i.process
 ```
 
-## Known Bugs and Limitations
+If you need to perform more complex elaborations on the data, you can
+also directly access all data read, using the `table` method, which
+returns an array of hashes (see next section for the details).
+
+```ruby
+i.table
+```
+
+For further details and a couple of working examples, look in the
+examples directory.
+
+
+## Digging deeper
+
+The `read` method fills a `@table` instance variable with an array of
+hashes.  Each hash represents a line of the file. 
+
+Each hash contains one key per column, following your specification.
+Its value is, in turn, a hash with the following structure:
+
+```ruby
+{ 
+  value: ...,     # the result of calling process on the cell
+  row_number: ... # the row number
+  col_number: ... # the column number
+  error: ...      # the result of calling check on the cell processed value
+}
+```
+
+Thus, for instance, given the example above:
+
+```ruby
+i.table
+[ { name: { value: "John", row_number: 1, col_number: 1, errors: nil },
+    age:  { value: 30, row_number: 1, col_number: 2, errors: nil } },
+  { name: { value: "Jane", row_number: 2, col_number: 1, errors: nil },
+    age:  { value: 31, row_number: 2, col_number: 2, errors: nil } } ]
+```
+
+## Simplifying the data read
+
+The `Dreader::Util` class provides some functions to simplify and
+restructure the hashes built by `dreader`.
+
+More in details:
+
+`Dreader::Util.simplify hash` simplifies the hash passed as input by
+removing all information but the value and making the value
+accessible directly from the name of the column.
+
+```ruby
+Dreader::Util.simplify i.table[0]
+{name: "John", age: 30}
+```
+
+`Dreader::Util.slice hash, keys` and `Dreader::Util.slice hash,
+keys`, where `keys` is an arrays of keys, are respectively used to
+select or remove some keys from `hash`.
+  
+```ruby
+Dreader::Util.slice i.table[0], [:age]
+{age: 30}
+  
+Dreader::Util.clean i.table[0], [:age]
+{name: "John"}
+```
+    
+Finally, the `Dreader::Util.restructure` method helps building hashes
+to create
+[ActiveModel](http://api.rubyonrails.org/classes/ActiveModel/Model.html)
+objects with nested attributes:
+
+```ruby
+hash = {name: "John", surname: "Doe", address: "Unknown", city: "NY" }
+   
+Dreader::Util.restructure hash, [:name, :surname], :address_attributes, [:address, :city]
+{name: "John", surname: "Doe", address_attributes: {address: "Unknonw", city: "NY"}}
+```
+
+
+## Debugging your specification
+
+If you are not sure about what is going on (like I often am when
+reading tabular data), you can use the `debug` function, which prints
+the current configuration, reads some records from your files, and
+shows them to standard output:
+
+
+```ruby
+i.debug
+i.debug 40 # read 40 lines (from first_row, if the option is declared)
+i.debug 40, filename # like above, but read from filename
+```
+
+Another possibility is getting the value of the `@table` variable,
+which contains all the data read.
+
+## Known Limitations
+
+At the moment:
+
+- it is not possible to specify column references using header names
+  (like Roo does).
+- it is not possible to pass options to the file readers.  As a
+  consequence tab-separated files must have the `.tsv` extension to be
+  correctly parsed.
+- some testing wouldn't hurt.
+
+## Known Bugs
 
 No known bugs and an unknown number of unknown bugs.
 

data/examples/wikipedia_big_us_cities/big_us_cities.rb

@@ -0,0 +1,48 @@
+# simple example which shows processing data from LibreOffice spreadsheets
+
+require 'dreader'
+
+processor = Dreader::Engine.new
+
+processor.options do
+  first_row 2
+  filename "cities_by_state.ods"
+end
+
+processor.column :state do |col|
+  col.colref 'A'
+end
+
+processor.column :cities do |col|
+  col.colref 'B'
+  col.check do |data|
+    data.class == Integer
+  end
+end
+
+processor.mapping do |row|
+  hash = Dreader::Util.simplify row
+  puts "  states: #{hash[:state]} number of cities: #{hash[:cities]}"
+end
+
+printf "Loading the spreadsheet..."
+processor.load
+puts "done!"
+
+puts "Checking errors..."
+errors = processor.errors
+if errors.size > 0
+  puts "  Errors found:"
+  processor.errors.each do |error|
+    puts error
+  end
+else
+  puts "  No errors found."
+end
+puts "done!"
+
+puts "Processing the spreadsheet..."
+processor.process
+puts "... done"
+    
+  

data/examples/wikipedia_us_cities/us_cities.rb

@@ -0,0 +1,84 @@
+require 'dreader'
+
+# this is the class which will contain all the data we read from the file
+class City
+  [:city, :state, :population, :lat, :lon].each do |var|
+    attr_accessor var
+  end
+
+  def initialize(hash)
+    hash.each do |k, v|
+      self.send("#{k}=", v)
+    end
+  end
+end
+
+importer = Dreader::Engine.new
+
+# read from us_cities.tsv, lines from 2 to 10 (included)
+importer.options do
+  filename "us_cities.tsv"
+  first_row 2
+  last_row  10
+end
+
+# these are the columns for which we only need to specify column and name
+[[:city, 2], [:state, 3], [:latlon, 11]].each do |val|
+  # val[0] -> :city
+  # val[1] -> 2
+  importer.column val[0] do |col|
+    col.colref val[1]
+    col.process do |val|
+      val.strip
+    end
+  end
+end
+
+# the population column requires more work
+importer.column :population do |col|
+  col.colref 4
+
+  # make "3,000" into 3000 (int)
+  col.process do |value|
+    value.gsub(",", "").to_i
+  end
+
+end
+
+cities = []
+
+importer.mapping do |row|
+  # remove all additional information stored in each cell
+  r = Dreader::Util.simplify row
+
+  # make latlon into the lat, lon fields
+  r[:lat], r[:lon] = r[:latlon].split(" ")
+
+  # now r contains something like
+  # {lat: ..., lon: ..., city: ..., state: ..., population: ..., latlon: ...}
+
+  # remove fields which are not understood by the Cities class and
+  # make a new instance
+  cleaned = Dreader::Util.clean r, [:latlon]
+
+  # you must declare an array cities before calling importer.process
+  cities << City.new(cleaned)
+end
+
+# print to stdout what we told dreader to read
+# (useful only for ... debugging!)
+importer.debug 10
+
+# load and process
+importer.load
+cities = []
+importer.process
+
+# output everything to see whether it works
+puts "First ten cities in the US (source Wikipedia)"
+cities.each do |city|
+  [:city, :state, :population, :lat, :lon].each do |var|
+    puts "#{var.to_s.capitalize}: #{city.send(var)}"
+  end
+  puts ""
+end

data/examples/wikipedia_us_cities/us_cities.tsv

@@ -0,0 +1,308 @@
+2016 rank 	City 	State[5] 	2016 estimate 	2010 Census 	Change 	2016 land area 	2016 land area	2016 pop density	2016 pop density	Location
+1	New York[6] 	 New York 	8537673	8175133	+4.43% 	301.5 sq mi 	780.9 km2 	28,317/sq mi 	10,933/km2 	40.6635°N 73.9387°W
+2	Los Angeles 	 California 	3976322	3792621	+4.84% 	468.7 sq mi 	1,213.9 km2 	8,484/sq mi 	3,276/km2 	34.0194°N 118.4108°W
+3	Chicago 	 Illinois 	2704958	2695598	+0.35% 	227.3 sq mi 	588.7 km2 	11,900/sq mi 	4,600/km2 	41.8376°N 87.6818°W
+4	Houston[7] 	 Texas 	2303482	2100263	+9.68% 	637.5 sq mi 	1,651.1 km2 	3,613/sq mi 	1,395/km2 	29.7866°N 95.3909°W
+5	Phoenix 	 Arizona 	1615017	1445632	+11.72% 	517.6 sq mi 	1,340.6 km2 	3,120/sq mi 	1,200/km2 	33.5722°N 112.0901°W
+6	Philadelphia[8] 	 Pennsylvania 	1567872	1526006	+2.74% 	134.2 sq mi 	347.6 km2 	11,683/sq mi 	4,511/km2 	40.0094°N 75.1333°W
+7	San Antonio 	 Texas 	1492510	1327407	+12.44% 	461.0 sq mi 	1,194.0 km2 	3,238/sq mi 	1,250/km2 	29.4724°N 98.5251°W
+8	San Diego 	 California 	1406630	1307402	+7.59% 	325.2 sq mi 	842.3 km2 	4,325/sq mi 	1,670/km2 	32.8153°N 117.1350°W
+9	Dallas 	 Texas 	1317929	1197816	+10.03% 	340.9 sq mi 	882.9 km2 	3,866/sq mi 	1,493/km2 	32.7933°N 96.7665°W
+10	San Jose 	 California 	1025350	945942	+8.39% 	177.5 sq mi 	459.7 km2 	5,777/sq mi 	2,231/km2 	37.2967°N 121.8189°W
+11	Austin 	 Texas 	947890	790390	+19.93% 	312.7 sq mi 	809.9 km2 	3,031/sq mi 	1,170/km2 	30.3039°N 97.7544°W
+12	Jacksonville[9] 	 Florida 	880619	821784	+7.16% 	747.4 sq mi 	1,935.8 km2 	1,178/sq mi 	455/km2 	30.3369°N 81.6616°W
+13	San Francisco[10] 	 California 	870887	805235	+8.15% 	46.9 sq mi 	121.5 km2 	18,569/sq mi 	7,170/km2 	37.7272°N 123.0322°W
+14	Columbus 	 Ohio 	860090	787033	+9.28% 	218.5 sq mi 	565.9 km2 	3,936/sq mi 	1,520/km2 	39.9852°N 82.9848°W
+15	Indianapolis[11] 	 Indiana 	855164	820445	+4.23% 	361.5 sq mi 	936.3 km2 	2,366/sq mi 	914/km2 	39.7767°N 86.1459°W
+16	Fort Worth 	 Texas 	854113	741206	+15.23% 	342.9 sq mi 	888.1 km2 	2,491/sq mi 	962/km2 	32.7815°N 97.3467°W
+17	Charlotte 	 North Carolina 	842051	731424	+15.12% 	305.4 sq mi 	791.0 km2 	2,757/sq mi 	1,064/km2 	35.2078°N 80.8310°W
+18	Seattle 	 Washington 	704352	608660	+15.72% 	83.8 sq mi 	217.0 km2 	8,405/sq mi 	3,245/km2 	47.6205°N 122.3509°W
+19	Denver[12] 	 Colorado 	693060	600158	+15.48% 	153.3 sq mi 	397.0 km2 	4,521/sq mi 	1,746/km2 	39.7619°N 104.8811°W
+20	El Paso 	 Texas 	683080	649121	+5.23% 	256.8 sq mi 	665.1 km2 	2,660/sq mi 	1,030/km2 	31.8484°N 106.4270°W
+21	Washington[13] 	 District of Columbia 	681170	601723	+13.20% 	61.1 sq mi 	158.2 km2 	11,148/sq mi 	4,304/km2 	38.9041°N 77.0172°W
+22	Boston 	 Massachusetts 	673184	617594	+9.00% 	48.3 sq mi 	125.1 km2 	13,938/sq mi 	5,381/km2 	42.3320°N 71.0202°W
+23	Detroit 	 Michigan 	672795	713777	−5.74% 	138.8 sq mi 	359.5 km2 	4,847/sq mi 	1,871/km2 	42.3830°N 83.1022°W
+24	Nashville[14] 	 Tennessee 	660388	601222	+9.84% 	475.9 sq mi 	1,232.6 km2 	1,388/sq mi 	536/km2 	36.1718°N 86.7850°W
+25	Memphis 	 Tennessee 	652717	646889	+0.90% 	317.4 sq mi 	822.1 km2 	2,056/sq mi 	794/km2 	35.1028°N 89.9774°W
+26	Portland 	 Oregon 	639863	583776	+9.61% 	133.5 sq mi 	345.8 km2 	4,793/sq mi 	1,851/km2 	45.5370°N 122.6500°W
+27	Oklahoma City 	 Oklahoma 	638367	579999	+10.06% 	606.3 sq mi 	1,570.3 km2 	1,053/sq mi 	407/km2 	35.4671°N 97.5137°W
+28	Las Vegas 	 Nevada 	632912	583756	+8.42% 	134.4 sq mi 	348.1 km2 	4,709/sq mi 	1,818/km2 	36.2292°N 115.2601°W
+29	Louisville[15] 	 Kentucky 	616261	597337	+3.17% 	263.5 sq mi 	682.5 km2 	2,339/sq mi 	903/km2 	38.1654°N 85.6474°W
+30	Baltimore[16] 	 Maryland 	614664	620961	−1.01% 	80.9 sq mi 	209.5 km2 	7,598/sq mi 	2,934/km2 	39.3000°N 76.6105°W
+31	Milwaukee 	 Wisconsin 	595047	594833	+0.04% 	96.2 sq mi 	249.2 km2 	6,186/sq mi 	2,388/km2 	43.0633°N 87.9667°W
+32	Albuquerque 	 New Mexico 	559277	545852	+2.46% 	188.2 sq mi 	487.4 km2 	2,972/sq mi 	1,147/km2 	35.1056°N 106.6474°W
+33	Tucson 	 Arizona 	530706	520116	+2.04% 	230.8 sq mi 	597.8 km2 	2,299/sq mi 	888/km2 	32.1531°N 110.8706°W
+34	Fresno 	 California 	522053	494665	+5.54% 	114.4 sq mi 	296.3 km2 	4,563/sq mi 	1,762/km2 	36.7836°N 119.7934°W
+35	Sacramento 	 California 	495234	466488	+6.16% 	97.9 sq mi 	253.6 km2 	5,059/sq mi 	1,953/km2 	38.5666°N 121.4686°W
+36	Mesa 	 Arizona 	484587	439041	+10.37% 	137.9 sq mi 	357.2 km2 	3,514/sq mi 	1,357/km2 	33.4019°N 111.7174°W
+37	Kansas City 	 Missouri 	481420	459787	+4.71% 	315.0 sq mi 	815.8 km2 	1,528/sq mi 	590/km2 	39.1251°N 94.5510°W
+38	Atlanta 	 Georgia 	472522	420003	+12.50% 	133.5 sq mi 	345.8 km2 	3,539/sq mi 	1,366/km2 	33.7629°N 84.4227°W
+39	Long Beach 	 California 	470130	462257	+1.70% 	50.3 sq mi 	130.3 km2 	9,347/sq mi 	3,609/km2 	33.8092°N 118.1553°W
+40	Colorado Springs 	 Colorado 	465101	416427	+11.69% 	195.6 sq mi 	506.6 km2 	2,378/sq mi 	918/km2 	38.8673°N 104.7607°W
+41	Raleigh 	 North Carolina 	458880	403892	+13.61% 	145.1 sq mi 	375.8 km2 	3,163/sq mi 	1,221/km2 	35.8306°N 78.6418°W
+42	Miami 	 Florida 	453579	399457	+13.55% 	36.0 sq mi 	93.2 km2 	12,599/sq mi 	4,865/km2 	25.7752°N 80.2086°W
+43	Virginia Beach[16] 	 Virginia 	452602	437994	+3.34% 	244.7 sq mi 	633.8 km2 	1,850/sq mi 	710/km2 	36.7800°N 76.0252°W
+44	Omaha 	 Nebraska 	446970	408958	+9.29% 	133.2 sq mi 	345.0 km2 	3,356/sq mi 	1,296/km2 	41.2644°N 96.0451°W
+45	Oakland 	 California 	420005	390724	+7.49% 	55.9 sq mi 	144.8 km2 	7,514/sq mi 	2,901/km2 	37.7698°N 122.2257°W
+46	Minneapolis 	 Minnesota 	413651	382578	+8.12% 	54.0 sq mi 	139.9 km2 	7,660/sq mi 	2,960/km2 	44.9633°N 93.2683°W
+47	Tulsa 	 Oklahoma 	403090	391906	+2.85% 	196.8 sq mi 	509.7 km2 	2,048/sq mi 	791/km2 	36.1279°N 95.9023°W
+48	Arlington 	 Texas 	392772	365438	+7.48% 	95.8 sq mi 	248.1 km2 	4,100/sq mi 	1,600/km2 	32.7007°N 97.1247°W
+49	New Orleans[17] 	 Louisiana 	391495	343829	+13.86% 	169.4 sq mi 	438.7 km2 	2,311/sq mi 	892/km2 	30.0534°N 89.9345°W
+50	Wichita 	 Kansas 	389902	382368	+1.97% 	160.4 sq mi 	415.4 km2 	2,431/sq mi 	939/km2 	37.6907°N 97.3459°W
+51	Cleveland 	 Ohio 	385809	396815	−2.77% 	77.7 sq mi 	201.2 km2 	4,965/sq mi 	1,917/km2 	41.4785°N 81.6794°W
+52	Tampa 	 Florida 	377165	335709	+12.35% 	113.4 sq mi 	293.7 km2 	3,326/sq mi 	1,284/km2 	27.9701°N 82.4797°W
+53	Bakersfield 	 California 	376380	347483	+8.32% 	148.8 sq mi 	385.4 km2 	2,529/sq mi 	976/km2 	35.3212°N 119.0183°W
+54	Aurora 	 Colorado 	361710	325078	+11.27% 	153.5 sq mi 	397.6 km2 	2,356/sq mi 	910/km2 	39.6880°N 104.6897°W
+55	Honolulu[2] 	 Hawaii 	351792	337256	+4.31% 	60.5 sq mi 	156.7 km2 	5,815/sq mi 	2,245/km2 	21.3243°N 157.8476°W
+56	Anaheim 	 California 	351043	336265	+4.39% 	50.0 sq mi 	129.5 km2 	7,021/sq mi 	2,711/km2 	33.8555°N 117.7601°W
+57	Santa Ana 	 California 	334217	324528	+2.99% 	27.1 sq mi 	70.2 km2 	12,333/sq mi 	4,762/km2 	33.7363°N 117.8830°W
+58	Corpus Christi 	 Texas 	325733	305215	+6.72% 	174.6 sq mi 	452.2 km2 	1,866/sq mi 	720/km2 	27.7543°N 97.1734°W
+59	Riverside 	 California 	324722	303871	+6.86% 	81.2 sq mi 	210.3 km2 	3,999/sq mi 	1,544/km2 	33.9381°N 117.3932°W
+60	Lexington[18] 	 Kentucky 	318449	295803	+7.66% 	283.6 sq mi 	734.5 km2 	1,123/sq mi 	434/km2 	38.0407°N 84.4583°W
+61	St. Louis[16] 	 Missouri 	311404	319294	−2.47% 	62.0 sq mi 	160.6 km2 	5,023/sq mi 	1,939/km2 	38.6357°N 90.2446°W
+62	Stockton 	 California 	307072	291707	+5.27% 	61.7 sq mi 	159.8 km2 	4,977/sq mi 	1,922/km2 	37.9763°N 121.3133°W
+63	Pittsburgh 	 Pennsylvania 	303625	305704	−0.68% 	55.4 sq mi 	143.5 km2 	5,481/sq mi 	2,116/km2 	40.4398°N 79.9766°W
+64	Saint Paul 	 Minnesota 	302398	285068	+6.08% 	52.0 sq mi 	134.7 km2 	5,815/sq mi 	2,245/km2 	44.9489°N 93.1041°W
+65	Cincinnati 	 Ohio 	298800	296943	+0.63% 	77.4 sq mi 	200.5 km2 	3,860/sq mi 	1,490/km2 	39.1402°N 84.5058°W
+66	Anchorage[19] 	 Alaska 	298192	291826	+2.18% 	1,706.6 sq mi 	4,420.1 km2 	175/sq mi 	68/km2 	61.1743°N 149.2843°W
+67	Henderson 	 Nevada 	292969	257729	+13.67% 	104.7 sq mi 	271.2 km2 	2,798/sq mi 	1,080/km2 	36.0097°N 115.0357°W
+68	Greensboro 	 North Carolina 	287027	269666	+6.44% 	128.3 sq mi 	332.3 km2 	2,237/sq mi 	864/km2 	36.0951°N 79.8270°W
+69	Plano 	 Texas 	286057	259841	+10.09% 	71.7 sq mi 	185.7 km2 	3,990/sq mi 	1,540/km2 	33.0508°N 96.7479°W
+70	Newark 	 New Jersey 	281764	277140	+1.67% 	24.1 sq mi 	62.4 km2 	11,691/sq mi 	4,514/km2 	40.7242°N 74.1726°W
+71	Lincoln 	 Nebraska 	280364	258379	+8.51% 	92.1 sq mi 	238.5 km2 	3,044/sq mi 	1,175/km2 	40.8105°N 96.6803°W
+72	Toledo 	 Ohio 	278508	287208	−3.03% 	80.7 sq mi 	209.0 km2 	3,451/sq mi 	1,332/km2 	41.6641°N 83.5819°W
+73	Orlando 	 Florida 	277173	238300	+16.31% 	105.2 sq mi 	272.5 km2 	2,635/sq mi 	1,017/km2 	28.4166°N 81.2736°W
+74	Chula Vista 	 California 	267172	243916	+9.53% 	49.6 sq mi 	128.5 km2 	5,387/sq mi 	2,080/km2 	32.6277°N 117.0152°W
+75	Irvine 	 California 	266122	212375	+25.31% 	65.6 sq mi 	169.9 km2 	4,057/sq mi 	1,566/km2 	33.6784°N 117.7713°W
+76	Fort Wayne 	 Indiana 	264488	253691	+4.26% 	110.6 sq mi 	286.5 km2 	2,391/sq mi 	923/km2 	41.0882°N 85.1439°W
+77	Jersey City 	 New Jersey 	264152	247597	+6.69% 	14.8 sq mi 	38.3 km2 	17,848/sq mi 	6,891/km2 	40.7114°N 74.0648°W
+78	Durham 	 North Carolina 	263016	228330	+15.19% 	109.8 sq mi 	284.4 km2 	2,395/sq mi 	925/km2 	35.9811°N 78.9029°W
+79	St. Petersburg 	 Florida 	260999	244769	+6.63% 	61.8 sq mi 	160.1 km2 	4,223/sq mi 	1,631/km2 	27.7620°N 82.6441°W
+80	Laredo 	 Texas 	257156	236091	+8.92% 	101.1 sq mi 	261.8 km2 	2,544/sq mi 	982/km2 	27.5604°N 99.4892°W
+81	Buffalo 	 New York 	256902	261310	−1.69% 	40.4 sq mi 	104.6 km2 	6,359/sq mi 	2,455/km2 	42.8925°N 78.8597°W
+82	Madison 	 Wisconsin 	252551	233209	+8.29% 	77.0 sq mi 	199.4 km2 	3,280/sq mi 	1,270/km2 	43.0878°N 89.4299°W
+83	Lubbock 	 Texas 	252506	229573	+9.99% 	124.6 sq mi 	322.7 km2 	2,027/sq mi 	783/km2 	33.5656°N 101.8867°W
+84	Chandler 	 Arizona 	247477	236123	+4.81% 	64.9 sq mi 	168.1 km2 	3,813/sq mi 	1,472/km2 	33.2829°N 111.8549°W
+85	Scottsdale 	 Arizona 	246645	217385	+13.46% 	183.9 sq mi 	476.3 km2 	1,341/sq mi 	518/km2 	33.6843°N 111.8611°W
+86	Glendale 	 Arizona 	245895	226721	+8.46% 	59.1 sq mi 	153.1 km2 	4,161/sq mi 	1,607/km2 	33.5331°N 112.1899°W
+87	Reno 	 Nevada 	245255	225221	+8.90% 	107.3 sq mi 	277.9 km2 	2,286/sq mi 	883/km2 	39.5491°N 119.8499°W
+88	Norfolk[16] 	 Virginia 	245115	242803	+0.95% 	53.3 sq mi 	138.0 km2 	4,599/sq mi 	1,776/km2 	36.9230°N 76.2446°W
+89	Winston–Salem 	 North Carolina 	242203	229617	+5.48% 	132.5 sq mi 	343.2 km2 	1,828/sq mi 	706/km2 	36.1027°N 80.2610°W
+90	North Las Vegas 	 Nevada 	238702	216961	+10.02% 	98.0 sq mi 	253.8 km2 	2,436/sq mi 	941/km2 	36.2857°N 115.0939°W
+91	Irving 	 Texas 	238289	216290	+10.17% 	67.0 sq mi 	173.5 km2 	3,557/sq mi 	1,373/km2 	32.8577°N 96.9700°W
+92	Chesapeake[16] 	 Virginia 	237940	222209	+7.08% 	338.5 sq mi 	876.7 km2 	703/sq mi 	271/km2 	36.6794°N 76.3018°W
+93	Gilbert[20] 	 Arizona 	237133	208453	+13.76% 	68.0 sq mi 	176.1 km2 	3,487/sq mi 	1,346/km2 	33.3103°N 111.7431°W
+94	Hialeah 	 Florida 	236387	224669	+5.22% 	21.5 sq mi 	55.7 km2 	10,995/sq mi 	4,245/km2 	25.8699°N 80.3029°W
+95	Garland 	 Texas 	234943	226876	+3.56% 	57.0 sq mi 	147.6 km2 	4,122/sq mi 	1,592/km2 	32.9098°N 96.6303°W
+96	Fremont 	 California 	233136	214089	+8.90% 	77.5 sq mi 	200.7 km2 	3,008/sq mi 	1,161/km2 	37.4945°N 121.9412°W
+97	Baton Rouge[21] 	 Louisiana 	227715	229493	−0.77% 	85.9 sq mi 	222.5 km2 	2,651/sq mi 	1,024/km2 	30.4422°N 91.1309°W
+98	Richmond[16] 	 Virginia 	223170	204214	+9.28% 	59.8 sq mi 	154.9 km2 	3,732/sq mi 	1,441/km2 	37.5314°N 77.4760°W
+99	Boise[22] 	 Idaho 	223154	205671	+8.50% 	82.1 sq mi 	212.6 km2 	2,718/sq mi 	1,049/km2 	43.6002°N 116.2317°W
+100	San Bernardino 	 California 	216239	209924	+3.01% 	61.5 sq mi 	159.3 km2 	3,516/sq mi 	1,358/km2 	34.1416°N 117.2936°W
+101	Spokane 	 Washington 	215973	208916	+3.38% 	68.7 sq mi 	177.9 km2 	3,144/sq mi 	1,214/km2 	47.6669°N 117.4333°W
+102	Des Moines 	 Iowa 	215472	203433	+5.92% 	88.9 sq mi 	230.2 km2 	2,424/sq mi 	936/km2 	41.5726°N 93.6102°W
+103	Modesto 	 California 	212175	201165	+5.47% 	43.0 sq mi 	111.4 km2 	4,934/sq mi 	1,905/km2 	37.6375°N 121.0030°W
+104	Birmingham 	 Alabama 	212157	212237	−0.04% 	146.1 sq mi 	378.4 km2 	1,452/sq mi 	561/km2 	33.5274°N 86.7990°W
+105	Tacoma 	 Washington 	211277	198397	+6.49% 	49.7 sq mi 	128.7 km2 	4,251/sq mi 	1,641/km2 	47.2522°N 122.4598°W
+106	Fontana 	 California 	209665	196069	+6.93% 	43.0 sq mi 	111.4 km2 	4,876/sq mi 	1,883/km2 	34.1090°N 117.4629°W
+107	Rochester 	 New York 	208880	210565	−0.80% 	35.8 sq mi 	92.7 km2 	5,835/sq mi 	2,253/km2 	43.1699°N 77.6169°W
+108	Oxnard 	 California 	207906	197899	+5.06% 	26.9 sq mi 	69.7 km2 	7,729/sq mi 	2,984/km2 	34.2023°N 119.2046°W
+109	Moreno Valley 	 California 	205499	193365	+6.28% 	51.3 sq mi 	132.9 km2 	4,006/sq mi 	1,547/km2 	33.9233°N 117.2057°W
+110	Fayetteville 	 North Carolina 	204759	200564	+2.09% 	147.7 sq mi 	382.5 km2 	1,386/sq mi 	535/km2 	35.0828°N 78.9735°W
+111	Aurora 	 Illinois 	201110	197899	+1.62% 	44.9 sq mi 	116.3 km2 	4,479/sq mi 	1,729/km2 	41.7635°N 88.2901°W
+112	Glendale 	 California 	200831	191719	+4.75% 	30.4 sq mi 	78.7 km2 	6,606/sq mi 	2,551/km2 	34.1814°N 118.2458°W
+113	Yonkers 	 New York 	200807	195976	+2.47% 	18.0 sq mi 	46.6 km2 	11,156/sq mi 	4,307/km2 	40.9459°N 73.8674°W
+114	Huntington Beach 	 California 	200652	189992	+5.61% 	26.9 sq mi 	69.7 km2 	7,459/sq mi 	2,880/km2 	33.6906°N 118.0093°W
+115	Montgomery 	 Alabama 	200022	205764	−2.79% 	159.8 sq mi 	413.9 km2 	1,252/sq mi 	483/km2 	32.3472°N 86.2661°W
+116	Amarillo 	 Texas 	199582	190695	+4.66% 	101.4 sq mi 	262.6 km2 	1,968/sq mi 	760/km2 	35.1999°N 101.8302°W
+117	Little Rock 	 Arkansas 	198541	193524	+2.59% 	118.7 sq mi 	307.4 km2 	1,673/sq mi 	646/km2 	34.7254°N 92.3586°W
+118	Akron 	 Ohio 	197633	199110	−0.74% 	62.0 sq mi 	160.6 km2 	3,188/sq mi 	1,231/km2 	41.0805°N 81.5214°W
+119	Columbus[23] 	 Georgia 	197485	189885	+4.00% 	216.4 sq mi 	560.5 km2 	913/sq mi 	353/km2 	32.5102°N 84.8749°W
+120	Augusta[24] 	 Georgia 	197081	195844	+0.63% 	302.5 sq mi 	783.5 km2 	652/sq mi 	252/km2 	33.3655°N 82.0734°W
+121	Grand Rapids 	 Michigan 	196445	188040	+4.47% 	44.4 sq mi 	115.0 km2 	4,424/sq mi 	1,708/km2 	42.9612°N 85.6556°W
+122	Shreveport 	 Louisiana 	194920	199311	−2.20% 	107.1 sq mi 	277.4 km2 	1,820/sq mi 	700/km2 	32.4669°N 93.7922°W
+123	Salt Lake City 	 Utah 	193744	186440	+3.92% 	111.2 sq mi 	288.0 km2 	1,742/sq mi 	673/km2 	40.7769°N 111.9310°W
+124	Huntsville 	 Alabama 	193079	180105	+7.20% 	213.4 sq mi 	552.7 km2 	905/sq mi 	349/km2 	34.6990°N 86.6730°W
+125	Mobile 	 Alabama 	192904	195111	−1.13% 	139.4 sq mi 	361.0 km2 	1,384/sq mi 	534/km2 	30.6684°N 88.1002°W
+126	Tallahassee 	 Florida 	190894	181376	+5.25% 	100.4 sq mi 	260.0 km2 	1,901/sq mi 	734/km2 	30.4551°N 84.2534°W
+127	Grand Prairie 	 Texas 	190682	175396	+8.72% 	72.3 sq mi 	187.3 km2 	2,637/sq mi 	1,018/km2 	32.6869°N 97.0211°W
+128	Overland Park 	 Kansas 	188966	173372	+8.99% 	75.1 sq mi 	194.5 km2 	2,516/sq mi 	971/km2 	38.8890°N 94.6906°W
+129	Knoxville 	 Tennessee 	186239	178874	+4.12% 	98.5 sq mi 	255.1 km2 	1,891/sq mi 	730/km2 	35.9707°N 83.9493°W
+130	Port St. Lucie 	 Florida 	185132	164603	+12.47% 	118.9 sq mi 	307.9 km2 	1,557/sq mi 	601/km2 	27.2806°N 80.3883°W
+131	Worcester 	 Massachusetts 	184508	181045	+1.91% 	37.4 sq mi 	96.9 km2 	4,933/sq mi 	1,905/km2 	42.2695°N 71.8078°W
+132	Brownsville 	 Texas 	183823	175023	+5.03% 	132.5 sq mi 	343.2 km2 	1,387/sq mi 	536/km2 	25.9991°N 97.4550°W
+133	Tempe 	 Arizona 	182498	161719	+12.85% 	40.0 sq mi 	103.6 km2 	4,562/sq mi 	1,761/km2 	33.3884°N 111.9318°W
+134	Santa Clarita 	 California 	181972	176320	+3.21% 	52.8 sq mi 	136.8 km2 	3,446/sq mi 	1,331/km2 	34.4030°N 118.5042°W
+135	Newport News[16] 	 Virginia 	181825	180719	+0.61% 	69.1 sq mi 	179.0 km2 	2,631/sq mi 	1,016/km2 	37.0762°N 76.5220°W
+136	Cape Coral 	 Florida 	179804	154305	+16.53% 	105.6 sq mi 	273.5 km2 	1,703/sq mi 	658/km2 	26.6432°N 81.9974°W
+137	Providence 	 Rhode Island 	179219	178042	+0.66% 	18.4 sq mi 	47.7 km2 	9,740/sq mi 	3,760/km2 	41.8231°N 71.4188°W
+138	Fort Lauderdale 	 Florida 	178752	165521	+7.99% 	34.6 sq mi 	89.6 km2 	5,166/sq mi 	1,995/km2 	26.1412°N 80.1467°W
+139	Chattanooga 	 Tennessee 	177571	167674	+5.90% 	143.1 sq mi 	370.6 km2 	1,241/sq mi 	479/km2 	35.0660°N 85.2484°W
+140	Rancho Cucamonga 	 California 	176534	165269	+6.82% 	40.0 sq mi 	103.6 km2 	4,413/sq mi 	1,704/km2 	34.1233°N 117.5642°W
+141	Oceanside 	 California 	175464	167086	+5.01% 	41.3 sq mi 	107.0 km2 	4,249/sq mi 	1,641/km2 	33.2245°N 117.3062°W
+142	Santa Rosa 	 California 	175155	167815	+4.37% 	41.3 sq mi 	107.0 km2 	4,241/sq mi 	1,637/km2 	38.4468°N 122.7061°W
+143	Garden Grove 	 California 	174858	170883	+2.33% 	18.0 sq mi 	46.6 km2 	9,714/sq mi 	3,751/km2 	33.7788°N 117.9605°W
+144	Vancouver 	 Washington 	174826	161791	+8.06% 	46.9 sq mi 	121.5 km2 	3,728/sq mi 	1,439/km2 	45.6349°N 122.5957°W
+145	Sioux Falls 	 South Dakota 	174360	153888	+13.30% 	75.4 sq mi 	195.3 km2 	2,312/sq mi 	893/km2 	43.5383°N 96.7320°W
+146	Ontario 	 California 	173212	163924	+5.67% 	49.9 sq mi 	129.2 km2 	3,471/sq mi 	1,340/km2 	34.0394°N 117.6042°W
+147	McKinney 	 Texas 	172298	131117	+31.41% 	63.0 sq mi 	163.2 km2 	2,735/sq mi 	1,056/km2 	33.1985°N 96.6680°W
+148	Elk Grove 	 California 	169743	153015	+10.93% 	42.2 sq mi 	109.3 km2 	4,022/sq mi 	1,553/km2 	38.4146°N 121.3850°W
+149	Jackson 	 Mississippi 	169148	173514	−2.52% 	111.0 sq mi 	287.5 km2 	1,524/sq mi 	588/km2 	32.3158°N 90.2128°W
+150	Pembroke Pines 	 Florida 	168587	154019	+9.46% 	33.0 sq mi 	85.5 km2 	5,109/sq mi 	1,973/km2 	26.0210°N 80.3404°W
+151	Salem 	 Oregon 	167419	154637	+8.27% 	48.6 sq mi 	125.9 km2 	3,445/sq mi 	1,330/km2 	44.9237°N 123.0232°W
+152	Springfield 	 Missouri 	167319	159498	+4.90% 	82.3 sq mi 	213.2 km2 	2,033/sq mi 	785/km2 	37.1942°N 93.2913°W
+153	Corona 	 California 	166785	152374	+9.46% 	39.5 sq mi 	102.3 km2 	4,222/sq mi 	1,630/km2 	33.8620°N 117.5655°W
+154	Eugene 	 Oregon 	166575	156185	+6.65% 	44.1 sq mi 	114.2 km2 	3,777/sq mi 	1,458/km2 	44.0567°N 123.1162°W
+155	Fort Collins 	 Colorado 	164207	143986	+14.04% 	55.8 sq mi 	144.5 km2 	2,943/sq mi 	1,136/km2 	40.5482°N 105.0648°W
+156	Peoria 	 Arizona 	164173	154065	+6.56% 	175.7 sq mi 	455.1 km2 	934/sq mi 	361/km2 	33.7862°N 112.3080°W
+157	Frisco 	 Texas 	163656	116989	+39.89% 	67.7 sq mi 	175.3 km2 	2,417/sq mi 	933/km2 	33.1554°N 96.8226°W
+158	Cary[25] 	 North Carolina 	162320	135234	+20.03% 	56.5 sq mi 	146.3 km2 	2,873/sq mi 	1,109/km2 	35.7809°N 78.8133°W
+159	Lancaster 	 California 	160106	156633	+2.22% 	94.3 sq mi 	244.2 km2 	1,698/sq mi 	656/km2 	34.6936°N 118.1753°W
+160	Hayward 	 California 	158937	144186	+10.23% 	45.5 sq mi 	117.8 km2 	3,493/sq mi 	1,349/km2 	37.6287°N 122.1024°W
+161	Palmdale 	 California 	157356	152750	+3.02% 	106.0 sq mi 	274.5 km2 	1,484/sq mi 	573/km2 	34.5910°N 118.1054°W
+162	Salinas 	 California 	157218	150441	+4.50% 	23.6 sq mi 	61.1 km2 	6,662/sq mi 	2,572/km2 	36.6902°N 121.6337°W
+163	Alexandria[16] 	 Virginia 	155810	139966	+11.32% 	15.0 sq mi 	38.8 km2 	10,387/sq mi 	4,010/km2 	38.8201°N 77.0841°W
+164	Lakewood 	 Colorado 	154393	142980	+7.98% 	42.9 sq mi 	111.1 km2 	3,599/sq mi 	1,390/km2 	39.6989°N 105.1176°W
+165	Springfield 	 Massachusetts 	154074	153060	+0.66% 	31.9 sq mi 	82.6 km2 	4,830/sq mi 	1,860/km2 	42.1155°N 72.5400°W
+166	Pasadena 	 Texas 	153351	149043	+2.89% 	43.5 sq mi 	112.7 km2 	3,525/sq mi 	1,361/km2 	29.6586°N 95.1506°W
+167	Sunnyvale 	 California 	152771	140081	+9.06% 	22.0 sq mi 	57.0 km2 	6,944/sq mi 	2,681/km2 	37.3858°N 122.0263°W
+168	Macon[26] 	 Georgia 	152555	91351	+67.00% 	249.3 sq mi 	645.7 km2 	612/sq mi 	236/km2 	32.8088°N 83.6942°W
+169	Pomona 	 California 	152494	149058	+2.31% 	23.0 sq mi 	59.6 km2 	6,630/sq mi 	2,560/km2 	34.0585°N 117.7611°W
+170	Hollywood 	 Florida 	151998	140768	+7.98% 	27.3 sq mi 	70.7 km2 	5,568/sq mi 	2,150/km2 	26.0310°N 80.1646°W
+171	Kansas City[27] 	 Kansas 	151709	145786	+4.06% 	124.8 sq mi 	323.2 km2 	1,216/sq mi 	470/km2 	39.1225°N 94.7418°W
+172	Escondido 	 California 	151613	143911	+5.35% 	37.1 sq mi 	96.1 km2 	4,087/sq mi 	1,578/km2 	33.1331°N 117.0740°W
+173	Clarksville 	 Tennessee 	150287	132929	+13.06% 	98.3 sq mi 	254.6 km2 	1,529/sq mi 	590/km2 	36.5664°N 87.3452°W
+174	Joliet 	 Illinois 	148262	147433	+0.56% 	64.4 sq mi 	166.8 km2 	2,302/sq mi 	889/km2 	41.5177°N 88.1488°W
+175	Rockford 	 Illinois 	147651	152871	−3.41% 	63.5 sq mi 	164.5 km2 	2,325/sq mi 	898/km2 	42.2588°N 89.0646°W
+176	Torrance 	 California 	147195	145438	+1.21% 	20.5 sq mi 	53.1 km2 	7,180/sq mi 	2,770/km2 	33.8350°N 118.3414°W
+177	Naperville 	 Illinois 	147122	141853	+3.71% 	38.7 sq mi 	100.2 km2 	3,802/sq mi 	1,468/km2 	41.7492°N 88.1620°W
+178	Paterson 	 New Jersey 	147000	146199	+0.55% 	8.4 sq mi 	21.8 km2 	17,500/sq mi 	6,800/km2 	40.9148°N 74.1628°W
+179	Savannah 	 Georgia 	146763	136286	+7.69% 	103.6 sq mi 	268.3 km2 	1,417/sq mi 	547/km2 	32.0025°N 81.1536°W
+180	Bridgeport 	 Connecticut 	145936	144229	+1.18% 	16.1 sq mi 	41.7 km2 	9,064/sq mi 	3,500/km2 	41.1874°N 73.1958°W
+181	Mesquite 	 Texas 	143736	139824	+2.80% 	47.2 sq mi 	122.2 km2 	3,045/sq mi 	1,176/km2 	32.7629°N 96.5888°W
+182	Killeen 	 Texas 	143400	127921	+12.10% 	53.5 sq mi 	138.6 km2 	2,680/sq mi 	1,030/km2 	31.0777°N 97.7320°W
+183	Syracuse 	 New York 	143378	145170	−1.23% 	25.0 sq mi 	64.7 km2 	5,735/sq mi 	2,214/km2 	43.0410°N 76.1436°W
+184	McAllen 	 Texas 	142212	129877	+9.50% 	58.4 sq mi 	151.3 km2 	2,435/sq mi 	940/km2 	26.2322°N 98.2464°W
+185	Pasadena 	 California 	142059	137122	+3.60% 	23.0 sq mi 	59.6 km2 	6,176/sq mi 	2,385/km2 	34.1606°N 118.1396°W
+186	Bellevue 	 Washington 	141400	122363	+15.56% 	33.5 sq mi 	86.8 km2 	4,221/sq mi 	1,630/km2 	47.5979°N 122.1565°W
+187	Fullerton 	 California 	140721	135161	+4.11% 	22.4 sq mi 	58.0 km2 	6,282/sq mi 	2,425/km2 	33.8857°N 117.9280°W
+188	Orange 	 California 	140504	136416	+3.00% 	25.4 sq mi 	65.8 km2 	5,532/sq mi 	2,136/km2 	33.7870°N 117.8613°W
+189	Dayton 	 Ohio 	140489	141527	−0.73% 	55.7 sq mi 	144.3 km2 	2,522/sq mi 	974/km2 	39.7774°N 84.1996°W
+190	Miramar 	 Florida 	138449	122041	+13.44% 	29.4 sq mi 	76.1 km2 	4,709/sq mi 	1,818/km2 	25.9770°N 80.3358°W
+191	Thornton 	 Colorado 	136703	118772	+15.10% 	35.7 sq mi 	92.5 km2 	3,829/sq mi 	1,478/km2 	39.9194°N 104.9428°W
+192	West Valley City 	 Utah 	136574	129480	+5.48% 	35.5 sq mi 	91.9 km2 	3,847/sq mi 	1,485/km2 	40.6885°N 112.0118°W
+193	Olathe 	 Kansas 	135473	125872	+7.63% 	60.9 sq mi 	157.7 km2 	2,225/sq mi 	859/km2 	38.8843°N 94.8195°W
+194	Hampton[16] 	 Virginia 	135410	137436	−1.47% 	51.5 sq mi 	133.4 km2 	2,629/sq mi 	1,015/km2 	37.0480°N 76.2971°W
+195	Warren 	 Michigan 	135125	134056	+0.80% 	34.4 sq mi 	89.1 km2 	3,928/sq mi 	1,517/km2 	42.4929°N 83.0250°W
+196	Midland 	 Texas 	134610	111147	+21.11% 	74.4 sq mi 	192.7 km2 	1,809/sq mi 	698/km2 	32.0246°N 102.1135°W
+197	Waco 	 Texas 	134432	124805	+7.71% 	89.0 sq mi 	230.5 km2 	1,510/sq mi 	580/km2 	31.5601°N 97.1860°W
+198	Charleston 	 South Carolina 	134385	120083	+11.91% 	109.0 sq mi 	282.3 km2 	1,233/sq mi 	476/km2 	32.8179°N 79.9590°W
+199	Columbia 	 South Carolina 	134309	129272	+3.90% 	133.5 sq mi 	345.8 km2 	1,006/sq mi 	388/km2 	34.0291°N 80.8980°W
+200	Denton 	 Texas 	133808	113383	+18.01% 	93.4 sq mi 	241.9 km2 	1,433/sq mi 	553/km2 	33.2166°N 97.1414°W
+201	Carrollton 	 Texas 	133351	119097	+11.97% 	36.3 sq mi 	94.0 km2 	3,674/sq mi 	1,419/km2 	32.9884°N 96.8998°W
+202	Surprise 	 Arizona 	132677	117517	+12.90% 	107.9 sq mi 	279.5 km2 	1,230/sq mi 	470/km2 	33.6706°N 112.4527°W
+203	Roseville 	 California 	132671	118788	+11.69% 	43.0 sq mi 	111.4 km2 	3,085/sq mi 	1,191/km2 	38.7690°N 121.3189°W
+204	Sterling Heights 	 Michigan 	132427	129699	+2.10% 	36.5 sq mi 	94.5 km2 	3,628/sq mi 	1,401/km2 	42.5812°N 83.0303°W
+205	Murfreesboro 	 Tennessee 	131947	108755	+21.32% 	55.9 sq mi 	144.8 km2 	2,360/sq mi 	910/km2 	35.8522°N 86.4160°W
+206	Gainesville 	 Florida 	131591	124354	+5.82% 	62.3 sq mi 	161.4 km2 	2,112/sq mi 	815/km2 	29.6788°N 82.3461°W
+207	Cedar Rapids 	 Iowa 	131127	126326	+3.80% 	70.8 sq mi 	183.4 km2 	1,852/sq mi 	715/km2 	41.9670°N 91.6778°W
+208	Visalia 	 California 	131074	124442	+5.33% 	37.5 sq mi 	97.1 km2 	3,495/sq mi 	1,349/km2 	36.3273°N 119.3289°W
+209	Coral Springs 	 Florida 	130059	121096	+7.40% 	23.8 sq mi 	61.6 km2 	5,465/sq mi 	2,110/km2 	26.2707°N 80.2593°W
+210	New Haven 	 Connecticut 	129934	129779	+0.12% 	18.7 sq mi 	48.4 km2 	6,948/sq mi 	2,683/km2 	41.3108°N 72.9250°W
+211	Stamford 	 Connecticut 	129113	122643	+5.28% 	37.6 sq mi 	97.4 km2 	3,434/sq mi 	1,326/km2 	41.0799°N 73.5460°W
+212	Thousand Oaks 	 California 	128888	126683	+1.74% 	55.2 sq mi 	143.0 km2 	2,335/sq mi 	902/km2 	34.1933°N 118.8742°W
+213	Concord 	 California 	128726	122067	+5.46% 	30.5 sq mi 	79.0 km2 	4,221/sq mi 	1,630/km2 	37.9722°N 122.0016°W
+214	Elizabeth 	 New Jersey 	128640	124969	+2.94% 	12.3 sq mi 	31.9 km2 	10,459/sq mi 	4,038/km2 	40.6664°N 74.1935°W
+215	Lafayette[28] 	 Louisiana 	127626	120623	+5.81% 	53.8 sq mi 	139.3 km2 	2,372/sq mi 	916/km2 	30.2074°N 92.0285°W
+216	Kent 	 Washington 	127514	92411	+37.99% 	33.7 sq mi 	87.3 km2 	3,784/sq mi 	1,461/km2 	47.3880°N 122.2127°W
+217	Topeka 	 Kansas 	126808	127473	−0.52% 	61.5 sq mi 	159.3 km2 	2,062/sq mi 	796/km2 	39.0347°N 95.6962°W
+218	Simi Valley 	 California 	126327	124237	+1.68% 	41.5 sq mi 	107.5 km2 	3,044/sq mi 	1,175/km2 	34.2669°N 118.7485°W
+219	Santa Clara 	 California 	125948	116468	+8.14% 	18.4 sq mi 	47.7 km2 	6,845/sq mi 	2,643/km2 	37.3646°N 121.9679°W
+220	Athens[29] 	 Georgia 	123371	115452	+6.86% 	116.4 sq mi 	301.5 km2 	1,060/sq mi 	410/km2 	33.9496°N 83.3701°W
+221	Hartford 	 Connecticut 	123243	124775	−1.23% 	17.4 sq mi 	45.1 km2 	7,083/sq mi 	2,735/km2 	41.7659°N 72.6816°W
+222	Victorville 	 California 	122265	115903	+5.49% 	73.3 sq mi 	189.8 km2 	1,668/sq mi 	644/km2 	34.5277°N 117.3536°W
+223	Abilene 	 Texas 	122225	117063	+4.41% 	106.7 sq mi 	276.4 km2 	1,146/sq mi 	442/km2 	32.4545°N 99.7381°W
+224	Norman 	 Oklahoma 	122180	110925	+10.15% 	178.8 sq mi 	463.1 km2 	683/sq mi 	264/km2 	35.2406°N 97.3453°W
+225	Vallejo 	 California 	121299	115942	+4.62% 	30.7 sq mi 	79.5 km2 	3,951/sq mi 	1,525/km2 	38.1079°N 122.2640°W
+226	Berkeley 	 California 	121240	112580	+7.69% 	10.5 sq mi 	27.2 km2 	11,547/sq mi 	4,458/km2 	37.8670°N 122.2991°W
+227	Round Rock 	 Texas 	120892	99887	+21.03% 	35.6 sq mi 	92.2 km2 	3,396/sq mi 	1,311/km2 	30.5252°N 97.6660°W
+228	Ann Arbor 	 Michigan 	120782	113934	+6.01% 	28.1 sq mi 	72.8 km2 	4,298/sq mi 	1,659/km2 	42.2761°N 83.7309°W
+229	Fargo 	  North Dakota 	120762	105549	+14.41% 	49.3 sq mi 	127.7 km2 	2,450/sq mi 	950/km2 	46.8652°N 96.8290°W
+230	Columbia 	 Missouri 	120612	108500	+11.16% 	65.0 sq mi 	168.3 km2 	1,856/sq mi 	717/km2 	38.9473°N 92.3264°W
+231	Allentown 	 Pennsylvania 	120443	118032	+2.04% 	17.5 sq mi 	45.3 km2 	6,882/sq mi 	2,657/km2 	40.5936°N 75.4784°W
+232	Evansville 	 Indiana 	119477	117429	+1.74% 	47.3 sq mi 	122.5 km2 	2,526/sq mi 	975/km2 	37.9877°N 87.5347°W
+233	Beaumont 	 Texas 	118299	118296	0.00% 	82.1 sq mi 	212.6 km2 	1,441/sq mi 	556/km2 	30.0849°N 94.1453°W
+234	Odessa 	 Texas 	117871	99940	+17.94% 	45.2 sq mi 	117.1 km2 	2,608/sq mi 	1,007/km2 	31.8838°N 102.3411°W
+235	Wilmington 	 North Carolina 	117525	106476	+10.38% 	51.6 sq mi 	133.6 km2 	2,278/sq mi 	880/km2 	34.2092°N 77.8858°W
+236	Arvada 	 Colorado 	117453	106433	+10.35% 	38.6 sq mi 	100.0 km2 	3,043/sq mi 	1,175/km2 	39.8337°N 105.1503°W
+237	Independence 	 Missouri 	117030	116830	+0.17% 	77.8 sq mi 	201.5 km2 	1,504/sq mi 	581/km2 	39.0855°N 94.3521°W
+238	Provo 	 Utah 	116868	112488	+3.89% 	41.7 sq mi 	108.0 km2 	2,803/sq mi 	1,082/km2 	40.2453°N 111.6448°W
+239	Lansing 	 Michigan 	116020	114297	+1.51% 	39.1 sq mi 	101.3 km2 	2,967/sq mi 	1,146/km2 	42.7143°N 84.5593°W
+240	El Monte 	 California 	115807	113475	+2.06% 	9.6 sq mi 	24.9 km2 	12,063/sq mi 	4,658/km2 	34.0746°N 118.0291°W
+241	Springfield 	 Illinois 	115715	116250	−0.46% 	60.1 sq mi 	155.7 km2 	1,925/sq mi 	743/km2 	39.7911°N 89.6446°W
+242	Fairfield 	 California 	114756	105321	+8.96% 	40.9 sq mi 	105.9 km2 	2,806/sq mi 	1,083/km2 	38.2593°N 122.0321°W
+243	Clearwater 	 Florida 	114361	107685	+6.20% 	25.9 sq mi 	67.1 km2 	4,415/sq mi 	1,705/km2 	27.9789°N 82.7666°W
+244	Peoria 	 Illinois 	114265	115007	−0.65% 	48.2 sq mi 	124.8 km2 	2,371/sq mi 	915/km2 	40.7515°N 89.6174°W
+245	Rochester 	 Minnesota 	114011	106769	+6.78% 	54.6 sq mi 	141.4 km2 	2,088/sq mi 	806/km2 	44.0154°N 92.4772°W
+246	Carlsbad 	 California 	113952	105328	+8.19% 	37.7 sq mi 	97.6 km2 	3,023/sq mi 	1,167/km2 	33.1239°N 117.2828°W
+247	Westminster 	 Colorado 	113875	106114	+7.31% 	31.7 sq mi 	82.1 km2 	3,592/sq mi 	1,387/km2 	39.8822°N 105.0644°W
+248	West Jordan 	 Utah 	113699	103712	+9.63% 	32.3 sq mi 	83.7 km2 	3,520/sq mi 	1,360/km2 	40.6024°N 112.0008°W
+249	Pearland 	 Texas 	113570	91252	+24.46% 	46.3 sq mi 	119.9 km2 	2,453/sq mi 	947/km2 	29.5558°N 95.3231°W
+250	Richardson 	 Texas 	113347	99223	+14.23% 	28.6 sq mi 	74.1 km2 	3,963/sq mi 	1,530/km2 	32.9723°N 96.7081°W
+251	Downey 	 California 	113267	111772	+1.34% 	12.4 sq mi 	32.1 km2 	9,134/sq mi 	3,527/km2 	33.9382°N 118.1309°W
+252	Miami Gardens 	 Florida 	113058	107167	+5.50% 	18.2 sq mi 	47.1 km2 	6,212/sq mi 	2,398/km2 	25.9489°N 80.2436°W
+253	Temecula 	 California 	113054	100097	+12.94% 	37.3 sq mi 	96.6 km2 	3,031/sq mi 	1,170/km2 	33.4931°N 117.1317°W
+254	Costa Mesa 	 California 	112822	109960	+2.60% 	15.7 sq mi 	40.7 km2 	7,186/sq mi 	2,775/km2 	33.6659°N 117.9123°W
+255	College Station 	 Texas 	112141	93857	+19.48% 	51.0 sq mi 	132.1 km2 	2,199/sq mi 	849/km2 	30.5852°N 96.2964°W
+256	Elgin 	 Illinois 	112123	108188	+3.64% 	37.4 sq mi 	96.9 km2 	2,998/sq mi 	1,158/km2 	42.0396°N 88.3217°W
+257	Murrieta 	 California 	111674	103466	+7.93% 	33.6 sq mi 	87.0 km2 	3,324/sq mi 	1,283/km2 	33.5721°N 117.1904°W
+258	Gresham 	 Oregon 	111523	105594	+5.61% 	23.3 sq mi 	60.3 km2 	4,786/sq mi 	1,848/km2 	45.5023°N 122.4416°W
+259	High Point 	 North Carolina 	111223	104371	+6.57% 	55.2 sq mi 	143.0 km2 	2,015/sq mi 	778/km2 	35.9900°N 79.9905°W
+260	Antioch 	 California 	110898	102372	+8.33% 	29.4 sq mi 	76.1 km2 	3,772/sq mi 	1,456/km2 	37.9791°N 121.7962°W
+261	Inglewood 	 California 	110654	109673	+0.89% 	9.1 sq mi 	23.6 km2 	12,160/sq mi 	4,700/km2 	33.9561°N 118.3443°W
+262	Cambridge 	 Massachusetts 	110651	105162	+5.22% 	6.4 sq mi 	16.6 km2 	17,289/sq mi 	6,675/km2 	42.3760°N 71.1187°W
+263	Lowell 	 Massachusetts 	110558	106519	+3.79% 	13.6 sq mi 	35.2 km2 	8,129/sq mi 	3,139/km2 	42.6390°N 71.3211°W
+264	Manchester 	 New Hampshire 	110506	109565	+0.86% 	33.1 sq mi 	85.7 km2 	3,339/sq mi 	1,289/km2 	42.9849°N 71.4441°W
+265	Billings 	 Montana 	110323	104170	+5.91% 	43.7 sq mi 	113.2 km2 	2,525/sq mi 	975/km2 	45.7885°N 108.5499°W
+266	Pueblo 	 Colorado 	110291	106595	+3.47% 	53.6 sq mi 	138.8 km2 	2,058/sq mi 	795/km2 	38.2699°N 104.6123°W
+267	Palm Bay 	 Florida 	110104	103190	+6.70% 	65.7 sq mi 	170.2 km2 	1,676/sq mi 	647/km2 	27.9856°N 80.6626°W
+268	Centennial 	 Colorado 	109932	100377	+9.52% 	29.5 sq mi 	76.4 km2 	3,727/sq mi 	1,439/km2 	39.5906°N 104.8691°W
+269	Richmond 	 California 	109813	103701	+5.89% 	30.1 sq mi 	78.0 km2 	3,648/sq mi 	1,409/km2 	37.9523°N 122.3606°W
+270	Ventura[30] 	 California 	109592	106433	+2.97% 	21.8 sq mi 	56.5 km2 	5,027/sq mi 	1,941/km2 	34.2678°N 119.2542°W
+271	Pompano Beach 	 Florida 	109393	99845	+9.56% 	24.0 sq mi 	62.2 km2 	4,558/sq mi 	1,760/km2 	26.2416°N 80.1339°W
+272	North Charleston 	 South Carolina 	109298	97471	+12.13% 	73.7 sq mi 	190.9 km2 	1,483/sq mi 	573/km2 	32.9178°N 80.0650°W
+273	Everett 	 Washington 	109043	103019	+5.85% 	33.3 sq mi 	86.2 km2 	3,275/sq mi 	1,264/km2 	47.9566°N 122.1914°W
+274	Waterbury 	 Connecticut 	108272	110366	−1.90% 	28.5 sq mi 	73.8 km2 	3,799/sq mi 	1,467/km2 	41.5585°N 73.0367°W
+275	West Palm Beach 	 Florida 	108161	99919	+8.25% 	55.1 sq mi 	142.7 km2 	1,963/sq mi 	758/km2 	26.7464°N 80.1251°W
+276	Boulder 	 Colorado 	108090	97385	+10.99% 	24.8 sq mi 	64.2 km2 	4,358/sq mi 	1,683/km2 	40.0270°N 105.2519°W
+277	West Covina 	 California 	107847	106098	+1.65% 	16.0 sq mi 	41.4 km2 	6,740/sq mi 	2,600/km2 	34.0559°N 117.9099°W
+278	Broken Arrow 	 Oklahoma 	107403	98850	+8.65% 	61.7 sq mi 	159.8 km2 	1,741/sq mi 	672/km2 	36.0365°N 95.7810°W
+279	Clovis 	 California 	106583	95631	+11.45% 	24.2 sq mi 	62.7 km2 	4,404/sq mi 	1,700/km2 	36.8282°N 119.6849°W
+280	Daly City 	 California 	106472	101123	+5.29% 	7.6 sq mi 	19.7 km2 	14,009/sq mi 	5,409/km2 	37.7009°N 122.4650°W
+281	Lakeland 	 Florida 	106420	97422	+9.24% 	65.9 sq mi 	170.7 km2 	1,615/sq mi 	624/km2 	28.0555°N 81.9549°W
+282	Santa Maria 	 California 	106290	99553	+6.77% 	22.8 sq mi 	59.1 km2 	4,662/sq mi 	1,800/km2 	34.9332°N 120.4438°W
+283	Norwalk 	 California 	106178	105549	+0.60% 	9.7 sq mi 	25.1 km2 	10,946/sq mi 	4,226/km2 	33.9076°N 118.0835°W
+284	Sandy Springs 	 Georgia 	105703	93853	+12.63% 	37.7 sq mi 	97.6 km2 	2,804/sq mi 	1,083/km2 	33.9315°N 84.3687°W
+285	Hillsboro 	 Oregon 	105164	91611	+14.79% 	25.0 sq mi 	64.7 km2 	4,207/sq mi 	1,624/km2 	45.5280°N 122.9357°W
+286	Green Bay 	 Wisconsin 	105139	104057	+1.04% 	45.4 sq mi 	117.6 km2 	2,316/sq mi 	894/km2 	44.5207°N 87.9842°W
+287	Tyler 	 Texas 	104798	96900	+8.15% 	56.6 sq mi 	146.6 km2 	1,852/sq mi 	715/km2 	32.3173°N 95.3059°W
+288	Wichita Falls 	 Texas 	104724	104553	+0.16% 	72.2 sq mi 	187.0 km2 	1,450/sq mi 	560/km2 	33.9067°N 98.5259°W
+289	Lewisville 	 Texas 	104659	95290	+9.83% 	36.7 sq mi 	95.1 km2 	2,852/sq mi 	1,101/km2 	33.0466°N 96.9818°W
+290	Burbank 	 California 	104447	103340	+1.07% 	17.4 sq mi 	45.1 km2 	6,003/sq mi 	2,318/km2 	34.1901°N 118.3264°W
+291	Greeley 	 Colorado 	103990	92889	+11.95% 	47.8 sq mi 	123.8 km2 	2,176/sq mi 	840/km2 	40.4153°N 104.7697°W
+292	San Mateo 	 California 	103959	97207	+6.95% 	12.1 sq mi 	31.3 km2 	8,592/sq mi 	3,317/km2 	37.5603°N 122.3106°W
+293	El Cajon 	 California 	103768	99478	+4.31% 	14.5 sq mi 	37.6 km2 	7,156/sq mi 	2,763/km2 	32.8017°N 116.9604°W
+294	Jurupa Valley 	 California 	103541	0	NA[31] 	42.9 sq mi 	111.1 km2 	2,414/sq mi 	932/km2 	34.0026°N 117.4676°W
+295	Rialto 	 California 	103314	99171	+4.18% 	22.3 sq mi 	57.8 km2 	4,633/sq mi 	1,789/km2 	34.1118°N 117.3883°W
+296	Davenport 	 Iowa 	102612	99685	+2.94% 	62.9 sq mi 	162.9 km2 	1,631/sq mi 	630/km2 	41.5541°N 90.6040°W
+297	League City 	 Texas 	102010	83560	+22.08% 	51.2 sq mi 	132.6 km2 	1,992/sq mi 	769/km2 	29.4901°N 95.1091°W
+298	Edison[32] 	 New Jersey 	101996	99967	+2.03% 	30.1 sq mi 	78.0 km2 	3,389/sq mi 	1,309/km2 	40.5040°N 74.3494°W
+299	Davie[33] 	 Florida 	101871	91992	+10.74% 	34.9 sq mi 	90.4 km2 	2,919/sq mi 	1,127/km2 	26.0791°N 80.2850°W
+300	Las Cruces 	 New Mexico 	101759	97618	+4.24% 	76.9 sq mi 	199.2 km2 	1,323/sq mi 	511/km2 	32.3264°N 106.7897°W
+301	South Bend 	 Indiana 	101735	101168	+0.56% 	41.4 sq mi 	107.2 km2 	2,457/sq mi 	949/km2 	41.6769°N 86.2690°W
+302	Vista 	 California 	101659	93834	+8.34% 	18.7 sq mi 	48.4 km2 	5,436/sq mi 	2,099/km2 	33.1895°N 117.2386°W
+303	Woodbridge[32] 	 New Jersey 	101389	99585	+1.81% 	23.3 sq mi 	60.3 km2 	4,351/sq mi 	1,680/km2 	40.5607°N 74.2927°W
+304	Renton 	 Washington 	100953	90927	+11.03% 	23.4 sq mi 	60.6 km2 	4,314/sq mi 	1,666/km2 	47.4761°N 122.1920°W
+305	Lakewood[32] 	 New Jersey 	100758	92843	+8.53% 	24.7 sq mi 	64.0 km2 	4,079/sq mi 	1,575/km2 	40.0771°N 74.2004°W
+306	San Angelo 	 Texas 	100702	93200	+8.05% 	59.9 sq mi 	155.1 km2 	1,681/sq mi 	649/km2 	31.4411°N 100.4505°W
+307	Clinton[34] 	 Michigan 	100392	96796	+3.72% 	28.1 sq mi 	72.8 km2 	3,573/sq mi 	1,380/km2 	42.5903°N 82.9170°W

data/lib/dreader/version.rb

@@ -1,3 +1,3 @@
 module Dreader
-  VERSION = "0.1.2"
+  VERSION = "0.2.0"
 end

data/lib/dreader.rb

@@ -4,6 +4,10 @@ require 'roo'
 module Dreader
   # service class to implement the column DSL language
   class Column
+    def colref colref
+      @colref = colref
+    end
+
     def process &block
       @process = block
     end
@@ -13,7 +17,7 @@ module Dreader
     end
 
     def to_hash
-      {process: @process, check: @check }
+      {process: @process, check: @check, colref: @colref }
     end
   end
 
@@ -32,11 +36,67 @@ module Dreader
     end
   end
 
+  # Utilities function to simplify importing data into
+  # ActiveRecords
+  class Util
+    # given a hash returned by Engine, return the same hash with
+    # keys directly bound to the content of the :value sub-key
+    #
+    # Example
+    #
+    # hash = {name: {value: "A", ...}, surname: {value: "B", ...}}
+    # simplify hash
+    # {name: "A", surname: "B"}
+    def self.simplify hash
+      new_hash = {}
+      hash.keys.map { |k| new_hash[k] = hash[k][:value] }
+      new_hash
+    end
+    
+    # given a hash returned by Engine, keep the "kept" keys in the top
+    # of the hierarchy and move the "moved_key" below the
+    # "subordinate_key"
+    #
+    # Example
+    #
+    # hash = {name: "A", surname: "B", address: "via XX Settembre", city: "Genoa"}
+    # restructure hash, [:name, :surname], :address_attributes, [:address, :city]
+    # {name: "A", surname: "B", address_attributes: {address: "via XX Settembre", city: "Genoa"}}
+    #
+    def self.restructure hash, kept, subordinate_key, moved_keys
+      head = hash.slice kept
+      subordinate = self.prepend subordinate_key, hash.slice(moved_keys)
+      head.merge subordinate
+    end
+
+    # an alias for Hash.slice
+    def self.slice hash, *keys 
+      hash.slice keys
+    end
+
+    # remove all `keys` from `hash`
+    def self.clean hash, keys
+      hash.reject { |k, v| keys.include?(k) }
+    end
+
+    # given a hash, return a new hash with key and whose value is
+    # the hash
+    #
+    # Example:
+    #
+    #    hash = {name: "A", size: 10}
+    #    prepend hash, :product_attributes
+    #    {product_attributes: {name: "A", size: 10}}
+    #
+    def self.prepend hash, key
+      {key => hash}
+    end
+  end
+
   #
   # This is where the real stuff begins
   #
-  class Engine
-    
+  class Engine    
     # readable for debugging purposes
     # the options we passed 
     attr_reader :options
@@ -83,39 +143,73 @@ module Dreader
 
     # read a file and store it internally
     # return the data we read in the form of an array of hashes
-    def read filename=nil
+    def read filename = nil
       spreadsheet = Dreader::Engine.open_spreadsheet (filename || @options[:filename])
       sheet = spreadsheet.sheet(@options[:sheet] || 0)
 
-      @array = Array.new
+      @table = Array.new
       @errors = Array.new
-      first_row = @options[:start_at] || 1
-      (first_row..spreadsheet.last_row).each do |row_number|
-        row = spreadsheet.row(row_number)
+
+      first_row = @options[:first_row] || 1
+      last_row = @options[:last_row] || sheet.last_row
+
+      (first_row..last_row).each do |row_number|
 
         r = Hash.new
         @colspec.each_with_index do |colspec, index|
+          cell = sheet.cell(row_number, colspec[:colref])
+          
           colname = colspec[:name]
 
           r[colname] = Hash.new
-
           r[colname][:row_number] = row_number
-          r[colname][:col_number] = index + 1
+          r[colname][:col_number] = colspec[:colref]
 
-          r[colname][:value] = value = colspec[:process] ? colspec[:process].call(row[index]) : row[index]
+          r[colname][:value] = value = colspec[:process] ? colspec[:process].call(cell) : cell
 
           if colspec[:check] and not colspec[:check].call(value) then
             r[colname][:error] = true
-            @errors << "Error: value \"#{row[index]}\" for #{colname} at row #{row_number} (col #{index + 1}) is incorrect"
+            @errors << "Error: value \"#{cell}\" for #{colname} at row #{row_number} (col #{index + 1}) does not pass the check function"
           else
             r[colname][:error] = false
           end
         end
 
-        @array << r
+        @table << r
+      end
+
+      @table
+    end
+
+    alias_method :load, :read
+
+    # show to stdout the first `n` records we read from the file given the current
+    # configuration
+    def debug n = 10, filename = nil
+      spreadsheet = Dreader::Engine.open_spreadsheet (filename || @options[:filename])
+      sheet = spreadsheet.sheet(@options[:sheet] || 0)
+
+      puts "Current configuration:"
+      @options.each do |k, v|
+        puts "  #{k}: #{v}"
+      end
+      if filename and @options[:filename] and filename != @options[:filename]
+        puts "Warning: you asked me to load a file different from the one specified in the otptions."
       end
 
-      @array
+      first_row = @options[:first_row] || 1
+      puts "I will read records #{first_row} to #{n} from #{filename || @options[:filename]}"
+      last_row = first_row + n - 1
+      (first_row..last_row).each do |row_number|
+        puts "Record #{row_number} is:"
+        r = Hash.new
+        @colspec.each_with_index do |colspec, index|
+          cell = sheet.cell(row_number, colspec[:colref])
+          colname = colspec[:name]
+
+          puts "  Field #{colname}: column %d, value \"#{cell}\"" % colspec[:colref]
+        end
+      end
     end
 
     # return an array of strings with all the errors we have encounterd
@@ -127,24 +221,25 @@ module Dreader
     # apply the mapping code to the array
     # it makes sense to invoke it only 
     def process
-      @array.each do |r|
+      @table.each do |r|
         @mapping.call(r)
       end
     end
 
     def to_s
-      @array.to_s
+      @table.to_s
     end
 
     private
 
     def self.open_spreadsheet(filename)
       case File.extname(filename)
-      when ".csv" then Csv.new(filename)
+      when ".csv" then Roo::CSV.new(filename)
+      when ".tsv" then Roo::CSV.new(filename, csv_options: {col_sep: "\t"})
       when ".ods" then Roo::OpenOffice.new(filename)
       when ".xls" then Roo::Excel.new(filename)
       when ".xlsx" then Roo::Excelx.new(filename)
-      else raise "Unknown file type: #{filename}"
+      else raise "Unknown extension: #{File.extname(filename)}"
       end
     end
   end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: dreader
 version: !ruby/object:Gem::Version
-  version: 0.1.2
+  version: 0.2.0
 platform: ruby
 authors:
 - Adolfo Villafiorita
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2018-03-22 00:00:00.000000000 Z
+date: 2018-03-23 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -70,12 +70,17 @@ extra_rdoc_files: []
 files:
 - ".gitignore"
 - Gemfile
+- Gemfile.lock
 - LICENSE.txt
 - README.md
 - Rakefile
 - bin/console
 - bin/setup
 - dreader.gemspec
+- examples/wikipedia_big_us_cities/big_us_cities.rb
+- examples/wikipedia_big_us_cities/cities_by_state.ods
+- examples/wikipedia_us_cities/us_cities.rb
+- examples/wikipedia_us_cities/us_cities.tsv
 - lib/dreader.rb
 - lib/dreader/version.rb
 homepage: http://github.com/avillafiorita/dreader