remote_table 1.4.0 → 2.0.0

data/CHANGELOG

@@ -1,3 +1,20 @@
+2.0.0 / 2012-05-08
+
+* Breaking changes
+
+  * New names for options... (not really breaking, these deprecated options are still accepted)
+    :errata -> :errata_settings
+    :transform -> :transform_settings
+    :select -> :pre_select (to avoid conflict with Enumerable#select)
+    :reject -> :pre_reject
+    :encoding -> :internal_encoding
+
+* Enhancements
+
+  * Every option is documented
+  * Refactored to simplify and DRY
+  * Thread safe
+
 1.4.0 / 2012-04-12
 
 * Enhancements

data/README.markdown

@@ -1,25 +1,38 @@
 # remote_table
 
-Open local or remote XLSX, XLS, ODS, CSV and fixed-width files.
+Open Google Docs spreadsheets, local or remote XLSX, XLS, ODS, CSV (comma separated), TSV (tab separated), other delimited, fixed-width files.
 
-## Production usage
+Tested on MRI 1.8, MRI 1.9, and JRuby 1.6.7+. Thread-safe.
 
-Used by [the Brighter Planet Reference Data web service](http://data.brighterplanet.com), the [`data_miner` gem](https://github.com/seamusabshere/data_miner), and the [`earth` gem](https://github.com/brighterplanet/earth).
+## Real-world usage
+
+<p><a href="http://brighterplanet.com"><img src="https://s3.amazonaws.com/static.brighterplanet.com/assets/logos/flush-left/inline/green/rasterized/brighter_planet-160-transparent.png" alt="Brighter Planet logo"/></a></p>
+
+We use `remote_table` for [data science at Brighter Planet](http://brighterplanet.com/research) and in production at
+
+* [Brighter Planet's impact estimate web service](http://impact.brighterplanet.com)
+* [Brighter Planet's reference data web service](http://data.brighterplanet.com)
+
+It's also a big part of
+
+* the [`data_miner`](https://github.com/seamusabshere/data_miner) library
+* the [`earth`](https://github.com/brighterplanet/earth) library
 
 ## Example
 
-    $ irb
-    1.9.3-p0 :001 > require 'remote_table'
+    >> require 'remote_table'
+    remote_table.rb:8:in `<top (required)>': iconv will be deprecated in the future, use String#encode instead.
+    [remote_table] Apologies - using iconv because Ruby 1.9.x's String#encode doesn't have transliteration tables (yet)
     => true
-    1.9.3-p0 :002 > t = RemoteTable.new 'http://www.fueleconomy.gov/FEG/epadata/98guide6.zip', :filename => '98guide6.csv'
-    => #<RemoteTable:0x00000100851d98 @options={:filename=>"98guide6.csv"}, @url="http://www.fueleconomy.gov/FEG/epadata/98guide6.zip">
-    1.9.3-p0 :003 > t.rows.length
+    >> t = RemoteTable.new 'http://www.fueleconomy.gov/FEG/epadata/98guide6.zip', :filename => '98guide6.csv'
+    => #<RemoteTable:0x00000101b87390 @download_count_mutex=#<Mutex:0x00000101b87228>, @iconv_mutex=#<Mutex:0x00000101b87200>, @extend_bang_mutex=#<Mutex:0x00000101b871d8>, @errata_mutex=#<Mutex:0x00000101b871b0>, @cache=[], @download_count=0, @url="http://www.fueleconomy.gov/FEG/epadata/98guide6.zip", @format=nil, @headers=:first_row, @compression=:zip, @packing=nil, @streaming=false, @warn_on_multiple_downloads=true, @delimiter=",", @sheet=nil, @keep_blank_rows=false, @form_data=nil, @skip=0, @internal_encoding="UTF-8", @row_xpath=nil, @column_xpath=nil, @row_css=nil, @column_css=nil, @glob=nil, @filename="98guide6.csv", @transform_settings=nil, @cut=nil, @crop=nil, @schema=nil, @schema_name=nil, @pre_select=nil, @pre_reject=nil, @errata_settings=nil, @other_options={}, @transformer=#<RemoteTable::Transformer:0x00000101b8c2f0 @t=#<RemoteTable:0x00000101b87390 ...>, @legacy_transformer_mutex=#<Mutex:0x00000101b8c2a0>>, @local_copy=#<RemoteTable::LocalCopy:0x00000101b8bf58 @t=#<RemoteTable:0x00000101b87390 ...>, @encoded_io_mutex=#<Mutex:0x00000101b8be18>, @generate_mutex=#<Mutex:0x00000101b8bdc8>>>
+    >> t.rows.length
     => 806
-    1.9.3-p0 :004 > t.rows.first.length
+    >> t.rows.first.length
     => 26
-    1.9.3-p0 :005 > require 'pp'
+    >> require 'pp'
     => true
-    1.9.3-p0 :006 > pp t[23]
+    >> pp t[23]
     {"Class"=>"TWO SEATERS",
      "Manufacturer"=>"PORSCHE",
      "carline name"=>"BOXSTER",
@@ -47,7 +60,19 @@ Used by [the Brighter Planet Reference Data web service](http://data.brighterpla
      "eng dscr"=>"",
      "trans dscr"=>""}
 
-You get an <code>Array</code> of <code>Hash</code>es with **string keys**. If you set <code>:headers => false</code>, then you get an <code>Array</code> of <code>Array</code>s.
+## Columns and rows
+
+* If there are headers, you get an <code>Array</code> of <code>Hash</code>es with **string keys**.
+* If you set <code>:headers => false</code>, then you get an <code>Array</code> of <code>Array</code>s.
+
+## Row keys
+
+Row keys are **strings**. Row keys are NOT symbolized.
+
+    row['foobar'] # correct
+    row[:foobar]  # incorrect
+
+You can call <code>symbolize_keys</code> yourself, but we don't do it automatically to avoid creating loads of garbage symbols.
 
 ## Supported formats
 
@@ -59,7 +84,7 @@ You get an <code>Array</code> of <code>Hash</code>es with **string keys**. If yo
   </tr>
   <tr>
     <td>Delimited (CSV, TSV, etc.)</td>
-    <td>All <code>RemoteTable::Format::Delimited::FASTERCSV_OPTIONS</code>, for example <code>:col_sep</code>, are passed directly to fastercsv.</td>
+    <td>All <code>RemoteTable::Delimited::PASSTHROUGH_CSV_SETTINGS</code>, for example <code>:col_sep</code>, are passed directly to fastercsv.</td>
     <td>
       <a href="http://fastercsv.rubyforge.org/">fastercsv</a> (1.8);
       <a href="http://www.ruby-doc.org/stdlib-1.9.3/libdoc/csv/rdoc/index.html">stdlib</code></a> (1.9)
@@ -147,12 +172,12 @@ Everything is forced into UTF-8. You can improve the quality of the conversion b
                     :row_xpath => '//table/tr[2]/td/table/tr',
                     :column_xpath => 'td',
                     :errata => { RemoteTable.new('https://spreadsheets.google.com/spreadsheet/pub?key=0AoQJbWqPrREqdGVBRnhkRGhSaVptSDJ5bXJGbkpUSWc&output=csv', :responder => Aircraft::Guru.new },
-                    :select => lambda { |record| manufacturer_whitelist? record['Manufacturer'] })
+                    :select => proc { |record| manufacturer_whitelist? record['Manufacturer'] })
 
     # OpenFlights.org airports database
     RemoteTable.new('https://openflights.svn.sourceforge.net/svnroot/openflights/openflights/data/airports.dat',
                     :headers => %w{ id name city country_name iata_code icao_code latitude longitude altitude timezone daylight_savings },
-                    :select => lambda { |record| record['iata_code'].present? },
+                    :select => proc { |record| record['iata_code'].present? },
                     :errata => { RemoteTable.new('https://spreadsheets.google.com/pub?key=0AoQJbWqPrREqdFc2UzhQYU5PWEQ0N21yWFZGNmc2a3c&gid=0&output=csv', :responder => Airport::Guru.new }) # see https://github.com/brighterplanet/earth/blob/master/lib/earth/air/aircraft/data_miner.rb
 
     # T100 flight segment data for #{month.strftime('%B %Y')}
@@ -162,7 +187,7 @@ Everything is forced into UTF-8. You can improve the quality of the conversion b
                     :compression => :zip,
                     :glob => '/*.csv',
                     :errata => { RemoteTable.new('https://spreadsheets.google.com/spreadsheet/pub?key=0AoQJbWqPrREqdGxpYU1qWFR3d0syTVMyQVVOaDd0V3c&output=csv', :responder => FlightSegment::Guru.new },
-                    :select => lambda { |record| record['DEPARTURES_PERFORMED'].to_i > 0 })
+                    :select => proc { |record| record['DEPARTURES_PERFORMED'].to_i > 0 })
 
     # 1995 Fuel Economy Guide
     # for definition of `:fuel_economy_guide_b` and `AutomobileMakeModelYearVariant::ParserB` see https://github.com/brighterplanet/earth/blob/master/lib/earth/automobile/automobile_make_model_year_variant/data_miner.rb
@@ -171,7 +196,7 @@ Everything is forced into UTF-8. You can improve the quality of the conversion b
                     :format => :fixed_width,
                     :cut => '13-',
                     :schema_name => :fuel_economy_guide_b,
-                    :select => lambda { |row| row['model'].present? and (row['suppress_code'].blank? or row['suppress_code'].to_f == 0) and row['state_code'] == 'F' },
+                    :select => proc { |row| row['model'].present? and (row['suppress_code'].blank? or row['suppress_code'].to_f == 0) and row['state_code'] == 'F' },
                     :transform => { :class => AutomobileMakeModelYearVariant::ParserB, :year => 1995 },
                     :errata => { :url => "https://docs.google.com/spreadsheet/pub?key=0AoQJbWqPrREqdDkxTElWRVlvUXB3Uy04SDhSYWkzakE&output=csv", :responder => AutomobileMakeModelYearVariant::Guru.new })
 
@@ -181,30 +206,30 @@ Everything is forced into UTF-8. You can improve the quality of the conversion b
                     :filename => '98guide6.csv',
                     :transform => { :class => AutomobileMakeModelYearVariant::ParserC, :year => 1998 },
                     :errata => { :url => "https://docs.google.com/spreadsheet/pub?key=0AoQJbWqPrREqdDkxTElWRVlvUXB3Uy04SDhSYWkzakE&output=csv", :responder => AutomobileMakeModelYearVariant::Guru.new },
-                    :select => lambda { |row| row['model'].present? })
+                    :select => proc { |row| row['model'].present? })
 
     # annual corporate average fuel economy data for domestic and imported vehicle fleets from the NHTSA
     RemoteTable.new('https://spreadsheets.google.com/pub?key=0AoQJbWqPrREqdEdXWXB6dkVLWkowLXhYSFVUT01sS2c&hl=en&gid=0&output=csv',
                     :errata => { 'url' => 'http://static.brighterplanet.com/science/data/transport/automobiles/make_fleet_years/errata.csv' },
-                    :select => lambda { |row| row['volume'].to_i > 0 })
+                    :select => proc { |row| row['volume'].to_i > 0 })
 
     # total vehicle miles travelled by gasoline passenger cars from the 2010 EPA GHG Inventory
     RemoteTable.new('http://www.epa.gov/climatechange/emissions/downloads10/2010-Inventory-Annex-Tables.zip',
                     :filename => 'Annex Tables/Annex 3/Table A-87.csv',
                     :skip => 1,
-                    :select => lambda { |row| row['Year'].to_i.to_s == row['Year'] })
+                    :select => proc { |row| row['Year'].to_i.to_s == row['Year'] })
 
     # total vehicle miles travelled from the 2010 EPA GHG Inventory
     RemoteTable.new('http://www.epa.gov/climatechange/emissions/downloads10/2010-Inventory-Annex-Tables.zip',
                     :filename => 'Annex Tables/Annex 3/Table A-87.csv',
                     :skip => 1,
-                    :select => lambda { |row| row['Year'].to_i.to_s == row['Year'] })
+                    :select => proc { |row| row['Year'].to_i.to_s == row['Year'] })
 
     # total travel distribution from the 2010 EPA GHG Inventory
     RemoteTable.new('http://www.epa.gov/climatechange/emissions/downloads10/2010-Inventory-Annex-Tables.zip',
                     :filename => 'Annex Tables/Annex 3/Table A-93.csv',
                     :skip => 1,
-                    :select => lambda { |row| row['Vehicle Age'].to_i.to_s == row['Vehicle Age'] })
+                    :select => proc { |row| row['Vehicle Age'].to_i.to_s == row['Vehicle Age'] })
 
     # building characteristics from the 2003 EIA Commercial Building Energy Consumption Survey
     RemoteTable.new('http://www.eia.gov/emeu/cbecs/cbecs2003/public_use_2003/data/FILE02.csv',
@@ -215,7 +240,7 @@ Everything is forced into UTF-8. You can improve the quality of the conversion b
     # for definition of `CbecsEnergyIntensity::NAICS_CODE_SYNTHESIZER` see https://github.com/brighterplanet/earth/blob/master/lib/earth/industry/cbecs_energy_intensity/data_miner.rb
     RemoteTable.new("http://www.eia.gov/emeu/cbecs/cbecs2003/detailed_tables_2003/2003set10/2003excel/C17.xls",
                     :headers => false,
-                    :select => ::Proc.new { |row| CbecsEnergyIntensity::NAICS_CODE_SYNTHESIZER.call(row) },
+                    :select => proc { |row| CbecsEnergyIntensity::NAICS_CODE_SYNTHESIZER.call(row) },
                     :crop => (21..37))
 
     # U.S. Census 2002 NAICS code list
@@ -238,13 +263,13 @@ Everything is forced into UTF-8. You can improve the quality of the conversion b
     RemoteTable.new('http://www.census.gov/popest/about/geo/state_geocodes_v2009.txt',
                     :skip => 6,
                     :headers => %w{ Region Division FIPS Name },
-                    :select => ::Proc.new { |row| row['Division'].to_i > 0 and row['FIPS'].to_i == 0 })
+                    :select => proc { |row| row['Division'].to_i > 0 and row['FIPS'].to_i == 0 })
 
     # state census divisions from the U.S. Census
     RemoteTable.new('http://www.census.gov/popest/about/geo/state_geocodes_v2009.txt',
                     :skip => 8,
                     :headers => ['Region', 'Division', 'State FIPS', 'Name'],
-                    :select => ::Proc.new { |row| row['State FIPS'].to_i > 0 })
+                    :select => proc { |row| row['State FIPS'].to_i > 0 })
 
     # OpenGeoCode.org's Country Codes to Country Names list
     RemoteTable.new('http://opengeocode.org/download/countrynames.txt',
@@ -267,19 +292,19 @@ Everything is forced into UTF-8. You can improve the quality of the conversion b
     RemoteTable.new('http://www.epa.gov/cleanenergy/documents/egridzips/eGRID2010V1_1_STIE_USGC.xls',
                     :sheet => 'STIE07',
                     :skip => 4,
-                    :select => lambda { |row| row['eGRID2010 year 2007 file state sequence number'].to_i.between?(1, 51) })
+                    :select => proc { |row| row['eGRID2010 year 2007 file state sequence number'].to_i.between?(1, 51) })
 
     # eGRID 2010 subregions and electricity emission factors
     RemoteTable.new('http://www.epa.gov/cleanenergy/documents/egridzips/eGRID2010_Version1-1_xls_only.zip',
                     :filename => 'eGRID2010V1_1_year07_AGGREGATION.xls',
                     :sheet => 'SRL07',
                     :skip => 4,
-                    :select => lambda { |row| row['SEQSRL07'].to_i.between?(1, 26) })
+                    :select => proc { |row| row['SEQSRL07'].to_i.between?(1, 26) })
 
     # U.S. Census State ANSI Code file
     RemoteTable.new('http://www.census.gov/geo/www/ansi/state.txt',
                     :delimiter => '|',
-                    :select => lambda { |record| record['STATE'].to_i < 60 })
+                    :select => proc { |record| record['STATE'].to_i < 60 })
 
     # Mapping Hacks zipcode database
     RemoteTable.new('http://mappinghacks.com/data/zipcode.zip',
@@ -295,18 +320,18 @@ Everything is forced into UTF-8. You can improve the quality of the conversion b
     # Brighter Planet's list of cat and dog breeds, genders, and weights
     RemoteTable.new('http://static.brighterplanet.com/science/data/consumables/pets/breed_genders.csv',
                     :encoding => 'ISO-8859-1',
-                    :select => lambda { |row| row['gender'].present? })
+                    :select => proc { |row| row['gender'].present? })
 
     # residential electricity prices from the EIA
     RemoteTable.new('http://www.eia.doe.gov/cneaf/electricity/page/sales_revenue.xls',
-                    :select => lambda { |row| row['Year'].to_s.first(4).to_i > 1989 })
+                    :select => proc { |row| row['Year'].to_s.first(4).to_i > 1989 })
 
     # residential natural gas prices from the EIA
     # for definition of `NaturalGasParser` see https://github.com/brighterplanet/earth/blob/master/lib/earth/residence/residence_fuel_price/data_miner.rb
     RemoteTable.new('http://tonto.eia.doe.gov/dnav/ng/xls/ng_pri_sum_a_EPG0_FWA_DMcf_a.xls',
                     :sheet => 'Data 1',
                     :skip => 2,
-                    :select => lambda { |row| row['year'].to_i > 1989 },
+                    :select => proc { |row| row['year'].to_i > 1989 },
                     :transform => { :class => NaturalGasParser })
 
     # 2005 EIA Residential Energy Consumption Survey microdata
@@ -375,7 +400,7 @@ Everything is forced into UTF-8. You can improve the quality of the conversion b
                     :format => :fixed_width,
                     :crop => 21..26, # inclusive
                     :cut => '2-',
-                    :select => lambda { |row| /\A[A-Z]/.match row['code'] },
+                    :select => proc { |row| /\A[A-Z]/.match row['code'] },
                     :schema => [[ 'code',   2, { :type => :string }  ],
                                 [ 'spacer', 2 ],
                                 [ 'name',   52, { :type => :string } ]]
@@ -420,14 +445,11 @@ Everything is forced into UTF-8. You can improve the quality of the conversion b
 
 ## Requirements
 
-* MRI (not JRuby)
-* Unix tools like curl, iconv, perl, cat, cut, tail, etc. accessible from `ENV['PATH']`
-
-As this library matures, that requirement should go away.
+* Unix tools like curl, iconv, perl, cat, cut, tail, etc. accessible from your `$PATH`
 
 ## Wishlist
 
-* JRuby and Win32 compat
+* Win32 compat
 * The new "custom parser" syntax (aka transformer) hasn't been defined yet... only the old-style syntax is available
 
 ## Authors

data/lib/remote_table.rb

@@ -3,6 +3,14 @@ if ::RUBY_VERSION < '1.9' and $KCODE != 'UTF8'
   $KCODE = 'UTF8'
 end
 
+require 'thread'
+
+require 'iconv'
+if RUBY_VERSION >= '1.9'
+  # for an excellent explanation see http://blog.segment7.net/2010/12/17/from-iconv-iconv-to-string-encode
+  Kernel.warn "[remote_table] Apologies - using iconv because Ruby 1.9.x's String#encode doesn't have transliteration tables (yet)"
+end
+
 require 'active_support'
 require 'active_support/version'
 if ::ActiveSupport::VERSION::MAJOR >= 3
@@ -10,78 +18,432 @@ if ::ActiveSupport::VERSION::MAJOR >= 3
 end
 require 'hash_digest'
 
-require 'remote_table/format'
-require 'remote_table/config'
-require 'remote_table/local_file'
+require 'remote_table/local_copy'
 require 'remote_table/transformer'
 
+require 'remote_table/plaintext'
+require 'remote_table/processed_by_roo'
+require 'remote_table/processed_by_nokogiri'
+require 'remote_table/xls'
+require 'remote_table/xlsx'
+require 'remote_table/delimited'
+require 'remote_table/ods'
+require 'remote_table/fixed_width'
+require 'remote_table/html'
+require 'remote_table/xml'
+require 'remote_table/yaml'
+
 class Hash
+  # Added by remote_table to store a hash (think checksum) of the data with which a particular Hash is initialized.
+  # @return [String]
   attr_accessor :row_hash
 end
 
 class Array
+  # Added by remote_table to store a hash (think checksum) of the data with which a particular Array is initialized.
+  # @return [String]
   attr_accessor :row_hash
 end
 
-class RemoteTable  
-  # Legacy
+# Open Google Docs spreadsheets, local or remote XLSX, XLS, ODS, CSV (comma separated), TSV (tab separated), other delimited, fixed-width files.
+class RemoteTable
+  class << self
+    # Guess compression based on URL. Used internally.
+    # @return [Symbol,nil]
+    def guess_compression(url)
+      extname = ::File.extname(::URI.parse(url).path).downcase
+      case extname
+      when /gz/, /gunzip/
+        :gz
+      when /zip/
+        :zip
+      when /bz2/, /bunzip2/
+        :bz2
+      when /exe/
+        :exe
+      end
+    end
+
+    # Guess packing from URL. Used internally.
+    # @return [Symbol,nil]
+    def guess_packing(url)
+      basename = ::File.basename(::URI.parse(url).path).downcase
+      if basename.include?('.tar') or basename.include?('.tgz')
+        :tar
+      end
+    end
+
+    # Guess file format from the basename. Since a file might be decompressed and/or pulled out of an archive with a glob, this usually can't be called until a file is downloaded.
+    # @return [Symbol,nil]
+    def guess_format(basename)
+      case basename.to_s.downcase
+      when /ods/, /open_?office/
+        :ods
+      when /xlsx/, /excelx/
+        :xlsx
+      when /xls/, /excel/
+        :xls
+      when /csv/, /tsv/, /delimited/
+        # note that there is no RemoteTable::Csv class - it's normalized to :delimited
+        :delimited
+      when /fixed_?width/
+        :fixed_width
+      when /htm/
+        :html
+      when /xml/
+        :xml
+      when /yaml/, /yml/
+        :yaml
+      end
+    end
+
+    # Given a Google Docs spreadsheet URL, make sure it uses CSV output.
+    # @return [String]
+    def google_spreadsheet_csv_url(url)
+      uri = ::URI.parse url
+      params = uri.query.split('&')
+      params.delete_if { |param| param.start_with?('output=') }
+      params << 'output=csv'
+      uri.query = params.join('&')
+      uri.to_s
+    end
+  end
+
+  # @private
+  # Here to support legacy code.
   class Transform
     def self.row_hash(row)
       ::HashDigest.hexdigest row
     end
   end
 
+  EXTERNAL_ENCODING = 'UTF-8'
+  EXTERNAL_ENCODING_ICONV = 'UTF-8//TRANSLIT'
+  GOOGLE_DOCS_SPREADSHEET = [
+    /docs.google.com/i,
+    /spreadsheets.google.com/i
+  ]
+  VALID = {
+    :compression => [:gz, :zip, :bz2, :exe],
+    :packing => [:tar],
+    :format => [:xlsx, :xls, :delimited, :ods, :fixed_width, :html, :xml, :yaml, :csv],
+  }
+  DEFAULT = {
+    :streaming => false,
+    :warn_on_multiple_downloads => true,
+    :headers => :first_row,
+    :keep_blank_rows => false,
+    :skip => 0,
+    :internal_encoding => 'UTF-8',
+    :delimiter => ','
+  }
+  OLD_SETTING_NAMES = {
+    :internal_encoding => [:encoding],
+    :transform_settings => [:transform],
+    :pre_select => [:select],
+    :pre_reject => [:reject],
+    :errata_settings => [:errata],
+  }
+
   include ::Enumerable
   
+  # The URL of the local or remote file.
+  #
+  # * Local: "file:///Users/myuser/Desktop/holidays.csv"
+  # * Remote: "http://data.brighterplanet.com/countries.csv"
+  #
+  # @return [String]
   attr_reader :url
-  attr_reader :config
+
+  # @private
+  # A cache of rows, created unless +:streaming+ is enabled.
+  # @return [Array<Hash,Array>]
+  attr_reader :cache
+
+  # @private
+  # How many times this file has been downloaded. RemoteTable will emit a warning if you download it more than once.
+  # @return [Integer]
+  attr_reader :download_count
+
+  # @private
+  # Used internally to access the transformer (aka parser).
+  attr_reader :transformer
+
+  # @private
+  # Used internally to access to a downloaded copy of the file.
+  # @return [RemoteTable::LocalCopy]
+  attr_reader :local_copy
+
+  # Whether to stream the rows without caching them. Saves memory, but you have to re-download the file every time you enumerate its rows. Defaults to false.
+  # @return [true,false]
+  attr_reader :streaming
+
+  # Whether to warn the user on multiple downloads. Defaults to true.
+  # @return [true,false]
+  attr_reader :warn_on_multiple_downloads
+  
+  # Headers specified by the user: +:first_row+ (the default), +false+, or a list of headers.
+  # @return [:first_row,false,Array<String>]
+  attr_reader :headers
+    
+  # The sheet specified by the user as a number or a string.
+  # @return[String,Integer]
+  attr_reader :sheet
+  
+  # Whether to keep blank rows. Default is false.
+  # @return [true,false]
+  attr_reader :keep_blank_rows
+  
+  # Form data to POST in the download request. It should probably be in +application/x-www-form-urlencoded+.
+  # @return [String]
+  attr_reader :form_data
+  
+  # How many rows to skip at the beginning of the file or table. Default is 0.
+  # @return [Integer]
+  attr_reader :skip
+
+  # The original encoding of the source file. Default is UTF-8. Previously passed as +:encoding+.
+  # @return [String]
+  attr_reader :internal_encoding
+  
+  # The delimiter, a.k.a. column separator. Passed to Ruby CSV as +:col_sep+. Default is :delimited.
+  # @return [String]
+  attr_reader :delimiter
+  
+  # The XPath used to find rows in HTML or XML.
+  # @return [String]
+  attr_reader :row_xpath
+  
+  # The XPath used to find columns in HTML or XML.
+  # @return [String]
+  attr_reader :column_xpath
+
+  # The CSS selector used to find rows in HTML or XML.
+  # @return [String]
+  attr_reader :row_css
+  
+  # The CSS selector used to find columns in HTML or XML.
+  # @return [String]
+  attr_reader :column_css
+  
+  # The format of the source file. Can be +:xlsx+, +:xls+, +:delimited+, +:ods+, +:fixed_width+, +:html+, +:xml+, +:yaml+.
+  # @return [Symbol]
+  attr_reader :format
+
+  # The compression type. Guessed from URL if not provided. +:gz+, +:zip+, +:bz2+, and +:exe+ (treated as +:zip+) are supported.
+  # @return [Symbol]
+  attr_reader :compression
+
+  # The packing type. Guessed from URL if not provided. Only +:tar+ is supported.
+  # @return [Symbol]
+  attr_reader :packing
+  
+  # The glob used to pick a file out of an archive.
+  #
+  # @return [String]
+  #
+  # @example Pick out the only CSV in a ZIP file
+  #   RemoteTable.new 'http://www.fueleconomy.gov/FEG/epadata/08data.zip', :glob => '/*.csv'
+  attr_reader :glob
+  
+  # The filename, which can be used to pick a file out of an archive.
+  #
+  # @return [String]
+  #
+  # @example Specify the filename to get out of a ZIP file
+  #   RemoteTable.new 'http://www.fueleconomy.gov/FEG/epadata/08data.zip', :filename => '2008_FE_guide_ALL_rel_dates_-no sales-for DOE-5-1-08.csv'
+  attr_reader :filename
+
+  # Pick specific columns out of a plaintext file using an argument to the UNIX [+cut+ utility](http://en.wikipedia.org/wiki/Cut_%28Unix%29).
+  #
+  # @return [String]
+  #
+  # @example Pick ALMOST out of ABCDEFGHIJKLMNOPQRSTUVWXYZ
+  #   # $ echo ABCDEFGHIJKLMNOPQRSTUVWXYZ | cut -c '1,12,13,15,19,20'
+  #   # ALMOST
+  #   RemoteTable.new 'file:///atoz.txt', :cut => '1,12,13,15,19,20'
+  attr_reader :cut
+  
+  # Use a range of rows in a plaintext file.
+  #
+  # @return [Range]
+  #
+  # @example Only take rows 21 through 37
+  #   RemoteTable.new("http://www.eia.gov/emeu/cbecs/cbecs2003/detailed_tables_2003/2003set10/2003excel/C17.xls",
+  #                   :headers => false,
+  #                   :select => proc { |row| CbecsEnergyIntensity::NAICS_CODE_SYNTHESIZER.call(row) },
+  #                   :crop => (21..37))
+  attr_reader :crop
+  
+  # The fixed-width schema, given as a multi-dimensional array.
+  #
+  # @return [Array<Array{String,Integer,Hash}>]
+  #
+  # @example From the tests
+  #   RemoteTable.new('http://cloud.github.com/downloads/seamusabshere/remote_table/test2.fixed_width.txt',
+  #                    :format => :fixed_width,
+  #                    :skip => 1,
+  #                    :schema => [[ 'header4', 10, { :type => :string }  ],
+  #                                [  'spacer',  1 ],
+  #                                [  'header5', 10, { :type => :string } ],
+  #                                [  'spacer',  12 ],
+  #                                [  'header6', 10, { :type => :string } ]])
+  attr_reader :schema
   
-  # Create a new RemoteTable.
+  # If you somehow already defined a fixed-width schema (so you can re-use it?), specify it here.
+  # @return [String,Symbol]
+  attr_reader :schema_name
+  
+  # A proc that decides whether to include a row. Previously passed as +:select+.
+  # @return [Proc]
+  attr_reader :pre_select
+  
+  # A proc that decides whether to include a row. Previously passed as +:reject+.
+  # @return [Proc]
+  attr_reader :pre_reject
+
+  # Settings to create a transformer.
+  # @return [Hash]
+  attr_reader :transform_settings
+  
+  # A hash of settings to initialize an Errata instance to be used on every row. Previously passed as +:errata+.
+  #
+  # See the Errata library at https://github.com/seamusabshere/errata
+  #
+  # @return [Hash]
+  attr_reader :errata_settings
+  
+  # The format of the source file. Can be specified as: :xlsx, :xls, :delimited (aka :csv), :ods, :fixed_width, :html, :xml, :yaml
+  #
+  # Note: treats all +docs.google.com+ and +spreadsheets.google.com+ URLs as +:delimited+.
   #
-  #     RemoteTable.new(url, options = {})
+  # Default: guessed from file extension (which is usually the same as the URL, but sometimes not if you pick out a specific file from an archive)
   #
-  # New syntax:
-  #     RemoteTable.new('www.customerreferenceprogram.org/uploads/CRP_RFP_template.xlsx', :foo => 'bar')
-  # Old syntax:
-  #     RemoteTable.new(:url => 'www.customerreferenceprogram.org/uploads/CRP_RFP_template.xlsx', :foo => 'bar')
+  # @return [Hash]
+  attr_reader :format
+
+  # Options passed by the user that may be passed through to the underlying parsing library.
+  # @return [Hash]
+  attr_reader :other_options
+
+  # Create a new RemoteTable, which is an Enumerable.
+  #
+  # Does not immediately download/parse... it's lazy-loading.
+  #
+  # @overload initialize(settings)
+  #   @param [Hash] settings Settings including +:url+.
   #
-  # See the <tt>Config</tt> object for the sorts of options you can pass.
+  # @overload initialize(url, settings)
+  #   @param [String] url The URL to the local or remote file.
+  #   @param [Hash] settings Settings.
+  #
+  # @example Open an XLSX
+  #   RemoteTable.new('http://www.customerreferenceprogram.org/uploads/CRP_RFP_template.xlsx')
+  #
+  # @example Open a CSV inside a ZIP file
+  #   RemoteTable.new 'http://www.epa.gov/climatechange/emissions/downloads10/2010-Inventory-Annex-Tables.zip',
+  #                   :filename => 'Annex Tables/Annex 3/Table A-93.csv',
+  #                   :skip => 1,
+  #                   :pre_select => proc { |row| row['Vehicle Age'].strip =~ /^\d+$/ }
   def initialize(*args)
-    options = args.last.is_a?(::Hash) ? args.last.symbolize_keys : {}
-    
+    @download_count_mutex = ::Mutex.new
+    @iconv_mutex = ::Mutex.new
+    @extend_bang_mutex = ::Mutex.new
+    @errata_mutex = ::Mutex.new
+
+    @cache = []
+    @download_count = 0
+
+    settings = args.last.is_a?(::Hash) ? args.last.symbolize_keys : {}
+
     @url = if args.first.is_a? ::String
-      args.first.dup
+      args.first
     else
-      options[:url].dup
+      grab settings, :url
+    end
+    @format = RemoteTable.guess_format grab(settings, :format)
+    if GOOGLE_DOCS_SPREADSHEET.any? { |regex| regex =~ url }
+      @url = RemoteTable.google_spreadsheet_csv_url url
+      @format = :delimited
+    end
+
+    @headers = grab settings, :headers
+    if headers.is_a?(::Array) and headers.any?(&:blank?)
+      raise ::ArgumentError, "[remote_table] If you specify headers, none of them can be blank"
     end
-    @config = Config.new self, options
+
+    @compression = grab(settings, :compression) || RemoteTable.guess_compression(url)
+    @packing = grab(settings, :packing) || RemoteTable.guess_packing(url)
+
+    @streaming = grab settings, :streaming
+    @warn_on_multiple_downloads = grab settings, :warn_on_multiple_downloads
+    @delimiter = grab settings, :delimiter
+    @sheet = grab settings, :sheet
+    @keep_blank_rows = grab settings, :keep_blank_rows
+    @form_data = grab settings, :form_data
+    @skip = grab settings, :skip
+    @internal_encoding = grab settings, :internal_encoding
+    @row_xpath = grab settings, :row_xpath
+    @column_xpath = grab settings, :column_xpath
+    @row_css = grab settings, :row_css
+    @column_css = grab settings, :column_css
+    @glob = grab settings, :glob
+    @filename = grab settings, :filename
+    @transform_settings = grab settings, :transform_settings
+    @cut = grab settings, :cut
+    @crop = grab settings, :crop
+    @schema = grab settings, :schema
+    @schema_name = grab settings, :schema_name
+    @pre_select = grab settings, :pre_select
+    @pre_reject = grab settings, :pre_reject
+    @errata_settings = grab settings, :errata_settings
+
+    @other_options = settings
+    
+    @transformer = Transformer.new self
+    @local_copy = LocalCopy.new self
   end
-  
-  # not thread safe
-  def each(&blk)
+
+  # Yield each row.
+  #
+  # @return [nil]
+  #
+  # @yield [Hash,Array] A hash or an array depending on whether the RemoteTable has named headers (column names).
+  def each
+    extend!
     if fully_cached?
-      cache.each(&blk)
+      cache.each do |row|
+        yield row
+      end
     else
       mark_download!
-      retval = format.each do |row|
+      memo = _each do |row|
         transformer.transform(row).each do |virtual_row|
           virtual_row.row_hash = ::HashDigest.hexdigest row
-          if config.errata
-            next if config.errata.rejects? virtual_row
-            config.errata.correct! virtual_row
+          if errata
+            next if errata.rejects? virtual_row
+            errata.correct! virtual_row
+          end
+          next if pre_select and !pre_select.call(virtual_row)
+          next if pre_reject and pre_reject.call(virtual_row)
+          unless streaming
+            cache.push virtual_row
           end
-          next if config.select and !config.select.call(virtual_row)
-          next if config.reject and config.reject.call(virtual_row)
-          cache.push virtual_row unless config.streaming
           yield virtual_row
         end
       end
-      fully_cached! unless config.streaming
-      retval
+      unless streaming
+        fully_cached!
+      end
+      memo
     end
+    nil
   end
+
+  # @deprecated
   alias :each_row :each
   
+  # @return [Array<Hash,Array>] All rows.
   def to_a
     if fully_cached?
       cache.dup
@@ -89,9 +451,13 @@ class RemoteTable
       map { |row| row }
     end
   end
+
+  # @deprecated
   alias :rows :to_a
   
-  # Get a row by row number
+  # Get a row by row number. Zero-based.
+  #
+  # @return [Hash,Array]
   def [](row_number)
     if fully_cached?
       cache[row_number]
@@ -100,35 +466,37 @@ class RemoteTable
     end
   end
   
-  # clear the row cache to save memory
+  # Clear the row cache in case it helps your GC.
+  #
+  # @return [nil]
   def free
+    @fully_cached = false
+    @errata = nil
     cache.clear
     nil
   end
-  
-  # Used internally to access to a downloaded copy of the file
-  def local_file
-    @local_file ||= LocalFile.new self
-  end
-  
-  # Used internally to access to the driver that reads the format
-  def format
-    @format ||= config.format.new self
-  end
-  
-  # Used internally to access the transformer (aka parser).
-  def transformer
-    @transformer ||= Transformer.new self
+
+  # @private
+  def errata
+    @errata || @errata_mutex.synchronize do
+      @errata ||= begin
+        if defined?(::Errata) and errata_settings.is_a?(::Errata)
+          ::Kernel.warn %{[remote_table] Passing :errata_settings as an Errata object is deprecated. Please pass a Hash of settings instead.}
+          errata_settings
+        elsif errata_settings.is_a?(::Hash)
+          ::Errata.new errata_settings
+        end
+      end
+    end
   end
-  
-  attr_reader :download_count
-  
+
   private
   
   def mark_download!
-    @download_count ||= 0
-    @download_count += 1
-    if config.warn_on_multiple_downloads and download_count > 1
+    @download_count_mutex.synchronize do
+      @download_count += 1
+    end
+    if warn_on_multiple_downloads and download_count > 1
       ::Kernel.warn "[remote_table] #{url} has been downloaded #{download_count} times."
     end
   end 
@@ -140,8 +508,62 @@ class RemoteTable
   def fully_cached?
     !!@fully_cached
   end
-  
-  def cache
-    @cache ||= []
+
+  def iconv
+    @iconv || @iconv_mutex.synchronize do
+      @iconv ||= ::Iconv.new(EXTERNAL_ENCODING_ICONV, internal_encoding)
+    end
+  end
+
+  def transliterate_to_utf8(str)
+    if str.is_a?(::String)
+      [ iconv.iconv(str), iconv.iconv(nil) ].join
+    end
+  end
+
+  def assume_utf8(str)
+    if str.is_a?(::String) and ::RUBY_VERSION >= '1.9'
+      str.encode! EXTERNAL_ENCODING
+    else
+      str
+    end
+  end
+
+  def grab(settings, k)
+    user_specified = false
+    memo = nil
+    if (old_names = OLD_SETTING_NAMES[k]) and old_names.any? { |old_k| settings.has_key?(old_k) }
+      user_specified = true
+      memo = old_names.map { |old_k| settings.delete(old_k) }.compact.first
+    end
+    if settings.has_key?(k)
+      user_specified = true
+      memo = settings.delete k
+    end
+    if not user_specified and DEFAULT.has_key?(k)
+      memo = DEFAULT[k]
+    end
+    if memo and (valid = VALID[k]) and not valid.include?(memo.to_sym)
+      raise ::ArgumentError, %{[remote_table] #{k.inspect} => #{memo.inspect} is not a valid setting. Valid settings are #{valid.inspect}.}
+    end
+    memo
+  end
+
+  def extend!
+    return if @extend_bang
+    @extend_bang_mutex.synchronize do
+      return if @extend_bang
+      @extend_bang = true
+      format_module = if format
+        RemoteTable.const_get format.to_s.camelcase
+      elsif format = RemoteTable.guess_format(local_copy.path)
+        @format = format
+        RemoteTable.const_get format.to_s.camelcase
+      else
+        Delimited
+      end
+      extend format_module
+      after_extend if respond_to?(:after_extend)
+    end
   end
 end