csvreader 0.7.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: c61a8e62f99e1a06c119b4995e0e4e1d3c829d71
-  data.tar.gz: 1b59f3415f3f0fe449a8c2395d2cefc7a7bd855c
+  metadata.gz: aa8aec6ffb59bb3e27d09889ebd1294364d288eb
+  data.tar.gz: 913002d3c342651381bf38fc952b56913f2554da
 SHA512:
-  metadata.gz: 52e3e8effa09f492c38736f1fb16552341237ebe0e8e5452a72c080d03f48e47b9d66fbfa1b8f42e3ccd86d37038975254da0e3038f297174636a3b34ce57542
-  data.tar.gz: 66c17daaa22d6e5d526c76b9568737b4982da9f02b11e9264fdb9d9644d5e33881255f623a174515c78714352b2df54c811cb8b6e0222aa5cc79682ac320ee62
+  metadata.gz: 23d5bedb995926f464a4bd95e62c52eb50dd8ee109ae883b934c616b62cbbf9b9239f184e89027dda2fdf14d41af63c7d41a26f869951544c90ed6ad662be8b3
+  data.tar.gz: c212ad5acdc55f5105bd5c412fcfef36f18370282f84295b47ae517cbcf5f03d9bb78440709cf6137c657feb67028de266c18cd3cb577877547249413cc1783a

data/Manifest.txt

@@ -6,6 +6,7 @@ Rakefile
 lib/csvreader.rb
 lib/csvreader/buffer.rb
 lib/csvreader/builder.rb
+lib/csvreader/converter.rb
 lib/csvreader/parser.rb
 lib/csvreader/parser_std.rb
 lib/csvreader/parser_strict.rb
@@ -20,6 +21,7 @@ test/data/cities11.csv
 test/data/customers11.csv
 test/data/shakespeare.csv
 test/helper.rb
+test/test_converter.rb
 test/test_parser.rb
 test/test_parser_formats.rb
 test/test_parser_java.rb
@@ -27,4 +29,6 @@ test/test_parser_null.rb
 test/test_parser_strict.rb
 test/test_parser_tab.rb
 test/test_reader.rb
+test/test_reader_converters.rb
 test/test_reader_hash.rb
+test/test_reader_hash_converters.rb

data/README.md

@@ -40,6 +40,67 @@ end
 ```
 
 
+### What about converters?
+
+Use the converters keyword option to (auto-)convert strings to nulls, booleans, integers, floats, dates, etc.
+Example:
+
+``` ruby
+txt <<=TXT
+1,2,3
+true,false,null
+TXT
+
+records = Csv.parse( txt, :converters => :all )     ## or CsvReader.parse
+pp records
+# => [[1,2,3],
+#     [true,false,nil]]
+```
+
+
+Built-in converters include:
+
+| Converter    | Comments          |
+|--------------|-------------------|
+| `:integer`   |   convert matching strings to integer |
+| `:float`     |   convert matching strings to float   |
+| `:numeric`   |   shortcut for `[:integer, :float]`   |
+| `:date`      |   convert matching strings to `Date` (year/month/day) |
+| `:date_time` |   convert matching strings to `DateTime` |
+| `:null`      |   convert matching strings to null (`nil`) |
+| `:boolean`   |   convert matching strings to boolean (`true` or `false`) |
+| `:all`       |   shortcut for `[:null, :boolean, :date_time, :numeric]` |
+
+
+
+### What about Enumerable?
+
+Yes, every reader includes `Enumerable` and runs on `each`.
+Use `new` or `open` without a block
+to get the enumerator (iterator).
+Example:
+
+
+``` ruby
+csv = Csv.new( "a,b,c" )
+it  = csv.to_enum
+pp it.next  
+# => ["a","b","c"]
+
+# -or-
+
+csv = Csv.open( "values.csv" )
+it  = csv.to_enum
+pp it.next
+# => ["1","2","3"]
+pp it.next
+# => ["5","6","7"]
+```
+
+
+
+
+
 ### What about headers?
 
 Use the `CsvHash`
@@ -87,6 +148,41 @@ end
 ```
 
 
+### What about symbol keys for hashes?
+
+Yes, use can use the header_converters keyword option.
+Use `:symbol` for (auto-)converting header (strings) to symbols.
+Note: the symbol converter will also downcase all letters and
+remove all non-alphanumeric (e.g. `!?$%`) chars
+and replace spaces with underscores.
+
+Example:
+
+``` ruby
+txt <<=TXT
+a,b,c
+1,2,3
+true,false,null
+TXT
+
+records = CsvHash.parse( txt, :converters => :all, :header_converters => :symbol )  
+pp records
+# => [{a: 1,    b: 2,     c: 3},
+#     {a: true, b: false, c: nil}]
+```
+
+Built-in header converters include:
+
+| Converter    | Comments            |
+|--------------|---------------------|
+| `:downcase`  |   downcase strings  |
+| `:symbol`    |   convert strings to symbols (and downcase and remove non-alphanumerics) |
+
+
+
+
+
+
 
 ## Frequently Asked Questions (FAQ) and Answers
 
@@ -128,25 +224,94 @@ Staatliches Hofbräuhaus München,München,Hofbräu Oktoberfestbier,6.3%
 
 
 
-### Q: How can I change the separator to semicolon (`;`) or pipe (`|`)?
+### Q: How can I change the default format / dialect?
+
+The reader includes more than half a dozen pre-configured formats,
+dialects.
 
-Pass in the `sep` keyword option. Example:
+Use strict if you do NOT want to trim leading and trailing spaces
+and if you do NOT want to skip blank lines. Example:
 
 ``` ruby
-Csv.parse( ..., sep: ';' )
-Csv.read( ..., sep: ';' )
-# ...
-Csv.parse( ..., sep: '|' )
-Csv.read( ..., sep: '|' )
+txt <<=TXT
+1, 2,3
+4,5 ,6
+
+TXT
+
+records = Csv.strict.parse( txt )
+pp records
+# => [["1","•2","3"],
+#     ["4","5•","6"],
+#     [""]]
+```
+
+More strict pre-configured variants include:
+
+`Csv.mysql` uses:
+
+``` ruby
+ParserStrict.new( sep: "\t",
+                  quote: false,
+                  escape: true,
+                  null: "\\N" )
+```
+
+`Csv.postgres` or `Csv.postgresql` uses:
+
+``` ruby
+ParserStrict.new( doublequote: false,
+                  escape: true,
+                  null: "" )
+```
+
+`Csv.postgres_text` or `Csv.postgresql_text` uses:
+
+``` ruby
+ParserStrict.new( sep: "\t",
+                  quote: false,
+                  escape: true,
+                  null: "\\N" )
+```
+
+and so on.
+
+
+### Q: How can I change the separator to semicolon (`;`) or pipe (`|`) or tab (`\t`)?
+
+Pass in the `sep` keyword option
+to the "strict" parser. Example:
+
+``` ruby
+Csv.strict.parse( ..., sep: ';' )
+Csv.strict.read( ..., sep: ';' )
 # ...
+Csv.strict.parse( ..., sep: '|' )
+Csv.strict.read( ..., sep: '|' )
 # and so on
 ```
 
-
-Note: If you use tab (`\t`) use the `TabReader`! Why? Tab =! CSV. Yes, tab is
+Note: If you use tab (`\t`) use the `TabReader`
+(or for your convenience the built-in `Csv.tab` alias)!
+Why? Tab =! CSV. Yes, tab is
 its own (even) simpler format
 (e.g. no escape rules, no newlines in values, etc.),
-see [`TabReader` »](https://github.com/datatxt/tabreader).
+see [`TabReader` »](https://github.com/csv11/tabreader).
+
+``` ruby
+Csv.tab.parse( ... )  # note: "classic" strict tab format
+Csv.tab.read( ... )
+# ...
+```
+
+If you want double quote escape rules, newlines in quotes values, etc. use
+the "strict" parser with the separator (`sep`) changed to tab (`\t`).
+
+``` ruby
+Csv.strict.parse( ..., sep: "\t" )  # note: csv-like tab format with quotes
+Csv.strict.read( ..., sep: "\t" )
+# ...
+```
 
 
 

data/lib/csvreader.rb

@@ -4,6 +4,8 @@
 require 'pp'
 require 'logger'
 require 'forwardable'
+require 'stringio'
+require 'date'    ## use for Date.parse and DateTime.parse
 
 
 ###
@@ -17,6 +19,113 @@ require 'csvreader/parser'
 require 'csvreader/builder'
 require 'csvreader/reader'
 require 'csvreader/reader_hash'
+require 'csvreader/converter'
+
+
+
+class CsvReader
+class Parser
+
+  ## use/allow different "backends" e.g. ParserStd, ParserStrict, ParserTab, etc.
+  ##   parser must support parse method (with and without block)
+  ##    e.g.  records = parse( data )
+  ##             -or-
+  ##          parse( data ) do |record|
+  ##          end
+
+  DEFAULT = ParserStd.new
+
+  RFC4180 = ParserStrict.new
+  STRICT  = ParserStrict.new  ## note: make strict its own instance (so you can change config without "breaking" rfc4180)
+  EXCEL   = ParserStrict.new   ## note: make excel its own instance (so you can change configs without "breaking" rfc4180/strict)
+
+  MYSQL   = ParserStrict.new( sep: "\t",
+                              quote: false,
+                              escape: true,
+                              null: "\\N" )
+
+  POSTGRES = POSTGRESQL = ParserStrict.new( doublequote: false,
+                                            escape: true,
+                                            null: "" )
+
+  POSTGRES_TEXT = POSTGRESQL_TEXT = ParserStrict.new( sep: "\t",
+                                                      quote: false,
+                                                      escape: true,
+                                                      null: "\\N" )
+
+  TAB     = ParserTab.new
+
+
+  def self.default()         DEFAULT;         end ## alternative alias for DEFAULT
+  def self.strict()          STRICT;          end ## alternative alias for STRICT
+  def self.rfc4180()         RFC4180;         end ## alternative alias for RFC4180
+  def self.excel()           EXCEL;           end ## alternative alias for EXCEL
+  def self.mysql()           MYSQL;           end
+  def self.postgresql()      POSTGRESQL;      end
+  def self.postgres()        postgresql;      end
+  def self.postgresql_text() POSTGRESQL_TEXT; end
+  def self.postgres_text()   postgresql_text; end
+  def self.tab()             TAB;             end
+end # class Parser
+end # class CsvReader
+
+
+
+class CsvReader
+  ### pre-define CsvReader (built-in) formats/dialect
+  DEFAULT = CsvBuilder.new( Parser::DEFAULT )
+
+  STRICT  = CsvBuilder.new( Parser::STRICT )
+  RFC4180 = CsvBuilder.new( Parser::RFC4180 )
+  EXCEL   = CsvBuilder.new( Parser::EXCEL )
+
+  MYSQL                           = CsvBuilder.new( Parser::MYSQL )
+  POSTGRES = POSTGRESQL           = CsvBuilder.new( Parser::POSTGRESQL )
+  POSTGRES_TEXT = POSTGRESQL_TEXT = CsvBuilder.new( Parser::POSTGRESQL_TEXT )
+
+  TAB = CsvBuilder.new( Parser::TAB )
+
+
+  def self.default()         DEFAULT;         end ## alternative alias for DEFAULT
+  def self.strict()          STRICT;          end ## alternative alias for STRICT
+  def self.rfc4180()         RFC4180;         end ## alternative alias for RFC4180
+  def self.excel()           EXCEL;           end ## alternative alias for EXCEL
+  def self.mysql()           MYSQL;           end
+  def self.postgresql()      POSTGRESQL;      end
+  def self.postgres()        postgresql;      end
+  def self.postgresql_text() POSTGRESQL_TEXT; end
+  def self.postgres_text()   postgresql_text; end
+  def self.tab()             TAB;             end
+end # class CsvReader
+
+
+class CsvHashReader
+  ### pre-define CsvReader (built-in) formats/dialect
+  DEFAULT = CsvHashBuilder.new( CsvReader::Parser::DEFAULT )
+
+  STRICT  = CsvHashBuilder.new( CsvReader::Parser::STRICT )
+  RFC4180 = CsvHashBuilder.new( CsvReader::Parser::RFC4180 )
+  EXCEL   = CsvHashBuilder.new( CsvReader::Parser::EXCEL )
+
+  MYSQL                           = CsvHashBuilder.new( CsvReader::Parser::MYSQL )
+  POSTGRES = POSTGRESQL           = CsvHashBuilder.new( CsvReader::Parser::POSTGRESQL )
+  POSTGRES_TEXT = POSTGRESQL_TEXT = CsvHashBuilder.new( CsvReader::Parser::POSTGRESQL_TEXT )
+
+  TAB = CsvHashBuilder.new( CsvReader::Parser::TAB )
+
+
+  def self.default()         DEFAULT;         end ## alternative alias for DEFAULT
+  def self.strict()          STRICT;          end ## alternative alias for STRICT
+  def self.rfc4180()         RFC4180;         end ## alternative alias for RFC4180
+  def self.excel()           EXCEL;           end ## alternative alias for EXCEL
+  def self.mysql()           MYSQL;           end
+  def self.postgresql()      POSTGRESQL;      end
+  def self.postgres()        postgresql;      end
+  def self.postgresql_text() POSTGRESQL_TEXT; end
+  def self.postgres_text()   postgresql_text; end
+  def self.tab()             TAB;             end
+end # class CsvHashReader
+
 
 
 

data/lib/csvreader/builder.rb

@@ -2,21 +2,21 @@
 
 
 class CsvBuilder  ## rename to CsvReaderBuilder - why? why not?
+
+
   def initialize( parser )
     @parser = parser
   end
 
+  def config() @parser.config; end   ## (auto-)forward to wrapped parser
 
   ## todo/fix:
   ##   add parser config (attribute) setter e.g.
   ##   - sep=(value)
   ##   - comment=(value)
   ##   - and so on!!!
-  ##
-  ##   add config too - why? why not?
-
 
-  def open( path, mode='r:bom|utf-8',
+  def open( path, mode=nil,
                  sep: nil,
                  converters: nil,
                  parser: @parser, &block )
@@ -54,3 +54,67 @@ class CsvBuilder  ## rename to CsvReaderBuilder - why? why not?
                    parser: @parser, &block )
   end
 end # class CsvBuilder
+
+
+
+
+class CsvHashBuilder  ## rename to CsvHashReaderBuilder - why? why not?
+  def initialize( parser )
+    @parser = parser
+  end
+
+  def config() @parser.config; end   ## (auto-)forward to wrapped parser
+
+  ## todo/fix:
+  ##   add parser config (attribute) setter e.g.
+  ##   - sep=(value)
+  ##   - comment=(value)
+  ##   - and so on!!!
+
+
+  def open( path, mode=nil,
+                 headers: nil,
+                 sep: nil,
+                 converters: nil,
+                 header_converters: nil,
+                 parser: @parser, &block )
+      CsvHashReader.open( path, mode,
+                      headers: headers, sep: sep, converters: converters,
+                      header_converters: header_converters,
+                      parser: @parser, &block )
+  end
+
+  def read( path, headers: nil,
+                  sep: nil,
+                  converters: nil,
+                  header_converters: nil )
+    CsvHashReader.read( path,
+                  headers: headers,
+                  sep: sep, converters: converters,
+                  header_converters: header_converters,
+                  parser: @parser )
+  end
+
+  def foreach( path, headers: nil,
+                     sep: nil,
+                     converters: nil,
+                     header_converters: nil, &block )
+    CsvHashReader.foreach( path,
+                     headers: headers,
+                     sep: sep, converters: converters,
+                     header_converters: header_converters,
+                     parser: @parser, &block )
+  end
+
+
+  def parse( data, headers: nil,
+                   sep: nil,
+                   converters: nil,
+                   header_converters: nil, &block )
+    CsvHashReader.parse( data,
+                   headers: headers,
+                   sep: sep, converters: converters,
+                   header_converters: header_converters,
+                   parser: @parser, &block )
+  end
+end # class CsvHashBuilder