csvlint 0.1.4 → 0.2.0

checksums.yaml

@@ -1,15 +1,15 @@
 ---
 !binary "U0hBMQ==":
   metadata.gz: !binary |-
-    NTcwMjkzODNkNWM4ZDg1ZjU2MjQzZjdhYzJhNDE0OGFlZWZmMjQ0ZA==
+    YjlmZmFlNGZjOWQ5MmNlNDZiOTUxMWY0NGExYTRkYjhhNzdlNjAyNA==
   data.tar.gz: !binary |-
-    N2I0YWJjYzgzM2UxZDk1ODdkY2M0ZTRmNGU2ZDI3ZGNhMTgxNGFiMA==
+    ODFjZmJkZmI0Nzg2NmMzN2ViOGNiNDlmODA0NDcxMzM0Zjk4NTgwOQ==
 SHA512:
   metadata.gz: !binary |-
-    NDM5MmU5MDEwNjQyOWFlMGUwNWFmNGNjNTc0YTU4NmE0ZWY2YjMyMDRkODY2
-    NmIyODJhMzEyOGIyYjlkMTQwM2FhYjJmNzU3NGVjY2E3OTgyMjg0NWRkZGEx
-    YzU1MjAyYjJhYzUxNGFmZTc2YmVkYjQ0MjI3ZjdkZmNlYzJkZjM=
+    ZTIyMGVkYjIyMjc2ZWViNTBhYmZkMWIxN2E1OTU0OTFhNGMxNzBlYzg0OTI4
+    NDRkMzY2YzgxNmQwZGZiZDE5M2M2NzYwMzk3ZWZjMDc3YWM0YzQ0NTczY2U3
+    MGZjNTUwMGI2MzgzZDQxYzkzMzBiNzI3NmJkZTIxYjZiYjc5MDA=
   data.tar.gz: !binary |-
-    MzVjNWU4MDFjYWEyNWNhNWY0OGM3NTUxNmU4MzAxZTYwMGYyZTBiOTdmYzhl
-    MmJiNWM0OTQ0YWRlZWY3MDAyN2RjN2RmMWIyNTk5NWJlODgwZDVkODUwN2E0
-    NTJlYjMxNTg5Y2I4ZDcyNzUyN2EzNjNkNmRiOTRiMzlkOTdhZjg=
+    NTI1M2I5Yzc3NGNhOTg3Y2VkMmM3ZGM1ZTdiZWNmMzM0ZTY5ODljODNmNWYy
+    MDA0NGVlMGFhNDQ2ZjZjYjI0Nzc2OTdhMWRmODI5YTEzMGRmNTQxZjAyOTA5
+    YjVmMjk4NDIyOWEzMzIxMTBlYjQ4YTgwZmE4MWZlYTQ4MjMzZmE=

data/.gitignore

@@ -19,4 +19,10 @@ coverage/
 /.rspec
 
 .idea
-.DS_Store
+.DS_Store
+features/csvw_validation_tests.feature
+features/fixtures/csvw
+
+bin/run-csvw-tests
+
+features/csvw_json_transformation_tests.feature

data/CHANGELOG.md

@@ -2,7 +2,25 @@
 
 ## [Unreleased](https://github.com/theodi/csvlint.rb/tree/HEAD)
 
-[Full Changelog](https://github.com/theodi/csvlint.rb/compare/0.1.3...HEAD)
+[Full Changelog](https://github.com/theodi/csvlint.rb/compare/0.1.4...HEAD)
+
+**Closed issues:**
+
+- CSV on the web support [\#141](https://github.com/theodi/csvlint.rb/issues/141)
+
+**Merged pull requests:**
+
+- Recover from `ArgumentError`s when attempting to locate a schema and detect bad schema when JSON is malformed [\#152](https://github.com/theodi/csvlint.rb/pull/152) ([pezholio](https://github.com/pezholio))
+
+- Catch errors if link headers are don't have particular values [\#151](https://github.com/theodi/csvlint.rb/pull/151) ([pezholio](https://github.com/pezholio))
+
+- Rescue excel warning [\#149](https://github.com/theodi/csvlint.rb/pull/149) ([quadrophobiac](https://github.com/quadrophobiac))
+
+- CSVW-based validation! [\#142](https://github.com/theodi/csvlint.rb/pull/142) ([JeniT](https://github.com/JeniT))
+
+## [0.1.4](https://github.com/theodi/csvlint.rb/tree/0.1.4) (2015-08-06)
+
+[Full Changelog](https://github.com/theodi/csvlint.rb/compare/0.1.3...0.1.4)
 
 **Merged pull requests:**
 

data/README.md

@@ -31,13 +31,13 @@ You can either use this gem within your own Ruby code, or as a standolone comman
 After installing the gem, you can validate a CSV on the command line like so:
 
 	csvlint myfile.csv
-	
+
 You will then see the validation result, together with any warnings or errors e.g.
 
 ```
 myfile.csv is INVALID
 1. blank_rows. Row: 3
-1. title_row. 
+1. title_row.
 2. inconsistent_values. Column: 14
 ```
 
@@ -50,40 +50,40 @@ You can also optionally pass a schema file like so:
 Currently the gem supports retrieving a CSV accessible from a URL, File, or an IO-style object (e.g. StringIO)
 
 	require 'csvlint'
-	
+
 	validator = Csvlint::Validator.new( "http://example.org/data.csv" )
 	validator = Csvlint::Validator.new( File.new("/path/to/my/data.csv" ))
 	validator = Csvlint::Validator.new( StringIO.new( my_data_in_a_string ) )
 
-When validating from a URL the range of errors and warnings is wider as the library will also check HTTP headers for 
+When validating from a URL the range of errors and warnings is wider as the library will also check HTTP headers for
 best practices
-	
-	#invoke the validation	
+
+	#invoke the validation
 	validator.validate
-	
+
 	#check validation status
 	validator.valid?
-	
+
 	#access array of errors, each is an Csvlint::ErrorMessage object
 	validator.errors
-	
+
 	#access array of warnings
 	validator.warnings
-	
+
 	#access array of information messages
 	validator.info_messages
-	
+
 	#get some information about the CSV file that was validated
 	validator.encoding
 	validator.content_type
 	validator.extension
-	
+
 	#retrieve HTTP headers from request
 	validator.headers
 
 ## Controlling CSV Parsing
 
-The validator supports configuration of the [CSV Dialect](http://dataprotocols.org/csv-dialect/) used in a data file. This is specified by 
+The validator supports configuration of the [CSV Dialect](http://dataprotocols.org/csv-dialect/) used in a data file. This is specified by
 passing a dialect hash to the constructor:
 
     dialect = {
@@ -94,17 +94,17 @@ passing a dialect hash to the constructor:
 
 The options should be a Hash that conforms to the [CSV Dialect](http://dataprotocols.org/csv-dialect/) JSON structure.
 
-While these options configure the parser to correctly process the file, the validator will still raise errors or warnings for CSV 
+While these options configure the parser to correctly process the file, the validator will still raise errors or warnings for CSV
 structure that it considers to be invalid, e.g. a missing header or different delimiters.
 
-Note that the parser will also check for a `header` parameter on the `Content-Type` header returned when fetching a remote CSV file. As 
+Note that the parser will also check for a `header` parameter on the `Content-Type` header returned when fetching a remote CSV file. As
 specified in [RFC 4180](http://www.ietf.org/rfc/rfc4180.txt) the values for this can be `present` and `absent`, e.g:
 
 	Content-Type: text/csv; header=present
 
 ## Error Reporting
 
-The validator provides feedback on a validation result using instances of `Csvlint::ErrorMessage`. Errors are divided into errors, warnings and information 
+The validator provides feedback on a validation result using instances of `Csvlint::ErrorMessage`. Errors are divided into errors, warnings and information
 messages. A validation attempt is successful if there are no errors.
 
 Messages provide context including:
@@ -122,7 +122,7 @@ The following types of error can be reported:
 * `:wrong_content_type` -- content type is not `text/csv`
 * `:ragged_rows` -- row has a different number of columns (than the first row in the file)
 * `:blank_rows` -- completely empty row, e.g. blank line or a line where all column values are empty
-* `:invalid_encoding` -- encoding error when parsing row, e.g. because of invalid characters 
+* `:invalid_encoding` -- encoding error when parsing row, e.g. because of invalid characters
 * `:not_found` -- HTTP 404 error when retrieving the data
 * `:stray_quote` -- missing or stray quote
 * `:unclosed_quote` -- unclosed quoted field
@@ -153,36 +153,66 @@ There are also information messages available:
 
 ## Schema Validation
 
-The library supports validating data against a schema. A schema configuration can be provided as a Hash or parsed from JSON. The structure currently 
-follows JSON Table Schema with some extensions.
+The library supports validating data against a schema. A schema configuration can be provided as a Hash or parsed from JSON. The structure currently
+follows JSON Table Schema with some extensions and rudinmentary [CSV on the Web Metadata](http://www.w3.org/TR/tabular-metadata/).
 
-An example schema file is:
+An example JSON Table Schema schema file is:
 
 	{
 		"fields": [
-			{ 
-				"name": "id", 
-			  	"constraints": { "required": true } 
+			{
+				"name": "id",
+			  	"constraints": { "required": true }
 			},
-            { 
-               	"name": "price", 
-               	"constraints": { "required": true, "minLength": 1 } 
+            {
+               	"name": "price",
+               	"constraints": { "required": true, "minLength": 1 }
             },
-            { 
-            	"name": "postcode", 
-            	"constraints": { 
-            		"required": true, 
-            		"pattern": "[A-Z]{1,2}[0-9][0-9A-Z]? ?[0-9][A-Z]{2}" 
-            	} 
+            {
+            	"name": "postcode",
+            	"constraints": {
+            		"required": true,
+            		"pattern": "[A-Z]{1,2}[0-9][0-9A-Z]? ?[0-9][A-Z]{2}"
+            	}
             }
         ]
 	}
 
-Parsing and validating with a schema:
+An equivalent CSV on the Web Metadata file is:
+
+	{
+		"@context": "http://www.w3.org/ns/csvw",
+		"url": "http://example.com/example1.csv",
+		"tableSchema": {
+			"columns": [
+				{
+					"name": "id",
+					"required": true
+				},
+				{
+					"name": "price",
+					"required": true,
+					"datatype": { "base": "string", "minLength": 1 }
+				},
+				{
+					"name": "postcode",
+					"required": true
+				}
+			]
+		}
+	}
 
-	schema = Csvlint::Schema.load_from_json_table(uri)
+Parsing and validating with a schema (of either kind):
+
+	schema = Csvlint::Schema.load_from_json(uri)
 	validator = Csvlint::Validator.new( "http://example.org/data.csv", nil, schema )
 
+### CSV on the Web Validation Support
+
+This gem passes all the validation tests in the [official CSV on the Web test suite](http://w3c.github.io/csvw/tests/) (though there might still be errors or parts of the [CSV on the Web standard](http://www.w3.org/TR/tabular-metadata/) that aren't tested by that test suite).
+
+### JSON Table Schema Support
+
 Supported constraints:
 
 * `required` -- there must be a value for this field in every row
@@ -192,7 +222,7 @@ Supported constraints:
 * `pattern` -- values must match the provided regular expression
 * `type` -- specifies an XML Schema data type. Values of the column must be a valid value for that type
 * `minimum` -- specify a minimum range for values, the value will be parsed as specified by `type`
-* `maximum` -- specify a maximum range for values, the value will be parsed as specified by `type` 
+* `maximum` -- specify a maximum range for values, the value will be parsed as specified by `type`
 * `datePattern` -- specify a `strftime` compatible date pattern to be used when parsing date values and min/max constraints
 
 Supported data types (this is still a work in progress):
@@ -214,7 +244,7 @@ Supported data types (this is still a work in progress):
 * Time -- `http://www.w3.org/2001/XMLSchema#time`
 
 Use of an unknown data type will result in the column failing to validate.
-	
+
 Schema validation provides some additional types of error and warning messages:
 
 * `:missing_value` (error) -- a column marked as `required` in the schema has no value
@@ -248,3 +278,30 @@ validator = Csvlint::Validator.new( "http://example.org/data.csv", nil, nil, opt
 3. Commit your changes (`git commit -am 'Add some feature'`)
 4. Push to the branch (`git push origin my-new-feature`)
 5. Create new Pull Request
+
+### Testing
+
+The codebase includes both rspec and cucumber tests, which can be run together using:
+
+    $ rake
+
+or separately:
+
+    $ rake spec
+    $ rake features
+
+When the cucumber tests are first run, a script will create tests based on the latest version of the [CSV on the Web test suite](http://w3c.github.io/csvw/tests/), including creating a local cache of the test files. This requires an internet connection and some patience. Following that download, the tests will run locally; there's also a batch script:
+
+    $ bin/run-csvw-tests
+
+which will run the tests from the command line.
+
+If you need to refresh the CSV on the Web tests:
+
+    $ rm bin/run-csvw-tests
+    $ rm features/csvw_validation_tests.feature
+    $ rm -r features/fixtures/csvw
+
+and then run the cucumber tests again or:
+
+    $ ruby features/support/load_tests.rb

data/bin/csvlint

@@ -16,8 +16,8 @@ opts.on("-d", "--dump-errors", "Pretty print error and warning objects.") do |d|
   options[:dump] = d
 end
 
-opts.on("-s", "--schema-file FILENAME", "Schema file") do |s|
-  options[:schema_file] = s
+opts.on("-s", "--schema FILENAME", "Schema file") do |s|
+  options[:schema] = s
 end
 
 opts.on_tail("-h", "--help",
@@ -35,14 +35,15 @@ rescue OptionParser::InvalidOption => e
 end
 
 def print_error(index, error, dump, color)
-
   location = ""
   location += error.row.to_s if error.row
   location += "#{error.row ? "," : ""}#{error.column.to_s}" if error.column
   if error.row || error.column
     location = "#{error.row ? "Row" : "Column"}: #{location}"
   end
-  output_string = "#{index+1}. #{error.type}. #{location}"
+  output_string = "#{index+1}. #{error.type}"
+  output_string += ". #{location}" unless location.empty?
+  output_string += ". #{error.content}" if error.content
 
   if $stdout.tty?
     puts output_string.colorize(color)
@@ -56,6 +57,30 @@ def print_error(index, error, dump, color)
 
 end
 
+def validate_csv(source, schema, dump)
+  validator = Csvlint::Validator.new( source, nil, schema )
+
+  if $stdout.tty?
+    puts "#{source.path || source || "CSV"} is #{validator.valid? ? "VALID".green : "INVALID".red}"
+  else
+    puts "#{source.path || source || "CSV"} is #{validator.valid? ? "VALID" : "INVALID"}"
+  end
+
+  if validator.errors.size > 0
+    validator.errors.each_with_index do |error, i|
+      print_error(i, error, dump, :red)
+    end
+  end
+
+  if validator.warnings.size > 0
+    validator.warnings.each_with_index do |error, i|
+      print_error(i, error, dump, :yellow)
+    end
+  end
+
+  return validator.valid?
+end
+
 if ARGV.length == 0 && !$stdin.tty?
   source = StringIO.new(ARGF.read)
 else
@@ -63,13 +88,13 @@ else
     source = ARGV[0]
     unless source =~ /^http(s)?/
       begin
-        source = File.new( source ) unless source =~ /^http(s)?/ 
+        source = File.new( source ) unless source =~ /^http(s)?/
       rescue Errno::ENOENT
         puts "#{source} not found"
         exit 1
       end
     end
-  else
+  elsif !options[:schema]
     puts "No CSV data to validate."
     puts opts
     exit 1
@@ -77,34 +102,50 @@ else
 end
 
 schema = nil
-if options[:schema_file]
+if options[:schema]
   begin
-    schemafile = File.read( options[:schema_file] )
+    schema = Csvlint::Schema.load_from_json(options[:schema])
+  rescue JSON::ParserError => e
+    output_string = "invalid metadata: malformed JSON"
+    if $stdout.tty?
+      puts output_string.colorize(:red)
+    else
+      puts output_string
+    end
+    exit 1
+  rescue Csvlint::Csvw::MetadataError => e
+    output_string = "invalid metadata: #{e.message}#{" at " + e.path if e.path}"
+    if $stdout.tty?
+      puts output_string.colorize(:red)
+    else
+      puts output_string
+    end
+    exit 1
   rescue Errno::ENOENT
-    puts "#{options[:schema_file]} not found"
+    puts "#{options[:schema]} not found"
     exit 1
   end
-  schema = Csvlint::Schema.from_json_table(nil, JSON.parse(schemafile))
-end
-
-validator = Csvlint::Validator.new( source, nil, schema )
-
-if $stdout.tty?
-  puts "#{ARGV[0] || "CSV"} is #{validator.valid? ? "VALID".green : "INVALID".red}"
-else
-  puts "#{ARGV[0] || "CSV"} is #{validator.valid? ? "VALID" : "INVALID"}"
 end
 
-if validator.errors.size > 0
-  validator.errors.each_with_index do |error, i|
-    print_error(i, error, options[:dump], :red)
+valid = true
+if source.nil?
+  unless schema.instance_of? Csvlint::Csvw::TableGroup
+    puts "No CSV data to validate."
+    puts opts
+    exit 1
   end
-end
-
-if validator.warnings.size > 0
-  validator.warnings.each_with_index do |error, i|
-    print_error(i, error, options[:dump], :yellow)
+  schema.tables.keys.each do |source|
+    begin
+      source = source.sub("file:","")
+      source = File.new( source )
+    rescue Errno::ENOENT
+      puts "#{source} not found"
+      exit 1
+    end unless source =~ /^http(s)?/
+    valid &= validate_csv(source, schema, options[:dump])
   end
+else
+  valid = validate_csv(source, schema, options[:dump])
 end
 
-exit 1 unless validator.valid?
+exit 1 unless valid

data/csvlint.gemspec

@@ -23,6 +23,8 @@ Gem::Specification.new do |spec|
   spec.add_dependency "open_uri_redirections"
   spec.add_dependency "activesupport"
   spec.add_dependency "addressable"
+  spec.add_dependency "escape_utils"
+  spec.add_dependency "uri_template"
 
   spec.add_development_dependency "bundler", "~> 1.3"
   spec.add_development_dependency "rake"

data/features/csvw_schema_validation.feature

@@ -0,0 +1,127 @@
+Feature: CSVW Schema Validation
+
+  Scenario: Valid CSV
+    Given I have a CSV with the following content:
+    """
+"Bob","1234","bob@example.org"
+"Alice","5","alice@example.com"
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    And I have metadata with the following content:
+    """
+{
+  "@context": "http://www.w3.org/ns/csvw",
+  "url": "http://example.com/example1.csv",
+  "dialect": { "header": false },
+  "tableSchema": {
+    "columns": [
+            { "name": "Name", "required": true },
+            { "name": "Id", "required": true, "datatype": { "base": "string", "minLength": 1 } },
+            { "name": "Email", "required": true }
+      ]
+  }
+}
+    """
+    When I ask if there are errors
+    Then there should be 0 error
+
+  Scenario: Schema invalid CSV
+    Given I have a CSV with the following content:
+    """
+"Bob","1234","bob@example.org"
+"Alice","5","alice@example.com"
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    And I have metadata with the following content:
+    """
+{
+  "@context": "http://www.w3.org/ns/csvw",
+  "url": "http://example.com/example1.csv",
+  "dialect": { "header": false },
+  "tableSchema": {
+    "columns": [
+            { "name": "Name", "required": true },
+            { "name": "Id", "required": true, "datatype": { "base": "string", "minLength": 3 } },
+            { "name": "Email", "required": true }
+      ]
+  }
+}
+    """
+    When I ask if there are errors
+    Then there should be 1 error
+
+  Scenario: CSV with incorrect header
+    Given I have a CSV with the following content:
+    """
+"name","id","contact"
+"Bob","1234","bob@example.org"
+"Alice","5","alice@example.com"
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    And I have metadata with the following content:
+    """
+{
+  "@context": "http://www.w3.org/ns/csvw",
+  "url": "http://example.com/example1.csv",
+  "tableSchema": {
+    "columns": [
+            { "titles": "name", "required": true },
+            { "titles": "id", "required": true, "datatype": { "base": "string", "minLength": 1 } },
+            { "titles": "email", "required": true }
+      ]
+  }
+}
+    """
+    When I ask if there are errors
+    Then there should be 1 error
+
+  Scenario: Schema with valid regex
+    Given I have a CSV with the following content:
+    """
+  "firstname","id","email"
+  "Bob","1234","bob@example.org"
+  "Alice","5","alice@example.com"
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    And I have metadata with the following content:
+    """
+{
+  "@context": "http://www.w3.org/ns/csvw",
+  "url": "http://example.com/example1.csv",
+  "tableSchema": {
+    "columns": [
+            { "titles": "firstname", "required": true, "datatype": { "base": "string", "format": "^[A-Za-z0-9_]*$" } },
+            { "titles": "id", "required": true, "datatype": { "base": "string", "minLength": 1 } },
+            { "titles": "email", "required": true }
+      ]
+  }
+}
+    """
+    When I ask if there are warnings
+    Then there should be 0 warnings
+
+  Scenario: Schema with invalid regex
+    Given I have a CSV with the following content:
+    """
+  "firstname","id","email"
+  "Bob","1234","bob@example.org"
+  "Alice","5","alice@example.com"
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    And I have metadata with the following content:
+    """
+{
+  "@context": "http://www.w3.org/ns/csvw",
+  "url": "http://example.com/example1.csv",
+  "tableSchema": {
+    "columns": [
+            { "titles": "firstname", "required": true, "datatype": { "base": "string", "format": "((" } },
+            { "titles": "id", "required": true, "datatype": { "base": "string", "minLength": 1 } },
+            { "titles": "email", "required": true }
+      ]
+  }
+}
+    """
+    When I ask if there are warnings
+    Then there should be 1 warnings
+    And that warning should have the type "invalid_regex"