wjordan213-csvlint 0.2.8

data/features/cli.feature

@@ -0,0 +1,210 @@
+Feature: CSVlint CLI
+
+  Scenario: Valid CSV from url
+    Given I have a CSV with the following content:
+    """
+"Foo","Bar","Baz"
+"1","2","3"
+"3","2","1"
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    When I run `csvlint http://example.com/example1.csv`
+    Then the output should contain "http://example.com/example1.csv is VALID"
+
+  Scenario: Valid CSV from file
+    When I run `csvlint ../../features/fixtures/valid.csv`
+    Then the output should contain "valid.csv is VALID"
+
+  # This is a hacky way of saying to run `cat features/fixtures/valid.csv | csvlint`
+  Scenario: Valid CSV from pipe
+    Given I have stubbed stdin to contain "features/fixtures/valid.csv"
+    When I run `csvlint`
+    Then the output should contain "CSV is VALID"
+
+  Scenario: URL that 404s
+    Given there is no file at the url "http://example.com/example1.csv"
+    And there is no file at the url "http://example.com/.well-known/csvm"
+    And there is no file at the url "http://example.com/example1.csv-metadata.json"
+    And there is no file at the url "http://example.com/csv-metadata.json"
+    When I run `csvlint http://example.com/example1.csv`
+    Then the output should contain "http://example.com/example1.csv is INVALID"
+    And the output should contain "not_found"
+
+  Scenario: File doesn't exist
+    When I run `csvlint ../../features/fixtures/non-existent-file.csv`
+    Then the output should contain "non-existent-file.csv not found"
+
+  Scenario: No file or URL specified
+    Given I have stubbed stdin to contain nothing
+    When I run `csvlint`
+    Then the output should contain "No CSV data to validate"
+
+  Scenario: No file or URL specified, but schema specified
+    Given I have stubbed stdin to contain nothing
+    And I have a schema with the following content:
+    """
+{
+  "fields": [
+          { "name": "Name", "constraints": { "required": true } },
+          { "name": "Id", "constraints": { "required": true, "minLength": 1 } },
+          { "name": "Email", "constraints": { "required": true } }
+    ]
+}
+    """
+    And the schema is stored at the url "http://example.com/schema.json"
+    When I run `csvlint --schema http://example.com/schema.json`
+    Then the output should contain "No CSV data to validate"
+
+  Scenario: Invalid CSV from url
+    Given I have a CSV with the following content:
+    """
+    "Foo",	"Bar"	,	"Baz"
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    When I run `csvlint http://example.com/example1.csv`
+    Then the output should contain "http://example.com/example1.csv is INVALID"
+    And the output should contain "whitespace"
+
+  Scenario: Specify schema
+    Given I have a CSV with the following content:
+    """
+"Bob","1234","bob@example.org"
+"Alice","5","alice@example.com"
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    And I have a schema with the following content:
+    """
+{
+	"fields": [
+          { "name": "Name", "constraints": { "required": true } },
+          { "name": "Id", "constraints": { "required": true, "minLength": 1 } },
+          { "name": "Email", "constraints": { "required": true } }
+    ]
+}
+    """
+    And the schema is stored at the url "http://example.com/schema.json"
+    When I run `csvlint http://example.com/example1.csv --schema http://example.com/schema.json`
+    Then the output should contain "http://example.com/example1.csv is VALID"
+
+  Scenario: Schema errors
+    Given I have a CSV with the following content:
+    """
+"Bob","1234","bob@example.org"
+"Alice","5","alice@example.com"
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    And I have a schema with the following content:
+    """
+{
+  "fields": [
+          { "name": "Name", "constraints": { "required": true } },
+          { "name": "Id", "constraints": { "required": true, "minLength": 3 } },
+          { "name": "Email", "constraints": { "required": true } }
+    ]
+}
+    """
+    And the schema is stored at the url "http://example.com/schema.json"
+    When I run `csvlint http://example.com/example1.csv --schema http://example.com/schema.json`
+    Then the output should contain "http://example.com/example1.csv is INVALID"
+    And the output should contain "1. Id: min_length. Row: 2,2. 5"
+    And the output should contain "1. malformed_header. Row: 1. Bob,1234,bob@example.org"
+
+  Scenario: Invalid schema
+    Given I have a CSV with the following content:
+    """
+"Bob","1234","bob@example.org"
+"Alice","5","alice@example.com"
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    And I have a schema with the following content:
+    """
+NO JSON HERE SON
+    """
+    And the schema is stored at the url "http://example.com/schema.json"
+    Then nothing should be outputted to STDERR
+    When I run `csvlint http://example.com/example1.csv --schema http://example.com/schema.json`
+    And the output should contain "invalid metadata: malformed JSON"
+
+  Scenario: Schema that 404s
+    Given I have a CSV with the following content:
+    """
+"Bob","1234","bob@example.org"
+"Alice","5","alice@example.com"
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    And there is no file at the url "http://example.com/schema404.json"
+    When I run `csvlint http://example.com/example1.csv --schema http://example.com/schema404.json`
+    Then the output should contain "http://example.com/schema404.json not found"
+
+  Scenario: Schema that doesn't exist
+    Given I have a CSV with the following content:
+    """
+"Bob","1234","bob@example.org"
+"Alice","5","alice@example.com"
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    When I run `csvlint http://example.com/example1.csv --schema /fake/file/path.json`
+    Then the output should contain "/fake/file/path.json not found"
+
+  Scenario: Valid CSVw schema
+    Given I have a CSV with the following content:
+    """
+"Bob","1234","bob@example.org"
+"Alice","5","alice@example.com"
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    And I have metadata with the following content:
+    """
+{
+  "@context": "http://www.w3.org/ns/csvw",
+  "url": "http://example.com/example1.csv",
+  "dialect": { "header": false },
+  "tableSchema": {
+    "columns": [
+            { "name": "Name", "required": true },
+            { "name": "Id", "required": true, "datatype": { "base": "string", "minLength": 1 } },
+            { "name": "Email", "required": true }
+      ]
+  }
+}
+    """
+    And the schema is stored at the url "http://example.com/schema.json"
+    When I run `csvlint http://example.com/example1.csv --schema http://example.com/schema.json`
+    Then the output should contain "http://example.com/example1.csv is VALID"
+
+  Scenario: CSVw schema with invalid CSV
+    Given I have a CSV with the following content:
+    """
+"Bob","1234","bob@example.org"
+"Alice","5","alice@example.com"
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    And I have metadata with the following content:
+    """
+{
+  "@context": "http://www.w3.org/ns/csvw",
+  "url": "http://example.com/example1.csv",
+  "dialect": { "header": false },
+  "tableSchema": {
+    "columns": [
+            { "name": "Name", "required": true },
+            { "name": "Id", "required": true, "datatype": { "base": "string", "minLength": 3 } },
+            { "name": "Email", "required": true }
+      ]
+  }
+}
+    """
+    And the schema is stored at the url "http://example.com/schema.json"
+    When I run `csvlint http://example.com/example1.csv --schema http://example.com/schema.json`
+    Then the output should contain "http://example.com/example1.csv is INVALID"
+    And the output should contain "1. min_length. Row: 2,2. 5"
+
+  Scenario: CSVw table Schema
+    Given I have stubbed stdin to contain nothing
+    And I have a metadata file called "csvw/countries.json"
+    And the metadata is stored at the url "http://w3c.github.io/csvw/tests/countries.json"
+    And I have a file called "csvw/countries.csv" at the url "http://w3c.github.io/csvw/tests/countries.csv"
+    And I have a file called "csvw/country_slice.csv" at the url "http://w3c.github.io/csvw/tests/country_slice.csv"
+    When I run `csvlint --schema http://w3c.github.io/csvw/tests/countries.json`
+    Then the output should contain "http://w3c.github.io/csvw/tests/countries.csv is VALID"
+    And the output should contain "http://w3c.github.io/csvw/tests/country_slice.csv is VALID"

data/features/csv_options.feature

@@ -0,0 +1,35 @@
+Feature: CSV options
+
+  Scenario: Sucessfully parse a valid CSV
+    Given I have a CSV with the following content:
+    """
+'Foo';'Bar';'Baz'
+'1';'2';'3'
+'3';'2';'1'
+    """
+    And I set the delimiter to ";"
+    And I set quotechar to "'" 
+    And it is stored at the url "http://example.com/example1.csv"
+    When I ask if the CSV is valid
+    Then I should get the value of true
+
+  Scenario: Warn if options seem to return invalid data
+    Given I have a CSV with the following content:
+    """
+'Foo';'Bar';'Baz'
+'1';'2';'3'
+'3';'2';'1'
+    """
+    And I set the delimiter to ","
+    And I set quotechar to """ 
+    And it is stored at the url "http://example.com/example1.csv"
+    And I ask if there are warnings
+    Then there should be 1 warnings
+    And that warning should have the type "check_options"
+
+  Scenario: Use esoteric line endings
+    Given I have a CSV file called "windows-line-endings.csv"
+    And it is stored at the url "http://example.com/example1.csv"
+    When I ask if the CSV is valid
+    Then I should get the value of true
+    

data/features/csvupload.feature

@@ -0,0 +1,145 @@
+Feature: Collect all the tests that should trigger dialect check related errors
+
+  Scenario: Title rows, I wish to trigger a :title_row type message
+    Given I have a CSV file called "title-row.csv"
+    And it is stored at the url "http://example.com/example1.csv"
+    And I ask if there are warnings
+    Then there should be 1 warnings
+    And that warning should have the type "title_row"
+
+#    :nonrfc_line_breaks
+
+  Scenario: LF line endings in file give an info message of type :nonrfc_line_breaks
+    Given I have a CSV file called "lf-line-endings.csv"
+    And it is stored at the url "http://example.com/example1.csv"
+    And I set header to "true"
+    And I ask if there are info messages
+    Then there should be 1 info message
+    And one of the messages should have the type "nonrfc_line_breaks"
+
+  Scenario: CRLF line endings in file produces no info messages of type :nonrfc_line_breaks
+    Given I have a CSV file called "crlf-line-endings.csv"
+    And it is stored at the url "http://example.com/example1.csv"
+    And I set header to "true"
+    And I ask if there are info messages
+    Then there should be 0 info messages
+
+#  :line_breaks
+
+  Scenario: Incorrect line endings specified in settings
+    Given I have a CSV file called "lf-line-endings.csv"
+    And I set the line endings to carriage return
+    And it is stored at the url "http://example.com/example1.csv"
+    And I ask if there are errors
+    Then there should be 1 error
+    And that error should have the type "line_breaks"
+
+  Scenario: inconsistent line endings in file cause an error
+    Given I have a CSV file called "inconsistent-line-endings.csv"
+    And it is stored at the url "http://example.com/example1.csv"
+    And I ask if there are errors
+    Then there should be 1 error
+    And that error should have the type "line_breaks"
+
+
+  Scenario: inconsistent line endings with unquoted fields in file cause an error
+    Given I have a CSV file called "inconsistent-line-endings-unquoted.csv"
+    And it is stored at the url "http://example.com/example1.csv"
+    And I ask if there are errors
+    Then there should be 1 error
+    And that error should have the type "line_breaks"
+
+#:unclosed_quote
+
+  Scenario: CSV with incorrect quoting
+    Given I have a CSV with the following content:
+    """
+"col1","col2","col3"
+"Foo","Bar","Baz
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    When I ask if there are errors
+    Then there should be 1 error
+    And that error should have the type "unclosed_quote"
+    And that error should have the row "2"
+    And that error should have the content ""Foo","Bar","Baz"
+
+#  :invalid_encoding
+
+  Scenario: Report invalid Encoding
+    Given I have a CSV file called "invalid-byte-sequence.csv"
+    And I set an encoding header of "UTF-8"
+    And it is stored at the url "http://example.com/example1.csv"
+    When I ask if there are errors
+    Then there should be 1 error
+    And that error should have the type "invalid_encoding"
+
+  Scenario: Report invalid file
+#should this throw an excel error?
+    Given I have a CSV file called "spreadsheet.xls"
+    And it is stored at the url "http://example.com/example1.csv"
+    When I ask if there are errors
+    Then there should be 1 error
+    And that error should have the type "invalid_encoding"
+
+#  :blank_rows
+
+  Scenario: Successfully report a CSV with blank rows
+    Given I have a CSV with the following content:
+    """
+"col1","col2","col3"
+"Foo","Bar","Baz"
+"","",
+"Baz","Bar","Foo"
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    When I ask if there are errors
+    Then there should be 1 error
+    And that error should have the type "blank_rows"
+    And that error should have the row "3"
+    And that error should have the content ""","","
+
+  Scenario: Successfully report a CSV with multiple trailing empty rows
+    Given I have a CSV with the following content:
+    """
+"col1","col2","col3"
+"Foo","Bar","Baz"
+"Foo","Bar","Baz"
+
+
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    When I ask if there are errors
+    Then there should be 1 error
+    And that error should have the type "blank_rows"
+    And that error should have the row "4"
+
+  Scenario: Successfully report a CSV with an empty row
+    Given I have a CSV with the following content:
+    """
+"col1","col2","col3"
+"Foo","Bar","Baz"
+
+"Foo","Bar","Baz"
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    When I ask if there are errors
+    Then there should be 1 error
+    And that error should have the type "blank_rows"
+    And that error should have the row "3"
+
+#:check_options
+
+  Scenario: Warn if options seem to return invalid data
+    Given I have a CSV with the following content:
+    """
+'Foo';'Bar';'Baz'
+'1';'2';'3'
+'3';'2';'1'
+    """
+    And I set the delimiter to ","
+    And I set quotechar to """
+    And it is stored at the url "http://example.com/example1.csv"
+    And I ask if there are warnings
+    Then there should be 1 warnings
+    And that warning should have the type "check_options"

data/features/csvw_schema_validation.feature

@@ -0,0 +1,127 @@
+Feature: CSVW Schema Validation
+
+  Scenario: Valid CSV
+    Given I have a CSV with the following content:
+    """
+"Bob","1234","bob@example.org"
+"Alice","5","alice@example.com"
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    And I have metadata with the following content:
+    """
+{
+  "@context": "http://www.w3.org/ns/csvw",
+  "url": "http://example.com/example1.csv",
+  "dialect": { "header": false },
+  "tableSchema": {
+    "columns": [
+            { "name": "Name", "required": true },
+            { "name": "Id", "required": true, "datatype": { "base": "string", "minLength": 1 } },
+            { "name": "Email", "required": true }
+      ]
+  }
+}
+    """
+    When I ask if there are errors
+    Then there should be 0 error
+
+  Scenario: Schema invalid CSV
+    Given I have a CSV with the following content:
+    """
+"Bob","1234","bob@example.org"
+"Alice","5","alice@example.com"
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    And I have metadata with the following content:
+    """
+{
+  "@context": "http://www.w3.org/ns/csvw",
+  "url": "http://example.com/example1.csv",
+  "dialect": { "header": false },
+  "tableSchema": {
+    "columns": [
+            { "name": "Name", "required": true },
+            { "name": "Id", "required": true, "datatype": { "base": "string", "minLength": 3 } },
+            { "name": "Email", "required": true }
+      ]
+  }
+}
+    """
+    When I ask if there are errors
+    Then there should be 1 error
+
+  Scenario: CSV with incorrect header
+    Given I have a CSV with the following content:
+    """
+"name","id","contact"
+"Bob","1234","bob@example.org"
+"Alice","5","alice@example.com"
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    And I have metadata with the following content:
+    """
+{
+  "@context": "http://www.w3.org/ns/csvw",
+  "url": "http://example.com/example1.csv",
+  "tableSchema": {
+    "columns": [
+            { "titles": "name", "required": true },
+            { "titles": "id", "required": true, "datatype": { "base": "string", "minLength": 1 } },
+            { "titles": "email", "required": true }
+      ]
+  }
+}
+    """
+    When I ask if there are errors
+    Then there should be 1 error
+
+  Scenario: Schema with valid regex
+    Given I have a CSV with the following content:
+    """
+  "firstname","id","email"
+  "Bob","1234","bob@example.org"
+  "Alice","5","alice@example.com"
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    And I have metadata with the following content:
+    """
+{
+  "@context": "http://www.w3.org/ns/csvw",
+  "url": "http://example.com/example1.csv",
+  "tableSchema": {
+    "columns": [
+            { "titles": "firstname", "required": true, "datatype": { "base": "string", "format": "^[A-Za-z0-9_]*$" } },
+            { "titles": "id", "required": true, "datatype": { "base": "string", "minLength": 1 } },
+            { "titles": "email", "required": true }
+      ]
+  }
+}
+    """
+    When I ask if there are warnings
+    Then there should be 0 warnings
+
+  Scenario: Schema with invalid regex
+    Given I have a CSV with the following content:
+    """
+  "firstname","id","email"
+  "Bob","1234","bob@example.org"
+  "Alice","5","alice@example.com"
+    """
+    And it is stored at the url "http://example.com/example1.csv"
+    And I have metadata with the following content:
+    """
+{
+  "@context": "http://www.w3.org/ns/csvw",
+  "url": "http://example.com/example1.csv",
+  "tableSchema": {
+    "columns": [
+            { "titles": "firstname", "required": true, "datatype": { "base": "string", "format": "((" } },
+            { "titles": "id", "required": true, "datatype": { "base": "string", "minLength": 1 } },
+            { "titles": "email", "required": true }
+      ]
+  }
+}
+    """
+    When I ask if there are warnings
+    Then there should be 1 warnings
+    And that warning should have the type "invalid_regex"