ascii-data-tools 0.9

data/features/ascii-data-cat.feature

@@ -0,0 +1,110 @@
+Feature: ascii-data-cat support
+  In order to understand the contents of ascii-encoded record streams quicker
+  As a tester
+  I want to decode and pretty print the streams
+  
+  Background:
+    Given the following configuration:
+      """
+      record_type("ABC") do
+        field "RECORD_TYPE",   :length => 3, :constrained_to => "ABC"
+        field "RECORD_SIZE",   :length => 5
+        field "END_OF_RECORD", :length => 1
+      end
+      """
+  
+  Scenario: two fixed-length records
+    When ascii-data-cat is invoked on a record stream containing
+      """
+      ABC12345
+      ABC67890
+      
+      """
+    Then the following is printed out:
+      """
+      Record 01 (ABC)
+      01 RECORD_TYPE   : [ABC]-------
+      02 RECORD_SIZE   : [12345]-----
+      03 END_OF_RECORD : [\n]--------
+      
+      Record 02 (ABC)
+      01 RECORD_TYPE   : [ABC]-------
+      02 RECORD_SIZE   : [67890]-----
+      03 END_OF_RECORD : [\n]--------
+      
+      
+      """
+
+  Scenario: an unknown record
+    When ascii-data-cat is invoked on a record stream containing
+      """
+      XYZ123456789
+
+      """    
+    Then the following is printed out:
+      """
+      Record 01 (unknown)
+      01 UNKNOWN : [XYZ123456789\n]-----
+
+
+      """
+      
+  Scenario: record types can be limited to apply only to records contained in particular filenames
+    # In this example, the record should not be recognised with type XYZ because the source filename does not match /records[.]XYZ[.]gz/
+    Given the following configuration:
+      """
+      record_type("XYZ", :applies_for_filenames_matching => /records[.]XYZ[.]gz/) do
+        field "RECORD_TYPE",   :length => 3
+        field "RECORD_SIZE",   :length => 6
+        field "END_OF_RECORD", :length => 1
+      end
+      """
+    When ascii-data-cat is invoked on a file "records.ABC.gz" containing
+      """
+      ABC123456
+
+      """
+    Then the following is printed out:
+      """
+      Record 01 (unknown)
+      01 UNKNOWN : [ABC123456\n]-----
+
+
+      """
+      
+  Scenario: out-of-the-box example record types defined and configured
+    When ascii-data-cat is invoked on a file "records.gz" containing
+      """
+      EXAMPLE01MO 4912345678      442012345678    0012
+      EXAMPLE02internet    2010010112000007220156
+      EXAMPLE01SMS4998765432      55555           0099
+
+      """
+    Then the following is printed out:
+      """
+      Record 01 (EXAMPLE01)
+      01 RECORD_TYPE      : [EXAMPLE01]------------
+      02 USAGE            : [MO ]------------------
+      03 A_NUMBER         : [4912345678      ]-----
+      04 B_NUMBER         : [442012345678    ]-----
+      05 CHARGEABLE_UNITS : [0012]-----------------
+      06 END_OF_RECORD    : [\n]-------------------
+
+      Record 02 (EXAMPLE02)
+      01 RECORD_TYPE      : [EXAMPLE02]----------
+      02 APN              : [internet    ]-------
+      03 TIMESTAMP        : [20100101120000]-----
+      04 SESSION_DURATION : [0722]---------------
+      05 CHARGEABLE_UNITS : [0156]---------------
+      06 END_OF_RECORD    : [\n]-----------------
+
+      Record 03 (EXAMPLE01)
+      01 RECORD_TYPE      : [EXAMPLE01]------------
+      02 USAGE            : [SMS]------------------
+      03 A_NUMBER         : [4998765432      ]-----
+      04 B_NUMBER         : [55555           ]-----
+      05 CHARGEABLE_UNITS : [0099]-----------------
+      06 END_OF_RECORD    : [\n]-------------------
+
+
+      """

data/features/ascii-data-edit.feature

@@ -0,0 +1,91 @@
+Feature: ascii-data-edit support
+  In order to create and edit test data efficiently
+  As a tester
+  I want a tool to edit the streams in a readable form
+  
+  Background:
+    Given the following configuration:
+      """
+      record_type("ABC") do
+        field "RECORD_TYPE",   :length => 3, :constrained_to => "ABC"
+        field "RECORD_SIZE",   :length => 5
+        field "END_OF_RECORD", :length => 1
+      end
+      """
+  
+  Scenario: two fixed-length records opened
+    When ascii-data-edit is invoked on a record stream containing
+      """
+      ABC12345
+      ABC67890
+
+      """
+    Then the editor shows:
+      """
+      Record 01 (ABC)
+      01 RECORD_TYPE   : [ABC]-------
+      02 RECORD_SIZE   : [12345]-----
+      03 END_OF_RECORD : [\n]--------
+
+      Record 02 (ABC)
+      01 RECORD_TYPE   : [ABC]-------
+      02 RECORD_SIZE   : [67890]-----
+      03 END_OF_RECORD : [\n]--------
+
+
+      """
+  
+  Scenario: two fixed-length records changed
+    Given a record stream containing
+      """
+      ABC12345
+      ABC67890
+      
+      """
+    When the output is successfully ascii-edited to the following:
+      """
+      Record 01 (ABC)
+      01 RECORD_TYPE   : [ABC]-------
+      02 RECORD_SIZE   : [45678]-----
+      03 END_OF_RECORD : [\n]--------
+      
+      Record 02 (ABC)
+      01 RECORD_TYPE   : [ABC]-------
+      02 RECORD_SIZE   : [XXXXX]-----
+      03 END_OF_RECORD : [\n]--------
+      
+      
+      """
+    Then the encoded record stream contains:
+      """
+      ABC45678
+      ABCXXXXX
+    
+      """
+      
+  Scenario: files not resaved unless they are modified during editing
+    Given a record stream containing
+      """
+      ABC12345
+      ABC67890
+
+      """
+    When the output is ascii-edited without alteration
+    Then the user receives the following feedback:
+      """
+      The file is unmodified.
+      
+      """
+  Scenario: editing an unknown record
+    When ascii-data-edit is invoked on a record stream containing
+      """
+      XYZ123456789
+  
+      """    
+    Then the editor shows:
+      """
+      Record 01 (unknown)
+      01 UNKNOWN : [XYZ123456789\n]-----
+  
+  
+      """

data/features/ascii-data-qdiff.feature

@@ -0,0 +1,54 @@
+Feature: ascii-data-qdiff support
+  In order to see the difference between the contents of ascii-encoded record streams
+  As a tester
+  I want a tool to decode, normalise, sort, pretty print the streams and show them in a diffing editor
+  
+  Scenario: comparing identical files
+    When ascii-data-qdiff is invoked on files containing:
+      """
+      EXAMPLE01MO 1112345678      442012345678    0012\n  ||  EXAMPLE01MO 1112345678      442012345678    0012\n
+      """
+    Then the user receives the following feedback:
+      """
+      The files are identical.
+      
+      """
+
+  Scenario: normal execution
+    When ascii-data-qdiff is invoked on files containing:
+      """
+      EXAMPLE01MO 1112345678      442012345678    0012\n  ||  EXAMPLE01MO 9954321098      442012345678    0012\n
+      EXAMPLE01MO 9992345678      442012345678    0012\n  ||  EXAMPLE01MO 9992345678      442012345678    0012\n
+      EXAMPLE02internet    2010010112000007220156\n       ||  EXAMPLE02internet    2010010112000007220156\n
+      EXAMPLE02internet    2010010113000001500000\n       ||  EXAMPLE02internet    2010010113000001500156\n
+      EXAMPLE01SMS4998765432      55555           0099\n  ||  --------------------------------------------------
+      """
+    Then the diffed result should be:
+      """
+      Record (EXAMPLE01)                             ||  Record (EXAMPLE01)
+      01 RECORD_TYPE      : [EXAMPLE01]------------  ||  01 RECORD_TYPE      : [EXAMPLE01]------------
+      02 USAGE            : [MO ]------------------  ||  02 USAGE            : [MO ]------------------
+      03 A_NUMBER         : [1112345678      ]-----  ||  03 A_NUMBER         : [9954321098      ]-----
+      04 B_NUMBER         : [442012345678    ]-----  ||  04 B_NUMBER         : [442012345678    ]-----
+      05 CHARGEABLE_UNITS : [0012]-----------------  ||  05 CHARGEABLE_UNITS : [0012]-----------------
+      06 END_OF_RECORD    : [\n]-------------------  ||  06 END_OF_RECORD    : [\n]-------------------
+                                                     ||  
+      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~||  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+                                                     ||  -----------------------------------------------
+      Record (EXAMPLE01)                             ||  -----------------------------------------------
+      01 RECORD_TYPE      : [EXAMPLE01]------------  ||  -----------------------------------------------
+      02 USAGE            : [SMS]------------------  ||  -----------------------------------------------
+      03 A_NUMBER         : [4998765432      ]-----  ||  -----------------------------------------------
+      04 B_NUMBER         : [55555           ]-----  ||  -----------------------------------------------
+      05 CHARGEABLE_UNITS : [0099]-----------------  ||  -----------------------------------------------
+      06 END_OF_RECORD    : [\n]-------------------  ||  -----------------------------------------------
+                                                     ||  
+      Record (EXAMPLE02)                             ||  Record (EXAMPLE02)
+      01 RECORD_TYPE      : [EXAMPLE02]----------    ||  01 RECORD_TYPE      : [EXAMPLE02]----------
+      02 APN              : [internet    ]-------    ||  02 APN              : [internet    ]-------
+      03 TIMESTAMP        : [XXXXXXXXXXXXXX]-----    ||  03 TIMESTAMP        : [XXXXXXXXXXXXXX]-----
+      04 SESSION_DURATION : [0150]---------------    ||  04 SESSION_DURATION : [0150]---------------
+      05 CHARGEABLE_UNITS : [0000]---------------    ||  05 CHARGEABLE_UNITS : [0156]---------------
+      06 END_OF_RECORD    : [\n]-----------------    ||  06 END_OF_RECORD    : [\n]-----------------
+                                                     ||  
+      """

data/features/encoding_decoding.feature

@@ -0,0 +1,68 @@
+Feature: encoding and decoding of records
+  In order to understand the contents of ascii-encoded record streams quicker
+  As a tester
+  I want a tool to be able to handle different types of records
+  
+  Scenario: decoding fixed-length records
+    Given the following configuration:
+      """
+      record_type("ABC") do
+        field "RECORD_TYPE",   :length => 3
+        field "RECORD_SIZE",   :length => 5
+        field "END_OF_RECORD", :length => 1
+      end
+      """
+    When I decode an encoded record "ABC12345\n" of type "ABC"
+    Then I should have a decoded record of type "ABC" and contents:
+      | field name    | field value |
+      | RECORD_TYPE   | ABC         |
+      | RECORD_SIZE   | 12345       |
+      | END_OF_RECORD | \n          |
+
+  Scenario: encoding fixed-length records
+    Given the following configuration:
+      """
+      record_type("ABC") do
+        field "RECORD_TYPE",   :length => 3
+        field "RECORD_SIZE",   :length => 5
+        field "END_OF_RECORD", :length => 1
+      end
+      """
+    When I encode a record of type "ABC" and contents:
+      | field name    | field value |
+      | RECORD_TYPE   | ABC         |
+      | RECORD_SIZE   | 12345       |
+      | END_OF_RECORD | \n          |
+    Then I should have a encoded record "ABC12345\n"
+      
+  Scenario: decoding csv records
+    Given the following configuration:
+      """
+      record_type("ABC", :family => "csv", :divider => ",") do
+        field "RECORD_TYPE"
+        field "RECORD_SIZE"
+        field "UNITS"
+      end
+      """
+    When I decode an encoded record "ABC,12345,123\n" of type "ABC"
+    Then I should have a decoded record of type "ABC" and contents:
+      | field name    | field value |
+      | RECORD_TYPE   | ABC         |
+      | RECORD_SIZE   | 12345       |
+      | UNITS         | 123         |
+
+  Scenario: encoding csv records
+    Given the following configuration:
+      """
+      record_type("ABC", :family => "csv", :divider => ",") do
+        field "RECORD_TYPE"
+        field "RECORD_SIZE"
+        field "UNITS"
+      end
+      """
+    When I encode a record of type "ABC" and contents:
+      | field name    | field value |
+      | RECORD_TYPE   | ABC         |
+      | RECORD_SIZE   | 12345       |
+      | UNITS         | XX          |
+    Then I should have a encoded record "ABC,12345,XX\n"

data/features/normaliser.feature

@@ -0,0 +1,27 @@
+Feature: normalisation
+  In order to see differences between two streams of record streams with variable parameters like timestamps
+  As a tester
+  I want a tool to normalise which outputs normalised raw records
+  
+  Scenario: two fixed-length records
+    Given the following configuration:
+      """
+      record_type("ABC") do
+        field "RECORD_TYPE",   :length => 3, :constrained_to => "ABC"
+        field "RECORD_SIZE",   :length => 5
+        field "TIMESTAMP",     :length => 14, :normalised => true
+        field "END_OF_RECORD", :length => 1
+      end
+      """
+    When ascii-data-norm is invoked on a record stream containing
+      """
+      ABC1234520100101120000
+      ABC6789020100415180005
+      
+      """
+    Then the following is printed out:
+      """
+      ABC12345XXXXXXXXXXXXXX
+      ABC67890XXXXXXXXXXXXXX
+    
+      """

data/features/plugins.feature

@@ -0,0 +1,73 @@
+Feature: tools for plugins
+  In order to create a plugin that adds custom functionality to the ascii tools  
+  As a plugin writer
+  I want tools which support readable, compact configurations and configuration debugging
+
+  Scenario: defining new record types and printing the record type summary
+    Given the following configuration:
+      """
+      record_type("ABC") do
+        field "RECORD_TYPE",   :length => 3
+        field "RECORD_SIZE",   :length => 5
+        field "END_OF_RECORD", :length => 1
+      end
+      
+      record_type("DEF") do
+        field "RECORD_TYPE",   :length => 3
+        field "RECORD_SIZE",   :length => 3
+        field "END_OF_RECORD", :length => 1
+      end
+      """
+    When the record type configuration is printed
+    Then it should look like this:
+      """
+      +-----------+--------------+-------------------------+-------------------+
+      | type name | total length | constraints             | normalised fields |
+      +-----------+--------------+-------------------------+-------------------+
+      | DEF       | 7            |                         |                   |
+      | ABC       | 9            |                         |                   |
+      | EXAMPLE02 | 44           | RECORD_TYPE = EXAMPLE02 | TIMESTAMP         |
+      | EXAMPLE01 | 49           | RECORD_TYPE = EXAMPLE01 |                   |
+      +-----------+--------------+-------------------------+-------------------+
+
+      """
+      
+  Scenario: defining constraints on record types
+    Given the following configuration:
+      """
+      record_type("XYZ") do
+        field "RECORD_TYPE",      :length => 5,  :constrained_to => "REC01"
+        field "A_NUMBER",         :length => 16, :constrained_to => /44123/
+        field "END_OF_RECORD",    :length => 1
+      end
+      """
+    When the record type configuration is printed
+    Then it should look like this:
+      """
+      +-----------+--------------+------------------------------------------+-------------------+
+      | type name | total length | constraints                              | normalised fields |
+      +-----------+--------------+------------------------------------------+-------------------+
+      | XYZ       | 22           | RECORD_TYPE = REC01, A_NUMBER =~ /44123/ |                   |
+      | EXAMPLE02 | 44           | RECORD_TYPE = EXAMPLE02                  | TIMESTAMP         |
+      | EXAMPLE01 | 49           | RECORD_TYPE = EXAMPLE01                  |                   |
+      +-----------+--------------+------------------------------------------+-------------------+
+
+      """
+
+  Scenario: normalising and grepping record types
+    Given the following configuration:
+      """
+      for_names_matching(/EXAMPLE\d/) {|type| type.fields_with {|field| field.name =~ /RECORD_/}.should_be_normalised }
+      type("EXAMPLE01").field_with_index(2).should_be_normalised
+      """
+    When the record type configuration is printed
+    Then it should look like this:
+      """
+      +-----------+--------------+-------------------------+------------------------+
+      | type name | total length | constraints             | normalised fields      |
+      +-----------+--------------+-------------------------+------------------------+
+      | EXAMPLE02 | 44           | RECORD_TYPE = EXAMPLE02 | RECORD_TYPE, TIMESTAMP |
+      | EXAMPLE01 | 49           | RECORD_TYPE = EXAMPLE01 | RECORD_TYPE, USAGE     |
+      +-----------+--------------+-------------------------+------------------------+
+
+      """

data/features/record_recognition.feature

@@ -0,0 +1,61 @@
+Feature: intelligent record recognition
+  In order to understand the contents of ascii-encoded record streams quicker
+  As a tester
+  I want a tool to correctly recognise the record type without user intervention
+  
+  Background:
+    Given the following configuration:
+    """
+    record_type("ABC") do
+      field "RECORD_TYPE",   :length => 3, :constrained_to => "ABC"
+      field "RECORD_SIZE",   :length => 5
+      field "END_OF_RECORD", :length => 1
+    end
+
+    record_type("DEF") do
+      field "RECORD_TYPE",   :length => 3, :constrained_to => "DEF"
+      field "RECORD_SIZE",   :length => 5
+      field "END_OF_RECORD", :length => 1
+    end
+    
+    record_type("GXX") do
+      field "RECORD_TYPE",   :length => 3, :constrained_to => ["G01", "G02"]
+      field "RECORD_SIZE",   :length => 5
+      field "END_OF_RECORD", :length => 1
+    end
+
+    record_type("XYZ") do
+      field "RECORD_TYPE",   :length => 3
+      field "RECORD_SIZE",   :length => 3
+      field "END_OF_RECORD", :length => 1
+    end
+    
+    record_type("TXX_A", :applies_for_filenames_matching => /TXX_A/) do
+      field "RECORD_TYPE",   :length => 3, :constrained_to => "TXX"
+      field "RECORD_SIZE",   :length => 4
+      field "END_OF_RECORD", :length => 1
+    end
+
+    record_type("TXX_B", :applies_for_filenames_matching => /TXX_B/) do
+      field "RECORD_TYPE",   :length => 3, :constrained_to => "TXX"
+      field "RECORD_SIZE",   :length => 4
+      field "END_OF_RECORD", :length => 1
+    end
+    """
+  
+  Scenario Outline: fixed length record recognition
+    When record "<record>" coming from <filename> is analysed
+    Then its type should be recognised as "<expected type>"
+    
+    Examples:
+      | record     | filename    | expected type |
+      | ABC        | unspecified | unknown       |
+      | XYZ123\n   | unspecified | XYZ           |
+      | ABC12345\n | unspecified | ABC           |
+      | DEF12345\n | unspecified | DEF           |
+      | G0112345\n | unspecified | GXX           |
+      | G0212345\n | unspecified | GXX           |
+      | G0312345\n | unspecified | unknown       |
+      | TXX1234\n  | TXX_A.gz    | TXX_A         |
+      | TXX1234\n  | TXX_B.gz    | TXX_B         |
+      | TXX1234\n  | TXX_Z.gz    | unknown       |