cure 0.1.1 → 0.4.0

data/docs/builder/black_white_list.md

@@ -0,0 +1,83 @@
+[... go back to build contents](main.md)
+
+## Black/White List 
+
+### What is it?
+
+These builders operate at a sheet level, bulk changing the columns provided.
+- Blacklist will bulk remove any columns **in** the list provided.
+- Whitelist will bulk remove any columns **not in** the list provided. 
+
+### Why would you need it?
+
+If you have a large spreadsheet that you want to control the presence or removal of multiple columns,
+the quickest way to do so is via this option.
+
+### Full Configuration
+
+```ruby
+candidate(named_range: "mysheet") do
+  blacklist options: { columns: %w[col_a col_b] }
+  whitelist options: { columns: %w[col_c col_d] }
+end
+```
+
+- `named_range`: specifies the named range holding the column, if no named range has been set you can leave it blank.
+  - `options`:
+    - `columns`: mandatory, will perform filtering on these options. 
+
+### Example
+
+## Blacklist
+
+```ruby
+candidate(named_range: "mysheet") do
+  blacklist options: { columns: %w[col_a col_b] }
+end
+```
+
+Original input: 
+```
++-------+-------+-------+
+| col_a | col_b | col_c |
++-------+-------+-------+
+| a     | b     | c     |
++-------+-------+-------+
+```
+
+changes to: 
+
+```
++-------+
+| col_c |
++-------+
+| c     |
++-------+
+```
+
+## Whitelist
+
+```ruby
+candidate(named_range: "mysheet") do
+  whitelist options: { columns: %w[col_a col_b] }
+end
+```
+
+Original input:
+```
++-------+-------+-------+
+| col_a | col_b | col_c |
++-------+-------+-------+
+| a     | b     | c     |
++-------+-------+-------+
+```
+
+changes to:
+
+```
++-------+-------+
+| col_a | col_b |
++-------+-------+
+| a     | b     |
++-------+-------+
+```

data/docs/builder/copy.md

@@ -0,0 +1,48 @@
+[... go back to build contents](main.md)
+
+## Copy 
+
+### What is it?
+
+Copy builder will copy an entire column from the spreadsheet.
+
+### Why would you need it?
+
+Copy a column from the spreadsheet, useful if you want to transform or manipulate the column in the transform stage.
+
+### Full Configuration
+
+```ruby
+build do
+  candidate(column: "col_b", named_range: "_default") { copy options: { to_column: "col_b_copy" } }
+end
+```
+- `column`: represents the column name, mandatory.
+- `named_range`: specifies the named range holding the column, if no named range has been set you can leave it blank.
+  - `options`:
+    - `to_column`: column will be renamed to this value if set, otherwise will default to <column>_copy.
+    
+### Example
+
+```ruby
+build do
+  candidate(column: "col_a") { copy options: { to_column: "col_a_copy" } }
+end
+```
+
+Original input:
+```
++-------+
+| col_a |
++-------+
+| a     |
++-------+
+```
+changes to:
+```
++-------+------------+
+| col_a | col_a_copy |
++-------+------------+
+| a     | a          |
++-------+------------+
+```

data/docs/builder/explode.md

@@ -0,0 +1,70 @@
+[... go back to build contents](main.md)
+
+## Explode
+
+### What is it?
+
+Explode takes a JSON string and will break it out into columns intelligently.
+
+### Why would you need it?
+
+If you have a JSON column and you wish to treat the values as an individual column per key.  This is popular in technical
+
+### Full Configuration
+
+```yaml
+build:
+  candidates:
+    - column: "column_name"
+      named_range: "default"
+      action:
+        type: "explode"
+        options:
+          filter:
+            type: "whitelist|blacklist"
+            values:
+              - "example"
+```
+- 
+- `column`: represents the column name, mandatory.
+- `named_range`: specifies the named range holding the column, if no named range has been set you can leave it blank.
+- `action`: represents the action that will be taken on the data
+  - `type`: specifies the type of action, in this instance is explode
+  - `options`:
+    - `filter`: filters out the candidate columns, you can either use whitelist or blacklist.
+    - `values`: contains the names of the columns that will be filtered with.
+
+
+### Example
+
+```yaml
+build:
+  candidates:
+    - column: tags
+      action:
+        type: "explode"
+```
+
+Original input: 
+```
++---------------------------------+
+| tags                            |
++---------------------------------+
+| {"type":"string","name":"abcd"} |
++---------------------------------+
+| {"tier":"high"}                 | 
++---------------------------------+
+```
+
+changes to: 
+
+```
++----------------------------------+--------+-------+------+
+| tags                             | type   | name  | tier |
++----------------------------------+--------+-------+------+
+| {"type":"string","name":"abcde"} | string | abcde |      |
+| {"tier":"string"}                |        |       | high |
++----------------------------------+--------+-------+------+
+```
+
+**Note:** if you want to remove the original column (tags) you can do so with the [remove](remove.md) option.

data/docs/builder/main.md

@@ -0,0 +1,43 @@
+Source > Extract > Validate > **Build** > Query > Transform > Export
+
+Build
+=======
+
+### About
+
+The build step immediately follows the **extract** step, and operates at a column level on the spreadsheet. It provides
+an interface to manipulation that you may wish to occur across all columns.  Individual build steps are called
+candidates, and *multiple steps can be performed on a single column* if desired.  
+
+---
+
+**When you should use this**: You have a spreadsheet that requires changes to the column structure of the data. This 
+may be as trivial as adding or removing a column, or *exploding* a JSON object (Key => Val) into individual columns.
+
+See below an example configuration block:
+
+```ruby
+build do
+  # White/Blacklist - do not need to provide column into candidate
+  candidate do
+    blacklist options: { columns: %w[col_a col_b] }
+    whitelist options: { columns: %w[col_c col_d] }
+  end
+
+  # Simple addition of new column
+  candidate(column: "full_name") { add options: { default_value: "ABC" } }
+  # Simple renaming of existing column
+  candidate(column: "Tags") { rename options: { new_name: "test" } }
+end
+```
+
+- `column`: represents the column name, mandatory.
+- `named_range`: specifies the named range holding the column, if no named range has been set you can leave it blank.
+
+### Components
+
+There are four different types of operations you can perform in this step; 
+- [add](add.md)
+- [remove](remove.md)
+- [copy](copy.md)
+- [black_white_list](black_white_list.md)

data/docs/builder/remove.md

@@ -0,0 +1,46 @@
+[... go back to build contents](main.md)
+
+## Remove 
+
+### What is it?
+
+Remove builder will remove an entire column from the spreadsheet.
+
+### Why would you need it?
+
+Removes a column from the spreadsheet, useful if you want to remove entire columns from the output.
+
+### Full Configuration
+
+```ruby
+build do
+  candidate(column: "remove_this", named_range: "mysheet") { remove }
+end
+```
+- `column`: represents the column name, mandatory.
+- `named_range`: specifies the named range holding the column, if no named range has been set you can leave it blank.
+
+### Example
+
+```ruby
+build do
+  candidate(column: "col_b") { remove }
+end
+```
+
+Original input:
+```
++-------+-------+
+| col_a | col_b |
++-------+-------+
+| a     | b     |
++-------+-------+
+```
+changes to: 
+```
++-------+
+| col_a |
++-------+
+| a     |
++-------+
+```

data/docs/examples/examples.md

@@ -0,0 +1,164 @@
+## Examples
+
+Below are some examples of Cure being used to transform odd CSV formats, or unusual tasks.
+
+### Multi row grouping
+
+In the example below we want to:
+1. Group the rows on identifier
+2. Change gender to single letter, 
+3. Create a full name column that joins first_name and last_name, and capitalizes them.
+
+| id | identifier | first_name | last_name | age | gender |
+|----|------------|------------|-----------|-----|--------|
+| 1  | 1          | joe        | smith     | 20  |        |
+| 2  | 1          |            |           |     | male   |
+| 3  | 2          | lean       | davis     | 32  |        |
+| 4  | 2          |            |           |     | female |
+
+to
+
+| id | identifier | first_name | last_name | age | gender | full_name  |
+|----|------------|------------|-----------|-----|--------|------------|
+| 1  | 1          | joe        | smith     | 20  | M      | Joe Smith  |
+| 3  | 2          | lean       | davis     | 32  | F      | Lean Davis |
+
+using
+
+```ruby
+build do
+  candidate column: "full_name" do
+    add options: { default_value: "" }
+  end
+end
+
+transform do
+  from query: <<-SQL
+      SELECT 
+        id as id, 
+        identifier as identifier, 
+        group_concat(first_name, '') as first_name, 
+        group_concat(last_name, '') as last_name, 
+        group_concat(gender, '') as gender, 
+        group_concat(age, '') as age, 
+        full_name FROM _default 
+      GROUP BY identifier
+  SQL
+  
+  candidate column: "gender" do
+    with_translation { replace("full").with("case",
+      statement: {
+        switch: [
+          {
+            case: "male",
+            return_value: "M"
+          },{
+            case: "female",
+            return_value: "F"
+          }
+        ],
+        else: [
+          return_value: "<unknown gender>"
+        ]
+      })
+    }
+  end
+
+  candidate column: "full_name" do
+    with_translation { replace("full").with("erb", 
+      template: "<%= first_name.capitalize %> <%= last_name.capitalize %>")
+    }
+  end
+end
+
+export do
+  terminal title: "Exported", limit_rows: 5
+end
+
+```
+
+### AWS Cost and Usage Report Anonymization
+
+Below a small subset of the Cost and Usage Report provided by Amazon that hold information that we want to transform.
+
+Some thoughts;
+- the **identity/LineItemId** column has seemingly random characters that may be the same (see row ids 9 and 10)
+- **lineItem/ResourceId** has records that hold account numbers, we want to ensure that they are the same as **bill/PayerAccountId**
+and **lineItem/UsageAccountId** for consistent data.
+
+| id | identity/LineItemId                                  | bill/PayerAccountId | lineItem/UsageAccountId | lineItem/ProductCode | lineItem/ResourceId                          |
+|----|------------------------------------------------------|---------------------|-------------------------|----------------------|----------------------------------------------|
+| 1  | mggj00y7rig8p3xjma6rpzkvtrn98q4a0ortz9ddgquu0pv3xshq | 9876543210          | 9876543210              | AmazonS3             | cloudtrail-9876543210                        |
+| 2  | t8ubihdw6ad39awf1748v98yim4uh6wyjzr59bziwwcfnyu4rxhf | 9876543210          | 9876543210              | AmazonS3             | cloudtrail-9876543210                        |
+| 3  | 8c8u2fcetmrz3f0x52coe4wgjv77ffxx2ivgitg1a1nacpo8menv | 9876543210          | 9876543210              | AmazonCloudFront     | arn:aws:cloudfront::9876543210:Overhold      |
+| 4  | 9jqoasom8qnxma5rjqhawkncrhev0ocsp4ax5pngrp8l1yno03v3 | 9876543210          | 9876543210              | AmazonS3             | aws-cloudtrail-logs-9876543210               |
+| 5  | 35znibzyuoisze9x45377jqkbd7o677w4mhgl8hyte8born5h1h3 | 9876543210          | 9876543210              | AmazonCloudFront     | arn:aws:cloudfront::9876543210:Overhold      |
+| 6  | tb8qzhsrqu0z613jervo541l7p95b5pq2k80m7hcsnqjjjs6jnlx | 9876543210          | 9876543210              | awskms               | arn:aws:kms:ap-southeast-2:9876543210:Zoolab |
+| 7  | c0k9bpm5y5m1aoebsrlc2ozdgqoqjkyjy7z0hx7kv4y93gx8ioji | 9876543210          | 9876543210              | AWSLambda            | arn:aws:lambda:Trippledex                    |
+| 8  | ju8pmo0qqn5c2tapej4toy3c95w08ym6uar9hllyf3r0oj1hoiya | 9876543210          | 9876543210              | AmazonEC2            | vol-3ef2aece632                              |
+| 9  | f5kta3av4k5k2fve6l8g370bj41leqzkazsad28hjnu2xngn8f86 | 9876543210          | 9876543210              | AmazonS3             | cloudtrail-9876543210                        |
+| 10 | f5kta3av4k5k2fve6l8g370bj41leqzkazsad28hjnu2xngn8f86 | 9876543210          | 9876543210              | AmazonS3             | cloudtrail-9876543210                        |
+
+##### Configuration
+```ruby
+transform do
+  # Operate on the "identity/LineItemId" column
+  candidate column: "identity/LineItemId" do
+    # Replace the full record with a random character string of 52 length, only consisting of 
+    # lowercase and number values.
+    with_translation { replace("full").with("character", length: 52, types: %w[lowercase number]) }
+  end
+
+  candidate column: "bill/PayerAccountId" do
+    # Replace the full record with a placeholder named :account_number (See at bottom of file for placeholders)
+    with_translation { replace("full").with("placeholder", name: :account_number) }
+  end
+
+  candidate column: "lineItem/UsageAccountId" do
+    with_translation { replace("full").with("number", length: 10) }
+  end
+
+  candidate column: "lineItem/ResourceId" do
+    # If there is a match (i-[my-group]), replace just the match group with a hex string of 10 length
+    with_translation { replace("regex", regex_cg: "^i-(.*)").with("hex", length: 10) }
+    # If there is a match (vol-[my-group]), replace just the match group with a hex string of 10 length
+    with_translation { replace("regex", regex_cg: "^vol-(.*)").with("hex", length: 10) }
+    # If the string contains a token :, replace the 4th element with the account_number placeholder.
+    with_translation { replace("split", token: ":", index: 4).with("placeholder", name: :account_number) }
+    # If the string contains a token -, replace the last element with the account_number placeholder.
+    with_translation { replace("split", token: "-", index: -1).with("placeholder", name: :account_number) }
+    # If the string contains a token :, replace the last element with the a Faker value Faker::App.name.
+    with_translation { replace("split", token: ":", index: -1).with("faker", module: "App", method: "name") }
+
+    # If no match is found, replace the whole match with a prefix hidden_ along with a random 10 char hex string
+    if_no_match { replace("full").with("hex", prefix: "hidden_", length: 10) }
+  end
+
+  # Hardcoded values that we may wish to reference
+  place_holders({account_number: 1_234_567_890})
+end
+
+export do
+  # Export to terminal a table with only 10 rows.
+  terminal title: "Exported", row_count: 10
+end
+```
+
+With these rules, the above file becomes:
+
+Output:
+
+| id | identity/LineItemId                                  | bill/PayerAccountId | lineItem/UsageAccountId | lineItem/ProductCode | lineItem/ResourceId                          |
+|----|------------------------------------------------------|---------------------|-------------------------|----------------------|----------------------------------------------|
+| 1  | ozsmh5j4oqnfgnv7k82tx1yne4h62rt2rfiilo0clt306ts9ib9g | 1234567890          | 1234567890              | AmazonS3             | cloudtrail-1234567890                        |
+| 2  | soha1946igwsaz8iju4a6q9305yd1cj9gluqwxu6lmjor1wf4yb0 | 1234567890          | 1234567890              | AmazonS3             | cloudtrail-1234567890                        |
+| 3  | k5a29qle33aqoemi74m75pwmhv5xq4sau6e6pyc9pc93g6stzk8s | 1234567890          | 1234567890              | AmazonCloudFront     | arn:aws:cloudfront::1234567890:Latlux        |
+| 4  | 9i0pxzj7mgfy2nnjhalxatck9xidqt55vvmopiotv23raaol9wh1 | 1234567890          | 1234567890              | AmazonS3             | aws-cloudtrail-logs-1234567890               |
+| 5  | uvws7h5xqc8qov8ana6arxyr0urkhpgu9a0g3wzv1emq9z19bl9m | 1234567890          | 1234567890              | AmazonCloudFront     | arn:aws:cloudfront::1234567890:Latlux        |
+| 6  | lhv6swfx2ulsfs8mpfrjutgq45kixouh0xjfvfo40g42757r7mje | 1234567890          | 1234567890              | awskms               | arn:aws:kms:ap-southeast-2:1234567890:Sonair |
+| 7  | zm6gwy8c5qxbe24du6oipdls3iyjp83a3000z6p1l26xo44e0swa | 1234567890          | 1234567890              | AWSLambda            | arn:aws:lambda:Biodex                        |
+| 8  | xcpy7jqbash47ckhyv8bnaqrf1tvsmrqq325vbebu550v7nnhef5 | 1234567890          | 1234567890              | AmazonEC2            | vol-1234567890                               |
+| 9  | o1b4h0yvkw0jkbrhewqr1s0cd9abyqol1r90jtitu7vcr2e6qvcb | 1234567890          | 1234567890              | AmazonS3             | cloudtrail-1234567890                        |
+| 10 | o1b4h0yvkw0jkbrhewqr1s0cd9abyqol1r90jtitu7vcr2e6qvcb | 1234567890          | 1234567890              | AmazonS3             | cloudtrail-1234567890                        |
+
+Note that rows 9 and 10 have the same **identity/LineItemId**, and **lineItem/ResourceId** references our new made up account number.

data/docs/export/main.md

@@ -0,0 +1,37 @@
+Source > Extract > Validate > Build > Query > Transform > **Export**
+
+Export
+=======
+
+### About
+
+Exporting is the final step, where you are given each row at the end of each previous step. You can have multiple 
+exporters, that can each point to different named ranges, or the same.
+
+A common pattern is to export the first 10 rows to terminal, and export the larger dataset to a CSV.
+
+---
+
+**When you should use this**: You have transformed your data, and you want to save the results.
+
+See below an example configuration block:
+
+### Example
+
+```ruby
+export do
+  # Export to terminal window
+  terminal title: "Exported", limit_rows: 5, named_range: "mysheet"
+  
+  # Export to a single CSV
+  csv file_name: "mysheet", directory: "/tmp/cure", named_range: "mysheet"
+  
+  # Export to multiple CSVs each with 100 rows. 
+  # These will be exported as 1_mysheet.csv, 2_mysheet.csv... n_mysheet.csv
+  chunk_csv file_name_prefix: "mysheet", directory: "/tmp/cure", chunk_size: 100, named_range: "mysheet"
+  
+  # Yield out each row to a custom proc. This allows for the caller to do whatever they want
+  # with the row. You could use this to make a API call to insert data to remote system.
+  yield_row named_range: "mysheet", proc: proc { |row| puts row }
+end
+```

data/docs/extract/main.md

@@ -0,0 +1,89 @@
+Source > **Extract** > Validate > Build > Query > Transform > Export
+
+Extract
+=======
+
+### About
+
+The extract step is the first step that is undertaken on the spreadsheet.  If the spreadsheet is in the form you need,
+(where headers and rows are in the right place), this step is not necessary.
+
+There are two main processes that are available in this section; named ranges and variables.
+
+**Named ranges** are a subset of your spreadsheets data.  In some situations, spreadsheets may have more than one section
+of data that you are interested in. Using named ranges, and simple notation (eg. B2:G6), you can select as many ranges
+as needed, and format them back together at the end.
+
+**Variables** are a single row value that is extracted into a hash, and available at the transform stage. A common use
+for this would be to extract a value from somewhere in the spreadsheet to allow it to be added to each row.
+
+---
+
+**When you should use this**: You have a spreadsheet that has more data than you need, or is in a format that is not 
+strictly in a tabular format.  You may want to extract a part (or multiple parts) of the spreadsheet, and discard the
+rest.
+
+See below an example configuration block:
+
+### Example
+
+```ruby
+extract do
+  named_range name: "main", at: "B2:D4", headers: "B2:B4", ref_name: "_default"
+  named_range name: "secondary", at: "A2:D3", ref_name: "_default"
+  named_range name: "full", at: -1, ref_name: "_default"
+  
+  variable name: "my_string", at: "E5", ref_name: "_default"
+end
+```
+
+- `name`: represents what you want to call the named range, mandatory.
+- `at`: specifies the named range location in the sheet. -1 will collect the entire sheet.
+- `headers`: specifies the named range location of the headers. Leave off unless they are not on the top row.
+- `ref_name`: specifies the file to use to extract the named range. If you are only processing a single file you 
+do not need to supply (default ref_name is "_default").
+
+If you do not supply any named range, a default named range is given "_default" which encompasses the entire sheet.
+You do not need to supply this in other parts of the template as if they are not set, they will default to "_default".
+
+Original input:
+```
++----+----+----+----+----+
+| a1 | b1 | c1 | d1 | e1 |
+| a2 | b2 | c2 | d2 | e2 |
+| a3 | b3 | c3 | d3 | e3 |
+| a4 | b4 | c4 | d4 | e4 |
+| a5 | b5 | c5 | d5 | e5 |
++----+----+----+----+----+
+```
+changes to: 
+```
++--------------+
+| main         |
++----+----+----+
+| b2 | c2 | d2 |
+| b3 | c3 | d3 |
+| b4 | c4 | d4 |
++----+----+----+
+
++----+----+----+----+
+| secondary         |
++----+----+----+----+
+| a2 | b2 | c2 | d2 |
+| a3 | b3 | c3 | d3 |
++----+----+----+----+
+
++----+----+----+----+----+
+| full                   |
++----+----+----+----+----+
+| a1 | b1 | c1 | d1 | e1 |
+| a2 | b2 | c2 | d2 | e2 |
+| a3 | b3 | c3 | d3 | e3 |
+| a4 | b4 | c4 | d4 | e4 |
+| a5 | b5 | c5 | d5 | e5 |
++----+----+----+----+----+
+
+variables 
+  - my_string => "e5" 
+```
+

data/docs/metadata/main.md

@@ -0,0 +1,29 @@
+**Metadata** > Source > Extract > Validate > Build > Query > Transform > Export
+
+Metadata
+=======
+
+### About
+
+The metadata step will not affect the process, but allows you to document things you might want to in the template.
+
+---
+
+**When you should use this**: You want to record some information - version, author, date. 
+
+See below an example configuration block:
+
+### Example
+
+```ruby
+metadata do
+  name "My Dataset"
+  version 1
+  comments "A useless comment"
+  additional data: {
+    created_date: "2023-01-01 00:00",
+    author: "william"
+  }
+end
+```
+

data/docs/query/main.md

@@ -0,0 +1,45 @@
+Source > Extract > Validate > Build > **Query** > Transform > Export
+
+Query
+=======
+
+### About
+
+The query step allows you to customise what data is returned from the extract step.
+
+If this step is not provided, `SELECT * FROM _default` is run. Whatever you put in the SELECT (aliases etc) will
+be returned to you for transforming.
+
+---
+
+**When you should use this**: You want to harness the full power of SQL to return a more tailored response. 
+
+See below an example configuration block:
+
+### Example
+
+```ruby
+query do
+  with named_range: "data_log", query: <<-SQL
+    SELECT
+      *
+    FROM 
+      data_log
+    WHERE
+      Equipment = 'Raw'
+    AND
+      (Division = 'O' OR Division = 'Open')
+    AND
+      Event = 'SBD'
+    AND
+      ParentFederation = 'IPF'
+    AND
+      Sex = 'F'
+    AND
+      strftime('%Y', Date) > '2014'
+    ORDER BY Date DESC
+  SQL
+end
+
+```
+