remi 0.0.1 → 0.2.2

data/lib/remi/project/features/sample_job.feature

@@ -0,0 +1,304 @@
+Feature: This is a sample feature file.
+  It demonstrates some of the functionality of using Remi for BRDD.
+
+  Background:
+    Given the job is 'Sample'
+    And the job source 'Sample File'
+    And the job source 'Existing Contacts'
+    And the job target 'Contact Updates'
+    And the job target 'Contact Creates'
+
+    And the following example record called 'existing contact':
+      | Id                 | External_ID__c |
+      | 003G0030H1BxH0xIIF | SAMP12345      |
+
+    And the following example record called 'sample records to load':
+      | Program | Student Id |
+      | BIO     | 12345      |
+      | BIO     | 0          |
+
+    And the following example record called 'sample record to update':
+      | Program | Student Id |
+      | BIO     | 12345      |
+
+    And the following example record called 'sample record to create':
+      | Program | Student Id |
+      | BIO     | 0          |
+
+#####     File Specifications     #####
+
+  Scenario: Selecting and downloading the appropriate source files
+
+    Given the source 'Sample File'
+    And files with names matching the pattern /^SampleFile_(\d+)\.txt/
+    Then the file with the latest date stamp will be downloaded for processing
+
+    Given files with names that do not match the pattern /^SampleFile_(\d+)\.txt/
+    Then no files will be downloaded for processing
+
+
+  Scenario: In order to be parsed and properly processed, the file must conform
+    to expectations about its structure and content.
+
+    Given the source 'Sample File'
+    And the source file is delimited with a comma
+    And the source file is encoded using "ISO-8859-1" format
+    And the source file uses a double quote to quote embedded delimiters
+    And the source file uses a preceeding double quote to escape an embedded quoting character
+    And the source file uses windows or unix line endings
+    And the source file contains a header row
+    And the source file contains at least the following headers in no particular order:
+      | header                       |
+      | Student Id                   |
+      | School Id                    |
+      | School Name                  |
+      | Program                      |
+      | Last Name                    |
+      | First Name                   |
+      | Current Email                |
+      | Mailing Address Line 1       |
+      | Mailing Address Line 2       |
+      | Mailing City                 |
+      | Mailing State                |
+      | Mailing Postal Code          |
+      | Birthdate                    |
+      | Applied Date                 |
+
+
+#####     Record Acceptance     #####
+
+  Scenario Outline: 'Sample File' records are rejected for non-matching programs.
+    Records are rejected from all targets without error unless the
+    'Program' name is included in the list of acceptable program
+    names.  The examples below are not exhaustive.
+
+
+    Given the source 'Sample File'
+    And the target 'Contact Updates'
+    And the target 'Contact Creates'
+    And the example 'sample records to load' for 'Sample File'
+    And the example 'existing contact' for 'Existing Contacts'
+
+    When the source field 'Program' has the value "<Program>"
+    Then the record should be <Action> without error
+
+    Examples:
+      | Program      | Action    |
+      | BIO          | Retained  |
+      | Fake Biology | Rejected  |
+      | COMM         | Rejected  |
+      | CHEM         | Retained  |
+
+
+#####     Transformations common to both contact updates and creates     #####
+
+  Scenario Outline: Populating Major__c for updates and creates.
+    This scenario uses the program name mapping.  The examples below
+    are not exhaustive, but are intended for demonstration and testing
+    of edge cases.
+
+    Given the source 'Sample File'
+    And the target 'Contact Updates'
+    And the target 'Contact Creates'
+    And the example 'existing contact' for 'Existing Contacts'
+    And the example 'sample records to load' for 'Sample File'
+
+    When the source field 'Program' has the value "<Program>"
+    Then the target field 'Major__c' is set to the value "<Major__c>"
+
+    Examples:
+      | Program            | Major__c  |
+      | BIO                | Biology   |
+      | CHEM               | Chemistry |
+      | Chemistry          | Chemistry |
+      | Physical Chemistry | Chemistry |
+      | Microbiology       | Biology   |
+
+
+#####     Transformations for creating contact records     #####
+
+  Scenario: Fields that are present on create
+    Given the target 'Contact Creates'
+    And the example 'existing contact' for 'Existing Contacts'
+    And the example 'sample record to create' for 'Sample File'
+
+    Then only the following fields should be present on the target:
+      | Field Name        |
+      | External_ID__c    |
+      | School_ID__c      |
+      | School_Name__c    |
+      | School__c         |
+      | Major__c          |
+      | FirstName         |
+      | LastName          |
+      | Email             |
+      | MailingStreet     |
+      | MailingCity       |
+      | MailingState      |
+      | MailingPostalCode |
+      | Birthdate         |
+      | Applied_Date__c   |
+
+
+  Scenario: Records that are directed to Contact creates
+    This example shows how to create a scenario that simultaneously references
+    two sources.
+
+    Given the source 'Sample File'
+    And the source 'Existing Contacts'
+    And the target 'Contact Creates'
+    And the example 'sample record to create' for 'Sample File'
+    And the example 'existing contact' for 'Existing Contacts'
+
+    And the source field 'Sample File: Student Id'
+    And the source field 'Existing Contacts: External_ID__c' has the value "some arbitrary value"
+    Then a target record is created
+
+    When the source field 'Existing Contacts: External_ID__c' has the value in the source field 'Sample File: Student Id', prefixed with "SAMP"
+    Then a target record is not created
+
+
+  Scenario Outline: Fields that are copied from the 'Sample File' to 'Contact Creates'
+    This example shows how a scenario can be used to specify very simple
+    field mappings, like a simple copy with missing value handling.
+
+    Given the source 'Sample File'
+    And the target 'Contact Creates'
+    And the example 'sample record to create' for 'Sample File'
+
+    And the source field '<Source Field>'
+    Then the target field '<Target Field>' is copied from the source field
+
+    When the source field is blank
+    Then the target field '<Target Field>' is populated with "<If Blank>"
+
+    Examples:
+      | Source Field        | Target Field      | If Blank     |
+      | School Id           | School_ID__c      |              |
+      | School Name         | School_Name__c    |              |
+      | First Name          | FirstName         | Not Provided |
+      | Last Name           | LastName          | Not Provided |
+      | Mailing City        | MailingCity       |              |
+      | Mailing State       | MailingState      |              |
+      | Mailing Postal Code | MailingPostalCode |              |
+
+
+  Scenario Outline: Date fields that are parsed and then copied from the Sample File
+    to 'Contact Creates'
+
+    Given the source 'Sample File'
+    And the target 'Contact Creates'
+    And the example 'sample record to create' for 'Sample File'
+
+    And the source field '<Source Field>'
+    And the source field is parsed with the date format "<Source Format>"
+    Then the target field '<Target Field>' is populated from the source field using the format "<Target Format>"
+
+    When the source field is blank
+    Then the target field '<Target Field>' is populated with "<If Blank>" using the format "<Target Format>"
+
+    Examples:
+      | Source Field | Source Format | Target Field    | Target Format | If Blank       |
+      | Applied Date | %m/%d/%Y      | Applied_Date__c | %Y-%m-%d      | *Today's Date* |
+      | Birthdate    | %m/%d/%Y      | Birthdate       | %Y-%m-%d      |                |
+
+
+  Scenario: Populating School__c
+    This is an example of creating a field that is a delimited
+    concatenation of two fields, where we replace blank values with "Unknowns"
+
+    Given the source 'Sample File'
+    And the target 'Contact Creates'
+    And the example 'existing contact' for 'Existing Contacts'
+    And the example 'sample records to load' for 'Sample File'
+
+    And the source field 'School Id'
+    And the source field 'School Name'
+    And the target field 'School__c'
+    Then the target field is a concatenation of the source fields, delimited by "-"
+
+    When the source field 'School Id' is blank
+    And the source field 'School Name' has the value "some arbitrary value"
+    Then the target field is a concatenation of "Unknown" and 'School Name', delimited by "-"
+
+    When the source field 'School Id' has the value "some arbitrary value"
+    And the source field 'School Name' is blank
+    Then the target field is a concatenation of 'School Id' and "Unknown", delimited by "-"
+
+
+  Scenario: Populating Email
+    This example shows some pre-cleaning of e-mail addresses (replacing commas with periods)
+
+    Given the source 'Sample File'
+    And the target 'Contact Creates'
+    And the example 'sample record to create' for 'Sample File'
+
+    And the source field 'Current Email'
+    And the target field 'Email'
+
+    When the source field is a valid email address
+    Then the target field is copied from the source field
+
+    When the source field is not a valid email address
+    Then the target field is populated with ""
+
+    When the source field is a valid email address
+    And in the source field, periods have been used in place of commas
+    Then the target field is copied from the source field, but commas have been replaced by periods
+
+
+  Scenario: Populating External_ID__c
+
+    Given the source 'Sample File'
+    And the target 'Contact Creates'
+    And the example 'sample record to create' for 'Sample File'
+
+    And the source field 'Student Id'
+    And the target field 'External_ID__c'
+    Then the source field is prefixed with "SAMP" and loaded into the target field
+
+
+  Scenario: Populating MailingStreet
+    Given the source 'Sample File'
+    And the target 'Contact Creates'
+    And the example 'sample record to create' for 'Sample File'
+
+    And the source field 'Mailing Address Line 1'
+    And the source field 'Mailing Address Line 2'
+    And the target field 'MailingStreet'
+    Then the target field is a concatenation of the source fields, delimited by ", "
+
+    When the source field 'Mailing Address Line 1' is blank
+    Then the target field is populated with ""
+
+
+#####     Transformations for updating contact records     #####
+
+  Scenario: Fields that are present on update
+    The business rules specify that once a student record has been
+    created, only updates to the program are accepted (all other
+    changes to be done in Salesforce)
+
+    Given the target 'Contact Updates'
+    And the example 'existing contact' for 'Existing Contacts'
+    And the example 'sample record to create' for 'Sample File'
+
+    Then only the following fields should be present on the target:
+      | Field Name      |
+      | Id              |
+      | Major__c        |
+
+  Scenario: Records that are directed to Contact updates
+
+    Given the source 'Sample File'
+    And the source 'Existing Contacts'
+    And the target 'Contact Updates'
+    And the example 'sample record to update' for 'Sample File'
+    And the example 'existing contact' for 'Existing Contacts'
+
+    And the source field 'Sample File:  Student Id'
+    And the source field 'Existing Contacts: External_ID__c' has the value in the source field 'Sample File: Student Id', prefixed with "SAMP"
+    Then a target record is created
+
+    When the source field 'Existing Contacts: External_ID__c' has the value "some arbitrary value"
+    Then a target record is not created

data/lib/remi/project/features/step_definitions/remi_step.rb

@@ -0,0 +1,310 @@
+# Auto-generated from Remi.  Editing this file is not recommended.  Instead,
+# create another step definition file to contain application-specific steps.
+
+### Job and background setup
+
+Given /^the job is '([[:alnum:]\s]+)'$/ do |arg|
+  @brt = Remi::BusinessRules::Tester.new(arg)
+end
+
+Given /^the job source '([[:alnum:]\s\-_]+)'$/ do |arg|
+  @brt.add_job_source arg
+end
+
+Given /^the job target '([[:alnum:]\s\-_]+)'$/ do |arg|
+  @brt.add_job_target arg
+end
+
+Given /^the following example record called '([[:alnum:]\s\-_]+)':$/ do |arg, example_table|
+  @brt.add_example arg, example_table
+end
+
+Given /^the job parameter '([[:alnum:]\s\-_]+)' is "([^"]*)"$/ do |param, value|
+  @brt.set_job_parameter(param, value)
+end
+
+### Setting up example data
+
+Given /^the following example record for '([[:alnum:]\s\-_]+)':$/ do |source_name, example_table|
+  example_name = source_name
+  @brt.add_example example_name, example_table
+  @brt.job_sources[source_name].stub_data_with(@brt.examples[example_name])
+end
+
+Given /^the example '([[:alnum:]\s\-_]+)' for '([[:alnum:]\s\-_]+)'$/ do |example_name, source_name|
+  @brt.job_sources[source_name].stub_data_with(@brt.examples[example_name])
+end
+
+
+### Source file processing
+
+Given /^files with names matching the pattern \/(.*)\/$/ do |pattern|
+  @brt.filestore.pattern(Regexp.new(pattern))
+end
+
+Given /^files with names that do not match the pattern \/(.*)\/$/ do |pattern|
+  @brt.filestore.anti_pattern(Regexp.new(pattern))
+end
+
+Given /^files delivered within the last (\d+) hours$/ do |hours|
+  @brt.filestore.delivered_since(Time.now - hours.to_i * 3600)
+end
+
+Given /^files were delivered more than (\d+) hours ago$/ do |hours|
+  @brt.filestore.delivered_before(Time.now - hours.to_i * 3600)
+end
+
+Then /^the file with the latest date stamp will be downloaded for processing$/ do
+  @brt.filestore.generate
+  @brt.source.mock_extractor(@brt.filestore)
+  expect(@brt.source.extract).to match_array Array(@brt.filestore.latest)
+end
+
+Then /^no files will be downloaded for processing$/ do
+  @brt.filestore.generate
+  @brt.source.mock_extractor(@brt.filestore)
+  expect { @brt.source.extract }.to raise_error Remi::Extractor::SftpFile::FileNotFoundError
+end
+
+
+Given /^the source file is delimited with a (\w+)$/ do |delimiter|
+  expect(@brt.source.csv_options[:col_sep]).to eq Remi::BusinessRules.csv_opt_map[delimiter]
+end
+
+Given /^the source file is encoded using "([^"]+)" format$/ do |encoding|
+  expect(@brt.source.csv_options[:encoding].split(':').first).to eq encoding
+end
+
+Given /^the source file uses a ([\w ]+) to quote embedded delimiters$/ do |quote_char|
+  expect(@brt.source.csv_options[:quote_char]).to eq Remi::BusinessRules.csv_opt_map[quote_char]
+end
+
+Given /^the source file uses a preceeding ([\w ]+) to escape an embedded quoting character$/ do |escape_char|
+  expect(@brt.source.csv_options[:quote_char]).to eq Remi::BusinessRules.csv_opt_map[escape_char]
+end
+
+Given /^the source file uses ([\w ]+) line endings$/ do |line_endings|
+  expect(@brt.source.csv_options[:row_sep]).to eq Remi::BusinessRules.csv_opt_map[line_endings]
+end
+
+Given /^the source file (contains|does not contain) a header row$/ do |header|
+  expect(@brt.source.csv_options[:headers]).to eq (header == 'contains')
+end
+
+Given /^the source file contains at least the following headers in no particular order:$/ do |table|
+  table.rows.each do |row|
+    field = row.first
+    step "the source field '#{field}'"
+  end
+  expect(@brt.source.data_obj.fields.keys).to include(*@brt.source.fields.names)
+end
+
+
+### Source
+
+Given /^the source '([[:alnum:]\s\-_]+)'$/ do |arg|
+  @brt.add_source(arg)
+end
+
+Given /^the source field '(.+)'$/ do |arg|
+  @brt.sources.add_field(arg)
+end
+
+Given /^the source field has the value "([^"]*)"$/ do |arg|
+  @brt.source.field.value = Remi::BusinessRules::ParseFormula.parse(arg)
+end
+
+When /^the source field (?:has an empty value|is blank)$/ do
+  @brt.source.field.value = ''
+end
+
+When /^the source field '(.+)' (?:has an empty value|is blank)$/ do |source_field|
+  step "the source field '#{source_field}'"
+  @brt.source.fields[source_field].value = ''
+end
+
+Given /^the source field '([^:]+)' (?:has|is set to) the value "([^"]*)"$/ do |source_field, value|
+  step "the source field '#{source_field}'"
+  @brt.source.fields[source_field].value = Remi::BusinessRules::ParseFormula.parse(value)
+end
+
+Given /^the source field '(.+:.+)' (?:has|is set to) the value "([^"]*)"$/ do |source_field, value|
+  step "the source field '#{source_field}'"
+  source_name, field_name = *Remi::BusinessRules.parse_full_field(source_field)
+  @brt.sources[source_name].fields[field_name].value = Remi::BusinessRules::ParseFormula.parse(value)
+end
+
+Given /^the source field '(.+:.+)' (?:has|is set to) the value in the source field '(.+:.+)', prefixed with "([^"]*)"$/ do |source_field, other_source_field, prefix|
+  step "the source field '#{source_field}'"
+  step "the source field '#{other_source_field}'"
+  source_name, field_name = *Remi::BusinessRules.parse_full_field(source_field)
+  other_source_name, other_field_name = *Remi::BusinessRules.parse_full_field(other_source_field)
+
+  prefixed = "#{prefix}#{@brt.sources[other_source_name].fields[other_field_name].value}"
+  @brt.sources[source_name].fields[field_name].value = prefixed
+end
+
+Given /^the source field is parsed with the date format "([^"]*)"$/ do |date_format|
+  expect(@brt.source.field.metadata[:format]).to eq date_format
+end
+
+Given /^the source field is a valid email address$/ do
+  @brt.source.field.value = 'valid@example.com'
+end
+
+Given /^the source field is not a valid email address$/ do
+  @brt.source.field.value = 'invalid!example.com'
+end
+
+### Target
+
+Given /^the target '([[:alnum:]\s\-_]+)'$/ do |arg|
+  @brt.add_target(arg)
+end
+
+Given /^the target field '([^']+)'$/ do |arg|
+  @brt.targets.add_field(arg)
+end
+
+Then /^the target field '(.+)' is copied from the source field$/ do |arg|
+  step "the target field '#{arg}'"
+  step "the target field is copied from the source field"
+end
+
+Then /^the target field is copied from the source field$/ do
+  @brt.run_transforms
+  expect(@brt.target.field.value).to eq (@brt.source.field.value)
+end
+
+Then /^the target field '(.+)' is copied from the source field '(.+:.+)'$/ do |target_field, source_field|
+  step "the target field '#{target_field}'"
+  step "the source field '#{source_field}'"
+
+  source_name, source_field_name = *Remi::BusinessRules.parse_full_field(source_field)
+
+  @brt.run_transforms
+  expect(@brt.target.field.value).to eq (@brt.sources[source_name].fields[source_field_name].value)
+end
+
+Then /^the target field is (?:set to the value|populated with) "([^"]*)"$/ do |value|
+  @brt.run_transforms
+  expect(@brt.target.field.value).to eq Remi::BusinessRules::ParseFormula.parse(value)
+end
+
+Then /^the target field '(.+)' is (?:set to the value|populated with) "([^"]*)"$/ do |target_field, value|
+  @brt.targets.add_field(target_field)
+  @brt.run_transforms
+  expect(@brt.targets.fields[target_field].values.uniq).to eq [Remi::BusinessRules::ParseFormula.parse(value)]
+end
+
+Then /^the target field '(.+)' is the date (.+)$/ do |target_field, date_reference|
+  step "the target field '#{target_field}' is set to the value \"*#{date_reference}*\""
+end
+
+
+### Transforms
+
+Then /^the target field is a concatenation of the source fields, delimited by "([^"]*)"$/ do |delimiter|
+  concatenated_source = @brt.sources.fields.values.uniq.map do |row|
+    Array(row.join(delimiter))
+  end
+
+  @brt.run_transforms
+  expect(@brt.targets.fields.values.uniq).to eq concatenated_source
+end
+
+Then /^the target field is a concatenation of "([^"]*)" and '(.+)', delimited by "([^"]*)"$/ do |constant, source_field, delimiter|
+  expected_value = [constant, @brt.sources.fields[source_field].value].join(delimiter)
+  @brt.run_transforms
+  expect(@brt.targets.fields.values.uniq).to eq [[expected_value]]
+end
+
+Then /^the target field is a concatenation of '(.+)' and "([^"]*)", delimited by "([^"]*)"$/ do |source_field, constant, delimiter|
+  expected_value = [@brt.sources.fields[source_field].value, constant].join(delimiter)
+  @brt.run_transforms
+  expect(@brt.targets.fields.values.uniq).to eq [[expected_value]]
+end
+
+Then /^the source field is prefixed with "([^"]*)" and loaded into the target field$/ do |prefix|
+  prefixed_source = "#{prefix}#{@brt.source.field.value}"
+  @brt.run_transforms
+  expect(@brt.target.field.value).to eq prefixed_source
+end
+
+Then /^the target field '(.+)' is populated from the source field using the format "([^"]*)"$/ do |target_field, target_format|
+  source_format = @brt.source.field.metadata[:format]
+  source_reformatted = Remi::Transform[:format_date].(from_fmt: source_format, to_fmt: target_format)
+    .call(@brt.source.field.value)
+
+  step "the target field '#{target_field}'"
+  @brt.run_transforms
+  expect(@brt.target.field.value).to eq source_reformatted
+end
+
+Then /^the target field '(.+)' is populated with "([^"]*)" using the format "([^"]*)"$/ do |target_field, target_value, target_format|
+  source_format = @brt.source.field.metadata[:format]
+  target_value_source_format = target_value == "*Today's Date*" ? Date.today.strftime(source_format) : target_value
+  target_reformatted = Remi::Transform[:format_date].(from_fmt: source_format, to_fmt: target_format)
+   .call(target_value_source_format)
+
+  step "the target field '#{target_field}'"
+  @brt.run_transforms
+  expect(@brt.target.field.value).to eq target_reformatted
+end
+
+
+When /^in the source field, periods have been used in place of commas$/ do
+  @brt.source.field.value = @brt.source.field.value.gsub(/\./, ',')
+end
+
+Then /^the target field is copied from the source field, but commas have been replaced by periods$/ do
+  source_field_value = @brt.source.field.value
+
+  @brt.run_transforms
+  expect(@brt.target.field.value).to eq source_field_value.gsub(/,/, '.')
+end
+
+
+### Field presence
+
+Then /^only the following fields should be present on the target:$/ do |table|
+  table.rows.each do |row|
+    field = row.first
+    step "the target field '#{field}'"
+  end
+
+  @brt.run_transforms
+  expect(@brt.target.data_obj.fields.keys).to match_array @brt.target.fields.names
+end
+
+### Record-level expectations
+
+Then /^the record should be (?i)(Retained|Rejected)(?-i)(?: without error|)$/ do |action|
+  source_size  = @brt.source.size
+  @brt.run_transforms
+  targets_size = @brt.targets.total_size
+
+  case
+  when action.downcase == 'retained'
+    expect(targets_size).to eq source_size
+  when action.downcase == 'rejected'
+    expect(targets_size).to eq 0
+  else
+    raise "Unknown action #{action}"
+  end
+end
+
+Then /^the record should (not be|be) present on the target$/ do |action|
+  map_action = { 'not be' => 'rejected', 'be' => 'retained' }
+  step "the record should be #{map_action[action]}"
+end
+
+Then /^a target record is created$/ do
+  @brt.run_transforms
+  expect(@brt.targets.total_size).to be > 0
+end
+
+Then /^a target record is not created$/ do
+  @brt.run_transforms
+  expect(@brt.targets.total_size).to be 0
+end