cure 0.1.1 → 0.4.0

data/docs/sources/main.md

@@ -0,0 +1,36 @@
+**Sources** > Extract > Validate > Build > **Query** > Transform > Export
+
+Sources
+=======
+
+### About
+
+The sources step allows you to define the files that will be extracted. You can supply it in the template,
+or you can provide it when you are initializing `Cure::Launcher`.
+
+As Cure templates are just Ruby, you can define the files dynamically, and provide them to the template DSL.
+
+---
+
+**When you should use this**: You want to load a CSV file, but you don't want to do it at initialize.
+
+See below an example configuration block:
+
+### Example
+
+```ruby
+sources do
+  csv :pathname, Pathname.new("loc/to/my/file_1.csv"), ref_name: "file_1"
+  csv :pathname, Pathname.new("loc/to/my/file_2.csv"), ref_name: "file_2"
+end
+```
+
+or using Ruby (this is a contrived example)
+
+```ruby
+[Pathname.new("loc/to/my/file_1.csv"), Pathname.new("loc/to/my/file_2.csv")].each_with_index do |file_path, idx|
+  sources do
+    csv :pathname, file_path, ref_name: "file_#{idx}"
+  end
+end
+```

data/docs/transform/main.md

@@ -0,0 +1,53 @@
+Source > Extract > Validate > Build > Query > **Transform** > Export
+
+Transform
+=======
+
+### About
+
+The transform step is the arguably the most powerful step in the chain.  It is used when you want to *change* a value.
+The process can be undertaken in multiple steps, performing transforms as you go. It is quite sophisticated, and does
+not require homologous data for each value.
+
+**When you should use this**: You have a value in a row, and you wish to change it.  
+
+Transform is made up of many translations.  There are two main parts that make up a translations step, the 
+**strategy** and the **generator** which are defined on a specific column.
+
+**Strategy** is the means in which a value will be transformed. It is responsible for how and what part of the value 
+will be transformed. The strategy will first look to extract the value that you wish to transform.  This can either
+be simple, like a full replacement, or complex using regex.  Once it has the value extracted, it will attempt to find
+any previously translated mapping before running it through the generator step.
+
+**Generator** is the process of creating that transformed value.  It could be as simple as replace all characters with
+the character "x", fill it with Faker data, set it to a random guid, modify the date format, inject a value from 
+elsewhere on the spreadsheet (see [Build](../builder/main.md)).  
+
+Once this value has been generated, it is stored against the existing value so any other translation will get the same.
+This is particularly useful if you want to keep data integrity. For example, if you had two columns, one was `account
+number`, and the other was `reference_id` (which was constructed by the format `account_number/identifier`).  Once you 
+had set a replacement value for the account number (`abcde => defgh`), you wouldn't want a different value for the 
+account number part of the `reference_id` column. 
+
+```
++----------------+--------------+
+| account_number | reference_id |
++----------------+--------------+
+| abcde          | abcde/123456 |
++----------------+--------------+
+```
+changes to
+```
++----------------+--------------+
+| account_number | reference_id |
++----------------+--------------+
+| defgh          | defgh/123456 |
++----------------+--------------+
+```
+
+---
+
+See below an example configuration block:
+
+### Example
+

data/docs/validate/main.md

@@ -0,0 +1,42 @@
+Source > Extract > **Validate** > Build > Query > Transform > Export
+
+Validate
+=======
+
+### About
+
+The validator step checks that data in the spreadsheet is what is expected. This may involve checking
+that data is of a certain format, and if it isn't, it can either fail or warn.
+
+---
+
+**When you should use this**: You have a spreadsheet that has data that needs to be in a certain format
+before transforming or exporting. 
+
+See below an example configuration block:
+
+### Example
+
+```ruby
+validate do
+  candidate column: "new_column", named_range: "section_1", options: { fail_on_error: false } do
+    with_rule :not_null
+    with_rule :length, { min: 0, max: 5 }
+    with_rule :custom, { proc: Proc.new { |x| x.size >= 0 && x.size <= 5 } } # Proc version of above.
+  end
+end
+```
+
+Original input:
+```
++------------+
+| new_column | 
++------------+
+| test       | <- Is valid
+| test value | <- Is invalid (too long)
+|            | <- Is invalid (null)
++------------+
+```
+
+As `fail_on_error` is set to false, this would warn that 2 values are invalid.
+

data/exe/cure

@@ -1,42 +1,17 @@
 #!/usr/bin/env ruby
-# frozen_string_literal: true
 
-require "cure"
-require "optparse"
+require "cure/cli/command"
+require "cure/cli/new_command"
+require "cure/cli/run_command"
+require "cure/cli/generate_command"
 
-include Cure::Log
-
-class CureConfig
-  include Cure::Configuration
-end
-
-conf = {}
-
-OptionParser.new do |opts|
-  opts.banner = "Usage: cure_cli [options]"
-
-  opts.on("-t", "--template_file=template_file", "Template definition file") do |t|
-    conf[:template_file_location] = t
-  end
-
-  opts.on("-s", "--source_file=source_file", "Source file") do |s|
-    conf[:source_file_location] = s
-  end
-
-  opts.on("-o", "--output_dir=output_dir", "Output directory") do |o|
-    conf[:output_dir] = o
-  end
-
-  opts.on("-h", "--help", "Prints this help") do
-    puts opts
-    exit
-  end
-end.parse!
-
-log_info "Config loaded successfully, initialising environment ..."
-main = Cure::Main.init(conf[:template_file_location], conf[:source_file_location], conf[:output_dir])
-
-log_info "... set up complete. Beginning process"
-main.process
+short_hand = {
+  "n" => "new",
+  "r" => "run",
+  "g" => "generate"
+}
 
+command = ARGV.shift
+command = short_hand[command] || command
 
+Cure::Cli::Command.invoke(command, ARGV)

data/exe/cure.old

@@ -0,0 +1,59 @@
+#!/usr/bin/env ruby
+
+# frozen_string_literal: true
+
+# rubocop:disable Style/MixinUsage
+
+require "cure"
+require "optparse"
+
+include Cure::Log
+
+class CureConfig
+  include Cure::Configuration
+end
+
+conf = {}
+
+opts_parser = OptionParser.new do |opts|
+  opts.banner = "Usage: cure [options]"
+
+  opts.on("-t", "--template_file=template_file", "Template definition file") do |t|
+    conf[:template_file_location] = t
+  end
+
+  opts.on("-s", "--source_file=source_file", "Source file") do |s|
+    conf[:source_file_location] = s
+  end
+
+  opts.on("-l", "--list_plugins=list_plugins", "List plugins") do
+    ObjectSpace.each_object(Class) do |klass|
+      log_info "Generator: #{klass.name}" if klass.ancestors.include?(Cure::Generator::BaseGenerator)
+      log_info "Strategy: #{klass.name}" if klass.ancestors.include?(Cure::Strategy::BaseStrategy)
+    end
+
+    exit
+  end
+
+  opts.on("-h", "--help", "Prints this help") do
+    puts opts
+    exit
+  end
+end
+
+opts_parser.parse!
+
+if conf.empty?
+  puts opts_parser
+  exit
+end
+
+log_info "Config loaded successfully, initialising environment ..."
+main = Cure.init_from_file(conf[:template_file_location])
+           .with_csv_file(:pathname, Pathname.new(conf[:source_file_location]))
+           .setup
+
+log_info "... set up complete. Beginning process"
+main.run_export
+
+# rubocop:enable Style/MixinUsage

data/lib/cure/builder/base_builder.rb

@@ -0,0 +1,151 @@
+# frozen_string_literal: true
+
+require "cure/log"
+require "cure/config"
+require "cure/database"
+require "cure/helpers/file_helpers"
+require "cure/extract/extractor"
+
+require "rcsv"
+
+module Cure
+  module Builder
+    include Database
+
+    class BaseBuilder
+      include Database
+
+      # @param [String] named_range
+      # @param [String] column
+      # @param [Hash] opts
+      def initialize(named_range, column, opts)
+        @named_range = named_range
+        @column = column
+        @opts = opts
+      end
+
+      # @return [void]
+      def call
+        raise NotImplementedError, "#{self.class} has not implemented method '#{__method__}'"
+      end
+
+      # @return [String (frozen)]
+      def to_s
+        "Base Builder"
+      end
+
+      # @yield [DatabaseService]
+      # @raise StandardError
+      def with_database(&block)
+        raise "Missing block" unless block
+
+        yield database_service
+      end
+    end
+
+    class AddBuilder < BaseBuilder
+
+      # @return [void]
+      def call
+        with_database do |db_svc|
+          db_svc.add_column(@named_range.to_sym, @column.to_sym, default: @opts.fetch(:default_value, nil))
+        end
+      end
+
+      # @return [String (frozen)]
+      def to_s
+        "Add Builder"
+      end
+    end
+
+    class RemoveBuilder < BaseBuilder
+
+      # @return [void]
+      def call
+        with_database do |db_svc|
+          db_svc.remove_column(@named_range.to_sym, @column.to_sym)
+        end
+      end
+
+      # @return [String (frozen)]
+      def to_s
+        "Remove Builder"
+      end
+    end
+
+    class RenameBuilder < BaseBuilder
+
+      # @return [void]
+      def call
+        with_database do |db_svc|
+          db_svc.rename_column(@named_range.to_sym, @column.to_sym, @opts.fetch("new_name"))
+        end
+      end
+
+      # @return [String (frozen)]
+      def to_s
+        "Rename Builder"
+      end
+    end
+
+    class CopyBuilder < BaseBuilder
+
+      # @return [void]
+      def call
+        with_database do |db_svc|
+          to_column = @opts.fetch(:to_column)
+          db_svc.copy_column(
+            @named_range.to_sym,
+            @column.to_sym,
+            to_column
+          )
+        end
+      end
+
+      # @return [String (frozen)]
+      def to_s
+        "Copy Builder"
+      end
+    end
+
+    class BlacklistBuilder < BaseBuilder
+
+      # @return [void]
+      def call
+        @opts[:columns].each do |column|
+          with_database do |db_svc|
+            db_svc.remove_column(@named_range.to_sym, column.to_sym)
+          end
+        end
+      end
+
+      # @return [String (frozen)]
+      def to_s
+        "Blacklist Builder"
+      end
+    end
+
+    class WhitelistBuilder < BaseBuilder
+
+      # @return [void]
+      def call
+        with_database do |db_svc|
+          whitelist_columns = (@opts[:columns]).map(&:to_sym)
+          all_columns = db_svc.list_columns(@named_range.to_sym)
+
+          # Remove cols that aren't defined in white list or sys columns
+          candidate_cols = all_columns - whitelist_columns - [:_id]
+
+          candidate_cols.each do |column|
+            db_svc.remove_column(@named_range.to_sym, column.to_sym)
+          end
+        end
+      end
+
+      # @return [String (frozen)]
+      def to_s
+        "Whitelist Builder"
+      end
+    end
+  end
+end

data/lib/cure/builder/candidate.rb

@@ -0,0 +1,56 @@
+# frozen_string_literal: true
+
+require "cure/helpers/object_helpers"
+require "cure/builder/base_builder"
+require "cure/extract/extractor"
+require "cure/log"
+
+module Cure
+  module Builder
+    # Per row, we will have a candidate for each transformation that needs to be made
+    class Candidate
+      include Helpers::ObjectHelpers
+      include Log
+
+      # Named range that column exists in
+      # @return [String]
+      attr_reader :named_range
+
+      # Lookup column name for CSV.
+      # @return [String,nil]
+      attr_reader :column
+
+      # What sort of data needs to be generated.
+      # @return [Cure::Builder::BaseBuilder]
+      attr_reader :action
+
+      # @param [String,nil] column
+      # @param [String] named_range
+      def initialize(column, named_range)
+        @column = column
+        @named_range = named_range || "_default"
+      end
+
+      # @return [void]
+      def perform
+        @action.call
+      end
+
+      # @param [String,Symbol] _method_name
+      # @param [TrueClass,FalseClass] _include_private
+      # @return [TrueClass]
+      def respond_to_missing?(_method_name, _include_private=false)
+        true
+      end
+
+      # @param [Symbol] method_name
+      # @return [Cure::Builder::BaseBuilder]
+      def method_missing(method_name, args)
+        klass_name = "Cure::Builder::#{method_name.to_s.capitalize}Builder"
+        raise "#{method_name} is not valid" unless class_exists?(klass_name)
+
+        @action = Kernel.const_get(klass_name).new(@named_range, @column, args[:options] || {})
+      end
+    end
+  end
+end

data/lib/cure/cli/command.rb

@@ -0,0 +1,105 @@
+# frozen_string_literal: true
+
+require "cure"
+require "cure/log"
+
+module Cure
+  module Cli
+    class Command
+      include Log
+
+      # @param [String] command
+      # @param [Array] argv
+      def self.invoke(command, argv)
+        extend Log
+
+        handler = nil
+
+        case command
+        when "new"
+          handler = NewCommand.new(argv)
+        when "generate"
+            handler = GenerateCommand.new(argv)
+        when "run"
+          handler = RunCommand.new(argv)
+        else
+          new.help
+          return
+        end
+
+        handler.call
+      rescue ArgumentError => aex
+        handler ? handler.help(ex: aex) : new.help
+      rescue StandardError => ex
+        log_error(ex.message)
+      end
+
+      # @param [Array] argv
+      def initialize(argv = [])
+        @argv = argv
+      end
+
+      def call
+        validate
+        execute
+      end
+
+      def help(ex: nil)
+        log_error ex.message if ex && ex.is_a?(StandardError)
+        log_error "Error: unknown request"
+        log_info "\nUsage: cure <command> [options]"
+        log_info "\tRun:         cure run -t [template] -f [file]"
+        log_info "\tNew Project: cure new [name]"
+      end
+
+      private
+
+      # @return [void]
+      def execute
+        raise NotImplementedError
+      end
+
+      # @return [void]
+      # @raise [ArgumentError, NotImplementedError]
+      def validate
+        raise NotImplementedError
+      end
+
+      protected
+
+
+      def make_directory(dir_name)
+        log_info "Creating directory #{dir_name}"
+
+        FileUtils.mkdir_p(dir_name)
+      end
+
+      def make_file(dir_name, file_name, template: nil, binding: nil)
+        file = File.join(dir_name, file_name)
+
+        log_info "Creating file #{file}"
+
+        unless template
+          FileUtils.touch(file)
+          return
+        end
+
+        content = retrieve_template(template)
+        if binding
+          erb = ERB.new(content)
+          content = erb.result_with_hash(binding)
+        end
+
+        File.open(file, "w") do |f|
+          f.write(content)
+        end
+      end
+
+      def retrieve_template(template)
+        File.read(
+          File.join(File.dirname(__FILE__), "templates", "#{template}.erb")
+        )
+      end
+    end
+  end
+end

data/lib/cure/cli/generate_command.rb

@@ -0,0 +1,54 @@
+# frozen_string_literal: true
+
+require "cure/cli/command"
+
+# TODO: Generate script - inc File and init and Run export.
+
+module Cure
+  module Cli
+    class GenerateCommand < Command
+      def initialize(args)
+        super(args)
+      end
+
+      def help(ex: nil)
+        log_error ex.message if ex && ex.is_a?(StandardError)
+        log_info "\nUsage: cure generate template [name]"
+      end
+
+      private
+
+      def execute
+        raise "You are not in a Cure project! Cannot create template." unless cure_project?
+
+        log_info "Creating new template: #{params[:name]}"
+
+        root_dir = File.join(Dir.pwd, "templates")
+        make_file(root_dir, "#{params[:name]}_template.rb", template: "new_template.rb")
+      end
+
+      def validate
+        # Theres a nicer way to do this, I'll do it later.
+        raise ArgumentError, "Missing arguments" if @argv.empty?
+        raise ArgumentError, "Invalid operation #{@argv.first}" if @argv.first != "template"
+        raise ArgumentError, "Invalid name #{@argv.last}" if @argv.last.nil? || @argv.last.empty?
+      end
+
+      attr_reader :params
+
+      def params
+        {
+          operation: @argv.first,
+          name: @argv.last
+        }
+      end
+
+      # Naive implementation, could do better. Who cares for now.
+      # @return [TrueClass, FalseClass]
+      def cure_project?
+        gf = File.join(Dir.pwd, 'Gemfile')
+        File.exist?(gf) && File.read(gf).include?('cure')
+      end
+    end
+  end
+end

data/lib/cure/cli/new_command.rb

@@ -0,0 +1,52 @@
+# frozen_string_literal: true
+
+require "cure/version"
+require "cure/cli/command"
+
+require "fileutils"
+require "erb"
+
+module Cure
+  module Cli
+    class NewCommand < Command
+      def initialize(argv)
+        super(argv)
+      end
+
+      def help(ex: nil)
+        log_error ex.message if ex && ex.is_a?(StandardError)
+        log_info "\nUsage: cure new [name]"
+      end
+
+      private
+
+      def execute
+        root_dir = File.join(Dir.pwd, @argv[0])
+
+        raise "Project already exists in directory!" if File.exist?(root_dir)
+
+        log_info "Creating new project: #{@argv[0]}"
+
+        make_directory(root_dir)
+        make_directory(File.join(root_dir, "input"))
+        make_directory(File.join(root_dir, "output"))
+        make_directory(File.join(root_dir, "scripts"))
+        make_directory(File.join(root_dir, "templates"))
+        make_directory(File.join(root_dir, "utilities"))
+
+        make_file(root_dir, ".gitignore", template: "gitignore")
+        make_file(root_dir, ".tool-versions", template: "tool-versions")
+        make_file(root_dir, "README.md", template: "README.md")
+        make_file(root_dir, "Gemfile", template: "gemfile", binding: { version: Cure::VERSION })
+        make_file("#{root_dir}/input", ".gitkeep")
+        make_file("#{root_dir}/output", ".gitkeep")
+      end
+
+      # @return [void]
+      # @raise [ArgumentError]
+      def validate
+        raise ArgumentError, "Error: No project name given" if @argv.empty?
+      end
+    end
+  end
+end

data/lib/cure/cli/run_command.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+require "cure/cli/command"
+
+module Cure
+  module Cli
+    class RunCommand < Command
+      def initialize(args)
+        super(args)
+      end
+
+      private
+
+      def execute
+        log_info "Not implemented yet!"
+      end
+    end
+  end
+end

data/lib/cure/cli/templates/README.md.erb

@@ -0,0 +1 @@
+## New Cure Project

data/lib/cure/cli/templates/gemfile.erb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+source "https://rubygems.org"
+
+gem "cure", "~> <%= version %>"