chronicle-etl 0.4.1 → 0.4.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8a267de435b41b579e36128b7392729ef499eb37f05fabaead7811f089938ddb
-  data.tar.gz: d4af2f62f3f5de926bdfbb0e3d6dbe2c952ec286c07317af4dca8d98f665d6da
+  metadata.gz: f041e90fc6019ecbc2424e8e75e7f21fa5ba715c2cdbde73d61c298e9ca82e07
+  data.tar.gz: 2feda7669f8fefc7ad80fe7918530a86ad2ced431f75e70ad42653c871b67d90
 SHA512:
-  metadata.gz: c78080cce008340f0b2795be46da2b5eb6562b2bffd97728150960343870f2bea4699e4efa07905710dd0e2eba7aaa1e803d8c0f727196f5d9d655b28a04f02e
-  data.tar.gz: cae3a3ffb6527f5c0b3ff89c75dc98d9cd66157ee6230c9db797f4683f90e2146daadf291108e55d3090d0120d3c9e25135cb21c4e9078bcaf4d1edf2172c930
+  metadata.gz: 4a40c72dcb6514037c6e53214dc0af3bfba20c272b959c3e83496658b4b8dc3f841d7399aa215a5fcc5c5cd494278223666b8b881f645ed2c61b667351ccde94
+  data.tar.gz: 5b5450b76a8c03d7cb8405888b2e059390c1585a66524dd7403b458c828a1936422e03a9654ec6d270b634bf6bfe17d6dafda54e05aca8a8732207366d03ffd2

data/.rubocop.yml

@@ -27,6 +27,9 @@ Style/OpenStructUse:
 Style/Copyright:
   Enabled: false
 
+Style/MissingElse:
+  Enabled: false
+
 Style/SymbolArray:
   EnforcedStyle: brackets
 

data/README.md

@@ -1,12 +1,14 @@
 ## A CLI toolkit for extracting and working with your digital history
 
+![chronicle-etl-banner](https://user-images.githubusercontent.com/6291/157330518-0f934c9a-9ec4-43d9-9cc2-12f156d09b37.png)
+
 [![Gem Version](https://badge.fury.io/rb/chronicle-etl.svg)](https://badge.fury.io/rb/chronicle-etl) [![Ruby](https://github.com/chronicle-app/chronicle-etl/actions/workflows/ruby.yml/badge.svg)](https://github.com/chronicle-app/chronicle-etl/actions/workflows/ruby.yml)
 
 Are you trying to archive your digital history or incorporate it into your own projects? You’ve probably discovered how frustrating it is to get machine-readable access to your own data. While [building a memex](https://hyfen.net/memex/), I learned first-hand what great efforts must be made before you can begin using the data in interesting ways.
 
 If you don’t want to spend all your time writing scrapers, reverse-engineering APIs, or parsing takeout data, this project is for you! (*If you do enjoy these things, please see the [open issues](https://github.com/chronicle-app/chronicle-etl/issues).*)
 
-`chronicle-etl` is a CLI tool that gives you the ability to easily access your personal data. It uses the ETL pattern to **extract** it from a source (e.g. your local browser history, a directory of images, goodreads.com reading history), **transform** it (into a given schema), and **load** it to a source (e.g. a CSV file, JSON, external API).
+**`chronicle-etl` is a CLI tool that gives you a unified interface for accessing your personal data.** It uses the ETL pattern to *extract* it from a source (e.g. your local browser history, a directory of images, goodreads.com reading history), *transform* it (into a given schema), and *load* it to a source (e.g. a CSV file, JSON, external API).
 
 ## What does `chronicle-etl` give you?
 * **CLI tool for working with personal data**. You can monitor progress of exports, manipulate the output, set up recurring jobs, manage credentials, and more.
@@ -86,7 +88,16 @@ Plugins provide access to data from third-party platforms, services, or formats.
 
 ```bash
 # Install a plugin
-$ chronicle-etl connectors:install NAME
+$ chronicle-etl plugins:install NAME
+
+# Install the imessage plugin
+$ chronicle-etl plugins:install imessage
+
+# List installed plugins
+$ chronicle-etl plugins:list
+
+# Uninstall a plugin
+$ chronicle-etl plugins:uninstall NAME
 ```
 
 A few dozen importers exist [in my Memex project](https://hyfen.net/memex/) and they’re being ported over to the Chronicle system. This table shows what’s available now and what’s coming. Rows are sorted in very rough order of priority.
@@ -99,8 +110,8 @@ If you want to work together on a connector, please [get in touch](#get-in-touch
 | [shell](https://github.com/chronicle-app/chronicle-shell)       | Shell command history                                                                       | Available (zsh support pending)  |
 | [email](https://github.com/chronicle-app/chronicle-email)       | Emails and attachments from IMAP or .mbox files                                             | Available (imap support pending) |
 | [pinboard](https://github.com/chronicle-app/chronicle-email)    | Bookmarks and tags                                                                          | Available                        |
+| [safari](https://github.com/chronicle-app/chronicle-safari)     | Browser history from local sqlite db                                                        | Available                        |
 | github                                                          | Github user and repo activity                                                               | In progress                      |
-| safari                                                          | Browser history from local sqlite db                                                        | Needs porting                    |
 | chrome                                                          | Browser history from local sqlite db                                                        | Needs porting                    |
 | whatsapp                                                        | Messaging history (via individual chat exports) or reverse-engineered local desktop install | Unstarted                        |
 | anki                                                            | Studying and card creation history                                                          | Needs porting                    |
@@ -186,4 +197,4 @@ Bug reports and pull requests are welcome on GitHub at https://github.com/chroni
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
 
 ## Code of Conduct
-Everyone interacting in the Chronicle::ETL project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/chronicle-app/chronicle-etl/blob/master/CODE_OF_CONDUCT.md).
+Everyone interacting in the Chronicle::ETL project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/chronicle-app/chronicle-etl/blob/main/CODE_OF_CONDUCT.md).

data/chronicle-etl.gemspec

@@ -47,8 +47,11 @@ Gem::Specification.new do |spec|
   spec.add_dependency "sequel", "~> 5.35"
   spec.add_dependency "sqlite3", "~> 1.4"
   spec.add_dependency "thor", "~> 1.2"
+  spec.add_dependency "thor-hollaback", "~> 0.2"
   spec.add_dependency "tty-progressbar", "~> 0.17"
+  spec.add_dependency "tty-spinner"
   spec.add_dependency "tty-table", "~> 0.11"
+  spec.add_dependency "tty-prompt", "~> 0.23"
 
   spec.add_development_dependency "bundler", "~> 2.1"
   spec.add_development_dependency "pry-byebug", "~> 3.9"

data/lib/chronicle/etl/cli/connectors.rb

@@ -8,11 +8,6 @@ module Chronicle
         default_task 'list'
         namespace :connectors
 
-        desc "install NAME", "Installs connector NAME"
-        def install(name)
-          Chronicle::ETL::Registry.install_connector(name)
-        end
-
         desc "list", "Lists available connectors"
         # Display all available connectors that chronicle-etl has access to
         def list
@@ -44,21 +39,21 @@ module Chronicle
         desc "show PHASE IDENTIFIER", "Show information about a connector"
         def show(phase, identifier)
           unless ['extractor', 'transformer', 'loader'].include?(phase)
-            puts "phase argument must be one of: [extractor, transformer, loader]"
-            return
+            Chronicle::ETL::Logger.fatal("Phase argument must be one of: [extractor, transformer, loader]")
+            exit 1
           end
 
           begin
             connector = Chronicle::ETL::Registry.find_by_phase_and_identifier(phase.to_sym, identifier)
-          rescue Chronicle::ETL::ConnectorNotAvailableError
-            puts "Could not find #{phase} #{identifier}"
-            return
+          rescue Chronicle::ETL::ConnectorNotAvailableError, Chronicle::ETL::PluginError
+            Chronicle::ETL::Logger.fatal("Could not find #{phase} #{identifier}")
+            exit 1
           end
 
           puts connector.klass.to_s.bold
           puts "  #{connector.descriptive_phrase}"
           puts
-          puts "OPTIONS"
+          puts "Settings:"
 
           headers = ['name', 'default', 'required'].map{ |h| h.to_s.upcase.bold }
 

data/lib/chronicle/etl/cli/jobs.rb

@@ -1,4 +1,5 @@
 require 'pp'
+require 'tty-prompt'
 
 module Chronicle
   module ETL
@@ -6,7 +7,7 @@ module Chronicle
       # CLI commands for working with ETL jobs
       class Jobs < SubcommandBase
         default_task "start"
-        namespace :jobs 
+        namespace :jobs
 
         class_option :name, aliases: '-j', desc: 'Job configuration name'
 
@@ -25,16 +26,11 @@ module Chronicle
 
         class_option :output, aliases: '-o', desc: 'Output filename', type: 'string'
         class_option :fields, desc: 'Output only these fields', type: 'array', banner: 'field1 field2 ...'
-
-        class_option :log_level, desc: 'Log level (debug, info, warn, error, fatal)', default: 'info'
-        class_option :verbose, aliases: '-v', desc: 'Set log level to verbose', type: :boolean
-        class_option :silent, desc: 'Silence all output', type: :boolean
+        class_option :header_row, desc: 'Output the header row of tabular output', type: 'boolean'
 
         # Thor doesn't like `run` as a command name
         map run: :start
         desc "run", "Start a job"
-        option :log_level, desc: 'Log level (debug, info, warn, error, fatal)', default: 'info'
-        option :verbose, aliases: '-v', desc: 'Set log level to verbose', type: :boolean
         option :dry_run, desc: 'Only run the extraction and transform steps, not the loading', type: :boolean
         long_desc <<-LONG_DESC
           This will run an ETL job. Each job needs three parts:
@@ -49,25 +45,40 @@ module Chronicle
 LONG_DESC
         # Run an ETL job
         def start
-          setup_log_level
-          job_definition = build_job_definition(options)
-          job = Chronicle::ETL::Job.new(job_definition)
-          runner = Chronicle::ETL::Runner.new(job)
-          runner.run!
+          run_job(options)
+        rescue Chronicle::ETL::JobDefinitionError => e
+          missing_plugins = e.job_definition.errors
+            .select { |error| error.is_a?(Chronicle::ETL::PluginLoadError) }
+            .map(&:name)
+            .uniq
+
+          install_missing_plugins(missing_plugins)
+          run_job(options)
         end
 
         desc "create", "Create a job"
         # Create an ETL job
         def create
           job_definition = build_job_definition(options)
+          job_definition.validate!
+
           path = File.join('chronicle', 'etl', 'jobs', options[:name])
           Chronicle::ETL::Config.write(path, job_definition.definition)
+        rescue Chronicle::ETL::JobDefinitionError => e
+          Chronicle::ETL::Logger.debug(e.full_message)
+          Chronicle::ETL::Logger.fatal("Job definition error".red)
         end
 
         desc "show", "Show details about a job"
         # Show an ETL job
         def show
-          puts Chronicle::ETL::Job.new(build_job_definition(options))
+          job_definition = build_job_definition(options)
+          job_definition.validate!
+          puts Chronicle::ETL::Job.new(job_definition)
+        rescue Chronicle::ETL::JobDefinitionError => e
+          Chronicle::ETL::Logger.debug(e.full_message)
+          Chronicle::ETL::Logger.fatal("Job definition error".red)
+          exit 1
         end
 
         desc "list", "List all available jobs"
@@ -87,21 +98,43 @@ LONG_DESC
 
           headers = ['name', 'extractor', 'transformer', 'loader'].map { |h| h.upcase.bold }
 
+          puts "Available jobs:"
           table = TTY::Table.new(headers, job_details)
           puts table.render(indent: 0, padding: [0, 2])
+        rescue Chronicle::ETL::ConfigError => e
+          Chronicle::ETL::Logger.debug(e.full_message)
+          Chronicle::ETL::Logger.fatal("Error reading config. #{e.message}".red)
+          exit 1
         end
 
         private
 
-        def setup_log_level
-          if options[:silent]
-            Chronicle::ETL::Logger.log_level = Chronicle::ETL::Logger::SILENT
-          elsif options[:verbose]
-            Chronicle::ETL::Logger.log_level = Chronicle::ETL::Logger::DEBUG
-          elsif options[:log_level]
-            level = Chronicle::ETL::Logger.const_get(options[:log_level].upcase)
-            Chronicle::ETL::Logger.log_level = level 
+        def run_job(options)
+          job_definition = build_job_definition(options)
+          job = Chronicle::ETL::Job.new(job_definition)
+          runner = Chronicle::ETL::Runner.new(job)
+          runner.run!
+        end
+
+        # TODO: probably could merge this with something in cli/plugin
+        def install_missing_plugins(missing_plugins)
+          prompt = TTY::Prompt.new
+          message = "Plugin#{'s' if missing_plugins.count > 1} specified by job not installed.\n"
+          message += "Do you want to install "
+          message += missing_plugins.map { |name| "chronicle-#{name}".bold}.join(", ")
+          message += " and start the job?"
+          install = prompt.yes?(message)
+          return unless install
+
+          spinner = TTY::Spinner.new("[:spinner] Installing plugins...", format: :dots_2)
+          spinner.auto_spin
+          missing_plugins.each do |plugin|
+            Chronicle::ETL::Registry::PluginRegistry.install(plugin)
           end
+          spinner.success("(#{'successful'.green})")
+        rescue Chronicle::ETL::PluginNotAvailableError => e
+          spinner.error("Error".red)
+          Chronicle::ETL::Logger.fatal("Plugin '#{e.name}' could not be installed".red)
         end
 
         # Create job definition by reading config file and then overwriting with flag options
@@ -129,6 +162,7 @@ LONG_DESC
 
           loader_options = options[:'loader-opts'].merge({
             output: options[:output],
+            header_row: options[:header_row],
             fields: options[:fields]
           }.compact)
 

data/lib/chronicle/etl/cli/main.rb

@@ -5,6 +5,14 @@ module Chronicle
     module CLI
       # Main entrypoint for CLI app
       class Main < ::Thor
+        class_before :set_log_level
+        class_before :set_color_output
+
+        class_option :log_level, desc: 'Log level (debug, info, warn, error, fatal, silent)', default: 'info'
+        class_option :verbose, aliases: '-v', desc: 'Set log level to verbose', type: :boolean
+        class_option :silent, desc: 'Silence all output', type: :boolean
+        class_option :'no-color', desc: 'Disable colour output', type: :boolean
+
         default_task "jobs"
 
         desc 'connectors:COMMAND', 'Connectors available for ETL jobs', hide: true
@@ -13,6 +21,9 @@ module Chronicle
         desc 'jobs:COMMAND', 'Configure and run jobs', hide: true
         subcommand 'jobs', Jobs
 
+        desc 'plugins:COMMAND', 'Configure plugins', hide: true
+        subcommand 'plugins', Plugins
+
         # Entrypoint for the CLI
         def self.start(given_args = ARGV, config = {})
           # take a subcommand:command and splits them so Thor knows how to hand off to the subcommand class
@@ -79,6 +90,23 @@ module Chronicle
             shell.say
           end
         end
+
+        no_commands do
+          def set_color_output
+            String.disable_colorization true if options[:'no-color'] || ENV['NO_COLOR']
+          end
+
+          def set_log_level
+            if options[:silent]
+              Chronicle::ETL::Logger.log_level = Chronicle::ETL::Logger::SILENT
+            elsif options[:verbose]
+              Chronicle::ETL::Logger.log_level = Chronicle::ETL::Logger::DEBUG
+            elsif options[:log_level]
+              level = Chronicle::ETL::Logger.const_get(options[:log_level].upcase)
+              Chronicle::ETL::Logger.log_level = level
+            end
+          end
+        end
       end
     end
   end

data/lib/chronicle/etl/cli/plugins.rb

@@ -0,0 +1,62 @@
+# frozen_string_literal: true
+
+require "tty-prompt"
+require "tty-spinner"
+
+
+module Chronicle
+  module ETL
+    module CLI
+      # CLI commands for working with ETL plugins
+      class Plugins < SubcommandBase
+        default_task 'list'
+        namespace :plugins
+
+        desc "install", "Install a plugin"
+        def install(name)
+          spinner = TTY::Spinner.new("[:spinner] Installing plugin #{name}...", format: :dots_2)
+          spinner.auto_spin
+          Chronicle::ETL::Registry::PluginRegistry.install(name)
+          spinner.success("(#{'successful'.green})")
+        rescue Chronicle::ETL::PluginError => e
+          spinner.error("Error".red)
+          Chronicle::ETL::Logger.debug(e.full_message)
+          Chronicle::ETL::Logger.fatal("Plugin '#{name}' could not be installed".red)
+          exit 1
+        end
+
+        desc "uninstall", "Unintall a plugin"
+        def uninstall(name)
+          spinner = TTY::Spinner.new("[:spinner] Uninstalling plugin #{name}...", format: :dots_2)
+          spinner.auto_spin
+          Chronicle::ETL::Registry::PluginRegistry.uninstall(name)
+          spinner.success("(#{'successful'.green})")
+        rescue Chronicle::ETL::PluginError => e
+          spinner.error("Error".red)
+          Chronicle::ETL::Logger.debug(e.full_message)
+          Chronicle::ETL::Logger.fatal("Plugin '#{name}' could not be uninstalled (was it installed?)".red)
+          exit 1
+        end
+
+        desc "list", "Lists available plugins"
+        # Display all available plugins that chronicle-etl has access to
+        def list
+          plugins = Chronicle::ETL::Registry::PluginRegistry.all_installed_latest
+
+          info = plugins.map do |plugin|
+            {
+              name: plugin.name.sub("chronicle-", ""),
+              description: plugin.description,
+              version: plugin.version
+            }
+          end
+
+          headers = ['name', 'description', 'latest version'].map{ |h| h.to_s.upcase.bold }
+          table = TTY::Table.new(headers, info.map(&:values))
+          puts "Installed plugins:"
+          puts table.render(indent: 2, padding: [0, 0])
+        end
+      end
+    end
+  end
+end

data/lib/chronicle/etl/cli.rb

@@ -1,7 +1,9 @@
 require 'thor'
+require 'thor/hollaback'
 require 'chronicle/etl'
 
 require 'chronicle/etl/cli/subcommand_base'
 require 'chronicle/etl/cli/connectors'
 require 'chronicle/etl/cli/jobs'
+require 'chronicle/etl/cli/plugins'
 require 'chronicle/etl/cli/main'

data/lib/chronicle/etl/config.rb

@@ -24,16 +24,14 @@ module Chronicle
 
       # Returns all jobs available in ~/.config/chronicle/etl/jobs/*.yml
       def available_jobs
-        job_directory = Runcom::Config.new('chronicle/etl/jobs').current
-        Dir.glob(File.join(job_directory, "*.yml")).map do |filename|
+        Dir.glob(File.join(config_directory("jobs"), "*.yml")).map do |filename|
           File.basename(filename, ".*")
         end
       end
 
       # Returns all available credentials available in ~/.config/chronicle/etl/credentials/*.yml
       def available_credentials
-        job_directory = Runcom::Config.new('chronicle/etl/credentials').current
-        Dir.glob(File.join(job_directory, "*.yml")).map do |filename|
+        Dir.glob(File.join(config_directory("credentials"), "*.yml")).map do |filename|
           File.basename(filename, ".*")
         end
       end
@@ -48,6 +46,11 @@ module Chronicle
       def load_credentials(name)
         config = self.load("chronicle/etl/credentials/#{name}.yml")
       end
+
+      def config_directory(type)
+        path = "chronicle/etl/#{type}"
+        Runcom::Config.new(path).current || raise(Chronicle::ETL::ConfigError, "Could not access config directory (#{path})")
+      end
     end
   end
 end

data/lib/chronicle/etl/configurable.rb

@@ -57,7 +57,7 @@ module Chronicle
 
           options.each do |name, value|
             setting = self.class.all_settings[name]
-            raise(Chronicle::ETL::ConfigurationError, "Unrecognized setting: #{name}") unless setting
+            raise(Chronicle::ETL::ConnectorConfigurationError, "Unrecognized setting: #{name}") unless setting
 
             @config[name] = coerced_value(setting, value)
           end
@@ -78,7 +78,7 @@ module Chronicle
 
         def validate_config
           missing = (self.class.all_required_settings.keys - @config.compacted_h.keys)
-          raise Chronicle::ETL::ConfigurationError, "Missing options: #{missing}" if missing.count.positive?
+          raise Chronicle::ETL::ConnectorConfigurationError, "Missing options: #{missing}" if missing.count.positive?
         end
 
         def coerced_value(setting, value)
@@ -89,6 +89,11 @@ module Chronicle
           value.to_s
         end
 
+        # TODO: think about whether to split up float, integer
+        def coerce_numeric(value)
+          value.to_f
+        end
+
         def coerce_boolean(value)
           if value.is_a?(String)
             value.downcase == "true"

data/lib/chronicle/etl/exceptions.rb

@@ -2,10 +2,32 @@ module Chronicle
   module ETL
     class Error < StandardError; end
 
-    class ConfigurationError < Error; end
+    class ConfigError < Error; end
 
     class RunnerTypeError < Error; end
 
+    class JobDefinitionError < Error
+      attr_reader :job_definition
+
+      def initialize(job_definition)
+        @job_definition = job_definition
+        super
+      end
+    end
+
+    class PluginError < Error
+      attr_reader :name
+
+      def initialize(name)
+        @name = name
+      end
+    end
+
+    class PluginNotAvailableError < PluginError; end
+    class PluginLoadError < PluginError; end
+
+    class ConnectorConfigurationError < Error; end
+
     class ConnectorNotAvailableError < Error
       def initialize(message, provider: nil, name: nil)
         super(message)

data/lib/chronicle/etl/extractors/extractor.rb

@@ -9,7 +9,7 @@ module Chronicle
 
       setting :since, type: :time
       setting :until, type: :time
-      setting :limit
+      setting :limit, type: :numeric
       setting :load_after_id
       setting :input
 

data/lib/chronicle/etl/job.rb

@@ -1,6 +1,11 @@
 require 'forwardable'
+
 module Chronicle
   module ETL
+    # A runner job
+    #
+    # TODO: this can probably be merged with JobDefinition. Not clear
+    # where the boundaries are
     class Job
       extend Forwardable
 
@@ -12,7 +17,8 @@ module Chronicle
                     :transformer_klass,
                     :transformer_options,
                     :loader_klass,
-                    :loader_options
+                    :loader_options,
+                    :job_definition
 
       # TODO: build a proper id system
       alias id name

data/lib/chronicle/etl/job_definition.rb

@@ -19,12 +19,31 @@ module Chronicle
         }
       }.freeze
 
+      attr_reader :errors
       attr_accessor :definition
 
       def initialize()
         @definition = SKELETON_DEFINITION
       end
 
+      def validate
+        @errors = []
+
+        Chronicle::ETL::Registry::PHASES.each do |phase|
+          __send__("#{phase}_klass".to_sym)
+        rescue Chronicle::ETL::PluginError => e
+          @errors << e
+        end
+
+        @errors.empty?
+      end
+
+      def validate!
+        raise(Chronicle::ETL::JobDefinitionError.new(self), "Job definition is invalid") unless validate
+
+        true
+      end
+
       # Add config hash to this definition
       def add_config(config = {})
         @definition = @definition.deep_merge(config)
@@ -80,10 +99,6 @@ module Chronicle
           end
         end
       end
-
-      def validate
-        return true   # TODO
-      end
     end
   end
 end

data/lib/chronicle/etl/loaders/csv_loader.rb

@@ -7,22 +7,49 @@ module Chronicle
         r.description = 'CSV'
       end
 
-      def initialize(options={})
-        super(options)
-        @rows = []
+      setting :output, default: $stdout
+      setting :headers, default: true
+      setting :header_row, default: true
+
+      def records
+        @records ||= []
       end
 
       def load(record)
-        @rows << record.to_h_flattened.values
+        records << record.to_h_flattened
       end
 
       def finish
-        z = $stdout
-        CSV(z) do |csv|
-          @rows.each do |row|
-            csv << row
+        return unless records.any?
+
+        headers = build_headers(records)
+
+        csv_options = {}
+        if @config.headers
+          csv_options[:write_headers] = @config.header_row
+          csv_options[:headers] = headers
+        end
+
+        if @config.output.is_a?(IO)
+          # This might seem like a duplication of the default value ($stdout)
+          # but it's because rspec overwrites $stdout (in helper #capture) to
+          # capture output.
+          io = $stdout.dup
+        else
+          io = File.open(@config.output, "w+")
+        end
+
+        output = CSV.generate(**csv_options) do |csv|
+          records.each do |record|
+            csv << record
+              .transform_keys(&:to_sym)
+              .values_at(*headers)
+              .map { |value| force_utf8(value) }
           end
         end
+
+        io.write(output)
+        io.close
       end
     end
   end

data/lib/chronicle/etl/loaders/helpers/encoding_helper.rb

@@ -0,0 +1,18 @@
+require 'pathname'
+
+module Chronicle
+  module ETL
+    module Loaders
+      module Helpers
+        module EncodingHelper
+          # Mostly useful for handling loading with binary data from a raw extraction
+          def force_utf8(value)
+            return value unless value.is_a?(String)
+
+            value.encode('UTF-8', invalid: :replace, undef: :replace, replace: '')
+          end
+        end
+      end
+    end
+  end
+end

data/lib/chronicle/etl/loaders/json_loader.rb

@@ -25,7 +25,7 @@ module Chronicle
         encoded = serialized.transform_values do |value|
           next value unless value.is_a?(String)
 
-          value.encode('UTF-8', invalid: :replace, undef: :replace, replace: '?')
+          force_utf8(value)
         end
         @output.puts encoded.to_json
       end

data/lib/chronicle/etl/loaders/loader.rb

@@ -1,11 +1,17 @@
+require_relative 'helpers/encoding_helper'
+
 module Chronicle
   module ETL
     # Abstract class representing a Loader for an ETL job
     class Loader
       extend Chronicle::ETL::Registry::SelfRegistering
       include Chronicle::ETL::Configurable
+      include Chronicle::ETL::Loaders::Helpers::EncodingHelper
 
       setting :output
+      setting :fields
+      setting :fields_limit, default: nil
+      setting :fields_exclude
 
       # Construct a new instance of this loader. Options are passed in from a Runner
       # == Parameters:
@@ -25,6 +31,23 @@ module Chronicle
 
       # Called once there are no more records to process
       def finish; end
+
+      private
+
+      def build_headers(records)
+        headers =
+          if @config.fields && @config.fields.any?
+            Set[*@config.fields]
+          else
+            # use all the keys of the flattened record hash
+            Set[*records.map(&:keys).flatten.map(&:to_s).uniq]
+          end
+
+        headers = headers.delete_if { |header| header.end_with?(*@config.fields_exclude) }
+        headers = headers.first(@config.fields_limit) if @config.fields_limit
+
+        headers.to_a.map(&:to_sym)
+      end
     end
   end
 end

data/lib/chronicle/etl/loaders/table_loader.rb

@@ -9,11 +9,10 @@ module Chronicle
         r.description = 'an ASCII table'
       end
 
-      setting :fields_limit, default: nil
-      setting :fields_exclude, default: ['lids', 'type']
-      setting :fields, default: []
       setting :truncate_values_at, default: 40
       setting :table_renderer, default: :basic
+      setting :fields_exclude, default: ['lids', 'type']
+      setting :header_row, default: true
 
       def load(record)
         records << record.to_h_flattened
@@ -25,7 +24,7 @@ module Chronicle
         headers = build_headers(records)
         rows = build_rows(records, headers)
 
-        @table = TTY::Table.new(header: headers, rows: rows)
+        @table = TTY::Table.new(header: (headers if @config.header_row), rows: rows)
         puts @table.render(
           @config.table_renderer.to_sym,
           padding: [0, 2, 0, 0]
@@ -38,25 +37,10 @@ module Chronicle
 
       private
 
-      def build_headers(records)
-        headers =
-          if @config.fields.any?
-            Set[*@config.fields]
-          else
-            # use all the keys of the flattened record hash
-            Set[*records.map(&:keys).flatten.map(&:to_s).uniq]
-          end
-
-        headers = headers.delete_if { |header| header.end_with?(*@config.fields_exclude) } if @config.fields_exclude.any?
-        headers = headers.first(@config.fields_limit) if @config.fields_limit
-
-        headers.to_a.map(&:to_sym)
-      end
-
       def build_rows(records, headers)
         records.map do |record|
           values = record.transform_keys(&:to_sym).values_at(*headers).map{|value| value.to_s }
-
+          values = values.map { |value| force_utf8(value) }
           if @config.truncate_values_at
             values = values.map{ |value| value.truncate(@config.truncate_values_at) }
           end

data/lib/chronicle/etl/logger.rb

@@ -13,7 +13,6 @@ module Chronicle
       attr_accessor :log_level
 
       @log_level = INFO
-      @destination = $stderr
 
       def output message, level
         return unless level >= @log_level
@@ -21,10 +20,14 @@ module Chronicle
         if @progress_bar
           @progress_bar.log(message)
         else
-          @destination.puts(message)
+          $stderr.puts(message)
         end
       end
 
+      def fatal(message)
+        output(message, FATAL)
+      end
+
       def error(message)
         output(message, ERROR)
       end

data/lib/chronicle/etl/registry/connector_registration.rb

@@ -44,6 +44,11 @@ module Chronicle
           @provider || (built_in? ? 'chronicle' : '')
         end
 
+        # TODO: allow overriding here. Maybe through self-registration process
+        def plugin
+          @provider
+        end
+
         def descriptive_phrase
           prefix = case phase
           when :extractor

data/lib/chronicle/etl/registry/plugin_registry.rb

@@ -0,0 +1,70 @@
+require 'rubygems'
+require 'rubygems/command'
+require 'rubygems/commands/install_command'
+require 'rubygems/uninstaller'
+
+module Chronicle
+  module ETL
+    module Registry
+      # Responsible for managing plugins available to chronicle-etl
+      #
+      # @todo Better validation for whether a gem is actually a plugin
+      # @todo Add ways to load a plugin that don't require a gem on rubygems.org
+      module PluginRegistry
+        # Does this plugin exist?
+        def self.exists?(name)
+          # TODO: implement this. Could query rubygems.org or have a
+          # hardcoded approved list
+          true
+        end
+
+        # All versions of all plugins currently installed
+        def self.all_installed
+          # TODO: add check for chronicle-etl dependency
+          Gem::Specification.filter { |s| s.name.match(/^chronicle-/) && s.name != "chronicle-etl" }
+        end
+
+        # Latest version of each installed plugin
+        def self.all_installed_latest
+          all_installed.group_by(&:name)
+            .transform_values { |versions| versions.sort_by(&:version).reverse.first }
+            .values
+        end
+
+        # Activate a plugin with given name by `require`ing it
+        def self.activate(name)
+          # By default, activates the latest available version of a gem
+          # so don't have to run Kernel#gem separately
+          require "chronicle/#{name}"
+        rescue LoadError
+          raise Chronicle::ETL::PluginLoadError.new(name), "Plugin #{name} couldn't be loaded" if exists?(name)
+
+          raise Chronicle::ETL::PluginNotAvailableError.new(name), "Plugin #{name} doesn't exist"
+        end
+
+        # Install a plugin to local gems
+        def self.install(name)
+          gem_name = "chronicle-#{name}"
+          raise(Chronicle::ETL::PluginNotAvailableError.new(gem_name), "Plugin #{name} doesn't exist") unless exists?(gem_name)
+
+          Gem::DefaultUserInteraction.ui = Gem::SilentUI.new
+          Gem.install(gem_name)
+        rescue Gem::UnsatisfiableDependencyError
+          # TODO: we need to catch a lot more than this here
+          raise Chronicle::ETL::PluginNotAvailableError.new(name), "Plugin #{name} doesn't exist"
+        end
+
+        # Uninstall a plugin
+        def self.uninstall(name)
+          gem_name = "chronicle-#{name}"
+          Gem::DefaultUserInteraction.ui = Gem::SilentUI.new
+          uninstaller = Gem::Uninstaller.new(gem_name)
+          uninstaller.uninstall
+        rescue Gem::InstallError
+          # TODO: strengthen this exception handling
+          raise(Chronicle::ETL::PluginError.new(name), "Plugin #{name} wasn't uninstalled")
+        end
+      end
+    end
+  end
+end

data/lib/chronicle/etl/registry/registry.rb

@@ -20,28 +20,40 @@ module Chronicle
           end
         end
 
-        def install_connector name
-          gem_name = "chronicle-#{name}"
-          Gem.install(gem_name)
+        def register connector
+          connectors << connector
         end
 
-        def register connector
+        def connectors
           @connectors ||= []
-          @connectors << connector
         end
 
         def find_by_phase_and_identifier(phase, identifier)
-          connector = find_within_loaded_connectors(phase, identifier)
-          unless connector
-            # Only load external connectors (slow) if not found in built-in connectors
-            load_all!
-            connector = find_within_loaded_connectors(phase, identifier)
+          # Simple case: built in connector
+          connector = connectors.find { |c| c.phase == phase && c.identifier == identifier }
+          return connector if connector
+
+          # if not available in built-in connectors, try to activate a
+          # relevant plugin and try again
+          if identifier.include?(":")
+            plugin, name = identifier.split(":")
+          else
+            # This case handles the case where the identifier is a 
+            # shorthand (ie `imessage`) because there's only one default
+            # connector.
+            plugin = identifier
           end
-          connector || raise(ConnectorNotAvailableError.new("Connector '#{identifier}' not found"))
-        end
 
-        def find_within_loaded_connectors(phase, identifier)
-          @connectors.find { |c| c.phase == phase && c.identifier == identifier }
+          PluginRegistry.activate(plugin)
+
+          candidates = connectors.select { |c| c.phase == phase && c.plugin == plugin }
+          # if no name given, just use first connector with right phase/plugin
+          # TODO: set up a property for connectors to specify that they're the
+          # default connector for the plugin
+          candidates = candidates.select { |c| c.identifier == name } if name
+          connector = candidates.first
+
+          connector || raise(ConnectorNotAvailableError, "Connector '#{identifier}' not found")
         end
       end
     end
@@ -50,3 +62,4 @@ end
 
 require_relative 'self_registering'
 require_relative 'connector_registration'
+require_relative 'plugin_registry'

data/lib/chronicle/etl/runner.rb

@@ -8,19 +8,41 @@ class Chronicle::ETL::Runner
   end
 
   def run!
-    extractor = @job.instantiate_extractor
-    loader = @job.instantiate_loader
+    validate_job
+    instantiate_connectors
+    prepare_job
+    prepare_ui
+    run_extraction
+    finish_job
+  end
+
+  private
+
+  def validate_job
+    @job.job_definition.validate!
+  end
+
+  def instantiate_connectors
+    @extractor = @job.instantiate_extractor
+    @loader = @job.instantiate_loader
+  end
 
+  def prepare_job
+    Chronicle::ETL::Logger.info(tty_log_job_start)
     @job_logger.start
-    loader.start
+    @loader.start
+    @extractor.prepare
+  end
 
-    extractor.prepare
-    total = extractor.results_count
+  def prepare_ui
+    total = @extractor.results_count
     @progress_bar = Chronicle::ETL::Utils::ProgressBar.new(title: 'Running job', total: total)
     Chronicle::ETL::Logger.attach_to_progress_bar(@progress_bar)
+  end
 
-    Chronicle::ETL::Logger.info(tty_log_job_start)
-    extractor.extract do |extraction|
+  # TODO: refactor this further
+  def run_extraction
+    @extractor.extract do |extraction|
       unless extraction.is_a?(Chronicle::ETL::Extraction)
         raise Chronicle::ETL::RunnerTypeError, "Extracted should be a Chronicle::ETL::Extraction"
       end
@@ -28,15 +50,10 @@ class Chronicle::ETL::Runner
       transformer = @job.instantiate_transformer(extraction)
       record = transformer.transform
 
-      # TODO: rethink this
-      # unless record.is_a?(Chronicle::ETL::Models)
-      #   raise Chronicle::ETL::RunnerTypeError, "Transformed data should be a type of Chronicle::ETL::Models"
-      # end
-
       Chronicle::ETL::Logger.info(tty_log_transformation(transformer))
       @job_logger.log_transformation(transformer)
 
-      loader.load(record) unless @job.dry_run?
+      @loader.load(record) unless @job.dry_run?
     rescue Chronicle::ETL::TransformationError => e
       Chronicle::ETL::Logger.error(tty_log_transformation_failure(e))
     ensure
@@ -44,22 +61,22 @@ class Chronicle::ETL::Runner
     end
 
     @progress_bar.finish
-    loader.finish
+    @loader.finish
     @job_logger.finish
   rescue Interrupt
     Chronicle::ETL::Logger.error("\n#{'Job interrupted'.red}")
     @job_logger.error
   rescue StandardError => e
     raise e
-  ensure
+  end
+
+  def finish_job
     @job_logger.save
     @progress_bar&.finish
     Chronicle::ETL::Logger.detach_from_progress_bar
     Chronicle::ETL::Logger.info(tty_log_completion)
   end
 
-  private
-
   def tty_log_job_start
     output = "Beginning job "
     output += "'#{@job.name}'".bold if @job.name

data/lib/chronicle/etl/version.rb

@@ -1,5 +1,5 @@
 module Chronicle
   module ETL
-    VERSION = "0.4.1"
+    VERSION = "0.4.2"
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: chronicle-etl
 version: !ruby/object:Gem::Version
-  version: 0.4.1
+  version: 0.4.2
 platform: ruby
 authors:
 - Andrew Louis
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2022-03-05 00:00:00.000000000 Z
+date: 2022-03-12 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -150,6 +150,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '1.2'
+- !ruby/object:Gem::Dependency
+  name: thor-hollaback
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.2'
 - !ruby/object:Gem::Dependency
   name: tty-progressbar
   requirement: !ruby/object:Gem::Requirement
@@ -164,6 +178,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '0.17'
+- !ruby/object:Gem::Dependency
+  name: tty-spinner
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: tty-table
   requirement: !ruby/object:Gem::Requirement
@@ -178,6 +206,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '0.11'
+- !ruby/object:Gem::Dependency
+  name: tty-prompt
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.23'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.23'
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
@@ -320,6 +362,7 @@ files:
 - lib/chronicle/etl/cli/connectors.rb
 - lib/chronicle/etl/cli/jobs.rb
 - lib/chronicle/etl/cli/main.rb
+- lib/chronicle/etl/cli/plugins.rb
 - lib/chronicle/etl/cli/subcommand_base.rb
 - lib/chronicle/etl/config.rb
 - lib/chronicle/etl/configurable.rb
@@ -336,6 +379,7 @@ files:
 - lib/chronicle/etl/job_log.rb
 - lib/chronicle/etl/job_logger.rb
 - lib/chronicle/etl/loaders/csv_loader.rb
+- lib/chronicle/etl/loaders/helpers/encoding_helper.rb
 - lib/chronicle/etl/loaders/json_loader.rb
 - lib/chronicle/etl/loaders/loader.rb
 - lib/chronicle/etl/loaders/rest_loader.rb
@@ -347,6 +391,7 @@ files:
 - lib/chronicle/etl/models/entity.rb
 - lib/chronicle/etl/models/raw.rb
 - lib/chronicle/etl/registry/connector_registration.rb
+- lib/chronicle/etl/registry/plugin_registry.rb
 - lib/chronicle/etl/registry/registry.rb
 - lib/chronicle/etl/registry/self_registering.rb
 - lib/chronicle/etl/runner.rb
@@ -384,7 +429,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.1.6
+rubygems_version: 3.3.9
 signing_key: 
 specification_version: 4
 summary: ETL tool for personal data