chronicle-etl 0.4.0 → 0.4.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5fd411a9a41a645b85780230c79b09f361e121d0e8ca7f3270ca8eba55a76ca8
-  data.tar.gz: c09053715910ab4f027fbdc3a5b7d10c042eee962f7fa93c6571ce8359f51009
+  metadata.gz: f2b6fdca3723ec52287c070a0dd08d0cfaf825f5e8f46da0d5a34172c0008573
+  data.tar.gz: e15181ba7edc1698404af8ff8c05d5367786ea809360393e825ca5ee5eef6c75
 SHA512:
-  metadata.gz: 2c9ec14b6c0a51f1c5ec77ee8d9a7f016d16bdc35db5634f9fa5d38aabc30dec201cd4b8bef06a31b86773a0c1cda2d271d7008dcb247a86d956c094919f3c0f
-  data.tar.gz: 0dca41e1654e5b2b98a148f853492a67126cdac767000b3c5f97c5c8ff88b77464e17a2fab38b72c1f014f3515c911e5f3f391eaf68d64e73dcfcff5d8e6cb6a
+  metadata.gz: c5508dcfc5e1367122ebbc191dc60a76cc7b1f088ae9ebd52bae41e07420f54ea7977a35f2b25839933b06d4673e2a120feecbd958efee07e0d313eaa7a5d167
+  data.tar.gz: adcb90549af364189c5ae3b0811c039277aba7dc6fbf2fbd6de8b89c572948d0dc00ba8da59db03252f4770737423e62c3a1175ef018742ef8bd7aee14837f63

data/.github/workflows/ruby.yml

@@ -9,9 +9,9 @@ name: Ruby
 
 on:
   push:
-    branches: [ master ]
+    branches: [ main ]
   pull_request:
-    branches: [ master ]
+    branches: [ main ]
 
 jobs:
   test:

data/.rubocop.yml

@@ -27,6 +27,9 @@ Style/OpenStructUse:
 Style/Copyright:
   Enabled: false
 
+Style/MissingElse:
+  Enabled: false
+
 Style/SymbolArray:
   EnforcedStyle: brackets
 

data/README.md

@@ -1,125 +1,200 @@
-# Chronicle::ETL
+## A CLI toolkit for extracting and working with your digital history
+
+![chronicle-etl-banner](https://user-images.githubusercontent.com/6291/157330518-0f934c9a-9ec4-43d9-9cc2-12f156d09b37.png)
 
 [![Gem Version](https://badge.fury.io/rb/chronicle-etl.svg)](https://badge.fury.io/rb/chronicle-etl) [![Ruby](https://github.com/chronicle-app/chronicle-etl/actions/workflows/ruby.yml/badge.svg)](https://github.com/chronicle-app/chronicle-etl/actions/workflows/ruby.yml)
 
-Chronicle ETL is a utility that helps you archive and processes personal data. You can *extract* it from a variety of sources, *transform* it, and *load* it to an external API, file, or stdout.
+Are you trying to archive your digital history or incorporate it into your own projects? You’ve probably discovered how frustrating it is to get machine-readable access to your own data. While [building a memex](https://hyfen.net/memex/), I learned first-hand what great efforts must be made before you can begin using the data in interesting ways.
 
-This tool is an adaptation of Andrew Louis's experimental [Memex project](https://hyfen.net/memex) and the dozens of existing importers are being migrated to Chronicle.
+If you don’t want to spend all your time writing scrapers, reverse-engineering APIs, or parsing takeout data, this project is for you! (*If you do enjoy these things, please see the [open issues](https://github.com/chronicle-app/chronicle-etl/issues).*)
 
-## Installation
+**`chronicle-etl` is a CLI tool that gives you a unified interface for accessing your personal data.** It uses the ETL pattern to *extract* it from a source (e.g. your local browser history, a directory of images, goodreads.com reading history), *transform* it (into a given schema), and *load* it to a source (e.g. a CSV file, JSON, external API).
 
-```bash
-$ gem install chronicle-etl
+## What does `chronicle-etl` give you?
+* **CLI tool for working with personal data**. You can monitor progress of exports, manipulate the output, set up recurring jobs, manage credentials, and more.
+* **Plugins for many third-party providers**. A plugin system allows you to access data from third-party providers and hook it into the shared CLI infrastructure.
+* **A common, opinionated schema**: You can normalize different datasets into a single schema so that, for example, all your iMessages and emails are stored in a common schema. Don’t want to use the schema? `chronicle-etl` always allows you to fall back on working with the raw extraction data.
+
+## Installation
+```sh
+# Install chronicle-etl
+gem install chronicle-etl
 ```
 
-## Usage
+After installation, the `chronicle-etl` command will be available in your shell. Homebrew support [is coming soon](https://github.com/chronicle-app/chronicle-etl/issues/13).
 
-After installing the gem, `chronicle-etl` is available to run in your shell.
+## Basic usage and running jobs
 
-```bash
-# read test.csv and display it as a table
-$ chronicle-etl jobs:run --extractor csv --extractor-opts filename:test.csv --loader table
+```sh
+# Display help
+$ chronicle-etl help
 
-# Display help for the jobs:run command
-$ chronicle-etl jobs help run
+# Basic job usage
+$ chronicle-etl --extractor NAME --transformer NAME --loader NAME
+
+# Read test.csv and display it to stdout as a table 
+$ chronicle-etl --extractor csv --input ./data.csv --loader table
 ```
 
-## Connectors
+### Common options
+```sh
+Options:
+  -j, [--name=NAME]                    # Job configuration name
+  -e, [--extractor=EXTRACTOR-NAME]     # Extractor class. Default: stdin
+      [--extractor-opts=key:value]     # Extractor options
+  -t, [--transformer=TRANFORMER-NAME]  # Transformer class. Default: null
+      [--transformer-opts=key:value]   # Transformer options
+  -l, [--loader=LOADER-NAME]           # Loader class. Default: stdout
+      [--loader-opts=key:value]        # Loader options
+  -i, [--input=FILENAME]               # Input filename or directory
+      [--since=DATE]                   # Load records SINCE this date. Overrides job's `load_since` configuration option in extractor's options
+      [--until=DATE]                   # Load records UNTIL this date
+      [--limit=N]                      # Only extract the first LIMIT records
+  -o, [--output=OUTPUT]                # Output filename
+      [--fields=field1 field2 ...]     # Output only these fields
+      [--log-level=LOG_LEVEL]          # Log level (debug, info, warn, error, fatal)
+                                       # Default: info
+  -v, [--verbose], [--no-verbose]      # Set log level to verbose
+      [--silent], [--no-silent]        # Silence all output
+```
 
+## Connectors
 Connectors are available to read, process, and load data from different formats or external services.
 
-```bash
+```sh
 # List all available connectors
 $ chronicle-etl connectors:list
-
-# Install a connector
-$ chronicle-etl connectors:install imessage
 ```
 
-Built in connectors:
-
-### Extractors
-- `stdin` - (default) Load records from line-separated stdin
-- `csv`
-- `file` - load from a single file or directory (with a glob pattern)
-
-### Transformers
-- `null` - (default) Don't do anything
-
-### Loaders
-- `stdout` - (default) output records to stdout serialized as JSON
-- `csv` - Load records to a csv file
-- `rest` - Serialize records with [JSONAPI](https://jsonapi.org/) and send to a REST API
-- `table` - Output an ascii table of records. Useful for debugging.
+### Built-in Connectors
+`chronicle-etl` comes with several built-in connectors for common formats and sources.
 
-### Provider-specific importers
+#### Extractors
+- [`csv`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/extractors/csv_extractor.rb) - Load records from CSV files or stdin
+- [`json`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/extractors/json_extractor.rb) - Load JSON (either [line-separated objects](https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON) or one object)
+- [`file`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/extractors/file_extractor.rb) - load from a single file or directory (with a glob pattern)
 
-In addition to the built-in importers, importers for third-party platforms are available. They are packaged as individual Ruby gems.
+#### Transformers
+- [`null`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/transformers/null_transformer.rb) - (default) Don’t do anything and pass on raw extraction data
 
-- [email](https://github.com/chronicle-app/chronicle-email). Extractors for `mbox` and other email files
-- [shell](https://github.com/chronicle-app/chronicle-shell). Extract shell history from Bash or Zsh`
-- [imessage](https://github.com/chronicle-app/chronicle-imessage). Extract iMessage messages from a local macOS installation
+#### Loaders
+- [`table`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/loaders/table_loader.rb) - (default) Output an ascii table of records. Useful for exploring data.
+- [`csv`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/extractors/csv_extractor.rb) - Load records to CSV
+- [`json`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/loaders/json_loader.rb) - Load records serialized as JSON
+- [`rest`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/loaders/rest_loader.rb) - Serialize records with [JSONAPI](https://jsonapi.org/) and send to a REST API
 
-To install any of these, run `gem install chronicle-PROVIDER`. 
+### Plugins
+Plugins provide access to data from third-party platforms, services, or formats. 
 
-If you don't want to use the available rubygem importers, `chronicle-etl` can use `stdin` as an Extractor source (newline separated records). You can also use `stdout` as a loader — transformed records will be outputted separated by newlines.
-
-I'll be open-sourcing more importers. Please [contact me](mailto:andrew@hyfen.net) to chat about what will be available!
-
-## Full commands
+```bash
+# Install a plugin
+$ chronicle-etl plugins:install NAME
 
-```
-$ chronicle-etl help 
-
-ALL COMMANDS
-  help                       # This help menu
-  connectors help [COMMAND]  # Describe subcommands or one specific subcommand
-  connectors:install NAME    # Installs connector NAME
-  connectors:list            # Lists available connectors
-  jobs help [COMMAND]        # Describe subcommands or one specific subcommand
-  jobs:create                # Create a job
-  jobs:list                  # List all available jobs
-  jobs:run                   # Start a job
-  jobs:show                  # Show details about a job
-```
+# Install the imessage plugin
+$ chronicle-etl plugins:install imessage
 
-### Running a job
+# List installed plugins
+$ chronicle-etl plugins:list
 
+# Uninstall a plugin
+$ chronicle-etl plugins:uninstall NAME
 ```
-Usage:
-  chronicle-etl jobs:run
 
-Options:
-      [--log-level=LOG_LEVEL]           # Log level (debug, info, warn, error, fatal)
-                                        # Default: info
-  -v, [--verbose], [--no-verbose]       # Set log level to verbose
-      [--dry-run], [--no-dry-run]       # Only run the extraction and transform steps, not the loading
-  -e, [--extractor=extractor-name]      # Extractor class. Default: stdin
-      [--extractor-opts=key:value]      # Extractor options
-  -t, [--transformer=transformer-name]  # Transformer class. Default: null
-      [--transformer-opts=key:value]    # Transformer options
-  -l, [--loader=loader-name]            # Loader class. Default: stdout
-      [--loader-opts=key:value]         # Loader options
-  -j, [--name=NAME]                     # Job configuration name
-
-
-Runs an ETL job
+A few dozen importers exist [in my Memex project](https://hyfen.net/memex/) and they’re being ported over to the Chronicle system. This table shows what’s available now and what’s coming. Rows are sorted in very rough order of priority.
+
+If you want to work together on a connector, please [get in touch](#get-in-touch)!
+
+| Name                                                            | Description                                                                                 | Availability                     |
+|-----------------------------------------------------------------|---------------------------------------------------------------------------------------------|----------------------------------|
+| [imessage](https://github.com/chronicle-app/chronicle-imessage) | iMessage messages and attachments                                                           | Available                        |
+| [shell](https://github.com/chronicle-app/chronicle-shell)       | Shell command history                                                                       | Available (zsh support pending)  |
+| [email](https://github.com/chronicle-app/chronicle-email)       | Emails and attachments from IMAP or .mbox files                                             | Available (imap support pending) |
+| [pinboard](https://github.com/chronicle-app/chronicle-email)    | Bookmarks and tags                                                                          | Available                        |
+| [safari](https://github.com/chronicle-app/chronicle-safari)     | Browser history from local sqlite db                                                        | Available                        |
+| github                                                          | Github user and repo activity                                                               | In progress                      |
+| chrome                                                          | Browser history from local sqlite db                                                        | Needs porting                    |
+| whatsapp                                                        | Messaging history (via individual chat exports) or reverse-engineered local desktop install | Unstarted                        |
+| anki                                                            | Studying and card creation history                                                          | Needs porting                    |
+| facebook                                                        | Messaging and history posting via data export files                                         | Needs porting                    |
+| twitter                                                         | History via API or export data files                                                        | Needs porting                    |
+| foursquare                                                      | Location history via API                                                                    | Needs porting                    |
+| goodreads                                                       | Reading history via export csv (RIP goodreads API)                                          | Needs porting                    |
+| lastfm                                                          | Listening history via API                                                                   | Needs porting                    |
+| images                                                          | Process image files                                                                         | Needs porting                    |
+| arc                                                             | Location history from synced icloud backup files                                            | Needs porting                    |
+| firefox                                                         | Browser history from local sqlite db                                                        | Needs porting                    |
+| fitbit                                                          | Personal analytics via API                                                                  | Needs porting                    |
+| git                                                             | Commit history on a repo                                                                    | Needs porting                    |
+| google-calendar                                                 | Calendar events via API                                                                     | Needs porting                    |
+| instagram                                                       | Posting and messaging history via export data                                               | Needs porting                    |
+| shazam                                                          | Song tags via reverse-engineered API                                                        | Needs porting                    |
+| slack                                                           | Messaging history via API                                                                   | Need rethinking                  |
+| strava                                                          | Activity history via API                                                                    | Needs porting                    |
+| things                                                          | Task activity via local sqlite db                                                           | Needs porting                    |
+| bear                                                            | Note taking activity via local sqlite db                                                    | Needs porting                    |
+| youtube                                                         | Video activity via takeout data and API                                                     | Needs porting                    |
+
+### Writing your own connector
+
+Additional connectors are packaged as separate ruby gems. You can view the [iMessage plugin](https://github.com/chronicle-app/chronicle-imessage) for an example.
+
+If you want to load a custom connector without creating a gem, you can help by [completing this issue](https://github.com/chronicle-app/chronicle-etl/issues/23).
+
+If you want to work together on a connector, please [get in touch](#get-in-touch)! 
+
+#### Sample custom Extractor class
+```ruby
+module Chronicle
+  module FooService
+    class FooExtractor < Chronicle::ETL::Extractor
+      register_connector do |r|
+        r.identifier = 'foo'
+        r.description = 'From foo.com'
+      end
+
+      setting :access_token, required: true
+
+      def prepare
+        @records = # load from somewhere
+      end
+
+      def extract
+        @records.each do |record|
+          yield Chronicle::ETL::Extraction.new(data: row.to_h)
+        end
+      end
+    end
+  end
+end
 ```
 
 ## Development
-
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
 To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
-## Contributing
+### Additional development commands
+```bash
+# run tests
+bundle exec rake spec
+
+# generate docs
+bundle exec rake yard
+
+# use Guard to run specs automatically
+bundle exec guard
+```
 
+## Get in touch
+- [@hyfen](https://twitter.com/hyfen) on Twitter
+- [@hyfen](https://github.com/hyfen) on Github
+- Email: andrew@hyfen.net
+
+## Contributing
 Bug reports and pull requests are welcome on GitHub at https://github.com/chronicle-app/chronicle-etl. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
 
 ## License
-
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
 
 ## Code of Conduct
-
-Everyone interacting in the Chronicle::ETL project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/chronicle-app/chronicle-etl/blob/master/CODE_OF_CONDUCT.md).
+Everyone interacting in the Chronicle::ETL project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/chronicle-app/chronicle-etl/blob/main/CODE_OF_CONDUCT.md).

data/chronicle-etl.gemspec

@@ -47,8 +47,11 @@ Gem::Specification.new do |spec|
   spec.add_dependency "sequel", "~> 5.35"
   spec.add_dependency "sqlite3", "~> 1.4"
   spec.add_dependency "thor", "~> 1.2"
+  spec.add_dependency "thor-hollaback", "~> 0.2"
   spec.add_dependency "tty-progressbar", "~> 0.17"
+  spec.add_dependency "tty-spinner"
   spec.add_dependency "tty-table", "~> 0.11"
+  spec.add_dependency "tty-prompt", "~> 0.23"
 
   spec.add_development_dependency "bundler", "~> 2.1"
   spec.add_development_dependency "pry-byebug", "~> 3.9"

data/lib/chronicle/etl/cli/cli_base.rb

@@ -0,0 +1,31 @@
+module Chronicle
+  module ETL
+    module CLI
+      # Base class for CLI commands
+      class CLIBase < ::Thor
+        no_commands do
+          # Shorthand for cli_exit(status: :failure)
+          def cli_fail(message: nil, exception: nil)
+            cli_exit(status: :failure, message: message, exception: exception)
+          end
+
+          # Exit from CLI
+          #
+          # @params status Can be eitiher :success or :failure
+          # @params message to print
+          # @params exception stacktrace if log_level is set to debug
+          def cli_exit(status: :success, message: nil, exception: nil)
+            exit_code = status == :success ? 0 : 1
+            log_level = status == :success ? :info : :fatal
+
+            message = message.red if status != :success
+
+            Chronicle::ETL::Logger.debug(exception.full_message) if exception
+            Chronicle::ETL::Logger.send(log_level, message) if message
+            exit(exit_code)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/chronicle/etl/cli/connectors.rb

@@ -8,11 +8,6 @@ module Chronicle
         default_task 'list'
         namespace :connectors
 
-        desc "install NAME", "Installs connector NAME"
-        def install(name)
-          Chronicle::ETL::Registry.install_connector(name)
-        end
-
         desc "list", "Lists available connectors"
         # Display all available connectors that chronicle-etl has access to
         def list
@@ -44,21 +39,19 @@ module Chronicle
         desc "show PHASE IDENTIFIER", "Show information about a connector"
         def show(phase, identifier)
           unless ['extractor', 'transformer', 'loader'].include?(phase)
-            puts "phase argument must be one of: [extractor, transformer, loader]"
-            return
+            cli_fail(message: "Phase argument must be one of: [extractor, transformer, loader]")
           end
 
           begin
             connector = Chronicle::ETL::Registry.find_by_phase_and_identifier(phase.to_sym, identifier)
-          rescue Chronicle::ETL::ConnectorNotAvailableError
-            puts "Could not find #{phase} #{identifier}"
-            return
+          rescue Chronicle::ETL::ConnectorNotAvailableError, Chronicle::ETL::PluginError => e
+            cli_fail(message: "Could not find #{phase} #{identifier}", exception: e)
           end
 
           puts connector.klass.to_s.bold
           puts "  #{connector.descriptive_phrase}"
           puts
-          puts "OPTIONS"
+          puts "Settings:"
 
           headers = ['name', 'default', 'required'].map{ |h| h.to_s.upcase.bold }
 

data/lib/chronicle/etl/cli/jobs.rb

@@ -1,4 +1,5 @@
 require 'pp'
+require 'tty-prompt'
 
 module Chronicle
   module ETL
@@ -10,30 +11,26 @@ module Chronicle
 
         class_option :name, aliases: '-j', desc: 'Job configuration name'
 
-        class_option :extractor, aliases: '-e', desc: "Extractor class. Default: stdin", banner: 'extractor-name'
+        class_option :extractor, aliases: '-e', desc: "Extractor class. Default: stdin", banner: 'NAME'
         class_option :'extractor-opts', desc: 'Extractor options', type: :hash, default: {}
-        class_option :transformer, aliases: '-t', desc: 'Transformer class. Default: null', banner: 'transformer-name'
+        class_option :transformer, aliases: '-t', desc: 'Transformer class. Default: null', banner: 'NAME'
         class_option :'transformer-opts', desc: 'Transformer options', type: :hash, default: {}
-        class_option :loader, aliases: '-l', desc: 'Loader class. Default: stdout', banner: 'loader-name'
+        class_option :loader, aliases: '-l', desc: 'Loader class. Default: table', banner: 'NAME'
         class_option :'loader-opts', desc: 'Loader options', type: :hash, default: {}
 
         # This is an array to deal with shell globbing
         class_option :input, aliases: '-i', desc: 'Input filename or directory', default: [], type: 'array', banner: 'FILENAME'
-        class_option :since, desc: "Load records SINCE this date. Overrides job's `load_since` configuration option in extractor's options", banner: 'DATE'
+        class_option :since, desc: "Load records SINCE this date", banner: 'DATE'
         class_option :until, desc: "Load records UNTIL this date", banner: 'DATE'
         class_option :limit, desc: "Only extract the first LIMIT records", banner: 'N'
 
         class_option :output, aliases: '-o', desc: 'Output filename', type: 'string'
         class_option :fields, desc: 'Output only these fields', type: 'array', banner: 'field1 field2 ...'
-
-        class_option :log_level, desc: 'Log level (debug, info, warn, error, fatal)', default: 'info'
-        class_option :verbose, aliases: '-v', desc: 'Set log level to verbose', type: :boolean
+        class_option :header_row, desc: 'Output the header row of tabular output', type: 'boolean'
 
         # Thor doesn't like `run` as a command name
         map run: :start
         desc "run", "Start a job"
-        option :log_level, desc: 'Log level (debug, info, warn, error, fatal)', default: 'info'
-        option :verbose, aliases: '-v', desc: 'Set log level to verbose', type: :boolean
         option :dry_run, desc: 'Only run the extraction and transform steps, not the loading', type: :boolean
         long_desc <<-LONG_DESC
           This will run an ETL job. Each job needs three parts:
@@ -48,25 +45,41 @@ module Chronicle
 LONG_DESC
         # Run an ETL job
         def start
-          setup_log_level
           job_definition = build_job_definition(options)
-          job = Chronicle::ETL::Job.new(job_definition)
-          runner = Chronicle::ETL::Runner.new(job)
-          runner.run!
+
+          if job_definition.plugins_missing?
+            missing_plugins = job_definition.errors[:plugins]
+              .select { |error| error.is_a?(Chronicle::ETL::PluginLoadError) }
+              .map(&:name)
+              .uniq
+            install_missing_plugins(missing_plugins)
+          end
+
+          run_job(job_definition)
+        rescue Chronicle::ETL::JobDefinitionError => e
+          cli_fail(message: "Error running job.\n#{job_definition.errors}", exception: e)
         end
 
         desc "create", "Create a job"
         # Create an ETL job
         def create
           job_definition = build_job_definition(options)
+          job_definition.validate!
+
           path = File.join('chronicle', 'etl', 'jobs', options[:name])
           Chronicle::ETL::Config.write(path, job_definition.definition)
+        rescue Chronicle::ETL::JobDefinitionError => e
+          cli_fail(message: "Job definition error", exception: e)
         end
 
         desc "show", "Show details about a job"
         # Show an ETL job
         def show
-          puts Chronicle::ETL::Job.new(build_job_definition(options))
+          job_definition = build_job_definition(options)
+          job_definition.validate!
+          puts Chronicle::ETL::Job.new(job_definition)
+        rescue Chronicle::ETL::JobDefinitionError => e
+          cli_fail(message: "Job definition error", exception: e)
         end
 
         desc "list", "List all available jobs"
@@ -86,19 +99,32 @@ LONG_DESC
 
           headers = ['name', 'extractor', 'transformer', 'loader'].map { |h| h.upcase.bold }
 
+          puts "Available jobs:"
           table = TTY::Table.new(headers, job_details)
           puts table.render(indent: 0, padding: [0, 2])
+        rescue Chronicle::ETL::ConfigError => e
+          cli_fail(message: "Config error. #{e.message}", exception: e)
         end
 
         private
 
-        def setup_log_level
-          if options[:verbose]
-            Chronicle::ETL::Logger.log_level = Chronicle::ETL::Logger::DEBUG
-          elsif options[:log_level]
-            level = Chronicle::ETL::Logger.const_get(options[:log_level].upcase)
-            Chronicle::ETL::Logger.log_level = level 
-          end
+        def run_job(job_definition)
+          job = Chronicle::ETL::Job.new(job_definition)
+          runner = Chronicle::ETL::Runner.new(job)
+          runner.run!
+        end
+
+        # TODO: probably could merge this with something in cli/plugin
+        def install_missing_plugins(missing_plugins)
+          prompt = TTY::Prompt.new
+          message = "Plugin#{'s' if missing_plugins.count > 1} specified by job not installed.\n"
+          message += "Do you want to install "
+          message += missing_plugins.map { |name| "chronicle-#{name}".bold}.join(", ")
+          message += " and start the job?"
+          will_install = prompt.yes?(message)
+          cli_fail(message: "Must install #{missing_plugins.join(", ")} plugin to run job") unless will_install
+
+          Chronicle::ETL::CLI::Plugins.new.install(*missing_plugins)
         end
 
         # Create job definition by reading config file and then overwriting with flag options
@@ -116,7 +142,7 @@ LONG_DESC
         # Takes flag options and turns them into a runner config
         def process_flag_options options
           extractor_options = options[:'extractor-opts'].merge({
-            filename: (options[:input] if options[:input].any?),
+            input: (options[:input] if options[:input].any?),
             since: options[:since],
             until: options[:until],
             limit: options[:limit],
@@ -126,6 +152,7 @@ LONG_DESC
 
           loader_options = options[:'loader-opts'].merge({
             output: options[:output],
+            header_row: options[:header_row],
             fields: options[:fields]
           }.compact)
 

data/lib/chronicle/etl/cli/main.rb

@@ -4,7 +4,15 @@ module Chronicle
   module ETL
     module CLI
       # Main entrypoint for CLI app
-      class Main < ::Thor
+      class Main < Chronicle::ETL::CLI::CLIBase
+        class_before :set_log_level
+        class_before :set_color_output
+
+        class_option :log_level, desc: 'Log level (debug, info, warn, error, fatal, silent)', default: 'info'
+        class_option :verbose, aliases: '-v', desc: 'Set log level to verbose', type: :boolean
+        class_option :silent, desc: 'Silence all output', type: :boolean
+        class_option :'no-color', desc: 'Disable colour output', type: :boolean
+
         default_task "jobs"
 
         desc 'connectors:COMMAND', 'Connectors available for ETL jobs', hide: true
@@ -13,6 +21,9 @@ module Chronicle
         desc 'jobs:COMMAND', 'Configure and run jobs', hide: true
         subcommand 'jobs', Jobs
 
+        desc 'plugins:COMMAND', 'Configure plugins', hide: true
+        subcommand 'plugins', Plugins
+
         # Entrypoint for the CLI
         def self.start(given_args = ARGV, config = {})
           # take a subcommand:command and splits them so Thor knows how to hand off to the subcommand class
@@ -79,6 +90,26 @@ module Chronicle
             shell.say
           end
         end
+
+        no_commands do
+          def testb
+            puts "hi"
+          end
+          def set_color_output
+            String.disable_colorization true if options[:'no-color'] || ENV['NO_COLOR']
+          end
+
+          def set_log_level
+            if options[:silent]
+              Chronicle::ETL::Logger.log_level = Chronicle::ETL::Logger::SILENT
+            elsif options[:verbose]
+              Chronicle::ETL::Logger.log_level = Chronicle::ETL::Logger::DEBUG
+            elsif options[:log_level]
+              level = Chronicle::ETL::Logger.const_get(options[:log_level].upcase)
+              Chronicle::ETL::Logger.log_level = level
+            end
+          end
+        end
       end
     end
   end