chronicle-etl 0.5.3 → 0.5.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d0f305f15f4eda7a5851dfff2155da2c12ee010d4619346a13551f298d5b7991
-  data.tar.gz: d44f82b2bd06521740ad2b0e58cad0db840884fc5616858ef857d78fccb2b5dd
+  metadata.gz: 8f6b4272cd7f2cfcc12e6327324b5d2f11e76036dcf2442de1fe9ad08e041bb2
+  data.tar.gz: 4558bf48b7de7c64b691e8ef6403304c864e818360bc7c741a290f7650a7eb8c
 SHA512:
-  metadata.gz: 99214409831e2799dffe2e3b096e9406222cf571d4e49fe71d3e3c645ad635e73c3aa42cb6af6569431064ce2750dc38ba051122a826b9feb6b21724ebd31db8
-  data.tar.gz: 77a30ecb069906b0e992adbcb1b7470642bcda65b8767dd8ba0145d63e834cb83e2a0a7bc5212edcf8c5028c637b2edbd8476b559f8e28765267516308b160eb
+  metadata.gz: 6f7d7f4fd89d284a3a7ad5bffc05cf50ba6ea4e909457585dccfe9071ee91fb36bea505b3fa559a9b6ddab8f37845cf45651c1d1abc4045282c95c99ca9a5944
+  data.tar.gz: e2cf8a277c463d3b8ddef811e398fbae2e2649fb8227d14874443969b659ee68e74603f438ef4294b5caa59289f26dd5674dd28f9befe537b86b72ecf2ce7b40

data/README.md

@@ -6,14 +6,14 @@
 
 Are you trying to archive your digital history or incorporate it into your own projects? You’ve probably discovered how frustrating it is to get machine-readable access to your own data. While [building a memex](https://hyfen.net/memex/), I learned first-hand what great efforts must be made before you can begin using the data in interesting ways.
 
-If you don’t want to spend all your time writing scrapers, reverse-engineering APIs, or parsing takeout data, this project is for you! (*If you do enjoy these things, please see the [open issues](https://github.com/chronicle-app/chronicle-etl/issues).*)
+If you don’t want to spend all your time writing scrapers, reverse-engineering APIs, or parsing takeout data, this tool is for you! (*If you do enjoy these things, please see the [open issues](https://github.com/chronicle-app/chronicle-etl/issues).*)
 
-**`chronicle-etl` is a CLI tool that gives you a unified interface for accessing your personal data.** It uses the ETL pattern to *extract* it from a source (e.g. your local browser history, a directory of images, goodreads.com reading history), *transform* it (into a given schema), and *load* it to a destination (e.g. a CSV file, JSON, external API).
+**`chronicle-etl` is a CLI tool that gives you a unified interface to your personal data.** It uses the ETL pattern to *extract* data from a source (e.g. your local browser history, a directory of images, goodreads.com reading history), *transform* it (into a given schema), and *load* it to a destination (e.g. a CSV file, JSON, external API).
 
 ## What does `chronicle-etl` give you?
-* **CLI tool for working with personal data**. You can monitor progress of exports, manipulate the output, set up recurring jobs, manage credentials, and more.
-* **Plugins for many third-party providers**. A plugin system allows you to access data from third-party providers and hook it into the shared CLI infrastructure.
-* **A common, opinionated schema**: You can normalize different datasets into a single schema so that, for example, all your iMessages and emails are stored in a common schema. Don’t want to use the schema? `chronicle-etl` always allows you to fall back on working with the raw extraction data.
+* **A CLI tool for working with personal data**. You can monitor progress of exports, manipulate the output, set up recurring jobs, manage credentials, and more.
+* **Plugins for many third-party providers**. This plugin system allows you to access data from dozens of third-party services, all accessible through a common CLI interface.
+* **A common, opinionated schema**: You can normalize different datasets into a single schema so that, for example, all your iMessages and emails are represented in a common schema. (Don’t want to use this schema? `chronicle-etl` always allows you to fall back on working with the raw extraction data.)
 
 ## Chronicle-ETL in action
 
@@ -45,7 +45,7 @@ $ chronicle-etl --version
 # Display help
 $ chronicle-etl help
 
-# Basic job usage
+# Run a basic job 
 $ chronicle-etl --extractor NAME --transformer NAME --loader NAME
 
 # Read test.csv and display it to stdout as a table 
@@ -88,7 +88,7 @@ Options:
 
 ### Saving jobs
 
-You can save details about a job to a local config file (saved by default in `~/.config/chronicle/etl/jobs/job_name.yml`) to save yourself the trouble of setting the CLI flags for each run.
+You can save details about a job to a local config file (saved by default in `~/.config/chronicle/etl/jobs/JOB_NAME.yml`) to save yourself the trouble specifying options each time.
 
 ```sh
 # Save a job named 'sample' to ~/.config/chronicle/etl/jobs/sample.yml
@@ -99,8 +99,6 @@ $ chronicle-etl jobs:show sample
 
 # Run the job
 $ chronicle-etl jobs:run sample
-# Or more simply:
-$ chronicle-etl sample
 
 # Show all saved jobs
 $ chronicle-etl jobs:list
@@ -132,7 +130,7 @@ $ chronicle-etl connectors:list
 - [`rest`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/loaders/rest_loader.rb) - Serialize records with [JSONAPI](https://jsonapi.org/) and send to a REST API
 
 ## Chronicle Plugins
-Plugins provide access to data from third-party platforms, services, or formats. Plugins are packaged as separate rubygems and can be installed through the CLI (which installs the Gems under the hood).
+Plugins provide access to data from third-party platforms, services, or formats. Plugins are packaged as separate rubygems and can be installed through the CLI (under the hood, it's a `gem install chronicle-PLUGINNAME`)
 
 ### Plugin usage
 
@@ -167,6 +165,8 @@ If you don't see a plugin for a third-party provider or data source that you're
 | [pinboard](https://github.com/chronicle-app/chronicle-email)    | Bookmarks and tags                                                                          | Available                        |
 | [safari](https://github.com/chronicle-app/chronicle-safari)     | Browser history from local sqlite db                                                        | Available                        |
 | [shell](https://github.com/chronicle-app/chronicle-shell)       | Shell command history                                                                       | Available (still needs zsh support)  |
+| [zulip](https://github.com/chronicle-app/chronicle-zulip)       | Zulip message history                                                                       | Available (for private messages) |
+
 
 #### Coming soon
 
@@ -243,8 +243,9 @@ $ chronicle-etl secrets:unset pinboard access_token
 ## Roadmap
 
 - Keep tackling **new plugins**. See: [Chronicle Plugin Tracker](https://github.com/orgs/chronicle-app/projects/1)
-- Add support for **incremental extractions** #37
-- **Improve stdin extractor and shell command transformer** (#5) so that users can easily integrate their own scripts/tools into jobs
+- Add an **OAuth2 authorizer** for services that require this type of authorization ([#48](https://github.com/chronicle-app/chronicle-etl/issues/48))
+- Add support for **incremental extractions** ([#37](https://github.com/chronicle-app/chronicle-etl/issues/37))
+- **Improve stdin extractor and shell command transformer** so that users can easily integrate their own scripts/languages/tools into jobs ([#5](https://github.com/chronicle-app/chronicle-etl/issues/48))
 - **Add documentation for Chronicle Schema**. It's found throughout this project but never explained.
 
 ## Development

data/chronicle-etl.gemspec

@@ -40,10 +40,13 @@ Gem::Specification.new do |spec|
   spec.add_dependency "activesupport", "~> 7.0"
   spec.add_dependency "chronic_duration", "~> 0.10.6"
   spec.add_dependency "colorize", "~> 0.8.1"
+  spec.add_dependency 'launchy'
   spec.add_dependency "marcel", "~> 1.0.2"
   spec.add_dependency "mini_exiftool", "~> 2.10"
   spec.add_dependency "nokogiri", "~> 1.13"
+  spec.add_dependency 'omniauth', "~> 2"
   spec.add_dependency "sequel", "~> 5.35"
+  spec.add_dependency 'sinatra', "~> 2"
   spec.add_dependency "sqlite3", "~> 1.4"
   spec.add_dependency "thor", "~> 1.2"
   spec.add_dependency "thor-hollaback", "~> 0.2"
@@ -54,8 +57,8 @@ Gem::Specification.new do |spec|
   spec.add_dependency "xdg", ">= 4.0"
 
   spec.add_development_dependency "bundler", "~> 2.1"
-  spec.add_development_dependency "guard-rspec", "~> 4.7.3"
   spec.add_development_dependency "fakefs"
+  spec.add_development_dependency "guard-rspec", "~> 4.7.3"
   spec.add_development_dependency "pry-byebug", "~> 3.9"
   spec.add_development_dependency "rake", "~> 13.0"
   spec.add_development_dependency "rspec", "~> 3.9"

data/lib/chronicle/etl/authorization_server.rb

@@ -0,0 +1,48 @@
+require 'sinatra'
+require 'omniauth'
+
+module Chronicle
+  module ETL
+    class AuthorizationServer < Sinatra::Base
+      class << self
+        attr_accessor :latest_authorization
+      end
+
+      configure do
+        set :inline_templates, true
+        set :dump_errors, false
+        set :raise_errors, true
+        disable :logging
+        set :sessions, true
+        set :quiet, true
+        set :threaded, true
+        set :environment, ENV['APP_ENV'] == 'test' ? :test : :production
+      end
+
+      use OmniAuth::Builder do
+        Chronicle::ETL::OauthAuthorizer.all.each do |klass|
+          args = [klass.client_id, klass.client_secret, klass.options].compact
+          provider(
+            klass.strategy,
+            *args
+          )
+        end
+      end
+
+      OmniAuth.config.logger = Chronicle::ETL::Logger
+      OmniAuth.config.silence_get_warning = true
+      OmniAuth.config.allowed_request_methods = %i[get]
+
+      get '/auth/:provider/callback' do
+        authorization = request.env['omniauth.auth'].to_h.deep_transform_keys(&:to_sym)
+        self.class.latest_authorization = authorization
+        erb "<h1>Settings saved for #{params[:provider]}</h1><p>You can now close this tab and return to your terminal!</p>"
+      end
+
+      get '/auth/failure' do
+        # TODO: handle this
+        erb "<h1>Authentication Failed:</h1><h3>message:<h3> <pre>#{params}</pre>"
+      end
+    end
+  end
+end

data/lib/chronicle/etl/authorizer.rb

@@ -0,0 +1,37 @@
+module Chronicle
+  module ETL
+    # An authorization strategy for a third-party data source
+    class Authorizer
+      class << self
+        attr_reader :provider_name
+
+        # Macro for setting provider on an Authorizer
+        def provider(provider_name)
+          @provider_name = provider_name
+        end
+
+        # From all loaded Authorizers, return the first one that matches
+        # a given provider
+        #
+        # @todo Have a proper identifier system for authorizers
+        #   (to have more than one per plugin)
+        def find_by_provider(provider)
+          ObjectSpace.each_object(::Class).select {|klass| klass < self }.find do |authorizer|
+            authorizer.provider_name == provider
+          end
+        end
+      end
+
+      # Construct a new authorizer
+      def initialize(args)
+      end
+
+      # Main entry-point for authorization flows. Implemented by subclass
+      def authorize!
+        raise NotImplementedError
+      end
+    end
+  end
+end
+
+require_relative 'oauth_authorizer'

data/lib/chronicle/etl/cli/authorizations.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+require 'sinatra'
+require 'launchy'
+require 'pp'
+
+module Chronicle
+  module ETL
+    module CLI
+      # CLI commands for authorizing chronicle-etl with third-party services
+      class Authorizations < SubcommandBase
+        default_task 'new'
+        namespace :authorizations
+
+        desc "authorize", "Authorize with a third-party provider"
+        option :port, desc: 'Port to run authorization server on', type: :numeric, default: 4567
+        option :credentials, desc: 'Secrets namespace for where to read credentials from (default: PROVIDER)', type: :string, banner: 'NAMESPACE'
+        option :secrets, desc: 'Secrets namespace for where authorization should be saved to (default: PROVIDER)', type: :string, banner: 'NAMESPACE'
+        option :print, desc: 'Show authorization results (instead of just saving secrets)', type: :boolean, default: false
+        def new(provider)
+          authorizer_klass = find_authorizer_klass(provider)
+          credentials = load_credentials(provider: provider, credentials_source: options[:credentials])
+          authorizer = authorizer_klass.new(port: options[:port], credentials: credentials)
+
+          secrets = authorizer.authorize!
+          secrets_namespace = options[:secrets] || provider
+          Chronicle::ETL::Secrets.set_all(secrets_namespace, secrets)
+
+          pp secrets if options[:print]
+
+          cli_exit(message: "Authorization saved to '#{secrets_namespace}' secrets")
+        rescue StandardError => e
+          cli_fail(message: "Authorization not successful.\n" + e.message, exception: e)
+        end
+
+        private
+
+        def find_authorizer_klass(provider)
+          # TODO: this assumes provider:plugin one-to-one
+          unless Chronicle::ETL::Registry::PluginRegistry.installed?(provider)
+            cli_fail(message: "Plugin for #{provider} is not installed.")
+          end
+
+          begin
+            Chronicle::ETL::Registry::PluginRegistry.activate(provider)
+          rescue PluginError => e
+            cli_fail(message: "Could not load plugin '#{provider}'.\n" + e.message, exception: e)
+          end
+
+          Authorizer.find_by_provider(provider.to_sym) || cli_fail(message: "No authorizer available for '#{provider}'")
+        end
+
+        def load_credentials(provider:, credentials_source: nil)
+          if credentials_source && !Chronicle::ETL::Secrets.exists?(credentials_source)
+            cli_fail(message: "OAuth credentials specified as '#{credentials_source}' but a secrets namespace with that name does not exist.")
+          end
+
+          Chronicle::ETL::Secrets.read(credentials_source || provider)
+        end
+      end
+    end
+  end
+end

data/lib/chronicle/etl/cli/cli_base.rb

@@ -6,7 +6,10 @@ module Chronicle
         no_commands do
           # Shorthand for cli_exit(status: :failure)
           def cli_fail(message: nil, exception: nil)
-            message += "\nRe-run the command with --verbose to see details." if Chronicle::ETL::Logger.log_level > Chronicle::ETL::Logger::DEBUG
+            if exception && Chronicle::ETL::Logger.log_level > Chronicle::ETL::Logger::DEBUG
+              message += "\nRe-run the command with --verbose to see details."
+            end
+
             cli_exit(status: :failure, message: message, exception: exception)
           end
 

data/lib/chronicle/etl/cli/jobs.rb

@@ -52,6 +52,8 @@ LONG_DESC
             cli_exit
           end
 
+          cli_fail(message: "Job '#{name}' does not exist") if name && !Chronicle::ETL::Config.exists?("jobs", name)
+
           job_definition = build_job_definition(name, options)
 
           if job_definition.plugins_missing?
@@ -102,6 +104,8 @@ LONG_DESC
         desc "show", "Show details about a job"
         # Show an ETL job
         def show(name = nil)
+          cli_fail(message: "Job '#{name}' does not exist") if name && !Chronicle::ETL::Config.exists?("jobs", name)
+
           job_definition = build_job_definition(name, options)
           job_definition.validate!
           puts Chronicle::ETL::Job.new(job_definition)
@@ -140,7 +144,6 @@ LONG_DESC
           job_definition.validate!
           # FIXME: clumsy to make CLI responsible for setting secrets here. Think about a better way to do this
           job_definition.apply_default_secrets
-
           job = Chronicle::ETL::Job.new(job_definition)
           runner = Chronicle::ETL::Runner.new(job)
           runner.run!

data/lib/chronicle/etl/cli/main.rb

@@ -27,6 +27,9 @@ module Chronicle
         desc 'secrets:COMMAND', 'Manage secrets', hide: true
         subcommand 'secrets', Secrets
 
+        desc 'authorizations', 'Authorize', hide: true
+        subcommand 'authorizations', Authorizations
+
         # Entrypoint for the CLI
         def self.start(given_args = ARGV, config = {})
           # take a subcommand:command and splits them so Thor knows how to hand off to the subcommand class

data/lib/chronicle/etl/cli/secrets.rb

@@ -46,7 +46,7 @@ module Chronicle
           all_secrets.each do |namespace, secrets|
             rows += secrets.map do |key, value|
               # hidden_value = (value[0..5] + ("*" * [0, [value.length - 5, 30].min].max)).truncate(30)
-              truncated_value = value.truncate(30)
+              truncated_value = value&.truncate(30)
               [namespace, key, truncated_value]
             end
           end

data/lib/chronicle/etl/cli.rb

@@ -4,6 +4,7 @@ require 'chronicle/etl'
 
 require 'chronicle/etl/cli/cli_base'
 require 'chronicle/etl/cli/subcommand_base'
+require 'chronicle/etl/cli/authorizations'
 require 'chronicle/etl/cli/connectors'
 require 'chronicle/etl/cli/jobs'
 require 'chronicle/etl/cli/plugins'

data/lib/chronicle/etl/exceptions.rb

@@ -4,6 +4,8 @@ module Chronicle
 
     class SecretsError < Error; end
 
+    class AuthorizationError < Error; end
+
     class ConfigError < Error; end
 
     class RunnerError < Error; end

data/lib/chronicle/etl/logger.rb

@@ -17,8 +17,8 @@ module Chronicle
       def output message, level
         return unless level >= @log_level
 
-        if @progress_bar
-          @progress_bar.log(message)
+        if @ui_element
+          @ui_element.log(message)
         else
           $stderr.puts(message)
         end
@@ -40,12 +40,12 @@ module Chronicle
         output(message, DEBUG)
       end
 
-      def attach_to_progress_bar(progress_bar)
-        @progress_bar = progress_bar
+      def attach_to_ui(ui_element)
+        @ui_elemenet = ui_element
       end
 
-      def detach_from_progress_bar
-        @progress_bar = nil
+      def detach_from_ui
+        @ui_element = nil
       end
     end
   end

data/lib/chronicle/etl/models/entity.rb

@@ -16,6 +16,8 @@ module Chronicle
           :aboutables, # inverse of above
           :depicts,
           :consumers,
+          :creators,
+          :creations,
           :contains,
           :containers # inverse of above
         ].freeze  # TODO: add these to reflect Chronicle Schema

data/lib/chronicle/etl/oauth_authorizer.rb

@@ -0,0 +1,142 @@
+require 'omniauth'
+require 'tty-spinner'
+
+module Chronicle
+  module ETL
+    # An authorization strategy that uses oauth2 (and omniauth under the hood)
+    class OauthAuthorizer < Authorizer
+      class << self
+        attr_reader :strategy, :provider_name, :authorization_to_secret_map
+        attr_accessor :client_id, :client_secret
+
+        # Macro for specifying which omniauth strategy to use
+        def omniauth_strategy(strategy)
+          @strategy = strategy
+        end
+
+        # Macro for specifying which omniauth scopes to request
+        def scope(value)
+          options[:scope] = value
+        end
+
+        # Macro for specifying hash of returned authorization to secrets hash
+        def pluck_secrets(map)
+          @authorization_to_secret_map = map
+        end
+
+        # # Macro for specifying options to pass to omniauth
+        def options
+          @options ||= {}
+        end
+
+        # Returns all subclasses of OauthAuthorizer
+        # (Used by AuthorizationServer to build omniauth providers)
+        def all
+          ObjectSpace.each_object(::Class).select { |klass| klass < self }
+        end
+      end
+
+      attr_reader :authorization
+
+      # Create a new instance of OauthAuthorizer
+      def initialize(port:, credentials: {})
+        @port = port
+        @credentials = credentials
+        super
+      end
+
+      # Start up an authorization server and handle the oauth flow
+      def authorize!
+        associate_oauth_credentials
+        @server = load_server
+        spinner = TTY::Spinner.new(":spinner :title", format: :dots_2)
+        Chronicle::ETL::Logger.attach_to_ui(spinner)
+        spinner.auto_spin
+        spinner.update(title: "Starting temporary authorization server on port #{@port}""")
+
+        server_thread = start_authorization_server(port: @port)
+        start_oauth_flow
+
+        spinner.update(title: "Waiting for authorization to complete in your browser")
+        sleep 0.1 while authorization_pending?(server_thread)
+
+        @server.quit!
+        server_thread.join
+        spinner.success("(#{'successful'.green})")
+        Chronicle::ETL::Logger.detach_from_ui
+
+        # TODO: properly handle failed authorizations
+        raise Chronicle::ETL::AuthorizationError unless @server.latest_authorization
+
+        @authorization = @server.latest_authorization
+
+        extract_secrets(authorization: @authorization, pluck_values: self.class.authorization_to_secret_map)
+      end
+
+      private
+
+      def authorization_pending?(server_thread)
+        server_thread.status && !@server.latest_authorization
+      end
+
+      def associate_oauth_credentials
+        self.class.client_id = @credentials[:client_id]
+        self.class.client_secret = @credentials[:client_secret]
+      end
+
+      def load_server
+        # Load at runtime so that we can set omniauth strategies based on
+        # which chronicle plugin has been loaded.
+        require_relative './authorization_server'
+        Chronicle::ETL::AuthorizationServer
+      end
+
+      def start_authorization_server(port:)
+        @server.settings.port = port
+        suppress_webrick_logging(@server)
+        Thread.abort_on_exception = true
+        Thread.report_on_exception = false
+
+        Thread.new do
+          @server.run!({ port: @port }) do |s|
+            s.silent = true if s.class.to_s == "Thin::Server"
+          end
+        end
+      end
+
+      def start_oauth_flow
+        url = "http://localhost:#{@port}/auth/#{omniauth_strategy}"
+        Launchy.open(url)
+      rescue Launchy::CommandNotFoundError
+        Chronicle::ETL::Logger.info("Please open #{url} in a browser to continue")
+      end
+
+      def suppress_webrick_logging(server)
+        require 'webrick'
+        server.set(
+          :server_settings,
+          {
+            AccessLog: [],
+            # TODO: make this windows friendly
+            # https://github.com/winton/stasis/commit/77da36f43285fda129300e382f18dfaff48571b0
+            Logger: WEBrick::Log::new("/dev/null")
+          }
+        )
+      rescue LoadError
+        # no worries if we're not using WEBrick
+      end
+
+      def extract_secrets(authorization:, pluck_values:)
+        return authorization unless pluck_values&.any?
+
+        pluck_values.each_with_object({}) do |(key, identifiers), secrets|
+          secrets[key] = authorization.dig(*identifiers)
+        end
+      end
+
+      def omniauth_strategy
+        self.class.strategy
+      end
+    end
+  end
+end

data/lib/chronicle/etl/registry/plugin_registry.rb

@@ -11,6 +11,18 @@ module Chronicle
       # @todo Better validation for whether a gem is actually a plugin
       # @todo Add ways to load a plugin that don't require a gem on rubygems.org
       module PluginRegistry
+        class << self
+          # Start of a system for having non-gem plugins. Right now, we just
+          # make registry aware of existenc of name of non-gem plugin
+          def register_standalone(name)
+            standalones << name
+          end
+
+          def standalones
+            @standalones ||= []
+          end
+        end
+
         # Does this plugin exist?
         def self.exists?(name)
           # TODO: implement this. Could query rubygems.org or use a hardcoded
@@ -33,8 +45,7 @@ module Chronicle
 
         # Check whether a given plugin is installed
         def self.installed?(name)
-          gem_name = "chronicle-#{name}"
-          all_installed.map(&:name).include?(gem_name)
+          (standalones + all_installed.map { |gem| gem.name.gsub("chronicle-", "") }).include?(name)
         end
 
         # Activate a plugin with given name by `require`ing it

data/lib/chronicle/etl/runner.rb

@@ -63,7 +63,7 @@ class Chronicle::ETL::Runner
   def prepare_ui
     total = @extractor.results_count
     @progress_bar = Chronicle::ETL::Utils::ProgressBar.new(title: 'Running job', total: total)
-    Chronicle::ETL::Logger.attach_to_progress_bar(@progress_bar)
+    Chronicle::ETL::Logger.attach_to_ui(@progress_bar)
   end
 
   def run_extraction
@@ -99,7 +99,7 @@ class Chronicle::ETL::Runner
   def finish_job
     @job_logger.save
     @progress_bar&.finish
-    Chronicle::ETL::Logger.detach_from_progress_bar
+    Chronicle::ETL::Logger.detach_from_ui
     Chronicle::ETL::Logger.info(tty_log_completion)
   end
 

data/lib/chronicle/etl/secrets.rb

@@ -1,9 +1,16 @@
+require "active_support/core_ext/hash/keys"
+
 module Chronicle
   module ETL
     # Secret management module
     module Secrets
       module_function
 
+      # Whether a given namespace exists
+      def exists?(namespace)
+        Chronicle::ETL::Config.exists?("secrets", namespace)
+      end
+
       # Save a setting to a namespaced config file
       def set(namespace, key, value)
         config = read(namespace)
@@ -11,6 +18,13 @@ module Chronicle
         write(namespace, config)
       end
 
+      # Save a hash to a secrets namespace
+      def set_all(namespace, secrets)
+        config = read(namespace)
+        config = config.merge(secrets.deep_stringify_keys)
+        write(namespace, config)
+      end
+
       # Remove a setting from a namespaced config file
       def unset(namespace, key)
         config = read(namespace)
@@ -42,7 +56,7 @@ module Chronicle
         data = {
           secrets: (secrets || {}).transform_keys(&:to_s),
           chronicle_etl_version: Chronicle::ETL::VERSION
-        }.transform_keys(&:to_s) # Should I implement deeply_transform_keys...?
+        }.deep_stringify_keys
         Chronicle::ETL::Config.write("secrets", namespace, data)
       end
 

data/lib/chronicle/etl/version.rb

@@ -1,5 +1,5 @@
 module Chronicle
   module ETL
-    VERSION = "0.5.3"
+    VERSION = "0.5.4"
   end
 end

data/lib/chronicle/etl.rb

@@ -1,4 +1,5 @@
 require_relative 'etl/registry/registry'
+require_relative 'etl/authorizer'
 require_relative 'etl/config'
 require_relative 'etl/configurable'
 require_relative 'etl/exceptions'

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: chronicle-etl
 version: !ruby/object:Gem::Version
-  version: 0.5.3
+  version: 0.5.4
 platform: ruby
 authors:
 - Andrew Louis
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2022-04-04 00:00:00.000000000 Z
+date: 2022-04-23 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -52,6 +52,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: 0.8.1
+- !ruby/object:Gem::Dependency
+  name: launchy
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: marcel
   requirement: !ruby/object:Gem::Requirement
@@ -94,6 +108,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '1.13'
+- !ruby/object:Gem::Dependency
+  name: omniauth
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2'
 - !ruby/object:Gem::Dependency
   name: sequel
   requirement: !ruby/object:Gem::Requirement
@@ -108,6 +136,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '5.35'
+- !ruby/object:Gem::Dependency
+  name: sinatra
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2'
 - !ruby/object:Gem::Dependency
   name: sqlite3
   requirement: !ruby/object:Gem::Requirement
@@ -235,33 +277,33 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '2.1'
 - !ruby/object:Gem::Dependency
-  name: guard-rspec
+  name: fakefs
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: 4.7.3
+        version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: 4.7.3
+        version: '0'
 - !ruby/object:Gem::Dependency
-  name: fakefs
+  name: guard-rspec
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 4.7.3
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 4.7.3
 - !ruby/object:Gem::Dependency
   name: pry-byebug
   requirement: !ruby/object:Gem::Requirement
@@ -372,7 +414,10 @@ files:
 - chronicle-etl.gemspec
 - exe/chronicle-etl
 - lib/chronicle/etl.rb
+- lib/chronicle/etl/authorization_server.rb
+- lib/chronicle/etl/authorizer.rb
 - lib/chronicle/etl/cli.rb
+- lib/chronicle/etl/cli/authorizations.rb
 - lib/chronicle/etl/cli/cli_base.rb
 - lib/chronicle/etl/cli/connectors.rb
 - lib/chronicle/etl/cli/jobs.rb
@@ -407,6 +452,7 @@ files:
 - lib/chronicle/etl/models/base.rb
 - lib/chronicle/etl/models/entity.rb
 - lib/chronicle/etl/models/raw.rb
+- lib/chronicle/etl/oauth_authorizer.rb
 - lib/chronicle/etl/registry/connector_registration.rb
 - lib/chronicle/etl/registry/plugin_registry.rb
 - lib/chronicle/etl/registry/registry.rb