chronicle-etl 0.5.4 → 0.5.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8f6b4272cd7f2cfcc12e6327324b5d2f11e76036dcf2442de1fe9ad08e041bb2
-  data.tar.gz: 4558bf48b7de7c64b691e8ef6403304c864e818360bc7c741a290f7650a7eb8c
+  metadata.gz: a2de46efc3c5fbdc7ac120137bef56e13a138c8a95c8dd7d0a3542a65be65959
+  data.tar.gz: e8e3e9ae236e270b2926037419d5349170f85b8597c640c0b7b899552257fdb9
 SHA512:
-  metadata.gz: 6f7d7f4fd89d284a3a7ad5bffc05cf50ba6ea4e909457585dccfe9071ee91fb36bea505b3fa559a9b6ddab8f37845cf45651c1d1abc4045282c95c99ca9a5944
-  data.tar.gz: e2cf8a277c463d3b8ddef811e398fbae2e2649fb8227d14874443969b659ee68e74603f438ef4294b5caa59289f26dd5674dd28f9befe537b86b72ecf2ce7b40
+  metadata.gz: 58b293d45d1a7f4589aee080d0fb1348e45d76753a8d48bb33e694d6a2b6ea123d1495a12a42e98ed6d3e65926ad48bd7de1fb98dac9bc7bac9225ef00d32fb3
+  data.tar.gz: e73c73b67b4e3790347da3df22ed7a11357fdf43e91b734925503f8d0c189aabd07e26f51ed25cf1409fd68ba74dd72e1e7e01c113f773832c509172f0b4ee84

data/README.md

@@ -12,7 +12,7 @@ If you don’t want to spend all your time writing scrapers, reverse-engineering
 
 ## What does `chronicle-etl` give you?
 * **A CLI tool for working with personal data**. You can monitor progress of exports, manipulate the output, set up recurring jobs, manage credentials, and more.
-* **Plugins for many third-party providers**. This plugin system allows you to access data from dozens of third-party services, all accessible through a common CLI interface.
+* **Plugins for many third-party providers** (see [list](#available-plugins-and-connectors)). This plugin system allows you to access data from dozens of third-party services, all accessible through a common CLI interface.
 * **A common, opinionated schema**: You can normalize different datasets into a single schema so that, for example, all your iMessages and emails are represented in a common schema. (Don’t want to use this schema? `chronicle-etl` always allows you to fall back on working with the raw extraction data.)
 
 ## Chronicle-ETL in action
@@ -51,6 +51,10 @@ $ chronicle-etl --extractor NAME --transformer NAME --loader NAME
 # Read test.csv and display it to stdout as a table 
 $ chronicle-etl --extractor csv --input data.csv --loader table
 
+# Show available plugins and install one
+$ chronicle-etl plugins:list
+$ chronicle-etl plugins:install shell
+
 # Retrieve shell commands run in the last 5 hours
 $ chronicle-etl -e shell --since 5h
 
@@ -86,35 +90,36 @@ Options:
       [--silent], [--no-silent]          # Silence all output
 ```
 
-### Saving jobs
-
+### Saving a job
 You can save details about a job to a local config file (saved by default in `~/.config/chronicle/etl/jobs/JOB_NAME.yml`) to save yourself the trouble specifying options each time.
 
 ```sh
 # Save a job named 'sample' to ~/.config/chronicle/etl/jobs/sample.yml
 $ chronicle-etl jobs:save sample --extractor pinboard --since 10d
 
-# Show details about the job
-$ chronicle-etl jobs:show sample
-
 # Run the job
 $ chronicle-etl jobs:run sample
 
+# Show details about the job
+$ chronicle-etl jobs:show sample
+
 # Show all saved jobs
 $ chronicle-etl jobs:list
 ```
 
-## Connectors
-Connectors are available to read, process, and load data from different formats or external services.
+## Connectors and plugins
+
+Connectors let you work with different data formats or third-party providers.
+
+### Built-in Connectors
+
+`chronicle-etl` comes with several built-in connectors for common formats and sources.
 
 ```sh
 # List all available connectors
 $ chronicle-etl connectors:list
 ```
 
-### Built-in Connectors
-`chronicle-etl` comes with several built-in connectors for common formats and sources.
-
 #### Extractors
 - [`csv`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/extractors/csv_extractor.rb) - Load records from CSV files or stdin
 - [`json`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/extractors/json_extractor.rb) - Load JSON (either [line-separated objects](https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON) or one object)
@@ -129,18 +134,19 @@ $ chronicle-etl connectors:list
 - [`json`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/loaders/json_loader.rb) - Load records serialized as JSON
 - [`rest`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/loaders/rest_loader.rb) - Serialize records with [JSONAPI](https://jsonapi.org/) and send to a REST API
 
-## Chronicle Plugins
-Plugins provide access to data from third-party platforms, services, or formats. Plugins are packaged as separate rubygems and can be installed through the CLI (under the hood, it's a `gem install chronicle-PLUGINNAME`)
+### Chronicle Plugins for third-party services
+
+Plugins provide access to data from third-party platforms, services, or formats. Plugins are packaged as separate gems and can be installed through the CLI (under the hood, it's a `gem install chronicle-PLUGINNAME`)
 
-### Plugin usage
+#### Plugin usage
 
 ```bash
+# List available plugins
+$ chronicle-etl plugins:list
+
 # Install a plugin
 $ chronicle-etl plugins:install NAME
 
-# List installed plugins
-$ chronicle-etl plugins:list
-
 # Use a plugin
 $ chronicle-etl plugins:install shell
 $ chronicle-etl --extractor shell:history --limit 10
@@ -148,33 +154,45 @@ $ chronicle-etl --extractor shell:history --limit 10
 # Uninstall a plugin
 $ chronicle-etl plugins:uninstall NAME
 ```
-
-### Status
+#### Available plugins and connectors
+
+The following are the officially-supported list of plugins and their available connectors:
+
+| Plugin                                                              | Type        | Identifier       | Description                               | Description                               |
+|---------------------------------------------------------------------|-------------|------------------|-------------------------------------------|-------------------------------------------|
+| [email](https://github.com/chronicle-app/chronicle-email)           | extractor   | imap             | emails over an IMAP connection            | emails over an IMAP connection            |
+| [email](https://github.com/chronicle-app/chronicle-email)           | extractor   | mbox             | emails from an .mbox file                 | emails from an .mbox file                 |
+| [email](https://github.com/chronicle-app/chronicle-email)           | transformer | email            | email to Chronicle Schema                 | email to Chronicle Schema                 |
+| [foursquare](https://github.com/chronicle-app/chronicle-foursquare) | extractor   | checkins         | Foursqure visits                          | Foursqure visits                          |
+| [foursquare](https://github.com/chronicle-app/chronicle-foursquare) | transformer | checkin          | checkin to Chronicle Schema               | checkin to Chronicle Schema               |
+| [github](https://github.com/chronicle-app/chronicle-github)         | extractor   | activity         | user activity stream                      | user activity stream                      |
+| [imessage](https://github.com/chronicle-app/chronicle-imessage)     | extractor   | messages         | imessages from local macOS                | imessages from local macOS                |
+| [imessage](https://github.com/chronicle-app/chronicle-imessage)     | transformer | message          | imessage to Chronicle Schema              | imessage to Chronicle Schema              |
+| [pinboard](https://github.com/chronicle-app/chronicle-pinboard)     | extractor   | bookmarks        | Pinboard.in bookmarks                     | Pinboard.in bookmarks                     |
+| [pinboard](https://github.com/chronicle-app/chronicle-pinboard)     | transformer | bookmark         | bookmark to Chronicle Schema              | bookmark to Chronicle Schema              |
+| [safari](https://github.com/chronicle-app/chronicle-safari)         | extractor   | browser-history  | browser history                           | browser history                           |
+| [safari ](https://github.com/chronicle-app/chronicle-safari )       | transformer | browser-history  | browser history to Chronicle Schema       | browser history to Chronicle Schema       |
+| [shell](https://github.com/chronicle-app/chronicle-shell)           | extractor   | history          | shell command history (bash / zsh)        | shell command history (bash / zsh)        |
+| [shell](https://github.com/chronicle-app/chronicle-shell)           | transformer | command          | command to Chronicle Schema               | command to Chronicle Schema               |
+| [spotify](https://github.com/chronicle-app/chronicle-spotify)       | extractor   | liked-tracks     | liked tracks                              | liked tracks                              |
+| [spotify](https://github.com/chronicle-app/chronicle-spotify)       | extractor   | saved-albums     | saved albums                              | saved albums                              |
+| [spotify](https://github.com/chronicle-app/chronicle-spotify)       | extractor   | listens          | recently listened tracks (last 50 tracks) | recently listened tracks (last 50 tracks) |
+| [spotify](https://github.com/chronicle-app/chronicle-spotify)       | transformer | like             | like to Chronicle Schema                  | like to Chronicle Schema                  |
+| [spotify](https://github.com/chronicle-app/chronicle-spotify)       | transformer | listen           | listen to Chronicle Schema                | listen to Chronicle Schema                |
+| [spotify](https://github.com/chronicle-app/chronicle-spotify)       | authorizer  |                  | OAuth authorizer                          | OAuth authorizer                          |
+| [zulip](https://github.com/chronicle-app/chronicle-zulip)           | extractor   | private-messages | private messages                          | private messages                          |
+| [zulip](https://github.com/chronicle-app/chronicle-zulip)           | transformer | message          | message to Chronicle Schema               | message to Chronicle Schema               |
+
+
+### Coming soon
 
 A few dozen importers exist [in my Memex project](https://hyfen.net/memex/) and I'm porting them over to the Chronicle system. The [Chronicle Plugin Tracker](https://github.com/orgs/chronicle-app/projects/1/views/1) lets you keep track what's available and what's coming soon.
 
 If you don't see a plugin for a third-party provider or data source that you're interested in using with `chronicle-etl`, [please open an issue](https://github.com/chronicle-app/chronicle-etl/issues/new). If you want to work together on a plugin, please [get in touch](#get-in-touch)!
 
-#### Currently available
-
-| Name                                                            | Description                                                                                 | Availability                     |
-|-----------------------------------------------------------------|---------------------------------------------------------------------------------------------|----------------------------------|
-| [email](https://github.com/chronicle-app/chronicle-email)       | Emails and attachments from IMAP or .mbox files                                             | Available |
-| [github](https://github.com/chronicle-app/chronicle-github)     | Github activity stream                                                                      | Available                        |
-| [imessage](https://github.com/chronicle-app/chronicle-imessage) | iMessage messages and attachments                                                           | Available                        |
-| [pinboard](https://github.com/chronicle-app/chronicle-email)    | Bookmarks and tags                                                                          | Available                        |
-| [safari](https://github.com/chronicle-app/chronicle-safari)     | Browser history from local sqlite db                                                        | Available                        |
-| [shell](https://github.com/chronicle-app/chronicle-shell)       | Shell command history                                                                       | Available (still needs zsh support)  |
-| [zulip](https://github.com/chronicle-app/chronicle-zulip)       | Zulip message history                                                                       | Available (for private messages) |
-
-
-#### Coming soon
-
 In summary, the following **are coming soon**:
 anki, arc, bear, chrome, facebook, firefox, fitbit, foursquare, git, github, goodreads, google-calendar, images, instagram, lastfm, shazam, slack, strava, things, twitter, whatsapp, youtube.
 
-Please check the [Chronicle Plugin Tracker](https://github.com/orgs/chronicle-app/projects/1/views/1) for details.
-
 ### Writing your own plugin
 
 Additional connectors are packaged as separate ruby gems. You can view the [iMessage plugin](https://github.com/chronicle-app/chronicle-imessage) for an example.
@@ -209,6 +227,7 @@ module Chronicle
 end
 ```
 
+
 ## Secrets Management
 
 If your job needs secrets such as access tokens or passwords, `chronicle-etl` has a built-in secret management system. 
@@ -243,7 +262,6 @@ $ chronicle-etl secrets:unset pinboard access_token
 ## Roadmap
 
 - Keep tackling **new plugins**. See: [Chronicle Plugin Tracker](https://github.com/orgs/chronicle-app/projects/1)
-- Add an **OAuth2 authorizer** for services that require this type of authorization ([#48](https://github.com/chronicle-app/chronicle-etl/issues/48))
 - Add support for **incremental extractions** ([#37](https://github.com/chronicle-app/chronicle-etl/issues/37))
 - **Improve stdin extractor and shell command transformer** so that users can easily integrate their own scripts/languages/tools into jobs ([#5](https://github.com/chronicle-app/chronicle-etl/issues/48))
 - **Add documentation for Chronicle Schema**. It's found throughout this project but never explained.

data/chronicle-etl.gemspec

@@ -40,13 +40,14 @@ Gem::Specification.new do |spec|
   spec.add_dependency "activesupport", "~> 7.0"
   spec.add_dependency "chronic_duration", "~> 0.10.6"
   spec.add_dependency "colorize", "~> 0.8.1"
-  spec.add_dependency 'launchy'
+  spec.add_dependency "gems", ">= 1"
+  spec.add_dependency "launchy"
   spec.add_dependency "marcel", "~> 1.0.2"
   spec.add_dependency "mini_exiftool", "~> 2.10"
   spec.add_dependency "nokogiri", "~> 1.13"
-  spec.add_dependency 'omniauth', "~> 2"
+  spec.add_dependency "omniauth", "~> 2"
   spec.add_dependency "sequel", "~> 5.35"
-  spec.add_dependency 'sinatra', "~> 2"
+  spec.add_dependency "sinatra", "~> 2"
   spec.add_dependency "sqlite3", "~> 1.4"
   spec.add_dependency "thor", "~> 1.2"
   spec.add_dependency "thor-hollaback", "~> 0.2"
@@ -57,12 +58,14 @@ Gem::Specification.new do |spec|
   spec.add_dependency "xdg", ">= 4.0"
 
   spec.add_development_dependency "bundler", "~> 2.1"
-  spec.add_development_dependency "fakefs"
+  spec.add_development_dependency "fakefs", "~> 1.4"
   spec.add_development_dependency "guard-rspec", "~> 4.7.3"
   spec.add_development_dependency "pry-byebug", "~> 3.9"
   spec.add_development_dependency "rake", "~> 13.0"
   spec.add_development_dependency "rspec", "~> 3.9"
   spec.add_development_dependency "rubocop", "~> 1.25.1"
   spec.add_development_dependency "simplecov", "~> 0.21"
+  spec.add_development_dependency "vcr", "~> 6.1"
+  spec.add_development_dependency "webmock", "~> 3"
   spec.add_development_dependency "yard", "~> 0.9.7"
 end

data/lib/chronicle/etl/cli/authorizations.rb

@@ -37,12 +37,12 @@ module Chronicle
 
         def find_authorizer_klass(provider)
           # TODO: this assumes provider:plugin one-to-one
-          unless Chronicle::ETL::Registry::PluginRegistry.installed?(provider)
+          unless Chronicle::ETL::Registry::Plugins.installed?(provider)
             cli_fail(message: "Plugin for #{provider} is not installed.")
           end
 
           begin
-            Chronicle::ETL::Registry::PluginRegistry.activate(provider)
+            Chronicle::ETL::Registry::Plugins.activate(provider)
           rescue PluginError => e
             cli_fail(message: "Could not load plugin '#{provider}'.\n" + e.message, exception: e)
           end

data/lib/chronicle/etl/cli/connectors.rb

@@ -13,7 +13,7 @@ module Chronicle
         desc "list", "Lists available connectors"
         # Display all available connectors that chronicle-etl has access to
         def list
-          connector_info = Chronicle::ETL::Registry.connectors.map do |connector_registration|
+          connector_info = Chronicle::ETL::Registry::Connectors.connectors.map do |connector_registration|
             {
               identifier: connector_registration.identifier,
               phase: connector_registration.phase,
@@ -43,7 +43,7 @@ module Chronicle
           end
 
           begin
-            connector = Chronicle::ETL::Registry.find_by_phase_and_identifier(phase.to_sym, identifier)
+            connector = Chronicle::ETL::Registry::Connectors.find_by_phase_and_identifier(phase.to_sym, identifier)
           rescue Chronicle::ETL::ConnectorNotAvailableError, Chronicle::ETL::PluginError => e
             cli_fail(message: "Could not find #{phase} #{identifier}", exception: e)
           end

data/lib/chronicle/etl/cli/plugins.rb

@@ -16,7 +16,7 @@ module Chronicle
           cli_fail(message: "Please specify a plugin to install") unless plugins.any?
 
           installed, not_installed = plugins.partition do |plugin|
-            Chronicle::ETL::Registry::PluginRegistry.installed?(plugin)
+            Chronicle::ETL::Registry::Plugins.installed?(plugin)
           end
 
           puts "Already installed: #{installed.join(", ")}" if installed.any?
@@ -27,7 +27,7 @@ module Chronicle
 
           not_installed.each do |plugin|
             spinner.update(title: "Installing #{plugin}")
-            Chronicle::ETL::Registry::PluginRegistry.install(plugin)
+            Chronicle::ETL::Registry::Plugins.install(plugin)
 
           rescue Chronicle::ETL::PluginError => e
             spinner.error("Error".red)
@@ -41,7 +41,7 @@ module Chronicle
         def uninstall(name)
           spinner = TTY::Spinner.new("[:spinner] Uninstalling plugin #{name}...", format: :dots_2)
           spinner.auto_spin
-          Chronicle::ETL::Registry::PluginRegistry.uninstall(name)
+          Chronicle::ETL::Registry::Plugins.uninstall(name)
           spinner.success("(#{'successful'.green})")
         rescue Chronicle::ETL::PluginError => e
           spinner.error("Error".red)
@@ -51,20 +51,24 @@ module Chronicle
         desc "list", "Lists available plugins"
         # Display all available plugins that chronicle-etl has access to
         def list
-          plugins = Chronicle::ETL::Registry::PluginRegistry.all_installed_latest
-
-          info = plugins.map do |plugin|
-            {
-              name: plugin.name.sub("chronicle-", ""),
-              description: plugin.description,
-              version: plugin.version
-            }
+          values = Chronicle::ETL::Registry::Plugins.all
+            .map do |plugin|
+            [
+              plugin.name, 
+              plugin.description,
+              plugin.installed ? '✓' : '',
+              plugin.version
+            ]
           end
 
-          headers = ['name', 'description', 'version'].map{ |h| h.to_s.upcase.bold }
-          table = TTY::Table.new(headers, info.map(&:values))
-          puts "Installed plugins:"
-          puts table.render(indent: 2, padding: [0, 0])
+          headers = ['name', 'description', 'installed', 'version'].map{ |h| h.to_s.upcase.bold }
+          table = TTY::Table.new(headers, values)
+          puts "Available plugins:"
+          puts table.render(
+            indent: 2,
+            padding: [0, 0],
+            alignments: [:left, :left, :center, :left]
+          )
         end
       end
     end

data/lib/chronicle/etl/config.rb

@@ -1,3 +1,4 @@
+require "active_support/core_ext/hash/keys"
 require 'fileutils'
 require 'yaml'
 
@@ -21,6 +22,8 @@ module Chronicle
       def write(type, identifier, data)
         base = config_pathname_for_type(type)
         path = base.join("#{identifier}.yml")
+
+        data.deep_stringify_keys!
         FileUtils.mkdir_p(File.dirname(path))
         File.open(path, 'w', 0o600) do |f|
           # Ruby likes to add --- separators when writing yaml files

data/lib/chronicle/etl/configurable.rb

@@ -108,6 +108,10 @@ module Chronicle
         end
 
         def coerce_time(value)
+          # parsing yml files might result in us getting Date objects
+          # we convert to DateTime first to to ensure UTC
+          return value.to_datetime.to_time if value.is_a?(Date)
+
           return value unless value.is_a?(String)
 
           # Hacky check for duration strings like "60m"

data/lib/chronicle/etl/job_definition.rb

@@ -34,7 +34,7 @@ module Chronicle
       def validate
         @errors = {}
 
-        Chronicle::ETL::Registry::PHASES.each do |phase|
+        Chronicle::ETL::Registry::Connectors::PHASES.each do |phase|
           __send__("#{phase}_klass".to_sym)
         rescue Chronicle::ETL::PluginError => e
           @errors[:plugins] ||= []
@@ -66,7 +66,7 @@ module Chronicle
 
       # For each connector in this job, mix in secrets into the options
       def apply_default_secrets
-        Chronicle::ETL::Registry::PHASES.each do |phase|
+        Chronicle::ETL::Registry::Connectors::PHASES.each do |phase|
           # If the option have a `secrets` key, we look up those secrets and
           # mix them in. If not, use the connector's plugin name and look up 
           # secrets with the same namespace
@@ -124,11 +124,11 @@ module Chronicle
       private
 
       def load_klass(phase, identifier)
-        Chronicle::ETL::Registry.find_by_phase_and_identifier(phase, identifier).klass
+        Chronicle::ETL::Registry::Connectors.find_by_phase_and_identifier(phase, identifier).klass
       end
 
       def load_credentials
-        Chronicle::ETL::Registry::PHASES.each do |phase|
+        Chronicle::ETL::Registry::Connectors::PHASES.each do |phase|
           credentials_name = @definition[phase].dig(:options, :credentials)
           if credentials_name
             credentials = Chronicle::ETL::Config.load_credentials(credentials_name)

data/lib/chronicle/etl/logger.rb

@@ -41,7 +41,7 @@ module Chronicle
       end
 
       def attach_to_ui(ui_element)
-        @ui_elemenet = ui_element
+        @ui_element = ui_element
       end
 
       def detach_from_ui

data/lib/chronicle/etl/oauth_authorizer.rb

@@ -50,7 +50,6 @@ module Chronicle
         associate_oauth_credentials
         @server = load_server
         spinner = TTY::Spinner.new(":spinner :title", format: :dots_2)
-        Chronicle::ETL::Logger.attach_to_ui(spinner)
         spinner.auto_spin
         spinner.update(title: "Starting temporary authorization server on port #{@port}""")
 
@@ -63,7 +62,6 @@ module Chronicle
         @server.quit!
         server_thread.join
         spinner.success("(#{'successful'.green})")
-        Chronicle::ETL::Logger.detach_from_ui
 
         # TODO: properly handle failed authorizations
         raise Chronicle::ETL::AuthorizationError unless @server.latest_authorization

data/lib/chronicle/etl/registry/connectors.rb

@@ -0,0 +1,60 @@
+require 'rubygems'
+
+module Chronicle
+  module ETL
+    module Registry
+      # A singleton class that acts as a registry of connector classes available for ETL jobs
+      module Connectors
+        PHASES = [:extractor, :transformer, :loader].freeze
+        public_constant :PHASES
+
+        class << self
+          attr_accessor :connectors
+
+          def register(connector)
+            connectors << connector
+          end
+
+          def connectors
+            @connectors ||= []
+          end
+
+          # Find connector from amongst those currently loaded
+          def find_by_phase_and_identifier_local(phase, identifier)
+            connector = connectors.find { |c| c.phase == phase && c.identifier == identifier }
+          end
+
+          # Find connector and load relevant plugin to find it if necessary
+          def find_by_phase_and_identifier(phase, identifier)
+            connector = find_by_phase_and_identifier_local(phase, identifier)
+            return connector if connector
+
+            # if not available in built-in connectors, try to activate a
+            # relevant plugin and try again
+            if identifier.include?(":")
+              plugin, name = identifier.split(":")
+            else
+              # This case handles the case where the identifier is a 
+              # shorthand (ie `imessage`) because there's only one default
+              # connector.
+              plugin = identifier
+            end
+
+            raise(Chronicle::ETL::PluginNotInstalledError.new(plugin)) unless Chronicle::ETL::Registry::Plugins.installed?(plugin)
+
+            Chronicle::ETL::Registry::Plugins.activate(plugin)
+
+            candidates = connectors.select { |c| c.phase == phase && c.plugin == plugin }
+            # if no name given, just use first connector with right phase/plugin
+            # TODO: set up a property for connectors to specify that they're the
+            # default connector for the plugin
+            candidates = candidates.select { |c| c.identifier == name } if name
+            connector = candidates.first
+
+            connector || raise(ConnectorNotAvailableError, "Connector '#{identifier}' not found")
+          end
+        end
+      end
+    end
+  end
+end

data/lib/chronicle/etl/registry/plugin_registration.rb

@@ -0,0 +1,19 @@
+module Chronicle
+  module ETL
+    module Registry
+      class PluginRegistration
+        attr_accessor :name, :description, :gem, :version, :installed, :gemspec
+
+        def initialize(name=nil)
+          @installed = false
+          @name = name
+          yield self if block_given?
+        end
+
+        def installed?
+          @installed || false
+        end
+      end
+    end
+  end
+end

data/lib/chronicle/etl/registry/plugins.rb

@@ -0,0 +1,163 @@
+require 'rubygems'
+require 'rubygems/command'
+require 'rubygems/commands/install_command'
+require 'rubygems/uninstaller'
+require 'gems'
+require 'active_support/core_ext/hash/deep_merge'
+
+module Chronicle
+  module ETL
+    module Registry
+      # Responsible for managing plugins available to chronicle-etl
+      #
+      # @todo Better validation for whether a gem is actually a plugin
+      # @todo Add ways to load a plugin that don't require a gem on rubygems.org
+      module Plugins
+        KNOWN_PLUGINS = [
+          'email',
+          'foursquare',
+          'github',
+          'imessage',
+          'pinboard',
+          'safari',
+          'shell',
+          'spotify',
+          'zulip'
+        ].freeze
+        public_constant :KNOWN_PLUGINS
+
+        # Start of a system for having non-gem plugins. Right now, we just
+        # make registry aware of existence of name of non-gem plugin
+        def self.register_standalone(name:)
+          plugin = Chronicle::ETL::Registry::PluginRegistration.new do |p|
+            p.name = name
+            p.installed = true
+          end
+
+          installed_standalone << plugin
+        end
+
+        # Plugins either installed as gems or manually loaded/registered
+        def self.installed
+          installed_standalone + installed_as_gem
+        end
+
+        # Check whether a given plugin is installed
+        def self.installed?(name)
+          installed.map(&:name).include?(name)
+        end
+
+        # List of plugins installed as standalone
+        def self.installed_standalone
+          @standalones ||= []
+        end
+
+        # List of plugins installed as gems
+        def self.installed_as_gem
+          installed_gemspecs_latest.map do |gem|
+            Chronicle::ETL::Registry::PluginRegistration.new do |p|
+              p.name = gem.name.sub("chronicle-", "")
+              p.gem = gem.name
+              p.description = gem.description
+              p.version = gem.version.to_s
+              p.installed = true
+            end
+          end
+        end
+
+        # List of all plugins available to chronicle-etl
+        def self.available
+          available_as_gem
+        end
+
+        # List of plugins available through rubygems
+        # TODO: make this concurrent
+        def self.available_as_gem
+          KNOWN_PLUGINS.map do |name|
+            info = gem_info(name)
+            Chronicle::ETL::Registry::PluginRegistration.new do |p|
+              p.name = name
+              p.gem = info['name']
+              p.version = info['version']
+              p.description = info['info']
+            end
+          end
+        end
+
+        # Load info about a gem plugin from rubygems API
+        def self.gem_info(name)
+          gem_name = "chronicle-#{name}"
+          Gems.info(gem_name)
+        end
+
+        # Union of installed gems (latest version) + available gems
+        def self.all
+          (installed + available)
+            .group_by(&:name)
+            .transform_values { |plugin| plugin.find(&:installed) || plugin.first }
+            .values
+        end
+
+        # Does a plugin with a given name exist?
+        def self.exists?(name)
+          KNOWN_PLUGINS.include?(name)
+        end
+
+        # All versions of all plugins currently installed
+        def self.installed_gemspecs
+          # TODO: add check for chronicle-etl dependency
+          Gem::Specification.filter { |s| s.name.match(/^chronicle-/) && s.name != "chronicle-etl" }
+        end
+
+        # Latest version of each installed plugin
+        def self.installed_gemspecs_latest
+          installed_gemspecs.group_by(&:name)
+            .transform_values { |versions| versions.sort_by(&:version).reverse.first }
+            .values
+        end
+
+        # Activate a plugin with given name by `require`ing it
+        def self.activate(name)
+          # By default, activates the latest available version of a gem
+          # so don't have to run Kernel#gem separately
+          require "chronicle/#{name}"
+        rescue Gem::ConflictError => e
+          # TODO: figure out if there's more we can do here
+          raise Chronicle::ETL::PluginConflictError.new(name), "Plugin '#{name}' couldn't be loaded. #{e.message}"
+        rescue StandardError, LoadError => e
+          # StandardError to catch random non-loading problems that might occur
+          # when requiring the plugin (eg class macro invoked the wrong way)
+          # TODO: decide if this should be separated
+          raise Chronicle::ETL::PluginLoadError.new(name), "Plugin '#{name}' couldn't be loaded"
+        end
+
+        # Install a plugin to local gems
+        def self.install(name)
+          return if installed?(name)
+          raise(Chronicle::ETL::PluginNotAvailableError.new(name), "Plugin #{name} doesn't exist") unless exists?(name)
+
+          gem_name = "chronicle-#{name}"
+
+          Gem::DefaultUserInteraction.ui = Gem::SilentUI.new
+          Gem.install(gem_name)
+
+          activate(name)
+        rescue Gem::UnsatisfiableDependencyError
+          # TODO: we need to catch a lot more than this here
+          raise Chronicle::ETL::PluginNotAvailableError.new(name), "Plugin #{name} could not be installed."
+        end
+
+        # Uninstall a plugin
+        def self.uninstall(name)
+          gem_name = "chronicle-#{name}"
+          Gem::DefaultUserInteraction.ui = Gem::SilentUI.new
+          uninstaller = Gem::Uninstaller.new(gem_name)
+          uninstaller.uninstall
+        rescue Gem::InstallError
+          # TODO: strengthen this exception handling
+          raise(Chronicle::ETL::PluginError.new(name), "Plugin #{name} wasn't uninstalled")
+        end
+      end
+    end
+  end
+end

data/lib/chronicle/etl/registry/registry.rb

@@ -1,61 +1,12 @@
-require 'rubygems'
-
 module Chronicle
   module ETL
-    # A singleton class that acts as a registry of connector classes available for ETL jobs
     module Registry
-      PHASES = [:extractor, :transformer, :loader]
-
-      class << self
-        attr_accessor :connectors
-
-        def register(connector)
-          connectors << connector
-        end
-
-        def connectors
-          @connectors ||= []
-        end
-
-        # Find connector from amongst those currently loaded
-        def find_by_phase_and_identifier_local(phase, identifier)
-          connector = connectors.find { |c| c.phase == phase && c.identifier == identifier }
-        end
-
-        # Find connector and load relevant plugin to find it if necessary
-        def find_by_phase_and_identifier(phase, identifier)
-          connector = find_by_phase_and_identifier_local(phase, identifier)
-          return connector if connector
-
-          # if not available in built-in connectors, try to activate a
-          # relevant plugin and try again
-          if identifier.include?(":")
-            plugin, name = identifier.split(":")
-          else
-            # This case handles the case where the identifier is a 
-            # shorthand (ie `imessage`) because there's only one default
-            # connector.
-            plugin = identifier
-          end
-
-          raise(Chronicle::ETL::PluginNotInstalledError.new(plugin)) unless PluginRegistry.installed?(plugin)
-
-          PluginRegistry.activate(plugin)
-
-          candidates = connectors.select { |c| c.phase == phase && c.plugin == plugin }
-          # if no name given, just use first connector with right phase/plugin
-          # TODO: set up a property for connectors to specify that they're the
-          # default connector for the plugin
-          candidates = candidates.select { |c| c.identifier == name } if name
-          connector = candidates.first
-
-          connector || raise(ConnectorNotAvailableError, "Connector '#{identifier}' not found")
-        end
-      end
     end
   end
 end
 
 require_relative 'self_registering'
 require_relative 'connector_registration'
-require_relative 'plugin_registry'
+require_relative 'connectors'
+require_relative 'plugin_registration'
+require_relative 'plugins'

data/lib/chronicle/etl/registry/self_registering.rb

@@ -17,7 +17,7 @@ module Chronicle
         def register_connector
           @connector_registration ||= ::Chronicle::ETL::Registry::ConnectorRegistration.new(self)
           yield @connector_registration if block_given?
-          ::Chronicle::ETL::Registry.register(@connector_registration)
+          ::Chronicle::ETL::Registry::Connectors.register(@connector_registration)
         end
       end
     end

data/lib/chronicle/etl/secrets.rb

@@ -56,7 +56,7 @@ module Chronicle
         data = {
           secrets: (secrets || {}).transform_keys(&:to_s),
           chronicle_etl_version: Chronicle::ETL::VERSION
-        }.deep_stringify_keys
+        }
         Chronicle::ETL::Config.write("secrets", namespace, data)
       end
 

data/lib/chronicle/etl/version.rb

@@ -1,5 +1,5 @@
 module Chronicle
   module ETL
-    VERSION = "0.5.4"
+    VERSION = "0.5.5"
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: chronicle-etl
 version: !ruby/object:Gem::Version
-  version: 0.5.4
+  version: 0.5.5
 platform: ruby
 authors:
 - Andrew Louis
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2022-04-23 00:00:00.000000000 Z
+date: 2022-05-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -52,6 +52,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: 0.8.1
+- !ruby/object:Gem::Dependency
+  name: gems
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1'
 - !ruby/object:Gem::Dependency
   name: launchy
   requirement: !ruby/object:Gem::Requirement
@@ -280,16 +294,16 @@ dependencies:
   name: fakefs
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '1.4'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '1.4'
 - !ruby/object:Gem::Dependency
   name: guard-rspec
   requirement: !ruby/object:Gem::Requirement
@@ -374,6 +388,34 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '0.21'
+- !ruby/object:Gem::Dependency
+  name: vcr
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '6.1'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '6.1'
+- !ruby/object:Gem::Dependency
+  name: webmock
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3'
 - !ruby/object:Gem::Dependency
   name: yard
   requirement: !ruby/object:Gem::Requirement
@@ -454,7 +496,9 @@ files:
 - lib/chronicle/etl/models/raw.rb
 - lib/chronicle/etl/oauth_authorizer.rb
 - lib/chronicle/etl/registry/connector_registration.rb
-- lib/chronicle/etl/registry/plugin_registry.rb
+- lib/chronicle/etl/registry/connectors.rb
+- lib/chronicle/etl/registry/plugin_registration.rb
+- lib/chronicle/etl/registry/plugins.rb
 - lib/chronicle/etl/registry/registry.rb
 - lib/chronicle/etl/registry/self_registering.rb
 - lib/chronicle/etl/runner.rb

data/lib/chronicle/etl/registry/plugin_registry.rb

@@ -1,95 +0,0 @@
-require 'rubygems'
-require 'rubygems/command'
-require 'rubygems/commands/install_command'
-require 'rubygems/uninstaller'
-
-module Chronicle
-  module ETL
-    module Registry
-      # Responsible for managing plugins available to chronicle-etl
-      #
-      # @todo Better validation for whether a gem is actually a plugin
-      # @todo Add ways to load a plugin that don't require a gem on rubygems.org
-      module PluginRegistry
-        class << self
-          # Start of a system for having non-gem plugins. Right now, we just
-          # make registry aware of existenc of name of non-gem plugin
-          def register_standalone(name)
-            standalones << name
-          end
-
-          def standalones
-            @standalones ||= []
-          end
-        end
-
-        # Does this plugin exist?
-        def self.exists?(name)
-          # TODO: implement this. Could query rubygems.org or use a hardcoded
-          # list somewhere
-          true
-        end
-
-        # All versions of all plugins currently installed
-        def self.all_installed
-          # TODO: add check for chronicle-etl dependency
-          Gem::Specification.filter { |s| s.name.match(/^chronicle-/) && s.name != "chronicle-etl" }
-        end
-
-        # Latest version of each installed plugin
-        def self.all_installed_latest
-          all_installed.group_by(&:name)
-            .transform_values { |versions| versions.sort_by(&:version).reverse.first }
-            .values
-        end
-
-        # Check whether a given plugin is installed
-        def self.installed?(name)
-          (standalones + all_installed.map { |gem| gem.name.gsub("chronicle-", "") }).include?(name)
-        end
-
-        # Activate a plugin with given name by `require`ing it
-        def self.activate(name)
-          # By default, activates the latest available version of a gem
-          # so don't have to run Kernel#gem separately
-          require "chronicle/#{name}"
-        rescue Gem::ConflictError => e
-          # TODO: figure out if there's more we can do here
-          raise Chronicle::ETL::PluginConflictError.new(name), "Plugin '#{name}' couldn't be loaded. #{e.message}"
-        rescue StandardError, LoadError => e
-          # StandardError to catch random non-loading problems that might occur
-          # when requiring the plugin (eg class macro invoked the wrong way)
-          # TODO: decide if this should be separated
-          raise Chronicle::ETL::PluginLoadError.new(name), "Plugin '#{name}' couldn't be loaded"
-        end
-
-        # Install a plugin to local gems
-        def self.install(name)
-          return if installed?(name)
-
-          gem_name = "chronicle-#{name}"
-          raise(Chronicle::ETL::PluginNotAvailableError.new(gem_name), "Plugin #{name} doesn't exist") unless exists?(gem_name)
-
-          Gem::DefaultUserInteraction.ui = Gem::SilentUI.new
-          Gem.install(gem_name)
-
-          activate(name)
-        rescue Gem::UnsatisfiableDependencyError
-          # TODO: we need to catch a lot more than this here
-          raise Chronicle::ETL::PluginNotAvailableError.new(name), "Plugin #{name} could not be installed."
-        end
-
-        # Uninstall a plugin
-        def self.uninstall(name)
-          gem_name = "chronicle-#{name}"
-          Gem::DefaultUserInteraction.ui = Gem::SilentUI.new
-          uninstaller = Gem::Uninstaller.new(gem_name)
-          uninstaller.uninstall
-        rescue Gem::InstallError
-          # TODO: strengthen this exception handling
-          raise(Chronicle::ETL::PluginError.new(name), "Plugin #{name} wasn't uninstalled")
-        end
-      end
-    end
-  end
-end