chronicle-etl 0.5.3 → 0.5.4

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: d0f305f15f4eda7a5851dfff2155da2c12ee010d4619346a13551f298d5b7991
4
- data.tar.gz: d44f82b2bd06521740ad2b0e58cad0db840884fc5616858ef857d78fccb2b5dd
3
+ metadata.gz: 8f6b4272cd7f2cfcc12e6327324b5d2f11e76036dcf2442de1fe9ad08e041bb2
4
+ data.tar.gz: 4558bf48b7de7c64b691e8ef6403304c864e818360bc7c741a290f7650a7eb8c
5
5
  SHA512:
6
- metadata.gz: 99214409831e2799dffe2e3b096e9406222cf571d4e49fe71d3e3c645ad635e73c3aa42cb6af6569431064ce2750dc38ba051122a826b9feb6b21724ebd31db8
7
- data.tar.gz: 77a30ecb069906b0e992adbcb1b7470642bcda65b8767dd8ba0145d63e834cb83e2a0a7bc5212edcf8c5028c637b2edbd8476b559f8e28765267516308b160eb
6
+ metadata.gz: 6f7d7f4fd89d284a3a7ad5bffc05cf50ba6ea4e909457585dccfe9071ee91fb36bea505b3fa559a9b6ddab8f37845cf45651c1d1abc4045282c95c99ca9a5944
7
+ data.tar.gz: e2cf8a277c463d3b8ddef811e398fbae2e2649fb8227d14874443969b659ee68e74603f438ef4294b5caa59289f26dd5674dd28f9befe537b86b72ecf2ce7b40
data/README.md CHANGED
@@ -6,14 +6,14 @@
6
6
 
7
7
  Are you trying to archive your digital history or incorporate it into your own projects? You’ve probably discovered how frustrating it is to get machine-readable access to your own data. While [building a memex](https://hyfen.net/memex/), I learned first-hand what great efforts must be made before you can begin using the data in interesting ways.
8
8
 
9
- If you don’t want to spend all your time writing scrapers, reverse-engineering APIs, or parsing takeout data, this project is for you! (*If you do enjoy these things, please see the [open issues](https://github.com/chronicle-app/chronicle-etl/issues).*)
9
+ If you don’t want to spend all your time writing scrapers, reverse-engineering APIs, or parsing takeout data, this tool is for you! (*If you do enjoy these things, please see the [open issues](https://github.com/chronicle-app/chronicle-etl/issues).*)
10
10
 
11
- **`chronicle-etl` is a CLI tool that gives you a unified interface for accessing your personal data.** It uses the ETL pattern to *extract* it from a source (e.g. your local browser history, a directory of images, goodreads.com reading history), *transform* it (into a given schema), and *load* it to a destination (e.g. a CSV file, JSON, external API).
11
+ **`chronicle-etl` is a CLI tool that gives you a unified interface to your personal data.** It uses the ETL pattern to *extract* data from a source (e.g. your local browser history, a directory of images, goodreads.com reading history), *transform* it (into a given schema), and *load* it to a destination (e.g. a CSV file, JSON, external API).
12
12
 
13
13
  ## What does `chronicle-etl` give you?
14
- * **CLI tool for working with personal data**. You can monitor progress of exports, manipulate the output, set up recurring jobs, manage credentials, and more.
15
- * **Plugins for many third-party providers**. A plugin system allows you to access data from third-party providers and hook it into the shared CLI infrastructure.
16
- * **A common, opinionated schema**: You can normalize different datasets into a single schema so that, for example, all your iMessages and emails are stored in a common schema. Don’t want to use the schema? `chronicle-etl` always allows you to fall back on working with the raw extraction data.
14
+ * **A CLI tool for working with personal data**. You can monitor progress of exports, manipulate the output, set up recurring jobs, manage credentials, and more.
15
+ * **Plugins for many third-party providers**. This plugin system allows you to access data from dozens of third-party services, all accessible through a common CLI interface.
16
+ * **A common, opinionated schema**: You can normalize different datasets into a single schema so that, for example, all your iMessages and emails are represented in a common schema. (Don’t want to use this schema? `chronicle-etl` always allows you to fall back on working with the raw extraction data.)
17
17
 
18
18
  ## Chronicle-ETL in action
19
19
 
@@ -45,7 +45,7 @@ $ chronicle-etl --version
45
45
  # Display help
46
46
  $ chronicle-etl help
47
47
 
48
- # Basic job usage
48
+ # Run a basic job
49
49
  $ chronicle-etl --extractor NAME --transformer NAME --loader NAME
50
50
 
51
51
  # Read test.csv and display it to stdout as a table
@@ -88,7 +88,7 @@ Options:
88
88
 
89
89
  ### Saving jobs
90
90
 
91
- You can save details about a job to a local config file (saved by default in `~/.config/chronicle/etl/jobs/job_name.yml`) to save yourself the trouble of setting the CLI flags for each run.
91
+ You can save details about a job to a local config file (saved by default in `~/.config/chronicle/etl/jobs/JOB_NAME.yml`) to save yourself the trouble specifying options each time.
92
92
 
93
93
  ```sh
94
94
  # Save a job named 'sample' to ~/.config/chronicle/etl/jobs/sample.yml
@@ -99,8 +99,6 @@ $ chronicle-etl jobs:show sample
99
99
 
100
100
  # Run the job
101
101
  $ chronicle-etl jobs:run sample
102
- # Or more simply:
103
- $ chronicle-etl sample
104
102
 
105
103
  # Show all saved jobs
106
104
  $ chronicle-etl jobs:list
@@ -132,7 +130,7 @@ $ chronicle-etl connectors:list
132
130
  - [`rest`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/loaders/rest_loader.rb) - Serialize records with [JSONAPI](https://jsonapi.org/) and send to a REST API
133
131
 
134
132
  ## Chronicle Plugins
135
- Plugins provide access to data from third-party platforms, services, or formats. Plugins are packaged as separate rubygems and can be installed through the CLI (which installs the Gems under the hood).
133
+ Plugins provide access to data from third-party platforms, services, or formats. Plugins are packaged as separate rubygems and can be installed through the CLI (under the hood, it's a `gem install chronicle-PLUGINNAME`)
136
134
 
137
135
  ### Plugin usage
138
136
 
@@ -167,6 +165,8 @@ If you don't see a plugin for a third-party provider or data source that you're
167
165
  | [pinboard](https://github.com/chronicle-app/chronicle-email) | Bookmarks and tags | Available |
168
166
  | [safari](https://github.com/chronicle-app/chronicle-safari) | Browser history from local sqlite db | Available |
169
167
  | [shell](https://github.com/chronicle-app/chronicle-shell) | Shell command history | Available (still needs zsh support) |
168
+ | [zulip](https://github.com/chronicle-app/chronicle-zulip) | Zulip message history | Available (for private messages) |
169
+
170
170
 
171
171
  #### Coming soon
172
172
 
@@ -243,8 +243,9 @@ $ chronicle-etl secrets:unset pinboard access_token
243
243
  ## Roadmap
244
244
 
245
245
  - Keep tackling **new plugins**. See: [Chronicle Plugin Tracker](https://github.com/orgs/chronicle-app/projects/1)
246
- - Add support for **incremental extractions** #37
247
- - **Improve stdin extractor and shell command transformer** (#5) so that users can easily integrate their own scripts/tools into jobs
246
+ - Add an **OAuth2 authorizer** for services that require this type of authorization ([#48](https://github.com/chronicle-app/chronicle-etl/issues/48))
247
+ - Add support for **incremental extractions** ([#37](https://github.com/chronicle-app/chronicle-etl/issues/37))
248
+ - **Improve stdin extractor and shell command transformer** so that users can easily integrate their own scripts/languages/tools into jobs ([#5](https://github.com/chronicle-app/chronicle-etl/issues/48))
248
249
  - **Add documentation for Chronicle Schema**. It's found throughout this project but never explained.
249
250
 
250
251
  ## Development
@@ -40,10 +40,13 @@ Gem::Specification.new do |spec|
40
40
  spec.add_dependency "activesupport", "~> 7.0"
41
41
  spec.add_dependency "chronic_duration", "~> 0.10.6"
42
42
  spec.add_dependency "colorize", "~> 0.8.1"
43
+ spec.add_dependency 'launchy'
43
44
  spec.add_dependency "marcel", "~> 1.0.2"
44
45
  spec.add_dependency "mini_exiftool", "~> 2.10"
45
46
  spec.add_dependency "nokogiri", "~> 1.13"
47
+ spec.add_dependency 'omniauth', "~> 2"
46
48
  spec.add_dependency "sequel", "~> 5.35"
49
+ spec.add_dependency 'sinatra', "~> 2"
47
50
  spec.add_dependency "sqlite3", "~> 1.4"
48
51
  spec.add_dependency "thor", "~> 1.2"
49
52
  spec.add_dependency "thor-hollaback", "~> 0.2"
@@ -54,8 +57,8 @@ Gem::Specification.new do |spec|
54
57
  spec.add_dependency "xdg", ">= 4.0"
55
58
 
56
59
  spec.add_development_dependency "bundler", "~> 2.1"
57
- spec.add_development_dependency "guard-rspec", "~> 4.7.3"
58
60
  spec.add_development_dependency "fakefs"
61
+ spec.add_development_dependency "guard-rspec", "~> 4.7.3"
59
62
  spec.add_development_dependency "pry-byebug", "~> 3.9"
60
63
  spec.add_development_dependency "rake", "~> 13.0"
61
64
  spec.add_development_dependency "rspec", "~> 3.9"
@@ -0,0 +1,48 @@
1
+ require 'sinatra'
2
+ require 'omniauth'
3
+
4
+ module Chronicle
5
+ module ETL
6
+ class AuthorizationServer < Sinatra::Base
7
+ class << self
8
+ attr_accessor :latest_authorization
9
+ end
10
+
11
+ configure do
12
+ set :inline_templates, true
13
+ set :dump_errors, false
14
+ set :raise_errors, true
15
+ disable :logging
16
+ set :sessions, true
17
+ set :quiet, true
18
+ set :threaded, true
19
+ set :environment, ENV['APP_ENV'] == 'test' ? :test : :production
20
+ end
21
+
22
+ use OmniAuth::Builder do
23
+ Chronicle::ETL::OauthAuthorizer.all.each do |klass|
24
+ args = [klass.client_id, klass.client_secret, klass.options].compact
25
+ provider(
26
+ klass.strategy,
27
+ *args
28
+ )
29
+ end
30
+ end
31
+
32
+ OmniAuth.config.logger = Chronicle::ETL::Logger
33
+ OmniAuth.config.silence_get_warning = true
34
+ OmniAuth.config.allowed_request_methods = %i[get]
35
+
36
+ get '/auth/:provider/callback' do
37
+ authorization = request.env['omniauth.auth'].to_h.deep_transform_keys(&:to_sym)
38
+ self.class.latest_authorization = authorization
39
+ erb "<h1>Settings saved for #{params[:provider]}</h1><p>You can now close this tab and return to your terminal!</p>"
40
+ end
41
+
42
+ get '/auth/failure' do
43
+ # TODO: handle this
44
+ erb "<h1>Authentication Failed:</h1><h3>message:<h3> <pre>#{params}</pre>"
45
+ end
46
+ end
47
+ end
48
+ end
@@ -0,0 +1,37 @@
1
+ module Chronicle
2
+ module ETL
3
+ # An authorization strategy for a third-party data source
4
+ class Authorizer
5
+ class << self
6
+ attr_reader :provider_name
7
+
8
+ # Macro for setting provider on an Authorizer
9
+ def provider(provider_name)
10
+ @provider_name = provider_name
11
+ end
12
+
13
+ # From all loaded Authorizers, return the first one that matches
14
+ # a given provider
15
+ #
16
+ # @todo Have a proper identifier system for authorizers
17
+ # (to have more than one per plugin)
18
+ def find_by_provider(provider)
19
+ ObjectSpace.each_object(::Class).select {|klass| klass < self }.find do |authorizer|
20
+ authorizer.provider_name == provider
21
+ end
22
+ end
23
+ end
24
+
25
+ # Construct a new authorizer
26
+ def initialize(args)
27
+ end
28
+
29
+ # Main entry-point for authorization flows. Implemented by subclass
30
+ def authorize!
31
+ raise NotImplementedError
32
+ end
33
+ end
34
+ end
35
+ end
36
+
37
+ require_relative 'oauth_authorizer'
@@ -0,0 +1,63 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'sinatra'
4
+ require 'launchy'
5
+ require 'pp'
6
+
7
+ module Chronicle
8
+ module ETL
9
+ module CLI
10
+ # CLI commands for authorizing chronicle-etl with third-party services
11
+ class Authorizations < SubcommandBase
12
+ default_task 'new'
13
+ namespace :authorizations
14
+
15
+ desc "authorize", "Authorize with a third-party provider"
16
+ option :port, desc: 'Port to run authorization server on', type: :numeric, default: 4567
17
+ option :credentials, desc: 'Secrets namespace for where to read credentials from (default: PROVIDER)', type: :string, banner: 'NAMESPACE'
18
+ option :secrets, desc: 'Secrets namespace for where authorization should be saved to (default: PROVIDER)', type: :string, banner: 'NAMESPACE'
19
+ option :print, desc: 'Show authorization results (instead of just saving secrets)', type: :boolean, default: false
20
+ def new(provider)
21
+ authorizer_klass = find_authorizer_klass(provider)
22
+ credentials = load_credentials(provider: provider, credentials_source: options[:credentials])
23
+ authorizer = authorizer_klass.new(port: options[:port], credentials: credentials)
24
+
25
+ secrets = authorizer.authorize!
26
+ secrets_namespace = options[:secrets] || provider
27
+ Chronicle::ETL::Secrets.set_all(secrets_namespace, secrets)
28
+
29
+ pp secrets if options[:print]
30
+
31
+ cli_exit(message: "Authorization saved to '#{secrets_namespace}' secrets")
32
+ rescue StandardError => e
33
+ cli_fail(message: "Authorization not successful.\n" + e.message, exception: e)
34
+ end
35
+
36
+ private
37
+
38
+ def find_authorizer_klass(provider)
39
+ # TODO: this assumes provider:plugin one-to-one
40
+ unless Chronicle::ETL::Registry::PluginRegistry.installed?(provider)
41
+ cli_fail(message: "Plugin for #{provider} is not installed.")
42
+ end
43
+
44
+ begin
45
+ Chronicle::ETL::Registry::PluginRegistry.activate(provider)
46
+ rescue PluginError => e
47
+ cli_fail(message: "Could not load plugin '#{provider}'.\n" + e.message, exception: e)
48
+ end
49
+
50
+ Authorizer.find_by_provider(provider.to_sym) || cli_fail(message: "No authorizer available for '#{provider}'")
51
+ end
52
+
53
+ def load_credentials(provider:, credentials_source: nil)
54
+ if credentials_source && !Chronicle::ETL::Secrets.exists?(credentials_source)
55
+ cli_fail(message: "OAuth credentials specified as '#{credentials_source}' but a secrets namespace with that name does not exist.")
56
+ end
57
+
58
+ Chronicle::ETL::Secrets.read(credentials_source || provider)
59
+ end
60
+ end
61
+ end
62
+ end
63
+ end
@@ -6,7 +6,10 @@ module Chronicle
6
6
  no_commands do
7
7
  # Shorthand for cli_exit(status: :failure)
8
8
  def cli_fail(message: nil, exception: nil)
9
- message += "\nRe-run the command with --verbose to see details." if Chronicle::ETL::Logger.log_level > Chronicle::ETL::Logger::DEBUG
9
+ if exception && Chronicle::ETL::Logger.log_level > Chronicle::ETL::Logger::DEBUG
10
+ message += "\nRe-run the command with --verbose to see details."
11
+ end
12
+
10
13
  cli_exit(status: :failure, message: message, exception: exception)
11
14
  end
12
15
 
@@ -52,6 +52,8 @@ LONG_DESC
52
52
  cli_exit
53
53
  end
54
54
 
55
+ cli_fail(message: "Job '#{name}' does not exist") if name && !Chronicle::ETL::Config.exists?("jobs", name)
56
+
55
57
  job_definition = build_job_definition(name, options)
56
58
 
57
59
  if job_definition.plugins_missing?
@@ -102,6 +104,8 @@ LONG_DESC
102
104
  desc "show", "Show details about a job"
103
105
  # Show an ETL job
104
106
  def show(name = nil)
107
+ cli_fail(message: "Job '#{name}' does not exist") if name && !Chronicle::ETL::Config.exists?("jobs", name)
108
+
105
109
  job_definition = build_job_definition(name, options)
106
110
  job_definition.validate!
107
111
  puts Chronicle::ETL::Job.new(job_definition)
@@ -140,7 +144,6 @@ LONG_DESC
140
144
  job_definition.validate!
141
145
  # FIXME: clumsy to make CLI responsible for setting secrets here. Think about a better way to do this
142
146
  job_definition.apply_default_secrets
143
-
144
147
  job = Chronicle::ETL::Job.new(job_definition)
145
148
  runner = Chronicle::ETL::Runner.new(job)
146
149
  runner.run!
@@ -27,6 +27,9 @@ module Chronicle
27
27
  desc 'secrets:COMMAND', 'Manage secrets', hide: true
28
28
  subcommand 'secrets', Secrets
29
29
 
30
+ desc 'authorizations', 'Authorize', hide: true
31
+ subcommand 'authorizations', Authorizations
32
+
30
33
  # Entrypoint for the CLI
31
34
  def self.start(given_args = ARGV, config = {})
32
35
  # take a subcommand:command and splits them so Thor knows how to hand off to the subcommand class
@@ -46,7 +46,7 @@ module Chronicle
46
46
  all_secrets.each do |namespace, secrets|
47
47
  rows += secrets.map do |key, value|
48
48
  # hidden_value = (value[0..5] + ("*" * [0, [value.length - 5, 30].min].max)).truncate(30)
49
- truncated_value = value.truncate(30)
49
+ truncated_value = value&.truncate(30)
50
50
  [namespace, key, truncated_value]
51
51
  end
52
52
  end
@@ -4,6 +4,7 @@ require 'chronicle/etl'
4
4
 
5
5
  require 'chronicle/etl/cli/cli_base'
6
6
  require 'chronicle/etl/cli/subcommand_base'
7
+ require 'chronicle/etl/cli/authorizations'
7
8
  require 'chronicle/etl/cli/connectors'
8
9
  require 'chronicle/etl/cli/jobs'
9
10
  require 'chronicle/etl/cli/plugins'
@@ -4,6 +4,8 @@ module Chronicle
4
4
 
5
5
  class SecretsError < Error; end
6
6
 
7
+ class AuthorizationError < Error; end
8
+
7
9
  class ConfigError < Error; end
8
10
 
9
11
  class RunnerError < Error; end
@@ -17,8 +17,8 @@ module Chronicle
17
17
  def output message, level
18
18
  return unless level >= @log_level
19
19
 
20
- if @progress_bar
21
- @progress_bar.log(message)
20
+ if @ui_element
21
+ @ui_element.log(message)
22
22
  else
23
23
  $stderr.puts(message)
24
24
  end
@@ -40,12 +40,12 @@ module Chronicle
40
40
  output(message, DEBUG)
41
41
  end
42
42
 
43
- def attach_to_progress_bar(progress_bar)
44
- @progress_bar = progress_bar
43
+ def attach_to_ui(ui_element)
44
+ @ui_elemenet = ui_element
45
45
  end
46
46
 
47
- def detach_from_progress_bar
48
- @progress_bar = nil
47
+ def detach_from_ui
48
+ @ui_element = nil
49
49
  end
50
50
  end
51
51
  end
@@ -16,6 +16,8 @@ module Chronicle
16
16
  :aboutables, # inverse of above
17
17
  :depicts,
18
18
  :consumers,
19
+ :creators,
20
+ :creations,
19
21
  :contains,
20
22
  :containers # inverse of above
21
23
  ].freeze # TODO: add these to reflect Chronicle Schema
@@ -0,0 +1,142 @@
1
+ require 'omniauth'
2
+ require 'tty-spinner'
3
+
4
+ module Chronicle
5
+ module ETL
6
+ # An authorization strategy that uses oauth2 (and omniauth under the hood)
7
+ class OauthAuthorizer < Authorizer
8
+ class << self
9
+ attr_reader :strategy, :provider_name, :authorization_to_secret_map
10
+ attr_accessor :client_id, :client_secret
11
+
12
+ # Macro for specifying which omniauth strategy to use
13
+ def omniauth_strategy(strategy)
14
+ @strategy = strategy
15
+ end
16
+
17
+ # Macro for specifying which omniauth scopes to request
18
+ def scope(value)
19
+ options[:scope] = value
20
+ end
21
+
22
+ # Macro for specifying hash of returned authorization to secrets hash
23
+ def pluck_secrets(map)
24
+ @authorization_to_secret_map = map
25
+ end
26
+
27
+ # # Macro for specifying options to pass to omniauth
28
+ def options
29
+ @options ||= {}
30
+ end
31
+
32
+ # Returns all subclasses of OauthAuthorizer
33
+ # (Used by AuthorizationServer to build omniauth providers)
34
+ def all
35
+ ObjectSpace.each_object(::Class).select { |klass| klass < self }
36
+ end
37
+ end
38
+
39
+ attr_reader :authorization
40
+
41
+ # Create a new instance of OauthAuthorizer
42
+ def initialize(port:, credentials: {})
43
+ @port = port
44
+ @credentials = credentials
45
+ super
46
+ end
47
+
48
+ # Start up an authorization server and handle the oauth flow
49
+ def authorize!
50
+ associate_oauth_credentials
51
+ @server = load_server
52
+ spinner = TTY::Spinner.new(":spinner :title", format: :dots_2)
53
+ Chronicle::ETL::Logger.attach_to_ui(spinner)
54
+ spinner.auto_spin
55
+ spinner.update(title: "Starting temporary authorization server on port #{@port}""")
56
+
57
+ server_thread = start_authorization_server(port: @port)
58
+ start_oauth_flow
59
+
60
+ spinner.update(title: "Waiting for authorization to complete in your browser")
61
+ sleep 0.1 while authorization_pending?(server_thread)
62
+
63
+ @server.quit!
64
+ server_thread.join
65
+ spinner.success("(#{'successful'.green})")
66
+ Chronicle::ETL::Logger.detach_from_ui
67
+
68
+ # TODO: properly handle failed authorizations
69
+ raise Chronicle::ETL::AuthorizationError unless @server.latest_authorization
70
+
71
+ @authorization = @server.latest_authorization
72
+
73
+ extract_secrets(authorization: @authorization, pluck_values: self.class.authorization_to_secret_map)
74
+ end
75
+
76
+ private
77
+
78
+ def authorization_pending?(server_thread)
79
+ server_thread.status && !@server.latest_authorization
80
+ end
81
+
82
+ def associate_oauth_credentials
83
+ self.class.client_id = @credentials[:client_id]
84
+ self.class.client_secret = @credentials[:client_secret]
85
+ end
86
+
87
+ def load_server
88
+ # Load at runtime so that we can set omniauth strategies based on
89
+ # which chronicle plugin has been loaded.
90
+ require_relative './authorization_server'
91
+ Chronicle::ETL::AuthorizationServer
92
+ end
93
+
94
+ def start_authorization_server(port:)
95
+ @server.settings.port = port
96
+ suppress_webrick_logging(@server)
97
+ Thread.abort_on_exception = true
98
+ Thread.report_on_exception = false
99
+
100
+ Thread.new do
101
+ @server.run!({ port: @port }) do |s|
102
+ s.silent = true if s.class.to_s == "Thin::Server"
103
+ end
104
+ end
105
+ end
106
+
107
+ def start_oauth_flow
108
+ url = "http://localhost:#{@port}/auth/#{omniauth_strategy}"
109
+ Launchy.open(url)
110
+ rescue Launchy::CommandNotFoundError
111
+ Chronicle::ETL::Logger.info("Please open #{url} in a browser to continue")
112
+ end
113
+
114
+ def suppress_webrick_logging(server)
115
+ require 'webrick'
116
+ server.set(
117
+ :server_settings,
118
+ {
119
+ AccessLog: [],
120
+ # TODO: make this windows friendly
121
+ # https://github.com/winton/stasis/commit/77da36f43285fda129300e382f18dfaff48571b0
122
+ Logger: WEBrick::Log::new("/dev/null")
123
+ }
124
+ )
125
+ rescue LoadError
126
+ # no worries if we're not using WEBrick
127
+ end
128
+
129
+ def extract_secrets(authorization:, pluck_values:)
130
+ return authorization unless pluck_values&.any?
131
+
132
+ pluck_values.each_with_object({}) do |(key, identifiers), secrets|
133
+ secrets[key] = authorization.dig(*identifiers)
134
+ end
135
+ end
136
+
137
+ def omniauth_strategy
138
+ self.class.strategy
139
+ end
140
+ end
141
+ end
142
+ end
@@ -11,6 +11,18 @@ module Chronicle
11
11
  # @todo Better validation for whether a gem is actually a plugin
12
12
  # @todo Add ways to load a plugin that don't require a gem on rubygems.org
13
13
  module PluginRegistry
14
+ class << self
15
+ # Start of a system for having non-gem plugins. Right now, we just
16
+ # make registry aware of existenc of name of non-gem plugin
17
+ def register_standalone(name)
18
+ standalones << name
19
+ end
20
+
21
+ def standalones
22
+ @standalones ||= []
23
+ end
24
+ end
25
+
14
26
  # Does this plugin exist?
15
27
  def self.exists?(name)
16
28
  # TODO: implement this. Could query rubygems.org or use a hardcoded
@@ -33,8 +45,7 @@ module Chronicle
33
45
 
34
46
  # Check whether a given plugin is installed
35
47
  def self.installed?(name)
36
- gem_name = "chronicle-#{name}"
37
- all_installed.map(&:name).include?(gem_name)
48
+ (standalones + all_installed.map { |gem| gem.name.gsub("chronicle-", "") }).include?(name)
38
49
  end
39
50
 
40
51
  # Activate a plugin with given name by `require`ing it
@@ -63,7 +63,7 @@ class Chronicle::ETL::Runner
63
63
  def prepare_ui
64
64
  total = @extractor.results_count
65
65
  @progress_bar = Chronicle::ETL::Utils::ProgressBar.new(title: 'Running job', total: total)
66
- Chronicle::ETL::Logger.attach_to_progress_bar(@progress_bar)
66
+ Chronicle::ETL::Logger.attach_to_ui(@progress_bar)
67
67
  end
68
68
 
69
69
  def run_extraction
@@ -99,7 +99,7 @@ class Chronicle::ETL::Runner
99
99
  def finish_job
100
100
  @job_logger.save
101
101
  @progress_bar&.finish
102
- Chronicle::ETL::Logger.detach_from_progress_bar
102
+ Chronicle::ETL::Logger.detach_from_ui
103
103
  Chronicle::ETL::Logger.info(tty_log_completion)
104
104
  end
105
105
 
@@ -1,9 +1,16 @@
1
+ require "active_support/core_ext/hash/keys"
2
+
1
3
  module Chronicle
2
4
  module ETL
3
5
  # Secret management module
4
6
  module Secrets
5
7
  module_function
6
8
 
9
+ # Whether a given namespace exists
10
+ def exists?(namespace)
11
+ Chronicle::ETL::Config.exists?("secrets", namespace)
12
+ end
13
+
7
14
  # Save a setting to a namespaced config file
8
15
  def set(namespace, key, value)
9
16
  config = read(namespace)
@@ -11,6 +18,13 @@ module Chronicle
11
18
  write(namespace, config)
12
19
  end
13
20
 
21
+ # Save a hash to a secrets namespace
22
+ def set_all(namespace, secrets)
23
+ config = read(namespace)
24
+ config = config.merge(secrets.deep_stringify_keys)
25
+ write(namespace, config)
26
+ end
27
+
14
28
  # Remove a setting from a namespaced config file
15
29
  def unset(namespace, key)
16
30
  config = read(namespace)
@@ -42,7 +56,7 @@ module Chronicle
42
56
  data = {
43
57
  secrets: (secrets || {}).transform_keys(&:to_s),
44
58
  chronicle_etl_version: Chronicle::ETL::VERSION
45
- }.transform_keys(&:to_s) # Should I implement deeply_transform_keys...?
59
+ }.deep_stringify_keys
46
60
  Chronicle::ETL::Config.write("secrets", namespace, data)
47
61
  end
48
62
 
@@ -1,5 +1,5 @@
1
1
  module Chronicle
2
2
  module ETL
3
- VERSION = "0.5.3"
3
+ VERSION = "0.5.4"
4
4
  end
5
5
  end
data/lib/chronicle/etl.rb CHANGED
@@ -1,4 +1,5 @@
1
1
  require_relative 'etl/registry/registry'
2
+ require_relative 'etl/authorizer'
2
3
  require_relative 'etl/config'
3
4
  require_relative 'etl/configurable'
4
5
  require_relative 'etl/exceptions'
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: chronicle-etl
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.5.3
4
+ version: 0.5.4
5
5
  platform: ruby
6
6
  authors:
7
7
  - Andrew Louis
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2022-04-04 00:00:00.000000000 Z
11
+ date: 2022-04-23 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: activesupport
@@ -52,6 +52,20 @@ dependencies:
52
52
  - - "~>"
53
53
  - !ruby/object:Gem::Version
54
54
  version: 0.8.1
55
+ - !ruby/object:Gem::Dependency
56
+ name: launchy
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - ">="
60
+ - !ruby/object:Gem::Version
61
+ version: '0'
62
+ type: :runtime
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - ">="
67
+ - !ruby/object:Gem::Version
68
+ version: '0'
55
69
  - !ruby/object:Gem::Dependency
56
70
  name: marcel
57
71
  requirement: !ruby/object:Gem::Requirement
@@ -94,6 +108,20 @@ dependencies:
94
108
  - - "~>"
95
109
  - !ruby/object:Gem::Version
96
110
  version: '1.13'
111
+ - !ruby/object:Gem::Dependency
112
+ name: omniauth
113
+ requirement: !ruby/object:Gem::Requirement
114
+ requirements:
115
+ - - "~>"
116
+ - !ruby/object:Gem::Version
117
+ version: '2'
118
+ type: :runtime
119
+ prerelease: false
120
+ version_requirements: !ruby/object:Gem::Requirement
121
+ requirements:
122
+ - - "~>"
123
+ - !ruby/object:Gem::Version
124
+ version: '2'
97
125
  - !ruby/object:Gem::Dependency
98
126
  name: sequel
99
127
  requirement: !ruby/object:Gem::Requirement
@@ -108,6 +136,20 @@ dependencies:
108
136
  - - "~>"
109
137
  - !ruby/object:Gem::Version
110
138
  version: '5.35'
139
+ - !ruby/object:Gem::Dependency
140
+ name: sinatra
141
+ requirement: !ruby/object:Gem::Requirement
142
+ requirements:
143
+ - - "~>"
144
+ - !ruby/object:Gem::Version
145
+ version: '2'
146
+ type: :runtime
147
+ prerelease: false
148
+ version_requirements: !ruby/object:Gem::Requirement
149
+ requirements:
150
+ - - "~>"
151
+ - !ruby/object:Gem::Version
152
+ version: '2'
111
153
  - !ruby/object:Gem::Dependency
112
154
  name: sqlite3
113
155
  requirement: !ruby/object:Gem::Requirement
@@ -235,33 +277,33 @@ dependencies:
235
277
  - !ruby/object:Gem::Version
236
278
  version: '2.1'
237
279
  - !ruby/object:Gem::Dependency
238
- name: guard-rspec
280
+ name: fakefs
239
281
  requirement: !ruby/object:Gem::Requirement
240
282
  requirements:
241
- - - "~>"
283
+ - - ">="
242
284
  - !ruby/object:Gem::Version
243
- version: 4.7.3
285
+ version: '0'
244
286
  type: :development
245
287
  prerelease: false
246
288
  version_requirements: !ruby/object:Gem::Requirement
247
289
  requirements:
248
- - - "~>"
290
+ - - ">="
249
291
  - !ruby/object:Gem::Version
250
- version: 4.7.3
292
+ version: '0'
251
293
  - !ruby/object:Gem::Dependency
252
- name: fakefs
294
+ name: guard-rspec
253
295
  requirement: !ruby/object:Gem::Requirement
254
296
  requirements:
255
- - - ">="
297
+ - - "~>"
256
298
  - !ruby/object:Gem::Version
257
- version: '0'
299
+ version: 4.7.3
258
300
  type: :development
259
301
  prerelease: false
260
302
  version_requirements: !ruby/object:Gem::Requirement
261
303
  requirements:
262
- - - ">="
304
+ - - "~>"
263
305
  - !ruby/object:Gem::Version
264
- version: '0'
306
+ version: 4.7.3
265
307
  - !ruby/object:Gem::Dependency
266
308
  name: pry-byebug
267
309
  requirement: !ruby/object:Gem::Requirement
@@ -372,7 +414,10 @@ files:
372
414
  - chronicle-etl.gemspec
373
415
  - exe/chronicle-etl
374
416
  - lib/chronicle/etl.rb
417
+ - lib/chronicle/etl/authorization_server.rb
418
+ - lib/chronicle/etl/authorizer.rb
375
419
  - lib/chronicle/etl/cli.rb
420
+ - lib/chronicle/etl/cli/authorizations.rb
376
421
  - lib/chronicle/etl/cli/cli_base.rb
377
422
  - lib/chronicle/etl/cli/connectors.rb
378
423
  - lib/chronicle/etl/cli/jobs.rb
@@ -407,6 +452,7 @@ files:
407
452
  - lib/chronicle/etl/models/base.rb
408
453
  - lib/chronicle/etl/models/entity.rb
409
454
  - lib/chronicle/etl/models/raw.rb
455
+ - lib/chronicle/etl/oauth_authorizer.rb
410
456
  - lib/chronicle/etl/registry/connector_registration.rb
411
457
  - lib/chronicle/etl/registry/plugin_registry.rb
412
458
  - lib/chronicle/etl/registry/registry.rb