chronicle-etl 0.5.4 → 0.5.5
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +55 -37
- data/chronicle-etl.gemspec +7 -4
- data/lib/chronicle/etl/cli/authorizations.rb +2 -2
- data/lib/chronicle/etl/cli/connectors.rb +2 -2
- data/lib/chronicle/etl/cli/plugins.rb +19 -15
- data/lib/chronicle/etl/config.rb +3 -0
- data/lib/chronicle/etl/configurable.rb +4 -0
- data/lib/chronicle/etl/job_definition.rb +4 -4
- data/lib/chronicle/etl/logger.rb +1 -1
- data/lib/chronicle/etl/oauth_authorizer.rb +0 -2
- data/lib/chronicle/etl/registry/connectors.rb +60 -0
- data/lib/chronicle/etl/registry/plugin_registration.rb +19 -0
- data/lib/chronicle/etl/registry/plugins.rb +163 -0
- data/lib/chronicle/etl/registry/registry.rb +3 -52
- data/lib/chronicle/etl/registry/self_registering.rb +1 -1
- data/lib/chronicle/etl/secrets.rb +1 -1
- data/lib/chronicle/etl/version.rb +1 -1
- metadata +51 -7
- data/lib/chronicle/etl/registry/plugin_registry.rb +0 -95
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: a2de46efc3c5fbdc7ac120137bef56e13a138c8a95c8dd7d0a3542a65be65959
|
4
|
+
data.tar.gz: e8e3e9ae236e270b2926037419d5349170f85b8597c640c0b7b899552257fdb9
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 58b293d45d1a7f4589aee080d0fb1348e45d76753a8d48bb33e694d6a2b6ea123d1495a12a42e98ed6d3e65926ad48bd7de1fb98dac9bc7bac9225ef00d32fb3
|
7
|
+
data.tar.gz: e73c73b67b4e3790347da3df22ed7a11357fdf43e91b734925503f8d0c189aabd07e26f51ed25cf1409fd68ba74dd72e1e7e01c113f773832c509172f0b4ee84
|
data/README.md
CHANGED
@@ -12,7 +12,7 @@ If you don’t want to spend all your time writing scrapers, reverse-engineering
|
|
12
12
|
|
13
13
|
## What does `chronicle-etl` give you?
|
14
14
|
* **A CLI tool for working with personal data**. You can monitor progress of exports, manipulate the output, set up recurring jobs, manage credentials, and more.
|
15
|
-
* **Plugins for many third-party providers
|
15
|
+
* **Plugins for many third-party providers** (see [list](#available-plugins-and-connectors)). This plugin system allows you to access data from dozens of third-party services, all accessible through a common CLI interface.
|
16
16
|
* **A common, opinionated schema**: You can normalize different datasets into a single schema so that, for example, all your iMessages and emails are represented in a common schema. (Don’t want to use this schema? `chronicle-etl` always allows you to fall back on working with the raw extraction data.)
|
17
17
|
|
18
18
|
## Chronicle-ETL in action
|
@@ -51,6 +51,10 @@ $ chronicle-etl --extractor NAME --transformer NAME --loader NAME
|
|
51
51
|
# Read test.csv and display it to stdout as a table
|
52
52
|
$ chronicle-etl --extractor csv --input data.csv --loader table
|
53
53
|
|
54
|
+
# Show available plugins and install one
|
55
|
+
$ chronicle-etl plugins:list
|
56
|
+
$ chronicle-etl plugins:install shell
|
57
|
+
|
54
58
|
# Retrieve shell commands run in the last 5 hours
|
55
59
|
$ chronicle-etl -e shell --since 5h
|
56
60
|
|
@@ -86,35 +90,36 @@ Options:
|
|
86
90
|
[--silent], [--no-silent] # Silence all output
|
87
91
|
```
|
88
92
|
|
89
|
-
### Saving
|
90
|
-
|
93
|
+
### Saving a job
|
91
94
|
You can save details about a job to a local config file (saved by default in `~/.config/chronicle/etl/jobs/JOB_NAME.yml`) to save yourself the trouble specifying options each time.
|
92
95
|
|
93
96
|
```sh
|
94
97
|
# Save a job named 'sample' to ~/.config/chronicle/etl/jobs/sample.yml
|
95
98
|
$ chronicle-etl jobs:save sample --extractor pinboard --since 10d
|
96
99
|
|
97
|
-
# Show details about the job
|
98
|
-
$ chronicle-etl jobs:show sample
|
99
|
-
|
100
100
|
# Run the job
|
101
101
|
$ chronicle-etl jobs:run sample
|
102
102
|
|
103
|
+
# Show details about the job
|
104
|
+
$ chronicle-etl jobs:show sample
|
105
|
+
|
103
106
|
# Show all saved jobs
|
104
107
|
$ chronicle-etl jobs:list
|
105
108
|
```
|
106
109
|
|
107
|
-
## Connectors
|
108
|
-
|
110
|
+
## Connectors and plugins
|
111
|
+
|
112
|
+
Connectors let you work with different data formats or third-party providers.
|
113
|
+
|
114
|
+
### Built-in Connectors
|
115
|
+
|
116
|
+
`chronicle-etl` comes with several built-in connectors for common formats and sources.
|
109
117
|
|
110
118
|
```sh
|
111
119
|
# List all available connectors
|
112
120
|
$ chronicle-etl connectors:list
|
113
121
|
```
|
114
122
|
|
115
|
-
### Built-in Connectors
|
116
|
-
`chronicle-etl` comes with several built-in connectors for common formats and sources.
|
117
|
-
|
118
123
|
#### Extractors
|
119
124
|
- [`csv`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/extractors/csv_extractor.rb) - Load records from CSV files or stdin
|
120
125
|
- [`json`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/extractors/json_extractor.rb) - Load JSON (either [line-separated objects](https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON) or one object)
|
@@ -129,18 +134,19 @@ $ chronicle-etl connectors:list
|
|
129
134
|
- [`json`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/loaders/json_loader.rb) - Load records serialized as JSON
|
130
135
|
- [`rest`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/loaders/rest_loader.rb) - Serialize records with [JSONAPI](https://jsonapi.org/) and send to a REST API
|
131
136
|
|
132
|
-
|
133
|
-
|
137
|
+
### Chronicle Plugins for third-party services
|
138
|
+
|
139
|
+
Plugins provide access to data from third-party platforms, services, or formats. Plugins are packaged as separate gems and can be installed through the CLI (under the hood, it's a `gem install chronicle-PLUGINNAME`)
|
134
140
|
|
135
|
-
|
141
|
+
#### Plugin usage
|
136
142
|
|
137
143
|
```bash
|
144
|
+
# List available plugins
|
145
|
+
$ chronicle-etl plugins:list
|
146
|
+
|
138
147
|
# Install a plugin
|
139
148
|
$ chronicle-etl plugins:install NAME
|
140
149
|
|
141
|
-
# List installed plugins
|
142
|
-
$ chronicle-etl plugins:list
|
143
|
-
|
144
150
|
# Use a plugin
|
145
151
|
$ chronicle-etl plugins:install shell
|
146
152
|
$ chronicle-etl --extractor shell:history --limit 10
|
@@ -148,33 +154,45 @@ $ chronicle-etl --extractor shell:history --limit 10
|
|
148
154
|
# Uninstall a plugin
|
149
155
|
$ chronicle-etl plugins:uninstall NAME
|
150
156
|
```
|
151
|
-
|
152
|
-
|
157
|
+
#### Available plugins and connectors
|
158
|
+
|
159
|
+
The following are the officially-supported list of plugins and their available connectors:
|
160
|
+
|
161
|
+
| Plugin | Type | Identifier | Description | Description |
|
162
|
+
|---------------------------------------------------------------------|-------------|------------------|-------------------------------------------|-------------------------------------------|
|
163
|
+
| [email](https://github.com/chronicle-app/chronicle-email) | extractor | imap | emails over an IMAP connection | emails over an IMAP connection |
|
164
|
+
| [email](https://github.com/chronicle-app/chronicle-email) | extractor | mbox | emails from an .mbox file | emails from an .mbox file |
|
165
|
+
| [email](https://github.com/chronicle-app/chronicle-email) | transformer | email | email to Chronicle Schema | email to Chronicle Schema |
|
166
|
+
| [foursquare](https://github.com/chronicle-app/chronicle-foursquare) | extractor | checkins | Foursqure visits | Foursqure visits |
|
167
|
+
| [foursquare](https://github.com/chronicle-app/chronicle-foursquare) | transformer | checkin | checkin to Chronicle Schema | checkin to Chronicle Schema |
|
168
|
+
| [github](https://github.com/chronicle-app/chronicle-github) | extractor | activity | user activity stream | user activity stream |
|
169
|
+
| [imessage](https://github.com/chronicle-app/chronicle-imessage) | extractor | messages | imessages from local macOS | imessages from local macOS |
|
170
|
+
| [imessage](https://github.com/chronicle-app/chronicle-imessage) | transformer | message | imessage to Chronicle Schema | imessage to Chronicle Schema |
|
171
|
+
| [pinboard](https://github.com/chronicle-app/chronicle-pinboard) | extractor | bookmarks | Pinboard.in bookmarks | Pinboard.in bookmarks |
|
172
|
+
| [pinboard](https://github.com/chronicle-app/chronicle-pinboard) | transformer | bookmark | bookmark to Chronicle Schema | bookmark to Chronicle Schema |
|
173
|
+
| [safari](https://github.com/chronicle-app/chronicle-safari) | extractor | browser-history | browser history | browser history |
|
174
|
+
| [safari ](https://github.com/chronicle-app/chronicle-safari ) | transformer | browser-history | browser history to Chronicle Schema | browser history to Chronicle Schema |
|
175
|
+
| [shell](https://github.com/chronicle-app/chronicle-shell) | extractor | history | shell command history (bash / zsh) | shell command history (bash / zsh) |
|
176
|
+
| [shell](https://github.com/chronicle-app/chronicle-shell) | transformer | command | command to Chronicle Schema | command to Chronicle Schema |
|
177
|
+
| [spotify](https://github.com/chronicle-app/chronicle-spotify) | extractor | liked-tracks | liked tracks | liked tracks |
|
178
|
+
| [spotify](https://github.com/chronicle-app/chronicle-spotify) | extractor | saved-albums | saved albums | saved albums |
|
179
|
+
| [spotify](https://github.com/chronicle-app/chronicle-spotify) | extractor | listens | recently listened tracks (last 50 tracks) | recently listened tracks (last 50 tracks) |
|
180
|
+
| [spotify](https://github.com/chronicle-app/chronicle-spotify) | transformer | like | like to Chronicle Schema | like to Chronicle Schema |
|
181
|
+
| [spotify](https://github.com/chronicle-app/chronicle-spotify) | transformer | listen | listen to Chronicle Schema | listen to Chronicle Schema |
|
182
|
+
| [spotify](https://github.com/chronicle-app/chronicle-spotify) | authorizer | | OAuth authorizer | OAuth authorizer |
|
183
|
+
| [zulip](https://github.com/chronicle-app/chronicle-zulip) | extractor | private-messages | private messages | private messages |
|
184
|
+
| [zulip](https://github.com/chronicle-app/chronicle-zulip) | transformer | message | message to Chronicle Schema | message to Chronicle Schema |
|
185
|
+
|
186
|
+
|
187
|
+
### Coming soon
|
153
188
|
|
154
189
|
A few dozen importers exist [in my Memex project](https://hyfen.net/memex/) and I'm porting them over to the Chronicle system. The [Chronicle Plugin Tracker](https://github.com/orgs/chronicle-app/projects/1/views/1) lets you keep track what's available and what's coming soon.
|
155
190
|
|
156
191
|
If you don't see a plugin for a third-party provider or data source that you're interested in using with `chronicle-etl`, [please open an issue](https://github.com/chronicle-app/chronicle-etl/issues/new). If you want to work together on a plugin, please [get in touch](#get-in-touch)!
|
157
192
|
|
158
|
-
#### Currently available
|
159
|
-
|
160
|
-
| Name | Description | Availability |
|
161
|
-
|-----------------------------------------------------------------|---------------------------------------------------------------------------------------------|----------------------------------|
|
162
|
-
| [email](https://github.com/chronicle-app/chronicle-email) | Emails and attachments from IMAP or .mbox files | Available |
|
163
|
-
| [github](https://github.com/chronicle-app/chronicle-github) | Github activity stream | Available |
|
164
|
-
| [imessage](https://github.com/chronicle-app/chronicle-imessage) | iMessage messages and attachments | Available |
|
165
|
-
| [pinboard](https://github.com/chronicle-app/chronicle-email) | Bookmarks and tags | Available |
|
166
|
-
| [safari](https://github.com/chronicle-app/chronicle-safari) | Browser history from local sqlite db | Available |
|
167
|
-
| [shell](https://github.com/chronicle-app/chronicle-shell) | Shell command history | Available (still needs zsh support) |
|
168
|
-
| [zulip](https://github.com/chronicle-app/chronicle-zulip) | Zulip message history | Available (for private messages) |
|
169
|
-
|
170
|
-
|
171
|
-
#### Coming soon
|
172
|
-
|
173
193
|
In summary, the following **are coming soon**:
|
174
194
|
anki, arc, bear, chrome, facebook, firefox, fitbit, foursquare, git, github, goodreads, google-calendar, images, instagram, lastfm, shazam, slack, strava, things, twitter, whatsapp, youtube.
|
175
195
|
|
176
|
-
Please check the [Chronicle Plugin Tracker](https://github.com/orgs/chronicle-app/projects/1/views/1) for details.
|
177
|
-
|
178
196
|
### Writing your own plugin
|
179
197
|
|
180
198
|
Additional connectors are packaged as separate ruby gems. You can view the [iMessage plugin](https://github.com/chronicle-app/chronicle-imessage) for an example.
|
@@ -209,6 +227,7 @@ module Chronicle
|
|
209
227
|
end
|
210
228
|
```
|
211
229
|
|
230
|
+
|
212
231
|
## Secrets Management
|
213
232
|
|
214
233
|
If your job needs secrets such as access tokens or passwords, `chronicle-etl` has a built-in secret management system.
|
@@ -243,7 +262,6 @@ $ chronicle-etl secrets:unset pinboard access_token
|
|
243
262
|
## Roadmap
|
244
263
|
|
245
264
|
- Keep tackling **new plugins**. See: [Chronicle Plugin Tracker](https://github.com/orgs/chronicle-app/projects/1)
|
246
|
-
- Add an **OAuth2 authorizer** for services that require this type of authorization ([#48](https://github.com/chronicle-app/chronicle-etl/issues/48))
|
247
265
|
- Add support for **incremental extractions** ([#37](https://github.com/chronicle-app/chronicle-etl/issues/37))
|
248
266
|
- **Improve stdin extractor and shell command transformer** so that users can easily integrate their own scripts/languages/tools into jobs ([#5](https://github.com/chronicle-app/chronicle-etl/issues/48))
|
249
267
|
- **Add documentation for Chronicle Schema**. It's found throughout this project but never explained.
|
data/chronicle-etl.gemspec
CHANGED
@@ -40,13 +40,14 @@ Gem::Specification.new do |spec|
|
|
40
40
|
spec.add_dependency "activesupport", "~> 7.0"
|
41
41
|
spec.add_dependency "chronic_duration", "~> 0.10.6"
|
42
42
|
spec.add_dependency "colorize", "~> 0.8.1"
|
43
|
-
spec.add_dependency
|
43
|
+
spec.add_dependency "gems", ">= 1"
|
44
|
+
spec.add_dependency "launchy"
|
44
45
|
spec.add_dependency "marcel", "~> 1.0.2"
|
45
46
|
spec.add_dependency "mini_exiftool", "~> 2.10"
|
46
47
|
spec.add_dependency "nokogiri", "~> 1.13"
|
47
|
-
spec.add_dependency
|
48
|
+
spec.add_dependency "omniauth", "~> 2"
|
48
49
|
spec.add_dependency "sequel", "~> 5.35"
|
49
|
-
spec.add_dependency
|
50
|
+
spec.add_dependency "sinatra", "~> 2"
|
50
51
|
spec.add_dependency "sqlite3", "~> 1.4"
|
51
52
|
spec.add_dependency "thor", "~> 1.2"
|
52
53
|
spec.add_dependency "thor-hollaback", "~> 0.2"
|
@@ -57,12 +58,14 @@ Gem::Specification.new do |spec|
|
|
57
58
|
spec.add_dependency "xdg", ">= 4.0"
|
58
59
|
|
59
60
|
spec.add_development_dependency "bundler", "~> 2.1"
|
60
|
-
spec.add_development_dependency "fakefs"
|
61
|
+
spec.add_development_dependency "fakefs", "~> 1.4"
|
61
62
|
spec.add_development_dependency "guard-rspec", "~> 4.7.3"
|
62
63
|
spec.add_development_dependency "pry-byebug", "~> 3.9"
|
63
64
|
spec.add_development_dependency "rake", "~> 13.0"
|
64
65
|
spec.add_development_dependency "rspec", "~> 3.9"
|
65
66
|
spec.add_development_dependency "rubocop", "~> 1.25.1"
|
66
67
|
spec.add_development_dependency "simplecov", "~> 0.21"
|
68
|
+
spec.add_development_dependency "vcr", "~> 6.1"
|
69
|
+
spec.add_development_dependency "webmock", "~> 3"
|
67
70
|
spec.add_development_dependency "yard", "~> 0.9.7"
|
68
71
|
end
|
@@ -37,12 +37,12 @@ module Chronicle
|
|
37
37
|
|
38
38
|
def find_authorizer_klass(provider)
|
39
39
|
# TODO: this assumes provider:plugin one-to-one
|
40
|
-
unless Chronicle::ETL::Registry::
|
40
|
+
unless Chronicle::ETL::Registry::Plugins.installed?(provider)
|
41
41
|
cli_fail(message: "Plugin for #{provider} is not installed.")
|
42
42
|
end
|
43
43
|
|
44
44
|
begin
|
45
|
-
Chronicle::ETL::Registry::
|
45
|
+
Chronicle::ETL::Registry::Plugins.activate(provider)
|
46
46
|
rescue PluginError => e
|
47
47
|
cli_fail(message: "Could not load plugin '#{provider}'.\n" + e.message, exception: e)
|
48
48
|
end
|
@@ -13,7 +13,7 @@ module Chronicle
|
|
13
13
|
desc "list", "Lists available connectors"
|
14
14
|
# Display all available connectors that chronicle-etl has access to
|
15
15
|
def list
|
16
|
-
connector_info = Chronicle::ETL::Registry.connectors.map do |connector_registration|
|
16
|
+
connector_info = Chronicle::ETL::Registry::Connectors.connectors.map do |connector_registration|
|
17
17
|
{
|
18
18
|
identifier: connector_registration.identifier,
|
19
19
|
phase: connector_registration.phase,
|
@@ -43,7 +43,7 @@ module Chronicle
|
|
43
43
|
end
|
44
44
|
|
45
45
|
begin
|
46
|
-
connector = Chronicle::ETL::Registry.find_by_phase_and_identifier(phase.to_sym, identifier)
|
46
|
+
connector = Chronicle::ETL::Registry::Connectors.find_by_phase_and_identifier(phase.to_sym, identifier)
|
47
47
|
rescue Chronicle::ETL::ConnectorNotAvailableError, Chronicle::ETL::PluginError => e
|
48
48
|
cli_fail(message: "Could not find #{phase} #{identifier}", exception: e)
|
49
49
|
end
|
@@ -16,7 +16,7 @@ module Chronicle
|
|
16
16
|
cli_fail(message: "Please specify a plugin to install") unless plugins.any?
|
17
17
|
|
18
18
|
installed, not_installed = plugins.partition do |plugin|
|
19
|
-
Chronicle::ETL::Registry::
|
19
|
+
Chronicle::ETL::Registry::Plugins.installed?(plugin)
|
20
20
|
end
|
21
21
|
|
22
22
|
puts "Already installed: #{installed.join(", ")}" if installed.any?
|
@@ -27,7 +27,7 @@ module Chronicle
|
|
27
27
|
|
28
28
|
not_installed.each do |plugin|
|
29
29
|
spinner.update(title: "Installing #{plugin}")
|
30
|
-
Chronicle::ETL::Registry::
|
30
|
+
Chronicle::ETL::Registry::Plugins.install(plugin)
|
31
31
|
|
32
32
|
rescue Chronicle::ETL::PluginError => e
|
33
33
|
spinner.error("Error".red)
|
@@ -41,7 +41,7 @@ module Chronicle
|
|
41
41
|
def uninstall(name)
|
42
42
|
spinner = TTY::Spinner.new("[:spinner] Uninstalling plugin #{name}...", format: :dots_2)
|
43
43
|
spinner.auto_spin
|
44
|
-
Chronicle::ETL::Registry::
|
44
|
+
Chronicle::ETL::Registry::Plugins.uninstall(name)
|
45
45
|
spinner.success("(#{'successful'.green})")
|
46
46
|
rescue Chronicle::ETL::PluginError => e
|
47
47
|
spinner.error("Error".red)
|
@@ -51,20 +51,24 @@ module Chronicle
|
|
51
51
|
desc "list", "Lists available plugins"
|
52
52
|
# Display all available plugins that chronicle-etl has access to
|
53
53
|
def list
|
54
|
-
|
55
|
-
|
56
|
-
|
57
|
-
|
58
|
-
|
59
|
-
|
60
|
-
|
61
|
-
|
54
|
+
values = Chronicle::ETL::Registry::Plugins.all
|
55
|
+
.map do |plugin|
|
56
|
+
[
|
57
|
+
plugin.name,
|
58
|
+
plugin.description,
|
59
|
+
plugin.installed ? '✓' : '',
|
60
|
+
plugin.version
|
61
|
+
]
|
62
62
|
end
|
63
63
|
|
64
|
-
headers = ['name', 'description', 'version'].map{ |h| h.to_s.upcase.bold }
|
65
|
-
table = TTY::Table.new(headers,
|
66
|
-
puts "
|
67
|
-
puts table.render(
|
64
|
+
headers = ['name', 'description', 'installed', 'version'].map{ |h| h.to_s.upcase.bold }
|
65
|
+
table = TTY::Table.new(headers, values)
|
66
|
+
puts "Available plugins:"
|
67
|
+
puts table.render(
|
68
|
+
indent: 2,
|
69
|
+
padding: [0, 0],
|
70
|
+
alignments: [:left, :left, :center, :left]
|
71
|
+
)
|
68
72
|
end
|
69
73
|
end
|
70
74
|
end
|
data/lib/chronicle/etl/config.rb
CHANGED
@@ -1,3 +1,4 @@
|
|
1
|
+
require "active_support/core_ext/hash/keys"
|
1
2
|
require 'fileutils'
|
2
3
|
require 'yaml'
|
3
4
|
|
@@ -21,6 +22,8 @@ module Chronicle
|
|
21
22
|
def write(type, identifier, data)
|
22
23
|
base = config_pathname_for_type(type)
|
23
24
|
path = base.join("#{identifier}.yml")
|
25
|
+
|
26
|
+
data.deep_stringify_keys!
|
24
27
|
FileUtils.mkdir_p(File.dirname(path))
|
25
28
|
File.open(path, 'w', 0o600) do |f|
|
26
29
|
# Ruby likes to add --- separators when writing yaml files
|
@@ -108,6 +108,10 @@ module Chronicle
|
|
108
108
|
end
|
109
109
|
|
110
110
|
def coerce_time(value)
|
111
|
+
# parsing yml files might result in us getting Date objects
|
112
|
+
# we convert to DateTime first to to ensure UTC
|
113
|
+
return value.to_datetime.to_time if value.is_a?(Date)
|
114
|
+
|
111
115
|
return value unless value.is_a?(String)
|
112
116
|
|
113
117
|
# Hacky check for duration strings like "60m"
|
@@ -34,7 +34,7 @@ module Chronicle
|
|
34
34
|
def validate
|
35
35
|
@errors = {}
|
36
36
|
|
37
|
-
Chronicle::ETL::Registry::PHASES.each do |phase|
|
37
|
+
Chronicle::ETL::Registry::Connectors::PHASES.each do |phase|
|
38
38
|
__send__("#{phase}_klass".to_sym)
|
39
39
|
rescue Chronicle::ETL::PluginError => e
|
40
40
|
@errors[:plugins] ||= []
|
@@ -66,7 +66,7 @@ module Chronicle
|
|
66
66
|
|
67
67
|
# For each connector in this job, mix in secrets into the options
|
68
68
|
def apply_default_secrets
|
69
|
-
Chronicle::ETL::Registry::PHASES.each do |phase|
|
69
|
+
Chronicle::ETL::Registry::Connectors::PHASES.each do |phase|
|
70
70
|
# If the option have a `secrets` key, we look up those secrets and
|
71
71
|
# mix them in. If not, use the connector's plugin name and look up
|
72
72
|
# secrets with the same namespace
|
@@ -124,11 +124,11 @@ module Chronicle
|
|
124
124
|
private
|
125
125
|
|
126
126
|
def load_klass(phase, identifier)
|
127
|
-
Chronicle::ETL::Registry.find_by_phase_and_identifier(phase, identifier).klass
|
127
|
+
Chronicle::ETL::Registry::Connectors.find_by_phase_and_identifier(phase, identifier).klass
|
128
128
|
end
|
129
129
|
|
130
130
|
def load_credentials
|
131
|
-
Chronicle::ETL::Registry::PHASES.each do |phase|
|
131
|
+
Chronicle::ETL::Registry::Connectors::PHASES.each do |phase|
|
132
132
|
credentials_name = @definition[phase].dig(:options, :credentials)
|
133
133
|
if credentials_name
|
134
134
|
credentials = Chronicle::ETL::Config.load_credentials(credentials_name)
|
data/lib/chronicle/etl/logger.rb
CHANGED
@@ -50,7 +50,6 @@ module Chronicle
|
|
50
50
|
associate_oauth_credentials
|
51
51
|
@server = load_server
|
52
52
|
spinner = TTY::Spinner.new(":spinner :title", format: :dots_2)
|
53
|
-
Chronicle::ETL::Logger.attach_to_ui(spinner)
|
54
53
|
spinner.auto_spin
|
55
54
|
spinner.update(title: "Starting temporary authorization server on port #{@port}""")
|
56
55
|
|
@@ -63,7 +62,6 @@ module Chronicle
|
|
63
62
|
@server.quit!
|
64
63
|
server_thread.join
|
65
64
|
spinner.success("(#{'successful'.green})")
|
66
|
-
Chronicle::ETL::Logger.detach_from_ui
|
67
65
|
|
68
66
|
# TODO: properly handle failed authorizations
|
69
67
|
raise Chronicle::ETL::AuthorizationError unless @server.latest_authorization
|
@@ -0,0 +1,60 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
|
3
|
+
module Chronicle
|
4
|
+
module ETL
|
5
|
+
module Registry
|
6
|
+
# A singleton class that acts as a registry of connector classes available for ETL jobs
|
7
|
+
module Connectors
|
8
|
+
PHASES = [:extractor, :transformer, :loader].freeze
|
9
|
+
public_constant :PHASES
|
10
|
+
|
11
|
+
class << self
|
12
|
+
attr_accessor :connectors
|
13
|
+
|
14
|
+
def register(connector)
|
15
|
+
connectors << connector
|
16
|
+
end
|
17
|
+
|
18
|
+
def connectors
|
19
|
+
@connectors ||= []
|
20
|
+
end
|
21
|
+
|
22
|
+
# Find connector from amongst those currently loaded
|
23
|
+
def find_by_phase_and_identifier_local(phase, identifier)
|
24
|
+
connector = connectors.find { |c| c.phase == phase && c.identifier == identifier }
|
25
|
+
end
|
26
|
+
|
27
|
+
# Find connector and load relevant plugin to find it if necessary
|
28
|
+
def find_by_phase_and_identifier(phase, identifier)
|
29
|
+
connector = find_by_phase_and_identifier_local(phase, identifier)
|
30
|
+
return connector if connector
|
31
|
+
|
32
|
+
# if not available in built-in connectors, try to activate a
|
33
|
+
# relevant plugin and try again
|
34
|
+
if identifier.include?(":")
|
35
|
+
plugin, name = identifier.split(":")
|
36
|
+
else
|
37
|
+
# This case handles the case where the identifier is a
|
38
|
+
# shorthand (ie `imessage`) because there's only one default
|
39
|
+
# connector.
|
40
|
+
plugin = identifier
|
41
|
+
end
|
42
|
+
|
43
|
+
raise(Chronicle::ETL::PluginNotInstalledError.new(plugin)) unless Chronicle::ETL::Registry::Plugins.installed?(plugin)
|
44
|
+
|
45
|
+
Chronicle::ETL::Registry::Plugins.activate(plugin)
|
46
|
+
|
47
|
+
candidates = connectors.select { |c| c.phase == phase && c.plugin == plugin }
|
48
|
+
# if no name given, just use first connector with right phase/plugin
|
49
|
+
# TODO: set up a property for connectors to specify that they're the
|
50
|
+
# default connector for the plugin
|
51
|
+
candidates = candidates.select { |c| c.identifier == name } if name
|
52
|
+
connector = candidates.first
|
53
|
+
|
54
|
+
connector || raise(ConnectorNotAvailableError, "Connector '#{identifier}' not found")
|
55
|
+
end
|
56
|
+
end
|
57
|
+
end
|
58
|
+
end
|
59
|
+
end
|
60
|
+
end
|
@@ -0,0 +1,19 @@
|
|
1
|
+
module Chronicle
|
2
|
+
module ETL
|
3
|
+
module Registry
|
4
|
+
class PluginRegistration
|
5
|
+
attr_accessor :name, :description, :gem, :version, :installed, :gemspec
|
6
|
+
|
7
|
+
def initialize(name=nil)
|
8
|
+
@installed = false
|
9
|
+
@name = name
|
10
|
+
yield self if block_given?
|
11
|
+
end
|
12
|
+
|
13
|
+
def installed?
|
14
|
+
@installed || false
|
15
|
+
end
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
19
|
+
end
|
@@ -0,0 +1,163 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'rubygems/command'
|
3
|
+
require 'rubygems/commands/install_command'
|
4
|
+
require 'rubygems/uninstaller'
|
5
|
+
require 'gems'
|
6
|
+
require 'active_support/core_ext/hash/deep_merge'
|
7
|
+
|
8
|
+
module Chronicle
|
9
|
+
module ETL
|
10
|
+
module Registry
|
11
|
+
# Responsible for managing plugins available to chronicle-etl
|
12
|
+
#
|
13
|
+
# @todo Better validation for whether a gem is actually a plugin
|
14
|
+
# @todo Add ways to load a plugin that don't require a gem on rubygems.org
|
15
|
+
module Plugins
|
16
|
+
KNOWN_PLUGINS = [
|
17
|
+
'email',
|
18
|
+
'foursquare',
|
19
|
+
'github',
|
20
|
+
'imessage',
|
21
|
+
'pinboard',
|
22
|
+
'safari',
|
23
|
+
'shell',
|
24
|
+
'spotify',
|
25
|
+
'zulip'
|
26
|
+
].freeze
|
27
|
+
public_constant :KNOWN_PLUGINS
|
28
|
+
|
29
|
+
# Start of a system for having non-gem plugins. Right now, we just
|
30
|
+
# make registry aware of existence of name of non-gem plugin
|
31
|
+
def self.register_standalone(name:)
|
32
|
+
plugin = Chronicle::ETL::Registry::PluginRegistration.new do |p|
|
33
|
+
p.name = name
|
34
|
+
p.installed = true
|
35
|
+
end
|
36
|
+
|
37
|
+
installed_standalone << plugin
|
38
|
+
end
|
39
|
+
|
40
|
+
# Plugins either installed as gems or manually loaded/registered
|
41
|
+
def self.installed
|
42
|
+
installed_standalone + installed_as_gem
|
43
|
+
end
|
44
|
+
|
45
|
+
# Check whether a given plugin is installed
|
46
|
+
def self.installed?(name)
|
47
|
+
installed.map(&:name).include?(name)
|
48
|
+
end
|
49
|
+
|
50
|
+
# List of plugins installed as standalone
|
51
|
+
def self.installed_standalone
|
52
|
+
@standalones ||= []
|
53
|
+
end
|
54
|
+
|
55
|
+
# List of plugins installed as gems
|
56
|
+
def self.installed_as_gem
|
57
|
+
installed_gemspecs_latest.map do |gem|
|
58
|
+
Chronicle::ETL::Registry::PluginRegistration.new do |p|
|
59
|
+
p.name = gem.name.sub("chronicle-", "")
|
60
|
+
p.gem = gem.name
|
61
|
+
p.description = gem.description
|
62
|
+
p.version = gem.version.to_s
|
63
|
+
p.installed = true
|
64
|
+
end
|
65
|
+
end
|
66
|
+
end
|
67
|
+
|
68
|
+
# List of all plugins available to chronicle-etl
|
69
|
+
def self.available
|
70
|
+
available_as_gem
|
71
|
+
end
|
72
|
+
|
73
|
+
# List of plugins available through rubygems
|
74
|
+
# TODO: make this concurrent
|
75
|
+
def self.available_as_gem
|
76
|
+
KNOWN_PLUGINS.map do |name|
|
77
|
+
info = gem_info(name)
|
78
|
+
Chronicle::ETL::Registry::PluginRegistration.new do |p|
|
79
|
+
p.name = name
|
80
|
+
p.gem = info['name']
|
81
|
+
p.version = info['version']
|
82
|
+
p.description = info['info']
|
83
|
+
end
|
84
|
+
end
|
85
|
+
end
|
86
|
+
|
87
|
+
# Load info about a gem plugin from rubygems API
|
88
|
+
def self.gem_info(name)
|
89
|
+
gem_name = "chronicle-#{name}"
|
90
|
+
Gems.info(gem_name)
|
91
|
+
end
|
92
|
+
|
93
|
+
# Union of installed gems (latest version) + available gems
|
94
|
+
def self.all
|
95
|
+
(installed + available)
|
96
|
+
.group_by(&:name)
|
97
|
+
.transform_values { |plugin| plugin.find(&:installed) || plugin.first }
|
98
|
+
.values
|
99
|
+
end
|
100
|
+
|
101
|
+
# Does a plugin with a given name exist?
|
102
|
+
def self.exists?(name)
|
103
|
+
KNOWN_PLUGINS.include?(name)
|
104
|
+
end
|
105
|
+
|
106
|
+
# All versions of all plugins currently installed
|
107
|
+
def self.installed_gemspecs
|
108
|
+
# TODO: add check for chronicle-etl dependency
|
109
|
+
Gem::Specification.filter { |s| s.name.match(/^chronicle-/) && s.name != "chronicle-etl" }
|
110
|
+
end
|
111
|
+
|
112
|
+
# Latest version of each installed plugin
|
113
|
+
def self.installed_gemspecs_latest
|
114
|
+
installed_gemspecs.group_by(&:name)
|
115
|
+
.transform_values { |versions| versions.sort_by(&:version).reverse.first }
|
116
|
+
.values
|
117
|
+
end
|
118
|
+
|
119
|
+
# Activate a plugin with given name by `require`ing it
|
120
|
+
def self.activate(name)
|
121
|
+
# By default, activates the latest available version of a gem
|
122
|
+
# so don't have to run Kernel#gem separately
|
123
|
+
require "chronicle/#{name}"
|
124
|
+
rescue Gem::ConflictError => e
|
125
|
+
# TODO: figure out if there's more we can do here
|
126
|
+
raise Chronicle::ETL::PluginConflictError.new(name), "Plugin '#{name}' couldn't be loaded. #{e.message}"
|
127
|
+
rescue StandardError, LoadError => e
|
128
|
+
# StandardError to catch random non-loading problems that might occur
|
129
|
+
# when requiring the plugin (eg class macro invoked the wrong way)
|
130
|
+
# TODO: decide if this should be separated
|
131
|
+
raise Chronicle::ETL::PluginLoadError.new(name), "Plugin '#{name}' couldn't be loaded"
|
132
|
+
end
|
133
|
+
|
134
|
+
# Install a plugin to local gems
|
135
|
+
def self.install(name)
|
136
|
+
return if installed?(name)
|
137
|
+
raise(Chronicle::ETL::PluginNotAvailableError.new(name), "Plugin #{name} doesn't exist") unless exists?(name)
|
138
|
+
|
139
|
+
gem_name = "chronicle-#{name}"
|
140
|
+
|
141
|
+
Gem::DefaultUserInteraction.ui = Gem::SilentUI.new
|
142
|
+
Gem.install(gem_name)
|
143
|
+
|
144
|
+
activate(name)
|
145
|
+
rescue Gem::UnsatisfiableDependencyError
|
146
|
+
# TODO: we need to catch a lot more than this here
|
147
|
+
raise Chronicle::ETL::PluginNotAvailableError.new(name), "Plugin #{name} could not be installed."
|
148
|
+
end
|
149
|
+
|
150
|
+
# Uninstall a plugin
|
151
|
+
def self.uninstall(name)
|
152
|
+
gem_name = "chronicle-#{name}"
|
153
|
+
Gem::DefaultUserInteraction.ui = Gem::SilentUI.new
|
154
|
+
uninstaller = Gem::Uninstaller.new(gem_name)
|
155
|
+
uninstaller.uninstall
|
156
|
+
rescue Gem::InstallError
|
157
|
+
# TODO: strengthen this exception handling
|
158
|
+
raise(Chronicle::ETL::PluginError.new(name), "Plugin #{name} wasn't uninstalled")
|
159
|
+
end
|
160
|
+
end
|
161
|
+
end
|
162
|
+
end
|
163
|
+
end
|
@@ -1,61 +1,12 @@
|
|
1
|
-
require 'rubygems'
|
2
|
-
|
3
1
|
module Chronicle
|
4
2
|
module ETL
|
5
|
-
# A singleton class that acts as a registry of connector classes available for ETL jobs
|
6
3
|
module Registry
|
7
|
-
PHASES = [:extractor, :transformer, :loader]
|
8
|
-
|
9
|
-
class << self
|
10
|
-
attr_accessor :connectors
|
11
|
-
|
12
|
-
def register(connector)
|
13
|
-
connectors << connector
|
14
|
-
end
|
15
|
-
|
16
|
-
def connectors
|
17
|
-
@connectors ||= []
|
18
|
-
end
|
19
|
-
|
20
|
-
# Find connector from amongst those currently loaded
|
21
|
-
def find_by_phase_and_identifier_local(phase, identifier)
|
22
|
-
connector = connectors.find { |c| c.phase == phase && c.identifier == identifier }
|
23
|
-
end
|
24
|
-
|
25
|
-
# Find connector and load relevant plugin to find it if necessary
|
26
|
-
def find_by_phase_and_identifier(phase, identifier)
|
27
|
-
connector = find_by_phase_and_identifier_local(phase, identifier)
|
28
|
-
return connector if connector
|
29
|
-
|
30
|
-
# if not available in built-in connectors, try to activate a
|
31
|
-
# relevant plugin and try again
|
32
|
-
if identifier.include?(":")
|
33
|
-
plugin, name = identifier.split(":")
|
34
|
-
else
|
35
|
-
# This case handles the case where the identifier is a
|
36
|
-
# shorthand (ie `imessage`) because there's only one default
|
37
|
-
# connector.
|
38
|
-
plugin = identifier
|
39
|
-
end
|
40
|
-
|
41
|
-
raise(Chronicle::ETL::PluginNotInstalledError.new(plugin)) unless PluginRegistry.installed?(plugin)
|
42
|
-
|
43
|
-
PluginRegistry.activate(plugin)
|
44
|
-
|
45
|
-
candidates = connectors.select { |c| c.phase == phase && c.plugin == plugin }
|
46
|
-
# if no name given, just use first connector with right phase/plugin
|
47
|
-
# TODO: set up a property for connectors to specify that they're the
|
48
|
-
# default connector for the plugin
|
49
|
-
candidates = candidates.select { |c| c.identifier == name } if name
|
50
|
-
connector = candidates.first
|
51
|
-
|
52
|
-
connector || raise(ConnectorNotAvailableError, "Connector '#{identifier}' not found")
|
53
|
-
end
|
54
|
-
end
|
55
4
|
end
|
56
5
|
end
|
57
6
|
end
|
58
7
|
|
59
8
|
require_relative 'self_registering'
|
60
9
|
require_relative 'connector_registration'
|
61
|
-
require_relative '
|
10
|
+
require_relative 'connectors'
|
11
|
+
require_relative 'plugin_registration'
|
12
|
+
require_relative 'plugins'
|
@@ -17,7 +17,7 @@ module Chronicle
|
|
17
17
|
def register_connector
|
18
18
|
@connector_registration ||= ::Chronicle::ETL::Registry::ConnectorRegistration.new(self)
|
19
19
|
yield @connector_registration if block_given?
|
20
|
-
::Chronicle::ETL::Registry.register(@connector_registration)
|
20
|
+
::Chronicle::ETL::Registry::Connectors.register(@connector_registration)
|
21
21
|
end
|
22
22
|
end
|
23
23
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: chronicle-etl
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.5.
|
4
|
+
version: 0.5.5
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Andrew Louis
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2022-
|
11
|
+
date: 2022-05-19 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activesupport
|
@@ -52,6 +52,20 @@ dependencies:
|
|
52
52
|
- - "~>"
|
53
53
|
- !ruby/object:Gem::Version
|
54
54
|
version: 0.8.1
|
55
|
+
- !ruby/object:Gem::Dependency
|
56
|
+
name: gems
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - ">="
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '1'
|
62
|
+
type: :runtime
|
63
|
+
prerelease: false
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - ">="
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: '1'
|
55
69
|
- !ruby/object:Gem::Dependency
|
56
70
|
name: launchy
|
57
71
|
requirement: !ruby/object:Gem::Requirement
|
@@ -280,16 +294,16 @@ dependencies:
|
|
280
294
|
name: fakefs
|
281
295
|
requirement: !ruby/object:Gem::Requirement
|
282
296
|
requirements:
|
283
|
-
- - "
|
297
|
+
- - "~>"
|
284
298
|
- !ruby/object:Gem::Version
|
285
|
-
version: '
|
299
|
+
version: '1.4'
|
286
300
|
type: :development
|
287
301
|
prerelease: false
|
288
302
|
version_requirements: !ruby/object:Gem::Requirement
|
289
303
|
requirements:
|
290
|
-
- - "
|
304
|
+
- - "~>"
|
291
305
|
- !ruby/object:Gem::Version
|
292
|
-
version: '
|
306
|
+
version: '1.4'
|
293
307
|
- !ruby/object:Gem::Dependency
|
294
308
|
name: guard-rspec
|
295
309
|
requirement: !ruby/object:Gem::Requirement
|
@@ -374,6 +388,34 @@ dependencies:
|
|
374
388
|
- - "~>"
|
375
389
|
- !ruby/object:Gem::Version
|
376
390
|
version: '0.21'
|
391
|
+
- !ruby/object:Gem::Dependency
|
392
|
+
name: vcr
|
393
|
+
requirement: !ruby/object:Gem::Requirement
|
394
|
+
requirements:
|
395
|
+
- - "~>"
|
396
|
+
- !ruby/object:Gem::Version
|
397
|
+
version: '6.1'
|
398
|
+
type: :development
|
399
|
+
prerelease: false
|
400
|
+
version_requirements: !ruby/object:Gem::Requirement
|
401
|
+
requirements:
|
402
|
+
- - "~>"
|
403
|
+
- !ruby/object:Gem::Version
|
404
|
+
version: '6.1'
|
405
|
+
- !ruby/object:Gem::Dependency
|
406
|
+
name: webmock
|
407
|
+
requirement: !ruby/object:Gem::Requirement
|
408
|
+
requirements:
|
409
|
+
- - "~>"
|
410
|
+
- !ruby/object:Gem::Version
|
411
|
+
version: '3'
|
412
|
+
type: :development
|
413
|
+
prerelease: false
|
414
|
+
version_requirements: !ruby/object:Gem::Requirement
|
415
|
+
requirements:
|
416
|
+
- - "~>"
|
417
|
+
- !ruby/object:Gem::Version
|
418
|
+
version: '3'
|
377
419
|
- !ruby/object:Gem::Dependency
|
378
420
|
name: yard
|
379
421
|
requirement: !ruby/object:Gem::Requirement
|
@@ -454,7 +496,9 @@ files:
|
|
454
496
|
- lib/chronicle/etl/models/raw.rb
|
455
497
|
- lib/chronicle/etl/oauth_authorizer.rb
|
456
498
|
- lib/chronicle/etl/registry/connector_registration.rb
|
457
|
-
- lib/chronicle/etl/registry/
|
499
|
+
- lib/chronicle/etl/registry/connectors.rb
|
500
|
+
- lib/chronicle/etl/registry/plugin_registration.rb
|
501
|
+
- lib/chronicle/etl/registry/plugins.rb
|
458
502
|
- lib/chronicle/etl/registry/registry.rb
|
459
503
|
- lib/chronicle/etl/registry/self_registering.rb
|
460
504
|
- lib/chronicle/etl/runner.rb
|
@@ -1,95 +0,0 @@
|
|
1
|
-
require 'rubygems'
|
2
|
-
require 'rubygems/command'
|
3
|
-
require 'rubygems/commands/install_command'
|
4
|
-
require 'rubygems/uninstaller'
|
5
|
-
|
6
|
-
module Chronicle
|
7
|
-
module ETL
|
8
|
-
module Registry
|
9
|
-
# Responsible for managing plugins available to chronicle-etl
|
10
|
-
#
|
11
|
-
# @todo Better validation for whether a gem is actually a plugin
|
12
|
-
# @todo Add ways to load a plugin that don't require a gem on rubygems.org
|
13
|
-
module PluginRegistry
|
14
|
-
class << self
|
15
|
-
# Start of a system for having non-gem plugins. Right now, we just
|
16
|
-
# make registry aware of existenc of name of non-gem plugin
|
17
|
-
def register_standalone(name)
|
18
|
-
standalones << name
|
19
|
-
end
|
20
|
-
|
21
|
-
def standalones
|
22
|
-
@standalones ||= []
|
23
|
-
end
|
24
|
-
end
|
25
|
-
|
26
|
-
# Does this plugin exist?
|
27
|
-
def self.exists?(name)
|
28
|
-
# TODO: implement this. Could query rubygems.org or use a hardcoded
|
29
|
-
# list somewhere
|
30
|
-
true
|
31
|
-
end
|
32
|
-
|
33
|
-
# All versions of all plugins currently installed
|
34
|
-
def self.all_installed
|
35
|
-
# TODO: add check for chronicle-etl dependency
|
36
|
-
Gem::Specification.filter { |s| s.name.match(/^chronicle-/) && s.name != "chronicle-etl" }
|
37
|
-
end
|
38
|
-
|
39
|
-
# Latest version of each installed plugin
|
40
|
-
def self.all_installed_latest
|
41
|
-
all_installed.group_by(&:name)
|
42
|
-
.transform_values { |versions| versions.sort_by(&:version).reverse.first }
|
43
|
-
.values
|
44
|
-
end
|
45
|
-
|
46
|
-
# Check whether a given plugin is installed
|
47
|
-
def self.installed?(name)
|
48
|
-
(standalones + all_installed.map { |gem| gem.name.gsub("chronicle-", "") }).include?(name)
|
49
|
-
end
|
50
|
-
|
51
|
-
# Activate a plugin with given name by `require`ing it
|
52
|
-
def self.activate(name)
|
53
|
-
# By default, activates the latest available version of a gem
|
54
|
-
# so don't have to run Kernel#gem separately
|
55
|
-
require "chronicle/#{name}"
|
56
|
-
rescue Gem::ConflictError => e
|
57
|
-
# TODO: figure out if there's more we can do here
|
58
|
-
raise Chronicle::ETL::PluginConflictError.new(name), "Plugin '#{name}' couldn't be loaded. #{e.message}"
|
59
|
-
rescue StandardError, LoadError => e
|
60
|
-
# StandardError to catch random non-loading problems that might occur
|
61
|
-
# when requiring the plugin (eg class macro invoked the wrong way)
|
62
|
-
# TODO: decide if this should be separated
|
63
|
-
raise Chronicle::ETL::PluginLoadError.new(name), "Plugin '#{name}' couldn't be loaded"
|
64
|
-
end
|
65
|
-
|
66
|
-
# Install a plugin to local gems
|
67
|
-
def self.install(name)
|
68
|
-
return if installed?(name)
|
69
|
-
|
70
|
-
gem_name = "chronicle-#{name}"
|
71
|
-
raise(Chronicle::ETL::PluginNotAvailableError.new(gem_name), "Plugin #{name} doesn't exist") unless exists?(gem_name)
|
72
|
-
|
73
|
-
Gem::DefaultUserInteraction.ui = Gem::SilentUI.new
|
74
|
-
Gem.install(gem_name)
|
75
|
-
|
76
|
-
activate(name)
|
77
|
-
rescue Gem::UnsatisfiableDependencyError
|
78
|
-
# TODO: we need to catch a lot more than this here
|
79
|
-
raise Chronicle::ETL::PluginNotAvailableError.new(name), "Plugin #{name} could not be installed."
|
80
|
-
end
|
81
|
-
|
82
|
-
# Uninstall a plugin
|
83
|
-
def self.uninstall(name)
|
84
|
-
gem_name = "chronicle-#{name}"
|
85
|
-
Gem::DefaultUserInteraction.ui = Gem::SilentUI.new
|
86
|
-
uninstaller = Gem::Uninstaller.new(gem_name)
|
87
|
-
uninstaller.uninstall
|
88
|
-
rescue Gem::InstallError
|
89
|
-
# TODO: strengthen this exception handling
|
90
|
-
raise(Chronicle::ETL::PluginError.new(name), "Plugin #{name} wasn't uninstalled")
|
91
|
-
end
|
92
|
-
end
|
93
|
-
end
|
94
|
-
end
|
95
|
-
end
|