chronicle-etl 0.5.4 → 0.5.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +55 -37
- data/chronicle-etl.gemspec +7 -4
- data/lib/chronicle/etl/cli/authorizations.rb +2 -2
- data/lib/chronicle/etl/cli/connectors.rb +2 -2
- data/lib/chronicle/etl/cli/plugins.rb +19 -15
- data/lib/chronicle/etl/config.rb +3 -0
- data/lib/chronicle/etl/configurable.rb +4 -0
- data/lib/chronicle/etl/job_definition.rb +4 -4
- data/lib/chronicle/etl/logger.rb +1 -1
- data/lib/chronicle/etl/oauth_authorizer.rb +0 -2
- data/lib/chronicle/etl/registry/connectors.rb +60 -0
- data/lib/chronicle/etl/registry/plugin_registration.rb +19 -0
- data/lib/chronicle/etl/registry/plugins.rb +163 -0
- data/lib/chronicle/etl/registry/registry.rb +3 -52
- data/lib/chronicle/etl/registry/self_registering.rb +1 -1
- data/lib/chronicle/etl/secrets.rb +1 -1
- data/lib/chronicle/etl/version.rb +1 -1
- metadata +51 -7
- data/lib/chronicle/etl/registry/plugin_registry.rb +0 -95
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: a2de46efc3c5fbdc7ac120137bef56e13a138c8a95c8dd7d0a3542a65be65959
|
4
|
+
data.tar.gz: e8e3e9ae236e270b2926037419d5349170f85b8597c640c0b7b899552257fdb9
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 58b293d45d1a7f4589aee080d0fb1348e45d76753a8d48bb33e694d6a2b6ea123d1495a12a42e98ed6d3e65926ad48bd7de1fb98dac9bc7bac9225ef00d32fb3
|
7
|
+
data.tar.gz: e73c73b67b4e3790347da3df22ed7a11357fdf43e91b734925503f8d0c189aabd07e26f51ed25cf1409fd68ba74dd72e1e7e01c113f773832c509172f0b4ee84
|
data/README.md
CHANGED
@@ -12,7 +12,7 @@ If you don’t want to spend all your time writing scrapers, reverse-engineering
|
|
12
12
|
|
13
13
|
## What does `chronicle-etl` give you?
|
14
14
|
* **A CLI tool for working with personal data**. You can monitor progress of exports, manipulate the output, set up recurring jobs, manage credentials, and more.
|
15
|
-
* **Plugins for many third-party providers
|
15
|
+
* **Plugins for many third-party providers** (see [list](#available-plugins-and-connectors)). This plugin system allows you to access data from dozens of third-party services, all accessible through a common CLI interface.
|
16
16
|
* **A common, opinionated schema**: You can normalize different datasets into a single schema so that, for example, all your iMessages and emails are represented in a common schema. (Don’t want to use this schema? `chronicle-etl` always allows you to fall back on working with the raw extraction data.)
|
17
17
|
|
18
18
|
## Chronicle-ETL in action
|
@@ -51,6 +51,10 @@ $ chronicle-etl --extractor NAME --transformer NAME --loader NAME
|
|
51
51
|
# Read test.csv and display it to stdout as a table
|
52
52
|
$ chronicle-etl --extractor csv --input data.csv --loader table
|
53
53
|
|
54
|
+
# Show available plugins and install one
|
55
|
+
$ chronicle-etl plugins:list
|
56
|
+
$ chronicle-etl plugins:install shell
|
57
|
+
|
54
58
|
# Retrieve shell commands run in the last 5 hours
|
55
59
|
$ chronicle-etl -e shell --since 5h
|
56
60
|
|
@@ -86,35 +90,36 @@ Options:
|
|
86
90
|
[--silent], [--no-silent] # Silence all output
|
87
91
|
```
|
88
92
|
|
89
|
-
### Saving
|
90
|
-
|
93
|
+
### Saving a job
|
91
94
|
You can save details about a job to a local config file (saved by default in `~/.config/chronicle/etl/jobs/JOB_NAME.yml`) to save yourself the trouble specifying options each time.
|
92
95
|
|
93
96
|
```sh
|
94
97
|
# Save a job named 'sample' to ~/.config/chronicle/etl/jobs/sample.yml
|
95
98
|
$ chronicle-etl jobs:save sample --extractor pinboard --since 10d
|
96
99
|
|
97
|
-
# Show details about the job
|
98
|
-
$ chronicle-etl jobs:show sample
|
99
|
-
|
100
100
|
# Run the job
|
101
101
|
$ chronicle-etl jobs:run sample
|
102
102
|
|
103
|
+
# Show details about the job
|
104
|
+
$ chronicle-etl jobs:show sample
|
105
|
+
|
103
106
|
# Show all saved jobs
|
104
107
|
$ chronicle-etl jobs:list
|
105
108
|
```
|
106
109
|
|
107
|
-
## Connectors
|
108
|
-
|
110
|
+
## Connectors and plugins
|
111
|
+
|
112
|
+
Connectors let you work with different data formats or third-party providers.
|
113
|
+
|
114
|
+
### Built-in Connectors
|
115
|
+
|
116
|
+
`chronicle-etl` comes with several built-in connectors for common formats and sources.
|
109
117
|
|
110
118
|
```sh
|
111
119
|
# List all available connectors
|
112
120
|
$ chronicle-etl connectors:list
|
113
121
|
```
|
114
122
|
|
115
|
-
### Built-in Connectors
|
116
|
-
`chronicle-etl` comes with several built-in connectors for common formats and sources.
|
117
|
-
|
118
123
|
#### Extractors
|
119
124
|
- [`csv`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/extractors/csv_extractor.rb) - Load records from CSV files or stdin
|
120
125
|
- [`json`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/extractors/json_extractor.rb) - Load JSON (either [line-separated objects](https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON) or one object)
|
@@ -129,18 +134,19 @@ $ chronicle-etl connectors:list
|
|
129
134
|
- [`json`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/loaders/json_loader.rb) - Load records serialized as JSON
|
130
135
|
- [`rest`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/loaders/rest_loader.rb) - Serialize records with [JSONAPI](https://jsonapi.org/) and send to a REST API
|
131
136
|
|
132
|
-
|
133
|
-
|
137
|
+
### Chronicle Plugins for third-party services
|
138
|
+
|
139
|
+
Plugins provide access to data from third-party platforms, services, or formats. Plugins are packaged as separate gems and can be installed through the CLI (under the hood, it's a `gem install chronicle-PLUGINNAME`)
|
134
140
|
|
135
|
-
|
141
|
+
#### Plugin usage
|
136
142
|
|
137
143
|
```bash
|
144
|
+
# List available plugins
|
145
|
+
$ chronicle-etl plugins:list
|
146
|
+
|
138
147
|
# Install a plugin
|
139
148
|
$ chronicle-etl plugins:install NAME
|
140
149
|
|
141
|
-
# List installed plugins
|
142
|
-
$ chronicle-etl plugins:list
|
143
|
-
|
144
150
|
# Use a plugin
|
145
151
|
$ chronicle-etl plugins:install shell
|
146
152
|
$ chronicle-etl --extractor shell:history --limit 10
|
@@ -148,33 +154,45 @@ $ chronicle-etl --extractor shell:history --limit 10
|
|
148
154
|
# Uninstall a plugin
|
149
155
|
$ chronicle-etl plugins:uninstall NAME
|
150
156
|
```
|
151
|
-
|
152
|
-
|
157
|
+
#### Available plugins and connectors
|
158
|
+
|
159
|
+
The following are the officially-supported list of plugins and their available connectors:
|
160
|
+
|
161
|
+
| Plugin | Type | Identifier | Description | Description |
|
162
|
+
|---------------------------------------------------------------------|-------------|------------------|-------------------------------------------|-------------------------------------------|
|
163
|
+
| [email](https://github.com/chronicle-app/chronicle-email) | extractor | imap | emails over an IMAP connection | emails over an IMAP connection |
|
164
|
+
| [email](https://github.com/chronicle-app/chronicle-email) | extractor | mbox | emails from an .mbox file | emails from an .mbox file |
|
165
|
+
| [email](https://github.com/chronicle-app/chronicle-email) | transformer | email | email to Chronicle Schema | email to Chronicle Schema |
|
166
|
+
| [foursquare](https://github.com/chronicle-app/chronicle-foursquare) | extractor | checkins | Foursqure visits | Foursqure visits |
|
167
|
+
| [foursquare](https://github.com/chronicle-app/chronicle-foursquare) | transformer | checkin | checkin to Chronicle Schema | checkin to Chronicle Schema |
|
168
|
+
| [github](https://github.com/chronicle-app/chronicle-github) | extractor | activity | user activity stream | user activity stream |
|
169
|
+
| [imessage](https://github.com/chronicle-app/chronicle-imessage) | extractor | messages | imessages from local macOS | imessages from local macOS |
|
170
|
+
| [imessage](https://github.com/chronicle-app/chronicle-imessage) | transformer | message | imessage to Chronicle Schema | imessage to Chronicle Schema |
|
171
|
+
| [pinboard](https://github.com/chronicle-app/chronicle-pinboard) | extractor | bookmarks | Pinboard.in bookmarks | Pinboard.in bookmarks |
|
172
|
+
| [pinboard](https://github.com/chronicle-app/chronicle-pinboard) | transformer | bookmark | bookmark to Chronicle Schema | bookmark to Chronicle Schema |
|
173
|
+
| [safari](https://github.com/chronicle-app/chronicle-safari) | extractor | browser-history | browser history | browser history |
|
174
|
+
| [safari ](https://github.com/chronicle-app/chronicle-safari ) | transformer | browser-history | browser history to Chronicle Schema | browser history to Chronicle Schema |
|
175
|
+
| [shell](https://github.com/chronicle-app/chronicle-shell) | extractor | history | shell command history (bash / zsh) | shell command history (bash / zsh) |
|
176
|
+
| [shell](https://github.com/chronicle-app/chronicle-shell) | transformer | command | command to Chronicle Schema | command to Chronicle Schema |
|
177
|
+
| [spotify](https://github.com/chronicle-app/chronicle-spotify) | extractor | liked-tracks | liked tracks | liked tracks |
|
178
|
+
| [spotify](https://github.com/chronicle-app/chronicle-spotify) | extractor | saved-albums | saved albums | saved albums |
|
179
|
+
| [spotify](https://github.com/chronicle-app/chronicle-spotify) | extractor | listens | recently listened tracks (last 50 tracks) | recently listened tracks (last 50 tracks) |
|
180
|
+
| [spotify](https://github.com/chronicle-app/chronicle-spotify) | transformer | like | like to Chronicle Schema | like to Chronicle Schema |
|
181
|
+
| [spotify](https://github.com/chronicle-app/chronicle-spotify) | transformer | listen | listen to Chronicle Schema | listen to Chronicle Schema |
|
182
|
+
| [spotify](https://github.com/chronicle-app/chronicle-spotify) | authorizer | | OAuth authorizer | OAuth authorizer |
|
183
|
+
| [zulip](https://github.com/chronicle-app/chronicle-zulip) | extractor | private-messages | private messages | private messages |
|
184
|
+
| [zulip](https://github.com/chronicle-app/chronicle-zulip) | transformer | message | message to Chronicle Schema | message to Chronicle Schema |
|
185
|
+
|
186
|
+
|
187
|
+
### Coming soon
|
153
188
|
|
154
189
|
A few dozen importers exist [in my Memex project](https://hyfen.net/memex/) and I'm porting them over to the Chronicle system. The [Chronicle Plugin Tracker](https://github.com/orgs/chronicle-app/projects/1/views/1) lets you keep track what's available and what's coming soon.
|
155
190
|
|
156
191
|
If you don't see a plugin for a third-party provider or data source that you're interested in using with `chronicle-etl`, [please open an issue](https://github.com/chronicle-app/chronicle-etl/issues/new). If you want to work together on a plugin, please [get in touch](#get-in-touch)!
|
157
192
|
|
158
|
-
#### Currently available
|
159
|
-
|
160
|
-
| Name | Description | Availability |
|
161
|
-
|-----------------------------------------------------------------|---------------------------------------------------------------------------------------------|----------------------------------|
|
162
|
-
| [email](https://github.com/chronicle-app/chronicle-email) | Emails and attachments from IMAP or .mbox files | Available |
|
163
|
-
| [github](https://github.com/chronicle-app/chronicle-github) | Github activity stream | Available |
|
164
|
-
| [imessage](https://github.com/chronicle-app/chronicle-imessage) | iMessage messages and attachments | Available |
|
165
|
-
| [pinboard](https://github.com/chronicle-app/chronicle-email) | Bookmarks and tags | Available |
|
166
|
-
| [safari](https://github.com/chronicle-app/chronicle-safari) | Browser history from local sqlite db | Available |
|
167
|
-
| [shell](https://github.com/chronicle-app/chronicle-shell) | Shell command history | Available (still needs zsh support) |
|
168
|
-
| [zulip](https://github.com/chronicle-app/chronicle-zulip) | Zulip message history | Available (for private messages) |
|
169
|
-
|
170
|
-
|
171
|
-
#### Coming soon
|
172
|
-
|
173
193
|
In summary, the following **are coming soon**:
|
174
194
|
anki, arc, bear, chrome, facebook, firefox, fitbit, foursquare, git, github, goodreads, google-calendar, images, instagram, lastfm, shazam, slack, strava, things, twitter, whatsapp, youtube.
|
175
195
|
|
176
|
-
Please check the [Chronicle Plugin Tracker](https://github.com/orgs/chronicle-app/projects/1/views/1) for details.
|
177
|
-
|
178
196
|
### Writing your own plugin
|
179
197
|
|
180
198
|
Additional connectors are packaged as separate ruby gems. You can view the [iMessage plugin](https://github.com/chronicle-app/chronicle-imessage) for an example.
|
@@ -209,6 +227,7 @@ module Chronicle
|
|
209
227
|
end
|
210
228
|
```
|
211
229
|
|
230
|
+
|
212
231
|
## Secrets Management
|
213
232
|
|
214
233
|
If your job needs secrets such as access tokens or passwords, `chronicle-etl` has a built-in secret management system.
|
@@ -243,7 +262,6 @@ $ chronicle-etl secrets:unset pinboard access_token
|
|
243
262
|
## Roadmap
|
244
263
|
|
245
264
|
- Keep tackling **new plugins**. See: [Chronicle Plugin Tracker](https://github.com/orgs/chronicle-app/projects/1)
|
246
|
-
- Add an **OAuth2 authorizer** for services that require this type of authorization ([#48](https://github.com/chronicle-app/chronicle-etl/issues/48))
|
247
265
|
- Add support for **incremental extractions** ([#37](https://github.com/chronicle-app/chronicle-etl/issues/37))
|
248
266
|
- **Improve stdin extractor and shell command transformer** so that users can easily integrate their own scripts/languages/tools into jobs ([#5](https://github.com/chronicle-app/chronicle-etl/issues/48))
|
249
267
|
- **Add documentation for Chronicle Schema**. It's found throughout this project but never explained.
|
data/chronicle-etl.gemspec
CHANGED
@@ -40,13 +40,14 @@ Gem::Specification.new do |spec|
|
|
40
40
|
spec.add_dependency "activesupport", "~> 7.0"
|
41
41
|
spec.add_dependency "chronic_duration", "~> 0.10.6"
|
42
42
|
spec.add_dependency "colorize", "~> 0.8.1"
|
43
|
-
spec.add_dependency
|
43
|
+
spec.add_dependency "gems", ">= 1"
|
44
|
+
spec.add_dependency "launchy"
|
44
45
|
spec.add_dependency "marcel", "~> 1.0.2"
|
45
46
|
spec.add_dependency "mini_exiftool", "~> 2.10"
|
46
47
|
spec.add_dependency "nokogiri", "~> 1.13"
|
47
|
-
spec.add_dependency
|
48
|
+
spec.add_dependency "omniauth", "~> 2"
|
48
49
|
spec.add_dependency "sequel", "~> 5.35"
|
49
|
-
spec.add_dependency
|
50
|
+
spec.add_dependency "sinatra", "~> 2"
|
50
51
|
spec.add_dependency "sqlite3", "~> 1.4"
|
51
52
|
spec.add_dependency "thor", "~> 1.2"
|
52
53
|
spec.add_dependency "thor-hollaback", "~> 0.2"
|
@@ -57,12 +58,14 @@ Gem::Specification.new do |spec|
|
|
57
58
|
spec.add_dependency "xdg", ">= 4.0"
|
58
59
|
|
59
60
|
spec.add_development_dependency "bundler", "~> 2.1"
|
60
|
-
spec.add_development_dependency "fakefs"
|
61
|
+
spec.add_development_dependency "fakefs", "~> 1.4"
|
61
62
|
spec.add_development_dependency "guard-rspec", "~> 4.7.3"
|
62
63
|
spec.add_development_dependency "pry-byebug", "~> 3.9"
|
63
64
|
spec.add_development_dependency "rake", "~> 13.0"
|
64
65
|
spec.add_development_dependency "rspec", "~> 3.9"
|
65
66
|
spec.add_development_dependency "rubocop", "~> 1.25.1"
|
66
67
|
spec.add_development_dependency "simplecov", "~> 0.21"
|
68
|
+
spec.add_development_dependency "vcr", "~> 6.1"
|
69
|
+
spec.add_development_dependency "webmock", "~> 3"
|
67
70
|
spec.add_development_dependency "yard", "~> 0.9.7"
|
68
71
|
end
|
@@ -37,12 +37,12 @@ module Chronicle
|
|
37
37
|
|
38
38
|
def find_authorizer_klass(provider)
|
39
39
|
# TODO: this assumes provider:plugin one-to-one
|
40
|
-
unless Chronicle::ETL::Registry::
|
40
|
+
unless Chronicle::ETL::Registry::Plugins.installed?(provider)
|
41
41
|
cli_fail(message: "Plugin for #{provider} is not installed.")
|
42
42
|
end
|
43
43
|
|
44
44
|
begin
|
45
|
-
Chronicle::ETL::Registry::
|
45
|
+
Chronicle::ETL::Registry::Plugins.activate(provider)
|
46
46
|
rescue PluginError => e
|
47
47
|
cli_fail(message: "Could not load plugin '#{provider}'.\n" + e.message, exception: e)
|
48
48
|
end
|
@@ -13,7 +13,7 @@ module Chronicle
|
|
13
13
|
desc "list", "Lists available connectors"
|
14
14
|
# Display all available connectors that chronicle-etl has access to
|
15
15
|
def list
|
16
|
-
connector_info = Chronicle::ETL::Registry.connectors.map do |connector_registration|
|
16
|
+
connector_info = Chronicle::ETL::Registry::Connectors.connectors.map do |connector_registration|
|
17
17
|
{
|
18
18
|
identifier: connector_registration.identifier,
|
19
19
|
phase: connector_registration.phase,
|
@@ -43,7 +43,7 @@ module Chronicle
|
|
43
43
|
end
|
44
44
|
|
45
45
|
begin
|
46
|
-
connector = Chronicle::ETL::Registry.find_by_phase_and_identifier(phase.to_sym, identifier)
|
46
|
+
connector = Chronicle::ETL::Registry::Connectors.find_by_phase_and_identifier(phase.to_sym, identifier)
|
47
47
|
rescue Chronicle::ETL::ConnectorNotAvailableError, Chronicle::ETL::PluginError => e
|
48
48
|
cli_fail(message: "Could not find #{phase} #{identifier}", exception: e)
|
49
49
|
end
|
@@ -16,7 +16,7 @@ module Chronicle
|
|
16
16
|
cli_fail(message: "Please specify a plugin to install") unless plugins.any?
|
17
17
|
|
18
18
|
installed, not_installed = plugins.partition do |plugin|
|
19
|
-
Chronicle::ETL::Registry::
|
19
|
+
Chronicle::ETL::Registry::Plugins.installed?(plugin)
|
20
20
|
end
|
21
21
|
|
22
22
|
puts "Already installed: #{installed.join(", ")}" if installed.any?
|
@@ -27,7 +27,7 @@ module Chronicle
|
|
27
27
|
|
28
28
|
not_installed.each do |plugin|
|
29
29
|
spinner.update(title: "Installing #{plugin}")
|
30
|
-
Chronicle::ETL::Registry::
|
30
|
+
Chronicle::ETL::Registry::Plugins.install(plugin)
|
31
31
|
|
32
32
|
rescue Chronicle::ETL::PluginError => e
|
33
33
|
spinner.error("Error".red)
|
@@ -41,7 +41,7 @@ module Chronicle
|
|
41
41
|
def uninstall(name)
|
42
42
|
spinner = TTY::Spinner.new("[:spinner] Uninstalling plugin #{name}...", format: :dots_2)
|
43
43
|
spinner.auto_spin
|
44
|
-
Chronicle::ETL::Registry::
|
44
|
+
Chronicle::ETL::Registry::Plugins.uninstall(name)
|
45
45
|
spinner.success("(#{'successful'.green})")
|
46
46
|
rescue Chronicle::ETL::PluginError => e
|
47
47
|
spinner.error("Error".red)
|
@@ -51,20 +51,24 @@ module Chronicle
|
|
51
51
|
desc "list", "Lists available plugins"
|
52
52
|
# Display all available plugins that chronicle-etl has access to
|
53
53
|
def list
|
54
|
-
|
55
|
-
|
56
|
-
|
57
|
-
|
58
|
-
|
59
|
-
|
60
|
-
|
61
|
-
|
54
|
+
values = Chronicle::ETL::Registry::Plugins.all
|
55
|
+
.map do |plugin|
|
56
|
+
[
|
57
|
+
plugin.name,
|
58
|
+
plugin.description,
|
59
|
+
plugin.installed ? '✓' : '',
|
60
|
+
plugin.version
|
61
|
+
]
|
62
62
|
end
|
63
63
|
|
64
|
-
headers = ['name', 'description', 'version'].map{ |h| h.to_s.upcase.bold }
|
65
|
-
table = TTY::Table.new(headers,
|
66
|
-
puts "
|
67
|
-
puts table.render(
|
64
|
+
headers = ['name', 'description', 'installed', 'version'].map{ |h| h.to_s.upcase.bold }
|
65
|
+
table = TTY::Table.new(headers, values)
|
66
|
+
puts "Available plugins:"
|
67
|
+
puts table.render(
|
68
|
+
indent: 2,
|
69
|
+
padding: [0, 0],
|
70
|
+
alignments: [:left, :left, :center, :left]
|
71
|
+
)
|
68
72
|
end
|
69
73
|
end
|
70
74
|
end
|
data/lib/chronicle/etl/config.rb
CHANGED
@@ -1,3 +1,4 @@
|
|
1
|
+
require "active_support/core_ext/hash/keys"
|
1
2
|
require 'fileutils'
|
2
3
|
require 'yaml'
|
3
4
|
|
@@ -21,6 +22,8 @@ module Chronicle
|
|
21
22
|
def write(type, identifier, data)
|
22
23
|
base = config_pathname_for_type(type)
|
23
24
|
path = base.join("#{identifier}.yml")
|
25
|
+
|
26
|
+
data.deep_stringify_keys!
|
24
27
|
FileUtils.mkdir_p(File.dirname(path))
|
25
28
|
File.open(path, 'w', 0o600) do |f|
|
26
29
|
# Ruby likes to add --- separators when writing yaml files
|
@@ -108,6 +108,10 @@ module Chronicle
|
|
108
108
|
end
|
109
109
|
|
110
110
|
def coerce_time(value)
|
111
|
+
# parsing yml files might result in us getting Date objects
|
112
|
+
# we convert to DateTime first to to ensure UTC
|
113
|
+
return value.to_datetime.to_time if value.is_a?(Date)
|
114
|
+
|
111
115
|
return value unless value.is_a?(String)
|
112
116
|
|
113
117
|
# Hacky check for duration strings like "60m"
|
@@ -34,7 +34,7 @@ module Chronicle
|
|
34
34
|
def validate
|
35
35
|
@errors = {}
|
36
36
|
|
37
|
-
Chronicle::ETL::Registry::PHASES.each do |phase|
|
37
|
+
Chronicle::ETL::Registry::Connectors::PHASES.each do |phase|
|
38
38
|
__send__("#{phase}_klass".to_sym)
|
39
39
|
rescue Chronicle::ETL::PluginError => e
|
40
40
|
@errors[:plugins] ||= []
|
@@ -66,7 +66,7 @@ module Chronicle
|
|
66
66
|
|
67
67
|
# For each connector in this job, mix in secrets into the options
|
68
68
|
def apply_default_secrets
|
69
|
-
Chronicle::ETL::Registry::PHASES.each do |phase|
|
69
|
+
Chronicle::ETL::Registry::Connectors::PHASES.each do |phase|
|
70
70
|
# If the option have a `secrets` key, we look up those secrets and
|
71
71
|
# mix them in. If not, use the connector's plugin name and look up
|
72
72
|
# secrets with the same namespace
|
@@ -124,11 +124,11 @@ module Chronicle
|
|
124
124
|
private
|
125
125
|
|
126
126
|
def load_klass(phase, identifier)
|
127
|
-
Chronicle::ETL::Registry.find_by_phase_and_identifier(phase, identifier).klass
|
127
|
+
Chronicle::ETL::Registry::Connectors.find_by_phase_and_identifier(phase, identifier).klass
|
128
128
|
end
|
129
129
|
|
130
130
|
def load_credentials
|
131
|
-
Chronicle::ETL::Registry::PHASES.each do |phase|
|
131
|
+
Chronicle::ETL::Registry::Connectors::PHASES.each do |phase|
|
132
132
|
credentials_name = @definition[phase].dig(:options, :credentials)
|
133
133
|
if credentials_name
|
134
134
|
credentials = Chronicle::ETL::Config.load_credentials(credentials_name)
|
data/lib/chronicle/etl/logger.rb
CHANGED
@@ -50,7 +50,6 @@ module Chronicle
|
|
50
50
|
associate_oauth_credentials
|
51
51
|
@server = load_server
|
52
52
|
spinner = TTY::Spinner.new(":spinner :title", format: :dots_2)
|
53
|
-
Chronicle::ETL::Logger.attach_to_ui(spinner)
|
54
53
|
spinner.auto_spin
|
55
54
|
spinner.update(title: "Starting temporary authorization server on port #{@port}""")
|
56
55
|
|
@@ -63,7 +62,6 @@ module Chronicle
|
|
63
62
|
@server.quit!
|
64
63
|
server_thread.join
|
65
64
|
spinner.success("(#{'successful'.green})")
|
66
|
-
Chronicle::ETL::Logger.detach_from_ui
|
67
65
|
|
68
66
|
# TODO: properly handle failed authorizations
|
69
67
|
raise Chronicle::ETL::AuthorizationError unless @server.latest_authorization
|
@@ -0,0 +1,60 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
|
3
|
+
module Chronicle
|
4
|
+
module ETL
|
5
|
+
module Registry
|
6
|
+
# A singleton class that acts as a registry of connector classes available for ETL jobs
|
7
|
+
module Connectors
|
8
|
+
PHASES = [:extractor, :transformer, :loader].freeze
|
9
|
+
public_constant :PHASES
|
10
|
+
|
11
|
+
class << self
|
12
|
+
attr_accessor :connectors
|
13
|
+
|
14
|
+
def register(connector)
|
15
|
+
connectors << connector
|
16
|
+
end
|
17
|
+
|
18
|
+
def connectors
|
19
|
+
@connectors ||= []
|
20
|
+
end
|
21
|
+
|
22
|
+
# Find connector from amongst those currently loaded
|
23
|
+
def find_by_phase_and_identifier_local(phase, identifier)
|
24
|
+
connector = connectors.find { |c| c.phase == phase && c.identifier == identifier }
|
25
|
+
end
|
26
|
+
|
27
|
+
# Find connector and load relevant plugin to find it if necessary
|
28
|
+
def find_by_phase_and_identifier(phase, identifier)
|
29
|
+
connector = find_by_phase_and_identifier_local(phase, identifier)
|
30
|
+
return connector if connector
|
31
|
+
|
32
|
+
# if not available in built-in connectors, try to activate a
|
33
|
+
# relevant plugin and try again
|
34
|
+
if identifier.include?(":")
|
35
|
+
plugin, name = identifier.split(":")
|
36
|
+
else
|
37
|
+
# This case handles the case where the identifier is a
|
38
|
+
# shorthand (ie `imessage`) because there's only one default
|
39
|
+
# connector.
|
40
|
+
plugin = identifier
|
41
|
+
end
|
42
|
+
|
43
|
+
raise(Chronicle::ETL::PluginNotInstalledError.new(plugin)) unless Chronicle::ETL::Registry::Plugins.installed?(plugin)
|
44
|
+
|
45
|
+
Chronicle::ETL::Registry::Plugins.activate(plugin)
|
46
|
+
|
47
|
+
candidates = connectors.select { |c| c.phase == phase && c.plugin == plugin }
|
48
|
+
# if no name given, just use first connector with right phase/plugin
|
49
|
+
# TODO: set up a property for connectors to specify that they're the
|
50
|
+
# default connector for the plugin
|
51
|
+
candidates = candidates.select { |c| c.identifier == name } if name
|
52
|
+
connector = candidates.first
|
53
|
+
|
54
|
+
connector || raise(ConnectorNotAvailableError, "Connector '#{identifier}' not found")
|
55
|
+
end
|
56
|
+
end
|
57
|
+
end
|
58
|
+
end
|
59
|
+
end
|
60
|
+
end
|
@@ -0,0 +1,19 @@
|
|
1
|
+
module Chronicle
|
2
|
+
module ETL
|
3
|
+
module Registry
|
4
|
+
class PluginRegistration
|
5
|
+
attr_accessor :name, :description, :gem, :version, :installed, :gemspec
|
6
|
+
|
7
|
+
def initialize(name=nil)
|
8
|
+
@installed = false
|
9
|
+
@name = name
|
10
|
+
yield self if block_given?
|
11
|
+
end
|
12
|
+
|
13
|
+
def installed?
|
14
|
+
@installed || false
|
15
|
+
end
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
19
|
+
end
|
@@ -0,0 +1,163 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'rubygems/command'
|
3
|
+
require 'rubygems/commands/install_command'
|
4
|
+
require 'rubygems/uninstaller'
|
5
|
+
require 'gems'
|
6
|
+
require 'active_support/core_ext/hash/deep_merge'
|
7
|
+
|
8
|
+
module Chronicle
|
9
|
+
module ETL
|
10
|
+
module Registry
|
11
|
+
# Responsible for managing plugins available to chronicle-etl
|
12
|
+
#
|
13
|
+
# @todo Better validation for whether a gem is actually a plugin
|
14
|
+
# @todo Add ways to load a plugin that don't require a gem on rubygems.org
|
15
|
+
module Plugins
|
16
|
+
KNOWN_PLUGINS = [
|
17
|
+
'email',
|
18
|
+
'foursquare',
|
19
|
+
'github',
|
20
|
+
'imessage',
|
21
|
+
'pinboard',
|
22
|
+
'safari',
|
23
|
+
'shell',
|
24
|
+
'spotify',
|
25
|
+
'zulip'
|
26
|
+
].freeze
|
27
|
+
public_constant :KNOWN_PLUGINS
|
28
|
+
|
29
|
+
# Start of a system for having non-gem plugins. Right now, we just
|
30
|
+
# make registry aware of existence of name of non-gem plugin
|
31
|
+
def self.register_standalone(name:)
|
32
|
+
plugin = Chronicle::ETL::Registry::PluginRegistration.new do |p|
|
33
|
+
p.name = name
|
34
|
+
p.installed = true
|
35
|
+
end
|
36
|
+
|
37
|
+
installed_standalone << plugin
|
38
|
+
end
|
39
|
+
|
40
|
+
# Plugins either installed as gems or manually loaded/registered
|
41
|
+
def self.installed
|
42
|
+
installed_standalone + installed_as_gem
|
43
|
+
end
|
44
|
+
|
45
|
+
# Check whether a given plugin is installed
|
46
|
+
def self.installed?(name)
|
47
|
+
installed.map(&:name).include?(name)
|
48
|
+
end
|
49
|
+
|
50
|
+
# List of plugins installed as standalone
|
51
|
+
def self.installed_standalone
|
52
|
+
@standalones ||= []
|
53
|
+
end
|
54
|
+
|
55
|
+
# List of plugins installed as gems
|
56
|
+
def self.installed_as_gem
|
57
|
+
installed_gemspecs_latest.map do |gem|
|
58
|
+
Chronicle::ETL::Registry::PluginRegistration.new do |p|
|
59
|
+
p.name = gem.name.sub("chronicle-", "")
|
60
|
+
p.gem = gem.name
|
61
|
+
p.description = gem.description
|
62
|
+
p.version = gem.version.to_s
|
63
|
+
p.installed = true
|
64
|
+
end
|
65
|
+
end
|
66
|
+
end
|
67
|
+
|
68
|
+
# List of all plugins available to chronicle-etl
|
69
|
+
def self.available
|
70
|
+
available_as_gem
|
71
|
+
end
|
72
|
+
|
73
|
+
# List of plugins available through rubygems
|
74
|
+
# TODO: make this concurrent
|
75
|
+
def self.available_as_gem
|
76
|
+
KNOWN_PLUGINS.map do |name|
|
77
|
+
info = gem_info(name)
|
78
|
+
Chronicle::ETL::Registry::PluginRegistration.new do |p|
|
79
|
+
p.name = name
|
80
|
+
p.gem = info['name']
|
81
|
+
p.version = info['version']
|
82
|
+
p.description = info['info']
|
83
|
+
end
|
84
|
+
end
|
85
|
+
end
|
86
|
+
|
87
|
+
# Load info about a gem plugin from rubygems API
|
88
|
+
def self.gem_info(name)
|
89
|
+
gem_name = "chronicle-#{name}"
|
90
|
+
Gems.info(gem_name)
|
91
|
+
end
|
92
|
+
|
93
|
+
# Union of installed gems (latest version) + available gems
|
94
|
+
def self.all
|
95
|
+
(installed + available)
|
96
|
+
.group_by(&:name)
|
97
|
+
.transform_values { |plugin| plugin.find(&:installed) || plugin.first }
|
98
|
+
.values
|
99
|
+
end
|
100
|
+
|
101
|
+
# Does a plugin with a given name exist?
|
102
|
+
def self.exists?(name)
|
103
|
+
KNOWN_PLUGINS.include?(name)
|
104
|
+
end
|
105
|
+
|
106
|
+
# All versions of all plugins currently installed
|
107
|
+
def self.installed_gemspecs
|
108
|
+
# TODO: add check for chronicle-etl dependency
|
109
|
+
Gem::Specification.filter { |s| s.name.match(/^chronicle-/) && s.name != "chronicle-etl" }
|
110
|
+
end
|
111
|
+
|
112
|
+
# Latest version of each installed plugin
|
113
|
+
def self.installed_gemspecs_latest
|
114
|
+
installed_gemspecs.group_by(&:name)
|
115
|
+
.transform_values { |versions| versions.sort_by(&:version).reverse.first }
|
116
|
+
.values
|
117
|
+
end
|
118
|
+
|
119
|
+
# Activate a plugin with given name by `require`ing it
|
120
|
+
def self.activate(name)
|
121
|
+
# By default, activates the latest available version of a gem
|
122
|
+
# so don't have to run Kernel#gem separately
|
123
|
+
require "chronicle/#{name}"
|
124
|
+
rescue Gem::ConflictError => e
|
125
|
+
# TODO: figure out if there's more we can do here
|
126
|
+
raise Chronicle::ETL::PluginConflictError.new(name), "Plugin '#{name}' couldn't be loaded. #{e.message}"
|
127
|
+
rescue StandardError, LoadError => e
|
128
|
+
# StandardError to catch random non-loading problems that might occur
|
129
|
+
# when requiring the plugin (eg class macro invoked the wrong way)
|
130
|
+
# TODO: decide if this should be separated
|
131
|
+
raise Chronicle::ETL::PluginLoadError.new(name), "Plugin '#{name}' couldn't be loaded"
|
132
|
+
end
|
133
|
+
|
134
|
+
# Install a plugin to local gems
|
135
|
+
def self.install(name)
|
136
|
+
return if installed?(name)
|
137
|
+
raise(Chronicle::ETL::PluginNotAvailableError.new(name), "Plugin #{name} doesn't exist") unless exists?(name)
|
138
|
+
|
139
|
+
gem_name = "chronicle-#{name}"
|
140
|
+
|
141
|
+
Gem::DefaultUserInteraction.ui = Gem::SilentUI.new
|
142
|
+
Gem.install(gem_name)
|
143
|
+
|
144
|
+
activate(name)
|
145
|
+
rescue Gem::UnsatisfiableDependencyError
|
146
|
+
# TODO: we need to catch a lot more than this here
|
147
|
+
raise Chronicle::ETL::PluginNotAvailableError.new(name), "Plugin #{name} could not be installed."
|
148
|
+
end
|
149
|
+
|
150
|
+
# Uninstall a plugin
|
151
|
+
def self.uninstall(name)
|
152
|
+
gem_name = "chronicle-#{name}"
|
153
|
+
Gem::DefaultUserInteraction.ui = Gem::SilentUI.new
|
154
|
+
uninstaller = Gem::Uninstaller.new(gem_name)
|
155
|
+
uninstaller.uninstall
|
156
|
+
rescue Gem::InstallError
|
157
|
+
# TODO: strengthen this exception handling
|
158
|
+
raise(Chronicle::ETL::PluginError.new(name), "Plugin #{name} wasn't uninstalled")
|
159
|
+
end
|
160
|
+
end
|
161
|
+
end
|
162
|
+
end
|
163
|
+
end
|
@@ -1,61 +1,12 @@
|
|
1
|
-
require 'rubygems'
|
2
|
-
|
3
1
|
module Chronicle
|
4
2
|
module ETL
|
5
|
-
# A singleton class that acts as a registry of connector classes available for ETL jobs
|
6
3
|
module Registry
|
7
|
-
PHASES = [:extractor, :transformer, :loader]
|
8
|
-
|
9
|
-
class << self
|
10
|
-
attr_accessor :connectors
|
11
|
-
|
12
|
-
def register(connector)
|
13
|
-
connectors << connector
|
14
|
-
end
|
15
|
-
|
16
|
-
def connectors
|
17
|
-
@connectors ||= []
|
18
|
-
end
|
19
|
-
|
20
|
-
# Find connector from amongst those currently loaded
|
21
|
-
def find_by_phase_and_identifier_local(phase, identifier)
|
22
|
-
connector = connectors.find { |c| c.phase == phase && c.identifier == identifier }
|
23
|
-
end
|
24
|
-
|
25
|
-
# Find connector and load relevant plugin to find it if necessary
|
26
|
-
def find_by_phase_and_identifier(phase, identifier)
|
27
|
-
connector = find_by_phase_and_identifier_local(phase, identifier)
|
28
|
-
return connector if connector
|
29
|
-
|
30
|
-
# if not available in built-in connectors, try to activate a
|
31
|
-
# relevant plugin and try again
|
32
|
-
if identifier.include?(":")
|
33
|
-
plugin, name = identifier.split(":")
|
34
|
-
else
|
35
|
-
# This case handles the case where the identifier is a
|
36
|
-
# shorthand (ie `imessage`) because there's only one default
|
37
|
-
# connector.
|
38
|
-
plugin = identifier
|
39
|
-
end
|
40
|
-
|
41
|
-
raise(Chronicle::ETL::PluginNotInstalledError.new(plugin)) unless PluginRegistry.installed?(plugin)
|
42
|
-
|
43
|
-
PluginRegistry.activate(plugin)
|
44
|
-
|
45
|
-
candidates = connectors.select { |c| c.phase == phase && c.plugin == plugin }
|
46
|
-
# if no name given, just use first connector with right phase/plugin
|
47
|
-
# TODO: set up a property for connectors to specify that they're the
|
48
|
-
# default connector for the plugin
|
49
|
-
candidates = candidates.select { |c| c.identifier == name } if name
|
50
|
-
connector = candidates.first
|
51
|
-
|
52
|
-
connector || raise(ConnectorNotAvailableError, "Connector '#{identifier}' not found")
|
53
|
-
end
|
54
|
-
end
|
55
4
|
end
|
56
5
|
end
|
57
6
|
end
|
58
7
|
|
59
8
|
require_relative 'self_registering'
|
60
9
|
require_relative 'connector_registration'
|
61
|
-
require_relative '
|
10
|
+
require_relative 'connectors'
|
11
|
+
require_relative 'plugin_registration'
|
12
|
+
require_relative 'plugins'
|
@@ -17,7 +17,7 @@ module Chronicle
|
|
17
17
|
def register_connector
|
18
18
|
@connector_registration ||= ::Chronicle::ETL::Registry::ConnectorRegistration.new(self)
|
19
19
|
yield @connector_registration if block_given?
|
20
|
-
::Chronicle::ETL::Registry.register(@connector_registration)
|
20
|
+
::Chronicle::ETL::Registry::Connectors.register(@connector_registration)
|
21
21
|
end
|
22
22
|
end
|
23
23
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: chronicle-etl
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.5.
|
4
|
+
version: 0.5.5
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Andrew Louis
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2022-
|
11
|
+
date: 2022-05-19 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activesupport
|
@@ -52,6 +52,20 @@ dependencies:
|
|
52
52
|
- - "~>"
|
53
53
|
- !ruby/object:Gem::Version
|
54
54
|
version: 0.8.1
|
55
|
+
- !ruby/object:Gem::Dependency
|
56
|
+
name: gems
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - ">="
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '1'
|
62
|
+
type: :runtime
|
63
|
+
prerelease: false
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - ">="
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: '1'
|
55
69
|
- !ruby/object:Gem::Dependency
|
56
70
|
name: launchy
|
57
71
|
requirement: !ruby/object:Gem::Requirement
|
@@ -280,16 +294,16 @@ dependencies:
|
|
280
294
|
name: fakefs
|
281
295
|
requirement: !ruby/object:Gem::Requirement
|
282
296
|
requirements:
|
283
|
-
- - "
|
297
|
+
- - "~>"
|
284
298
|
- !ruby/object:Gem::Version
|
285
|
-
version: '
|
299
|
+
version: '1.4'
|
286
300
|
type: :development
|
287
301
|
prerelease: false
|
288
302
|
version_requirements: !ruby/object:Gem::Requirement
|
289
303
|
requirements:
|
290
|
-
- - "
|
304
|
+
- - "~>"
|
291
305
|
- !ruby/object:Gem::Version
|
292
|
-
version: '
|
306
|
+
version: '1.4'
|
293
307
|
- !ruby/object:Gem::Dependency
|
294
308
|
name: guard-rspec
|
295
309
|
requirement: !ruby/object:Gem::Requirement
|
@@ -374,6 +388,34 @@ dependencies:
|
|
374
388
|
- - "~>"
|
375
389
|
- !ruby/object:Gem::Version
|
376
390
|
version: '0.21'
|
391
|
+
- !ruby/object:Gem::Dependency
|
392
|
+
name: vcr
|
393
|
+
requirement: !ruby/object:Gem::Requirement
|
394
|
+
requirements:
|
395
|
+
- - "~>"
|
396
|
+
- !ruby/object:Gem::Version
|
397
|
+
version: '6.1'
|
398
|
+
type: :development
|
399
|
+
prerelease: false
|
400
|
+
version_requirements: !ruby/object:Gem::Requirement
|
401
|
+
requirements:
|
402
|
+
- - "~>"
|
403
|
+
- !ruby/object:Gem::Version
|
404
|
+
version: '6.1'
|
405
|
+
- !ruby/object:Gem::Dependency
|
406
|
+
name: webmock
|
407
|
+
requirement: !ruby/object:Gem::Requirement
|
408
|
+
requirements:
|
409
|
+
- - "~>"
|
410
|
+
- !ruby/object:Gem::Version
|
411
|
+
version: '3'
|
412
|
+
type: :development
|
413
|
+
prerelease: false
|
414
|
+
version_requirements: !ruby/object:Gem::Requirement
|
415
|
+
requirements:
|
416
|
+
- - "~>"
|
417
|
+
- !ruby/object:Gem::Version
|
418
|
+
version: '3'
|
377
419
|
- !ruby/object:Gem::Dependency
|
378
420
|
name: yard
|
379
421
|
requirement: !ruby/object:Gem::Requirement
|
@@ -454,7 +496,9 @@ files:
|
|
454
496
|
- lib/chronicle/etl/models/raw.rb
|
455
497
|
- lib/chronicle/etl/oauth_authorizer.rb
|
456
498
|
- lib/chronicle/etl/registry/connector_registration.rb
|
457
|
-
- lib/chronicle/etl/registry/
|
499
|
+
- lib/chronicle/etl/registry/connectors.rb
|
500
|
+
- lib/chronicle/etl/registry/plugin_registration.rb
|
501
|
+
- lib/chronicle/etl/registry/plugins.rb
|
458
502
|
- lib/chronicle/etl/registry/registry.rb
|
459
503
|
- lib/chronicle/etl/registry/self_registering.rb
|
460
504
|
- lib/chronicle/etl/runner.rb
|
@@ -1,95 +0,0 @@
|
|
1
|
-
require 'rubygems'
|
2
|
-
require 'rubygems/command'
|
3
|
-
require 'rubygems/commands/install_command'
|
4
|
-
require 'rubygems/uninstaller'
|
5
|
-
|
6
|
-
module Chronicle
|
7
|
-
module ETL
|
8
|
-
module Registry
|
9
|
-
# Responsible for managing plugins available to chronicle-etl
|
10
|
-
#
|
11
|
-
# @todo Better validation for whether a gem is actually a plugin
|
12
|
-
# @todo Add ways to load a plugin that don't require a gem on rubygems.org
|
13
|
-
module PluginRegistry
|
14
|
-
class << self
|
15
|
-
# Start of a system for having non-gem plugins. Right now, we just
|
16
|
-
# make registry aware of existenc of name of non-gem plugin
|
17
|
-
def register_standalone(name)
|
18
|
-
standalones << name
|
19
|
-
end
|
20
|
-
|
21
|
-
def standalones
|
22
|
-
@standalones ||= []
|
23
|
-
end
|
24
|
-
end
|
25
|
-
|
26
|
-
# Does this plugin exist?
|
27
|
-
def self.exists?(name)
|
28
|
-
# TODO: implement this. Could query rubygems.org or use a hardcoded
|
29
|
-
# list somewhere
|
30
|
-
true
|
31
|
-
end
|
32
|
-
|
33
|
-
# All versions of all plugins currently installed
|
34
|
-
def self.all_installed
|
35
|
-
# TODO: add check for chronicle-etl dependency
|
36
|
-
Gem::Specification.filter { |s| s.name.match(/^chronicle-/) && s.name != "chronicle-etl" }
|
37
|
-
end
|
38
|
-
|
39
|
-
# Latest version of each installed plugin
|
40
|
-
def self.all_installed_latest
|
41
|
-
all_installed.group_by(&:name)
|
42
|
-
.transform_values { |versions| versions.sort_by(&:version).reverse.first }
|
43
|
-
.values
|
44
|
-
end
|
45
|
-
|
46
|
-
# Check whether a given plugin is installed
|
47
|
-
def self.installed?(name)
|
48
|
-
(standalones + all_installed.map { |gem| gem.name.gsub("chronicle-", "") }).include?(name)
|
49
|
-
end
|
50
|
-
|
51
|
-
# Activate a plugin with given name by `require`ing it
|
52
|
-
def self.activate(name)
|
53
|
-
# By default, activates the latest available version of a gem
|
54
|
-
# so don't have to run Kernel#gem separately
|
55
|
-
require "chronicle/#{name}"
|
56
|
-
rescue Gem::ConflictError => e
|
57
|
-
# TODO: figure out if there's more we can do here
|
58
|
-
raise Chronicle::ETL::PluginConflictError.new(name), "Plugin '#{name}' couldn't be loaded. #{e.message}"
|
59
|
-
rescue StandardError, LoadError => e
|
60
|
-
# StandardError to catch random non-loading problems that might occur
|
61
|
-
# when requiring the plugin (eg class macro invoked the wrong way)
|
62
|
-
# TODO: decide if this should be separated
|
63
|
-
raise Chronicle::ETL::PluginLoadError.new(name), "Plugin '#{name}' couldn't be loaded"
|
64
|
-
end
|
65
|
-
|
66
|
-
# Install a plugin to local gems
|
67
|
-
def self.install(name)
|
68
|
-
return if installed?(name)
|
69
|
-
|
70
|
-
gem_name = "chronicle-#{name}"
|
71
|
-
raise(Chronicle::ETL::PluginNotAvailableError.new(gem_name), "Plugin #{name} doesn't exist") unless exists?(gem_name)
|
72
|
-
|
73
|
-
Gem::DefaultUserInteraction.ui = Gem::SilentUI.new
|
74
|
-
Gem.install(gem_name)
|
75
|
-
|
76
|
-
activate(name)
|
77
|
-
rescue Gem::UnsatisfiableDependencyError
|
78
|
-
# TODO: we need to catch a lot more than this here
|
79
|
-
raise Chronicle::ETL::PluginNotAvailableError.new(name), "Plugin #{name} could not be installed."
|
80
|
-
end
|
81
|
-
|
82
|
-
# Uninstall a plugin
|
83
|
-
def self.uninstall(name)
|
84
|
-
gem_name = "chronicle-#{name}"
|
85
|
-
Gem::DefaultUserInteraction.ui = Gem::SilentUI.new
|
86
|
-
uninstaller = Gem::Uninstaller.new(gem_name)
|
87
|
-
uninstaller.uninstall
|
88
|
-
rescue Gem::InstallError
|
89
|
-
# TODO: strengthen this exception handling
|
90
|
-
raise(Chronicle::ETL::PluginError.new(name), "Plugin #{name} wasn't uninstalled")
|
91
|
-
end
|
92
|
-
end
|
93
|
-
end
|
94
|
-
end
|
95
|
-
end
|