chronicle-etl 0.5.2 → 0.5.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +72 -46
- data/chronicle-etl.gemspec +7 -1
- data/lib/chronicle/etl/authorization_server.rb +48 -0
- data/lib/chronicle/etl/authorizer.rb +37 -0
- data/lib/chronicle/etl/cli/authorizations.rb +63 -0
- data/lib/chronicle/etl/cli/cli_base.rb +4 -0
- data/lib/chronicle/etl/cli/connectors.rb +2 -2
- data/lib/chronicle/etl/cli/jobs.rb +16 -3
- data/lib/chronicle/etl/cli/main.rb +32 -20
- data/lib/chronicle/etl/cli/plugins.rb +19 -15
- data/lib/chronicle/etl/cli/secrets.rb +1 -1
- data/lib/chronicle/etl/cli.rb +1 -0
- data/lib/chronicle/etl/config.rb +3 -0
- data/lib/chronicle/etl/configurable.rb +4 -0
- data/lib/chronicle/etl/exceptions.rb +5 -0
- data/lib/chronicle/etl/job_definition.rb +4 -4
- data/lib/chronicle/etl/logger.rb +6 -6
- data/lib/chronicle/etl/models/base.rb +1 -1
- data/lib/chronicle/etl/models/entity.rb +3 -1
- data/lib/chronicle/etl/oauth_authorizer.rb +140 -0
- data/lib/chronicle/etl/registry/connectors.rb +60 -0
- data/lib/chronicle/etl/registry/plugin_registration.rb +19 -0
- data/lib/chronicle/etl/registry/plugins.rb +163 -0
- data/lib/chronicle/etl/registry/registry.rb +3 -52
- data/lib/chronicle/etl/registry/self_registering.rb +1 -1
- data/lib/chronicle/etl/runner.rb +53 -26
- data/lib/chronicle/etl/secrets.rb +15 -1
- data/lib/chronicle/etl/transformers/transformer.rb +4 -0
- data/lib/chronicle/etl/version.rb +1 -1
- data/lib/chronicle/etl.rb +1 -0
- metadata +101 -11
- data/lib/chronicle/etl/registry/plugin_registry.rb +0 -84
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: a2de46efc3c5fbdc7ac120137bef56e13a138c8a95c8dd7d0a3542a65be65959
|
|
4
|
+
data.tar.gz: e8e3e9ae236e270b2926037419d5349170f85b8597c640c0b7b899552257fdb9
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 58b293d45d1a7f4589aee080d0fb1348e45d76753a8d48bb33e694d6a2b6ea123d1495a12a42e98ed6d3e65926ad48bd7de1fb98dac9bc7bac9225ef00d32fb3
|
|
7
|
+
data.tar.gz: e73c73b67b4e3790347da3df22ed7a11357fdf43e91b734925503f8d0c189aabd07e26f51ed25cf1409fd68ba74dd72e1e7e01c113f773832c509172f0b4ee84
|
data/README.md
CHANGED
|
@@ -6,21 +6,28 @@
|
|
|
6
6
|
|
|
7
7
|
Are you trying to archive your digital history or incorporate it into your own projects? You’ve probably discovered how frustrating it is to get machine-readable access to your own data. While [building a memex](https://hyfen.net/memex/), I learned first-hand what great efforts must be made before you can begin using the data in interesting ways.
|
|
8
8
|
|
|
9
|
-
If you don’t want to spend all your time writing scrapers, reverse-engineering APIs, or parsing takeout data, this
|
|
9
|
+
If you don’t want to spend all your time writing scrapers, reverse-engineering APIs, or parsing takeout data, this tool is for you! (*If you do enjoy these things, please see the [open issues](https://github.com/chronicle-app/chronicle-etl/issues).*)
|
|
10
10
|
|
|
11
|
-
**`chronicle-etl` is a CLI tool that gives you a unified interface
|
|
11
|
+
**`chronicle-etl` is a CLI tool that gives you a unified interface to your personal data.** It uses the ETL pattern to *extract* data from a source (e.g. your local browser history, a directory of images, goodreads.com reading history), *transform* it (into a given schema), and *load* it to a destination (e.g. a CSV file, JSON, external API).
|
|
12
12
|
|
|
13
13
|
## What does `chronicle-etl` give you?
|
|
14
|
-
* **CLI tool for working with personal data**. You can monitor progress of exports, manipulate the output, set up recurring jobs, manage credentials, and more.
|
|
15
|
-
* **Plugins for many third-party providers
|
|
16
|
-
* **A common, opinionated schema**: You can normalize different datasets into a single schema so that, for example, all your iMessages and emails are
|
|
14
|
+
* **A CLI tool for working with personal data**. You can monitor progress of exports, manipulate the output, set up recurring jobs, manage credentials, and more.
|
|
15
|
+
* **Plugins for many third-party providers** (see [list](#available-plugins-and-connectors)). This plugin system allows you to access data from dozens of third-party services, all accessible through a common CLI interface.
|
|
16
|
+
* **A common, opinionated schema**: You can normalize different datasets into a single schema so that, for example, all your iMessages and emails are represented in a common schema. (Don’t want to use this schema? `chronicle-etl` always allows you to fall back on working with the raw extraction data.)
|
|
17
|
+
|
|
18
|
+
## Chronicle-ETL in action
|
|
19
|
+
|
|
20
|
+
|
|
21
|
+
|
|
22
|
+
### Longer screencast
|
|
23
|
+
|
|
24
|
+
[](https://asciinema.org/a/483455)
|
|
17
25
|
|
|
18
26
|
## Installation
|
|
19
27
|
|
|
20
28
|
Using homebrew:
|
|
21
29
|
```sh
|
|
22
30
|
$ brew install chronicle-app/etl/chronicle-etl
|
|
23
|
-
|
|
24
31
|
```
|
|
25
32
|
Using rubygems:
|
|
26
33
|
```sh
|
|
@@ -38,11 +45,15 @@ $ chronicle-etl --version
|
|
|
38
45
|
# Display help
|
|
39
46
|
$ chronicle-etl help
|
|
40
47
|
|
|
41
|
-
#
|
|
48
|
+
# Run a basic job
|
|
42
49
|
$ chronicle-etl --extractor NAME --transformer NAME --loader NAME
|
|
43
50
|
|
|
44
51
|
# Read test.csv and display it to stdout as a table
|
|
45
|
-
$ chronicle-etl --extractor csv --input
|
|
52
|
+
$ chronicle-etl --extractor csv --input data.csv --loader table
|
|
53
|
+
|
|
54
|
+
# Show available plugins and install one
|
|
55
|
+
$ chronicle-etl plugins:list
|
|
56
|
+
$ chronicle-etl plugins:install shell
|
|
46
57
|
|
|
47
58
|
# Retrieve shell commands run in the last 5 hours
|
|
48
59
|
$ chronicle-etl -e shell --since 5h
|
|
@@ -79,37 +90,36 @@ Options:
|
|
|
79
90
|
[--silent], [--no-silent] # Silence all output
|
|
80
91
|
```
|
|
81
92
|
|
|
82
|
-
### Saving
|
|
83
|
-
|
|
84
|
-
You can save details about a job to a local config file (saved by default in `~/.config/chronicle/etl/jobs/job_name.yml`) to save yourself the trouble of setting the CLI flags for each run.
|
|
93
|
+
### Saving a job
|
|
94
|
+
You can save details about a job to a local config file (saved by default in `~/.config/chronicle/etl/jobs/JOB_NAME.yml`) to save yourself the trouble specifying options each time.
|
|
85
95
|
|
|
86
96
|
```sh
|
|
87
97
|
# Save a job named 'sample' to ~/.config/chronicle/etl/jobs/sample.yml
|
|
88
98
|
$ chronicle-etl jobs:save sample --extractor pinboard --since 10d
|
|
89
99
|
|
|
90
|
-
# Show details about the job
|
|
91
|
-
$ chronicle-etl jobs:show sample
|
|
92
|
-
|
|
93
100
|
# Run the job
|
|
94
101
|
$ chronicle-etl jobs:run sample
|
|
95
|
-
|
|
96
|
-
|
|
102
|
+
|
|
103
|
+
# Show details about the job
|
|
104
|
+
$ chronicle-etl jobs:show sample
|
|
97
105
|
|
|
98
106
|
# Show all saved jobs
|
|
99
107
|
$ chronicle-etl jobs:list
|
|
100
108
|
```
|
|
101
109
|
|
|
102
|
-
## Connectors
|
|
103
|
-
|
|
110
|
+
## Connectors and plugins
|
|
111
|
+
|
|
112
|
+
Connectors let you work with different data formats or third-party providers.
|
|
113
|
+
|
|
114
|
+
### Built-in Connectors
|
|
115
|
+
|
|
116
|
+
`chronicle-etl` comes with several built-in connectors for common formats and sources.
|
|
104
117
|
|
|
105
118
|
```sh
|
|
106
119
|
# List all available connectors
|
|
107
120
|
$ chronicle-etl connectors:list
|
|
108
121
|
```
|
|
109
122
|
|
|
110
|
-
### Built-in Connectors
|
|
111
|
-
`chronicle-etl` comes with several built-in connectors for common formats and sources.
|
|
112
|
-
|
|
113
123
|
#### Extractors
|
|
114
124
|
- [`csv`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/extractors/csv_extractor.rb) - Load records from CSV files or stdin
|
|
115
125
|
- [`json`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/extractors/json_extractor.rb) - Load JSON (either [line-separated objects](https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON) or one object)
|
|
@@ -124,18 +134,19 @@ $ chronicle-etl connectors:list
|
|
|
124
134
|
- [`json`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/loaders/json_loader.rb) - Load records serialized as JSON
|
|
125
135
|
- [`rest`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/loaders/rest_loader.rb) - Serialize records with [JSONAPI](https://jsonapi.org/) and send to a REST API
|
|
126
136
|
|
|
127
|
-
|
|
128
|
-
|
|
137
|
+
### Chronicle Plugins for third-party services
|
|
138
|
+
|
|
139
|
+
Plugins provide access to data from third-party platforms, services, or formats. Plugins are packaged as separate gems and can be installed through the CLI (under the hood, it's a `gem install chronicle-PLUGINNAME`)
|
|
129
140
|
|
|
130
|
-
|
|
141
|
+
#### Plugin usage
|
|
131
142
|
|
|
132
143
|
```bash
|
|
144
|
+
# List available plugins
|
|
145
|
+
$ chronicle-etl plugins:list
|
|
146
|
+
|
|
133
147
|
# Install a plugin
|
|
134
148
|
$ chronicle-etl plugins:install NAME
|
|
135
149
|
|
|
136
|
-
# List installed plugins
|
|
137
|
-
$ chronicle-etl plugins:list
|
|
138
|
-
|
|
139
150
|
# Use a plugin
|
|
140
151
|
$ chronicle-etl plugins:install shell
|
|
141
152
|
$ chronicle-etl --extractor shell:history --limit 10
|
|
@@ -143,31 +154,45 @@ $ chronicle-etl --extractor shell:history --limit 10
|
|
|
143
154
|
# Uninstall a plugin
|
|
144
155
|
$ chronicle-etl plugins:uninstall NAME
|
|
145
156
|
```
|
|
146
|
-
|
|
147
|
-
|
|
157
|
+
#### Available plugins and connectors
|
|
158
|
+
|
|
159
|
+
The following are the officially-supported list of plugins and their available connectors:
|
|
160
|
+
|
|
161
|
+
| Plugin | Type | Identifier | Description | Description |
|
|
162
|
+
|---------------------------------------------------------------------|-------------|------------------|-------------------------------------------|-------------------------------------------|
|
|
163
|
+
| [email](https://github.com/chronicle-app/chronicle-email) | extractor | imap | emails over an IMAP connection | emails over an IMAP connection |
|
|
164
|
+
| [email](https://github.com/chronicle-app/chronicle-email) | extractor | mbox | emails from an .mbox file | emails from an .mbox file |
|
|
165
|
+
| [email](https://github.com/chronicle-app/chronicle-email) | transformer | email | email to Chronicle Schema | email to Chronicle Schema |
|
|
166
|
+
| [foursquare](https://github.com/chronicle-app/chronicle-foursquare) | extractor | checkins | Foursqure visits | Foursqure visits |
|
|
167
|
+
| [foursquare](https://github.com/chronicle-app/chronicle-foursquare) | transformer | checkin | checkin to Chronicle Schema | checkin to Chronicle Schema |
|
|
168
|
+
| [github](https://github.com/chronicle-app/chronicle-github) | extractor | activity | user activity stream | user activity stream |
|
|
169
|
+
| [imessage](https://github.com/chronicle-app/chronicle-imessage) | extractor | messages | imessages from local macOS | imessages from local macOS |
|
|
170
|
+
| [imessage](https://github.com/chronicle-app/chronicle-imessage) | transformer | message | imessage to Chronicle Schema | imessage to Chronicle Schema |
|
|
171
|
+
| [pinboard](https://github.com/chronicle-app/chronicle-pinboard) | extractor | bookmarks | Pinboard.in bookmarks | Pinboard.in bookmarks |
|
|
172
|
+
| [pinboard](https://github.com/chronicle-app/chronicle-pinboard) | transformer | bookmark | bookmark to Chronicle Schema | bookmark to Chronicle Schema |
|
|
173
|
+
| [safari](https://github.com/chronicle-app/chronicle-safari) | extractor | browser-history | browser history | browser history |
|
|
174
|
+
| [safari ](https://github.com/chronicle-app/chronicle-safari ) | transformer | browser-history | browser history to Chronicle Schema | browser history to Chronicle Schema |
|
|
175
|
+
| [shell](https://github.com/chronicle-app/chronicle-shell) | extractor | history | shell command history (bash / zsh) | shell command history (bash / zsh) |
|
|
176
|
+
| [shell](https://github.com/chronicle-app/chronicle-shell) | transformer | command | command to Chronicle Schema | command to Chronicle Schema |
|
|
177
|
+
| [spotify](https://github.com/chronicle-app/chronicle-spotify) | extractor | liked-tracks | liked tracks | liked tracks |
|
|
178
|
+
| [spotify](https://github.com/chronicle-app/chronicle-spotify) | extractor | saved-albums | saved albums | saved albums |
|
|
179
|
+
| [spotify](https://github.com/chronicle-app/chronicle-spotify) | extractor | listens | recently listened tracks (last 50 tracks) | recently listened tracks (last 50 tracks) |
|
|
180
|
+
| [spotify](https://github.com/chronicle-app/chronicle-spotify) | transformer | like | like to Chronicle Schema | like to Chronicle Schema |
|
|
181
|
+
| [spotify](https://github.com/chronicle-app/chronicle-spotify) | transformer | listen | listen to Chronicle Schema | listen to Chronicle Schema |
|
|
182
|
+
| [spotify](https://github.com/chronicle-app/chronicle-spotify) | authorizer | | OAuth authorizer | OAuth authorizer |
|
|
183
|
+
| [zulip](https://github.com/chronicle-app/chronicle-zulip) | extractor | private-messages | private messages | private messages |
|
|
184
|
+
| [zulip](https://github.com/chronicle-app/chronicle-zulip) | transformer | message | message to Chronicle Schema | message to Chronicle Schema |
|
|
185
|
+
|
|
186
|
+
|
|
187
|
+
### Coming soon
|
|
148
188
|
|
|
149
189
|
A few dozen importers exist [in my Memex project](https://hyfen.net/memex/) and I'm porting them over to the Chronicle system. The [Chronicle Plugin Tracker](https://github.com/orgs/chronicle-app/projects/1/views/1) lets you keep track what's available and what's coming soon.
|
|
150
190
|
|
|
151
191
|
If you don't see a plugin for a third-party provider or data source that you're interested in using with `chronicle-etl`, [please open an issue](https://github.com/chronicle-app/chronicle-etl/issues/new). If you want to work together on a plugin, please [get in touch](#get-in-touch)!
|
|
152
192
|
|
|
153
|
-
#### Currently available
|
|
154
|
-
|
|
155
|
-
| Name | Description | Availability |
|
|
156
|
-
|-----------------------------------------------------------------|---------------------------------------------------------------------------------------------|----------------------------------|
|
|
157
|
-
| [imessage](https://github.com/chronicle-app/chronicle-imessage) | iMessage messages and attachments | Available |
|
|
158
|
-
| [shell](https://github.com/chronicle-app/chronicle-shell) | Shell command history | Available (still needs zsh support) |
|
|
159
|
-
| [email](https://github.com/chronicle-app/chronicle-email) | Emails and attachments from IMAP or .mbox files | Available (still needs IMAP support) |
|
|
160
|
-
| [pinboard](https://github.com/chronicle-app/chronicle-email) | Bookmarks and tags | Available |
|
|
161
|
-
| [safari](https://github.com/chronicle-app/chronicle-safari) | Browser history from local sqlite db | Available |
|
|
162
|
-
| [github](https://github.com/chronicle-app/chronicle-github) | Github activity stream | Available |
|
|
163
|
-
|
|
164
|
-
#### Coming soon
|
|
165
|
-
|
|
166
193
|
In summary, the following **are coming soon**:
|
|
167
194
|
anki, arc, bear, chrome, facebook, firefox, fitbit, foursquare, git, github, goodreads, google-calendar, images, instagram, lastfm, shazam, slack, strava, things, twitter, whatsapp, youtube.
|
|
168
195
|
|
|
169
|
-
Please check the [Chronicle Plugin Tracker](https://github.com/orgs/chronicle-app/projects/1/views/1) for details.
|
|
170
|
-
|
|
171
196
|
### Writing your own plugin
|
|
172
197
|
|
|
173
198
|
Additional connectors are packaged as separate ruby gems. You can view the [iMessage plugin](https://github.com/chronicle-app/chronicle-imessage) for an example.
|
|
@@ -202,6 +227,7 @@ module Chronicle
|
|
|
202
227
|
end
|
|
203
228
|
```
|
|
204
229
|
|
|
230
|
+
|
|
205
231
|
## Secrets Management
|
|
206
232
|
|
|
207
233
|
If your job needs secrets such as access tokens or passwords, `chronicle-etl` has a built-in secret management system.
|
|
@@ -236,8 +262,8 @@ $ chronicle-etl secrets:unset pinboard access_token
|
|
|
236
262
|
## Roadmap
|
|
237
263
|
|
|
238
264
|
- Keep tackling **new plugins**. See: [Chronicle Plugin Tracker](https://github.com/orgs/chronicle-app/projects/1)
|
|
239
|
-
- Add support for **incremental extractions** #37
|
|
240
|
-
- **Improve stdin extractor and shell command transformer**
|
|
265
|
+
- Add support for **incremental extractions** ([#37](https://github.com/chronicle-app/chronicle-etl/issues/37))
|
|
266
|
+
- **Improve stdin extractor and shell command transformer** so that users can easily integrate their own scripts/languages/tools into jobs ([#5](https://github.com/chronicle-app/chronicle-etl/issues/48))
|
|
241
267
|
- **Add documentation for Chronicle Schema**. It's found throughout this project but never explained.
|
|
242
268
|
|
|
243
269
|
## Development
|
data/chronicle-etl.gemspec
CHANGED
|
@@ -40,10 +40,14 @@ Gem::Specification.new do |spec|
|
|
|
40
40
|
spec.add_dependency "activesupport", "~> 7.0"
|
|
41
41
|
spec.add_dependency "chronic_duration", "~> 0.10.6"
|
|
42
42
|
spec.add_dependency "colorize", "~> 0.8.1"
|
|
43
|
+
spec.add_dependency "gems", ">= 1"
|
|
44
|
+
spec.add_dependency "launchy"
|
|
43
45
|
spec.add_dependency "marcel", "~> 1.0.2"
|
|
44
46
|
spec.add_dependency "mini_exiftool", "~> 2.10"
|
|
45
47
|
spec.add_dependency "nokogiri", "~> 1.13"
|
|
48
|
+
spec.add_dependency "omniauth", "~> 2"
|
|
46
49
|
spec.add_dependency "sequel", "~> 5.35"
|
|
50
|
+
spec.add_dependency "sinatra", "~> 2"
|
|
47
51
|
spec.add_dependency "sqlite3", "~> 1.4"
|
|
48
52
|
spec.add_dependency "thor", "~> 1.2"
|
|
49
53
|
spec.add_dependency "thor-hollaback", "~> 0.2"
|
|
@@ -54,12 +58,14 @@ Gem::Specification.new do |spec|
|
|
|
54
58
|
spec.add_dependency "xdg", ">= 4.0"
|
|
55
59
|
|
|
56
60
|
spec.add_development_dependency "bundler", "~> 2.1"
|
|
61
|
+
spec.add_development_dependency "fakefs", "~> 1.4"
|
|
57
62
|
spec.add_development_dependency "guard-rspec", "~> 4.7.3"
|
|
58
|
-
spec.add_development_dependency "fakefs"
|
|
59
63
|
spec.add_development_dependency "pry-byebug", "~> 3.9"
|
|
60
64
|
spec.add_development_dependency "rake", "~> 13.0"
|
|
61
65
|
spec.add_development_dependency "rspec", "~> 3.9"
|
|
62
66
|
spec.add_development_dependency "rubocop", "~> 1.25.1"
|
|
63
67
|
spec.add_development_dependency "simplecov", "~> 0.21"
|
|
68
|
+
spec.add_development_dependency "vcr", "~> 6.1"
|
|
69
|
+
spec.add_development_dependency "webmock", "~> 3"
|
|
64
70
|
spec.add_development_dependency "yard", "~> 0.9.7"
|
|
65
71
|
end
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
require 'sinatra'
|
|
2
|
+
require 'omniauth'
|
|
3
|
+
|
|
4
|
+
module Chronicle
|
|
5
|
+
module ETL
|
|
6
|
+
class AuthorizationServer < Sinatra::Base
|
|
7
|
+
class << self
|
|
8
|
+
attr_accessor :latest_authorization
|
|
9
|
+
end
|
|
10
|
+
|
|
11
|
+
configure do
|
|
12
|
+
set :inline_templates, true
|
|
13
|
+
set :dump_errors, false
|
|
14
|
+
set :raise_errors, true
|
|
15
|
+
disable :logging
|
|
16
|
+
set :sessions, true
|
|
17
|
+
set :quiet, true
|
|
18
|
+
set :threaded, true
|
|
19
|
+
set :environment, ENV['APP_ENV'] == 'test' ? :test : :production
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
use OmniAuth::Builder do
|
|
23
|
+
Chronicle::ETL::OauthAuthorizer.all.each do |klass|
|
|
24
|
+
args = [klass.client_id, klass.client_secret, klass.options].compact
|
|
25
|
+
provider(
|
|
26
|
+
klass.strategy,
|
|
27
|
+
*args
|
|
28
|
+
)
|
|
29
|
+
end
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
OmniAuth.config.logger = Chronicle::ETL::Logger
|
|
33
|
+
OmniAuth.config.silence_get_warning = true
|
|
34
|
+
OmniAuth.config.allowed_request_methods = %i[get]
|
|
35
|
+
|
|
36
|
+
get '/auth/:provider/callback' do
|
|
37
|
+
authorization = request.env['omniauth.auth'].to_h.deep_transform_keys(&:to_sym)
|
|
38
|
+
self.class.latest_authorization = authorization
|
|
39
|
+
erb "<h1>Settings saved for #{params[:provider]}</h1><p>You can now close this tab and return to your terminal!</p>"
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
get '/auth/failure' do
|
|
43
|
+
# TODO: handle this
|
|
44
|
+
erb "<h1>Authentication Failed:</h1><h3>message:<h3> <pre>#{params}</pre>"
|
|
45
|
+
end
|
|
46
|
+
end
|
|
47
|
+
end
|
|
48
|
+
end
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
module Chronicle
|
|
2
|
+
module ETL
|
|
3
|
+
# An authorization strategy for a third-party data source
|
|
4
|
+
class Authorizer
|
|
5
|
+
class << self
|
|
6
|
+
attr_reader :provider_name
|
|
7
|
+
|
|
8
|
+
# Macro for setting provider on an Authorizer
|
|
9
|
+
def provider(provider_name)
|
|
10
|
+
@provider_name = provider_name
|
|
11
|
+
end
|
|
12
|
+
|
|
13
|
+
# From all loaded Authorizers, return the first one that matches
|
|
14
|
+
# a given provider
|
|
15
|
+
#
|
|
16
|
+
# @todo Have a proper identifier system for authorizers
|
|
17
|
+
# (to have more than one per plugin)
|
|
18
|
+
def find_by_provider(provider)
|
|
19
|
+
ObjectSpace.each_object(::Class).select {|klass| klass < self }.find do |authorizer|
|
|
20
|
+
authorizer.provider_name == provider
|
|
21
|
+
end
|
|
22
|
+
end
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
# Construct a new authorizer
|
|
26
|
+
def initialize(args)
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
# Main entry-point for authorization flows. Implemented by subclass
|
|
30
|
+
def authorize!
|
|
31
|
+
raise NotImplementedError
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
end
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
require_relative 'oauth_authorizer'
|
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'sinatra'
|
|
4
|
+
require 'launchy'
|
|
5
|
+
require 'pp'
|
|
6
|
+
|
|
7
|
+
module Chronicle
|
|
8
|
+
module ETL
|
|
9
|
+
module CLI
|
|
10
|
+
# CLI commands for authorizing chronicle-etl with third-party services
|
|
11
|
+
class Authorizations < SubcommandBase
|
|
12
|
+
default_task 'new'
|
|
13
|
+
namespace :authorizations
|
|
14
|
+
|
|
15
|
+
desc "authorize", "Authorize with a third-party provider"
|
|
16
|
+
option :port, desc: 'Port to run authorization server on', type: :numeric, default: 4567
|
|
17
|
+
option :credentials, desc: 'Secrets namespace for where to read credentials from (default: PROVIDER)', type: :string, banner: 'NAMESPACE'
|
|
18
|
+
option :secrets, desc: 'Secrets namespace for where authorization should be saved to (default: PROVIDER)', type: :string, banner: 'NAMESPACE'
|
|
19
|
+
option :print, desc: 'Show authorization results (instead of just saving secrets)', type: :boolean, default: false
|
|
20
|
+
def new(provider)
|
|
21
|
+
authorizer_klass = find_authorizer_klass(provider)
|
|
22
|
+
credentials = load_credentials(provider: provider, credentials_source: options[:credentials])
|
|
23
|
+
authorizer = authorizer_klass.new(port: options[:port], credentials: credentials)
|
|
24
|
+
|
|
25
|
+
secrets = authorizer.authorize!
|
|
26
|
+
secrets_namespace = options[:secrets] || provider
|
|
27
|
+
Chronicle::ETL::Secrets.set_all(secrets_namespace, secrets)
|
|
28
|
+
|
|
29
|
+
pp secrets if options[:print]
|
|
30
|
+
|
|
31
|
+
cli_exit(message: "Authorization saved to '#{secrets_namespace}' secrets")
|
|
32
|
+
rescue StandardError => e
|
|
33
|
+
cli_fail(message: "Authorization not successful.\n" + e.message, exception: e)
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
private
|
|
37
|
+
|
|
38
|
+
def find_authorizer_klass(provider)
|
|
39
|
+
# TODO: this assumes provider:plugin one-to-one
|
|
40
|
+
unless Chronicle::ETL::Registry::Plugins.installed?(provider)
|
|
41
|
+
cli_fail(message: "Plugin for #{provider} is not installed.")
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
begin
|
|
45
|
+
Chronicle::ETL::Registry::Plugins.activate(provider)
|
|
46
|
+
rescue PluginError => e
|
|
47
|
+
cli_fail(message: "Could not load plugin '#{provider}'.\n" + e.message, exception: e)
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
Authorizer.find_by_provider(provider.to_sym) || cli_fail(message: "No authorizer available for '#{provider}'")
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
def load_credentials(provider:, credentials_source: nil)
|
|
54
|
+
if credentials_source && !Chronicle::ETL::Secrets.exists?(credentials_source)
|
|
55
|
+
cli_fail(message: "OAuth credentials specified as '#{credentials_source}' but a secrets namespace with that name does not exist.")
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
Chronicle::ETL::Secrets.read(credentials_source || provider)
|
|
59
|
+
end
|
|
60
|
+
end
|
|
61
|
+
end
|
|
62
|
+
end
|
|
63
|
+
end
|
|
@@ -6,6 +6,10 @@ module Chronicle
|
|
|
6
6
|
no_commands do
|
|
7
7
|
# Shorthand for cli_exit(status: :failure)
|
|
8
8
|
def cli_fail(message: nil, exception: nil)
|
|
9
|
+
if exception && Chronicle::ETL::Logger.log_level > Chronicle::ETL::Logger::DEBUG
|
|
10
|
+
message += "\nRe-run the command with --verbose to see details."
|
|
11
|
+
end
|
|
12
|
+
|
|
9
13
|
cli_exit(status: :failure, message: message, exception: exception)
|
|
10
14
|
end
|
|
11
15
|
|
|
@@ -13,7 +13,7 @@ module Chronicle
|
|
|
13
13
|
desc "list", "Lists available connectors"
|
|
14
14
|
# Display all available connectors that chronicle-etl has access to
|
|
15
15
|
def list
|
|
16
|
-
connector_info = Chronicle::ETL::Registry.connectors.map do |connector_registration|
|
|
16
|
+
connector_info = Chronicle::ETL::Registry::Connectors.connectors.map do |connector_registration|
|
|
17
17
|
{
|
|
18
18
|
identifier: connector_registration.identifier,
|
|
19
19
|
phase: connector_registration.phase,
|
|
@@ -43,7 +43,7 @@ module Chronicle
|
|
|
43
43
|
end
|
|
44
44
|
|
|
45
45
|
begin
|
|
46
|
-
connector = Chronicle::ETL::Registry.find_by_phase_and_identifier(phase.to_sym, identifier)
|
|
46
|
+
connector = Chronicle::ETL::Registry::Connectors.find_by_phase_and_identifier(phase.to_sym, identifier)
|
|
47
47
|
rescue Chronicle::ETL::ConnectorNotAvailableError, Chronicle::ETL::PluginError => e
|
|
48
48
|
cli_fail(message: "Could not find #{phase} #{identifier}", exception: e)
|
|
49
49
|
end
|
|
@@ -43,6 +43,17 @@ module Chronicle
|
|
|
43
43
|
LONG_DESC
|
|
44
44
|
# Run an ETL job
|
|
45
45
|
def start(name = nil)
|
|
46
|
+
# If someone runs `$ chronicle-etl` with no arguments, show help menu.
|
|
47
|
+
# TODO: decide if we should check that there's nothing in stdin pipe
|
|
48
|
+
# in case user wants to actually run this sort of job stdin->null->stdout
|
|
49
|
+
if name.nil? && options[:extractor].nil?
|
|
50
|
+
m = Chronicle::ETL::CLI::Main.new
|
|
51
|
+
m.help
|
|
52
|
+
cli_exit
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
cli_fail(message: "Job '#{name}' does not exist") if name && !Chronicle::ETL::Config.exists?("jobs", name)
|
|
56
|
+
|
|
46
57
|
job_definition = build_job_definition(name, options)
|
|
47
58
|
|
|
48
59
|
if job_definition.plugins_missing?
|
|
@@ -82,11 +93,10 @@ LONG_DESC
|
|
|
82
93
|
|
|
83
94
|
if write_config
|
|
84
95
|
Chronicle::ETL::Config.write("jobs", name, job_definition.definition)
|
|
85
|
-
cli_exit(message: "Job saved. Run it with `$chronicle-etl jobs:run #{name}`")
|
|
96
|
+
cli_exit(message: "Job saved. Run it with `$ chronicle-etl jobs:run #{name}`")
|
|
86
97
|
else
|
|
87
98
|
cli_fail(message: "\nJob not saved")
|
|
88
99
|
end
|
|
89
|
-
|
|
90
100
|
rescue Chronicle::ETL::JobDefinitionError => e
|
|
91
101
|
cli_fail(message: "Job definition error", exception: e)
|
|
92
102
|
end
|
|
@@ -94,6 +104,8 @@ LONG_DESC
|
|
|
94
104
|
desc "show", "Show details about a job"
|
|
95
105
|
# Show an ETL job
|
|
96
106
|
def show(name = nil)
|
|
107
|
+
cli_fail(message: "Job '#{name}' does not exist") if name && !Chronicle::ETL::Config.exists?("jobs", name)
|
|
108
|
+
|
|
97
109
|
job_definition = build_job_definition(name, options)
|
|
98
110
|
job_definition.validate!
|
|
99
111
|
puts Chronicle::ETL::Job.new(job_definition)
|
|
@@ -132,10 +144,11 @@ LONG_DESC
|
|
|
132
144
|
job_definition.validate!
|
|
133
145
|
# FIXME: clumsy to make CLI responsible for setting secrets here. Think about a better way to do this
|
|
134
146
|
job_definition.apply_default_secrets
|
|
135
|
-
|
|
136
147
|
job = Chronicle::ETL::Job.new(job_definition)
|
|
137
148
|
runner = Chronicle::ETL::Runner.new(job)
|
|
138
149
|
runner.run!
|
|
150
|
+
rescue RunnerError => e
|
|
151
|
+
cli_fail(message: "#{e.message}", exception: e)
|
|
139
152
|
end
|
|
140
153
|
|
|
141
154
|
# TODO: probably could merge this with something in cli/plugin
|
|
@@ -27,6 +27,9 @@ module Chronicle
|
|
|
27
27
|
desc 'secrets:COMMAND', 'Manage secrets', hide: true
|
|
28
28
|
subcommand 'secrets', Secrets
|
|
29
29
|
|
|
30
|
+
desc 'authorizations', 'Authorize', hide: true
|
|
31
|
+
subcommand 'authorizations', Authorizations
|
|
32
|
+
|
|
30
33
|
# Entrypoint for the CLI
|
|
31
34
|
def self.start(given_args = ARGV, config = {})
|
|
32
35
|
# take a subcommand:command and splits them so Thor knows how to hand off to the subcommand class
|
|
@@ -54,24 +57,40 @@ module Chronicle
|
|
|
54
57
|
klass, task = ::Thor::Util.find_class_and_task_by_namespace("#{meth}:#{meth}")
|
|
55
58
|
klass.start(['-h', task].compact, shell: shell)
|
|
56
59
|
else
|
|
57
|
-
shell.say "ABOUT".bold
|
|
58
|
-
shell.say " #{'chronicle-etl'.italic} is a
|
|
60
|
+
shell.say "ABOUT:".bold
|
|
61
|
+
shell.say " #{'chronicle-etl'.italic} is a toolkit for extracting and working with your digital"
|
|
62
|
+
shell.say " history. 📜"
|
|
63
|
+
shell.say
|
|
64
|
+
shell.say " A job #{'extracts'.underline} personal data from a source, #{'transforms'.underline} it (Chronicle"
|
|
65
|
+
shell.say " Schema or preserves raw data), and then #{'loads'.underline} it to a destination. Use"
|
|
66
|
+
shell.say " built-in extractors (json, csv, stdin) and loaders (csv, json, table,"
|
|
67
|
+
shell.say " rest) or use plugins to connect to third-party services."
|
|
68
|
+
shell.say
|
|
69
|
+
shell.say " Plugins: https://github.com/chronicle-app/chronicle-etl#currently-available"
|
|
59
70
|
shell.say
|
|
60
|
-
shell.say "USAGE".bold
|
|
61
|
-
shell.say "
|
|
71
|
+
shell.say "USAGE:".bold
|
|
72
|
+
shell.say " # Basic job usage:".italic.light_black
|
|
73
|
+
shell.say " $ chronicle-etl --extractor NAME --transformer NAME --loader NAME"
|
|
62
74
|
shell.say
|
|
63
|
-
shell.say "
|
|
64
|
-
shell.say "
|
|
65
|
-
shell.say " $ chronicle-etl connectors:list"
|
|
75
|
+
shell.say " # Read test.csv and display it to stdout as a table:".italic.light_black
|
|
76
|
+
shell.say " $ chronicle-etl --extractor csv --input data.csv --loader table"
|
|
66
77
|
shell.say
|
|
67
|
-
shell.say "
|
|
68
|
-
shell.say " $ chronicle-etl
|
|
78
|
+
shell.say " # Show available plugins:".italic.light_black
|
|
79
|
+
shell.say " $ chronicle-etl plugins:list"
|
|
69
80
|
shell.say
|
|
70
|
-
shell.say "
|
|
81
|
+
shell.say " # Save an access token as a secret and use it in a job:".italic.light_black
|
|
82
|
+
shell.say " $ chronicle-etl secrets:set pinboard access_token username:foo123"
|
|
83
|
+
shell.say " $ chronicle-etl secrets:list"
|
|
84
|
+
shell.say " $ chronicle-etl -e pinboard --since 1mo"
|
|
85
|
+
shell.say
|
|
86
|
+
shell.say " # Show full job options:".italic.light_black
|
|
71
87
|
shell.say " $ chronicle-etl jobs help run"
|
|
88
|
+
shell.say
|
|
89
|
+
shell.say "FULL DOCUMENTATION:".bold
|
|
90
|
+
shell.say " https://github.com/chronicle-app/chronicle-etl".blue
|
|
91
|
+
shell.say
|
|
72
92
|
|
|
73
93
|
list = []
|
|
74
|
-
|
|
75
94
|
::Thor::Util.thor_classes_in(Chronicle::ETL::CLI).each do |thor_class|
|
|
76
95
|
list += thor_class.printable_tasks(false)
|
|
77
96
|
end
|
|
@@ -79,25 +98,18 @@ module Chronicle
|
|
|
79
98
|
list.unshift ["help", "# This help menu"]
|
|
80
99
|
|
|
81
100
|
shell.say
|
|
82
|
-
shell.say 'ALL COMMANDS'.bold
|
|
101
|
+
shell.say 'ALL COMMANDS:'.bold
|
|
83
102
|
shell.print_table(list, indent: 2, truncate: true)
|
|
84
103
|
shell.say
|
|
85
|
-
shell.say "VERSION".bold
|
|
104
|
+
shell.say "VERSION:".bold
|
|
86
105
|
shell.say " #{Chronicle::ETL::VERSION}"
|
|
87
106
|
shell.say
|
|
88
107
|
shell.say " Display current version:".italic.light_black
|
|
89
108
|
shell.say " $ chronicle-etl --version"
|
|
90
|
-
shell.say
|
|
91
|
-
shell.say "FULL DOCUMENTATION".bold
|
|
92
|
-
shell.say " https://github.com/chronicle-app/chronicle-etl".blue
|
|
93
|
-
shell.say
|
|
94
109
|
end
|
|
95
110
|
end
|
|
96
111
|
|
|
97
112
|
no_commands do
|
|
98
|
-
def testb
|
|
99
|
-
puts "hi"
|
|
100
|
-
end
|
|
101
113
|
def set_color_output
|
|
102
114
|
String.disable_colorization true if options[:'no-color'] || ENV['NO_COLOR']
|
|
103
115
|
end
|
|
@@ -16,7 +16,7 @@ module Chronicle
|
|
|
16
16
|
cli_fail(message: "Please specify a plugin to install") unless plugins.any?
|
|
17
17
|
|
|
18
18
|
installed, not_installed = plugins.partition do |plugin|
|
|
19
|
-
Chronicle::ETL::Registry::
|
|
19
|
+
Chronicle::ETL::Registry::Plugins.installed?(plugin)
|
|
20
20
|
end
|
|
21
21
|
|
|
22
22
|
puts "Already installed: #{installed.join(", ")}" if installed.any?
|
|
@@ -27,7 +27,7 @@ module Chronicle
|
|
|
27
27
|
|
|
28
28
|
not_installed.each do |plugin|
|
|
29
29
|
spinner.update(title: "Installing #{plugin}")
|
|
30
|
-
Chronicle::ETL::Registry::
|
|
30
|
+
Chronicle::ETL::Registry::Plugins.install(plugin)
|
|
31
31
|
|
|
32
32
|
rescue Chronicle::ETL::PluginError => e
|
|
33
33
|
spinner.error("Error".red)
|
|
@@ -41,7 +41,7 @@ module Chronicle
|
|
|
41
41
|
def uninstall(name)
|
|
42
42
|
spinner = TTY::Spinner.new("[:spinner] Uninstalling plugin #{name}...", format: :dots_2)
|
|
43
43
|
spinner.auto_spin
|
|
44
|
-
Chronicle::ETL::Registry::
|
|
44
|
+
Chronicle::ETL::Registry::Plugins.uninstall(name)
|
|
45
45
|
spinner.success("(#{'successful'.green})")
|
|
46
46
|
rescue Chronicle::ETL::PluginError => e
|
|
47
47
|
spinner.error("Error".red)
|
|
@@ -51,20 +51,24 @@ module Chronicle
|
|
|
51
51
|
desc "list", "Lists available plugins"
|
|
52
52
|
# Display all available plugins that chronicle-etl has access to
|
|
53
53
|
def list
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
54
|
+
values = Chronicle::ETL::Registry::Plugins.all
|
|
55
|
+
.map do |plugin|
|
|
56
|
+
[
|
|
57
|
+
plugin.name,
|
|
58
|
+
plugin.description,
|
|
59
|
+
plugin.installed ? '✓' : '',
|
|
60
|
+
plugin.version
|
|
61
|
+
]
|
|
62
62
|
end
|
|
63
63
|
|
|
64
|
-
headers = ['name', 'description', '
|
|
65
|
-
table = TTY::Table.new(headers,
|
|
66
|
-
puts "
|
|
67
|
-
puts table.render(
|
|
64
|
+
headers = ['name', 'description', 'installed', 'version'].map{ |h| h.to_s.upcase.bold }
|
|
65
|
+
table = TTY::Table.new(headers, values)
|
|
66
|
+
puts "Available plugins:"
|
|
67
|
+
puts table.render(
|
|
68
|
+
indent: 2,
|
|
69
|
+
padding: [0, 0],
|
|
70
|
+
alignments: [:left, :left, :center, :left]
|
|
71
|
+
)
|
|
68
72
|
end
|
|
69
73
|
end
|
|
70
74
|
end
|