chronicle-etl 0.5.4 → 0.5.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 8f6b4272cd7f2cfcc12e6327324b5d2f11e76036dcf2442de1fe9ad08e041bb2
4
- data.tar.gz: 4558bf48b7de7c64b691e8ef6403304c864e818360bc7c741a290f7650a7eb8c
3
+ metadata.gz: a2de46efc3c5fbdc7ac120137bef56e13a138c8a95c8dd7d0a3542a65be65959
4
+ data.tar.gz: e8e3e9ae236e270b2926037419d5349170f85b8597c640c0b7b899552257fdb9
5
5
  SHA512:
6
- metadata.gz: 6f7d7f4fd89d284a3a7ad5bffc05cf50ba6ea4e909457585dccfe9071ee91fb36bea505b3fa559a9b6ddab8f37845cf45651c1d1abc4045282c95c99ca9a5944
7
- data.tar.gz: e2cf8a277c463d3b8ddef811e398fbae2e2649fb8227d14874443969b659ee68e74603f438ef4294b5caa59289f26dd5674dd28f9befe537b86b72ecf2ce7b40
6
+ metadata.gz: 58b293d45d1a7f4589aee080d0fb1348e45d76753a8d48bb33e694d6a2b6ea123d1495a12a42e98ed6d3e65926ad48bd7de1fb98dac9bc7bac9225ef00d32fb3
7
+ data.tar.gz: e73c73b67b4e3790347da3df22ed7a11357fdf43e91b734925503f8d0c189aabd07e26f51ed25cf1409fd68ba74dd72e1e7e01c113f773832c509172f0b4ee84
data/README.md CHANGED
@@ -12,7 +12,7 @@ If you don’t want to spend all your time writing scrapers, reverse-engineering
12
12
 
13
13
  ## What does `chronicle-etl` give you?
14
14
  * **A CLI tool for working with personal data**. You can monitor progress of exports, manipulate the output, set up recurring jobs, manage credentials, and more.
15
- * **Plugins for many third-party providers**. This plugin system allows you to access data from dozens of third-party services, all accessible through a common CLI interface.
15
+ * **Plugins for many third-party providers** (see [list](#available-plugins-and-connectors)). This plugin system allows you to access data from dozens of third-party services, all accessible through a common CLI interface.
16
16
  * **A common, opinionated schema**: You can normalize different datasets into a single schema so that, for example, all your iMessages and emails are represented in a common schema. (Don’t want to use this schema? `chronicle-etl` always allows you to fall back on working with the raw extraction data.)
17
17
 
18
18
  ## Chronicle-ETL in action
@@ -51,6 +51,10 @@ $ chronicle-etl --extractor NAME --transformer NAME --loader NAME
51
51
  # Read test.csv and display it to stdout as a table
52
52
  $ chronicle-etl --extractor csv --input data.csv --loader table
53
53
 
54
+ # Show available plugins and install one
55
+ $ chronicle-etl plugins:list
56
+ $ chronicle-etl plugins:install shell
57
+
54
58
  # Retrieve shell commands run in the last 5 hours
55
59
  $ chronicle-etl -e shell --since 5h
56
60
 
@@ -86,35 +90,36 @@ Options:
86
90
  [--silent], [--no-silent] # Silence all output
87
91
  ```
88
92
 
89
- ### Saving jobs
90
-
93
+ ### Saving a job
91
94
  You can save details about a job to a local config file (saved by default in `~/.config/chronicle/etl/jobs/JOB_NAME.yml`) to save yourself the trouble specifying options each time.
92
95
 
93
96
  ```sh
94
97
  # Save a job named 'sample' to ~/.config/chronicle/etl/jobs/sample.yml
95
98
  $ chronicle-etl jobs:save sample --extractor pinboard --since 10d
96
99
 
97
- # Show details about the job
98
- $ chronicle-etl jobs:show sample
99
-
100
100
  # Run the job
101
101
  $ chronicle-etl jobs:run sample
102
102
 
103
+ # Show details about the job
104
+ $ chronicle-etl jobs:show sample
105
+
103
106
  # Show all saved jobs
104
107
  $ chronicle-etl jobs:list
105
108
  ```
106
109
 
107
- ## Connectors
108
- Connectors are available to read, process, and load data from different formats or external services.
110
+ ## Connectors and plugins
111
+
112
+ Connectors let you work with different data formats or third-party providers.
113
+
114
+ ### Built-in Connectors
115
+
116
+ `chronicle-etl` comes with several built-in connectors for common formats and sources.
109
117
 
110
118
  ```sh
111
119
  # List all available connectors
112
120
  $ chronicle-etl connectors:list
113
121
  ```
114
122
 
115
- ### Built-in Connectors
116
- `chronicle-etl` comes with several built-in connectors for common formats and sources.
117
-
118
123
  #### Extractors
119
124
  - [`csv`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/extractors/csv_extractor.rb) - Load records from CSV files or stdin
120
125
  - [`json`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/extractors/json_extractor.rb) - Load JSON (either [line-separated objects](https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON) or one object)
@@ -129,18 +134,19 @@ $ chronicle-etl connectors:list
129
134
  - [`json`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/loaders/json_loader.rb) - Load records serialized as JSON
130
135
  - [`rest`](https://github.com/chronicle-app/chronicle-etl/blob/main/lib/chronicle/etl/loaders/rest_loader.rb) - Serialize records with [JSONAPI](https://jsonapi.org/) and send to a REST API
131
136
 
132
- ## Chronicle Plugins
133
- Plugins provide access to data from third-party platforms, services, or formats. Plugins are packaged as separate rubygems and can be installed through the CLI (under the hood, it's a `gem install chronicle-PLUGINNAME`)
137
+ ### Chronicle Plugins for third-party services
138
+
139
+ Plugins provide access to data from third-party platforms, services, or formats. Plugins are packaged as separate gems and can be installed through the CLI (under the hood, it's a `gem install chronicle-PLUGINNAME`)
134
140
 
135
- ### Plugin usage
141
+ #### Plugin usage
136
142
 
137
143
  ```bash
144
+ # List available plugins
145
+ $ chronicle-etl plugins:list
146
+
138
147
  # Install a plugin
139
148
  $ chronicle-etl plugins:install NAME
140
149
 
141
- # List installed plugins
142
- $ chronicle-etl plugins:list
143
-
144
150
  # Use a plugin
145
151
  $ chronicle-etl plugins:install shell
146
152
  $ chronicle-etl --extractor shell:history --limit 10
@@ -148,33 +154,45 @@ $ chronicle-etl --extractor shell:history --limit 10
148
154
  # Uninstall a plugin
149
155
  $ chronicle-etl plugins:uninstall NAME
150
156
  ```
151
-
152
- ### Status
157
+ #### Available plugins and connectors
158
+
159
+ The following are the officially-supported list of plugins and their available connectors:
160
+
161
+ | Plugin | Type | Identifier | Description | Description |
162
+ |---------------------------------------------------------------------|-------------|------------------|-------------------------------------------|-------------------------------------------|
163
+ | [email](https://github.com/chronicle-app/chronicle-email) | extractor | imap | emails over an IMAP connection | emails over an IMAP connection |
164
+ | [email](https://github.com/chronicle-app/chronicle-email) | extractor | mbox | emails from an .mbox file | emails from an .mbox file |
165
+ | [email](https://github.com/chronicle-app/chronicle-email) | transformer | email | email to Chronicle Schema | email to Chronicle Schema |
166
+ | [foursquare](https://github.com/chronicle-app/chronicle-foursquare) | extractor | checkins | Foursqure visits | Foursqure visits |
167
+ | [foursquare](https://github.com/chronicle-app/chronicle-foursquare) | transformer | checkin | checkin to Chronicle Schema | checkin to Chronicle Schema |
168
+ | [github](https://github.com/chronicle-app/chronicle-github) | extractor | activity | user activity stream | user activity stream |
169
+ | [imessage](https://github.com/chronicle-app/chronicle-imessage) | extractor | messages | imessages from local macOS | imessages from local macOS |
170
+ | [imessage](https://github.com/chronicle-app/chronicle-imessage) | transformer | message | imessage to Chronicle Schema | imessage to Chronicle Schema |
171
+ | [pinboard](https://github.com/chronicle-app/chronicle-pinboard) | extractor | bookmarks | Pinboard.in bookmarks | Pinboard.in bookmarks |
172
+ | [pinboard](https://github.com/chronicle-app/chronicle-pinboard) | transformer | bookmark | bookmark to Chronicle Schema | bookmark to Chronicle Schema |
173
+ | [safari](https://github.com/chronicle-app/chronicle-safari) | extractor | browser-history | browser history | browser history |
174
+ | [safari ](https://github.com/chronicle-app/chronicle-safari ) | transformer | browser-history | browser history to Chronicle Schema | browser history to Chronicle Schema |
175
+ | [shell](https://github.com/chronicle-app/chronicle-shell) | extractor | history | shell command history (bash / zsh) | shell command history (bash / zsh) |
176
+ | [shell](https://github.com/chronicle-app/chronicle-shell) | transformer | command | command to Chronicle Schema | command to Chronicle Schema |
177
+ | [spotify](https://github.com/chronicle-app/chronicle-spotify) | extractor | liked-tracks | liked tracks | liked tracks |
178
+ | [spotify](https://github.com/chronicle-app/chronicle-spotify) | extractor | saved-albums | saved albums | saved albums |
179
+ | [spotify](https://github.com/chronicle-app/chronicle-spotify) | extractor | listens | recently listened tracks (last 50 tracks) | recently listened tracks (last 50 tracks) |
180
+ | [spotify](https://github.com/chronicle-app/chronicle-spotify) | transformer | like | like to Chronicle Schema | like to Chronicle Schema |
181
+ | [spotify](https://github.com/chronicle-app/chronicle-spotify) | transformer | listen | listen to Chronicle Schema | listen to Chronicle Schema |
182
+ | [spotify](https://github.com/chronicle-app/chronicle-spotify) | authorizer | | OAuth authorizer | OAuth authorizer |
183
+ | [zulip](https://github.com/chronicle-app/chronicle-zulip) | extractor | private-messages | private messages | private messages |
184
+ | [zulip](https://github.com/chronicle-app/chronicle-zulip) | transformer | message | message to Chronicle Schema | message to Chronicle Schema |
185
+
186
+
187
+ ### Coming soon
153
188
 
154
189
  A few dozen importers exist [in my Memex project](https://hyfen.net/memex/) and I'm porting them over to the Chronicle system. The [Chronicle Plugin Tracker](https://github.com/orgs/chronicle-app/projects/1/views/1) lets you keep track what's available and what's coming soon.
155
190
 
156
191
  If you don't see a plugin for a third-party provider or data source that you're interested in using with `chronicle-etl`, [please open an issue](https://github.com/chronicle-app/chronicle-etl/issues/new). If you want to work together on a plugin, please [get in touch](#get-in-touch)!
157
192
 
158
- #### Currently available
159
-
160
- | Name | Description | Availability |
161
- |-----------------------------------------------------------------|---------------------------------------------------------------------------------------------|----------------------------------|
162
- | [email](https://github.com/chronicle-app/chronicle-email) | Emails and attachments from IMAP or .mbox files | Available |
163
- | [github](https://github.com/chronicle-app/chronicle-github) | Github activity stream | Available |
164
- | [imessage](https://github.com/chronicle-app/chronicle-imessage) | iMessage messages and attachments | Available |
165
- | [pinboard](https://github.com/chronicle-app/chronicle-email) | Bookmarks and tags | Available |
166
- | [safari](https://github.com/chronicle-app/chronicle-safari) | Browser history from local sqlite db | Available |
167
- | [shell](https://github.com/chronicle-app/chronicle-shell) | Shell command history | Available (still needs zsh support) |
168
- | [zulip](https://github.com/chronicle-app/chronicle-zulip) | Zulip message history | Available (for private messages) |
169
-
170
-
171
- #### Coming soon
172
-
173
193
  In summary, the following **are coming soon**:
174
194
  anki, arc, bear, chrome, facebook, firefox, fitbit, foursquare, git, github, goodreads, google-calendar, images, instagram, lastfm, shazam, slack, strava, things, twitter, whatsapp, youtube.
175
195
 
176
- Please check the [Chronicle Plugin Tracker](https://github.com/orgs/chronicle-app/projects/1/views/1) for details.
177
-
178
196
  ### Writing your own plugin
179
197
 
180
198
  Additional connectors are packaged as separate ruby gems. You can view the [iMessage plugin](https://github.com/chronicle-app/chronicle-imessage) for an example.
@@ -209,6 +227,7 @@ module Chronicle
209
227
  end
210
228
  ```
211
229
 
230
+
212
231
  ## Secrets Management
213
232
 
214
233
  If your job needs secrets such as access tokens or passwords, `chronicle-etl` has a built-in secret management system.
@@ -243,7 +262,6 @@ $ chronicle-etl secrets:unset pinboard access_token
243
262
  ## Roadmap
244
263
 
245
264
  - Keep tackling **new plugins**. See: [Chronicle Plugin Tracker](https://github.com/orgs/chronicle-app/projects/1)
246
- - Add an **OAuth2 authorizer** for services that require this type of authorization ([#48](https://github.com/chronicle-app/chronicle-etl/issues/48))
247
265
  - Add support for **incremental extractions** ([#37](https://github.com/chronicle-app/chronicle-etl/issues/37))
248
266
  - **Improve stdin extractor and shell command transformer** so that users can easily integrate their own scripts/languages/tools into jobs ([#5](https://github.com/chronicle-app/chronicle-etl/issues/48))
249
267
  - **Add documentation for Chronicle Schema**. It's found throughout this project but never explained.
@@ -40,13 +40,14 @@ Gem::Specification.new do |spec|
40
40
  spec.add_dependency "activesupport", "~> 7.0"
41
41
  spec.add_dependency "chronic_duration", "~> 0.10.6"
42
42
  spec.add_dependency "colorize", "~> 0.8.1"
43
- spec.add_dependency 'launchy'
43
+ spec.add_dependency "gems", ">= 1"
44
+ spec.add_dependency "launchy"
44
45
  spec.add_dependency "marcel", "~> 1.0.2"
45
46
  spec.add_dependency "mini_exiftool", "~> 2.10"
46
47
  spec.add_dependency "nokogiri", "~> 1.13"
47
- spec.add_dependency 'omniauth', "~> 2"
48
+ spec.add_dependency "omniauth", "~> 2"
48
49
  spec.add_dependency "sequel", "~> 5.35"
49
- spec.add_dependency 'sinatra', "~> 2"
50
+ spec.add_dependency "sinatra", "~> 2"
50
51
  spec.add_dependency "sqlite3", "~> 1.4"
51
52
  spec.add_dependency "thor", "~> 1.2"
52
53
  spec.add_dependency "thor-hollaback", "~> 0.2"
@@ -57,12 +58,14 @@ Gem::Specification.new do |spec|
57
58
  spec.add_dependency "xdg", ">= 4.0"
58
59
 
59
60
  spec.add_development_dependency "bundler", "~> 2.1"
60
- spec.add_development_dependency "fakefs"
61
+ spec.add_development_dependency "fakefs", "~> 1.4"
61
62
  spec.add_development_dependency "guard-rspec", "~> 4.7.3"
62
63
  spec.add_development_dependency "pry-byebug", "~> 3.9"
63
64
  spec.add_development_dependency "rake", "~> 13.0"
64
65
  spec.add_development_dependency "rspec", "~> 3.9"
65
66
  spec.add_development_dependency "rubocop", "~> 1.25.1"
66
67
  spec.add_development_dependency "simplecov", "~> 0.21"
68
+ spec.add_development_dependency "vcr", "~> 6.1"
69
+ spec.add_development_dependency "webmock", "~> 3"
67
70
  spec.add_development_dependency "yard", "~> 0.9.7"
68
71
  end
@@ -37,12 +37,12 @@ module Chronicle
37
37
 
38
38
  def find_authorizer_klass(provider)
39
39
  # TODO: this assumes provider:plugin one-to-one
40
- unless Chronicle::ETL::Registry::PluginRegistry.installed?(provider)
40
+ unless Chronicle::ETL::Registry::Plugins.installed?(provider)
41
41
  cli_fail(message: "Plugin for #{provider} is not installed.")
42
42
  end
43
43
 
44
44
  begin
45
- Chronicle::ETL::Registry::PluginRegistry.activate(provider)
45
+ Chronicle::ETL::Registry::Plugins.activate(provider)
46
46
  rescue PluginError => e
47
47
  cli_fail(message: "Could not load plugin '#{provider}'.\n" + e.message, exception: e)
48
48
  end
@@ -13,7 +13,7 @@ module Chronicle
13
13
  desc "list", "Lists available connectors"
14
14
  # Display all available connectors that chronicle-etl has access to
15
15
  def list
16
- connector_info = Chronicle::ETL::Registry.connectors.map do |connector_registration|
16
+ connector_info = Chronicle::ETL::Registry::Connectors.connectors.map do |connector_registration|
17
17
  {
18
18
  identifier: connector_registration.identifier,
19
19
  phase: connector_registration.phase,
@@ -43,7 +43,7 @@ module Chronicle
43
43
  end
44
44
 
45
45
  begin
46
- connector = Chronicle::ETL::Registry.find_by_phase_and_identifier(phase.to_sym, identifier)
46
+ connector = Chronicle::ETL::Registry::Connectors.find_by_phase_and_identifier(phase.to_sym, identifier)
47
47
  rescue Chronicle::ETL::ConnectorNotAvailableError, Chronicle::ETL::PluginError => e
48
48
  cli_fail(message: "Could not find #{phase} #{identifier}", exception: e)
49
49
  end
@@ -16,7 +16,7 @@ module Chronicle
16
16
  cli_fail(message: "Please specify a plugin to install") unless plugins.any?
17
17
 
18
18
  installed, not_installed = plugins.partition do |plugin|
19
- Chronicle::ETL::Registry::PluginRegistry.installed?(plugin)
19
+ Chronicle::ETL::Registry::Plugins.installed?(plugin)
20
20
  end
21
21
 
22
22
  puts "Already installed: #{installed.join(", ")}" if installed.any?
@@ -27,7 +27,7 @@ module Chronicle
27
27
 
28
28
  not_installed.each do |plugin|
29
29
  spinner.update(title: "Installing #{plugin}")
30
- Chronicle::ETL::Registry::PluginRegistry.install(plugin)
30
+ Chronicle::ETL::Registry::Plugins.install(plugin)
31
31
 
32
32
  rescue Chronicle::ETL::PluginError => e
33
33
  spinner.error("Error".red)
@@ -41,7 +41,7 @@ module Chronicle
41
41
  def uninstall(name)
42
42
  spinner = TTY::Spinner.new("[:spinner] Uninstalling plugin #{name}...", format: :dots_2)
43
43
  spinner.auto_spin
44
- Chronicle::ETL::Registry::PluginRegistry.uninstall(name)
44
+ Chronicle::ETL::Registry::Plugins.uninstall(name)
45
45
  spinner.success("(#{'successful'.green})")
46
46
  rescue Chronicle::ETL::PluginError => e
47
47
  spinner.error("Error".red)
@@ -51,20 +51,24 @@ module Chronicle
51
51
  desc "list", "Lists available plugins"
52
52
  # Display all available plugins that chronicle-etl has access to
53
53
  def list
54
- plugins = Chronicle::ETL::Registry::PluginRegistry.all_installed_latest
55
-
56
- info = plugins.map do |plugin|
57
- {
58
- name: plugin.name.sub("chronicle-", ""),
59
- description: plugin.description,
60
- version: plugin.version
61
- }
54
+ values = Chronicle::ETL::Registry::Plugins.all
55
+ .map do |plugin|
56
+ [
57
+ plugin.name,
58
+ plugin.description,
59
+ plugin.installed ? '✓' : '',
60
+ plugin.version
61
+ ]
62
62
  end
63
63
 
64
- headers = ['name', 'description', 'version'].map{ |h| h.to_s.upcase.bold }
65
- table = TTY::Table.new(headers, info.map(&:values))
66
- puts "Installed plugins:"
67
- puts table.render(indent: 2, padding: [0, 0])
64
+ headers = ['name', 'description', 'installed', 'version'].map{ |h| h.to_s.upcase.bold }
65
+ table = TTY::Table.new(headers, values)
66
+ puts "Available plugins:"
67
+ puts table.render(
68
+ indent: 2,
69
+ padding: [0, 0],
70
+ alignments: [:left, :left, :center, :left]
71
+ )
68
72
  end
69
73
  end
70
74
  end
@@ -1,3 +1,4 @@
1
+ require "active_support/core_ext/hash/keys"
1
2
  require 'fileutils'
2
3
  require 'yaml'
3
4
 
@@ -21,6 +22,8 @@ module Chronicle
21
22
  def write(type, identifier, data)
22
23
  base = config_pathname_for_type(type)
23
24
  path = base.join("#{identifier}.yml")
25
+
26
+ data.deep_stringify_keys!
24
27
  FileUtils.mkdir_p(File.dirname(path))
25
28
  File.open(path, 'w', 0o600) do |f|
26
29
  # Ruby likes to add --- separators when writing yaml files
@@ -108,6 +108,10 @@ module Chronicle
108
108
  end
109
109
 
110
110
  def coerce_time(value)
111
+ # parsing yml files might result in us getting Date objects
112
+ # we convert to DateTime first to to ensure UTC
113
+ return value.to_datetime.to_time if value.is_a?(Date)
114
+
111
115
  return value unless value.is_a?(String)
112
116
 
113
117
  # Hacky check for duration strings like "60m"
@@ -34,7 +34,7 @@ module Chronicle
34
34
  def validate
35
35
  @errors = {}
36
36
 
37
- Chronicle::ETL::Registry::PHASES.each do |phase|
37
+ Chronicle::ETL::Registry::Connectors::PHASES.each do |phase|
38
38
  __send__("#{phase}_klass".to_sym)
39
39
  rescue Chronicle::ETL::PluginError => e
40
40
  @errors[:plugins] ||= []
@@ -66,7 +66,7 @@ module Chronicle
66
66
 
67
67
  # For each connector in this job, mix in secrets into the options
68
68
  def apply_default_secrets
69
- Chronicle::ETL::Registry::PHASES.each do |phase|
69
+ Chronicle::ETL::Registry::Connectors::PHASES.each do |phase|
70
70
  # If the option have a `secrets` key, we look up those secrets and
71
71
  # mix them in. If not, use the connector's plugin name and look up
72
72
  # secrets with the same namespace
@@ -124,11 +124,11 @@ module Chronicle
124
124
  private
125
125
 
126
126
  def load_klass(phase, identifier)
127
- Chronicle::ETL::Registry.find_by_phase_and_identifier(phase, identifier).klass
127
+ Chronicle::ETL::Registry::Connectors.find_by_phase_and_identifier(phase, identifier).klass
128
128
  end
129
129
 
130
130
  def load_credentials
131
- Chronicle::ETL::Registry::PHASES.each do |phase|
131
+ Chronicle::ETL::Registry::Connectors::PHASES.each do |phase|
132
132
  credentials_name = @definition[phase].dig(:options, :credentials)
133
133
  if credentials_name
134
134
  credentials = Chronicle::ETL::Config.load_credentials(credentials_name)
@@ -41,7 +41,7 @@ module Chronicle
41
41
  end
42
42
 
43
43
  def attach_to_ui(ui_element)
44
- @ui_elemenet = ui_element
44
+ @ui_element = ui_element
45
45
  end
46
46
 
47
47
  def detach_from_ui
@@ -50,7 +50,6 @@ module Chronicle
50
50
  associate_oauth_credentials
51
51
  @server = load_server
52
52
  spinner = TTY::Spinner.new(":spinner :title", format: :dots_2)
53
- Chronicle::ETL::Logger.attach_to_ui(spinner)
54
53
  spinner.auto_spin
55
54
  spinner.update(title: "Starting temporary authorization server on port #{@port}""")
56
55
 
@@ -63,7 +62,6 @@ module Chronicle
63
62
  @server.quit!
64
63
  server_thread.join
65
64
  spinner.success("(#{'successful'.green})")
66
- Chronicle::ETL::Logger.detach_from_ui
67
65
 
68
66
  # TODO: properly handle failed authorizations
69
67
  raise Chronicle::ETL::AuthorizationError unless @server.latest_authorization
@@ -0,0 +1,60 @@
1
+ require 'rubygems'
2
+
3
+ module Chronicle
4
+ module ETL
5
+ module Registry
6
+ # A singleton class that acts as a registry of connector classes available for ETL jobs
7
+ module Connectors
8
+ PHASES = [:extractor, :transformer, :loader].freeze
9
+ public_constant :PHASES
10
+
11
+ class << self
12
+ attr_accessor :connectors
13
+
14
+ def register(connector)
15
+ connectors << connector
16
+ end
17
+
18
+ def connectors
19
+ @connectors ||= []
20
+ end
21
+
22
+ # Find connector from amongst those currently loaded
23
+ def find_by_phase_and_identifier_local(phase, identifier)
24
+ connector = connectors.find { |c| c.phase == phase && c.identifier == identifier }
25
+ end
26
+
27
+ # Find connector and load relevant plugin to find it if necessary
28
+ def find_by_phase_and_identifier(phase, identifier)
29
+ connector = find_by_phase_and_identifier_local(phase, identifier)
30
+ return connector if connector
31
+
32
+ # if not available in built-in connectors, try to activate a
33
+ # relevant plugin and try again
34
+ if identifier.include?(":")
35
+ plugin, name = identifier.split(":")
36
+ else
37
+ # This case handles the case where the identifier is a
38
+ # shorthand (ie `imessage`) because there's only one default
39
+ # connector.
40
+ plugin = identifier
41
+ end
42
+
43
+ raise(Chronicle::ETL::PluginNotInstalledError.new(plugin)) unless Chronicle::ETL::Registry::Plugins.installed?(plugin)
44
+
45
+ Chronicle::ETL::Registry::Plugins.activate(plugin)
46
+
47
+ candidates = connectors.select { |c| c.phase == phase && c.plugin == plugin }
48
+ # if no name given, just use first connector with right phase/plugin
49
+ # TODO: set up a property for connectors to specify that they're the
50
+ # default connector for the plugin
51
+ candidates = candidates.select { |c| c.identifier == name } if name
52
+ connector = candidates.first
53
+
54
+ connector || raise(ConnectorNotAvailableError, "Connector '#{identifier}' not found")
55
+ end
56
+ end
57
+ end
58
+ end
59
+ end
60
+ end
@@ -0,0 +1,19 @@
1
+ module Chronicle
2
+ module ETL
3
+ module Registry
4
+ class PluginRegistration
5
+ attr_accessor :name, :description, :gem, :version, :installed, :gemspec
6
+
7
+ def initialize(name=nil)
8
+ @installed = false
9
+ @name = name
10
+ yield self if block_given?
11
+ end
12
+
13
+ def installed?
14
+ @installed || false
15
+ end
16
+ end
17
+ end
18
+ end
19
+ end
@@ -0,0 +1,163 @@
1
+ require 'rubygems'
2
+ require 'rubygems/command'
3
+ require 'rubygems/commands/install_command'
4
+ require 'rubygems/uninstaller'
5
+ require 'gems'
6
+ require 'active_support/core_ext/hash/deep_merge'
7
+
8
+ module Chronicle
9
+ module ETL
10
+ module Registry
11
+ # Responsible for managing plugins available to chronicle-etl
12
+ #
13
+ # @todo Better validation for whether a gem is actually a plugin
14
+ # @todo Add ways to load a plugin that don't require a gem on rubygems.org
15
+ module Plugins
16
+ KNOWN_PLUGINS = [
17
+ 'email',
18
+ 'foursquare',
19
+ 'github',
20
+ 'imessage',
21
+ 'pinboard',
22
+ 'safari',
23
+ 'shell',
24
+ 'spotify',
25
+ 'zulip'
26
+ ].freeze
27
+ public_constant :KNOWN_PLUGINS
28
+
29
+ # Start of a system for having non-gem plugins. Right now, we just
30
+ # make registry aware of existence of name of non-gem plugin
31
+ def self.register_standalone(name:)
32
+ plugin = Chronicle::ETL::Registry::PluginRegistration.new do |p|
33
+ p.name = name
34
+ p.installed = true
35
+ end
36
+
37
+ installed_standalone << plugin
38
+ end
39
+
40
+ # Plugins either installed as gems or manually loaded/registered
41
+ def self.installed
42
+ installed_standalone + installed_as_gem
43
+ end
44
+
45
+ # Check whether a given plugin is installed
46
+ def self.installed?(name)
47
+ installed.map(&:name).include?(name)
48
+ end
49
+
50
+ # List of plugins installed as standalone
51
+ def self.installed_standalone
52
+ @standalones ||= []
53
+ end
54
+
55
+ # List of plugins installed as gems
56
+ def self.installed_as_gem
57
+ installed_gemspecs_latest.map do |gem|
58
+ Chronicle::ETL::Registry::PluginRegistration.new do |p|
59
+ p.name = gem.name.sub("chronicle-", "")
60
+ p.gem = gem.name
61
+ p.description = gem.description
62
+ p.version = gem.version.to_s
63
+ p.installed = true
64
+ end
65
+ end
66
+ end
67
+
68
+ # List of all plugins available to chronicle-etl
69
+ def self.available
70
+ available_as_gem
71
+ end
72
+
73
+ # List of plugins available through rubygems
74
+ # TODO: make this concurrent
75
+ def self.available_as_gem
76
+ KNOWN_PLUGINS.map do |name|
77
+ info = gem_info(name)
78
+ Chronicle::ETL::Registry::PluginRegistration.new do |p|
79
+ p.name = name
80
+ p.gem = info['name']
81
+ p.version = info['version']
82
+ p.description = info['info']
83
+ end
84
+ end
85
+ end
86
+
87
+ # Load info about a gem plugin from rubygems API
88
+ def self.gem_info(name)
89
+ gem_name = "chronicle-#{name}"
90
+ Gems.info(gem_name)
91
+ end
92
+
93
+ # Union of installed gems (latest version) + available gems
94
+ def self.all
95
+ (installed + available)
96
+ .group_by(&:name)
97
+ .transform_values { |plugin| plugin.find(&:installed) || plugin.first }
98
+ .values
99
+ end
100
+
101
+ # Does a plugin with a given name exist?
102
+ def self.exists?(name)
103
+ KNOWN_PLUGINS.include?(name)
104
+ end
105
+
106
+ # All versions of all plugins currently installed
107
+ def self.installed_gemspecs
108
+ # TODO: add check for chronicle-etl dependency
109
+ Gem::Specification.filter { |s| s.name.match(/^chronicle-/) && s.name != "chronicle-etl" }
110
+ end
111
+
112
+ # Latest version of each installed plugin
113
+ def self.installed_gemspecs_latest
114
+ installed_gemspecs.group_by(&:name)
115
+ .transform_values { |versions| versions.sort_by(&:version).reverse.first }
116
+ .values
117
+ end
118
+
119
+ # Activate a plugin with given name by `require`ing it
120
+ def self.activate(name)
121
+ # By default, activates the latest available version of a gem
122
+ # so don't have to run Kernel#gem separately
123
+ require "chronicle/#{name}"
124
+ rescue Gem::ConflictError => e
125
+ # TODO: figure out if there's more we can do here
126
+ raise Chronicle::ETL::PluginConflictError.new(name), "Plugin '#{name}' couldn't be loaded. #{e.message}"
127
+ rescue StandardError, LoadError => e
128
+ # StandardError to catch random non-loading problems that might occur
129
+ # when requiring the plugin (eg class macro invoked the wrong way)
130
+ # TODO: decide if this should be separated
131
+ raise Chronicle::ETL::PluginLoadError.new(name), "Plugin '#{name}' couldn't be loaded"
132
+ end
133
+
134
+ # Install a plugin to local gems
135
+ def self.install(name)
136
+ return if installed?(name)
137
+ raise(Chronicle::ETL::PluginNotAvailableError.new(name), "Plugin #{name} doesn't exist") unless exists?(name)
138
+
139
+ gem_name = "chronicle-#{name}"
140
+
141
+ Gem::DefaultUserInteraction.ui = Gem::SilentUI.new
142
+ Gem.install(gem_name)
143
+
144
+ activate(name)
145
+ rescue Gem::UnsatisfiableDependencyError
146
+ # TODO: we need to catch a lot more than this here
147
+ raise Chronicle::ETL::PluginNotAvailableError.new(name), "Plugin #{name} could not be installed."
148
+ end
149
+
150
+ # Uninstall a plugin
151
+ def self.uninstall(name)
152
+ gem_name = "chronicle-#{name}"
153
+ Gem::DefaultUserInteraction.ui = Gem::SilentUI.new
154
+ uninstaller = Gem::Uninstaller.new(gem_name)
155
+ uninstaller.uninstall
156
+ rescue Gem::InstallError
157
+ # TODO: strengthen this exception handling
158
+ raise(Chronicle::ETL::PluginError.new(name), "Plugin #{name} wasn't uninstalled")
159
+ end
160
+ end
161
+ end
162
+ end
163
+ end
@@ -1,61 +1,12 @@
1
- require 'rubygems'
2
-
3
1
  module Chronicle
4
2
  module ETL
5
- # A singleton class that acts as a registry of connector classes available for ETL jobs
6
3
  module Registry
7
- PHASES = [:extractor, :transformer, :loader]
8
-
9
- class << self
10
- attr_accessor :connectors
11
-
12
- def register(connector)
13
- connectors << connector
14
- end
15
-
16
- def connectors
17
- @connectors ||= []
18
- end
19
-
20
- # Find connector from amongst those currently loaded
21
- def find_by_phase_and_identifier_local(phase, identifier)
22
- connector = connectors.find { |c| c.phase == phase && c.identifier == identifier }
23
- end
24
-
25
- # Find connector and load relevant plugin to find it if necessary
26
- def find_by_phase_and_identifier(phase, identifier)
27
- connector = find_by_phase_and_identifier_local(phase, identifier)
28
- return connector if connector
29
-
30
- # if not available in built-in connectors, try to activate a
31
- # relevant plugin and try again
32
- if identifier.include?(":")
33
- plugin, name = identifier.split(":")
34
- else
35
- # This case handles the case where the identifier is a
36
- # shorthand (ie `imessage`) because there's only one default
37
- # connector.
38
- plugin = identifier
39
- end
40
-
41
- raise(Chronicle::ETL::PluginNotInstalledError.new(plugin)) unless PluginRegistry.installed?(plugin)
42
-
43
- PluginRegistry.activate(plugin)
44
-
45
- candidates = connectors.select { |c| c.phase == phase && c.plugin == plugin }
46
- # if no name given, just use first connector with right phase/plugin
47
- # TODO: set up a property for connectors to specify that they're the
48
- # default connector for the plugin
49
- candidates = candidates.select { |c| c.identifier == name } if name
50
- connector = candidates.first
51
-
52
- connector || raise(ConnectorNotAvailableError, "Connector '#{identifier}' not found")
53
- end
54
- end
55
4
  end
56
5
  end
57
6
  end
58
7
 
59
8
  require_relative 'self_registering'
60
9
  require_relative 'connector_registration'
61
- require_relative 'plugin_registry'
10
+ require_relative 'connectors'
11
+ require_relative 'plugin_registration'
12
+ require_relative 'plugins'
@@ -17,7 +17,7 @@ module Chronicle
17
17
  def register_connector
18
18
  @connector_registration ||= ::Chronicle::ETL::Registry::ConnectorRegistration.new(self)
19
19
  yield @connector_registration if block_given?
20
- ::Chronicle::ETL::Registry.register(@connector_registration)
20
+ ::Chronicle::ETL::Registry::Connectors.register(@connector_registration)
21
21
  end
22
22
  end
23
23
  end
@@ -56,7 +56,7 @@ module Chronicle
56
56
  data = {
57
57
  secrets: (secrets || {}).transform_keys(&:to_s),
58
58
  chronicle_etl_version: Chronicle::ETL::VERSION
59
- }.deep_stringify_keys
59
+ }
60
60
  Chronicle::ETL::Config.write("secrets", namespace, data)
61
61
  end
62
62
 
@@ -1,5 +1,5 @@
1
1
  module Chronicle
2
2
  module ETL
3
- VERSION = "0.5.4"
3
+ VERSION = "0.5.5"
4
4
  end
5
5
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: chronicle-etl
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.5.4
4
+ version: 0.5.5
5
5
  platform: ruby
6
6
  authors:
7
7
  - Andrew Louis
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2022-04-23 00:00:00.000000000 Z
11
+ date: 2022-05-19 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: activesupport
@@ -52,6 +52,20 @@ dependencies:
52
52
  - - "~>"
53
53
  - !ruby/object:Gem::Version
54
54
  version: 0.8.1
55
+ - !ruby/object:Gem::Dependency
56
+ name: gems
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - ">="
60
+ - !ruby/object:Gem::Version
61
+ version: '1'
62
+ type: :runtime
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - ">="
67
+ - !ruby/object:Gem::Version
68
+ version: '1'
55
69
  - !ruby/object:Gem::Dependency
56
70
  name: launchy
57
71
  requirement: !ruby/object:Gem::Requirement
@@ -280,16 +294,16 @@ dependencies:
280
294
  name: fakefs
281
295
  requirement: !ruby/object:Gem::Requirement
282
296
  requirements:
283
- - - ">="
297
+ - - "~>"
284
298
  - !ruby/object:Gem::Version
285
- version: '0'
299
+ version: '1.4'
286
300
  type: :development
287
301
  prerelease: false
288
302
  version_requirements: !ruby/object:Gem::Requirement
289
303
  requirements:
290
- - - ">="
304
+ - - "~>"
291
305
  - !ruby/object:Gem::Version
292
- version: '0'
306
+ version: '1.4'
293
307
  - !ruby/object:Gem::Dependency
294
308
  name: guard-rspec
295
309
  requirement: !ruby/object:Gem::Requirement
@@ -374,6 +388,34 @@ dependencies:
374
388
  - - "~>"
375
389
  - !ruby/object:Gem::Version
376
390
  version: '0.21'
391
+ - !ruby/object:Gem::Dependency
392
+ name: vcr
393
+ requirement: !ruby/object:Gem::Requirement
394
+ requirements:
395
+ - - "~>"
396
+ - !ruby/object:Gem::Version
397
+ version: '6.1'
398
+ type: :development
399
+ prerelease: false
400
+ version_requirements: !ruby/object:Gem::Requirement
401
+ requirements:
402
+ - - "~>"
403
+ - !ruby/object:Gem::Version
404
+ version: '6.1'
405
+ - !ruby/object:Gem::Dependency
406
+ name: webmock
407
+ requirement: !ruby/object:Gem::Requirement
408
+ requirements:
409
+ - - "~>"
410
+ - !ruby/object:Gem::Version
411
+ version: '3'
412
+ type: :development
413
+ prerelease: false
414
+ version_requirements: !ruby/object:Gem::Requirement
415
+ requirements:
416
+ - - "~>"
417
+ - !ruby/object:Gem::Version
418
+ version: '3'
377
419
  - !ruby/object:Gem::Dependency
378
420
  name: yard
379
421
  requirement: !ruby/object:Gem::Requirement
@@ -454,7 +496,9 @@ files:
454
496
  - lib/chronicle/etl/models/raw.rb
455
497
  - lib/chronicle/etl/oauth_authorizer.rb
456
498
  - lib/chronicle/etl/registry/connector_registration.rb
457
- - lib/chronicle/etl/registry/plugin_registry.rb
499
+ - lib/chronicle/etl/registry/connectors.rb
500
+ - lib/chronicle/etl/registry/plugin_registration.rb
501
+ - lib/chronicle/etl/registry/plugins.rb
458
502
  - lib/chronicle/etl/registry/registry.rb
459
503
  - lib/chronicle/etl/registry/self_registering.rb
460
504
  - lib/chronicle/etl/runner.rb
@@ -1,95 +0,0 @@
1
- require 'rubygems'
2
- require 'rubygems/command'
3
- require 'rubygems/commands/install_command'
4
- require 'rubygems/uninstaller'
5
-
6
- module Chronicle
7
- module ETL
8
- module Registry
9
- # Responsible for managing plugins available to chronicle-etl
10
- #
11
- # @todo Better validation for whether a gem is actually a plugin
12
- # @todo Add ways to load a plugin that don't require a gem on rubygems.org
13
- module PluginRegistry
14
- class << self
15
- # Start of a system for having non-gem plugins. Right now, we just
16
- # make registry aware of existenc of name of non-gem plugin
17
- def register_standalone(name)
18
- standalones << name
19
- end
20
-
21
- def standalones
22
- @standalones ||= []
23
- end
24
- end
25
-
26
- # Does this plugin exist?
27
- def self.exists?(name)
28
- # TODO: implement this. Could query rubygems.org or use a hardcoded
29
- # list somewhere
30
- true
31
- end
32
-
33
- # All versions of all plugins currently installed
34
- def self.all_installed
35
- # TODO: add check for chronicle-etl dependency
36
- Gem::Specification.filter { |s| s.name.match(/^chronicle-/) && s.name != "chronicle-etl" }
37
- end
38
-
39
- # Latest version of each installed plugin
40
- def self.all_installed_latest
41
- all_installed.group_by(&:name)
42
- .transform_values { |versions| versions.sort_by(&:version).reverse.first }
43
- .values
44
- end
45
-
46
- # Check whether a given plugin is installed
47
- def self.installed?(name)
48
- (standalones + all_installed.map { |gem| gem.name.gsub("chronicle-", "") }).include?(name)
49
- end
50
-
51
- # Activate a plugin with given name by `require`ing it
52
- def self.activate(name)
53
- # By default, activates the latest available version of a gem
54
- # so don't have to run Kernel#gem separately
55
- require "chronicle/#{name}"
56
- rescue Gem::ConflictError => e
57
- # TODO: figure out if there's more we can do here
58
- raise Chronicle::ETL::PluginConflictError.new(name), "Plugin '#{name}' couldn't be loaded. #{e.message}"
59
- rescue StandardError, LoadError => e
60
- # StandardError to catch random non-loading problems that might occur
61
- # when requiring the plugin (eg class macro invoked the wrong way)
62
- # TODO: decide if this should be separated
63
- raise Chronicle::ETL::PluginLoadError.new(name), "Plugin '#{name}' couldn't be loaded"
64
- end
65
-
66
- # Install a plugin to local gems
67
- def self.install(name)
68
- return if installed?(name)
69
-
70
- gem_name = "chronicle-#{name}"
71
- raise(Chronicle::ETL::PluginNotAvailableError.new(gem_name), "Plugin #{name} doesn't exist") unless exists?(gem_name)
72
-
73
- Gem::DefaultUserInteraction.ui = Gem::SilentUI.new
74
- Gem.install(gem_name)
75
-
76
- activate(name)
77
- rescue Gem::UnsatisfiableDependencyError
78
- # TODO: we need to catch a lot more than this here
79
- raise Chronicle::ETL::PluginNotAvailableError.new(name), "Plugin #{name} could not be installed."
80
- end
81
-
82
- # Uninstall a plugin
83
- def self.uninstall(name)
84
- gem_name = "chronicle-#{name}"
85
- Gem::DefaultUserInteraction.ui = Gem::SilentUI.new
86
- uninstaller = Gem::Uninstaller.new(gem_name)
87
- uninstaller.uninstall
88
- rescue Gem::InstallError
89
- # TODO: strengthen this exception handling
90
- raise(Chronicle::ETL::PluginError.new(name), "Plugin #{name} wasn't uninstalled")
91
- end
92
- end
93
- end
94
- end
95
- end