journaled 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/LICENSE +19 -0
- data/README.md +194 -0
- data/Rakefile +32 -0
- data/app/controllers/concerns/journaled/actor.rb +18 -0
- data/app/models/concerns/journaled/changes.rb +38 -0
- data/app/models/journaled/change.rb +34 -0
- data/app/models/journaled/change_definition.rb +8 -0
- data/app/models/journaled/change_writer.rb +81 -0
- data/app/models/journaled/delivery.rb +61 -0
- data/app/models/journaled/event.rb +65 -0
- data/app/models/journaled/job_priority.rb +5 -0
- data/app/models/journaled/json_schema_model/validator.rb +38 -0
- data/app/models/journaled/not_truly_exceptional_error.rb +2 -0
- data/app/models/journaled/writer.rb +63 -0
- data/config/routes.rb +2 -0
- data/lib/journaled.rb +24 -0
- data/lib/journaled/engine.rb +4 -0
- data/lib/journaled/version.rb +3 -0
- metadata +238 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 067a7a010e80a13c5a9204e8e761e949af52243e
|
4
|
+
data.tar.gz: e5e072be8c838c63dfbbc082b51d709ed11eef6a
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 93726fa7a7c41e57cb7c7a73c6d5bee57d83562e7d52e51306cb1fc0bc25760ca959c5ba256c5415be8df0c45084486eea809372d701d33b5386e3d0dbba5b2c
|
7
|
+
data.tar.gz: 9c8aea433e5c9ffbdd5039036ea860a9c8cd464ed29fa9330bbbb936ac725b97e3b1c98eb20ea4e612ea33bddf926b80bceb02c412e55f2d0dca78ac3472d666
|
data/LICENSE
ADDED
@@ -0,0 +1,19 @@
|
|
1
|
+
Copyright (c) 2017-2019 Betterment
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy of
|
4
|
+
this software and associated documentation files (the "Software"), to deal in
|
5
|
+
the Software without restriction, including without limitation the rights to
|
6
|
+
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
|
7
|
+
of the Software, and to permit persons to whom the Software is furnished to do
|
8
|
+
so, subject to the following conditions:
|
9
|
+
|
10
|
+
The above copyright notice and this permission notice shall be included in all
|
11
|
+
copies or substantial portions of the Software.
|
12
|
+
|
13
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
14
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
15
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
16
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
17
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
18
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
19
|
+
SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,194 @@
|
|
1
|
+
# Journaled
|
2
|
+
|
3
|
+
A Rails engine to durably deliver schematized events to Amazon Kinesis via DelayedJob.
|
4
|
+
|
5
|
+
More specifically, `journaled` is composed of three opinionated pieces:
|
6
|
+
schema definition/validation via JSON Schema, transactional enqueueing
|
7
|
+
via Delayed::Job (specifically `delayed_job_active_record`), and event
|
8
|
+
transmission via Amazon Kinesis. Our current use-cases include
|
9
|
+
transmitting audit events for durable storage in S3 and/or analytical
|
10
|
+
querying in Amazon Redshift.
|
11
|
+
|
12
|
+
Journaled provides an at-least-once event delivery guarantee assuming
|
13
|
+
Delayed::Job is configured not to delete jobs on failure.
|
14
|
+
|
15
|
+
Note: Do not use the journaled gem to build an event sourcing solution
|
16
|
+
as it does not guarantee total ordering of events. It's possible we'll
|
17
|
+
add scoped ordering capability at a future date (and would gladly
|
18
|
+
entertain pull requests), but it is presently only designed to provide a
|
19
|
+
durable, eventually consistent record that discrete events happened.
|
20
|
+
|
21
|
+
## Installation
|
22
|
+
|
23
|
+
1. [Install `delayed_job_active_record`](https://github.com/collectiveidea/delayed_job_active_record#installation)
|
24
|
+
if you haven't already.
|
25
|
+
|
26
|
+
|
27
|
+
2. To integrate Journaled into your application, simply include the gem in your
|
28
|
+
app's Gemfile.
|
29
|
+
```
|
30
|
+
gem 'journaled', https_github: 'Betterment/journaled'
|
31
|
+
```
|
32
|
+
3. You will also need to define the following environment variables to allow Journaled to publish events to your AWS Kinesis event stream:
|
33
|
+
|
34
|
+
* `JOURNALED_STREAM_NAME`
|
35
|
+
|
36
|
+
Special case: if your `Journaled::Event` objects override the
|
37
|
+
`#journaled_app_name` method to a non-nil value e.g. `my_app`, you will
|
38
|
+
instead need to provide a corresponding
|
39
|
+
`[upcased_app_name]_JOURNALED_STREAM_NAME` variable for each distinct
|
40
|
+
value, e.g. `MY_APP_JOURNALED_STREAM_NAME`. You can provide a default value
|
41
|
+
for all `Journaled::Event`s in an initializer like this:
|
42
|
+
|
43
|
+
```ruby
|
44
|
+
Journaled.default_app_name = 'my_app'
|
45
|
+
```
|
46
|
+
|
47
|
+
You may optionally define the following ENV vars to specify AWS
|
48
|
+
credentials outside of the locations that the AWS SDK normally looks:
|
49
|
+
|
50
|
+
* `RUBY_AWS_ACCESS_KEY_ID`
|
51
|
+
* `RUBY_AWS_SECRET_ACCESS_KEY`
|
52
|
+
|
53
|
+
You may also specify the region to target your AWS stream by setting
|
54
|
+
`AWS_DEFAULT_REGION`. If you don't specify, Journaled will default to
|
55
|
+
`us-east-1`.
|
56
|
+
|
57
|
+
Journaled::Event provides a `commit_hash` method which you may journal
|
58
|
+
if you like. If you choose to use it, you must provide a `GIT_COMMIT`
|
59
|
+
environment variable.
|
60
|
+
|
61
|
+
## Usage
|
62
|
+
|
63
|
+
### Change Journaling
|
64
|
+
|
65
|
+
Out of the box, `Journaled` provides an event type and ActiveRecord
|
66
|
+
mix-in for durably journaling changes to your model, implemented via
|
67
|
+
ActiveRecord hooks. Use it like so:
|
68
|
+
|
69
|
+
```
|
70
|
+
class User < ApplicationRecord
|
71
|
+
include Journaled::Changes
|
72
|
+
|
73
|
+
journal_changes_to :email, :first_name, :last_name, as: :identity_change
|
74
|
+
end
|
75
|
+
```
|
76
|
+
|
77
|
+
Add the following to your controller base class for attribution:
|
78
|
+
|
79
|
+
```
|
80
|
+
class ApplicationController < ActionController::Base
|
81
|
+
include Journaled::Actor
|
82
|
+
|
83
|
+
self.journaled_actor = :current_user # Or your authenticated entity
|
84
|
+
end
|
85
|
+
```
|
86
|
+
|
87
|
+
Your authenticated entity must respond to `#to_global_id`, which
|
88
|
+
ActiveRecords do by default.
|
89
|
+
|
90
|
+
Every time any of the specified attributes is modified, or a `User`
|
91
|
+
record is created or destroyed, an event will be sent to Kinesis with the following attributes:
|
92
|
+
|
93
|
+
* `id` - a random event-specific UUID
|
94
|
+
* `event_type` - the constant value `journaled_change`
|
95
|
+
* `created_at`- when the event was created
|
96
|
+
* `table_name` - the table name backing the ActiveRecord (e.g. `users`)
|
97
|
+
* `record_id` - the primary key of the record, as a string (e.g.
|
98
|
+
`"300"`)
|
99
|
+
* `database_operation` - one of `create`, `update`, `delete`
|
100
|
+
* `logical_operation` - whatever logical operation you specified in
|
101
|
+
your `journal_changes_to` declaration (e.g. `identity_change`)
|
102
|
+
* `changes` - a serialized JSON object representing the latest values
|
103
|
+
of any new or changed attributes from the specified set (e.g.
|
104
|
+
`{"email":"mynewemail@example.com"}`)
|
105
|
+
* `actor` - a string (usually a rails global_id) representing who
|
106
|
+
performed the action.
|
107
|
+
|
108
|
+
### Custom Journaling
|
109
|
+
|
110
|
+
For every custom implementation of journaling in your application, define the JSON schema for the attributes in your event.
|
111
|
+
This schema file should live in your Rails application at the top level and should be named in snake case to match the
|
112
|
+
class being journaled.
|
113
|
+
E.g.: `your_app/journaled_schemas/my_class.json)`
|
114
|
+
|
115
|
+
In each class you intend to use Journaled, include the `Journaled::Event` module and define the attributes you want
|
116
|
+
captured. After completing the above steps, you can call the `journal!` method in the model code and the declared
|
117
|
+
attributes will be published to the Kinesis stream. Be sure to call
|
118
|
+
`journal!` within the same transaction as any database side effects of
|
119
|
+
your business logic operation to ensure that the event will eventually
|
120
|
+
be delivered if-and-only-if your transaction commits.
|
121
|
+
|
122
|
+
Example:
|
123
|
+
|
124
|
+
```js
|
125
|
+
// journaled_schemas/contract_acceptance_event.json
|
126
|
+
|
127
|
+
{
|
128
|
+
"type": "object",
|
129
|
+
"title": "contract_acceptance_event",
|
130
|
+
"required": [
|
131
|
+
"user_id",
|
132
|
+
"signature"
|
133
|
+
],
|
134
|
+
"properties": {
|
135
|
+
"user_id": {
|
136
|
+
"type": "integer"
|
137
|
+
},
|
138
|
+
"signature": {
|
139
|
+
"type": "string"
|
140
|
+
}
|
141
|
+
}
|
142
|
+
}
|
143
|
+
```
|
144
|
+
|
145
|
+
```ruby
|
146
|
+
# app/models/contract_acceptance_event.rb
|
147
|
+
|
148
|
+
ContractAcceptanceEvent = Struct.new(:user_id, :signature) do
|
149
|
+
include Journaled::Event
|
150
|
+
|
151
|
+
journal_attributes :user_id, :signature
|
152
|
+
end
|
153
|
+
```
|
154
|
+
|
155
|
+
```ruby
|
156
|
+
# app/models/contract_acceptance.rb
|
157
|
+
|
158
|
+
class ContractAcceptance
|
159
|
+
include ActiveModel::Model
|
160
|
+
|
161
|
+
attr_accessor :user_id, :signature
|
162
|
+
|
163
|
+
def user
|
164
|
+
@user ||= User.find(user_id)
|
165
|
+
end
|
166
|
+
|
167
|
+
def contract_acceptance_event
|
168
|
+
@contract_acceptance_event ||= ContractAcceptanceEvent.new(user_id, signature)
|
169
|
+
end
|
170
|
+
|
171
|
+
def save!
|
172
|
+
User.transaction do
|
173
|
+
user.update!(contract_accepted: true)
|
174
|
+
contract_acceptance_event.journal!
|
175
|
+
end
|
176
|
+
end
|
177
|
+
end
|
178
|
+
```
|
179
|
+
|
180
|
+
An event like the following will be journaled to kinesis:
|
181
|
+
|
182
|
+
```js
|
183
|
+
{
|
184
|
+
"id": "bc7cb6a6-88cf-4849-a4f0-a31b0b199c47", // A random event ID for idempotency filtering
|
185
|
+
"event_type": "contract_acceptance_event",
|
186
|
+
"created_at": "2019-01-28T11:06:54.928-05:00",
|
187
|
+
"user_id": 123,
|
188
|
+
"signature": "Sarah T. User"
|
189
|
+
}
|
190
|
+
```
|
191
|
+
|
192
|
+
## Future improvements & issue tracking
|
193
|
+
Suggestions for enhancements to this engine are currently being tracked via Github Issues. Please feel free to open an
|
194
|
+
issue for a desired feature, as well as for any observed bugs.
|
data/Rakefile
ADDED
@@ -0,0 +1,32 @@
|
|
1
|
+
begin
|
2
|
+
require 'bundler/setup'
|
3
|
+
rescue LoadError
|
4
|
+
puts 'You must `gem install bundler` and `bundle install` to run rake tasks'
|
5
|
+
end
|
6
|
+
|
7
|
+
require 'rdoc/task'
|
8
|
+
|
9
|
+
RDoc::Task.new(:rdoc) do |rdoc|
|
10
|
+
rdoc.rdoc_dir = 'rdoc'
|
11
|
+
rdoc.title = 'Journaled'
|
12
|
+
rdoc.options << '--line-numbers'
|
13
|
+
rdoc.rdoc_files.include('README.rdoc')
|
14
|
+
rdoc.rdoc_files.include('lib/**/*.rb')
|
15
|
+
end
|
16
|
+
|
17
|
+
APP_RAKEFILE = File.expand_path('spec/dummy/Rakefile', __dir__)
|
18
|
+
load 'rails/tasks/engine.rake'
|
19
|
+
|
20
|
+
Bundler::GemHelper.install_tasks
|
21
|
+
|
22
|
+
if %w(development test).include? Rails.env
|
23
|
+
require 'rspec/core'
|
24
|
+
require 'rspec/core/rake_task'
|
25
|
+
RSpec::Core::RakeTask.new
|
26
|
+
|
27
|
+
require 'rubocop/rake_task'
|
28
|
+
RuboCop::RakeTask.new
|
29
|
+
|
30
|
+
task(:default).clear
|
31
|
+
task default: %i(rubocop spec)
|
32
|
+
end
|
@@ -0,0 +1,18 @@
|
|
1
|
+
module Journaled::Actor
|
2
|
+
extend ActiveSupport::Concern
|
3
|
+
|
4
|
+
included do
|
5
|
+
class_attribute :_journaled_actor_method_name, instance_accessor: false, instance_predicate: false
|
6
|
+
before_action do
|
7
|
+
RequestStore.store[:journaled_actor_proc] = self.class._journaled_actor_method_name &&
|
8
|
+
-> { send(self.class._journaled_actor_method_name) }
|
9
|
+
end
|
10
|
+
end
|
11
|
+
|
12
|
+
class_methods do
|
13
|
+
def journaled_actor=(method_name)
|
14
|
+
raise "Must provide a symbol method name" unless method_name.is_a?(Symbol)
|
15
|
+
self._journaled_actor_method_name = method_name
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
@@ -0,0 +1,38 @@
|
|
1
|
+
module Journaled::Changes
|
2
|
+
extend ActiveSupport::Concern
|
3
|
+
|
4
|
+
included do
|
5
|
+
cattr_accessor :_journaled_change_definitions
|
6
|
+
self._journaled_change_definitions = []
|
7
|
+
|
8
|
+
after_create do
|
9
|
+
self.class._journaled_change_definitions.each do |definition|
|
10
|
+
Journaled::ChangeWriter.new(model: self, change_definition: definition).create
|
11
|
+
end
|
12
|
+
end
|
13
|
+
|
14
|
+
after_save unless: :saved_change_to_id? do
|
15
|
+
self.class._journaled_change_definitions.each do |definition|
|
16
|
+
Journaled::ChangeWriter.new(model: self, change_definition: definition).update
|
17
|
+
end
|
18
|
+
end
|
19
|
+
|
20
|
+
after_destroy do
|
21
|
+
self.class._journaled_change_definitions.each do |definition|
|
22
|
+
Journaled::ChangeWriter.new(model: self, change_definition: definition).delete
|
23
|
+
end
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
class_methods do
|
28
|
+
def journal_changes_to(*attribute_names, as:) # rubocop:disable Naming/UncommunicativeMethodParamName
|
29
|
+
if attribute_names.empty? || attribute_names.any? { |n| !n.is_a?(Symbol) }
|
30
|
+
raise "one or more symbol attribute_name arguments is required"
|
31
|
+
end
|
32
|
+
|
33
|
+
raise "as: must be a symbol" unless as.is_a?(Symbol)
|
34
|
+
|
35
|
+
_journaled_change_definitions << Journaled::ChangeDefinition.new(attribute_names: attribute_names, logical_operation: as)
|
36
|
+
end
|
37
|
+
end
|
38
|
+
end
|
@@ -0,0 +1,34 @@
|
|
1
|
+
class Journaled::Change
|
2
|
+
include Journaled::Event
|
3
|
+
|
4
|
+
attr_reader :table_name,
|
5
|
+
:record_id,
|
6
|
+
:database_operation,
|
7
|
+
:logical_operation,
|
8
|
+
:changes,
|
9
|
+
:journaled_app_name,
|
10
|
+
:actor
|
11
|
+
|
12
|
+
journal_attributes :table_name,
|
13
|
+
:record_id,
|
14
|
+
:database_operation,
|
15
|
+
:logical_operation,
|
16
|
+
:changes,
|
17
|
+
:actor
|
18
|
+
|
19
|
+
def initialize(table_name:, # rubocop:disable Metrics/ParameterLists
|
20
|
+
record_id:,
|
21
|
+
database_operation:,
|
22
|
+
logical_operation:,
|
23
|
+
changes:,
|
24
|
+
journaled_app_name:,
|
25
|
+
actor:)
|
26
|
+
@table_name = table_name
|
27
|
+
@record_id = record_id
|
28
|
+
@database_operation = database_operation
|
29
|
+
@logical_operation = logical_operation
|
30
|
+
@changes = changes
|
31
|
+
@journaled_app_name = journaled_app_name
|
32
|
+
@actor = actor
|
33
|
+
end
|
34
|
+
end
|
@@ -0,0 +1,81 @@
|
|
1
|
+
class Journaled::ChangeWriter
|
2
|
+
attr_reader :model, :change_definition
|
3
|
+
delegate :logical_operation, to: :change_definition
|
4
|
+
|
5
|
+
def initialize(model:, change_definition:)
|
6
|
+
@model = model
|
7
|
+
@change_definition = change_definition
|
8
|
+
end
|
9
|
+
|
10
|
+
def attribute_names
|
11
|
+
@attribute_names ||= change_definition.attribute_names.map(&:to_s)
|
12
|
+
end
|
13
|
+
|
14
|
+
def create
|
15
|
+
journaled_change_for("create", relevant_attributes).journal!
|
16
|
+
end
|
17
|
+
|
18
|
+
def update
|
19
|
+
journaled_change_for("update", relevant_changes).journal! if relevant_changes.present?
|
20
|
+
end
|
21
|
+
|
22
|
+
def delete
|
23
|
+
journaled_change_for("delete", {}).journal!
|
24
|
+
end
|
25
|
+
|
26
|
+
def journaled_change_for(database_operation, changes)
|
27
|
+
Journaled::Change.new(
|
28
|
+
table_name: model.class.table_name,
|
29
|
+
record_id: model.id.to_s,
|
30
|
+
database_operation: database_operation,
|
31
|
+
logical_operation: logical_operation,
|
32
|
+
changes: JSON.dump(changes),
|
33
|
+
journaled_app_name: journaled_app_name,
|
34
|
+
actor: actor_uri
|
35
|
+
)
|
36
|
+
end
|
37
|
+
|
38
|
+
def relevant_attributes
|
39
|
+
model.attributes.slice(*attribute_names)
|
40
|
+
end
|
41
|
+
|
42
|
+
def relevant_changes
|
43
|
+
relevant_changes_with_previous_values.each_with_object({}) do |(k, v), result|
|
44
|
+
result[k] = v[1]
|
45
|
+
end
|
46
|
+
end
|
47
|
+
|
48
|
+
def actor_uri
|
49
|
+
actor_global_id_uri || fallback_global_id_uri
|
50
|
+
end
|
51
|
+
|
52
|
+
private
|
53
|
+
|
54
|
+
def actor_global_id_uri
|
55
|
+
actor.to_global_id.to_s if actor
|
56
|
+
end
|
57
|
+
|
58
|
+
def actor
|
59
|
+
@actor ||= RequestStore.store[:journaled_actor_proc]&.call
|
60
|
+
end
|
61
|
+
|
62
|
+
def fallback_global_id_uri
|
63
|
+
if defined?(::Rails::Console) || File.basename($PROGRAM_NAME) == "rake"
|
64
|
+
"gid://local/#{Etc.getlogin}"
|
65
|
+
else
|
66
|
+
"gid://#{Rails.application.config.global_id.app}"
|
67
|
+
end
|
68
|
+
end
|
69
|
+
|
70
|
+
def relevant_changes_with_previous_values
|
71
|
+
model.saved_changes.slice(*attribute_names)
|
72
|
+
end
|
73
|
+
|
74
|
+
def journaled_app_name
|
75
|
+
if model.class.respond_to?(:journaled_app_name)
|
76
|
+
model.class.journaled_app_name
|
77
|
+
else
|
78
|
+
Journaled.default_app_name
|
79
|
+
end
|
80
|
+
end
|
81
|
+
end
|
@@ -0,0 +1,61 @@
|
|
1
|
+
class Journaled::Delivery
|
2
|
+
DEFAULT_REGION = 'us-east-1'.freeze
|
3
|
+
|
4
|
+
def initialize(serialized_event:, partition_key:, app_name:)
|
5
|
+
@serialized_event = serialized_event
|
6
|
+
@partition_key = partition_key
|
7
|
+
@app_name = app_name
|
8
|
+
end
|
9
|
+
|
10
|
+
def perform
|
11
|
+
kinesis_client.put_record record if Journaled.enabled?
|
12
|
+
rescue Aws::Kinesis::Errors::InternalFailure, Aws::Kinesis::Errors::ServiceUnavailable, Aws::Kinesis::Errors::Http503Error => e
|
13
|
+
Rails.logger.error "Kinesis Error - Server Error occurred - #{e.class}"
|
14
|
+
raise KinesisTemporaryFailure
|
15
|
+
rescue Seahorse::Client::NetworkingError => e
|
16
|
+
Rails.logger.error "Kinesis Error - Networking Error occurred - #{e.class}"
|
17
|
+
raise KinesisTemporaryFailure
|
18
|
+
end
|
19
|
+
|
20
|
+
def stream_name
|
21
|
+
env_var_name = [app_name&.upcase, 'JOURNALED_STREAM_NAME'].compact.join('_')
|
22
|
+
ENV.fetch(env_var_name)
|
23
|
+
end
|
24
|
+
|
25
|
+
def kinesis_client_config
|
26
|
+
{
|
27
|
+
region: ENV.fetch('AWS_DEFAULT_REGION', DEFAULT_REGION),
|
28
|
+
retry_limit: 0
|
29
|
+
}.merge(legacy_credentials_hash)
|
30
|
+
end
|
31
|
+
|
32
|
+
private
|
33
|
+
|
34
|
+
attr_reader :serialized_event, :partition_key, :app_name
|
35
|
+
|
36
|
+
def record
|
37
|
+
{
|
38
|
+
stream_name: stream_name,
|
39
|
+
data: serialized_event,
|
40
|
+
partition_key: partition_key
|
41
|
+
}
|
42
|
+
end
|
43
|
+
|
44
|
+
def kinesis_client
|
45
|
+
@kinesis_client ||= Aws::Kinesis::Client.new(kinesis_client_config)
|
46
|
+
end
|
47
|
+
|
48
|
+
def legacy_credentials_hash
|
49
|
+
if ENV.key?('RUBY_AWS_ACCESS_KEY_ID')
|
50
|
+
{
|
51
|
+
access_key_id: ENV.fetch('RUBY_AWS_ACCESS_KEY_ID'),
|
52
|
+
secret_access_key: ENV.fetch('RUBY_AWS_SECRET_ACCESS_KEY')
|
53
|
+
}
|
54
|
+
else
|
55
|
+
{}
|
56
|
+
end
|
57
|
+
end
|
58
|
+
|
59
|
+
class KinesisTemporaryFailure < NotTrulyExceptionalError
|
60
|
+
end
|
61
|
+
end
|
@@ -0,0 +1,65 @@
|
|
1
|
+
module Journaled::Event
|
2
|
+
extend ActiveSupport::Concern
|
3
|
+
|
4
|
+
def journal!
|
5
|
+
Journaled::Writer.new(journaled_event: self).journal!
|
6
|
+
end
|
7
|
+
|
8
|
+
# Base attributes
|
9
|
+
|
10
|
+
def id
|
11
|
+
@id ||= SecureRandom.uuid
|
12
|
+
end
|
13
|
+
|
14
|
+
def event_type
|
15
|
+
@event_type ||= self.class.event_type
|
16
|
+
end
|
17
|
+
|
18
|
+
def created_at
|
19
|
+
@created_at ||= Time.zone.now
|
20
|
+
end
|
21
|
+
|
22
|
+
def commit_hash
|
23
|
+
@commit_hash ||= ENV.fetch('GIT_COMMIT')
|
24
|
+
end
|
25
|
+
|
26
|
+
# Event metadata and configuration (not serialized)
|
27
|
+
|
28
|
+
def journaled_schema_name
|
29
|
+
self.class.to_s.underscore
|
30
|
+
end
|
31
|
+
|
32
|
+
def journaled_attributes
|
33
|
+
self.class.public_send(:journaled_attributes).each_with_object({}) do |attribute, memo|
|
34
|
+
memo[attribute] = public_send(attribute)
|
35
|
+
end
|
36
|
+
end
|
37
|
+
|
38
|
+
def journaled_partition_key
|
39
|
+
event_type
|
40
|
+
end
|
41
|
+
|
42
|
+
def journaled_app_name
|
43
|
+
Journaled.default_app_name
|
44
|
+
end
|
45
|
+
|
46
|
+
private
|
47
|
+
|
48
|
+
class_methods do
|
49
|
+
def journal_attributes(*args)
|
50
|
+
journaled_attributes.concat(args)
|
51
|
+
end
|
52
|
+
|
53
|
+
def journaled_attributes
|
54
|
+
@journaled_attributes ||= []
|
55
|
+
end
|
56
|
+
|
57
|
+
def event_type
|
58
|
+
name.underscore.parameterize(separator: '_')
|
59
|
+
end
|
60
|
+
end
|
61
|
+
|
62
|
+
included do
|
63
|
+
journal_attributes :id, :event_type, :created_at
|
64
|
+
end
|
65
|
+
end
|
@@ -0,0 +1,5 @@
|
|
1
|
+
module Journaled::JobPriority
|
2
|
+
INTERACTIVE = 0 # These jobs will actively hinder end-user interactions until complete, e.g. assembling a report a user is polling for.
|
3
|
+
USER_VISIBLE = 10 # These jobs have end-user-visible side effects that will not obviously impact customers, e.g. welcome emails
|
4
|
+
EVENTUAL = 20 # These jobs affect business process that are tolerant to some degree of queue backlog, e.g. desk record synchronization
|
5
|
+
end
|
@@ -0,0 +1,38 @@
|
|
1
|
+
class Journaled::JsonSchemaModel::Validator
|
2
|
+
def initialize(schema_name)
|
3
|
+
@schema_name = schema_name
|
4
|
+
end
|
5
|
+
|
6
|
+
def validate!(json_to_validate)
|
7
|
+
JSON::Validator.validate!(json_schema, json_to_validate)
|
8
|
+
end
|
9
|
+
|
10
|
+
private
|
11
|
+
|
12
|
+
attr_reader :schema_name
|
13
|
+
|
14
|
+
def json_schema
|
15
|
+
@json_schema ||= JSON.parse(json_schema_file)
|
16
|
+
end
|
17
|
+
|
18
|
+
def json_schema_file
|
19
|
+
@json_schema_file ||= File.read(json_schema_path)
|
20
|
+
end
|
21
|
+
|
22
|
+
def json_schema_path
|
23
|
+
@json_schema_path ||= gem_paths.detect { |path| File.exist?(path) } || raise(<<~ERROR)
|
24
|
+
journaled_schemas/#{schema_name}.json not found in any of #{Journaled.schema_providers.map { |sp| "#{sp}.root" }.join(', ')}
|
25
|
+
|
26
|
+
You can add schema providers as follows:
|
27
|
+
|
28
|
+
# config/initializers/journaled.rb
|
29
|
+
Journaled.schema_providers << MyGem::Engine
|
30
|
+
ERROR
|
31
|
+
end
|
32
|
+
|
33
|
+
def gem_paths
|
34
|
+
Journaled.schema_providers.map do |engine|
|
35
|
+
engine.root.join "journaled_schemas/#{schema_name}.json"
|
36
|
+
end
|
37
|
+
end
|
38
|
+
end
|
@@ -0,0 +1,63 @@
|
|
1
|
+
class Journaled::Writer
|
2
|
+
EVENT_METHOD_NAMES = %i(
|
3
|
+
journaled_schema_name
|
4
|
+
journaled_partition_key
|
5
|
+
journaled_attributes
|
6
|
+
journaled_app_name
|
7
|
+
).freeze
|
8
|
+
|
9
|
+
def initialize(journaled_event:, priority: Journaled::JobPriority::EVENTUAL)
|
10
|
+
raise "An enqueued event must respond to: #{EVENT_METHOD_NAMES.to_sentence}" unless respond_to_all?(journaled_event, EVENT_METHOD_NAMES)
|
11
|
+
|
12
|
+
unless journaled_event.journaled_schema_name.present? &&
|
13
|
+
journaled_event.journaled_partition_key.present? &&
|
14
|
+
journaled_event.journaled_attributes.present?
|
15
|
+
raise <<~ERROR
|
16
|
+
An enqueued event must have a non-nil response to:
|
17
|
+
#json_schema_name,
|
18
|
+
#partition_key, and
|
19
|
+
#journaled_attributes
|
20
|
+
ERROR
|
21
|
+
end
|
22
|
+
|
23
|
+
@journaled_event = journaled_event
|
24
|
+
@priority = priority
|
25
|
+
end
|
26
|
+
|
27
|
+
def journal!
|
28
|
+
base_event_json_schema_validator.validate! serialized_event
|
29
|
+
json_schema_validator.validate! serialized_event
|
30
|
+
Delayed::Job.enqueue journaled_delivery, priority: priority
|
31
|
+
end
|
32
|
+
|
33
|
+
private
|
34
|
+
|
35
|
+
attr_reader :journaled_event, :priority
|
36
|
+
delegate :journaled_schema_name, :journaled_attributes, :journaled_partition_key, :journaled_app_name, to: :journaled_event
|
37
|
+
|
38
|
+
def journaled_delivery
|
39
|
+
@journaled_delivery ||= Journaled::Delivery.new(
|
40
|
+
serialized_event: serialized_event,
|
41
|
+
partition_key: journaled_partition_key,
|
42
|
+
app_name: journaled_app_name
|
43
|
+
)
|
44
|
+
end
|
45
|
+
|
46
|
+
def serialized_event
|
47
|
+
@serialized_event ||= journaled_attributes.to_json
|
48
|
+
end
|
49
|
+
|
50
|
+
def json_schema_validator
|
51
|
+
@json_schema_validator ||= Journaled::JsonSchemaModel::Validator.new(journaled_schema_name)
|
52
|
+
end
|
53
|
+
|
54
|
+
def base_event_json_schema_validator
|
55
|
+
@base_event_json_schema_validator ||= Journaled::JsonSchemaModel::Validator.new('base_event')
|
56
|
+
end
|
57
|
+
|
58
|
+
def respond_to_all?(object, method_names)
|
59
|
+
method_names.all? do |method_name|
|
60
|
+
object.respond_to?(method_name)
|
61
|
+
end
|
62
|
+
end
|
63
|
+
end
|
data/config/routes.rb
ADDED
data/lib/journaled.rb
ADDED
@@ -0,0 +1,24 @@
|
|
1
|
+
require "aws-sdk-resources"
|
2
|
+
require "delayed_job"
|
3
|
+
require "json-schema"
|
4
|
+
require "request_store"
|
5
|
+
|
6
|
+
require "journaled/engine"
|
7
|
+
|
8
|
+
module Journaled
|
9
|
+
mattr_accessor :default_app_name
|
10
|
+
|
11
|
+
def development_or_test?
|
12
|
+
%w(development test).include?(Rails.env)
|
13
|
+
end
|
14
|
+
|
15
|
+
def enabled?
|
16
|
+
!['0', 'false', false, 'f', ''].include?(ENV.fetch('JOURNALED_ENABLED', !development_or_test?))
|
17
|
+
end
|
18
|
+
|
19
|
+
def schema_providers
|
20
|
+
@schema_providers ||= [Journaled::Engine, Rails]
|
21
|
+
end
|
22
|
+
|
23
|
+
module_function :development_or_test?, :enabled?, :schema_providers
|
24
|
+
end
|
metadata
ADDED
@@ -0,0 +1,238 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: journaled
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 1.0.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Jake Lipson
|
8
|
+
- Corey Alexander
|
9
|
+
- Cyrus Eslami
|
10
|
+
- John Mileham
|
11
|
+
autorequire:
|
12
|
+
bindir: bin
|
13
|
+
cert_chain: []
|
14
|
+
date: 2019-01-31 00:00:00.000000000 Z
|
15
|
+
dependencies:
|
16
|
+
- !ruby/object:Gem::Dependency
|
17
|
+
name: aws-sdk-resources
|
18
|
+
requirement: !ruby/object:Gem::Requirement
|
19
|
+
requirements:
|
20
|
+
- - "<"
|
21
|
+
- !ruby/object:Gem::Version
|
22
|
+
version: '4'
|
23
|
+
type: :runtime
|
24
|
+
prerelease: false
|
25
|
+
version_requirements: !ruby/object:Gem::Requirement
|
26
|
+
requirements:
|
27
|
+
- - "<"
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
version: '4'
|
30
|
+
- !ruby/object:Gem::Dependency
|
31
|
+
name: delayed_job
|
32
|
+
requirement: !ruby/object:Gem::Requirement
|
33
|
+
requirements:
|
34
|
+
- - ">="
|
35
|
+
- !ruby/object:Gem::Version
|
36
|
+
version: '0'
|
37
|
+
type: :runtime
|
38
|
+
prerelease: false
|
39
|
+
version_requirements: !ruby/object:Gem::Requirement
|
40
|
+
requirements:
|
41
|
+
- - ">="
|
42
|
+
- !ruby/object:Gem::Version
|
43
|
+
version: '0'
|
44
|
+
- !ruby/object:Gem::Dependency
|
45
|
+
name: json-schema
|
46
|
+
requirement: !ruby/object:Gem::Requirement
|
47
|
+
requirements:
|
48
|
+
- - ">="
|
49
|
+
- !ruby/object:Gem::Version
|
50
|
+
version: '0'
|
51
|
+
type: :runtime
|
52
|
+
prerelease: false
|
53
|
+
version_requirements: !ruby/object:Gem::Requirement
|
54
|
+
requirements:
|
55
|
+
- - ">="
|
56
|
+
- !ruby/object:Gem::Version
|
57
|
+
version: '0'
|
58
|
+
- !ruby/object:Gem::Dependency
|
59
|
+
name: rails
|
60
|
+
requirement: !ruby/object:Gem::Requirement
|
61
|
+
requirements:
|
62
|
+
- - "~>"
|
63
|
+
- !ruby/object:Gem::Version
|
64
|
+
version: '5.1'
|
65
|
+
type: :runtime
|
66
|
+
prerelease: false
|
67
|
+
version_requirements: !ruby/object:Gem::Requirement
|
68
|
+
requirements:
|
69
|
+
- - "~>"
|
70
|
+
- !ruby/object:Gem::Version
|
71
|
+
version: '5.1'
|
72
|
+
- !ruby/object:Gem::Dependency
|
73
|
+
name: request_store
|
74
|
+
requirement: !ruby/object:Gem::Requirement
|
75
|
+
requirements:
|
76
|
+
- - ">="
|
77
|
+
- !ruby/object:Gem::Version
|
78
|
+
version: '0'
|
79
|
+
type: :runtime
|
80
|
+
prerelease: false
|
81
|
+
version_requirements: !ruby/object:Gem::Requirement
|
82
|
+
requirements:
|
83
|
+
- - ">="
|
84
|
+
- !ruby/object:Gem::Version
|
85
|
+
version: '0'
|
86
|
+
- !ruby/object:Gem::Dependency
|
87
|
+
name: delayed_job_active_record
|
88
|
+
requirement: !ruby/object:Gem::Requirement
|
89
|
+
requirements:
|
90
|
+
- - ">="
|
91
|
+
- !ruby/object:Gem::Version
|
92
|
+
version: '0'
|
93
|
+
type: :development
|
94
|
+
prerelease: false
|
95
|
+
version_requirements: !ruby/object:Gem::Requirement
|
96
|
+
requirements:
|
97
|
+
- - ">="
|
98
|
+
- !ruby/object:Gem::Version
|
99
|
+
version: '0'
|
100
|
+
- !ruby/object:Gem::Dependency
|
101
|
+
name: pg
|
102
|
+
requirement: !ruby/object:Gem::Requirement
|
103
|
+
requirements:
|
104
|
+
- - ">="
|
105
|
+
- !ruby/object:Gem::Version
|
106
|
+
version: '0'
|
107
|
+
type: :development
|
108
|
+
prerelease: false
|
109
|
+
version_requirements: !ruby/object:Gem::Requirement
|
110
|
+
requirements:
|
111
|
+
- - ">="
|
112
|
+
- !ruby/object:Gem::Version
|
113
|
+
version: '0'
|
114
|
+
- !ruby/object:Gem::Dependency
|
115
|
+
name: rspec-rails
|
116
|
+
requirement: !ruby/object:Gem::Requirement
|
117
|
+
requirements:
|
118
|
+
- - ">="
|
119
|
+
- !ruby/object:Gem::Version
|
120
|
+
version: '0'
|
121
|
+
type: :development
|
122
|
+
prerelease: false
|
123
|
+
version_requirements: !ruby/object:Gem::Requirement
|
124
|
+
requirements:
|
125
|
+
- - ">="
|
126
|
+
- !ruby/object:Gem::Version
|
127
|
+
version: '0'
|
128
|
+
- !ruby/object:Gem::Dependency
|
129
|
+
name: rspec_junit_formatter
|
130
|
+
requirement: !ruby/object:Gem::Requirement
|
131
|
+
requirements:
|
132
|
+
- - ">="
|
133
|
+
- !ruby/object:Gem::Version
|
134
|
+
version: '0'
|
135
|
+
type: :development
|
136
|
+
prerelease: false
|
137
|
+
version_requirements: !ruby/object:Gem::Requirement
|
138
|
+
requirements:
|
139
|
+
- - ">="
|
140
|
+
- !ruby/object:Gem::Version
|
141
|
+
version: '0'
|
142
|
+
- !ruby/object:Gem::Dependency
|
143
|
+
name: rubocop-betterment
|
144
|
+
requirement: !ruby/object:Gem::Requirement
|
145
|
+
requirements:
|
146
|
+
- - '='
|
147
|
+
- !ruby/object:Gem::Version
|
148
|
+
version: 1.3.0
|
149
|
+
type: :development
|
150
|
+
prerelease: false
|
151
|
+
version_requirements: !ruby/object:Gem::Requirement
|
152
|
+
requirements:
|
153
|
+
- - '='
|
154
|
+
- !ruby/object:Gem::Version
|
155
|
+
version: 1.3.0
|
156
|
+
- !ruby/object:Gem::Dependency
|
157
|
+
name: timecop
|
158
|
+
requirement: !ruby/object:Gem::Requirement
|
159
|
+
requirements:
|
160
|
+
- - ">="
|
161
|
+
- !ruby/object:Gem::Version
|
162
|
+
version: '0'
|
163
|
+
type: :development
|
164
|
+
prerelease: false
|
165
|
+
version_requirements: !ruby/object:Gem::Requirement
|
166
|
+
requirements:
|
167
|
+
- - ">="
|
168
|
+
- !ruby/object:Gem::Version
|
169
|
+
version: '0'
|
170
|
+
- !ruby/object:Gem::Dependency
|
171
|
+
name: webmock
|
172
|
+
requirement: !ruby/object:Gem::Requirement
|
173
|
+
requirements:
|
174
|
+
- - ">="
|
175
|
+
- !ruby/object:Gem::Version
|
176
|
+
version: '0'
|
177
|
+
type: :development
|
178
|
+
prerelease: false
|
179
|
+
version_requirements: !ruby/object:Gem::Requirement
|
180
|
+
requirements:
|
181
|
+
- - ">="
|
182
|
+
- !ruby/object:Gem::Version
|
183
|
+
version: '0'
|
184
|
+
description: A Rails engine to durably deliver schematized events to Amazon Kinesis
|
185
|
+
via DelayedJob.
|
186
|
+
email:
|
187
|
+
- jacob.lipson@betterment.com
|
188
|
+
- corey@betterment.com
|
189
|
+
- cyrus@betterment.com
|
190
|
+
- john@betterment.com
|
191
|
+
executables: []
|
192
|
+
extensions: []
|
193
|
+
extra_rdoc_files: []
|
194
|
+
files:
|
195
|
+
- LICENSE
|
196
|
+
- README.md
|
197
|
+
- Rakefile
|
198
|
+
- app/controllers/concerns/journaled/actor.rb
|
199
|
+
- app/models/concerns/journaled/changes.rb
|
200
|
+
- app/models/journaled/change.rb
|
201
|
+
- app/models/journaled/change_definition.rb
|
202
|
+
- app/models/journaled/change_writer.rb
|
203
|
+
- app/models/journaled/delivery.rb
|
204
|
+
- app/models/journaled/event.rb
|
205
|
+
- app/models/journaled/job_priority.rb
|
206
|
+
- app/models/journaled/json_schema_model/validator.rb
|
207
|
+
- app/models/journaled/not_truly_exceptional_error.rb
|
208
|
+
- app/models/journaled/writer.rb
|
209
|
+
- config/routes.rb
|
210
|
+
- lib/journaled.rb
|
211
|
+
- lib/journaled/engine.rb
|
212
|
+
- lib/journaled/version.rb
|
213
|
+
homepage: http://github.com/Betterment/journaled
|
214
|
+
licenses:
|
215
|
+
- MIT
|
216
|
+
metadata: {}
|
217
|
+
post_install_message:
|
218
|
+
rdoc_options: []
|
219
|
+
require_paths:
|
220
|
+
- lib
|
221
|
+
- spec/support
|
222
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
223
|
+
requirements:
|
224
|
+
- - ">="
|
225
|
+
- !ruby/object:Gem::Version
|
226
|
+
version: '0'
|
227
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
228
|
+
requirements:
|
229
|
+
- - ">="
|
230
|
+
- !ruby/object:Gem::Version
|
231
|
+
version: '0'
|
232
|
+
requirements: []
|
233
|
+
rubyforge_project:
|
234
|
+
rubygems_version: 2.5.1
|
235
|
+
signing_key:
|
236
|
+
specification_version: 4
|
237
|
+
summary: Journaling for Betterment apps.
|
238
|
+
test_files: []
|