journaled 1.0.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/LICENSE +19 -0
- data/README.md +194 -0
- data/Rakefile +32 -0
- data/app/controllers/concerns/journaled/actor.rb +18 -0
- data/app/models/concerns/journaled/changes.rb +38 -0
- data/app/models/journaled/change.rb +34 -0
- data/app/models/journaled/change_definition.rb +8 -0
- data/app/models/journaled/change_writer.rb +81 -0
- data/app/models/journaled/delivery.rb +61 -0
- data/app/models/journaled/event.rb +65 -0
- data/app/models/journaled/job_priority.rb +5 -0
- data/app/models/journaled/json_schema_model/validator.rb +38 -0
- data/app/models/journaled/not_truly_exceptional_error.rb +2 -0
- data/app/models/journaled/writer.rb +63 -0
- data/config/routes.rb +2 -0
- data/lib/journaled.rb +24 -0
- data/lib/journaled/engine.rb +4 -0
- data/lib/journaled/version.rb +3 -0
- metadata +238 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 067a7a010e80a13c5a9204e8e761e949af52243e
|
4
|
+
data.tar.gz: e5e072be8c838c63dfbbc082b51d709ed11eef6a
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 93726fa7a7c41e57cb7c7a73c6d5bee57d83562e7d52e51306cb1fc0bc25760ca959c5ba256c5415be8df0c45084486eea809372d701d33b5386e3d0dbba5b2c
|
7
|
+
data.tar.gz: 9c8aea433e5c9ffbdd5039036ea860a9c8cd464ed29fa9330bbbb936ac725b97e3b1c98eb20ea4e612ea33bddf926b80bceb02c412e55f2d0dca78ac3472d666
|
data/LICENSE
ADDED
@@ -0,0 +1,19 @@
|
|
1
|
+
Copyright (c) 2017-2019 Betterment
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy of
|
4
|
+
this software and associated documentation files (the "Software"), to deal in
|
5
|
+
the Software without restriction, including without limitation the rights to
|
6
|
+
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
|
7
|
+
of the Software, and to permit persons to whom the Software is furnished to do
|
8
|
+
so, subject to the following conditions:
|
9
|
+
|
10
|
+
The above copyright notice and this permission notice shall be included in all
|
11
|
+
copies or substantial portions of the Software.
|
12
|
+
|
13
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
14
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
15
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
16
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
17
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
18
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
19
|
+
SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,194 @@
|
|
1
|
+
# Journaled
|
2
|
+
|
3
|
+
A Rails engine to durably deliver schematized events to Amazon Kinesis via DelayedJob.
|
4
|
+
|
5
|
+
More specifically, `journaled` is composed of three opinionated pieces:
|
6
|
+
schema definition/validation via JSON Schema, transactional enqueueing
|
7
|
+
via Delayed::Job (specifically `delayed_job_active_record`), and event
|
8
|
+
transmission via Amazon Kinesis. Our current use-cases include
|
9
|
+
transmitting audit events for durable storage in S3 and/or analytical
|
10
|
+
querying in Amazon Redshift.
|
11
|
+
|
12
|
+
Journaled provides an at-least-once event delivery guarantee assuming
|
13
|
+
Delayed::Job is configured not to delete jobs on failure.
|
14
|
+
|
15
|
+
Note: Do not use the journaled gem to build an event sourcing solution
|
16
|
+
as it does not guarantee total ordering of events. It's possible we'll
|
17
|
+
add scoped ordering capability at a future date (and would gladly
|
18
|
+
entertain pull requests), but it is presently only designed to provide a
|
19
|
+
durable, eventually consistent record that discrete events happened.
|
20
|
+
|
21
|
+
## Installation
|
22
|
+
|
23
|
+
1. [Install `delayed_job_active_record`](https://github.com/collectiveidea/delayed_job_active_record#installation)
|
24
|
+
if you haven't already.
|
25
|
+
|
26
|
+
|
27
|
+
2. To integrate Journaled into your application, simply include the gem in your
|
28
|
+
app's Gemfile.
|
29
|
+
```
|
30
|
+
gem 'journaled', https_github: 'Betterment/journaled'
|
31
|
+
```
|
32
|
+
3. You will also need to define the following environment variables to allow Journaled to publish events to your AWS Kinesis event stream:
|
33
|
+
|
34
|
+
* `JOURNALED_STREAM_NAME`
|
35
|
+
|
36
|
+
Special case: if your `Journaled::Event` objects override the
|
37
|
+
`#journaled_app_name` method to a non-nil value e.g. `my_app`, you will
|
38
|
+
instead need to provide a corresponding
|
39
|
+
`[upcased_app_name]_JOURNALED_STREAM_NAME` variable for each distinct
|
40
|
+
value, e.g. `MY_APP_JOURNALED_STREAM_NAME`. You can provide a default value
|
41
|
+
for all `Journaled::Event`s in an initializer like this:
|
42
|
+
|
43
|
+
```ruby
|
44
|
+
Journaled.default_app_name = 'my_app'
|
45
|
+
```
|
46
|
+
|
47
|
+
You may optionally define the following ENV vars to specify AWS
|
48
|
+
credentials outside of the locations that the AWS SDK normally looks:
|
49
|
+
|
50
|
+
* `RUBY_AWS_ACCESS_KEY_ID`
|
51
|
+
* `RUBY_AWS_SECRET_ACCESS_KEY`
|
52
|
+
|
53
|
+
You may also specify the region to target your AWS stream by setting
|
54
|
+
`AWS_DEFAULT_REGION`. If you don't specify, Journaled will default to
|
55
|
+
`us-east-1`.
|
56
|
+
|
57
|
+
Journaled::Event provides a `commit_hash` method which you may journal
|
58
|
+
if you like. If you choose to use it, you must provide a `GIT_COMMIT`
|
59
|
+
environment variable.
|
60
|
+
|
61
|
+
## Usage
|
62
|
+
|
63
|
+
### Change Journaling
|
64
|
+
|
65
|
+
Out of the box, `Journaled` provides an event type and ActiveRecord
|
66
|
+
mix-in for durably journaling changes to your model, implemented via
|
67
|
+
ActiveRecord hooks. Use it like so:
|
68
|
+
|
69
|
+
```
|
70
|
+
class User < ApplicationRecord
|
71
|
+
include Journaled::Changes
|
72
|
+
|
73
|
+
journal_changes_to :email, :first_name, :last_name, as: :identity_change
|
74
|
+
end
|
75
|
+
```
|
76
|
+
|
77
|
+
Add the following to your controller base class for attribution:
|
78
|
+
|
79
|
+
```
|
80
|
+
class ApplicationController < ActionController::Base
|
81
|
+
include Journaled::Actor
|
82
|
+
|
83
|
+
self.journaled_actor = :current_user # Or your authenticated entity
|
84
|
+
end
|
85
|
+
```
|
86
|
+
|
87
|
+
Your authenticated entity must respond to `#to_global_id`, which
|
88
|
+
ActiveRecords do by default.
|
89
|
+
|
90
|
+
Every time any of the specified attributes is modified, or a `User`
|
91
|
+
record is created or destroyed, an event will be sent to Kinesis with the following attributes:
|
92
|
+
|
93
|
+
* `id` - a random event-specific UUID
|
94
|
+
* `event_type` - the constant value `journaled_change`
|
95
|
+
* `created_at`- when the event was created
|
96
|
+
* `table_name` - the table name backing the ActiveRecord (e.g. `users`)
|
97
|
+
* `record_id` - the primary key of the record, as a string (e.g.
|
98
|
+
`"300"`)
|
99
|
+
* `database_operation` - one of `create`, `update`, `delete`
|
100
|
+
* `logical_operation` - whatever logical operation you specified in
|
101
|
+
your `journal_changes_to` declaration (e.g. `identity_change`)
|
102
|
+
* `changes` - a serialized JSON object representing the latest values
|
103
|
+
of any new or changed attributes from the specified set (e.g.
|
104
|
+
`{"email":"mynewemail@example.com"}`)
|
105
|
+
* `actor` - a string (usually a rails global_id) representing who
|
106
|
+
performed the action.
|
107
|
+
|
108
|
+
### Custom Journaling
|
109
|
+
|
110
|
+
For every custom implementation of journaling in your application, define the JSON schema for the attributes in your event.
|
111
|
+
This schema file should live in your Rails application at the top level and should be named in snake case to match the
|
112
|
+
class being journaled.
|
113
|
+
E.g.: `your_app/journaled_schemas/my_class.json)`
|
114
|
+
|
115
|
+
In each class you intend to use Journaled, include the `Journaled::Event` module and define the attributes you want
|
116
|
+
captured. After completing the above steps, you can call the `journal!` method in the model code and the declared
|
117
|
+
attributes will be published to the Kinesis stream. Be sure to call
|
118
|
+
`journal!` within the same transaction as any database side effects of
|
119
|
+
your business logic operation to ensure that the event will eventually
|
120
|
+
be delivered if-and-only-if your transaction commits.
|
121
|
+
|
122
|
+
Example:
|
123
|
+
|
124
|
+
```js
|
125
|
+
// journaled_schemas/contract_acceptance_event.json
|
126
|
+
|
127
|
+
{
|
128
|
+
"type": "object",
|
129
|
+
"title": "contract_acceptance_event",
|
130
|
+
"required": [
|
131
|
+
"user_id",
|
132
|
+
"signature"
|
133
|
+
],
|
134
|
+
"properties": {
|
135
|
+
"user_id": {
|
136
|
+
"type": "integer"
|
137
|
+
},
|
138
|
+
"signature": {
|
139
|
+
"type": "string"
|
140
|
+
}
|
141
|
+
}
|
142
|
+
}
|
143
|
+
```
|
144
|
+
|
145
|
+
```ruby
|
146
|
+
# app/models/contract_acceptance_event.rb
|
147
|
+
|
148
|
+
ContractAcceptanceEvent = Struct.new(:user_id, :signature) do
|
149
|
+
include Journaled::Event
|
150
|
+
|
151
|
+
journal_attributes :user_id, :signature
|
152
|
+
end
|
153
|
+
```
|
154
|
+
|
155
|
+
```ruby
|
156
|
+
# app/models/contract_acceptance.rb
|
157
|
+
|
158
|
+
class ContractAcceptance
|
159
|
+
include ActiveModel::Model
|
160
|
+
|
161
|
+
attr_accessor :user_id, :signature
|
162
|
+
|
163
|
+
def user
|
164
|
+
@user ||= User.find(user_id)
|
165
|
+
end
|
166
|
+
|
167
|
+
def contract_acceptance_event
|
168
|
+
@contract_acceptance_event ||= ContractAcceptanceEvent.new(user_id, signature)
|
169
|
+
end
|
170
|
+
|
171
|
+
def save!
|
172
|
+
User.transaction do
|
173
|
+
user.update!(contract_accepted: true)
|
174
|
+
contract_acceptance_event.journal!
|
175
|
+
end
|
176
|
+
end
|
177
|
+
end
|
178
|
+
```
|
179
|
+
|
180
|
+
An event like the following will be journaled to kinesis:
|
181
|
+
|
182
|
+
```js
|
183
|
+
{
|
184
|
+
"id": "bc7cb6a6-88cf-4849-a4f0-a31b0b199c47", // A random event ID for idempotency filtering
|
185
|
+
"event_type": "contract_acceptance_event",
|
186
|
+
"created_at": "2019-01-28T11:06:54.928-05:00",
|
187
|
+
"user_id": 123,
|
188
|
+
"signature": "Sarah T. User"
|
189
|
+
}
|
190
|
+
```
|
191
|
+
|
192
|
+
## Future improvements & issue tracking
|
193
|
+
Suggestions for enhancements to this engine are currently being tracked via Github Issues. Please feel free to open an
|
194
|
+
issue for a desired feature, as well as for any observed bugs.
|
data/Rakefile
ADDED
@@ -0,0 +1,32 @@
|
|
1
|
+
begin
|
2
|
+
require 'bundler/setup'
|
3
|
+
rescue LoadError
|
4
|
+
puts 'You must `gem install bundler` and `bundle install` to run rake tasks'
|
5
|
+
end
|
6
|
+
|
7
|
+
require 'rdoc/task'
|
8
|
+
|
9
|
+
RDoc::Task.new(:rdoc) do |rdoc|
|
10
|
+
rdoc.rdoc_dir = 'rdoc'
|
11
|
+
rdoc.title = 'Journaled'
|
12
|
+
rdoc.options << '--line-numbers'
|
13
|
+
rdoc.rdoc_files.include('README.rdoc')
|
14
|
+
rdoc.rdoc_files.include('lib/**/*.rb')
|
15
|
+
end
|
16
|
+
|
17
|
+
APP_RAKEFILE = File.expand_path('spec/dummy/Rakefile', __dir__)
|
18
|
+
load 'rails/tasks/engine.rake'
|
19
|
+
|
20
|
+
Bundler::GemHelper.install_tasks
|
21
|
+
|
22
|
+
if %w(development test).include? Rails.env
|
23
|
+
require 'rspec/core'
|
24
|
+
require 'rspec/core/rake_task'
|
25
|
+
RSpec::Core::RakeTask.new
|
26
|
+
|
27
|
+
require 'rubocop/rake_task'
|
28
|
+
RuboCop::RakeTask.new
|
29
|
+
|
30
|
+
task(:default).clear
|
31
|
+
task default: %i(rubocop spec)
|
32
|
+
end
|
@@ -0,0 +1,18 @@
|
|
1
|
+
module Journaled::Actor
|
2
|
+
extend ActiveSupport::Concern
|
3
|
+
|
4
|
+
included do
|
5
|
+
class_attribute :_journaled_actor_method_name, instance_accessor: false, instance_predicate: false
|
6
|
+
before_action do
|
7
|
+
RequestStore.store[:journaled_actor_proc] = self.class._journaled_actor_method_name &&
|
8
|
+
-> { send(self.class._journaled_actor_method_name) }
|
9
|
+
end
|
10
|
+
end
|
11
|
+
|
12
|
+
class_methods do
|
13
|
+
def journaled_actor=(method_name)
|
14
|
+
raise "Must provide a symbol method name" unless method_name.is_a?(Symbol)
|
15
|
+
self._journaled_actor_method_name = method_name
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
@@ -0,0 +1,38 @@
|
|
1
|
+
module Journaled::Changes
|
2
|
+
extend ActiveSupport::Concern
|
3
|
+
|
4
|
+
included do
|
5
|
+
cattr_accessor :_journaled_change_definitions
|
6
|
+
self._journaled_change_definitions = []
|
7
|
+
|
8
|
+
after_create do
|
9
|
+
self.class._journaled_change_definitions.each do |definition|
|
10
|
+
Journaled::ChangeWriter.new(model: self, change_definition: definition).create
|
11
|
+
end
|
12
|
+
end
|
13
|
+
|
14
|
+
after_save unless: :saved_change_to_id? do
|
15
|
+
self.class._journaled_change_definitions.each do |definition|
|
16
|
+
Journaled::ChangeWriter.new(model: self, change_definition: definition).update
|
17
|
+
end
|
18
|
+
end
|
19
|
+
|
20
|
+
after_destroy do
|
21
|
+
self.class._journaled_change_definitions.each do |definition|
|
22
|
+
Journaled::ChangeWriter.new(model: self, change_definition: definition).delete
|
23
|
+
end
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
class_methods do
|
28
|
+
def journal_changes_to(*attribute_names, as:) # rubocop:disable Naming/UncommunicativeMethodParamName
|
29
|
+
if attribute_names.empty? || attribute_names.any? { |n| !n.is_a?(Symbol) }
|
30
|
+
raise "one or more symbol attribute_name arguments is required"
|
31
|
+
end
|
32
|
+
|
33
|
+
raise "as: must be a symbol" unless as.is_a?(Symbol)
|
34
|
+
|
35
|
+
_journaled_change_definitions << Journaled::ChangeDefinition.new(attribute_names: attribute_names, logical_operation: as)
|
36
|
+
end
|
37
|
+
end
|
38
|
+
end
|
@@ -0,0 +1,34 @@
|
|
1
|
+
class Journaled::Change
|
2
|
+
include Journaled::Event
|
3
|
+
|
4
|
+
attr_reader :table_name,
|
5
|
+
:record_id,
|
6
|
+
:database_operation,
|
7
|
+
:logical_operation,
|
8
|
+
:changes,
|
9
|
+
:journaled_app_name,
|
10
|
+
:actor
|
11
|
+
|
12
|
+
journal_attributes :table_name,
|
13
|
+
:record_id,
|
14
|
+
:database_operation,
|
15
|
+
:logical_operation,
|
16
|
+
:changes,
|
17
|
+
:actor
|
18
|
+
|
19
|
+
def initialize(table_name:, # rubocop:disable Metrics/ParameterLists
|
20
|
+
record_id:,
|
21
|
+
database_operation:,
|
22
|
+
logical_operation:,
|
23
|
+
changes:,
|
24
|
+
journaled_app_name:,
|
25
|
+
actor:)
|
26
|
+
@table_name = table_name
|
27
|
+
@record_id = record_id
|
28
|
+
@database_operation = database_operation
|
29
|
+
@logical_operation = logical_operation
|
30
|
+
@changes = changes
|
31
|
+
@journaled_app_name = journaled_app_name
|
32
|
+
@actor = actor
|
33
|
+
end
|
34
|
+
end
|
@@ -0,0 +1,81 @@
|
|
1
|
+
class Journaled::ChangeWriter
|
2
|
+
attr_reader :model, :change_definition
|
3
|
+
delegate :logical_operation, to: :change_definition
|
4
|
+
|
5
|
+
def initialize(model:, change_definition:)
|
6
|
+
@model = model
|
7
|
+
@change_definition = change_definition
|
8
|
+
end
|
9
|
+
|
10
|
+
def attribute_names
|
11
|
+
@attribute_names ||= change_definition.attribute_names.map(&:to_s)
|
12
|
+
end
|
13
|
+
|
14
|
+
def create
|
15
|
+
journaled_change_for("create", relevant_attributes).journal!
|
16
|
+
end
|
17
|
+
|
18
|
+
def update
|
19
|
+
journaled_change_for("update", relevant_changes).journal! if relevant_changes.present?
|
20
|
+
end
|
21
|
+
|
22
|
+
def delete
|
23
|
+
journaled_change_for("delete", {}).journal!
|
24
|
+
end
|
25
|
+
|
26
|
+
def journaled_change_for(database_operation, changes)
|
27
|
+
Journaled::Change.new(
|
28
|
+
table_name: model.class.table_name,
|
29
|
+
record_id: model.id.to_s,
|
30
|
+
database_operation: database_operation,
|
31
|
+
logical_operation: logical_operation,
|
32
|
+
changes: JSON.dump(changes),
|
33
|
+
journaled_app_name: journaled_app_name,
|
34
|
+
actor: actor_uri
|
35
|
+
)
|
36
|
+
end
|
37
|
+
|
38
|
+
def relevant_attributes
|
39
|
+
model.attributes.slice(*attribute_names)
|
40
|
+
end
|
41
|
+
|
42
|
+
def relevant_changes
|
43
|
+
relevant_changes_with_previous_values.each_with_object({}) do |(k, v), result|
|
44
|
+
result[k] = v[1]
|
45
|
+
end
|
46
|
+
end
|
47
|
+
|
48
|
+
def actor_uri
|
49
|
+
actor_global_id_uri || fallback_global_id_uri
|
50
|
+
end
|
51
|
+
|
52
|
+
private
|
53
|
+
|
54
|
+
def actor_global_id_uri
|
55
|
+
actor.to_global_id.to_s if actor
|
56
|
+
end
|
57
|
+
|
58
|
+
def actor
|
59
|
+
@actor ||= RequestStore.store[:journaled_actor_proc]&.call
|
60
|
+
end
|
61
|
+
|
62
|
+
def fallback_global_id_uri
|
63
|
+
if defined?(::Rails::Console) || File.basename($PROGRAM_NAME) == "rake"
|
64
|
+
"gid://local/#{Etc.getlogin}"
|
65
|
+
else
|
66
|
+
"gid://#{Rails.application.config.global_id.app}"
|
67
|
+
end
|
68
|
+
end
|
69
|
+
|
70
|
+
def relevant_changes_with_previous_values
|
71
|
+
model.saved_changes.slice(*attribute_names)
|
72
|
+
end
|
73
|
+
|
74
|
+
def journaled_app_name
|
75
|
+
if model.class.respond_to?(:journaled_app_name)
|
76
|
+
model.class.journaled_app_name
|
77
|
+
else
|
78
|
+
Journaled.default_app_name
|
79
|
+
end
|
80
|
+
end
|
81
|
+
end
|
@@ -0,0 +1,61 @@
|
|
1
|
+
class Journaled::Delivery
|
2
|
+
DEFAULT_REGION = 'us-east-1'.freeze
|
3
|
+
|
4
|
+
def initialize(serialized_event:, partition_key:, app_name:)
|
5
|
+
@serialized_event = serialized_event
|
6
|
+
@partition_key = partition_key
|
7
|
+
@app_name = app_name
|
8
|
+
end
|
9
|
+
|
10
|
+
def perform
|
11
|
+
kinesis_client.put_record record if Journaled.enabled?
|
12
|
+
rescue Aws::Kinesis::Errors::InternalFailure, Aws::Kinesis::Errors::ServiceUnavailable, Aws::Kinesis::Errors::Http503Error => e
|
13
|
+
Rails.logger.error "Kinesis Error - Server Error occurred - #{e.class}"
|
14
|
+
raise KinesisTemporaryFailure
|
15
|
+
rescue Seahorse::Client::NetworkingError => e
|
16
|
+
Rails.logger.error "Kinesis Error - Networking Error occurred - #{e.class}"
|
17
|
+
raise KinesisTemporaryFailure
|
18
|
+
end
|
19
|
+
|
20
|
+
def stream_name
|
21
|
+
env_var_name = [app_name&.upcase, 'JOURNALED_STREAM_NAME'].compact.join('_')
|
22
|
+
ENV.fetch(env_var_name)
|
23
|
+
end
|
24
|
+
|
25
|
+
def kinesis_client_config
|
26
|
+
{
|
27
|
+
region: ENV.fetch('AWS_DEFAULT_REGION', DEFAULT_REGION),
|
28
|
+
retry_limit: 0
|
29
|
+
}.merge(legacy_credentials_hash)
|
30
|
+
end
|
31
|
+
|
32
|
+
private
|
33
|
+
|
34
|
+
attr_reader :serialized_event, :partition_key, :app_name
|
35
|
+
|
36
|
+
def record
|
37
|
+
{
|
38
|
+
stream_name: stream_name,
|
39
|
+
data: serialized_event,
|
40
|
+
partition_key: partition_key
|
41
|
+
}
|
42
|
+
end
|
43
|
+
|
44
|
+
def kinesis_client
|
45
|
+
@kinesis_client ||= Aws::Kinesis::Client.new(kinesis_client_config)
|
46
|
+
end
|
47
|
+
|
48
|
+
def legacy_credentials_hash
|
49
|
+
if ENV.key?('RUBY_AWS_ACCESS_KEY_ID')
|
50
|
+
{
|
51
|
+
access_key_id: ENV.fetch('RUBY_AWS_ACCESS_KEY_ID'),
|
52
|
+
secret_access_key: ENV.fetch('RUBY_AWS_SECRET_ACCESS_KEY')
|
53
|
+
}
|
54
|
+
else
|
55
|
+
{}
|
56
|
+
end
|
57
|
+
end
|
58
|
+
|
59
|
+
class KinesisTemporaryFailure < NotTrulyExceptionalError
|
60
|
+
end
|
61
|
+
end
|
@@ -0,0 +1,65 @@
|
|
1
|
+
module Journaled::Event
|
2
|
+
extend ActiveSupport::Concern
|
3
|
+
|
4
|
+
def journal!
|
5
|
+
Journaled::Writer.new(journaled_event: self).journal!
|
6
|
+
end
|
7
|
+
|
8
|
+
# Base attributes
|
9
|
+
|
10
|
+
def id
|
11
|
+
@id ||= SecureRandom.uuid
|
12
|
+
end
|
13
|
+
|
14
|
+
def event_type
|
15
|
+
@event_type ||= self.class.event_type
|
16
|
+
end
|
17
|
+
|
18
|
+
def created_at
|
19
|
+
@created_at ||= Time.zone.now
|
20
|
+
end
|
21
|
+
|
22
|
+
def commit_hash
|
23
|
+
@commit_hash ||= ENV.fetch('GIT_COMMIT')
|
24
|
+
end
|
25
|
+
|
26
|
+
# Event metadata and configuration (not serialized)
|
27
|
+
|
28
|
+
def journaled_schema_name
|
29
|
+
self.class.to_s.underscore
|
30
|
+
end
|
31
|
+
|
32
|
+
def journaled_attributes
|
33
|
+
self.class.public_send(:journaled_attributes).each_with_object({}) do |attribute, memo|
|
34
|
+
memo[attribute] = public_send(attribute)
|
35
|
+
end
|
36
|
+
end
|
37
|
+
|
38
|
+
def journaled_partition_key
|
39
|
+
event_type
|
40
|
+
end
|
41
|
+
|
42
|
+
def journaled_app_name
|
43
|
+
Journaled.default_app_name
|
44
|
+
end
|
45
|
+
|
46
|
+
private
|
47
|
+
|
48
|
+
class_methods do
|
49
|
+
def journal_attributes(*args)
|
50
|
+
journaled_attributes.concat(args)
|
51
|
+
end
|
52
|
+
|
53
|
+
def journaled_attributes
|
54
|
+
@journaled_attributes ||= []
|
55
|
+
end
|
56
|
+
|
57
|
+
def event_type
|
58
|
+
name.underscore.parameterize(separator: '_')
|
59
|
+
end
|
60
|
+
end
|
61
|
+
|
62
|
+
included do
|
63
|
+
journal_attributes :id, :event_type, :created_at
|
64
|
+
end
|
65
|
+
end
|
@@ -0,0 +1,5 @@
|
|
1
|
+
module Journaled::JobPriority
|
2
|
+
INTERACTIVE = 0 # These jobs will actively hinder end-user interactions until complete, e.g. assembling a report a user is polling for.
|
3
|
+
USER_VISIBLE = 10 # These jobs have end-user-visible side effects that will not obviously impact customers, e.g. welcome emails
|
4
|
+
EVENTUAL = 20 # These jobs affect business process that are tolerant to some degree of queue backlog, e.g. desk record synchronization
|
5
|
+
end
|
@@ -0,0 +1,38 @@
|
|
1
|
+
class Journaled::JsonSchemaModel::Validator
|
2
|
+
def initialize(schema_name)
|
3
|
+
@schema_name = schema_name
|
4
|
+
end
|
5
|
+
|
6
|
+
def validate!(json_to_validate)
|
7
|
+
JSON::Validator.validate!(json_schema, json_to_validate)
|
8
|
+
end
|
9
|
+
|
10
|
+
private
|
11
|
+
|
12
|
+
attr_reader :schema_name
|
13
|
+
|
14
|
+
def json_schema
|
15
|
+
@json_schema ||= JSON.parse(json_schema_file)
|
16
|
+
end
|
17
|
+
|
18
|
+
def json_schema_file
|
19
|
+
@json_schema_file ||= File.read(json_schema_path)
|
20
|
+
end
|
21
|
+
|
22
|
+
def json_schema_path
|
23
|
+
@json_schema_path ||= gem_paths.detect { |path| File.exist?(path) } || raise(<<~ERROR)
|
24
|
+
journaled_schemas/#{schema_name}.json not found in any of #{Journaled.schema_providers.map { |sp| "#{sp}.root" }.join(', ')}
|
25
|
+
|
26
|
+
You can add schema providers as follows:
|
27
|
+
|
28
|
+
# config/initializers/journaled.rb
|
29
|
+
Journaled.schema_providers << MyGem::Engine
|
30
|
+
ERROR
|
31
|
+
end
|
32
|
+
|
33
|
+
def gem_paths
|
34
|
+
Journaled.schema_providers.map do |engine|
|
35
|
+
engine.root.join "journaled_schemas/#{schema_name}.json"
|
36
|
+
end
|
37
|
+
end
|
38
|
+
end
|
@@ -0,0 +1,63 @@
|
|
1
|
+
class Journaled::Writer
|
2
|
+
EVENT_METHOD_NAMES = %i(
|
3
|
+
journaled_schema_name
|
4
|
+
journaled_partition_key
|
5
|
+
journaled_attributes
|
6
|
+
journaled_app_name
|
7
|
+
).freeze
|
8
|
+
|
9
|
+
def initialize(journaled_event:, priority: Journaled::JobPriority::EVENTUAL)
|
10
|
+
raise "An enqueued event must respond to: #{EVENT_METHOD_NAMES.to_sentence}" unless respond_to_all?(journaled_event, EVENT_METHOD_NAMES)
|
11
|
+
|
12
|
+
unless journaled_event.journaled_schema_name.present? &&
|
13
|
+
journaled_event.journaled_partition_key.present? &&
|
14
|
+
journaled_event.journaled_attributes.present?
|
15
|
+
raise <<~ERROR
|
16
|
+
An enqueued event must have a non-nil response to:
|
17
|
+
#json_schema_name,
|
18
|
+
#partition_key, and
|
19
|
+
#journaled_attributes
|
20
|
+
ERROR
|
21
|
+
end
|
22
|
+
|
23
|
+
@journaled_event = journaled_event
|
24
|
+
@priority = priority
|
25
|
+
end
|
26
|
+
|
27
|
+
def journal!
|
28
|
+
base_event_json_schema_validator.validate! serialized_event
|
29
|
+
json_schema_validator.validate! serialized_event
|
30
|
+
Delayed::Job.enqueue journaled_delivery, priority: priority
|
31
|
+
end
|
32
|
+
|
33
|
+
private
|
34
|
+
|
35
|
+
attr_reader :journaled_event, :priority
|
36
|
+
delegate :journaled_schema_name, :journaled_attributes, :journaled_partition_key, :journaled_app_name, to: :journaled_event
|
37
|
+
|
38
|
+
def journaled_delivery
|
39
|
+
@journaled_delivery ||= Journaled::Delivery.new(
|
40
|
+
serialized_event: serialized_event,
|
41
|
+
partition_key: journaled_partition_key,
|
42
|
+
app_name: journaled_app_name
|
43
|
+
)
|
44
|
+
end
|
45
|
+
|
46
|
+
def serialized_event
|
47
|
+
@serialized_event ||= journaled_attributes.to_json
|
48
|
+
end
|
49
|
+
|
50
|
+
def json_schema_validator
|
51
|
+
@json_schema_validator ||= Journaled::JsonSchemaModel::Validator.new(journaled_schema_name)
|
52
|
+
end
|
53
|
+
|
54
|
+
def base_event_json_schema_validator
|
55
|
+
@base_event_json_schema_validator ||= Journaled::JsonSchemaModel::Validator.new('base_event')
|
56
|
+
end
|
57
|
+
|
58
|
+
def respond_to_all?(object, method_names)
|
59
|
+
method_names.all? do |method_name|
|
60
|
+
object.respond_to?(method_name)
|
61
|
+
end
|
62
|
+
end
|
63
|
+
end
|
data/config/routes.rb
ADDED
data/lib/journaled.rb
ADDED
@@ -0,0 +1,24 @@
|
|
1
|
+
require "aws-sdk-resources"
|
2
|
+
require "delayed_job"
|
3
|
+
require "json-schema"
|
4
|
+
require "request_store"
|
5
|
+
|
6
|
+
require "journaled/engine"
|
7
|
+
|
8
|
+
module Journaled
|
9
|
+
mattr_accessor :default_app_name
|
10
|
+
|
11
|
+
def development_or_test?
|
12
|
+
%w(development test).include?(Rails.env)
|
13
|
+
end
|
14
|
+
|
15
|
+
def enabled?
|
16
|
+
!['0', 'false', false, 'f', ''].include?(ENV.fetch('JOURNALED_ENABLED', !development_or_test?))
|
17
|
+
end
|
18
|
+
|
19
|
+
def schema_providers
|
20
|
+
@schema_providers ||= [Journaled::Engine, Rails]
|
21
|
+
end
|
22
|
+
|
23
|
+
module_function :development_or_test?, :enabled?, :schema_providers
|
24
|
+
end
|
metadata
ADDED
@@ -0,0 +1,238 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: journaled
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 1.0.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Jake Lipson
|
8
|
+
- Corey Alexander
|
9
|
+
- Cyrus Eslami
|
10
|
+
- John Mileham
|
11
|
+
autorequire:
|
12
|
+
bindir: bin
|
13
|
+
cert_chain: []
|
14
|
+
date: 2019-01-31 00:00:00.000000000 Z
|
15
|
+
dependencies:
|
16
|
+
- !ruby/object:Gem::Dependency
|
17
|
+
name: aws-sdk-resources
|
18
|
+
requirement: !ruby/object:Gem::Requirement
|
19
|
+
requirements:
|
20
|
+
- - "<"
|
21
|
+
- !ruby/object:Gem::Version
|
22
|
+
version: '4'
|
23
|
+
type: :runtime
|
24
|
+
prerelease: false
|
25
|
+
version_requirements: !ruby/object:Gem::Requirement
|
26
|
+
requirements:
|
27
|
+
- - "<"
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
version: '4'
|
30
|
+
- !ruby/object:Gem::Dependency
|
31
|
+
name: delayed_job
|
32
|
+
requirement: !ruby/object:Gem::Requirement
|
33
|
+
requirements:
|
34
|
+
- - ">="
|
35
|
+
- !ruby/object:Gem::Version
|
36
|
+
version: '0'
|
37
|
+
type: :runtime
|
38
|
+
prerelease: false
|
39
|
+
version_requirements: !ruby/object:Gem::Requirement
|
40
|
+
requirements:
|
41
|
+
- - ">="
|
42
|
+
- !ruby/object:Gem::Version
|
43
|
+
version: '0'
|
44
|
+
- !ruby/object:Gem::Dependency
|
45
|
+
name: json-schema
|
46
|
+
requirement: !ruby/object:Gem::Requirement
|
47
|
+
requirements:
|
48
|
+
- - ">="
|
49
|
+
- !ruby/object:Gem::Version
|
50
|
+
version: '0'
|
51
|
+
type: :runtime
|
52
|
+
prerelease: false
|
53
|
+
version_requirements: !ruby/object:Gem::Requirement
|
54
|
+
requirements:
|
55
|
+
- - ">="
|
56
|
+
- !ruby/object:Gem::Version
|
57
|
+
version: '0'
|
58
|
+
- !ruby/object:Gem::Dependency
|
59
|
+
name: rails
|
60
|
+
requirement: !ruby/object:Gem::Requirement
|
61
|
+
requirements:
|
62
|
+
- - "~>"
|
63
|
+
- !ruby/object:Gem::Version
|
64
|
+
version: '5.1'
|
65
|
+
type: :runtime
|
66
|
+
prerelease: false
|
67
|
+
version_requirements: !ruby/object:Gem::Requirement
|
68
|
+
requirements:
|
69
|
+
- - "~>"
|
70
|
+
- !ruby/object:Gem::Version
|
71
|
+
version: '5.1'
|
72
|
+
- !ruby/object:Gem::Dependency
|
73
|
+
name: request_store
|
74
|
+
requirement: !ruby/object:Gem::Requirement
|
75
|
+
requirements:
|
76
|
+
- - ">="
|
77
|
+
- !ruby/object:Gem::Version
|
78
|
+
version: '0'
|
79
|
+
type: :runtime
|
80
|
+
prerelease: false
|
81
|
+
version_requirements: !ruby/object:Gem::Requirement
|
82
|
+
requirements:
|
83
|
+
- - ">="
|
84
|
+
- !ruby/object:Gem::Version
|
85
|
+
version: '0'
|
86
|
+
- !ruby/object:Gem::Dependency
|
87
|
+
name: delayed_job_active_record
|
88
|
+
requirement: !ruby/object:Gem::Requirement
|
89
|
+
requirements:
|
90
|
+
- - ">="
|
91
|
+
- !ruby/object:Gem::Version
|
92
|
+
version: '0'
|
93
|
+
type: :development
|
94
|
+
prerelease: false
|
95
|
+
version_requirements: !ruby/object:Gem::Requirement
|
96
|
+
requirements:
|
97
|
+
- - ">="
|
98
|
+
- !ruby/object:Gem::Version
|
99
|
+
version: '0'
|
100
|
+
- !ruby/object:Gem::Dependency
|
101
|
+
name: pg
|
102
|
+
requirement: !ruby/object:Gem::Requirement
|
103
|
+
requirements:
|
104
|
+
- - ">="
|
105
|
+
- !ruby/object:Gem::Version
|
106
|
+
version: '0'
|
107
|
+
type: :development
|
108
|
+
prerelease: false
|
109
|
+
version_requirements: !ruby/object:Gem::Requirement
|
110
|
+
requirements:
|
111
|
+
- - ">="
|
112
|
+
- !ruby/object:Gem::Version
|
113
|
+
version: '0'
|
114
|
+
- !ruby/object:Gem::Dependency
|
115
|
+
name: rspec-rails
|
116
|
+
requirement: !ruby/object:Gem::Requirement
|
117
|
+
requirements:
|
118
|
+
- - ">="
|
119
|
+
- !ruby/object:Gem::Version
|
120
|
+
version: '0'
|
121
|
+
type: :development
|
122
|
+
prerelease: false
|
123
|
+
version_requirements: !ruby/object:Gem::Requirement
|
124
|
+
requirements:
|
125
|
+
- - ">="
|
126
|
+
- !ruby/object:Gem::Version
|
127
|
+
version: '0'
|
128
|
+
- !ruby/object:Gem::Dependency
|
129
|
+
name: rspec_junit_formatter
|
130
|
+
requirement: !ruby/object:Gem::Requirement
|
131
|
+
requirements:
|
132
|
+
- - ">="
|
133
|
+
- !ruby/object:Gem::Version
|
134
|
+
version: '0'
|
135
|
+
type: :development
|
136
|
+
prerelease: false
|
137
|
+
version_requirements: !ruby/object:Gem::Requirement
|
138
|
+
requirements:
|
139
|
+
- - ">="
|
140
|
+
- !ruby/object:Gem::Version
|
141
|
+
version: '0'
|
142
|
+
- !ruby/object:Gem::Dependency
|
143
|
+
name: rubocop-betterment
|
144
|
+
requirement: !ruby/object:Gem::Requirement
|
145
|
+
requirements:
|
146
|
+
- - '='
|
147
|
+
- !ruby/object:Gem::Version
|
148
|
+
version: 1.3.0
|
149
|
+
type: :development
|
150
|
+
prerelease: false
|
151
|
+
version_requirements: !ruby/object:Gem::Requirement
|
152
|
+
requirements:
|
153
|
+
- - '='
|
154
|
+
- !ruby/object:Gem::Version
|
155
|
+
version: 1.3.0
|
156
|
+
- !ruby/object:Gem::Dependency
|
157
|
+
name: timecop
|
158
|
+
requirement: !ruby/object:Gem::Requirement
|
159
|
+
requirements:
|
160
|
+
- - ">="
|
161
|
+
- !ruby/object:Gem::Version
|
162
|
+
version: '0'
|
163
|
+
type: :development
|
164
|
+
prerelease: false
|
165
|
+
version_requirements: !ruby/object:Gem::Requirement
|
166
|
+
requirements:
|
167
|
+
- - ">="
|
168
|
+
- !ruby/object:Gem::Version
|
169
|
+
version: '0'
|
170
|
+
- !ruby/object:Gem::Dependency
|
171
|
+
name: webmock
|
172
|
+
requirement: !ruby/object:Gem::Requirement
|
173
|
+
requirements:
|
174
|
+
- - ">="
|
175
|
+
- !ruby/object:Gem::Version
|
176
|
+
version: '0'
|
177
|
+
type: :development
|
178
|
+
prerelease: false
|
179
|
+
version_requirements: !ruby/object:Gem::Requirement
|
180
|
+
requirements:
|
181
|
+
- - ">="
|
182
|
+
- !ruby/object:Gem::Version
|
183
|
+
version: '0'
|
184
|
+
description: A Rails engine to durably deliver schematized events to Amazon Kinesis
|
185
|
+
via DelayedJob.
|
186
|
+
email:
|
187
|
+
- jacob.lipson@betterment.com
|
188
|
+
- corey@betterment.com
|
189
|
+
- cyrus@betterment.com
|
190
|
+
- john@betterment.com
|
191
|
+
executables: []
|
192
|
+
extensions: []
|
193
|
+
extra_rdoc_files: []
|
194
|
+
files:
|
195
|
+
- LICENSE
|
196
|
+
- README.md
|
197
|
+
- Rakefile
|
198
|
+
- app/controllers/concerns/journaled/actor.rb
|
199
|
+
- app/models/concerns/journaled/changes.rb
|
200
|
+
- app/models/journaled/change.rb
|
201
|
+
- app/models/journaled/change_definition.rb
|
202
|
+
- app/models/journaled/change_writer.rb
|
203
|
+
- app/models/journaled/delivery.rb
|
204
|
+
- app/models/journaled/event.rb
|
205
|
+
- app/models/journaled/job_priority.rb
|
206
|
+
- app/models/journaled/json_schema_model/validator.rb
|
207
|
+
- app/models/journaled/not_truly_exceptional_error.rb
|
208
|
+
- app/models/journaled/writer.rb
|
209
|
+
- config/routes.rb
|
210
|
+
- lib/journaled.rb
|
211
|
+
- lib/journaled/engine.rb
|
212
|
+
- lib/journaled/version.rb
|
213
|
+
homepage: http://github.com/Betterment/journaled
|
214
|
+
licenses:
|
215
|
+
- MIT
|
216
|
+
metadata: {}
|
217
|
+
post_install_message:
|
218
|
+
rdoc_options: []
|
219
|
+
require_paths:
|
220
|
+
- lib
|
221
|
+
- spec/support
|
222
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
223
|
+
requirements:
|
224
|
+
- - ">="
|
225
|
+
- !ruby/object:Gem::Version
|
226
|
+
version: '0'
|
227
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
228
|
+
requirements:
|
229
|
+
- - ">="
|
230
|
+
- !ruby/object:Gem::Version
|
231
|
+
version: '0'
|
232
|
+
requirements: []
|
233
|
+
rubyforge_project:
|
234
|
+
rubygems_version: 2.5.1
|
235
|
+
signing_key:
|
236
|
+
specification_version: 4
|
237
|
+
summary: Journaling for Betterment apps.
|
238
|
+
test_files: []
|