journaled 2.2.0 → 3.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +42 -6
- data/Rakefile +7 -1
- data/app/jobs/journaled/application_job.rb +4 -0
- data/app/jobs/journaled/delivery_job.rb +96 -0
- data/app/models/concerns/journaled/changes.rb +5 -5
- data/app/models/journaled/change.rb +3 -0
- data/app/models/journaled/change_writer.rb +1 -0
- data/app/models/journaled/delivery.rb +5 -2
- data/app/models/journaled/event.rb +5 -2
- data/app/models/journaled/writer.rb +10 -8
- data/lib/journaled.rb +26 -5
- data/lib/journaled/engine.rb +5 -0
- data/lib/journaled/relation_change_protection.rb +1 -1
- data/lib/journaled/version.rb +1 -1
- data/spec/dummy/config/application.rb +1 -2
- data/spec/dummy/config/database.yml +4 -19
- data/spec/dummy/config/environments/development.rb +0 -13
- data/spec/dummy/config/environments/test.rb +3 -5
- data/spec/dummy/db/schema.rb +3 -16
- data/spec/jobs/journaled/delivery_job_spec.rb +221 -0
- data/spec/lib/journaled_spec.rb +39 -0
- data/spec/models/concerns/journaled/changes_spec.rb +11 -0
- data/spec/models/database_change_protection_spec.rb +19 -25
- data/spec/models/journaled/change_writer_spec.rb +5 -0
- data/spec/models/journaled/delivery_spec.rb +33 -0
- data/spec/models/journaled/event_spec.rb +23 -16
- data/spec/models/journaled/writer_spec.rb +34 -18
- data/spec/rails_helper.rb +1 -2
- data/spec/spec_helper.rb +1 -3
- metadata +38 -62
- data/config/routes.rb +0 -2
- data/lib/journaled/job_priority.rb +0 -5
- data/spec/dummy/config/environments/production.rb +0 -78
- data/spec/dummy/config/initializers/assets.rb +0 -8
- data/spec/dummy/db/migrate/20180606205114_create_delayed_jobs.rb +0 -18
- data/spec/dummy/log/development.log +0 -29
- data/spec/dummy/log/test.log +0 -2482
- data/spec/support/delayed_job_spec_helper.rb +0 -11
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 358c3ced2b976724c62e951ca88b635709d1fb7da0276ed4538bdc44bc02f468
|
4
|
+
data.tar.gz: ca65a5b0531ba4540de1b79fd36d247c7c164f3aef8b397254ed7b34503ebf0f
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 236c6169ef41b1af9043f2549248ae43fad665a91d43cdd2f579e891685788aa5a008cf4f1dc8aba13abcce5dccfa6513c82232fdaaf3ab4f42f8a7b1b222bf7
|
7
|
+
data.tar.gz: 9e325f6c1e41eb577f5e0d7320517f5435b9570d7b0274c5b6ad9c593abc2a4a66dc6b6eb10d668617e645e5731804ecd65d4b6403fda63005de129443ed9332
|
data/README.md
CHANGED
@@ -1,16 +1,16 @@
|
|
1
1
|
# Journaled
|
2
2
|
|
3
|
-
A Rails engine to durably deliver schematized events to Amazon Kinesis via
|
3
|
+
A Rails engine to durably deliver schematized events to Amazon Kinesis via ActiveJob.
|
4
4
|
|
5
5
|
More specifically, `journaled` is composed of three opinionated pieces:
|
6
6
|
schema definition/validation via JSON Schema, transactional enqueueing
|
7
|
-
via
|
7
|
+
via ActiveJob (specifically, via a DB-backed queue adapter), and event
|
8
8
|
transmission via Amazon Kinesis. Our current use-cases include
|
9
9
|
transmitting audit events for durable storage in S3 and/or analytical
|
10
10
|
querying in Amazon Redshift.
|
11
11
|
|
12
12
|
Journaled provides an at-least-once event delivery guarantee assuming
|
13
|
-
|
13
|
+
ActiveJob's queue adapter is not configured to delete jobs on failure.
|
14
14
|
|
15
15
|
Note: Do not use the journaled gem to build an event sourcing solution
|
16
16
|
as it does not guarantee total ordering of events. It's possible we'll
|
@@ -20,9 +20,20 @@ durable, eventually consistent record that discrete events happened.
|
|
20
20
|
|
21
21
|
## Installation
|
22
22
|
|
23
|
-
1.
|
24
|
-
|
23
|
+
1. If you haven't already,
|
24
|
+
[configure ActiveJob](https://guides.rubyonrails.org/active_job_basics.html)
|
25
|
+
to use one of the following queue adapters:
|
25
26
|
|
27
|
+
- `:delayed_job` (via `delayed_job_active_record`)
|
28
|
+
- `:que`
|
29
|
+
- `:good_job`
|
30
|
+
- `:delayed`
|
31
|
+
|
32
|
+
Ensure that your queue adapter is not configured to delete jobs on failure.
|
33
|
+
|
34
|
+
**If you launch your application in production mode and the gem detects that
|
35
|
+
`ActiveJob::Base.queue_adapter` is not in the above list, it will raise an exception
|
36
|
+
and prevent your application from performing unsafe journaling.**
|
26
37
|
|
27
38
|
2. To integrate Journaled into your application, simply include the gem in your
|
28
39
|
app's Gemfile.
|
@@ -85,9 +96,34 @@ Journaling provides a number of different configuation options that can be set i
|
|
85
96
|
|
86
97
|
#### `Journaled.job_priority` (default: 20)
|
87
98
|
|
88
|
-
This can be used to configure what `priority` the
|
99
|
+
This can be used to configure what `priority` the ActiveJobs are enqueued with. This will be applied to all the `Journaled::DeliveryJob`s that are created by this application.
|
89
100
|
Ex: `Journaled.job_priority = 14`
|
90
101
|
|
102
|
+
_Note that job priority is only supported on Rails 6.0+. Prior Rails versions will ignore this parameter and enqueue jobs with the underlying ActiveJob adapter's default priority._
|
103
|
+
|
104
|
+
#### `Journaled.http_idle_timeout` (default: 1 second)
|
105
|
+
|
106
|
+
The number of seconds a persistent connection is allowed to sit idle before it should no longer be used.
|
107
|
+
|
108
|
+
#### `Journaled.http_open_timeout` (default: 2 seconds)
|
109
|
+
|
110
|
+
The number of seconds before the :http_handler should timeout while trying to open a new HTTP session.
|
111
|
+
|
112
|
+
#### `Journaled.http_read_timeout` (default: 60 seconds)
|
113
|
+
|
114
|
+
The number of seconds before the :http_handler should timeout while waiting for a HTTP response.
|
115
|
+
|
116
|
+
#### ActiveJob `set` options
|
117
|
+
|
118
|
+
Both model-level directives accept additional options to be passed into ActiveJob's `set` method:
|
119
|
+
|
120
|
+
```ruby
|
121
|
+
# For change journaling:
|
122
|
+
journal_changes_to :email, as: :identity_change, enqueue_with: { priority: 10 }
|
123
|
+
|
124
|
+
# Or for custom journaling:
|
125
|
+
journal_attributes :email, enqueue_with: { priority: 20, queue: 'journaled' }
|
126
|
+
```
|
91
127
|
|
92
128
|
### Change Journaling
|
93
129
|
|
data/Rakefile
CHANGED
@@ -28,7 +28,13 @@ if %w(development test).include? Rails.env
|
|
28
28
|
RuboCop::RakeTask.new
|
29
29
|
|
30
30
|
task(:default).clear
|
31
|
-
|
31
|
+
if ENV['APPRAISAL_INITIALIZED'] || ENV['CI']
|
32
|
+
task default: %i(rubocop spec)
|
33
|
+
else
|
34
|
+
require 'appraisal'
|
35
|
+
Appraisal::Task.new
|
36
|
+
task default: :appraisal
|
37
|
+
end
|
32
38
|
|
33
39
|
task 'db:test:prepare' => 'db:setup'
|
34
40
|
end
|
@@ -0,0 +1,96 @@
|
|
1
|
+
module Journaled
|
2
|
+
class DeliveryJob < ApplicationJob
|
3
|
+
DEFAULT_REGION = 'us-east-1'.freeze
|
4
|
+
|
5
|
+
rescue_from(Aws::Kinesis::Errors::InternalFailure, Aws::Kinesis::Errors::ServiceUnavailable, Aws::Kinesis::Errors::Http503Error) do |e|
|
6
|
+
Rails.logger.error "Kinesis Error - Server Error occurred - #{e.class}"
|
7
|
+
raise KinesisTemporaryFailure
|
8
|
+
end
|
9
|
+
|
10
|
+
rescue_from(Seahorse::Client::NetworkingError) do |e|
|
11
|
+
Rails.logger.error "Kinesis Error - Networking Error occurred - #{e.class}"
|
12
|
+
raise KinesisTemporaryFailure
|
13
|
+
end
|
14
|
+
|
15
|
+
def perform(serialized_event:, partition_key:, app_name:)
|
16
|
+
@serialized_event = serialized_event
|
17
|
+
@partition_key = partition_key
|
18
|
+
@app_name = app_name
|
19
|
+
|
20
|
+
journal!
|
21
|
+
end
|
22
|
+
|
23
|
+
def self.stream_name(app_name:)
|
24
|
+
env_var_name = [app_name&.upcase, 'JOURNALED_STREAM_NAME'].compact.join('_')
|
25
|
+
ENV.fetch(env_var_name)
|
26
|
+
end
|
27
|
+
|
28
|
+
def kinesis_client_config
|
29
|
+
{
|
30
|
+
region: ENV.fetch('AWS_DEFAULT_REGION', DEFAULT_REGION),
|
31
|
+
retry_limit: 0,
|
32
|
+
http_idle_timeout: Journaled.http_idle_timeout,
|
33
|
+
http_open_timeout: Journaled.http_open_timeout,
|
34
|
+
http_read_timeout: Journaled.http_read_timeout,
|
35
|
+
}.merge(credentials)
|
36
|
+
end
|
37
|
+
|
38
|
+
private
|
39
|
+
|
40
|
+
attr_reader :serialized_event, :partition_key, :app_name
|
41
|
+
|
42
|
+
def journal!
|
43
|
+
kinesis_client.put_record record if Journaled.enabled?
|
44
|
+
end
|
45
|
+
|
46
|
+
def record
|
47
|
+
{
|
48
|
+
stream_name: self.class.stream_name(app_name: app_name),
|
49
|
+
data: serialized_event,
|
50
|
+
partition_key: partition_key,
|
51
|
+
}
|
52
|
+
end
|
53
|
+
|
54
|
+
def kinesis_client
|
55
|
+
Aws::Kinesis::Client.new(kinesis_client_config)
|
56
|
+
end
|
57
|
+
|
58
|
+
def credentials
|
59
|
+
if ENV.key?('JOURNALED_IAM_ROLE_ARN')
|
60
|
+
{
|
61
|
+
credentials: iam_assume_role_credentials,
|
62
|
+
}
|
63
|
+
else
|
64
|
+
legacy_credentials_hash_if_present
|
65
|
+
end
|
66
|
+
end
|
67
|
+
|
68
|
+
def legacy_credentials_hash_if_present
|
69
|
+
if ENV.key?('RUBY_AWS_ACCESS_KEY_ID')
|
70
|
+
{
|
71
|
+
access_key_id: ENV.fetch('RUBY_AWS_ACCESS_KEY_ID'),
|
72
|
+
secret_access_key: ENV.fetch('RUBY_AWS_SECRET_ACCESS_KEY'),
|
73
|
+
}
|
74
|
+
else
|
75
|
+
{}
|
76
|
+
end
|
77
|
+
end
|
78
|
+
|
79
|
+
def sts_client
|
80
|
+
Aws::STS::Client.new({
|
81
|
+
region: ENV.fetch('AWS_DEFAULT_REGION', DEFAULT_REGION),
|
82
|
+
}.merge(legacy_credentials_hash_if_present))
|
83
|
+
end
|
84
|
+
|
85
|
+
def iam_assume_role_credentials
|
86
|
+
@iam_assume_role_credentials ||= Aws::AssumeRoleCredentials.new(
|
87
|
+
client: sts_client,
|
88
|
+
role_arn: ENV.fetch('JOURNALED_IAM_ROLE_ARN'),
|
89
|
+
role_session_name: "JournaledAssumeRoleAccess",
|
90
|
+
)
|
91
|
+
end
|
92
|
+
|
93
|
+
class KinesisTemporaryFailure < NotTrulyExceptionalError
|
94
|
+
end
|
95
|
+
end
|
96
|
+
end
|
@@ -2,10 +2,9 @@ module Journaled::Changes
|
|
2
2
|
extend ActiveSupport::Concern
|
3
3
|
|
4
4
|
included do
|
5
|
-
cattr_accessor
|
6
|
-
cattr_accessor
|
7
|
-
|
8
|
-
self.journaled_attribute_names = []
|
5
|
+
cattr_accessor(:_journaled_change_definitions) { [] }
|
6
|
+
cattr_accessor(:journaled_attribute_names) { [] }
|
7
|
+
cattr_accessor(:journaled_enqueue_opts, instance_writer: false) { {} }
|
9
8
|
|
10
9
|
after_create do
|
11
10
|
self.class._journaled_change_definitions.each do |definition|
|
@@ -57,7 +56,7 @@ module Journaled::Changes
|
|
57
56
|
end
|
58
57
|
|
59
58
|
class_methods do
|
60
|
-
def journal_changes_to(*attribute_names, as:) # rubocop:disable Naming/UncommunicativeMethodParamName
|
59
|
+
def journal_changes_to(*attribute_names, as:, enqueue_with: {}) # rubocop:disable Naming/UncommunicativeMethodParamName
|
61
60
|
if attribute_names.empty? || attribute_names.any? { |n| !n.is_a?(Symbol) }
|
62
61
|
raise "one or more symbol attribute_name arguments is required"
|
63
62
|
end
|
@@ -66,6 +65,7 @@ module Journaled::Changes
|
|
66
65
|
|
67
66
|
_journaled_change_definitions << Journaled::ChangeDefinition.new(attribute_names: attribute_names, logical_operation: as)
|
68
67
|
journaled_attribute_names.concat(attribute_names)
|
68
|
+
journaled_enqueue_opts.merge!(enqueue_with)
|
69
69
|
end
|
70
70
|
|
71
71
|
if Rails::VERSION::MAJOR > 5 || (Rails::VERSION::MAJOR == 5 && Rails::VERSION::MINOR >= 2)
|
@@ -7,6 +7,7 @@ class Journaled::Change
|
|
7
7
|
:logical_operation,
|
8
8
|
:changes,
|
9
9
|
:journaled_app_name,
|
10
|
+
:journaled_enqueue_opts,
|
10
11
|
:actor
|
11
12
|
|
12
13
|
journal_attributes :table_name,
|
@@ -22,6 +23,7 @@ class Journaled::Change
|
|
22
23
|
logical_operation:,
|
23
24
|
changes:,
|
24
25
|
journaled_app_name:,
|
26
|
+
journaled_enqueue_opts:,
|
25
27
|
actor:)
|
26
28
|
@table_name = table_name
|
27
29
|
@record_id = record_id
|
@@ -29,6 +31,7 @@ class Journaled::Change
|
|
29
31
|
@logical_operation = logical_operation
|
30
32
|
@changes = changes
|
31
33
|
@journaled_app_name = journaled_app_name
|
34
|
+
@journaled_enqueue_opts = journaled_enqueue_opts
|
32
35
|
@actor = actor
|
33
36
|
end
|
34
37
|
end
|
@@ -1,4 +1,4 @@
|
|
1
|
-
class Journaled::Delivery
|
1
|
+
class Journaled::Delivery # rubocop:disable Betterment/ActiveJobPerformable
|
2
2
|
DEFAULT_REGION = 'us-east-1'.freeze
|
3
3
|
|
4
4
|
def initialize(serialized_event:, partition_key:, app_name:)
|
@@ -26,6 +26,9 @@ class Journaled::Delivery
|
|
26
26
|
{
|
27
27
|
region: ENV.fetch('AWS_DEFAULT_REGION', DEFAULT_REGION),
|
28
28
|
retry_limit: 0,
|
29
|
+
http_idle_timeout: Journaled.http_idle_timeout,
|
30
|
+
http_open_timeout: Journaled.http_open_timeout,
|
31
|
+
http_read_timeout: Journaled.http_read_timeout,
|
29
32
|
}.merge(credentials)
|
30
33
|
end
|
31
34
|
|
@@ -80,6 +83,6 @@ class Journaled::Delivery
|
|
80
83
|
)
|
81
84
|
end
|
82
85
|
|
83
|
-
class KinesisTemporaryFailure < NotTrulyExceptionalError
|
86
|
+
class KinesisTemporaryFailure < ::Journaled::NotTrulyExceptionalError
|
84
87
|
end
|
85
88
|
end
|
@@ -2,7 +2,7 @@ module Journaled::Event
|
|
2
2
|
extend ActiveSupport::Concern
|
3
3
|
|
4
4
|
def journal!
|
5
|
-
Journaled::Writer.new(journaled_event: self
|
5
|
+
Journaled::Writer.new(journaled_event: self).journal!
|
6
6
|
end
|
7
7
|
|
8
8
|
# Base attributes
|
@@ -42,8 +42,9 @@ module Journaled::Event
|
|
42
42
|
private
|
43
43
|
|
44
44
|
class_methods do
|
45
|
-
def journal_attributes(*args)
|
45
|
+
def journal_attributes(*args, enqueue_with: {})
|
46
46
|
journaled_attributes.concat(args)
|
47
|
+
journaled_enqueue_opts.merge!(enqueue_with)
|
47
48
|
end
|
48
49
|
|
49
50
|
def journaled_attributes
|
@@ -56,6 +57,8 @@ module Journaled::Event
|
|
56
57
|
end
|
57
58
|
|
58
59
|
included do
|
60
|
+
cattr_accessor(:journaled_enqueue_opts, instance_writer: false) { {} }
|
61
|
+
|
59
62
|
journal_attributes :id, :event_type, :created_at
|
60
63
|
end
|
61
64
|
end
|
@@ -4,9 +4,10 @@ class Journaled::Writer
|
|
4
4
|
journaled_partition_key
|
5
5
|
journaled_attributes
|
6
6
|
journaled_app_name
|
7
|
+
journaled_enqueue_opts
|
7
8
|
).freeze
|
8
9
|
|
9
|
-
def initialize(journaled_event
|
10
|
+
def initialize(journaled_event:)
|
10
11
|
raise "An enqueued event must respond to: #{EVENT_METHOD_NAMES.to_sentence}" unless respond_to_all?(journaled_event, EVENT_METHOD_NAMES)
|
11
12
|
|
12
13
|
unless journaled_event.journaled_schema_name.present? &&
|
@@ -21,26 +22,27 @@ class Journaled::Writer
|
|
21
22
|
end
|
22
23
|
|
23
24
|
@journaled_event = journaled_event
|
24
|
-
@priority = priority
|
25
25
|
end
|
26
26
|
|
27
27
|
def journal!
|
28
28
|
base_event_json_schema_validator.validate! serialized_event
|
29
29
|
json_schema_validator.validate! serialized_event
|
30
|
-
|
30
|
+
Journaled::DeliveryJob
|
31
|
+
.set(journaled_enqueue_opts.reverse_merge(priority: Journaled.job_priority))
|
32
|
+
.perform_later(delivery_perform_args)
|
31
33
|
end
|
32
34
|
|
33
35
|
private
|
34
36
|
|
35
|
-
attr_reader :journaled_event
|
36
|
-
delegate
|
37
|
+
attr_reader :journaled_event
|
38
|
+
delegate(*EVENT_METHOD_NAMES, to: :journaled_event)
|
37
39
|
|
38
|
-
def
|
39
|
-
|
40
|
+
def delivery_perform_args
|
41
|
+
{
|
40
42
|
serialized_event: serialized_event,
|
41
43
|
partition_key: journaled_partition_key,
|
42
44
|
app_name: journaled_app_name,
|
43
|
-
|
45
|
+
}
|
44
46
|
end
|
45
47
|
|
46
48
|
def serialized_event
|
data/lib/journaled.rb
CHANGED
@@ -1,14 +1,19 @@
|
|
1
|
-
require "aws-sdk-
|
2
|
-
require "
|
1
|
+
require "aws-sdk-kinesis"
|
2
|
+
require "active_job"
|
3
3
|
require "json-schema"
|
4
4
|
require "request_store"
|
5
5
|
|
6
6
|
require "journaled/engine"
|
7
|
-
require "journaled/job_priority"
|
8
7
|
|
9
8
|
module Journaled
|
9
|
+
SUPPORTED_QUEUE_ADAPTERS = %w(delayed delayed_job good_job que).freeze
|
10
|
+
|
10
11
|
mattr_accessor :default_app_name
|
11
|
-
mattr_accessor(:job_priority) {
|
12
|
+
mattr_accessor(:job_priority) { 20 }
|
13
|
+
mattr_accessor(:http_idle_timeout) { 5 }
|
14
|
+
mattr_accessor(:http_open_timeout) { 2 }
|
15
|
+
mattr_accessor(:http_read_timeout) { 60 }
|
16
|
+
mattr_accessor(:job_base_class_name) { 'ActiveJob::Base' }
|
12
17
|
|
13
18
|
def development_or_test?
|
14
19
|
%w(development test).include?(Rails.env)
|
@@ -30,5 +35,21 @@ module Journaled
|
|
30
35
|
Journaled::ActorUriProvider.instance.actor_uri
|
31
36
|
end
|
32
37
|
|
33
|
-
|
38
|
+
def detect_queue_adapter!
|
39
|
+
adapter = job_base_class_name.constantize.queue_adapter.class.name.split('::').last.underscore.gsub("_adapter", "")
|
40
|
+
unless SUPPORTED_QUEUE_ADAPTERS.include?(adapter)
|
41
|
+
raise <<~MSG
|
42
|
+
Journaled has detected an unsupported ActiveJob queue adapter: `:#{adapter}`
|
43
|
+
|
44
|
+
Journaled jobs must be enqueued transactionally to your primary database.
|
45
|
+
|
46
|
+
Please install the appropriate gems and set `queue_adapter` to one of the following:
|
47
|
+
#{SUPPORTED_QUEUE_ADAPTERS.map { |a| "- `:#{a}`" }.join("\n")}
|
48
|
+
|
49
|
+
Read more at https://github.com/Betterment/journaled
|
50
|
+
MSG
|
51
|
+
end
|
52
|
+
end
|
53
|
+
|
54
|
+
module_function :development_or_test?, :enabled?, :schema_providers, :commit_hash, :actor_uri, :detect_queue_adapter!
|
34
55
|
end
|
data/lib/journaled/engine.rb
CHANGED
@@ -1,5 +1,5 @@
|
|
1
1
|
module Journaled::RelationChangeProtection
|
2
|
-
def update_all(updates, force: false) # rubocop:disable Metrics/
|
2
|
+
def update_all(updates, force: false) # rubocop:disable Metrics/AbcSize, Metrics/PerceivedComplexity
|
3
3
|
unless force || !@klass.respond_to?(:journaled_attribute_names) || @klass.journaled_attribute_names.empty?
|
4
4
|
conflicting_journaled_attribute_names = if updates.is_a?(Hash)
|
5
5
|
@klass.journaled_attribute_names & updates.keys.map(&:to_sym)
|
data/lib/journaled/version.rb
CHANGED