journaled 2.2.0 → 3.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (39) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +42 -6
  3. data/Rakefile +7 -1
  4. data/app/jobs/journaled/application_job.rb +4 -0
  5. data/app/jobs/journaled/delivery_job.rb +96 -0
  6. data/app/models/concerns/journaled/changes.rb +5 -5
  7. data/app/models/journaled/change.rb +3 -0
  8. data/app/models/journaled/change_writer.rb +1 -0
  9. data/app/models/journaled/delivery.rb +5 -2
  10. data/app/models/journaled/event.rb +5 -2
  11. data/app/models/journaled/writer.rb +10 -8
  12. data/lib/journaled.rb +26 -5
  13. data/lib/journaled/engine.rb +5 -0
  14. data/lib/journaled/relation_change_protection.rb +1 -1
  15. data/lib/journaled/version.rb +1 -1
  16. data/spec/dummy/config/application.rb +1 -2
  17. data/spec/dummy/config/database.yml +4 -19
  18. data/spec/dummy/config/environments/development.rb +0 -13
  19. data/spec/dummy/config/environments/test.rb +3 -5
  20. data/spec/dummy/db/schema.rb +3 -16
  21. data/spec/jobs/journaled/delivery_job_spec.rb +221 -0
  22. data/spec/lib/journaled_spec.rb +39 -0
  23. data/spec/models/concerns/journaled/changes_spec.rb +11 -0
  24. data/spec/models/database_change_protection_spec.rb +19 -25
  25. data/spec/models/journaled/change_writer_spec.rb +5 -0
  26. data/spec/models/journaled/delivery_spec.rb +33 -0
  27. data/spec/models/journaled/event_spec.rb +23 -16
  28. data/spec/models/journaled/writer_spec.rb +34 -18
  29. data/spec/rails_helper.rb +1 -2
  30. data/spec/spec_helper.rb +1 -3
  31. metadata +38 -62
  32. data/config/routes.rb +0 -2
  33. data/lib/journaled/job_priority.rb +0 -5
  34. data/spec/dummy/config/environments/production.rb +0 -78
  35. data/spec/dummy/config/initializers/assets.rb +0 -8
  36. data/spec/dummy/db/migrate/20180606205114_create_delayed_jobs.rb +0 -18
  37. data/spec/dummy/log/development.log +0 -29
  38. data/spec/dummy/log/test.log +0 -2482
  39. data/spec/support/delayed_job_spec_helper.rb +0 -11
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 5f46775bfd5621ba7978529c64da7768d471ee6fd73cc8cf315d8fa60e98c537
4
- data.tar.gz: f0bbb3d98be52d715d847a1ee44c843d8c4f654826939167a710b04a23c4adb6
3
+ metadata.gz: 358c3ced2b976724c62e951ca88b635709d1fb7da0276ed4538bdc44bc02f468
4
+ data.tar.gz: ca65a5b0531ba4540de1b79fd36d247c7c164f3aef8b397254ed7b34503ebf0f
5
5
  SHA512:
6
- metadata.gz: 82c607495e4dc58e70005a73b32d473d84d85baaa4b3004d8ce018ea65d178a4eafafee7c64336ba3616dc4d6b65349bbc4d2209a770aa858cfdd92f9c07043d
7
- data.tar.gz: 803c3f09b67ac93ff28df87ad40680f7c05daed85c7da7110c35c817ccd2cd4f2a4b1995540f5659ce5b3a839b124f4fccbeed91686f540280aa0abe25bb05c9
6
+ metadata.gz: 236c6169ef41b1af9043f2549248ae43fad665a91d43cdd2f579e891685788aa5a008cf4f1dc8aba13abcce5dccfa6513c82232fdaaf3ab4f42f8a7b1b222bf7
7
+ data.tar.gz: 9e325f6c1e41eb577f5e0d7320517f5435b9570d7b0274c5b6ad9c593abc2a4a66dc6b6eb10d668617e645e5731804ecd65d4b6403fda63005de129443ed9332
data/README.md CHANGED
@@ -1,16 +1,16 @@
1
1
  # Journaled
2
2
 
3
- A Rails engine to durably deliver schematized events to Amazon Kinesis via DelayedJob.
3
+ A Rails engine to durably deliver schematized events to Amazon Kinesis via ActiveJob.
4
4
 
5
5
  More specifically, `journaled` is composed of three opinionated pieces:
6
6
  schema definition/validation via JSON Schema, transactional enqueueing
7
- via Delayed::Job (specifically `delayed_job_active_record`), and event
7
+ via ActiveJob (specifically, via a DB-backed queue adapter), and event
8
8
  transmission via Amazon Kinesis. Our current use-cases include
9
9
  transmitting audit events for durable storage in S3 and/or analytical
10
10
  querying in Amazon Redshift.
11
11
 
12
12
  Journaled provides an at-least-once event delivery guarantee assuming
13
- Delayed::Job is configured not to delete jobs on failure.
13
+ ActiveJob's queue adapter is not configured to delete jobs on failure.
14
14
 
15
15
  Note: Do not use the journaled gem to build an event sourcing solution
16
16
  as it does not guarantee total ordering of events. It's possible we'll
@@ -20,9 +20,20 @@ durable, eventually consistent record that discrete events happened.
20
20
 
21
21
  ## Installation
22
22
 
23
- 1. [Install `delayed_job_active_record`](https://github.com/collectiveidea/delayed_job_active_record#installation)
24
- if you haven't already.
23
+ 1. If you haven't already,
24
+ [configure ActiveJob](https://guides.rubyonrails.org/active_job_basics.html)
25
+ to use one of the following queue adapters:
25
26
 
27
+ - `:delayed_job` (via `delayed_job_active_record`)
28
+ - `:que`
29
+ - `:good_job`
30
+ - `:delayed`
31
+
32
+ Ensure that your queue adapter is not configured to delete jobs on failure.
33
+
34
+ **If you launch your application in production mode and the gem detects that
35
+ `ActiveJob::Base.queue_adapter` is not in the above list, it will raise an exception
36
+ and prevent your application from performing unsafe journaling.**
26
37
 
27
38
  2. To integrate Journaled into your application, simply include the gem in your
28
39
  app's Gemfile.
@@ -85,9 +96,34 @@ Journaling provides a number of different configuation options that can be set i
85
96
 
86
97
  #### `Journaled.job_priority` (default: 20)
87
98
 
88
- This can be used to configure what `priority` the Delayed Jobs are enqueued with. This will be applied to all the Journaled::Devivery jobs that are created by this application.
99
+ This can be used to configure what `priority` the ActiveJobs are enqueued with. This will be applied to all the `Journaled::DeliveryJob`s that are created by this application.
89
100
  Ex: `Journaled.job_priority = 14`
90
101
 
102
+ _Note that job priority is only supported on Rails 6.0+. Prior Rails versions will ignore this parameter and enqueue jobs with the underlying ActiveJob adapter's default priority._
103
+
104
+ #### `Journaled.http_idle_timeout` (default: 1 second)
105
+
106
+ The number of seconds a persistent connection is allowed to sit idle before it should no longer be used.
107
+
108
+ #### `Journaled.http_open_timeout` (default: 2 seconds)
109
+
110
+ The number of seconds before the :http_handler should timeout while trying to open a new HTTP session.
111
+
112
+ #### `Journaled.http_read_timeout` (default: 60 seconds)
113
+
114
+ The number of seconds before the :http_handler should timeout while waiting for a HTTP response.
115
+
116
+ #### ActiveJob `set` options
117
+
118
+ Both model-level directives accept additional options to be passed into ActiveJob's `set` method:
119
+
120
+ ```ruby
121
+ # For change journaling:
122
+ journal_changes_to :email, as: :identity_change, enqueue_with: { priority: 10 }
123
+
124
+ # Or for custom journaling:
125
+ journal_attributes :email, enqueue_with: { priority: 20, queue: 'journaled' }
126
+ ```
91
127
 
92
128
  ### Change Journaling
93
129
 
data/Rakefile CHANGED
@@ -28,7 +28,13 @@ if %w(development test).include? Rails.env
28
28
  RuboCop::RakeTask.new
29
29
 
30
30
  task(:default).clear
31
- task default: %i(rubocop spec)
31
+ if ENV['APPRAISAL_INITIALIZED'] || ENV['CI']
32
+ task default: %i(rubocop spec)
33
+ else
34
+ require 'appraisal'
35
+ Appraisal::Task.new
36
+ task default: :appraisal
37
+ end
32
38
 
33
39
  task 'db:test:prepare' => 'db:setup'
34
40
  end
@@ -0,0 +1,4 @@
1
+ module Journaled
2
+ class ApplicationJob < Journaled.job_base_class_name.constantize
3
+ end
4
+ end
@@ -0,0 +1,96 @@
1
+ module Journaled
2
+ class DeliveryJob < ApplicationJob
3
+ DEFAULT_REGION = 'us-east-1'.freeze
4
+
5
+ rescue_from(Aws::Kinesis::Errors::InternalFailure, Aws::Kinesis::Errors::ServiceUnavailable, Aws::Kinesis::Errors::Http503Error) do |e|
6
+ Rails.logger.error "Kinesis Error - Server Error occurred - #{e.class}"
7
+ raise KinesisTemporaryFailure
8
+ end
9
+
10
+ rescue_from(Seahorse::Client::NetworkingError) do |e|
11
+ Rails.logger.error "Kinesis Error - Networking Error occurred - #{e.class}"
12
+ raise KinesisTemporaryFailure
13
+ end
14
+
15
+ def perform(serialized_event:, partition_key:, app_name:)
16
+ @serialized_event = serialized_event
17
+ @partition_key = partition_key
18
+ @app_name = app_name
19
+
20
+ journal!
21
+ end
22
+
23
+ def self.stream_name(app_name:)
24
+ env_var_name = [app_name&.upcase, 'JOURNALED_STREAM_NAME'].compact.join('_')
25
+ ENV.fetch(env_var_name)
26
+ end
27
+
28
+ def kinesis_client_config
29
+ {
30
+ region: ENV.fetch('AWS_DEFAULT_REGION', DEFAULT_REGION),
31
+ retry_limit: 0,
32
+ http_idle_timeout: Journaled.http_idle_timeout,
33
+ http_open_timeout: Journaled.http_open_timeout,
34
+ http_read_timeout: Journaled.http_read_timeout,
35
+ }.merge(credentials)
36
+ end
37
+
38
+ private
39
+
40
+ attr_reader :serialized_event, :partition_key, :app_name
41
+
42
+ def journal!
43
+ kinesis_client.put_record record if Journaled.enabled?
44
+ end
45
+
46
+ def record
47
+ {
48
+ stream_name: self.class.stream_name(app_name: app_name),
49
+ data: serialized_event,
50
+ partition_key: partition_key,
51
+ }
52
+ end
53
+
54
+ def kinesis_client
55
+ Aws::Kinesis::Client.new(kinesis_client_config)
56
+ end
57
+
58
+ def credentials
59
+ if ENV.key?('JOURNALED_IAM_ROLE_ARN')
60
+ {
61
+ credentials: iam_assume_role_credentials,
62
+ }
63
+ else
64
+ legacy_credentials_hash_if_present
65
+ end
66
+ end
67
+
68
+ def legacy_credentials_hash_if_present
69
+ if ENV.key?('RUBY_AWS_ACCESS_KEY_ID')
70
+ {
71
+ access_key_id: ENV.fetch('RUBY_AWS_ACCESS_KEY_ID'),
72
+ secret_access_key: ENV.fetch('RUBY_AWS_SECRET_ACCESS_KEY'),
73
+ }
74
+ else
75
+ {}
76
+ end
77
+ end
78
+
79
+ def sts_client
80
+ Aws::STS::Client.new({
81
+ region: ENV.fetch('AWS_DEFAULT_REGION', DEFAULT_REGION),
82
+ }.merge(legacy_credentials_hash_if_present))
83
+ end
84
+
85
+ def iam_assume_role_credentials
86
+ @iam_assume_role_credentials ||= Aws::AssumeRoleCredentials.new(
87
+ client: sts_client,
88
+ role_arn: ENV.fetch('JOURNALED_IAM_ROLE_ARN'),
89
+ role_session_name: "JournaledAssumeRoleAccess",
90
+ )
91
+ end
92
+
93
+ class KinesisTemporaryFailure < NotTrulyExceptionalError
94
+ end
95
+ end
96
+ end
@@ -2,10 +2,9 @@ module Journaled::Changes
2
2
  extend ActiveSupport::Concern
3
3
 
4
4
  included do
5
- cattr_accessor :_journaled_change_definitions
6
- cattr_accessor :journaled_attribute_names
7
- self._journaled_change_definitions = []
8
- self.journaled_attribute_names = []
5
+ cattr_accessor(:_journaled_change_definitions) { [] }
6
+ cattr_accessor(:journaled_attribute_names) { [] }
7
+ cattr_accessor(:journaled_enqueue_opts, instance_writer: false) { {} }
9
8
 
10
9
  after_create do
11
10
  self.class._journaled_change_definitions.each do |definition|
@@ -57,7 +56,7 @@ module Journaled::Changes
57
56
  end
58
57
 
59
58
  class_methods do
60
- def journal_changes_to(*attribute_names, as:) # rubocop:disable Naming/UncommunicativeMethodParamName
59
+ def journal_changes_to(*attribute_names, as:, enqueue_with: {}) # rubocop:disable Naming/UncommunicativeMethodParamName
61
60
  if attribute_names.empty? || attribute_names.any? { |n| !n.is_a?(Symbol) }
62
61
  raise "one or more symbol attribute_name arguments is required"
63
62
  end
@@ -66,6 +65,7 @@ module Journaled::Changes
66
65
 
67
66
  _journaled_change_definitions << Journaled::ChangeDefinition.new(attribute_names: attribute_names, logical_operation: as)
68
67
  journaled_attribute_names.concat(attribute_names)
68
+ journaled_enqueue_opts.merge!(enqueue_with)
69
69
  end
70
70
 
71
71
  if Rails::VERSION::MAJOR > 5 || (Rails::VERSION::MAJOR == 5 && Rails::VERSION::MINOR >= 2)
@@ -7,6 +7,7 @@ class Journaled::Change
7
7
  :logical_operation,
8
8
  :changes,
9
9
  :journaled_app_name,
10
+ :journaled_enqueue_opts,
10
11
  :actor
11
12
 
12
13
  journal_attributes :table_name,
@@ -22,6 +23,7 @@ class Journaled::Change
22
23
  logical_operation:,
23
24
  changes:,
24
25
  journaled_app_name:,
26
+ journaled_enqueue_opts:,
25
27
  actor:)
26
28
  @table_name = table_name
27
29
  @record_id = record_id
@@ -29,6 +31,7 @@ class Journaled::Change
29
31
  @logical_operation = logical_operation
30
32
  @changes = changes
31
33
  @journaled_app_name = journaled_app_name
34
+ @journaled_enqueue_opts = journaled_enqueue_opts
32
35
  @actor = actor
33
36
  end
34
37
  end
@@ -28,6 +28,7 @@ class Journaled::ChangeWriter
28
28
  logical_operation: logical_operation,
29
29
  changes: JSON.dump(changes),
30
30
  journaled_app_name: journaled_app_name,
31
+ journaled_enqueue_opts: model.journaled_enqueue_opts,
31
32
  actor: actor_uri,
32
33
  )
33
34
  end
@@ -1,4 +1,4 @@
1
- class Journaled::Delivery
1
+ class Journaled::Delivery # rubocop:disable Betterment/ActiveJobPerformable
2
2
  DEFAULT_REGION = 'us-east-1'.freeze
3
3
 
4
4
  def initialize(serialized_event:, partition_key:, app_name:)
@@ -26,6 +26,9 @@ class Journaled::Delivery
26
26
  {
27
27
  region: ENV.fetch('AWS_DEFAULT_REGION', DEFAULT_REGION),
28
28
  retry_limit: 0,
29
+ http_idle_timeout: Journaled.http_idle_timeout,
30
+ http_open_timeout: Journaled.http_open_timeout,
31
+ http_read_timeout: Journaled.http_read_timeout,
29
32
  }.merge(credentials)
30
33
  end
31
34
 
@@ -80,6 +83,6 @@ class Journaled::Delivery
80
83
  )
81
84
  end
82
85
 
83
- class KinesisTemporaryFailure < NotTrulyExceptionalError
86
+ class KinesisTemporaryFailure < ::Journaled::NotTrulyExceptionalError
84
87
  end
85
88
  end
@@ -2,7 +2,7 @@ module Journaled::Event
2
2
  extend ActiveSupport::Concern
3
3
 
4
4
  def journal!
5
- Journaled::Writer.new(journaled_event: self, priority: Journaled.job_priority).journal!
5
+ Journaled::Writer.new(journaled_event: self).journal!
6
6
  end
7
7
 
8
8
  # Base attributes
@@ -42,8 +42,9 @@ module Journaled::Event
42
42
  private
43
43
 
44
44
  class_methods do
45
- def journal_attributes(*args)
45
+ def journal_attributes(*args, enqueue_with: {})
46
46
  journaled_attributes.concat(args)
47
+ journaled_enqueue_opts.merge!(enqueue_with)
47
48
  end
48
49
 
49
50
  def journaled_attributes
@@ -56,6 +57,8 @@ module Journaled::Event
56
57
  end
57
58
 
58
59
  included do
60
+ cattr_accessor(:journaled_enqueue_opts, instance_writer: false) { {} }
61
+
59
62
  journal_attributes :id, :event_type, :created_at
60
63
  end
61
64
  end
@@ -4,9 +4,10 @@ class Journaled::Writer
4
4
  journaled_partition_key
5
5
  journaled_attributes
6
6
  journaled_app_name
7
+ journaled_enqueue_opts
7
8
  ).freeze
8
9
 
9
- def initialize(journaled_event:, priority:)
10
+ def initialize(journaled_event:)
10
11
  raise "An enqueued event must respond to: #{EVENT_METHOD_NAMES.to_sentence}" unless respond_to_all?(journaled_event, EVENT_METHOD_NAMES)
11
12
 
12
13
  unless journaled_event.journaled_schema_name.present? &&
@@ -21,26 +22,27 @@ class Journaled::Writer
21
22
  end
22
23
 
23
24
  @journaled_event = journaled_event
24
- @priority = priority
25
25
  end
26
26
 
27
27
  def journal!
28
28
  base_event_json_schema_validator.validate! serialized_event
29
29
  json_schema_validator.validate! serialized_event
30
- Delayed::Job.enqueue journaled_delivery, priority: priority
30
+ Journaled::DeliveryJob
31
+ .set(journaled_enqueue_opts.reverse_merge(priority: Journaled.job_priority))
32
+ .perform_later(delivery_perform_args)
31
33
  end
32
34
 
33
35
  private
34
36
 
35
- attr_reader :journaled_event, :priority
36
- delegate :journaled_schema_name, :journaled_attributes, :journaled_partition_key, :journaled_app_name, to: :journaled_event
37
+ attr_reader :journaled_event
38
+ delegate(*EVENT_METHOD_NAMES, to: :journaled_event)
37
39
 
38
- def journaled_delivery
39
- @journaled_delivery ||= Journaled::Delivery.new(
40
+ def delivery_perform_args
41
+ {
40
42
  serialized_event: serialized_event,
41
43
  partition_key: journaled_partition_key,
42
44
  app_name: journaled_app_name,
43
- )
45
+ }
44
46
  end
45
47
 
46
48
  def serialized_event
data/lib/journaled.rb CHANGED
@@ -1,14 +1,19 @@
1
- require "aws-sdk-resources"
2
- require "delayed_job"
1
+ require "aws-sdk-kinesis"
2
+ require "active_job"
3
3
  require "json-schema"
4
4
  require "request_store"
5
5
 
6
6
  require "journaled/engine"
7
- require "journaled/job_priority"
8
7
 
9
8
  module Journaled
9
+ SUPPORTED_QUEUE_ADAPTERS = %w(delayed delayed_job good_job que).freeze
10
+
10
11
  mattr_accessor :default_app_name
11
- mattr_accessor(:job_priority) { Journaled::JobPriority::EVENTUAL }
12
+ mattr_accessor(:job_priority) { 20 }
13
+ mattr_accessor(:http_idle_timeout) { 5 }
14
+ mattr_accessor(:http_open_timeout) { 2 }
15
+ mattr_accessor(:http_read_timeout) { 60 }
16
+ mattr_accessor(:job_base_class_name) { 'ActiveJob::Base' }
12
17
 
13
18
  def development_or_test?
14
19
  %w(development test).include?(Rails.env)
@@ -30,5 +35,21 @@ module Journaled
30
35
  Journaled::ActorUriProvider.instance.actor_uri
31
36
  end
32
37
 
33
- module_function :development_or_test?, :enabled?, :schema_providers, :commit_hash, :actor_uri
38
+ def detect_queue_adapter!
39
+ adapter = job_base_class_name.constantize.queue_adapter.class.name.split('::').last.underscore.gsub("_adapter", "")
40
+ unless SUPPORTED_QUEUE_ADAPTERS.include?(adapter)
41
+ raise <<~MSG
42
+ Journaled has detected an unsupported ActiveJob queue adapter: `:#{adapter}`
43
+
44
+ Journaled jobs must be enqueued transactionally to your primary database.
45
+
46
+ Please install the appropriate gems and set `queue_adapter` to one of the following:
47
+ #{SUPPORTED_QUEUE_ADAPTERS.map { |a| "- `:#{a}`" }.join("\n")}
48
+
49
+ Read more at https://github.com/Betterment/journaled
50
+ MSG
51
+ end
52
+ end
53
+
54
+ module_function :development_or_test?, :enabled?, :schema_providers, :commit_hash, :actor_uri, :detect_queue_adapter!
34
55
  end
@@ -1,4 +1,9 @@
1
1
  module Journaled
2
2
  class Engine < ::Rails::Engine
3
+ config.after_initialize do
4
+ ActiveSupport.on_load(:active_job) do
5
+ Journaled.detect_queue_adapter! unless Journaled.development_or_test?
6
+ end
7
+ end
3
8
  end
4
9
  end
@@ -1,5 +1,5 @@
1
1
  module Journaled::RelationChangeProtection
2
- def update_all(updates, force: false) # rubocop:disable Metrics/MethodLength, Metrics/AbcSize, Metrics/PerceivedComplexity
2
+ def update_all(updates, force: false) # rubocop:disable Metrics/AbcSize, Metrics/PerceivedComplexity
3
3
  unless force || !@klass.respond_to?(:journaled_attribute_names) || @klass.journaled_attribute_names.empty?
4
4
  conflicting_journaled_attribute_names = if updates.is_a?(Hash)
5
5
  @klass.journaled_attribute_names & updates.keys.map(&:to_sym)
@@ -1,3 +1,3 @@
1
1
  module Journaled
2
- VERSION = "2.2.0".freeze
2
+ VERSION = "3.0.0".freeze
3
3
  end