journaled 4.1.0 → 5.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 88602263644ea51719a18c5ee3e719d2c13a9e69049e1f20a52a44edc38a0835
-  data.tar.gz: c7cafdb9f305e51c3590d09815d5d6f909db496866c774ab442ffaa334be927c
+  metadata.gz: ac63d8988a5cbbd63340c019b82642a039a9dfee81f46d83b15e5a71d83e9cb7
+  data.tar.gz: ae809e4b05901d6eb73eb0196119999be2b0fd414c3af3c4bb30d280216098cf
 SHA512:
-  metadata.gz: 6bf229efd53882ed50931c964c1564b01603fa22ea2083b8a6039d0a0ed972d2cdc7bfc5913a9e9539d7dd023294c2a226963e2e2a526bf857926afe3e0c76c0
-  data.tar.gz: cf7b28c58c40c9ee8348df60095a7b544c0a8b0ea448ffa20021aceda61ae2feda61110ef9685756d4bac7d3d887e12a181f1840d435b2d2f85aeba5d8a52e45
+  metadata.gz: 72bcbe0ae43717280eb8a0b3383672cc87762e4c2b9b082778f1beac249004ef90a41e32e6fbabd693377a4d0af6cf6fc78df8a2b8ff44067276dd502cece63a
+  data.tar.gz: b21401ba2cf6155a25f50f1c58c634e3036754ec69b8d7f287c86408f69819aed5cfac1a43d55636a2be461545096b1510318a6ce665bf47595f30c8b90bceb9

data/README.md

@@ -179,52 +179,6 @@ journaling. Note that the less-frequently-used methods `toggle`,
 `increment*`, `decrement*`, and `update_counters` are not intercepted at
 this time.
 
-### Tagged Events
-
-Events may be optionally marked as "tagged." This will add a `tags` field, intended for tracing and
-auditing purposes.
-
-```ruby
-class MyEvent
-  include Journaled::Event
-
-  journal_attributes :attr_1, :attr_2, tagged: true
-end
-```
-
-You may then use `Journaled.tag!` and `Journaled.tagged` inside of your
-`ApplicationController` and `ApplicationJob` classes (or anywhere else!) to tag
-all events with request and job metadata:
-
-```ruby
-class ApplicationController < ActionController::Base
-  before_action do
-    Journaled.tag!(request_id: request.request_id, current_user_id: current_user&.id)
-  end
-end
-
-class ApplicationJob < ActiveJob::Base
-  around_perform do |job, perform|
-    Journaled.tagged(job_id: job.id) { perform.call }
-  end
-end
-```
-
-This feature relies on `ActiveSupport::CurrentAttributes` under the hood, so these tags are local to
-the current thread, and will be cleared at the end of each request request/job.
-
-#### Testing
-
-If you use RSpec (and have required `journaled/rspec` in your
-`spec/rails_helper.rb`), you can regression-protect important journaling
-config with the `journal_changes_to` matcher:
-
-```ruby
-it "journals exactly these things or there will be heck to pay" do
-  expect(User).to journal_changes_to(:email, :first_name, :last_name, as: :identity_change)
-end
-```
-
 ### Custom Journaling
 
 For every custom implementation of journaling in your application, define the JSON schema for the attributes in your event.
@@ -309,6 +263,40 @@ An event like the following will be journaled to kinesis:
 }
 ```
 
+### Tagged Events
+
+Events may be optionally marked as "tagged." This will add a `tags` field, intended for tracing and
+auditing purposes.
+
+```ruby
+class MyEvent
+  include Journaled::Event
+
+  journal_attributes :attr_1, :attr_2, tagged: true
+end
+```
+
+You may then use `Journaled.tag!` and `Journaled.tagged` inside of your
+`ApplicationController` and `ApplicationJob` classes (or anywhere else!) to tag
+all events with request and job metadata:
+
+```ruby
+class ApplicationController < ActionController::Base
+  before_action do
+    Journaled.tag!(request_id: request.request_id, current_user_id: current_user&.id)
+  end
+end
+
+class ApplicationJob < ActiveJob::Base
+  around_perform do |job, perform|
+    Journaled.tagged(job_id: job.id) { perform.call }
+  end
+end
+```
+
+This feature relies on `ActiveSupport::CurrentAttributes` under the hood, so these tags are local to
+the current thread, and will be cleared at the end of each request request/job.
+
 ### Helper methods for custom events
 
 Journaled provides a couple helper methods that may be useful in your
@@ -352,6 +340,102 @@ Returns one of the following in order of preference:
 In order for this to be most useful, you must configure your controller
 as described in [Change Journaling](#change-journaling) above.
 
+### Testing
+
+If you use RSpec, you can test for journaling behaviors with the
+`journal_event(s)_including` and `journal_changes_to` matchers. First, make
+sure to require `journaled/rspec` in your spec setup (e.g.
+`spec/rails_helper.rb`):
+
+```ruby
+require 'journaled/rspec'
+```
+
+#### Checking for specific events
+
+The `journal_event_including` and `journal_events_including` matchers allow you
+to check for one or more matching event being journaled:
+
+```ruby
+expect { my_code }
+  .to journal_event_including(name: 'foo')
+expect { my_code }
+  .to journal_events_including({ name: 'foo', value: 1 }, { name: 'foo', value: 2 })
+```
+
+This will only perform matches on the specified fields (and will not match one
+way or the other against unspecified fields). These matchers will also ignore
+any extraneous events that are not positively matched (as they may be unrelated
+to behavior under test).
+
+When writing tests, pairing every positive assertion with a negative assertion
+is a good practice, and so negative matching is also supported (via both
+`.not_to` and `.to not_`):
+
+```ruby
+expect { my_code }
+  .not_to journal_events_including({ name: 'foo' }, { name: 'bar' })
+expect { my_code }
+  .to raise_error(SomeError)
+  .and not_journal_event_including(name: 'foo') # the `not_` variant can chain off of `.and`
+```
+
+Several chainable modifiers are also available:
+
+```ruby
+expect { my_code }.to journal_event_including(name: 'foo')
+  .with_schema_name('my_event_schema')
+  .with_partition_key(user.id)
+  .with_stream_name('my_stream_name')
+  .with_enqueue_opts(run_at: future_time)
+  .with_priority(999)
+```
+
+All of this can be chained together to test for multiple sets of events with
+multiple sets of options:
+
+```ruby
+expect { subject.journal! }
+  .to journal_events_including({ name: 'event1', value: 300 }, { name: 'event2', value: 200 })
+    .with_priority(10)
+  .and journal_event_including(name: 'event3', value: 100)
+    .with_priority(20)
+  .and not_journal_event_including(name: 'other_event')
+```
+
+#### Checking for `Journaled::Changes` declarations
+
+The `journal_changes_to` matcher checks against the list of attributes specified
+on the model. It does not actually test that an event is emitted within a given
+codepath, and is instead intended to guard against accidental regressions that
+may impact external consumers of these events:
+
+```ruby
+it "journals exactly these things or there will be heck to pay" do
+  expect(User).to journal_changes_to(:email, :first_name, :last_name, as: :identity_change)
+end
+```
+
+### Instrumentation
+
+When an event is enqueued, an `ActiveSupport::Notification` titled
+`journaled.event.enqueue` is emitted. Its payload will include the `:event` and
+its background job `:priority`.
+
+This can be forwarded along to your preferred monitoring solution via a Rails
+initializer:
+
+```ruby
+ActiveSupport::Notifications.subscribe('journaled.event.enqueue') do |*args|
+  payload = ActiveSupport::Notifications::Event.new(*args).payload
+  journaled_event = payload[:event]
+
+  tags = { priority: payload[:priority], event_type: journaled_event.journaled_attributes[:event_type] }
+
+  Statsd.increment('journaled.event.enqueue', tags: tags.map { |k,v| "#{k.to_s[0..64]}:#{v.to_s[0..255]}" })
+end
+```
+
 ## Upgrades
 
 Since this gem relies on background jobs (which can remain in the queue across
@@ -360,6 +444,24 @@ gem version.
 
 As such, **we always recommend upgrading only one major version at a time.**
 
+### Upgrading from 4.3.0
+
+Versions of Journaled prior to 5.0 would enqueue events one at a time, but 5.0
+introduces a new transaction-aware feature that will bundle up all events
+emitted within a transaction and enqueue them all in a single "batch" job
+directly before the SQL `COMMIT` statement. This reduces the database impact of
+emitting a large volume of events at once.
+
+This feature can be disabled conditionally:
+
+```ruby
+Journaled.transactional_batching_enabled = false
+```
+
+Backwards compatibility has been included for background jobs enqueued by
+version 4.0 and above, but **has been dropped for jobs emitted by versions prior
+to 4.0**. (Again, be sure to upgrade only one major version at a time.)
+
 ### Upgrading from 3.1.0
 
 Versions of Journaled prior to 4.0 relied directly on environment variables for stream names, but now stream names are configured directly.

data/Rakefile

@@ -4,37 +4,23 @@ rescue LoadError
   puts 'You must `gem install bundler` and `bundle install` to run rake tasks'
 end
 
-require 'rdoc/task'
-
-RDoc::Task.new(:rdoc) do |rdoc|
-  rdoc.rdoc_dir = 'rdoc'
-  rdoc.title    = 'Journaled'
-  rdoc.options << '--line-numbers'
-  rdoc.rdoc_files.include('README.rdoc')
-  rdoc.rdoc_files.include('lib/**/*.rb')
-end
-
-APP_RAKEFILE = File.expand_path('spec/dummy/Rakefile', __dir__)
-load 'rails/tasks/engine.rake'
-
 Bundler::GemHelper.install_tasks
 
-if %w(development test).include? Rails.env
-  require 'rspec/core'
-  require 'rspec/core/rake_task'
-  RSpec::Core::RakeTask.new
+require 'rubocop/rake_task'
+RuboCop::RakeTask.new
 
-  require 'rubocop/rake_task'
-  RuboCop::RakeTask.new
+require 'rspec/core'
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec)
 
-  task(:default).clear
+def default_task
   if ENV['APPRAISAL_INITIALIZED'] || ENV['CI']
-    task default: %i(rubocop spec)
+    %i(rubocop spec)
   else
     require 'appraisal'
     Appraisal::Task.new
-    task default: :appraisal
+    %i(appraisal)
   end
-
-  task 'db:test:prepare' => 'db:setup'
 end
+
+task(:default).clear.enhance(default_task)

data/app/jobs/journaled/delivery_job.rb

@@ -12,26 +12,11 @@ module Journaled
       raise KinesisTemporaryFailure
     end
 
-    UNSPECIFIED = Object.new
-    private_constant :UNSPECIFIED
-
-    def perform(serialized_event:, partition_key:, stream_name: UNSPECIFIED, app_name: UNSPECIFIED)
-      @serialized_event = serialized_event
-      @partition_key = partition_key
-      if app_name != UNSPECIFIED
-        @stream_name = self.class.legacy_computed_stream_name(app_name: app_name)
-      elsif stream_name != UNSPECIFIED && !stream_name.nil?
-        @stream_name = stream_name
-      else
-        raise(ArgumentError, 'missing keyword: stream_name')
-      end
-
-      journal!
-    end
+    def perform(*events, **legacy_kwargs)
+      events << legacy_kwargs if legacy_kwargs.present?
+      @kinesis_records = events.map { |e| KinesisRecord.new(**e.delete_if { |_k, v| v.nil? }) }
 
-    def self.legacy_computed_stream_name(app_name:)
-      env_var_name = [app_name&.upcase, 'JOURNALED_STREAM_NAME'].compact.join('_')
-      ENV.fetch(env_var_name)
+      journal! if Journaled.enabled?
     end
 
     def kinesis_client_config
@@ -46,18 +31,22 @@ module Journaled
 
     private
 
-    attr_reader :serialized_event, :partition_key, :stream_name
+    KinesisRecord = Struct.new(:serialized_event, :partition_key, :stream_name, keyword_init: true) do
+      def initialize(serialized_event:, partition_key:, stream_name:)
+        super(serialized_event: serialized_event, partition_key: partition_key, stream_name: stream_name)
+      end
 
-    def journal!
-      kinesis_client.put_record record if Journaled.enabled?
+      def to_h
+        { stream_name: stream_name, data: serialized_event, partition_key: partition_key }
+      end
     end
 
-    def record
-      {
-        stream_name: stream_name,
-        data: serialized_event,
-        partition_key: partition_key,
-      }
+    attr_reader :kinesis_records
+
+    def journal!
+      kinesis_records.map do |record|
+        kinesis_client.put_record(**record.to_h)
+      end
     end
 
     def kinesis_client

data/app/models/concerns/journaled/changes.rb

@@ -39,8 +39,8 @@ module Journaled::Changes
       end
     end
 
-    def update_columns(attributes, force: false)
-      unless force || self.class.journaled_attribute_names.empty?
+    def update_columns(attributes, opts = { force: false })
+      unless opts[:force] || self.class.journaled_attribute_names.empty?
         conflicting_journaled_attribute_names = self.class.journaled_attribute_names & attributes.keys.map(&:to_sym)
         raise(<<~ERROR) if conflicting_journaled_attribute_names.present?
           #update_columns aborted by Journaled::Changes due to journaled attributes:
@@ -56,7 +56,7 @@ module Journaled::Changes
   end
 
   class_methods do
-    def journal_changes_to(*attribute_names, as:, enqueue_with: {}) # rubocop:disable Naming/UncommunicativeMethodParamName
+    def journal_changes_to(*attribute_names, as:, enqueue_with: {}) # rubocop:disable Naming/MethodParameterName
       if attribute_names.empty? || attribute_names.any? { |n| !n.is_a?(Symbol) }
         raise "one or more symbol attribute_name arguments is required"
       end
@@ -69,8 +69,8 @@ module Journaled::Changes
     end
 
     if Rails::VERSION::MAJOR > 5 || (Rails::VERSION::MAJOR == 5 && Rails::VERSION::MINOR >= 2)
-      def delete(id_or_array, force: false)
-        if force || journaled_attribute_names.empty?
+      def delete(id_or_array, opts = { force: false })
+        if opts[:force] || journaled_attribute_names.empty?
           where(primary_key => id_or_array).delete_all(force: true)
         else
           raise(<<~ERROR)

data/app/models/journaled/change.rb

@@ -2,20 +2,20 @@ class Journaled::Change
   include Journaled::Event
 
   attr_reader :table_name,
-    :record_id,
-    :database_operation,
-    :logical_operation,
-    :changes,
-    :journaled_stream_name,
-    :journaled_enqueue_opts,
-    :actor
+              :record_id,
+              :database_operation,
+              :logical_operation,
+              :changes,
+              :journaled_stream_name,
+              :journaled_enqueue_opts,
+              :actor
 
   journal_attributes :table_name,
-    :record_id,
-    :database_operation,
-    :logical_operation,
-    :changes,
-    :actor
+                     :record_id,
+                     :database_operation,
+                     :logical_operation,
+                     :changes,
+                     :actor
 
   def initialize(table_name:,
                  record_id:,

data/app/models/journaled/change_writer.rb

@@ -1,5 +1,6 @@
 class Journaled::ChangeWriter
   attr_reader :model, :change_definition
+
   delegate :attribute_names, :logical_operation, to: :change_definition
 
   def initialize(model:, change_definition:)
@@ -52,8 +53,8 @@ class Journaled::ChangeWriter
   private
 
   def pluck_changed_values(change_hash, index:)
-    change_hash.each_with_object({}) do |(k, v), result|
-      result[k] = v[index]
+    change_hash.transform_values do |v|
+      v[index]
     end
   end
 

data/app/models/journaled/event.rb

@@ -63,7 +63,7 @@ module Journaled::Event
   end
 
   included do
-    cattr_accessor(:journaled_enqueue_opts, instance_writer: false) { {} }
+    class_attribute :journaled_enqueue_opts, default: {}
 
     journal_attributes :id, :event_type, :created_at
   end

data/app/models/journaled/writer.rb

@@ -26,34 +26,51 @@ class Journaled::Writer
 
   def journal!
     validate!
-    Journaled::DeliveryJob
-      .set(journaled_enqueue_opts.reverse_merge(priority: Journaled.job_priority))
-      .perform_later(delivery_perform_args)
+
+    ActiveSupport::Notifications.instrument('journaled.event.stage', event: journaled_event, **journaled_enqueue_opts) do
+      if Journaled::Connection.available?
+        Journaled::Connection.stage!(journaled_event)
+      else
+        self.class.enqueue!(journaled_event)
+      end
+    end
+  end
+
+  def self.enqueue!(*events)
+    events.group_by(&:journaled_enqueue_opts).each do |enqueue_opts, batch|
+      job_opts = enqueue_opts.reverse_merge(priority: Journaled.job_priority)
+      ActiveSupport::Notifications.instrument('journaled.batch.enqueue', batch: batch, **job_opts) do
+        Journaled::DeliveryJob.set(job_opts).perform_later(*delivery_perform_args(batch))
+
+        batch.each { |event| ActiveSupport::Notifications.instrument('journaled.event.enqueue', event: event, **job_opts) }
+      end
+    end
+  end
+
+  def self.delivery_perform_args(events)
+    events.map do |event|
+      {
+        serialized_event: event.journaled_attributes.to_json,
+        partition_key: event.journaled_partition_key,
+        stream_name: event.journaled_stream_name,
+      }
+    end
   end
 
   private
 
   attr_reader :journaled_event
+
   delegate(*EVENT_METHOD_NAMES, to: :journaled_event)
 
   def validate!
+    serialized_event = journaled_event.journaled_attributes.to_json
+
     schema_validator('base_event').validate! serialized_event
     schema_validator('tagged_event').validate! serialized_event if journaled_event.tagged?
     schema_validator(journaled_schema_name).validate! serialized_event
   end
 
-  def delivery_perform_args
-    {
-      serialized_event: serialized_event,
-      partition_key: journaled_partition_key,
-      stream_name: journaled_stream_name,
-    }
-  end
-
-  def serialized_event
-    @serialized_event ||= journaled_attributes.to_json
-  end
-
   def schema_validator(schema_name)
     Journaled::JsonSchemaModel::Validator.new(schema_name)
   end

data/lib/journaled/connection.rb

@@ -0,0 +1,48 @@
+module Journaled
+  module Connection
+    class << self
+      def available?
+        Journaled.transactional_batching_enabled && transaction_open?
+      end
+
+      def stage!(event)
+        raise TransactionSafetyError, <<~MSG unless transaction_open?
+          Transaction not available! By default, journaled event batching requires an open database transaction.
+        MSG
+
+        connection.current_transaction._journaled_staged_events << event
+      end
+
+      private
+
+      def transaction_open?
+        connection.transaction_open?
+      end
+
+      def connection
+        if Journaled.queue_adapter.in? %w(delayed delayed_job)
+          Delayed::Job.connection
+        elsif Journaled.queue_adapter == 'good_job'
+          GoodJob::BaseRecord.connection
+        elsif Journaled.queue_adapter == 'que'
+          Que::ActiveRecord::Model.connection
+        elsif Journaled.queue_adapter == 'test' && Rails.env.test?
+          ActiveRecord::Base.connection
+        else
+          raise "Unsupported adapter: #{Journaled.queue_adapter}"
+        end
+      end
+    end
+
+    module TestOnlyBehaviors
+      def transaction_open?
+        # Transactional fixtures wrap all tests in an outer, non-joinable transaction:
+        super && (connection.open_transactions > 1 || connection.current_transaction.joinable?)
+      end
+    end
+
+    class << self
+      prepend TestOnlyBehaviors if Rails.env.test?
+    end
+  end
+end

data/lib/journaled/engine.rb

@@ -4,6 +4,11 @@ module Journaled
       ActiveSupport.on_load(:active_job) do
         Journaled.detect_queue_adapter! unless Journaled.development_or_test?
       end
+
+      ActiveSupport.on_load(:active_record) do
+        require 'journaled/transaction_ext'
+        ActiveRecord::ConnectionAdapters::Transaction.prepend Journaled::TransactionExt
+      end
     end
   end
 end

data/lib/journaled/errors.rb

@@ -0,0 +1,3 @@
+module Journaled
+  class TransactionSafetyError < StandardError; end
+end

data/lib/journaled/relation_change_protection.rb

@@ -1,14 +1,15 @@
 module Journaled::RelationChangeProtection
-  def update_all(updates, force: false) # rubocop:disable Metrics/AbcSize, Metrics/PerceivedComplexity
-    unless force || !@klass.respond_to?(:journaled_attribute_names) || @klass.journaled_attribute_names.empty?
-      conflicting_journaled_attribute_names = if updates.is_a?(Hash)
-                                                @klass.journaled_attribute_names & updates.keys.map(&:to_sym)
-                                              elsif updates.is_a?(String)
-                                                @klass.journaled_attribute_names.select do |a|
-                                                  updates.match?(/\b(?<!')#{a}(?!')\b/)
-                                                end
-                                              else
-                                                raise "unsupported type '#{updates.class}' for 'updates'"
+  def update_all(updates, opts = { force: false }) # rubocop:disable Metrics/AbcSize
+    unless opts[:force] || !@klass.respond_to?(:journaled_attribute_names) || @klass.journaled_attribute_names.empty?
+      conflicting_journaled_attribute_names = case updates
+                                                when Hash
+                                                  @klass.journaled_attribute_names & updates.keys.map(&:to_sym)
+                                                when String
+                                                  @klass.journaled_attribute_names.select do |a|
+                                                    updates.match?(/\b(?<!')#{a}(?!')\b/)
+                                                  end
+                                                else
+                                                  raise "unsupported type '#{updates.class}' for 'updates'"
                                               end
       raise(<<~ERROR) if conflicting_journaled_attribute_names.present?
         .update_all aborted by Journaled::Changes due to journaled attributes: