journaled 4.2.0 → 5.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: bfdabb8b24bdfdf23c4f77ec819f03e89a4a0239ed997368eb1f07ac0bc2cd5b
-  data.tar.gz: 97eeec1dfa22f7b683f23e322eb663e6fae4b47e0c2786690c4f44166b8cca51
+  metadata.gz: 5d0222aba969718f085949e0bf6b1fee163c70ed9512190c2b53abf75d0a120c
+  data.tar.gz: d3d2ac0e99d142aeb608f66f7d1a1ae15831d56483a8975bb7a991d3447058ac
 SHA512:
-  metadata.gz: 78c8b888f9c00a084e8d60c2f271d56a9a188f60287cb30edfbadc7a4f9fc1c3fc3210c763e757658a0040fb57f6f94f1a713fc1f95942db801e2e114dd50ec3
-  data.tar.gz: f0e56609680ecfc3e1f3f3a9d2ba801515762311307f0a208757a70a82ff7c37ba6af040f8ee5a64ebbf7a08a910579c27d65e940680b2406383b8eca022f323
+  metadata.gz: a6b4789d6314447dad04cc152b76cdf861f21d92cf27af5030f2ce500a8f2c6b791f7adfe86b975e614cba3f3af5b811022e6895feb32a831893851fabc5426d
+  data.tar.gz: 735a2305bb0599d1b6db9b355e39c1523079df60cbd975627eb5b273c155228d3740a3215341983ad15a90ff53dd6772f81a154829d13237ea8dde0eed92d34b

data/README.md

@@ -179,52 +179,6 @@ journaling. Note that the less-frequently-used methods `toggle`,
 `increment*`, `decrement*`, and `update_counters` are not intercepted at
 this time.
 
-### Tagged Events
-
-Events may be optionally marked as "tagged." This will add a `tags` field, intended for tracing and
-auditing purposes.
-
-```ruby
-class MyEvent
-  include Journaled::Event
-
-  journal_attributes :attr_1, :attr_2, tagged: true
-end
-```
-
-You may then use `Journaled.tag!` and `Journaled.tagged` inside of your
-`ApplicationController` and `ApplicationJob` classes (or anywhere else!) to tag
-all events with request and job metadata:
-
-```ruby
-class ApplicationController < ActionController::Base
-  before_action do
-    Journaled.tag!(request_id: request.request_id, current_user_id: current_user&.id)
-  end
-end
-
-class ApplicationJob < ActiveJob::Base
-  around_perform do |job, perform|
-    Journaled.tagged(job_id: job.id) { perform.call }
-  end
-end
-```
-
-This feature relies on `ActiveSupport::CurrentAttributes` under the hood, so these tags are local to
-the current thread, and will be cleared at the end of each request request/job.
-
-#### Testing
-
-If you use RSpec (and have required `journaled/rspec` in your
-`spec/rails_helper.rb`), you can regression-protect important journaling
-config with the `journal_changes_to` matcher:
-
-```ruby
-it "journals exactly these things or there will be heck to pay" do
-  expect(User).to journal_changes_to(:email, :first_name, :last_name, as: :identity_change)
-end
-```
-
 ### Custom Journaling
 
 For every custom implementation of journaling in your application, define the JSON schema for the attributes in your event.
@@ -309,6 +263,40 @@ An event like the following will be journaled to kinesis:
 }
 ```
 
+### Tagged Events
+
+Events may be optionally marked as "tagged." This will add a `tags` field, intended for tracing and
+auditing purposes.
+
+```ruby
+class MyEvent
+  include Journaled::Event
+
+  journal_attributes :attr_1, :attr_2, tagged: true
+end
+```
+
+You may then use `Journaled.tag!` and `Journaled.tagged` inside of your
+`ApplicationController` and `ApplicationJob` classes (or anywhere else!) to tag
+all events with request and job metadata:
+
+```ruby
+class ApplicationController < ActionController::Base
+  before_action do
+    Journaled.tag!(request_id: request.request_id, current_user_id: current_user&.id)
+  end
+end
+
+class ApplicationJob < ActiveJob::Base
+  around_perform do |job, perform|
+    Journaled.tagged(job_id: job.id) { perform.call }
+  end
+end
+```
+
+This feature relies on `ActiveSupport::CurrentAttributes` under the hood, so these tags are local to
+the current thread, and will be cleared at the end of each request request/job.
+
 ### Helper methods for custom events
 
 Journaled provides a couple helper methods that may be useful in your
@@ -352,6 +340,102 @@ Returns one of the following in order of preference:
 In order for this to be most useful, you must configure your controller
 as described in [Change Journaling](#change-journaling) above.
 
+### Testing
+
+If you use RSpec, you can test for journaling behaviors with the
+`journal_event(s)_including` and `journal_changes_to` matchers. First, make
+sure to require `journaled/rspec` in your spec setup (e.g.
+`spec/rails_helper.rb`):
+
+```ruby
+require 'journaled/rspec'
+```
+
+#### Checking for specific events
+
+The `journal_event_including` and `journal_events_including` matchers allow you
+to check for one or more matching event being journaled:
+
+```ruby
+expect { my_code }
+  .to journal_event_including(name: 'foo')
+expect { my_code }
+  .to journal_events_including({ name: 'foo', value: 1 }, { name: 'foo', value: 2 })
+```
+
+This will only perform matches on the specified fields (and will not match one
+way or the other against unspecified fields). These matchers will also ignore
+any extraneous events that are not positively matched (as they may be unrelated
+to behavior under test).
+
+When writing tests, pairing every positive assertion with a negative assertion
+is a good practice, and so negative matching is also supported (via both
+`.not_to` and `.to not_`):
+
+```ruby
+expect { my_code }
+  .not_to journal_events_including({ name: 'foo' }, { name: 'bar' })
+expect { my_code }
+  .to raise_error(SomeError)
+  .and not_journal_event_including(name: 'foo') # the `not_` variant can chain off of `.and`
+```
+
+Several chainable modifiers are also available:
+
+```ruby
+expect { my_code }.to journal_event_including(name: 'foo')
+  .with_schema_name('my_event_schema')
+  .with_partition_key(user.id)
+  .with_stream_name('my_stream_name')
+  .with_enqueue_opts(run_at: future_time)
+  .with_priority(999)
+```
+
+All of this can be chained together to test for multiple sets of events with
+multiple sets of options:
+
+```ruby
+expect { subject.journal! }
+  .to journal_events_including({ name: 'event1', value: 300 }, { name: 'event2', value: 200 })
+    .with_priority(10)
+  .and journal_event_including(name: 'event3', value: 100)
+    .with_priority(20)
+  .and not_journal_event_including(name: 'other_event')
+```
+
+#### Checking for `Journaled::Changes` declarations
+
+The `journal_changes_to` matcher checks against the list of attributes specified
+on the model. It does not actually test that an event is emitted within a given
+codepath, and is instead intended to guard against accidental regressions that
+may impact external consumers of these events:
+
+```ruby
+it "journals exactly these things or there will be heck to pay" do
+  expect(User).to journal_changes_to(:email, :first_name, :last_name, as: :identity_change)
+end
+```
+
+### Instrumentation
+
+When an event is enqueued, an `ActiveSupport::Notification` titled
+`journaled.event.enqueue` is emitted. Its payload will include the `:event` and
+its background job `:priority`.
+
+This can be forwarded along to your preferred monitoring solution via a Rails
+initializer:
+
+```ruby
+ActiveSupport::Notifications.subscribe('journaled.event.enqueue') do |*args|
+  payload = ActiveSupport::Notifications::Event.new(*args).payload
+  journaled_event = payload[:event]
+
+  tags = { priority: payload[:priority], event_type: journaled_event.journaled_attributes[:event_type] }
+
+  Statsd.increment('journaled.event.enqueue', tags: tags.map { |k,v| "#{k.to_s[0..64]}:#{v.to_s[0..255]}" })
+end
+```
+
 ## Upgrades
 
 Since this gem relies on background jobs (which can remain in the queue across
@@ -360,6 +444,24 @@ gem version.
 
 As such, **we always recommend upgrading only one major version at a time.**
 
+### Upgrading from 4.3.0
+
+Versions of Journaled prior to 5.0 would enqueue events one at a time, but 5.0
+introduces a new transaction-aware feature that will bundle up all events
+emitted within a transaction and enqueue them all in a single "batch" job
+directly before the SQL `COMMIT` statement. This reduces the database impact of
+emitting a large volume of events at once.
+
+This feature can be disabled conditionally:
+
+```ruby
+Journaled.transactional_batching_enabled = false
+```
+
+Backwards compatibility has been included for background jobs enqueued by
+version 4.0 and above, but **has been dropped for jobs emitted by versions prior
+to 4.0**. (Again, be sure to upgrade only one major version at a time.)
+
 ### Upgrading from 3.1.0
 
 Versions of Journaled prior to 4.0 relied directly on environment variables for stream names, but now stream names are configured directly.

data/app/jobs/journaled/delivery_job.rb

@@ -12,26 +12,11 @@ module Journaled
       raise KinesisTemporaryFailure
     end
 
-    UNSPECIFIED = Object.new
-    private_constant :UNSPECIFIED
-
-    def perform(serialized_event:, partition_key:, stream_name: UNSPECIFIED, app_name: UNSPECIFIED)
-      @serialized_event = serialized_event
-      @partition_key = partition_key
-      if app_name != UNSPECIFIED
-        @stream_name = self.class.legacy_computed_stream_name(app_name: app_name)
-      elsif stream_name != UNSPECIFIED && !stream_name.nil?
-        @stream_name = stream_name
-      else
-        raise(ArgumentError, 'missing keyword: stream_name')
-      end
-
-      journal!
-    end
+    def perform(*events, **legacy_kwargs)
+      events << legacy_kwargs if legacy_kwargs.present?
+      @kinesis_records = events.map { |e| KinesisRecord.new(**e.delete_if { |_k, v| v.nil? }) }
 
-    def self.legacy_computed_stream_name(app_name:)
-      env_var_name = [app_name&.upcase, 'JOURNALED_STREAM_NAME'].compact.join('_')
-      ENV.fetch(env_var_name)
+      journal! if Journaled.enabled?
     end
 
     def kinesis_client_config
@@ -46,18 +31,22 @@ module Journaled
 
     private
 
-    attr_reader :serialized_event, :partition_key, :stream_name
+    KinesisRecord = Struct.new(:serialized_event, :partition_key, :stream_name, keyword_init: true) do
+      def initialize(serialized_event:, partition_key:, stream_name:)
+        super(serialized_event: serialized_event, partition_key: partition_key, stream_name: stream_name)
+      end
 
-    def journal!
-      kinesis_client.put_record record if Journaled.enabled?
+      def to_h
+        { stream_name: stream_name, data: serialized_event, partition_key: partition_key }
+      end
     end
 
-    def record
-      {
-        stream_name: stream_name,
-        data: serialized_event,
-        partition_key: partition_key,
-      }
+    attr_reader :kinesis_records
+
+    def journal!
+      kinesis_records.map do |record|
+        kinesis_client.put_record(**record.to_h)
+      end
     end
 
     def kinesis_client

data/app/models/concerns/journaled/changes.rb

@@ -56,7 +56,7 @@ module Journaled::Changes
   end
 
   class_methods do
-    def journal_changes_to(*attribute_names, as:, enqueue_with: {}) # rubocop:disable Naming/MethodParameterName
+    def journal_changes_to(*attribute_names, as:, enqueue_with: {})
       if attribute_names.empty? || attribute_names.any? { |n| !n.is_a?(Symbol) }
         raise "one or more symbol attribute_name arguments is required"
       end

data/app/models/journaled/audit_log/event.rb

@@ -0,0 +1,87 @@
+# FIXME: This cannot be included in lib/ because Journaled::Event is autoloaded via app/models
+#        Autoloading Journaled::Event isn't strictly necessary, and for compatibility it would
+#        make sense to move it to lib/.
+module Journaled
+  module AuditLog
+    Event = Struct.new(:record, :database_operation, :unfiltered_changes) do
+      include Journaled::Event
+
+      journal_attributes :class_name, :table_name, :record_id,
+                         :database_operation, :changes, :snapshot, :actor, tagged: true
+
+      def journaled_stream_name
+        AuditLog.default_stream_name || super
+      end
+
+      def created_at
+        case database_operation
+          when 'insert'
+            record_created_at
+          when 'update'
+            record_updated_at
+          when 'delete'
+            Time.zone.now
+          else
+            raise "Unhandled database operation type: #{database_operation}"
+        end
+      end
+
+      def record_created_at
+        record.try(:created_at) || Time.zone.now
+      end
+
+      def record_updated_at
+        record.try(:updated_at) || Time.zone.now
+      end
+
+      def class_name
+        record.class.name
+      end
+
+      def table_name
+        record.class.table_name
+      end
+
+      def record_id
+        record.id
+      end
+
+      def changes
+        filtered_changes = unfiltered_changes.deep_dup.deep_symbolize_keys
+        filtered_changes.each do |key, value|
+          filtered_changes[key] = value.map { |val| '[FILTERED]' if val } if filter_key?(key)
+        end
+      end
+
+      def snapshot
+        filtered_attributes if record._log_snapshot || AuditLog.snapshots_enabled
+      end
+
+      def actor
+        Journaled.actor_uri
+      end
+
+      private
+
+      def filter_key?(key)
+        filter_params.include?(key) || encrypted_column?(key)
+      end
+
+      def encrypted_column?(key)
+        key.to_s.end_with?('_crypt', '_hmac') ||
+          (Rails::VERSION::MAJOR >= 7 && record.encrypted_attribute?(key))
+      end
+
+      def filter_params
+        Rails.application.config.filter_parameters
+      end
+
+      def filtered_attributes
+        attrs = record.attributes.dup.symbolize_keys
+        attrs.each do |key, _value|
+          attrs[key] = '[FILTERED]' if filter_key?(key)
+        end
+      end
+    end
+  end
+end

data/app/models/journaled/event.rb

@@ -63,7 +63,7 @@ module Journaled::Event
   end
 
   included do
-    cattr_accessor(:journaled_enqueue_opts, instance_writer: false) { {} }
+    class_attribute :journaled_enqueue_opts, default: {}
 
     journal_attributes :id, :event_type, :created_at
   end

data/app/models/journaled/writer.rb

@@ -26,9 +26,35 @@ class Journaled::Writer
 
   def journal!
     validate!
-    Journaled::DeliveryJob
-      .set(journaled_enqueue_opts.reverse_merge(priority: Journaled.job_priority))
-      .perform_later(**delivery_perform_args)
+
+    ActiveSupport::Notifications.instrument('journaled.event.stage', event: journaled_event, **journaled_enqueue_opts) do
+      if Journaled::Connection.available?
+        Journaled::Connection.stage!(journaled_event)
+      else
+        self.class.enqueue!(journaled_event)
+      end
+    end
+  end
+
+  def self.enqueue!(*events)
+    events.group_by(&:journaled_enqueue_opts).each do |enqueue_opts, batch|
+      job_opts = enqueue_opts.reverse_merge(priority: Journaled.job_priority)
+      ActiveSupport::Notifications.instrument('journaled.batch.enqueue', batch: batch, **job_opts) do
+        Journaled::DeliveryJob.set(job_opts).perform_later(*delivery_perform_args(batch))
+
+        batch.each { |event| ActiveSupport::Notifications.instrument('journaled.event.enqueue', event: event, **job_opts) }
+      end
+    end
+  end
+
+  def self.delivery_perform_args(events)
+    events.map do |event|
+      {
+        serialized_event: event.journaled_attributes.to_json,
+        partition_key: event.journaled_partition_key,
+        stream_name: event.journaled_stream_name,
+      }
+    end
   end
 
   private
@@ -38,23 +64,13 @@ class Journaled::Writer
   delegate(*EVENT_METHOD_NAMES, to: :journaled_event)
 
   def validate!
+    serialized_event = journaled_event.journaled_attributes.to_json
+
     schema_validator('base_event').validate! serialized_event
     schema_validator('tagged_event').validate! serialized_event if journaled_event.tagged?
     schema_validator(journaled_schema_name).validate! serialized_event
   end
 
-  def delivery_perform_args
-    {
-      serialized_event: serialized_event,
-      partition_key: journaled_partition_key,
-      stream_name: journaled_stream_name,
-    }
-  end
-
-  def serialized_event
-    @serialized_event ||= journaled_attributes.to_json
-  end
-
   def schema_validator(schema_name)
     Journaled::JsonSchemaModel::Validator.new(schema_name)
   end

data/journaled_schemas/journaled/audit_log/event.json

@@ -0,0 +1,31 @@
+{
+  "type": "object",
+  "title": "audit_log_event",
+  "additionalProperties": false,
+  "required": [
+    "id",
+    "event_type",
+    "created_at",
+    "class_name",
+    "table_name",
+    "record_id",
+    "database_operation",
+    "changes",
+    "snapshot",
+    "actor",
+    "tags"
+  ],
+  "properties": {
+    "id": { "type": "string" },
+    "event_type": { "type": "string" },
+    "created_at": { "type": "string" },
+    "class_name": { "type": "string" },
+    "table_name": { "type": "string" },
+    "record_id": { "type": ["string", "integer"] },
+    "database_operation": { "type": "string" },
+    "changes": { "type": "object", "additionalProperties": true },
+    "snapshot": { "type": ["object", "null"], "additionalProperties": true },
+    "actor": { "type": "string" },
+    "tags": { "type": "object", "additionalProperties": true }
+  }
+}

data/lib/journaled/audit_log.rb

@@ -0,0 +1,194 @@
+require 'active_support/core_ext/module/attribute_accessors_per_thread'
+
+module Journaled
+  module AuditLog
+    extend ActiveSupport::Concern
+
+    DEFAULT_EXCLUDED_CLASSES = %w(
+      Delayed::Job
+      PaperTrail::Version
+      ActiveStorage::Attachment
+      ActiveStorage::Blob
+      ActiveRecord::InternalMetadata
+      ActiveRecord::SchemaMigration
+    ).freeze
+
+    mattr_accessor(:default_ignored_columns) { %i(created_at updated_at) }
+    mattr_accessor(:default_stream_name) { Journaled.default_stream_name }
+    mattr_accessor(:excluded_classes) { DEFAULT_EXCLUDED_CLASSES.dup }
+    thread_mattr_accessor(:snapshots_enabled) { false }
+    thread_mattr_accessor(:_disabled) { false }
+    thread_mattr_accessor(:_force) { false }
+
+    class << self
+      def exclude_classes!
+        excluded_classes.each do |name|
+          if Rails::VERSION::MAJOR >= 6 && Rails.autoloaders.zeitwerk_enabled?
+            zeitwerk_exclude!(name)
+          else
+            classic_exclude!(name)
+          end
+        end
+      end
+
+      def with_snapshots
+        snapshots_enabled_was = snapshots_enabled
+        self.snapshots_enabled = true
+        yield
+      ensure
+        self.snapshots_enabled = snapshots_enabled_was
+      end
+
+      def without_audit_logging
+        disabled_was = _disabled
+        self._disabled = true
+        yield
+      ensure
+        self._disabled = disabled_was
+      end
+
+      private
+
+      def zeitwerk_exclude!(name)
+        if Object.const_defined?(name)
+          name.constantize.skip_audit_log
+        else
+          Rails.autoloaders.main.on_load(name) { |klass, _path| klass.skip_audit_log }
+        end
+      end
+
+      def classic_exclude!(name)
+        name.constantize.skip_audit_log
+      rescue NameError
+        nil
+      end
+    end
+
+    Config = Struct.new(:enabled, :ignored_columns) do
+      private :enabled
+      def enabled?
+        !AuditLog._disabled && self[:enabled].present?
+      end
+    end
+
+    included do
+      prepend BlockedMethods
+      singleton_class.prepend BlockedClassMethods
+
+      class_attribute :audit_log_config, default: Config.new(false, AuditLog.default_ignored_columns)
+      attr_accessor :_log_snapshot
+
+      after_create { _emit_audit_log!('insert') }
+      after_update { _emit_audit_log!('update') if _audit_log_changes.any? }
+      after_destroy { _emit_audit_log!('delete') }
+    end
+
+    class_methods do
+      def has_audit_log(ignore: [])
+        ignored_columns = _audit_log_inherited_ignored_columns + [ignore].flatten(1)
+        self.audit_log_config = Config.new(true, ignored_columns.uniq)
+      end
+
+      def skip_audit_log
+        self.audit_log_config = Config.new(false, _audit_log_inherited_ignored_columns.uniq)
+      end
+
+      private
+
+      def _audit_log_inherited_ignored_columns
+        (superclass.try(:audit_log_config)&.ignored_columns || []) + audit_log_config.ignored_columns
+      end
+    end
+
+    module BlockedMethods
+      BLOCKED_METHODS = {
+        delete: '#destroy',
+        update_column: '#update!',
+        update_columns: '#update!',
+      }.freeze
+
+      def delete(**kwargs)
+        _journaled_audit_log_check!(:delete, **kwargs) do
+          super()
+        end
+      end
+
+      def update_column(name, value, **kwargs)
+        _journaled_audit_log_check!(:update_column, **kwargs.merge(name => value)) do
+          super(name, value)
+        end
+      end
+
+      def update_columns(args = {}, **kwargs)
+        _journaled_audit_log_check!(:update_columns, **args.merge(kwargs)) do
+          super(args.merge(kwargs).except(:_force))
+        end
+      end
+
+      def _journaled_audit_log_check!(method, **kwargs) # rubocop:disable Metrics/AbcSize
+        force_was = AuditLog._force
+        AuditLog._force = kwargs.delete(:_force) if kwargs.key?(:_force)
+        audited_columns = kwargs.keys - audit_log_config.ignored_columns
+
+        if method == :delete || audited_columns.any?
+          column_message = <<~MSG if kwargs.any?
+            You are attempting to change the following audited columns:
+              #{audited_columns.inspect}
+
+          MSG
+          raise <<~MSG if audit_log_config.enabled? && !AuditLog._force
+            #{column_message}Using `#{method}` is blocked because it skips audit logging (and other Rails callbacks)!
+            Consider using `#{BLOCKED_METHODS[method]}` instead, or pass `_force: true` as an argument.
+          MSG
+        end
+
+        yield
+      ensure
+        AuditLog._force = force_was
+      end
+    end
+
+    module BlockedClassMethods
+      BLOCKED_METHODS = {
+        delete_all: '.destroy_all',
+        insert: '.create!',
+        insert_all: '.each { create!(...) }',
+        update_all: '.find_each { update!(...) }',
+        upsert: '.create_or_find_by!',
+        upsert_all: '.each { create_or_find_by!(...) }',
+      }.freeze
+
+      BLOCKED_METHODS.each do |method, alternative|
+        define_method(method) do |*args, **kwargs, &block|
+          force_was = AuditLog._force
+          AuditLog._force = kwargs.delete(:_force) if kwargs.key?(:_force)
+
+          raise <<~MSG if audit_log_config.enabled? && !AuditLog._force
+            `#{method}` is blocked because it skips callbacks and audit logs!
+            Consider using `#{alternative}` instead, or pass `_force: true` as an argument.
+          MSG
+
+          super(*args, **kwargs, &block)
+        ensure
+          AuditLog._force = force_was
+        end
+      end
+    end
+
+    def _emit_audit_log!(database_operation)
+      if audit_log_config.enabled?
+        event = Journaled::AuditLog::Event.new(self, database_operation, _audit_log_changes)
+        ActiveSupport::Notifications.instrument('journaled.audit_log.journal', event: event) do
+          event.journal!
+        end
+      end
+    end
+
+    def _audit_log_changes
+      previous_changes.except(*audit_log_config.ignored_columns)
+    end
+  end
+end
+
+ActiveSupport.on_load(:active_record) { include Journaled::AuditLog }
+Journaled::Engine.config.after_initialize { Journaled::AuditLog.exclude_classes! }