journaled 5.0.0 → 5.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ac63d8988a5cbbd63340c019b82642a039a9dfee81f46d83b15e5a71d83e9cb7
-  data.tar.gz: ae809e4b05901d6eb73eb0196119999be2b0fd414c3af3c4bb30d280216098cf
+  metadata.gz: 4d27e8931e64963620ecd1fd44c4a46c244470a6715dc19c56a7a1a29aaa97b6
+  data.tar.gz: f95df32775abd3f65cb8d526c60fae948fc69425957a378e4007765fdade14c0
 SHA512:
-  metadata.gz: 72bcbe0ae43717280eb8a0b3383672cc87762e4c2b9b082778f1beac249004ef90a41e32e6fbabd693377a4d0af6cf6fc78df8a2b8ff44067276dd502cece63a
-  data.tar.gz: b21401ba2cf6155a25f50f1c58c634e3036754ec69b8d7f287c86408f69819aed5cfac1a43d55636a2be461545096b1510318a6ce665bf47595f30c8b90bceb9
+  metadata.gz: 1cd4a144b2cc6dbb398b082dfe97b93829ab5d0e2ad7c22fb807ff63be9d9f5e1b24278db14eb947ce247c92f8fd09e135010ce9bc546e6e34589f98fdccaeb9
+  data.tar.gz: 10da7fde07cfe2e7d654978a99c0734976f7199efd1d1d5e6aa11684d30d537d625acf238e0f636f8dec75b298920a8bf3f64a8c41105ddab40f624b03f1e8d6

data/README.md

@@ -122,25 +122,19 @@ Both model-level directives accept additional options to be passed into ActiveJo
 # For change journaling:
 journal_changes_to :email, as: :identity_change, enqueue_with: { priority: 10 }
 
+# For audit logging:
+has_audit_log enqueue_with: { priority: 30 }
+
 # Or for custom journaling:
 journal_attributes :email, enqueue_with: { priority: 20, queue: 'journaled' }
 ```
 
-### Change Journaling
-
-Out of the box, `Journaled` provides an event type and ActiveRecord
-mix-in for durably journaling changes to your model, implemented via
-ActiveRecord hooks. Use it like so:
-
-```ruby
-class User < ApplicationRecord
-  include Journaled::Changes
-
-  journal_changes_to :email, :first_name, :last_name, as: :identity_change
-end
-```
+### Attribution
 
-Add the following to your controller base class for attribution:
+Before using `Journaled::Changes` or `Journaled::AuditLog`, you will want to
+set up automatic "actor" attribution (i.e. tracking the current user session).
+To enable this feature, add the following to your controller base class for
+attribution:
 
 ```ruby
 class ApplicationController < ActionController::Base
@@ -153,6 +147,20 @@ end
 Your authenticated entity must respond to `#to_global_id`, which ActiveRecords do by default.
 This feature relies on `ActiveSupport::CurrentAttributes` under the hood.
 
+### Change Journaling with `Journaled::Changes`
+
+Out of the box, `Journaled` provides an event type and ActiveRecord
+mix-in for durably journaling changes to your model, implemented via
+ActiveRecord hooks. Use it like so:
+
+```ruby
+class User < ApplicationRecord
+  include Journaled::Changes
+
+  journal_changes_to :email, :first_name, :last_name, as: :identity_change
+end
+```
+
 Every time any of the specified attributes is modified, or a `User`
 record is created or destroyed, an event will be sent to Kinesis with the following attributes:
 
@@ -179,6 +187,213 @@ journaling. Note that the less-frequently-used methods `toggle`,
 `increment*`, `decrement*`, and `update_counters` are not intercepted at
 this time.
 
+
+### Audit Logging with `Journaled::AuditLog`
+
+Journaled includes a feature for producing audit logs of changes to your model.
+Unlike `Journaled::Changes`, which will emit individual sets of changes as
+"logical" events, `Journaled::AuditLog` will log all changes in their entirety,
+unless otherwise told to ignore changes to specific columns.
+
+This behavior is similar to
+[papertrail](https://github.com/paper-trail-gem/paper_trail),
+[audited](https://github.com/collectiveidea/audited), and
+[logidze](https://github.com/palkan/logidze), except instead of storing
+changes/versions locally (in your application's database), it emits them to
+Kinesis (as Journaled events).
+
+#### Audit Log Configuration
+
+To enable audit logging for a given record, use the `has_audit_log` directive:
+
+```ruby
+class MyModel < ApplicationRecord
+  has_audit_log
+
+  # This class will now be audited,
+  # but will ignore changes to `created_at` and `updated_at`.
+end
+```
+
+To ignore changes to additional columns, use the `ignore` option:
+
+```ruby
+class MyModel < ApplicationRecord
+  has_audit_log ignore: :last_synced_at
+
+  # This class will be audited,
+  # and will ignore changes to `created_at`, `updated_at`, and `last_synced_at`.
+end
+```
+
+By default, changes to `updated_at` and `created_at` will be ignored (since
+these generally change on every update), but this behavior can be reconfigured:
+
+```ruby
+# change the defaults:
+Journaled::AuditLog.default_ignored_columns = %i(createdAt updatedAt)
+
+# or append new defaults:
+Journaled::AuditLog.default_ignored_columns += %i(modified_at)
+
+# or disable defaults entirely:
+Journaled::AuditLog.default_ignored_columns = []
+```
+
+Subclasses will inherit audit log configs:
+
+```ruby
+class MyModel < ApplicationRecord
+  has_audit_log ignore: :last_synced_at
+end
+
+class MySubclass < MyModel
+  # this class will be audited,
+  # and will ignore `created_at`, `updated_at`, and `last_synced_at`.
+end
+```
+
+To disable audit logs on subclasses, use `skip_audit_log`:
+
+```ruby
+class MySubclass < MyModel
+  skip_audit_log
+end
+```
+
+Subclasses may specify additional columns to ignore (which will be merged into
+the inherited list):
+
+```ruby
+class MySubclass < MyModel
+  has_audit_log ignore: :another_field
+
+  # this class will ignore `another_field`, IN ADDITION TO `created_at`, `updated_at`,
+  # and any other fields specified by the parent class.
+end
+```
+
+To temporarily disable audit logging globally, use the `without_audit_logging` directive:
+
+```ruby
+Journaled::AuditLog.without_audit_logging do
+  # Any operation in here will skip audit logging
+end
+```
+
+#### Audit Log Events
+
+Whenever an audited record is created, updated, or destroyed, a
+`journaled_audit_log` event is emitted. For example, calling
+`user.update!(name: 'Bart')` would result in an event that looks something like
+this:
+
+```json
+{
+  "id": "bc7cb6a6-88cf-4849-a4f0-a31b0b199c47",
+  "event_type": "journaled_audit_log",
+  "created_at": "2022-01-28T11:06:54.928-05:00",
+  "class_name": "User",
+  "table_name": "users",
+  "record_id": "123",
+  "database_operation": "update",
+  "changes": { "name": ["Homer", "Bart"] },
+  "snapshot": null,
+  "actor": "gid://app_name/AdminUser/456",
+  "tags": {}
+}
+```
+
+The field breakdown is as follows:
+
+- `id`: a randomly-generated ID for the event itself
+- `event_type`: the type of event (always `journaled_audit_log`)
+- `created_at`: the time that the action occurred (should match `updated_at` on
+  the ActiveRecord)
+- `class_name`: the name of the ActiveRecord class
+- `table_name`: the underlying table that the class interfaces with
+- `record_id`: the primary key of the ActiveRecord
+- `database_operation`: the type of operation (`insert`, `update`, or `delete`)
+- `changes`: the changes to the record, in the form of `"field_name":
+  ["from_value", "to_value"]`
+- `snapshot`: an (optional) snapshot of all of the record's columns and their
+  values (see below).
+- `actor`: the current `Journaled.actor`
+- `tags`: the current `Journaled.tags`
+
+#### Snapshots
+
+When records are created, updated, and deleted, the `changes` field is populated
+with only the columns that changed. While this keeps event payload size down, it
+may make it harder to reconstruct the state of the record at a given point in
+time.
+
+This is where the `snapshot` field comes in! To produce a full snapshot of a
+record as part of an update, set use the virtual `_log_snapshot` attribute, like
+so:
+
+```ruby
+my_user.update!(name: 'Bart', _log_snapshot: true)
+```
+
+Or to produce snapshots for all records that change for a given operation,
+wrap it a `with_snapshots` block, like so:
+
+```ruby
+Journaled::AuditLog.with_snapshots do
+  ComplicatedOperation.run!
+end
+```
+
+Events with snapshots will continue to populate the `changes` field, but will
+additionally contain a snapshot with the full state of the user:
+
+```json
+{
+  "...": "...",
+  "changes": { "name": ["Homer", "Bart"] },
+  "snapshot": { "name": "Bart", "email": "simpson@example.com", "favorite_food": "pizza" },
+  "...": "..."
+}
+```
+
+#### Handling Sensitive Data
+
+Both `changes` and `snapshot` will filter out sensitive fields, as defined by
+your `Rails.application.config.filter_parameters` list:
+
+```json
+{
+  "...": "...",
+  "changes": { "ssn": ["[FILTERED]", "[FILTERED]"] },
+  "snapshot": { "ssn": "[FILTERED]" },
+  "...": "..."
+}
+```
+
+They will also filter out any fields whose name ends in `_crypt` or `_hmac`, as
+well as fields that rely on Active Record Encryption / `encrypts` ([introduced
+in Rails 7](https://edgeguides.rubyonrails.org/active_record_encryption.html)).
+
+This is done to avoid emitting values to locations where it is difficult or
+impossible to rotate encryption keys (or otherwise scrub values after the
+fact), and currently there is no built-in configuration to bypass this
+behavior. If you need to track changes to sensitive/encrypted fields, it is
+recommended that you store the values in a local history table (still
+encrypted, of course!).
+
+#### Caveats
+
+Because Journaled events are not guaranteed to arrive in order, events emitted
+by `Journaled::AuditLog` must be sorted by their `created_at` value, which
+should correspond roughly to the time that the SQL statement was issued.
+**There is currently no other means of globally ordering audit log events**,
+making them susceptible to clock drift and race conditions.
+
+These issues may be mitigated on a per-model basis via
+`ActiveRecord::Locking::Optimistic` (and its auto-incrementing `lock_version`
+column), and/or by careful use of other locking mechanisms.
+
 ### Custom Journaling
 
 For every custom implementation of journaling in your application, define the JSON schema for the attributes in your event.
@@ -338,7 +553,7 @@ Returns one of the following in order of preference:
 * a string of the form `gid://[app_name]` as a fallback
 
 In order for this to be most useful, you must configure your controller
-as described in [Change Journaling](#change-journaling) above.
+as described in [Attribution](#attribution) above.
 
 ### Testing
 

data/app/models/concerns/journaled/changes.rb

@@ -56,7 +56,7 @@ module Journaled::Changes
   end
 
   class_methods do
-    def journal_changes_to(*attribute_names, as:, enqueue_with: {}) # rubocop:disable Naming/MethodParameterName
+    def journal_changes_to(*attribute_names, as:, enqueue_with: {})
       if attribute_names.empty? || attribute_names.any? { |n| !n.is_a?(Symbol) }
         raise "one or more symbol attribute_name arguments is required"
       end

data/app/models/journaled/audit_log/event.rb

@@ -0,0 +1,91 @@
+# FIXME: This cannot be included in lib/ because Journaled::Event is autoloaded via app/models
+#        Autoloading Journaled::Event isn't strictly necessary, and for compatibility it would
+#        make sense to move it to lib/.
+module Journaled
+  module AuditLog
+    Event = Struct.new(:record, :database_operation, :unfiltered_changes, :enqueue_opts) do
+      include Journaled::Event
+
+      journal_attributes :class_name, :table_name, :record_id,
+                         :database_operation, :changes, :snapshot, :actor, tagged: true
+
+      def journaled_stream_name
+        AuditLog.default_stream_name || super
+      end
+
+      def journaled_enqueue_opts
+        record.class.audit_log_config.enqueue_opts
+      end
+
+      def created_at
+        case database_operation
+          when 'insert'
+            record_created_at
+          when 'update'
+            record_updated_at
+          when 'delete'
+            Time.zone.now
+          else
+            raise "Unhandled database operation type: #{database_operation}"
+        end
+      end
+
+      def record_created_at
+        record.try(:created_at) || Time.zone.now
+      end
+
+      def record_updated_at
+        record.try(:updated_at) || Time.zone.now
+      end
+
+      def class_name
+        record.class.name
+      end
+
+      def table_name
+        record.class.table_name
+      end
+
+      def record_id
+        record.id
+      end
+
+      def changes
+        filtered_changes = unfiltered_changes.deep_dup.deep_symbolize_keys
+        filtered_changes.each do |key, value|
+          filtered_changes[key] = value.map { |val| '[FILTERED]' if val } if filter_key?(key)
+        end
+      end
+
+      def snapshot
+        filtered_attributes if record._log_snapshot || AuditLog.snapshots_enabled
+      end
+
+      def actor
+        Journaled.actor_uri
+      end
+
+      private
+
+      def filter_key?(key)
+        filter_params.include?(key) || encrypted_column?(key)
+      end
+
+      def encrypted_column?(key)
+        key.to_s.end_with?('_crypt', '_hmac') ||
+          (Rails::VERSION::MAJOR >= 7 && record.encrypted_attribute?(key))
+      end
+
+      def filter_params
+        Rails.application.config.filter_parameters
+      end
+
+      def filtered_attributes
+        attrs = record.attributes.dup.symbolize_keys
+        attrs.each do |key, _value|
+          attrs[key] = '[FILTERED]' if filter_key?(key)
+        end
+      end
+    end
+  end
+end

data/journaled_schemas/journaled/audit_log/event.json

@@ -0,0 +1,31 @@
+{
+  "type": "object",
+  "title": "audit_log_event",
+  "additionalProperties": false,
+  "required": [
+    "id",
+    "event_type",
+    "created_at",
+    "class_name",
+    "table_name",
+    "record_id",
+    "database_operation",
+    "changes",
+    "snapshot",
+    "actor",
+    "tags"
+  ],
+  "properties": {
+    "id": { "type": "string" },
+    "event_type": { "type": "string" },
+    "created_at": { "type": "string" },
+    "class_name": { "type": "string" },
+    "table_name": { "type": "string" },
+    "record_id": { "type": ["string", "integer"] },
+    "database_operation": { "type": "string" },
+    "changes": { "type": "object", "additionalProperties": true },
+    "snapshot": { "type": ["object", "null"], "additionalProperties": true },
+    "actor": { "type": "string" },
+    "tags": { "type": "object", "additionalProperties": true }
+  }
+}

data/lib/journaled/audit_log.rb

@@ -0,0 +1,211 @@
+require 'active_support/core_ext/module/attribute_accessors_per_thread'
+
+module Journaled
+  module AuditLog
+    extend ActiveSupport::Concern
+
+    DEFAULT_EXCLUDED_CLASSES = %w(
+      Delayed::Job
+      PaperTrail::Version
+      ActiveStorage::Attachment
+      ActiveStorage::Blob
+      ActiveRecord::InternalMetadata
+      ActiveRecord::SchemaMigration
+    ).freeze
+
+    mattr_accessor(:default_ignored_columns) { %i(created_at updated_at) }
+    mattr_accessor(:default_stream_name) { Journaled.default_stream_name }
+    mattr_accessor(:default_enqueue_opts) { {} }
+    mattr_accessor(:excluded_classes) { DEFAULT_EXCLUDED_CLASSES.dup }
+    thread_mattr_accessor(:snapshots_enabled) { false }
+    thread_mattr_accessor(:_disabled) { false }
+    thread_mattr_accessor(:_force) { false }
+
+    class << self
+      def exclude_classes!
+        excluded_classes.each do |name|
+          if Rails::VERSION::MAJOR >= 6 && Rails.autoloaders.zeitwerk_enabled?
+            zeitwerk_exclude!(name)
+          else
+            classic_exclude!(name)
+          end
+        end
+      end
+
+      def with_snapshots
+        snapshots_enabled_was = snapshots_enabled
+        self.snapshots_enabled = true
+        yield
+      ensure
+        self.snapshots_enabled = snapshots_enabled_was
+      end
+
+      def without_audit_logging
+        disabled_was = _disabled
+        self._disabled = true
+        yield
+      ensure
+        self._disabled = disabled_was
+      end
+
+      private
+
+      def zeitwerk_exclude!(name)
+        if Object.const_defined?(name)
+          name.constantize.skip_audit_log
+        else
+          Rails.autoloaders.main.on_load(name) { |klass, _path| klass.skip_audit_log }
+        end
+      end
+
+      def classic_exclude!(name)
+        name.constantize.skip_audit_log
+      rescue NameError
+        nil
+      end
+    end
+
+    Config = Struct.new(:enabled, :ignored_columns, :enqueue_opts) do
+      def self.default
+        new(false, AuditLog.default_ignored_columns.dup, AuditLog.default_enqueue_opts.dup)
+      end
+
+      def initialize(*)
+        super
+        self.ignored_columns ||= []
+        self.enqueue_opts ||= {}
+      end
+
+      def enabled?
+        !AuditLog._disabled && self[:enabled].present?
+      end
+
+      def dup
+        super.tap do |config|
+          config.ignored_columns = ignored_columns.dup
+          config.enqueue_opts = enqueue_opts.dup
+        end
+      end
+
+      private :enabled
+    end
+
+    included do
+      prepend BlockedMethods
+      singleton_class.prepend BlockedClassMethods
+
+      class_attribute :audit_log_config, default: Config.default
+
+      attr_accessor :_log_snapshot
+
+      after_create { _emit_audit_log!('insert') }
+      after_update { _emit_audit_log!('update') if _audit_log_changes.any? }
+      after_destroy { _emit_audit_log!('delete') }
+    end
+
+    class_methods do
+      def has_audit_log(ignore: [], enqueue_with: {})
+        self.audit_log_config = audit_log_config.dup
+        audit_log_config.enabled = true
+        audit_log_config.ignored_columns |= [ignore].flatten(1)
+        audit_log_config.enqueue_opts.merge!(enqueue_with)
+      end
+
+      def skip_audit_log
+        self.audit_log_config = audit_log_config.dup
+        audit_log_config.enabled = false
+      end
+    end
+
+    module BlockedMethods
+      BLOCKED_METHODS = {
+        delete: '#destroy',
+        update_column: '#update!',
+        update_columns: '#update!',
+      }.freeze
+
+      def delete(**kwargs)
+        _journaled_audit_log_check!(:delete, **kwargs) do
+          super()
+        end
+      end
+
+      def update_column(name, value, **kwargs)
+        _journaled_audit_log_check!(:update_column, **kwargs.merge(name => value)) do
+          super(name, value)
+        end
+      end
+
+      def update_columns(args = {}, **kwargs)
+        _journaled_audit_log_check!(:update_columns, **args.merge(kwargs)) do
+          super(args.merge(kwargs).except(:_force))
+        end
+      end
+
+      def _journaled_audit_log_check!(method, **kwargs) # rubocop:disable Metrics/AbcSize
+        force_was = AuditLog._force
+        AuditLog._force = kwargs.delete(:_force) if kwargs.key?(:_force)
+        audited_columns = kwargs.keys - audit_log_config.ignored_columns
+
+        if method == :delete || audited_columns.any?
+          column_message = <<~MSG if kwargs.any?
+            You are attempting to change the following audited columns:
+              #{audited_columns.inspect}
+
+          MSG
+          raise <<~MSG if audit_log_config.enabled? && !AuditLog._force
+            #{column_message}Using `#{method}` is blocked because it skips audit logging (and other Rails callbacks)!
+            Consider using `#{BLOCKED_METHODS[method]}` instead, or pass `_force: true` as an argument.
+          MSG
+        end
+
+        yield
+      ensure
+        AuditLog._force = force_was
+      end
+    end
+
+    module BlockedClassMethods
+      BLOCKED_METHODS = {
+        delete_all: '.destroy_all',
+        insert: '.create!',
+        insert_all: '.each { create!(...) }',
+        update_all: '.find_each { update!(...) }',
+        upsert: '.create_or_find_by!',
+        upsert_all: '.each { create_or_find_by!(...) }',
+      }.freeze
+
+      BLOCKED_METHODS.each do |method, alternative|
+        define_method(method) do |*args, **kwargs, &block|
+          force_was = AuditLog._force
+          AuditLog._force = kwargs.delete(:_force) if kwargs.key?(:_force)
+
+          raise <<~MSG if audit_log_config.enabled? && !AuditLog._force
+            `#{method}` is blocked because it skips callbacks and audit logs!
+            Consider using `#{alternative}` instead, or pass `_force: true` as an argument.
+          MSG
+
+          super(*args, **kwargs, &block)
+        ensure
+          AuditLog._force = force_was
+        end
+      end
+    end
+
+    def _emit_audit_log!(database_operation)
+      if audit_log_config.enabled?
+        event = Journaled::AuditLog::Event.new(self, database_operation, _audit_log_changes, audit_log_config.enqueue_opts)
+        ActiveSupport::Notifications.instrument('journaled.audit_log.journal', event: event) do
+          event.journal!
+        end
+      end
+    end
+
+    def _audit_log_changes
+      previous_changes.except(*audit_log_config.ignored_columns)
+    end
+  end
+end
+
+ActiveSupport.on_load(:active_record) { include Journaled::AuditLog }
+Journaled::Engine.config.after_initialize { Journaled::AuditLog.exclude_classes! }

data/lib/journaled/version.rb

@@ -1,3 +1,3 @@
 module Journaled
-  VERSION = "5.0.0".freeze
+  VERSION = "5.1.1".freeze
 end

data/lib/journaled.rb

@@ -69,3 +69,5 @@ module Journaled
     Current.tags = Current.tags.merge(tags)
   end
 end
+
+require 'journaled/audit_log'

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: journaled
 version: !ruby/object:Gem::Version
-  version: 5.0.0
+  version: 5.1.1
 platform: ruby
 authors:
 - Jake Lipson
@@ -11,7 +11,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2022-08-23 00:00:00.000000000 Z
+date: 2022-09-16 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -41,6 +41,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: activesupport
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: aws-sdk-kinesis
   requirement: !ruby/object:Gem::Requirement
@@ -242,6 +256,7 @@ files:
 - app/jobs/journaled/delivery_job.rb
 - app/models/concerns/journaled/changes.rb
 - app/models/journaled/actor_uri_provider.rb
+- app/models/journaled/audit_log/event.rb
 - app/models/journaled/change.rb
 - app/models/journaled/change_definition.rb
 - app/models/journaled/change_writer.rb
@@ -252,9 +267,11 @@ files:
 - config/initializers/change_protection.rb
 - config/spring.rb
 - journaled_schemas/base_event.json
+- journaled_schemas/journaled/audit_log/event.json
 - journaled_schemas/journaled/change.json
 - journaled_schemas/tagged_event.json
 - lib/journaled.rb
+- lib/journaled/audit_log.rb
 - lib/journaled/connection.rb
 - lib/journaled/current.rb
 - lib/journaled/engine.rb