journaled 2.4.0 → 4.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a57088ffa5913d3e80f1d6d978bfe3f0a4ff090bd78f20f3fcbed1c32b84bdbc
-  data.tar.gz: 75c470723ae03dd2695300c0a8a2aaf0d05202fac8fdb5157efd248214a7eb6d
+  metadata.gz: 4cefb08a9d7ab294f9084f150f89cda921982fffdb0ec7b0a5bc702ede1d23b8
+  data.tar.gz: 21a484a10c58a3f551e95af064b5f718b9496ba7840ac6c1ac84142b859942db
 SHA512:
-  metadata.gz: ffcbd7201f572423fa304c440d18b2993e7a97050b5ac7b1be68143da64a8c7caaa430aa3ef64d8a99940380b67361e50ef55cb9ce01cecb9dd017f977e08934
-  data.tar.gz: a6c33d7c56b1fbd9558d219617a12dbfa0b704dac7b9ca3171cd2387477a3b00a6fe6dac592ed6a8cc4a4f6de2cbb806ff49d670134131a6e2eddfe5d64be0d7
+  metadata.gz: 59d07f836a83dde4e4a24a8a5f326ff5cbc7a751d92fb6df66ee22b5b31ac1aa63326675167d45bcd8c572a0e4afeb831774a468eb45097a71c6940a79ce2681
+  data.tar.gz: 807c603b1085819ed003ed8053f81b6fec59ba895c920bc24ac178b0d3188fac554a2ed9b2dcb80ceceb2a26fc5effe6af5d52819f66d041ad7959cfc30957da

data/README.md

@@ -1,16 +1,16 @@
 # Journaled
 
-A Rails engine to durably deliver schematized events to Amazon Kinesis via DelayedJob.
+A Rails engine to durably deliver schematized events to Amazon Kinesis via ActiveJob.
 
 More specifically, `journaled` is composed of three opinionated pieces:
 schema definition/validation via JSON Schema, transactional enqueueing
-via Delayed::Job (specifically `delayed_job_active_record`), and event
+via ActiveJob (specifically, via a DB-backed queue adapter), and event
 transmission via Amazon Kinesis. Our current use-cases include
 transmitting audit events for durable storage in S3 and/or analytical
 querying in Amazon Redshift.
 
 Journaled provides an at-least-once event delivery guarantee assuming
-Delayed::Job is configured not to delete jobs on failure.
+ActiveJob's queue adapter is not configured to delete jobs on failure.
 
 Note: Do not use the journaled gem to build an event sourcing solution
 as it does not guarantee total ordering of events. It's possible we'll
@@ -20,9 +20,20 @@ durable, eventually consistent record that discrete events happened.
 
 ## Installation
 
-1. [Install `delayed_job_active_record`](https://github.com/collectiveidea/delayed_job_active_record#installation)
-if you haven't already.
+1. If you haven't already,
+[configure ActiveJob](https://guides.rubyonrails.org/active_job_basics.html)
+to use one of the following queue adapters:
 
+    - `:delayed_job` (via `delayed_job_active_record`)
+    - `:que`
+    - `:good_job`
+    - `:delayed`
+
+    Ensure that your queue adapter is not configured to delete jobs on failure.
+
+    **If you launch your application in production mode and the gem detects that
+    `ActiveJob::Base.queue_adapter` is not in the above list, it will raise an exception
+    and prevent your application from performing unsafe journaling.**
 
 2. To integrate Journaled into your application, simply include the gem in your
 app's Gemfile.
@@ -40,20 +51,21 @@ app's Gemfile.
     require 'journaled/rspec'
     ```
 
-3. You will also need to define the following environment variables to allow Journaled to publish events to your AWS Kinesis event stream:
+3. You will need to set the following config in an initializer to allow Journaled to publish events to your AWS Kinesis event stream:
 
-    * `JOURNALED_STREAM_NAME`
+    ```ruby
+    Journaled.default_stream_name = "my_app_#{Rails.env}_events"
+    ```
 
-    Special case: if your `Journaled::Event` objects override the
-    `#journaled_app_name` method to a non-nil value e.g. `my_app`, you will
-    instead need to provide a corresponding
-    `[upcased_app_name]_JOURNALED_STREAM_NAME` variable for each distinct
-    value, e.g. `MY_APP_JOURNALED_STREAM_NAME`. You can provide a default value
-    for all `Journaled::Event`s in an initializer like this:
+    You may also define a `#journaled_stream_name` method on `Journaled::Event` instances:
 
     ```ruby
-    Journaled.default_app_name = 'my_app'
-    ```
+    def journaled_stream_name
+      "my_app_#{Rails.env}_alternate_events"
+    end
+    ````
+
+3. You may also need to define environment variables to allow Journaled to publish events to your AWS Kinesis event stream:
 
     You may optionally define the following ENV vars to specify AWS
     credentials outside of the locations that the AWS SDK normally looks:
@@ -71,6 +83,41 @@ app's Gemfile.
 
     The AWS principal whose credentials are in the environment will need to be allowed to assume this role.
 
+### Upgrading from 3.1.0
+
+Versions of Journaled prior to 4.0 relied directly on environment variables for stream names, but now stream names are configured directly.
+When upgrading, you can use the following configuration to maintain the previous behavior:
+
+```ruby
+Journaled.default_stream_name = ENV['JOURNALED_STREAM_NAME']
+```
+
+If you previously specified a `Journaled.default_app_name`, you would have required a more precise environment variable name (substitute `{{upcase_app_name}}`):
+
+```ruby
+Journaled.default_stream_name = ENV["{{upcase_app_name}}_JOURNALED_STREAM_NAME"]
+```
+
+And if you had defined any `journaled_app_name` methods on `Journaled::Event` instances, you can replace them with the following:
+
+```ruby
+def journaled_stream_name
+  ENV['{{upcase_app_name}}_JOURNALED_STREAM_NAME']
+end
+```
+
+When upgrading from 3.1 or below, `Journaled::DeliveryJob` will handle any jobs that remain in the queue by accepting an `app_name` argument. **This behavior will be removed in version 5.0**, so it is recommended to upgrade one major version at a time.
+
+### Upgrading from 2.5.0
+
+Versions of Journaled prior to 3.0 relied direclty on `delayed_job` and a "performable" class called `Journaled::Delivery`.
+In 3.0, this was superceded by an ActiveJob class called `Journaled::DeliveryJob`, but the `Journaled::Delivery` class was not removed until 4.0.
+
+Therefore, when upgrading from 2.5.0 or below, it is recommended to first upgrade to 3.1.0 (to allow any `Journaled::Delivery` jobs to finish working off) before upgrading to 4.0+.
+
+The upgrade to 3.1.0 will require a working ActiveJob config. ActiveJob can be configured globally by setting `ActiveJob::Base.queue_adapter`, or just for Journaled jobs by setting `Journaled::DeliveryJob.queue_adapter`.
+The `:delayed_job` queue adapter will allow you to continue relying on `delayed_job`. You may also consider switching your app(s) to [`delayed`](https://github.com/Betterment/delayed) and using the `:delayed` queue adapter.
+
 ## Usage
 
 ### Configuration
@@ -85,14 +132,16 @@ Journaling provides a number of different configuation options that can be set i
 
 #### `Journaled.job_priority` (default: 20)
 
-  This can be used to configure what `priority` the Delayed Jobs are enqueued with. This will be applied to all the Journaled::Devivery jobs that are created by this application.
+  This can be used to configure what `priority` the ActiveJobs are enqueued with. This will be applied to all the `Journaled::DeliveryJob`s that are created by this application.
   Ex: `Journaled.job_priority = 14`
 
+  _Note that job priority is only supported on Rails 6.0+. Prior Rails versions will ignore this parameter and enqueue jobs with the underlying ActiveJob adapter's default priority._
+
 #### `Journaled.http_idle_timeout` (default: 1 second)
 
   The number of seconds a persistent connection is allowed to sit idle before it should no longer be used.
 
-#### `Journaled.http_open_timeout` (default: 15 seconds)
+#### `Journaled.http_open_timeout` (default: 2 seconds)
 
   The number of seconds before the :http_handler should timeout while trying to open a new HTTP session.
 
@@ -100,9 +149,9 @@ Journaling provides a number of different configuation options that can be set i
 
   The number of seconds before the :http_handler should timeout while waiting for a HTTP response.
 
-#### DJ `enqueue` options
+#### ActiveJob `set` options
 
-Both model-level directives accept additional options to be passed into DelayedJob's `enqueue` method:
+Both model-level directives accept additional options to be passed into ActiveJob's `set` method:
 
 ```ruby
 # For change journaling:

data/Rakefile

@@ -28,7 +28,13 @@ if %w(development test).include? Rails.env
   RuboCop::RakeTask.new
 
   task(:default).clear
-  task default: %i(rubocop spec)
+  if ENV['APPRAISAL_INITIALIZED'] || ENV['CI']
+    task default: %i(rubocop spec)
+  else
+    require 'appraisal'
+    Appraisal::Task.new
+    task default: :appraisal
+  end
 
   task 'db:test:prepare' => 'db:setup'
 end

data/app/jobs/journaled/application_job.rb

@@ -0,0 +1,4 @@
+module Journaled
+  class ApplicationJob < Journaled.job_base_class_name.constantize
+  end
+end

data/app/jobs/journaled/delivery_job.rb

@@ -0,0 +1,105 @@
+module Journaled
+  class DeliveryJob < ApplicationJob
+    DEFAULT_REGION = 'us-east-1'.freeze
+
+    rescue_from(Aws::Kinesis::Errors::InternalFailure, Aws::Kinesis::Errors::ServiceUnavailable, Aws::Kinesis::Errors::Http503Error) do |e|
+      Rails.logger.error "Kinesis Error - Server Error occurred - #{e.class}"
+      raise KinesisTemporaryFailure
+    end
+
+    rescue_from(Seahorse::Client::NetworkingError) do |e|
+      Rails.logger.error "Kinesis Error - Networking Error occurred - #{e.class}"
+      raise KinesisTemporaryFailure
+    end
+
+    UNSPECIFIED = Object.new
+    private_constant :UNSPECIFIED
+
+    def perform(serialized_event:, partition_key:, stream_name: UNSPECIFIED, app_name: UNSPECIFIED)
+      @serialized_event = serialized_event
+      @partition_key = partition_key
+      if app_name != UNSPECIFIED
+        @stream_name = self.class.legacy_computed_stream_name(app_name: app_name)
+      elsif stream_name != UNSPECIFIED && !stream_name.nil?
+        @stream_name = stream_name
+      else
+        raise(ArgumentError, 'missing keyword: stream_name')
+      end
+
+      journal!
+    end
+
+    def self.legacy_computed_stream_name(app_name:)
+      env_var_name = [app_name&.upcase, 'JOURNALED_STREAM_NAME'].compact.join('_')
+      ENV.fetch(env_var_name)
+    end
+
+    def kinesis_client_config
+      {
+        region: ENV.fetch('AWS_DEFAULT_REGION', DEFAULT_REGION),
+        retry_limit: 0,
+        http_idle_timeout: Journaled.http_idle_timeout,
+        http_open_timeout: Journaled.http_open_timeout,
+        http_read_timeout: Journaled.http_read_timeout,
+      }.merge(credentials)
+    end
+
+    private
+
+    attr_reader :serialized_event, :partition_key, :stream_name
+
+    def journal!
+      kinesis_client.put_record record if Journaled.enabled?
+    end
+
+    def record
+      {
+        stream_name: stream_name,
+        data: serialized_event,
+        partition_key: partition_key,
+      }
+    end
+
+    def kinesis_client
+      Aws::Kinesis::Client.new(kinesis_client_config)
+    end
+
+    def credentials
+      if ENV.key?('JOURNALED_IAM_ROLE_ARN')
+        {
+          credentials: iam_assume_role_credentials,
+        }
+      else
+        legacy_credentials_hash_if_present
+      end
+    end
+
+    def legacy_credentials_hash_if_present
+      if ENV.key?('RUBY_AWS_ACCESS_KEY_ID')
+        {
+          access_key_id: ENV.fetch('RUBY_AWS_ACCESS_KEY_ID'),
+          secret_access_key: ENV.fetch('RUBY_AWS_SECRET_ACCESS_KEY'),
+        }
+      else
+        {}
+      end
+    end
+
+    def sts_client
+      Aws::STS::Client.new({
+        region: ENV.fetch('AWS_DEFAULT_REGION', DEFAULT_REGION),
+      }.merge(legacy_credentials_hash_if_present))
+    end
+
+    def iam_assume_role_credentials
+      @iam_assume_role_credentials ||= Aws::AssumeRoleCredentials.new(
+        client: sts_client,
+        role_arn: ENV.fetch('JOURNALED_IAM_ROLE_ARN'),
+        role_session_name: "JournaledAssumeRoleAccess",
+      )
+    end
+
+    class KinesisTemporaryFailure < NotTrulyExceptionalError
+    end
+  end
+end

data/app/models/journaled/change.rb

@@ -6,7 +6,7 @@ class Journaled::Change
     :database_operation,
     :logical_operation,
     :changes,
-    :journaled_app_name,
+    :journaled_stream_name,
     :journaled_enqueue_opts,
     :actor
 
@@ -22,7 +22,7 @@ class Journaled::Change
                  database_operation:,
                  logical_operation:,
                  changes:,
-                 journaled_app_name:,
+                 journaled_stream_name:,
                  journaled_enqueue_opts:,
                  actor:)
     @table_name = table_name
@@ -30,7 +30,7 @@ class Journaled::Change
     @database_operation = database_operation
     @logical_operation = logical_operation
     @changes = changes
-    @journaled_app_name = journaled_app_name
+    @journaled_stream_name = journaled_stream_name
     @journaled_enqueue_opts = journaled_enqueue_opts
     @actor = actor
   end

data/app/models/journaled/change_writer.rb

@@ -27,7 +27,7 @@ class Journaled::ChangeWriter
       database_operation: database_operation,
       logical_operation: logical_operation,
       changes: JSON.dump(changes),
-      journaled_app_name: journaled_app_name,
+      journaled_stream_name: journaled_stream_name,
       journaled_enqueue_opts: model.journaled_enqueue_opts,
       actor: actor_uri,
     )
@@ -57,11 +57,11 @@ class Journaled::ChangeWriter
     end
   end
 
-  def journaled_app_name
-    if model.class.respond_to?(:journaled_app_name)
-      model.class.journaled_app_name
+  def journaled_stream_name
+    if model.class.respond_to?(:journaled_stream_name)
+      model.class.journaled_stream_name
     else
-      Journaled.default_app_name
+      Journaled.default_stream_name
     end
   end
 end

data/app/models/journaled/event.rb

@@ -35,8 +35,8 @@ module Journaled::Event
     event_type
   end
 
-  def journaled_app_name
-    Journaled.default_app_name
+  def journaled_stream_name
+    Journaled.default_stream_name
   end
 
   private

data/app/models/journaled/writer.rb

@@ -3,7 +3,7 @@ class Journaled::Writer
     journaled_schema_name
     journaled_partition_key
     journaled_attributes
-    journaled_app_name
+    journaled_stream_name
     journaled_enqueue_opts
   ).freeze
 
@@ -27,7 +27,9 @@ class Journaled::Writer
   def journal!
     base_event_json_schema_validator.validate! serialized_event
     json_schema_validator.validate! serialized_event
-    Journaled.enqueue!(journaled_delivery, journaled_enqueue_opts)
+    Journaled::DeliveryJob
+      .set(journaled_enqueue_opts.reverse_merge(priority: Journaled.job_priority))
+      .perform_later(delivery_perform_args)
   end
 
   private
@@ -35,12 +37,12 @@ class Journaled::Writer
   attr_reader :journaled_event
   delegate(*EVENT_METHOD_NAMES, to: :journaled_event)
 
-  def journaled_delivery
-    @journaled_delivery ||= Journaled::Delivery.new(
+  def delivery_perform_args
+    {
       serialized_event: serialized_event,
       partition_key: journaled_partition_key,
-      app_name: journaled_app_name,
-    )
+      stream_name: journaled_stream_name,
+    }
   end
 
   def serialized_event

data/lib/journaled/engine.rb

@@ -1,4 +1,9 @@
 module Journaled
   class Engine < ::Rails::Engine
+    config.after_initialize do
+      ActiveSupport.on_load(:active_job) do
+        Journaled.detect_queue_adapter! unless Journaled.development_or_test?
+      end
+    end
   end
 end

data/lib/journaled/relation_change_protection.rb

@@ -1,5 +1,5 @@
 module Journaled::RelationChangeProtection
-  def update_all(updates, force: false) # rubocop:disable Metrics/MethodLength, Metrics/AbcSize, Metrics/PerceivedComplexity
+  def update_all(updates, force: false) # rubocop:disable Metrics/AbcSize, Metrics/PerceivedComplexity
     unless force || !@klass.respond_to?(:journaled_attribute_names) || @klass.journaled_attribute_names.empty?
       conflicting_journaled_attribute_names = if updates.is_a?(Hash)
                                                 @klass.journaled_attribute_names & updates.keys.map(&:to_sym)

data/lib/journaled/version.rb

@@ -1,3 +1,3 @@
 module Journaled
-  VERSION = "2.4.0".freeze
+  VERSION = "4.0.0".freeze
 end

data/lib/journaled.rb

@@ -1,17 +1,19 @@
 require "aws-sdk-kinesis"
-require "delayed_job"
+require "active_job"
 require "json-schema"
 require "request_store"
 
 require "journaled/engine"
-require 'journaled/enqueue'
 
 module Journaled
-  mattr_accessor :default_app_name
+  SUPPORTED_QUEUE_ADAPTERS = %w(delayed delayed_job good_job que).freeze
+
+  mattr_accessor :default_stream_name
   mattr_accessor(:job_priority) { 20 }
   mattr_accessor(:http_idle_timeout) { 5 }
-  mattr_accessor(:http_open_timeout) { 15 }
+  mattr_accessor(:http_open_timeout) { 2 }
   mattr_accessor(:http_read_timeout) { 60 }
+  mattr_accessor(:job_base_class_name) { 'ActiveJob::Base' }
 
   def development_or_test?
     %w(development test).include?(Rails.env)
@@ -33,5 +35,21 @@ module Journaled
     Journaled::ActorUriProvider.instance.actor_uri
   end
 
-  module_function :development_or_test?, :enabled?, :schema_providers, :commit_hash, :actor_uri
+  def detect_queue_adapter!
+    adapter = job_base_class_name.constantize.queue_adapter_name
+    unless SUPPORTED_QUEUE_ADAPTERS.include?(adapter)
+      raise <<~MSG
+        Journaled has detected an unsupported ActiveJob queue adapter: `:#{adapter}`
+
+        Journaled jobs must be enqueued transactionally to your primary database.
+
+        Please install the appropriate gems and set `queue_adapter` to one of the following:
+        #{SUPPORTED_QUEUE_ADAPTERS.map { |a| "- `:#{a}`" }.join("\n")}
+
+        Read more at https://github.com/Betterment/journaled
+      MSG
+    end
+  end
+
+  module_function :development_or_test?, :enabled?, :schema_providers, :commit_hash, :actor_uri, :detect_queue_adapter!
 end

data/spec/dummy/config/application.rb

@@ -1,10 +1,9 @@
 require File.expand_path('boot', __dir__)
 
 require "active_record/railtie"
+require "active_job/railtie"
 require "active_model/railtie"
 require "action_controller/railtie"
-require "action_mailer/railtie"
-require "sprockets/railtie"
 
 Bundler.require(*Rails.groups)
 require "journaled"

data/spec/dummy/config/database.yml

@@ -1,21 +1,6 @@
-default: &default
-  pool: 5
-  encoding: unicode
-
-postgresql:
-  default: &postgres_default
-    adapter: postgresql
-    url: <%= ENV['DATABASE_URL'] %>
-  test: &postgres_test
-    <<: *postgres_default
-    url: <%= ENV['DATABASE_URL'] || "postgresql://localhost:#{ENV.fetch('PGPORT', 5432)}/journaled_test" %>
-    database: journaled_test
-  development: &postgres_development
-    <<: *postgres_default
-    url: <%= ENV['DATABASE_URL'] || "postgresql://localhost:#{ENV.fetch('PGPORT', 5432)}/journaled_development" %>
-    database: journaled_development
-
 development:
-  <<: *postgres_development
+  adapter: sqlite3
+  database: ":memory:"
 test:
-  <<: *postgres_test
+  adapter: sqlite3
+  database: ":memory:"

data/spec/dummy/config/environments/development.rb

@@ -13,25 +13,12 @@ Rails.application.configure do
   config.consider_all_requests_local       = true
   config.action_controller.perform_caching = false
 
-  # Don't care if the mailer can't send.
-  config.action_mailer.raise_delivery_errors = false
-
   # Print deprecation notices to the Rails logger.
   config.active_support.deprecation = :log
 
   # Raise an error on page load if there are pending migrations.
   config.active_record.migration_error = :page_load
 
-  # Debug mode disables concatenation and preprocessing of assets.
-  # This option may cause significant delays in view rendering with a large
-  # number of complex assets.
-  config.assets.debug = true
-
-  # Adds additional error checking when serving assets at runtime.
-  # Checks for improperly declared sprockets dependencies.
-  # Raises helpful error messages.
-  config.assets.raise_runtime_errors = true
-
   # Raises error for missing translations
   # config.action_view.raise_on_missing_translations = true
 end

data/spec/dummy/config/environments/test.rb

@@ -26,14 +26,12 @@ Rails.application.configure do
   # Disable request forgery protection in test environment.
   config.action_controller.allow_forgery_protection = false
 
-  # Tell Action Mailer not to deliver emails to the real world.
-  # The :test delivery method accumulates sent emails in the
-  # ActionMailer::Base.deliveries array.
-  config.action_mailer.delivery_method = :test
-
   # Print deprecation notices to the stderr.
   config.active_support.deprecation = :stderr
 
   # Raises error for missing translations
   # config.action_view.raise_on_missing_translations = true
+
+  # Use ActiveJob test adapter
+  config.active_job.queue_adapter = :test
 end

data/spec/dummy/db/schema.rb

@@ -11,21 +11,8 @@
 # It's strongly recommended that you check this file into your version control system.
 
 ActiveRecord::Schema.define(version: 20180606205114) do
-  # These are extensions that must be enabled in order to support this database
-  enable_extension "plpgsql"
-
-  create_table "delayed_jobs", force: :cascade do |t|
-    t.integer "priority", default: 0, null: false
-    t.integer "attempts", default: 0, null: false
-    t.text "handler", null: false
-    t.text "last_error"
-    t.datetime "run_at"
-    t.datetime "locked_at"
-    t.datetime "failed_at"
-    t.string "locked_by"
-    t.string "queue"
-    t.datetime "created_at"
-    t.datetime "updated_at"
-    t.index ["priority", "run_at"], name: "delayed_jobs_priority"
+  create_table "widgets", force: :cascade do |t|
+    t.string "name"
+    t.string "other_column"
   end
 end