munster 0.3.1 → 0.4.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 70c4c7bb98c04fb6e5d429ac03cd9dd8734e8a8d55d8934a1eb41d1a4e359aec
-  data.tar.gz: 4d2ec2780fe9900c61512b2cdeb7c70351ec8a80d69bf4a23734e600b69d3a83
+  metadata.gz: 7d3f84bd06c033ef585c3282701915793b981142351824cc7bba2f41c5bcde4b
+  data.tar.gz: 9e614a61681b3bed5667905ae378a357fcfa96d85c79b7d85c32473c2b763c4f
 SHA512:
-  metadata.gz: bb7319560e29abf5011b73e595cba880100402ff03713d9b4d051c8d06b27ff17591703f077f5c5366d2824cf8d8ec4fab93bc72fb74b8c4517215a4aea5e5e2
-  data.tar.gz: 0712b43343c58e637d90c4171de3b9e4c5cd8c96815d3afa84ccdf7718ccf3115cc7a2ec2ee552878c88ec307c021b9e9d905dff0e345f29aca18de55683b92f
+  metadata.gz: 616dc8e6585239309106444bb65de90d572e946e08479044459e5f8deb764d0d308d22da895152e3d7250345a81640f627ea72485daf17d6d51b03299d29f5c3
+  data.tar.gz: 369a9b8ce33c83e876836f652b57fc618a82e3028818531f6bc98d8732fad53cc7a926d6faf2ad8711c5e08ac3e0c9ec23de06a95b597087f86091dfb78e6240

data/CHANGELOG.md

@@ -3,31 +3,37 @@ All notable changes to this project will be documented in this file.
 
 This format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## 0.3.1
+## 0.4.1
+
+- Webhook processor now requires `active_job/railtie`, instead of `active_job`. It requires GlobalID to work and that get's required only with railtie.
+- Adding a state transition from `error` to `received`. Proccessing job will be enqueued automatic after this transition was executed.
+
+## 0.4.0
 
-### Changed
+- Limit the size of the request body, since otherwise there can be a large attack vector where random senders can spam the database with data and cause a denial of service. With background validation, this is one of the few cases where we want to reject the payload without persisting it.
+- Manage the `state` of the `ReceivedWebhook` from the background job itself. This frees up the handler to actually do the work associated with processing only. The job will manage the rest.
+- Use `valid?` in the background job instead of the controller. Most common configuration issue is an incorrectly specified signing secret, or an incorrectly implemented input validation. When these happen, it is better to allow the webhook to be reprocessed
+- Use instance methods in handlers instead of class methods, as they are shorter to define. Assume a handler module supports `.new` - with a module using singleton methods it may return `self` from `new`.
+- In the config, allow the handlers specified as strings. Module resolution in Rails happens after the config gets loaded, because the config may alter the Zeitwerk load paths. To allow the config to get loaded and to allow handlers to be autoloaded using Zeitwerk, the handler modules have to be resolved lazily. This also permits the handlers to be reloadable, like any module under Rails' autoloading control.
+- Simplify the Rails app used in tests to be small and keep it in a single file
+- If a handler is willing to expose errors to the caller, let Rails rescue the error and display an error page or do whatever else is configured for Rails globally.
+- Store request headers with the received webhook to allow for async validation. Run `bin/rails g munster:install` to add the required migration.
+
+## 0.3.1
 
 - BaseHandler#expose_errors_to_sender? default to true now.
 
 ## 0.3.0
 
-### Changed
 - state_machine_enum library was moved in it's own library/gem.
-
-### Fixed
 - Provide handled: true attribute for Rails.error.report method, because it is required in Rails 7.0.
 
 ## 0.2.0
 
-### Changed
-
 - Handler methods are now defined as instance methods for simplicity.
 - Define service_id in initializer with active_handlers, instead of handler class.
 - Use ruby 3.0 as a base for standard/rubocop, format all code according to it.
-
-### Added
-
-- Introduce Rails common error reporter ( https://guides.rubyonrails.org/error_reporting.html )
+- Use Rails common error reporter ( https://guides.rubyonrails.org/error_reporting.html )
 
 ## 0.1.0
 

data/README.md

@@ -2,9 +2,11 @@
 
 Munster is a Rails engine that provides a webhook endpoint for receiving and processing webhooks from various services. Engine stores received webhook first and later processes webhook in a separete async process.
 
-Source code is extracted from https://cheddar.me/ main service to be used in internal microservices. Code here could be a subject to change while we flesh out details.
-
-Due to that, support for this gem is limited.
+> [!CAUTION]
+> At the moment Munster is only used internally at Cheddar. Any support to external parties is on best-effort
+> basis. While we are happy to see issues and pull requests, we can't guarantee that those will be addressed
+> quickly. The engine does receive rapid updates which may break your application if you come to depend on
+> the library. That is to be expected.
 
 ## Installation
 
@@ -20,11 +22,25 @@ If bundler is not being used to manage dependencies, install the gem by executin
 
 Generate migrations and initializer file.
 
-`munster:install`
+`bin/rails g munster:install`
 
 Mount munster engine in your routes.
 
-`mount Munster::Engine, at: "/webhooks"`
+```ruby
+mount Munster::Engine, at: "/webhooks"
+```
+
+Define a class for your first handler (let's call it `ExampleHandler`) and inherit it from `Munster::BaseHandler`. Place it somewhere where Rails autoloading can find it, and add it to your `munster.rb` config file:
+
+```ruby
+config.active_handlers = {
+  "example" => "ExampleHandler"
+}
+```
+
+## Example handlers
+
+We provide a number of webhook handlers which demonstrate certain features of Munster. You will find them in `handler-examples`.
 
 ## Requirements
 
@@ -38,7 +54,8 @@ This project depends on two dependencies:
 This gem uses [Rails common error reporter](https://guides.rubyonrails.org/error_reporting.html) to report any possible error to services like Honeybadger, Appsignal, Sentry and etc. Most of those services already support this common interface, if not - it's not that hard to add this support on your own.
 
 It's possible to provide additional context for every error. e.g.
-```
+
+```ruby
 Munster.configure do |config|
   config.error_context = { appsignal: { namespace: "webhooks" } }
 end

data/Rakefile

@@ -1,12 +1,7 @@
 # frozen_string_literal: true
 
 require "bundler/setup"
-
-APP_RAKEFILE = File.expand_path("test/dummy/Rakefile", __dir__)
-load "rails/tasks/engine.rake"
-
-load "rails/tasks/statistics.rake"
-
+require "rake/testtask"
 require "bundler/gem_tasks"
 require "standard/rake"
 
@@ -15,4 +10,17 @@ task :format do
   `bundle exec magic_frozen_string_literal .`
 end
 
-task default: %i[standard]
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+
+  file_name = ARGV[1]
+
+  t.test_files = if file_name
+    [file_name]
+  else
+    FileList["test/**/*_test.rb"]
+  end
+end
+
+task default: [:test, :standard]

data/example/app/webhooks/webhook_test_handler.rb

@@ -2,17 +2,11 @@
 
 # This handler accepts webhooks from our integration tests. This webhook gets dispatched
 # if a banking provider test fails, indicating that the bank might be having an incident
-
 class WebhookTestHandler < Munster::BaseHandler
   def valid?(request) = true
 
   def process(webhook)
-    return unless webhook.received?
-    webhook.update!(status: "processing")
-    webhook.update!(status: "processed")
-  rescue
-    webhook.update!(status: "error")
-    raise
+    Rails.logger.info { webhook.request.params.fetch(:payment_id) }
   end
 
   def expose_errors_to_sender? = true

data/example/config/initializers/munster.rb

@@ -3,5 +3,7 @@
 require_relative "../../app/webhooks/webhook_test_handler"
 
 Munster.configure do |config|
-  config.active_handlers = [WebhookTestHandler]
+  config.active_handlers = {
+    "test-handler" => "WebhookTestHandler"
+  }
 end

data/handler-examples/customer_io_handler.rb

@@ -0,0 +1,36 @@
+# This is an example handler for Customer.io reporting webhooks. You
+# can find more documentation here https://customer.io/docs/api/webhooks/#operation/reportingWebhook
+class Webhooks::CustomerIoHandler < Munster::BaseHandler
+  def process(webhook)
+    json = JSON.parse(webhook.body, symbolize_names: true)
+    case json[:metric]
+    when "subscribed"
+      # ...
+    when "unsubscribed"
+      # ...
+    when "cio_subscription_preferences_changed"
+      # ...
+    end
+  end
+
+  def extract_event_id_from_request(action_dispatch_request)
+    action_dispatch_request.params.fetch(:event_id)
+  end
+
+  # Verify that request is actually comming from customer.io here
+  # @see https://customer.io/docs/api/webhooks/#section/Securely-Verifying-Requests
+  #
+  # - Should have "X-CIO-Signature", "X-CIO-Timestamp" headers.
+  # - Combine the version number, timestamp and body delimited by colons to form a string in the form v0:<timestamp>:<body>
+  # - Using HMAC-SHA256, hash the string using your webhook signing secret as the hash key.
+  # - Compare this value to the value of the X-CIO-Signature header sent with the request to confirm
+  def valid?(action_dispatch_request)
+    signing_key = Rails.application.secrets.customer_io_webhook_signing_key
+    xcio_signature = action_dispatch_request.headers["HTTP_X_CIO_SIGNATURE"]
+    xcio_timestamp = action_dispatch_request.headers["HTTP_X_CIO_TIMESTAMP"]
+    request_body = action_dispatch_request.body.read
+    string_to_sign = "v0:#{xcio_timestamp}:#{request_body}"
+    hmac = OpenSSL::HMAC.hexdigest("SHA256", signing_key, string_to_sign)
+    Rack::Utils.secure_compare(hmac, xcio_signature)
+  end
+end

data/handler-examples/revolut_business_v1_handler.rb

@@ -0,0 +1,29 @@
+# This is for Revolut V1 API for webhooks - https://developer.revolut.com/docs/business/webhooks-v-1-deprecated
+class RevolutBusinessV1Handler < Munster::BaseHandler
+  def valid?(_)
+    # V1 of Revolut webhooks does not support signatures
+    true
+  end
+
+  def self.process(webhook)
+    parsed_payload = JSON.parse(webhook.body)
+    topic = parsed_payload.fetch("Topic")
+    case topic
+    when "tokens" # Account access revocation payload
+      # ...
+    when "draftpayments/transfers" # Draft payment transfer notification payload
+      # ...
+    else
+      # ...
+    end
+  end
+
+  def self.extract_event_id_from_request(action_dispatch_request)
+    # Since b-tree indices generally divide from the start of the string, place the highest
+    # entropy component at the start (the EventId)
+    key_components = %w[EventId Topic Version]
+    key_components.map do |key|
+      action_dispatch_request.params.fetch(key)
+    end.join("-")
+  end
+end

data/handler-examples/revolut_business_v2_handler.rb

@@ -0,0 +1,42 @@
+# This is for Revolut V2 API for webhooks - https://developer.revolut.com/docs/business/webhooks-v-2
+class RevolutBusinessV2Handler < Munster::BaseHandler
+  def valid?(request)
+    # 1 - Validate the timestamp of the request. Prevent replay attacks.
+    # "To validate the event, make sure that the Revolut-Request-Timestamp date-time is within a 5-minute time tolerance of the current universal time (UTC)".
+    # Their examples list `timestamp = '1683650202360'` as a sample value, so their timestamp is in millis - not in seconds
+    timestamp_str_from_headers = request.headers["HTTP_REVOLUT_REQUEST_TIMESTAMP"]
+    delta_t_seconds = (timestamp_str_from_headers / 1000) - Time.now.to_i
+    return false unless delta_t_seconds.abs < (5 * 60)
+
+    # 2 - Validate the signature
+    # https://developer.revolut.com/docs/guides/manage-accounts/tutorials/work-with-webhooks/verify-the-payload-signature
+    string_to_sign = [
+      "v1",
+      timestamp_str_from_headers,
+      request.body.read
+    ].join(".")
+    computed_signature = "v1=" + OpenSSL::HMAC.hexdigest("SHA256", Rails.application.secrets.revolut_business_webhook_signing_key, string_to_sign)
+    # Note: "This means that in the period when multiple signing secrets remain valid, multiple signatures are sent."
+    # https://developer.revolut.com/docs/guides/manage-accounts/tutorials/work-with-webhooks/manage-webhooks#rotate-a-webhook-signing-secret
+    # https://developer.revolut.com/docs/guides/manage-accounts/tutorials/work-with-webhooks/about-webhooks#security
+    # An HTTP header may contain multiple values if it gets sent multiple times. But it does mean we need to test for multiple provided
+    # signatures in case of rotation.
+    provided_signatures = request.headers["HTTP_REVOLUT_SIGNATURE"].split(",")
+    # Use #select instead of `find` to compare all signatures even if only one matches - this to avoid timing leaks.
+    # Small effort but might be useful.
+    matches = provided_signatures.select do |provided_signature|
+      ActiveSupport::SecurityUtils.secure_compare(provided_signature, computed_signature)
+    end
+    matches.any?
+  end
+
+  def self.process(webhook)
+    Rails.logger.info { "Processing Revolut webhook #{webhook.body.inspect}" }
+  end
+
+  def self.extract_event_id_from_request(action_dispatch_request)
+    # The event ID is only available when you retrieve the failed webhooks, which is sad.
+    # We can divinate a synthetic ID though by taking a hash of the entire payload though.
+    Digest::SHA256.hexdigest(action_dispatch_request.body.read)
+  end
+end

data/handler-examples/starling_payments_handler.rb

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+# This handler is an example for Starling Payments API,
+# you can find the documentation here https://developer.starlingbank.com/payments/docs#account-and-address-structure-1
+class StarlingPaymentsHandler < Munster::BaseHandler
+  # This method will be used to process webhook by async worker.
+  def process(received_webhook)
+    Rails.logger.info { received_webhook.body }
+  end
+
+  # Starling supplies signatures in the form SHA512(secret + request_body)
+  def valid?(action_dispatch_request)
+    supplied_signature = action_dispatch_request.headers.fetch("X-Hook-Signature")
+    supplied_digest_bytes = Base64.strict_decode64(supplied_signature)
+    sha512 = Digest::SHA2.new(512)
+    signing_secret = Rails.credentials.starling_payments_webhook_signing_secret
+    computed_digest_bytes = sha512.digest(signing_secret.b + action_dispatch_request.body.b)
+    ActiveSupport::SecurityUtils.secure_compare(computed_digest_bytes, supplied_digest_bytes)
+  end
+
+  # Some Starling webhooks do not provide a notification UID, but for those which do we can deduplicate
+  def extract_event_id_from_request(action_dispatch_request)
+    action_dispatch_request.params.fetch("notificationUid", SecureRandom.uuid)
+  end
+end

data/lib/munster/base_handler.rb

@@ -4,40 +4,74 @@ require_relative "jobs/processing_job"
 
 module Munster
   class BaseHandler
-    # Gets called from the background job
-    def self.process(...)
-      new.process(...)
+    # `handle` accepts the ActionDispatch HTTP request and saves the webhook for later processing. It then
+    # enqueues an ActiveJob which will perform the processing using `process`.
+    #
+    # @param action_dispatch_request[ActionDispatch::Request] the request from the controller
+    # @return [void]
+    def handle(action_dispatch_request)
+      handler_module_name = is_a?(Munster::BaseHandler) ? self.class.name : to_s
+      handler_event_id = extract_event_id_from_request(action_dispatch_request)
+
+      webhook = Munster::ReceivedWebhook.new(request: action_dispatch_request, handler_event_id: handler_event_id, handler_module_name: handler_module_name)
+      webhook.save!
+
+      enqueue(webhook)
+    rescue ActiveRecord::RecordNotUnique # Webhook deduplicated
+      Rails.logger.info { "#{inspect} Webhook #{handler_event_id} is a duplicate delivery and will not be stored." }
     end
 
-    # Reimplement this method, it's being used in WebhooksController to store incoming webhook.
-    # Also que for processing in the end.
+    # Enqueues the processing job to process webhook asynchronously. The job class could be configured.
+    #
+    # @param webhook [Munster::ReceivedWebhook]
     # @return [void]
-    def handle(action_dispatch_request)
-      binary_body_str = action_dispatch_request.body.read.force_encoding(Encoding::BINARY)
-      attrs = {
-        body: binary_body_str,
-        handler_module_name: self.class.name,
-        handler_event_id: extract_event_id_from_request(action_dispatch_request)
-      }
-      webhook = Munster::ReceivedWebhook.create!(**attrs)
-
-      Munster.configuration.processing_job_class.perform_later(webhook)
-    rescue ActiveRecord::RecordNotUnique # Deduplicated
-      nil
+    def enqueue(webhook)
+      # The configured job class can be a class name or a module, to support lazy loading
+      job_class_or_module_name = Munster.configuration.processing_job_class
+      job_class = if job_class_or_module_name.respond_to?(:perform_later)
+        job_class_or_module_name
+      else
+        job_class_or_module_name.constantize
+      end
+
+      job_class.perform_later(webhook)
     end
 
-    # This method will be used to process webhook by async worker.
+    # This is the heart of your webhook processing. Override this method and define your processing inside of it.
+    # The `received_webhook` will provide access to the `ReceivedWebhook` model, which contains the received
+    # body of the webhook request, but also the full (as-full-as-possible) clone of the original ActionDispatch::Request
+    # that you can use.
+    #
+    # @param received_webhook[Munster::ReceivedWebhook]
+    # @return [void]
     def process(received_webhook)
     end
 
-    # This method verifies that request actually comes from provider:
-    # signature validation, HTTP authentication, IP whitelisting and the like
+    # This method verifies that request is not malformed and actually comes from the webhook sender:
+    # signature validation, HTTP authentication, IP whitelisting and the like. There is a difference depending
+    # on whether you validate sync (in the receiving controller) or async (in the processing job):
+    # Validation is async - it takes place in the background job that gets enqueued to process the webhook.
+    # The `action_dispatch_request` will be reconstructed from the `ReceivedWebhook` data. Background validation
+    # is used because the most common misconfiguration that may occur is usually forgetting or misidentifying the
+    # signature for signed webhooks. If such a misconfiguration has taken place, the background validation
+    # (instead of rejecting the webhook at input) permits you to still process the webhook once the secrets
+    # have been configured correctly.
+    #
+    # If this method returns `false`, the webhook will be marked as `failed_validation` in the database. If this
+    # method returns `true`, the `process` method of the handler is going to be called.
+    #
+    # @see Munster::ReceivedWebhook#request
+    # @param action_dispatch_request[ActionDispatch::Request] the reconstructed request from the controller
+    # @return [Boolean]
     def valid?(action_dispatch_request)
       true
     end
 
-    # Default implementation just generates UUID, but if the webhook sender sends us
-    # an event ID we use it for deduplication.
+    # Default implementation just generates a random UUID, but if the webhook sender sends us
+    # an event ID we use it for deduplication. A duplicate webhook is not going to be
+    # stored in the database if it is already present there.
+    #
+    # @return [String]
     def extract_event_id_from_request(action_dispatch_request)
       SecureRandom.uuid
     end
@@ -47,6 +81,8 @@ module Munster
     # data and do not disable your endpoint forcibly. We allow this to be configured
     # on a per-handler basis - a better webhooks sender will be able to make out
     # some sense of the errors.
+    #
+    # @return [Boolean]
     def expose_errors_to_sender?
       true
     end
@@ -55,6 +91,8 @@ module Munster
     # to deactivate a particular handler via feature flags for example, or use other
     # logic to determine whether the handler may be used to create new received webhooks
     # in the system. This is primarily needed for load shedding.
+    #
+    # @return [Boolean]
     def active?
       true
     end

data/lib/munster/controllers/receive_webhooks_controller.rb

@@ -2,55 +2,54 @@
 
 module Munster
   class ReceiveWebhooksController < ActionController::API
-    class HandlerRefused < StandardError
+    class HandlerInactive < StandardError
     end
 
-    class HandlerInactive < StandardError
+    class UnknownHandler < StandardError
     end
 
     def create
-      handler = lookup_handler(params[:service_id]).new
-
+      Rails.error.set_context(**Munster.configuration.error_context)
+      handler = lookup_handler(service_id)
       raise HandlerInactive unless handler.active?
-      raise HandlerRefused unless handler.valid?(request)
-
       handler.handle(request)
-      head :ok
-    rescue KeyError # handler was not found, so we return generic 404 error.
-      render_error("Required parameters were not present in the request", :not_found)
+      render(json: {ok: true, error: nil})
+    rescue UnknownHandler => e
+      Rails.error.report(e, handled: true, severity: :error)
+      render_error_with_status("No handler found for #{service_id.inspect}", status: :not_found)
+    rescue HandlerInactive => e
+      Rails.error.report(e, handled: true, severity: :error)
+      render_error_with_status("Webhook handler #{service_id.inspect} is inactive", status: :service_unavailable)
     rescue => e
-      Rails.error.set_context(**Munster.configuration.error_context)
-      # Rails 7.1 only requires `error` attribute for .report method, but Rails 7.0 requires `handled:` attribute additionally.
-      # We're setting `handled:` and `severity:` attributes to maintain compatibility with all versions of > rails 7.
+      raise e unless handler
+      raise e if handler.expose_errors_to_sender?
       Rails.error.report(e, handled: true, severity: :error)
-
-      if handler&.expose_errors_to_sender?
-        error_for_sender_from_exception(e)
-      else
-        head :ok
-      end
+      render_error_with_status("Internal error (#{e})")
     end
 
-    def error_for_sender_from_exception(e)
-      case e
-      when HandlerRefused
-        render_error("Webhook handler did not validate the request (signature or authentication may be invalid)", :forbidden)
-      when HandlerInactive
-        render_error("Webhook handler is inactive", :service_unavailable)
-      when JSON::ParserError
-        render_error("Request body is not a valid JSON", :bad_request)
-      else
-        render_error("Internal error", :internal_server_error)
-      end
+    def service_id
+      params.require(:service_id)
     end
 
-    def render_error(message_str, status_sym)
-      json = {error: message_str}.to_json
-      render(json: json, status: status_sym)
+    def render_error_with_status(message_str, status: :ok)
+      json = {ok: false, error: message_str}.to_json
+      render(json: json, status: status)
     end
 
     def lookup_handler(service_id_str)
-      Munster.configuration.active_handlers.with_indifferent_access.fetch(service_id_str)
+      active_handlers = Munster.configuration.active_handlers.with_indifferent_access
+      # The config can specify a mapping of:
+      # {"service-1" => MyHandler }
+      # or
+      # {"service-2" => "MyOtherHandler"}
+      # We need to support both, because `MyHandler` is not loaded yet when Rails initializers run.
+      # Zeitwerk takes over after the initializers. So we can't really use a module in the init cycle just yet.
+      # We can, however, use the module name - and resolve it lazily, later.
+      handler_class_or_class_name = active_handlers.fetch(service_id_str)
+      handler_class = handler_class_or_class_name.respond_to?(:constantize) ? handler_class_or_class_name.constantize : handler_class_or_class_name
+      handler_class.new
+    rescue KeyError
+      raise UnknownHandler
     end
   end
 end

data/lib/munster/engine.rb

@@ -1,6 +1,5 @@
 # frozen_string_literal: true
 
-require_relative "../munster"
 require_relative "controllers/receive_webhooks_controller"
 require_relative "jobs/processing_job"
 require_relative "models/received_webhook"
@@ -10,7 +9,6 @@ module Munster
   class Engine < ::Rails::Engine
     isolate_namespace Munster
 
-    autoload :Munster, "munster"
     autoload :ReceiveWebhooksController, "munster/controllers/receive_webhooks_controller"
     autoload :ProcessingJob, "munster/jobs/processing_job"
     autoload :BaseHandler, "munster/base_handler"
@@ -18,5 +16,9 @@ module Munster
     generators do
       require_relative "install_generator"
     end
+
+    routes do
+      post "/:service_id" => "received_webhooks#create"
+    end
   end
 end

data/lib/munster/install_generator.rb

@@ -15,6 +15,7 @@ module Munster
 
     def create_migration_file
       migration_template "create_munster_tables.rb.erb", File.join(db_migrate_path, "create_munster_tables.rb")
+      migration_template "add_headers_to_munster_webhooks.rb.erb", File.join(db_migrate_path, "add_headers_to_munster_webhooks.rb")
     end
 
     def copy_files

data/lib/munster/jobs/processing_job.rb

@@ -1,14 +1,31 @@
 # frozen_string_literal: true
 
-require "active_job" if defined?(Rails)
+require "active_job/railtie"
 
 module Munster
   class ProcessingJob < ActiveJob::Base
+    class WebhookPayloadInvalid < StandardError
+    end
+
     def perform(webhook)
-      # TODO: there should be some sort of locking or concurrency control here, but it's outside of
-      # Munsters scope of responsibility. Developer implementing this should decide how this should be handled.
-      webhook.handler.process(webhook)
-      # TODO: remove process attribute
+      Rails.error.set_context(munster_handler_module_name: webhook.handler_module_name, **Munster.configuration.error_context)
+
+      webhook.with_lock do
+        return unless webhook.received?
+        webhook.processing!
+      end
+
+      if webhook.handler.valid?(webhook.request)
+        webhook.handler.process(webhook)
+        webhook.processed! if webhook.processing?
+      else
+        e = WebhookPayloadInvalid.new("#{webhook.class} #{webhook.id} did not pass validation and was skipped")
+        Rails.error.report(e, handled: true, severity: :error)
+        webhook.failed_validation!
+      end
+    rescue => e
+      webhook.error!
+      raise e
     end
   end
 end

data/lib/munster/models/received_webhook.rb

@@ -1,6 +1,6 @@
 # frozen_string_literal: true
 
-require 'state_machine_enum'
+require "state_machine_enum"
 
 module Munster
   class ReceivedWebhook < ActiveRecord::Base
@@ -11,13 +11,89 @@ module Munster
 
     state_machine_enum :status do |s|
       s.permit_transition(:received, :processing)
+      s.permit_transition(:processing, :failed_validation)
       s.permit_transition(:processing, :skipped)
       s.permit_transition(:processing, :processed)
       s.permit_transition(:processing, :error)
+      s.permit_transition(:error, :received)
+
+      s.after_committed_transition_to(:received) do |webhook|
+        webhook.handler.enqueue(webhook)
+      end
+    end
+
+    # Store the pertinent data from an ActionDispatch::Request into the webhook.
+    # @param [ActionDispatch::Request]
+    def request=(action_dispatch_request)
+      # Filter out all Rack-specific headers such as "rack.input" and the like. We are
+      # only interested in the actual HTTP headers presented by the webserver. Mostly...
+      headers = action_dispatch_request.env.filter_map do |(header_name, header_value)|
+        if header_name.is_a?(String) && header_name.upcase == header_name && header_value.is_a?(String)
+          [header_name, header_value]
+        end
+      end.to_h
+
+      # ...except the path parameters - they do not get parsed from the headers, but instead get set by Journey - the Rails
+      # router - when the ActionDispatch::Request object gets instantiated. They need to be preserved separately in case the Munster
+      # controller gets mounted under a parametrized path - and the path component actually is a parameter that the webhook
+      # handler either needs for validation or for processing
+      headers["action_dispatch.request.path_parameters"] = action_dispatch_request.env.fetch("action_dispatch.request.path_parameters")
+
+      # ...and the raw request body - because we already save it separately
+      headers.delete("RAW_POST_DATA")
+
+      # Verify the request body is not too large
+      request_body_io = action_dispatch_request.env.fetch("rack.input")
+      if request_body_io.size > Munster.configuration.request_body_size_limit
+        raise "Cannot accept the webhook as the request body is larger than #{limit} bytes"
+      end
+
+      write_attribute("body", request_body_io.read.force_encoding(Encoding::BINARY))
+      write_attribute("request_headers", headers)
+    ensure
+      request_body_io.rewind
+    end
+
+    # A Munster handler is, in a way, a tiny Rails controller which runs in a background job. To allow this,
+    # we need to provide access not only to the webhook payload (the HTTP request body, usually), but also
+    # to the rest of the HTTP request - such as headers and route params. For example, imagine you use
+    # a system where your multiple tenants (users) may receive webhooks from the same sender. However, you
+    # need to dispatch those webhooks to those particular tenants of your application. Instead of mounting
+    # Munster as an engine under a common route, you mount it like so:
+    #
+    #    post "/incoming-webhooks/:user_id/:service_id" => "munster/receive_webhooks#create"
+    #
+    # This way, the tenant ID (the `user_id`) parameter is not going to be provided to you inside the webhook
+    # payload, as the sender is not sending it to you at all. However, you do have that parameter in your
+    # route. When processing the webhook, it is important for you to know which tenant has received the
+    # webhook - so that you can manipulate their data, and not the data belonging to another tenant. With
+    # validation, it is important too - in such a multitenant setup every user is likely to have their own,
+    # specific signing secret that they have set up. To find that secret and compare the signature, you
+    # need access to that `user_id` parameter.
+    #
+    # To allow access to these, Munster allows the ActionDispatch::Request object to be persisted. The
+    # persistence is not 1:1 - the Request is a fairly complex object, with lots of things injected into it
+    # by the Rails stack. Not all of those injected properties (Rack headers) are marshalable, some of them
+    # depend on the Rails application configuration, etc. However, we do retain the most important things
+    # for webhooks to be correctly handled.
+    #
+    # * The HTTP request body
+    # * The headers set by the webserver and the downstream proxies
+    # * The request body and query string params, depending on the MIME type
+    # * The route params. These are set by Journey (the Rails router) and cannot be reconstructed from a "bare" request
+    #
+    # While this reconstruction is best-effort it might not be lossless. For example, there might be no access
+    # to Rack hijack, streaming APIs, the cookie jar or other more high-level Rails request access features.
+    # You will, however, have the basics in place - such as the params, the request body, the path params
+    # (as were decoded by your routes) etc. But it should be sufficient to do the basic tasks to process a webhook.
+    #
+    # @return [ActionDispatch::Request]
+    def request
+      ActionDispatch::Request.new(request_headers.merge!("rack.input" => StringIO.new(body.to_s.b)))
     end
 
     def handler
-      handler_module_name.constantize
+      handler_module_name.constantize.new
     end
   end
 end

data/lib/munster/templates/add_headers_to_munster_webhooks.rb.erb

@@ -0,0 +1,5 @@
+class AddHeadersToMunsterWebhooks < ActiveRecord::Migration<%= migration_version %>
+  def change
+    add_column :received_webhooks, :request_headers, :json, null: true
+  end
+end

data/lib/munster/templates/create_munster_tables.rb.erb

@@ -6,7 +6,7 @@ class CreateMunsterTables < ActiveRecord::Migration<%= migration_version %>
       t.string :handler_module_name, null: false
       t.string :status, default: "received", null: false
 
-      # We don't assume, that we can always parse received body as JSON. Body could be invalid or partly missing,
+      # We don't assume that we can always parse the received body as JSON. Body could be invalid or partly missing,
       # we can argue how we can handle that for different integrations, but we still should be able to save this data
       # if it's required. Hence, we don't use :jsonb, but :binary type column here.
       t.binary :body, null: false

data/lib/munster/templates/munster.rb

@@ -1,10 +1,15 @@
 Munster.configure do |config|
   # Active Handlers are defined as hash with key as a service_id and handler class  that would handle webhook request.
+  # A Handler must respond to `.new` and return an object roughly matching `Munster::BaseHandler` in terms of interface.
+  # Use module names (strings) here to allow the handler modules to be lazy-loaded by Rails.
+  #
   # Example:
-  #   {:test => TestHandler, :inactive => InactiveHandler}
+  #   {:test => "TestHandler", :inactive => "InactiveHandler"}
   config.active_handlers = {}
 
-  # It's possible to overwrite default processing job to enahance it. As example if you want to add proper locking or retry mechanism.
+  # It's possible to overwrite default processing job to enahance it. As example if you want to add custom
+  # locking or retry mechanism. You want to inherit that job from Munster::ProcessingJob because the background
+  # job also manages the webhook state.
   #
   # Example:
   #
@@ -15,9 +20,9 @@ Munster.configure do |config|
   #     end
   #   end
   #
-  # This is how you can change processing job:
+  # In the config a string with your job' class name can be used so that the job can be lazy-loaded by Rails:
   #
-  # config.processing_job_class = WebhookProcessingJob
+  # config.processing_job_class = "WebhookProcessingJob"
 
   # We're using a common interface for error reporting provided by Rails, e.g Rails.error.report. In some cases
   # you want to enhance those errors with additional context. As example to provide a namespace:
@@ -25,4 +30,11 @@ Munster.configure do |config|
   # { appsignal: { namespace: "webhooks" } }
   #
   # config.error_context = { appsignal: { namespace: "webhooks" } }
+
+  # Incoming webhooks will be written into your DB without any prior validation. By default, Munster limits the
+  # request body size for webhooks to 512 KiB, so that it would not be too easy for an attacker to fill your
+  # database with junk. However, if you are receiving very large webhook payloads you might need to increase
+  # that limit (or make it even smaller for extra security)
+  #
+  # config.request_body_size_limit = 2.megabytes
 end

data/lib/munster/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Munster
-  VERSION = "0.3.1"
+  VERSION = "0.4.1"
 end

data/lib/munster.rb

@@ -18,7 +18,8 @@ end
 class Munster::Configuration
   include ActiveSupport::Configurable
 
-  config_accessor(:processing_job_class) { Munster::ProcessingJob }
-  config_accessor(:active_handlers) { [] }
-  config_accessor(:error_context) { {} }
+  config_accessor(:processing_job_class, default: Munster::ProcessingJob)
+  config_accessor(:active_handlers, default: {})
+  config_accessor(:error_context, default: {})
+  config_accessor(:request_body_size_limit, default: 512.kilobytes)
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: munster
 version: !ruby/object:Gem::Version
-  version: 0.3.1
+  version: 0.4.1
 platform: ruby
 authors:
 - Stanislav Katkov
 autorequire:
-bindir: exe
+bindir: bin
 cert_chain: []
-date: 2024-06-07 00:00:00.000000000 Z
+date: 2024-07-29 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rails
@@ -94,7 +94,7 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '5.0'
-description: Webhooks framework for Rails applications
+description: Webhooks processing engine for Rails applications
 email:
 - skatkov@cheddar.me
 executables: []
@@ -106,7 +106,6 @@ files:
 - ".standard.yml"
 - CHANGELOG.md
 - LICENSE
-- LICENSE.txt
 - README.md
 - Rakefile
 - config/routes.rb
@@ -171,6 +170,10 @@ files:
 - example/tmp/.keep
 - example/tmp/pids/.keep
 - example/vendor/.keep
+- handler-examples/customer_io_handler.rb
+- handler-examples/revolut_business_v1_handler.rb
+- handler-examples/revolut_business_v2_handler.rb
+- handler-examples/starling_payments_handler.rb
 - lib/munster.rb
 - lib/munster/base_handler.rb
 - lib/munster/controllers/receive_webhooks_controller.rb
@@ -178,17 +181,18 @@ files:
 - lib/munster/install_generator.rb
 - lib/munster/jobs/processing_job.rb
 - lib/munster/models/received_webhook.rb
+- lib/munster/templates/add_headers_to_munster_webhooks.rb.erb
 - lib/munster/templates/create_munster_tables.rb.erb
 - lib/munster/templates/munster.rb
 - lib/munster/version.rb
 - lib/tasks/munster_tasks.rake
-- sig/munster.rbs
 homepage: https://www.cheddar.me/
 licenses:
 - MIT
 metadata:
   homepage_uri: https://www.cheddar.me/
   source_code_uri: https://github.com/cheddar-me/munster
+  changelog_uri: https://github.com/cheddar-me/munster/blob/main/CHANGELOG.md
 post_install_message:
 rdoc_options: []
 require_paths:
@@ -204,8 +208,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.3
+rubygems_version: 3.5.9
 signing_key:
 specification_version: 4
-summary: Webhooks framework for Rails applications
+summary: Webhooks processing engine for Rails applications
 test_files: []

data/LICENSE.txt

@@ -1,21 +0,0 @@
-The MIT License (MIT)
-
-Copyright (c) 2024 Stanislav Katkov
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in
-all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-THE SOFTWARE.

data/sig/munster.rbs

@@ -1,4 +0,0 @@
-module Munster
-  VERSION: String
-  # See the writing guide of rbs: https://github.com/ruby/rbs#guides
-end