actioncable-enhanced-postgresql-adapter 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: d4b7f64d6bbcae86b24c05c84cc7d182d260c7dd2c6b41300ee5de5e4a87d31a
+  data.tar.gz: ab38f315f5f5db4901c951c53e51d475da2fce45a50fc607cc2b56cb9e89b6c2
+SHA512:
+  metadata.gz: 3e2d3ebd77fef8816c622f47075ab51e89eb6f10529e9bdd95c60d7d34b64bc2a19146cc1931c264ae57dd1c51b438e5e926cab1eacba29837f406811e9209bc
+  data.tar.gz: 81cc41a2e93e420069ff42f6deddb8766bea59e0689ced9c89d5c58e9809bff385c37e8e58660ed78b70012a92ce3ea22caece7033329243659be271a2eeaed3

data/CHANGELOG.md

@@ -0,0 +1,4 @@
+1.0.0
+
+- Support > 8000 byte payloads
+- Remove hard dependency on ActiveRecord

data/README.md

@@ -0,0 +1,61 @@
+# actioncable-enhanced-postgresql-adapter
+
+This gem provides an enhanced PostgreSQL adapter for ActionCable. It is based on the original PostgreSQL adapter, but includes the following enhancements:
+- Ability to broadcast payloads larger than 8000 bytes
+- Not dependent on ActiveRecord (but can still integrate with it if available)
+
+### Approach
+
+To overcome the 8000 bytes limit, we temporarily store large payloads in an [unlogged](https://www.crunchydata.com/blog/postgresl-unlogged-tables) database table named `action_cable_large_payloads`. The table is lazily created on first broadcast.
+
+We then broadcast a payload in the style of `__large_payload:<encrypted-payload-id>`. The listener client then decrypts incoming ID's, fetches the original payload from the database, and replaces the temporary payload before invoking the subscriber callback.
+
+ID encryption is done to prevent spoofing large payloads by manually broadcasting messages prefixed with `__large_payload:` with just an auto incrementing integer.
+
+Note that payloads smaller than 8000 bytes are sent directly via NOTIFY, as per the original adapter.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem "actioncable-enhanced-postgresql-adapter", git: "https://github.com/reclaim-the-stack/actioncable-enhanced-postgresql-adapter"
+```
+
+## Usage
+
+In your `config/cable.yml` file, change the adapter for relevant environments to `enhanced_postgresql`:
+
+```yaml
+development:
+  adapter: enhanced_postgresql
+
+production:
+  adapter: enhanced_postgresql
+```
+
+## Configuration
+
+The following configuration options are available:
+
+- `payload_encryptor_secret` - The secret used to encrypt large payload ID's. Defaults to `Rails.application.secret_key_base` or the `SECRET_KEY_BASE` environment variable unless explicitly specified.
+- `url` - Set this if you want to use a different database than the one provided by ActiveRecord. Must be a valid PostgreSQL connection string.
+- `connection_pool_size` - Set this in conjunction with `url` to set the size of the postgres connection pool used for broadcasts. Defaults to `RAILS_MAX_THREADS` environment variable or falls back to 5.
+
+## Performance
+
+For payloads smaller than 8000 bytes, which should cover the majority of cases, performance is identical to the original adapter.
+
+When broadcasting large payloads, one has to consider the overhead of storing and fetching the payload from the database. For low frequency broadcasting, this overhead is likely negligible. But take care if you're doing very high frequency broadcasting.
+
+Note that whichever ActionCable adapter you're using, sending large payloads with high frequency is an anti-pattern. Even Redis pub/sub has [limitations](https://redis.io/docs/reference/clients/#output-buffer-limits) to be aware of.
+
+### Cleanup of large payloads
+
+Deletion of stale payloads (2 minutes or older) are triggered every 100 large payload inserts. We do this by looking at the incremental ID generated on insert and checking if it is evenly divisible by 100. This approach avoids having to manually schedule cleanup jobs while striking a balance between performance and cleanup frequency.
+
+## Development
+
+- Clone repo
+- `bundle install` to install dependencies
+- `bundle exec ruby test/postgresql_test.rb` to run tests

data/actioncable-enhanced-postgresql-adapter.gemspec

@@ -0,0 +1,24 @@
+Gem::Specification.new do |spec|
+  spec.name = "actioncable-enhanced-postgresql-adapter"
+  spec.version = "1.0.0"
+  spec.authors = ["David Backeus"]
+  spec.email = ["david.backeus@mynewsdesk.com"]
+
+  spec.summary = "ActionCable adapter for PostgreSQL that enhances the default."
+  spec.description = "Handles the 8000 byte limit for PostgreSQL NOTIFY payloads"
+  spec.homepage = "https://github.com/dbackeus/actioncable-enhanced-postgresql-adapter"
+  spec.license = "MIT"
+  spec.required_ruby_version = ">= 2.7.0"
+
+  spec.metadata = {
+    "homepage_uri" => spec.homepage,
+    "source_code_uri" => spec.homepage,
+    "changelog_uri" => "#{spec.homepage}/CHANGELOG.md"
+  }
+
+  spec.files = %w[README.md CHANGELOG.md actioncable-enhanced-postgresql-adapter.gemspec] + Dir["lib/**/*"]
+
+  spec.add_dependency "actioncable", ">= 6.0"
+  spec.add_dependency "connection_pool", ">= 2.2.5" # Ruby 2.7 compatible version
+  spec.add_dependency "pg", "~> 1.5"
+end

data/lib/action_cable/subscription_adapter/enhanced_postgresql.rb

@@ -0,0 +1,131 @@
+# freeze_string_literal: true
+
+require "action_cable/subscription_adapter/postgresql"
+require "connection_pool"
+
+module ActionCable
+  module SubscriptionAdapter
+    class EnhancedPostgresql < PostgreSQL
+      MAX_NOTIFY_SIZE = 7997 # documented as 8000 bytes, but there appears to be some overhead in transit
+      LARGE_PAYLOAD_PREFIX = "__large_payload:"
+      INSERTS_PER_DELETE = 100 # execute DELETE query every N inserts
+
+      LARGE_PAYLOADS_TABLE = "action_cable_large_payloads"
+      CREATE_LARGE_TABLE_QUERY = <<~SQL
+        CREATE UNLOGGED TABLE IF NOT EXISTS #{LARGE_PAYLOADS_TABLE} (
+          id SERIAL PRIMARY KEY,
+          payload TEXT NOT NULL,
+          created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
+        )
+      SQL
+      CREATE_CREATED_AT_INDEX_QUERY = <<~SQL
+        CREATE INDEX IF NOT EXISTS index_action_cable_large_payloads_on_created_at
+        ON #{LARGE_PAYLOADS_TABLE} (created_at)
+      SQL
+      INSERT_LARGE_PAYLOAD_QUERY = "INSERT INTO #{LARGE_PAYLOADS_TABLE} (payload, created_at) VALUES ($1, CURRENT_TIMESTAMP) RETURNING id"
+      SELECT_LARGE_PAYLOAD_QUERY = "SELECT payload FROM #{LARGE_PAYLOADS_TABLE} WHERE id = $1"
+      DELETE_LARGE_PAYLOAD_QUERY = "DELETE FROM #{LARGE_PAYLOADS_TABLE} WHERE created_at < CURRENT_TIMESTAMP - INTERVAL '2 minutes'"
+
+      def initialize(*)
+        super
+
+        @url = @server.config.cable[:url]
+        @connection_pool_size = @server.config.cable[:connection_pool_size] || ENV["RAILS_MAX_THREADS"] || 5
+      end
+
+      def broadcast(channel, payload)
+        channel = channel_with_prefix(channel)
+
+        with_broadcast_connection do |pg_conn|
+          channel = pg_conn.escape_identifier(channel_identifier(channel))
+          payload = pg_conn.escape_string(payload)
+
+          if payload.bytesize > MAX_NOTIFY_SIZE
+            payload_id = insert_large_payload(pg_conn, payload)
+
+            if payload_id % INSERTS_PER_DELETE == 0
+              pg_conn.exec(DELETE_LARGE_PAYLOAD_QUERY)
+            end
+
+            # Encrypt payload_id to prevent simple integer ID spoofing
+            encrypted_payload_id = payload_encryptor.encrypt_and_sign(payload_id)
+
+            payload = "#{LARGE_PAYLOAD_PREFIX}#{encrypted_payload_id}"
+          end
+
+          pg_conn.exec("NOTIFY #{channel}, '#{payload}'")
+        end
+      end
+
+      def payload_encryptor
+        @payload_encryptor ||= begin
+          secret = @server.config.cable[:payload_encryptor_secret]
+          secret ||= Rails.application.secret_key_base if defined? Rails
+          secret ||= ENV["SECRET_KEY_BASE"]
+
+          raise ArgumentError, "Missing payload_encryptor_secret configuration for ActionCable EnhancedPostgresql adapter. You need to either explicitly configure it in cable.yml or set the SECRET_KEY_BASE environment variable." unless secret
+
+          secret_32_byte = Digest::SHA256.digest(secret)
+          ActiveSupport::MessageEncryptor.new(secret_32_byte)
+        end
+      end
+
+      def with_broadcast_connection(&block)
+        return super unless @url
+
+        connection_pool.with do |pg_conn|
+          yield pg_conn
+        end
+      end
+
+      # Called from the Listener thread
+      def with_subscriptions_connection(&block)
+        return super unless @url
+
+        pg_conn = PG::Connection.new(@url)
+        pg_conn.exec("SET application_name = #{pg_conn.escape_identifier(identifier)}")
+        yield pg_conn
+      ensure
+        pg_conn&.close
+      end
+
+      private
+
+      def connection_pool
+        @connection_pool ||= ConnectionPool.new(size: @connection_pool_size, timeout: 5) do
+          PG::Connection.new(@url)
+        end
+      end
+
+      def insert_large_payload(pg_conn, payload)
+        result = pg_conn.exec_params(INSERT_LARGE_PAYLOAD_QUERY, [payload])
+        result.first.fetch("id").to_i
+      rescue PG::UndefinedTable
+        pg_conn.exec(CREATE_LARGE_TABLE_QUERY)
+        pg_conn.exec(CREATE_CREATED_AT_INDEX_QUERY)
+        retry
+      end
+
+      # Override needed to ensure we reference our local Listener class
+      def listener
+        @listener || @server.mutex.synchronize { @listener ||= Listener.new(self, @server.event_loop) }
+      end
+
+      class Listener < PostgreSQL::Listener
+        def invoke_callback(callback, message)
+          if message.start_with?(LARGE_PAYLOAD_PREFIX)
+            encrypted_payload_id = message.delete_prefix(LARGE_PAYLOAD_PREFIX)
+            payload_id = @adapter.payload_encryptor.decrypt_and_verify(encrypted_payload_id)
+
+            @adapter.with_broadcast_connection do |pg_conn|
+              result = pg_conn.exec_params(SELECT_LARGE_PAYLOAD_QUERY, [payload_id])
+              message = result.first.fetch("payload")
+            end
+          end
+
+          @event_loop.post { super }
+        end
+      end
+    end
+  end
+end

data/lib/actioncable-enhanced-postgresql-adapter.rb

@@ -0,0 +1,2 @@
+require_relative "action_cable/subscription_adapter/enhanced_postgresql"
+require_relative "railtie" if defined? Rails::Railtie

data/lib/railtie.rb

@@ -0,0 +1,10 @@
+class ActionCable::SubscriptionAdapter::EnhancedPostgresql
+  class Railtie < ::Rails::Railtie
+    initializer "action_cable.enhanced_postgresql_adapter" do
+      ActiveSupport.on_load(:active_record) do
+        large_payloads_table = ActionCable::SubscriptionAdapter::EnhancedPostgresql::LARGE_PAYLOADS_TABLE
+        ActiveRecord::SchemaDumper.ignore_tables << large_payloads_table
+      end
+    end
+  end
+end

metadata

@@ -0,0 +1,94 @@
+--- !ruby/object:Gem::Specification
+name: actioncable-enhanced-postgresql-adapter
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- David Backeus
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2023-11-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: actioncable
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '6.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '6.0'
+- !ruby/object:Gem::Dependency
+  name: connection_pool
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.2.5
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.2.5
+- !ruby/object:Gem::Dependency
+  name: pg
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.5'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.5'
+description: Handles the 8000 byte limit for PostgreSQL NOTIFY payloads
+email:
+- david.backeus@mynewsdesk.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- README.md
+- actioncable-enhanced-postgresql-adapter.gemspec
+- lib/action_cable/subscription_adapter/enhanced_postgresql.rb
+- lib/actioncable-enhanced-postgresql-adapter.rb
+- lib/railtie.rb
+homepage: https://github.com/dbackeus/actioncable-enhanced-postgresql-adapter
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/dbackeus/actioncable-enhanced-postgresql-adapter
+  source_code_uri: https://github.com/dbackeus/actioncable-enhanced-postgresql-adapter
+  changelog_uri: https://github.com/dbackeus/actioncable-enhanced-postgresql-adapter/CHANGELOG.md
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.7.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.4.10
+signing_key:
+specification_version: 4
+summary: ActionCable adapter for PostgreSQL that enhances the default.
+test_files: []