activejob-compressible 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 28d3f54a6ca82b0bfbc6d00db5ff31159f7ce19fd56e157e2805d913cec79165
+  data.tar.gz: dd256e7a07ff11bdb73180d05a4f85c1bfb94aab754b7c8d64ab22517aa95bb0
+SHA512:
+  metadata.gz: d716916f8c7d9d4d8ee07a0ac08caedc9445e9cd6f8a0cd8946f1cc8f635b47fe87b0faa9a9f8c40953061d0bc035a9e0dcb1f0d53b7e444ebc17644c98faf26
+  data.tar.gz: ed162b58f90dbc68136bd2b5e6132ad4d64afa9d2291904733dd86baa1b87d94c4536334dbaa9822f1d0dffe1a093875d8dc1cb28aa200c476d54a61416257fe

data/CHANGELOG.md

@@ -0,0 +1,34 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.0.0] - 2025-06-05
+
+### Added
+- Initial release of ActiveJob::Compressible gem
+- Transparent compression/decompression for large ActiveJob payloads
+- Automatic detection of Rails cache backend limits for smart defaults
+- Configurable compression threshold via `ActiveJob::Compressible.configure`
+- Support for zlib compression algorithm with base64 encoding
+- Backward compatibility for existing uncompressed jobs
+- Safe compression markers (`"_compressed"` key) for reliable detection
+- Comprehensive test suite with dynamic configuration testing
+- Support for Ruby >= 3.1 and Rails 6.0-8.x
+- Detailed documentation and usage examples
+
+### Technical Details
+- Uses zlib compression with base64 encoding for safety
+- Only compresses Hash arguments that exceed the configured threshold
+- Defaults to cache `value_max_bytes` minus 100KB for safety margin
+- Graceful fallback to 948KB default if cache limits not detected
+- Thread-safe configuration system
+- Minimal performance overhead (~1-5ms for typical payloads)
+
+### Documentation
+- Complete README with usage examples and configuration options
+- API documentation with inline comments
+- Development setup and testing instructions
+- Performance characteristics and compatibility notes

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Evan Lee
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,158 @@
+# ActiveJob::Compressible
+
+Transparent compression for large ActiveJob payloads to avoid cache backend limits.
+
+## Overview
+
+The `ActiveJob::Compressible` concern provides automatic compression and decompression for large job arguments to prevent exceeding cache or queue backend size limits (such as Redis/Memcached). This is especially useful for jobs that handle large payloads like webhook events, API responses, or bulk data processing.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'activejob-compressible'
+```
+
+And then execute:
+
+    $ bundle install
+
+Or install it yourself as:
+
+    $ gem install activejob-compressible
+
+## Usage
+
+Include the concern in your job classes:
+
+```ruby
+class WebhookJob < ApplicationJob
+  include ActiveJob::Compressible
+
+  def perform(large_webhook_payload)
+    # Process the payload - compression/decompression is transparent
+    process_webhook(large_webhook_payload)
+  end
+end
+```
+
+## How It Works
+
+- **Automatic Compression**: When jobs are serialized (enqueued), the first argument is checked. If it's a Hash and its JSON representation exceeds the compression threshold, it's automatically compressed using zlib and base64 encoding.
+
+- **Transparent Decompression**: When jobs are deserialized (dequeued), compressed arguments are automatically detected and decompressed.
+
+- **Backward Compatibility**: Old jobs (created before adding the concern) continue to work without any changes during rolling deployments.
+
+- **Safe Markers**: Compressed payloads are marked with a `"_compressed"` key to ensure reliable detection.
+
+## Configuration
+
+```ruby
+ActiveJob::Compressible.configure do |config|
+  config.compression_threshold = 500_000  # Compress payloads > 500KB
+end
+```
+
+### Configuration Options
+
+- **`compression_threshold`**: Size in bytes above which payloads will be compressed. Defaults to your cache backend's `value_max_bytes` minus 100KB for safety, or 948KB if not detected.
+
+- **`compression_algorithm`**: Currently only `:zlib` is supported.
+
+### Default Behavior
+
+The gem automatically detects your Rails cache configuration and sets a safe default threshold:
+
+```ruby
+# Automatically uses your cache backend's limits
+max_bytes = Rails.cache.options[:value_max_bytes] || 1_048_576
+default_threshold = max_bytes - 100_000  # Safe margin
+```
+
+## Examples
+
+### Basic Usage
+
+```ruby
+class DataProcessingJob < ApplicationJob
+  include ActiveJob::Compressible
+
+  def perform(large_dataset)
+    # large_dataset is automatically compressed if > threshold
+    process_data(large_dataset)
+  end
+end
+
+# Enqueue with large payload
+DataProcessingJob.perform_later({
+  "records" => large_array_of_data,
+  "metadata" => additional_info
+})
+```
+
+### Custom Configuration
+
+```ruby
+# config/initializers/activejob_compressible.rb
+ActiveJob::Compressible.configure do |config|
+  config.compression_threshold = 750_000  # 750KB threshold
+end
+```
+
+### Multiple Job Classes
+
+```ruby
+class ApiResponseJob < ApplicationJob
+  include ActiveJob::Compressible
+  # Inherits default configuration
+end
+
+class BulkImportJob < ApplicationJob
+  include ActiveJob::Compressible
+  # Also inherits default configuration
+end
+```
+
+## Requirements
+
+- Ruby >= 2.7.0
+- ActiveJob >= 6.0
+- ActiveSupport >= 6.0
+
+## Compatibility
+
+- **Rails 6.0-8.x**: Full support
+- **Queue Adapters**: Works with any ActiveJob queue adapter (Sidekiq, Resque, etc.)
+- **Cache Backends**: Automatically detects Redis, Memcached, and other cache limits
+- **Rolling Deployments**: Safe to deploy incrementally
+
+## Performance
+
+- **Compression Ratio**: Typically 60-90% size reduction for JSON payloads
+- **Speed**: zlib compression/decompression adds ~1-5ms for typical payloads
+- **Memory**: Minimal overhead, only processes large payloads
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests.
+
+```bash
+# Run tests
+bundle exec rake test
+
+# Run tests quietly
+bundle exec rake test TESTOPTS="--quiet"
+
+# Run RuboCop
+bundle exec rubocop
+```
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/Archetypically/activejob-compressible.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/lib/activejob/compressible/configuration.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+module ActiveJob
+  # Compressible module provides transparent compression for large ActiveJob payloads.
+  module Compressible
+    # Configuration class for ActiveJob::Compressible settings.
+    #
+    # Manages compression behavior including threshold limits and algorithm selection.
+    # The compression threshold determines when job arguments should be compressed
+    # based on their serialized size.
+    class Configuration
+      attr_accessor :compression_threshold
+      attr_reader :compression_algorithm
+
+      def initialize
+        @compression_threshold = default_compression_threshold
+        @compression_algorithm = :zlib
+      end
+
+      def compression_algorithm=(algorithm)
+        raise ArgumentError, "Only :zlib compression algorithm is currently supported" unless algorithm == :zlib
+
+        @compression_algorithm = algorithm
+      end
+
+      private
+
+      def default_compression_threshold
+        max_bytes = begin
+          Rails.cache.options[:value_max_bytes] if defined?(Rails)
+        rescue StandardError
+          nil
+        end || 1_048_576
+        max_bytes - 100_000
+      end
+    end
+
+    def self.configuration
+      @configuration ||= Configuration.new
+    end
+
+    def self.configure
+      yield(configuration)
+    end
+  end
+end

data/lib/activejob/compressible/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module ActiveJob
+  module Compressible
+    VERSION = "1.0.0"
+  end
+end

data/lib/activejob/compressible.rb

@@ -0,0 +1,78 @@
+# frozen_string_literal: true
+
+require "active_support/concern"
+require "zlib"
+require "base64"
+require "json"
+
+module ActiveJob
+  # CompressibleJob is a concern for ActiveJob classes that transparently compresses
+  # and decompresses large job arguments to avoid exceeding cache or queue backend limits
+  # (such as Dalli/Memcached or Redis). This is especially useful for jobs that handle
+  # large payloads, such as webhook events or API responses.
+  #
+  # Usage:
+  #   class MyJob < ApplicationJob
+  #     include ActiveJob::Compressible
+  #     # ...
+  #   end
+  #
+  # How it works:
+  #   - When the job is serialized (enqueued), the first argument is checked.
+  #   - If the argument is a Hash and its JSON representation exceeds the compression threshold,
+  #     it is compressed using zlib and base64, and marked with a special key.
+  #   - When the job is deserialized (dequeued), the argument is checked for the marker and
+  #     transparently decompressed if needed.
+  #   - Old jobs (with uncompressed payloads) are still supported for backward compatibility.
+  #
+  # Configuration:
+  #   - The compression threshold is configurable via ActiveJob::Compressible.configuration
+  #   - Defaults to cache backend's :value_max_bytes minus 100,000 bytes for safety
+  #
+  # This concern is safe for rolling deployments and can be included in any job class that may
+  # enqueue large payloads.
+  module Compressible
+    extend ActiveSupport::Concern
+
+    included do
+      def serialize
+        super.tap do |h|
+          if h["arguments"] && h["arguments"].first.is_a?(Hash)
+            h["arguments"][0] = self.class.compress_if_needed(h["arguments"].first)
+          end
+        end
+      end
+    end
+
+    class_methods do
+      def deserialize(job_data)
+        arg = job_data["arguments"].first
+        job_data = job_data.dup
+        job_data["arguments"][0] = decompress_if_needed(arg)
+        super
+      end
+
+      def compress_if_needed(obj)
+        json_str = obj.to_json
+        if json_str.bytesize > compression_threshold
+          compressed = Base64.encode64(Zlib::Deflate.deflate(json_str))
+          { "_compressed" => true, "data" => compressed }
+        else
+          obj
+        end
+      end
+
+      def decompress_if_needed(arg)
+        if arg.is_a?(Hash) && arg["_compressed"]
+          JSON.parse(Zlib::Inflate.inflate(Base64.decode64(arg["data"])))
+        else
+          arg
+        end
+      end
+
+      def compression_threshold
+        ActiveJob::Compressible.configuration.compression_threshold
+      end
+    end
+  end
+end

data/lib/activejob-compressible.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+require "activejob/compressible/version"
+require "activejob/compressible/configuration"
+require "activejob/compressible"

data/sig/activejob/compressible/configuration.rbs

@@ -0,0 +1,16 @@
+module ActiveJob
+  module Compressible
+    class Configuration
+      attr_accessor compression_threshold: Integer
+      attr_reader compression_algorithm: Symbol
+
+      def initialize: () -> void
+
+      def compression_algorithm=: (Symbol algorithm) -> Symbol
+
+      private
+
+      def default_compression_threshold: () -> Integer
+    end
+  end
+end

data/sig/activejob/compressible/version.rbs

@@ -0,0 +1,5 @@
+module ActiveJob
+  module Compressible
+    VERSION: String
+  end
+end

data/sig/activejob/compressible.rbs

@@ -0,0 +1,23 @@
+module ActiveJob
+  module Compressible
+    extend ActiveSupport::Concern
+
+    def serialize: () -> Hash[String, untyped]
+
+    module ClassMethods
+      def deserialize: (Hash[String, untyped] job_data) -> ActiveJob::Base
+
+      def compress_if_needed: (Hash[String, untyped] obj) -> (Hash[String, untyped] | Hash[String, String | bool])
+
+      def decompress_if_needed: (untyped arg) -> untyped
+
+      def compression_threshold: () -> Integer
+    end
+
+    VERSION: String
+
+    def self.configuration: () -> Configuration
+
+    def self.configure: () { (Configuration) -> void } -> void
+  end
+end

metadata

@@ -0,0 +1,94 @@
+--- !ruby/object:Gem::Specification
+name: activejob-compressible
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- Evan Lee
+bindir: exe
+cert_chain: []
+date: 2025-06-05 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: activejob
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '6.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '6.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9.0'
+- !ruby/object:Gem::Dependency
+  name: activesupport
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '6.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '6.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9.0'
+description: Automatically compress/decompress large job arguments to avoid size restrictions
+email:
+- evan.lee@shopify.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE
+- README.md
+- lib/activejob-compressible.rb
+- lib/activejob/compressible.rb
+- lib/activejob/compressible/configuration.rb
+- lib/activejob/compressible/version.rb
+- sig/activejob/compressible.rbs
+- sig/activejob/compressible/configuration.rbs
+- sig/activejob/compressible/version.rbs
+homepage: https://github.com/Archetypically/activejob-compressible
+licenses:
+- MIT
+metadata:
+  rubygems_mfa_required: 'true'
+  homepage_uri: https://github.com/Archetypically/activejob-compressible
+  source_code_uri: https://github.com/Archetypically/activejob-compressible
+  changelog_uri: https://github.com/Archetypically/activejob-compressible/blob/main/CHANGELOG.md
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.1'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.6.2
+specification_version: 4
+summary: Transparent compression for large ActiveJob payloads
+test_files: []