rack-smart_compress 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 31b1ea77cbc0e6ff01ec184c4c99cc68c712ed24e94f8db1214f1cff24250ff8
+  data.tar.gz: b4fc25908f2e0b73872a7ca38a4276b41f75cc592a470f3144aa853063b2b8ca
+SHA512:
+  metadata.gz: b7562181678286e9507ae01edbc8a01779c61ab0bfa97b94bde0768ddbf7d78bb8790090391b26355dae97ae4f3d25f4c0f61098367e7672ba2b552f303b85d2
+  data.tar.gz: a5ae38ec9123cc4d4350d55004c57128b89130254f1ccc9e831d0c61318065b131b55d22a95f42057bbedc4d5932b4afe489dc4d1c74f8a4369858d30160f3a3

data/CHANGELOG.md

@@ -0,0 +1,30 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.2.0] - 2026-08-16
+
+### Added
+- Pre-Compressed Static Asset Middleware (`Rack::SmartCompress::Static`) supporting build-time `.zst`, `.br`, and `.gz` static files with 0% runtime CPU overhead.
+- Configuration DSL via `Rack::SmartCompress.configure { |c| ... }` and `Rack::SmartCompress.configuration`.
+- Dedicated Rails configuration hook via `config.smart_compress` namespace.
+- RFC 9110 §8.8.1 ETag weakening (`W/"..."`) for dynamically compressed responses.
+- RFC 9111 `Cache-Control: no-transform` support (skips compression when present).
+- 206 Partial Content and `Content-Range` skipping to protect byte-range requests.
+- Wildcard `Accept-Encoding: *` and explicit `q=0` rejection handling.
+- Real chunked streaming for Brotli (`Brotli::Compressor`).
+- ActiveSupport / Custom callable telemetry instrumentation (`rack_smart_compress.compress`).
+- GitHub Actions CI workflow supporting Ruby 3.0, 3.1, 3.2, 3.3, and 3.4.
+
+### Fixed
+- Fixed Zstd streaming class typo (`Zstd::StreamingCompress` instead of non-existent `Zstd::StreamingCompressor`).
+- Fixed `Rack::BodyProxy` streaming misclassification in Rails applications.
+- Relieved LRU cache mutex contention by computing compression outside the lock.
+- Optimized LRU cache key hashing to eliminate large memory allocations.
+- Added 1-second load average sampling cooldown to prevent `/proc/loadavg` syscall overhead on high-frequency requests.
+
+## [0.1.0] - Initial Release
+- Initial release with Zstandard, Brotli, Gzip, and Deflate compression support.

data/README.md

@@ -0,0 +1,198 @@
+# ⚡ `rack-smart_compress`
+
+> High-performance **Zstandard (`zstd`)**, **Brotli (`br`)**, and **Gzip (`gzip`)** dynamic HTTP compression & pre-compressed static asset middleware for Rack and Ruby on Rails applications.
+
+[![Gem Version](https://badge.fury.io/rb/rack-smart_compress.svg)](https://badge.fury.io/rb/rack-smart_compress)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+`Rack::SmartCompress` replaces traditional `Rack::Deflater` by automatically negotiating modern, hyper-efficient compression algorithms like **Zstd** and **Brotli** with fallback to Gzip. It applies smart payload thresholding, LRU caching, and content-type filtering to cut outgoing server bandwidth by up to **40%** with zero setup friction.
+
+---
+
+## 🥊 Why `Rack::SmartCompress` vs `Rack::Deflater`?
+
+| Feature | `Rack::Deflater` | `rack-brotli` | `Rack::SmartCompress` ⚡ |
+| :--- | :---: | :---: | :---: |
+| **Zstandard (`zstd`) Support** | ❌ | ❌ | ✅ **Native** |
+| **Brotli (`br`) Support** | ❌ | ✅ | ✅ **Native** |
+| **Gzip & Deflate Fallback** | ✅ | ❌ | ✅ **Native** |
+| **Chunked Stream Compression** | ⚠️ Buffers | ❌ | ✅ **True Chunk Streaming** |
+| **Pre-Compressed Static Assets (`.zst`, `.br`, `.gz`)** | ❌ | ❌ | ✅ **0% CPU Overhead** |
+| **RFC 9110 ETag Weakening (`W/"..."`)** | ⚠️ Partial | ❌ | ✅ **RFC Compliant** |
+| **`Cache-Control: no-transform` Respect** | ❌ | ❌ | ✅ **RFC Compliant** |
+| **206 Partial Content / Range Protection** | ⚠️ Buggy | ❌ | ✅ **Protected** |
+| **In-Memory LRU Payload Cache** | ❌ | ❌ | ✅ **Thread-Safe LRU** |
+| **Dynamic CPU Load Advisor** | ❌ | ❌ | ✅ **Automatic** |
+| **Rails Railtie Auto-Inject** | ❌ | ❌ | ✅ **`config.smart_compress`** |
+| **ActiveSupport Telemetry Instrumentation** | ❌ | ❌ | ✅ **Built-in** |
+
+---
+
+## ✨ Features
+
+- 🏎️ **Next-Gen Encodings:** Native support for **Zstandard (`zstd`)**, **Brotli (`br`)**, **Gzip**, and **Deflate**.
+- 📦 **Pre-Compressed Static Assets:** `Rack::SmartCompress::Static` serves build-time `.zst`, `.br`, and `.gz` files with **0% runtime CPU overhead**.
+- 🌊 **True Chunked Streaming:** Progressive, non-buffering stream compression for Zstandard (`Zstd::StreamingCompress`), Brotli (`Brotli::Compressor`), and Gzip.
+- 🧠 **Smart Payload Filtering:** Automatically skips tiny payloads (< 1KB) where compression overhead increases latency without savings.
+- 🛡️ **RFC 9110 & 9111 Compliant:** Automatic strong-to-weak ETag conversion (`W/"..."`), `Cache-Control: no-transform` respect, and 206 Partial Content / `Content-Range` protection.
+- 🛑 **Media & Binary Exclusion:** Skips already compressed assets (`images`, `audio`, `video`, `zip`, `pdf`).
+- ⚡ **Auto-Negotiation:** Respects HTTP `Accept-Encoding` quality weights (`qvalue`) and `*` wildcard sent by modern browsers and HTTP clients.
+- 📦 **Rails Auto-Integration:** Includes a built-in Railtie with `config.smart_compress` support.
+- 📊 **Telemetry & Observability:** Emits `rack_smart_compress.compress` notifications via `ActiveSupport::Notifications` or custom callbacks.
+- 🛡️ **Rack 2 & Rack 3 Compatible:** Fully compliant with modern Rack specifications.
+
+---
+
+## 📦 Installation
+
+Add this line to your application's `Gemfile`:
+
+```ruby
+gem "rack-smart_compress"
+```
+
+To enable native **Brotli** and **Zstandard** compression drivers, add their gems to your Gemfile:
+
+```ruby
+gem "brotli", "~> 0.4"     # Optional: enables Brotli (br) compression
+gem "zstd-ruby", "~> 1.5"  # Optional: enables Zstandard (zstd) compression
+```
+
+Then execute:
+
+```bash
+$ bundle install
+```
+
+---
+
+## 🚀 Quick Start
+
+### Ruby on Rails Integration
+
+In Rails, `Rack::SmartCompress` automatically attaches to your middleware stack via Railtie.
+
+Configure options in `config/initializers/smart_compress.rb` or `config/environments/production.rb`:
+
+```ruby
+Rails.application.configure do
+  config.smart_compress.min_size = 1024                  # Only compress responses >= 1 KB
+  config.smart_compress.encodings = %w[zstd br gzip deflate]
+  config.smart_compress.cache = true                     # Enable in-memory LRU payload caching
+  config.smart_compress.cache_size = 200                 # Cache up to 200 warm response payloads
+  config.smart_compress.dynamic_levels = true            # Dynamically drop compression levels under high CPU load
+  config.smart_compress.if = ->(env, status, headers) { env["HTTP_X_NO_COMPRESS"].nil? }
+
+  # Optional: Enable pre-compressed static asset serving (.zst, .br, .gz)
+  config.smart_compress.static_assets = true
+  config.smart_compress.static_headers = { "Cache-Control" => "public, max-age=31536000, immutable" }
+end
+```
+
+### Global Configuration (Sinatra / Hanami / Pure Rack)
+
+```ruby
+# config.ru or boot.rb
+require "rack/smart_compress"
+
+Rack::SmartCompress.configure do |config|
+  config.min_size = 1024
+  config.cache = true
+  config.encodings = %w[zstd br gzip]
+end
+
+use Rack::SmartCompress::Middleware
+run YourRackApp.new
+```
+
+### Serving Pre-Compressed Static Assets (`.zst`, `.br`, `.gz`)
+
+If your asset pipeline (Vite Ruby, Propshaft, Webpacker, esbuild) pre-compresses static assets into `.zst`, `.br`, and `.gz` files:
+
+```ruby
+# config.ru
+use Rack::SmartCompress::Static,
+  root: "public",
+  urls: ["/assets", "/packs"],
+  headers: { "Cache-Control" => "public, max-age=31536000, immutable" }
+
+run YourRackApp.new
+```
+
+When a browser requests `/assets/bundle.js` with `Accept-Encoding: zstd, br, gzip`, `Rack::SmartCompress::Static` will:
+1. Detect `bundle.js.zst` on disk.
+2. Serve it directly via streaming IO with **0% runtime CPU usage**.
+3. Set `Content-Type: application/javascript`, `Content-Encoding: zstd`, and `Vary: Accept-Encoding`.
+
+---
+
+## ⚙️ Configuration Options
+
+| Option | Default | Description |
+| :--- | :--- | :--- |
+| `min_size` | `1024` (1 KB) | Minimum response body size in bytes required to trigger compression. |
+| `encodings` | `%w[zstd br gzip deflate]` | Priority order of compression algorithms to allow. |
+| `cache` | `false` | Enable thread-safe in-memory LRU payload caching for warm endpoints. |
+| `cache_size` | `200` | Maximum number of compressed payloads to cache in memory. |
+| `dynamic_levels` | `false` | Automatically drops compression levels under high CPU load to preserve low latency. |
+| `if` | `nil` | Proc/Lambda `->(env, status, headers)` returning true to compress. |
+| `unless` | `nil` | Proc/Lambda `->(env, status, headers)` returning true to skip compression. |
+| `mime_types` | `[text/*, application/json, ...]` | List of compressible MIME types. Supports `+json` and `+xml` suffixes. |
+| `exclude_mime_types` | `[image/*, video/*, audio/*, zip, pdf]` | List of MIME type prefixes to explicitly skip. |
+| `zstd_level` | `3` | Zstandard compression level (1-22). |
+| `brotli_level` | `4` | Brotli compression quality level (0-11). |
+| `instrumentation` | `true` | Emits `rack_smart_compress.compress` via `ActiveSupport::Notifications` or block callback. |
+| `static_assets` | `false` | Enables pre-compressed static asset serving in Rails. |
+| `static_root` | `"public"` | Root directory for static asset resolution. |
+| `static_urls` | `["/"]` | URL prefixes intercepted by static pre-compressed asset middleware. |
+
+---
+
+## 📊 Benchmark & Performance
+
+Run the built-in benchmark script to compare performance against your payloads:
+
+```bash
+$ bundle exec rake benchmark
+```
+
+**Sample Benchmark Output (108 KB JSON API payload):**
+
+```
+============================================================
+  Rack::SmartCompress Performance Benchmark  
+============================================================
+Algorithm    | Compressed Size | Compression % | Avg Speed (ms)
+------------------------------------------------------------
+Zstd         | 3.07 KB         | 97.16%        | 0.085 ms
+Brotli       | 2.66 KB         | 97.54%        | 0.355 ms
+Gzip         | 8.16 KB         | 92.45%        | 0.498 ms
+Deflate      | 8.15 KB         | 92.46%        | 0.478 ms
+============================================================
+```
+
+- **Zstandard (`zstd`)** delivers the fastest compression/decompression throughput (sub-0.1ms).
+- **Brotli (`br`)** delivers the smallest payload size over the wire.
+
+---
+
+## 🧪 Running Tests
+
+Run all unit and end-to-end integration tests:
+
+```bash
+$ bundle exec rake
+```
+
+Or individual suites:
+
+```bash
+$ bundle exec rspec       # RSpec unit tests (45 examples)
+$ bundle exec rake e2e    # Comprehensive End-to-End integration suite (18 tests)
+```
+
+---
+
+## 📄 License
+
+This gem is available as open source under the terms of the [MIT License](LICENSE.txt).

data/lib/rack/smart_compress/configuration.rb

@@ -0,0 +1,134 @@
+# frozen_string_literal: true
+
+require "zlib"
+
+module Rack
+  module SmartCompress
+    class Configuration
+      DEFAULT_MIN_SIZE = 1024 # 1 KB
+
+      DEFAULT_ENCODINGS = %w[zstd br gzip deflate].freeze
+
+      DEFAULT_MIME_TYPES = [
+        "text/html",
+        "text/plain",
+        "text/css",
+        "text/javascript",
+        "text/xml",
+        "application/json",
+        "application/javascript",
+        "application/x-javascript",
+        "application/xml",
+        "application/xhtml+xml",
+        "application/rss+xml",
+        "application/atom+xml",
+        "application/wasm",
+        "image/svg+xml"
+      ].freeze
+
+      DEFAULT_EXCLUDED_MIME_TYPES = [
+        "image/jpeg",
+        "image/png",
+        "image/gif",
+        "image/webp",
+        "image/avif",
+        "audio/",
+        "video/",
+        "application/zip",
+        "application/x-gzip",
+        "application/pdf"
+      ].freeze
+
+      attr_accessor :min_size,
+                    :encodings,
+                    :mime_types,
+                    :exclude_mime_types,
+                    :cache,
+                    :cache_size,
+                    :dynamic_levels,
+                    :zstd_level,
+                    :brotli_level,
+                    :gzip_level,
+                    :deflate_level,
+                    :if_condition,
+                    :unless_condition,
+                    :instrumentation,
+                    :static_assets,
+                    :static_root,
+                    :static_urls,
+                    :static_headers,
+                    :static_cascade
+
+      def initialize
+        @min_size = DEFAULT_MIN_SIZE
+        @encodings = DEFAULT_ENCODINGS.dup
+        @mime_types = DEFAULT_MIME_TYPES.dup
+        @exclude_mime_types = DEFAULT_EXCLUDED_MIME_TYPES.dup
+        @cache = false
+        @cache_size = 200
+        @dynamic_levels = false
+        @zstd_level = 3
+        @brotli_level = 4
+        @gzip_level = Zlib::DEFAULT_COMPRESSION
+        @deflate_level = Zlib::DEFAULT_COMPRESSION
+        @if_condition = nil
+        @unless_condition = nil
+        @instrumentation = true
+        @static_assets = false
+        @static_root = "public"
+        @static_urls = ["/"]
+        @static_headers = {}
+        @static_cascade = true
+      end
+
+      # Convenience aliases for :if and :unless
+      def if(&block)
+        if block_given?
+          @if_condition = block
+        else
+          @if_condition
+        end
+      end
+
+      def if=(val)
+        @if_condition = val
+      end
+
+      def unless(&block)
+        if block_given?
+          @unless_condition = block
+        else
+          @unless_condition
+        end
+      end
+
+      def unless=(val)
+        @unless_condition = val
+      end
+
+      def to_h
+        {
+          min_size: @min_size,
+          encodings: @encodings,
+          mime_types: @mime_types,
+          exclude_mime_types: @exclude_mime_types,
+          cache: @cache,
+          cache_size: @cache_size,
+          dynamic_levels: @dynamic_levels,
+          zstd_level: @zstd_level,
+          brotli_level: @brotli_level,
+          gzip_level: @gzip_level,
+          deflate_level: @deflate_level,
+          if: @if_condition,
+          unless: @unless_condition,
+          instrumentation: @instrumentation,
+          static_assets: @static_assets,
+          static_root: @static_root,
+          static_urls: @static_urls,
+          static_headers: @static_headers,
+          static_cascade: @static_cascade
+        }
+      end
+    end
+  end
+end

data/lib/rack/smart_compress/cpu_advisor.rb

@@ -0,0 +1,74 @@
+# frozen_string_literal: true
+
+require "etc"
+require "zlib"
+
+module Rack
+  module SmartCompress
+    class CpuAdvisor
+      SAMPLE_INTERVAL = 1.0 # Sample at most once per second
+
+      class << self
+        def high_load?
+          now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+          if @last_check.nil? || (now - @last_check) >= SAMPLE_INTERVAL
+            @last_check = now
+            @cached_high_load = compute_high_load
+          end
+          @cached_high_load
+        end
+
+        def ncpu_count
+          @ncpu_count ||= begin
+            Etc.nprocessors
+          rescue StandardError
+            2
+          end
+        end
+
+        def adjusted_level(encoder_name, default_level)
+          return default_level unless high_load?
+
+          case encoder_name
+          when "zstd"
+            1 # Fastest Zstd level
+          when "br"
+            1 # Fastest Brotli quality level
+          when "gzip", "deflate"
+            Zlib::BEST_SPEED # Fastest Gzip level (1)
+          else
+            default_level
+          end
+        end
+
+        def reset!
+          @last_check = nil
+          @cached_high_load = nil
+          @ncpu_count = nil
+        end
+
+        private
+
+        def compute_high_load
+          load_1min = current_load_avg
+          return false if load_1min.nil?
+
+          ncpu = ncpu_count
+          load_1min > (ncpu * 0.85) # High load threshold (> 85% capacity)
+        rescue StandardError
+          false
+        end
+
+        def current_load_avg
+          if File.exist?("/proc/loadavg")
+            File.read("/proc/loadavg").split.first.to_f
+          else
+            nil
+          end
+        rescue StandardError
+          nil
+        end
+      end
+    end
+  end
+end

data/lib/rack/smart_compress/encoders/base.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module Rack
+  module SmartCompress
+    module Encoders
+      class Base
+        class << self
+          def available?
+            true
+          end
+
+          def compress(body, level: nil)
+            raise NotImplementedError, "#{name}.compress must be implemented"
+          end
+
+          def encode_body(body, level: nil)
+            compressed = []
+            if body.respond_to?(:each)
+              buffer = String.new
+              body.each { |chunk| buffer << chunk.to_s }
+              compressed << compress(buffer, level: level)
+            else
+              compressed << compress(body.to_s, level: level)
+            end
+            compressed
+          end
+        end
+      end
+    end
+  end
+end

data/lib/rack/smart_compress/encoders/brotli.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+require_relative "base"
+
+module Rack
+  module SmartCompress
+    module Encoders
+      class Brotli < Base
+        ENCODING_NAME = "br"
+
+        class << self
+          def available?
+            return @available if defined?(@available)
+
+            @available = begin
+              require "brotli"
+              true
+            rescue LoadError
+              false
+            end
+          end
+
+          def compress(content, level: nil)
+            raise LoadError, "brotli gem is not available" unless available?
+
+            compression_level = level || 4
+            ::Brotli.deflate(content, quality: compression_level)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/rack/smart_compress/encoders/deflate.rb

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+require "zlib"
+require_relative "base"
+
+module Rack
+  module SmartCompress
+    module Encoders
+      class Deflate < Base
+        ENCODING_NAME = "deflate"
+
+        class << self
+          def available?
+            true
+          end
+
+          def compress(content, level: nil)
+            compression_level = level || Zlib::DEFAULT_COMPRESSION
+            Zlib::Deflate.deflate(content, compression_level)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/rack/smart_compress/encoders/gzip.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+require "zlib"
+require "stringio"
+require_relative "base"
+
+module Rack
+  module SmartCompress
+    module Encoders
+      class Gzip < Base
+        ENCODING_NAME = "gzip"
+
+        class << self
+          def available?
+            true
+          end
+
+          def compress(content, level: nil)
+            compression_level = level || Zlib::DEFAULT_COMPRESSION
+            io = StringIO.new
+            io.set_encoding(Encoding::BINARY)
+            writer = Zlib::GzipWriter.new(io, compression_level)
+            writer.write(content)
+            writer.close
+            io.string
+          end
+        end
+      end
+    end
+  end
+end

data/lib/rack/smart_compress/encoders/zstd.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+require_relative "base"
+
+module Rack
+  module SmartCompress
+    module Encoders
+      class Zstd < Base
+        ENCODING_NAME = "zstd"
+
+        class << self
+          def available?
+            return @available if defined?(@available)
+
+            @available = begin
+              require "zstd-ruby"
+              true
+            rescue LoadError
+              false
+            end
+          end
+
+          def compress(content, level: nil)
+            raise LoadError, "zstd-ruby gem is not available" unless available?
+
+            compression_level = level || 3
+            ::Zstd.compress(content, level: compression_level)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/rack/smart_compress/lru_cache.rb

@@ -0,0 +1,87 @@
+# frozen_string_literal: true
+
+require "digest"
+require "monitor"
+
+module Rack
+  module SmartCompress
+    class LruCache
+      include MonitorMixin
+
+      DEFAULT_MAX_SIZE = 200
+
+      attr_reader :max_size, :hits, :misses
+
+      def initialize(max_size = DEFAULT_MAX_SIZE)
+        super()
+        @max_size = max_size
+        @cache = {}
+        @hits = 0
+        @misses = 0
+      end
+
+      def get(key)
+        mon_synchronize do
+          if @cache.key?(key)
+            @hits += 1
+            value = @cache.delete(key)
+            @cache[key] = value # Move to end (MRU)
+            value
+          else
+            @misses += 1
+            nil
+          end
+        end
+      end
+
+      def fetch(key)
+        cached = get(key)
+        return cached unless cached.nil?
+        return nil unless block_given?
+
+        # Execute expensive compression outside the mutex lock
+        value = yield
+        put(key, value)
+        value
+      end
+
+      def put(key, value)
+        mon_synchronize do
+          @cache.delete(key)
+          @cache[key] = value
+
+          if @cache.size > @max_size
+            @cache.shift # Remove oldest (LRU)
+          end
+          value
+        end
+      end
+
+      def delete(key)
+        mon_synchronize { @cache.delete(key) }
+      end
+
+      def clear
+        mon_synchronize do
+          @cache.clear
+          @hits = 0
+          @misses = 0
+        end
+      end
+
+      def size
+        mon_synchronize { @cache.size }
+      end
+
+      def build_key(encoder_name, level, content)
+        digest = Digest::SHA256.new
+        digest.update(encoder_name.to_s)
+        digest.update(":")
+        digest.update(level.to_s)
+        digest.update(":")
+        digest.update(content.to_s)
+        digest.hexdigest
+      end
+    end
+  end
+end