io_monitor 0.1.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 55da4008002fc3ddd2846fda47f10c76910a29b370754f02772e33635c73d156
-  data.tar.gz: be06ae7bf31a7051a7e0e113cf16f1c2a55300bf4519d083d400e948e2b4a760
+  metadata.gz: 1ee7933c9976ebde6bcd8a84dd206a4020cd297edc96ad762ed9345b4a912a82
+  data.tar.gz: e17935f41005f34bda26cbc6021fa77b337ec876f0b3440a45a4b0085376a0fd
 SHA512:
-  metadata.gz: 4120f7fad9b19fe8ef2be71a5c14fd54fca42141ed205cdd468086956b4232bb401fca94d47c906fa2fc371df3288f16b6b46ee2a520ef023d9fdf9fbe61a7d2
-  data.tar.gz: 5eab6fe8f0e5eb4072d8c9e092259896cadb8034dbf4baf8ec99460febbc0fe3bb592f7e0f120e14fdf6f1ed0e4f2684df54b2cd27830281b37eabe36c31a3d4
+  metadata.gz: 96ed1d20af1c8a20a101f04d389a9d9d5922c5bd9757a0460913f36347ba4a82c15abec0b6f0128079df44d3296d94d1e3e405a464883ae72a503d36c086e7dd
+  data.tar.gz: 4765a09692f9788c2fb394be13f161b9bce18a753bd3fa327874e00dcfc359b7ffd0d79f424dbbd6cb0d4df51e2e996f5cc48bf0107de8c2f498354264bc300f

data/CHANGELOG.md

@@ -2,6 +2,18 @@
 
 ## main
 
+## 1.0.0 (2023-05-06)
+
+- [PR#22](https://github.com/DmitryTsepelev/io_monitor/pull/22) Handle zero payload ([@DmitryTsepelev])
+- [PR#17](https://github.com/DmitryTsepelev/io_monitor/pull/17) Prometheus publisher ([@maxshend])
+- [PR#10](https://github.com/DmitryTsepelev/io_monitor/pull/10) Per–action monitoring ([@DmitryTsepelev])
+- [PR#15](https://github.com/DmitryTsepelev/io_monitor/pull/15) Allow configure more than one publisher ([@DmitryTsepelev])
+- [PR#9](https://github.com/DmitryTsepelev/io_monitor/pull/9) Restrict minimum Rails version to 6.1, adjust test matrix, and related changes ([@Envek])
+
+## 0.2.0 (2022-05-29)
+
+- [PR#8](https://github.com/DmitryTsepelev/io_monitor/pull/8) Add Redis adapter ([@DmitryTsepelev])
+
 ## 0.1.0 (2022-05-24)
 
 - [PR#7](https://github.com/DmitryTsepelev/io_monitor/pull/7) Add HTTP adapter ([@maxshend])
@@ -12,3 +24,5 @@
 [@baygeldin]: https://github.com/baygeldin
 [@prog-supdex]: https://github.com/prog-supdex
 [@maxshend]: https://github.com/maxshend
+[@DmitryTsepelev]: https://github.com/DmitryTsepelev
+[@Envek]: https://github.com/Envek

data/README.md

@@ -1,5 +1,9 @@
 # IoMonitor
 
+[![Gem Version](https://badge.fury.io/rb/io_monitor.svg)](https://rubygems.org/gems/io_monitor)
+[![Tests status](https://github.com/DmitryTsepelev/io_monitor/actions/workflows/test.yml/badge.svg)](https://github.com/DmitryTsepelev/io_monitor/actions/workflows/test.yml)
+![](https://ruby-gem-downloads-badge.herokuapp.com/io_monitor?type=total)
+
 A gem that helps to detect potential memory bloats.
 
 When your controller loads a lot of data to the memory but returns a small response to the client it might mean that you're using the IO in the non–optimal way. In this case, you'll see the following message in your logs:
@@ -8,12 +12,6 @@ When your controller loads a lot of data to the memory but returns a small respo
 Completed 200 OK in 349ms (Views: 2.1ms | ActiveRecord: 38.7ms | ActiveRecord Payload: 866.00 B | Response Payload: 25.00 B | Allocations: 72304)
 ```
 
-<p align="center">
-  <a href="https://evilmartians.com/?utm_source=io_monitor">
-    <img src="https://evilmartians.com/badges/sponsored-by-evil-martians.svg" alt="Sponsored by Evil Martians" width="236" height="54">
-  </a>
-</p>
-
 ## Usage
 
 Add this line to your application's Gemfile:
@@ -22,13 +20,15 @@ Add this line to your application's Gemfile:
 gem 'io_monitor'
 ```
 
+Currently gem can collect the data from `ActiveRecord`, `Net::HTTP` and `Redis`.
+
 Change configuration in an initializer if you need:
 
 ```ruby
 IoMonitor.configure do |config|
-  config.publish = :notifications # defaults to :logs
+  config.publish = [:logs, :notifications, :prometheus] # defaults to :logs
   config.warn_threshold = 0.8 # defaults to 0
-  config.adapters = [:active_record, :net_http] # defaults to [:active_record]
+  config.adapters = [:active_record, :net_http, :redis] # defaults to [:active_record]
 end
 ```
 
@@ -45,6 +45,31 @@ Depending on configuration when IO payload size to response payload size ratio r
 ```
 ActiveRecord I/O to response payload ratio is 0.1, while threshold is 0.8
 ```
+Prometheus metrics example:
+```
+...
+# TYPE io_monitor_ratio histogram
+# HELP io_monitor_ratio IO payload size to response payload size ratio
+io_monitor_ratio_bucket{adapter="active_record",le="0.01"} 0.0
+io_monitor_ratio_bucket{adapter="active_record",le="5"} 2.0
+io_monitor_ratio_bucket{adapter="active_record",le="10"} 2.0
+io_monitor_ratio_bucket{adapter="active_record",le="+Inf"} 2.0
+io_monitor_ratio_sum{adapter="active_record"} 0.15779381908414167
+io_monitor_ratio_count{adapter="active_record"} 2.0
+...
+```
+If you want to customize Prometheus publisher you can pass it as object:
+```ruby
+IoMonitor.configure do |config|
+  config.publish = [
+    IoMonitor::PrometheusPublisher.new(
+      registry: custom_registry, # defaults to Prometheus::Client.registry
+      aggregation: :max, # defaults to nil
+      buckets: [0.1, 5, 10] # defaults to Prometheus::Client::Histogram::DEFAULT_BUCKETS
+    )
+  ]
+end
+```
 
 In addition, if `publish` is set to logs, additional data will be logged on each request:
 
@@ -60,6 +85,18 @@ ActiveSupport::Notifications.subscribe("process_action.action_controller") do |n
 end
 ```
 
+## Per–action monitoring
+
+Since this approach can lead to false–positives or other things you don't want or cannot fix, there is a way to configure monitoring only for specific actions:
+
+```ruby
+class MyController < ApplicationController
+  include IoMonitor::Controller
+
+  monitor_io_for :index, :show
+end
+```
+
 ## Custom publishers
 
 Implement your custom publisher by inheriting from `BasePublisher`:
@@ -108,6 +145,10 @@ end
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests.
 
+## Credits
+
+Initially sponsored by [Evil Martians](http://evilmartians.com).
+
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/DmitryTsepelev/io_monitor.

data/lib/io_monitor/adapters/active_record_adapter.rb

@@ -17,7 +17,7 @@ module IoMonitor
         # but it makes a lot of unnecessary allocations.
         io_payload_size = rows.sum(0) do |row|
           row.sum(0) do |val|
-            (String === val ? val : val.to_s).bytesize
+            ((String === val) ? val : val.to_s).bytesize
           end
         end
 

data/lib/io_monitor/adapters/redis_adapter.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+require "io_monitor/patches/redis_patch"
+
+module IoMonitor
+  class RedisAdapter < BaseAdapter
+    def self.kind
+      :redis
+    end
+
+    def initialize!
+      ActiveSupport.on_load(:after_initialize) do
+        Redis.prepend(RedisPatch)
+      end
+    end
+  end
+end

data/lib/io_monitor/aggregator.rb

@@ -13,6 +13,12 @@ module IoMonitor
       InputPayload.active.present?
     end
 
+    def collect
+      start!
+      yield
+      stop!
+    end
+
     def start!
       InputPayload.active = true
     end

data/lib/io_monitor/configuration.rb

@@ -5,22 +5,15 @@ module IoMonitor
     DEFAULT_WARN_THRESHOLD = 0.0
 
     def initialize
-      @publisher = LogsPublisher.new
+      @publishers = [LogsPublisher.new]
       @adapters = [ActiveRecordAdapter.new]
       @warn_threshold = DEFAULT_WARN_THRESHOLD
     end
 
-    attr_reader :publisher, :adapters, :warn_threshold
+    attr_reader :publishers, :adapters, :warn_threshold
 
-    def publish=(value)
-      if value.is_a?(BasePublisher)
-        @publisher = value
-      elsif (publisher_type = resolve(IoMonitor::PUBLISHERS, value))
-        @publisher = publisher_type.new
-      else
-        supported = IoMonitor::PUBLISHERS.map(&:kind)
-        raise ArgumentError, "Only the following publishers are supported: #{supported}."
-      end
+    def publish=(values)
+      @publishers = [*values].map { |value| value_to_publisher(value) }
     end
 
     def adapters=(value)
@@ -50,5 +43,16 @@ module IoMonitor
     def resolve(list, kind)
       list.find { |p| p.kind == kind }
     end
+
+    def value_to_publisher(value)
+      if value.is_a?(BasePublisher)
+        value
+      elsif (publisher_type = resolve(IoMonitor::PUBLISHERS, value))
+        publisher_type.new
+      else
+        supported = IoMonitor::PUBLISHERS.map(&:kind)
+        raise ArgumentError, "Only the following publishers are supported: #{supported}."
+      end
+    end
   end
 end

data/lib/io_monitor/controller.rb

@@ -4,24 +4,47 @@ module IoMonitor
   module Controller
     extend ActiveSupport::Concern
 
-    def process_action(*)
-      IoMonitor.aggregator.start!
+    delegate :aggregator, to: IoMonitor
 
-      super
+    ALL_ACTIONS = Object.new
+
+    class_methods do
+      def monitor_io_for(*actions_to_monitor_io)
+        @actions_to_monitor_io = actions_to_monitor_io
+      end
 
-      IoMonitor.aggregator.stop!
+      def actions_to_monitor_io
+        @actions_to_monitor_io || ALL_ACTIONS
+      end
+    end
+
+    def process_action(*)
+      if monitors_action?(action_name)
+        aggregator.collect { super }
+      else
+        super
+      end
     end
 
     def append_info_to_payload(payload)
       super
 
+      return unless monitors_action?(action_name)
+
       data = payload[IoMonitor::NAMESPACE] = {}
 
-      IoMonitor.aggregator.sources.each do |source|
-        data[source] = IoMonitor.aggregator.get(source)
+      aggregator.sources.each do |source|
+        data[source] = aggregator.get(source)
       end
 
-      data[:response] = payload[:response]&.body&.bytesize || 0
+      data[:response] = payload[:response].body.bytesize
+    end
+
+    private
+
+    def monitors_action?(action_name)
+      actions = self.class.actions_to_monitor_io
+      actions == ALL_ACTIONS || actions.include?(action_name.to_sym)
     end
   end
 end

data/lib/io_monitor/patches/action_controller_base_patch.rb

@@ -4,7 +4,7 @@ module IoMonitor
   module ActionControllerBasePatch
     def log_process_action(payload)
       super.tap do |messages|
-        next unless IoMonitor.config.publisher.is_a?(LogsPublisher)
+        next unless IoMonitor.config.publishers.any? { |publisher| publisher.is_a?(LogsPublisher) }
 
         data = payload[IoMonitor::NAMESPACE]
         next unless data

data/lib/io_monitor/patches/redis_patch.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+module IoMonitor
+  module RedisPatch
+    def send_command(command, &block)
+      super(command, &block).tap do |reply|
+        # we need to check QUEUED because of https://github.com/redis/redis-rb/blob/cbdb53e8c2f0be53c91404cb7ff566a36fc8ebf5/lib/redis/client.rb#L164
+        if reply != "QUEUED" && !reply.is_a?(Redis::CommandError) && IoMonitor.aggregator.active?
+          IoMonitor.aggregator.increment(RedisAdapter.kind, reply.bytesize)
+        end
+      end
+    end
+  end
+end

data/lib/io_monitor/publishers/base_publisher.rb

@@ -16,15 +16,15 @@ module IoMonitor
       (payload.keys - [:response]).each do |source|
         ratio = ratio(payload[:response], payload[source])
 
-        if ratio < IoMonitor.config.warn_threshold
-          publish(source, ratio)
-        end
+        publish(source, ratio) if ratio < IoMonitor.config.warn_threshold
       end
     end
 
     private
 
     def ratio(response_size, io_size)
+      return 0 if io_size.to_f.zero?
+
       response_size.to_f / io_size.to_f
     end
   end

data/lib/io_monitor/publishers/prometheus_publisher.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+module IoMonitor
+  class PrometheusPublisher < BasePublisher
+    HELP_MESSAGE = "IO payload size to response payload size ratio"
+
+    def initialize(registry: nil, aggregation: nil, buckets: nil)
+      registry ||= ::Prometheus::Client.registry
+      @metric = registry.histogram(
+        "#{IoMonitor::NAMESPACE}_ratio".to_sym,
+        labels: %i[adapter],
+        buckets: buckets || ::Prometheus::Client::Histogram::DEFAULT_BUCKETS,
+        store_settings: store_settings(aggregation),
+        docstring: HELP_MESSAGE
+      )
+    end
+
+    def self.kind
+      :prometheus
+    end
+
+    def publish(source, ratio)
+      metric.observe(ratio, labels: {adapter: source})
+    end
+
+    private
+
+    attr_reader :metric
+
+    # From https://github.com/yabeda-rb/yabeda-prometheus/blob/v0.8.0/lib/yabeda/prometheus/adapter.rb#L101
+    def store_settings(aggregation)
+      case ::Prometheus::Client.config.data_store
+      when ::Prometheus::Client::DataStores::Synchronized, ::Prometheus::Client::DataStores::SingleThreaded
+        {} # Default synchronized store doesn't allow to pass any options
+      when ::Prometheus::Client::DataStores::DirectFileStore, ::Object # Anything else
+        {aggregation: aggregation}.compact
+      end
+    end
+  end
+end

data/lib/io_monitor/railtie.rb

@@ -15,7 +15,7 @@ module IoMonitor
         payload = args.last[IoMonitor::NAMESPACE]
         next unless payload
 
-        IoMonitor.config.publisher.process_action(payload)
+        IoMonitor.config.publishers.each { |publisher| publisher.process_action(payload) }
       end
     end
   end

data/lib/io_monitor/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module IoMonitor
-  VERSION = "0.1.0"
+  VERSION = "1.0.0"
 end

data/lib/io_monitor.rb

@@ -12,13 +12,22 @@ require "io_monitor/adapters/net_http_adapter"
 require "io_monitor/publishers/base_publisher"
 require "io_monitor/publishers/logs_publisher"
 require "io_monitor/publishers/notifications_publisher"
+require "io_monitor/publishers/prometheus_publisher"
 
 require "io_monitor/railtie"
 
 module IoMonitor
   NAMESPACE = :io_monitor
-  ADAPTERS = [ActiveRecordAdapter, NetHttpAdapter].freeze
-  PUBLISHERS = [LogsPublisher, NotificationsPublisher].freeze
+
+  adapters = [ActiveRecordAdapter, NetHttpAdapter]
+
+  if defined? Redis
+    require "io_monitor/adapters/redis_adapter"
+    adapters << RedisAdapter
+  end
+  ADAPTERS = adapters.freeze
+
+  PUBLISHERS = [LogsPublisher, NotificationsPublisher, PrometheusPublisher].freeze
 
   class << self
     def aggregator

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: io_monitor
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 1.0.0
 platform: ruby
 authors:
 - baygeldin
@@ -11,7 +11,7 @@ authors:
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2022-05-23 00:00:00.000000000 Z
+date: 2023-05-06 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rails
@@ -19,14 +19,42 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '6.0'
+        version: '6.1'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '6.0'
+        version: '6.1'
+- !ruby/object:Gem::Dependency
+  name: redis
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '4.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '4.0'
+- !ruby/object:Gem::Dependency
+  name: prometheus-client
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description:
 email:
 - dmitry.a.tsepelev@gmail.com
@@ -41,6 +69,7 @@ files:
 - lib/io_monitor/adapters/active_record_adapter.rb
 - lib/io_monitor/adapters/base_adapter.rb
 - lib/io_monitor/adapters/net_http_adapter.rb
+- lib/io_monitor/adapters/redis_adapter.rb
 - lib/io_monitor/aggregator.rb
 - lib/io_monitor/configuration.rb
 - lib/io_monitor/controller.rb
@@ -48,9 +77,11 @@ files:
 - lib/io_monitor/patches/action_controller_base_patch.rb
 - lib/io_monitor/patches/future_result_patch.rb
 - lib/io_monitor/patches/net_http_adapter_patch.rb
+- lib/io_monitor/patches/redis_patch.rb
 - lib/io_monitor/publishers/base_publisher.rb
 - lib/io_monitor/publishers/logs_publisher.rb
 - lib/io_monitor/publishers/notifications_publisher.rb
+- lib/io_monitor/publishers/prometheus_publisher.rb
 - lib/io_monitor/railtie.rb
 - lib/io_monitor/version.rb
 homepage: https://github.com/DmitryTsepelev/io_monitor
@@ -70,14 +101,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: 2.6.0
+      version: 2.7.0
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.2.33
+rubygems_version: 3.2.15
 signing_key:
 specification_version: 4
 summary: A gem that helps to detect potential memory bloats