io_monitor 0.1.0 → 1.0.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 55da4008002fc3ddd2846fda47f10c76910a29b370754f02772e33635c73d156
4
- data.tar.gz: be06ae7bf31a7051a7e0e113cf16f1c2a55300bf4519d083d400e948e2b4a760
3
+ metadata.gz: 1ee7933c9976ebde6bcd8a84dd206a4020cd297edc96ad762ed9345b4a912a82
4
+ data.tar.gz: e17935f41005f34bda26cbc6021fa77b337ec876f0b3440a45a4b0085376a0fd
5
5
  SHA512:
6
- metadata.gz: 4120f7fad9b19fe8ef2be71a5c14fd54fca42141ed205cdd468086956b4232bb401fca94d47c906fa2fc371df3288f16b6b46ee2a520ef023d9fdf9fbe61a7d2
7
- data.tar.gz: 5eab6fe8f0e5eb4072d8c9e092259896cadb8034dbf4baf8ec99460febbc0fe3bb592f7e0f120e14fdf6f1ed0e4f2684df54b2cd27830281b37eabe36c31a3d4
6
+ metadata.gz: 96ed1d20af1c8a20a101f04d389a9d9d5922c5bd9757a0460913f36347ba4a82c15abec0b6f0128079df44d3296d94d1e3e405a464883ae72a503d36c086e7dd
7
+ data.tar.gz: 4765a09692f9788c2fb394be13f161b9bce18a753bd3fa327874e00dcfc359b7ffd0d79f424dbbd6cb0d4df51e2e996f5cc48bf0107de8c2f498354264bc300f
data/CHANGELOG.md CHANGED
@@ -2,6 +2,18 @@
2
2
 
3
3
  ## main
4
4
 
5
+ ## 1.0.0 (2023-05-06)
6
+
7
+ - [PR#22](https://github.com/DmitryTsepelev/io_monitor/pull/22) Handle zero payload ([@DmitryTsepelev])
8
+ - [PR#17](https://github.com/DmitryTsepelev/io_monitor/pull/17) Prometheus publisher ([@maxshend])
9
+ - [PR#10](https://github.com/DmitryTsepelev/io_monitor/pull/10) Per–action monitoring ([@DmitryTsepelev])
10
+ - [PR#15](https://github.com/DmitryTsepelev/io_monitor/pull/15) Allow configure more than one publisher ([@DmitryTsepelev])
11
+ - [PR#9](https://github.com/DmitryTsepelev/io_monitor/pull/9) Restrict minimum Rails version to 6.1, adjust test matrix, and related changes ([@Envek])
12
+
13
+ ## 0.2.0 (2022-05-29)
14
+
15
+ - [PR#8](https://github.com/DmitryTsepelev/io_monitor/pull/8) Add Redis adapter ([@DmitryTsepelev])
16
+
5
17
  ## 0.1.0 (2022-05-24)
6
18
 
7
19
  - [PR#7](https://github.com/DmitryTsepelev/io_monitor/pull/7) Add HTTP adapter ([@maxshend])
@@ -12,3 +24,5 @@
12
24
  [@baygeldin]: https://github.com/baygeldin
13
25
  [@prog-supdex]: https://github.com/prog-supdex
14
26
  [@maxshend]: https://github.com/maxshend
27
+ [@DmitryTsepelev]: https://github.com/DmitryTsepelev
28
+ [@Envek]: https://github.com/Envek
data/README.md CHANGED
@@ -1,5 +1,9 @@
1
1
  # IoMonitor
2
2
 
3
+ [![Gem Version](https://badge.fury.io/rb/io_monitor.svg)](https://rubygems.org/gems/io_monitor)
4
+ [![Tests status](https://github.com/DmitryTsepelev/io_monitor/actions/workflows/test.yml/badge.svg)](https://github.com/DmitryTsepelev/io_monitor/actions/workflows/test.yml)
5
+ ![](https://ruby-gem-downloads-badge.herokuapp.com/io_monitor?type=total)
6
+
3
7
  A gem that helps to detect potential memory bloats.
4
8
 
5
9
  When your controller loads a lot of data to the memory but returns a small response to the client it might mean that you're using the IO in the non–optimal way. In this case, you'll see the following message in your logs:
@@ -8,12 +12,6 @@ When your controller loads a lot of data to the memory but returns a small respo
8
12
  Completed 200 OK in 349ms (Views: 2.1ms | ActiveRecord: 38.7ms | ActiveRecord Payload: 866.00 B | Response Payload: 25.00 B | Allocations: 72304)
9
13
  ```
10
14
 
11
- <p align="center">
12
- <a href="https://evilmartians.com/?utm_source=io_monitor">
13
- <img src="https://evilmartians.com/badges/sponsored-by-evil-martians.svg" alt="Sponsored by Evil Martians" width="236" height="54">
14
- </a>
15
- </p>
16
-
17
15
  ## Usage
18
16
 
19
17
  Add this line to your application's Gemfile:
@@ -22,13 +20,15 @@ Add this line to your application's Gemfile:
22
20
  gem 'io_monitor'
23
21
  ```
24
22
 
23
+ Currently gem can collect the data from `ActiveRecord`, `Net::HTTP` and `Redis`.
24
+
25
25
  Change configuration in an initializer if you need:
26
26
 
27
27
  ```ruby
28
28
  IoMonitor.configure do |config|
29
- config.publish = :notifications # defaults to :logs
29
+ config.publish = [:logs, :notifications, :prometheus] # defaults to :logs
30
30
  config.warn_threshold = 0.8 # defaults to 0
31
- config.adapters = [:active_record, :net_http] # defaults to [:active_record]
31
+ config.adapters = [:active_record, :net_http, :redis] # defaults to [:active_record]
32
32
  end
33
33
  ```
34
34
 
@@ -45,6 +45,31 @@ Depending on configuration when IO payload size to response payload size ratio r
45
45
  ```
46
46
  ActiveRecord I/O to response payload ratio is 0.1, while threshold is 0.8
47
47
  ```
48
+ Prometheus metrics example:
49
+ ```
50
+ ...
51
+ # TYPE io_monitor_ratio histogram
52
+ # HELP io_monitor_ratio IO payload size to response payload size ratio
53
+ io_monitor_ratio_bucket{adapter="active_record",le="0.01"} 0.0
54
+ io_monitor_ratio_bucket{adapter="active_record",le="5"} 2.0
55
+ io_monitor_ratio_bucket{adapter="active_record",le="10"} 2.0
56
+ io_monitor_ratio_bucket{adapter="active_record",le="+Inf"} 2.0
57
+ io_monitor_ratio_sum{adapter="active_record"} 0.15779381908414167
58
+ io_monitor_ratio_count{adapter="active_record"} 2.0
59
+ ...
60
+ ```
61
+ If you want to customize Prometheus publisher you can pass it as object:
62
+ ```ruby
63
+ IoMonitor.configure do |config|
64
+ config.publish = [
65
+ IoMonitor::PrometheusPublisher.new(
66
+ registry: custom_registry, # defaults to Prometheus::Client.registry
67
+ aggregation: :max, # defaults to nil
68
+ buckets: [0.1, 5, 10] # defaults to Prometheus::Client::Histogram::DEFAULT_BUCKETS
69
+ )
70
+ ]
71
+ end
72
+ ```
48
73
 
49
74
  In addition, if `publish` is set to logs, additional data will be logged on each request:
50
75
 
@@ -60,6 +85,18 @@ ActiveSupport::Notifications.subscribe("process_action.action_controller") do |n
60
85
  end
61
86
  ```
62
87
 
88
+ ## Per–action monitoring
89
+
90
+ Since this approach can lead to false–positives or other things you don't want or cannot fix, there is a way to configure monitoring only for specific actions:
91
+
92
+ ```ruby
93
+ class MyController < ApplicationController
94
+ include IoMonitor::Controller
95
+
96
+ monitor_io_for :index, :show
97
+ end
98
+ ```
99
+
63
100
  ## Custom publishers
64
101
 
65
102
  Implement your custom publisher by inheriting from `BasePublisher`:
@@ -108,6 +145,10 @@ end
108
145
 
109
146
  After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests.
110
147
 
148
+ ## Credits
149
+
150
+ Initially sponsored by [Evil Martians](http://evilmartians.com).
151
+
111
152
  ## Contributing
112
153
 
113
154
  Bug reports and pull requests are welcome on GitHub at https://github.com/DmitryTsepelev/io_monitor.
@@ -17,7 +17,7 @@ module IoMonitor
17
17
  # but it makes a lot of unnecessary allocations.
18
18
  io_payload_size = rows.sum(0) do |row|
19
19
  row.sum(0) do |val|
20
- (String === val ? val : val.to_s).bytesize
20
+ ((String === val) ? val : val.to_s).bytesize
21
21
  end
22
22
  end
23
23
 
@@ -0,0 +1,17 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "io_monitor/patches/redis_patch"
4
+
5
+ module IoMonitor
6
+ class RedisAdapter < BaseAdapter
7
+ def self.kind
8
+ :redis
9
+ end
10
+
11
+ def initialize!
12
+ ActiveSupport.on_load(:after_initialize) do
13
+ Redis.prepend(RedisPatch)
14
+ end
15
+ end
16
+ end
17
+ end
@@ -13,6 +13,12 @@ module IoMonitor
13
13
  InputPayload.active.present?
14
14
  end
15
15
 
16
+ def collect
17
+ start!
18
+ yield
19
+ stop!
20
+ end
21
+
16
22
  def start!
17
23
  InputPayload.active = true
18
24
  end
@@ -5,22 +5,15 @@ module IoMonitor
5
5
  DEFAULT_WARN_THRESHOLD = 0.0
6
6
 
7
7
  def initialize
8
- @publisher = LogsPublisher.new
8
+ @publishers = [LogsPublisher.new]
9
9
  @adapters = [ActiveRecordAdapter.new]
10
10
  @warn_threshold = DEFAULT_WARN_THRESHOLD
11
11
  end
12
12
 
13
- attr_reader :publisher, :adapters, :warn_threshold
13
+ attr_reader :publishers, :adapters, :warn_threshold
14
14
 
15
- def publish=(value)
16
- if value.is_a?(BasePublisher)
17
- @publisher = value
18
- elsif (publisher_type = resolve(IoMonitor::PUBLISHERS, value))
19
- @publisher = publisher_type.new
20
- else
21
- supported = IoMonitor::PUBLISHERS.map(&:kind)
22
- raise ArgumentError, "Only the following publishers are supported: #{supported}."
23
- end
15
+ def publish=(values)
16
+ @publishers = [*values].map { |value| value_to_publisher(value) }
24
17
  end
25
18
 
26
19
  def adapters=(value)
@@ -50,5 +43,16 @@ module IoMonitor
50
43
  def resolve(list, kind)
51
44
  list.find { |p| p.kind == kind }
52
45
  end
46
+
47
+ def value_to_publisher(value)
48
+ if value.is_a?(BasePublisher)
49
+ value
50
+ elsif (publisher_type = resolve(IoMonitor::PUBLISHERS, value))
51
+ publisher_type.new
52
+ else
53
+ supported = IoMonitor::PUBLISHERS.map(&:kind)
54
+ raise ArgumentError, "Only the following publishers are supported: #{supported}."
55
+ end
56
+ end
53
57
  end
54
58
  end
@@ -4,24 +4,47 @@ module IoMonitor
4
4
  module Controller
5
5
  extend ActiveSupport::Concern
6
6
 
7
- def process_action(*)
8
- IoMonitor.aggregator.start!
7
+ delegate :aggregator, to: IoMonitor
9
8
 
10
- super
9
+ ALL_ACTIONS = Object.new
10
+
11
+ class_methods do
12
+ def monitor_io_for(*actions_to_monitor_io)
13
+ @actions_to_monitor_io = actions_to_monitor_io
14
+ end
11
15
 
12
- IoMonitor.aggregator.stop!
16
+ def actions_to_monitor_io
17
+ @actions_to_monitor_io || ALL_ACTIONS
18
+ end
19
+ end
20
+
21
+ def process_action(*)
22
+ if monitors_action?(action_name)
23
+ aggregator.collect { super }
24
+ else
25
+ super
26
+ end
13
27
  end
14
28
 
15
29
  def append_info_to_payload(payload)
16
30
  super
17
31
 
32
+ return unless monitors_action?(action_name)
33
+
18
34
  data = payload[IoMonitor::NAMESPACE] = {}
19
35
 
20
- IoMonitor.aggregator.sources.each do |source|
21
- data[source] = IoMonitor.aggregator.get(source)
36
+ aggregator.sources.each do |source|
37
+ data[source] = aggregator.get(source)
22
38
  end
23
39
 
24
- data[:response] = payload[:response]&.body&.bytesize || 0
40
+ data[:response] = payload[:response].body.bytesize
41
+ end
42
+
43
+ private
44
+
45
+ def monitors_action?(action_name)
46
+ actions = self.class.actions_to_monitor_io
47
+ actions == ALL_ACTIONS || actions.include?(action_name.to_sym)
25
48
  end
26
49
  end
27
50
  end
@@ -4,7 +4,7 @@ module IoMonitor
4
4
  module ActionControllerBasePatch
5
5
  def log_process_action(payload)
6
6
  super.tap do |messages|
7
- next unless IoMonitor.config.publisher.is_a?(LogsPublisher)
7
+ next unless IoMonitor.config.publishers.any? { |publisher| publisher.is_a?(LogsPublisher) }
8
8
 
9
9
  data = payload[IoMonitor::NAMESPACE]
10
10
  next unless data
@@ -0,0 +1,14 @@
1
+ # frozen_string_literal: true
2
+
3
+ module IoMonitor
4
+ module RedisPatch
5
+ def send_command(command, &block)
6
+ super(command, &block).tap do |reply|
7
+ # we need to check QUEUED because of https://github.com/redis/redis-rb/blob/cbdb53e8c2f0be53c91404cb7ff566a36fc8ebf5/lib/redis/client.rb#L164
8
+ if reply != "QUEUED" && !reply.is_a?(Redis::CommandError) && IoMonitor.aggregator.active?
9
+ IoMonitor.aggregator.increment(RedisAdapter.kind, reply.bytesize)
10
+ end
11
+ end
12
+ end
13
+ end
14
+ end
@@ -16,15 +16,15 @@ module IoMonitor
16
16
  (payload.keys - [:response]).each do |source|
17
17
  ratio = ratio(payload[:response], payload[source])
18
18
 
19
- if ratio < IoMonitor.config.warn_threshold
20
- publish(source, ratio)
21
- end
19
+ publish(source, ratio) if ratio < IoMonitor.config.warn_threshold
22
20
  end
23
21
  end
24
22
 
25
23
  private
26
24
 
27
25
  def ratio(response_size, io_size)
26
+ return 0 if io_size.to_f.zero?
27
+
28
28
  response_size.to_f / io_size.to_f
29
29
  end
30
30
  end
@@ -0,0 +1,40 @@
1
+ # frozen_string_literal: true
2
+
3
+ module IoMonitor
4
+ class PrometheusPublisher < BasePublisher
5
+ HELP_MESSAGE = "IO payload size to response payload size ratio"
6
+
7
+ def initialize(registry: nil, aggregation: nil, buckets: nil)
8
+ registry ||= ::Prometheus::Client.registry
9
+ @metric = registry.histogram(
10
+ "#{IoMonitor::NAMESPACE}_ratio".to_sym,
11
+ labels: %i[adapter],
12
+ buckets: buckets || ::Prometheus::Client::Histogram::DEFAULT_BUCKETS,
13
+ store_settings: store_settings(aggregation),
14
+ docstring: HELP_MESSAGE
15
+ )
16
+ end
17
+
18
+ def self.kind
19
+ :prometheus
20
+ end
21
+
22
+ def publish(source, ratio)
23
+ metric.observe(ratio, labels: {adapter: source})
24
+ end
25
+
26
+ private
27
+
28
+ attr_reader :metric
29
+
30
+ # From https://github.com/yabeda-rb/yabeda-prometheus/blob/v0.8.0/lib/yabeda/prometheus/adapter.rb#L101
31
+ def store_settings(aggregation)
32
+ case ::Prometheus::Client.config.data_store
33
+ when ::Prometheus::Client::DataStores::Synchronized, ::Prometheus::Client::DataStores::SingleThreaded
34
+ {} # Default synchronized store doesn't allow to pass any options
35
+ when ::Prometheus::Client::DataStores::DirectFileStore, ::Object # Anything else
36
+ {aggregation: aggregation}.compact
37
+ end
38
+ end
39
+ end
40
+ end
@@ -15,7 +15,7 @@ module IoMonitor
15
15
  payload = args.last[IoMonitor::NAMESPACE]
16
16
  next unless payload
17
17
 
18
- IoMonitor.config.publisher.process_action(payload)
18
+ IoMonitor.config.publishers.each { |publisher| publisher.process_action(payload) }
19
19
  end
20
20
  end
21
21
  end
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module IoMonitor
4
- VERSION = "0.1.0"
4
+ VERSION = "1.0.0"
5
5
  end
data/lib/io_monitor.rb CHANGED
@@ -12,13 +12,22 @@ require "io_monitor/adapters/net_http_adapter"
12
12
  require "io_monitor/publishers/base_publisher"
13
13
  require "io_monitor/publishers/logs_publisher"
14
14
  require "io_monitor/publishers/notifications_publisher"
15
+ require "io_monitor/publishers/prometheus_publisher"
15
16
 
16
17
  require "io_monitor/railtie"
17
18
 
18
19
  module IoMonitor
19
20
  NAMESPACE = :io_monitor
20
- ADAPTERS = [ActiveRecordAdapter, NetHttpAdapter].freeze
21
- PUBLISHERS = [LogsPublisher, NotificationsPublisher].freeze
21
+
22
+ adapters = [ActiveRecordAdapter, NetHttpAdapter]
23
+
24
+ if defined? Redis
25
+ require "io_monitor/adapters/redis_adapter"
26
+ adapters << RedisAdapter
27
+ end
28
+ ADAPTERS = adapters.freeze
29
+
30
+ PUBLISHERS = [LogsPublisher, NotificationsPublisher, PrometheusPublisher].freeze
22
31
 
23
32
  class << self
24
33
  def aggregator
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: io_monitor
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0
4
+ version: 1.0.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - baygeldin
@@ -11,7 +11,7 @@ authors:
11
11
  autorequire:
12
12
  bindir: bin
13
13
  cert_chain: []
14
- date: 2022-05-23 00:00:00.000000000 Z
14
+ date: 2023-05-06 00:00:00.000000000 Z
15
15
  dependencies:
16
16
  - !ruby/object:Gem::Dependency
17
17
  name: rails
@@ -19,14 +19,42 @@ dependencies:
19
19
  requirements:
20
20
  - - ">="
21
21
  - !ruby/object:Gem::Version
22
- version: '6.0'
22
+ version: '6.1'
23
23
  type: :runtime
24
24
  prerelease: false
25
25
  version_requirements: !ruby/object:Gem::Requirement
26
26
  requirements:
27
27
  - - ">="
28
28
  - !ruby/object:Gem::Version
29
- version: '6.0'
29
+ version: '6.1'
30
+ - !ruby/object:Gem::Dependency
31
+ name: redis
32
+ requirement: !ruby/object:Gem::Requirement
33
+ requirements:
34
+ - - ">="
35
+ - !ruby/object:Gem::Version
36
+ version: '4.0'
37
+ type: :development
38
+ prerelease: false
39
+ version_requirements: !ruby/object:Gem::Requirement
40
+ requirements:
41
+ - - ">="
42
+ - !ruby/object:Gem::Version
43
+ version: '4.0'
44
+ - !ruby/object:Gem::Dependency
45
+ name: prometheus-client
46
+ requirement: !ruby/object:Gem::Requirement
47
+ requirements:
48
+ - - ">="
49
+ - !ruby/object:Gem::Version
50
+ version: '0'
51
+ type: :development
52
+ prerelease: false
53
+ version_requirements: !ruby/object:Gem::Requirement
54
+ requirements:
55
+ - - ">="
56
+ - !ruby/object:Gem::Version
57
+ version: '0'
30
58
  description:
31
59
  email:
32
60
  - dmitry.a.tsepelev@gmail.com
@@ -41,6 +69,7 @@ files:
41
69
  - lib/io_monitor/adapters/active_record_adapter.rb
42
70
  - lib/io_monitor/adapters/base_adapter.rb
43
71
  - lib/io_monitor/adapters/net_http_adapter.rb
72
+ - lib/io_monitor/adapters/redis_adapter.rb
44
73
  - lib/io_monitor/aggregator.rb
45
74
  - lib/io_monitor/configuration.rb
46
75
  - lib/io_monitor/controller.rb
@@ -48,9 +77,11 @@ files:
48
77
  - lib/io_monitor/patches/action_controller_base_patch.rb
49
78
  - lib/io_monitor/patches/future_result_patch.rb
50
79
  - lib/io_monitor/patches/net_http_adapter_patch.rb
80
+ - lib/io_monitor/patches/redis_patch.rb
51
81
  - lib/io_monitor/publishers/base_publisher.rb
52
82
  - lib/io_monitor/publishers/logs_publisher.rb
53
83
  - lib/io_monitor/publishers/notifications_publisher.rb
84
+ - lib/io_monitor/publishers/prometheus_publisher.rb
54
85
  - lib/io_monitor/railtie.rb
55
86
  - lib/io_monitor/version.rb
56
87
  homepage: https://github.com/DmitryTsepelev/io_monitor
@@ -70,14 +101,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
70
101
  requirements:
71
102
  - - ">="
72
103
  - !ruby/object:Gem::Version
73
- version: 2.6.0
104
+ version: 2.7.0
74
105
  required_rubygems_version: !ruby/object:Gem::Requirement
75
106
  requirements:
76
107
  - - ">="
77
108
  - !ruby/object:Gem::Version
78
109
  version: '0'
79
110
  requirements: []
80
- rubygems_version: 3.2.33
111
+ rubygems_version: 3.2.15
81
112
  signing_key:
82
113
  specification_version: 4
83
114
  summary: A gem that helps to detect potential memory bloats