io_monitor 0.1.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 55da4008002fc3ddd2846fda47f10c76910a29b370754f02772e33635c73d156
4
+ data.tar.gz: be06ae7bf31a7051a7e0e113cf16f1c2a55300bf4519d083d400e948e2b4a760
5
+ SHA512:
6
+ metadata.gz: 4120f7fad9b19fe8ef2be71a5c14fd54fca42141ed205cdd468086956b4232bb401fca94d47c906fa2fc371df3288f16b6b46ee2a520ef023d9fdf9fbe61a7d2
7
+ data.tar.gz: 5eab6fe8f0e5eb4072d8c9e092259896cadb8034dbf4baf8ec99460febbc0fe3bb592f7e0f120e14fdf6f1ed0e4f2684df54b2cd27830281b37eabe36c31a3d4
data/CHANGELOG.md ADDED
@@ -0,0 +1,14 @@
1
+ # Change log
2
+
3
+ ## main
4
+
5
+ ## 0.1.0 (2022-05-24)
6
+
7
+ - [PR#7](https://github.com/DmitryTsepelev/io_monitor/pull/7) Add HTTP adapter ([@maxshend])
8
+ - [PR#6](https://github.com/DmitryTsepelev/io_monitor/pull/6) Add support for ActiveRecord::Relation#load_async method ([@maxshend])
9
+ - [PR#5](https://github.com/DmitryTsepelev/io_monitor/pull/5) Use ActiveSupport::CurrentAttributes to store input payload ([@maxshend])
10
+ - [PR#2](https://github.com/DmitryTsepelev/io_monitor/pull/2), [PR#3](https://github.com/DmitryTsepelev/io_monitor/pull/3), [PR#4](https://github.com/DmitryTsepelev/io_monitor/pull/4) Initial implementation ([@prog-supdex], [@maxshend], [@baygeldin])
11
+
12
+ [@baygeldin]: https://github.com/baygeldin
13
+ [@prog-supdex]: https://github.com/prog-supdex
14
+ [@maxshend]: https://github.com/maxshend
data/LICENSE.txt ADDED
@@ -0,0 +1,21 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2022 DmitryTsepelev
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in
13
+ all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,121 @@
1
+ # IoMonitor
2
+
3
+ A gem that helps to detect potential memory bloats.
4
+
5
+ When your controller loads a lot of data to the memory but returns a small response to the client it might mean that you're using the IO in the non–optimal way. In this case, you'll see the following message in your logs:
6
+
7
+ ```
8
+ Completed 200 OK in 349ms (Views: 2.1ms | ActiveRecord: 38.7ms | ActiveRecord Payload: 866.00 B | Response Payload: 25.00 B | Allocations: 72304)
9
+ ```
10
+
11
+ <p align="center">
12
+ <a href="https://evilmartians.com/?utm_source=io_monitor">
13
+ <img src="https://evilmartians.com/badges/sponsored-by-evil-martians.svg" alt="Sponsored by Evil Martians" width="236" height="54">
14
+ </a>
15
+ </p>
16
+
17
+ ## Usage
18
+
19
+ Add this line to your application's Gemfile:
20
+
21
+ ```ruby
22
+ gem 'io_monitor'
23
+ ```
24
+
25
+ Change configuration in an initializer if you need:
26
+
27
+ ```ruby
28
+ IoMonitor.configure do |config|
29
+ config.publish = :notifications # defaults to :logs
30
+ config.warn_threshold = 0.8 # defaults to 0
31
+ config.adapters = [:active_record, :net_http] # defaults to [:active_record]
32
+ end
33
+ ```
34
+
35
+ Then include the concern into your controller:
36
+
37
+ ```ruby
38
+ class MyController < ApplicationController
39
+ include IoMonitor::Controller
40
+ end
41
+ ```
42
+
43
+ Depending on configuration when IO payload size to response payload size ratio reaches the threshold either a `warn_threshold_reached.io_monitor` notification will be sent or a following warning will be logged:
44
+
45
+ ```
46
+ ActiveRecord I/O to response payload ratio is 0.1, while threshold is 0.8
47
+ ```
48
+
49
+ In addition, if `publish` is set to logs, additional data will be logged on each request:
50
+
51
+ ```
52
+ Completed 200 OK in 349ms (Views: 2.1ms | ActiveRecord: 38.7ms | ActiveRecord Payload: 866.00 B | Response Payload: 25.00 B | Allocations: 72304)
53
+ ```
54
+
55
+ If you want to inspect payload sizes, check out payload data for the `process_action.action_controller` event:
56
+
57
+ ```ruby
58
+ ActiveSupport::Notifications.subscribe("process_action.action_controller") do |name, start, finish, id, payload|
59
+ payload[:io_monitor] # { active_record: 866, response: 25 }
60
+ end
61
+ ```
62
+
63
+ ## Custom publishers
64
+
65
+ Implement your custom publisher by inheriting from `BasePublisher`:
66
+
67
+ ```ruby
68
+ class MyPublisher < IoMonitor::BasePublisher
69
+ def publish(source, ratio)
70
+ puts "Warn threshold reched for #{source} at #{ratio}!"
71
+ end
72
+ end
73
+ ```
74
+
75
+ Then specify it in the configuration:
76
+
77
+ ```ruby
78
+ IoMonitor.configure do |config|
79
+ config.publish = MyPublisher.new
80
+ end
81
+ ```
82
+
83
+ ## Custom adapters
84
+
85
+ Implement your custom adapter by inheriting from `BaseAdapter`:
86
+
87
+ ```ruby
88
+ class MyAdapter < IoMonitor::BaseAdapter
89
+ def self.kind
90
+ :my_source
91
+ end
92
+
93
+ def initialize!
94
+ # Take a look at `AbstractAdapterPatch` for an example.
95
+ end
96
+ end
97
+ ```
98
+
99
+ Then specify it in the configuration:
100
+
101
+ ```ruby
102
+ IoMonitor.configure do |config|
103
+ config.adapters = [:active_record, MyAdapter.new]
104
+ end
105
+ ```
106
+
107
+ ## Development
108
+
109
+ After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests.
110
+
111
+ ## Contributing
112
+
113
+ Bug reports and pull requests are welcome on GitHub at https://github.com/DmitryTsepelev/io_monitor.
114
+
115
+ ## License
116
+
117
+ The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
118
+
119
+ ## Credits
120
+
121
+ Thanks to [@prog-supdex](https://github.com/prog-supdex) and [@maxshend](https://github.com/maxshend) for building the initial implementations (see [PR#2](https://github.com/DmitryTsepelev/io_monitor/pull/2) and [PR#3](https://github.com/DmitryTsepelev/io_monitor/pull/3)).
@@ -0,0 +1,38 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "io_monitor/patches/abstract_adapter_patch"
4
+ require "io_monitor/patches/future_result_patch"
5
+
6
+ module IoMonitor
7
+ class ActiveRecordAdapter < BaseAdapter
8
+ class << self
9
+ def kind
10
+ :active_record
11
+ end
12
+
13
+ def aggregate_result(rows:)
14
+ return unless IoMonitor.aggregator.active?
15
+
16
+ # `.flatten.join.bytesize` would look prettier,
17
+ # but it makes a lot of unnecessary allocations.
18
+ io_payload_size = rows.sum(0) do |row|
19
+ row.sum(0) do |val|
20
+ (String === val ? val : val.to_s).bytesize
21
+ end
22
+ end
23
+
24
+ IoMonitor.aggregator.increment(kind, io_payload_size)
25
+ end
26
+ end
27
+
28
+ def initialize!
29
+ ActiveSupport.on_load(:active_record) do
30
+ ActiveRecord::ConnectionAdapters::AbstractAdapter.prepend(AbstractAdapterPatch)
31
+
32
+ if Rails::VERSION::MAJOR >= 7
33
+ ActiveRecord::FutureResult.prepend(FutureResultPatch)
34
+ end
35
+ end
36
+ end
37
+ end
38
+ end
@@ -0,0 +1,15 @@
1
+ # frozen_string_literal: true
2
+
3
+ module IoMonitor
4
+ class BaseAdapter
5
+ # :nocov:
6
+ def self.kind
7
+ raise NotImplementedError
8
+ end
9
+
10
+ def initialize!
11
+ raise NotImplementedError
12
+ end
13
+ # :nocov:
14
+ end
15
+ end
@@ -0,0 +1,24 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "net/http"
4
+ require "io_monitor/patches/net_http_adapter_patch"
5
+
6
+ module IoMonitor
7
+ class NetHttpAdapter < BaseAdapter
8
+ def self.kind
9
+ :net_http
10
+ end
11
+
12
+ def initialize!
13
+ ActiveSupport.on_load(:after_initialize) do
14
+ Net::HTTP.prepend(NetHttpAdapterPatch)
15
+
16
+ if defined?(::WebMock)
17
+ WebMock::HttpLibAdapters::NetHttpAdapter
18
+ .instance_variable_get(:@webMockNetHTTP)
19
+ .prepend(NetHttpAdapterPatch)
20
+ end
21
+ end
22
+ end
23
+ end
24
+ end
@@ -0,0 +1,46 @@
1
+ # frozen_string_literal: true
2
+
3
+ module IoMonitor
4
+ # Thread-safe payload size aggregator.
5
+ class Aggregator
6
+ def initialize(sources)
7
+ @sources = sources
8
+ end
9
+
10
+ attr_reader :sources
11
+
12
+ def active?
13
+ InputPayload.active.present?
14
+ end
15
+
16
+ def start!
17
+ InputPayload.active = true
18
+ end
19
+
20
+ def stop!
21
+ InputPayload.active = false
22
+ end
23
+
24
+ def increment(source, val)
25
+ return unless active?
26
+
27
+ InputPayload.state ||= empty_state
28
+ InputPayload.state[source.to_sym] += val
29
+ end
30
+
31
+ def get(source)
32
+ InputPayload.state ||= empty_state
33
+ InputPayload.state[source.to_sym]
34
+ end
35
+
36
+ private
37
+
38
+ def empty_state
39
+ sources.map { |kind| [kind, 0] }.to_h
40
+ end
41
+
42
+ class InputPayload < ActiveSupport::CurrentAttributes
43
+ attribute :state, :active
44
+ end
45
+ end
46
+ end
@@ -0,0 +1,54 @@
1
+ # frozen_string_literal: true
2
+
3
+ module IoMonitor
4
+ class Configuration
5
+ DEFAULT_WARN_THRESHOLD = 0.0
6
+
7
+ def initialize
8
+ @publisher = LogsPublisher.new
9
+ @adapters = [ActiveRecordAdapter.new]
10
+ @warn_threshold = DEFAULT_WARN_THRESHOLD
11
+ end
12
+
13
+ attr_reader :publisher, :adapters, :warn_threshold
14
+
15
+ def publish=(value)
16
+ if value.is_a?(BasePublisher)
17
+ @publisher = value
18
+ elsif (publisher_type = resolve(IoMonitor::PUBLISHERS, value))
19
+ @publisher = publisher_type.new
20
+ else
21
+ supported = IoMonitor::PUBLISHERS.map(&:kind)
22
+ raise ArgumentError, "Only the following publishers are supported: #{supported}."
23
+ end
24
+ end
25
+
26
+ def adapters=(value)
27
+ @adapters = [*value].map do |adapter|
28
+ if adapter.is_a?(BaseAdapter)
29
+ adapter
30
+ elsif (adapter_type = resolve(IoMonitor::ADAPTERS, adapter))
31
+ adapter_type.new
32
+ else
33
+ supported = IoMonitor::ADAPTERS.map(&:kind)
34
+ raise ArgumentError, "Only the following adapters are supported: #{supported}."
35
+ end
36
+ end
37
+ end
38
+
39
+ def warn_threshold=(value)
40
+ case value
41
+ when 0..1
42
+ @warn_threshold = value.to_f
43
+ else
44
+ raise ArgumentError, "Warn threshold should be within 0..1 range."
45
+ end
46
+ end
47
+
48
+ private
49
+
50
+ def resolve(list, kind)
51
+ list.find { |p| p.kind == kind }
52
+ end
53
+ end
54
+ end
@@ -0,0 +1,27 @@
1
+ # frozen_string_literal: true
2
+
3
+ module IoMonitor
4
+ module Controller
5
+ extend ActiveSupport::Concern
6
+
7
+ def process_action(*)
8
+ IoMonitor.aggregator.start!
9
+
10
+ super
11
+
12
+ IoMonitor.aggregator.stop!
13
+ end
14
+
15
+ def append_info_to_payload(payload)
16
+ super
17
+
18
+ data = payload[IoMonitor::NAMESPACE] = {}
19
+
20
+ IoMonitor.aggregator.sources.each do |source|
21
+ data[source] = IoMonitor.aggregator.get(source)
22
+ end
23
+
24
+ data[:response] = payload[:response]&.body&.bytesize || 0
25
+ end
26
+ end
27
+ end
@@ -0,0 +1,11 @@
1
+ # frozen_string_literal: true
2
+
3
+ module IoMonitor
4
+ module AbstractAdapterPatch
5
+ def build_result(*args, **kwargs, &block)
6
+ ActiveRecordAdapter.aggregate_result rows: kwargs[:rows]
7
+
8
+ super
9
+ end
10
+ end
11
+ end
@@ -0,0 +1,19 @@
1
+ # frozen_string_literal: true
2
+
3
+ module IoMonitor
4
+ module ActionControllerBasePatch
5
+ def log_process_action(payload)
6
+ super.tap do |messages|
7
+ next unless IoMonitor.config.publisher.is_a?(LogsPublisher)
8
+
9
+ data = payload[IoMonitor::NAMESPACE]
10
+ next unless data
11
+
12
+ data.each do |source, bytes|
13
+ size = ActiveSupport::NumberHelper.number_to_human_size(bytes)
14
+ messages << "#{source.to_s.camelize} Payload: #{size}"
15
+ end
16
+ end
17
+ end
18
+ end
19
+ end
@@ -0,0 +1,15 @@
1
+ # frozen_string_literal: true
2
+
3
+ module IoMonitor
4
+ module FutureResultPatch
5
+ def result
6
+ # @event_buffer is used to send ActiveSupport notifications related to async queries
7
+ return super unless @event_buffer
8
+
9
+ res = super
10
+ ActiveRecordAdapter.aggregate_result rows: res.rows
11
+
12
+ res
13
+ end
14
+ end
15
+ end
@@ -0,0 +1,13 @@
1
+ # frozen_string_literal: true
2
+
3
+ module IoMonitor
4
+ module NetHttpAdapterPatch
5
+ def request(*args, &block)
6
+ super do |response|
7
+ if response.body && IoMonitor.aggregator.active?
8
+ IoMonitor.aggregator.increment(NetHttpAdapter.kind, response.body.bytesize)
9
+ end
10
+ end
11
+ end
12
+ end
13
+ end
@@ -0,0 +1,31 @@
1
+ # frozen_string_literal: true
2
+
3
+ module IoMonitor
4
+ class BasePublisher
5
+ # :nocov:
6
+ def self.kind
7
+ raise NotImplementedError
8
+ end
9
+
10
+ def publish(source, ratio)
11
+ raise NotImplementedError
12
+ end
13
+ # :nocov:
14
+
15
+ def process_action(payload)
16
+ (payload.keys - [:response]).each do |source|
17
+ ratio = ratio(payload[:response], payload[source])
18
+
19
+ if ratio < IoMonitor.config.warn_threshold
20
+ publish(source, ratio)
21
+ end
22
+ end
23
+ end
24
+
25
+ private
26
+
27
+ def ratio(response_size, io_size)
28
+ response_size.to_f / io_size.to_f
29
+ end
30
+ end
31
+ end
@@ -0,0 +1,16 @@
1
+ # frozen_string_literal: true
2
+
3
+ module IoMonitor
4
+ class LogsPublisher < BasePublisher
5
+ def self.kind
6
+ :logs
7
+ end
8
+
9
+ def publish(source, ratio)
10
+ Rails.logger.warn <<~HEREDOC.squish
11
+ #{source.to_s.camelize} I/O to response payload ratio is #{ratio},
12
+ while threshold is #{IoMonitor.config.warn_threshold}
13
+ HEREDOC
14
+ end
15
+ end
16
+ end
@@ -0,0 +1,25 @@
1
+ # frozen_string_literal: true
2
+
3
+ module IoMonitor
4
+ class NotificationsPublisher < BasePublisher
5
+ WARN_THRESHOLD_REACHED_EVENT = "warn_threshold_reached"
6
+
7
+ def self.kind
8
+ :notifications
9
+ end
10
+
11
+ def publish(source, ratio)
12
+ ActiveSupport::Notifications.instrument(
13
+ full_event_name(WARN_THRESHOLD_REACHED_EVENT),
14
+ source: source,
15
+ ratio: ratio
16
+ )
17
+ end
18
+
19
+ private
20
+
21
+ def full_event_name(event_name)
22
+ "#{event_name}.#{IoMonitor::NAMESPACE}"
23
+ end
24
+ end
25
+ end
@@ -0,0 +1,22 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "io_monitor/patches/action_controller_base_patch"
4
+
5
+ module IoMonitor
6
+ class Railtie < Rails::Railtie
7
+ config.after_initialize do
8
+ IoMonitor.config.adapters.each(&:initialize!)
9
+
10
+ ActiveSupport.on_load(:action_controller) do
11
+ ActionController::Base.singleton_class.prepend(ActionControllerBasePatch)
12
+ end
13
+
14
+ ActiveSupport::Notifications.subscribe("process_action.action_controller") do |*args|
15
+ payload = args.last[IoMonitor::NAMESPACE]
16
+ next unless payload
17
+
18
+ IoMonitor.config.publisher.process_action(payload)
19
+ end
20
+ end
21
+ end
22
+ end
@@ -0,0 +1,5 @@
1
+ # frozen_string_literal: true
2
+
3
+ module IoMonitor
4
+ VERSION = "0.1.0"
5
+ end
data/lib/io_monitor.rb ADDED
@@ -0,0 +1,38 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "io_monitor/version"
4
+ require "io_monitor/configuration"
5
+ require "io_monitor/aggregator"
6
+ require "io_monitor/controller"
7
+
8
+ require "io_monitor/adapters/base_adapter"
9
+ require "io_monitor/adapters/active_record_adapter"
10
+ require "io_monitor/adapters/net_http_adapter"
11
+
12
+ require "io_monitor/publishers/base_publisher"
13
+ require "io_monitor/publishers/logs_publisher"
14
+ require "io_monitor/publishers/notifications_publisher"
15
+
16
+ require "io_monitor/railtie"
17
+
18
+ module IoMonitor
19
+ NAMESPACE = :io_monitor
20
+ ADAPTERS = [ActiveRecordAdapter, NetHttpAdapter].freeze
21
+ PUBLISHERS = [LogsPublisher, NotificationsPublisher].freeze
22
+
23
+ class << self
24
+ def aggregator
25
+ @aggregator ||= Aggregator.new(
26
+ config.adapters.map { |a| a.class.kind }
27
+ )
28
+ end
29
+
30
+ def config
31
+ @config ||= Configuration.new
32
+ end
33
+
34
+ def configure
35
+ yield config
36
+ end
37
+ end
38
+ end
metadata ADDED
@@ -0,0 +1,84 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: io_monitor
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.1.0
5
+ platform: ruby
6
+ authors:
7
+ - baygeldin
8
+ - prog-supdex
9
+ - maxshend
10
+ - DmitryTsepelev
11
+ autorequire:
12
+ bindir: bin
13
+ cert_chain: []
14
+ date: 2022-05-23 00:00:00.000000000 Z
15
+ dependencies:
16
+ - !ruby/object:Gem::Dependency
17
+ name: rails
18
+ requirement: !ruby/object:Gem::Requirement
19
+ requirements:
20
+ - - ">="
21
+ - !ruby/object:Gem::Version
22
+ version: '6.0'
23
+ type: :runtime
24
+ prerelease: false
25
+ version_requirements: !ruby/object:Gem::Requirement
26
+ requirements:
27
+ - - ">="
28
+ - !ruby/object:Gem::Version
29
+ version: '6.0'
30
+ description:
31
+ email:
32
+ - dmitry.a.tsepelev@gmail.com
33
+ executables: []
34
+ extensions: []
35
+ extra_rdoc_files: []
36
+ files:
37
+ - CHANGELOG.md
38
+ - LICENSE.txt
39
+ - README.md
40
+ - lib/io_monitor.rb
41
+ - lib/io_monitor/adapters/active_record_adapter.rb
42
+ - lib/io_monitor/adapters/base_adapter.rb
43
+ - lib/io_monitor/adapters/net_http_adapter.rb
44
+ - lib/io_monitor/aggregator.rb
45
+ - lib/io_monitor/configuration.rb
46
+ - lib/io_monitor/controller.rb
47
+ - lib/io_monitor/patches/abstract_adapter_patch.rb
48
+ - lib/io_monitor/patches/action_controller_base_patch.rb
49
+ - lib/io_monitor/patches/future_result_patch.rb
50
+ - lib/io_monitor/patches/net_http_adapter_patch.rb
51
+ - lib/io_monitor/publishers/base_publisher.rb
52
+ - lib/io_monitor/publishers/logs_publisher.rb
53
+ - lib/io_monitor/publishers/notifications_publisher.rb
54
+ - lib/io_monitor/railtie.rb
55
+ - lib/io_monitor/version.rb
56
+ homepage: https://github.com/DmitryTsepelev/io_monitor
57
+ licenses:
58
+ - MIT
59
+ metadata:
60
+ bug_tracker_uri: https://github.com/DmitryTsepelev/io_monitor/issues
61
+ changelog_uri: https://github.com/DmitryTsepelev/io_monitor/blob/master/CHANGELOG.md
62
+ documentation_uri: https://github.com/DmitryTsepelev/io_monitor/blob/master/README.md
63
+ homepage_uri: https://github.com/DmitryTsepelev/io_monitor
64
+ source_code_uri: https://github.com/DmitryTsepelev/io_monitor
65
+ post_install_message:
66
+ rdoc_options: []
67
+ require_paths:
68
+ - lib
69
+ required_ruby_version: !ruby/object:Gem::Requirement
70
+ requirements:
71
+ - - ">="
72
+ - !ruby/object:Gem::Version
73
+ version: 2.6.0
74
+ required_rubygems_version: !ruby/object:Gem::Requirement
75
+ requirements:
76
+ - - ">="
77
+ - !ruby/object:Gem::Version
78
+ version: '0'
79
+ requirements: []
80
+ rubygems_version: 3.2.33
81
+ signing_key:
82
+ specification_version: 4
83
+ summary: A gem that helps to detect potential memory bloats
84
+ test_files: []