active_monitoring 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/MIT-LICENSE +20 -0
- data/README.md +182 -0
- data/Rakefile +22 -0
- data/app/controllers/active_monitoring/application_controller.rb +5 -0
- data/app/controllers/active_monitoring/dashboard_controller.rb +9 -0
- data/app/models/active_monitoring/application_record.rb +5 -0
- data/app/models/active_monitoring/dashboard.rb +33 -0
- data/app/models/active_monitoring/metric.rb +7 -0
- data/app/views/active_monitoring/dashboard/show.html.erb +23 -0
- data/app/views/layouts/active_monitoring/application.html.erb +13 -0
- data/config/routes.rb +3 -0
- data/db/migrate/20200401191741_create_active_monitoring_metrics.rb +13 -0
- data/lib/active_monitoring.rb +6 -0
- data/lib/active_monitoring/core_extensions/action_controller.rb +19 -0
- data/lib/active_monitoring/core_extensions/active_record.rb +13 -0
- data/lib/active_monitoring/current.rb +31 -0
- data/lib/active_monitoring/engine.rb +47 -0
- data/lib/active_monitoring/notifications.rb +19 -0
- data/lib/active_monitoring/notifier.rb +33 -0
- data/lib/active_monitoring/sql_normalizer.rb +23 -0
- data/lib/active_monitoring/sql_query.rb +23 -0
- data/lib/active_monitoring/sql_tracker.rb +33 -0
- data/lib/active_monitoring/version.rb +3 -0
- metadata +158 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: d713464ea142fd4d40db7b8b12b786f3fd21c85d052cc9fbe2eb9fcb5ece2a87
|
4
|
+
data.tar.gz: 94c592b0e41fe8be9c609525eb429979cb67f382179baaae902abcd4b256b031
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 56afc9fc77d56d52437671acfe644308e83f13d1a0fe91b2b38ded82b0f007156eec4784d75d6ba041c57bed97ba301a3334bb1dd88a8ce4a4e824993d01ff02
|
7
|
+
data.tar.gz: b1b763e859633c3255693fa5b3453cadccf6184788f5482f18913bbcc8c7cc8dc88d1429fe4a9fa50b721f646ca39a4befb36fb20fc1f76a27122803a87b63fb
|
data/MIT-LICENSE
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
Copyright 2020 Christian Bruckmayer
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
4
|
+
a copy of this software and associated documentation files (the
|
5
|
+
"Software"), to deal in the Software without restriction, including
|
6
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
7
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
8
|
+
permit persons to whom the Software is furnished to do so, subject to
|
9
|
+
the following conditions:
|
10
|
+
|
11
|
+
The above copyright notice and this permission notice shall be
|
12
|
+
included in all copies or substantial portions of the Software.
|
13
|
+
|
14
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
15
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
16
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
17
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
18
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
19
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
20
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,182 @@
|
|
1
|
+
# ActiveMonitoring
|
2
|
+
|
3
|
+
I gave this talk at [RailsConf 2020 Couch Edition](https://railsconf.com/) and this blog article contains the source code and step by step tutorial.
|
4
|
+
The talk and tutorial is inspired by my work on [influxdb-rails](https://github.com/influxdata/influxdb-rails) and contains several patterns and learnings from it.
|
5
|
+
|
6
|
+
By the end of this tutorial, we will have a very basic Performance Monitoring tool which measures the response and SQL query time of a Rails application.
|
7
|
+
|
8
|
+
## Abstract
|
9
|
+
> Building a Performance Analytics Tool with ActiveSupport
|
10
|
+
|
11
|
+
> Setting up a performance analytics tool like NewRelic or Skylight is one of the first things many developers do in a new project. However, have you ever wondered how these tools work under the hood?
|
12
|
+
|
13
|
+
> In this talk, we will build a basic performance analytics tool from scratch. We will deep dive into ActiveSupport instrumentations to collect, group and normalise metrics from your app. To persist these metrics, we will use a time series database and visualise the data on a dashboard. By the end of the talk, you will know how your favourite performance analytics tool works.
|
14
|
+
|
15
|
+
## Chapter 1: Collecting data
|
16
|
+
### Notification Framework
|
17
|
+
As a first step we need to implement a notification framework which we can use to instrument and subscribe to performance metrics.
|
18
|
+
|
19
|
+
A basic example of the functionality of our notification framework would be this code:
|
20
|
+
```ruby
|
21
|
+
scenario "It can susbcribe and instrument events" do
|
22
|
+
events = []
|
23
|
+
ActiveMonitoring::Notifications.subscribe("test_event") do |name, start, finish, id, payload|
|
24
|
+
events << { name: name, start: start, finish: finish, id: id, payload: payload }
|
25
|
+
end
|
26
|
+
|
27
|
+
ActiveMonitoring::Notifications.instrument("test_event", payload: :payload) do
|
28
|
+
1 + 1
|
29
|
+
end
|
30
|
+
|
31
|
+
expect(metrics[0].name).to eq("test_event")
|
32
|
+
expect(metrics[0].payload).to include(payload: :payload)
|
33
|
+
end
|
34
|
+
```
|
35
|
+
|
36
|
+
The implementation can be found in commit [32c3](https://github.com/ChrisBr/active_monitoring/commit/32c32dcec7529ac7a8e0125b1aaab28cd4219363).
|
37
|
+
For the sake of simplicity, this framework is not thread safe of course.
|
38
|
+
|
39
|
+
This framework is inspired by [ActiveSupport Notifications](https://edgeguides.rubyonrails.org/active_support_instrumentation.html) which already provides various hooks into Rails internals.
|
40
|
+
The implementatition of Active Support Notifications can be found [here](https://github.com/rails/rails/tree/master/activesupport/lib/active_support/notifications).
|
41
|
+
|
42
|
+
### Instrument events
|
43
|
+
[ActiveSupport Notifications](https://edgeguides.rubyonrails.org/active_support_instrumentation.html) provides already
|
44
|
+
several hooks to subscribe to e.g. processing of controller actions, SQL queries or view rendering.
|
45
|
+
However, Rails does not provide hooks for external libraries like Redis or Memcache.
|
46
|
+
In [ef20](https://github.com/ChrisBr/active_monitoring/commit/ef200ae5bf58f9afd2a69003e2c3620d0d9c86eb) and [46c3](https://github.com/ChrisBr/active_monitoring/commit/46c338f8fa5870bae7dc0e251e637d1d4be551db) we monkey patch Rails
|
47
|
+
to subscribe to controller actions and SQL queries.
|
48
|
+
|
49
|
+
## Chapter 2: Storage
|
50
|
+
In chapter two we discussed an appropriate storage engine for our data.
|
51
|
+
Our data would look for example like this
|
52
|
+
|
53
|
+
| Timestamp | Hook | Location | Request ID | Value |
|
54
|
+
| ------------- | ------------- |------------- |------------- |------------- |
|
55
|
+
| 01.04.2020 00:31:30 CET | process_action | BooksController#show | cf6d6dcd | 100 ms
|
56
|
+
| 01.04.2020 00:32:30 CET | sql | BooksController#show | c7c7c841 | 80 ms
|
57
|
+
|
58
|
+
This data looks tabular, so using a relational database to store it would make sense.
|
59
|
+
However, there are several downsides to this approach.
|
60
|
+
|
61
|
+
### Relational Database
|
62
|
+
|
63
|
+
Relational databases perform very well with applications which map to the basic CRUD operations of the database like a Rails controller.
|
64
|
+
|
65
|
+
```ruby
|
66
|
+
class BooksController < ApplicationController
|
67
|
+
def new; end
|
68
|
+
def create; end
|
69
|
+
def show; end
|
70
|
+
def edit; end
|
71
|
+
def update; end
|
72
|
+
def destroy; end
|
73
|
+
end
|
74
|
+
```
|
75
|
+
|
76
|
+
However, our plugin does not really use all of these traditional database operations or in a different way we would in a normal Rails app.
|
77
|
+
|
78
|
+
#### Create
|
79
|
+
For each request, we will write one process action and several SQL query data points.
|
80
|
+
If we extend this application to also cover view render times or background jobs, it's not unlikely we would write 10+ metrics per request.
|
81
|
+
Depending of how many requests your app does process, we could easily create millions of metrics per year.
|
82
|
+
|
83
|
+
#### Read
|
84
|
+
We will read the metrics in a dashboard for a certain time range like today or last three hours but not randomly.
|
85
|
+
|
86
|
+
#### Update
|
87
|
+
We will almost never go back to update on of the metrics.
|
88
|
+
|
89
|
+
#### Delete
|
90
|
+
We will only delete metrics in batches but not single metrics.
|
91
|
+
For instance, we are only interested in metrics for the last three months so once a month we can do a 'garbage collection' and delete metrics older than three months.
|
92
|
+
|
93
|
+
### Time Series Database
|
94
|
+
|
95
|
+
With this all said, a relational database might not be the best storage engine for this kind of data as
|
96
|
+
|
97
|
+
* We will heavily write to the database which would cause to frequently create new pages in a [B-Tree](https://en.wikipedia.org/wiki/B-tree) like most relational databases use internally
|
98
|
+
* We would need to implement some sort of garbage collection to remove old data
|
99
|
+
* Compression of data would not be very efficient
|
100
|
+
|
101
|
+
For this kind of data, a [log based storage engine](https://en.wikipedia.org/wiki/Log-structured_merge-tree) like most [Time Series databases](https://db-engines.com/en/ranking/time+series+dbms) implement it would be more efficient.
|
102
|
+
|
103
|
+
However, for the sake of simplicity, we store our metrics in a relational database in this tutorial, the implementation can be found in [5967](https://github.com/ChrisBr/active_monitoring/commit/5967190a1484009476f2378c02e3e0ea96d21624).
|
104
|
+
|
105
|
+
## Chapter 3: Cleaning, Normalizing and Grouping
|
106
|
+
In the previous chapter we already stored the controller metrics.
|
107
|
+
Before we can store SQL metrics, we need to do some more additional work.
|
108
|
+
|
109
|
+
### Cleaning
|
110
|
+
ActiveRecord does a lot of 'magic' behind the scenes to abstract persisting objects to the database.
|
111
|
+
Some of this 'magic' requires to execute additional database queries, for instance
|
112
|
+
|
113
|
+
* Current version of the database engine
|
114
|
+
* Which migrations did already run
|
115
|
+
* Which environment
|
116
|
+
|
117
|
+
We're only interested to store application metrics so we need to filter these queries out.
|
118
|
+
The implementation can be found in [156d](https://github.com/ChrisBr/active_monitoring/commit/156d8f7fb0bd3b013524a20effc65dad05506b3d).
|
119
|
+
|
120
|
+
### Normalizing
|
121
|
+
Depending on the database adapter you use, the queries might contain values.
|
122
|
+
To group the same queries together, we need to normalize these queries.
|
123
|
+
|
124
|
+
```SQL
|
125
|
+
SELECT * FROM books WHERE id = 1;
|
126
|
+
SELECT * FROM books WHERE id = 2;
|
127
|
+
```
|
128
|
+
|
129
|
+
For our performance monitoring tool, these two queries should be treated the same and we need to normalize them to
|
130
|
+
```SQL
|
131
|
+
SELECT * FROM books WHERE id = xxx;
|
132
|
+
```
|
133
|
+
|
134
|
+
In [66a8](https://github.com/ChrisBr/active_monitoring/commit/66a833a46b40072a8888fb6b6c6bb02ee60bda17) we implement a simple query normalizer.
|
135
|
+
|
136
|
+
### Grouping
|
137
|
+
ActiveRecord is a standalone framework, therefore the payload of the SQL event does not contain a `request_id`.
|
138
|
+
We can use ActiveRecord for instance in migrations or background jobs so it is perfectly valid to use it outside of a request response cycle.
|
139
|
+
However, in our Performance Monitoring tool we would like to group requests and SQL queries together so we can see if a query causes a slow response.
|
140
|
+
|
141
|
+
Luckily, we also implemented a `start_processing` event in [ef20](https://github.com/ChrisBr/active_monitoring/commit/ef200ae5bf58f9afd2a69003e2c3620d0d9c86eb).
|
142
|
+
We can now subscribe to this event and set the `request_id` and `location` in a cache which we later read
|
143
|
+
in when writing the SQL metrics.
|
144
|
+
In [b1f1](https://github.com/ChrisBr/active_monitoring/commit/b1f1847270935beacc236bc54535ced1eb83c5ef) we implement a `CurrentAttributes` class and eventually write the SQL metrics.
|
145
|
+
|
146
|
+
ActiveSupport ships with [CurrentAttributes](https://github.com/rails/rails/blob/157920aead96865e3135f496c09ace607d5620dc/activesupport/lib/active_support/current_attributes.rb) out of the box since Rails 5.
|
147
|
+
|
148
|
+
## Chapter 4: Visualization
|
149
|
+
So we now have several metrics in our data base written which would look something like
|
150
|
+
|
151
|
+
| Timestamp | Hook | Location | Request ID | Value |
|
152
|
+
| ------------- | ------------- |------------- |------------- |------------- |
|
153
|
+
| 01.04.2020 00:31:30 CET | process_action | BooksController#show | cf6d6dcd | 100 ms
|
154
|
+
| 01.04.2020 00:32:30 CET | sql | BooksController#show | c7c7c841 | 80 ms
|
155
|
+
|
156
|
+
It would be very hard to spot now problems or discover pattern so eventually we need to visualize our collected data.
|
157
|
+
In [cfca](https://github.com/ChrisBr/active_monitoring/commit/cfcafd2f4664e6927e7d297a4b7da2e18bdd5205) we implement a dashboard to show percentiles and the slowest queries of our Rails app.
|
158
|
+
A very good blog article about data visualization for performance metrics from Richard Schneemann can be found [here](https://www.schneems.com/2020/03/17/lies-damned-lies-and-averages-perc50-perc95-explained-for-programmers/).
|
159
|
+
|
160
|
+
In a real world application, I would strongly recommend to use a dashboard software like [Grafana](https://github.com/grafana/grafana).
|
161
|
+
|
162
|
+
## Summary
|
163
|
+
Congratulations, we implemented a very basic Performance Monitoring tool in just a few hundred lines of code now.
|
164
|
+
We deep dived into [ActiveSupport Notifications](https://edgeguides.rubyonrails.org/active_support_instrumentation.html) and hooked into Rails events to write request and SQL query metrics in our data storage.
|
165
|
+
As data storage, we compared relational databases with time series databases like InfluxDB.
|
166
|
+
Before we could visualize our data, we needed to clean, normalize and group the metrics.
|
167
|
+
|
168
|
+
From here, we can easily add more metrics like [rendering of views](https://edgeguides.rubyonrails.org/active_support_instrumentation.html#action-view),
|
169
|
+
[caching](https://edgeguides.rubyonrails.org/active_support_instrumentation.html#active-support) or [background jobs](https://edgeguides.rubyonrails.org/active_support_instrumentation.html#active-job).
|
170
|
+
|
171
|
+
As initially mentioned, this tutorial is heavily influenced by our work on https://github.com/influxdata/influxdb-rails.
|
172
|
+
If this made you curious, we always look for new contributors.
|
173
|
+
|
174
|
+
## Further information
|
175
|
+
* [Profiling and Benchmarking 101 by Nate Berkopec](https://youtu.be/XL51vf-XBTs)
|
176
|
+
* [Lies, Damned Lies, and Averages: Perc50, Perc95 explained for Programmers by Richard Schneeman](https://www.schneems.com/2020/03/17/lies-damned-lies-and-averages-perc50-perc95-explained-for-programmers/)
|
177
|
+
* https://github.com/influxdata/influxdb-ruby
|
178
|
+
* https://github.com/influxdata/influxdb-rails
|
179
|
+
* https://docs.influxdata.com/influxdb/v1.7/concepts/storage_engine/
|
180
|
+
|
181
|
+
## License
|
182
|
+
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
|
data/Rakefile
ADDED
@@ -0,0 +1,22 @@
|
|
1
|
+
begin
|
2
|
+
require "bundler/setup"
|
3
|
+
rescue LoadError
|
4
|
+
puts "You must `gem install bundler` and `bundle install` to run rake tasks"
|
5
|
+
end
|
6
|
+
|
7
|
+
require "rdoc/task"
|
8
|
+
|
9
|
+
RDoc::Task.new(:rdoc) do |rdoc|
|
10
|
+
rdoc.rdoc_dir = "rdoc"
|
11
|
+
rdoc.title = "ActiveMonitoring"
|
12
|
+
rdoc.options << "--line-numbers"
|
13
|
+
rdoc.rdoc_files.include("README.md")
|
14
|
+
rdoc.rdoc_files.include("lib/**/*.rb")
|
15
|
+
end
|
16
|
+
|
17
|
+
APP_RAKEFILE = File.expand_path("spec/dummy/Rakefile", __dir__)
|
18
|
+
load "rails/tasks/engine.rake"
|
19
|
+
|
20
|
+
load "rails/tasks/statistics.rake"
|
21
|
+
|
22
|
+
require "bundler/gem_tasks"
|
@@ -0,0 +1,33 @@
|
|
1
|
+
module ActiveMonitoring
|
2
|
+
class Dashboard
|
3
|
+
LIMIT = 10
|
4
|
+
|
5
|
+
def initialize(date = Date.current)
|
6
|
+
@date = date
|
7
|
+
end
|
8
|
+
|
9
|
+
def percentile(value)
|
10
|
+
response_metrics.percentile(value)
|
11
|
+
end
|
12
|
+
|
13
|
+
def slow_sql_queries
|
14
|
+
sql_metrics.order(:value).limit(LIMIT)
|
15
|
+
end
|
16
|
+
|
17
|
+
private
|
18
|
+
|
19
|
+
attr_reader :date
|
20
|
+
|
21
|
+
def sql_metrics
|
22
|
+
metrics.where(name: "sql.active_record")
|
23
|
+
end
|
24
|
+
|
25
|
+
def response_metrics
|
26
|
+
metrics.where(name: "process_action.action_controller")
|
27
|
+
end
|
28
|
+
|
29
|
+
def metrics
|
30
|
+
Metric.where(created_at: date.beginning_of_day..date.end_of_day)
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
@@ -0,0 +1,23 @@
|
|
1
|
+
<h1>Dashboard</h1>
|
2
|
+
|
3
|
+
<h2>Response time</h2>
|
4
|
+
<ul>
|
5
|
+
<li>90th Percentile: <%= @dashboard.percentile(9) %><li>
|
6
|
+
<li>50th Percentile: <%= @dashboard.percentile(5) %><li>
|
7
|
+
</ul>
|
8
|
+
|
9
|
+
<h2>Slow queriess</h2>
|
10
|
+
<table>
|
11
|
+
<tr>
|
12
|
+
<th>SQL</th>
|
13
|
+
<th>Location</th>
|
14
|
+
<th>Time</th>
|
15
|
+
</tr>
|
16
|
+
<% @dashboard.slow_sql_queries.each do |metric| %>
|
17
|
+
<tr>
|
18
|
+
<td><%= metric.sql_query %></td>
|
19
|
+
<td><%= metric.location %></td>
|
20
|
+
<td><%= metric.value %></td>
|
21
|
+
</tr>
|
22
|
+
<% end %>
|
23
|
+
<table>
|
data/config/routes.rb
ADDED
@@ -0,0 +1,13 @@
|
|
1
|
+
class CreateActiveMonitoringMetrics < ActiveRecord::Migration[6.0]
|
2
|
+
def change
|
3
|
+
create_table :active_monitoring_metrics do |t|
|
4
|
+
t.string :name
|
5
|
+
t.string :request_id
|
6
|
+
t.string :location
|
7
|
+
t.string :sql_query
|
8
|
+
t.integer :value
|
9
|
+
|
10
|
+
t.timestamps
|
11
|
+
end
|
12
|
+
end
|
13
|
+
end
|
@@ -0,0 +1,19 @@
|
|
1
|
+
module ActiveMonitoring
|
2
|
+
module CoreExtensions
|
3
|
+
module ActionController
|
4
|
+
module Instrumentation
|
5
|
+
def process_action(*)
|
6
|
+
payload = {
|
7
|
+
controller: self.class.name,
|
8
|
+
action: action_name,
|
9
|
+
request_id: request.uuid
|
10
|
+
}
|
11
|
+
::ActiveMonitoring::Notifications.instrument("start_processing.action_controller", payload)
|
12
|
+
::ActiveMonitoring::Notifications.instrument("process_action.action_controller", payload) do
|
13
|
+
super
|
14
|
+
end
|
15
|
+
end
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
19
|
+
end
|
@@ -0,0 +1,13 @@
|
|
1
|
+
module ActiveMonitoring
|
2
|
+
module CoreExtensions
|
3
|
+
module ActiveRecord
|
4
|
+
module Instrumentation
|
5
|
+
def log(sql, name = "SQL", binds = [], type_casted_binds = [], statement_name = nil)
|
6
|
+
::ActiveMonitoring::Notifications.instrument("sql.active_record", sql: sql, name: name) do
|
7
|
+
super
|
8
|
+
end
|
9
|
+
end
|
10
|
+
end
|
11
|
+
end
|
12
|
+
end
|
13
|
+
end
|
@@ -0,0 +1,31 @@
|
|
1
|
+
require_relative "core_extensions/action_controller"
|
2
|
+
require_relative "core_extensions/active_record"
|
3
|
+
require_relative "sql_query"
|
4
|
+
require_relative "current"
|
5
|
+
|
6
|
+
module ActiveMonitoring
|
7
|
+
class Current
|
8
|
+
class << self
|
9
|
+
def request_id
|
10
|
+
store[:request_id]
|
11
|
+
end
|
12
|
+
|
13
|
+
def request_id=(value)
|
14
|
+
store[:request_id] = value
|
15
|
+
end
|
16
|
+
|
17
|
+
def location
|
18
|
+
store[:location]
|
19
|
+
end
|
20
|
+
|
21
|
+
def location=(value)
|
22
|
+
store[:location] = value
|
23
|
+
end
|
24
|
+
|
25
|
+
def store
|
26
|
+
Thread.current[:active_monitoring_store] ||= {}
|
27
|
+
Thread.current[:active_monitoring_store]
|
28
|
+
end
|
29
|
+
end
|
30
|
+
end
|
31
|
+
end
|
@@ -0,0 +1,47 @@
|
|
1
|
+
require_relative "core_extensions/action_controller"
|
2
|
+
require_relative "core_extensions/active_record"
|
3
|
+
require_relative "sql_query"
|
4
|
+
require_relative "current"
|
5
|
+
|
6
|
+
module ActiveMonitoring
|
7
|
+
class Engine < ::Rails::Engine
|
8
|
+
isolate_namespace ActiveMonitoring
|
9
|
+
|
10
|
+
ActiveSupport.on_load(:active_record) do
|
11
|
+
ActiveRecord::ConnectionAdapters::AbstractAdapter.prepend(::ActiveMonitoring::CoreExtensions::ActiveRecord::Instrumentation)
|
12
|
+
end
|
13
|
+
|
14
|
+
ActiveSupport.on_load(:action_controller) do
|
15
|
+
ActionController::Base.prepend(::ActiveMonitoring::CoreExtensions::ActionController::Instrumentation)
|
16
|
+
|
17
|
+
ActiveMonitoring::Notifications.subscribe("start_processing.action_controller") do |_, _, _, _, payload|
|
18
|
+
Current.request_id = payload[:request_id]
|
19
|
+
Current.location = "#{payload[:controller]}##{payload[:action]}"
|
20
|
+
end
|
21
|
+
|
22
|
+
ActiveMonitoring::Notifications.subscribe("process_action.action_controller") do |name, start, finish, _id, payload|
|
23
|
+
Metric.create(
|
24
|
+
name: name,
|
25
|
+
value: finish - start,
|
26
|
+
request_id: payload[:request_id],
|
27
|
+
location: "#{payload[:controller]}##{payload[:action]}",
|
28
|
+
created_at: finish
|
29
|
+
)
|
30
|
+
end
|
31
|
+
|
32
|
+
ActiveMonitoring::Notifications.subscribe("sql.active_record") do |name, start, finish, _id, payload|
|
33
|
+
query = ActiveMonitoring::SqlQuery.new(name: payload[:name].dup, query: payload[:sql].dup)
|
34
|
+
if query.track?
|
35
|
+
Metric.create(
|
36
|
+
name: name,
|
37
|
+
value: finish - start,
|
38
|
+
request_id: Current.request_id,
|
39
|
+
sql_query: query.normalized_query,
|
40
|
+
location: Current.location,
|
41
|
+
created_at: finish
|
42
|
+
)
|
43
|
+
end
|
44
|
+
end
|
45
|
+
end
|
46
|
+
end
|
47
|
+
end
|
@@ -0,0 +1,19 @@
|
|
1
|
+
require_relative "notifier"
|
2
|
+
|
3
|
+
module ActiveMonitoring
|
4
|
+
class Notifications
|
5
|
+
class << self
|
6
|
+
attr_accessor :notifier
|
7
|
+
|
8
|
+
def subscribe(name, &block)
|
9
|
+
notifier.subscribe(name, &block)
|
10
|
+
end
|
11
|
+
|
12
|
+
def instrument(name, payload = {}, &block)
|
13
|
+
notifier.instrument(name, payload, &block)
|
14
|
+
end
|
15
|
+
end
|
16
|
+
|
17
|
+
self.notifier = ActiveMonitoring::Notifier.new
|
18
|
+
end
|
19
|
+
end
|
@@ -0,0 +1,33 @@
|
|
1
|
+
module ActiveMonitoring
|
2
|
+
class Notifier
|
3
|
+
def initialize
|
4
|
+
@subscribers = {}
|
5
|
+
end
|
6
|
+
|
7
|
+
def subscribe(name, &block)
|
8
|
+
subscribers[name] ||= []
|
9
|
+
subscribers[name] << block
|
10
|
+
end
|
11
|
+
|
12
|
+
def instrument(name, payload)
|
13
|
+
id = SecureRandom.hex(10)
|
14
|
+
start = Time.current
|
15
|
+
result = yield if block_given?
|
16
|
+
finish = Time.current
|
17
|
+
|
18
|
+
subscribers_for(name).each do |callback|
|
19
|
+
callback.call(name, start, finish, id, payload)
|
20
|
+
end
|
21
|
+
|
22
|
+
result
|
23
|
+
end
|
24
|
+
|
25
|
+
private
|
26
|
+
|
27
|
+
attr_reader :subscribers
|
28
|
+
|
29
|
+
def subscribers_for(name)
|
30
|
+
subscribers[name].to_a
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
@@ -0,0 +1,23 @@
|
|
1
|
+
module ActiveMonitoring
|
2
|
+
class SqlNormalizer
|
3
|
+
def initialize(query:)
|
4
|
+
@query = query
|
5
|
+
end
|
6
|
+
|
7
|
+
def to_s
|
8
|
+
query.squish!
|
9
|
+
query.gsub!(/(\s(=|>|<|>=|<=|<>|!=)\s)('[^']+'|[\$\+\-\w\.]+)/, '\1xxx')
|
10
|
+
query.gsub!(/(\sIN\s)\([^\(\)]+\)/i, '\1(xxx)')
|
11
|
+
regex = /(\sBETWEEN\s)('[^']+'|[\+\-\w\.]+)(\sAND\s)('[^']+'|[\+\-\w\.]+)/i
|
12
|
+
query.gsub!(regex, '\1xxx\3xxx')
|
13
|
+
query.gsub!(/(\sVALUES\s)\(.+\)/i, '\1(xxx)')
|
14
|
+
query.gsub!(/(\s(LIKE|ILIKE|SIMILAR TO|NOT SIMILAR TO)\s)('[^']+')/i, '\1xxx')
|
15
|
+
query.gsub!(/(\s(LIMIT|OFFSET)\s)(\d+)/i, '\1xxx')
|
16
|
+
query
|
17
|
+
end
|
18
|
+
|
19
|
+
private
|
20
|
+
|
21
|
+
attr_reader :query
|
22
|
+
end
|
23
|
+
end
|
@@ -0,0 +1,23 @@
|
|
1
|
+
require_relative "sql_tracker"
|
2
|
+
require_relative "sql_normalizer"
|
3
|
+
|
4
|
+
module ActiveMonitoring
|
5
|
+
class SqlQuery
|
6
|
+
def initialize(query:, name:)
|
7
|
+
@query = query
|
8
|
+
@name = name
|
9
|
+
end
|
10
|
+
|
11
|
+
def track?
|
12
|
+
SqlTracker.new(query: query, name: name).track?
|
13
|
+
end
|
14
|
+
|
15
|
+
def normalized_query
|
16
|
+
SqlNormalizer.new(query: query).to_s
|
17
|
+
end
|
18
|
+
|
19
|
+
private
|
20
|
+
|
21
|
+
attr_reader :query, :name
|
22
|
+
end
|
23
|
+
end
|
@@ -0,0 +1,33 @@
|
|
1
|
+
module ActiveMonitoring
|
2
|
+
class SqlTracker
|
3
|
+
TRACKED_SQL_COMMANDS = %w(SELECT INSERT UPDATE DELETE).freeze
|
4
|
+
UNTRACKED_NAMES = %w(SCHEMA).freeze
|
5
|
+
UNTRACKED_TABLES = %w(
|
6
|
+
SCHEMA_MIGRATIONS
|
7
|
+
SQLITE_MASTER
|
8
|
+
ACTIVE_MONITORING_METRICS
|
9
|
+
SQLITE_TEMP_MASTER
|
10
|
+
SQLITE_VERSION
|
11
|
+
AR_INTERNAL_METADATA
|
12
|
+
).freeze
|
13
|
+
|
14
|
+
def initialize(query:, name:)
|
15
|
+
@query = query.to_s.upcase
|
16
|
+
@name = name.to_s.upcase
|
17
|
+
end
|
18
|
+
|
19
|
+
def track?
|
20
|
+
query.start_with?(*TRACKED_SQL_COMMANDS) &&
|
21
|
+
!name.start_with?(*UNTRACKED_NAMES) &&
|
22
|
+
!untracked_tables?
|
23
|
+
end
|
24
|
+
|
25
|
+
private
|
26
|
+
|
27
|
+
attr_reader :query, :name
|
28
|
+
|
29
|
+
def untracked_tables?
|
30
|
+
UNTRACKED_TABLES.any? { |table| query.include?(table) }
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
metadata
ADDED
@@ -0,0 +1,158 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: active_monitoring
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.1.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Christian Bruckmayer
|
8
|
+
autorequire:
|
9
|
+
bindir: bin
|
10
|
+
cert_chain: []
|
11
|
+
date: 2020-04-17 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: rails
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - "~>"
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: 6.0.2
|
20
|
+
- - ">="
|
21
|
+
- !ruby/object:Gem::Version
|
22
|
+
version: 6.0.2.2
|
23
|
+
type: :runtime
|
24
|
+
prerelease: false
|
25
|
+
version_requirements: !ruby/object:Gem::Requirement
|
26
|
+
requirements:
|
27
|
+
- - "~>"
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
version: 6.0.2
|
30
|
+
- - ">="
|
31
|
+
- !ruby/object:Gem::Version
|
32
|
+
version: 6.0.2.2
|
33
|
+
- !ruby/object:Gem::Dependency
|
34
|
+
name: byebug
|
35
|
+
requirement: !ruby/object:Gem::Requirement
|
36
|
+
requirements:
|
37
|
+
- - ">="
|
38
|
+
- !ruby/object:Gem::Version
|
39
|
+
version: '0'
|
40
|
+
type: :development
|
41
|
+
prerelease: false
|
42
|
+
version_requirements: !ruby/object:Gem::Requirement
|
43
|
+
requirements:
|
44
|
+
- - ">="
|
45
|
+
- !ruby/object:Gem::Version
|
46
|
+
version: '0'
|
47
|
+
- !ruby/object:Gem::Dependency
|
48
|
+
name: rspec-rails
|
49
|
+
requirement: !ruby/object:Gem::Requirement
|
50
|
+
requirements:
|
51
|
+
- - ">="
|
52
|
+
- !ruby/object:Gem::Version
|
53
|
+
version: '0'
|
54
|
+
type: :development
|
55
|
+
prerelease: false
|
56
|
+
version_requirements: !ruby/object:Gem::Requirement
|
57
|
+
requirements:
|
58
|
+
- - ">="
|
59
|
+
- !ruby/object:Gem::Version
|
60
|
+
version: '0'
|
61
|
+
- !ruby/object:Gem::Dependency
|
62
|
+
name: rubocop-performance
|
63
|
+
requirement: !ruby/object:Gem::Requirement
|
64
|
+
requirements:
|
65
|
+
- - ">="
|
66
|
+
- !ruby/object:Gem::Version
|
67
|
+
version: '0'
|
68
|
+
type: :development
|
69
|
+
prerelease: false
|
70
|
+
version_requirements: !ruby/object:Gem::Requirement
|
71
|
+
requirements:
|
72
|
+
- - ">="
|
73
|
+
- !ruby/object:Gem::Version
|
74
|
+
version: '0'
|
75
|
+
- !ruby/object:Gem::Dependency
|
76
|
+
name: rubocop-rails
|
77
|
+
requirement: !ruby/object:Gem::Requirement
|
78
|
+
requirements:
|
79
|
+
- - ">="
|
80
|
+
- !ruby/object:Gem::Version
|
81
|
+
version: '0'
|
82
|
+
type: :development
|
83
|
+
prerelease: false
|
84
|
+
version_requirements: !ruby/object:Gem::Requirement
|
85
|
+
requirements:
|
86
|
+
- - ">="
|
87
|
+
- !ruby/object:Gem::Version
|
88
|
+
version: '0'
|
89
|
+
- !ruby/object:Gem::Dependency
|
90
|
+
name: sqlite3
|
91
|
+
requirement: !ruby/object:Gem::Requirement
|
92
|
+
requirements:
|
93
|
+
- - ">="
|
94
|
+
- !ruby/object:Gem::Version
|
95
|
+
version: '0'
|
96
|
+
type: :development
|
97
|
+
prerelease: false
|
98
|
+
version_requirements: !ruby/object:Gem::Requirement
|
99
|
+
requirements:
|
100
|
+
- - ">="
|
101
|
+
- !ruby/object:Gem::Version
|
102
|
+
version: '0'
|
103
|
+
description: his is an example gem used in my RailsConf 2020 talk. Please don't use
|
104
|
+
in production! https://bruckmayer.net/rails-conf-2020
|
105
|
+
email:
|
106
|
+
- christian@bruckmayer.net
|
107
|
+
executables: []
|
108
|
+
extensions: []
|
109
|
+
extra_rdoc_files: []
|
110
|
+
files:
|
111
|
+
- MIT-LICENSE
|
112
|
+
- README.md
|
113
|
+
- Rakefile
|
114
|
+
- app/controllers/active_monitoring/application_controller.rb
|
115
|
+
- app/controllers/active_monitoring/dashboard_controller.rb
|
116
|
+
- app/models/active_monitoring/application_record.rb
|
117
|
+
- app/models/active_monitoring/dashboard.rb
|
118
|
+
- app/models/active_monitoring/metric.rb
|
119
|
+
- app/views/active_monitoring/dashboard/show.html.erb
|
120
|
+
- app/views/layouts/active_monitoring/application.html.erb
|
121
|
+
- config/routes.rb
|
122
|
+
- db/migrate/20200401191741_create_active_monitoring_metrics.rb
|
123
|
+
- lib/active_monitoring.rb
|
124
|
+
- lib/active_monitoring/core_extensions/action_controller.rb
|
125
|
+
- lib/active_monitoring/core_extensions/active_record.rb
|
126
|
+
- lib/active_monitoring/current.rb
|
127
|
+
- lib/active_monitoring/engine.rb
|
128
|
+
- lib/active_monitoring/notifications.rb
|
129
|
+
- lib/active_monitoring/notifier.rb
|
130
|
+
- lib/active_monitoring/sql_normalizer.rb
|
131
|
+
- lib/active_monitoring/sql_query.rb
|
132
|
+
- lib/active_monitoring/sql_tracker.rb
|
133
|
+
- lib/active_monitoring/version.rb
|
134
|
+
homepage: https://github.com/ChrisBr/active_monitoring
|
135
|
+
licenses:
|
136
|
+
- MIT
|
137
|
+
metadata: {}
|
138
|
+
post_install_message:
|
139
|
+
rdoc_options: []
|
140
|
+
require_paths:
|
141
|
+
- lib
|
142
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
143
|
+
requirements:
|
144
|
+
- - ">="
|
145
|
+
- !ruby/object:Gem::Version
|
146
|
+
version: '0'
|
147
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
148
|
+
requirements:
|
149
|
+
- - ">="
|
150
|
+
- !ruby/object:Gem::Version
|
151
|
+
version: '0'
|
152
|
+
requirements: []
|
153
|
+
rubygems_version: 3.1.2
|
154
|
+
signing_key:
|
155
|
+
specification_version: 4
|
156
|
+
summary: This is an example gem used in my RailsConf 2020 talk. Please don't use in
|
157
|
+
production!
|
158
|
+
test_files: []
|