active_monitoring 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/MIT-LICENSE +20 -0
- data/README.md +182 -0
- data/Rakefile +22 -0
- data/app/controllers/active_monitoring/application_controller.rb +5 -0
- data/app/controllers/active_monitoring/dashboard_controller.rb +9 -0
- data/app/models/active_monitoring/application_record.rb +5 -0
- data/app/models/active_monitoring/dashboard.rb +33 -0
- data/app/models/active_monitoring/metric.rb +7 -0
- data/app/views/active_monitoring/dashboard/show.html.erb +23 -0
- data/app/views/layouts/active_monitoring/application.html.erb +13 -0
- data/config/routes.rb +3 -0
- data/db/migrate/20200401191741_create_active_monitoring_metrics.rb +13 -0
- data/lib/active_monitoring.rb +6 -0
- data/lib/active_monitoring/core_extensions/action_controller.rb +19 -0
- data/lib/active_monitoring/core_extensions/active_record.rb +13 -0
- data/lib/active_monitoring/current.rb +31 -0
- data/lib/active_monitoring/engine.rb +47 -0
- data/lib/active_monitoring/notifications.rb +19 -0
- data/lib/active_monitoring/notifier.rb +33 -0
- data/lib/active_monitoring/sql_normalizer.rb +23 -0
- data/lib/active_monitoring/sql_query.rb +23 -0
- data/lib/active_monitoring/sql_tracker.rb +33 -0
- data/lib/active_monitoring/version.rb +3 -0
- metadata +158 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: d713464ea142fd4d40db7b8b12b786f3fd21c85d052cc9fbe2eb9fcb5ece2a87
|
4
|
+
data.tar.gz: 94c592b0e41fe8be9c609525eb429979cb67f382179baaae902abcd4b256b031
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 56afc9fc77d56d52437671acfe644308e83f13d1a0fe91b2b38ded82b0f007156eec4784d75d6ba041c57bed97ba301a3334bb1dd88a8ce4a4e824993d01ff02
|
7
|
+
data.tar.gz: b1b763e859633c3255693fa5b3453cadccf6184788f5482f18913bbcc8c7cc8dc88d1429fe4a9fa50b721f646ca39a4befb36fb20fc1f76a27122803a87b63fb
|
data/MIT-LICENSE
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
Copyright 2020 Christian Bruckmayer
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
4
|
+
a copy of this software and associated documentation files (the
|
5
|
+
"Software"), to deal in the Software without restriction, including
|
6
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
7
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
8
|
+
permit persons to whom the Software is furnished to do so, subject to
|
9
|
+
the following conditions:
|
10
|
+
|
11
|
+
The above copyright notice and this permission notice shall be
|
12
|
+
included in all copies or substantial portions of the Software.
|
13
|
+
|
14
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
15
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
16
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
17
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
18
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
19
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
20
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,182 @@
|
|
1
|
+
# ActiveMonitoring
|
2
|
+
|
3
|
+
I gave this talk at [RailsConf 2020 Couch Edition](https://railsconf.com/) and this blog article contains the source code and step by step tutorial.
|
4
|
+
The talk and tutorial is inspired by my work on [influxdb-rails](https://github.com/influxdata/influxdb-rails) and contains several patterns and learnings from it.
|
5
|
+
|
6
|
+
By the end of this tutorial, we will have a very basic Performance Monitoring tool which measures the response and SQL query time of a Rails application.
|
7
|
+
|
8
|
+
## Abstract
|
9
|
+
> Building a Performance Analytics Tool with ActiveSupport
|
10
|
+
|
11
|
+
> Setting up a performance analytics tool like NewRelic or Skylight is one of the first things many developers do in a new project. However, have you ever wondered how these tools work under the hood?
|
12
|
+
|
13
|
+
> In this talk, we will build a basic performance analytics tool from scratch. We will deep dive into ActiveSupport instrumentations to collect, group and normalise metrics from your app. To persist these metrics, we will use a time series database and visualise the data on a dashboard. By the end of the talk, you will know how your favourite performance analytics tool works.
|
14
|
+
|
15
|
+
## Chapter 1: Collecting data
|
16
|
+
### Notification Framework
|
17
|
+
As a first step we need to implement a notification framework which we can use to instrument and subscribe to performance metrics.
|
18
|
+
|
19
|
+
A basic example of the functionality of our notification framework would be this code:
|
20
|
+
```ruby
|
21
|
+
scenario "It can susbcribe and instrument events" do
|
22
|
+
events = []
|
23
|
+
ActiveMonitoring::Notifications.subscribe("test_event") do |name, start, finish, id, payload|
|
24
|
+
events << { name: name, start: start, finish: finish, id: id, payload: payload }
|
25
|
+
end
|
26
|
+
|
27
|
+
ActiveMonitoring::Notifications.instrument("test_event", payload: :payload) do
|
28
|
+
1 + 1
|
29
|
+
end
|
30
|
+
|
31
|
+
expect(metrics[0].name).to eq("test_event")
|
32
|
+
expect(metrics[0].payload).to include(payload: :payload)
|
33
|
+
end
|
34
|
+
```
|
35
|
+
|
36
|
+
The implementation can be found in commit [32c3](https://github.com/ChrisBr/active_monitoring/commit/32c32dcec7529ac7a8e0125b1aaab28cd4219363).
|
37
|
+
For the sake of simplicity, this framework is not thread safe of course.
|
38
|
+
|
39
|
+
This framework is inspired by [ActiveSupport Notifications](https://edgeguides.rubyonrails.org/active_support_instrumentation.html) which already provides various hooks into Rails internals.
|
40
|
+
The implementatition of Active Support Notifications can be found [here](https://github.com/rails/rails/tree/master/activesupport/lib/active_support/notifications).
|
41
|
+
|
42
|
+
### Instrument events
|
43
|
+
[ActiveSupport Notifications](https://edgeguides.rubyonrails.org/active_support_instrumentation.html) provides already
|
44
|
+
several hooks to subscribe to e.g. processing of controller actions, SQL queries or view rendering.
|
45
|
+
However, Rails does not provide hooks for external libraries like Redis or Memcache.
|
46
|
+
In [ef20](https://github.com/ChrisBr/active_monitoring/commit/ef200ae5bf58f9afd2a69003e2c3620d0d9c86eb) and [46c3](https://github.com/ChrisBr/active_monitoring/commit/46c338f8fa5870bae7dc0e251e637d1d4be551db) we monkey patch Rails
|
47
|
+
to subscribe to controller actions and SQL queries.
|
48
|
+
|
49
|
+
## Chapter 2: Storage
|
50
|
+
In chapter two we discussed an appropriate storage engine for our data.
|
51
|
+
Our data would look for example like this
|
52
|
+
|
53
|
+
| Timestamp | Hook | Location | Request ID | Value |
|
54
|
+
| ------------- | ------------- |------------- |------------- |------------- |
|
55
|
+
| 01.04.2020 00:31:30 CET | process_action | BooksController#show | cf6d6dcd | 100 ms
|
56
|
+
| 01.04.2020 00:32:30 CET | sql | BooksController#show | c7c7c841 | 80 ms
|
57
|
+
|
58
|
+
This data looks tabular, so using a relational database to store it would make sense.
|
59
|
+
However, there are several downsides to this approach.
|
60
|
+
|
61
|
+
### Relational Database
|
62
|
+
|
63
|
+
Relational databases perform very well with applications which map to the basic CRUD operations of the database like a Rails controller.
|
64
|
+
|
65
|
+
```ruby
|
66
|
+
class BooksController < ApplicationController
|
67
|
+
def new; end
|
68
|
+
def create; end
|
69
|
+
def show; end
|
70
|
+
def edit; end
|
71
|
+
def update; end
|
72
|
+
def destroy; end
|
73
|
+
end
|
74
|
+
```
|
75
|
+
|
76
|
+
However, our plugin does not really use all of these traditional database operations or in a different way we would in a normal Rails app.
|
77
|
+
|
78
|
+
#### Create
|
79
|
+
For each request, we will write one process action and several SQL query data points.
|
80
|
+
If we extend this application to also cover view render times or background jobs, it's not unlikely we would write 10+ metrics per request.
|
81
|
+
Depending of how many requests your app does process, we could easily create millions of metrics per year.
|
82
|
+
|
83
|
+
#### Read
|
84
|
+
We will read the metrics in a dashboard for a certain time range like today or last three hours but not randomly.
|
85
|
+
|
86
|
+
#### Update
|
87
|
+
We will almost never go back to update on of the metrics.
|
88
|
+
|
89
|
+
#### Delete
|
90
|
+
We will only delete metrics in batches but not single metrics.
|
91
|
+
For instance, we are only interested in metrics for the last three months so once a month we can do a 'garbage collection' and delete metrics older than three months.
|
92
|
+
|
93
|
+
### Time Series Database
|
94
|
+
|
95
|
+
With this all said, a relational database might not be the best storage engine for this kind of data as
|
96
|
+
|
97
|
+
* We will heavily write to the database which would cause to frequently create new pages in a [B-Tree](https://en.wikipedia.org/wiki/B-tree) like most relational databases use internally
|
98
|
+
* We would need to implement some sort of garbage collection to remove old data
|
99
|
+
* Compression of data would not be very efficient
|
100
|
+
|
101
|
+
For this kind of data, a [log based storage engine](https://en.wikipedia.org/wiki/Log-structured_merge-tree) like most [Time Series databases](https://db-engines.com/en/ranking/time+series+dbms) implement it would be more efficient.
|
102
|
+
|
103
|
+
However, for the sake of simplicity, we store our metrics in a relational database in this tutorial, the implementation can be found in [5967](https://github.com/ChrisBr/active_monitoring/commit/5967190a1484009476f2378c02e3e0ea96d21624).
|
104
|
+
|
105
|
+
## Chapter 3: Cleaning, Normalizing and Grouping
|
106
|
+
In the previous chapter we already stored the controller metrics.
|
107
|
+
Before we can store SQL metrics, we need to do some more additional work.
|
108
|
+
|
109
|
+
### Cleaning
|
110
|
+
ActiveRecord does a lot of 'magic' behind the scenes to abstract persisting objects to the database.
|
111
|
+
Some of this 'magic' requires to execute additional database queries, for instance
|
112
|
+
|
113
|
+
* Current version of the database engine
|
114
|
+
* Which migrations did already run
|
115
|
+
* Which environment
|
116
|
+
|
117
|
+
We're only interested to store application metrics so we need to filter these queries out.
|
118
|
+
The implementation can be found in [156d](https://github.com/ChrisBr/active_monitoring/commit/156d8f7fb0bd3b013524a20effc65dad05506b3d).
|
119
|
+
|
120
|
+
### Normalizing
|
121
|
+
Depending on the database adapter you use, the queries might contain values.
|
122
|
+
To group the same queries together, we need to normalize these queries.
|
123
|
+
|
124
|
+
```SQL
|
125
|
+
SELECT * FROM books WHERE id = 1;
|
126
|
+
SELECT * FROM books WHERE id = 2;
|
127
|
+
```
|
128
|
+
|
129
|
+
For our performance monitoring tool, these two queries should be treated the same and we need to normalize them to
|
130
|
+
```SQL
|
131
|
+
SELECT * FROM books WHERE id = xxx;
|
132
|
+
```
|
133
|
+
|
134
|
+
In [66a8](https://github.com/ChrisBr/active_monitoring/commit/66a833a46b40072a8888fb6b6c6bb02ee60bda17) we implement a simple query normalizer.
|
135
|
+
|
136
|
+
### Grouping
|
137
|
+
ActiveRecord is a standalone framework, therefore the payload of the SQL event does not contain a `request_id`.
|
138
|
+
We can use ActiveRecord for instance in migrations or background jobs so it is perfectly valid to use it outside of a request response cycle.
|
139
|
+
However, in our Performance Monitoring tool we would like to group requests and SQL queries together so we can see if a query causes a slow response.
|
140
|
+
|
141
|
+
Luckily, we also implemented a `start_processing` event in [ef20](https://github.com/ChrisBr/active_monitoring/commit/ef200ae5bf58f9afd2a69003e2c3620d0d9c86eb).
|
142
|
+
We can now subscribe to this event and set the `request_id` and `location` in a cache which we later read
|
143
|
+
in when writing the SQL metrics.
|
144
|
+
In [b1f1](https://github.com/ChrisBr/active_monitoring/commit/b1f1847270935beacc236bc54535ced1eb83c5ef) we implement a `CurrentAttributes` class and eventually write the SQL metrics.
|
145
|
+
|
146
|
+
ActiveSupport ships with [CurrentAttributes](https://github.com/rails/rails/blob/157920aead96865e3135f496c09ace607d5620dc/activesupport/lib/active_support/current_attributes.rb) out of the box since Rails 5.
|
147
|
+
|
148
|
+
## Chapter 4: Visualization
|
149
|
+
So we now have several metrics in our data base written which would look something like
|
150
|
+
|
151
|
+
| Timestamp | Hook | Location | Request ID | Value |
|
152
|
+
| ------------- | ------------- |------------- |------------- |------------- |
|
153
|
+
| 01.04.2020 00:31:30 CET | process_action | BooksController#show | cf6d6dcd | 100 ms
|
154
|
+
| 01.04.2020 00:32:30 CET | sql | BooksController#show | c7c7c841 | 80 ms
|
155
|
+
|
156
|
+
It would be very hard to spot now problems or discover pattern so eventually we need to visualize our collected data.
|
157
|
+
In [cfca](https://github.com/ChrisBr/active_monitoring/commit/cfcafd2f4664e6927e7d297a4b7da2e18bdd5205) we implement a dashboard to show percentiles and the slowest queries of our Rails app.
|
158
|
+
A very good blog article about data visualization for performance metrics from Richard Schneemann can be found [here](https://www.schneems.com/2020/03/17/lies-damned-lies-and-averages-perc50-perc95-explained-for-programmers/).
|
159
|
+
|
160
|
+
In a real world application, I would strongly recommend to use a dashboard software like [Grafana](https://github.com/grafana/grafana).
|
161
|
+
|
162
|
+
## Summary
|
163
|
+
Congratulations, we implemented a very basic Performance Monitoring tool in just a few hundred lines of code now.
|
164
|
+
We deep dived into [ActiveSupport Notifications](https://edgeguides.rubyonrails.org/active_support_instrumentation.html) and hooked into Rails events to write request and SQL query metrics in our data storage.
|
165
|
+
As data storage, we compared relational databases with time series databases like InfluxDB.
|
166
|
+
Before we could visualize our data, we needed to clean, normalize and group the metrics.
|
167
|
+
|
168
|
+
From here, we can easily add more metrics like [rendering of views](https://edgeguides.rubyonrails.org/active_support_instrumentation.html#action-view),
|
169
|
+
[caching](https://edgeguides.rubyonrails.org/active_support_instrumentation.html#active-support) or [background jobs](https://edgeguides.rubyonrails.org/active_support_instrumentation.html#active-job).
|
170
|
+
|
171
|
+
As initially mentioned, this tutorial is heavily influenced by our work on https://github.com/influxdata/influxdb-rails.
|
172
|
+
If this made you curious, we always look for new contributors.
|
173
|
+
|
174
|
+
## Further information
|
175
|
+
* [Profiling and Benchmarking 101 by Nate Berkopec](https://youtu.be/XL51vf-XBTs)
|
176
|
+
* [Lies, Damned Lies, and Averages: Perc50, Perc95 explained for Programmers by Richard Schneeman](https://www.schneems.com/2020/03/17/lies-damned-lies-and-averages-perc50-perc95-explained-for-programmers/)
|
177
|
+
* https://github.com/influxdata/influxdb-ruby
|
178
|
+
* https://github.com/influxdata/influxdb-rails
|
179
|
+
* https://docs.influxdata.com/influxdb/v1.7/concepts/storage_engine/
|
180
|
+
|
181
|
+
## License
|
182
|
+
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
|
data/Rakefile
ADDED
@@ -0,0 +1,22 @@
|
|
1
|
+
begin
|
2
|
+
require "bundler/setup"
|
3
|
+
rescue LoadError
|
4
|
+
puts "You must `gem install bundler` and `bundle install` to run rake tasks"
|
5
|
+
end
|
6
|
+
|
7
|
+
require "rdoc/task"
|
8
|
+
|
9
|
+
RDoc::Task.new(:rdoc) do |rdoc|
|
10
|
+
rdoc.rdoc_dir = "rdoc"
|
11
|
+
rdoc.title = "ActiveMonitoring"
|
12
|
+
rdoc.options << "--line-numbers"
|
13
|
+
rdoc.rdoc_files.include("README.md")
|
14
|
+
rdoc.rdoc_files.include("lib/**/*.rb")
|
15
|
+
end
|
16
|
+
|
17
|
+
APP_RAKEFILE = File.expand_path("spec/dummy/Rakefile", __dir__)
|
18
|
+
load "rails/tasks/engine.rake"
|
19
|
+
|
20
|
+
load "rails/tasks/statistics.rake"
|
21
|
+
|
22
|
+
require "bundler/gem_tasks"
|
@@ -0,0 +1,33 @@
|
|
1
|
+
module ActiveMonitoring
|
2
|
+
class Dashboard
|
3
|
+
LIMIT = 10
|
4
|
+
|
5
|
+
def initialize(date = Date.current)
|
6
|
+
@date = date
|
7
|
+
end
|
8
|
+
|
9
|
+
def percentile(value)
|
10
|
+
response_metrics.percentile(value)
|
11
|
+
end
|
12
|
+
|
13
|
+
def slow_sql_queries
|
14
|
+
sql_metrics.order(:value).limit(LIMIT)
|
15
|
+
end
|
16
|
+
|
17
|
+
private
|
18
|
+
|
19
|
+
attr_reader :date
|
20
|
+
|
21
|
+
def sql_metrics
|
22
|
+
metrics.where(name: "sql.active_record")
|
23
|
+
end
|
24
|
+
|
25
|
+
def response_metrics
|
26
|
+
metrics.where(name: "process_action.action_controller")
|
27
|
+
end
|
28
|
+
|
29
|
+
def metrics
|
30
|
+
Metric.where(created_at: date.beginning_of_day..date.end_of_day)
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
@@ -0,0 +1,23 @@
|
|
1
|
+
<h1>Dashboard</h1>
|
2
|
+
|
3
|
+
<h2>Response time</h2>
|
4
|
+
<ul>
|
5
|
+
<li>90th Percentile: <%= @dashboard.percentile(9) %><li>
|
6
|
+
<li>50th Percentile: <%= @dashboard.percentile(5) %><li>
|
7
|
+
</ul>
|
8
|
+
|
9
|
+
<h2>Slow queriess</h2>
|
10
|
+
<table>
|
11
|
+
<tr>
|
12
|
+
<th>SQL</th>
|
13
|
+
<th>Location</th>
|
14
|
+
<th>Time</th>
|
15
|
+
</tr>
|
16
|
+
<% @dashboard.slow_sql_queries.each do |metric| %>
|
17
|
+
<tr>
|
18
|
+
<td><%= metric.sql_query %></td>
|
19
|
+
<td><%= metric.location %></td>
|
20
|
+
<td><%= metric.value %></td>
|
21
|
+
</tr>
|
22
|
+
<% end %>
|
23
|
+
<table>
|
data/config/routes.rb
ADDED
@@ -0,0 +1,13 @@
|
|
1
|
+
class CreateActiveMonitoringMetrics < ActiveRecord::Migration[6.0]
|
2
|
+
def change
|
3
|
+
create_table :active_monitoring_metrics do |t|
|
4
|
+
t.string :name
|
5
|
+
t.string :request_id
|
6
|
+
t.string :location
|
7
|
+
t.string :sql_query
|
8
|
+
t.integer :value
|
9
|
+
|
10
|
+
t.timestamps
|
11
|
+
end
|
12
|
+
end
|
13
|
+
end
|
@@ -0,0 +1,19 @@
|
|
1
|
+
module ActiveMonitoring
|
2
|
+
module CoreExtensions
|
3
|
+
module ActionController
|
4
|
+
module Instrumentation
|
5
|
+
def process_action(*)
|
6
|
+
payload = {
|
7
|
+
controller: self.class.name,
|
8
|
+
action: action_name,
|
9
|
+
request_id: request.uuid
|
10
|
+
}
|
11
|
+
::ActiveMonitoring::Notifications.instrument("start_processing.action_controller", payload)
|
12
|
+
::ActiveMonitoring::Notifications.instrument("process_action.action_controller", payload) do
|
13
|
+
super
|
14
|
+
end
|
15
|
+
end
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
19
|
+
end
|
@@ -0,0 +1,13 @@
|
|
1
|
+
module ActiveMonitoring
|
2
|
+
module CoreExtensions
|
3
|
+
module ActiveRecord
|
4
|
+
module Instrumentation
|
5
|
+
def log(sql, name = "SQL", binds = [], type_casted_binds = [], statement_name = nil)
|
6
|
+
::ActiveMonitoring::Notifications.instrument("sql.active_record", sql: sql, name: name) do
|
7
|
+
super
|
8
|
+
end
|
9
|
+
end
|
10
|
+
end
|
11
|
+
end
|
12
|
+
end
|
13
|
+
end
|
@@ -0,0 +1,31 @@
|
|
1
|
+
require_relative "core_extensions/action_controller"
|
2
|
+
require_relative "core_extensions/active_record"
|
3
|
+
require_relative "sql_query"
|
4
|
+
require_relative "current"
|
5
|
+
|
6
|
+
module ActiveMonitoring
|
7
|
+
class Current
|
8
|
+
class << self
|
9
|
+
def request_id
|
10
|
+
store[:request_id]
|
11
|
+
end
|
12
|
+
|
13
|
+
def request_id=(value)
|
14
|
+
store[:request_id] = value
|
15
|
+
end
|
16
|
+
|
17
|
+
def location
|
18
|
+
store[:location]
|
19
|
+
end
|
20
|
+
|
21
|
+
def location=(value)
|
22
|
+
store[:location] = value
|
23
|
+
end
|
24
|
+
|
25
|
+
def store
|
26
|
+
Thread.current[:active_monitoring_store] ||= {}
|
27
|
+
Thread.current[:active_monitoring_store]
|
28
|
+
end
|
29
|
+
end
|
30
|
+
end
|
31
|
+
end
|
@@ -0,0 +1,47 @@
|
|
1
|
+
require_relative "core_extensions/action_controller"
|
2
|
+
require_relative "core_extensions/active_record"
|
3
|
+
require_relative "sql_query"
|
4
|
+
require_relative "current"
|
5
|
+
|
6
|
+
module ActiveMonitoring
|
7
|
+
class Engine < ::Rails::Engine
|
8
|
+
isolate_namespace ActiveMonitoring
|
9
|
+
|
10
|
+
ActiveSupport.on_load(:active_record) do
|
11
|
+
ActiveRecord::ConnectionAdapters::AbstractAdapter.prepend(::ActiveMonitoring::CoreExtensions::ActiveRecord::Instrumentation)
|
12
|
+
end
|
13
|
+
|
14
|
+
ActiveSupport.on_load(:action_controller) do
|
15
|
+
ActionController::Base.prepend(::ActiveMonitoring::CoreExtensions::ActionController::Instrumentation)
|
16
|
+
|
17
|
+
ActiveMonitoring::Notifications.subscribe("start_processing.action_controller") do |_, _, _, _, payload|
|
18
|
+
Current.request_id = payload[:request_id]
|
19
|
+
Current.location = "#{payload[:controller]}##{payload[:action]}"
|
20
|
+
end
|
21
|
+
|
22
|
+
ActiveMonitoring::Notifications.subscribe("process_action.action_controller") do |name, start, finish, _id, payload|
|
23
|
+
Metric.create(
|
24
|
+
name: name,
|
25
|
+
value: finish - start,
|
26
|
+
request_id: payload[:request_id],
|
27
|
+
location: "#{payload[:controller]}##{payload[:action]}",
|
28
|
+
created_at: finish
|
29
|
+
)
|
30
|
+
end
|
31
|
+
|
32
|
+
ActiveMonitoring::Notifications.subscribe("sql.active_record") do |name, start, finish, _id, payload|
|
33
|
+
query = ActiveMonitoring::SqlQuery.new(name: payload[:name].dup, query: payload[:sql].dup)
|
34
|
+
if query.track?
|
35
|
+
Metric.create(
|
36
|
+
name: name,
|
37
|
+
value: finish - start,
|
38
|
+
request_id: Current.request_id,
|
39
|
+
sql_query: query.normalized_query,
|
40
|
+
location: Current.location,
|
41
|
+
created_at: finish
|
42
|
+
)
|
43
|
+
end
|
44
|
+
end
|
45
|
+
end
|
46
|
+
end
|
47
|
+
end
|
@@ -0,0 +1,19 @@
|
|
1
|
+
require_relative "notifier"
|
2
|
+
|
3
|
+
module ActiveMonitoring
|
4
|
+
class Notifications
|
5
|
+
class << self
|
6
|
+
attr_accessor :notifier
|
7
|
+
|
8
|
+
def subscribe(name, &block)
|
9
|
+
notifier.subscribe(name, &block)
|
10
|
+
end
|
11
|
+
|
12
|
+
def instrument(name, payload = {}, &block)
|
13
|
+
notifier.instrument(name, payload, &block)
|
14
|
+
end
|
15
|
+
end
|
16
|
+
|
17
|
+
self.notifier = ActiveMonitoring::Notifier.new
|
18
|
+
end
|
19
|
+
end
|
@@ -0,0 +1,33 @@
|
|
1
|
+
module ActiveMonitoring
|
2
|
+
class Notifier
|
3
|
+
def initialize
|
4
|
+
@subscribers = {}
|
5
|
+
end
|
6
|
+
|
7
|
+
def subscribe(name, &block)
|
8
|
+
subscribers[name] ||= []
|
9
|
+
subscribers[name] << block
|
10
|
+
end
|
11
|
+
|
12
|
+
def instrument(name, payload)
|
13
|
+
id = SecureRandom.hex(10)
|
14
|
+
start = Time.current
|
15
|
+
result = yield if block_given?
|
16
|
+
finish = Time.current
|
17
|
+
|
18
|
+
subscribers_for(name).each do |callback|
|
19
|
+
callback.call(name, start, finish, id, payload)
|
20
|
+
end
|
21
|
+
|
22
|
+
result
|
23
|
+
end
|
24
|
+
|
25
|
+
private
|
26
|
+
|
27
|
+
attr_reader :subscribers
|
28
|
+
|
29
|
+
def subscribers_for(name)
|
30
|
+
subscribers[name].to_a
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
@@ -0,0 +1,23 @@
|
|
1
|
+
module ActiveMonitoring
|
2
|
+
class SqlNormalizer
|
3
|
+
def initialize(query:)
|
4
|
+
@query = query
|
5
|
+
end
|
6
|
+
|
7
|
+
def to_s
|
8
|
+
query.squish!
|
9
|
+
query.gsub!(/(\s(=|>|<|>=|<=|<>|!=)\s)('[^']+'|[\$\+\-\w\.]+)/, '\1xxx')
|
10
|
+
query.gsub!(/(\sIN\s)\([^\(\)]+\)/i, '\1(xxx)')
|
11
|
+
regex = /(\sBETWEEN\s)('[^']+'|[\+\-\w\.]+)(\sAND\s)('[^']+'|[\+\-\w\.]+)/i
|
12
|
+
query.gsub!(regex, '\1xxx\3xxx')
|
13
|
+
query.gsub!(/(\sVALUES\s)\(.+\)/i, '\1(xxx)')
|
14
|
+
query.gsub!(/(\s(LIKE|ILIKE|SIMILAR TO|NOT SIMILAR TO)\s)('[^']+')/i, '\1xxx')
|
15
|
+
query.gsub!(/(\s(LIMIT|OFFSET)\s)(\d+)/i, '\1xxx')
|
16
|
+
query
|
17
|
+
end
|
18
|
+
|
19
|
+
private
|
20
|
+
|
21
|
+
attr_reader :query
|
22
|
+
end
|
23
|
+
end
|
@@ -0,0 +1,23 @@
|
|
1
|
+
require_relative "sql_tracker"
|
2
|
+
require_relative "sql_normalizer"
|
3
|
+
|
4
|
+
module ActiveMonitoring
|
5
|
+
class SqlQuery
|
6
|
+
def initialize(query:, name:)
|
7
|
+
@query = query
|
8
|
+
@name = name
|
9
|
+
end
|
10
|
+
|
11
|
+
def track?
|
12
|
+
SqlTracker.new(query: query, name: name).track?
|
13
|
+
end
|
14
|
+
|
15
|
+
def normalized_query
|
16
|
+
SqlNormalizer.new(query: query).to_s
|
17
|
+
end
|
18
|
+
|
19
|
+
private
|
20
|
+
|
21
|
+
attr_reader :query, :name
|
22
|
+
end
|
23
|
+
end
|
@@ -0,0 +1,33 @@
|
|
1
|
+
module ActiveMonitoring
|
2
|
+
class SqlTracker
|
3
|
+
TRACKED_SQL_COMMANDS = %w(SELECT INSERT UPDATE DELETE).freeze
|
4
|
+
UNTRACKED_NAMES = %w(SCHEMA).freeze
|
5
|
+
UNTRACKED_TABLES = %w(
|
6
|
+
SCHEMA_MIGRATIONS
|
7
|
+
SQLITE_MASTER
|
8
|
+
ACTIVE_MONITORING_METRICS
|
9
|
+
SQLITE_TEMP_MASTER
|
10
|
+
SQLITE_VERSION
|
11
|
+
AR_INTERNAL_METADATA
|
12
|
+
).freeze
|
13
|
+
|
14
|
+
def initialize(query:, name:)
|
15
|
+
@query = query.to_s.upcase
|
16
|
+
@name = name.to_s.upcase
|
17
|
+
end
|
18
|
+
|
19
|
+
def track?
|
20
|
+
query.start_with?(*TRACKED_SQL_COMMANDS) &&
|
21
|
+
!name.start_with?(*UNTRACKED_NAMES) &&
|
22
|
+
!untracked_tables?
|
23
|
+
end
|
24
|
+
|
25
|
+
private
|
26
|
+
|
27
|
+
attr_reader :query, :name
|
28
|
+
|
29
|
+
def untracked_tables?
|
30
|
+
UNTRACKED_TABLES.any? { |table| query.include?(table) }
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
metadata
ADDED
@@ -0,0 +1,158 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: active_monitoring
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.1.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Christian Bruckmayer
|
8
|
+
autorequire:
|
9
|
+
bindir: bin
|
10
|
+
cert_chain: []
|
11
|
+
date: 2020-04-17 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: rails
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - "~>"
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: 6.0.2
|
20
|
+
- - ">="
|
21
|
+
- !ruby/object:Gem::Version
|
22
|
+
version: 6.0.2.2
|
23
|
+
type: :runtime
|
24
|
+
prerelease: false
|
25
|
+
version_requirements: !ruby/object:Gem::Requirement
|
26
|
+
requirements:
|
27
|
+
- - "~>"
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
version: 6.0.2
|
30
|
+
- - ">="
|
31
|
+
- !ruby/object:Gem::Version
|
32
|
+
version: 6.0.2.2
|
33
|
+
- !ruby/object:Gem::Dependency
|
34
|
+
name: byebug
|
35
|
+
requirement: !ruby/object:Gem::Requirement
|
36
|
+
requirements:
|
37
|
+
- - ">="
|
38
|
+
- !ruby/object:Gem::Version
|
39
|
+
version: '0'
|
40
|
+
type: :development
|
41
|
+
prerelease: false
|
42
|
+
version_requirements: !ruby/object:Gem::Requirement
|
43
|
+
requirements:
|
44
|
+
- - ">="
|
45
|
+
- !ruby/object:Gem::Version
|
46
|
+
version: '0'
|
47
|
+
- !ruby/object:Gem::Dependency
|
48
|
+
name: rspec-rails
|
49
|
+
requirement: !ruby/object:Gem::Requirement
|
50
|
+
requirements:
|
51
|
+
- - ">="
|
52
|
+
- !ruby/object:Gem::Version
|
53
|
+
version: '0'
|
54
|
+
type: :development
|
55
|
+
prerelease: false
|
56
|
+
version_requirements: !ruby/object:Gem::Requirement
|
57
|
+
requirements:
|
58
|
+
- - ">="
|
59
|
+
- !ruby/object:Gem::Version
|
60
|
+
version: '0'
|
61
|
+
- !ruby/object:Gem::Dependency
|
62
|
+
name: rubocop-performance
|
63
|
+
requirement: !ruby/object:Gem::Requirement
|
64
|
+
requirements:
|
65
|
+
- - ">="
|
66
|
+
- !ruby/object:Gem::Version
|
67
|
+
version: '0'
|
68
|
+
type: :development
|
69
|
+
prerelease: false
|
70
|
+
version_requirements: !ruby/object:Gem::Requirement
|
71
|
+
requirements:
|
72
|
+
- - ">="
|
73
|
+
- !ruby/object:Gem::Version
|
74
|
+
version: '0'
|
75
|
+
- !ruby/object:Gem::Dependency
|
76
|
+
name: rubocop-rails
|
77
|
+
requirement: !ruby/object:Gem::Requirement
|
78
|
+
requirements:
|
79
|
+
- - ">="
|
80
|
+
- !ruby/object:Gem::Version
|
81
|
+
version: '0'
|
82
|
+
type: :development
|
83
|
+
prerelease: false
|
84
|
+
version_requirements: !ruby/object:Gem::Requirement
|
85
|
+
requirements:
|
86
|
+
- - ">="
|
87
|
+
- !ruby/object:Gem::Version
|
88
|
+
version: '0'
|
89
|
+
- !ruby/object:Gem::Dependency
|
90
|
+
name: sqlite3
|
91
|
+
requirement: !ruby/object:Gem::Requirement
|
92
|
+
requirements:
|
93
|
+
- - ">="
|
94
|
+
- !ruby/object:Gem::Version
|
95
|
+
version: '0'
|
96
|
+
type: :development
|
97
|
+
prerelease: false
|
98
|
+
version_requirements: !ruby/object:Gem::Requirement
|
99
|
+
requirements:
|
100
|
+
- - ">="
|
101
|
+
- !ruby/object:Gem::Version
|
102
|
+
version: '0'
|
103
|
+
description: his is an example gem used in my RailsConf 2020 talk. Please don't use
|
104
|
+
in production! https://bruckmayer.net/rails-conf-2020
|
105
|
+
email:
|
106
|
+
- christian@bruckmayer.net
|
107
|
+
executables: []
|
108
|
+
extensions: []
|
109
|
+
extra_rdoc_files: []
|
110
|
+
files:
|
111
|
+
- MIT-LICENSE
|
112
|
+
- README.md
|
113
|
+
- Rakefile
|
114
|
+
- app/controllers/active_monitoring/application_controller.rb
|
115
|
+
- app/controllers/active_monitoring/dashboard_controller.rb
|
116
|
+
- app/models/active_monitoring/application_record.rb
|
117
|
+
- app/models/active_monitoring/dashboard.rb
|
118
|
+
- app/models/active_monitoring/metric.rb
|
119
|
+
- app/views/active_monitoring/dashboard/show.html.erb
|
120
|
+
- app/views/layouts/active_monitoring/application.html.erb
|
121
|
+
- config/routes.rb
|
122
|
+
- db/migrate/20200401191741_create_active_monitoring_metrics.rb
|
123
|
+
- lib/active_monitoring.rb
|
124
|
+
- lib/active_monitoring/core_extensions/action_controller.rb
|
125
|
+
- lib/active_monitoring/core_extensions/active_record.rb
|
126
|
+
- lib/active_monitoring/current.rb
|
127
|
+
- lib/active_monitoring/engine.rb
|
128
|
+
- lib/active_monitoring/notifications.rb
|
129
|
+
- lib/active_monitoring/notifier.rb
|
130
|
+
- lib/active_monitoring/sql_normalizer.rb
|
131
|
+
- lib/active_monitoring/sql_query.rb
|
132
|
+
- lib/active_monitoring/sql_tracker.rb
|
133
|
+
- lib/active_monitoring/version.rb
|
134
|
+
homepage: https://github.com/ChrisBr/active_monitoring
|
135
|
+
licenses:
|
136
|
+
- MIT
|
137
|
+
metadata: {}
|
138
|
+
post_install_message:
|
139
|
+
rdoc_options: []
|
140
|
+
require_paths:
|
141
|
+
- lib
|
142
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
143
|
+
requirements:
|
144
|
+
- - ">="
|
145
|
+
- !ruby/object:Gem::Version
|
146
|
+
version: '0'
|
147
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
148
|
+
requirements:
|
149
|
+
- - ">="
|
150
|
+
- !ruby/object:Gem::Version
|
151
|
+
version: '0'
|
152
|
+
requirements: []
|
153
|
+
rubygems_version: 3.1.2
|
154
|
+
signing_key:
|
155
|
+
specification_version: 4
|
156
|
+
summary: This is an example gem used in my RailsConf 2020 talk. Please don't use in
|
157
|
+
production!
|
158
|
+
test_files: []
|