query_police 0.1.4.beta → 0.1.5.beta
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +408 -242
- data/examples/rules/json/absent_rule.json +13 -0
- data/examples/rules/json/basic_rule.json +22 -0
- data/examples/rules/json/complex_detailed_rule.json +18 -0
- data/examples/rules/json/threshold_rule.json +15 -0
- data/examples/rules/yaml/absent_rule.yml +11 -0
- data/examples/rules/yaml/basic_rule.yml +18 -0
- data/examples/rules/yaml/complex_detailed_rule.yml +13 -0
- data/examples/rules/yaml/threshold_rule.yml +12 -0
- data/lib/query_police/analyse.rb +16 -16
- data/lib/query_police/analysis/dynamic_message.rb +3 -3
- data/lib/query_police/analysis.rb +67 -39
- data/lib/query_police/config.rb +11 -6
- data/lib/query_police/constants.rb +6 -3
- data/lib/query_police/explain.rb +15 -2
- data/lib/query_police/helper.rb +13 -3
- data/lib/query_police/rules.json +66 -57
- data/lib/query_police/version.rb +1 -1
- data/lib/query_police.rb +58 -25
- metadata +12 -4
data/README.md
CHANGED
@@ -1,16 +1,16 @@
|
|
1
1
|
# QueryPolice
|
2
2
|
|
3
|
-
It is a rule-based engine with custom rules to Analyze Active-Record relations using explain results
|
3
|
+
It is a rule-based engine with custom rules to Analyze Active-Record relations using explain results and detect bad queries.
|
4
4
|
|
5
5
|
## Installation
|
6
6
|
|
7
7
|
Install the gem and add it to the application's Gemfile by executing:
|
8
8
|
|
9
|
-
$
|
9
|
+
$ gem 'query_police', '~> 0.1.5.beta'
|
10
10
|
|
11
11
|
If the bundler is not being used to manage dependencies, install the gem by executing:
|
12
12
|
|
13
|
-
$ gem install query_police
|
13
|
+
$ gem install query_police --pre
|
14
14
|
|
15
15
|
---
|
16
16
|
|
@@ -20,320 +20,318 @@ If the bundler is not being used to manage dependencies, install the gem by exec
|
|
20
20
|
|
21
21
|
Use `QueryPolice.analyse` to generate an analysis object for your Active-Record relation or query and then you can use pretty print on the object.
|
22
22
|
|
23
|
-
```
|
24
|
-
analysis = QueryPolice.analyse(<
|
25
|
-
puts analysis.
|
23
|
+
```ruby
|
24
|
+
analysis = QueryPolice.analyse(<query>)
|
25
|
+
puts analysis.pretty_analysis
|
26
26
|
```
|
27
27
|
|
28
28
|
**Eg.**
|
29
|
-
```
|
29
|
+
```ruby
|
30
30
|
analysis = QueryPolice.analyse(
|
31
|
-
User.joins('join orders on orders.user_id = users.id')
|
31
|
+
User.joins('join orders on orders.user_id = users.id') # analyse query using active record relation
|
32
32
|
)
|
33
|
-
puts analysis.pretty_analysis_for('negative')
|
34
33
|
# or
|
35
|
-
|
34
|
+
analysis = QueryPolice.analyse(
|
35
|
+
"select * from users join orders on orders.user_id = users.id" # analyse query using query string
|
36
|
+
)
|
37
|
+
|
38
|
+
puts analysis.pretty_analysis
|
36
39
|
```
|
37
40
|
**Results**
|
38
41
|
|
42
|
+
**Note:** [Query debt significance](#significance)
|
43
|
+
|
44
|
+
|
39
45
|
```
|
40
|
-
|
46
|
+
query_debt: 330.0
|
41
47
|
|
42
48
|
+----------------------------------------------------------------------------------------------------------------------------------+
|
43
49
|
| orders |
|
44
50
|
+------------+---------------------------------------------------------------------------------------------------------------------+
|
45
|
-
|
|
51
|
+
| debt | 200.0 |
|
46
52
|
+------------+---------------------------------------------------------------------------------------------------------------------+
|
47
53
|
| column | type |
|
48
54
|
| impact | negative |
|
49
|
-
|
|
55
|
+
| tag_debt | 100.0 |
|
50
56
|
| message | Entire orders table is scanned to find matching rows, you have 0 possible keys to use. |
|
51
57
|
| suggestion | Use index here. You can use index from possible key: absent or add new one to orders table as per the requirements. |
|
52
58
|
+------------+---------------------------------------------------------------------------------------------------------------------+
|
53
59
|
| column | possible_keys |
|
54
60
|
| impact | negative |
|
55
|
-
|
|
61
|
+
| tag_debt | 50.0 |
|
56
62
|
| message | There are no possible keys for orders table to be used, can result into full scan |
|
57
63
|
| suggestion | Please add index keys for orders table |
|
58
64
|
+------------+---------------------------------------------------------------------------------------------------------------------+
|
59
65
|
| column | key |
|
60
66
|
| impact | negative |
|
61
|
-
|
|
67
|
+
| tag_debt | 50.0 |
|
62
68
|
| message | There is no index key used for orders table, and can result into full scan of the orders table |
|
63
69
|
| suggestion | Please use index from possible_keys: absent or add new one to orders table as per the requirements. |
|
64
70
|
+------------+---------------------------------------------------------------------------------------------------------------------+
|
65
71
|
+------------------------------------------------------------------------------------+
|
66
72
|
| users |
|
67
73
|
+------------+-----------------------------------------------------------------------+
|
68
|
-
|
|
74
|
+
| debt | 130.0 |
|
69
75
|
+------------+-----------------------------------------------------------------------+
|
70
76
|
| column | detailed#used_columns |
|
71
77
|
| impact | negative |
|
72
|
-
|
|
78
|
+
| tag_debt | 130.0 |
|
73
79
|
| message | You have selected 18 columns, You should not select too many columns. |
|
74
80
|
| suggestion | Please only select required columns. |
|
75
81
|
+------------+-----------------------------------------------------------------------+
|
76
82
|
```
|
77
83
|
|
78
|
-
###
|
79
|
-
|
80
|
-
Add `QueryPolice.subscribe_logger` to your initial load file like `application.rb`
|
81
|
-
|
82
|
-
You can make logger silence of error using `QueryPolice.subscribe_logger silent: true`.
|
83
|
-
|
84
|
-
You can change logger config using `QueryPolice logger_config: <config>`, default logger_config `{'negative' => true}`, options `positive: <Boolean>, caution: <Boolean>`.
|
85
|
-
|
86
|
-
---
|
84
|
+
### Analysis for a Impact
|
87
85
|
|
88
|
-
|
86
|
+
To print pretty analysis for different impacts
|
87
|
+
```ruby
|
88
|
+
analysis = QueryPolice.analyse("select * from users")
|
89
|
+
puts analysis.pretty_analysis_for('positive') # impact negative, positive, caution
|
89
90
|
|
90
|
-
|
91
|
-
|
92
|
-
|
93
|
-
|
94
|
-
|
95
|
-
|
96
|
-
|
97
|
-
|
98
|
-
|
99
|
-
|
100
|
-
|
101
|
-
|
102
|
-
|
103
|
-
We have 2 possible execution plans:-
|
104
|
-
|
105
|
-
Normal - using `EXPLAIN`
|
106
|
-
|
107
|
-
Detailed - using `EXPLAIN format=JSON`
|
108
|
-
|
109
|
-
**NOTE:** By default Detailed execution plan is added in the final execution plan, you can remove that by `QueryPolice.detailed=false`
|
110
|
-
|
111
|
-
### Normal execution plan
|
112
|
-
|
113
|
-
Generated using `EXPAIN <query>`
|
114
|
-
|
115
|
-
**Result**
|
116
|
-
|
117
|
-
| id | select_type | table | partitions | type. | possible_keys | key | key_len | ref | rows | filtered | Extra |
|
118
|
-
|----|-------------|---------|------------|--------|---------------------------|---------------------|---------|-----------------------------|------|----------|--------------------------|
|
119
|
-
| 1 | SIMPLE | profile | NULL. | index | fk_rails_249a7ebca1 | fk_rails_249a7ebca1 | 5 | NULL | 603 | 100.00 | Using where; Using index |
|
120
|
-
| 1 | SIMPLE | users | NULL | eq_ref | PRIMARY,index_users_on_id | PRIMARY | 4 | development.profile.user_id |1 | 100.00 | NULL |
|
121
|
-
|
122
|
-
|
123
|
-
The result for this is added as it is in the final execution plan
|
91
|
+
# puts
|
92
|
+
# +----------------------------------------------------------+
|
93
|
+
# | users |
|
94
|
+
# +-----------+----------------------------------------------+
|
95
|
+
# | debt | 330.0 |
|
96
|
+
# +-----------+----------------------------------------------+
|
97
|
+
# | column | select_type |
|
98
|
+
# | impact | positive |
|
99
|
+
# | tag_debt | 0 |
|
100
|
+
# | message | A simple query without subqueries or unions. |
|
101
|
+
# +-----------+----------------------------------------------+
|
102
|
+
```
|
124
103
|
|
125
|
-
|
104
|
+
### Analysis for Multiple Impacts
|
126
105
|
|
106
|
+
To print pretty analysis for multiple impacts, default: `{ 'negative' => true, 'caution' => true }`
|
107
|
+
```ruby
|
108
|
+
analysis = QueryPolice.analyse("select * from users")
|
109
|
+
puts analysis.pretty_analysis({'negative' => true, 'positive' => true})
|
127
110
|
```
|
128
|
-
{
|
129
|
-
"profile" => {
|
130
|
-
"id" => 1,
|
131
|
-
"select_type" => "SIMPLE",
|
132
|
-
"table" => "profile",
|
133
|
-
"partitions" => nil,
|
134
|
-
"type" => "index",
|
135
|
-
"possible_keys" => "fk_rails_249a7ebca1",
|
136
|
-
"key" => "fk_rails_249a7ebca1",
|
137
|
-
"key_len" => "5",
|
138
|
-
"ref" => nil,
|
139
|
-
"rows" => 603,
|
140
|
-
"filtered" => 100.0,
|
141
|
-
"Extra" => "Using where; Using index"
|
142
|
-
},
|
143
|
-
"users" => {
|
144
|
-
"id" => 1,
|
145
|
-
"select_type" => "SIMPLE",
|
146
|
-
"table" => "users",
|
147
|
-
"partitions" => nil,
|
148
|
-
"type" => "eq_ref",
|
149
|
-
"possible_keys" => "PRIMARY,index_users_on_id",
|
150
|
-
"key" => "PRIMARY",
|
151
|
-
"key_len" => "4",
|
152
|
-
"ref" => "development.profile.user_id",
|
153
|
-
"rows" => 1,
|
154
|
-
"filtered" => 100.0,
|
155
|
-
"Extra" => nil
|
156
|
-
}
|
157
|
-
}
|
158
|
-
```
|
159
|
-
|
160
|
-
### Detailed execution plan
|
161
111
|
|
162
|
-
|
112
|
+
### Query debt
|
163
113
|
|
164
|
-
|
114
|
+
To get the final debt of the query
|
115
|
+
```ruby
|
116
|
+
analysis = QueryPolice.analyse("select * from users")
|
117
|
+
puts analysis.query_debt
|
165
118
|
|
119
|
+
# puts
|
120
|
+
# 100.0
|
166
121
|
```
|
167
|
-
{
|
168
|
-
"query_block": {
|
169
|
-
"select_id": 1,
|
170
|
-
"cost_info": {
|
171
|
-
"query_cost": "850.20"
|
172
|
-
},
|
173
|
-
"nested_loop": [
|
174
|
-
{
|
175
|
-
"table": {
|
176
|
-
"table_name": "profile",
|
177
|
-
"access_type": "index",
|
178
|
-
"key_length": "5",
|
179
|
-
"cost_info": {
|
180
|
-
"read_cost": "6.00",
|
181
|
-
"eval_cost": "120.60",
|
182
|
-
"prefix_cost": "126.60",
|
183
|
-
"data_read_per_join": "183K"
|
184
|
-
},
|
185
|
-
"used_columns": [
|
186
|
-
"user_id"
|
187
|
-
],
|
188
|
-
"attached_condition": "(`development`.`profile`.`user_id` is not null)"
|
189
|
-
}
|
190
|
-
}
|
191
|
-
]
|
192
|
-
}
|
193
|
-
}
|
194
|
-
```
|
195
|
-
|
196
122
|
|
197
|
-
|
123
|
+
#### Significance
|
124
|
+
Query debt signifies the quality of the query, high value represents a bad query.
|
125
|
+
- `0 - 199` - Good Query
|
126
|
+
- `200 - 499` - Potentially Bad query
|
127
|
+
- `>=500` - Bad query
|
198
128
|
|
199
|
-
|
129
|
+
### Word wrap
|
200
130
|
|
201
|
-
|
202
|
-
|
203
|
-
|
204
|
-
|
205
|
-
|
206
|
-
|
207
|
-
|
208
|
-
|
209
|
-
|
210
|
-
|
211
|
-
|
212
|
-
|
213
|
-
|
131
|
+
To change the word wrap width for the pretty analysis result, default value: `100`
|
132
|
+
```ruby
|
133
|
+
analysis = QueryPolice.analyse("select * from users")
|
134
|
+
puts analysis.pretty_analysis_for('positive', {'wrap_width' => 40})
|
135
|
+
# or
|
136
|
+
puts analysis.pretty_analysis({'positive' => true, 'wrap_width' => 40})
|
137
|
+
|
138
|
+
# puts
|
139
|
+
# +--------------------------------------------------+
|
140
|
+
# | users |
|
141
|
+
# +-----------+--------------------------------------+
|
142
|
+
# | debt | 330.0 |
|
143
|
+
# +-----------+--------------------------------------+
|
144
|
+
# | column | select_type |
|
145
|
+
# | impact | positive |
|
146
|
+
# | tag_debt | 0 |
|
147
|
+
# | message | A simple query without subqueries or |
|
148
|
+
# | | unions. |
|
149
|
+
# +-----------+--------------------------------------+
|
150
|
+
|
151
|
+
puts analysis.pretty_analysis_for('positive', {'wrap_width' => 20})
|
152
|
+
# or
|
153
|
+
puts analysis.pretty_analysis({'positive' => true, 'wrap_width' => 20})
|
154
|
+
|
155
|
+
# puts
|
156
|
+
# +--------------------------------+
|
157
|
+
# | users |
|
158
|
+
# +-----------+--------------------+
|
159
|
+
# | debt | 330.0 |
|
160
|
+
# +-----------+--------------------+
|
161
|
+
# | column | select_type |
|
162
|
+
# | impact | positive |
|
163
|
+
# | tag_debt | 0 |
|
164
|
+
# | message | A simple query |
|
165
|
+
# | | without subqueries |
|
166
|
+
# | | or unions. |
|
167
|
+
# +-----------+--------------------+
|
168
|
+
```
|
169
|
+
|
170
|
+
### Skip footer
|
171
|
+
|
172
|
+
To skip the footer (added in [query police config](#analysis-footer)) after an analysis of a query
|
173
|
+
```ruby
|
174
|
+
analysis = QueryPolice.analyse("select * from users")
|
175
|
+
puts analysis.pretty_analysis_for('positive', {'skip_footer' => true})
|
176
|
+
# or
|
177
|
+
puts analysis.pretty_analysis({'positive' => true, 'skip_footer' => true})
|
214
178
|
```
|
215
179
|
|
180
|
+
### Analysis Footer
|
181
|
+
To define a footer text that will be added after an analysis for a query, by default there is no footer
|
182
|
+
```ruby
|
183
|
+
QueryPolice.analysis_footer = 'Please check more details with this link...'
|
184
|
+
# or
|
185
|
+
QueryPolice.configure do |config|
|
186
|
+
config.analysis_footer = 'Please check more details with this link...'
|
187
|
+
end
|
188
|
+
|
189
|
+
# puts
|
190
|
+
# +----------------------------------------------------------+
|
191
|
+
# | users |
|
192
|
+
# +-----------+----------------------------------------------+
|
193
|
+
# | debt | 330.0 |
|
194
|
+
# +-----------+----------------------------------------------+
|
195
|
+
# | column | select_type |
|
196
|
+
# | impact | positive |
|
197
|
+
# | tag_debt | 0 |
|
198
|
+
# | message | A simple query without subqueries or unions. |
|
199
|
+
# +-----------+----------------------------------------------+
|
200
|
+
# Please check more details with this link...
|
201
|
+
```
|
202
|
+
|
203
|
+
### Custom rules path
|
204
|
+
|
205
|
+
To define custom rules path (More details about [how to define custom rules](#how-to-define-custom-rules))
|
206
|
+
```ruby
|
207
|
+
QueryPolice.rules_path = 'path/to/rules/file.<json/yml>'
|
208
|
+
# or
|
209
|
+
QueryPolice.configure do |config|
|
210
|
+
config.rules_path = 'path/to/rules/file.<json/yml>'
|
211
|
+
end
|
212
|
+
```
|
216
213
|
|
214
|
+
### Verbosity
|
217
215
|
|
218
|
-
|
219
|
-
|
220
|
-
`
|
216
|
+
Verbosity defines which `EXPLAIN` result should be used for analysis. (More details about [EXPLAIN vs Detailed EXPLAIN](#execution-plan))
|
217
|
+
- `basic` - It uses only `EXPLAIN`(basic) result
|
218
|
+
- `detailed` - It uses both `EXPLAIN`(basic) and `EXPLAIN format=json`(detailed) results `(default value)`
|
219
|
+
```ruby
|
220
|
+
QueryPolice.verbosity = "detailed"
|
221
|
+
# or
|
222
|
+
QueryPolice.configure do |config|
|
223
|
+
config.verbosity = "detailed"
|
224
|
+
end
|
225
|
+
```
|
221
226
|
|
227
|
+
### Disable Actions
|
222
228
|
|
223
|
-
|
229
|
+
To disable actions that are performed on each query.
|
230
|
+
```ruby
|
231
|
+
# Note: A logger action is already added, so query police will log pretty analysis after each query by default
|
232
|
+
QueryPolice.action_enabled = false
|
233
|
+
# or
|
234
|
+
QueryPolice.configure do |config|
|
235
|
+
config.action_enabled = false
|
236
|
+
end
|
237
|
+
```
|
224
238
|
|
225
|
-
|
239
|
+
### Custom Actions
|
226
240
|
|
227
|
-
|
241
|
+
To add custom actions, by default a logger action is already added and enabled (More details about [Analysis Object](#analysis-object))
|
242
|
+
```ruby
|
243
|
+
QueryPolice.add_action do |analysis| # analysis object for the query
|
244
|
+
puts analysis.tables
|
245
|
+
end
|
246
|
+
```
|
228
247
|
|
229
|
-
|
248
|
+
### Logger config
|
230
249
|
|
231
|
-
|
232
|
-
|
233
|
-
|
234
|
-
|
235
|
-
|
236
|
-
|
237
|
-
|
238
|
-
'type' => { # attribute name
|
239
|
-
'value' => <string>, # raw value of attribute in execution plan
|
240
|
-
'tags' => {
|
241
|
-
'all' => { # tag based on the value of a attribute
|
242
|
-
'impact'=> <string>, # negative, positive, cautions
|
243
|
-
'warning'=> <string>, # Eg. 'warning to represent the issue'
|
244
|
-
'suggestions'=> <string> # Eg. 'some follow-up suggestions'
|
245
|
-
'score' => <float> # score for the tag
|
246
|
-
}
|
247
|
-
}
|
248
|
-
}
|
249
|
-
}
|
250
|
-
}
|
251
|
-
}
|
252
|
-
```
|
253
|
-
**summary [Hash] - hash of analysis summary**
|
250
|
+
To change the logger options which will be used to generate analysis
|
251
|
+
```ruby
|
252
|
+
QueryPolice.logger_options = {'negative' => true}
|
253
|
+
# or
|
254
|
+
QueryPolice.configure do |config|
|
255
|
+
config.logger_options = {'negative' => true}
|
256
|
+
end
|
254
257
|
|
255
|
-
|
256
|
-
|
257
|
-
'cardinality' => {
|
258
|
-
'amount' => 10,
|
259
|
-
'warning' => 'warning to represent the issue',
|
260
|
-
'suggestions' => 'some follow up suggestions',
|
261
|
-
'score' => 100.0
|
262
|
-
}
|
263
|
-
}
|
258
|
+
# default logger_config: {'negative' => true, 'caution' => true}
|
259
|
+
# options negative: <Boolean>, positive: <Boolean>, caution: <Boolean>, wrap_width: <Integer>, skip_footer: <Boolean>
|
264
260
|
```
|
265
261
|
|
262
|
+
---
|
266
263
|
|
264
|
+
## How to define custom rules?
|
267
265
|
|
268
|
-
|
269
|
-
|
270
|
-
Rules defined in the json file at config_path is applied to the execution plan. We have variety of option to define rules.
|
266
|
+
Rules defined in the `json/yaml` file at rules_path is applied to the execution plan. Query Police have variety of option to define rules.
|
271
267
|
|
272
268
|
You can change this by `QueryPolice.rules_path=<path>` and define your own rules
|
273
269
|
|
274
270
|
### Rule Structure
|
271
|
+
**Note:** Check Query Police default rules defined at [rules.json](lib/query_police/rules.json) or examples at [examples/rules/](examples/rules/) for better clarity
|
275
272
|
|
276
273
|
A basic rule structure -
|
277
|
-
|
274
|
+
|
275
|
+
JSON
|
276
|
+
```json
|
278
277
|
"<column_name>": {
|
279
|
-
"description": <string
|
280
|
-
"value_type": <string
|
281
|
-
"delimiter": <string
|
278
|
+
"description": "<string>",
|
279
|
+
"value_type": "<string>",
|
280
|
+
"delimiter": "<string>",
|
282
281
|
"rules": {
|
283
282
|
"<rule>": {
|
284
|
-
"amount": <integer>
|
285
|
-
"impact": <string
|
286
|
-
"message": <string
|
287
|
-
"suggestion": <string
|
288
|
-
"
|
289
|
-
"value": <integer
|
290
|
-
"type": <string>
|
283
|
+
"amount": "<integer>",
|
284
|
+
"impact": "<string>",
|
285
|
+
"message": "<string>",
|
286
|
+
"suggestion": "<string>",
|
287
|
+
"debt": {
|
288
|
+
"value": "<integer>",
|
289
|
+
"type": "<string>"
|
291
290
|
}
|
292
291
|
}
|
293
292
|
}
|
294
293
|
}
|
295
294
|
```
|
296
|
-
|
297
|
-
|
295
|
+
YAML
|
296
|
+
```yaml
|
297
|
+
<column_name>:
|
298
|
+
description: <string>
|
299
|
+
value_type: <string>
|
300
|
+
delimiter: <string>
|
301
|
+
rules:
|
302
|
+
<rule>:
|
303
|
+
amount: <integer>
|
304
|
+
impact: <string>
|
305
|
+
message: <string>
|
306
|
+
suggestion: <string>
|
307
|
+
debt:
|
308
|
+
value: <integer>
|
309
|
+
type: <string>
|
310
|
+
```
|
311
|
+
- `<column_name>` - attribute name in the final execution plan. (more details about [attributes](#attributes-for-rules))
|
298
312
|
- `description` - description of the attribute
|
299
|
-
|
300
313
|
- `value_type` - value type of the attribute
|
301
|
-
|
302
314
|
- `delimiter` - delimiter to parse array type attribute values, if no delimiter is passed engine will consider value is already in array form.
|
303
|
-
|
304
315
|
- `<rule>` - kind of rule for the attribute
|
305
|
-
|
306
316
|
- `<tag>` - direct value match eg. ALL, SIMPLE
|
307
|
-
|
308
317
|
- `absent` - when the value is missing
|
309
|
-
|
310
318
|
- `threshold` - a greater than threshold check based on the amount set inside the rule.
|
311
|
-
|
312
319
|
- `amount` - the amount of threshold that needs to check for
|
313
|
-
|
314
320
|
- length for string
|
315
|
-
|
316
321
|
- value for number
|
317
|
-
|
318
322
|
- size for the array
|
319
|
-
|
320
323
|
- `impact` - impact of the rule
|
321
|
-
|
322
324
|
- `negative`
|
323
|
-
|
324
325
|
- `postive`
|
325
|
-
|
326
326
|
- `caution`
|
327
|
-
|
328
327
|
- `message` - the message needs to provide the significance of the rule
|
329
|
-
|
330
328
|
- `suggestion` - suggestion on how we can fix the issue
|
331
|
-
- `
|
332
|
-
- `value` - value that will be added to the query
|
333
|
-
- `type` - the type of scoring that will be added to the query
|
329
|
+
- `debt` - debt-related config that will be affected to final query debt
|
330
|
+
- `value` - value that will be added to the query debt
|
331
|
+
- `type` - the type of scoring that will be added to the query debt
|
334
332
|
- `base`- value
|
335
333
|
- `relative` - value * (amount for that column in query)
|
336
|
-
- `
|
334
|
+
- `threshold_relative` - (value - (threshold amount)) * (amount for that column in query)
|
337
335
|
|
338
336
|
|
339
337
|
|
@@ -342,34 +340,22 @@ A basic rule structure -
|
|
342
340
|
We can define dynamic messages and suggestions with variables provided by the engine.
|
343
341
|
|
344
342
|
- `$amount` - the amount of the value
|
345
|
-
|
346
343
|
- length for string
|
347
|
-
|
348
344
|
- value for number
|
349
|
-
|
350
345
|
- size for the array
|
351
|
-
|
352
346
|
- `$column` - attribute name
|
353
|
-
|
354
347
|
- `$impact` - impact for the rule
|
355
|
-
|
356
348
|
- `$table` - table alias used in the plan
|
357
|
-
|
358
349
|
- `$tag` - tag for which rule is applied
|
359
|
-
|
360
350
|
- `$value` - original parsed value
|
361
|
-
|
362
351
|
- `$<column_name>` - the value of that specific column in that table
|
363
|
-
|
364
352
|
- `$amount_<column_name>` - amount of that specific column
|
365
353
|
|
366
|
-
|
367
|
-
|
368
354
|
### Rules Examples
|
369
355
|
|
370
356
|
#### Basic rule example
|
371
|
-
|
372
|
-
```
|
357
|
+
File: [JSON](examples/rules/json/basic_rule.json) | [YAML](examples/rules/yaml/basic_rule.yml)
|
358
|
+
```json
|
373
359
|
"type": {
|
374
360
|
"description": "Join used in the query for a specific table.",
|
375
361
|
"value_type": "string",
|
@@ -383,11 +369,12 @@ We can define dynamic messages and suggestions with variables provided by the en
|
|
383
369
|
"impact": "negative",
|
384
370
|
"message": "Entire $table table is scanned to find matching rows, you have $amount_possible_keys possible keys to use.",
|
385
371
|
"suggestion": "Use index here. You can use index from possible key: $possible_keys or add new one to $table table as per the requirements.",
|
386
|
-
"
|
372
|
+
"debt": {
|
387
373
|
"value": 200,
|
388
374
|
"type": "base"
|
389
375
|
}
|
390
376
|
}
|
377
|
+
}
|
391
378
|
}
|
392
379
|
```
|
393
380
|
For the above rule, dynamic message will be generated as-
|
@@ -401,8 +388,8 @@ Use index here. You can use index from possible key: ["PRIMARY", "user_email"] o
|
|
401
388
|
|
402
389
|
|
403
390
|
#### Absent rule example
|
404
|
-
|
405
|
-
```
|
391
|
+
File: [JSON](examples/rules/json/absent_rule.json) | [YAML](examples/rules/yaml/absent_rule.yml)
|
392
|
+
```json
|
406
393
|
"key": {
|
407
394
|
"description": "index key used for the table",
|
408
395
|
"value_type": "string",
|
@@ -427,8 +414,8 @@ Please use index from possible_keys: ["PRIMARY", "user_email"] or add new one to
|
|
427
414
|
|
428
415
|
|
429
416
|
#### Threshold rule example
|
430
|
-
|
431
|
-
```
|
417
|
+
File: [JSON](examples/rules/json/threshold_rule.json) | [YAML](examples/rules/yaml/threshold_rule.yml)
|
418
|
+
```json
|
432
419
|
"possible_keys": {
|
433
420
|
"description": "Index keys possible for a specifc table",
|
434
421
|
"value_type": "array",
|
@@ -454,8 +441,8 @@ Please check if there are extra indexes in users table.
|
|
454
441
|
|
455
442
|
|
456
443
|
#### Complex Detailed rule example
|
457
|
-
|
458
|
-
```
|
444
|
+
File: [JSON](examples/rules/json/complex_detailed_rule.json) | [YAML](examples/rules/yaml/complex_detailed_rule.yml)
|
445
|
+
```json
|
459
446
|
"detailed#used_columns": {
|
460
447
|
"description": "",
|
461
448
|
"value_type": "array",
|
@@ -465,9 +452,9 @@ Please check if there are extra indexes in users table.
|
|
465
452
|
"impact": "negative",
|
466
453
|
"message": "You have selected $amount columns, You should not select too many columns.",
|
467
454
|
"suggestion": "Please only select required columns.",
|
468
|
-
"
|
455
|
+
"debt": {
|
469
456
|
"value": 10,
|
470
|
-
"type": "
|
457
|
+
"type": "threshold_relative"
|
471
458
|
}
|
472
459
|
}
|
473
460
|
}
|
@@ -483,9 +470,9 @@ Please only select required columns.
|
|
483
470
|
```
|
484
471
|
|
485
472
|
|
486
|
-
### Summary
|
473
|
+
### Summary in Analysis
|
487
474
|
|
488
|
-
You can define similar rules for the summary. Current
|
475
|
+
You can define similar rules for the summary. Current only one attribute is supported in summary -
|
489
476
|
|
490
477
|
- `cardinality` - cardinality based on all tables
|
491
478
|
|
@@ -493,7 +480,7 @@ You can define similar rules for the summary. Current summary attribute supporte
|
|
493
480
|
|
494
481
|
|
495
482
|
|
496
|
-
### Attributes
|
483
|
+
### Attributes for Rules
|
497
484
|
|
498
485
|
There are a lot of attributes for you to use based on the final execution plan.
|
499
486
|
|
@@ -509,6 +496,185 @@ To check more keys you can use `EXPLAIN format=JSON <query>`
|
|
509
496
|
|
510
497
|
---
|
511
498
|
|
499
|
+
## How Query Police works?
|
500
|
+
|
501
|
+
1. Query police converts the relation into SQL query
|
502
|
+
|
503
|
+
2. Query police generates an execution plan using EXPLAIN and EXPLAIN format=JSON based on the configuration.
|
504
|
+
|
505
|
+
3. Query police load rules from the config file.
|
506
|
+
|
507
|
+
4. Query police apply rules on the execution plan and generate a new analysis object.
|
508
|
+
|
509
|
+
5. Analysis object provides different methods to print the analysis in a more descriptive format.
|
510
|
+
|
511
|
+
|
512
|
+
## Execution plan
|
513
|
+
|
514
|
+
We have 2 possible execution plans:-
|
515
|
+
|
516
|
+
Normal - using `EXPLAIN`
|
517
|
+
|
518
|
+
Detailed - using `EXPLAIN format=JSON`
|
519
|
+
|
520
|
+
**NOTE:** By default Detailed execution plan is added in the final execution plan, you can remove that by `QueryPolice.detailed=false`
|
521
|
+
|
522
|
+
### Normal execution plan
|
523
|
+
|
524
|
+
Generated using `EXPAIN <query>`
|
525
|
+
|
526
|
+
**Result**
|
527
|
+
|
528
|
+
| id | select_type | table | partitions | type. | possible_keys | key | key_len | ref | rows | filtered | Extra |
|
529
|
+
|----|-------------|---------|------------|--------|---------------------------|---------------------|---------|-----------------------------|------|----------|--------------------------|
|
530
|
+
| 1 | SIMPLE | profile | NULL. | index | fk_rails_249a7ebca1 | fk_rails_249a7ebca1 | 5 | NULL | 603 | 100.00 | Using where; Using index |
|
531
|
+
| 1 | SIMPLE | users | NULL | eq_ref | PRIMARY,index_users_on_id | PRIMARY | 4 | development.profile.user_id |1 | 100.00 | NULL |
|
532
|
+
|
533
|
+
|
534
|
+
The result for this is added as it is in the final execution plan
|
535
|
+
|
536
|
+
**Eg.**
|
537
|
+
|
538
|
+
```ruby
|
539
|
+
{
|
540
|
+
"profile" => {
|
541
|
+
"id" => 1,
|
542
|
+
"select_type" => "SIMPLE",
|
543
|
+
"table" => "profile",
|
544
|
+
"partitions" => nil,
|
545
|
+
"type" => "index",
|
546
|
+
"possible_keys" => "fk_rails_249a7ebca1",
|
547
|
+
"key" => "fk_rails_249a7ebca1",
|
548
|
+
"key_len" => "5",
|
549
|
+
"ref" => nil,
|
550
|
+
"rows" => 603,
|
551
|
+
"filtered" => 100.0,
|
552
|
+
"Extra" => "Using where; Using index"
|
553
|
+
},
|
554
|
+
"users" => {
|
555
|
+
"id" => 1,
|
556
|
+
"select_type" => "SIMPLE",
|
557
|
+
"table" => "users",
|
558
|
+
"partitions" => nil,
|
559
|
+
"type" => "eq_ref",
|
560
|
+
"possible_keys" => "PRIMARY,index_users_on_id",
|
561
|
+
"key" => "PRIMARY",
|
562
|
+
"key_len" => "4",
|
563
|
+
"ref" => "development.profile.user_id",
|
564
|
+
"rows" => 1,
|
565
|
+
"filtered" => 100.0,
|
566
|
+
"Extra" => nil
|
567
|
+
}
|
568
|
+
}
|
569
|
+
```
|
570
|
+
|
571
|
+
### Detailed execution plan
|
572
|
+
|
573
|
+
Generated using `EXPAIN format=JSON <query>`
|
574
|
+
|
575
|
+
**Truncated Result**
|
576
|
+
|
577
|
+
```ruby
|
578
|
+
{
|
579
|
+
"query_block": {
|
580
|
+
"select_id": 1,
|
581
|
+
"cost_info": {
|
582
|
+
"query_cost": "850.20"
|
583
|
+
},
|
584
|
+
"nested_loop": [
|
585
|
+
{
|
586
|
+
"table": {
|
587
|
+
"table_name": "profile",
|
588
|
+
"access_type": "index",
|
589
|
+
"key_length": "5",
|
590
|
+
"cost_info": {
|
591
|
+
"read_cost": "6.00",
|
592
|
+
"eval_cost": "120.60",
|
593
|
+
"prefix_cost": "126.60",
|
594
|
+
"data_read_per_join": "183K"
|
595
|
+
},
|
596
|
+
"used_columns": [
|
597
|
+
"user_id"
|
598
|
+
],
|
599
|
+
"attached_condition": "(`development`.`profile`.`user_id` is not null)"
|
600
|
+
}
|
601
|
+
}
|
602
|
+
]
|
603
|
+
}
|
604
|
+
}
|
605
|
+
```
|
606
|
+
|
607
|
+
|
608
|
+
The result for this is added in the flattened form to the final execution plan, where the `detailed#` prefix is added before each key.
|
609
|
+
|
610
|
+
**Truncated Eg.**
|
611
|
+
|
612
|
+
```ruby
|
613
|
+
{
|
614
|
+
"detailed#key_length" => "5",
|
615
|
+
"detailed#rows_examined_per_scan" => 603,
|
616
|
+
"detailed#rows_produced_per_join" => 603,
|
617
|
+
"detailed#filtered" => "100.00",
|
618
|
+
"detailed#using_index" => true,
|
619
|
+
"detailed#cost_info#read_cost" => "6.00",
|
620
|
+
"detailed#cost_info#eval_cost" => "120.60",
|
621
|
+
"detailed#cost_info#prefix_cost" => "126.60",
|
622
|
+
"detailed#cost_info#data_read_per_join" => "183K",
|
623
|
+
"detailed#used_columns" => ["user_id"]
|
624
|
+
...
|
625
|
+
```
|
626
|
+
|
627
|
+
|
628
|
+
|
629
|
+
##### Flatten
|
630
|
+
|
631
|
+
`{a: {b: 1}, c: 2}` is converted into `{a#b: 1, c: 2}`.
|
632
|
+
|
633
|
+
---
|
634
|
+
|
635
|
+
## Analysis object
|
636
|
+
|
637
|
+
Analysis object stores a detailed analysis report of a relation inside `:tables :summary attributes`.
|
638
|
+
|
639
|
+
#### Attributes
|
640
|
+
|
641
|
+
**tables [Hash] - detailed table analysis**
|
642
|
+
|
643
|
+
```ruby
|
644
|
+
{
|
645
|
+
'users' => {
|
646
|
+
'id'=>1,
|
647
|
+
'name' => 'users', # table alias user in the execution plan
|
648
|
+
'debt' => <float> # debt for the table
|
649
|
+
'analysis' => {
|
650
|
+
'type' => { # attribute name
|
651
|
+
'value' => <string>, # raw value of attribute in execution plan
|
652
|
+
'tags' => {
|
653
|
+
'all' => { # tag based on the value of a attribute
|
654
|
+
'impact'=> <string>, # negative, positive, cautions
|
655
|
+
'warning'=> <string>, # Eg. 'warning to represent the issue'
|
656
|
+
'suggestions'=> <string> # Eg. 'some follow-up suggestions'
|
657
|
+
'debt' => <float> # debt for the tag
|
658
|
+
}
|
659
|
+
}
|
660
|
+
}
|
661
|
+
}
|
662
|
+
}
|
663
|
+
}
|
664
|
+
```
|
665
|
+
**summary [Hash] - hash of analysis summary**
|
666
|
+
|
667
|
+
```ruby
|
668
|
+
{
|
669
|
+
'cardinality' => {
|
670
|
+
'amount' => 10,
|
671
|
+
'warning' => 'warning to represent the issue',
|
672
|
+
'suggestions' => 'some follow up suggestions',
|
673
|
+
'debt' => 100.0
|
674
|
+
}
|
675
|
+
}
|
676
|
+
```
|
677
|
+
|
512
678
|
## Development
|
513
679
|
|
514
680
|
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
|