query_police 0.1.4.beta → 0.1.5.beta
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +408 -242
- data/examples/rules/json/absent_rule.json +13 -0
- data/examples/rules/json/basic_rule.json +22 -0
- data/examples/rules/json/complex_detailed_rule.json +18 -0
- data/examples/rules/json/threshold_rule.json +15 -0
- data/examples/rules/yaml/absent_rule.yml +11 -0
- data/examples/rules/yaml/basic_rule.yml +18 -0
- data/examples/rules/yaml/complex_detailed_rule.yml +13 -0
- data/examples/rules/yaml/threshold_rule.yml +12 -0
- data/lib/query_police/analyse.rb +16 -16
- data/lib/query_police/analysis/dynamic_message.rb +3 -3
- data/lib/query_police/analysis.rb +67 -39
- data/lib/query_police/config.rb +11 -6
- data/lib/query_police/constants.rb +6 -3
- data/lib/query_police/explain.rb +15 -2
- data/lib/query_police/helper.rb +13 -3
- data/lib/query_police/rules.json +66 -57
- data/lib/query_police/version.rb +1 -1
- data/lib/query_police.rb +58 -25
- metadata +12 -4
data/README.md
CHANGED
@@ -1,16 +1,16 @@
|
|
1
1
|
# QueryPolice
|
2
2
|
|
3
|
-
It is a rule-based engine with custom rules to Analyze Active-Record relations using explain results
|
3
|
+
It is a rule-based engine with custom rules to Analyze Active-Record relations using explain results and detect bad queries.
|
4
4
|
|
5
5
|
## Installation
|
6
6
|
|
7
7
|
Install the gem and add it to the application's Gemfile by executing:
|
8
8
|
|
9
|
-
$
|
9
|
+
$ gem 'query_police', '~> 0.1.5.beta'
|
10
10
|
|
11
11
|
If the bundler is not being used to manage dependencies, install the gem by executing:
|
12
12
|
|
13
|
-
$ gem install query_police
|
13
|
+
$ gem install query_police --pre
|
14
14
|
|
15
15
|
---
|
16
16
|
|
@@ -20,320 +20,318 @@ If the bundler is not being used to manage dependencies, install the gem by exec
|
|
20
20
|
|
21
21
|
Use `QueryPolice.analyse` to generate an analysis object for your Active-Record relation or query and then you can use pretty print on the object.
|
22
22
|
|
23
|
-
```
|
24
|
-
analysis = QueryPolice.analyse(<
|
25
|
-
puts analysis.
|
23
|
+
```ruby
|
24
|
+
analysis = QueryPolice.analyse(<query>)
|
25
|
+
puts analysis.pretty_analysis
|
26
26
|
```
|
27
27
|
|
28
28
|
**Eg.**
|
29
|
-
```
|
29
|
+
```ruby
|
30
30
|
analysis = QueryPolice.analyse(
|
31
|
-
User.joins('join orders on orders.user_id = users.id')
|
31
|
+
User.joins('join orders on orders.user_id = users.id') # analyse query using active record relation
|
32
32
|
)
|
33
|
-
puts analysis.pretty_analysis_for('negative')
|
34
33
|
# or
|
35
|
-
|
34
|
+
analysis = QueryPolice.analyse(
|
35
|
+
"select * from users join orders on orders.user_id = users.id" # analyse query using query string
|
36
|
+
)
|
37
|
+
|
38
|
+
puts analysis.pretty_analysis
|
36
39
|
```
|
37
40
|
**Results**
|
38
41
|
|
42
|
+
**Note:** [Query debt significance](#significance)
|
43
|
+
|
44
|
+
|
39
45
|
```
|
40
|
-
|
46
|
+
query_debt: 330.0
|
41
47
|
|
42
48
|
+----------------------------------------------------------------------------------------------------------------------------------+
|
43
49
|
| orders |
|
44
50
|
+------------+---------------------------------------------------------------------------------------------------------------------+
|
45
|
-
|
|
51
|
+
| debt | 200.0 |
|
46
52
|
+------------+---------------------------------------------------------------------------------------------------------------------+
|
47
53
|
| column | type |
|
48
54
|
| impact | negative |
|
49
|
-
|
|
55
|
+
| tag_debt | 100.0 |
|
50
56
|
| message | Entire orders table is scanned to find matching rows, you have 0 possible keys to use. |
|
51
57
|
| suggestion | Use index here. You can use index from possible key: absent or add new one to orders table as per the requirements. |
|
52
58
|
+------------+---------------------------------------------------------------------------------------------------------------------+
|
53
59
|
| column | possible_keys |
|
54
60
|
| impact | negative |
|
55
|
-
|
|
61
|
+
| tag_debt | 50.0 |
|
56
62
|
| message | There are no possible keys for orders table to be used, can result into full scan |
|
57
63
|
| suggestion | Please add index keys for orders table |
|
58
64
|
+------------+---------------------------------------------------------------------------------------------------------------------+
|
59
65
|
| column | key |
|
60
66
|
| impact | negative |
|
61
|
-
|
|
67
|
+
| tag_debt | 50.0 |
|
62
68
|
| message | There is no index key used for orders table, and can result into full scan of the orders table |
|
63
69
|
| suggestion | Please use index from possible_keys: absent or add new one to orders table as per the requirements. |
|
64
70
|
+------------+---------------------------------------------------------------------------------------------------------------------+
|
65
71
|
+------------------------------------------------------------------------------------+
|
66
72
|
| users |
|
67
73
|
+------------+-----------------------------------------------------------------------+
|
68
|
-
|
|
74
|
+
| debt | 130.0 |
|
69
75
|
+------------+-----------------------------------------------------------------------+
|
70
76
|
| column | detailed#used_columns |
|
71
77
|
| impact | negative |
|
72
|
-
|
|
78
|
+
| tag_debt | 130.0 |
|
73
79
|
| message | You have selected 18 columns, You should not select too many columns. |
|
74
80
|
| suggestion | Please only select required columns. |
|
75
81
|
+------------+-----------------------------------------------------------------------+
|
76
82
|
```
|
77
83
|
|
78
|
-
###
|
79
|
-
|
80
|
-
Add `QueryPolice.subscribe_logger` to your initial load file like `application.rb`
|
81
|
-
|
82
|
-
You can make logger silence of error using `QueryPolice.subscribe_logger silent: true`.
|
83
|
-
|
84
|
-
You can change logger config using `QueryPolice logger_config: <config>`, default logger_config `{'negative' => true}`, options `positive: <Boolean>, caution: <Boolean>`.
|
85
|
-
|
86
|
-
---
|
84
|
+
### Analysis for a Impact
|
87
85
|
|
88
|
-
|
86
|
+
To print pretty analysis for different impacts
|
87
|
+
```ruby
|
88
|
+
analysis = QueryPolice.analyse("select * from users")
|
89
|
+
puts analysis.pretty_analysis_for('positive') # impact negative, positive, caution
|
89
90
|
|
90
|
-
|
91
|
-
|
92
|
-
|
93
|
-
|
94
|
-
|
95
|
-
|
96
|
-
|
97
|
-
|
98
|
-
|
99
|
-
|
100
|
-
|
101
|
-
|
102
|
-
|
103
|
-
We have 2 possible execution plans:-
|
104
|
-
|
105
|
-
Normal - using `EXPLAIN`
|
106
|
-
|
107
|
-
Detailed - using `EXPLAIN format=JSON`
|
108
|
-
|
109
|
-
**NOTE:** By default Detailed execution plan is added in the final execution plan, you can remove that by `QueryPolice.detailed=false`
|
110
|
-
|
111
|
-
### Normal execution plan
|
112
|
-
|
113
|
-
Generated using `EXPAIN <query>`
|
114
|
-
|
115
|
-
**Result**
|
116
|
-
|
117
|
-
| id | select_type | table | partitions | type. | possible_keys | key | key_len | ref | rows | filtered | Extra |
|
118
|
-
|----|-------------|---------|------------|--------|---------------------------|---------------------|---------|-----------------------------|------|----------|--------------------------|
|
119
|
-
| 1 | SIMPLE | profile | NULL. | index | fk_rails_249a7ebca1 | fk_rails_249a7ebca1 | 5 | NULL | 603 | 100.00 | Using where; Using index |
|
120
|
-
| 1 | SIMPLE | users | NULL | eq_ref | PRIMARY,index_users_on_id | PRIMARY | 4 | development.profile.user_id |1 | 100.00 | NULL |
|
121
|
-
|
122
|
-
|
123
|
-
The result for this is added as it is in the final execution plan
|
91
|
+
# puts
|
92
|
+
# +----------------------------------------------------------+
|
93
|
+
# | users |
|
94
|
+
# +-----------+----------------------------------------------+
|
95
|
+
# | debt | 330.0 |
|
96
|
+
# +-----------+----------------------------------------------+
|
97
|
+
# | column | select_type |
|
98
|
+
# | impact | positive |
|
99
|
+
# | tag_debt | 0 |
|
100
|
+
# | message | A simple query without subqueries or unions. |
|
101
|
+
# +-----------+----------------------------------------------+
|
102
|
+
```
|
124
103
|
|
125
|
-
|
104
|
+
### Analysis for Multiple Impacts
|
126
105
|
|
106
|
+
To print pretty analysis for multiple impacts, default: `{ 'negative' => true, 'caution' => true }`
|
107
|
+
```ruby
|
108
|
+
analysis = QueryPolice.analyse("select * from users")
|
109
|
+
puts analysis.pretty_analysis({'negative' => true, 'positive' => true})
|
127
110
|
```
|
128
|
-
{
|
129
|
-
"profile" => {
|
130
|
-
"id" => 1,
|
131
|
-
"select_type" => "SIMPLE",
|
132
|
-
"table" => "profile",
|
133
|
-
"partitions" => nil,
|
134
|
-
"type" => "index",
|
135
|
-
"possible_keys" => "fk_rails_249a7ebca1",
|
136
|
-
"key" => "fk_rails_249a7ebca1",
|
137
|
-
"key_len" => "5",
|
138
|
-
"ref" => nil,
|
139
|
-
"rows" => 603,
|
140
|
-
"filtered" => 100.0,
|
141
|
-
"Extra" => "Using where; Using index"
|
142
|
-
},
|
143
|
-
"users" => {
|
144
|
-
"id" => 1,
|
145
|
-
"select_type" => "SIMPLE",
|
146
|
-
"table" => "users",
|
147
|
-
"partitions" => nil,
|
148
|
-
"type" => "eq_ref",
|
149
|
-
"possible_keys" => "PRIMARY,index_users_on_id",
|
150
|
-
"key" => "PRIMARY",
|
151
|
-
"key_len" => "4",
|
152
|
-
"ref" => "development.profile.user_id",
|
153
|
-
"rows" => 1,
|
154
|
-
"filtered" => 100.0,
|
155
|
-
"Extra" => nil
|
156
|
-
}
|
157
|
-
}
|
158
|
-
```
|
159
|
-
|
160
|
-
### Detailed execution plan
|
161
111
|
|
162
|
-
|
112
|
+
### Query debt
|
163
113
|
|
164
|
-
|
114
|
+
To get the final debt of the query
|
115
|
+
```ruby
|
116
|
+
analysis = QueryPolice.analyse("select * from users")
|
117
|
+
puts analysis.query_debt
|
165
118
|
|
119
|
+
# puts
|
120
|
+
# 100.0
|
166
121
|
```
|
167
|
-
{
|
168
|
-
"query_block": {
|
169
|
-
"select_id": 1,
|
170
|
-
"cost_info": {
|
171
|
-
"query_cost": "850.20"
|
172
|
-
},
|
173
|
-
"nested_loop": [
|
174
|
-
{
|
175
|
-
"table": {
|
176
|
-
"table_name": "profile",
|
177
|
-
"access_type": "index",
|
178
|
-
"key_length": "5",
|
179
|
-
"cost_info": {
|
180
|
-
"read_cost": "6.00",
|
181
|
-
"eval_cost": "120.60",
|
182
|
-
"prefix_cost": "126.60",
|
183
|
-
"data_read_per_join": "183K"
|
184
|
-
},
|
185
|
-
"used_columns": [
|
186
|
-
"user_id"
|
187
|
-
],
|
188
|
-
"attached_condition": "(`development`.`profile`.`user_id` is not null)"
|
189
|
-
}
|
190
|
-
}
|
191
|
-
]
|
192
|
-
}
|
193
|
-
}
|
194
|
-
```
|
195
|
-
|
196
122
|
|
197
|
-
|
123
|
+
#### Significance
|
124
|
+
Query debt signifies the quality of the query, high value represents a bad query.
|
125
|
+
- `0 - 199` - Good Query
|
126
|
+
- `200 - 499` - Potentially Bad query
|
127
|
+
- `>=500` - Bad query
|
198
128
|
|
199
|
-
|
129
|
+
### Word wrap
|
200
130
|
|
201
|
-
|
202
|
-
|
203
|
-
|
204
|
-
|
205
|
-
|
206
|
-
|
207
|
-
|
208
|
-
|
209
|
-
|
210
|
-
|
211
|
-
|
212
|
-
|
213
|
-
|
131
|
+
To change the word wrap width for the pretty analysis result, default value: `100`
|
132
|
+
```ruby
|
133
|
+
analysis = QueryPolice.analyse("select * from users")
|
134
|
+
puts analysis.pretty_analysis_for('positive', {'wrap_width' => 40})
|
135
|
+
# or
|
136
|
+
puts analysis.pretty_analysis({'positive' => true, 'wrap_width' => 40})
|
137
|
+
|
138
|
+
# puts
|
139
|
+
# +--------------------------------------------------+
|
140
|
+
# | users |
|
141
|
+
# +-----------+--------------------------------------+
|
142
|
+
# | debt | 330.0 |
|
143
|
+
# +-----------+--------------------------------------+
|
144
|
+
# | column | select_type |
|
145
|
+
# | impact | positive |
|
146
|
+
# | tag_debt | 0 |
|
147
|
+
# | message | A simple query without subqueries or |
|
148
|
+
# | | unions. |
|
149
|
+
# +-----------+--------------------------------------+
|
150
|
+
|
151
|
+
puts analysis.pretty_analysis_for('positive', {'wrap_width' => 20})
|
152
|
+
# or
|
153
|
+
puts analysis.pretty_analysis({'positive' => true, 'wrap_width' => 20})
|
154
|
+
|
155
|
+
# puts
|
156
|
+
# +--------------------------------+
|
157
|
+
# | users |
|
158
|
+
# +-----------+--------------------+
|
159
|
+
# | debt | 330.0 |
|
160
|
+
# +-----------+--------------------+
|
161
|
+
# | column | select_type |
|
162
|
+
# | impact | positive |
|
163
|
+
# | tag_debt | 0 |
|
164
|
+
# | message | A simple query |
|
165
|
+
# | | without subqueries |
|
166
|
+
# | | or unions. |
|
167
|
+
# +-----------+--------------------+
|
168
|
+
```
|
169
|
+
|
170
|
+
### Skip footer
|
171
|
+
|
172
|
+
To skip the footer (added in [query police config](#analysis-footer)) after an analysis of a query
|
173
|
+
```ruby
|
174
|
+
analysis = QueryPolice.analyse("select * from users")
|
175
|
+
puts analysis.pretty_analysis_for('positive', {'skip_footer' => true})
|
176
|
+
# or
|
177
|
+
puts analysis.pretty_analysis({'positive' => true, 'skip_footer' => true})
|
214
178
|
```
|
215
179
|
|
180
|
+
### Analysis Footer
|
181
|
+
To define a footer text that will be added after an analysis for a query, by default there is no footer
|
182
|
+
```ruby
|
183
|
+
QueryPolice.analysis_footer = 'Please check more details with this link...'
|
184
|
+
# or
|
185
|
+
QueryPolice.configure do |config|
|
186
|
+
config.analysis_footer = 'Please check more details with this link...'
|
187
|
+
end
|
188
|
+
|
189
|
+
# puts
|
190
|
+
# +----------------------------------------------------------+
|
191
|
+
# | users |
|
192
|
+
# +-----------+----------------------------------------------+
|
193
|
+
# | debt | 330.0 |
|
194
|
+
# +-----------+----------------------------------------------+
|
195
|
+
# | column | select_type |
|
196
|
+
# | impact | positive |
|
197
|
+
# | tag_debt | 0 |
|
198
|
+
# | message | A simple query without subqueries or unions. |
|
199
|
+
# +-----------+----------------------------------------------+
|
200
|
+
# Please check more details with this link...
|
201
|
+
```
|
202
|
+
|
203
|
+
### Custom rules path
|
204
|
+
|
205
|
+
To define custom rules path (More details about [how to define custom rules](#how-to-define-custom-rules))
|
206
|
+
```ruby
|
207
|
+
QueryPolice.rules_path = 'path/to/rules/file.<json/yml>'
|
208
|
+
# or
|
209
|
+
QueryPolice.configure do |config|
|
210
|
+
config.rules_path = 'path/to/rules/file.<json/yml>'
|
211
|
+
end
|
212
|
+
```
|
216
213
|
|
214
|
+
### Verbosity
|
217
215
|
|
218
|
-
|
219
|
-
|
220
|
-
`
|
216
|
+
Verbosity defines which `EXPLAIN` result should be used for analysis. (More details about [EXPLAIN vs Detailed EXPLAIN](#execution-plan))
|
217
|
+
- `basic` - It uses only `EXPLAIN`(basic) result
|
218
|
+
- `detailed` - It uses both `EXPLAIN`(basic) and `EXPLAIN format=json`(detailed) results `(default value)`
|
219
|
+
```ruby
|
220
|
+
QueryPolice.verbosity = "detailed"
|
221
|
+
# or
|
222
|
+
QueryPolice.configure do |config|
|
223
|
+
config.verbosity = "detailed"
|
224
|
+
end
|
225
|
+
```
|
221
226
|
|
227
|
+
### Disable Actions
|
222
228
|
|
223
|
-
|
229
|
+
To disable actions that are performed on each query.
|
230
|
+
```ruby
|
231
|
+
# Note: A logger action is already added, so query police will log pretty analysis after each query by default
|
232
|
+
QueryPolice.action_enabled = false
|
233
|
+
# or
|
234
|
+
QueryPolice.configure do |config|
|
235
|
+
config.action_enabled = false
|
236
|
+
end
|
237
|
+
```
|
224
238
|
|
225
|
-
|
239
|
+
### Custom Actions
|
226
240
|
|
227
|
-
|
241
|
+
To add custom actions, by default a logger action is already added and enabled (More details about [Analysis Object](#analysis-object))
|
242
|
+
```ruby
|
243
|
+
QueryPolice.add_action do |analysis| # analysis object for the query
|
244
|
+
puts analysis.tables
|
245
|
+
end
|
246
|
+
```
|
228
247
|
|
229
|
-
|
248
|
+
### Logger config
|
230
249
|
|
231
|
-
|
232
|
-
|
233
|
-
|
234
|
-
|
235
|
-
|
236
|
-
|
237
|
-
|
238
|
-
'type' => { # attribute name
|
239
|
-
'value' => <string>, # raw value of attribute in execution plan
|
240
|
-
'tags' => {
|
241
|
-
'all' => { # tag based on the value of a attribute
|
242
|
-
'impact'=> <string>, # negative, positive, cautions
|
243
|
-
'warning'=> <string>, # Eg. 'warning to represent the issue'
|
244
|
-
'suggestions'=> <string> # Eg. 'some follow-up suggestions'
|
245
|
-
'score' => <float> # score for the tag
|
246
|
-
}
|
247
|
-
}
|
248
|
-
}
|
249
|
-
}
|
250
|
-
}
|
251
|
-
}
|
252
|
-
```
|
253
|
-
**summary [Hash] - hash of analysis summary**
|
250
|
+
To change the logger options which will be used to generate analysis
|
251
|
+
```ruby
|
252
|
+
QueryPolice.logger_options = {'negative' => true}
|
253
|
+
# or
|
254
|
+
QueryPolice.configure do |config|
|
255
|
+
config.logger_options = {'negative' => true}
|
256
|
+
end
|
254
257
|
|
255
|
-
|
256
|
-
|
257
|
-
'cardinality' => {
|
258
|
-
'amount' => 10,
|
259
|
-
'warning' => 'warning to represent the issue',
|
260
|
-
'suggestions' => 'some follow up suggestions',
|
261
|
-
'score' => 100.0
|
262
|
-
}
|
263
|
-
}
|
258
|
+
# default logger_config: {'negative' => true, 'caution' => true}
|
259
|
+
# options negative: <Boolean>, positive: <Boolean>, caution: <Boolean>, wrap_width: <Integer>, skip_footer: <Boolean>
|
264
260
|
```
|
265
261
|
|
262
|
+
---
|
266
263
|
|
264
|
+
## How to define custom rules?
|
267
265
|
|
268
|
-
|
269
|
-
|
270
|
-
Rules defined in the json file at config_path is applied to the execution plan. We have variety of option to define rules.
|
266
|
+
Rules defined in the `json/yaml` file at rules_path is applied to the execution plan. Query Police have variety of option to define rules.
|
271
267
|
|
272
268
|
You can change this by `QueryPolice.rules_path=<path>` and define your own rules
|
273
269
|
|
274
270
|
### Rule Structure
|
271
|
+
**Note:** Check Query Police default rules defined at [rules.json](lib/query_police/rules.json) or examples at [examples/rules/](examples/rules/) for better clarity
|
275
272
|
|
276
273
|
A basic rule structure -
|
277
|
-
|
274
|
+
|
275
|
+
JSON
|
276
|
+
```json
|
278
277
|
"<column_name>": {
|
279
|
-
"description": <string
|
280
|
-
"value_type": <string
|
281
|
-
"delimiter": <string
|
278
|
+
"description": "<string>",
|
279
|
+
"value_type": "<string>",
|
280
|
+
"delimiter": "<string>",
|
282
281
|
"rules": {
|
283
282
|
"<rule>": {
|
284
|
-
"amount": <integer>
|
285
|
-
"impact": <string
|
286
|
-
"message": <string
|
287
|
-
"suggestion": <string
|
288
|
-
"
|
289
|
-
"value": <integer
|
290
|
-
"type": <string>
|
283
|
+
"amount": "<integer>",
|
284
|
+
"impact": "<string>",
|
285
|
+
"message": "<string>",
|
286
|
+
"suggestion": "<string>",
|
287
|
+
"debt": {
|
288
|
+
"value": "<integer>",
|
289
|
+
"type": "<string>"
|
291
290
|
}
|
292
291
|
}
|
293
292
|
}
|
294
293
|
}
|
295
294
|
```
|
296
|
-
|
297
|
-
|
295
|
+
YAML
|
296
|
+
```yaml
|
297
|
+
<column_name>:
|
298
|
+
description: <string>
|
299
|
+
value_type: <string>
|
300
|
+
delimiter: <string>
|
301
|
+
rules:
|
302
|
+
<rule>:
|
303
|
+
amount: <integer>
|
304
|
+
impact: <string>
|
305
|
+
message: <string>
|
306
|
+
suggestion: <string>
|
307
|
+
debt:
|
308
|
+
value: <integer>
|
309
|
+
type: <string>
|
310
|
+
```
|
311
|
+
- `<column_name>` - attribute name in the final execution plan. (more details about [attributes](#attributes-for-rules))
|
298
312
|
- `description` - description of the attribute
|
299
|
-
|
300
313
|
- `value_type` - value type of the attribute
|
301
|
-
|
302
314
|
- `delimiter` - delimiter to parse array type attribute values, if no delimiter is passed engine will consider value is already in array form.
|
303
|
-
|
304
315
|
- `<rule>` - kind of rule for the attribute
|
305
|
-
|
306
316
|
- `<tag>` - direct value match eg. ALL, SIMPLE
|
307
|
-
|
308
317
|
- `absent` - when the value is missing
|
309
|
-
|
310
318
|
- `threshold` - a greater than threshold check based on the amount set inside the rule.
|
311
|
-
|
312
319
|
- `amount` - the amount of threshold that needs to check for
|
313
|
-
|
314
320
|
- length for string
|
315
|
-
|
316
321
|
- value for number
|
317
|
-
|
318
322
|
- size for the array
|
319
|
-
|
320
323
|
- `impact` - impact of the rule
|
321
|
-
|
322
324
|
- `negative`
|
323
|
-
|
324
325
|
- `postive`
|
325
|
-
|
326
326
|
- `caution`
|
327
|
-
|
328
327
|
- `message` - the message needs to provide the significance of the rule
|
329
|
-
|
330
328
|
- `suggestion` - suggestion on how we can fix the issue
|
331
|
-
- `
|
332
|
-
- `value` - value that will be added to the query
|
333
|
-
- `type` - the type of scoring that will be added to the query
|
329
|
+
- `debt` - debt-related config that will be affected to final query debt
|
330
|
+
- `value` - value that will be added to the query debt
|
331
|
+
- `type` - the type of scoring that will be added to the query debt
|
334
332
|
- `base`- value
|
335
333
|
- `relative` - value * (amount for that column in query)
|
336
|
-
- `
|
334
|
+
- `threshold_relative` - (value - (threshold amount)) * (amount for that column in query)
|
337
335
|
|
338
336
|
|
339
337
|
|
@@ -342,34 +340,22 @@ A basic rule structure -
|
|
342
340
|
We can define dynamic messages and suggestions with variables provided by the engine.
|
343
341
|
|
344
342
|
- `$amount` - the amount of the value
|
345
|
-
|
346
343
|
- length for string
|
347
|
-
|
348
344
|
- value for number
|
349
|
-
|
350
345
|
- size for the array
|
351
|
-
|
352
346
|
- `$column` - attribute name
|
353
|
-
|
354
347
|
- `$impact` - impact for the rule
|
355
|
-
|
356
348
|
- `$table` - table alias used in the plan
|
357
|
-
|
358
349
|
- `$tag` - tag for which rule is applied
|
359
|
-
|
360
350
|
- `$value` - original parsed value
|
361
|
-
|
362
351
|
- `$<column_name>` - the value of that specific column in that table
|
363
|
-
|
364
352
|
- `$amount_<column_name>` - amount of that specific column
|
365
353
|
|
366
|
-
|
367
|
-
|
368
354
|
### Rules Examples
|
369
355
|
|
370
356
|
#### Basic rule example
|
371
|
-
|
372
|
-
```
|
357
|
+
File: [JSON](examples/rules/json/basic_rule.json) | [YAML](examples/rules/yaml/basic_rule.yml)
|
358
|
+
```json
|
373
359
|
"type": {
|
374
360
|
"description": "Join used in the query for a specific table.",
|
375
361
|
"value_type": "string",
|
@@ -383,11 +369,12 @@ We can define dynamic messages and suggestions with variables provided by the en
|
|
383
369
|
"impact": "negative",
|
384
370
|
"message": "Entire $table table is scanned to find matching rows, you have $amount_possible_keys possible keys to use.",
|
385
371
|
"suggestion": "Use index here. You can use index from possible key: $possible_keys or add new one to $table table as per the requirements.",
|
386
|
-
"
|
372
|
+
"debt": {
|
387
373
|
"value": 200,
|
388
374
|
"type": "base"
|
389
375
|
}
|
390
376
|
}
|
377
|
+
}
|
391
378
|
}
|
392
379
|
```
|
393
380
|
For the above rule, dynamic message will be generated as-
|
@@ -401,8 +388,8 @@ Use index here. You can use index from possible key: ["PRIMARY", "user_email"] o
|
|
401
388
|
|
402
389
|
|
403
390
|
#### Absent rule example
|
404
|
-
|
405
|
-
```
|
391
|
+
File: [JSON](examples/rules/json/absent_rule.json) | [YAML](examples/rules/yaml/absent_rule.yml)
|
392
|
+
```json
|
406
393
|
"key": {
|
407
394
|
"description": "index key used for the table",
|
408
395
|
"value_type": "string",
|
@@ -427,8 +414,8 @@ Please use index from possible_keys: ["PRIMARY", "user_email"] or add new one to
|
|
427
414
|
|
428
415
|
|
429
416
|
#### Threshold rule example
|
430
|
-
|
431
|
-
```
|
417
|
+
File: [JSON](examples/rules/json/threshold_rule.json) | [YAML](examples/rules/yaml/threshold_rule.yml)
|
418
|
+
```json
|
432
419
|
"possible_keys": {
|
433
420
|
"description": "Index keys possible for a specifc table",
|
434
421
|
"value_type": "array",
|
@@ -454,8 +441,8 @@ Please check if there are extra indexes in users table.
|
|
454
441
|
|
455
442
|
|
456
443
|
#### Complex Detailed rule example
|
457
|
-
|
458
|
-
```
|
444
|
+
File: [JSON](examples/rules/json/complex_detailed_rule.json) | [YAML](examples/rules/yaml/complex_detailed_rule.yml)
|
445
|
+
```json
|
459
446
|
"detailed#used_columns": {
|
460
447
|
"description": "",
|
461
448
|
"value_type": "array",
|
@@ -465,9 +452,9 @@ Please check if there are extra indexes in users table.
|
|
465
452
|
"impact": "negative",
|
466
453
|
"message": "You have selected $amount columns, You should not select too many columns.",
|
467
454
|
"suggestion": "Please only select required columns.",
|
468
|
-
"
|
455
|
+
"debt": {
|
469
456
|
"value": 10,
|
470
|
-
"type": "
|
457
|
+
"type": "threshold_relative"
|
471
458
|
}
|
472
459
|
}
|
473
460
|
}
|
@@ -483,9 +470,9 @@ Please only select required columns.
|
|
483
470
|
```
|
484
471
|
|
485
472
|
|
486
|
-
### Summary
|
473
|
+
### Summary in Analysis
|
487
474
|
|
488
|
-
You can define similar rules for the summary. Current
|
475
|
+
You can define similar rules for the summary. Current only one attribute is supported in summary -
|
489
476
|
|
490
477
|
- `cardinality` - cardinality based on all tables
|
491
478
|
|
@@ -493,7 +480,7 @@ You can define similar rules for the summary. Current summary attribute supporte
|
|
493
480
|
|
494
481
|
|
495
482
|
|
496
|
-
### Attributes
|
483
|
+
### Attributes for Rules
|
497
484
|
|
498
485
|
There are a lot of attributes for you to use based on the final execution plan.
|
499
486
|
|
@@ -509,6 +496,185 @@ To check more keys you can use `EXPLAIN format=JSON <query>`
|
|
509
496
|
|
510
497
|
---
|
511
498
|
|
499
|
+
## How Query Police works?
|
500
|
+
|
501
|
+
1. Query police converts the relation into SQL query
|
502
|
+
|
503
|
+
2. Query police generates an execution plan using EXPLAIN and EXPLAIN format=JSON based on the configuration.
|
504
|
+
|
505
|
+
3. Query police load rules from the config file.
|
506
|
+
|
507
|
+
4. Query police apply rules on the execution plan and generate a new analysis object.
|
508
|
+
|
509
|
+
5. Analysis object provides different methods to print the analysis in a more descriptive format.
|
510
|
+
|
511
|
+
|
512
|
+
## Execution plan
|
513
|
+
|
514
|
+
We have 2 possible execution plans:-
|
515
|
+
|
516
|
+
Normal - using `EXPLAIN`
|
517
|
+
|
518
|
+
Detailed - using `EXPLAIN format=JSON`
|
519
|
+
|
520
|
+
**NOTE:** By default Detailed execution plan is added in the final execution plan, you can remove that by `QueryPolice.detailed=false`
|
521
|
+
|
522
|
+
### Normal execution plan
|
523
|
+
|
524
|
+
Generated using `EXPAIN <query>`
|
525
|
+
|
526
|
+
**Result**
|
527
|
+
|
528
|
+
| id | select_type | table | partitions | type. | possible_keys | key | key_len | ref | rows | filtered | Extra |
|
529
|
+
|----|-------------|---------|------------|--------|---------------------------|---------------------|---------|-----------------------------|------|----------|--------------------------|
|
530
|
+
| 1 | SIMPLE | profile | NULL. | index | fk_rails_249a7ebca1 | fk_rails_249a7ebca1 | 5 | NULL | 603 | 100.00 | Using where; Using index |
|
531
|
+
| 1 | SIMPLE | users | NULL | eq_ref | PRIMARY,index_users_on_id | PRIMARY | 4 | development.profile.user_id |1 | 100.00 | NULL |
|
532
|
+
|
533
|
+
|
534
|
+
The result for this is added as it is in the final execution plan
|
535
|
+
|
536
|
+
**Eg.**
|
537
|
+
|
538
|
+
```ruby
|
539
|
+
{
|
540
|
+
"profile" => {
|
541
|
+
"id" => 1,
|
542
|
+
"select_type" => "SIMPLE",
|
543
|
+
"table" => "profile",
|
544
|
+
"partitions" => nil,
|
545
|
+
"type" => "index",
|
546
|
+
"possible_keys" => "fk_rails_249a7ebca1",
|
547
|
+
"key" => "fk_rails_249a7ebca1",
|
548
|
+
"key_len" => "5",
|
549
|
+
"ref" => nil,
|
550
|
+
"rows" => 603,
|
551
|
+
"filtered" => 100.0,
|
552
|
+
"Extra" => "Using where; Using index"
|
553
|
+
},
|
554
|
+
"users" => {
|
555
|
+
"id" => 1,
|
556
|
+
"select_type" => "SIMPLE",
|
557
|
+
"table" => "users",
|
558
|
+
"partitions" => nil,
|
559
|
+
"type" => "eq_ref",
|
560
|
+
"possible_keys" => "PRIMARY,index_users_on_id",
|
561
|
+
"key" => "PRIMARY",
|
562
|
+
"key_len" => "4",
|
563
|
+
"ref" => "development.profile.user_id",
|
564
|
+
"rows" => 1,
|
565
|
+
"filtered" => 100.0,
|
566
|
+
"Extra" => nil
|
567
|
+
}
|
568
|
+
}
|
569
|
+
```
|
570
|
+
|
571
|
+
### Detailed execution plan
|
572
|
+
|
573
|
+
Generated using `EXPAIN format=JSON <query>`
|
574
|
+
|
575
|
+
**Truncated Result**
|
576
|
+
|
577
|
+
```ruby
|
578
|
+
{
|
579
|
+
"query_block": {
|
580
|
+
"select_id": 1,
|
581
|
+
"cost_info": {
|
582
|
+
"query_cost": "850.20"
|
583
|
+
},
|
584
|
+
"nested_loop": [
|
585
|
+
{
|
586
|
+
"table": {
|
587
|
+
"table_name": "profile",
|
588
|
+
"access_type": "index",
|
589
|
+
"key_length": "5",
|
590
|
+
"cost_info": {
|
591
|
+
"read_cost": "6.00",
|
592
|
+
"eval_cost": "120.60",
|
593
|
+
"prefix_cost": "126.60",
|
594
|
+
"data_read_per_join": "183K"
|
595
|
+
},
|
596
|
+
"used_columns": [
|
597
|
+
"user_id"
|
598
|
+
],
|
599
|
+
"attached_condition": "(`development`.`profile`.`user_id` is not null)"
|
600
|
+
}
|
601
|
+
}
|
602
|
+
]
|
603
|
+
}
|
604
|
+
}
|
605
|
+
```
|
606
|
+
|
607
|
+
|
608
|
+
The result for this is added in the flattened form to the final execution plan, where the `detailed#` prefix is added before each key.
|
609
|
+
|
610
|
+
**Truncated Eg.**
|
611
|
+
|
612
|
+
```ruby
|
613
|
+
{
|
614
|
+
"detailed#key_length" => "5",
|
615
|
+
"detailed#rows_examined_per_scan" => 603,
|
616
|
+
"detailed#rows_produced_per_join" => 603,
|
617
|
+
"detailed#filtered" => "100.00",
|
618
|
+
"detailed#using_index" => true,
|
619
|
+
"detailed#cost_info#read_cost" => "6.00",
|
620
|
+
"detailed#cost_info#eval_cost" => "120.60",
|
621
|
+
"detailed#cost_info#prefix_cost" => "126.60",
|
622
|
+
"detailed#cost_info#data_read_per_join" => "183K",
|
623
|
+
"detailed#used_columns" => ["user_id"]
|
624
|
+
...
|
625
|
+
```
|
626
|
+
|
627
|
+
|
628
|
+
|
629
|
+
##### Flatten
|
630
|
+
|
631
|
+
`{a: {b: 1}, c: 2}` is converted into `{a#b: 1, c: 2}`.
|
632
|
+
|
633
|
+
---
|
634
|
+
|
635
|
+
## Analysis object
|
636
|
+
|
637
|
+
Analysis object stores a detailed analysis report of a relation inside `:tables :summary attributes`.
|
638
|
+
|
639
|
+
#### Attributes
|
640
|
+
|
641
|
+
**tables [Hash] - detailed table analysis**
|
642
|
+
|
643
|
+
```ruby
|
644
|
+
{
|
645
|
+
'users' => {
|
646
|
+
'id'=>1,
|
647
|
+
'name' => 'users', # table alias user in the execution plan
|
648
|
+
'debt' => <float> # debt for the table
|
649
|
+
'analysis' => {
|
650
|
+
'type' => { # attribute name
|
651
|
+
'value' => <string>, # raw value of attribute in execution plan
|
652
|
+
'tags' => {
|
653
|
+
'all' => { # tag based on the value of a attribute
|
654
|
+
'impact'=> <string>, # negative, positive, cautions
|
655
|
+
'warning'=> <string>, # Eg. 'warning to represent the issue'
|
656
|
+
'suggestions'=> <string> # Eg. 'some follow-up suggestions'
|
657
|
+
'debt' => <float> # debt for the tag
|
658
|
+
}
|
659
|
+
}
|
660
|
+
}
|
661
|
+
}
|
662
|
+
}
|
663
|
+
}
|
664
|
+
```
|
665
|
+
**summary [Hash] - hash of analysis summary**
|
666
|
+
|
667
|
+
```ruby
|
668
|
+
{
|
669
|
+
'cardinality' => {
|
670
|
+
'amount' => 10,
|
671
|
+
'warning' => 'warning to represent the issue',
|
672
|
+
'suggestions' => 'some follow up suggestions',
|
673
|
+
'debt' => 100.0
|
674
|
+
}
|
675
|
+
}
|
676
|
+
```
|
677
|
+
|
512
678
|
## Development
|
513
679
|
|
514
680
|
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
|