query_police 0.1.2.beta → 0.1.3.beta

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: b3ba602c5b4a73be62b2660559a20d3314acbe30215cdb501163d4ced4851112
4
- data.tar.gz: defbfd46a5b7702205e0a3acd7c426374eea75ceb477e8e403972ef8519fb9e8
3
+ metadata.gz: 27f13db4425088a65a440e6f5bc0092898f45b3bda80e20a8f02c810153eca9b
4
+ data.tar.gz: 7a75948db5efee3b2a21b4b2f1479f0640c1f270b8b95cebe2d04079d43f9eb9
5
5
  SHA512:
6
- metadata.gz: 06dde4464e7020f9bfb187980f77463b96e7b18d25dacfaf6b58f42cfe538d49ad27e49e62a3294b1b8f312e573d8b252a8c7163c83a0f7ee21c59fd95ee2888
7
- data.tar.gz: 94950f4b507c314b9cd2f8966516826034028a02d38e089a506067f4466f8273031c94b41c6c998299ca24c5315f061ca5315a79504d921bb7b5f1058d74ae9a
6
+ metadata.gz: 37bf74b29105ed8a838019bda964395daadfe9241f4b14f91dee9ac306d5374c6b8faf08fb090df9fdcc7f52394c47d5973079c2f31a9ff4fc20434f9bff33df
7
+ data.tar.gz: b02df2bee3c473856e8a2edd0956cf39670d58d20b7f8f66f6b63d7e749e2fa01de82bd0f898b08ff223992ebb1936599765c3dc090423c7e974776e895e10de
data/.rubocop.yml CHANGED
@@ -9,6 +9,15 @@ Style/StringLiteralsInInterpolation:
9
9
  Enabled: true
10
10
  EnforcedStyle: double_quotes
11
11
 
12
+ Style/HashTransformKeys:
13
+ Enabled: true
14
+
15
+ Style/HashTransformValues:
16
+ Enabled: true
17
+
18
+ Style/HashEachMethods:
19
+ Enabled: true
20
+
12
21
  Metrics/MethodLength:
13
22
  Max: 15
14
23
 
data/README.md CHANGED
@@ -4,11 +4,11 @@ It is a rule-based engine with custom rules to Analyze Active-Record relations u
4
4
 
5
5
  ## Installation
6
6
 
7
- Install the gem and add to the application's Gemfile by executing:
7
+ Install the gem and add it to the application's Gemfile by executing:
8
8
 
9
9
  $ bundle add query_police
10
10
 
11
- If bundler is not being used to manage dependencies, install the gem by executing:
11
+ If the bundler is not being used to manage dependencies, install the gem by executing:
12
12
 
13
13
  $ gem install query_police
14
14
 
@@ -28,8 +28,7 @@ puts analysis.pretty_analysis_for(<impact>)
28
28
  **Eg.**
29
29
  ```
30
30
  analysis = QueryPolice.analyse(
31
- User.joins('join sessions on sessions.user_email = users.email ')
32
- .where('sessions.created_at < ?', Time.now - 5.months).order('sessions.created_at')
31
+ User.joins('join orders on orders.user_id = users.id')
33
32
  )
34
33
  puts analysis.pretty_analysis_for('negative')
35
34
  # or
@@ -38,22 +37,45 @@ puts analysis.pretty_analysis({'negative' => true, 'positive' => true})
38
37
  **Results**
39
38
 
40
39
  ```
41
- table: sessions
42
- column: type
43
- impact: negative
44
- message: Entire sessions table is scanned to find matching rows, you have 1 possible keys to use.
45
- suggestion: Use index here. You can use index from possible key: ["index_sessions_on_user_email"] or add new one to sessions table as per the requirements.
46
- column: key
47
- impact: negative
48
- message: There is no index key used for sessions table, and can result into full scan of the sessions table
49
- suggestion: Please use index from possible_keys: ["index_sessions_on_user_email"] or add new one to sessions table as per the requirements.
50
- column: rows
51
- impact: negative
52
- message: 2982924 rows are being scanned per join for sessions table.
53
- suggestion: Please see if it is possible to use index from ["index_sessions_on_user_email"] or add new one to sessions table as per the requirements to reduce the number of rows scanned.
54
- ```
55
-
56
- ### Add logger for every query
40
+ query_score: 330.0
41
+
42
+ +----------------------------------------------------------------------------------------------------------------------------------+
43
+ | orders |
44
+ +------------+---------------------------------------------------------------------------------------------------------------------+
45
+ | score | 200.0 |
46
+ +------------+---------------------------------------------------------------------------------------------------------------------+
47
+ | column | type |
48
+ | impact | negative |
49
+ | tag_score | 100.0 |
50
+ | message | Entire orders table is scanned to find matching rows, you have 0 possible keys to use. |
51
+ | suggestion | Use index here. You can use index from possible key: absent or add new one to orders table as per the requirements. |
52
+ +------------+---------------------------------------------------------------------------------------------------------------------+
53
+ | column | possible_keys |
54
+ | impact | negative |
55
+ | tag_score | 50.0 |
56
+ | message | There are no possible keys for orders table to be used, can result into full scan |
57
+ | suggestion | Please add index keys for orders table |
58
+ +------------+---------------------------------------------------------------------------------------------------------------------+
59
+ | column | key |
60
+ | impact | negative |
61
+ | tag_score | 50.0 |
62
+ | message | There is no index key used for orders table, and can result into full scan of the orders table |
63
+ | suggestion | Please use index from possible_keys: absent or add new one to orders table as per the requirements. |
64
+ +------------+---------------------------------------------------------------------------------------------------------------------+
65
+ +------------------------------------------------------------------------------------+
66
+ | users |
67
+ +------------+-----------------------------------------------------------------------+
68
+ | score | 130.0 |
69
+ +------------+-----------------------------------------------------------------------+
70
+ | column | detailed#used_columns |
71
+ | impact | negative |
72
+ | tag_score | 130.0 |
73
+ | message | You have selected 18 columns, You should not select too many columns. |
74
+ | suggestion | Please only select required columns. |
75
+ +------------+-----------------------------------------------------------------------+
76
+ ```
77
+
78
+ ### Add a logger for every query
57
79
 
58
80
  Add `QueryPolice.subscribe_logger` to your initial load file like `application.rb`
59
81
 
@@ -61,24 +83,24 @@ You can make logger silence of error using `QueryPolice.subscribe_logger silent:
61
83
 
62
84
  You can change logger config using `QueryPolice logger_config: <config>`, default logger_config `{'negative' => true}`, options `positive: <Boolean>, caution: <Boolean>`.
63
85
 
64
-
86
+ ---
65
87
 
66
88
  ## How it works?
67
89
 
68
- 1. Query police converts the relation into sql query
90
+ 1. Query police converts the relation into SQL query
69
91
 
70
- 2. Query police generates execution plan using EXPLAIN and EXPLAIN format=JSON based on the configuration.
92
+ 2. Query police generates an execution plan using EXPLAIN and EXPLAIN format=JSON based on the configuration.
71
93
 
72
94
  3. Query police load rules from the config file.
73
95
 
74
96
  4. Query police apply rules on the execution plan and generate a new analysis object.
75
97
 
76
- 5. Analysis object provide different methods to print the analysis in more descriptive format.
98
+ 5. Analysis object provides different methods to print the analysis in a more descriptive format.
77
99
 
78
100
 
79
101
  ## Execution plan
80
102
 
81
- We have 2 possible execution plan:-
103
+ We have 2 possible execution plans:-
82
104
 
83
105
  Normal - using `EXPLAIN`
84
106
 
@@ -98,7 +120,7 @@ Generated using `EXPAIN <query>`
98
120
  | 1 | SIMPLE | users | NULL | eq_ref | PRIMARY,index_users_on_id | PRIMARY | 4 | development.profile.user_id |1 | 100.00 | NULL |
99
121
 
100
122
 
101
- Result for this is added as it is in the final execution plan
123
+ The result for this is added as it is in the final execution plan
102
124
 
103
125
  **Eg.**
104
126
 
@@ -172,7 +194,7 @@ Generated using `EXPAIN format=JSON <query>`
172
194
  ```
173
195
 
174
196
 
175
- Result for this is added in flatten form to final execution plan, where `detailed#` prefix is added before each key.
197
+ The result for this is added in the flattened form to the final execution plan, where the `detailed#` prefix is added before each key.
176
198
 
177
199
  **Truncated Eg.**
178
200
 
@@ -200,27 +222,27 @@ Result for this is added in flatten form to final execution plan, where `detaile
200
222
 
201
223
  ## Analysis object
202
224
 
203
- Analysis object stores a detailed analysis report of a relation inside `:tables :table_count :summary attributes`.
225
+ Analysis object stores a detailed analysis report of a relation inside `:tables :summary attributes`.
204
226
 
205
227
  #### Attributes
206
228
 
207
- **table_count [Integer] - No. Tables used in the relation**
208
-
209
229
  **tables [Hash] - detailed table analysis**
210
230
 
211
231
  ```
212
232
  {
213
233
  'users' => {
214
234
  'id'=>1,
215
- 'name'=>'users', # table alias user in the execution plan
216
- 'analysis'=>{
217
- 'type'=>{ # attribute name
235
+ 'name' => 'users', # table alias user in the execution plan
236
+ 'score' => <float> # score for the table
237
+ 'analysis' => {
238
+ 'type' => { # attribute name
218
239
  'value' => <string>, # raw value of attribute in execution plan
219
240
  'tags' => {
220
241
  'all' => { # tag based on the value of a attribute
221
242
  'impact'=> <string>, # negative, positive, cautions
222
243
  'warning'=> <string>, # Eg. 'warning to represent the issue'
223
- 'suggestions'=> <string> # Eg. 'some follow up suggestions'
244
+ 'suggestions'=> <string> # Eg. 'some follow-up suggestions'
245
+ 'score' => <float> # score for the tag
224
246
  }
225
247
  }
226
248
  }
@@ -232,10 +254,11 @@ Analysis object stores a detailed analysis report of a relation inside `:tables
232
254
 
233
255
  ```
234
256
  {
235
- 'cardinality'=>{
236
- 'amount'=>10,
237
- 'warning'=>'warning to represent the issue',
238
- 'suggestions'=>'some follow up suggestions'
257
+ 'cardinality' => {
258
+ 'amount' => 10,
259
+ 'warning' => 'warning to represent the issue',
260
+ 'suggestions' => 'some follow up suggestions',
261
+ 'score' => 100.0
239
262
  }
240
263
  }
241
264
  ```
@@ -261,7 +284,11 @@ A basic rule structure -
261
284
  "amount": <integer>
262
285
  "impact": <string>,
263
286
  "message": <string>,
264
- "suggestion": <string>
287
+ "suggestion": <string>,
288
+ "score": {
289
+ "value": <integer>,
290
+ "type": <string>
291
+ }
265
292
  }
266
293
  }
267
294
  }
@@ -278,19 +305,19 @@ A basic rule structure -
278
305
 
279
306
  - `<tag>` - direct value match eg. ALL, SIMPLE
280
307
 
281
- - `absent` - when value is missing
308
+ - `absent` - when the value is missing
282
309
 
283
310
  - `threshold` - a greater than threshold check based on the amount set inside the rule.
284
311
 
285
- - `amount` - amount of threshold need to check for
312
+ - `amount` - the amount of threshold that needs to check for
286
313
 
287
314
  - length for string
288
315
 
289
316
  - value for number
290
317
 
291
- - size for array
318
+ - size for the array
292
319
 
293
- - `impact` - impact for the rule
320
+ - `impact` - impact of the rule
294
321
 
295
322
  - `negative`
296
323
 
@@ -298,23 +325,29 @@ A basic rule structure -
298
325
 
299
326
  - `caution`
300
327
 
301
- - `message` - message need to provide the significance of the rule
328
+ - `message` - the message needs to provide the significance of the rule
302
329
 
303
330
  - `suggestion` - suggestion on how we can fix the issue
331
+ - `score` - score-related config that will be affected to final query score
332
+ - `value` - value that will be added to the query score
333
+ - `type` - the type of scoring that will be added to the query score
334
+ - `base`- value
335
+ - `relative` - value * (amount for that column in query)
336
+ - `treshold_relative` - (value - (threshold amount)) * (amount for that column in query)
304
337
 
305
338
 
306
339
 
307
340
  ### Dynamic messages and suggestion
308
341
 
309
- We can define dynamic messages and suggestion with variables provided by the engine.
342
+ We can define dynamic messages and suggestions with variables provided by the engine.
310
343
 
311
- - `$amount` - amount of the value
344
+ - `$amount` - the amount of the value
312
345
 
313
346
  - length for string
314
347
 
315
348
  - value for number
316
349
 
317
- - size for array
350
+ - size for the array
318
351
 
319
352
  - `$column` - attribute name
320
353
 
@@ -326,7 +359,7 @@ We can define dynamic messages and suggestion with variables provided by the eng
326
359
 
327
360
  - `$value` - original parsed value
328
361
 
329
- - `$<column_name>` - value of that specific column in that table
362
+ - `$<column_name>` - the value of that specific column in that table
330
363
 
331
364
  - `$amount_<column_name>` - amount of that specific column
332
365
 
@@ -349,15 +382,19 @@ We can define dynamic messages and suggestion with variables provided by the eng
349
382
  "ALL": {
350
383
  "impact": "negative",
351
384
  "message": "Entire $table table is scanned to find matching rows, you have $amount_possible_keys possible keys to use.",
352
- "suggestion": "Use index here. You can use index from possible key: $possible_keys or add new one to $table table as per the requirements."
385
+ "suggestion": "Use index here. You can use index from possible key: $possible_keys or add new one to $table table as per the requirements.",
386
+ "score": {
387
+ "value": 200,
388
+ "type": "base"
389
+ }
353
390
  }
354
391
  }
355
392
  ```
356
- For above rule dynamic message will be generated as-
393
+ For the above rule, dynamic message will be generated as-
357
394
  ```
358
395
  Entire users table is scanned to find matching rows, you have 1 possible keys to use
359
396
  ```
360
- For above rule dynamic suggestion will be generated as-
397
+ For the above rule, dynamic suggestion will be generated as-
361
398
  ```
362
399
  Use index here. You can use index from possible key: ["PRIMARY", "user_email"] or add new one to users table as per the requirements.
363
400
  ```
@@ -379,11 +416,11 @@ Use index here. You can use index from possible key: ["PRIMARY", "user_email"] o
379
416
  }
380
417
  ```
381
418
 
382
- For above rule dynamic message will be generated as-
419
+ For the above rule, dynamic message will be generated as-
383
420
  ```
384
421
  There is no index key used for users table, and can result into full scan of the users table
385
422
  ```
386
- For above rule dynamic suggestion will be generated as-
423
+ For the above rule, dynamic suggestion will be generated as-
387
424
  ```
388
425
  Please use index from possible_keys: ["PRIMARY", "user_email"] or add new one to users table as per the requirements.
389
426
  ```
@@ -406,11 +443,11 @@ Please use index from possible_keys: ["PRIMARY", "user_email"] or add new one to
406
443
  }
407
444
  }
408
445
  ```
409
- For above rule dynamic message will be generated as-
446
+ For the above rule, dynamic message will be generated as-
410
447
  ```
411
448
  There are 10 possible keys for users table, having too many index keys can be unoptimal
412
449
  ```
413
- For above rule dynamic suggestion will be generated as-
450
+ For the above rule, dynamic suggestion will be generated as-
414
451
  ```
415
452
  Please check if there are extra indexes in users table.
416
453
  ```
@@ -427,16 +464,20 @@ Please check if there are extra indexes in users table.
427
464
  "amount": 7,
428
465
  "impact": "negative",
429
466
  "message": "You have selected $amount columns, You should not select too many columns.",
430
- "suggestion": "Please only select required columns."
467
+ "suggestion": "Please only select required columns.",
468
+ "score": {
469
+ "value": 10,
470
+ "type": "treshold_relative"
471
+ }
431
472
  }
432
473
  }
433
474
  }
434
475
  ```
435
- For above rule dynamic message will be generated as-
476
+ For the above rule, dynamic message will be generated as-
436
477
  ```
437
478
  You have selected 10 columns, You should not select too many columns.
438
479
  ```
439
- For above rule dynamic suggestion will be generated as-
480
+ For the above rule, dynamic suggestions will be generated as-
440
481
  ```
441
482
  Please only select required columns.
442
483
  ```
@@ -444,24 +485,24 @@ Please only select required columns.
444
485
 
445
486
  ### Summary
446
487
 
447
- You can define similar rules for summary. Current summary attribute supported -
488
+ You can define similar rules for the summary. Current summary attribute supported -
448
489
 
449
- - `cardinality` - cardinality based on the all tables
490
+ - `cardinality` - cardinality based on all tables
450
491
 
451
- **NOTE:** You can add custom summary attributes by defining how to calculate them in `QueryPolice.add_summary` for a attribute key.
492
+ **NOTE:** You can add custom summary attributes by defining how to calculate them in `QueryPolice.add_summary` for an attribute key.
452
493
 
453
494
 
454
495
 
455
496
  ### Attributes
456
497
 
457
- There all lot of attributes for you to use based on the final execution plan.
498
+ There are a lot of attributes for you to use based on the final execution plan.
458
499
 
459
- You can use normal execution plan attribute directly.
500
+ You can use the normal execution plan attribute directly.
460
501
  Eg. `select_type, type, Extra, possible_keys`
461
502
 
462
503
  To check more keys you can use `EXPLAIN <query>`
463
504
 
464
- You can use detailed execution plan attribute can be used in flatten form with `detailed#` prefix.
505
+ You can use the detailed execution plan attribute can be used in flattened form with the `detailed#` prefix.
465
506
  Eg. `detailed#used_columns, detailed#cost_info#read_cost`
466
507
 
467
508
  To check more keys you can use `EXPLAIN format=JSON <query>`
@@ -484,4 +525,4 @@ The gem is available as open source under the terms of the [MIT License](https:/
484
525
 
485
526
  ## Code of Conduct
486
527
 
487
- Everyone interacting in the QueryPolice project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/query_police/blob/master/CODE_OF_CONDUCT.md).
528
+ Everyone interacting in the QueryPolice project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/query_police/blob/master/CODE_OF_CONDUCT.md).
@@ -0,0 +1,100 @@
1
+ # frozen_string_literal: true
2
+
3
+ # QueryPolice::Analyse
4
+ module QueryPolice
5
+ # This module define analyse methods for query police
6
+ module Analyse
7
+ def table(table, summary, rules_config)
8
+ table_analysis = {}
9
+ table_score = 0
10
+
11
+ table.each do |column, value|
12
+ summary = add_summary(summary, column, value)
13
+ next unless rules_config.dig(column).present?
14
+
15
+ table_analysis.merge!({ column => apply_rules(rules_config.dig(column), value) })
16
+ table_score += table_analysis.dig(column, "tags").map { |_, tag| tag.dig("score") }.sum.to_f
17
+ end
18
+
19
+ [table_analysis, summary, table_score]
20
+ end
21
+
22
+ def generate_summary(rules_config, summary)
23
+ summary_analysis = {}
24
+ summary_score = 0
25
+
26
+ summary.each do |column, value|
27
+ next unless rules_config.dig(column).present?
28
+
29
+ summary_analysis.merge!({ column => apply_rules(rules_config.dig(column), value) })
30
+ summary_score += summary_analysis.dig(column, "tags").map { |_, tag| tag.dig("score") }.sum.to_f
31
+ end
32
+
33
+ [summary_analysis, summary_score]
34
+ end
35
+
36
+ class << self
37
+ private
38
+
39
+ def add_summary(summary, column_name, value)
40
+ summary["cardinality"] = (summary.dig("cardinality") || 1) + value.to_f if column_name.eql?("rows")
41
+
42
+ summary
43
+ end
44
+
45
+ def apply_rules(column_rules, value)
46
+ column_rules = Constants::DEFAULT_COLUMN_RULES.merge(column_rules)
47
+ value = Transform.value(value, column_rules)
48
+ amount = Transform.amount(value, column_rules)
49
+
50
+ column_analyse = { "value" => value, "amount" => amount, "tags" => {} }
51
+
52
+ [*value].each do |tag|
53
+ tag_rule = column_rules.dig("rules", tag)
54
+ next unless tag_rule.present?
55
+
56
+ column_analyse["tags"].merge!(
57
+ { tag => Transform.tag_rule(tag_rule).merge!({ "score" => generate_score(tag_rule, amount) }) }
58
+ )
59
+ end
60
+
61
+ column_analyse["tags"].merge!(apply_threshold_rule(column_rules, amount))
62
+
63
+ column_analyse
64
+ end
65
+
66
+ def apply_threshold_rule(column_rules, amount)
67
+ threshold_rule = column_rules.dig("rules", "threshold")
68
+
69
+ if threshold_rule.present? && amount >= threshold_rule.dig("amount")
70
+ return {
71
+ "threshold" => Transform.tag_rule(threshold_rule).merge(
72
+ { "amount" => amount, "score" => generate_score(threshold_rule, amount) }
73
+ )
74
+ }
75
+ end
76
+
77
+ {}
78
+ end
79
+
80
+ def generate_score(tag_rule, amount)
81
+ score = tag_rule.dig("score", "value")
82
+
83
+ case tag_rule.dig("score", "type").to_s
84
+ when "base"
85
+ score.to_f
86
+ when "relative"
87
+ amount.to_f * score.to_f
88
+ when "treshold_relative"
89
+ (amount - tag_rule.dig("amount")).to_f * score.to_f
90
+ else
91
+ 0
92
+ end
93
+ end
94
+ end
95
+
96
+ module_function :table, :generate_summary
97
+ end
98
+
99
+ private_constant :Analyse
100
+ end
@@ -14,13 +14,13 @@ module QueryPolice
14
14
  # @return [String]
15
15
  def dynamic_message(opts)
16
16
  table, column, tag, type = opts.values_at("table", "column", "tag", "type")
17
- message = tables.dig(table, "analysis", column, "tags", tag, type) || ""
17
+ message = query_analytic.dig(table, "analysis", column, "tags", tag, type) || ""
18
18
 
19
19
  variables = message.scan(/\$(\w+)/).uniq.map { |var| var[0] }
20
20
  variables.each do |var|
21
21
  value = dynamic_value_of(var, opts)
22
22
 
23
- message.gsub!(/\$#{var}/, value.to_s) if value.present?
23
+ message = message.gsub(/\$#{var}/, value.to_s) unless value.nil?
24
24
  end
25
25
 
26
26
  message
@@ -32,14 +32,14 @@ module QueryPolice
32
32
 
33
33
  def relative_value_of(var, table)
34
34
  value_type = var.match(/amount_/).present? ? "amount" : "value"
35
- tables.dig(table, "analysis", var.gsub(/amount_/, ""), value_type)
35
+ query_analytic.dig(table, "analysis", var.gsub(/amount_/, ""), value_type)
36
36
  end
37
37
 
38
38
  # dynamic variable methods
39
39
  def amount(opts)
40
40
  table, column = opts.values_at("table", "column")
41
41
 
42
- tables.dig(table, "analysis", column, "amount")
42
+ query_analytic.dig(table, "analysis", column, "amount")
43
43
  end
44
44
 
45
45
  def column(opts)
@@ -49,9 +49,18 @@ module QueryPolice
49
49
  def impact(opts)
50
50
  table, column, tag = opts.values_at("table", "column", "tag")
51
51
 
52
- impact = tables.dig(table, "analysis", column, "tags", tag, "impact")
52
+ impact = query_analytic.dig(table, "analysis", column, "tags", tag, "impact")
53
53
 
54
- opts.dig("colours").present? ? impact.send(IMPACTS[impact].colour) : impact
54
+ opts.dig("colours").present? ? impact.send(IMPACTS.dig(impact, "colour")) : impact
55
+ end
56
+
57
+ def score(opts)
58
+ table, column, tag = opts.values_at("table", "column", "tag")
59
+
60
+ impact = query_analytic.dig(table, "analysis", column, "tags", tag, "impact")
61
+ score = query_analytic.dig(table, "analysis", column, "tags", tag, "score")
62
+
63
+ opts.dig("colours").present? ? score.to_s.send(IMPACTS.dig(impact, "colour")) : score
55
64
  end
56
65
 
57
66
  def table(opts)
@@ -65,7 +74,7 @@ module QueryPolice
65
74
  def value(opts)
66
75
  table, column = opts.values_at("table", "column")
67
76
 
68
- tables.dig(table, "analysis", column, "value")
77
+ query_analytic.dig(table, "analysis", column, "value")
69
78
  end
70
79
  end
71
80
  end
@@ -18,16 +18,18 @@ module QueryPolice
18
18
  # Eg.
19
19
  # {
20
20
  # "users" => {
21
- # "id"=>1,
22
- # "name"=>"users",
23
- # "analysis"=>{
24
- # "type"=>{
25
- # "value" => "all",
21
+ # "id" => 1,
22
+ # "name" => "users",
23
+ # "score" => 100.0,
24
+ # "analysis" => {
25
+ # "type" => {
26
+ # "value" => all",
26
27
  # "tags" => {
27
28
  # "all" => {
28
- # "impact"=>"negative",
29
- # "warning"=>"warning to represent the issue",
30
- # "suggestions"=>"some follow up suggestions"
29
+ # "impact" => "negative",
30
+ # "warning" => "warning to represent the issue",
31
+ # "suggestions" => "some follow up suggestions",
32
+ # "score" => 100.0
31
33
  # }
32
34
  # }
33
35
  # }
@@ -37,55 +39,65 @@ module QueryPolice
37
39
  # summary [Hash] hash of analysis summary
38
40
  # Eg.
39
41
  # {
40
- # "cardinality"=>{
41
- # "amount"=>10,
42
- # "warning"=>"warning to represent the issue",
43
- # "suggestions"=>"some follow up suggestions"
42
+ # "cardinality" => {
43
+ # "amount" => 10,
44
+ # "warning" => "warning to represent the issue",
45
+ # "suggestions" => "some follow up suggestions",
46
+ # "score" => 100.0
44
47
  # }
45
48
  # }
46
49
  def initialize
47
50
  @table_count = 0
48
51
  @tables = {}
52
+ @table_score = 0
49
53
  @summary = {}
54
+ @summary_score = 0
50
55
  end
51
56
 
52
- attr_accessor :table_count, :tables, :summary
57
+ attr_accessor :tables, :summary
53
58
 
54
59
  # register a table analysis in analysis object
55
60
  # @param name [String] name of the table
56
61
  # @param table_analysis [Hash] analysis of a table
62
+ # @param score [Integer] score for that table
57
63
  # Eg.
58
64
  # {
59
- # "id"=>1,
60
- # "name"=>"users",
61
- # "analysis"=>{
62
- # "type"=>[
65
+ # "id" => 1,
66
+ # "name" => "users",
67
+ # "score" => 100.0
68
+ # "analysis" => {
69
+ # "type" => [
63
70
  # {
64
- # "tag"=>"all",
65
- # "impact"=>"negative",
66
- # "warning"=>"warning to represent the issue",
67
- # "suggestions"=>"some follow up suggestions"
71
+ # "tag" => "all",
72
+ # "impact" => "negative",
73
+ # "warning" => "warning to represent the issue",
74
+ # "suggestions" => "some follow up suggestions",
75
+ # "score" => 100.0
68
76
  # }
69
77
  # ]
70
78
  # }
71
79
  # }
72
- def register_table(name, table_analysis)
73
- self.table_count += 1
80
+ def register_table(name, table_analysis, score)
81
+ @table_count += 1
74
82
  tables.merge!(
75
83
  {
76
84
  name => {
77
- "id" => self.table_count,
85
+ "id" => @table_count,
78
86
  "name" => name,
87
+ "score" => score,
79
88
  "analysis" => table_analysis
80
89
  }
81
90
  }
82
91
  )
92
+
93
+ @table_score += score
83
94
  end
84
95
 
85
96
  # register summary based in different attributes
86
97
  # @param summary [Hash] hash of summary of analysis
87
- def register_summary(summary)
98
+ def register_summary(summary, score)
88
99
  self.summary.merge!(summary)
100
+ @summary_score += score
89
101
  end
90
102
 
91
103
  # to get analysis in pretty format with warnings and suggestions
@@ -93,6 +105,7 @@ module QueryPolice
93
105
  # @return [String] pretty analysis
94
106
  def pretty_analysis(opts)
95
107
  final_message = ""
108
+ opts = opts.with_indifferent_access
96
109
 
97
110
  opts.slice(*IMPACTS.keys).each do |impact, value|
98
111
  final_message += pretty_analysis_for(impact) if value.present?
@@ -105,49 +118,80 @@ module QueryPolice
105
118
  # @param impact [String]
106
119
  # @return [String] pretty analysis
107
120
  def pretty_analysis_for(impact)
108
- final_message = ""
121
+ final_message = "query_score: #{query_score}\n\n"
109
122
 
110
- tables.keys.each do |table|
111
- table_message = table_pretty_analysis(table, { impact => true })
123
+ query_analytic.each_key do |table|
124
+ table_message = query_pretty_analysis(table, { impact => true })
112
125
 
113
- final_message += "table: #{table}\n#{table_message}\n" if table_message.present?
126
+ final_message += "#{table_message}\n" if table_message.present?
114
127
  end
115
128
 
116
129
  final_message
117
130
  end
118
131
 
132
+ # to get the final score
133
+ def query_score
134
+ @table_score + @summary_score
135
+ end
136
+
119
137
  # to get analysis in pretty format with warnings and suggestions for a table
120
138
  # @param table [String] - table name
121
139
  # @param opts [Hash] - possible options [positive: <boolean>, negative: <boolean>, caution: <boolean>]
122
140
  # @return [String] pretty analysis
123
- def table_pretty_analysis(table, opts)
124
- table_message = ""
141
+ def query_pretty_analysis(table, opts)
142
+ table_analytics = Terminal::Table.new(title: table)
143
+ table_analytics_present = false
144
+ table_analytics.add_row(["score", query_analytic.dig(table, "score")])
125
145
 
126
- tables.dig(table, "analysis").each do |column, column_analysis|
127
- tags_message = ""
128
- column_analysis.dig("tags").each do |tag, tag_analysis|
129
- next unless opts.dig(tag_analysis.dig("impact")).present?
146
+ opts = opts.with_indifferent_access
130
147
 
131
- tags_message += tag_pretty_analysis(table, column, tag)
132
- end
148
+ query_analytic.dig(table, "analysis").each do |column, _|
149
+ column_analytics = column_analytic(table, column, opts)
150
+ next unless column_analytics.present?
133
151
 
134
- table_message += "column: #{column}\n#{tags_message}" if tags_message.present?
152
+ table_analytics_present = true
153
+ table_analytics.add_separator
154
+ table_analytics.add_row(["column", column])
155
+ column_analytics.each { |row| table_analytics.add_row(row) }
135
156
  end
136
157
 
137
- table_message
158
+ table_analytics_present ? table_analytics : nil
138
159
  end
139
160
 
140
161
  private
141
162
 
142
- def tag_pretty_analysis(table, column, tag)
143
- tag_message = ""
163
+ def column_analytic(table, column, opts)
164
+ column_analytics = []
165
+
166
+ query_analytic.dig(table, "analysis", column, "tags").each do |tag, tag_analysis|
167
+ next unless opts.dig(tag_analysis.dig("impact")).present?
168
+
169
+ column_analytics += tag_analytic(table, column, tag)
170
+ end
171
+
172
+ column_analytics
173
+ end
174
+
175
+ def query_analytic
176
+ tables.merge(
177
+ "summary" => {
178
+ "name" => "summary",
179
+ "score" => @summary_score,
180
+ "analysis" => summary
181
+ }
182
+ )
183
+ end
184
+
185
+ def tag_analytic(table, column, tag)
186
+ tag_message = []
144
187
 
145
188
  opts = { "table" => table, "column" => column, "tag" => tag }
146
189
  message = dynamic_message(opts.merge({ "type" => "message" }))
147
190
  suggestion = dynamic_message(opts.merge({ "type" => "suggestion" }))
148
- tag_message += "impact: #{impact(opts.merge({ "colours" => true }))}\n"
149
- tag_message += "message: #{message}\n"
150
- tag_message += "suggestion: #{suggestion}\n" if suggestion.present?
191
+ tag_message << ["impact", impact(opts.merge({ "colours" => true }))]
192
+ tag_message << ["tag_score", score(opts.merge({ "colours" => true }))]
193
+ tag_message << ["message", message]
194
+ tag_message << ["suggestion", suggestion] if suggestion.present?
151
195
 
152
196
  tag_message
153
197
  end
@@ -62,7 +62,11 @@
62
62
  "ALL": {
63
63
  "impact": "negative",
64
64
  "message": "Entire $table table is scanned to find matching rows, you have $amount_possible_keys possible keys to use.",
65
- "suggestion": "Use index here. You can use index from possible key: $possible_keys or add new one to $table table as per the requirements."
65
+ "suggestion": "Use index here. You can use index from possible key: $possible_keys or add new one to $table table as per the requirements.",
66
+ "score": {
67
+ "value": 100,
68
+ "type": "base"
69
+ }
66
70
  }
67
71
  }
68
72
  },
@@ -86,13 +90,21 @@
86
90
  "absent": {
87
91
  "impact": "negative",
88
92
  "message": "There are no possible keys for $table table to be used, can result into full scan",
89
- "suggestion": "Please add index keys for $table table"
93
+ "suggestion": "Please add index keys for $table table",
94
+ "score": {
95
+ "value": 50,
96
+ "type": "base"
97
+ }
90
98
  },
91
99
  "threshold": {
92
100
  "amount": 5,
93
101
  "impact": "negative",
94
102
  "message": "There are $amount possible keys for $table table, having too many index keys can be unoptimal",
95
- "suggestion": "Please check if there are extra indexes in $table table."
103
+ "suggestion": "Please check if there are extra indexes in $table table.",
104
+ "score": {
105
+ "value": 20,
106
+ "type": "treshold_relative"
107
+ }
96
108
  }
97
109
  }
98
110
  },
@@ -103,7 +115,11 @@
103
115
  "absent": {
104
116
  "impact": "negative",
105
117
  "message": "There is no index key used for $table table, and can result into full scan of the $table table",
106
- "suggestion": "Please use index from possible_keys: $possible_keys or add new one to $table table as per the requirements."
118
+ "suggestion": "Please use index from possible_keys: $possible_keys or add new one to $table table as per the requirements.",
119
+ "score": {
120
+ "value": 50,
121
+ "type": "base"
122
+ }
107
123
  }
108
124
  }
109
125
  },
@@ -122,15 +138,14 @@
122
138
  "value_type": "array",
123
139
  "delimiter": ";",
124
140
  "rules": {
125
- "Using temporary": {
126
- "impact": "",
127
- "message": "",
128
- "suggestion": ""
129
- },
130
141
  "Using filesort": {
131
142
  "impact": "negative",
132
143
  "message": "A file-based algorithm in being applied over your result, This can be inefficient and result into long query time.",
133
- "suggestion": "Please ensure either result set is small or use proper index."
144
+ "suggestion": "Please ensure either result set is small or use proper index.",
145
+ "score": {
146
+ "value": 50,
147
+ "type": "base"
148
+ }
134
149
  },
135
150
  "Using join buffer": {
136
151
  "impact": "",
@@ -145,26 +160,50 @@
145
160
  }
146
161
  },
147
162
  "detailed#used_columns": {
148
- "description": "",
163
+ "description": "number of column used to execute the query",
149
164
  "value_type": "array",
150
165
  "rules": {
151
166
  "threshold": {
152
- "amount": 7,
167
+ "amount": 5,
153
168
  "impact": "negative",
154
169
  "message": "You have selected $amount columns, You should not select too many columns.",
155
- "suggestion": "Please only select required columns."
170
+ "suggestion": "Please only select required columns.",
171
+ "score": {
172
+ "value": 10,
173
+ "type": "treshold_relative"
174
+ }
156
175
  }
157
176
  }
158
177
  },
159
- "cardinality": {
160
- "description": "",
178
+ "detailed#cost_info#read_cost ": {
179
+ "description": "read cost to execute the query",
161
180
  "value_type": "number",
162
181
  "rules": {
163
182
  "threshold": {
164
183
  "amount": 100,
165
184
  "impact": "negative",
185
+ "message": "The read cost of query is to high, most likely you are scanning to many rows",
186
+ "suggestion": "Please use proper index, query only requried data and ensure you are using proper joins.",
187
+ "score": {
188
+ "value": 0.01,
189
+ "type": "relative"
190
+ }
191
+ }
192
+ }
193
+ },
194
+ "cardinality": {
195
+ "description": "total cardinality of the query",
196
+ "value_type": "number",
197
+ "rules": {
198
+ "threshold": {
199
+ "amount": 500,
200
+ "impact": "negative",
166
201
  "message": "The cardinality of table is $amount, and its too high.",
167
- "suggestion": "Please use proper index, query only requried data and ensure you are using proper joins."
202
+ "suggestion": "Please use proper index, query only requried data and ensure you are using proper joins.",
203
+ "score": {
204
+ "value": 0.01,
205
+ "type": "relative"
206
+ }
168
207
  }
169
208
  }
170
209
  }
@@ -0,0 +1,31 @@
1
+ # frozen_string_literal: true
2
+
3
+ # QueryPolice::Transform
4
+ module QueryPolice
5
+ # This module define transformer methods for query police
6
+ module Transform
7
+ def amount(value, column_rules)
8
+ return 0 if value.eql?("absent")
9
+
10
+ column_rules.dig("value_type").eql?("number") ? value.to_f : value.size
11
+ end
12
+
13
+ def tag_rule(tag_rule)
14
+ tag_rule.slice("impact", "suggestion", "message")
15
+ end
16
+
17
+ def value(value, column_rules)
18
+ return "absent" if value.nil?
19
+
20
+ if column_rules.dig("value_type").eql?("array") && column_rules.dig("delimiter").present?
21
+ value = value.split(column_rules.dig("delimiter")).map(&:strip)
22
+ end
23
+
24
+ value
25
+ end
26
+
27
+ module_function :amount, :tag_rule, :value
28
+ end
29
+
30
+ private_constant :Transform
31
+ end
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module QueryPolice
4
- VERSION = "0.1.2.beta"
4
+ VERSION = "0.1.3.beta"
5
5
  end
data/lib/query_police.rb CHANGED
@@ -3,14 +3,18 @@
3
3
  require "active_record"
4
4
  require "active_support/notifications"
5
5
  require "active_support/core_ext"
6
- require "json"
6
+ require "colorize"
7
7
  require "forwardable"
8
+ require "json"
9
+ require "terminal-table"
8
10
 
11
+ require_relative "query_police/analyse"
9
12
  require_relative "query_police/analysis"
10
- require_relative "query_police/constants"
11
13
  require_relative "query_police/config"
14
+ require_relative "query_police/constants"
12
15
  require_relative "query_police/explain"
13
16
  require_relative "query_police/helper"
17
+ require_relative "query_police/transform"
14
18
  require_relative "query_police/version"
15
19
 
16
20
  # This module provides tools to analyse your queries based on custom rules
@@ -40,12 +44,12 @@ module QueryPolice
40
44
  query_plan = Explain.full_explain(relation, config.detailed?)
41
45
 
42
46
  query_plan.each do |table|
43
- table_analysis, summary = analyse_table(table, summary, rules_config)
47
+ table_analysis, summary, table_score = Analyse.table(table, summary, rules_config)
44
48
 
45
- analysis.register_table(table.dig("table"), table_analysis)
49
+ analysis.register_table(table.dig("table"), table_analysis, table_score)
46
50
  end
47
51
 
48
- analysis.register_summary(generate_summary_analysis(rules_config, summary))
52
+ analysis.register_summary(*Analyse.generate_summary(rules_config, summary))
49
53
 
50
54
  analysis
51
55
  end
@@ -71,90 +75,6 @@ module QueryPolice
71
75
 
72
76
  class << self
73
77
  attr_accessor :config
74
-
75
- private
76
-
77
- def add_summary(summary, column_name, value)
78
- summary["cardinality"] = (summary.dig("cardinality") || 1) + value.to_f if column_name.eql?("rows")
79
-
80
- summary
81
- end
82
-
83
- def analyse_table(table, summary, rules_config)
84
- table_analysis = {}
85
-
86
- table.each do |column, value|
87
- summary = add_summary(summary, column, value)
88
- next unless rules_config.dig(column).present?
89
-
90
- table_analysis.merge!({ column => apply_rules(rules_config.dig(column), value) })
91
- end
92
-
93
- [table_analysis, summary]
94
- end
95
-
96
- def apply_rules(column_rules, value)
97
- column_rules = Constants::DEFAULT_COLUMN_RULES.merge(column_rules)
98
- value = transform_value(value, column_rules)
99
- amount = transform_amount(value, column_rules)
100
-
101
- column_analyse = { "value" => value, "amount" => amount, "tags" => {} }
102
-
103
- [*value].each do |tag|
104
- tag_rule = column_rules.dig("rules", tag)
105
- next unless tag_rule.present?
106
-
107
- column_analyse["tags"].merge!({ tag => transform_tag_rule(tag_rule) })
108
- end
109
-
110
- column_analyse["tags"].merge!(apply_threshold_rule(column_rules, amount))
111
-
112
- column_analyse
113
- end
114
-
115
- def apply_threshold_rule(column_rules, amount)
116
- threshold_rule = column_rules.dig("rules", "threshold")
117
-
118
- if threshold_rule.present? && amount >= threshold_rule.dig("amount")
119
- return {
120
- "threshold" => transform_tag_rule(threshold_rule).merge(
121
- { "amount" => amount }
122
- )
123
- }
124
- end
125
-
126
- {}
127
- end
128
-
129
- def generate_summary_analysis(rules_config, summary)
130
- summary_analysis = {}
131
-
132
- summary.each do |column, value|
133
- next unless rules_config.dig(column).present?
134
-
135
- summary_analysis.merge!({ column => apply_rules(rules_config.dig(column), value) })
136
- end
137
-
138
- summary_analysis
139
- end
140
-
141
- def transform_amount(value, column_rules)
142
- column_rules.dig("value_type").eql?("number") ? value.to_f : value.size
143
- end
144
-
145
- def transform_tag_rule(tag_rule)
146
- tag_rule.slice("impact", "suggestion", "message")
147
- end
148
-
149
- def transform_value(value, column_rules)
150
- value ||= "absent"
151
-
152
- if column_rules.dig("value_type").eql?("array") && column_rules.dig("delimiter").present?
153
- value = value.split(column_rules.dig("delimiter")).map(&:strip)
154
- end
155
-
156
- value
157
- end
158
78
  end
159
79
 
160
80
  module_function :analyse, :subscribe_logger, *CONFIG_METHODS
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: query_police
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.2.beta
4
+ version: 0.1.3.beta
5
5
  platform: ruby
6
6
  authors:
7
7
  - strikeraryu
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2023-04-24 00:00:00.000000000 Z
11
+ date: 2023-08-16 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: activerecord
@@ -19,7 +19,7 @@ dependencies:
19
19
  version: 3.0.0
20
20
  - - "<"
21
21
  - !ruby/object:Gem::Version
22
- version: 6.0.0
22
+ version: 8.0.0
23
23
  type: :runtime
24
24
  prerelease: false
25
25
  version_requirements: !ruby/object:Gem::Requirement
@@ -29,7 +29,7 @@ dependencies:
29
29
  version: 3.0.0
30
30
  - - "<"
31
31
  - !ruby/object:Gem::Version
32
- version: 6.0.0
32
+ version: 8.0.0
33
33
  - !ruby/object:Gem::Dependency
34
34
  name: activesupport
35
35
  requirement: !ruby/object:Gem::Requirement
@@ -39,7 +39,7 @@ dependencies:
39
39
  version: 3.0.0
40
40
  - - "<"
41
41
  - !ruby/object:Gem::Version
42
- version: 6.0.0
42
+ version: 8.0.0
43
43
  type: :runtime
44
44
  prerelease: false
45
45
  version_requirements: !ruby/object:Gem::Requirement
@@ -49,7 +49,47 @@ dependencies:
49
49
  version: 3.0.0
50
50
  - - "<"
51
51
  - !ruby/object:Gem::Version
52
- version: 6.0.0
52
+ version: 8.0.0
53
+ - !ruby/object:Gem::Dependency
54
+ name: colorize
55
+ requirement: !ruby/object:Gem::Requirement
56
+ requirements:
57
+ - - ">="
58
+ - !ruby/object:Gem::Version
59
+ version: 0.5.0
60
+ - - "<"
61
+ - !ruby/object:Gem::Version
62
+ version: 0.8.1
63
+ type: :runtime
64
+ prerelease: false
65
+ version_requirements: !ruby/object:Gem::Requirement
66
+ requirements:
67
+ - - ">="
68
+ - !ruby/object:Gem::Version
69
+ version: 0.5.0
70
+ - - "<"
71
+ - !ruby/object:Gem::Version
72
+ version: 0.8.1
73
+ - !ruby/object:Gem::Dependency
74
+ name: terminal-table
75
+ requirement: !ruby/object:Gem::Requirement
76
+ requirements:
77
+ - - ">="
78
+ - !ruby/object:Gem::Version
79
+ version: 3.0.0
80
+ - - "<"
81
+ - !ruby/object:Gem::Version
82
+ version: 3.0.2
83
+ type: :runtime
84
+ prerelease: false
85
+ version_requirements: !ruby/object:Gem::Requirement
86
+ requirements:
87
+ - - ">="
88
+ - !ruby/object:Gem::Version
89
+ version: 3.0.0
90
+ - - "<"
91
+ - !ruby/object:Gem::Version
92
+ version: 3.0.2
53
93
  description:
54
94
  email:
55
95
  - striker.aryu56@gmail.com
@@ -66,6 +106,7 @@ files:
66
106
  - README.md
67
107
  - Rakefile
68
108
  - lib/query_police.rb
109
+ - lib/query_police/analyse.rb
69
110
  - lib/query_police/analysis.rb
70
111
  - lib/query_police/analysis/dynamic_message.rb
71
112
  - lib/query_police/config.rb
@@ -73,6 +114,7 @@ files:
73
114
  - lib/query_police/explain.rb
74
115
  - lib/query_police/helper.rb
75
116
  - lib/query_police/rules.json
117
+ - lib/query_police/transform.rb
76
118
  - lib/query_police/version.rb
77
119
  - sig/query_police.rbs
78
120
  homepage: https://github.com/strikeraryu/query_police.git
@@ -100,5 +142,6 @@ requirements: []
100
142
  rubygems_version: 3.2.3
101
143
  signing_key:
102
144
  specification_version: 4
103
- summary: This gem provides tools to analyze your queries based on custom rules.
145
+ summary: This gem provides tools to analyze your queries based on custom rules and
146
+ detect bad queries.
104
147
  test_files: []