pgdexter 0.2.1 → 0.3.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +8 -0
- data/README.md +98 -25
- data/guides/Hosted-Postgres.md +1 -1
- data/lib/dexter/client.rb +4 -1
- data/lib/dexter/collector.rb +2 -1
- data/lib/dexter/csv_log_parser.rb +13 -0
- data/lib/dexter/indexer.rb +107 -82
- data/lib/dexter/log_parser.rb +2 -2
- data/lib/dexter/processor.rb +8 -2
- data/lib/dexter/query.rb +10 -1
- data/lib/dexter/version.rb +1 -1
- data/lib/dexter.rb +1 -0
- metadata +4 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 16f61626f5003a801ec12cda3f663390a57e69d8
|
4
|
+
data.tar.gz: 238ee1b24dab3f473accc2ca9a5156fd31e8d19e
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: ccae6266196fc84ea0fc69177b80144f8747996ff0693b6a7492e34d095b2cb400e9ce3299741326c50a5eb267ef48be368384a2ab27553002eb93ac7b6d0922
|
7
|
+
data.tar.gz: 944bc27bb1ffff546c2f33aa10aa1b8371ba43ae2be889a6b9445c3d3a93ca36fee0cfd57487134a5ea807e37583f74b1e5e9de7714fa30364ab28f64a5679e8
|
data/CHANGELOG.md
CHANGED
data/README.md
CHANGED
@@ -12,8 +12,8 @@ First, install [HypoPG](https://github.com/dalibo/hypopg) on your database serve
|
|
12
12
|
|
13
13
|
```sh
|
14
14
|
cd /tmp
|
15
|
-
curl -L https://github.com/dalibo/hypopg/archive/1.
|
16
|
-
cd hypopg-1.
|
15
|
+
curl -L https://github.com/dalibo/hypopg/archive/1.1.0.tar.gz | tar xz
|
16
|
+
cd hypopg-1.1.0
|
17
17
|
make
|
18
18
|
make install # may need sudo
|
19
19
|
```
|
@@ -45,23 +45,23 @@ tail -F -n +1 <log-file> | dexter <connection-options>
|
|
45
45
|
This finds slow queries and generates output like:
|
46
46
|
|
47
47
|
```
|
48
|
-
|
49
|
-
|
50
|
-
|
51
|
-
|
52
|
-
|
53
|
-
|
54
|
-
|
55
|
-
|
56
|
-
|
48
|
+
Started
|
49
|
+
Processing 189 new query fingerprints
|
50
|
+
Index found: public.genres_movies (genre_id)
|
51
|
+
Index found: public.genres_movies (movie_id)
|
52
|
+
Index found: public.movies (title)
|
53
|
+
Index found: public.ratings (movie_id)
|
54
|
+
Index found: public.ratings (rating)
|
55
|
+
Index found: public.ratings (user_id)
|
56
|
+
Processing 12 new query fingerprints
|
57
57
|
```
|
58
58
|
|
59
59
|
To be safe, Dexter will not create indexes unless you pass the `--create` flag. In this case, you’ll see:
|
60
60
|
|
61
61
|
```
|
62
|
-
|
63
|
-
|
64
|
-
|
62
|
+
Index found: public.ratings (user_id)
|
63
|
+
Creating index: CREATE INDEX CONCURRENTLY ON "public"."ratings" ("user_id")
|
64
|
+
Index created: 15243 ms
|
65
65
|
```
|
66
66
|
|
67
67
|
## Connection Options
|
@@ -84,30 +84,58 @@ and connection strings:
|
|
84
84
|
host=localhost port=5432 dbname=mydb
|
85
85
|
```
|
86
86
|
|
87
|
-
##
|
87
|
+
## Collecting Queries
|
88
88
|
|
89
|
-
|
90
|
-
--- | --- | ---
|
91
|
-
exclude | prevent specific tables from being indexed | None
|
92
|
-
interval | time to wait between processing queries, in seconds | 60
|
93
|
-
log-level | `debug` gives additional info for suggested indexes<br />`debug2` gives additional info for processed queries<br />`error` suppresses logging | info
|
94
|
-
log-sql | log SQL statements executed | false
|
95
|
-
min-time | only process queries consuming a min amount of DB time, in minutes | 0
|
89
|
+
There are many ways to collect queries. For real-time indexing, pipe your logfile:
|
96
90
|
|
97
|
-
|
91
|
+
```sh
|
92
|
+
tail -F -n +1 <log-file> | dexter <connection-options>
|
93
|
+
```
|
98
94
|
|
99
|
-
|
95
|
+
Pass a single statement with:
|
100
96
|
|
101
97
|
```sh
|
102
98
|
dexter <connection-options> -s "SELECT * FROM ..."
|
103
99
|
```
|
104
100
|
|
105
|
-
or files
|
101
|
+
or pass files:
|
106
102
|
|
107
103
|
```sh
|
108
104
|
dexter <connection-options> <file1> <file2>
|
109
105
|
```
|
110
106
|
|
107
|
+
or use the [pg_stat_statements](https://www.postgresql.org/docs/current/static/pgstatstatements.html) extension:
|
108
|
+
|
109
|
+
```sh
|
110
|
+
dexter <connection-options> --pg-stat-statements
|
111
|
+
```
|
112
|
+
|
113
|
+
### Collection Options
|
114
|
+
|
115
|
+
To prevent one-off queries from being indexed, specify a minimum number of calls before a query is considered for indexing
|
116
|
+
|
117
|
+
```sh
|
118
|
+
dexter --min-calls 100
|
119
|
+
```
|
120
|
+
|
121
|
+
You can do the same for total time a query has run
|
122
|
+
|
123
|
+
```sh
|
124
|
+
dexter --min-time 10 # minutes
|
125
|
+
```
|
126
|
+
|
127
|
+
Specify the format
|
128
|
+
|
129
|
+
```sh
|
130
|
+
dexter --input-format csv
|
131
|
+
```
|
132
|
+
|
133
|
+
When steaming logs, specify the time to wait between processing queries
|
134
|
+
|
135
|
+
```sh
|
136
|
+
dexter --interval 60 # seconds
|
137
|
+
```
|
138
|
+
|
111
139
|
## Examples
|
112
140
|
|
113
141
|
Ubuntu with PostgreSQL 9.6
|
@@ -122,6 +150,36 @@ Homebrew on Mac
|
|
122
150
|
tail -F -n +1 /usr/local/var/postgres/server.log | dexter dbname
|
123
151
|
```
|
124
152
|
|
153
|
+
## Tables
|
154
|
+
|
155
|
+
You can exclude large or write-heavy tables from indexing with:
|
156
|
+
|
157
|
+
```sh
|
158
|
+
dexter --exclude table1,table2
|
159
|
+
```
|
160
|
+
|
161
|
+
Alternatively, you can specify which tables to index with:
|
162
|
+
|
163
|
+
```sh
|
164
|
+
dexter --include table3,table4
|
165
|
+
```
|
166
|
+
|
167
|
+
## Debugging
|
168
|
+
|
169
|
+
See how Dexter is processing queries with:
|
170
|
+
|
171
|
+
```sh
|
172
|
+
dexter --log-sql --log-level debug2
|
173
|
+
```
|
174
|
+
|
175
|
+
## Analyze
|
176
|
+
|
177
|
+
For best results, make sure your tables have been recently analyzed so statistics are up-to-date. You can ask Dexter to analyze tables it comes across that haven’t been analyzed in the past hour with:
|
178
|
+
|
179
|
+
```sh
|
180
|
+
dexter --analyze
|
181
|
+
```
|
182
|
+
|
125
183
|
## Hosted Postgres
|
126
184
|
|
127
185
|
Some hosted providers like Amazon RDS and Heroku do not support the HypoPG extension, which Dexter needs to run. See [how to use Dexter](guides/Hosted-Postgres.md) in these cases.
|
@@ -130,6 +188,21 @@ Some hosted providers like Amazon RDS and Heroku do not support the HypoPG exten
|
|
130
188
|
|
131
189
|
[Here are some ideas](https://github.com/ankane/dexter/issues/1)
|
132
190
|
|
191
|
+
## Upgrading
|
192
|
+
|
193
|
+
Run:
|
194
|
+
|
195
|
+
```sh
|
196
|
+
gem install pgdexter
|
197
|
+
```
|
198
|
+
|
199
|
+
To use master, run:
|
200
|
+
|
201
|
+
```sh
|
202
|
+
gem install specific_install
|
203
|
+
gem specific_install https://github.com/ankane/dexter.git
|
204
|
+
```
|
205
|
+
|
133
206
|
## Thanks
|
134
207
|
|
135
208
|
This software wouldn’t be possible without [HypoPG](https://github.com/dalibo/hypopg), which allows you to create hypothetical indexes, and [pg_query](https://github.com/lfittl/pg_query), which allows you to parse and fingerprint queries. A big thanks to Dalibo and Lukas Fittl respectively.
|
data/guides/Hosted-Postgres.md
CHANGED
data/lib/dexter/client.rb
CHANGED
@@ -29,10 +29,13 @@ module Dexter
|
|
29
29
|
dexter [options]
|
30
30
|
|
31
31
|
Options:)
|
32
|
+
o.boolean "--analyze", "analyze tables that haven't been analyzed in the past hour", default: false
|
32
33
|
o.boolean "--create", "create indexes", default: false
|
33
34
|
o.array "--exclude", "prevent specific tables from being indexed"
|
34
35
|
o.string "--include", "only include specific tables"
|
36
|
+
o.string "--input-format", "input format", default: "stderr"
|
35
37
|
o.integer "--interval", "time to wait between processing queries, in seconds", default: 60
|
38
|
+
o.float "--min-calls", "only process queries that have been called a certain number of times", default: 0
|
36
39
|
o.float "--min-time", "only process queries that have consumed a certain amount of DB time, in minutes", default: 0
|
37
40
|
o.boolean "--pg-stat-statements", "use pg_stat_statements", default: false, help: false
|
38
41
|
o.boolean "--log-explain", "log explain", default: false, help: false
|
@@ -63,7 +66,7 @@ Options:)
|
|
63
66
|
|
64
67
|
# TODO don't use global var
|
65
68
|
$log_level = options[:log_level].to_s.downcase
|
66
|
-
abort "Unknown log level" unless ["error", "info", "debug", "debug2"].include?($log_level)
|
69
|
+
abort "Unknown log level" unless ["error", "info", "debug", "debug2", "debug3"].include?($log_level)
|
67
70
|
|
68
71
|
[arguments, options]
|
69
72
|
rescue Slop::Error => e
|
data/lib/dexter/collector.rb
CHANGED
@@ -5,6 +5,7 @@ module Dexter
|
|
5
5
|
@new_queries = Set.new
|
6
6
|
@mutex = Mutex.new
|
7
7
|
@min_time = options[:min_time] * 60000 # convert minutes to ms
|
8
|
+
@min_calls = options[:min_calls]
|
8
9
|
end
|
9
10
|
|
10
11
|
def add(query, duration)
|
@@ -36,7 +37,7 @@ module Dexter
|
|
36
37
|
|
37
38
|
queries = []
|
38
39
|
@top_queries.each do |k, v|
|
39
|
-
if new_queries.include?(k) && v[:total_time]
|
40
|
+
if new_queries.include?(k) && v[:total_time] >= @min_time && v[:calls] >= @min_calls
|
40
41
|
query = Query.new(v[:query], k)
|
41
42
|
query.total_time = v[:total_time]
|
42
43
|
query.calls = v[:calls]
|
data/lib/dexter/indexer.rb
CHANGED
@@ -10,6 +10,8 @@ module Dexter
|
|
10
10
|
@log_sql = options[:log_sql]
|
11
11
|
@log_explain = options[:log_explain]
|
12
12
|
@min_time = options[:min_time] || 0
|
13
|
+
@min_calls = options[:min_calls] || 0
|
14
|
+
@analyze = options[:analyze]
|
13
15
|
@options = options
|
14
16
|
|
15
17
|
create_extension unless extension_exists?
|
@@ -26,24 +28,39 @@ module Dexter
|
|
26
28
|
# reset hypothetical indexes
|
27
29
|
reset_hypothetical_indexes
|
28
30
|
|
29
|
-
|
30
|
-
tables = possible_tables(queries)
|
31
|
-
queries.each do |query|
|
32
|
-
query.missing_tables = !query.tables.all? { |t| tables.include?(t) }
|
33
|
-
end
|
31
|
+
tables = Set.new(database_tables)
|
34
32
|
|
35
33
|
if @include_tables
|
36
|
-
|
34
|
+
include_set = Set.new(@include_tables)
|
35
|
+
tables.keep_if { |t| include_set.include?(t) || include_set.include?(t.split(".")[-1]) }
|
36
|
+
end
|
37
|
+
|
38
|
+
if @exclude_tables.any?
|
39
|
+
exclude_set = Set.new(@exclude_tables)
|
40
|
+
tables.delete_if { |t| exclude_set.include?(t) || exclude_set.include?(t.split(".")[-1]) }
|
41
|
+
end
|
42
|
+
|
43
|
+
# map tables without schema to schema
|
44
|
+
no_schema_tables = {}
|
45
|
+
search_path_index = Hash[search_path.map.with_index.to_a]
|
46
|
+
tables.group_by { |t| t.split(".")[-1] }.each do |group, t2|
|
47
|
+
no_schema_tables[group] = t2.sort_by { |t| search_path_index[t.split(".")[0]] || 1000000 }[0]
|
37
48
|
end
|
38
49
|
|
39
|
-
#
|
40
|
-
|
41
|
-
|
42
|
-
tables.
|
50
|
+
# filter queries from other databases and system tables
|
51
|
+
queries.each do |query|
|
52
|
+
# add schema to table if needed
|
53
|
+
query.tables = query.tables.map { |t| no_schema_tables[t] || t }
|
54
|
+
|
55
|
+
# check for missing tables
|
56
|
+
query.missing_tables = !query.tables.all? { |t| tables.include?(t) }
|
43
57
|
end
|
44
58
|
|
59
|
+
# set tables
|
60
|
+
tables = Set.new(queries.reject(&:missing_tables).flat_map(&:tables))
|
61
|
+
|
45
62
|
# analyze tables if needed
|
46
|
-
analyze_tables(tables) if tables.any?
|
63
|
+
analyze_tables(tables) if tables.any? && (@analyze || @log_level == "debug2")
|
47
64
|
|
48
65
|
# create hypothetical indexes and explain queries
|
49
66
|
candidates = tables.any? ? create_hypothetical_indexes(queries.reject(&:missing_tables), tables) : {}
|
@@ -81,14 +98,13 @@ module Dexter
|
|
81
98
|
|
82
99
|
analyze_stats = execute <<-SQL
|
83
100
|
SELECT
|
84
|
-
schemaname AS
|
85
|
-
relname AS table,
|
101
|
+
schemaname || '.' || relname AS table,
|
86
102
|
last_analyze,
|
87
103
|
last_autoanalyze
|
88
104
|
FROM
|
89
105
|
pg_stat_user_tables
|
90
106
|
WHERE
|
91
|
-
relname IN (#{tables.map { |t| quote(t) }.join(", ")})
|
107
|
+
schemaname || '.' || relname IN (#{tables.map { |t| quote(t) }.join(", ")})
|
92
108
|
SQL
|
93
109
|
|
94
110
|
last_analyzed = {}
|
@@ -97,7 +113,14 @@ module Dexter
|
|
97
113
|
end
|
98
114
|
|
99
115
|
tables.each do |table|
|
100
|
-
|
116
|
+
la = last_analyzed[table]
|
117
|
+
|
118
|
+
if @log_level == "debug2"
|
119
|
+
time_str = la ? la.iso8601 : "Unknown"
|
120
|
+
log "Last analyze: #{table} : #{time_str}"
|
121
|
+
end
|
122
|
+
|
123
|
+
if @analyze && (!la || la < Time.now - 3600)
|
101
124
|
statement = "ANALYZE #{quote_ident(table)}"
|
102
125
|
log "Running analyze: #{statement}"
|
103
126
|
execute(statement)
|
@@ -137,6 +160,7 @@ module Dexter
|
|
137
160
|
# try to parse out columns
|
138
161
|
possible_columns = Set.new
|
139
162
|
explainable_queries.each do |query|
|
163
|
+
log "Finding columns: #{query.statement}" if @log_level == "debug3"
|
140
164
|
find_columns(query.tree).each do |col|
|
141
165
|
last_col = col["fields"].last
|
142
166
|
if last_col["String"]
|
@@ -296,72 +320,75 @@ module Dexter
|
|
296
320
|
end
|
297
321
|
|
298
322
|
def show_and_create_indexes(new_indexes, queries, tables)
|
323
|
+
# print summary
|
299
324
|
if new_indexes.any?
|
300
325
|
new_indexes.each do |index|
|
301
326
|
log "Index found: #{index[:table]} (#{index[:columns].join(", ")})"
|
302
327
|
end
|
328
|
+
else
|
329
|
+
log "No new indexes found"
|
330
|
+
end
|
303
331
|
|
304
|
-
|
305
|
-
|
306
|
-
|
307
|
-
|
308
|
-
|
309
|
-
|
310
|
-
|
311
|
-
|
312
|
-
|
313
|
-
|
314
|
-
|
315
|
-
|
316
|
-
|
317
|
-
|
318
|
-
|
319
|
-
|
320
|
-
|
321
|
-
|
322
|
-
|
323
|
-
|
324
|
-
|
325
|
-
|
326
|
-
|
327
|
-
elsif query.fingerprint == "unknown"
|
328
|
-
log "Could not parse query"
|
329
|
-
elsif query.tables.empty?
|
330
|
-
log "No tables"
|
331
|
-
elsif query.missing_tables
|
332
|
-
log "Tables not present in current database"
|
333
|
-
else
|
334
|
-
log "Could not run explain"
|
332
|
+
# debug info
|
333
|
+
if @log_level.start_with?("debug")
|
334
|
+
index_queries = new_indexes.flat_map { |i| i[:queries].sort_by(&:fingerprint) }
|
335
|
+
if @log_level == "debug2"
|
336
|
+
fingerprints = Set.new(index_queries.map(&:fingerprint))
|
337
|
+
index_queries.concat(queries.reject { |q| fingerprints.include?(q.fingerprint) }.sort_by(&:fingerprint))
|
338
|
+
end
|
339
|
+
index_queries.each do |query|
|
340
|
+
log "-" * 80
|
341
|
+
log "Query #{query.fingerprint}"
|
342
|
+
log "Total time: #{(query.total_time / 60000.0).round(1)} min, avg time: #{(query.total_time / query.calls.to_f).round} ms, calls: #{query.calls}" if query.total_time
|
343
|
+
if tables.empty?
|
344
|
+
log "No candidate tables for indexes"
|
345
|
+
elsif query.explainable? && !query.high_cost?
|
346
|
+
log "Low initial cost: #{query.initial_cost}"
|
347
|
+
elsif query.explainable?
|
348
|
+
query_indexes = query.indexes || []
|
349
|
+
log "Start: #{query.costs[0]}"
|
350
|
+
log "Pass1: #{query.costs[1]} : #{log_indexes(query.pass1_indexes || [])}"
|
351
|
+
log "Pass2: #{query.costs[2]} : #{log_indexes(query.pass2_indexes || [])}"
|
352
|
+
log "Final: #{query.new_cost} : #{log_indexes(query.suggest_index ? query_indexes : [])}"
|
353
|
+
if query_indexes.any? && !query.suggest_index
|
354
|
+
log "Need 50% cost savings to suggest index"
|
335
355
|
end
|
336
|
-
|
337
|
-
log query
|
338
|
-
|
356
|
+
elsif query.fingerprint == "unknown"
|
357
|
+
log "Could not parse query"
|
358
|
+
elsif query.tables.empty?
|
359
|
+
log "No tables"
|
360
|
+
elsif query.missing_tables
|
361
|
+
log "Tables not present in current database"
|
362
|
+
else
|
363
|
+
log "Could not run explain"
|
339
364
|
end
|
365
|
+
log
|
366
|
+
log query.statement
|
367
|
+
log
|
340
368
|
end
|
369
|
+
end
|
341
370
|
|
342
|
-
|
343
|
-
|
344
|
-
|
345
|
-
|
346
|
-
|
347
|
-
|
348
|
-
|
349
|
-
|
350
|
-
|
351
|
-
|
352
|
-
|
353
|
-
|
354
|
-
|
355
|
-
|
356
|
-
|
357
|
-
|
358
|
-
|
371
|
+
# create
|
372
|
+
if @create && new_indexes.any?
|
373
|
+
# 1. create lock
|
374
|
+
# 2. refresh existing index list
|
375
|
+
# 3. create indexes that still don't exist
|
376
|
+
# 4. release lock
|
377
|
+
with_advisory_lock do
|
378
|
+
new_indexes.each do |index|
|
379
|
+
unless index_exists?(index)
|
380
|
+
statement = "CREATE INDEX CONCURRENTLY ON #{quote_ident(index[:table])} (#{index[:columns].map { |c| quote_ident(c) }.join(", ")})"
|
381
|
+
log "Creating index: #{statement}"
|
382
|
+
started_at = Time.now
|
383
|
+
begin
|
384
|
+
execute(statement)
|
385
|
+
log "Index created: #{((Time.now - started_at) * 1000).to_i} ms"
|
386
|
+
rescue PG::LockNotAvailable
|
387
|
+
log "Could not acquire lock: #{index[:table]}"
|
359
388
|
end
|
360
389
|
end
|
361
390
|
end
|
362
391
|
end
|
363
|
-
else
|
364
|
-
log "No new indexes found"
|
365
392
|
end
|
366
393
|
|
367
394
|
new_indexes
|
@@ -417,7 +444,7 @@ module Dexter
|
|
417
444
|
def database_tables
|
418
445
|
result = execute <<-SQL
|
419
446
|
SELECT
|
420
|
-
table_name
|
447
|
+
table_schema || '.' || table_name AS table_name
|
421
448
|
FROM
|
422
449
|
information_schema.tables
|
423
450
|
WHERE
|
@@ -439,16 +466,13 @@ module Dexter
|
|
439
466
|
WHERE
|
440
467
|
datname = current_database()
|
441
468
|
AND total_time >= #{@min_time * 60000}
|
469
|
+
AND calls >= #{@min_calls}
|
442
470
|
ORDER BY
|
443
471
|
1
|
444
472
|
SQL
|
445
473
|
result.map { |q| q["query"] }
|
446
474
|
end
|
447
475
|
|
448
|
-
def possible_tables(queries)
|
449
|
-
Set.new(queries.flat_map(&:tables).uniq & database_tables)
|
450
|
-
end
|
451
|
-
|
452
476
|
def with_advisory_lock
|
453
477
|
lock_id = 123456
|
454
478
|
first_time = true
|
@@ -480,14 +504,13 @@ module Dexter
|
|
480
504
|
def columns(tables)
|
481
505
|
columns = execute <<-SQL
|
482
506
|
SELECT
|
483
|
-
table_name,
|
507
|
+
table_schema || '.' || table_name AS table_name,
|
484
508
|
column_name,
|
485
509
|
data_type
|
486
510
|
FROM
|
487
511
|
information_schema.columns
|
488
512
|
WHERE
|
489
|
-
table_schema
|
490
|
-
table_name IN (#{tables.map { |t| quote(t) }.join(", ")})
|
513
|
+
table_schema || '.' || table_name IN (#{tables.map { |t| quote(t) }.join(", ")})
|
491
514
|
ORDER BY
|
492
515
|
1, 2
|
493
516
|
SQL
|
@@ -498,8 +521,7 @@ module Dexter
|
|
498
521
|
def indexes(tables)
|
499
522
|
execute(<<-SQL
|
500
523
|
SELECT
|
501
|
-
schemaname AS
|
502
|
-
t.relname AS table,
|
524
|
+
schemaname || '.' || t.relname AS table,
|
503
525
|
ix.relname AS name,
|
504
526
|
regexp_replace(pg_get_indexdef(i.indexrelid), '^[^\\(]*\\((.*)\\)$', '\\1') AS columns,
|
505
527
|
regexp_replace(pg_get_indexdef(i.indexrelid), '.* USING ([^ ]*) \\(.*', '\\1') AS using
|
@@ -512,8 +534,7 @@ module Dexter
|
|
512
534
|
LEFT JOIN
|
513
535
|
pg_stat_user_indexes ui ON ui.indexrelid = i.indexrelid
|
514
536
|
WHERE
|
515
|
-
t.relname IN (#{tables.map { |t| quote(t) }.join(", ")}) AND
|
516
|
-
schemaname IS NOT NULL AND
|
537
|
+
schemaname || '.' || t.relname IN (#{tables.map { |t| quote(t) }.join(", ")}) AND
|
517
538
|
indisvalid = 't' AND
|
518
539
|
indexprs IS NULL AND
|
519
540
|
indpred IS NULL
|
@@ -523,8 +544,12 @@ module Dexter
|
|
523
544
|
).map { |v| v["columns"] = v["columns"].sub(") WHERE (", " WHERE ").split(", ").map { |c| unquote(c) }; v }
|
524
545
|
end
|
525
546
|
|
547
|
+
def search_path
|
548
|
+
execute("SHOW search_path")[0]["search_path"].split(",").map(&:strip)
|
549
|
+
end
|
550
|
+
|
526
551
|
def unquote(part)
|
527
|
-
if part && part.start_with?('"')
|
552
|
+
if part && part.start_with?('"') && part.end_with?('"')
|
528
553
|
part[1..-2]
|
529
554
|
else
|
530
555
|
part
|
@@ -532,7 +557,7 @@ module Dexter
|
|
532
557
|
end
|
533
558
|
|
534
559
|
def quote_ident(value)
|
535
|
-
conn.quote_ident(
|
560
|
+
value.split(".").map { |v| conn.quote_ident(v) }.join(".")
|
536
561
|
end
|
537
562
|
|
538
563
|
def quote(value)
|
data/lib/dexter/log_parser.rb
CHANGED
@@ -1,6 +1,6 @@
|
|
1
1
|
module Dexter
|
2
2
|
class LogParser
|
3
|
-
REGEX = /duration: (\d+\.\d+) ms (statement|execute <unnamed>): (.+)/
|
3
|
+
REGEX = /duration: (\d+\.\d+) ms (statement|execute <unnamed>|parse <unnamed>): (.+)/
|
4
4
|
LINE_SEPERATOR = ": ".freeze
|
5
5
|
|
6
6
|
def initialize(logfile, collector)
|
@@ -22,7 +22,7 @@ module Dexter
|
|
22
22
|
end
|
23
23
|
end
|
24
24
|
|
25
|
-
if !active_line && m = REGEX.match(line.chomp)
|
25
|
+
if !active_line && (m = REGEX.match(line.chomp))
|
26
26
|
duration = m[1].to_f
|
27
27
|
active_line = m[3]
|
28
28
|
end
|
data/lib/dexter/processor.rb
CHANGED
@@ -5,8 +5,14 @@ module Dexter
|
|
5
5
|
def initialize(logfile, options)
|
6
6
|
@logfile = logfile
|
7
7
|
|
8
|
-
@collector = Collector.new(min_time: options[:min_time])
|
9
|
-
@log_parser =
|
8
|
+
@collector = Collector.new(min_time: options[:min_time], min_calls: options[:min_calls])
|
9
|
+
@log_parser =
|
10
|
+
if options[:input_format] == "csv"
|
11
|
+
CsvLogParser.new(logfile, @collector)
|
12
|
+
else
|
13
|
+
LogParser.new(logfile, @collector)
|
14
|
+
end
|
15
|
+
|
10
16
|
@indexer = Indexer.new(options)
|
11
17
|
|
12
18
|
@starting_interval = 3
|
data/lib/dexter/query.rb
CHANGED
@@ -1,6 +1,7 @@
|
|
1
1
|
module Dexter
|
2
2
|
class Query
|
3
3
|
attr_reader :statement, :fingerprint, :plans
|
4
|
+
attr_writer :tables
|
4
5
|
attr_accessor :missing_tables, :new_cost, :total_time, :calls, :indexes, :suggest_index, :pass1_indexes, :pass2_indexes
|
5
6
|
|
6
7
|
def initialize(statement, fingerprint = nil)
|
@@ -13,7 +14,15 @@ module Dexter
|
|
13
14
|
end
|
14
15
|
|
15
16
|
def tables
|
16
|
-
@tables ||=
|
17
|
+
@tables ||= begin
|
18
|
+
parse ? parse.tables : []
|
19
|
+
rescue => e
|
20
|
+
# possible pg_query bug
|
21
|
+
$stderr.puts "Error extracting tables. Please report to https://github.com/ankane/dexter/issues"
|
22
|
+
$stderr.puts "#{e.class.name}: #{e.message}"
|
23
|
+
$stderr.puts statement
|
24
|
+
[]
|
25
|
+
end
|
17
26
|
end
|
18
27
|
|
19
28
|
def tree
|
data/lib/dexter/version.rb
CHANGED
data/lib/dexter.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: pgdexter
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.3.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Andrew Kane
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2017-
|
11
|
+
date: 2017-12-23 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: slop
|
@@ -115,6 +115,7 @@ files:
|
|
115
115
|
- lib/dexter.rb
|
116
116
|
- lib/dexter/client.rb
|
117
117
|
- lib/dexter/collector.rb
|
118
|
+
- lib/dexter/csv_log_parser.rb
|
118
119
|
- lib/dexter/indexer.rb
|
119
120
|
- lib/dexter/log_parser.rb
|
120
121
|
- lib/dexter/logging.rb
|
@@ -141,7 +142,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
141
142
|
version: '0'
|
142
143
|
requirements: []
|
143
144
|
rubyforge_project:
|
144
|
-
rubygems_version: 2.6.
|
145
|
+
rubygems_version: 2.6.13
|
145
146
|
signing_key:
|
146
147
|
specification_version: 4
|
147
148
|
summary: The automatic indexer for Postgres
|