pgdexter 0.2.1 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +8 -0
- data/README.md +98 -25
- data/guides/Hosted-Postgres.md +1 -1
- data/lib/dexter/client.rb +4 -1
- data/lib/dexter/collector.rb +2 -1
- data/lib/dexter/csv_log_parser.rb +13 -0
- data/lib/dexter/indexer.rb +107 -82
- data/lib/dexter/log_parser.rb +2 -2
- data/lib/dexter/processor.rb +8 -2
- data/lib/dexter/query.rb +10 -1
- data/lib/dexter/version.rb +1 -1
- data/lib/dexter.rb +1 -0
- metadata +4 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 16f61626f5003a801ec12cda3f663390a57e69d8
|
4
|
+
data.tar.gz: 238ee1b24dab3f473accc2ca9a5156fd31e8d19e
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: ccae6266196fc84ea0fc69177b80144f8747996ff0693b6a7492e34d095b2cb400e9ce3299741326c50a5eb267ef48be368384a2ab27553002eb93ac7b6d0922
|
7
|
+
data.tar.gz: 944bc27bb1ffff546c2f33aa10aa1b8371ba43ae2be889a6b9445c3d3a93ca36fee0cfd57487134a5ea807e37583f74b1e5e9de7714fa30364ab28f64a5679e8
|
data/CHANGELOG.md
CHANGED
data/README.md
CHANGED
@@ -12,8 +12,8 @@ First, install [HypoPG](https://github.com/dalibo/hypopg) on your database serve
|
|
12
12
|
|
13
13
|
```sh
|
14
14
|
cd /tmp
|
15
|
-
curl -L https://github.com/dalibo/hypopg/archive/1.
|
16
|
-
cd hypopg-1.
|
15
|
+
curl -L https://github.com/dalibo/hypopg/archive/1.1.0.tar.gz | tar xz
|
16
|
+
cd hypopg-1.1.0
|
17
17
|
make
|
18
18
|
make install # may need sudo
|
19
19
|
```
|
@@ -45,23 +45,23 @@ tail -F -n +1 <log-file> | dexter <connection-options>
|
|
45
45
|
This finds slow queries and generates output like:
|
46
46
|
|
47
47
|
```
|
48
|
-
|
49
|
-
|
50
|
-
|
51
|
-
|
52
|
-
|
53
|
-
|
54
|
-
|
55
|
-
|
56
|
-
|
48
|
+
Started
|
49
|
+
Processing 189 new query fingerprints
|
50
|
+
Index found: public.genres_movies (genre_id)
|
51
|
+
Index found: public.genres_movies (movie_id)
|
52
|
+
Index found: public.movies (title)
|
53
|
+
Index found: public.ratings (movie_id)
|
54
|
+
Index found: public.ratings (rating)
|
55
|
+
Index found: public.ratings (user_id)
|
56
|
+
Processing 12 new query fingerprints
|
57
57
|
```
|
58
58
|
|
59
59
|
To be safe, Dexter will not create indexes unless you pass the `--create` flag. In this case, you’ll see:
|
60
60
|
|
61
61
|
```
|
62
|
-
|
63
|
-
|
64
|
-
|
62
|
+
Index found: public.ratings (user_id)
|
63
|
+
Creating index: CREATE INDEX CONCURRENTLY ON "public"."ratings" ("user_id")
|
64
|
+
Index created: 15243 ms
|
65
65
|
```
|
66
66
|
|
67
67
|
## Connection Options
|
@@ -84,30 +84,58 @@ and connection strings:
|
|
84
84
|
host=localhost port=5432 dbname=mydb
|
85
85
|
```
|
86
86
|
|
87
|
-
##
|
87
|
+
## Collecting Queries
|
88
88
|
|
89
|
-
|
90
|
-
--- | --- | ---
|
91
|
-
exclude | prevent specific tables from being indexed | None
|
92
|
-
interval | time to wait between processing queries, in seconds | 60
|
93
|
-
log-level | `debug` gives additional info for suggested indexes<br />`debug2` gives additional info for processed queries<br />`error` suppresses logging | info
|
94
|
-
log-sql | log SQL statements executed | false
|
95
|
-
min-time | only process queries consuming a min amount of DB time, in minutes | 0
|
89
|
+
There are many ways to collect queries. For real-time indexing, pipe your logfile:
|
96
90
|
|
97
|
-
|
91
|
+
```sh
|
92
|
+
tail -F -n +1 <log-file> | dexter <connection-options>
|
93
|
+
```
|
98
94
|
|
99
|
-
|
95
|
+
Pass a single statement with:
|
100
96
|
|
101
97
|
```sh
|
102
98
|
dexter <connection-options> -s "SELECT * FROM ..."
|
103
99
|
```
|
104
100
|
|
105
|
-
or files
|
101
|
+
or pass files:
|
106
102
|
|
107
103
|
```sh
|
108
104
|
dexter <connection-options> <file1> <file2>
|
109
105
|
```
|
110
106
|
|
107
|
+
or use the [pg_stat_statements](https://www.postgresql.org/docs/current/static/pgstatstatements.html) extension:
|
108
|
+
|
109
|
+
```sh
|
110
|
+
dexter <connection-options> --pg-stat-statements
|
111
|
+
```
|
112
|
+
|
113
|
+
### Collection Options
|
114
|
+
|
115
|
+
To prevent one-off queries from being indexed, specify a minimum number of calls before a query is considered for indexing
|
116
|
+
|
117
|
+
```sh
|
118
|
+
dexter --min-calls 100
|
119
|
+
```
|
120
|
+
|
121
|
+
You can do the same for total time a query has run
|
122
|
+
|
123
|
+
```sh
|
124
|
+
dexter --min-time 10 # minutes
|
125
|
+
```
|
126
|
+
|
127
|
+
Specify the format
|
128
|
+
|
129
|
+
```sh
|
130
|
+
dexter --input-format csv
|
131
|
+
```
|
132
|
+
|
133
|
+
When steaming logs, specify the time to wait between processing queries
|
134
|
+
|
135
|
+
```sh
|
136
|
+
dexter --interval 60 # seconds
|
137
|
+
```
|
138
|
+
|
111
139
|
## Examples
|
112
140
|
|
113
141
|
Ubuntu with PostgreSQL 9.6
|
@@ -122,6 +150,36 @@ Homebrew on Mac
|
|
122
150
|
tail -F -n +1 /usr/local/var/postgres/server.log | dexter dbname
|
123
151
|
```
|
124
152
|
|
153
|
+
## Tables
|
154
|
+
|
155
|
+
You can exclude large or write-heavy tables from indexing with:
|
156
|
+
|
157
|
+
```sh
|
158
|
+
dexter --exclude table1,table2
|
159
|
+
```
|
160
|
+
|
161
|
+
Alternatively, you can specify which tables to index with:
|
162
|
+
|
163
|
+
```sh
|
164
|
+
dexter --include table3,table4
|
165
|
+
```
|
166
|
+
|
167
|
+
## Debugging
|
168
|
+
|
169
|
+
See how Dexter is processing queries with:
|
170
|
+
|
171
|
+
```sh
|
172
|
+
dexter --log-sql --log-level debug2
|
173
|
+
```
|
174
|
+
|
175
|
+
## Analyze
|
176
|
+
|
177
|
+
For best results, make sure your tables have been recently analyzed so statistics are up-to-date. You can ask Dexter to analyze tables it comes across that haven’t been analyzed in the past hour with:
|
178
|
+
|
179
|
+
```sh
|
180
|
+
dexter --analyze
|
181
|
+
```
|
182
|
+
|
125
183
|
## Hosted Postgres
|
126
184
|
|
127
185
|
Some hosted providers like Amazon RDS and Heroku do not support the HypoPG extension, which Dexter needs to run. See [how to use Dexter](guides/Hosted-Postgres.md) in these cases.
|
@@ -130,6 +188,21 @@ Some hosted providers like Amazon RDS and Heroku do not support the HypoPG exten
|
|
130
188
|
|
131
189
|
[Here are some ideas](https://github.com/ankane/dexter/issues/1)
|
132
190
|
|
191
|
+
## Upgrading
|
192
|
+
|
193
|
+
Run:
|
194
|
+
|
195
|
+
```sh
|
196
|
+
gem install pgdexter
|
197
|
+
```
|
198
|
+
|
199
|
+
To use master, run:
|
200
|
+
|
201
|
+
```sh
|
202
|
+
gem install specific_install
|
203
|
+
gem specific_install https://github.com/ankane/dexter.git
|
204
|
+
```
|
205
|
+
|
133
206
|
## Thanks
|
134
207
|
|
135
208
|
This software wouldn’t be possible without [HypoPG](https://github.com/dalibo/hypopg), which allows you to create hypothetical indexes, and [pg_query](https://github.com/lfittl/pg_query), which allows you to parse and fingerprint queries. A big thanks to Dalibo and Lukas Fittl respectively.
|
data/guides/Hosted-Postgres.md
CHANGED
data/lib/dexter/client.rb
CHANGED
@@ -29,10 +29,13 @@ module Dexter
|
|
29
29
|
dexter [options]
|
30
30
|
|
31
31
|
Options:)
|
32
|
+
o.boolean "--analyze", "analyze tables that haven't been analyzed in the past hour", default: false
|
32
33
|
o.boolean "--create", "create indexes", default: false
|
33
34
|
o.array "--exclude", "prevent specific tables from being indexed"
|
34
35
|
o.string "--include", "only include specific tables"
|
36
|
+
o.string "--input-format", "input format", default: "stderr"
|
35
37
|
o.integer "--interval", "time to wait between processing queries, in seconds", default: 60
|
38
|
+
o.float "--min-calls", "only process queries that have been called a certain number of times", default: 0
|
36
39
|
o.float "--min-time", "only process queries that have consumed a certain amount of DB time, in minutes", default: 0
|
37
40
|
o.boolean "--pg-stat-statements", "use pg_stat_statements", default: false, help: false
|
38
41
|
o.boolean "--log-explain", "log explain", default: false, help: false
|
@@ -63,7 +66,7 @@ Options:)
|
|
63
66
|
|
64
67
|
# TODO don't use global var
|
65
68
|
$log_level = options[:log_level].to_s.downcase
|
66
|
-
abort "Unknown log level" unless ["error", "info", "debug", "debug2"].include?($log_level)
|
69
|
+
abort "Unknown log level" unless ["error", "info", "debug", "debug2", "debug3"].include?($log_level)
|
67
70
|
|
68
71
|
[arguments, options]
|
69
72
|
rescue Slop::Error => e
|
data/lib/dexter/collector.rb
CHANGED
@@ -5,6 +5,7 @@ module Dexter
|
|
5
5
|
@new_queries = Set.new
|
6
6
|
@mutex = Mutex.new
|
7
7
|
@min_time = options[:min_time] * 60000 # convert minutes to ms
|
8
|
+
@min_calls = options[:min_calls]
|
8
9
|
end
|
9
10
|
|
10
11
|
def add(query, duration)
|
@@ -36,7 +37,7 @@ module Dexter
|
|
36
37
|
|
37
38
|
queries = []
|
38
39
|
@top_queries.each do |k, v|
|
39
|
-
if new_queries.include?(k) && v[:total_time]
|
40
|
+
if new_queries.include?(k) && v[:total_time] >= @min_time && v[:calls] >= @min_calls
|
40
41
|
query = Query.new(v[:query], k)
|
41
42
|
query.total_time = v[:total_time]
|
42
43
|
query.calls = v[:calls]
|
data/lib/dexter/indexer.rb
CHANGED
@@ -10,6 +10,8 @@ module Dexter
|
|
10
10
|
@log_sql = options[:log_sql]
|
11
11
|
@log_explain = options[:log_explain]
|
12
12
|
@min_time = options[:min_time] || 0
|
13
|
+
@min_calls = options[:min_calls] || 0
|
14
|
+
@analyze = options[:analyze]
|
13
15
|
@options = options
|
14
16
|
|
15
17
|
create_extension unless extension_exists?
|
@@ -26,24 +28,39 @@ module Dexter
|
|
26
28
|
# reset hypothetical indexes
|
27
29
|
reset_hypothetical_indexes
|
28
30
|
|
29
|
-
|
30
|
-
tables = possible_tables(queries)
|
31
|
-
queries.each do |query|
|
32
|
-
query.missing_tables = !query.tables.all? { |t| tables.include?(t) }
|
33
|
-
end
|
31
|
+
tables = Set.new(database_tables)
|
34
32
|
|
35
33
|
if @include_tables
|
36
|
-
|
34
|
+
include_set = Set.new(@include_tables)
|
35
|
+
tables.keep_if { |t| include_set.include?(t) || include_set.include?(t.split(".")[-1]) }
|
36
|
+
end
|
37
|
+
|
38
|
+
if @exclude_tables.any?
|
39
|
+
exclude_set = Set.new(@exclude_tables)
|
40
|
+
tables.delete_if { |t| exclude_set.include?(t) || exclude_set.include?(t.split(".")[-1]) }
|
41
|
+
end
|
42
|
+
|
43
|
+
# map tables without schema to schema
|
44
|
+
no_schema_tables = {}
|
45
|
+
search_path_index = Hash[search_path.map.with_index.to_a]
|
46
|
+
tables.group_by { |t| t.split(".")[-1] }.each do |group, t2|
|
47
|
+
no_schema_tables[group] = t2.sort_by { |t| search_path_index[t.split(".")[0]] || 1000000 }[0]
|
37
48
|
end
|
38
49
|
|
39
|
-
#
|
40
|
-
|
41
|
-
|
42
|
-
tables.
|
50
|
+
# filter queries from other databases and system tables
|
51
|
+
queries.each do |query|
|
52
|
+
# add schema to table if needed
|
53
|
+
query.tables = query.tables.map { |t| no_schema_tables[t] || t }
|
54
|
+
|
55
|
+
# check for missing tables
|
56
|
+
query.missing_tables = !query.tables.all? { |t| tables.include?(t) }
|
43
57
|
end
|
44
58
|
|
59
|
+
# set tables
|
60
|
+
tables = Set.new(queries.reject(&:missing_tables).flat_map(&:tables))
|
61
|
+
|
45
62
|
# analyze tables if needed
|
46
|
-
analyze_tables(tables) if tables.any?
|
63
|
+
analyze_tables(tables) if tables.any? && (@analyze || @log_level == "debug2")
|
47
64
|
|
48
65
|
# create hypothetical indexes and explain queries
|
49
66
|
candidates = tables.any? ? create_hypothetical_indexes(queries.reject(&:missing_tables), tables) : {}
|
@@ -81,14 +98,13 @@ module Dexter
|
|
81
98
|
|
82
99
|
analyze_stats = execute <<-SQL
|
83
100
|
SELECT
|
84
|
-
schemaname AS
|
85
|
-
relname AS table,
|
101
|
+
schemaname || '.' || relname AS table,
|
86
102
|
last_analyze,
|
87
103
|
last_autoanalyze
|
88
104
|
FROM
|
89
105
|
pg_stat_user_tables
|
90
106
|
WHERE
|
91
|
-
relname IN (#{tables.map { |t| quote(t) }.join(", ")})
|
107
|
+
schemaname || '.' || relname IN (#{tables.map { |t| quote(t) }.join(", ")})
|
92
108
|
SQL
|
93
109
|
|
94
110
|
last_analyzed = {}
|
@@ -97,7 +113,14 @@ module Dexter
|
|
97
113
|
end
|
98
114
|
|
99
115
|
tables.each do |table|
|
100
|
-
|
116
|
+
la = last_analyzed[table]
|
117
|
+
|
118
|
+
if @log_level == "debug2"
|
119
|
+
time_str = la ? la.iso8601 : "Unknown"
|
120
|
+
log "Last analyze: #{table} : #{time_str}"
|
121
|
+
end
|
122
|
+
|
123
|
+
if @analyze && (!la || la < Time.now - 3600)
|
101
124
|
statement = "ANALYZE #{quote_ident(table)}"
|
102
125
|
log "Running analyze: #{statement}"
|
103
126
|
execute(statement)
|
@@ -137,6 +160,7 @@ module Dexter
|
|
137
160
|
# try to parse out columns
|
138
161
|
possible_columns = Set.new
|
139
162
|
explainable_queries.each do |query|
|
163
|
+
log "Finding columns: #{query.statement}" if @log_level == "debug3"
|
140
164
|
find_columns(query.tree).each do |col|
|
141
165
|
last_col = col["fields"].last
|
142
166
|
if last_col["String"]
|
@@ -296,72 +320,75 @@ module Dexter
|
|
296
320
|
end
|
297
321
|
|
298
322
|
def show_and_create_indexes(new_indexes, queries, tables)
|
323
|
+
# print summary
|
299
324
|
if new_indexes.any?
|
300
325
|
new_indexes.each do |index|
|
301
326
|
log "Index found: #{index[:table]} (#{index[:columns].join(", ")})"
|
302
327
|
end
|
328
|
+
else
|
329
|
+
log "No new indexes found"
|
330
|
+
end
|
303
331
|
|
304
|
-
|
305
|
-
|
306
|
-
|
307
|
-
|
308
|
-
|
309
|
-
|
310
|
-
|
311
|
-
|
312
|
-
|
313
|
-
|
314
|
-
|
315
|
-
|
316
|
-
|
317
|
-
|
318
|
-
|
319
|
-
|
320
|
-
|
321
|
-
|
322
|
-
|
323
|
-
|
324
|
-
|
325
|
-
|
326
|
-
|
327
|
-
elsif query.fingerprint == "unknown"
|
328
|
-
log "Could not parse query"
|
329
|
-
elsif query.tables.empty?
|
330
|
-
log "No tables"
|
331
|
-
elsif query.missing_tables
|
332
|
-
log "Tables not present in current database"
|
333
|
-
else
|
334
|
-
log "Could not run explain"
|
332
|
+
# debug info
|
333
|
+
if @log_level.start_with?("debug")
|
334
|
+
index_queries = new_indexes.flat_map { |i| i[:queries].sort_by(&:fingerprint) }
|
335
|
+
if @log_level == "debug2"
|
336
|
+
fingerprints = Set.new(index_queries.map(&:fingerprint))
|
337
|
+
index_queries.concat(queries.reject { |q| fingerprints.include?(q.fingerprint) }.sort_by(&:fingerprint))
|
338
|
+
end
|
339
|
+
index_queries.each do |query|
|
340
|
+
log "-" * 80
|
341
|
+
log "Query #{query.fingerprint}"
|
342
|
+
log "Total time: #{(query.total_time / 60000.0).round(1)} min, avg time: #{(query.total_time / query.calls.to_f).round} ms, calls: #{query.calls}" if query.total_time
|
343
|
+
if tables.empty?
|
344
|
+
log "No candidate tables for indexes"
|
345
|
+
elsif query.explainable? && !query.high_cost?
|
346
|
+
log "Low initial cost: #{query.initial_cost}"
|
347
|
+
elsif query.explainable?
|
348
|
+
query_indexes = query.indexes || []
|
349
|
+
log "Start: #{query.costs[0]}"
|
350
|
+
log "Pass1: #{query.costs[1]} : #{log_indexes(query.pass1_indexes || [])}"
|
351
|
+
log "Pass2: #{query.costs[2]} : #{log_indexes(query.pass2_indexes || [])}"
|
352
|
+
log "Final: #{query.new_cost} : #{log_indexes(query.suggest_index ? query_indexes : [])}"
|
353
|
+
if query_indexes.any? && !query.suggest_index
|
354
|
+
log "Need 50% cost savings to suggest index"
|
335
355
|
end
|
336
|
-
|
337
|
-
log query
|
338
|
-
|
356
|
+
elsif query.fingerprint == "unknown"
|
357
|
+
log "Could not parse query"
|
358
|
+
elsif query.tables.empty?
|
359
|
+
log "No tables"
|
360
|
+
elsif query.missing_tables
|
361
|
+
log "Tables not present in current database"
|
362
|
+
else
|
363
|
+
log "Could not run explain"
|
339
364
|
end
|
365
|
+
log
|
366
|
+
log query.statement
|
367
|
+
log
|
340
368
|
end
|
369
|
+
end
|
341
370
|
|
342
|
-
|
343
|
-
|
344
|
-
|
345
|
-
|
346
|
-
|
347
|
-
|
348
|
-
|
349
|
-
|
350
|
-
|
351
|
-
|
352
|
-
|
353
|
-
|
354
|
-
|
355
|
-
|
356
|
-
|
357
|
-
|
358
|
-
|
371
|
+
# create
|
372
|
+
if @create && new_indexes.any?
|
373
|
+
# 1. create lock
|
374
|
+
# 2. refresh existing index list
|
375
|
+
# 3. create indexes that still don't exist
|
376
|
+
# 4. release lock
|
377
|
+
with_advisory_lock do
|
378
|
+
new_indexes.each do |index|
|
379
|
+
unless index_exists?(index)
|
380
|
+
statement = "CREATE INDEX CONCURRENTLY ON #{quote_ident(index[:table])} (#{index[:columns].map { |c| quote_ident(c) }.join(", ")})"
|
381
|
+
log "Creating index: #{statement}"
|
382
|
+
started_at = Time.now
|
383
|
+
begin
|
384
|
+
execute(statement)
|
385
|
+
log "Index created: #{((Time.now - started_at) * 1000).to_i} ms"
|
386
|
+
rescue PG::LockNotAvailable
|
387
|
+
log "Could not acquire lock: #{index[:table]}"
|
359
388
|
end
|
360
389
|
end
|
361
390
|
end
|
362
391
|
end
|
363
|
-
else
|
364
|
-
log "No new indexes found"
|
365
392
|
end
|
366
393
|
|
367
394
|
new_indexes
|
@@ -417,7 +444,7 @@ module Dexter
|
|
417
444
|
def database_tables
|
418
445
|
result = execute <<-SQL
|
419
446
|
SELECT
|
420
|
-
table_name
|
447
|
+
table_schema || '.' || table_name AS table_name
|
421
448
|
FROM
|
422
449
|
information_schema.tables
|
423
450
|
WHERE
|
@@ -439,16 +466,13 @@ module Dexter
|
|
439
466
|
WHERE
|
440
467
|
datname = current_database()
|
441
468
|
AND total_time >= #{@min_time * 60000}
|
469
|
+
AND calls >= #{@min_calls}
|
442
470
|
ORDER BY
|
443
471
|
1
|
444
472
|
SQL
|
445
473
|
result.map { |q| q["query"] }
|
446
474
|
end
|
447
475
|
|
448
|
-
def possible_tables(queries)
|
449
|
-
Set.new(queries.flat_map(&:tables).uniq & database_tables)
|
450
|
-
end
|
451
|
-
|
452
476
|
def with_advisory_lock
|
453
477
|
lock_id = 123456
|
454
478
|
first_time = true
|
@@ -480,14 +504,13 @@ module Dexter
|
|
480
504
|
def columns(tables)
|
481
505
|
columns = execute <<-SQL
|
482
506
|
SELECT
|
483
|
-
table_name,
|
507
|
+
table_schema || '.' || table_name AS table_name,
|
484
508
|
column_name,
|
485
509
|
data_type
|
486
510
|
FROM
|
487
511
|
information_schema.columns
|
488
512
|
WHERE
|
489
|
-
table_schema
|
490
|
-
table_name IN (#{tables.map { |t| quote(t) }.join(", ")})
|
513
|
+
table_schema || '.' || table_name IN (#{tables.map { |t| quote(t) }.join(", ")})
|
491
514
|
ORDER BY
|
492
515
|
1, 2
|
493
516
|
SQL
|
@@ -498,8 +521,7 @@ module Dexter
|
|
498
521
|
def indexes(tables)
|
499
522
|
execute(<<-SQL
|
500
523
|
SELECT
|
501
|
-
schemaname AS
|
502
|
-
t.relname AS table,
|
524
|
+
schemaname || '.' || t.relname AS table,
|
503
525
|
ix.relname AS name,
|
504
526
|
regexp_replace(pg_get_indexdef(i.indexrelid), '^[^\\(]*\\((.*)\\)$', '\\1') AS columns,
|
505
527
|
regexp_replace(pg_get_indexdef(i.indexrelid), '.* USING ([^ ]*) \\(.*', '\\1') AS using
|
@@ -512,8 +534,7 @@ module Dexter
|
|
512
534
|
LEFT JOIN
|
513
535
|
pg_stat_user_indexes ui ON ui.indexrelid = i.indexrelid
|
514
536
|
WHERE
|
515
|
-
t.relname IN (#{tables.map { |t| quote(t) }.join(", ")}) AND
|
516
|
-
schemaname IS NOT NULL AND
|
537
|
+
schemaname || '.' || t.relname IN (#{tables.map { |t| quote(t) }.join(", ")}) AND
|
517
538
|
indisvalid = 't' AND
|
518
539
|
indexprs IS NULL AND
|
519
540
|
indpred IS NULL
|
@@ -523,8 +544,12 @@ module Dexter
|
|
523
544
|
).map { |v| v["columns"] = v["columns"].sub(") WHERE (", " WHERE ").split(", ").map { |c| unquote(c) }; v }
|
524
545
|
end
|
525
546
|
|
547
|
+
def search_path
|
548
|
+
execute("SHOW search_path")[0]["search_path"].split(",").map(&:strip)
|
549
|
+
end
|
550
|
+
|
526
551
|
def unquote(part)
|
527
|
-
if part && part.start_with?('"')
|
552
|
+
if part && part.start_with?('"') && part.end_with?('"')
|
528
553
|
part[1..-2]
|
529
554
|
else
|
530
555
|
part
|
@@ -532,7 +557,7 @@ module Dexter
|
|
532
557
|
end
|
533
558
|
|
534
559
|
def quote_ident(value)
|
535
|
-
conn.quote_ident(
|
560
|
+
value.split(".").map { |v| conn.quote_ident(v) }.join(".")
|
536
561
|
end
|
537
562
|
|
538
563
|
def quote(value)
|
data/lib/dexter/log_parser.rb
CHANGED
@@ -1,6 +1,6 @@
|
|
1
1
|
module Dexter
|
2
2
|
class LogParser
|
3
|
-
REGEX = /duration: (\d+\.\d+) ms (statement|execute <unnamed>): (.+)/
|
3
|
+
REGEX = /duration: (\d+\.\d+) ms (statement|execute <unnamed>|parse <unnamed>): (.+)/
|
4
4
|
LINE_SEPERATOR = ": ".freeze
|
5
5
|
|
6
6
|
def initialize(logfile, collector)
|
@@ -22,7 +22,7 @@ module Dexter
|
|
22
22
|
end
|
23
23
|
end
|
24
24
|
|
25
|
-
if !active_line && m = REGEX.match(line.chomp)
|
25
|
+
if !active_line && (m = REGEX.match(line.chomp))
|
26
26
|
duration = m[1].to_f
|
27
27
|
active_line = m[3]
|
28
28
|
end
|
data/lib/dexter/processor.rb
CHANGED
@@ -5,8 +5,14 @@ module Dexter
|
|
5
5
|
def initialize(logfile, options)
|
6
6
|
@logfile = logfile
|
7
7
|
|
8
|
-
@collector = Collector.new(min_time: options[:min_time])
|
9
|
-
@log_parser =
|
8
|
+
@collector = Collector.new(min_time: options[:min_time], min_calls: options[:min_calls])
|
9
|
+
@log_parser =
|
10
|
+
if options[:input_format] == "csv"
|
11
|
+
CsvLogParser.new(logfile, @collector)
|
12
|
+
else
|
13
|
+
LogParser.new(logfile, @collector)
|
14
|
+
end
|
15
|
+
|
10
16
|
@indexer = Indexer.new(options)
|
11
17
|
|
12
18
|
@starting_interval = 3
|
data/lib/dexter/query.rb
CHANGED
@@ -1,6 +1,7 @@
|
|
1
1
|
module Dexter
|
2
2
|
class Query
|
3
3
|
attr_reader :statement, :fingerprint, :plans
|
4
|
+
attr_writer :tables
|
4
5
|
attr_accessor :missing_tables, :new_cost, :total_time, :calls, :indexes, :suggest_index, :pass1_indexes, :pass2_indexes
|
5
6
|
|
6
7
|
def initialize(statement, fingerprint = nil)
|
@@ -13,7 +14,15 @@ module Dexter
|
|
13
14
|
end
|
14
15
|
|
15
16
|
def tables
|
16
|
-
@tables ||=
|
17
|
+
@tables ||= begin
|
18
|
+
parse ? parse.tables : []
|
19
|
+
rescue => e
|
20
|
+
# possible pg_query bug
|
21
|
+
$stderr.puts "Error extracting tables. Please report to https://github.com/ankane/dexter/issues"
|
22
|
+
$stderr.puts "#{e.class.name}: #{e.message}"
|
23
|
+
$stderr.puts statement
|
24
|
+
[]
|
25
|
+
end
|
17
26
|
end
|
18
27
|
|
19
28
|
def tree
|
data/lib/dexter/version.rb
CHANGED
data/lib/dexter.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: pgdexter
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.3.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Andrew Kane
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2017-
|
11
|
+
date: 2017-12-23 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: slop
|
@@ -115,6 +115,7 @@ files:
|
|
115
115
|
- lib/dexter.rb
|
116
116
|
- lib/dexter/client.rb
|
117
117
|
- lib/dexter/collector.rb
|
118
|
+
- lib/dexter/csv_log_parser.rb
|
118
119
|
- lib/dexter/indexer.rb
|
119
120
|
- lib/dexter/log_parser.rb
|
120
121
|
- lib/dexter/logging.rb
|
@@ -141,7 +142,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
141
142
|
version: '0'
|
142
143
|
requirements: []
|
143
144
|
rubyforge_project:
|
144
|
-
rubygems_version: 2.6.
|
145
|
+
rubygems_version: 2.6.13
|
145
146
|
signing_key:
|
146
147
|
specification_version: 4
|
147
148
|
summary: The automatic indexer for Postgres
|