pgdexter 0.4.3 → 0.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +17 -0
- data/README.md +97 -40
- data/lib/dexter/client.rb +12 -8
- data/lib/dexter/collector.rb +3 -3
- data/lib/dexter/indexer.rb +100 -113
- data/lib/dexter/log_parser.rb +0 -4
- data/lib/dexter/pg_stat_activity_parser.rb +34 -11
- data/lib/dexter/processor.rb +1 -8
- data/lib/dexter/stderr_log_parser.rb +3 -0
- data/lib/dexter/version.rb +1 -1
- data/lib/dexter.rb +0 -2
- metadata +6 -8
- data/lib/dexter/csv_log_table_parser.rb +0 -20
- data/lib/dexter/stderr_log_table_parser.rb +0 -7
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: c68746f6134f6603c5549b561886b45ee3432df5d1e07e65a5907559901d333a
|
4
|
+
data.tar.gz: 699c2744f3e2c9a79f8fa9ca6ae7a77586421243cbf7a1f92642f76724cc0cb0
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 343bc52539ef09541fd0774667034ea02fb88b0e4f820e52c86d77df46d059c253576c22272b91a407b764965f66c554278154da6ab923249ce9979ab3a5aed0
|
7
|
+
data.tar.gz: a9b86931b4fc58d2b89c87f46f5008dfd47274cc39f096ec2b5143bea8858e29a786e387f9ccc29fe74475b5b0317e9d348d36f2a686cb19457ee79179a270f5
|
data/CHANGELOG.md
CHANGED
@@ -1,3 +1,20 @@
|
|
1
|
+
## 0.5.1 (2023-05-27)
|
2
|
+
|
3
|
+
- Fixed `JSON::NestingError`
|
4
|
+
|
5
|
+
## 0.5.0 (2023-04-18)
|
6
|
+
|
7
|
+
- Added support for normalized queries
|
8
|
+
- Added `--stdin` option (now required to read from stdin)
|
9
|
+
- Added `--enable-hypopg` option (now required to enable HypoPG)
|
10
|
+
- Improved output when HypoPG not installed
|
11
|
+
- Changed `--pg-stat-activity` to sample 10 times and exit
|
12
|
+
- Detect input format based on file extension
|
13
|
+
- Dropped support for experimental `--log-table` option
|
14
|
+
- Dropped support for Linux packages for Ubuntu 18.04 and Debian 10
|
15
|
+
- Dropped support for Ruby < 2.7
|
16
|
+
- Dropped support for Postgres < 11
|
17
|
+
|
1
18
|
## 0.4.3 (2023-03-26)
|
2
19
|
|
3
20
|
- Added experimental `--log-table` option
|
data/README.md
CHANGED
@@ -12,21 +12,21 @@ First, install [HypoPG](https://github.com/HypoPG/hypopg) on your database serve
|
|
12
12
|
|
13
13
|
```sh
|
14
14
|
cd /tmp
|
15
|
-
curl -L https://github.com/HypoPG/hypopg/archive/1.
|
16
|
-
cd hypopg-1.
|
15
|
+
curl -L https://github.com/HypoPG/hypopg/archive/1.4.0.tar.gz | tar xz
|
16
|
+
cd hypopg-1.4.0
|
17
17
|
make
|
18
18
|
make install # may need sudo
|
19
19
|
```
|
20
20
|
|
21
|
-
|
21
|
+
And enable it in databases where you want to use Dexter:
|
22
22
|
|
23
|
-
|
24
|
-
|
25
|
-
```ini
|
26
|
-
log_min_duration_statement = 10 # ms
|
23
|
+
```sql
|
24
|
+
CREATE EXTENSION hypopg;
|
27
25
|
```
|
28
26
|
|
29
|
-
|
27
|
+
See the [installation notes](#hypopg-installation-notes) if you run into issues.
|
28
|
+
|
29
|
+
Then install the command line tool with:
|
30
30
|
|
31
31
|
```sh
|
32
32
|
gem install pgdexter
|
@@ -36,10 +36,10 @@ The command line tool is also available with [Docker](#docker), [Homebrew](#home
|
|
36
36
|
|
37
37
|
## How to Use
|
38
38
|
|
39
|
-
Dexter needs a connection to your database and a
|
39
|
+
Dexter needs a connection to your database and a source of queries (like [pg_stat_statements](https://www.postgresql.org/docs/current/pgstatstatements.html)) to process.
|
40
40
|
|
41
41
|
```sh
|
42
|
-
|
42
|
+
dexter -d dbname --pg-stat-statements
|
43
43
|
```
|
44
44
|
|
45
45
|
This finds slow queries and generates output like:
|
@@ -53,7 +53,6 @@ Index found: public.movies (title)
|
|
53
53
|
Index found: public.ratings (movie_id)
|
54
54
|
Index found: public.ratings (rating)
|
55
55
|
Index found: public.ratings (user_id)
|
56
|
-
Processing 12 new query fingerprints
|
57
56
|
```
|
58
57
|
|
59
58
|
To be safe, Dexter will not create indexes unless you pass the `--create` flag. In this case, you’ll see:
|
@@ -84,70 +83,95 @@ and connection strings:
|
|
84
83
|
host=localhost port=5432 dbname=mydb
|
85
84
|
```
|
86
85
|
|
86
|
+
Always make sure your [connection is secure](https://ankane.org/postgres-sslmode-explained) when connecting to a database over a network you don’t fully trust.
|
87
|
+
|
87
88
|
## Collecting Queries
|
88
89
|
|
89
|
-
|
90
|
+
Dexter can collect queries from a number of sources.
|
90
91
|
|
91
|
-
|
92
|
-
|
93
|
-
|
92
|
+
- [Query stats](#query-stats)
|
93
|
+
- [Live queries](#live-queries)
|
94
|
+
- [Log files](#log-file)
|
95
|
+
- [SQL files](#sql-files)
|
94
96
|
|
95
|
-
|
97
|
+
### Query Stats
|
96
98
|
|
97
|
-
|
98
|
-
|
99
|
+
Enable [pg_stat_statements](https://www.postgresql.org/docs/current/pgstatstatements.html) in your database.
|
100
|
+
|
101
|
+
```psql
|
102
|
+
CREATE EXTENSION pg_stat_statements;
|
99
103
|
```
|
100
104
|
|
101
|
-
|
105
|
+
And use:
|
102
106
|
|
103
107
|
```sh
|
104
|
-
dexter <connection-options>
|
108
|
+
dexter <connection-options> --pg-stat-statements
|
105
109
|
```
|
106
110
|
|
107
|
-
|
111
|
+
### Live Queries
|
112
|
+
|
113
|
+
Get live queries from the [pg_stat_activity](https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEW) view with:
|
108
114
|
|
109
115
|
```sh
|
110
116
|
dexter <connection-options> --pg-stat-activity
|
111
117
|
```
|
112
118
|
|
113
|
-
###
|
119
|
+
### Log Files
|
114
120
|
|
115
|
-
|
121
|
+
Enable logging for slow queries in your Postgres config file.
|
122
|
+
|
123
|
+
```ini
|
124
|
+
log_min_duration_statement = 10 # ms
|
125
|
+
```
|
126
|
+
|
127
|
+
And use:
|
116
128
|
|
117
129
|
```sh
|
118
|
-
dexter
|
130
|
+
dexter <connection-options> postgresql.log
|
119
131
|
```
|
120
132
|
|
121
|
-
|
133
|
+
Supports `stderr`, `csvlog`, and `jsonlog` formats.
|
134
|
+
|
135
|
+
For real-time indexing, pipe your logfile:
|
122
136
|
|
123
137
|
```sh
|
124
|
-
|
138
|
+
tail -F -n +1 postgresql.log | dexter <connection-options> --stdin
|
125
139
|
```
|
126
140
|
|
127
|
-
|
141
|
+
And pass `--input-format csvlog` or `--input-format jsonlog` if needed.
|
142
|
+
|
143
|
+
### SQL Files
|
144
|
+
|
145
|
+
Pass a SQL file with:
|
128
146
|
|
129
147
|
```sh
|
130
|
-
dexter
|
148
|
+
dexter <connection-options> queries.sql
|
131
149
|
```
|
132
150
|
|
133
|
-
|
151
|
+
Pass a single query with:
|
134
152
|
|
135
153
|
```sh
|
136
|
-
dexter
|
154
|
+
dexter <connection-options> -s "SELECT * FROM ..."
|
137
155
|
```
|
138
156
|
|
139
|
-
##
|
157
|
+
## Collection Options
|
158
|
+
|
159
|
+
To prevent one-off queries from being indexed, specify a minimum number of calls before a query is considered for indexing
|
160
|
+
|
161
|
+
```sh
|
162
|
+
dexter --min-calls 100
|
163
|
+
```
|
140
164
|
|
141
|
-
|
165
|
+
You can do the same for total time a query has run
|
142
166
|
|
143
167
|
```sh
|
144
|
-
|
168
|
+
dexter --min-time 10 # minutes
|
145
169
|
```
|
146
170
|
|
147
|
-
|
171
|
+
When streaming logs, specify the time to wait between processing queries
|
148
172
|
|
149
173
|
```sh
|
150
|
-
|
174
|
+
dexter --interval 60 # seconds
|
151
175
|
```
|
152
176
|
|
153
177
|
## Analyze
|
@@ -186,9 +210,32 @@ The `hypopg` extension, which Dexter needs to run, is available on [these provid
|
|
186
210
|
|
187
211
|
For other providers, see [this guide](guides/Hosted-Postgres.md). To request a new extension:
|
188
212
|
|
189
|
-
- Amazon RDS - follow the instructions on [this page](https://aws.amazon.com/rds/postgresql/faqs/)
|
190
213
|
- Google Cloud SQL - vote or comment on [this page](https://issuetracker.google.com/issues/69250435)
|
191
|
-
- DigitalOcean Managed Databases - vote or comment on [this page](https://ideas.digitalocean.com/
|
214
|
+
- DigitalOcean Managed Databases - vote or comment on [this page](https://ideas.digitalocean.com/managed-database/p/support-hypopg-for-postgres)
|
215
|
+
|
216
|
+
## HypoPG Installation Notes
|
217
|
+
|
218
|
+
### Postgres Location
|
219
|
+
|
220
|
+
If your machine has multiple Postgres installations, specify the path to [pg_config](https://www.postgresql.org/docs/current/app-pgconfig.html) with:
|
221
|
+
|
222
|
+
```sh
|
223
|
+
export PG_CONFIG=/Applications/Postgres.app/Contents/Versions/latest/bin/pg_config
|
224
|
+
```
|
225
|
+
|
226
|
+
Then re-run the installation instructions (run `make clean` before `make` if needed)
|
227
|
+
|
228
|
+
### Missing Header
|
229
|
+
|
230
|
+
If compilation fails with `fatal error: postgres.h: No such file or directory`, make sure Postgres development files are installed on the server.
|
231
|
+
|
232
|
+
For Ubuntu and Debian, use:
|
233
|
+
|
234
|
+
```sh
|
235
|
+
sudo apt-get install postgresql-server-dev-15
|
236
|
+
```
|
237
|
+
|
238
|
+
Note: Replace `15` with your Postgres server version
|
192
239
|
|
193
240
|
## Additional Installation Methods
|
194
241
|
|
@@ -213,12 +260,12 @@ For databases on the host machine, use `host.docker.internal` as the hostname (o
|
|
213
260
|
With Homebrew, you can use:
|
214
261
|
|
215
262
|
```sh
|
216
|
-
brew install
|
263
|
+
brew install dexter
|
217
264
|
```
|
218
265
|
|
219
266
|
## Future Work
|
220
267
|
|
221
|
-
[Here are some ideas](https://github.com/ankane/dexter/issues/
|
268
|
+
[Here are some ideas](https://github.com/ankane/dexter/issues/45)
|
222
269
|
|
223
270
|
## Upgrading
|
224
271
|
|
@@ -235,9 +282,19 @@ gem install specific_install
|
|
235
282
|
gem specific_install https://github.com/ankane/dexter.git
|
236
283
|
```
|
237
284
|
|
285
|
+
## Upgrade Notes
|
286
|
+
|
287
|
+
### 0.5.0
|
288
|
+
|
289
|
+
The `--stdin` option is now required to read queries from stdin.
|
290
|
+
|
291
|
+
```sh
|
292
|
+
tail -F -n +1 postgresql.log | dexter <connection-options> --stdin
|
293
|
+
```
|
294
|
+
|
238
295
|
## Thanks
|
239
296
|
|
240
|
-
This software wouldn’t be possible without [HypoPG](https://github.com/HypoPG/hypopg), which allows you to create hypothetical indexes, and [pg_query](https://github.com/lfittl/pg_query), which allows you to parse and fingerprint queries. A big thanks to Dalibo and Lukas Fittl respectively.
|
297
|
+
This software wouldn’t be possible without [HypoPG](https://github.com/HypoPG/hypopg), which allows you to create hypothetical indexes, and [pg_query](https://github.com/lfittl/pg_query), which allows you to parse and fingerprint queries. A big thanks to Dalibo and Lukas Fittl respectively. Also, thanks to YugabyteDB for [this article](https://dev.to/yugabyte/explain-from-pgstatstatements-normalized-queries-how-to-always-get-the-generic-plan-in--5cfi) on how to explain normalized queries.
|
241
298
|
|
242
299
|
## Research
|
243
300
|
|
data/lib/dexter/client.rb
CHANGED
@@ -7,7 +7,7 @@ module Dexter
|
|
7
7
|
|
8
8
|
def self.start
|
9
9
|
Dexter::Client.new(ARGV).perform
|
10
|
-
rescue Dexter::Abort, PG::UndefinedFile => e
|
10
|
+
rescue Dexter::Abort, PG::UndefinedFile, PG::FeatureNotSupported => e
|
11
11
|
abort colorize(e.message.strip, :red)
|
12
12
|
end
|
13
13
|
|
@@ -27,13 +27,17 @@ module Dexter
|
|
27
27
|
Indexer.new(options).process_stat_statements
|
28
28
|
elsif options[:pg_stat_activity]
|
29
29
|
Processor.new(:pg_stat_activity, options).perform
|
30
|
-
elsif options[:log_table]
|
31
|
-
Processor.new(:log_table, options).perform
|
32
30
|
elsif arguments.any?
|
33
31
|
ARGV.replace(arguments)
|
32
|
+
if !options[:input_format]
|
33
|
+
ext = ARGV.map { |v| File.extname(v) }.uniq
|
34
|
+
options[:input_format] = ext.first[1..-1] if ext.size == 1
|
35
|
+
end
|
34
36
|
Processor.new(ARGF, options).perform
|
35
|
-
|
37
|
+
elsif options[:stdin]
|
36
38
|
Processor.new(STDIN, options).perform
|
39
|
+
else
|
40
|
+
raise Dexter::Abort, "Specify a source of queries: --pg-stat-statements, --pg-stat-activity, --stdin, or a path"
|
37
41
|
end
|
38
42
|
end
|
39
43
|
|
@@ -44,10 +48,10 @@ module Dexter
|
|
44
48
|
o.separator ""
|
45
49
|
|
46
50
|
o.separator "Input options:"
|
47
|
-
o.string "--input-format", "input format"
|
48
|
-
o.
|
49
|
-
o.boolean "--pg-stat-activity", "use pg_stat_activity", default: false, help: false
|
51
|
+
o.string "--input-format", "input format"
|
52
|
+
o.boolean "--pg-stat-activity", "use pg_stat_activity", default: false
|
50
53
|
o.boolean "--pg-stat-statements", "use pg_stat_statements", default: false, help: false
|
54
|
+
o.boolean "--stdin", "use stdin", default: false
|
51
55
|
o.string "-s", "--statement", "process a single statement"
|
52
56
|
o.separator ""
|
53
57
|
|
@@ -62,12 +66,12 @@ module Dexter
|
|
62
66
|
o.integer "--interval", "time to wait between processing queries, in seconds", default: 60
|
63
67
|
o.float "--min-calls", "only process queries that have been called a certain number of times", default: 0
|
64
68
|
o.float "--min-time", "only process queries that have consumed a certain amount of DB time, in minutes", default: 0
|
65
|
-
o.boolean "--once", "run once", default: false, help: false
|
66
69
|
o.separator ""
|
67
70
|
|
68
71
|
o.separator "Indexing options:"
|
69
72
|
o.boolean "--analyze", "analyze tables that haven't been analyzed in the past hour", default: false
|
70
73
|
o.boolean "--create", "create indexes", default: false
|
74
|
+
o.boolean "--enable-hypopg", "enable the HypoPG extension", default: false
|
71
75
|
o.array "--exclude", "prevent specific tables from being indexed"
|
72
76
|
o.string "--include", "only include specific tables"
|
73
77
|
o.integer "--min-cost-savings-pct", default: 50, help: false
|
data/lib/dexter/collector.rb
CHANGED
@@ -8,7 +8,7 @@ module Dexter
|
|
8
8
|
@min_calls = options[:min_calls]
|
9
9
|
end
|
10
10
|
|
11
|
-
def add(query,
|
11
|
+
def add(query, total_time, calls = 1)
|
12
12
|
fingerprint =
|
13
13
|
begin
|
14
14
|
PgQuery.fingerprint(query)
|
@@ -19,8 +19,8 @@ module Dexter
|
|
19
19
|
return unless fingerprint
|
20
20
|
|
21
21
|
@top_queries[fingerprint] ||= {calls: 0, total_time: 0}
|
22
|
-
@top_queries[fingerprint][:calls] +=
|
23
|
-
@top_queries[fingerprint][:total_time] +=
|
22
|
+
@top_queries[fingerprint][:calls] += calls
|
23
|
+
@top_queries[fingerprint][:total_time] += total_time
|
24
24
|
@top_queries[fingerprint][:query] = query
|
25
25
|
@mutex.synchronize do
|
26
26
|
@new_queries << fingerprint
|
data/lib/dexter/indexer.rb
CHANGED
@@ -14,11 +14,15 @@ module Dexter
|
|
14
14
|
@min_calls = options[:min_calls] || 0
|
15
15
|
@analyze = options[:analyze]
|
16
16
|
@min_cost_savings_pct = options[:min_cost_savings_pct].to_i
|
17
|
-
@log_table = options[:log_table]
|
18
17
|
@options = options
|
19
18
|
@mutex = Mutex.new
|
20
19
|
|
21
|
-
|
20
|
+
if server_version_num < 110000
|
21
|
+
raise Dexter::Abort, "This version of Dexter requires Postgres 11+"
|
22
|
+
end
|
23
|
+
|
24
|
+
check_extension
|
25
|
+
|
22
26
|
execute("SET lock_timeout = '5s'")
|
23
27
|
end
|
24
28
|
|
@@ -28,53 +32,6 @@ module Dexter
|
|
28
32
|
process_queries(queries)
|
29
33
|
end
|
30
34
|
|
31
|
-
def stat_activity
|
32
|
-
execute <<~SQL
|
33
|
-
SELECT
|
34
|
-
pid || ':' || COALESCE(query_start, xact_start) AS id,
|
35
|
-
query,
|
36
|
-
EXTRACT(EPOCH FROM NOW() - COALESCE(query_start, xact_start)) * 1000.0 AS duration_ms
|
37
|
-
FROM
|
38
|
-
pg_stat_activity
|
39
|
-
WHERE
|
40
|
-
datname = current_database()
|
41
|
-
AND state = 'active'
|
42
|
-
AND pid != pg_backend_pid()
|
43
|
-
ORDER BY
|
44
|
-
1
|
45
|
-
SQL
|
46
|
-
end
|
47
|
-
|
48
|
-
# works with
|
49
|
-
# file_fdw: https://www.postgresql.org/docs/current/file-fdw.html
|
50
|
-
# log_fdw: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.Extensions.foreign-data-wrappers.html
|
51
|
-
def csvlog_activity(last_log_time)
|
52
|
-
query = <<~SQL
|
53
|
-
SELECT
|
54
|
-
log_time,
|
55
|
-
message,
|
56
|
-
detail
|
57
|
-
FROM
|
58
|
-
#{conn.quote_ident(@log_table)}
|
59
|
-
WHERE
|
60
|
-
log_time >= \$1
|
61
|
-
SQL
|
62
|
-
execute(query, params: [last_log_time])
|
63
|
-
end
|
64
|
-
|
65
|
-
# works with
|
66
|
-
# file_fdw: https://www.postgresql.org/docs/current/file-fdw.html
|
67
|
-
# log_fdw: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.Extensions.foreign-data-wrappers.html
|
68
|
-
def stderr_activity
|
69
|
-
query = <<~SQL
|
70
|
-
SELECT
|
71
|
-
log_entry
|
72
|
-
FROM
|
73
|
-
#{conn.quote_ident(@log_table)}
|
74
|
-
SQL
|
75
|
-
execute(query)
|
76
|
-
end
|
77
|
-
|
78
35
|
def process_queries(queries)
|
79
36
|
# reset hypothetical indexes
|
80
37
|
reset_hypothetical_indexes
|
@@ -150,19 +107,20 @@ module Dexter
|
|
150
107
|
|
151
108
|
private
|
152
109
|
|
153
|
-
def
|
154
|
-
execute("
|
155
|
-
|
156
|
-
|
157
|
-
rescue PG::UndefinedFile
|
110
|
+
def check_extension
|
111
|
+
extension = execute("SELECT installed_version FROM pg_available_extensions WHERE name = 'hypopg'").first
|
112
|
+
|
113
|
+
if extension.nil?
|
158
114
|
raise Dexter::Abort, "Install HypoPG first: https://github.com/ankane/dexter#installation"
|
159
|
-
rescue PG::InsufficientPrivilege
|
160
|
-
raise Dexter::Abort, "Use a superuser to run: CREATE EXTENSION hypopg"
|
161
115
|
end
|
162
|
-
end
|
163
116
|
|
164
|
-
|
165
|
-
|
117
|
+
if extension["installed_version"].nil?
|
118
|
+
if @options[:enable_hypopg]
|
119
|
+
execute("CREATE EXTENSION hypopg")
|
120
|
+
else
|
121
|
+
raise Dexter::Abort, "Run `CREATE EXTENSION hypopg` or pass --enable-hypopg"
|
122
|
+
end
|
123
|
+
end
|
166
124
|
end
|
167
125
|
|
168
126
|
def reset_hypothetical_indexes
|
@@ -172,7 +130,7 @@ module Dexter
|
|
172
130
|
def analyze_tables(tables)
|
173
131
|
tables = tables.to_a.sort
|
174
132
|
|
175
|
-
|
133
|
+
query = <<~SQL
|
176
134
|
SELECT
|
177
135
|
schemaname || '.' || relname AS table,
|
178
136
|
last_analyze,
|
@@ -180,8 +138,9 @@ module Dexter
|
|
180
138
|
FROM
|
181
139
|
pg_stat_user_tables
|
182
140
|
WHERE
|
183
|
-
schemaname || '.' || relname IN (#{tables.map { |
|
141
|
+
schemaname || '.' || relname IN (#{tables.size.times.map { |i| "$#{i + 1}" }.join(", ")})
|
184
142
|
SQL
|
143
|
+
analyze_stats = execute(query, params: tables.to_a)
|
185
144
|
|
186
145
|
last_analyzed = {}
|
187
146
|
analyze_stats.each do |stats|
|
@@ -212,10 +171,6 @@ module Dexter
|
|
212
171
|
end
|
213
172
|
begin
|
214
173
|
query.plans << plan(query.statement)
|
215
|
-
if @log_explain
|
216
|
-
# Pass format to prevent ANALYZE
|
217
|
-
puts execute("EXPLAIN (FORMAT TEXT) #{safe_statement(query.statement)}", pretty: false).map { |r| r["QUERY PLAN"] }.join("\n")
|
218
|
-
end
|
219
174
|
rescue PG::Error, JSON::NestingError => e
|
220
175
|
if @log_explain
|
221
176
|
log e.message
|
@@ -242,10 +197,16 @@ module Dexter
|
|
242
197
|
possible_columns = Set.new
|
243
198
|
explainable_queries.each do |query|
|
244
199
|
log "Finding columns: #{query.statement}" if @log_level == "debug3"
|
245
|
-
|
246
|
-
|
247
|
-
|
248
|
-
|
200
|
+
begin
|
201
|
+
find_columns(query.tree).each do |col|
|
202
|
+
last_col = col["fields"].last
|
203
|
+
if last_col["String"]
|
204
|
+
possible_columns << last_col["String"]["sval"]
|
205
|
+
end
|
206
|
+
end
|
207
|
+
rescue JSON::NestingError
|
208
|
+
if @log_level.start_with?("debug")
|
209
|
+
log colorize("ERROR: Cannot get columns", :red)
|
249
210
|
end
|
250
211
|
end
|
251
212
|
end
|
@@ -271,7 +232,7 @@ module Dexter
|
|
271
232
|
end
|
272
233
|
|
273
234
|
def find_columns(plan)
|
274
|
-
plan = JSON.parse(plan.to_json)
|
235
|
+
plan = JSON.parse(plan.to_json, max_nesting: 1000)
|
275
236
|
find_by_key(plan, "ColumnRef")
|
276
237
|
end
|
277
238
|
|
@@ -541,7 +502,7 @@ module Dexter
|
|
541
502
|
def conn
|
542
503
|
@conn ||= begin
|
543
504
|
# set connect timeout if none set
|
544
|
-
ENV["PGCONNECT_TIMEOUT"] ||= "
|
505
|
+
ENV["PGCONNECT_TIMEOUT"] ||= "3"
|
545
506
|
|
546
507
|
if @options[:dbname] =~ /\Apostgres(ql)?:\/\//
|
547
508
|
config = @options[:dbname]
|
@@ -569,16 +530,56 @@ module Dexter
|
|
569
530
|
# as an extra defense against SQL-injection attacks.
|
570
531
|
# https://www.postgresql.org/docs/current/static/libpq-exec.html
|
571
532
|
query = squish(query) if pretty
|
572
|
-
log colorize("[sql] #{query}", :cyan) if @log_sql
|
533
|
+
log colorize("[sql] #{query}#{params.any? ? " /*#{params.to_json}*/" : ""}", :cyan) if @log_sql
|
573
534
|
|
574
535
|
@mutex.synchronize do
|
575
|
-
conn.exec_params(query, params).to_a
|
536
|
+
conn.exec_params("#{query} /*dexter*/", params).to_a
|
576
537
|
end
|
577
538
|
end
|
578
539
|
|
579
540
|
def plan(query)
|
541
|
+
prepared = false
|
542
|
+
transaction = false
|
543
|
+
|
544
|
+
# try to EXPLAIN normalized queries
|
545
|
+
# https://dev.to/yugabyte/explain-from-pgstatstatements-normalized-queries-how-to-always-get-the-generic-plan-in--5cfi
|
546
|
+
explain_normalized = query.include?("$1")
|
547
|
+
if explain_normalized
|
548
|
+
prepared_name = "dexter_prepared"
|
549
|
+
execute("PREPARE #{prepared_name} AS #{safe_statement(query)}", pretty: false)
|
550
|
+
prepared = true
|
551
|
+
params = execute("SELECT array_length(parameter_types, 1) AS params FROM pg_prepared_statements WHERE name = $1", params: [prepared_name]).first["params"].to_i
|
552
|
+
query = "EXECUTE #{prepared_name}(#{params.times.map { "NULL" }.join(", ")})"
|
553
|
+
|
554
|
+
execute("BEGIN")
|
555
|
+
transaction = true
|
556
|
+
|
557
|
+
if server_version_num >= 120000
|
558
|
+
execute("SET LOCAL plan_cache_mode = force_generic_plan")
|
559
|
+
else
|
560
|
+
execute("SET LOCAL cpu_operator_cost = 1e42")
|
561
|
+
5.times do
|
562
|
+
execute("EXPLAIN (FORMAT JSON) #{safe_statement(query)}", pretty: false)
|
563
|
+
end
|
564
|
+
execute("ROLLBACK")
|
565
|
+
execute("BEGIN")
|
566
|
+
end
|
567
|
+
end
|
568
|
+
|
580
569
|
# strip semi-colons as another measure of defense
|
581
|
-
JSON.parse(execute("EXPLAIN (FORMAT JSON) #{safe_statement(query)}", pretty: false).first["QUERY PLAN"], max_nesting: 1000).first["Plan"]
|
570
|
+
plan = JSON.parse(execute("EXPLAIN (FORMAT JSON) #{safe_statement(query)}", pretty: false).first["QUERY PLAN"], max_nesting: 1000).first["Plan"]
|
571
|
+
|
572
|
+
if @log_explain
|
573
|
+
# Pass format to prevent ANALYZE
|
574
|
+
puts execute("EXPLAIN (FORMAT TEXT) #{safe_statement(query)}", pretty: false).map { |r| r["QUERY PLAN"] }.join("\n")
|
575
|
+
end
|
576
|
+
|
577
|
+
plan
|
578
|
+
ensure
|
579
|
+
if explain_normalized
|
580
|
+
execute("ROLLBACK") if transaction
|
581
|
+
execute("DEALLOCATE #{prepared_name}") if prepared
|
582
|
+
end
|
582
583
|
end
|
583
584
|
|
584
585
|
# TODO for multicolumn indexes, use ordering
|
@@ -608,17 +609,13 @@ module Dexter
|
|
608
609
|
end
|
609
610
|
|
610
611
|
def materialized_views
|
611
|
-
|
612
|
-
|
613
|
-
|
614
|
-
|
615
|
-
|
616
|
-
|
617
|
-
|
618
|
-
result.map { |r| r["table_name"] }
|
619
|
-
else
|
620
|
-
[]
|
621
|
-
end
|
612
|
+
result = execute <<~SQL
|
613
|
+
SELECT
|
614
|
+
schemaname || '.' || matviewname AS table_name
|
615
|
+
FROM
|
616
|
+
pg_matviews
|
617
|
+
SQL
|
618
|
+
result.map { |r| r["table_name"] }
|
622
619
|
end
|
623
620
|
|
624
621
|
def server_version_num
|
@@ -652,7 +649,7 @@ module Dexter
|
|
652
649
|
|
653
650
|
def stat_statements
|
654
651
|
total_time = server_version_num >= 130000 ? "(total_plan_time + total_exec_time)" : "total_time"
|
655
|
-
|
652
|
+
sql = <<~SQL
|
656
653
|
SELECT
|
657
654
|
DISTINCT query
|
658
655
|
FROM
|
@@ -661,18 +658,18 @@ module Dexter
|
|
661
658
|
pg_database ON pg_database.oid = pg_stat_statements.dbid
|
662
659
|
WHERE
|
663
660
|
datname = current_database()
|
664
|
-
AND #{total_time} >=
|
665
|
-
AND calls >=
|
661
|
+
AND #{total_time} >= \$1
|
662
|
+
AND calls >= \$2
|
666
663
|
ORDER BY
|
667
664
|
1
|
668
665
|
SQL
|
669
|
-
|
666
|
+
execute(sql, params: [@min_time * 60000, @min_calls]).map { |q| q["query"] }
|
670
667
|
end
|
671
668
|
|
672
669
|
def with_advisory_lock
|
673
670
|
lock_id = 123456
|
674
671
|
first_time = true
|
675
|
-
while execute("SELECT pg_try_advisory_lock(
|
672
|
+
while execute("SELECT pg_try_advisory_lock($1)", params: [lock_id]).first["pg_try_advisory_lock"] != "t"
|
676
673
|
if first_time
|
677
674
|
log "Waiting for lock..."
|
678
675
|
first_time = false
|
@@ -681,16 +678,19 @@ module Dexter
|
|
681
678
|
end
|
682
679
|
yield
|
683
680
|
ensure
|
684
|
-
|
685
|
-
execute("SELECT pg_advisory_unlock(
|
681
|
+
suppress_messages do
|
682
|
+
execute("SELECT pg_advisory_unlock($1)", params: [lock_id])
|
686
683
|
end
|
687
684
|
end
|
688
685
|
|
689
|
-
def
|
690
|
-
|
686
|
+
def suppress_messages
|
687
|
+
conn.set_notice_processor do |message|
|
688
|
+
# do nothing
|
689
|
+
end
|
691
690
|
yield
|
692
691
|
ensure
|
693
|
-
|
692
|
+
# clear notice processor
|
693
|
+
conn.set_notice_processor
|
694
694
|
end
|
695
695
|
|
696
696
|
def index_exists?(index)
|
@@ -698,7 +698,7 @@ module Dexter
|
|
698
698
|
end
|
699
699
|
|
700
700
|
def columns(tables)
|
701
|
-
|
701
|
+
query = <<~SQL
|
702
702
|
SELECT
|
703
703
|
s.nspname || '.' || t.relname AS table_name,
|
704
704
|
a.attname AS column_name,
|
@@ -708,11 +708,11 @@ module Dexter
|
|
708
708
|
JOIN pg_namespace s on t.relnamespace = s.oid
|
709
709
|
WHERE a.attnum > 0
|
710
710
|
AND NOT a.attisdropped
|
711
|
-
AND s.nspname || '.' || t.relname IN (#{tables.map { |
|
711
|
+
AND s.nspname || '.' || t.relname IN (#{tables.size.times.map { |i| "$#{i + 1}" }.join(", ")})
|
712
712
|
ORDER BY
|
713
713
|
1, 2
|
714
714
|
SQL
|
715
|
-
|
715
|
+
columns = execute(query, params: tables.to_a)
|
716
716
|
columns.map { |v| {table: v["table_name"], column: v["column_name"], type: v["data_type"]} }
|
717
717
|
end
|
718
718
|
|
@@ -732,14 +732,14 @@ module Dexter
|
|
732
732
|
LEFT JOIN
|
733
733
|
pg_stat_user_indexes ui ON ui.indexrelid = i.indexrelid
|
734
734
|
WHERE
|
735
|
-
schemaname || '.' || t.relname IN (#{tables.map { |
|
735
|
+
schemaname || '.' || t.relname IN (#{tables.size.times.map { |i| "$#{i + 1}" }.join(", ")}) AND
|
736
736
|
indisvalid = 't' AND
|
737
737
|
indexprs IS NULL AND
|
738
738
|
indpred IS NULL
|
739
739
|
ORDER BY
|
740
740
|
1, 2
|
741
741
|
SQL
|
742
|
-
execute(query).map { |v| v["columns"] = v["columns"].sub(") WHERE (", " WHERE ").split(", ").map { |c| unquote(c) }; v }
|
742
|
+
execute(query, params: tables.to_a).map { |v| v["columns"] = v["columns"].sub(") WHERE (", " WHERE ").split(", ").map { |c| unquote(c) }; v }
|
743
743
|
end
|
744
744
|
|
745
745
|
def search_path
|
@@ -758,19 +758,6 @@ module Dexter
|
|
758
758
|
value.split(".").map { |v| conn.quote_ident(v) }.join(".")
|
759
759
|
end
|
760
760
|
|
761
|
-
def quote(value)
|
762
|
-
if value.is_a?(String)
|
763
|
-
"'#{quote_string(value)}'"
|
764
|
-
else
|
765
|
-
value
|
766
|
-
end
|
767
|
-
end
|
768
|
-
|
769
|
-
# from activerecord
|
770
|
-
def quote_string(s)
|
771
|
-
s.gsub(/\\/, '\&\&').gsub(/'/, "''")
|
772
|
-
end
|
773
|
-
|
774
761
|
# from activesupport
|
775
762
|
def squish(str)
|
776
763
|
str.to_s.gsub(/\A[[:space:]]+/, "").gsub(/[[:space:]]+\z/, "").gsub(/[[:space:]]+/, " ")
|
data/lib/dexter/log_parser.rb
CHANGED
@@ -1,27 +1,50 @@
|
|
1
1
|
module Dexter
|
2
2
|
class PgStatActivityParser < LogParser
|
3
3
|
def perform
|
4
|
-
|
4
|
+
previous_queries = {}
|
5
5
|
|
6
|
-
|
7
|
-
|
8
|
-
|
9
|
-
|
6
|
+
10.times do
|
7
|
+
active_queries = {}
|
8
|
+
processed_queries = {}
|
9
|
+
|
10
|
+
stat_activity.each do |row|
|
11
|
+
if row["state"] == "active"
|
12
|
+
active_queries[row["id"]] = row
|
13
|
+
else
|
14
|
+
process_entry(row["query"], row["duration_ms"].to_f)
|
15
|
+
processed_queries[row["id"]] = true
|
16
|
+
end
|
10
17
|
end
|
11
18
|
|
12
19
|
# store queries after they complete
|
13
|
-
|
14
|
-
|
20
|
+
previous_queries.each do |id, row|
|
21
|
+
if !active_queries[id] && !processed_queries[id]
|
15
22
|
process_entry(row["query"], row["duration_ms"].to_f)
|
16
23
|
end
|
17
24
|
end
|
18
25
|
|
19
|
-
|
26
|
+
previous_queries = active_queries
|
20
27
|
|
21
|
-
|
22
|
-
|
23
|
-
sleep(1)
|
28
|
+
sleep(0.1)
|
24
29
|
end
|
25
30
|
end
|
31
|
+
|
32
|
+
def stat_activity
|
33
|
+
sql = <<~SQL
|
34
|
+
SELECT
|
35
|
+
pid || ':' || COALESCE(query_start, xact_start) AS id,
|
36
|
+
query,
|
37
|
+
state,
|
38
|
+
EXTRACT(EPOCH FROM NOW() - COALESCE(query_start, xact_start)) * 1000.0 AS duration_ms
|
39
|
+
FROM
|
40
|
+
pg_stat_activity
|
41
|
+
WHERE
|
42
|
+
datname = current_database()
|
43
|
+
AND pid != pg_backend_pid()
|
44
|
+
ORDER BY
|
45
|
+
1
|
46
|
+
SQL
|
47
|
+
@logfile.send(:execute, sql)
|
48
|
+
end
|
26
49
|
end
|
27
50
|
end
|
data/lib/dexter/processor.rb
CHANGED
@@ -11,12 +11,6 @@ module Dexter
|
|
11
11
|
@log_parser =
|
12
12
|
if @logfile == :pg_stat_activity
|
13
13
|
PgStatActivityParser.new(@indexer, @collector)
|
14
|
-
elsif @logfile == :log_table
|
15
|
-
if options[:input_format] == "csv"
|
16
|
-
CsvLogTableParser.new(@indexer, @collector)
|
17
|
-
else
|
18
|
-
StderrLogTableParser.new(@indexer, @collector)
|
19
|
-
end
|
20
14
|
elsif options[:input_format] == "csv"
|
21
15
|
CsvLogParser.new(logfile, @collector)
|
22
16
|
elsif options[:input_format] == "json"
|
@@ -29,7 +23,6 @@ module Dexter
|
|
29
23
|
|
30
24
|
@starting_interval = 3
|
31
25
|
@interval = options[:interval]
|
32
|
-
@log_parser.once = options[:once]
|
33
26
|
|
34
27
|
@mutex = Mutex.new
|
35
28
|
@last_checked_at = {}
|
@@ -38,7 +31,7 @@ module Dexter
|
|
38
31
|
end
|
39
32
|
|
40
33
|
def perform
|
41
|
-
if [STDIN
|
34
|
+
if [STDIN].include?(@logfile)
|
42
35
|
Thread.abort_on_exception = true
|
43
36
|
Thread.new do
|
44
37
|
sleep(@starting_interval)
|
data/lib/dexter/version.rb
CHANGED
data/lib/dexter.rb
CHANGED
@@ -21,12 +21,10 @@ require_relative "dexter/version"
|
|
21
21
|
# parsers
|
22
22
|
require_relative "dexter/log_parser"
|
23
23
|
require_relative "dexter/csv_log_parser"
|
24
|
-
require_relative "dexter/csv_log_table_parser"
|
25
24
|
require_relative "dexter/json_log_parser"
|
26
25
|
require_relative "dexter/pg_stat_activity_parser"
|
27
26
|
require_relative "dexter/sql_log_parser"
|
28
27
|
require_relative "dexter/stderr_log_parser"
|
29
|
-
require_relative "dexter/stderr_log_table_parser"
|
30
28
|
|
31
29
|
module Dexter
|
32
30
|
class Abort < StandardError; end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: pgdexter
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.5.1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Andrew Kane
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2023-
|
11
|
+
date: 2023-05-27 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: pg
|
@@ -30,14 +30,14 @@ dependencies:
|
|
30
30
|
requirements:
|
31
31
|
- - "~>"
|
32
32
|
- !ruby/object:Gem::Version
|
33
|
-
version: '
|
33
|
+
version: '4'
|
34
34
|
type: :runtime
|
35
35
|
prerelease: false
|
36
36
|
version_requirements: !ruby/object:Gem::Requirement
|
37
37
|
requirements:
|
38
38
|
- - "~>"
|
39
39
|
- !ruby/object:Gem::Version
|
40
|
-
version: '
|
40
|
+
version: '4'
|
41
41
|
- !ruby/object:Gem::Dependency
|
42
42
|
name: slop
|
43
43
|
requirement: !ruby/object:Gem::Requirement
|
@@ -67,7 +67,6 @@ files:
|
|
67
67
|
- lib/dexter/client.rb
|
68
68
|
- lib/dexter/collector.rb
|
69
69
|
- lib/dexter/csv_log_parser.rb
|
70
|
-
- lib/dexter/csv_log_table_parser.rb
|
71
70
|
- lib/dexter/indexer.rb
|
72
71
|
- lib/dexter/json_log_parser.rb
|
73
72
|
- lib/dexter/log_parser.rb
|
@@ -77,7 +76,6 @@ files:
|
|
77
76
|
- lib/dexter/query.rb
|
78
77
|
- lib/dexter/sql_log_parser.rb
|
79
78
|
- lib/dexter/stderr_log_parser.rb
|
80
|
-
- lib/dexter/stderr_log_table_parser.rb
|
81
79
|
- lib/dexter/version.rb
|
82
80
|
homepage: https://github.com/ankane/dexter
|
83
81
|
licenses:
|
@@ -91,14 +89,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
91
89
|
requirements:
|
92
90
|
- - ">="
|
93
91
|
- !ruby/object:Gem::Version
|
94
|
-
version: '2.
|
92
|
+
version: '2.7'
|
95
93
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
96
94
|
requirements:
|
97
95
|
- - ">="
|
98
96
|
- !ruby/object:Gem::Version
|
99
97
|
version: '0'
|
100
98
|
requirements: []
|
101
|
-
rubygems_version: 3.4.
|
99
|
+
rubygems_version: 3.4.10
|
102
100
|
signing_key:
|
103
101
|
specification_version: 4
|
104
102
|
summary: The automatic indexer for Postgres
|
@@ -1,20 +0,0 @@
|
|
1
|
-
module Dexter
|
2
|
-
class CsvLogTableParser < CsvLogParser
|
3
|
-
def perform
|
4
|
-
last_log_time = Time.at(0).iso8601(3)
|
5
|
-
|
6
|
-
loop do
|
7
|
-
@logfile.csvlog_activity(last_log_time).each do |row|
|
8
|
-
process_csv_row(row["message"], row["detail"])
|
9
|
-
last_log_time = row["log_time"]
|
10
|
-
end
|
11
|
-
|
12
|
-
break
|
13
|
-
|
14
|
-
# possibly enable later
|
15
|
-
# break if once
|
16
|
-
# sleep(1)
|
17
|
-
end
|
18
|
-
end
|
19
|
-
end
|
20
|
-
end
|