pgdexter 0.4.3 → 0.5.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: d2b2528c0ff8201ce676b1a4ca7ec11de52013bfc608fab1b574122a31dd5ae9
4
- data.tar.gz: 570f24c171e02425a5c492068ccedf48e318194333d825cc384ce6cfeeb6c30a
3
+ metadata.gz: c68746f6134f6603c5549b561886b45ee3432df5d1e07e65a5907559901d333a
4
+ data.tar.gz: 699c2744f3e2c9a79f8fa9ca6ae7a77586421243cbf7a1f92642f76724cc0cb0
5
5
  SHA512:
6
- metadata.gz: e0975c42916c22b0d8f0b12f0144865370372771526ce32574a4602918aadd786b8ab39cb092ad28590c61c86820c51dcac4f7a41cc5930412868ce2a0590498
7
- data.tar.gz: e91d272a0d6bc737634615118a1a33cbbf2ac2fe957aabb17edae9ae42fa17f596052e83a9caa80b71effa6b20316b12e551bd8aea7734648b3210bdb2bf00ec
6
+ metadata.gz: 343bc52539ef09541fd0774667034ea02fb88b0e4f820e52c86d77df46d059c253576c22272b91a407b764965f66c554278154da6ab923249ce9979ab3a5aed0
7
+ data.tar.gz: a9b86931b4fc58d2b89c87f46f5008dfd47274cc39f096ec2b5143bea8858e29a786e387f9ccc29fe74475b5b0317e9d348d36f2a686cb19457ee79179a270f5
data/CHANGELOG.md CHANGED
@@ -1,3 +1,20 @@
1
+ ## 0.5.1 (2023-05-27)
2
+
3
+ - Fixed `JSON::NestingError`
4
+
5
+ ## 0.5.0 (2023-04-18)
6
+
7
+ - Added support for normalized queries
8
+ - Added `--stdin` option (now required to read from stdin)
9
+ - Added `--enable-hypopg` option (now required to enable HypoPG)
10
+ - Improved output when HypoPG not installed
11
+ - Changed `--pg-stat-activity` to sample 10 times and exit
12
+ - Detect input format based on file extension
13
+ - Dropped support for experimental `--log-table` option
14
+ - Dropped support for Linux packages for Ubuntu 18.04 and Debian 10
15
+ - Dropped support for Ruby < 2.7
16
+ - Dropped support for Postgres < 11
17
+
1
18
  ## 0.4.3 (2023-03-26)
2
19
 
3
20
  - Added experimental `--log-table` option
data/README.md CHANGED
@@ -12,21 +12,21 @@ First, install [HypoPG](https://github.com/HypoPG/hypopg) on your database serve
12
12
 
13
13
  ```sh
14
14
  cd /tmp
15
- curl -L https://github.com/HypoPG/hypopg/archive/1.3.1.tar.gz | tar xz
16
- cd hypopg-1.3.1
15
+ curl -L https://github.com/HypoPG/hypopg/archive/1.4.0.tar.gz | tar xz
16
+ cd hypopg-1.4.0
17
17
  make
18
18
  make install # may need sudo
19
19
  ```
20
20
 
21
- > Note: If you have issues, make sure `postgresql-server-dev-*` is installed.
21
+ And enable it in databases where you want to use Dexter:
22
22
 
23
- Enable logging for slow queries in your Postgres config file.
24
-
25
- ```ini
26
- log_min_duration_statement = 10 # ms
23
+ ```sql
24
+ CREATE EXTENSION hypopg;
27
25
  ```
28
26
 
29
- And install the command line tool with:
27
+ See the [installation notes](#hypopg-installation-notes) if you run into issues.
28
+
29
+ Then install the command line tool with:
30
30
 
31
31
  ```sh
32
32
  gem install pgdexter
@@ -36,10 +36,10 @@ The command line tool is also available with [Docker](#docker), [Homebrew](#home
36
36
 
37
37
  ## How to Use
38
38
 
39
- Dexter needs a connection to your database and a log file to process.
39
+ Dexter needs a connection to your database and a source of queries (like [pg_stat_statements](https://www.postgresql.org/docs/current/pgstatstatements.html)) to process.
40
40
 
41
41
  ```sh
42
- tail -F -n +1 <log-file> | dexter <connection-options>
42
+ dexter -d dbname --pg-stat-statements
43
43
  ```
44
44
 
45
45
  This finds slow queries and generates output like:
@@ -53,7 +53,6 @@ Index found: public.movies (title)
53
53
  Index found: public.ratings (movie_id)
54
54
  Index found: public.ratings (rating)
55
55
  Index found: public.ratings (user_id)
56
- Processing 12 new query fingerprints
57
56
  ```
58
57
 
59
58
  To be safe, Dexter will not create indexes unless you pass the `--create` flag. In this case, you’ll see:
@@ -84,70 +83,95 @@ and connection strings:
84
83
  host=localhost port=5432 dbname=mydb
85
84
  ```
86
85
 
86
+ Always make sure your [connection is secure](https://ankane.org/postgres-sslmode-explained) when connecting to a database over a network you don’t fully trust.
87
+
87
88
  ## Collecting Queries
88
89
 
89
- There are many ways to collect queries. For real-time indexing, pipe your logfile:
90
+ Dexter can collect queries from a number of sources.
90
91
 
91
- ```sh
92
- tail -F -n +1 <log-file> | dexter <connection-options>
93
- ```
92
+ - [Query stats](#query-stats)
93
+ - [Live queries](#live-queries)
94
+ - [Log files](#log-file)
95
+ - [SQL files](#sql-files)
94
96
 
95
- Pass a single statement with:
97
+ ### Query Stats
96
98
 
97
- ```sh
98
- dexter <connection-options> -s "SELECT * FROM ..."
99
+ Enable [pg_stat_statements](https://www.postgresql.org/docs/current/pgstatstatements.html) in your database.
100
+
101
+ ```psql
102
+ CREATE EXTENSION pg_stat_statements;
99
103
  ```
100
104
 
101
- or pass files:
105
+ And use:
102
106
 
103
107
  ```sh
104
- dexter <connection-options> <file1> <file2>
108
+ dexter <connection-options> --pg-stat-statements
105
109
  ```
106
110
 
107
- or collect running queries with:
111
+ ### Live Queries
112
+
113
+ Get live queries from the [pg_stat_activity](https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEW) view with:
108
114
 
109
115
  ```sh
110
116
  dexter <connection-options> --pg-stat-activity
111
117
  ```
112
118
 
113
- ### Collection Options
119
+ ### Log Files
114
120
 
115
- To prevent one-off queries from being indexed, specify a minimum number of calls before a query is considered for indexing
121
+ Enable logging for slow queries in your Postgres config file.
122
+
123
+ ```ini
124
+ log_min_duration_statement = 10 # ms
125
+ ```
126
+
127
+ And use:
116
128
 
117
129
  ```sh
118
- dexter --min-calls 100
130
+ dexter <connection-options> postgresql.log
119
131
  ```
120
132
 
121
- You can do the same for total time a query has run
133
+ Supports `stderr`, `csvlog`, and `jsonlog` formats.
134
+
135
+ For real-time indexing, pipe your logfile:
122
136
 
123
137
  ```sh
124
- dexter --min-time 10 # minutes
138
+ tail -F -n +1 postgresql.log | dexter <connection-options> --stdin
125
139
  ```
126
140
 
127
- Specify the format
141
+ And pass `--input-format csvlog` or `--input-format jsonlog` if needed.
142
+
143
+ ### SQL Files
144
+
145
+ Pass a SQL file with:
128
146
 
129
147
  ```sh
130
- dexter --input-format csv
148
+ dexter <connection-options> queries.sql
131
149
  ```
132
150
 
133
- When streaming logs, specify the time to wait between processing queries
151
+ Pass a single query with:
134
152
 
135
153
  ```sh
136
- dexter --interval 60 # seconds
154
+ dexter <connection-options> -s "SELECT * FROM ..."
137
155
  ```
138
156
 
139
- ## Examples
157
+ ## Collection Options
158
+
159
+ To prevent one-off queries from being indexed, specify a minimum number of calls before a query is considered for indexing
160
+
161
+ ```sh
162
+ dexter --min-calls 100
163
+ ```
140
164
 
141
- Ubuntu with PostgreSQL 12
165
+ You can do the same for total time a query has run
142
166
 
143
167
  ```sh
144
- tail -F -n +1 /var/log/postgresql/postgresql-12-main.log | sudo -u postgres dexter dbname
168
+ dexter --min-time 10 # minutes
145
169
  ```
146
170
 
147
- Homebrew on Mac
171
+ When streaming logs, specify the time to wait between processing queries
148
172
 
149
173
  ```sh
150
- tail -F -n +1 /usr/local/var/postgres/server.log | dexter dbname
174
+ dexter --interval 60 # seconds
151
175
  ```
152
176
 
153
177
  ## Analyze
@@ -186,9 +210,32 @@ The `hypopg` extension, which Dexter needs to run, is available on [these provid
186
210
 
187
211
  For other providers, see [this guide](guides/Hosted-Postgres.md). To request a new extension:
188
212
 
189
- - Amazon RDS - follow the instructions on [this page](https://aws.amazon.com/rds/postgresql/faqs/)
190
213
  - Google Cloud SQL - vote or comment on [this page](https://issuetracker.google.com/issues/69250435)
191
- - DigitalOcean Managed Databases - vote or comment on [this page](https://ideas.digitalocean.com/app-framework-services/p/support-hypopg-for-postgres)
214
+ - DigitalOcean Managed Databases - vote or comment on [this page](https://ideas.digitalocean.com/managed-database/p/support-hypopg-for-postgres)
215
+
216
+ ## HypoPG Installation Notes
217
+
218
+ ### Postgres Location
219
+
220
+ If your machine has multiple Postgres installations, specify the path to [pg_config](https://www.postgresql.org/docs/current/app-pgconfig.html) with:
221
+
222
+ ```sh
223
+ export PG_CONFIG=/Applications/Postgres.app/Contents/Versions/latest/bin/pg_config
224
+ ```
225
+
226
+ Then re-run the installation instructions (run `make clean` before `make` if needed)
227
+
228
+ ### Missing Header
229
+
230
+ If compilation fails with `fatal error: postgres.h: No such file or directory`, make sure Postgres development files are installed on the server.
231
+
232
+ For Ubuntu and Debian, use:
233
+
234
+ ```sh
235
+ sudo apt-get install postgresql-server-dev-15
236
+ ```
237
+
238
+ Note: Replace `15` with your Postgres server version
192
239
 
193
240
  ## Additional Installation Methods
194
241
 
@@ -213,12 +260,12 @@ For databases on the host machine, use `host.docker.internal` as the hostname (o
213
260
  With Homebrew, you can use:
214
261
 
215
262
  ```sh
216
- brew install ankane/brew/dexter
263
+ brew install dexter
217
264
  ```
218
265
 
219
266
  ## Future Work
220
267
 
221
- [Here are some ideas](https://github.com/ankane/dexter/issues/1)
268
+ [Here are some ideas](https://github.com/ankane/dexter/issues/45)
222
269
 
223
270
  ## Upgrading
224
271
 
@@ -235,9 +282,19 @@ gem install specific_install
235
282
  gem specific_install https://github.com/ankane/dexter.git
236
283
  ```
237
284
 
285
+ ## Upgrade Notes
286
+
287
+ ### 0.5.0
288
+
289
+ The `--stdin` option is now required to read queries from stdin.
290
+
291
+ ```sh
292
+ tail -F -n +1 postgresql.log | dexter <connection-options> --stdin
293
+ ```
294
+
238
295
  ## Thanks
239
296
 
240
- This software wouldn’t be possible without [HypoPG](https://github.com/HypoPG/hypopg), which allows you to create hypothetical indexes, and [pg_query](https://github.com/lfittl/pg_query), which allows you to parse and fingerprint queries. A big thanks to Dalibo and Lukas Fittl respectively.
297
+ This software wouldn’t be possible without [HypoPG](https://github.com/HypoPG/hypopg), which allows you to create hypothetical indexes, and [pg_query](https://github.com/lfittl/pg_query), which allows you to parse and fingerprint queries. A big thanks to Dalibo and Lukas Fittl respectively. Also, thanks to YugabyteDB for [this article](https://dev.to/yugabyte/explain-from-pgstatstatements-normalized-queries-how-to-always-get-the-generic-plan-in--5cfi) on how to explain normalized queries.
241
298
 
242
299
  ## Research
243
300
 
data/lib/dexter/client.rb CHANGED
@@ -7,7 +7,7 @@ module Dexter
7
7
 
8
8
  def self.start
9
9
  Dexter::Client.new(ARGV).perform
10
- rescue Dexter::Abort, PG::UndefinedFile => e
10
+ rescue Dexter::Abort, PG::UndefinedFile, PG::FeatureNotSupported => e
11
11
  abort colorize(e.message.strip, :red)
12
12
  end
13
13
 
@@ -27,13 +27,17 @@ module Dexter
27
27
  Indexer.new(options).process_stat_statements
28
28
  elsif options[:pg_stat_activity]
29
29
  Processor.new(:pg_stat_activity, options).perform
30
- elsif options[:log_table]
31
- Processor.new(:log_table, options).perform
32
30
  elsif arguments.any?
33
31
  ARGV.replace(arguments)
32
+ if !options[:input_format]
33
+ ext = ARGV.map { |v| File.extname(v) }.uniq
34
+ options[:input_format] = ext.first[1..-1] if ext.size == 1
35
+ end
34
36
  Processor.new(ARGF, options).perform
35
- else
37
+ elsif options[:stdin]
36
38
  Processor.new(STDIN, options).perform
39
+ else
40
+ raise Dexter::Abort, "Specify a source of queries: --pg-stat-statements, --pg-stat-activity, --stdin, or a path"
37
41
  end
38
42
  end
39
43
 
@@ -44,10 +48,10 @@ module Dexter
44
48
  o.separator ""
45
49
 
46
50
  o.separator "Input options:"
47
- o.string "--input-format", "input format", default: "stderr"
48
- o.string "--log-table", "log table (experimental)"
49
- o.boolean "--pg-stat-activity", "use pg_stat_activity", default: false, help: false
51
+ o.string "--input-format", "input format"
52
+ o.boolean "--pg-stat-activity", "use pg_stat_activity", default: false
50
53
  o.boolean "--pg-stat-statements", "use pg_stat_statements", default: false, help: false
54
+ o.boolean "--stdin", "use stdin", default: false
51
55
  o.string "-s", "--statement", "process a single statement"
52
56
  o.separator ""
53
57
 
@@ -62,12 +66,12 @@ module Dexter
62
66
  o.integer "--interval", "time to wait between processing queries, in seconds", default: 60
63
67
  o.float "--min-calls", "only process queries that have been called a certain number of times", default: 0
64
68
  o.float "--min-time", "only process queries that have consumed a certain amount of DB time, in minutes", default: 0
65
- o.boolean "--once", "run once", default: false, help: false
66
69
  o.separator ""
67
70
 
68
71
  o.separator "Indexing options:"
69
72
  o.boolean "--analyze", "analyze tables that haven't been analyzed in the past hour", default: false
70
73
  o.boolean "--create", "create indexes", default: false
74
+ o.boolean "--enable-hypopg", "enable the HypoPG extension", default: false
71
75
  o.array "--exclude", "prevent specific tables from being indexed"
72
76
  o.string "--include", "only include specific tables"
73
77
  o.integer "--min-cost-savings-pct", default: 50, help: false
@@ -8,7 +8,7 @@ module Dexter
8
8
  @min_calls = options[:min_calls]
9
9
  end
10
10
 
11
- def add(query, duration)
11
+ def add(query, total_time, calls = 1)
12
12
  fingerprint =
13
13
  begin
14
14
  PgQuery.fingerprint(query)
@@ -19,8 +19,8 @@ module Dexter
19
19
  return unless fingerprint
20
20
 
21
21
  @top_queries[fingerprint] ||= {calls: 0, total_time: 0}
22
- @top_queries[fingerprint][:calls] += 1
23
- @top_queries[fingerprint][:total_time] += duration
22
+ @top_queries[fingerprint][:calls] += calls
23
+ @top_queries[fingerprint][:total_time] += total_time
24
24
  @top_queries[fingerprint][:query] = query
25
25
  @mutex.synchronize do
26
26
  @new_queries << fingerprint
@@ -14,11 +14,15 @@ module Dexter
14
14
  @min_calls = options[:min_calls] || 0
15
15
  @analyze = options[:analyze]
16
16
  @min_cost_savings_pct = options[:min_cost_savings_pct].to_i
17
- @log_table = options[:log_table]
18
17
  @options = options
19
18
  @mutex = Mutex.new
20
19
 
21
- create_extension unless extension_exists?
20
+ if server_version_num < 110000
21
+ raise Dexter::Abort, "This version of Dexter requires Postgres 11+"
22
+ end
23
+
24
+ check_extension
25
+
22
26
  execute("SET lock_timeout = '5s'")
23
27
  end
24
28
 
@@ -28,53 +32,6 @@ module Dexter
28
32
  process_queries(queries)
29
33
  end
30
34
 
31
- def stat_activity
32
- execute <<~SQL
33
- SELECT
34
- pid || ':' || COALESCE(query_start, xact_start) AS id,
35
- query,
36
- EXTRACT(EPOCH FROM NOW() - COALESCE(query_start, xact_start)) * 1000.0 AS duration_ms
37
- FROM
38
- pg_stat_activity
39
- WHERE
40
- datname = current_database()
41
- AND state = 'active'
42
- AND pid != pg_backend_pid()
43
- ORDER BY
44
- 1
45
- SQL
46
- end
47
-
48
- # works with
49
- # file_fdw: https://www.postgresql.org/docs/current/file-fdw.html
50
- # log_fdw: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.Extensions.foreign-data-wrappers.html
51
- def csvlog_activity(last_log_time)
52
- query = <<~SQL
53
- SELECT
54
- log_time,
55
- message,
56
- detail
57
- FROM
58
- #{conn.quote_ident(@log_table)}
59
- WHERE
60
- log_time >= \$1
61
- SQL
62
- execute(query, params: [last_log_time])
63
- end
64
-
65
- # works with
66
- # file_fdw: https://www.postgresql.org/docs/current/file-fdw.html
67
- # log_fdw: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.Extensions.foreign-data-wrappers.html
68
- def stderr_activity
69
- query = <<~SQL
70
- SELECT
71
- log_entry
72
- FROM
73
- #{conn.quote_ident(@log_table)}
74
- SQL
75
- execute(query)
76
- end
77
-
78
35
  def process_queries(queries)
79
36
  # reset hypothetical indexes
80
37
  reset_hypothetical_indexes
@@ -150,19 +107,20 @@ module Dexter
150
107
 
151
108
  private
152
109
 
153
- def create_extension
154
- execute("SET client_min_messages = warning")
155
- begin
156
- execute("CREATE EXTENSION IF NOT EXISTS hypopg")
157
- rescue PG::UndefinedFile
110
+ def check_extension
111
+ extension = execute("SELECT installed_version FROM pg_available_extensions WHERE name = 'hypopg'").first
112
+
113
+ if extension.nil?
158
114
  raise Dexter::Abort, "Install HypoPG first: https://github.com/ankane/dexter#installation"
159
- rescue PG::InsufficientPrivilege
160
- raise Dexter::Abort, "Use a superuser to run: CREATE EXTENSION hypopg"
161
115
  end
162
- end
163
116
 
164
- def extension_exists?
165
- execute("SELECT * FROM pg_available_extensions WHERE name = 'hypopg' AND installed_version IS NOT NULL").any?
117
+ if extension["installed_version"].nil?
118
+ if @options[:enable_hypopg]
119
+ execute("CREATE EXTENSION hypopg")
120
+ else
121
+ raise Dexter::Abort, "Run `CREATE EXTENSION hypopg` or pass --enable-hypopg"
122
+ end
123
+ end
166
124
  end
167
125
 
168
126
  def reset_hypothetical_indexes
@@ -172,7 +130,7 @@ module Dexter
172
130
  def analyze_tables(tables)
173
131
  tables = tables.to_a.sort
174
132
 
175
- analyze_stats = execute <<~SQL
133
+ query = <<~SQL
176
134
  SELECT
177
135
  schemaname || '.' || relname AS table,
178
136
  last_analyze,
@@ -180,8 +138,9 @@ module Dexter
180
138
  FROM
181
139
  pg_stat_user_tables
182
140
  WHERE
183
- schemaname || '.' || relname IN (#{tables.map { |t| quote(t) }.join(", ")})
141
+ schemaname || '.' || relname IN (#{tables.size.times.map { |i| "$#{i + 1}" }.join(", ")})
184
142
  SQL
143
+ analyze_stats = execute(query, params: tables.to_a)
185
144
 
186
145
  last_analyzed = {}
187
146
  analyze_stats.each do |stats|
@@ -212,10 +171,6 @@ module Dexter
212
171
  end
213
172
  begin
214
173
  query.plans << plan(query.statement)
215
- if @log_explain
216
- # Pass format to prevent ANALYZE
217
- puts execute("EXPLAIN (FORMAT TEXT) #{safe_statement(query.statement)}", pretty: false).map { |r| r["QUERY PLAN"] }.join("\n")
218
- end
219
174
  rescue PG::Error, JSON::NestingError => e
220
175
  if @log_explain
221
176
  log e.message
@@ -242,10 +197,16 @@ module Dexter
242
197
  possible_columns = Set.new
243
198
  explainable_queries.each do |query|
244
199
  log "Finding columns: #{query.statement}" if @log_level == "debug3"
245
- find_columns(query.tree).each do |col|
246
- last_col = col["fields"].last
247
- if last_col["String"]
248
- possible_columns << last_col["String"]["str"]
200
+ begin
201
+ find_columns(query.tree).each do |col|
202
+ last_col = col["fields"].last
203
+ if last_col["String"]
204
+ possible_columns << last_col["String"]["sval"]
205
+ end
206
+ end
207
+ rescue JSON::NestingError
208
+ if @log_level.start_with?("debug")
209
+ log colorize("ERROR: Cannot get columns", :red)
249
210
  end
250
211
  end
251
212
  end
@@ -271,7 +232,7 @@ module Dexter
271
232
  end
272
233
 
273
234
  def find_columns(plan)
274
- plan = JSON.parse(plan.to_json)
235
+ plan = JSON.parse(plan.to_json, max_nesting: 1000)
275
236
  find_by_key(plan, "ColumnRef")
276
237
  end
277
238
 
@@ -541,7 +502,7 @@ module Dexter
541
502
  def conn
542
503
  @conn ||= begin
543
504
  # set connect timeout if none set
544
- ENV["PGCONNECT_TIMEOUT"] ||= "2"
505
+ ENV["PGCONNECT_TIMEOUT"] ||= "3"
545
506
 
546
507
  if @options[:dbname] =~ /\Apostgres(ql)?:\/\//
547
508
  config = @options[:dbname]
@@ -569,16 +530,56 @@ module Dexter
569
530
  # as an extra defense against SQL-injection attacks.
570
531
  # https://www.postgresql.org/docs/current/static/libpq-exec.html
571
532
  query = squish(query) if pretty
572
- log colorize("[sql] #{query}", :cyan) if @log_sql
533
+ log colorize("[sql] #{query}#{params.any? ? " /*#{params.to_json}*/" : ""}", :cyan) if @log_sql
573
534
 
574
535
  @mutex.synchronize do
575
- conn.exec_params(query, params).to_a
536
+ conn.exec_params("#{query} /*dexter*/", params).to_a
576
537
  end
577
538
  end
578
539
 
579
540
  def plan(query)
541
+ prepared = false
542
+ transaction = false
543
+
544
+ # try to EXPLAIN normalized queries
545
+ # https://dev.to/yugabyte/explain-from-pgstatstatements-normalized-queries-how-to-always-get-the-generic-plan-in--5cfi
546
+ explain_normalized = query.include?("$1")
547
+ if explain_normalized
548
+ prepared_name = "dexter_prepared"
549
+ execute("PREPARE #{prepared_name} AS #{safe_statement(query)}", pretty: false)
550
+ prepared = true
551
+ params = execute("SELECT array_length(parameter_types, 1) AS params FROM pg_prepared_statements WHERE name = $1", params: [prepared_name]).first["params"].to_i
552
+ query = "EXECUTE #{prepared_name}(#{params.times.map { "NULL" }.join(", ")})"
553
+
554
+ execute("BEGIN")
555
+ transaction = true
556
+
557
+ if server_version_num >= 120000
558
+ execute("SET LOCAL plan_cache_mode = force_generic_plan")
559
+ else
560
+ execute("SET LOCAL cpu_operator_cost = 1e42")
561
+ 5.times do
562
+ execute("EXPLAIN (FORMAT JSON) #{safe_statement(query)}", pretty: false)
563
+ end
564
+ execute("ROLLBACK")
565
+ execute("BEGIN")
566
+ end
567
+ end
568
+
580
569
  # strip semi-colons as another measure of defense
581
- JSON.parse(execute("EXPLAIN (FORMAT JSON) #{safe_statement(query)}", pretty: false).first["QUERY PLAN"], max_nesting: 1000).first["Plan"]
570
+ plan = JSON.parse(execute("EXPLAIN (FORMAT JSON) #{safe_statement(query)}", pretty: false).first["QUERY PLAN"], max_nesting: 1000).first["Plan"]
571
+
572
+ if @log_explain
573
+ # Pass format to prevent ANALYZE
574
+ puts execute("EXPLAIN (FORMAT TEXT) #{safe_statement(query)}", pretty: false).map { |r| r["QUERY PLAN"] }.join("\n")
575
+ end
576
+
577
+ plan
578
+ ensure
579
+ if explain_normalized
580
+ execute("ROLLBACK") if transaction
581
+ execute("DEALLOCATE #{prepared_name}") if prepared
582
+ end
582
583
  end
583
584
 
584
585
  # TODO for multicolumn indexes, use ordering
@@ -608,17 +609,13 @@ module Dexter
608
609
  end
609
610
 
610
611
  def materialized_views
611
- if server_version_num >= 90300
612
- result = execute <<~SQL
613
- SELECT
614
- schemaname || '.' || matviewname AS table_name
615
- FROM
616
- pg_matviews
617
- SQL
618
- result.map { |r| r["table_name"] }
619
- else
620
- []
621
- end
612
+ result = execute <<~SQL
613
+ SELECT
614
+ schemaname || '.' || matviewname AS table_name
615
+ FROM
616
+ pg_matviews
617
+ SQL
618
+ result.map { |r| r["table_name"] }
622
619
  end
623
620
 
624
621
  def server_version_num
@@ -652,7 +649,7 @@ module Dexter
652
649
 
653
650
  def stat_statements
654
651
  total_time = server_version_num >= 130000 ? "(total_plan_time + total_exec_time)" : "total_time"
655
- result = execute <<~SQL
652
+ sql = <<~SQL
656
653
  SELECT
657
654
  DISTINCT query
658
655
  FROM
@@ -661,18 +658,18 @@ module Dexter
661
658
  pg_database ON pg_database.oid = pg_stat_statements.dbid
662
659
  WHERE
663
660
  datname = current_database()
664
- AND #{total_time} >= #{@min_time * 60000}
665
- AND calls >= #{@min_calls}
661
+ AND #{total_time} >= \$1
662
+ AND calls >= \$2
666
663
  ORDER BY
667
664
  1
668
665
  SQL
669
- result.map { |q| q["query"] }
666
+ execute(sql, params: [@min_time * 60000, @min_calls]).map { |q| q["query"] }
670
667
  end
671
668
 
672
669
  def with_advisory_lock
673
670
  lock_id = 123456
674
671
  first_time = true
675
- while execute("SELECT pg_try_advisory_lock(#{lock_id})").first["pg_try_advisory_lock"] != "t"
672
+ while execute("SELECT pg_try_advisory_lock($1)", params: [lock_id]).first["pg_try_advisory_lock"] != "t"
676
673
  if first_time
677
674
  log "Waiting for lock..."
678
675
  first_time = false
@@ -681,16 +678,19 @@ module Dexter
681
678
  end
682
679
  yield
683
680
  ensure
684
- with_min_messages("error") do
685
- execute("SELECT pg_advisory_unlock(#{lock_id})")
681
+ suppress_messages do
682
+ execute("SELECT pg_advisory_unlock($1)", params: [lock_id])
686
683
  end
687
684
  end
688
685
 
689
- def with_min_messages(value)
690
- execute("SET client_min_messages = #{quote(value)}")
686
+ def suppress_messages
687
+ conn.set_notice_processor do |message|
688
+ # do nothing
689
+ end
691
690
  yield
692
691
  ensure
693
- execute("SET client_min_messages = warning")
692
+ # clear notice processor
693
+ conn.set_notice_processor
694
694
  end
695
695
 
696
696
  def index_exists?(index)
@@ -698,7 +698,7 @@ module Dexter
698
698
  end
699
699
 
700
700
  def columns(tables)
701
- columns = execute <<~SQL
701
+ query = <<~SQL
702
702
  SELECT
703
703
  s.nspname || '.' || t.relname AS table_name,
704
704
  a.attname AS column_name,
@@ -708,11 +708,11 @@ module Dexter
708
708
  JOIN pg_namespace s on t.relnamespace = s.oid
709
709
  WHERE a.attnum > 0
710
710
  AND NOT a.attisdropped
711
- AND s.nspname || '.' || t.relname IN (#{tables.map { |t| quote(t) }.join(", ")})
711
+ AND s.nspname || '.' || t.relname IN (#{tables.size.times.map { |i| "$#{i + 1}" }.join(", ")})
712
712
  ORDER BY
713
713
  1, 2
714
714
  SQL
715
-
715
+ columns = execute(query, params: tables.to_a)
716
716
  columns.map { |v| {table: v["table_name"], column: v["column_name"], type: v["data_type"]} }
717
717
  end
718
718
 
@@ -732,14 +732,14 @@ module Dexter
732
732
  LEFT JOIN
733
733
  pg_stat_user_indexes ui ON ui.indexrelid = i.indexrelid
734
734
  WHERE
735
- schemaname || '.' || t.relname IN (#{tables.map { |t| quote(t) }.join(", ")}) AND
735
+ schemaname || '.' || t.relname IN (#{tables.size.times.map { |i| "$#{i + 1}" }.join(", ")}) AND
736
736
  indisvalid = 't' AND
737
737
  indexprs IS NULL AND
738
738
  indpred IS NULL
739
739
  ORDER BY
740
740
  1, 2
741
741
  SQL
742
- execute(query).map { |v| v["columns"] = v["columns"].sub(") WHERE (", " WHERE ").split(", ").map { |c| unquote(c) }; v }
742
+ execute(query, params: tables.to_a).map { |v| v["columns"] = v["columns"].sub(") WHERE (", " WHERE ").split(", ").map { |c| unquote(c) }; v }
743
743
  end
744
744
 
745
745
  def search_path
@@ -758,19 +758,6 @@ module Dexter
758
758
  value.split(".").map { |v| conn.quote_ident(v) }.join(".")
759
759
  end
760
760
 
761
- def quote(value)
762
- if value.is_a?(String)
763
- "'#{quote_string(value)}'"
764
- else
765
- value
766
- end
767
- end
768
-
769
- # from activerecord
770
- def quote_string(s)
771
- s.gsub(/\\/, '\&\&').gsub(/'/, "''")
772
- end
773
-
774
761
  # from activesupport
775
762
  def squish(str)
776
763
  str.to_s.gsub(/\A[[:space:]]+/, "").gsub(/[[:space:]]+\z/, "").gsub(/[[:space:]]+/, " ")
@@ -3,10 +3,6 @@ module Dexter
3
3
  include Logging
4
4
 
5
5
  REGEX = /duration: (\d+\.\d+) ms (statement|execute [^:]+): (.+)/
6
- LINE_SEPERATOR = ": ".freeze
7
- DETAIL_LINE = "DETAIL: ".freeze
8
-
9
- attr_accessor :once
10
6
 
11
7
  def initialize(logfile, collector)
12
8
  @logfile = logfile
@@ -1,27 +1,50 @@
1
1
  module Dexter
2
2
  class PgStatActivityParser < LogParser
3
3
  def perform
4
- queries = {}
4
+ previous_queries = {}
5
5
 
6
- loop do
7
- new_queries = {}
8
- @logfile.stat_activity.each do |row|
9
- new_queries[row["id"]] = row
6
+ 10.times do
7
+ active_queries = {}
8
+ processed_queries = {}
9
+
10
+ stat_activity.each do |row|
11
+ if row["state"] == "active"
12
+ active_queries[row["id"]] = row
13
+ else
14
+ process_entry(row["query"], row["duration_ms"].to_f)
15
+ processed_queries[row["id"]] = true
16
+ end
10
17
  end
11
18
 
12
19
  # store queries after they complete
13
- queries.each do |id, row|
14
- unless new_queries[id]
20
+ previous_queries.each do |id, row|
21
+ if !active_queries[id] && !processed_queries[id]
15
22
  process_entry(row["query"], row["duration_ms"].to_f)
16
23
  end
17
24
  end
18
25
 
19
- queries = new_queries
26
+ previous_queries = active_queries
20
27
 
21
- break if once
22
-
23
- sleep(1)
28
+ sleep(0.1)
24
29
  end
25
30
  end
31
+
32
+ def stat_activity
33
+ sql = <<~SQL
34
+ SELECT
35
+ pid || ':' || COALESCE(query_start, xact_start) AS id,
36
+ query,
37
+ state,
38
+ EXTRACT(EPOCH FROM NOW() - COALESCE(query_start, xact_start)) * 1000.0 AS duration_ms
39
+ FROM
40
+ pg_stat_activity
41
+ WHERE
42
+ datname = current_database()
43
+ AND pid != pg_backend_pid()
44
+ ORDER BY
45
+ 1
46
+ SQL
47
+ @logfile.send(:execute, sql)
48
+ end
26
49
  end
27
50
  end
@@ -11,12 +11,6 @@ module Dexter
11
11
  @log_parser =
12
12
  if @logfile == :pg_stat_activity
13
13
  PgStatActivityParser.new(@indexer, @collector)
14
- elsif @logfile == :log_table
15
- if options[:input_format] == "csv"
16
- CsvLogTableParser.new(@indexer, @collector)
17
- else
18
- StderrLogTableParser.new(@indexer, @collector)
19
- end
20
14
  elsif options[:input_format] == "csv"
21
15
  CsvLogParser.new(logfile, @collector)
22
16
  elsif options[:input_format] == "json"
@@ -29,7 +23,6 @@ module Dexter
29
23
 
30
24
  @starting_interval = 3
31
25
  @interval = options[:interval]
32
- @log_parser.once = options[:once]
33
26
 
34
27
  @mutex = Mutex.new
35
28
  @last_checked_at = {}
@@ -38,7 +31,7 @@ module Dexter
38
31
  end
39
32
 
40
33
  def perform
41
- if [STDIN, :pg_stat_activity, :log_table].include?(@logfile) && !@log_parser.once
34
+ if [STDIN].include?(@logfile)
42
35
  Thread.abort_on_exception = true
43
36
  Thread.new do
44
37
  sleep(@starting_interval)
@@ -1,5 +1,8 @@
1
1
  module Dexter
2
2
  class StderrLogParser < LogParser
3
+ LINE_SEPERATOR = ": ".freeze
4
+ DETAIL_LINE = "DETAIL: ".freeze
5
+
3
6
  def perform
4
7
  process_stderr(@logfile.each_line)
5
8
  end
@@ -1,3 +1,3 @@
1
1
  module Dexter
2
- VERSION = "0.4.3"
2
+ VERSION = "0.5.1"
3
3
  end
data/lib/dexter.rb CHANGED
@@ -21,12 +21,10 @@ require_relative "dexter/version"
21
21
  # parsers
22
22
  require_relative "dexter/log_parser"
23
23
  require_relative "dexter/csv_log_parser"
24
- require_relative "dexter/csv_log_table_parser"
25
24
  require_relative "dexter/json_log_parser"
26
25
  require_relative "dexter/pg_stat_activity_parser"
27
26
  require_relative "dexter/sql_log_parser"
28
27
  require_relative "dexter/stderr_log_parser"
29
- require_relative "dexter/stderr_log_table_parser"
30
28
 
31
29
  module Dexter
32
30
  class Abort < StandardError; end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: pgdexter
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.4.3
4
+ version: 0.5.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Andrew Kane
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2023-03-27 00:00:00.000000000 Z
11
+ date: 2023-05-27 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: pg
@@ -30,14 +30,14 @@ dependencies:
30
30
  requirements:
31
31
  - - "~>"
32
32
  - !ruby/object:Gem::Version
33
- version: '2.1'
33
+ version: '4'
34
34
  type: :runtime
35
35
  prerelease: false
36
36
  version_requirements: !ruby/object:Gem::Requirement
37
37
  requirements:
38
38
  - - "~>"
39
39
  - !ruby/object:Gem::Version
40
- version: '2.1'
40
+ version: '4'
41
41
  - !ruby/object:Gem::Dependency
42
42
  name: slop
43
43
  requirement: !ruby/object:Gem::Requirement
@@ -67,7 +67,6 @@ files:
67
67
  - lib/dexter/client.rb
68
68
  - lib/dexter/collector.rb
69
69
  - lib/dexter/csv_log_parser.rb
70
- - lib/dexter/csv_log_table_parser.rb
71
70
  - lib/dexter/indexer.rb
72
71
  - lib/dexter/json_log_parser.rb
73
72
  - lib/dexter/log_parser.rb
@@ -77,7 +76,6 @@ files:
77
76
  - lib/dexter/query.rb
78
77
  - lib/dexter/sql_log_parser.rb
79
78
  - lib/dexter/stderr_log_parser.rb
80
- - lib/dexter/stderr_log_table_parser.rb
81
79
  - lib/dexter/version.rb
82
80
  homepage: https://github.com/ankane/dexter
83
81
  licenses:
@@ -91,14 +89,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
91
89
  requirements:
92
90
  - - ">="
93
91
  - !ruby/object:Gem::Version
94
- version: '2.5'
92
+ version: '2.7'
95
93
  required_rubygems_version: !ruby/object:Gem::Requirement
96
94
  requirements:
97
95
  - - ">="
98
96
  - !ruby/object:Gem::Version
99
97
  version: '0'
100
98
  requirements: []
101
- rubygems_version: 3.4.6
99
+ rubygems_version: 3.4.10
102
100
  signing_key:
103
101
  specification_version: 4
104
102
  summary: The automatic indexer for Postgres
@@ -1,20 +0,0 @@
1
- module Dexter
2
- class CsvLogTableParser < CsvLogParser
3
- def perform
4
- last_log_time = Time.at(0).iso8601(3)
5
-
6
- loop do
7
- @logfile.csvlog_activity(last_log_time).each do |row|
8
- process_csv_row(row["message"], row["detail"])
9
- last_log_time = row["log_time"]
10
- end
11
-
12
- break
13
-
14
- # possibly enable later
15
- # break if once
16
- # sleep(1)
17
- end
18
- end
19
- end
20
- end
@@ -1,7 +0,0 @@
1
- module Dexter
2
- class StderrLogTableParser < StderrLogParser
3
- def perform
4
- process_stderr(@logfile.stderr_activity.map { |r| r["log_entry"] })
5
- end
6
- end
7
- end