pgdexter 0.4.2 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8e8097505929a24e8c038c7fe69ac667faaa05008a3aa94971501fc4b9f15157
-  data.tar.gz: a08bf061493ed984103ce768af1347d81250c51b4f068f69d3688a1e4fd25514
+  metadata.gz: 65822d0d98c9641efdc3146295e098e09e83348ed109b856670a03d74ed2d70b
+  data.tar.gz: 6d16c9019172e5e69df056ac358588151fa52ae9e8256159547e1842e2f9b97c
 SHA512:
-  metadata.gz: 9c2d99ecbd5fd68e460f25982fe31d1149d9682a7065cfb674d24785f78138ab04d94bcdbf3b81f2d8065468f6531bbaea1e0593bd8fd7ab4dcdb2b061276c1e
-  data.tar.gz: 0e55cc589470760d317e3ba0e895d7ce5e24cd79517cf02c59aaba1caf36eb1f9785725eedc61e7402d480a61870828aaf2b33683f8ca58223f380d518c7df9c
+  metadata.gz: 4991adea5ee65493ea99abe94c19360fc6cc718048784431409abc08fbaf1b1efe3b304dedbd0994d8b66b38294b41ea6400bf5de1f03f09694723a7b709e77c
+  data.tar.gz: 00f47a3efd2de6565dd5f5a3a64caa4b610e282d16620975376876501130a6d0cdbbef9412254b8f3d09db2df7a1f3f62d3a8e581c4fcfecd27ea3224cb6f20f

data/CHANGELOG.md

@@ -1,3 +1,22 @@
+## 0.5.0 (2023-04-18)
+
+- Added support for normalized queries
+- Added `--stdin` option (now required to read from stdin)
+- Added `--enable-hypopg` option (now required to enable HypoPG)
+- Improved output when HypoPG not installed
+- Changed `--pg-stat-activity` to sample 10 times and exit
+- Detect input format based on file extension
+- Dropped support for experimental `--log-table` option
+- Dropped support for Linux packages for Ubuntu 18.04 and Debian 10
+- Dropped support for Ruby < 2.7
+- Dropped support for Postgres < 11
+
+## 0.4.3 (2023-03-26)
+
+- Added experimental `--log-table` option
+- Improved help
+- Require pg_query < 4
+
 ## 0.4.2 (2023-01-29)
 
 - Fixed `--pg-stat-statements` option for Postgres 13+

data/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright (c) 2017-2021 Andrew Kane
+Copyright (c) 2017-2023 Andrew Kane
 
 MIT License
 

data/README.md

@@ -18,15 +18,15 @@ make
 make install # may need sudo
 ```
 
-> Note: If you have issues, make sure `postgresql-server-dev-*` is installed.
+And enable it in databases where you want to use Dexter:
 
-Enable logging for slow queries in your Postgres config file.
-
-```ini
-log_min_duration_statement = 10 # ms
+```sql
+CREATE EXTENSION hypopg;
 ```
 
-And install the command line tool with:
+See the [installation notes](#hypopg-installation-notes) if you run into issues.
+
+Then install the command line tool with:
 
 ```sh
 gem install pgdexter
@@ -36,10 +36,10 @@ The command line tool is also available with [Docker](#docker), [Homebrew](#home
 
 ## How to Use
 
-Dexter needs a connection to your database and a log file to process.
+Dexter needs a connection to your database and a source of queries (like [pg_stat_statements](https://www.postgresql.org/docs/current/pgstatstatements.html)) to process.
 
 ```sh
-tail -F -n +1 <log-file> | dexter <connection-options>
+dexter -d dbname --pg-stat-statements
 ```
 
 This finds slow queries and generates output like:
@@ -53,7 +53,6 @@ Index found: public.movies (title)
 Index found: public.ratings (movie_id)
 Index found: public.ratings (rating)
 Index found: public.ratings (user_id)
-Processing 12 new query fingerprints
 ```
 
 To be safe, Dexter will not create indexes unless you pass the `--create` flag. In this case, you’ll see:
@@ -84,41 +83,78 @@ and connection strings:
 host=localhost port=5432 dbname=mydb
 ```
 
+Always make sure your [connection is secure](https://ankane.org/postgres-sslmode-explained) when connecting to a database over a network you don’t fully trust.
+
 ## Collecting Queries
 
-There are many ways to collect queries. For real-time indexing, pipe your logfile:
+Dexter can collect queries from a number of sources.
+
+- [Query stats](#query-stats)
+- [Live queries](#live-queries)
+- [Log files](#log-file)
+- [SQL files](#sql-files)
+
+### Query Stats
+
+Enable [pg_stat_statements](https://www.postgresql.org/docs/current/pgstatstatements.html) in your database.
+
+```psql
+CREATE EXTENSION pg_stat_statements;
+```
+
+And use:
 
 ```sh
-tail -F -n +1 <log-file> | dexter <connection-options>
+dexter <connection-options> --pg-stat-statements
 ```
 
-Pass a single statement with:
+### Live Queries
+
+Get live queries from the [pg_stat_activity](https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEW) view with:
 
 ```sh
-dexter <connection-options> -s "SELECT * FROM ..."
+dexter <connection-options> --pg-stat-activity
+```
+
+### Log Files
+
+Enable logging for slow queries in your Postgres config file.
+
+```ini
+log_min_duration_statement = 10 # ms
 ```
 
-or pass files:
+And use:
 
 ```sh
-dexter <connection-options> <file1> <file2>
+dexter <connection-options> postgresql.log
 ```
 
-or collect running queries with:
+Supports `stderr`, `csvlog`, and `jsonlog` formats.
+
+For real-time indexing, pipe your logfile:
 
 ```sh
-dexter <connection-options> --pg-stat-activity
+tail -F -n +1 postgresql.log | dexter <connection-options> --stdin
 ```
 
-or use the [pg_stat_statements](https://www.postgresql.org/docs/current/static/pgstatstatements.html) extension:
+And pass `--input-format csvlog` or `--input-format jsonlog` if needed.
+
+### SQL Files
+
+Pass a SQL file with:
 
 ```sh
-dexter <connection-options> --pg-stat-statements
+dexter <connection-options> queries.sql
 ```
 
-> Note: Logs or running queries are highly preferred over pg_stat_statements, as pg_stat_statements often doesn’t store enough information to optimize queries.
+Pass a single query with:
+
+```sh
+dexter <connection-options> -s "SELECT * FROM ..."
+```
 
-### Collection Options
+## Collection Options
 
 To prevent one-off queries from being indexed, specify a minimum number of calls before a query is considered for indexing
 
@@ -132,12 +168,6 @@ You can do the same for total time a query has run
 dexter --min-time 10 # minutes
 ```
 
-Specify the format
-
-```sh
-dexter --input-format csv
-```
-
 When streaming logs, specify the time to wait between processing queries
 
 ```sh
@@ -146,16 +176,22 @@ dexter --interval 60 # seconds
 
 ## Examples
 
-Ubuntu with PostgreSQL 12
+Postgres package on Ubuntu 22.04
 
 ```sh
-tail -F -n +1 /var/log/postgresql/postgresql-12-main.log | sudo -u postgres dexter dbname
+sudo -u postgres dexter -d dbname /var/log/postgresql/postgresql-14-main.log
 ```
 
-Homebrew on Mac
+Homebrew Postgres on Mac ARM
 
 ```sh
-tail -F -n +1 /usr/local/var/postgres/server.log | dexter dbname
+dexter -d dbname /opt/homebrew/var/log/postgresql@14.log
+```
+
+Homebrew Postgres on Mac x86-64
+
+```sh
+dexter -d dbname /usr/local/var/log/postgresql@14.log
 ```
 
 ## Analyze
@@ -198,6 +234,30 @@ For other providers, see [this guide](guides/Hosted-Postgres.md). To request a n
 - Google Cloud SQL - vote or comment on [this page](https://issuetracker.google.com/issues/69250435)
 - DigitalOcean Managed Databases - vote or comment on [this page](https://ideas.digitalocean.com/app-framework-services/p/support-hypopg-for-postgres)
 
+## HypoPG Installation Notes
+
+### Postgres Location
+
+If your machine has multiple Postgres installations, specify the path to [pg_config](https://www.postgresql.org/docs/current/app-pgconfig.html) with:
+
+```sh
+export PG_CONFIG=/Applications/Postgres.app/Contents/Versions/latest/bin/pg_config
+```
+
+Then re-run the installation instructions (run `make clean` before `make` if needed)
+
+### Missing Header
+
+If compilation fails with `fatal error: postgres.h: No such file or directory`, make sure Postgres development files are installed on the server.
+
+For Ubuntu and Debian, use:
+
+```sh
+sudo apt-get install postgresql-server-dev-15
+```
+
+Note: Replace `15` with your Postgres server version
+
 ## Additional Installation Methods
 
 ### Docker
@@ -214,19 +274,19 @@ And run it with:
 docker run -ti ankane/dexter <connection-options>
 ```
 
-For databases on the host machine, use `host.docker.internal` as the hostname (on Linux, this requires Docker 20.04 and `--add-host=host.docker.internal:host-gateway`).
+For databases on the host machine, use `host.docker.internal` as the hostname (on Linux, this requires Docker 20.04+ and `--add-host=host.docker.internal:host-gateway`).
 
 ### Homebrew
 
 With Homebrew, you can use:
 
 ```sh
-brew install ankane/brew/dexter
+brew install dexter
 ```
 
 ## Future Work
 
-[Here are some ideas](https://github.com/ankane/dexter/issues/1)
+[Here are some ideas](https://github.com/ankane/dexter/issues/45)
 
 ## Upgrading
 
@@ -243,9 +303,19 @@ gem install specific_install
 gem specific_install https://github.com/ankane/dexter.git
 ```
 
+## Upgrade Notes
+
+### 0.5.0
+
+The `--stdin` option is now required to read queries from stdin.
+
+```sh
+tail -F -n +1 postgresql.log | dexter <connection-options> --stdin
+```
+
 ## Thanks
 
-This software wouldn’t be possible without [HypoPG](https://github.com/HypoPG/hypopg), which allows you to create hypothetical indexes, and [pg_query](https://github.com/lfittl/pg_query), which allows you to parse and fingerprint queries. A big thanks to Dalibo and Lukas Fittl respectively.
+This software wouldn’t be possible without [HypoPG](https://github.com/HypoPG/hypopg), which allows you to create hypothetical indexes, and [pg_query](https://github.com/lfittl/pg_query), which allows you to parse and fingerprint queries. A big thanks to Dalibo and Lukas Fittl respectively. Also, thanks to YugabyteDB for [this article](https://dev.to/yugabyte/explain-from-pgstatstatements-normalized-queries-how-to-always-get-the-generic-plan-in--5cfi) on how to explain normalized queries.
 
 ## Research
 

data/lib/dexter/client.rb

@@ -7,7 +7,7 @@ module Dexter
 
     def self.start
       Dexter::Client.new(ARGV).perform
-    rescue Dexter::Abort, PG::UndefinedFile => e
+    rescue Dexter::Abort, PG::UndefinedFile, PG::FeatureNotSupported => e
       abort colorize(e.message.strip, :red)
     end
 
@@ -29,9 +29,15 @@ module Dexter
         Processor.new(:pg_stat_activity, options).perform
       elsif arguments.any?
         ARGV.replace(arguments)
+        if !options[:input_format]
+          ext = ARGV.map { |v| File.extname(v) }.uniq
+          options[:input_format] = ext.first[1..-1] if ext.size == 1
+        end
         Processor.new(ARGF, options).perform
-      else
+      elsif options[:stdin]
         Processor.new(STDIN, options).perform
+      else
+        raise Dexter::Abort, "Specify a source of queries: --pg-stat-statements, --pg-stat-activity, --stdin, or a path"
       end
     end
 
@@ -40,23 +46,45 @@ module Dexter
         o.banner = %(Usage:
     dexter [options])
         o.separator ""
-        o.separator "Options:"
+
+        o.separator "Input options:"
+        o.string "--input-format", "input format"
+        o.boolean "--pg-stat-activity", "use pg_stat_activity", default: false
+        o.boolean "--pg-stat-statements", "use pg_stat_statements", default: false, help: false
+        o.boolean "--stdin", "use stdin", default: false
+        o.string "-s", "--statement", "process a single statement"
+        o.separator ""
+
+        o.separator "Connection options:"
+        o.string "-d", "--dbname", "database name"
+        o.string "-h", "--host", "database host"
+        o.integer "-p", "--port", "database port"
+        o.string "-U", "--username", "database user"
+        o.separator ""
+
+        o.separator "Processing options:"
+        o.integer "--interval", "time to wait between processing queries, in seconds", default: 60
+        o.float "--min-calls", "only process queries that have been called a certain number of times", default: 0
+        o.float "--min-time", "only process queries that have consumed a certain amount of DB time, in minutes", default: 0
+        o.separator ""
+
+        o.separator "Indexing options:"
         o.boolean "--analyze", "analyze tables that haven't been analyzed in the past hour", default: false
         o.boolean "--create", "create indexes", default: false
+        o.boolean "--enable-hypopg", "enable the HypoPG extension", default: false
         o.array "--exclude", "prevent specific tables from being indexed"
         o.string "--include", "only include specific tables"
-        o.string "--input-format", "input format", default: "stderr"
-        o.integer "--interval", "time to wait between processing queries, in seconds", default: 60
+        o.integer "--min-cost-savings-pct", default: 50, help: false
+        o.string "--tablespace", "tablespace to create indexes"
+        o.separator ""
+
+        o.separator "Logging options:"
         o.boolean "--log-explain", "log explain", default: false, help: false
         o.string "--log-level", "log level", default: "info"
         o.boolean "--log-sql", "log sql", default: false
-        o.float "--min-calls", "only process queries that have been called a certain number of times", default: 0
-        o.float "--min-time", "only process queries that have consumed a certain amount of DB time, in minutes", default: 0
-        o.integer "--min-cost-savings-pct", default: 50, help: false
-        o.boolean "--pg-stat-activity", "use pg_stat_activity", default: false, help: false
-        o.boolean "--pg-stat-statements", "use pg_stat_statements", default: false, help: false
-        o.string "-s", "--statement", "process a single statement"
-        o.string "--tablespace", "tablespace to create indexes"
+        o.separator ""
+
+        o.separator "Other options:"
         o.on "-v", "--version", "print the version" do
           log Dexter::VERSION
           exit
@@ -65,12 +93,6 @@ module Dexter
           log o
           exit
         end
-        o.separator ""
-        o.separator "Connection options:"
-        o.string "-d", "--dbname", "database name"
-        o.string "-h", "--host", "database host"
-        o.integer "-p", "--port", "database port"
-        o.string "-U", "--username", "database user"
       end
 
       arguments = opts.arguments

data/lib/dexter/collector.rb

@@ -8,7 +8,7 @@ module Dexter
       @min_calls = options[:min_calls]
     end
 
-    def add(query, duration)
+    def add(query, total_time, calls = 1)
       fingerprint =
         begin
           PgQuery.fingerprint(query)
@@ -19,8 +19,8 @@ module Dexter
       return unless fingerprint
 
       @top_queries[fingerprint] ||= {calls: 0, total_time: 0}
-      @top_queries[fingerprint][:calls] += 1
-      @top_queries[fingerprint][:total_time] += duration
+      @top_queries[fingerprint][:calls] += calls
+      @top_queries[fingerprint][:total_time] += total_time
       @top_queries[fingerprint][:query] = query
       @mutex.synchronize do
         @new_queries << fingerprint

data/lib/dexter/csv_log_parser.rb

@@ -1,22 +1,24 @@
-require "csv"
-
 module Dexter
   class CsvLogParser < LogParser
     FIRST_LINE_REGEX = /\A.+/
 
     def perform
       CSV.new(@logfile.to_io).each do |row|
-        if (m = REGEX.match(row[13]))
-          # replace first line with match
-          # needed for multiline queries
-          active_line = row[13].sub(FIRST_LINE_REGEX, m[3])
-
-          add_parameters(active_line, row[14]) if row[14]
-          process_entry(active_line, m[1].to_f)
-        end
+        process_csv_row(row[13], row[14])
       end
     rescue CSV::MalformedCSVError => e
       raise Dexter::Abort, "ERROR: #{e.message}"
     end
+
+    def process_csv_row(message, detail)
+      if (m = REGEX.match(message))
+        # replace first line with match
+        # needed for multiline queries
+        active_line = message.sub(FIRST_LINE_REGEX, m[3])
+
+        add_parameters(active_line, detail) if detail
+        process_entry(active_line, m[1].to_f)
+      end
+    end
   end
 end

data/lib/dexter/indexer.rb

@@ -17,7 +17,12 @@ module Dexter
       @options = options
       @mutex = Mutex.new
 
-      create_extension unless extension_exists?
+      if server_version_num < 110000
+        raise Dexter::Abort, "This version of Dexter requires Postgres 11+"
+      end
+
+      check_extension
+
       execute("SET lock_timeout = '5s'")
     end
 
@@ -27,23 +32,6 @@ module Dexter
       process_queries(queries)
     end
 
-    def stat_activity
-      execute <<-SQL
-        SELECT
-          pid || ':' || COALESCE(query_start, xact_start) AS id,
-          query,
-          EXTRACT(EPOCH FROM NOW() - COALESCE(query_start, xact_start)) * 1000.0 AS duration_ms
-        FROM
-          pg_stat_activity
-        WHERE
-          datname = current_database()
-          AND state = 'active'
-          AND pid != pg_backend_pid()
-        ORDER BY
-          1
-      SQL
-    end
-
     def process_queries(queries)
       # reset hypothetical indexes
       reset_hypothetical_indexes
@@ -119,19 +107,20 @@ module Dexter
 
     private
 
-    def create_extension
-      execute("SET client_min_messages = warning")
-      begin
-        execute("CREATE EXTENSION IF NOT EXISTS hypopg")
-      rescue PG::UndefinedFile
+    def check_extension
+      extension = execute("SELECT installed_version FROM pg_available_extensions WHERE name = 'hypopg'").first
+
+      if extension.nil?
         raise Dexter::Abort, "Install HypoPG first: https://github.com/ankane/dexter#installation"
-      rescue PG::InsufficientPrivilege
-        raise Dexter::Abort, "Use a superuser to run: CREATE EXTENSION hypopg"
       end
-    end
 
-    def extension_exists?
-      execute("SELECT * FROM pg_available_extensions WHERE name = 'hypopg' AND installed_version IS NOT NULL").any?
+      if extension["installed_version"].nil?
+        if @options[:enable_hypopg]
+          execute("CREATE EXTENSION hypopg")
+        else
+          raise Dexter::Abort, "Run `CREATE EXTENSION hypopg` or pass --enable-hypopg"
+        end
+      end
     end
 
     def reset_hypothetical_indexes
@@ -141,7 +130,7 @@ module Dexter
     def analyze_tables(tables)
       tables = tables.to_a.sort
 
-      analyze_stats = execute <<-SQL
+      query = <<~SQL
         SELECT
           schemaname || '.' || relname AS table,
           last_analyze,
@@ -149,8 +138,9 @@ module Dexter
         FROM
           pg_stat_user_tables
         WHERE
-          schemaname || '.' || relname IN (#{tables.map { |t| quote(t) }.join(", ")})
+          schemaname || '.' || relname IN (#{tables.size.times.map { |i| "$#{i + 1}" }.join(", ")})
       SQL
+      analyze_stats = execute(query, params: tables.to_a)
 
       last_analyzed = {}
       analyze_stats.each do |stats|
@@ -181,10 +171,6 @@ module Dexter
         end
         begin
           query.plans << plan(query.statement)
-          if @log_explain
-            # Pass format to prevent ANALYZE
-            puts execute("EXPLAIN (FORMAT TEXT) #{safe_statement(query.statement)}", pretty: false).map { |r| r["QUERY PLAN"] }.join("\n")
-          end
         rescue PG::Error, JSON::NestingError => e
           if @log_explain
             log e.message
@@ -214,7 +200,7 @@ module Dexter
           find_columns(query.tree).each do |col|
             last_col = col["fields"].last
             if last_col["String"]
-              possible_columns << last_col["String"]["str"]
+              possible_columns << last_col["String"]["sval"]
             end
           end
         end
@@ -510,7 +496,7 @@ module Dexter
     def conn
       @conn ||= begin
         # set connect timeout if none set
-        ENV["PGCONNECT_TIMEOUT"] ||= "2"
+        ENV["PGCONNECT_TIMEOUT"] ||= "3"
 
         if @options[:dbname] =~ /\Apostgres(ql)?:\/\//
           config = @options[:dbname]
@@ -529,7 +515,7 @@ module Dexter
       raise Dexter::Abort, e.message
     end
 
-    def execute(query, pretty: true)
+    def execute(query, pretty: true, params: [])
       # use exec_params instead of exec for security
       #
       # Unlike PQexec, PQexecParams allows at most one SQL command in the given string.
@@ -538,16 +524,56 @@ module Dexter
       # as an extra defense against SQL-injection attacks.
       # https://www.postgresql.org/docs/current/static/libpq-exec.html
       query = squish(query) if pretty
-      log colorize("[sql] #{query}", :cyan) if @log_sql
+      log colorize("[sql] #{query}#{params.any? ? " /*#{params.to_json}*/" : ""}", :cyan) if @log_sql
 
       @mutex.synchronize do
-        conn.exec_params(query, []).to_a
+        conn.exec_params("#{query} /*dexter*/", params).to_a
       end
     end
 
     def plan(query)
+      prepared = false
+      transaction = false
+
+      # try to EXPLAIN normalized queries
+      # https://dev.to/yugabyte/explain-from-pgstatstatements-normalized-queries-how-to-always-get-the-generic-plan-in--5cfi
+      explain_normalized = query.include?("$1")
+      if explain_normalized
+        prepared_name = "dexter_prepared"
+        execute("PREPARE #{prepared_name} AS #{safe_statement(query)}", pretty: false)
+        prepared = true
+        params = execute("SELECT array_length(parameter_types, 1) AS params FROM pg_prepared_statements WHERE name = $1", params: [prepared_name]).first["params"].to_i
+        query = "EXECUTE #{prepared_name}(#{params.times.map { "NULL" }.join(", ")})"
+
+        execute("BEGIN")
+        transaction = true
+
+        if server_version_num >= 120000
+          execute("SET LOCAL plan_cache_mode = force_generic_plan")
+        else
+          execute("SET LOCAL cpu_operator_cost = 1e42")
+          5.times do
+            execute("EXPLAIN (FORMAT JSON) #{safe_statement(query)}", pretty: false)
+          end
+          execute("ROLLBACK")
+          execute("BEGIN")
+        end
+      end
+
       # strip semi-colons as another measure of defense
-      JSON.parse(execute("EXPLAIN (FORMAT JSON) #{safe_statement(query)}", pretty: false).first["QUERY PLAN"], max_nesting: 1000).first["Plan"]
+      plan = JSON.parse(execute("EXPLAIN (FORMAT JSON) #{safe_statement(query)}", pretty: false).first["QUERY PLAN"], max_nesting: 1000).first["Plan"]
+
+      if @log_explain
+        # Pass format to prevent ANALYZE
+        puts execute("EXPLAIN (FORMAT TEXT) #{safe_statement(query)}", pretty: false).map { |r| r["QUERY PLAN"] }.join("\n")
+      end
+
+      plan
+    ensure
+      if explain_normalized
+        execute("ROLLBACK") if transaction
+        execute("DEALLOCATE #{prepared_name}") if prepared
+      end
     end
 
     # TODO for multicolumn indexes, use ordering
@@ -565,7 +591,7 @@ module Dexter
     end
 
     def database_tables
-      result = execute <<-SQL
+      result = execute <<~SQL
         SELECT
           table_schema || '.' || table_name AS table_name
         FROM
@@ -577,17 +603,13 @@ module Dexter
     end
 
     def materialized_views
-      if server_version_num >= 90300
-        result = execute <<-SQL
-          SELECT
-            schemaname || '.' || matviewname AS table_name
-          FROM
-            pg_matviews
-        SQL
-        result.map { |r| r["table_name"] }
-      else
-        []
-      end
+      result = execute <<~SQL
+        SELECT
+          schemaname || '.' || matviewname AS table_name
+        FROM
+          pg_matviews
+      SQL
+      result.map { |r| r["table_name"] }
     end
 
     def server_version_num
@@ -595,7 +617,7 @@ module Dexter
     end
 
     def database_view_tables
-      result = execute <<-SQL
+      result = execute <<~SQL
         SELECT
           schemaname || '.' || viewname AS table_name,
           definition
@@ -621,7 +643,7 @@ module Dexter
 
     def stat_statements
       total_time = server_version_num >= 130000 ? "(total_plan_time + total_exec_time)" : "total_time"
-      result = execute <<-SQL
+      sql = <<~SQL
         SELECT
           DISTINCT query
         FROM
@@ -630,18 +652,18 @@ module Dexter
           pg_database ON pg_database.oid = pg_stat_statements.dbid
         WHERE
           datname = current_database()
-          AND #{total_time} >= #{@min_time * 60000}
-          AND calls >= #{@min_calls}
+          AND #{total_time} >= \$1
+          AND calls >= \$2
         ORDER BY
           1
       SQL
-      result.map { |q| q["query"] }
+      execute(sql, params: [@min_time * 60000, @min_calls]).map { |q| q["query"] }
     end
 
     def with_advisory_lock
       lock_id = 123456
       first_time = true
-      while execute("SELECT pg_try_advisory_lock(#{lock_id})").first["pg_try_advisory_lock"] != "t"
+      while execute("SELECT pg_try_advisory_lock($1)", params: [lock_id]).first["pg_try_advisory_lock"] != "t"
         if first_time
           log "Waiting for lock..."
           first_time = false
@@ -650,16 +672,19 @@ module Dexter
       end
       yield
     ensure
-      with_min_messages("error") do
-        execute("SELECT pg_advisory_unlock(#{lock_id})")
+      suppress_messages do
+        execute("SELECT pg_advisory_unlock($1)", params: [lock_id])
       end
     end
 
-    def with_min_messages(value)
-      execute("SET client_min_messages = #{quote(value)}")
+    def suppress_messages
+      conn.set_notice_processor do |message|
+        # do nothing
+      end
       yield
     ensure
-      execute("SET client_min_messages = warning")
+      # clear notice processor
+      conn.set_notice_processor
     end
 
     def index_exists?(index)
@@ -667,7 +692,7 @@ module Dexter
     end
 
     def columns(tables)
-      columns = execute <<-SQL
+      query = <<~SQL
         SELECT
           s.nspname || '.' || t.relname AS table_name,
           a.attname AS column_name,
@@ -677,16 +702,16 @@ module Dexter
           JOIN pg_namespace s on t.relnamespace = s.oid
         WHERE a.attnum > 0
           AND NOT a.attisdropped
-          AND s.nspname || '.' || t.relname IN (#{tables.map { |t| quote(t) }.join(", ")})
+          AND s.nspname || '.' || t.relname IN (#{tables.size.times.map { |i| "$#{i + 1}" }.join(", ")})
         ORDER BY
           1, 2
       SQL
-
+      columns = execute(query, params: tables.to_a)
       columns.map { |v| {table: v["table_name"], column: v["column_name"], type: v["data_type"]} }
     end
 
     def indexes(tables)
-      execute(<<-SQL
+      query = <<~SQL
         SELECT
           schemaname || '.' || t.relname AS table,
           ix.relname AS name,
@@ -701,14 +726,14 @@ module Dexter
         LEFT JOIN
           pg_stat_user_indexes ui ON ui.indexrelid = i.indexrelid
         WHERE
-          schemaname || '.' || t.relname IN (#{tables.map { |t| quote(t) }.join(", ")}) AND
+          schemaname || '.' || t.relname IN (#{tables.size.times.map { |i| "$#{i + 1}" }.join(", ")}) AND
           indisvalid = 't' AND
           indexprs IS NULL AND
           indpred IS NULL
         ORDER BY
           1, 2
       SQL
-      ).map { |v| v["columns"] = v["columns"].sub(") WHERE (", " WHERE ").split(", ").map { |c| unquote(c) }; v }
+      execute(query, params: tables.to_a).map { |v| v["columns"] = v["columns"].sub(") WHERE (", " WHERE ").split(", ").map { |c| unquote(c) }; v }
     end
 
     def search_path
@@ -727,19 +752,6 @@ module Dexter
       value.split(".").map { |v| conn.quote_ident(v) }.join(".")
     end
 
-    def quote(value)
-      if value.is_a?(String)
-        "'#{quote_string(value)}'"
-      else
-        value
-      end
-    end
-
-    # from activerecord
-    def quote_string(s)
-      s.gsub(/\\/, '\&\&').gsub(/'/, "''")
-    end
-
     # from activesupport
     def squish(str)
       str.to_s.gsub(/\A[[:space:]]+/, "").gsub(/[[:space:]]+\z/, "").gsub(/[[:space:]]+/, " ")

data/lib/dexter/json_log_parser.rb

@@ -1,5 +1,3 @@
-require "json"
-
 module Dexter
   class JsonLogParser < LogParser
     FIRST_LINE_REGEX = /\A.+/

data/lib/dexter/log_parser.rb

@@ -3,38 +3,12 @@ module Dexter
     include Logging
 
     REGEX = /duration: (\d+\.\d+) ms  (statement|execute [^:]+): (.+)/
-    LINE_SEPERATOR = ":  ".freeze
-    DETAIL_LINE = "DETAIL:  ".freeze
 
     def initialize(logfile, collector)
       @logfile = logfile
       @collector = collector
     end
 
-    def perform
-      active_line = nil
-      duration = nil
-
-      @logfile.each_line do |line|
-        if active_line
-          if line.include?(DETAIL_LINE)
-            add_parameters(active_line, line.chomp.split(DETAIL_LINE)[1])
-          elsif line.include?(LINE_SEPERATOR)
-            process_entry(active_line, duration)
-            active_line = nil
-          else
-            active_line << line
-          end
-        end
-
-        if !active_line && (m = REGEX.match(line.chomp))
-          duration = m[1].to_f
-          active_line = m[3]
-        end
-      end
-      process_entry(active_line, duration) if active_line
-    end
-
     private
 
     def process_entry(query, duration)

data/lib/dexter/pg_stat_activity_parser.rb

@@ -1,25 +1,50 @@
 module Dexter
   class PgStatActivityParser < LogParser
     def perform
-      queries = {}
+      previous_queries = {}
 
-      loop do
-        new_queries = {}
-        @logfile.stat_activity.each do |row|
-          new_queries[row["id"]] = row
+      10.times do
+        active_queries = {}
+        processed_queries = {}
+
+        stat_activity.each do |row|
+          if row["state"] == "active"
+            active_queries[row["id"]] = row
+          else
+            process_entry(row["query"], row["duration_ms"].to_f)
+            processed_queries[row["id"]] = true
+          end
         end
 
         # store queries after they complete
-        queries.each do |id, row|
-          unless new_queries[id]
+        previous_queries.each do |id, row|
+          if !active_queries[id] && !processed_queries[id]
             process_entry(row["query"], row["duration_ms"].to_f)
           end
         end
 
-        queries = new_queries
+        previous_queries = active_queries
 
-        sleep(1)
+        sleep(0.1)
       end
     end
+
+    def stat_activity
+      sql = <<~SQL
+        SELECT
+          pid || ':' || COALESCE(query_start, xact_start) AS id,
+          query,
+          state,
+          EXTRACT(EPOCH FROM NOW() - COALESCE(query_start, xact_start)) * 1000.0 AS duration_ms
+        FROM
+          pg_stat_activity
+        WHERE
+          datname = current_database()
+          AND pid != pg_backend_pid()
+        ORDER BY
+          1
+      SQL
+      @logfile.send(:execute, sql)
+    end
   end
 end

data/lib/dexter/processor.rb

@@ -18,7 +18,7 @@ module Dexter
         elsif options[:input_format] == "sql"
           SqlLogParser.new(logfile, @collector)
         else
-          LogParser.new(logfile, @collector)
+          StderrLogParser.new(logfile, @collector)
         end
 
       @starting_interval = 3
@@ -31,7 +31,7 @@ module Dexter
     end
 
     def perform
-      if [STDIN, :pg_stat_activity].include?(@logfile)
+      if [STDIN].include?(@logfile)
         Thread.abort_on_exception = true
         Thread.new do
           sleep(@starting_interval)

data/lib/dexter/stderr_log_parser.rb

@@ -0,0 +1,34 @@
+module Dexter
+  class StderrLogParser < LogParser
+    LINE_SEPERATOR = ":  ".freeze
+    DETAIL_LINE = "DETAIL:  ".freeze
+
+    def perform
+      process_stderr(@logfile.each_line)
+    end
+
+    def process_stderr(rows)
+      active_line = nil
+      duration = nil
+
+      rows.each do |line|
+        if active_line
+          if line.include?(DETAIL_LINE)
+            add_parameters(active_line, line.chomp.split(DETAIL_LINE)[1])
+          elsif line.include?(LINE_SEPERATOR)
+            process_entry(active_line, duration)
+            active_line = nil
+          else
+            active_line << line
+          end
+        end
+
+        if !active_line && (m = REGEX.match(line.chomp))
+          duration = m[1].to_f
+          active_line = m[3]
+        end
+      end
+      process_entry(active_line, duration) if active_line
+    end
+  end
+end

data/lib/dexter/version.rb

@@ -1,3 +1,3 @@
 module Dexter
-  VERSION = "0.4.2"
+  VERSION = "0.5.0"
 end

data/lib/dexter.rb

@@ -4,23 +4,27 @@ require "pg_query"
 require "slop"
 
 # stdlib
+require "csv"
 require "json"
 require "set"
 require "time"
 
 # modules
-require "dexter/version"
-require "dexter/logging"
-require "dexter/client"
-require "dexter/collector"
-require "dexter/indexer"
-require "dexter/log_parser"
-require "dexter/csv_log_parser"
-require "dexter/json_log_parser"
-require "dexter/pg_stat_activity_parser"
-require "dexter/sql_log_parser"
-require "dexter/processor"
-require "dexter/query"
+require_relative "dexter/logging"
+require_relative "dexter/client"
+require_relative "dexter/collector"
+require_relative "dexter/indexer"
+require_relative "dexter/processor"
+require_relative "dexter/query"
+require_relative "dexter/version"
+
+# parsers
+require_relative "dexter/log_parser"
+require_relative "dexter/csv_log_parser"
+require_relative "dexter/json_log_parser"
+require_relative "dexter/pg_stat_activity_parser"
+require_relative "dexter/sql_log_parser"
+require_relative "dexter/stderr_log_parser"
 
 module Dexter
   class Abort < StandardError; end

metadata

@@ -1,57 +1,57 @@
 --- !ruby/object:Gem::Specification
 name: pgdexter
 version: !ruby/object:Gem::Version
-  version: 0.4.2
+  version: 0.5.0
 platform: ruby
 authors:
 - Andrew Kane
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2023-01-30 00:00:00.000000000 Z
+date: 2023-04-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: slop
+  name: pg
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 4.8.2
+        version: 0.18.2
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 4.8.2
+        version: 0.18.2
 - !ruby/object:Gem::Dependency
-  name: pg
+  name: pg_query
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.18.2
+        version: '4'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.18.2
+        version: '4'
 - !ruby/object:Gem::Dependency
-  name: pg_query
+  name: slop
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '2.1'
+        version: 4.10.1
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '2.1'
+        version: 4.10.1
 description:
 email: andrew@ankane.org
 executables:
@@ -75,6 +75,7 @@ files:
 - lib/dexter/processor.rb
 - lib/dexter/query.rb
 - lib/dexter/sql_log_parser.rb
+- lib/dexter/stderr_log_parser.rb
 - lib/dexter/version.rb
 homepage: https://github.com/ankane/dexter
 licenses:
@@ -88,14 +89,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '2.5'
+      version: '2.7'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.4.1
+rubygems_version: 3.4.10
 signing_key:
 specification_version: 4
 summary: The automatic indexer for Postgres