pgdexter 0.4.3 → 0.5.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d2b2528c0ff8201ce676b1a4ca7ec11de52013bfc608fab1b574122a31dd5ae9
-  data.tar.gz: 570f24c171e02425a5c492068ccedf48e318194333d825cc384ce6cfeeb6c30a
+  metadata.gz: c68746f6134f6603c5549b561886b45ee3432df5d1e07e65a5907559901d333a
+  data.tar.gz: 699c2744f3e2c9a79f8fa9ca6ae7a77586421243cbf7a1f92642f76724cc0cb0
 SHA512:
-  metadata.gz: e0975c42916c22b0d8f0b12f0144865370372771526ce32574a4602918aadd786b8ab39cb092ad28590c61c86820c51dcac4f7a41cc5930412868ce2a0590498
-  data.tar.gz: e91d272a0d6bc737634615118a1a33cbbf2ac2fe957aabb17edae9ae42fa17f596052e83a9caa80b71effa6b20316b12e551bd8aea7734648b3210bdb2bf00ec
+  metadata.gz: 343bc52539ef09541fd0774667034ea02fb88b0e4f820e52c86d77df46d059c253576c22272b91a407b764965f66c554278154da6ab923249ce9979ab3a5aed0
+  data.tar.gz: a9b86931b4fc58d2b89c87f46f5008dfd47274cc39f096ec2b5143bea8858e29a786e387f9ccc29fe74475b5b0317e9d348d36f2a686cb19457ee79179a270f5

data/CHANGELOG.md

@@ -1,3 +1,20 @@
+## 0.5.1 (2023-05-27)
+
+- Fixed `JSON::NestingError`
+
+## 0.5.0 (2023-04-18)
+
+- Added support for normalized queries
+- Added `--stdin` option (now required to read from stdin)
+- Added `--enable-hypopg` option (now required to enable HypoPG)
+- Improved output when HypoPG not installed
+- Changed `--pg-stat-activity` to sample 10 times and exit
+- Detect input format based on file extension
+- Dropped support for experimental `--log-table` option
+- Dropped support for Linux packages for Ubuntu 18.04 and Debian 10
+- Dropped support for Ruby < 2.7
+- Dropped support for Postgres < 11
+
 ## 0.4.3 (2023-03-26)
 
 - Added experimental `--log-table` option

data/README.md

@@ -12,21 +12,21 @@ First, install [HypoPG](https://github.com/HypoPG/hypopg) on your database serve
 
 ```sh
 cd /tmp
-curl -L https://github.com/HypoPG/hypopg/archive/1.3.1.tar.gz | tar xz
-cd hypopg-1.3.1
+curl -L https://github.com/HypoPG/hypopg/archive/1.4.0.tar.gz | tar xz
+cd hypopg-1.4.0
 make
 make install # may need sudo
 ```
 
-> Note: If you have issues, make sure `postgresql-server-dev-*` is installed.
+And enable it in databases where you want to use Dexter:
 
-Enable logging for slow queries in your Postgres config file.
-
-```ini
-log_min_duration_statement = 10 # ms
+```sql
+CREATE EXTENSION hypopg;
 ```
 
-And install the command line tool with:
+See the [installation notes](#hypopg-installation-notes) if you run into issues.
+
+Then install the command line tool with:
 
 ```sh
 gem install pgdexter
@@ -36,10 +36,10 @@ The command line tool is also available with [Docker](#docker), [Homebrew](#home
 
 ## How to Use
 
-Dexter needs a connection to your database and a log file to process.
+Dexter needs a connection to your database and a source of queries (like [pg_stat_statements](https://www.postgresql.org/docs/current/pgstatstatements.html)) to process.
 
 ```sh
-tail -F -n +1 <log-file> | dexter <connection-options>
+dexter -d dbname --pg-stat-statements
 ```
 
 This finds slow queries and generates output like:
@@ -53,7 +53,6 @@ Index found: public.movies (title)
 Index found: public.ratings (movie_id)
 Index found: public.ratings (rating)
 Index found: public.ratings (user_id)
-Processing 12 new query fingerprints
 ```
 
 To be safe, Dexter will not create indexes unless you pass the `--create` flag. In this case, you’ll see:
@@ -84,70 +83,95 @@ and connection strings:
 host=localhost port=5432 dbname=mydb
 ```
 
+Always make sure your [connection is secure](https://ankane.org/postgres-sslmode-explained) when connecting to a database over a network you don’t fully trust.
+
 ## Collecting Queries
 
-There are many ways to collect queries. For real-time indexing, pipe your logfile:
+Dexter can collect queries from a number of sources.
 
-```sh
-tail -F -n +1 <log-file> | dexter <connection-options>
-```
+- [Query stats](#query-stats)
+- [Live queries](#live-queries)
+- [Log files](#log-file)
+- [SQL files](#sql-files)
 
-Pass a single statement with:
+### Query Stats
 
-```sh
-dexter <connection-options> -s "SELECT * FROM ..."
+Enable [pg_stat_statements](https://www.postgresql.org/docs/current/pgstatstatements.html) in your database.
+
+```psql
+CREATE EXTENSION pg_stat_statements;
 ```
 
-or pass files:
+And use:
 
 ```sh
-dexter <connection-options> <file1> <file2>
+dexter <connection-options> --pg-stat-statements
 ```
 
-or collect running queries with:
+### Live Queries
+
+Get live queries from the [pg_stat_activity](https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEW) view with:
 
 ```sh
 dexter <connection-options> --pg-stat-activity
 ```
 
-### Collection Options
+### Log Files
 
-To prevent one-off queries from being indexed, specify a minimum number of calls before a query is considered for indexing
+Enable logging for slow queries in your Postgres config file.
+
+```ini
+log_min_duration_statement = 10 # ms
+```
+
+And use:
 
 ```sh
-dexter --min-calls 100
+dexter <connection-options> postgresql.log
 ```
 
-You can do the same for total time a query has run
+Supports `stderr`, `csvlog`, and `jsonlog` formats.
+
+For real-time indexing, pipe your logfile:
 
 ```sh
-dexter --min-time 10 # minutes
+tail -F -n +1 postgresql.log | dexter <connection-options> --stdin
 ```
 
-Specify the format
+And pass `--input-format csvlog` or `--input-format jsonlog` if needed.
+
+### SQL Files
+
+Pass a SQL file with:
 
 ```sh
-dexter --input-format csv
+dexter <connection-options> queries.sql
 ```
 
-When streaming logs, specify the time to wait between processing queries
+Pass a single query with:
 
 ```sh
-dexter --interval 60 # seconds
+dexter <connection-options> -s "SELECT * FROM ..."
 ```
 
-## Examples
+## Collection Options
+
+To prevent one-off queries from being indexed, specify a minimum number of calls before a query is considered for indexing
+
+```sh
+dexter --min-calls 100
+```
 
-Ubuntu with PostgreSQL 12
+You can do the same for total time a query has run
 
 ```sh
-tail -F -n +1 /var/log/postgresql/postgresql-12-main.log | sudo -u postgres dexter dbname
+dexter --min-time 10 # minutes
 ```
 
-Homebrew on Mac
+When streaming logs, specify the time to wait between processing queries
 
 ```sh
-tail -F -n +1 /usr/local/var/postgres/server.log | dexter dbname
+dexter --interval 60 # seconds
 ```
 
 ## Analyze
@@ -186,9 +210,32 @@ The `hypopg` extension, which Dexter needs to run, is available on [these provid
 
 For other providers, see [this guide](guides/Hosted-Postgres.md). To request a new extension:
 
-- Amazon RDS - follow the instructions on [this page](https://aws.amazon.com/rds/postgresql/faqs/)
 - Google Cloud SQL - vote or comment on [this page](https://issuetracker.google.com/issues/69250435)
-- DigitalOcean Managed Databases - vote or comment on [this page](https://ideas.digitalocean.com/app-framework-services/p/support-hypopg-for-postgres)
+- DigitalOcean Managed Databases - vote or comment on [this page](https://ideas.digitalocean.com/managed-database/p/support-hypopg-for-postgres)
+
+## HypoPG Installation Notes
+
+### Postgres Location
+
+If your machine has multiple Postgres installations, specify the path to [pg_config](https://www.postgresql.org/docs/current/app-pgconfig.html) with:
+
+```sh
+export PG_CONFIG=/Applications/Postgres.app/Contents/Versions/latest/bin/pg_config
+```
+
+Then re-run the installation instructions (run `make clean` before `make` if needed)
+
+### Missing Header
+
+If compilation fails with `fatal error: postgres.h: No such file or directory`, make sure Postgres development files are installed on the server.
+
+For Ubuntu and Debian, use:
+
+```sh
+sudo apt-get install postgresql-server-dev-15
+```
+
+Note: Replace `15` with your Postgres server version
 
 ## Additional Installation Methods
 
@@ -213,12 +260,12 @@ For databases on the host machine, use `host.docker.internal` as the hostname (o
 With Homebrew, you can use:
 
 ```sh
-brew install ankane/brew/dexter
+brew install dexter
 ```
 
 ## Future Work
 
-[Here are some ideas](https://github.com/ankane/dexter/issues/1)
+[Here are some ideas](https://github.com/ankane/dexter/issues/45)
 
 ## Upgrading
 
@@ -235,9 +282,19 @@ gem install specific_install
 gem specific_install https://github.com/ankane/dexter.git
 ```
 
+## Upgrade Notes
+
+### 0.5.0
+
+The `--stdin` option is now required to read queries from stdin.
+
+```sh
+tail -F -n +1 postgresql.log | dexter <connection-options> --stdin
+```
+
 ## Thanks
 
-This software wouldn’t be possible without [HypoPG](https://github.com/HypoPG/hypopg), which allows you to create hypothetical indexes, and [pg_query](https://github.com/lfittl/pg_query), which allows you to parse and fingerprint queries. A big thanks to Dalibo and Lukas Fittl respectively.
+This software wouldn’t be possible without [HypoPG](https://github.com/HypoPG/hypopg), which allows you to create hypothetical indexes, and [pg_query](https://github.com/lfittl/pg_query), which allows you to parse and fingerprint queries. A big thanks to Dalibo and Lukas Fittl respectively. Also, thanks to YugabyteDB for [this article](https://dev.to/yugabyte/explain-from-pgstatstatements-normalized-queries-how-to-always-get-the-generic-plan-in--5cfi) on how to explain normalized queries.
 
 ## Research
 

data/lib/dexter/client.rb

@@ -7,7 +7,7 @@ module Dexter
 
     def self.start
       Dexter::Client.new(ARGV).perform
-    rescue Dexter::Abort, PG::UndefinedFile => e
+    rescue Dexter::Abort, PG::UndefinedFile, PG::FeatureNotSupported => e
       abort colorize(e.message.strip, :red)
     end
 
@@ -27,13 +27,17 @@ module Dexter
         Indexer.new(options).process_stat_statements
       elsif options[:pg_stat_activity]
         Processor.new(:pg_stat_activity, options).perform
-      elsif options[:log_table]
-        Processor.new(:log_table, options).perform
       elsif arguments.any?
         ARGV.replace(arguments)
+        if !options[:input_format]
+          ext = ARGV.map { |v| File.extname(v) }.uniq
+          options[:input_format] = ext.first[1..-1] if ext.size == 1
+        end
         Processor.new(ARGF, options).perform
-      else
+      elsif options[:stdin]
         Processor.new(STDIN, options).perform
+      else
+        raise Dexter::Abort, "Specify a source of queries: --pg-stat-statements, --pg-stat-activity, --stdin, or a path"
       end
     end
 
@@ -44,10 +48,10 @@ module Dexter
         o.separator ""
 
         o.separator "Input options:"
-        o.string "--input-format", "input format", default: "stderr"
-        o.string "--log-table", "log table (experimental)"
-        o.boolean "--pg-stat-activity", "use pg_stat_activity", default: false, help: false
+        o.string "--input-format", "input format"
+        o.boolean "--pg-stat-activity", "use pg_stat_activity", default: false
         o.boolean "--pg-stat-statements", "use pg_stat_statements", default: false, help: false
+        o.boolean "--stdin", "use stdin", default: false
         o.string "-s", "--statement", "process a single statement"
         o.separator ""
 
@@ -62,12 +66,12 @@ module Dexter
         o.integer "--interval", "time to wait between processing queries, in seconds", default: 60
         o.float "--min-calls", "only process queries that have been called a certain number of times", default: 0
         o.float "--min-time", "only process queries that have consumed a certain amount of DB time, in minutes", default: 0
-        o.boolean "--once", "run once", default: false, help: false
         o.separator ""
 
         o.separator "Indexing options:"
         o.boolean "--analyze", "analyze tables that haven't been analyzed in the past hour", default: false
         o.boolean "--create", "create indexes", default: false
+        o.boolean "--enable-hypopg", "enable the HypoPG extension", default: false
         o.array "--exclude", "prevent specific tables from being indexed"
         o.string "--include", "only include specific tables"
         o.integer "--min-cost-savings-pct", default: 50, help: false

data/lib/dexter/collector.rb

@@ -8,7 +8,7 @@ module Dexter
       @min_calls = options[:min_calls]
     end
 
-    def add(query, duration)
+    def add(query, total_time, calls = 1)
       fingerprint =
         begin
           PgQuery.fingerprint(query)
@@ -19,8 +19,8 @@ module Dexter
       return unless fingerprint
 
       @top_queries[fingerprint] ||= {calls: 0, total_time: 0}
-      @top_queries[fingerprint][:calls] += 1
-      @top_queries[fingerprint][:total_time] += duration
+      @top_queries[fingerprint][:calls] += calls
+      @top_queries[fingerprint][:total_time] += total_time
       @top_queries[fingerprint][:query] = query
       @mutex.synchronize do
         @new_queries << fingerprint

data/lib/dexter/indexer.rb

@@ -14,11 +14,15 @@ module Dexter
       @min_calls = options[:min_calls] || 0
       @analyze = options[:analyze]
       @min_cost_savings_pct = options[:min_cost_savings_pct].to_i
-      @log_table = options[:log_table]
       @options = options
       @mutex = Mutex.new
 
-      create_extension unless extension_exists?
+      if server_version_num < 110000
+        raise Dexter::Abort, "This version of Dexter requires Postgres 11+"
+      end
+
+      check_extension
+
       execute("SET lock_timeout = '5s'")
     end
 
@@ -28,53 +32,6 @@ module Dexter
       process_queries(queries)
     end
 
-    def stat_activity
-      execute <<~SQL
-        SELECT
-          pid || ':' || COALESCE(query_start, xact_start) AS id,
-          query,
-          EXTRACT(EPOCH FROM NOW() - COALESCE(query_start, xact_start)) * 1000.0 AS duration_ms
-        FROM
-          pg_stat_activity
-        WHERE
-          datname = current_database()
-          AND state = 'active'
-          AND pid != pg_backend_pid()
-        ORDER BY
-          1
-      SQL
-    end
-
-    # works with
-    # file_fdw: https://www.postgresql.org/docs/current/file-fdw.html
-    # log_fdw: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.Extensions.foreign-data-wrappers.html
-    def csvlog_activity(last_log_time)
-      query = <<~SQL
-        SELECT
-          log_time,
-          message,
-          detail
-        FROM
-          #{conn.quote_ident(@log_table)}
-        WHERE
-          log_time >= \$1
-      SQL
-      execute(query, params: [last_log_time])
-    end
-
-    # works with
-    # file_fdw: https://www.postgresql.org/docs/current/file-fdw.html
-    # log_fdw: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.Extensions.foreign-data-wrappers.html
-    def stderr_activity
-      query = <<~SQL
-        SELECT
-          log_entry
-        FROM
-          #{conn.quote_ident(@log_table)}
-      SQL
-      execute(query)
-    end
-
     def process_queries(queries)
       # reset hypothetical indexes
       reset_hypothetical_indexes
@@ -150,19 +107,20 @@ module Dexter
 
     private
 
-    def create_extension
-      execute("SET client_min_messages = warning")
-      begin
-        execute("CREATE EXTENSION IF NOT EXISTS hypopg")
-      rescue PG::UndefinedFile
+    def check_extension
+      extension = execute("SELECT installed_version FROM pg_available_extensions WHERE name = 'hypopg'").first
+
+      if extension.nil?
         raise Dexter::Abort, "Install HypoPG first: https://github.com/ankane/dexter#installation"
-      rescue PG::InsufficientPrivilege
-        raise Dexter::Abort, "Use a superuser to run: CREATE EXTENSION hypopg"
       end
-    end
 
-    def extension_exists?
-      execute("SELECT * FROM pg_available_extensions WHERE name = 'hypopg' AND installed_version IS NOT NULL").any?
+      if extension["installed_version"].nil?
+        if @options[:enable_hypopg]
+          execute("CREATE EXTENSION hypopg")
+        else
+          raise Dexter::Abort, "Run `CREATE EXTENSION hypopg` or pass --enable-hypopg"
+        end
+      end
     end
 
     def reset_hypothetical_indexes
@@ -172,7 +130,7 @@ module Dexter
     def analyze_tables(tables)
       tables = tables.to_a.sort
 
-      analyze_stats = execute <<~SQL
+      query = <<~SQL
         SELECT
           schemaname || '.' || relname AS table,
           last_analyze,
@@ -180,8 +138,9 @@ module Dexter
         FROM
           pg_stat_user_tables
         WHERE
-          schemaname || '.' || relname IN (#{tables.map { |t| quote(t) }.join(", ")})
+          schemaname || '.' || relname IN (#{tables.size.times.map { |i| "$#{i + 1}" }.join(", ")})
       SQL
+      analyze_stats = execute(query, params: tables.to_a)
 
       last_analyzed = {}
       analyze_stats.each do |stats|
@@ -212,10 +171,6 @@ module Dexter
         end
         begin
           query.plans << plan(query.statement)
-          if @log_explain
-            # Pass format to prevent ANALYZE
-            puts execute("EXPLAIN (FORMAT TEXT) #{safe_statement(query.statement)}", pretty: false).map { |r| r["QUERY PLAN"] }.join("\n")
-          end
         rescue PG::Error, JSON::NestingError => e
           if @log_explain
             log e.message
@@ -242,10 +197,16 @@ module Dexter
         possible_columns = Set.new
         explainable_queries.each do |query|
           log "Finding columns: #{query.statement}" if @log_level == "debug3"
-          find_columns(query.tree).each do |col|
-            last_col = col["fields"].last
-            if last_col["String"]
-              possible_columns << last_col["String"]["str"]
+          begin
+            find_columns(query.tree).each do |col|
+              last_col = col["fields"].last
+              if last_col["String"]
+                possible_columns << last_col["String"]["sval"]
+              end
+            end
+          rescue JSON::NestingError
+            if @log_level.start_with?("debug")
+              log colorize("ERROR: Cannot get columns", :red)
             end
           end
         end
@@ -271,7 +232,7 @@ module Dexter
     end
 
     def find_columns(plan)
-      plan = JSON.parse(plan.to_json)
+      plan = JSON.parse(plan.to_json, max_nesting: 1000)
       find_by_key(plan, "ColumnRef")
     end
 
@@ -541,7 +502,7 @@ module Dexter
     def conn
       @conn ||= begin
         # set connect timeout if none set
-        ENV["PGCONNECT_TIMEOUT"] ||= "2"
+        ENV["PGCONNECT_TIMEOUT"] ||= "3"
 
         if @options[:dbname] =~ /\Apostgres(ql)?:\/\//
           config = @options[:dbname]
@@ -569,16 +530,56 @@ module Dexter
       # as an extra defense against SQL-injection attacks.
       # https://www.postgresql.org/docs/current/static/libpq-exec.html
       query = squish(query) if pretty
-      log colorize("[sql] #{query}", :cyan) if @log_sql
+      log colorize("[sql] #{query}#{params.any? ? " /*#{params.to_json}*/" : ""}", :cyan) if @log_sql
 
       @mutex.synchronize do
-        conn.exec_params(query, params).to_a
+        conn.exec_params("#{query} /*dexter*/", params).to_a
       end
     end
 
     def plan(query)
+      prepared = false
+      transaction = false
+
+      # try to EXPLAIN normalized queries
+      # https://dev.to/yugabyte/explain-from-pgstatstatements-normalized-queries-how-to-always-get-the-generic-plan-in--5cfi
+      explain_normalized = query.include?("$1")
+      if explain_normalized
+        prepared_name = "dexter_prepared"
+        execute("PREPARE #{prepared_name} AS #{safe_statement(query)}", pretty: false)
+        prepared = true
+        params = execute("SELECT array_length(parameter_types, 1) AS params FROM pg_prepared_statements WHERE name = $1", params: [prepared_name]).first["params"].to_i
+        query = "EXECUTE #{prepared_name}(#{params.times.map { "NULL" }.join(", ")})"
+
+        execute("BEGIN")
+        transaction = true
+
+        if server_version_num >= 120000
+          execute("SET LOCAL plan_cache_mode = force_generic_plan")
+        else
+          execute("SET LOCAL cpu_operator_cost = 1e42")
+          5.times do
+            execute("EXPLAIN (FORMAT JSON) #{safe_statement(query)}", pretty: false)
+          end
+          execute("ROLLBACK")
+          execute("BEGIN")
+        end
+      end
+
       # strip semi-colons as another measure of defense
-      JSON.parse(execute("EXPLAIN (FORMAT JSON) #{safe_statement(query)}", pretty: false).first["QUERY PLAN"], max_nesting: 1000).first["Plan"]
+      plan = JSON.parse(execute("EXPLAIN (FORMAT JSON) #{safe_statement(query)}", pretty: false).first["QUERY PLAN"], max_nesting: 1000).first["Plan"]
+
+      if @log_explain
+        # Pass format to prevent ANALYZE
+        puts execute("EXPLAIN (FORMAT TEXT) #{safe_statement(query)}", pretty: false).map { |r| r["QUERY PLAN"] }.join("\n")
+      end
+
+      plan
+    ensure
+      if explain_normalized
+        execute("ROLLBACK") if transaction
+        execute("DEALLOCATE #{prepared_name}") if prepared
+      end
     end
 
     # TODO for multicolumn indexes, use ordering
@@ -608,17 +609,13 @@ module Dexter
     end
 
     def materialized_views
-      if server_version_num >= 90300
-        result = execute <<~SQL
-          SELECT
-            schemaname || '.' || matviewname AS table_name
-          FROM
-            pg_matviews
-        SQL
-        result.map { |r| r["table_name"] }
-      else
-        []
-      end
+      result = execute <<~SQL
+        SELECT
+          schemaname || '.' || matviewname AS table_name
+        FROM
+          pg_matviews
+      SQL
+      result.map { |r| r["table_name"] }
     end
 
     def server_version_num
@@ -652,7 +649,7 @@ module Dexter
 
     def stat_statements
       total_time = server_version_num >= 130000 ? "(total_plan_time + total_exec_time)" : "total_time"
-      result = execute <<~SQL
+      sql = <<~SQL
         SELECT
           DISTINCT query
         FROM
@@ -661,18 +658,18 @@ module Dexter
           pg_database ON pg_database.oid = pg_stat_statements.dbid
         WHERE
           datname = current_database()
-          AND #{total_time} >= #{@min_time * 60000}
-          AND calls >= #{@min_calls}
+          AND #{total_time} >= \$1
+          AND calls >= \$2
         ORDER BY
           1
       SQL
-      result.map { |q| q["query"] }
+      execute(sql, params: [@min_time * 60000, @min_calls]).map { |q| q["query"] }
     end
 
     def with_advisory_lock
       lock_id = 123456
       first_time = true
-      while execute("SELECT pg_try_advisory_lock(#{lock_id})").first["pg_try_advisory_lock"] != "t"
+      while execute("SELECT pg_try_advisory_lock($1)", params: [lock_id]).first["pg_try_advisory_lock"] != "t"
         if first_time
           log "Waiting for lock..."
           first_time = false
@@ -681,16 +678,19 @@ module Dexter
       end
       yield
     ensure
-      with_min_messages("error") do
-        execute("SELECT pg_advisory_unlock(#{lock_id})")
+      suppress_messages do
+        execute("SELECT pg_advisory_unlock($1)", params: [lock_id])
       end
     end
 
-    def with_min_messages(value)
-      execute("SET client_min_messages = #{quote(value)}")
+    def suppress_messages
+      conn.set_notice_processor do |message|
+        # do nothing
+      end
       yield
     ensure
-      execute("SET client_min_messages = warning")
+      # clear notice processor
+      conn.set_notice_processor
     end
 
     def index_exists?(index)
@@ -698,7 +698,7 @@ module Dexter
     end
 
     def columns(tables)
-      columns = execute <<~SQL
+      query = <<~SQL
         SELECT
           s.nspname || '.' || t.relname AS table_name,
           a.attname AS column_name,
@@ -708,11 +708,11 @@ module Dexter
           JOIN pg_namespace s on t.relnamespace = s.oid
         WHERE a.attnum > 0
           AND NOT a.attisdropped
-          AND s.nspname || '.' || t.relname IN (#{tables.map { |t| quote(t) }.join(", ")})
+          AND s.nspname || '.' || t.relname IN (#{tables.size.times.map { |i| "$#{i + 1}" }.join(", ")})
         ORDER BY
           1, 2
       SQL
-
+      columns = execute(query, params: tables.to_a)
       columns.map { |v| {table: v["table_name"], column: v["column_name"], type: v["data_type"]} }
     end
 
@@ -732,14 +732,14 @@ module Dexter
         LEFT JOIN
           pg_stat_user_indexes ui ON ui.indexrelid = i.indexrelid
         WHERE
-          schemaname || '.' || t.relname IN (#{tables.map { |t| quote(t) }.join(", ")}) AND
+          schemaname || '.' || t.relname IN (#{tables.size.times.map { |i| "$#{i + 1}" }.join(", ")}) AND
           indisvalid = 't' AND
           indexprs IS NULL AND
           indpred IS NULL
         ORDER BY
           1, 2
       SQL
-      execute(query).map { |v| v["columns"] = v["columns"].sub(") WHERE (", " WHERE ").split(", ").map { |c| unquote(c) }; v }
+      execute(query, params: tables.to_a).map { |v| v["columns"] = v["columns"].sub(") WHERE (", " WHERE ").split(", ").map { |c| unquote(c) }; v }
     end
 
     def search_path
@@ -758,19 +758,6 @@ module Dexter
       value.split(".").map { |v| conn.quote_ident(v) }.join(".")
     end
 
-    def quote(value)
-      if value.is_a?(String)
-        "'#{quote_string(value)}'"
-      else
-        value
-      end
-    end
-
-    # from activerecord
-    def quote_string(s)
-      s.gsub(/\\/, '\&\&').gsub(/'/, "''")
-    end
-
     # from activesupport
     def squish(str)
       str.to_s.gsub(/\A[[:space:]]+/, "").gsub(/[[:space:]]+\z/, "").gsub(/[[:space:]]+/, " ")

data/lib/dexter/log_parser.rb

@@ -3,10 +3,6 @@ module Dexter
     include Logging
 
     REGEX = /duration: (\d+\.\d+) ms  (statement|execute [^:]+): (.+)/
-    LINE_SEPERATOR = ":  ".freeze
-    DETAIL_LINE = "DETAIL:  ".freeze
-
-    attr_accessor :once
 
     def initialize(logfile, collector)
       @logfile = logfile

data/lib/dexter/pg_stat_activity_parser.rb

@@ -1,27 +1,50 @@
 module Dexter
   class PgStatActivityParser < LogParser
     def perform
-      queries = {}
+      previous_queries = {}
 
-      loop do
-        new_queries = {}
-        @logfile.stat_activity.each do |row|
-          new_queries[row["id"]] = row
+      10.times do
+        active_queries = {}
+        processed_queries = {}
+
+        stat_activity.each do |row|
+          if row["state"] == "active"
+            active_queries[row["id"]] = row
+          else
+            process_entry(row["query"], row["duration_ms"].to_f)
+            processed_queries[row["id"]] = true
+          end
         end
 
         # store queries after they complete
-        queries.each do |id, row|
-          unless new_queries[id]
+        previous_queries.each do |id, row|
+          if !active_queries[id] && !processed_queries[id]
             process_entry(row["query"], row["duration_ms"].to_f)
           end
         end
 
-        queries = new_queries
+        previous_queries = active_queries
 
-        break if once
-
-        sleep(1)
+        sleep(0.1)
       end
     end
+
+    def stat_activity
+      sql = <<~SQL
+        SELECT
+          pid || ':' || COALESCE(query_start, xact_start) AS id,
+          query,
+          state,
+          EXTRACT(EPOCH FROM NOW() - COALESCE(query_start, xact_start)) * 1000.0 AS duration_ms
+        FROM
+          pg_stat_activity
+        WHERE
+          datname = current_database()
+          AND pid != pg_backend_pid()
+        ORDER BY
+          1
+      SQL
+      @logfile.send(:execute, sql)
+    end
   end
 end

data/lib/dexter/processor.rb

@@ -11,12 +11,6 @@ module Dexter
       @log_parser =
         if @logfile == :pg_stat_activity
           PgStatActivityParser.new(@indexer, @collector)
-        elsif @logfile == :log_table
-          if options[:input_format] == "csv"
-            CsvLogTableParser.new(@indexer, @collector)
-          else
-            StderrLogTableParser.new(@indexer, @collector)
-          end
         elsif options[:input_format] == "csv"
           CsvLogParser.new(logfile, @collector)
         elsif options[:input_format] == "json"
@@ -29,7 +23,6 @@ module Dexter
 
       @starting_interval = 3
       @interval = options[:interval]
-      @log_parser.once = options[:once]
 
       @mutex = Mutex.new
       @last_checked_at = {}
@@ -38,7 +31,7 @@ module Dexter
     end
 
     def perform
-      if [STDIN, :pg_stat_activity, :log_table].include?(@logfile) && !@log_parser.once
+      if [STDIN].include?(@logfile)
         Thread.abort_on_exception = true
         Thread.new do
           sleep(@starting_interval)

data/lib/dexter/stderr_log_parser.rb

@@ -1,5 +1,8 @@
 module Dexter
   class StderrLogParser < LogParser
+    LINE_SEPERATOR = ":  ".freeze
+    DETAIL_LINE = "DETAIL:  ".freeze
+
     def perform
       process_stderr(@logfile.each_line)
     end

data/lib/dexter/version.rb

@@ -1,3 +1,3 @@
 module Dexter
-  VERSION = "0.4.3"
+  VERSION = "0.5.1"
 end

data/lib/dexter.rb

@@ -21,12 +21,10 @@ require_relative "dexter/version"
 # parsers
 require_relative "dexter/log_parser"
 require_relative "dexter/csv_log_parser"
-require_relative "dexter/csv_log_table_parser"
 require_relative "dexter/json_log_parser"
 require_relative "dexter/pg_stat_activity_parser"
 require_relative "dexter/sql_log_parser"
 require_relative "dexter/stderr_log_parser"
-require_relative "dexter/stderr_log_table_parser"
 
 module Dexter
   class Abort < StandardError; end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: pgdexter
 version: !ruby/object:Gem::Version
-  version: 0.4.3
+  version: 0.5.1
 platform: ruby
 authors:
 - Andrew Kane
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2023-03-27 00:00:00.000000000 Z
+date: 2023-05-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: pg
@@ -30,14 +30,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '2.1'
+        version: '4'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '2.1'
+        version: '4'
 - !ruby/object:Gem::Dependency
   name: slop
   requirement: !ruby/object:Gem::Requirement
@@ -67,7 +67,6 @@ files:
 - lib/dexter/client.rb
 - lib/dexter/collector.rb
 - lib/dexter/csv_log_parser.rb
-- lib/dexter/csv_log_table_parser.rb
 - lib/dexter/indexer.rb
 - lib/dexter/json_log_parser.rb
 - lib/dexter/log_parser.rb
@@ -77,7 +76,6 @@ files:
 - lib/dexter/query.rb
 - lib/dexter/sql_log_parser.rb
 - lib/dexter/stderr_log_parser.rb
-- lib/dexter/stderr_log_table_parser.rb
 - lib/dexter/version.rb
 homepage: https://github.com/ankane/dexter
 licenses:
@@ -91,14 +89,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '2.5'
+      version: '2.7'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.4.6
+rubygems_version: 3.4.10
 signing_key:
 specification_version: 4
 summary: The automatic indexer for Postgres

data/lib/dexter/csv_log_table_parser.rb

@@ -1,20 +0,0 @@
-module Dexter
-  class CsvLogTableParser < CsvLogParser
-    def perform
-      last_log_time = Time.at(0).iso8601(3)
-
-      loop do
-        @logfile.csvlog_activity(last_log_time).each do |row|
-          process_csv_row(row["message"], row["detail"])
-          last_log_time = row["log_time"]
-        end
-
-        break
-
-        # possibly enable later
-        # break if once
-        # sleep(1)
-      end
-    end
-  end
-end

data/lib/dexter/stderr_log_table_parser.rb

@@ -1,7 +0,0 @@
-module Dexter
-  class StderrLogTableParser < StderrLogParser
-    def perform
-      process_stderr(@logfile.stderr_activity.map { |r| r["log_entry"] })
-    end
-  end
-end