pgdexter 0.3.2 → 0.3.7

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 714767248afe28ad9e354ebb6485b34a8d6aadd0
-  data.tar.gz: 545eaaea1df0312049f4cecff3fe9b03b80a6cd9
+SHA256:
+  metadata.gz: a4245b0c02667e411a915ef4c19ef472925048180d5882bc7b63b90037cb9d45
+  data.tar.gz: '0887ae99de87c186c1bfc52edbdbb59546df61de93bfbccd0974dff5c15b80f0'
 SHA512:
-  metadata.gz: '096684887c5d5a48a2df74a2dae6cf70bc2b5e98dd5dc6d7d6d8b583bfb76193ce86166972d9553f89452f16629a8576a30f2627fd8d0b16a9672ce76a316835'
-  data.tar.gz: d128734845414672a3f470f138c6a422dc2b24024a66866e50524fe3bb3e949bd4a899305a93f71fb14af3b56820f66e20a00d5ba5788cd0c88bd0afaa064e32
+  metadata.gz: 45ec7b462d925c2e5aa313734d1109b59e65e743c28e3471ef604d430037949675c615ab9004b98f8b055b0a575cdc169470d89e64694ef39cdd2bee60674548
+  data.tar.gz: 8d33afbd03ba250ae7dd4dfe6c1947e8860fece0925b58a098fb7a54f6f8fe76aaa3bc93f663b10282ef1a9b3bb2c059ac26014f9a3cff3798ece22dbb8d62be

data/CHANGELOG.md

@@ -1,16 +1,42 @@
-## 0.3.2
+## 0.3.7 (2020-07-10)
+
+- Fixed help output
+
+## 0.3.6 (2020-03-30)
+
+- Fixed warning with Ruby 2.7
+
+## 0.3.5 (2018-04-30)
+
+- Added `sql` input format
+- Fixed error for queries with double dash comments
+- Fixed connection threading issue with `--pg-stat-activity` option
+
+## 0.3.4 (2018-04-09)
+
+- Fixed `--username` option
+- Fixed `JSON::NestingError`
+- Added `--pg-stat-activity` option
+
+## 0.3.3 (2018-02-22)
+
+- Added support for views and materialized views
+- Better handle case when multiple indexes are found for a query
+- Added `--min-cost-savings-pct` option
+
+## 0.3.2 (2018-01-04)
 
 - Fixed parsing issue with named prepared statements
 - Fixed parsing issue with multiline queries in csv format
 - Better explanations for indexing decisions
 
-## 0.3.1
+## 0.3.1 (2017-12-28)
 
 - Added support for queries with bind variables
 - Fixed error with streaming logs as csv format
 - Handle malformed CSV gracefully
 
-## 0.3.0
+## 0.3.0 (2017-12-22)
 
 - Added support for schemas
 - Added support for csv format
@@ -18,12 +44,12 @@
 - Added `--min-calls` option
 - Fixed debug output when indexes not found
 
-## 0.2.1
+## 0.2.1 (2017-09-02)
 
 - Fixed bad suggestions
 - Improved debugging output
 
-## 0.2.0
+## 0.2.0 (2017-08-27)
 
 - Added same connection options as `psql`
 - Added support for multiple files
@@ -34,38 +60,38 @@ Breaking
 
 - `-h` option changed to `--host` instead of `--help` for consistency with `psql`
 
-## 0.1.6
+## 0.1.6 (2017-08-26)
 
 - Significant performance improvements
 - Added `--include` option
 
-## 0.1.5
+## 0.1.5 (2017-08-14)
 
 - Added support for non-`SELECT` queries
 - Added `--pg-stat-statements` option
 - Added advisory locks
 - Added support for running as a non-superuser
 
-## 0.1.4
+## 0.1.4 (2017-07-02)
 
 - Added support for multicolumn indexes
 
-## 0.1.3
+## 0.1.3 (2017-06-30)
 
 - Fixed error with non-lowercase columns
 - Fixed error with `json` columns
 
-## 0.1.2
+## 0.1.2 (2017-06-26)
 
 - Added `--exclude` option
 - Added `--log-sql` option
 
-## 0.1.1
+## 0.1.1 (2017-06-25)
 
 - Added `--interval` option
 - Added `--min-time` option
 - Added `--log-level` option
 
-## 0.1.0
+## 0.1.0 (2017-06-24)
 
 - Launched

data/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright (c) 2017 Andrew Kane
+Copyright (c) 2017-2020 Andrew Kane
 
 MIT License
 

data/README.md

@@ -2,18 +2,18 @@
 
 The automatic indexer for Postgres
 
-[Read about how it works](https://medium.com/@ankane/introducing-dexter-the-automatic-indexer-for-postgres-5f8fa8b28f27)
+[Read about how it works](https://ankane.org/introducing-dexter)
 
 [![Build Status](https://travis-ci.org/ankane/dexter.svg?branch=master)](https://travis-ci.org/ankane/dexter)
 
 ## Installation
 
-First, install [HypoPG](https://github.com/dalibo/hypopg) on your database server. This doesn’t require a restart.
+First, install [HypoPG](https://github.com/HypoPG/hypopg) on your database server. This doesn’t require a restart.
 
 ```sh
 cd /tmp
-curl -L https://github.com/dalibo/hypopg/archive/1.1.0.tar.gz | tar xz
-cd hypopg-1.1.0
+curl -L https://github.com/HypoPG/hypopg/archive/1.1.4.tar.gz | tar xz
+cd hypopg-1.1.4
 make
 make install # may need sudo
 ```
@@ -104,12 +104,20 @@ or pass files:
 dexter <connection-options> <file1> <file2>
 ```
 
+or collect running queries with:
+
+```sh
+dexter <connection-options> --pg-stat-activity
+```
+
 or use the [pg_stat_statements](https://www.postgresql.org/docs/current/static/pgstatstatements.html) extension:
 
 ```sh
 dexter <connection-options> --pg-stat-statements
 ```
 
+> Note: Logs or running queries are highly preferred over pg_stat_statements, as pg_stat_statements often doesn’t store enough information to optimize queries.
+
 ### Collection Options
 
 To prevent one-off queries from being indexed, specify a minimum number of calls before a query is considered for indexing
@@ -207,6 +215,10 @@ gem specific_install https://github.com/ankane/dexter.git
 
 This software wouldn’t be possible without [HypoPG](https://github.com/dalibo/hypopg), which allows you to create hypothetical indexes, and [pg_query](https://github.com/lfittl/pg_query), which allows you to parse and fingerprint queries. A big thanks to Dalibo and Lukas Fittl respectively.
 
+## Research
+
+This is known as the Index Selection Problem (ISP).
+
 ## Contributing
 
 Everyone is encouraged to help improve this project. Here are a few ways you can help:
@@ -216,17 +228,18 @@ Everyone is encouraged to help improve this project. Here are a few ways you can
 - Write, clarify, or fix documentation
 - Suggest or add new features
 
-To get started, run:
+To get started with development, run:
 
 ```sh
 git clone https://github.com/ankane/dexter.git
 cd dexter
-bundle
-rake install
+bundle install
+bundle exec rake install
 ```
 
 To run tests, use:
 
 ```sh
-rake test
+createdb dexter_test
+bundle exec rake test
 ```

data/exe/dexter

@@ -1,10 +1,7 @@
 #!/usr/bin/env ruby
 
+# handle interrupts
+trap("SIGINT") { abort }
+
 require "dexter"
-begin
-  Dexter::Client.new(ARGV).perform
-rescue Dexter::Abort => e
-  abort e.message
-rescue Interrupt => e
-  # do nothing
-end
+Dexter::Client.start

data/lib/dexter.rb

@@ -1,16 +1,22 @@
-require "dexter/version"
-require "slop"
+# dependencies
 require "pg"
 require "pg_query"
-require "time"
+require "slop"
+
+# stdlib
 require "set"
-require "thread"
+require "time"
+
+# modules
+require "dexter/version"
 require "dexter/logging"
 require "dexter/client"
 require "dexter/collector"
 require "dexter/indexer"
 require "dexter/log_parser"
 require "dexter/csv_log_parser"
+require "dexter/pg_stat_activity_parser"
+require "dexter/sql_log_parser"
 require "dexter/processor"
 require "dexter/query"
 

data/lib/dexter/client.rb

@@ -4,6 +4,12 @@ module Dexter
 
     attr_reader :arguments, :options
 
+    def self.start
+      Dexter::Client.new(ARGV).perform
+    rescue Dexter::Abort => e
+      abort e.message
+    end
+
     def initialize(args)
       @arguments, @options = parse_args(args)
     end
@@ -18,6 +24,8 @@ module Dexter
       elsif options[:pg_stat_statements]
         # TODO support streaming option
         Indexer.new(options).process_stat_statements
+      elsif options[:pg_stat_activity]
+        Processor.new(:pg_stat_activity, options).perform
       elsif arguments.any?
         ARGV.replace(arguments)
         Processor.new(ARGF, options).perform
@@ -29,9 +37,9 @@ module Dexter
     def parse_args(args)
       opts = Slop.parse(args) do |o|
         o.banner = %(Usage:
-    dexter [options]
-
-Options:)
+    dexter [options])
+        o.separator ""
+        o.separator "Options:"
         o.boolean "--analyze", "analyze tables that haven't been analyzed in the past hour", default: false
         o.boolean "--create", "create indexes", default: false
         o.array "--exclude", "prevent specific tables from being indexed"
@@ -43,11 +51,10 @@ Options:)
         o.boolean "--log-sql", "log sql", default: false
         o.float "--min-calls", "only process queries that have been called a certain number of times", default: 0
         o.float "--min-time", "only process queries that have consumed a certain amount of DB time, in minutes", default: 0
+        o.integer "--min-cost-savings-pct", default: 50, help: false
+        o.boolean "--pg-stat-activity", "use pg_stat_activity", default: false, help: false
         o.boolean "--pg-stat-statements", "use pg_stat_statements", default: false, help: false
         o.string "-s", "--statement", "process a single statement"
-        # separator must go here to show up correctly - slop bug?
-        o.separator ""
-        o.separator "Connection options:"
         o.on "-v", "--version", "print the version" do
           log Dexter::VERSION
           exit
@@ -56,10 +63,12 @@ Options:)
           log o
           exit
         end
-        o.string "-U", "--username"
-        o.string "-d", "--dbname"
-        o.string "-h", "--host"
-        o.integer "-p", "--port"
+        o.separator ""
+        o.separator "Connection options:"
+        o.string "-d", "--dbname", "database name"
+        o.string "-h", "--host", "database host"
+        o.integer "-p", "--port", "database port"
+        o.string "-U", "--username", "database user"
       end
 
       arguments = opts.arguments

data/lib/dexter/indexer.rb

@@ -12,7 +12,9 @@ module Dexter
       @min_time = options[:min_time] || 0
       @min_calls = options[:min_calls] || 0
       @analyze = options[:analyze]
+      @min_cost_savings_pct = options[:min_cost_savings_pct].to_i
       @options = options
+      @mutex = Mutex.new
 
       create_extension unless extension_exists?
       execute("SET lock_timeout = '5s'")
@@ -24,11 +26,28 @@ module Dexter
       process_queries(queries)
     end
 
+    def stat_activity
+      execute <<-SQL
+        SELECT
+          pid || ':' || COALESCE(query_start, xact_start) AS id,
+          query,
+          EXTRACT(EPOCH FROM NOW() - COALESCE(query_start, xact_start)) * 1000.0 AS duration_ms
+        FROM
+          pg_stat_activity
+        WHERE
+          datname = current_database()
+          AND state = 'active'
+          AND pid != pg_backend_pid()
+        ORDER BY
+          1
+      SQL
+    end
+
     def process_queries(queries)
       # reset hypothetical indexes
       reset_hypothetical_indexes
 
-      tables = Set.new(database_tables)
+      tables = Set.new(database_tables + materialized_views)
 
       # map tables without schema to schema
       no_schema_tables = {}
@@ -37,11 +56,28 @@ module Dexter
         no_schema_tables[group] = t2.sort_by { |t| [search_path_index[t.split(".")[0]] || 1000000, t] }[0]
       end
 
+      # add tables from views
+      view_tables = database_view_tables
+      view_tables.each do |v, vt|
+        view_tables[v] = vt.map { |t| no_schema_tables[t] || t }
+      end
+
+      # fully resolve tables
+      # make sure no views in result
+      view_tables.each do |v, vt|
+        view_tables[v] = vt.flat_map { |t| view_tables[t] || [t] }.uniq
+      end
+
       # filter queries from other databases and system tables
       queries.each do |query|
         # add schema to table if needed
         query.tables = query.tables.map { |t| no_schema_tables[t] || t }
 
+        # substitute view tables
+        new_tables = query.tables.flat_map { |t| view_tables[t] || [t] }.uniq
+        query.tables_from_views = new_tables - query.tables
+        query.tables = new_tables
+
         # check for missing tables
         query.missing_tables = !query.tables.all? { |t| tables.include?(t) }
       end
@@ -71,13 +107,13 @@ module Dexter
       analyze_tables(tables) if tables.any? && (@analyze || @log_level == "debug2")
 
       # create hypothetical indexes and explain queries
-      candidates = tables.any? ? create_hypothetical_indexes(queries.select(&:candidate_tables), tables) : {}
+      candidates = tables.any? ? create_hypothetical_indexes(queries.select(&:candidate_tables)) : {}
 
       # see if new indexes were used and meet bar
       new_indexes = determine_indexes(queries, candidates, tables)
 
       # display and create new indexes
-      show_and_create_indexes(new_indexes, queries, tables)
+      show_and_create_indexes(new_indexes, queries)
     end
 
     private
@@ -146,9 +182,9 @@ module Dexter
           query.plans << plan(query.statement)
           if @log_explain
             # Pass format to prevent ANALYZE
-            puts execute("EXPLAIN (FORMAT TEXT) #{safe_statement(query.statement)}").map { |r| r["QUERY PLAN"] }.join("\n")
+            puts execute("EXPLAIN (FORMAT TEXT) #{safe_statement(query.statement)}", pretty: false).map { |r| r["QUERY PLAN"] }.join("\n")
           end
-        rescue PG::Error => e
+        rescue PG::Error, JSON::NestingError => e
           if @log_explain
             log e.message
           end
@@ -157,7 +193,7 @@ module Dexter
       end
     end
 
-    def create_hypothetical_indexes(queries, tables)
+    def create_hypothetical_indexes(queries)
       candidates = {}
 
       # get initial costs for queries
@@ -166,6 +202,7 @@ module Dexter
 
       # filter tables for performance
       tables = Set.new(explainable_queries.flat_map(&:tables))
+      tables_from_views = Set.new(explainable_queries.flat_map(&:tables_from_views))
 
       if tables.any?
         # since every set of multi-column indexes are expensive
@@ -182,7 +219,8 @@ module Dexter
         end
 
         # create hypothetical indexes
-        columns_by_table = columns(tables).select { |c| possible_columns.include?(c[:column]) }.group_by { |c| c[:table] }
+        # use all columns in tables from views
+        columns_by_table = columns(tables).select { |c| possible_columns.include?(c[:column]) || tables_from_views.include?(c[:table]) }.group_by { |c| c[:table] }
 
         # create single column indexes
         create_hypothetical_indexes_helper(columns_by_table, 1, candidates)
@@ -265,14 +303,16 @@ module Dexter
         end
       end
 
+      savings_ratio = (1 - @min_cost_savings_pct / 100.0)
+
       queries.each do |query|
         if query.explainable? && query.high_cost?
           new_cost, new_cost2 = query.costs[1..2]
 
-          cost_savings = new_cost < query.initial_cost * 0.5
+          cost_savings = new_cost < query.initial_cost * savings_ratio
 
           # set high bar for multicolumn indexes
-          cost_savings2 = new_cost > 100 && new_cost2 < new_cost * 0.5
+          cost_savings2 = new_cost > 100 && new_cost2 < new_cost * savings_ratio
 
           key = cost_savings2 ? 2 : 1
           query_indexes = hypo_indexes_from_plan(index_name_to_columns, query.plans[key], index_set)
@@ -283,10 +323,56 @@ module Dexter
             cost_savings2 = false
           end
 
-          # TODO if multiple indexes are found (for either single or multicolumn)
+          suggest_index = cost_savings || cost_savings2
+
+          cost_savings3 = false
+          new_cost3 = nil
+
+          # if multiple indexes are found (for either single or multicolumn)
           # determine the impact of each individually
-          # for now, be conservative and don't suggest if more than one index
-          suggest_index = (cost_savings || cost_savings2) && query_indexes.size == 1
+          # there may be a better single index that we're not considering
+          # that didn't get picked up by pass1 or pass2
+          # TODO clean this up
+          # TODO suggest more than one index from this if savings are there
+          if suggest_index && query_indexes.size > 1
+            winning_index = nil
+            winning_cost = nil
+            winning_plan = nil
+
+            query_indexes.each do |query_index|
+              reset_hypothetical_indexes
+              create_hypothetical_index(query_index[:table], query_index[:columns].map { |v| {column: v} })
+              plan3 = plan(query.statement)
+              cost3 = plan3["Total Cost"]
+
+              if !winning_cost || cost3 < winning_cost
+                winning_cost = cost3
+                winning_index = query_index
+                winning_plan = plan3
+              end
+            end
+
+            query.plans << winning_plan
+
+            # duplicated from above
+            # TODO DRY
+            use_winning =
+              if cost_savings2
+                new_cost > 100 && winning_cost < new_cost * savings_ratio
+              else
+                winning_cost < query.initial_cost * savings_ratio
+              end
+
+            query_indexes = [winning_index]
+            new_cost3 = winning_cost
+            query.pass3_indexes = query_indexes
+
+            if use_winning
+              cost_savings3 = true
+            else
+              suggest_index = false
+            end
+          end
 
           if suggest_index
             query_indexes.each do |index|
@@ -299,7 +385,7 @@ module Dexter
           query.suggest_index = suggest_index
           query.new_cost =
             if suggest_index
-              cost_savings2 ? new_cost2 : new_cost
+              cost_savings3 ? new_cost3 : (cost_savings2 ? new_cost2 : new_cost)
             else
               query.initial_cost
             end
@@ -331,7 +417,7 @@ module Dexter
       end
     end
 
-    def show_and_create_indexes(new_indexes, queries, tables)
+    def show_and_create_indexes(new_indexes, queries)
       # print summary
       if new_indexes.any?
         new_indexes.each do |index|
@@ -368,9 +454,12 @@ module Dexter
             log "Start: #{query.costs[0]}"
             log "Pass1: #{query.costs[1]} : #{log_indexes(query.pass1_indexes || [])}"
             log "Pass2: #{query.costs[2]} : #{log_indexes(query.pass2_indexes || [])}"
+            if query.costs[3]
+              log "Pass3: #{query.costs[3]} : #{log_indexes(query.pass3_indexes || [])}"
+            end
             log "Final: #{query.new_cost} : #{log_indexes(query.suggest_index ? query_indexes : [])}"
-            if query_indexes.size == 1 && !query.suggest_index
-              log "Need 50% cost savings to suggest index"
+            if (query.pass1_indexes.any? || query.pass2_indexes.any?) && !query.suggest_index
+              log "Need #{@min_cost_savings_pct}% cost savings to suggest index"
             end
           else
             log "Could not run explain"
@@ -409,6 +498,9 @@ module Dexter
 
     def conn
       @conn ||= begin
+        # set connect timeout if none set
+        ENV["PGCONNECT_TIMEOUT"] ||= "2"
+
         if @options[:dbname] =~ /\Apostgres(ql)?:\/\//
           config = @options[:dbname]
         else
@@ -416,7 +508,7 @@ module Dexter
             host: @options[:host],
             port: @options[:port],
             dbname: @options[:dbname],
-            user: @options[:user]
+            user: @options[:username]
           }.reject { |_, value| value.to_s.empty? }
           config = config[:dbname] if config.keys == [:dbname] && config[:dbname].include?("=")
         end
@@ -426,7 +518,7 @@ module Dexter
       abort e.message
     end
 
-    def execute(query)
+    def execute(query, pretty: true)
       # use exec_params instead of exec for security
       #
       # Unlike PQexec, PQexecParams allows at most one SQL command in the given string.
@@ -434,14 +526,17 @@ module Dexter
       # This is a limitation of the underlying protocol, but has some usefulness
       # as an extra defense against SQL-injection attacks.
       # https://www.postgresql.org/docs/current/static/libpq-exec.html
-      query = squish(query)
+      query = squish(query) if pretty
       log "SQL: #{query}" if @log_sql
-      conn.exec_params(query, []).to_a
+
+      @mutex.synchronize do
+        conn.exec_params(query, []).to_a
+      end
     end
 
     def plan(query)
       # strip semi-colons as another measure of defense
-      JSON.parse(execute("EXPLAIN (FORMAT JSON) #{safe_statement(query)}").first["QUERY PLAN"]).first["Plan"]
+      JSON.parse(execute("EXPLAIN (FORMAT JSON) #{safe_statement(query)}", pretty: false).first["QUERY PLAN"], max_nesting: 1000).first["Plan"]
     end
 
     # TODO for multicolumn indexes, use ordering
@@ -449,11 +544,15 @@ module Dexter
       columns_by_table.each do |table, cols|
         # no reason to use btree index for json columns
         cols.reject { |c| ["json", "jsonb"].include?(c[:type]) }.permutation(n) do |col_set|
-          candidates[col_set] = execute("SELECT * FROM hypopg_create_index('CREATE INDEX ON #{quote_ident(table)} (#{col_set.map { |c| quote_ident(c[:column])  }.join(", ")})')").first["indexname"]
+          candidates[col_set] = create_hypothetical_index(table, col_set)
         end
       end
     end
 
+    def create_hypothetical_index(table, col_set)
+      execute("SELECT * FROM hypopg_create_index('CREATE INDEX ON #{quote_ident(table)} (#{col_set.map { |c| quote_ident(c[:column])  }.join(", ")})')").first["indexname"]
+    end
+
     def database_tables
       result = execute <<-SQL
         SELECT
@@ -466,6 +565,43 @@ module Dexter
       result.map { |r| r["table_name"] }
     end
 
+    def materialized_views
+      if server_version_num >= 90300
+        result = execute <<-SQL
+          SELECT
+            schemaname || '.' || matviewname AS table_name
+          FROM
+            pg_matviews
+        SQL
+        result.map { |r| r["table_name"] }
+      else
+        []
+      end
+    end
+
+    def server_version_num
+      execute("SHOW server_version_num").first["server_version_num"].to_i
+    end
+
+    def database_view_tables
+      result = execute <<-SQL
+        SELECT
+          schemaname || '.' || viewname AS table_name,
+          definition
+        FROM
+          pg_views
+        WHERE
+          schemaname NOT IN ('information_schema', 'pg_catalog')
+      SQL
+
+      view_tables = {}
+      result.each do |row|
+        view_tables[row["table_name"]] = PgQuery.parse(row["definition"]).tables
+      end
+
+      view_tables
+    end
+
     def stat_statements
       result = execute <<-SQL
         SELECT
@@ -515,13 +651,15 @@ module Dexter
     def columns(tables)
       columns = execute <<-SQL
         SELECT
-          table_schema || '.' || table_name AS table_name,
-          column_name,
-          data_type
-        FROM
-          information_schema.columns
-        WHERE
-          table_schema || '.' || table_name IN (#{tables.map { |t| quote(t) }.join(", ")})
+          s.nspname || '.' || t.relname AS table_name,
+          a.attname AS column_name,
+          pg_catalog.format_type(a.atttypid, a.atttypmod) AS data_type
+        FROM pg_attribute a
+          JOIN pg_class t on a.attrelid = t.oid
+          JOIN pg_namespace s on t.relnamespace = s.oid
+        WHERE a.attnum > 0
+          AND NOT a.attisdropped
+          AND s.nspname || '.' || t.relname IN (#{tables.map { |t| quote(t) }.join(", ")})
         ORDER BY
           1, 2
       SQL

data/lib/dexter/pg_stat_activity_parser.rb

@@ -0,0 +1,25 @@
+module Dexter
+  class PgStatActivityParser < LogParser
+    def perform
+      queries = {}
+
+      loop do
+        new_queries = {}
+        @logfile.stat_activity.each do |row|
+          new_queries[row["id"]] = row
+        end
+
+        # store queries after they complete
+        queries.each do |id, row|
+          unless new_queries[id]
+            process_entry(row["query"], row["duration_ms"].to_f)
+          end
+        end
+
+        queries = new_queries
+
+        sleep(1)
+      end
+    end
+  end
+end

data/lib/dexter/processor.rb

@@ -6,15 +6,19 @@ module Dexter
       @logfile = logfile
 
       @collector = Collector.new(min_time: options[:min_time], min_calls: options[:min_calls])
+      @indexer = Indexer.new(options)
+
       @log_parser =
-        if options[:input_format] == "csv"
+        if @logfile == :pg_stat_activity
+          PgStatActivityParser.new(@indexer, @collector)
+        elsif options[:input_format] == "csv"
           CsvLogParser.new(logfile, @collector)
+        elsif options[:input_format] == "sql"
+          SqlLogParser.new(logfile, @collector)
         else
           LogParser.new(logfile, @collector)
         end
 
-      @indexer = Indexer.new(options)
-
       @starting_interval = 3
       @interval = options[:interval]
 
@@ -25,7 +29,7 @@ module Dexter
     end
 
     def perform
-      if @logfile == STDIN
+      if [STDIN, :pg_stat_activity].include?(@logfile)
         Thread.abort_on_exception = true
         Thread.new do
           sleep(@starting_interval)

data/lib/dexter/query.rb

@@ -2,7 +2,7 @@ module Dexter
   class Query
     attr_reader :statement, :fingerprint, :plans
     attr_writer :tables
-    attr_accessor :missing_tables, :new_cost, :total_time, :calls, :indexes, :suggest_index, :pass1_indexes, :pass2_indexes, :candidate_tables
+    attr_accessor :missing_tables, :new_cost, :total_time, :calls, :indexes, :suggest_index, :pass1_indexes, :pass2_indexes, :pass3_indexes, :candidate_tables, :tables_from_views
 
     def initialize(statement, fingerprint = nil)
       @statement = statement
@@ -11,6 +11,7 @@ module Dexter
       end
       @fingerprint = fingerprint
       @plans = []
+      @tables_from_views = []
     end
 
     def tables

data/lib/dexter/sql_log_parser.rb

@@ -0,0 +1,10 @@
+module Dexter
+  class SqlLogParser < LogParser
+    def perform
+      # TODO support streaming
+      @logfile.read.split(";").each do |statement|
+        process_entry(statement, 0)
+      end
+    end
+  end
+end

data/lib/dexter/version.rb

@@ -1,3 +1,3 @@
 module Dexter
-  VERSION = "0.3.2"
+  VERSION = "0.3.7"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: pgdexter
 version: !ruby/object:Gem::Version
-  version: 0.3.2
+  version: 0.3.7
 platform: ruby
 authors:
 - Andrew Kane
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2018-01-05 00:00:00.000000000 Z
+date: 2020-07-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: slop
@@ -16,28 +16,28 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 4.2.0
+        version: 4.8.2
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 4.2.0
+        version: 4.8.2
 - !ruby/object:Gem::Dependency
   name: pg
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 0.18.2
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 0.18.2
 - !ruby/object:Gem::Dependency
   name: pg_query
   requirement: !ruby/object:Gem::Requirement
@@ -95,23 +95,16 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
 description: 
-email:
-- andrew@chartkick.com
+email: andrew@chartkick.com
 executables:
 - dexter
 extensions: []
 extra_rdoc_files: []
 files:
-- ".gitignore"
-- ".travis.yml"
 - CHANGELOG.md
-- Gemfile
 - LICENSE.txt
 - README.md
-- Rakefile
 - exe/dexter
-- guides/Hosted-Postgres.md
-- guides/Linux.md
 - lib/dexter.rb
 - lib/dexter/client.rb
 - lib/dexter/collector.rb
@@ -119,12 +112,14 @@ files:
 - lib/dexter/indexer.rb
 - lib/dexter/log_parser.rb
 - lib/dexter/logging.rb
+- lib/dexter/pg_stat_activity_parser.rb
 - lib/dexter/processor.rb
 - lib/dexter/query.rb
+- lib/dexter/sql_log_parser.rb
 - lib/dexter/version.rb
-- pgdexter.gemspec
 homepage: https://github.com/ankane/dexter
-licenses: []
+licenses:
+- MIT
 metadata: {}
 post_install_message: 
 rdoc_options: []
@@ -134,15 +129,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '0'
+      version: '2.2'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.6.13
+rubygems_version: 3.1.2
 signing_key: 
 specification_version: 4
 summary: The automatic indexer for Postgres

data/.gitignore

@@ -1,9 +0,0 @@
-/.bundle/
-/.yardoc
-/Gemfile.lock
-/_yardoc/
-/coverage/
-/doc/
-/pkg/
-/spec/reports/
-/tmp/

data/.travis.yml

@@ -1,18 +0,0 @@
-language: ruby
-rvm: 2.4.1
-cache: bundler
-script: bundle exec rake test
-addons:
-  postgresql: "9.6"
-before_script:
-  - sudo apt-get install postgresql-server-dev-9.6
-  - wget https://github.com/dalibo/hypopg/archive/1.0.0.tar.gz
-  - tar xf 1.0.0.tar.gz
-  - cd hypopg-1.0.0
-  - make
-  - sudo make install
-  - psql -c 'create database dexter_test;' -U postgres
-notifications:
-  email:
-    on_success: never
-    on_failure: change

data/Gemfile

@@ -1,4 +0,0 @@
-source "https://rubygems.org"
-
-# Specify your gem's dependencies in dexter.gemspec
-gemspec

data/Rakefile

@@ -1,11 +0,0 @@
-require "bundler/gem_tasks"
-require "rake/testtask"
-
-Rake::TestTask.new(:test) do |t|
-  t.libs << "test"
-  t.libs << "lib"
-  t.test_files = FileList["test/**/*_test.rb"]
-  t.warning = false
-end
-
-task default: :test

data/guides/Hosted-Postgres.md

@@ -1,102 +0,0 @@
-# Hosted Postgres
-
-Some hosted providers like Amazon RDS and Heroku do not support the HypoPG extension, which Dexter needs to run. Hopefully this will change with time. For now, we can spin up a separate database instance to run Dexter. It’s not super convenient, but can be useful to do from time to time.
-
-### Install Postgres and Ruby
-
-Linux
-
-```sh
-sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
-sudo apt-get install -y wget ca-certificates
-wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
-sudo apt-get update
-sudo apt-get install -y postgresql-9.6 postgresql-server-dev-9.6
-sudo -u postgres createuser $(whoami) -s
-sudo apt-get install -y ruby2.2 ruby2.2-dev
-```
-
-Mac
-
-```sh
-brew install postgresql
-brew install ruby
-```
-
-### Install HypoPG and Dexter
-
-HypoPG
-
-```sh
-cd /tmp
-curl -L https://github.com/dalibo/hypopg/archive/1.0.0.tar.gz | tar xz
-cd hypopg-1.0.0
-make
-make install # may need sudo
-```
-
-Dexter
-
-```sh
-gem install pgdexter # may need sudo
-```
-
-### Download logs
-
-#### Amazon RDS
-
-Create an IAM user with the policy below:
-
-```
-{
-  "Statement": [
-    {
-      "Action": [
-        "rds:DescribeDBLogFiles",
-        "rds:DownloadDBLogFilePortion"
-      ],
-      "Effect": "Allow",
-      "Resource": "*"
-    }
-  ]
-}
-```
-
-And run:
-
-```sh
-aws configure
-gem install pghero_logs # may need sudo
-pghero_logs download <instance-id>
-```
-
-#### Heroku
-
-Production-tier databases only
-
-```sh
-heroku logs -p postgres > postgresql.log
-```
-
-### Dump and restore
-
-We recommend creating a new instance from a snapshot for the dump to avoid affecting customers.
-
-```sh
-pg_dump -v -j 8 -Fd -f /tmp/newout.dir <connection-options>
-```
-
-Then shutdown the dump instance. Restore with:
-
-```sh
-createdb dexter_restore
-pg_restore -v -j 8 -x -O --format=d -d dexter_restore /tmp/newout.dir/
-```
-
-### Run Dexter
-
-```sh
-dexter dexter_restore postgresql.log* --analyze
-```
-
-:tada:

data/guides/Linux.md

@@ -1,59 +0,0 @@
-# Linux Packages
-
-Distributions
-
-- [Ubuntu 16.04 (Xenial)](#ubuntu-1604-xenial)
-- [Ubuntu 14.04 (Trusty)](#ubuntu-1404-trusty)
-- [Debian 8 (Jesse)](#debian-8-jesse)
-- [CentOS / RHEL 7](#centos--rhel-7)
-- [SUSE Linux Enterprise Server 12](#suse-linux-enterprise-server-12)
-
-### Ubuntu 16.04 (Xenial)
-
-```sh
-wget -qO- https://dl.packager.io/srv/pghero/dexter/key | sudo apt-key add -
-sudo wget -O /etc/apt/sources.list.d/dexter.list \
-  https://dl.packager.io/srv/pghero/dexter/master/installer/ubuntu/16.04.repo
-sudo apt-get update
-sudo apt-get -y install dexter
-```
-
-### Ubuntu 14.04 (Trusty)
-
-```sh
-wget -qO- https://dl.packager.io/srv/pghero/dexter/key | sudo apt-key add -
-sudo wget -O /etc/apt/sources.list.d/dexter.list \
-  https://dl.packager.io/srv/pghero/dexter/master/installer/ubuntu/14.04.repo
-sudo apt-get update
-sudo apt-get install dexter
-```
-
-### Debian 8 (Jesse)
-
-```sh
-wget -qO- https://dl.packager.io/srv/pghero/dexter/key | sudo apt-key add -
-sudo wget -O /etc/apt/sources.list.d/dexter.list \
-  https://dl.packager.io/srv/pghero/dexter/master/installer/debian/8.repo
-sudo apt-get update
-sudo apt-get install dexter
-```
-
-### CentOS / RHEL 7
-
-```sh
-sudo wget -O /etc/yum.repos.d/dexter.repo \
-  https://dl.packager.io/srv/pghero/dexter/master/installer/el/7.repo
-sudo yum install dexter
-```
-
-### SUSE Linux Enterprise Server 12
-
-```sh
-sudo wget -O /etc/zypp/repos.d/dexter.repo \
-  https://dl.packager.io/srv/pghero/dexter/master/installer/sles/12.repo
-sudo zypper install dexter
-```
-
-## Credits
-
-:heart: Made possible by [Packager](https://packager.io/)

data/pgdexter.gemspec

@@ -1,30 +0,0 @@
-# coding: utf-8
-
-lib = File.expand_path("../lib", __FILE__)
-$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
-require "dexter/version"
-
-Gem::Specification.new do |spec|
-  spec.name          = "pgdexter"
-  spec.version       = Dexter::VERSION
-  spec.authors       = ["Andrew Kane"]
-  spec.email         = ["andrew@chartkick.com"]
-
-  spec.summary       = "The automatic indexer for Postgres"
-  spec.homepage      = "https://github.com/ankane/dexter"
-
-  spec.files         = `git ls-files -z`.split("\x0").reject do |f|
-    f.match(%r{^(test|spec|features)/})
-  end
-  spec.bindir        = "exe"
-  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
-  spec.require_paths = ["lib"]
-
-  spec.add_dependency "slop", ">= 4.2.0"
-  spec.add_dependency "pg"
-  spec.add_dependency "pg_query"
-
-  spec.add_development_dependency "bundler"
-  spec.add_development_dependency "rake"
-  spec.add_development_dependency "minitest"
-end