pgdexter 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: c513808e2cdd9690c477c30548979fad122575a4
-  data.tar.gz: 6f4c605a0b68baa0275a523155b7be4ea9bc4d56
+  metadata.gz: 16f61626f5003a801ec12cda3f663390a57e69d8
+  data.tar.gz: 238ee1b24dab3f473accc2ca9a5156fd31e8d19e
 SHA512:
-  metadata.gz: 3b38a53e96516a485394f3ec594ee0418c1a7ab94fe8726b3228cc5443843dd7f7ba2bd1fe4bb0cc5c4dbd877bd9b9b036d5bfd5b4cda275dc7176dbdf9da82c
-  data.tar.gz: cbe64d0cf9a40b96bf2644a80519bb837d235d6956ca9b61b24533867130958c95932ffbf4d69672218a10628c32b87a1f1e87f86a4968577f1629fc871ef440
+  metadata.gz: ccae6266196fc84ea0fc69177b80144f8747996ff0693b6a7492e34d095b2cb400e9ce3299741326c50a5eb267ef48be368384a2ab27553002eb93ac7b6d0922
+  data.tar.gz: 944bc27bb1ffff546c2f33aa10aa1b8371ba43ae2be889a6b9445c3d3a93ca36fee0cfd57487134a5ea807e37583f74b1e5e9de7714fa30364ab28f64a5679e8

data/CHANGELOG.md

@@ -1,3 +1,16 @@
+## 0.3.0
+
+- Added support for schemas
+- Added support for csv format
+- Added `--analyze` option and do not analyze by default
+- Added `--min-calls` option
+- Fixed debug output when indexes not found
+
+## 0.2.1
+
+- Fixed bad suggestions
+- Improved debugging output
+
 ## 0.2.0
 
 - Added same connection options as `psql`

data/README.md

@@ -12,8 +12,8 @@ First, install [HypoPG](https://github.com/dalibo/hypopg) on your database serve
 
 ```sh
 cd /tmp
-curl -L https://github.com/dalibo/hypopg/archive/1.0.0.tar.gz | tar xz
-cd hypopg-1.0.0
+curl -L https://github.com/dalibo/hypopg/archive/1.1.0.tar.gz | tar xz
+cd hypopg-1.1.0
 make
 make install # may need sudo
 ```
@@ -45,23 +45,23 @@ tail -F -n +1 <log-file> | dexter <connection-options>
 This finds slow queries and generates output like:
 
 ```
-2017-06-25T17:52:19+00:00 Started
-2017-06-25T17:52:22+00:00 Processing 189 new query fingerprints
-2017-06-25T17:52:22+00:00 Index found: genres_movies (genre_id)
-2017-06-25T17:52:22+00:00 Index found: genres_movies (movie_id)
-2017-06-25T17:52:22+00:00 Index found: movies (title)
-2017-06-25T17:52:22+00:00 Index found: ratings (movie_id)
-2017-06-25T17:52:22+00:00 Index found: ratings (rating)
-2017-06-25T17:52:22+00:00 Index found: ratings (user_id)
-2017-06-25T17:53:22+00:00 Processing 12 new query fingerprints
+Started
+Processing 189 new query fingerprints
+Index found: public.genres_movies (genre_id)
+Index found: public.genres_movies (movie_id)
+Index found: public.movies (title)
+Index found: public.ratings (movie_id)
+Index found: public.ratings (rating)
+Index found: public.ratings (user_id)
+Processing 12 new query fingerprints
 ```
 
 To be safe, Dexter will not create indexes unless you pass the `--create` flag. In this case, you’ll see:
 
 ```
-2017-06-25T17:52:22+00:00 Index found: ratings (user_id)
-2017-06-25T17:52:22+00:00 Creating index: CREATE INDEX CONCURRENTLY ON "ratings" ("user_id")
-2017-06-25T17:52:37+00:00 Index created: 15243 ms
+Index found: public.ratings (user_id)
+Creating index: CREATE INDEX CONCURRENTLY ON "public"."ratings" ("user_id")
+Index created: 15243 ms
 ```
 
 ## Connection Options
@@ -84,30 +84,58 @@ and connection strings:
 host=localhost port=5432 dbname=mydb
 ```
 
-## Options
+## Collecting Queries
 
-Name | Description | Default
---- | --- | ---
-exclude | prevent specific tables from being indexed | None
-interval | time to wait between processing queries, in seconds | 60
-log-level | `debug` gives additional info for suggested indexes<br />`debug2` gives additional info for processed queries<br />`error` suppresses logging | info
-log-sql | log SQL statements executed | false
-min-time | only process queries consuming a min amount of DB time, in minutes | 0
+There are many ways to collect queries. For real-time indexing, pipe your logfile:
 
-## Non-Streaming Modes
+```sh
+tail -F -n +1 <log-file> | dexter <connection-options>
+```
 
-You can pass a single statement with:
+Pass a single statement with:
 
 ```sh
 dexter <connection-options> -s "SELECT * FROM ..."
 ```
 
-or files with:
+or pass files:
 
 ```sh
 dexter <connection-options> <file1> <file2>
 ```
 
+or use the [pg_stat_statements](https://www.postgresql.org/docs/current/static/pgstatstatements.html) extension:
+
+```sh
+dexter <connection-options> --pg-stat-statements
+```
+
+### Collection Options
+
+To prevent one-off queries from being indexed, specify a minimum number of calls before a query is considered for indexing
+
+```sh
+dexter --min-calls 100
+```
+
+You can do the same for total time a query has run
+
+```sh
+dexter --min-time 10 # minutes
+```
+
+Specify the format
+
+```sh
+dexter --input-format csv
+```
+
+When steaming logs, specify the time to wait between processing queries
+
+```sh
+dexter --interval 60 # seconds
+```
+
 ## Examples
 
 Ubuntu with PostgreSQL 9.6
@@ -122,6 +150,36 @@ Homebrew on Mac
 tail -F -n +1 /usr/local/var/postgres/server.log | dexter dbname
 ```
 
+## Tables
+
+You can exclude large or write-heavy tables from indexing with:
+
+```sh
+dexter --exclude table1,table2
+```
+
+Alternatively, you can specify which tables to index with:
+
+```sh
+dexter --include table3,table4
+```
+
+## Debugging
+
+See how Dexter is processing queries with:
+
+```sh
+dexter --log-sql --log-level debug2
+```
+
+## Analyze
+
+For best results, make sure your tables have been recently analyzed so statistics are up-to-date. You can ask Dexter to analyze tables it comes across that haven’t been analyzed in the past hour with:
+
+```sh
+dexter --analyze
+```
+
 ## Hosted Postgres
 
 Some hosted providers like Amazon RDS and Heroku do not support the HypoPG extension, which Dexter needs to run. See [how to use Dexter](guides/Hosted-Postgres.md) in these cases.
@@ -130,6 +188,21 @@ Some hosted providers like Amazon RDS and Heroku do not support the HypoPG exten
 
 [Here are some ideas](https://github.com/ankane/dexter/issues/1)
 
+## Upgrading
+
+Run:
+
+```sh
+gem install pgdexter
+```
+
+To use master, run:
+
+```sh
+gem install specific_install
+gem specific_install https://github.com/ankane/dexter.git
+```
+
 ## Thanks
 
 This software wouldn’t be possible without [HypoPG](https://github.com/dalibo/hypopg), which allows you to create hypothetical indexes, and [pg_query](https://github.com/lfittl/pg_query), which allows you to parse and fingerprint queries. A big thanks to Dalibo and Lukas Fittl respectively.

data/guides/Hosted-Postgres.md

@@ -8,12 +8,12 @@ Linux
 
 ```sh
 sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
-sudo apt-get install wget ca-certificates
+sudo apt-get install -y wget ca-certificates
 wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
 sudo apt-get update
-sudo apt-get install postgresql-9.6 postgresql-server-dev-9.6
+sudo apt-get install -y postgresql-9.6 postgresql-server-dev-9.6
 sudo -u postgres createuser $(whoami) -s
-sudo apt-get install ruby2.2 ruby2.2-dev
+sudo apt-get install -y ruby2.2 ruby2.2-dev
 ```
 
 Mac
@@ -83,7 +83,7 @@ heroku logs -p postgres > postgresql.log
 We recommend creating a new instance from a snapshot for the dump to avoid affecting customers.
 
 ```sh
-pg_dump -v -j 8 -Fd -f /tmp/newout.dir <connection-string>
+pg_dump -v -j 8 -Fd -f /tmp/newout.dir <connection-options>
 ```
 
 Then shutdown the dump instance. Restore with:
@@ -96,7 +96,7 @@ pg_restore -v -j 8 -x -O --format=d -d dexter_restore /tmp/newout.dir/
 ### Run Dexter
 
 ```sh
-dexter dexter_restore postgresql.log*
+dexter dexter_restore postgresql.log* --analyze
 ```
 
 :tada:

data/lib/dexter/client.rb

@@ -29,16 +29,20 @@ module Dexter
     dexter [options]
 
 Options:)
+        o.boolean "--analyze", "analyze tables that haven't been analyzed in the past hour", default: false
         o.boolean "--create", "create indexes", default: false
         o.array "--exclude", "prevent specific tables from being indexed"
         o.string "--include", "only include specific tables"
+        o.string "--input-format", "input format", default: "stderr"
         o.integer "--interval", "time to wait between processing queries, in seconds", default: 60
+        o.float "--min-calls", "only process queries that have been called a certain number of times", default: 0
         o.float "--min-time", "only process queries that have consumed a certain amount of DB time, in minutes", default: 0
         o.boolean "--pg-stat-statements", "use pg_stat_statements", default: false, help: false
         o.boolean "--log-explain", "log explain", default: false, help: false
         o.string "--log-level", "log level", default: "info"
         o.boolean "--log-sql", "log sql", default: false
         o.string "-s", "--statement", "process a single statement"
+        # separator must go here to show up correctly - slop bug?
         o.separator ""
         o.separator "Connection options:"
         o.on "-v", "--version", "print the version" do
@@ -62,7 +66,7 @@ Options:)
 
       # TODO don't use global var
       $log_level = options[:log_level].to_s.downcase
-      abort "Unknown log level" unless ["error", "info", "debug", "debug2"].include?($log_level)
+      abort "Unknown log level" unless ["error", "info", "debug", "debug2", "debug3"].include?($log_level)
 
       [arguments, options]
     rescue Slop::Error => e

data/lib/dexter/collector.rb

@@ -5,6 +5,7 @@ module Dexter
       @new_queries = Set.new
       @mutex = Mutex.new
       @min_time = options[:min_time] * 60000 # convert minutes to ms
+      @min_calls = options[:min_calls]
     end
 
     def add(query, duration)
@@ -36,8 +37,11 @@ module Dexter
 
       queries = []
       @top_queries.each do |k, v|
-        if new_queries.include?(k) && v[:total_time] > @min_time
-          queries << Query.new(v[:query], k)
+        if new_queries.include?(k) && v[:total_time] >= @min_time && v[:calls] >= @min_calls
+          query = Query.new(v[:query], k)
+          query.total_time = v[:total_time]
+          query.calls = v[:calls]
+          queries << query
         end
       end
 

data/lib/dexter/csv_log_parser.rb

@@ -0,0 +1,13 @@
+require "csv"
+
+module Dexter
+  class CsvLogParser < LogParser
+    def perform
+      CSV.foreach(@logfile.file) do |row|
+        if (m = REGEX.match(row[13]))
+          process_entry(m[3], m[1].to_f)
+        end
+      end
+    end
+  end
+end

data/lib/dexter/indexer.rb

@@ -10,6 +10,8 @@ module Dexter
       @log_sql = options[:log_sql]
       @log_explain = options[:log_explain]
       @min_time = options[:min_time] || 0
+      @min_calls = options[:min_calls] || 0
+      @analyze = options[:analyze]
       @options = options
 
       create_extension unless extension_exists?
@@ -26,24 +28,39 @@ module Dexter
       # reset hypothetical indexes
       reset_hypothetical_indexes
 
-      # filter queries from other databases and system tables
-      tables = possible_tables(queries)
-      queries.each do |query|
-        query.missing_tables = !query.tables.all? { |t| tables.include?(t) }
-      end
+      tables = Set.new(database_tables)
 
       if @include_tables
-        tables = Set.new(tables.to_a & @include_tables)
+        include_set = Set.new(@include_tables)
+        tables.keep_if { |t| include_set.include?(t) || include_set.include?(t.split(".")[-1]) }
+      end
+
+      if @exclude_tables.any?
+        exclude_set = Set.new(@exclude_tables)
+        tables.delete_if { |t| exclude_set.include?(t) || exclude_set.include?(t.split(".")[-1]) }
       end
 
-      # exclude user specified tables
-      # TODO exclude write-heavy tables
-      @exclude_tables.each do |table|
-        tables.delete(table)
+      # map tables without schema to schema
+      no_schema_tables = {}
+      search_path_index = Hash[search_path.map.with_index.to_a]
+      tables.group_by { |t| t.split(".")[-1] }.each do |group, t2|
+        no_schema_tables[group] = t2.sort_by { |t| search_path_index[t.split(".")[0]] || 1000000 }[0]
+      end
+
+      # filter queries from other databases and system tables
+      queries.each do |query|
+        # add schema to table if needed
+        query.tables = query.tables.map { |t| no_schema_tables[t] || t }
+
+        # check for missing tables
+        query.missing_tables = !query.tables.all? { |t| tables.include?(t) }
       end
 
+      # set tables
+      tables = Set.new(queries.reject(&:missing_tables).flat_map(&:tables))
+
       # analyze tables if needed
-      analyze_tables(tables) if tables.any?
+      analyze_tables(tables) if tables.any? && (@analyze || @log_level == "debug2")
 
       # create hypothetical indexes and explain queries
       candidates = tables.any? ? create_hypothetical_indexes(queries.reject(&:missing_tables), tables) : {}
@@ -52,7 +69,7 @@ module Dexter
       new_indexes = determine_indexes(queries, candidates, tables)
 
       # display and create new indexes
-      show_and_create_indexes(new_indexes)
+      show_and_create_indexes(new_indexes, queries, tables)
     end
 
     private
@@ -81,14 +98,13 @@ module Dexter
 
       analyze_stats = execute <<-SQL
         SELECT
-          schemaname AS schema,
-          relname AS table,
+          schemaname || '.' || relname AS table,
           last_analyze,
           last_autoanalyze
         FROM
           pg_stat_user_tables
         WHERE
-          relname IN (#{tables.map { |t| quote(t) }.join(", ")})
+          schemaname || '.' || relname IN (#{tables.map { |t| quote(t) }.join(", ")})
       SQL
 
       last_analyzed = {}
@@ -97,7 +113,14 @@ module Dexter
       end
 
       tables.each do |table|
-        if !last_analyzed[table] || last_analyzed[table] < Time.now - 3600
+        la = last_analyzed[table]
+
+        if @log_level == "debug2"
+          time_str = la ? la.iso8601 : "Unknown"
+          log "Last analyze: #{table} : #{time_str}"
+        end
+
+        if @analyze && (!la || la < Time.now - 3600)
           statement = "ANALYZE #{quote_ident(table)}"
           log "Running analyze: #{statement}"
           execute(statement)
@@ -133,17 +156,11 @@ module Dexter
       tables = Set.new(explainable_queries.flat_map(&:tables))
 
       if tables.any?
-        # get existing indexes
-        index_set = Set.new
-        indexes(tables).each do |index|
-          # TODO make sure btree
-          index_set << [index["table"], index["columns"]]
-        end
-
         # since every set of multi-column indexes are expensive
         # try to parse out columns
         possible_columns = Set.new
         explainable_queries.each do |query|
+          log "Finding columns: #{query.statement}" if @log_level == "debug3"
           find_columns(query.tree).each do |col|
             last_col = col["fields"].last
             if last_col["String"]
@@ -156,13 +173,13 @@ module Dexter
         columns_by_table = columns(tables).select { |c| possible_columns.include?(c[:column]) }.group_by { |c| c[:table] }
 
         # create single column indexes
-        create_hypothetical_indexes_helper(columns_by_table, 1, index_set, candidates)
+        create_hypothetical_indexes_helper(columns_by_table, 1, candidates)
 
         # get next round of costs
         calculate_plan(explainable_queries)
 
         # create multicolumn indexes
-        create_hypothetical_indexes_helper(columns_by_table, 2, index_set, candidates)
+        create_hypothetical_indexes_helper(columns_by_table, 2, candidates)
 
         # get next round of costs
         calculate_plan(explainable_queries)
@@ -196,56 +213,145 @@ module Dexter
       indexes
     end
 
+    def hypo_indexes_from_plan(index_name_to_columns, plan, index_set)
+      query_indexes = []
+
+      find_indexes(plan).uniq.sort.each do |index_name|
+        col_set = index_name_to_columns[index_name]
+
+        if col_set
+          index = {
+            table: col_set[0][:table],
+            columns: col_set.map { |c| c[:column] }
+          }
+
+          unless index_set.include?([index[:table], index[:columns]])
+            query_indexes << index
+          end
+        end
+      end
+
+      query_indexes
+    end
+
     def determine_indexes(queries, candidates, tables)
       new_indexes = {}
       index_name_to_columns = candidates.invert
 
+      # filter out existing indexes
+      # this must happen at end of process
+      # since sometimes hypothetical indexes
+      # can give lower cost than actual indexes
+      index_set = Set.new
+      if tables.any?
+        indexes(tables).each do |index|
+          if index["using"] == "btree"
+            # don't add indexes that are already covered
+            index_set << [index["table"], index["columns"].first(1)]
+            index_set << [index["table"], index["columns"].first(2)]
+          end
+        end
+      end
+
       queries.each do |query|
         if query.explainable? && query.high_cost?
           new_cost, new_cost2 = query.costs[1..2]
 
           cost_savings = new_cost < query.initial_cost * 0.5
+
           # set high bar for multicolumn indexes
           cost_savings2 = new_cost > 100 && new_cost2 < new_cost * 0.5
 
-          query.new_cost = cost_savings2 ? new_cost2 : new_cost
-
-          query_indexes = []
           key = cost_savings2 ? 2 : 1
-          indexes = find_indexes(query.plans[key]).uniq.sort
+          query_indexes = hypo_indexes_from_plan(index_name_to_columns, query.plans[key], index_set)
 
-          indexes.each do |index_name|
-            col_set = index_name_to_columns[index_name]
+          # likely a bad suggestion, so try single column
+          if cost_savings2 && query_indexes.size > 1
+            query_indexes = hypo_indexes_from_plan(index_name_to_columns, query.plans[1], index_set)
+            cost_savings2 = false
+          end
 
-            if col_set
-              index = {
-                table: col_set[0][:table],
-                columns: col_set.map { |c| c[:column] }
-              }
-              query_indexes << index
+          # TODO if multiple indexes are found (for either single or multicolumn)
+          # determine the impact of each individually
+          # for now, be conservative and don't suggest if more than one index
+          suggest_index = (cost_savings || cost_savings2) && query_indexes.size == 1
 
-              if cost_savings
-                new_indexes[index] ||= index.dup
-                (new_indexes[index][:queries] ||= []) << query
-              end
+          if suggest_index
+            query_indexes.each do |index|
+              new_indexes[index] ||= index.dup
+              (new_indexes[index][:queries] ||= []) << query
             end
           end
+
+          query.indexes = query_indexes
+          query.suggest_index = suggest_index
+          query.new_cost =
+            if suggest_index
+              cost_savings2 ? new_cost2 : new_cost
+            else
+              query.initial_cost
+            end
+
+          # TODO optimize
+          if @log_level.start_with?("debug")
+            query.pass1_indexes = hypo_indexes_from_plan(index_name_to_columns, query.plans[1], index_set)
+            query.pass2_indexes = hypo_indexes_from_plan(index_name_to_columns, query.plans[2], index_set)
+          end
+        end
+      end
+
+      # filter out covered indexes
+      covered = Set.new
+      new_indexes.values.each do |index|
+        if index[:columns].size > 1
+          covered << [index[:table], index[:columns].first(1)]
         end
+      end
 
+      new_indexes.values.reject { |i| covered.include?([i[:table], i[:columns]]) }.sort_by(&:to_a)
+    end
+
+    def log_indexes(indexes)
+      if indexes.any?
+        indexes.map { |i| "#{i[:table]} (#{i[:columns].join(", ")})" }.join(", ")
+      else
+        "None"
+      end
+    end
+
+    def show_and_create_indexes(new_indexes, queries, tables)
+      # print summary
+      if new_indexes.any?
+        new_indexes.each do |index|
+          log "Index found: #{index[:table]} (#{index[:columns].join(", ")})"
+        end
+      else
+        log "No new indexes found"
+      end
+
+      # debug info
+      if @log_level.start_with?("debug")
+        index_queries = new_indexes.flat_map { |i| i[:queries].sort_by(&:fingerprint) }
         if @log_level == "debug2"
-          log "Processed #{query.fingerprint}"
+          fingerprints = Set.new(index_queries.map(&:fingerprint))
+          index_queries.concat(queries.reject { |q| fingerprints.include?(q.fingerprint) }.sort_by(&:fingerprint))
+        end
+        index_queries.each do |query|
+          log "-" * 80
+          log "Query #{query.fingerprint}"
+          log "Total time: #{(query.total_time / 60000.0).round(1)} min, avg time: #{(query.total_time / query.calls.to_f).round} ms, calls: #{query.calls}" if query.total_time
           if tables.empty?
             log "No candidate tables for indexes"
           elsif query.explainable? && !query.high_cost?
             log "Low initial cost: #{query.initial_cost}"
           elsif query.explainable?
-            log "Cost: #{query.initial_cost} -> #{query.new_cost}"
-
-            if query_indexes.any?
-              log "Indexes: #{query_indexes.map { |i| "#{i[:table]} (#{i[:columns].join(", ")})" }.join(", ")}"
-              log "Need 50% cost savings to suggest index" unless cost_savings || cost_savings2
-            else
-              log "Indexes: None"
+            query_indexes = query.indexes || []
+            log "Start: #{query.costs[0]}"
+            log "Pass1: #{query.costs[1]} : #{log_indexes(query.pass1_indexes || [])}"
+            log "Pass2: #{query.costs[2]} : #{log_indexes(query.pass2_indexes || [])}"
+            log "Final: #{query.new_cost} : #{log_indexes(query.suggest_index ? query_indexes : [])}"
+            if query_indexes.any? && !query.suggest_index
+              log "Need 50% cost savings to suggest index"
             end
           elsif query.fingerprint == "unknown"
             log "Could not parse query"
@@ -256,54 +362,33 @@ module Dexter
           else
             log "Could not run explain"
           end
-
-          puts
-          puts query.statement
-          puts
+          log
+          log query.statement
+          log
         end
       end
 
-      new_indexes.values.sort_by(&:to_a)
-    end
-
-    def show_and_create_indexes(new_indexes)
-      if new_indexes.any?
-        new_indexes.each do |index|
-          log "Index found: #{index[:table]} (#{index[:columns].join(", ")})"
-
-          if @log_level.start_with?("debug")
-            index[:queries].sort_by(&:fingerprint).each do |query|
-              log "Query #{query.fingerprint} (Cost: #{query.initial_cost} -> #{query.new_cost})"
-              puts
-              puts query.statement
-              puts
-            end
-          end
-        end
-
-        if @create
-          # 1. create lock
-          # 2. refresh existing index list
-          # 3. create indexes that still don't exist
-          # 4. release lock
-          with_advisory_lock do
-            new_indexes.each do |index|
-              unless index_exists?(index)
-                statement = "CREATE INDEX CONCURRENTLY ON #{quote_ident(index[:table])} (#{index[:columns].map { |c| quote_ident(c) }.join(", ")})"
-                log "Creating index: #{statement}"
-                started_at = Time.now
-                begin
-                  execute(statement)
-                  log "Index created: #{((Time.now - started_at) * 1000).to_i} ms"
-                rescue PG::LockNotAvailable => e
-                  log "Could not acquire lock: #{index[:table]}"
-                end
+      # create
+      if @create && new_indexes.any?
+        # 1. create lock
+        # 2. refresh existing index list
+        # 3. create indexes that still don't exist
+        # 4. release lock
+        with_advisory_lock do
+          new_indexes.each do |index|
+            unless index_exists?(index)
+              statement = "CREATE INDEX CONCURRENTLY ON #{quote_ident(index[:table])} (#{index[:columns].map { |c| quote_ident(c) }.join(", ")})"
+              log "Creating index: #{statement}"
+              started_at = Time.now
+              begin
+                execute(statement)
+                log "Index created: #{((Time.now - started_at) * 1000).to_i} ms"
+              rescue PG::LockNotAvailable
+                log "Could not acquire lock: #{index[:table]}"
               end
             end
           end
         end
-      else
-        log "No new indexes found"
       end
 
       new_indexes
@@ -347,13 +432,11 @@ module Dexter
     end
 
     # TODO for multicolumn indexes, use ordering
-    def create_hypothetical_indexes_helper(columns_by_table, n, index_set, candidates)
+    def create_hypothetical_indexes_helper(columns_by_table, n, candidates)
       columns_by_table.each do |table, cols|
         # no reason to use btree index for json columns
         cols.reject { |c| ["json", "jsonb"].include?(c[:type]) }.permutation(n) do |col_set|
-          if !index_set.include?([table, col_set.map { |col| col[:column] }])
-            candidates[col_set] = execute("SELECT * FROM hypopg_create_index('CREATE INDEX ON #{quote_ident(table)} (#{col_set.map { |c| quote_ident(c[:column])  }.join(", ")})')").first["indexname"]
-          end
+          candidates[col_set] = execute("SELECT * FROM hypopg_create_index('CREATE INDEX ON #{quote_ident(table)} (#{col_set.map { |c| quote_ident(c[:column])  }.join(", ")})')").first["indexname"]
         end
       end
     end
@@ -361,7 +444,7 @@ module Dexter
     def database_tables
       result = execute <<-SQL
         SELECT
-          table_name
+          table_schema || '.' || table_name AS table_name
         FROM
           information_schema.tables
         WHERE
@@ -383,16 +466,13 @@ module Dexter
         WHERE
           datname = current_database()
           AND total_time >= #{@min_time * 60000}
+          AND calls >= #{@min_calls}
         ORDER BY
           1
       SQL
       result.map { |q| q["query"] }
     end
 
-    def possible_tables(queries)
-      Set.new(queries.flat_map(&:tables).uniq & database_tables)
-    end
-
     def with_advisory_lock
       lock_id = 123456
       first_time = true
@@ -424,14 +504,13 @@ module Dexter
     def columns(tables)
       columns = execute <<-SQL
         SELECT
-          table_name,
+          table_schema || '.' || table_name AS table_name,
           column_name,
           data_type
         FROM
           information_schema.columns
         WHERE
-          table_schema = 'public' AND
-          table_name IN (#{tables.map { |t| quote(t) }.join(", ")})
+          table_schema || '.' || table_name IN (#{tables.map { |t| quote(t) }.join(", ")})
         ORDER BY
           1, 2
       SQL
@@ -442,8 +521,7 @@ module Dexter
     def indexes(tables)
       execute(<<-SQL
         SELECT
-          schemaname AS schema,
-          t.relname AS table,
+          schemaname || '.' || t.relname AS table,
           ix.relname AS name,
           regexp_replace(pg_get_indexdef(i.indexrelid), '^[^\\(]*\\((.*)\\)$', '\\1') AS columns,
           regexp_replace(pg_get_indexdef(i.indexrelid), '.* USING ([^ ]*) \\(.*', '\\1') AS using
@@ -456,8 +534,7 @@ module Dexter
         LEFT JOIN
           pg_stat_user_indexes ui ON ui.indexrelid = i.indexrelid
         WHERE
-          t.relname IN (#{tables.map { |t| quote(t) }.join(", ")}) AND
-          schemaname IS NOT NULL AND
+          schemaname || '.' || t.relname IN (#{tables.map { |t| quote(t) }.join(", ")}) AND
           indisvalid = 't' AND
           indexprs IS NULL AND
           indpred IS NULL
@@ -467,8 +544,12 @@ module Dexter
       ).map { |v| v["columns"] = v["columns"].sub(") WHERE (", " WHERE ").split(", ").map { |c| unquote(c) }; v }
     end
 
+    def search_path
+      execute("SHOW search_path")[0]["search_path"].split(",").map(&:strip)
+    end
+
     def unquote(part)
-      if part && part.start_with?('"')
+      if part && part.start_with?('"') && part.end_with?('"')
         part[1..-2]
       else
         part
@@ -476,7 +557,7 @@ module Dexter
     end
 
     def quote_ident(value)
-      conn.quote_ident(value)
+      value.split(".").map { |v| conn.quote_ident(v) }.join(".")
     end
 
     def quote(value)

data/lib/dexter/log_parser.rb

@@ -1,6 +1,6 @@
 module Dexter
   class LogParser
-    REGEX = /duration: (\d+\.\d+) ms  (statement|execute <unnamed>): (.+)/
+    REGEX = /duration: (\d+\.\d+) ms  (statement|execute <unnamed>|parse <unnamed>): (.+)/
     LINE_SEPERATOR = ":  ".freeze
 
     def initialize(logfile, collector)
@@ -12,7 +12,7 @@ module Dexter
       active_line = nil
       duration = nil
 
-      each_line do |line|
+      @logfile.each_line do |line|
         if active_line
           if line.include?(LINE_SEPERATOR)
             process_entry(active_line, duration)
@@ -22,7 +22,7 @@ module Dexter
           end
         end
 
-        if !active_line && m = REGEX.match(line.chomp)
+        if !active_line && (m = REGEX.match(line.chomp))
           duration = m[1].to_f
           active_line = m[3]
         end
@@ -32,12 +32,6 @@ module Dexter
 
     private
 
-    def each_line
-      @logfile.each_line do |line|
-        yield line
-      end
-    end
-
     def process_entry(query, duration)
       @collector.add(query, duration)
     end

data/lib/dexter/logging.rb

@@ -1,7 +1,7 @@
 module Dexter
   module Logging
-    def log(message)
-      puts "#{Time.now.iso8601} #{message}" unless $log_level == "error"
+    def log(message = "")
+      puts message unless $log_level == "error"
     end
   end
 end

data/lib/dexter/processor.rb

@@ -5,8 +5,14 @@ module Dexter
     def initialize(logfile, options)
       @logfile = logfile
 
-      @collector = Collector.new(min_time: options[:min_time])
-      @log_parser = LogParser.new(logfile, @collector)
+      @collector = Collector.new(min_time: options[:min_time], min_calls: options[:min_calls])
+      @log_parser =
+        if options[:input_format] == "csv"
+          CsvLogParser.new(logfile, @collector)
+        else
+          LogParser.new(logfile, @collector)
+        end
+
       @indexer = Indexer.new(options)
 
       @starting_interval = 3

data/lib/dexter/query.rb

@@ -1,7 +1,8 @@
 module Dexter
   class Query
     attr_reader :statement, :fingerprint, :plans
-    attr_accessor :missing_tables, :new_cost
+    attr_writer :tables
+    attr_accessor :missing_tables, :new_cost, :total_time, :calls, :indexes, :suggest_index, :pass1_indexes, :pass2_indexes
 
     def initialize(statement, fingerprint = nil)
       @statement = statement
@@ -13,7 +14,15 @@ module Dexter
     end
 
     def tables
-      @tables ||= parse ? parse.tables : []
+      @tables ||= begin
+        parse ? parse.tables : []
+      rescue => e
+        # possible pg_query bug
+        $stderr.puts "Error extracting tables. Please report to https://github.com/ankane/dexter/issues"
+        $stderr.puts "#{e.class.name}: #{e.message}"
+        $stderr.puts statement
+        []
+      end
     end
 
     def tree

data/lib/dexter/version.rb

@@ -1,3 +1,3 @@
 module Dexter
-  VERSION = "0.2.0"
+  VERSION = "0.3.0"
 end

data/lib/dexter.rb

@@ -10,5 +10,6 @@ require "dexter/client"
 require "dexter/collector"
 require "dexter/indexer"
 require "dexter/log_parser"
+require "dexter/csv_log_parser"
 require "dexter/processor"
 require "dexter/query"

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: pgdexter
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.0
 platform: ruby
 authors:
 - Andrew Kane
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2017-08-28 00:00:00.000000000 Z
+date: 2017-12-23 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: slop
@@ -115,6 +115,7 @@ files:
 - lib/dexter.rb
 - lib/dexter/client.rb
 - lib/dexter/collector.rb
+- lib/dexter/csv_log_parser.rb
 - lib/dexter/indexer.rb
 - lib/dexter/log_parser.rb
 - lib/dexter/logging.rb
@@ -141,7 +142,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.6.11
+rubygems_version: 2.6.13
 signing_key: 
 specification_version: 4
 summary: The automatic indexer for Postgres