pgdexter 0.3.3 → 0.3.8

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 14bc122b136301535793c29b19336a2d0614f640
-  data.tar.gz: 624c8c6ae5aabd8a5e2150887aa83efe6e67f9b4
+SHA256:
+  metadata.gz: 3940bf892014f2337cc5cd95c85b66f245b3725ad7ce84397c4eb4f9e3751c3c
+  data.tar.gz: b8fc834cda0161c8ef0b44853c14f5f608c00f373e6a95481321e45611d2d5b4
 SHA512:
-  metadata.gz: 44224c687d8590c0441d587b432fde3b85cef77728980059d497ade992cd953e8f8445747de61c984d559f55d74461ac284c6c8423c0f1f8b08e439e361c141d
-  data.tar.gz: 42ec40ec7527dfc3093853ec013c5b30bacbaee5cbe8d27a1530649bfbdc16c639d9d93d65e72b062d53b33bff1d0a1bec67e971d283f5d5a2e6fea7d884fccd
+  metadata.gz: 463430e69b2a1f08ca411db0a4938c7243839e78f81db72455d508ff1e9de1a752889bff8a3e42486cf4859042065b0847f0b6c0216c80c9b9a36a10cf360753
+  data.tar.gz: d31949fc719e863aa56672fd807141dfbeb521d2ce2525ca551109e09840115c3091e7130cc47617a6b836ff31853dbf12a1bb9fb4dea492b17cbf7c9bfd5c8e

data/CHANGELOG.md

@@ -1,22 +1,47 @@
-## 0.3.3
+## 0.3.8 (2020-08-17)
+
+- Colorize output
+- Fixed error when unable to parse view definitions
+
+## 0.3.7 (2020-07-10)
+
+- Fixed help output
+
+## 0.3.6 (2020-03-30)
+
+- Fixed warning with Ruby 2.7
+
+## 0.3.5 (2018-04-30)
+
+- Added `sql` input format
+- Fixed error for queries with double dash comments
+- Fixed connection threading issue with `--pg-stat-activity` option
+
+## 0.3.4 (2018-04-09)
+
+- Fixed `--username` option
+- Fixed `JSON::NestingError`
+- Added `--pg-stat-activity` option
+
+## 0.3.3 (2018-02-22)
 
 - Added support for views and materialized views
 - Better handle case when multiple indexes are found for a query
 - Added `--min-cost-savings-pct` option
 
-## 0.3.2
+## 0.3.2 (2018-01-04)
 
 - Fixed parsing issue with named prepared statements
 - Fixed parsing issue with multiline queries in csv format
 - Better explanations for indexing decisions
 
-## 0.3.1
+## 0.3.1 (2017-12-28)
 
 - Added support for queries with bind variables
 - Fixed error with streaming logs as csv format
 - Handle malformed CSV gracefully
 
-## 0.3.0
+## 0.3.0 (2017-12-22)
 
 - Added support for schemas
 - Added support for csv format
@@ -24,12 +49,12 @@
 - Added `--min-calls` option
 - Fixed debug output when indexes not found
 
-## 0.2.1
+## 0.2.1 (2017-09-02)
 
 - Fixed bad suggestions
 - Improved debugging output
 
-## 0.2.0
+## 0.2.0 (2017-08-27)
 
 - Added same connection options as `psql`
 - Added support for multiple files
@@ -40,38 +65,38 @@ Breaking
 
 - `-h` option changed to `--host` instead of `--help` for consistency with `psql`
 
-## 0.1.6
+## 0.1.6 (2017-08-26)
 
 - Significant performance improvements
 - Added `--include` option
 
-## 0.1.5
+## 0.1.5 (2017-08-14)
 
 - Added support for non-`SELECT` queries
 - Added `--pg-stat-statements` option
 - Added advisory locks
 - Added support for running as a non-superuser
 
-## 0.1.4
+## 0.1.4 (2017-07-02)
 
 - Added support for multicolumn indexes
 
-## 0.1.3
+## 0.1.3 (2017-06-30)
 
 - Fixed error with non-lowercase columns
 - Fixed error with `json` columns
 
-## 0.1.2
+## 0.1.2 (2017-06-26)
 
 - Added `--exclude` option
 - Added `--log-sql` option
 
-## 0.1.1
+## 0.1.1 (2017-06-25)
 
 - Added `--interval` option
 - Added `--min-time` option
 - Added `--log-level` option
 
-## 0.1.0
+## 0.1.0 (2017-06-24)
 
 - Launched

data/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright (c) 2017 Andrew Kane
+Copyright (c) 2017-2020 Andrew Kane
 
 MIT License
 

data/README.md

@@ -2,18 +2,18 @@
 
 The automatic indexer for Postgres
 
-[Read about how it works](https://medium.com/@ankane/introducing-dexter-the-automatic-indexer-for-postgres-5f8fa8b28f27)
+[Read about how it works](https://ankane.org/introducing-dexter)
 
 [![Build Status](https://travis-ci.org/ankane/dexter.svg?branch=master)](https://travis-ci.org/ankane/dexter)
 
 ## Installation
 
-First, install [HypoPG](https://github.com/dalibo/hypopg) on your database server. This doesn’t require a restart.
+First, install [HypoPG](https://github.com/HypoPG/hypopg) on your database server. This doesn’t require a restart.
 
 ```sh
 cd /tmp
-curl -L https://github.com/dalibo/hypopg/archive/1.1.0.tar.gz | tar xz
-cd hypopg-1.1.0
+curl -L https://github.com/HypoPG/hypopg/archive/1.1.4.tar.gz | tar xz
+cd hypopg-1.1.4
 make
 make install # may need sudo
 ```
@@ -104,13 +104,19 @@ or pass files:
 dexter <connection-options> <file1> <file2>
 ```
 
+or collect running queries with:
+
+```sh
+dexter <connection-options> --pg-stat-activity
+```
+
 or use the [pg_stat_statements](https://www.postgresql.org/docs/current/static/pgstatstatements.html) extension:
 
 ```sh
 dexter <connection-options> --pg-stat-statements
 ```
 
-> Note: Logs are highly preferred over pg_stat_statements, as pg_stat_statements often doesn’t store enough information to optimize queries.
+> Note: Logs or running queries are highly preferred over pg_stat_statements, as pg_stat_statements often doesn’t store enough information to optimize queries.
 
 ### Collection Options
 
@@ -140,10 +146,10 @@ dexter --interval 60 # seconds
 
 ## Examples
 
-Ubuntu with PostgreSQL 9.6
+Ubuntu with PostgreSQL 12
 
 ```sh
-tail -F -n +1 /var/log/postgresql/postgresql-9.6-main.log | sudo -u postgres dexter dbname
+tail -F -n +1 /var/log/postgresql/postgresql-12-main.log | sudo -u postgres dexter dbname
 ```
 
 Homebrew on Mac
@@ -209,6 +215,10 @@ gem specific_install https://github.com/ankane/dexter.git
 
 This software wouldn’t be possible without [HypoPG](https://github.com/dalibo/hypopg), which allows you to create hypothetical indexes, and [pg_query](https://github.com/lfittl/pg_query), which allows you to parse and fingerprint queries. A big thanks to Dalibo and Lukas Fittl respectively.
 
+## Research
+
+This is known as the Index Selection Problem (ISP).
+
 ## Contributing
 
 Everyone is encouraged to help improve this project. Here are a few ways you can help:
@@ -218,17 +228,18 @@ Everyone is encouraged to help improve this project. Here are a few ways you can
 - Write, clarify, or fix documentation
 - Suggest or add new features
 
-To get started, run:
+To get started with development, run:
 
 ```sh
 git clone https://github.com/ankane/dexter.git
 cd dexter
-bundle
-rake install
+bundle install
+bundle exec rake install
 ```
 
 To run tests, use:
 
 ```sh
-rake test
+createdb dexter_test
+bundle exec rake test
 ```

data/exe/dexter

@@ -1,10 +1,7 @@
 #!/usr/bin/env ruby
 
+# handle interrupts
+trap("SIGINT") { abort }
+
 require "dexter"
-begin
-  Dexter::Client.new(ARGV).perform
-rescue Dexter::Abort => e
-  abort e.message
-rescue Interrupt => e
-  # do nothing
-end
+Dexter::Client.start

data/lib/dexter.rb

@@ -1,16 +1,22 @@
-require "dexter/version"
-require "slop"
+# dependencies
 require "pg"
 require "pg_query"
-require "time"
+require "slop"
+
+# stdlib
 require "set"
-require "thread"
+require "time"
+
+# modules
+require "dexter/version"
 require "dexter/logging"
 require "dexter/client"
 require "dexter/collector"
 require "dexter/indexer"
 require "dexter/log_parser"
 require "dexter/csv_log_parser"
+require "dexter/pg_stat_activity_parser"
+require "dexter/sql_log_parser"
 require "dexter/processor"
 require "dexter/query"
 

data/lib/dexter/client.rb

@@ -1,9 +1,16 @@
 module Dexter
   class Client
+    extend Logging
     include Logging
 
     attr_reader :arguments, :options
 
+    def self.start
+      Dexter::Client.new(ARGV).perform
+    rescue Dexter::Abort, PG::UndefinedFile => e
+      abort colorize(e.message.strip, :red)
+    end
+
     def initialize(args)
       @arguments, @options = parse_args(args)
     end
@@ -18,6 +25,8 @@ module Dexter
       elsif options[:pg_stat_statements]
         # TODO support streaming option
         Indexer.new(options).process_stat_statements
+      elsif options[:pg_stat_activity]
+        Processor.new(:pg_stat_activity, options).perform
       elsif arguments.any?
         ARGV.replace(arguments)
         Processor.new(ARGF, options).perform
@@ -29,9 +38,9 @@ module Dexter
     def parse_args(args)
       opts = Slop.parse(args) do |o|
         o.banner = %(Usage:
-    dexter [options]
-
-Options:)
+    dexter [options])
+        o.separator ""
+        o.separator "Options:"
         o.boolean "--analyze", "analyze tables that haven't been analyzed in the past hour", default: false
         o.boolean "--create", "create indexes", default: false
         o.array "--exclude", "prevent specific tables from being indexed"
@@ -44,11 +53,9 @@ Options:)
         o.float "--min-calls", "only process queries that have been called a certain number of times", default: 0
         o.float "--min-time", "only process queries that have consumed a certain amount of DB time, in minutes", default: 0
         o.integer "--min-cost-savings-pct", default: 50, help: false
+        o.boolean "--pg-stat-activity", "use pg_stat_activity", default: false, help: false
         o.boolean "--pg-stat-statements", "use pg_stat_statements", default: false, help: false
         o.string "-s", "--statement", "process a single statement"
-        # separator must go here to show up correctly - slop bug?
-        o.separator ""
-        o.separator "Connection options:"
         o.on "-v", "--version", "print the version" do
           log Dexter::VERSION
           exit
@@ -57,10 +64,12 @@ Options:)
           log o
           exit
         end
-        o.string "-U", "--username"
-        o.string "-d", "--dbname"
-        o.string "-h", "--host"
-        o.integer "-p", "--port"
+        o.separator ""
+        o.separator "Connection options:"
+        o.string "-d", "--dbname", "database name"
+        o.string "-h", "--host", "database host"
+        o.integer "-p", "--port", "database port"
+        o.string "-U", "--username", "database user"
       end
 
       arguments = opts.arguments
@@ -70,11 +79,11 @@ Options:)
 
       # TODO don't use global var
       $log_level = options[:log_level].to_s.downcase
-      abort "Unknown log level" unless ["error", "info", "debug", "debug2", "debug3"].include?($log_level)
+      raise Dexter::Abort, "Unknown log level" unless ["error", "info", "debug", "debug2", "debug3"].include?($log_level)
 
       [arguments, options]
     rescue Slop::Error => e
-      abort e.message
+      raise Dexter::Abort, e.message
     end
   end
 end

data/lib/dexter/csv_log_parser.rb

@@ -16,7 +16,7 @@ module Dexter
         end
       end
     rescue CSV::MalformedCSVError => e
-      abort "ERROR: #{e.message}"
+      raise Dexter::Abort, "ERROR: #{e.message}"
     end
   end
 end

data/lib/dexter/indexer.rb

@@ -14,6 +14,7 @@ module Dexter
       @analyze = options[:analyze]
       @min_cost_savings_pct = options[:min_cost_savings_pct].to_i
       @options = options
+      @mutex = Mutex.new
 
       create_extension unless extension_exists?
       execute("SET lock_timeout = '5s'")
@@ -25,6 +26,23 @@ module Dexter
       process_queries(queries)
     end
 
+    def stat_activity
+      execute <<-SQL
+        SELECT
+          pid || ':' || COALESCE(query_start, xact_start) AS id,
+          query,
+          EXTRACT(EPOCH FROM NOW() - COALESCE(query_start, xact_start)) * 1000.0 AS duration_ms
+        FROM
+          pg_stat_activity
+        WHERE
+          datname = current_database()
+          AND state = 'active'
+          AND pid != pg_backend_pid()
+        ORDER BY
+          1
+      SQL
+    end
+
     def process_queries(queries)
       # reset hypothetical indexes
       reset_hypothetical_indexes
@@ -89,13 +107,13 @@ module Dexter
       analyze_tables(tables) if tables.any? && (@analyze || @log_level == "debug2")
 
       # create hypothetical indexes and explain queries
-      candidates = tables.any? ? create_hypothetical_indexes(queries.select(&:candidate_tables), tables) : {}
+      candidates = tables.any? ? create_hypothetical_indexes(queries.select(&:candidate_tables)) : {}
 
       # see if new indexes were used and meet bar
       new_indexes = determine_indexes(queries, candidates, tables)
 
       # display and create new indexes
-      show_and_create_indexes(new_indexes, queries, tables)
+      show_and_create_indexes(new_indexes, queries)
     end
 
     private
@@ -105,9 +123,9 @@ module Dexter
       begin
         execute("CREATE EXTENSION IF NOT EXISTS hypopg")
       rescue PG::UndefinedFile
-        abort "Install HypoPG first: https://github.com/ankane/dexter#installation"
+        raise Dexter::Abort, "Install HypoPG first: https://github.com/ankane/dexter#installation"
       rescue PG::InsufficientPrivilege
-        abort "Use a superuser to run: CREATE EXTENSION hypopg"
+        raise Dexter::Abort, "Use a superuser to run: CREATE EXTENSION hypopg"
       end
     end
 
@@ -164,9 +182,9 @@ module Dexter
           query.plans << plan(query.statement)
           if @log_explain
             # Pass format to prevent ANALYZE
-            puts execute("EXPLAIN (FORMAT TEXT) #{safe_statement(query.statement)}").map { |r| r["QUERY PLAN"] }.join("\n")
+            puts execute("EXPLAIN (FORMAT TEXT) #{safe_statement(query.statement)}", pretty: false).map { |r| r["QUERY PLAN"] }.join("\n")
           end
-        rescue PG::Error => e
+        rescue PG::Error, JSON::NestingError => e
           if @log_explain
             log e.message
           end
@@ -175,7 +193,7 @@ module Dexter
       end
     end
 
-    def create_hypothetical_indexes(queries, tables)
+    def create_hypothetical_indexes(queries)
       candidates = {}
 
       # get initial costs for queries
@@ -345,11 +363,12 @@ module Dexter
                 winning_cost < query.initial_cost * savings_ratio
               end
 
+            query_indexes = [winning_index]
+            new_cost3 = winning_cost
+            query.pass3_indexes = query_indexes
+
             if use_winning
-              query_indexes = [winning_index]
               cost_savings3 = true
-              new_cost3 = winning_cost
-              query.pass3_indexes = query_indexes
             else
               suggest_index = false
             end
@@ -398,11 +417,11 @@ module Dexter
       end
     end
 
-    def show_and_create_indexes(new_indexes, queries, tables)
+    def show_and_create_indexes(new_indexes, queries)
       # print summary
       if new_indexes.any?
         new_indexes.each do |index|
-          log "Index found: #{index[:table]} (#{index[:columns].join(", ")})"
+          log colorize("Index found: #{index[:table]} (#{index[:columns].join(", ")})", :green)
         end
       else
         log "No new indexes found"
@@ -439,7 +458,7 @@ module Dexter
               log "Pass3: #{query.costs[3]} : #{log_indexes(query.pass3_indexes || [])}"
             end
             log "Final: #{query.new_cost} : #{log_indexes(query.suggest_index ? query_indexes : [])}"
-            if query_indexes.size == 1 && !query.suggest_index
+            if (query.pass1_indexes.any? || query.pass2_indexes.any?) && !query.suggest_index
               log "Need #{@min_cost_savings_pct}% cost savings to suggest index"
             end
           else
@@ -479,6 +498,9 @@ module Dexter
 
     def conn
       @conn ||= begin
+        # set connect timeout if none set
+        ENV["PGCONNECT_TIMEOUT"] ||= "2"
+
         if @options[:dbname] =~ /\Apostgres(ql)?:\/\//
           config = @options[:dbname]
         else
@@ -486,17 +508,17 @@ module Dexter
             host: @options[:host],
             port: @options[:port],
             dbname: @options[:dbname],
-            user: @options[:user]
+            user: @options[:username]
           }.reject { |_, value| value.to_s.empty? }
           config = config[:dbname] if config.keys == [:dbname] && config[:dbname].include?("=")
         end
         PG::Connection.new(config)
       end
     rescue PG::ConnectionBad => e
-      abort e.message
+      raise Dexter::Abort, e.message
     end
 
-    def execute(query)
+    def execute(query, pretty: true)
       # use exec_params instead of exec for security
       #
       # Unlike PQexec, PQexecParams allows at most one SQL command in the given string.
@@ -504,14 +526,17 @@ module Dexter
       # This is a limitation of the underlying protocol, but has some usefulness
       # as an extra defense against SQL-injection attacks.
       # https://www.postgresql.org/docs/current/static/libpq-exec.html
-      query = squish(query)
-      log "SQL: #{query}" if @log_sql
-      conn.exec_params(query, []).to_a
+      query = squish(query) if pretty
+      log colorize("[sql] #{query}", :cyan) if @log_sql
+
+      @mutex.synchronize do
+        conn.exec_params(query, []).to_a
+      end
     end
 
     def plan(query)
       # strip semi-colons as another measure of defense
-      JSON.parse(execute("EXPLAIN (FORMAT JSON) #{safe_statement(query)}").first["QUERY PLAN"]).first["Plan"]
+      JSON.parse(execute("EXPLAIN (FORMAT JSON) #{safe_statement(query)}", pretty: false).first["QUERY PLAN"], max_nesting: 1000).first["Plan"]
     end
 
     # TODO for multicolumn indexes, use ordering
@@ -571,7 +596,13 @@ module Dexter
 
       view_tables = {}
       result.each do |row|
-        view_tables[row["table_name"]] = PgQuery.parse(row["definition"]).tables
+        begin
+          view_tables[row["table_name"]] = PgQuery.parse(row["definition"]).tables
+        rescue PgQuery::ParseError
+          if @log_level.start_with?("debug")
+            log colorize("ERROR: Cannot parse view definition: #{row["table_name"]}", :red)
+          end
+        end
       end
 
       view_tables

data/lib/dexter/logging.rb

@@ -1,11 +1,26 @@
 module Dexter
   module Logging
+    COLOR_CODES = {
+      red: 31,
+      green: 32,
+      yellow: 33,
+      cyan: 36
+    }
+
+    def output
+      $stdout
+    end
+
     def log(message = "")
-      puts message unless $log_level == "error"
+      output.puts(message) unless $log_level == "error"
     end
 
-    def abort(message)
-      raise Dexter::Abort, message
+    def colorize(message, color)
+      if output.tty?
+        "\e[#{COLOR_CODES[color]}m#{message}\e[0m"
+      else
+        message
+      end
     end
   end
 end

data/lib/dexter/pg_stat_activity_parser.rb

@@ -0,0 +1,25 @@
+module Dexter
+  class PgStatActivityParser < LogParser
+    def perform
+      queries = {}
+
+      loop do
+        new_queries = {}
+        @logfile.stat_activity.each do |row|
+          new_queries[row["id"]] = row
+        end
+
+        # store queries after they complete
+        queries.each do |id, row|
+          unless new_queries[id]
+            process_entry(row["query"], row["duration_ms"].to_f)
+          end
+        end
+
+        queries = new_queries
+
+        sleep(1)
+      end
+    end
+  end
+end

data/lib/dexter/processor.rb

@@ -6,15 +6,19 @@ module Dexter
       @logfile = logfile
 
       @collector = Collector.new(min_time: options[:min_time], min_calls: options[:min_calls])
+      @indexer = Indexer.new(options)
+
       @log_parser =
-        if options[:input_format] == "csv"
+        if @logfile == :pg_stat_activity
+          PgStatActivityParser.new(@indexer, @collector)
+        elsif options[:input_format] == "csv"
           CsvLogParser.new(logfile, @collector)
+        elsif options[:input_format] == "sql"
+          SqlLogParser.new(logfile, @collector)
         else
           LogParser.new(logfile, @collector)
         end
 
-      @indexer = Indexer.new(options)
-
       @starting_interval = 3
       @interval = options[:interval]
 
@@ -25,7 +29,7 @@ module Dexter
     end
 
     def perform
-      if @logfile == STDIN
+      if [STDIN, :pg_stat_activity].include?(@logfile)
         Thread.abort_on_exception = true
         Thread.new do
           sleep(@starting_interval)
@@ -33,7 +37,7 @@ module Dexter
             begin
               process_queries
             rescue PG::ServerError => e
-              log "ERROR: #{e.class.name}: #{e.message}"
+              log colorize("ERROR: #{e.class.name}: #{e.message}", :red)
             end
             sleep(@interval)
           end
@@ -43,7 +47,7 @@ module Dexter
       begin
         @log_parser.perform
       rescue Errno::ENOENT => e
-        abort "ERROR: #{e.message}"
+        raise Dexter::Abort, "ERROR: #{e.message}"
       end
 
       process_queries

data/lib/dexter/sql_log_parser.rb

@@ -0,0 +1,10 @@
+module Dexter
+  class SqlLogParser < LogParser
+    def perform
+      # TODO support streaming
+      @logfile.read.split(";").each do |statement|
+        process_entry(statement, 0)
+      end
+    end
+  end
+end

data/lib/dexter/version.rb

@@ -1,3 +1,3 @@
 module Dexter
-  VERSION = "0.3.3"
+  VERSION = "0.3.8"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: pgdexter
 version: !ruby/object:Gem::Version
-  version: 0.3.3
+  version: 0.3.8
 platform: ruby
 authors:
 - Andrew Kane
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2018-02-23 00:00:00.000000000 Z
+date: 2020-08-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: slop
@@ -16,28 +16,28 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 4.2.0
+        version: 4.8.2
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 4.2.0
+        version: 4.8.2
 - !ruby/object:Gem::Dependency
   name: pg
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 0.18.2
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 0.18.2
 - !ruby/object:Gem::Dependency
   name: pg_query
   requirement: !ruby/object:Gem::Requirement
@@ -95,23 +95,16 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
 description: 
-email:
-- andrew@chartkick.com
+email: andrew@chartkick.com
 executables:
 - dexter
 extensions: []
 extra_rdoc_files: []
 files:
-- ".gitignore"
-- ".travis.yml"
 - CHANGELOG.md
-- Gemfile
 - LICENSE.txt
 - README.md
-- Rakefile
 - exe/dexter
-- guides/Hosted-Postgres.md
-- guides/Linux.md
 - lib/dexter.rb
 - lib/dexter/client.rb
 - lib/dexter/collector.rb
@@ -119,12 +112,14 @@ files:
 - lib/dexter/indexer.rb
 - lib/dexter/log_parser.rb
 - lib/dexter/logging.rb
+- lib/dexter/pg_stat_activity_parser.rb
 - lib/dexter/processor.rb
 - lib/dexter/query.rb
+- lib/dexter/sql_log_parser.rb
 - lib/dexter/version.rb
-- pgdexter.gemspec
 homepage: https://github.com/ankane/dexter
-licenses: []
+licenses:
+- MIT
 metadata: {}
 post_install_message: 
 rdoc_options: []
@@ -134,15 +129,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '0'
+      version: '2.2'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.6.13
+rubygems_version: 3.1.2
 signing_key: 
 specification_version: 4
 summary: The automatic indexer for Postgres

data/.gitignore

@@ -1,9 +0,0 @@
-/.bundle/
-/.yardoc
-/Gemfile.lock
-/_yardoc/
-/coverage/
-/doc/
-/pkg/
-/spec/reports/
-/tmp/

data/.travis.yml

@@ -1,17 +0,0 @@
-language: ruby
-rvm: 2.4.1
-cache: bundler
-script: bundle exec rake test
-addons:
-  postgresql: "9.6"
-before_script:
-  - sudo apt-get install postgresql-server-dev-9.6
-  - git clone https://github.com/dalibo/hypopg.git
-  - cd hypopg
-  - make
-  - sudo make install
-  - psql -c 'create database dexter_test;' -U postgres
-notifications:
-  email:
-    on_success: never
-    on_failure: change

data/Gemfile

@@ -1,4 +0,0 @@
-source "https://rubygems.org"
-
-# Specify your gem's dependencies in dexter.gemspec
-gemspec

data/Rakefile

@@ -1,11 +0,0 @@
-require "bundler/gem_tasks"
-require "rake/testtask"
-
-Rake::TestTask.new(:test) do |t|
-  t.libs << "test"
-  t.libs << "lib"
-  t.test_files = FileList["test/**/*_test.rb"]
-  t.warning = false
-end
-
-task default: :test

data/guides/Hosted-Postgres.md

@@ -1,102 +0,0 @@
-# Hosted Postgres
-
-Some hosted providers like Amazon RDS and Heroku do not support the HypoPG extension, which Dexter needs to run. Hopefully this will change with time. For now, we can spin up a separate database instance to run Dexter. It’s not super convenient, but can be useful to do from time to time.
-
-### Install Postgres and Ruby
-
-Linux
-
-```sh
-sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
-sudo apt-get install -y wget ca-certificates
-wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
-sudo apt-get update
-sudo apt-get install -y postgresql-9.6 postgresql-server-dev-9.6
-sudo -u postgres createuser $(whoami) -s
-sudo apt-get install -y ruby2.2 ruby2.2-dev
-```
-
-Mac
-
-```sh
-brew install postgresql
-brew install ruby
-```
-
-### Install HypoPG and Dexter
-
-HypoPG
-
-```sh
-cd /tmp
-curl -L https://github.com/dalibo/hypopg/archive/1.0.0.tar.gz | tar xz
-cd hypopg-1.0.0
-make
-make install # may need sudo
-```
-
-Dexter
-
-```sh
-gem install pgdexter # may need sudo
-```
-
-### Download logs
-
-#### Amazon RDS
-
-Create an IAM user with the policy below:
-
-```
-{
-  "Statement": [
-    {
-      "Action": [
-        "rds:DescribeDBLogFiles",
-        "rds:DownloadDBLogFilePortion"
-      ],
-      "Effect": "Allow",
-      "Resource": "*"
-    }
-  ]
-}
-```
-
-And run:
-
-```sh
-aws configure
-gem install pghero_logs # may need sudo
-pghero_logs download <instance-id>
-```
-
-#### Heroku
-
-Production-tier databases only
-
-```sh
-heroku logs -p postgres > postgresql.log
-```
-
-### Dump and restore
-
-We recommend creating a new instance from a snapshot for the dump to avoid affecting customers.
-
-```sh
-pg_dump -v -j 8 -Fd -f /tmp/newout.dir <connection-options>
-```
-
-Then shutdown the dump instance. Restore with:
-
-```sh
-createdb dexter_restore
-pg_restore -v -j 8 -x -O --format=d -d dexter_restore /tmp/newout.dir/
-```
-
-### Run Dexter
-
-```sh
-dexter dexter_restore postgresql.log* --analyze
-```
-
-:tada:

data/guides/Linux.md

@@ -1,70 +0,0 @@
-# Linux Packages
-
-Distributions
-
-- [Ubuntu 16.04 (Xenial)](#ubuntu-1604-xenial)
-- [Ubuntu 14.04 (Trusty)](#ubuntu-1404-trusty)
-- [Debian 9 (Stretch)](#debian-9-stretch)
-- [Debian 8 (Jesse)](#debian-8-jesse)
-- [CentOS / RHEL 7](#centos--rhel-7)
-- [SUSE Linux Enterprise Server 12](#suse-linux-enterprise-server-12)
-
-### Ubuntu 16.04 (Xenial)
-
-```sh
-wget -qO- https://dl.packager.io/srv/pghero/dexter/key | sudo apt-key add -
-sudo wget -O /etc/apt/sources.list.d/dexter.list \
-  https://dl.packager.io/srv/pghero/dexter/master/installer/ubuntu/16.04.repo
-sudo apt-get update
-sudo apt-get -y install dexter
-```
-
-### Ubuntu 14.04 (Trusty)
-
-```sh
-wget -qO- https://dl.packager.io/srv/pghero/dexter/key | sudo apt-key add -
-sudo wget -O /etc/apt/sources.list.d/dexter.list \
-  https://dl.packager.io/srv/pghero/dexter/master/installer/ubuntu/14.04.repo
-sudo apt-get update
-sudo apt-get install dexter
-```
-
-### Debian 9 (Stretch)
-
-```sh
-wget -qO- https://dl.packager.io/srv/pghero/dexter/key | sudo apt-key add -
-sudo wget -O /etc/apt/sources.list.d/dexter.list \
-  https://dl.packager.io/srv/pghero/dexter/master/installer/debian/9.repo
-sudo apt-get update
-sudo apt-get install dexter
-```
-
-### Debian 8 (Jesse)
-
-```sh
-wget -qO- https://dl.packager.io/srv/pghero/dexter/key | sudo apt-key add -
-sudo wget -O /etc/apt/sources.list.d/dexter.list \
-  https://dl.packager.io/srv/pghero/dexter/master/installer/debian/8.repo
-sudo apt-get update
-sudo apt-get install dexter
-```
-
-### CentOS / RHEL 7
-
-```sh
-sudo wget -O /etc/yum.repos.d/dexter.repo \
-  https://dl.packager.io/srv/pghero/dexter/master/installer/el/7.repo
-sudo yum install dexter
-```
-
-### SUSE Linux Enterprise Server 12
-
-```sh
-sudo wget -O /etc/zypp/repos.d/dexter.repo \
-  https://dl.packager.io/srv/pghero/dexter/master/installer/sles/12.repo
-sudo zypper install dexter
-```
-
-## Credits
-
-:heart: Made possible by [Packager](https://packager.io/)

data/pgdexter.gemspec

@@ -1,30 +0,0 @@
-# coding: utf-8
-
-lib = File.expand_path("../lib", __FILE__)
-$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
-require "dexter/version"
-
-Gem::Specification.new do |spec|
-  spec.name          = "pgdexter"
-  spec.version       = Dexter::VERSION
-  spec.authors       = ["Andrew Kane"]
-  spec.email         = ["andrew@chartkick.com"]
-
-  spec.summary       = "The automatic indexer for Postgres"
-  spec.homepage      = "https://github.com/ankane/dexter"
-
-  spec.files         = `git ls-files -z`.split("\x0").reject do |f|
-    f.match(%r{^(test|spec|features)/})
-  end
-  spec.bindir        = "exe"
-  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
-  spec.require_paths = ["lib"]
-
-  spec.add_dependency "slop", ">= 4.2.0"
-  spec.add_dependency "pg"
-  spec.add_dependency "pg_query"
-
-  spec.add_development_dependency "bundler"
-  spec.add_development_dependency "rake"
-  spec.add_development_dependency "minitest"
-end