pgdexter 0.1.6 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 496685bbacfac7387182be3ebda3515a492723cc
-  data.tar.gz: b99548fccab8337b9ff5765a6182472abba9bb06
+  metadata.gz: c513808e2cdd9690c477c30548979fad122575a4
+  data.tar.gz: 6f4c605a0b68baa0275a523155b7be4ea9bc4d56
 SHA512:
-  metadata.gz: 3f9b30e7355c26cf7084b8deee9735206da484cd9f4725a5c1f751db5808dc32d66954341c5be3ba46cd783fa2e360e2b9c32ab0667aa2372213ebca695b29ff
-  data.tar.gz: 41346b8ed9906e24c68847c26bc9ffe54176eb0e74dc771138ec266ebccfca3e0d18ebd37223b73f1f2b8a9f512320502e4ad9597982bc253846881ea7e93e8d
+  metadata.gz: 3b38a53e96516a485394f3ec594ee0418c1a7ab94fe8726b3228cc5443843dd7f7ba2bd1fe4bb0cc5c4dbd877bd9b9b036d5bfd5b4cda275dc7176dbdf9da82c
+  data.tar.gz: cbe64d0cf9a40b96bf2644a80519bb837d235d6956ca9b61b24533867130958c95932ffbf4d69672218a10628c32b87a1f1e87f86a4968577f1629fc871ef440

data/CHANGELOG.md

@@ -1,3 +1,14 @@
+## 0.2.0
+
+- Added same connection options as `psql`
+- Added support for multiple files
+- Added `error` log level
+- Better error messages when cannot connect
+
+Breaking
+
+- `-h` option changed to `--host` instead of `--help` for consistency with `psql`
+
 ## 0.1.6
 
 - Significant performance improvements

data/README.md

@@ -11,7 +11,8 @@ The automatic indexer for Postgres
 First, install [HypoPG](https://github.com/dalibo/hypopg) on your database server. This doesn’t require a restart.
 
 ```sh
-curl -L https://github.com/dalibo/hypopg/archive/1.0.0.tar.gz | tar -x
+cd /tmp
+curl -L https://github.com/dalibo/hypopg/archive/1.0.0.tar.gz | tar xz
 cd hypopg-1.0.0
 make
 make install # may need sudo
@@ -38,7 +39,7 @@ The command line tool is also available as a [Linux package](guides/Linux.md).
 Dexter needs a connection to your database and a log file to process.
 
 ```sh
-tail -F -n +1 <log-file> | dexter <connection-string>
+tail -F -n +1 <log-file> | dexter <connection-options>
 ```
 
 This finds slow queries and generates output like:
@@ -63,18 +64,24 @@ To be safe, Dexter will not create indexes unless you pass the `--create` flag.
 2017-06-25T17:52:37+00:00 Index created: 15243 ms
 ```
 
-## Connection String
+## Connection Options
 
-The connection string is a URI with the format:
+Dexter supports the same connection options as psql.
+
+```
+-h host -U user -p 5432 -d dbname
+```
+
+This includes URIs:
 
 ```
 postgresql://user:pass@host:5432/dbname
 ```
 
-To connect through a socket, just pass the database name.
+and connection strings:
 
 ```
-dbname
+host=localhost port=5432 dbname=mydb
 ```
 
 ## Options
@@ -83,16 +90,22 @@ Name | Description | Default
 --- | --- | ---
 exclude | prevent specific tables from being indexed | None
 interval | time to wait between processing queries, in seconds | 60
-log-level | `debug` gives additional info for suggested indexes<br />`debug2` gives additional info for processed queries | info
+log-level | `debug` gives additional info for suggested indexes<br />`debug2` gives additional info for processed queries<br />`error` suppresses logging | info
 log-sql | log SQL statements executed | false
 min-time | only process queries consuming a min amount of DB time, in minutes | 0
 
-## Single Statement Mode
+## Non-Streaming Modes
 
 You can pass a single statement with:
 
 ```sh
-dexter <connection-string> -s "SELECT * FROM ..."
+dexter <connection-options> -s "SELECT * FROM ..."
+```
+
+or files with:
+
+```sh
+dexter <connection-options> <file1> <file2>
 ```
 
 ## Examples
@@ -109,6 +122,10 @@ Homebrew on Mac
 tail -F -n +1 /usr/local/var/postgres/server.log | dexter dbname
 ```
 
+## Hosted Postgres
+
+Some hosted providers like Amazon RDS and Heroku do not support the HypoPG extension, which Dexter needs to run. See [how to use Dexter](guides/Hosted-Postgres.md) in these cases.
+
 ## Future Work
 
 [Here are some ideas](https://github.com/ankane/dexter/issues/1)

data/guides/Hosted-Postgres.md

@@ -0,0 +1,102 @@
+# Hosted Postgres
+
+Some hosted providers like Amazon RDS and Heroku do not support the HypoPG extension, which Dexter needs to run. Hopefully this will change with time. For now, we can spin up a separate database instance to run Dexter. It’s not super convenient, but can be useful to do from time to time.
+
+### Install Postgres and Ruby
+
+Linux
+
+```sh
+sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
+sudo apt-get install wget ca-certificates
+wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
+sudo apt-get update
+sudo apt-get install postgresql-9.6 postgresql-server-dev-9.6
+sudo -u postgres createuser $(whoami) -s
+sudo apt-get install ruby2.2 ruby2.2-dev
+```
+
+Mac
+
+```sh
+brew install postgresql
+brew install ruby
+```
+
+### Install HypoPG and Dexter
+
+HypoPG
+
+```sh
+cd /tmp
+curl -L https://github.com/dalibo/hypopg/archive/1.0.0.tar.gz | tar xz
+cd hypopg-1.0.0
+make
+make install # may need sudo
+```
+
+Dexter
+
+```sh
+gem install pgdexter # may need sudo
+```
+
+### Download logs
+
+#### Amazon RDS
+
+Create an IAM user with the policy below:
+
+```
+{
+  "Statement": [
+    {
+      "Action": [
+        "rds:DescribeDBLogFiles",
+        "rds:DownloadDBLogFilePortion"
+      ],
+      "Effect": "Allow",
+      "Resource": "*"
+    }
+  ]
+}
+```
+
+And run:
+
+```sh
+aws configure
+gem install pghero_logs # may need sudo
+pghero_logs download <instance-id>
+```
+
+#### Heroku
+
+Production-tier databases only
+
+```sh
+heroku logs -p postgres > postgresql.log
+```
+
+### Dump and restore
+
+We recommend creating a new instance from a snapshot for the dump to avoid affecting customers.
+
+```sh
+pg_dump -v -j 8 -Fd -f /tmp/newout.dir <connection-string>
+```
+
+Then shutdown the dump instance. Restore with:
+
+```sh
+createdb dexter_restore
+pg_restore -v -j 8 -x -O --format=d -d dexter_restore /tmp/newout.dir/
+```
+
+### Run Dexter
+
+```sh
+dexter dexter_restore postgresql.log*
+```
+
+:tada:

data/lib/dexter/client.rb

@@ -12,20 +12,21 @@ module Dexter
 
       if options[:statement]
         query = Query.new(options[:statement])
-        Indexer.new(arguments[0], options).process_queries([query])
+        Indexer.new(options).process_queries([query])
       elsif options[:pg_stat_statements]
-        Indexer.new(arguments[0], options).process_stat_statements
-      elsif arguments[1]
-        Processor.new(arguments[0], arguments[1], options).perform
+        Indexer.new(options).process_stat_statements
+      elsif arguments.any?
+        ARGV.replace(arguments)
+        Processor.new(ARGF, options).perform
       else
-        Processor.new(arguments[0], STDIN, options).perform
+        Processor.new(STDIN, options).perform
       end
     end
 
     def parse_args(args)
       opts = Slop.parse(args) do |o|
         o.banner = %(Usage:
-    dexter <database-url> [options]
+    dexter [options]
 
 Options:)
         o.boolean "--create", "create indexes", default: false
@@ -38,28 +39,32 @@ Options:)
         o.string "--log-level", "log level", default: "info"
         o.boolean "--log-sql", "log sql", default: false
         o.string "-s", "--statement", "process a single statement"
+        o.separator ""
+        o.separator "Connection options:"
         o.on "-v", "--version", "print the version" do
           log Dexter::VERSION
           exit
         end
-        o.on "-h", "--help", "prints help" do
+        o.on "--help", "prints help" do
           log o
           exit
         end
+        o.string "-U", "--username"
+        o.string "-d", "--dbname"
+        o.string "-h", "--host"
+        o.integer "-p", "--port"
       end
 
       arguments = opts.arguments
+      options = opts.to_hash
 
-      if arguments.empty?
-        log opts
-        exit
-      end
-
-      abort "Too many arguments" if arguments.size > 2
+      options[:dbname] = arguments.shift unless options[:dbname]
 
-      abort "Unknown log level" unless ["info", "debug", "debug2"].include?(opts.to_hash[:log_level].to_s.downcase)
+      # TODO don't use global var
+      $log_level = options[:log_level].to_s.downcase
+      abort "Unknown log level" unless ["error", "info", "debug", "debug2"].include?($log_level)
 
-      [arguments, opts.to_hash]
+      [arguments, options]
     rescue Slop::Error => e
       abort e.message
     end

data/lib/dexter/indexer.rb

@@ -2,8 +2,7 @@ module Dexter
   class Indexer
     include Logging
 
-    def initialize(database_url, options)
-      @database_url = database_url
+    def initialize(options)
       @create = options[:create]
       @log_level = options[:log_level]
       @exclude_tables = options[:exclude]
@@ -11,6 +10,7 @@ module Dexter
       @log_sql = options[:log_sql]
       @log_explain = options[:log_explain]
       @min_time = options[:min_time] || 0
+      @options = options
 
       create_extension unless extension_exists?
       execute("SET lock_timeout = '5s'")
@@ -311,19 +311,21 @@ module Dexter
 
     def conn
       @conn ||= begin
-        uri = URI.parse(@database_url)
-        config = {
-          host: uri.host,
-          port: uri.port,
-          dbname: uri.path.sub(/\A\//, ""),
-          user: uri.user,
-          password: uri.password,
-          connect_timeout: 3
-        }.reject { |_, value| value.to_s.empty? }
+        if @options[:dbname] =~ /\Apostgres(ql)?:\/\//
+          config = @options[:dbname]
+        else
+          config = {
+            host: @options[:host],
+            port: @options[:port],
+            dbname: @options[:dbname],
+            user: @options[:user]
+          }.reject { |_, value| value.to_s.empty? }
+          config = config[:dbname] if config.keys == [:dbname] && config[:dbname].include?("=")
+        end
         PG::Connection.new(config)
       end
-    rescue PG::ConnectionBad
-      abort "Can't connect to database"
+    rescue PG::ConnectionBad => e
+      abort e.message
     end
 
     def execute(query)

data/lib/dexter/log_parser.rb

@@ -6,8 +6,6 @@ module Dexter
     def initialize(logfile, collector)
       @logfile = logfile
       @collector = collector
-
-      abort "Log file not found" unless File.exist?(logfile)
     end
 
     def perform
@@ -35,14 +33,8 @@ module Dexter
     private
 
     def each_line
-      if @logfile == STDIN
-        STDIN.each_line do |line|
-          yield line
-        end
-      else
-        File.foreach(@logfile) do |line|
-          yield line
-        end
+      @logfile.each_line do |line|
+        yield line
       end
     end
 

data/lib/dexter/logging.rb

@@ -1,7 +1,7 @@
 module Dexter
   module Logging
     def log(message)
-      puts "#{Time.now.iso8601} #{message}"
+      puts "#{Time.now.iso8601} #{message}" unless $log_level == "error"
     end
   end
 end

data/lib/dexter/processor.rb

@@ -2,12 +2,12 @@ module Dexter
   class Processor
     include Logging
 
-    def initialize(database_url, logfile, options)
+    def initialize(logfile, options)
       @logfile = logfile
 
       @collector = Collector.new(min_time: options[:min_time])
       @log_parser = LogParser.new(logfile, @collector)
-      @indexer = Indexer.new(database_url, options)
+      @indexer = Indexer.new(options)
 
       @starting_interval = 3
       @interval = options[:interval]
@@ -34,7 +34,11 @@ module Dexter
         end
       end
 
-      @log_parser.perform
+      begin
+        @log_parser.perform
+      rescue Errno::ENOENT => e
+        abort e.message
+      end
 
       process_queries
     end

data/lib/dexter/version.rb

@@ -1,3 +1,3 @@
 module Dexter
-  VERSION = "0.1.6"
+  VERSION = "0.2.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: pgdexter
 version: !ruby/object:Gem::Version
-  version: 0.1.6
+  version: 0.2.0
 platform: ruby
 authors:
 - Andrew Kane
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2017-08-26 00:00:00.000000000 Z
+date: 2017-08-28 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: slop
@@ -110,6 +110,7 @@ files:
 - README.md
 - Rakefile
 - exe/dexter
+- guides/Hosted-Postgres.md
 - guides/Linux.md
 - lib/dexter.rb
 - lib/dexter/client.rb