ruby-pg-extras 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 771a487899452aae45b3491ef7b5c4704ff627abe1768c9342d64a6ae5c2e050
+  data.tar.gz: 98b7ccbc8b3c74e4b623be5cc932d73110676cf8de6610ceeffa0127c25c750b
+SHA512:
+  metadata.gz: 4b7f48a2907f36db39033aa81801eef6cc397c8e2670024ad2b0f5d06984792bc7cfc664e3aab34b5f6d84268dc6bb312c345a078f7914fa858eb7dfa0aa1fe2
+  data.tar.gz: cf675a7c8364d01468602aee27961f008a33a6168ab11afd406cfaa7eb9b97c1197f6e0e455ba1801d9c4282d95502cb8041644b992e4455431e5eed6c5c4f1a

data/.gitignore

@@ -0,0 +1,6 @@
+Gemfile.lock
+.ruby-version
+pkg/
+*.gem
+docker-compose.yml
+

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+The MIT License (MIT)
+
+Copyright © Paweł Urbanek 2019
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,373 @@
+# Rails PG Extras [![Gem Version](https://badge.fury.io/rb/ruby-pg-extras.svg)](https://badge.fury.io/rb/ruby-pg-extras) [![CircleCI](https://circleci.com/gh/pawurb/ruby-pg-extras.svg?style=svg)](https://circleci.com/gh/pawurb/ruby-pg-extras)
+
+Rails port of [Heroku PG Extras](https://github.com/heroku/heroku-pg-extras). The goal of this project is to provide a powerful insights into PostgreSQL database for Ruby on Rails apps that are not using the default Heroku PostgreSQL plugin.
+
+Included rake tasks and Ruby methods can be used to obtain information about a Postgres instance, that may be useful when analyzing performance issues. This includes information about locks, index usage, buffer cache hit ratios and vacuum statistics. Ruby API enables developers to easily integrate the tool into e.g. automatic monitoring tasks.
+
+### Installation
+
+In your Gemfile
+
+```ruby
+gem 'ruby-pg-extras'
+```
+
+### Usage
+
+Each command can be used as a rake task, or a directly from the Ruby code.
+
+```bash
+rake pg_extras:cache_hit
+```
+
+```ruby
+RubyPGExtras.cache_hit
+```
+```bash
++----------------+------------------------+
+|        Index and table hit rate         |
++----------------+------------------------+
+| name           | ratio                  |
++----------------+------------------------+
+| index hit rate | 0.97796610169491525424 |
+| table hit rate | 0.96724294813466787989 |
++----------------+------------------------+
+```
+
+
+By default the ASCII table is displayed, to change to format you need to specify the `in_format` parameter (`[:display_table, :hash, :array, :raw]` options are available):
+
+```ruby
+RubyPGExtras.cache_hit(in_format: :hash) =>
+
+ [{"name"=>"index hit rate", "ratio"=>"0.97796610169491525424"}, {"name"=>"table hit rate", "ratio"=>"0.96724294813466787989"}]
+
+RubyPGExtras.cache_hit(in_format: :array) =>
+
+ [["index hit rate", "0.97796610169491525424"], ["table hit rate", "0.96724294813466787989"]]
+
+RubyPGExtras.cache_hit(in_format: :raw) =>
+
+ #<PG::Result:0x00007f75777f7328 status=PGRES_TUPLES_OK ntuples=2 nfields=2 cmd_tuples=2>
+```
+
+### Available methods
+
+#### `cache_hit`
+
+```bash
+$ rake pg_extras:cache_hit
+      name      |         ratio
+----------------+------------------------
+ index hit rate | 0.99957765013541945832
+ table hit rate |                   1.00
+(2 rows)
+```
+
+This command provides information on the efficiency of the buffer cache, for both index reads (`index hit rate`) as well as table reads (`table hit rate`). A low buffer cache hit ratio can be a sign that the Postgres instance is too small for the workload.
+
+#### `index_usage`
+
+```
+$ rake pg_extras:index_usage
+       relname       | percent_of_times_index_used | rows_in_table
+---------------------+-----------------------------+---------------
+ events              |                          65 |       1217347
+ app_infos           |                          74 |        314057
+ app_infos_user_info |                           0 |        198848
+ user_info           |                           5 |         94545
+ delayed_jobs        |                          27 |             0
+(5 rows)
+```
+
+This command provides information on the efficiency of indexes, represented as what percentage of total scans were index scans. A low percentage can indicate under indexing, or wrong data being indexed.
+
+### `locks`
+
+```
+$ rake pg_extras:locks
+ procpid | relname | transactionid | granted |     query_snippet     |       age
+---------+---------+---------------+---------+-----------------------+-----------------
+   31776 |         |               | t       | <IDLE> in transaction | 00:19:29.837898
+   31776 |         |          1294 | t       | <IDLE> in transaction | 00:19:29.837898
+   31912 |         |               | t       | select * from hello;  | 00:19:17.94259
+    3443 |         |               | t       |                      +| 00:00:00
+         |         |               |         |    select            +|
+         |         |               |         |      pg_stat_activi   |
+(4 rows)
+```
+
+This command displays queries that have taken out an exlusive lock on a relation. Exclusive locks typically prevent other operations on that relation from taking place, and can be a cause of "hung" queries that are waiting for a lock to be granted.
+
+### `all_locks`
+
+```
+$ rake pg_extras:all_locks
+```
+
+This command displays all the current locks, regardless of their type.
+
+### `outliers`
+
+```
+$ rake pg_extras:outliers
+                   qry                   |    exec_time     | prop_exec_time |   ncalls    | sync_io_time
+-----------------------------------------+------------------+----------------+-------------+--------------
+ SELECT * FROM archivable_usage_events.. | 154:39:26.431466 | 72.2%          | 34,211,877  | 00:00:00
+ COPY public.archivable_usage_events (.. | 50:38:33.198418  | 23.6%          | 13          | 13:34:21.00108
+ COPY public.usage_events (id, reporte.. | 02:32:16.335233  | 1.2%           | 13          | 00:34:19.784318
+ INSERT INTO usage_events (id, retaine.. | 01:42:59.436532  | 0.8%           | 12,328,187  | 00:00:00
+ SELECT * FROM usage_events WHERE (alp.. | 01:18:10.754354  | 0.6%           | 102,114,301 | 00:00:00
+ UPDATE usage_events SET reporter_id =.. | 00:52:35.683254  | 0.4%           | 23,786,348  | 00:00:00
+ INSERT INTO usage_events (id, retaine.. | 00:49:24.952561  | 0.4%           | 21,988,201  | 00:00:00
+ COPY public.app_ownership_events (id,.. | 00:37:14.31082   | 0.3%           | 13          | 00:12:32.584754
+ INSERT INTO app_ownership_events (id,.. | 00:26:59.808212  | 0.2%           | 383,109     | 00:00:00
+ SELECT * FROM app_ownership_events   .. | 00:19:06.021846  | 0.1%           | 744,879     | 00:00:00
+(10 rows)
+```
+
+This command displays statements, obtained from `pg_stat_statements`, ordered by the amount of time to execute in aggregate. This includes the statement itself, the total execution time for that statement, the proportion of total execution time for all statements that statement has taken up, the number of times that statement has been called, and the amount of time that statement spent on synchronous I/O (reading/writing from the filesystem).
+
+Typically, an efficient query will have an appropriate ratio of calls to total execution time, with as little time spent on I/O as possible. Queries that have a high total execution time but low call count should be investigated to improve their performance. Queries that have a high proportion of execution time being spent on synchronous I/O should also be investigated.
+
+### `calls`
+
+```
+$ rake pg_extras:calls
+                   qry                   |    exec_time     | prop_exec_time |   ncalls    | sync_io_time
+-----------------------------------------+------------------+----------------+-------------+--------------
+ SELECT * FROM usage_events WHERE (alp.. | 01:18:11.073333  | 0.6%           | 102,120,780 | 00:00:00
+ BEGIN                                   | 00:00:51.285988  | 0.0%           | 47,288,662  | 00:00:00
+ COMMIT                                  | 00:00:52.31724   | 0.0%           | 47,288,615  | 00:00:00
+ SELECT * FROM  archivable_usage_event.. | 154:39:26.431466 | 72.2%          | 34,211,877  | 00:00:00
+ UPDATE usage_events SET reporter_id =.. | 00:52:35.986167  | 0.4%           | 23,788,388  | 00:00:00
+ INSERT INTO usage_events (id, retaine.. | 00:49:25.260245  | 0.4%           | 21,990,326  | 00:00:00
+ INSERT INTO usage_events (id, retaine.. | 01:42:59.436532  | 0.8%           | 12,328,187  | 00:00:00
+ SELECT * FROM app_ownership_events   .. | 00:19:06.289521  | 0.1%           | 744,976     | 00:00:00
+ INSERT INTO app_ownership_events(id, .. | 00:26:59.885631  | 0.2%           | 383,153     | 00:00:00
+ UPDATE app_ownership_events SET app_i.. | 00:01:22.282337  | 0.0%           | 359,741     | 00:00:00
+(10 rows)
+```
+
+This command is much like `pg:outliers`, but ordered by the number of times a statement has been called.
+
+### `blocking`
+
+```
+$ rake pg_extras:blocking
+ blocked_pid |    blocking_statement    | blocking_duration | blocking_pid |                                        blocked_statement                           | blocked_duration
+-------------+--------------------------+-------------------+--------------+------------------------------------------------------------------------------------+------------------
+         461 | select count(*) from app | 00:00:03.838314   |        15682 | UPDATE "app" SET "updated_at" = '2013-03-04 15:07:04.746688' WHERE "id" = 12823149 | 00:00:03.821826
+(1 row)
+```
+
+This command displays statements that are currently holding locks that other statements are waiting to be released. This can be used in conjunction with `pg:locks` to determine which statements need to be terminated in order to resolve lock contention.
+
+#### `total_index_size`
+
+```
+$ rake pg_extras:total_index_size
+  size
+-------
+ 28194 MB
+(1 row)
+```
+
+This command displays the total size of all indexes on the database, in MB. It is calculated by taking the number of pages (reported in `relpages`) and multiplying it by the page size (8192 bytes).
+
+### `index_size`
+
+```
+$ rake pg_extras:index_size
+                             name                              |  size
+---------------------------------------------------------------+---------
+ idx_activity_attemptable_and_type_lesson_enrollment           | 5196 MB
+ index_enrollment_attemptables_by_attempt_and_last_in_group    | 4045 MB
+ index_attempts_on_student_id                                  | 2611 MB
+ enrollment_activity_attemptables_pkey                         | 2513 MB
+ index_attempts_on_student_id_final_attemptable_type           | 2466 MB
+ attempts_pkey                                                 | 2466 MB
+ index_attempts_on_response_id                                 | 2404 MB
+ index_attempts_on_enrollment_id                               | 1957 MB
+ index_enrollment_attemptables_by_enrollment_activity_id       | 1789 MB
+ enrollment_activities_pkey                                    |  458 MB
+ index_enrollment_activities_by_lesson_enrollment_and_activity |  402 MB
+ index_placement_attempts_on_response_id                       |  109 MB
+ index_placement_attempts_on_placement_test_id                 |  108 MB
+ index_placement_attempts_on_grade_level_id                    |   97 MB
+ index_lesson_enrollments_on_lesson_id                         |   93 MB
+(truncated results for brevity)
+```
+
+This command displays the size of each each index in the database, in MB. It is calculated by taking the number of pages (reported in `relpages`) and multiplying it by the page size (8192 bytes).
+
+### `table_size`
+
+```
+$ rake pg_extras:table_size
+                             name                              |  size
+---------------------------------------------------------------+---------
+ learning_coaches                                              |  196 MB
+ states                                                        |  145 MB
+ grade_levels                                                  |  111 MB
+ charities_customers                                           |   73 MB
+ charities                                                     |   66 MB
+(truncated results for brevity)
+```
+
+This command displays the size of each table in the database, in MB. It is calculated by using the system administration function `pg_table_size()`, which includes the size of the main data fork, free space map, visibility map and TOAST data.
+
+### `table_indexes_size`
+
+```
+$ rake pg_extras:table-indexes-size
+                             table                             | indexes_size
+---------------------------------------------------------------+--------------
+ learning_coaches                                              |    153 MB
+ states                                                        |    125 MB
+ charities_customers                                           |     93 MB
+ charities                                                     |     16 MB
+ grade_levels                                                  |     11 MB
+(truncated results for brevity)
+```
+
+This command displays the total size of indexes for each table, in MB. It is calcualtes by using the system administration function `pg_indexes_size()`.
+
+### `total_table_size`
+
+```
+$ rake pg_extras:total_table_size
+                             name                              |  size
+---------------------------------------------------------------+---------
+ learning_coaches                                              |  349 MB
+ states                                                        |  270 MB
+ charities_customers                                           |  166 MB
+ grade_levels                                                  |  122 MB
+ charities                                                     |   82 MB
+(truncated results for brevity)
+```
+
+This command displays the total size of each table in the database, in MB. It is calculated by using the system administration function `pg_total_relation_size()`, which includes table size, total index size and TOAST data.
+
+### `unused_indexes`
+
+```
+$ rake pg_extras:unused_indexes
+          table      |                       index                | index_size | index_scans
+---------------------+--------------------------------------------+------------+-------------
+ public.grade_levels | index_placement_attempts_on_grade_level_id | 97 MB      |           0
+ public.observations | observations_attrs_grade_resources         | 33 MB      |           0
+ public.messages     | user_resource_id_idx                       | 12 MB      |           0
+(3 rows)
+```
+
+This command displays indexes that have < 50 scans recorded against them, and are greater than 5 pages in size, ordered by size relative to the number of index scans. This command is generally useful for eliminating indexes that are unused, which can impact write performance, as well as read performance should they occupy space in memory.
+
+### `seq_scans`
+
+```
+$ rake pg_extras:seq_scans
+
+               name                |  count
+-----------------------------------+----------
+ learning_coaches                  | 44820063
+ states                            | 36794975
+ grade_levels                      | 13972293
+ charities_customers               |  8615277
+ charities                         |  4316276
+ messages                          |  3922247
+ contests_customers                |  2915972
+ classroom_goals                   |  2142014
+ contests                          |  1370267
+ goals                             |  1112659
+ districts                         |   158995
+ rollup_reports                    |   115942
+ customers                         |    93847
+ schools                           |    92984
+ classrooms                        |    92982
+ customer_settings                 |    91226
+(truncated results for brevity)
+```
+
+This command displays the number of sequential scans recorded against all tables, descending by count of sequential scans. Tables that have very high numbers of sequential scans may be underindexed, and it may be worth investigating queries that read from these tables.
+
+### long_running_queries
+
+```
+$ rake pg_extras:long_running_queries
+
+  pid  |    duration     |                                      query
+-------+-----------------+---------------------------------------------------------------------------------------
+ 19578 | 02:29:11.200129 | EXPLAIN SELECT  "students".* FROM "students"  WHERE "students"."id" = 1450645 LIMIT 1
+ 19465 | 02:26:05.542653 | EXPLAIN SELECT  "students".* FROM "students"  WHERE "students"."id" = 1889881 LIMIT 1
+ 19632 | 02:24:46.962818 | EXPLAIN SELECT  "students".* FROM "students"  WHERE "students"."id" = 1581884 LIMIT 1
+(truncated results for brevity)
+```
+
+This command displays currently running queries, that have been running for longer than 5 minutes, descending by duration. Very long running queries can be a source of multiple issues, such as preventing DDL statements completing or vacuum being unable to update `relfrozenxid`.
+
+### records_rank
+
+```
+$ rake pg_extras:records_rank
+               name                | estimated_count
+-----------------------------------+-----------------
+ tastypie_apiaccess                |          568891
+ notifications_event               |          381227
+ core_todo                         |          178614
+ core_comment                      |          123969
+ notifications_notification        |          102101
+ django_session                    |           68078
+ (truncated results for brevity)
+```
+
+This command displays an estimated count of rows per table, descending by estimated count. The estimated count is derived from `n_live_tup`, which is updated by vacuum operations. Due to the way `n_live_tup` is populated, sparse vs. dense pages can result in estimations that are significantly out from the real count of rows.
+
+### bloat
+
+```
+$ rake pg_extras:bloat
+
+ type  | schemaname |           object_name         | bloat |   waste
+-------+------------+-------------------------------+-------+----------
+ table | public     | bloated_table                 |   1.1 | 98 MB
+ table | public     | other_bloated_table           |   1.1 | 58 MB
+ index | public     | bloated_table::bloated_index  |   3.7 | 34 MB
+ table | public     | clean_table                   |   0.2 | 3808 kB
+ table | public     | other_clean_table             |   0.3 | 1576 kB
+```
+
+This command displays an estimation of table "bloat" – space allocated to a relation that is full of dead tuples, that has yet to be reclaimed. Tables that have a high bloat ratio, typically 10 or greater, should be investigated to see if vacuuming is aggressive enough, and can be a sign of high table churn.
+
+### vacuum_stats
+
+```
+$ rake pg_extras:vacuum_stats
+ schema |         table         | last_vacuum | last_autovacuum  |    rowcount    | dead_rowcount  | autovacuum_threshold | expect_autovacuum
+--------+-----------------------+-------------+------------------+----------------+----------------+----------------------+-------------------
+ public | log_table             |             | 2013-04-26 17:37 |         18,030 |              0 |          3,656       |
+ public | data_table            |             | 2013-04-26 13:09 |             79 |             28 |             66       |
+ public | other_table           |             | 2013-04-26 11:41 |             41 |             47 |             58       |
+ public | queue_table           |             | 2013-04-26 17:39 |             12 |          8,228 |             52       | yes
+ public | picnic_table          |             |                  |             13 |              0 |             53       |
+```
+
+This command displays statistics related to vacuum operations for each table, including an estiamtion of dead rows, last autovacuum and the current autovacuum threshold. This command can be useful when determining if current vacuum thresholds require adjustments, and to determine when the table was last vacuumed.
+
+### mandelbrot
+
+```
+$ rake pg_extras:mandelbrot
+```
+
+This command outputs the Mandelbrot set, calculated through SQL.
+
+## FAQ
+
+* Does is not violate the Heroku PG Extras license?
+
+The original plugin is MIT based so it means that copying and redistribution in any format is permitted.
+
+## Disclaimer
+
+This tool is in beta state.

data/Rakefile

@@ -0,0 +1,5 @@
+require "bundler/gem_tasks"
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+

data/lib/ruby-pg-extras.rb

@@ -0,0 +1,94 @@
+# frozen_string_literal: true
+
+require 'terminal-table'
+
+module RubyPGExtras
+  QUERIES = %i(
+    bloat blocking cache_hit
+    calls extensions
+    index_size index_usage locks all_locks
+    long_running_queries mandelbrot outliers
+    records_rank seq_scans table_indexes_size
+    table_size total_index_size total_table_size
+    unused_indexes vacuum_stats
+  )
+
+  QUERIES.each do |query_name|
+    define_singleton_method query_name do |options = { in_format: :display_table }|
+      run_query(
+        query_name: query_name,
+        in_format: options.fetch(:in_format)
+      )
+    end
+  end
+
+  def self.run_query(query_name:, in_format:)
+    result = connection.execute(
+      sql_for(query_name: query_name)
+    )
+
+    display_result(
+      result,
+      title: description_for(query_name: query_name),
+      in_format: in_format
+    )
+  end
+
+  def self.display_result(result, title:, in_format:)
+    case in_format
+    when :array
+      result.values
+    when :hash
+      result.to_a
+    when :raw
+      result
+    when :display_table
+      headings = if result.count > 0
+        result[0].keys
+      else
+        ["No results"]
+      end
+
+      puts Terminal::Table.new(
+        title: title,
+        headings: headings,
+        rows: result.values
+      )
+    else
+      raise "Invalid in_format option"
+    end
+  end
+
+  def self.description_for(query_name:)
+    first_line = File.open(
+      sql_path_for(query_name: query_name)
+    ) { |f| f.readline }
+
+    first_line[/\/\*(.*?)\*\//m, 1].strip
+  end
+
+  def self.sql_for(query_name:)
+    File.read(
+      sql_path_for(query_name: query_name)
+    )
+  end
+
+  def self.sql_path_for(query_name:)
+    File.join(File.dirname(__FILE__), "/ruby-pg-extras/queries/#{query_name}.sql")
+  end
+
+  def self.connection
+    ActiveRecord::Base.connection
+  end
+
+  %i(
+    connection
+    display_result
+    sql_for
+    sql_path_for
+  ).each do |method_name|
+    private_class_method method_name
+  end
+end
+
+require 'ruby-pg-extras/railtie' if defined?(Rails)

data/lib/ruby-pg-extras/queries/all_locks.sql

@@ -0,0 +1,16 @@
+/* Queries with active locks */
+
+SELECT
+  pg_stat_activity.pid,
+  pg_class.relname,
+  pg_locks.transactionid,
+  pg_locks.granted,
+  pg_locks.mode,
+  pg_stat_activity.query AS query_snippet,
+  age(now(),pg_stat_activity.query_start) AS "age"
+FROM pg_stat_activity,pg_locks left
+OUTER JOIN pg_class
+  ON (pg_locks.relation = pg_class.oid)
+WHERE pg_stat_activity.query <> '<insufficient privilege>'
+  AND pg_locks.pid = pg_stat_activity.pid
+  AND pg_stat_activity.pid <> pg_backend_pid() order by query_start;

data/lib/ruby-pg-extras/queries/bloat.sql

@@ -0,0 +1,63 @@
+/* Table and index bloat in your database ordered by most wasteful */
+
+WITH constants AS (
+  SELECT current_setting('block_size')::numeric AS bs, 23 AS hdr, 4 AS ma
+), bloat_info AS (
+  SELECT
+    ma,bs,schemaname,tablename,
+    (datawidth+(hdr+ma-(case when hdr%ma=0 THEN ma ELSE hdr%ma END)))::numeric AS datahdr,
+    (maxfracsum*(nullhdr+ma-(case when nullhdr%ma=0 THEN ma ELSE nullhdr%ma END))) AS nullhdr2
+  FROM (
+    SELECT
+      schemaname, tablename, hdr, ma, bs,
+      SUM((1-null_frac)*avg_width) AS datawidth,
+      MAX(null_frac) AS maxfracsum,
+      hdr+(
+        SELECT 1+count(*)/8
+        FROM pg_stats s2
+        WHERE null_frac<>0 AND s2.schemaname = s.schemaname AND s2.tablename = s.tablename
+      ) AS nullhdr
+    FROM pg_stats s, constants
+    GROUP BY 1,2,3,4,5
+  ) AS foo
+), table_bloat AS (
+  SELECT
+    schemaname, tablename, cc.relpages, bs,
+    CEIL((cc.reltuples*((datahdr+ma-
+      (CASE WHEN datahdr%ma=0 THEN ma ELSE datahdr%ma END))+nullhdr2+4))/(bs-20::float)) AS otta
+  FROM bloat_info
+  JOIN pg_class cc ON cc.relname = bloat_info.tablename
+  JOIN pg_namespace nn ON cc.relnamespace = nn.oid AND nn.nspname = bloat_info.schemaname AND nn.nspname <> 'information_schema'
+), index_bloat AS (
+  SELECT
+    schemaname, tablename, bs,
+    COALESCE(c2.relname,'?') AS iname, COALESCE(c2.reltuples,0) AS ituples, COALESCE(c2.relpages,0) AS ipages,
+    COALESCE(CEIL((c2.reltuples*(datahdr-12))/(bs-20::float)),0) AS iotta -- very rough approximation, assumes all cols
+  FROM bloat_info
+  JOIN pg_class cc ON cc.relname = bloat_info.tablename
+  JOIN pg_namespace nn ON cc.relnamespace = nn.oid AND nn.nspname = bloat_info.schemaname AND nn.nspname <> 'information_schema'
+  JOIN pg_index i ON indrelid = cc.oid
+  JOIN pg_class c2 ON c2.oid = i.indexrelid
+)
+SELECT
+  type, schemaname, object_name, bloat, pg_size_pretty(raw_waste) as waste
+FROM
+(SELECT
+  'table' as type,
+  schemaname,
+  tablename as object_name,
+  ROUND(CASE WHEN otta=0 THEN 0.0 ELSE table_bloat.relpages/otta::numeric END,1) AS bloat,
+  CASE WHEN relpages < otta THEN '0' ELSE (bs*(table_bloat.relpages-otta)::bigint)::bigint END AS raw_waste
+FROM
+  table_bloat
+    UNION
+SELECT
+  'index' as type,
+  schemaname,
+  tablename || '::' || iname as object_name,
+  ROUND(CASE WHEN iotta=0 OR ipages=0 THEN 0.0 ELSE ipages/iotta::numeric END,1) AS bloat,
+  CASE WHEN ipages < iotta THEN '0' ELSE (bs*(ipages-iotta))::bigint END AS raw_waste
+FROM
+  index_bloat) bloat_summary
+ORDER BY raw_waste DESC, bloat DESC;
+

data/lib/ruby-pg-extras/queries/blocking.sql

@@ -0,0 +1,16 @@
+/* Queries holding locks other queries are waiting to be released */
+
+SELECT bl.pid AS blocked_pid,
+  ka.query AS blocking_statement,
+  now() - ka.query_start AS blocking_duration,
+  kl.pid AS blocking_pid,
+  a.query AS blocked_statement,
+  now() - a.query_start AS blocked_duration
+FROM pg_catalog.pg_locks bl
+JOIN pg_catalog.pg_stat_activity a
+  ON bl.pid = a.pid
+JOIN pg_catalog.pg_locks kl
+  JOIN pg_catalog.pg_stat_activity ka
+    ON kl.pid = ka.pid
+ON bl.transactionid = kl.transactionid AND bl.pid != kl.pid
+WHERE NOT bl.granted;

data/lib/ruby-pg-extras/queries/cache_hit.sql

@@ -0,0 +1,11 @@
+/* Index and table hit rate */
+
+SELECT
+  'index hit rate' AS name,
+  (sum(idx_blks_hit)) / nullif(sum(idx_blks_hit + idx_blks_read),0) AS ratio
+FROM pg_statio_user_indexes
+UNION ALL
+SELECT
+ 'table hit rate' AS name,
+  sum(heap_blks_hit) / nullif(sum(heap_blks_hit) + sum(heap_blks_read),0) AS ratio
+FROM pg_statio_user_tables;

data/lib/ruby-pg-extras/queries/calls.sql

@@ -0,0 +1,9 @@
+/* 10 queries that have longest execution time in aggregate */
+
+SELECT query AS qry,
+interval '1 millisecond' * total_time AS exec_time,
+to_char((total_time/sum(total_time) OVER()) * 100, 'FM90D0') || '%'  AS prop_exec_time,
+to_char(calls, 'FM999G999G990') AS ncalls,
+interval '1 millisecond' * (blk_read_time + blk_write_time) AS sync_io_time
+FROM pg_stat_statements WHERE userid = (SELECT usesysid FROM pg_user WHERE usename = current_user LIMIT 1)
+ORDER BY calls DESC LIMIT 10;

data/lib/ruby-pg-extras/queries/extensions.sql

@@ -0,0 +1,4 @@
+/* Available and installed extensions */
+
+SELECT * FROM pg_available_extensions ORDER BY installed_version;
+

data/lib/ruby-pg-extras/queries/index_size.sql

@@ -0,0 +1,11 @@
+/* The size of indexes, descending by size */
+
+SELECT c.relname AS name,
+  pg_size_pretty(sum(c.relpages::bigint*8192)::bigint) AS size
+FROM pg_class c
+LEFT JOIN pg_namespace n ON (n.oid = c.relnamespace)
+WHERE n.nspname NOT IN ('pg_catalog', 'information_schema')
+AND n.nspname !~ '^pg_toast'
+AND c.relkind='i'
+GROUP BY c.relname
+ORDER BY sum(c.relpages) DESC;

data/lib/ruby-pg-extras/queries/index_usage.sql

@@ -0,0 +1,12 @@
+/* Index hit rate (effective databases are at 99% and up) */
+
+SELECT relname,
+   CASE idx_scan
+     WHEN 0 THEN 'Insufficient data'
+     ELSE (100 * idx_scan / (seq_scan + idx_scan))::text
+   END percent_of_times_index_used,
+   n_live_tup rows_in_table
+ FROM
+   pg_stat_user_tables
+ ORDER BY
+   n_live_tup DESC;

data/lib/ruby-pg-extras/queries/locks.sql

@@ -0,0 +1,16 @@
+/* Queries with active exclusive locks */
+
+SELECT
+  pg_stat_activity.pid,
+  pg_class.relname,
+  pg_locks.transactionid,
+  pg_locks.granted,
+  pg_stat_activity.query AS query_snippet,
+  age(now(),pg_stat_activity.query_start) AS "age"
+FROM pg_stat_activity,pg_locks left
+OUTER JOIN pg_class
+  ON (pg_locks.relation = pg_class.oid)
+WHERE pg_stat_activity.query <> '<insufficient privilege>'
+  AND pg_locks.pid = pg_stat_activity.pid
+  AND pg_locks.mode = 'ExclusiveLock'
+  AND pg_stat_activity.pid <> pg_backend_pid() order by query_start;

data/lib/ruby-pg-extras/queries/long_running_queries.sql

@@ -0,0 +1,14 @@
+/* All queries longer than five minutes by descending duration */
+
+SELECT
+  pid,
+  now() - pg_stat_activity.query_start AS duration,
+  query AS query
+FROM
+  pg_stat_activity
+WHERE
+  pg_stat_activity.query <> ''::text
+  AND state <> 'idle'
+  AND now() - pg_stat_activity.query_start > interval '5 minutes'
+ORDER BY
+  now() - pg_stat_activity.query_start DESC;

data/lib/ruby-pg-extras/queries/mandelbrot.sql

@@ -0,0 +1,21 @@
+/* The mandelbrot set */
+
+WITH RECURSIVE Z(IX, IY, CX, CY, X, Y, I) AS (
+          SELECT IX, IY, X::float, Y::float, X::float, Y::float, 0
+          FROM (select -2.2 + 0.031 * i, i from generate_series(0,101) as i) as xgen(x,ix),
+               (select -1.5 + 0.031 * i, i from generate_series(0,101) as i) as ygen(y,iy)
+          UNION ALL
+          SELECT IX, IY, CX, CY, X * X - Y * Y + CX AS X, Y * X * 2 + CY, I + 1
+          FROM Z
+          WHERE X * X + Y * Y < 16::float
+          AND I < 100
+    )
+SELECT array_to_string(array_agg(SUBSTRING(' .,,,-----++++%%%%@@@@#### ', LEAST(GREATEST(I,1),27), 1)),'')
+FROM (
+      SELECT IX, IY, MAX(I) AS I
+      FROM Z
+      GROUP BY IY, IX
+      ORDER BY IY, IX
+     ) AS ZT
+GROUP BY IY
+ORDER BY IY;

data/lib/ruby-pg-extras/queries/outliers.sql

@@ -0,0 +1,10 @@
+/* 10 queries that have longest execution time in aggregate */
+
+SELECT interval '1 millisecond' * total_time AS total_exec_time,
+to_char((total_time/sum(total_time) OVER()) * 100, 'FM90D0') || '%'  AS prop_exec_time,
+to_char(calls, 'FM999G999G999G990') AS ncalls,
+interval '1 millisecond' * (blk_read_time + blk_write_time) AS sync_io_time,
+query AS query
+FROM pg_stat_statements WHERE userid = (SELECT usesysid FROM pg_user WHERE usename = current_user LIMIT 1)
+ORDER BY total_time DESC
+LIMIT 10;

data/lib/ruby-pg-extras/queries/records_rank.sql

@@ -0,0 +1,9 @@
+/* All tables and the number of rows in each ordered by number of rows descending */
+
+SELECT
+  relname AS name,
+  n_live_tup AS estimated_count
+FROM
+  pg_stat_user_tables
+ORDER BY
+  n_live_tup DESC;

data/lib/ruby-pg-extras/queries/seq_scans.sql

@@ -0,0 +1,7 @@
+/* Count of sequential scans by table descending by order */
+
+SELECT relname AS name,
+       seq_scan as count
+FROM
+  pg_stat_user_tables
+ORDER BY seq_scan DESC;

data/lib/ruby-pg-extras/queries/table_indexes_size.sql

@@ -0,0 +1,10 @@
+/* Total size of all the indexes on each table, descending by size */
+
+SELECT c.relname AS table,
+  pg_size_pretty(pg_indexes_size(c.oid)) AS index_size
+FROM pg_class c
+LEFT JOIN pg_namespace n ON (n.oid = c.relnamespace)
+WHERE n.nspname NOT IN ('pg_catalog', 'information_schema')
+AND n.nspname !~ '^pg_toast'
+AND c.relkind='r'
+ORDER BY pg_indexes_size(c.oid) DESC;

data/lib/ruby-pg-extras/queries/table_size.sql

@@ -0,0 +1,10 @@
+/* Size of the tables (excluding indexes), descending by size */
+
+SELECT c.relname AS name,
+  pg_size_pretty(pg_table_size(c.oid)) AS size
+FROM pg_class c
+LEFT JOIN pg_namespace n ON (n.oid = c.relnamespace)
+WHERE n.nspname NOT IN ('pg_catalog', 'information_schema')
+AND n.nspname !~ '^pg_toast'
+AND c.relkind='r'
+ORDER BY pg_table_size(c.oid) DESC;

data/lib/ruby-pg-extras/queries/total_index_size.sql

@@ -0,0 +1,8 @@
+/* Total size of all indexes in MB */
+
+SELECT pg_size_pretty(sum(c.relpages::bigint*8192)::bigint) AS size
+FROM pg_class c
+LEFT JOIN pg_namespace n ON (n.oid = c.relnamespace)
+WHERE n.nspname NOT IN ('pg_catalog', 'information_schema')
+AND n.nspname !~ '^pg_toast'
+AND c.relkind='i';

data/lib/ruby-pg-extras/queries/total_table_size.sql

@@ -0,0 +1,10 @@
+/* Size of the tables (including indexes), descending by size */
+
+SELECT c.relname AS name,
+  pg_size_pretty(pg_total_relation_size(c.oid)) AS size
+FROM pg_class c
+LEFT JOIN pg_namespace n ON (n.oid = c.relnamespace)
+WHERE n.nspname NOT IN ('pg_catalog', 'information_schema')
+AND n.nspname !~ '^pg_toast'
+AND c.relkind='r'
+ORDER BY pg_total_relation_size(c.oid) DESC;

data/lib/ruby-pg-extras/queries/unused_indexes.sql

@@ -0,0 +1,16 @@
+/* Unused and almost unused indexes */
+/* Ordered by their size relative to the number of index scans.
+Exclude indexes of very small tables (less than 5 pages),
+where the planner will almost invariably select a sequential scan,
+but may not in the future as the table grows */
+
+SELECT
+  schemaname || '.' || relname AS table,
+  indexrelname AS index,
+  pg_size_pretty(pg_relation_size(i.indexrelid)) AS index_size,
+  idx_scan as index_scans
+FROM pg_stat_user_indexes ui
+JOIN pg_index i ON ui.indexrelid = i.indexrelid
+WHERE NOT indisunique AND idx_scan < 50 AND pg_relation_size(relid) > 5 * 8192
+ORDER BY pg_relation_size(i.indexrelid) / nullif(idx_scan, 0) DESC NULLS FIRST,
+pg_relation_size(i.indexrelid) DESC;

data/lib/ruby-pg-extras/queries/vacuum_stats.sql

@@ -0,0 +1,40 @@
+/* Dead rows and whether an automatic vacuum is expected to be triggered */
+
+WITH table_opts AS (
+  SELECT
+    pg_class.oid, relname, nspname, array_to_string(reloptions, '') AS relopts
+  FROM
+     pg_class INNER JOIN pg_namespace ns ON relnamespace = ns.oid
+), vacuum_settings AS (
+  SELECT
+    oid, relname, nspname,
+    CASE
+      WHEN relopts LIKE '%autovacuum_vacuum_threshold%'
+        THEN substring(relopts, '.*autovacuum_vacuum_threshold=([0-9.]+).*')::integer
+        ELSE current_setting('autovacuum_vacuum_threshold')::integer
+      END AS autovacuum_vacuum_threshold,
+    CASE
+      WHEN relopts LIKE '%autovacuum_vacuum_scale_factor%'
+        THEN substring(relopts, '.*autovacuum_vacuum_scale_factor=([0-9.]+).*')::real
+        ELSE current_setting('autovacuum_vacuum_scale_factor')::real
+      END AS autovacuum_vacuum_scale_factor
+  FROM
+    table_opts
+)
+SELECT
+  vacuum_settings.nspname AS schema,
+  vacuum_settings.relname AS table,
+  to_char(psut.last_vacuum, 'YYYY-MM-DD HH24:MI') AS last_vacuum,
+  to_char(psut.last_autovacuum, 'YYYY-MM-DD HH24:MI') AS last_autovacuum,
+  to_char(pg_class.reltuples, '9G999G999G999') AS rowcount,
+  to_char(psut.n_dead_tup, '9G999G999G999') AS dead_rowcount,
+  to_char(autovacuum_vacuum_threshold
+       + (autovacuum_vacuum_scale_factor::numeric * pg_class.reltuples), '9G999G999G999') AS autovacuum_threshold,
+  CASE
+    WHEN autovacuum_vacuum_threshold + (autovacuum_vacuum_scale_factor::numeric * pg_class.reltuples) < psut.n_dead_tup
+    THEN 'yes'
+  END AS expect_autovacuum
+FROM
+  pg_stat_user_tables psut INNER JOIN pg_class ON psut.relid = pg_class.oid
+    INNER JOIN vacuum_settings ON pg_class.oid = vacuum_settings.oid
+ORDER BY 1;

data/lib/ruby-pg-extras/tasks/all.rake

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+require 'ruby-pg-extras'
+
+namespace :pg_extras do
+  RubyPGExtras::QUERIES.each do |query_name|
+    desc RubyPGExtras.description_for(query_name: query_name)
+    task query_name.to_sym => :environment do
+      RubyPGExtras.public_send(query_name)
+    end
+  end
+end

data/lib/ruby-pg-extras/version.rb

@@ -0,0 +1,3 @@
+module RubyPGExtras
+  VERSION = "0.0.1"
+end

data/ruby-pg-extras.gemspec

@@ -0,0 +1,22 @@
+# -*- encoding: utf-8 -*-
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'ruby-pg-extras/version'
+
+Gem::Specification.new do |gem|
+  gem.name          = "ruby-pg-extras"
+  gem.version       = RubyPGExtras::VERSION
+  gem.authors       = ["pawurb"]
+  gem.email         = ["contact@pawelurbanek.com"]
+  gem.summary       = %q{ Ruby PostgreSQL performance database insights }
+  gem.description   = %q{ Ruby port of Heroku PG Extras. The goal of this project is to provide a powerful insights into PostgreSQL database for Ruby on Rails apps that are not using the default Heroku PostgreSQL plugin. }
+  gem.homepage      = "http://github.com/pawurb/ruby-pg-extras"
+  gem.files         = `git ls-files`.split("\n")
+  gem.test_files    = gem.files.grep(%r{^(spec)/})
+  gem.require_paths = ["lib"]
+  gem.license       = "MIT"
+  gem.add_dependency "pg"
+  gem.add_dependency "terminal-table"
+  gem.add_development_dependency "rake"
+  gem.add_development_dependency "rspec"
+end

data/spec/smoke_spec.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+require 'spec_helper'
+
+describe RubyPGExtras do
+  RubyPGExtras::QUERIES.each do |query_name|
+    it "#{query_name} description can be read" do
+      expect do
+        RubyPGExtras.description_for(
+          query_name: query_name
+        )
+      end.not_to raise_error
+    end
+  end
+
+  PG_STATS_DEPENDENT_QUERIES = %i(calls outliers)
+
+  (RubyPGExtras::QUERIES - PG_STATS_DEPENDENT_QUERIES).each do |query_name|
+    it "#{query_name} query can be executed" do
+      expect do
+        RubyPGExtras.run_query(
+          query_name: query_name,
+          in_format: :hash
+        )
+      end.not_to raise_error
+    end
+  end
+end

data/spec/spec_helper.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+require 'rubygems'
+require 'bundler/setup'
+require_relative '../lib/ruby-pg-extras'
+
+ENV["DATABASE_URL"] ||= "postgresql://postgres:secret@localhost:5432/ruby-pg-extras-test"
+
+RSpec.configure do |config|
+  config.before :suite do
+    ActiveRecord::Base.establish_connection(
+      ENV.fetch("DATABASE_URL")
+    )
+  end
+
+  config.after :suite do
+    ActiveRecord::Base.remove_connection
+  end
+end

metadata

@@ -0,0 +1,134 @@
+--- !ruby/object:Gem::Specification
+name: ruby-pg-extras
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+platform: ruby
+authors:
+- pawurb
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2020-01-08 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: pg
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: terminal-table
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: " Ruby port of Heroku PG Extras. The goal of this project is to provide
+  a powerful insights into PostgreSQL database for Ruby on Rails apps that are not
+  using the default Heroku PostgreSQL plugin. "
+email:
+- contact@pawelurbanek.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- lib/ruby-pg-extras.rb
+- lib/ruby-pg-extras/queries/all_locks.sql
+- lib/ruby-pg-extras/queries/bloat.sql
+- lib/ruby-pg-extras/queries/blocking.sql
+- lib/ruby-pg-extras/queries/cache_hit.sql
+- lib/ruby-pg-extras/queries/calls.sql
+- lib/ruby-pg-extras/queries/extensions.sql
+- lib/ruby-pg-extras/queries/index_size.sql
+- lib/ruby-pg-extras/queries/index_usage.sql
+- lib/ruby-pg-extras/queries/locks.sql
+- lib/ruby-pg-extras/queries/long_running_queries.sql
+- lib/ruby-pg-extras/queries/mandelbrot.sql
+- lib/ruby-pg-extras/queries/outliers.sql
+- lib/ruby-pg-extras/queries/records_rank.sql
+- lib/ruby-pg-extras/queries/seq_scans.sql
+- lib/ruby-pg-extras/queries/table_indexes_size.sql
+- lib/ruby-pg-extras/queries/table_size.sql
+- lib/ruby-pg-extras/queries/total_index_size.sql
+- lib/ruby-pg-extras/queries/total_table_size.sql
+- lib/ruby-pg-extras/queries/unused_indexes.sql
+- lib/ruby-pg-extras/queries/vacuum_stats.sql
+- lib/ruby-pg-extras/tasks/all.rake
+- lib/ruby-pg-extras/version.rb
+- ruby-pg-extras.gemspec
+- spec/smoke_spec.rb
+- spec/spec_helper.rb
+homepage: http://github.com/pawurb/ruby-pg-extras
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.0.6
+signing_key: 
+specification_version: 4
+summary: Ruby PostgreSQL performance database insights
+test_files:
+- spec/smoke_spec.rb
+- spec/spec_helper.rb