rails-pg-extras 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 00bbc417cc97e72d3815a72a4411d3ce44216259011f27f661d92bd433673731
-  data.tar.gz: 9fc774e5852ce88293dd0350783ac17f4c426777e9a1a87b31cd350eb3368c9e
+  metadata.gz: 7d015c34f17329bb7986f95030416c916c6fbbe017d417f41ef5989f2c6d3540
+  data.tar.gz: fdd19bc750196bf2d1943ec2c4136c5075736fb21d3e4bed1e13c3a6aa0156bb
 SHA512:
-  metadata.gz: 00a4ae363c93f55fed8d03fab9f8f558cff3967d010434fd20e712f4145a9d56feb6afdf4269b3cb567673592daaf60daf22e6ea929198a78bf44a4f6844a6ab
-  data.tar.gz: a64be50d53b081ff70db4512f4ea14ffe09bc0c73c18b62f38af5d9909d030c577c2176686bcb24d4ffa694431fd8ccfb9189b770a1cc547990c1d83483f038d
+  metadata.gz: e77d66aa4817bafc1abfa034327ba4463df0fce00199ea7728acd4284ecace4fbdef8d0b3754ff5136c2f083cec18cec2a13b4c95547ea959ca37d5e86a34bde
+  data.tar.gz: abe72eeffe9ea727d91e01581129ff66664b40aa5448630a50b31edf06c2114c4904f8bfdc7149ea37b51bc234f950f849585c31f9cab1a62195bda97ab8ddd3

data/README.md

@@ -1 +1,324 @@
 # Rails PG Extras
+
+Inspired by (a shameless ripoff of) [Heroku PG Extras](https://github.com/heroku/heroku-pg-extras). The goal of this project is to provide a powerful insights into PostgreSQL database for Ruby on Rails apps that are not using the default Heroku PostgreSQL plugin.
+
+Included rake tasks and Ruby methods can be used to obtain information about a Postgres instance, that may be useful when analyzing performance issues. This includes information about locks, index usage, buffer cache hit ratios and vacuum statistics. Ruby API enables developers to easily integrate the tool into e.g. automatic monitoring tasks.
+
+### Installation
+
+In your Gemfile
+
+```ruby
+gem 'rails-pg-extras'
+```
+
+### Usage
+
+Each command can be used as a rake task, or a directly from the Ruby code.
+
+#### `cache_hit`
+
+```bash
+$ rake pg_extras:cache_hit
+      name      |         ratio
+----------------+------------------------
+ index hit rate | 0.99957765013541945832
+ table hit rate |                   1.00
+(2 rows)
+```
+
+This command provides information on the efficiency of the buffer cache, for both index reads (`index hit rate`) as well as table reads (`table hit rate`). A low buffer cache hit ratio can be a sign that the Postgres instance is too small for the workload.
+
+#### `index_usage`
+
+```
+$ rake pg_extras:index_usage
+       relname       | percent_of_times_index_used | rows_in_table
+---------------------+-----------------------------+---------------
+ events              |                          65 |       1217347
+ app_infos           |                          74 |        314057
+ app_infos_user_info |                           0 |        198848
+ user_info           |                           5 |         94545
+ delayed_jobs        |                          27 |             0
+(5 rows)
+```
+
+This command provides information on the efficiency of indexes, represented as what percentage of total scans were index scans. A low percentage can indicate under indexing, or wrong data being indexed.
+
+### `locks`
+
+```
+$ rake pg_extras:locks
+ procpid | relname | transactionid | granted |     query_snippet     |       age
+---------+---------+---------------+---------+-----------------------+-----------------
+   31776 |         |               | t       | <IDLE> in transaction | 00:19:29.837898
+   31776 |         |          1294 | t       | <IDLE> in transaction | 00:19:29.837898
+   31912 |         |               | t       | select * from hello;  | 00:19:17.94259
+    3443 |         |               | t       |                      +| 00:00:00
+         |         |               |         |    select            +|
+         |         |               |         |      pg_stat_activi   |
+(4 rows)
+```
+
+This command displays queries that have taken out an exlusive lock on a relation. Exclusive locks typically prevent other operations on that relation from taking place, and can be a cause of "hung" queries that are waiting for a lock to be granted.
+
+### `pg:outliers`
+
+```
+$ rake pg_extras:outliers
+                   qry                   |    exec_time     | prop_exec_time |   ncalls    | sync_io_time
+-----------------------------------------+------------------+----------------+-------------+--------------
+ SELECT * FROM archivable_usage_events.. | 154:39:26.431466 | 72.2%          | 34,211,877  | 00:00:00
+ COPY public.archivable_usage_events (.. | 50:38:33.198418  | 23.6%          | 13          | 13:34:21.00108
+ COPY public.usage_events (id, reporte.. | 02:32:16.335233  | 1.2%           | 13          | 00:34:19.784318
+ INSERT INTO usage_events (id, retaine.. | 01:42:59.436532  | 0.8%           | 12,328,187  | 00:00:00
+ SELECT * FROM usage_events WHERE (alp.. | 01:18:10.754354  | 0.6%           | 102,114,301 | 00:00:00
+ UPDATE usage_events SET reporter_id =.. | 00:52:35.683254  | 0.4%           | 23,786,348  | 00:00:00
+ INSERT INTO usage_events (id, retaine.. | 00:49:24.952561  | 0.4%           | 21,988,201  | 00:00:00
+ COPY public.app_ownership_events (id,.. | 00:37:14.31082   | 0.3%           | 13          | 00:12:32.584754
+ INSERT INTO app_ownership_events (id,.. | 00:26:59.808212  | 0.2%           | 383,109     | 00:00:00
+ SELECT * FROM app_ownership_events   .. | 00:19:06.021846  | 0.1%           | 744,879     | 00:00:00
+(10 rows)
+```
+
+This command displays statements, obtained from `pg_stat_statements`, ordered by the amount of time to execute in aggregate. This includes the statement itself, the total execution time for that statement, the proportion of total execution time for all statements that statement has taken up, the number of times that statement has been called, and the amount of time that statement spent on synchronous I/O (reading/writing from the filesystem).
+
+Typically, an efficient query will have an appropriate ratio of calls to total execution time, with as little time spent on I/O as possible. Queries that have a high total execution time but low call count should be investigated to improve their performance. Queries that have a high proportion of execution time being spent on synchronous I/O should also be investigated.
+
+### `calls`
+
+```
+$ rake pg_extras:calls
+                   qry                   |    exec_time     | prop_exec_time |   ncalls    | sync_io_time
+-----------------------------------------+------------------+----------------+-------------+--------------
+ SELECT * FROM usage_events WHERE (alp.. | 01:18:11.073333  | 0.6%           | 102,120,780 | 00:00:00
+ BEGIN                                   | 00:00:51.285988  | 0.0%           | 47,288,662  | 00:00:00
+ COMMIT                                  | 00:00:52.31724   | 0.0%           | 47,288,615  | 00:00:00
+ SELECT * FROM  archivable_usage_event.. | 154:39:26.431466 | 72.2%          | 34,211,877  | 00:00:00
+ UPDATE usage_events SET reporter_id =.. | 00:52:35.986167  | 0.4%           | 23,788,388  | 00:00:00
+ INSERT INTO usage_events (id, retaine.. | 00:49:25.260245  | 0.4%           | 21,990,326  | 00:00:00
+ INSERT INTO usage_events (id, retaine.. | 01:42:59.436532  | 0.8%           | 12,328,187  | 00:00:00
+ SELECT * FROM app_ownership_events   .. | 00:19:06.289521  | 0.1%           | 744,976     | 00:00:00
+ INSERT INTO app_ownership_events(id, .. | 00:26:59.885631  | 0.2%           | 383,153     | 00:00:00
+ UPDATE app_ownership_events SET app_i.. | 00:01:22.282337  | 0.0%           | 359,741     | 00:00:00
+(10 rows)
+```
+
+This command is much like `pg:outliers`, but ordered by the number of times a statement has been called.
+
+### `blocking`
+
+```
+$ rake pg_extras:blocking
+ blocked_pid |    blocking_statement    | blocking_duration | blocking_pid |                                        blocked_statement                           | blocked_duration
+-------------+--------------------------+-------------------+--------------+------------------------------------------------------------------------------------+------------------
+         461 | select count(*) from app | 00:00:03.838314   |        15682 | UPDATE "app" SET "updated_at" = '2013-03-04 15:07:04.746688' WHERE "id" = 12823149 | 00:00:03.821826
+(1 row)
+```
+
+This command displays statements that are currently holding locks that other statements are waiting to be released. This can be used in conjunction with `pg:locks` to determine which statements need to be terminated in order to resolve lock contention.
+
+#### `total_index_size`
+
+```
+$ rake pg_extras:total_index_size
+  size
+-------
+ 28194 MB
+(1 row)
+```
+
+This command displays the total size of all indexes on the database, in MB. It is calculated by taking the number of pages (reported in `relpages`) and multiplying it by the page size (8192 bytes).
+
+### `index_size`
+
+```
+$ rake pg_extras:index_size
+                             name                              |  size
+---------------------------------------------------------------+---------
+ idx_activity_attemptable_and_type_lesson_enrollment           | 5196 MB
+ index_enrollment_attemptables_by_attempt_and_last_in_group    | 4045 MB
+ index_attempts_on_student_id                                  | 2611 MB
+ enrollment_activity_attemptables_pkey                         | 2513 MB
+ index_attempts_on_student_id_final_attemptable_type           | 2466 MB
+ attempts_pkey                                                 | 2466 MB
+ index_attempts_on_response_id                                 | 2404 MB
+ index_attempts_on_enrollment_id                               | 1957 MB
+ index_enrollment_attemptables_by_enrollment_activity_id       | 1789 MB
+ enrollment_activities_pkey                                    |  458 MB
+ index_enrollment_activities_by_lesson_enrollment_and_activity |  402 MB
+ index_placement_attempts_on_response_id                       |  109 MB
+ index_placement_attempts_on_placement_test_id                 |  108 MB
+ index_placement_attempts_on_grade_level_id                    |   97 MB
+ index_lesson_enrollments_on_lesson_id                         |   93 MB
+(truncated results for brevity)
+```
+
+This command displays the size of each each index in the database, in MB. It is calculated by taking the number of pages (reported in `relpages`) and multiplying it by the page size (8192 bytes).
+
+### `table_size`
+
+```
+$ rake pg_extras:table_size
+                             name                              |  size
+---------------------------------------------------------------+---------
+ learning_coaches                                              |  196 MB
+ states                                                        |  145 MB
+ grade_levels                                                  |  111 MB
+ charities_customers                                           |   73 MB
+ charities                                                     |   66 MB
+(truncated results for brevity)
+```
+
+This command displays the size of each table in the database, in MB. It is calculated by using the system administration function `pg_table_size()`, which includes the size of the main data fork, free space map, visibility map and TOAST data.
+
+### `table_indexes_size`
+
+```
+$ rake pg_extras:table-indexes-size
+                             table                             | indexes_size
+---------------------------------------------------------------+--------------
+ learning_coaches                                              |    153 MB
+ states                                                        |    125 MB
+ charities_customers                                           |     93 MB
+ charities                                                     |     16 MB
+ grade_levels                                                  |     11 MB
+(truncated results for brevity)
+```
+
+This command displays the total size of indexes for each table, in MB. It is calcualtes by using the system administration function `pg_indexes_size()`.
+
+### `total_table_size`
+
+```
+$ rake pg_extras:total_table_size
+                             name                              |  size
+---------------------------------------------------------------+---------
+ learning_coaches                                              |  349 MB
+ states                                                        |  270 MB
+ charities_customers                                           |  166 MB
+ grade_levels                                                  |  122 MB
+ charities                                                     |   82 MB
+(truncated results for brevity)
+```
+
+This command displays the total size of each table in the database, in MB. It is calculated by using the system administration function `pg_total_relation_size()`, which includes table size, total index size and TOAST data.
+
+### `unused_indexes`
+
+```
+$ rake pg_extras:unused_indexes
+          table      |                       index                | index_size | index_scans
+---------------------+--------------------------------------------+------------+-------------
+ public.grade_levels | index_placement_attempts_on_grade_level_id | 97 MB      |           0
+ public.observations | observations_attrs_grade_resources         | 33 MB      |           0
+ public.messages     | user_resource_id_idx                       | 12 MB      |           0
+(3 rows)
+```
+
+This command displays indexes that have < 50 scans recorded against them, and are greater than 5 pages in size, ordered by size relative to the number of index scans. This command is generally useful for eliminating indexes that are unused, which can impact write performance, as well as read performance should they occupy space in memory.
+
+### `seq_scans`
+
+```
+$ rake pg_extras:seq_scans
+
+               name                |  count
+-----------------------------------+----------
+ learning_coaches                  | 44820063
+ states                            | 36794975
+ grade_levels                      | 13972293
+ charities_customers               |  8615277
+ charities                         |  4316276
+ messages                          |  3922247
+ contests_customers                |  2915972
+ classroom_goals                   |  2142014
+ contests                          |  1370267
+ goals                             |  1112659
+ districts                         |   158995
+ rollup_reports                    |   115942
+ customers                         |    93847
+ schools                           |    92984
+ classrooms                        |    92982
+ customer_settings                 |    91226
+(truncated results for brevity)
+```
+
+This command displays the number of sequential scans recorded against all tables, descending by count of sequential scans. Tables that have very high numbers of sequential scans may be underindexed, and it may be worth investigating queries that read from these tables.
+
+### long_running_queries
+
+```
+$ rake pg_extras:long_running_queries
+
+  pid  |    duration     |                                      query
+-------+-----------------+---------------------------------------------------------------------------------------
+ 19578 | 02:29:11.200129 | EXPLAIN SELECT  "students".* FROM "students"  WHERE "students"."id" = 1450645 LIMIT 1
+ 19465 | 02:26:05.542653 | EXPLAIN SELECT  "students".* FROM "students"  WHERE "students"."id" = 1889881 LIMIT 1
+ 19632 | 02:24:46.962818 | EXPLAIN SELECT  "students".* FROM "students"  WHERE "students"."id" = 1581884 LIMIT 1
+(truncated results for brevity)
+```
+
+This command displays currently running queries, that have been running for longer than 5 minutes, descending by duration. Very long running queries can be a source of multiple issues, such as preventing DDL statements completing or vacuum being unable to update `relfrozenxid`.
+
+### records_rank
+
+```
+$ rake pg_extras:records_rank
+               name                | estimated_count
+-----------------------------------+-----------------
+ tastypie_apiaccess                |          568891
+ notifications_event               |          381227
+ core_todo                         |          178614
+ core_comment                      |          123969
+ notifications_notification        |          102101
+ django_session                    |           68078
+ (truncated results for brevity)
+```
+
+This command displays an estimated count of rows per table, descending by estimated count. The estimated count is derived from `n_live_tup`, which is updated by vacuum operations. Due to the way `n_live_tup` is populated, sparse vs. dense pages can result in estimations that are significantly out from the real count of rows.
+
+### bloat
+
+```
+$ rake pg_extras:bloat
+
+ type  | schemaname |           object_name         | bloat |   waste
+-------+------------+-------------------------------+-------+----------
+ table | public     | bloated_table                 |   1.1 | 98 MB
+ table | public     | other_bloated_table           |   1.1 | 58 MB
+ index | public     | bloated_table::bloated_index  |   3.7 | 34 MB
+ table | public     | clean_table                   |   0.2 | 3808 kB
+ table | public     | other_clean_table             |   0.3 | 1576 kB
+```
+
+This command displays an estimation of table "bloat" – space allocated to a relation that is full of dead tuples, that has yet to be reclaimed. Tables that have a high bloat ratio, typically 10 or greater, should be investigated to see if vacuuming is aggressive enough, and can be a sign of high table churn.
+
+### vacuum_stats
+
+```
+$ rake pg_extras:vacuum_stats
+ schema |         table         | last_vacuum | last_autovacuum  |    rowcount    | dead_rowcount  | autovacuum_threshold | expect_autovacuum
+--------+-----------------------+-------------+------------------+----------------+----------------+----------------------+-------------------
+ public | log_table             |             | 2013-04-26 17:37 |         18,030 |              0 |          3,656       |
+ public | data_table            |             | 2013-04-26 13:09 |             79 |             28 |             66       |
+ public | other_table           |             | 2013-04-26 11:41 |             41 |             47 |             58       |
+ public | queue_table           |             | 2013-04-26 17:39 |             12 |          8,228 |             52       | yes
+ public | picnic_table          |             |                  |             13 |              0 |             53       |
+```
+
+This command displays statistics related to vacuum operations for each table, including an estiamtion of dead rows, last autovacuum and the current autovacuum threshold. This command can be useful when determining if current vacuum thresholds require adjustments, and to determine when the table was last vacuumed.
+
+### pg:mandelbrot
+
+```
+$ rake pg_extras:mandelbrot
+```
+
+This command outputs the Mandelbrot set, calculated through SQL.
+
+## FAQ
+
+* Does is not violate the Heroku PG Extras license?
+
+The original plugin is [MIT based](https://github.com/heroku/heroku-pg-extras/blob/master/LICENSE) so it means that copying and redistribution in any format is permitted.

data/lib/rails-pg-extras/queries/bloat.rb

@@ -0,0 +1,73 @@
+# frozen_string_literal: true
+
+module RailsPGExtras
+  def self.bloat_description
+    "Table and index bloat in your database ordered by most wasteful"
+  end
+
+  def self.bloat_sql
+    <<-EOS
+WITH constants AS (
+  SELECT current_setting('block_size')::numeric AS bs, 23 AS hdr, 4 AS ma
+), bloat_info AS (
+  SELECT
+    ma,bs,schemaname,tablename,
+    (datawidth+(hdr+ma-(case when hdr%ma=0 THEN ma ELSE hdr%ma END)))::numeric AS datahdr,
+    (maxfracsum*(nullhdr+ma-(case when nullhdr%ma=0 THEN ma ELSE nullhdr%ma END))) AS nullhdr2
+  FROM (
+    SELECT
+      schemaname, tablename, hdr, ma, bs,
+      SUM((1-null_frac)*avg_width) AS datawidth,
+      MAX(null_frac) AS maxfracsum,
+      hdr+(
+        SELECT 1+count(*)/8
+        FROM pg_stats s2
+        WHERE null_frac<>0 AND s2.schemaname = s.schemaname AND s2.tablename = s.tablename
+      ) AS nullhdr
+    FROM pg_stats s, constants
+    GROUP BY 1,2,3,4,5
+  ) AS foo
+), table_bloat AS (
+  SELECT
+    schemaname, tablename, cc.relpages, bs,
+    CEIL((cc.reltuples*((datahdr+ma-
+      (CASE WHEN datahdr%ma=0 THEN ma ELSE datahdr%ma END))+nullhdr2+4))/(bs-20::float)) AS otta
+  FROM bloat_info
+  JOIN pg_class cc ON cc.relname = bloat_info.tablename
+  JOIN pg_namespace nn ON cc.relnamespace = nn.oid AND nn.nspname = bloat_info.schemaname AND nn.nspname <> 'information_schema'
+), index_bloat AS (
+  SELECT
+    schemaname, tablename, bs,
+    COALESCE(c2.relname,'?') AS iname, COALESCE(c2.reltuples,0) AS ituples, COALESCE(c2.relpages,0) AS ipages,
+    COALESCE(CEIL((c2.reltuples*(datahdr-12))/(bs-20::float)),0) AS iotta -- very rough approximation, assumes all cols
+  FROM bloat_info
+  JOIN pg_class cc ON cc.relname = bloat_info.tablename
+  JOIN pg_namespace nn ON cc.relnamespace = nn.oid AND nn.nspname = bloat_info.schemaname AND nn.nspname <> 'information_schema'
+  JOIN pg_index i ON indrelid = cc.oid
+  JOIN pg_class c2 ON c2.oid = i.indexrelid
+)
+SELECT
+  type, schemaname, object_name, bloat, pg_size_pretty(raw_waste) as waste
+FROM
+(SELECT
+  'table' as type,
+  schemaname,
+  tablename as object_name,
+  ROUND(CASE WHEN otta=0 THEN 0.0 ELSE table_bloat.relpages/otta::numeric END,1) AS bloat,
+  CASE WHEN relpages < otta THEN '0' ELSE (bs*(table_bloat.relpages-otta)::bigint)::bigint END AS raw_waste
+FROM
+  table_bloat
+    UNION
+SELECT
+  'index' as type,
+  schemaname,
+  tablename || '::' || iname as object_name,
+  ROUND(CASE WHEN iotta=0 OR ipages=0 THEN 0.0 ELSE ipages/iotta::numeric END,1) AS bloat,
+  CASE WHEN ipages < iotta THEN '0' ELSE (bs*(ipages-iotta))::bigint END AS raw_waste
+FROM
+  index_bloat) bloat_summary
+ORDER BY raw_waste DESC, bloat DESC
+EOS
+  end
+end
+

data/lib/rails-pg-extras/queries/blocking.rb

@@ -0,0 +1,26 @@
+# frozen_string_literal: true
+
+module RailsPGExtras
+  def self.blocking_description
+    "Queries holding locks other queries are waiting to be released"
+  end
+
+  def self.blocking_sql
+    <<-EOS
+SELECT bl.pid AS blocked_pid,
+  ka.query AS blocking_statement,
+  now() - ka.query_start AS blocking_duration,
+  kl.pid AS blocking_pid,
+  a.query AS blocked_statement,
+  now() - a.query_start AS blocked_duration
+FROM pg_catalog.pg_locks bl
+JOIN pg_catalog.pg_stat_activity a
+  ON bl.pid = a.pid
+JOIN pg_catalog.pg_locks kl
+  JOIN pg_catalog.pg_stat_activity ka
+    ON kl.pid = ka.pid
+ON bl.transactionid = kl.transactionid AND bl.pid != kl.pid
+WHERE NOT bl.granted
+EOS
+  end
+end

data/lib/rails-pg-extras/queries/cache_hit.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+module RailsPGExtras
+  def self.cache_hit_description
+    "Index and table hit rate"
+  end
+
+  def self.cache_hit_sql
+    <<-EOS
+SELECT
+  'index hit rate' AS name,
+  (sum(idx_blks_hit)) / nullif(sum(idx_blks_hit + idx_blks_read),0) AS ratio
+FROM pg_statio_user_indexes
+UNION ALL
+SELECT
+ 'table hit rate' AS name,
+  sum(heap_blks_hit) / nullif(sum(heap_blks_hit) + sum(heap_blks_read),0) AS ratio
+FROM pg_statio_user_tables;
+EOS
+  end
+end

data/lib/rails-pg-extras/queries/calls.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+module RailsPGExtras
+  def self.calls_description
+    "10 queries that have longest execution time in aggregate"
+  end
+
+  def self.calls_sql
+    <<-EOS
+SELECT query AS qry,
+interval '1 millisecond' * total_time AS exec_time,
+to_char((total_time/sum(total_time) OVER()) * 100, 'FM90D0') || '%'  AS prop_exec_time,
+to_char(calls, 'FM999G999G990') AS ncalls,
+interval '1 millisecond' * (blk_read_time + blk_write_time) AS sync_io_time
+FROM pg_stat_statements WHERE userid = (SELECT usesysid FROM pg_user WHERE usename = current_user LIMIT 1)
+ORDER BY calls DESC LIMIT 10
+EOS
+  end
+end

data/lib/rails-pg-extras/queries/extensions.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+module RailsPGExtras
+  def self.extensions_description
+    "Available and installed extensions"
+  end
+
+  def self.extensions_sql
+    <<-EOS
+SELECT * FROM pg_available_extensions ORDER BY installed_version;
+EOS
+  end
+end

data/lib/rails-pg-extras/queries/index_size.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+module RailsPGExtras
+  def self.index_size_description
+    "The size of indexes, descending by size"
+  end
+
+  def self.index_size_sql
+    <<-EOS
+SELECT c.relname AS name,
+  pg_size_pretty(sum(c.relpages::bigint*8192)::bigint) AS size
+FROM pg_class c
+LEFT JOIN pg_namespace n ON (n.oid = c.relnamespace)
+WHERE n.nspname NOT IN ('pg_catalog', 'information_schema')
+AND n.nspname !~ '^pg_toast'
+AND c.relkind='i'
+GROUP BY c.relname
+ORDER BY sum(c.relpages) DESC;
+EOS
+  end
+end

data/lib/rails-pg-extras/queries/index_usage.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+module RailsPGExtras
+  def self.index_usage_description
+    "Index hit rate (effective databases are at 99% and up)"
+  end
+
+  def self.index_usage_sql
+    <<-EOS
+SELECT relname,
+   CASE idx_scan
+     WHEN 0 THEN 'Insufficient data'
+     ELSE (100 * idx_scan / (seq_scan + idx_scan))::text
+   END percent_of_times_index_used,
+   n_live_tup rows_in_table
+ FROM
+   pg_stat_user_tables
+ ORDER BY
+   n_live_tup DESC;
+EOS
+  end
+end

data/lib/rails-pg-extras/queries/locks.rb

@@ -0,0 +1,26 @@
+# frozen_string_literal: true
+
+module RailsPGExtras
+  def self.locks_description
+    "Queries with active locks"
+  end
+
+  def self.locks_sql
+    <<-EOS
+SELECT
+  pg_stat_activity.pid,
+  pg_class.relname,
+  pg_locks.transactionid,
+  pg_locks.granted,
+  pg_stat_activity.query AS query_snippet,
+  age(now(),pg_stat_activity.query_start) AS "age"
+FROM pg_stat_activity,pg_locks left
+OUTER JOIN pg_class
+  ON (pg_locks.relation = pg_class.oid)
+WHERE pg_stat_activity.query <> '<insufficient privilege>'
+  AND pg_locks.pid = pg_stat_activity.pid
+  AND pg_locks.mode = 'ExclusiveLock'
+  AND pg_stat_activity.pid <> pg_backend_pid() order by query_start;
+EOS
+  end
+end

data/lib/rails-pg-extras/queries/long_running_queries.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module RailsPGExtras
+  def self.long_running_queries_description
+    "All queries longer than five minutes by descending duration"
+  end
+
+  def self.long_running_queries_sql
+    <<-EOS
+SELECT
+  pid,
+  now() - pg_stat_activity.query_start AS duration,
+  query AS query
+FROM
+  pg_stat_activity
+WHERE
+  pg_stat_activity.query <> ''::text
+  AND state <> 'idle'
+  AND now() - pg_stat_activity.query_start > interval '5 minutes'
+ORDER BY
+  now() - pg_stat_activity.query_start DESC;
+EOS
+  end
+end

data/lib/rails-pg-extras/queries/mandelbrot.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module RailsPGExtras
+  def self.mandelbrot_description
+    "The mandelbrot set"
+  end
+
+  def self.mandelbrot_sql
+    <<-EOS
+WITH RECURSIVE Z(IX, IY, CX, CY, X, Y, I) AS (
+          SELECT IX, IY, X::float, Y::float, X::float, Y::float, 0
+          FROM (select -2.2 + 0.031 * i, i from generate_series(0,101) as i) as xgen(x,ix),
+               (select -1.5 + 0.031 * i, i from generate_series(0,101) as i) as ygen(y,iy)
+          UNION ALL
+          SELECT IX, IY, CX, CY, X * X - Y * Y + CX AS X, Y * X * 2 + CY, I + 1
+          FROM Z
+          WHERE X * X + Y * Y < 16::float
+          AND I < 100
+    )
+SELECT array_to_string(array_agg(SUBSTRING(' .,,,-----++++%%%%@@@@#### ', LEAST(GREATEST(I,1),27), 1)),'')
+FROM (
+      SELECT IX, IY, MAX(I) AS I
+      FROM Z
+      GROUP BY IY, IX
+      ORDER BY IY, IX
+     ) AS ZT
+GROUP BY IY
+ORDER BY IY
+EOS
+  end
+end

data/lib/rails-pg-extras/queries/outliers.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+module RailsPGExtras
+  def self.outliers_description
+    "10 queries that have longest execution time in aggregate"
+  end
+
+  def self.outliers_sql
+    <<-EOS
+SELECT interval '1 millisecond' * total_time AS total_exec_time,
+to_char((total_time/sum(total_time) OVER()) * 100, 'FM90D0') || '%'  AS prop_exec_time,
+to_char(calls, 'FM999G999G999G990') AS ncalls,
+interval '1 millisecond' * (blk_read_time + blk_write_time) AS sync_io_time,
+query AS query
+FROM pg_stat_statements WHERE userid = (SELECT usesysid FROM pg_user WHERE usename = current_user LIMIT 1)
+ORDER BY total_time DESC
+LIMIT 10
+EOS
+  end
+end

data/lib/rails-pg-extras/queries/records_rank.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+module RailsPGExtras
+  def self.records_rank_description
+    "All tables and the number of rows in each ordered by number of rows descending"
+  end
+
+  def self.records_rank_sql
+    <<-EOS
+SELECT
+  relname AS name,
+  n_live_tup AS estimated_count
+FROM
+  pg_stat_user_tables
+ORDER BY
+  n_live_tup DESC;
+EOS
+  end
+end

data/lib/rails-pg-extras/queries/seq_scans.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+module RailsPGExtras
+  def self.seq_scans_description
+    "Count of sequential scans by table descending by order"
+  end
+
+  def self.seq_scans_sql
+    <<-EOS
+SELECT relname AS name,
+       seq_scan as count
+FROM
+  pg_stat_user_tables
+ORDER BY seq_scan DESC;
+EOS
+  end
+end

data/lib/rails-pg-extras/queries/table_indexes_size.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+module RailsPGExtras
+  def self.table_indexes_size_description
+    "Total size of all the indexes on each table, descending by size"
+  end
+
+  def self.table_indexes_size_sql
+    <<-EOS
+SELECT c.relname AS table,
+  pg_size_pretty(pg_indexes_size(c.oid)) AS index_size
+FROM pg_class c
+LEFT JOIN pg_namespace n ON (n.oid = c.relnamespace)
+WHERE n.nspname NOT IN ('pg_catalog', 'information_schema')
+AND n.nspname !~ '^pg_toast'
+AND c.relkind='r'
+ORDER BY pg_indexes_size(c.oid) DESC;
+EOS
+  end
+end

data/lib/rails-pg-extras/queries/table_size.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+module RailsPGExtras
+  def self.table_size_description
+    "Size of the tables (excluding indexes), descending by size"
+  end
+
+  def self.table_size_sql
+    <<-EOS
+SELECT c.relname AS name,
+  pg_size_pretty(pg_table_size(c.oid)) AS size
+FROM pg_class c
+LEFT JOIN pg_namespace n ON (n.oid = c.relnamespace)
+WHERE n.nspname NOT IN ('pg_catalog', 'information_schema')
+AND n.nspname !~ '^pg_toast'
+AND c.relkind='r'
+ORDER BY pg_table_size(c.oid) DESC;
+EOS
+  end
+end

data/lib/rails-pg-extras/queries/total_index_size.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+module RailsPGExtras
+  def self.total_index_size_description
+    "Total size of all indexes in MB"
+  end
+
+  def self.total_index_size_sql
+    <<-EOS
+SELECT pg_size_pretty(sum(c.relpages::bigint*8192)::bigint) AS size
+FROM pg_class c
+LEFT JOIN pg_namespace n ON (n.oid = c.relnamespace)
+WHERE n.nspname NOT IN ('pg_catalog', 'information_schema')
+AND n.nspname !~ '^pg_toast'
+AND c.relkind='i';
+EOS
+  end
+end

data/lib/rails-pg-extras/queries/total_table_size.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+module RailsPGExtras
+  def self.total_table_size_description
+    "Size of the tables (including indexes), descending by size"
+  end
+
+  def self.total_table_size_sql
+    <<-EOS
+SELECT c.relname AS name,
+  pg_size_pretty(pg_total_relation_size(c.oid)) AS size
+FROM pg_class c
+LEFT JOIN pg_namespace n ON (n.oid = c.relnamespace)
+WHERE n.nspname NOT IN ('pg_catalog', 'information_schema')
+AND n.nspname !~ '^pg_toast'
+AND c.relkind='r'
+ORDER BY pg_total_relation_size(c.oid) DESC;
+EOS
+  end
+end

data/lib/rails-pg-extras/queries/unused_indexes.rb

@@ -0,0 +1,26 @@
+# frozen_string_literal: true
+
+module RailsPGExtras
+  def self.unused_indexes_description
+    "Unused and almost unused indexes"
+  end
+  # Ordered by their size relative to the number of index scans.
+  # Exclude indexes of very small tables (less than 5 pages),
+  # where the planner will almost invariably select a sequential scan,
+  # but may not in the future as the table grows
+
+  def self.unused_indexes_sql
+    <<-EOS
+SELECT
+  schemaname || '.' || relname AS table,
+  indexrelname AS index,
+  pg_size_pretty(pg_relation_size(i.indexrelid)) AS index_size,
+  idx_scan as index_scans
+FROM pg_stat_user_indexes ui
+JOIN pg_index i ON ui.indexrelid = i.indexrelid
+WHERE NOT indisunique AND idx_scan < 50 AND pg_relation_size(relid) > 5 * 8192
+ORDER BY pg_relation_size(i.indexrelid) / nullif(idx_scan, 0) DESC NULLS FIRST,
+pg_relation_size(i.indexrelid) DESC;
+EOS
+  end
+end

data/lib/rails-pg-extras/queries/vacuum_stats.rb

@@ -0,0 +1,51 @@
+# frozen_string_literal: true
+
+module RailsPGExtras
+  def self.vacuum_stats_description
+    "Dead rows and whether an automatic vacuum is expected to be triggered"
+  end
+
+  def self.vacuum_stats_sql
+    <<-EOS
+WITH table_opts AS (
+  SELECT
+    pg_class.oid, relname, nspname, array_to_string(reloptions, '') AS relopts
+  FROM
+     pg_class INNER JOIN pg_namespace ns ON relnamespace = ns.oid
+), vacuum_settings AS (
+  SELECT
+    oid, relname, nspname,
+    CASE
+      WHEN relopts LIKE '%autovacuum_vacuum_threshold%'
+        THEN substring(relopts, '.*autovacuum_vacuum_threshold=([0-9.]+).*')::integer
+        ELSE current_setting('autovacuum_vacuum_threshold')::integer
+      END AS autovacuum_vacuum_threshold,
+    CASE
+      WHEN relopts LIKE '%autovacuum_vacuum_scale_factor%'
+        THEN substring(relopts, '.*autovacuum_vacuum_scale_factor=([0-9.]+).*')::real
+        ELSE current_setting('autovacuum_vacuum_scale_factor')::real
+      END AS autovacuum_vacuum_scale_factor
+  FROM
+    table_opts
+)
+SELECT
+  vacuum_settings.nspname AS schema,
+  vacuum_settings.relname AS table,
+  to_char(psut.last_vacuum, 'YYYY-MM-DD HH24:MI') AS last_vacuum,
+  to_char(psut.last_autovacuum, 'YYYY-MM-DD HH24:MI') AS last_autovacuum,
+  to_char(pg_class.reltuples, '9G999G999G999') AS rowcount,
+  to_char(psut.n_dead_tup, '9G999G999G999') AS dead_rowcount,
+  to_char(autovacuum_vacuum_threshold
+       + (autovacuum_vacuum_scale_factor::numeric * pg_class.reltuples), '9G999G999G999') AS autovacuum_threshold,
+  CASE
+    WHEN autovacuum_vacuum_threshold + (autovacuum_vacuum_scale_factor::numeric * pg_class.reltuples) < psut.n_dead_tup
+    THEN 'yes'
+  END AS expect_autovacuum
+FROM
+  pg_stat_user_tables psut INNER JOIN pg_class ON psut.relid = pg_class.oid
+    INNER JOIN vacuum_settings ON pg_class.oid = vacuum_settings.oid
+ORDER BY 1
+EOS
+  end
+end
+

data/lib/rails-pg-extras/railtie.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+class RailsPGExtras::Railtie < Rails::Railtie
+  rake_tasks do
+    load 'rails-pg-extras/tasks/all.rake'
+  end
+end

data/lib/rails-pg-extras/tasks/all.rake

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+require 'rails-pg-extras'
+
+namespace :pg_extras do
+  RailsPGExtras::QUERIES.each do |query_name|
+    desc RailsPGExtras.public_send("#{query_name}_description")
+    task query_name.to_sym => :environment do
+      RailsPGExtras.public_send(query_name)
+    end
+  end
+end

data/lib/rails-pg-extras/version.rb

@@ -1,3 +1,3 @@
 module RailsPGExtras
-  VERSION = "0.1.0"
+  VERSION = "0.2.0"
 end

data/lib/rails-pg-extras.rb

@@ -1,2 +1,65 @@
+# frozen_string_literal: true
+
+require 'terminal-table'
+require 'rails-pg-extras/railtie' if defined?(Rails)
+
 module RailsPGExtras
+  QUERIES =   %i(
+    bloat blocking cache_hit
+     calls extensions
+     index_size index_usage locks
+     long_running_queries mandelbrot outliers
+     records_rank seq_scans table_indexes_size
+     table_size total_index_size total_table_size
+     unused_indexes vacuum_stats
+  )
+
+  QUERIES.each do |query_name|
+    require "rails-pg-extras/queries/#{query_name}"
+
+    define_singleton_method query_name do |options = { in_format: :display_table }|
+      run_query(
+        query_name: query_name,
+        in_format: options.fetch(:in_format)
+      )
+    end
+  end
+
+  def self.run_query(query_name:, in_format:)
+    result = connection.execute(self.public_send("#{query_name}_sql"))
+    title = self.public_send("#{query_name}_description")
+    display_result(result, title: title, in_format: in_format)
+  end
+
+  def self.display_result(result, title:, in_format:)
+    case in_format
+    when :array
+      result.values
+    when :hash
+      result.to_a
+    when :raw
+      result
+    when :display_table
+      headings = if result.count > 0
+        result[0].keys
+      else
+        ["No results"]
+      end
+
+      puts Terminal::Table.new(
+        title: title,
+        headings: headings,
+        rows: result.values
+      )
+    else
+      raise "Invalid in_format option"
+    end
+  end
+
+  def self.connection
+    ActiveRecord::Base.connection
+  end
+
+  private_class_method :connection
+  private_class_method :display_result
 end

data/rails-pg-extras.gemspec

@@ -15,4 +15,5 @@ Gem::Specification.new do |gem|
   gem.require_paths = ["lib"]
   gem.license       = "MIT"
   gem.add_dependency "activerecord"
+  gem.add_dependency "terminal-table"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rails-pg-extras
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - pawurb
@@ -24,6 +24,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: terminal-table
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: " A bunch of rake tasks for showing what's going on inside your Rails
   PostgreSQL database "
 email:
@@ -41,7 +55,27 @@ files:
 - coverage/.resultset.json
 - coverage/.resultset.json.lock
 - lib/rails-pg-extras.rb
-- lib/rails-pg-extras/main.rb
+- lib/rails-pg-extras/queries/bloat.rb
+- lib/rails-pg-extras/queries/blocking.rb
+- lib/rails-pg-extras/queries/cache_hit.rb
+- lib/rails-pg-extras/queries/calls.rb
+- lib/rails-pg-extras/queries/extensions.rb
+- lib/rails-pg-extras/queries/index_size.rb
+- lib/rails-pg-extras/queries/index_usage.rb
+- lib/rails-pg-extras/queries/locks.rb
+- lib/rails-pg-extras/queries/long_running_queries.rb
+- lib/rails-pg-extras/queries/mandelbrot.rb
+- lib/rails-pg-extras/queries/outliers.rb
+- lib/rails-pg-extras/queries/records_rank.rb
+- lib/rails-pg-extras/queries/seq_scans.rb
+- lib/rails-pg-extras/queries/table_indexes_size.rb
+- lib/rails-pg-extras/queries/table_size.rb
+- lib/rails-pg-extras/queries/total_index_size.rb
+- lib/rails-pg-extras/queries/total_table_size.rb
+- lib/rails-pg-extras/queries/unused_indexes.rb
+- lib/rails-pg-extras/queries/vacuum_stats.rb
+- lib/rails-pg-extras/railtie.rb
+- lib/rails-pg-extras/tasks/all.rake
 - lib/rails-pg-extras/version.rb
 - rails-pg-extras.gemspec
 homepage: http://github.com/pawurb/rails-pg-extras