awesome_explain 0.3.0 → 1.0.0

data/README.md

@@ -1,11 +1,15 @@
 # AwesomeExplain
 
-Awesome explain is a simple global method that provides quick insights into mongodb's query plan and execution stats.
-Currently the explain functionality only supports `Mongo::Collection::View::Aggregation` & ``Mongoid::Criteria`.
+AwesomeExplain provides the same APM's level of query analysis under your development and test Rails environments.
 
-[![Build Status](https://travis-ci.com/sandboxws/awesome_explain.svg?branch=master)](https://travis-ci.com/sandboxws/awesome_explain)
-[![Maintainability](https://api.codeclimate.com/v1/badges/75e1a5cb4b6a5c1ba4c8/maintainability)](https://codeclimate.com/github/sandboxws/awesome_explain/maintainability)
-[![Test Coverage](https://api.codeclimate.com/v1/badges/75e1a5cb4b6a5c1ba4c8/test_coverage)](https://codeclimate.com/github/sandboxws/awesome_explain/test_coverage)
+## Main Features
+
+* A set of utilities for analyzing MongoDB and SQL queries from Rails console.
+* Tracking queries under your database of choice (SQLite3 or PostgreSQL)
+which can be viewed under [Athena's](https://github.com/sandboxws/athena_dashboard) dashboard.
+
+![Build Status](https://github.com/sandboxws/awesome_explain/actions/workflows/mongodb.yml/badge.svg)
+![Build Status](https://github.com/sandboxws/awesome_explain/actions/workflows/postgres.yml/badge.svg)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
 ## Installation
@@ -14,7 +18,21 @@ Add the following line to your application's Gemfile:
 
 `gem 'awesome_explain', require: true`
 
-## Usage
+## Console Utility Methods
+
+* **ae**: Prints a query's execution plan using a user friendly terminal table
+  * Currently supports the following classes:
+    * `Mongo::Collection::View::Aggregation`
+    * `Mongoid::Criteria`
+    * `ActiveRecord::Relation`
+* **analyze**: Prints a summary of all MongoDB queries passed to the Ruby block
+* **analyze_ar**: Prints a summary of all ActiveRecord queries passed to the Ruby block
+
+*Detailed usage examples can be found below.*
+
+## MongoDB
+
+### Usage
 
 `ae Article.where(author_id: '5b9ec484d5cc2e697189d7c9')`
 
@@ -61,7 +79,29 @@ Add the following line to your application's Gemfile:
 +--------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
 ```
 
-## Winning Plan Examples
+`ae Product.where(_id: 22)`
+
+```
++--------------------+----------------+
+| Winning Plan       | IDHACK (1 / 1) |
++--------------------+----------------+
+| Used Indexes       |                |
++--------------------+----------------+
+| Rejected Plans     | 0              |
++--------------------+----------------+
+| Documents Returned | 1              |
++--------------------+----------------+
+| Documents Examined | 1              |
++--------------------+----------------+
+| Keys Examined      | 1              |
++--------------------+----------------+
+| Execution time(ms) | 90             |
++--------------------+----------------+
+| Execution time(s)  | 0.09           |
++--------------------+----------------+
+```
+
+### Winning Plan Examples
 
 `FETCH (7 / 7) -> IXSCAN (7)`
 
@@ -72,3 +112,111 @@ Below is a breakdown of the above winning plan:
 
 For information about MongoDB's explain output, please refer to the official MongoDB Explain documentation:
 https://docs.mongodb.com/manual/reference/explain-results/
+
+## PostgreSQL
+
+### ae Usage
+
+#### Query using PK index
+
+`ae Film.where(film_id: 1)`
+
+```
++--------------------+-------+
+|       General Stats        |
++--------------------+-------+
+| Table              | Count |
++--------------------+-------+
+| Total Rows Planned | 1     |
+| Total Rows         | 1     |
+| Total Loops        | 1     |
+| Seq Scans          | 0     |
+| Indexes Used       | 1     |
++--------------------+-------+
+|        Table Stats         |
++--------------------+-------+
+| Table              | Count |
++--------------------+-------+
+| film               | 1     |
++--------------------+-------+
+|      Node Type Stats       |
++--------------------+-------+
+| Node Type          | Count |
++--------------------+-------+
+| Index Scan         | 1     |
++--------------------+-------+
+|        Index Stats         |
++--------------------+-------+
+| Index Name         | Count |
++--------------------+-------+
+| film_pkey          | 1     |
++--------------------+-------+
+```
+
+#### Query not using any index
+
+`ae Film.where(description: 'Alien Center')`
+
+```
++--------------------+-------+
+|       General Stats        |
++--------------------+-------+
+| Table              | Count |
++--------------------+-------+
+| Total Rows Planned | 1     |
+| Total Rows         | 0     |
+| Total Loops        | 1     |
+| Seq Scans          | 1     |
+| Indexes Used       | 0     |
++--------------------+-------+
+|        Table Stats         |
++--------------------+-------+
+| Table              | Count |
++--------------------+-------+
+| film               | 1     |
++--------------------+-------+
+|      Node Type Stats       |
++--------------------+-------+
+| Node Type          | Count |
++--------------------+-------+
+| Seq Scan           | 1     |
++--------------------+-------+
+```
+
+### analyze_ar Usage
+
+`analyze_ar { Film.where(film_id: 1).to_a; Actor.where(last_name: 'Cage').to_a };0`
+
+```
++--------------------+----------------+
+| Time (sec)         | 0.0            |
++--------------------+----------------+
+| Total Rows Planned | 3              |
++--------------------+----------------+
+| Total Rows         | 3              |
++--------------------+----------------+
+| Total Loops        | 2              |
++--------------------+----------------+
+| Seq Scans          | 1              |
++--------------------+----------------+
+| Tables             | film (1)       |
+|                    | actor (1)      |
++--------------------+----------------+
+| Node Types         | Index Scan (1) |
+|                    | Seq Scan (1)   |
++--------------------+----------------+
+| Indexes            | film_pkey (1)  |
++--------------------+----------------+
+```
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/sandboxws/awesome_explain. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+
+## Code of Conduct
+
+Everyone interacting in the AwesomeExplain project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/sandboxws/awesome_explain/blob/master/CODE_OF_CONDUCT.md).

data/Rakefile

@@ -1,6 +1,40 @@
 require "bundler/gem_tasks"
 require "rspec/core/rake_task"
+require "wwtd/tasks"
 
 RSpec::Core::RakeTask.new(:spec)
 
-task :default => :spec
+# task :default => :spec
+
+begin
+  require 'bundler/setup'
+rescue LoadError
+  puts 'You must `gem install bundler` and `bundle install` to run rake tasks'
+end
+
+require 'rdoc/task'
+
+RDoc::Task.new(:rdoc) do |rdoc|
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title    = 'AwesomeExplain'
+  rdoc.options << '--line-numbers'
+  rdoc.rdoc_files.include('README.md')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+APP_RAKEFILE = File.expand_path("test/dummy/Rakefile", __dir__)
+# load 'rails/tasks/engine.rake'
+
+load 'rails/tasks/statistics.rake'
+
+require 'bundler/gem_tasks'
+
+require 'rake/testtask'
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << 'test'
+  t.pattern = 'test/**/*_test.rb'
+  t.verbose = false
+end
+
+task default: :test

data/app/models/awesome_explain/application_record.rb

@@ -0,0 +1,5 @@
+module AwesomeExplain
+  class ApplicationRecord < ActiveRecord::Base
+    self.abstract_class = true
+  end
+end

data/app/models/awesome_explain/controller.rb

@@ -0,0 +1,20 @@
+class AwesomeExplain::Controller < ActiveRecord::Base
+  establish_connection AwesomeExplain::Config.instance.db_config
+  self.table_name = 'controllers'
+
+  has_many :logs
+  has_many :explains
+
+  def total_duration
+    logs.sum(:duration).round(3)
+  end
+
+  def gql?
+    path =~ /graphql/ || name =~ /GraphqlController/
+  end
+
+  def formatted_params
+    h_params = JSON.parse(params)
+    gql? ? h_params.dig('graphql') : h_params
+  end
+end

data/app/models/awesome_explain/delayed_job.rb

@@ -0,0 +1,7 @@
+class AwesomeExplain::DelayedJob < ActiveRecord::Base
+  establish_connection AwesomeExplain::Config.instance.db_config
+  self.table_name = 'delayed_jobs'
+
+  has_many :logs
+  has_many :sql_queries
+end

data/app/models/awesome_explain/explain.rb

@@ -0,0 +1,23 @@
+class AwesomeExplain::Explain < ActiveRecord::Base
+  establish_connection AwesomeExplain::Config.instance.db_config
+  self.table_name = 'explains'
+
+  belongs_to :stacktrace
+  before_create :init_collscan
+
+  def to_s
+    "collection: #{collection}, winning_plan: #{winning_plan}, duration: #{duration}, documents_returned: #{documents_returned}, documents_examined: #{documents_examined}"
+  end
+
+  def init_collscan
+    self.collscan = winning_plan_tree.collscan?
+  end
+
+  def treeviz
+    winning_plan_tree.treeviz.to_json
+  end
+
+  def winning_plan_tree
+    @tree ||= AwesomeExplain::PlanTree.build(JSON.parse(winning_plan_raw).with_indifferent_access)
+  end
+end

data/app/models/awesome_explain/log.rb

@@ -0,0 +1,7 @@
+class AwesomeExplain::Log < ActiveRecord::Base
+  establish_connection AwesomeExplain::Config.instance.db_config
+  self.table_name = 'logs'
+
+  belongs_to :stacktrace
+  belongs_to :explain
+end

data/app/models/awesome_explain/pg_dml_stat.rb

@@ -0,0 +1,4 @@
+class AwesomeExplain::PgDmlStat < ActiveRecord::Base
+  establish_connection AwesomeExplain::Config.instance.db_config
+  self.table_name = 'pg_dml_stats'
+end

data/app/models/awesome_explain/pg_seq_scan.rb

@@ -0,0 +1,4 @@
+class AwesomeExplain::PgSeqScan < ActiveRecord::Base
+  establish_connection AwesomeExplain::Config.instance.db_config
+  self.table_name = 'pg_seq_scans'
+end

data/app/models/awesome_explain/plan_node.rb

@@ -0,0 +1,52 @@
+module AwesomeExplain
+  class PlanNode
+    attr_accessor :id,
+      :parent,
+      :children,
+      :label,
+      :documents_returned,
+      :n_returned,
+      :documents_examined,
+      :duration,
+      :keys_examined,
+      :index_name,
+      :treeviz
+
+    def self.build(data, parent = nil)
+      instance = PlanNode.new
+      instance.label = data.dig(:stage)
+      instance.documents_returned = data.dig(:docsReturned)
+      instance.n_returned = data.dig(:nReturned)
+      instance.documents_examined = data.dig(:docsExamined)
+      instance.keys_examined = data.dig(:keysExamined)
+      instance.duration = data.dig(:executionTimeMillisEstimate)
+      instance.index_name = data.dig(:indexName)
+      instance.parent = parent
+      instance.children = []
+      instance
+    end
+
+    def collscan?
+      label.downcase == 'collscan'
+    end
+
+    def treeviz
+      { id: id, text_1: label, text_2: meta_data_str, father: parent&.id, color: '#ffffff' }
+    end
+
+    def meta_data_str
+      meta_data.join('<hr />')
+    end
+
+    def meta_data
+      data = []
+      data << "<strong>Docs Returned:</strong> #{documents_returned}" if documents_returned.present?
+      data << "<strong>N Returned:</strong> #{n_returned}" if n_returned.present?
+      data << "<strong>Docs Examined:</strong> #{documents_examined}" if documents_examined.present?
+      data << "<strong>Keys Examined</strong> #{keys_examined}" if keys_examined.present?
+      data << "<strong>Duration</strong> #{duration}" if duration.present?
+      data << "<strong>Index</strong> #{index_name}" if index_name.present?
+      data
+    end
+  end
+end

data/app/models/awesome_explain/plan_tree.rb

@@ -0,0 +1,66 @@
+module AwesomeExplain
+  class PlanTree
+    attr_accessor :root, :ids, :stages_count, :collscan
+
+    def collscan?
+      collscan
+    end
+
+    def self.build(plan)
+      tree = PlanTree.new
+      tree.ids = (2..500).to_a
+      root = PlanNode.build(plan)
+      root.id = 1
+      tree.stages_count = 1
+      build_recursive(plan.dig('inputStage'), root, tree)
+      tree.root = root
+      tree
+    end
+
+    def self.build_recursive(data, parent, tree)
+      return unless data.present?
+      if data.dig('inputStages').present?
+        # Parent doesn't change
+        data.dig('inputStages').each do |stage|
+          node = PlanNode.build(stage, parent)
+          node.id = tree.ids.shift
+          parent.children << node
+          tree.stages_count += 1
+          tree.collscan = 1 if node.collscan? && !tree.collscan
+          build_recursive(stage.dig('inputStage'), node, tree)
+        end
+      elsif data.dig('inputStage').present?
+        # Parent changes
+        node = PlanNode.build(data, parent)
+        node.id = tree.ids.shift
+        parent.children << node
+        tree.stages_count += 1
+        tree.collscan = 1 if node.collscan? && !tree.collscan
+        build_recursive(data.dig('inputStage'), node, tree)
+      elsif data.dig('inputStage').nil?
+        # Parent doesn't change
+        node = PlanNode.build(data, parent)
+        node.id = tree.ids.shift
+        tree.stages_count += 1
+        tree.collscan = 1 if node.collscan? && !tree.collscan
+        parent.children << node
+      end
+    end
+
+    # Breadth First Traversal
+    def treeviz
+      return unless root.present?
+      output = []
+      queue = [root]
+      while(!queue.empty?) do
+        node = queue.shift
+        output << node.treeviz
+        node.children.each do |child|
+          queue << child
+        end
+      end
+
+      output
+    end
+  end
+end

data/app/models/awesome_explain/sidekiq_worker.rb

@@ -0,0 +1,7 @@
+class AwesomeExplain::SidekiqWorker < ActiveRecord::Base
+  establish_connection AwesomeExplain::Config.instance.db_config
+  self.table_name = 'sidekiq_workers'
+
+  has_many :logs
+  has_many :sql_queries
+end

data/app/models/awesome_explain/sql_explain.rb

@@ -0,0 +1,14 @@
+class AwesomeExplain::SqlExplain < ActiveRecord::Base
+  establish_connection AwesomeExplain::Config.instance.db_config
+  self.table_name = 'sql_explains'
+
+  belongs_to :stacktrace
+
+  def tree
+    @tree ||= ::AwesomeExplain::SqlPlanTree.build(JSON.parse(explain_output))
+  end
+
+  def tree_root
+    tree.root
+  end
+end