lookout-query_reviewer 0.1.5

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2007 [name of plugin creator]
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,136 @@
+# QueryReviewer #
+
+## Introduction ##
+
+QueryReviewer is an advanced SQL query analyzer.  It accomplishes the following goals:
+
+ * View all EXPLAIN output for all SELECT queries to generate a page
+ * Rate a page's SQL usage into one of three categories: OK, WARNING, CRITICAL
+ * Attach meaningful warnings to individual queries, and collections of queries
+ * Display interactive summary on page
+
+## Use This Fork ##
+
+I use this utility for most of my MySQL-backed Rails projects. Still the best out there in my opinion for analyzing and understanding your generated SQL queries. 
+I forked the original [query_reviewer](https://github.com/dsboulder/query_reviewer) and applied a collection of patches that have been made since the plugin was created. 
+A list of the biggest additions below:
+
+ * Snazzed up the README into markdown for better readability
+ * Full compatibility for Rails 3 (including Railtie)
+ * Cleanup and move rake task to `lib/tasks` to fix deprecation warnings
+ * Added gemspec for use with Bundler (as a gem)
+ * Fixed missing tags and additional XHTML escaping
+ * Fix SQL escaping for better XHTML compatibility
+ * Fixes for deprecation warnings and for 1.9 compatiblity
+ * Converts templates to more modern foo.html.erb naming
+
+As of August 2011, This has become the "official" version to use. David has deprecated the [original repo](https://github.com/dsboulder/query_reviewer) and recommends that people looking for this utility use this fork instead. 
+This fork compiles a variety of patches that were made since that time along with additional work to support compatibility with 1.9 and Rails 3. 
+**Also:** If anyone else creates generally useful enhancements to this utility please start by forking this and then issue me a pull request.
+
+**Note:** This plugin should work for Rails 2.X and Rails 3. If you run into issues, please file a bug report or send a pull request. I consider this new fork to be a group effort
+so if you want to be added as a contributor to help maintain this, please let me know!
+
+## Database Limitations ##
+
+As of right now, this plugin  supports analyzing queries **only for MySQL**. I consider this the primary **limitation** of this gem. 
+I am now using PostgreSQL for a number of my projects and the "EXPLAIN" output is totally different. Several people have started a PostgreSQL adapter for
+this plugin, but none that are mature enough to include here.
+
+If anyone is interested at taking a stab at implementing PostgreSQL support for this utility, that would be hugely appreciated! I hope this plugin can come to support
+all the major Rails adapters over time.
+
+## Installation ##
+
+All you have to do is install it into your Rails 2 or 3 project.
+
+    gem install query_reviewer
+
+Right now if you use bundler, simply add this to your Gemfile:
+
+    # Gemfile
+    gem "query_reviewer", :git => "git://github.com/nesquena/query_reviewer.git"
+
+If you are not using bundler, you might want to [start using it](http://gembundler.com/rails23.html). You can also install this as a plugin:
+
+    script/plugin install git://github.com/nesquena/query_reviewer.git
+
+In Rails 2, the rake tasks are not loaded automatically (as a gem), you’ll need to add the following to your Rakefile:
+
+    # Rakefile
+    begin
+      require 'query_reviewer/tasks'
+    rescue LoadError
+      STDERR.puts "The query_reviewer gem could not be found!"
+    end
+
+You can then run:
+
+    $ rake query_reviewer:setup
+
+Which will create `config/query_reviewer.yml` in your application, see below for what these options mean.
+If you don't create a config file, the gem will use the default in `vendor/plugins/query_reviewer`.
+
+## Configuration ##
+
+The configuration file allows you to set configuration parameters shared across all rails environment, as well as overriding those shared parameteres with environment-specific parameters (such as disabling analysis on production!)
+
+ * `enabled`: whether any output or query analysis is performed.  Set this false in production!
+ * `inject_view`: controls whether the output automatically is injected before the </body> in HTML output.
+ * `profiling`: when enabled, runs the MySQL SET PROFILING=1 for queries longer than the `warn_duration_threshold` / 2.0
+ * `production_data`: whether the duration of a query should be taken into account
+ * `stack_trace_lines`: number of lines of call stack to include in the "short" version of the stack trace
+ * `trace_includes_vendor`: whether the "short" version of the stack trace should include files in /vendor
+ * `trace_includes_lib`: whether the "short" version of the stack trace should include files in /lib
+ * `warn_severity`: the severity of problem that merits "WARNING" status
+ * `critical_severity`: the severity of problem that merits "CRITICAL" status
+ * `warn_query_count`: the number of queries in a single request that merits "WARNING" status
+ * `critical_query_count`: the number of queries in a single request that merits "CRITICAL" status
+ * `warn_duration_threshold`: how long a query must take in seconds (float) before it's considered "WARNING"
+ * `critical_duration_threshold`: how long a query must take in seconds (float) before it's considered "CRITICIAL"
+
+## Example ##
+
+If you disable the inject_view option above, you'll need to manually put the analyzer's output into your view:
+
+    # view.html.haml
+    = query_review_output
+
+and that will display the analyzer view!
+
+## Resources ##
+
+Random collection of resources that might be interesting related to this utility:
+
+ * <http://blog.purifyapp.com/2010/06/15/optimise-your-mysql/>
+ * <http://www.tatvartha.com/2009/09/rails-optimizing-database-indexes-using-query_analyzer-and-query_reviewer/>
+ * <http://www.geekskillz.com/articles/using-indexes-to-improve-rails-performance>
+ * <http://www.williambharding.com/blog/rails/rails-mysql-indexes-step-1-in-pitiful-to-prime-performance/>
+ * <http://guides.rubyonrails.org/performance_testing.html>
+
+Other related gems that prove useful for database optimization:
+
+ * [bullet](https://github.com/flyerhzm/bullet)
+ * [slim-scrooge](https://github.com/sdsykes/slim_scrooge)
+ * [slim-attributes](https://github.com/sdsykes/slim-attributes)
+
+## Alternatives ##
+
+There have been other alternatives created since this was originally released. A few of the best are listed below. I for one still prefer this utility over the other options:
+
+ * [rack-bug](https://github.com/brynary/rack-bug)
+ * [rails-footnotes](https://github.com/josevalim/rails-footnotes)
+ * [newrelic-development](http://support.newrelic.com/kb/docs/developer-mode)
+ * [palmist](https://github.com/flyingmachine/palmist)
+ * [query_diet](https://github.com/makandra/query_diet)
+ * [query_trace](https://github.com/ntalbott/query_trace)
+
+Know of a better alternative? Let me know!
+
+## Acknowledgements ##
+
+Created by Kongregate & David Stevenson.
+Refactorings and compilations of all fixes since was done by Nathan Esquenazi.
+Also, ajvargo for helping with some fixes.
+
+Copyright (c) 2007-2008 Kongregate & David Stevenson, released under the MIT license

data/Rakefile

@@ -0,0 +1,24 @@
+require 'rake'
+require 'rake/testtask'
+require 'rake/rdoctask'
+require 'bundler'
+Bundler::GemHelper.install_tasks
+
+desc 'Default: run unit tests.'
+task :default => :test
+
+desc 'Test the query_reviewer plugin.'
+Rake::TestTask.new(:test) do |t|
+  t.libs << 'lib'
+  t.pattern = 'test/**/*_test.rb'
+  t.verbose = true
+end
+
+desc 'Generate documentation for the query_reviewer plugin.'
+Rake::RDocTask.new(:rdoc) do |rdoc|
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title    = 'QueryReviewer'
+  rdoc.options << '--line-numbers' << '--inline-source'
+  rdoc.rdoc_files.include('README')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end

data/lib/query_reviewer.rb

@@ -0,0 +1,66 @@
+# QueryReviewer
+require "ostruct"
+require 'erb'
+require 'yaml'
+
+module QueryReviewer
+  CONFIGURATION = {}
+
+  def self.load_configuration
+    default_config = YAML::load(ERB.new(IO.read(File.join(File.dirname(__FILE__), "..", "query_reviewer_defaults.yml"))).result)
+
+    CONFIGURATION.merge!(default_config["all"] || {})
+    CONFIGURATION.merge!(default_config[Rails.env || "test"] || {})
+
+    app_config_file = Rails.root + "config/query_reviewer.yml"
+
+    if app_config_file.exist?
+      app_config = YAML.load(ERB.new(IO.read(app_config_file)).result)
+      CONFIGURATION.merge!(app_config["all"] || {})
+      CONFIGURATION.merge!(app_config[Rails.env || "test"] || {})
+    end
+
+    if enabled?
+      begin
+        CONFIGURATION["uv"] ||= if Gem::Specification.respond_to?(:find_all_by_name)
+          !Gem::Specification.find_all_by_name('uv').empty?
+        else # RubyGems < 1.8.0
+          !Gem.searcher.find("uv").nil?
+        end
+
+        if CONFIGURATION["uv"]
+          require "uv"
+        end
+      rescue
+        CONFIGURATION["uv"] ||= false
+      end
+
+      require "query_reviewer/query_warning"
+      require "query_reviewer/array_extensions"
+      require "query_reviewer/sql_query"
+      require "query_reviewer/mysql_analyzer"
+      require "query_reviewer/sql_sub_query"
+      require "query_reviewer/mysql_adapter_extensions"
+      require "query_reviewer/controller_extensions"
+      require "query_reviewer/sql_query_collection"
+    end
+  end
+
+  def self.enabled?
+    CONFIGURATION["enabled"]
+  end
+
+  def self.safe_log(&block)
+    if @logger.nil?
+      yield
+    elsif @logger.respond_to?(:quietly)
+      @logger.quietly { yield }
+    elsif @logger.respond_to?(:silence)
+      @logger.silence { yield }
+    end
+  end
+
+end
+
+# Rails Integration
+require 'query_reviewer/rails' if defined?(Rails)

data/lib/query_reviewer/array_extensions.rb

@@ -0,0 +1,29 @@
+module QueryReviewer
+  module ArrayExtensions  #taken from query_analyser plugin
+    protected
+      def qa_columnized_row(fields, sized)
+        row = []
+        fields.each_with_index do |f, i|
+          row << sprintf("%0-#{sized[i]}s", f.to_s)
+        end
+        row.join(' | ')
+      end
+
+    public
+
+    def qa_columnized
+      sized = {}
+      self.each do |row|
+        row.values.each_with_index do |value, i|
+          sized[i] = [sized[i].to_i, row.keys[i].length, value.to_s.length].max
+        end
+      end
+
+      table = []
+      table << qa_columnized_row(self.first.keys, sized)
+      table << '-' * table.first.length
+      self.each { |row| table << qa_columnized_row(row.values, sized) }
+      table.join("\n   ") # Spaces added to work with format_log_entry
+    end
+  end  
+end

data/lib/query_reviewer/controller_extensions.rb

@@ -0,0 +1,67 @@
+require "action_view"
+require File.join(File.dirname(__FILE__), "views", "query_review_box_helper")
+
+module QueryReviewer
+  module ControllerExtensions
+    class QueryViewBase < ActionView::Base
+      include QueryReviewer::Views::QueryReviewBoxHelper
+    end
+
+    def self.included(base)
+      if QueryReviewer::CONFIGURATION["inject_view"]
+        alias_name = defined?(Rails::Railtie) ? :process_action : :perform_action
+        base.alias_method_chain(alias_name, :query_review)
+      end
+      base.alias_method_chain :process, :query_review
+      base.helper_method :query_review_output
+    end
+
+    def query_review_output(ajax = false, total_time = nil)
+      faux_view = QueryViewBase.new([File.join(File.dirname(__FILE__), "views")], {}, self)
+      queries = Thread.current["queries"]
+      queries.analyze!
+      faux_view.instance_variable_set("@queries", queries)
+      faux_view.instance_variable_set("@total_time", total_time)
+      if ajax
+        js = faux_view.render(:partial => "/box_ajax.js")
+      else
+        html = faux_view.render(:partial => "/box")
+      end
+    end
+
+    def add_query_output_to_view(total_time)
+      if request.xhr?
+        if cookies["query_review_enabled"]
+          if !response.content_type || response.content_type.include?("text/html")
+            response.body += "<script type=\"text/javascript\">"+query_review_output(true, total_time)+"</script>"
+          elsif response.content_type && response.content_type.include?("text/javascript")
+            response.body += ";\n"+query_review_output(true, total_time)
+          end
+        end
+      else
+        if response.body.is_a?(String) && response.body.match(/<\/body>/i)
+          idx = (response.body =~ /<\/body>/i)
+          html = query_review_output(false, total_time)
+          response.body = response.body.insert(idx, html)
+        end
+      end
+    end
+
+    def perform_action_with_query_review(*args)
+      return unless (CONFIGURATION["enabled"] == true               and cookies["query_review_enabled"]) ||
+                    (CONFIGURATION["enabled"] == "based_on_session" and session["query_review_enabled"])
+
+      t1 = Time.now
+      r = defined?(Rails::Railtie) ? process_action_without_query_review(*args) : perform_action_without_query_review(*args)
+      t2 = Time.now
+      add_query_output_to_view(t2 - t1)
+      r
+    end
+    alias_method :process_action_with_query_review, :perform_action_with_query_review
+
+    def process_with_query_review(*args) #:nodoc:
+      Thread.current["queries"] = SqlQueryCollection.new
+      process_without_query_review(*args)
+    end
+  end
+end

data/lib/query_reviewer/mysql_adapter_extensions.rb

@@ -0,0 +1,92 @@
+module QueryReviewer
+  module MysqlAdapterExtensions
+    def self.included(base)
+      base.alias_method_chain :select, :review
+      base.alias_method_chain :update, :review
+      base.alias_method_chain :insert, :review
+      base.alias_method_chain :delete, :review
+    end
+
+    def update_with_review(sql, *args)
+      t1 = Time.now
+      result = update_without_review(sql, *args)
+      t2 = Time.now
+
+      create_or_add_query_to_query_reviewer!(sql, nil, t2 - t1, nil, "UPDATE", result)
+
+      result
+    end
+
+    def insert_with_review(sql, *args)
+      t1 = Time.now
+      result = insert_without_review(sql, *args)
+      t2 = Time.now
+
+      create_or_add_query_to_query_reviewer!(sql, nil, t2 - t1, nil, "INSERT")
+
+      result
+    end
+
+    def delete_with_review(sql, *args)
+      t1 = Time.now
+      result = delete_without_review(sql, *args)
+      t2 = Time.now
+
+      create_or_add_query_to_query_reviewer!(sql, nil, t2 - t1, nil, "DELETE", result)
+
+      result
+    end
+
+    def select_with_review(sql, *args)
+      return select_without_review(sql, *args) unless query_reviewer_enabled?
+
+      sql.gsub!(/^SELECT /i, "SELECT SQL_NO_CACHE ") if QueryReviewer::CONFIGURATION["disable_sql_cache"]
+      QueryReviewer.safe_log { execute("SET PROFILING=1") } if QueryReviewer::CONFIGURATION["profiling"]
+      t1 = Time.now
+      query_results = select_without_review(sql, *args)
+      t2 = Time.now
+
+      if @logger && sql =~ /^select/i && query_reviewer_enabled?
+        use_profiling = QueryReviewer::CONFIGURATION["profiling"]
+        use_profiling &&= (t2 - t1) >= QueryReviewer::CONFIGURATION["warn_duration_threshold"].to_f / 2.0 if QueryReviewer::CONFIGURATION["production_data"]
+
+        if use_profiling
+          t5 = Time.now
+          QueryReviewer.safe_log { execute("SET PROFILING=1") }
+          t3 = Time.now
+          select_without_review(sql, *args)
+          t4 = Time.now
+          profile = QueryReviewer.safe_log { select_without_review("SHOW PROFILE ALL", *args) }
+          QueryReviewer.safe_log { execute("SET PROFILING=0") }
+          t6 = Time.now
+          Thread.current["queries"].overhead_time += t6 - t5
+        else
+          profile = nil
+        end
+
+        cols = QueryReviewer.safe_log do
+          select_without_review("explain #{sql}", *args)
+        end
+
+        duration = t3 ? [t2 - t1, t4 - t3].min : t2 - t1
+        create_or_add_query_to_query_reviewer!(sql, cols, duration, profile)
+
+        #@logger.debug(format_log_entry("Analyzing #{name}\n", query.to_table)) if @logger.level <= Logger::INFO
+      end
+      query_results
+    end
+
+    def query_reviewer_enabled?
+      Thread.current["queries"] && Thread.current["queries"].respond_to?(:find_or_create_sql_query) && Thread.current["query_reviewer_enabled"]
+    end
+
+    def create_or_add_query_to_query_reviewer!(sql, cols, run_time, profile, command = "SELECT", affected_rows = 1)
+      if query_reviewer_enabled?
+        t1 = Time.now
+        Thread.current["queries"].find_or_create_sql_query(sql, cols, run_time, profile, command, affected_rows)
+        t2 = Time.now
+        Thread.current["queries"].overhead_time += t2 - t1
+      end
+    end
+  end
+end

data/lib/query_reviewer/mysql_analyzer.rb

@@ -0,0 +1,62 @@
+module QueryReviewer
+  module MysqlAnalyzer
+    def do_mysql_analysis!
+      analyze_select_type!
+      analyze_query_type!
+      analyze_key!
+      analyze_extras!
+      analyze_keylen!
+    end
+
+    def analyze_select_type!
+      if select_type.match /uncacheable subquery/
+        warn(:severity => 10, :field => "select_type", :desc => "Subquery must be run once for EVERY row in main query")
+      elsif select_type.match /dependent/
+        warn(:severity => 2, :field => "select_type", :desc => "Dependent subqueries can not be executed while the main query is running")
+      end
+    end
+
+    def analyze_query_type!
+      case query_type
+      when "system", "const", "eq_ref" then
+        praise("Yay")
+      when "ref", "ref_or_null", "range", "index_merge" then
+        praise("Not bad eh...")
+      when "unique_subquery", "index_subquery" then
+        #NOT SURE
+      when "index" then
+        warn(:severity => 8, :field => "query_type", :desc => "Full index tree scan (slightly faster than a full table scan)") unless !extra.include?("using where")
+      when "all" then
+        warn(:severity => 9, :field => "query_type", :desc => "Full table scan") unless !extra.include?("using where")
+      end
+    end
+
+    def analyze_key!
+      if self.key == "const"
+        praise "Way to go!"
+      elsif self.key.blank? && !self.extra.include?("select tables optimized away")
+        warn :severity => 6, :field => "key", :desc => "No index was used here. In this case, that meant scanning #{self.rows} rows."
+      end
+    end
+
+    def analyze_extras!
+      if self.extra.match(/range checked for each record/)
+        warn :severity => 4, :problem => "Range checked for each record", :desc => "MySQL found no good index to use, but found that some of indexes might be used after column values from preceding tables are known"
+      end
+
+      if self.extra.match(/using filesort/)
+        warn :severity => 2, :problem => "Using filesort", :desc => "MySQL must do an extra pass to find out how to retrieve the rows in sorted order."
+      end
+
+      if self.extra.match(/using temporary/)
+        warn :severity => 10, :problem => "Using temporary table", :desc => "To resolve the query, MySQL needs to create a temporary table to hold the result."
+      end
+    end
+
+    def analyze_keylen!
+      if self.key_len && !self.key_len.to_i.nil? && (self.key_len.to_i > QueryReviewer::CONFIGURATION["max_safe_key_length"])
+        warn :severity => 4, :problem => "Long key length (#{self.key_len.to_i})", :desc => "The key used for the index was rather long, potentially affecting indices in memory"
+      end
+    end
+  end
+end