shiba 0.1.2 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: f6436b087e61bcce7488d2fbb04003c3fd32fb36
-  data.tar.gz: 348c1fa32744f1879e489fa5530294a4ea70fe4e
+  metadata.gz: 4cd7eeaf515680a21faf4b12533d160db2b86a89
+  data.tar.gz: c44357f6abd8e38ec6c8df9139e81826b39aabcf
 SHA512:
-  metadata.gz: e84dcdbd7cb06d3a2dfa7fa87e9c0bf40e8a3059a8396284c8503599d3d94cbb4c45681b081e8074abc700a7285f64239777cd30c5d1f5eed17f6302756ac98a
-  data.tar.gz: be47f54c87cad50e7260d64468b2c0489f55163a57db59173c3302da9e82a9f62380b239a65eecab6d6db849b044b38d1c0821fdb66457d8497d73f29a6b34f0
+  metadata.gz: 46d865b67f759ed0e694a9c88aaaf4dbbc932c006b2f0ea088a715d55d07d45409d441b0b518ebd10cc7cf9b5eb3dc861437f8473f1eb293c443e54373476f2b
+  data.tar.gz: 6a5bba249b95220a9fbd8bf1f84d4c04a4a8ba74ec500ddc826994f7b2b07c7d23ef67d6e0ad9cbda9aaebc99983eb81eac3a656688bdb612bd4505168e653e4

data/.travis.yml

@@ -0,0 +1,13 @@
+language: ruby
+cache: bundler
+
+rvm:
+ - 2.3
+
+services:
+ - mysql
+
+before_script:
+ - cp .travis/my.cnf ~/.my.cnf
+ - mysql -e 'create database shiba_test;'
+ - mysql shiba_test -e 'source test/structure.sql'

data/.travis/my.cnf

@@ -0,0 +1,3 @@
+[client]
+user = travis
+password =

data/Gemfile.lock

@@ -1,14 +1,27 @@
 PATH
   remote: .
   specs:
-    shiba (0.1.2)
+    shiba (0.2.0)
+      activesupport
 
 GEM
   remote: https://rubygems.org/
   specs:
+    activesupport (5.2.2)
+      concurrent-ruby (~> 1.0, >= 1.0.2)
+      i18n (>= 0.7, < 2)
+      minitest (~> 5.1)
+      tzinfo (~> 1.1)
     byebug (10.0.2)
+    concurrent-ruby (1.1.4)
+    i18n (1.5.3)
+      concurrent-ruby (~> 1.0)
+    minitest (5.11.3)
     mysql2 (0.5.2)
     rake (10.5.0)
+    thread_safe (0.3.6)
+    tzinfo (1.2.5)
+      thread_safe (~> 0.1)
 
 PLATFORMS
   ruby

data/README.md

@@ -1,56 +1,119 @@
+[![Build Status](https://travis-ci.com/burrito-brothers/shiba.svg?branch=master)](https://travis-ci.com/burrito-brothers/shiba)
+
 # Shiba
 
-Shiba is a tool that helps you to understand and write better SQL.  Integrate
-the gem into your test suite, give Shiba a bit of data about your indexes, and Shiba
-will let you know the impact of your queries on production, with the goal of catching
-poorly performing queries before they hit production.
+Shiba is a tool that helps catch poorly performing queries before they cause problems in production, including:
+
+* Full table scans
+* Non selective indexes
+
+By default, it will pretty much only detect queries that miss indexes. As it's fed more information, it warns about advanced problems, such as queries that use indexes but are still very expensive.
+
+To help find such queries, Shiba monitors test runs for ActiveRecord queries. A warning and report are then generated. Shiba is further capable of only warning on changes that occured on a particular git branch/pull request to allow for CI integration.
 
 ## Installation
 
-You can run shiba either as a gem in your test suite, or as a standalone utility.
+Install using bundler. Note: this gem is not designed to be run on production.
 
-### Gem
+```ruby
+gem 'shiba', :group => :test, :require => true
+```
 
-Install
+## Usage
 
 ```ruby
-bundle add shiba --group "development, test"
+# Install
+bundle
+
+# Run some tests using to generate a SQL report
+rake test:functional
+rails test test/controllers/users_controller_test.rb
+
+# 1 problematic query detected
+# Report available at /tmp/shiba-explain.log-1550099512
 ```
 
-Run shiba
+## Typical query problems
+
+Here are some typical query problems Shiba can detect. We'll assume the following schema:
 
 ```ruby
-# Run some some code using shiba to generate a SQL report
-bundle exec shiba analyze rake test:functional
+create_table :users do |t|
+  t.string :name
+  t.string :email
+  # add an organization_id column with an index
+  t.references :organization, index: true
+
+  t.timestamps
+end
+```
 
-# Or run a single test
-bundle exec shiba analyze rails test test/controllers/users_controller_test.rb
+#### Full table scans
+
+The most simple case to detect are queries that don't utilize indexes. While it isn't a problem to scan small tables, often tables will grow large enough where this can become a serious issue.
+
+```ruby
+user = User.where(email: 'squirrel@example.com').limit(1)
 ```
 
-### Standalone:
+Without an index, the database will read every row in the table until it finds one with an email address that matches. By adding an index, the database can perform a quick lookup for the record.
+
+#### Non selective indexes
+
+Another common case is queries that use an index, and work fine in the average case, but the distribution is non normal. These issues can be hard to track down and often impact large customers.
 
+```ruby
+users = User.where(organization_id: 1)
+users.size
+# => 75
+
+users = User.where(organization_id: 42)
+users.size
+# => 52,000
 ```
-# 1.  Get shiba.
-local:$ git clone git@github.com:burrito-brothers/shiba.git
 
-# 2.  Get production data.
-# Shiba *can* work without any further data, but it's really best if you can
-# dump index statistics from a production database, or a staging database with
-# that resembles production.
+Normally a query like this would only become a problem as the app grows in popularity. Fixes include adding `limit` or `find_each`.
 
-local:$ ssh production_host
-production_host:$ mysql -ABe "select * from information_schema.statistics where table_schema = 'DATABASE'" > shiba_schema_stats.tsv
-local:$ scp production_host:shiba_schema_stats.tsv shiba/
+With more data, Shiba can help detect this issue when it appears in a pull request.
 
-# 3.  Analyze your queries.
-# set shiba loose on your queries!
-# If you can't do step #2, just leave off the '-s' option
+## Going beyond table scans
 
-local:$ cd shiba
-local:$ bin/analyze.rb -h 127.0.0.1 -d TESTDB -u MYSQLUSER -p MYSQLPASS -s shiba_schema_stats.tsv -f ~/src/MYPROJECT/log/test.log > results.json
+For smarter analysis, Shiba requires general statistics about production data, such as the number of rows in a table and how unique columns are.
 
-# analyze the results with `jq`, whynot
+This information can be obtained by running the bin/dump_stats command in production.
 
-local:$ jq -C -s 'sort_by(.cost) | reverse' results.json | less -R
+```console
+production$ 
+git clone https://github.com/burrito-brothers/shiba.git
+cd shiba ; bundle
+bin/dump_stats DATABASE_NAME [MYSQLOPTS] > ~/shiba_index.yml
+
+local$
+scp production:~/shiba_index.yml RAILS_PROJECT/config
+```
 
+The stats file will look similar to the following:
+
+```yaml
+users:
+  count: 10000
+  indexes:
+    PRIMARY:
+      name: PRIMARY
+      columns:
+      - column: id
+        rows_per: 1
+      unique: true
+    index_users_on_login:
+      name: index_users_on_login
+      columns:
+      - column: login
+        rows_per: 1
+      unique: true
+    index_users_on_created_by_id:
+      name: index_users_on_created_by_id
+      columns:
+      - column: created_by_id
+        rows_per: 3
+      unique: false
 ```

data/Rakefile

@@ -1,2 +1,10 @@
 require "bundler/gem_tasks"
-task :default => :spec
+require 'rake/testtask'
+
+Rake::TestTask.new do |t|
+    t.libs << "test"
+    t.test_files = FileList['test/*_test.rb']
+    t.verbose = true
+  end
+
+task :default => :test

data/TODO

@@ -1,12 +1,30 @@
 ===
-this query is throwing the optimizer for a serious loop;
-it says it can use an index on `ipb_address` but when we force key it
-still table scans.  Not clear on whether the OR in there is fucking us over or
-if it's a test-data issue.
 
+# we screw this up because of the table aliasing
+SELECT COUNT(*) AS count_all, group_id AS group_id FROM `users` INNER JOIN `groups_users` `users_groups_users_join` ON `users_groups_users_join`.`user_id` = `users`.`id` INNER JOIN `users` `groups_users` ON `groups_users`.`id` = `users_groups_users_join`.`group_id` AND `groups_users`.`type` IN ('Group', 'GroupBuiltin', 'GroupAnonymous', 'GroupNonMember') WHERE `users`.`type` IN ('User', 'AnonymousUser') GROUP BY group_id
+
+
+# UI
+
+SQL Query -> where clause
+throw tags in
+
+# Add "ignore me" button next to query to show how to block stuff.
+
+# Fuzzer
+
+see how feasible a basic-ass cardinality estimator would be
+
+# Docs
+
+- main docs (readme.md)
+- link from call to acion about fuzzed data
+
+# Explain Stuff
+
+Add index-walk detection
+Add filesort badge
+Postgres!
 
-# SELECT  ipb_id,ipb_address,ipb_timestamp,ipb_auto,ipb_anon_only,ipb_create_account,ipb_enable_autoblock,ipb_expiry,ipb_deleted,ipb_block_email,ipb_allow_usertalk,ipb_parent_block_id,ipb_sitewide,comment_ipb_reason.comment_text AS `ipb_reason_text`,comment_ipb_reason.comment_data AS `ipb_reason_data`,comment_ipb_reason.comment_id AS `ipb_reason_cid`,ipb_by,ipb_by_text,NULL AS `ipb_by_actor`  FROM `ipblocks` FORCE KEY(`ipb_address`) JOIN `comment` `comment_ipb_reason` ON ((comment_ipb_reason.comment_id = ipb_reason_id))   WHERE ipb_address = '127.0.0.1' OR ((ipb_range_start  LIKE '7F00%' ESCAPE '`' ) AND (ipb_range_start <= '7F000001') AND (ipb_range_end >= '7F000001'));
 
-===
-need to use format=json and see how much of an index we're using.
 

data/bin/dump_stats

@@ -0,0 +1,38 @@
+#!/bin/bash
+
+case "$1" in
+  -t|--tables)
+    TABLES=$(echo "$2" | tr -d ' ' | tr ',' '\n')
+    shift 2
+    ;;
+esac
+
+DATABASE=$1
+if [ -z "$DATABASE" ]
+then
+  echo "usage: dump_stats -tables [table1,table2...] DATABASE [ ...mysql_args]"
+  exit 1
+fi
+
+shift
+
+#for x in `mysql $* -NABe "show tables from $DATABASE"` ; do
+#  mysql $* -e "ANALYZE TABLE $DATABASE.$x" >/dev/null 2>&1
+#done
+
+MYSQL_STATS=`mktemp`
+mysql $* -ABe "select * from information_schema.statistics where table_schema = '$DATABASE'" > $MYSQL_STATS
+
+if [ "$TABLES" ]
+then
+  filtered=`mktemp`
+  head -n 1 $MYSQL_STATS >> $filtered
+  for t in $TABLES
+  do
+    awk -v tbl=$t '{ if ($3 == tbl) print $0 }' $MYSQL_STATS >> $filtered
+  done
+
+  MYSQL_STATS=$filtered
+fi
+
+bundle exec ruby -e "require 'shiba/index'; puts Shiba::Index.parse('$MYSQL_STATS').to_yaml"

data/bin/explain

@@ -3,7 +3,7 @@
 require 'bundler/setup'
 require 'shiba'
 require 'shiba/analyzer'
-require 'shiba/index'
+require 'shiba/table_stats'
 require 'shiba/configure'
 require 'shiba/output'
 
@@ -15,20 +15,51 @@ parser = Shiba::Configure.make_options_parser(options)
 parser.banner = "Run a list of queries through shiba's analyzer."
 parser.parse!
 
-["database", "username"].each do |opt|
-  if !options[opt]
-    $stderr.puts "Required: #{opt}"
-    $stderr.puts parser.banner
-    exit 1
+option_path = Shiba::Configure.mysql_config_path
+
+if option_path
+  puts "Found config at #{option_path}" if options["verbose"]
+  options['default_file'] ||= option_path
+end
+
+option_file = if options['default_file'] && File.exist?(options['default_file'])
+   File.read(options['default_file'])
+else
+  ""
+end
+
+if options['json'] && options['html']
+  $stderr.puts "Can only output to json or html, not both"
+  $stderr.puts parser.banner
+  exit 1
+end
+
+if option_file && !options['default_group']
+  if option_file.include?("[client]")
+    options['default_group'] = 'client'
   end
+  if option_file.include?("[mysql]")
+    options['default_group'] = 'mysql'
+  end
+end
+
+def require_option(parser, name)
+  $stderr.puts "Required: #{name}"
+  $stderr.puts parser.banner
+  exit 1
+end
+
+if !options["username"] && !option_file.include?('user')
+  require_option(parser, 'username')
+end
+
+if !options["database"] && !option_file.include?('database')
+  require_option(parser, 'database')
 end
 
 file = options.delete("file")
 file = File.open(file, "r") if file
 
-output = options.delete("explain")
-output = File.open(output, 'w') if output
-
 Shiba.configure(options)
 
 schema_stats_fname = options["stats"]
@@ -38,27 +69,35 @@ if schema_stats_fname && !File.exist?(schema_stats_fname)
   exit 1
 end
 
-if schema_stats_fname
-  schema_stats = Shiba::Index.parse(schema_stats_fname)
+file = $stdin if file.nil?
+json = options['json']
+json = File.open('/dev/null', 'w') if json.nil?
+
+if options["verbose"]
+  $stderr.puts "Reading queries from '#{file.inspect}'..."
+  $stderr.puts "Analyzing SQL to '#{json.inspect}'..."
+end
 
-  local_db_stats = Shiba::Index.query(Shiba.connection)
-  Shiba::Index.fuzz!(local_db_stats)
-  local_db_stats.each do |table, values|
-    schema_stats[table] = values unless schema_stats[table]
-  end
-else
-  schema_stats = Shiba::Index.query(Shiba.connection)
+table_stats = Shiba::TableStats.new(Shiba.index_config, Shiba.connection, {})
+queries = Shiba::Analyzer.analyze(file, json, table_stats, options)
 
-  if Shiba::Index.insufficient_stats?(schema_stats)
-    $stderr.puts "WARN: insufficient stats available in the #{options["database"]} database, guessing at numbers."
-    $stderr.puts "To get better analysis please specify an index statistics file."
-    sleep 0.5
-    Shiba::Index.fuzz!(schema_stats)
+problems = queries.select { |q| q[:cost] && q[:cost] > 100 }
+
+if problems.any?
+  query_word = problems.size == 1 ? 'query' : 'queries'
+  $stderr.puts "#{problems.size} problematic #{query_word} detected"
+
+  if options['json']
+    exit 3
   end
-end
 
-file = $stdin if file.nil?
-output = $stdout if output.nil?
+  page = Shiba::Output.new(queries, { 'output' => options['html'] }).make_web!
 
-queries = Shiba::Analyzer.analyze(file, output, schema_stats, options)
-Shiba::Output.new(queries, { 'output' => options['output'] }).make_web!
+  if !File.exist?(page)
+    $stderr.puts("Failed to generate #{page}")
+    exit 1
+  end
+
+  $stderr.puts "Report available at #{page}"
+  exit 3
+end

data/bin/shiba

@@ -5,8 +5,8 @@ require 'optionparser'
 APP = File.basename(__FILE__)
 
 commands = {
-    "analyze" => { :bin => "analyze", :description => "Analyze logged SQL queries" },
-    "watch"   => { :bin => "watch.rb", :description => "Log queries from Rails command" }
+    "explain" => "Generate a report from logged SQL queries",
+    "check"   => "Check staged files for query problems",
 }
 
 global = OptionParser.new do |opts|
@@ -14,7 +14,7 @@ global = OptionParser.new do |opts|
   opts.separator ""
   opts.separator "These are the commands available:"
   commands.each do |name,info|
-    opts.separator "  #{name}     #{info[:description]}"
+    opts.separator "  #{name}     #{info}"
   end
   opts.separator ""
   opts.separator "See #{APP} --help <command> to read about a specific command."
@@ -35,6 +35,6 @@ if !commands.key?(command)
     exit 1
 end
 
-path = "#{File.dirname(__FILE__)}/#{commands[command][:bin]}"
+path = File.join(File.dirname(__FILE__), command)
 
 Kernel.exec(path, *ARGV)