mctop 0.0.3

data/.gitignore

@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in mctop.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2012 Etsy
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,68 @@
+# mctop
+
+Inspired by "top", mctop passively sniffs the network traffic passing in and out of a
+server's network interface and tracks the keys responding to memcache get commands. The output
+is presented on the terminal and allows sorting by total calls, requests/sec and
+bandwidth.
+
+You can read more detail about why this tool evovled over on our
+[code as craft](http://codeascraft.etsy.com/2012/12/13/mctop-a-tool-for-analyzing-memcache-get-traffic) blog.
+
+mctop depends on the [ruby-pcap](https://rubygems.org/gems/ruby-pcap) gem, if you don't have 
+this installed you'll need to ensure you have the development pcap libraries (libpcap-devel 
+package on most linux distros) to build the native gem.
+
+## How it works
+
+mctop sniffs network traffic collecting memcache `VALUE` responses and calculates from
+traffic statistics for each key seen.  It currently reports on the following metrics per key:
+
+* **calls** - the number of times the key has been called since mctop started 
+* **objsize** - the size of the object stored for that key
+* **req/sec** - the number of requests per second for the key
+* **bw (kbps)** - the estimated netowrk bandwidth consumed by this key in kilobits-per-second
+
+## Getting it running
+
+the quickest way to get it running is to:
+
+* ensure you have libpcap-devel installed
+* git clone this repo
+* in the top level directory of this repo `bundle install` (this will install the deps)
+* then either:
+    * install it locally `rake install`; or
+    * run it from the repo (good for hacking) `sudo ./bin/mctop --help`
+
+## Command line options
+
+    Usage: mctop [options]
+        -i, --interface=NIC              Network interface to sniff (required)
+        -d, --discard=THRESH             Discard keys with request/sec rate below THRESH
+        -r, --refresh=MS                 Refresh the stats display every MS milliseconds
+        -h, --help                       Show usage info
+
+## User interface commands
+
+The following key commands are available in the console UI:
+
+* `C` - sort by number of calls
+* `S` - sort by object size
+* `R` - sort by requests/sec
+* `B` - sort by bandwidth
+* `T` - toggle sorting by ascending / descending order
+* `Q` - quits
+
+## Status bar
+
+The following details are displayed in the status bar
+
+* `sort mode` - the current sort mode and ordering
+* `keys` - total number of keys in the metrics table
+* `packets` - packets received and dropped by libpcap (% is percentage of packets dropped)
+* `rt` - the time taken to sort and render the stats
+
+## Known issues / Gotchas
+
+### ruby-pcap drops packets at high volume
+from my testing the ruby-pcap native interface to libpcap struggles to keep up with high packet rates (in what we see on a production memcache instance) you can keep an eye on the packets recv/drop and loss percentage on the status bar at the bottom of the UI to get an idea of the packet 
+

data/Rakefile

@@ -0,0 +1 @@
+require "bundler/gem_tasks"

data/bin/mctop

@@ -0,0 +1,71 @@
+#!/usr/bin/env ruby
+
+# mctop - A command line memcached traffic analyzer
+# 
+# Author:: Marcus Barczak (<marcus@etsy.com>)
+
+$:.unshift File.join(File.dirname(__FILE__),'..','lib')
+
+require 'cmdline'
+require 'sniffer'
+require 'ui'
+
+@config = CmdLine.parse(ARGV)
+
+# instantiate a sniffer and user interface object
+sniffer = MemcacheSniffer.new(@config)
+ui      = UI.new(@config)
+
+# set default display options
+sort_mode  = :reqsec
+sort_order = :desc
+done       = false
+
+# trap most of the typical signals
+%w[ INT QUIT HUP KILL ].each do |sig|
+    Signal.trap(sig) do
+        puts "** Caught signal #{sig} - exiting"
+        done = true
+    end
+end
+
+# kick the sniffer thread off
+sniff_thread = Thread.new { sniffer.start }
+
+# main loop
+until done do
+    ui.header
+    ui.footer
+    ui.render_stats(sniffer, sort_mode, sort_order)
+    refresh
+
+    key = ui.input_handler
+    case key
+        when /[Qq]/
+            done = true
+        when /[Cc]/
+            sort_mode = :calls
+        when /[Ss]/
+            sort_mode = :objsize
+        when /[Rr]/
+            sort_mode = :reqsec
+        when /[Bb]/
+            sort_mode = :bw
+        when /[Tt]/
+            if sort_order == :desc
+                sort_order = :asc
+            else
+                sort_order = :desc
+            end
+    end
+end
+
+## cleanup
+ui.done
+sniffer.done
+
+## if sniffer thread doesn't join immediately kill it off the
+## capture.each loop blocks if no packets have been seen
+if sniff_thread.join(0)
+    sniff_thread.kill
+end

data/lib/cmdline.rb

@@ -0,0 +1,51 @@
+require 'optparse'
+require 'pcap'
+
+class CmdLine
+    def self.parse(args)
+        @config = {}
+
+        opts = OptionParser.new do |opt|
+            opt.on('-i', '--interface=NIC', 'Network interface to sniff (required)') do |nic|
+                @config[:nic] = nic
+            end
+
+            opt.on '-d', '--discard=THRESH', Float, 'Discard keys with request/sec rate below THRESH' do |discard_thresh|
+                @config[:discard_thresh] = discard_thresh
+            end
+
+            opt.on '-r', '--refresh=MS', Float, 'Refresh the stats display every MS milliseconds' do |refresh_rate|
+                @config[:refresh_rate] = refresh_rate
+            end
+
+            opt.on_tail '-h', '--help', 'Show usage info' do 
+                puts opts
+                exit
+            end
+        end
+
+        opts.parse!
+
+        # bail if we're not root
+        unless Process::Sys.getuid == 0
+            puts "** ERROR: needs to run as root to capture packets"
+            exit 1
+        end
+
+        # we need need a nic to listen on
+        unless @config.has_key?(:nic)
+            puts "** ERROR: You must specify a network interface to listen on"
+            puts opts
+            exit 1
+        end
+
+        # we can't do 'any' interface just yet due to weirdness with ruby pcap libs
+        if @config[:nic] =~ /any/i
+            puts "** ERROR: can't bind to any interface due to odd issues with ruby-pcap"
+            puts opts
+            exit 1
+        end
+
+        @config
+    end
+end

data/lib/sniffer.rb

@@ -0,0 +1,55 @@
+require 'pcap'
+
+class MemcacheSniffer 
+    attr_accessor :metrics, :semaphore
+
+    def initialize(config)
+        @source    = config[:nic]
+
+        @metrics = {}
+        @metrics[:calls]   = {}
+        @metrics[:objsize] = {}
+        @metrics[:reqsec]  = {}
+        @metrics[:bw]      = {}
+        @metrics[:stats]   = { :recv => 0, :drop => 0 }
+
+        @semaphore = Mutex.new
+    end
+
+    def start
+        cap = Pcap::Capture.open_live(@source, 1500)
+        
+        @metrics[:start_time] = Time.new.to_f
+
+        @done      = false
+
+        cap.setfilter('port 11211')
+        cap.loop do |packet|
+            @metrics[:stats] = cap.stats
+
+            # parse key name, and size from VALUE responses
+            if packet.raw_data =~ /VALUE (\S+) \S+ (\S+)/
+                key   = $1
+                bytes = $2
+
+                @semaphore.synchronize do
+                    if @metrics[:calls].has_key?(key)
+                        @metrics[:calls][key] += 1
+                    else
+                        @metrics[:calls][key] = 1
+                    end
+
+                    @metrics[:objsize][key] = bytes.to_i
+                end
+            end
+
+            break if @done
+        end
+
+        cap.close
+    end
+
+    def done
+        @done = true
+    end
+end

data/lib/ui.rb

@@ -0,0 +1,161 @@
+require 'curses'
+
+include Curses
+
+class UI
+    def initialize(config)
+        init_screen
+        cbreak
+        curs_set(0)
+
+        # set keyboard input timeout - sneaky way to manage refresh rate
+        Curses.timeout = config.has_key?(:refresh_rate) ? config[:refresh_rate] : 500
+
+        if can_change_color?
+            start_color
+            init_pair(0, COLOR_WHITE, COLOR_BLACK)
+            init_pair(1, COLOR_WHITE, COLOR_BLUE)
+            init_pair(2, COLOR_WHITE, COLOR_RED)
+        end
+
+        @stat_cols      = %w[ calls objsize req/sec bw(kbps) ]
+        @stat_col_width = 10 
+        @key_col_width  = 0
+
+        # we will delete any keys from the metrics table whose req/sec rate is below discard_rate
+        @discard_thresh = config.has_key?(:discard_thresh) ? config[:discard_thresh] : 0
+
+        @commands = {
+            'Q' => "quit",
+            'C' => "sort by calls",
+            'S' => "sort by size",
+            'R' => "sort by req/sec",
+            'B' => "sort by bandwidth",
+            'T' => "toggle sort order (asc|desc)"
+        }
+    end
+
+    def header
+        # pad stat columns to @stat_col_width
+        @stat_cols = @stat_cols.map { |c| sprintf("%#{@stat_col_width}s", c) }
+
+        # key column width is whatever is left over
+        @key_col_width = cols - (@stat_cols.length * @stat_col_width)
+
+        attrset(color_pair(1))
+        setpos(0,0)
+        addstr(sprintf "%-#{@key_col_width}s%s", "memcache key", @stat_cols.join)
+    end
+
+    def footer
+        footer_text = @commands.map { |k,v| "#{k}:#{v}" }.join(' | ')
+        setpos(lines-1, 0)
+        attrset(color_pair(2))
+        addstr(sprintf "%-#{cols}s", footer_text)
+    end
+
+    def render_stats(sniffer, sort_mode, sort_order = :desc)
+        render_start_t = Time.now.to_f * 1000
+
+        # subtract header + footer lines
+        maxlines = lines - 3
+        offset   = 1
+
+        # calculate packet loss ratio
+        if sniffer.metrics[:stats][:recv] > 0
+            loss = (sniffer.metrics[:stats][:drop].to_f / sniffer.metrics[:stats][:recv].to_f) * 100
+        else
+            loss = 0
+        end
+
+        # construct and render footer stats line
+        setpos(lines-2,0)
+        attrset(color_pair(2))
+        header_summary = sprintf "%-28s %-14s %-30s", 
+            "sort mode: #{sort_mode.to_s} (#{sort_order.to_s})",
+            "keys: #{sniffer.metrics[:calls].keys.count}",
+            "packets (recv/dropped): #{sniffer.metrics[:stats][:recv]} / #{sniffer.metrics[:stats][:drop]} (#{loss.round(2)}%)" 
+        addstr(sprintf "%-#{cols}s", header_summary)
+
+        # reset colours for main key display
+        attrset(color_pair(0))
+
+        top = []
+
+        sniffer.semaphore.synchronize do 
+            # we may have seen no packets received on the sniffer thread 
+            return if sniffer.metrics[:start_time].nil?
+
+            elapsed = Time.now.to_f - sniffer.metrics[:start_time]
+
+            # iterate over all the keys in the metrics hash and calculate some values
+            sniffer.metrics[:calls].each do |k,v|
+                    reqsec = v / elapsed
+
+                    # if req/sec is <= the discard threshold delete those keys from
+                    # the metrics hash - this is a hack to manage the size of the
+                    # metrics hash in high volume environments
+                    if reqsec <= @discard_thresh 
+                        sniffer.metrics[:calls].delete(k)
+                        sniffer.metrics[:objsize].delete(k)
+                        sniffer.metrics[:reqsec].delete(k)
+                        sniffer.metrics[:bw].delete(k)
+                    else
+                        sniffer.metrics[:reqsec][k]  = v / elapsed
+                        sniffer.metrics[:bw][k]      = ((sniffer.metrics[:objsize][k] * sniffer.metrics[:reqsec][k]) * 8) / 1000
+                    end
+            end
+
+            top = sniffer.metrics[sort_mode].sort { |a,b| a[1] <=> b[1] }
+        end
+
+        unless sort_order == :asc
+            top.reverse!
+        end
+
+        for i in 0..maxlines-1
+            if i < top.length
+                k = top[i][0]
+                v = top[i][1]
+
+                # if the key is too wide for the column truncate it and add an ellipsis
+                if k.length > @key_col_width
+                    display_key = k[0..@key_col_width-4]
+                    display_key = "#{display_key}..."
+                else
+                    display_key = k
+                end
+           
+                # render each key
+                line = sprintf "%-#{@key_col_width}s %9.d %9.d %9.2f %9.2f",
+                                 display_key,
+                                 sniffer.metrics[:calls][k],
+                                 sniffer.metrics[:objsize][k],
+                                 sniffer.metrics[:reqsec][k],
+                                 sniffer.metrics[:bw][k]
+            else
+                # we're not clearing the display between renders so erase past
+                # keys with blank lines if there's < maxlines of results
+                line = " "*cols
+            end
+
+            setpos(1+i, 0)
+            addstr(line)
+        end
+
+        # print render time in status bar
+        runtime = (Time.now.to_f * 1000) - render_start_t
+        attrset(color_pair(2))
+        setpos(lines-2, cols-18)
+        addstr(sprintf "rt: %8.3f (ms)", runtime)
+    end
+
+    def input_handler
+        getch
+    end
+
+    def done
+        nocbreak
+        close_screen
+    end
+end

data/mctop.gemspec

@@ -0,0 +1,19 @@
+# -*- encoding: utf-8 -*-
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+
+Gem::Specification.new do |gem|
+  gem.name          = "mctop"
+  gem.version       = "0.0.3"
+  gem.authors       = ["Marcus Barczak"]
+  gem.email         = ["marcus@etsy.com"]
+  gem.description   = %q{mctop - a realtime memcache key analyzer}
+  gem.summary       = %q{mctop - an interactive terminal app for analyzing memcache key activity breaking it out by requests per second, calls, and estimated bandwidth make sure you have the libpcap development libraries installed for the dependencies}
+  gem.homepage      = "https://github.com/etsy/mctop/"
+
+  gem.files         = `git ls-files`.split($/)
+  gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  gem.require_paths = ["lib"]
+
+  gem.add_runtime_dependency 'ruby-pcap', '~> 0.7.8'
+end

metadata

@@ -0,0 +1,74 @@
+--- !ruby/object:Gem::Specification
+name: mctop
+version: !ruby/object:Gem::Version
+  version: 0.0.3
+  prerelease: 
+platform: ruby
+authors:
+- Marcus Barczak
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2012-12-13 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: ruby-pcap
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.7.8
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.7.8
+description: mctop - a realtime memcache key analyzer
+email:
+- marcus@etsy.com
+executables:
+- mctop
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- Gemfile
+- LICENSE
+- README.md
+- Rakefile
+- bin/mctop
+- lib/cmdline.rb
+- lib/sniffer.rb
+- lib/ui.rb
+- mctop.gemspec
+homepage: https://github.com/etsy/mctop/
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.23
+signing_key: 
+specification_version: 3
+summary: mctop - an interactive terminal app for analyzing memcache key activity breaking
+  it out by requests per second, calls, and estimated bandwidth make sure you have
+  the libpcap development libraries installed for the dependencies
+test_files: []