spanx 0.1.0

data/.gitignore

@@ -0,0 +1,20 @@
+*.gem
+*.rbc
+.idea
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+
+spanx-config.yml

data/.pairs

@@ -0,0 +1,13 @@
+pairs:
+  ag: Atasay Gokkaya; atasay
+  km: Kaan Meralan; kaan
+  kg: Konstantin Gredeskoul; kig
+  ph: Paul Henry; paul
+  sf: Sean Flannagan; sean
+  es: Eric Saxby; sax
+  tn: Truong Nguyen; constantx
+  cc: Cihan Cimen; cihan
+  sc: Server Cimen; server
+email:
+  prefix: pair
+  domain: wanelo.com

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--format progress

data/.rvmrc

@@ -0,0 +1 @@
+rvm use 1.9.3@spanx --create

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in spanx.gemspec
+gemspec

data/Guardfile

@@ -0,0 +1,13 @@
+#!/usr/bin/env ruby
+#^syntax detection
+
+# A sample Guardfile
+# More info at https://github.com/guard/guard#readme
+
+guard 'rspec' do
+  watch(%r{^spanx\.gemspec}) { "spec"}
+  watch(%r{^spec/.+_spec\.rb$})
+  watch(%r{^lib/(.+)\.rb$}) { "spec" }
+  watch('spec/spec_helper.rb')  { "spec" }
+end
+

data/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2012 Konstantin Gredeskoul
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,175 @@
+# Spanx
+
+Spank down IP spam: IP-based rate limiting for web applications behind HTTP server such as nginx or Apache.
+
+Spanx is a simple Redis-based web request rate limiter, which integrates into any web application simply by monitoring
+one or more HTTP server access log file(s) in real time (think Apache/nginx access.log).
+
+Basic flow is as follows:
+
+* Spanx tails the access.log file(s)
+* parses out IP addresses of each request
+* maintains a tally of request counts per IP, and per a time slice.
+* Spanx is then able to detect when one or more IPs exceed the rate limiting configuration thresholds provided
+(multiple thresholds are supported).
+  * When such IP is detected, Spanx immediately writes it out into a block-list file (suitable for consumption by nginx or
+apache, in format eg "deny 127.0.0.1;"), and then
+  * executes a pre-configured command, presumed to reload HTTP server configuration (such as HUP nginx, etc) and activate new blocking rules.
+
+Spanx additionally supports regular expression based white list file, that can be used to eliminate certain log lines
+from the consideration (for example, you Googlebot based on User-Agent).
+
+### Design
+
+Spanx can be integrated into part of your application, or can run as a standalone ruby app.  Spanx requires ruby
+1.9.3, and it uses ruby threads to work on a few things in parallel.
+
+Spanx has two main components:
+
+1. *watcher* is a process that monitors HTTP server log files, and updates Redis periodically with most recent counts.
+   Watcher also writes out the blocked IP file, if blocked IPs are found in Redis database.
+
+2. *analyzer* is a process that reads up to date information on IP addresses from Redis, and analyzes it. If any rate
+   limit-exceeding IPs are found, it writes them to the Redis DB, with an expiration TTL set.
+
+If you have only one web server, you can run both watcher and analyzer as a single ruby process.
+
+If you have multiple web servers, you need to run watcher on each server, and analyzer only once (somewhere).
+
+### Alerts
+
+Besides actually writing out IPs to a block list file, Spanx supports notifiers that will be called when a new IP
+is blocked.  Currently supported are audit log notifier (that writes that information to a log file), a Campfire
+Chat notifier, which will print IP blocking information into your Campfire chat room, and an Email notifier. It is
+very easy to write additional notifiers.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'spanx'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install spanx
+
+### Dependencies
+
+Spanx uses the Pause gem to persist state. This depends on Redis to save state and do set logic on the information it finds.
+
+## Usage
+
+Spanx has a single executable with several sub-commands. In practice, multiple commands will
+be run concurrently to do all of the necessary calculations.
+
+Configuration can be provided via a YAML file (see example), and/or via command line options. Not
+all configuration can be set via command line. If an option is provided in both YAML file and command line,
+then latter is chosen.
+
+
+### watch
+
+This command watches an HTTP server log file and writes out blocked IPs to a file specified.
+
+```bash
+  Usage: [bundle exec] spanx watch [options]
+    -f, --file ACCESS_LOG            Apache/nginx access log file to scan continuously
+    -z, --analyze                    Analyze IPs also (as opposed to running `spanx analyze` in another process)
+    -b, --block_file BLOCK_FILE      Output file to store NGINX block list
+    -c, --config CONFIG              Path to config file (YML) (required)
+    -d, --daemonize                  Detach from TTY and run as a daemon
+    -g, --debug                      Log to STDOUT status of execution and some time metrics
+    -r, --run <shell command>        Shell command to run anytime blocked ip file changes, for example "sudo pkill -HUP nginx"
+    -w, --whitelist WHITELIST        File with newline separated reg exps, to exclude lines from access log
+    -h, --help                       Show this message
+```
+
+### analyze
+
+Analyzes IPs found by the `watch` command. If an IP exceeds its maximum count for a time
+period check (as set in the config file), the IP is written into Redis with a TTL defined by the
+period check.
+
+```bash
+Usage: [bundle exec] spanx analyze [options]
+    -a, --audit AUDIT_FILE           Historical record of IP blocking decisions
+    -c, --config CONFIG              Path to config file (YML) (required)
+    -d, --daemonize
+    -g, --debug                      Log status to STDOUT
+    -h, --help                       Show this message
+```
+
+### disable
+
+Disables IP blocking. Note that this only effects the actual writing out
+of block files, not of IP tracking or analysis. Note that this requires
+a connection to redis, and thus requires the same config file used in
+`analyze` and `watch`.
+
+```bash
+Usage: [bundle exec] spanx disable [options]
+    -c, --config CONFIG              Path to config file (YML) (required)
+    -g, --debug                      Log status to STDOUT
+    -h, --help                       Show this message
+```
+
+### disable
+
+Reenables IP blocking if disabled. As with `disable`, the config file is
+required to connect to redis.
+
+```bash
+Usage: [bundle exec] spanx enable [options]
+    -c, --config CONFIG              Path to config file (YML) (required)
+    -g, --debug                      Log status to STDOUT
+    -h, --help                       Show this message
+```
+
+### flush
+
+This removes the persistence data around current IP blocks. Use this
+when you want to remove all data around current blocks without (or in
+addition to) disabling the blocker.
+
+```bash
+Usage: [bundle exec] spanx flush [options]
+    -c, --config CONFIG              Path to config file (YML) (required)
+    -g, --debug                      Log status to STDOUT
+    -h, --help                       Show this message
+```
+
+## Examples
+
+If you have only one load balancer, you may want to centralize all work into a single process, as such:
+
+```bash
+ $ spanx watch -w /path/to/whitelist -c /path/to/spanx.conf.yml -z -d
+```
+
+With multiple load balancers, this may not be desirable. All hosts will need to process their own access
+log, but a minimum number of hosts should analyze the IP traffic.
+
+```bash
+ lb1 $ spanx watch -c spanx.conf.yml -r "sudo pkill -HUP nginx" --debug 2>&1 >> /var/log/spanx.watch.log &
+ lb2 $ spanx watch -c spanx.conf.yml -r "sudo pkill -HUP nginx" --debug 2>&1 >> /var/log/spanx.watch.log &
+
+ lb2 $ spanx analyze -c spanx.conf.yml -a spanx.audit.log --debug 2>&1 >> /var/log/spanx.analyze.log &
+```
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Added some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request
+
+## Maintainers
+
+Konstantin Gredeskoul (@kigster) and Eric Saxby (@sax) at Wanelo, Inc (http://github.com/wanelo)
+
+(c) 2012, All rights reserved.

data/Rakefile

@@ -0,0 +1,2 @@
+#!/usr/bin/env rake
+require "bundler/gem_tasks"

data/bin/spanx

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby
+ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../../Gemfile', __FILE__)
+require 'rubygems'
+require 'bundler/setup' if File.exists?(ENV['BUNDLE_GEMFILE'])
+require 'spanx'
+
+Spanx::CLI.new.run

data/conf/spanx-config.yml.example

@@ -0,0 +1,44 @@
+---
+:access_log: ""
+:block_file: "block-ips.conf"
+:audit_file: "spanx-audit.log"
+:redis:
+  :host: "127.0.0.1"
+  :port: 6379
+  :db: 1
+:collector:
+  :resolution: 300       # seconds
+  :history: 21600        # seconds
+  :flush_interval: 5
+:log_reader:
+  :tail_interval: 1
+:analyzer:
+  :analyze_interval: 20
+  :blocked_ip_notifiers:
+    - "Spanx::Notifier::AuditLog"
+    - "Spanx::Notifier::Campfire"
+    - "Spanx::Notifier::Email"
+  :period_checks:
+    - :period_seconds: 3600
+      :max_allowed: 2000
+      :block_ttl: 7200
+    - :period_seconds: 600
+      :max_allowed: 600
+      :block_ttl: 1200
+    - :period_seconds: 21600
+      :max_allowed: 8000
+      :block_ttl: 64800
+:writer:
+  :write_interval: 10
+:campfire:
+  :enabled: true
+  :room_id: 1111
+  :token: aaffdfsdfadfasdfasdfasdf
+  :account: test
+:email:
+  :enabled: true
+  :to: "everyone@mycompany.com"
+  :from: "spanx@mycompany.com"
+  :password: "s3cVr3p4ssw0rd"
+  :domain: "mycompany.com"
+  :gateway: "smtp.gmail.com"

data/conf/spanx-whitelist.txt.example

@@ -0,0 +1,2 @@
+^(8.8.8.[8-9])
+msnbot|Googlebot|bingbot|Yahoo. Slurp|facebookexternalhit

data/lib/spanx.rb

@@ -0,0 +1,38 @@
+require 'redis'
+require 'pause'
+require 'spanx/version'
+require 'spanx/helper'
+require 'spanx/logger'
+require 'spanx/config'
+require 'spanx/usage'
+
+require 'spanx/ip_checker'
+
+require 'spanx/cli'
+require 'spanx/notifier/base'
+require 'spanx/notifier/campfire'
+require 'spanx/notifier/audit_log'
+require 'spanx/notifier/email'
+
+require 'spanx/actor/log_reader'
+require 'spanx/actor/collector'
+require 'spanx/actor/analyzer'
+require 'spanx/actor/writer'
+require 'spanx/whitelist'
+
+require 'spanx/runner'
+
+module Spanx
+end
+
+class String
+  def constantize
+    camel_cased_word = self
+    unless /\A(?:::)?([A-Z]\w*(?:::[A-Z]\w*)*)\z/ =~ camel_cased_word
+      raise NameError, "#{camel_cased_word.inspect} is not a valid constant name!"
+    end
+
+    Object.module_eval("::#{$1}", __FILE__, __LINE__)
+  end
+end
+

data/lib/spanx/actor/analyzer.rb

@@ -0,0 +1,94 @@
+require 'spanx/logger'
+require 'spanx/helper/timing'
+require 'spanx/notifier/base'
+require 'spanx/notifier/campfire'
+require 'spanx/notifier/audit_log'
+require 'spanx/notifier/email'
+
+module Spanx
+  module Actor
+    class Analyzer
+      include Spanx::Helper::Timing
+
+      attr_accessor :config, :notifiers, :blocked_ips
+
+      def initialize config
+        @config = config
+        @audit_file = config[:audit_file]
+        @notifiers = []
+        initialize_notifiers(config) if config[:analyzer][:blocked_ip_notifiers]
+
+        @blocked_ips = []
+        @previously_blocked_ips = []
+      end
+
+      def run
+        Thread.new do
+          Thread.current[:name] = "analyzer"
+          Logger.log "starting analyzer loop..."
+          loop do
+            analyze_all_ips()
+            sleep config[:analyzer][:analyze_interval]
+          end
+        end
+      end
+
+      # Look through every IP on the stack. IPs that fulfill a PeriodCheck
+      # are pushed onto a redis block list.
+      def analyze_all_ips
+        return unless Spanx::IPChecker.enabled?
+
+        @previously_blocked_ips = Spanx::IPChecker.blocked_identifiers
+
+        ips = Spanx::IPChecker.tracked_identifiers
+
+        Logger.logging "analyzed #{ips.size} IPs" do
+          ips.each do |ip|
+            blocked_ip = analyze_ip(ip)
+            blocked_ips << blocked_ip if blocked_ip
+          end
+        end
+
+        Logger.log "blocking [#{blocked_ips.size}] ips" unless blocked_ips.empty?
+        call_notifiers(blocked_ips)
+        blocked_ips.clear
+      end
+
+      # Analyze individual IP for all defined periods.  As soon as one
+      # rule is triggered, exit the method
+      def analyze_ip(ip)
+        Spanx::IPChecker.new(ip).analyze
+      end
+
+      private
+
+      def initialize_notifiers(config)
+        notifiers_to_initialize = config[:analyzer][:blocked_ip_notifiers]
+        notifiers_to_initialize.each do |class_name|
+          Logger.logging "instantiating notifier #{class_name}" do
+            begin
+              notifier = class_name.constantize.new(config)
+              self.notifiers << notifier
+            rescue => e
+              Logger.log "error instantiating #{class_name}: #{e.inspect}, notifier disabled."
+            end
+          end
+        end
+      end
+
+      def call_notifiers(blocked_ips)
+        unless notifiers.empty?
+          blocked_ips.reject { |b| @previously_blocked_ips.include?(b.identifier) }.each do |blocked_ip|
+            self.notifiers.each do |notifier|
+              begin
+                notifier.publish(blocked_ip)
+              rescue => e
+                Logger.log "error notifying #{notifier.inspect} about blocked IP #{blocked_ip}: #{e.inspect}"
+              end
+            end
+          end
+        end
+      end
+    end
+  end
+end