degrade 0.1.0

data/.document

@@ -0,0 +1,5 @@
+README.rdoc
+lib/**/*.rb
+bin/*
+features/**/*.feature
+LICENSE

data/.gitignore

@@ -0,0 +1,21 @@
+## MAC OS
+.DS_Store
+
+## TEXTMATE
+*.tmproj
+tmtags
+
+## EMACS
+*~
+\#*
+.\#*
+
+## VIM
+*.swp
+
+## PROJECT::GENERAL
+coverage
+rdoc
+pkg
+
+## PROJECT::SPECIFIC

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 James Golick
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,84 @@
+= degrade
+
+Keep track of error rates using redis. Degrade functionality if they're too high.
+
+== Install it
+
+  gem install degrade
+
+== How it works
+
+Setup one instance per feature.
+
+  $redis             = Redis.new
+  $rollout           = Rollout.new($redis) # see http://github.com/jamesgolick/rollout
+  $degrade_cassandra = Degrade.new(@redis, :name             => :cassandra,
+                                           :sample           => 5000,            # optional - 5000 is the default
+                                           :minimum          => 100,             # optional - 100 is the default
+                                           :threshold        => 0.1,             # optional - 0.1 is the default
+                                           :errors           => [StandardError], # optional - [StandardError] is the default
+                                           :failure_strategy => lambda { $rollout.deactivate_all(:cassandra) }
+
+There are a bunch of options here:
+
+  * name: The name of the feature.
+  * sample: We zero the counters every n requests. Define n here.
+  * minimum: What's the minimum number of requests we need to see before we decide that the threshold has been met? Without this, if the first request of a given sample is a failure, we'd be failing the service.
+  * threshold: Percentage of failure requests necessary to trigger a failure.
+  * errors: Which errors should cause us to mark a failures?
+  * failure_strategy: The proc that will get called when the threshold is met. See http://github.com/jamesgolick/rollout if you don't have an existing system for disabling features on the fly.
+  
+The only thing left to do is wrap calls to whatever it is that might fail in a call to perform():
+
+  $degrade_cassandra.perform { $cassandra.get("Timelines", "1/everything") }
+
+Note that degrade doesn't actually handle the degradation for you. You still have to wrap sections of functionality with whatever mechanism you use to disable features (i.e. rollout).
+
+== Driver decorators
+
+Wrapping every call to a service in $degrade.perform() is a hassle and will require a lot of changes to your code. You're better off decorating your driver.
+
+Let's say we had a service called NewsService:
+
+  class NewsService < SomeRPCMechanism
+    def get_news(*args)
+      # ...
+    end
+  end
+
+We'd decorate it like this:
+
+  class NewsServiceWithDegradation
+    def initialize(news_service, degrade)
+      @news_service = news_service
+      @degrade      = degrade
+    end
+
+    def get_news(*args)
+      @degrade.perform { @news_service.get_news(*args) }
+    end
+  end
+
+We're doing this for one of our services and it's working pretty well.
+
+== Known issues / concerns
+
+Making a bunch of requests to services whose drivers are wrapped in this could cause problems if your services / site are extremely high throughput. This could be improved somewhat by adding some randomness to threshold checking and sample resetting so that those things aren't checked every request. The service we're currently using this with runs at about 25 req / s which is a _long_ way before any problems are likely. 
+
+Also, obviously calls to redis add latency to your requests. Make sure you're okay with this.
+
+Don't try to use this on redis itself. If you do, the universe will enter a state of infinite recursion.
+
+== Note on Patches/Pull Requests
+ 
+* Fork the project.
+* Make your feature addition or bug fix.
+* Add tests for it. This is important so I don't break it in a
+  future version unintentionally.
+* Commit, do not mess with rakefile, version, or history.
+  (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
+* Send me a pull request. Bonus points for topic branches.
+
+== Copyright
+
+Copyright (c) 2010 James Golick. See LICENSE for details.

data/Rakefile

@@ -0,0 +1,47 @@
+require 'rubygems'
+require 'rake'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gem|
+    gem.name = "degrade"
+    gem.summary = %Q{Keep track of error rates using redis. Degrade functionality if they're too high.}
+    gem.description = %Q{Keep track of error rates using redis. Degrade functionality if they're too high.}
+    gem.email = "jamesgolick@gmail.com"
+    gem.homepage = "http://github.com/jamesgolick/degrade"
+    gem.authors = ["James Golick"]
+    gem.add_development_dependency "rspec", "1.2.9"
+    gem.add_development_dependency "bourne", "1.0.0"
+    gem.add_development_dependency "redis", "0.1.0"
+    # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
+  end
+  Jeweler::GemcutterTasks.new
+rescue LoadError
+  puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
+end
+
+require 'spec/rake/spectask'
+Spec::Rake::SpecTask.new(:spec) do |spec|
+  spec.libs << 'lib' << 'spec'
+  spec.spec_files = FileList['spec/**/*_spec.rb']
+end
+
+Spec::Rake::SpecTask.new(:rcov) do |spec|
+  spec.libs << 'lib' << 'spec'
+  spec.pattern = 'spec/**/*_spec.rb'
+  spec.rcov = true
+end
+
+task :spec => :check_dependencies
+
+task :default => :spec
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+  version = File.exist?('VERSION') ? File.read('VERSION') : ""
+
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "degrade #{version}"
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end

data/VERSION

@@ -0,0 +1 @@
+0.1.0

data/lib/degrade.rb

@@ -0,0 +1,62 @@
+class Degrade
+  def initialize(redis, options)
+    @redis            = redis
+    @name             = options[:name]
+    @minimum          = options[:minimum] || 100
+    @sample           = options[:sample] || 5000
+    @threshold        = options[:threshold] || 0.1
+    @errors           = options[:errors] || [StandardError]
+    @failure_strategy = options[:failure_strategy]
+  end
+
+  def perform
+    begin
+      mark_request
+      yield
+    rescue *@errors => e
+      mark_failure
+      raise e
+    end
+  end
+
+  def requests
+    @redis.get(requests_key).to_f
+  end
+
+  def failures
+    @redis.get(failures_key).to_f
+  end
+
+  private
+    def requests_key
+      "status:#{@name}:requests"
+    end
+
+    def failures_key
+      "status:#{@name}:failures"
+    end
+
+    def mark_request
+      @redis.incr(requests_key)
+      reset_sample
+    end
+
+    def mark_failure
+      @redis.incr(failures_key)
+      check_threshold
+    end
+
+    def check_threshold
+      total_requests = requests
+      return if total_requests < @minimum
+    
+      @failure_strategy.call if failures / total_requests >= @threshold
+    end
+
+    def reset_sample
+      if requests > @sample
+        @redis.del(requests_key)
+        @redis.del(failures_key)
+      end
+    end
+end

data/spec/degrade_spec.rb

@@ -0,0 +1,57 @@
+require File.expand_path(File.dirname(__FILE__) + '/spec_helper')
+
+describe "Degrade" do
+  before do
+    @redis        = Redis.new
+    @failure_proc = stub(:call => nil)
+    @degrade      = Degrade.new(@redis, :name             => :chat_service, 
+                                        :sample           => 100,
+                                        :minimum          => 100,
+                                        :threshold        => 0.1,
+                                        :errors           => [StandardError],
+                                        :failure_strategy => @failure_proc)
+  end
+
+  describe "when there's an error" do
+    it "reraises the error" do
+      lambda { @degrade.perform { raise "asdf" } }.should raise_error
+    end
+  end
+
+  describe "when the threshold is reached before the minimum number of requests" do
+    before do
+      9.times { @degrade.perform { } }
+      @degrade.perform { raise "" } rescue nil
+    end
+
+    it "doesn't call the failure strategy" do
+      @failure_proc.should_not have_received(:call)
+    end
+  end
+
+  describe "when the threshold is reached" do
+    before do
+      90.times { @degrade.perform {} }
+      10.times { @degrade.perform { raise "" } rescue nil }
+    end
+
+    it "calls the failure strategy" do
+      @failure_proc.should have_received(:call)
+    end
+  end
+
+  describe "when the requests grow above the sample" do
+    before do
+      @degrade.perform { raise "" } rescue nil
+      100.times { @degrade.perform { } }
+    end
+
+    it "resets the counter" do
+      @degrade.requests.should == 0
+    end
+
+    it "resets the requests" do
+      @degrade.failures.should == 0
+    end
+  end
+end

data/spec/spec.opts

@@ -0,0 +1 @@
+--color

data/spec/spec_helper.rb

@@ -0,0 +1,12 @@
+$LOAD_PATH.unshift(File.dirname(__FILE__))
+$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
+require 'degrade'
+require 'spec'
+require 'spec/autorun'
+require 'bourne'
+require 'redis'
+
+Spec::Runner.configure do |config|
+  config.mock_with :mocha
+  config.before { Redis.new.flushdb }
+end

metadata

@@ -0,0 +1,125 @@
+--- !ruby/object:Gem::Specification 
+name: degrade
+version: !ruby/object:Gem::Version 
+  hash: 27
+  prerelease: false
+  segments: 
+  - 0
+  - 1
+  - 0
+  version: 0.1.0
+platform: ruby
+authors: 
+- James Golick
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2010-07-17 00:00:00 -07:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: rspec
+  prerelease: false
+  requirement: &id001 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - "="
+      - !ruby/object:Gem::Version 
+        hash: 13
+        segments: 
+        - 1
+        - 2
+        - 9
+        version: 1.2.9
+  type: :development
+  version_requirements: *id001
+- !ruby/object:Gem::Dependency 
+  name: bourne
+  prerelease: false
+  requirement: &id002 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - "="
+      - !ruby/object:Gem::Version 
+        hash: 23
+        segments: 
+        - 1
+        - 0
+        - 0
+        version: 1.0.0
+  type: :development
+  version_requirements: *id002
+- !ruby/object:Gem::Dependency 
+  name: redis
+  prerelease: false
+  requirement: &id003 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - "="
+      - !ruby/object:Gem::Version 
+        hash: 27
+        segments: 
+        - 0
+        - 1
+        - 0
+        version: 0.1.0
+  type: :development
+  version_requirements: *id003
+description: Keep track of error rates using redis. Degrade functionality if they're too high.
+email: jamesgolick@gmail.com
+executables: []
+
+extensions: []
+
+extra_rdoc_files: 
+- LICENSE
+- README.rdoc
+files: 
+- .document
+- .gitignore
+- LICENSE
+- README.rdoc
+- Rakefile
+- VERSION
+- lib/degrade.rb
+- spec/degrade_spec.rb
+- spec/spec.opts
+- spec/spec_helper.rb
+has_rdoc: true
+homepage: http://github.com/jamesgolick/degrade
+licenses: []
+
+post_install_message: 
+rdoc_options: 
+- --charset=UTF-8
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.3.7
+signing_key: 
+specification_version: 3
+summary: Keep track of error rates using redis. Degrade functionality if they're too high.
+test_files: 
+- spec/degrade_spec.rb
+- spec/spec_helper.rb