redisrank 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 750cd449e44719036541e0fd53226339bc06da26
+  data.tar.gz: 808d6302c856ae4f132da7420e6c45434b94819f
+SHA512:
+  metadata.gz: 412112104821d68d5603ff6df2a629b519c61f4286cbecd10896b273bbea978b334df366b63e77cf77a613cbfbc68cc27f890f63623abf62b29a4b19a474fa96
+  data.tar.gz: 3e35841285088e42ffc6ba54b753aa17d9e147fef79d0639384c0dfeb072dc1f58a6c88e4419e51a7ed0129d896c880d56a7aef8d02d4ec69a553f420dfd57d4

data/.document

@@ -0,0 +1,5 @@
+README.md
+lib/**/*.rb
+bin/*
+features/**/*.feature
+LICENSE

data/.gitignore

@@ -0,0 +1,27 @@
+## MAC OS
+.DS_Store
+
+## TEXTMATE
+*.tmproj
+tmtags
+
+## EMACS
+*~
+\#*
+.\#*
+
+## VIM
+*.swp
+
+## PROJECT::GENERAL
+coverage
+rdoc
+pkg/*
+*.gem
+.bundle
+Gemfile.lock
+
+## PROJECT::SPECIFIC
+.yardoc/*
+spec/db/*
+doc/*

data/.rspec

@@ -0,0 +1,2 @@
+--format documentation
+--color

data/Gemfile

@@ -0,0 +1,4 @@
+source 'http://rubygems.org/'
+
+# Specify your gem's dependencies in redisrank.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2011 Jim Myhrberg.
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,297 @@
+A Redis-backed statistics storage and querying library written in Ruby.
+
+Redisrank was created taking as reference a Gem called Redistat by Jimeh.
+The motivations for the gem creation were similar to the Redistat too, I had a
+collection solution which was MySQL-based with the following requirements.
+
+* Fetch the top most ... 
+* This ranks should be fetched in any time-range
+* Screamingly fast
+
+## Installation
+
+    gem install redisrank
+
+If you are using Ruby 1.8.x, it's recommended you also install the
+`SystemTimer` gem, as the Redis gem will otherwise complain.
+
+## Usage (Crash Course)
+
+view\_stats.rb:
+
+```ruby
+require 'redisrank'
+
+class ViewRank
+  include Redisrank::Model
+end
+
+# if using Redisrank in multiple threads set this
+# somewhere in the beginning of the execution stack
+Redisrank.thread_safe = true
+```
+
+
+### Simple Example
+
+Store:
+
+```ruby
+ViewRank.store('hello', {:world => 4})
+ViewRank.store('hello', {:world => 2}, 2.hours.ago)
+```
+
+Fetch:
+
+```ruby
+ViewRank.find('hello', 1.hour.ago, 1.hour.from_now).all
+  #=> [{'world' => 4}]
+ViewRank.find('hello', 3.hour.ago, 1.hour.from_now).rank
+  #=> {'world' => 4}
+ViewRank.find('hello', 3.hour.ago, 1.hour.ago).rank
+  #=> {'world' => 2}
+```
+
+### Other usefull Use Cases
+
+
+
+## Terminology
+
+### Scope
+
+A type of global-namespace for storing data. When using the `Redisrank::Model`
+wrapper, the scope is automatically set to the class name. In the examples
+above, the scope is `ViewRank`. Can be overridden by calling the `#scope`
+class method on your model class.
+
+### Label
+
+Identifier string to separate different types and groups of statistics from
+each other. The first argument of the `#store`, `#find`, and `#fetch` methods
+is the label that you're storing to, or fetching from.
+
+Labels support multiple grouping levels by splitting the label string with `/`
+and storing the same stats for each level. For example, when storing data to a
+label called `views/product/44`, the data is stored for the label you specify,
+and also for `views/product` and `views`. You may also configure a different
+group separator using the `Redisrank.group_separator=` method. For example:
+
+```ruby
+Redisrank.group_separator = '|'
+```
+
+A word of caution: Don't use a crazy number of group levels. As two levels
+causes twice as many `hincrby` calls to Redis as not using the grouping
+feature. Hence using 10 grouping levels, causes 10 times as many write calls
+to Redis.
+
+### Input Statistics Data
+
+You provide Redisrank with the data you want to store using a Ruby Hash. This
+data is then stored in a corresponding Redis hash with identical key/field
+names.
+
+Key names in the hash also support grouping features similar to those
+available for Labels. Again, the more levels you use, the more write calls to
+Redis, so avoid using 10-15 levels.
+
+### Depth (Storage Accuracy)
+
+Define how accurately data should be stored, and how accurately it's looked up
+when fetching it again. By default Redisrank uses a depth value of `:hour`,
+which means it's impossible to separate two events which were stored at 10:18
+and 10:23. In Redis they are both stored within a date key of `2011031610`.
+
+You can set depth within your model using the `#depth` class method. Available
+depths are: `:year`, `:month`, `:day`, `:hour`, `:min`, `:sec`
+
+### Time Ranges
+
+When you fetch data, you need to specify a start and an end time. The
+selection behavior can seem a bit weird at first when, but makes sense when
+you understand how Redisrank works internally.
+
+For example, if we are using a Depth value of `:hour`, and we trigger a fetch
+call starting at `1.hour.ago` (13:34), till `Time.now` (14:34), only stats
+from 13:00:00 till 13:59:59 are returned, as they were all stored within the
+key for the 13th hour. If both 13:00 and 14:00 was returned, you would get
+results from two whole hours. Hence if you want up to the second data, use an
+end time of `1.hour.from_now`.
+
+### The Finder Object
+
+Calling the `#find` method on a Redisrank model class returns a
+`Redisrank::Finder` object. The finder is a lazy-loaded gateway to your
+data. Meaning you can create a new finder, and modify instantiated finder's
+label, scope, dates, and more. It does not call Redis and fetch the data until
+you call `#total`, `#all`, `#map`, `#each`, or `#each_with_index` on the
+finder.
+
+This section does need further expanding as there's a lot to cover when it
+comes to the finder.
+
+
+## Key Expiry
+
+Support for expiring keys from Redis is available, allowing you too keep
+varying levels of details for X period of time. This allows you easily keep
+things nice and tidy by only storing varying levels detailed stats only for as
+long as you need.
+
+In the below example we define how long Redis keys for varying depths are
+stored. Second by second stats are available for 10 minutes, minute by minute
+stats for 6 hours, hourly stats for 3 months, daily stats for 2 years, and
+yearly stats are retained forever.
+
+```ruby
+class ViewRank
+  include Redisrank::Model
+
+  depth :sec
+
+  expire \
+    :sec => 10.minutes.to_i,
+    :min => 6.hours.to_i,
+    :hour => 3.months.to_i,
+    :day => 2.years.to_i
+end
+```
+
+Keep in mind that when storing stats for a custom date in the past for
+example, the expiry time for the keys will be relative to now. The values you
+specify are simply passed to the `Redis#expire` method.
+
+
+## Internals
+
+### Storing / Writing
+
+Redisrank stores all data into a Redis hash keys. The Redis key name the used
+consists of three parts. The scope, label, and datetime:
+
+    {scope}/{label}:{datetime}
+
+For example, this...
+
+```ruby
+ViewRank.store('views/product/44', {'count/chrome/11' => 1})
+```
+
+...would store the follow hash of data...
+
+```ruby
+{ 'count' => 1, 'count/chrome' => 1, 'count/chrome/11' => 1 }
+```
+
+...to all 12 of these Redis hash keys...
+
+    ViewRank/views:2011
+    ViewRank/views:201103
+    ViewRank/views:20110315
+    ViewRank/views:2011031510
+    ViewRank/views/product:2011
+    ViewRank/views/product:201103
+    ViewRank/views/product:20110315
+    ViewRank/views/product:2011031510
+    ViewRank/views/product/44:2011
+    ViewRank/views/product/44:201103
+    ViewRank/views/product/44:20110315
+    ViewRank/views/product/44:2011031510
+
+...by creating the Redis key, and/or hash field if needed, otherwise it simply
+increments the already existing data.
+
+It would also create the following Redis sets to keep track of which child
+labels are available:
+
+    ViewRank.label_index:
+    ViewRank.label_index:views
+    ViewRank.label_index:views/product
+
+It should now be more obvious to you why you should think about how you use
+the grouping capabilities so you don't go crazy and use 10-15 levels. Storing
+is done through Redis' `hincrby` call, which only supports a single key/field
+combo. Meaning the above example would call `hincrby` a total of 36 times to
+store the data, and `sadd` a total of 3 times to ensure the label index is
+accurate. 39 calls is however not a problem for Redis, most calls happen in
+less than 0.15ms (0.00015 seconds) on my local machine.
+
+
+### Fetching / Reading
+
+By default when fetching statistics, Redisrank will figure out how to do the
+least number of reads from Redis. First it checks how long range you're
+fetching. If whole days, months or years for example fit within the start and
+end dates specified, it will fetch the one key for the day/month/year in
+question. It further drills down to the smaller units.
+
+It is also intelligent enough to not fetch each day from 3-31 of a month,
+instead it would fetch the data for the whole month and the first two days,
+which are then removed from the summary of the whole month. This means three
+calls to `hgetall` instead of 29 if each whole day was fetched.
+
+### Buffer
+
+The buffer is a new, still semi-beta, feature aimed to reduce the number of
+Redis `hincrby` that Redisrank sends. This should only really be useful when
+you're hitting north of 30,000 Redis requests per second, if your Redis server
+has limited resources, or against my recommendation you've opted to use 10,
+20, or more label grouping levels.
+
+Buffering tries to fold together multiple `store` calls into as few as
+possible by merging the statistics hashes from all calls and groups them based
+on scope, label, date depth, and more. You configure the the buffer by setting
+`Redisrank.buffer_size` to an integer higher than 1. This basically tells
+Redisrank how many `store` calls to buffer in memory before writing all data to
+Redis.
+
+
+## Todo
+
+* More details in Readme.
+* Documentation.
+* Anything else that becomes apparent after real-world use.
+
+
+## Credits
+
+[Encore Alert](http://encorealert.com/) that allowed me to spend some
+company time to further develop the project. @jimeh for creating the Redistat 
+that was not a inspiration but the base version of Redisrank.
+
+
+## Note on Patches/Pull Requests
+
+* Fork the project.
+* Make your feature addition or bug fix.
+* Add tests for it. This is important so I don't break it in a
+  future version unintentionally.
+* Commit, do not mess with rakefile, version, or history.  (if you want to
+  have your own version, that is fine but bump version in a commit by itself I
+  can ignore when I pull)
+* Send me a pull request. Bonus points for topic branches.
+
+
+## License and Copyright
+
+Copyright (c) 2011 Jim Myhrberg.
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -0,0 +1,69 @@
+require 'bundler'
+Bundler::GemHelper.install_tasks
+
+
+#
+# Rspec
+#
+
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec) do |spec|
+  spec.pattern = 'spec/**/*_spec.rb'
+end
+
+RSpec::Core::RakeTask.new(:rcov) do |spec|
+  spec.pattern = 'spec/**/*_spec.rb'
+  spec.rcov = true
+  spec.rcov_opts = ['--exclude', 'spec']
+end
+
+task :default => [:start, :spec, :stop]
+
+
+#
+# Start/stop Redis test server
+#
+
+REDIS_DIR = File.expand_path(File.join("..", "spec"), __FILE__)
+REDIS_CNF = File.join(REDIS_DIR, "redis-test.conf")
+REDIS_PID = File.join(REDIS_DIR, "db", "redis.pid")
+
+desc "Start the Redis test server"
+task :start do
+  unless File.exists?(REDIS_PID)
+    system "redis-server #{REDIS_CNF}"
+  end
+end
+
+desc "Stop the Redis test server"
+task :stop do
+  if File.exists?(REDIS_PID)
+    system "kill #{File.read(REDIS_PID)}"
+    system "rm #{REDIS_PID}"
+  end
+end
+
+
+#
+# Yard
+#
+
+begin
+  require 'yard'
+  YARD::Rake::YardocTask.new
+rescue LoadError
+  task :yardoc do
+    abort "YARD is not available. In order to run yardoc, you must: sudo gem install yard"
+  end
+end
+
+
+#
+# Misc.
+#
+
+desc "Start an irb console with Redisrank pre-loaded."
+task :console do
+  exec "irb -r spec/spec_helper"
+end
+task :c => :console

data/lib/redisrank.rb

@@ -0,0 +1,106 @@
+
+require 'rubygems'
+require 'date'
+require 'time'
+require 'digest/sha1'
+require 'monitor'
+
+# Active Support 2.x or 3.x
+require 'active_support'
+if !{}.respond_to?(:with_indifferent_access)
+  require 'active_support/core_ext/hash/indifferent_access'
+  require 'active_support/core_ext/hash/reverse_merge'
+end
+
+require 'time_ext'
+require 'redis'
+require 'json'
+
+require 'redisrank/mixins/options'
+require 'redisrank/mixins/synchronize'
+require 'redisrank/mixins/database'
+require 'redisrank/mixins/date_helper'
+
+require 'redisrank/connection'
+require 'redisrank/buffer'
+require 'redisrank/collection'
+require 'redisrank/date'
+require 'redisrank/event'
+require 'redisrank/finder'
+require 'redisrank/key'
+require 'redisrank/label'
+require 'redisrank/model'
+require 'redisrank/result'
+require 'redisrank/scope'
+require 'redisrank/summary'
+require 'redisrank/version'
+
+require 'redisrank/core_ext'
+
+
+module Redisrank
+
+  KEY_NEXT_ID = ".next_id"
+  KEY_EVENT = ".event:"
+  KEY_LABELS = "Redisrank.labels:" # used for reverse label hash lookup
+  KEY_EVENT_IDS = ".event_ids"
+  LABEL_INDEX = ".label_index:"
+  GROUP_SEPARATOR = "/"
+
+  class InvalidOptions < ArgumentError; end
+  class RedisServerIsTooOld < Exception; end
+
+  class << self
+
+    def buffer
+      Buffer.instance
+    end
+
+    def buffer_size
+      buffer.size
+    end
+
+    def buffer_size=(size)
+      buffer.size = size
+    end
+
+    def thread_safe
+      Synchronize.thread_safe
+    end
+
+    def thread_safe=(value)
+      Synchronize.thread_safe = value
+    end
+
+    def connection(ref = nil)
+      Connection.get(ref)
+    end
+    alias :redis :connection
+
+    def connection=(connection)
+      Connection.add(connection)
+    end
+    alias :redis= :connection=
+
+    def connect(options)
+      Connection.create(options)
+    end
+
+    def flush
+      puts "WARNING: Redisrank.flush is deprecated. Use Redisrank.redis.flushdb instead."
+      connection.flushdb
+    end
+
+    def group_separator
+      @group_separator ||= GROUP_SEPARATOR
+    end
+    attr_writer :group_separator
+
+  end
+end
+
+
+# ensure buffer is flushed on program exit
+Kernel.at_exit do
+  Redisrank.buffer.flush(true)
+end