frivol 0.3.1 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: fb5b81ba6e03198c9bd7e33e3ce843dd3a0d48df
-  data.tar.gz: 1e2456b6e27f97eec9309fc7e638c498bc166f5a
+  metadata.gz: 49b60bcea96f41ca6dfd54dc9b3b4c63db50c144
+  data.tar.gz: e47b8d3203ca595433192dbe55673fff5d2ad13e
 SHA512:
-  metadata.gz: 741f725ccee1f4fd43e5edde0d2de85500511c84c547dbc1af85baa2a8eff8cbc9498809d38a7c60e28d6d22eb86a43bfe42e0db74bd8e6087c1cfa8f7f4e3da
-  data.tar.gz: 0d1cc7ddeaffc023ccb1449a28e1ce9b9f536667d2fea6c3f7f2e5ed96663162905dc009114adef5dcb7ea38cb2cc8a2b8594f1e95ba37261efc7e0383397a73
+  metadata.gz: b4ebac79ca4b09b7685580b630ad763c75766b9daaf70b624c85d856bbcb44282ace07cc6ec2f890fb3c6519bac2ab4116b36580d910e9c2d128365ff9313801
+  data.tar.gz: e523e418a679e1b81e0645314e482e592f01f23fa89db470ec52ef382f394043ea8afe355195771bdf657b8aead9d2b10381296c00674915c4294d2e40f07c41

data/.travis.yml

@@ -2,14 +2,15 @@ language: ruby
 bundler_args: --without development
 before_install:
 - 'echo ''gem: --no-ri --no-rdoc'' > ~/.gemrc'
+services:
+  - redis-server
+  - riak
 rvm:
-  - 1.8.7
-  - 1.9.2
   - 1.9.3
   - 2.0.0
-  # - 2.1.0
-  - jruby-18mode
-  - jruby-19mode
+  - 2.1.0
+  - 2.2.0
   - ruby-head
+  - jruby-19mode
   - jruby-head
-  - ree
+script: bundle exec rake test:all

data/CHANGES.md

@@ -0,0 +1,15 @@
+<!--- TODO: Date and version -->
+# March 2014 (v)
+<!--- TODO: Check on this version number -->
+- Drop support for Redis < 2.2.6
+- Add backends for Redis, Redis::Distributed and Riak
+- FIXED: counters never expire if created with increment or decrement methods
+
+# Previous
+<!--- TODO: Fill in -->
+
+
+# TODO/BUGS(?):
+- Hook AR reload method to clear_storage?
+- Add a way (maybe using MR) to expire keys in Riak
+- BUG: Now, counters always have their expiry reset

data/Gemfile

@@ -1,13 +1,16 @@
 source 'https://rubygems.org'
 
 gem 'multi_json'
-gem 'redis'
 gem 'rake'
 
 group :development do
   gem 'jeweler'
+  gem 'pry'
+  gem 'pry-debugger'
 end
 
 group :test do
   gem 'test-unit'
+  gem 'redis'
+  gem 'riak-client'
 end

data/README.rdoc

@@ -10,14 +10,27 @@ when a hot system is restarted. Frivol's design is such that it solves our probl
 believe it is generic enough to be used in many Rails web projects and even in other types of
 projects altogether.
 
+As of version 0.4.0, Frivol supports various backends, including Redis::Destributed and Riak.
+There's a Multi backend which will migrate keys from an old backend to a new one, like from
+Redis to Redis::Destributed.
+
 == Usage
+
+=== Configuration
 Configure Frivol in your configuration, for example in an initializer or in environment.rb
   REDIS_CONFIG = {
     :host => "localhost",
     :port => 6379
   }
-  Frivol::Config.redis_config = REDIS_CONFIG
+  Frivol::Config.backend = Frivol::Backend::Redis.new(REDIS_CONFIG)
+
+**Note:** This configuration has changed in version 0.4.0 with the introduction of <tt>Backend</tt>s
+
+You can also use RedisDestributed or Riak backends. These are configured similarly. Finally there is a
+Multi which works to migrate data from an old backend to a new one. For example, we used a Multi backend
+to migrate from a single Redis instance to a RedisDistributed pair.
 
+=== Standard Usage
 Now include Frivol in whichever classes you'd like to make use of temporary storage. You can optionally
 call the <tt>storage_expires_in(time)</tt> class method to set a default expiry. In your methods you can
 now call the <tt>store(keys_and_values)</tt> and <tt>retrieve(keys_and_defaults)</tt> methods.
@@ -29,6 +42,33 @@ The default is not to expire the storage, in which case it will live for as long
 <tt>delete_storage</tt>, as the name suggests will immediately delete the storage, while <tt>clear_storage</tt>
 will clear the cache that Frivol keeps and force the next <tt>retrieve</tt> to return to Redis for the data.
 
+==== Example
+  class BigComplexCalcer
+    include Frivol
+    storage_expires_in 600 # temporary storage expires in 10 minutes.
+    def initialize(key)
+      @key = key
+    end
+    def storage_key(bucket = nil)
+      "frivol-test-#{key}" # override the storage key because we don't respond_to? :id, and don't care about buckets
+    end
+    def big_complex_calc
+      retrieve :complex => :do_big_complex_calc # do_big_complex_calc is the method to get the default from
+    end
+    def last_calc_done
+      last = retrieve(:last => nil) # default is nil
+      return "never" if last.nil?
+      return "#{Time.now - Time.at(last)} seconds ago"
+    end
+    def do_big_complex_calc
+      # Wee! Do some really hard work here...
+      # ...still working...
+      store :complex => result, :last => Time.now.to_i # ...and let's keep the result for at least 10 minutes, as well as the last timme we did it
+    end
+  end
+
+
+=== Buckets
 Since version 0.1.5 Frivol can create different storage buckets. Note that this introduces a breaking change
 to the <tt>storage_key</tt> method if you have overriden it. It now takes a +bucket+ parameter.
 
@@ -42,6 +82,7 @@ exactly like the standard +store+ and +retrieve+ methods. There will also be <tt
 take a integer (value and default, respectively) and the increment does not take a parameter. Since version 0.2.1
 there is also <tt>increment_my_counter_by</tt>, <tt>decrement_my_counter</tt> and <tt>decrement_my_counter_by<tt>.
 
+=== Conditional retrieval
 Fine grained control of storing and retrieving values from buckets can be controlled using the :condition and
 :else options.
   storage_bucket :my_bucket,
@@ -61,30 +102,21 @@ Frivol uses the +storage_key+ method to create a base key for storage in Redis.
 <tt>"#{self.class.name}-#{id}"</tt> so you'll want to override that method if you have classes that don't
 respond to id.
 
-== Example
-  class BigComplexCalcer
-    include Frivol
-    storage_expires_in 600 # temporary storage expires in 10 minutes.
-    def initialize(key)
-      @key = key
-    end
-    def storage_key(bucket = nil)
-      "frivol-test-#{key}" # override the storage key because we don't respond_to? :id, and don't care about buckets
-    end
-    def big_complex_calc
-      retrieve :complex => :do_big_complex_calc # do_big_complex_calc is the method to get the default from
-    end
-    def last_calc_done
-      last = retrieve(:last => nil) # default is nil
-      return "never" if last.nil?
-      return "#{Time.now - Time.at(last)} seconds ago"
-    end
-    def do_big_complex_calc
-      # Wee! Do some really hard work here...
-      # ...still working...
-      store :complex => result, :last => Time.now.to_i # ...and let's keep the result for at least 10 minutes, as well as the last me we did it
-    end
+=== Frivolize
+The +frivolize+ method is similar to +memoize+ except that it uses the backend to store the data which can thus be
+shared amongst multiple processes.
+
+  def long_running_result
+    # do lots of hard work
+  end
+  frivolize :long_running_result
+
+The frivolize also is also able to take options to make counters and expire.
+
+  def long_running_count
+    # count something that takes a long time
   end
+  frivolize :long_running_count, counter: true, expires_in: 10.minutes
 
 == Time Extensions
 These extensions allow the storing and retrieving of <tt>Time</tt> and

data/Rakefile

@@ -24,6 +24,23 @@ Rake::TestTask.new(:test) do |test|
   test.verbose = true
 end
 
+namespace :test do
+  desc 'Test all backends and the Multi backend in all combinations'
+  task :all do
+    %w[
+      fake_redis redis redis_distributed riak multi
+      multi_redis_redis multi_redis_redis_distributed
+      multi_redis_riak multi_redis_distributed_riak
+      multi_riak_redis multi_riak_redis_distributed
+    ].each do |backend|
+      puts "Testing #{backend}"
+      ENV['backend'] = backend
+      Rake::Task["test"].reenable
+      Rake::Task["test"].invoke
+    end
+  end
+end
+
 task :default => :test
 
 begin

data/VERSION

@@ -1 +1 @@
-0.3.1
+0.4.0

data/frivol.gemspec

@@ -2,16 +2,16 @@
 # DO NOT EDIT THIS FILE DIRECTLY
 # Instead, edit Jeweler::Tasks in Rakefile, and run 'rake gemspec'
 # -*- encoding: utf-8 -*-
-# stub: frivol 0.3.1 ruby lib
+# stub: frivol 0.4.0 ruby lib
 
 Gem::Specification.new do |s|
   s.name = "frivol"
-  s.version = "0.3.1"
+  s.version = "0.4.0"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.require_paths = ["lib"]
   s.authors = ["Marc Heiligers"]
-  s.date = "2014-03-06"
+  s.date = "2015-06-28"
   s.description = "Simple Redis backed temporary storage intended primarily for use with ActiveRecord models to provide caching"
   s.email = "marc@eternal.co.za"
   s.extra_rdoc_files = [
@@ -21,6 +21,7 @@ Gem::Specification.new do |s|
   s.files = [
     ".document",
     ".travis.yml",
+    "CHANGES.md",
     "Gemfile",
     "LICENSE",
     "README.rdoc",
@@ -53,6 +54,10 @@ Gem::Specification.new do |s|
     "doc/rdoc-style.css",
     "frivol.gemspec",
     "lib/frivol.rb",
+    "lib/frivol/backend/multi.rb",
+    "lib/frivol/backend/redis.rb",
+    "lib/frivol/backend/redis_distributed.rb",
+    "lib/frivol/backend/riak.rb",
     "lib/frivol/class_methods.rb",
     "lib/frivol/config.rb",
     "lib/frivol/functor.rb",
@@ -60,6 +65,7 @@ Gem::Specification.new do |s|
     "lib/frivol/time_extensions.rb",
     "test/fake_redis.rb",
     "test/helper.rb",
+    "test/test_backend.rb",
     "test/test_buckets.rb",
     "test/test_condition.rb",
     "test/test_condition_with_counters.rb",
@@ -68,6 +74,10 @@ Gem::Specification.new do |s|
     "test/test_extensions.rb",
     "test/test_frivol.rb",
     "test/test_frivolize.rb",
+    "test/test_multi_backend.rb",
+    "test/test_multi_backend_counters.rb",
+    "test/test_multi_backend_expiry.rb",
+    "test/test_riak.rb",
     "test/test_seeds.rb",
     "test/test_threads.rb"
   ]
@@ -80,20 +90,23 @@ Gem::Specification.new do |s|
 
     if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
       s.add_runtime_dependency(%q<multi_json>, [">= 0"])
-      s.add_runtime_dependency(%q<redis>, [">= 0"])
       s.add_runtime_dependency(%q<rake>, [">= 0"])
       s.add_development_dependency(%q<jeweler>, [">= 0"])
+      s.add_development_dependency(%q<pry>, [">= 0"])
+      s.add_development_dependency(%q<pry-debugger>, [">= 0"])
     else
       s.add_dependency(%q<multi_json>, [">= 0"])
-      s.add_dependency(%q<redis>, [">= 0"])
       s.add_dependency(%q<rake>, [">= 0"])
       s.add_dependency(%q<jeweler>, [">= 0"])
+      s.add_dependency(%q<pry>, [">= 0"])
+      s.add_dependency(%q<pry-debugger>, [">= 0"])
     end
   else
     s.add_dependency(%q<multi_json>, [">= 0"])
-    s.add_dependency(%q<redis>, [">= 0"])
     s.add_dependency(%q<rake>, [">= 0"])
     s.add_dependency(%q<jeweler>, [">= 0"])
+    s.add_dependency(%q<pry>, [">= 0"])
+    s.add_dependency(%q<pry-debugger>, [">= 0"])
   end
 end
 

data/lib/frivol.rb

@@ -1,5 +1,3 @@
-require "redis"
-
 # == Frivol
 module Frivol
   require "frivol/config"
@@ -56,7 +54,7 @@ module Frivol
   # Expire the stored data in +time+ seconds.
   def expire_storage(time, bucket = nil)
     return if time.nil?
-    Frivol::Config.redis.expire storage_key(bucket), time
+    Frivol::Config.backend.expire storage_key(bucket), time
   end
 
   # The base key used for storage in Redis.

data/lib/frivol/backend/multi.rb

@@ -0,0 +1,159 @@
+module Frivol
+  module Backend
+    # == Configuration
+    # This backend is used to migrate from one backend to another (and another) (and another).
+    # Essentially, this backend will check for the existence of the key in the newest backend,
+    # and if not found check in the older backends in order. If it's found in an older backend
+    # it will move the value to the new backend.
+    #
+    # I have used this in production to move from Redis to RedisDistributed.
+    #   old_backend = Frivol::Backend::Redis.new(:db => 10)
+    #   new_backend = Frivol::Backend::RedisDistributed.new(["redis://127.0.0.1:6379/11", "redis://127.0.0.1:6379/12"])
+    #   Frivol::Config.backend = Frivol::Backend::Multi.new([ new_backend, old_backend ])
+    class Multi
+      # :nodoc:
+      BackendsNotUniqueError = Class.new(StandardError)
+
+      def initialize(backends)
+        unless backends.map(&:inspect).uniq.size == backends.size
+          raise BackendsNotUniqueError, "Backends are not unique: #{backends.map(&:inspect).join(', ')}"
+        end
+        @primary_backend = backends.shift
+        @other_backends = backends
+      end
+
+      # Hashes
+      def get(key, expiry = Frivol::NEVER_EXPIRE)
+        val = @primary_backend.get(key)
+        val = migrate(key, expiry) if val.nil?
+        val
+      end
+
+      def set(key, val, expiry = Frivol::NEVER_EXPIRE)
+        @other_backends.each { |be| be.del(key) }
+        @primary_backend.set(key, val, expiry)
+      end
+
+      def del(key)
+        @primary_backend.del(key)
+        @other_backends.each { |be| be.del(key) }
+      end
+
+      def exists(key)
+        @primary_backend.exists(key) ||
+          @other_backends.detect { |be| be.exists(key) }
+      end
+
+      # Counters
+      def getc(key, expiry = Frivol::NEVER_EXPIRE)
+        val = @primary_backend.getc(key)
+        val = migratec(key, :getc, 0, expiry) if val.nil?
+        val
+      end
+
+      def setc(key, val, expiry = Frivol::NEVER_EXPIRE)
+        @other_backends.each { |be| be.delc(key) }
+        @primary_backend.setc(key, val, expiry)
+      end
+
+      def delc(key)
+        @primary_backend.delc(key)
+        @other_backends.each { |be| be.delc(key) }
+      end
+
+      def existsc(key)
+        @primary_backend.existsc(key) ||
+          @other_backends.detect { |be| be.existsc(key) }
+      end
+
+      def incr(key, expiry = Frivol::NEVER_EXPIRE)
+        if @primary_backend.existsc(key)
+          @primary_backend.incr(key)
+        else
+          migratec(key, :incrby, 1, expiry)
+        end
+      end
+
+      def decr(key, expiry = Frivol::NEVER_EXPIRE)
+        if @primary_backend.existsc(key)
+          @primary_backend.decr(key)
+        else
+          migratec(key, :decrby, 1, expiry)
+        end
+      end
+
+      def incrby(key, amt, expiry = Frivol::NEVER_EXPIRE)
+        if @primary_backend.existsc(key)
+          @primary_backend.incrby(key, amt)
+        else
+          migratec(key, :incrby, amt, expiry)
+        end
+      end
+
+      def decrby(key, amt, expiry = Frivol::NEVER_EXPIRE)
+        if @primary_backend.existsc(key)
+          @primary_backend.decrby(key, amt)
+        else
+          migratec(key, :decrby, amt, expiry)
+        end
+      end
+
+      # Expiry/TTL
+      def expire(key, ttl)
+        @primary_backend.expire(key, ttl)
+      end
+
+      def ttl(key)
+        expiry = @primary_backend.ttl(key)
+        if expiry.nil?
+          @other_backends.each do |be|
+            expiry = be.ttl(key)
+            return expiry unless expiry.nil?
+          end
+          nil
+        else
+          expiry
+        end
+      end
+
+      # Connection
+      def flushdb
+        @primary_backend.flushdb
+        @other_backends.each { |be| be.flushdb }
+      end
+
+      def connection
+        @primary_backend.connection
+      end
+
+      # Migration
+      def migrate(key, expiry = Frivol::NEVER_EXPIRE)
+        backend = @other_backends.detect { |be| be.exists(key) }
+        if backend
+          val = backend.get(key)
+          ttl = backend.ttl(key)
+          @primary_backend.set(key, val, ttl)
+          @other_backends.each { |be| be.del key }
+          val
+        end
+      end
+
+      def migratec(key, method = :incrby, amt = 0, expiry = Frivol::NEVER_EXPIRE)
+        backend = @other_backends.detect { |be| be.existsc(key) }
+        if backend
+          val = backend.getc(key).to_i
+          ttl = backend.ttl(key)
+          @primary_backend.incrby(key, val) unless val.zero?
+          val = @primary_backend.send(method, key, amt) unless amt.zero? || method == :getc
+          @primary_backend.expire(key, ttl) if ttl
+          @other_backends.each { |be| be.delc key }
+          val
+        elsif method != :getc
+          @primary_backend.send(method, key, amt, expiry)
+        else
+          nil
+        end
+      end
+    end
+  end
+end