redis-bitops 0.2

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: a567e3b9af870d0a1875760e5c02ea1562b1204b
+  data.tar.gz: 0d27e8d824da9cc252aa3058c5c0fcb90c1af965
+SHA512:
+  metadata.gz: 9a127af0eda591bd37ad31378275c52ea1cd8d4280fe836520e67d4ae423947836316583edffb4f2a0a153ba67fe4cc7c75c741b98bea862761ba2b4f92f731c
+  data.tar.gz: 56f170a119fff911cc250597887c4e49b6d5c9a784817315d02b80fbd70db129ca39ceaeb2a3d999bf5bba51066d3df705184a9ec1971a12084d215565ad89c1

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright 2014 Martin Bilski
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,193 @@
+# Introduction
+
+This gem makes it easier to do bit-wise operations on large Redis bitsets, usually called bitmaps, with a natural expression syntax. It also supports huge **sparse bitmaps** by storing data in multiple keys, called chunks, per bitmap.
+
+The typical use is real-time web analytics where each bit in a bitmap/bitset corresponds to a user ([introductory article here](http://blog.getspool.com/2011/11/29/fast-easy-realtime-metrics-using-redis-bitmaps/)). This library isn't an analytic package though, it's more low level than that and you can use it for anything. 
+
+**This library is under development and its interface might change.**
+
+
+# Quick start/pitch
+
+Why use the library?
+
+    require 'redis/bitops'
+    redis = Redis.new
+
+    b1 = redis.sparse_bitmap("b1")
+    b1[128_000_000] = true
+    b2 = redis.sparse_bitmap("b2")
+    b2[128_000_000] = true
+    result = b1 & b2
+
+Memory usage: about 20kb because it uses a sparse bitmap implementation using chunks of data.
+    
+Let's go crazy with super-complicated expressions:
+
+    ...
+    result = (b1 & ~b2) | b3 (| (b4 & b5 & b6 & ~b7))
+    
+Imagine writing this expression using Redis#bitop!
+
+
+# Installation
+
+To install the gem:
+
+    gem install redis-bitops
+    
+To use it in your code:
+
+    require 'redis/bitops'
+
+# Usage
+
+Reference: [here](http://rdoc.info/github/bilus/redis-bitops/master/frames)
+
+## Basic example
+
+An example is often better than theory so here's one. Let's create a few bitmaps and set their individual bits; we'll use those bitmaps in the examples below:
+
+    redis = Redis.new
+
+    a = redis.bitmap("a") 
+    b = redis.bitmap("b") 
+    result = redis.bitmap("result")
+        
+    b[0] = true; b[2] = true; b[7] = true # 10100001
+    a[0] = true; a[1] = true; a[7] = true # 11000001
+
+So, now here's a very simple expression:
+
+    c = a & b 
+    
+You may be surprised but the above statement does not query Redis at all! The expression is lazy-evaluated when you access the result:
+    
+    puts c.bitcount # => 2
+    puts c[0] # => true
+    puts c[1] # => false
+    puts c[2] # => false
+    puts c[7] # => false
+
+So, in the above example, the call to `c.bitcount` happens to be the first moment when Redis is queried. The result is stored under a temporary unique key.
+
+    puts c.root_key # => "redis:bitops:8eef38u9o09334"
+    
+Let's delete the temporary result:
+
+    c.delete!
+
+If you want to store the result directly under a specific key:
+
+    result << c
+
+Or, more adventurously, we can use the following more complex one-liner:
+
+    result << (~c & (a | b))
+    
+**Note: ** expressions are optimized by reducing the number of Redis commands and using as few temporary keys to hold intermediate values as possible. See below for details.
+
+
+## Sparse bitmaps
+
+### Usage
+
+You don't have to do anything special, simply use `Redis#sparse_bitmap` instead of `Redis#bitmap`:
+
+    a = redis.sparse_bitmap("a") 
+    b = redis.sparse_bitmap("b") 
+    result = redis.sparse_bitmap("result")
+        
+    b[0] = true; b[2] = true; b[7] = true # 10100001
+    a[0] = true; a[1] = true; a[7] = true # 11000001
+
+    c = a & b 
+    
+    result << c
+    
+or just:
+
+    result << (a & b)
+
+You can specify the chunk size (in bytes).
+
+Use the size consistently. Note that it cannot be re-adjusted for data already saved to Redis:
+
+    x = redis.sparse_bitmap("x", 1024 * 1024) # 1 MB per chunk.
+    x[0] = true
+    x[1000] = true
+
+**Important:** Do not mix sparse bitmaps with regular ones and never mix sparse bitmaps with different chunk sizes in the same expressions.
+
+### Rationale
+
+If you want to store a lot of huge but sparse bitsets, with not many bits set, using regular Redis bitmaps doesn't work very well. It wastes a lot of space. In analytics, it's a reasonable requirement, to be able to store data about several million users. A bitmap for 10 million users weights over 1MB! Imagine storing hourly statistics and using up memory at a rate of 720MB per month. 
+
+For, say, 100 million users it becomes outright prohibitive!
+
+But even with a fairly popular websites, I dare say, you don't often have one million users per hour :) This means that the majority of those bits is never sets and a lot of space goes wasted.
+
+Enter sparse bitmaps. They divide each bitmap into chunks thus minimizing memory use (chunks' size can be configured, see Configuration below). 
+
+Creating and using sparse bitmaps is identical to using regular bitmaps:
+
+    huge = redis.sparse_bitmap("huge_bitmap")
+    huge[128_000_000] = true
+
+The only difference in the above example is that it will allocate two 32kb chunks as opposed to 1MB that would be allocated if we used a regular bitmap (Redis#bitmap). In addition, setting the bit is nearly instantaneous.
+
+Compare:
+
+    puts Benchmark.measure {
+      sparse = redis.sparse_bitmap("huge_sparse_bitmap")
+      sparse[500_000_000] = true      
+    }
+    
+which on my machine this generates:   
+    
+    0.000000   0.000000   0.000000 (  0.000366)
+
+It uses just 23kb memory as opposed to 120MB (megabytes!) to store the bit using a regular Redis bitmap:
+
+      regular = redis.bitmap("huge_regular_bitmap")
+      regular[500_000_000] = true      
+
+## Configuration
+
+Here's how to configure the gem:
+
+    Redis::Bitops.configure do |config|
+      config.default_bytes_per_chunk = 8096 # Eight kilobytes.
+      config.transaction_level = :bitmap # allowed values: :bitmap or :none.
+    end
+
+# Implementation & efficiency
+
+## Optimization phase
+
+Prior to evaluation, the expression is optimized by combining operators into single BITOP commands and reusing temporary keys (required to store intermediate results) as much as possible.
+
+This silly example:
+
+    result << (a & b & c | a | b)
+    
+translates into simply:
+
+    BITOP AND result a b c
+    BITOP OR result result a b
+
+and doesn't create any temporary keys at all!
+
+## Materialization phase
+
+At this point, the calculations are carried out and the result is saved under the destination key. Note that, for sparse bitmaps, multiple keys may be created.
+
+
+## Transaction levels
+
+TBD
+
+
+## Contributing/feedback
+
+Please send in your suggestions to [gyamtso@gmail.com](mailto:gyamtso@gmail.com). Pull requests, issues, comments are more than welcome.

data/lib/redis/bitops.rb

@@ -0,0 +1,26 @@
+require 'redis'
+require 'redis/bitops/queries/materialization_helpers'
+require 'redis/bitops/queries/tree_building_helpers'
+require 'redis/bitops/queries/lazy_evaluation'
+require 'redis/bitops/queries/binary_operator'
+require 'redis/bitops/queries/unary_operator'
+require 'redis/bitops/bitmap'
+require 'redis/bitops/sparse_bitmap'
+
+require 'redis/bitops/configuration'
+
+
+class Redis
+  
+  # Creates a new bitmap.
+  #
+  def bitmap(key)
+    Bitops::Bitmap.new(key, self)
+  end
+
+  # Creates a new sparse bitmap storing data in n chunks to conserve memory.
+  #
+  def sparse_bitmap(key, bytes_per_chunk = nil)
+    Bitops::SparseBitmap.new(key, self, bytes_per_chunk)
+  end
+end

data/lib/redis/bitops/bitmap.rb

@@ -0,0 +1,107 @@
+class Redis
+  module Bitops
+  
+    # A sparse bitmap using multiple key to store its data to save memory.
+    #
+    # Note: When adding new public methods, revise the LazyEvaluation module.
+    #
+    class Bitmap
+    
+      include Queries
+      include TreeBuildingHelpers # See for a list of supported operators.
+    
+      # Creates a new regular Redis bitmap stored in 'redis' under 'root_key'.
+      #
+      def initialize(root_key, redis)
+        @redis = redis
+        @root_key = root_key
+      end
+    
+      # Saves the result of the query in the bitmap.
+      #
+      def << (query)
+        query.evaluate(self)
+      end
+    
+      # Reads bit at position 'pos' returning a boolean.
+      #
+      def [] (pos)
+        i2b(@redis.getbit(key(pos), offset(pos)))
+      end
+    
+      # Sets bit at position 'pos' to 1 or 0 based on the boolean 'b'.
+      #
+      def []= (pos, b)
+        @redis.setbit(key(pos), offset(pos), b2i(b))
+      end
+    
+      # Returns the number of set bits.
+      #
+      def bitcount
+        @redis.bitcount(@root_key)
+      end
+    
+      # Deletes the bitmap and all its keys.
+      #
+      def delete!
+        @redis.del(@root_key)
+      end
+     
+      # Redis BITOP operator 'op' (one of :and, :or, :xor or :not) on operands
+      # (bitmaps). The result is stored in 'result'.
+      #
+      def bitop(op, *operands, result)
+        @redis.bitop(op, result.root_key, self.root_key, *operands.map(&:root_key))
+        result
+      end
+
+      # The key the bitmap is stored under.
+      #
+      def root_key
+        @root_key
+      end
+    
+      # Returns lambda creating Bitmap objects using @redis as the connection.
+      #
+      def bitmap_factory
+        lambda { |key| @redis.bitmap(key) }
+      end
+        
+      # Copy this bitmap to 'dest' bitmap.
+      #
+      def copy_to(dest)
+        copy(root_key, dest.root_key)
+      end
+        
+      protected
+    
+      def key(pos)
+        @root_key
+      end
+    
+      def offset(pos)
+        pos
+      end
+    
+      def b2i(b)
+        b ? 1 : 0
+      end
+    
+      def i2b(i)
+        i.to_i != 0 ? true : false
+      end
+    
+      COPY_SCRIPT = 
+        <<-EOS
+          redis.call("DEL", KEYS[2])
+          if redis.call("EXISTS", KEYS[1]) == 1 then
+            local val = redis.call("DUMP", KEYS[1])
+            redis.call("RESTORE", KEYS[2], 0, val)
+          end
+        EOS
+      def copy(source_key, dest_key)
+        @redis.eval(COPY_SCRIPT, [source_key, dest_key])      
+      end
+    end
+  end
+end

data/lib/redis/bitops/configuration.rb

@@ -0,0 +1,38 @@
+class Redis
+  module Bitops
+  
+    # Configurable settings.
+    #
+    class Configuration
+      
+      # Number of bytes per one sparse bitmap chunk.
+      #
+      attr_accessor :default_bytes_per_chunk
+      
+      # Granulatity of MULTI transactions. Currently supported values are :bitmap and nil.
+      #
+      attr_accessor :transaction_level
+    
+      def initialize
+        reset!
+      end
+
+      def reset!
+        @default_bytes_per_chunk = 32 * 1024
+        @transaction_level = :bitmap
+      end
+    end
+
+    extend self
+    attr_accessor :configuration
+  
+    # Call this method to modify defaults in your initializers.
+    #
+    def configure
+      self.configuration ||= Configuration.new
+      yield(configuration)
+    end
+  end
+
+  Bitops.configure {}
+end

data/lib/redis/bitops/queries/binary_operator.rb

@@ -0,0 +1,71 @@
+require 'securerandom'
+
+class Redis
+  module Bitops
+    module Queries
+    
+      # Binary bitwise operator.
+      #
+      class BinaryOperator
+        include MaterializationHelpers
+        include TreeBuildingHelpers
+        include LazyEvaluation
+     
+        # Creates a bitwise operator 'op' with left-hand operand, 'lhs', and right-hand operand, 'rhs'.
+        #
+        def initialize(op, lhs, rhs)
+          @args = [lhs, rhs]
+          @op = op
+        end
+      
+        # Runs the expression tree against the redis database, saving the results
+        # in bitmap 'dest'.
+        #
+        def materialize(dest)
+          # Resolve lhs and rhs operand, using 'dest' to store intermediate result so
+          # a maximum of one temporary Bitmap has to be created.
+          # Then apply the bitwise operator storing the final result in 'dest'.
+
+          intermediate = dest
+        
+          lhs, *other_args = @args
+          temp_intermediates = []
+        
+          # Side-effects: if a temp intermediate bitmap is created, it's added to 'temp_intermediates' 
+          # to be deleted in the "ensure" block. Marked with "<- SE".
+          
+          lhs_operand, intermediate = resolve_operand(lhs, intermediate, temp_intermediates) # <- SE
+          other_operands, *_ = other_args.inject([[], intermediate]) do |(operands, intermediate), arg|
+            operand, intermediate = resolve_operand(arg, intermediate, temp_intermediates) # <- SE
+            [operands << operand, intermediate]
+          end
+        
+          lhs_operand.bitop(@op, *other_operands, dest)
+        ensure
+          temp_intermediates.each(&:delete!)
+        end
+      
+        # Recursively optimizes the expression tree by combining operands for neighboring identical
+        # operators, so for instance a & b & c ultimately becomes BITOP :and dest a b c as opposed 
+        # to running two separate BITOP commands.
+        #
+        def optimize!(parent_op = nil)
+          @args.map! { |arg| arg.respond_to?(:optimize!) ? arg.optimize!(@op) : arg }.flatten!
+          if parent_op == @op
+            @args
+          else
+            self
+          end
+        end
+
+        # Finds the first bitmap factory  in the expression tree.
+        # Required by LazyEvaluation and MaterializationHelpers.
+        #
+        def bitmap_factory
+          arg = @args.find { |arg| arg.bitmap_factory } or raise "Internal error. Cannot find a bitmap factory."
+          arg.bitmap_factory
+        end
+      end
+    end
+  end
+end