redis-rdb 0.1.0

data/.gitignore

@@ -0,0 +1,2 @@
+*.gem
+*.aof

data/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2012 Daniele Alessandri
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,91 @@
+# redis-rdb #
+
+This library provides a set of modules and classes that make it easy to handle binary
+database dumps generated by [Redis](http://redis.io) (.rdb files) in Ruby.
+
+Currently redis-rdb allows developers to read .rdb files in a streamable flashion
+with `RDB::Reader` by providing a set of callbacks. Database objects can optionally
+be filtered by database / key / type using custom filters.
+
+```ruby
+require 'rdb'
+
+class MyCallbacks
+  include RDB::ReaderCallbacks
+
+  KEY_SELECTOR = Regexp.compile(/user:\d+/)
+
+  def accept_key?(state)
+    state.database == 15 && KEY_SELECTOR.match(state.key)
+  end
+
+  def set(key, value, state)
+    puts "SET \"#{key}\" \"#{value}\""
+  end
+end
+
+RDB::Reader.read_file('dump.rdb', callbacks: MyCallbacks.new)
+```
+
+For more details about the supported callbacks you can take a look at the source code in
+[lib/rdb/callbacks.rb](https://github.com/nrk/redis-rdb/blob/master/lib/rdb/callbacks.rb).
+
+### Data dumpers ###
+
+The `RDB::Dumper` module can be used to create classes that dump the data read from an .rdb file
+into a new file using a different format. An example would be to create an AOF file for Redis or
+to store the data into JSON or CSV. This is an example of using `RDB::Dumpers::AOF`:
+
+```ruby
+require 'rdb'
+
+source = 'test/rdb/database_multiple_logical_dbs.rdb'
+destination = File.basename(source, '.rdb') + '.aof'
+
+dumper = RDB::Dumpers::AOF.new(source, destination)
+dumper.dump
+```
+
+A dumper really is no more than a slightly augmented version of a class defining the callbacks
+for `RDB::Reader` with a few additional helper methods. You can find more about how to implement
+dumpers in [lib/rdb/dumper.rb](https://github.com/nrk/redis-rdb/blob/master/lib/rdb/dumper.rb)
+and [lib/rdb/dumpers/aof.rb](https://github.com/nrk/redis-rdb/blob/master/lib/rdb/dumpers/aof.rb).
+
+## RDB file format ##
+
+Right now there is still no official documentation about the binary format of .rdb files beside
+the code in [rdb.c](https://github.com/antirez/redis/blob/unstable/src/rdb.c) as the reference
+implementation.
+
+[An unofficial](https://github.com/sripathikrishnan/redis-rdb-tools/wiki/Redis-RDB-Dump-File-Format)
+but comprehensive description of the RDB format has been recently made available, but there is
+still no official documentation about it beside the actual implementation that can be found in
+[rdb.c](https://github.com/antirez/redis/blob/unstable/src/rdb.c).
+
+## Additional notes and credits ##
+
+Credit goes to [sripathikrishnan](https://github.com/sripathikrishnan) for his work on the
+[redis-rdb-tools](https://github.com/sripathikrishnan/redis-rdb-tools) Python library that
+proved to be quite an inspiration for the final design of `RDB::Reader`.We also reused the
+.rdb files shipped with his project for testing.
+
+## Dependencies ##
+- Ruby >= 1.9.0
+
+## Links ##
+
+### Project ###
+- [Source code](https://github.com/nrk/redis-rdb/)
+- [Issue tracker](https://github.com/nrk/redis-rdb/issues)
+
+### Related ###
+- [Redis](http://redis.io/)
+- [redis-rdb-tools](https://github.com/sripathikrishnan/redis-rdb-tools) (Python)
+
+## Author ##
+
+- [Daniele Alessandri](mailto:suppakilla@gmail.com) ([twitter](http://twitter.com/JoL1hAHN))
+
+## License ##
+
+The code for redis-rdb is distributed under the terms of the MIT license (see LICENSE).

data/Rakefile

@@ -0,0 +1,7 @@
+require 'cutest'
+
+task :test do
+  Cutest.run(Dir['test/test_*.rb'])
+end
+
+task :default => :test

data/examples/aof_dumper.rb

@@ -0,0 +1,8 @@
+$:.unshift File.expand_path('../lib', File.dirname(__FILE__))
+
+require 'rdb'
+
+source = 'test/rdb/database_multiple_logical_dbs.rdb'
+destination = File.basename(source, '.rdb') + '.aof'
+
+RDB::Dumpers::AOF.new(source, destination, variadic: true).dump

data/examples/read_rdb.rb

@@ -0,0 +1,9 @@
+$:.unshift File.expand_path('../lib', File.dirname(__FILE__))
+
+require 'rdb'
+
+options = {
+  callbacks: RDB::DebugCallbacks.new,
+}
+
+RDB::Reader.read_file('test/rdb/database_multiple_logical_dbs.rdb', options)

data/lib/rdb.rb

@@ -0,0 +1,10 @@
+require 'stringio'
+require 'rdb/version'
+require 'rdb/constants'
+require 'rdb/errors'
+require 'rdb/lzf'
+require 'rdb/callbacks'
+require 'rdb/reader-state'
+require 'rdb/reader'
+require 'rdb/dumper'
+require 'rdb/dumpers/aof'

data/lib/rdb/callbacks.rb

@@ -0,0 +1,129 @@
+module RDB
+  module ReaderCallbacks
+    def accept_key?(state)
+      true
+    end
+
+    def start_rdb(rdb_version); end
+
+    def end_rdb(); end
+
+    def start_database(database); end
+
+    def end_database(database); end
+
+    def pexpireat(key, expiration, state); end
+
+    def set(key, value, state); end
+
+    def start_list(key, length, state); end
+
+    def rpush(key, value, state); end
+
+    def end_list(key, state); end
+
+    def start_set(key, length, state); end
+
+    def sadd(key, value, state); end
+
+    def end_set(key, state); end
+
+    def start_sortedset(key, length, state); end
+
+    def zadd(key, score, value, state); end
+
+    def end_sortedset(key, state); end
+
+    def start_hash(key, length, state); end
+
+    def hset(key, field, value, state); end
+
+    def end_hash(key, state); end
+
+    def skip_object(key, state); end
+  end
+
+  class EmptyCallbacks
+    include ReaderCallbacks
+  end
+
+  class DebugCallbacks
+    include ReaderCallbacks
+
+    def start_rdb(version)
+      puts "Start RDB file - version #{version}"
+    end
+
+    def end_rdb()
+      puts "Close RDB file"
+    end
+
+    def start_database(database)
+      puts "Open database #{database}."
+    end
+
+    def end_database(database)
+      puts "Close database #{database}."
+    end
+
+    def pexpireat(key, expiration, state)
+      puts "PEXPIREAT \"#{key}\" \"#{expiration}\""
+    end
+
+    def set(key, value, state)
+      puts "SET \"#{key}\" \"#{value}\""
+    end
+
+    def start_list(key, length, state)
+      puts "Start list \"#{key}\" of #{length} items."
+    end
+
+    def rpush(key, value, state)
+      puts "RPUSH \"#{key}\" \"#{value}\""
+    end
+
+    def end_list(key, state)
+      puts "End list \"#{key}\"."
+    end
+
+    def start_set(key, length, state)
+      puts "Start set \"#{key}\" of #{length} members."
+    end
+
+    def sadd(key, value, state)
+      puts "SADD \"#{key}\" \"#{value}\""
+    end
+
+    def end_set(key, state)
+      puts "End set \"#{key}\"."
+    end
+
+    def start_sortedset(key, length, state)
+      puts "Start sortedset \"#{key}\" of #{length} members."
+    end
+
+    def zadd(key, score, value, state)
+      puts "ZADD \"#{key}\" \"#{score}\" \"#{value}\""
+    end
+
+    def end_sortedset(key, state)
+      puts "End sortedset \"#{key}\"."
+    end
+
+    def start_hash(key, length, state)
+      puts "Start hash \"#{key}\" of #{length} members."
+    end
+
+    def hset(key, field, value, state)
+      puts "HSET \"#{key}\" \"#{field}\" \"#{value}\""
+    end
+
+    def end_hash(key, state)
+      puts "End hash \"#{key}\"."
+    end
+
+    def skip_object(key, state)
+      puts "Skipping object for key #{key} of type #{state.type}"
+    end
+  end
+end

data/lib/rdb/constants.rb

@@ -0,0 +1,35 @@
+module RDB
+  module Length
+    BITS_6 = 0
+    BITS_14 = 1
+    BITS_32 = 2
+    ENCODED = 3
+  end
+
+  module Encoding
+    INT8 = 0
+    INT16 = 1
+    INT32 = 2
+    LZF = 3
+  end
+
+  module Opcode
+    EXPIRETIME_MS = 252
+    EXPIRETIME = 253
+    SELECTDB = 254
+    EOF = 255
+  end
+
+  module Type
+    STRING = 0
+    LIST = 1
+    SET = 2
+    ZSET = 3
+    HASH = 4
+    HASH_ZIPMAP = 9
+    LIST_ZIPLIST = 10
+    SET_INTSET = 11
+    ZSET_ZIPLIST = 12
+    HASH_ZIPLIST = 13
+  end
+end

data/lib/rdb/dumper.rb

@@ -0,0 +1,37 @@
+module RDB
+  module Dumper
+    include ReaderCallbacks
+
+    def initialize(source, destination, options = {})
+      @source = source
+      @destination = destination
+      @options = options
+      @output = nil
+    end
+
+    def <<(buffer)
+      @output << buffer unless @output.nil?; nil
+    end
+
+    def with_streams(&block)
+      input = open(@source, 'rb') unless @source.kind_of? IO
+      output = open(@destination, 'wb') unless @destination.kind_of? IO
+
+      begin
+        block.call(input, output)
+      ensure
+        input.close
+        output.close
+      end
+    end
+
+    def dump
+      raise RuntimeException, 'Output stream already opened' if @output
+
+      with_streams do |input, output|
+        @output = output
+        Reader.read(input, callbacks: self)
+      end
+    end
+  end
+end

data/lib/rdb/dumpers/aof.rb

@@ -0,0 +1,112 @@
+module RDB
+  module Dumpers
+    class AOF
+      include Dumper
+
+      REDIS_AOF_REWRITE_ITEMS_PER_CMD = 64
+
+      def start_database(database)
+        self << serialize_command(:select, [database])
+      end
+
+      def pexpireat(key, expiration, state)
+        command = if state.info[:precision] == :second
+          expiration = (expiration / 1000).to_i
+          :pexpire
+        else
+          :pexpireat
+        end
+        self << serialize_command(command, [key, expiration])
+      end
+
+      def set(key, value, state)
+        self << serialize_command(:set, [key, value])
+      end
+
+      def start_list(key, length, state)
+        reset_buffer(state)
+      end
+
+      def rpush(key, member, state)
+        handle(:rpush, state, key, member)
+      end
+
+      def end_list(key, state)
+        flush(:rpush, state)
+      end
+
+      def start_set(key, length, state)
+        reset_buffer(state)
+      end
+
+      def sadd(key, member, state)
+        handle(:sadd, state, key, member)
+      end
+
+      def end_set(key, state)
+        flush(:sadd, state)
+      end
+
+      def start_sortedset(key, length, state)
+        reset_buffer(state)
+      end
+
+      def zadd(key, score, member, state)
+        handle(:zadd, state, key, score, member)
+      end
+
+      def end_sortedset(key, state)
+        flush(:zadd, state)
+      end
+
+      def start_hash(key, length, state)
+        reset_buffer(state)
+      end
+
+      def hset(key, field, value, state)
+        handle(variadic? ? :hmset : :hset, state, key, field, value)
+      end
+
+      def end_hash(key, state)
+        flush(:hmset, state)
+      end
+
+      def handle(command, state, key, *arguments)
+        if variadic?
+          state.info[:buffer].push(arguments)
+          flush(command, state) if buffer_full?(state)
+        else
+          self << serialize_command(command, [key, *arguments])
+        end
+      end
+
+      def flush(command, state)
+        if buffer_some?(state)
+          self << serialize_command(command, [state.key] + state.info[:buffer].flatten)
+          reset_buffer(state)
+        end
+      end
+
+      def serialize_command(command, arguments)
+        buffer = "*#{arguments.length + 1}\r\n$#{command.length}\r\n#{command.upcase}\r\n"
+        buffer << arguments.map { |arg| "$#{arg.to_s.length}\r\n#{arg}\r\n" }.join
+      end
+
+      def variadic?
+        @options[:variadic] ||= false
+      end
+
+      def reset_buffer(state)
+        state.info[:buffer] = [];
+      end
+
+      def buffer_some?(state)
+        state.info[:buffer].length > 0
+      end
+
+      def buffer_full?(state)
+        state.info[:buffer].length == REDIS_AOF_REWRITE_ITEMS_PER_CMD
+      end
+    end
+  end
+end