rdkit 0.0.1 → 0.1.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: b7f8dab0976c2aaf2d8045793c9c52713a7ff867
-  data.tar.gz: 0cb9f68b9684fc6c2a5e553202a724396ae75d46
+  metadata.gz: 10a0bd157c17b8098e43a87046384ebc7e9ae534
+  data.tar.gz: 7ad179971146d74db65ffbc8a6999b1b9b8bb034
 SHA512:
-  metadata.gz: 2e1c7b165b6cc10f382afd74618dd5a5df2a1722e8bbad9a5eb69c2742449fe6f2ca6e39ec0a4b163574b4cce8ae8aaf78a45ccbafddd18e26dc7896e072956d
-  data.tar.gz: 2b6889a5764936fb592b0d29418aab64108c7cb53f1b22c2d2e87381a9259e2f8ab7f3f8f4e0e3366a5e554f1399d6a6218b8a28317ed5ef30f7b83ace5a964f
+  metadata.gz: f8fd1464e00be7ce386d4eefb1132984cbfb7632ff498eb7df6ddd97fd07127122f27b8828c1722ad22b515b606a686286fc5a5e550adc964c00409e7950aa62
+  data.tar.gz: 2d71bef8dfac46a9b7d4274ab755e83a3f944a05a401def43b1bda6932d69c1a2cef181a229a015faf5299d73d22361778502e9653c4e52df61d251572e304a6

data/.gitignore

@@ -8,3 +8,6 @@
 /spec/reports/
 /tmp/
 log
+.vagrant
+.env
+Gemfile.ci.lock

data/Gemfile

@@ -1,4 +1,13 @@
 source 'https://rubygems.org'
 
-# Specify your gem's dependencies in rdkit.gemspec
 gemspec
+
+group :development, :test do
+  gem 'pry'
+  gem 'coco'
+  gem 'timecop'
+  gem 'guard'
+  gem 'guard-rspec'
+end
+
+gem 'thread', '~> 0.2.1', github: 'meh/ruby-thread'

data/Gemfile.ci

@@ -0,0 +1,13 @@
+source 'http://ruby.taobao.org'
+
+gemspec
+
+group :development, :test do
+  gem 'pry'
+  gem 'coco'
+  gem 'timecop'
+  gem 'guard'
+  gem 'guard-rspec'
+end
+
+gem 'thread', '~> 0.2.1', github: 'meh/ruby-thread'

data/Guardfile

@@ -0,0 +1,40 @@
+# A sample Guardfile
+# More info at https://github.com/guard/guard#readme
+
+## Uncomment and set this to only include directories you want to watch
+# directories %w(app lib config test spec features)
+
+## Uncomment to clear the screen before every task
+# clearing :on
+
+## Guard internally checks for changes in the Guardfile and exits.
+## If you want Guard to automatically start up again, run guard in a
+## shell loop, e.g.:
+##
+##  $ while bundle exec guard; do echo "Restarting Guard..."; done
+##
+## Note: if you are using the `directories` clause above and you are not
+## watching the project directory ('.'), then you will want to move
+## the Guardfile to a watched dir and symlink it back, e.g.
+#
+#  $ mkdir config
+#  $ mv Guardfile config/
+#  $ ln -s config/Guardfile .
+#
+# and, you'll have to watch "config/Guardfile" instead of "Guardfile"
+
+# Note: The cmd option is now required due to the increasing number of ways
+#       rspec may be run, below are examples of the most common uses.
+#  * bundler: 'bundle exec rspec'
+#  * bundler binstubs: 'bin/rspec'
+#  * spring: 'bin/rspec' (This will use spring if running and you have
+#                          installed the spring binstubs per the docs)
+#  * zeus: 'zeus rspec' (requires the server to be started separately)
+#  * 'just' rspec: 'rspec'
+
+guard :rspec, cmd: "bundle exec rspec" do
+  watch(%r{^spec/.+_spec\.rb$})
+  watch(%r{^lib/(.+)\.rb$})       { |m| "spec/#{m[1]}_spec.rb" }
+
+  watch('spec/spec_helper.rb')    { "spec" }
+end

data/README.md

@@ -1,6 +1,6 @@
 # RDKit
 
-RDKit is a simple toolkit to write Redis-like, single-threaded multiplexing-IO server.
+`RDKit` is a simple toolkit to write Redis-like, single-threaded multiplexing-IO server.
 
 The server speaks [Redis RESP protocol](http://redis.io/topics/protocol), so you can reuse many Redis-compatible clients and tools such as:
 
@@ -10,11 +10,17 @@ The server speaks [Redis RESP protocol](http://redis.io/topics/protocol), so you
 
 And a lot more.
 
-`RDKit` is used to power the [520 Love Radio](http://s.weibo.com/weibo/same%2520%25E7%2594%25B5%25E5%258F%25B0) service of [same.com](http://same.com)
+`RDKit` is used to power:
+
+- [520 Love Radio](http://s.weibo.com/weibo/same%2520%25E7%2594%25B5%25E5%258F%25B0) service of [same.com](http://same.com)
+- AntiSpam blacklisted photo filtering service used at [same.com](http://same.com) (BK-Tree + pHash)
+- channel unread count service at [same.com](http://same.com)
 
 [![Code Climate](https://codeclimate.com/github/forresty/rdkit/badges/gpa.svg)](https://codeclimate.com/github/forresty/rdkit)
 [![Build Status](https://travis-ci.org/forresty/rdkit.svg?branch=master)](https://travis-ci.org/forresty/rdkit)
 
+`RDKit` should work without problem on `MRI` 2.2+, may encounter bugs on earlier version of `MRI` or `JRuby` or `Rubinus`, in that case, please kindly open an issue on GitHub
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -33,9 +39,100 @@ Or install it yourself as:
 
 ## Usage
 
-see examples under `example` folder.
+Generally, you should implement one subclass for each of the 3 classes: `RDKit::RESPResponder`, `RDKit::Core` and `RDKit::Server`, and spawn one object for each class.
+
+Your server object should have two instance variables `@responder` and `@core` pointed to your spawned instances.
+
+### RDKit::Server
+
+```ruby
+class YourServer < RDKit::Server
+  def initialize
+    super('0.0.0.0', 3721)
+
+    @core = YourCore.new
+    @responder = YourResponder.new(core)
+  end
+end
+
+server = YourServer.new
+
+trap(:INT) { server.stop }
+
+server.start
+```
+
+This will start a `TCPServer` on `0.0.0.0:3721` and stops when you `CTRL-C`.
+
+### RDKit::RESPResponder
+
+`@responder` maps Redis commands to its methods and arguments, for example `info` will be sent to `RESPResponder#info`, and `info all` to `RESPResponder#info` with `"all"` as its first argument.
+
+The return ruby object of each method will be marshaled as RESP strings, for example `'OK'` becomes `"+OK\r\n"`.
+
+For example, with following implementation in your `RESPResponder` subclass:
+
+```ruby
+def add(a, b)
+  a.to_i + b.to_i
+end
+```
+
+You implemented an adder using RDKit! See it in action:
+
+```shell
+$ redis-cli -p 3721
+127.0.0.1:3721> add 1 2
+(integer) 3
+127.0.0.1:3721> add 5
+(error) ERR wrong number of arguments for 'add' command
+127.0.0.1:3721>
+```
+
+The detailed algorithm can be found in `resp.rb`, at the time of writing it is like this:
+
+```ruby
+def compose(data)
+  case data
+  when *%w{ OK string list set hash zset none }
+    "+#{data}\r\n"
+  when true
+    ":1\r\n"
+  when false
+    ":0\r\n"
+  when Integer
+    ":#{data}\r\n"
+  when Array
+    "*#{data.size}\r\n" + data.map { |i| compose(i) }.join
+  when NilClass
+    # Null Bulk String, not Null Array of "*-1\r\n"
+    "$-1\r\n"
+  when WrongTypeError
+    "-WRONGTYPE #{data.message}\r\n"
+  when StandardError
+    "-ERR #{data.message}\r\n"
+  else
+    # always Bulk String
+    "$#{data.bytesize}\r\n#{data}\r\n"
+  end
+end
+```
+
+### RDKit::Core
+
+You are required to implement a `tick!` method. `RDKit` will call it periodically (currently roughly every 0.1 sec), this gives you a chance to do some house-keeping. For example:
+
+```ruby
+def tick!
+  save_non_critical_data! if server.cycles % 1000 == 0
+end
+```
+
+### Examples
+
+See examples under `example` folder.
 
-### Implementing a counter server
+#### Implementing a counter server
 
 A simple counter server source code listing:
 
@@ -123,7 +220,7 @@ server.start
 
 ```
 
-### Connect using `redis-cli`
+#### Connect using `redis-cli`
 
 ```shell
 $ redis-cli -p 3721
@@ -166,7 +263,9 @@ total_commands_processed:6
 (error) ERR unknown command 'xx'
 ```
 
-### Benchmarking with `redis-benchmark`
+Hint: if you are adventurous, try `info all`
+
+#### Benchmarking with `redis-benchmark`
 
 ```shell
 $ redis-benchmark -p 3721 incr
@@ -196,6 +295,183 @@ Since it is single-threaded, the count will be correct:
 (integer) 10000
 ```
 
+#### Implementing blocked commands
+
+Some commands will be blocking: they may either depend on external services or need some background tasks to be run.
+
+The clients will expect those commands to be blocking calls, they will not return until the commands are finished, but we don't want the server to be blocked as well.
+
+Therefore we introduce `Server#blocking` methods, execution wrapped in this method call will be run in a background thread pool, and the client will be on hold until that task is finished.
+
+Example: see `examples/blocking` folder.
+
+```ruby
+# blocking/command_runner.rb
+
+module Blocking
+  class CommandRunner < RDKit::RESPRunner
+    attr_reader :core
+
+    def initialize(core)
+      @core = core
+    end
+
+    def block_with_callback
+      core.block_with_callback
+
+      # this is ignored, instead `on_success` block of `core.block_with_callback` is evaluated and returned
+      'OK'
+    end
+
+    def block
+      core.block
+
+      'OK'
+    end
+
+    def nonblock
+      core.nonblock
+
+      'OK'
+    end
+  end
+end
+
+# blocking/core.rb
+
+module Blocking
+  class Core < RDKit::Core
+    def block_with_callback
+      on_success = lambda { 'success' }
+
+      server.blocking(on_success) { do_something }
+    end
+
+    def block
+      server.blocking { do_something }
+    end
+
+    def nonblock
+      do_something
+    end
+
+    def do_something
+      sleep 1
+    end
+
+    def tick!
+    end
+  end
+end
+```
+
+Running:
+
+```shell
+$ redis-cli -p 3721
+127.0.0.1:3721> block
+OK
+(1.03s)
+127.0.0.1:3721> nonblock
+OK
+(1.01s)
+127.0.0.1:3721> block_with_callback
+"success"
+(1.02s)
+```
+
+Benchmarking:
+
+```shell
+$ redis-benchmark -p 3721 -n 10 block
+====== block ======
+  10 requests completed in 1.03 seconds
+  50 parallel clients
+  3 bytes payload
+  keep alive: 1
+
+10.00% <= 1027 milliseconds
+100.00% <= 1027 milliseconds
+9.73 requests per second
+
+$ redis-benchmark -p 3721 -n 10 nonblock
+====== nonblock ======
+  10 requests completed in 10.04 seconds
+  50 parallel clients
+  3 bytes payload
+  keep alive: 1
+
+10.00% <= 1001 milliseconds
+20.00% <= 2005 milliseconds
+30.00% <= 3010 milliseconds
+40.00% <= 4013 milliseconds
+50.00% <= 5018 milliseconds
+60.00% <= 6022 milliseconds
+70.00% <= 7027 milliseconds
+80.00% <= 8030 milliseconds
+90.00% <= 9034 milliseconds
+100.00% <= 10039 milliseconds
+1.00 requests per second
+
+```
+
+See the difference between blocking and non-blocking commands?
+
+### Implemented Redis Commands
+
+| command     | support                              | note                                         |
+|-------------|--------------------------------------|----------------------------------------------|
+| `info`      | full                                 | additional `objspace` and `gc` commands      |
+| `ping`      | full                                 |                                              |
+| `echo`      | full                                 |                                              |
+| `time`      | full                                 |                                              |
+| `select`    | partial/compatible                   | `redis-benchmark` requires `select` command  |
+| `config`    | `get`, `set`, `resetstat`            |                                              |
+| `slowlog`   | full                                 |                                              |
+| `client`    | `getname`, `setname`, `list`, `kill` | `kill` filter only supports `id`, `addr`     |
+| `monitor`   | full                                 |                                              |
+| `debug`     | `sleep`, `segfault`                  |                                              |
+| `shutdown`  | full                                 |                                              |
+| `get`       | full                                 |                                              |
+| `set`       | without options                      |                                              |
+| `del`       | full                                 |                                              |
+| `keys`      | without pattern (return all)         |                                              |
+| `lpush`     | full                                 |                                              |
+| `lpop`      | full                                 |                                              |
+| `rpop`      | full                                 |                                              |
+| `llen`      | full                                 |                                              |
+| `lrange`    | partial (not fully tested)           |                                              |
+| `exists`    | full                                 |                                              |
+| `flushdb`   | full                                 |                                              |
+| `flushall`  | full                                 |                                              |
+| `mget`      | full                                 |                                              |
+| `mset`      | full                                 |                                              |
+| `strlen`    | full                                 |                                              |
+| `sadd`      | full                                 |                                              |
+| `scard`     | full                                 |                                              |
+| `smembers`  | full                                 |                                              |
+| `sismember` | full                                 |                                              |
+| `srem`      | full                                 |                                              |
+| `hset`      | full                                 |                                              |
+| `hget`      | full                                 |                                              |
+| `hexists`   | full                                 |                                              |
+| `hlen`      | full                                 |                                              |
+| `hstrlen`   | full                                 |                                              |
+| `hdel`      | full                                 |                                              |
+| `hkeys`     | full                                 |                                              |
+| `hvals`     | full                                 |                                              |
+| `setnx`     | full                                 |                                              |
+| `getset`    | full                                 |                                              |
+
+
+### Implemented Additional Commands
+
+| command     | description                                                                |
+|-------------|----------------------------------------------------------------------------|
+| `gc`        | start garbage collection immediately                                       |
+| `heapdump`  | `ObjectSpace.dump_all` to ./tmp                                            |
+
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `bin/console` for an interactive prompt that will allow you to experiment.

data/Vagrantfile

@@ -0,0 +1,13 @@
+VAGRANTFILE_API_VERSION = "2"
+
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+  config.vm.box = "ubuntu/trusty64"
+
+  config.vm.define 'redis' do |c|
+    config.vm.provision "shell", inline: "sudo apt-get install redis-tools -y"
+
+    c.vm.provision "docker" do |docker|
+      docker.run "redis", args: "-p 6379:6379"
+    end
+  end
+end

data/example/blocking.rb

@@ -0,0 +1,10 @@
+require 'rdkit'
+
+require_relative 'blocking/version'
+require_relative 'blocking/core'
+require_relative 'blocking/command_runner'
+require_relative 'blocking/server'
+
+server = Blocking::Server.new
+
+server.start