memcache 1.2.0 → 1.2.1
Sign up to get free protection for your applications and to get access to all the features.
- data/.gitignore +12 -1
- data/README.rdoc +65 -11
- data/Rakefile +14 -0
- data/VERSION.yml +2 -2
- data/bench/benchmark.rb +126 -0
- data/ext/extconf.rb +51 -2
- data/ext/libmemcached-0.38.tar.gz +0 -0
- data/ext/native_server.c +575 -0
- data/lib/memcache/base.rb +65 -0
- data/lib/memcache/local_server.rb +7 -53
- data/lib/memcache/migration.rb +7 -6
- data/lib/memcache/null_server.rb +1 -1
- data/lib/memcache/pg_server.rb +37 -20
- data/lib/memcache/{segmented_server.rb → segmented.rb} +25 -15
- data/lib/memcache/server.rb +44 -56
- data/lib/memcache.rb +131 -83
- data/test/memcache_local_server_test.rb +1 -1
- data/test/memcache_native_server_test.rb +31 -0
- data/test/memcache_null_server_test.rb +16 -8
- data/test/memcache_segmented_native_server_test.rb +18 -0
- data/test/memcache_segmented_server_test.rb +6 -9
- data/test/memcache_server_test.rb +9 -11
- data/test/memcache_server_test_helper.rb +89 -49
- data/test/memcache_test.rb +11 -19
- data/test/test_helper.rb +34 -9
- metadata +12 -4
- data/memcache.gemspec +0 -69
data/.gitignore
CHANGED
data/README.rdoc
CHANGED
@@ -2,8 +2,8 @@
|
|
2
2
|
|
3
3
|
This is the Geni memcached client. It started out as a fork of fiveruns/memcache-client,
|
4
4
|
which was a fork of seattle.rb's memcache-client, but over time, our client has diverged,
|
5
|
-
and I've rewritten
|
6
|
-
served as a starting point for this code.
|
5
|
+
and I've rewritten the majority of the code. Of course, a lot of credit is due to those
|
6
|
+
whose code served as a starting point for this code.
|
7
7
|
|
8
8
|
== Usage
|
9
9
|
|
@@ -16,11 +16,11 @@ served as a starting point for this code.
|
|
16
16
|
cache['things']
|
17
17
|
=> {:foo => "1", :bar => [1,2,3]}
|
18
18
|
|
19
|
-
== How is this different from memcache
|
19
|
+
== How is this different from other memcache clients?
|
20
20
|
|
21
|
-
Like memcache-client, _memcache_ (shown in italics when I am
|
22
|
-
library) is a memcached client, but it differs significantly from
|
23
|
-
several important ways.
|
21
|
+
Like fiveruns/memcache-client and fauna/memcached, _memcache_ (shown in italics when I am
|
22
|
+
referring to this library) is a memcached client, but it differs significantly from
|
23
|
+
these clients in several important ways.
|
24
24
|
|
25
25
|
=== Interface
|
26
26
|
|
@@ -80,8 +80,8 @@ changes:
|
|
80
80
|
cache.add('foo', 0)
|
81
81
|
cache.get('foo')
|
82
82
|
=> 1
|
83
|
-
|
84
|
-
cache.replace('foo', 2)
|
83
|
+
|
84
|
+
cache.replace('foo', 2)
|
85
85
|
cache.get('foo')
|
86
86
|
=> 2
|
87
87
|
|
@@ -111,13 +111,13 @@ changes:
|
|
111
111
|
|
112
112
|
The underlying architechture of _memcache_ is more modular than memcache-client.
|
113
113
|
A given +Memcache+ instance has a group of servers, just like before, but much more of the
|
114
|
-
functionality
|
114
|
+
functionality is encapsulated inside the <tt>Memcache::Server</tt> object. Really, a +Server+
|
115
115
|
object is a thin wrapper around an remote memcached server that takes care of the socket
|
116
116
|
and protocol details along with basic error handling. The +Memcache+ class handles the
|
117
117
|
partitioning algorithm, marshaling of ruby objects and various higher-level methods.
|
118
118
|
|
119
119
|
By encapsulating the protocol inside the +Server+ object, it becomes very easy to plug-in
|
120
|
-
alternate server implementations. Right now, there are
|
120
|
+
alternate backend server implementations. Right now, there are three basic, alternate servers:
|
121
121
|
|
122
122
|
[+LocalServer+] This is an in-process server for storing keys and values in local
|
123
123
|
memory. It is good for testing, when you don't want to spin up an instance
|
@@ -131,6 +131,9 @@ alternate server implementations. Right now, there are two basic, alternate serv
|
|
131
131
|
are very large. It can also be used in a multi-level cache setup with
|
132
132
|
<tt>Memcache::Server</tt> to provide persistence without sacrificing speed.
|
133
133
|
|
134
|
+
[+NativeServer+] This implementation uses native bindings to libmemcached. It is described
|
135
|
+
in more detail in the "Native Bindings" section below.
|
136
|
+
|
134
137
|
=== Very Large Values
|
135
138
|
|
136
139
|
Memcached limits the size of values to 1MB. This is done to reduce memory usage, but it
|
@@ -170,10 +173,61 @@ finally marking a server as dead and raising an exception. We will not attempt t
|
|
170
173
|
from a dead server for 5 seconds, but a write will always attempt to revive a dead server
|
171
174
|
by attempting to connect.
|
172
175
|
|
176
|
+
=== Keys, Namespaces, and Prefixes
|
177
|
+
|
178
|
+
Unlike the other ruby memcache clients, keys in _memcache_ can contain spaces. This is
|
179
|
+
possible because the backend transparently enscapes all space characters, and is
|
180
|
+
especially important if you are using method_cache[http://github.com/ninjudd/method_cache]
|
181
|
+
or record_cache[http://github.com/ninjudd/record_cache]. <tt>Memcache::Server</tt> implements
|
182
|
+
this escaping using gsub and it adds a slight performance penalty when escaping is
|
183
|
+
necessary. +NativeServer+ implements this escaping directly in C, and the performance
|
184
|
+
overhead is negligible.
|
185
|
+
|
186
|
+
You can also partition your keys into different namespaces for convenience. This is done
|
187
|
+
by prefixing all keys in the backend server with "namespace:". However, the hash keys
|
188
|
+
returned by multi gets do not contain the prefix. In this way, the namespace can be
|
189
|
+
totally transparent to your code. You can also determine whether the prefix is used for
|
190
|
+
hashing with the following option:
|
191
|
+
|
192
|
+
[+hash_with_prefix+] Determines whether the prefix/namespace is used when hashing keys to
|
193
|
+
determine which server to use. Defaults to true.
|
194
|
+
|
195
|
+
== Native Bindings
|
196
|
+
|
197
|
+
The <tt>Memcache::NativeServer</tt> backend provides native bindings to libmecached. This is
|
198
|
+
significantly faster than using <tt>Memcache::Server</tt> as demonstrated by runnning
|
199
|
+
bench/benchmark.rb. NativeServer encapsulates a set of remote servers and allows you to
|
200
|
+
use the various hashing methods in libmemcached.
|
201
|
+
|
202
|
+
You can use native bindings either by passing +NativeServer+ objects to +Memcache+, or you
|
203
|
+
can use the +native+ option. Native bindings are compatible with segmented values through
|
204
|
+
the +SegmentedNativeServer+ object or by combining the +native+ option with
|
205
|
+
+segment_large_values+.
|
206
|
+
|
207
|
+
server = Memcache::NativeServer.new(:servers => ['localhost:11211', 'localhost:11212'])
|
208
|
+
cache = Memcache.new(:server => server)
|
209
|
+
|
210
|
+
cache = Memcache.new(:servers => ['localhost:11211', 'localhost:11212'], :native => true)
|
211
|
+
|
212
|
+
NativeServer also accepts a few other options:
|
213
|
+
|
214
|
+
[+hash+] The libmemcached hashing method. See http://docs.tangent.org/libmemcached/index.html
|
215
|
+
for more detail. One of:
|
216
|
+
|
217
|
+
<tt>:default :md5 :crc :fnv1_64 :fnv1a_64 :fnv1_32 :fnv1a_32 :jenkins
|
218
|
+
:hsieh :murmur</tt>.
|
219
|
+
|
220
|
+
NOTE: Even though there is a libmemcached method named <tt>:default</tt> (which
|
221
|
+
is actually Jenkins's one-at-a-time hash), the default hashing method if you
|
222
|
+
don't specify one is <tt>:crc</tt>.
|
223
|
+
|
224
|
+
[+binary+] A boolean value specifying whether to use memcached's binary protocol instead
|
225
|
+
of the default ascii protocol. This is slightly slower, but should allow you to use unicode keys.
|
226
|
+
|
173
227
|
== Installation
|
174
228
|
|
175
229
|
$ sudo gem install memcache --source http://gemcutter.org
|
176
230
|
|
177
231
|
== License:
|
178
232
|
|
179
|
-
Copyright (c)
|
233
|
+
Copyright (c) 2010 Justin Balthrop, Geni.com; Published under The MIT License, see LICENSE
|
data/Rakefile
CHANGED
@@ -41,6 +41,10 @@ task :test => :check_dependencies
|
|
41
41
|
|
42
42
|
task :default => :test
|
43
43
|
|
44
|
+
task :clean do
|
45
|
+
`rm -rf ext/lib ext/bin ext/share ext/include`
|
46
|
+
end
|
47
|
+
|
44
48
|
require 'rake/rdoctask'
|
45
49
|
Rake::RDocTask.new do |rdoc|
|
46
50
|
if File.exist?('VERSION')
|
@@ -54,3 +58,13 @@ Rake::RDocTask.new do |rdoc|
|
|
54
58
|
rdoc.rdoc_files.include('README*')
|
55
59
|
rdoc.rdoc_files.include('lib/**/*.rb')
|
56
60
|
end
|
61
|
+
|
62
|
+
namespace :test do
|
63
|
+
Rake::TestTask.new(:native) do |t|
|
64
|
+
`cd ext && make && cp native_server.bundle native_server.o ../lib/memcache/`
|
65
|
+
t.libs << 'test'
|
66
|
+
t.pattern = 'test/memcache_native_server_test.rb'
|
67
|
+
t.verbose
|
68
|
+
end
|
69
|
+
end
|
70
|
+
|
data/VERSION.yml
CHANGED
data/bench/benchmark.rb
ADDED
@@ -0,0 +1,126 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'benchmark'
|
3
|
+
require 'pp'
|
4
|
+
|
5
|
+
$:.unshift(File.dirname(__FILE__) + '/../lib')
|
6
|
+
require 'memcache'
|
7
|
+
|
8
|
+
def array(*args)
|
9
|
+
arr = []
|
10
|
+
args.each do |arg|
|
11
|
+
if arg.kind_of?(Enumerable)
|
12
|
+
arr.concat(arg.to_a)
|
13
|
+
else
|
14
|
+
arr << arg
|
15
|
+
end
|
16
|
+
end
|
17
|
+
arr
|
18
|
+
end
|
19
|
+
|
20
|
+
CHARS = array('a'..'z', 'A'..'Z', '0'..'9', '_', '+', '-')
|
21
|
+
def rand_string(len)
|
22
|
+
str = ''
|
23
|
+
len.times do
|
24
|
+
str << pick_rand(CHARS)
|
25
|
+
end
|
26
|
+
str
|
27
|
+
end
|
28
|
+
|
29
|
+
def pick_rand(items)
|
30
|
+
i = rand(items.size)
|
31
|
+
items[i]
|
32
|
+
end
|
33
|
+
|
34
|
+
def pick_mod(items, i)
|
35
|
+
i = i % items.size
|
36
|
+
items[i]
|
37
|
+
end
|
38
|
+
|
39
|
+
class MemcacheBench
|
40
|
+
attr_reader :num_items, :key_length, :val_length, :keys, :vals, :n
|
41
|
+
|
42
|
+
def initialize(opts = {})
|
43
|
+
@n = opts[:n] || 100_000
|
44
|
+
@num_items = opts[:num_items] || 5000
|
45
|
+
@key_length = array(opts[:key_length] || 10)
|
46
|
+
@val_length = array(opts[:val_length] || 100)
|
47
|
+
|
48
|
+
puts "N = #{@n}"
|
49
|
+
puts "key_length: #{@key_length.join(' or ')}"
|
50
|
+
puts "val_length: #{@val_length.join(' or ')}"
|
51
|
+
puts "Generating #{@num_items} random keys and values..."
|
52
|
+
@keys = []
|
53
|
+
@vals = []
|
54
|
+
@num_items.times do
|
55
|
+
@keys << rand_string( pick_rand(@key_length) )
|
56
|
+
@vals << rand_string( pick_rand(@val_length) )
|
57
|
+
end
|
58
|
+
|
59
|
+
Benchmark.bm(36) do |x|
|
60
|
+
@bench = x
|
61
|
+
end
|
62
|
+
end
|
63
|
+
|
64
|
+
def bench(name, nkeys = 1, &block)
|
65
|
+
if nkeys > 1
|
66
|
+
keyseq = keys + keys
|
67
|
+
block = lambda do |i|
|
68
|
+
i = i % keys.size
|
69
|
+
yield(keyseq[i, nkeys])
|
70
|
+
end
|
71
|
+
name = "#{name}-#{nkeys}"
|
72
|
+
elsif block.arity == 1
|
73
|
+
block = lambda {|i| yield(pick_mod(keys, i))}
|
74
|
+
else
|
75
|
+
block = lambda {|i| yield(pick_mod(keys, i), pick_mod(vals, i))}
|
76
|
+
end
|
77
|
+
|
78
|
+
@bench.report(name) do
|
79
|
+
(n/nkeys).times do |i|
|
80
|
+
block.call(i)
|
81
|
+
end
|
82
|
+
end
|
83
|
+
end
|
84
|
+
end
|
85
|
+
|
86
|
+
def init_servers(*ports)
|
87
|
+
servers = []
|
88
|
+
ports.each do |port|
|
89
|
+
system("memcached -p #{port} -U 0 -d -P /tmp/memcached_#{port}.pid")
|
90
|
+
servers << "127.0.0.1:#{port}"
|
91
|
+
sleep 0.3
|
92
|
+
end
|
93
|
+
memcache = yield(servers)
|
94
|
+
memcache.flush_all
|
95
|
+
memcache
|
96
|
+
end
|
97
|
+
|
98
|
+
def ___
|
99
|
+
puts('=' * 81)
|
100
|
+
end
|
101
|
+
|
102
|
+
puts `uname -a`
|
103
|
+
puts "Ruby #{RUBY_VERSION}p#{RUBY_PATCHLEVEL}"
|
104
|
+
|
105
|
+
ns = 'namespace'
|
106
|
+
memcache = init_servers(10000,10001) {|s| Memcache.new(:servers => s, :namespace => ns)}
|
107
|
+
native = init_servers(10002,10003) {|s| Memcache.new(:servers => s, :namespace => ns, :native => true)}
|
108
|
+
native_nowrap = init_servers(10004,10005) {|s| Memcache::NativeServer.new(:servers => s, :prefix => "#{ns}:")}
|
109
|
+
|
110
|
+
b = MemcacheBench.new(:num_items => 5000, :n => 100_000, :key_length => 20, :val_length => 100)
|
111
|
+
|
112
|
+
2.times do
|
113
|
+
___
|
114
|
+
b.bench( 'set:native-nowrap' ) {|key, val| native_nowrap.set(key, val) }
|
115
|
+
b.bench( 'get:native-nowrap' ) {|key | native_nowrap.get(key) }
|
116
|
+
b.bench( 'get:native-nowrap', 100 ) {|keys | native_nowrap.get(keys) }
|
117
|
+
___
|
118
|
+
b.bench( 'set:native' ) {|key, val| native.set(key, val, :raw => true) }
|
119
|
+
b.bench( 'get:native' ) {|key | native.get(key, :raw => true) }
|
120
|
+
b.bench( 'get:native', 100 ) {|keys | native.get(keys, :raw => true) }
|
121
|
+
___
|
122
|
+
b.bench( 'set:ruby' ) {|key, val| memcache.set(key, val, :raw => true) }
|
123
|
+
b.bench( 'get:ruby' ) {|key | memcache.get(key, :raw => true) }
|
124
|
+
b.bench( 'get:ruby', 100 ) {|keys | memcache.get(keys, :raw => true) }
|
125
|
+
___
|
126
|
+
end
|
data/ext/extconf.rb
CHANGED
@@ -1,3 +1,52 @@
|
|
1
1
|
require 'mkmf'
|
2
|
-
|
3
|
-
|
2
|
+
require 'rbconfig'
|
3
|
+
|
4
|
+
# Code taken from Evan's Weaver memcached library: http://github.com/fauna/memcached
|
5
|
+
|
6
|
+
HERE = File.expand_path(File.dirname(__FILE__))
|
7
|
+
BUNDLE = Dir.glob("libmemcached-*.tar.gz").first
|
8
|
+
BUNDLE_PATH = BUNDLE.sub(".tar.gz", "")
|
9
|
+
|
10
|
+
$CXXFLAGS = " -std=gnu++98"
|
11
|
+
|
12
|
+
if !ENV["EXTERNAL_LIB"]
|
13
|
+
$includes = " -I#{HERE}/include"
|
14
|
+
$libraries = " -L#{HERE}/lib"
|
15
|
+
$CFLAGS = "#{$includes} #{$libraries} #{$CFLAGS}"
|
16
|
+
$LDFLAGS = "#{$libraries} #{$LDFLAGS}"
|
17
|
+
$LIBPATH = ["#{HERE}/lib"]
|
18
|
+
$DEFLIBPATH = []
|
19
|
+
|
20
|
+
Dir.chdir(HERE) do
|
21
|
+
if false and File.exist?("lib")
|
22
|
+
puts "Libmemcached already built; run 'rake clean' first if you need to rebuild."
|
23
|
+
else
|
24
|
+
puts "Building libmemcached."
|
25
|
+
puts(cmd = "tar xzf #{BUNDLE} 2>&1")
|
26
|
+
raise "'#{cmd}' failed" unless system(cmd)
|
27
|
+
|
28
|
+
Dir.chdir(BUNDLE_PATH) do
|
29
|
+
puts(cmd = "env CFLAGS='-fPIC' ./configure --prefix=#{HERE} --without-memcached --disable-shared --disable-utils --disable-dependency-tracking #{$EXTRA_CONF} 2>&1")
|
30
|
+
raise "'#{cmd}' failed" unless system(cmd)
|
31
|
+
|
32
|
+
puts(cmd = "make CXXFLAGS='#{$CXXFLAGS}' || true 2>&1")
|
33
|
+
raise "'#{cmd}' failed" unless system(cmd)
|
34
|
+
|
35
|
+
puts(cmd = "make install || true 2>&1")
|
36
|
+
raise "'#{cmd}' failed" unless system(cmd)
|
37
|
+
end
|
38
|
+
|
39
|
+
system("rm -rf #{BUNDLE_PATH}") unless ENV['DEBUG'] or ENV['DEV']
|
40
|
+
end
|
41
|
+
end
|
42
|
+
|
43
|
+
# Absolutely prevent the linker from picking up any other libmemcached
|
44
|
+
Dir.chdir("#{HERE}/lib") do
|
45
|
+
system('cp -f libmemcached.a libmemcached_gem.a')
|
46
|
+
system('cp -f libmemcached.la libmemcached_gem.la')
|
47
|
+
end
|
48
|
+
|
49
|
+
$LIBS << " -lmemcached_gem"
|
50
|
+
end
|
51
|
+
|
52
|
+
create_makefile('memcache/native_server')
|
Binary file
|