dalli 0.9.2 → 0.9.3
Sign up to get free protection for your applications and to get access to all the features.
Potentially problematic release.
This version of dalli might be problematic. Click here for more details.
- data/History.md +9 -0
- data/README.md +42 -3
- data/Rakefile +5 -0
- data/TODO.md +1 -0
- data/Upgrade.md +44 -0
- data/dalli.gemspec +2 -1
- data/lib/action_dispatch/middleware/session/dalli_store.rb +76 -0
- data/lib/active_support/cache/dalli_store.rb +4 -0
- data/lib/active_support/cache/dalli_store23.rb +172 -0
- data/lib/dalli/client.rb +23 -14
- data/lib/dalli/server.rb +0 -2
- data/lib/dalli/version.rb +1 -1
- data/test/abstract_unit.rb +285 -0
- data/test/{test_benchmark.rb → benchmark_test.rb} +0 -0
- data/test/helper.rb +23 -1
- data/test/memcached_mock.rb +2 -7
- data/test/test_active_support.rb +72 -54
- data/test/test_dalli.rb +5 -1
- data/test/test_session_store.rb +200 -0
- metadata +12 -8
- data/lib/dalli/sasl/base64.rb +0 -14
- data/lib/dalli/sasl/digest_md5.rb +0 -175
data/History.md
CHANGED
data/README.md
CHANGED
@@ -27,6 +27,7 @@ So a few notes. Dalli:
|
|
27
27
|
2. contains explicit "chokepoint" methods which handle all requests; these can be hooked into by monitoring tools (NewRelic, Rack::Bug, etc) to track memcached usage.
|
28
28
|
3. comes with hooks to replace memcache-client in Rails.
|
29
29
|
4. is approx 700 lines of Ruby. memcache-client is approx 1250 lines.
|
30
|
+
5. supports SASL for use in managed environments, e.g. Heroku.
|
30
31
|
|
31
32
|
|
32
33
|
Installation and Usage
|
@@ -44,23 +45,55 @@ Remember, Dalli **requires** memcached 1.4+. You can check the version with `me
|
|
44
45
|
The test suite requires memcached 1.4.3+ with SASL enabled (./configure --enable-sasl). Currently only supports the PLAIN mechanism.
|
45
46
|
|
46
47
|
|
47
|
-
Usage with Rails
|
48
|
+
Usage with Rails 3.0
|
48
49
|
---------------------------
|
49
50
|
|
50
51
|
In your Gemfile:
|
51
52
|
|
52
53
|
gem 'dalli'
|
53
54
|
|
54
|
-
In `config/environments/production.rb
|
55
|
+
In `config/environments/production.rb`:
|
56
|
+
|
57
|
+
config.cache_store = :dalli_store
|
58
|
+
|
59
|
+
A more comprehensive example (note that we are setting a reasonable default for maximum cache entry lifetime (one day), enabling compression for large values, and namespacing all entries for this rails app. Remove the namespace if you have multiple apps which share cached values):
|
55
60
|
|
56
|
-
require 'active_support/cache/dalli_store'
|
57
61
|
config.cache_store = :dalli_store, 'cache-1.example.com', 'cache-2.example.com',
|
58
62
|
:namespace => NAME_OF_RAILS_APP, :expires_in => 1.day, :compress => true, :compress_threshold => 64.kilobytes
|
59
63
|
|
60
64
|
|
65
|
+
Usage with Rails 2.3.x
|
66
|
+
----------------------------
|
67
|
+
|
68
|
+
In `config/environment.rb`:
|
69
|
+
|
70
|
+
config.gem 'dalli'
|
71
|
+
|
72
|
+
In `config/environments/production.rb`:
|
73
|
+
|
74
|
+
require 'active_support/cache/dalli_store23'
|
75
|
+
config.cache_store = :dalli_store
|
76
|
+
|
77
|
+
|
78
|
+
Usage with Passenger
|
79
|
+
------------------------
|
80
|
+
|
81
|
+
Put this at the bottom of `config/environment.rb`:
|
82
|
+
|
83
|
+
if defined?(PhusionPassenger)
|
84
|
+
PhusionPassenger.on_event(:starting_worker_process) do |forked|
|
85
|
+
# Only works with DalliStore
|
86
|
+
Rails.cache.reset if forked
|
87
|
+
end
|
88
|
+
end
|
89
|
+
|
90
|
+
|
61
91
|
Features and Changes
|
62
92
|
------------------------
|
63
93
|
|
94
|
+
Dalli is **NOT** 100% API compatible with memcache-client. If you have code which uses the MemCache
|
95
|
+
API directly, it will likely need small tweaks. Method parameters and return values changed slightly.
|
96
|
+
|
64
97
|
memcache-client allowed developers to store either raw or marshalled values with each API call. I feel this is needless complexity; Dalli allows you to control marshalling per-Client with the `:marshal => false` flag but you cannot explicitly set the raw flag for each API call. By default, marshalling is enabled.
|
65
98
|
|
66
99
|
I've removed support for key namespaces and automatic pruning of keys longer than 250 characters. ActiveSupport::Cache implements these features so there is little need for Dalli to reinvent them.
|
@@ -76,6 +109,12 @@ Helping Out
|
|
76
109
|
If you have a fix you wish to provide, please fork the code, fix in your local project and then send a pull request on github. Please ensure that you include a test which verifies your fix and update History.md with a one sentence description of your fix so you get credit as a contributor.
|
77
110
|
|
78
111
|
|
112
|
+
Thanks
|
113
|
+
------------
|
114
|
+
|
115
|
+
Brian Mitchell - for his remix-stash project which was helpful when implementing and testing the binary protocol support.
|
116
|
+
|
117
|
+
|
79
118
|
Author
|
80
119
|
----------
|
81
120
|
|
data/Rakefile
CHANGED
data/TODO.md
CHANGED
data/Upgrade.md
ADDED
@@ -0,0 +1,44 @@
|
|
1
|
+
Upgrading from memcache-client
|
2
|
+
========
|
3
|
+
|
4
|
+
Dalli is not meant to be 100% compatible with memcache-client, there are serveral differences in the API.
|
5
|
+
|
6
|
+
|
7
|
+
Marshalling
|
8
|
+
---------------
|
9
|
+
|
10
|
+
Dalli has removed support for specifying the marshalling behavior for each operation.
|
11
|
+
|
12
|
+
Take this typical operation:
|
13
|
+
|
14
|
+
cache = MemCache.new
|
15
|
+
cache.set('abc', 123)
|
16
|
+
|
17
|
+
Technically 123 is an Integer and presumably you want `cache.get('abc')` to return an Integer. Since memcached stores values as binary blobs, Dalli will serialize the value to a binary blob for storage. When you get() the value back, Ruby will deserialize it properly to an Integer and all will be well. Without marshalling, Dalli will convert values to Strings and so get() would return a String, not an Integer.
|
18
|
+
|
19
|
+
The memcache-client API allowed you to control marshalling on a per-method basis using a boolean 'raw' parameter to several of the API methods:
|
20
|
+
|
21
|
+
cache = MemCache.new
|
22
|
+
cache.set('abc', 123, 0, true)
|
23
|
+
cache.get('abc', true) => '123'
|
24
|
+
|
25
|
+
cache.set('abc', 123, 0)
|
26
|
+
cache.get('abc') => 123
|
27
|
+
|
28
|
+
Note that the last 'raw' parameter is set to true in the first two API calls and so `get` returns a string, not an integer. In the second example, we don't provide the raw parameter. Since it defaults to false, it works exactly like Dalli.
|
29
|
+
|
30
|
+
If the code specifies raw as false, you can simply remove that parameter. If the code is using raw = true, you will need to use the :marshal option to create a Dalli::Client instance that does not perform marshalling:
|
31
|
+
|
32
|
+
cache = Dalli::Client.new(servers, :marshal => false)
|
33
|
+
|
34
|
+
If the code is mixing marshal modes (performing operations where raw is both true and false), you will need to use two different client instances.
|
35
|
+
|
36
|
+
|
37
|
+
Return Values
|
38
|
+
----------------
|
39
|
+
|
40
|
+
In memcache-client, `set(key, value)` normally returns "STORED\r\n". This is an artifact of the text protocol used in earlier versions of memcached. Code that checks the return value will need to be updated. Dalli raises errors for exceptional cases but otherwise returns true or false depending on whether the operation succeeded or not. These methods are affected:
|
41
|
+
|
42
|
+
set
|
43
|
+
add
|
44
|
+
replace
|
data/dalli.gemspec
CHANGED
@@ -19,6 +19,7 @@ Gem::Specification.new do |s|
|
|
19
19
|
"Gemfile",
|
20
20
|
"dalli.gemspec",
|
21
21
|
"Performance.md",
|
22
|
+
"Upgrade.md",
|
22
23
|
]
|
23
24
|
s.homepage = %q{http://github.com/mperham/dalli}
|
24
25
|
s.rdoc_options = ["--charset=UTF-8"]
|
@@ -27,7 +28,7 @@ Gem::Specification.new do |s|
|
|
27
28
|
s.test_files = Dir.glob("test/**/*")
|
28
29
|
s.add_development_dependency(%q<shoulda>, [">= 0"])
|
29
30
|
s.add_development_dependency(%q<mocha>, [">= 0"])
|
30
|
-
s.add_development_dependency(%q<rails>, [">= 3.0.0
|
31
|
+
s.add_development_dependency(%q<rails>, [">= 3.0.0"])
|
31
32
|
s.add_development_dependency(%q<memcache-client>, [">= 1.8.5"])
|
32
33
|
end
|
33
34
|
|
@@ -0,0 +1,76 @@
|
|
1
|
+
require 'active_support/cache'
|
2
|
+
require 'action_dispatch/middleware/session/abstract_store'
|
3
|
+
require 'dalli'
|
4
|
+
|
5
|
+
# Dalli-based session store for Rails 3.0. Use like so:
|
6
|
+
#
|
7
|
+
# require 'action_dispatch/middleware/session/dalli_store'
|
8
|
+
# config.session_store ActionDispatch::Session::DalliStore, ['cache-1', 'cache-2'], :expire_after => 2.weeks
|
9
|
+
module ActionDispatch
|
10
|
+
module Session
|
11
|
+
class DalliStore < AbstractStore
|
12
|
+
def initialize(app, options = {})
|
13
|
+
# Support old :expires option
|
14
|
+
options[:expire_after] ||= options[:expires]
|
15
|
+
|
16
|
+
super
|
17
|
+
|
18
|
+
@default_options = {
|
19
|
+
:namespace => 'rack:session',
|
20
|
+
:memcache_server => 'localhost:11211',
|
21
|
+
:expires_in => options[:expire_after]
|
22
|
+
}.merge(@default_options)
|
23
|
+
|
24
|
+
@pool = options[:cache] || begin
|
25
|
+
Dalli::Client.new(
|
26
|
+
@default_options[:memcache_server], @default_options)
|
27
|
+
end
|
28
|
+
@namespace = @default_options[:namespace]
|
29
|
+
|
30
|
+
super
|
31
|
+
end
|
32
|
+
|
33
|
+
def reset
|
34
|
+
@pool.reset
|
35
|
+
end
|
36
|
+
|
37
|
+
private
|
38
|
+
|
39
|
+
def session_key(sid)
|
40
|
+
# Dalli does not support namespaces directly so we have
|
41
|
+
# to roll our own.
|
42
|
+
@namespace ? "#{@namespace}:#{sid}" : sid
|
43
|
+
end
|
44
|
+
|
45
|
+
def get_session(env, sid)
|
46
|
+
begin
|
47
|
+
session = @pool.get(session_key(sid)) || {}
|
48
|
+
rescue Dalli::DalliError => de
|
49
|
+
Rails.logger.warn("Session::DalliStore: #{$!.message}")
|
50
|
+
session = {}
|
51
|
+
end
|
52
|
+
[sid, session]
|
53
|
+
end
|
54
|
+
|
55
|
+
def set_session(env, sid, session_data)
|
56
|
+
options = env['rack.session.options']
|
57
|
+
expiry = options[:expire_after] || 0
|
58
|
+
@pool.set(session_key(sid), session_data, expiry)
|
59
|
+
sid
|
60
|
+
rescue Dalli::DalliError
|
61
|
+
Rails.logger.warn("Session::DalliStore: #{$!.message}")
|
62
|
+
false
|
63
|
+
end
|
64
|
+
|
65
|
+
def destroy(env)
|
66
|
+
if sid = current_session_id(env)
|
67
|
+
@pool.delete(session_key(sid))
|
68
|
+
end
|
69
|
+
rescue Dalli::DalliError
|
70
|
+
Rails.logger.warn("Session::DalliStore: #{$!.message}")
|
71
|
+
false
|
72
|
+
end
|
73
|
+
|
74
|
+
end
|
75
|
+
end
|
76
|
+
end
|
@@ -0,0 +1,172 @@
|
|
1
|
+
begin
|
2
|
+
require 'dalli'
|
3
|
+
rescue LoadError => e
|
4
|
+
$stderr.puts "You don't have dalli installed in your application: #{e.message}"
|
5
|
+
raise e
|
6
|
+
end
|
7
|
+
require 'digest/md5'
|
8
|
+
|
9
|
+
module ActiveSupport
|
10
|
+
module Cache
|
11
|
+
# A cache store implementation which stores data in Memcached:
|
12
|
+
# http://www.danga.com/memcached/
|
13
|
+
#
|
14
|
+
# DalliStore implements the Strategy::LocalCache strategy which implements
|
15
|
+
# an in memory cache inside of a block.
|
16
|
+
class DalliStore < Store
|
17
|
+
module Response # :nodoc:
|
18
|
+
STORED = "STORED\r\n"
|
19
|
+
NOT_STORED = "NOT_STORED\r\n"
|
20
|
+
EXISTS = "EXISTS\r\n"
|
21
|
+
NOT_FOUND = "NOT_FOUND\r\n"
|
22
|
+
DELETED = "DELETED\r\n"
|
23
|
+
end
|
24
|
+
|
25
|
+
ESCAPE_KEY_CHARS = /[\x00-\x20%\x7F-\xFF]/
|
26
|
+
|
27
|
+
def self.build_mem_cache(*addresses)
|
28
|
+
addresses = addresses.flatten
|
29
|
+
options = addresses.extract_options!
|
30
|
+
addresses = ["localhost"] if addresses.empty?
|
31
|
+
Dalli::Client.new(addresses, options)
|
32
|
+
end
|
33
|
+
|
34
|
+
# Creates a new DalliStore object, with the given memcached server
|
35
|
+
# addresses. Each address is either a host name, or a host-with-port string
|
36
|
+
# in the form of "host_name:port". For example:
|
37
|
+
#
|
38
|
+
# ActiveSupport::Cache::DalliStore.new("localhost", "server-downstairs.localnetwork:8229")
|
39
|
+
#
|
40
|
+
# If no addresses are specified, then DalliStore will connect to
|
41
|
+
# localhost port 11211 (the default memcached port).
|
42
|
+
#
|
43
|
+
def initialize(*addresses)
|
44
|
+
if addresses.first.respond_to?(:get)
|
45
|
+
@data = addresses.first
|
46
|
+
else
|
47
|
+
@data = self.class.build_mem_cache(*addresses)
|
48
|
+
end
|
49
|
+
|
50
|
+
extend Strategy::LocalCache
|
51
|
+
end
|
52
|
+
|
53
|
+
# Reads multiple keys from the cache using a single call to the
|
54
|
+
# servers for all keys. Options can be passed in the last argument.
|
55
|
+
def read_multi(*names)
|
56
|
+
keys_to_names = names.inject({}){|map, name| map[escape_key(name)] = name; map}
|
57
|
+
cache_keys = {}
|
58
|
+
# map keys to servers
|
59
|
+
names.each do |key|
|
60
|
+
cache_key = escape_key key
|
61
|
+
cache_keys[cache_key] = key
|
62
|
+
end
|
63
|
+
|
64
|
+
values = @data.get_multi keys_to_names.keys
|
65
|
+
results = {}
|
66
|
+
values.each do |key, value|
|
67
|
+
results[cache_keys[key]] = Marshal.load value
|
68
|
+
end
|
69
|
+
results
|
70
|
+
end
|
71
|
+
|
72
|
+
# Increment a cached value. This method uses the memcached incr atomic
|
73
|
+
# operator and can only be used on values written with the :raw option.
|
74
|
+
# Calling it on a value not stored with :raw will initialize that value
|
75
|
+
# to zero.
|
76
|
+
def increment(key, amount = 1) # :nodoc:
|
77
|
+
log("incrementing", key, amount)
|
78
|
+
|
79
|
+
response = @data.incr(escape_key(key), amount)
|
80
|
+
response == Response::NOT_FOUND ? nil : response
|
81
|
+
rescue Dalli::DalliError
|
82
|
+
nil
|
83
|
+
end
|
84
|
+
# Decrement a cached value. This method uses the memcached decr atomic
|
85
|
+
# operator and can only be used on values written with the :raw option.
|
86
|
+
# Calling it on a value not stored with :raw will initialize that value
|
87
|
+
# to zero.
|
88
|
+
def decrement(key, amount = 1) # :nodoc:
|
89
|
+
log("decrement", key, amount)
|
90
|
+
response = @data.decr(escape_key(key), amount)
|
91
|
+
response == Response::NOT_FOUND ? nil : response
|
92
|
+
rescue Dalli::DalliError
|
93
|
+
nil
|
94
|
+
end
|
95
|
+
|
96
|
+
def reset
|
97
|
+
@pool.reset
|
98
|
+
end
|
99
|
+
|
100
|
+
# Clear the entire cache on all memcached servers. This method should
|
101
|
+
# be used with care when using a shared cache.
|
102
|
+
def clear
|
103
|
+
@data.flush_all
|
104
|
+
end
|
105
|
+
|
106
|
+
# Get the statistics from the memcached servers.
|
107
|
+
def stats
|
108
|
+
@data.stats
|
109
|
+
end
|
110
|
+
|
111
|
+
# Read an entry from the cache.
|
112
|
+
def read(key, options = nil) # :nodoc:
|
113
|
+
super
|
114
|
+
value = @data.get(escape_key(key))
|
115
|
+
return nil if value.nil?
|
116
|
+
value = Marshal.load value
|
117
|
+
value
|
118
|
+
rescue Dalli::DalliError => e
|
119
|
+
logger.error("DalliError (#{e}): #{e.message}")
|
120
|
+
nil
|
121
|
+
end
|
122
|
+
|
123
|
+
# Writes a value to the cache.
|
124
|
+
#
|
125
|
+
# Possible options:
|
126
|
+
# - +:unless_exist+ - set to true if you don't want to update the cache
|
127
|
+
# if the key is already set.
|
128
|
+
# - +:expires_in+ - the number of seconds that this value may stay in
|
129
|
+
# the cache. See ActiveSupport::Cache::Store#write for an example.
|
130
|
+
def write(key, value, options = nil)
|
131
|
+
super
|
132
|
+
method = options && options[:unless_exist] ? :add : :set
|
133
|
+
# memcache-client will break the connection if you send it an integer
|
134
|
+
# in raw mode, so we convert it to a string to be sure it continues working.
|
135
|
+
value = Marshal.dump value
|
136
|
+
@data.send(method, escape_key(key), value, expires_in(options))
|
137
|
+
rescue Dalli::DalliError => e
|
138
|
+
logger.error("DalliError (#{e}): #{e.message}")
|
139
|
+
false
|
140
|
+
end
|
141
|
+
|
142
|
+
def delete(key, options = nil) # :nodoc:
|
143
|
+
super
|
144
|
+
@data.delete(escape_key(key))
|
145
|
+
rescue Dalli::DalliError => e
|
146
|
+
logger.error("DalliError (#{e}): #{e.message}")
|
147
|
+
false
|
148
|
+
end
|
149
|
+
|
150
|
+
def exist?(key, options = nil) # :nodoc:
|
151
|
+
# Doesn't call super, cause exist? in memcache is in fact a read
|
152
|
+
# But who cares? Reading is very fast anyway
|
153
|
+
# Local cache is checked first, if it doesn't know then memcache itself is read from
|
154
|
+
!read(key, options).nil?
|
155
|
+
end
|
156
|
+
|
157
|
+
def delete_matched(matcher, options = nil) # :nodoc:
|
158
|
+
# don't do any local caching at present, just pass
|
159
|
+
# through and let the error happen
|
160
|
+
super
|
161
|
+
raise "Not supported by Memcache"
|
162
|
+
end
|
163
|
+
|
164
|
+
private
|
165
|
+
def escape_key(key)
|
166
|
+
key = key.to_s.gsub(ESCAPE_KEY_CHARS){|match| "%#{match.getbyte(0).to_s(16).upcase}"}
|
167
|
+
key = "#{key[0, 213]}:md5:#{Digest::MD5.hexdigest(key)}" if key.size > 250
|
168
|
+
key
|
169
|
+
end
|
170
|
+
end
|
171
|
+
end
|
172
|
+
end
|
data/lib/dalli/client.rb
CHANGED
@@ -20,11 +20,8 @@ module Dalli
|
|
20
20
|
# Default: true.
|
21
21
|
#
|
22
22
|
def initialize(servers=nil, options={})
|
23
|
-
@
|
24
|
-
|
25
|
-
Dalli::Server.new(s)
|
26
|
-
end, options
|
27
|
-
)
|
23
|
+
@servers = servers
|
24
|
+
@options = options
|
28
25
|
self.extend(Dalli::Marshal) unless options[:marshal] == false
|
29
26
|
end
|
30
27
|
|
@@ -38,11 +35,11 @@ module Dalli
|
|
38
35
|
end
|
39
36
|
|
40
37
|
def get_multi(*keys)
|
41
|
-
|
38
|
+
ring.lock do
|
42
39
|
keys.flatten.each do |key|
|
43
40
|
perform(:getkq, key)
|
44
41
|
end
|
45
|
-
values =
|
42
|
+
values = ring.servers.inject({}) { |hash, s| hash.merge!(s.request(:noop)); hash }
|
46
43
|
values.inject(values) { |memo, (k,v)| memo[k] = deserialize(v); memo }
|
47
44
|
end
|
48
45
|
end
|
@@ -91,11 +88,12 @@ module Dalli
|
|
91
88
|
|
92
89
|
def flush(delay=0)
|
93
90
|
time = -delay
|
94
|
-
|
91
|
+
ring.servers.map { |s| s.request(:flush, time += delay) }
|
95
92
|
end
|
96
93
|
|
97
|
-
|
98
|
-
|
94
|
+
# deprecated, please use #flush.
|
95
|
+
def flush_all(delay=0)
|
96
|
+
flush(delay)
|
99
97
|
end
|
100
98
|
|
101
99
|
##
|
@@ -129,16 +127,27 @@ module Dalli
|
|
129
127
|
end
|
130
128
|
|
131
129
|
def stats
|
132
|
-
|
130
|
+
ring.servers.inject({}) { |memo, s| memo["#{s.hostname}:#{s.port}"] = s.request(:stats); memo }
|
133
131
|
end
|
134
132
|
|
135
133
|
def close
|
136
|
-
@ring
|
137
|
-
|
134
|
+
if @ring
|
135
|
+
@ring.servers.map { |s| s.close }
|
136
|
+
@ring = nil
|
137
|
+
end
|
138
138
|
end
|
139
|
+
alias_method :reset, :close
|
139
140
|
|
140
141
|
private
|
141
142
|
|
143
|
+
def ring
|
144
|
+
@ring ||= Dalli::Ring.new(
|
145
|
+
Array(env_servers || @servers).map do |s|
|
146
|
+
Dalli::Server.new(s)
|
147
|
+
end, @options
|
148
|
+
)
|
149
|
+
end
|
150
|
+
|
142
151
|
def serialize(value)
|
143
152
|
value.to_s
|
144
153
|
end
|
@@ -155,7 +164,7 @@ module Dalli
|
|
155
164
|
def perform(op, *args)
|
156
165
|
key = args.first
|
157
166
|
validate_key(key)
|
158
|
-
server =
|
167
|
+
server = ring.server_for_key(key)
|
159
168
|
server.request(op, *args)
|
160
169
|
end
|
161
170
|
|