redis-objects 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/ATOMICITY.rdoc +154 -0
- data/README.rdoc +144 -0
- data/lib/redis/counter.rb +105 -0
- data/lib/redis/data_types.rb +84 -0
- data/lib/redis/list.rb +95 -0
- data/lib/redis/lock.rb +47 -0
- data/lib/redis/objects.rb +103 -0
- data/lib/redis/objects/core.rb +0 -0
- data/lib/redis/objects/counters.rb +113 -0
- data/lib/redis/objects/lists.rb +34 -0
- data/lib/redis/objects/locks.rb +56 -0
- data/lib/redis/objects/sets.rb +9 -0
- data/lib/redis/objects/values.rb +37 -0
- data/lib/redis/set.rb +7 -0
- data/lib/redis/value.rb +39 -0
- data/spec/redis_objects_model_spec.rb +278 -0
- data/spec/spec_helper.rb +5 -0
- metadata +73 -0
data/ATOMICITY.rdoc
ADDED
@@ -0,0 +1,154 @@
|
|
1
|
+
= An Atomic Rant
|
2
|
+
|
3
|
+
== Brush Up Your Resume
|
4
|
+
|
5
|
+
You are probably not handling atomic operations properly in your app, and
|
6
|
+
probably have some nasty lurking race conditions. The worst part is these
|
7
|
+
will get worse as your user count increases, are difficult to reproduce,
|
8
|
+
and usually happen to your most critical pieces of code. (And no, your
|
9
|
+
rspec tests can't catch them either.)
|
10
|
+
|
11
|
+
Let's assume you're writing an app to enable students to enroll in courses.
|
12
|
+
You need to ensure that no more than 30 students can sign up for a given course.
|
13
|
+
In your enrollment code, you have something like this:
|
14
|
+
|
15
|
+
@course = Course.find(1)
|
16
|
+
if @course.num_students < 30
|
17
|
+
@course.course_students.create!(:student_id => 101)
|
18
|
+
@course.num_students += 1
|
19
|
+
@course.save!
|
20
|
+
else
|
21
|
+
# course is full
|
22
|
+
end
|
23
|
+
|
24
|
+
You're screwed. You now have 32 people in your 30 person class, and you have
|
25
|
+
no idea what happened.
|
26
|
+
|
27
|
+
"Well no duh," you're saying, "even the {ActiveRecord docs mention locking}[http://api.rubyonrails.org/classes/ActiveRecord/Locking/Pessimistic.html],
|
28
|
+
so I'll just use that."
|
29
|
+
|
30
|
+
@course = Course.find(1, :lock => true)
|
31
|
+
if @course.num_students < 30
|
32
|
+
# ...
|
33
|
+
|
34
|
+
Nice try, but now you've introduced other issues. Any other piece of code
|
35
|
+
in your entire app that needs to update _anything_ about the course - maybe
|
36
|
+
the course name, or start date, or location - is now serialized. If you need high
|
37
|
+
concurrency, you're still screwed.
|
38
|
+
|
39
|
+
You think, "ah-ha, the problem is having a separate counter!"
|
40
|
+
|
41
|
+
@course = Course.find(1)
|
42
|
+
if @course.course_students.count < 30
|
43
|
+
@course.course_students.create!(:student_id => 101)
|
44
|
+
else
|
45
|
+
# course is full
|
46
|
+
end
|
47
|
+
|
48
|
+
Nope. Still screwed.
|
49
|
+
|
50
|
+
== The Root Down
|
51
|
+
|
52
|
+
It's worth understanding the root issue, and how to address it.
|
53
|
+
|
54
|
+
Race conditions arise from the difference in time between *evaluating* and *altering*
|
55
|
+
a value. In our example, we fetched the record, then checked the value, then
|
56
|
+
changed it. The more lines of code between those operations, and the higher your user
|
57
|
+
count, the bigger the window of opportunity for other clients to get the data in an
|
58
|
+
inconsistent state.
|
59
|
+
|
60
|
+
Sometimes race conditions don't matter in practice, since often a user is
|
61
|
+
only operating their own data. This has a race condition, but is probably ok:
|
62
|
+
|
63
|
+
@post = Post.create(:user_id => @user.id, :title => "Whattup", ...)
|
64
|
+
@user.total_posts += 1 # update my post count
|
65
|
+
|
66
|
+
But this _would_ be problematic:
|
67
|
+
|
68
|
+
@post = Post.create(:user_id => @user.id, :title => "Whattup", ...)
|
69
|
+
@blog.total_posts += 1 # update post count across all users
|
70
|
+
|
71
|
+
As multiple users could be adding posts concurrently.
|
72
|
+
|
73
|
+
In a traditional RDBMS, you can increment counters atomically (but not return them)
|
74
|
+
by firing off an update statement that self-references the column:
|
75
|
+
|
76
|
+
update users set total_posts = total_posts + 1 where id = 372
|
77
|
+
|
78
|
+
You may have seen {ActiveRecord's increment_counter class method}[http://api.rubyonrails.org/classes/ActiveRecord/Base.html#M002278],
|
79
|
+
which wraps this functionality. But outside of being cumbersome, this has
|
80
|
+
the side effect that your object is no longer in sync with the DB, so you
|
81
|
+
get other issues:
|
82
|
+
|
83
|
+
Blog.increment_counter :total_posts, @blog.id
|
84
|
+
if @blog.total_posts == 1000
|
85
|
+
# the 1000th poster - award them a gold star!
|
86
|
+
|
87
|
+
The DB says 1000, but your @blog object still says 999, and the right person
|
88
|
+
doesn't get their gold star. Sad faces all around.
|
89
|
+
|
90
|
+
== A Better Way
|
91
|
+
|
92
|
+
Bottom line: Any operation that could alter a value *must* return that value in
|
93
|
+
the _same_ _operation_ for it to be atomic. If you do a separate get then set,
|
94
|
+
or set then get, you're open to a race condition. There are very few systems that
|
95
|
+
support an "increment and return" type operation, and Redis is one of them
|
96
|
+
(Oracle sequences are another).
|
97
|
+
|
98
|
+
When you think of the specific things that you need to ensure, many of these will
|
99
|
+
reduce to numeric operations:
|
100
|
+
|
101
|
+
* Ensuring there are no more than 30 students in a course
|
102
|
+
* Getting more than 2 but less than 6 people in a game
|
103
|
+
* Keeping a chat room to a max of 50 people
|
104
|
+
* Correctly recording the total number of blog posts
|
105
|
+
* Only allowing one piece of code to reorder a large dataset at a time
|
106
|
+
|
107
|
+
All except the last one can be implemented with counters. The last one
|
108
|
+
will need a carefully placed lock.
|
109
|
+
|
110
|
+
The best way I've found to balance atomicity and concurrency is, for each value,
|
111
|
+
actually create two counters:
|
112
|
+
|
113
|
+
* A counter you base logic on (eg, +slots_taken+)
|
114
|
+
* A counter users see (eg, +current_students+)
|
115
|
+
|
116
|
+
The reason you want two counters is you'll need to change the value of the logic
|
117
|
+
counter *first*, _before_ checking it, to address any race conditions. This means
|
118
|
+
the value can get wonky momentarily (eg, there could be 32 +slots_taken+ for a 30-person
|
119
|
+
course). This doesn't affect its function - indeed, it's part of what makes it work - but
|
120
|
+
does mean you don't want to display it.
|
121
|
+
|
122
|
+
So, taking our +Course+ example:
|
123
|
+
|
124
|
+
class Course < ActiveRecord::Base
|
125
|
+
include Redis::Atoms
|
126
|
+
|
127
|
+
counter :slots_taken
|
128
|
+
counter :current_students
|
129
|
+
end
|
130
|
+
|
131
|
+
Then:
|
132
|
+
|
133
|
+
@course = Course.find(1)
|
134
|
+
@course.slots_taken.increment do |val|
|
135
|
+
if val <= @course.max_students
|
136
|
+
@course.course_students.create!(:student_id => 101)
|
137
|
+
@course.current_students.increment
|
138
|
+
end
|
139
|
+
end
|
140
|
+
|
141
|
+
Race-condition free. And, with the separate +current_students+ counter, your
|
142
|
+
views get consistent information about the course, since it will only be
|
143
|
+
incremented on success. There is still a race condition where +current_students+
|
144
|
+
could be less than the real number of +CourseStudent+ records, but since you'll be
|
145
|
+
displaying these values in a view (after that block completes) you shouldn't see
|
146
|
+
this manifest in real-world usage.
|
147
|
+
|
148
|
+
Now you can sleep soundly, without fear of getting fired at 3am via an angry
|
149
|
+
phone call from your boss. (At least, not about this...)
|
150
|
+
|
151
|
+
== Author
|
152
|
+
|
153
|
+
Copyright (c) 2009 {Nate Wiger}[http://nate.wiger.org]. All Rights Reserved.
|
154
|
+
Rant released under {Creative Commons}[http://creativecommons.org/licenses/by/3.0/legalcode].
|
data/README.rdoc
ADDED
@@ -0,0 +1,144 @@
|
|
1
|
+
= Redis::Objects - Lightweight, atomic object layer around redis-rb
|
2
|
+
|
3
|
+
This is *not* an ORM. People that are wrapping ORM's around Redis are missing
|
4
|
+
the point.
|
5
|
+
|
6
|
+
The killer feature of Redis that it allows you to perform atomic operations
|
7
|
+
on _individual_ data structures, like counters, lists, and sets, that you can use
|
8
|
+
*with* your existing ActiveRecord/DataMapper/etc models, or in classes that have
|
9
|
+
nothing to do with an ORM or even a database. That's where this gem comes in.
|
10
|
+
|
11
|
+
|
12
|
+
This gem originally arose out of a need for high-concurrency operations for online
|
13
|
+
games; for a fun rant on the topic, see
|
14
|
+
{ATOMICITY}[http://github.com/nateware/redis-objects/blob/master/ATOMICITY.rdoc],
|
15
|
+
or scroll down to "Atomicity" in this README.
|
16
|
+
|
17
|
+
There are two ways to use Redis::Objects, either as an +include+ in a class, or
|
18
|
+
by using +new+ with the type of data structure you want to create.
|
19
|
+
|
20
|
+
== Installation
|
21
|
+
|
22
|
+
gem install gemcutter
|
23
|
+
gem tumble
|
24
|
+
gem install redis-objects
|
25
|
+
|
26
|
+
=== Initialization
|
27
|
+
|
28
|
+
# If on Rails, config/initializers/redis.rb is a good place for this
|
29
|
+
require 'redis'
|
30
|
+
require 'redis/objects'
|
31
|
+
Redis::Objects.redis = Redis.new(:host => 127.0.0.1, :port => 6379)
|
32
|
+
|
33
|
+
== Examples
|
34
|
+
|
35
|
+
=== Model Class Usage
|
36
|
+
|
37
|
+
class Team < ActiveRecord::Base
|
38
|
+
include Redis::Objects
|
39
|
+
|
40
|
+
counter :drafted_players
|
41
|
+
counter :active_players
|
42
|
+
counter :total_online_players, :global => true
|
43
|
+
list :on_base
|
44
|
+
set :outfielders
|
45
|
+
value :coach
|
46
|
+
end
|
47
|
+
|
48
|
+
Familiar Ruby operations Just Work:
|
49
|
+
|
50
|
+
@team = Team.find(1)
|
51
|
+
@team.on_base << 'player1'
|
52
|
+
@team.on_base << 'player2'
|
53
|
+
puts @team.on_base # ['player1', 'player2']
|
54
|
+
@team.on_base.pop
|
55
|
+
|
56
|
+
With one purposeful exception - counters cannot be set, only incremented/decremented:
|
57
|
+
|
58
|
+
@team.drafted_players.increment # or incr
|
59
|
+
@team.drafted_players.decrement # or decr
|
60
|
+
@team.drafted_players = 4 # exception
|
61
|
+
@team.drafted_players += 1 # exception
|
62
|
+
|
63
|
+
=== Instance Usage
|
64
|
+
|
65
|
+
=== Counters
|
66
|
+
|
67
|
+
@counter = Redis::Counter.new('counter_name')
|
68
|
+
@counter.increment
|
69
|
+
puts @counter
|
70
|
+
puts @counter.get # force re-fetch
|
71
|
+
|
72
|
+
=== Lists
|
73
|
+
|
74
|
+
@list = Redis::List.new('list_name')
|
75
|
+
@list << 'a'
|
76
|
+
@list << 'b'
|
77
|
+
puts @list
|
78
|
+
|
79
|
+
== Atomicity
|
80
|
+
|
81
|
+
You are probably not handling atomicity correctly in your app. For a fun rant
|
82
|
+
on the topic, see {ATOMICITY}[ATOMICITY.doc]
|
83
|
+
|
84
|
+
Atomic counters are a good way to handle concurrency:
|
85
|
+
|
86
|
+
@team = Team.find(1)
|
87
|
+
if @team.drafted_players.increment <= @team.max_players
|
88
|
+
# do stuff
|
89
|
+
@team.team_players.create!(:player_id => 221)
|
90
|
+
@team.active_players.increment
|
91
|
+
else
|
92
|
+
# reset counter state
|
93
|
+
@team.drafted_players.decrement
|
94
|
+
end
|
95
|
+
|
96
|
+
Atomic block - a cleaner way to do the above. Exceptions or return nil
|
97
|
+
rewind counter back to previous state:
|
98
|
+
|
99
|
+
@team.drafted_players.increment do |val|
|
100
|
+
raise Team::TeamFullError if val > @team.max_players
|
101
|
+
@team.team_players.create!(:player_id => 221)
|
102
|
+
@team.active_players.increment
|
103
|
+
end
|
104
|
+
|
105
|
+
Similar approach, using an if block (failure rewinds counter):
|
106
|
+
|
107
|
+
@team.drafted_players.increment do |val|
|
108
|
+
if val <= @team.max_players
|
109
|
+
@team.team_players.create!(:player_id => 221)
|
110
|
+
@team.active_players.increment
|
111
|
+
end
|
112
|
+
end
|
113
|
+
|
114
|
+
Class methods work too - notice we override ActiveRecord counters:
|
115
|
+
|
116
|
+
Team.increment_counter :drafted_players, team_id
|
117
|
+
Team.decrement_counter :drafted_players, team_id, 2
|
118
|
+
Team.increment_counter :total_online_players # no ID on global counter
|
119
|
+
|
120
|
+
Class-level atomic block (may save a DB fetch depending on your app):
|
121
|
+
|
122
|
+
Team.increment_counter(:drafted_players, team_id) do |val|
|
123
|
+
TeamPitcher.create!(:team_id => team_id, :pitcher_id => 181)
|
124
|
+
Team.increment_counter(:active_players, team_id)
|
125
|
+
end
|
126
|
+
|
127
|
+
Locks with Redis. On completion or exception the lock is released:
|
128
|
+
|
129
|
+
@team.reorder_lock.lock do
|
130
|
+
@team.reorder_all_players
|
131
|
+
end
|
132
|
+
|
133
|
+
Class-level lock (same concept)
|
134
|
+
|
135
|
+
Team.obtain_lock(:reorder, team_id) do
|
136
|
+
Team.reorder_all_players(team_id)
|
137
|
+
end
|
138
|
+
|
139
|
+
|
140
|
+
== Author
|
141
|
+
|
142
|
+
Copyright (c) 2009 {Nate Wiger}[http://nate.wiger.org]. All Rights Reserved.
|
143
|
+
Released under the {Artistic License}[http://www.opensource.org/licenses/artistic-license-2.0.php].
|
144
|
+
|
@@ -0,0 +1,105 @@
|
|
1
|
+
class Redis
|
2
|
+
#
|
3
|
+
# Class representing a Redis counter. This functions like a proxy class, in
|
4
|
+
# that you can say @object.counter_name to get the value and then
|
5
|
+
# @object.counter_name.increment to operate on it. You can use this
|
6
|
+
# directly, or you can use the counter :foo class method in your
|
7
|
+
# class to define a counter.
|
8
|
+
#
|
9
|
+
class Counter
|
10
|
+
attr_reader :key, :options, :redis
|
11
|
+
def initialize(key, options={})
|
12
|
+
@key = key
|
13
|
+
@options = options
|
14
|
+
@redis = options[:redis] || $redis || Redis::Objects.redis
|
15
|
+
@options[:start] ||= 0
|
16
|
+
@options[:type] ||= @options[:start] == 0 ? :increment : :decrement
|
17
|
+
@redis.setnx(key, @options[:start]) unless @options[:start] == 0 || @options[:init] === false
|
18
|
+
end
|
19
|
+
|
20
|
+
# Reset the counter to its starting value. Not atomic, so use with care.
|
21
|
+
# Normally only useful if you're discarding all sub-records associated
|
22
|
+
# with a parent and starting over (for example, restarting a game and
|
23
|
+
# disconnecting all players).
|
24
|
+
def reset(to=options[:start])
|
25
|
+
redis.set(key, to.to_i)
|
26
|
+
@value = to.to_i
|
27
|
+
end
|
28
|
+
|
29
|
+
# Re-gets the current value of the counter. Normally just calling the
|
30
|
+
# counter will lazily fetch the value, and only update it if increment
|
31
|
+
# or decrement is called. This forces a network call to redis-server
|
32
|
+
# to get the current value.
|
33
|
+
def get
|
34
|
+
@value = redis.get(key).to_i
|
35
|
+
end
|
36
|
+
|
37
|
+
# Delete a counter. Usage discouraged. Consider +reset+ instead.
|
38
|
+
def delete
|
39
|
+
redis.del(key)
|
40
|
+
@value = nil
|
41
|
+
end
|
42
|
+
alias_method :del, :delete
|
43
|
+
|
44
|
+
# Returns the (possibly cached) value of the counter. Use +get+ to
|
45
|
+
# force a re-get from the Redis server.
|
46
|
+
def value
|
47
|
+
@value ||= get
|
48
|
+
end
|
49
|
+
|
50
|
+
# Increment the counter atomically and return the new value. If passed
|
51
|
+
# a block, that block will be evaluated with the new value of the counter
|
52
|
+
# as an argument. If the block returns nil or throws an exception, the
|
53
|
+
# counter will automatically be decremented to its previous value. This
|
54
|
+
# method is aliased as incr() for brevity.
|
55
|
+
def increment(by=1, &block)
|
56
|
+
@value = redis.incr(key, by).to_i
|
57
|
+
block_given? ? rewindable_block(:decrement, @value, &block) : @value
|
58
|
+
end
|
59
|
+
alias_method :incr, :increment
|
60
|
+
|
61
|
+
# Decrement the counter atomically and return the new value. If passed
|
62
|
+
# a block, that block will be evaluated with the new value of the counter
|
63
|
+
# as an argument. If the block returns nil or throws an exception, the
|
64
|
+
# counter will automatically be incremented to its previous value. This
|
65
|
+
# method is aliased as incr() for brevity.
|
66
|
+
def decrement(by=1, &block)
|
67
|
+
@value = redis.decr(key, by).to_i
|
68
|
+
block_given? ? rewindable_block(:increment, @value, &block) : @value
|
69
|
+
end
|
70
|
+
alias_method :decr, :decrement
|
71
|
+
|
72
|
+
##
|
73
|
+
# Proxy methods to help make @object.counter == 10 work
|
74
|
+
def to_s; value.to_s; end
|
75
|
+
alias_method :to_str, :to_s
|
76
|
+
alias_method :to_i, :value
|
77
|
+
def nil?; value.nil? end
|
78
|
+
|
79
|
+
# Math ops
|
80
|
+
%w(== < > <= >=).each do |m|
|
81
|
+
class_eval <<-EndOverload
|
82
|
+
def #{m}(x)
|
83
|
+
value #{m} x
|
84
|
+
end
|
85
|
+
EndOverload
|
86
|
+
end
|
87
|
+
|
88
|
+
private
|
89
|
+
|
90
|
+
# Implements atomic increment/decrement blocks
|
91
|
+
def rewindable_block(rewind, value, &block)
|
92
|
+
raise ArgumentError, "Missing block to rewindable_block somehow" unless block_given?
|
93
|
+
ret = nil
|
94
|
+
begin
|
95
|
+
ret = yield value
|
96
|
+
rescue
|
97
|
+
send(rewind)
|
98
|
+
raise
|
99
|
+
end
|
100
|
+
send(rewind) if ret.nil?
|
101
|
+
ret
|
102
|
+
end
|
103
|
+
end
|
104
|
+
end
|
105
|
+
|
@@ -0,0 +1,84 @@
|
|
1
|
+
class Redis
|
2
|
+
module DataTypes
|
3
|
+
TYPES = %w(String Integer Float EpochTime DateTime Json Yaml IPAddress FilePath Uri Slug)
|
4
|
+
def self.included(klass)
|
5
|
+
TYPES.each do |data_type|
|
6
|
+
if Object.const_defined?(data_type)
|
7
|
+
klass = Object.const_get(data_type)
|
8
|
+
else
|
9
|
+
klass = Object.const_set(data_type, Class.new)
|
10
|
+
end
|
11
|
+
if const_defined?(data_type)
|
12
|
+
klass.extend const_get(data_type)
|
13
|
+
end
|
14
|
+
end
|
15
|
+
end
|
16
|
+
|
17
|
+
module String
|
18
|
+
def to_redis; to_s; end
|
19
|
+
end
|
20
|
+
|
21
|
+
module Integer
|
22
|
+
def from_redis(value); value && value.to_i end
|
23
|
+
end
|
24
|
+
|
25
|
+
module Float
|
26
|
+
def from_redis(value); value && value.to_f end
|
27
|
+
end
|
28
|
+
|
29
|
+
module EpochTime
|
30
|
+
def to_redis(value)
|
31
|
+
value.is_a?(DateTime) ? value.to_time.to_i : value.to_i
|
32
|
+
end
|
33
|
+
|
34
|
+
def from_redis(value) Time.at(value.to_i) end
|
35
|
+
end
|
36
|
+
|
37
|
+
module DateTime
|
38
|
+
def to_redis(value); value.strftime('%FT%T%z') end
|
39
|
+
def from_redis(value); value && ::DateTime.strptime(value, '%FT%T%z') end
|
40
|
+
end
|
41
|
+
|
42
|
+
module Json
|
43
|
+
def to_redis(value); Yajl::Encoder.encode(value) end
|
44
|
+
def from_redis(value); value && Yajl::Parser.parse(value) end
|
45
|
+
end
|
46
|
+
|
47
|
+
module Yaml
|
48
|
+
def to_redis(value); Yaml.dump(value) end
|
49
|
+
def from_redis(value); Yaml.load(value) end
|
50
|
+
end
|
51
|
+
|
52
|
+
module IPAddress
|
53
|
+
def from_redis(value)
|
54
|
+
return nil if value.nil?
|
55
|
+
if value.is_a?(String)
|
56
|
+
IPAddr.new(value.empty? ? '0.0.0.0' : value)
|
57
|
+
else
|
58
|
+
raise "+value+ must be nil or a String"
|
59
|
+
end
|
60
|
+
end
|
61
|
+
end
|
62
|
+
|
63
|
+
module FilePath
|
64
|
+
require 'pathname'
|
65
|
+
def from_redis(value)
|
66
|
+
value.blank? ? nil : Pathname.new(value)
|
67
|
+
end
|
68
|
+
end
|
69
|
+
|
70
|
+
module Uri
|
71
|
+
require 'addressable/uri'
|
72
|
+
def from_redis(value)
|
73
|
+
Addressable::URI.parse(value)
|
74
|
+
end
|
75
|
+
end
|
76
|
+
|
77
|
+
module Slug
|
78
|
+
require 'addressable/uri'
|
79
|
+
def to_redis(value)
|
80
|
+
Addressable::URI.parse(value).display_uri
|
81
|
+
end
|
82
|
+
end
|
83
|
+
end
|
84
|
+
end
|