redis-objects 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,154 @@
1
+ = An Atomic Rant
2
+
3
+ == Brush Up Your Resume
4
+
5
+ You are probably not handling atomic operations properly in your app, and
6
+ probably have some nasty lurking race conditions. The worst part is these
7
+ will get worse as your user count increases, are difficult to reproduce,
8
+ and usually happen to your most critical pieces of code. (And no, your
9
+ rspec tests can't catch them either.)
10
+
11
+ Let's assume you're writing an app to enable students to enroll in courses.
12
+ You need to ensure that no more than 30 students can sign up for a given course.
13
+ In your enrollment code, you have something like this:
14
+
15
+ @course = Course.find(1)
16
+ if @course.num_students < 30
17
+ @course.course_students.create!(:student_id => 101)
18
+ @course.num_students += 1
19
+ @course.save!
20
+ else
21
+ # course is full
22
+ end
23
+
24
+ You're screwed. You now have 32 people in your 30 person class, and you have
25
+ no idea what happened.
26
+
27
+ "Well no duh," you're saying, "even the {ActiveRecord docs mention locking}[http://api.rubyonrails.org/classes/ActiveRecord/Locking/Pessimistic.html],
28
+ so I'll just use that."
29
+
30
+ @course = Course.find(1, :lock => true)
31
+ if @course.num_students < 30
32
+ # ...
33
+
34
+ Nice try, but now you've introduced other issues. Any other piece of code
35
+ in your entire app that needs to update _anything_ about the course - maybe
36
+ the course name, or start date, or location - is now serialized. If you need high
37
+ concurrency, you're still screwed.
38
+
39
+ You think, "ah-ha, the problem is having a separate counter!"
40
+
41
+ @course = Course.find(1)
42
+ if @course.course_students.count < 30
43
+ @course.course_students.create!(:student_id => 101)
44
+ else
45
+ # course is full
46
+ end
47
+
48
+ Nope. Still screwed.
49
+
50
+ == The Root Down
51
+
52
+ It's worth understanding the root issue, and how to address it.
53
+
54
+ Race conditions arise from the difference in time between *evaluating* and *altering*
55
+ a value. In our example, we fetched the record, then checked the value, then
56
+ changed it. The more lines of code between those operations, and the higher your user
57
+ count, the bigger the window of opportunity for other clients to get the data in an
58
+ inconsistent state.
59
+
60
+ Sometimes race conditions don't matter in practice, since often a user is
61
+ only operating their own data. This has a race condition, but is probably ok:
62
+
63
+ @post = Post.create(:user_id => @user.id, :title => "Whattup", ...)
64
+ @user.total_posts += 1 # update my post count
65
+
66
+ But this _would_ be problematic:
67
+
68
+ @post = Post.create(:user_id => @user.id, :title => "Whattup", ...)
69
+ @blog.total_posts += 1 # update post count across all users
70
+
71
+ As multiple users could be adding posts concurrently.
72
+
73
+ In a traditional RDBMS, you can increment counters atomically (but not return them)
74
+ by firing off an update statement that self-references the column:
75
+
76
+ update users set total_posts = total_posts + 1 where id = 372
77
+
78
+ You may have seen {ActiveRecord's increment_counter class method}[http://api.rubyonrails.org/classes/ActiveRecord/Base.html#M002278],
79
+ which wraps this functionality. But outside of being cumbersome, this has
80
+ the side effect that your object is no longer in sync with the DB, so you
81
+ get other issues:
82
+
83
+ Blog.increment_counter :total_posts, @blog.id
84
+ if @blog.total_posts == 1000
85
+ # the 1000th poster - award them a gold star!
86
+
87
+ The DB says 1000, but your @blog object still says 999, and the right person
88
+ doesn't get their gold star. Sad faces all around.
89
+
90
+ == A Better Way
91
+
92
+ Bottom line: Any operation that could alter a value *must* return that value in
93
+ the _same_ _operation_ for it to be atomic. If you do a separate get then set,
94
+ or set then get, you're open to a race condition. There are very few systems that
95
+ support an "increment and return" type operation, and Redis is one of them
96
+ (Oracle sequences are another).
97
+
98
+ When you think of the specific things that you need to ensure, many of these will
99
+ reduce to numeric operations:
100
+
101
+ * Ensuring there are no more than 30 students in a course
102
+ * Getting more than 2 but less than 6 people in a game
103
+ * Keeping a chat room to a max of 50 people
104
+ * Correctly recording the total number of blog posts
105
+ * Only allowing one piece of code to reorder a large dataset at a time
106
+
107
+ All except the last one can be implemented with counters. The last one
108
+ will need a carefully placed lock.
109
+
110
+ The best way I've found to balance atomicity and concurrency is, for each value,
111
+ actually create two counters:
112
+
113
+ * A counter you base logic on (eg, +slots_taken+)
114
+ * A counter users see (eg, +current_students+)
115
+
116
+ The reason you want two counters is you'll need to change the value of the logic
117
+ counter *first*, _before_ checking it, to address any race conditions. This means
118
+ the value can get wonky momentarily (eg, there could be 32 +slots_taken+ for a 30-person
119
+ course). This doesn't affect its function - indeed, it's part of what makes it work - but
120
+ does mean you don't want to display it.
121
+
122
+ So, taking our +Course+ example:
123
+
124
+ class Course < ActiveRecord::Base
125
+ include Redis::Atoms
126
+
127
+ counter :slots_taken
128
+ counter :current_students
129
+ end
130
+
131
+ Then:
132
+
133
+ @course = Course.find(1)
134
+ @course.slots_taken.increment do |val|
135
+ if val <= @course.max_students
136
+ @course.course_students.create!(:student_id => 101)
137
+ @course.current_students.increment
138
+ end
139
+ end
140
+
141
+ Race-condition free. And, with the separate +current_students+ counter, your
142
+ views get consistent information about the course, since it will only be
143
+ incremented on success. There is still a race condition where +current_students+
144
+ could be less than the real number of +CourseStudent+ records, but since you'll be
145
+ displaying these values in a view (after that block completes) you shouldn't see
146
+ this manifest in real-world usage.
147
+
148
+ Now you can sleep soundly, without fear of getting fired at 3am via an angry
149
+ phone call from your boss. (At least, not about this...)
150
+
151
+ == Author
152
+
153
+ Copyright (c) 2009 {Nate Wiger}[http://nate.wiger.org]. All Rights Reserved.
154
+ Rant released under {Creative Commons}[http://creativecommons.org/licenses/by/3.0/legalcode].
@@ -0,0 +1,144 @@
1
+ = Redis::Objects - Lightweight, atomic object layer around redis-rb
2
+
3
+ This is *not* an ORM. People that are wrapping ORM's around Redis are missing
4
+ the point.
5
+
6
+ The killer feature of Redis that it allows you to perform atomic operations
7
+ on _individual_ data structures, like counters, lists, and sets, that you can use
8
+ *with* your existing ActiveRecord/DataMapper/etc models, or in classes that have
9
+ nothing to do with an ORM or even a database. That's where this gem comes in.
10
+
11
+
12
+ This gem originally arose out of a need for high-concurrency operations for online
13
+ games; for a fun rant on the topic, see
14
+ {ATOMICITY}[http://github.com/nateware/redis-objects/blob/master/ATOMICITY.rdoc],
15
+ or scroll down to "Atomicity" in this README.
16
+
17
+ There are two ways to use Redis::Objects, either as an +include+ in a class, or
18
+ by using +new+ with the type of data structure you want to create.
19
+
20
+ == Installation
21
+
22
+ gem install gemcutter
23
+ gem tumble
24
+ gem install redis-objects
25
+
26
+ === Initialization
27
+
28
+ # If on Rails, config/initializers/redis.rb is a good place for this
29
+ require 'redis'
30
+ require 'redis/objects'
31
+ Redis::Objects.redis = Redis.new(:host => 127.0.0.1, :port => 6379)
32
+
33
+ == Examples
34
+
35
+ === Model Class Usage
36
+
37
+ class Team < ActiveRecord::Base
38
+ include Redis::Objects
39
+
40
+ counter :drafted_players
41
+ counter :active_players
42
+ counter :total_online_players, :global => true
43
+ list :on_base
44
+ set :outfielders
45
+ value :coach
46
+ end
47
+
48
+ Familiar Ruby operations Just Work:
49
+
50
+ @team = Team.find(1)
51
+ @team.on_base << 'player1'
52
+ @team.on_base << 'player2'
53
+ puts @team.on_base # ['player1', 'player2']
54
+ @team.on_base.pop
55
+
56
+ With one purposeful exception - counters cannot be set, only incremented/decremented:
57
+
58
+ @team.drafted_players.increment # or incr
59
+ @team.drafted_players.decrement # or decr
60
+ @team.drafted_players = 4 # exception
61
+ @team.drafted_players += 1 # exception
62
+
63
+ === Instance Usage
64
+
65
+ === Counters
66
+
67
+ @counter = Redis::Counter.new('counter_name')
68
+ @counter.increment
69
+ puts @counter
70
+ puts @counter.get # force re-fetch
71
+
72
+ === Lists
73
+
74
+ @list = Redis::List.new('list_name')
75
+ @list << 'a'
76
+ @list << 'b'
77
+ puts @list
78
+
79
+ == Atomicity
80
+
81
+ You are probably not handling atomicity correctly in your app. For a fun rant
82
+ on the topic, see {ATOMICITY}[ATOMICITY.doc]
83
+
84
+ Atomic counters are a good way to handle concurrency:
85
+
86
+ @team = Team.find(1)
87
+ if @team.drafted_players.increment <= @team.max_players
88
+ # do stuff
89
+ @team.team_players.create!(:player_id => 221)
90
+ @team.active_players.increment
91
+ else
92
+ # reset counter state
93
+ @team.drafted_players.decrement
94
+ end
95
+
96
+ Atomic block - a cleaner way to do the above. Exceptions or return nil
97
+ rewind counter back to previous state:
98
+
99
+ @team.drafted_players.increment do |val|
100
+ raise Team::TeamFullError if val > @team.max_players
101
+ @team.team_players.create!(:player_id => 221)
102
+ @team.active_players.increment
103
+ end
104
+
105
+ Similar approach, using an if block (failure rewinds counter):
106
+
107
+ @team.drafted_players.increment do |val|
108
+ if val <= @team.max_players
109
+ @team.team_players.create!(:player_id => 221)
110
+ @team.active_players.increment
111
+ end
112
+ end
113
+
114
+ Class methods work too - notice we override ActiveRecord counters:
115
+
116
+ Team.increment_counter :drafted_players, team_id
117
+ Team.decrement_counter :drafted_players, team_id, 2
118
+ Team.increment_counter :total_online_players # no ID on global counter
119
+
120
+ Class-level atomic block (may save a DB fetch depending on your app):
121
+
122
+ Team.increment_counter(:drafted_players, team_id) do |val|
123
+ TeamPitcher.create!(:team_id => team_id, :pitcher_id => 181)
124
+ Team.increment_counter(:active_players, team_id)
125
+ end
126
+
127
+ Locks with Redis. On completion or exception the lock is released:
128
+
129
+ @team.reorder_lock.lock do
130
+ @team.reorder_all_players
131
+ end
132
+
133
+ Class-level lock (same concept)
134
+
135
+ Team.obtain_lock(:reorder, team_id) do
136
+ Team.reorder_all_players(team_id)
137
+ end
138
+
139
+
140
+ == Author
141
+
142
+ Copyright (c) 2009 {Nate Wiger}[http://nate.wiger.org]. All Rights Reserved.
143
+ Released under the {Artistic License}[http://www.opensource.org/licenses/artistic-license-2.0.php].
144
+
@@ -0,0 +1,105 @@
1
+ class Redis
2
+ #
3
+ # Class representing a Redis counter. This functions like a proxy class, in
4
+ # that you can say @object.counter_name to get the value and then
5
+ # @object.counter_name.increment to operate on it. You can use this
6
+ # directly, or you can use the counter :foo class method in your
7
+ # class to define a counter.
8
+ #
9
+ class Counter
10
+ attr_reader :key, :options, :redis
11
+ def initialize(key, options={})
12
+ @key = key
13
+ @options = options
14
+ @redis = options[:redis] || $redis || Redis::Objects.redis
15
+ @options[:start] ||= 0
16
+ @options[:type] ||= @options[:start] == 0 ? :increment : :decrement
17
+ @redis.setnx(key, @options[:start]) unless @options[:start] == 0 || @options[:init] === false
18
+ end
19
+
20
+ # Reset the counter to its starting value. Not atomic, so use with care.
21
+ # Normally only useful if you're discarding all sub-records associated
22
+ # with a parent and starting over (for example, restarting a game and
23
+ # disconnecting all players).
24
+ def reset(to=options[:start])
25
+ redis.set(key, to.to_i)
26
+ @value = to.to_i
27
+ end
28
+
29
+ # Re-gets the current value of the counter. Normally just calling the
30
+ # counter will lazily fetch the value, and only update it if increment
31
+ # or decrement is called. This forces a network call to redis-server
32
+ # to get the current value.
33
+ def get
34
+ @value = redis.get(key).to_i
35
+ end
36
+
37
+ # Delete a counter. Usage discouraged. Consider +reset+ instead.
38
+ def delete
39
+ redis.del(key)
40
+ @value = nil
41
+ end
42
+ alias_method :del, :delete
43
+
44
+ # Returns the (possibly cached) value of the counter. Use +get+ to
45
+ # force a re-get from the Redis server.
46
+ def value
47
+ @value ||= get
48
+ end
49
+
50
+ # Increment the counter atomically and return the new value. If passed
51
+ # a block, that block will be evaluated with the new value of the counter
52
+ # as an argument. If the block returns nil or throws an exception, the
53
+ # counter will automatically be decremented to its previous value. This
54
+ # method is aliased as incr() for brevity.
55
+ def increment(by=1, &block)
56
+ @value = redis.incr(key, by).to_i
57
+ block_given? ? rewindable_block(:decrement, @value, &block) : @value
58
+ end
59
+ alias_method :incr, :increment
60
+
61
+ # Decrement the counter atomically and return the new value. If passed
62
+ # a block, that block will be evaluated with the new value of the counter
63
+ # as an argument. If the block returns nil or throws an exception, the
64
+ # counter will automatically be incremented to its previous value. This
65
+ # method is aliased as incr() for brevity.
66
+ def decrement(by=1, &block)
67
+ @value = redis.decr(key, by).to_i
68
+ block_given? ? rewindable_block(:increment, @value, &block) : @value
69
+ end
70
+ alias_method :decr, :decrement
71
+
72
+ ##
73
+ # Proxy methods to help make @object.counter == 10 work
74
+ def to_s; value.to_s; end
75
+ alias_method :to_str, :to_s
76
+ alias_method :to_i, :value
77
+ def nil?; value.nil? end
78
+
79
+ # Math ops
80
+ %w(== < > <= >=).each do |m|
81
+ class_eval <<-EndOverload
82
+ def #{m}(x)
83
+ value #{m} x
84
+ end
85
+ EndOverload
86
+ end
87
+
88
+ private
89
+
90
+ # Implements atomic increment/decrement blocks
91
+ def rewindable_block(rewind, value, &block)
92
+ raise ArgumentError, "Missing block to rewindable_block somehow" unless block_given?
93
+ ret = nil
94
+ begin
95
+ ret = yield value
96
+ rescue
97
+ send(rewind)
98
+ raise
99
+ end
100
+ send(rewind) if ret.nil?
101
+ ret
102
+ end
103
+ end
104
+ end
105
+
@@ -0,0 +1,84 @@
1
+ class Redis
2
+ module DataTypes
3
+ TYPES = %w(String Integer Float EpochTime DateTime Json Yaml IPAddress FilePath Uri Slug)
4
+ def self.included(klass)
5
+ TYPES.each do |data_type|
6
+ if Object.const_defined?(data_type)
7
+ klass = Object.const_get(data_type)
8
+ else
9
+ klass = Object.const_set(data_type, Class.new)
10
+ end
11
+ if const_defined?(data_type)
12
+ klass.extend const_get(data_type)
13
+ end
14
+ end
15
+ end
16
+
17
+ module String
18
+ def to_redis; to_s; end
19
+ end
20
+
21
+ module Integer
22
+ def from_redis(value); value && value.to_i end
23
+ end
24
+
25
+ module Float
26
+ def from_redis(value); value && value.to_f end
27
+ end
28
+
29
+ module EpochTime
30
+ def to_redis(value)
31
+ value.is_a?(DateTime) ? value.to_time.to_i : value.to_i
32
+ end
33
+
34
+ def from_redis(value) Time.at(value.to_i) end
35
+ end
36
+
37
+ module DateTime
38
+ def to_redis(value); value.strftime('%FT%T%z') end
39
+ def from_redis(value); value && ::DateTime.strptime(value, '%FT%T%z') end
40
+ end
41
+
42
+ module Json
43
+ def to_redis(value); Yajl::Encoder.encode(value) end
44
+ def from_redis(value); value && Yajl::Parser.parse(value) end
45
+ end
46
+
47
+ module Yaml
48
+ def to_redis(value); Yaml.dump(value) end
49
+ def from_redis(value); Yaml.load(value) end
50
+ end
51
+
52
+ module IPAddress
53
+ def from_redis(value)
54
+ return nil if value.nil?
55
+ if value.is_a?(String)
56
+ IPAddr.new(value.empty? ? '0.0.0.0' : value)
57
+ else
58
+ raise "+value+ must be nil or a String"
59
+ end
60
+ end
61
+ end
62
+
63
+ module FilePath
64
+ require 'pathname'
65
+ def from_redis(value)
66
+ value.blank? ? nil : Pathname.new(value)
67
+ end
68
+ end
69
+
70
+ module Uri
71
+ require 'addressable/uri'
72
+ def from_redis(value)
73
+ Addressable::URI.parse(value)
74
+ end
75
+ end
76
+
77
+ module Slug
78
+ require 'addressable/uri'
79
+ def to_redis(value)
80
+ Addressable::URI.parse(value).display_uri
81
+ end
82
+ end
83
+ end
84
+ end